@getpipher/vision

Capability-aware vision + paste extension for the pi coding agent.

@getpipher/vision adds a describe_image tool to pi that is aware of the active primary model's input modalities — so image analysis is never wasteful.

Why

Two community pi extensions — pi-vision-tool and pi-paster — are load-bearing but leave real workflow friction. The biggest: pi-vision-tool always delegates image analysis to a second vision model, even when the active primary model is multimodal and can see images natively. That's a wasteful double-call: extra API roundtrip, extra tokens, and a hardcoded description that's worse than the model's own native image understanding.

@getpipher/vision fixes this with capability-aware delegation:

• Multimodal primary model (e.g. minimax-m3:cloud, qwen3.5:cloud) → the image passes through to the model natively. Zero delegation calls, zero extra tokens, zero extra latency. The describe_image tool is hidden from the model entirely, so the wasted call is structurally impossible — not just discouraged.
• Text-only primary model → describe_image delegates to a configured vision model and returns the text description (the existing, working pattern).

The capability check is automatic and silent — no config, no opt-in, no flag. It just works under the hood.

Install

pi install npm:@getpipher/vision

@getpipher/vision replaces pi-vision-tool and pi-paster. Uninstall those first to avoid duplicate tools and double-attachment:

pi uninstall npm:pi-vision-tool
pi uninstall npm:pi-paster

Configure

The vision tool needs a configured vision model for DELEGATE mode (text-only primaries). Run /vision to open an interactive settings panel (pi's native SettingsList — the same engine /settings uses): arrow keys navigate, Enter cycles a value or opens the vision-model picker, Escape exits. Changes apply live.

/vision

Or use the typed subcommands (power users / scripts):

/vision provider ollama
/vision model minimax-m3:cloud

The provider and model must be defined in ~/.pi/agent/models.json and the model should have "input": ["text", "image"]. Other /vision subcommands:

Subcommand	Purpose
/vision (no arg)	Open the interactive settings panel (like /settings)
/vision show	Display the current config as text
/vision model (no arg)	Open a picker of vision-capable authed models
/vision on / /vision off	Enable / disable the tool
/vision provider <name>	Set the vision provider
/vision model (no arg)	Open a picker of vision-capable authed models
/vision model <id>	Set the vision model id (typed fallback)
/vision max-dim <px>	Max image dimension for compression (1–8000)
/vision quality <1-100>	JPEG re-encode quality
/vision reasoning-effort <off|minimal|low|medium|high|xhigh>	Default reasoning effort for delegation
/vision system-prompt [<text>|clear]	Set/clear a custom system prompt for the vision model (no arg → multi-line editor)
/vision cache <clear|show>	Clear the cache or show stats (memory + disk entries)
/vision fallback <provider/model>|clear	Set/clear a fallback vision model
/vision clear	Reset config to defaults
/vision-use [provider/model]	Switch the DELEGATE vision model inline (no arg → picker). Hotkey: ctrl+shift+i (rebindable via keybindings.json; on Mac, alt-based combos need macos-option-as-alt=true)
/vision paste-mode [hint|auto|off]	Set how pasted images are handled on a text-only primary (no arg → cycle).
/vision marker-style [code|bold|plain]	Set the markdown style for [Image-#N] markers (no arg → show current).
/vision auto-prompt [<text>|clear]	Set/clear the generic auto-delegation prompt (no arg → multi-line editor).
/vision preview <path>	Open a full-screen TUI preview of an image (Kitty/iTerm2 graphics, text fallback on tmux).
/vision batch-concurrency [<1-20>]	Max parallel image delegations in a batch (describe_image image_paths + paste auto mode). 1 = serial; 20 = aggressive. Default 5.
/vision local-only [on|off]	Toggle local-only mode (no arg → show current). When on, image bytes never leave the machine — cache hits still work, a cache miss refuses with a clear error instead of a network call.
/vision audit <clear|show|path|on|off>	Audit log management. clear truncates; show tails the last 10 entries + count; path prints the resolved path; on/off toggle logging. Default on.
/vision audit-path [<path>|clear]	Set/clear a custom audit log path (power users; no arg → show current). Default ~/.pi/agent/vision-audit.log.

Config is stored at ~/.pi/agent/vision.json (not vision-tool.json, so it doesn't collide with the community package during transition).

Resilience (v0.2.0)

DELEGATE mode (text-only primary) is resilient + cheap:

• Caching. Successful delegation results are cached by a content-addressed key (image hash + compression params + prompt + vision model + reasoning). A second call on the same image costs zero vision-model API calls. In-memory by default; opt into cross-session persistence with Persist cache to disk (LRU-evicted at the configured max entries). Only successes are cached — failures never are. /vision cache clear wipes both layers.
• Retry + fallback. On a retryable failure (HTTP 5xx, 429, network), the primary vision model is retried with exponential backoff (abort-aware — a cancelled turn stops retrying immediately). On a non-retryable error or exhausted retries, a configured fallback vision model is tried once. Configure both via the /vision panel or /vision fallback <provider/model>.
• Custom system prompt. A per-workflow framing prepended to the vision-model request (/vision system-prompt <text>, or the panel row).
• Inline model switch. ctrl+shift+i (or /vision-use) switches the DELEGATE vision model mid-session without opening the full panel.

Paste UX (v0.3.0)

When you reference an image file path in your message, the paste hook makes it visible and actionable — capability-aware for both multimodal and text-only primaries.

• [Image-#N] markers. Image file paths in your message are replaced with [Image-#N] markers (sequential, 1-indexed). The marker style is configurable: code (inline code — default, visually distinct), bold, or plain. On a multimodal primary, the image is also attached as a native attachment (zero delegation). On a text-only primary, the marker is informational — the image is NOT attached (text-only models can't process images).
• Text-only paste modes. When the primary model is text-only, pasted images can't be processed natively. Three configurable modes:
  • hint (default): markers + a hint line nudging the model to call describe_image. Zero tokens — the model decides whether to delegate.
  • auto (opt-in): auto-delegates each image via the v0.2.x pipeline (cache/retry/fallback) and appends the descriptions to your message. Timeout-protected (own AbortController, default 30s) — falls back to hint on timeout or failure.
  • off: markers only — no attachment, no hint, no delegation.

Image Preview (v0.3.3)

@getpipher/vision can render images directly in the terminal — no external viewer needed.

Compose-time auto-preview. As you type a message referencing an image path, the image renders above the editor (WhatsApp/Telegram compose-box style) ~300ms after you stop typing. Clears when you remove the path or submit the message. Enable/disable via the /vision panel (Compose preview row) or the config field composePreview.

On-demand preview. /vision preview <path> opens a full panel showing the image + metadata (filename, dimensions, MIME, file size, detected protocol). Useful for checking an image before deciding to analyze it.

Terminal support:

Terminal	Image rendering
Kitty, Ghostty, WezTerm, Warp (standalone)	✅ Real graphics (Kitty protocol)
iTerm2 (standalone)	✅ Real graphics (iTerm2 protocol)
tmux (any terminal)	Text fallback ([Image: filename image/png WxH])
VSCode, Alacritty, other	Text fallback

The text fallback still shows useful metadata (filename, dimensions, format, file size) and confirms the image was found. Real graphics require running pi outside tmux on a graphics-capable terminal.

Batch + scale (v0.4.0)

describe_image accepts multiple images in a single call via image_paths (alongside the single image_path for back-compat). For text-only primaries (where the tool is visible), this lets the model analyze, compare, or cross-reference several images in one tool call instead of N serial round-trips:

describe_image(
  image_paths: ["/tmp/before.png", "/tmp/after.png", "/tmp/diff.png"],
  prompt: "Compare these three screenshots. What changed?"
)

Delegations run in parallel, bounded by batchConcurrency (default 5, configurable 1–20 via /vision batch-concurrency). 1 = serial escape hatch; 20 = aggressive (rate-limit risk is yours). Each image reuses the v0.2.x resilience pipeline (cache/retry/fallback) independently — a cache hit returns 0 vision-model calls, a failed image becomes an [error: …] section rather than failing the whole batch, and isError is set only if every image failed. The result is one structured, order-stable text block:

[Batch: 3 image(s)]

[Image 1] /tmp/before.png
<vision model description>

[Image 2] (cached) /tmp/after.png
<vision model description>

[Image 3] /tmp/diff.png
[error: not_found — image not found at /tmp/diff.png]

A hard cap of 50 images per call (MAX_BATCH_IMAGES) defends against an over-eager model passing an absurd array; split across calls if you need more.

Parallel auto-delegation (paste auto mode). When textOnlyPasteMode is "auto" and you paste multiple image paths, delegations now run in parallel (one batch-level timeout = the total budget, bounded by batchConcurrency) instead of serially — so a 5-screenshot paste completes in ~ceil(N/c) × per-call instead of N × per-call.

Hint mode now exposes paths. In text-only + "hint" mode (the default), the hint line now lists the image paths and names the image_paths batch affordance, so the model can actually invoke describe_image (previously the hint named the tool but the path markers erased the paths, leaving the model unable to call it). Paste 2+ images and the model learns it can pass them all to image_paths for batch analysis.

Clipboard paste just works. Pi binds ctrl+v (alt+v on Windows) to paste the system clipboard image: it reads the clipboard, writes the bytes to /tmp/pi-clipboard-<uuid>.<ext>, and inserts that path at the cursor. Our existing path-token pipeline detects it, renders a [Image-#N] marker, and attaches (multimodal) or delegates (text-only) — no separate clipboard code path needed. Multi-image clipboard = N ctrl+v presses = N paths = handled as a batch.

Config + security (v0.5.0)

v0.5.0 makes the tool trustable — audit where your images go, opt out of network delegation entirely, and get workflow-fit defaults out of the box.

Audit log

Every vision-model delegation (success, cache hit, fallback, or failure) is recorded in a persisted, append-only JSONL log at ~/.pi/agent/vision-audit.log (one line per delegation). The log answers "where did each image go?" without storing the image bytes or the full prompt.

Each line is a JSON object:

Field	Meaning
ts	ISO 8601 timestamp of the delegation event
provider	The configured primary provider (the attempted route)
model	The model that actually responded (the fallback on a fallback success)
image_path	The path the user passed (file paths full; data:URL/base64 truncated)
source_hash	SHA-256 of the original image bytes (a content fingerprint — not the bytes)
cached	true if served from the cache (0 network calls)
fallback	true if the result came from the fallback vision model
fallback_model	The fallback model id, if fallback was used
ok	true if the delegation succeeded
error_code	Error code on failure (local_only, vision_call_error, aborted, …)
latency_ms	Round-trip latency (0 for a cache hit / local-only refusal)
local_only	true if local-only mode was active

Privacy stance: the log records routing (where bytes went), never content. Image bytes are never logged (only the source_hash fingerprint); the full prompt is never logged (the conversation log already has it). A data: URL or raw base64 image_path is truncated to the first 64 chars + a size suffix. The audit log is on by default (opt-out, not opt-in — the security posture is traceability without you remembering to enable it):

/vision audit off        # disable logging
/vision audit show       # tail the last 10 entries + total count
/vision audit clear      # truncate the log
/vision audit path       # print the resolved log path
/vision audit-path /tmp/my-vision.log  # custom location (power users)

Local-only mode

When local-only mode is on, image bytes never leave the machine. This is a structural guarantee, not a polite request — the network-call code path is never entered. Cache hits still work (the cache is local — memory + disk under ~/.pi/agent), so local-only mode is effectively "cache-only mode": previously- seen images get their cached descriptions, new images refuse with a clear error.

/vision local-only on    # refuse all network delegation (cache hits OK)
/vision local-only off   # allow delegation

In paste auto mode + local-only, the hook skips delegation entirely and falls straight to the hint line (no autoDelegateTimeoutMs burned waiting for a refused call to abort). The describe_image tool stays visible so the model can still retrieve cached descriptions and report the local-only error on a cache miss. Every local-only event is audited with local_only: true (greppable proof: grep '"local_only":true' ~/.pi/agent/vision-audit.log).

Workflow-fit defaults (auto-detect)

On a fresh install (or after /vision clear), the extension auto-detects the vision model at session start from ~/.pi/agent/models.json — aligned to the AGENTS.md LLM-backend policy of "Ollama Cloud primary + frontier escalation":

• Primary: prefers the Ollama provider's vision-capable models (the :cloud ones — Ollama Cloud, flat-rate + private). On RECTOR's setup this picks Ollama/minimax-m3:cloud (first by sorted id).
• Fallback: NOT auto-populated (v0.5.1). The frontier-escalation mechanism (the fallback field + the retry→fallback pipeline) is still there — set it explicitly via /vision fallback <provider/model> if you want a secondary vision model tried when the primary fails.
• The auto-detected primary is persisted once to vision.json (you see the choice + can override it). /vision clear resets to defaults and re-triggers detection on the next session start.
• Escape hatch: /vision auto-detect off (set autoDetectVisionModel: false) disables auto-detection entirely — a fresh config stays unconfigured.

Auto-detect only fires when both provider and model are unset (a truly fresh state). A partial config (one set, one blank) is you mid-configuration — it's not overwritten.

No config migration from pi-paster

pi-paster is a paste-format extension — it has no configurable vision model (it doesn't delegate; it just marks + attaches). @getpipher/vision's markerStyle (code/bold/plain) is a superset of pi-paster's fixed format, so there's nothing to migrate. Uninstall pi-paster, run /vision, done.

How it works

Two mechanisms combine to guarantee the behavior:

1. Conditional tool availability (mechanism A). describe_image is always registered, but its visibility to the LLM is toggled with pi.setActiveTools() based on the active model's input capability. On a multimodal primary, the tool is removed from the active set — the LLM never sees it, so it can never waste a call. On a text-only primary, the tool is added back so the LLM can delegate. The model_select event re-syncs on mid-session /model switches.

2. Capability-aware paste hook. When the primary model is multimodal, the input hook detects image file paths in your message, attaches them as native image content, and renders [Image-#N] markers (mechanism B-lite, graduated in v0.3.0). On a text-only primary, the hook renders markers but does NOT attach images — instead it hints, auto-delegates, or stays silent based on textOnlyPasteMode. The hook dedups by data hash so it coexists cleanly with other paste extensions during transition.

Using describe_image

The tool accepts a file path, data URL, or raw base64. For a single image:

describe_image(image_path: "/tmp/screenshot.png", prompt: "What's in this image?")

For multiple images (batch — parallel delegation, one structured result):

describe_image(
  image_paths: ["/tmp/a.png", "/tmp/b.png", "/tmp/c.png"],
  prompt: "Compare these screenshots. What changed between them?"
)

Parameters:

Param	Type	Description
image_path	string?	Path to a single image, data: URL, or raw base64. Use for one image.
image_paths	string[]?	Multiple paths to analyze together (comparison/cross-reference). Up to 50.
prompt	string	What to analyze or answer about the image(s). For a batch, applies to each; describe what to compare.
compress	boolean?	Optimize the image(s) before delegation (default true)
reasoning	enum?	Reasoning effort for the delegation (off…xhigh)

When caching or fallback is active, the tool result details include cached: true (cache hit) and fallback: true (result from the fallback model) for traceability. For a batch, details.batch is an array of per-image results (index, path, ok, cached, fallback, errorCode) in input order.

For multimodal primaries you don't call describe_image — just reference the image path in your message and the model sees it natively.

Credits

Inspired by pi-vision-tool (xezpeleta) and pi-paster (beowulf11) — both filled this gap first and motivated this clean-room reimplementation. Superseded by @getpipher/vision.

This package contains no code copied from either project. The behavior is specified independently and implemented fresh.

License

MIT