pi-vision-route

Zero-switch image routing for text-only models in the pi coding agent.

Many strong coding models are text-only (GLM, DeepSeek, Kimi, ...). Send them an image and you get a 400, a silent hang, or — on some Anthropic-compat gateways — the image silently degraded to a URL placeholder the model never sees, inviting confident hallucination. This extension removes the manual "switch to a multimodal model before showing a screenshot" dance entirely:

• Fires on pi's context event (before every LLM call).
• If the current model can see images → pass through untouched, zero overhead.
• If it cannot → every image block in the conversation (user attachments and read-tool results — pi's main image entry point) is transcribed out-of-band by a VLM and swapped for its text transcription. The text model then answers normally.

You keep working in your favorite text model; images just work.

Install

pi install npm:pi-vision-route

Set one environment variable (the default chain needs a Zhipu key):

export ZAI_API_KEY=...            # default chain; or use VISION_ROUTE_API_KEY

Configuration

All optional. Unset = defaults below.

Env var	Default	Purpose
VISION_ROUTE_API_KEY	falls back to ZAI_API_KEY	API key for the VLM chain
VISION_ROUTE_VLM_CHAIN	Zhipu GLM chain (below)	JSON array [{url, model, timeoutMs?}] — any OpenAI-compatible /chat/completions endpoint works
VISION_ROUTE_PROMPT	strict OCR prompt (below)	Transcription prompt override

Default VLM chain

[
  { "url": "https://open.bigmodel.cn/api/coding/paas/v4/chat/completions", "model": "glm-4.6v", "timeoutMs": 120000 },
  { "url": "https://open.bigmodel.cn/api/paas/v4/chat/completions", "model": "glm-4v-plus", "timeoutMs": 60000 }
]

Primary first (subscription quota), pay-per-use fallback. To route through any other OpenAI-compatible vision model:

export VISION_ROUTE_VLM_CHAIN='[{"url":"https://api.deepseek.com/chat/completions","model":"your-vlm","timeoutMs":60000}]'

An invalid or empty chain logs a warning and falls back to the default.

Default prompt

Strict and OCR-oriented; the transcription answers in the image's own language. Override VISION_ROUTE_PROMPT to force a fixed language or detail level. The transcription is what your text model actually sees — tune it like any prompt.

How it earns its latency

• Parallel transcription — N images in one context are transcribed concurrently, not N serial VLM round-trips.
• In-flight dedup — the same image often appears twice in one context (user attachment + read toolResult); it is transcribed and billed once.
• Content-hash cache — the context event refires before every LLM call; repeat calls over the same image are served from cache, never re-billed.
• Fallback chain — primary VLM timeout/failure rolls to the next entry before giving up; failure degrades to an explicit note telling the model to inform the user, never a fabricated description.

Limitations

• The text model sees a description, not pixels — fine for screenshots/receipts/docs, not for pixel-perfect UI critique.
• First sight of an image adds VLM latency (typically 5–20 s) to that turn.
• Transcription cost follows your VLM pricing; the cache keeps repeat calls free.

Changelog

1.0.1

• Fix: drop empty text parts after image→text swap. An image-only user message leaves an empty-string text part next to the transcription; Zhipu/GLM upstream rejects messages containing empty text parts (1210/1214), so the swap itself caused the 400 it was meant to prevent. Gemini tolerates them, which is why it only surfaced on GLM paths.
• package.json: declare "type": "module" (removes Node's MODULE_TYPELESS_PACKAGE_JSON reparse warning when loading the extension).

Uninstall

pi remove npm:pi-vision-route

License

MIT