pi-vision-agents 🧩🎬

Vision agents for pi. If your pi model doesn't support images or videos, this package gives you two tools that hand the work to dedicated vision/video models:

• forward_to_pimage — describe, OCR, or analyze an image with a vision model
• forward_to_pideo — summarize/transcribe a YouTube or TikTok video

Both are file-bridge daemons: the extension writes a request, a small Python daemon picks it up, calls the provider API, and writes back the result. No terminal multiplexer required — works in any terminal where pi runs.

cmux note: you can run these in a plain pi session. The examples/ directory contains optional pane-launch scripts for cmux users who want a dedicated vision pane, but they are not required.

---

Install

pi install npm:pi-vision-agents

Then add your API keys to your shell environment (or pi's .env):

# pimage (image analysis) — OpenAI-compatible vision API
export OPENCODE_API_KEY="your-key-here"

# pideo (video analysis) — OpenRouter
export OPENROUTER_API_KEY="your-key-here"

Dependencies

Requirement	Needed for	Install
python3 (≥3.10)	both daemons	system Python
yt-dlp	pideo (video metadata + subtitles)	pip install yt-dlp or brew install yt-dlp

Both daemons start lazily on first tool call and stop on session end — no manual daemon management needed.

---

Usage

In pi, just ask:

forward_to_pimage: /path/to/screenshot.png — "Extract all text from this image"
forward_to_pideo:  https://www.youtube.com/watch?v=... — "Summarize this video"

The extension also auto-detects when you attach an image in a non-vision model session and tells the model to use forward_to_pimage.

---

Configuration

Everything is env-var driven. No hardcoded paths, no baked-in secrets.

Env var	Default	Purpose
PIMAGE_DIR	~/.pimage	pimage state dir (inbox/outbox/signal)
PIMAGE_MODEL	opencode-go/mimo-v2.5	Vision model
PIMAGE_API_BASE	https://opencode.ai/zen/go/v1	OpenAI-compatible API base
PIMAGE_DAEMON	package daemons/	Explicit daemon path override
PIMAGE_TIMEOUT_MS	120000	Response timeout
OPENCODE_API_KEY	—	Required for pimage
PIDEO_DIR	~/.pideo	pideo state dir
PIDEO_MODEL	google/gemini-2.5-flash	Video analysis model
PIDEO_API_BASE	OpenRouter chat completions	API base URL
PIDEO_DAEMON	package daemons/	Explicit daemon path override
PIDEO_TIMEOUT_MS	300000	Response timeout
OPENROUTER_API_KEY	—	Required for pideo

---

How it works

pi session                          python daemon
┌──────────────────┐   file write   ┌──────────────────┐   HTTPS   ┌─────────────┐
│ forward_to_pimage│ ─────────────▶ │ pimage-daemon.py │ ────────▶ │ vision API  │
│ extension        │ ◀───────────── │ polls ~/.pimage  │ ◀──────── │ (OpenAI-    │
│ polls result     │   result.json  │                  │  result   │  compatible)│
└──────────────────┘                └──────────────────┘           └─────────────┘

1. Tool call writes {id, image_path/url, instruction} to $DIR/in/<uuid>/
2. Signal file $DIR/current wakes the daemon
3. Daemon calls the provider (base64 image / yt-dlp transcript)
4. Result written to $DIR/out/<uuid>/result.json, extension returns it

---

Security & privacy

• No secrets in the package. All API keys come from your environment.
• No PII. The package collects nothing — requests are only sent to the provider API you configure.
• Files stay local. Images/transcripts pass through your local state dir and are deleted after processing ($DIR/in/<uuid>/ is cleaned up on success/timeout).
• Your data goes to whichever provider you configure (OPENCODE_API_KEY provider for pimage, OpenRouter for pideo). Review their policies.

---

Development

git clone https://github.com/ezadEzanee/pi-vision-agents
# try locally without installing:
pi -e ./path/to/pi-vision-agents

Run a daemon manually for debugging:

PIMAGE_DIR=/tmp/pimage-test OPENCODE_API_KEY=... python3 daemons/pimage-daemon.py
PIDEO_DIR=/tmp/pideo-test OPENROUTER_API_KEY=... python3 daemons/pideo-daemon.py

---

License

MIT