pi-comfyui-paint

ComfyUI image/video generation extension for pi.

Install

pi install npm:pi-comfyui-paint

Or install a pinned version:

pi install npm:pi-comfyui-paint@0.3.0

Development/git install:

pi install git:github.com/MacroSony/pi-comfyui-paint@v0.3.0

Configuration

Configuration is resolved in this order (highest priority first):

1. COMFYUI_* environment variables
2. Project config: <project>/.pi/comfyui-paint.json
3. Global config: ~/.pi/agent/comfyui-paint.json
4. Defaults

JSON config keys use camelCase equivalents, for example:

{
  "backends": [
    {
      "id": "gpu-a",
      "url": "http://127.0.0.1:8188",
      "capabilities": ["video", "h3", "image", "krea"]
    },
    {
      "id": "gpu-b",
      "url": "http://127.0.0.1:8189",
      "capabilities": ["image", "anima"]
    }
  ],
  "outputDir": ".pi/paint-jobs",
  "syncTimeoutSeconds": 600,
  "interruptOnAbort": false,
  "jobIdStyle": "timestamp",
  "reconcileIntervalSeconds": 30,
  "backendOutputDirs": {
    "gpu-a": "/home/me/ComfyUI/output",
    "gpu-b": "/home/me/ComfyUI/output"
  }
}

Backend capabilities are declared in JSON config only (the flat COMFYUI_BACKENDS=id=url env form cannot carry a list). A backend without a capabilities field accepts every workflow; capabilities: [] accepts nothing (a soft-disable). Workflows declare their required tags with a [CAPABILITY] marker node (see below), and paint only auto-selects among backends that offer every required tag — an explicit backend: that cannot run the workflow fails fast with a clear message.

Relative paths in a project config resolve from the project root; relative paths in the global config resolve from the user home directory. Environment variables remain the override mechanism for one-off runs.

Env var	JSON key	Default	Description
COMFYUI_URL	url	http://127.0.0.1:8188	ComfyUI server URL. https:// URLs are supported; legacy host:port values are treated as http://host:port.
COMFYUI_BACKENDS	backends	(unset)	Named ComfyUI backends in id=url,id=url form, or a JSON [{id,url}] array. When set, this replaces COMFYUI_URL for generation and enables least-queued direct assignment. Capability tags are JSON-config-only: add "capabilities": ["video", "h3"] to a backends entry (see above).
COMFYUI_WORKFLOW_DIR	workflowDir	(auto)	Custom workflow directory
COMFYUI_OUTPUT_DIR	outputDir	<temp>/pi-comfyui-paint-<user>	Root for private job records and output folders. Relative paths are resolved from the project directory. Use a persistent location when job recovery must survive OS temp cleanup.
COMFYUI_OUTPUT_RETENTION_HOURS	outputRetentionHours	168	Delete terminal extension-managed jobs/outputs after this many hours. Active and uncertain jobs are retained. Set to 0 to disable extension cleanup.
COMFYUI_SYNC_TIMEOUT_SECONDS	syncTimeoutSeconds	600	Maximum synchronous wait. On timeout, ComfyUI keeps running and paint returns a durable job ID instead of losing the generation.
COMFYUI_INTERRUPT_ON_ABORT	interruptOnAbort	off	Set to 1, true, yes, or on to attempt targeted job cancellation when a synchronous paint call is cancelled. By default, cancellation only stops Pi from polling; ComfyUI continues running.
COMFYUI_INLINE_IMAGE_LIMIT	inlineImageLimit	1	Number of generated images returned to the model as inline previews. Clamped to 0–4; set to 0 for path-only results.
COMFYUI_IMAGE_QUALITY	imageQuality	80	Initial JPEG quality for inline previews, clamped to 1–100. Quality and dimensions are reduced further when needed to meet byte limits.
COMFYUI_IMAGE_MAX_DIMENSION	imageMaxDimension	2000	Maximum width or height of an inline preview.
COMFYUI_IMAGE_MAX_BYTES	imageMaxBytes	4718592	Maximum base64-encoded bytes for one inline preview (4.5 MiB).
COMFYUI_IMAGE_TOTAL_MAX_BYTES	imageTotalMaxBytes	8388608	Maximum base64-encoded bytes across all previews returned by one paint call (8 MiB).
COMFYUI_JOB_ID_STYLE	jobIdStyle	timestamp	Durable job ID format: timestamp (YYYYMMDD-HHMMSSZ-xxxxxx) or legacy uuid.
COMFYUI_BACKEND_OUTPUT_DIRS	backendOutputDirs	(unset)	Optional local/mounted ComfyUI output directories in id=/path,id=/path form, used to recover job-scoped outputs if ComfyUI history is lost.
COMFYUI_RECONCILE_INTERVAL_SECONDS	reconcileIntervalSeconds	30	Background sweep interval for non-terminal durable jobs. Set to 0 to disable the sweeper.

Every generated original is returned by local path. Up to the configured number of image outputs are also returned as bounded JPEG previews so the agent can inspect them without another read call. Videos and other non-image outputs are path-only. Preview processing never modifies the originals and does not make an otherwise successful generation fail.

The default output root is user-specific and private. Each job uses a short timestamped subdirectory with user-only directory/file permissions where the platform supports POSIX modes. Paths are temporary by default; configure COMFYUI_OUTPUT_DIR or outputDir for durable recovery across OS temp cleanup.

Background Jobs

Pass background: true to paint for long generations such as H3 video workflows. The tool returns after ComfyUI accepts the prompt, including a durable jobId and the assigned backend. The accepted prompt lives in ComfyUI's native queue and continues if Pi exits.

Use paint_job_status with the job ID to check progress. Once ComfyUI finishes, that tool persists an output manifest before streaming outputs to private local files. Repeated status calls reuse already-downloaded files. If ComfyUI's in-memory history is lost after a restart, the saved manifest can still drive /view downloads. New jobs also write server outputs under a job-scoped paint/<jobId>/ prefix; with backendOutputDirs configured for local or mounted backends, status can rebuild a manifest from that prefix even if no manifest was captured in time.

A background reconciler also sweeps non-terminal jobs every reconcileIntervalSeconds while Pi is running. This shrinks the window between ComfyUI completion and manifest persistence. paint_job_cancel uses ComfyUI's atomic per-job cancellation API when available. On older backends it can safely remove pending prompts, but it deliberately leaves running work alone because the legacy /interrupt endpoint is backend-wide; paint_interrupt remains the explicit escape hatch.

Submission failures with an uncertain outcome are recorded but never retried automatically, preventing duplicate expensive generations.

Multiple Backends

Configure named backends with:

COMFYUI_BACKENDS="gpu-a=http://gpu-a:8188,gpu-b=http://gpu-b:8188"

For each generation, the extension queries all native queues concurrently and directly submits to the reachable backend with the fewest running, pending, and locally-reserved submissions. Ties rotate within the Pi process. If every backend is busy, the job is still assigned immediately to the shortest native queue, so ComfyUI—not the Pi process—durably owns the wait.

Automatic selection initially assumes the backends have compatible models, custom nodes, and workflows. Pass backend: "gpu-a" to paint to force one server. Backend-aware tools such as paint_get_models, paint_get_details, and paint_interrupt also accept a backend ID; paint_interrupt requires one when multiple backends are configured.

Once backends declare capabilities, automatic selection filters to the capable subset first, then picks the least-queued one among them. paint_get_details reports the workflow's required tags and whether the requested backend can run it; paint_validate_workflow accepts a backend argument for the same fit check; paint_server_status lists each backend's capabilities and every workflow's requirements.

Workflow Resolution

Workflows are resolved in this order:

1. COMFYUI_WORKFLOW_DIR env var or workflowDir JSON config (if set)
2. .pi/comfyui_workflows/ in your project root
3. workflows/ bundled with this package (per-file fallback)

Resolution is per file: a workflow name is first looked up in the active directory, then falls back to the bundled directory. This means bundled workflows (T2I_Anime_Anima.json, T2I_Anime_Anima_hires.json, I2I_General_QwenImageEdit.json, …) are usable by name directly — no copy step required. A same-named file in the project directory always wins.

Place your own .json workflow files in any of these locations. To customize a bundled workflow, copy the .json (and its *.loras.json sidecar, if any) into .pi/comfyui_workflows/ and edit the copy.

Tools

Tool	Description
paint_list_workflows	List available workflow JSON files with a one-line summary each (variables, file slots, LoRA slots, outputs)
paint_get_details	Inspect a workflow's variables, notes, outputs, file slots, and LoRA metadata
paint_validate_workflow	Validate a workflow's JSON structure and pi-comfyui-paint annotations
paint_server_status	Check every configured backend, queue, durable jobs, and effective extension configuration
paint_get_models	Query a selected ComfyUI backend for available models (checkpoints, LoRAs, etc.)
paint_interrupt	Interrupt the current generation on a selected backend (backend-wide escape hatch)
paint	Generate synchronously or submit a durable background job, with automatic or explicit backend selection
paint_job_status	List recent durable jobs or reconcile one job and retrieve completed outputs
paint_job_cancel	Target one recorded prompt for safe cancellation without interrupting unrelated work
paint_search_danbooru_tags	Search Danbooru to confirm tags and find related tags (supports multiple queries)

paint_search_danbooru_tags defaults to wildcard tag-name search. Pass mode: "related" to use Danbooru's related-tag endpoint for tags that commonly appear with a tag or search; optional related-mode parameters include categories, order, search_sample_size, and tag_sample_size. The tool warns when an input is not exact Danbooru tag spelling, and reports Danbooru request failures separately from successful empty results.

If your environment uses HTTP_PROXY or HTTPS_PROXY for outbound access, Node may require NODE_USE_ENV_PROXY=1 for Danbooru requests to use those proxy settings.

Skills

This package ships the pi-comfyui-paint-custom-workflow skill. It teaches the agent how to write and run custom ComfyUI API-format workflow JSONs through paint: the [VAR] / [OUTPUT] / [FILE] / [LORA] annotation system, the write → validate (paint_validate_workflow) → inspect (paint_get_details) → generate loop, a two-pass hires-fix walkthrough (bundled as T2I_Anime_Anima_hires.json), and debugging tips. The skill loads on demand when you ask the agent to build or tweak a custom workflow — no setup needed.

ComfyUI Custom Node Dependencies

Most bundled workflows only require standard ComfyUI nodes plus the models listed in paint_get_details. LoRA-enabled workflows that use Power Lora Loader (rgthree) require rgthree/rgthree-comfy to be installed in your ComfyUI custom_nodes/ directory.

Power Lora Loader (rgthree) is preferred for LoRA workflows because it can load multiple LoRAs in one node, avoiding the need to manually edit workflows when combining style, character, detail, or concept LoRAs.

Workflow Format

Workflow JSONs use _meta.title annotations:

• [VAR] Name — Customizable variable (exposed as a prompt parameter)
• [NOTE] — Documentation shown in paint_get_details
• [OUTPUT:type] — Tagged output node
• [CAPABILITY] — Marker node: a PrimitiveStringMultiline whose title is exactly [CAPABILITY] and whose value input holds the comma-separated tags (e.g. image, anima) this workflow requires for backend selection. A workflow is only auto-assigned to backends that offer every tag; workflows without a [CAPABILITY] node run on any backend. Recommended vocabulary: domain tags (image, video, audio, anime, upscale) plus model/pack tags (h3, krea, anima).
• [FILE:type:order] — Input file slot for paint.input_files. Files are matched by media type, not strict position: [FILE:image:1] + [FILE:audio:13] can both be filled with only two entries, skipping all slots in between.
• [FILE:type:order:optional] — Optional input file slot. When no input_files entry covers it, the node is removed from the graph (and all downstream links to it are stripped) instead of failing on its placeholder default. Use this for optional image inputs like MiniMax H3 first_frame/last_frame/ref_image_N — one workflow can serve t2v/i2v/fl2v depending on how many files are passed.
• [LORA:slot] — LoRA loader slot for paint.loras overrides. Intended for Power Lora Loader (rgthree) nodes.

For workflows with [FILE:type:order] nodes, pass local files to paint as input_files. Each entry is a bare path or an object; files are routed to the lowest-numbered uncovered slot of their type, so slots never need to be filled in order:

{
  "workflow": "minimax_h3.json",
  "prompt": "...",
  "input_files": [
    "first_frame.png",
    { "path": "bgm.mp3", "type": "audio" },
    { "path": "ref.png", "slot": 9 }
  ]
}

• "ref.png" — type inferred from the extension (png/jpg/webp/gif/… → image; mp4/webm/mov → video; mp3/wav/flac/… → audio; unknown extensions match any slot)
• { "path": "bgm.mp3", "type": "audio" } — explicit type (image / video / audio / file), routed to the first uncovered slot of that type
• { "path": "ref.png", "slot": 9 } — pin an exact [FILE:type:order] slot; the file's type must match that slot's expected type (or be "file"), otherwise paint errors instead of silently mis-feeding a node

This lets one workflow serve sparse inputs without editing it: a MiniMax H3 node with 9 image + 3 video + 3 audio slots takes [{ "path": "first.png", "type": "image" }, { "path": "bgm.mp3", "type": "audio" }] and only those two slots are filled — no dummy files needed. Uncovered :optional slots are disconnected from the graph; uncovered required slots keep their defaults (with a warning). Bare paths preserve legacy positional behavior for single-slot and all-image workflows. Relative paths are resolved from the current project directory, uploaded to ComfyUI, and inserted into the annotated nodes.

LoRA Workflows

LoRA-enabled workflows should use Power Lora Loader (rgthree) and annotate each loader with a simple slot name:

[LORA:base_style] Power Lora Loader (rgthree)
[LORA:hires_detail] Power Lora Loader (rgthree)
[LORA:inpaint_character] Power Lora Loader (rgthree)

paint_get_details detects these slots and returns LoRA slot info together with workflow variables, notes, outputs, and sidecar metadata.

Optional LoRA metadata can be stored next to a workflow:

.pi/comfyui_workflows/T2I_Anime_Anima_lora.json
.pi/comfyui_workflows/T2I_Anime_Anima_lora.loras.json

Example sidecar entry:

{
  "file": "anima/[Style]saio_ga_ushi_v1.safetensors",
  "displayName": "Saio ga Ushi Style",
  "activationPrompt": "@saio ga ushi",
  "defaultStrength": 0.7,
  "description": "Artist/style LoRA for Anima. Add the activation tag to the prompt when you want this style."
}

Use LoRA overrides in paint like this:

{
  "workflow": "T2I_Anime_Anima_lora.json",
  "prompt": "masterpiece, best quality, score_7, safe, 1girl, @saio ga ushi, smile",
  "loras": {
    "base_style": {
      "file": "anima/[Style]saio_ga_ushi_v1.safetensors",
      "strength": 0.7
    }
  }
}

To load multiple LoRAs into one Power Lora Loader slot, use an array:

{
  "loras": {
    "base_style": [
      { "file": "anima/[Style]saio_ga_ushi_v1.safetensors", "strength": 0.7 },
      { "file": "anima/[Detail]some_detail_lora.safetensors", "strength": 0.35 }
    ]
  }
}

LoRA overrides replace the contents of the named slot. Activation tags are not added automatically; put them in prompt yourself based on paint_get_details metadata.

Path separators: Always use forward slashes (/) in file values, matching ComfyUI's lora_name list (e.g. Krea2/KNPV3.safetensors). Backslash paths are normalized to forward slashes automatically, but prefer the canonical form. Slot names come from paint_get_details (LoRA slots); valid file names come from paint_get_models (LoRA category) or the "Usable LoRA metadata" in paint_get_details. If a LoRA's sidecar metadata declares an activationPrompt, copy that tag into prompt — it is not injected automatically.