pi-coding-agent-eval

Compare arbitrary agent profiles against the same real Pi Coding Agent tasks.

Summary

pi-coding-agent-eval runs repeatable evaluations through real Pi sessions. It balances agent profile order, prepares isolated workspaces, validates outcomes, records agent telemetry, and calculates per-agent-profile and pairwise statistics.

The package defines only one agent profile: vanilla, with no additional extensions and all registered tools. Consumers define every other agent profile and explicitly select the profiles to run.

Executable behavior contracts and runnable comparisons live together under ./test/examples/.

Installation

npm install pi-coding-agent-eval

The install includes the TypeScript API, the pi-eval CLI, and all npm dependencies. Eval uses pi-coding-agent-test to run the pi executable available on the system PATH.

From a checkout, install the repository dependencies instead:

npm install

Optional Pi skills

The package includes three skills for creating and inspecting evaluations:

• using-pi-coding-agent-eval — orient before changing a benchmark;
• write-pi-coding-agent-eval — write suites, presets, and profile comparisons;
• inspect-pi-coding-agent-eval — inspect artifacts, reports, and pairwise results.

Install the published package as a Pi package:

pi install npm:pi-coding-agent-eval

From a checkout, install the local package with the helper script:

npm run skills:install
npm run skills:install -- --local

The first command updates user settings. The second writes a project-local Pi setting.

Prerequisites

• Node.js 22.19 or newer;
• Pi installed with the pi executable available on PATH;
• model credentials configured for paid model runs.

Run an evaluation

The package does not provide a built-in benchmark suite. A suite is supplied by the caller through a config file. The repository includes a complete deterministic config at ./test/examples/tool-profile-comparison/eval.config.mjs. It registers the real tool-comparison suite and two scripted profiles, so it makes no model request.

Run it from the repository root:

npm run example:cli

The equivalent direct CLI command is:

npm run build
node ./dist/cli.mjs run tool-comparison easy \
  --config ./test/examples/tool-profile-comparison/eval.config.mjs \
  --agent-profiles filesystem-tools,bash-only \
  --run-id cli-example

To exercise the global model, thinking, skill, and prompt flags without making a model request:

node ./dist/cli.mjs run tool-comparison easy \
  --config ./test/examples/tool-profile-comparison/eval.config.mjs \
  --agent-profiles filesystem-tools,bash-only \
  --model scripted/scripted-model \
  --thinking off \
  --skill "$(pwd)/skills/write-pi-coding-agent-eval" \
  --system-prompt "Use only the tools exposed by your profile." \
  --append-system-prompt "Complete the task and stop when it is done." \
  --run-id cli-settings-example

The run writes its artifacts to ./.tmp/tool-profile-comparison/cli/results/cli-example. Rebuild that report without rerunning Pi:

node ./dist/cli.mjs report ./.tmp/tool-profile-comparison/cli/results/cli-example

The CLI accepts the global --model, --thinking, --skill, --system-prompt, and --append-system-prompt settings. Repeat --skill and --append-system-prompt when you need more than one value. A profile can override these settings in the config file. The evaluator calculates every supplied agent profile pair as right - left.

Thinking off enforcement

When a profile's effective thinking level is off, the evaluator automatically loads its built-in disable-thinking extension. The extension checks the selected model's metadata and applies the provider's real hard-off request control only when off is supported. It never substitutes minimal or low reasoning.

The control covers Pi's OpenAI, Anthropic, DeepSeek, Qwen, Google, Vertex, Together, Baseten, Mistral, Bedrock, and compatible chat-template request formats. Each trial records control evidence in agent/provider-control.jsonl; reported reasoning tokens or thinking blocks make an off trial fail.

The extension path is also exported for direct Pi use:

import { disableThinkingExtension } from "pi-coding-agent-eval";

The extension's package subpath is pi-coding-agent-eval/disable-thinking.

TypeScript API

The complete direct API example is ./test/examples/tool-profile-comparison/run-api.mjs. It uses the exported runEvaluation function with the real tool-comparison suite and scripted profiles, so it makes no model request.

Run the exact example:

npm run example:api

The example uses a global model, thinking level, system prompt, and appended system prompt. It overrides thinking for bash-only, calculates a custom exact-files metric, writes a custom api-report.json, and checks that both profiles pass. This is the canonical runnable example for the direct API.

The evaluator keeps these responsibilities separate:

• a suite owns task selection, workspace preparation, prompts, validation, and cleanup;
• a benchmark preset names the task set, seed, and default attempts;
• an agent profile owns the model, thinking level, skills, extensions, tools, prompt settings, and optional scripted conversation;
• run-level model, thinking, skills, and prompt settings are inherited by profiles unless a profile overrides them.

The public API is documented with TSDoc comments next to its source declarations. The published package exposes the same contract through its dist/*.d.ts files. There is no generated documentation site or manually maintained API copy.

Runnable examples

From a checkout, run all free deterministic checks, including the integration suite and the executable API/CLI examples:

npm test

To run only the integration suite:

npm run test:integration

The exact CLI and direct API examples use the same suite without model requests:

npm run example:cli
npm run example:api

The CLI example writes ./.tmp/tool-profile-comparison/cli/results/cli-example; the API example writes ./.tmp/tool-profile-comparison/api/results/api-example and adds api-report.json.

Example	Covers
tool-profile-comparison	Filesystem tools versus Bash, scripted and real-model live modes
multi-profile-comparison	Filesystem tools, Bash, unrestricted vanilla, and every pair among three agent profiles

Watch the exact scripted tests in Pi's native TUI:

npm run example:scripted-live
npm run example:multi-scripted-live

Run the three-profile comparison with a real model:

npm run example:multi-live -- \
  --model opencode-go/deepseek-v4-flash \
  --thinking high \
  --attempts 3

License

MIT