pi-webaio

All-in-one web access tools for pi: search, fetch, crawl, extract, map, cache, chunk, and render web content for AI agents.

What It Does

pi-webaio registers eight pi tools:

aio-websearch — search DuckDuckGo, Brave, Yahoo, Bing, and Google in parallel
aio-webfetch — fetch one or many URLs into markdown or structured formats, with an opt-in heading outline, query-focused answer mode, and multi-source cited answers
aio-webcontent — retrieve cached content by URL (with opt-in section-level diff)
aio-webresult — retrieve cached results by response ID
aio-webmap — discover site pages or map GitHub repositories without fetching
aio-webpull — crawl/pull sites into local markdown files
aio-webquery — BM25 search over a locally-pulled corpus (offline, no re-fetching)
aio-webresearch — single-round research bundle: search → rank → fetch → cited evidence bundle on disk

It includes anti-bot TLS fingerprinting, browser fallback, 21 API-first extractors (GitHub, YouTube, npm/PyPI and other package registries, Context7, DeepWiki, and more), RAG chunking, TUI progress rendering, phase-aware errors, and opt-in paywall bypass support.

Install

pi install npm:pi-webaio

Or from git:

pi install git:github.com/apmantza/pi-webaio

Documentation

Features — overview, extraction pipeline, GitHub/YouTube handling, output formats, chunking, errors, and search ranking
Custom vertical extractors — add your own site extractors (company wikis, niche sites) without forking
Usage guide — common pi prompts and examples
Tools reference — tool names, parameters, and defaults
Architecture — build, TUI rendering, FetchError system, CI, and security notes
MCP server — use the tools from Claude Code, Claude Desktop, and other MCP clients without pi (npx -y pi-webaio-mcp)
PageMap inspiration — future extraction and structured-output ideas

Contributing

We especially welcome contributors for new vertical extractors, search engines, site-specific fetch fixes, anti-bot/paywall resilience, and docs. See CONTRIBUTING.md for setup, testing, and contribution checklists.