Fetcher

Fetcher is the project-owned retrieval contract for agents that need public HTTP/HTTPS URLs, local files, PDFs, or explicitly enabled anonymous FTP resources converted into deterministic artifacts. It is not the right tool for OAuth connectors, private SaaS APIs, credentialed scraping, or unbounded recursive crawls.

First Command

Check the installed contract before fetching:

# fetcher-doc-smoke: version-json
fetcher version --json

Require:

• package.name == "fetcher".
• schemas.consumer_summary includes fetcher.consumer_summary.v1.
• entrypoints.fetcher and entrypoints.fetcher-etl are present.
• Any optional capability you need is available or explicitly enabled.

Consumer CLI

Use the consumer CLI for agent-facing retrieval and artifact handoff:

fetcher get https://example.com --json --out run/fetcher/example
fetcher get-manifest urls.txt --json --out run/fetcher/batch
fetcher get-manifest - --json --out run/fetcher/stdin < urls.txt

The primary artifact is always:

<out>/consumer_summary.json

When --json is used, stdout is exactly the same summary object. Preserve the process exit status and parse the JSON before using artifacts.

Terminal acceptance:

• run_status is one of completed, completed_with_failures, capability_unavailable, usage_error, or fatal_error.
• exit_code matches the process exit code.
• schema == "fetcher.consumer_summary.v1".
• items has one terminal item per requested URL in stable input order.
• For synthesis, use items[].artifacts.extracted_text_path or items[].artifacts.markdown_path, not HTTP status or raw HTML alone.
• Treat item warnings and errors as part of the result, not as log noise.

Exit codes:

• 0: every required requested item was accepted.
• 2: usage, manifest, or validation error.
• 3: completed with one or more failed or rejected items.
• 4: requested capability unavailable before it could run.
• 5: fatal internal/orchestration error.

Tested Smoke Commands

These commands are intentionally side-effect-light and are executed by scripts/ci/fetcher_skill_contract_smoke.py against a clean wheel:

# fetcher-doc-smoke: doctor
fetcher doctor

# fetcher-doc-smoke: dry-run-single
fetcher get "$FETCHER_SMOKE_URL" --dry-run --json --out "$FETCHER_SMOKE_ROOT/dry-single"

# fetcher-doc-smoke: dry-run-manifest
fetcher get-manifest "$FETCHER_SMOKE_MANIFEST" --dry-run --json --out "$FETCHER_SMOKE_ROOT/dry-manifest"

# fetcher-doc-smoke: ftp-disabled exit=4
fetcher get "ftp://ftp.example.com/pub/data.txt" --json --out "$FETCHER_SMOKE_ROOT/ftp-disabled"

# fetcher-doc-smoke: etl-find
fetcher-etl --find metrics

FTP

FTP is disabled by default. Enable it only when the caller explicitly needs anonymous read-only ftp:// retrieval:

fetcher get "ftp://ftp.gnu.org/README" --enable-ftp --json --out run/fetcher/ftp
FETCHER_ENABLE_FTP=1 fetcher-etl --manifest ftp-urls.txt --out run/fetcher/ftp-etl

Fetcher rejects ftps://, SFTP, embedded credentials, authentication, active mode, writes, and recursive directory crawling. Private/local destinations are denied unless FETCHER_FTP_ALLOW_PRIVATE=1 is set for a trusted fixture.

ETL Mode

Use fetcher-etl when you need full pipeline controls, metrics, resolver knobs, inventory JSONL, or ETL audit files:

fetcher-etl --manifest urls.txt --out run/fetcher/etl
fetcher-etl --inventory urls.jsonl --output run/fetcher/results.jsonl --audit run/fetcher/audit.json
fetcher-etl --help-full
fetcher-etl --find metrics

Consumer and ETL artifacts are different contracts. Do not assume ETL results.jsonl fields are the same shape as consumer_summary.json.

Python API

Use Python only when a CLI process is not the right integration boundary. A complete async example must read metadata through FetchResult.metadata or to_dict():

import asyncio

from fetcher.workflows.web_fetch import FetchConfig, URLFetcher


async def main() -> None:
    fetcher = URLFetcher(FetchConfig(concurrency=2, per_domain=1))
    results, audit = await fetcher.fetch_many([{"url": "https://example.com"}])
    result = results[0]
    payload = result.to_dict()
    print(payload["status"])
    print((result.metadata or {}).get("content_verdict"))
    print(audit.get("requested"))


asyncio.run(main())

Failure Reporting

For each degraded or failed item, report:

• requested_url and final_downloaded_url.
• status, method, content_type, and verdict.
• warnings, errors, paywall_verdict, and alternate_provider.
• Selected artifact paths and whether they exist and are non-empty.
• Relevant failure_summary buckets such as fallback reason or content verdict.

Do not call a run successful from HTTP 200 alone. Use the summary verdict and artifact existence.

References

• references/USAGE_CONTRACT.md: artifact selection and acceptance rules.
• references/ETL_AND_CONFIG.md: ETL-only flags, cache knobs, proxy rotation, alternates, PDF discovery, and Python API details.
• references/TRIGGER_EVAL.md: should-trigger and should-not-trigger prompts.
• docs/DOWNSTREAM_WRAPPER_CONTRACT.md: contract for downstream skill wrappers.