Docs

Quickstart

Run this in a repo that calls a hosted model. It needs no keys and no config, and it spends nothing.

npx @allthingsai/swapcheck     # or: uvx swapcheck

It finds model IDs in your code and config, shows which are changing with dates and source links, and writes models.lock. Commit that file. Then:

# Fail CI when something changes within 30 days
npx @allthingsai/swapcheck watch --fail-within-days 30

# Shortlist replacements for a model
npx @allthingsai/swapcheck find gemini-2.5-pro

# Write contract checks and drafts from your code
npx @allthingsai/swapcheck cases init
npx @allthingsai/swapcheck cases accept --all

# Measure your current model, then compare candidates
export GEMINI_API_KEY=...
npx @allthingsai/swapcheck baseline
npx @allthingsai/swapcheck test --to candidate-a,candidate-b

# Edit the IDs and open a pull request
npx @allthingsai/swapcheck bump gemini-2.5-pro --to candidate-b --open-pr

Commands that spend show the maximum cost and the vendors first, and ask. Off a terminal, they refuse to spend without --yes. Keys come from OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY or GOOGLE_API_KEY, and OPENROUTER_API_KEY.

Commands

Options on every command

--format text|json|markdown|sarif, --cwd <dir>, --config <file>, --registry-url <url>, --offline, --yes, --no-color, --debug.

Options on commands that spend

--runs <n> runs per case, --max-usd <amount> hard spending limit, --judge <model> model that grades meaning checks, --route <route> where the model is served, such as aws-bedrock.

Config

swapcheck.yaml is optional. Everything has a default.

models:
  - id: gemini-2.5-pro
    route: google-ai-studio
    paths: ["app/**"]
cases: swapcheck/cases
runs: 5
budget:
  max_usd_per_run: 5
policy:
  providers: [google, anthropic]
  regions: [us, eu]
  licenses: [proprietary]
  max_usd_per_million_output: 15
watch:
  fail_within_days: 30
include_text_in_reports: false
exclude: ["vendor/**"]

models.lock

Hosted model IDs are not pinned by anything in your package manager, so Swapcheck writes a lockfile for them. It records each model, where it is served, its status and when it was last verified.

{
  "version": 1,
  "models": [
    {
      "id": "gemini-2.5-pro",
      "route": "google-ai-studio",
      "status": "reroute_scheduled",
      "reroute_to": "gemini-3.8-flash",
      "effective": "2026-12-01",
      "verified_at": null,
      "baseline": null
    }
  ]
}

Commit it. watch compares it to the registry, so a change shows up in CI even though your code did not move. It cannot stop a provider changing a model. It makes the change visible.

Files in your repo

swapcheck.yaml (optional), models.lock, swapcheck/cases/*.json, swapcheck/reports/<id>.json, swapcheck/baselines/<hash>.json.

Cases

A case is a request plus checks on the response. Available checks: json_valid, json_schema, label_in, contains, not_contains, regex, tool_called, tool_not_called, tool_args_valid, no_refusal, refusal, max_tokens_output and meaning, which a judge model grades.

Generated cases are excluded until you accept them. Multi-turn cases replay only up to each decision point. Later logged user turns are never replayed against a new model's replies.

GitHub Action

name: Model watch
on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"
permissions:
  contents: read
jobs:
  watch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: allthingsN/swapcheck@v1
        with:
          fail-within-days: 30

A scheduled run matters, because the point is to notice a change when your code has not moved.

Outputs: verdict, changes-due, report-path, spent-usd and recommended-model. In check mode, recommended-model is the cheapest candidate with no change detected. Pull requests from forks get the free scan only.

MCP server

{ "mcpServers": { "swapcheck": {
  "command": "npx", "args": ["-y", "@allthingsai/swapcheck", "mcp"]
} } }

Tools: scan, watch, find and explain_event. Tools that spend money, such as test, appear only when SWAPCHECK_MCP_ALLOW_SPEND=1 is set. The budget for a test call is the lowest of the requested max_usd, budget.max_usd_per_run in your config, and SWAPCHECK_MCP_MAX_USD if set, so the agent can never raise it.

Verdicts

How a verdict is reached

1. Baseline: the current model, N runs per case (default 5).
2. Noise: compare two independent halves or a second baseline, and report the false-change rate.
3. Candidate: N runs per case on each candidate.
4. Per case: two-sided Fisher exact test on pass counts, Benjamini-Hochberg across the suite.
5. Suite: paired permutation test across cases on pass-rate difference, with a bootstrap 95% interval.
6. Any case with q < 0.1 and a lower candidate pass rate is re-run on a fresh sample of N, and must reproduce to count as confirmed.

Exit codes

Limits

• It only knows what your cases cover. A behaviour you do not test can change without any verdict noticing.
• Five runs detect large drops, not small ones. Raise --runs to find smaller differences, at more cost.
• The registry is what providers announce. An unannounced change will not appear. Dates can move.
• The scanner reads source statically. IDs built at runtime are reported as unresolved. Declare them in swapcheck.yaml.
• LiteLLM route names are not model IDs. In a proxy config or Router, Swapcheck checks litellm_params.model and skips model_name, also where other code calls the proxy by that name.
• Bedrock changes are per region. A change is reported when it covers the region in aws_region_name or region_name, or the regions an inference profile such as us. reaches.
• An Azure deployment hides its version. For a deployment such as gpt-4o, the soonest retirement among Azure's dated versions is shown.
• A result describes one day. A hosted model can change after you test it.
• bump edits IDs and parameters. It does not rewrite your prompts.
• Accuracy has not been published yet. The test suite targets under 5% false alarms and 80% of seeded regressions caught, in simulation.