Every check passed. The model changed anyway.

Model IDs are not in a lockfile. A hosted model can be retired, re-routed or repriced while your code and CI stay green. Swapcheck scans your repo, tells you what is changing and when, tests what you could move to on your own cases, and opens a PR with the evidence.

npx @allthingsai/swapcheck

or uvx swapcheck. Run it inside a project. Needs no keys and spends nothing to scan.

Your repo model = "gemini-2.5-pro" git diff: 0 files changed

CI run

Date

gemini-2.5-pro

lint
types
unit
e2e
deploy

5 of 5 green

Provider side. Your CI cannot see past this line.

You asked for

It answered with

Status

Billed at the replacement's rates. Calls keep succeeding.

A real change, dated. On 2026-12-01 gemini-2.5-pro re-routes to gemini-3.8-flash. Nothing fails, so nothing tells you.

Now showing

Dated changes, none of them in your diff

Three real model changes with their dates and effects
Date	Model or API	Change	What happens
2026-11-30in 19 days	`claude-sonnet-4-5-20250929`	Retires	Requests fail.
2026-12-01in 20 days	`gemini-2.5-pro`to `gemini-3.8-flash`	Re-routes	Calls keep succeeding on a different model, at the replacement's rates.
2026-10-2121 days ago	`anthropic-version: 2026-10-21`	Parameters	`temperature`, `top_p` and `top_k` removed. Requests using them return 400s.

See every upcoming change on the calendar

How it works

From an announcement to a pull request

One line, five stops. Only test spends money, and it shows you the maximum first.

Registry

A free, signed list of what providers announced, with a source link for each entry.
- retires claude-sonnet-4-5-20250929
- re-routes gemini-2.5-pro
- 400s temperature, top_p, top_k
scan
Finds model IDs in TS, JS, Python and config. Writes models.lock.
watch
Compares the lock to the registry. Fails CI inside your window.
find
Shortlists models that do what your calls do and your policy allows.
test
Repeated runs on your cases, compared against noise.
bump
Edits the IDs, drops parameters the target rejects.
Pull request

One command per stop. Plain swapcheck runs the scan. The GitHub Action can run them on a schedule.

find

Many models in. Five out. One tested.

The registry can tell you what exists. Only your cases can tell you what works for you, so the shortlist is where the listing stops and the testing starts.

Registryevery model it lists
Capabilitytools, JSON, vision, context
Policyproviders, regions, licence, price
Retirementnot scheduled to go away itself
Shortlistthe top five, each with a reason

Testedon your cases, repeated runs

CHANGED
REGRESSED
NO CHANGE DETECTED
REGRESSED
NOT ENOUGH EVIDENCE

Tick counts and verdicts are illustrative. Your shortlist depends on the registry and your policy.

test

A difference has to survive the maths, then happen again

The same prompt gives different answers from one run to the next, so one run of each model proves nothing.

Run each case several times on both models.
Test each case with Fisher's exact test, then correct across the suite with Benjamini-Hochberg.
Re-run anything flagged on a fresh sample. A drop that does not repeat does not count.

refund-policy0.06

json-invoice1.00

tool-lookup1.00

tone-check1.00

date-parse1.00

summary-len1.00

label-ticket1.00

safety-refuse1.00

passed failed. Each dot is one run.

Verdict

refund-policy fell from 5 of 5 to 0 of 5 (p = 0.008, q = 0.063), then passed 1 of 5 on a fresh sample (p = 0.024). The drop happened twice, so it counts. Exit code 1.

The runs are illustrative. The p and q values are what Swapcheck's statistics module returns for these counts.

Verdicts

Four verdicts. None of them is "safe".

Swapcheck only knows what your cases cover. A clean result means nothing moved on those cases, at that sample size, on that day.

NO CHANGE DETECTED: Nothing differed beyond noise on your cases. It is not a guarantee about anything your cases do not test.exit 0
CHANGED: Check results, tool calls or token use moved beyond noise, with no confirmed drop in pass rate. Read the report and decide.exit 0
REGRESSED: At least one case dropped, and dropped again on a fresh sample. CI fails.exit 1
NOT ENOUGH EVIDENCE: Over 20% of calls failed, the budget ran out, or there were fewer than six usable cases. It will not guess.exit 3

The commands

What it looks like on your screen

Scanning, watching and finding are free. Testing spends your provider credit, after showing the most it could cost.

$ npx @allthingsai/swapcheck
Scanned 214 files. Found 2 models.
Registry: 2026-11-09 (remote)
  claude-sonnet-4-5-20250929  retires 2026-11-30, in 21 days
    src/agents/planner.ts:14  src/agents/review.ts:9
    source: https://docs.anthropic.com/en/docs/about-claude/model-deprecations
  gemini-2.5-pro  reroutes to gemini-3.8-flash 2026-12-01, in 22 days
    app/summarize.py:31
    source: https://ai.google.dev/gemini-api/docs/deprecations
2 of 2 models have a change in the registry.
Wrote models.lock.
$ npx @allthingsai/swapcheck watch --fail-within-days 30
Watching 2 models from models.lock.
2 models changing within 30 days.
# exit code 1
$ npx @allthingsai/swapcheck test --to example-model-a,example-model-b
Spend up to $4.20? [y/N] y
Compared claude-sonnet-4-5-20250929 on 24 cases, 5 runs each.
Noise: model against itself flagged 0 cases (false-change rate 0%).
  example-model-a  Regressed
    1 of 24 cases passed less often on the candidate and the drop reproduced on a fresh sample.
  example-model-b  No change detected
    No difference beyond the measured noise was detected on the 24 cases tested. This covers only those cases.
Spent $2.87 of a $4.20 limit.
Limits of this measurement:
  - Only the behaviour these cases cover was measured.
$ npx @allthingsai/swapcheck bump claude-sonnet-4-5-20250929 --to example-model-b --open-pr
Edited 2 places in 2 files. Opened a pull request with the report attached.

Real commands, in the CLI's output format, shortened. The file names, costs and example-model results are made up for this replay.

Security posture

It runs on your machine, with your keys

There is no Swapcheck server to send anything to. The only network calls are the registry fetch, your model providers, and the GitHub API when you ask for a PR.

Read the threat model

telemetry: none
hosted service: none
api keys: read from environment variables, redacted from all output
tool calls: recorded and mocked, never executed
scanned code: read as text, never run
registry: Ed25519 signed, public key pinned in the client
releases: published to npm and PyPI from GitHub Actions, with provenance
spending: maximum cost shown first; refused without --yes off a terminal

Comparison

Two other open-source tools do this well

ModelBump and Modelpin got here first, and both publish measured accuracy, which Swapcheck has not done yet. Read from their READMEs on 2026-11-09. Swapcheck can import suites from either.

Swapcheck compared with ModelBump and Modelpin
	Swapcheck	ModelBump	Modelpin
Pick it when	You want every kind of provider change on one board, in TS or Python repos	You want Dependabot-style upgrades and prompts repaired for you	You want your app's real requests replayed and drift caught in CI
Runs on	Node 22 via `npx`; binary wheels via `uvx`	Python 3.12+	Python 3.11+
Watches for	Retirements, re-routes, alias moves, prices, parameters, capabilities	Retirements	Retirements
Registry updates	Signed snapshot, fetched daily, bundled fallback	Registry of provider retirement dates	Ships inside each release; can point at a newer file
Finds replacements	Shortlist by capability, policy and retirement	Uses the replacement the registry suggests	Names the vendor's successor; you choose the candidate
Statistics	Fisher exact, Benjamini-Hochberg, must repeat on a fresh sample	Sequential tests or fixed runs, with Benjamini-Hochberg	Exact permutation test, minimum effect size, must repeat on fresh runs
Repairs prompts	No	Yes, and verifies the fix	Not in its README
Opens the PR	Yes	Yes	No, by design
Free local models	Through OpenAI-compatible routes	Ollama by default	Through OpenAI-compatible hosts
MCP server	Yes	Not in its README	Not in its README
Published accuracy	Not yet	Scanner, judge and repair results	False-alarm and detection rates
Licence	Apache-2.0	Apache-2.0	Apache-2.0, open-core

Install

Nothing to sign up for

Run it once in a repo that calls a hosted model. It needs no keys and no config, and it spends nothing.

Read the quickstart

Node

npx @allthingsai/swapcheck

Python

uvx swapcheck

GitHub Action

- uses: allthingsN/swapcheck@v1
  with:
    fail-within-days: 30

MCP server

{ "mcpServers": { "swapcheck": {
  "command": "npx", "args": ["-y", "@allthingsai/swapcheck", "mcp"]
} } }

Every check passed. The model changed anyway.

Dated changes, none of them in your diff

From an announcement to a pull request

Registry

`scan`

`watch`

`find`

`test`

`bump`

Pull request