swapcheck

Security

Swapcheck reads your source code and holds your provider keys while it runs, so it should be held to a high standard. Here is what it does, what it refuses to do, and what it cannot protect you from.

It runs on your machine

There is no Swapcheck server, account or hosted service. It runs from the command line or in your own CI. It makes exactly three kinds of network call:

  • a fetch of the public registry snapshot and its signature,
  • calls to the model providers you configure, when you run a command that spends,
  • GitHub API calls when you ask bump to open a pull request.

When you test, your case content goes to the providers you chose to test. Each run lists those vendors first, and your policy file can restrict them. Reports store case hashes and scores, not your prompts, unless you opt in to including text.

No telemetry

No usage data, crash reports or analytics of any kind, and no switch to turn any on. This website has no trackers or third-party scripts either.

Your keys

API keys are read from environment variables only and are not written to config or lock files. All logs, errors and reports pass through a redaction step before they are shown or saved.

Tool calls are mocked and never executed

When a model asks to call a tool during a test, Swapcheck records the request and compares it. It never runs it. Replays feed back a mocked result, and only where your case supplies one. A model under test cannot send an email, write to a database or issue a refund through Swapcheck.

Everything it reads is treated as data

Source files, cases, prompts, model output and registry entries are all untrusted input.

  • Nothing found in them is executed. Swapcheck does not use eval or shell strings.
  • Paths must stay inside your project. Symlinks that escape are refused.
  • File sizes, response bodies and finding counts are capped.
  • Regular expressions from user data are length-limited and protected against runaway matching.

In your CI

Text from models, cases and registries is escaped before it reaches workflow commands, step summaries or pull requests, so it cannot inject commands or markup. Pull requests from forks get the free scan only, and paid modes run only on pull_request, push, schedule, workflow_dispatch, merge_group and release. They refuse every other trigger, including pull_request_target, workflow_run and issue_comment, so a fork cannot spend your money.

The registry is signed

The client verifies each snapshot with an Ed25519 signature against a public key pinned in the client. A snapshot that fails verification is never parsed. Swapcheck falls back to a cached copy, which is verified the same way, or to the snapshot bundled in the release. Reports state the snapshot date and warn when it is more than 7 days old.

Releases

npm and PyPI packages are published from GitHub Actions through OIDC trusted publishing, with provenance, so no long-lived publishing token exists to steal. Every release file carries a GitHub build attestation you can check with gh attestation verify. The published npm package has no runtime dependencies.

Spending

Every paid command works out the worst-case cost first, including retries. Off a terminal it refuses to spend without --yes. The limit you set is a hard stop, and when it is reached the verdict is NOT_ENOUGH_EVIDENCE.

What this does not protect you from

  • A compromised machine or CI runner can read your keys, as with any tool that uses them.
  • Your case content is sent to the providers you test against, under their terms.
  • The registry is only as accurate as what providers announce. A silent, unannounced change will not appear on it.
  • Trace imports are redacted by pattern, which can miss things. Review them, and treat --allow-unredacted with care.
  • NO_CHANGE_DETECTED means nothing moved on your cases. It does not mean the model is safe.

Report a vulnerability

Please report privately through GitHub's security advisories, and do not open a public issue. Include what you found, how to reproduce it, and the version. The goal is to acknowledge a report within three working days and say what will happen within ten. This is a very small team, so those are goals and not guarantees. Only the latest release receives security fixes. For a problem with registry data or signing, use the advisories on the registry repository.