CI/CD

Run LLM tests in CI without API keys

Calling a real LLM API from CI works — until it doesn't. Pipelines get flaky, bills grow with every push, rate limits fail builds at random, and pull requests from forks can't run the tests at all. Here's how to make LLM tests in CI deterministic and free, while still noticing when the model changes.

What goes wrong with live calls in CI

• Flakiness. Model output varies between runs, so the same commit can pass and then fail. Teams learn to click "re-run", and stop trusting red builds.
• Cost. Every push, every branch, every matrix job and every retry pays for tokens. Agent tests that make dozens of calls each are the worst offenders.
• Rate limits. Parallel CI jobs hit 429 Too Many Requests and fail for reasons that have nothing to do with your change.
• Secrets and forks. On GitHub, workflows triggered by pull requests from forks don't receive your repository secrets. Tests that need an API key simply can't run for outside contributors — or you're tempted into risky workarounds.
• Speed. Seconds per call turn a 30-second suite into a 10-minute one.

The strategy: replay in CI, record locally

1. Developers record API calls once with a real key, using a record/replay proxy such as retake.
2. The recordings (cassettes) are committed with the code, and prompt changes show up in code review.
3. CI replays them strictly: no network access to the LLM API, no key, and any call without a recording fails the build instead of silently going live.
4. A scheduled job runs live once a week to catch model drift.

GitHub Actions setup

retake ships a GitHub Action that installs a verified binary on Linux, macOS or Windows runners:

name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - uses: Zecrari/retake@v0
      - run: retake check                 # cassettes are valid and secret-free
      - run: retake replay -- npm test    # no key, no network, no cost

That's the whole change. Notice what's missing: there's no OPENAI_API_KEY or ANTHROPIC_API_KEY secret. In replay mode retake sets placeholder keys so SDKs start normally, and since nothing reaches the real API, fork pull requests run the full suite too.

What a failure looks like

If someone changes a prompt without re-recording, the build fails with a precise diff instead of a mysterious assertion error:

no recording in session `default` matches POST /openai/v1/chat/completions (closest: step 1)
-      "content": "Summarize in one short sentence.",
+      "content": "Summarize in one sentence of at most 20 words.",
retake: 0 replayed, 0 recorded, 1 unmatched

The fix is to run retake auto -- npm test locally (it records only what's missing), and commit the updated cassette — which reviewers then see in the pull request.

Keeping secrets out of cassettes

Committing recordings of API traffic needs care. retake handles the main risks for you:

• Request headers — Authorization, x-api-key and the rest — are never written to disk.
• Strings in request and response bodies that look like API keys, tokens or private keys are replaced with [REDACTED].
• retake check fails the build if a cassette still looks like it contains a secret. Run it in CI (as above) and as a pre-commit hook.

A weekly live job for model drift

Replays freeze model behavior at recording time. To find out when the provider changes something, run the suite live on a schedule — with the API key stored as a secret that only this trusted, scheduled workflow uses:

name: Live LLM check
on:
  schedule:
    - cron: "0 6 * * 1"     # Mondays 06:00 UTC
  workflow_dispatch:

jobs:
  live:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - run: npm ci
      - uses: Zecrari/retake@v0
      - run: retake record -- npm test
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - run: retake check
      - name: Show what changed
        run: git diff --stat -- cassettes/

If the live run fails, or the cassette diff shows meaningful changes, the model's behavior has moved. Review the diff, then commit the new recordings (or open a pull request with them) when you're happy.

GitLab CI and others

Anywhere you can run a shell, you can install retake with the script:

# .gitlab-ci.yml
test:
  image: node:22
  script:
    - curl -fsSL https://zecrari.github.io/retake/install.sh | sh
    - export PATH="$HOME/.local/bin:$PATH"
    - npm ci
    - retake check
    - retake replay -- npm test

Pin a version for reproducible pipelines with RETAKE_VERSION=v0.1.0. The script verifies the download's SHA-256 checksum.