Concepts

VCR for LLMs: record/replay vs mocks vs live calls

Updated October 20267 min read

"VCR" testing — record real HTTP traffic once, replay it in later test runs — has been around for over a decade. LLM APIs make it more useful than ever, and also expose where general-purpose recorders fall short. Here's how the approaches compare, and how to choose.

What is VCR-style testing?

The idea comes from Ruby's VCR library and spread to VCR.py, Polly.js, nock's back mode and others. The first time a test runs, real HTTP requests are made and the responses are saved to a "cassette" file. Every later run serves responses from the cassette instead of the network.

You get real data — not responses you invented — with the speed and determinism of a mock.

Mocks vs live calls vs recordings

	Hand-written mocks	Live API calls	Recordings
Uses real model output	No	Yes	Yes, from when recorded
Deterministic	Yes	No	Yes
Speed	Instant	Seconds per call	Instant
Cost per run	Free	Tokens	Free
Needs a key in CI	No	Yes	No
Effort for multi-step agents	High	None	One recording run
Notices model changes	No	Yes	Only when re-recorded

No single column wins everything. Recordings cover the most ground for everyday tests; a scheduled live run covers the last row.

Why LLM APIs are a special case

LLM traffic stresses record/replay in ways typical REST APIs don't:

Streaming. Most LLM apps stream. Responses arrive as server-sent events, and replay has to reproduce the event stream faithfully — not just a final JSON body — or streaming code isn't really tested.
Huge, nested JSON requests. A request carries the whole conversation, tool definitions and schemas. Matching must ignore formatting and key order, and when a match fails you need a readable diff, not "no cassette found".
Secrets. API keys travel in headers, and prompts sometimes contain tokens. Cassettes get committed to git, so they must be scrubbed by default.
Retries. SDKs automatically retry 429 and 5xx responses. A recorder that saves those — or a replayer that fails with a 500 — produces confusing, slow failures.
Many languages. One team may call models from a Python backend, a TypeScript frontend and a Go worker. Library-based recorders need one setup per language.

Can I use VCR.py, Polly.js or nock?

Yes, and they're excellent general-purpose tools. If your whole codebase is in one language and you're happy to configure header filtering, body matching and streaming support yourself, they can work well.

A dedicated LLM recorder like retake takes a different approach: it runs as a local HTTP proxy that the official SDKs reach through OPENAI_BASE_URL and ANTHROPIC_BASE_URL. That gives you:

One setup for every language and test runner.
Byte-for-byte replay of streams.
Order-insensitive JSON matching, with a diff against the closest recording when something changes.
Headers never stored and secrets redacted, plus a retake check command for CI.
Mismatches returned as 404, so SDKs don't retry them.

How to choose

Testing a parser, validator or prompt template in isolation? A small mock is simplest.
Testing code that depends on what the model actually says — an agent loop, a RAG pipeline, structured extraction? Record and replay.
Need to know whether output is good, or whether a new model is better? That's an eval, and it should call the model live (on a schedule or on demand, not every push).
Want to know when the provider changed the model? A weekly live re-record, as described in the CI guide.