Your tests passed. Your migration still broke production.

Mocks encode the past

A mock, stub or recorded fixture is a snapshot of what an API returned when someone wrote the test. That is its job: it makes tests fast and deterministic. The cost is that the snapshot does not update when the provider does.

During a migration, engineers update the code and then update the fixtures until the tests compile and pass. The new fixtures are written by the same person, from the same understanding of the change, as the new code. If that understanding is wrong, the code and the fixtures are wrong together, and the tests confirm the mistake.

This is not an argument against mocks. It is an argument against treating tests that use them as proof that a migration preserved behavior. They prove the code agrees with the fixtures. Nothing more.

Contract breaks and behavioral breaks

It helps to separate two kinds of breaking change:

• Contract breaks change the shape: a field is renamed or removed, a type changes, a parameter becomes required. Compilers, type checkers and OpenAPI diff tools catch most of these.
• Behavioral breaks keep the shape and change the meaning: a unit changes, a default changes, an enum value is replaced. The schema validates and the code compiles, yet the result is wrong.

Contract breaks are loud. Behavioral breaks are quiet, and a migration usually contains both. The loud ones get fixed because the build fails. The quiet ones survive because nothing fails.

A worked example: a billing API change

Imagine a billing provider ships a new version of its invoices endpoint with three changes. Here is the old response:

{
  "id": "inv_123",
  "customer": "cus_42",
  "amount": 19.99,
  "currency": "usd",
  "status": "succeeded"
}

And the new one:

{
  "id": "inv_123",
  "customer_id": "cus_42",
  "amount": 1999,
  "currency": "usd",
  "status": "paid"
}

Three changes, three different kinds of risk:

1. customer becomes customer_id. A contract break. With typed clients, every reader fails to compile until it is fixed.
2. amount moves from dollars to minor units (cents for USD). A behavioral break. The type is still a number, so the code compiles either way.
3. status value succeeded becomes paid. A behavioral break. The type is still a string, so a check like status === "succeeded" compiles and now always returns false.

How the migration goes wrong

The engineer doing the migration reads the changelog, renames customer to customer_id everywhere the compiler complains, and updates the fixtures to the new shape. They notice the unit change and divide by 100 in the invoice rendering module. Tests pass. Review approves.

What they missed:

• A revenue report reads amount from a cached copy of the response, in a different module, and sums it without dividing. The fixture for that report test was updated to 1999, and the expected total was updated to match. Reported revenue is now 100 times too high.
• A webhook handler marks orders fulfilled when status === "succeeded". Its fixture still says succeeded, so the test passes. In production, no order is ever marked fulfilled.
• For currencies with no minor unit, such as JPY, dividing by 100 is wrong. No fixture used JPY, so no test noticed.

Every test is green. Every one of these is a production incident, and none of them throws an error.

What actually catches it

Each layer of testing catches a different slice:

• Types and specification diffs catch the customer_id rename. They cannot see units or enum meaning.
• Consumer-driven [contract testing](/glossary/contract-testing) catches shape changes the consumer declared it relies on. It still depends on what the consumer thought to declare.
• Behavioral comparison on real inputs catches the rest. Replay recorded production requests against the old code and the migrated code, and compare the full outputs: status, headers, body and timing.

In the example, a replay would show the revenue report total changing by a factor of 100, the fulfilment state never flipping, and JPY invoices rendering with a wrong total. Those diffs exist whether or not anyone wrote a test for them, because the inputs came from production rather than from the engineer’s imagination. See traffic replay testing.

Practical steps for your next migration

1. Treat updated fixtures as part of the change under review, not as test scaffolding. Ask who decided the new expected values and from what source.
2. Search for every read of a changed field, including caches, queues and serialised copies, not only direct API calls.
3. Grep for every literal comparison against a changed enum value.
4. Build a small table of edge cases from production data: zero-decimal currencies, legacy accounts, empty lists. Make sure they are exercised.
5. Before merge, replay a sample of real traffic against old and new code and require an empty diff, or a diff that matches the expected change exactly.

Releo runs that last step on every migration pull request it opens and blocks the merge when behavior diverges. If that would help your team, join the waitlist.

Frequently asked questions

Related guides

• What is an API breaking change?
• Traffic replay testing, explained
• How to migrate to a new API version safely

More from the blog

• How to ship a breaking API change without burning your consumers
• The hidden cost of an API breaking change, for the people who consume it