Inside a behavioral diff: what to compare and what to ignore when replaying API traffic

Start by comparing everything

A behavioral diff should begin from the full response and subtract noise, rather than starting from a few fields and adding coverage. For each replayed request, capture:

• Status code. A 200 becoming a 201, or a 404 becoming a 200 with an empty body, is a behavioral change clients branch on.
• Headers. Content type, caching headers, pagination links, rate limit headers and Deprecation or Sunset headers all carry meaning.
• Body. Parsed structurally, not compared as text, so that key order and whitespace do not matter but values, types and presence do.
• Timing. Latency for each side, compared against a budget rather than for equality.
• Side effects, where observable. Outbound calls, emitted events or database writes made while handling the request.

Everything in that list is in scope by default. Every exclusion that follows is an explicit, reviewed rule.

Normalizing values that are supposed to differ

Some values legitimately change on every request. The goal is to neutralise them without hiding changes in their shape or meaning.

Timestamps

Do not drop timestamp fields; replace them with a placeholder after checking they still parse in the same format and time zone. A field that moves from 2026-10-21T09:00:00Z to a Unix integer, or from UTC to local time, is a real change. A field whose value differs by a few milliseconds is not.

Generated identifiers

Replace generated IDs with stable tokens assigned in order of first appearance, so inv_8f2a becomes <id:1> everywhere it appears in both responses. This preserves referential structure: if the old response used the same ID in two places and the new one uses two different IDs, the diff still catches it. Check the prefix and format separately, since a change from inv_ to in_ breaks clients that parse IDs.

Nonces, ETags and request IDs

Values that exist only to be unique (request IDs, trace IDs, nonces) can be masked entirely, as long as their presence is still asserted.

Ordering: decide which collections are sets

Arrays are ordered in JSON, but many API collections are semantically unordered: the tags on a resource, the scopes on a token, the set of enabled features. If the new code returns them in a different order, a strict diff fails for no reason.

The fix is to declare, per path, which arrays are sets and sort them by a stable key before comparing. The declaration must be explicit, because the opposite mistake is worse: list endpoints usually have a documented or implied order, and a change in default sort is one of the classic breaking changes. Treat $.data on a list endpoint as ordered unless you have a reason not to.

rules:
  - path: "$.tags"
    compare: unordered
  - path: "$.data[*].line_items"
    compare: unordered
    key: "sku"
  - path: "$.data"
    compare: ordered   # default sort is part of the contract

Header noise and header signal

Headers mix pure infrastructure noise with fields clients depend on. A reasonable default split:

• Ignore: Date, Server, Via, X-Request-Id, trace headers, Set-Cookie values for session tokens (but not their presence or attributes), and load balancer or CDN diagnostics.
• Compare exactly: Content-Type, Location, Link (pagination), Allow, Deprecation, Sunset, Retry-After, and any X- header documented in the API reference.
• Compare semantically: Cache-Control and Vary directives as unordered sets, and rate limit headers by name and presence rather than by remaining count.

When in doubt, compare. A header that turns out to be noise produces a false positive you can add a rule for. A header that turns out to matter, and was ignored, produces an incident.

Latency budgets, secrets and PII

Latency

Timing is never equal between two runs, so compare it against a budget. A practical rule is to flag a request when the new latency exceeds the old by both a relative margin and an absolute floor, so a jump from 2ms to 5ms does not fire but a jump from 200ms to 900ms does. Look at percentiles across the replay set as well as individual requests, since a change that adds a database query often shows up as a shifted tail rather than one slow call.

Redaction

Recorded production traffic contains credentials and personal data, so redaction is part of the design, not an afterthought. Strip or tokenise Authorization headers, API keys, cookies and known sensitive fields at capture time, before anything is stored. Use format-preserving placeholders where the format matters for behavior (an email address should still look like an email) and keep the mapping consistent within a request so that relationships survive.

Deciding pass or fail

After normalization, each request lands in one of three states: identical, divergent, or divergent but expected. The last category is what makes the system usable for real migrations, where some behavior is supposed to change.

1. Every divergence is reported with the request, the path, the old value and the new value.
2. Expected changes are declared up front, tied to a specific changelog entry or migration step, and scoped to specific paths. "Ignore $.amount" is too broad; "$.amount is multiplied by 100" is a rule you can check.
3. Any divergence not covered by a declared expectation fails the run and blocks the merge.
4. Normalization rules and expectations are versioned with the code and reviewed like code, because a loose rule is a silent hole in coverage.

This is how Releo gates its own migration PRs: replayed traffic, diffed under explicit rules, with any unexplained divergence blocking the merge. The same rules apply to any team building replay in-house: compare everything, normalise only what you can justify, and make every exception visible.

Frequently asked questions

Related guides

• Traffic replay testing, explained
• What is an API breaking change?
• How to migrate to a new API version safely

More from the blog

• Deterministic verification vs LLM-as-judge for code changes
• How API providers can cut their customers' migration time