← All insights

Reproducing agent failures from sealed inputs

A finding becomes actionable when an engineer can replay the same world snapshot, seed, event schedule, and dependency behavior—and verify the same trace root.

2026-09-016 minute readEASAP Research

Record the world around the agent

Saving the prompt and response is insufficient for an agent that reads clocks, calls tools, sees changing state, and waits on people. Exact replay needs a sealed subject, a versioned world snapshot, a typed scenario, the random seed, the event schedule, dependency responses, and the harness version.

Each observation should capture the actor, event, state before and after, payload digest, and causal relationship at the harness boundary. That is the point where an attempted capability becomes evidence.

Minimize without changing the predicate

Large campaign traces are expensive to debug. A minimizer can remove state, events, payload fields, and dependencies while rerunning the failure predicate. Every accepted and rejected transformation belongs in the evidence so a reviewer can see that the reduced case still represents the original defect.

Close the incident loop

When production monitoring finds a new behavior, protected values should be redacted, provenance recorded, and a privacy reviewer should approve the derived synthetic scenario. The resulting regression then joins the sealed release suite without copying sensitive production data into an evaluation corpus.

Primary referencesModel Context Protocol specification ↗A2A specification ↗