Reproducible Trajectories in Multi-Step LLM Tool-Use Environments

Reproducing a multi-step LLM agent trajectory is an environment problem, not a model problem. A single inference call is reproducible by construction: fix the prompt and the weights, and the output is fixed. A trajectory is only reproducible if the world each step lands in is also fixed, and most testing setups leave that world unmanaged. The result is that scores move for reasons nobody can attribute. A stateful, deterministic execution environment is the precondition for evaluating, debugging, or training an agent with any confidence.

Why reproducing a multi-step agent trajectory is harder than reproducing a single LLM call

The LLM itself is stateless by architecture. Every inference call starts from a blank slate, and whatever appears to be memory is the application layer re-injecting prior context as tokens. That property makes a single call easy to reproduce: the same prompt and the same weights produce the same distribution over outputs, and with decoding temperature fixed, the same output. The tools an agent calls do not share this property. A data store remembers what a prior call inserted. A payment processor carries account balances across requests. A messaging platform retains channels and threads. The asymmetry between the stateless model and the stateful tools it drives is the source of the reproducibility problem.

An agent trajectory is not a sequence of independent inferences. Each tool call reads state, mutates it, and hands a changed world to the next step. The trajectory is therefore the execution history of a stateful system. Reproducing it requires reproducing not just the sequence of calls but the intermediate world each call operated against. TraceSafe, a benchmark presented at COLM 2026, makes this concrete from a safety angle: reproducibly evaluating a guard model requires static trajectories with precise, step-level annotations. Without those annotations, it is impossible to know whether a score moved because the agent changed or because the environment changed under it.

How shared mutable state turns a trace into a moving target

The mechanism is straightforward. Because each tool call can mutate shared state, replaying the same sequence of calls against a live or stateless environment does not reproduce the same intermediate world. It produces a world conditioned on whatever state happened to exist at replay time. Step three's tool call is correct only relative to the world that steps one and two left behind. If that world differs by even one record, step three's input is different, its output may differ, and every downstream step inherits the deviation. A single divergence compounds at each subsequent step, so the final trajectory can diverge widely from the reference run while sharing the same call sequence.

This is why bugs in multi-step agents often surface only after three or four interactions. The faulty condition only exists once prior steps have accumulated it. A test that exercises one turn will never see it, because the state never reaches the configuration where the failure becomes possible. The same logic applies to evaluation. Live systems are structurally poor test fixtures: customer records change, documents are updated, external APIs return different data on different days. If the world changes between runs, a score change is uninterpretable. The agent may have improved, or the environment may have simply been easier that day. Rystic, which maintains stateful replicas of external APIs with verified fidelity to the real service, addresses this by making replay deterministic and isolated, so that a score change reflects a change in the agent rather than a change in the world.

Why stateless mocking does not solve the problem even though it looks like it should

The obvious objection is to mock the tools. Stateless mocks are reproducible in a narrow sense: the same input returns the same output, every time. For a single-turn tool-selection test, that is exactly what is needed. The model is asked to pick the right tool, the mock confirms the choice, and the test ends. But a trajectory is not a single turn. A stateless mock for a data store returns the same response regardless of what prior calls have added, updated, or deleted. It cannot represent the state a real system would be in after step two, so step three runs against a fiction, not a replay.

The consequence for evaluation is that a trajectory scored against stateless mocks may pass because the mock always returns the expected record, not because the agent correctly created it in step one. The test confirms the agent's outputs but not its reasoning or its effect on the world. The trajectory-judge project, under review at a NeurIPS 2026 workshop, establishes the same point at the judge level. A judge shown only the final reply scores meaningful recall on faults that never appear in the reply. Its discrimination is zero, because outcome-level evaluation cannot see intermediate failures. The same logic applies to stateless mocks: they suppress the intermediate world where failures live. The hybrid the field has converged on, a stateless LLM inside a stateful runtime, is not an accident. It preserves the LLM's reproducibility while delegating world-state to a layer that can actually track it. Tools designed for stateful integration testing, rather than generic mocking libraries, are necessary to reproduce what an agent actually experiences.

What a stateful, deterministic execution environment must do to guarantee trajectory reproducibility

Guaranteeing that replaying the same call sequence produces the same intermediate world requires four things from the environment. The first is persistent state across calls. Add a record, list it, delete it, and the world updates accordingly. The environment must remember every mutation made within a session so that subsequent calls land in the world prior calls created. Without persistence, the environment is a stateless mock under another name.

The second requirement is deterministic initial conditions. The environment must start from pre-seeded realistic data, not an empty sandbox and not an unpredictably seeded one. An empty environment makes many agent behaviors untestable because the agent has nothing to act on. An unpredictably seeded one makes scores uninterpretable across runs, because the starting world itself is a random variable.

The third requirement is isolation between runs. Each evaluation run must start from a known, reset state so that a prior run's mutations do not contaminate the next. This is what makes a score from run N comparable to a score from run N+1. The fourth is continuous verification against the live API the environment represents. A simulator that is not continuously verified against the real API will silently drift. The environment will diverge from what the real service does, and tests will pass against a fiction while production breaks. TraceSafe's benchmark design makes the same requirement explicit from the safety side. It uses an editing methodology that deterministically injects targeted risks into natural trajectories, preserving realistic planning logic while providing precise step-level ground truth. Determinism and realism are both required, not one or the other.

Two complementary scoring methods and why each requires a stateful environment to mean anything

Two established methods score multi-step trajectories, and they answer different questions. Step-wise scoring, also called teacher-forced scoring, replays the reference trajectory and checks at each step whether the model's next call matches the expected one given the correct history. This isolates per-step accuracy and prevents one early mistake from cascading, which makes it the right tool for diagnosing where things break. End-to-end scoring lets the agent run to completion against simulated tools and checks only the final state. This measures real-world success but conflates errors across steps, since a failure early in the trajectory and a failure late in the trajectory both show up as the same unsuccessful final state.

Both methods presuppose a stateful environment. Step-wise scoring requires injecting the correct history at each step, which means the environment must be able to represent and reset to any intermediate world state, not just the initial one. End-to-end scoring requires that the final state be checkable, which means the environment must have tracked every mutation that led to it. The trajectory-judge testbed demonstrates the cost of skipping step-wise evaluation. Its judge, shown only the final reply, scores near-zero paired discrimination on faults that do not change the reply. These silent faults are the most dangerous kind, and they are invisible to outcome-only evaluation. A loud fault changes the reply and is at least detectable. A silent fault corrupts intermediate state while leaving the reply unchanged, and no output-level check can see it. Step-level visibility requires step-level state.

How fault injection exposes resilience gaps that only appear under realistic state

Systematic fault injection into tool-call responses reveals failure modes that never appear under nominal testing. The trajectory-judge testbed uses a deterministic support-desk environment with a scripted oracle and a one-step fault injector. The injector makes one edit to an oracle run at a known step and replays the edited calls. All 400 trajectories carried an exact label at no annotation cost, which is only possible because the environment is stateful and the oracle is correct by construction. This methodology generalizes beyond the judge evaluation that motivated it.

The most consequential faults are those that return a normal HTTP 200 while corrupting intermediate state. They pass every output-level check while leaving the agent in a world that diverges from what the trajectory assumes. Rate-limit responses, HTTP 429, and server errors, HTTP 503, are also first-class testing concerns. An agent that cannot handle transient tool failures gracefully will fail in production in ways that are impossible to reproduce against a stateless mock that always returns success. Variable tool response latency is another dimension that only a controllable environment can expose. Timeout handling and retry logic are behaviors, not outputs, and they change depending on how long a tool takes to respond. Testing them requires controlling the environment's behavior, not just its outputs.

What the RL environment use case reveals about the infrastructure requirements at scale

Reinforcement learning over tool-using agents makes the stateful environment requirement non-negotiable. RL rollouts require thousands of independent, reproducible environment instances. Any shared mutable state or live API dependency makes that impossible. The synthetic-to-real gap, authentic tool semantics, auth, rate limits, schema drift, remains largely unsolved in RL environments. Most approaches either sacrifice realism for scale or sacrifice scale for realism. The run-to-verify paradigm states the requirement plainly: every step must be independently checkable against programmatic ground truth to generate verifiable rewards. This is only possible when the environment's state is observable and deterministic, so the reward signal reflects what actually happened rather than what was assumed to happen.

EnvSimBench's central insight pushes the same argument further. It recasts simulation as a fully observable MDP, providing the explicit before-state and action logic and requiring the model to predict the resulting state and observation. Each step is then independently verifiable against a programmatic label. Decoupling simulation fidelity from state tracking makes fidelity objectively measurable rather than assumed. The infrastructure burden is the practical constraint. Running thousands of containers for RL rollouts against live APIs is not feasible: rate limits, credentials, and cost make it prohibitive. A horizontally scalable stateful simulator that does not touch live credentials is the only practical architecture. Pre-built, continuously verified stateful simulators for commonly integrated services, payment processors, communication platforms, market data feeds, let teams focus on agent logic rather than maintaining faithful replicas.

Fidelity verification as the mechanism that keeps a stateful simulator trustworthy over time

A stateful simulator that is not continuously verified against the live API it represents will silently diverge. An agent that passes against a drifted simulator has been given false confidence. The verification methodology is therefore part of the environment, not a separate concern. API drift is not hypothetical: providers update schemas, change response shapes, add required fields, and alter error codes. Any simulator that is not tested against the real API before each release will eventually disagree with it on details that matter to agent behavior.

Verification depth is proportional to the breadth of the probe corpus. A simulator verified across a wide probe corpus covers more of the API surface than one verified against a narrow set of happy-path calls. Rystic publishes fidelity scores against real probe corpora for each supported API: Slack at 96.3% across 538 probes, GitHub at 91.4% across 452 probes, Kalshi at 91.7% across 361 probes, PayPal at 97.8% across 90 probes, Lob at 100.0% across 32 probes, Okta at 98.1% across 159 probes. These scores encode a meaningful argument. The gap between Lob at 100% across 32 probes and GitHub at 91.4% across 452 probes reflects both API surface complexity and probe corpus depth. Higher fidelity at lower probe counts is not a stronger result. It is an underexplored surface. Transparency in fidelity reporting, publishing scores, probe counts, and the methodology behind them, is what distinguishes a verified simulator from an assumed one. That transparency is the evidence base that makes a simulator a trustworthy foundation for CI and RL environments alike.

What a reproducible trajectory evaluation workflow looks like end to end

A reproducible trajectory evaluation workflow is a closed loop, not a collection of separate tools. A stateful environment, a scenario bank, step-wise and end-to-end scoring, and continuous fidelity verification each depend on the others. The scenario bank is the critical artifact. It is a fixed, versioned set of scenarios with known initial states. Every scenario defines the starting world, the agent's task, and the expected trajectory, so a score is always interpretable relative to a known baseline.

Evaluation gates run at two points. Pre-merge gates catch regressions before they land. Pre-promotion gates enforce higher thresholds before production. Step-wise trajectory matching is the primary signal. End-to-end task completion is the secondary. The loop closes when failing live traces become regression fixtures. Every previously fixed bug becomes a scenario in the bank, and after sustained operation that regression set encodes everything the team learned the hard way about how the agent fails. For teams building RL pipelines, the same stateful simulator that serves CI also serves rollout generation. The environment resets to a known state for each episode, reward signals are verifiable against programmatic ground truth, and the simulator scales horizontally without touching live credentials or rate limits. The same environment architecture serves both testing and training, which is the strongest argument for investing in a stateful, verified simulator rather than treating the two infrastructure problems as separate.