Harness-Native Rollout Capture

also known as In-Process LLM Proxying, Gateway-Captured Rollout, Token-Level Rollout Provenance

Capture reinforcement-learning rollouts by intercepting the model-call boundary inside the harness the agent already runs in, recording exact prompt and response token ids rather than reconstructing them from the transcript.

Context

A team post-trains an agent with reinforcement learning. In production the agent runs inside a harness — a long-running loop that manages tool integration, repository context, execution feedback and its own context budget. Rollouts for training can come from two places: a training-only reimplementation of that loop inside the trainer, or the real harness itself. The harness rewrites the conversation between turns; it compacts older messages, summarises sub-trajectories, truncates tool output and re-serialises the message list before each call. What the trainer sees afterwards is the rendered transcript, not the byte sequence the model was handed.

Problem

The prompt a trainer rebuilds from a transcript is not the prompt the model was asked. Harness-side compaction, summarisation and re-serialisation change which turns are present and how they are written, so the reconstructed token sequence diverges from the one that was generated against. Policy-gradient methods then recompute log-probabilities over tokens the model never saw, and the update is off-policy in a way no metric reports. The alternative — reimplementing the agent loop as a training-only environment — removes the mismatch by optimising a loop that is not the one shipped, so the two drift apart as the production harness evolves. Environmental crashes and reward hacking add a second corruption: a failed episode reaches the trainer as a low-reward rollout rather than as a discarded one.

Forces

The harness must keep rewriting the conversation to stay inside a context window, and every rewrite widens the gap between the transcript and what was actually sent.
A training-only reimplementation of the loop is easy to instrument but optimises a policy for an environment that never serves a request, while the production harness is faithful but was not built to emit training data.
Token ids are the only representation that survives re-serialisation; text survives it in appearance only, because a re-tokenisation can shift boundaries without changing a single visible character.
Interception at the model-call boundary leaves the harness's control flow untouched, but it also inherits the harness's failures — a crashed or reward-hacked episode is recorded as faithfully as a good one.
Rollouts are wanted as retained assets that can be resampled across runs, yet harnesses treat interaction traces as temporary runtime logs that are rotated away.

When to use

Rollouts are produced by a long-running harness that compacts, summarises, truncates or re-serialises the conversation between turns.
The training target is the agent as it runs in production, and maintaining a separate training-only copy of the loop would let the two drift apart.
The update method needs token-level alignment, such as a policy-gradient method that recomputes log-probabilities over the sampled tokens.
The harness is third-party or otherwise not open to modification, but its inference endpoint can be redirected.

When not to use

The interaction is a single turn against a fixed prompt, where the transcript and the sent prompt are the same object.
The harness exposes no model-call boundary and cannot be pointed at a proxy endpoint, so there is nothing to intercept without rewriting it.
Traces are wanted only for debugging or regression scoring, where a rendered transcript is enough and replay tooling already covers the need.
Prompts routinely carry material that must not be retained, and no redaction or retention control can be applied to the rollout pool.

Example

A team fine-tunes a coding agent that runs inside an off-the-shelf harness. Halfway through a long task the harness compacts the older turns to stay inside the context window, so the transcript the trainer reads afterwards is not the prompt the model was handed. Gradients are computed against a prompt that never existed, and the reward curve climbs for a week while the deployed agent does not improve. Recording the token ids at the model-call boundary instead of rebuilding them removes the gap.

Diagram

flowchart TD A["Unmodified agent harness"] -->|"compaction, summarisation, re-serialisation"| B["Model call"] B --> C["Proxy at the LLM API boundary"] C --> D["Inference engine"] D --> C C --> E["Step record: prompt token ids, response token ids, reward, metadata"] E --> F["Rollout pool"] F --> G{"Crashed or reward-hacked episode?"} G -- "yes" --> H["Excluded from the update"] G -- "no" --> I["Trainer recomputes log-probs on the recorded tokens"] I --> J["Probability agreement check"] J --> K["Policy update"] A -.-> L["Rendered transcript"] L -.->|"reconstructed prompt differs"| M["Never used as a training source"]

Solution

Therefore:

Leave the harness alone and tap it. A proxy sits at the single point where the harness reaches an inference endpoint, forwards the request unchanged, and writes one step record per generation: the exact prompt token ids that were sent, the exact response token ids that came back, the reward once it is known, and metadata such as episode id, step index, model version and sampling configuration. The proxy adds no control flow of its own, so the harness keeps compacting and summarising exactly as it does in production; whatever it decided to send is what gets stored. Step records are collected into a pool that outlives the run, so an episode can be resampled, filtered or re-weighted later instead of being consumed once and discarded. The trainer reads token ids straight from the record and recomputes log-probabilities over them, never rebuilding a prompt from rendered text. The agreement between those recomputed values and the ones the serving path recorded at generation time becomes the health metric for the capture itself: a reported rollout-training probability correlation above 0.99 says the tap is faithful, and a drop says something in the path has started to re-tokenise or reorder. A filtering stage sits between the pool and the update, marking episodes that ended in a harness crash or that scored through reward hacking so they are excluded rather than learned from. Because the tap is at an API boundary rather than inside the loop, the same capture works across unmodified third-party harnesses; one reported setup trains through three of them without patching any.

What it gives you

The tokens used for the update are the tokens the model was given, so harness-side compaction, summarisation and re-serialisation no longer decouple rollout behaviour from the gradient.
Training runs against the harness that is actually shipped, so the policy is not tuned for a training-only reimplementation of the loop that drifts away from production.
The correlation between trainer-recomputed and generation-time log-probabilities becomes a single number that reports whether the capture is still faithful; above 0.99 has been reported as the healthy range.
Interaction traces become retained data assets organised as step-level records rather than runtime logs that are rotated away, so an episode can be resampled or re-weighted across runs.
Because the tap is an API boundary and not a loop rewrite, third-party harnesses can be trained through without being modified.

What it costs you

The proxy sees every prompt the harness assembles, including repository content and credentials pulled into context, so the rollout pool becomes a concentrated store of sensitive material with the harness's own access scope.
Token-id records for long-horizon episodes are much larger than text transcripts, and retaining them for resampling multiplies that cost across runs.
The proxy sits on the latency path of a live harness, so a slow or failing recorder degrades the agent it is meant to observe.
A harness with no single model-call entry point, or one that offers no supported way to redirect its endpoint, has to be given one before the capture can be installed at all.
The tap records failures as faithfully as successes, so a separate filtering stage for crashed and reward-hacked episodes is mandatory rather than optional.

What this pattern forbids. The trainer must not rebuild a prompt from the rendered transcript; only the token ids the proxy recorded at the model-call boundary may be consumed for a policy update, and the harness's control flow cannot be replaced by a training-only reimplementation of the loop.

And the patterns that stand alongside it, or against it —

complementsServed-Policy Anchoring★— Treat the policy the inference engine actually serves, not the one the training engine computes, as the object being optimised and gated, and measure their probability agreement continuously.
complementsJournaled LLM Call★— Record the output of every non-deterministic step on first execution and replay that recorded value during crash-recovery instead of re-invoking the model.
complementsReplay / Time-Travel★★— Re-run a past agent trace from any step with modified inputs/prompts/tools to debug or branch.
complementsEval Harness★★— Run a held-out dataset against agent versions to detect regressions and measure improvement.
complementsManaged Agent Runtime★— Offer the agent loop itself as a managed cloud primitive so a caller supplies a model, system prompt, and tools and the platform runs the orchestration in an isolated, session-scoped runtime.
complementsContext Folding·— Let the agent branch into a temporary sub-context for a subtask and fold it back into a short summary on completion, so a long-horizon task stays within a small active window.
complementsReward Hacking✕— Anti-pattern: optimise the agent against a single proxy metric and assume the metric remains a faithful proxy after optimisation pressure.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance

Source: patterns/harness-native-rollout-capture.md on GitHub · commit 4eec3f8 · view history
Added to catalog: 2026-09-09
Last updated: 2026-09-09
Contribute: open an issue or PR at github.com/agentpatternscatalog/patterns.