Harness-Native Rollout Capture

Problem

The prompt a trainer rebuilds from a transcript is not the prompt the model was asked. Harness-side compaction, summarisation and re-serialisation change which turns are present and how they are written, so the reconstructed token sequence diverges from the one that was generated against. Policy-gradient methods then recompute log-probabilities over tokens the model never saw, and the update is off-policy in a way no metric reports. The alternative — reimplementing the agent loop as a training-only environment — removes the mismatch by optimising a loop that is not the one shipped, so the two drift apart as the production harness evolves. Environmental crashes and reward hacking add a second corruption: a failed episode reaches the trainer as a low-reward rollout rather than as a discarded one.

Solution

Leave the harness alone and tap it. A proxy sits at the single point where the harness reaches an inference endpoint, forwards the request unchanged, and writes one step record per generation: the exact prompt token ids that were sent, the exact response token ids that came back, the reward once it is known, and metadata such as episode id, step index, model version and sampling configuration. The proxy adds no control flow of its own, so the harness keeps compacting and summarising exactly as it does in production; whatever it decided to send is what gets stored. Step records are collected into a pool that outlives the run, so an episode can be resampled, filtered or re-weighted later instead of being consumed once and discarded. The trainer reads token ids straight from the record and recomputes log-probabilities over them, never rebuilding a prompt from rendered text. The agreement between those recomputed values and the ones the serving path recorded at generation time becomes the health metric for the capture itself: a reported rollout-training probability correlation above 0.99 says the tap is faithful, and a drop says something in the path has started to re-tokenise or reorder. A filtering stage sits between the pool and the update, marking episodes that ended in a harness crash or that scored through reward hacking so they are excluded rather than learned from. Because the tap is at an API boundary rather than inside the loop, the same capture works across unmodified third-party harnesses; one reported setup trains through three of them without patching any.

When to use

• Rollouts are produced by a long-running harness that compacts, summarises, truncates or re-serialises the conversation between turns.
• The training target is the agent as it runs in production, and maintaining a separate training-only copy of the loop would let the two drift apart.
• The update method needs token-level alignment, such as a policy-gradient method that recomputes log-probabilities over the sampled tokens.
• The harness is third-party or otherwise not open to modification, but its inference endpoint can be redirected.