Served-Policy Anchoring

also known as Training-Inference Mismatch Control, Rollout-Engine Policy Gap, Inference-Policy Objective

Treat the policy the inference engine actually serves, not the one the training engine computes, as the object being optimised and gated, and measure their probability agreement continuously.

Context

An agent is post-trained with reinforcement learning on a hybrid stack. Trajectories are generated by a throughput-tuned inference engine such as vLLM or SGLang, which uses fused low-precision kernels, continuous batching and its own expert-routing decisions. Gradients are computed by a separate training engine such as Megatron or a sharded-data-parallel backend at higher precision, with a different kernel path. Weights are copied between the two after every update, so both hold identical parameters, and the same inference engine will later serve production traffic.

Problem

Identical parameters do not make one policy. The two engines assign different probabilities to the same trajectory even when synchronised, so every batch of rollouts is off-policy with respect to the gradients computed from it, on every step of every run. The consequence is an objective misalignment: an update that improves the training-engine policy does not necessarily improve the inference policy, the one that is deployed. A reward curve can therefore rise for weeks while describing a policy no request is ever served by, and the run can destabilise or collapse without the metric that is being watched showing why.

Forces

The inference engine is chosen for generation throughput — low-precision quantisation, fused kernels, continuous batching — and each of those choices is a source of numerical divergence from the higher-precision path that computes gradients.
An effective update to the policy in the training engine does not necessarily improve the inference policy, so optimising the measurable quantity and optimising the deployed artefact are not the same task.
The gap is not a static numerical offset but a dynamic failure coupled to optimisation: gradient noise and mismatch escalate together as training progresses, so standard importance-sampling corrections that hold early can fail during extended runs.
The gap can be eliminated rather than corrected — under batch-invariant kernels the measured disagreement rates fall to exactly zero — but determinism is bought with throughput, so the size of the gap is a purchased quantity rather than a fixed fact of the stack.
Decoupling rollout generation from optimisation raises throughput and adds staleness compounded by policy lag, engine delays and expert routing, while clipping constrains only sampled outward updates and so acts as a sampled surrogate rather than a full-policy constraint.

When to use

Rollouts are generated by an inference engine that is not the engine computing gradients.
Inference runs at lower precision, or with different kernels, batching or expert routing, than the training path.
Rollout generation is decoupled from optimisation, so batches arrive with lag behind the weights being updated.
The checkpoint being trained will be deployed through the same inference engine that produced its rollouts.

When not to use

Generation and gradients share one engine, one precision and one kernel path, so there is no gap between two runtimes to measure.
The model is consumed only through a third-party API and no post-training loop is owned, in which case attesting the serving stack is the available remedy instead.
The run is short enough to finish before mismatch and gradient noise escalate, and the throughput lost to instrumentation would cost more than the accuracy it recovers.

Example

A team post-trains a tool-using agent with reinforcement learning and watches the reward curve climb steadily for three weeks. The checkpoint is promoted, and in production the agent performs no better than the one it replaced. The rollouts had been generated by the throughput-tuned inference server while the gradients came from the training framework, and the two assigned different probabilities to the same trajectories, so the curve described a policy that never answered a single request.

Diagram

flowchart TD W["Synchronised weights"] --> R["Rollout engine: low precision, batched, own routing"] W --> T["Training engine: higher precision, different kernels"] R --> TR["Trajectories + recorded log-probs"] TR --> T T --> G{"Per-token probability gap"} G -- "within band" --> U["Apply update"] G -- "rising / tail heavy" --> C["Correct: importance weight, staleness-scaled trust region, smaller update"] C --> U G -- "fidelity over throughput" --> B["Eliminate: batch-invariant kernels, shared model definition"] B --> U U --> RG{"Release gate"} RG -- "score from trainer forward pass" --> X["Rejected: policy nobody serves"] RG -- "score reproduced through serving path" --> S["Promote checkpoint"]

Solution

Therefore:

Name the served policy as the optimisation target and instrument the distance to it. For each batch of rollouts, recompute the log-probability of the sampled tokens under the training engine and compare it against the log-probability the rollout engine recorded when it generated them. The resulting log-ratio distribution — its disagreement rate, its tail mass, and its trend across steps — is logged next to reward as a training metric with its own thresholds, so a rising mismatch is visible before the run destabilises. The correction is then sized to that measurement rather than assumed: importance weighting, a trust region contracted in proportion to observed staleness, or a reduction in update size when an early-warning signal fires. Where correctness outweighs generation throughput, the gap is removed instead of corrected by running the rollout engine in a batch-invariant mode and sharing one model definition between the two paths, which drives the disagreement to zero. Release is gated on the same principle: a candidate checkpoint is scored by re-running the evaluation through the inference engine, precision and decoding configuration that production will use, and that served-policy score, not the trainer's forward pass, is what is compared against the current release. Every reported number carries the engine name, version, precision and routing configuration it was produced under, because a threshold measured on one configuration does not transfer to another.

What it gives you

The number that gates a release describes the artefact that ships, so a training curve cannot rise for a policy that is never served.
Instability becomes visible as a rising mismatch metric before the reward curve collapses, giving the run an early-warning signal rather than a post-mortem.
The correction term is sized to a measured quantity instead of a guessed one, and reported results carry the engine, precision and routing configuration that produced them.
The choice between correcting the gap and eliminating it with batch-invariant kernels becomes an explicit trade of throughput against fidelity rather than an unexamined default.

What it costs you

Recomputing log-probabilities for every rollout batch under the training engine costs an extra forward pass per step, and batch-invariant inference costs generation throughput, so the discipline is paid for in wall-clock time.
A threshold set too tight stalls training on mismatch that the correction would have absorbed, and one set too loose admits the collapse it was meant to catch.
The metric measures agreement between two runtimes, not correctness: the engines can agree closely on a policy that is getting worse.
The size of the gap depends on model, precision, batching and hardware, so a threshold calibrated on one configuration is not evidence about another and has to be re-measured after any engine or precision change.

What this pattern forbids. A checkpoint must not be promoted on a score the training engine produced; the released number is admitted only from a run through the serving engine, precision and decoding configuration production will use, and no rollout batch may be consumed as on-policy without its measured training-inference probability gap.

And the patterns that stand alongside it, or against it —

complementsServing-Stack Attestation★— Treat the served model, inference backend and generation defaults behind an endpoint as an unverified claim, and establish them by black-box behavioural probe recorded alongside every result.
complementsDeterminism-Tiered Replay Gate·— Classify an agent into a reproducibility tier by re-running identical inputs, require the strictest decision-determinism tier for regulated decisions, and gate deployment and validation-sample size on the measured tier.
complementsConfident Inconsistency✕— Anti-pattern: in a regulated workflow the same query produces materially different outputs at different times, each looking correct and passing review, so the variance stays invisible unless outputs are deliberately re-run and compared across time.
complementsEval Harness★★— Run a held-out dataset against agent versions to detect regressions and measure improvement.
complementsDual Evaluation (Offline + Online)★— Run two parallel evaluation tracks — offline benchmark gates before deploy AND online production-traffic monitoring after — so drift is caught even when pre-deploy benchmarks pass.
complementsRL-Trained Conductor Orchestrator·— Train a small meta-model with reinforcement learning to dispatch sub-tasks across a pool of frontier LLM workers, learning the communication topology end-to-end and allowing the conductor to recursively invoke itself as a worker.
complementsHarness-Native Rollout Capture★— Capture reinforcement-learning rollouts by intercepting the model-call boundary inside the harness the agent already runs in, recording exact prompt and response token ids rather than reconstructing them from the transcript.