Served-Policy Anchoring

Problem

Identical parameters do not make one policy. The two engines assign different probabilities to the same trajectory even when synchronised, so every batch of rollouts is off-policy with respect to the gradients computed from it, on every step of every run. The consequence is an objective misalignment: an update that improves the training-engine policy does not necessarily improve the inference policy, the one that is deployed. A reward curve can therefore rise for weeks while describing a policy no request is ever served by, and the run can destabilise or collapse without the metric that is being watched showing why.

Solution

Name the served policy as the optimisation target and instrument the distance to it. For each batch of rollouts, recompute the log-probability of the sampled tokens under the training engine and compare it against the log-probability the rollout engine recorded when it generated them. The resulting log-ratio distribution — its disagreement rate, its tail mass, and its trend across steps — is logged next to reward as a training metric with its own thresholds, so a rising mismatch is visible before the run destabilises. The correction is then sized to that measurement rather than assumed: importance weighting, a trust region contracted in proportion to observed staleness, or a reduction in update size when an early-warning signal fires. Where correctness outweighs generation throughput, the gap is removed instead of corrected by running the rollout engine in a batch-invariant mode and sharing one model definition between the two paths, which drives the disagreement to zero. Release is gated on the same principle: a candidate checkpoint is scored by re-running the evaluation through the inference engine, precision and decoding configuration that production will use, and that served-policy score, not the trainer's forward pass, is what is compared against the current release. Every reported number carries the engine name, version, precision and routing configuration it was produced under, because a threshold measured on one configuration does not transfer to another.

When to use

• Rollouts are generated by an inference engine that is not the engine computing gradients.
• Inference runs at lower precision, or with different kernels, batching or expert routing, than the training path.
• Rollout generation is decoupled from optimisation, so batches arrive with lag behind the weights being updated.
• The checkpoint being trained will be deployed through the same inference engine that produced its rollouts.