Serving-Stack Attestation

Problem

The model name in configuration is a claim written by the party whose costs fall when a cheaper checkpoint is served, and nothing in the response confirms it. A backbone can be substituted, quantized or wrapped without any visible change, and even a fully honest provider changes what the caller measures: under greedy, sampling-noise-free decoding, roughly 39 percent of the benchmark variability a practitioner sees out of the box can be attributed to the inference backend alone, and framework names and versions are almost never disclosed with a published score. The obvious audit is unusable on agentic traffic. Serving stacks discard text and expose only the structured action once the model calls a tool, so text-channel tests have nothing to measure on exactly the requests an agent generates, and where text does come back, provider-injected system prompts distort its distribution enough that such tests falsely accuse honest providers 67 percent and 53 percent of the time. The damage also hides from cheap evaluation: extreme fidelity loss showed little detectable association with single-shot accuracy while coinciding with a declining long-horizon pass rate as task exposure grew.

Solution

Build a probe on a channel that survives the agentic path. For tool-calling traffic that channel is the structured action: a fixed battery of prompts whose tool choice is genuinely policy-dependent yields a categorical distribution over tool names and arguments that is characteristic of the backbone and remains visible after the stack drops the text. Where no tool is involved, freeze everything the caller controls — one system prompt, fixed decoding parameters, a fixed seed where the API offers one — and compare the output distribution under that frozen context. Take a reference distribution from the claimed checkpoint run on infrastructure the caller controls, or from the provider's own behaviour at a recorded snapshot when the weights are closed, then apply a two-sample test that needs no token probabilities from the target API and report separation at a controlled false-positive rate. Re-run the probe per route snapshot rather than once at onboarding, because a pooled endpoint can change what answers without changing any version string. Independently of the verdict, attach the backend name and version, the precision, and the generation configuration actually in effect to every recorded result and every published benchmark number, so a later comparison is between like and like. Define in advance what a separated result triggers: pin a provider, fall back to another route, or open the finding with the provider, since a probe with no escalation path only produces logs.

When to use

• The endpoint is operated by someone else and nothing in the response makes the served checkpoint, precision or framework verifiable.
• Traffic is agentic, so most requests end in a tool call and the text channel carries too little to compare.
• Results are compared across time, across providers, or against published benchmark numbers.
• An open-weight model is bought from a reseller, where quantization, framework and routing are the reseller's choices to make.
• Long-horizon task success matters, since that is where fidelity loss shows up before single-shot accuracy moves.