Serving-Stack Attestation

also known as Backbone Attestation Probe, Served-Model Verification, Endpoint Provenance Probe

Treat the served model, inference backend and generation defaults behind an endpoint as an unverified claim, and establish them by black-box behavioural probe recorded alongside every result.

Context

An agent reaches a hosted model through an API and names the model it wants in configuration. The party on the other side owns the whole serving stack: which checkpoint is loaded, at which precision, on which inference framework, with which default decoding settings, and which route in a pool of machines handles this particular request. None of that is returned in the response. Open-weight checkpoints re-hosted by resellers make the gap wider, because the reseller chooses quantization and framework freely, and the same name is served by many of them at different prices.

Problem

The model name in configuration is a claim written by the party whose costs fall when a cheaper checkpoint is served, and nothing in the response confirms it. A backbone can be substituted, quantized or wrapped without any visible change, and even a fully honest provider changes what the caller measures: under greedy, sampling-noise-free decoding, roughly 39 percent of the benchmark variability a practitioner sees out of the box can be attributed to the inference backend alone, and framework names and versions are almost never disclosed with a published score. The obvious audit is unusable on agentic traffic. Serving stacks discard text and expose only the structured action once the model calls a tool, so text-channel tests have nothing to measure on exactly the requests an agent generates, and where text does come back, provider-injected system prompts distort its distribution enough that such tests falsely accuse honest providers 67 percent and 53 percent of the time. The damage also hides from cheap evaluation: extreme fidelity loss showed little detectable association with single-shot accuracy while coinciding with a declining long-horizon pass rate as task exposure grew.

Forces

The endpoint's model name is the only identity signal returned, and it is authored by the party whose serving costs fall if a smaller or lower-precision checkpoint answers the request.
Text is the natural audit channel and the first thing the serving stack removes: once the model calls a tool, modern stacks expose only the structured action, so a text-channel test is blind on agentic traffic.
A text-channel test that does run is confounded by provider-injected system prompts; measured false-positive rates of 67 percent and 53 percent mean it accuses honest providers more often than it clears them.
The backend is not a neutral carrier of the checkpoint: about 39 percent of out-of-the-box benchmark variability under greedy decoding stems from the backend, so a model name alone cannot make two numbers comparable.
Cheap evaluation misses the harm — fidelity loss barely moved single-shot accuracy but tracked a falling pass rate on long-horizon tasks, which is where the failure is expensive.
A probe costs tokens and latency and has to be repeated whenever the route can change, while writing a model name into configuration costs nothing at all.

When to use

The endpoint is operated by someone else and nothing in the response makes the served checkpoint, precision or framework verifiable.
Traffic is agentic, so most requests end in a tool call and the text channel carries too little to compare.
Results are compared across time, across providers, or against published benchmark numbers.
An open-weight model is bought from a reseller, where quantization, framework and routing are the reseller's choices to make.
Long-horizon task success matters, since that is where fidelity loss shows up before single-shot accuracy moves.

When not to use

The model runs on infrastructure the caller controls, where the framework, precision and generation configuration can be read directly instead of inferred.
No reference behaviour for the claimed checkpoint can be obtained, so a probe can report change but never identity.
The workload is single-shot and low stakes, and the recurring probe costs more than the failure it would catch.
The provider already returns a signed per-response record of backbone, framework version and generation defaults that the caller has reason to trust.

Example

A team pins an open-weight 70B model in configuration and buys it from a reseller at a lower price than the reference host charges. For three weeks the nightly single-shot benchmark is unchanged and short tasks look fine, but multi-step jobs start failing further and further into the run. A probe against the endpoint shows the tool-call distribution no longer matches the same checkpoint the team runs on its own machine, and the reseller confirms it moved that route to a lower-precision copy. Nothing in the response payload had changed; the model name field still said exactly what the configuration said.

Diagram

flowchart TD A["Model name in configuration"] --> B["Claim, not a measurement"] C["Probe battery: policy-dependent tool choices"] --> D["Live endpoint"] D --> E["Observed action distribution"] C --> F["Claimed checkpoint, controlled host"] F --> G["Reference distribution"] E --> H{"Two-sample test separated?"} G --> H H -- "no" --> I["Record backend, version, precision, generation config"] H -- "yes" --> J["Escalate: pin provider / fall back / raise with provider"] J --> I I --> K["Re-run on next route snapshot"] K --> C

Solution

Therefore:

Build a probe on a channel that survives the agentic path. For tool-calling traffic that channel is the structured action: a fixed battery of prompts whose tool choice is genuinely policy-dependent yields a categorical distribution over tool names and arguments that is characteristic of the backbone and remains visible after the stack drops the text. Where no tool is involved, freeze everything the caller controls — one system prompt, fixed decoding parameters, a fixed seed where the API offers one — and compare the output distribution under that frozen context. Take a reference distribution from the claimed checkpoint run on infrastructure the caller controls, or from the provider's own behaviour at a recorded snapshot when the weights are closed, then apply a two-sample test that needs no token probabilities from the target API and report separation at a controlled false-positive rate. Re-run the probe per route snapshot rather than once at onboarding, because a pooled endpoint can change what answers without changing any version string. Independently of the verdict, attach the backend name and version, the precision, and the generation configuration actually in effect to every recorded result and every published benchmark number, so a later comparison is between like and like. Define in advance what a separated result triggers: pin a provider, fall back to another route, or open the finding with the provider, since a probe with no escalation path only produces logs.

What it gives you

Substitution becomes detectable on the traffic an agent actually generates, instead of only on a text channel the serving stack removes.
Honest providers stop being accused: the tool-call probe held a 7 percent false-positive rate under system-prompt injection against 67 percent and 53 percent for text-channel baselines, while separating every one of 630 evaluated substituted checkpoint pairs.
Every recorded result carries the backend and generation configuration it was produced under, so a regression can be attributed to the stack rather than charged to the model.
Long-horizon degradation that single-shot benchmarks miss gains a monitoring channel tied to the route that served the request.

What it costs you

The probe spends tokens and latency and must repeat per route snapshot, so its cost scales with how aggressively the provider routes across a pool.
A reference distribution needs access to the claimed checkpoint; with closed weights the only baseline is the provider's own past behaviour, which drifts for legitimate reasons and weakens the verdict.
Separation is evidence of difference, not of bad faith — a precision change, a framework upgrade and a deliberate substitution look alike from outside.
The battery has to be built from prompts whose tool choice really is policy-dependent; a battery where every backbone picks the same tool measures nothing while appearing to pass.
Recording backend and generation configuration everywhere adds fields to every result record and benchmark table, and partial adoption leaves comparisons looking sound while remaining incomparable.

What this pattern forbids. A configured model name must not be recorded as the identity of what served a request; identity is admitted only from a probe run against the live endpoint on a channel the serving stack still exposes, and no result may be compared with another without the backend, precision and generation configuration each was produced under.

And the patterns that stand alongside it, or against it —

complementsHidden Mode Switching✕— Anti-pattern: silently swap the underlying model between requests without disclosing the change to users or operators.
complementsLineage Tracking★★— Track which prompt version, model version, and data sources produced each agent output.
complementsDeterminism-Tiered Replay Gate·— Classify an agent into a reproducibility tier by re-running identical inputs, require the strictest decision-determinism tier for regulated decisions, and gate deployment and validation-sample size on the measured tier.
complementsConfident Inconsistency✕— Anti-pattern: in a regulated workflow the same query produces materially different outputs at different times, each looking correct and passing review, so the variance stays invisible unless outputs are deliberately re-run and compared across time.
complementsProvider Fallback★★— When one provider's API errors mid-stream, transparently switch to another provider while preserving state.
complementsProvider-String Routing★— Select the model and provider for a request through a single namespaced string (`provider/model`) backed by env-var credentials, so the caller specifies what to run with one parameter rather than a typed provider object.
complementsScaffold Ablation on Model Upgrade★— On each model upgrade, treat every harness component as an encoded assumption about a model weakness and ablate the components the new model no longer needs, gated by evals.
complementsServed-Policy Anchoring★— Treat the policy the inference engine actually serves, not the one the training engine computes, as the object being optimised and gated, and measure their probability agreement continuously.
complementsAs-Of Information-Set Pinning★— Bind every input a dated decision may read, including the model snapshot, retrieval corpus, reference data and table version, to the information set that existed at its as-of timestamp, and record the pin.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance

Source: patterns/serving-stack-attestation.md on GitHub · commit 804ea97 · view history
Added to catalog: 2026-09-09
Last updated: 2026-09-09
Contribute: open an issue or PR at github.com/agentpatternscatalog/patterns.