X · Governance & ObservabilityEmerging★

Simulated-User Rollout Evaluation

also known as User Simulator Eval, Dual-Control Rollout Evaluation, Synthetic Customer Agent Validation, Tau-Bench-Style Evaluation

Evaluate a conversational agent by pairing it with a second model that plays the user from a persona and a hidden goal, then score the final environment state rather than the transcript.

Context

A conversational or tool-using agent is being taken toward deployment in a domain that has its own database, its own tool APIs and its own policy document — retail returns, telecom support, retail banking. Quality in such a domain is a property of whole episodes rather than of single replies: the agent has to ask for the account number it was never given, notice that the stated travel dates contradict the booking on file, and refuse the discount the policy forbids. A team preparing a release needs to measure that property before real customers do, and needs to measure it again on every model, prompt or tool change.

Problem

A held-out dataset of fixed inputs cannot express a counterpart who withholds a detail until asked, misstates a date, changes their mind halfway through, or has to be talked through an action on their own device. Recruiting people to play that counterpart produces the right signal at the wrong cadence and cost, so it happens once a quarter rather than once a commit. Scoring the transcript instead of the outcome rewards an agent that sounds cooperative while leaving the database wrong, and a single run per task hides the fact that the same task succeeds on one attempt and fails on the next.

Forces

Multi-turn quality only appears against a partner who withholds, misstates and changes their mind, and a fixed list of inputs cannot hold that behaviour.
Human trials give the most trustworthy counterpart but cannot be re-run on every commit, while a model playing the user finishes an episode in minutes for a few cents.
The model playing the user is itself an instrument with error: on identical retail tasks, measured agent success moves by up to 9 percentage points depending only on which model plays the user.
Comparing the final database state against an annotated goal state is cheap and objective, but it only covers what the task annotation anticipated; judging the transcript covers the rest and is an opinion.
One episode per task is affordable and misleading, while repeating each task exposes inconsistency and multiplies the bill.

When to use

The agent's job spans several turns and depends on information the counterpart holds back until asked.
The domain has an inspectable end state — a database row, a ticket, a booking — that can be compared against an annotated goal.
Quality has to be re-measured on every model, prompt or tool change, at a cadence human testing cannot match.
The counterpart can also act on the shared state, so guiding a person through a step is part of what needs measuring.
The same episodes are wanted as a training environment as well as an evaluation.

When not to use

The task is single-turn and a held-out dataset of fixed inputs already expresses it.
There is no inspectable end state, so grading would fall back to opinion about the transcript.
The number is meant to stand as a human-user result and no validation against real people is planned.
The user population that matters speaks in ways the simulator is known to model poorly, and the score would be read as covering them.
The episode has irreversible side effects that cannot be run against a disposable copy of the environment.

Example

A team is about to ship a change to an airline support agent. Instead of replaying fifty saved questions, they start fifty episodes in which a second model plays a passenger who wants to move a flight, does not mention the second bag until asked, and has already spent one of two travel vouchers. Each episode runs to the end, and the score is whether the booking in the test database matches what the passenger actually wanted. Two personas that always passed now fail on the voucher split, which no saved question had covered.

Diagram

sequenceDiagram participant P as Persona + hidden goal participant U as User simulator participant A as Agent under test participant E as Shared environment participant G as State grader P->>U: persona, scenario, withheld facts loop until goal met, stopping rule, or turn budget U->>A: next user turn A->>E: domain tool call E-->>A: result A-->>U: reply U->>E: user tool call (dual control) end E->>G: final environment state G->>G: compare with annotated goal state G-->>P: outcome, repeated k times for consistency

Solution

Therefore:

Give the evaluation two policies instead of one. The agent under test keeps its usual tools and its usual policy document, and is not told that the episode is a test. Opposite it sits a second model instantiated from a persona — who is speaking, how they speak, what they will volunteer unprompted — and a scenario stating what they want and which facts stay back until asked. Where the domain lets the user act as well, such as resetting a router or toggling a setting on their own phone, the simulated user is given its own tool set and its own view of the shared state, so its behaviour is bounded by what the environment permits rather than by prompt wording alone. The two converse until the goal is met, a stopping rule fires, or the turn budget runs out. Grading then compares the final environment state — the database rows, the created ticket, the amended booking — against the goal state annotated with the task, so an episode that reads well but leaves the world wrong scores nothing. Each task is run several times and reported as a consistency figure over the repeats rather than as a single pass. Personas and scenarios are enumerated from named dimensions instead of free-form prompting, and the simulator is periodically checked against real people, because it is an instrument with a bias of its own.

What it gives you

Behaviour that only appears under a partner who withholds and changes their mind becomes measurable at commit cadence instead of once a release.
Grading the final environment state against an annotated goal state gives an objective outcome that a fluent but wrong episode cannot pass.
Repeating each task exposes run-to-run inconsistency a single pass rate conceals: capable function-calling agents were reported succeeding on under 50% of retail tasks and holding across eight repeats on under 25%.
Episodes are reproducible and cheap enough to re-run per model change, and the same harness doubles as a training environment for reinforcement learning.
Giving the simulated user its own tools surfaces a separate skill — guiding a person through an action — that single-control benchmarks never test, and agent scores drop sharply when moved from no-user to dual-control settings.

What it costs you

The simulated user is an unvalidated proxy: measured agent success rates shift by up to 9 percentage points across different models playing the user, on identical tasks.
Results are miscalibrated in a direction that varies with difficulty, underestimating agent performance on hard tasks and overestimating it on moderately difficult ones.
The proxy is not equally good for everyone: African American Vernacular English speakers show consistently worse success rates and larger calibration error than Standard American English speakers, and the gap widens with age.
State-based grading only covers what the annotation anticipated, so tone failures, policy leaks and needless escalations pass unless a separate transcript check is added.
Simulated users introduce conversational artefacts and surface different failure patterns than people do, so a clean scoreboard can sit alongside unaddressed deployment problems.
Annotating a goal state for every task is manual work, and a task whose goal state is wrong silently marks correct agents as failures.

What this pattern forbids. The agent under test must not be told that its counterpart is simulated, and the simulated user may not receive the annotated goal state or the agent's private reasoning. A rollout cannot be scored from the transcript alone: grading compares the final environment state against the annotated goal state. No single episode may stand as a task's result — each task is repeated and reported as a consistency figure — and a simulated-user number must not be published as a human-user result before the simulator has been checked against real people.

The smaller patterns that complete this one —

usesAgent Persona Profile★— Treat agent identity as a structured profile object — persona, primary motivator, allowed actions, knowledge bindings — rather than a free-form role sentence in the system prompt.

And the patterns that stand alongside it, or against it —

complementsEval Harness★★— Run a held-out dataset against agent versions to detect regressions and measure improvement.
complementsLLM-as-Judge★★— Use an LLM to score open-ended outputs against rubric criteria when no exact-match metric applies.
complementsAgent Evaluator★— A dedicated agent or harness whose sole job is running tests against another agent's outputs to evaluate performance; distinct from eval-harness (offline batch) and llm-as-judge (per-output).
complementsCAMEL Role-Playing·— Have two agents role-play a user-assistant interaction to autonomously complete a task neither could solve alone.
complementsDimensional Synthetic Eval Set★— Generate evaluation inputs not by free-form LLM prompting (which mode-collapses) but by enumerating tuples over explicitly named dimensions and seeding generation from each tuple.
complementsChecklist Partial-Credit Scoring★— Score each long-horizon task against task-specific graded checkpoints instead of a single pass/fail bit, so progress and regression that leave the binary outcome unchanged still show up in the report.
complementsChance-Corrected Judge Validation★— Measure an automated judge against human labels on a held-out meta-evaluation set with a chance-corrected agreement statistic, and publish that number beside every score the judge is allowed to gate.
complementsDual Evaluation (Offline + Online)★— Run two parallel evaluation tracks — offline benchmark gates before deploy AND online production-traffic monitoring after — so drift is caught even when pre-deploy benchmarks pass.