Simulated-User Rollout Evaluation

Problem

A held-out dataset of fixed inputs cannot express a counterpart who withholds a detail until asked, misstates a date, changes their mind halfway through, or has to be talked through an action on their own device. Recruiting people to play that counterpart produces the right signal at the wrong cadence and cost, so it happens once a quarter rather than once a commit. Scoring the transcript instead of the outcome rewards an agent that sounds cooperative while leaving the database wrong, and a single run per task hides the fact that the same task succeeds on one attempt and fails on the next.

Solution

Give the evaluation two policies instead of one. The agent under test keeps its usual tools and its usual policy document, and is not told that the episode is a test. Opposite it sits a second model instantiated from a persona — who is speaking, how they speak, what they will volunteer unprompted — and a scenario stating what they want and which facts stay back until asked. Where the domain lets the user act as well, such as resetting a router or toggling a setting on their own phone, the simulated user is given its own tool set and its own view of the shared state, so its behaviour is bounded by what the environment permits rather than by prompt wording alone. The two converse until the goal is met, a stopping rule fires, or the turn budget runs out. Grading then compares the final environment state — the database rows, the created ticket, the amended booking — against the goal state annotated with the task, so an episode that reads well but leaves the world wrong scores nothing. Each task is run several times and reported as a consistency figure over the repeats rather than as a single pass. Personas and scenarios are enumerated from named dimensions instead of free-form prompting, and the simulator is periodically checked against real people, because it is an instrument with a bias of its own.

When to use

• The agent's job spans several turns and depends on information the counterpart holds back until asked.
• The domain has an inspectable end state — a database row, a ticket, a booking — that can be compared against an annotated goal.
• Quality has to be re-measured on every model, prompt or tool change, at a cadence human testing cannot match.
• The counterpart can also act on the shared state, so guiding a person through a step is part of what needs measuring.
• The same episodes are wanted as a training environment as well as an evaluation.