Conversational Feedback Extraction Loop
Turn noisy in-chat behaviour, such as regenerations, edits, deletes, and thumbs, into a clean feedback stream that drives the evaluation and improvement loop.
Description
Collect the signals users give off in a chat-style LLM application and feed them to your evaluation pipeline. The signals include leaving early, asking for a regeneration, fixing an error, and how users organise chats, such as share, save, and delete. A chat interface produces lots of feedback, but it is messy. Without a deliberate loop, the signal gets logged and never read. The thing you keep is a per-turn feedback stream with each signal labelled as explicit or implicit. That stream then feeds test-set curation and a list of fine-tuning candidates.
When to apply
Use this for any live chat-style LLM application, such as a chatbot, agent, coding assistant, or voice agent, where users take several turns and you can see what they do. Apply it early. If you bolt the schema on after launch, you lose all the earlier signal. Do not apply it to single-shot endpoints where users see one response and leave, because the signal is too thin. One exception: even single-shot endpoints can capture thumbs and whether the task got done. Treat those as a stripped-down one-turn case.
What it involves
- Instrument the UI affordances
- Author the feedback schema
- Stream events to the feedback pipeline
- Compute aggregate signals
- Surface negative-signal turns for review
- Close the loop into evaluation
Open the full interactive page →
Diagram, neighbourhood map, code examples, related patterns and full provenance.