Braintrust
Braintrust is a platform for evaluating LLM applications offline against datasets and scorers and for scoring production traces online without adding latency.
Description
Braintrust is a platform for building, evaluating, and monitoring LLM applications. Offline, each evaluation runs a task against a dataset of test cases and measures the output with scoring functions, producing an experiment that records inputs, outputs, scores, and metadata for comparing prompt and model versions. Online, configured scorers run against production traces automatically as they are logged, evaluating asynchronously in the background. This gives both pre-deploy regression checks and continuous production quality monitoring.
Solution
Braintrust evaluates rather than runs the application. Offline, an Eval pairs a dataset, a task function under test, and scoring functions; running it produces an experiment that records inputs, outputs, and scores for comparison across versions. Online, scorers run asynchronously against production traces as they arrive, logging quality metrics without regenerating output or adding latency to the application.
Primary use cases
- offline evaluation of LLM applications against datasets
- comparing prompt and model versions across experiments
- catching regressions in CI
- online scoring of production traces
Open the full interactive page →
Diagram, neighbourhood map, code examples, related patterns and full provenance.