Skip to content

Braintrust

Braintrust compares AI outputs across the same test cases so you can find which examples improved and which got worse after a change.

Visit Braintrust

How does Braintrust show what changed between two runs?

Keep one set of examples and one scoring rule while you change the prompt or model. Braintrust calls each saved run an experiment. Choose a trusted experiment as the baseline, then open cases whose scores changed to see the answers behind the numbers.

In a Braintrust project, the Playground lets you configure a prompt, model, dataset and scorer without changing your app. A scorer applies a rule to each answer. Testing the app's full behavior can require your developer to connect its actual function through the SDK. Enable tracing for the app steps you want to inspect.

Can two runs have the same score and fail different cases?

Try a paper comparison for a help form that labels messages Billing or Shipping. Use two fictional cases and keep the exact expected label with each case.

  1. Set the correct label
    Mark “Why was I charged twice?” as Billing and “Where is my parcel?” as Shipping. Give one point when the chosen label matches the expected label.
  2. Compare a baseline and a proposed change
    Imagine the baseline labels both examples Billing. The new version labels both Shipping. Write down each output beside its expected label. Shipping now passes, and Billing now fails.
  3. Decide which case needs attention
    Both runs get one of two correct. Check the billing row, which got worse even though the overall score stayed the same. Keep the expected labels fixed for the next comparison.

Where should you remove private data?

Clean inputs, answers, expected values and metadata before sending them. Hiding a field from teammates can leave its full value stored. Braintrust's project-log ingestion redaction covers certain text values and does not cover direct dataset or experiment writes or attachment contents.

If you try the workflow in an app, verify the disposable workspace, model connection and tracing destination before opening it. Use fictional data and disable live services or connect test services. Use the paper version if any of those connections are uncertain.

What counts toward Starter usage?

On November 12, 2026, Starter listed no platform fee, 1 GB of processed data and 10,000 scores per month. Processed data is the volume Braintrust receives for logs and experiments. Each recorded score counts separately. Usage above included limits is blocked unless you enable paid on-demand usage.

The listed Starter overage rates were US$4 per additional GB and US$2.50 per 1,000 additional scores. Built-in model credits and any separate model-provider bill are distinct from those meters. Starter requires a workspace owner with a work email or a payment method on file for access to built-in models. Spend alerts notify you but do not act as a hard cap.

Starter lists up to 14 days for logs and Playground data. Saved experiments have a separate retention period of up to 365 days on all plans.

What can you export and where can it be hosted?

With the required permissions, you can download selected dataset rows or experiment results from Braintrust. Experiment exports include inputs, outputs, scores and metadata. Check the files for private values before sharing them.

Braintrust offers a hosted service. Enterprise can keep application data in your cloud account under its bring-your-own-cloud option. Braintrust still runs the sign-in and management service, which receives some metadata and operational measurements. Check where any model or export integration sends data separately.