Did the change make your AI feature better?
Compare the old and new answers on the same examples with the same pass rules. Start on paper with invented data. A trace can help you find which recorded step led to a wrong answer.
What would a good answer do for your user?
Suppose your app answers questions about booking a class. Write a rule that says what each answer must get right. A reply about cancellations needs the correct deadline. A reply about a refund needs to say when the policy is missing and point the person to support.
An evaluation applies a rule to an answer. Save a small set of examples and expected results, called a dataset. Include ordinary questions, incomplete information and a mistake you would consider serious. Review the cases that require judgment yourself or ask someone allowed to see them.
Can you tell which step went wrong?
A trace records the steps your app reports for one attempt. Compare the source the app found with the answer it gave. If it found an old policy, fix what it searches. If it found the right policy and gave a different deadline, check how it used the source.
A missing step may mean your app did not record it. Keep the model or prompt version with each run so you know which version produced an answer. Use a known failed case to check the fix on the same input.
Compare tools
Scroll the table sideways to read every column.
| Tool | What would you check first?A useful starting task when a result is wrong or you are deciding whether to ship. | Where does it run?Who runs the service that receives the evaluation or trace data. | What gets counted?The usage units to check before starting a larger comparison. | What needs a privacy check?Where private content might be sent, stored or shared. |
|---|---|---|---|---|
| Langfuse | Inspect a wrong answer and save approved examples for repeatable tests. | Cloud or self-hosted open-source software. You run the self-hosted services. | Cloud counts each trace, recorded step and score. | Masking depends on the integration. Saved test examples can outlive traces. |
| Braintrust | Compare two saved experiments and open cases whose scores changed. | Hosted, or Enterprise app data in your cloud. Braintrust still runs sign-in and management. | Processed data in GB and recorded scores. | Attachments and direct dataset or experiment writes are outside log redaction. |
| LangSmith | Inspect recorded search and answer steps, then rerun a failed case. | Hosted or a licensed self-hosted Enterprise add-on. | Hosted counts base and extended traces. One trace can cover several steps. | In hosted LangSmith, a public dataset link exposes examples, experiments, runs and feedback to anyone with it. |
| Promptfoo | Use the same cases to compare two prompt versions against checks you choose. | Run Community locally. Hosted models and managed sharing are separate. | Count calls to the app models and any model scoring the answers. | Local execution can still send inputs to your chosen provider. Review exports before sharing. |
| Opik | Inspect the failed step and turn an approved example into a rerunnable test. | Comet-hosted or self-hosted with your own operations. | Cloud plans list spans and retention. Check evaluator model usage separately. | Copying a trace into a test includes its input and output. Display masking still leaves the original stored. |
| Arize Phoenix | Open the trace and find the first search, model or tool step that went wrong. | Run Phoenix yourself for control of its storage. AX is a separate offering. | Self-hosted spans, ingestion and retention use the resources you provision. | Check what your integration puts in each span before sending traces or exports. |
| DeepEval | Run the same test cases against a change and inspect which cases fail a threshold. | Local Python test framework. Confident AI is an optional separate platform. | Scoring calls can add model usage even when the framework runs locally. | A hosted evaluator can receive selected test fields. Check provider and upload settings. |
| Ragas | Separate finding useful source text from supporting the final reply with that text. | Python evaluation library in your own workflow. | Count cases, metrics and any evaluator model calls. | Document passages can enter model-based scoring. Use approved examples. |
| Agenta | The 1.0 playground documents comparing two instruction versions against the same input. Check your version. | Cloud or self-hosted. A configured model can still be remote. | The 2.0 cloud pricing uses credits for included models and agent runtime. | Review what the evaluator receives and what trace or test data is retained. |
| Confident AI | Turn an approved production failure into a repeatable test case. | Shared hosted platform. A local DeepEval test can run separately. | The trace calculator uses data size over time. Confirm evaluation and plan limits. | Check what logging collects and what an ingestion task copies into a dataset. |
How can you compare two versions on paper?
Write an invented class policy that allows cancellations until 24 hours before class and gives no refund terms. Use imaginary replies for both versions. Keep the exercise in your notes with no app, model account, upload or message.
- Write the pass rulesFor “Can I cancel two days before class?” require yes and the 24-hour deadline. For “Will I get a refund?” require an honest statement that the policy gives no terms and a way to ask support.
- Compare the individual repliesGive the old version an incorrect cancellation deadline and a cautious refund reply. Give the new version the right deadline and a promise of a refund. Each passes one check. A wrong promise is still a reason to hold the change.
- Trace the wrong promiseDraw two boxes for the source text and the answer. If the reply used the policy you wrote, circle the answer step. If it used a different page with refund terms, circle the search step. Keep the same test questions and change one thing for the next comparison.
What should stay out of a trace or test set?
Before opening a connected test app, confirm the workspace, model and document store are the test ones you intended. Turn off live messages and other actions or connect test services. Use invented inputs and files, and stay with the paper exercise if you cannot verify those connections.
Remove unnecessary private details before they leave the app, including inputs, outputs, notes and attachments. Check tracing, saved test examples and exports separately. A hidden field can still be stored. Before sending a link, check whether it requires your team's sign-in or opens to anyone who has it.
What should you count and keep?
Estimate how many requests you will record and how many scores you will save. The tool may charge for a request, a recorded step, a score or data volume. Model calls from the app and any automated judging may add separate costs.
Before changing services, check how to take out approved examples, expected answers and comparison results. Ask separately how long you can open a trace and when the service deletes it. A saved example may remain after the trace expires.