How can you tell which step went wrong?

Open the trace for one request. LangSmith calls a recorded step a run. Look at the run that found the source text, then at the run that used that text to write an answer. You can inspect the inputs and outputs your app chose to send.

Your developer connects the relevant server code to a LangSmith workspace or tracing project and records the steps you need. The SDK supports custom code as well as LangChain applications. Keep the secret key on the server. A test dataset with a reference answer gives you a way to rerun the case after a fix.

What would you check in two traces with the same wrong answer?

Use a fictional community center. Its class guide says doors open at 9 a.m. Its staff guide says staff arrive at 8 a.m. The test question is “When do class doors open?” The expected answer is 9 a.m.

Read the source step first
On paper, imagine the first trace shows the app found only the staff guide, then answered 8 a.m. Mark the search step as failing to find the relevant class guide.
Read the answer step separately
Imagine the second trace shows the app found the right line in the class guide but still answered 8 a.m. Mark the answer as inconsistent with the text it received.
Write a repeatable test case
Keep the question, expected time and allowed source with your case. After a proposed change, you would check that the app found the class guide and gave the correct time. This exercise uses no real records or model calls.

What should stay out of the trace and dataset?

Leave out customer records, secrets and private documents that the investigation does not need. Configure the tracing library to omit or redact inputs, outputs and metadata before sending them. Verify that the exact test workspace receives only fictional data and that any model, document store or other connected service is a test service or is disabled.

Use the paper exercise if you cannot verify those connections before running the app. Copy only approved fields into a dataset. Its examples can remain after their source traces expire. In hosted LangSmith, anyone with a publicly shared dataset link can see the examples, experiments and their associated runs, and feedback. Check the sharing setting and review those records before you send a link.

What does hosted tracing count and how long does it keep a trace?

LangSmith counts a trace for an operation that can contain several runs. On November 12, 2026, the hosted Developer plan listed one seat with 5,000 base traces per month included. With no payment method, the plan stops accepting new traces at its monthly 5,000-trace limit. The price list shows pay-as-you-go tracing beyond included usage when billing is enabled.

Base traces are kept for 14 days. The current default for new hosted extended traces is 180 days. Experiments use extended retention by default, and online evaluators and automation rules can extend traces and add cost when that setting is enabled. Model-provider and deployment charges can be separate.

Which evaluation records can you take with you?

Download dataset examples as CSV or JSONL from the dataset page. Keep the expected answers and source notes you need for your test. A dataset download is a copy of examples. Check separately which trace details you still need before they expire.

LangSmith offers a separate self-hosted Enterprise add-on that runs in your own Kubernetes cluster. It requires a license and its own infrastructure and operations. The hosted trace allowances above apply to the hosted Developer plan.