What can a trace tell you that the final answer cannot?
A trace records a request as individual steps, called spans. Separate the part that searched for a document from the part that wrote the answer. If the model only saw an old policy, changing the reply wording alone will leave the old source in place.
What changes if you run Phoenix yourself?
Your team sets up the trace collector and manages its storage, access and retention. Arize lists Phoenix separately from its AX hosted tiers. Budget for the system you run and for any scoring model you call. Check which attributes your integration sends before connecting live data.
What can you take from a trace to the next test?
A Phoenix dataset can hold an input and an optional reference answer. Keep your own approved case and expected result even if the original trace expires. Arize documents how to export Phoenix traces. Check fields and access before moving an export.
Which step gave the assistant an old policy?
Sketch a fictional returns assistant that finds a 30-day policy from an old document and repeats it accurately.
- Find the first discrepancyWrite the current fictional 14-day policy beside the retrieved 30-day one. Mark the search step for review.
- Keep a clean test caseSave only the fictional question and expected source or answer in the paper test. Leave personal details out of the trace plan.