Did the app find the right text, and did it use it?
Context precision checks how useful the retrieved passages are in the order returned. Faithfulness checks whether the reply’s claims are supported by those passages. A reply can faithfully repeat an old rule if the app retrieved an old document. Check both the passages and the reply.
What does the evaluator need to see?
Keep each question, the answer and the passages the app retrieved. Some scoring methods also need a reference answer or an evaluation model. Check which fields your selected metric uses, where it sends them and what calls the model provider may charge for.
What should you save so someone can compare another version?
Keep the same approved examples and metric setup when you change the app. The documented local experiment template saves CSV results. Keep the date and version of the documents as well, so a changed result can be traced back to an actual change.
Could an answer be faithful and still wrong?
Draw a fictional shipping guide with a retired price and a current price. Give an assistant only the old guide and write the answer it would return.
- Check the retrieved textCircle the passage it saw. Mark whether this is the right version of the guide.
- Check the answer separatelyWrite whether the reply accurately repeats that passage. Keep this finding separate from whether it gives a useful current price.