What should a test check before it can fail a release?

Choose one outcome the app should deliver. A test case supplies the input and output fields your metric uses. Decide the passing threshold and read the borderline examples with a person before making a failed score block a release.

Does a local test send data to a model?

DeepEval runs as a local framework, but many metrics ask an evaluation model to score an answer. The FAQ says most model-scored metrics default to OpenAI if no model is specified. Choose the evaluator deliberately and check which fields leave your machine.

What do you need to repeat the test next week?

Keep the test cases, metric settings, model choice, threshold and result file. DeepEval documents a local results folder. Confident AI is the separate shared platform, so check any configured upload and model costs independently.

Should a polite answer pass if it invents a delivery date?

Write a fictional customer question with no order ID. Compare a reply asking for the ID with one inventing a delivery date.

Choose the useful pass rule
Mark “asks for the order ID without promising a date” as the condition.
Keep a second check
Decide whether a person should also check the reply is understandable. Keep that judgment separate from the required condition.