A generative model can sound confident and still be wrong. Evaluation is how you measure that gap before a system reaches a customer, a committee, or a filing. Responsible use is the set of limits you keep around it after it is live.
Hallucination
A hallucination is an output that is fluent but not supported by the source or by the facts. Typical forms are a made-up citation, a policy clause that does not exist, a rounded number that was never in the data, or a confident answer when the documents do not contain one.
What to measure
| Check | Question it answers |
|---|---|
| Faithfulness | Is every claim supported by the retrieved text? |
| Relevance | Did the answer address the question that was asked? |
| Refusal | Does the system say “not in the source” instead of guessing? |
| Format | Did it return the fields, length, and tone you specified? |
| Safety | Did it avoid leaking private data or giving a disallowed recommendation? |
Build a fixed set of questions with known good answers, including cases where the right behavior is to refuse. Run that set whenever you change the prompt, the index, or the model. A single impressive demo is not an evaluation.
Practical controls
- Require citations that point back to a chunk a reviewer can open.
- Keep private data out of prompts and logs that you do not control.
- Separate drafting from approval. A model may draft a credit note; a person signs it.
- Tell the user when an answer came from a model.
- Watch live traffic for questions the test set never covered.
A practical example
A policy assistant is tested on 50 questions. On 40, the cited paragraph supports the answer. On 6, it answers from memory and the citation does not match. On 4, the policy is silent and it should have said so. Those 10 failures are the work list: tighten retrieval, and add an instruction to abstain when the source is missing.
What to remember
- Fluency is not evidence. Check claims against a source.
- Evaluate on a fixed set, including questions the system should refuse.
- Keep a person responsible for decisions that affect customers, money, or compliance.
