Evaluate LLM behavior and hallucination risk
Section titled “Evaluate LLM behavior and hallucination risk”LLM output can be fluent and plausible while being unsupported or incorrect. Fluency is not a reliability signal.
flowchart LR
Cases[Representative evaluation cases] --> Run[Run system]
Run --> Grade[Deterministic and calibrated grading]
Grade --> Failures[Classify failures]
Failures --> Improve[Change model, prompt, retrieval, tools, or schema]
Improve --> Run
The evaluation loop should track application behavior, not only a model version in isolation.
Evaluation
Section titled “Evaluation”Build an evaluation set from representative tasks, difficult cases, known failures, and important invariants. Measure the behavior that matters to the application rather than only generic benchmark scores.
Use deterministic checks where possible. Add human or model-based grading only for qualities that cannot be reduced to a reliable rule, and calibrate those graders against examples.
Hallucination controls
Section titled “Hallucination controls”Require external verification for claims that can be checked against authoritative data. Prefer tools or retrieval when the answer depends on current state.
For high-impact actions, separate generation from authorization and execution.
Regression testing
Section titled “Regression testing”Treat model, prompt, retrieval, tool, and schema changes as behavior changes. Re-run evaluations and compare failure categories before deployment.
No evaluation set proves universal correctness. Maintain it as production failures reveal new cases.
Sources
Section titled “Sources”- National Institute of Standards and Technology. AI Risk Management Framework.