AI Engineering · Evaluation and Observability¶

Offline evaluation compares changes under specified dataset and grading conditions; production observability records real calls and quality signals. This module explains paired samples, holdout sets, confidence intervals, and judge bias, and distinguishes sampled traces from SLO metrics covering all eligible traffic. Neither an average score nor an error rate alone is enough to judge a system.