A demo that works once is easy. A system that keeps working, for thousands of real people, across languages, seasons and edge cases, is hard. The difference is evaluation and operations. This chapter shows how professionals measure AI properly, test language models, ship safely and notice when a model quietly starts to fail.
By the end of this chapter you can
- Read a confusion matrix and choose between accuracy, precision and recall.
- Evaluate an LLM application with test sets, rubrics, groundedness checks and red teaming.
- Describe the MLOps lifecycle and safe deployment patterns such as canary releases.
- Explain data drift and concept drift, and what to monitor after launch.
Why "95% accurate" can be meaningless
Imagine a model that screens 1,000 people for a disease that only 50 of them actually have. Every prediction falls into one of four boxes, together called a confusion matrix:
Swipe sideways to see the full diagram.
- Accuracy is the share of all predictions that are right. It is misleading whenever one outcome is rare (fraud, disease, defects).
- Recall (sensitivity) asks: of all the real cases, how many did we catch? Prioritise it when missing a case is dangerous, as in disease screening or child-safety alerts.
- Precision asks: of everything we flagged, how much was real? Prioritise it when false alarms are expensive or harmful, for example wrongly accusing someone of fraud.
- The F1 score combines precision and recall into one number when you need a balance.
Most models output a probability, and you choose the threshold that turns it into a decision. Lowering the threshold catches more real cases (higher recall) but raises more false alarms (lower precision). Choosing that trade-off is a business and ethical decision, so write down who made it and why.
Fairness is a metric too
An average can hide serious problems. A crop-disease model that is 92% accurate overall might be 97% accurate on photos from well-lit phones and 70% on photos from older, low-resolution phones, which often means poorer farmers. Always break results down by meaningful groups (slice-based evaluation): region, language, gender, device, age group. If one group is consistently worse off, you have found a fairness issue before your users find it for you.
Evaluating language-model applications
LLM outputs are open-ended text, so there is rarely a single right answer to compare against. Professionals combine several methods:
Your own test set
Hundreds of realistic questions from your actual users, including hard and unusual ones, with what a good answer must contain. Public benchmarks alone are not enough.
Rubrics and reviewers
Score answers against clear criteria (correct, complete, safe, polite, in the right language). Humans review samples; an LLM "judge" can scale scoring if you check that it agrees with people.
Groundedness
Check that every claim is supported by the retrieved sources, and that the system says "I don't know" when the sources are silent.
Red teaming
Deliberately try to break the system: jailbreaks, prompt injection, requests for harmful content, attempts to extract personal data, and abuse in local languages.
Beware benchmark contamination
Famous public test sets often end up inside a model's training data, so a high benchmark score can partly reflect memorisation. That is another reason to keep a private, regularly refreshed test set built from your own domain.
MLOps: the lifecycle after the notebook
MLOps (machine learning operations) applies software engineering discipline to AI: versioning, automated testing, controlled releases and continuous monitoring. It turns model building from a one-off experiment into a repeatable, auditable process.
Swipe sideways to see the full diagram.
Teams keep a model registry that records every version, its training data, its evaluation results and who approved it. New versions are released carefully:
Monitoring: models decay in the real world
Unlike ordinary software, an AI system can get worse without a single line of code changing, because the world it learned from has moved on.
Data drift
- The inputs change.
- Example: users switch to a new phone camera, or a new district with different crops joins.
- Detect by comparing today's input statistics with the training data.
Concept drift
- The relationship between inputs and the right answer changes.
- Example: a new pest appears, or a policy change alters who repays loans.
- Detect by tracking real outcomes and accuracy over time.
A good monitoring dashboard tracks: input drift, output distribution, accuracy on labelled samples, fairness by group, latency, cost per request, user complaints and safety incidents. Each metric needs an owner and an alert threshold, so a problem triggers action rather than sitting unnoticed in a chart.
What does an AI incident response look like?Go deeper
Treat serious AI failures like any other operational incident: detect (alert or user report), contain (switch to a fallback, pause the feature, or roll back), assess the harm and who was affected, fix the root cause, notify affected people and, where the law requires, regulators, and learn by updating tests so the same failure is caught next time. Public collections such as the AI Incident Database show how often similar failures repeat across organisations.
Ready-to-launch checklist
- Metrics chosen to match the real cost of each type of error, with the threshold decision documented.
- Results broken down by region, language, gender and device, with gaps explained or fixed.
- A private test set and red-team results, including attempts in local languages.
- A canary or shadow release plan, plus a tested rollback.
- Monitoring for drift, quality, fairness, cost and incidents, each with an owner and an alert.
Key takeaways
- Accuracy hides rare-event failures; choose recall or precision based on which mistake is more harmful, and document the threshold.
- Evaluate by group, not just on average: slice-based evaluation reveals fairness problems.
- Test LLM apps with your own test set, rubrics, groundedness checks and red teaming.
- MLOps makes AI repeatable and auditable; monitor for data and concept drift because models decay as the world changes.