Model Evaluation — Explain Like I'm 5

The Test You Never Trained For

Imagine a student who memorizes every past exam their teacher ever gave. On test day, they ace the practice exam perfectly. But when the teacher writes new questions — slightly different from any in the past — the student fails completely.

Did the student actually learn the material? No. They memorized answers.

Machine learning models have the same problem. If you test a model on the same data you trained it on, it will look incredibly good. But that tells you nothing about whether it can handle new situations.

Good model evaluation means testing on data the model has never seen during training.

The Basic Setup

When building an AI model, you split your data into three groups:

• Training data: Used to teach the model (60-70%)
• Validation data: Used while tuning — checking how you’re doing (10-20%)
• Test data: Locked away; only used for the final grade (10-20%)

The test set gives you an honest grade. The model never saw it during training or tuning.

Different Metrics for Different Problems

Accuracy (“what fraction did I get right?”): Works when each type of mistake is equally bad. For classifying emails as spam or not-spam, accuracy tells you how often you’re correct.

Precision and Recall: When mistakes have different costs.

• A cancer diagnosis AI that misses cancer (false negative) is dangerous — high recall matters
• A fraud detection AI that flags too many legitimate transactions (false positive) annoys customers — high precision matters

F1 Score: Combines precision and recall into one number. Useful when both matter.

For AI assistants (LLMs): Accuracy on multiple-choice questions, human preference ratings, benchmark scores on specific tasks. This is harder than classification — how do you grade a written response?

One thing to remember: The most important rule of model evaluation is simple — test on data you never trained on. Everything else is about choosing the right way to measure “good” for your specific problem.