Model Evaluation Metrics in Python — Core Concepts

Why One Metric Is Not Enough

Every metric answers a specific question. Choosing the wrong one can make a bad model look good — or a good model look bad. The right choice depends on what your application cares about most.

Classification Metrics

Accuracy

The percentage of correct predictions. Works well when classes are balanced (roughly equal numbers of each category) but fails badly on imbalanced datasets. A fraud detector that always says “not fraud” on a dataset with 0.1 percent fraud achieves 99.9 percent accuracy while catching zero actual fraud.

Precision

Of all the times the model said “yes,” how often was it right?

Precision = True Positives / (True Positives + False Positives)

High precision matters when false alarms are expensive. A spam filter with low precision deletes legitimate emails. An alert system with low precision causes alert fatigue.

Recall (Sensitivity)

Of all the actual “yes” cases, how many did the model catch?

Recall = True Positives / (True Positives + False Negatives)

High recall matters when missing a positive case is dangerous. A cancer screening tool with low recall misses sick patients. A security system with low recall fails to detect real threats.

F1 Score

The harmonic mean of precision and recall. It balances both concerns into a single number:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

The F1 score is useful when you want a single metric that penalizes models with an extreme gap between precision and recall.

The Precision-Recall Tradeoff

You can usually increase precision by making the model more conservative (only say “yes” when very confident), but this lowers recall. Conversely, a model that says “yes” more freely catches more true cases but also makes more false alarms. The optimal balance depends on the business cost of each type of error.

Regression Metrics

Mean Absolute Error (MAE)

The average of absolute differences between predictions and actual values. Easy to interpret: “on average, the prediction is off by X units.”

Root Mean Squared Error (RMSE)

Like MAE but penalizes large errors more heavily because it squares the differences before averaging. If a few big misses are much worse than many small ones, RMSE is more informative.

R² (Coefficient of Determination)

Measures how much variance the model explains compared to a baseline that always predicts the mean. An R² of 0.85 means the model explains 85 percent of the variability. An R² of 0 means it is no better than guessing the average.

Choosing the Right Metric

Problem Type	Priority	Recommended Metric
Balanced classification	Overall correctness	Accuracy or F1
Imbalanced classification	Catching positives	Recall, then F1
Imbalanced classification	Avoiding false alarms	Precision, then F1
Regression, uniform errors	Average error size	MAE
Regression, big errors costly	Penalize outliers	RMSE
Ranking or probability	Discrimination quality	AUC-ROC

Common Misconception

“High accuracy means the model is good.” On imbalanced datasets, accuracy is nearly meaningless. A model predicting the majority class every time can score above 95 percent accuracy while having zero predictive value for the minority class. Always pair accuracy with precision, recall, or F1.

Practical Tips

• Start by asking: “What is the cost of a false positive vs. a false negative?” The answer guides your metric choice.
• Report multiple metrics, not just one. Stakeholders need the full picture.
• Use the same metric consistently when comparing models. Switching metrics mid-project makes comparisons invalid.
• For multi-class problems, report per-class metrics and a weighted average.

One thing to remember: Metrics are not just numbers — they encode business priorities. Picking the right metric is as important as picking the right model.