Overfitting — Core Concepts

When Your Model Is Too Smart For Its Own Good

Every machine learning project eventually hits the same wall: the model performs beautifully on the training data and falls apart on anything new. That gap between training performance and real-world performance has a name — overfitting — and it’s probably the most common reason ML projects fail quietly without anyone understanding why.

Here’s what’s actually happening under the hood.

The Bias-Variance Tradeoff

There’s a fundamental tension in ML models between two failure modes:

• Underfitting (high bias): The model is too simple. It can’t capture the patterns in the data. Like trying to fit a straight line through data that curves.
• Overfitting (high variance): The model is too complex. It captures the patterns and all the noise. Like a curvy line that wiggles through every single data point exactly.

The goal is the middle — a model complex enough to learn the real patterns, simple enough not to chase the noise.

Most beginners intuitively fear underfitting (“my model isn’t learning!”) but overfitting is where projects actually die, because it looks like success until you deploy.

What “Noise” Means Here

Your training data isn’t perfect. It contains:

• Random measurement errors
• One-off weird examples that don’t generalize
• Artifacts of how you collected the data

A simple model ignores this noise. A too-complex model treats every data point as equally meaningful and twists itself into knots to fit all of them. The result is a model that has essentially memorized your dataset rather than learned from it.

This is why a decision tree with no depth limit on a small dataset will often hit 100% training accuracy. It just builds a lookup table. Every path through the tree ends in “I’ve seen this exact example before.”

Diagnosing Overfitting

The standard way to detect it: plot your training loss and validation loss over epochs (training steps).

In a healthy training run:

• Both curves go down together
• They level off at similar values

In an overfit model:

• Training loss keeps going down
• Validation loss starts going up at some point

That divergence is the moment the model stopped learning generalizable patterns and started memorizing. The technical term for the point just before validation loss rises is the sweet spot — and most practitioners use early stopping to freeze training there automatically.

Common Fixes

More data. Almost always the best answer. It’s hard to memorize 1 million examples. It’s easy to memorize 500.

Regularization. L1 and L2 regularization add a penalty term to the loss function that punishes models for having large weights. The model is forced to stay simpler. L2 (Ridge) shrinks all weights. L1 (Lasso) pushes some weights to exactly zero, effectively removing features.

Dropout. Specifically for neural networks. During training, randomly “turn off” some percentage of neurons each pass. The network can’t rely on any single neuron always being there, so it’s forced to learn redundant, distributed representations. Introduced in a famous 2014 paper by Srivastava et al. — now completely standard.

Cross-validation. Instead of one train/test split, use k-fold validation: split data into k groups, train k times (each time leaving one group out as validation). You get a much better estimate of real-world performance.

Simpler model. Sometimes the model architecture is overkill for the data. A 7-layer neural network on a 1,000-row dataset is going to overfit no matter what you do.

The Common Misconception

Most people think “more training = better model.” It’s actually “more training = better generalization, up to a point, then worse.”

There’s also a newer twist that upends classical wisdom: for very large neural networks (modern LLMs), researchers have observed double descent — training even longer past the overfit point sometimes leads the model back into good generalization territory. This is a genuinely weird phenomenon that’s not fully understood yet. It’s one reason extremely large language models can train for so long without collapsing.

But for 99% of practical ML work, the old rule holds: watch your validation loss curve, stop when it diverges.

One thing to remember: Overfitting is the model learning your training data’s quirks instead of the world’s patterns. A great training score means nothing if validation performance is terrible — that’s not success, it’s a warning.