PyTorch Quantization — ELI5

Imagine you’re describing temperatures to a friend. You could say “it’s 72.3847 degrees” — super precise but slow to say. Or you could say “it’s about 72” — close enough, and much quicker. Your friend still grabs the right jacket either way.

Quantization does this to the numbers inside a neural network. During training, models use very precise numbers (32-bit floating point — think 8 decimal places). But for actually using the model (inference), you can round those numbers down to simpler ones (8-bit integers — basically whole numbers from -128 to 127).

Why bother? Three big reasons:

1. Speed. Smaller numbers mean faster math. An 8-bit calculation is 2-4× faster than a 32-bit one on most hardware.
2. Size. The model file shrinks by about 75%. A 4 GB model becomes 1 GB — fitting on a phone instead of needing a server.
3. Cost. Less memory and fewer computations mean lower cloud bills and longer battery life.

The amazing part is that for most models, this rounding barely changes the results. An image classifier that was 95% accurate at full precision might be 94.5% accurate after quantization. For practical purposes, that’s identical.

This is why AI can run on your phone’s camera, in your car’s autopilot, and on tiny devices at the edge of the network — quantization makes big models small enough to go anywhere.

The one thing to remember: Quantization rounds a model’s precise numbers to simpler ones, making it 2-4× faster and 4× smaller while barely affecting accuracy — it’s how AI runs on phones and cheap hardware.