Model Quantization — Explain Like I'm 5

How AI models get shrunk to run on your phone — the precision-tradeoff trick that makes 70 billion parameter models fit in consumer hardware.

Numbers That Don’t Need to Be Precise

Imagine you’re measuring ingredients for a recipe. You can use a precision scientific scale accurate to 0.001 grams, or a kitchen scale accurate to 1 gram.

For most recipes, the kitchen scale is plenty. Salt doesn’t need to be measured to the microgram. You’re not making pharmaceuticals.

Neural networks store billions of numbers — the “weights” that encode what the model has learned. By default, these numbers are stored as 32-bit floating point numbers (very precise). Quantization asks: do we really need that precision?

For most purposes, the answer is no. You can store each weight as a much smaller number — 8-bit integers instead of 32-bit floats — with only a tiny drop in quality. This makes the model 4x smaller and dramatically faster.

What the Numbers Actually Look Like

A 32-bit float can represent numbers like 0.00847239123… with extreme precision. An 8-bit integer can represent only 256 different values (typically -128 to 127).

The trick: you figure out the range of values a layer uses (say, mostly between -2.0 and 2.0), and map that range to -128 to 127. Each step is about 0.016. You’ve lost some precision, but you’ve compressed the storage 4x.

For a 7 billion parameter model like Llama 2:

  • Full precision (32-bit): 28 GB — too large for most consumer GPUs
  • 8-bit quantized: 7 GB — fits in a mid-range GPU
  • 4-bit quantized: 3.5 GB — fits in a gaming PC or even a MacBook Pro with 8GB RAM

This is why you can run surprisingly capable AI models on your laptop.

Where Quality Gets Lost

The key insight: not all numbers are equally important. Some weights vary across a wide range; others cluster tightly. Applying the same crude approximation uniformly loses more quality than necessary.

Researchers have found clever ways to quantize that preserve the most important information — targeting which parts of the model need more precision.

The result: a 4-bit quantized model often achieves 95–99% of the quality of the full-precision version, while fitting in one-quarter the memory.

One thing to remember: Quantization trades precision for speed and memory — and because neural networks are surprisingly robust to small changes in individual weights, this tradeoff often costs almost nothing in quality while making deployment dramatically more practical.

model-quantizationmodel-compressionefficiencyint8llmon-device-ai

See Also

  • Knowledge Distillation How AI companies shrink massive models down to phone-sized ones without losing much intelligence — the teacher-student trick that powers on-device AI.
  • Model Pruning How AI models lose weight without losing intelligence — removing the neurons that don't actually do anything useful to make models faster and smaller.
  • Speculative Decoding The clever trick that makes large AI models generate text 2-4x faster — using a small 'draft' model to guess tokens that a big model then quickly verifies.
  • Activation Functions Why neural networks need these tiny mathematical functions — and how ReLU's simplicity accidentally made deep learning possible.
  • Ai Agents Architecture How AI systems go from answering questions to actually doing things — the design patterns that turn language models into autonomous agents that browse, code, and plan.

Related Topics

ab-testing How tech companies run thousands of experiments at once to improve their products — the scientific method applied to everything from button colors to recommendation algorithms. 3 levels → activation-functions Why neural networks need these tiny mathematical functions — and how ReLU's simplicity accidentally made deep learning possible. 3 levels → ai-agents-architecture How AI systems go from answering questions to actually doing things — the design patterns that turn language models into autonomous agents that browse, code, and plan. 3 levels →