GPU Computing — Explain Like I'm 5

1 Expert vs. 5,000 Workers

Imagine you need to paint 10,000 small squares. One option: hire one master painter who’s incredibly skilled. They do each square perfectly, one at a time — 10,000 squares × 2 minutes each = 3 weeks.

Another option: hire 5,000 regular painters. Give each a couple of squares. They’re not as skilled individually, but together they finish in 2 minutes.

For this specific task — doing the same thing to many items simultaneously — more workers is almost always better than one skilled worker.

A CPU is like the master painter: powerful, flexible, excellent at complex tasks, but mostly one thing at a time. A GPU is like those 5,000 workers: thousands of simpler processors all running simultaneously.

Why AI Needs This

Training a neural network means doing millions of “multiply these numbers and add them up” operations. Not complex operations — just the same simple math on enormous lists of numbers.

This is exactly what GPUs are built for. NVIDIA’s A100 GPU has 6,912 CUDA cores, each capable of doing floating-point math. Together, they can do 77.6 trillion operations per second — thousands of times faster than a CPU for this specific task.

Training GPT-3 would have taken 355 years on a single CPU. On thousands of A100 GPUs, it took a few weeks.

The Gaming Origin Story

GPUs were invented to render video game graphics — calculating the color of millions of pixels simultaneously, 60+ times per second. Game developers wanted smoother, more detailed graphics, which required more parallel processing power.

In 2006, NVIDIA released CUDA — software that let people use these graphics processors for general computing, not just graphics. Researchers discovered GPU math was perfect for neural networks. The combination of cheaper, faster GPUs and bigger neural networks is what launched the deep learning revolution in 2012.

Today’s AI chips (NVIDIA H100, Google TPU, Apple Neural Engine) are purpose-built for AI operations — but they evolved from gaming hardware.

One thing to remember: GPUs accelerate AI not through raw intelligence but through brute parallel force — thousands of simple processors doing the same simple math on millions of numbers simultaneously.