How a Giant AI Model Fits on Your Laptop
Something quietly impossible happens every time you run an AI model on your laptop.
A capable modern model is, at its heart, a giant pile of numbers, its “weights.” A small one has around 8 billion of them. Stored the normal way, that’s roughly 16 gigabytes of pure numbers, and bigger models run to hundreds of gigabytes. Your laptop’s memory is nowhere near enough to hold that comfortably. By rights, it shouldn’t fit. And yet you type one command and the thing runs, smoothly, answering you in real time.
How? The answer is one of the most elegant tricks in all of AI, and it’s called quantization. The one-line version: store each of those billions of numbers using fewer bits. It’s the exact same idea as saving a photo as a compact JPEG instead of a massive RAW file, you lose a tiny sliver of quality you can barely perceive, and in return the thing shrinks to a fraction of its size.
I touched this in my Ollama post, but it deserves the full treatment, because once you actually see how it works, it stops feeling like magic and starts feeling like clever engineering you could have thought of yourself. Let me build it up from the very bottom: what a single weight even is, how you shrink it, why that barely hurts, and where the trick finally falls apart.
The problem, drawn to scale
First, feel the size mismatch that makes this necessary:
Step 1: what is a single weight, really?
Before you can shrink billions of numbers, you have to know what one of them looks like to the computer. A weight is just a number like 0.4732. But the computer doesn’t store “0.4732” as text, it stores it in bits, using a format called a floating-point number. The standard format for AI, FP16, uses 16 bits per number.
Step 2: the ladder of precision
Here’s the key realization: you don’t have to store every weight in 16 bits. You can use fewer, and the fewer you use, the smaller the model. This is a ladder, and each step down roughly halves the size:
But wait, this should bother you. A 16-bit float can represent a smooth, near-infinite range of values. A 4-bit integer can only represent 16 distinct values (2 to the power of 4). How on earth do you cram a delicate weight like 0.4732 into a system that only has 16 possible slots? That’s the heart of the trick, and it’s beautifully simple.
Step 3: the actual trick, mapping floats onto a grid
Here’s how you turn a precise float into a low-bit integer. Imagine the possible integer values as a grid of evenly-spaced pegs. Quantization takes each real weight and snaps it to the nearest peg. You lose the tiny distance between the true value and the peg, that’s the “rounding error”, but you gain the ability to store just which peg it landed on, which takes far fewer bits.
But there’s a catch you may have spotted: those pegs are evenly spaced from -max to +max. How does the computer know what max is, so it knows how wide to space the pegs? That’s where the one crucial extra number comes in: the scale factor.
The recipe (this is the whole mechanic, in plain arithmetic):
The elegant part: you don’t store a scale for every weight (that would defeat the purpose). You group weights into blocks, say 32 at a time, and store one scale factor per block. So the real cost of “4-bit” is a touch more than 4 bits: 32 four-bit weights (128 bits) plus one 16-bit scale is 144 bits over 32 weights, about 4.5 bits each. Honest math, and still a massive saving.
Why this barely hurts: the JPEG intuition
Now the reassuring part. You might expect throwing away precision to wreck the model. It mostly doesn’t, and the reason is the same reason JPEG works on photos:
And “almost identically” is measurable, not a hope. Here’s roughly how much of the original quality survives at each level (from large-scale evaluations):
The wrinkle the pros handle: outliers
Here’s where it gets genuinely interesting, and where naive quantization breaks. In a real model, the weights aren’t all similar sizes. A tiny fraction of them, often around 0.1%, are wild outliers: values far bigger than the rest. And they matter enormously to the output.
The clever fixes are why modern quantization is so good. Techniques like LLM.int8() keep just those rare outlier weights in full precision while quantizing the other 99.9% aggressively, a mixed-precision compromise. Others, like AWQ, analyze which weights actually matter most to the output (only about 1% are “salient”) and protect those while squeezing the rest. And the popular K-quant scheme (the “K” in Q4_K_M) uses smart block structures with shared meta-constants so even the scale factors are stored efficiently. This is why a well-made 4-bit model is so much better than a crude one: it’s not quantizing everything blindly, it’s protecting what matters.
Decoding the cryptic names
Now those mysterious model filenames make sense. When you see Q4_K_M, you can read it:
| Piece | Means |
|---|---|
Q4 | Quantized to about 4 bits per weight |
K | Uses the smart "K-quant" block structure (better than the old plain method) |
M | Medium variant, a size within the family (S = small, M = medium, L = large) |
One honest detail: it unpacks to compute
A subtle point worth knowing so you’re not misled. The weights are stored in 4 bits to save space, but during the actual math, the computer usually unpacks them back to full precision for each calculation (this is called dequantization). So quantization mainly saves memory and loading time, not necessarily the raw math speed, though smaller data moving around does often make things faster too. The headline win is the one that lets it run at all: it fits.
The full picture
| Level | Size (8B model) | Quality kept | Use when |
|---|---|---|---|
| FP16 | ~16 GB | 100% | You have the hardware and want max fidelity |
| Q8 | ~8.5 GB | ~99%+ | Plenty of memory, want near-perfect |
| Q4_K_M | ~5 GB | ~97-99% | The default. Best balance for most laptops |
| Q3 | ~4 GB | ~90% | Tight on memory, can tolerate some slip |
| Q2 | ~3 GB | drops sharply | Rarely worth it, quality falls off |
The takeaway
The reason a model that “shouldn’t fit” runs on your laptop isn’t magic, it’s a chain of clever, understandable ideas. A weight is just a number in bits. You don’t need all those bits, most of the precision was never doing real work, exactly like the detail a JPEG throws away. So you snap each weight to the nearest peg on a grid, store just the peg plus one shared scale factor per block, and reverse it when needed. The rare outlier weights that actually matter get protected. The result: a quarter of the size, nearly all of the intelligence.
The next time you watch a genuinely capable AI answer you with your Wi-Fi off, on a laptop that has no business running something that large, you’ll know the trick underneath. Billions of numbers, each quietly rounded to the nearest peg, small enough to fit, sharp enough to still think. That’s quantization, and it’s one of the quiet pieces of engineering that put real AI in everyone’s hands.