You’ve found a model you want to try: Llama 8B, Qwen 32B, something with “70B” in the name. The question is always the same: will it run on my machine, and will it be fast enough to use?

You don’t need to download 40 GB to find out. With four back-of-the-envelope formulas, you can answer it in about a minute.

The one idea that explains everything

Think of a language model as a giant book of numbers. To write each new word, the computer reads through the entire book once. Then it does it again for the next word.

That gives you the only two questions that matter:

Question	What it really means	Measured in
Does the book fit on your shelf?	Is there enough memory to hold the model?	GB
How fast can you read it?	How quickly can the hardware pull data out of memory?	GB/s

Notice what’s missing: raw computing power. For generating text, the bottleneck is almost never the math. It’s fetching billions of numbers from memory, over and over.

Step 1: How big is the model?

Weights (GB) ≈ billions of parameters × bytes per parameter

The parameter count is in the model’s name: “8B” means 8 billion. The bytes per parameter depends on how the model is stored, called its precision or quantization:

Precision	Bytes per parameter	8B model	70B model
16-bit (FP16 / BF16)	2	16 GB	140 GB
8-bit	~1	~8.5 GB	~75 GB
4-bit (e.g. Q4_K_M)	~0.6	~5 GB	~42 GB

Why ~0.6 and not 0.5 for 4-bit? Quantized files carry some bookkeeping, and a few sensitive layers stay at higher precision.

4-bit is the everyday default for running models at home. It cuts the size by about two-thirds with only a small loss in quality. And for the same memory budget, a bigger model at 4-bit usually beats a smaller one at 8-bit.

The shortcut worth memorizing:

At 4-bit, GB ≈ 0.6 × billions of parameters. 8B → 5 GB · 14B → 9 GB · 32B → 20 GB · 70B → 42 GB

Treat this as a floor. Some models ship large embedding tables or image and audio encoders in the same file. Always glance at the real download size.

Step 2: Add the context cache

The model file isn’t the only thing in memory. While it works, the model keeps notes on everything said so far in the conversation, so it doesn’t have to reprocess it for every new word. This is the KV cache, and it grows with every token.

KV cache (bytes) = 2 × layers × KV heads × head dimension × tokens × bytes per value

You’ll find layers, KV heads and head dimension in the model’s config.json on Hugging Face. The “2” is because each layer stores a key and a value, and bytes per value is 2 at the default 16-bit.

For a typical 8B model (32 layers, 8 KV heads, head dimension 128):

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB per token
128 KB × 8,192 tokens       ≈ 1 GB

That gives a handy rule of thumb for mid-size models:

Context	Extra memory (8B model)
8K tokens (~12 pages)	~1 GB
32K tokens	~4 GB
128K tokens	~16 GB

At long contexts, the cache can be as large as the model itself. If memory is tight, most tools can store it at 8-bit to halve it: OLLAMA_KV_CACHE_TYPE=q8_0 in Ollama, or the K/V cache quantization setting in LM Studio.

Step 3: Does it fit?

Weights + KV cache + 1.5 GB headroom ≤ your usable memory

The 1.5 GB covers the software itself and its buffers. What counts as “your memory” depends on the machine:

Your machine	Memory that counts
PC with a graphics card	The card’s VRAM (8, 12, 16, 24 GB…)
PC or laptop without one	System RAM, minus what the OS and apps use
Apple Silicon Mac	About 70–75% of unified memory, the GPU’s default share

VRAM is a hard wall. A model that needs 9 GB doesn’t run cleanly on an 8 GB card. Leave real margin.

If it doesn’t fit, try in this order:

1. A lower precision (8-bit → 4-bit)
2. A smaller model
3. A shorter context
4. An 8-bit KV cache

On a PC you can also split a model between the graphics card and system RAM. It works, but the spilled part runs at RAM speed. It’s a way to squeeze in a model that’s slightly too big, not one that’s twice your VRAM.

Step 4: How fast will it run?

Every new token means reading all the weights once, so:

Tokens per second ≈ 0.6 × memory bandwidth (GB/s) ÷ model size (GB)

The 0.6 accounts for real-world overhead. Typical bandwidths:

Hardware	Bandwidth	8B at 4-bit (5 GB)	70B at 4-bit (42 GB)
Laptop / desktop DDR5 RAM	60–90 GB/s	~10 tok/s	~1 tok/s
Apple M-series Pro	~250 GB/s	~30 tok/s	~4 tok/s
Apple M-series Max	~500 GB/s	~60 tok/s	~7 tok/s
RTX 3090 / 4090	~1,000 GB/s	~120 tok/s	doesn’t fit on one card

What those numbers feel like:

Tokens per second	How it feels
Under 3	Painful; you watch every word appear
5–10	About reading speed; fine for chat
20+	Comfortable
50+	Instant

The formula also explains why smaller is faster: halve the model size and you roughly double the speed.

So, do you need a GPU?

Model size	Answer
Up to ~8B	No. A laptop with 16 GB of RAM runs them at ~10 tok/s.
~14–32B	It helps a lot. CPU-only gives 3–6 tok/s; a 16–24 GB card or a Mac makes it comfortable.
70B and up	Effectively yes. It loads with enough RAM but crawls. You’ll want a 64 GB+ Mac or several GPUs.

Macs are the middle path. Unified memory gives them lots of room, and their bandwidth sits between regular RAM and a high-end graphics card. A 64 GB Mac can run models that would otherwise need two expensive GPUs.

There’s one place where raw compute does matter: reading a long prompt before the model starts answering. That step runs in parallel and is limited by compute, roughly:

Seconds ≈ 2 × parameters × prompt tokens ÷ achievable FLOP/s

GPUs are often 10× or more faster here. If you plan to paste in long documents or codebases, a GPU or Mac matters even for small models.

A special case: Mixture-of-Experts models

Some models list two numbers, like 30B-A3B (30 billion total, 3 billion active). These are Mixture-of-Experts models: each token is routed through a few specialist sub-networks, called experts.

Memory follows the total parameters. Speed follows the active parameters.

A 30B-A3B model needs memory for all 30B (~18 GB at 4-bit) but reads only ~3B per token, so it runs at small-model speed. On machines with lots of memory but modest bandwidth, like a CPU with 32–64 GB of RAM or a Mac, they’re often the best choice.

The common mistake: “only 3B active, so it’ll fit on my 4 GB card.” It won’t. All 30B still have to sit in memory.

Putting it together: one worked example

“Can I run a 32B model on my 16 GB laptop with no graphics card?”

1. Weights: 32 × 0.6 ≈ 19 GB
2. Plus cache and headroom: ~22 GB
3. Fit: 16 GB of RAM → doesn’t fit

Try a 14B model instead:

1. Weights: 14 × 0.6 ≈ 8.5 GB
2. Plus cache and headroom: ~11 GB
3. Fit: fits in 16 GB, leaving room for the OS
4. Speed: 0.6 × 80 ÷ 8.5 ≈ 5–6 tok/s, usable but slow

If speed matters more than smarts, an 8B model gives you about 10 tok/s on the same laptop.

When the rule of thumb breaks

Real models aren’t always average. Google’s Gemma 4 E4B is a good reminder:

• It needs more memory than the shortcut says. 8B × 0.6 ≈ 4.8 GB, but the 4-bit download is 6.6 GB. Embedding tables make up about 44% of the model and are kept at higher precision, and the image and audio encoders ship in the same file.
• Its KV cache is about 5× smaller than you’d expect. Only 24 of its 42 layers store a cache, and most of those keep only the last 512 tokens. That’s about 0.14 GB at 8K context instead of the usual ~1 GB.

Neither breaks the formulas. They just mean you plug in the model’s real numbers: the actual file size for Step 1, and the actual layer setup from config.json for Step 2.

The whole thing in four lines

Fit:    weights + KV cache + 1.5 GB ≤ usable memory
Size:   weights ≈ billions of params × bytes per param   (4-bit: × 0.6)
Cache:  2 × layers × KV heads × head dim × tokens × bytes
Speed:  tok/s ≈ 0.6 × bandwidth ÷ GB read per token

Download the one-page cheat sheet (PDF) to keep these handy, or view it online.

Or plug your own model and machine into the LLM Sizing Calculator.

All of these are estimates. They usually land within 20–30% of reality, which is close enough to make the right call before you download anything.

---

Want to go deeper? LLM Sizing 101 is a free eight-lesson course that walks through each step, covers the edge cases, and ends with practice exercises.