Model family · 6 sizes

Granite: which size runs locally?

Granite comes in 6 sizes, from 2.53B to 32B. Some sizes are Mixture-of-Experts, so they run faster than their memory footprint suggests. Here is each size with its Q4_K_M weight, the memory it needs, and the hardware that runs it.

Sizes: 6
Smallest: 2.53B
Largest: 32B
Runs from: 8GB

The Granite lineup

Smallest first Q4_K_M · min memory

"Needs" is the sourced minimum memory for Q4_K_M with a small context. Larger context needs more.

Which Granite fits your memory

8GB

Largest that fits: Granite 4.1 3B (3.4B), best case on Apple M1 (8GB).

Yes

16GB

Largest that fits: Granite 4.1 8B (8.8B), best case on Nvidia GeForce RTX 4080 (16GB).

Yes

24GB

Largest that fits: Granite 4.0 H Small (32B), best case on Nvidia GeForce RTX 4090 (24GB).

Yes

32GB

Largest that fits: Granite 4.0 H Small (32B), best case on Nvidia GeForce RTX 5090 (32GB).

Yes

48GB

Largest that fits: Granite 4.0 H Small (32B), best case on Apple M5 Pro (48GB).

Yes

64GB

Largest that fits: Granite 4.0 H Small (32B), best case on Apple M4 Max (64GB).

Yes

128GB

Largest that fits: Granite 4.0 H Small (32B), best case on Apple M5 Max (128GB).

Yes

256GB

Largest that fits: Granite 4.0 H Small (32B), best case on Apple M3 Ultra (256GB).

Yes

Best case means the most capable device at that size (usually a discrete GPU). A Mac at the same size sits roughly one rung lower; see the per-size breakdown on each memory budget page.

FAQ

Which Granite size should I run locally?

Pick the largest size your memory allows. On 8GB (best case) up to Granite 4.1 3B; On 16GB (best case) up to Granite 4.1 8B; On 24GB (best case) up to Granite 4.0 H Small; On 32GB (best case) up to Granite 4.0 H Small; On 48GB (best case) up to Granite 4.0 H Small; On 64GB (best case) up to Granite 4.0 H Small; On 128GB (best case) up to Granite 4.0 H Small; On 256GB (best case) up to Granite 4.0 H Small. Smaller sizes run faster and leave headroom for context.

What is the smallest Granite model?

Granite 3.1 2B at 2.53B parameters, about 1.55 GB on disk at Q4_K_M and roughly 3 GB of memory to run. It is the one to use on phones and 8 GB machines.

What is the largest Granite model and what does it need?

Granite 4.0 H Small at 32B (mixture of experts), about 18.14 GB at Q4_K_M. It fits a high-memory desktop GPU or Mac.

Understand the numbers

Short guides to the ideas behind Granite's memory and quant figures.

Sources

Memory figures are estimates at Q4_K_M. See methodology.