Skip to content
localmodel.run

Text model · Granite

Granite 4.0 H Small: RAM and VRAM requirements

Granite 4.0 H Small needs about 20.3 GB to run at Q4_K_M (the Q4_K_M GGUF file is ~18.14 GB to download; KV cache and overhead add the rest), or about 34.1 GB at Q8_0. The lightest hardware that runs it is Nvidia GeForce RTX 4090 (24GB).

Granite family · 32B params (Mixture-of-Experts: activates only 9B of 32B params per token, so generation is faster than the total size suggests) · released Oct 2025.

Shopping for hardware? See what runs Granite 4.0 H Small →

License Apache-2.0 · Commercial OK ↓ 132.5K/mo ♥ 308 on HuggingFace
Q4_K_M GGUF
18.14 GB
Q8_0 GGUF
31.91 GB
Memory @ Q4 (4k)
~20.3 GB
Context
128 k

Q4_K_M and Q8_0 are GGUF download (file) sizes. Memory @ Q4 includes the KV cache and overhead, so it runs larger than the file on disk.

Will it run on your device?

Granite 4.0 H Small runs on 13 of 40 tracked devices at Q4_K_M.

12 run well 1 tight fit 27 too small
Fit check Q4_K_M
No, not enough memory
needs 20.3 GB usable 10.5 GB
See the full breakdown

Memory breakdown

Weights (Q4_K_M)18.14 GB
+
KV cache (4k)1.4 GB
+
Overhead0.8 GB
=
Total20.3 GB

How context length changes it

4k context ~20.3 GB 32k context ~29.8 GB 128k context ~62.3 GB

Longer context grows the KV cache, which for Granite 4.0 H Small is sized by its 9B active params, not the full 32B. It needs ~20.3 GB at 4k and ~62.3 GB at 128k.

Quantization sizes

GGUF quantson disk
Quantization Size on disk
Q2_K 13.4 GB est
Q3_K_M 15.6 GB est
Q4_K_M (default) 18.14 GB
Q5_K_M 22.8 GB est
Q6_K 26.2 GB est
Q8_0 31.91 GB
FP16 64.4 GB

Lower quant = smaller and faster, slightly lower quality. Q4_K_M is the common default.

Run it

Ollama
$ ollama run granite4:32b-a9b-h
llama.cpp
$ llama-cli -hf ibm-granite/granite-4.0-h-small-GGUF:Q4_K_M
LM Studio
$ lms get ibm-granite/granite-4.0-h-small-GGUF

Which devices can run Granite 4.0 H Small?

Apple Silicon Macs

RAM-only laptops

iPhone & iPad

Android

NVIDIA GPUs

AMD GPUs

Same job, different size

Pick by what fits your memory: step down to free up VRAM, or step up if you have headroom.

Next size down North Mini Code 1.0 30.5B Next size up Gemma 4 31B 32.7B

Similar models

Head-to-head

FAQ

How much VRAM or RAM does Granite 4.0 H Small need?

At Q4_K_M, Granite 4.0 H Small needs about 20.3 GB (weights ~18.14 GB + KV cache + overhead) at a 4k context. At Q8_0 budget ~34.1 GB.

What is the Q4_K_M GGUF file size of Granite 4.0 H Small?

The Q4_K_M GGUF file is about 18.14 GB to download, and the Q8_0 GGUF is about 31.91 GB. That is the weights file on disk; to run it you also need room for the KV cache and overhead, so budget ~20.3 GB of memory at Q4_K_M.

Can Granite 4.0 H Small run on a laptop?

Granite 4.0 H Small is large; you need a 24 GB+ GPU or a 32-48 GB Mac at Q4_K_M.

Is Granite 4.0 H Small cheaper to run because it is a MoE model?

It is faster, not lighter. Granite 4.0 H Small activates only 9B of 32B params per token (so it runs quickly), but all experts must stay in memory, so it still needs memory for the full 32B.

Can I use Granite 4.0 H Small commercially?

Yes. Granite 4.0 H Small is licensed Apache-2.0, which permits commercial use.

Understand the numbers

Short guides to the ideas behind Granite 4.0 H Small's memory and quant figures.

IBM Granite 4.0: a hybrid Mamba-2 + Transformer MoE, 32B total / 9B active, 128K context. Apache-2.0.

Sources

Last validated 2026-08-03. Memory figures are estimates. See methodology.