Guide · The basics
How local LLMs work
Running a model locally means its weights download to your machine and every token is computed on your own CPU, GPU or Apple Silicon. Nothing leaves your device. The catch is that your hardware now has to hold the whole model in memory while it runs.
A model is a big file of numbers (weights). To run it, those weights load into memory, plus a KV cache for your context and a little runtime overhead. Quantization shrinks the weights so they fit. Whether it runs on your machine is just: does that total fit in your usable memory?
What "running locally" actually means
A language model is, at its core, a large file of learned numbers called weights. To use it, a runtime reads those weights from disk into memory, then does matrix math on your input to produce output one token at a time. The process is the same on a MacBook, a Windows gaming PC or a Linux box with a data-center GPU. The only difference is how fast the hardware moves numbers. After the one-time download, the model never contacts a server again.
Memory: weights + KV cache + overhead
The memory a model needs is roughly its parameters times the bytes per parameter. At full 16-bit precision each parameter costs 2 bytes, so a 7B model needs about 14 GB and a 70B needs ~140 GB. That alone rules out most consumer hardware at full precision, which is where quantization comes in.
On top of the weights sits the KV cache, which holds the attention state for every token in your context. It grows with context length and can add several GB. Then a few hundred MB of runtime overhead. The site adds these three up against your device's usable memory; the exact formula is on the methodology page.
Quantization: fit a big model in less memory
Quantization stores each weight in fewer bits. The common local format is GGUF, and Q4_K_M is the practical default: it keeps most weights at 4 bits, cutting a 7B model from ~14 GB at full precision to around 4-5 GB while holding most of the quality. Go higher (Q5, Q6, Q8) for more fidelity if you have the headroom, or lower (Q3, Q2) to squeeze into a tight budget. The quantization guide covers each level with concrete sizes.
Why some hardware is much faster
Local inference speed is set mainly by memory bandwidth, not raw compute: every token loads the full set of weights from memory. GPUs have wide, fast memory and run many times quicker than CPU-only inference. Apple Silicon shares one unified memory pool between CPU and GPU, so a model that fits runs fully accelerated with no transfer cost. On a discrete GPU, if the model does not fit in VRAM the runtime spills layers to system RAM and speed drops sharply. See VRAM vs RAM.
The runtimes
Ollama
One command pulls a model and starts a local API. The easiest first choice; uses MLX on Apple Silicon.
LM Studio
A desktop app with a model browser and chat UI, plus a headless server mode. Best if you want a GUI.
llama.cpp
The C/C++ engine under both, and the origin of GGUF. A tiny static binary that runs on almost anything; use it directly for the lowest layer.
MLX
Apple's framework built for M-series unified memory. The fastest path on a Mac; use mlx-lm directly for maximum throughput or fine-tuning.
Per-platform picks, including iOS and Android apps, are on the tools page.
Local vs cloud
Local inference buys you three things a cloud API cannot: privacy (data stays on your machine), offline use, and no per-token cost. The trade is hardware. A 7-8B model at Q4_K_M runs comfortably on a 16 GB machine today and is plenty for a private assistant. Frontier-level reasoning at the very top end still favours hosted APIs, because the largest open models need server-class memory to run. This site exists to tell you exactly where your hardware lands on that line.
FAQ
How much RAM do I need to run a local LLM?
About 16 GB of memory is the practical entry point. A 7-8B model at Q4_K_M needs roughly 5-7 GB, so it fits a 16 GB Mac or a 12 GB GPU. Smaller 1-3B models run in a few GB. A 70B model needs ~40 GB+ at Q4_K_M, which means a high-memory Mac or a multi-GPU rig. The exact per-model number is on each model page.
Does the model need an internet connection to run?
Only for the initial download. Once the weights file is on your machine, inference is fully offline. Nothing you type leaves your device.
What is GGUF and why does everyone use it?
GGUF is a single-file weights format from the llama.cpp project. It holds every quantization level in one self-describing file and is read natively by the major runtimes (Ollama, LM Studio, llama.cpp, GPT4All). Most open models on HuggingFace ship a GGUF release alongside the original weights.
What is the KV cache and why does it matter?
The KV cache stores the attention state for every token in your context window, so it grows with context length. At a long context it can add several GB, sometimes more than the weights themselves, which is why setting a very long context can push you over your memory limit.
Which runtime should I start with?
Ollama if you want a one-command install and an API for your own code; LM Studio if you prefer a GUI with a chat window. Both run on Mac, Windows and Linux, and both use MLX on Apple Silicon for near-native speed. Reach for llama.cpp or MLX directly only when you want the lowest layer or maximum throughput.
Can I run local models on a phone?
Yes, within limits. Apps like PocketPal, LLM Farm and Private LLM run quantized models on iPhone and Android. A 1-4B model at Q4 is the realistic ceiling; anything larger thermally throttles or runs out of memory. See the per-device pages for what fits.