How to Run an LLM Locally (2026 Guide)

Zakir Hossen·Jun 16, 2026·6 min read

You can run a capable AI model on your own computer right now, offline, for free. No API key, no monthly bill, no data leaving your machine. Five years ago this needed a research lab. Today it needs about ten minutes and a laptop made in the last few years. This guide shows you exactly what hardware you need, which model to pick for that hardware, and the commands to get a local LLM running today.

The reason this works now is two-fold: open models like Llama, Qwen, Mistral, and Gemma have gotten very good, and a tool called Ollama made running them as simple as installing an app. The engine underneath, llama.cpp, has crossed 116,000 GitHub stars and turned consumer laptops into respectable AI machines.

Can Your Computer Run a Local LLM?

The honest answer for most people: yes, a smaller one. The size of model you can run depends mostly on memory (RAM, or VRAM if you have a dedicated GPU). Here is the rule of thumb.

Models are measured in billions of parameters: 7B, 14B, 70B, and so on. Bigger is smarter but heavier. The trick that makes local LLMs practical is quantization, which shrinks a model by storing its numbers at lower precision. A 4-bit quantized model is roughly a quarter the size of the full version with only a small quality drop. Almost everyone runs quantized models locally.

Memory Requirements (Quantized, 4-bit)

Model size	Memory needed	What it can do	Typical hardware
1B to 3B	2 to 4 GB	Basic chat, simple tasks	Any modern laptop, even CPU-only
7B to 8B	5 to 8 GB	Solid daily assistant	16GB laptop, Apple Silicon, 8GB GPU
14B	9 to 12 GB	Strong general use, coding	16 to 32GB machine, 12GB+ GPU
32B to 34B	18 to 24 GB	Near-pro quality	32GB+ unified memory, 24GB GPU
70B	40 to 48 GB	Top open-model quality	64GB unified memory, dual GPUs

Two practical notes:

Apple Silicon punches above its weight. Macs share memory between CPU and GPU, so a Mac with 32GB of unified memory can run a 32B model that would need an expensive dedicated GPU on a PC. An M-series Mac is one of the best value local-AI machines you can buy.
No GPU? You can still run small models. A 7B model runs on a modern CPU, just slower (a few words per second instead of many). For chat that is often fine.

Step 1: Install Ollama

Ollama is the simplest way to run models locally. It handles downloading, memory management, and serving the model through a local API.

Go to the Ollama website and install it for your operating system. On Mac and Windows it is a normal installer. On Linux it is a single command. That is the whole setup.

To confirm it works, open a terminal and run:

ollama --version

Step 2: Pull a Model That Fits Your Hardware

Now download a model. Use the table above to pick a size. Some safe choices for 2026:

# Great default for a 16GB machine
ollama pull llama3.1:8b

# Strong all-rounder for 32GB machines
ollama pull qwen2.5:14b

# Excellent for coding help
ollama pull qwen2.5-coder:7b

# Tiny and fast, for weak hardware
ollama pull llama3.2:3b

The first download takes a few minutes. After that the model lives on your disk and loads instantly.

Step 3: Chat With It

Start a conversation right in the terminal:

ollama run qwen2.5:14b

Type a question, press enter, and the model answers locally. No internet required after the download. To exit, type /bye.

That is a working local LLM. Everything past this point is about making it nicer to use.

Step 4: Add a ChatGPT-Style Interface (Optional)

Talking to AI in a terminal gets old. To get a real chat window with history, document upload, and multiple users, add a web interface. The most popular is Open WebUI (over 141,000 GitHub stars), which runs in Docker and connects to Ollama automatically.

Once it is running, you open a browser tab and get a private ChatGPT clone backed by your local model. We cover the full set of interface options in self-hosted ChatGPT alternatives.

Which Model Should You Actually Pick?

Model names change fast, so here is how to think about it rather than a list that will date:

General chat and writing: the latest Llama or Qwen instruct model in the largest size your memory allows.
Coding: a coder-tuned model like Qwen Coder or a DeepSeek coder variant. These beat general models on programming.
Speed over quality: a 3B model. It will not reason as deeply, but it is fast and runs anywhere.
Maximum quality on a Mac: a 32B model on 32GB+ of unified memory hits a sweet spot of smart and usable.

When in doubt, start with a 7B or 8B model. It runs on almost anything and is genuinely useful. Move up only if your hardware allows and you want more.

How Fast Will It Be?

Speed is measured in tokens per second (roughly, words per second). What to expect:

Apple Silicon (M2/M3/M4): comfortably fast on 7B to 14B models, very usable on 32B.
Dedicated GPU (RTX-class, 12GB+): very fast on models that fit in VRAM.
CPU only: a few tokens per second on 7B models. Fine for chat, slow for long outputs.

If responses feel sluggish, drop to a smaller model or a more aggressive quantization. Speed and size trade off directly.

Common Problems and Fixes

"Out of memory" or the model crashes: the model is too big for your RAM or VRAM. Pull a smaller size or a more compressed quantization.

Responses are too slow: switch to a smaller model, close memory-hungry apps, or rent a cloud GPU if you need heavy throughput. See self-hosting AI on a budget for cheap options.

Quality feels weak: you are probably on a small model. Move up a size, or use a task-specific model (coder model for code, for example).

Want it to answer about your own files: that is retrieval, not raw chat. See self-hosted RAG.

Why Bother Running It Locally?

Three reasons people make the switch:

Privacy. Your prompts and data never leave the machine. For sensitive work, this is the whole point.
Cost. After the hardware, it is free. Heavy users and teams save real money versus per-seat cloud fees. The full comparison is in is self-hosting worth it.
Control. No rate limits, no surprise price changes, no model getting quietly swapped out from under you.

The trade-off is that the best open models still trail the best cloud models on the hardest tasks, and you handle your own setup. For most everyday work, you will not notice the gap.

Start Now

The shortest path: install Ollama, run ollama pull llama3.1:8b, then ollama run llama3.1:8b, and start typing. You will have a private AI assistant running on your own machine in less time than it takes to read this article twice.

When you are ready to make it nicer, add Open WebUI and explore the wider ecosystem in our best self-hosted AI tools roundup.