Run a local LLM with Ollama: setup in 3 commands, and the memory math

The short version

• Ollama gets a local model running in three commands.
• It serves an OpenAI-compatible API at http://localhost:11434/v1, so existing code needs only a new base_url and model.
• Memory decides what you can run: parameters times bytes per parameter, plus 20 to 30 percent headroom.
• At 4-bit, an 8B model needs about 4 GB and a 70B model about 35 GB. Frontier models do not fit on a personal machine.
• Keep both: route sensitive data to the local model and quality-critical work to a hosted API, from the same function.

Three commands

# 1. Install (macOS)
brew install ollama

# 2. Start the server
ollama serve

# 3. Pull and run a model
ollama run qwen3:8b

Linux has an official install script and Windows has an installer. ollama run downloads the model on first use and loads it from disk after that. Type /bye to leave the prompt, and ollama list shows the models you have.

Connect existing code

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, which makes it useful beyond the terminal:

from openai import OpenAI

client = OpenAI(
    api_key="ollama",  # any non-empty string; Ollama does not check it
    base_url="http://localhost:11434/v1",
)

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Only base_url and model change. Code written against the OpenAI format can prototype on a local model and move to a hosted API with no other edits.

The memory wall

Memory is the first limit you hit. The weights need roughly:

memory ≈ parameters × bytes per parameter

Quantization (storing weights at fewer bits) sets the bytes per parameter:

These figures cover the weights only. The context (the KV cache) and the operating system need memory on top, so allow 20 to 30 percent more.

In practice:

• 8B class: about 4 GB at 4-bit. Runs on most laptops.
• 32B class: about 16 GB at 4-bit. Realistic on a 32 GB machine.
• 70B class: about 35 GB even at 4-bit. Needs 64 GB or more.
• Beyond that: frontier-scale models do not fit on a personal machine at all.

Apple Silicon's unified memory helps here. The GPU and CPU share one pool, so most of the installed memory is available to the model.

Quantization costs quality

A 4-bit model fits where the full one does not, and it gives up some output quality to get there. For many tasks 4-bit is fine; the gap shows in consistency over long contexts and in following detailed instructions.

The same model name at two quantization levels behaves like two different models. When a local model feels weaker than its reputation, quantization is the usual cause. Published benchmark scores are normally measured at or near full precision.

Where local fits

Good fits:

• Confidential data that must never leave the machine. This is the strongest reason to run locally.
• Offline work, with no network.
• High-volume simple jobs, such as classifying or tagging tens of thousands of items, where electricity is the only running cost.
• Learning and experiments, to see how models behave.

Poor fits:

• Work that needs the best available quality, since frontier models do not fit.
• Occasional use: the hardware costs money whether or not you use it.
• Concurrent users. A local setup mostly serves one request at a time and loses badly to hosted batching.
• Trying new models the day they ship, since each one means a download and setup.

The break-even math, with its assumptions stated, is in local LLM vs hosted API and on the FastMetal vs a local LLM page.

Use both

The two share one API format, so you can route by the nature of the request:

import os
from openai import OpenAI

local = OpenAI(api_key="ollama", base_url="http://localhost:11434/v1")
cloud = OpenAI(api_key=os.environ["FASTMETAL_API_KEY"], base_url="https://api.fastmetal.ai/v1")

def ask(text: str, sensitive: bool = False):
    client, model = (local, "qwen3:8b") if sensitive else (cloud, "anthropic-claude-sonnet-5-5")
    r = client.chat.completions.create(
        model=model,
        max_tokens=500,
        messages=[{"role": "user", "content": text}],
    )
    return r.choices[0].message.content

Sensitive data stays on the machine; work that needs quality goes to the API.

The hosted side also serves open-weight models of the kind you would run locally. If the open model you want does not fit in your memory, a hosted copy of it is often the simple answer. The model catalog lists them.

Common problems

• Very slow output with no error. The model does not fit in memory and the machine is swapping. Work out the memory first.
• An empty api_key. The OpenAI SDK refuses an empty key. Ollama ignores the value, so any string works.
• A missing tag. qwen3 and qwen3:8b are different; without a tag, Ollama picks a default size.
• A shorter context than you expect. Local models often ship with a short default context length. Check the setting before you send long inputs.
• Connection refused. ollama serve is not running.

FAQ

Is a local LLM free?

The software and the open models are free. The hardware and electricity are not, and you pay for the hardware even when it sits idle. At low volume, a pay-per-token API usually costs less.

How much memory do I need?

About parameters times bytes per parameter. At 4-bit: ~4 GB for 8B, ~16 GB for 32B, ~35 GB for 70B. Add 20 to 30 percent for context and the operating system.

Is a local model as good as the hosted one?

Quantization lowers the quality of the same model, and frontier models do not fit locally at all, so the ceiling is lower.

Can I move Ollama code to a hosted API?

Yes. Both are OpenAI-compatible; change base_url and model.

Try it

Run a model with Ollama, then point the same code at FastMetal: create an account, issue a key, and change two lines. The catalog shows what you can call.