Area 07 · Deployment · Concept 11 of 20

Ollama for Local Models

Run open models on your own laptop with one command. Good for testing and privacy.

11–20 · Production levelPhase 5 · Weeks 18–20≈ 45 min

The idea

Ollama downloads quantized open models and runs them locally with a simple CLI and an HTTP API, including an OpenAI-compatible endpoint. It runs on macOS (using Apple Silicon GPUs), Linux and Windows.

Use it for offline development, cheap testing of pipelines, privacy-sensitive experiments, and CI jobs that need a model without API keys. A Modelfile lets you set a system prompt and parameters and share a configured model.

Local models are smaller and weaker than frontier APIs; do not judge your product’s final quality on them.

Why it matters

It removes API cost and network dependency from your inner development loop.

Words to know

  • ModelfileOllama’s recipe for a configured model.

Try it without code

Chat offline

  1. Install Ollama.
  2. Run: ollama run llama3.2
  3. Turn off Wi-Fi and keep chatting.

Use it in code

Point the OpenAI client at Ollama so your app can switch by config.

python
from openai import OpenAI

local = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = local.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Give me three test questions about refunds."}],
)
print(reply.choices[0].message.content)
# In settings: CHAT_BASE_URL=http://localhost:11434/v1 for dev, provider URL for prod
PRACTICE · TICK IT AFTER THIS

Make your app’s model endpoint configurable and run your integration tests against Ollama in CI.

Common mistakes

  • Developing prompts only on a local model and being surprised in production.

Go deeper