The idea
Ollama downloads quantized open models and runs them locally with a simple CLI and an HTTP API, including an OpenAI-compatible endpoint. It runs on macOS (using Apple Silicon GPUs), Linux and Windows.
Use it for offline development, cheap testing of pipelines, privacy-sensitive experiments, and CI jobs that need a model without API keys. A Modelfile lets you set a system prompt and parameters and share a configured model.
Local models are smaller and weaker than frontier APIs; do not judge your product’s final quality on them.
It removes API cost and network dependency from your inner development loop.
Words to know
- ModelfileOllama’s recipe for a configured model.
Try it without code
Chat offline
- Install Ollama.
- Run: ollama run llama3.2
- Turn off Wi-Fi and keep chatting.
Use it in code
Point the OpenAI client at Ollama so your app can switch by config.
from openai import OpenAI
local = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = local.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Give me three test questions about refunds."}],
)
print(reply.choices[0].message.content)
# In settings: CHAT_BASE_URL=http://localhost:11434/v1 for dev, provider URL for prodMake your app’s model endpoint configurable and run your integration tests against Ollama in CI.
Common mistakes
- Developing prompts only on a local model and being surprised in production.