Running Models Locally with Ollama

Ollama wraps llama.cpp with model management and a local API server. Install it, then pull and chat with a small open-weight model:

# install from ollama.com, then:
ollama run llama3.2
# first run downloads the quantized model (~2GB); then you're chatting locally
# also try: qwen2.5-coder, mistral, deepseek-r1

Ollama serves an OpenAI-compatible endpoint on localhost:11434 — your existing client code works by changing the base URL:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

r = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Classify: 'refund my order' → intent?"}],
)
print(r.choices[0].message.content)

A Modelfile bakes a system prompt and parameters into a named local model — project memory, local edition:

# Modelfile
FROM llama3.2
PARAMETER temperature 0.2
SYSTEM """You are the support-ticket classifier for Acme.
Output exactly one label: billing | bug | feature_request | other."""

# build and run it:
#   ollama create acme-classifier -f Modelfile
#   ollama run acme-classifier

Local wins on: privacy/compliance (data never leaves), cost at sustained volume (hardware amortizes), offline/edge, and latency for small models near the user. Cloud frontier models win on: peak capability, zero ops, and burst scale. The honest middle path most teams land on — local small models for high-volume simple tasks (classification, extraction, PII scrubbing pre-cloud), frontier models for the hard reasoning. Quantization makes local practical; know it trades a little quality for a lot of memory.