Inference

Inference is the act of running a trained model: feed it a context window full of tokens, get back new tokens. Nothing in the weights changes; the model is being used, not trained. Each model provider request triggers one inference pass, and inside that pass the model emits its answer token by token, which is why responses stream in rather than appear at once.

Inference is where cost and latency come from. The provider runs the model on GPUs, charges per token processed, and needs time proportional to how much it has to read (input tokens) and how much it has to write (output tokens). A long session with a bloated context makes every single inference pass slower and more expensive, because the model re-reads the whole window each time.

You can run inference locally with tools like Ollama or llama.cpp, trading capability for privacy and zero per-token cost. For agentic coding that's usually a step down: the models that fit on a laptop are noticeably weaker at multi-step work. The practical levers are elsewhere: keep context lean, use a cheaper model for routine steps, and let the prefix cache absorb what repeats.