How LLMs Actually Work
Tokens, next-token prediction, context windows, and why temperature exists — the mental model everything else builds on.
An LLM does one thing: given a sequence of tokens, it predicts a probability distribution over the next token. Everything else — chat, agents, reasoning — is scaffolding around that loop. Tokens are subword chunks (roughly ¾ of an English word each), and the model sees nothing but their ids: no letters, no words, no meaning it wasn't trained into. This is why models miscount letters in words and why token-efficient prompts are cheaper — you pay per token in, per token out.
The context window is the model's entire working memory: system prompt, conversation, retrieved documents, tool results — all of it competes for the same budget (hundreds of thousands of tokens on frontier models in 2026, still finite). Nothing outside the window exists to the model. There is no hidden database of your conversation; if it scrolled out, it's gone. Most 'the model forgot' bugs are context-budget bugs.
Sampling settings shape how the distribution becomes output. Temperature near 0 makes the model pick the most likely token nearly every time — good for extraction and structured output. Higher temperature spreads probability across alternatives — useful for brainstorming, dangerous for JSON. When you need determinism-ish behavior, lower the temperature and pin the prompt; when you need diversity, raise it and sample more than once.