Next-token prediction

Next-token prediction is the single operation a model performs. Given the whole context window so far, it produces a probability for every possible next token, one is chosen, it's appended to the sequence, and the whole thing runs again. A paragraph of prose, a hundred-line function and a structured tool call are all produced the same way, one piece at a time, with no separate 'planning' step outside the text.

This explains a lot of otherwise puzzling behaviour. The model can't go back and edit what it already emitted, so an early wrong assumption propagates through the rest of the answer. It can't 'run' code in its head; it predicts what running code would print, which is a very different thing. And because each token is sampled from a distribution, the same prompt can go a different way next time (non-determinism).

Two habits follow. Get the important constraints in front of the model before it starts writing, because what comes first shapes everything after. And when it goes wrong early in a long output, stop and re-prompt rather than hoping it self-corrects mid-stream; asking it to reason first (effort) gives it text to condition on before the final answer.