The model doesn't output a word. It outputs a score for every possible token. How you pick from those scores decides whether the text is safe, dull, creative or wild.
5 minute read
In short
The model outputs logits; softmax turns them into probabilities; sampling picks one token.
Temperature sharpens (low) or flattens (high) the probabilities.
Top-k keeps a fixed number of tokens; top-p keeps as many as needed to reach a probability total.
In plain words
At each step the model produces one number per token in its vocabulary. These raw scores are called logits. A function called softmax turns them into probabilities that add up to 1, and then one token is picked. Add it to the text, run the model again, and repeat.
You control the picking with a few settings. Temperature makes the probabilities sharper or flatter. Top-k and top-p throw away unlikely tokens before picking.
Worked example: I'd like a cup of ...
Pretend the vocabulary has four tokens, and the model gives these logits for the next one.
token
logit
probability
tea
2.0
0.609
coffee
1.0
0.224
water
0.5
0.136
sand
−1.0
0.030
The probability column is softmax: raise e to each logit and divide by the total. Now change the temperature. It divides every logit before softmax.
token
T = 0.5
T = 1
T = 2
tea
0.842
0.609
0.434
coffee
0.114
0.224
0.263
water
0.042
0.136
0.205
sand
0.002
0.030
0.097
A low temperature makes the favourite stronger: tea goes from 61% to 84%. A high temperature flattens everything: sand, a bad answer, goes from 3% to almost 10%. At temperature 0 the model always takes the top token, which is called greedy decoding.
1
Top-k: keep only the k most likely tokens
With k = 2, keep tea and coffee, drop the rest, and rescale so they add up to 1.
Top-p: keep the smallest set that reaches probability p
With p = 0.9, add from the top: tea 0.609, plus coffee 0.834, plus water 0.970. That passes 0.9, so keep three and drop sand.
tea 0.629 coffee 0.231 water 0.140
Top-p adapts by itself. When the model is sure, a single token may reach 0.9 and it keeps just that one. When the model is unsure, it keeps many. Top-k always keeps the same number, whether or not that makes sense.
Your turn
You're extracting invoice totals into a database and need the same answer every time. Which setting fits?
What to remember in practice
Use low temperature for extraction, classification and code. Use moderate temperature for writing and ideas. Change temperature or top-p, not both at once, so you know which change had the effect.
Even at temperature 0, answers can differ slightly between runs, because of small numerical differences in how the hardware does the arithmetic. If you need exactly repeatable results, store the output instead of regenerating it.
Common mistakes
Saying temperature makes the model 'more creative' without explaining that it reshapes the probabilities.
Using a high temperature for tasks that need exact answers, like code or JSON.
Thinking temperature 0 guarantees identical outputs; batching and hardware can still cause small differences.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
What does temperature do, mathematically?
The model outputs a score, called a logit, for every token in the vocabulary. Softmax turns those into probabilities. Temperature divides every logit by T before the softmax. With T below 1, the differences between logits are stretched, so the top tokens get even more of the probability and outputs become more predictable. With T above 1, differences shrink and the distribution flattens, so less likely tokens get picked more often. As T approaches 0, it becomes greedy decoding, always taking the top token. It's often combined with top-k or top-p, which cut off the unlikely tail entirely before sampling, so high temperatures don't produce nonsense.
More to think about
What's the difference between top-k and top-p sampling?
Why might a model give different answers at temperature 0?