Learn · Module 16 · ML from scratch · Lesson 5 of 5
Sampling strategies in code
Temperature, top-k and top-p in one function, and a quick check that it matches the maths.
4 minute read
- Divide by temperature, softmax, filter with top-k or top-p, renormalise, sample.
- Temperature 0 is greedy decoding: always the top token.
- Test random code by fixing the seed and checking frequencies over many draws.
The idea
From the Choosing the next token lesson: divide logits by temperature, softmax, keep only the top-k tokens or the smallest set reaching top-p, renormalise, then draw one. Temperature 0 means always take the top token.
The code
import numpy as np
def softmax(x):
e = np.exp(x - x.max())
return e / e.sum()
def sample(logits, temperature=1.0, top_k=None, top_p=None, rng=None):
rng = rng or np.random.default_rng()
if temperature == 0:
return int(np.argmax(logits)) # greedy
probs = softmax(np.asarray(logits, dtype=float) / temperature)
order = np.argsort(probs)[::-1] # most likely first
keep = np.ones(len(probs), dtype=bool)
if top_k is not None:
keep[order[top_k:]] = False
if top_p is not None:
cumulative = np.cumsum(probs[order])
cutoff = np.searchsorted(cumulative, top_p) + 1 # smallest set reaching top_p
keep[order[cutoff:]] = False
probs = np.where(keep, probs, 0)
probs /= probs.sum()
return int(rng.choice(len(probs), p=probs))logits = np.array([2.0, 1.0, 0.5, -1.0]) # tea, coffee, water, sand rng = np.random.default_rng(0) print(sample(logits, temperature=0)) # 0 (tea) counts = np.bincount([sample(logits, top_p=0.9, rng=rng) for _ in range(10000)], minlength=4) print(counts.tolist()) # [6266, 2357, 1377, 0] counts = np.bincount([sample(logits, top_k=2, rng=rng) for _ in range(10000)], minlength=4) print(counts.tolist()) # [7269, 2731, 0, 0]
- 1top-p 0.9 keeps three tokens
Over 10,000 draws: 62.7%, 23.6%, 13.8%, and sand never. The lesson predicted 62.9%, 23.1% and 14.0%. Random sampling lands close.
- 2top-k 2 keeps two
72.7% and 27.3%, against the lesson's 73.1% and 26.9%.
- 3Testing randomness
Fix the random seed and check frequencies over many draws, rather than a single output. That's how to test anything random.
Why add 1 after searchsorted in the top-p code?
Take it further
Work on a batch of sequences at once. Add a repetition penalty (lower the logits of tokens already used). Explain why top-p adapts better than top-k when the model is very sure or very unsure.
Common mistakes
- Dividing by a temperature of zero instead of treating it as greedy decoding.
- Forgetting to renormalise after removing tokens with top-k or top-p.
- Testing random code with one sample instead of checking frequencies over many draws with a fixed seed.
Answer in your own words, out loud or on paper, before you open the model answer.
Implement temperature, top-k and top-p sampling.
I'd write one function taking the logits, a temperature and optional top-k and top-p. If the temperature is zero, return the argmax. Otherwise divide the logits by the temperature and apply a stable softmax. Sort the probabilities from most to least likely. For top-k, zero out everything after the first k. For top-p, take the cumulative sum of the sorted probabilities and keep the smallest set whose total reaches p, including the token that crosses it. Renormalise what's left so it sums to one and draw a token with the random generator. To test it, I'd fix the seed and check that over thousands of draws the frequencies match the expected probabilities.
- How would you test a function that returns random results?
- When does top-p behave differently from top-k?
Go further
- PaperThe Curious Case of Neural Text Degeneration (top-p)Holtzman et al., 2019
- ArticleHow to generate text: decoding methodsHugging Face
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.