epochs

Learn · Module 06 · Transformers · Lesson 3 of 6

Choosing the next token

The model doesn't output a word. It outputs a score for every possible token. How you pick from those scores decides whether the text is safe, dull, creative or wild.

5 minute read

In short
  • The model outputs logits; softmax turns them into probabilities; sampling picks one token.
  • Temperature sharpens (low) or flattens (high) the probabilities.
  • Top-k keeps a fixed number of tokens; top-p keeps as many as needed to reach a probability total.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.