epochs

Learn · Module 16 · ML from scratch · Lesson 4 of 5

A BPE tokenizer

Turn the tokenization lesson into code: learn merges from word counts, then apply them to new words.

4 minute read

In short
  • Learning: count weighted neighbouring pairs, merge the most common, repeat.
  • Tokenizing: apply merges in learned order; unseen words split into known pieces.
  • Real tokenizers work on bytes so every text is covered.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.