epochs

Projects · Beginner · 2–3 hours

Build a tokenizer

Teach a program to split text into tokens the way GPT-style models do, by learning merges from a book.

What you'll build

A byte-pair encoding tokenizer that learns its vocabulary from any text file and tokenizes new text with it.

You'll need

Python 3 and NumPy where the code imports it. Any plain-text book, such as Pride and Prejudice from Project Gutenberg (gutenberg.org/ebooks/1342).

Learn first

Steps

Tick each step off as you finish it. Your progress is saved on this device, and to your account when you sign in.

  1. Count the words

    Read the book into one string and count every word. BPE learns from these counts, so a word that appears 500 times counts 500 times.

    Python
    import re
    from collections import Counter
    
    def word_counts(text):
        """How often each word appears. BPE learns from these counts."""
        return Counter(re.findall(r"[a-z']+", text.lower()))

    Check: word_counts(book).most_common(5) shows very common words like 'the' and 'and'.

  2. Count neighbouring pairs

    Each word is a tuple of symbols, starting as single letters. Count every pair of neighbours, weighted by the word's count.

    Python
    def pair_counts(words):
        """Count neighbouring symbol pairs, weighted by how often each word appears."""
        pairs = Counter()
        for symbols, n in words.items():
            for a, b in zip(symbols, symbols[1:]):
                pairs[(a, b)] += n
        return pairs

    Check: For {('c','h','a','t'): 4}, each of (c,h), (h,a) and (a,t) is counted 4 times.

  3. Learn the merges

    Repeatedly merge the most common pair everywhere and record it. The list of merges, in order, is your tokenizer.

    Python
    def merge_word(symbols, pair):
        out, i = [], 0
        while i < len(symbols):
            if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
                out.append(symbols[i] + symbols[i + 1])
                i += 2  # skip ahead so one symbol is never used in two merges
            else:
                out.append(symbols[i])
                i += 1
        return tuple(out)
    
    def learn(text, merges):
        words = {tuple(w): n for w, n in word_counts(text).items()}
        rules = []
        for _ in range(merges):
            pairs = pair_counts(words)
            if not pairs:
                break
            best = max(pairs, key=pairs.get)
            rules.append(best)
            merged = {}
            for w, n in words.items():
                new = merge_word(w, best)
                merged[new] = merged.get(new, 0) + n
            words = merged
        return rules

    Check: learn(book, 300) finishes in under a minute, and the first merges are very common English letter pairs.

  4. Tokenize new text

    Apply the merges to each word, in the order they were learned.

    Python
    def tokenize(text, rules):
        tokens = []
        for word in re.findall(r"[a-z']+", text.lower()):
            symbols = tuple(word)
            for pair in rules:  # merges in the order they were learned
                symbols = merge_word(symbols, pair)
            tokens.extend(symbols)
        return tokens

    Check: A word the book never contains still tokenizes, into smaller known pieces.

  5. Measure it

    More merges means a bigger vocabulary and fewer tokens per word.

    Python
    def report(text, rules):
        words = re.findall(r"[a-z']+", text.lower())
        tokens = tokenize(text, rules)
        print(f"{len(words)} words -> {len(tokens)} tokens ({len(tokens) / len(words):.2f} per word)")
        print("longest tokens:", sorted(set(tokens), key=len, reverse=True)[:8])

    Check: Try 100, 300 and 1,000 merges: tokens per word falls each time.

You're done when
  • Learns 300 merges from a whole book
  • Tokenizes a sentence it has never seen
  • Shows tokens per word falling as merges increase

Go further

  • Work on bytes instead of letters, so emoji and every language are covered
  • Save the merges to JSON and load them back
  • Compare your token counts with a real tokenizer such as tiktoken