Projects · Beginner · 2–3 hours
Build a tokenizer
Teach a program to split text into tokens the way GPT-style models do, by learning merges from a book.
A byte-pair encoding tokenizer that learns its vocabulary from any text file and tokenizes new text with it.
Python 3 and NumPy where the code imports it. Any plain-text book, such as Pride and Prejudice from Project Gutenberg (gutenberg.org/ebooks/1342).
Steps
Tick each step off as you finish it. Your progress is saved on this device, and to your account when you sign in.
Count the words
Read the book into one string and count every word. BPE learns from these counts, so a word that appears 500 times counts 500 times.
Python import re from collections import Counter def word_counts(text): """How often each word appears. BPE learns from these counts.""" return Counter(re.findall(r"[a-z']+", text.lower()))Check: word_counts(book).most_common(5) shows very common words like 'the' and 'and'.
Count neighbouring pairs
Each word is a tuple of symbols, starting as single letters. Count every pair of neighbours, weighted by the word's count.
Python def pair_counts(words): """Count neighbouring symbol pairs, weighted by how often each word appears.""" pairs = Counter() for symbols, n in words.items(): for a, b in zip(symbols, symbols[1:]): pairs[(a, b)] += n return pairsCheck: For {('c','h','a','t'): 4}, each of (c,h), (h,a) and (a,t) is counted 4 times.
Learn the merges
Repeatedly merge the most common pair everywhere and record it. The list of merges, in order, is your tokenizer.
Python def merge_word(symbols, pair): out, i = [], 0 while i < len(symbols): if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair: out.append(symbols[i] + symbols[i + 1]) i += 2 # skip ahead so one symbol is never used in two merges else: out.append(symbols[i]) i += 1 return tuple(out) def learn(text, merges): words = {tuple(w): n for w, n in word_counts(text).items()} rules = [] for _ in range(merges): pairs = pair_counts(words) if not pairs: break best = max(pairs, key=pairs.get) rules.append(best) merged = {} for w, n in words.items(): new = merge_word(w, best) merged[new] = merged.get(new, 0) + n words = merged return rulesCheck: learn(book, 300) finishes in under a minute, and the first merges are very common English letter pairs.
Tokenize new text
Apply the merges to each word, in the order they were learned.
Python def tokenize(text, rules): tokens = [] for word in re.findall(r"[a-z']+", text.lower()): symbols = tuple(word) for pair in rules: # merges in the order they were learned symbols = merge_word(symbols, pair) tokens.extend(symbols) return tokensCheck: A word the book never contains still tokenizes, into smaller known pieces.
Measure it
More merges means a bigger vocabulary and fewer tokens per word.
Python def report(text, rules): words = re.findall(r"[a-z']+", text.lower()) tokens = tokenize(text, rules) print(f"{len(words)} words -> {len(tokens)} tokens ({len(tokens) / len(words):.2f} per word)") print("longest tokens:", sorted(set(tokens), key=len, reverse=True)[:8])Check: Try 100, 300 and 1,000 merges: tokens per word falls each time.
- Learns 300 merges from a whole book
- Tokenizes a sentence it has never seen
- Shows tokens per word falling as merges increase
Go further
- Work on bytes instead of letters, so emoji and every language are covered
- Save the merges to JSON and load them back
- Compare your token counts with a real tokenizer such as tiktoken