Projects · Beginner · 2–3 hours
A tiny language model
Generate new names by predicting one letter at a time, the same idea as an LLM predicting tokens.
A character-level model that learns which letter tends to follow which, generates new names, and measures how well it predicts.
Python 3 and NumPy where the code imports it. A list of names, one per line, such as names.txt from Andrej Karpathy's makemore project on GitHub (github.com/karpathy/makemore).
Steps
Tick each step off as you finish it. Your progress is saved on this device, and to your account when you sign in.
Load names and build a vocabulary
Every character gets an id. A dot marks both the start and the end of a name, so the model also learns how names begin and end.
Python import numpy as np def load_names(path): with open(path, encoding="utf-8") as f: return [line.strip().lower() for line in f if line.strip()] def vocab(names): chars = sorted(set("".join(names))) itos = ["."] + chars # "." marks both the start and the end of a name return itos, {c: i for i, c in enumerate(itos)}Check: itos[0] is '.', and the rest are the letters that appear in your names.
Count which letter follows which
This table is the whole model: row a, column b counts how often b came straight after a. Dividing each row by its total turns counts into probabilities.
Python def bigram_counts(names, stoi): N = np.zeros((len(stoi), len(stoi))) for name in names: seq = ["."] + list(name) + ["."] for a, b in zip(seq, seq[1:]): N[stoi[a], stoi[b]] += 1 return N def probabilities(N, smoothing=1): P = N + smoothing # add-one smoothing: nothing has probability zero return P / P.sum(axis=1, keepdims=True)Check: Every row of P adds up to 1.
Generate names
Start at the dot, sample the next letter from that row, and repeat until you sample the dot again. Temperature reshapes the probabilities exactly as it does in an LLM.
Python def generate(P, itos, temperature=1.0, rng=None): rng = rng or np.random.default_rng() out, i = [], 0 while True: logits = np.log(P[i]) / temperature p = np.exp(logits - logits.max()) i = rng.choice(len(p), p=p / p.sum()) if i == 0: return "".join(out) out.append(itos[i])Check: Temperature 0.5 gives safer, common-looking names; 1.5 gives stranger ones.
Measure how good it is
Loss is the average surprise per character. A model that guessed uniformly would score log(vocabulary size).
Python def avg_loss(names, P, stoi): """Average negative log-likelihood per character: lower means the model predicts names better.""" total, count = 0.0, 0 for name in names: seq = ["."] + list(name) + ["."] for a, b in zip(seq, seq[1:]): total -= np.log(P[stoi[a], stoi[b]]) count += 1 return total / countCheck: Your loss is clearly lower than np.log(len(itos)).
- Generates names that look plausible
- Shows how temperature changes the output
- Reports a loss lower than uniform guessing
Go further
- Use the previous two letters instead of one and compare the loss
- Hold out 10% of names and measure loss on those
- Replace the count table with a small neural network trained by gradient descent