epochs

Projects · Beginner · 2–3 hours

A tiny language model

Generate new names by predicting one letter at a time, the same idea as an LLM predicting tokens.

What you'll build

A character-level model that learns which letter tends to follow which, generates new names, and measures how well it predicts.

You'll need

Python 3 and NumPy where the code imports it. A list of names, one per line, such as names.txt from Andrej Karpathy's makemore project on GitHub (github.com/karpathy/makemore).

Learn first

Steps

Tick each step off as you finish it. Your progress is saved on this device, and to your account when you sign in.

  1. Load names and build a vocabulary

    Every character gets an id. A dot marks both the start and the end of a name, so the model also learns how names begin and end.

    Python
    import numpy as np
    
    def load_names(path):
        with open(path, encoding="utf-8") as f:
            return [line.strip().lower() for line in f if line.strip()]
    
    def vocab(names):
        chars = sorted(set("".join(names)))
        itos = ["."] + chars            # "." marks both the start and the end of a name
        return itos, {c: i for i, c in enumerate(itos)}

    Check: itos[0] is '.', and the rest are the letters that appear in your names.

  2. Count which letter follows which

    This table is the whole model: row a, column b counts how often b came straight after a. Dividing each row by its total turns counts into probabilities.

    Python
    def bigram_counts(names, stoi):
        N = np.zeros((len(stoi), len(stoi)))
        for name in names:
            seq = ["."] + list(name) + ["."]
            for a, b in zip(seq, seq[1:]):
                N[stoi[a], stoi[b]] += 1
        return N
    
    def probabilities(N, smoothing=1):
        P = N + smoothing               # add-one smoothing: nothing has probability zero
        return P / P.sum(axis=1, keepdims=True)

    Check: Every row of P adds up to 1.

  3. Generate names

    Start at the dot, sample the next letter from that row, and repeat until you sample the dot again. Temperature reshapes the probabilities exactly as it does in an LLM.

    Python
    def generate(P, itos, temperature=1.0, rng=None):
        rng = rng or np.random.default_rng()
        out, i = [], 0
        while True:
            logits = np.log(P[i]) / temperature
            p = np.exp(logits - logits.max())
            i = rng.choice(len(p), p=p / p.sum())
            if i == 0:
                return "".join(out)
            out.append(itos[i])

    Check: Temperature 0.5 gives safer, common-looking names; 1.5 gives stranger ones.

  4. Measure how good it is

    Loss is the average surprise per character. A model that guessed uniformly would score log(vocabulary size).

    Python
    def avg_loss(names, P, stoi):
        """Average negative log-likelihood per character: lower means the model predicts names better."""
        total, count = 0.0, 0
        for name in names:
            seq = ["."] + list(name) + ["."]
            for a, b in zip(seq, seq[1:]):
                total -= np.log(P[stoi[a], stoi[b]])
                count += 1
        return total / count

    Check: Your loss is clearly lower than np.log(len(itos)).

You're done when
  • Generates names that look plausible
  • Shows how temperature changes the output
  • Reports a loss lower than uniform guessing

Go further

  • Use the previous two letters instead of one and compare the loss
  • Hold out 10% of names and measure loss on those
  • Replace the count table with a small neural network trained by gradient descent