Vector search compares meanings, so it handles different wording.
Keyword search wins on exact names and codes; vector search on paraphrases.
In plain words
Keyword search scores documents by the words they share with the query. Rare words count for more: matching 'refund' says a lot, matching 'the' says almost nothing. The classic scoring method is BM25, a refined version of TF-IDF.
Vector search, from the Tokens and embeddings module, compares meanings instead. It finds 'how do I get my money back' for a question about refunds, even with no shared words.
Worked example: TF-IDF by hand
Search 1,000 documents for 'refund policy'. A word's weight, its IDF, is ln(total documents ÷ documents containing it). Rare words get large weights.
word
in how many documents
IDF
refund
10
ln(1000 ÷ 10) = 4.605
policy
200
ln(1000 ÷ 200) = 1.609
the
990
ln(1000 ÷ 990) = 0.010
1
Score each document
For each query word: times it appears in the document × its IDF. Add them up.
2
Document A: 'refund' twice, 'policy' once
2 × 4.605 + 1 × 1.609 = 10.82
3
Document B: 'policy' three times, no 'refund'
0 + 3 × 1.609 = 4.83
4
Document C: 'refund' once
1 × 4.605 + 0 = 4.61
A wins clearly. B mentions 'policy' three times but never the rare, telling word. BM25 adds two refinements: repeated words give diminishing returns, and long documents are penalised slightly so they don't win just by being long.
Which one when?
query
keyword search
vector search
Error code E-4012
Finds it exactly
May return similar-looking codes
Product name or person's name
Strong
Often weak
"how do I get my money back"
Misses 'refund' documents
Finds them
Different wording, same idea
Weak
Strong
Your turn
Users search your docs for exact error codes like 'E-4012', and vector search keeps returning the wrong codes. What's the best fix?
What to remember
Neither method wins everywhere. Keyword search is fast, exact and great for names, codes and rare terms. Vector search handles paraphrases and vague questions. Production systems usually run both.
Common mistakes
Forgetting document length normalisation, which BM25 adds and plain TF-IDF lacks.
Dismissing keyword search as outdated; it's still essential for exact terms.
Not adding synonyms or query expansion when users and documents use different words.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Explain TF-IDF or BM25.
TF-IDF scores a document for a query by how often each query word appears in it, term frequency, multiplied by how rare that word is across all documents, inverse document frequency. So a rare word like 'refund' counts more than a common word like 'the'. BM25 improves on this in two ways. Term frequency saturates, so the tenth mention of a word adds much less than the first, and scores are normalised by document length, so long documents don't win just by repeating words. It's fast, needs no training, and is excellent for exact matches like names, codes and error messages, which is why it's still used alongside vector search in most production systems.
More to think about
When does keyword search beat vector search?
How would you search a mix of error codes and natural-language questions?