A language model always produces an answer, even when it doesn't know one. Understanding why is the first step to designing around it.
4 minute read
In short
Models produce the most likely text, not the true text, so they guess fluently when they don't know.
Rare facts, outdated knowledge and leading questions make hallucination more likely.
Ground answers in sources, allow 'I don't know', and verify high-stakes claims.
In plain words
A hallucination is a fluent, confident answer that is false: a made-up citation, a wrong date, an invented function. It isn't a bug in one model. It follows from how all language models work.
The model was trained to produce the most likely next token. It was never trained to say 'I don't know' unless its training data taught it to. When it lacks the fact, it still produces something that sounds like the right kind of answer.
Worked example: a fact the model half-knows
Ask a model the founding year of a small company it saw only a few times in training. Its probabilities for the next token might look like this.
next token
probability
2014
0.35
2015
0.30
2013
0.20
2016
0.15
1
The model is unsure
No year has more than 35%. The model's own numbers say it doesn't know.
2
Decoding hides that
Greedy decoding picks 2014 and writes it as plainly as a fact it knows for certain. The text carries no trace of the 35%.
"The company was founded in 2014."
3
Ask again, get another answer
With temperature above 0, a second run may say 2015. Different answers to the same factual question is a strong hallucination signal.
Why it happens
cause
what it means
Rare facts
Seen too few times in training to be stored reliably.
No 'I don't know' habit
Training rewards plausible answers, so guessing is the default.
Outdated knowledge
Anything after the training cutoff simply isn't there.
Pressure from the prompt
A question that assumes something false often gets an answer that plays along.
Your turn
Your support bot invents refund rules that don't exist. Which fix attacks the cause most directly?
What to do about it
Ground answers in retrieved sources and ask for citations, so claims can be checked. Give the model permission to abstain. For high-stakes outputs, verify: check citations exist, compare several samples, or have a second step check claims against the source.
Measure it. Build a test set of questions where you know the answer, including ones the model can't answer, and track how often it makes something up instead of declining.
Common mistakes
Treating hallucination as a bug that a better prompt can fully remove.
Trusting a model's stated confidence; models often sound equally sure when wrong.
Adding retrieval but not checking that answers actually use the retrieved text.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Why do language models hallucinate?
A language model is trained to produce likely continuations of text, not to check facts. When it lacks the knowledge, the most fluent continuation is still a confident-sounding answer, so it fills the gap with something plausible. Training data contains errors and gaps, rare facts are learned weakly, and training often rewards giving an answer over saying 'I don't know'. Sampling adds randomness on top. You can't remove it entirely, but you can reduce it: ground answers in retrieved sources and ask for citations, let the model say it doesn't know, use lower temperature for factual tasks, check claims automatically against the sources, and measure hallucination rates in your evaluations.
More to think about
How would you reduce hallucinations in a customer-facing assistant?
How would you measure the hallucination rate of a system?