A hallucination is fluent, confident-sounding AI content that is incorrect or unsupported by sources or the material provided. It may be an invented number, source, or quotation; a detail missing from the material; or a contradiction of an earlier answer in the same conversation. The US National Institute of Standards and Technology (NIST) uses the term confabulation for this phenomenon.
Why it happens
A language model generates text that follows patterns in its training data, predicting one piece of text at a time. The same process can produce accurate text or errors. As a result, an incorrect answer can sound just as confident as a correct one.
Where to expect it
- Long, open-ended answers and subjects that require specialized knowledge.
- Citations and explanations: an answer may include a source or line of reasoning that appears to support it even when the answer is wrong.
- Tasks that should rely on material you provided: the output may contain details absent from that material.
How to reduce the risk
- Allow AI to say when information is unavailable, and ask it to mark uncertain points.
- Limit the answer to the material you provided and ask for exact quotations to support claims.
- Open the original source for an important claim yourself. An AI citation is a lead, not proof.
These methods reduce errors but do not eliminate them. Check any answer you plan to rely on for something important.
A hallucination is not a lie
The word sounds human, but this is not an intention to mislead. Errors arise through the same process as correct answers, so confidence does not show whether an answer is accurate.