AI hallucinations: why you must always check the output (and how)

The biggest risk of AI is not that it makes mistakes, every tool does. The risk is that AI makes mistakes with full conviction. An AI model can present a non-existent source, a wrong amount, or an incorrect legal rule in exactly the same confident tone as a correct answer. We call that a “hallucination”, and it is the reason checking is not an option but a necessity.

What exactly is a hallucination?

AI language models predict the most likely next word; they do not “know” anything in the human sense. Usually that produces a fine answer, but sometimes the model fills a gap with something that sounds plausible but is wrong. The model is not lying deliberately, it simply has no sense of true or false.

Where it is most dangerous

• Facts, figures, and calculations, especially in financial work.
• Sources and citations, AI sometimes invents references that do not exist (a real risk for lawyers).
• Legal or medical information, where a mistake has major consequences.
• Anything you forward one-to-one to a client without reading.

A practical checking routine

1. Treat AI output like a draft from an intern, not a finished product. Useful, but always reviewed.
2. Verify facts at the source. An amount, date, or citation you check against the original document.
3. Be extra critical when it suits you. An answer that says exactly what you wanted to hear deserves a second look.
4. Have sensitive work signed off by a human. For client, financial, or legal work, a human final check is non-negotiable.
5. Ask for reasoning. Have the AI explain what an answer is based on; that often makes errors visible.

Checking as part of your AI policy

This routine does not belong in someone’s head, but in your AI policy: record that output is checked, and by whom. That way you benefit from the speed of AI without importing its risks.

Adopt AI responsibly?

We help you set up AI so that speed and care go together, with clear checkpoints. See our AI & automation service or book a conversation.