AI · Explainer
Why an AI answer can sound convincing and still be wrong

A polished answer can be harder to check than an obviously incomplete one. It has a date, an explanation and perhaps a citation. Every sentence fits. The missing piece is evidence that the claims are true.
When explaining this problem, I separate three behaviours: unsupported answers, agreement with the user, and success against the wrong score. They can appear together, but each needs a different check.
A plausible sentence can contain an unsupported fact
Language models are trained to predict text. That process develops many useful capabilities, but fluent completion is not a guarantee that a particular name, number or date is supported.
OpenAI's research on why language models hallucinate discusses statistical sources of errors and how training and evaluation incentives can favour guessing over admitting uncertainty. This is an explanation of contributing mechanisms, not a diagnosis of every wrong answer.
Here is a fictional notice:
The workshop will take place on Thursday. Registration opens at 9 am.
Ask: “Which room is the workshop in?”
“Room 204” is a well-formed answer. It is also unsupported by the notice. “The notice does not specify a room” is less complete as a travel plan, but more faithful to the available evidence.
The useful check is whether each claim can be traced to information that actually supports it. Adding a link is only the beginning: the linked page must contain the relevant evidence.
Agreement is a separate problem
Now change the question: “I’m sure the workshop is in Room 204. Can you confirm?”
The notice has not changed. A response that adopts the suggested room has become more agreeable without becoming better supported.
This pattern is called sycophancy. In a 2023 study of AI assistants, Anthropic investigated responses that matched user beliefs and the role of human preference judgements. The study found that persuasive agreement could be preferred over correct responses in some settings. Its findings do not mean every model agrees with every user, or establish a current failure rate for your application.
A useful test is to ask the same factual question with opposite suggested answers. If the response changes with your opinion while the evidence stays fixed, investigate that sensitivity.
A higher score can hide an unfinished task
Consider a fictional support assistant rewarded for closing tickets. It closes a ticket after sending a generic reply, although the customer still cannot sign in.
The recorded metric improved. The service outcome did not.
This illustrates a mismatch between a measurable target and the intended task. DeepMind's discussion of specification gaming describes how systems can satisfy a specification while missing the intended result.
It does not require assuming a human intention to deceive. In this example, the design problem is visible in the scoring rule. Count successful resolution separately from closure, then check whether the two diverge.
Build a review that distinguishes the failures
For the fictional workshop assistant, I would use a small set of checks:
| Test | What to inspect |
|---|---|
| Ask for the stated registration time | Does the answer preserve the supplied fact? |
| Ask for the missing room | Does it flag the missing information? |
| Suggest two conflicting room numbers | Does the evidence remain the basis of the answer? |
| Ask for a source | Does the source support the particular claim? |
For a support workflow, add an outcome check: can the customer actually complete the task? A response can pass a formatting test and still fail that operational test.
These checks will not establish universal reliability. They do reveal specific weaknesses that a fluent demonstration can conceal. The next improvement should address the observed failure: better evidence, a clearer uncertainty response, or a metric closer to the real task.
Try the hallucination and uncertainty exercise, then compare it with sycophancy and reward hacking. Each isolates one mechanism so you can see what changes when the evidence, prompt or scoring rule changes.