Spotting the AI's confident mistakes

When a colleague isn't sure, you can hear it. They hesitate, or say 'let me check with accounts'. AI has no such tone. It gives an invented fact in the same confident voice as a true one. So the checking is your job.

The mistake most people make

People assume that if the AI gives a reference number, a section or a percentage, it must be real. Here is what happens when someone asks it for a tax notification.

What was asked

“Cite the exact CBIC notification that exempts transport services from reverse charge under GST.”

What the AI said

“Under CBIC Notification No. 99/2031-Central Tax (Rate) dated 12th July 2031, goods transport agencies operating under forward charge with registered GSTIN are exempt from reverse charge liability under Section 9(3).”

What went wrong: We invented this notification number and date for the lesson. A real AI reply looks just like it. From the words alone, you can't tell whether such a notification exists or what it covers. The AI didn't look anything up. It wrapped real tax words around a number that fits the pattern. Only the official text, read by your CA, tells you what applies.

How it works: the eager new intern

Picture an intern who joined yesterday. They are scared of looking foolish in front of the managing director. Someone asks, 'What did we spend on packaging last quarter?' They don't go to the files or say they don't know. They make up a likely number and hand it over as fact. You keep the intern, because they type faster than anyone. But you never forward their memo to the board without checking the bills yourself.

Three kinds of AI mistake

1. Invented things

The AI makes up court cases, book titles, website links or registration numbers. They look like the real thing because they follow the same pattern.

2. Invented links between real things

The AI takes two real things and makes up a connection. It might say Company A bought Company B because both work in the same industrial area, when they are rivals.

3. Wrong sums

When the AI adds up rows, it doesn't calculate like a calculator. It predicts digits that look right. The total can look fine and still be wrong.

A quick check before you use any answer

Check every nameMark the names of people, companies, laws and places. Confirm each one appears in your own notes or documents.

Redo every totalNever trust a total the AI worked out. Put the numbers in a spreadsheet or your phone calculator.

Ask where each claim came fromTell the AI: 'For every claim, quote the exact sentence from my document where it appears.'

The Strict Checker

Act as a strict checker looking for mistakes in this text:
"""
[Paste the AI draft or text you want to check here]
"""

Do these three checks:
1. List every claim stated as fact that isn't backed by something in the text or a document I gave you.
2. List every name, number, date, sum, section or rule that a person needs to check. Mark each one [CHECK].
3. Give two reasons the conclusion might be wrong.
Give your findings as a bulleted list. Don't correct any law, tax rule or figure yourself.

Before acting on anything legal, tax or financial in this text, have a CA or lawyer confirm it.

Changing the AI's job from writer to checker makes it point at the claims that have nothing behind them. You still do the final check.

Check your understanding

Why does AI invent a realistic notification number or court case instead of saying it doesn't have one?

Show the answer

The AI predicts words that fit the pattern of the question. If you ask for a citation, the likeliest reply is something shaped like a citation, whether or not it exists.