AI Explained

Why AI makes things up, and how to build systems that don’t

Deloitte refunded an Australian government client after shipping a report with fabricated citations. A BC tribunal made Air Canada pay for a policy its chatbot invented. Neither was bad luck. Here is why hallucinations happen, why dumping raw data into a chatbot makes them worse, and how we engineer around them.

By Adi Huric, founder of Most AI Labs·July 2026·10 min read

In October 2025, Deloitte Australia refunded part of a AU$440,000 government contract after its report was found to contain academic references that do not exist and a quote attributed to a Federal Court judgment that was never written. The report was produced with a GPT-4o toolchain. If a Big Four firm with a review process can ship fabricated sources to a national government, the question for your business is not “is AI safe to use.” It is “what has to be around the AI so this cannot happen to us.”

This article explains, in plain language, why AI systems confidently invent things, what the measured rates actually are, why pasting your raw business data into a chatbot makes everything worse, and the specific engineering that keeps hallucinations out of the systems we build for clients. Every claim is cited. Vendor self-reports are labeled.

What a hallucination actually is

A language model does not look facts up in a database. It generates text by predicting, one token at a time, what plausibly comes next based on patterns learned from training data. That is why its errors do not look like errors: a fabricated court citation has perfect formatting, a plausible case name, and a confident tone, because the model has learned exactly what real citations look like. The fluency is the danger.

In September 2025 OpenAI published the clearest explanation to date of why this keeps happening. The paper’s argument: models are “optimized to be good test-takers,” and on almost every benchmark the industry uses, guessing when uncertain scores better than saying “I don’t know.” We train AI the way we train students facing a multiple-choice exam: a wrong guess might earn points, a blank never does. An earlier theoretical result by the same lead author showed something stronger: a well-calibrated language model must hallucinate at some rate on facts that appear rarely in its training data. Hallucination is not a bug that the next version patches out. It is a statistical property of the method.

Newer models do not automatically fix it

The most counterintuitive evidence comes from OpenAI’s own system cards. Its April 2025 reasoning models hallucinated more than the models they replaced: o3 invented answers on 33% of a person-facts benchmark and o4-mini on 48%, roughly double the older o1’s 16%. Those are the vendor’s own published numbers. GPT-5’s system card (August 2025) reports real progress, claiming over five times fewer factual errors than o3, but the improved rate is still not zero, and it is still a self-report.

Independent measurement says the same thing with more precision. Vectara’s hallucination leaderboard tests the easiest possible task: summarize a document the model was just handed. On the 2026 dataset of 7,700+ real legal, medical, and financial documents, the best models still hallucinate on roughly 2 to 3% of summaries, and several reasoning-focused models run above 20%. And on hard knowledge questions, a November 2025 benchmark across 42 topics found all but three frontier models were more likely to hallucinate than answer correctly. One number to distrust on sight: any vendor quoting a single universal “hallucination rate.” The real rate ranges from about 2% to over 80% depending entirely on the task.

The receipts: what this costs real organizations

·Deloitte Australia, October 2025. Fabricated references and an invented court quote in a welfare-compliance report; AU$97,587 repaid (the widely reported “$290,000 refund” confuses the contract value with the actual repayment). The government learned about the errors from the press, not from the firm.
·Air Canada, February 2024, right here in BC. The airline’s chatbot invented a retroactive bereavement-fare policy. The BC Civil Resolution Tribunal rejected Air Canada’s argument that the chatbot was “a separate legal entity responsible for its own actions” and ordered the airline to pay. The award was small ($812.02). The precedent is not: your business is liable for what your AI tells your customers.
·The courts, at scale. A researcher-maintained database counted 1,598 court decisions worldwide involving hallucinated AI material by June 2026, up from about 200 a year earlier. On one day in March 2026, 17 separate US decisions flagged suspected AI fabrications. Vancouver contributed a famous early example: a lawyer who filed two nonexistent ChatGPT-generated cases and was personally ordered to pay costs (Zhang v. Chen).
·Finance is next in line. FINRA’s 2026 oversight report added a dedicated generative AI section requiring firms to test and continuously monitor AI outputs. In the UK, a July 2026 regulator-commissioned review urged deciding within months whether AI chatbots giving financial guidance need to be brought inside the regulatory perimeter, after chatbots were documented telling savers the annual ISA limit was £25,000 when the legal limit is £20,000. Plausible, specific, and wrong is exactly the failure mode money cannot tolerate.

Even the tools built specifically to eliminate hallucinations do not eliminate them: a peer-reviewed Stanford study found dedicated AI legal research platforms still hallucinated on 17 to 33% of queries, while one vendor marketed “100% hallucination-free” citations. Treat every accuracy claim as marketing until someone independent has measured it.

Why dumping raw data into a chatbot makes it worse

The most common AI mistake we see in businesses is also the most intuitive-feeling one: paste the spreadsheet, the ledger, the contract pile into a chatbot and ask questions. The evidence says this fails in four distinct ways.

·Models do not reliably read what you give them. On FinanceBench, a benchmark of questions about real company filings, GPT-4-Turbo paired with retrieval answered incorrectly or refused 81% of the time, and even with the entire document loaded into its context window it still failed 21% of questions. Research on long inputs shows why: accuracy drops when the relevant fact sits in the middle of a long document, and a 2025 study found 11 of 13 models advertising 128K-token windows lost over half their accuracy by just 32K tokens when the question did not share literal wording with the source.
·Models are bad at math on your data. A 2025 study of LLMs on tabular data found they manage simple lookups but show large deficits on exactly what a business needs: averages, totals, correlations. The reliable pattern, established since 2023, is having the AI write code that computes the answer: on hard arithmetic benchmarks, code generation held around 61% accuracy while direct in-the-head reasoning collapsed to roughly 20%.
·Messy input multiplies error. A 2025 finance benchmark that added realistic messiness (typos, OCR noise, incomplete questions) found even the most robust model hallucinated in 41% of the perturbed cases. Your raw exports are the messy case, not the clean one.
·The privacy exposure is now enforced, not theoretical. In May 2026, Canada’s federal privacy commissioner, jointly with BC, Alberta, and Quebec, found OpenAI’s training-data collection “overbroad” and without valid consent under Canadian law; BC’s commissioner went further, finding consent for scraped training data cannot be obtained under BC’s PIPA. Client data pasted into a consumer chatbot may become someone else’s training data, which is why Samsung banned the tools company-wide after engineers pasted proprietary code into ChatGPT, and why the IRS now tells tax practitioners that client data belongs only in vetted, secured tools.

How we build systems that don’t make things up

Hallucination cannot be eliminated from the model, so it has to be engineered out of the system. This is the difference between handing an employee a chatbot subscription and deploying AI that your business can stand behind. The stack we use at Most AI Labs maps directly onto the published evidence:

·Ground every answer in your verified sources. The AI answers from your actual documents, policies, and data, retrieved at question time, never from its memory of the internet. Grounding is the single best-evidenced mitigation: a 2025 clinical study got hallucination down to 5.8% with a well-built retrieval pipeline. But the Stanford legal-tools numbers (17 to 33% with retrieval) are why grounding alone is never our whole answer.
·Numbers come from code, not from the model’s head. When a client system needs a total, a margin, or a comparison, the AI writes the query and deterministic code computes it against the database. The model orchestrates; it does not do mental arithmetic. This is the pattern with the strongest benchmark evidence behind it.
·“I don’t know” is a designed outcome. We configure systems to abstain and escalate to a human rather than guess, the exact behavior OpenAI’s research says the industry under-rewards. The best calibration results in independent benchmarks come from models tuned to refuse when uncertain, and we treat a refusal as a feature, not a failure.
·Structure in, structure out. Inputs get cleaned and structured before the model sees them; outputs conform to fixed schemas that downstream code validates. No free-text answer ever writes directly into an invoice, a booking, or a client message.
·Humans review where the stakes are. Regulators have converged on this: the IRS now tells tax practitioners AI due diligence “cannot be delegated to an algorithmic process,” and Ontario’s broker regulator holds licensees accountable regardless of what tool produced the output. We design the review step into the workflow instead of hoping someone remembers to check.
·We verify our own AI in public. The system that drafts our Facebook posts is fact-checked against the source article before anything publishes, and drafts get rejected for invented numbers regularly. This week it tried to price a drive-thru AI system at $40,000, a figure that appears nowhere in our research. The draft died in review. That is the workflow working.

Watch out for this

One myth worth killing: turning the model’s “temperature” down to zero does not stop hallucinations. It makes outputs more repeatable, which means the same wrong answer comes back consistently. A 2026 study across 90 model and hardware combinations found zero temperature gave the best accuracy only 60% of the time. Determinism is not truth.

The honest bottom line

·Hallucination is inherent to how language models work. The research question is settled; the engineering question is what surrounds the model.
·Never judge an AI system by its demo. Ask the vendor: what grounds the answers, what happens to numbers, when does it say “I don’t know,” where does a human review, and where does our data go? A vendor without crisp answers to all five is selling you a chatbot with a logo.
·Keep raw data out of consumer chatbots. The accuracy evidence is bad, the arithmetic evidence is worse, and in Canada the privacy exposure is now a matter of regulatory findings, not opinion.
·The chatbot speaks for your company. A BC tribunal has already ruled that what your AI tells a customer binds you. Build accordingly.

If you want AI that generates leads and answers customers without inventing policies or prices, that is what our AI implementation work builds: grounded, verified, human-checked systems on your data, in your systems. It starts with the free 7-day audit. Fixed price. You own what gets built.

Sources

If any of this is your week

Start with the 7-day audit.

7 business days. A real document. Yours to keep — whether you hire us or not.

Start with the 7-day audit Or talk to the founder first

Not all AI is the same. Which model fits which job?

The tier system behind every tool your business will evaluate.

Read→

Accounting

AI for accounting firms: the profession your clients expect to use it

Where hallucination risk meets the new IRS rules for tax practice.

Read→