Assay

Documented, sourced, and checked against what Assay can and cannot do

When AI made things up

Text written by a chatbot was wrong in a way that cost someone money or standing. Here is what happened in four cases, what each cost, and, honestly, whether Assay would have caught it.

Written from a court decision, a tribunal decision, published studies and press reports. Figures are as reported there, not a measure of the full harm. This is not legal advice: read the disclaimer. Spotted a mistake? Open an issue on GitHub.

  1. 2023

    At a product launch · 8 February 2023

    Google’s Bard demonstration

    Worldwide

    $100bnabout this much market value fell that day, with other factors at work

    What happened

    A promotional post for Google’s chatbot showed it answering that the James Webb Space Telescope took the first pictures of a planet outside our solar system. The first such pictures were taken by a different telescope in 2004. Reuters pointed out the error.

    What it cost

    Alphabet’s shares fell about 7.7% that day, roughly $100 billion of market value.

    The fall cannot be pinned on the error alone. The same week Microsoft announced a chatbot for its search engine, and investors were also reacting to worries that Google was losing ground. Read the figure as the size of the reaction, not the price of one mistake.

    Would Assay have caught it? No, not on its own.

    This was a wrong general-knowledge statement with no source supplied, and none of Assay’s checks compares a statement with the world at large. Given a source, such as the telescope’s mission page, the support check could test the sentence against it. The lesson is that Assay needs something to compare against.

    Sources: Global News: Bard chatbot wrong answer, shares plummet; The Week: AI chatbot makes $100bn mistake.

  2. 2023

    In court · Sanctions order, 22 June 2023

    Mata v. Avianca

    US District Court, Southern District of New York

    $5,000sanction on the lawyers and their firm

    What happened

    Lawyers for a passenger suing an airline filed a brief citing court decisions that did not exist. A chatbot had written them, complete with plausible names and citations. The court found the citations fictitious and, more seriously, that the lawyers kept vouching for them after the court asked about them.

    What it cost

    A $5,000 fine, an order to write to each judge whose name had been put on a fake opinion, and the passenger’s case dismissed as filed too late.

    The dollar figure is small. The reputational cost, and the lesson for every lawyer after them, is not.

    Would Assay have caught it? Yes, for the fake citations.

    The legal check looks every case citation up in CourtListener and catches a real citation attached to the wrong case. It has not yet been run live with a token. No check fixes the second failure: standing by a citation after being asked about it.

    Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), 678 F. Supp. 3d 443 (S.D.N.Y. 2023). Sources: Bloomberg Law: Phony ChatGPT brief leads to $5,000 fine.

  3. 2024

    To a customer · Decision, 14 February 2024

    Moffatt v. Air Canada

    Civil Resolution Tribunal of British Columbia

    $812ordered in total ($650.88 in damages, plus interest and fees)

    What happened

    After a family death, a customer asked the airline’s website chatbot about bereavement fares and was told he could claim the discount after travelling. The airline’s actual policy did not allow that. The airline argued the chatbot was a separate entity responsible for its own words. The tribunal rejected that.

    What it cost

    The tribunal found negligent misrepresentation and ordered the difference in fare: $650.88 in damages, about $812 in total with interest and fees. Small money, but the ruling is that a company answers for what its chatbot tells customers.

    Secondary reports sometimes call the $812 figure the damages. The damages were $650.88; the total includes interest and tribunal fees.

    Would Assay have caught it? Partly.

    The policy was written down on the airline’s own site. Checking each chatbot answer against that approved text, with grounding (every statement traces to an approved fact) or support (a judge you supply, with its evidence verified against the policy), would have flagged a refund rule the policy did not contain. Both need the policy text to be supplied.

    Moffatt v. Air Canada, 2024 BCCRT 149. Sources: Civil Resolution Tribunal decision, 2024 BCCRT 149; Manatt: Airline has to honor a refund policy its chatbot fabricated.

  4. 2025

    In a government report · Commissioned December 2024; refund reported October 2025

    Deloitte Australia’s review for the Department of Employment and Workplace Relations

    Australia

    A$290,000reported refund on a report of about A$440,000; the amount was not officially disclosed

    What happened

    A report of about A$440,000 on a welfare compliance system contained references to academic works that did not exist and a misquoted federal court judgment, found by an academic reader. A corrected version disclosed that a generative AI tool had been used in producing it.

    What it cost

    The department said the firm agreed to repay the final instalment under the contract. The amount was not officially disclosed; some outlets put it at roughly A$290,000. The firm maintained that the findings of the report were unchanged.

    A researcher who raised the alarm argued that some corrections replaced fake references with other doubtful ones, which is what happens when a model is asked to fix its own citations without a check.

    Would Assay have caught it? Yes.

    The citations check looks up every reference. The quotes check compares a quotation with the text of the judgment it is attributed to. The repair loop asks the model again, but only keeps a replacement that the checks then verify.

    Sources: Malay Mail: Deloitte to pay partial refund to Australian government; TechSpot: Deloitte admits AI hallucinated quotes in government report.

Measured, not just reported

Studies that counted the problem instead of waiting for it to be reported. Each is a rate for particular models, prompts and dates.

  • 55%of GPT-3.5 citations fabricated

    Walters and Wilder (2023): 636 citations across 42 topics. For GPT-4, 18% were fabricated; of the real citations, 43% (GPT-3.5) and 24% (GPT-4) had substantive errors.

    Read the study
  • 47%of references fabricated

    Bhattacharyya and colleagues (2023): 115 references in medical papers written by ChatGPT-3.5. 46% were real but inaccurate; only 7% were real and accurate.

    Read the study
  • ≥58%hallucination on legal questions

    Dahl and colleagues (2024): public language models hallucinated at least 58% of the time on legal queries and often accepted users’ wrong legal assumptions.

    Read the study
  • 17–33%in tools sold as “hallucination-free”

    Magesh and colleagues (2025): commercial legal research tools, tested in a preregistered study. Fewer than general chatbots, far from none.

    Read the study

Courts are keeping count too. Damien Charlotin maintains a public database of court decisions in which a judge addressed AI-generated content, mostly invented citations; press coverage in mid-2026 put it above 1,600 entries. It lists only what was caught and reported, so it is a floor. See the database.

What the cases have in common

  1. The text looked finished.

    Fluent, formatted, confident. Nothing in it said “unchecked”.

  2. The facts were checkable.

    A citation exists or it does not. A refund policy is written down. A quotation is in the document or it is not.

  3. Nobody checked, or someone checked and kept going.

    No tool replaces the person who decides. Assay’s job is to put the unchecked things in front of that person, in plain words, before the filing, the report or the launch.

Try the offline checksWhat counts as evidenceWhat Assay cannot do