The Reality of AI Hallucinations: Recent Examples

Last week, the AI coding assistant tool Cursor encountered a crisis when its front-line AI support bot named, “Sam”, confidently provided a user with a completely fabricated policy.

The user raised a support ticket to resolve an issue with ending Cursor sessions when switching between multiple devices. Sam responded, “Cursor is designed to work with one device per subscription as a core security feature,” according to a now-deleted Reddit post. The incident quickly gained traction among frustrated developers on Reddit and Hacker News, with many threatening to cancel their subscriptions. Cursor’s leadership issued a retraction explaining that Sam, a front-line AI support bot, had “hallucinated” the policy—it had essentially fabricated information that didn’t exist.

Cursor, AI assistant tool for coding.

This incident involving Cursor’s AI support bot, ‘Sam’, highlights critical issues facing organisations deploying AI:

The lack of clear AI labeling, which led to user confusion—Sam’s response did not indicate that it was generated by an AI system. The company has since confirmed it will rectify this in future communications.
The fact that AI hallucinations aren’t isolated incidents but a systemic challege—even for AI-focused companies.

This isn’t unique. In February 2025, the Morgan & Morgan law firm admitted to citing fabricated legal cases made up by AI in a lawsuit against Walmart. This incident is described as one of the most shocking instances where AI confidently generated false information and illustrates potential consequences such as courtroom sanctions, professional embarrassment, and potential job losses. Similarly, Air Canada lost a legal battle in 2024 after its AI-powered chat assistant fabricated a refund policy that did not exist. The misinformation caused confusion and inconvenience for the traveler. These examples emphasize the tangible risks of LLM hallucinations, where AI confidently ouputs information ungrounded in its knowledge base.

Let’s first dig into the term AI “Hallucination” to understand it more before we dive into how enterprises can address this issue.

Understanding AI Hallucinations

The term “hallucination” in the AI context refers to a phenomenon where an Artificial Intelligence system, particularly large language models (LLMs) and Generative AI, generates outputs that are incorrect, misleading, or entirely fabricated. These outputs often appear plausible, coherent, or confident, especially in customer interactions, even though they are not grounded in factual accuracy or real-world information.

It's an anthropomorphic term that implies the AI is experiencing something similar to human hallucinations—seeing or believing things that aren't there. However, this anthropomorphizing is somewhat misleading. What's actually happening is a fundamental error in the model's pattern recognition and text generation processes.

When pressed for an answer, these models will attempt to provide one based on statistical patterns they've observed—even when they have insufficient data to form a reliable response. They don't have a concept of "I don't know" unless explicitly trained to admit uncertainty. Large language models aren't "imagining" things; they're failing to properly map their outputs, resulting in erroneously constructed responses (“confabulation”) rather than perceived experiences.

Personally, I think the term “hallucination” is misleading or personifies AI systems. This tendency to anthropomorphize is widespread - Sam Altman from OpenAI even noted the cost of users treating ChatGPT politely saying ‘please’ and ‘thank you’ is costing them millions of dollars of compute power. It is evident how easily we project human traits onto these systems. Are we failing to see the machines for what they are? Another term suggested to describe this phenomenon is “confabulation,” which means representing a process of creative gap-filling. Why not simply state them as “factual pattern error”? Too wordy?

How Common Are AI Hallucinations

Hallucinations are not unique to Cursor - they affect virtually all generative AI systems:

Diagram highlighting the hallucination rates of LLM models.

In a benchmarking study, OpenAI’s technology had a hallucination rate of about 3%, Meta’s systems around 5%, Anthropic’s Claude 2 over 8%, and Google’s Palm Chat as high as 27%.
Even top-performing models like GPT-4 and Gemini 2.0 Flash Exp exhibit hallucination rates between 1.3% and 1.8%.
One analysis found factual errors in 46% of chatbot outputs, and by 2023, chatbots were estimated to hallucinate up to 27% of the time depending on the model and task.

Concerningly, as models become more fluent and confident, their errors can become more difficult to detect because they're presented with the same authoritative tone as factual information.

How to Spot AI Hallucinations

Recognizing when an AI system is hallucinating requires vigilance and often domain expertise. Here are key indicators:

Overly specific details that seem improbable or too convenient
References to non-existent sources, studies, or publications
Inconsistencies within the same response or across multiple responses
Perfect but suspicious answers to ambiguous or challenging questions
Confident assertions about recent events that fall outside the model's training data
Too-good-to-be-true solutions to complex problems

As a technical architect using AI in daily work tasks, I've found that cross-referencing critical information and implementing systematic verification processes are essential when leveraging AI outputs for business decisions.

Enterprise Mitigation Strategies

For organizations racing to adopt AI technologies, hallucinations represent a significant business risk. Here's how enterprises can navigate this challenge:

Retrieval-Augmented Generation (RAG)

Cross-checks LLM outputs against external databases in real time, reducing hallucinations by 20–30% in tasks like medical advice or factual queries. For example, RAG verifies current data (e.g., gas prices) instead of relying on outdated training sets.

Implementing Verification Pipelines

Implement self-verification mechanisms within your system architecture to ensure AI outputs are validated prior to delivery. A few techniques to leverage:

CoVe: Encourages models to draft verification questions and independently answers those questions to check consistency.
EVER: Employs external evidence retrieval and verification steps. Corrects outputs by comparing them against verified statements, improving accuracy in high-stake domains.

Develop Domain-Specific Evaluation Benchmarks

A visual representation of the steps in AI Hallucination Benchmark Framework.

Develop a database of factual information relevant to your industry domain, products, and operations. This should include edge cases and common misconceptions. The knowledge base must be updated regularly as the industry facts evolve.
Design prompts that probe the boundaries of the AI’s knowledge. This should include variations of queries with verifiable answers alongside questions where the expected response should be “I don’t know”.
Track both false positives (hallucinated information) and false negatives (refusal to answer known information). Measure hallucination severity using a scoring system to define a benchmark.
Use the domain-specific benchmark before your system deployment as an entry criterion.
Develop feedback loops where hallucinations are identified to improve both the model and the benchmark.

Advanced Prompting

A quick win that requires no architectural redesign is to develop chain-of-thought prompting and negative prompting:

Chain-of-thought prompting is where your query guides the AI models to break down their reasoning process into explicit steps, slowing down inference and creating verification points before arriving at a conclusion.
- Example:
  
  Instead of asking, "What caused the 2008 financial crisis?"
  You would prompt, "Let's think step by step about the causes of the 2008 financial crisis. First, identify the main sectors involved. Then, analyze the key events chronologically. Finally, explain how these factors interconnected to create the crisis."
Negative prompting involves setting explicit boundaries in the instruction, priming for uncertainty, and establishing fallback procedures.
- Example:
  
  "Answer the following question about our company policy. If you're unsure about any details or if the information might not be in your training data, explicitly state that you don't know rather than attempting to guess. Do not reference specific policy numbers, dates, or provisions unless you are certain they exist. If you need to make an assumption, clearly label it as such."

💡Bonus: For those interested, Hugging Face maintains a leaderboard that tracks, ranks, and evaluates hallucinations in LLMs - Click here.

Moving Forward: AI Architecture & Governance

The Cursor incident demonstrates that even AI-focused companies are not immune to the risks posed by AI hallucinations. As enterprises accelerate AI adoption, robust architectural governance becomes increasingly critical.

The solution isn't to abandon AI adoption but to approach it with architectural rigor. By designing systems that complement AI capabilities with appropriate verification mechanisms and human oversight, organizations can harness the benefits while mitigating the risks of hallucinations.

As technology architects, our role has expanded beyond traditional system design to include crafting resilient sociotechnical systems where AI and human intelligence work in concert—each compensating for the limitations of the other.

The race for AI adoption need not be a reckless sprint but a strategic marathon where governance and architecture evolve alongside technological capabilities.

What strategies has your organization implemented to address AI hallucinations? I'd love to hear your experiences in the comments below.

Mitigating AI Hallucinations: Strategies & Best Practices for Tech Leaders