TL;DR — the essentials in 30 seconds
- A hallucination is an AI answer that sounds plausible but is based on made-up or wrongly attributed information, not on a visible system error.
- The three most common triggers in customer service: an incomplete or outdated knowledge base, missing guardrails and a model unsuited to the use case.
- Retrieval-augmented generation (RAG) significantly reduces hallucinations but doesn't eliminate them completely. That applies to every provider.
- Hallucination rates can't be measured with BLEU. Established approaches are faithfulness metrics such as Vectara HHEM or RAGAS, plus manual sampling against real tickets.
- According to a 2026 ServiceNow study, consumers in Germany lose over 470 million hours a year to poor customer service. Trust is the scarce resource no company should squander with made-up answers.
Definition:
In an AI customer service system, a hallucination is an answer that sounds linguistically correct and convincing but is based on made-up, outdated or wrongly attributed information. Unlike a technical error, it often stays invisible to customers because it looks like a plausible, genuine answer.
An AI chatbot that tells a customer the wrong return period or invents a discount rule that never existed hasn't “made a mistake” in the classic sense. It has hallucinated: produced an answer that's grammatically flawless and sounds confident but is entirely made up. That's exactly what makes hallucinations in customer service so tricky – they look like correct answers until a customer complains or an invoice doesn't add up.
What are LLM hallucinations, and why are they especially critical in customer service?
A hallucination happens because a language model (LLM) is trained to produce the statistically most likely next word, not the factually correct one. If the model lacks reliable knowledge on a question, it fills the gap with a plausible-sounding guess without flagging it as a guess.
In customer service, this weighs more heavily than in many other AI applications, for three reasons. First, the answer reaches a real person with a real concern – wrong information about the return period or a warranty claim has legal and financial consequences. Second, the error often goes unnoticed: unlike a chatbot that obviously crashes, a hallucination sounds just as confident as a correct answer. Third, the problem scales: a single support agent might occasionally make a mistake, but an automated system can repeat the same mistake across thousands of tickets before anyone notices.
That trust in AI-powered customer service is fragile anyway is shown by a recent study by ServiceNow and ThoughtLab (“The CX Shift: Customer Expectations in the AI Era”, 2026): consumers in Germany lose more than 470 million hours every year to poor customer service, and 42 percent switch to a competitor after a single negative experience. The study measures general service quality, not hallucination rates specifically, but it shows how little buffer companies have when it comes to trust. In this climate, a made-up answer is no minor slip. (Source: Callcenterprofi)
The three most common causes of hallucinations in customer service AI
An incomplete or outdated knowledge base is the most common cause. If an AI agent finds no suitable information in its knowledge base for a question but is still expected to answer, the underlying model falls back on its general training knowledge, which naturally doesn't apply to your specific product range, your terms and conditions or your current discount campaigns. Outdated FAQ articles, duplicate or contradictory entries and missing product data make the problem worse.
Missing guardrails are the second cause. These are technical and prompt-level boundaries that force a system to answer only from verified sources and to escalate rather than guess when in doubt. Without such guardrails, a model also answers questions for which it has no solid basis, often without any recognizable uncertainty in its wording.
The wrong choice of model is the third, often underestimated cause. Not every powerful model is automatically the most factually accurate. An interesting finding from the current Vectara hallucination leaderboard: reasoning models, which invest extra compute in “thinking”, tend to hallucinate more when summarizing documents than simpler models, because while reasoning they add conclusions and connections that aren't in the source text at all. For customer service answers that should be based strictly on your own knowledge base, this is an effect providers need to actively counter when choosing a model. (Source: Suprmind)
RAG as the standard countermeasure: what retrieval-augmented generation really does
Retrieval-augmented generation (RAG) connects a language model to an external, controlled knowledge base. Instead of drawing on trained-in, often outdated model knowledge, the system first searches your own data for the right content before every answer – such as product data, terms and conditions or return policies – and passes it to the model as context. The answer is then based on concrete, traceable sources rather than pure language memory. The concept goes back to an influential paper by Lewis et al. from 2020 and has since become the standard approach against hallucinations in production AI systems. How it works technically is explained in our glossary entry on retrieval-augmented generation (RAG).
Important for setting expectations: RAG noticeably reduces hallucinations; it doesn't eliminate them. This is also confirmed by Fraunhofer IESE in an analysis of LLM hallucinations: even with a RAG architecture, a model can weight or misunderstand the retrieved information. Two factors decide quality in practice: how well the knowledge base is maintained and whether the system actually escalates when sources are missing instead of forcing an answer.
Checklist: six measures before an AI customer service system goes live
- Clean up the knowledge base and keep it up to date. Identify and remove duplicate, contradictory or outdated content before the system goes live.
- Define escalation rules. Decide at what level of uncertainty a case is handed over to a human agent instead of forcing an uncertain answer.
- Sharpen the system prompt and guardrails. Explicitly instruct the model to answer only on the basis of the knowledge base provided, not general training knowledge.
- Representative sampling before go-live. Run a sample of real, historical tickets through the system and manually check the answers against the correct solution.
- Ongoing monitoring after go-live. Regular spot checks instead of one-off acceptance, since product range, terms and conditions keep changing.
- Add a transparency notice. Users must be able to recognize that they're talking to an AI system. This is mandatory from August 2, 2026 under Art. 50 EU AI Act; you'll find the classification for customer service chatbots in our glossary entry on the EU AI Act risk categories. The notice also builds trust, because customers know where they stand.
How do you measure hallucination rates in practice?
A common misconception: BLEU scores aren't suitable for this. BLEU measures how much a generated text overlaps with a reference text in word choice – a measure of translation and text quality, not factual accuracy. A model can achieve a good BLEU score and still claim something completely wrong, as long as the wording sounds similar.
Instead, specialized faithfulness metrics have become established. The best-known example is the Vectara Hallucination Evaluation Model (HHEM), which checks whether a generated answer is actually supported by the source documents. For RAG systems specifically, faithfulness evaluation has also become common: checking whether every statement in the answer can be traced back to the retrieved documents. Manual sampling remains indispensable as well: a sample of real tickets in which trained staff check every AI answer against the actually correct information. No automated method fully replaces this human review, especially for more complex or ambiguous requests.
Which models are more prone to hallucinations in a customer service context?
An important caveat first: the only publicly and continuously updated benchmark in this area, the Vectara leaderboard, measures factual accuracy when summarizing short to medium-length documents, not open customer service dialogs directly. According to Vectara, though, it's a good indicator of accuracy in RAG and agent pipelines, because the model summarizes search results there too. With this caveat in mind, it's worth looking at individual model families, as of May 2026.
For OpenAI, GPT-4.1 has a hallucination rate of 5.6 percent, while GPT-4o (version 2024-08-06) is at 9.6 percent – a sign that newer versions aren't automatically more accurate. The Mistral models tested perform comparatively well: Mistral Large (2411) reaches 4.5 percent, Mistral Small (2501) 5.1 percent. For Meta, Llama 3.3 70B Instruct is at 4.1 percent, while the newer Llama 4 models (Scout and Maverick) are slightly higher at 7.7 and 8.2 percent respectively. For Anthropic, Claude Haiku 4.5 is at 9.8 percent and Claude Sonnet 4 at 10.3 percent. (Source: Vectara hallucination leaderboard on GitHub)
The key message for customer service decision-makers isn't “model X is the safest” but that the differences between model families are smaller than the effect of a good or bad RAG architecture. In practice, a weaker model with a clean knowledge base and strict guardrails regularly beats a stronger model without either.
How melibo and other providers address the problem technically
Providers developing for DACH customer service, such as melibo, Userlike or moinAI, now almost all rely on RAG architectures in which the model only accesses the knowledge base provided by the company instead of drawing on free model knowledge. The differences between providers show less in the basic principle than in the details: how granularly escalation rules can be configured, how closely the system is connected to existing systems like Shopware, Zendesk or Freshdesk, and how transparently the provider documents its architecture for customers.
Legally, the same basis applies to all providers with German or European customers. Processing personal data in an AI customer service system must follow the principles of Art. 5 GDPR and be based on a data processing agreement under Art. 28 GDPR. For companies that value hosting transparency, it's also worth looking at the BSI Cloud Computing Compliance Criteria Catalogue (C5), which has been available in the revised C5:2026 version since 2026 and defines minimum information security requirements for cloud services.
Hallucination or AI slop?
Not every unsatisfying AI answer is a hallucination. When a chatbot answers in smooth language but with empty content, without inventing false facts, that's called AI slop. The difference matters in practice: a better knowledge base and RAG help against hallucinations, while clearer prompts and quality control of the answer structure mainly help against AI slop. Both problems can occur at the same time but require different countermeasures.
Conclusion
Hallucinations aren't a fringe problem that will disappear with the next model update; they're a structural property of language models that has to be contained in customer service with concrete technical and organizational measures. A clean, well-maintained knowledge base, clear escalation rules and a RAG architecture reduce the risk considerably but don't eliminate it completely. If you test before go-live, measure after go-live and escalate to people rather than guess when in doubt, you protect not just individual tickets but the trust that's scarce in customer service anyway.
