LLM (large language model)
LLM (large language model): definition, how it works and use in customer service
Definition.
A large language model (LLM) is an AI language model trained on huge amounts of text to understand and generate human language. It recognizes statistical patterns in language and, based on them, predicts the most likely next word in an answer. Well-known examples are GPT-4o, Mistral and Llama. In customer service, LLMs form the technical foundation of modern AI agents and chatbots.
An LLM doesn't answer questions by looking them up in a database. It generates text word by word, based on probabilities it has learned from billions of training examples. That's exactly what makes LLMs so versatile – but it also brings a well-known weakness: without additional safeguards, they can invent information that sounds plausible but is wrong.
How does an LLM work?
An LLM breaks text into small units called tokens and learns from huge amounts of text which tokens are statistically most likely to follow each other. When generating an answer, the model predicts token by token what comes next and assembles coherent sentences from them.
The context window is important for understanding this. It describes how much text (conversation, documents, instructions) an LLM can process at once before older information drops out of view. Larger context windows allow longer conversations and more background knowledge per request, which is especially relevant in customer service when an AI agent has to take the previous conversation into account.
Which LLMs are available today?
OpenAI's GPT-4o is one of the best-known LLMs and is used both in consumer products such as ChatGPT and in business applications. Mistral is a European model often chosen for its efficiency and the option of hosting within the EU. Meta's Llama is an openly available model that companies can run on their own infrastructure. Anthropic's Claude is also increasingly used in professional settings, partly because of its focus on reliable behavior. For DACH companies, where a model is hosted is relevant when choosing a provider, because it directly affects GDPR compliance.
LLM or chatbot: what's the difference?
An LLM is the underlying technology; a chatbot or AI agent is the application that uses an LLM. A classic, rule-based chatbot works with predefined decision trees and can only answer questions that were programmed in advance. An LLM-powered chatbot, by contrast, also understands unfamiliar phrasing and can respond flexibly to new questions because it generates language instead of just working through rules. The difference shows especially in special cases: a rule-based bot fails with unexpected phrasing, while an LLM-based system interprets the intent behind it and formulates a suitable answer. You'll find a detailed comparison in the article AI agents vs. chatbots.
What are the limits of LLMs?
The best-known weakness of LLMs is hallucination. The model generates statements that sound linguistically correct and confident but are wrong in substance, because it fills gaps in its own knowledge with statistically plausible but invented information. A second point is timeliness: an LLM only knows the information in its training data and by default knows nothing about events after that, such as current prices, stock levels or company policies. Third, an LLM only processes what lies within its context window. In very long conversations or with extensive documents, relevant context can be lost.
How are LLMs used in customer service?
In customer service, LLMs provide the language understanding of chatbots and AI agents. To make them answer reliably and based on facts, most professional providers combine the LLM with retrieval-augmented generation (RAG). The model then specifically accesses a verified, company-owned knowledge base instead of answering freely from its training knowledge. This significantly reduces hallucinations, because the LLM only processes information that actually belongs to the company. Providers like melibo use this principle to connect AI agents in customer service to current order data, return status or contract information instead of relying on the general world knowledge of the underlying LLM. The glossary entry Retrieval-augmented generation (RAG) explains more about how it works technically.
What do the GDPR and EU AI Act mean for using LLMs?
As soon as an LLM processes personal data – which in customer service is practically always the case – the GDPR applies. Companies need a data processing agreement (DPA) with the provider, and customer data must not be used to train third-party models. The melibo article How to use chatbots in your company in a GDPR-compliant way explains how to do this in practice. In addition, the EU AI Act under Art. 50 (Regulation 2024/1689) imposes a transparency obligation: users must be able to recognize that they are talking to an AI system. For sensitive industries, Germany's BSI explicitly recommends hosting within the EU, ideally in Germany, to minimize data protection risks.
Conclusion
An LLM is the language technology behind modern chatbots and AI agents, not the application itself. It generates answers based on statistical probabilities and can therefore be both surprisingly flexible and error-prone, especially when it comes to hallucinations and outdated information. For use in customer service, what matters is how a provider safeguards the LLM: through retrieval-augmented generation, a verified knowledge base and GDPR-compliant hosting within the EU. Providers like melibo connect the LLM to company-owned data in exactly this way, so answers stay fact-based instead of made up. If you're choosing an AI system for customer service, you shouldn't just ask which LLM runs in the background but above all how it is connected to your own knowledge base.
