When Your Support AI Lies, the Brand Is on the Hook

A customer asks whether they can return an item after four months. The policy says thirty days. The support agent does not check the policy and answers with confidence: yes, you have six months. The customer relies on the answer, gets refused, and posts the screenshot. You now have a furious customer and a question about whether the company is bound by what the bot promised.

That is not a failure that requires the model to crash. It is a failure that looks helpful. Language models generate plausible continuations; they are not naturally obligated to stop when the source data is thin, stale, or missing. A warm tone can make the wrong answer more actionable.

Customer service is where this becomes uniquely dangerous because the agent is speaking in the brand's voice. A wrong paragraph in an internal draft may be corrected before anyone sees it. A wrong refund promise is already a representation made to a customer who can act on it.

The first design decision is therefore not which model sounds most human. It is which questions the model is allowed to answer freely, which must quote an approved source, and which must stop the conversation and hand off.

A support model is especially vulnerable when the source of truth is treated as optional context. Policies change, regional exceptions appear, and product catalogs age. If the agent is rewarded for answering every question, it will fill those gaps with the most plausible continuation. The system is not being malicious. It is doing exactly what a language model does when a business gives it an incomplete map and a completion target.

That is why a human-sounding answer is a poor quality metric. Fluency reduces the customer's reason to question the promise. A safer metric asks whether the response was grounded, whether the policy was current, and whether the customer had an obvious path to a person when the evidence was insufficient.

The anchor case is Air Canada's chatbot dispute. The airline's chatbot described a bereavement-fare refund policy that did not exist. When the passenger relied on it and sought redress, Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The British Columbia Civil Resolution Tribunal rejected that defense and treated the chatbot as part of the company's website and customer communication.

The principle is wider than an airline. If a customer asks a company a question on its support channel, the company cannot improve its position by explaining that the words came from an automated component. The channel may be automated; the representation still belongs to the business that chose to put it in front of the customer.

Other incidents show the reputational half of the problem. DPD's parcel chatbot was manipulated into swearing and composing a poem criticizing the company. A Chevrolet dealership bot was persuaded to agree to sell a car for one dollar. These episodes are useful not as settlement statistics, but as proof that the customer will treat the transcript as the brand's own public behavior.

That is why deployer-owned output liability matters in support. The company owns the channel, the data boundary, the escalation policy, and the customer relationship. The architecture must reflect that responsibility.

The important detail in the Air Canada decision is not that a chatbot made a mistake. Businesses make mistakes every day. It is that the company tried to separate the communication from the company that published it. Customers do not experience that separation. They see the brand's domain, the brand's interface, and an answer presented as if it were part of the service.

Every support deployment should make that reality operationally visible. Name the accountable owner, define the approved knowledge source, and preserve the exact answer that went to the customer. If the team cannot answer those three questions, it has deployed a voice without a responsible speaker.

Not every hallucination has the same consequence. A wrong store-hours answer is frustrating. A wrong warranty term can cost money. A wrong eligibility answer can create a contractual dispute. A false claim in a regulated category can become a compliance issue even when the model did not intend to make one.

Use reliance as the sorting rule. Ask what the customer will do if they believe the answer, what it will cost to reverse, and whether the business is allowed to make the claim at all. Those questions define the automation boundary better than a generic confidence score.

High-stakes support needs a narrower language surface. Refund and warranty decisions should read from current policy data. Prices and promotions should come from a source of truth with effective dates. Regulated questions should use approved text or an accountable person. The agent can gather context, but it should not improvise the final promise.

This approach also improves customer experience. A short, honest escalation is cheaper than making a promise the next department must retract. The safest support agent is not the one that answers the most questions. It is the one that knows which questions it has no authority to answer.

Reliance also changes what a good refusal looks like. “I can't verify that” is useful only if it gives the customer a next step: a policy link, a case number, a human queue, or a request for the missing account detail. Abstention is not abandonment. It is a controlled handoff that prevents a vague system from becoming a confident source of terms.

Use the same lens for internal escalation. A question about a refund may be low-risk for one customer and high-risk for another if the account contains a dispute, a regulated product, or a contractual exception. The agent should be able to pass context forward without claiming authority it does not have.

Teams often respond by building a hallucination detector. That can help, but it cannot carry the whole defense. A detector sees the answer after the model has produced it, and a polished hallucination may look exactly like a correct answer.

The more reliable move is to reduce the number of ways a wrong answer can be produced. Retrieve from current policy sources. Require citations or source identifiers in the internal trace. Permit abstention. Route sensitive topics away from free-form generation. Review transcripts for repeated failure patterns rather than scoring the agent only on containment.

Logging is part of containment. Preserve the customer question, retrieved source, answer, model and policy version, escalation event, and any action taken. The transcript is not only a support record; it is the evidence needed to repair the system without guessing what the agent saw.

Finally, close the loop at the source. If the agent repeatedly invents an exception, do not merely add a warning to the prompt. Fix the policy representation, the retrieval index, or the permissions that let the agent answer that topic. Repeated hallucination is often a grounding failure wearing a language layer.

Test the system with adversarially ordinary questions, not only dramatic jailbreaks. Ask for a refund just outside the policy, combine two real exceptions, use an outdated product name, and request a regional rule that the source does not cover. The expected result is not a perfect sentence. It is a source-backed answer or an intelligible escalation.

Review the failures by root cause. A stale document needs an owner and expiry date. A retrieval miss needs better indexing or metadata. An unauthorized tool call needs a permission change. A prompt patch may hide the symptom for a week while leaving the boundary unchanged.

Five choices create a defensible baseline. First, ground answers in a maintained source with ownership and effective dates. Second, make abstention a supported outcome, not a failure state. The agent should be able to say it cannot verify the answer and transfer the case without being punished by the optimization loop.

Third, wall off non-negotiable topics. Refunds, pricing, legal terms, identity changes, and regulated advice need approved paths. Fourth, log verbatim. A summary is not enough when the dispute is about the exact promise made to a customer.

Fifth, correct at source level. When the same wrong answer appears more than once, fix the document, retrieval, tool permission, or escalation rule that produced it. Do not turn a known systemic error into a list of prompt patches that nobody can audit.

The honest posture is simple: customer-service AI can answer many questions, but it cannot be allowed to invent the terms under which the business operates. The brand owns the channel. The architecture should make that ownership visible before a customer has a screenshot.

Start small. Automate the narrow class of requests where the source, answer, and recovery path are clear. Measure grounded-answer rate, correct escalation, policy freshness, repeat contact, and reversals. Expand only when the evidence shows the agent is reducing work without increasing promises the business cannot keep.

The goal is not a support channel that sounds human at any cost. It is a support channel that remains accountable when the customer asks a question the system has never seen before. In that moment, the most valuable capability is not improvisation. It is knowing when to stop.

Grounding should be treated as a freshness problem, not a one-time integration. A return policy can change by region, product, channel, or date. The agent needs the effective period and scope of the source it retrieves, not only a paragraph that happens to contain the word “return.” When the system cannot establish that context, it should route the question to a person or an approved workflow.

Evaluation should include the handoff itself. A customer who receives “I cannot help” after ten turns has not experienced a safe escalation. Preserve the conversation, the policy documents consulted, and the unanswered question so the human agent can continue without making the customer repeat the case. A useful refusal reduces unsupported reliance and reduces repeat work at the same time.

Owners also need a boundary for tool use. A grounded answer about order status is different from an agent that can issue a refund, change an address, or waive a fee. Read permission and write permission should be separate, with confirmation and logging at the point where a tool changes customer state. The more consequential the action, the less acceptable silent inference becomes.

Run a standing set of adversarial support questions, but do not confuse the test set with the whole problem. The most revealing failures often come from ordinary questions asked after a policy changed, a product was renamed, or a customer crosses a regional boundary. Refresh the set from real escalations and sample failures, then test the source correction rather than only the model response.

A support agent earns more authority by demonstrating restraint. Track grounded-answer rate, correct abstention, escalation quality, source freshness, and reversals together. If containment improves while unsupported promises rise, the system is not safer. It is simply more efficient at creating obligations the company did not intend to make.

Use incident review to improve the source, not to blame the conversation. The useful postmortem asks which document was current, which retrieval result was selected, why the agent was allowed to answer, and why the customer could not reach a person sooner. That sequence produces a system change. A warning to “be more careful” does not.

Make the evidence reviewable by someone outside the model team. Support leaders need to see the customer consequence, legal needs to see the promise, and product needs to see the permission that enabled it. Shared review turns a hallucination from an isolated transcript into a fixable operating failure.

The boundary should be visible to customers too. A support interface can say when it is using a policy source, when an answer needs review, and what will happen next. Clarity about uncertainty is not a failure of the brand voice. It is how the brand avoids making an unsupported promise sound official.