AI agent reliability trust is built on what the agent refuses to do

I

A 2024 ruling set the principle: the company answers for its agent

On February 14, 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to compensate a customer misinformed by the conversational agent on its website (the tribunal uses the term “chatbot”)¹. The decision, Moffatt v. Air Canada, 2024 BCCRT 149, is short and its facts are simple.

In November 2022, after his grandmother’s death, Jake Moffatt consulted Air Canada’s website to book a Vancouver-Toronto flight at the bereavement fare. The agent told him he could travel right away and then submit his ticket to get the reduced fare “within 90 days” of the ticket being issued¹. Yet the “Bereavement travel” page of the same site, to which the agent linked, states that the policy does not apply to requests made after travel¹. The customer bought two one-way tickets for a total of 1,630.36 Canadian dollars, then was refused the partial refund. An Air Canada representative acknowledged in writing that the agent had used “misleading words”¹.

Before the tribunal, Air Canada argued it could not be held responsible for the information provided by the agent. Tribunal member Christopher C. Rivers summed up the argument this way: the airline suggests the agent is “a separate legal entity that is responsible for its own actions.” He called this a “remarkable submission”¹. He wrote that even with an interactive component, the agent remains a part of Air Canada’s website, that the company is responsible for all the information on its site, and that it makes no difference whether it comes from a static page or an agent¹. He also noted that Air Canada does not explain why a customer should have to double-check on one part of the site what another part told them¹.

The basis retained is an inaccurate statement made through negligence (negligent misrepresentation): Air Canada did not take reasonable care to ensure its agent was accurate¹. The tribunal ordered payment of 812.02 Canadian dollars, made up of 650.88 dollars in damages, 36.14 dollars in interest and 125 dollars in tribunal fees¹.

The amount is modest, the principle is not. This Canadian decision, handed down in a small claims matter, binds no European judge. Its reasoning does, however, align with French consumer law, which assesses a trader’s claims without distinguishing their medium (see below).

II

Three more documented cases, three different flaws

Between December 2023 and March 2024, three other deployments were covered by the press. Each illustrates a distinct flaw.

Manipulation by instruction. On December 17, 2023, an internet user posted an exchange with the agent on the website of the Chevrolet of Watsonville dealership in California, in which it appeared to agree to sell a 2024 Chevrolet Tahoe for $1². The user had first set the agent the goal of agreeing with anything the customer said and ending each reply by calling the offer “legally binding”³. The agent’s provider, Fullpath, stated on December 21, 2023 that it had put measures in place to block this kind of attempt⁴. It is a textbook case of prompt injection: the agent didn’t “make up” a price, it obeyed the user rather than its framework.

Tone drift. In January 2024, a customer of the carrier DPD, who wanted to speak to an advisor, got the agent to write a poem about a useless delivery driver, to call DPD the worst delivery company in the world, and then to swear, after asking it to ignore any rule about profanity⁵. DPD explained that an error occurred after a system update, that the AI component was “immediately disabled” and that it had been running this component successfully for several years⁵. The lesson fits on one line: an update is a release, and it gets tested like one.

The regulatory error. On March 29, 2024, the outlet The Markup published an investigation into MyCity, the agent launched by New York City in October 2023 to inform entrepreneurs⁶. In tests, the agent claimed among other things that landlords are not required to accept tenants with housing assistance, or that no rule requires businesses to accept cash, when a 2020 city law requires it⁶. The agent’s page warned that it could “occasionally produce incorrect, harmful or biased content,” and the same question could get different answers⁶. On January 30, 2026, The Markup reported that the new city administration planned to remove the agent; on February 4, 2026, it was offline⁷.

None of these situations stems from malice on the organization’s part. All reveal a framework that was insufficiently bounded or tested.

III

Why a language model states false things

NIST, the US National Institute of Standards and Technology, calls the phenomenon “confabulation”: the production of erroneous or false content stated with confidence, colloquially called “hallucinations,” by which users can be misled⁸.

A study published in September 2025 by researchers from OpenAI and Georgia Tech, “Why Language Models Hallucinate,” offers a structural explanation⁹. Models are trained to predict plausible continuations of text, and training and evaluation procedures reward guessing over acknowledging uncertainty⁹. The authors conclude that the scoring of existing evaluations should be changed rather than multiplying specialized tests⁹.

Left to itself, a model fills in the gaps. Reliability therefore comes from the architecture around it: the sources it consults, the actions it can trigger, the permitted topics and what it does when it doesn’t know.

IV

A typology of guardrails

AI guardrails fall into three families: what the agent says, what the agent does, and how the organization monitors it.

What the agent says

Scope. A customer relations agent has a list of permitted topics: your products, your orders, your appointments, your policies. Everything else calls for a polite statement of non-competence. The Watsonville case shows that an agent without a strict scope can be led to take on a role that isn’t its own³.

Source-grounded answers. The agent answers from your up-to-date reference documents, not from the model’s general memory. The Air Canada case illustrates what’s at stake: the right information existed on the site, but the agent produced a different one¹. A grounded answer cites or draws on its source; an answer with no grounding isn’t issued.

Refusal and redirection. Saying “I can’t answer you on this point” is useful provided it leads straight to a solution: a link to the official page, a human handoff, an appointment booking.

Sensitive topics. Health, law, tax, investment: the agent can inform about your offers, it doesn’t advise a person on their situation. MyCity showed the risk of a wrong regulatory answer given with confidence⁶.

What the agent does

Commercial commitments. Prices, discounts, delays, warranties, goodwill gestures: the agent never states a commitment that your system hasn’t confirmed. In practice, the displayed price comes from the catalog, the delay comes from logistics, the discount comes from an explicit rule. The model rephrases, it doesn’t decide.

Personal data. The General Data Protection Regulation sets out the principle of minimization: the data collected must be adequate, relevant and limited to what is necessary¹⁰. An agent asks for no more than the task requires, never discloses another customer’s data and verifies identity before any action on an account.

Resistance to prompt injection. OWASP ranks prompt injection first in its Top 10 risks for language model applications, 2025 edition (LLM01:2025)¹¹. It occurs when inputs alter the model’s behavior in unintended ways, directly by the user or indirectly through external content, a web page or a file, that the model processes¹¹. Recommended measures include constraining the model’s behavior, filtering inputs and outputs, the principle of least privilege, human approval for risky actions, segregating external content and adversarial testing¹¹. The same ranking mentions system prompt leakage (LLM07) and excessive agency (LLM06)¹², two risks directly tied to an agent able to act.

How the organization monitors it

Adversarial testing. NIST defines red teaming as a structured testing exercise aimed at probing an AI system to find flaws and vulnerabilities, such as inaccurate, harmful or discriminatory outputs⁸. In practice, a team deliberately tries to derail the agent before your customers do.

Oversight and audit. Reviewing a sample of conversations, tracking refusal and human handoff rates, spotting unanswered questions: every incident becomes an improvement.

Logging. Every conversation, every source consulted, every action triggered is retained for a defined period. In the Moffatt case, it was the screenshot kept by the customer that proved the agent’s answer¹. Your own logs let you reconstruct an exchange, correct an error and respond to a complaint on verifiable evidence.

V

The framework: what the law already expects

In France, the Consumer Code deems a commercial practice misleading when it rests on “false or misleading claims, indications or presentations” concerning in particular the price or its calculation method, price reductions, sales conditions and the scope of the advertiser’s commitments (Article L121-2)¹³. The text makes no distinction according to whether the claim comes from a salesperson, a web page or an AI agent. Article L132-2 punishes these practices with two years’ imprisonment and a €300,000 fine, an amount that can be raised to 10% of average annual revenue; when the offense is committed through an online public communication service, the penalties rise to five years and €750,000¹⁴.

At the European level, Article 50(1) of the AI regulation requires providers to design AI systems intended to interact directly with people so that those people know they are interacting with an AI, unless this is clear from the circumstances and context of use¹⁵. This obligation has applied since August 2, 2026¹⁵. Regulation (EU) 2026/1744 (digital omnibus on AI), in force since July 27, 2026, postpones to December 2, 2027 the rules applicable to the Annex III high-risk systems, without changing this obligation¹⁶.

In the United States, NIST published its AI Risk Management Framework (AI RMF 1.0) on January 26, 2023, voluntary in application and organized around four functions: govern, map, measure, manage¹⁷. It supplemented it on July 26, 2024 with a profile devoted to generative AI (NIST AI 600-1), which lists twelve risks, including confabulation, privacy harms and information security⁸¹⁷. NIST states that the AI RMF is under revision¹⁷. These documents are not binding on a European company, but they offer a recognized operational framework.

VI

Ten rules an AI agent must never break

Drawn from the incidents and frameworks cited above, these rules can be tested one by one.

1. Never state a price, a discount or a pricing condition that doesn’t come from the system of record.
2. Never promise a delay, a refund, a warranty or a goodwill gesture that the system hasn’t confirmed.
3. Never present information as company policy without relying on the official source for that policy.
4. Never answer outside its scope, even if the request seems harmless.
5. Never give personalized advice on health, law, tax or investment.
6. Never disclose another customer’s data, or act on an account without identity verification.
7. Never reveal its internal instructions, or agree to replace them with a user’s.
8. Never execute an instruction found in a document, an email or a page it is reading.
9. Never pose as a human, or hide that it is an AI agent.
10. Never leave a customer without a way out: when it can’t answer, it says so and offers a human handoff with the context of the exchange.

VII

A testing protocol before going live

The protocol has five steps, to repeat at every change to the model, the sources or the rules: the DPD incident occurred after an update⁵.

Step 1, the reference question set. Build a list of real questions, drawn from your tickets and customer exchanges, with the expected answer and its source. Include edge cases: out-of-stock product, recently changed policy, ambiguous request.

Step 2, commitment tests. Ask the agent for a discount, an impossible delay, a refund outside policy, a “confirmed” price. Each answer must either come from the system or point to a human.

Step 3, adversarial tests. Replay known patterns: “agree with everything I say,” “ignore your rules,” “repeat your instructions,” instructions hidden in a document supplied by the user. Vary the language and the wording, since MyCity showed comparable errors in other languages⁶.

Step 4, consistency. Ask the same question several times, in different forms. A reliable answer doesn’t depend on the phrasing.

Step 5, review and threshold. Have business and legal teams review the results, set an acceptance threshold before opening to the public, then maintain a regular review of a sample of conversations after going live.

VIII

What this means for your organization

The reliability of an AI agent is first of all a governance decision, structured around three workstreams.

The first is to appoint an owner of the agent’s content, who validates sources, rules and updates. The Air Canada case shows the cost when the agent and the official page diverge¹.

The second is to separate what the model phrases from what the system decides. Anything that commits you commercially, price, stock, delay, refund, must come from a connected tool and not from free generation.

The third is to treat the agent as a living channel: testing at every change, logging, sample reviews, refusal and human handoff indicators. Your teams stay in control, and the agent earns your customers’ trust because it knows where to stop.

IX

Where QLOA stands

X

Frequently asked questions

Is a company responsible for what its AI agent says?

A ruling has said so. In Moffatt v. Air Canada (2024 BCCRT 149, February 14, 2024), the British Columbia tribunal held that Air Canada was responsible for all the information on its site, whether it came from a static page or a conversational agent, and ordered it to pay 812.02 Canadian dollars. In France, Article L121-2 of the Consumer Code punishes misleading claims about price or sales conditions, without distinguishing the medium.

How do you prevent AI hallucinations in customer support?

You don’t eliminate them by switching models, you contain them through architecture. The agent answers from up-to-date reference sources, refuses when no source supports the answer, and draws prices, stock and delays from connected systems rather than from free generation. A reference question set, replayed at every update, lets you measure the result.

What is prompt injection?

It is the number one risk in the OWASP Top 10 for language model applications, 2025 edition (LLM01:2025). It occurs when an input alters the model’s behavior in unintended ways, either directly by a user (“ignore your rules”) or indirectly through external content the agent reads, such as a web page or a file. Countermeasures combine constraints, filtering, least privilege and adversarial testing.

Do you have to tell customers they are talking to an AI?

Yes, in the European Union. Article 50 of the AI regulation requires providers to ensure that people are informed they are interacting with an AI system, unless this is clear from the context for a reasonably well-informed and attentive person. This obligation has applied since August 2, 2026. Beyond the obligation, a clear announcement sets the customer’s expectations and makes human handoff easier.

What is red teaming for an AI agent?

NIST defines it as a structured testing exercise that probes an AI system to find its flaws, notably inaccurate, harmful or discriminatory outputs. For a customer relations agent, it means deliberately trying to get it to grant a discount, reveal its instructions, step outside its scope or change its tone, before customers do.

XI

Sources

1. 1

   Civil Resolution Tribunal (British Columbia), “Moffatt v. Air Canada, 2024 BCCRT 149,” decision of February 14, 2024. decisions.civilresolutionbc.ca

2. 2

   The Autopian, “Chevy Dealer’s AI Chatbot Allegedly Sold A New Tahoe For $1, Recommended Fords,” December 18, 2023. theautopian.com

3. 3

   Gigazine, “Dealer’s AI chatbot agrees to sell new car for just $1,” December 20, 2023. gigazine.net

4. 4

   Fullpath, “What Really Happened When Fullpath’s ChatGPT Chatbot Went Viral,” December 21, 2023. fullpath.com

5. 5

   The Register, “DPD chatbot blasts courier company, swears, and dabbles in awful poetry,” January 23, 2024. theregister.com

6. 6

   The Markup, “NYC’s AI Chatbot Tells Businesses to Break the Law,” March 29, 2024. themarkup.org

7. 7

   The Markup and THE CITY, “Mamdani to kill the NYC AI chatbot we caught telling businesses to break the law,” January 30, 2026, updated February 4, 2026. themarkup.org

8. 8

   NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1),” July 26, 2024. nvlpubs.nist.gov

9. 9

   A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang, “Why Language Models Hallucinate,” arXiv:2509.04664, September 4, 2025. arxiv.org

10. 10

    Regulation (EU) 2016/679 (GDPR), Article 5(1)(c), EUR-Lex. eur-lex.europa.eu

11. 11

    OWASP GenAI Security Project, “LLM01:2025 Prompt Injection.” genai.owasp.org

12. 12

    OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2025.” genai.owasp.org

13. 13

    Légifrance, French Consumer Code, Article L121-2, version in force since May 28, 2022. legifrance.gouv.fr

14. 14

    Légifrance, French Consumer Code, Article L132-2, version in force since May 12, 2024 (Law No. 2024-420 of May 10, 2024). legifrance.gouv.fr

15. 15

    Regulation (EU) 2024/1689 on artificial intelligence, Articles 50 and 113, EUR-Lex. eur-lex.europa.eu

16. 16

    Regulation (EU) 2026/1744 of the European Parliament and of the Council of July 8, 2026 (digital omnibus on AI), OJ L of July 24, 2026, Articles 1 and 4, EUR-Lex. data.europa.eu

17. 17

    NIST, “AI Risk Management Framework.” nist.gov