AI agent KPIs what the deflection rate doesn’t tell you

I

The deflection rate measures an absence, not a result

According to a Gartner survey published in August 2024, only 14% of customer service issues are fully resolved in self-service, and that rate rises to just 36% for requests that customers themselves consider “very simple”¹. The same study indicates that 73% of customers use a self-service channel at some point in their journey¹. This gap between usage and resolution illustrates what the deflection rate does not see.

The deflection rate (or containment rate) is calculated as follows: conversations not handed off to a human, divided by total conversations. Its limitation lies in its definition. A customer who gets an answer and a customer who closes the window out of weariness are counted the same way. So is a customer who gives up on the conversation and calls the switchboard the next day. The harder it is to reach a human handoff, the more the indicator improves, while the experience gets worse.

This indicator remains useful for sizing teams, not for judging success. Zendesk’s CX Trends 2026 report recommends going beyond speed and volume metrics and treating resolution as the true unit of measurement, by asking whether the problem was fully solved the first time².

II

Why the question arises now

In that same report, based on two surveys conducted in June 2025 across 22 countries with 6,182 consumers and 5,115 customer service leaders and agents, 47% of organizations say they currently track AI-specific KPIs, and 86% plan to do so within 12 to 24 months². 78% of leaders believe AI forces them to rethink their success metrics, while 84% consider customer satisfaction (CSAT) to remain the benchmark indicator². Dashboards are being built right now.

Two other signals raise the stakes. First, Gartner predicts that by 2029, agentic AI will autonomously resolve 80% of common customer service issues³. This is a projection, not a finding, but it points to the growing share of exchanges whose measurement will shape the experience. Second, trust is not a given: in a Gartner survey conducted in December 2023 among 5,728 customers, 64% said they would prefer that companies not use AI for their customer service, and their first concern is that it will become harder to reach a person⁴. A dashboard that rewards deflection pushes in precisely that direction.

III

The foundation: the verified resolution rate

Definition. The share of conversations in which the customer’s request was actually handled, and where this handling is attested by an observable signal, not merely presumed from the absence of escalation.

Formula. Resolved and verified conversations, divided by eligible conversations (excluding spam, tests and empty messages).

The difficulty lies in the word “verified”. Three signals, ideally combined, make it possible to establish it.

The completed action is the strongest signal. The agent created the return, changed the delivery address, confirmed the appointment, issued the ticket: the event exists in your management system, timestamped. For transactional requests, it is the most direct proof.

Explicit confirmation comes from the customer, who states in the conversation that their question is settled. Useful for information requests, it remains declarative and partial.

The absence of recontact within a defined window completes the setup. In their 2010 Harvard Business Review article, Matthew Dixon, Karen Freeman and Nick Toman recommend tracking repeat contacts over a period of seven to fourteen days⁵. That work focused on the contact centers of the time: the window is a starting point, to be adjusted to your cycle.

Pitfall. Letting the model itself judge whether the conversation is resolved, with no human review or external signal. An automatic evaluation can help with triage, but it must be calibrated on a sample reviewed by your teams before it serves as an indicator.

IV

Experience quality indicators

Recontact rate

Definition. The share of customers who come back, on any channel, for the same reason within the chosen window. Formula. Customers who recontacted for the same reason within X days, divided by customers who had a conversation. Pitfall. Looking at the same channel only. The customer who abandons the conversation to call or send an email is the typical case that deflection hides. The HBR study finds that 22% of callbacks concern problems downstream of the initial problem, even when it had been handled correctly the first time⁵. The authors also warn that tracking recontacts mechanically makes performance look worse than a first-contact resolution indicator⁵.

Post-conversation CSAT and CES

Definition. CSAT measures the satisfaction reported after the exchange. The Customer Effort Score (CES), introduced in the HBR article, rests on a single question: “How much personal effort did you have to put forth to handle your request?” (our translation), rated from 1 (very low effort) to 5 (very high effort)⁵. Formula. For CSAT, the share of positive responses out of total responses. For the Customer Effort Score, the average score or the share of “low effort” responses.

The authors compared the predictive power of CSAT, Net Promoter Score and the Customer Effort Score on loyalty in service interactions. CSAT proved to be a poor predictor. More broadly, 20% of the “satisfied” customers in their study reported an intention to leave, and 28% of the “dissatisfied” ones an intention to stay⁵. CES predicted loyalty better: among customers reporting low effort, 94% expressed an intention to repurchase and 88% to increase their spending, while 81% of customers who had difficulty resolving their issue reported an intention to speak negatively about it⁵. These results concern stated intentions, collected from more than 75,000 people, and not observed behavior.

Pitfall. Three biases weigh on these surveys. Non-response: customers who gave up rarely answer the questionnaire. Timing: at the end of an exchange, the customer does not yet know whether the promise will be kept. Object: the customer sometimes rates the company’s policy (a refused refund) rather than the agent. Always track the response rate alongside the score, and read CSAT together with the recontact rate.

Time to resolution

Definition. The delay between the customer’s first message and the verified resolution, human handoff included. Formula. Median of the gaps between the timestamp of the first message and the timestamp of the resolution signal. Pitfall. Measuring the AI agent’s first response time, which says nothing about the real handling time.

V

Orchestration indicators

Qualified human handoff rate

Definition. The share of handoffs to a human that are justified: sensitive complaint, out-of-scope case, explicit customer request, decision committing the company. Formula. Justified handoffs, divided by total handoffs, with qualification established on a reviewed sample. Pitfall. Aiming for the lowest possible handoff rate. A justified, fast handoff is a success. The real warning sign is twofold: unjustified handoffs, which reveal a gap in the agent, and missed handoffs, those conversations that should have been transferred and were not.

Handoff quality

Definition. The handoff’s ability to spare the customer from repeating themselves. The HBR article cites, among the recurring effort-related complaints, having to repeat information and switch channels⁵. Formula. Share of handoffs for which the advisor receives the reason, the history of the exchange and the data already collected, and does not have to ask for them again. Pitfall. Only checking that the context is transmitted technically. The useful question is whether the advisor read and used it.

Analysis of uncovered intents

Definition. The mapping of contact reasons the agent does not know how to handle. In the same Gartner survey, 45% of customers who started in self-service felt the company had not understood what they were trying to do, and the most frequent failure was being unable to find relevant content, in 43% of cases¹. Formula. Volume per uncovered intent, relative to total volume, ranked by frequency and by value. Pitfall. Making it a report rather than a work queue. This indicator is only valuable if it feeds the knowledge base and the agent’s rules every month.

VI

Reliability indicators

Error rate audited on a sample

Definition. The share of responses containing false information, a promise that does not comply with your rules or an erroneous action. Formula. Non-compliant responses, divided by audited responses, on a random sample reviewed by people who know your policies, using a written grid. Pitfall. Auditing only the conversations flagged by customers, which do not represent the whole, or only short conversations. The sample must be drawn at random, then reinforced on risk topics: prices, refunds, warranties, contractual terms.

The stakes are not theoretical. In February 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to compensate a customer after the chatbot on its website suggested that the customer could request, after the fact, a bereavement travel fare, which the airline’s policy did not allow⁶. The tribunal rejected the argument that the tool was a separate entity, noting that it should be obvious to Air Canada that it is responsible for all the information on its website⁶. What your AI agent says commits your company: the error rate is a matter of risk management.

VII

Economic indicators

Conversion and assisted order value

Definition. The conversion rate and average order value of visitors who talked with the agent. Formula. Orders placed after a conversation within a defined window, divided by conversations with purchase intent; for order value, revenue from those orders divided by their number. Pitfall. Attributing to the agent every sale that follows a conversation. Visitors who start a conversation may already be further along in their decision. The most reliable measure of the agent’s own effect is a comparison with a control group, to whom the agent is not offered during the test period. Failing that, speak of “assisted” sales rather than “generated” sales.

Revenue per conversation

Definition. Assisted revenue divided by the number of conversations. Formula. Assisted revenue over the period, divided by conversations over the same period. Pitfall. Reading it alone. An increase can come from an agent that pushes the sale at the expense of advice: pair it with the product return rate.

Cost per resolution

Definition. What a request that is truly resolved costs, not a conversation that is merely handled. Formula. Total cost of the setup (licenses, integration, supervision and review time, human handoffs), divided by the number of verified resolutions. Pitfall. Dividing by the number of conversations, or forgetting the human time spent on supervision. It is this indicator, not deflection, that lets you calculate the ROI of a conversational agent in a way that holds up in front of a finance department.

VIII

Summary table

IX

What this means for your organization

We propose building the dashboard in three stages over ninety days, to move from a single indicator to full steering without launching everything at once.

Days 1 to 30: establish the baseline. Choose a written definition of resolution for each major family of requests, and have it validated by the customer relations teams. Link conversations to the events in your tools (orders, tickets, appointments) to have the “completed action” signal. Set the recontact window and make sure you can match the same customer across channels. Start measuring recontact and time to resolution on your existing channels now, to have a point of comparison.

Days 31 to 60: put quality control in place. Launch the weekly audit on a random sample, with a written grid and two reviewers on part of the conversations to check that they judge in the same way. Qualify human handoffs on this same sample. Deploy the CES or CSAT survey while tracking its response rate. Open the list of uncovered intents and assign it to an owner.

Days 61 to 90: link quality and economics. Calculate the cost per resolution with all costs, human time included. If the agent takes part in the purchase journey, set up a control group to measure its own effect on conversion. Set a monthly review bringing together customer relations, e-commerce and finance, with a simple rule: no indicator is presented alone, each is read with its counterweight (resolution and recontact, revenue and return rate, deflection and qualified handoff).

The question asked in committee then becomes: how many customers did the agent really help, at what cost, and with what reliability? Your teams keep control over the definition of each indicator, which makes the figures credible.

X

Where QLOA stands

XI

Frequently asked questions

Which KPIs should you track for a customer service AI agent?

The foundation is the verified resolution rate, attested by a completed action, a customer confirmation or the absence of recontact. It is complemented by the recontact rate, CES or CSAT, time to resolution, the quality of human handoffs, the error rate audited on a sample and the cost per resolution. For an agent that takes part in sales, add assisted conversion, measured with a control group.

Why is the deflection rate misleading?

It counts as successful any conversation that was not handed off to a human. A customer who gives up, or who calls back by phone the next day, is therefore counted as a success. The harder it is to reach a human handoff, the more deflection rises, while the experience gets worse. It remains useful for sizing teams, but it does not measure resolution.

How do you check that a conversation with an AI agent is truly resolved?

Combine three signals. The completed action, recorded in your tools (return created, appointment confirmed). The customer’s explicit confirmation in the conversation. The absence of recontact for the same reason, across all channels, within a defined window: a Harvard Business Review article recommended in 2010 a period of seven to fourteen days to track repeat contacts.

Is CSAT enough to measure an AI agent?

No. CSAT suffers from non-response bias: customers who gave up rarely answer the questionnaire. In the study published by Harvard Business Review in 2010, 20% of satisfied customers said they intended to leave. The Customer Effort Score predicted loyalty better there. Always read satisfaction together with the response rate and the recontact rate.

How do you calculate the ROI of a conversational agent?

Start from the cost per resolution: total cost of the setup, supervision and handoff human time included, divided by the number of verified resolutions, and compare it with the cost of a resolution through your existing channels. For the commercial part, measure the agent’s own effect on conversion with a control group, rather than attributing to it every sale that follows a conversation.

XII

Sources

1. 1

   Gartner, “Gartner Survey Finds Only 14% of Customer Service Issues Are Fully Resolved in Self-Service”, August 19, 2024. gartner.com

2. 2

   Zendesk, “CX Trends 2026. Leading in the AI era: Building competitive edge with contextual intelligence”, November 2025 (surveys conducted in June 2025). cxtrends.zendesk.com

3. 3

   Gartner, “Gartner Predicts Agentic AI Will Autonomously Resolve 80% of Common Customer Service Issues Without Human Intervention by 2029”, March 5, 2025. gartner.com

4. 4

   Gartner, “Gartner Survey Finds 64% of Customers Would Prefer That Companies Didn’t Use AI for Customer Service”, July 9, 2024. gartner.com

5. 5

   Matthew Dixon, Karen Freeman, Nick Toman, “Stop Trying to Delight Your Customers”, Harvard Business Review, July–August 2010. hbr.org

6. 6

   British Columbia Civil Resolution Tribunal, Moffatt v. Air Canada, 2024 BCCRT 149, February 14, 2024. decisions.civilresolutionbc.ca