Your Chatbot's Containment Rate Might Be Lying to You

Containment rate is one of the most common KPIs for chatbots and AI assistants. It measures the percentage of conversations that didn't end with a transfer to a human agent. On the surface, that sounds like a reasonable thing to optimize for: fewer transfers, better bot, right?

Not quite.

A chatbot can post a great containment rate while customers are quietly frustrated, giving up, or going off to search for the answer themselves. And a transfer to a human agent isn't automatically a failure — sometimes it's exactly what the customer needed.

That gap between "the conversation stayed inside the bot" and "the customer actually got what they came for" is why we think it's time to measure something better: Containment Quality (CQ).

What Is Containment Quality?

Containment Quality measures whether a chatbot resolved a user's request from start to finish, without an unnecessarytransfer to another channel or agent.

Notice the shift in emphasis: it's about resolution, not just whether the conversation technically stayed inside the chatbot. A conversation has high CQ when the user reaches their goal directly in the chat, or through a smooth handoff that finishes the job — not one that dumps them somewhere else to start over.

The important word here is unnecessary. Not every exit from the chatbot is a failure.

Not Every Exit Is a Failure

Customer journeys naturally span multiple channels, and that's fine — as long as the bot is actually helping.

Say a user asks: "Can you send me the privacy policy?" The chatbot links the right page, the customer opens it, finds what they need, and moves on. That's not a containment failure. The bot did its job — it got the customer to the answer.

Now compare that to a user asking: "What's my account IBAN?" and the chatbot replying, "Please log in to the mobile banking app."

Technically, that's an answer. Practically, it solved nothing. The customer still has to switch apps, log in again, and dig around for the number themselves. From their point of view, the task didn't get done — and that's a real Containment Quality problem.

It Comes Down to Effort

The real difference between these two examples is customer effort.

Good containment removes effort. Poor containment just relocates it — from the bot to the customer.

A chatbot that keeps pointing people to webpages, apps, PDFs, or call centers instead of solving the problem itself can still rack up a strong containment score, even as the actual experience gets worse. CQ cuts through that by asking one blunt question: did the assistant actually help the customer finish what they came to do?

Why the Traditional Metric Falls Short

Standard containment tracking usually sorts every conversation into one of two buckets: stayed in the bot, or got transferred. That binary hides a lot.

It can't tell the difference between:

• A chatbot confidently giving wrong information and never escalating

• A chatbot looping the same non-answer until the user gives up

• A chatbot punting to another channel when it could have just solved the problem

• A customer silently abandoning the chat out of frustration

All of these can look perfectly fine on a containment dashboard. None of them are fine. CQ catches these hidden failures because it's evaluating the outcome, not just the channel.

How You'd Measure It

Measuring CQ means understanding what the user actually wanted and checking whether they got it.

The common thread: always judge against what the customer actually set out to do.

Why This Matters

As conversational AI gets more capable, people expect it to complete tasks, not just answer questions. Teams that optimize purely for containment risk rewarding the wrong things — dodging transfers instead of solving problems, cutting agent load instead of cutting customer effort, chasing an operational number instead of an actual outcome.

Containment Quality realigns that incentive with what customers actually care about: getting their problem solved, quickly and without extra work.

Beyond Containment

CQ isn't the whole picture of chatbot performance, but it might be the most important piece. A bot that resolves requests on its own creates real value. A bot that just keeps the conversation inside its own window doesn't.

The question worth asking isn't "did the conversation stay in the bot?" It's "did the customer actually get what they came for?" That's the question Containment Quality is built to answer.