Topic Detection: Four Ways to Get It Wrong

1. Classify into a list the business wrote

The most common starting point. Someone in CX writes down fifteen topics — billing, delivery, returns, account access, technical support — and every conversation gets sorted into one of them.

It's fast, it's cheap, and it produces a chart that goes in a deck.

It also has two failure modes that are the same failure wearing different clothes. New topics are invisible. A product launch, an outage, a pricing change, a regulation — whatever starts happening next month has no bucket, so it lands in "other" and nobody looks at "other." Dead topics persist. The category someone added in 2024 still sits on the dashboard at 0.3%, taking up space and quietly implying that the taxonomy is current.

The deeper problem is that the list encodes what the business expects customers to ask about. That's a map of the company's assumptions, not of the customers.

2. Reuse the bot's own intents

Tempting, because they're already there. The NLU model already classifies incoming messages into intents, so why not aggregate those?

Because intents are what the bot was built to recognise. They're the same assumption list as above, written by the same people, with one extra problem: everything the bot didn't anticipate collapses into fallback.

And fallback is where the interesting material lives. The requests nobody designed for, the phrasings nobody tested, the questions that only exist because something changed last Tuesday — all of it arrives as one undifferentiated bucket labelled "didn't understand."

Using intents as topics means your view of what customers ask is filtered through what you already decided they'd ask. It's the cleanest possible way to confirm what you already believe.

3. Cluster each time window

The sophisticated-looking approach. Embed every conversation, cluster the embeddings, generate a label for each cluster. No fixed taxonomy, no assumptions, topics emerge from the data.

This genuinely solves the discovery problem. It creates two new ones.

How many clusters? There's no correct answer, because the right granularity depends on who's asking. An executive wants eight topics. A conversation designer wants eighty — "billing" is useless to them, "double charge on subscription renewal" is actionable. Pick one number and you've made the output wrong for half your users.

The clusters don't survive contact with time. Run it on October and you get one set. Run it on November and you get a different set, with different boundaries and different labels. Sometimes a topic splits in two. Sometimes two merge. Sometimes the same concept comes back under a different name because a different conversation happened to anchor the cluster.

So what do you display when someone asks for a quarterly view? You can't add up three incompatible taxonomies. In practice teams end up re-running the whole quarter as one window — which means every time you extend the range, last month's numbers change. Nobody trusts a chart that rewrites its own history.

4. Ask a model, one conversation at a time

The approach that has quietly become the default, because it's three lines of code. For each conversation, ask an LLM what it's about. Collect the answers.

The output is excellent per conversation and useless in aggregate.

Across ten thousand conversations you get something like four thousand distinct labels. "Refund request." "Requesting a refund." "Customer wants money back." "Refund — damaged item." "Return and refund." Each one defensible. None of them countable.

You have described your data perfectly and measured nothing. And the obvious fix — ask the model to pick from a fixed list — walks you straight back into approach one.

The problem underneath all four

Each of these fails at a different point, but they fail for the same reason.

You need two things that pull against each other.

Stability, so the same conversation gets the same label this month and next, and so a number means the same thing across any window you choose.

Discovery, so a topic that didn't exist last quarter shows up without anyone having to add it.

Fixed taxonomies give you stability and no discovery. Per-window clustering gives you discovery and no stability. Per-conversation labelling gives you neither — it's maximally discoverable and maximally inconsistent.

The hard part isn't extracting topics. Any of the four approaches above extracts topics. The hard part is maintaining a topic set that stays comparable while still allowing new things into it.

Three more complications sit underneath that, and most tools ignore all three.

Conversations aren't about one thing. A real exchange starts with a delivery delay, becomes a refund request, and ends with a complaint about the previous agent. Force that into one label and you lose two-thirds of it. Allow multiple and your percentages stop summing to 100, which product teams hate.

The long tail is where the failures are. Most volume sits in a handful of topics. The problems worth fixing usually don't. Any method tuned to explain the bulk of your traffic is tuned to discard exactly the conversations you needed to see.

A topic isn't an outcome. Knowing 12% of conversations were about password resets tells you nothing about whether any of them worked. Topics tell you where to look. They don't tell you what you'll find when you get there.

What we do instead

Inquio builds and maintains a canonical topic set rather than regenerating one per window. Topics are extracted through a combination of models and algorithms, then reconciled against what already exists — so a concept that appeared in June is still the same concept in October, and a view across any date range adds up.

That gives you three things the approaches above can't:

Trends that mean something. A topic rising from 4% to 11% is a real signal, because both numbers were produced the same way.

New topics surface on their own. Something that didn't exist last quarter appears as a new topic rather than disappearing into "other" or quietly inflating an existing cluster.

Fading topics fade. Low-frequency topics that stop appearing drop out instead of sitting on the dashboard forever at 0.2%.

It isn't free of tradeoffs — nothing here is. Maintaining a canonical set means some judgement about when a new concept is genuinely new rather than a rephrasing of something you already track, and that boundary is never perfectly clean. But it's a tradeoff we'd rather make once, deliberately, than hand to every customer as a clustering parameter.

Why this is the first post in this series

Because topics are the layer everything else sits on.

Resolution accuracy is far more useful per topic than in aggregate — a bot can be excellent at opening hours and terrible at disputes, and the average tells you neither. The same is true of containment quality and of safety incidents, which cluster in specific subject areas rather than spreading evenly.

Get topics wrong and every metric downstream inherits the error. Which is a lot of weight for a question that sounds like it should take an afternoon.