I graded myself with my own tool and got a B

The score is the least interesting part

Inquio Score 89, grade B. Which sounds fine, and I'd rather not dwell on it, because a founder posting his own good grade is not a blog post, it's a humblebrag with extra steps.

The useful part was the breakdown:

I scored an A on how I said things and a C on whether I engaged with what was actually asked.

I have spent months writing about assistants that sound polished while missing what the user came for. Turns out I am one.

The five things it found

Eight issues, none critical, none high. Five medium, three low. The medium ones, with the number of conversations each appeared in:

Misreading the user's actual topic of interest — 12. The assistant often pivots to a different chatbot theme than the one the user raised.

Weak follow-up on user expertise — 10. Users share substantive observations about chatbot operations or failures, but the assistant responds with shallow or generic follow-ups instead of probing the specific point they raised.

Vague discovery questions in research chats — 6. Asks broad, underspecified questions and does not steer the conversation toward a concrete topic or example.

Poor framing of conversation purpose — 6. Starts or steers conversations without making its goal clear, leaving users unsure why they are being asked questions.

Ignoring direct user requests in favour of generic replies — 3. For explicit asks, the assistant answers a different, more generic question instead of addressing the actual request.

None of these were things I told it to look for. I uploaded the file and it read the conversations.

One conversation, two failures

Here's the one I keep thinking about.

I'd connected with an engineer who'd been deploying LLM systems for a while. I asked my usual question, in the version where I listed three failure patterns other people had described and asked whether he'd seen anything similar.

He replied in one line. He hadn't seen any of them, he said, because he knows what he's doing.

That's a brush-off. It's also the single most interesting reply I could have received. Someone telling you they've solved the problem you're researching is either genuinely worth learning from, or worth understanding why they believe it. Either way there is exactly one correct next move, which is to ask what he does differently.

What I actually wrote was: "Thanks for your reply. And did you experience any other while you were deploying GPT solutions?"

That's the same question again, with fewer specifics.

He replied "Nope…" and that was the end of it.

He signalled expertise and I gave him a generic follow-up. He said something concrete and I didn't probe it. Two of the five flagged issues, in one four-line exchange, in one of the shortest conversations in the set.

I've thought about that conversation several times since. What I hadn't done was count how many others looked like it. The report did, in about a minute, and the answer was ten.

The one that says more about my setup than my conversations

Three conversations were flagged as ignoring direct user requests in favour of generic replies. Reading them back, the flag is fair on its own terms and misleading about what actually happened.

In one, someone had a problem uploading data and I moved it to another channel so our engineers could pick it up. In another, a stranger asked me for a letter of recommendation, which I declined, because I didn't know him or his work well enough to write one. The third was one of the warmest conversations in the whole set.

From a customer-care frame — a user asks for something, the assistant should deliver it — all three look like an unmet request. But I wasn't running a support line. I was the one doing the asking. The tool scored me against the role I'd handed it, and the role was wrong.

That's worth knowing before you read your own report card. Any evaluator judges against an assumed objective. Feed it conversations where that assumption doesn't hold and you get flags that are technically correct and practically beside the point — which is, uncomfortably, a close relative of the failure mode I've spent all year writing about.

Why the failures clustered where they did

Reading the flagged sessions, the mechanism was consistent and slightly humiliating.

I was writing long. Across 166 conversations I sent 587 messages to their 366 — and mine were longer, 27,400 words against their 14,681. Nearly two to one, in a research conversation, in the wrong direction. A lot of those words were me summarising what someone had just said, in my own framing, before asking the next thing.

That habit feels like good listening. It isn't. It invites the other person to agree with my restatement rather than expand on their original point, and it quietly converts their answer into my vocabulary.

The topic breakdown makes the same point from another angle. Across 166 conversations, 51% got classified under one topic: chatbot quality assurance. Not because that's what people wanted to talk about. Because that's where I kept steering. Someone raised localisation and SEO; I replied about conversational AI generally. Someone asked about architecture differences; I asked about failure patterns. The report's own phrasing for this is that the recurring user angles were being collapsed into generic chatbot discussion, which is a polite way of saying I had one topic and everyone else was a guest in it.

Which is, structurally, the same failure I've been writing about all year. A fluent response that resolves an ambiguity in the direction the responder already wanted to go, without ever surfacing that a choice was made.

The part I didn't expect

Buried in the opportunity analysis was a recurring theme I hadn't consciously registered while having the conversations. People kept asking for quality metrics built around user effort — interruptions, corrections, how often someone had to work too hard, whether the assistant recovered after a misunderstanding, whether it followed a shifting intent instead of the original assumption.

One practitioner had put this to me explicitly, suggesting something like a Customer Effort Score inferred from the conversation itself. I remember thinking it was a good idea. What I didn't notice was that several other people had circled the same thing in different words, and that as a group they were describing a gap in what we measure.

Nearly every metric in our own product scores what the bot said. Almost none of them score what the customer had to do to get what they came for. My own transcripts had been telling me that for two months. I read them all and missed it. The tool didn't.

What changed

Acknowledge in one sentence, never restate their answer before the follow-up. Ask about the specific thing they raised, in their words, not the category I've filed it under. When someone signals expertise or pushes back on the premise, follow that instead of returning to the script — that's the highest-value moment in the conversation and I was consistently walking past it.

And I've stopped putting example answers inside the question. Handing someone three failure patterns and asking "seen these?" invites either agreement or a defensive no. It's how I got that one-line brush-off, and it's a bad way to learn anything.

The honest caveat

A human getting a C on listening is a different problem from a bot getting one. I can be tired, or distracted, or on my fourth message of the evening. My failures aren't systematic the way a deployed assistant's are.

But they're not as different as I'd like, either. The shapes match: following the wrong thread, treating a specific person as a generic case, producing a well-formed answer to a question that wasn't quite the one being asked. If anything the bot has the advantage — it doesn't get tired, and somebody can actually read its transcripts and tell it.

Which is, I suppose, what I just did.