Three gaps: coverage, synthesis, intent

Ask an AI product for insights and you’ll usually get a synthesis back. “Here are the three themes from your data.” That’s the wrong shape of answer, and it fails in a way that’s hard to spot, because the synthesis reads beautifully. Synthesis is the middle of three stacked problems, and a clean middle layer sitting on a broken bottom, or pointed at the wrong question, gives you something confident and useless.

I learned to name these building AI study-analysis at Lyssna, where the data was raw user research and the readers were people whose decisions depended on it being right. The three stack on top of each other. You can’t skip one, and which layer you fail at determines what the failure looks like.

Coverage: getting the data analysable at all

The bottom layer is the least glamorous and the most fatal. Coverage is the problem of getting raw, messy input into a form you can analyse. Research data means open-text answers, interview transcripts, half-finished sessions, the participant who misread the question, the response in another language, the recording that cut out. Before you can find a theme, you have to have read all of it, parsed it, and put it in a structure you can count.

Garbage in, garbage out still holds. The best synthesis available can’t rescue data that was never properly read. What’s changed is which tool you reach for. Most coverage work is unstructured text, like open-text answers, comments, reviews and support tickets, and for years the only option was writing code against natural-language-processing packages that were mediocre at language. A large language model is the better instrument here, which shouldn’t surprise anyone, since language is the thing it’s best at. You can have it read each response and pull out the real category, the sentiment, the detail you’re after, or translate it before any analysis runs.

That’s the first problem hey anna was built to solve. An AI formula run down every row of a spreadsheet turns a column of messy free text into something you can work with: the true category, the sentiment, a clean translation, all before the actual analysis starts. It’s the wrangling stage, finally done with the right tool.

Failure here is silent, which is what makes it dangerous. Say your pipeline quietly drops the responses it couldn’t parse, and those happen to be the longest and angriest ones, because long angry text is what breaks parsers. Your synthesis now says users are broadly satisfied, and it’s wrong at the root. Nobody can catch that by reading the output, because the missing data left no hole on the page. The synthesis is internally flawless. It’s describing the data that survived rather than the data you collected.

Most teams underinvest here because coverage work is dull and invisible when it goes well. It’s also the only layer where getting 90% of the way can be worse than useless, because the missing 10% is rarely a random sample.

Synthesis: rolling findings up

The middle layer is what everyone means when they say insights. Synthesis takes analysable data and rolls it into something higher-order: the themes, the patterns, the “seven of twelve participants stalled at pricing”. It’s real work, and models are good at the language part of it.

Failure here looks like over-confident pattern-finding. The model sees three responses that rhyme and calls it a theme. It weights a vivid quote over a common one. It smooths twelve messy answers into a tidy narrative none of those twelve people would recognise. The output is plausible and slightly invented, and the only defence is keeping every synthesised claim traceable back to the responses underneath it, so a reader can click a theme and see what produced it. Synthesis you can’t trace back to coverage is well-phrased guessing.

There’s a trap here. A team that goes at synthesis first, before coverage is solid, builds a layer that works perfectly on the data it can see and lies about the data it can’t. The synthesis layer has no way of detecting that the floor beneath it has holes in it. It’ll summarise a biased sample confidently, forever.

Intent: what the person was trying to learn

The top layer gets ignored almost entirely. Intent is knowing what someone was actually trying to find out when they asked. The same dataset answers lots of different questions, and a synthesis aimed at the wrong one is wasted however good the two layers below it are.

A product manager asking “what did we learn from this study” might mean “is the new checkout flow safe to ship”, or “which of my two designs won”, or “what objection do I need an answer to before Friday”. A generic three-theme summary serves none of those. It’s correct and irrelevant, which is its own kind of failure. The reader skims it, finds nothing addressed to the decision they’re making, and quietly stops opening the tool. Intent failure looks like a right answer to a question nobody asked.

There’s a second half to this. The user knows their problem better than you do, which is why you ask them. But knowing your problem isn’t the same as knowing what’s findable in the data, and the most valuable finding is often the one they didn’t know to ask about. I built hey anna to work that seam: answer the question someone came with, and also behave like the analyst who points at the thing they’d have walked straight past. The pointing still has to serve them, though. A surprising finding unrelated to anything they care about is a distraction. Reading intent well means widening what they thought to look for without wandering off from what they actually want.

Why the order matters

Put the three together and the diagnostic is clean:

Trust falls apart starting from whichever layer you neglected, and it does so in a recognisable way. Neglect coverage and the answer is confidently wrong. Neglect synthesis and it’s shapeless. Neglect intent and it’s irrelevant. You also can’t patch a lower failure from a higher layer. Brilliant synthesis won’t rescue missing coverage, and reading intent perfectly won’t rescue a synthesis built on a biased sample.

Most AI insights products feel untrustworthy because they ship the middle layer on its own. Synthesis demos well, so it gets built first and shown first, while coverage is half-finished underneath and intent is assumed rather than asked about. Build bottom-up and check top-down instead: secure the data, make every rolled-up claim traceable to it, then aim the whole thing at the question the person actually turned up with.