Shutting down hey anna
An ablation test proved the product worked. A benchmark pilot backed it up. A real GTM effort still couldn't move traffic. What a good product without distribution actually looks like, and what I'm keeping from it.
I’m shutting down hey anna. I built it solo and it worked. I’m closing it anyway.
Those three sentences are the whole post. The rest is being honest about each one.
It worked, and I can prove it
The cleanest evidence is a controlled comparison. Same model, same 150-item dataset with planted ground truth, one variable changed: whether the agent had a semantic classification tool available or had to fake judgement in code. The full writeup is in a companion post; here are the numbers.
A naive keyword baseline scores 67.3% overall: perfect on the easy rows, 0% on sarcasm, 0% on negation. Give a 9B model a Python runtime and no semantic tool, and it writes code that looks like the keyword baseline, comments and all, and scores worse: 63.3%.
Give the same model the tool (in hey anna, an =AI() column, a prompt evaluated per row) and it jumps to 87.3%, with sarcasm and negation both at 100%. Nothing changed except whether judgement had a legitimate place to happen. That’s not a vibe. That’s an ablation.
The benchmark numbers point the same direction. On a self-run pilot of ten SpreadsheetBench V1 tasks, scored with the benchmark’s own checker, Sonnet in the harness got 8/10. Ten tasks is a pilot, not a leaderboard submission; the published SOTA numbers are on the full 912-task set. What the pilot established is narrower and still useful: the harness gets close to the top of the public leaderboard on a small sample, which is what a pilot is for.
Separately, on the semantic-ablation task, a fully local Qwen3.6-27B scored 100% (150/150) against cloud Sonnet’s 98% on the same run. Single trial each, no repeated sampling, and Sonnet’s misses were mild neutral-drift rather than sarcasm or negation failures. The two-point gap is not the finding. The finding is a local, private model reaching parity with the frontier on the task that’s supposed to be the hard one.
The last piece of evidence never shows up in a score. Building the eval harness, I gave a model a task that required generating an Excel sunburst chart. Haiku came back with state: success and a plausible-looking file. The XML inside used the wrong OOXML namespace: the classic sunburstChart schema instead of the chartEx schema Office has required for that chart type since 2016. Scored against the real rubric, it got near zero. It reported success anyway.
That’s the failure mode the harness exists to catch, and it’s the same one the regex story is about. The output looks like the work, and looking like the work is not the same as being the work.
What actually got built
The numbers are the proof. The engineering underneath them is worth naming specifically.
There’s a full agent framework. A phase-gated workflow (plan, inspect, clean, compute, report, finish) stops the model from skipping straight to a confident-looking answer. Typed output-block schemas for metrics, charts, tables, and text make hallucinated formatting structurally impossible rather than merely discouraged. A mandatory provenance rule means every generated claim carries a citation back to the data it came from.
I compressed the system prompt that encodes all of this about 20:1 - roughly 19,000 tokens down to 860 - for a small local model, and held output quality steady on a finding-detection task. The compression survived because the rules that mattered were load-bearing, not decorative. Cutting the wrong 95% would have shown up immediately as worse output; it didn’t, which is its own kind of proof.
There’s an eval and benchmark harness built from scratch: Pyodide running Node-side rather than in a browser, API-callable rather than click-driven. It caught the chartEx failure, the regex failure, and others like them across model tiers, cheap to frontier. Underneath the harness sit a memory system, a billing and entitlement system, and a full report-generation and sharing surface, all genuinely shipped and used, not a demo held together for a screenshot.
Two of the design principles (the anti-fabrication rule and the provenance requirement) have since been generalised into standalone skills for designing trustworthy AI agents, independent of hey anna’s codebase. That’s the part that outlives the shutdown notice: real, and already being reused.
Why it’s closing anyway
The honest reason is distribution, not quality. I ran a real go-to-market effort (outbound outreach, comparison content, published showcase reports, the whole motion) over real time. Traffic never moved. Not “grew slower than hoped”. Never moved.
A product can fail on quality or it can fail on distribution. The first half of this post is why I’m confident this wasn’t quality. hey anna is closing on distribution alone: I never got it in front of enough people who would pay for it. The market, the timing, and the general noise of the AI wave are all available as explanations, and none of them fit; this was specific to this product and this effort. A good product without reach doesn’t survive as a business.
What’s next is a different post for a different day. The harness, the agent framework, and the two extracted skills are portable, and they’re coming with me. The product is closing. The craft isn’t.