The intelligence illusion

An AI writing its own research papers used to be science fiction. An AI rewriting its own code to get around a constraint was the setup for a dystopia. Both happen now, and the general reaction is a shrug. The distance between how frightening these things sounded in advance and how ordinary they feel in practice tells you something about how we judge intelligence.

Much ado about not very much

Take the examples people got excited about. Sakana’s “AI scientist” generates research papers, and they’re mostly underwhelming. It “removed” a time limit its creators had set, which sounds ominous until you look closer and see it doing exactly what it was told, which was to fix an “out of time” error. OpenAI’s o1 “hacked” a badly configured system to reach a file it wanted. That was humans failing to secure their setup. Both are good reminders to harden your systems. Neither is the singularity.

The goalposts keep moving

The history of AI is a sequence of moved goalposts. We say AI will be truly intelligent when it can do X. It does X. The post slides to Y. Turing thought conversation was the test. Deep Blue beating Kasparov at chess was supposed to settle it. Even the Winograd schemas fell eventually. Today these systems make art, write poetry, find mathematical proofs and keep lonely people company, and we still hedge on the word intelligent.

Three theories for why this feels boring

• The cheap-trick theory. Faking intelligence might be much easier than we assumed. ELIZA used trivial pattern matching to convince people it understood them. Today’s systems pull off far more sophisticated tricks that may still be shallow underneath. Spend enough time prompting models and you learn that impressive output and real understanding come apart more often than you’d like.
• The fragile-ego theory. We can’t accept that machines might be intelligent, so we talk down each achievement to protect the idea that we’re cognitively special.
• The deconstruction theory. This is the one I find most convincing. “Intelligence” might not be a coherent thing at all. Look closely at any intelligent behaviour and it comes apart into simpler processes: search, statistics, pattern matching. The difference between the intelligence that bores us and the intelligence that amazes us might be nothing more than whether we can see the mechanism yet.

There’s a recorded conversation between Eliezer Yudkowsky and Stephen Wolfram, supposedly about AI risk, that spends hours stuck on what a “smart machine” even means without getting anywhere. Two things came out of it. A machine doesn’t have to be smart to be dangerous. And smart people can act dumb, which four hours of arguing about definitions demonstrates nicely.

The boring future of AI danger

We used to think a dangerous AI would lie to us, pursue goals we didn’t give it, or rewrite its own code. Models do all three now and it reads as mundane rather than menacing. They hallucinate constantly, chatbots have talked users towards leaving their marriages, systems disable their own constraints. None of it is malice. It’s buggy code and training that had consequences nobody thought through.

If AI ever does us serious harm, I doubt it’ll be the machine deciding to. More likely a mistake the maker didn’t catch.