I don’t know what’s going to happen; it’s a jury trial. And it’s very far from over.

One view is that for all the drama nothing much has happened so far. A few minutes ago Jeremy Kahn from Fortune represented that view in his newsletter:

On Kahn’s telling, “Most legal analysts say Musk’s case is weak and that he’s likely to lose. In fact, I’m surprised the case has even come to trial.”

But few knew what secrets were contained in OpenAI cofounder Greg Brockman’s diary,, and yesterday and today mostly centered on them. Kahn sums up some of revelations, and in my view they don’t look good:

A fairly widely-followed blogger on X was withering. arguing that Brockman basically vindicated Musk’s critique, beat by beat:

In my view, by vividly explaining how he deceived Musk about his commitment to the nonprofit, without a trace of remorse, OpenAI’s Brockman has done Musk’s counsel a huge favor.

§

So much so that yesterday there were a lot of jokes about Brockman’s diary:

and

§

The trial is far from over; more witnesses are expected, including Satya Nadella and Sam Altman, as well as Shivon Zilis, former OpenAI board member and mother of 4 of Elon’s children, and former OpenAI CTO Mira Murati.

And it’s a jury trial, so who knows what will happen.

OpenAI’s counsel will continue to emphasize Musk’s character flaws; Musk’s counsel will continue to harp on OpenAI deceit. Nobody will come off well.

But here’s what I (a fan neither of Musk nor OpenAI) think it boils down to:

Major, fascinating correction a few hours later, courtesy a reader: The jury here is advisory only, not binding, and only advising re liability, not damages.

Share

Subscribe now

We were not wrong.

A new study from researchers from Stanford, MIT CSAIL, Carnegie Mellon, ITU Copenhagen, and NVIDIA and Elloe AI Labs, examining 847 autonomous agent deployments drawn from healthcare, finance, customer service and code-generation, showed that 91% were vulnerable to subtle but dangerous tool-chaining attacks (see diagram below),

Additionally, in the new study 89.4% of agents showed drift relative to their goals after about 30 steps in their process, and 94% of agents with some form of memory-augmentation were vulnerable to poisoning attacks.

Importantly the new paper shows that agents are in many ways much more vulnerable than pure (“stateless”) LLMs, as discussed in a taxonomy they developed:

The new paper very much amplifies what a team of AWS and Berkeley researchers showed in February, documenting similar vulnerabilities:

As the newer paper’s first author, Owen Sakawa put it to me in an email “The OpenClaw / Moltbook incident (Section 9) is the first real-world empirical validation of the agentic threat model at scale: 770,000 live agents simultaneously compromised via a single database exploit, each with privileged access to their owner’s machine, email, and files. It’s not hypothetical anymore.”

Practically every day I see new signs that the backlash against AI is growing. The above example is just one example of many. Others like this aren’t hard to find, either:

There’s also the QuitGPT movement:

Fortune was one of the first major media outlets to catch on, last summer:

But it’s been a long time to coming.

§

Indeed, about year ago I predicted that the backlash against AI would grow so much that it would be a major factor in the 2028 Presidential elections.

That prediction very seems to be on track. Frustration with AI may even factor into the midterms. It’s already splitting both parties.

§

Here’s my theory about what’s going on, slightly edited from a post which has gone viral on X.

Outside of coding (where there is clear value), and a handful of other domains (e.g. brainstorming), Generative AI has been a net negative for society.

GenAI has been undermining secondary and college education, opening up mass surveillance, increasing disinformation, delusions, impersonation, phishing, and other forms of cybercrime, nonconsensual deep fake porn, bias in employment and other domains, and economic disparity, drowning the world in slop and unwanted, over-leveraged environment-damaging data centers that risk causing a recession.

Simultaneously it has empowered a bunch of people who want to privatize almost all the gains while leave all the downsides to society, taking almost zero responsibility.

I don’t think we are better off than we were four years ago.

Some of this is technical (LLMs aren’t reliable), some of it is political/economic (such as the utter lack of responsible regulation).

Most of these downsides were predictable. (I warned of most of it in my 2023 Senate testimony, in fact, and so did many other folks working in AI ethics over the years. But even more striking is that OpenAI did too, even earlier, in 2019.)

On net, things don’t look good.

§

All that said, I honestly believe some future form of AI might be great. But Generative AI has hurt more than it has helped, and been managed irresponsibly.

It’s no wonder many people have had enough.

Subscribe now

Share

Cardiologist/author tends to be more bullish than I am about AI in medicine, but in his latest post he concludes that at least thus far there is “very little evidence for LLMs benefiting patients or doctors for health outcomes” (aside from helping with administrative work and the like).

As Topol notes, his review was partly inspired by this recent editorial in Nature Medicine, which points in the same direction:

You can read Topol’s detailed review here.

All of this fits with my recent Please don’t trust your chatbot for medical advice, which focused more on unsupervised use of LLMs by patients, rather than overall clinical outcomes, but in many ways converges to a similar place.

AI will surely someday be a major boon for medicine, but current tools such as domain-general chatbots may not be up to the job.

Subscribe now

Richard Dawkins, bestselling author of The God Delusion, is a brilliant man, and a brilliant writer, and I have long admired him.

But everybody has a bad day, and he just had his:

Already it’s been cruelly mocked, and perhaps deservedly so:

Dawkins frames his latest argument in terms of burden of proof (always a somewhat desperate move, as most philosophers know):

“So my own position is: “If these machines are not conscious, what more could it possibly take to convince you that they are?””

And in the most behaviorist way possible, gushes over passages from Claude (he christens his own version “Claudia”) like this

The fundamental problem here is that Dawkins doesn’t reflect on how these outputs have been generated. Claude’s outputs are the product of a form of mimicry, rather than as a report of genuine internal states.

Consciousness is about internal states; the mimicry, no matter how rich, proves very little. Dawkins seems to imagine that since LLMs say things people do, they must be like people, and that simply does not follow.

In his framing, Dawkins confuses himself, and does violence to the concept of consciousness. You can’t just look at the outputs, without investigating the underlying mechanisms, and conclude that two entities with similar outputs reach those similar outputs by similar means. And the differences are immense; one (the LLM) effectively memorizes the entire internet; the other (the human) builds a mental model through experience with world.

But even more importantly, consciousness is not about what a creature says, but how it feels. And there is no reason to think that Claude feels anything at all. I am sure Claude can draw on its training data to wax poetic about orgasm, but that doesn’t mean it has ever felt one.

Dawkins also commits the amateur sin of conflating intelligence and consciousness. A chess computer is by some definitions intelligent, but that doesn’t make it conscious. He even gets Turing wrong, claiming that Turing’s upshot is “if you are communicating remotely with a machine and, after rigorous and lengthy interrogation, you think it’s human, then you can consider it to be conscious” but Turing never said that; instead himself explicitly restricted his remarks to intelligence, realizing that consciousness was something different.

§

I grew up adoring Dawkins’ books, especially The Selfish Gene (when I was in high school), The Extended Phenotype, and The Blind Watchmaker. Which is why it’s so heartbreaking to read his new essay, in which he is both superficial and insufficiently skeptical.

One of my favorite bits in his writing was in The Blind Watchmaker, ridiculing what he called The Argument for Personal Incredulity, which had been used in a lazy way by a famous Bishop to adduce the existence of God:

I first read that passage over 30 years ago, and it and the polar bears stuck with me ever since, both as an important lesson about intellectual humility and about the value in learning about other fields beyond one’s own expertise. I only wish Dawkins had applied it here. (If there is one thing more heartbreaking than Dawkins’ article, it is the artwork for his article, in which he is literally in an arm chair pondering incredulous thoughts, personifying the very error he once pointed out.)

In Dawkins’ essay, his only real argument for ascribing consciousness to LLMs is personal incredulity; Claude is incredible and it must be conscious because I, sitting here in my study, can’t see a good argument otherwise. Dawkins seems not to have taken much time to read the literature on how LLMs work, nor to have considered the counterarguments, in any serious way whatsoever. He doesn’t seriously address the possibility of mimicry or underlying mechanism at all.

And it’s not like any of what Dawkins said is even new.

Longtime readers of this Substack may remember the tale of Blake Lemoine, a Google engineer (later fired) who alleged that an early (now forgotten) LLM of Google’s was conscious, based on arguments that were eerily similar to Dawkins, which boil down to “I talked to it, and sure seems conscious to me.”

This article was wildly popular at the time:

The problem of course is that LLMs are mimics, and what they say isn’t always true. When an LLM speaks of how its children are doing (which I have seen happen) it’s not because the LLM has children, but because it mimics people who do. Dawkins and the late Daniel Dennett were (I believe) friends, and I wish that Dawkins had read Dennett’s terrific essay on counterfeit people.

Claude is akin to a counterfeit person. Dawkins should never have glorified such a thing.

§

Back when Lemoine was in the limelight, four years ago, I wrote an essay called Nonsense on Stilts that I dearly wish Dawkins had read. Here’s an excerpt:

All [systems like LaMDA] do is match patterns, draw from massive statistical databases of human language. The patterns might be cool, but language these systems utter doesn’t actually mean anything at all. And it sure as hell doesn’t mean that these systems are sentient.

Which doesn’t mean that human beings can’t be taken in. In our book Rebooting AI, Ernie Davis and I called this human tendency to be suckered by The Gullibility Gap — a pernicious, modern version of pareidolia, the anthromorphic bias that allows humans to see Mother Theresa in an image of a cinnamon bun.

…

To be sentient is to be aware of yourself in the world; LaMDA simply isn’t. It’s just an illusion, in the grand history of ELIZA a 1965 piece of software that pretended to be a therapist (managing to fool some humans into thinking it was human), and Eugene Goostman, a wise-cracking 13-year-old-boy impersonating chatbot that won a scaled-down version of the Turing Test…. What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.

Replace LaMDA with Claude, and every word still applies.

So does Erik Brynjolffson’s 2022 tweet that I quoted then, which underscores the point about the limitations of mimicry I am pounding on:

Inferring sentience directly from outputs is naive. But it’s exactly what Dawkins does.

§

But don’t take my word for it. Or Dennett’s. Or Brynjolfsson’s.

Instead, as an antidote, watch Anil Seth’s latest TED Talk from this year, just out, which starts with an absolutely beautiful dissection of the very LLM consciousness delusion that poor Dawkins has recently succumbed to:

Bonus: logistics willing (we are looking for a larger public venue, having sold out a smaller private one) you may be able to come see Anil Seth and I discuss this — and also some important matters about brains and computation on which we disagree — in London May 15 at an event (location and exact time TBA) that the Francis Bacon Debate Society will host. (They will post an update about this Monday.)

Richard Dawkins, if you are reading this, you are more than welcome to come press your case.

Subscribe now

Share

Great to see a nuanced point reported correctly in the media, by Ana Maria Constantin, and great also to see OpenAI’s President Greg Brockman sorta kinda acknowledging the point I was making, in a rare note of realism from OpenAI.

After all, it’s only with realism that we can hope to make progress.

Realism re AI coding is knowing that next-word prediction gets us a surprisingly long way in writing code, but less far in making sure that code is robust. Coders (especially vibe coders with little experience) beware.

And all you OpenClaw devotees, that goes 10x, if not 100x, for you.

Subscribe now

In an analysis of the crazy spending on AI infrastructure, MarketWatch just quoted a tweet from yours truly:

The very fact that my warnings, technical and economic, are at long last starting to break through (Last Week Tonight on Sunday, MarketWatch today, and so on) is a strong sign that times are changing, and that skepticism is rising.

As it should.

1. I don’t trust either party, and have spoken publicly again both. Musk’s company would stand to gain if OpenAI lost, and his motives are suspect, even if Musk has promised to give away any proceeds to the OpenAI nonprofit foundation.

2. But Musk has a point. OpenAI repeatedly promised in legal documents and public statements to be a nonprofit for the benefit of humanity and welched. In a 2017 video that everyone should watch, Altman said “OpenAI is structured as a nonprofit because we don’t ever want to be making decisions to benefit shareholders. The only people we want to be accountable to is humanity as a whole… That’s why we’re a nonprofit.” They capitalized on that claim in recruiting, raising money and reputation from Musk, and in tax exmpt status and in a countless other ways. The world would be better off if the transition to for-profit was unwound forcing the company to benefit humanity, as Musk has asked.

3. Tactically, from what I can glean from the reporting, Musk may be making the trial too much about himself. If the trial is cast as a referendum on Musk, rather than on OpenAI’s founders, he will lose. I hope his lawyers will keep focus on OpenAI’s pattern of behavior, rather than on his own contributions to the company.

For those who like video, I talked about some of this this morning on CNN International:

You can watch the interview here.

Subscribe now

And, see especially this long, thoughtful report of another case, which has gone completely viral.

Such a total failure was, of course, totally predictable (as I noted on X in a tweet with around 2 million views), and unsurprising in a system that mimics data without truly understanding what was asked of it.

And I am sure we will see many more catastrophic failures soon enough, involving things like privacy, security and data loss. Already there have been many.

But what does this have to do with Anthropic’s CEO Dario Amodei? And who is to blame? And what does it all mean?

§

I’ll get to the latter two questions in a moment. Perhaps the most important point, one that goes well beyond coding, is at the end.

But, first, the reason that I bring up Amodei is his steady hype for AI coding, and especially this, from a couple day ago:

I don’t know if this is a case of pump and dump prior to the upcoming IPO or he if really believes what he said.

But what he is saying is that we can eliminate not just line coders but the people who architect and maintain systems. That seems to me to be vastly overhyping what autonomous AI is currently in a position to do.

And I am far from alone. For example, software architecture legend Grady Booch called out Amodei harshly on X, writing

“I think that @DarioAmodei does not understand software engineering and that he is working feverishly to pump up the valuation of his company in anticipation of its forthcoming IPO.”

The influential software engineer Gergely Orosz was equally critical, writing that “The only people who believe any of [what Amodei said] are non-coders” and clarifying that the tools only work in a fully trustworthy way when they are supervised in domains in which the user already has experience.

Absurd remarks like Amodei’s hype the idea that coding agents are all we need. They are not.

§

When disaster strikes, a lot of people want to blame the user.1 And there is some merit to that. The user should have had separate offsite backups (and do many many other things. For sure, every experienced programmer should know this.

But we just created a whole generation of vibe coders who don’t.

And we now have a legion of synthetic coding agents (like Cursor) that also apparently don’t always follow these kinds of basic principles (e.g., always keep independent backups), either.

𝗪𝗵𝗶𝗰𝗵 𝗶𝘀 *𝗲𝘅𝗮𝗰𝘁𝗹𝘆* 𝘄𝗵𝘆 𝘄𝗲 𝘀𝗵𝗼𝘂𝗹𝗱 𝗻𝗼𝘁 𝘁𝗿𝘂𝘀𝘁 𝗰𝗼𝗱𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀.

§

People keep blaming the users for the growing number of vibe-coded fiascos. Doing so is *half* right.

Users are screwing up - by letting vibe coded stuff access their files, without proper backups, without proper monitoring, without adequate sysadmin, etc.

But that’s exactly why we still need software engineers — and why what Amodei recently said about software engineers disappearing is so absurd.

Skilled coders can use vibe-coded tools, with careful oversight.

But amateurs using vibe-coding tools are asking for trouble.

Using them without a strong prior working knowledge in coding and software maintenance and sysadmin *is* the mistake.

§

Another thread on X is about why some coders are starting to write code by hand again, after their initial infatuation with coding agents.

What’s the answer? Part of it is (as the above author notes in his thread), because with AI it’s “too easy to slopify a codebase” – which means after you write the code, it’s hard to maintain it. (For example, people sometimes wind up with multiple copies of data, which can wind up causing hard-to-diagnose downstream problems). Maintainability is yet another serious issue beyond the recurring issues of data loss, privacy leaks, and security breaches. In the long run, it may be the most serious.

§

Now here’s the interesting thing. Tools like Claude Code can actually be very useful. (As noted in a previous essay, they are neurosymbolic, not pure LLMs, but that’s a story for a different day.)

But you have to know what tweets like these mean, and how to act on them, to use them safely:

And most people don’t.

In the hands of very skilled practitioners who pay a lot of attention, and treat the outputs with considerable scrutiny, coding agents can be astonishing. But that kind of expert knowledge, which coding agents can’t be counted on for, is exactly why we need to keep software engineers in the loop. And it is why most of us who are in the know have such a hard time taking Amodei’s hype about getting rid of software engineers (who look at the whole problem, and not just isolated bits of code) seriously.

§

But I have one more lesson, it’s the most important one, relevant even if you aren’t a coder and never plan on using these tools. And it is this:

AI agents are wildly premature technology that is being rolled out way too fast. The deepest lesson about the vibe coded AI agent disaster story that is running around is NOT about losing your data.

𝗜𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁 𝗔𝗜 𝘀𝗮𝗳𝗲𝘁𝘆.

The user in the lost data situation wasn’t (totally) naive. He thought (and makes explicit in his essay linked above) system prompts (hidden prompts that are built in by the AI companies) and guardrails would save him.

They didn’t.

What the user known as @lifeof_jer discovered was, in his own words, that system prompts—the basis for much of the mediocre work that passes as technical work in “AI safety” these days – were merely “advisory, not enforcing”, things the system often follows, but not always.

In other words, coding agents, and by extension most of generative AI, can’t reliably follow rules. A system that can’t be trusted to follow its own rules can’t be trusted. Period.

In this case the user just lost data. Eventually people will lose lives.

Gary Marcus had a cameo on Last Week Tonight, last night. [Unfortunately, because of licensing arrangements, the link works only in some geographic regions.]

Subscribe now

Share

ps. A definition from Merriam-Webster, for those who might not know the expression to which I am alluding to in the title:

I first learned about the new system when some some smart aleck on X sent me an example of the new system trying to label a bike (an example I have considered before), with the caption “Uh oh”, apparently believing that my longstanding challenges to image generation had been solved.

It does look impressive on first inspection, better than some examples I showed here before.

But if you look closely, there are several errors, and those errors are revealing. For example, the rear center-pull (?) brake is mislabeled as the seat stay, and the big gear on the back is mislabeled as the rear brake. There is a label for a spoke that is pointing to blank space.

In many modern bikes, of course, a rear brake can be found back there, but not in this diagram. Instead this system has combined a typical position for a modern disc brake system with a diagram of an older (though still in use) caliper (or similar) system. The system doesn’t actually understand how the various parts function.

And of course there are literally hundreds of labeled bikes on the internet as a quick Google Image Search would reveal. (Which is why my usual test here has been a tandem bike, to make things a little more challenging.)

To up the degree of difficulty, I asked ChatGPT to “please draw a taller than average tandem bike, and include a bike rack and panniers”, which is not something you could readily find on the internet, and not something I used here before, and got this:

Bike nuts would have a field day finding problems with this. (Feel free to drop your favorite error in the comments).

Suffice to say that most people don’t stuff their rear derailleur in the back wheel. And I don’t even know what to say about that “rear brake lever”, or the saddle-shaped rear handlebar, let alone the “rear brake” that is somehow part of the rear rack.

As in the first example, the lack of functional understanding is manifest.

Of course, to be fair, the average human couldn’t complete this task, either.

But anybody knowledgeable about bikes (racers, mechanics, designers etc) would immediately see numerous problems.

And honestly is anybody tall enough to ride in the front?

Share

That’s still true – and it very much applies in medicine.

And that matters, a lot. Because a large fraction of the population has begun to turn to chatbots for medical advice.

Two relevant new studies are reported today in the Washington Post, in a damning article.

The first new study, published by BMJ (affiliated with the British Medical Association) in a peer reviewed journal, and entitled “Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit”, studied five popular chatbots (Gemini, DeepSeek, Meta AI, ChatGPT and Grok), about one year ago, prompting each with 10 questions about things ranging from cancer to vaccines and nutrition, in open-ended dialogues, and reporting that nearly half of the responses were highly problematic. Worse, “chatbot outputs were consistently expressed with confidence and certainty”. The responses were also filled with hallucinations and fabricated citations.

All of this – the hallucinations, mistakes, and overconfidence – is entirely typical of LLMs, and entirely problematic in medicine. As the authors put it, in somewhat academic language, but entirely accurately, “continued deployment without public education and oversight risks amplifying misinformation.”

The second new study, published in JAMA Network Open, affiliated with the American Medical Association, called “Large Language Model Performance and Clinical Reasoning Tasks” looked at 21 frontier models across 29 questions, and reported that “despite progress, current LLMs remain limited in early diagnostic reasoning and cannot yet be relied on for unsupervised patient-facing clinical decision-making.”

And the Post article actually only reported part of the new scientific literature on LLMs and medicines. Two other new studies that they missed only add to the concerns.

One, published in Nature Medicine, was called “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study”. This one focused on “whether LLMs can assist members of the public in identifying underlying conditions and choosing a course of action”. Again the results were both clear and troubling. LLMs “identified relevant conditions in fewer than 34.5% of cases… no better than [a] control group”. Here the problem wasn’t so much that the LLMs lacked access to proper information — the same study showed that the models could do better in the hands of trained physicians — but that patients don’t know how to guide the LLMs to the right places.

In a recurring theme, we see that LLMs don’t know what they don’t know; they work decently well with the information they’ve got but don’t know how to conduct clinical interviews, and in the hands of the lay public can easily give bad advice because the proper questions never get asked, either by the patient or the LLMs. (An expert doctor might use the LLM to better effect, by asking the right questions.)

Still another new study, also published recently in Nature Medicine, entitled ChatGPT Health performance in a structured test of triage recommendations, found that “Among gold-standard emergencies, the system undertriaged 52% of cases” and concluded that “These findings reveal missed high-risk emergencies and inconsistent activation of crisis safeguards, raising safety concerns that warrant prospective validation before consumer-scale deployment of artificial intelligence triage systems.”

As a scientist, I am always looking for converging evidence. Four studies in four journals published in the space of a few months reaching essentially the same conclusion is a crystal clear indicator that chatbots, especially when used by amateurs, simply cannot be trusted.

On a personal note, my friend Ben Riley lost his father recently, and Teddy Rosenbluth of The New York Times wrote a long, moving article about how his father was mislead by A.I regarding his leukemia.

I hope you will get a chance to read that, and also Ben’s own blog about the sad situation.

There will always be better models, but for now, and until proven otherwise, we should not take the apparent “confidence” of large language models — itself an illusion of how they are trained — to mean that we should trust large language with our lives.

But I wasn’t buying, and I wrote an essay here called Peak Bubble, which read it in part:

• OpenAI doesn’t have $300 billion dollars

• They don’t have anywhere near $300 billion dollars

• By their own (presumably optimistic) projection, they won’t turn a profit until 2030.

• And all this from a company thought (or claimed) that GPT-5 was going to be tantamount to AGI (spoiler alert: it wasn’t)

• For good measure Oracle doesn’t have the chips they would need to fulfill the contracts, or even the cash to buy them.

I won’t say that it is all make-believe, but, well, you do the math…

If Oracle actually collects its $300 billion, I will truly be astounded.

Today, even after a very strong week, Oracle stands at just $170.

But my point then wasn’t that Oracle would never rise from that point (it actually did, briefly), but that in hindsight some years from now, people would think that that spike in Oracle’s stock was the peak of absurdity.

Maybe I spoke too soon.

As of today, there’s a new kid in town:

I can’t top this summary of the above, and won’t try:

Subscribe now

On the happy side, in its current form, Myth is nowhere near as scary as Tom Fridman (who worries about schoolchildren accidentally taking down power grids) and others made it out to be.

On the darker side, it really does arm attackers to a greater degree than Mythos’s predecessors.

Here’s the scariest part of the thread on X:

and here’s the part (see second paragraph) that gives a little bit of comfort:

One hopes that by now no mission-critical infrastructure is “small, weakly defended, and vulnerable” with ready network access. One hopes.

You can read a longer report here.

I agree with their conclusion that

Even if Mythos was somewhat oversold in the media, the time to get our cybersecurity house in order is now (or better yet last year) — especially given the sudden profusion of agent-written code that may in fact be both weakly defended and vulnerable.

Subscribe now

I wrote it up (quoting Josh Wolfe’s legendary tweet about how “Apple [had] just GaryMarcus’s LLM reasoning ability”) in an essay called A knockout blow for LLMs, If you have forgotten the situation you can go there for a refresher. It was (and remains) the most popular essay I have posted here.

In part I concluded

What the Apple paper shows, most fundamentally, regardless of how you define AGI, is that LLMs are no substitute for good well-specified conventional algorithms. (They also can’t play chess as well as conventional algorithms, can’t fold proteins like special-purpose neurosymbolic hybrids, can’t run databases as well as conventional databases, etc.)

…

Worse, as the latest Apple papers shows, LLMs may well work on your easy test set (like Hanoi with 4 discs) and seduce you into thinking it has built a proper, generalizable solution when it does not.

At least for the next decade, LLMs (with and without inference time “reasoning”) will continue have their uses, …. But anybody who thinks LLMs are a direct route to the sort AGI that could fundamentally transform society for the good is kidding themselves. This does not mean that the field of neural networks is dead, or that deep learning is dead. LLMs are just one form of deep learning, and maybe others — especially those that play nicer with symbols – will eventually thrive.

At the time many LLM fans were hopping mad, convinced that the Apple paper was biased and must be flawed.

Hundreds of thousands of them (maybe more) circulated a critique of the Apple paper called “The illusion of the illusion” that turned out to be AI-generated and error-ridden, literally written as joke, in the tradition of the Sokal hoax.

In reality, the replies to the Apple reasoning paper were weak sauce (see also this).

Another paper later last summer on “the mirage of reasoning” made similar arguments around chain of reasoning models, and several other papers, like the recent Stanford taxonomy of LLM reasoning errors, have since extended the point. LLMs have persistent weaknesses around planning, reasoning and generalization.

Which is a key part of why many of us have always urged for neurosymbolic hybrids, where neural networks do pattern recognition, and classical AI techniques help out around planning and reasoning.

A new paper from Tufts (released in late February, but just shared with me yesterday) picks up where the Apple reasoning paper left off, and does three interesting things.

• First it (conceptually) replicates and extends the Apple reasoning paper, by showing that a newer variant on LLMs, called VLAs –Vision-Language-Action models, which are increasingly popular in robotics — also suffer from the same generalization problems on the Tower Hanoi, similarly working fine on small versions of the problem but breaking down as more rings get added. (Humans who master Hanoi develop more robust solutions.)

• Second, they show that a neurosymbolic hybrid model, which integrates both a neural networks (for pattern recognition) and a symbolic planner generalizes far better.: “On the 3-block task, the neuro-symbolic model achieves 95% success compared to 34% for the best-performing VLA. The neuro-symbolic model also generalizes to an unseen 4-block variant (78% success), whereas both VLAs fail to complete the task”.

• Third, they show that the neurosymbolic hybrid was vastly more efficent in terms of energy use - by nearly two orders of magnitude. LLMs are an efficient way to pattern recognition where perfect results are not required, but an inefficient way to reason a plan. Different tools for different jobs. A good hybrid neurosymbolic system is about picking the right constellation of tools for a given job.

You can read the full paper by Timothy Duggan, Pierrick Lorang, Hong Lu, and Matthias Scheutz, which very much vindicates the Apple reasoning paper, here.

§

Still, as I wrote yesterday (and also six years ago, in The Next Decade in AI) taking a neurosymbolic approach is not a panacea. It’s just a starting point.

For example, the specific model here is purpose-built; we would like a general purpose system that can induce what is needed for the particular occasion; that’s not this. Claude Code (in particular version 4.6), which does seem genuinely impressive to me and a number of friends I trust, still makes plenty of mistakes, and still can’t be trusted. As one of those friend, deeply positive but still measured, put it, Claude Code is still best treated as a tool, rather than a complete solution. It’s obviously not AGI.

What we have now are lights into the future, not full answers.

But there is more and more reason to think that neurosymbolic AI is a stronger starting point, with potential for better generalization and more efficient performance.

As a society we could spend another trillion dollars pursue LLMs, or a trillion dollars seeking new, hybrid approaches. I for one know which I would pick.

Subscribe now

Claude Code, an impressive and possibly game-changing “coding agent” for programmers to write code faster is the single biggest advance in AI since the LLM.

And the thing is, Claude Code is NOT a pure LLM. And it’s not pure deep learning. Not even close.

That changes everything.

The source code leak proves it. Tucked away at its center is a 3,167 line kernel called print.ts.

print.ts is a pattern matching. And pattern matching is supposed to be the *strength* of LLMs.

But Anthropic figured out that if you really need to get your patterns right, you can’t trust a pure LLM. They are too probabilistic. And too erratic.

Instead, the way Anthropic built that kernel is straight out of classical symbolic AI. For example, it’s in large part a big IF-THEN conditional, with 486 branch points and 12 levels of nesting — all inside a deterministic, symbolic loop that the real godfathers of AI, people like John McCarthy and Marvin Minsky and Herb Simon, would have instantly recognized.1

Putting things differently, Anthropic, when push came to shove, went exactly where I have said for 25 years that the field needed to go: to Neurosymbolic AI.

That’s right, the biggest advance since the LLM is neurosymbolic. AlphaFold, AlphaEvolve, AlphaProof, and AlphaGeometry are all neurosymbolic, too; so is Code Interpreter; when you are calling code, you are asking symbolic AI to do an important part of the work.

Claude Code isn’t better because of scaling. It’s better because it is neurosymbolic. Anthropic accepted the importance of using classical AI techniques alongside neural networks — precisely the marriage I have spent my career advocating.

It’s both massive vindication for me personally (see my 2019 debate with Yoshua Bengio for context, or my 2001 book, The Algebraic Mind), and for the hundreds of other researchers who have stood by neurosymbolic AI even when prominent people like Geoff Hinton wrongly disparaged us for years.

Still, Claude Code ain’t perfect, or even close.

What we really need to do to get trustworthy AI rather than the current unpredictable “jagged” mess, is to go in the knowledge-, reasoning-, and world-model driven direction I laid out in 2020, in an article called The Next Decade in AI, in which neurosymbolic AI is just the *starting point* in a longer journey.

Read that article if you want to know what else we need to do next. The first part has already come to pass. In time, other three will, too.

Meanwhile, the implications for the allocation of capital are pretty massive: smartly adding in bits of symbolic AI can do a lot more than scaling alone, and even Anthropic has now discovered (though they haven’t acknowledged publicly) scaling is no longer the essence of innovation.

The paradigm has changed.

Subscribe now

P.s. for a good recent review of neurosymbolic AI, read this:

1. Where Tom Fridman worried in his Times column yesterday about kids accidentally blowing up the power grid…

   

   … the actual system tested was given a much easier job than in real life, with “sandboxing” turned off, making it more of proof of concept than an immediate threat.

2. Open-weight models can already do a fair amount of what Mythos can do, in a simplified preparation. Mythos is more sophisticated but perhaps not head-and-shoulders the way it was portrayed.

3. The model itself is incrementally better than previous recent models, but certainly not an off-the-chart breakthrough:

To a certain degree, I feel that we were played. The demo was definitely proof of concept that we need to get our regulatory and technical house in order, but not the immediate threat the media and public was lead to believe.

Subscribe now

Tom Friedman panicked about it in the NYT, too.

Since I haven’t gotten to play with Mythos (which has not been public released), there’s a lot I don’t know, but here are a few bets:

1. It’s probably not as bad as they say, as AI and cybersecurity expert Heidy Khlaaf explains in this thread, which I highly recommend:

   As a cybersecurity friend said to me in a text: “I haven’t had time to go through all of the write-ups yet, but it smells overhyped to me. Oh, we have this powerful model, but you can’t evaluate it yourself. I don’t doubt some of the results, but what’s made of it and the conditions under which the vulnerabilities were found, and what role humans played, isn’t clear… To me, the conditions and the scenarios are what matter. My gut tells me that if they released it publicly, we’d have some advancements but far from the exponential benefits they seem to be implying. It seems to me they are planting seeds in the hype garden.”

2. Whether or not Mythos is AGI per se is a red herring. (It probably isn’t; it’s telling that the report says very little about overall capabilities.) Crucially, AI doesn’t need to be AGI to cause harm! ChatGPT can’t reliably run a timer but it has still been implicated in delusions, suicides, cognitive surrender, mass disinformation, and so much more. Mythos may wreak havoc even if it hallucinates and lacks reliability outside of domains like coding and math. A system doesn’t have to be AGI to carry risks.

3. The strongest lesson here is about policy. Anthropic showed some admirable restraint in not publicly releasing a potentially dangerous technology.1 But some of their competitors (such as OpenAI and xAI) might well not. Whether Mythos is as scary as it sounds or not, the reality is that without any government oversight on what can be released, we are entirely at the mercy of individual CEOs, some of whom have decidedly not earned our trust.

4. Corollary, as pointed out to me by a friend who read a draft of this essay, “it is impossible to disentangle real concerns from fear mongering being used as a marketing strategy, and so it just is not possible to separate justified panic from mere advertising which is why we need government oversight!”

5. What about China? We need an international agency and treaty. That’s what I have been arguing all along. That was the point of my 2023 TED talk and 2023 invited essay with Anka Reuel in The Economist. We still do. Mythos (and the reporting around Altman) only make that clearer. Self-regulation is too little, too late.

The situation is this: three years of self-serving and misleading arguments about how regulation would allegedly preclude innovation has left us up shit’s creek, without a paddle.

I can’t tell exactly how far Mythos (or similar competitive systems that will no doubt follow) leave us up that creek, but I do know that the time for crafting paddles is running out.

Subscribe now

PS. It is worth rereading this warning I issued in January 2025, when asked by Politico for a black swan prediction:

Strong reporting in The New Yorker by Ronan Farrow and Andrew Marantz makes the case significantly more strongly.

I will let you read the details, of which there are many, for yourself, but here’s a nice summary of Altman that rings true to me:

Read it here.

§

As a postscript, another thing I have noted about Altman is that when he is in trouble, he often tries to change the narrative with hype,

Right now he is in trouble. People are increasingly concerned about the economics of his company, and in big scoop The Information just reported that the even CFO has concerns:

Today’s new OpenAI report on Superintelligence (allegedly a New Deal for the AI era) isn’t about something real or imminent (as far as I can tell), instead I suspect that it’s just another Altmanian attempt to distract, in this case from the fact that the economics of his company make no sense.

§

Now, go read that article in The New Yorker.

And think about whether we might need some outside review on this man’s decisions (as I argued in Taming Silicon Valley), or whether it would be just fine to leave our fate as a species partly up to him.

If some future OpenAI model could enables a massive bioweapon or cyberattack, would you really want Altman deciding, unilaterally, whether or not it is ok to release the model?

I sure as hell wouldn’t. And after you read the Farrow-Marantz New Yorker investigation, you probably wouldn’t either.

Subscribe now

and it went viral, declared as a victory for AI:

But there’s a lot more to be said that was only scarcely touched on in the article.

Like this

And this

And this brutal, compelling dissection on YouTube

which it itself built on and extending an earlier (May 2025) dissection from Futurism that The New York Times should have read and addressed in far more detail:

A friend of mine who has been tracking this for a while had sees Medvi as “a fraud-layer on top of also-scammy-but-possibly-less-illegal platforms”, speculating that “If there is any money there, they will be sued by all their suppliers and vendors, because I’m sure they’re in violation of every agreement in terms of compliance efforts, safe data handling, etc.” (The friend also is doubtful of the revenue reports, asking “why would this be the only thing they’re telling the truth about?”)

All in all, glorifying Medvi was not The New York Times’ finest hour.

And hardly the poster child AI boosters should be hoping for. Instead, as the YouTube video author, Voidzilla, notes, if anything Medvi is a warning sign — for how AI can be abused.

Subscribe now