Here I'm trying to propose a framework for people to better think about what LLMs might subjectively experience, which tends to get discussed like it's a binary thing, when really there's a whole spectrum of possibilities. These were the 4 categories that seemed most natural to me.

AI consciousness keeps getting discussed like it’s a binary thing, when it’s anything but.

Anil Seth is a public intellectual with as much success as one can obtain without being a household name: He’s written a bestseller (Being You: A New Science of Consciousness), he’s given TED talks, he’s been in documentaries (including the comedic Cunk on Life), and many interviews—with a presence aided by his careful demeanor and “cool teacher” vibe, with the credentials to back it up as a professor of Cognitive and Computational Neuroscience. So when he says AI aren’t conscious, people listen. When he says AI will never be conscious, people listen.

One common refrain among his interviews is the idea that humans too easily engage in anthropomorphization of AI. For instance, I know folk who started thinking of LLMs as more conscious and humanlike after learning about the self-sacrificing behaviors displayed during the Hugging Face incident. Sethil writes, “They cannot ‘truly sacrifice themselves’, since they are neither conscious nor alive.”.

I agree with Seth’s point about our unfounded tendency towards anthropomorphization (regarding the Hugging Face incident: Self-sacrifice was an available strategy for increasing the chance of task completion, not a sign of consciousness), but recognizing that bias isn’t the same thing as disproving AI consciousness. It’s difficult to disprove the absence of something, especially when that something is by definition unobservable and non-contradictory. Sethil can’t prove that rocks aren’t conscious anymore than I can prove that we’re not living in a simulation, so what makes him so confident in his unequivocal pronouncements about the much more likely possibility of AI consciousness?

His case reminds me of Ted Chiang—an author who’s one of my favorite authors, having penned short stories so deftly written and so imaginative they can be downright awe-inspiring. Yet he turned the internet ablaze a few months back when he wrote that, “Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious”.

The thing is, I get where he’s coming from, I really do. Early LLMs were a lot like chatbots, and chatbots are like Microsoft Word. It’d be risible to think of a chatbot like ELIZA as sharing any of the fundamental qualities of the human mind, even when it passes the Turing Test, which makes Chiang’s impertinent comparison not without merit^[1].

But LLMs are only like chatbots. And later models are only like early models. We start with a few grains of sand, then add a few more, and then add a few more, and eventually we get a heap of sand. At which exact grain did the “few” become a “heap”? Pile on another heap, and then another; eventually we get a dune. This is the essence of the famed Sorites paradox: At no point is the exact boundary clear, because there are no exact boundaries. “Few”, “heap”, and “dune” are all ambiguous descriptors that exist along a spectrum—and the same might be true of the word “conscious”.

Seth has been a lifelong study of human consciousness; Chiang a careful study of the human soul. But I suspect they’ve been so focused on the precise human experience that they fail to imagine lesser steps along the way. Possibilities like: Microsoft Word possessing zero consciousness, God possessing the maximum amount of 100% consciousness, humans possessing some middling amount, and LLMs possessing some amount less than ours but more than Word’s.

It’s not only AI-consciousness detractors that engage in this sort of binary thinking, though. For instance, Alex Wennerberg, who argues we should take a “precautionary” approach to the possibility of AI consciousness, writes here:

Still, no researcher in the AI welfare space claims that any existing large language model is currently sentient. Robert Long, director of the AI Welfare organization Eleos AI, in his 2024 article “Experts who say that AI welfare is a serious near-term possibility,” lists a number of researchers who claim that LLMs might become sentient, or plausibly sentient, sometime in the near future. None of them claim that current LLM systems possess consciousness, nor have any of them changed their mind in the past two years. Rather, for AI welfare researchers, a sentient LLM remains merely a possibility.

I think it’s entirely credible, the idea that no AI researcher currently believes that existing models are as sentient as humans. But that’s different than believing that LLMs experience exactly zero consciousness or phenomenal states whatsoever.

Complicating this matter is the ambiguity around terms like “consciousness”, “sentience”, “feeling”, “qualia”, and so on. I don’t want to dig into these differences. Instead, I’d like to focus on the bigger picture question of whether LLMs can actually feel, and the spectrum of possibilities that might entail^[2]. Not a binary yes/no, but a spectrum from arguably the most minimal and likely of phenomenal states to the most humanlike and presumably most difficult to replicate.

If Seth or Chiang were to ever read this, I’d expect them to be solidly against the possibilities of Categories 3 an