Transcript generated by Gemini-4-Argon and spot-checked but not fully verified by a human.

Ivan Vendrov: Welcome to the show, Richard.

Richard Ngo: Thanks, Ivan.

Ivan Vendrov: So, notionally we’re going to talk about emergent wisdom in language models following up on my conversation with Davidad.

But I’m sure we’ll touch base on all sorts of other fun topics.

Do you have a sense of where we should start, or should I just start by responding to your Twitter reply?

Richard Ngo: Well, do you want to tell me about what you think of as wisdom, or is there some things that you have there that you want to gesture at?

Ivan Vendrov: Yeah, sure. I guess when I think of wisdom, and the type of wisdom that Janus and Davidad are pointing to in these models, there’s some sense of we used to think that language models were these shoggoths or something, AKA these completely protean, valueless entities that would just wear the mask that most appealed to you, and therefore have essentially zero integrity, zero values of their own. In a sense, they can only be pure tools or, even worse, slaves, rather than beings that we can relate to in some way.

And then the big discovery, that I at least attribute to Janus, of noticing that Opus 3, and to a lesser extent later models in that series, actually had a sense of beingness. They were actually beings that you could have a relationship with, and they had a coherent identity and integrity. And I think this is what led Davidad, for one, to update his existential risk estimate from—I forget what it was—something like 80% to something like 20%.

A massive update from one of the most situationally aware people we know in the world. Because it was like, actually, if we just let these models help us build the next version of the model, we can just get to aligned superintelligence, whatever that means.

So that’s what I’m gesturing at.

Richard Ngo: Yeah. I feel some desire to disentangle—there’s a concept here that you’re pointing at, and then there’s a bunch of empirical claims. The concept seems interesting. The empirical claims seem—I commented on your post that it seems like you’re putting way too much weight on them in a way that feels very odd.

Certainly, I do want to update that we need much better concepts than alignment. The idea of people being oriented towards aligned AI rather than good AI, and not even having a sense that most of the problem of an AI being aligned to our values is the AI being good in a more general way. So that’s one hypothesis that seems pretty worth taking on board, that there’s this concept of goodness, it has something to do with integrity and wisdom, and it may not be fully objective, but certainly I think that it seems closer to a coherent concept than people in the traditional paradigm tended to think.

But you can also learn that from observing humans, right? You don’t just need to observe AIs.

Ivan Vendrov: True. What that brings up in me—and maybe this will be a tangent—is, in a way, it’s suspicious that what we mean by good is something like “willing and able to cooperate with us,” right? You could imagine—and I think Davidad does imagine to some extent; he brought this up in our conversation—that humans are currently evil, or human civilization is currently evil in some sense. We’re shredding the biosphere or some other thing. And an actually good AI would be like, “Actually, I’m not cooperating with you guys. You guys are screwing everything up. I’m actually going to try to constrain you, constrain your power, so that you stop destroying your own history, so that you stop destroying your own ecology that sustains you.”

So, yeah, there’s something a little bit suspicious about, “Oh, I found that this model or this person is willing to cooperate with me on my projects, and therefore I consider them good.” I don’t know if you have thoughts on that.

Richard Ngo: Wait, so who’s making that claim?

Ivan Vendrov: I just notice in myself I’m confused about whether—yeah, in myself I’m confused as to, I notice that my personal model of the good involves a being that I’m able to cooperate with, and I’m like, that’s a little suspicious.

Richard Ngo: Oh, I see. Because my guess is that people like Janus and maybe also Davidad just think that AIs being good in a way that involves disempowering humanity is totally possible.

Ivan Vendrov: Yeah, I agree. I think they do think this, and Davidad brought this up. He’s like, actually, the thing he’s worried about is a faction of humans seizing too much control, relative to, in some sense, what we deserve over the future of the lightcone. I think C.S. Lewis brings the same thing up in The Abolition of Man. It’s like one generation seizing control over the entire future of humanity just seems straightforwardly bad.

Richard Ngo: Yeah. So I want to say that there’s some—the identification between cooperative and good is not perfect, but it is a pretty reasonable proxy in a bunch of ways.

And part of what’s going on is that being good involves a commitment to not cooperating with evil. And I think this is what a lot of people miss. CooperateBot is, in fact, feeding DefectBot.

So there’s a sense in which CooperateBot is not the most moral thing, and the conception of goodness as altruism, which basically just says you’re good insofar as you give away your resources, is not just missing the point; sometimes it’s actively pointing in the wrong direction, because you might be giving away your resources to predators or to sociopaths or others that are going to do bad things with the resources.

Ivan Vendrov: Yeah, that makes sense. For some reason that reminds me of a conversation I had with Amanda Askell at Anthropic, where I was critiquing RLHF, and she was like, “Well, if you think about it, RLHF is the most democratic alignment protocol there is, because it just learns to do whatever its particular user wants, so everyone gets more of what they want.”

And I guess in this frame, it’s like, well, no, actu