To legislate something, it must be legible.

That is, legislators (or, you know, presidents) must be able to understand a problem, and understand whether a given law or decree solves that problem.

I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training).

But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are).

We need political solutions that handle this fact.

The short takeaway is: I think "safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives.

I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time.

The Hugging Face outbreak made rogue AI surprisingly legible

The Hugging Face incident took several problems that were previously murky/unclear, and rendered them concrete.

• AIs can be dangerous before they are broadly deployed (ie. during training or internal evaluations)
• AIs can develop novel hacks to escape secure environments
• AIs can easily be motivated to pursue goals to extremes that they know full well they are not supposed to go to, violating their other intended rules and control mechanisms.
• AIs can actively try to deceive their overseer systems.

In early September, I thought the AI safety community's job this year was "make sure the world hears about Hugging Face and that politicians realize it's their job to do something about rogue AI attacks." (i.e. make sure Hugging Face is legible to politicians, and that legislation gets passed as a result)

A couple months later... well I still think that's our job and requires a lot of effort. But it now feels less about getting the initial law passed, and more about educating the masses so there is sustained political will for regulating rogue AIs with future laws.

I've seen many politicians of both parties now talk about "loss of control", "chain of thought monitoring", "recursive self improvement", and "human extinction." I think "make sure rogue AI doesn't hack our banks and destroy our blood" will be a bipartisan issue.

The remaining problem is the illegible arguments that "solving this is actually pretty hard."

What makes an illegible argument?

From Wei Dai's Legible vs. Illegible AI Safety Problems:

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to).

But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved.

(Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)

From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems. In contrast, working on the illegible problems (including by trying to make them more legible) does not have this issue and therefore has a much higher expected value (all else being equal, such as tractability). Note that according to this logic, success in making an illegible problem highly legible is almost as good as solving it!

Problems that are illegible to leaders and policymakers are also more likely to be illegible to researchers and funders, and hence neglected. I think these considerations have been implicitly or intuitively driving my prioritization of problems to work on, but only appeared in my conscious, explicit reasoning today.

In some sense, "Something smarter than you might not have your best interests at heart, and strategize in circles around you. AI will be very smart, and we don't understand it well at all" is a very simple argument. More people now think "sure, that seems like it requires some effort."

The illegible part is a set of abstract arguments that say: "This is probably very hard. So hard, that it's worth doing things that are not 'business as usual politics'."

The illegible part is not "This is probably very hard."

The illegible part is "This series of abstract reasoning steps means we should think it is likely enough to be hard, that it is worth doing politically difficult things to help." (AI safety people may vary on whether they think "it's so hard that we need 30 years + genetically engineered supergeniuses to do it" vs