I created these notes for use when I facilitate AI safety courses. I'm sharing these notes early because this issue is topical and I suspect it is better to make an imperfect version available early rather than a much more polished version available later.

Please note: some of this content was taken from Additional Course Context. I decided recently that the content on take-off speeds really belonged in the same document as the content on RSI, so I'll be releasing an new version of Additional Course Context at some point (to avoid spamming LW, it'll probably be Substack only; none of my work is paywalled). Some of the other content was taken from Notes on “We’re in Triage Mode for AI Safety”. This Miles' article has been removed from the governance course, so I was looking for a new home for this content. I wanted to include it in a document that was intended to be shared in all the courses I facilitate.

Take-off Speed from Take-off Speeds Rules Everything Around Me:

Whilst there isn’t a strict definition of fast vs. slow takeoff, possibly the most influential definition comes from Paul Christiano (2018):

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)

At risk of oversimplifying, Paul argues that we should prima facie expect a slow takeoff because someone will generally figure out how to produce a “slightly worse” version of AI before someone figures out how to build the better version (he claims this is supported by the historical record of technological development overall).

Yudkowsky disagrees (2021), criticising the intuition that “design precursors with 95% of the technology should at least have 10% of the impact. He writes: "An awful lot of my model of deployment lag is adoption lag and regulatory lag and bureaucratic sclerosis across companies and countries” and “My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips"

How should we characterise progress up until now?

Eliezer recently wrote:

If Hanson or Christiano or Karnofsky etc etc believed this was what a "slow takeoff" looked like, they failed to convey that to their reassured, cheerful audience.

Tom Davidson responded:

If you thought this was what fast takeoff looked like, you also failed to convey that.

For reference, Eliezer previously defined a fast as meaning, “a timescale of weeks or hours rather than years or decades”.

“Bumpy” intelligence explosions

Forethought defines an additional scenario. For example, imagine if AI dramatically accelerates algorithm progress for a while, but then this plateaus whilst we figure out how to make robots that can automate the construction of chip fabs.

(Diagram note: the types of intelligence explosion will be explained in the next section)

In contrast, Arvind Narayanan (of AI as a Normal Technology) argues any RSI we do achieve will be narrow (that is jagged): ie. progress on verifiable tasks will continue “for a few more years” or even accelerate whilst progress on unverifiable tasks more or less remains “frustratingly hard”.

Richard Ngo also expects jaggedness to persist, as opposed to everything “going vertical”. Why? Because of a poorly understood “divergence” between “measured capabilities” and “real-world impacts” (nonetheless he asserts “things are going to look much weirder than almost anyone expects”):

Getting into the weeds

Whilst this is arguably less relevant these days, take-off speed is distinct from timelines. Take-off speeds are about how fast an intelligence explosion occurs if/when it happens whilst timelines are about when a particular milestone is achieved. Both slow-takeoff/short-timelines and fast-takeoff/long-timelines are conceptually possible.

Types of “Intelligence Explosion”:

Forethought examines three k

I created these notes for use when I facilitate AI safety courses. I'm sharing these notes early because this issue is topical and I suspect it is better to make an imperfect version available early rather than a much more polished version available later.

Please note: some of this content was taken from Additional Course Context. I decided recently that the content on take-off speeds really belonged in the same document as the content on RSI, so I'll be releasing an new version of Additional Course Context at some point (to avoid spamming LW, it'll probably be Substack only; none of my work is paywalled). Some of the other content was taken from Notes on “We’re in Triage Mode for AI Safety”. This Miles' article has been removed from the governance course, so I was looking for a new home for this content. I wanted to include it in a document that was intended to be shared in all the courses I facilitate.

Take-off Speed from Take-off Speeds Rules Everything Around Me:

Whilst there isn’t a strict definition of fast vs. slow takeoff, possibly the most influential definition comes from Paul Christiano (2018):

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)

At risk of oversimplifying, Paul argues that we should prima facie expect a slow takeoff because someone will generally figure out how to produce a “slightly worse” version of AI before someone figures out how to build the better version (he claims this is supported by the historical record of technological development overall).

Yudkowsky disagrees (2021), criticising the intuition that “design precursors with 95% of the technology should at least have 10% of the impact. He writes: "An awful lot of my model of deployment lag is adoption lag and regulatory lag and bureaucratic sclerosis across companies and countries” and “My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips"

How should we characterise progress up until now?

Eliezer recently wrote:

If Hanson or Christiano or Karnofsky etc etc believed this was what a "slow takeoff" looked like, they failed to convey that to their reassured, cheerful audience.

Tom Davidson responded:

If you thought this was what fast takeoff looked like, you also failed to convey that.

For reference, Eliezer previously defined a fast as meaning, “a timescale of weeks or hours rather than years or decades”.

“Bumpy” intelligence explosions

Forethought defines an additional scenario. For example, imagine if AI dramatically accelerates algorithm progress for a while, but then this plateaus whilst we figure out how to make robots that can automate the construction of chip fabs.

(Diagram note: the types of intelligence explosion will be explained in the next section)

In contrast, Arvind Narayanan (of AI as a Normal Technology) argues any RSI we do achieve will be narrow (that is jagged): ie. progress on verifiable tasks will continue “for a few more years” or even accelerate whilst progress on unverifiable tasks more or less remains “frustratingly hard”.

Richard Ngo also expects jaggedness to persist, as opposed to everything “going vertical”. Why? Because of a poorly understood “divergence” between “measured capabilities” and “real-world impacts” (nonetheless he asserts “things are going to look much weirder than almost anyone expects”):

Getting into the weeds

Whilst this is arguably less relevant these days, take-off speed is distinct from timelines. Take-off speeds are about how fast an intelligence explosion occurs if/when it happens whilst timelines are about when a particular milestone is achieved. Both slow-takeoff/short-timelines and fast-takeoff/long-timelines are conceptually possible.

Types of “Intelligence Explosion”:

Forethought examines three k