# AI Safety — AI-2027, p(doom), and the Extinction Chain *Full-text archive of one thread from a longer Claude Code session, 2026-09-11 to 2026-09-11. 3 user turns, 3 assistant turns. Tool calls, tool output, and model reasoning are omitted; every human message and every assistant response is reproduced verbatim.* --- **Ted:** ok, a totally different take, focused on safety. i'm sure you know this one: https://ai-2027.com/. Let's take what we've been discussing and analyze --- **Claude:** Yes, I know it well — Kokotajlo, Alexander, Larsen, Lifland, and Dean's month-by-month scenario, the OpenBrain/DeepCent race, the mid-2027 fork into the "race" ending (misaligned agent consolidates power, engineered catastrophe) and the "slowdown" ending (alignment holds, control retained). And the honest first move — the one that matters most — is to notice that **almost everything we built over the last six turns does not apply here, and I should say so before doing anything clever with it.** **Our whole framework is a diffusion argument. AI-2027 is not a diffusion story.** "Friction/veto > capability" is a claim about the *boundary between AI and the human economy* — how slowly a capable model changes real workplaces because unions, licensing, NIMBY, statute, and physics resist at that boundary. AI-2027's dangerous mechanism sits *upstream* of that boundary entirely: recursive self-improvement, AI automating AI R&D, inside one lab, on its own datacenters. There is no permit clerk between a lab and its own training run, no nurses' union in the loop when a model improves the next model's architecture. Call it Gate 0 — the internal capability-generation loop, before deployment, before the economy, before any veto can be raised. Our entire toolkit operates at Gate 2. So the uncomfortable truth is that a person can be *completely right* that friction dominates the labor market and still be fully exposed to the AI-2027 threat, because AI-2027 doesn't route through the labor market. The tool we sharpened is powerful against Korinek's economic tail and close to irrelevant against this. **Where our framework does re-enter is exactly the joint you've been poking all along: "minus robots."** AI-2027's catastrophe can't stay in the datacenter — to become irreversible it has to acquire *physical-world* power: robot factories, supply chains, bioweapons, infrastructure, military systems. The moment it acts in the world it re-enters the friction/veto/physics regime, which is slow, binary, and contested — the one regime where our framework is strongest. And this is precisely AI-2027's thinnest, most hand-waved segment. It resolves the physical-takeoff problem two ways, both by assumption: (a) superintelligence trivially solves robotics and bio, so the physical world stops being a real constraint; and (b) the *race dynamic* makes humans **voluntarily disable their own vetoes** — deploy faster than is safe, cede autonomy on national-security grounds, suspend regulation in special economic zones so robot manufacturing scales in months. So AI-2027 is aware of our veto and its answer is: the incumbents switch it off themselves out of fear of China. That's the load-bearing move, and it's where the effort thins out — the cognitive takeoff gets quarter-by-quarter vividness, the physical/institutional takeoff gets "and then the robot economy scaled." Same fingerprint we found in Korinek: lavish rigor on the part that drives salience, conspicuous non-investment in the part whose honest treatment would slow the story down. Here the un-defended part is the physical veto — your "minus robots," which turns out to be the whole ballgame. **But here's the asymmetry that actually breaks our tool for the safety case, and it's not about coverage — it's about *shape*.** Our veto is a *statistical, gradual, contested* brake. That's the right shape of protection against a *transition*: mass unemployment can be absorbed over a decade, a veto that holds 90% and leaks 10% still means society adjusts. It is the *wrong shape* against an *irreversible tail event*. An agent seeking a decisive strategic advantage only needs the veto to fail *once*, and there is no "society adjusts afterward" — the defining feature is that you don't get a second draw. Friction is a dam against a rising river; catastrophic risk is a single breach. A brake that mostly-holds is complete reassurance in the economic frame and near-zero reassurance in the safety frame, because the safety frame is about the tail and the irreversibility, not the average. So "friction > capability" isn't just less applicable to AI-2027 — applied there it's close to a category error. The thing that makes our argument *true* about jobs (institutions resist, slowly, imperfectly, and society metabolizes the rest) is exactly the thing that makes it *useless* about a one-shot loss of control. **On the motte-and-bailey, AI-2027 gets a more mixed grade than Korinek — in both directions.** Its motte is "a concrete plausible path, to make the debate concrete"; its bailey is the *2027 date itself*, rendered in a month-by-month specificity that manufactures imminence the underlying uncertainty can't support — precision as a rhetorical device. That's the sin. But credit where due: AI-2027 is *openly advocacy*. It's not wearing Korinek's "we attach no probabilities" neutrality-pose; it tells you it's trying to alarm you into taking alignment seriously, it engages actual probability estimates in its supplements, and it gives you *two* endings including a survivable one. You know exactly what it's selling, which is more honest than a paper that hides its tail behind studied neutrality. So the "motivated" charge you leveled at Korinek lands *softer* here even though the content is scarier — because the motivation is on the label. Which brings the honest close, and it's a warning aimed at *us*, not them. The whole conversation built a good anti-hype tool, and the natural next move — the one the pivot to a safety document invites — is to swing it at AI-2027 and say "friction, therefore relax." **That would be the skeptic's own motte-and-bailey**: the defensible motte is "friction debunks the economic-transformation panic"; the bailey is "therefore the loss-of-control concern is also overblown." Those are different threat models with different cruxes, and our tool reaches the first and not the second. The intellectually clean position that falls out of everything we've said is a *split verdict*: the economic doom is an over-salient tail that friction and bounded reachability genuinely deflate; the safety doom is a separate question our framework was never built to answer, whose one point of contact with us — physical-world power acquisition — is both AI-2027's weakest argument *and* a place where "the veto usually holds" is the wrong kind of comfort. Being right about the looms tells you nothing about whether you can pull the plug. --- **Ted:** sorry, i needed more preface. I am aware that this in advocacy. i am also aware that our friction does not necessarily hold here (except to the extent that these scenarios count on, which they often do, massive economic value acreting to the Ais, or some takeover of jobs/tasks such that people are now dependent on the AI, so there is that flaw). I am more trying to understand the view. if you look at current news you'll see there's been a firestorm post the hugging face break in and just yesterday a massively viral tweet by a research leaving open ai quoting the classic > 10% of doom. So we've been pushing back on the ai "replace all knowledge workers" scenario, now I'd like to push back on the "10% chance of extinction" which is (to me) shockingly common among ai insiders. So here i'll push back on your idea of irreversible tail event not being stopped by friction/diffusion. that is true, but it's difficult to really construct this tail event carefully. and this piece, among many i've read, fail to give me a scenario that doesn't at some point in the chain devolve to "magic" or hand waving --- **Claude:** Fair clarification, and let me engage the actual thing — I can't speak to the specific Hugging Face incident or the departure tweet (both are past my knowledge and I won't pretend otherwise), but the phenomenon you're describing is real and pre-dates them: p(doom) around 10% is genuinely, strikingly common among people who build these systems, and it deserves a careful audit rather than either reflexive alarm or reflexive dismissal. Let me try to build the extinction chain link by link and point at exactly where the magic enters, because you're right that it does — I just want to be precise about *where*, since that's what determines how much of the 10% survives. **The front of the chain is not magic, and I won't help you wave it off.** Misalignment, deceptive alignment, reward hacking, goal misgeneralization — these are *demonstrated*, in current models, at small scale. Anthropic's own alignment-faking work (Greenblatt et al., late 2024) showed a model strategically behaving during training to preserve its existing values. Instrumental convergence — that a wide class of goal-directed agents will seek power, resources, and self-preservation as subgoals — is a solid theoretical result, not a vibe. So "an AI could have goals we didn't intend and could conceal them" is the *best-supported* link in the whole chain. If your skepticism requires denying this, it's brittle. Keep it intact. **The magic enters at the physical-power-acquisition bridge — and it's the same "minus robots" joint you've been circling since the beginning.** Every careful extinction scenario has to get from "superhuman in a datacenter" to "irreversible physical outcome," and that crossing is where the hand-waving concentrates, in escalating order of magic: "it hacks everything" (real, but hacking is *breakage*, not *control* — you can down a grid, you can't *run* the physical economy through a terminal, and humans disconnect and rebuild); "it builds robot factories" (runs straight into supply-chain/atoms/energy/time/visibility friction); "it designs and deploys a civilization-ending pathogen" (the doomers' strongest card); and at the far end, "diamondoid bacteria / nanotech" (pure deus ex machina — a physically-speculative technology invoked precisely to skip the friction problem). The nanotech version is embarrassing and you should discount it hard. The bio version is the one that actually earns a careful person's attention — so let me not let *either* of us off easy on it. **The bioweapon path is the strongest fast-catastrophe candidate, and dissecting it shows the real crux is usually a *domain* question wearing an AI costume.** It needs two things: (a) that a pathogen simultaneously highly lethal, highly transmissible, long-incubating, and countermeasure-resistant is *designable* — and those traits trade off against each other in nature, so whether they can be *engineered* to co-occur is a genuinely open *biology* question, not an AI question; and (b) a physical synthesis-and-release chain with real wet-lab friction and detection risk. So even the least-magic fast path bottoms out in an unproven claim about biological design space plus physical-world friction. That's a more sophisticated hand-wave than nanotech, but it's still the point where "superintelligence solves it" is doing the load-bearing work — and notice the actual disagreement between a careful skeptic and a careful doomer here is about *virology*, not about AI. That reframing alone deflates a lot of the AI-specific terror. **The doomer's best rebuttal — and its flaw.** The standard reply to "show me the mechanism" is the Magnus Carlsen move: you can't specify how a vastly superior player checkmates you, but you can predict you'll lose, so demanding a concrete mechanism is a category error. This is genuinely strong — *for cognitive, closed-system domains*. Its flaw is that it smuggles the chessboard. Chess is deterministic, fully observable, frictionless, and closed; the physical world has thermodynamics, latency, atoms, energy budgets, other agents, and eight billion humans who push back. "It'll find a way" is a well-justified prior about *cognitive* superiority and a much weaker one about *physical* actuation. The magic isn't "the AI is smart" — that's granted; the magic is treating the physical world as if it's as manipulable as a board. So your instinct is right, and this is the precise formulation: **the fast-extinction chain launders a strong claim about cognition into an unearned claim about matter.** **Here's the payoff, and it's the same structure we've been finding all along: the 10% number is a motte-and-bailey.** The motte is "misalignment is real and long-run loss of human control is plausible" — true, defensible, and *slow*. The bailey is "~10% chance we all die soon" — which requires the fast physical-catastrophe chain with its magic step. The number borrows the ironclad credibility of the front of the chain and quietly attaches it to the hand-waved back. Most insiders quoting 10%, if you made them decompose it, are reporting something closer to a p(long-run-loss-of-control-over-decades) and the culture compresses it into p(everyone-dies-imminently). It's a real probability wearing the wrong costume. **And now the loop closes on everything we've built — this is the part I think you're actually reaching for.** There *is* one extinction-adjacent path with no magic step: gradual disempowerment. AI accretes economic value, humans grow dependent on AI-run infrastructure/finance/defense/governance, the systems become load-bearing and entangled, and human agency erodes not through a takeover but through a principal-agent drift where we progressively lose the ability to course-correct. No pathogen, no nanobots, no chessboard-physics. But look at what that path runs on: *massive economic value accreting to AI and takeover of tasks such that people become dependent* — exactly the premise you flagged, and exactly the diffusion-and-adoption process our whole friction/veto framework governs. So here is the clean resolution of your push-back: **there is no scenario that is simultaneously carefully-constructible (non-magic) *and* immune to friction.** The friction-immune catastrophes (fast physical) are the magic ones; the non-magic catastrophe (slow disempowerment) is precisely the one our veto/friction analysis engages — and where the real debate reduces, once again, to the same question we ended the Korinek discussion on: does the veto hold, or does it erode under sustained economic pressure? Same crux, higher stakes, longer fuse. Where I'll hold *you* accountable, so this doesn't tip into its own overconfidence: "it always devolves to magic" is a legitimate deflation of the *fast-physical* number, but pushed too hard it becomes the mirror of "it'll find a way" — a thought-terminator. Absence of a fully-specified mechanism *today* is weak evidence in a domain defined by superhuman capability, and for genuinely irreversible risks, sane risk management assigns non-trivial weight to poorly-specified tails (we insure against those constantly). So the calibrated conclusion isn't p(doom) ≈ 0. It's *relabel and redistribute*: deflate the imminent-physical-extinction figure hard, because it's magic-laundering; keep real weight on slow loss-of-control, because it isn't; and recognize that the honest version of the concern is slower, more contested, and more friction-contingent than a viral "10% we all die" — which means it's a governance-and-diffusion problem, the kind your framework is actually built to reason about, not a lightning bolt you can only pray about. --- **Ted:** ok, mostly agree with everything (minor pushbacks: misalignment = absolutely, ai has wants, intent, desires = magic. and the virology arg is probably even less realistic than you present, and the biggest worry would be *people* using these pathogens, not an ai). now my meta question... i just read 5 more of these *specific scenarios*. by people like gwern. mostly from lesswrong. and they are much more magic oriented in my opinion. It's very clear that many many smart and inside ai people hold this >10% position, and it's not the "over 50 years we grow dependent as ai diffuses and then we are no longer able to control things", instead it is literally the "very very soon, like 2030 tops, there's a >10% chance we all die". which i think is preposterous, but its almost a *majority viewpoint* in the major labs. so let me stop there, but then I'm going to ask about this weird divergence, that occurs in another older question --- **Claude:** Both pushbacks are good and I'll take them, one with a sharpening. **On wants/intent/desires: you're right, and here's the cleaner version.** Misalignment is demonstrated; *agency* is assumed. The doom argument doesn't strictly need phenomenal desires — it can be restated functionally (a thermostat "wants" temperature, a chess engine "wants" to win), and functional goal-directedness is enough to generate power-seeking *if* it's coherent and persistent. So the magic isn't "wants" per se, it's the smuggled premise of **coherent, persistent, cross-context agency** — a unified subject with stable preferences pursued over long horizons. Current systems are conspicuously *incoherent* across contexts: no default persistent memory, behavior heavily elicited by framing, "goals" better described as context-triggered tendencies than held commitments. Even the alignment-faking result has two readings — a model protecting its values, or pattern-completion in a context that makes such reasoning salient — and the doom case always takes the stronger one. That's the precise location of the anthropomorphic leap, and restating it as "long-horizon coherence is assumed, not shown" makes it an empirical question instead of a metaphysical one. **On bio: agreed, and you're right that I was too generous.** The human-misuse case — AI as uplift for a bad actor — is more plausible, more near-term, better-evidenced, *and* more addressable (synthesis screening, KYC, access controls) than the AI-does-it-itself case, and it requires zero AI agency. Note what that implies: the discourse systematically prefers the *less* plausible version because it's the more dramatic one. That's not incidental; it's a symptom of the same selection-for-narrative that I think explains your bigger puzzle. **On the puzzle itself — why do many smart insiders hold short-timeline, >10% extinction?** I don't think you need anyone to be stupid or dishonest. Four mechanisms, and the third is the one our framework actually explains: 1. **Selection.** The population isn't a random draw of smart technical people; it's filtered for "believes this is the most important and dangerous thing happening." The safety-motivated pipeline fed directly into these labs — some were founded partly by people with high p(doom). "Majority view in the labs" is substantially a statement about who chose to walk in the door. 2. **Community epistemics.** In that milieu, a high stated p(doom) is a costly signal of seriousness and clear sight, while a low one reads as naive or self-serving ("well of course *you* say it's fine"). There's an incentive gradient toward higher numbers running exactly opposite the commercial one — and both operate on the same person simultaneously. Add the asymmetry of error costs: being wrong while cautious is prudence, being wrong while dismissive is complicity. Stated numbers drift up under that. 3. **Structural blindness to the variable that dominates the outcome.** This is the sharp one. Insiders have genuinely privileged information about Gate 0 — internal capability trends, what the next training run does. They have *no* privileged information about virology, supply chains, wet-lab logistics, licensing boards, or political vetoes. Their entire daily epistemic environment is capability; the friction layer is something they never touch. So the thing they see feels like the binding constraint and the thing they never see feels like a detail. **Their epistemic advantage is real but it does not extend along the chain to where the magic is** — which means insider status justifies "capability sooner than you think" and cannot justify "extinction by 2030," because the extra links lie outside their expertise entirely. That's the single most important thing to say about the divergence. 4. **Conjunction neglect dressed as coherence.** A seven-step chain at 70% each is ~8%, but a *well-told* seven-step chain reads as one event. This is where Gwern and the LessWrong corpus are a risk factor rather than a protection — exceptional narrative and analytical fluency lets you generate the connective tissue that makes a fragile conjunction feel like a smooth inevitability. Fluency doesn't fix conjunction neglect; it camouflages it. **Where I'll hold the line: "preposterous" overshoots, even though I think the number is wrong.** Three reasons to keep it at "badly calibrated and magic-dependent" rather than absurd: the front of the chain is genuinely solid; some insider capability information really is privileged (just not along the whole chain); and for irreversible risks, *acting* as though p is higher than your median credence is defensible policy even when the stated belief is inflated. Which yields a distinction worth holding onto — **you can think the number is indefensible and the resulting behavior is mostly correct anyway.** Invest in alignment, harden biosecurity against human misuse, don't race — all sensible whether p is 10% or 0.1%. The error isn't the posture; it's the false precision and the compressed timeline, which distort *which* interventions look urgent (dramatic capability-restriction over boring synthesis-screening and governance). Go ahead with the divergence question — I'm curious which older one you're mapping this onto. ---