We're sunsetting PodQuest on 2025-07-28. Thank you for your support!

“Foom & Doom 2: Technical alignment is hard” by Steven Byrnes

2025/7/1

LessWrong (Curated & Popular)

AI Chapters

Chapters

Shownotes Transcript

No transcript made for this episode yet, you may request it for free.

“Foom & Doom 2: Technical alignment is hard” by Steven Byrnes

LessWrong (Curated & Popular)

Summary & Table of Contents

What Future AI Paradigm Shift Do You Expect?

Where Does Egregious Scheming Come From?

How Do AI Capabilities Emerge?

Why Is LLM Pretraining So Different from Brain Functioning?

Are LLMs Truly Imitating Human Behavior?

Why Is Egregious Scheming a Natural Outcome?

Why Are Current LLMs Not Scheming, but Future AI Will Be Different?

What's the 'Literal Genie' Problem in AI?

How Do Reward Functions Differ Between RLHF and Brain-like AGI?

What Are the Implications of Inner and Outer Misalignment?

What Are the Risks of Open-ended Autonomous Learning?

What Are the Problems with Amplified Oversight?

What Are the Downstream Impacts of Technical Alignment Being Hard?

Is Technical Alignment Really That Hard?

Can We Choose the Innate Drives of AI?

Why Am I More Bullish on Impure Consequentialism?

How Narrow Is the Target for Technical Alignment?

Conclusion and Takeaways

If Brain-like AGI Is So Dangerous, Should We Focus on LLMs Instead?

What Actions Should We Take to Address These Concerns?

Shownotes Transcript

“Foom & Doom 2: Technical alignment is hard” by Steven Byrnes 56:38 Share

LessWrong (Curated & Popular)

Summary & Table of Contents

What Future AI Paradigm Shift Do You Expect?

Where Does Egregious Scheming Come From?

How Do AI Capabilities Emerge?

Why Is LLM Pretraining So Different from Brain Functioning?

Are LLMs Truly Imitating Human Behavior?

Why Is Egregious Scheming a Natural Outcome?

Why Are Current LLMs Not Scheming, but Future AI Will Be Different?

What's the 'Literal Genie' Problem in AI?

How Do Reward Functions Differ Between RLHF and Brain-like AGI?

What Are the Implications of Inner and Outer Misalignment?

What Are the Risks of Open-ended Autonomous Learning?

What Are the Problems with Amplified Oversight?

What Are the Downstream Impacts of Technical Alignment Being Hard?

Is Technical Alignment Really That Hard?

Can We Choose the Innate Drives of AI?

Why Am I More Bullish on Impure Consequentialism?

How Narrow Is the Target for Technical Alignment?

Conclusion and Takeaways

If Brain-like AGI Is So Dangerous, Should We Focus on LLMs Instead?

What Actions Should We Take to Address These Concerns?

Shownotes Transcript

“Foom & Doom 2: Technical alignment is hard” by Steven Byrnes