Audio note: this article contains 172 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

This post was written during Alex Altair's agent foundations fellowship program, funded by LTFF. Thank you Alex Altair, Alfred Harwood, Daniel C for feedback and comments.

Introduction

The selection theorems agenda aims to prove statements of the following form: "agents selected under criteria <span>_X_</span> provably has property <span>_Y_</span>," where <span>_Y_</span> are things such as world models, general purpose search, modularity, etc. We're going to focus on world models.

But what is the intuition that makes us expect to be able to prove such things in the first place? Why expect world models?

Because: assuming the world is a Causal Bayesian Network with the agent's actions corresponding to the <span>_D_</span> (decision) node, if its actions can robustly control the [...]