Work done with James Faina, Evan Hubinger and Ethan Perez.

This is ongoing research on speculative threats, which might be of interest to researchers in the field of AI safety. We ask that readers treat this as a work in progress, rather than as a mature research paper.

To prevent highly capable and potentially misaligned LLMs from taking bad actions, we might want to monitor all of their outputs (Inan et al. 2023, Greenblatt 2024, Balesni 2024). How do we ensure that highly capable LLMs can’t disable the monitoring system?

Well-configured permissions and good cybersecurity may prevent highly capable LLMs from modifying inference code they are not supposed to modify, but highly capable LLMs might be used to improve the inference code, which opens up the risk of LLMs subtly disabling the monitor.