Language Agents Reduce the Risk of Existential Catastrophe

AI and Society:1-11 (forthcoming)
  Copy   BIBTEX


Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability.

Similar books and articles

How Much Should Governments Pay to Prevent Catastrophes? Longtermism's Limited Role.Carl Shulman & Elliott Thornley - forthcoming - In Jacob Barrett, Hilary Greaves & David Thorstad (eds.), Essays on Longtermism. Oxford University Press.
Existential Risks: Exploring a Robust Risk Reduction Strategy.Karim Jebari - 2015 - Science and Engineering Ethics 21 (3):541-554.
Agential Risks: A Comprehensive Introduction.Phil Torres - 2016 - Journal of Evolution and Technology 26 (2):31-47.
Existential risks: a philosophical analysis.Phil Torres - 2023 - Inquiry: An Interdisciplinary Journal of Philosophy 66 (4):614-639.


Added to PP

485 (#27,161)

6 months
485 (#1,087)

Historical graph of downloads
How can I increase my downloads?

Author Profiles

Simon Goldstein
Australian Catholic University
Cameron Domenico Kirk-Giannini
Rutgers University - Newark

Citations of this work

No citations found.

Add more citations

References found in this work

No references found.

Add more references