The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents [Book Review]

Minds and Machines 22 (2):71-85 (2012)
Authors
Nick Bostrom
Oxford University
Abstract
This paper discusses the relation between intelligence and motivation in artificial agents, developing and briefly arguing for two theses. The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. In combination, the two theses help us understand the possible range of behavior of superintelligent agents, and they point to some potential dangers in building such an agent
Keywords Superintelligence  Artificial intelligence  AI  Goal  Instrumental reason  Intelligent agent
Categories (categorize this paper)
DOI 10.1007/s11023-012-9281-3
Options
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

Our Archive


Upload a copy of this paper     Check publisher's policy     Papers currently archived: 33,755
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

On What Matters: Two-Volume Set.Derek Parfit - 2011 - Oxford University Press.
Reasons and Persons.Derek Parfit - 1984 - Oxford University Press.
The Singularity: A Philosophical Analysis.David J. Chalmers - 2010 - Journal of Consciousness Studies 17 (9-10):9 - 10.
The Humean Theory of Motivation.Michael Smith - 1987 - Mind 96 (381):36-61.

View all 8 references / Add more references

Citations of this work BETA

Add more citations

Similar books and articles

Instrumental Rationality.Ralph Wedgwood - 2011 - Oxford Studies in Metaethics 6:280-309.
The Varieties of Instrumental Rationality.Stephen Ellis - 2008 - Southern Journal of Philosophy 46 (2):199-220.
Instrumental Rationality.Markos Valaris - 2012 - European Journal of Philosophy 20 (4):443-462.
Intention, Belief, and Instrumental Rationality.Michael Bratman - 2009 - In David Sobel & Steven Wall (eds.), Reasons for Action. Cambridge University Press. pp. 13--36.
Evidence and Normativity: Reply to Leite.Thomas Kelly - 2007 - Philosophy and Phenomenological Research 75 (2):465–474.
Epistemic Rationality as Instrumental Rationality: A Critique.Thomas Kelly - 2003 - Philosophy and Phenomenological Research 66 (3):612–640.
Instrumental Desires, Instrumental Rationality.Edward Harcourt - 2004 - Aristotelian Society Supplementary Volume 78 (1):111–129.
Humean Motivation and Humean Rationality.Mark Van Roojen - 1995 - Philosophical Studies 79 (1):37-57.

Analytics

Added to PP index
2012-06-14

Total downloads
112 ( #52,593 of 2,263,119 )

Recent downloads (6 months)
10 ( #36,637 of 2,263,119 )

How can I increase my downloads?

Monthly downloads

My notes

Sign in to use this feature