Minds and Machines 22 (4):299-324 (2012)

Nick Bostrom
Oxford University
There is no strong reason to believe that human-level intelligence represents an upper limit of the capacity of artificial intelligence, should it be realized. This poses serious safety issues, since a superintelligent system would have great power to direct the future according to its possibly flawed motivation system. Solving this issue in general has proven to be considerably harder than expected. This paper looks at one particular approach, Oracle AI. An Oracle AI is an AI that does not act in the world except by answering questions. Even this narrow approach presents considerable challenges. In this paper, we analyse and critique various methods of controlling the AI. In general an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous.
Keywords Artificial intelligence  Superintelligence  Security  Risks  Motivational control  Capability control
Categories (categorize this paper)
DOI 10.1007/s11023-012-9282-2
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

PhilArchive copy

Upload a copy of this paper     Check publisher's policy     Papers currently archived: 71,379
Through your library

References found in this work BETA

Superintelligence: Paths, Dangers, Strategies.Nick Bostrom (ed.) - 2014 - Oxford University Press.
Theory of Games and Economic Behavior.John Von Neumann & Oskar Morgenstern - 1944 - Princeton, NJ, USA: Princeton University Press.
Judgement Under Uncertainty: Heuristics and Biases.Daniel Kahneman, Paul Slovic & Amos Tversky - 1985 - British Journal for the Philosophy of Science 36 (3):331-340.

View all 28 references / Add more references

Citations of this work BETA

Is It Time for Robot Rights? Moral Status in Artificial Entities.Vincent C. Müller - 2021 - Ethics and Information Technology 23 (3):579–587.

View all 15 citations / Add more citations

Similar books and articles

Trapped Inside the Box? Five Questions for Ben Fine.Michael A. Lebowitz - 2010 - Historical Materialism 18 (1):131-149.
The Meta-Newcomb Problem.Nick Bostrom - 2001 - Analysis 61 (4):309–310.
A General Black Box Theory.Mario Bunge - 1963 - Philosophy of Science 30 (4):346-358.


Added to PP index

Total views
260 ( #42,705 of 2,519,679 )

Recent downloads (6 months)
9 ( #79,144 of 2,519,679 )

How can I increase my downloads?


My notes