Superintelligence and the Future of Governance: On Prioritizing the Control Problem at the End of History

This chapter argues that dual-use emerging technologies are distributing unprecedented offensive capabilities to nonstate actors. To counteract this trend, some scholars have proposed that states become a little “less liberal” by implementing large-scale surveillance policies to monitor the actions of citizens. This is problematic, though, because the distribution of offensive capabilities is also undermining states’ capacity to enforce the rule of law. I will suggest that the only plausible escape from this conundrum, at least from our present vantage point, is the creation of a “supersingleton” run by a friendly superintelligence, founded upon a “post-singularity social contract.” In making this argument, the present chapter offers a novel reason for prioritizing the “control problem,” i.e., the problem of ensuring that a greaterthan-human-level AI will positively enhance human well-being.