In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart’s Law as “Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity”, and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart’s Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that (...) aims to be informative for evaluating specific arguments for Extinction-level Goodhart’s Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods. (shrink)

We describe a significant practical consequence of taking anthropic biases into account in deriving predictions for rare stochastic catastrophic events. The risks associated with catastrophes such as asteroidal/cometary impacts, supervolcanic episodes, and explosions of supernovae/gamma-ray bursts are based on their observed frequencies. As a result, the frequencies of catastrophes that destroy or are otherwise incompatible with the existence of observers are systematically underestimated. We describe the consequences of the anthropic bias for estimation of catastrophic risks, and suggest some directions for (...) future work. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future (...) people. Lastly, we suggest that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

In this century a number of events could extinguish humanity. The probability of these events may be very low, but the expected value of preventing them could be high, as it represents the value of all future human lives. We review the challenges to studying human extinction risks and, by way of example, estimate the cost effectiveness of preventing extinction-level asteroid impacts.

If it survives for a little longer, the human race will probably start to spread across its galaxy. Germ warfare, though, or environmental collapse or many another factor might shortly drive humans to extinction. Are they likely to avoid it? Well, suppose they spread across the galaxy. Of all humans who would ever have been born, maybe only one in a hundred thousand would have lived as early as you. If, in contrast, humans soon became extinct then because of (...) the population explosion you would have been 'fairly ordinary'. Roughly ten per cent of all humans would have been your contemporaries. Now (as the cosmologist Brandon Carter saw to his dismay) a scientific principle tells us not to treat observations as highly extraordinary when they could easily be fairly ordinary. How to apply the principle is controversial, yet it seems we can safely conclude that humanity's chances of galactic colonization cannot be high. Still, we should work to make them as high as possible, resisting those philosophers who argue that human extinction would be no tragedy. (shrink)

“Existential risks” are risks that threaten the destruction of humanity’s long-term potential: risks of nuclear wars, pandemics, supervolcano eruptions, and so on. On standard utilitarianism, it seems, the reduction of such risks should be a key global priority today. Many effective altruists agree with this verdict. But how should the importance of these risks be assessed on a Christian moral theory? In this paper, I begin to answer this question – taking Thomas Aquinas as a reference, and the risks of (...) anthropogenic extinction as an example. I’ll suggest that on a Thomist framework, driving ourselves extinct would constitute a complete failure to fulfil our God-given nature, a radical form of hubris, and a loss of cosmological significance. So the reduction of such risks seems vital for Thomists, or Christians more broadly. Indeed, I end with the thought that these considerations are not parochially Christian, or indeed specifically religious: everyone with a similar view of mankind – as immensely valuable yet fallen and fragile – should agree that existential risks are a pressing moral concern. (shrink)

Longtermism is the view that the most urgent global priorities, and those to which we should devote the largest portion of our resources, are those that focus on (i) ensuring a long future for humanity, and perhaps sentient or intelligent life more generally, and (ii) improving the quality of the lives that inhabit that long future. While it is by no means the only one, the argument most commonly given for this conclusion is that these interventions have greater expected goodness (...) per unit of resource devoted to them than each of the other available interventions, including those that focus on the health and well-being of the current population. In this paper, I argue that, even if we grant the consequentialist ethics upon which this argument depends, and even if we grant one of the axiologies that are typically paired with that ethics to give the argument, we are not morally required to choose an option that maximises expected utility; indeed, we might not even be permitted to do so. Instead, I will argue, if the argument's consequentialism is correct, we should choose using a decision theory that is sensitive to risk, and allows us to give greater weight to worse-case outcomes than expected utility theory does. And, I will show, such decision theories do not always recommend longtermist interventions. Indeed, sometimes, they recommend exactly the opposite: sometimes, they recommend hastening human extinction. Many, though not all, will take this as a reductio of the consequentialism or the axiology of the argument. I remain agnostic on the conclusion we should draw. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk avoidant risk function. This, in turn, has been claimed to require the use of a risk avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the (...) significant risks associated with continued human survival. I raise objections to the claim that respect for others’ risk attitudes requires risk avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others’ risk attitudes in fixed population cases. GPI Working Paper No . 20-2022. (shrink)

This review provides an overview of the ethics of extinctions with a focus on the Western analytical environmental ethics literature. It thereby gives special attention to the possible philosophical grounds for Michael Soulé’s assertion that the untimely ‘extinction of populations and species is bad’. Illustrating such debates in environmental ethics, the guiding question for this review concerns why – or when – anthropogenic extinctions are bad or wrong, which also includes the question of when that might not be the (...) case (i.e. which extinctions are even desirable). After providing an explanation of the disciplinary perspective taken (section “Introduction”), the concept of extinction and its history within that literature are introduced (section “Understanding extinction”). Then, in section “Why (or when) might anthropogenic extinctions be morally problematic?”, different reasons for why anthropogenic extinctions might be morally problematic are presented based on the loss of species’ value, harm to nonhuman individuals, the loss of valuable biological variety and duties to future generations. This section concludes by also considering cases where anthropogenic extinctions might be justified. Section “How to respond to extinctions?” then addresses a selection of topics concerning risks and de-extinction technologies. Finally, the section on “Extinction studies” introduces other viewpoints on the ethics of extinction from the extinction studies literature, followed by the “Conclusion”. (shrink)

In a laboratory on a university campus in Santa Cruz, California, Ben Novak is doing everything he can to bring Ectopistes migratorius back from the dead. Using techniques now available in genome reading and gene synthesis, he and paleogenomicist Beth Shapiro hope that, by 2032, a flock of passenger pigeons ten thousand or more strong will have resumed an ecologically significant role in the mast forests of the Eastern United States. Novak knows—and the International Union for Conservation of Nature (IUCN) (...) affirms—that the challenges involved in making de‐extinction work are far from solely genetic. Novak is doing an ecological risk assessment of passenger pigeon reintroduction to figure out whether flocks of the resurrected species would pose any special hazards to ecosystems. Ecological harm is one of several worries attending the prospect of de‐extinction. Among other concerns are the possible harm that individuals born through this process might suffer and the possible introduction of disease vectors. But I want to step back from these immediate questions and think about some conceptual ones that operate in the background. Technology can be a grand shaper of cultural norms and expectations, and de‐extinction should be looked at in relation to a number of emerging technologies. This paper will examine the degree to which de‐extinction is part of a more widespread restructuring of ethical relationships to the surrounding world that are under way at the hands of emerging technologies. This restructuring is part of an ongoing shift in how to think about conservation in the new epoch of the Anthropocene. (shrink)

Are we in imminent danger of extinction? Yes, we probably are, argues John Leslie in his chilling account of the dangers facing the human race as we approach the second millenium. The End of the World is a sobering assessment of the many disasters that scientists have predicted and speculated on as leading to apocalypse. In the first comprehensive survey, potential catastrophes - ranging from deadly diseases to high-energy physics experiments - are explored to help us understand the risks. (...) One of the greatest threats facing humankind, however, is the insurmountable fact that we are a relatively young species, a risk which is at the heart of the 'Doomsday Argument'. This argument, if correct, makes the dangers we face more serious than we could have ever imagined. This more than anything makes the arrogance and ignorance of politicians, and indeed philosophers, so disturbing as they continue to ignore the manifest dangers facing future generations. (shrink)

A small but growing number of studies have aimed to understand, assess and reduce existential risks, or risks that threaten the continued existence of mankind. However, most attention has been focused on known and tangible risks. This paper proposes a heuristic for reducing the risk of black swan extinction events. These events are, as the name suggests, stochastic and unforeseen when they happen. Decision theory based on a fixed model of possible outcomes cannot properly deal with this kind (...) of event. Neither can probabilistic risk analysis. This paper will argue that the approach that is referred to as engineering safety could be applied to reducing the risk from black swan extinction events. It will also propose a conceptual sketch of how such a strategy may be implemented: isolated, self-sufficient, and continuously manned underground refuges. Some characteristics of such refuges are also described, in particular the psychosocial aspects. Furthermore, it is argued that this implementation of the engineering safety strategy safety barriers would be effective and plausible and could reduce the risk of an extinction event in a wide range of possible scenarios. Considering the staggering opportunity cost of an existential catastrophe, such strategies ought to be explored more vigorously. (shrink)

Many people assume that the extinction of humanity would be a bad thing. This article scrutinizes this apparent badness and demonstrates that on most plausible consequentialist frameworks, the extinction of humanity is not necessarily bad. The best accounts of the badness of the extinction of humanity focus on the loss of potential utility, but this loss can be offset if it is the result of sufficiently large gains by the present generation. Plausible means of calculating the goodness (...) of outcomes accordingly suggest hastening extinction even in some circumstances where the alternative is a long period of human existence at a high level. (shrink)

This paper examines and analyzes five definitions of ‘existential risk.’ It tentatively adopts a pluralistic approach according to which the definition that scholars employ should depend upon the particular context of use. More specifically, the notion that existential risks are ‘risks of human extinction or civilizational collapse’ is best when communicating with the public, whereas equating existential risks with a ‘significant loss of expected value’ may be the most effective definition for establishing existential risk studies as a (...) legitimate field of scientific and philosophical inquiry. In making these arguments, the present paper hopes to provide a modicum of clarity to foundational issues relating to the central concept of arguably the most important discussion of our times. (shrink)

Most current techniques to deal with invasive species are ineffective or have highly damaging side effects. To this end suppression-drives based on clustered regularly inter-spaced short palindromic repeats (CRISPR/Cas9) have been touted as a potential silver bullet for the problem, allowing for a highly focused, humane and cost-effective means of removing a target species from an environment. Suppression-drives come with serious risks, however, such that the precautionary principle seems to warrant us not deploying this technology. The focus of this paper (...) is on one such risk – the danger of a suppression-drive escaping containment and wiping out the target species globally. Here, I argue that in most cases this risk is significant enough to warrant not using a gene-drive. In some cases, however, we can bypass the precautionary principle by using an approach that hinges on what I term the ‘Worst-Case Clause’. This clause, in turn, provides us with a litmus test that can be fruitfully used to determine what species are viable targets for suppression-drives in the wild. Using this metric in concert with other considerations, I suggest that only three species are currently possible viable targets – the European rabbit, ship rat and Caribbean Tree Frog. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk avoidant risk function. This, in turn, has been claimed to require the use of a risk avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the (...) significant risks associated with continued human survival. I raise objections to the claim that respect for others’ risk attitudes requires risk avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others’ risk attitudes in fixed population cases. GPI Working Paper No. 20-2022. (shrink)

We explore the science of mass extinction, link it to industrial civilization, use the concept of the banality of evil to explain the ethical situation, and then explain the various ways in which mass extinction poses further ethical problems within that situation, especially of environmental justice and the loss of value. Overall, humankind risks a profound failure of autonomy, perhaps our greatest achievement. For those who want to take action, we recommend the project of anthroponomy and large-unit/deep-branching conservation.

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure (...) that the prospect of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

I have been involved with the current interest in de‐extinction since early 2012, nearly its beginning. I have given a lot of thought to the potential risks and benefits of de‐extinction. But only recently, after deep immersion in discussions around CRISPR‐Cas9, the hottest new tool in bioscience since polymerase chain reaction, have I thought about a more fundamental question: how, if at all, is de‐extinction special? Are “revived species” just another kind of genetically modified organism, raising essentially (...) the same general concerns? I answer, for the most part, yes. De‐extinction is not (very) special. New biotechnologies are giving humans even more power to change the biosphere but more directly, more quickly, and more utterly than ever before. De‐extinction is just one possible, and probably small, use of those technologies. Our attention, for the most part, should be on the bigger issues of regulating this power, rather than focusing specifically on their application to de‐extinction. (shrink)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the prospects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a “posthuman” society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could cause our extinction or destroy the potential of Earth-originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

We outline an argument favoring voluntary moral bioenhancement as a response to existential risks humanity exposes itself to. We consider this type of enhancement a solution to the antithesis between the extinction of humanity and the imperative of humanity to survive at any cost. By opting for voluntary moral bioenhancement; we refrain from advocating illiberal or even totalitarian strategies that would allegedly help humanity preserve itself. We argue that such strategies; by encroaching upon the freedom of individuals; already inflict (...) a degree of existential harm on human beings. We also give some pointers as to the desirable direction for morally enhanced post-personhood. (shrink)

Existence value refers to the value humans ascribe to the existence of something, regard­less of whether it is or will be of any particular use to them. This existence value based on preference satisfaction should be taken into account in evaluating activities that come with a risk of species extinction. There are two main objections. The first is that on the preference satisfaction interpretation, the concept lacks moral importance because satisfying people’s preferences may involve no good or well-being (...) for them. However, existence value can be based on a restricted version of the preference satisfaction theory, which is not vulnerable to the skeptical arguments about the link between preference satisfaction and well-being. The second objection is that even if preference satisfaction can be linked to well-being, understanding existence value in terms of individual preference satisfaction is incoherent, because existence value reflects disinterested preferences that involve no benefits to the individual. However, the fact that existence value may involve disinterested preferences does not threaten the coherence of the concept, but suggests that it does not fit smoothly into the “utilitarian” or “welfarist” framework it is commonly considered within. A pluralistic normative approach based on prima facie duties can be an alternative to standard utilitarian-style approaches for considering existence value in concrete cases involving a risk of species extinction, such as through deep sea mining. (shrink)

How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time, misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language (...) models and game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk‐avoidant risk function. This, in turn, has been claimed to require the use of a risk‐avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks (...) associated with continued human survival. I raise objections to the claim that respect for others' risk attitudes requires risk‐avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others' risk attitudes in fixed population cases. (shrink)

Part I discusses what kind of ‘advances’ occur in Hegel's works, particularly his Philosophy of Nature. I then discuss evolution and extinction in relation to these advances. I summarize Errol Harris' view that Hegel's advances are consistent with current evolutionary theory and then critique this view using articles by Cinzia Ferinni and Alison Stone. I discuss an alternative, post-Kantian Hegelianism which dialectically unites the nature of our cognition with us as subjects that cognize (spirit). For that, I draw on (...) Hegel's claim that ‘The True is just as much Substance as Subject’, and I show how that claim is complicated in a helpful way when applied to the multiple kinds of ‘subjects’ in the Philosophy of Nature. Part II highlights the implications of biological mass extinction for these differing Hegelian theories. Part III brings the above post-Kantian dialectical Hegelianism together with the final chapter of the Phenomenology of Spirit in order to align Hegelian terminology with current solutions for mass extinction: ‘Absolute Knowing’ is a self-sacrificing and a recollecting that reinstates the subject in and through nature; such knowing generates dialectically holistic logics of survival. This recollection of life produces neither abstract spiritual self-consciousness nor a mere ‘gallery’ of extinct life forms. It is philosophical responsibility, identity through natural differences, Absolute Spirit ‘tarrying’ with its world. 21st century Hegelians should be environmentalists on Hegel's terms, a surprising conclusion given that Hegel did not anticipate environmental problems. Hegel's Philosophy of Nature may in turn help environmentalists argue against unacknowledged and badly conceived philosophies of nature. Hegel's work is not reducible to either naturalism or metaphysical idealism: reading Hegel's Philosophy of Nature in conjunction with his Phenomenology of Spirit reveals ways in which a subject, widely and complexly defined at different levels (including different species), is dialectically inseparable from substance, equally widely and complexly defined. This helps us to understand the natural depths of Spirit, which we are at risk of losing. (shrink)

In The Expanse, the future of humanity is constantly at stake. In The Expanse vestiges of an ancient alien civilization with incredibly advanced technology have been found—which eventually permits human interstellar expansion through the gates. James Lenman argues that, even if we agree that biodiversity is a good thing, it only means that it's good that there should be natural diversity while life exists on Earth. While we might not be facing interplanetary war or the unpredictable consequences of ancient alien (...) technology, many serious dangers face the human species today. We are quite resilient and have survived numerous disasters in the past, including floods, earthquakes, famine, pandemics, and wars. Small population size and restricted distribution range are factors that increase a species’ extinction risk. Interplanetary expansion could potentially bring about speciation, which means the separation of Homo sapiens into different species. (shrink)

This review essay contrasts two of the most notable recent contributions to literature on space and society: Daniel Deudney's Dark Skies (2020) and Brian Patrick Green's Space Ethics (2022). The Green volume is a course textbook, geared to giving students an overview of some of the key ethical issues concerning space and how the arguments on these matters are shaping up. Its aim is to provide an overview rather than a specific line of argument. Deudney's text, by contrast, is an (...) example of a book proposing space skepticism. It argues that we should relinquish many of our current ambitions for space expansion on the grounds that they will increase the already significant degree of extinction-level risk that we face. The essay marks the distinction between these texts by contrasting normal- and special-domain approaches. Normal-domain approaches seek to extend familiar ethico-political issues into the discussion about space expansion without regarding space expansion as the road to utopia or extinction. Special-domain approaches hold to some such optimistic or pessimistic view. The essay goes on to highlight the way in which Green's text would benefit from more social critique, along the lines of Deudney. Ultimately, the normal-domain approach presupposed by Green and rejected by Deudney is upheld. (shrink)

Conservation biology is a discipline with the explicit goal of protecting species from extinction. We examine how conservation biologists represent at-risk species, how they navigate values and ethical tensions in the discipline, and how they might be more effective in reaching conservation goals. While these topics are discussed in the literature, we offer a unique empirical examination of how individuals perceive and perform conservation work. We conducted 29 interviews with conservation biologists and found that most respondents viewed their (...) work as providing information but also felt that other species have intrinsic value and we should extend our ethical standards to include other species. However, many attempted to separate science from values, and some felt it was necessary to hide their values and ethical positions and avoid advocacy. While conservation biologists navigate these tensions differently, those who engage in advocacy will likely be more effective in reaching conservation goals. Current societal values and views on ethical extension, rather than a lack of science, represent the most significant impediment to addressing the extinction crisis. (shrink)

Eighty percent of (commercial) genetically engineered seeds (GES) are designed only to resist herbicides. Letting farmers use more chemicals, they cut labor costs. But developing nations say GES cause food shortages, unemployment, resistant weeds, and extinction of native cultivars when “volunteers” drift nearby. While GES patents are reasonable, this paper argues many patent policies are not. The paper surveys GE technology, outlines John Locke’s classic account of property rights, and argues that current patent policies must be revised to take (...) account of Lockean ethical constraints. After answering a key objection, it provides concrete suggestions for implementing its ethical conclusions. (shrink)

The Buddhist Jātaka tells the tale of a hare lounging under a palm tree who becomes convinced the Earth is coming to an end when a ripe bael fruit falls on its head. Soon all the hares are running; other animals join them, forming a stampede of deer, boar, elk, buffalo, wild oxen, rhinoceros, tigers and elephants, loudly proclaiming the earth is ending.1 In the American retelling, the hare is ‘chicken little,’ and the exaggerated fear is that the sky is (...) falling. The story offers a cautionary tale for considering the trend towards calamity thinking in artificial intelligence (AI). A growing chorus of tech leaders has warned that AI poses existential risk (X-Risk) that could result in the extinction of the human species, the collapse of civilisation, or a colossal decline in human potential and culture. In 2014, Hawking told the Washington Post that AI, ‘could spell the end of the human race’.2 Musk has repeatedly warned about AI’s perils, calling AI, ‘more dangerous than nukes’, recommending colonising Mars to ensure ‘a bolt-hole if AI goes rogue and turns on humanity’,3 and donating 10 million dollars to the Future of Life Institute to jumpstart research on AI’s X-Risk. Gates has stated, ‘I am in the camp…concerned about super intelligence…I agree with … Musk and some others on this and don’t understand why some people are not concerned’.4 Altman, CEO of OpenAI, has fretted that technologies OpenAI was building could endanger humanity—‘even destroying the world as we know it’.5 This paper offers a critical appraisal of the rise of calamity thinking in the scholarly AI ethics literature. It cautions against viewing X-Risk in isolation and highlights ethical considerations sidelined when X-Risk takes centre stage. Section I introduces a working definition of X-Risk, considers its likelihood and …. (shrink)

ExcerptRecent advances in the functional power of artificial intelligence (AI) have prompted an urgent warning from industry leaders and researchers concerning its “profound risks to society and humanity.”1 Their open letter is admirable not only for its succinct identification of said risks, which include the mass dissemination of misinformation, loss of jobs, and even the possible extinction of our species, but also for its clear normative framing of the problem: “Should we let machines flood our information channels with propaganda (...) and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?”2. (shrink)

The asymmetry is the view in population ethics that, while we ought to avoid creating additional bad lives, there is no requirement to create additional good ones. The question is how to embed this intuitively compelling view in a more complete normative theory, and in particular one that treats uncertainty in a plausible way. While arguing against existing approaches, I present new and general principles for thinking about welfarist choice under uncertainty. Together, these reduce arbitrary choices to uncertainty-free ones, regardless (...) of how the latter should be made. I illustrate these principles by developing two theories of the asymmetry, reflecting different views about the non-identity problem. In doing so, I clarify some other major choice-points, presenting new arguments that creating additional good lives can justify (but not require) doing harm. Finally, I consider what the developed theories have to say about the importance of extinction risk and the long-run future. (shrink)

Scientists have been arguing for more than 25 years about whether it is a good idea to collect voucher specimens from particularly vulnerable biological populations. Some think that, obviously, scientists should not be harvesting organisms from, for instance, critically endangered species. Others think that, obviously, it is the special job of scientists to collect precisely such information before any chance of retrieving it is forever lost. The character, extent, longevity, and span of the ongoing disagreement indicates that this is likely (...) to be a hard problem to solve. Nonetheless, the aim of this paper is to help field biologists figure out what do to when collecting a voucher specimen risks extinction. Here I present and assess varying practices of specimen collection for both extant and extinct species in order to compare and contrast cases where extinction risk both is and is not a problem. When it comes to taking vouchers from extant species at some risk of extinction, I determine that those advocating for conservative approaches to collection as well as those advocating for liberal information-gathering practices have good reasons to assess things in the way they each do. This means that there is unlikely to be a decisive, one-size-fits-all response to this problem. Still, progress can be made. We can acknowledge the risks of proceeding in either manner, as well as the uncertainty about how best to proceed. We can proceed as thoughtfully as possible, and be ready to articulate a rationale for whichever procedure is used in any particular case. (shrink)

One argument that is sometimes made against pursuing radical forms of human life extension is that such interventions will make the species less evolvable, which would be morally undesirable. In this article, I discuss the empirical and evaluative claims of this argument. I argue that radical increases in life expectancy could, in principle, reduce the evolutionary potential of human populations through both biological and cultural mechanisms. I further argue that if life extension did reduce the evolvability of the species, this (...) will be undesirable for three reasons: it may increase the species’ susceptibility to extinction risks, it may adversely affect institutions and practices that promote well-being, and it may impede moral progress. (shrink)

Group decisions raise a number of substantial philosophical and methodological issues. We focus on the goal of the group decision exercise itself. We ask: What should be counted as a good group decision-making result? The right decision might not be accessible to, or please, any of the group members. Conversely, a popular decision can fail to be the correct decision. In this paper we discuss what it means for a decision to be "right" and what components are required in a (...) decision process to produce happy decision-makers. Importantly, we discuss how "right" decisions can produce happy decision-makers, or rather, the conditions under which happy decision-makers and right decisions coincide. In a large range of contexts, we argue for the adoption of formal consensus models to assist in the group decision-making process. In particular, we advocate the formal consensus convergence model of Lehrer and Wagner (1981), because a strong case can be made as to why the underlying algorithm produces a result that should make each of the experts in a group happy. Arguably, this model facilitates true consensus, where the group choice is effectively each person's individual choice. We analyse Lehrer and Wagner's algorithm for reaching consensus on group probabilities/utilities in the context of complex decision-making for conservation biology. While many conservation decisions are driven by a search for objective utility/probability distributions (regarding extinction risks of species and the like), other components of conservation management primarily concern the interests of stakeholders. We conclude with cautionary notes on mandating consensus in decision scenarios for which no fact of the matter exists. For such decision settings alternative types of social choice methods are more appropriate. (shrink)

“Offsetting” habitat destruction has widespread appeal as an instrument for balancing economic growth with biodiversity conservation. Requiring proponents to pay the nontrivial costs of habitat loss encourages sensitive planning approaches. Offsetting, biobanking, and biodiverse carbon sequestration schemes will play an important role in conserving biodiversity under increasing human pressures. However, untenable assumptions in existing schemes are undermining their benefits. Policies that allow habitat destruction to be offset by the protection of existing habitat are guaranteed to result in further loss of (...) biodiversity. Similarly, schemes that allow trading the immediate loss of existing habitat for restoration projects that promise future habitat will, at best, result in time lags in the availability of habitat that increases extinction risks, or at worst, fail to achieve the offset at all. We detail concerns about existing approaches and describe how offsetting and trading policies can be improved to provide genuine benefits for biodiversity. Due to uncertainties about the way in which restored vegetation matures, we propose that the biodiversity bank should be a savings bank. Accrued biodiversity values should be demonstrated before they can be used to offset biodiversity losses. We provide recommendations about how this could be achieved in practice. (shrink)

This essay uses the global impact of the Coronavirus as a heuristic semiotic for exploring the future of the church. Unlike the pandemic of 1918, which left few dents on the world's economic, social, and cultural systems, almost all the nations of the world have passed laws and implemented procedures that are only comparable to world wars in their impact on entire populations. Nations are acting in unison, but not in unity. This post-COVID, post-Corona world is the 'time that is (...) given' to the church. But it will not be a post-pandemic world. We may become COVID-proof, but we will never be pandemic-proof. There is no pre-COVID reset. There is only risk assessment from natural extinction risks to existential dangers of our own creation that are catching up to us. Disruption is the new status that is never quo; stability is the new abnormality; global cataclysm is the ever-present peril. The only way to prepare for a future of constant 'the end of the world as we know it?' moments is by developing a high Contextual Quotient, and deepening our Contextual Intelligence so we can choose 'the next right thing' in a world of volcanic volatility. CONTRIBUTION: This essay frames the semiotics of a missional ecclesiology in the COVIDian wake from the hermeneutics of blessings not curses. What virtues might we expect to come out of a virus that is fast-forwarding the future, virtues that will shape the contours of Christianity. What if the pandemic is a shock treatment that is putting the world, and the church, back in a new and better equilibrium? What if there are goldmines on the other side of the landmines and minefields? (shrink)

According to an intriguing though somewhat enigmatic line of thought first proposed by Jonathan Bennett, if humanity went extinct any time soon this would be unfortunate because important business would be left unfinished. This line of thought remains largely unexplored. I offer an interpretation of the idea that captures its intuitive appeal, is consistent with plausible constraints, and makes it non-redundant to other views in the literature. The resulting view contrasts with a welfare-promotion perspective, according to which extinction would (...) be bad because it would come at a massive opportunity cost in terms of feasibly achievable welfare over the lifetime of our species. (shrink)

Current planetary defense policy prioritizes a probability assessment of risk of Earth impact by an asteroid or a comet in the planning of detection and mitigation strategies and in setting the levels of urgency and budgeting to operationalize them. The result has been a focus on asteroids of Tunguska size, which could destroy a city or a region, since this is the most likely sort of object we would need to defend against. However a complete risk assessment would (...) consider not only the probability of an impact but also the magnitude of its consequences, which in the case of an object of Chicxulub size could be the end of civilization or even human extinction. This paper argues that a planetary defense policy based on a complete (or one could say genuine) risk assessment would justify expenditures much higher than at present. (shrink)

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI (...) systems capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

We are constantly making choices about how to invest our time and resources. From a moral perspective, we must ask which moral concerns are most deserving of our attention. Longtermism, as e.g. defined by Greaves and MacAskill, holds that our moral focus should be on the long-term future, and that current and medium-term moral problems are comparatively insignificant. This theory is centrally based on the assumption that the moral importance of individuals can be aggregated. Since the number of individuals of (...) future generations far exceeds the number of current individuals and those closer in time, future generations are to bemorally prioritised, according to longtermism. This paper explores the implications of rejecting the premise of moral aggregation of individuals and adopting a strongly non-aggregationist position instead. It is argued that, according to strong non-aggregationism, the magnitude of the probability with which our intervention actually make a difference, as well as whether we look at the available interventions from an ex ante or ex post perspective, are relevant factors in their moral assessment. Ultimately, the conclusion is reached that strong non-aggregationism does likely not support strongly longtermist conclusions. (shrink)

Recently many methods for reducing the risk of human extinction have been suggested, including building refuges underground and in space. Here we will discuss the perspective of using military nuclear submarines or their derivatives to ensure the survival of a small portion of humanity who will be able to rebuild human civilization after a large catastrophe. We will show that it is a very cost-effective way to build refuges, and viable solutions exist for various budgets and timeframes. Nuclear (...) submarines are surface independent, and could provide energy, oxygen, fresh water and perhaps even food for their inhabitants for years. They are able to withstand close nuclear explosions and radiation. They are able to maintain isolation from biological attacks and most known weapons. They already exist and need only small adaptation to be used as refuges. But building refuges is only “Plan B” of existential risk preparation; it is better to eliminate such risks than try to survive them. (shrink)

This chapter makes the case for strong longtermism: the claim that, in many situations, impact on the long-run future is the most important feature of our actions. Our case begins with the observation that an astronomical number of people could exist in the aeons to come. Even on conservative estimates, the expected future population is enormous. We then add a moral claim: all the consequences of our actions matter. In particular, the moral importance of what happens does not depend on (...) when it happens. That pushes us toward strong longtermism. We then address a few potential concerns, the first of which is that it is impossible to have any sufficiently predictable influence on the course of the long-run future. We argue that this is not true. Some actions can reasonably be expected to improve humanity’s long-term prospects. These include reducing the risk of human extinction, preventing climate change, guiding the development of artificial intelligence, and investing funds for later use. We end by arguing that these actions are more than just extremely effective ways to do good. Since the benefits of longtermist efforts are large and the personal costs are comparatively small, we are morally required to take up these efforts. (shrink)

One of Derek Parfit’s greatest legacies was the search for Theory X, a theory of population ethics that avoided all the implausible conclusions and paradoxes that have dogged the field since its inception: the Absurd Conclusion, the Repugnant Conclusion, the Non-Identity Problem, and the Mere Addition Paradox. In recent years, it has been argued that this search is doomed to failure and no satisfactory population axiology is possible. This chapter reviews Parfit’s life’s work in the field and argues that he (...) provided all the necessary components necessary for a Theory X. It then shows how these components can be combined together and applied to the global challenges Parfit argued matter most: preventing human extinction, managing catastrophic risks, and eradicating global poverty and suffering. Finally, it identifies a number of challenges facing his theory and suggests how these may be overcome. (shrink)

Coping with the existential risk of synthetic biology Synthetic biology (synbio) is a domain in full development, with a potentially big impact on humanity, both in a positive and a negative way. In that sense, synbio is not unique. After a short introduction into synbio, we survey the natural phenomena and technologies that can lead to a premature extinction of Earth-originating life or that can lead to the permanent and drastic reduction of its potential. Within that framework, synbio (...) can be classified as an anthropogenic existential risk in the short term. This leads to the question of how to deal with such risky technologies taking into account existential risks. For emerging technologies, Europe has chosen to be guided by the precautionary principle, focussing heavily on the prevention of possible harm. Implementing the principle has however not been able to offset the increase in anthropogenic existential risks. In addition the principle prevents synbio to play an important role in mitigating intermediate existential risks. Hence, there are reasons to abandon the precautionary principle in favour of a European version of the proactionary principle. (shrink)

Utilitarianism may well be the most influential secular ethical theory in the world today. It is also one of the most controversial. It clashes, or is widely thought to clash, with many conventional moral views, and with human rights when they are seen as inviolable. Would it, for example, be right to torture a suspected terrorist in order to prevent an attack that could kill and injure a large number of innocent people? In this Very Short Introduction Peter Singer and (...) Katarzyna de Lazari-Radek provide an authoritative account of the nature of utilitarianism, from its nineteenth-century origins, to its justification and its varieties. Considering how utilitarians can respond to objections that are often regarded as devastating, they explore the utilitarian answer to the question of whether torture can ever be justified. They also discuss what it is that utilitarians should seek to maximize, paying special attention to the classical utilitarian view that only pleasure or happiness is of intrinsic value. Singer and de Lazari-Radek conclude by analysing the continuing importance of utilitarianism in the world, indicating how it is a force for new thinking on contemporary moral challenges like global poverty, the treatment of animals, climate change, reducing the risk of human extinction, end-of-life decisions for terminally-ill patients, and the shift towards assessing the success of government policies in terms of their impact on happiness. ABOUT THE SERIES: The Very Short Introductions series from Oxford University Press contains hundreds of titles in almost every subject area. These pocket-sized books are the perfect way to get ahead in a new subject quickly. Our expert authors combine facts, analysis, perspective, new ideas, and enthusiasm to make interesting and challenging topics highly readable. (shrink)

> Wealthy nations must step up support for Africa and vulnerable countries in addressing past, present and future impacts of climate change The 2022 report of the Intergovernmental Panel on Climate Change paints a dark picture of the future of life on earth, characterised by ecosystem collapse, species extinction and climate hazards such as heatwaves and floods.1 These are all linked to physical and mental health problems, with direct and indirect consequences of increased morbidity and mortality. To avoid these (...) catastrophic health effects across all regions of the globe, there is broad agreement—as 231 health journals argued together in 2021—that the rise in global temperature must be limited to <1.5°C compared with pre-industrial levels. While the Paris Agreement of 2015 outlines a global action framework that incorporates providing climate finance to low-income and middle-income countries, this support has yet to materialise.2 COP27 is the fifth Conference of the Parties (COP) to be organised in Africa since its inception in 1995. Ahead of this meeting, we—as health journal editors from across the continent—call for urgent action to ensure it is the COP that finally delivers climate justice for Africa and vulnerable countries. This is essential for the health of those countries, and for the health of the whole world. The climate crisis has had an impact on the environmental and social determinants of health across Africa, leading to devastating health effects.3 Impacts on health can result directly from environmental shocks and indirectly through socially mediated effects.4 Climate change-related risks in Africa include flooding, drought, heatwaves, reduced food production and reduced labour productivity.5 Droughts in sub-Saharan Africa …. (shrink)

The threat to the survival of humankind posed by nuclear weapons has been a frightening and essential focus of public debate for the last four decades and must continue to be so if we are to avoid destroying ourselves and the natural world around us. One unfortunate result of preoccupation with the nuclear threat, however, has been a new kind of "respectability" accorded to conventional war. In this radical and cogent argument for pacifism, Robert Holmes asserts that all war--not just (...) nuclear war--has become morally impermissible in the modern world. Addressing a wide audience of informed and concerned readers, he raises dramatic questions about the concepts of "political realism" and nuclear deterrence, makes a number of persuasive suggestions for nonviolent alternatives to war, and presents a rich panorama of thinking about war from St. Augustine to Reinhold Niebuhr and Herman Kahn. Holmes's positions are compellingly presented and will provoke discussion both among convinced pacifists and among those whom he calls "militarists." "Militarists," we realize after reading this book, include the majority of us who live a friendly and peaceful personal life while supporting a system which, if Holmes is correct, guarantees war and risks eventual human extinction. Originally published in 1989. The Princeton Legacy Library uses the latest print-on-demand technology to again make available previously out-of-print books from the distinguished backlist of Princeton University Press. These editions preserve the original texts of these important books while presenting them in durable paperback and hardcover editions. The goal of the Princeton Legacy Library is to vastly increase access to the rich scholarly heritage found in the thousands of books published by Princeton University Press since its founding in 1905. (shrink)