Purpose This paper aims to formalize long-term trajectories of human civilization as a scientific and ethical field of study. The long-term trajectory of human civilization can be defined as the path that human civilization takes during the entire future time period in which human civilization could continue to exist. -/- Design/methodology/approach This paper focuses on four types of trajectories: status quo trajectories, in which human civilization persists in a state broadly similar to its current state into the distant future; catastrophe (...) trajectories, in which one or more events cause significant harm to human civilization; technological transformation trajectories, in which radical technological breakthroughs put human civilization on a fundamentally different course; and astronomical trajectories, in which human civilization expands beyond its home planet and into the accessible portions of the cosmos. -/- Findings Status quo trajectories appear unlikely to persist into the distant future, especially in light of long-term astronomical processes. Several catastrophe, technological transformation and astronomical trajectories appear possible. -/- Originality/value Some current actions may be able to affect the long-term trajectory. Whether these actions should be pursued depends on a mix of empirical and ethical factors. For some ethical frameworks, these actions may be especially important to pursue. (shrink)

Many thinkers believe that the next transformational change in human organisation will be the onset of human-level artificial intelligence, and that the most likely method of achieving this will come through brain emulations or "ems": the ability to scan human brains and program their connections into ever faster computers. Taking this as his starting point, Hanson describes what a world dominated by these ems will be like.

Humanity seems to have a bright future, i.e., a non-trivial chance of expanding to fill the universe with lasting life. But the fact that space near us seems dead now tells us that any given piece of dead matter faces an astronomically low chance of begating such a future. There thus exists a great filter between death and expanding lasting life, and humanity faces the ominous question: how far along this filter are we?

Policy disputes arise at all scales of governance: in clubs, non-profits, firms, nations, and alliances of nations. Both the means and ends of policy are disputed. While many, perhaps most, policy disputes arise from conflicting ends, important disputes also arise from differing beliefs on how to achieve shared ends. In fact, according to many experts in economics and development, governments often choose policies that are “inefficient” in the sense that most everyone could expect to gain from other feasible policies. Many (...) other kinds of experts also see existing policies as often clearly inferior to known alternatives. (shrink)

What if we someday learn how to model small brain units, and so can "upload" ourselves into new computer brains? What if this happens before we learn how to make human-level artificial intelligences? The result could be a sharp transition to an upload-dominated world, with many dramatic consequences. In particular, fast and cheap replication may once again make Darwinian evolution of human values a powerful force in human history. With evolved values, most uploads would value life even when life is (...) hard or short, uploads would reproduce quickly, and wages would fall. But total wealth should rise, so we could all do better by accepting uploads, or at worse taxing them, rather than trying to delay or segregate them. (shrink)

If you might be living in a simulation then all else equal you should care less about others, live more for today, make your world look more likely to become rich, expect to and try more to participate in pivotal events, be more entertaining and praiseworthy, and keep the famous people around you happier and more interested in you.

The pace of scientific progress may be hindered by the tendency of our academic institutions to reward being popular rather than being right. A market-based alternative, where scientists can more formally 'stake their reputation', is presented here. It offers clear incentives to be careful and honest while contributing to a visible, self-consistent consensus on controversial scientific questions. In addition, it allows patrons to choose questions to be researched without choosing people or methods. The bulk of this paper is spent in (...) examining potential problems with the proposed approach. After this examination, the idea still seems to be plausible and worthy of further study. (shrink)

Attempts to model interstellar colonization may seem hopelessly compromised by uncertainties regarding the technologies and preferences of advanced civilizations. If light speed limits travel speeds, however, then a selection effect may eventually determine frontier behavior. Making weak assumptions about colonization technology, we use this selection effect to predict colonists’ behavior, including which oases they colonize, how long they stay there, how many seeds they then launch, how fast and far those seeds fly, and how behavior changes with increasing congestion. This (...) colonization model explains several astrophysical puzzles, predicting lone oases like ours, amid large quiet regions with vast unused resources. (shrink)

Department of Economics, George Mason University, MSN 1D3, Carow Hall, Fairfax VA 22030, USA E-mail: [email protected] (http://hanson.gmu.edu).

A simple exogenous growth model gives conservative estimates of the economic implications of machine intelligence. Machines complement human labor when they become more productive at the jobs they perform, but machines also substitute for human labor by taking over human jobs. At first, expensive hardware and software does only the few jobs where computers have the strongest advantage over humans. Eventually, computers do most jobs. At first, complementary effects dominate, and human wages rise with computer productivity. But eventually substitution can (...) dominate, making wages fall as fast as computer prices now do. An intelligence population explosion makes per-intelligence consumption fall this fast, while economic growth rates rise by an order of magnitude or more. These results are robust to automating incrementally, and to distinguishing hardware, software, and human capital from other forms of capital. (shrink)

In standard belief models, priors are always common knowledge. This prevents such models from representing agents’ probabilistic beliefs about the origins of their priors. By embedding standard models in a larger standard model, however, pre-priors can describe such beliefs. When an agent’s prior and pre-prior are mutually consistent, he must believe that his prior would only have been different in situations where relevant event chances were different, but that variations in other agents’ priors are otherwise completely unrelated to which events (...) are how likely. Due to this, Bayesians who agree enough about the origins of their priors must have the same priors. (shrink)

Engineers’ love of technology often gets in the way of their being useful. Consider Post-it Notes or, better yet, plain paper notepads. These probably seemed like trivial ideas, but they turned out to be terribly useful. Why? Because the marvel that is the human brain has a horrible short-term memory, which means that dumb-as-dirt memory aids can make people substantially smarter.

Human behavior regarding medicine seems strange; assumptions and models that seem workable in other areas seem less so in medicine. Perhaps we need to rethink the basics. Toward this end, I have collected many puzzling stylized facts about behavior regarding medicine, and have sought a small number of simple assumptions which might together account for as many puzzles as possible.

In practice, scoring rules elicit good probability estimates from individuals, while betting markets elicit good consensus estimates from groups. Market scoring rules combine these features, eliciting estimates from individuals or groups, with groups costing no more than individuals. Regarding a bet on one event given another event, only logarithmic versions preserve the probability of the given event. Logarithmic versions also preserve the conditional probabilities of other events, and so preserve conditional independence relations. Given logarithmic rules that elicit relative probabilities of (...) base event pairs, it costs no more to elicit estimates on all combinations of these base events. (shrink)

In Everett's many worlds interpretation, quantum measurements are considered to be decoherence events. If so, then inexact decoherence may allow large worlds to mangle the memory of observers in small worlds, creating a cutoff in observable world size. Smaller world are mangled and so not observed. If this cutoff is much closer to the median measure size than to the median world size, the distribution of outcomes seen in unmangled worlds follows the Born rule. Thus deviations from exact decoherence can (...) allow the Born rule to be derived via world counting, with a finite number of worlds and no new fundamental physics. (shrink)

Humans have slowly built more productive societies by slowly acquiring various kinds of capital, and by carefully matching them to each other. Because disruptions can disturb this careful matching, and discourage social coordination, large disruptions can cause a “social collapse,” i.e., a reduction in productivity out of proportion to the disruption. For many types of disasters, severity seems to follow a power law distribution. For some of types, such as wars and earthquakes, most of the expected harm is predicted to (...) occur in extreme events, which kill most people on Earth. So if we are willing to worry about any war or earthquake, we should worry especially about extreme versions. If individuals varied little in their resistance to such disruptions, events a little stronger than extreme ones would eliminate humanity, and our only hope would be to prevent such events. If individuals vary a lot in their resistance, however, then it may pay to increase the variance in such resistance, such as by creating special sanctuaries from which the few remaining humans could rebuild society. (shrink)

Consider two agents who want to be Bayesians with a common prior, but who cannot due to computational limitations. If these agents agree that their estimates are consistent with certain easy-to-compute consistency constraints, then they can agree to disagree about any random variable only if they also agree to disagree, to a similar degree and in a stronger sense, about an average error. Yet average error is a state-independent random variable, and one agent's estimate of it is also agreed to (...) be state-independent. Thus suggests that disagreements are not fundamentally due to differing information about the state of the world. (shrink)

Given common priors, no agent can publicly estimate a non-zero sign for the difference between his estimate and another agent’s future estimate. Thus rational agents cannot publicly anticipate the direction in which other agents will disagree with them.  2002 Elsevier Science B.V. All rights reserved.

There is a widespread feeling that health is special; the rules that are usually used in other policy areas are not applied in health policy. Health economists, for example, tend to be reluctant to offer economists’ usual prescription of competition and consumer choice, even though they have largely failed to justify this reluctance by showing that health economics involves special features such as public goods, externalities, adverse selection, poor consumer information, or unusually severe consequences. Similarly, while some philosophers argue for (...) bioethical conclusions based on very general ethical intuitions,1 many others rely on moral intuitions that are specific to health and medicine to draw conclusions that are meant to apply mainly in health and medicine. For example, many authors appear to start from the strong moral intuition that it typically seems wrong to deny poor people access to health care, and then seek moral principles that can both account for such intuitions and justify the claim that people have some sort of right to health care.2 In metaethics, opinions on moral intuitions range from an extreme intuitionism, which accepts all case-specific moral intuitions at face value as reliable moral guides, to an extreme foundationalism, which rejects such intuitions as evidence regarding correct general moral principles. Between these extremes, opinions vary on how severe the errors in our moral intuitions are. The practice of bioethics seems to favor the extreme intuitionist end of this spectrum, and thus implicitly expects mild errors.3 In contrast, this essay will suggest that common practice in bioethics has seriously underestimated the errors in our moral intuitions. In this essay, I consider the evolutionary origin of our moral intuitions, but avoid the extreme positions of moral skepticism and “whatever evolved must be good,” both of which are commonly associated with evolution-. (shrink)

A world product time series covering two million years is well fit by either a sum of four exponentials, or a constant elasticity of substitution (CES) combination of three exponential growth modes: “hunting,” “farming,” and “industry.” The CES parameters suggest that farming substituted for hunting, while industry complemented farming, making the industrial revolution a smoother transition. Each mode grew world product by a factor of a few hundred, and grew a hundred times faster than its predecessor. This weakly suggests that (...) within the next century a new mode might appear with a doubling time measured in days, not years. (shrink)

There is a widespread feeling that health is special; the rules that are usually used in other policy areas are not applied in health policy. Health economists, for example, tend to be reluctant to offer economists' usual prescription of competition and consumer choice, even though they have largely failed to justify this reluctance by showing that health economics involves special features such as public goods, externalities, adverse selection, poor consumer information, or unusually severe consequences.

Given a finite state space and common priors, common knowledge of the identity of an agent with the minimal (or maximal) expectation of a random variable implies ‘consensus’, i.e., common knowledge of common expectations. This ‘extremist’ statistic induces consensus when repeatedly announced, and yet, with n agents, requires at most log2 n bits to broadcast.

Prediction markets are increasingly being considered as methods for gathering, summarizing and aggregating diffuse information by governments and businesses alike. Critics worry that these markets are susceptible to price manipulation by agents who wish to distort decision making. We study the effect of manipulators on an experimental market, and find that manipulators are unable to distort price accuracy. Subjects without manipulation incentives compensate for the bias in offers from manipulators by setting a different threshold at which they are willing to (...) accept trades. (shrink)

If we are not to conclude that most planets like Earth have evolved life as intelligent as we are, we must presume Earth is not random. This selection effect, however, also implies that the origin of life need not be as easy as the early appearance of life on Earth suggests. If a series of major evolutionary transitions were required to produce intelligent life, selection implies that a subset of these were “critical steps,” with durations that are similarly distributed. The (...) time remaining from now until simple life is no longer possible on Earth must also be similarly distributed. I show how these results provide timing tests to constrain models of critical evolutionary transitions. (shrink)

In Everett’s many-worlds interpretation, where quantum measurements are seen as decoherence events, inexact decoherence may let large worlds mangle the memories of observers in small worlds, creating a cutoff in observable world measure. I solve a growth–drift–diffusion–absorption model of such a mangled worlds scenario, and show that it reproduces the Born probability rule closely, though not exactly. Thus, inexact decoherence may allow the Born rule to be derived in a many-worlds approach via world counting, using a finite number of worlds (...) and no new fundamental physics. (shrink)

We study experimental markets where privately informed traders exchange simple assets, and where uninformed third parties are asked to forecast the values of these assets, guided only by market prices. Although prices only partially aggregate information, they significantly improve the forecasts of third parties. In a second treatment, a portion of traders are given preferences over the forecasts made by observers. Although we find evidence that these traders attempt to manipulate prices in order to influence the beliefs of observers, we (...) find no evidence that observers make less accurate forecasts as a result. (shrink)

Chalmers is right: we should expect our civilization to, within centuries, have vastly increasedmental capacities, surely in total and probably also for individual creatures and devices.We should also expect to see the conflicts he describes between creatures and devices with more versus less capacity. But Chalmers' main prediction follows simply by extrapolating historical trends, and the conflicts he identifies are common between differing generations. There is value in highlighting these issues, but once one knows of such simple extrapolations and standard (...) conflicts, it is hard to see much value in Chalmers' added analysis. (shrink)

Since utilities and probabilities jointly determine choices, event-dependent utilities complicate the elicitation of subjective event probabilities. However, for the usual purpose of obtaining the information embodied in agent beliefs, it is sufficient to elicit objective probabilities, i.e., probabilities obtained by updating a known common prior with that agent’s further information. Bayesians who play a Nash equilibrium of a certain insurance game before they obtain relevant information will afterward act regarding lottery ticket payments as if they had event-independent risk-neutral utility and (...) a known common prior. Proper scoring rules paid in lottery tickets can then elicit objective probabilities. (shrink)

You are in a grocery store, and thinking of buying some meat. You think you know what buying and eating this meat would mean for your taste buds, your nutrition, and your pocketbook, and let's assume that on those grounds it looks like a good deal. But now you want to think about the..

Humans clearly have trouble thinking about death. This trouble is often used to explain behavior like delay in writing wills or buying life insurance, or interest in odd medical and religious beliefs. But the problem is far worse than most people imagine. Fear of death makes us spend fifteen percent of our income on medicine, from which we get little or no health benefit, while we neglect things like exercise, which offer large health benefits.

Are you fascinated by some basic questions about science, technology, and our future? Questions like: Is cryonics technically feasible? When will nano‐assemblers be feasible and how quickly will resulting changes come? Does a larger population help or hinder the world environment and economy? Will uploading be possible, and if so when? When can I live in space? Where will I be able to live free from tyranny? When will AIs be bucking for my job? Is there intelligent life beyond earth? (...) If you are like most Extropy readers, such questions matter to you. (shrink)

The Matrix is a story of AIs who keep humans as slaves, by keeping them in a dream world, and of rebels who fight to teach people this truth and destroy this dream world. But we humans are today slaves to alien hyper-rational entities who care little about us, and who distract us with a dream world. We do not want to know this truth, and if anything fight to preserve our dream world. Go figure.

Vernor Vinge's “singularity” is a worthy contribution to the long tradition of contemplations about human transcendence. Throughout history, most of these musings have dwelled upon the spiritual – the notion that human beings can achieve a higher state through prayer, moral behavior, or mental discipline.

This chapter assesses realistic social implications of emulation (em). It takes the somewhat unusual approach of using basic social theory, in addition to common sense and trend projection, to forecast future societies. Em cities are likely toxic to ordinary humans, who, controlling most of the rest of the Earth, mostly live comfortably on their em‐economy investments. Em's extended lifespans induce greater wealth inequality among ems. Ems focus their identity less on individual personalities and abilities, and more on being part of (...) a particular team. While copy clans coordinate to show off common clan abilities, individual ems focus on showing their identities, abilities, and loyalties as team players. The analysis in this chapter suggests that lives in the next great era might be as different from what we are accustomed to as ours are from farmers' lives, or farmers' from foragers'. (shrink)

Salop’s “Circular City” model of spatial competition is generalized to higher dimensions, and to “transportation” costs which are a power of distance. Assuming free entry, mill pricing is compared to location-based price discrimination. For dimensions above one, there is some too little entry below some cutoff power, and too much entry above it. This cutoff cost-power rises with dimension, and is larger under price discrimination. Mill pricing induces more entry for powers of four or less, and less entry for powers (...) of five or more. Overall, too much entry seems a more severe problem, which tends to price discrimination. (shrink)

Economic growth is determined by the supply and demand of investment capital; technology determines the demand for capital; while human nature determines the supply. The supply curve has two distinct parts; giving the world economy two distinct modes. In the familiar slow growth mode; rates of return are limited by human discount rates. In the fast growth mode; investment is limited by the world's wealth. Historical trends suggest that we may transition to the fast mode in roughly another century and (...) a half. Can some new technology switch us to the fast mode more quickly than this? Perhaps; but such a technology must greatly raise the rate of return for the world's expected worst investment project. It must thus be very broadly applicable; improving almost all forms of capital and investment. Furthermore; investment externalities must remain within certain limits. (shrink)

This paper describes a Bayesian method for constructing a super-resolved surface model by combining information from a set of images of the given surface. We develop the theory and algorithms in detail for the 2-D reconstruction problem, appropriate for the case where all images are taken from roughly the same direction and under similar lighting conditions. We show the results of this 2-D reconstruction on Viking Martian data. These results show dramatic improvements in both spatial and gray-scale resolution. The Bayesian (...) approach uses a neighbor correlation model as well as pixel data from the image set. Some extensions of this method are discussed, including 3-D surface reconstruction and the resolution of diffraction blurred images. (shrink)

He who pays the piper calls the tune, but he can only succesfully call for a tune that he will recognize upon hearing. Previous models, of two candidates impressing a voter and of firm managers impressing stock speculators, found experts ignoring costly superior information in favor of client preconceptions. Similar result hold when we greatly generalize the agents, choices, information structures, and preferences. When experts must pay to acquire information, have no intrinsic interest in client topics, and can coordinate to (...) acquire the same information, no expert ever pays to know more than any client will know when rewarding those experts. (shrink)

In their article, ‘That is not dead which can eternal lie: the aestivation hypothesis for resolving Fermi’s paradox’, Sandberg et al. try to explain the Fermi paradox by claiming that Landauer’s principle implies that a civilization can in principle perform far more times more) irreversible logical operations if it conserves its resources until the distant future when the cosmic background temperature is very low. So perhaps aliens are out there, but quietly waiting. Sandberg et al. implicitly assume, however, that computer-generated (...) entropy can only be disposed of by transferring it to the cosmological background. In fact, while this assumption may apply in the distant future, our universe today contains vast reservoirs and other physical systems in non-maximal entropy states, and computer-generated entropy can be transferred to them at the adiabatic conversion rate of one bit of negentropy to erase one bit of error. This can be done at any time, and is not improved by waiting for a low cosmic background temperature. Thus aliens need not wait to be active. As Sandberg et al. do not provide a concrete model of the effect they assert, we construct one and show where their informal argument goes wrong. (shrink)

Why do we regulate the substances we can ingest, the advisors we can hear, and the products we can buy far more than similarly-important non-health choices? I review many possible arguments for such paternalistic policies, as well many possible holes in such arguments. I argue we should either be clearer about what justifies our paternalism, or we should back off and be less paternalistic.

Before an election, two candidates choose policies which are lotteries over electionday distributive positions. I find conditions under which there exist mixed-strategy probabilistic-voting equilibria which are independent, treating voter groups independently.

Being read is not the same as being believed. Most reviewers have praised the book as original, well-written, thought-provoking, etc., and then gone on to take issue with one or more of Penrose's main theses. Penrose seems unfamiliar with the existing literature in cognitive science, philosophy of mind, and AI. The handful of reviewers who agree with Penrose don't seem to have paid much attention to his specific arguments - they always thought AI was bogus. See, for example, the 37 (...) reviews in Behavioral and Brain Sciences (BBS), Dec. 1990, V13, pp.643-705. (shrink)

Are you fascinated by some basic questions about science, technology, and our future? Questions like: Is cryonics technically feasible? When will nanoassemblers be feasible and how quickly will resulting changes come? Does a larger population help or hinder the world environment and economy? Will uploading be possible, and if so when? When can I live in space? Where will I be able to live free from tyranny? When will A.I.s be bucking for my job? Is there intelligent life beyond earth? (...) If you are like most Extropy readers, such question matter to you. Now how do we, as a society, go about answering such questions? People who have an appropriate background, and who are interested enough in a particular question, can research that subject in depth themselves, and come to a considered opinion. And people who happen to know, respect, and trust such a person can simply take those opinions as their own, avoiding all the hard work. But what is everyone else to do, people whose actions often implicitly depend on such questions? In practice, people usually defer to larger social institutions on most questions, institutions which combine and evaluate contributions from many specialists, and which offer apparent institutional consensus estimates on many different questions. (shrink)

Extraordinary claims require extraordinary evidence. But on uninteresting topics, surprising claims usually are surprising evidence; we rarely make claims without suffi- cient evidence. On interesting topics, however, we can have interests in exaggerating or downplaying our evidence, and our actions often deviate from our interests. In a simple model of noisy humans reporting on extraordinary evidence, we find that extraordinary claims from low noise people are extraordinary evidence, but such claims from high noise people are not; their claims are more (...) likely unusual noise than unusual truth. When people are organized into a reporting chain, noise levels grow exponentially with chain length; long chains seem incapable of communicating extraordinary evidence. (shrink)

Artificial Intelligence Research Branch NASA Ames Research Center, Mail Stop 244-17 Moffet Field, CA 94035, USA Email: @ptolemy.arc.nasa.gov..

In a simple model of conflict, two agents fight over a fixed prize, and how hard they fight depends on what they believe about their abilities. To this model I add “preagents,” representing parents, leaders, or natural selection, who choose each agent’s confidence in his ability. Depending on the reason for such confidence, I find five different patterns in how confidence varies with ability. Agents who estimate their ability with error have under-confidence when ability is high and over-confidence when ability (...) is low, while strategic commitment incentives induce the opposite pattern. Agents who misjudge their value for the prize, relative to their cost of effort, induce an overor under-confidence that is independent of ability, while cooperating pre-agents choose extreme under-confidence. Agents who use confidence to signal ability have a relatively uniform over-confidence. (shrink)