The base rate fallacy is a fallacy that occurs in probabilistic reasoning when available general information is omitted from the calculations and attention is given to specific information only. To illustrate this concept, this chapter discusses the scenario in Philip K. Dick's short story “Minority Report”, with some details augmented by the 2002 movie daptation by Steven Spielberg. The legislative approach seems to be based not just on the base rate fallacy but on assuming a (...) class='Hi'>base rate that is divorced from reality, that the drug use rate among welfare applicants is significantly higher than the national rate. Once the accurate base rate is applied (be it the proportions between the races in the US population, or the percentage of drug users among the people receiving government welfare, etc.), one can clearly see that the original conclusion does not follow. (shrink)

The phenomenon of base-rate neglect has elicited much debate. One arena of debate concerns how people make judgments under conditions of uncertainty. Another more controversial arena concerns human rationality. In this target article, we attempt to unpack the perspectives in the literature on both kinds of issues and evaluate their ability to explain existing data and their conceptual coherence. From this evaluation we conclude that the best account of the data should be framed in terms of a dual-process (...) model of judgment, which attributes base-rate neglect to associative judgment strategies that fail to adequately represent the set structure of the problem. Base-rate neglect is reduced when problems are presented in a format that affords accurate representation in terms of nested sets of individuals. (shrink)

We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. At the empirical level, a thorough examination of the base rate literature (including the famous lawyer–engineer problem) does not support the conventional wisdom that people routinely ignore base rates. Quite the contrary, the literature shows that base rates are almost always used and that their degree of use depends on task structure and representation. Specifically, (...) class='Hi'>base rates play a relatively larger role in tasks where base rates are implicitly learned or can be represented in frequentist terms. Base rates are also used more when they are reliable and relatively more diagnostic than available individuating information. At the normative level, the base rate fallacy should be rejected because few tasks map unambiguously into the narrow framework that is held up as the standard of good decision making. Mechanical applications of Bayes's theorem to identify performance errors are inappropriate when (1) key assumptions of the model are either unchecked or grossly violated, and (2) no attempt is made to identify the decision maker's goals, values, and task assumptions. Methodologically, the current approach is criticized for its failure to consider how the ambiguous, unreliable, and unstable base rates of the real world are and should be used. Where decision makers' assumptions and goals vary, and where performance criteria are complex, the traditional Bayesian standard is insufficient. Even where predictive accuracy is the goal in commonly defined problems, there may be situations (e.g., informationally redundant environments) in which base rates can be ignored with impunity. A more ecologically valid research program is called for. This program should emphasize the development of prescriptive theory in rich, realistic decision environments. (shrink)

The commentaries indicate a general agreement that one source of reduction of base-rate neglect involves making structural relations among relevant sets transparent. There is much less agreement, however, that this entails dual systems of reasoning. In this response, we make the case for our perspective on dual systems. We compare and contrast our view to the natural frequency hypothesis as formulated in the commentaries.

Base rates have no necessary relation to judgments that are not themselves probabilities. There is no logical imperative, for instance, that behavioral base rates must affect causal attributions or that base rate information should affect judgments of legal liability. Decision theorists should be cautious in arguing that base rates place normative constraints on judgments of anything other than posterior probabilities.

We investigated the influence of stimulus base rates on the Implicit Association Test . Using an East/West-German attitude-IAT, we demonstrated that both overall response speed and differential response speed underlying IAT effects depend on the relative frequencies of the stimulus categories. First, when those stimuli that are more common in reality also occurred more frequently in the stimulus list, response speed generally increased. Second, IAT effects increased when congruent blocks profited from the compatibility of frequency-based response biases , whereas (...) IAT effects decreased when incongruent trial blocks profited from response compatibility. These findings demonstrate that the stimulus context moderates the magnitude of the IAT effect. Simultaneously, they highlight the need to explore the extent to which implicit measures reflect properties of the task or the environment rather than attributes of test-takers. (shrink)

There is a theorem that shows that it is impossible for an algorithm to jointly satisfy the statistical fairness criteria of Calibration and Equalised Odds non-trivially. But what about the recently advocated alternative to Calibration, Base Rate Tracking? Here, we show that Base Rate Tracking is strictly weaker than Calibration, and then take up the question of whether it is possible to jointly satisfy Base Rate Tracking and Equalised Odds in non-trivial scenarios. We show (...) that it is not, thereby establishing an even more general impossibility theorem. (shrink)

The no-miracles argument and the pessimistic induction are arguably the main considerations for and against scientific realism. Recently these arguments have been accused of embodying a familiar, seductive fallacy. In each case, we are tricked by a base rate fallacy, one much-discussed in the psychological literature. In this paper we consider this accusation and use it as an explanation for why the two most prominent `wholesale' arguments in the literature seem irresolvable. Framed probabilistically, we can see very clearly (...) why realists and anti-realists have been talking past one another. We then formulate a dilemma for advocates of either argument, answer potential objections to our criticism, discuss what remains (if anything) of these two major arguments, and then speculate about a future philosophy of science freed from these two arguments. In so doing, we connect the point about base rates to the wholesale/retail distinction; we believe it hints at an answer of how to distinguish profitable from unprofitable realism debates. In short, we offer a probabilistic analysis of the feeling of ennui afflicting contemporary philosophy of science. (shrink)

The tendency to neglect base-rates in judgment under uncertainty may be as Barbey & Sloman (B&S) suggest, but it is neither inevitable (as they document; see also Koehler 1996) nor unique. Here we would like to point out another line of evidence connecting ecological rationality to dual processes, the failure of individuals to appropriately judge cumulative probability.

The base rate fallacy is directly dependent on a particular judgment paradigm in which information may be unambiguously designated as either “base rate” or “individuating,” and in which subjects make two-stage sequential judgments. The paradigm may be a poor match for real world settings, and the fallacy may thus be undefined for natural ecologies of judgment.

The base rate literature has an opposite twin in the social psychological literature on stereotypes, which concludes that people use their preexisting beliefs about probabilistic category attributes too much, rather than not enough. This ironic discrepancy arises because beliefs about category attributes enhance accuracy when the beliefs are accurate and diminish accuracy when they are not. To determine the accuracy of base rate/stereotype beliefs requires research that addresses specific content.

In base rate problems the estimated probability must equal the base rate only where random sampling is assumed. Otherwise there is uncertainty over and above that which can be included in any probability model and inductive inference is involved. Subjects should use base rates to the extent that the problem suggests a simple random sampling model.

Decision-makers in institutional and non-institutional contexts are sometimes confronted with the issue of whether to use generalisations expressing the statistical incidence of a negative trait in a disadvantaged and discriminated-against social group in order to draw an inference concerning a member of that group. If a criminal court were confronted with such a question, what answer should it give? First, the article argues that, our qualms notwithstanding, morality does not demand that these generalisations be disregarded. In doing so, the article (...) addresses the relationship between factual accuracy and the demands of morality in criminal trials. Second, the article considers the implications of this conclusion for the legal question as to whether the evidence at issue should be excluded, in particular, on grounds of unfairness — pursuant to section 78 of the Police and Criminal Evidence Act 1984. (shrink)

While improving the theoretical account of base-rate neglect, Barbey & Sloman's (B&S's) target article suffers from affect neglect by failing to consider the fundamental role of emotional processes in decisions. We illustrate how affective influences are fundamental to decision making, and discuss how the dual process model can be a useful framework for understanding hot and cold cognition in reasoning.

Setting the two hypotheses of complete neglect and full use of base rates against each other is inappropriate. The proper question concerns the degree to which base rates are used (or neglected), and under what conditions. We outline alternative approaches and recommend regression analysis. Koehler's conclusion that we have been oversold on the base rate fallacy seems to be premature.

The important question is how people process probabilistic information, not whether they process it in accordance with a normative model that we never should have expected them to be capable of following. Experience is not the cure, as widely thought, to problems with utilizing base rate information.

Koehler's call for a reanalysis of the base rate fallacy is particularly important in the applied domain of accounting, since base rate data appear to be an important input for many accounting tasks. In this commentary I discuss the use of base rates in accounting and explain why more flexible standards of performance are important when judging the use of base rates.

Recent literature in the scientific realism debate has been concerned with a particular species of statistical fallacy concerning base-rates, and the worry that no matter how predictively successful our contemporary scientific theories may be, this will tell us absolutely nothing about the likelihood of their truth if our overall sample space contains enough empirically adequate theories that are nevertheless false. In response, both realists and anti-realists have switched their focus from general arguments concerning the reliability and historical track-records of (...) our scientific methodology, to a series of specific arguments and case-studies concerning our reasons to believe individual scientific theories to be true. Such a development however sits in tension with the usual understanding of the scientific realism debate as offering a second-order assessment of our first-order scientific practices, and threatens to undermine the possibility of a distinctive philosophical debate over the approximate truth of our scientific theories. I illustrate this concern with three recent attempts to offer a more localised understanding of the scientific realism debate—due to Stathis Psillos, Juha Saatsi, and Kyle Stanford—and argue that none of these alternatives offer a satisfactory response to the problem. (shrink)

Three experiments examine the effect of base rate consistency under direct experience. Base rate consistency was manipulated by blocking trials and setting base rate choice reinforcement to be either consistent or inconsistent across trial blocks. Experiment 1 shows that, contrary to the usual finding, participants use base rate information more than individuating information when it is consistent, but less when it is inconsistent. In Experiment 2, this effect was replicated, and transferred in (...) verbal questions posed subsequently. Despite experience with consistent base rates increasing sensitivity to base rates in word problems, verbal responses were far from normative. In Experiment 3, participants’ use of base rates was once again moderated by its consistency, but this effect was itself moderated by the diagnosticity of base rate information. Participants were highly accurate in estimating experienced base rates. These studies demonstrate that base rate usage is complex and a function of how base rates are presented (experienced versus summary statistics) and response format (choice proportions versus probability estimates). Knowledge of base rates was insufficient for proper usage in verbal word problems. Although choice proportions showed a sophisticated sensitivity to experienced base rate information, participants seemed unable to demonstrate a similar sophistication when given typical word problems indicating that base rate neglect is a function of information representation and not an inherent processing bias. (shrink)

We believe that when assessing the likelihood of uncertain events, statistically unsophisticated people utilize a coarse internal scale that only has a limited number of categories. The success of the nested sets hypothesis may lie in its ability to provide an appropriate set structure of the problem by reducing the computational demands.

Philosophy & Public Affairs, Volume 50, Issue 2, Page 239-266, Spring 2022.

Base rates are vital in predicting violent criminal recidivism. However, both lay people given simulated prediction tasks and professionals milking real life predictions appear insensitive to variations in the base rate of violent recidivism. Although there are techniques to help decision makers attend to base rates, increased decision accuracy is better sought in improved actuarial models as opposed to improved clinicians.

When base rates are learned and used in an experiential manner subjects show better base rate use, perhaps because the implicit learning system is engaged. A causal framework in which base rates are relevant might also be necessary. Humans might thus perform better on more ecologically valid tasks, which are likely to contain those three components.

Koehler is right that base rate information is used, to various degrees, both in laboratory tasks and in everyday life. However, it is not time to turn our backs on laboratory tasks and focus solely on ecologically valid decision making. Tightly controlled experimental data are still needed to understandhowbase rate information is used, and how this varies among groups.

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the field (...) of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science. (shrink)

We agree that current evolutionary accounts of base-rate neglect are unparsimonious, but we dispute the authors' account of the effect in terms of parallel associative and rule-based processes. We also question their assumption that cueing of nested set relations facilitates performance due to recruitment of explicit reasoning processes. In our account, such reasoning is always involved, but usually unsuccessful.

Koehler's target article attempts a balanced view of the relevance of knowledge of base rates to judgments of subjective or credal probability, but he is not sensitive enough to the difference between requiring and permitting the equation of probability judgments with base rates, the interaction between precision of base rate and reference class information, and the possibility of indeterminate probability judgment.

Consistent with Koehler's position, we propose a generalization of the base rate fallacy and earlier conservatism literatures. In studies using both traditional tasks and new tasks based on ecologically valid base rates, our subjects typically underweight individuating information at least as much as they underweight base rates. The implications of cue consistency for averaging heuristics are discussed.

Barbey & Sloman (B&S) claim that frequency formats and other task manipulations induce people to substitute associative thinking for rule-based thinking about nested sets. My critique focuses on the substitution assumption. B&S demonstrate that nested sets are important to solve base-rate problems but they do not show that thinking about these nested sets relies on a different type of reasoning.

We present evidence supporting the target article's assertion that while the presentation of base-rate information in a natural frequency format can be helpful in enhancing sensitivity to base rates, method of presentation is not a panacea. Indeed, we review studies demonstrating that when subjects directly experience base rates as natural frequencies in a trial-by-trial setting, they evince large base-rate neglect.

A recent study showed physicians' reasoning about a realistic case to be ignorant of base rate. It also showed physicians interpreting information pertinent to base rate differently, depending on whether it was presented early or late in the case. Although these adult reasoners might do better if given hints through talk of relative frequencies, this would not prove that they had no problem of base rate neglect.

Barbey & Sloman (B&S) attribute base-rate neglect to associative processes (like retrieval from memory) that fail to adequately represent the set structure of the problem. This commentary notes that associative responses can also lead to base-rate overweighting. We suggest that the difference between the two patterns is related to the distinction between decisions from experience and decisions from description.

The no miracles argument is one of the main arguments for scientific realism. Recently it has been alleged that the no miracles argument is fundamentally flawed because it commits the base rate fallacy. The allegation is based on the idea that the appeal of the no miracles argument arises from inappropriate neglect of the base rate of approximate truth among the relevant population of theories. However, the base rate fallacy allegation relies on an assumption (...) of random sampling of individuals from the population which cannot be made in the case of the no miracles argument. Therefore the base rate fallacy objection to the no miracles argument fails. I distinguish between a “local” and a “global” form of the no miracles argument. The base rate fallacy objection has been leveled at the local version. I argue that the global argument plays a key role in supporting a base-rate-fallacy-free formulation of the local version of the argument. (shrink)

The underutilization of base rates is a consistent finding. The strong claim that base rates are ignored has been rejected and this needs no further emphasis. Following the path of “normal science,” research examines the conditions predicting changes in the degree of underutilization. A scientific revolution that might dethrone the heuristics and biases paradigm is not in sight.

In many base rate studies, a judgment is required for which the base rates are relevant, and subjects do not use them. It is inferred that the base rates are ignored; I question this inference. Second, I argue that the base rate fallacy is not less significant for what it reveals about human reasoning, if it occurs less frequently than has been alleged.

Abstract(1) There is enough contradictory evidence regarding the role of base rates in category learning to confirm the nonexistence of biases in such learning. (2) It is not always possible to activate statistical reasoning through frequentist representation. (3) It is necessary to use the concept of systematic processing in reconsidering the published work on biases.

According to an argument by Colin Howson, the no-miracles argument is contingent on committing the base-rate fallacy and is therefore bound to fail. We demonstrate that Howson’s argument only applies to one of two versions of the NMA. The other version, which resembles the form in which the argument was initially presented by Putnam and Boyd, remains unaffected by his line of reasoning. We provide a formal reconstruction of that version of the NMA and show that it is (...) valid. Finally, we demonstrate that the use of subjective priors is consistent with the realist implication of the NMA and show that a core worry with respect to the suggested form of the NMA can be dispelled. (shrink)

The meaning of an experimental result depends on the experiment's conceptual backdrop, particularly its null hypothesis. This observation provides the basis for a functional interpretation of belief in the base rate fallacy. On this interpretation, if the base rate fallacy is to be labelled a “myth,” then it should be recognized that this label is not necessarily a disparaging one.

This article calls into question the charge that frequentist testing is susceptible to the base-rate fallacy. It is argued that the apparent similarity between examples like the Harvard Medical School test and frequentist testing is highly misleading. A closer scrutiny reveals that such examples have none of the basic features of a proper frequentist test, such as legitimate data, hypotheses, test statistics, and sampling distributions. Indeed, the relevant error probabilities are replaced with the false positive/negative rates that constitute (...) deductive calculations based on known probabilities among events. As a result, the ampliative dimension of frequentist induction—learning from data about the underlying data-generating mechanism—is missing. *Received August 2009; revised January 2010. †To contact the author, please write to: Department of Economics, Virginia Tech, Blacksburg, VA 24061; e-mail: [email protected]. (shrink)

Commentators agree that simple conclusions about a general base rate fallacy are not appropriate. It is more constructive to identify conditions under which base rates are differentially weighted. Commentators also agree that improving the ecological validity of the research is desirable, although this is less important to those interested exclusively in psychological processes. The philosophers and ecologists among the commentators offer a kinder perspective on base rate reasoning than the psychologists. My own perspective is that (...) the interesting questions (both psychological and normative/prescriptive) are best answered through empirical studies that use real-world performance standards. (shrink)

Is science in the midst of a crisis of replicability and false discoveries? In a recent article, Alexander Bird offers an explanation for the apparent lack of replicability in the biomedical sciences. Bird argues that the surprise at the failure to replicate biomedical research is a result of the fallacy of neglecting the base rate. The base-rate fallacy arises in situations in which one ignores the base rate—or prior probability—of an event when assessing the (...) probability of this event in the light of some observed evidence. By extension, the replication crisis would result from ignoring the low prior probability of biomedical hypotheses. In this paper, my response to Bird’s claim is twofold. First, I show that the argument according to which the replication crisis is due to the low prior of biomedical hypotheses is incomplete. Second, I claim that a simple base-rate fallacy model does not account for some important methodological insights that have emerged in discussions of the replication crisis. (shrink)

Patients’ beliefs about the effectiveness of their treatments are key to the success of any intervention. However, since these beliefs are usually formed by sequentially accumulating evidence in the form of the covariation between the treatment use and the symptoms, it is not always easy to detect when a treatment is actually working. In Experiments 1 and 2, we presented participants with a contingency learning task in which a fictitious treatment was actually effective to reduce the symptoms of fictitious patients. (...) However, the base-rate of the symptoms was manipulated so that, for half of participants, the symptoms were very frequent before the treatment, whereas for the rest of participants, the symptoms were less frequently observed. Although the treatment was equally effective in all cases according to the objective contingency between the treatment and healings, the participants’ beliefs on the effectiveness of the treatment were influenced by the base-rate of the symptoms, so that those who observed frequent symptoms before the treatment tended to produce lower judgments of effectiveness. Experiment 3 showed that participants were probably basing their judgments on an estimate of effectiveness relative to the symptom base-rate, rather than on contingency in absolute terms. Data, materials, and R scripts to reproduce the figures are publicly available at the Open Science Framework. (shrink)