Philosophy of Statistics

In the context of discovery-oriented hypothesis testing research, behavioral scientists widely accept a convention for false positive (α) and false negative error rates (β) proposed by Jacob Cohen, who deemed the general relative seriousness of the antecedently accepted α = 0.05 to be matched by β = 0.20. Cohen’s convention not only ignores contexts of hypothesis testing where the more serious error is the β-error. Cohen’s convention also implies for discovery-oriented hypothesis testing research that a statistically significant observed effect is (...) four times more probable to be a mistaken discovery than for a statistically significant true observed effect to be independently replicable. In the long run, Cohen’s convention thus is epistemically harmful to the development of a progressive science of human behavior, making its acceptance crucial in explaining the replication crisis in behavioral science. The balance between α- and β-errors generally ought to be struck using both epistemic and practical considerations. Yet epistemic considerations alone imply that making a genuine contribution to the body of knowledge in behavioral science requires error rates that are not only small but also symmetrical. (shrink)

I argue for a connection between two debates in the philosophy of probability. On the one hand, there is disagreement about conditional probability. Is it to be defined in terms of unconditional probability, or should we instead take conditional probability as the primitive notion? On the other hand, there is disagreement about how additive probability is. Is it merely finitely additive, or is it additionally countably additive? My thesis is that, if conditional probability is primitive, then it is not countably (...) additive. (shrink)

This article concerns the psychology of the paradoxical Two Envelope Problem. The goal is to find instructive variants of the envelope switching problem that are capable of clear-cut resolution, while still retaining paradoxical features. By relocating the original problem into different contexts involving commutes and playing cards the reader is presented with a succession of resolved paradoxes that reduce the confusion arising from the parent paradox. The goal is to reduce confusion by understanding how we sometimes misread mathematical statements; or, (...) to completely avoid confusion, either by reforming language, or adopting an unambiguous notation for switching problems. This article also suggests that an illusion close in character to the figure/ground illusion hampers our understanding of switching problems in general and helps account for the intense confusion that switching problems sometimes generate. (shrink)

Ex ante predicted outcomes should be interpreted as counterfactuals (potential histories), with errors as the spread between outcomes. But error rates have error rates. We reapply measurements of uncertainty about the estimation errors of the estimation errors of an estimation treated as branching counterfactuals. Such recursions of epistemic uncertainty have markedly different distributial properties from conventional sampling error, and lead to fatter tails in the projections than in past realizations. Counterfactuals of error rates always lead to fat tails, regardless of (...) the probability distribution used. A mere .01% branching error rate about the STD (itself an error rate), and .01% branching error rate about that error rate, etc. (recursing all the way) results in explosive (and infinite) moments higher than 1. Missing any degree of regress leads to the underestimation of small probabilities and concave payoffs (a standard example of which is Fukushima). The paper states the conditions under which higher order rates of uncertainty (expressed in spreads of counterfactuals) alters the shapes the of final distribution and shows which a priori beliefs about conterfactuals are needed to accept the reliability of conventional probabilistic methods (thin tails or mildly fat tails). (shrink)

In order to perform certain actions – such as incarcerating a person or revoking parental rights – the state must establish certain facts to a particular standard of proof. These standards – such as preponderance of evidence and beyond reasonable doubt – are often interpreted as likelihoods or epistemic confidences. Many theorists construe them numerically; beyond reasonable doubt, for example, is often construed as 90 to 95% confidence in the guilt of the defendant. -/- A family of influential cases suggests (...) standards of proof should not be interpreted numerically. These ‘proof paradoxes’ illustrate that purely statistical evidence can warrant high credence in a disputed fact without satisfying the relevant legal standard. In this essay I evaluate three influential attempts to explain why merely statistical evidence cannot satisfy legal standards. (shrink)

I defend the thesis that legal standards of proof are reducible to thresholds of probability. Many have rejected this thesis because it seems to entail that defendants can be found liable solely on the basis of statistical evidence. I argue that this inference is invalid. I do so by developing a view, called Legal Causalism, that combines Thomson's (1986) causal analysis of evidence with recent work in formal theories of causal inference. On this view, legal standards of proof can be (...) reduced to probabilities, but deriving these probabilities involves more than just statistics. (shrink)

This book investigates the interplay between two new and influential subdisciplines in the philosophy of science and philosophy: contemporary scientific metaphysics and the philosophy of information. Scientific metaphysics embodies various scientific realisms and has a partial intellectual heritage in some forms of neo-positivism, but is far more attuned than the latter to statistical science, theory defeasibility, scale variability, and pluralist ontological and explanatory commitments, and is averse to a-priori conceptual analysis. The philosophy of information is the combination of what has (...) been called the informational turn in philosophy and ontology, with the informational turn in logic. The book explores the intersecting theoretical commitments and metaphysical basis of both. It applies scientific metaphysics to the philosophy of information, and also correspondingly proposes a new informational interpretation of scientific metaphysics. These are applied to numerous outstanding ontological, philosophical, and theoretical problems in the philosophy of information, scientific realism, cognitive science, theories of semantic information, and informational logic. The book investigates known and new problems, and advances debates and insights in the philosophy of information. It will be of interest to philosophers of information and science, metaphysicians, and cognitive scientists. (shrink)

This book provides a new all-round perspective on the life and work of Edgar Zilsel (1891-1944) as a philosopher, historian, and sociologist. He was close to the Vienna Circle and has been hitherto almost exclusively referred to in terms of the so-called “Zilsel thesis” on the origins of modern science. Much beyond this “thesis”, Zilsel’s brilliant work provides original insights on a broad number of topics, ranging from the philosophy of probability and statistics to the concept of “genius”, from the (...) issues of scientific laws and theories to the sociological background of science and philosophy, and to the political analysis of the problems of his time. Praised by Herbert Feigl as an “outstanding brilliant mind”, Zilsel, being as a Social-Democrat of Jewish origins, mostly led a life of hardship marked by emigration and coming to a sudden and tragic end by suicide in 1944. The impossibility of an academic career has hindered the reception of Zilsel’s scientific work for a long time. This volume is a contribution to its late reception, providing new insights especially into his work during his years in Vienna; moreover, it shows the heuristic value of Zilsel’s ideas for future Scholar research – in philosophy, history, and sociology. -/- (ISSN: 0929-6328). (shrink)

We study the phenomenon of merging of opinions for computationally limited Bayesian agents from the perspective of algorithmic randomness. When they agree on which data streams are algorithmically random, two Bayesian agents beginning the learning process with different priors may be seen as having compatible beliefs about the global uniformity of nature. This is because the algorithmically random data streams are of necessity globally regular: they are precisely the sequences that satisfy certain important statistical laws. By virtue of agreeing on (...) what data streams are algorithmically random, two Bayesian agents can thus be taken to concur on what global regularities they expect to see in the data. We show that this type of compatibility between priors suffices to ensure that two computable Bayesian agents will reach inter-subjective agreement with increasing information. In other words, it guarantees that their respective probability assignments will almost surely become arbitrarily close to each other as the number of observations increases. Thus, when shared by computable Bayesian learners with different subjective priors, the beliefs about uniformity captured by algorithmic randomness provably lead to merging of opinions. (shrink)

Methods of machine learning (ML) are gradually complementing and sometimes even replacing methods of classical statistics in science. This raises the question whether ML faces the same methodological problems as classical statistics. This paper sheds light on this question by investigating a long-standing challenge to classical statistics: the reference class problem (RCP). It arises whenever statistical evidence is applied to an individual object, since the individual belongs to several reference classes and evidence might vary across them. Thus, the problem consists (...) in choosing a suitable reference class for the individual. I argue that deep neural networks (DNNs) are able to overcome specific instantiations of the RCP. Whereas the criteria of narrowness, reliability, and homogeneity, that have been proposed to determine a suitable reference class, pose an inextricable tradeoff to classical statistics, DNNs are able to satisfy them in some situations. On the one hand, they can exploit the high dimensionality in big-data settings. I argue that this corresponds to the criteria of narrowness and reliability. On the other hand, ML research indicates that DNNs are generally not susceptible to overfitting. I argue that this property is related to a particular form of homogeneity. Taking both aspects together reveals that there are specific settings in which DNNs can overcome the RCP. (shrink)

We argue that subjective Bayesians face a dilemma: they must offend against the spirit of their permissivism about rational credence or reject the principle that one should avoid accuracy dominance.

A view that emancipates free will by means of quantum indeterminism is frequently rejected based on arguments pointing out its incompatibility with what we know about quantum physics. However, if one carefully examines what classical physical causal determinism and quantum indeterminism are according to physics, it becomes clear what they really imply–and, especially, what they do not imply–for agent-causation theories. Here, we will make necessary conceptual clarifications on some aspects of physical determinism and indeterminism, review some of the major objections (...) against libertarian conjectures, and show that there is no conceptual incompatibility preventing us from taking a ‘quantum-libertarian’ approach to the problem of free will. In particular, we will illustrate the possible role of self-causation (causa sui) as a potential solution to otherwise apparently incompatible or even paradoxical statements concerning free will and quantum indeterminism. (shrink)

The framework of Solomonoff prediction assigns prior probability to hypotheses inversely proportional to their Kolmogorov complexity. There are two well-known problems. First, the Solomonoff prior is relative to a choice of Universal Turing machine. Second, the Solomonoff prior is not computable. However, there are responses to both problems. Different Solomonoff priors converge with more and more data. Further, there are computable approximations to the Solomonoff prior. I argue that there is a tension between these two responses. This is because computable (...) approximations to Solomonoff prediction do not always converge. (shrink)

It is one thing to hold that merely statistical evidence is _sometimes_ insufficient for rational belief, as in typical lottery and profiling cases. It is another thing to hold that merely statistical evidence is _always_ insufficient for rational belief. Indeed, there are cases where statistical evidence plainly does justify belief. This project develops a dispositional account of the normativity of statistical evidence, where the dispositions that ground justifying statistical evidence are connected to the goals (= proper function) of objects. There (...) are strong intuitive motivations for doing this. For we can turn almost any case of _non-justifying_ merely statistical evidence into a case of _justifying_ merely statistical evidence by adding information about the dispositions and goals of the objects involved. The resulting view not only helps us understand when and why merely statistical evidence is normatively significant, but it also helps us understand how statistical evidence relates to more standard forms of evidence (perceptual, testimonial). The emerging view also has surprising applications, as it imposes limitations on the epistemic value of fine-tuning arguments for theism as well as undermines a standard class of case-based arguments for moral encroachment. (shrink)

Acknowledging that many members of the SM3D Portal need reference documents related to Bayesian Mindsponge Framework (BMF) analytics to conduct research projects effectively, we present the essential materials and most up-to-date studies employing the method in this post. By summarizing all the publications and preprints associated with BMF analytics, we also aim to help researchers reduce the time and effort for information seeking, enhance proactive self-learning, and facilitate knowledge exchange and community dialogue through transparency.

Personalized medicine relies on two points: 1) causal knowledge about the possible effects of X in a given statistical population; 2) assignment of the given individual to a suitable reference class. Regarding point 1, standard approaches to causal inference are generally considered to be characterized by a trade-off between how confidently one can establish causality in any given study (internal validity) and extrapolating such knowledge to specific target groups (external validity). Regarding point 2, it is uncertain which reference class leads (...) to the most reliable inferences. -/- Instead, pharmacovigilance focuses on both elements of the individual prediction at the same time, that is, the establishment of the possible causal link between a given drug and an observed adverse event, and the identification of possible subgroups, where such links may arise. We develop an epistemic framework that exploits the joint contribution of different dimensions of evidence and allows one to deal with the reference class problem not only by relying on statistical data about covariances, but also by drawing on causal knowledge. That is, the probability that a given individual will face a given side effect, will probabilistically depend on his characteristics and the plausible causal models in which such features become relevant. The evaluation of the causal models is grounded on the available evidence and theory. (shrink)

Over the last twenty-five years, climate scientists working on the attribution of climate change to humans have developed increasingly sophisticated statistical models in a process that can be understood as a kind of calibration: the gradual changes to the statistical models employed in attribution studies served as iterative revisions to a measurement(-like) procedure motivated primarily by the aim of neutralizing particularly troublesome sources of error or uncertainty. This practice is in keeping with recent work on the evaluation of models more (...) generally that views models as tools for particular tasks: what drives the process is the desire for models that provide more reliable grounds for inference rather than accuracy to the underlying mechanisms of data-generation. (shrink)

Climate scientists often apply statistical tools to a set of different estimates generated by an “ensemble” of models. In this paper, I argue that the resulting inferences are justified in the same way as any other statistical inference: what must be demonstrated is that the statistical model that licenses the inferences accurately represents the probabilistic relationship between data and target. This view of statistical practice is appropriately termed “model-based,” and I examine the use of statistics in climate fingerprinting to show (...) how the difficulties that climate scientists encounter in applying statistics to ensemble-generated data are the practical difficulties of normal statistical practice. The upshot is that whether the application of statistics to ensemble-generated data yields trustworthy results should be expected to vary from case to case. (shrink)

This essay presents a unified account of safety, sensitivity, and severe testing. S’s belief is safe iff, roughly, S could not easily have falsely believed p, and S’s belief is sensitive iff were p false S would not believe p. These two conditions are typically viewed as rivals but, we argue, they instead play symbiotic roles. Safety and sensitivity are both valuable epistemic conditions, and the relevant alternatives framework provides the scaffolding for their mutually supportive roles. The relevant alternatives condition (...) holds that a belief is warranted only if the evidence rules out relevant error possibilities. The safety condition helps categorise relevant from irrelevant possibilities. The sensitivity condition captures ‘ruling out’. Safety, sensitivity, and the relevant alternatives condition are typically presented as conditions on warranted belief or knowledge. But these properties, once generalised, help characterise other epistemic phenomena, including warranted inference, legal verdicts, scientific claims, reaching conclusions, addressing questions, warranted assertion, and the epistemic force of corroborating evidence. We introduce and explain Mayo’s severe testing account of statistical inference. A hypothesis is severely tested to the extent it passes tests that probably would have found errors, were they present. We argue Mayo’s account is fruitfully understood using the resulting relevant alternatives framework. Recasting Mayo’s condition using the conceptual framework of contemporary epistemology helps forge fruitful connections between two research areas—philosophy of statistics and the analysis of knowledge—not currently in sufficient dialogue. The resulting union benefits both research areas. (shrink)

While practitioners think highly of randomized studies, some philosophers argue that there is no epistemic reason to randomize. Here I show that their arguments do not entail their conclusion. Moreover, I provide novel reasons for randomizing in the context of interventional studies. The overall discussion provides a unified framework for assessing baseline balance, one that holds for interventional and observational studies alike. The upshot: practitioners’ strong preference for randomized studies can be defended in some cases, while still offering a nuanced (...) approach to evidence-appraisal, one where not all non-randomized studies are treated equally. (shrink)

In his recent book, John Norton has created a theory of inference to the best explanation, within the context of his "material theory of induction". I apply it to the problem of scientific explanations that are false: if we want the theories in our explanations to be true, then why do historians and scientists often say that false theories explained phenomena? I also defend Norton's theory against some possible objections.

As part of a book symposium on Erwin Dekker's Jan Tinbergen (1903–1994) and the Rise of Economic Expertise (2021), William Peden reflects on shared views on the objectivity and nature of statistics between Tinbergen and Keynes underlying the Tinbergen-Keynes debates.

According to John D. Norton's Material Theory of Induction, all reasonable inductive inferences are justified in virtue of background knowledge about local uniformities in nature. These local uniformities indicate that our samples are likely to be representative of our target population in our inductions. However, a variety of critics have noted that there are many circumstances in which induction seems to be reasonable, yet such background knowledge is apparently absent. I call such an absence of circumstances ‘the frontiers of science', (...) where background scientific theories do not provide information about such local uniformities. I argue that the Material Theory of Induction can be reconciled with our intuitions in favour of these inductions. I adapt an attempted justification of induction in general, the Combinatoric Justification of Induction, into a more modest rationalisation at the less foundational level that the critics discuss. Subject to a number of conditions, we can extrapolate from large samples using our knowledge of facts about the minimum proportions of representative subsets of finite sets. I also discuss some of Norton's own criticisms of his theory and argue that he is overly pessimistic. I conclude that Norton's theory at least performs well at the frontiers of science. (shrink)

Bayesians often appeal to “merging of opinions” to rebut charges of excessive subjectivity. But what happens in the short run is often of greater interest than what happens in the limit. Seidenfeld and coauthors use this observation as motivation for investigating the counterintuitive short run phenomenon of dilation, since, they allege, dilation is “the opposite” of asymptotic merging of opinions. The measure of uncertainty relevant for dilation, however, is not the one relevant for merging of opinions. We explicitly investigate the (...) short run behavior of the metric relevant for merging, and show that dilation is independent of the opposite of merging. (shrink)

Being a researcher is challenging, especially in the beginning. Early Career Researchers (ECRs) need achievements to secure and expand their careers. In today’s academic landscape, researchers are under many pressures: data collection costs, the expectation of novelty, analytical skill requirements, lengthy publishing process, and the overall competitiveness of the career. Innovative thinking and the ability to turn good ideas into good papers are the keys to success.

"Correlation is not causation" is one of the mantras of the sciences—a cautionary warning especially to fields like epidemiology and pharmacology where the seduction of compelling correlations naturally leads to causal hypotheses. The standard view from the epistemology of causation is that to tell whether one correlated variable is causing the other, one needs to intervene on the system—the best sort of intervention being a trial that is both randomized and controlled. In this paper, we argue that some purely correlational (...) data contains information that allows us to draw causal inferences: statistical noise. Methods for extracting causal knowledge from noise provide us with an alternative to randomized controlled trials that allows us to reach causal conclusions from purely correlational data. (shrink)

From 1950 to 1952, statisticians W.G. Cochran, C.F. Mosteller, and J.W. Tukey reviewed A.C. Kinsey and colleagues’ methodology. Neither the history-and-philosophy of science literature nor contemporary theories of interdisciplinarity seem to offer a conceptual model that fits this forced interaction, which was characterized by significant power asymmetries and disagreements on multiple levels. The statisticians initially attempted to exclude all non-technical matters from their evaluation, but their political and personal investments interfered with this agenda. In the face of McCarthy’s witch hunts, (...) negotiations with Kinsey and his funding institutions became integral to the review group’s work. This paper analyzes the heavy burden of emotional and affective labor in this collaboration, the conflicts caused by competing visions of objectivity, and the uses of statistical knowledge to gain and sustain authority. Kinsey’s refusal to adopt the recommended probability sample damaged his already precarious position even further and marked him as a biased researcher who put his personal agenda above methodological rigor. Kinsey’s uncooperative demeanor can be explained by distrust resulting from numerous adverse reactions to his work and by fear of having his sexuality exposed. This case study illustrates that the very concept of valid numbers can become an arena for power struggles and that quantification alone does not guarantee productive exchanges across disciplines. It calls for a deeper conceptual analysis of the prerequisites for successful scientific collaborations. (shrink)

For computer simulation models to usefully inform climate risk management, uncertainties in model projections must be explored and characterized. Because doing so requires running the model many ti...

A prevalent narrative locates the discovery of the statistical phenomenon of regression to the mean in the work of Francis Galton. It is claimed that after 1885, Galton came to explain the fact that offspring deviated less from the mean value of the population than their parents did as a population-level statistical phenomenon and not as the result of the processes of inheritance. Arguing against this claim, we show that Galton did not explain regression towards mediocrity statistically, and did not (...) give up on his ideas regarding an inheritance process that caused offspring to revert to the mean. While the common narrative focuses almost exclusively on Galton’s statistics, our arguments emphasize the anthropological and biological questions that Galton addressed. Galton used regression towards mediocrity to support the claim that some biological types were more stable than others and hence were resistant to evolutionary change. This view had implications concerning both natural selection and eugenics. The statistical explanation attributed to Galton appeared later, during the biometrician-mutationist debate in the early 1900s. It was in the context of this debate and specifically by the biometricians, that the development of the statistical explanation was originally attributed to Galton. (shrink)

In this work, we explore the epistemic import of the value-ladenness of Neyman-Pearson’s Theory of Testing Hypotheses by reconstructing and extending Daniel Steel’s argument for the legitimate influence of pragmatic values on scientific inference. We focus on how to properly understand N-P’s pragmatic value-ladenness and the epistemic reliability of N-P. We develop an account of the twofold influence of pragmatic values on N-P’s epistemic reliability and replicability. We refer to these two distinguished aspects as “direct” and “indirect”. We discuss the (...) replicability of experiments in terms of the indirect aspect and the replicability of outcomes in terms of the direct aspect. We argue that the influence of pragmatic values is beneficial to N-P’s epistemic reliability and replicability indirectly. We show that while the direct influence of pragmatic values can be beneficial, its negative effects on reliability and replicability are also unavoidable in some cases, with the direct and indirect aspects possibly being incongruent. (shrink)

A worldview that does not involve religion or science seems to be incomplete. However, a worldview that includes both religion and science may arouse concern of incompatibility. This paper looks at the particular religion, Christianity, and proceeds to develop a worldview in which Christianity and Science are compatible with each other. The worldview may make use of some ideas of Christianity and may involve some author’s own ideas on Christianity. It is thought that Christianity and Science are in harmony in (...) the sense that science can support beliefs in Christianity and in turn beliefs in Christianity can support science. To avoid future unnecessary conflicts between science and religion, it is suggested that a core faith Christianity worldview should be taken. However, this does not mean that certain parts of scripture are abandoned. (shrink)

The history of statistics is filled with many controversies, in which the prime focus has been the difference in the “interpretation of probability” between Fre- quentist and Bayesian theories. Many philosophical arguments have been elabo- rated to examine the problems of both theories based on this dichotomized view of statistics, including the well-known stopping-rule problem and the catch-all hy- pothesis problem. However, there are also several “hybrid” approaches in theory, practice, and philosophical analysis. This poses many fundamental questions. This paper (...) reviews three cases and argues that the interpretation problem of probabil- ity is insufficient to begin a philosophical analysis of the current issues in the field of statistics. A novel viewpoint is proposed to examine the relationship between the stopping-rule problem and the catch-all hypothesis problem. (shrink)

The origins of testing scientific models with statistical techniques go back to 18th century mathematics. However, the modern theory of statistical testing was primarily developed through the work of Sir R.A. Fisher, Jerzy Neyman, and Egon Pearson in the inter-war period. Some of Fisher's papers on testing were published in economics journals (Fisher, 1923, 1935) and exerted a notable influence on the discipline. The development of econometrics and the rise of quantitative economic models in the mid-20th century made statistical significance (...) testing a commonplace, albeit controversial tool within economics. -/- In the debate about significance testing, methodological controversies intertwine with epistemological issues and sociological developments. Our aim in this chapter is to expound these connections and to show how the use of, and the debate about, significance testing in economics differs from other social sciences, such as psychology. (shrink)

The debates between Bayesian, frequentist, and other methodologies of statistics have tended to focus on conceptual justifications, sociological arguments, or mathematical proofs of their long run properties. Both Bayesian statistics and frequentist (“classical”) statistics have strong cases on these grounds. In this article, we instead approach the debates in the “Statistics Wars” from a largely unexplored angle: simulations of different methodologies’ performance in the short to medium run. We conducted a large number of simulations using a straightforward decision problem based (...) around tossing a coin with unknown bias and then placing bets. In this simulation, we programmed four players, inspired by Bayesian statistics, frequentist statistics, Jon Williamson’s version of Objective Bayesianism, and a player who simply extrapolates from observed frequencies to general frequencies. The last player functions as a benchmark: a statistical methodology should at least outperform a crude form of induction. We focused on the performance of these methodologies in guiding the players towards good decisions. Unlike an earlier simulation study of this type, we found no systematic difference in performance between the Bayesian and frequentist players, provided the Bayesian used a flat prior and the frequentist used a low confidence level. Unlike that study, we were able to use Big Data methods to mitigate problems of random error in the simulation results. The Williamsonian player, who is a novel element of our study, also had no systematic differences in their performance, provided that they use a low confidence level. These players performed similarly even in the very short run, when players were making different decisions. Our study indicates that all three methodologies should be taken seriously by philosophers and practitioners of statistics. However, the frequentist and Williamsonian performed poorly when their confidence levels were high, and the Bayesian was surprisingly harmed by biased priors, providing some unexpected lessons for these methodologies when facing this type of decision problem. (shrink)

Ian Hacking (born in 1936, Vancouver, British Columbia) is most well-known for his work in the philosophy of the natural and social sciences, but his contributions to philosophy are broad, spanning many areas and traditions. In his detailed case studies of the development of probabilistic and statistical reasoning, Hacking pioneered the naturalistic approach in the philosophy of science. Hacking’s research on social constructionism, transient mental illnesses, and the looping effect of the human kinds make use of historical materials to shed (...) light on how developments in the social, medical, and behavioural sciences have shaped our contemporary conceptions of identity and agency. Hacking’s other contributions to philosophy include his work on the philosophy of mathematics (Hacking, 2014), philosophy of statistics, philosophy of logic, inductive logic (Hacking, 1965, 1979, 2001) and natural kinds (Hacking, 1991, 2007a). (shrink)

My aim in this paper is to show how the problem of inflated effect sizes corrupts the severity measure of evidence. This has never been done. In fact, the Winner’s Curse is barely mentioned in the philosophical literature. Since the severity score is the predominant measure of evidence for frequentist tests in the philosophical literature, it is important to underscore its flaws. It is also crucial to bring the philosophical literature up to speed with the limits of classical testing. The (...) Winner’s Curse is one of them. The problem is that when a significant result is obtained by using an underpowered test, the severity score becomes particularly high for large discrepancies from the null-hypothesis. This means that such discrepancies are very well supported by the evidence according to that measure. However, it is now well documented that significant tests with low power display inflated effect sizes. They systematically show departures from the null hypothesis H0 that are much greater than they really are. From an epistemological point of view this means that a significant result produced by an underpowered test does not provide evidence for large discrepancies from H0. Therefore, the severity score is an inadequate measure of evidence. Given that we are now aware of the phenomenon of inflated effect sizes, it would be irresponsible to rely on the severity score to measure the strength of the evidence against the null. Instead, one must take appropriate measures to try and avoid using underpowered tests by setting a threshold for the sample size or by replicating the results of the experiment. (shrink)

Scientists often adjust their significance threshold during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha (...) adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses. (shrink)

The popular science of phrenology is known for its preoccupation with geniuses and criminals, but this article shows that phrenologists also introduced ideas about the ‘average’ person. Popular phrenologists in the US and the UK examined the heads of their clients to give an indication of their character. Based on the publications of phrenologists and on a large collection of standardized charts with clients’ scores, this article analyses their definition of what they considered to be the ‘average’. It can be (...) concluded that phrenologists were some of the first to teach individuals to see their identity in relation to an imagined statistical community. (shrink)

The social sciences face a problem of sample non-representation, where the majority of samples consist of undergraduate students from Euro-American institutions. The problem has been identified for decades with little trend of improvement. In this paper, I trace the history of sampling theory. The dominant framework, called the design-based approach, takes random sampling as the gold standard. The idea is that a sampling procedure that is maximally uninformative prevents samplers from introducing arbitrary bias, thus preserving sample representation. I show how (...) this framework, while good in theory, faces many challenges in application. Instead, I advocate for an alternative framework, called the model-based approach to sampling, where representative samples are those balanced in composition, however they were drawn. I argue that the model-based framework is more appropriate in the social sciences because it allows for systematic assessment of imperfect samples and methodical improvement in resource-limited scientific contexts. I end with practical proposals of improving sample quality in the social sciences. (shrink)

We address the need for a model by considering two competing theories regarding the origin of life: (i) the Metabolism First theory, and (ii) the RNA World theory. We discuss two interrelated points, namely: (i) Models are valuable tools for understanding both the processes and intricacies of origin-of-life issues, and (ii) Insights from models also help us to evaluate the core objection to origin-of-life theories, called “the inefficiency objection”, which is commonly raised by proponents of both the Metabolism First theory (...) and the RNA World theory against each other. We use Simpson’s Paradox (SP) as a tool for challenging this objection. We will use models in various senses, ranging from taking them as representations of reality to treating them as theories/accounts that provide heuristics for probing reality. In this paper, we will frequently use models and theories interchangeably. Additionally, we investigate Conway’s Game of Life and contrast it with our SP-based approach to emergence-of-life issues. Finally, we discuss some of the consequences of our view. A scientific model is testable in three senses: (i) a logical sense, (ii) a nomological sense, and (iii) a current technological sense. The SP-based model is testable in the first two senses but it is not feasible to test it using current technology. (shrink)

How we can reliably draw inferences from data, evidence and/or experience has been and continues to be a pressing question in everyday life, the sciences, politics and a number of branches in philosophy (traditional epistemology, social epistemology, formal epistemology, logic and philosophy of the sciences). In a world in which we can now longer fully rely on our experiences, interlocutors, measurement instruments, data collection and storage systems and even news outlets to draw reliable inferences, the issue becomes even more pressing. (...) While we were working on this question using a formal epistemology approach Landes and Osimani (2020); De Pretis et al. (2019); Osimani and Landes (2020); Osimani (2020), we realised that the width of current active interests in the notion of reliability was much broader than we initially thought. Given the breadth of approaches and angles present in philosophy (even in this journal Schubert 2012; Avigad 2021; Claveau and Grenier 2019; Kummerfeld and Danks 2014; Landes 2021; Trpin et al. 2021; Schippers 2014; Schindler 2011; Kelly et al. 2016; Mayo-Wilson 2014; Olsson and Schubert 2007; Pittard 2017), we thought that it would be beneficial to provide a forum for an open exchange of ideas, in which philosophers working in different paradigms could come together. Our call for expression of interest received a great variety of promised manuscripts, the variety is reflected in the published papers. They range from fields far away from our own interests such as quantum probabilities de Ronde et al. (2021), evolvable software systems Primiero et al. (2021), to topics closer to our own research in the philosophy of medicine Lalumera et al. (2020), psychology Dutilh et al. (2021), traditional epistemology Dunn (2021); Tolly (2021) to finally close shared interests in formal epistemology Romero and Sprenger (2021) even within our own department Merdes et al. (2021). Our job is now to reliably inform you about all the contributions in the papers in this special issue. Unfortunately, that task is beyond our capabilities. What we can and instead do is to summarise the contributed papers to inform your reliable inference to read them all in great detail. (shrink)

A brief idea on statistics and epistemology.

In statistics, there are two main paradigms: classical and Bayesian statistics. The purpose of this article is to investigate the extent to which classicists and Bayesians can agree. My conclusion is that, in certain situations, they cannot. The upshot is that, if we assume that the classicist is not allowed to have a higher degree of belief in a null hypothesis after he has rejected it than before, then he has to either have trivial or incoherent credences to begin with (...) or fail to update his credences by conditionalization. (shrink)

In this essay we examine the relationships between Big Data and cognition, in particular human cognition. The reason for exploring such relationships lies in two aspects. First, because in the domain of cognitive science, many speculate about the benefits that the uses of Big Data analysis techniques can provide to the characterization and understanding of cognition. Secondly, because the scientific and technological sectors that promote data analysis activities, particularly statistics, computer science and data science, naturally accustomed to working with Big (...) Data, have in recent years used the idea of ​​cognition. and its related concepts as terms that seem capable of remedying the deficiencies of the era of symbolic artificial intelligence, as a response to the automation of typically human activities in the process of analyzing data and using its results. of data science, seeking to understand what Big Data really is and to investigate whether the use of cognition models could be applied to the sector. On the other hand, we will analyze the potential of using Big Data to help cognitive science. We also tried to provide a certain didactic character in some points, aiming to help the understanding, by non-specialists, regarding some principles and aspects of the methodology of data science and statistics that will eventually be mentioned in the development of our analysis. (Translated from original). (shrink)

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are (...) typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study. (shrink)

Fisher criticised the Neyman-Pearson approach to hypothesis testing by arguing that it relies on the assumption of “repeated sampling from the same population.” The present article considers the responses to this criticism provided by Pearson and Neyman. Pearson interpreted alpha levels in relation to imaginary replications of the original test. This interpretation is appropriate when test users are sure that their replications will be equivalent to one another. However, by definition, scientific researchers do not possess sufficient knowledge about the relevant (...) and irrelevant aspects of their tests and populations to be sure that their replications will be equivalent to one another. Pearson also interpreted the alpha level as a personal rule that guides researchers’ behavior during hypothesis testing. However, this interpretation fails to acknowledge that the same researcher may use different alpha levels in different testing situations. Addressing this problem, Neyman proposed that the average alpha level adopted by a particular researcher can be viewed as an indicator of that researcher’s typical Type I error rate. Researchers’ average alpha levels may be informative from a metascientific perspective. However, they are not useful from a scientific perspective. Scientists are more concerned with the error rates of specific tests of specific hypotheses, rather than the error rates of their colleagues. It is concluded that neither Neyman nor Pearson adequately rebutted Fisher’s “repeated sampling” criticism. Fisher’s significance testing approach is briefly considered as an alternative to the Neyman-Pearson approach. (shrink)

Why are conditional degrees of belief in an observation E, given a statistical hypothesis H, aligned with the objective probabilities expressed by H? After showing that standard replies are not satisfactory, I develop a suppositional analysis of conditional degree of belief, transferring Ramsey’s classical proposal to statistical inference. The analysis saves the alignment, explains the role of chance-credence coordination, and rebuts the charge of arbitrary assessment of evidence in Bayesian inference. Finally, I explore the implications of this analysis for Bayesian (...) reasoning with idealized models in science. (shrink)