Unfortunately, reading Chow's work is likely to leave the reader more confused than enlightened. My preferred solutions to the “controversy” about null- hypothesis testing are: (1) recognize that we really want to test the hypothesis that an effect is “small,” not null, and (2) use Bayesian methods, which are much more in keeping with the way humans naturally think than are classical statistical methods.

Although many researchers may perceive empirical hypothesis testing using inferential statistics to be a value free process, I argue that any conclusion based on inferential statistics contains an important and intractable value judgment. Consequently, I conclude that researchers should use the same rationale for examining the ethical ramifications of committing errors in statistical inference that they use to examine the ethical parameters of a proposed research design.

Although logical consistency is desirable in scientific research, standard statistical hypothesis tests are typically logically inconsistent. To address this issue, previous work introduced agnostic hypothesis tests and proved that they can be logically consistent while retaining statistical optimality properties. This article characterizes the credal modalities in agnostic hypothesis tests and uses the hexagon of oppositions to explain the logical relations between these modalities. Geometric solids that are composed of hexagons of oppositions illustrate the conditions for (...) these modalities to be logically consistent. Prisms composed of hexagons of oppositions show how the credal modalities obtained from two agnostic tests vary according to their threshold values. Nested hexagons of oppositions summarize logical relations between the credal modalities in these tests and prove new relations. (shrink)

Simultaneous hypothesis tests can fail to provide results that meet logical requirements. For example, if A and B are two statements such that A implies B, there exist tests that, based on the same data, reject B but not A. Such outcomes are generally inconvenient to statisticians (who want to communicate the results to practitioners in a simple fashion) and non-statisticians (confused by conflicting pieces of information). Based on this inconvenience, one might want to use tests that satisfy logical (...) requirements. However, Izbicki and Esteves shows that the only tests that are in accordance with three logical requirements (monotonicity, invertibility and consonance) are trivial tests based on point estimation, which generally lack statistical optimality. As a possible solution to this dilemma, this paper adapts the above logical requirements to agnostic tests, in which one can accept, reject or remain agnostic with respect to a given hypothesis. Each of the logical requirements is characterized in terms of a Bayesian decision theoretic perspective. Contrary to the results obtained for regular hypothesis tests, there exist agnostic tests that satisfy all logical requirements and also perform well statistically. In particular, agnostic tests that fulfill all logical requirements are characterized as region estimator-based tests. Examples of such tests are provided. (shrink)

“Dutch Book” arguments and references to gambling theorems are typical in the debate between Bayesians and scientists committed to “classical” statistical methods. These arguments have rarely convinced non-Bayesian scientists to abandon certain conventional practices, partially because many scientists feel that gambling theorems have little relevance to their research activities. In other words, scientists “don’t bet.” This article examines one attempt, by Schervish, Seidenfeld, and Kadane, to progress beyond such apparent stalemates by connecting “Dutch Book”–type mathematical results with principles actually (...) endorsed by practicing experimentalists. (shrink)

This paper addresses the role of specification tests in the selection of a statistically admissible model used to evaluate economic hypotheses. The issue is formulated in the context of recent philosophical accounts on the nature of models and related to some results in the literature on specification search. In contrast to enumerative induction and a priori theory, powerful search methodologies are often adequate substitutes for experimental methods. They underwrite and support, rather than distort, statistical hypothesis tests. Their success (...) is grounded in a systematic effort to mold appropriate models to, and test them against, constraints of the data. (shrink)

In the past, hypothesis testing in medicine has employed the paradigm of the repeatable experiment. In statistical hypothesis testing, an unbiased sample is drawn from a larger source population, and a calculated statistic is compared to a preassigned critical region, on the assumption that the comparison could be repeated an indefinite number of times. However, repeated experiments often cannot be performed on human beings, due to ethical or economic constraints. We describe a new paradigm for (...) hypothesis testing which uses only rearrangements of data present within the observed data set. The token swap test, based on this new paradigm, is applied to three data sets from cardiovascular pathology, and computational experiments suggest that the token swap test satisfies the Neyman Pearson condition. (shrink)

We show that if among the tested hypotheses the number of true hypotheses is not equal to the number of false hypotheses, then Neyman-Pearson theory of testing hypotheses does not warrant minimal epistemic reliability. We also argue that N-P does not protect from the possible negative effects of the pragmatic value-laden unequal setting of error probabilities on N-P’s epistemic reliability. Most importantly, we argue that in the case of a negative impact no methodological adjustment is available to neutralize it, (...) so in such cases the discussed pragmatic-value-ladenness of N-P inevitably compromises the goal of attaining truth. (shrink)

Chow's endorsement of a limited role for null hypothesis significance testing is a needed corrective of research malpractice, but his decision to place this procedure in a hypothetico-deductive framework of Popperian cast is unwise. Various failures of this version of the hypothetico-deductive method have negative implications for Chow's treatment of significance testing, meta-analysis, and theory evaluation.

Professor Meehl [2] has pointed out a very significant problem in the methodology of psychological research, indicating that statistical tests of psychological hypotheses against a null hypothesis are loaded in favor of eventual success at rejecting the null hypothesis. In my opinion this is not, however, a contrast between physics and psychology, but rather between the method of parameter estimation and that of the null hypothesis in the tradition of Fisher. A physicist could use the null (...) hypothesis method as well as the psychologist. The fact that he doesn't is probably related to the more advanced state of his measurement techniques and theoretical constructs. (shrink)

In this article I give a critical evaluation of the use and limitations of null-model-based hypothesis testing as a research strategy in the biological sciences. According to this strategy, the null model based on a randomization procedure provides an appropriate null hypothesis stating that the existence of a pattern is the result of random processes or can be expected by chance alone, and proponents of other hypotheses should first try to reject this null hypothesis in order (...) to demonstrate their own hypotheses. Using as an example the controversy over the use of null hypotheses and null models in species co-occurrence studies, I argue that null-model-based hypothesis testing fails to work as a proper analog to traditional statistical null-hypothesis testing as used in well-controlled experimental research, and that the random process hypothesis should not be privileged as a null hypothesis. Instead, the possible use of the null model resides in its role of providing a way to challenge scientists’ commonsense judgments about how a seemingly unusual pattern could have come to be. Despite this possible use, null-model-based hypothesis testing still carries certain limitations, and it should not be regarded as an obligation for biologists who are interested in explaining patterns in nature to first conduct such a test before pursuing their own hypotheses. (shrink)

Evidence-based medicine frequently uses statistical hypothesis testing. In this paradigm, data can only disconfirm a research hypothesis’ competitors: One tests the negation of a statistical hypothesis that is supposed to correspond to the research hypothesis. In practice, these hypotheses are often misaligned. For instance, directional research hypotheses are often paired with non-directional statistical hypotheses. Prima facie, one cannot gain proper evidence for one’s research hypothesis employing a misaligned statistical hypothesis. (...) This paper sheds lights on the nature of and the reasons for such misalignments and it provides a thorough analysis of whether they pose a threat to evidence-based medicine. The upshots are that the misalignments are often hidden for clinicians and that although some cases of misalignments can be partially counterbalanced, the overall threat is non-negligible. The counterbalances either lead to methodological inadequacy, loss of statistical power, or involve a lack of information that could be crucial for decision making. This result casts doubt on various findings of medical studies in addition to issues associated with under-powered studies or the replication crisis. (shrink)

Several of Krueger & Funder's (K&F's) suggestions may promote more balanced social cognition research, but reconsidered null hypothesis statistical testing (NHST) is not one of them. Although NHST has primarily supported negative conclusions, this is simply because most conclusions have been negative. NHST can support positive, negative, and even balanced conclusions. Better NHST practices would benefit psychology, but would not alter the balance between positive and negative approaches.

Asking and answering questions is the cornerstone of science yet formal training in understanding this key process is often overlooked. "Asking Questions in Biology" unpacks this crucial process of enquiry, from a biological perspective, at its various stages. It begins with an overview of scientific question-asking in general, before moving on to demonstrate how to derive hypotheses from unstructured observations. It then explains in the main sections of the book, how to use statistical tests as tools to analyse data (...) and answer those questions before, finally, showing the best practice in presenting scientific reports. As such, it is an indispensable companion to all students of biology, but particularly those enrolled in courses concerning experimental design; data analysis; hypothesis testing; research methods; or any practical project work. The late Chris Barnard was Professor of Animal Behaviour at Nottingham University. Francis Gilbert is Associate Professor of Ecology at Nottingham University. Peter McGregor isa Reader in Applied Zoologyat Cornwall College. ". (shrink)

The interpretation of tests of a point null hypothesis against an unspecified alternative is a classical and yet unresolved issue in statistical methodology. This paper approaches the problem from the perspective of Lindley's Paradox: the divergence of Bayesian and frequentist inference in hypothesis tests with large sample size. I contend that the standard approaches in both frameworks fail to resolve the paradox. As an alternative, I suggest the Bayesian Reference Criterion: it targets the predictive performance of the (...) null hypothesis in future experiments; it provides a proper decision-theoretic model for testing a point null hypothesis and it convincingly accounts for Lindley's Paradox. (shrink)

The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null (...) class='Hi'>hypothesis is never used as a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the apriori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics. (shrink)

Chow's book provides a thorough analysis of the confusing array of issues surrounding conventional tests of statistical significance. This book should be required reading for behavioral and social scientists. Chow concludes that the null-hypothesis significance-testing procedure (NHSTP) plays a limited, but necessary, role in the experimental sciences. Another possibility is that – owing in part to its metaphorical underpinnings and convoluted logic – the NHSTP is declining in importance in those few sciences in which it ever played (...) a role. (shrink)

Setting optimal significance levels that minimize Type I and Type II errors allows for more transparent and well‐considered statistical decision making compared to the traditional α = 0.05 significance level. We use the optimal α approach to re‐assess conclusions reached by three recently published tests of the pace‐of‐life syndrome hypothesis, which attempts to unify occurrences of different physiological, behavioral, and life history characteristics under one theory, over different scales of biological organization. While some of the conclusions reached using (...) optimal α were consistent to those previously reported using the traditional α = 0.05 threshold, opposing conclusions were also frequently reached. The optimal α approach reduced probabilities of Type I and Type II errors, and ensured statistical significance was associated with biological relevance. Biologists should seriously consider their choice of α when conducting null hypothesis significance tests, as there are serious disadvantages with consistent reliance on the traditional but arbitrary α = 0.05 significance level. (shrink)

Gignac and Zajenkowski (2020) find that “the degree to which people mispredicted their objectively measured intelligence was equal across the whole spectrum of objectively measured intelligence”. This Comment shows that Gignac and Zajenkowski’s (2020) finding of homoscedasticity is likely the result of a recoding choice by the experimenters and does not in fact indicate that the Dunning-Kruger Effect is a mere statistical artifact. Specifically, Gignac and Zajenkowski (2020) recoded test subjects’ responses to a question regarding self-assessed comparative IQ onto (...) a linear IQ scale when a normal IQ scale would likely have been more appropriate. More generally, researchers studying self-assessed intelligence should be aware of potential measurement problems that may arise when transforming an ordinal scale onto an interval scale. (shrink)

A pure significance test would check the agreement of a statistical model with the observed data even when no alternative model was available. The paper proposes the use of a modified p -value to make such a test. The model will be rejected if something surprising is observed. It is shown that the relation between this measure of surprise and the surprise indices of Weaver and Good is similar to the relationship between a p -value, a corresponding odds-ratio, and (...) a logit or log-odds statistic. The s -value is always larger than the corresponding p -value, and is not uniformly distributed. Difficulties with the whole approach are discussed. (shrink)

To estimate causal relationships, time series econometricians must be aware of spurious correlation, a problem first mentioned by Yule (1926). To deal with this problem, one can work either with differenced series or multivariate models: VAR (VEC or VECM) models. These models usually include at least one cointegration relation. Although the Bayesian literature on VAR/VEC is quite advanced, Bauwens et al. (1999) highlighted that “the topic of selecting the cointegrating rank has not yet given very useful and convincing results”. The (...) present article applies the Full Bayesian Significance Test (FBST), especially designed to deal with sharp hypotheses, to cointegration rank selection tests in VECM time series models. It shows the FBST implementation using both simulated and available (in the literature) data sets. As illustration, standard non informative priors are used. (shrink)

The explanatory filter is a proposed method to detect design in nature with the aim of refuting Darwinian evolution. The explanatory filter borrows its logical structure from the theory of statistical hypothesis testing but we argue that, when viewed within this context, the filter runs into serious trouble in any interesting biological application. Although the explanatory filter has been extensively criticized from many angles, we present the first rigorous criticism based on the theory of mathematical statistics.

The first of 3 objectives in this study was to address the major problem with Null Hypothesis Significance Testing (NHST) and 2 common misconceptions related to NHST that cause confusion for students and researchers. The misconcep- tions are (a) a smaller p indicates a stronger relationship and (b) statistical signifi- cance indicates practical importance. The second objective was to determine how this problem and the misconceptions were treated in 12 recent textbooks used in edu- cation research methods (...) and statistics classes. The third objective was to examine how the textbooks’ presentations relate to current best practices and how much help they provide for students. The results show that almost all of the textbooks fail to acknowledge that there is controversy surrounding NHST. Most of the textbooks dealt, at least minimally, with the alleged misconceptions of interest, but they pro- vided relatively little help for students. (shrink)

“Evidence” in the form of data collected and analysis thereof is fundamental to medicine, health and science. In this paper, we discuss the “evidence-based” aspect of evidence-based medicine in terms of statistical inference, acknowledging that this latter field of statistical inference often also goes by various near-synonymous names—such as inductive inference (amongst philosophers), econometrics (amongst economists), machine learning (amongst computer scientists) and, in more recent times, data mining (in some circles). Three central issues to this discussion of “evidence-based” (...) are (i) whether or not the statistical analysis can and/or should be objective and/or whether or not (subjective) prior knowledge can and/or should be incorporated, (ii) whether or not the analysis should be invariant to the framing of the problem (e.g. does it matter whether we analyse the ratio of proportions of morbidity to non-morbidity rather than simply the proportion of morbidity?), and (iii) whether or not, as we get more and more data, our analysis should be able to converge arbitrarily closely to the process which is generating our observed data. For many problems of data analysis, it would appear that desiderata (ii) and (iii) above require us to invoke at least some form of subjective (Bayesian) prior knowledge. This sits uncomfortably with the understandable but perhaps impossible desire of many medical publications that at least all the statistical hypothesis testing has to be classical non-Bayesian—i.e. it is not permitted to use any (subjective) prior knowledge. (shrink)

In this paper, the notion of degree of inconsistency is introduced as a tool to evaluate the sensitivity of the Full Bayesian Significance Test (FBST) value of evidence with respect to changes in the prior or reference density. For that, both the definition of the FBST, a possibilistic approach to hypothesis testing based on Bayesian probability procedures, and the use of bilattice structures, as introduced by Ginsberg and Fitting, in paraconsistent logics, are reviewed. The computational and theoretical advantages (...) of using the proposed degree of inconsistency based sensitivity evaluation as an alternative to traditional statistical power analysis is also discussed. (shrink)

Statistical learning enables humans to involuntarily process and utilize different kinds of patterns from the environment. However, the cognitive mechanisms underlying the simultaneous acquisition of multiple regularities from different perceptual modalities remain unclear. A novel multidimensional serial reaction time task was developed to test 40 participants’ ability to learn simple first‐order and complex second‐order relations between uni‐modal visual and cross‐modal audio‐visual stimuli. Using the difference in reaction times between sequenced and random stimuli as the index of domain‐general statistical (...) learning, a significant difference and dissociation of learning occurred between the initial and final learning phases. Furthermore, we used a negative and positive occurrence‐frequency‐and‐reaction‐time correlation to indicate implicit and explicit learning, respectively, and found that learning simple uni‐modal patterns involved an implicit‐to‐explicit segue, while acquiring complex cross‐modal patterns required an explicit‐to‐implicit segue, resulting in a X‐shape crossing of regularity learning. Thus, we propose an X‐way hypothesis to elucidate the dynamic interplay between the implicit and explicit systems at two distinct stages when acquiring various regularities in a multidimensional probability space. (shrink)

This paper introduces pragmatic hypotheses and relates this concept to the spiral of scientific evolution. Previous works determined a characterization of logically consistent statistical hypothesis tests and showed that the modal operators obtained from this test can be represented in the hexagon of oppositions. However, despite the importance of precise hypothesis in science, they cannot be accepted by logically consistent tests. Here, we show that this dilemma can be overcome by the use of pragmatic versions of precise (...) hypotheses. These pragmatic versions allow a level of imprecision in the hypothesis that is small relative to other experimental conditions. The introduction of pragmatic hypotheses allows the evolution of scientific theories based on statistical hypothesis testing to be interpreted using the narratological structure of hexagonal spirals, as defined by Pierre Gallais. (shrink)

Statistical Default Logic (SDL) is an expansion of classical (i.e., Reiter) default logic that allows us to model common inference patterns found in standard inferential statistics, e.g., hypothesis testing and the estimation of a population‘s mean, variance and proportions. This paper presents an embedding of an important subset of SDL theories, called literal statistical default theories, into stable model semantics. The embedding is designed to compute the signature set of literals that uniquely distinguishes each extension on (...) a statistical default theory at a pre-assigned error-bound probability. (shrink)

I consider the error-statistical account as both a theory of evidence and as a theory of inference. I seek to show how inferences regarding the truth of hypotheses can be upheld by avoiding a certain kind of alternative hypothesis problem. In addition to the testing of assumptions behind the experimental model, I discuss the role of judgments of implausibility. A benefit of my analysis is that it reveals a continuity in the application of error-statistical assessment to (...) low-level empirical hypotheses and highly general theoretical principles. This last point is illustrated with a brief sketch of the issues involved in the parametric framework analysis of tests of physical theories such as General Relativity and of fundamental physical principles such as the Einstein Equivalence Principle. (shrink)

Despite frequent calls for the overhaul of null hypothesis significance testing (NHST), this controversial procedure remains ubiquitous in behavioral, social and biomedical teaching and research. Little change seems possible once the procedure becomes well ingrained in the minds and current practice of researchers; thus, the optimal opportunity for such change is at the time the procedure is taught, be this at undergraduate or at postgraduate levels. This paper presents a tutorial for the teaching of data testing procedures, (...) often referred to as hypothesis testing theories. The first procedure introduced is Fisher's approach to data testing—tests of significance; the second is Neyman-Pearson's approach—tests of acceptance; the final procedure is the incongruent combination of the previous two theories into the current approach—NSHT. For those researchers sticking with the latter, two compromise solutions on how to improve NHST conclude the tutorial. (shrink)

In this paper, I offer a new defense of scientific realism, tailored for the Akaikean paradigm of statistical hypothesis testing. After proposing definitions of verisimilitude and predictive success, I use computer simulations to show how the latter depends on the former, even in the kind of case featured in a recent argument for instrumentalism.

The study of social justice asks: what sorts of social arrangements are equitable ones? But also: how do we derive the inequitable arrangements we often observe in human societies? In particular, in spite of explicitly stated equity norms, categorical inequity tends to be the rule rather than the exception. The cultural Red King hypothesis predicts that differentials in group size may lead to inequitable outcomes for minority groups even in the absence of explicit or implicit bias. We test this (...) prediction in an experimental context where subjects divided into groups engage in repeated play of a bargaining game. We ran 14 trials involving a total of 112 participants. The results of the experiments are statistically significant and suggestive: individuals in minority groups in these experiments end up receiving fewer resources than those in majority groups. Combined with previous theoretical findings, these results give some reason to think that the cultural Red King may occur in real human groups. (shrink)

Entertaining diverse assumptions about empirical research, commentators give a wide range of verdicts on the NHSTP defence in Statistical significance. The null-hypothesis significance- test procedure is defended in a framework in which deductive and inductive rules are deployed in theory corroboration in the spirit of Popper's Conjectures and refutations. The defensible hypothetico-deductive structure of the framework is used to make explicit the distinctions between substantive and statistical hypotheses, statistical alternative and conceptual alternative hypotheses, and making (...) class='Hi'>statistical decisions and drawing theoretical conclusions. These distinctions make it easier to show that H0 can be true, the effect size is irrelevant to theory corroboration, and “strong” hypotheses make no difference to NHSTP. Reservations about statistical power, meta-analysis, and the Bayesian approach are still warranted. (shrink)

This study investigated visual statistical learning (VSL) in 24 deaf signers and 24 hearing non‐signers. Previous research with hearing individuals suggests that SL mechanisms support literacy. Our first goal was to assess whether VSL was associated with reading ability in deaf individuals, and whether this relation was sustained by a link between VSL and sign language skill. Our second goal was to test the Auditory Scaffolding Hypothesis, which makes the prediction that deaf people should be impaired in sequential (...) processing tasks. For the VSL task, we adopted a modified version of the triplet learning paradigm, with stimuli presented sequentially across space and time. Results revealed that measures of sign language skill (sentence comprehension/repetition) did not correlate with VSL scores, possibly due to the sequential nature of our VSL task. Reading comprehension scores (PIAT‐R) were a significant predictor of VSL accuracy in hearing but not deaf people. This finding might be due to the sequential nature of the VSL task and to a less salient role of the sequential orthography‐to‐phonology mapping in deaf readers compared to hearing readers. The two groups did not differ in VSL scores. However, when reading ability was taken into account, VSL scores were higher for the deaf group than the hearing group. Overall, this evidence is inconsistent with the Auditory Scaffolding Hypothesis, suggesting that humans can develop efficient sequencing abilities even in the absence of sound. (shrink)

Conventional discussion of research methodology contrast two approaches, the quantitative and the qualitative, presented as collectively exhaustive. But if qualitative is taken as the understanding of lifeworlds, the two approaches between them cover only a tiny fraction of research methodologies; and the quantitative, taken as the routine application to controlled experiments of frequentist statistics by way of the Null Hypothesis Significance Testing Procedure, is seriously flawed. It is contrary to the advice both of Fisher and of Neyman and (...) Pearson, the two sources from which it is drawn. (shrink)

Conventional discussion of research methodology contrast two approaches, the quantitative and the qualitative, presented as collectively exhaustive. But if qualitative is taken as the understanding of lifeworlds, the two approaches between them cover only a tiny fraction of research methodologies; and the quantitative, taken as the routine application to controlled experiments of frequentist statistics by way of the Null Hypothesis Significance Testing Procedure, is seriously flawed. It is contrary to the advice both of Fisher and of Neyman and (...) Pearson, the two sources from which it is drawn. (shrink)

Researchers in the social and behavioral sciences often have clear expectations about the order/direction of the parameters in their statistical model. For example, a researcher might expect that regression coefficient β 1 is larger than β 2 and β 3. The corresponding hypothesis is H : β 1 > {β 2, β 3 } and this is known as an (order) constrained hypothesis. A major advantage of testing such a hypothesis is that power can be (...) gained and inherently a smaller sample size is needed. This article discusses this gain in sample size reduction, when an increasing number of constraints is included into the hypothesis. The main goal is to present sample-size tables for constrained hypotheses. A sample-size table contains the necessary sample-size at a pre-specified power (say, 0.80) for an increasing number of constraints. To obtain sample-size tables, two Monte Carlo simulations were performed, one for ANOVA and one for multiple regression. Three results are salient. First, in an ANOVA the needed sample-size decreases with 30–50% when complete ordering of the parameters is taken into account. Second, small deviations from the imposed order have only a minor impact on the power. Third, at the maximum number of constraints, the linear regression results are comparable with the ANOVA results. However, in the case of fewer constraints, ordering the parameters (e.g., β 1 > β 2 ) results in a higher power than assigning a positive or a negative sign to the parameters (e.g., β 1 > 0). (shrink)

The Full Bayesian Significance Test (FBST) for precise hypotheses is presented, with some applications relevant to reliability theory. The FBST is an alternative to significance tests or, equivalently, to p-ualue.s. In the FBST we compute the evidence of the precise hypothesis. This evidence is the probability of the complement of a credible set "tangent" to the sub-manifold (of the para,rreter space) that defines the null hypothesis. We use the FBST in an application requiring a quality control of used (...) components, based on remaining life statistics. (shrink)

Attempts by 20th century philosophers of science to define inductive concepts and methods concerning the support provided to scientific theories by empirical data have been unsuccessful. Although 20th century philosophers of science largely ignored statistical methods for testing theories, when they did address them they argued against rather than for their use. In contrast, this study demonstrates that traditional statistical methods used for validating computer simulation models provide tests of the scientific theories that those models may embody. (...) This study shows that these statistical methods are applicable regardless of whether the scientific theory or model is deterministic, stochastic, or chaos-based in nature. Traditional statistical methods, such as hypothesis testing, test theories by providing measures of their goodness-of-fit with the portion of the world the theories describe. Nothing more is needed to portray the agreement between theory and the world. Furthermore, the statistical measures of goodness-of-fit are not ampliative inductive measures. Therefore, the ultimate value and direction of the 20 th century ampliative inductive program within philosophy of science is thrown into question. This study also suggests that advanced statistical, visualization, and hybrid quantitative qualitative methods hold promise for offering scientists additional methods for testing scientific theories through validating their computer model representations. As a prerequisite to the above discussions, this study proves that scientific theories, when applied to concrete physical systems, can indeed be represented by computer models. (shrink)

After some general remarks about the interrelation between philosophical and statistical thinking, the discussion centres largely on significance tests. These are defined as the calculation of p-values rather than as formal procedures for ‘acceptance‘ and ‘rejection‘. A number of types of null hypothesis are described and a principle for evidential interpretation set out governing the implications of p- values in the specific circumstances of each application, as contrasted with a long-run interpretation. A number of more complicated situ- ations (...) are discussed in which modification of the simple p-value may be essential. (shrink)

Several authors, e.g. Patrick Suppes and I. J. Good, have recently argued that the paradox of confirmation can be resolved within the developing subjective Bayesian account of inductive reasoning. The aim of this paper is to show that the paradox can also be resolved by the rival orthodox account of hypothesis testing currently employed by most statisticians and scientists. The key to the orthodox statistical resolution is the rejection of a generalized version of Hempel's instantiation condition, namely, (...) the condition that a PQ is inductively relevant to the hypothesis $(x)(Px\supset Qx)$ even in the absence of all further information. Though their reasons differ, it turns out that Bayesian and orthodox statisticians agree that this condition lies at the heart of the paradox. (shrink)

The unit root problem plays a central role in empirical applications in the time series econometric literature. However, significance tests developed under the frequentist tradition present various conceptual problems that jeopardize the power of these tests, especially for small samples. Bayesian alternatives, although having interesting interpretations and being precisely defined, experience problems due to the fact that that the hypothesis of interest in this case is sharp or precise. The Bayesian significance test used in this article, for the unit (...) root hypothesis, is based solely on the posterior density function, without the need of imposing positive probabilities to sets of zero Lebesgue measure. Furthermore, it is conducted under strict observance of the likelihood principle. It was designed mainly for testing sharp null hypotheses and it is called FBST for Full Bayesian Significance Test. (shrink)

In both theoretical and applied modeling in behavioral sciences, it is common to choose a mathematical specification of functional form and distribution of unobservables on grounds of analytic convenience without support from explicit theoretical postulates. This article discusses the issue of deriving particular qualitative hypotheses about functional form restrictions in structural models from intuitive theoretical axioms. In particular, we focus on a family of postulates known as dimensional invariance. Subsequently, we discuss how specific qualitative postulates can be reformulated so as (...) to allow for conventional statistical hypothesis testing, and we also derive details of a particular likelihood ratio test procedure. An empirical application of the testing procedure is carried out, based on data from a Stated Preference survey. (shrink)

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “H1,1 or H1,2”). However, in some cases, they do not make this inference. Instead, they make separate inferences about each of the individual hypotheses that comprise the joint hypothesis (e.g., H1,1 and H1,2). For example, a researcher might use a Bonferroni correction to adjust their alpha level from the conventional level (...) of 0.050 to 0.025 when testing H1,1 and H1,2, find a significant result for H1,1 (p < 0.025) and not for H1,2 (p > .0.025), and so claim support for H1,1 and not for H1,2. However, these separate individual inferences do not require an alpha adjustment. Only a statistical inference about the union alternative hypothesis “H1,1 or H1,2” requires an alpha adjustment because it is based on “at least one” significant result among the two tests, and so it refers to the familywise error rate. Hence, an inconsistent correction occurs when a researcher corrects their alpha level during multiple testing but does not make an inference about a union alternative hypothesis. In the present article, I discuss this inconsistent correction problem, including its reduction in statistical power for tests of individual hypotheses and its potential causes vis-à-vis error rate confusions and the alpha adjustment ritual. I also provide three illustrations of inconsistent corrections from recent psychology studies. I conclude that inconsistent corrections represent a symptom of statisticism, and I call for a more nuanced inference-based approach to multiple testing corrections. (shrink)

A new Evidence Test is applied to the problem of testing whether two Poisson random variables are dependent. The dependence structure is that of Holgate’s bivariate distribution. These bivariate distribution depends on three parameters, 0 < theta_1, theta_2 < infty, and 0 < theta_3 < min(theta_1, theta_2). The Evidence Test was originally developed as a Bayesian test, but in the present paper it is compared to the best known test of the hypothesis of independence in a frequentist framework. (...) It is shown that the Evidence Test is considerably more powerful when the correlation is not too close to zero, even for small samples. (shrink)

It has been claimed that if statistical power and p-values are both used to measure the strength of our evidence for the null-hypothesis when the results of our tests are not significant, then they can also be used to derive inconsistent epistemic judgements as we compare two different experiments. Those problematic derivations are known as power approach paradoxes. The consensus is that we can avoid them if we abandon the idea that statistical power can measure the strength (...) of our evidence. In this paper however, I put forward a different solution. I argue that every power approach paradox rests on an equivocation on "strong evidence". The main idea is that we need to make a careful distinction between the evidence provided by the quality of the test and the evidence provided by the outcome of the test. Both provide different types of evidence and their respective strength are to be evaluated differently. (shrink)