I suggest that the recent, highly visible, and often heated debate over failures to replicate the results in the social sciences reveals more than the need for greater attention to the pragmatics and value of empirical falsification. It also is a symptom of a serious issue -- the underdeveloped state of theory in many areas of psychology. While I focus on the phenomenon of “social priming” -- since it figures centrally in current debate -- it is not the only area (...) of psychological inquiry to which my critique applies. I first discuss some of the key issues in the “social priming” debate and then attempt to show that many of the problems thus far identified are traceable to a lack of specificity of theory. Finally, I hint at the possibility that adherence to the materialist tenets of modern psychological theory may have a limiting effect on our full appreciation of the phenomena under scrutiny. (shrink)

Previous research reported that college students' symbolic addition and subtraction fluency improved after training with non-symbolic, approximate addition and subtraction. These findings were widely interpreted as strong support for the hypothesis that the Approximate Number System (ANS) plays a causal role in symbolic mathematics, and that this relation holds into adulthood. Here we report four experiments that fail to find evidence for this causal relation. Experiment 1 examined whether the approximate arithmetic training effect exists within a shorter training period than (...) originally reported (2 vs 6 days of training). Experiment 2 attempted to replicate and compare the approximate arithmetic training effect to a control training condition matched in working memory load. Experiments 3 and 4 replicated the original approximate arithmetic training experiments with a larger sample size. Across all four experiments (N = 318) approximate arithmetic training was no more effective at improving the arithmetic fluency of adults than training with control tasks. Results call into question any causal relationship between approximate, non-symbolic arithmetic and precise symbolic arithmetic. (shrink)

The ongoing debate about a “replication crisis” has put scientific failure in the spotlight, not only in psychological research and the social sciences but also in the life sciences. However, despite this increased salience of failure in research, the concept itself has so far received little attention in the literature (for an exception, see Ref. 1). The lack of a systematic perspective on scientific failure—a daily experience for researchers—hampers our understanding of this complex phenomenon and the (...) development of efficient policies and measures to address it. Without a better grasp of the multiple dimensions of scientific failure, there is a risk that necessary measures will be neglected or that inadequate policies will be adopted because different kinds of failures require different responses. Developing a basic taxonomy of scientific failure will help to identify connections between different types of failures and benefit the formulation of policy measures for improving replicability. (shrink)

English uses the horizontal spatial metaphors to express time (e.g., the good days ahead of us). Chinese also uses the vertical metaphors (e.g., 'the month above' to mean last month). Do Chinese speakers, then, think about time in a different way than English speakers? Boroditsky [Boroditsky, L. (2001). Does language shape thought? Mandarin and English speakers' conceptions of time. Cognitive Psychology, 43(1), 1-22] claimed that they do, and went on to conclude that 'language is a powerful tool in shaping habitual (...) thought about abstract domains' (such as time). By estimating the frequency of usage, we found that Chinese speakers actually use the horizontal spatial metaphors more often than the vertical metaphors. This offered no logical ground for Boroditsky's claim. We were also unable to replicate her experiments in four different attempts. We conclude that Chinese speakers do not think about time in a different way than English speakers just because Chinese also uses the vertical spatial metaphors to express time. (shrink)

Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures. In this article, I contrast Popper’s approach with Lakatos’ (1978) approach and a related approach called naïve methodological falsificationism (NMF; Lakatos, 1978). The Popperian approach (...) is powerful because it is based on logical refutation, but its theories are noncausal and, therefore, lacking in scientific value. In contrast, the Lakatosian approach considers causal theories, but it concedes that these theories are not logically refutable. Finally, the NMF approach subjects Lakatosian causal theories to Popperian logical refutations. However, its approach of temporarily accepting a ceteris paribus clause during theory testing may be viewed as scientifically inappropriate, epistemically inconsistent, and “completely redundant” (Lakatos, 1978, p. 40). I conclude that the replication “crisis” makes the most sense in the context of the Popperian and NMF approaches because it is only in these two approaches that replication failures represent logical refutations of theories. In contrast, replication failures are less problematic in the Lakatosian approach because they do not logically refute theories. Indeed, in the Lakatosian approach, replication failures can be legitimately ignored or used to motivate theory development. (shrink)

Scientists often respond to failures to replicate by citing differences between the experimental components of an original study and those of its attempted replication. In this paper, we investigate these purported mismatch explanations. We assess a body of failures to replicate in neuroscience studies on spinal cord injury. We argue that a defensible mismatch explanation is one where a mismatch of components is a difference maker for a mismatch of outcomes, and the components are relevantly different in the follow-up (...) study, given the scope of the original study. With this account, we argue that not all differences between studies are meaningful, even if they are difference makers. As our examples show, focusing only on these differences results in disregarding the representativeness of the original experiment’s components and the scope of its outcomes, undercutting other epistemic aims, such as translation, in the process. (shrink)

The replicability of experiment is routinely offered as the gold standard of evidence. I argue that it is not supported by a universal principle of replicability in inductive logic. A failure of replication may not impugn a credible experimental result; and a successful replication can fail to vindicate an incredible experimental result. Rather, employing a material approach to inductive inference, the evidential import of successful replication of an experiment is determined by the prevailing background facts. Commonly, (...) these background facts do support successful replication as a good evidential guide and this has fostered the illusion of a deeper, exceptionless principle. (shrink)

The idea of replication is based on the premise that there are permanent laws to be replicated and verified, and the scientific method is adequate for doing so. Scientific truth, however, is not absolute but relative to time and context, and the method used. Time and context are inextricably interwoven, in that time creates different contexts and contexts (e.g., Christmas Day vs. New Year’s Day) create different experiences of time, rendering psychological phenomena inherently variable. This means that internal and (...) external conditions fluctuate and are different in a replication study vs. the original. Thus, a replication experiment is just another empirical investigation that has no special status in the establishment of scientific truth. It is not the final arbiter of whether or not something exists. In their pursuit of homogeneous external conditions, replications have ignored the homogeneity of internal conditions. There is not a single replication reported in the literature that would have shown participants’ feelings and thoughts—both conscious and nonconscious—to be identical to those of the original participants. Experimental instructions can create varying ratios of conscious over nonconscious processing from one study to another. Ironically, every replication is a failure at the fundamentals of human psychology. While patterns can be discovered, they are not permanent or unchangeable laws of human behavior to be proven by the pinpoint statistical verification through replication. As scientific knowledge in physics is temporary and incomplete, should it be any surprise that science can only provide “temporary winners” for psychological knowledge of human behavior? (shrink)

Replication protein A (RPA) is the main eukaryotic single‐stranded DNA (ssDNA) binding protein, having essential roles in all DNA metabolic reactions involving ssDNA. RPA binds ssDNA with high affinity, thereby preventing the formation of secondary structures and protecting ssDNA from the action of nucleases, and directly interacts with other DNA processing proteins. Here, we discuss recent results supporting the idea that one function of RPA is to prevent annealing between short repeats that can lead to chromosome rearrangements by microhomology‐mediated (...) end joining or the formation of hairpin structures that are substrates for structure‐selective nucleases. We suggest that replication fork catastrophe caused by depletion of RPA could result from cleavage of secondary structures by nucleases, and that failure to cleave hairpin structures formed at DNA ends could lead to gene amplification. These studies highlight the important role RPA plays in maintaining genome integrity. (shrink)

In “Normativity and Epistemic Intuitions”, Weinberg, Nichols and Stich famously argue from empirical data that East Asians and Westerners have different intuitions about Gettier -style cases. We attempted to replicate their study about the Car case, but failed to detect a cross - cultural difference. Our study used the same methods and case taken verbatim, but sampled an East Asian population 2.5 times greater than NEI’s 23 participants. We found no evidence supporting the existence of cross - cultural difference about (...) the intuition concerning the Gettier car case. Taken together with the failures of both of the existing replication studies, our results provide strong evidence that the purported cross - cultural difference in Gettier intuitions does not exist. (shrink)

In bacteria, PriA protein, a conserved DEXH‐type DNA helicase, plays a central role in replication restart at stalled replication forks. Its unique DNA‐binding property allows it to recognize and stabilize stalled forks and the structures derived from them. Cells must cope with fork stalls caused by various replication stresses to complete replication of the entire genome. Failure of the stalled fork stabilization process and eventual restart could lead to various forms of genomic instability. The low (...) viability of priA null cells indicates a frequent occurrence of fork stall during normal growth that needs to be properly processed. PriA specifically recognizes the 3′‐terminus of the nascent leading strand or the invading strand in a displacement (D)‐loop by the three‐prime terminus binding pocket (TT‐pocket) present in its unique DNA binding domain. Elucidation of the structural basis for recognition of arrested forks by PriA should provide useful insight into how stalled forks are recognized in eukaryotes. (shrink)

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the (...) field of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science. (shrink)

The replicability crisis refers to the apparent failures to replicate both important and typical positive experimental claims in psychological science and biomedicine, failures which have gained increasing attention in the past decade. In order to provide evidence that there is a replicability crisis in the first place, scientists have developed various measures of replication that help quantify or “count” whether one study replicates another. In this nontechnical essay, I critically examine five types of replication measures used in the (...) landmark article “Estimating the reproducibility of psychological science” (Open Science Collaboration, Science, 349, ac4716, 2015) based on the following techniques: subjective assessment, null hypothesis significance testing, comparing effect sizes, comparing the original effect size with the replication confidence interval, and meta-analysis. The first four, I argue, remain unsatisfactory for a variety of conceptual or formal reasons, even taking into account various improvements. By contrast, at least one version of the meta-analytic measure does not suffer from these problems. It differs from the others in rejecting dichotomous conclusions, the assumption that one study replicates another or not simpliciter. I defend it from other recent criticisms, concluding however that it is not a panacea for all the multifarious problems that the crisis has highlighted. (shrink)

Cunha and Caldieraro (2009) investigated whether sunk-cost effects, which are well documented in hypothetical situations involving monetary investments, also occur in choice situations with purely behavioral investments. Their results suggest that decision makers indeed fall prey to behavioral sunk-cost effects under certain circumstances. I have been unable to replicate their pattern of results in three separate investigations. In these studies, I attempted to recover the effect using two other behavioral effort manipulations in addition to the manipulation used by Cunha and (...) Caldieraro. This failure to replicate the pattern of results calls into question the robustness of the initial findings. (shrink)

Grau and Pury (Review of Philosophy and Psychology, 5, 155–168, 2014) reported that people’s views about love are related to their views about reference. This surprising effect was however not replicated in Cova et al.’s (in press) replication study. In this article, we show that the replication failure is probably due to the replication’s low power and that a metaanalytic reanalysis of the result in Cova et al. suggests that the effect reported in Grau and Pury (...) is real. We then report a large, highly powered replication that successfully replicates Grau and Pury 2014. This successful replication is a case study in the challenges involved in replicating scientific work, and our article contributes to the discussion of these challenges. (shrink)

The simian virus 40 chromosome, a model for the mammalian replicon, is a uniquely powerful system for the study of drugs and treatments which target enzymes of the mammalian replication apparatus. High resolution gel electrophoretic analysis of normal and aberrant viral replication intermediates can be used effectively to understand the molecular events of replication failure. These events include breakage of replication forks, aberrant topoisomerase action, failure to separate daughter chromosomes, protein‐DNA crosslinking, single and double (...) strand DNA breakage, alterations in topology and inactivation of replication intermediates. The SV40 replication system can also be used to study the recombinational events which often follow drug‐induced replication failure. (shrink)

Is science in the midst of a crisis of replicability and false discoveries? In a recent article, Alexander Bird offers an explanation for the apparent lack of replicability in the biomedical sciences. Bird argues that the surprise at the failure to replicate biomedical research is a result of the fallacy of neglecting the base rate. The base-rate fallacy arises in situations in which one ignores the base rate—or prior probability—of an event when assessing the probability of this event in (...) the light of some observed evidence. By extension, the replication crisis would result from ignoring the low prior probability of biomedical hypotheses. In this paper, my response to Bird’s claim is twofold. First, I show that the argument according to which the replication crisis is due to the low prior of biomedical hypotheses is incomplete. Second, I claim that a simple base-rate fallacy model does not account for some important methodological insights that have emerged in discussions of the replication crisis. (shrink)

While it is widely acknowledged that psychology is in the throes of a replication ‘crisis’, relatively little attention has been paid to the role theory plays in our evaluation of replications as ‘failed’ or ‘successful’. This paper applies well-known arguments in philosophy of science about the interplay between theory and experiment to a contemporary case study of infants’ understanding of false belief (Onishi and Baillargeon [2005]), and attempts to replicate it. It argues that the lack of consensus about over-arching (...) theories informing both the concepts under study and the methodologies used to track them means that researchers disagree over which experiments constitute replications of the original. The second part of the paper places this specific debate within a broader discussion of the replication crisis as a crisis of ‘theory’, developing work by Muthukrishna and Henrich ([2018]) and Bird ([2018]). Bird argues that the lack of agreed over-arching theories in psychology means that a high rate of replication failure is to be expected; this paper agrees with his diagnosis but challenges his proposal that more replication will resolve the problem. (shrink)

This commentary tests a methodology proposed by Munk et al. (2022) for using failed predictions in machine learning as a method to identify ambiguous and rich cases for qualitative analysis. Using a dataset describing actions performed by fictional characters interacting with machine vision technologies in 500 artworks, movies, novels and videogames, I trained a simple machine learning algorithm (using the kNN algorithm in R) to predict whether or not an action was active or passive using only information about the fictional (...) characters. Predictable actions were generally unemotional and unambiguous activities where machine vision technologies were treated as simple tools. Unpredictable actions, that is, actions that the algorithm could not correctly predict, were more ambivalent and emotionally loaded, with more complex power relationships between characters and technologies. The results thus support Munk et al.'s theory that failed predictions can be productively used to identify rich cases for qualitative analysis. This test goes beyond simply replicating Munk et al.'s results by demonstrating that the method can be applied to a broader humanities domain, and that it does not require complex neural networks but can also work with a simpler machine learning algorithm. Further research is needed to develop an understanding of what kinds of data the method is useful for and which kinds of machine learning are most generative. To support this, the R code required to produce the results is included so the test can be replicated. The code can also be reused or adapted to test the method on other datasets. (shrink)

. Reproducible results and repeatable measurements at the same location are fundamental to science, yet of grave concern to scientists. Involvement in biological re-surveys under MVZ-Berkeley, Harvard-LTER and Hamaarag elucidated “replication” and “location” and untangled “incommensurability” from “no fact of the matter” and “indeterminacy.” All cases revealed incommensurability without indeterminacy on the smallest scale and indeterminacy without incommensurability on higher scales, with communication failure in the former and successful workarounds in the latter. I argue that an involved philosophy (...) helps clarify fundamental concepts in this case, and since it could also help other cases it therefore should be examined in every case before being ruled out. (shrink)

The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an (...) analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study. (shrink)

The aim of the present research was twofold. First, we examined the effects of perceived organizational support on workplace conflict (i.e., relationship conflict and task conflict). Second, we identified one mechanism explaining these relationships, namely failure-related trust. Using a sample of 263 teachers from Belgium, the results of Study 1 indicated that perceived organizational support is negatively related to relationship conflict and is also, unexpectedly, negatively related to task conflict. Furthermore, using a sample of 477 Belgian employees, Study 2 (...) replicated these results and further demonstrated that failure-related trust mediates the negative relationship between perceived organizational support and both types of workplace conflict. Theoretical and practical insights of this research are discussed. (shrink)

In 1669, Malpighi published the first systematic dissection of an insect. The manuscript of this work contains a striking water-colour of the silkworm, which is described here for the first time. On repeating Malpighi's pioneering investigation, Swammerdam found what he thought were a number of errors, but was hampered by Malpighi's failure to explain his techniques. This may explain Swammerdam's subsequent description of his methods. In 1675, as he was about to abandon his scientific researches for a life of (...) religious contemplation, Swammerdam destroyed his manuscript on the silkworm, but not before sending the drawings to Malpighi. These figures, with their rich and unique use of colour, are studied here for the first time. The role played by Henry Oldenburg, secretary of the Royal Society, in encouraging contact between the two men is emphasized and the way this exchange reveals the development of some key features of modern science -- replication and modern scientific illustration -- is discussed. (shrink)

In football it was recently demonstrated, that patterns of motivational constructs in young talented football players are relatively stable in early adolescence, and are associated with specific performance related outcomes (Zuber, Zibung, & Conzelmann, 2015). The aim of the present study was to check whether the motivational patterns found in youth elite football also re-emerge in ice hockey, showing similar relations to performance. 135 young male ice hockey talents (MAge = 17.26, SD = 1.24) playing on the highest and second (...) highest level of their age group were questioned about five motivational constructs. Six months later, their coaches rated their players’ current game performance. The results demonstrated a very high similarity of the patterns in ice hockey with the ones earlier found in football. In terms of the transition to performance levels, the highly intrinsically achievement-oriented players were rated by their coaches as belonging to the top-level players significantly more often (OR = 2.6), whereas the non-achievement-oriented failure-fearing players show a higher chance to be rated as part of the lowest performance group (OR = 3.8). The results indicate the importance of achievement motivation for performance in ice hockey also, and this generalizability thus provides relevant clues which are important for consideration in national programs for talent identification across several sports. (shrink)

Transcranial direct current stimulation (tDCS) has been recognized as a promising tool for investigating the causal relationship between specific brain areas of interest and behavior. However, the reproducibility of previous tDCS studies is often questioned because of failures in replication. This study focused on the effects of tDCS on one cognitive domain: beauty perception. To date, the modulation of beauty perception by tDCS has been shown in two studies:Cattaneo et al. (2014)andNakamura and Kawabata (2015). Here, we aimed at replicating (...) their studies and investigating the effects of tDCS on beauty perception using the following parameters: (1) cathodal stimulation over the medial prefrontal cortex (mPFC) (Nakamura and Kawabata, 2015); (2) anodal stimulation over the left dorsolateral prefrontal cortex (lDLPFC) (Cattaneo et al., 2014). We also performed a more focal stimulation targeting the orbitofrontal cortex (OFC) to determine the optimal stimulation site for modulating beauty perception (3). Participants rated the subjectively-perceived beauty of the images before and after the tDCS administration. We divided images into four clusters according to the obtained scores in our preliminary study and examined changes in beauty ratings in each image cluster separately to exclude factors, such as stimuli attributions that may reduce tDCS effects. The results showed no strong effects of tDCS with the same parameters as in previous studies on beauty rating scores in any image cluster. Likewise, anodal stimulation over the OFC did not result in a change in rating scores. In contrast to previous studies, the current study did not corroborate the effects of tDCS on beauty perception. Our findings provide evidence regarding the recent reproducibility issue of tDCS effects and suggest the possible inflation of its effects on cognitive domains. (shrink)

We examined all 208 closed cases involving official findings of research misconduct published by the US Office of Research Integrity from 1992 to 2011 to determine how often scientists mention in a retraction or correction notice that there was an ethical problem with an associated article. 75 of these cases cited at least one published article affected by misconduct for a total of 174 articles. For 127 of these 174, we found both the article and a retraction or correction statement. (...) Since eight of the 127 published statements consisted of simply the word ‘retracted,’ our analysis focused on the remaining 119 for which a more detailed retraction or correction was published. Of these 119 statements, only 41.2% mentioned ethics at all (and only 32.8% named a specific ethical problem such as fabrication, falsification or plagiarism), whereas the other 58.8% described the reason for retraction or correction as error, loss of data or replication failure when misconduct was actually at issue. Among the published statements in response to an official finding of misconduct (within the time frame studied), the proportion that mentioned ethics was significantly higher in recent years than in earlier years, as was the proportion that named a specific problem. To promote research integrity, scientific journals should consider adopting policies concerning retractions and corrections similar to the guidelines developed by the Committee on Publication Ethics. Funding agencies and institutions should take steps to ensure that articles affected by misconduct are retracted or corrected. (shrink)

Theoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad perspective on scientific error, which may lead to replication failures. This broad perspective focuses on reform history and on opportunities for future reform. We organize our perspective along four main themes: institutional reform, methodological reform, statistical reform and publishing reform. For each theme, we illustrate potential errors by narrating the story of a (...) fictional researcher during the research cycle. We discuss future opportunities for reform. The resulting agenda provides a resource to usher in an era that is marked by a research culture that is less error-prone and a scientific publication landscape with fewer spurious findings. (shrink)

The ‘psychologist’s green thumb’ refers to the argument that an experimenter needs an indeterminate set of skills to successfully replicate an effect. This argument is sometimes invoked by psychological researchers to explain away failures of independent replication attempts of their work. In this paper, I assess the psychologist’s green thumb as a candidate explanation for individual replication failure and argue that it is potentially costly for psychology as a field. I also present other, more likely reasons for (...) these replication failures. I conclude that appealing to a psychologist’s green thumb is not a convincing explanation for replication failure. (shrink)

Psychological distance effects have attracted the attention of behavioral economists in the context of descriptive modeling and behavioral policy. Indeed, psychological distance effects have been shown for an increasing number of domains and applications relevant to economic decision-making. The current paper questions whether these effects are robust enough for economists to apply them to relevant policy questions. We demonstrate systematic replication failures for the distance-from-a-distance effect shown by Maglio et al., and relate them to theoretical arguments suggesting that psychological (...) distance theories are currently too poorly specified to make predictions that are precise enough for economic analyses. (shrink)