Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias also foregrounds a range (...) of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Journal peer review is an essential part of academic practices.1 But how well does it serve its purpose and which factors have an influence on how close it comes to achieving its aims? Peer review has been widely discussed in empirical literature: it has been studied both qualitatively and quantitatively (e.g., by Cole, who in his 1992 book uses data on how grant applications submitted to National Science Foundation were...

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

Realising the societal gains from publicly funded health and medical research requires a model for a reflexive evaluation precedent for the societal impact of research. This research explores UK Research Excellence Framework evaluators’ values and opinions and assessing societal impact, prior to the assessment taking place. Specifically, we discuss the characteristics of two different impact assessment extremes – the “quality-focused” evaluation and “societal impact-focused” evaluation. We show the wide range of evaluator views about impact, and that these views could be (...) conceptually reflected in a range of different positions along a conceptual evaluation scale. We describe the characteristics of these extremes in detail, and discuss the different beliefs evaluators had which could influence where they positioned themselves along the scale. These decisions, we argue, when considered together, form a dominant definition of societal impact that influences the direction of its evaluation by the panel. (shrink)

The goal of this dissertation is to reconstruct, critically evaluate, and apply the pluralism of Paul Feyerabend. I conclude by suggesting future points of contact between Feyerabend’s pluralism and topics of interest in contemporary philosophy of science. I begin, in Chapter 1, by reconstructing Feyerabend’s critical philosophy. I show how his published works from 1948 until 1970 show a remarkably consistent argumentative strategy which becomes more refined and general as Feyerabend’s thought matures. Specifically, I argue that Feyerabend develops a persuasive (...) case against rationalism, or the thesis that there exist normative and exclusive rules of scientific rationality. In Chapter 2, I reconstruct Feyerabend’s pluralism and detail its relationship to his humanitarianism and epistemological anarchism. I understand Feyerabend’s pluralism as the combination of the principles of proliferation and tenacity. I show the evolution and justification of these principles from Feyerabend’s early papers until the late 1970s. In Chapter 3, I defend Feyerabend’s pluralism from its most prominent criticisms. I then clarify that Feyerabend’s pluralism amounts to a conception of the logic of theory pursuit and modify his view using insights from C.S. Peirce, Pierre Duhem, and Michael Polanyi. From Peirce, I show how economic, sociological, and value-laden features of theory pursuit may be used to constrain proliferation and tenacity. From Duhem and Polanyi, I try to show the proper role of tacit knowledge within a Feyerabendian framework. Finally, I show what implications Feyerabend’s pluralism has for models of distributing funds within scientific communities. I contend that it provides a more promising model that the ‘well-ordered science’ proposal advanced by numerous philosophers and social scientists. Specifically, I aim to understand what taking Feyerabend’s pluralism seriously entails for principles of balancing funding allocation decisions and the role of peer-review in evaluating the potential success of research proposals. I conclude by suggesting future lines of research for further analyzing and applying Feyerabend’s pluralism. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications (...) of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

There is an increasing push by journals to ensure that data and products related to published papers are shared as part of a cultural move to promote transparency, reproducibility, and trust in the scientific literature. Yet few journals commit to evaluating their effectiveness in implementing reporting standards aimed at meeting those goals (1, 2). Similarly, though the vast majority of journals endorse peer review as an approach to ensure trust in the literature, few make their peer review data available to (...) evaluate effectiveness toward achieving concrete measures of quality, including consistency and completeness in meeting reporting standards. Remedying these apparent disconnects is critical for closing the gap between guidance recommendations and actual reporting behavior. We see this as a collective action problem requiring leadership and investment by publishers, who can be incentivized through mechanisms that allow them to manage reputational risk and through continued innovation in journal assessment. (shrink)