Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, (...) raises questions about the normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding (...) in science. Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Peer reviews of 84 organic farming grant applications from Sweden were analyzed to determine whether the reviewers’ affiliation to one of two types of agriculture (i.e., organic and conventional) influenced their reviews. Fifteen reviewers were divided into three groups: (1) scientists with experience in organic farming research; (2) scientists with no experience in organic farming research; and (3) users of organic farming research. The two groups of scientists assessed the societal relevance and scientific quality of the grant applications based (...) on three criteria (i.e., presentation, methodology, qualifications), whereas the user group only assessed societal relevance. The analysis showed that the two groups of scientists provided very different reviews. Scientist reviewers with experience in organic farming research agreed more with the user group on research relevance than did scientist reviewers without such experience, and the assessment of relevance was closely correlated to the assessment of scientific quality within both scientific groups. As both scientific groups did not clearly distinguish between societal relevance and scientific quality, the idea of an objective science is challenged. The contextual values associated with the norms of good agriculture were not clearly distinguished from the constitutive values of science associated with the traditional norms of good science. This raises the question of whether organic and conventional grant applications should be mixed for review regardless of the reviewers. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials (...) we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it (...) is in many cases impossible to avoid commensuration bias. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion (...) platforms such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review. (shrink)

Peer review is a widely accepted instrument for raising the quality of science. Peer review limits the enormous unstructured influx of information and the sheer amount of dubious data, which in its absence would plunge science into chaos. In particular, peer review offers the benefit of eliminating papers that suffer from poor craftsmanship or methodological shortcomings, especially in the experimental sciences. However, we believe that peer review is not always appropriate for the (...) evaluation of controversial hypothetical science. We argue that the process of peer review can be prone to bias towards ideas that affirm the prior convictions of reviewers and against innovation and radical new ideas. Innovative hypotheses are thus highly vulnerable to being “filtered out” or made to accord with conventional wisdom by the peer review process. Consequently, having introduced peer review, the Elsevier journal Medical Hypotheses may be unable to continue its tradition as a radical journal allowing discussion of improbable or unconventional ideas. Hence we conclude by asking the publisher to consider re-introducing the system of editorial review to Medical Hypotheses. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as (...) an alternative decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

In a recent article in the British Journal for the Philosophy of Science, Heesen and Bright argue that prepublication peer review should be abolished and replaced with postpublication peer review (provided the matter is judged purely on epistemic grounds). In this article, I show that there are three problems with their argument. First, it fails to consider the epistemic cost of implementing the change to postpublication peer review. Second, it fails to consider some potential (...) epistemic benefits of prepublication peer review, which involve avoiding bias. Third, it fails to consider some potential epistemic disadvantages of postpublication peer review, which stem from the greater number of papers that would be published under that system. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument (...) rests on two key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

BackgroundThe Patient-Centered Outcomes Research Institute is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion (...) of reports in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 submitted research reports. Types of spin included reporting bias, inappropriate interpretation, inappropriate extrapolation of results, and inappropriate attribution of causality. Authors addressed comments about spin related to 47/55 of the reports.Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 articles, of which 27/44 contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review.DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support. (shrink)

This article reports the results of an anonymous survey of researchers at a government research institution concerning their perceptions about ethical problems with journal peer review. Incompetent review was the most common ethical problem reported by the respondents, with 61.8% (SE = 3.3%) claiming to have experienced this at some point during peer review. Bias (50.5%, SE = 3.4%) was the next most common problem. About 22.7% (SE = 2.8%) of respondents said that a (...) reviewer had required them to include unnecessary references to his/her publication(s), 17.7% (SE = 2.6%) said that comments from reviewers had included personal attacks, and 9.6% (SE = 2.0%) stated that reviewers had delayed publication to publish a paper on the same topic. Two of the most serious violations of peer review ethics, breach of confidentiality (6.8%, SE = 1.7%) and using ideas, data, or methods without permission (5%, SE = 1.5%) were perceived less often than the other problems. We recommend that other investigators follow up on our exploratory research with additional studies on the ethics of peer review. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since (...) they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

A recent commentary argued for arbitration to resolve authorship disputes within academic research settings explaining that current mechanisms to resolve conflicts result in unclear outcomes and institutional power vested in senior investigators could compromise fairness. We argue here that arbitration is not a suitable means to resolve disputes among researchers in academia because it remains unclear who will assume the costs of arbitration, the rules of evidence do not apply to arbitration, and decisions are binding and very difficult to appeal. (...) Instead of arbitration, we advocate for peer-based approaches involving a peer review committee and research ethics consultation to help resolve authorship disagreements. We describe the composition of an institutional peer review committee to address authorship disputes. Both of these mechanisms are found, or can be formed, within academic institutions and offer several advantages to researchers who are likely to shy away from legalistic processes and gravitate towards those handled by their peers. Peer-based approaches are cheaper than arbitration and the experts involved have knowledge about academic publishing and the culture of research in the specific field. Decisions by knowledgeable and neutral experts could reduce bias, have greater authority, and could be appealed. Not only can peer-based approaches be leveraged to resolve authorship disagreements, but they may also enhance collegiality and promote a healthy team environment. (shrink)

BackgroundThe process of peer-review in academia has attracted criticism surrounding issues of bias, fairness, and professionalism; however, frequency of occurrence of such comments is unknown.MethodsWe evaluated 1491 sets of reviewer comments from the fields of “Ecology and Evolution” and “Behavioural Medicine,” of which 920 were retrieved from the online review repository Publons and 571 were obtained from six early career investigators. Comment sets were coded for the occurrence of “unprofessional comments” and “incomplete, inaccurate or unsubstantiated critiques” (...) using an a-prior rubric based on our published research. Results are presented as absolute numbers and percentages.ResultsOverall, 12% of comment sets included at least one unprofessional comment towards the author or their work, and 41% contained incomplete, inaccurate of unsubstantiated critiques.ConclusionsThe large number of unprofessional comments, and IIUCs observed could heighten psychological distress among investigators, particularly those at an early stage in their career. We suggest that development and adherence to a universally agreed upon reviewer code of conduct is necessary to improve the quality and professional experience of peer review. (shrink)

Impositiion of selection and opportunity for censorship meust be regarded as aberrations of a communication system for science. Future historians might wonder why these faults evinced so little concern. Because editorial decisiions pre-empt scientific debate, editors and their advisers assume a heavy responsibility for nurturing fresh conjectures and for maintaining unbiased speedy communication. Evidently this responsibility has not always been honoured.Available evidence of inappropriate rejection confirms the expectable, if not adequately anticipated, tendency for reviewers to oppose innovation; so that although (...) the statistics of rejection might seem acceptable, there is an unfortunate selectivity working against new ideas. It follows that the perceptive and inventive must work harder to be heard. This is quite the wrong bias: the ingenious should be encouraged, not opposed.The strategy of ‘peer review’ is attractive to publishers and disbursers of funds because it is cheap and practical; but there are fundamental weaknesses in the notion of peer review: peer status is illusory, preception is fickle, and impartiality is unnatural. Peer opinion is not, ipso facto, scientific argument.Unchecked whimsical censorship, which is most likely to affect the more perceptive and innovative offerings, disregards the fundamental uncertainty of scientific knowledge and denies the value of general debate. ‘Peer’ review should not usurp that debate, but should rather act as a strictly preliminary filter atuned to acceptance of fresh ideas.The psychology of anonymous ‘peer’ review is inimical to science. Authority automatically accorded to reviewers' comment and defence of ‘reputation’ are antitheses of scientific method. Proper functioning of review as a preliminary filter depends entirely on the good sense and integrity of the few people involved. Unfortunately there is no guarantee that criticism will be rational, relevant and refutable. The accidental authoeity of the uncertain review process encourages obstruction and fraud.In order to safeguard communication, the burden of proof in refereeing should be on the criticisers. Disagreement is no reason for suppression. Reviewers' objections, which should be factual and specific, should always be tested. Failure to ebserve these elementary principles of scientific assessment, by admitting anonymous unsupported opinion as valid criticism, has caused and is causing constraint on communication and on exposure of new ideas. (shrink)

In the past few decades, peer review has come to dominate virtually all professionally respectable academic and scientific publications. However, despite its near-universal acceptance, no code of conduct has been developed to which peer reviewers and their editors are encouraged to adhere. This paper proposes such a code of conduct.

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes (...) including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. (shrink)

Background: In biomedical research, there have been numerous scandals highlighting conflicts of interest (COIs) leading to significant bias in judgment and questionable practices. Academic institutions, journals, and funding agencies have developed and enforced policies to mitigate issues related to COI, especially surrounding financial interests. After a case of editorial COI in a prominent bioethics journal, there is concern that the same level of oversight regarding COIs in the biomedical sciences may not apply to the field of bioethics. In this (...) study, we examined the availability and comprehensiveness of COI policies for authors, peer reviewers, and editors of bioethics journals. Methods: After developing a codebook, we analyzed the content of online COI policies of 63 bioethics journals, along with policy information provided by journal editors that was not publicly available. Results: Just over half of the bioethics journals had COI policies for authors (57%), and only 25% for peer reviewers and 19% for editors. There was significant variation among policies regarding definitions, the types of COIs described, the management mechanisms, and the consequences for noncompliance. Definitions and descriptions centered on financial COIs, followed by personal and professional relationships. Almost all COI policies required disclosure of interests for authors as the primary management mechanism. Very few journals outlined consequences for noncompliance with COI policies or provided additional resources. Conclusion: Compared to other studies of biomedical journals, a much lower percentage of bioethics journals have COI policies and these vary substantially in content. The bioethics publishing community needs to develop robust policies for authors, peer reviewers, and editors and these should be made publicly available to enhance academic and public trust in bioethics scholarship. (shrink)

Implicit biases involve associations outside conscious awareness that lead to a negative evaluation of a person on the basis of irrelevant characteristics such as race or gender. This review examines the evidence that healthcare professionals display implicit biases towards patients. PubMed, PsychINFO, PsychARTICLE and CINAHL were searched for peer-reviewed articles published between 1st March 2003 and 31st March 2013. Two reviewers assessed the eligibility of the identified papers based on precise content and quality criteria. The references of eligible (...) papers were examined to identify further eligible studies. Forty two articles were identified as eligible. Seventeen used an implicit measure, to test the biases of healthcare professionals. Twenty five articles employed a between-subjects design, using vignettes to examine the influence of patient characteristics on healthcare professionals’ attitudes, diagnoses, and treatment decisions. The second method was included although it does not isolate implicit attitudes because it is recognised by psychologists who specialise in implicit cognition as a way of detecting the possible presence of implicit bias. Twenty seven studies examined racial/ethnic biases; ten other biases were investigated, including gender, age and weight. Thirty five articles found evidence of implicit bias in healthcare professionals; all the studies that investigated correlations found a significant positive relationship between level of implicit bias and lower quality of care. The evidence indicates that healthcare professionals exhibit the same levels of implicit bias as the wider population. The interactions between multiple patient characteristics and between healthcare professional and patient characteristics reveal the complexity of the phenomenon of implicit bias and its influence on clinician-patient interaction. The most convincing studies from our review are those that combine the IAT and a method measuring the quality of treatment in the actual world. Correlational evidence indicates that biases are likely to influence diagnosis and treatment decisions and levels of care in some circumstances and need to be further investigated. Our review also indicates that there may sometimes be a gap between the norm of impartiality and the extent to which it is embraced by healthcare professionals for some of the tested characteristics. Our findings highlight the need for the healthcare profession to address the role of implicit biases in disparities in healthcare. More research in actual care settings and a greater homogeneity in methods employed to test implicit biases in healthcare is needed. (shrink)

BackgroundDescriptive studies examining publication rates and citation counts demonstrate a geographic skew toward high-income countries (HIC), and research from low- or middle-income countries (LMICs) is generally underrepresented. This has been suggested to be due in part to reviewers’ and editors’ preference toward HIC sources; however, in the absence of controlled studies, it is impossible to assert whether there is bias or whether variations in the quality or relevance of the articles being reviewed explains the geographic divide. This study synthesizes (...) the evidence from randomized and controlled studies that explore geographic bias in the peer review process.MethodsA systematic review was conducted to identify research studies that explicitly explore the role of geographic bias in the assessment of the quality of research articles. Only randomized and controlled studies were included in the review. Five databases were searched to locate relevant articles. A narrative synthesis of included articles was performed to identify common findings.ResultsThe systematic literature search yielded 3501 titles from which 12 full texts were reviewed, and a further eight were identified through searching reference lists of the full texts. Of these articles, only three were randomized and controlled studies that examined variants of geographic bias. One study found that abstracts attributed to HIC sources elicited a higher review score regarding relevance of the research and likelihood to recommend the research to a colleague, than did abstracts attributed to LIC sources. Another study found that the predicted odds of acceptance for a submission to a computer science conference were statistically significantly higher for submissions from a “Top University.” Two of the studies showed the presence of geographic bias between articles from “high” or “low” prestige institutions.ConclusionsTwo of the three included studies identified that geographic bias in some form was impacting on peer review; however, further robust, experimental evidence is needed to adequately inform practice surrounding this topic. Reviewers and researchers should nonetheless be aware of whether author and institutional characteristics are interfering in their judgement of research. (shrink)

In 2014 Jennifer Cramblett, a white lesbian, filed a Complaint for Wrongful Birth alleging that the Midwest Sperm Bank mistakenly provided sperm from an African–American donor. In this article, we trace the complex and overlapping lines of legal and social inheritance that have conditioned not only the possibility of such a lawsuit, but also the legal language and arguments within the Complaint itself. First, we trace the racial politics of homonormativity, which set the conditions of possibility for an out, white (...) lesbian to bring this case forward. Second, we trace the inheritance of wrongful birth tort law, reviewing its prior race and disability-related uses, and its basis in feminist reproductive rights. Third, we trace how disability, race and sexuality interlock within the eugenic inheritance of both ‘wrongful birth’ and reproductive technologies. Finally, we follow traces of racial inheritance, namely, the loss of white property and proximity to whiteness. (shrink)

The questions of scientific choice which were left unresolved when the rapid expansion of academic science in the United States began in the early 1960s have come back to trouble the scientific community. There is now widespread dissatisfaction with the process of review by peers as one of the major systems for the allocation of public funds for research. While earlier criticisms had been brushed off by the assertion—unsupported by facts—that no other systems existed, the present situation cannot be (...) so easily dismissed.A serious examination of other national and international arrangements shows that a wide variety of procedures are in use and there is no research which shows that one system is either more productive scientifically, or more cost-effective in bringing about valuable scientific research. New systems which may be considered should avoid the major defects of the system of peer review as now practised: the enormous waste of scientists' time, the great potential for conflicts of interest, and the inherent bias against innovation.The principal system which I have proposed here combines the best elements of peer review with the simplicity and efficiency of the use of a formula. Moreover, this formula based on peer review of performance incorporates all the elements for which the academic scientific establishment should be accountable to its patron, which is the public treasury. A final virtue of the proposed system is that it provides simple and convenient procedures through the use of numerical weighting factors for the policy-maker to guide the support of scientific research as a whole. (shrink)

High percentages of submitted papers are rejected at editorial levels without offering a second chance to authors by sending their papers for further peer-reviews. In most cases, the rejections are typical quick answers without helpful argumentations related to the content of the rejected material. More surprisingly, some journals vaunt their high rejection rates as a “mark of prestige”!However, journals that reject high percentages of submitted papers have built their prominent positions based on a flawed measure, the impact factor, and (...) from a long and favorable historical context. Their shareholders may think that they are allowed to have a large margin of rejection rates without affecting their sponsorship or funding sources thanks to an extended anchorage since tens, or in some cases hundreds, of years compared to unknown or new journals that struggle to pave a way in the scientific publication world. Historical anchorage of some journals also makes it unfair to compare old and new .. (shrink)

During the early years of the COVID-19 pandemic, preclinical and clinical research were sped up and scaled up in both the public and private sectors and in partnerships between them. This resulted in some extraordinary advances, but it also raised a range of issues regarding the ethics, rigour, and integrity of scientific research, academic publication, and public communication. Many of the failures of scientific rigour and integrity that occurred during the pandemic were exacerbated by the rush to generate, disseminate, and (...) implement research findings, which not only created opportunities for unscrupulous actors but also compromised the methodological, peer review, and advisory processes that would usually identify sub-standard research and prevent compromised clinical or policy-level decisions. While it would be tempting to attribute these failures of science and its translation solely to the “unprecedented” circumstances of the COVID-19 pandemic, the reality is that they preceded the pandemic and will continue to arise once it is over. Existing strategies for promoting scientific rigour and integrity need to be made more rigorous, better integrated into research training and institutional cultures, and made more sophisticated. They might also need to be modified or supplemented with other strategies that are fit for purpose not only in public health emergencies but in any research that is sped-up and scaled up to address urgent unmet medical needs. (shrink)

Remco Heesen has recently argued in favor of the editorial practice of triple-anonymous review on the grounds that ``an injustice is committed against certain authors'' under non-anonymous review. On the other hand, he concedes that the information waste of triple-anonymous review does handicap editors, in particular sacrificing a boost in the average quality of accepted papers that would otherwise be conferred by non-anonymous review. In this paper it is observed that by devoting comparatively greater reviewing resources (...) to the papers of unfamiliar authors, editors practicing non-anonymous review can, without loss of information, avoid subjecting authors to the sorts of injustices observed by Heesen. Thus they can reap the efficiency gains of non-anonymous review without sacrificing fairness. (shrink)

Does hindsight knowledge make research seem more ethical and predictable? In line with the notion of hindsight bias, students in 3 experiments knowing the outcome of an animal experiment judged the results as more foreseeable and ethical relative to students who did not know the outcome. Via self to other comparisons, students evaluate themselves more favorably compared to a peer but exhibited hindsight bias in doing so. Uniquely, the findings reveal the possibility that students deem themselves to (...) be more skeptical and objective than their peers. Implications for teaching animal ethics and for Institutional Review Boards are discussed. (shrink)

This letter contains suggestions for editors and software providers to help avoid affiliation bias in the initial and concluding stages of the peer review process. Submission management systems have a responsibility to ensure protection against affiliation bias. This can be achieved by automatically withholding the author’s identity and affiliation information from all editors, including the Editor-in-Chief, until a decision about publication has been made. Journals relying on email-based submissions are in a more difficult situation. Not having (...) external support in confirming the submitting author’s identity, this dilemma can be resolved through the implementation of strict instructions and guidelines concerning manuscript submission and handling. In this case the “human factor” cannot be fully eliminated, while automated processes can only be changed intentionally and with a certain delay. Therefore, with the suggested steps, submission management systems could become a strong guard against unconscious bias while also making conscious bias more difficult to impact the peer review process. (shrink)

BackgroundFunding agencies have long used panel discussion in the peer review of research grant proposals as a way to utilize a set of expertise and perspectives in making funding decisions. Little research has examined the quality of panel discussions and how effectively they are facilitated.MethodsHere, we present a mixed-method analysis of data from a survey of reviewers focused on their perceptions of the quality, effectiveness, and influence of panel discussion from their last peer review experience.ResultsReviewers indicated (...) that panel discussions were viewed favorably in terms of participation, clarifying differing opinions, informing unassigned reviewers, and chair facilitation. However, some reviewers mentioned issues with panel discussions, including an uneven focus, limited participation from unassigned reviewers, and short discussion times. Most reviewers felt the discussions affected the review outcome, helped in choosing the best science, and were generally fair and balanced. However, those who felt the discussion did not affect the outcome were also more likely to evaluate panel communication negatively, and several reviewers mentioned potential sources of bias related to the discussion. While respondents strongly acknowledged the importance of the chair in ensuring appropriate facilitation of the discussion to influence scoring and to limit the influence of potential sources of bias from the discussion on scoring, nearly a third of respondents did not find the chair of their most recent panel to have performed these roles effectively.ConclusionsIt is likely that improving chair training in the management of discussion as well as creating review procedures that are informed by the science of leadership and team communication would improve review processes and proposal review reliability. (shrink)

Rooted in altruism theory, the purpose of the double-blind academic journal peer-review process is to: assess the quality of scientific research, minimize the potential for nepotism, and; advance the standards of research through high-quality, constructive feedback. However, considering the limited, if any, public recognition and monetary incentives that referees receive for reviewing manuscripts, academics are often reluctant to squander their limited time toward peer reviewing manuscripts. If they do accept such invitations, referees, at times, do not invest (...) the appropriate time needed and, as a result, scantily review manuscripts, which adversely affects the quality of the review. In addition, given that authors’ identities are not blind to journal editors, there is the potential for bias toward well-established academics from highly-ranked institutions. As a result of these issues, the aims of the academic journal review process are currently not being fulfilled. To rectify these issues, several recommendations, namely: single-blind the editors, pay reviewers, standardize the review process, increase the acceptance standards at academic conferences, and provide constructive feedback, are offered. (shrink)

This article focuses on emergency medical care in black urban populations, suggesting that the classification of a “community” within clinical trial language is problematic. The article references a cultural history of black Americans with pre-hospital emergency medical treatment as relevant to contemporary emergency medicine paradigms. Part I explores a relationship between “autonomy” and “community.” The idea of community emerges as a displacement for the ethical principle of autonomy precisely at the moment that institutionalized medicine focuses on diversity. Part II examines (...) a clinical trial for the blood substitute PolyHeme® (Northfield Laboratories, Inc., Evanston, IL). It illustrates the ways in which bias in research paradigms and Institutional Review Board decisions attach to the notion and utility of the language of “community.” The conclusion's contemporary anecdote makes apparent the vitality of the issues of prehospital emergency medical care and the ways in which decisions and practices fall too easily into a narrative of culturally biased treatment. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are (...) either positive or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

Background Implicit biases are present in the general population and among professionals in various domains, where they can lead to discrimination. Many interventions are used to reduce implicit bias. However, uncertainties remain as to their effectiveness. -/- Methods We conducted a systematic review by searching ERIC, PUBMED and PSYCHINFO for peer-reviewed studies conducted on adults between May 2005 and April 2015, testing interventions designed to reduce implicit bias, with results measured using the Implicit Association Test (IAT) (...) or sufficiently similar methods. -/- Results 30 articles were identified as eligible. Some techniques, such as engaging with others’ perspective, appear unfruitful, at least in short term implicit bias reduction, while other techniques, such as exposure to counterstereotypical exemplars, are more promising. Robust data is lacking for many of these interventions. -/- Conclusions Caution is thus advised when it comes to programs aiming at reducing biases. This does not weaken the case for implementing widespread structural and institutional changes that are multiply justified. (shrink)

Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this confidentiality. If such confidentiality (...) is breached by unethical reviewers who might steal or plagiarize the authors’ ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors’ and reviewers’ responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors’ intellectual ideas from predatory reviewers, and increase researchers’ awareness of the responsible reviewing of journal articles and reporting of biomedical research. (shrink)

BackgroundDespite its ubiquity in academic research, the phrase ‘ethical challenge(s)’ appears to lack an agreed definition. A lack of a definition risks introducing confusion or avoidable bias. Conceptual clarity is a key component of research, both theoretical and empirical. Using a rapid review methodology, we sought to review definitions of ‘ethical challenge(s)’ and closely related terms as used in current healthcare research literature.MethodsRapid review to identify peer-reviewed reports examining ‘ethical challenge(s)’ in any context, extracting data (...) on definitions of ‘ethical challenge(s)’ in use, and synonymous use of closely related terms in the general manuscript text. Data were analysed using content analysis. Four databases (MEDLINE, Philosopher’s Index, EMBASE, CINAHL) were searched from April 2016 to April 2021.Results393 records were screened, with 72 studies eligible and included: 53 empirical studies, 17 structured reviews and 2 review protocols. 12/72 (17%) contained an explicit definition of ‘ethical challenge(s), two of which were shared, resulting in 11 unique definitions. Within these 11 definitions, four approaches were identified: definition through concepts; reference to moral conflict, moral uncertainty or difficult choices; definition by participants; and challenges linked to emotional or moral distress. Each definition contained one or more of these approaches, but none contained all four. 68/72 (94%) included studies used terms closely related to synonymously refer to ‘ethical challenge(s)’ within their manuscript text, with 32 different terms identified and between one and eight different terms mentioned per study.ConclusionsOnly 12/72 studies contained an explicit definition of ‘ethical challenge(s)’, with significant variety in scope and complexity. This variation risks confusion and biasing data analysis and results, reducing confidence in research findings. Further work on establishing acceptable definitional content is needed to inform future bioethics research. (shrink)

BackgroundDouble-blind peer review has been proposed as a possible solution to avoid implicit referee bias in academic publishing. The aims of this study are to analyse the demographics of corresponding authors choosing double-blind peer review and to identify differences in the editorial outcome of manuscripts depending on their review model.MethodsData includes 128,454 manuscripts received between March 2015 and February 2017 by 25 Nature-branded journals. We investigated the uptake of double-blind review in relation to (...) journal tier, as well as gender, country, and institutional prestige of the corresponding author. We then studied the manuscripts’ editorial outcome in relation to review model and author’s characteristics. The gender (male, female, or NA) of the corresponding authors was determined from their first name using a third-party service (Gender API). The prestige of the corresponding author’s institutions was measured from the data of the Global Research Identifier Database (GRID) by dividing institutions in three prestige groups with reference to the 2016 Times Higher Education (THE) ranking. We employed descriptive statistics for data exploration, and we tested our hypotheses using Pearson’s chi-square and binomial tests. We also performed logistic regression modelling with author update, out-to-review, and acceptance as response, and journal tier, author gender, author country, and institution as predictors.ResultsAuthor uptake for double-blind submissions was 12% (12,631 out of 106,373). We found a small but significant association between journal tier and review type (p value < 0.001, Cramer’s V = 0.054, df = 2). We had gender information for 50,533 corresponding authors and found no statistically significant difference in the distribution of peer review model between males and females (p value = 0.6179). We had 58,920 records with normalised institutions and a THE rank, and we found that corresponding authors from the less prestigious institutions are more likely to choose double-blind review (p value < 0.001, df = 2, Cramer’s V = 0.106). In the ten countries with the highest number of submissions, we found a large significant association between country and review type (p value < 0.001, df = 10, Cramer’s V = 0.189). The outcome both at first decision and post review is significantly more negative (i.e. a higher likelihood for rejection) for double-blind than single-blind papers (p value < 0.001, df = 1, Cramer’s V = 0.112 for first decision; p value < 0.001; df = 1, Cramer’s V = 0.082 for post-review decision).ConclusionsThe proportion of authors that choose double-blind review is higher when they submit to more prestigious journals, they are affiliated with less prestigious institutions, or they are from specific countries; the double-blind option is also linked to less successful editorial outcomes. (shrink)

BackgroundArtificial intelligence has been described as the “fourth industrial revolution” with transformative and global implications, including in healthcare, public health, and global health. AI approaches hold promise for improving health systems worldwide, as well as individual and population health outcomes. While AI may have potential for advancing health equity within and between countries, we must consider the ethical implications of its deployment in order to mitigate its potential harms, particularly for the most vulnerable. This scoping review addresses the following (...) question: What ethical issues have been identified in relation to AI in the field of health, including from a global health perspective?MethodsEight electronic databases were searched for peer reviewed and grey literature published before April 2018 using the concepts of health, ethics, and AI, and their related terms. Records were independently screened by two reviewers and were included if they reported on AI in relation to health and ethics and were written in the English language. Data was charted on a piloted data charting form, and a descriptive and thematic analysis was performed.ResultsUpon reviewing 12,722 articles, 103 met the predetermined inclusion criteria. The literature was primarily focused on the ethics of AI in health care, particularly on carer robots, diagnostics, and precision medicine, but was largely silent on ethics of AI in public and population health. The literature highlighted a number of common ethical concerns related to privacy, trust, accountability and responsibility, and bias. Largely missing from the literature was the ethics of AI in global health, particularly in the context of low- and middle-income countries.ConclusionsThe ethical issues surrounding AI in the field of health are both vast and complex. While AI holds the potential to improve health and health systems, our analysis suggests that its introduction should be approached with cautious optimism. The dearth of literature on the ethics of AI within LMICs, as well as in public health, also points to a critical need for further research into the ethical implications of AI within both global and public health, to ensure that its development and implementation is ethical for everyone, everywhere. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will (...) argue that the publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

We examine the ethical, social, and regulatory barriers that may hinder research on therapeutic potential of certain controversial controlled substances like marijuana, heroin, or ketamine. Hazards for individuals and society and potential adverse effects on communities may be good reasons for limiting access and justify careful monitoring of these substances. Overly strict regulations, fear of legal consequences, stigma associated with abuse and populations using illicit drugs, and lack of funding may, however, limit research on their considerable therapeutic potential. We (...) class='Hi'>review the surprisingly sparse literature and address the particular ethical concerns pertinent to research with illicit and addictive substances, such as undue inducement, informed consent, therapeutic misconception, and risk to participants, researchers, and institutions. We consider the perspectives of key research stakeholders and explore whether they may be infected with bias. We conclude by proposing an empirical research agenda to provide an evidentiary basis for ethical reasoning. (shrink)

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings (...) such as invited papers, early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. (shrink)

On the surface, developing a social psychology of science seems compelling as a way to understand how individual social cognition – in aggregate – contributes towards individual and group behavior within scientific communities (Kitcher, 2002). However, in cases where the functional input-output profile of psychological processes cannot be mapped directly onto the observed behavior of working scientists, it becomes clear that the relationship between psychological claims and normative philosophy of science should be refined. For example, a robust body of social (...) psychological research demonstrates implicit gender bias in the evaluation of others (e.g., Steinpreis, Anders, & Ritzke, 1999). Many expected implicit bias to be a major cause of women’s underrepresentation in math intensive fields of science; however, quantitative sociological research of hiring and manuscript and grant evaluation has discovered no gender disparity in outcomes (Ceci & Williams, 2011). Why might this be so? This paper will discuss methodological challenges that go beyond classic problems of external validity in extrapolating psychological effects and explanations to scientific communities. These problems include more complex external validity issues raised by the introduction of multi-process models of cognition (e.g., implicit versus explicit social cognition) as well as the reflexive role that folk and experimental theories of social psychology play in guiding the behavior of scientists at the individual and community level. (shrink)

Peer review and publication is one of the factors proposed in Daubert v. Merrell Dow Pharmaceuticals, Inc. as indicia of the reliability of scientific testimony. This Article traces the origins of the peer-review system, the process by which it became standard at scientific and medical journals, and the many roles it now plays. Additionally, the Author articulates the epistemological rationale for pre-publication peer-review and the inherent limitations of the system as a scientific quality-control mechanism. (...) The Article explores recent changes in science, in scientific publishing, and in the academy that have put the system under strain. The Author argues that Justice Blackmun's advice to courts - that peer-reviewed publication is relevant, but is not dispositive - is of little practical help. Instead, the Author suggests questions that courts should ask in assessing the significance of the fact that testimony is, or is not, based on peer-reviewed publication and illustrates with reference to another Bendectin case, Blum v. Merrell Dow Pharmaceuticals, Inc., where some of these questions were asked. (shrink)

Pre-publication certification through peer review stands in need of philosophical examination. In this paper, philosopher-psychologist Steven James Bartlett recalls the arguments marshalled four hundred years ago by English poet John Milton against restraint of publication by the "gatekeepers of publication," AKA today's peer reviewers.

Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data (as is increasingly required) to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this (...) confidentiality. If such confidentiality is breached by unethical reviewers who might steal or plagiarize the authors’ ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors’ and reviewers’ responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors’ intellectual ideas from predatory reviewers, and increase researchers’ awareness of the responsible reviewing of journal articles and reporting of biomedical research. (shrink)