“Quality” refers nominatively to a standard of performance. Quality is the central idea that differentiates speech protected by academic freedom (the right to worthwhile utterances) from constitutionally protected speech (the right to say anything at all). Extant documents and discussions state that professional peers determine quality based on norms of a field. But professional peers deem utterances and activities as consonant with quality only in reference to criteria that establish meaning of the term. In the absence of articulation, these criteria (...) are ambiguous. Consequently, there exists recurrent confusion about what faculty members have a defensible right to say and do. This article develops an ontology of quality in reference to higher education teaching, a component of academic careers generally not subject to extensive peer review and where instructors thereby exercise considerable autonomy. The ontology identifies three criteria that bound quality: constraint, context, and amplitude. Boundedness exists only insofar as boundaries are controlled. The article examines two types of problems in professional control that affect quality: slippage and overreach. Both are instances of organizational deviance and abrogation of professional ethics. It is argued that the patterns threaten the structural integrity and public confidence of faculty, fields, and higher education institutions. (shrink)

Charles Perrow used the term normal accidents to characterize a type of catastrophic failure that resulted when complex, tightly coupled production systems encountered a certain kind of anomalous event. These were events in which systems failures interacted with one another in a way that could not be anticipated, and could not be easily understood and corrected. Systems of the production of expert knowledge are increasingly becoming tightly coupled. Unlike classical science, which operated with a long time horizon, many current forms (...) of expert knowledge are directed at immediate solutions to complex problems. These are prone to breakdowns like the kind discussed by Perrow. The example of the Homestake mine experiment shows that even in modern physics complex systems can produce knowledge failures that last for decades. The concept of knowledge risk is introduced, and used to characterize the risk of failure in such systems of knowledge production. (shrink)

Prediction markets, which trade contracts based on the results of predictions, have been remarkably successful in predicting the results of political events. A number of proposals have been made to extend prediction markets to scientific questions, and some small-scale science prediction markets have been implemented. Advocates for science prediction markets argue that they could alleviate problems in science such as bias in peer review and epistemically unjustified consensus. I argue that bias in peer review and epistemically unjustified consensuses are genuine (...) problems in science, and that current attempts to solve them fail in practical circumstances. Prediction markets do show some promise for answering scientific questions while avoiding the pitfalls of consensus and peer review. However, there are strong reasons to believe that science prediction markets will not perform nearly so well as election prediction markets, and implementing prediction markets on a large scale could have significant unintended consequences for the organization of scientific research. The promise of prediction markets to solve current problems in assessing scientific claims is therefore largely illusory. (shrink)

Large differences in scientific productivity between male and female researchers have not yet been explained satisfactorily. This study finds that child care and lack of research collaboration are the two factors that cause significant gender differences in scientific publishing. Women with young children and women who do not collaborate in research with other scientists are clearly less productive than both their male and female colleagues.

This article analyzes the transformation of Minerva from an intellectual towards a scholarly journal by making use of bibliometric methods. The aim is to provide some empirical insights that help to understand what properties of the journal changed in the course of this transformation process. Minerva was one of the first journals that reflected on science and its role in society and science policy in particular. Analyzing the development of the journal sheds light on the emergence of science (policy) studies (...) and on Minerva’s role as a forerunner in this field. In a first step, the methods will be described. The second section provides some empirical results of the publication output of Minerva and its relations to other journals in the field. The empirical findings are put into a broader perspective in the concluding third section. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Peer review is a topic of considerable concern to many researchers, and there is a correspondingly large body of research on the topic. This issue of Science and Engineering Ethics presents recent work on peer review that is both grounded in empirical science and is applicable to policy decisions. This research raises two basic questions; (a) how does current peer review operate, and (b) how can it be improved? Topics addressed include descriptions of how peer review is used in Federal (...) agencies. whether peer review leads to better manuscripts, demographic characteristics of authors or reviewers (status or institutional affiliation), blinding of reviewers, authors, or results, reliability and consistency of reviews, accepting a paper before the study is done, simultaneous submission, and use of dispute resolution procedures such as scientific dialectical and pleading protocols. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Two analogies are at the foundation of editions of writings of scientists, technologists and physicians. Both are exemplified in the collection of ‘works’, texts of printed finished versions of contributions. The literary analogy is that of authorship, of the creation of a significant assemblage of words and other symbols. Assemblages of monographs and articles of a scientist are functionally no different than comparable arrays of the writings of theologians, philosophers, poets, novelists and historians.

The conventional history of modern scientific peer review begins with the censorship practices of the Royal Society of London in the 1660s. This article traces one strand of the “prehistory” of peer review in the writings of John Amos Comenius and other members of the Hartlib circle, a precursor group to the Royal Society of London. These reformers appear to have first envisioned peer review as a technique for theologians, only later proposing to apply it to philosophy. The importance of (...) peer review was as a technique that would permit a community of theologians or philosophers to resolve disputes internally rather than publicly, since public disputation would (they believed) sow doubt, error, and confusion, and disrupt the social order. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)

Rooted in altruism theory, the purpose of the double-blind academic journal peer-review process is to: assess the quality of scientific research, minimize the potential for nepotism, and; advance the standards of research through high-quality, constructive feedback. However, considering the limited, if any, public recognition and monetary incentives that referees receive for reviewing manuscripts, academics are often reluctant to squander their limited time toward peer reviewing manuscripts. If they do accept such invitations, referees, at times, do not invest the appropriate time (...) needed and, as a result, scantily review manuscripts, which adversely affects the quality of the review. In addition, given that authors’ identities are not blind to journal editors, there is the potential for bias toward well-established academics from highly-ranked institutions. As a result of these issues, the aims of the academic journal review process are currently not being fulfilled. To rectify these issues, several recommendations, namely: single-blind the editors, pay reviewers, standardize the review process, increase the acceptance standards at academic conferences, and provide constructive feedback, are offered. (shrink)

The main aim of this article is to start a discussion of social pattern, a term that is commonly used in sociology but not specified or defined. The key question can be phrased as follows: Is it possible to transform the notion of social pattern from its current status in sociology as a proto-concept into a fully worked out concept? And if so, how can this be done? To provide material for the discussion we begin by introducing a few different (...) types of patterns that are currently being used (patterns in nature, cultural patterns, statistical patterns, and computationally generated patterns). This is followed by a suggestion for what a strictlysociologicalconcept of social pattern may look like. A useful and theoretically solid concept of social pattern can in our view be constructed by basing it on Weber’s concept of social action. This means thatboththe behavior of the actorsandthe meaning these invest their behavior with must be taken into account. The article ends with a brief discussion of how to use the concept of social patterns in an effective way and what may endanger such a use. (shrink)

For over 50 years, Minerva has been one of the leading independent journals in the study of ‘science, learning and policy’. Its pages have much to say about the origins and conduct of the ‘intellectual Cold War’, the defence of academic freedom, the emergence of modernization theory, and pioneering strategies in the social studies of science. This paper revisits Minerva through the life and times of its founding Editor, Edward Shils, and traces his influence on its early years – from (...) its association with the Congress for Cultural Freedom in the 1950s, to the higher reaches of research policy in the 1990s. At the close of his life, Minerva continued to espouse Shils’ commitment to what he saw as the fundamental Enlightenment traditions of consensus, civility, and community. In the 21st century, with the achievements of science producing rapid change in every walk of life, his legacy so far retains an established place in the history of scholarship. Whether that legacy will endure – and if so, what role Minerva will play in its defence – remain key questions for the coming generation. (shrink)

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain (...) unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review procedures and (...) their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Select groups and organizations embrace practices that perpetuate their inferiority. The result is the phenomenon we call “mediocrity.” This article examines the conditions under which mediocrity is selected and maintained by groups over time. Mediocrity is maintained by a key social process: the marginalization of the adept, which is a response to the group problem of what to do with the highly able. The problem arises when a majority of a group is comprised of average members who must decide what (...) to do with high performers in the group. To solve this problem, reward systems are subverted to benefit the less able and the adept are cast as deviant. Marginalization is a resolution of two tensions: marginalization of the adept for their behavior, and protection from the adept for the mediocre. The American research university is used as an example to describe the phenomenon and to formulate a theoretic argument. The forms and consequences of marginalization are discussed. Marginalizing the adept illustrates an anti-meritocratic behavioral pattern which serves to sustain social systems on which all people, however able, depend. (shrink)

Efforts to foster transparency in biopharmaceutical regulation are well underway: drug manufacturers are, for example, legally required to register clinical trials and share research results in the United States and Europe. Recently, the policy conversation has shifted toward the disclosure of clinical trial data, not just trial designs and basic results. Here, I argue that clinical trial registration and disclosure of clinical trial data are necessary but insufficient. There is also a need to ensure that regulatory decisions that flow from (...) clinical trials — whether positive (i.e., product approvals) or negative (i.e., abandoned products, product refusals, and withdrawals) — are open to outside scrutiny. Further, a jurisprudence of drug regulation is needed. I develop two arguments motivated by (1) innovation concerns and (2) the value of good governance in support of openly publishing all final decisions for approved, abandoned, refused, and withdrawn products. After articulating why greater transparency in regulatory decision-making is needed, I distil four essential features of a jurisprudence of drug regulation that prescribe policy changes in terms not only of the transparency of regulatory outcomes and the underlying reasoning, but also regulatory organization. (shrink)

Efforts to ensure greater transparency in the regulation of “drugs” are well underway. For example, laws in the United States and Europe now require registration of most clinical trials beyond phase 1. Yet instances of avoidable harm to patients continue to arise. In response, calls for disclosure of clinical trial data in the form of “clinical study reports,” not just trial designs and basic results, are growing. In this paper, I argue that disclosure of clinical trial data is necessary but (...) insufficient. Rather, the regulatory decisions that flow from those trial data —whether positive or negative —should also be open to outside scrutiny provided they are final in nature. (shrink)

The communist norm requires that scientists widely share the results of their work. Where did this norm come from, and how does it persist? Michael Strevens provides a partial answer to these questions by showing that scientists should be willing to sign a social contract that mandates sharing. However, he also argues that it is not in an individual credit-maximizing scientist's interest to follow this norm. I argue against Strevens that individual scientists can rationally conform to the communist norm, even (...) in the absence of a social contract or other ways of socially enforcing the norm, by proving results to this effect in a game-theoretic model. This shows that the incentives provided to scientists through the priority rule are sufficient to explain both the origins and the persistence of the communist norm, adding to previous results emphasizing the benefits of the incentive structure created by the priority rule. (shrink)

This study seeks to examine the impact of alumni connections between the evaluators and evaluatees on the results of peer review ratings for the Korean national R&D project and selection success rate. Specifically, this study analyzed the evaluation results of 8,402 research proposal entries submitted between 2007 and 2011 for the “general researchers support project,” all in the Natural Science and Engineering areas and sponsored by the National Research Foundation of Korea. Each proposal entry was evaluated by three evaluators, and (...) approximately 39 percent of the proposals had at least one evaluator from the same university. The results of this study showed that evaluators have a tendency to give relatively high scores to research proposals submitted by the alumni of the same universities as their alma mater. Also, when an evaluator from the same university as an evaluatee was included in the evaluator group, the results of this study showed that the percentage of entry submissions was higher compared to the contrary. Such results show that in the process of peer review–based research proposal evaluations for national R&D projects, alumni connections have significant influence on evaluation results in South Korea. (shrink)

BackgroundTriggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform (...) the implementation of innovations.MethodsWe carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software.ResultsAt the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives.ConclusionOur study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)