A Kuhnian Critique of Psychometric Research on Peer Review

Philosophy of Science 79 (5):859-870 (2012)
  Copy   BIBTEX

Abstract

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers.

Links

PhilArchive



    Upload a copy of this work     Papers currently archived: 92,907

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.
The principles and practices of Peer review.Ronald N. Kostoff - 1997 - Science and Engineering Ethics 3 (1):19-34.
Evidence for the effectiveness of Peer review.Robert H. Fletcher & Suzanne W. Fletcher - 1997 - Science and Engineering Ethics 3 (1):35-50.
Using a dialectical scientific brief in Peer review.Arthur Stamps - 1997 - Science and Engineering Ethics 3 (1):85-98.
‘Peer review’ culture.Malcolm Atkinson - 2001 - Science and Engineering Ethics 7 (2):193-204.

Analytics

Added to PP
2012-07-24

Downloads
90 (#193,469)

6 months
14 (#199,860)

Historical graph of downloads
How can I increase my downloads?

Author's Profile

Carole J. Lee
University of Washington

Citations of this work

Is Peer Review a Good Idea?Remco Heesen & Liam Kofi Bright - 2021 - British Journal for the Philosophy of Science 72 (3):635-663.
Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.
The social dimensions of scientific knowledge.Helen Longino - 2008 - Stanford Encyclopedia of Philosophy.
Commensuration Bias in Peer Review.Carole J. Lee - 2015 - Philosophy of Science 82 (5):1272-1283,.

View all 12 citations / Add more citations

References found in this work

Studies in the Way of Words.Paul Grice - 1989 - Philosophy 65 (251):111-113.
Logic and Conversation.H. Paul Grice - 1989 - In Studies in the Way of Words. Harvard University Press. pp. 22-40.
Bias in Peer Review.Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang & Blaise Cronin - 2013 - Journal of the American Society for Information Science and Technology 64 (1):2-17.

View all 15 references / Add more references