Skip to main content
SpringerLink
Log in

Peer review for journals: Evidence on quality control, fairness, and innovation

  • Published:
Science and Engineering Ethics Aims and scope Submit manuscript

Abstract

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings such as invited papers, early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abramowitz, S. I., Gomes, B. & Abramowitz, C. V. (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5, No. 3: 187–200. EX

    Article  Google Scholar 

  • Abrams, P. A. (1991) The predictive ability of peer review of grant proposals: The case of ecology and the US National Science Foundation. Social Studies of Science 21: 111–132. ER

    Google Scholar 

  • Arkes, H. (1996) The persistence of management folklore. Interfaces 26, No. 4: 42–44.

    Google Scholar 

  • Armstrong, J. S. (1980) Unintelligible management research and academic prestige. Interfaces 10 (April): 80–86. EXR

    Google Scholar 

  • Armstrong, J. S. (1982) Barriers to scientific contributions: The author’s formula. The Behavioral and Brain Sciences 5: 197–199. ER

    Google Scholar 

  • Armstrong, J. S. (1985) Long-Range Forecasting. John Wiley, New York.

    Google Scholar 

  • Armstrong, J. S. (1996) Management folklore and management science: On portfolio planning, escalation bias, and such (with commentaries). Interfaces 26, No. 4: 25–55.

    Google Scholar 

  • Armstrong, J. S. & Hubbard, R. (1991) Does the need for agreement among reviewers inhibit the publication of controversial findings? Behavioral and Brain Sciences 14 (March): 136–137. ER

    Google Scholar 

  • Atkinson, D. R., Furlong, M. J. & Wampold, B. E. (1982) Statistical significance, reviewer evaluations, and the scientific process: Is there a statistically significant relationship? Journal of Counseling Psychology 29, No. 2: 189–194. EX

    Article  Google Scholar 

  • Bakanic, V., McPhail, C. & Simon, R. J. (1990) If at first you don’t succeed: Review procedures for revised and resubmitted manuscripts. American Sociologist 21, No 4: 373–391. E

    Google Scholar 

  • Barber, B. (1961) Resistance by scientists to scientific discovery. Science 134: 596–602.

    Article  Google Scholar 

  • Begg, C. B. & Berlin, J. A. (1988) Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A 151: 419–463. E

    Google Scholar 

  • Beyer, J. M., Chanove, R. G. & Fox, W. B. (1995) The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal 38: 1219–1260. ER

    Article  Google Scholar 

  • Blank, R. M. (1991) The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. American Economic Review 81: 1041–1067. EXR

    Google Scholar 

  • Bradley, J. V. (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18: 31–34. E

    Google Scholar 

  • Broad, W. & Wade, N. (1982) Betrayers of the Truth. Simon and Schuster, New York.

    Google Scholar 

  • Campanario, J. M. (1995) On influential books and journal articles initially rejected because of negative referees’ evaluations. Science Communication 16 (March): 304–325. ER

    Google Scholar 

  • Chalmers, I. (1990) Underreporting research is scientific misconduct. Journal of the American Medical Association 263, No. 10: 1405–1408.

    Article  Google Scholar 

  • Chalmers, T. C., Frank, C. S. & Reitman, D. (1990) Minimizing the three stages of publication bias. Journal of the American Medical Association 263, No. 10: 1392–1395. E

    Article  Google Scholar 

  • Christiansen, D. (1978) The perils of publishing. IEEE Spectrum 15, No. 5: 27.

    Google Scholar 

  • Cho, M. K. & Bero, L. A. (1996) The quality of drug studies published in symposium proceedings. Annals of Internal Medicine 124, No. 5: 485–489. E

    Google Scholar 

  • Cicchetti, D. V. (1991) The reliability of peer review for manuscript and grant submissions: A cross disciplinary investigation. Behavioral and Brain Sciences 14 (March): 119–186. E

    Google Scholar 

  • Cohen, J. (1994) The earth is round (p<.05). American Psychologist 49: 997–1003.

    Article  Google Scholar 

  • Davidson, R. A. (1986) Source of funding and outcome of clinical trials. Journal of General Internal Medicine 1: 155–158. ER

    Google Scholar 

  • Dewald, W. G., Thursby, J. G. & Anderson, R. G. (1986) Replication in empirical economics: The Journal of Money, Credit, and Banking project. American Economic Review 76: 587–603. E

    Google Scholar 

  • Eichorn, P. & Yankauer, A. (1987) Do authors check their references? A survey of accuracy of references in three public health journals. American Journal of Public Health 77: 1011–1012. ER

    Google Scholar 

  • Epstein, W. M. (1990) Confirmational response bias among social work journals. Science, Technology, and Human Values 15: 9–38. EXR

    Google Scholar 

  • Evans, J. T., Nadjari, H. I. & Burchell, S. A. (1990) Quotational and reference accuracy in surgical journals: A continuing peer review problem. Journal of the American Medical Association 263, No. 10: 1353–1354. E

    Article  Google Scholar 

  • Fagan, W. T. (1990) To accept or reject: Peer review. Journal of Educational Thought 24: 103–113. ER

    Google Scholar 

  • Fletcher, R. H. & Fletcher, S. W. (1997) Evidence for the Effectiveness of Peer Review. Science and Engineering Ethics 3: 35–50. EX

    Google Scholar 

  • Franke, R. H., Edlund, T. W. & Oster, F. (1990) The development of strategic management: Journal quality and article impact. Strategic Management Journal 11: 243–253. ER

    Google Scholar 

  • Fuess, S. M. (1996) On replication in business and economics research: The QJBE case. Quarterly Journal of Business and Economics 35, No. 2: 3–13. E

    Google Scholar 

  • Friedman, P. J. (1990) Correcting the literature following fraudulent publication. Journal of the American Medical Association 263: 1416–1419. ER

    Article  Google Scholar 

  • Gans, J. S. & Shepherd, G. B. (1994) How are the mighty fallen: Rejected classic articles by leading economists. Journal of Economic Perspectives 8, No. 1: 165–179. E

    Google Scholar 

  • Garcia, J. (1981) Tilting at the paper mills of academe. American Psychologist 36, No. 2: 149–158.

    Article  Google Scholar 

  • Garfield, E. & Welljams-Dorof, A. (1990) The impact of fraudulent research on the scientific literature. Journal of the American Medical Association 263, No. 10: 1424–1426. E

    Article  Google Scholar 

  • Goodstein, L. & Brazis, K. (1970) Credibility of psychologists: An empirical study. Psychological Reports 27, No. 3: 835–838. E

    Google Scholar 

  • Gottfredson, S. D. (1978) Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934. EX

    Article  Google Scholar 

  • Greenwald, A. G. (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20. E

    Article  Google Scholar 

  • Hargens, L. L. (1990) Variation in journal peer review systems: possible causes and consequences. Journal of the American Medical Association 263: 1348–1352.

    Article  Google Scholar 

  • Harnad, S. (1979) Creative disagreement. The Sciences 19: 18–20.

    Google Scholar 

  • Hendrick, C. (1976) Editorial comment. Personality and Social Psychology Bulletin 2, No. 3: 207–208.

    Google Scholar 

  • Hibbitts, B. J. (1996a) Last writes? Re-assessing the law review in the age of cyberspace. http://www.law.pitt.edu/hibbitts/lastrev.htm; version 1.1, June 4, 1996; New York University Law Review 17: 615–688.

    Google Scholar 

  • Hibbitts, B. J. (1996b) Yesterday once more: Skeptics, scribes and the demise of law reviews. Akron Law Review 30 (forthcoming).

  • Holub, H. W., Tappeiner, G. & Eberharter, V. (1991) The iron law of important articles. Southern Economic Journal 58: 317–328. ER

    Article  Google Scholar 

  • Horrobin, D. F. (1982) A philosophically faulty concept which is proving disastrous for science. Behavioral and Brain Sciences 5, No. 2: 217–218.

    Google Scholar 

  • Horrobin, D. F. (1990) The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association 263 (March 9): 1438–1441.

    Article  Google Scholar 

  • Horrobin, D. F. (1996) Peer review of research grant application. Lancet (forthcoming).

  • Hubbard, R. & Armstrong, J. S. (1992) Are null results becoming an endangered species in marketing? Marketing Letters 3: 127–136. ER

    Article  Google Scholar 

  • Hubbard, R. & Armstrong, J. S. (1994) Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11: 233–248. ER

    Article  Google Scholar 

  • Hubbard, R. & Vetter, D. (1996) An empirical comparison of published replication research in accounting, economics, finance, management and marketing. Journal of Business Research 35: 153–164. ER

    Article  Google Scholar 

  • Jauch, L. R. & Wall, J. L. (1989) What they do when they get your manuscript: A survey of Academy of Management reviewer practices. Academy of Management Journal 32: 157–173. E

    Article  Google Scholar 

  • Juhasz, S., Calvert, E., Jackson, T., Kronick, D. A. & Shipton, J. (1975) Acceptance and rejection of manuscripts. IEEE Transactions of Professional Communications PC18: 177–184. E

    Google Scholar 

  • Kerr, S., Tolliver, J. & Petree, D. (1972) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20, No. 1: 132–141. ER

    Article  Google Scholar 

  • King, D. W., McDonald, D. D. & Roderer, N. K. (1981) Scientific Journals in the United States: Their Production, Use, and Economics. Hutchison Ross, Stroudsburg, Pa. E

    Google Scholar 

  • Kochler, J. J. (1993) The influence of prior beliefs on scientific judgments of evidence quality. Organizational Behavior and Human Decision Processes 56: 28–55. EXR

    Article  Google Scholar 

  • Kuhn, T. S. (1962) The Structure of Scientific Revolutions. University of Chicago Press, Chicago.

    Google Scholar 

  • Kupfersmid, J. & Wonderly, D. M. (1994) An Author’s Guide to Publishing Better Articles in Better Journals in the Behavioral Sciences. Clinical Psychology Publishing Co., Brandon, Vermont. ER

    Google Scholar 

  • Lau, R. R. (1994) An analysis of the accuracy of “trial heat” polls during the 1992 presidential election. Public Opinion Quarterly 58: 2–20.

    Article  Google Scholar 

  • Lindsey, D. (1978) The Scientific Publication System in Social Science. Jossey Bass, San Francisco. E

    Google Scholar 

  • Liversidge, A. (1989) PNAS publication of AIDS article spurs debate over peer review. The Scientist 3, No. 7: 4–5, 19.

    Google Scholar 

  • Lock, S. & Smith, J. (1986) Peer review at work. Scholarly Publishing 17, No. 4: 303–316. E

    Google Scholar 

  • Lock, S. & Smith, J. (1990) What do peer reviewers do? Journal of the American Medical Association 263, No. 10: 1341–1343. E

    Article  Google Scholar 

  • Lloyd, M. E. (1990) Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis 23: 539–543. EX

    Article  Google Scholar 

  • MacNealy, M. S., Speck, B. W. & Clements, N. (1994) Publishing in technical communication journals from the successful author’s point of view. Technical communication 41, No. 2: 240–259. E

    Google Scholar 

  • Mahoney, M. (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1: 161–175. EXR

    Article  Google Scholar 

  • Marsh, H. W. & Ball, S. (1989) The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education 57, No. 2: 151–169. E

    Google Scholar 

  • McCloskey, D. N. & Ziliak, S. T. (1996) The standard error of regressions. Journal of Economic Literature 34 (March): 97–114. E

    Google Scholar 

  • McNutt, R. A., Evans, A. T., Fletcher, R. H. & Fletcher, S. W. (1990) The effects of blinding on the quality of peer review: A randomized trial. Journal of the American Medical Association 263, No. 10: 1371–1376. EX.

    Article  Google Scholar 

  • Munley, P. H., Sharkin, B. & Gelso, C. J. (1988) Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology. Journal of Counseling Psychology 35, No. 2: 198–202. E

    Article  Google Scholar 

  • Murray, G. D. (1988) The task of a statistical referce. British Journal of Surgery 75: 664–667. E

    Google Scholar 

  • Needleman, H. L. (1992) Salem comes to the National Institutes of Health: Notes from inside the crucible of scientific integrity. Pediatrics 90, No. 6: 977–981.

    Google Scholar 

  • Newcombe, R. G. (1987) Towards a reduction in publication bias. British Medical Journal 295 (12 September): 656–659.

    Google Scholar 

  • Patterson, S. C. & Smithey, S. K. (1990) Monitoring scholarly journal publication in political science: The role of the APSR. PS: Political Science and Politics 23: 647–656. ER

    Article  Google Scholar 

  • Perlman, D. (1982) Reviewer “bias”: Do Peters and Ceci protest too much? The Behavioral and Brain Sciences 5: 231–232. E

    Article  Google Scholar 

  • Pfeifer, M. P. & Snodgrass, G. L. (1990) The continued use of retracted, invalid scientific literature. Journal of the American Medical Association 263, No. 10: 1420–1423. E

    Article  Google Scholar 

  • Peters, D. P. & Ceci, S. J. (1982) Peer-review practices of psychology journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195. EX

    Article  Google Scholar 

  • Pressman, S. (1994) Simultaneous multiple journal submissions: The case against. American Journal of Economics and Sociology 53: 316–333.

    Article  Google Scholar 

  • Rodman, H. & Mancini, J. A. (1977) Errors, manuscripts, and equal treatment. Research in Higher Education 7: 369–374. ER

    Article  Google Scholar 

  • Roediger, H. L. (1987) The role of journal editors in the scientific process. In: Jackson, D. N. & Rushton, J. P. (eds.), Scientific Excellence. Sage Publications, London.

    Google Scholar 

  • Rowney, J. A. & Zenisek, T. J. (1980) Manuscript characteristics influencing reviewers’ decisions. Canadian Psychology 21, 17–21. ER

    Google Scholar 

  • Rushton, J. P. & Ankney, C. D. (1996) Brain size and cognitive ability: Correlations with age, sex, social class, and race. Psychonomic Bulletin and Review 3, No. 1: 21–36.

    Google Scholar 

  • Salsburg, D. S. (1985) The religion of statistics as practiced in medical journals. American Statistician 39: 220–223. E

    Article  Google Scholar 

  • Shea, C. (1996) Psychologists debate accuracy of ‘significance test. The Chronicle of Higher Education 42 (August 16): A12 & A17.

  • Simon, R., Bakanic, V. & McPhail, C. (1986) Who complains to editors and what happens. Sociological Inquiry 56: 259–271. ER

    Article  Google Scholar 

  • Smart, R. G. (1964) The importance of negative results in psychological research. Canadian Psychologist 5: 225–232. E

    Article  Google Scholar 

  • Speck, B. W. (1993) Publication Peer Review. Westport, Connecticut.

    Google Scholar 

  • Spiegel-Rosing, I. (1977) Bibliometric and content analysis. Social Studies of Science 7: 97–113. E

    Google Scholar 

  • Stamps, A. E. III (1997) Advances in Reer Review Research: An introduction. Science and Engineering Ethics 3: 3–10. ER

    Google Scholar 

  • Stamps, A. E. III (1997) Using a Dialectical Scientific Brief in Peer Review. Science and Engineering Ethics 3: 85–98.

    Google Scholar 

  • Sterling, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa. Journal of the American Statistical Association, 54: 30–34. ER

    Article  Google Scholar 

  • Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician 49: 108–112. ER

    Article  Google Scholar 

  • Stewart, W. W. & Feder, N. (1987) The integrity of the scientific literature. Nature 325: 207–214. ER

    Article  Google Scholar 

  • Szenberg, M. (1994) Disseminating scholarly output: The case for eliminating the exclusivity of journal submissions. American Journal of Economics and Sociology 53: 303–315.

    Article  Google Scholar 

  • Taubes, G. (1994) Peer review in cyberspace. Science 266: 967.

    Article  Google Scholar 

  • Tetlock, P. E. & Kim, J. I. (1987) Accountability and judgment process in a personality prediction task. Journal of Personality and Social Psychology 52: 700–709.

    Article  Google Scholar 

  • Wade, N. (1976) IQ and heredity: Suspicion of fraud beclouds classic experiment. Science 194: 916–919.

    Article  Google Scholar 

  • Weiss, D. J. (1989) An experiment in publication: Advance publication review. Applied Psychological Measurement 13: 1–7.

    Google Scholar 

  • Wilson, J. D. (1978) Peer review and publication. Journal of Clinical Investigation 61, 1697–1701.

    Article  Google Scholar 

  • Yankauer, A. (1985) Peering at peer review. CBE Views 8, No. 2: 7–10. ER

    Google Scholar 

  • Yankauer, A. (1990) Who are the peer reviewers and how much do they review? Journal of the American Medical Association 263: 1338–1340. ER

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Wharton School, University of Pennsylvania, 19104, Philadelphia, PA, USA

    J. Scott Armstrong

Authors
  1. J. Scott Armstrong
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

The author, a professor of marketing at the Wharton School since 1968, was a founder editor of the Journal of Forecasting and the International Journal of Forecasting.

This paper is based on a presentation at a workshop, “Advances in Peer Review Research”, American Association for the Advancement of Science Meeting, Baltimore, MD, February 9, 1996.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Armstrong, J.S. Peer review for journals: Evidence on quality control, fairness, and innovation. SCI ENG ETHICS 3, 63–84 (1997). https://doi.org/10.1007/s11948-997-0017-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11948-997-0017-3

Keywords

Search

Navigation