Skip to main content
SpringerLink
Log in

The principles and practices of peer review

  • Published:
Science and Engineering Ethics Aims and scope Submit manuscript

Abstract

The principles and practices of research peer review are described. While the principles are fundamentally generic and apply to peer review across the full spectrum of performing institutions as well as to manuscript/proposal/program peer review, the focus of this paper is peer review of proposed and ongoing programs in federal agencies. The paper describes desireable characteristics and important intangible factors in successful peer review. Also presented is a heuristic protocol for the conduct of successful peer review research evaluations and impact assessments. Problems with peer review are then outlined, followed by examples of peer review of proposed and existing programs in selected federal agencies. Some peer review variants, such as the Science Court, are described, and then research requirements to improve peer review are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Salasin, J. et al. (June 1980) The Evaluation of Federal Research Programs, MITRE Technical Report MTR-80W123.

  2. Logsdon, J. M. and Rubin, C. B. (April 1985) An Overview of Federal Research Evaluation Activities, Report, The George Washington University, Wash., D. C. See also Logsdon, J. M. and Rubin. C. B. (1985) Federal Research Evaluation Activities, Abt Associates, Cambridge, MA.

    Google Scholar 

  3. Chubin, D. E. and Hackett, E. J. (1990) Peerless Science: Peer Review and U. S. Science Policy, State University of New York Press, Albany, NY.

    Google Scholar 

  4. Chubin, D. E. (February 1994) Grants Peer Review in Theory and Practice, in: Kostoff, R. N., (ed.), Evaluation Review, Special Issue on Research Impact Assessment, 18:1.

    Google Scholar 

  5. Kostoff, R. N. (1995) Federal Research Impact Assessment: Axioms, Approaches, Applications, Scientometrics 34:2.

    Article  Google Scholar 

  6. Barker, K. (1992) The ‘British Model’ — Evaluation by Professionals, in: Laredo, P. and Mustar. P. (eds), EC Handbook on Evaluation.

  7. Cicchetti, D. V. (1991) The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation, Behavioral and Brain Sciences 14:1.

    Google Scholar 

  8. Cole, S., Cole, J. and Simon, G. (November 1981) Chance and Consensus in Peer Review, Science 214.

  9. DOE, Procedures for Peer Review Assessments, Office of Energy Research, Office of Program Analysis, Report No. DOE/ST-0007P. Revised January 1993.

  10. Frazier, S. P. (March 1987) University Funding: Information on the Role of Peer Review at NSF and NIH. U.S. General Accounting Office Report No. GAO/RCED-87-87FS.

  11. Kostoff, R. N. (Summer 1996) The Handbook of Research Impact Assessment, Sixth Edition, DTIC Report Number ADA-296021.

  12. Ormala, E. (1989) Nordic Experiences of the Evaluation of Technical Research and Development. Research Policy 18.

  13. NAS (1992) The Government Role in Civilian Technology: Building a New Alliance, Committee on Science, Engineering, and Public Policy, National Academy of Sciences, National Academy Press, USA.

    Google Scholar 

  14. Kostoff, R. N. (1988) Evaluation of Proposed and Existing Accelerated Research Programs by the Office of Naval Research. IEEE Trans. of Engineering Management 35:4.

    Google Scholar 

  15. NIST (January 1991) Annual Report, 1990, Visiting Committee on Advanced Technology.

  16. Kostoff, R. N. (February 1994) Research Impact Assessment: Federal Peer Review Practices, in: Kostoff, R. N. (ed.), Evaluation Review, Special Issue on Research Impact Assessment, 18:1.

    Google Scholar 

  17. Roy, R. (1985) Funding Science: The Real Defects of Peer Review and an Alternative to It, Science, Technology, and Human Values 10:3.

    Google Scholar 

  18. King, J. (1987) A Review of Bibliometric and Other Science Indicators and Their Role in Research Evaluation, Journal of Information Science 13.

  19. Kostoff, R. N. (February 1994) Quantitative/Qualitative Federal Research Impact Evaluation Practices, Technological Forecasting and Social Change 45:2.

    Article  Google Scholar 

  20. DOE (March 1982) An Assessment of the Basic Energy Sciences Program, Office of Energy Research. Office of Program Analysis, Report No. DOE/ER-0123.

  21. Kostoff, R. N., Research Impact Assessment, Proceedings: Third International Conference on Management of Technology, Miami, FL, February 17–21, 1992. Larger text available from author.

  22. Bornstein, R. F. (1991) The Predictive Validity of Peer Review: A Neglected Issue, Behavioral and Brain Sciences 14:1.

    Google Scholar 

  23. Bornstein, R. F. (1991) Manuscript Review in Psychology: Psychometrics, Demand Characteristics, and an Alternative Model. Journal of Mind and Behaviour 12.

  24. NSF (1989) Science and Engineering Indicators — 1989, National Science Board Report NSB 89-1, GPO, Wash., D.C.

    Google Scholar 

  25. Kostoff, R. N. and Stanford, L. B. (April 1991) Program Funding Profiles under Budgetary Constraints, Research Evaluation 1:1.

    Google Scholar 

  26. Cozzens, S. E. (1987) Expert Review in Evaluating Programs, Science and Public Policy 14:2.

    Google Scholar 

  27. Kerpelman, L. C., and Fitzsimmons, S. J., Methods for the Strategic Evaluation of Research Programs: The State-of-the Art, and Annotated Bibliography, NSF Contract No. PRA 8400688. Abt Associates Inc., Cambridge, MA.

  28. Kostoff, R. N. (1993) Evaluating Federal R&D in the U. S., in: Bozeman, B. and Melkers, J., Eds., Assessing R&D Impacts: Method and Practice, Kluwer Academic Publishers, Norwell, MA.

    Google Scholar 

  29. Luukkonen-Gronow, T. (1987) Scientific Research Evaluation: A Review of Methods and Various Contexts of Their Application, R&D Management 17:3.

    Google Scholar 

  30. OTA, Research Funding as an Investment: Can We Measure the Returns, U. S. Congress, Office of Technology Assessment, OTA-TM-SET-36 (Wash., DC: U. S. GPO, April 1986).

    Google Scholar 

  31. NIST (1991) An Assessment of the National Institute of Standards and Technology Programs: FY 1990, Board on Assessment of NIST Programs, National Research Council, National Academy Press, Washington, D.C.

    Google Scholar 

  32. DOE (September 1988) Multiprogram Laboratory Appraisals, DOE ORDER 5000.2A.

  33. Ormala, E. (February 1994) Impact Assessment: European Experience of Qualitative Methods and Practices, in: Kostoff, R. N., (ed.), Evaluation Review, Special Issue on Research Impact Assessment 18:1.

    Google Scholar 

  34. DOE (May 1978) Evaluation of Alternate Magnetic Fusion Concepts 1977, DOE/ET-0047, U.S. Department of Energy, Assistant Secretary for Energy Technology, Office of Fusion Energy.

  35. Odeyale, C. O. (1993) Design and Development of a Knowledge-Based System for Effective and Unbiased Military Biomedical R&D Management, Ph.D. Dissertation, Walden University, 155 Fifth Ave. South. Minneapolis, MN 55401. USA.

    Google Scholar 

  36. Odeyale, C. O. and Kostoff, R. N. (1994) R&D Management Expert Networks: I. Knowledge Representation and Inference Strategies, HEURISTICS, the Journal of Knowledge Engineering and Technology 7:1.

    Google Scholar 

  37. Odeyale, C. O. and Kostoff, R. N. (1994) R&D Management Expert Networks: II. Prototype Construction and Validation. HEURISTICS, the Journal of Knowledge Engineering and Technology 7:1.

    Google Scholar 

  38. Luukkonen, T., and Stahle, B., (1993) Evaluation of Research Fields — Scientists’ Views, Nord 1993: 15, Nordic Council of Ministers, Copenhagen.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Office of Naval Research, 800 N. Quincy Street, 22217, Arlington, VA, USA

    Ronald N. Kostoff

Authors
  1. Ronald N. Kostoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

The views in this paper are solely those of the author and do not represent the views of the U.S. Department of the Navy.

This paper is based on a presentation at a workshop, “Advances in Peer Review Research”. American Association for the Advancement of Science Meeting, Baltimore, MD, February 9, 1996.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kostoff, R.N. The principles and practices of peer review. SCI ENG ETHICS 3, 19–34 (1997). https://doi.org/10.1007/s11948-997-0014-6

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11948-997-0014-6

Keywords

Search

Navigation