Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Cognitive Science 39 (2):405-433 (2015)

Abstract

Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots . Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal considerable variance within performance categories, differences in the process used as evidence in support of a performance rating, different numbers and types of facts identified, and differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations

Download options

PhilArchive



    Upload a copy of this work     Papers currently archived: 72,855

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Analytics

Added to PP
2014-07-23

Downloads
15 (#704,433)

6 months
1 (#386,001)

Historical graph of downloads
How can I increase my downloads?

Citations of this work

No citations found.

Add more citations

Similar books and articles

Peer Assessment of Children's Prosocial Behaviour.Susan Hayes Greener - 2000 - Journal of Moral Education 29 (1):47-60.
Lyotard’s Performance.Robin Usher - 2006 - Studies in Philosophy and Education 25 (4):279-288.
Future of Ethically Effective Leadership.Chaudhary Imran Sarwar - 2013 - Journal of Business Ethics 113 (1):81-89.
Expanding Expertise: Investigating a Musician’s Experience of Music Performance.Andrew Geeves, Doris Mcllwain, John Sutton & Wayne Christensen - 2010 - ASCS09: Proceedings of the 9th Conference of the Australasian Society for Cognitive Science:106-113.
Performance, Promotion and Information.Zajac Jaroslav - 2004 - Journal of Business Ethics 50 (2):187-198.
Use and Misuse of Metrics in Research Evaluation.Ronald N. Kostoff - 1997 - Science and Engineering Ethics 3 (2):109-120.
Performing Society.Peter Snow - 2010 - Thesis Eleven 103 (1):78-87.