Skip to main content
SpringerLink
Log in

The Group Calibration Index: a group-based approach for assessing forecasters’ expertise when external outcome data are missing

  • Published:
Theory and Decision Aims and scope Submit manuscript

Abstract

The Group Calibration Index (GCI) provides a means of assessing the quality of forecasters’ predictions in situations that lack external feedback or outcome data. The GCI replaces the missing outcome data with aggregated ratings of a well-defined reference group. A simulation study and two experiments show how the GCI classifies forecaster performance and distinguishes between forecasters with restricted information and those with complete information. The results also show that under certain circumstances, where members of the reference group have high-quality information, the new GCI will outperform expert classification that is based on traditional calibration indices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Brier G. W. (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 75: 1–3

    Article  Google Scholar 

  • Dawes R. M. (1994) House of cards: Psychology and psychotherapy built of myth. Free Press, New York

    Google Scholar 

  • Fischer I., Budescu D. V. (2005) When “do those who know more also know more about how much they know”: The development of confidence and performance in decision-making. Organizational Behavior and Human Decision Processes 98(1): 39–53

    Article  Google Scholar 

  • Gustafson D. H., Shukla R. K., Delbecq A., Walster G. W. (1973) A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance 9: 280–291

    Article  Google Scholar 

  • Harvey N., Fischer I. (2005) Development of experience-based judgment and decision making: The role of Outcome Feedback. In: Betsch T., Haberstroh S. (eds) The routines of decision making. Erlbaum, Mahwah, NJ, pp 119–137

    Google Scholar 

  • Keren G. (1987) Facing uncertainty in the game of bridge: A calibration study. Organizational Behavior and Human Decision Processes 39: 98–114

    Article  Google Scholar 

  • Larrick R. P., Soll B. J. (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science 52(1): 111–127

    Article  Google Scholar 

  • Lichtenstein S., Fischhoff B. (1977) Do those who know more also know more about they know?. Organizational Behavior and Human Performance 20: 159–183

    Article  Google Scholar 

  • Murphy A. H. (1973) A new vector partition of the probability score. Journal of Applied Meteorology 12: 595–600

    Article  Google Scholar 

  • Oskamp, S. (1965). Overconfidence in case-study judgments. The Journal of Consulting Psychology, 29, 261–265. Reprinted in: Kahneman, D., Slovic, P., Tversky, A. (1982). Judgment under Uncertainty, 287–293. (Cambridge: Cambridge University Press).

    Google Scholar 

  • Prelec D. (2004) A Baysian truth serum for subjective data. Science 306: 462–466

    Article  Google Scholar 

  • Rowe G., Wright G. (2001) Expert opinions in Forecasting: The Role of the Delphi Technique. In: Armstrong J. S. (eds) Principles of forecasting—a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston, pp 125–144

    Google Scholar 

  • Shuford E. H. (1961) Absolute judgments of discrete quantities randomly distributed over time. Journal of Experimental Psychology 61: 430–436

    Article  Google Scholar 

  • Sterman J. (1994) Learning in and about complex systems. System dynamics Review 16: 291–330

    Article  Google Scholar 

  • Stevens S. S. (1975) Psychophysics: Introduction to its perceptual, neural, and social prospects. John Wiley & Sons, New York, pp 13–36

    Google Scholar 

  • Van de Ven A. H., Delbecq A. L. (1974) The effectiveness of nominal, delphi, and interacting group decision making processes. The Academy of Management Journal 17: 605–621

    Article  Google Scholar 

  • Vincenz, S. (1955). Bałaguły, On the High Uplands, translated by H. C. Stevens (pp. 248–267). New York: Roy

  • Wagenaar W. A., Keren G. B. (1985) Calibration of probability assessment by professional Blackjack dealers, statistical experts, and lay people. Organizational Behavior and Human Decision Processes 36: 406–416

    Article  Google Scholar 

  • Wallsten T. S., Budescu D. V. (1983) Encoding subjective probabilities: A psychological and psychometric review. Management Science 29: 151–173

    Article  Google Scholar 

  • Yates J. F. (1982) External correspondence: Decompositions of the mean probability score. Organizational-Behavior-and-Human-Decision-Processes 30: 132–156

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Haifa, Haifa, Israel

    Ilan Fischer

  2. The Open University of Israel, Raanana, Israel

    Ravid Bogaire

Authors
  1. Ilan Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ravid Bogaire
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilan Fischer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fischer, I., Bogaire, R. The Group Calibration Index: a group-based approach for assessing forecasters’ expertise when external outcome data are missing. Theory Decis 73, 671–685 (2012). https://doi.org/10.1007/s11238-011-9265-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11238-011-9265-4

Keywords

Search

Navigation