Skip to main content
SpringerLink
Log in

Accommodation, prediction and replication: model selection in scale construction

  • Published:
Synthese Aims and scope Submit manuscript

Abstract

In psychology, measurement instruments are constructed from scales, which are obtained on the grounds of exploratory and confirmatory factor analysis. Looking at the literature, one can find various recommendations regarding how these techniques should be used during the scale construction process. Some authors suggest to use exploratory factor analysis on the entire data set while others advice to perform an internal cross-validation by randomly splitting the data set in two and then either perform exploratory factor analysis on both parts or exploratory factor analysis on the first part and confirmatory factor analysis on the other. In spite of all these divergent recommendations, there is no consensus on which method yields the best result. In this paper, we analyze this issue in light of the prediction versus accommodation debate and argue that the answer to this question depends on one’s conception of the criteria that should be used to achieve the goals of the scientific enterprise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. This paper is primarily concerned with the issue of model selection during scale construction in psychology. With that being said, our analysis should not be considered as restricted only to that field. Our analysis can be extended to other disciplines that use factor analytic methods for the development of measurement instruments (e.g. sociology, education), although some idiosyncratic differences might occur.

  2. Some items might be reversed score items.

  3. See Hood (2013) for an analysis of the issue of realism with respect to measurement models.

  4. In such a measurement model, the items are not used to predict the latent factors. As we will see, a factorial structure is a space where the items are the points.

  5. Note that Penny’s and Rosie’s partitions of D can be different.

  6. Hitchcock and Sober’s analysis takes place within the context of an instrumentalist perspective and, as such, they consider that science should aim at predictive accuracy (see Hitchcock and Sober 2004, pp. 2–3). It should be noted, however, that our analysis is independent of the realism/instrumentalism debate. As we will see, our analysis only relies on the assumption that empirical adequacy is, among others, one goal of the scientific enterprise, and that predictive accuracy is, among others, an indicator of the overall empirical adequacy of a theory.

  7. See also Gardner (1982) for a discussion of novelty.

  8. Conceptually, part of the unexplained variance might be due to something different than error.

  9. Even though the hypothesis of continuity is often violated, it has been shown that, under the assumption of normality, violation of the assumption of continuity is likely negligible (cf. Byrne 2012).

  10. There are other extraction techniques that can be used in cases where the assumption of normality is violated (cf. Flora et al. 2012). Consequently, despite their importance, these two conditions are not necessary for the use of factor analysis during scale construction.

  11. It has been argued by Michell (1997) that, given the violation of the assumption of continuity, measurement of psychological attributes is not possible (see also Michell 2003, 2004). His argument revolves around the assumption that only continuous quantities can be measured and, since the hypothesis that psychological attributes are continuous is not tested, it follows that one is not justified to believe that it is possible to measure psychological attributes. Borsboom and Mellenbergh (2004) answered this objection and showed that this hypothesis is actually tested, though not in isolation.

  12. It should be mentioned that some authors argued that an algorithm that searches for pure measurement models should be used rather than factor analysis techniques to obtain a measurement model (e.g. Silva et al. 2006; Kummerfeld et al. 2014; see also Murray-Watters and Glymour 2015). The analysis of these alternative techniques is beyond the scope of this paper. We will concentrate on factor analysis given its place with regard to scale construction in the psychology literature.

  13. Once a scale has been properly validated, researchers can use different measurement models and study the relationships between the latent factors through structural equation modeling. We will leave the analysis of model selection with respect to structural models for future research.

  14. To some extent, trying to replicate results from previous studies can be understood as a prediction that the results can be generalized.

  15. Thus understood, replication amounts to the reproduction of a result using the same technique.

  16. This would not be the case if one were to use principal component analysis instead of EFA.

References

  • Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4), 433–459.

    Article  Google Scholar 

  • Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332.

    Article  Google Scholar 

  • Aron, A., Coups, E. J., & Aron, E. N. (2013). Statistics for psychology (6th ed.). London: Pearson.

    Google Scholar 

  • Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119.

    Article  Google Scholar 

  • Autzen, B. (2016). Significance testing, p-values and the principle of total evidence. European Journal for Philosophy of Science, 6(2), 281–295.

    Article  Google Scholar 

  • Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825–829.

    Article  Google Scholar 

  • Borsboom, D. (2008). Latent variable theory. Measurement, 6(1–2), 25–53.

    Google Scholar 

  • Borsboom, D., & Mellenbergh, G. J. (2004). Why psychometrics is not pathological: A comment on Michell. Theory & Psychology, 14(1), 105–120.

    Article  Google Scholar 

  • Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219.

    Article  Google Scholar 

  • Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., et al. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.

    Article  Google Scholar 

  • Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York: Guilford Publications.

    Google Scholar 

  • Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258.

    Article  Google Scholar 

  • Byrne, B. M. (2012). Structural equation modeling with Mplus. New York: Routledge.

    Google Scholar 

  • Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56(5), 754–761.

    Article  Google Scholar 

  • Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1–9.

    Google Scholar 

  • Cudeck, R., & Browne, M. W. (1983). Cross-validation of covariance structures. Multivariate Behavioral Research, 18(2), 147–167.

    Article  Google Scholar 

  • Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174.

    Article  Google Scholar 

  • Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

    Article  Google Scholar 

  • Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.

    Google Scholar 

  • Finch, J. F., & West, S. G. (1997). The investigation of personality structure: Statistical models. Journal of Research in Personality, 31(4), 439–485.

    Article  Google Scholar 

  • Flora, D. B., LaBrish, C., & Chalmers, R. P. (2012). Old and new ideas for data screening and assumption testing for exploratory and confirmatory factor analysis. Frontiers in Psychology, 3(55), 1–21.

    Google Scholar 

  • Forster, M. R. (2002). Predictive accuracy as an achievable goal of science. Philosophy of Science, 69(3), S124–S134.

    Article  Google Scholar 

  • Forster, M., & Sober, E. (1994). How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. British Journal for the Philosophy of Science, 45(1), 1–35.

    Article  Google Scholar 

  • Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19(6), 975–991.

    Article  Google Scholar 

  • Gardner, M. R. (1982). Predicting novel facts. The British Journal for the Philosophy of Science, 33(1), 1–15.

    Article  Google Scholar 

  • Glymour, C. (2001). The mind’s arrows: Bayes nets and graphical causal models in psychology. Cambridge: MIT Press.

    Book  Google Scholar 

  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121.

    Article  Google Scholar 

  • Hitchcock, C., & Sober, E. (2004). Preduction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science, 55(1), 1–34.

    Article  Google Scholar 

  • Hood, S. B. (2013). Psychological measurement and methodological realism. Erkenntnis, 78(4), 739–761.

    Article  Google Scholar 

  • Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185.

    Article  Google Scholar 

  • Howell, D. C. (2010). Statistical methods for psychology (7th ed.). Wadsworth: Cengage Learning.

    Google Scholar 

  • Hu, L.-T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453.

    Article  Google Scholar 

  • Huck, S. W. (2012). Reading statistics and research (6th ed.). London: Pearson.

    Google Scholar 

  • Hurley, A. E., Scandura, T. A., Schriesheim, C. A., Brannick, M. T., Seers, A., Vandenberg, R. J., et al. (1997). Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives. Journal of Organizational Behavior, 18(6), 667–683.

    Article  Google Scholar 

  • Johnson, K. (2016). Realism and uncertainty of unobservable common causes in factor analysis. Noûs, 50(2), 329–355.

    Article  Google Scholar 

  • Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). Berlin: Springer.

    Google Scholar 

  • Jolliffe, I. T., & Morgan, B. J. T. (1992). Principal component analysis and exploratory factor analysis. Statistical Methods in Medical Research, 1(1), 69–95.

    Article  Google Scholar 

  • Kline, R. B. (2005). Structural equation modeling. New York: The Guilford Press.

    Google Scholar 

  • Kuhn, T. S. (1983). Rationality and theory choice. The Journal of Philosophy, 80(10), 563–570.

    Article  Google Scholar 

  • Kummerfeld, E., Ramsey, J., Yang, R., Spirtes, P., & Scheines, R. (2014). Causal clustering for 2-factor measurement models. In T. Calders, F. Esposito, E. Hüllermeier, & R. Meo (Eds.), Machine learning and knowledge discovery in databases, volume 8725 of lecture notes in computer science (pp. 34–49). Berlin: Springer.

  • MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504.

    Article  Google Scholar 

  • Maher, P. (1988). Prediction, accommodation and the logic of discovery. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1, 273–285.

    Google Scholar 

  • Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research. Perspectives on Psychological Science, 7(6), 537–542.

    Article  Google Scholar 

  • Maslach, C., & Jackson, S. E. (1981). The measurement of experienced burnout. Journal of Occupational Behavior, 2(2), 99–113.

    Article  Google Scholar 

  • Mellenbergh, G. J. (1994). Generalized linear item response theory. Psychological Bulletin, 115(2), 300–307.

    Article  Google Scholar 

  • Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88(3), 355–383.

    Article  Google Scholar 

  • Michell, J. (2003). The quantitative imperative positivism, naive realism and the place of qualitative methods in psychology. Theory & Psychology, 13(1), 5–31.

    Article  Google Scholar 

  • Michell, J. (2004). The place of qualitative research in psychology. Qualitative Research in Psychology, 1(4), 307–319.

    Article  Google Scholar 

  • Mulaik, S. A. (1991). Factor analysis, information-transforming instruments, and objectivity: A reply and discussion. The British Journal for the Philosophy of Science, 42(1), 87–100.

    Article  Google Scholar 

  • Murray-Watters, A., & Glymour, C. (2015). What is going on inside the arrows? Discovering the hidden springs in causal models. Philosophy of Science, 82(4), 556–586.

    Article  Google Scholar 

  • Musgrave, A. (1974). Logical versus historical theories of confirmation. British Journal for the Philosophy of Science, 25(1), 1–23.

    Article  Google Scholar 

  • Myrvold, W. C., & Harper, W. L. (2002). Model selection, simplicity, and scientific inference. Philosophy of Science, 69(S3), S135–S149.

    Article  Google Scholar 

  • Norton, J. D. (2015). Replicability of experiment. Theoria, 30(2), 229–248.

    Article  Google Scholar 

  • O’Connor, B. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavioral Research Methods, Intruments and Computers, 32(3), 396–402.

    Article  Google Scholar 

  • Park, H. S., Dailey, R., & Lemus, D. (2002). The use of exploratory factor analysis and principal component analysis in communication research. Human Communication Research, 28(4), 562–577.

    Article  Google Scholar 

  • Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48(1), 28–56.

    Article  Google Scholar 

  • Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338.

    Article  Google Scholar 

  • Schurz, G. (2014). Bayesian pseudo-confirmation, use-novelty, and genuine confirmation. Studies in History and Philosophy of Science, 45(1), 87–96.

    Article  Google Scholar 

  • Silva, R., Scheines, R., Glymour, C., & Spirtes, P. (2006). Learning the structure of linear latent variable models. The Journal of Machine Learning Research, 7, 191–246.

    Google Scholar 

  • Sober, E. (2004). Likelihood, model selection, and the Duhem-Quine problem. Journal of Philosophy, 101(5), 221–241.

    Article  Google Scholar 

  • Spirtes, P., Glymour, C., & Scheines, R. (1991). From probability to causality. Philosophical Studies, 64(1), 1–36.

    Article  Google Scholar 

  • Spirtes, P., Glymour, C., & Scheines, R. (2000). Causation, Prediction, and Search. Cambridge: MIT Press.

    Google Scholar 

  • Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680.

    Article  Google Scholar 

  • Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36(2), 111–147.

    Google Scholar 

  • Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society B, 39(1), 44–47.

    Google Scholar 

  • Suppes, P. (2007). Statistical concepts in philosophy of science. Synthese, 154(3), 485–496.

    Article  Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). London: Pearson.

    Google Scholar 

  • Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62(2), 157–176.

    Article  Google Scholar 

  • Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.

    Book  Google Scholar 

  • Thurstone, L. L. (1947). Multiple factor analysis. Chicago: University of Chicago Press.

    Google Scholar 

  • Thurstone, L. L. (1954). An analytical method for simple structure. Psychometrika, 19(3), 173–182.

    Article  Google Scholar 

  • Ullman, J. B. (2013). Structural equation modeling. In B. G. Tabachnick & L. S. Fidell (Eds.), Using multivariate statistics (6th ed.). London: Pearson.

    Google Scholar 

  • van Fraassen, B. C. (1980). The scientific image. New York: Oxford University Press.

    Book  Google Scholar 

  • Velicer, W. F., & Jackson, D. N. (1990a). Component analysis versus common factor analysis: Some further observations. Multivariate Behavioral Research, 25(1), 97–114.

    Article  Google Scholar 

  • Velicer, W. F., & Jackson, D. N. (1990b). Component analysis versus common factor analysis: Some issues in selecting an appropriate procedure. Multivariate Behavioral Research, 25(1), 1–28.

    Article  Google Scholar 

  • White, R. (2003). The epistemic advantage of prediction over accommodation. Mind, 112(448), 653–683.

    Article  Google Scholar 

  • Worrall, J. (2002). New evidence for old. In P. Gärdenfors, J. Wolenski, & K. Kijana-Placek (Eds.), In the scope of logic, methodology and philosophy of science (pp. 191–209). Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

Download references

Acknowledgements

I would like to thank Stephan Hartmann for valuable comments and suggestions made on a previous draft of this paper. I am also grateful to anonymous referees, whose comments and suggestions helped to improve this article, and to Sarah-Geneviève Trépanier, for enlightening discussions on the subject. This research was financially supported by the Social Sciences and Humanities Research Council of Canada.

Author information

Authors and Affiliations

  1. Munich Center for Mathematical Philosophy, Ludwig-Maximilians-Universität München, Munich, Germany

    Clayton Peterson

Authors
  1. Clayton Peterson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Clayton Peterson.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peterson, C. Accommodation, prediction and replication: model selection in scale construction. Synthese 196, 4329–4350 (2019). https://doi.org/10.1007/s11229-017-1660-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11229-017-1660-0

Keywords

Search

Navigation