Skip to main content

Advertisement

SpringerLink
Log in

Addressing confounding errors when using non-experimental, observational data to make causal claims

  • Published:
Synthese Aims and scope Submit manuscript

Abstract

In their recent book, Is Inequality Bad for Our Health?, Daniels, Kennedy, and Kawachi claim that to “act justly in health policy, we must have knowledge about the causal pathways through which socioeconomic (and other) inequalities work to produce differential health outcomes.” One of the central problems with this approach is its dependency on “knowledge about the causal pathways.” A widely held belief is that the randomized clinical trial (RCT) is, and ought to be the “gold standard” of evaluating the causal efficacy of interventions. However, often the only data available are non-experimental, observational data. For such data, the necessary randomization is missing. Because the randomization is missing, it seems to follow that it is not possible to make epistemically warranted claims about the causal pathways. Although we are not sanguine about the difficulty in using observational data to make warranted causal claims, we are not as pessimistic as those who believe that the only warranted causal claims are claims based on data from (idealized) RCTs. We argue that careful, thoughtful study design, informed by expert knowledge, that incorporates propensity score matching methods in conjunction with instrumental variable analyses, provides the possibility of warranted causal claims using observational data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Angrist J.D., Imbens G.W., Rubin D.B. (1996). Identification of effects using instrumental variables. Journal of the American Statistical Association 91(434): 444–455

    Article  Google Scholar 

  • Angrist J.D., Krueger A.B. (2001). Instrumental variables and the search for identification: From supply and demand to natural experiments. Journal of Economic Perspectives 15(4): 69–85

    Article  Google Scholar 

  • Austin P.C., Grootendorst P., Anderson G.M. (2007). A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine 26(4): 734–753

    Article  Google Scholar 

  • Bellman R. (1961). Adaptive control processes. Princeton University Press, Princeton, NJ

    Google Scholar 

  • Berk R.A. (2004). Regression analysis: A constructive critique. Sage Publications, Thousand Oaks, CA

    Google Scholar 

  • Bond S.J., White I.R., Walker A.S. (2007). Instrumental variables and interactions in the causal analysis of a complex clinical trial. Statistics in Medicine 26(7): 1473–1496

    Article  Google Scholar 

  • Clogg C.C., Haritou A. (1997). The regression method of causal inference and a dilemma confronting this method. In: McKim V., Turner S. (eds) Causality in crisis? Statistical methods and the search for causal knowledge in the social sciences. University of Notre Dame, Press, Notre Dame IN, pp 83–112

    Google Scholar 

  • D’Agostino R.B., Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine 17(19): 2265–2281

    Article  Google Scholar 

  • D’Agostino R.B., Jr., Rubin D.B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association 95(451): 749–759

    Article  Google Scholar 

  • Daniels D., Kennedy B., Kawachi I. (2000). Justice is good for our health. In: Cohen J., Rogers J. (eds) Is inequality bad for our health?. Beacon Press, Boston, MA, pp 3–33

    Google Scholar 

  • Davidson R., MacKinnon J.G. (1993). Estimation and inference in econometrics. Oxford University Press, New York, NY

    Google Scholar 

  • DiPrete T.A., Gangl M. (2004). Assessing bias in the estimation of causal effects: Rosenbaum bounds on matching estimators and instrumental variables estimation with imperfect instruments. Sociological Methodology 34: 271–310

    Article  Google Scholar 

  • Freedman D.A. (1999). From association to causation: Some remarks on the history of statistics. Statistical Science 14(3): 243–258

    Article  Google Scholar 

  • Freedman D.A. (2005). Statistical models: Theory and practice. Cambridge University Press, Cambridge

    Google Scholar 

  • Glymour M.M. (2006). Natural experiments and instrumental variable analyses in social epidemiology. In: Oakes J.M., Kaufman J. (eds) Methods in social epidemiology. Jossey-Bass, San Francisco, CA, pp 429–460

    Google Scholar 

  • Greenland S. (1990). Randomization, statistics, and causal inference. Epidemiology 1: 421–429

    Article  Google Scholar 

  • Greenland S. (2000). An introduction to instrumental variables for epidemiologists. International Journal of Epidemiology 29: 722–729

    Article  Google Scholar 

  • Greenland S., Robins J.M. (1986). Identifiability, exchangeability, and epidemiological confounding. International Journal of Epidemiology 15(3): 413–419

    Article  Google Scholar 

  • Haukoos J.S., Newgard C.D. (2007). Advanced statistics: Missing data in clinical research – Part 1: An introduction and conceptual framework. Academic Emergency Medicine 14(7): 662–668

    Google Scholar 

  • Heckman J.J. (1997). Instrumental variables: A study of implicit behavioral assumptions used in making program evaluations. The Journal of Human Resources 32(3): 441–462

    Article  Google Scholar 

  • Heckman, J. J. (2005). The scientific model of causality. In R. Stolzenberg (Ed.), Sociological methodology (Vol. 35, pp. 1–97). Oxford: Basil Blackwell (for the American Sociological Association).

  • Hernán M.A. (2004). A definition of causal effect for epidemiological research. Journal of Epidemiology and Community Health 58: 265–271

    Article  Google Scholar 

  • Hernán, M.A., Robins J.M. (2006). Instruments for causal inference: An epidemiologist’s dream? Epidemiology 17(4): 360–372

    Article  Google Scholar 

  • Hintikka J. (1975). The intentions of intentionality and other new models for modalities. D. Reidel Publishing Co., Dordrecht

    Google Scholar 

  • Holland P. (1986). Statistics and causal inference. Journal of the American Statistical Association 81(396): 945–960

    Article  Google Scholar 

  • Humphreys P. (1986). Causation in the social sciences: An overview. Synthese 68: 1–12

    Google Scholar 

  • Imbens G.W., Angrist J.D. (1994). Identification and estimation of local average treatment effects. Econometrica 62(2): 467–475

    Article  Google Scholar 

  • Imbens G.W., Rosenbaum P.R. (2005). Robust, accurate confidence intervals with a weak instrument: Quarter of birth and education. Journal of the Royal Statistical Society, Series A 168(part 1): 109–126

    Google Scholar 

  • Kaufman J.S., Kaufman S., Poole C. (2003). Causal inference from randomized trials in social epidemiology. Social Science and Medicine 57: 2397–2409

    Article  Google Scholar 

  • Linden A., Adams J.L. (2006). Evaluating disease management programme effectiveness: An introduction to instrumental variables. Journal of Evaluation in Clinical Practice 12(2): 148–154

    Article  Google Scholar 

  • Little R.J., Rubin D.B. (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and approaches. Annual Review of Public Health 21: 121–145

    Article  Google Scholar 

  • Luellen J.K., Shadish W.R., Clark M.H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review 29(6): 530–558

    Article  Google Scholar 

  • Maldonado G., Greenland S. (2002). Estimating causal effects. International Journal of Epidemiology 31: 422–429

    Article  Google Scholar 

  • Manski, C. F. (1993). Identification problems in the social sciences. In P. Marsden (Ed.), Social methodology (Vol. 23, pp. 1–56). Oxford: Basil Blackwell (for the American Sociological Association).

  • Manski C.F. (1995). Identification problems in the social sciences. Harvard University Press, Cambridge, MA

    Google Scholar 

  • Moffitt R. (2005). Remarks on the analysis of causal relationships in population research. Demography 42(1): 91–108

    Article  Google Scholar 

  • Newgard C.D., Haukoos J.S. (2007). Advanced statistics: Missing data in clinical research – Part 2: Multiple imputation. Academic Emergency Medicine 14(7): 669–678

    Google Scholar 

  • Newgard C.D., Hedges J.R., Arthur M., Mullins R.J. (2004). Advanced statistics: The propensity score – A method for estimating treatment effect in observational research. Academic Emergency Medicine 11(9): 953–961

    Google Scholar 

  • Newhouse J.P., McClellan M. (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health 19: 17–34

    Article  Google Scholar 

  • Newman S.C. (2004). Commonalities in the classical, collapsibility and counterfactual concepts of confounding. Journal of Clinical Epidemiology 57: 325–329

    Article  Google Scholar 

  • Oakes M.J. (2004). The (Mis)estimation of neighborhood effects: Causal inference for a practicable social epidemiology. Social Science and Medicine 58(10): 1929–1952

    Article  Google Scholar 

  • Oakes M.J., Johnson P.J. (2006). Propensity score matching for social epidemiology. In: Oakes J.M., Kaufman J. (eds) Methods in social epidemiology. Jossey-Bass, San Francisco, CA, pp 370–392

    Google Scholar 

  • Pearl J. (2000). Causality: Models, reasoning, and inference. Cambridge University Press, Cambridge

    Google Scholar 

  • Pearl J. (2001). Causal inference in health sciences: A conceptual introduction. Health Services and Outcomes Research Methodology 2: 189–220

    Article  Google Scholar 

  • Randall Jr. J.H. (1940). The making of the modern mind – Revised edition. Houghton Mifflin Company, Boston, MA

    Google Scholar 

  • Reiter J. (2000). Using statistics to determine causal relationships. The American Mathematical Monthly 107(1): 24–32

    Article  Google Scholar 

  • Robins J.M., Scheines R., Spirtes P., Wasserman L. (2003). Uniform consistency in causal inference. Biometrika 90(3): 491–515

    Article  Google Scholar 

  • Rosenbaum P.R. (2002). Observational studies (2nd ed). Springer, New York, NY

    Google Scholar 

  • Rosenbaum P.R. (2004). Matching in observational studies. In: Gelman A., Meng X.-L. (eds) Applied Bayesian modeling and causal inference from incomplete-data perspectives. Wiley and Sons, Ltd, West Sussex, pp 15–24

    Chapter  Google Scholar 

  • Rosenbaum P.R., Rubin D.B. (1983). The central role of propensity scores in observational studies for causal effects. Biometrika 70(1): 41–55

    Article  Google Scholar 

  • Rosenbaum P.R., Rubin D.B. (1984). On the nature and discovery of structure: Comment. Journal of the American Statistical Association 79(385): 26–28

    Article  Google Scholar 

  • Rosenbaum P.R., Rubin D.B. (1985a). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician 39(1): 33–38

    Article  Google Scholar 

  • Rosenbaum P.R., Rubin D.B. (1985b). The bias due to incomplete matching. Biometrics 41(1): 103–116

    Article  Google Scholar 

  • Rothman K. J. (2002). Epidemiology: An introduction. Oxford University Press, Oxford

    Google Scholar 

  • Rubin D.B. (1986). Statistics and causal inference: Comment: Which ifs have causal answers. Journal of the American Statistical Association 81(396): 961–962

    Article  Google Scholar 

  • Rubin D.B. (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine 127(8): 757–763

    Google Scholar 

  • Rubin D.B. (2004). On principles for modeling propensity scores in medical research. Pharmacoepidemiology and Drug Safety 13(12): 855–857

    Article  Google Scholar 

  • Rubin D.B. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 26(1): 20–36

    Article  Google Scholar 

  • Rubin D.B., Thomas N. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics 52(1): 249–264

    Article  Google Scholar 

  • Shadish W.R., Cook T.D., Campbell D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company, Boston, MA

    Google Scholar 

  • Smith H.L. (1997). Matching with multiple controls to estimate treatment effects in observational studies. Sociological Methodology 27: 325–353

    Article  Google Scholar 

  • Smith H.L. (2003). Some thoughts on causation as it relates to demography and population studies. Population and Development Review 29(3): 459–469

    Article  Google Scholar 

  • Smith J.A., Todd P.E. (2001). Reconciling conflicting evidence on the performance of propensity-score matching methods. American Economics Review 91(2): 112–118

    Article  Google Scholar 

  • Sobel, M. E. (2005). Discussion: The scientific model of causality. In R. Stolzenberg (Ed.), Social methodology (Vol. 35, pp. 99–133). Oxford: Basil Blackwell (for the American Sociological Association).

  • Urbach P. (1985). Randomization and the design of experiments. Philosophy of Science 52(2): 256–273

    Article  Google Scholar 

  • Weitzen S., Lapane K.L., Toledano A.Y., Hume A.L., Mor V. (2005). Weaknesses of goodness-of-fit tests for evaluating propensity score models: The case of the omitted confounder. Pharmacoepidemiology and Drug Safety 14(4): 227–238

    Article  Google Scholar 

  • Winship C., Morgan S.L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology 25: 659–706

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Minnesota Population Center, University of Minnesota, Room 50 Willey Hall, 225, 19th Avenue South, Minneapolis, MN, 55455, USA

    Andrew Ward

  2. Health Policy and Management, 2221 University Avenue S.E., Suite 345, 55414, Minneapolis, MN, USA

    Pamela Jo Johnson

Authors
  1. Andrew Ward
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pamela Jo Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew Ward.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ward, A., Johnson, P.J. Addressing confounding errors when using non-experimental, observational data to make causal claims. Synthese 163, 419–432 (2008). https://doi.org/10.1007/s11229-007-9292-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11229-007-9292-4

Keywords

Search

Navigation