There have recently been novel applications of medical systematic review guidelines to economic policy interventions which contain controversial methodological assumptions that require further scrutiny. A landmark 2017 Cochrane review of unconditional cash transfer (UCT) studies, based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE), exemplifies both the possibilities and limitations of applying medical systematic review guidelines to UCT and universal basic income (UBI) studies. Recognizing the need to upgrade GRADE to incorporate the differences between medical and policy interventions, (...) the GRADE Public Health Project Group (PHPG) was convened to enumerate and address these methodological challenges. However, in light of our analysis of additional methodological challenges that arise for UCT and UBI studies, we argue that the adaptation of medical systematic review guidelines to economic methodology is far from straightforward and is in fact more challenging than claimed by the PHPG. (shrink)

Explainable artificial intelligence (XAI) is an emerging, multidisciplinary field of research that seeks to develop methods and tools for making AI systems more explainable or interpretable. XAI researchers increasingly recognise explainability as a context-, audience- and purpose-sensitive phenomenon, rather than a single well-defined property that can be directly measured and optimised. However, since there is currently no overarching definition of explainability, this poses a risk of miscommunication between the many different researchers within this multidisciplinary space. This is the problem we (...) seek to address in this paper. We outline a framework, called Explanatory Pragmatism, which we argue has two attractive features. First, it allows us to conceptualise explainability in explicitly context-, audience- and purpose-relative terms, while retaining a unified underlying definition of explainability. Second, it makes visible any normative disagreements that may underpin conflicting claims about explainability regarding the purposes for which explanations are sought. Third, it allows us to distinguish several dimensions of AI explainability. We illustrate this framework by applying it to a case study involving a machine learning model for predicting whether patients suffering disorders of consciousness were likely to recover consciousness. (shrink)

While the history and methodology of laboratory experiments in economics have been extensively studied by philosophers, those of field experiments have not attracted much attention until recently. What is the historical context in which field experiments have been advocated? And what are the methodological rationales for conducting experiments in the field as opposed to in the lab? This article addresses these questions by combining historical and methodological perspectives. In terms of history, we show that the movement toward field experiments in (...) economics has two distinct roots. One is the general orientation of medical and social sciences to evidence-based policy evaluation, which gave rise to randomized field experiments in economics. The other is an awareness of several methodological limitations of lab experiments in economics, which required practitioners to get out of the lab and into the field. In these senses, the movement is a consequence of influences from both outside and inside economics: the general evidence-based trend in policy science and an internal methodological development of experimental economics. In terms of methodology, we show that these two roots resulted in two somewhat different notions of “external validity” as methodological rationales of field experiment. Finally, we suggest that analysis of experiments as exhibits highlights a methodological strategy in which both strands complement each other. (shrink)

I envision new directions in the methodology of experimental games in the field of developmental, environmental and resource economics. Although there have been extensive discussions on experimenta...

Extrapolating causal effects is becoming an increasingly important kind of inference in Evidence-Based Policy, development economics, and microeconometrics more generally. While several strategies have been proposed to aid with extrapolation, the existing methodological literature has left our understanding of what extrapolation consists of and what constitutes successful extrapolation underdeveloped. This paper addresses this lack in understanding by offering a novel account of successful extrapolation. Building on existing contributions pertaining to the challenges involved in extrapolation, this more nuanced and comprehensive account (...) seeks to provide tools that facilitate the scrutiny of specific extrapolative inferences and general strategies for extrapolation. Offering such resources is important especially in view of the increasing amounts of real-world decision-making in policy, development, and beyond that involve extrapolation. (shrink)

This paper argues that an important type of experiment-target inference, extrapolating causal effects, requires models to be successful. Focusing on extrapolation in Evidence-Based Policy, it is ar...

Extrapolating causal effects from experiments to novel populations is a common practice in evidence-based-policy, development economics and other social science areas. Drawing on experimental evidence of policy effectiveness, analysts aim to predict the effects of policies in new populations, which might differ importantly from experimental populations. Existing approaches made progress in articulating the sorts of similarities one needs to assume to enable such inferences. It is also recognized, however, that many of these assumptions will remain surrounded by significant uncertainty in (...) practice. Unfortunately, the existing literature says little on how analysts may articulate and manage these uncertainties. This paper aims to make progress on these issues. First, it considers several existing ideas that bear on issues of uncertainty, elaborates the challenges they face, and extracts some useful rationales. Second, it outlines a novel approach, called the support graph approach, that builds on these rationales and allows analysts to articulate and manage uncertainty in extrapolation in a systematic and unified way. (shrink)

Is evidential pluralism possible when we move to the social sciences, and if so, to what degree? What are the analytical benefits? The answer put forward in this article is that there is a tradeoff between how serious social science methodologies take the study of mechanisms and the analytical benefits that flow from evidential pluralism. In the social sciences, there are a range of different approaches to studying mechanisms, differentiated by the degree to which the ‘process’ is unpacked theoretically, and (...) whether the approach takes seriously the particular nature of social phenomena and the epistemological consequences that flow from this, as in realist approaches to the study of mechanisms, or whether more neopositivist-based foundational assumptions are adopted. Depending on which approach to study mechanisms is used, evidential pluralism is either: easy but superficial, very productive but challenging, or almost impossible because of the fundamental differences between the types of claims being made and the forms of evidence used. (shrink)

Public policymakers and institutional decision-makers routinely face questions about whether interventions “work”: does universal basic income improve people’s welfare and stimulate entrepreneurial activity? Do gated alleyways reduce burglaries or merely shift the crime burden to neighbouring communities? What is the most cost-effective way to improve students’ reading abilities? These are empirical questions that seem best answered by looking at the world, rather than trusting speculations about what will be effective. Evidence-based policy (EBP) is a movement that concretizes this intuition. It (...) maintains that policy should be based on evidence of “what works.” Not any evidence will do, however. Following on the heels of its intellectual progenitor, evidence-based medicine (EBM), EBP insists on the use of high-quality evidence produced in accordance with rigorous methodological standards. Though intuitively compelling, EBP has attracted significant criticism from methodologists, political scientists, and philosophers. This chapter provides a critical overview of EBP through a philosophical lens, reviewing and discussing some of the most pressing challenges that EBP faces and outlining some proposals for improving it. I first provide a brief overview of EBP and distinguish a broader and narrower understanding of it. I then review existing lines of criticism of EBP, two of which are considered in detail. The first elaborates how EBP struggles with extrapolation, that is, the use of evidence from study populations to make inferences about the effects of policies on novel target populations. The second maintains that EBP’s methodological tenets are deeply entwined with moral and political values, which can threaten EBP’s promise to promote objectivity in policymaking. Finally, I make some proposals for how EBP could be improved going forward. (shrink)