Back to the big picture Anna Alexandrova, Robert Northcott, and Jack Wright1 Forthcoming in the Journal of Economic Methodology Abstract: We distinguish between two different strategies in economic methodology. The big picture strategy, dominant in the twentieth century, ascribed to economics a unified method and evaluated this method against a single criterion of 'science'. In the last thirty years a second strategy gained prominence: fine-grained studies of how some specific technique common in economics can achieve one or more epistemic goal. We argue that recent developments in philosophy of science and in economics warrant a return to big-picture – but now reinvented. It should focus on a new question, already intensely debated within the profession: is the organization of economics healthy and appropriate? 1. Introduction The history of methodology of economics has seen two different strategies. At times, methodologists have analyzed economics as a whole, ascribing to it a single epistemic approach and appealing to a standard against which this approach can be evaluated. At other times, they have pursued a more circumscribed enquiry, into how some specific technique common in economics can achieve one or more epistemic goal. Label the first strategy bigpicture, and the second strategy fine-grained. Roughly speaking, the big-picture strategy prevailed up to the 1990s, but in the last thirty years the dominant mode has been finegrained. We argue that recent developments in philosophy of science and in economics warrant a return to big-picture – but now reinvented. It should not inherit the old presumption that economics has a single method, or that there is a single criterion of 'science'. Instead, it should focus on a new question, already intensely debated within the profession: is the organization of economics healthy and appropriate? This question is big-picture. Although any answer to it must ride on the back of fine-grained work, fine-grained work alone is not enough. It is also ripe for explicit and systematic examination by methodologists because a proper answer requires the skills and knowledge of our community. We illustrate how a revived big-picture strategy is fruitful for two controversies: how much effort to devote to rational choice modeling, and how economics is socially organized. 2. Big-picture versus fine-grained The big-picture approach has a long, proud history. It stretches back to John Stuart Mill, who defended political economy as a science that studies phenomena that arise from the pursuit of wealth. In the twentieth century, big-picture theorists borrowed from influential general accounts of scientific method and scientific change, especially those of Karl Popper, Thomas Kuhn, and Imre Lakatos. The concepts of demarcation, paradigm, and progressive research program got deployed either to vindicate economics as a legitimate science or to criticize it for not being one. The heyday of big-picture methodology of economics was the 1970s to 1 The authors are equally and jointly responsible. They are listed in alphabetical order. 2 1990s, a period in which the best-known participants in our field, such as Mark Blaug, Bruce Caldwell, Jon Elster, Uskali Mäki, and Deidre McCloskey, all took a stance on the overall 'status of economics'. (Heterodox economists, such as Tony Lawson, Ben Fine, and Frederic Lee, have continued to pronounce on the status of the discipline as a whole, but they have had a different audience.) This heyday culminated in 1992, with Dan Hausman's Inexact and Separate Science of Economics and Alex Rosenberg's Mathematical Politics. Both of these exemplified the big-picture approach even as they reached vastly different verdicts: there is a distinctive strategy to economics, and this strategy can be judged against a compelling criterion. More recently there has been a turn towards the fine-grained, at least among scholars who publish in journals such as this one. The big-picture accounts do still appear on teaching syllabi and retain historical interest. But it is now rare to hear scholarly verdicts on whether economics as a whole is a science, or to encounter attempts to ascribe to it a single overall strategy. When such attempts do appear, such as Dani Rodrik's 2015 book Economics Rules, they are usually aimed at a general audience, with the goal of correcting common misconceptions. Professional scholarship in methodology of economics instead prizes more specific contributions, such as the precise inferences that may or may not be warranted by models, by randomized controlled trials, by economic experiments, or by measurement of indicators. Mireles-Flores's recent review paper (2018) picks out modeling and explanation, causal inference and evidence, and behavioral economics as the three most active research topics. Today's doctoral dissertations and journal articles display intricate knowledge of economic practice, and this level of detail in turn demands a focus on specific models or theories rather than on economics as a whole. Why has our field gone fine-grained? General philosophy of science has moved on from the classic mid-century focus on the demarcation problem, and has largely given up on the idea of a single scientific method or a unified criterion of scientific success. Historians, sociologists, and philosophers have delivered a consistent message that science is hospitable to distinctive epistemic cultures and that evaluating these cultures requires sensitivity to local circumstances and constraints. One field may rely on simulations, another on fieldwork; one looks for general theories, another for toy models; one prizes unification, another rich ethnography. None is superior to all others in all dimensions. To capture this complexity and richness, philosophers of physical and life sciences have often relied on case studies. The lessons from these are illuminating and valuable without being universal or general. This spirit of case-driven philosophy of science has percolated through. Methodologists have recognized that economics is more diverse than the original big-picture accounts made it look. Rational choice modeling typically hogged attention and determined the agenda. But powerful and influential though it is, rational choice modeling hardly defines all of economics, and paying close attention to econometrics, welfare analysis, or experimental work makes it harder to ascribe a single epistemic strategy to the discipline as a whole. There are many strategies and many criteria of success. A good economic experiment may be prized for virtues that are entirely distinct from the virtues of a good rational choice model, and this variety makes it hard to pass a single 'yea or nay' verdict on economics as a whole. No wonder today's methodologists delve into the details of specific economic projects such as mechanism design, causal inference using instrumental variables, or the measurement of consumption. 3 There were good reasons to go fine-grained. But the fine-grained status quo also has downsides. Methodologists focus on specific techniques and, as they debate with each other what these techniques can and cannot achieve, their stances grow nuanced and intricate. This emphasis on nuance takes attention away from trends in economics as a whole. Yet discipline-wide trends arguably impact on the fortunes of economics more strongly than do further nuances of fine-grained issues. These wider trends demand our attention. Armed with the rich understanding of detail that the fine-grained era has delivered, methodologists are now well-positioned to make good on that obligation. We offer two examples. 3. The efficiency question Recent methodology of economics has been preoccupied perhaps more than anything else by the status of rational choice models. These models dominate mainstream work. What are they, how do they represent, and perhaps above all: do they explain? If they do explain, in what sense? Fine-grained work has greatly elucidated these debates (Marchionni 2017). But regardless of whether economic models do indeed explain, all sides agree that they are sometimes useful and sometimes not, whether that usefulness consists in explanations or in something else. Practically speaking, what matters is a different debate: should economists do more rational choice modeling or should they invest their efforts elsewhere? We cannot answer this question without specifying alternatives: qualitative methods such as interviews and ethnographic observation; questionnaires; small-N causal inference, such as qualitative comparative analysis; causal process tracing; causal inference from observational statistics; machine learning from big data; historical case studies; randomized controlled trials; laboratory experiments; and natural and quasi-experiments. All of these alternative methods are widely practiced in other sciences. All of them can generate results that, in a virtuous circle, feed back into more theory development. The latter few of these methods have begun to be co-opted by mainstream economics already. But which mix of methods is optimal? Label this the efficiency question (Northcott 2018). It is a big-picture issue. To answer the efficiency question requires, so to speak, an epistemic cost-benefit analysis. The costs are the resources invested into modeling, such as mathematical training of students, and perhaps more notably the opportunity costs, such as fieldwork methods not taught and fieldwork not done. The benefits are whatever successful explanations, predictions and interventions such modeling leads to. What is the optimal balance between, on one hand, building up a library of orthodox models, and on the other hand, pursuing more applied, contextual work and utilizing a wider range of methods? Of course, this cost-benefit analysis can only be done imperfectly and approximately. It is hard to count up explanations and predictions in an objective way, hard to weigh these versus other goals of science, and hard also to evaluate the counterfactual of whether a different allocation of resources would be better. But, implicitly, such analyses are unavoidable and are being done already – every time a researcher chooses, or a graduate school teaches, one method rather than another, or journals or prizes or hirers choose one paper or candidate rather than another. (This is true even though other factors enter such decisions too.) The status quo is not inevitable, as shown by different practices in other social sciences. It is surely better for methodologists to tackle this question explicitly rather than to leave it to inertia and sociological winds. 4 Recently, the discipline itself has, in effect, been addressing the efficiency question, reflecting that question's importance. We have in mind the much-remarked 'empirical turn'. In the five most prestigious journals in economics, the percentage of papers that are purely theoretical – i.e. free of any empirical data – fell from 57% in 1983 to 19% in 2011 (Hamermesh 2013). Not only is there more empirical work in prestigious venues but also this empirical work is more often 'a-theoretical' rather than theory-based, i.e. it more often establishes previously untheorized causal relations rather than tests already-existing theoretical models. Biddle and Hamermesh (2016) report that whereas in the 1970s all microeconomic empirical papers in top-5 journals exhibited a theoretical framework, in the 2000s a-theoretical studies resurged. Citation numbers suggest that the a-theoretical work is at least as influential. Angrist and Pischke (2010) also report the rise of a-theoretical practice in several subfields. Other evidence besides, such as the work of recent Clark medal winners, shows the same trend. The empirical turn is an implicit answer to the efficiency question. It reflects a disciplinewide shift in research emphasis away from pure theory towards empirical application. It shows that the discipline's norms and incentives allow it to make such a shift. It now becomes incumbent on methodologists to evaluate this shift. So far, remarkably little has been said by methodologists about the empirical turn, even though it is the biggest recent trend in the discipline. Historians have described and contextualized this transformation, but without taking an evaluative stance (Backhouse and Cherrier 2017). Is the empirical turn a good thing? This is just another way of formulating the efficiency question. On either formulation, it is a big-picture not fine-grained issue. 4. Social organization of economics Some of the most voracious and contested topics at economics conferences and in economics media (podcasts, forums, Twitter) concern how economists relate to one another. Do female economists face an adverse environment? Do economists from certain universities receive undue attention? These topics have received scholarly attention: historians and social scientists, from anthropologists to economists themselves, have presented rich data on who gets attention, who governs, which kinds of people succeed, and how economists think of outsiders. But methodologists have said little. This is surprising because much recent work in philosophy of science, political philosophy, and social epistemology focuses on how social norms and participation impacts on knowledge-making practices (Kitcher 2001, Longino 2002, Anderson 2020). Inquiring into how economics is organized raises questions that deserve methodological attention. Does the distribution of power in economics aid progress? Does a lack of female economists lead to missing topics or less robust research? Label these social organizational questions. They are big-picture. Consider the distribution of prestige and influence. Publications in the top 5 journals play a significant role in determining career success (Heckman and Moktan 2020). Economists are prone to elevating star individuals disproportionately, as shown by the attention, authority, and citations afforded to those that win the big prizes. The American Economic Association is dominated by economists from a few top US departments, much more so than the governing bodies of other social sciences (Fourcade et al. 2015). Those employed or trained at the same few top departments also dominate among the authors and editors of the top journals (Colussi 2018). Hiring prestige, whereby departments hire only from similar or higher-ranked departments, exists in many academic fields, but it too is stronger in economics (Han 2003). These factors and more highlight large inequities of power and opportunity and a steep hierarchy within the discipline. 5 Why should methodologists and economists care? Because even if individual economists diligently follow appropriate methods – the subject matter of fine-grained methodology – hierarchy effects might still hinder economics as a whole from self-correcting and developing new knowledge. The dominance of the top 5 journals encourages instrumental reasoning at the beginning of research projects and creates incentives against innovative methods and topics (Heckman et al. 2017). And the dominance of a small subset of economists amplifies the common bias that academics have in favor of work like their own (Akerlof and Michaillat 2018). Both of these factors put a damper on new methods and topics for inquiry and on their ability to gain attention. As well as impeding new knowledge, these factors also undercut existing knowledge: with fewer new ideas, existing work is less likely to be tested and stretched by new forms of reasoning or new observations. This is compounded by the way a steep hierarchy makes it costlier to criticize those higher up in the discipline. As Mill noted long ago, pressure from criticism and new ideas is a crucial mechanism for identifying errors. We can reasonably assume that beliefs that survive such pressure are less likely to be false and so are more justified. Putting these points together, no matter how well individual methods are followed, the steep hierarchy in economics – a social factor – has a significant negative effect. Similar points can be made about the climate for women and underrepresented minorities; that economics is more insular than other social sciences; that it is largely anglophone; and that US departments and institutions dominate. Many of these social phenomena attract much attention within economics itself, and in the coverage of the profession in the media. But methodologists have been mute about them. Given that these phenomena impact on squarely methodological topics – objectivity, progress, bias, trust – this muteness is surprising, and doubly so given the wealth of readily available data. Do methodologists in fact consider social organization questions already, just under a different name – pluralism (Davis 2014, Salanti and Screpanti 1997)? The two issues are related but distinct. Pluralism questions typically concern whether economists do the right distribution of things. This is a general version of the efficiency question from earlier. Social organization questions are more agnostic about the specific things that economists do, such as whether a particular method or school of thought deserves more attention. They focus instead on what social inputs are likely to lead to a healthy balance, from the perspective of what we know about intellectual inquiry in general. The empirical base of social organization arguments is also easier to secure. It is a difficult thing to determine exactly how much pluralism is desirable and whether in economics we have the right amount of it, but it is much easier to find data showing that economics is hierarchical. Like the efficiency question, social organizational questions are difficult to answer, involving as they do complicated counterfactuals such as whether economics would produce better explanations if it were less hierarchical. But, as is also the case with the efficiency question, social organizational questions are routinely answered already – implicitly. If you ignore gender in a hiring process, implicitly you are assuming that gender representation is not important. Or if you highlight that authors in the main journals are concentrated at a few universities, implicitly you are asserting that this pattern is concerning. Social organizational questions offer avenues for exciting new fine-grained work, such as whether bigger authorship teams are more innovative or reliable, or whether internet-era networks of communication marginalize certain voices. But the social organizational 6 questions that we think especially ripe for methodologists' scrutiny bear on economics as a whole. They become visible only when taking a wider view. Hierarchy, insularity, and maleness are all good examples. Methodologists should take their cue from the recent philosophy that highlights how knowledge production is affected by different social practices. Is economics as currently organized likely to generate a diverse enough range of hypotheses, is it likely to spot problems with evidence, are certain criticisms likely to be heard or ignored? Careful analysis, grounded in the lessons that philosophers have learnt from other scientific endeavors, is surely preferable to dueling tribes on Twitter. 5. Conclusion: big picture reinvented Are we urging that future work in our field be about big-picture issues such as the efficiency question and the social organization of economics? Yes and no. Yes, because why shouldn't methodology of economics be focused on those aspects of the discipline that at the moment most preoccupy the profession itself? No, because, as ever, some philosophically important aspects of economics may not be those that practicing economists care to attend to. Besides, responsible big-picture work must ride on the back of solid fine-grained foundations, so there is a need for both. But we have highlighted a sense in which the balance has shifted. Whereas the old school big-picture strategy was based on a relatively narrow picture of science and was rightly abandoned in favor of fine-grained work, now a focus on fine-grained work carries its own danger. It risks blinding methodologists to seminal changes in economics, such as the empirical turn and the rising awareness of disciplinary politics. Each of these changes sorely needs to be evaluated, and that requires the return of big-picture work, duly reinvented. Just as every age needs its own art, so every age needs its own methodology of economics. After decades of investment in the fine-grained, our age is ready to step back and look again at the health of economics as a whole. 7 References Akerlof, G. A., & Michaillat, P. (2018). Persistence of false paradigms in low-power sciences. Proceedings of the National Academy of Sciences Proc Natl Acad Sci USA, 115(52), 13228. Anderson, E. (2020). Feminist Epistemology and Philosophy of Science. In E. N. Zalta (Ed.), Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University. Angrist, Joshua, and Jörn-Steffen Pischke 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics." Journal of Economic Perspectives 24:3-30. Backhouse, R. E., & Cherrier, B. (2017). The age of the applied economist: the transformation of economics since the 1970s. History of Political Economy, 49, 1-33. Biddle, Jeff, and Daniel Hamermesh 2016. "Theory and Measurement: Emergence, Consolidation and Erosion of a Consensus." NBER Working Paper 22253. Colussi, T. (2018). Social Ties in Academia: A Friend Is a Treasure. The Review of Economics and Statistics, 100(1), 45-50. Davis, J. B. (2014). 'Pluralism' In Economics? A Symposium. Review of Political Economy, 26(4), 477-478. Fourcade, M., Ollion, E., & Algan, Y. (2015). The superiority of economists. The Journal of Economic Perspectives, 29(1), 89-113. Hamermesh, Daniel 2013. "Six Decades of Top Economics Publishing: Who and How?" Journal of Economic Literature 51:162-172. Han, S.-K. (2003). Tribal regimes in academia: a comparative analysis of market structure across disciplines. Social Networks, 25(3), 251-280. Heckman, J., Akerlof, G., Deaton, A., Fudenberg, D., & Hansen, L. P. (2017). Publishing and Promotion in Economics: The Curse of the Top Five. Webcast from ASSA 2017 Annual Meeting. Heckman, J. J., & Moktan, S. (2020). Publishing and Promotion in Economics: The Tyranny of the Top Five. Journal of Economic Literature, 58(2), 419-470. Kitcher, P. (2001). Science, truth, and democracy. New York: Oxford University Press. Longino, H. E. (2002). The fate of knowledge. Princeton University Press. Marchionni, C. (2017). What is the Problem with Model-based Explanation in Economics? Disputatio 9(47), 603-630. Mireles-Flores, L. (2018). Recent Trends in Economic Methodology: A Literature Review. In Research in the History of Economic Thought and Methodology: Volume 36A, 93-126, Emerald Publishing Limited. Northcott, Robert 2018. 'The efficiency question in economics', Philosophy of Science 85.5, 1140-1151. Salanti, A. and Screpanti, E. (eds) (1997) Pluralism in Economics, Cheltenham: Edward Elgar.