Divergent Perspectives on Expert Disagreement: Preliminary Evidence from Climate Science, Climate Policy, Astrophysics, and Public Opinion James R. Beebe (University at Buffalo), Maria Baghramian (University College Dublin), Luke Drury (Dublin Institute for Advanced Study), Finnur Dellsén (Inland Norway University of Applied Sciences) We report the results of an exploratory study that examines the judgments of climate scientists, climate policy experts, astrophysicists, and non-experts (N = 3,367) about the factors that contribute to the creation and persistence of disagreement within climate science and astrophysics and about how one should respond to disagreements among experts. We found that, as compared to educated non-experts, climate experts believe that within climate science (i) there is less disagreement about climate change, (ii) methodological factors play less of a role in generating existing disagreements, (iii) fewer personal or institutional biases influence the nature and direction of climate research, (iv) there is more agreement among scientists about which methods or theoretical perspectives should be used to examine the relevant phenomena, (v) disagreements about climate change should not lead people to conclude that the scientific methods being employed are unreliable, and (vi) climate science is more settled than ideological pundits would have us believe and settled enough to base public policy on it. In addition, we observed that the uniquely American political context predicted experts' judgments about some of these factors. We also found that, in regard to disagreements concerning cosmic ray physics, and commensurate with the greater inherent uncertainty and data lacunae in their field, astrophysicists working on cosmic rays were generally more willing to acknowledge expert disagreement, more open to the idea that a set of data can have multiple valid interpretations, and generally less quick to dismiss someone articulating a non-standard view as non-expert, than climate scientists were in regard to climate science. Keywords: expert disagreement, expertise, climate science, climate change, global warming, astrophysics, public policy 1. Introduction In today's complex societies, many policy decisions depend crucially upon expert advice and opinion. But experts can and do disagree, sometimes vehemently, and not all their disagreements are easily adjudicated. An immediate question facing policy makers, and in particular those involved in decisions concerning some of the greatest challenges facing 2 humanity, such as environmental policy, is how to react to persistent disagreement among experts. A further issue concerns how to respond to the sorry corollary of scientific disagreements, viz., the frequent misrepresentation and misunderstanding of them in the media and civic society. The current paper is the output of an interdisciplinary investigation, involving scientists and philosophers, of the ill understood, but socially and politically significant phenomenon of expert peer disagreement. The ultimate goal of the project is to gain a better understanding of the role and consequences of disagreement among scientific experts and its impact on policy decisions by governmental agencies and the formation of public opinion. We examined cases of expert scientific disagreement in a field that is relatively free of economic and political pressures (cosmic ray physics) and contrasted it with expert disagreements in a field where significant economic and political interests are at stake (climate science).1 Existing studies of the opinions of climate scientists (Oreskes 2004, Bray & von Storch 2008, Anderegg et al. 2010, Cook et al. 2013, Stenhouse et al. 2014, Cook et al. 2016) and the general public (Nisbet & Myers 2007, Leiserowitz et al. 2010, Leiserowitz et al. 2012, Pew 2012, Leiserowitz et al. 2015, Pew 2016, Gallup 2017) have shown significant disparities in their assessments of the extent of disagreement or consensus within climate science. These studies have not, however, examined the views of climate experts and non-experts in regard to the various personal, methodological, or institutional factors that these individuals think generate expert disagreement or how they think such disagreements should be approached. These studies also do not compare disagreements 1 Cf. the online supplementary materials document that accompanies this article for a summary of the key disagreements among experts in cosmic ray physics. 3 about climate science to disagreements in other scientific domains. Thus, the central questions we investigated included the following: (D1) In the politically charged discipline of climate science, are there significant differences between the opinions of climate scientists, climate policy experts, and non-experts regarding the extent and causes of disagreement within that discipline? (D2) Are there significant differences between the opinions of climate scientists, climate policy experts, and non-experts about the extent to which disagreement is due to what we call "epistemic" factors-i.e., familiar forms of scientific uncertainty and legitimate differences in expert's views of the quality of the relevant data and methods-and "nonepistemic" factors-i.e., the influence of scientists' own ideologies, personal values, or non-scientific beliefs (such as their political views), which may serve to bias their scientific work in various ways.2 (D3) What do climate scientists, climate policy experts, and astrophysicists take to be the most appropriate reaction to scientific disagreements within their fields? In particular, (a) should persistent disagreement regarding a given theory decrease our confidence in that theory? And (b) does persistent disagreement lead lay people to have doubts about the possibility of objectivity within that domain? 2 What we refer to as 'non-epistemic' factors here corresponds roughly to what Longino (1990: 46) calls 'contextual values', i.e. the personal, social, and cultural values that belong to the broader context in which science is done. However, for those familiar with Longino's terminology, it is worth noting that our notion of an 'epistemic' factor is broader than Longino's notion of a 'constitutive value'. 4 (D4) What effects does scientific disagreement have on the field in question? In particular, are disagreements perceived as good for the health of the discipline itself and what effects does publicizing disagreement have on public trust in science? We examined these issues using two questionnaires. We observed significant betweengroup differences in perceptions of the extent and causes of disagreement in climate science and significant differences between responses to the two scientific disciplines we studied. Our research questions and interpretations are informed by recent discussions within philosophy regarding disagreement between equally competent and otherwise equally well-informed agents-so-called 'peer disagreements' (Christensen 2007, 2009; Feldman 2006; Goldman 2010; Kelly 2005, 2010; Matheson 2015). A central issue at stake in cases of peer disagreement is the normative question of how someone should respond when she realizes that she is in such a situation, i.e., when she discovers that a peer disagrees with her.3 Each of the participant groups that were asked about climate science were disinclined to move closer to the opinion of peers who disagree with them about climate science. Astrophysicists, by contrast, were significantly inclined to adopt a conciliatory stance toward peers who disagreed with them about cosmic ray physics. Relatedly, we found that climate scientists, compared with astrophysicists, were more inclined to doubt that experts in their field who disagree with the dominant theoretical perspectives were as well informed as they are. This should perhaps not be surprising given the prevalence of ideological pundits masquerading as climate science experts in the media and elsewhere. The inherent uncertainty and data lacunae in the field of cosmic rays may 3 Cf. the supplementary materials document for additional details about the philosophical debate. 5 also explain the willingness of astrophysicists working on cosmic rays to show greater tolerance towards non-standard views. 2. Research Materials In order to investigate the research questions described above, the [name redacted] project hosted workshops with astrophysicists, climate scientists, climate policy experts, and philosophers working on the epistemology of disagreement. At these workshops, we discussed our central research questions with each group and solicited their input on how we might investigate them using questionnaires. On the basis of these discussions, we constructed two questionnaires, the first of which (Questionnaire 1) focused on descriptive questions about factors that lead to persistent expert disagreement within climate science and cosmic ray physics. The second (Questionnaire 2) focused on normative questions such as the reasonability of expert disagreement within these fields and whether encounters with peers with whom one disagrees should make one less confident in one's opinions. Questionnaire 2 also investigated the implications of persistent disagreement for public policy decisions, public trust in science, and assessments of the reliability of the scientific field in question. Our goal was to construct questionnaires for each participant group that were as similar as possible-except for the domain of science about which we inquired-so that comparisons across groups would be possible. However, some differences in wording were necessary. These differences are noted with underlining, parentheses, and footnotes below. Climate scientists, climate policy experts, and the university students and alumni (whom we call 'educated non-experts') were asked about disagreement among climate science 6 experts. Our questions probed participants' judgments about climate change and climate science as a whole, as opposed to particular areas or issues within climate science.4 Astrophysicists who work on cosmic ray physics were asked only about disagreements within cosmic ray physics. The items featured in Questionnaire 1 appear in Table 1. We are interested in how climate scientists (astrophysicists) view disagreements about climate change (cosmic rays).5 1.1 How much disagreement about climate change (cosmic ray physics) is there among the experts in your field?6 When considering the scientific methods and practices of your field, how important do you think the following factors are in the creation and persistence of disagreement about climate change (cosmic ray physics)? 1.2 Those involved in the disagreement are not equally well informed. 1.3 Those involved in the disagreement begin from different starting points, prior assumptions, or theoretically motivated expectations. 1.4 Those involved in the disagreement highlight or focus on different kinds of data as evidence. 1.5 Those involved in the disagreement have differing views of the appropriate scientific method. 1.6 The issues about which experts in my field disagree are very complex. 1.7 The data about which experts in my field disagree involve a great deal of uncertainty. 1.8 It is difficult to obtain enough of the right kind of data needed to resolve the disagreements that arise. When considering the practices of other experts in your field, how important do you think the following factors are in the creation and persistence of disagreement about climate change (cosmic ray physics)? 4 Some climate experts indicated that they would have liked to have commented on different aspects of the climate debate in different ways. However, because many members of the general public are unfamiliar with different aspects of the science of climate change and because we wanted to examine between-group differences on the same questions, we pitched our questions in this study at a general level. In a subsequent study, the results of which we do not report here, we asked different participant groups about particular issues within the overall climate debate. 5 For climate policy experts, we added "and climate policy experts" to "climate scientists." The heading for educated non-experts read "We are interested in how members of the general public view disagreements about climate change among climate science experts." 6 Throughout both questionnaires, the phrases 'your field' and 'my field' were changed to 'climate science' for educated non-experts. 7 1.9 Those involved in the disagreement are motivated by political ideology to defend particular theories. 1.10 Those involved in the disagreement are motivated by financial incentives to defend particular theories. 1.11 Those involved in the disagreement defend certain theories because those theories represent their life's work and they cannot bear to give them up. 1.12 Those involved in the disagreement are trying to garner attention or make a name for themselves. 1.13 Those involved in the disagreement are simply being stubborn, closedminded, or unreasonable. Table 1. Elements of Questionnaire 1. The answer choices for Question 1.1 were 'None,' 'Very little,' 'Some,' and 'A great deal.' For purposes of analysis, these were scored as 0, 1, 2, and 3. The answer choices for Questions 1.2 through 1.13 were 'Not important,' 'Slightly important,' 'Moderately important,' 'Important,' and 'Very important' (scored as 0, 1, 2, 3, and 4). All participants were asked demographic questions about their age, sex, education, and ethnicity or nationality. Experts were asked to describe the nature of their training and expertise. In light of existing public opinion research (Nisbet & Myers 2007, Leiserowitz et al. 2010, Leiserowitz et al. 2012, Pew 2012, Leiserowitz et al. 2015, Pew 2016, Gallup 2017), we hypothesized that experts and lay people would make divergent judgments about how much disagreement there was within climate science. Since scientific disagreements are rarely publicized except when connected to ideological, political, or religious disputes, we hypothesized that the non-experts would think that debates about climate science were driven more by ideological, social and personal factors than scientists in that field do. The items featured on Questionnaire 2 appear in Table 2. 8 We are interested in how climate scientists (astrophysicists) view disagreements about climate change (cosmic rays). 2.1 How much disagreement about climate change (cosmic ray physics) is there among the experts in your field? Please indicate the extent to which you agree or disagree with the following claims: 2.2 Two experts who are equally well-informed about the science of climate change (cosmic ray physics) might look at the same data but reasonably arrive at different conclusions. 2.3 Persistent disagreement about climate change (cosmic ray physics) may indicate that the tools or methods scientists use to study this phenomenon are not sufficiently reliable. 2.4 Persistent disagreement about climate change (cosmic ray physics) may mean that there is no correct theory in this domain. 2.5 Persistent disagreement about climate change (cosmic ray physics) may mean that there is more than one correct theory in this domain. 2.6 When I find that other experts who are as well-informed as I am hold opinions about climate change (cosmic ray physics) that are significantly different from my own, this sometimes leads me to become less confident in my own opinions.7 2.7 When I encounter an expert who disagrees with the dominant paradigms or theoretical perspectives in my field, this usually causes me to wonder whether they are really as well informed as other experts in the field. 2.8 The peer review process in my field is biased against publishing controversial hypotheses about climate change (cosmic ray physics).8 2.9 Minority or dissenting perspectives on climate change (cosmic ray physics) are often inappropriately silenced or suppressed within my field.9 2.10 Disagreements about issues such as climate change (cosmic ray physics) can be good for the health of my field. 2.11 When experts in my field disagree about climate change, it is most often about which public policy recommendations should be made in light of the science rather than about the science itself. 2.12 Publicizing the extent of disagreement among experts in my field reduces public trust in science. 2.13 Scientific experts should present their science to the general public without making any policy recommendations about what society should do in light of the science. 2.14 The science of climate change is settled enough to base public policy on it. 7 For educated non-experts, 'other experts' was changed to 'other people.' 8 Educated non-experts were given "The peer review process in climate science is probably biased against publishing controversial hypotheses about climate change." 9 Educated non-experts were given "Minority or dissenting perspectives on climate change are probably often inappropriately silenced or suppressed within climate science." 9 2.15 How much trust do you have in climate science experts? 2.16 How much trust do you have in scientists working in other areas of science, e.g. astrophysics? Table 2. Elements of Questionnaire 2. Question 2.1 was identical to Question 1.1, so that we could compare participants' answers to other questions with their answers to this particular question on both questionnaires. The response choices for Questions 2.2 through 2.14 were 'Completely disagree,' 'Mostly disagree,' 'Slightly disagree,' 'Neither agree nor disagree,' 'Slightly agree,' 'Mostly agree,' and 'Completely agree' (scored as 1 through 7). Questions 2.11 and 2.14 were not used in the version of Questionnaire 2 that was given to astrophysicists because there is not a visible public policy debate concerning cosmic ray physics. Questions 2.15 and 2.16 were presented only to educated non-experts. Answer choices for these latter two questions were 'None,' 'Very little,' 'Some,' and 'A great deal' (scored as 0 through 3). All participants were asked the same demographic questions as in Questionnaire 1. We hypothesized that, on the basis of scientists' experience with ambiguous data, interpretive flexibility, and professional disagreements within their disciplines, scientists would give higher estimates than non-experts of how reasonable it can be for scientists within their fields to draw different conclusions from the same data set (Q2.2). We also tentatively hypothesized that scientists would be more circumspect or self-reflective when confronted with an equally informed but divergent opinion than non-experts (Q2.6 & Q2.7). In the domain of morality, a number of philosophers (e.g., Ayer, 1936, Stevenson 1944) have argued that there are no correct answers to questions about what is morally 10 right or wrong on the grounds that there is no objective way to resolve disagreements that arise concerning them. We hypothesized that persistent disagreement in climate science would lead lay people to have similar doubts about whether there are objective facts in this domain, or, at a minimum, whether they doubt that current scientific methods are capable of giving us knowledge of those facts (Q2.3, Q2.4 & Q2.5). We hypothesized that educated non-experts would suspect that there was more inappropriate silencing within academic and scientific institutions than scientists would (Q2.8 & Q2.9). We hypothesized that climate scientists would be more likely than nonexperts to respond that the disagreements surrounding climate science did not actually fall within the science itself but rather lay in political debates about what to do in light of the science (Q2.11). We hypothesized that both experts and lay people would think that public discussions about the extent of disagreement within climate science have had a negative effect on public trust in science (Q2.13). Relatedly, we hypothesized that non-experts would express greater confidence in the results, methods, and authority of scientists working in other areas than of those working in climate science (Q2.15 & Q2.16). 11 3. Participants We recruited a total of 3,367 participants for our study from January to April, 2017. Participants were invited to complete either Questionnaire 1 or Questionnaire 2, which were made available online. Climate scientists, climate policy experts, and astrophysicists were recruited via professional listservs and newsletters. 457 climate scientists were recruited via messages to the following listservs that were selected because they represent major communication forums for experts in climate science: ecolog-l (maintained by the Ecological Society of America), coral-list (maintained by the U.S. National Oceanic and Atmospheric Administration's Coral Reef Conservation Program and Atlantic Oceanographic and Meteorological Laboratory), cryolist (affiliated with the International Glaciological Society), arcticinfo (for arctic climate scientists), and marine-b (for marine biodiversity scientists). These participants were asked whether they considered themselves to be climate scientists and to provide the following information about the nature of their expertise: (E1) What scientific discipline do you work in? (E2) Number of years you have worked in this field (include years spent as Ph.D. student or postdoc): ______ (E3) Within your scientific discipline, what is your primary area of expertise? (E4) Highest degree earned: ______ Of the 457 respondents (41% female, ave. age = 43, ave. years experience = 17) who were deemed to be climate scientists, 92% had graduate degrees in their respective fields, with 23% reporting expertise in atmospheric science (e.g., meteorology, atmospheric physics), 44% in one of the earth sciences (incl. oceanography, glaciology, geology, hydrology), 12 44% in biological science (primarily ecology), and 5% in other sciences.10 200 climate policy experts (31% female, ave. age = 48, ave. years experience = 18) were recruited via messages to climate-l (maintained by the International Institute for Sustainable Development), which was selected because it is the primary English-language listserv for climate policy experts. Participants with expertise in climate policy were asked similar questions about the nature of their expertise. 92% of climate policy experts had graduate degrees in their fields and reported expertise in economics, law, political science, anthropology, resource management, conservation, agriculture, and philosophy. 99 astrophysicists with expertise in cosmic ray physics (17% female, ave. age = 49, ave. years experience = 22) were recruited via an invitation in a bulletin of the International Astronomical Union Division D (High Energy Phenomena and Fundamental Physics) and listservs for researchers affiliated or collaborating with scientists at the Dublin Institute for Advanced Studies, the Pierre Auger Observatory (Argentina), and the High Energy Stereoscopic System (Namibia). Because the total global community of cosmic ray physicists is only a few hundred, this sample represents a good proportion of the scientists in this field. 91% of experts in cosmic ray physics have earned a doctoral degree. Among climate scientists 45% reported the U.S. as their nationality, as did 30% of climate policy experts and 17% of astrophysicists. 697 undergraduates enrolled at University College Dublin (52% female, ave. age = 23, 83% white Irish) were recruited via email with assistance from the UCD IT Services and Student Services offices. 1,914 alumni from the University at Buffalo (44% female, ave. age = 52, 82% Caucasian) were recruited via email with assistance from the UB Office of Development and Alumni Communications. 10 These percentages sum to more than 100% because several scientists indicated expertise in more than one of our areas of classification. 13 4. Results and Discussion 4.1 Summary The mean responses of the five participant groups to each of the items in Questionnaires 1 and 2 are summarized below in Tables 3 and 4.11 For ease of reference, the rightmost column indicates the midpoint of the response scale for each item. 11 Throughout this paper, we report mean rather than median responses, even though we often use nonparametric statistics to analyze these data, because we believe that means in the present case are more informative than medians. Additional information about these statistics can be found in the supplementary materials document. 14 Question Content Climate scientists Climate policy experts Undergraduates Alumni Astrophysicists Midpoint Q1.1 How much disagreement 1.2 1.4 1.7 1.6 2.0 1.5 Q1.2 Not well-informed 2.3 2.4 3.0 3.0 1.8 2 Q1.3 Different starting points 2.6 2.9 2.9 3.0 2.9 2 Q1.4 Different data 2.6 2.3 3.0 3.0 2.5 2 Q1.5 Different methods 2.0 2.0 2.4 2.5 2.0 2 Q1.6 Complex issues 2.4 2.4 2.1 2.6 2.9 2 Q1.7 Uncertain data 2.3 2.7 2.3 2.3 2.6 2 Q1.8 Difficult to obtain data 2.1 2.3 2.3 2.2 3.0 2 Q1.9 Political ideology 2.5 2.7 3.1 3.3 0.9 2 Q1.10 Financial incentives 2.1 2.0 3.2 3.2 1.2 2 Q1.11 Life's work 1.9 2.2 2.6 2.6 2.3 2 Q1.12 Fame 1.7 1.7 1.8 2.0 1.9 2 Q1.13 Stubbornness 1.7 1.5 2.0 2.1 1.4 2 Table 3. Mean responses of the five participant groups to each of the items in Questionnaire 1, along with an additional column indicating the midpoint of the relevant response scale. 15 Question Content Climate scientists Climate policy experts Undergraduates Alumni Astrophysicists Midpoint Q2.1 How much disagreement 1.0 1.5 1.8 1.6 2.0 1.5 Q2.2 Different conclusions 3.6 4.1 4.6 4.6 4.8 4 Q2.3 Unreliable methods 3.1 3.5 4.0 4.0 4.8 4 Q2.4 No correct theory 2.3 2.7 2.9 3.2 4.3 4 Q2.5 More than one correct theory 3.3 3.7 4.6 4.2 4.1 4 Q2.6 Decreased confidence 3.6 3.2 3.5 2.9 4.5 4 Q2.7 Doubt their expertise 5.0 5.0 4.7 4.8 4.1 4 Q2.8 Biased peer review 3.1 3.7 4.4 3.8 4.0 4 Q2.9 Dissent silenced 2.7 3.4 4.2 3.8 3.2 4 Q2.10 Healthy disagreement 4.5 4.5 4.4 4.8 5.5 4 Q2.11 Public policy disagreement 5.1 5.2 4.8 4.8 n/a 4 Q2.12 Reduces trust 4.6 4.2 4.8 4.1 3.6 4 Q2.13 Neutral on policy 3.6 3.1 3.4 3.6 3.7 4 Q2.14 Settled enough 6.0 5.8 5.2 4.8 n/a 4 Q2.15 Trust in climate scientists n/a n/a 2.6 2.5 n/a 1.5 Q2.16 Trust in other scientists n/a n/a 2.7 2.7 n/a 1.5 Table 4. Mean responses of the five participant groups to each of the items in Questionnaire 2, along with an additional column indicating the midpoint of the relevant response scale. 4.2 Disagreement Questions 1.1 and 2.1 asked participants about the extent to which there is disagreement among experts about climate change or cosmic ray physics. Significant between-group 16 differences were observed (cf. Figure 1).12 Post-hoc tests performed on the answers of different pairs of participant groups indicated significant differences in every case, with the largest differences occurring between the answers of climate scientists and undergraduates (r = .37), climate scientists and alumni (r = .23), and climate scientists and astrophysicists (r = .44). The mean response of climate scientists fell significantly below the midpoint of 1.5 (r = .46), whereas those of undergraduates (r = .28), alumni (r = .11), and astrophysicists (r = .66) fell significantly above the midpoint. The mean response of climate policy experts did not differ significantly from the midpoint. Statistical note: Although r values are primarily known as measures of correlation strength, they can also be calculated as measures of effect size for a variety of statistical tests. While it is common in some disciplines to report different measures of effect size for different kinds of tests, an important benefit of reporting the same effect size measure for different tests is that it allows for easy comparisons between the sizes of different kinds of effects.13 12 Kruskal-Wallis tests were used on each question to examine overall between-group differences. Pairwise differences were then examined with post-hoc Mann-Whitney tests. Cf. the supplementary materials document for further details concerning these tests and other statistics. 13 For correlations, r values indicate the strength of the correlation. For one-sample t-tests, r values indicate how large was the difference between the observed mean was and the neutral midpoint. For MannWhitney tests, r values indicate how large was the difference between the two distributions being compared. 17 Figure 1. Mean responses to Questions 1.1 and 2.1 (combined), organized by participant group. Error bars in all figures represent 95% confidence intervals. Many climate scientists indicated expertise in more than one of the following three broad categories: atmospheric science, earth science, and ecology. Focusing only on the 79% of climate scientists whose expertise fell squarely within only one of these categories, we observed significant between-group differences in how participants responded to Questions 1.1 and 2.1 (cf. Figure 2).14 Post-hoc pairwise tests found significant differences between the responses of atmospheric scientists and ecologists (r = .25) and between earth scientists and ecologists (r = .17). Thus, within our sample of climate scientists, the closer 14 Kruskal-Wallis: H(2) = 17.48, p < .001. One-sample t-tests reveal that the mean response of atmospheric scientists did not differ significantly from the neutral midpoint of 1.5, whereas those of earth scientists (r = .42) and ecologists (r = .64) fell significantly below it. 18 a scientist's area of expertise was to what is considered to be the core of climate science, the greater the level of disagreement that scientist was likely to report there being within that field. Figure 2. Mean responses to Questions 1.1 and 2.1 (combined) of those climate scientists whose areas of expertise falls into a single broad category. One possible explanation of this interesting finding is that different aspects of the climate debate may be salient to scientists from different disciplines and that these differences may shape their responses to our question. For example, a marine biologist who knows that rising global temperatures stress the ecosystems she studies may focus on the fact that climate scientists do not disagree at all about whether the planet is warming. However, an atmospheric physicist participating in our study who works on 'climate 19 sensitivity'-i.e., the question of how much global surface temperatures would rise given a doubling of atmospheric carbon dioxide-may respond to our question in light of the fact that different models of climate sensitivity yield different estimates. It is widely noted that the debate about climate change within the United States is markedly different than in other parts of the world. It is perhaps surprising, then, that the mean response to Questions 1.1 and 2.1 (combined) of the American non-experts in our study fell significantly below that of the non-American non-experts (r = .08).15 We had expected that the greater controversy surrounding climate change in the U.S. would lead American participants to report greater disagreement among climate scientists than European non-experts. However, the opposite was the case. We also examined differences across nationalities in the responses of our climate experts. Again to our surprise, a MannWhitney test revealed that the responses of American climate experts (M = 1.0) fell significantly below those of non-American climate experts (M = 1.4, r = .23).1617 Participants' responses to Questions 1.1 and 2.1 may be shaped by the particular ways in which alleged disagreements within climate science are often leveraged for political ends, especially within the American context. Since experts in the U.S. know that any indication of disagreement or uncertainty about climate change that might appear in their responses could be used or even distorted by climate skeptics, they may feel pressure 15 U = 602,039.00, p < .0001. 16 U = 35,421.50, p < .00001. 17 Although we did not gather data on participants' political orientation, it seems likely that university alumni will be more conservative politically than either undergraduates or climate scientists and that this fact might lead alumni to report greater disagreement within climate science than either of the other groups. Although we did observe higher ratings on Q1.1 and Q2.1 among alumni than among climate climate scientists, the ratings of alumni were not higher than those of undergraduates. Nevertheless, political orientation is likely to interact with participants' ratings in a number of ways, some of which we are examining in a follow up study. Thanks to an anonymous reviewer for bringing some of these points to our attention. 20 to downplay the extent of disagreement that they know exists. Another possibility is that different aspects of the climate debate may be salient to participants from different countries. In the U.S., admitting the very basic fact that the planet is warming is viewed as a strongly liberal stance. In other contexts where everyone agrees that the planet is warming, a question about the extent of disagreement within climate science may bring to mind other issues or questions, some of which may involve more uncertainty or disagreement than the basic question of increasing average temperature. As compared to educated non-experts, climate experts more strongly agreed on Question 2.11 that disagreement within climate science concerned the public policy recommendations that should be made in light of the science rather than the science itself. Each pairwise comparison between one expert group and one non-expert group was significant (r's between .09 and .14). However, these differences were not as large as the results from Questions 1.1 and 2.1 led us to expect. Moreover, American climate experts (M = 5.4) more strongly agreed that disagreements about climate change were most often about public policy questions than about core scientific questions than non-American climate experts (M = 4.9, r = .19).18 4.3 Epistemic Factors Questions 1.2 through 1.8 asked participants how important they thought various epistemic factors, as defined above, were to the emergence and persistence of disagreement about climate change or cosmic ray physics. The first half of these questions concerned the behaviors or attitudes of individual experts who are involved in scientific disagreements. 18 U = 10,936.00, p < .01. 21 The remaining questions in this section concerned the phenomena studied by scientific experts-e.g., their complexity, uncertainty in the data, and how difficult it is to obtain the right kind of data. On the whole, climate scientists and climate policy experts rated the epistemic factors as being less important contributors to expert disagreement than nonexperts did. Participants' ratings of the importance of the epistemic factors represented in Questions 1.2 through 1.5 positively correlated with one another to a noteworthy extent (r's .23 to .46, all p's < .000001), and their ratings of the factors in 1.6 through 1.8 correlated even more strongly (r's .37 to .64, all p's < .000001). Correlations between factors from the two groups correlated less strongly or failed to correlate significantly at all. Participants' ratings on Questions 1.4 through 1.8 also significantly predicted how much disagreement they thought there was within climate science or cosmic ray physics (r's .18 to .36, all p's < .000001). There were no significant differences between the responses of the two groups of climate experts on any of these questions and the two groups of non-experts were generally in agreement about them. Importantly, the responses of climate experts in general differed from those of non-experts. Post-hoc comparisons between climate scientists and astrophysicists revealed significant differences in their responses to Questions 1.2, 1.6, and 1.8 (r's = .16, .15, .25), with astrophysicists being less likely to think that disagreements in their field were due to some of the experts being less well-informed and more likely to think that the issues about which experts disagree are very complex and that it is difficult to obtain enough of the right kind of data needed to resolve disagreements. In regard to the other epistemic factors, 22 climate scientists and astrophysicists gave largely similar responses. Experts in both fields indicated that there is agreement among experts about the methods, theoretical frameworks, and data that are needed to address the phenomena they study. One reason the mean response of climate experts to Question 1.2 fell significantly below those of educated non-experts may be that the general public is frequently exposed to poorly informed 'experts' through popular media-pundits who are in fact not climate scientists and thus not part of the community of scientists with which climate scientists would be having a scientific discussion. Hence, what gets perceived as expert disagreement by the public may not be taken as disagreement among genuine experts by those working within the field. In response to Questions 1.3, 1.4, and 1.5, climate scientists and climate policy experts gave lower ratings of the extent to which expert disagreement within climate science was due to different starting points, focusing on different kinds of data, or differing views about the appropriate methods to use than did undergraduates or alumni. Comparing the responses of experts and non-experts to Questions 1.2 through 1.5 to their responses to Questions 1.6 through 1.8, we see that both experts and non-experts largely agree that climate science involves uncertainties, deals with complex phenomena, and faces difficulties in obtaining the right kind of data. However, experts think that there is more theoretical or methodological agreement within their fields about how these epistemic fallibilities should be addressed or approached than non-experts do. Cosmic ray physicists gave largely similar answers to climate experts, except that the former rated the complexity of the phenomena they study and the difficulty of obtaining enough of the right 23 kind of data about them as more important contributors to scientific disagreement within their field. 4.4 Nonepistemic Factors Items in the second half of Questionnaire 1 examined participants' judgments about the extent to which disagreements about climate change and cosmic ray physics are nurtured by political ideology (Q1.9), financial incentives (Q1.10), professional stake (Q1.11), careerism (Q1.12), or the psychological factor of stubbornness (Q1.13). The opinions of the non-experts paint a more negative picture of the state of climate science than those of practitioners within the field. Participants' ratings of the importance of the nonepistemic factors represented in Questions 1.9 through 1.13 all strongly correlated with one another (r's .30 to .70, all p's < .000001). There were no significant differences between climate scientists and climate policy experts on any of the nonepistemic questions, and undergraduates and alumni differed slightly on only one question. By contrast, the responses of climate experts differed significantly from educated non-experts on almost all questions (r's .09 to .38). The responses of astrophysicists differed significantly from those of all groups on Questions 1.9 and 1.10 (r's .24 to .54) but exhibited fewer differences on the other questions. The responses of scientific experts to Question 1.9 and 1.10 differed sharply from those of the educated non-experts, with the latter giving substantially higher ratings of the importance of these factors. All participant groups largely agreed that trying to make a name for oneself (Q1.12) and stubbornness, closed-mindedness, and unreasonableness (Q1.13) were considerably less significant nonepistemic factors. 24 For each participant, we averaged together their response to each question about epistemic factors (Q1.2 – Q1.8) and their response to each nonepistemic question (Q1.9 – Q1.13), resulting in a composite epistemic score and a composite nonepistemic score. Mean composite scores for each group are represented in Figure 3. A two-way mixed ANOVA revealed significant main effects for participant group and composite score type (epistemic vs. nonepistemic), along with a significant interaction between group and score type.19 In other words, (i) for a given composite score type, the scores of the groups differed significantly, (ii) for a given participant group, there were significant differences between their epistemic and nonepistemic scores, and (iii) the extent to which a participant group's epistemic and nonepistemic scores differed varied across groups. Figure 3. Mean composite scores for questions about epistemic and nonepistemic factors on Questionnaire 1, organized by participant group. 19 Group: F(4, 1624) = 38.59, p < .00001. Score type: F(1, 1624) = 83.14, p < .00001. Group * score type: F(4, 1624) = 15.90, p < .00001. 25 The epistemic and nonepistemic composite scores were nearly identical for both undergraduates and alumni. In contrast, each of the three groups of experts gave lower ratings of the importance of nonepistemic factors in climate science and cosmic ray physics. Notably, climate experts' estimates of the importance of nonepistemic factors were a good bit higher than those of astrophysicists. Among climate experts, there were no significant differences between the composite epistemic scores of Americans and non-Americans, nor were there any differences across nationalities between the composite nonepistemic scores of American and non-American climate experts. On the whole, then, non-experts gave significantly higher ratings of the importance of both epistemic and nonepistemic factors in creating expert disagreement, and large differences were observed between experts and non-experts concerning non-epistemic factors such as political ideology, financial incentives, biases, and the need to defend one's life's work. We believe that educated non-experts' ratings of the importance of nonepistemic factors indicates a significant lack of trust in professional scientists. In many studies of public trust in science (e.g., Pew 2015), members of the general public often indicate a lack of trust in particular scientific findings or in the ability of scientists to understand certain phenomena (e.g., health). One can get the impression that it is perhaps the scientific method that they distrust. However, at least in our data, the lack of trust that educated non-experts display is more strongly directed more toward individuals working within scientific institutions (as represented by participants' composite nonepistemic scores) than toward 26 the methods employed by those scientists (as represented by their composite epistemic scores). 4.5 Normative Issues I: The Epistemology of Disagreement We turn now to data from Questionnaire 2 that concern normative epistemological issues such as whether it can be reasonable to draw different conclusions from the same data (Q2.2), whether participants decrease their confidence when faced with an equally wellinformed person who disagrees with them (Q2.6), and whether participants conclude that someone with an atypical view is not actually as well informed as those with more standard views (Q2.7). Participants' responses to Questions 2.2, 2.6, and 2.7 were significantly correlated. Participants who took a more permissive stance toward evidence (Q2.2)-by indicating that two equally well-informed experts might look at the same data but reasonably arrive at different conclusions (see White 2005)-were significantly more likely to take a conciliatory stance toward peer disagreement (Q2.6) (r = .18, p < .000001)20 and were significantly less likely to judge that someone with a divergent opinion was not actually well-informed (Q2.7) (r = -.20, p < .000001). Participants who adopted a conciliatory attitude (Q2.6) were significantly less likely to judge that people holding divergent opinions were not well-informed (Q2.7) (r = -.06, p = .02). On Question 2.2, the responses of climate scientists differed significantly from those of every other group (r's .14 to .26), with climate scientists being more likely to deny 20 In the philosophical literature on disagreement, there is some discussion of whether a positive answer to Q2.6-i.e., to what philosophers call conciliationism – entails, presupposes, or is otherwise strongly connected to a negative answer to Q2.2-an idea known as uniqueness (see White 2005, Kelly 2010, Ballantyne and Coffman 2012, Christensen 2016). It appears that our participants did not recognize this alleged connection between conciliationism and uniqueness, at least not within the domains in question. 27 there is more than one uniquely reasonable conclusion that can be drawn from a set of climate data. The responses of undergraduates, alumni, and astrophysicists did not differ significantly among themselves. Note that climate scientists were asked only about the science of climate change, so that it is possible that they might well adopt a more permissive approach to scientific evidence in other, less politicized domains. On Question 2.6, the mean responses of climate scientists, climate policy experts, undergraduates, and alumni all fell significantly below the midpoint, indicating a disinclination to modify their views in light of contrary opinions about climate science. By contrast, astrophysicists were significantly inclined to modify their views upon learning about other experts in cosmic ray physics who disagreed with them. Again, these data do not indicate that climate scientists and non-experts endorse a steadfast approach to peer disagreement across the board, since they might well adopt a more conciliatory approach in less politicized areas of science or everyday life. On Question 2.7, the mean responses of all four participant groups who were asked about climate change fell significantly above the midpoint, indicating a significant tendency to think that someone out of step with the mainstream with regard to climate science was not really a well-informed expert after all. The mean response of astrophysicists did not differ from the midpoint. 4.6 Normative Issues II: Objectivity Questions 2.3 through 2.5 probed participants' judgments about the ways in which expert disagreement might underwrite the conclusion that there is a lack of epistemological or metaphysical objectivity within the disputed domain. Participant groups were significantly divided on these questions. All pairwise post-hoc between-group comparisons on Question 28 2.3 were significant (r's .09 to .34), except for that between undergraduates and alumni. The mean responses of climate scientists, climate policy experts, undergraduates, and alumni all fell significantly below the midpoint on Question 2.4 (r's .36 to .74), but there were important differences regarding how far below the midpoint these scores fell. The mean response of climate scientists to this question was their lowest on Questionnaire 2. In response to Question 2.5, the mean responses of undergraduates and alumni fell significantly above the midpoint, while that of climate scientists fell significantly above. Thus, climate scientists and climate policy experts strongly rejected the idea that the often exaggerated and politicized disagreements about climate science that exist should lead people to skeptical conclusions about whether there are objective facts about the nature of climate change, whether those facts can be known, and whether current scientific methods are equal to the task of uncovering them. 4.7 Institutional Issues Questions 2.8, 2.9, and 2.10 examined participants' judgments about institutional biases and the benefits of expert disagreement within the relevant domains. The responses of climate scientists differed significantly from those of every other group (r's .16 to .79) on Questions 2.8 and 2.9. That is, climate experts indicated that they did not think the professional debate about climate change, as represented in academic journals and other venues, involved significant bias against minority views, with climate scientists expressing this opinion more strongly than climate policy experts. Undergraduates suspected to a certain degree that institutional bias was present, and alumni seemed uncertain about whether or not this was the case. 29 When we put the data from Questions 2.8 and 2.9 together with the data from the questions about nonepistemic factors in Section 4.4 above, we get the unfortunate conclusion that educated non-experts think there is more personal and institutional bias operative within climate science than do climate experts themselves.21 These data seem to point to a crisis of trust in the men and women who work in climate science today. The mean responses of all participant groups to Question 2.10 fell significantly above the midpoint, with that of astrophysicists being the highest. Thus, despite the fact that disagreements within science can sometimes be exploited for ideological aims, climate scientists and climate policy experts still agree that disagreement can be good for the health of their field. There were significant correlations between participants' answers to Question 2.10 and their answers to 2.2 and 2.6. The more someone agreed that two experts could look at the same data but reasonably arrive at different conclusions (Q2.2), the more likely they were to think that disagreement could be good for a scientific discipline (Q2.10) (r = .31, p < .000001). Participants who reported taking a conciliatory approach to peer disagreement (Q2.6) were also significantly more likely to think disagreement could be good for science (Q2.10) (r = .23, p < .000001). 4.8 Public Trust in Science As compared to other participant groups, climate scientists and undergraduates more strongly agreed that publicizing disagreement about climate change among scientific experts reduces public trust in science (Q2.12). The two groups of non-experts in our study 21 We suspect that among non-experts without a college education this disparity may be even more pronounced. 30 were asked how much trust they have in climate science experts (Q2.15) and how much trust they have in scientists working in other areas of science (Q2.16). The mean response of each group fell significantly and substantially above the neutral midpoint for both questions (r's .81 to .92-the largest effect sizes reported in this study). The confidence that each group expressed in climate scientists was significantly lower than the confidence they expressed in other scientists. There were no significant differences between the two participant groups. Thus, while non-experts express less confidence in climate scientists than other scientists, they express very high levels of confidence in both. 4.9 Public Policy The remaining two questions from Questionnaire 2 concerned the public policy implications of expert disagreement. The mean responses of all participants except astrophysicists to Question 2.13 fell significantly below the midpoint. Thus, participants disagreed to a modest extent that scientists should keep out of public policy discussions. All participant groups (excluding astrophysicists, who were not asked this question) agreed on Question 2.14 that the science of climate change was settled enough for policy purposes, with climate experts agreeing most strongly. 5. Conclusion The goals of our study were to investigate the views of climate experts, astrophysicists, and educated non-experts in regard to the various methodological, personal, and institutional factors that these individuals think generate and sustain expert disagreement within climate science and cosmic ray physics and to examine how these participant groups 31 think such disagreements should be approached. By investigating participants' judgments about the factors underlying disagreements among scientific experts, we aim to supplement existing findings regarding the opinions of scientific experts and the general public, which are typically pitched at a more general level. By examining individuals' opinions about issues pertaining to the epistemology of disagreement in specific contexts, we hope that our findings can inform philosophical discussions about these epistemological matters, which take place at an especially high level of abstraction and generality. We observed that, as compared with educated non-experts, climate experts believe that within climate science (i) there is less disagreement within climate science about climate change, (ii) more of the disagreement that does exist concerns public policy questions rather than the science itself, (iii) methodological factors play less of a role in generating existing disagreement among experts about climate science, (iv) fewer personal and institutional biases influence the nature and direction of climate science research, (v) there is more agreement among scientists about which methods or theoretical perspectives should be used to examine and explain the relevant phenomena, (vi) disagreements about climate change should not lead people to conclude that the scientific methods being employed today are unreliable or incapable of revealing the truth, and (vii) climate science is more settled than ideological pundits would have us believe and settled enough to base public policy on it. We also observed that the uniquely American political context predicted participants' judgments in some of these domains. Our study also reveals that, as compared to climate scientists, astrophysicists working in cosmic ray physics were generally more willing to acknowledge expert disagreement within cosmic ray physics, more open to the idea that a set of data can have 32 multiple valid interpretations, and generally less quick to dismiss someone articulating a non-standard view as non-expert, than climate scientists. This was commensurate with the greater inherent uncertainty and data lacunae in their field. Experts in both climate science and astrophysics indicated that there is strong agreement within their fields about the methods, theoretical frameworks, and data that are needed to address the phenomena they study. In line with existing studies of the opinions of climate scientists (Oreskes 2004, Bray & von Storch 2008, Anderegg et al. 2010, Cook et al. 2013, Stenhouse et al. 2014, Cook et al. 2016) and the general public (Nisbet & Myers 2007, Leiserowitz et al. 2010, Leiserowitz et al. 2012, Pew 2012, Leiserowitz et al. 2015, Pew 2016, Gallup 2017), our findings show that despite the existence of a significant consensus on the causes of climate change among climate scientists, the general public continues to believe that climate scientists disagree over the fundamental cause of global warming. Our study goes beyond previous studies in examining (i) differences between expert and non-expert populations in regard to the factors these different groups think underlie expert disagreement in climate science, (ii) the attitudes of experts and non-experts toward normative issues of expert disagreement, and (iii) and how experts from politicized and depoliticized areas of science compare on various metrics. It is hoped that these results will inform our understanding of the nature and broader social consequences of expert disagreement. References 33 Anderegg, William R. L., Prall, James W., Harold, Jacob, and Schneider, Stephen H. 2010. "Expert Credibility in Climate Change." Proceedings of the National Academy of Sciences of the United States of America 107:12107-9. Ayer, A. J. 1936. Language, Truth and Logic. London: Gollancz. Ballantyne, Nathan, and E. J. Coffman. 2012. "Conciliationism and Uniqueness." Australasian Journal of Philosophy 9:657-70. Bray, Dennis, and von Storch, Hans. 2008. "The Scientific Consensus of Climate Change Revisited." Environmental Science & Policy 13: 340-50. Christensen, David. 2007. "Epistemology of Disagreement: The Good News", Philosophical Review, 116: 187-217. Christensen, David. 2009. "Disagreement as Evidence: The Epistemology of Controversy." Philosophy Compass 4: 756–67. Christensen, David. 2016. "Conciliation, Uniqueness and Rational Toxicity", Noûs 50: 584-603. Cook, John, Nuccitelli, Dana, Green, Sarah A., Richardson, Mark, Winkler, Bärbel, Painting, Rob, Way, Robert, Jacobs, Peter, and Skuce, Andrew. 2013. "Quantifying the Consensus on Anthropogenic Global Warming in the Scientific Literature." Environmental Research Letters 8: 024024. Cook, John, Oreskes, Naomi, Doran, Peter T., Anderegg, William R. L., Verheggen, Bart, Maibach, Ed W., Carlton, J. Stuart, Lewandowsky, Stephan, Skuce, Andrew G., Green, and Sarah A. 2016. "Consensus on Consensus: A Synthesis of Consensus Estimates on Human-Caused Global Warming." Environmental Research Letters 11: 048002. 34 Feldman, Richard. 2006. "Epistemological Puzzles about Disagreement." In Stephen Hetherington (ed.), Epistemology Futures, New York: Oxford University Press, pp. 194-214. Gallup News. 2017. "Global Warming Concern at Three-Decade High in U.S." Mar. 14, 2017. Available at: http://news.gallup.com/poll/206030/global-warming-concernthree-decade-high.aspx. Goldman, Alvin I. 2010. "Epistemic Relativism and Reasonable Disagreement." In Richard Feldman and Ted Warfield (eds.), Disagreement. Oxford: Oxford University Press, pp. 187-215. Kelly, Thomas. 2005. "The Epistemic Significance of Disagreement." In Tamar Gendler and John Hawthorne (eds.), Oxford Studies in Epistemology vol. 1. Oxford: Oxford University Press, pp. 167-196. Kelly, Thomas. 2010. "Peer Disagreement and Higher Order Evidence." In Richard Feldman and Ted Warfield (eds.), Disagreement. Oxford: Oxford University Press, pp. 111-174. Leiserowitz, A., Maibach, E., and Roser-Renouf, C. 2010. Climate Change in the American Mind: Americans' Global Warming Beliefs and Attitudes in January 2010. New Haven, CT: Yale Program on Climate Change Communication. Leiserowitz, A., Maibach, E., Roser-Renouf, C., Feinberg, G., and Howe, P. 2012 Climate Change in the American Mind: Americans' Global Warming Beliefs and Attitudes in September 2012. New Haven, CT: Yale Program on Climate Change Communication. 35 Leiserowitz, A., Maibach, E., Roser-Renouf, C., Feinberg, G., & Rosenthal, S. 2015. Climate Change in the American Mind: October, 2015. New Haven, CT: Yale Program on Climate Change Communication. Longino, Helen E. 1990. Science as Social Knowledge: Values and Objectivity in Scientific Inquiry. Princeton: Princeton University Press. Matheson, Jonathan. 2015. The Epistemic Significance of Disagreement. Basingstoke: Palgrave MacMillan. Nisbet, Matthew C, and Myers, Terea. 2007. "The Polls-Trends: Twenty Years of Public Opinion about Global Warming." Public Opinion Quarterly 71: 444–70. Oreskes, Naomi. 2004. "The Scientific Consensus on Climate Change." Science 306: 1686. Pew Research Center. 2012. "More Say There is Solid Evidence of Global Warming." Washington, DC: Pew Research Center for the People & the Press. Available at: http://www.people-press.org/2012/10/15/more-say-there-is-solid-evidence-ofglobal-warming/. Pew Research Center. 2015. "Public and Scientists' Views on Science and Society." Available at: http://www.pewinternet.org/files/2015/01/PI_ScienceandSociety_Report_012915. pdf. Pew Research Center. 2016. "The Politics of Climate." Available at: http://www.pewinternet.org/2016/10/04/the-politics-of-climate/. Stenhouse, N., Maibach, E., Cobb, S., Ban, R., Bleistein, A., Croft, P., Bierly, E., Seitter, K., Rasmussen, G., and Leiserowitz, A. 2014. "Meteorologists' Views About 36 Global Warming: A Survey of American Meteorological Society Professional Members." Bulletin of the American Meteorological Society 95: 1029–40. Stevenson, C. L. 1944. Ethics and Language. New Haven: Yale University Press. White, Roger. 2005. "Epistemic Permissivism", Philosophical Perspectives 19: 445-59.