Understanding 'Understanding' in Public Understanding of Science This study examines the conflation of terms such as 'knowledge' and 'understanding' in peer-reviewed literature, and tests the hypothesis that little current research clearly distinguishes between importantly distinct epistemic states. Two sets of data are presented from papers published in the journal Public Understanding of Science (PUS). In the first set, the digital text analysis tool, Voyant, is used to analyze all papers published in 2014 for the use of epistemic success terms. In the second set of data, all papers published in PUS from 2010–2015 are systematically analyzed to identify instances in which epistemic states are empirically measured. The results indicate that epistemic success terms are inconsistently defined, and that measurement of understanding, in particular, is rarely achieved in public understanding of science studies. We suggest that more diligent attention to measuring understanding, as opposed to mere knowledge, will increase efficacy of scientific outreach and communication efforts. Keywords Public understanding of science, epistemology, understanding, literacy 1. Introduction One of the primary goals of science communication is to improve the public's epistemic relationship to science. This immediately raises two questions: 1) What kind of epistemic relationship to science do we want the public to have?1 2) What kind of communication is most effective for producing it? Without a clear answer to the first question, it's hard to do much with the second. If you don't know your destination, then it's hard to evaluate candidate routes-or to judge whether you've arrived at all. In other words, confusion or lack of clarity about the epistemic goals of science communication makes it difficult to develop or evaluate effective communication strategies. So, what exactly are the proper epistemic goals of science communication? Let us start here. From one angle, the answer might seem obvious: isn't the basic epistemic goal of science 1 This question, of course, can be thought of in a variety of ways depending on how we think of 'science': are we talking about scientific facts or theories, methods, community norms or conventions, and so on? 2 communication scientific literacy? Perhaps-but saying it hardly helps. As long as the term 'literacy' remains unclear, this is not much better than saying that we want the public to have "a good epistemic relation" to science.2 So, the question remains: what is this good epistemic relation? The goal of this article is not to defend an answer to this question. Our aim is more modest: we will argue that, in the literature on science communication, there is a persistent lack of clarity in the use of certain epistemic terms, especially 'knowledge' and 'understanding'. In some cases, this is probably harmless. However, as a general phenomenon, it is troubling, because-as we will argue-it suggests that researchers may be overlooking important differences between distinct epistemic states, and this can have far-reaching effects, shaping research questions and instruments, experimental design, the analysis and evaluation of data, and, in the end, even science communication and policy recommendations. But our aim here is not merely to flag a problem; we also aspire to shed some light. We hope that clearly distinguishing between relevant epistemic states will help researchers to improve or develop new research tools, opportunities, and perspectives, and thereby contribute to the collective effort to identify the proper epistemic goals of science communication and the best means to realizing them. In Section 2 of this article, we introduce several epistemic terms relevant to science communication, explain our focus on 'knowledge' and 'understanding', and further clarify the aim of our research. Section 3 addresses research methods and Section 4 presents our results and analysis. Finally, Section 5 leverages recent philosophical work on knowledge and understanding to explain the significance of our data within the context of research on science communication and concludes with thoughts about further directions. 2. Epistemic Terms and Epistemic States The term 'epistemic' derives from the Greek epistêmê, which is usually translated as 'knowledge' or 2 Cf. Jon Miller's remark that 'scientific literacy' is "one of those terms that is often used but seldom defined" (1983, 29). 3 'understanding'.3 The key idea is that a person with epistêmê has a "cognitive grasp" of some aspect of reality. So, an epistemic state is a mental state that involves-or purports to involve-some such grasp. Thus, on most accounts, merely to have a sensation such as cold or pain is not to be in an epistemic state-though, of course, a sensation may be the basis for a variety of epistemic states. Epistemic terms are words or phrases related to epistemic states; more specifically, we will say that 'e' is an epistemic term if and only if 'e' denotes, or refers to, an epistemic state. Examples of epistemic terms include 'belief', 'opinion', 'knowledge', 'understanding', 'comprehension', and 'being informed'. Note that it is an open question whether two epistemic terms refer to distinct epistemic states; for example, 'belief' and 'opinion' seem to refer to the same kind of state, even though they have different valences in everyday speech; the same may be said of 'understanding' and 'comprehension'. The object of an epistemic state is that aspect of reality the state purports to grasp. Many kinds of things can figure in or among the objects of epistemic states, including the contents of the natural world (trees, quasars, dolphins,...) as well as our best scientific theories about it (evolution, plate tectonics, relativity, anthropogenic climate change,...). As a start, we can say that being "scientifically literate" requires having enough of the right kinds of epistemic states about-that is, the right kind of cognitive grasp of-both relevant aspects of reality and the strategies that the scientific community uses to study these aspects. In the debates about scientific literacy-how we should think of it, what we should expect of the public, how to bring this about-much of the focus has been on determining the appropriate objects of these epistemic states. (With what aspects of science and reality should a scientifically literate person be familiar?) Significantly less attention has been paid to what exactly it means to be in the "right kind" of epistemic state vis-á-vis these objects. This is a critical question for the study and activity of science communication.4 3 For reasons to prefer 'understanding', see Zagzebski (2001, 353ff). 4 We will not enter the debate here about the specific content one should grasp in order to be properly regarded as scientifically literate. 4 We posit, first, that scientific literacy requires more than merely having beliefs and opinions about the world and about science. This is because beliefs and opinions can be false, but scientific literacy requires epistemic success; thus, it requires states such as true belief, knowledge, being informed, understanding, and comprehension-each of which entails a kind of successful cognitive grasp of its object. Because these states entail epistemic success, we call the terms that denote them epistemic success terms. So, in thinking about literacy, we should focus on epistemic success states. But again: on which ones, exactly? Perhaps the most basic issue is whether to focus on states that are best conceived atomistically or holistically. For example, paradigm cases of true belief and knowledge are atomistic in the sense that their objects-the aspects of reality they purport to grasp-are individual facts or individual bits of information. We might say that Sam has a true belief or knows something about evolution even if he grasps only the fact that evolution has something to do with mutation. By contrast, paradigm cases of understanding are holistic in two senses: first, their objects-the aspects of reality they purport to grasp-are bodies of facts; second, "grasping a body of facts" involves more than simply knowing/truly believing each of those facts individually. Sam may know quite a few facts about evolution (having done a fine job of memorizing them for an exam), yet altogether fail to understand evolution because he has no grasp of how those facts fit together.5 Arguably, it is this intuitive distinction between knowledge/true belief and understanding-and what follows from it-that explains why we accord greater value to the latter. This is also what makes it appealing to think that "scientific literacy" properly so-called involves not just a few bits of knowledge or true belief, but some measure of understanding. Hence, of course, the name of this journal. We return to this issue in Section 5. For the time being, however, there are two points: first, there is a non-trivial difference between knowledge (or, more weakly, true belief) and understanding; and second, this difference is highly relevant to research on science communication and literacy. Whether we treat the desired outcome of science communication atomistically 5 Some regard the difference between these states as having a phenomenological marker: the "Ah ha!" of understanding (Gopnik 2000; cf. Trout 2002). 5 (as a form of knowledge/true belief) or holistically (as a form of understanding) is no small matter. Communication strategies appropriate to one may be inappropriate for the other, and techniques for measuring one may do a poor job of measuring the other. Our question, then, is whether sensitivity to the difference between knowledge/true belief and understanding is reflected in the recent literature on science communication and literacy. Sections 3 and 4 describe our empirical investigation into this question. For the purposes of this research, we needed to specify a core conception of understanding that would distinguish it from knowledge/true belief at least partly in functional terms. Based on the uncontroversial idea that someone who understands a body of information is able to "work" with it-e.g., to make new connections and inferences-we adopted the following "minimal definition": One understands a subject (issue, concept, theory,...) only if one grasps how a constellation of facts relevant to that subject are related to one another (causally, inferentially, explanatorily, &c.) in such a way as to be able to make new connections or draw new inferences with novel information. As a result, the object of understanding is always a body-and never a single piece-of information.6 The 'only if' indicates that our "minimal definition" is not an analytical definition; we intend to be providing only necessary conditions for understanding, even if they may be sufficient in some cases as well. Moreover, it seems that more detail would be needed in order to determine in a particular case whether we should attribute understanding.7 Nevertheless, there is no serious disagreement in the philosophical literature about this conceptual core. Thus, we treat a definition or use of the term 'understanding' as "standard" if and only if it is consistent with our minimal definition. Equipped with this minimal definition, we sought to answer three research questions focusing on recent publications in the journal Public Understanding of Science (PUS): [RQ1] Are epistemic terms (such as 'knowledge', 'belief', and 'understanding') defined or differentiated 6 For more on the idea that understanding entails certain practical abilities, see Section 5. 7 We suspect that such attributions will have a contextual character (Wilkenfeld et al. 2016), thus aligning our stance on understanding with other contextualist or interest-relative approaches to the epistemology of knowledge attributions (Stanley 2005; Cohen 1999). 6 in articles in PUS in 2014; in particular, is 'understanding' (and cognates) used or defined in a manner consonant with the core conception captured by our minimal definition? [RQ2] How do recent (2010–2015) PUS articles attempting to measure epistemic states use and define the epistemic success terms they are targeting? [RQ3] Do papers attempting to measure epistemic states effectively do so-i.e., do they plausibly measure what they claim to be measuring?-and how do types of instruments used influence the epistemic state assessed? Finally, before turning to our methods and results, a caveat, a comment, and a clarification. First, we recognize that many researchers use terms such as 'knowledge' and 'understanding' relatively loosely and that their research may be no worse for it. In fact, we believe that many of the papers we select for semantic criticism are of very high quality. Our purpose here is not to put research results into question, but to use semantic analysis to determine whether sensitivity to the difference between two quite different states of mind-knowledge/true belief and understanding-is reflected in the recent literature on science communication and literacy. That the answer turns out to be, "No," may not indicate a problem for particular research articles; however, it does suggest a way to enhance the conceptual clarity and richness of a very important research domain. Indeed, one way to understand the purpose of this article is just to suggest that research on science communication would benefit from less casual use of terms such as 'knowledge' and 'understanding'. In this context, there is something to be learned from philosophy, which generally treats these as technical terms that refer to importantly distinct states of mind. Second, to treat a term such as 'understanding' as a piece of technical vocabulary is not to make its definition arbitrary. Of course, authors may use words as they like (provided that, unlike Humpty Dumpty, they communicate this to their readers). But the point of our technical treatment of 'understanding' is not to focus on our preferred sense of the term; rather, the point is to capture something like the conceptual core of the notion of understanding expressed in ordinary speech. Thus, even if people often use 'knowledge' and 'understanding' interchangeably, it remains part of our ordinary conception of understanding that it is holistic rather than atomistic, and that it is possible to know things about a subject without also understanding it. In 7 this respect, we think that our minimal definition of 'understanding' has some normative force vis-à-vis nontechnical uses of the term: deviations from the minimal definition are explained as non-standard uses. Third, in his "manifesto" for "The Simonyi Professorship Chair for the Public Understanding of Science" at Oxford, Charles Simonyi claimed that we should take 'understanding' in the context of the mission of the Chair "a little poetically as well as literally," suggesting that this poetic interpretation involved appreciating "the order and beauty of the abstract and natural worlds," sharing in scientists' "awe" in the face of riddles, and empathizing with "the scientists who are humbled by the grandeur of it all" (Simonyi 1995). To produce understanding of this sort is a wonderful goal for science communicators to have. But such "aesthetic" understanding presupposes the satisfaction of some more basic (and less glamorous) epistemic conditions-that is, it presupposes understanding in the "literal" sense. (You can hardly appreciate the elegance and power of Darwin's theory of evolution while remaining ignorant of the workings of natural selection.) Again, this "literal" notion of understanding is our focus here. 3. Methods Dataset 1: Definition and Conflation in 2014 The first dataset is a snapshot of the use of epistemic success terms in one year's worth of articles published in Public Understanding of Science (PUS). For the qualitative and quantitative analysis, we collected all 67 of the articles published in 2014. Each of the eight 2014 issues was assigned to an individual researcher. We employed the digital text analysis tool, Voyant, to identify and assess the use of terms of interest in context, using the Keyword in Context function (KWIC). The KWIC function allows for identification and analysis of keywords within their local context throughout a corpus. KWIC has been successfully used for this purpose in literary, historical, scientific, and social scientific context (Maramba et al. 2015; Fankhauser et al. 2014; Baskarada and Koronios 2013). Papers were analyzed for their use of the epistemic success terms 'knowledge' and 'understanding' (and their boolean constructs). Other epistemic success terms or ability terms used in 8 each article, such as 'scientific literacy' or 'comprehension', were also recorded. Researchers gathered definitions used, if any, as well as instances of conflation of epistemic success terms. Both explicit and incontext definitions were considered using the context tools available in Voyant, as well as individual researcher analysis. We compared the use of terms, definitions, and implied meanings to our minimal definition above. Following initial data collection, each of eight readers qualitatively analyzed a different 2014 issue of PUS as a check of consistency and reliability. As a result, two independent researchers read and analyzed each issue. The lead author then read, reviewed, and analyzed the entire dataset as a final reliability check, tied together ideas and trends across the data, and assigned categories. We present quantitative analysis of the qualitative data as well as in depth qualitative analysis in §§4–5 below.8 Dataset 2: Measurement of Epistemic States To identify the instances in which researchers attempted to measure epistemic states in the context of the public understanding of science, two independent readers systematically analyzed all abstracts from the journal Public Understanding of Science from the year 2010 through the year 2015. The readers identified and collected thirteen articles that claimed to measure an epistemic state (e.g., knowledge, understanding, or literacy) and then analyzed each article, collecting data on definitions, instances of conflation, and in-context usage of epistemic success terms, noting the papers' thematic use of these terms.9 After the first two rounds of reading and analysis, the lead author coded all the measurement papers. The coding captured each paper's use and discussion of the epistemic terms 'knowledge', 'understanding', and 'literacy'. The coding scheme included identifying instances in which 'understanding' and 'knowledge' were used interchangeably, when understanding was described as a deeper or more complex state than knowledge, 8 A complete list of papers analyzed in Dataset 1 can be found in the supporting information accompanying the online version of this article. 9 A complete list of papers analyzed in Dataset 2 can be found in the supporting information accompanying the online version of this article. 9 or when literacy was described as involving both knowledge of facts and knowledge of scientific methods. Three researchers, including the lead author, also examined instruments used in any empirical studies. Instrument type, implementation, and item construction were analyzed. Consistent with our minimal definition of 'understanding', items clearly assessing understanding required participants to make inferences, include or incorporate new information, or otherwise draw upon interconnected ideas. Open-ended or interview questions asking participants to describe concepts could also capture understanding. In contrast, true/false or multiple choice questions that simply required the recollection of facts were considered knowledge questions. Answering such questions correctly merely requires a subject to have a true belief (or make a lucky guess); no inference from other facts is necessary.10 We found no multiple choice questions in this dataset that clearly gauged understanding. We discuss possible strategies for gauging understanding in scalable ways in the final section. The next section presents our qualitative analysis and quantitative representations of the overall qualitative trends. 4. Results and Analysis Dataset 1: 2014 For the purposes of this analysis, we focus mainly on the definition and use of the term 'understanding', and its potential conflation with other epistemic success terms. We placed each article in Dataset 1 into one of five categories: papers that (1) explicitly drew the standard distinctions between terms; (2) implied distinctions consistent with our minimal definition; (3) were unclear about their conception of relevant epistemic states; (4) conflated epistemic success terms (e.g., used 'understanding' as synonymous with 'knowledge') in ways that were inconsistent with standard definitions; and (5) showed significant misunderstanding or misuse of epistemic success terms as compared to the standard definition. Only three papers of 67 included an explicit definition of 'understanding'. Frequency data from the analysis dataset 1 can be seen in Table 1 below. A 10 This is not to say, however, that no inference actually occurs. 10 complete list of the papers falling into each of the categories can be found in the supplemental material. Table 1: Number of articles published in PUS in 2014 grouped by analysis category determined by use of of the term "understanding". Analysis Category Frequency (1) Explicit Standard 1 (2) Implied Standard 6 (3) Unclear 47 (4) Conflated 11 (5) Non-Standard 2 Total 67 Unclear Use of 'Understanding' A full 70% of the papers published in PUS in the year 2014 were identified as exhibiting ambiguous use of the term 'understanding' and as lacking any clear distinction in usage of epistemic success terms (Category 3). In these cases, it is difficult to say what conceptions the authors have of understanding, knowledge, or literacy. And while papers in Category 3 sometimes conflate epistemic terms, it is not with enough frequency to warrant the claim that the author(s) have deviated from the standard distinctions. Consider, for example, a paper by Jasanoff (January 2014). Here the author criticizes early conceptions of public understanding as "oversimplified," an observation that might be consistent with the minimal definition of understanding. However, at no point in the paper does she explicitly define 'understanding'. She writes, for instance: The overwhelming conclusion from two decades of PUS research... is that PUS as originally conceived framed its object of inquiry too narrowly.... [I]t imagined a phantom public ignorant of basic factual knowledge and detached from science in its everyday doings; and it sought to bridge the perceived gap between the two with an oversimplified, even cartoonish, 11 notion of "understanding" that misconceived both what publics know and what they are capable of grasping. (pp. 22-23) It is notable in this example that Jasanoff argues for the prevalence of an "oversimplified notion of 'understanding'," by which she may mean to say that much of PUS research conflates knowledge and understanding. However, Jasanoff does not explain what an adequate notion of understanding consists in or how it might differ from oversimplified conceptions. Several research articles were published in 2014 in which understanding played a central role in the analysis presented but was never defined or distinguished from other epistemic states. For example, the arguments in Engdahl and Lidskog (August 2014) rest on the differences between public and scientific understandings of science and risk, but the authors do not define understanding or distinguish it from belief or knowledge. The majority of articles published in PUS in 2014 were placed into the "unclear" category simply; in most of these cases, the articles paid little direct attention to understanding or other epistemic states. Topics of these papers ranged from the cultural evolution of science fiction (Hrotic, November 2014) to public perceptions of competing scientific claims (Brewer & Ley, May 2014). Conflation or Non-Standard Use. In our analysis, we identified 13 papers (Categories 4 and 5) in which the terms 'knowledge', 'understanding', and 'literacy' were either used interchangeably or were defined in ways that were inconsistent with our minimal/standard definition. Eleven of these papers (Category 4) gestured at some possible distinction among epistemic success terms, but continued to use them interchangeably. The following examples from Buys et al. (February 2014) and Fang (April 2014) highlight the types of language and discourse surrounding epistemic success terms common in this category of papers. Buys et al. discusses public perception, engagement, and understanding of climate change. The authors survey common definitions of 'public knowledge', the deficit model, and 'public engagement', but the distinctions between these terms are subsequently blurred. When describing the intent of their study and 12 measurement instrument, the authors do not distinguish between knowledge and understanding. Consider this passage: Five open-ended questions were asked, with the use of prompts where necessary, to fully explore their understanding of, and opinions about, climate change. Questions tapped perceived knowledge of climate change including what interviewees believed about its main causes.... (p. 178) Yet given the design of this study, measuring understanding was an attainable goal, given the open-ended interview style. Although Fang (April 2014) intends to examine public understanding of nuclear power and local risks, the author clearly defines neither 'knowledge' nor 'understanding'. We place this paper into Category 4 (conflation), however, because this paper uses the terms 'knowledge' and 'understanding' interchangeably on multiple occasions, and in ways that make their individual meanings unclear (pgs. 284, 289–90, 291, 292). Fang does, however, state that public understanding is more complex than possessing mere scientific information, and that political processes can influence understanding. For this reason, this paper remains in Category 4, rather than in Category 5 (non-standard use). Such distinctions, though often subtle and few, separate papers such as Fang (2014) and Buys el al. (2014) from the two papers we have identified as adopting clearly non-standard conceptions of epistemic success terms (Category 5). Like most papers we examined, Dineshkumar et al. (August 2014), does not clearly define epistemic terms. However, the authors also use epistemic success terms indiscriminately when describing and analyzing their survey questions and data, labelling their qualitative survey questions "knowledge statements" but describing answers to those questions as revealing understanding (pgs. 665, 672). For example, they describe a section of their survey, saying that it "assessed knowledge and/or understanding of climate change" and go on to explain that it simply contained true-false statements (p.663). Allum et al. (October 2014), the other paper we place into Category 5, also contains numerous apparent conflations of understanding and knowledge. Passages such as the following illustrate how the authors use 'understanding' 13 and 'knowledge' interchangeably when trying to address literacy: If religious belief, or even pre-Enlightenment thinking, might be one of the bases of resistance to medical genetics, the key ingredient for acceptance is prototypically assumed to be scientific literacy-familiarity with and understanding of the critical facts and methods of science (Bodmer, 1985). Armed with such knowledge and a scientific, rationalist worldview, citizens should be no more concerned with developments in the field of medical genetics than they are about antibiotics or modern heart surgery. (p. 834) To be clear, we would certainly not deny that a legitimate conception of scientific literacy might presume possession of a variety of epistemic states concerning a variety of subject matter, as the Bodmer Report and others have suggested. The authors of this paper, though, treated these distinct epistemic states as essentially equivalent (and more frequently than in any of the other papers examined). Explicit or Implied Standard Distinction Of the papers published in PUS in 2014, we identified six in which the implied distinctions between epistemic states were great enough for us to conclude that the authors hold definitions of understanding consistent with our minimal definition (Category 2). In only one of these papers was understanding specifically defined (Category 1). The definition in Hine's paper (July 2014) is as follows: [P]ublic understandings of science viewed as active processes of weaving together diverse resources.... The model of public understanding of science as a contextual phenomenon, fashioned by circumstances and constructed within specific interactions shaped the approach taken in this paper to the analysis of online discussions about headlice. (p. 577) Note how this gloss emphasizes the gathering together of information and resources to create understanding consistent with the emphasis on a grasp of a body of knowledge. However, Hine does use 'knowledge' and 'understanding' interchangeably at a few points in the paper. Other papers in this set offer no explicit definitions of understanding or other epistemic states, but allude to the relative complexity of understanding vis-á-vis knowledge (Jaspal and Nerlich, April 2014), discuss problems of transmitting knowledge without context or connections between facts (Wibeck, February 2014), or represent knowledge as a component of understanding (Kapoon, November 2014). Several of these 14 papers contain some term conflation, but this is less frequent than in papers in Categories 3–5. The paper published in PUS in 2014 that comes closest to an explicit standard definition of understanding is Ludwig (November 2014), which defines understanding as a more robust and ambitious undertaking than knowledge. Ludwig's paper deals with the limits that the complexity and scope of contemporary science place on the knowledge of any one individual, and argues for an "extended" conception of scientific knowledge that takes into account the degree to which external resources such as the internet are embedded in our cognitive lives. Ludwig specifically addresses the difference between knowledge and understanding in the following passage: ...I want to suggest that we often use the term 'understanding' in a more ambitious way than 'knowledge'. In order to truly understand a complex issue, we not only need propositional knowledge but we also need to be able to consistently integrate this knowledge in our system of beliefs, values, and actions. (pp. 993–994) Ludwig does not explicitly explain how understanding of science entails the ability to connect individual pieces of knowledge and make inferences based on those connections, but the explicit and implicit definitions of his paper come closer to our minimal definition than those of any other paper published in 2014 in PUS. Dataset 2: 2010–2015 Our third research question asked whether the unclarity we observed could also be found in attempts to measure epistemic states (e.g., as part of a larger attempt to gauge the efficacy of a given communication strategy). The second dataset includes all papers published in PUS between 2010 and 2015 that claimed to measure an epistemic state. Of the 350 papers published in PUS in this time period, 13 made this claim. The authors of these papers used various instruments (i.e. surveys, interviews, observations of group discussions, etc.) to measure knowledge, understanding, or literacy. For this dataset, we first examine the use, definition, and conflation of epistemic success terms in the 13 papers we identified. We then present an analysis of the types of instruments used and evaluate whether or not the authors successfully measured the epistemic state they intended to measure. 15 Analysis of use of terms in Dataset 2 In the 13 papers aiming to measure an epistemic state, there were instances of major conflation of epistemic terms, minor conflation of epistemic terms, and implied standard distinctions between epistemic terms. The frequency of papers falling into each category can be seen in Table 2. Although some of these papers suggested a distinction between epistemic states consistent with our standard, none of the authors in this dataset explicitly defined understanding in a way that matches the standard definition.11 Table 2: Number of articles published in PUS measuring an epistemic state between 2010 and 2015, grouped by analysis category determined by use of of the term 'understanding'. Analysis Category Frequency Major conflation 5 Minor conflation 3 Implied Standard distinction 5 Total 13 In the five papers we identified as exemplifying a major conflation of epistemic terms, 'understanding', 'knowledge', and 'literacy' were often used interchangeably. In some cases, such as Dineskumar and Smardon (August 2014), the authors call their measurement items "knowledge questions", but then state that they measured understanding in their analysis of their results. Similarly, Vandermoer et al. (March 2011), use the term understanding in the title of their paper, but then analyze knowledge or literacy throughout the rest of the article, without clearly defining their terms. Such inconsistencies between survey items and analysis occur as well in Zia and Todd (March 2010) and Allum et al. (October 2014). As in Dataset 1, some of the articles included in Dataset 2 contain less obvious conflations of epistemic 11 A complete list of the papers falling into each of the categories represented in Table 2 can be found in the supplemental material accompanying the online version of this paper. 16 terms. While the lack of clear distinctions still prevails, the papers in this category exhibit less problematic confusions between epistemic success terms. Finally, a group of papers in Dataset 2 were more likely to differentiate between epistemic states. One such paper is Lach and Sanford (March 2010). The authors set out to measure understanding, and, as we discuss later, are successful in doing so. The authors' definition of scientific understanding includes knowledge of the scientific process as well as knowledge of the scientific facts, but does not include comprehension of how those facts or processes are connected to one another (pgs. 132-133). In this sense, their definition is still somewhat atomistic and for this reason, we do not identify their paper as having correct explicit distinction between epistemic states. Analysis of instruments for measuring epistemic states The papers in Dataset 2 utilize various forms of measurement instruments, including surveys or questionnaires, interviews, essays, mixed methods, and observational data from discussions. These different instruments were applied to measuring knowledge, understanding, and scientific literacy across the dataset with varying degrees of success. Here we analyze the instruments used, the epistemic states the authors set out to measure, and the efficacy of these instruments. Figure 1 shows the types of instrument used in each paper, the epistemic state each paper claimed to measure, and the epistemic state actually measured based on our analysis of the measurement instruments. As Figure 1 shows, a majority of the papers in dataset 2 aimed at measuring knowledge rather than understanding or another epistemic state. In fact, 9 of the 13 papers published between 2010 and 2015 in PUS attempting to measure an epistemic state aimed to measure knowledge. Only four papers aimed at measuring understanding in six years in PUS. In our analysis, two papers succeeded in measuring understanding (though only one of them explicitly aimed to do so). This again highlights the prevalence of conflation and non-standard usage of epistemic success terms in the literature. More importantly, understanding properly so-called was rarely a target of study. 17 Another key finding, visible in Figure 1, has to do with the types of instruments used. In none of the cases analyzed in dataset 2 was a survey or questionnaire, by itself, successfully deployed to measure understanding. Again, the majority of cases did not endeavor to gauge understanding in the first place, however the one attempt to do so with a survey or questionnaire was not effective. Surveys and questionnaires deployed in the papers in dataset 2 consisted mainly of multiple choice or true false questions, such as Mielby et al. (February 2013), Zia and Todd (March 2010), and Shein et al. (October 2014). Roberts et al. (July 2013) and Vandermoer et al. (March 2011) asked participants for perceived or self-reported knowledge with Likert scale responses to various scientific statements. In none of these cases are respondents asked to make Survey Questionnaire Interview Essay Instrument: Legend Mixed methods R oberts et al (2013) C acciatore et al (2014) Understanding Actually Measured... Shein et al (2014) R oos (2014) Allum et al (2014) Knowledge Understanding Lach and Sanford (2010) Claimed to Measure... Knowledge M ielby et al (2013) Vanderm oeret al (2011) M oghariya and Sm ardon (2014) Literacy Borderline Understanding Figure 1: Claimed versus actual epistemic states measured and instrument types used in papers published in PUS from 2010-2015. Each line represents a paper and connects claimed epistemic state to actual measured epistemic state. Colors of lines indicate the type of instrument used, as noted in the legend. 18 inferences or connections between facts, solve problems, or process novel information, as would be necessary to assess understanding. Only one of the papers in dataset 2 both attempted to and succeeded in measuring understanding. Lach and Sanford (March 2010) observed a discussion panel consisting of laypersons and experts on a scientific topic (nanotechnology). The discussion was recorded and coded, and the participants were also asked to answer preand post-test essay questions. The authors analyzed discussion and essay content for changes in comprehension over time, participants' ability to apply concepts locally, and their ability to metaphorically explain phenomena. The interconnectedness of multiple knowledge areas was explicitly examined. We regard these efforts as successfully gauging understanding according to our minimal, standard definition. The only other paper to successfully assess understanding did not explicitly endeavor to do so. Jee et al. (February 2015) conflate 'knowledge', 'belief', and 'understanding' throughout their paper, but state that they are intending to measure knowledge. They conducted semi-structured interviews, asking participants with varying levels of education open-ended questions about viruses. Semi-structured interviews offer an effective methodology for gauging understanding, as they allow researchers to probe further into respondents' views, grasp of connections, and models of an issue (Bruine de Bruin and Bostrom 2013). The authors claim in their methods section that interview participants were in fact asked further prompts to assess their understanding, and given the data collected, it appears that understanding was indeed the epistemic state measured in this paper. Again, this paper highlights some of the pitfalls of a non-standardized use of epistemic terms. The authors' efforts have uncovered a deeper epistemic state than advertised. 5. Discussion As we have documented, PUS researchers are typically not very clear about how they conceptualize the epistemic goals of science communication; nor is it typical to distinguish importantly distinct epistemic states. In particular, the idea that understanding is a distinct epistemic goal from knowledge is hardly, if ever, 19 discussed; instead, the literature on scientific outreach often treats understanding (taken "literally") as more or less synonymous with knowledge. More precisely, in synthesizing our results from Datasets 1 and 2, and attempting to answer our research questions, three main findings emerged. The first concerns our first and second research questions ([RQ1] and [RQ2]): It is rare for authors writing on the public understanding of science to exhibit sensitivity to the difference between epistemic states. There are exceptions to this observation, but this finding was consistent across both datasets. The second, and more surprising, finding is that very few of the papers published in the last six years in Public Understanding of Science actually measured understanding empirically, even if they explicitly set out to do so. The final finding addresses our third research question ([RQ3]), and shows that survey instruments can effectively measure knowledge, but measurement instruments designed to assess understanding tend to be more complex. Online surveys are one of the simplest ways for social scientists to gather data. Survey questions are relatively simple to construct and the data is easier to gather and analyze. But the only papers in this dataset that effectively measured understanding did so with semi-structured interviews, observations, and essay data-strategies that are more time-consuming to design and analyze and impractical for large sample sizes. Similarly, work grounded in mental model theory, often using these same methods, is an established way of assessing understanding (Bruine de Bruin and Bostrom 2013), but is generally too timeand resource-intensive for large samples. While it is relatively easy to assess knowledge/true belief using surveys, measuring understanding using comparable means requires considerable care in instrument design. All of this, of course, coheres with our second finding: if measuring understanding is difficult, then no wonder we tend to not do it-and instead measure other epistemic states as proxies. Once again, we want to speak to the worry that we have unfairly criticized quality research articles on the grounds that they fail to attend explicitly to an issue that simply may not interest them, and that may not be highly relevant to their particular research questions. Moreover, as one referee for this journal suggested to us, it may well be that some of these authors use 'understanding' in a non-epistemic sense. But first, we 20 remind readers that our intention is not to use our results as a metric of overall paper quality or value; nor do we recommend that all work in science education, communication, and cognate fields take up the matter of precisely defining epistemic states. Second, it is important to keep in mind that our definition of 'understanding' is a quite standard and minimal sense of the term when used to indicate a distinctive epistemic state. There are uses of the word 'understanding'-as in 'do you understand that your flight leaves in 10 minutes'-in which the word 'understanding' is being used to get at the concept of knowledge. And perhaps there are uses of 'understanding' that denote non-epistemic (or not-wholly-epistemic) states, as suggested by Simonyi above. We do not rule out that such concepts may be relevant to legitimate conceptions of scientific literacy. But we think that it is very plausible that there is a distinctive and standard sense of understanding as an epistemic state that should play a significant role in conceptions of scientific literacy. We conclude with some reflections on this idea. As we have conceived it, understanding represents a more robust epistemic achievement than knowledge (typically) does. As such, as any teacher knows all too well, it is a more difficult state to bring about. So, why think that it should be a central goal of science communication (which is, after all, hard enough)? For many of the same reasons why it is our proper goal in ordinary educational contexts: we want students to walk away not just with more true beliefs, but with some ownership over the material that enables their further growth. Granted, we often settle for mere knowledge (or true belief) in both education and science outreach, but as the emphasis in science communication has shifted away from the unidirectional transmission of dry facts to models that are more sensitive to context, framing, and engagement (Jasanoff 2004; McCright et al. 2016; Nisbet and Scheufele 2009; Roser-Renouf et al. 2015), it seems that science advocates are centrally concerned with helping the public understand both scientific issues that bear on their lives and enable them to function as engaged and informed citizens. For one thing, someone who understands will plausibly have an easier time integrating new information into their current stock. In the ideal case, understanding is poised to grow and improve over time and as our 21 stock of information develops. A related feature of understanding, as opposed to knowledge, is that it tends to confer what we call "flexible responsiveness." Someone who understands a subject matter is characteristically able to work within that domain in a variety of ways that go beyond the individual facts that they know. This may not be the case for someone who merely knows facts within that domain. As Elgin notes, understanding involves "an adeptness in using the information one has, not merely an appreciation that things are so" (2007, 35; see also Grimm 2012; Zagzebski 2001, 110–111). If this is true, then insofar as researchers are interested in deeper epistemic states than mere knowledge or belief, they would be better served by attending to understanding as a precise state (or in any case by not conflating distinct states). Moreover, as we argue in other work [REFERENCE REMOVED], there is reason to believe that understanding might carry further practical benefits for science communication-especially in cases where the science in question has become "ideologically-entangled" (sensu Kahan et al. 2011) and/or politically contested (Brulle 2014; McCright and Dunlap 2011; Torcello 2016). For example, consider a member of the American lay public who leans to the conservative side of the political spectrum culturally and suppose that this individual knows little about the science of climate change.12 Some apparently authoritative sources say that anthropogenic climate change (ACC) is underway and serious; friends and family say that it is not and offer reasons to not trust those sources, citing prominent media personalities. The person who merely believes that ACC is real-on the basis of objectively-reliable testimony, say-plausibly seems more vulnerable to such challenges than someone who understands something about the science of ACC and how climate scientists reach their conclusions, form a consensus, and so on (Ranney and Clark 2016; van der Linden et al. 2015). In short, we conjecture that the "epistemic ownership" involved with understanding leads to greater levels of epistemic resilience-in part, due to the flexible responsiveness that such ownership engenders. Be this as it may, the ownership/flexibility typically involved in understanding seems to account for our tendency to treat 12 We might think of this individual as belonging to one of the 45% of Americans belonging to the "Cautious", "Disengaged", and "Doubtful" segments of Leiserowitz et al.'s "Six Americas" of climate change concern (Leiserowitz et al. 2016; Roser-Renouf et al. 2015). 22 scientific literacy as an appropriate metaphor for public understanding of science. One is not literate if one merely knows how to recognize certain strings of characters; literacy requires a more flexible ability to employ the language in novel ways. Researchers can use these properties of understanding to their advantage in devising methods for gauging it. First, since understanding typically involves practical abilities-to assimilate new information, use the information one already has in novel ways, and other aspects of what we call flexible responsiveness-survey instruments measuring understanding can offer subjects scenarios they are unlikely to have already encountered. For example, investigators seeking to determine whether subjects understood how climate change works (more than simply truly believing some facts about climate change) could ask subjects to draw inferences from a certain body of facts (perhaps after providing an additional fact). Second, and relatedly, the ability to evaluate counterfactual conditionals related to the body of facts is generally a sign of flexible responsiveness. Subjects could, for example, be presented a hypothetical scenario involving a certain kind of intervention on the climate system-e.g., a system for changing the reflectance of seawater-that would interact with their existing stock of beliefs such that correct answers to multiple choice questions concerning such a scenario would be made more likely by one's holistic and flexible grasp of a body of facts. Such techniques are familiar to educators faced with the challenge of assessing the understanding of large numbers of students. They take ingenuity to develop, but represent reasonable ways of gauging understanding over and above true belief. One can readily envision further strategies for exploiting these features of understanding in order to study its production and dynamics. These suggestions represent only a starting point. But we believe that the additional effort is warranted and would fill a lacuna that currently exists in research on the public understanding of science. REFERENCES 23 Baskarada, Sasa, and Andy Koronios (2013) "Data, information, knowledge, wisdom (DIKW): a semiotic theoretical and empirical exploration of the hierarchy and its quality dimension", Australasian Journal of Information Systems 18. Bruine de Bruin, W., and A. Bostrom (2013) "Assessing What to Address in Science Communication", Proceedings of the National Academy of Sciences 110 (3):14062–14068 Brulle, Robert J. (2014) "Institutionalizing Delay: Foundation Funding and the Creation of U.S. Climate Change Counter-Movement Organizations", Climatic Change 122:681–694. Cohen, Stewart (1999) "Contextualism, Skepticism, and the Structure of Reasons", Philosophical Perspectives 13:57–89. Elgin, C. Z. (2007) "Understanding and the Facts", Philosophical Studies 32:33–42. Fankhauser, Peter, Jörg Knappen, and Elke Teich (2014) "Exploring and Visualizing Variation in Language Resources.", in, 4125–4128. Gopnik, Alison (2000) "Explanation as Orgasm and the Drive for Causal Understanding: The Evolution, Function and Phenomenology of the Theory-Formation System", in F. Keil and R. Wilson (eds.), Cognition and Explanation. Cambridge, MA: MIT Press. Grimm, Stephen (2012) "The Value of Understanding", Philosophy Compass 7 (2):103–117. Jasanoff, Sheila, ed. (2004) States of Knowledge: The Co-Production of Science and Social Order. Abingdon, UK: Taylor & Francis. Kahan, Dan M., Hank Jenkins-Smith, and Donald Braman (2011) "Cultural Cognition of Scientific Consensus", Journal of Risk Research 14 (2):147–174. Leiserowitz, Anthony, Edward Maibach, Connie Roser-Renouf, Geoff Feinberg, and Seth Rosenthal (2016) Climate Change in the American Mind: March, 2016. Yale University and George Mason University. New Haven, CT: Yale Program on Climate Change Communication. Maramba, Inocencio Daniel, Antoinette Davey, Marc N Elliott, Martin Roberts, Martin Roland, Finlay Brown, Jenni Burt, Olga Boiko, and John Campbell (2015) "Web-Based Textual Analysis of Free-Text Patient Experience Comments From a Survey in Primary Care", JMIR Medical Informatics 3:e20. McCright, Aaron M., Meghan Charters, Katherine Dentzman, and Thomas Dietz (2016) "Examining the Effectiveness of Climate Change Frames in the Face of a Climate Change Denial Counter-Frame", Topics in Cognitive Science 8:76–97. McCright, Aaron M., and Riley E. Dunlap (2011) "The Politicization of Climate Change and Polarization in the American Public's Views of Global Warming, 2001–2010", The Sociological Quarterly 52:155–194. Miller, Jon D. (1983) "Scientific Literacy: A Conceptual and Empirical Review", Daedalus 112:29–48. Nisbet, Matthew C., and Dietram A. Scheufele (2009) "What's Next for Science Communication? Promising Directions and Lingering Distractions", American Journal of Botany 96 (10):1767–1778. Ranney, Michael Andrew, and Dav Clark (2016) "Climate Change Conceptual Change: Scientific Information Can Transform Attitudes", Topics in Cognitive Science 8:49–75. Roser-Renouf, Connie, Neil Stenhouse, Justin Rolfe-Redding, Edward Maibach, and Anthony Leiserowitz (2015) "Engaging Diverse Audiences with Climate Change: Message Strategies for Global Warming's Six Americas", in Anders Hansen and Robert Coxs (eds.), The Routledge Handbook of Environment and Communication. New York: Routledge. Simonyi, Charles (1995) "Charles Simonyi's Manifesto", Manifesto for the Simonyi Professorship Chair for the Public Understanding of Science at Oxford University URL = <http://www.simonyi.ox.ac.uk/aims/charles-simonyis-manifesto>. Stanley, Jason (2005) Knowledge and Practical Interests. Oxford: Oxford University Press. Torcello, Lawrence (2016) "The Ethics of Belief, Cognition, and Climate Change Pseudoskepticism: Implications for Public Discourse", Topics in Cognitive Science 8:19–48. Trout, J.D. (2002) "Scientific Explanation and the Sense of Understanding", Philosophy of Science 69:212– 24 233. van der Linden, Sander L., Anthony A. Leiserowitz, Geoffrey D. Feinberg, and Edward W. Maibach (2015) "The Scientific Consensus on Climate Change as a Gateway Belief: Experimental Evidence", PLOS One 10 (2):e0118489. Wilkenfeld, Daniel A., Dillon Plunkett, and Tania Lombrozo (2016) "Depth and Deference: When and Why We Attribute Understanding", Philosophical Studies 173:373-393. Zagzebski, Linda T. (2001) "Recovering Understanding", in Mathias Steup (ed.), Knowledge, Truth, and Duty: Essays on Epistemic Justification, Responsibility, and Virtue. Oxford: Oxford University Press.