RESEARCH ARTICLE 10.1002/2017WR020609 Toward best practice framing of uncertainty in scientific publications: A review of Water Resources Research abstracts Joseph H. A. Guillaume1,2 , Casey Helgeson3 , Sondoss Elsawah2,4 , Anthony J. Jakeman2 , and Matti Kummu1 1Water and Development Research Group, Aalto University, Finland, 2Fenner School of Environment & Society, Australian National University, Australia, 3GREGHEC, CNRS, HEC Paris, France, 4Capability Systems Centre, School of Engineering and Information Technology, University of New South Wales, Australia Abstract Uncertainty is recognized as a key issue in water resources research, among other sciences. Discussions of uncertainty typically focus on tools and techniques applied within an analysis, e.g., uncertainty quantification and model validation. But uncertainty is also addressed outside the analysis, in writing scientific publications. The language that authors use conveys their perspective of the role of uncertainty when interpreting a claim-what we call here ''framing'' the uncertainty. This article promotes awareness of uncertainty framing in four ways. (1) It proposes a typology of eighteen uncertainty frames, addressing five questions about uncertainty. (2) It describes the context in which uncertainty framing occurs. This is an interdisciplinary topic, involving philosophy of science, science studies, linguistics, rhetoric, and argumentation. (3) We analyze the use of uncertainty frames in a sample of 177 abstracts from the Water Resources Research journal in 2015. This helped develop and tentatively verify the typology, and provides a snapshot of current practice. (4) We make provocative recommendations to achieve a more influential, dynamic science. Current practice in uncertainty framing might be described as carefully considered incremental science. In addition to uncertainty quantification and degree of belief (present in 5% of abstracts), uncertainty is addressed by a combination of limiting scope, deferring to further work (25%) and indicating evidence is sufficient (40%)-or uncertainty is completely ignored (8%). There is a need for public debate within our discipline to decide in what context different uncertainty frames are appropriate. Uncertainty framing cannot remain a hidden practice evaluated only by lone reviewers. Plain Language Summary Scientists address uncertainty not only in how they perform analyses, but also in how they write about their results. Not enough attention has been given to the ways in which scientists describe how uncertainty affects interpretation of their conclusions, which we call ''uncertainty frames.'' We aim to raise awareness of this issue by: (1) proposing a typology of eighteen uncertainty frames, addressing five questions about uncertainty, (2) describing the factors that affect how uncertainty is framed, (3) describing current practice by identifying and analyzing what frames are used in a sample of abstracts published in Water Resources Research in 2015, and (4) providing provocative recommendations on how uncertainty communication could improve. We hope to spark debate within our community about how scientists should be communicating the role of uncertainty in their results. 1. Introduction: The Concept of Uncertainty Framing The handling of uncertainty is central to the nature of research and progress in science. For a statement to be accepted as fact means that it is no longer considered uncertain. The credibility of scientific findings depends on how uncertainty has been addressed. Uncertainty can also be used as an excuse for delay and is blamed for contributing to the science-policy gap [Bradshaw and Borchers, 2000; Brugnach et al., 2007]. Accordingly, uncertainty has been increasingly discussed from the standpoints of both scientific methods and decision making [Morgan et al., 1990; Reichert and Borsuk, 2005; Pappenberger and Beven, 2006; Beven and Young, 2013; Reichert et al., 2015]. This discourse has notably included: Special Section: Engagement, Communication, and Decision-Making Under Uncertainty Key Points:  Uncertainty framing in scientific articles is an important means of addressing uncertainty; indicating role of uncertainty in a claim  A typology of frames describes conclusions in terms of maturity, scope, level of belief, depth of analysis, and relatability to the reader  Frequency of frames in abstracts is consistent with carefully considered incremental science; potential to be more influential, dynamic Supporting Information:  Supporting Information S1  Data Set S1 Correspondence to: J. H. A. Guillaume, joseph.guillaume@aalto.fi Citation: Guillaume, J. H. A., C. Helgeson, S. Elsawah, A. J. Jakeman, and M. Kummu (2017), Toward best practice framing of uncertainty in scientific publications: A review of Water Resources Research abstracts, Water Resour. Res., 53, 6744–6762, doi:10.1002/2017WR020609. Received 17 FEB 2017 Accepted 16 JUL 2017 Accepted article online 20 JUL 2017 Published online 9 AUG 2017 VC 2017. American Geophysical Union. All Rights Reserved. GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6744 Water Resources Research PUBLICATIONS 1. typologies and taxonomies of uncertainty [e.g., Walker et al., 2003; Brown, 2004; Krauss and Walker, 2006; Norton et al., 2006; Refsgaard et al., 2007], including discussion of ambiguity [Brugnach et al., 2008] and linguistic uncertainty in defining concepts [Regan et al., 2002]; 2. analysis of how to address uncertainty in modelling generally [e.g., Matott et al., 2009; Gupta et al., 2012; Guillaume et al., 2016]; 3. model identifiability and uncertainty in parameter estimation [e.g., Beck, 1987; Brun et al., 2001; Vrugt et al., 2002, 2008; Duan et al., 2003; Wagener and Gupta, 2005; Hill and Tiedeman, 2007]; 4. characterizing model accuracy or information [Gupta et al., 1998; e.g., Bennett et al., 2013; Nearing and Gupta, 2015]; 5. tackling cases in which probabilities are less meaningful, e.g., climate change [e.g., Lempert, 2002; Walker et al., 2013; Maier et al., 2016]; 6. analyses of how humans make decisions under uncertainty [Lipshitz and Strauss, 1997; Kahneman, 2011], and how uncertainty is viewed by decision makers [Isendahl et al., 2009]. Most of this literature focusses on the treatment of uncertainties within an analysis. Insights from real-world decision making also feed into efforts to improve analysis, and even discussion of linguistic uncertainty emphasizes its impact on applications [Regan et al., 2002]. But given the academic community's focus on publishing, it is useful also to reflect specifically on uncertainty in our core business, namely writing papers [McDonnell, 2016]. While typically not considered a part of the research and analysis proper, communication about uncertainty in the text of the academic paper plays a key role in how others (i.e., researchers, policy makers) interpret our conclusions and whether they accept them. Information is conveyed not just when reporting quantified uncertainty, but also through the writing choices adopted by an author, intentional, or otherwise. By our definition, uncertainty framing involves communicating how uncertainty affects the interpretation of a conclusion. An uncertainty frame provides a description of how uncertainty affects interpretation of a conclusion. This is a more specific issue than simply how uncertainty is viewed in general terms [e.g., Isendahl et al., 2009]. What it means will become more concrete with the description of frames in section 2. It should, however, already be clear that framing of uncertainty is an unavoidable part of research in conveying the reliability of our conclusions and how the reader should use them. Aspects of uncertainty framing have been researched by a variety of other disciplines (see section 3), including science studies, science education, linguistics, and philosophy. But this knowledge has not been translated into practice within our discipline, where (as in many fields) there is a disconnect between the treatment of uncertainties in our scientific methods and communication about uncertainty in our scientific papers. This communication-which largely comes down to uncertainty framing-receives far less scrutiny than do the methods. There is no public debate about the appropriateness of uncertainty frames for different situations: has the author chosen an appropriate description of how uncertainty affects interpretation of a conclusion? Frames are accepted solely on the reviewer's individual (often subconscious) judgement, rather than a more rigorous disciplinary consensus. There is a need for reflection specifically within our discipline, to look at what is actually being delivered and assess the potential for improvement. There are several specific ways that reflecting on framing of uncertainty can help the academic community, as researchers, readers, reviewers, journal editors, and authors. The ultimate goal is to encourage appropriate use of different uncertainty framing approaches to achieve intended aims. Best practices could define the circumstances in which different approaches should or should not be used. Awareness of the diversity of frames in use can help the reader to notice how authors have qualified their statements and to critically evaluate the claims they make. Reviewers have the opportunity to ask the author to better support or to further qualify their statements [Myers, 1985]. Greater understanding of framing might help journals to better train reviewers and authors to improve research quality. An author aims to balance several objectives, including advancing their field, appropriately justifying their conclusions, avoiding misinterpretation or misuse of results, and getting their work published. Uncertainty framing allows the author to make qualified claims even when they are not certain, thus advancing the field as much as possible while making a valid argument that is clear to the reader and acceptable to the reviewer. Authors should strive to improve their repertoire of framing approaches, their acuity in recognizing what frame suits what purpose, and their skill in predicting what effect it will have on the reader. Given that uncertainty framing is ubiquitous (though not always appropriate or explicit), it appears many authors learn this intuitively. Making framing practices Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6745 explicit can support teachers to help those who lag behind. A researcher who improves their ability to frame uncertainty improves their ability to contribute to knowledge. This paper therefore aims to raise awareness of uncertainty framing by: 1. proposing a typology of uncertainty frames that provides a vocabulary and framework to help select, interpret, or evaluate the information provided about uncertainty in conclusions; 2. placing the act of uncertainty framing within its broader context by drawing on literature from multiple disciplines; 3. analyzing the frequency of occurrence of frames in abstracts of Water Resources Research (WRR) to provide a snapshot of current practice, a preliminary validation of the typology, and to generate hypotheses for their use; 4. providing a provocative set of recommendations to spur debate about best practices in uncertainty framing. Our focus on the abstract of a paper assumes that is where authors have distilled their claims and accompanying uncertainty framing to the highest level. Frames used in the abstract should reflect the aspects of uncertainty that authors consider most important to convey to their readers, even if they differ from the frames used elsewhere in the article. The focus on articles published in WRR is justified by the authors' (interdisciplinary) interest in water resources, the reputation of the journal, and its sense of academic community through its affiliation with the American Geophysical Union and its hydrology section. Moreover, WRR has a long tradition of reflection on the state of the field [Hufschmidt, 1967; Klemes, 1986; Dooge, 1996; Gupta and Nearing, 2014; Sivapalan, 2015; Vogel et al., 2015]. The article begins by introducing the typology of uncertainty frames developed in this work (section 2). We then dive more deeply into the topic of uncertainty frames by describing the context in which uncertainty framing occurs, drawing on existing literature across disciplines (section 3). This provides background for the method used for our analysis (section 4). Results (section 5) describe the frequency of occurrence of frames and the form in which they occur in abstracts, as well as positing hypotheses regarding their use. Recommendations are drawn to improve the use and understanding of uncertainty framing, with the aim of initiating a discussion about best practices (section 6). 2. Introducing a Typology of Uncertainty Frames Fundamentally, framing of uncertainty is about ensuring readers have the information they need to evaluate an article's conclusions. Authors should anticipate readers' needs and ensure that the right information is available. Based on our analysis (see sections 4 and 5), we propose an uncertainty framing typology consisting of five core questions that can be asked about an article's conclusions (Figure 1). Answers to the questions are offered by 18 uncertainty frames. For each question, the subset of frames most useful for answering that question can, moreover, be arranged on a continuum (where the meaning of the continuum depends on the question). A rationale is provided for each core question, explaining its relevance to the management of uncertainty: 1. Is the conclusion ready to be used? Knowing to what extent a conclusion is ready for use (its maturity) allows remaining uncertainty to be anticipated. A definitive conclusion can be delayed where appropriate, and uncertainty can be reduced in the long term by investing in future work. 2. What limitations are there on how the conclusion can be used? If a conclusion was obtained by using assumptions to reduce uncertainty in the short term, the reader needs to know about the resulting limitations on the scope of the conclusion. 3. How certain is the author that the conclusion is true? Describing the author's level of belief in a conclusion (e.g., using probabilities) allows reasoning about uncertain information. 4. How thoroughly has the issue been examined? Uncertainty can alternatively be described by considering the effect of alternative assumptions (e.g., scenarios and uncertainty quantification), which provides an indication of how thoroughly an issue has been examined, referred to here as the depth of analysis. 5. Is the conclusion consistent with the reader's prior knowledge? Uncertainty can be handled in communication itself, by anticipating potential mismatch between the position of the author and the reader, and Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6746 presenting a conclusion in a way that facilitates its integration with the reader's prior knowledge, i.e., emphasizing relatability of a conclusion. Below, we discuss a range of answers to these core questions and how those answers are typically reflected in scientific publications. For each question, a table is provided listing the names of the associated frames, the answers that those frames provide, and example statements. To allow easy comparison, example statements are all modifications of the statement that ''Water supply is sufficient to meet demand,'' in which additional information about uncertainty has been introduced using the respective frame. There are two special frames that cut across questions (a) through (d), in each case anchoring the two ends of a continuum. The strongest statements are those framed as Expressed-as-Fact, meaning that no uncertainty is acknowledged. The weakest statements are those indicating that the only useful consequence of the work is for others to Learn-from-myproblems. Between these extremes, there is a broad range of ways to frame uncertainty. Further analysis may extend the list of frames, notably by looking at uncertainty framing outside of abstracts, or possibly by subdividing the frames already identified. Section 5 provides more detail, describing how we perceive each frame to be employed within the abstracts in our sample. It also speculates on some explanations as to why each frame might be chosen by an author. How certain is the author that the conclusion is true? To what extent is the conclusion ready to be used? What limitations are there on how the conclusion can be used? Is the conclusion consistent with the reader's prior knowledge? How thoroughly has the issue been examined? a) Maturity and utility Core question Continuum along which answers are placed Legend for sub-tiles Uncertainty frames Category of frames b) Scope of a claim c) Level of belief in a conclusion d) Depth of analysis e) Relatability less limitations more assumptions accepted increasing exploration of alternative assumptions increasing utility of results increasing degree of belief in assumptions x Uncertainty can be reduced in the long term by delaying a definitive conclusion and investing in future work Uncertainty can be described in terms that allow reasoning about uncertain information Uncertainty can be handled in communication by anticipating differences in opinion between the reader and author Rationale of core question Uncertainty can be described by considering the effect of alternative assumptions Uncertainty can be reduced in the short term by making assumptions that limit scope of applicability Expressed-as-Fact Degree-of-belief Take-it-or-leave-it Learn-from-my-problems Follow-the-contract Validate-and-Defend Learn-from-my-problems Suggest-a-new-research-agenda Step-towards-a-goal Build-the-foundations Expressed-as-Fact Repeat-with-shifted-scope New-view-on-existing-result Learn-from-my-problems Make-it-hypothetical Make-it-relative Restrict-scope-of-applicability Expressed-as-Fact Expressed-as-Fact Learn-from-my-problems Demonstrate-robustness Uncertainty quantification Triangulation Just-a-possibility Figure 1. Uncertainty framing typology, showing five categories of frames defined in terms of core questions, the answers to which fall onto a continuum. Uncertainty frames are mapped onto these continuums, describing the role of uncertainty in interpreting a conclusion. The rationale of each core question describes the relevance to the management of uncertainty, which in this context involves either reduction, description, or communication of uncertainty. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6747 2.1. Communicating Maturity and Utility Information should be used in ways commensurate with its limitations. Appropriate uncertainty framing helps to facilitate this. Answers to the question ''to what extent is the conclusion ready to be used?'' form a continuum where the results have increasingly ''mature'' uses, articulated here as one moves up the steps of Figure 1a. The first two frames only highlight problems and opportunities for future work. Here we make a distinction between stating that further work is needed (Learn-from-my-problems) and taking the extra step to suggest a particular hypothesis or method that could be pursued (Suggest-a-new-research-agenda). Next come preliminary or partial results, where the author may indicate that their conclusions provide only initial estimates (Step-towards-a-goal), or will enable others to further extend their work (Build-the-foundations). Finally, there are results that are presented as ready for the intended use (Expressed-as-Fact). Where authors refrain from commenting on maturity and readiness for use, the reader is left to make up their own mind with the information provided (likely to be interpreted as Expressed-as-Fact). Table 1 gives example statements illustrating the frames used to communicate maturity and utility. 2.2. Communicating Scope of a Claim Characterizing the applicability and limitations of a conclusion-what we call the ''scope''-is not traditionally appreciated as a means of addressing uncertainty, but is actually fundamental to it. A broad, sweeping statement is typically more uncertain than a specific, targeted one, requiring a greater number of supporting assumptions both explicit and implicit. It is for instance easier to demonstrate that a phenomenon has occurred once in particular circumstances than to describe the rules governing its occurrence. The frames most helpful for communicating scope lie on a continuum where caveats and qualifications decrease as more supporting assumptions are accepted (Figure 1b). At the bottom of this continuum, an author would comment only on the research process, perhaps highlighting challenges that prevent further conclusions from having any scope (Learn-from-my-problems). At the top, accepting all necessary assumptions allows for making general, unrestricted statements (Expressed-as-Fact). Between these endpoints lie intermediate indications of scope. Assumptions underlying a conclusion may limit its scope to a hypothetical rather than real world (Make-it-hypothetical). Accepting a subset of assumptions leads to real-world statements that can only be generalized to other circumstances where the same assumptions hold (Restrict-scope-of-applicability). These restrictions can be expressed by referencing conditions under which the conclusion holds, or by highlighting exceptions. Comparative statements require acceptance of assumptions that impact on the comparison while allowing assumptions to go unresolved if they have a similar effect in all cases (Make-it-relative). Such comparisons may or may not be accompanied by an explicit disclosure that the analysis allows only such comparative statements and does not support conclusions about the individual success of a method, model, or policy in absolute terms. Table 2 shows example statements for the frames used to communicate scope. Table 1. Uncertainty Frames Conveying Maturity and Utility of a Conclusion, With Answers That Those Frames Provide, and Example Statements Name of Frame To What Extent Is the Conclusion Ready to be Used? Example Statement Expressed-as-Fact The results are ready for use Water supply will not be an issue Build-the-foundations The results provide firm foundations for future work to build upon This analysis of consumption helps determine whether there is enough water Step-towards-a-goal The results are only preliminary, and more work is needed to achieve the goal The analysis provides an initial estimate of the water balance to determine if there is enough water Suggest-a-new-research-agenda While not directly useful, the results do suggest a new research agenda A water balance analysis could determine if there is enough water Learn-from-my-problems Problems need to be resolved before any action can be taken Further work is needed Table 2. Uncertainty Frames Conveying Scope of a Conclusion, With Answers That Those Frames Provide, and Example Statements Name of Frame What Limitations Are There on How The Conclusion Can be Used? Example Statement Expressed-as-Fact There are no restrictions; general statements can be made Water supply is sufficient Make-it-relative We can draw only comparative conclusions There is more water than without the proposed adaptations Restrict-scope-of-applicability Conclusions cannot be generalized; they have a restricted scope of applicability There is enough water as long as demand growth does not exceed 5% Make-it-hypothetical Conclusions are dependent on the hypothetical assumptions used Based on these assumptions, there would be enough water Learn-from-my-problems We can only make a statement about the research process Adequacy of water supply presented interesting challenges Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6748 Note that some indications of scope may simply reflect what the author considered to be interesting (and therefore Expressed-as-Fact), rather than a limitation on how the results can be used. Limitations may therefore be ambiguous in some cases. 2.3. Communicating Level of Belief in a Conclusion The idea of ''level of belief'' in results-or strength of support-is closely associated with the topic of uncertainty analysis, and falls onto a continuum perhaps best defined by the question ''How certain is the author that the statement is true?'' (Figure 1c, Table 3) The weakest framings communicate that the author is unwilling to form an opinion, or does not believe they can convince someone else. Intermediate degrees of belief convey a level of support while leaving the reader to make their own final judgment. Where authors believe the reader should accept the argument, the framing may invoke a notion of sufficiency. Authors may claim there is sufficient evidence either based on their own arguments, or by appeal to accepted standards. At the top of the continuum, evidence may be so strong that no explicit argument is required; the conclusion is beyond dispute. Within a text, the first two frames are often quite explicit, with meta-statements about insufficiency of evidence (Learn-from-my-problems), or personal opinion (Take-it-or-leave-it). Degree-of-belief can be expressed qualitatively, using terms like ''probably'' or ''unlikely,'' or expressed quantitatively, using confidence levels, p-values, or measures of model fit. Sufficiency of evidence may be less directly expressed, for example, by indicating that a model is validated, ''good'' results are obtained, or a hypothesis is considered supported (Validate-and-Defend). Follow-the-contract involves additional reference to standard or accepted methods or evaluation criteria, and an argument that these methods have been adequately applied. Accepted facts may appear not just as unjustified statements, but also with emphasis on their certainty (Expressed-as-Fact). 2.4. Communicating Depth of Analysis ''Depth of analysis'' is closely related to the concept of level of uncertainty, which is common in typologies of uncertainty, capturing a continuum between ''deterministic knowledge'' and total ignorance [Walker et al., 2003]. We consider depth of analysis to correspond to a continuum of increasing exploration of alternative assumptions (Figure 1d, Table 4). An author does not know if they are totally ignorant, but can communicate their recognized ignorance. A single analysis (and underlying set of assumptions) yields a single scenario from an uncertainty viewpoint. The use of multiple lines of evidence is referred to as triangulation in the social sciences. When the consideration of alternative assumptions is sufficiently thorough, their aggregate properties can be communicated, in the form of bounds or probability distributions. If the remaining unexplored assumptions do not have a significant effect, the conclusion is considered robust to that uncertainty. Finally, when assumptions are considered to be exhaustively explored, this means we have ''deterministic knowledge.'' Table 3. Uncertainty Frames Conveying Level of Belief in a Conclusion, With Answers That Those Frames Provide, and Example Statements Name of frame How Certain Is the Author That the Conclusion Is True? Example Statement Expressed-as-Fact This is accepted as truth There is clearly enough water Follow-the-contract There are clear requirements for acceptance of this kind of statement, and in this case, the contract is satisfied According to standard methods, there is enough water Validate-and-Defend This has been sufficiently validated that I would be willing to defend it The analysis demonstrates that there is enough water Degree-of-belief I've quantified my degree of belief in terms of the model R2, which in this case is 0.8 The model (R250.8) indicates that there is enough water Take-it-or-leave-it I'm convinced, but others can make up their own minds In my professional opinion, there is enough water Learn-from-my-problems There are too many problems to know We do not know whether there is sufficient water Table 4. Uncertainty Frames Conveying Depth of Analysis Supporting a Conclusion, With Answers That Those Frames Provide, and Example Statements Name of frame How Thoroughly Has the Issue Been Examined? Example Statement Expressed-as-Fact Examination was so thorough that it is considered perfect It has been conclusively shown that there is enough water Demonstrate-robustness Examination has been sufficiently thorough to demonstrate robustness There is enough water in all scenarios considered Uncertainty-quantification A broad examination allows the uncertainty to be quantified The available water ranges from 101% to 130% of requirements Triangulation The use of several approaches provides a triangulation Multiple methods agree that there is enough water Just-a-possibility We only know enough to know this is a possibility Based on this estimate, it is possible that there is enough water Learn-from-my-problems Problems have prevented useful examination The analysis was unable to determine whether there is enough water Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6749 Within a text, recognized ignorance takes the form of identifying knowledge gaps (Learn-from-my-problems). When a conclusion is framed as Just-a-possibility, it may be highlighted as a possible scenario or an estimate. Triangulation uses explicit references to multiple methods or data sources. Uncertainty-quantification takes a broad range of forms depending on the method used, including ranges, various representations of probability distributions, and less formal references to lower bounds or orders of magnitude. Demonstrate-robustness includes any reference to the notion of insensitivity to assumptions including, for example, in statistical hypothesis testing. To say that a conclusion is statistically significant means that it is insensitive to changes in the sample drawn from the population. Deterministic knowledge may be highlighted as a settled issue (Expressed-as-Fact). 2.5. Communicating Relatability: Relation of Results to Readers' Prior Knowledge The commonality among all the preceding frames is that they establish the status of a new claim. We identified two ''relatability'' frames that instead engage with the reader's existing opinion (Figure 1e, Table 5). Uncertainty is dealt with by anticipating potential mismatch between the author and the reader's opinions. The two frames deal with the opposing situations where the author either expects that the reader may reject their argument, or expects that the reader already believes the statement. The first frame, Repeat-with-shifted-scope, anticipates possible rejection by making another, more strongly supported, claim, usually with a narrower scope. If the reader does agree with the author, then the author has been able to make a strong claim despite uncertainty. Otherwise, the author still retains a weaker claim. The second frame, New-view-on-existing-result, highlights that the conclusion will not be a surprise. If the reader agrees with the author, then the lack of novelty has at least been acknowledged. If the reader disagrees, then the author has strengthened their claim by making the reference to existing work. 3. Multiple Disciplinary Perspectives on Framing Use of the frames introduced in the previous section should be understood within a broader context. A range of factors inside and outside the analysis influence the way an author frames uncertainty in a scientific publication. These factors have been discussed in various fields of study, and here we briefly introduce some of the ideas from these other fields that have shaped our approach to uncertainty framing. Specifically, we consider frame selection as a part of the activities of building a scientific argument, writing in a disciplinary specific way, and negotiating claims within the social context of the disciplinary community. 3.1. Building a Scientific Argument The presentation of uncertain findings can be viewed as a form of argumentation, in which the researcher refers to the methods they have used as part of the argument they are building. As described by Toulmin et al. [1979], arguments in general consist of a claim (conclusion) supported by selected grounds (premises), with the logic between them given by a warrant. The claim may be qualified, and the warrant itself may require justification as a method for drawing conclusions. Scientific arguments in particular have some characteristic features, including layers of ''stacked claims,'' with more complex claims building on observations or existing disciplinary knowledge as foundations [Latour, 1987; Kelly and Bazerman, 2003]. Attending to these structural features of arguments helped us to refine the set of uncertainty frames classified in this paper, and to recognize them within abstracts. Our analysis of each individual abstract included mapping of any visible argument structure (see Methods). Table 5. Uncertainty Frames Conveying Relatability of a Conclusion to the Reader, With Answers That Those Frames Provide, and Example Statements Name of frame Is the Conclusion Consistent With the Reader's Prior Knowledge? Example Statement Repeat-with-shifted-scope The reader may not accept a general statement, but the specific statement is believable There is enough water, at least in wet conditions New-view-on-existing-result The result will not be a surprise As expected from previous analyses, we see that there is enough water Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6750 Within the argument structure, the warrant may take a variety of forms [Macagno and Walton, 2015], with consequences for the status of the conclusion. Among warrants, key distinctions are made between types of reasoning (e.g., deductive, inductive) and between types of argument (e.g., from cause and effect, comparison, expert opinion etc.) [Macagno and Walton, 2015]. Outside deductive proof, there is a need to acknowledge the possibility of mistaken inferences (defeasible reasoning) and that new information may change the conclusion (nonmonotonic reasoning) [Koons, 2014]. Assessing the strength of an argument leads to evaluation of the weight of evidence, and whether it is sufficient for a particular purpose [Parker, 2009; Haasnoot et al., 2014]. We find that these features of argument warrants are reflected in the uncertainty frames chosen by authors. 3.2. Writing Within One's Discipline In academic writing, the form of an argument is also shaped by a disciplinary genre [Hyland, 2008]. Some parts of the argument can be taken for granted due to shared understanding with the reader [Gilbert, 1976]. Authors endeavor to make their argument more acceptable by countering potential attacks and identifying possible extensions for or against the position, giving their own argument the last word [Dung, 1995]. Key skills include expressing one's evaluation of a statement (conveying stance) and using various approaches for engagement with the reader [Hyland, 2005]. In particular, epistemic stance conveys the author's commitment to the reliability of their claims, notably with boosters (expressing certainty) and hedges (withholding commitment). These same linguistic features can, however, be used for other purposes, for example, hedges are used to convey politeness, such that it becomes necessary to look for the function of the hedge rather than taking it at face value [Crompton, 1997; Kaltenb€ock et al., 2010]. In the end, arguments are presented in a naturally flowing text that carries a large amount of information. Multiple implicit claims are often found within a single statement. From a linguistic point of view, identifying the purpose of specific textual features requires investigating the role of such features in a given discourse community [Askehave and Swales, 2001]. It is often difficult to make explicit the tacit understanding that comes naturally to authors and readers immersed in a given research community. Rather than teaching hard and fast rules, pedagogic approaches to teaching academic writing tend to emphasize strengthening the ability to reflect on one's expectations of the reader and discourse community, [Street, 2009]. We believe the same applies to uncertainty framing, and that the proposed typology of uncertainty frames can support such reflection. 3.3. Negotiating Claims in a Research Community Science is also a social activity and, throughout the research process, one's work is influenced by the feedback received from colleagues and reviewers (and one's prediction of their responses). The reader will similarly be influenced by the discourse community, but ultimately form their own subjective interpretation [Gilbert, 1976; Latour and Woolgar, 1986]. Of particular importance to framing is the review process, where the tension between demonstrating novelty and connecting to existing knowledge-as well as a need to address expectations specific to the journal-results in a negotiation with the editor and reviewer. The author is led to go down a hierarchy of claims, asking ''what level of claim can I persuade this audience to accept?'' [Myers, 1985], sometimes reframing the claim to make stronger statements depending on the venue, or even for different contexts within the same text. Finally, the uncertainty frames in the article are interpreted one last time when a reader evaluates the evidence on an issue. For example, an intervention can be judged according to whether it is ''efficacious'' in its original study, ''effective'' for the specific context, and ''ready for adoption'' [Flay et al., 2005]. These judgements might be informed by uncertainty framing, respectively, related to level of belief and depth of analysis, scope, and maturity and utility. 4. Methods Complex factors drive how frames are used, communicated, and interpreted. Our analysis focusses on the specific task of identifying the uncertainty frames in current use. This was an iterative process in which preconceptions about current framing practice were confronted with feedback from colleagues, the experience of analyzing abstracts, and the results of a literature review, leading to changes in the number of frames as well as their names and descriptions, and to clarification of their classification. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6751 An initial set of 14 frames was proposed in a conference paper at MODSIM2015, the International Congress on Modelling and Simulation, which traditionally has a large water resources presence [Guillaume et al., 2015]. The frames described were those that the authors recognized as being commonly used, based on their own experience as authors, readers and referees, rather than building on any preexisting theory or typology developed within our discipline or elsewhere. Each frame's description covered the authors' interpretation of what the frame involved, in what context it is useable, why it works in that context, and how it could be implemented. The conference paper provided opportunity for feedback from colleagues. Analysis of the abstracts in WRR involved identifying the claims made in the abstract as well as surrounding qualifiers and the structure of the argument supporting each claim. Using a standard approach from qualitative data analysis, these textual features were explicitly ''coded'' using the web-based brat tool for text annotation [Stenetorp et al., 2012]. That is, the abstract text was accessed within a web app that allowed spans of text to be highlighted and classified according to the type of feature. One or more frames were then assigned to each claim based on the textual features connected to that claim, interpreted with the aid of contextual information and background disciplinary knowledge. Only novel claims were considered, i.e., claims arising from results, not statements motivating the need for analysis or declaring the intention of the paper. While we expected the same frames to be used in the latter as the former, the frequency of the frames in each case was expected to differ significantly, and we are primarily interested in how authors portray their completed work rather than intentions. Consistent with the discussion in section 3, more than one frame might be used on a single claim, separate claims within the same abstract might be framed differently, and different readers might perceive different frames in the same text. Consequently, assignment of uncertainty frames based on textual features could not in general be performed automatically, but required the coder's discretion, making use of their disciplinary knowledge. To handle the inherent subjectivity of the classification process, each abstract was coded independently by at least two of this paper's authors/coders (sometimes three), followed by a validation process involving comparison and discussion [Potter and Levine-Donnerstein, 1999]. The coders were encouraged to resolve their differences, but in a few cases agreed to disagree about whether a frame was in use due to the ambiguity of the language in the abstract. Our statistical analyses track the presence of frames abstract-byabstract rather than sentence-by-sentence, recognizing that uncertainty framing sometimes happens between the lines or in the relationship between multiple statements. Intercoder reliability was calculated as the proportion of all frame assignments in which the second coder agreed that the frame in question was present in the abstract. The overall postvalidation agreement was 98%. Note that this calculation does not include the number of cases where coders agreed that a frame was not present, due to the very high number of such cases. A random sample of abstracts was coded from a set downloaded from ISI Web of Knowledge and recorded as published in WRR in 2015. This set included 433 records, from which we considered only Research Articles, Technical Notes, and Technical Reports, and excluded articles that appeared to be reviews, commentaries, or editorials. Preliminary analysis suggested that the type of claims made in those latter articles were significantly different from those of research articles, such that the framing practices would benefit from dedicated analysis. Results from an initial training phase with five abstracts were not used. Coding then proceeded in four batches of increasing size, of 10, 29, 52, and 86 randomly selected abstracts, for a total of 177 abstracts. (The lack of round numbers is due to removal of abstracts that were identified as reviews and commentaries only during the coding process). No prior information was available about how framing might differ across abstracts, so a random sample was deemed to be representative (the effect of heterogeneity was also partially evaluated by quantifying uncertainty in our results). Using several batches kept the number of abstracts in each batch manageable for comparison and discussion within the validation process, and provided an opportunity for reflection on the definition of frames and the coding process, in some cases requiring revision of previous batches. We then calculated the proportion of abstracts in which each frame was used, and ranked the frames according to this frequency of occurrence. The estimated frequency of occurrence depends on the sample, and to a much lesser extent the coder (despite the 98% agreement, as reported above). We therefore estimated uncertainty in each frame's frequency by drawing 1000 bootstrap samples, where each sample consisted of 177 coded abstracts drawn, with replacement, from the full set of coded abstracts. A sampled abstract therefore consisted of the frames assigned by any coder to any of the 177 abstracts. Frequency of frames Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6752 was evaluated for each sample, providing a distribution of frequencies rather than a single estimate. Similarly, the ranking of frames by frequency was performed pair-wise on bootstrapped samples. For each pair of frames, we determined which frame occurred more frequently in each sample. The ranking was considered definite if it occurred in more than 95% of samples, i.e., it was significant at a 5% confidence level. As expected, the estimated uncertainty in frequency of frames decreased as sample size increased across batches, and we consider the current sample size an adequate snapshot of uncertainty framing in abstracts of WRR research articles in 2015. We expect that the use of frames is likely to vary over time and between sections of an article, such that results should not be generalized beyond this scope. In summary, uncertainty in this analysis has been addressed with the following approaches, each of which justifies the use of specific frames in presenting our conclusions: 1. The typology and set of frames was iteratively developed and could evolve further in future (Steptowards-a-goal) 2. Assignment of frames to claims was verified through discussion between coders (Triangulation) 3. The effect of sample selection on frequency of frames was quantified through bootstrap resampling (Uncertainty quantification) 4. Ranking of frames assumed a 5% confidence level (Validate-and-defend), for the purpose of evaluating how different frequencies of frames were relative to each other 5. The frequency of frames is not considered generalizable beyond abstracts of WRR in 2015 (Restrict-scopeof-applicability), but provides a firm basis for debate about best practices related to framing (Build-thefoundations) 6. The underlying approach of textual analysis builds on accepted qualitative data analysis techniques (Follow-the-contract). The design of the overall analysis is considered consistent with the ideas of ''systemic functional linguistics,'' which aims to provide a guide to action by investigating how language-related functions (like framing of uncertainty) are achieved in diverse contexts as part of a social system (e.g., water resources research and management) [Fang, 2005; Coffin and Donohue, 2012] 7. Hypotheses are offered regarding use of the frames (Suggest-a-new-research-agenda) 8. Recommendations for action are given as the authors' professional opinion (Take-it-or-leave-it) 5. Results and Interpretation: Frame Frequency, Implementation, and Possible Explanations The uncertainty frames identified through our analysis were summarized in the typology presented in section 2. The following results provide a means to (1) evaluate the relative importance of these frames within abstracts, (2) provide a preliminary validation of the frames and (3) identify hypotheses that may explain current use of uncertainty frames, which could be further investigated. To address the first objective, section 5.1 presents results regarding the frequency of frames in the sampled abstracts in WRR in 2015. The second and third objectives are addressed in sections 5.2–5.6 by describing how we have seen each frame implemented in abstracts for each category, and by suggesting some hypothetical explanations for why the frames are used as they are. Within this discussion, references to rare or common occurrence of frames should be read as the author's personal, informal interpretation of the results. 5.1. Frequency of Occurrence of Frames Figure 2 shows the percentage of abstracts in which each frame is used, and ranks the frames by frequency of occurrence. Note that both the estimate of frequency and the ranking are uncertain due to sampling. Boxplots of frequency are shown, as well as a partial ranking of frames based on pair-wise statistical comparisons (as described in section 4). (Because of correlation in occurrence of frames, some frames are significantly more frequent than others even if their estimated frequencies substantially overlap). The frame Expressed-as-Fact stands out as occurring on average in 90% of abstracts. In other words, nearly all abstracts include at least one claim for which uncertainty is not explicitly acknowledged. Validate-andDefend is the second most frequent, occurring in 38% of abstracts; authors commonly tell the reader that there is sufficient evidence for their claim. There are then roughly two tiers of frames for which the ranking is uncertain. The first tier includes frames focusing on scope (Restrict-scope-of-applicability and Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6753 Make-it-relative) and maturity and utility (Step-towards-a-goal and Build-the-foundations). The four frames focused on depth of analysis are not uncommon, but less frequent (on average, 9–19% of abstracts). In the second tier, some frames indicate there is much uncertainty (e.g., Learn-from-my-problems) or make a weak claim (e.g., Take-it-or-leave-it, Make-it-hypothetical). It also includes frames that intuitively would be less likely to appear in an abstract, namely Degree-of-belief and Follow-the-contract (which was not found at all). One would assume that expressions of likelihood and justifications of choices using existing work are more likely to be found in the body of an article, but are often not significant enough to mention in the abstract. These results paint a picture of a diverse body of research which emphasizes scientific progress in welldefined steps. Weaker claims are allowed, as long as they are appropriately framed and make a contribution to the field. The frames most commonly found in the abstract are not generally associated with uncertainty analysis, but do seem typical of careful scientific endeavor. Beyond the frequencies of use across abstracts, our analysis also addresses the cooccurrence of frames within the same abstract. Most abstracts use more than one frame, either because they make multiple claims (possibly building on each other), or because they use multiple frames for a single claim. Figure 3a collects all the frames used in an abstract and reports the most frequent combinations (ordered by median and 75% percentile). The inset Figure 3b shows the number of frames used per abstract. On average, 7.9% of abstracts frame all their claims as Expressed-as-Fact. The next two most common cooccurrences combine Expressed-as-fact with Validate-and-Defend (on average 5%) and with Build-the-foundations (3%). The number of abstracts involved are, however, small, especially for the subsequent frames. There is a large diversity of combinations of frames, which is understandable given that on average 90% of abstracts use multiple frames, and roughly 20% use five or more. 0% 25% 50% 75% 100% Percentage of abstracts with frame Make−it−hypothetical New−view−on−existing−result Learn−from−my-problems Degree−of−belief Suggest−a−new−research−agenda Repeat−with−shifted−scope Uncertainty−quantification Triangulation Build−the−foundations Demonstrate−robustness Just−a−possibility Step−towards−a−goal Restrict−scope−of−applicability Validate−and−Defend Expressed−as-Fact Take−it−or−leave-it Make−it−relative Scope of a claim Depth of analysis Level of belief Maturity and utility Relatability Category of frames: 1 2 3-7 3-7 3-8 Rank 3-9 3-9 5-10 6-14 8-16 9-16 9-17 9-17 9-17 10-17 10-17 12-17 12-17 Range of possible ranks given uncertainty Vertical lines indicate frames that are statistically equally frequent: Bootstrapped uncertainty in frequency of occurrence of frames: Q25 1.5*Interquartile range Q50 Q75 Probability density function Figure 2. Frequency of occurrence of uncertainty frames and their ranking. Estimates of uncertainty in percentage of abstracts are obtained by bootstrap sampling (right hand side). Frequency is compared pair-wise among frames at a two-sided 5% confidence level, resulting in a partial ranking (left hand side). The frame Follow-the-contract was not found, and is not shown. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6754 This analysis does not allow for differentiating whether the frames used within an abstract are applied to a single claim or to several different ones. However, Figure 3a does highlight suggestive examples. The top three combinations necessarily occur on separate claims, as they are fundamentally incompatible. Observations might, for example, be framed as Expressed-as-Fact, while the interpretation of those observations is framed as Validate-and-Defend. Statements about a method or a transferrable finding might be framed as Expressed-as-Fact, but its utility framed as Build-the-foundations because of uncertainty in its generalization. The combination Expressed-as-Fact/Validate-and-Defend/Demonstrate-robustness/Restrict-scope-of-applicability was observed in seven coded abstracts, and in the bootstrap sample occurred on average in 2% of abstracts. It is of interest because it makes use of uncertainty frames from several categories. An anonymized, annotated example reads: ''(a) when heterogeneity is low to moderate, (b) the simpler method adequately captures expected solute transport behavior (c) despite the existence of heterogeneity at smaller scales.'' The sentence makes a statement using Restrict-scope-of-applicability (a), claiming that there is sufficient evidence to take a Validate-and-Defend attitude to the result (b), and that the result is not affected by a potential source of uncertainty, i.e., Demonstrate-robustness (c). Earlier in the abstract, a comparison between methods reported that one method was more accurate than another, Expressed-as-Fact. While each abstract used the four frames differently, this example illustrates how the four categories of frames fit together to convey to the reader the aspects of uncertainty that the author deems necessary. 5.2. Use of Maturity and Utility Frames Within abstracts, the frequency of frames conveying readiness for use of a conclusion (maturity and utility) appears to depend on the level of utility that they convey (Figure 2), with Expressed-as-Fact the most common, and Learn-from-my-problems the least common (see Figure 1a and Table 1 for definitions). Framing a statement as Expressed-as-Fact means that it is either deliberately framed as certain, or that the statement lacks any other indication of uncertainty, which may, respectively, be justified because the Scope of a claim Depth of analysis Level of belief Maturity and utility Relatability Build−the−foundations Restrict−scope−of−applicability Restrict−scope−of−applicability Validate−and−Defend Validate−and−Defend Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Validate−and−Defend Validate−and−Defend Expressed-as-Fact Expressed-as-Fact Expressed-as-Fact Demonstrate−robustness Demonstrate−robustness Just−a−possibility Just−a−possibility Step−towards−a−goal Repeat−with−shifted−scope Category of frames: Other Number (and percentage) of abstracts with this combination of frames )%03.11(02)%56.5(01)%00.0(0 30 (16.95%) b) 0% 10% 20% 30% 1 2 3 4 5 6 7 8 % a bs tra ct s Number of frames in abstract Frequency of occurrence of number of frames: Bootstrapped uncertainty in frequency of occurrence of combinations of frames: Q25 1.5*Interquartile range Q50 Q75 Probability density function Q25 1.5*Interquartile range Q50 Q75 Figure 3. Frequency of occurrence of combinations of frames, (a) top-10 most frequent unique combinations, inset (b) number of frames used per abstract. Estimates of uncertainty are obtained by bootstrap sampling (n 5 177 for 1000 repeated samples). Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6755 conclusion is beyond doubt (e.g., deductive reasoning), or because the author considers that the uncertainty is not worth discussing in that particular venue (in this case an abstract). It can take the form of adjectives indicating certainty (''exactly''), so-called ''factive reporting verbs'' (''show,'' ''demonstrate'') [Crompton, 1997], or more subtly in the use of present tense without qualifiers, or past tense in reporting observations. Its frequency of use is unsurprising in that an abstract cannot do justice to the uncertainty in every statement. Especially when observations using a standard ''black-box'' technique play a small supporting role to another claim, it is natural that the uncertainty involved be neglected [Latour, 1987]. At the other extreme, it is nevertheless not unusual for abstracts to acknowledge inaccuracies and inadequacies of their methods, and report lack of statistical significance, with the frame Learn-from-my-problems (3% of abstracts). Reporting failure is permitted, even if not encouraged [Andreassian et al., 2010]. However, the frame is (at least marginally) less frequent than Suggest-a-new-research-agenda (4%), Steptowards-a-goal (25%), and Build-the-foundations (13%). In addition to reporting a problem, these frames, respectively, suggest a possible future solution, that the problem may still allow the conclusion to be used, and that the problem does not undermine its instrumental value. Suggest-a-new-research-agenda might either explicitly or implicitly suggest a hypothesis or possible future work. It is often subjective whether a statement of a problem implies a suggested solution. It is unsurprising that Step-towards-a-goal and Build-the-foundations are among the most common frames, given that they both provide a means to make a contribution to the body of knowledge even when substantial uncertainty is unresolved and unquantified. To place the work's contribution and problems in the context of a broader goal, Step-towards-a-goal often uses hedges, e.g., ''shows potential,'' ''promising,'' ''appears to.'' By framing the results as provisional, these terms emphasize that the uncertainty is expected to be reducible [Walker et al., 2003], in contrast to framing results as only possible, without committing to whether the uncertainty can be resolved. Build-the-foundations is particularly common for methodological and data-focused papers. The frame allows the potential usefulness of a method or a new data set to be emphasized even if it has not been tested, other factors may be at play, it is still too incomplete to be used in practice, or that more could have been done to argue the case. 5.3. Use of Scope Frames Existing literature has highlighted that a generally applicable, externally valid statement requires a greater number of supporting assumptions [Pinch, 1985]. Therefore, by defining the applicability of their claim, an author can anticipate criticism by limiting what counterarguments are considered legitimate. In principle, an author already does this merely by selecting the focus of their paper. For the purposes of our analyses, we considered only those cases where the author has further restricted the scope of a statement, beyond the initial focus of the paper. Nevertheless, Restrict-scope-of-applicability and Make-it-relative were among the most common frames (25% of abstracts, Figure 2. See Figure 1b and Table 2 for definitions). Restrict-scope-of-applicability involves identifying exceptions to a statement (e.g., ''x unless a''), specifying circumstances in which the statement applies (e.g., ''x when a''), or making separate statements for different circumstances (e.g., ''if a, then x; if b, then y''). It also includes hedges like ''typically'' and ''generally,'' which imply that an exception might exist, without explicitly naming it. Make-it-relative makes use of the phenomenon that comparisons are often more accurate than absolute statements [e.g., Reichert and Borsuk, 2005], in order to avoid the uncertainty in the latter. The comparison may emphasize similarity (''consistent with''), improvement (''more representative,'' ''outperform''), or worsening (''less accurate than''), respectively, avoiding direct comparison with observations, and judgement of whether performance is actually sufficient, or inadequate. Such focus on improvement has been criticized in the context of decision making [Simon, 1956], climate modeling [Risbey and O'Kane, 2011], and sustainability [Princen, 2003], where it is important to know what is ''good enough.'' In the context of scientific progression, it does make sense to clarify whether a previous effort is equivalent, has been superseded, or remains superior to the new contribution. The final frame in this category, Make-it-hypothetical, is much less common (2%). Its utility is in allowing claims to be made about consequences of an uncertain assumption, enabling some planning before all the facts are known. It does, however, make a weaker claim in the sense that the conclusion is dependent on assumptions that may not, the author acknowledges, match reality. Examples include hypothetical case Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6756 studies used for testing (''synthetic example'') or method development (''reference model''). Authors may also emphasize that results depend on the specific methods used (''The reconstructions indicate that. . .''). Interestingly, conditionality on assumptions also underpins the idea of exploratory modeling [Bankes, 1993; Lempert, 2002], though in practice most studies do at least assume that their scenarios or model structure are plausible in order to draw conclusions about vulnerability or robust decision making. The infrequent use of this frame perhaps results from it being tied to the use of specific research methods which appear to be less common in this community (at least in 2015). 5.4. Use of Level of Belief Frames Frames conveying level of belief are relatively uncommon in the abstracts studied, with the exception of Validate-and-Defend (Figure 2. See Figure 1c and Table 3 for definitions). Given that it communicates that the author is sufficiently confident that the reader should accept a conclusion, it makes sense for Validateand-Defend to appear in the abstract, whereas other frames would perhaps be more likely to be used in the body of an article. In abstracts, Validate-and-Defend might be stated explicitly (''adequately captured,'' ''good results,'' ''successfully modelled'') or indirectly (''can be reproduced''), which may be relatively subtle (''able to,'' ''allow,'' ''can''). We also include implicit use of the frame when a result is deemed statistically significant without indicating the confidence level. The widespread use of this frame (30%) may result from the need for scientific articles to emphasize their success in order to strengthen their claim of constituting a novel contribution. Degree-of-belief (5% of abstracts) would be expected to be more common in the results section of articles, when statistical techniques are applied and in presenting intermediate evidence in support a conclusion. In statistical tests, this frame allows the reader to judge the level of risk involved, either as p-values (e.g., p 5 0.003), confidence levels (e.g., p< 0.001), or confidence or credible intervals (e.g., 95%). In some cases, performance measures can be reported with a similar consequence-allowing the reader to decide whether the level of support is adequate. The frame Take-it-or-leave-it is unsurprisingly very uncommon in the abstracts studied (2%), but is used, for example, when the author wishes to express their own professional opinion (''We believe'') or make a value judgement (''x is desirable''). It makes a weak claim by avoiding imposing one's conclusion on the reader. Follow-the-contract was not identified in the abstracts studied but experience suggests this frame is common in the body of articles. It is common practice to follow a generally accepted method with the expectation that most readers would be expected to accept the conclusion. While such methods are commonly mentioned in the abstracts, it is not generally necessary to defend their use. An informed reader will recognize the method, and an uninformed reader is not in a position to question its use. Both will expect to see justification of appropriateness of the method within the article in any case. 5.5. Use of Depth of Analysis Frames Frames conveying depth of analysis are less common than those related to scope, and are not ordered according to their continuum, unlike those related to maturity. The most common frames are Just-a-possibility and Demonstrate-robustness (20% of abstracts, Figure 2), while Triangulation and Uncertainty-quantification are less common (10% and 5%. See Figure 1d and Table 4 for definitions). Just-a-possibility is presumably common because it allows authors to propose ideas that are insufficiently supported, but nevertheless important. This is commonly conveyed using hedges like ''can'' and ''may.'' Providing an ''estimate'' also implies that other values would also be possible. Some uses of this frame exemplify the stereotypical academic hedge-the expert does not wish to commit to an answer, recognizing that results are always uncertain. From this perspective, it is perhaps surprising that it is not more common, perhaps reflecting both the need for authors to assert the contribution of their work, and the availability of uncertainty frames from other categories that similarly avoid commitment. Conversely, Demonstrate-robustness is common presumably because it provides a strong argument in support of a claim, while acknowledging that a level of uncertainty still exists that does not influence the conclusion in question. The frame can be tied to specific methods, including robust decision making and statistical hypothesis testing. The robustness associated with statistical significance is, however, not Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6757 definitive due to the use of distributions with infinite support. Demonstrate-robustness is therefore often used in conjunction with two other frames from the category level of belief (section 5.4). (Specifically, statistical significance is additionally framed by a confidence level or a p-value (Degree-of-belief), and absence of that framing implies that the author has deemed the confidence level sufficient (Validate-and-Defend).) Demonstrate-robustness can also be exploited less formally, for example with clauses expressing insensitivity (''x, even when a and b,'' ''x, regardless of a,'' ''x for all a and b tested''), or using follow-up statements (''We found x. We further tested a and b and still found x''). The idea that a conclusion is supported by multiple lines of evidence is easy to communicate and presumably important to highlight in an abstract, when applicable. The frequency of the frame Triangulation therefore probably reflects actual use of multiple methods or multiple data sets leading to a conclusion, or the use of previously published results and theory to add support to the primary method of the paper. Its occurrence in 10% of abstracts may therefore be surprisingly high, given that Triangulation is more commonly associated with the social sciences [Jick, 1979]. Uncertainty-quantification is the least common of the level of uncertainty frames (on average 4%). We saw it implemented by specifying ranges, confidence intervals, standard deviations or standard errors, (approximate) one-sided bounds (''at least,'' ''from x to more than y'') and orders of magnitude (''on the order of,'' ''between a few months and 30 years''). Uncertainty-quantification may be more likely to be seen in a results section given the limited space in an abstract, and the need for further interpretation of nonunique solutions, whether by the reader or author. There are studies, however, which argue that methods for quantification of uncertainty need to be more widely adopted [Pappenberger and Beven, 2006]. 5.6. Use of Relatability Frames Frames that anticipate a reader's reaction to a claim (relatability) are less common, but still noteworthy (see Figure 1e and Table 5 for definitions). Repeat-with-shifted-scope is not rare (7%), but does not appear to be mainstream. It takes the form of coupled claims with differing scope (e.g., ''x, especially if a''). An example would be: ''Fishers prefer streams with abundant fish, especially trout and salmon.'' The use of multiple scopes can allow an author to outline how an argument would change if the reader rejects some of its premises. Beyond this aim of defending the contribution of an article, the effect appears to be that the strongly and weakly supported claim mutually support each other, because the strong seems more plausible in comparison with the weak, and the weak seems more plausible if the strong is accepted. More broadly, this frame's use of two scopes suggests the possibility of other compound frames that this analysis may have missed, combining the preceding basic frames. The frame New-view-on-existing-result emphasizes that the conclusion is actually not new, and has already been accepted by the discipline (''reinforcing that. . .,'' ''as has previously been noted . . .''). Uncertainty has therefore already been addressed and the conclusion is invoked to support the argument, rather than the other way around. Given that it downplays novelty and relies on preexisting work, this frame is understandably rare (3%). It is, however, of interest for its ability to draw attention to method, and hence to indirectly support other claims arising from that method. There may be other approaches that similarly frame uncertainty by signaling the use of a particular type of argument. 6. Discussion This study represents a first major attempt at describing a typology of uncertainty frames, applicable across general scientific reporting. Our study has both descriptive and prescriptive aspects. As a descriptive exercise, analysis of frames that are detectable by a reader in a sample of abstracts in WRR provided the evidence required both to generate and to qualitatively verify the typology. The analysis supports the argument that our uncertainty framing typology covers a set of frames that are in routine use (at least in abstracts), and that are employed in ways consistent with both existing literature and how authors might be expected to behave (i.e., we have been able to propose plausible hypotheses for their use). In its prescriptive role, the typology provides a vocabulary and framework to help select and interpret uncertainty frames, and evaluate their use, while the analysis provides a snapshot of the status quo to enable its critique. As a first attempt at this critique, we propose a set of general recommendations on how to approach Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6758 the management and framing of uncertainty, as well as some provocative-specific recommendations about how current practice in framing uncertainty should change. 6.1. General Recommendations for Managing and Framing Uncertainty 1. Pay attention to framing: The continuing quality and efficiency of research depends on authors, reviewers, and readers being sensitive to the varied status of a conclusion, and providing appropriate feedback. It is therefore beneficial to encourage awareness of uncertainty framing in academic education and reviewing. Do the frames employed in a text appropriately communicate relevant uncertainties? 2. Cast a wide net: The broad range of uncertainty frames implies a broad range of approaches for managing uncertainty inside and outside the analysis, of which a user should be aware. The most common approaches (e.g., Validate-and-defend, Restrict-scope-of-applicability, Make-it-relative, Step-towards-a-goal) are not traditionally associated with uncertainty analysis, and may be equally (or even more) valid to invoke. 3. Focus on fitness for purpose: Use the right tool for the job. Avoid advocating the use of a single approach, but rather think critically about what method and frame is suitable for the specific circumstance. Existing practice shows that there is value in allowing weakly supported, but important, claims to be made, as long as they are visibly marked as such (e.g., using Suggest-a-new-research-agenda, Just-apossibility, Degree-of-belief, Make-it-hypothetical, Take-it-or-leave-it). 4. Announce uncertainty even in abstracts: When it comes to uncertainty, the attitude should be that even the abstract matters-it is not just about what method was used to address uncertainty, but also how it is presented. Abstracts should be structured and language used deliberately, avoiding ambiguity as far as possible. 5. Interpret uncertainty frames in context: Indications of uncertainty should be interpreted within the broader disciplinary and cultural context. Research articles cannot in general be treated as a selfcontained structured argument. Novices in particular should avoid being overly-sensitive to subtle uncertainty signals such as the use of single words like ''may'' or ''can.'' These subtle signals can be misinterpreted if the reader does not share the author's context. 6.2. Specific Recommendations About How Uncertainty Should be Framed Our specific recommendations are intended to be provocative. Our results suggest that the literature in WRR is roughly in line with one view of a typical, healthy scientific debate, involving what one might call carefully considered incremental science. Thus uncertainty is avoided where permitted, but otherwise commonly dealt with by limiting scope, deferring to further work, and indicating that evidence is sufficient. But perhaps some changes would still be beneficial? There is apparent potential for a more influential, dynamic form of science within a sophisticated community that is sensitive to the use of uncertainty frames, and whose researchers are capable of making judgments given the necessary information. Our recommendations reflect the authors' personal opinions, and we encourage others in the community to respond, with the aim of eventually crystallizing best practices for uncertainty framing. 1. Highlight uncertainty, don't hide it. Our analysis indicates that 8% of abstracts are only framed as Expressed-as-Fact. While it is normal to ignore uncertainty in some claims, there is nearly always some statement in which uncertainty should actually be highlighted to the reader. Our typology shows there is a broad range of ways in which to do that. 2. Think twice about Validate-and-Defend. Our analysis shows that 38% of abstracts make a claim that the evidence is sufficient. Would it not be better to list key performance indicators (Degree-of-belief) and let the reader decide whether performance was indeed ''good'' or ''adequate''? If a judgement of adequacy is really needed from the author, then at the very least say for what purpose. 3. Don't hide behind narrow scope. When possible, look for generalized conclusions and relate work to ambitious goals. Aim for influential claims that are likely to prompt useful debate. Reviewers likewise should accommodate this boldness, while verifying that uncertainty in wider-scope conclusions is appropriately acknowledged using suitable frames. 4. Aim for actionable conclusions. A passing reference to future work is too often a mask for poorly thought out implications. If appropriately qualified, even speculative suggestions of future uses add value over a vague promise that work might be useful later. Basic science benefits from concrete speculations about implications for practice. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6759 5. Test before publication. Methods are often framed as a building block that still needs further testing before being applied. The reader would be better informed if the abstract instead emphasized the testing that has already been performed. In this case, defining a narrow scope in which the method is known to work helps to avoid vague promises. 6. Think twice about Just-a-possibility. It is rather limiting for instance to know only that a result is just one estimate, or that alternative conclusions are also possible, yet this is a commonly used frame. A reader can do much more if they are additionally provided with a Degree-of-belief or Uncertainty-Quantification. Clearly acknowledged, simple approaches can provide added value with little extra cost, for example, using screening techniques that perform preliminary uncertainty analysis. 7. Define limits of uncertainty analysis, but assume it is incomplete. To avoid giving a misleadingly reassuring picture, Demonstrate-Robustness and Uncertainty-Quantification frames should adequately address crucial uncertainties, and clearly state what uncertainties are addressed. However, the default interpretation should still be that some source of uncertainty may still have been missed. 8. Avoid Repeat-with-shifted-scope. There is too great a risk that this frame acts as a throw-away line to convince the reader. When a stronger, less certain conclusion is worth making, clearly separate it from the weaker conclusion, and let it stand on its own merits. 9. Do publish speculation. A field is invigorated by new claims that might encourage new ways of tackling problems. If the claim is important enough, consider the frames: Suggest-a-new-research-agenda, Learnfrom-my-problems, New-view-on-existing-result, Make-it-hypothetical, Take-it-or-leave-it. 7. Conclusions This study has provided a first analysis of uncertainty framing practices, motivated by the desire to reflect on the treatment and framing of uncertainty in research publications in the field of water resources, and on improving understanding of how uncertainty is addressed outside the analysis, not just within. Specifically, we provided a brief overview of the context of uncertainty framing, introduced a specific set of frames and analyzed their frequency in a representative sample of abstracts in the journal Water Resources Research for 2015. Hypotheses were posited explaining their frequency of use. As might be expected, most abstracts include claims that are Expressed-as-Fact (i.e., as if there is no doubt about the statements made), and the most frequent frames conveying uncertainty are those associated with incremental progress on a topic with the contributions being carefully defined. Consistent with efforts to promote better treatment and communication of uncertainty, we proposed a set of provocative recommendations for changes to the status quo and ask the community to consider whether current uncertainty framing is meeting their expectations. At the same time, our general recommendations highlight the diversity of uncertainty frames and recognize that any of those frames can be valid in the appropriate context. The ultimate aim would be to achieve consensus among water resources researchers regarding best practices for uncertainty framing, ensuring that frames are indeed used in appropriate contexts. Our analysis is, however, only a beginning. There is potential for future work to explore how framing of uncertainty can be more effectively taught; to investigate the use of frames at a finer level by formal argument mapping linked with contextual variables; and to collaborate with the social sciences to better understand what authors actually mean when they use frames in a particular context and what readers actually understand. Deploying an uncertainty frame is a crucial move in establishing a conclusion's maturity and utility, scope, level of belief, depth of analysis, and relatability. As with other types of rhetorical move associated with academic writing, ''by recognizing the power of different moves in different contexts, you can then mobilize that power'' [Bazerman, 1988]. Mobilizing the power of scientific understanding will play a crucial role in the future of our planet, particularly in environmental issues, including water resources management. References Andreassian, V., C. Perrin, E. Parent, and A. Bardossy (2010), The court of miracles of hydrology: Can failure stories contribute to hydrological science?, Hydrol. Sci. J., 55(6), 849–856, doi:10.1080/02626667.2010.506050. Askehave, I., and J. Swales (2001), Genre identification and communicative purpose: A problem and a possible solution, Appl. Linguist., 22(2), 195–212, doi:10.1093/applin/22.2.195. Bankes, S. (1993), Exploratory modeling for policy analysis, Oper. Res., 41(3), 435–449, doi:10.2307/171847. Acknowledgments The authors would like to thank the three anonymous reviewers for their detailed comments, which lead to substantial changes to the presentation of our ideas. We are also grateful to all conference participants and colleagues who provided feedback on the uncertainty frames during their development. J. H. A. Guillaume was funded by the Academy of Finland project WASCO (grant 305471) and Emil Aaltonen Foundation funded project eat-lesswater. C. Helgeson received funding from the Arts and Humanities Research Council (AHRC) project Managing Severe Uncertainty (AH/ J006033/1) and L'Agence Nationale de la Recherche (ANR) project DUSUCA (ANR-14-CE29–0003-01). Data on presence of frames in abstracts are available as supporting information in CSV format with three columns: (1) abstract as a DOI identifier, e.g., ''2014WR015291'' corresponds to the abstract accessible at https://doi.org/ 10.1002/2014WR015291, (2) coder as a uniquely identifying numeric code (1–3), (3) frame-the name of an uncertainty frame applied to the abstract by the coder (N.B. usually more than one frame per abstract and coder). Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6760 Bazerman, C. (1988), Shaping written knowledge: The genre and activity of the experimental article in science, Univ. of Wis. Press, Madison. Beck, M. B. (1987), Water quality modeling: A review of the analysis of uncertainty, Water Resour. Res., 23(8), 1393–1442, doi:10.1029/ WR023i008p01393. Bennett, N. D. et al. (2013), Characterising performance of environmental models, Environ. Model. Software, 40, 1–20, doi:10.1016/ j.envsoft.2012.09.011. Beven, K., and P. Young (2013), A guide to good practice in modeling semantics for authors and referees, Water Resour. Res., 49, 5092– 5098, doi:10.1002/wrcr.20393. Bradshaw, G. A., and J. G. Borchers (2000), Uncertainty as information: Narrowing the science-policy gap, Conserv. Ecol., 4(1), 7, doi:10.5751/ ES-00174-040107. Brown, J. D. (2004), Knowledge, uncertainty and physical geography: Towards the development of methodologies for questioning belief, Trans. Inst. Br. Geogr., 29(3), 367–381, doi:10.1111/j.0020-2754.2004.00342.x. Brugnach, M., A. Tagg, F. Keil, and W. J. de Lange (2007), Uncertainty matters: Computer models at the science–policy interface, Water Resour. Manage., 21(7), 1075–1090, doi:10.1007/s11269-006-9099-y. Brugnach, M., A. Dewulf, C. Pahl-Wostl, and T. Taillieu (2008), Toward a relational concept of uncertainty: About knowing too little, knowing too differently, and accepting not to know, Ecol. Soc., 13(2), 30. Brun, R., P. Reichert, and H. R. K€unsch (2001), Practical identifiability analysis of large environmental simulation models, Water Resour. Res., 37(4), 1015–1030, doi:10.1029/2000WR900350. Coffin, C., and J. P. Donohue (2012), Academic Literacies and systemic functional linguistics: How do they relate?, J. Engl. Acad. Purp., 11(1), 64–75, doi:10.1016/j.jeap.2011.11.004. Crompton, P. (1997), Hedging in academic writing: Some theoretical problems, Engl. Specif. Purp., 16(4), 271–287, doi:10.1016/S08894906(97)00007-0. Dooge, J. C. I. (1996), Walter Langbein and the emergence of scientific hydrology, Water Resour. Res., 32(10), 2969–2977, doi:10.1029/ 96WR00273. Duan, Q., H. V. Gupta, S. Sorooshian, A. N. Rousseau, and R. Turcotte (Eds.) (2003), Calibration of Watershed Models, Water Science and Application, AGU, Washington, D. C. Dung, P. M. (1995), On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games, Artif. Intell., 77(2), 321–357, doi:10.1016/0004-3702(94)00041-X. Fang, Z. (2005), Scientific literacy: A systemic functional linguistics perspective, Sci. Educ., 89(2), 335–347, doi:10.1002/sce.20050. Flay, B. R., A. Biglan, R. F. Boruch, F. G. Castro, D. Gottfredson, S. Kellam, E. K. Moscicki, S. Schinke, J. C. Valentine, and P. Ji (2005), Standards of evidence: Criteria for efficacy, effectiveness and dissemination., Prev. Sci., 6(3), 151–75. Gilbert, G. N. (1976), The transformation of research findings into scientific knowledge, Soc. Stud. Sci., 6(3/4), 281–306. Guillaume, J. H. A., S. Elsawah, A. J. Jakeman, and M. Kummu (2015), Communicating uncertainty: Design patterns for framing results in scientific publications, in MODSIM2015, 21st International Congress on Modelling and Simulation, edited by T. Weber, M. J. McPhee, and R. S. Anderssen, pp. 1972–1978, Modell. and Simul. Soc. of Aust. and N. Z., Gold Coast. Guillaume, J. H. A., R. J. Hunt, A. Comunian, R. S. Blakers, and B. Fu (2016), Methods for Exploring Uncertainty in Groundwater Management Predictions, in Integrated Groundwater Management, edited by A. J. Jakeman et al., pp. 711–737, Springer, Cham. Gupta, H. V., and G. S. Nearing (2014), Debates-the future of hydrological sciences: A (common) path forward? Using models and data to learn: A systems theoretic perspective on the future of hydrological science, Water Resour. Res., 50, 5351–5359, doi:10.1002/ 2013WR015096. Gupta, H. V., S. Sorooshian, and P. O. Yapo (1998), Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information, Water Resour. Res., 34(4), 751–763, doi:10.1029/97WR03495. Gupta, H. V., M. P. Clark, J. A. Vrugt, G. Abramowitz, and M. Ye (2012), Towards a comprehensive assessment of model structural adequacy, Water Resour. Res., 48, W08301, doi:10.1029/2011WR011044. Haasnoot, M., W. P. A. van Deursen, J. H. A. Guillaume, J. H. Kwakkel, E. van Beek, and H. Middelkoop (2014), Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways, Environ. Modell. Software, 60, 99–120, doi:10.1016/ j.envsoft.2014.05.020. Hill, M. C., and C. R. Tiedeman (2007), Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions, and Uncertainty, Wiley-Interscience, N. J. Hufschmidt, M. M. (1967), The role of universities in water resources education: The social sciences, Water Resour. Res., 3(1), 3–9, doi: 10.1029/WR003i001p00003. Hyland, K. (2005), Stance and engagement: A model of interaction in academic discourse, Discourse Stud., 7(2), doi:10.1177/1461445605050365. Hyland, K. (2008), Genre and academic writing in the disciplines, Lang. Teach., 41(4), doi:10.1017/S0261444808005235. Isendahl, N., A. Dewulf, M. Brugnach, G. François, S. M€ollenkamp, and C. Pahl-Wostl (2009), Assessing framing of uncertainties in water management practice, Water Resour. Manage., 23(15), 3191–3205, doi:10.1007/s11269-009-9429-y. Jick, T. D. (1979), Mixing qualitative and quantitative methods: Triangulation in action, Adm. Sci. Q., 24(4), 602–611, doi:10.2307/2392366. Kahneman, D. (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux, New York. Kaltenb€ock, G., W. Mihatsch, and S. Schneider (2010), New Approaches to Hedging, Emerald Group Publishing, Bingley, U. K. Kelly, G. J., and C. Bazerman (2003), How students argue scientific claims: A rhetorical-semantic analysis, Appl. Linguist., 24(1), 28–55, doi: 10.1093/applin/24.1.28. Klemes, V. (1986), Dilettantism in hydrology: Transition or destiny?, Water Resour. Res., 22(9S), 177S–188S, doi:10.1029/WR022i09Sp0177S. Koons, R. (2014), Defeasible reasoning, in The Stanford Encyclopedia of Philosophy, (Spring 2014 Edition), edited by E. N. Zalta. [Available at https://plato.stanford.edu/archives/spr2014/entries/reasoning-defeasible/.] Krauss, M. K. Von, and W. Walker (2006), Response to ''To what extent, and how, might uncertainty be defined'' by Norgon, Brown, and Mysiak, Integr. Assess. J., 6(1), 89–94. Latour, B. (1987), Science in Action: How to Follow Scientists and Engineers Through Society, Harvard Univ. Press, Cambridge, Mass. Latour, B., and S. Woolgar (1986), Laboratory Life: The Construction of Scientific Facts, Sage Publications, Beverly Hills, LA. Lempert, R. J. (2002), A new decision sciences for complex systems., Proc. Natl. Acad. Sci. U. S. A., 99, suppl. 3, 7309–7313, doi:10.1073/ pnas.082081699. Lipshitz, R., and O. Strauss (1997), Coping with uncertainty: A naturalistic decision-making analysis, Organ. Behav. Hum. Decis. Process., 69(2), 149–163, doi:10.1006/obhd.1997.2679. Macagno, F., and D. Walton (2015), Classifying the patterns of natural arguments, Philos. Rhetor., 48(1), 26–53, doi:10.1353/par.2015.0005. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY 6761 Maier, H. R., J. H. A. Guillaume, H. van Delden, G. A. Riddell, M. Haasnoot, and J. H. Kwakkel (2016), An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together?, Environ. Modell. Software, 81, 154–164, doi:10.1016/j.envsoft.2016.03.014. Matott, L. S., J. E. Babendreier, and S. T. Purucker (2009), Evaluating uncertainty in integrated environmental models: A review of concepts and tools, Water Resour. Res., 45, W06421, doi:10.1029/2008WR007301. McDonnell, J. J. (2016), The 1-hour workday, Science, 353(6300), doi:10.1126/science.353.6300.718. Morgan, M. G., M. Henrion, and M. Small (1990), Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press, Cambridge, U. K. Myers, G. (1985), Texts as knowledge claims: The social construction of two biology articles, Soc. Stud. Sci., 15(4), 593–630, doi:10.1177/ 030631285015004002. Nearing, G. S., and H. V. Gupta (2015), The quantity and quality of information in hydrologic models, Water Resour. Res., 51, 524–538, doi: 10.1002/2014WR015895. Norton, J. P., J. D. Brown, and J. Mysiak (2006), To what extent, and how, might uncertainty be defined? Comments engendered by ''Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support,'' Integr. Assess. J., 6(1), 83–88. [Available at http://journals.sfu.ca/int_assess/index.php/iaj/article/viewPDFInterstitial/9/195.] Pappenberger, F., and K. J. Beven (2006), Ignorance is bliss: Or seven reasons not to use uncertainty analysis, Water Resour. Res., 42, W05302, doi:10.1029/2005WR004820. Parker, W. S. (2009), II Confirmation and adequacy-for-purpose in climate modelling, Proc. Aristot. Soc. Suppl. Vol., 83(1), 233–249, doi: 10.1111/j.1467-8349.2009.00180.x. Pinch, T. (1985), Towards an Analysis of Scientific Observation: The Externality and Evidential Significance of Observational Reports in Physics, Soc. Stud. Sci., 15(1), 3–36, doi:10.1177/030631285015001001. Potter, W. J., and D. Levine-Donnerstein (1999), Rethinking validity and reliability in content analysis, J. Appl. Commun. Res., 27(3), 258–284, doi:10.1080/00909889909365539. Princen, T. (2003), Principles for sustainability: From cooperation and efficiency to sufficiency, Glob. Environ. Polit., 3(1), 33–50, doi:10.1162/ 152638003763336374. Refsgaard, J. C., J. P. van der Sluijs, A. L. Højberg, and P. A. Vanrolleghem (2007), Uncertainty in the environmental modelling process – A framework and guidance, Environ. Modell. Software, 22(11), 1543–1556, doi:10.1016/j.envsoft.2007.02.004. Regan, H. M., M. Colyvan, and M. A. Burgman (2002), A taxonomy and treatment of uncertainty for ecology and conservation biology, Ecol. Appl., 12(2), 618–628, doi:10.2307/3060967. Reichert, P., and M. E. Borsuk (2005), Does high forecast uncertainty preclude effective decision support?, Environ. Modell. Software, 20(8), 991–1001, doi:10.1016/j.envsoft.2004.10.005. Reichert, P., S. D. Langhans, J. Lienert, and N. Schuwirth (2015), The conceptual foundation of environmental decision support, J. Environ. Manage., 154, 316–332, doi:10.1016/j.jenvman.2015.01.053. Risbey, J. S., and T. J. O'Kane (2011), Sources of knowledge and ignorance in climate research, Clim. Change, 108(4), 755–773, doi:10.1007/ s10584-011-0186-6. Simon, H. A. (1956), Rational choice and the structure of the environment., Psychol. Rev., 63(2), 129–138, doi:10.1037/h0042769. Sivapalan, M. (2015), Debates-Perspectives on socio-hydrology: Changing water systems and the ''tyranny of small problems''-Sociohydrology, Water Resour. Res., 51, 4795–4805, doi:10.1002/2015WR017080. Stenetorp, P., S. Pyysalo, G. Topić, T. Ohta, S. Ananiadou, and J. Tsujii (2012), BRAT: A web-based tool for {NLP}-assisted text annotation, in Proceedings of the Demonstrations Session at {EACL} 2012, Assoc. for Comput. Linguist., Avignon, France. Street, B. (2009), ''Hidden'' Features of Academic Paper Writing, 24(1). [Available at http://repository.upenn.edu/wpel/vol24/iss1/1.] Toulmin, S., R. Rieke, and A. Janik (1979), An Introduction to Reasoning, Macmillan, New York. Vogel, R. M., U. Lall, X. Cai, B. Rajagopalan, P. K. Weiskel, R. P. Hooper, and N. C. Matalas (2015), Hydrology: The interdisciplinary science of water, Water Resour. Res., 51, 4409–4430, doi:10.1002/2015WR017049. Vrugt, J. A., W. Bouten, H. V. Gupta, and S. Sorooshian (2002), Toward improved identifiability of hydrologic model parameters: The information content of experimental data, Water Resour. Res., 38(12), 1312, doi:10.1029/2001WR001118. Vrugt, J. A., C. J. F. ter Braak, M. P. Clark, J. M. Hyman, and B. A. Robinson (2008), Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation, Water Resour. Res., 44, W00B09, doi:10.1029/2007WR006720. Wagener, T., and H. V. Gupta (2005), Model identification for hydrological forecasting under uncertainty, Stoch. Environ. Res. Risk Assess., 19(6), 378–387, doi:10.1007/s00477-005-0006-5. Walker, W. E., P. Harremo€es, J. Rotmans, J. P. van der Sluijs, M. B. A. van Asselt, P. Janssen, and M. P. Krayer von Krauss (2003), Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support, Integr. Assess., 4(1), 5–17, doi:10.1076/ iaij.4.1.5.16466. Walker, W. E., M. Haasnoot, and J. Kwakkel (2013), Adapt or perish: A review of planning approaches for adaptation under deep uncertainty, Sustainability, 5(3), 955–979, doi:10.3390/su5030955. Water Resources Research 10.1002/2017WR020609 GUILLAUME ET AL. FRAMING OF UNCERTAINTY