Expert Judgment for Climate Change Adaptation Erica Thompson, Roman Frigg, and Casey Helgeson London School of Economics and Political Science Forthcoming in Philosophy of Science Abstract. Climate change adaptation is largely a local matter, and adaptation planning can benefit from local climate change projections. Such projections are typically generated by accepting climate model outputs in a relatively uncritical way. We argue, based on the IPCC's treatment of model outputs from the CMIP5 ensemble, that this approach is unwarranted and that subjective expert judgment should play a central role in the provision of local climate change projections intended to support decision-making. 1. Introduction. It is now widely accepted that global warming is real and in large part due to human activities.1 But the big picture of increasing global average temperature reveals few specifics about how climate change will impact human lives and natural ecosystems at a local scale. Regional climates are changing, and will continue to change, in their own unique ways. Residents of one region may experience more summer droughts, while others on the same continent experience fewer. One city may face higher storm surges, another, more severe heat waves, a third, reduced monsoon rains. To some degree, we can adapt to these changes by aligning our behavior, institutions, and infrastructure with the new reality. Farmers can plant different crops; foresters can adjust their management plans; a water company can add a new reservoir; a city might strengthen its flood protection; centers for disease control can prepare for migrating vectors; city heath departments can build cooling centers for vulnerable residents; insurers (and reinsurers) can recalculate their prices. Adapting, however, requires insight into where the local climate is headed. Is it 1 The evidence is presented in the most recent IPCC report (Stocker et al. 2013); the consensus is documented in Oreskes (2007). 2 worth investing in a new irrigation system? That depends on whether recent dry conditions are the new normal or just a streak of bad luck. Longer-term plans that are expensive to change once implemented would better be informed by projections further into the future. Construction of Britain's new high-speed train infrastructure, for instance, is set to begin in 2017. We would like to design the tunnels to avoid overheating during late twenty first century summers, projected to be warmer and drier than today's and with more intense hot weather events (High Speed Two Limited, 2013). Several coastal sites in the UK are currently being evaluated for new build nuclear power stations; the choice of site, as well as the physical design and adaptive management plan are sensitive to long term projections of local sea level rise (Wilby et al. 2011). And similar questions arise, of course, in other parts of the world. Our topic is how climate science can best inform these local adaptation decisions. We begin by reviewing a standard approach to local climate change projections (Section 2) and considering some reasons for caution (Section 3). An alternative emerges from a sideward look at how scientific knowledge has been incorporated into decisions on issues such as dam safety and regulation of toxic substances. The key idea is to introduce a process of structured expert elicitation (Section 4). We discuss specific issues that may arise in implementing such a procedure for local climate change adaptation decisions (Section 5) and then draw some conclusions (Section 6). 2. Answers Offered. Science-based information and advice that aims to inform local adaptation decisions goes by the general name climate services. The goods on offer include data sets and climate forecasts (sometimes called products) as well as guidance on how to approach and think about adaptation planning (decision support). A variety of organizations currently develop and deliver climate services, including private consultancies, NGOs, universities, and government agencies.2 While climate services products vary in what exactly they offer and in how the offerings are presented, the methodology behind them is, in broad strokes, often the same: Start with output from general circulation models (GCMs), which simulate the whole climate system at a coarse resolution; then use one or more downscaling methods, which interpolate the GCM output to give finer details over the region of interest; 2 For a brief overview of the state of climate services in the United States, see the American Meteorological Society's (2015) policy statement on climate services. 3 finally, downscaled projections may be combined with local historical data. Methods diverge in their selection of GCMs and approach to downscaling, and in how they manage and communicate the uncertainties that pile up along the way. An example of such a product offered by a government agency is The United Kingdom Climate Impacts Program's UKCP09 project. The project offers a set of quantitative projections for local variables such as the change in temperature of the warmest summer day in London or the change in total precipitation during the wettest winter season in Tewkesbury.3 These projections, at the scale of a 25km grid, can be directly accessed on UKCP09's website in the form of probabilities for different timescales (2020s/2050s/2080s) and alternative emissions scenarios (high/medium/low). The projections are advertised as relevant for practical decision support, and "worked examples" show how they should be used to tackle concrete problems such as energy use and sustainability in school buildings, overheating risk for buildings and flood management policy (Met Office, 2009). Projects of this kind are not restricted to the UK. A larger project called weADAPT provides small-scale information based on the CMIP5 archive4 of GCM runs at particular station locations and using a statistical downscaling method. The weADAPT portal describes some in-depth case studies for use and interpretation of the data. For example, one case study looks at Bagamoyo, Tanzania, and offers an answer to the question: "What do current trends in temperatures, rainfall and mean sea level, and mediumand long-term projections, suggest are the most serious climate risks that government officials in Bagamoyo district need to address?" (Besa 2013). Cal-Adapt offers adaptation information for California, Climate Wizard for the entire USA, Climate Impacts Online for Germany, and the list could go on.5 An ever-increasing number of private consultancies also offer similar climate change products, usually based on publicly available data.6 3 "UKCP" stands for United Kingdom Climate Projections and "09" indicates that it was launched for public use in 2009. 4 CMIP5 is the fifth instantiation of the internationally coordinated Coupled Model Intercomparison Project, which compares the results of different GCMs. 5 For information see, respectively, California Energy Commission (2016), The Nature Conservancy (2009), and ClimateImpactsOnline (2012). 6 An example is JBA Consulting, North Yorkshire, UK. 4 In sum, questions about future local climate are, today, mostly answered by using state-of-the-art GCMs combined with downscaling. 3. Do the Answers Provide Actionable Information? Its popularity notwithstanding, this method has questionable credentials as a provider of actionable local information. This conclusion can be reached from at least three different directions. We comment only briefly on the first two: detailed analysis of individual projects, and recognition of actual predictive failures. Our main focus here is a third path to the same conclusion: making explicit the implications of the most recent report from the Intergovernmental Panel on Climate Change (IPCC). Each climate projection project is based on a set of methodological assumptions, and these can be subjected to scrutiny. Frigg, Smith and Stainforth (2013, 2015) analyze the foundational assumptions of UKCP09 and argue that the methods used do not warrant treating model outputs as decision relevant projections. Similar in-depth analyses could be carried out for other projects. Needless to say, we can only speculate about the outcomes of such an exercise, but given the arguments we present below we would not expect the verdicts on other projects to differ significantly. Regarding the success of predictions, Mearns (2014) reports telling results from a study evaluating models for the purpose of predicting the future regional climate of North America. Mearns and colleagues compared a number of variations on the methodology described above, in each instance combining one of four GCMs with one of six regional climate models (simulation-based downscaling methods). They evaluated each model pair by comparing its simulation of 1980-2005 with the weather data of that period. Two such model evaluation exercises focused on (1) the North American Monsoon in Southwest US and Northern Mexico, and (2) summer precipitation in the Northeast. A surprising result was that the model pair that performed best on (1) performed worst on (2); and few model pairs performed respectably on both. The result suggests that local climate projections are more sensitive to the choice of models than previously supposed. The problem is compounded by the difficulty of identifying any factor (or set of factors) in the models that explains the differences in performance. Furthermore, past success is not necessarily a guide to future success (Reifen and Toumi 2009), making it difficult to learn from a model's success in reproducing past data. This suggests that interpreting regional model outputs as decision relevant projections is more problematic than previously assumed. In particular, it raises 5 doubts about using any single model, or even the same ensemble or ensemble-weighting, for general-purpose national climate projections where multiple regions, variables and timescales are of interest. Our main argument, however, is based on the most recent IPCC report.7 The report includes global mean temperature projections for 2100 with a clear assessment of their reliability. They give, for instance, a projection of 2.6-4.8°C of global mean temperature rise relative to the late twentieth century, supposing the RCP8.5 scenario, which describes an illustrative "possible future" in which greenhouse gas emissions continue to rise unabated. The IPCC quantifies uncertainty in a projection using standardized language referring to likelihood intervals, for example, "virtually certain" (> 99%), "very likely" (> 90%), or "more likely than not" (> 50%) (Stocker et al. (2013), Box TS.1, Technical Summary). The 2.6-4.8°C projection is qualified as "likely," which means there is a 66-100% chance of the real-world outcome falling into that range, given the scenario. The range 2.6-4.8°C is arrived at by looking at the CMIP5 ensemble of climate models: a set of state-of-the-art models that have each run the RCP8.5 scenario and supplied a value for the projected temperature change at 2100. This set of single values is then used to fit the parameters (mean and standard deviation) of a Gaussian probability distribution. The range 2.6-4.8°C is the interval which symmetrically spans 90% of that distribution. In the "model-world," therefore, this range is an estimate of the interval into which model runs chosen from some underlying distribution will fall 90% of the time. In IPCC terminology this interval would be referred to as a "very likely" range. The IPCC authors' next step is to note that even state-of-the-art climate models share systematic biases (Knutti et al 2010). All models are limited by the same computational constraints; they are based on the same limited set of parameterizations; and they are calibrated to reproduce features of twentieth-century climate which may be less relevant for twenty-first century developments. For these reasons, when translating model-world findings into real-world projections, the IPCC authors downgrade the model-derived "very likely" uncertainty quantification for the range 2.6-4.8°C to "likely." Thus while there is a >90% probability that a state-of the-art model run will 7 Our discussion is based on Stocker et al. (2013), Sections 12.2.3 and 12.4.1, and Summary for Policymakers Table SPM.2 (see notes c and d). 6 result in a temperature change between 2.6-4.8 degrees, there is only a >66% chance- in the authors' judgment-that the real climate system will produce this outcome. So the IPCC judge that there is a up to an additional 24% probability that the actual climate outcome (for the RCP8.5 scenario) will be outside the central range of the models and that we experience something entirely different than what the models suggest. This is a clear indication that climate scientists do not believe the raw outputs of model ensembles to represent real-world probability distributions. The reassignment of probabilities described above is based on an informal process of expert judgment: the discussions and iterations between authors during the long writing and review stages of the report. We commend the forthrightness with which the IPCC acknowledges the role of expert judgment in their assessment process. At the same time, we recognize some drawbacks to the (relatively unstructured) way in which they go about it (details below). The main conclusion we draw from the example is that the IPCC's own results and methods contain the assertion that there is significant uncertainty about climate projections, and that this uncertainty is best quantified by appeal to expert opinion. This conclusion should not be limited to global mean temperature. Indeed, global mean temperature is the variable in which there is (rightly) the most confidence about climate models' ability to represent relevant processes.8 So for the local-scale variables that matter to adaptation planners, we would expect to see a probability downgrade of at least the same magnitude as the one given for global mean temperature. In effect the 24% represents an imprecise lower bound to our uncertainty about other variables.9 The conclusion we draw from this is that the case for quantifying uncertainty through expert judgment is even stronger when it comes to local-scale variables, and that uncertainty in local climate projections should be assessed through a suitably structured process of expert elicitation. This conclusion stands in stark contrast with 8 See for example Stocker et al. (2013), Technical Summary, page 81: "Model agreement and confidence in projections depends on the variable and on spatial and temporal averaging, with better agreement for larger scales. Confidence is higher for temperature than for those quantities related to the water cycle or atmospheric circulation." 9 In passing we note that levels of uncertainty are not expected to decrease significantly in the short and medium term, ever-increasing computational resources notwithstanding (Maslin and Austin 2012; Knutti and Sedlacek 2013). 7 current practice, where model results are often interpreted directly as a probability distribution of future real-world climate outcomes (this happens, for instance, in UKCP09). 4. Lateral Knowledge Transfer. The climate services sector faces a situation in which decisions call for scientific input, but there is severe uncertainty about the modeling on which scientists can rest their advice. Climate change adaptation, however, is not the only field that satisfies this description. Indeed the situation is comparable in many policy-relevant areas of science and engineering, for example toxicology, public health, and nuclear safety. To illustrate what has become a standard approach to the problem, we now have a look at a concrete case in environmental regulation. Our claim is that the example will illustrate important lessons for climate services. Zoning plans for sites that use or manufacture large volumes of toxic substances typically include a buffer zone in which no housing development can occur. How large should the buffer zone be? A political process can determine the acceptable level of risk for residents of neighboring developments, but what distance limits the risk to this level? The Ministry of Housing, Physical Planning and Environment of the Netherlands sponsored a study to help answer this question for a handful of common industrial chemicals including chlorine, ammonia and hydrogen chloride (Goossens et al. 1998). The key uncertainties concerned quantities for which there are no direct empirical observations: concentrations and exposure times that are fatal for humans. The standard approach to estimating those quantities models human toxicology using data from animal studies, treating the laboratory animal as an analogue model for humans and extrapolating concentrations and exposure times by adjusting for body weight, lung capacity, etc. This modeling exercise delivers answers, but how much can they be trusted? Humans are not 70kg rodents, and different species may react through different physiological pathways altogether. Indeed, there are competing explanations of some chemicals' toxicity in humans. Moreover, extrapolation results are sensitive to debatable scaling assumptions. And of course there are no data against which to test the resulting predictions (if there were, these would have already resolved the issue). Furthermore, a sole reliance on scale modeling fails to take advantage of other relevant sources of knowledge. For many chemicals, there are well-developed, qualitative toxicological models that describe metabolic processes, functional changes in the impacted organs, and clinical expression of these organ-function disturbances. 8 There may also be quantitative chemical and biological understanding of their "kinetics" within the body: rates of absorption, distribution, metabolism, and elimination. This knowledge is relevant to estimating the required concentrations and exposure times, and to judging the reliability of the extrapolations from animal studies, but there is no established quantitative method of feeding everything into a hopper and turning the crank for an all-things-considered best estimate. To integrate the available evidence into actionable findings, the experimenters recruited 27 toxicologists from industry, academia, and government (about five experts per chemical). The experts reviewed the available research and provided, through a carefully structured interview, their all-things-considered subjective judgments for the required concentrations and exposure times. The interview process, or elicitation, draws on a large body of research on the formatting and sequencing of questions to reduce overconfidence and minimize the effects of cognitive biases. In this particular case, the experimenters also developed a set of seed questions (with known answers) to evaluate expert competence, and then pooled judgments mathematically, giving experts with higher marks on the seed questions greater weight, yielding a single best-estimate doseresponse curve for each chemical. From this, regulatory authorities could derive the buffer zone depth that limits the risk of lethal exposure to the safety standard previously set. The study just described is an example of structured expert elicitation (SEE), a family of methods designed to incorporate expert knowledge into risk management and strategic and policy decisions. SEE has spread from early applications in defense planning and aero-space engineering to wider use in the nuclear sector (Cooke, 2013) and in support of regulatory and other policy decisions (Aspinall, 2010; Morgan, 2014) especially environmental regulation, food safety, public health, and disaster management. There are a variety of methods; some include deliberation between experts, others (like the above) elicit judgments separately and then aggregate the results, yet others report a full range of opinion, disagreements and all (see Cook and Goossens (2000), Goossens et al. (2008) and Martini and Boumans (2014) for wideranging reviews of applications and discussions of the method). 5. Changing Paradigm. In Section 3 we argued that there are severe uncertainties in regional climate projection, and that these uncertainties are best assessed via expert judgment. Section 4 looked at an analogous case of environmental planning where SEE 9 is used to assess uncertainties and synthesize different sources of knowledge into actionable results. Putting these two strands of argument together leads to our core suggestion: that all local climate change projections intended to support decisionmaking should be produced through a process of SEE. To be clear, SEE does not replace modeling exercises on this approach. Rather, the suggestion is to quit regarding downscaled GCM outputs as the only (or the privileged) source of information, and to use model outputs alongside other sources of information. Experts have much to draw on, including physical understanding of large-scale processes such as the greenhouse effect, energy conservation principles (sometimes violated by model behavior), historical data sets, projections based on simple models and statistical methods, and reconstructions of climate change in the distant past. Knowledge about model development is also essential, including understanding of the effects of km-scale resolution, of including or excluding particular processes (or including them only through ad-hoc parameterization), of mathematical idealizations of conceptual relationships, and of the numerical approximations required to process the mathematical idealizations. Simulations of the recent past have also been statistically compared to corresponding observations and plausible reasons for inaccuracies debated. The relative strengths and weaknesses of alternative model formulations are continually debated as well. All of this knowledge and understanding (and more) can inform expert judgments about local-scale climate projections. There are many variations in SEE methodology and a large literature on its theory and application. If one accepts our proposal, many questions remain about how best to implement SEE for climate change projections, including how to pick the experts, what constitutes a reasonable quorum of relevant experts, and how best to incentivize participation and address conflicts of interest. Other questions concern the mathematical formalism used to encode an expert's judgments (e.g., precise versus imprecise probability), and whether the individual judgments are aggregated (Genest and Zidek, 1986) or presented as a group. Cooke (1991, 2014) and Aspinall (2010), for example, promote a version of the former, while Morgan (2014) generally favors the latter. We believe no one method is right or wrong in general. The challenge is finding the right tool for the job, and this is a task for future research. Irrespective of the specifics, there are a few general issues we wish to address. One knee-jerk reaction to our proposal may be that results produced with SEE are too 10 subjective to inform evidence-based policy. This is wrong. All climate modeling exercises involve subjective decisions that can significantly influence the outcome, from the selection of models to include in an ensemble, to the range of parameter perturbations and initial conditions explored in a sensitivity analysis, to the method of downscaling (not to mention the decisions made in constructing a GCM in the first place). By asking experts to assess and communicate the consequences of these decisions as best they can, considering all available information, the proposed approach is more transparent about the subjectivity, but it is not more subjective. The alternative to SEE is not objective calculation; the alternatives are unstructured expert judgment (as in the IPCC report), or unstructured non-expert judgment (when model-world results are communicated to policy-makers ill-equipped to assess their real-world importance). Reliable objective calculation of probabilities based on nonlinear model output has been argued elsewhere to be infeasible (Frigg, Bradley, Du, and Smith, 2014). SEE does not guarantee trustworthiness, but it can compensate for (some of) the (known) inadequacies of climate models. Echoing a notorious remark of Churchill's about democracy, we submit that SEE is the worst form of science advising, except for all those other forms that have been tried from time to time. Others seem to have reached a similar conclusion and called for the IPCC assessment process to incorporate SEE (Reilly et al., 2001; Oppenheimer et al., 2007; Shapiro et al., 2010; Yohe and Oppenheimer, 2011). IPCC author teams have so far made their collective judgments through an informal procedure that does little to address cognitive biases and adverse group dynamics such as deference to high status participants. And despite the recent emphasis on "traceable accounts," the expert judgment process is still largely opaque to outsiders. Standard SEE practice requires publishing the interview protocols, raw elicitation results (anonymized), and any procedures for facilitated deliberation and/or mathematical aggregation along with the final results. We support the incorporation of SEE into IPCC process and recommend that the same ought to be done in the case of local climate projections. Even those who agree in principle with our conclusion might worry that SEE will offer no new insights in the case of local climate change projections because the outcome is in effect a foregone conclusion. We know, the objection continues, that models underestimate uncertainty and so SEE will simply downgrade all probability distributions, the only question being by how much. The worry is misguided. First, even where the need to downgrade is clear from the start, the degree is not, and the degree 11 matters. Second, this is not the only thing that can happen. In some contexts (weather forecasting, for instance) models can overestimate uncertainty and experts will sharpen the probability distribution. In other cases uncertainties are asymmetrical and can lead to a skewing rather than a widening of the distribution, and in yet other cases the verdict could be that the distribution is systematically biased and needs to be "moved." Moreover, SEE need not enter the process only at the end of resource intensive modeling projects, but should also inform their aims and methodology earlier on. This could constrain the choice of variables, time scales, and spatial scales for which it is worth trying to provide decision-relevant model information. Undoubtedly, there are some such limits beyond which projections are simply not plausible; arguably, many climate services products already exceed those limits. If experts judge large swaths of model output useless or misinformative for decision-making, something has gone wrong at an earlier stage. 6. Conclusion. For the purpose of local climate change decision support, direct model outputs can be misleading. The IPCC's published expert judgment about modeled global mean temperature projections implies that probabilities derived directly from model output on smaller-scale variables are inappropriate inputs to decision-making. Better models and faster computers cannot by themselves make the problem go away, and provision of local climate change projections should instead be reconceptualized as a problem of eliciting expert judgment based on a number of sources including but not limited to model output. Many open questions remain about how best to implement structured expert elicitation for local climate projections. Acknowledgments. This work was supported by the AHRC's Managing Severe Uncertainty project, the Spanish Ministry of Science and Innovation (grant FFI201237354), and the Natural Environment Research Council (grant NE/M008304/1). References American Meteorological Society. 2015. "A policy statement of the American Meteorological Society: Climate Services," The AMS Council, https://www2.ametsoc.org/ams/index.cfm/about-ams/ams-statements/statementsof-the-ams-in-force/climate-services1/ (accessed January 10, 2016). 12 Aspinall, Willy. 2010. "A route to more tractable expert advice." Nature 463 (7279): 294-95. California Energy Commission. 2016. "Precipitation: Decadal averages map," CalAdapt, http://cal-adapt.org/precip/decadal/ (accessed January 10, 2016). Cooke, Roger and Louis Goossens. 2000. "Procedures guide for structured expert judgment." Project report, European Commission Nuclear Science and Technology. Cooke, Roger M. 1991. Experts in uncertainty: opinion and subjective probability in science. Oxford University Press. Cooke, Roger M. 2013. "Uncertainty analysis comes to integrated assessment models for climate change... and conversely." Climatic change 117 (3): 467-79. Cooke, Roger M. 2014. "Deep and shallow uncertainty in messaging climate change." In Safety, Reliability and Risk Analysis: Beyond the Horizon, ed. Raphael D. J. M. Steenbergen, Pieter H. A. J. M. van Gelder, Simona Miraglia, and A. C. W. M. Vrouwenvelder, 13-25. CRC Press. Frigg, Roman, Leonard A. Smith and David A. Stainforth. 2013. "The Myopia of Imperfect Climate Models: The Case of UKCP09." Philosophy of Science 80: 886–97. Frigg, Roman, Leonard A. Smith and David A. Stainforth. 2015. "An Assessment of the Foundational Assumptions in High-Resolution Climate Projections: The Case of UKCP09." Synthese 192 (12): 3979–4008. Frigg, Roman, Seamus Bradley, Hailiang Du, and Leonard A. Smith. 2014. "Laplace's Demon and the Adventures of his Apprentices," Philosophy of Science 81(1): 31-59. Genest, Christian and James V. Zidek. 1986. "Combining probability distributions: A critique and an annotated bibliography." Statistical Science 1 (1): 114–35. Goossens, Louis H. J., Roger M. Cooke , Fred Woudenberg and Pieter Van Der Torn. 1998. "Expert judgement and lethal toxicity of inhaled chemicals." Journal of Risk Research 1 (2): 117-33 Goossens, Louis H. J., Roger Cooke, Andrew R. Hale and Ljiliana Rodic -Wiersma. 2008. "Fifteen years of expert judgement at TUDelft." Safety Science 46 (2): 23444. High Speed Two (HS2) Limited. 2013. "London-West Midlands Environmental Statement," Volume 5, Technical Appendices: Resilience to impacts from 13 climate conditions, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/25 9614/Volume5_Climate_Change_Resilience_Report_CL-003_000.pdf (accessed January 10, 2016). Knutti, Reto, Reinhard Furrer, Claudia Tebaldi, Jan Cermak and Gerald A. Meehl. 2010. "Challenges in Combining Projections from Multiple Climate Models." Journal of Climate 23:2739-58. Knutti, Reto and Jan Sedlacek. 2013. "Robustness and uncertainties in the new CMIP5 coordinated climate model projections." Nature Climate Change 3: 36973 Maslin, Mark and Patrick Austin. 2012. "Uncertainty: Climate models at their limit?" Nature 486 (7402): 183-84. Martini, Carlo and Marcel Boumans. 2014. Experts and Consensus in Social Science. Ethical Economy. Studies in Economic Ethics and Philosophy. Heidelberg New York Dordrecht London: Springer. Mearns, Linda. 2014. "Credibility of Regional Climate Model Projections of Future Climate," Lecture, Knowledge and Models in Climate Science: Philosophical, Historical, and Scientific Perspectives, University of Western Ontario, October 2014. (Available online at http://youtu.be/uAM5bHH4VGo.) Met Office. 2009. "Worked Examples," UK Climate Projections, http://ukclimateprojections.metoffice.gov.uk/23102 (accessed January 10, 2016). Morgan, M. Granger. (2014). "Use (and abuse) of expert elicitation in support of decision making for public policy." Proceedings of the National Academy of Sciences 111 (20): 7176–84. Oppenheimer, Michael, Brian C. O'Neill, Mort Webster and Shardul Agrawala. 2007. "The limits of consensus." Science Magazine's State of the Planet 2008-2009: with a Special Section on Energy and Sustainability 317: 1505–06. Oreskes, Naomi. 2007. "The Scientific Consensus on Climate Change: How Do We Know We're Not Wrong?" In Climate Change: What It Means for Us, Our Children, and Our Grandchildren, ed. Joseph F. C. DiMento and Pamela Doughman, 65-99. Boston: MIT Press. ClimateImpactsOnline. 2012. "ClimateImpactsOnline," http://www.climateimpactsonline.com/ (accessed January 10, 2016). 14 Reifen, Catherine, and Ralf Toumi. 2009. "Climate projections: Past performance no guarantee of future skill?" Geophysical Research Letters 36 (13): L13704 Reilly, John, Peter H. Stone, Chris E. Forest, Mort D. Webster, Henry D. Jacoby and Ronald G. Prinn. 2001. "Uncertainty and climate change assessments." Science 293 (5529): 430–33. Shapiro, Harold T., Carlos Henrique de Brito Cruz, Maureen Cropper, Fang Jingyun, Louise O. Fresco, Syukuro Manabe, Goverdhan Mehta, Mario Molina, Peter Williams, Ernst-Ludwig Winnacker, and Zakari Abdul Hamid. 2010. "Climate change assessments: Review of the processes and procedures of the IPCC." Technical report, InterAcademy Council. Stocker, Thomas F., Dahe Qin, Gian-Kasper Plattner, Melinda M.B. Tignor, Simon K. Allen , Judith Boschung, Alexander Nauels, Yu Xia, Vincent Bex and Pauline M. Midgley, eds. 2013. Climate Change 2013: The Physical Science Basis. Working Group I Contribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press. The Nature Conservancy. 2009. "Climate Wizard," http://www.climatewizard.org/AboutUs.html (accessed January 10, 2016). Besa, Monica, C. 2013. "Using climate information to support adaptation planning and policy-making: A practical case study in Bagamojo district, coastal Tanzania," weADAPT, the Collaborative Platform on Climate Adaptation, (accessed January 10, 2016). Wilby, Robert, Robert J Nicholls, Rachel Warren, Howard S Wheater, Derek Clarke and Richard J Dawson. 2011. "New nuclear build: adaptation options over the full life-cycle." Proceedings of Institution of Civil Engineers: Civil Engineering 164 (4): 129-36. Yohe, Gary and Michael Oppenheimer. 2011. "Evaluation, characterization, and communication of uncertainty by the intergovernmental panel on climate change-an introductory essay." Climatic Change 108 (4): 629-39.