Technology development and innovation are fundamentally different from scientific research. However, in many circumstances, they are evaluated jointly and by the same processes. In these cases, peer review—the most usual procedure for evaluating research—is also applied to the evaluation of technological products and innovation activities. This can lead to unfair results and end up discouraging the involvement of researchers in these fields. This paper analyzes the evaluation processes in Uruguay's National System of Researchers. In this system, all members' activities, both (...) scientific and technological, are evaluated by peer committees. Based on documentary analysis and semi-structured interviews, the difficulties faced by evaluators in assessing technology products are explored. The article highlights the persistence of a linear conception of the link between science and technology and describes the obstacles to assimilate the particularities of technological activities. Refereed publications are presented as the only uncontested product. Other types of output are reviewed with suspicion. This study emphasizes the need for specific mechanisms to evaluate technological production within academic careers. (shrink)

Science and technology policy is both faced by unprecedented challenges and itself undergoing seismic shifts. First, policy is increasingly demanding of science that it fixes a set of epochal and global crises. On the other hand, practices of scientific research are changing rapidly regarding geographical dispersion, the institutions and identities of those involved and its forms of knowledge production and circulation. Furthermore, these changes are accelerated by the current upheavals in public funding of research, higher education and technology development in (...) the wake of the economic crisis. The paper outlines an agenda for science & technology policy studies in terms of a research programme of a ‘cultural political economy of research and innovation’ (CPERI). First, the implications of the overlapping crises for science policy analysis are discussed. Secondly, three rough constellations of contemporary approaches to science policy are critically compared, namely: a techno-statist Keynesian governance; a neoliberal marketplace of ideas; and co-productionist enabling of democratic debate. CPERI is then introduced, showing how it builds on the strengths of co-production while also specifically targeting two major weaknesses that are of heightened importance in an age of multiple crises, namely neglect of political economy and the concept of power. (shrink)

U.S. federal research funding is generally justified by promises of public benefits, but the specific natures and distribution of such benefits often remain vague and ambiguous. Furthermore, the metrics by which outcomes are reported often do not necessarily or strongly imply the achievement of public benefits. These ambiguities and discontinuities make it difficult to assess the public outcomes of federal research programs. This study maps the terms in which the purposes and the outcomes of Advanced Research Projects Agency-Energy -a relatively (...) young and innovatively structured federal research agency-have been discussed and reported. I find that ARPA-E’s creation and funding have been justified with reference to a broad repertoire of economic, environmental, and national security values, but that the agency has been evaluated only through intermediate scientific and economic metrics and study of internal organizational structure. I suggest that these means of assessment elide ARPA-E’s lack of the financial scale, long time horizon, built-in customer, and radical vision which have been historically important to high-impact federal innovation, and I recommend the tracking of metrics more directly related to ARPA-E’s public value purposes as the agency grows older. This discussion illustrates the inadequacy of currently widespread metrics and conventional wisdom for the design and assessment of societally relevant research. (shrink)

Studies of how scientists select research problems suggest the process involves weighing a number of factors, including funding availability, likelihood of success versus failure, and perceived publishability of likely results, among others. In some fields, a strong personal interest in conducting science to bring about particular social and environmental outcomes plays an important role. Conservation biologists are frequently motivated by a desire that their research will contribute to improved conservation outcomes, which introduces a pair of challenging questions for managers of (...) science and scholars of policies governing science: 1) How do scientists integrate that goal into their processes of research priority evaluation, and 2) How can managers and funders of science utilize that knowledge in designing and administering funding programs? I use Q method to uncover four distinct schools of thought amongst researchers and knowledge-users about the merits of possible research priorities for coral reefs; one of the axes along which these schools of thought differ is in their interpretation of how science can and should interact with policy. The results reveal that perceived severity of reef stressors plays a role for some participants. Disciplinary training does not appear to be a major influence on research priority evaluation, but individual participants indicated professional expediency does prevent some researchers from pursuing or advocating that others pursue topics outside of that disciplinary specialty. Influences on and processes in research prioritization uncovered in this study have the potential to lead to counter-productive disciplinary path dependencies. From these results and building on outside literature, I conclude that better coordination and communication about research priorities across disciplines and with broader stakeholders – including knowledge users – could improve the research enterprise’s ability to contribute to meaningful societal and conservation goals. These findings are relevant to researchers and research administrators across disciplines that seek to conduct or fund science that is useful in addressing specific goals. (shrink)

Science and technology (S&T) policy studies has explored the relationship between the structure of scientific research and the attainment of desired outcomes. Due to the difficulty of measuring them directly, S&T policy scholars have traditionally equated “outcomes” with several proxies for evaluation, including economic impact, and academic output such as papers published and citations received. More recently, scholars have evaluated science policies through the lens of Public Value Mapping, which assesses scientific programs against societal values. Missing from these approaches is (...) an examination of the social activities within the scientific enterprise that affect research outputs and outcomes. We contend that activities that significantly affect research trajectories take place at the levels of individual researchers and their communities, and that S&T policy scholars must take heed of this activity in their work in order to better inform policy. Based on primary research of two scientific communities—ecologists and sustainability scientists—we demonstrate that research agendas are actively shaped by parochial epistemic and normative concerns of the scientists and their disciplines. S&T policy scholarship that explores how scientists balance these concerns, alongside more formal science policies and incentive structures, will enhance understanding of why certain science policies fail or succeed and how to more effectively link science to beneficial social outcomes. (shrink)

Because of the gross difficulties in measuring the societal impact of academic research, qualitative approaches have been developed in the last decade mostly based on forms of interaction between university and other societal stakeholders. In this paper, we suggest a framework for qualitative analysis based on the distinction between three dimensions of societal impact: epistemological, artefactual and interactive-institutional. The epistemological dimension addresses what new research results and understanding of relevant phenomena have contributed to solving of technological and societal problems. The (...) artefactual dimension comprises analysis of the artefacts, methods, tools and services through the use of which societal impact is realized. Finally, in the interactional–institutional dimension the forms and forums of collaboration between university and other societal actors are explored. The model is elaborated by analyzing the work of three university research groups in different disciplines. The framework may be used in articulating formative and dialogical peer evaluation of research to foster learning and social improvement as well as in evaluation of research proposals and research programs. (shrink)

This paper examines the broad social purpose of US climate science, which has benefitted from a public investment of more than $30 billion over the last 20 years. A public values analysis identifies five core public values that underpin the interagency program. Drawing from interviews, meeting observations, and document analysis, I examine the decision processes and institutional structures that lead to the implementation of climate science policy, and identify a variety of public values failures accommodated by this system. In contrast (...) to other cases which find market values frameworks (the profit as progress assumption) at the root of public values failures, this case shows how science values ( knowledge as progress ) may serve as an inadequate or inappropriate basis for achieving broader public values. For both institutions and individual decision makers, the logic linking science to societal benefit is generally incomplete, incoherent, and tends to conflate intrinsic and instrumental values. I argue that to be successful with respect to its motivating public values, the US climate science enterprise must avoid the assumption that any advance in knowledge is inherently good, and offer a clearer account of the kinds of research and knowledge advance likely to generate desirable social outcomes. (shrink)

Using the Public Value Mapping framework, I address the values successes and failures of chemistry as compared to the emerging field of green chemistry, in which the promoters attempt to incorporate new and expanded values, such as health, safety, and environmental sustainability, to the processes of prioritizing and conducting chemistry research. I document how such values are becoming increasingly public. Moreover, analysis of the relations among the multiple values associated with green chemistry displays a greater internal coherence and logic than (...) for conventional chemistry. Although traditional chemistry research has successfully contributed to both economic and values gains, there have been public values failures due to imperfect values articulations, failure to take a longer-term view, and inertia within a system that places too much emphasis on science values. Green chemistry, if implemented effectively, has potential to remedy these failures. (shrink)

ArgumentWhile science and economy are undoubtedly interwoven, the nature of their relationship is often reduced to a positive correlation between economic and scientific prosperity. It seems that the modern scholarship focusing on “success stories” tends to neglect counterintuitive examples such as the impact of economic crises on research. We argue that economic difficulties, under certain circumstances, may also lead to the prosperous development of scientific institutions. This paper focuses on a particular organism, the Pine Institute in Bordeaux in France. Not (...) only was it a key actor in the process of defining the discipline of resin chemistry, but also it remained for years at the heart of the local resin producing industry. Interestingly, there is an actual inverse correlation between the Institute's budgets and the prices and production of resinous products. The Pine Institute's existence seemed to have been driven by the crisis of the resin industry. (shrink)

A robust body of literature analyzes the shift of academic science toward more business-oriented models. This paper presents the findings of an empirical study investigating basic scientists’ attitudes toward publicly funded basic research at the Harvard Medical School and affiliated institutions. The study finds that scientists at the Harvard Medical School construe publicly funded basic research as inquiries that, whether use oriented or not, must be governed by the cognitive and social norms of the traditional mode of knowledge production. They (...) recognize that academic science is vulnerable to access by external capital but maintain that it remains distinct from research done in the private sector. Overall, the study demonstrates that important segments of academia have preserved a traditional approach to knowledge production, which is yet to be transformed by the entrepreneurial turn. (shrink)

This paper presents a framework to understand the impact of scientific knowledge on the policy-making process, focusing on the conceptual impact. We note the continuing dissatisfaction with the quality and effects of science-policy interactions in both theory and practice. We critique the current literature’s emphasis on the efforts of scientists to generate policy impact, because it neglects the role of ‘user’ policymaking organisations. The framework offered in the paper develops an argument about the essential role of institutional conditions of policy (...) ‘users’ for scientific knowledge to achieve impact. The framework is informed by the reflexive institutionalist and the neo-institutionalist theoretical approaches. Its main contribution is in outlining the intra- and inter-organisational conditions of policymaking organisations, along with personal characteristics of individual policy officials that influence the likelihood of scientific knowledge to generate conceptual impact. We also offer an operationalisation of the framework. The wider relevance of the paper is in moving the focus from the activities of scientists and the incentive structure in scientific organisations to the policy user side. (shrink)

When one wants to use citizen input to inform policy, what should the standards of informedness on the part of the citizens be? While there are moral reasons to allow every citizen to participate and have a voice on every issue, regardless of education and involvement, designers of participatory assessments have to make decisions about how to structure deliberations as well as how much background information and deliberation time to provide to participants. After assessing different frameworks for the relationship between (...) science and society, we use Philip Kitcher's framework of Well-Ordered Science to propose an epistemic standard on how citizen deliberations should be structured. We explore what potential standards follow from this epistemic framework focusing on significance versus scientific and engineering expertise. We argue that citizens should be tutored on the historical context of why scientific questions became significant and deemed scientifically and socially valuable, and if citizens report that they are capable of weighing in on an issue then they should be able to do so. We explore what this standard can mean by looking at actual citizen deliberations tied to the 2014 NASA ECAST Asteroid Initiative Citizen forums. We code different vignettes of citizens debating alternative approaches for Mars exploration based upon what level of information seemed to be sufficient for them to feel comfortable in making a policy position. The analysis provides recommendations on how to design and assess future citizen assessments grounded in properly conveying the historical value context surrounding a scientific issue and trusting citizens to seek out sufficient information to deliberate. (shrink)

We provide a critical assessment of the National Science Foundation's “broader impacts criterion” for peer review, which has met with resistance from the scientific community and been characterized as unlikely to have much positive effect due to poor implementation and adherence to the linear model heuristic for innovation. In our view, the weakness of NSF's approach owes less to these issues than to the misguided assumption that the peer review process can be used to leverage more societal value from research. (...) This idea, although undoubtedly well-meaning, is fundamentally flawed. Retooling or refining the Broader Impacts Criterion does not alter the fact that conventional peer review, based on specialized scientific and technical expertise, is not up to the task of ensuring adequate judgements about social impact. We consider some possible alternative approaches to providing greater social impact in science and include in our assessment past and current efforts at NSF and throughout the federal research establishment that address, in some cases having addressed for decades, the intentions and goals of the Broader Impacts Criterion, albeit using alternate mechanisms. We conclude that institution-building and explicit and targeted policy-making are more useful and democratically legitimate approaches to ensuring broad social impacts. (shrink)

We provide a critical assessment of the National Science Foundation's (NSF) “broader impacts criterion” for peer review, which has met with resistance from the scientific community and been characterized as unlikely to have much positive effect due to poor implementation and adherence to the linear model heuristic for innovation. In our view, the weakness of NSF's approach owes less to these issues than to the misguided assumption that the peer review process can be used to leverage more societal value from (...) research. This idea, although undoubtedly well-meaning, is fundamentally flawed. Retooling or refining the Broader Impacts Criterion does not alter the fact that conventional peer review, based on specialized scientific and technical expertise, is not up to the task of ensuring adequate judgements about social impact. We consider some possible alternative approaches to providing greater social impact in science and include in our assessment past and current efforts at NSF and throughout the federal research establishment that address, in some cases having addressed for decades, the intentions and goals of the Broader Impacts Criterion, albeit using alternate mechanisms. We conclude that institution-building and explicit and targeted policy-making are more useful and democratically legitimate approaches to ensuring broad social impacts. (shrink)

Helping scientists and engineers challenge received assumptions about how science, engineering, and society relate is a critical cornerstone for macroethics education. Scientific and engineering research are frequently framed as first steps of a value-free linear model that inexorably leads to societal benefit. Social studies of science and assessments of scientific and engineering research speak to the need for a more critical approach to the noble intentions underlying these assumptions. “Science Outside the Lab” is a program designed to help early-career scientists (...) and engineers understand the complexities of science and engineering policy. Assessment of the program entailed a pre-, post-, and 1 year follow up survey to gauge student perspectives on relationships between science and society, as well as a pre–post concept map exercise to elicit student conceptualizations of science policy. Students leave Science Outside the Lab with greater humility about the role of scientific expertise in science and engineering policy; greater skepticism toward linear notions of scientific advances benefiting society; a deeper, more nuanced understanding of the actors involved in shaping science policy; and a continued appreciation of the contributions of science and engineering to society. The study presents an efficacious program that helps scientists and engineers make inroads into macroethical debates, reframe the ways in which they think about values of science and engineering in society, and more thoughtfully engage with critical mediators of science and society relationships: policy makers and policy processes. (shrink)