Episteme http://journals.cambridge.org/EPI Additional services for Episteme: Email alerts: Click here Subscriptions: Click here Commercial reprints: Click here Terms of use : Click here THE LIMITED EFFECTIVENESS OF PRESTIGE AS AN INTERVENTION ON THE HEALTH OF MEDICAL JOURNAL PUBLICATIONS Carole J. Lee Episteme / Volume 10 / Issue 04 / December 2013, pp 387 -- 402 DOI: 10.1017/epi.2013.35, Published online: 13 November 2013 Link to this article: http://journals.cambridge.org/abstract_S174236001300035X How to cite this article: Carole J. Lee (2013). THE LIMITED EFFECTIVENESS OF PRESTIGE AS AN INTERVENTION ON THE HEALTH OF MEDICAL JOURNAL PUBLICATIONS. Episteme, 10, pp 387--402 doi:10.1017/epi.2013.35 Request Permissions : Click here Downloaded from http://journals.cambridge.org/EPI, IP address: 128.95.193.174 on 13 Nov 2013 the limited effectiveness of prestige as an intervention on the health of medical journal publications carole j. lee c3@uw.edu abstract Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. Peer review is essential to the normative well-functioning of science. It calls upon domain experts to scrutinize the soundness, signicance, and novelty of research for journal publication, conference presentations, and grants. The rigor of this "system of institutionalized vigilance" (Merton 1973: 339) bestows credibility upon successful submissions and, in so doing, bestows credibility upon the community and its mechanisms for self-governance. In addition to policing the content of science, peer review provides a system of incentives motivating scientists to disclose ndings in exchange for the prestige that comes with successful peer-reviewed publication and grant awards. It adjudicates the reward system in science by providing a mechanism by which the prestige of journals, editorial bodies, and granting agencies can be conferred upon scientists who successfully meet their stringent standards. The more exclusive and rened the gate-keeping, the more incentive scientists have to pass through the gate. The beauty of the traditional system of peer-reviewed publication, from the perspective of the social structure of science, is that the same mechanism – peer review – serves normative and motivational functions simultaneously: ambitious scientists do well professionally by doing well epistemically. This suggests a mechanism by which scientic communities should be able to align author behavior with community-wide epistemic goals: we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the capacity to motivate improved author behavior (Zuckerman and Merton 1971: 66).1 1 Questions about the efcient allocation of research effort have traditionally been the province of consequentialist accounts of scientic rationality, which evaluate policies for their effectiveness at Episteme, 10, 4 (2013) 387–402 © Cambridge University Press doi:10.1017/epi.2013.35 episteme volume 10–4 387 This paper will evaluate the success of this strategy in the context of academic medicine. Medical journal editors have periodically updated publication standards as a direct response to emerging empirical research on varieties of publication bias discovered in prestigious, peer-reviewed medical journals. The publication bias literature has diagnosed two major ways in which author behavior fails to align with the disciplinary goal of improving clinical practice. First, the disproportionate publication of positive outcomes leads to false and exaggerated effect sizes in meta-analyses (Simes 1986). Second, authors have adopted dodgy strategies to craft positive outcomes that fail to answer the most clinically relevant questions (Smith 2003: 1204). Unfortunately this latter phenomenon belongs to a broader trend of publication gamesmanship by authors across the sciences. In the rst part of the paper, I will discuss the co-emergence of the motivational and epistemic functions of peer-reviewed publication, motivate the hypothesis that stricter publication standards should improve author behavior, and introduce the institutions (created, constituted, and endorsed by medical journal editors) to leverage this proposed mechanism for change. Second, I will articulate the discipline-wide epistemic goal that medical journal editors hope to achieve through their interventions and explain why publication bias is prima facie normatively appropriate in relation to this goal. Third, I will evaluate editor efforts to debias trial data available for meta-analysis. Fourth, I will evaluate editor efforts to stop more pernicious forms of publication gamesmanship. To close, I will identify further actions – by members of the scientic community and federal regulators – that might help to better align author behavior with academic medicine's goal of improving clinical practice. 1. changing author behavior by changing publication standards Peer-reviewed publication has long incentivized individual scientists in ways that have structured scholarly communication as we know it today. We can see how these incentives work by tracing peer review's history all the way to its rst implementation in The Philosophical Transactions of the Royal Society back in the 1750s.2 Before that time, scientists had set "a premium upon secrecy" as a way to safeguard their intellectual property from competing scientists (Zuckerman and Merton 1971: 69) – at least, until they had "amass[ed] a vast quantity of material, or. . . conceive[d] a complete new 'world system'" sufcient for publishing a monograph-length treatise (Ziman 1969: 318). In contrast, the Philosophical Transactions provided scientists the opportunity to archive and establish priority for ideas that t into article-length papers (Zuckerman and Merton 1971: 70). As the functioning editor Henry Oldenberg wrote to Robert Boyle, the journal "inviolably discovering signicant truths or empirically adequate theory. Note, however, that such questions are also properly raised within procedural accounts of scientic rationality for which "following proper procedures is constitutive of scientic rationality" (Solomon and Richardson 2005: 212). For those committed to a procedural approach, this paper may be understood as evaluating strategies for motivating scientists to develop data, methods, and hypotheses in ways that improve their sensitivity to criticism, which, if successful, would improve community responsiveness and procedural objectivity (Longino 1990). This instrumental perspective does not subsume the procedural account under a consequentialist framework so long as the epistemic goal of achieving community responsiveness is not characterized independently of procedural norms. 2 For a brief history, see Lee et al. (2013). carole j . lee 388 episteme volume 10–4 preserved to all posterity" a scientist's intellectual ownership of particular discoveries and experiments (Oldenberg 1966: 70). However, archiving and establishing priority for smaller pieces of work were not the sole motivations for scientists. The Royal Society, concerned to protect the reputation of their Society and journal, established a new form of social technology: namely, peer review. From a sociological point of view, this turned the scientists at the Royal Society – in their capacity as reviewers and editors – into "status judges" charged with evaluating and allocating rewards for successful role-performance (Merton and Zuckerman 1971: 467–8). The reward was not simply publication, but prestige: recognition in a forum valued by other scientists whose opinions were valued by submitting authors. In contemporary times, peer-reviewed publication continues to provide a system of incentives motivating scientists to formulate and disclose experiments and methods in exchange for peer recognition. Editors and editorial board members are selected for characteristics associated with prestige – such as educational background, scholarly contributions, and publication experience (Weller 2001: 111). Selecting editors for these qualities burnishes the reputation of the journal and the vetted contents therein. Selectivity is sometimes measured through proxies such as rejection rates.3 The rough idea behind this metric remains the same: the more selective the peer-run gate-keeping process, the more prestigious the venue, and the more incentive for individual authors to publish there.4 From a purely sociological point of view, we should expect journal editors to have the epistemic authority and political power to motivate authors in ways that realign their behavior with community-wide epistemic goals.5 Academic medicine is a good context for evaluating whether this expectation pans out. Prominent medical journal editors have explicitly sought to improve author behavior through the articulation and endorsement of revised publication norms. And, a thriving community of publication bias researchers has carefully tracked the impact of these interventions on the health of the publication record. Publication bias research, as a genre of empirical study, received an institutional home with the inauguration of the International Congress on Peer Review and Biomedical Research, rst started in 1986 and sponsored by the Journal of the American Medical Association. Soon after, in the early 1990s, editors for top general medical journals organized to articulate improved guidelines for the reporting of randomized controlled trials – guidelines that became the Consolidated Standards for Reporting Trials (CONSORT) statement of 1996.6 This document has been regularly updated and extended in response 3 The association between rejection rates and the prestige of a peer-reviewed journal is oft expressed by scientists, e.g., Casadevall and Fang (2012). 4 For a study on the relationship between perceived journal prestige versus rejection and citation rates in philosophy, see Lee and Schunn (2010). 5 For a discussion of how the perceived prestige of authors can inuence peer review, see Lee and Schunn (2011). 6 CONSORT 1996 declared that the success of its interventions should be evaluated through randomized controlled trials (Begg et al. 1996: 639). Randomized controlled trials (RCTs) measure the effectiveness of an experimental intervention relative to some comparison treatment such as placebo or standard treatment. Methodologically, an intervention is deemed successful insofar as it provides a statistically signicant improvement over the comparison treatment. In the case of publication bias, this means that CONSORT interventions would be deemed successful in cases involving a statistically signicant the limited effectiveness of prest ige episteme volume 10–4 389 to emerging evidence on harmful trial reporting behavior. The CONSORT statements are published by the most prestigious medical journals, endorsed by hundreds more, and explicitly referred to in journal instructions to authors (Altman 2005). A second institution, the International Committee of Medical Journal Editors (ICMJE), constituted by editors at top general medical journals across the world, publishes and periodically updates a statement on Uniform Requirements for Manuscripts Submitted to Biomedical Journals that is endorsed by thousands of medical journals. The statement rst emerged in 1979, primarily as a style guide, but has since shifted its focus towards methodological concerns (ICMJE 2004). The strategy of improving author behavior through changes in publication standards had an auspicious start. Large reviews demonstrated signicant improvements in reporting for those journals adopting the rst CONSORT statement compared with journals that did not (Plint et al. 2006; Kane et al. 2007); and, among CONSORT-adopting journals, a review found improvements in reporting post-CONSORT (Moher et al. 2001). The magnitude of the percentage changes suggest that the 1996 statement was successful in shifting author behavior: for example, failing to explain subject attrition (Kane et al. 2007) or report allocation concealment (Moher et al. 2001) decreased from the 60%+ range down to 16% and 31% respectively.7 This result provides a kind of proof of concept that this strategy for realigning author behavior with the community's goals can work in the real world, at least in certain conditions. 2. medicine's goals, publication norms, and publication bias Unlike many scientic disciplines (Kitcher 2001: 145), academic medicine has explicitly articulated its primary epistemic goal. Textbooks tell us that the "best research" evidence "invalidates previously accepted diagnostic tests and treatments and replaces them with new ones that are more powerful, more accurate, more efcacious, and safer" (Sackett et al. 2000: 1): the most scientically signicant research is that which improves clinical practice. This idea gets echoed in medical journal instructions to authors. The Lancet "prioritises reports of original research that are likely to change clinical practice or thinking about a disease" (The Lancet 2011). The Journal of the American Medical Association's "highest priority for publication is given to reports of original research that present important, potentially practice-changing ndings" (Fontanarosa and DeAngelis 2004). And, the British Medical Journal takes its mission to be "to engage, improvement in author behavior for journals that adopt the requirement versus those that do not – even if, despite the CONSORT intervention, the majority of authors continue to engage in inappropriate behavior. I think our normative expectations should be stronger than this: we should aim for CONSORT and ICMJE requirements to change norms of author behavior, where a simple proxy for "normal" author behavior might be something like majority author behavior. The reexive felicity of using RCTs to study the behavior of researchers in academic medicine does not imply its normative t. For another case where the reexive use of scientic methods to evaluate the scientic practice of peerreviewed publication fails to provide an appropriate normative t, see Lee (2012). 7 Allocation concealment procedures prevent patients and those recruiting them from gaining knowledge of individual patient treatment group assignment (e.g., placebo versus experimental treatment) in order to prevent allocation biases and protect random assignment. carole j . lee 390 episteme volume 10–4 inform, and stimulate doctors, researchers and other health professionals in ways that will improve outcomes for patients" (British Medical Journal 2011). How can we fulll this lofty goal? First off, authors must, when evaluating new treatments, provide studies demonstrating improved efcacy or safety – that is, they must demonstrate a positive direction in favor of the experimental hypothesis, where larger effects are more valued than smaller effects (Sackett et al. 2000: 4). This is not to say that negative results are not relevant to changing clinical practice. Demonstrating the inefcaciousness of currently adopted therapies can improve clinical practice: important examples include trials on the relative inefcaciousness of standard drugs used to treat depression (Kirsch et al. 2002) and the inefcaciousness of hormone replacement therapy to prevent heart disease in postmenopausal women (Hulley et al. 1998; Grady et al. 2002). However, the assumption here is that most currently adopted clinical practices are efcacious to some degree, which turns the primary task to be one of improving upon standard treatment. Secondly, in order for the conclusions of their studies to be sufciently robust to ground recommendations for changing current practice, researchers must provide statistical measures of signicance, power, or condence to demonstrate that the ndings were unlikely to result from chance alone. Statistical signicance measures provide an estimate of the strength of the evidence against the null hypothesis (Sackett et al. 2000: 233). In general, a publication preference for positive, statistically signicant outcomes favoring a new experimental treatment serves medicine's mission of improving and transforming current clinical practice by adding better treatments. So, it shouldn't be surprising that top general medical journals have a preference to publish positive outcomes as opposed to negative outcomes. "Positive outcomes" include results that have a positive direction (Bardy 1998: 147), are statistically signicant (Dickersin et al. 1992; Dickersin and Min 1993; Stern 1997), or both (Ioannidis 1998: 282). To be consistent with the literature, I'll use "positive outcomes" to refer to this disjunction of result types. Likewise, "negative outcomes" include a disjunction of result types, including results that have a negative direction, are not statistically signicant, or both. Publication bias is the tendency for journals to publish studies reporting positive outcomes. A large study revealed that, of the studies submitted to the Journal of the American Medical Association, 20.4% of those with statistically signicant results on the primary outcome were accepted. This is slightly higher than the base rate for publication at the journal, which was 17.9%. In comparison, 15.0% of studies with negative results were accepted, which lies below the base rate for publication (Olson et al. 2002). Reporting positive outcomes raises author chances of publication above the base rate, while reporting negative outcomes lowers author chances of publication below the base rate. We can see much more dramatic publication bias comparisons when we shift our focus from the population of studies submitted for journal review to the larger population of studies approved by ethics boards – including papers that will never get submitted for journal review. Table 1 suggests that, among studies approved by ethics boards, those with positive outcomes are much more likely to be published than those with negative outcomes. Why does the publication bias differential shift so dramatically when we look at the larger population of studies that are and are not submitted for journal review? It turns out that authors are selective in their choices about which trials to submit for journal the limited effectiveness of prest ige episteme volume 10–4 391 review. A large study at Johns Hopkins found that over 90% of trials approved by its institutional review boards remained unpublished due to the inaction of investigators (Dickersin et al. 1992: 377).9 Self-reported reasons for non-submission reveal both epistemically virtuous and sullied considerations: authors report non-submission due to insufcient importance or interest of negative results as well as unfavorable likelihood of journal acceptance (Easterbrook et al. 1991; Dickersin et al. 1992; Dickersin and Min 1993; Weber et al. 1998). Further considerations include positive outcomes' higher citation counts (Ioannidis 2005a) and the strength they lend to grant applications (Ross et al. 2009: 6). 3. editorial efforts to debias meta-analysis measures Unfortunately, the systematic publication of positive outcomes skews the sample of published results towards positive outcomes, which makes it difcult – if not impossible – to measure true treatment effects. Publication bias for results favoring the experimental hypothesis can create effects where there are none: for example, meta-analysis of published trials suggested a survival advantage when using combination chemotherapy over an initial alkylating agent in the treatment of ovarian cancer; but, when results from all registered trials were pooled, the survival advantage disappeared (Simes 1986). More subtly, a publication preference for statistically signicant results can lead to exaggerated effect size measurements via meta-analysis. Hypothetically speaking, if the published record provided a representative sample of all conducted trials, by the law of large numbers, we would expect variance in effect magnitude to decrease as sample size increases; and, we would expect the mean effect magnitude to remain constant across sample size. However, if there is a publication bias for statistically signicant results, then in cases where there is a moderate true effect, the vast majority of smaller studies reaching statistical signicance will have extreme magnitudes in the direction of the true effect (Palmer 2000), while very few small studies with results in the opposite direction Table 1. Percentage of trials approved by ethics boards that were later published! Positive outcome Negative outcome Institution granting ethics approval No. of studies Reference 90% 29% Swedish drug regulatory agency 42 (Melander et al. 2003: 1173) 97% 8% FDA 74 (Turner et al. 2008) 97.6% 85.1% National Institutes of Health 253 (Dickersin and Min 1993) 88.5% 69.4% Johns Hopkins Medical School 537 (Dickersin et al. 1992) 78% 54% University Hospital (Sydney, Australia) 218 (Stern 1997) 65% 16% Central Oxford Research Ethics Committee 487 (Easterbrook et al. 1991)8 8 The Central Oxford Research Ethics study might report lower rates of publication (for both positive and negative outcomes) because it allowed for less time between data collection and publication. 9 Authors are also responsible for the greater time lag between study completion and submission for journal review of negative versus positive results: see Ioannidis (1998). carole j . lee 392 episteme volume 10–4 of the true effect will reach statistical signicance (Jennions and Møller 2002a: 212). Within the publication record, this leads to the increase in mean effect sizes as sample size decreases (Jennions and Møller 2002b: 47) – something we would not expect to see had studies been sampled in an unbiased manner. Literature reviews demonstrate that small studies published in medical journals do report larger effects than large studies do in the prevention of heart attacks (Egger and Smith 1995) as well as the treatment of obesity (Allison et al. 1996), preeclampsia (Pipkin et al. 1996), and cancer (Berlin et al. 1989). Inaccurate meta-analysis measures prevent the community from being able to evaluate the true clinical costs and benets of new treatments. To address this problem, the ICMJE instituted a requirement, effective in 2005, that trials be registered in a public database with information about the study's intervention(s), comparison groups, primary and secondary outcome measures (including harms), and contact information. That way, researchers conducting meta-analyses could refer to the database to identify relevant trials and contact individuals with access to and responsibility for the data (Mathieu et al. 2009: 982). At rst, it appeared that the ICMJE requirement successfully shifted author behavior. After the institution of the requirement, the average number of new trials registered per week at ClinicalTrials.gov jumped from 30 to 220 – a pace that has continued (Zarin et al. 2007: 2114). However, a closer look reveals that the quality of the registration information compromises the purpose of the registration requirement. Since the institution of the 2005 ICMJE requirement, the vast majority of registered trials provided at least one primary outcome; however, 61.8% of trials failed to provide specic/meaningful measures for those outcomes (Viergever and Ghersi 2011). Among published studies subject to the 2005 ICJME requirement, fewer than half (45.5%) conformed to it: 27.6% of published studies failed to register altogether, while 12.9% of those registered had no or an unclear description of the primary outcome (Mathieu et al. 2009: 982). Likewise, reporting of mandatory contact information has been inadequate. Although 74–83% of registered trials provided a contact name as required, 60–62% of trials failed to report the requisite email address (Sekeres et al. 2008; Viergever and Ghersi 2011). These ndings perpetuate worries that trial data available for and recognizably relevant to answering specic questions continue to skew meta-analysis measures. Medical journal editor efforts were not the only ones that failed to improve trial registration. The FDA Amendments (FDA) Act of 2007 required that trial results be registered within a year of trial completion. However, a large study of trials subject to mandatory reporting found that 78% had failed to comply (Prayle et al. 2012). Why do authors out both legal and publication requirements? Unfortunately, further research on publication bias suggests that more pernicious motives and forms of gamesmanship are afoot. 4. editorial efforts to curb dysfunctional gamesmanship The incentive to publish in peer-reviewed journals has increasingly motivated dysfunctional forms of publication gamesmanship in science. Editors complain of practices – such as self-plagiarism, duplicate publication, and the carving up of large studies into the smallest publishable unit – that boost the number of authored publications by diluting or obscuring their intellectual contribution (Kravitz and Feldman 2011). Retractions have also been on the rise, even among elite journals, due to misconduct and questionable the limited effectiveness of prest ige episteme volume 10–4 393 practices (Fang and Casadevall 2011; Grieneisen and Zhang 2012). Medicine currently accounts for half the retractions in publications across the physical, social, and engineering sciences (Grieneisen and Zhang 2012: 6). In medicine, the publication bias literature has catalogued many types of gaming behavior in which authors meet the letter of the norm of publishing positive outcomes (for individual gain) while violating the norm's function of improving clinical practice (at community-wide cost). Gaming strategies include engaging in covert duplicate publication,10 conducting weaker study designs,11 undertaking improper dosing strategies,12 choosing and gerrymandering endpoints post hoc, and omitting mention of harms.13 In what follows, I will focus on the last two "signicance questing" (Rothman 1986: 445) strategies because of the availability of systematic literature reviews evaluating the effectiveness of CONSORT and ICMJE interventions designed to stop them. Post hoc selection of endpoints In their "chase of statistical signicance" (Ioannidis 2005b: 696), authors select endpoints post hoc. A large study comparing the protocols of trials approved by ethics committees discovered that 62% of trials changed, introduced, or omitted at least one primary outcome in the published paper in favor of statistically signicant ones and that these changes 10 A review of 42 clinical trials submitted to the Swedish Drug Regulatory Authority discovered that those with positive outcomes were published multiple times, where duplications could not be identied by readers for lack of cross-referencing (Melander et al. 2003: 1173). Covert republication increases the number of author publications at the cost of skewing future meta-analysis measures: one study estimated a 23% exaggeration of meta-analysis effect size due to covert republication for an anti-nausea medication (Tramèr et al. 1997). To make republication more transparent, ICMJE 2004 and CONSORT 2010 required that trial registration numbers be included in manuscripts. 11 Industry-funded studies are especially likely to compare their drugs to placebo as opposed to standard treatment (Easterbrook et al. 1991), despite standard treatment's being the most relevant clinical comparison to draw (Angell 2004: 107). Industry-funded studies are also more likely to run equivalence trials, which "demonstrate" that a new treatment is "equivalent" to standard treatment by rejecting the null hypothesis that the treatment effects for two treatments differ by more than some equivalence margin (Snapinn 2000). Authors have taken advantage of the conceptual difculty involved in establishing proper equivalence margins by using grossly generous margins and stingy sample sizes (Barbui et al. 2002; Garattini et al. 2003). 12 Authors adopt improper dosing amounts, schedules, and administration methods for standard treatment to reduce its efcacy or tolerance and, thereby, boost the relative efcacy or tolerance of the experimental treatment (Simpson and Lindenmayer 1997; Garattini et al. 2003; Smith 2003). This behavior has been discovered in studies on immunosuppressants (Bruijn and van der Woude 2002), antifungal agents (Johansen and Gøtzsche 1999), second-generation neuroleptic drugs (Safer 2002), selective serotonin reuptake inhibitors (Barbui et al. 2002), and nonsteroidal anti-inammatory agents (Rochon et al. 1994). Arguably, such studies do not provide sufcient ethical justication for involving human subjects and should be blocked by research ethics committees (Garattini et al. 2003). 13 All of these critiques are predicated on the rejection of a value-free notion of objectivity in academic medicine: they take for granted that moral and social values dictating what constitute signicant clinical outcomes provide the proper grounds for evaluating the proper domain of research questions to pursue (Longino 1990: 90–2), types of analyses and endpoints to study (Longino 1990: 86; Anderson 1995), and the necessary selection of sample sizes and statistical margins according to which a hypothesis is to be accepted (Rudner 1953: 3; Douglas 2000: 565). For more on how contextual values can inuence methodological and evidential standards in biomedical research, see Intemann and de Melo-Martin (2010). carole j . lee 394 episteme volume 10–4 were not submitted as protocol amendments or mentioned in the published study (Chan et al. 2004). Authors obscure a new treatment's weaknesses by lumping together strong and weak endpoints into a single one (CAPRIE Steering Committee 1996; Born and Collins 1997; Garattini et al. 2003: 1200–1). In light of these strategies, it should not be surprising that a review of 74 studies approved by the FDA found that 33% of results deemed to be negative by the regulatory agency were nonetheless published by authors "in a way that conveyed a positive outcome" (Turner et al. 2008: 256). Can changes to publication standards incentivize improved author behavior? An important feature of the aforementioned 2005 ICMJE requirement was that it demanded that trials be registered before the enrollment of the rst patient with information about the study's intervention(s), comparison groups, and primary and secondary outcome measures (including harms). Prospective registration was designed to curb the practice of choosing endpoints post hoc by enhancing transparency and accountability in the planning and reporting of clinical trials (Mathieu et al. 2009: 982). However, subsequent research has demonstrated that among papers published in highimpact medical journals in 2008, more than half (54.5%) had registered improperly by failing to describe clearly a primary outcome or by failing to register prospectively. Among studies that were registered properly, 31.3% had, in the published manuscript, changed the primary outcome post hoc in favor of one that turned out to be statistically signicant (Mathieu et al. 2009: 982). A study of registered trials (published and unpublished) at around the same time period shows even higher rates of failure when it comes to prospective registration and the adequate description of primary outcomes (Viergever and Ghersi 2011). Despite the ICMJE requirements, authors continue to game publication by leaving themselves the exibility to frame results in ways that report positive outcomes.14 Authors have an incentive not to comply with the ICMJE requirements. Prospective registration of trial protocol would obstruct signicance questing practices such as: failing to report all of a study's dependent measures, failing to report all of a study's conditions, and selectively reporting studies that "worked" (John et al. 2012: 527). Signicance questing is not only a problem for academic medicine, but a problem plaguing any scientic discipline that prefers to publish statistically signicant effects, including psychology (Neuliep and Crandall 1990, 1993) and ecology (Palmer 2000; Jennions and Møller 2002a: 212). One meta-analysis suggests that 33.7% of scientic researchers admit to using at least one questionable research practice (Fanelli 2009); a recent survey in psychology suggests that 94% have done so (John et al. 2012: 527). These practices are now "the steroids of scientic competition, articially enhancing performance and producing a kind of arms race in which researchers who strictly play by the rules are at a competitive disadvantage" (John et al. 2012: 524). Omission of Harms The adequate reporting of harms is essential in order for clinicians to compare the benet versus harm of a treatment. Despite the fact that large trials "amass an enormous amount of information about safety and adverse effects during their conduct" (Ioannidis and Lau 14 CONSORT 2010 added more requirements to increase transparency and accountability in trial design and reporting (Schulz et al. 2010). At this time, there are no systematic reviews on the impact of this intervention. the limited effectiveness of prest ige episteme volume 10–4 395 2001: 443), published reports fail to report adverse events and toxicity adequately: one large review of drug trials from 1967–1999 shows that 71% failed to report toxicity adequately while 61% failed to report clinically adverse events adequately (Ioannidis and Lau 2001).15 Literature reviews have also demonstrated the inadequate reporting of complications in the surgical literature (Martin II et al. 2002). In response to such ndings, CONSORT created the 2004 harms extension with requirements on how harms should be reported (Ioannidis et al. 2004).16 Unfortunately, a study of applicable trials published in top medical journals two years after the harms extension demonstrates that although most reported the existence and rate of adverse events per experimental arm (as required), 78.4% failed to use severity grades for harms and 47.4% failed to provide information about patient withdrawal owing to adverse events (Pitrou et al. 2009: 1759, table 2). These omissions obscure clinically pressing questions about the severity of harms, including the worst cases (Ioannidis 2009). Why was CONSORT 2004 unsuccessful in improving the reporting of harms? The encroaching power of industry sponsors in medicine has provided new incentives and mechanisms for publication bias. A 2007 funding analysis shows that industry accounted for about 58% of biomedical research funding, while NIH accounted for about 27% (Dorsey et al. 2010). Authors have a large incentive to cooperate with sponsor terms, which can include relinquishing full access to data, analyses, and rights to publish (Horton 2004: 8). The fact that sponsors have been known to seek legal action against authors seeking to publish negative or harm-related outcomes creates further incentive for author cooperation (Thompson et al. 2001). Indeed, the role of academic researchers is sometimes so marginalized that they are placed in rst and second author positions only after full manuscripts have been prepared by ghostwriters (Ross et al. 2009). All of these practices enable industry sponsors to hide harm-related results that would block FDA marketing approval or decrease the protability of their drug or device (Psaty and Kornmal 2008; Elliott 2010). And, hide they do: industry-sponsored studies that make it to print are more likely to report better harms outcomes (Ioannidis and Lau 2001: 441; Lundh et al. 2012) and efcacy (Lexchin et al. 2003; Lundh et al. 2012) than studies not funded by industry. The suggestion that industry sponsors publish more selectively is not meant to imply that they do not care for the prestige associated with peer-reviewed publication. It is simply that prestige is only instrumental towards nancial ends: the imprimatur of prominent journals brings an unparalleled form of visibility and credibility to their drugs and devices (Smith 2003: 1203).17 15 In published papers reporting trial results, more surface area is dedicated to reporting author names and afliations than harms (Ioannidis 2009). 16 In the same year, ICMJE specied that editors "may request" copies of author contracts with sponsors and may ask authors to sign statements assuring editors of their full access to data and responsibility for data analyses (ICMJE 2004). ICMJE encouraged authors to invoke this clause when negotiating research contracts with sponsors in the hope that authors would be able to wrest control over trial design, access to data, responsibility for data analysis and interpretation, and rights to publish (ICMJE 2007). Note that ICJME 2004 only grants editors permission to request such information. CONSORT 2010 goes further by requiring disclosure of the sponsor's role in trial design, conduct, analysis, and reporting. However, role-disclosure is still weaker than requiring that authors retain control over trial design, data, analyses, and rights to publish. carole j . lee 396 episteme volume 10–4 5. conclusion The beauty of the traditional system of incentives in science was that one mechanism – peer review – served normative and motivational functions simultaneously: the slogan was that ambitious scientists could do well professionally by doing well epistemically. Under this regime, we would expect that editors should be able to motivate others to raise their standards of performance (Zuckerman and Merton 1971: 66). However, this has not been the case in academic medicine. ICMJE 2005 tried to debias trial data available for meta-analysis by requiring trial registration. While this has increased the rate of trial registration, registration information is too often incomplete and inadequately specied (Mathieu et al. 2009; Viergever and Ghersi 2011) – practices that undermine the purpose of the registration requirement. This demonstrates that the mere articulation of a stricter publication standard is not sufcient. Editors and reviewers, when evaluating manuscripts, must check that trial registration information is complete and adequate. Furthermore, the FDA and National Library of Medicine (NLM) should verify and enforce the completeness and adequacy of registration information (Ross et al. 2009: 7). Proper assessment of the completeness and adequacy of trial protocol might require adding a separate peer review process to the domain of data registration – a practice that has been recommended in other scientic disciplines (Parsons et al. 2010; Lawrence et al. 2011). ICMJE 2005 required the prospective registration of trial protocol in an effort to increase transparency and deter the post hoc selection and gerrymandering of endpoints. However, more than half of published studies outed the purpose of the requirement by failing to register prospectively or failing to articulate a specic primary outcome (Mathieu et al. 2009; Viergever and Ghersi 2011). This result underscores the suggestion that editors and reviewers should compare protocols identied in submitted manuscripts to the original registration record. Furthermore, the FDA, NLM, and reviewers for registration databases (if needed) should verify and enforce the completeness and adequacy of registration data. Finally, CONSORT 2004 was designed to improve the reporting of harms and toxicity in published reports. However, subsequent literature reviews demonstrate the continued inadequacy of harms reporting (Pitrou et al. 2009). Industry sponsorship has provided a new system of incentives and mechanisms for publication bias; and editorial interventions thus far have failed to countenance them. Editors must state much stricter publication requirements that make compulsory author control over trial design, data, analyses, and interpretation as well as rights to publish. Federal agencies should create and enforce regulations designed to grant such control to authors and protect them from litigation by industry sponsors. Federal protections are also needed for medical journals and editors who publish unfavorable trial results (protections are also needed for reviewers who recommend their publication). We may even need to uproot the nancial pull of industry more directly by getting rid of medical patents (Brown 2004, 2008; Reiss and Kitcher 2009; Reiss 2010) and providing full public funding for medical research (Brown 2004, 2008). To combat publication bias in medicine, we must go well 17 Note that editors are not immune from dysfunctional industry-related incentives. Publishing positive results for industry-funded studies makes millions of dollars in reprints for medical journals; and, publishing negative results for industry-funded studies has led to threats of litigation (Smith 2003: 1202). the limited effectiveness of prest ige episteme volume 10–4 397 beyond the prestige of its gate-keepers, the content of its evaluative standards, and the expertise of its members by appealing to increased federal legislation, enforcement, and funding.18 references Allison, D. B., Faith, M. S. and Gorman B. S. 1996. 'Publication Bias in Obesity Treatment Trials?' International Journal of Obesity and Related Metabolic Disorders, 20: 931–7. Altman, D. G. for the CONSORT Group. 2005. 'Endorsement of the CONSORT Statement by High Impact Medical Journals: Survey of Instructions for Authors.' British Medical Journal, 330: 1056–7. Anderson, E. 1995. 'Feminist Epistemology: An Interpretation and Defense.' Hypatia, 10: 50–84. Angell, M. 2004. The Truth about the Drug Companies: How they Deceive us and What to Do about It. New York, NY: Random House. Barbui, C., Hotopf, M. and Garattini, S. 2002. 'Fluoxetine Dose and Outcome in Antidepressant Drug Trials.' European Journal of Clinical Pharmacology, 58: 379–86. Bardy, A. H. 1998. 'Bias in Reporting Clinical Trials.' British Journal of Clinical Pharmacology, 46: 147–50. Begg, C., Cho, M., Eastwood, S. et al. 1996. 'Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement.' Journal of the American Medical Association, 276: 637–9. Berlin, J. A., Begg, C. B. and Louis, T. A. 1989. 'An Assessment of Publication Bias Using a Sample of Published Clinical Trials.' Journal of the American Statistical Association, 84(406): 381–92. Born, G. V. R. and Collins, R. 1997. 'Aspirin versus Clopidogrel: The Wrong Question?' Lancet, 349(9054): 806–7. British Medical Journal. 2011. Welcome to Resources for Authors. http://resources.bmj.com/bmj/ authors. Brown, J. R. 2004. 'Money, Method and Medical Research.' Episteme, 1:49–59. ---. 2008. 'Politics, Method, and Medical Research.' Philosophy of Science, 75: 756–66. Bruijn, J. A. and van der Woude, F. 2002. 'Tacrolimus and Ciclosporin Microemulsion in Renal Transplantation.' Lancet, 360(9335): 800. CAPRIE Steering Committee. 1996. 'A Randomized, Blinded, Trial of Clopidogrel versus Aspirin in Patients at Risk of Aschaemic Events (CAPRIE).' Lancet, 348(9038): 1329–40. Casadevall, A. and Fang, F. C. 2012. 'Reforming Science: Methodological and Cultural Reforms.' Infection and Immunity, 80: 891–6. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C. and Altman, D. G. 2004. 'Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles.' Journal of the American Medical Association, 291: 2457–65. Dickersin, K. and Min, Y-.I. 1993. 'NIH Clinical Trials and Publication Bias.' Online Journal of Current Clinical Trials (Doc. No. 50). Dickersin, K., Min, Y.-I. and Meinert, C. L. 1992. 'Factors inuencing Publication of Research Results: Follow-up of Applications Submitted to Two Institutional Review Boards.' Journal of the American Medical Association, 267: 374–9. Dorsey, E. R., de Roulet, J., Thompson, J. P. et al. 2010. 'Funding of US Biomedical Research 2003– 2008.' Journal of the American Medical Association, 303: 137–43. Douglas, H. 2000. 'Inductive Risk and Values in Science.' Philosophy of Science, 67: 559–79. 18 I would like to thank both the Mellon Foundation and the Woodrow Wilson Foundation for their respective funding and administration of the Career Enhancement Fellowship that afforded research time to complete this article. For comments that sharpened the content of this work, I am indebted to Inmaculada de Melo-Martin, Karen Frost-Arnold, Maria Kronfeldner, Flavia Padovani, Alan Richardson, Miriam Solomon, Kyle Stanford, Jonathan Tsou, and the anonymous reviewers. carole j . lee 398 episteme volume 10–4 Easterbrook, P. J., Berlin, J. A., Gopalan, R. and Matthews, D. R. 1991. 'Publication Bias in Clinical Research.' Lancet, 337(8746): 867–72. Egger, M. and Smith, G. D. 1995. 'Misleading Meta-Analysis: Lessons from "an Effective, Safe, Simple" Intervention that Wasn't. (Study on Treatment of Heart Attacks).' British Medical Journal, 310(6982): 752–4. Elliott, C. 2010. 'Making a Killing.' Mother Jones, 35(5):54–63. Fanelli, D. 2009. 'How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data.' PLoS One, 4(5):1–11. Fang, F. C. and Casadevall, A. 2011. 'Retracted Science and the Retraction Index.' Infection and Immunity, 79: 3855–9. Fontanarosa, P. B. and DeAngelis, C. D. 2004. 'Information for JAMA Authors – 2004.' Journal of the American Medical Association, 291:107–8. Garattini, S., Bertele, V. and Bassi, L. L. 2003. 'How Can Research Ethics Committees Protect Patients Better?' British Medical Journal, 326: 1199–201. Grady, D., Herrington, D., Bittner, V. et al. 2002. 'Cardiovascular Disease Outcomes During 6.8 Years of Hormone Therapy: Heart and Estrogen/Progestin Replacement Study Follow-up (HERSII).' Journal of the American Medical Association, 288: 49–57. Grieneisen, M. L. and Zhang, M. 2012. 'A Comprehensive Survey of Retracted Articles from the Scholarly Literature.' PLoS One 7 (10). doi: 10.1371/journal.pone.0044118. Horton, R. 2004. 'Review: The Dawn of McScience.' The New York Review of Books, March 11: 7–9. Hulley, S., Grady, D., Bush, T. et al. 1998. 'Randomized Trial of Estrogen Plus Progestin for Secondary Prevention of Coronary Heart Diease in Postmenopausal Women.' Journal of the American Medical Association 280: 605–13. ICMJE. 'Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication [2004].' Accessed March 1, 2013. http://www.icmje.org/ 2004_urm.pdf. ---. 'Announcements, Updates & Editorials: Sponsorship, Authorship, and Accountability [2007].' Accessed March 1, 2013. http://www.icmje.org/update_sponsor.html. Intemann, K. and de Melo-Martin, I. 2010. 'Social Values and Scientic Evidence: The Case of the HPV Vaccines.' Biology and Philosophy, 25: 203–13. Ioannidis, J. P. A. 1998. 'Effect of the Statistical Signicance of Results on the Time to Completion and Publication of Randomized Efcacy Trials.' Journal of the American Medical Association, 279: 281–6. ---. 2005a. 'Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.' Journal of the American Medical Association, 294: 218–28. ---. 2005b. 'Why Most Published Research Findings are False.' PLoS Medicine, 2: 696–701. ---. 2009. 'Adverse Events in Randomized Trials: Neglected, Restricted, Distorted, and Silenced.' Archives of Internal Medicine, 169: 1737–9. Ioannidis, J. P. A., Evans, S. J. W., Gøtzsche, P. C. et al. 2004. 'Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement.' Annals of Internal Medicine, 141: 781–8. Ioannidis, J. P. A. and Lau, J. 2001. 'Completeness of Safety Reporting in Randomized Trials: An Evaluation of Medical Areas.' Journal of the American Medical Association 285: 437–43. Jennions, M. D. and Møller, A. P. 2002a. 'Publication Bias in Ecology and Evolution: An Empirical Assessment Using the 'Trim and Fill' Method.' Biological Reviews, 77: 211–22. ---. 2002b. 'Relationships Fade with Time: A Meta-Analysis of Temporal Trends in Publication in Ecology and Evolution.' Proceedings: Biological Sciences, 269(1486): 43–8. Johansen, H. K. and Gøtzsche, P. C. 1999. 'Problems in the Design and Reporting of Trials of Antifungal Agents Encountered During Meta-Analysis.' Journal of the American Medical Association, 282(18): 1752–9. John, L. K., Loewenstein, G. and Prelec, D. 2012. 'Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.' Perspectives on Psychological Science 23: 524–32. Kane, R., Wang, J. and Garrard, J. 2007. 'Reporting in Randomized Clinical Trials Improved after Adoption of the CONSORT Statement.' Journal of Clinical Epidemiology 60: 241–9. the limited effectiveness of prest ige episteme volume 10–4 399 Kirsch, I., Moore, T. J., Scoboria, A. and Nicholls, S. S. 2002. 'The Emperor's New Drugs: Effect Size and Moderation Effects.' Prevention and Treatment, 5(1): Art. 23. Kitcher, P. 2001. Science, Truth, and Democracy. New York, NY: Oxford University Press. Kravitz, R. L. and Feldman, M. D. 2011. 'From the Editor's Desk: Self-Plagiarism and Other Editorial Crimes and Misdemeanors.' Journal of General Internal Medicine, 26: 1. Lawrence, B., Jones, C., Matthews, B., Pepler, S. and Callaghan, S. 2011. 'Citation and Peer Review of Data: Moving towards Formal Data Publication.' International Journal of Digital Curation, 6(2): 4–37. Lee, C. J. 2012. 'A Kuhnian Critique of Psychometric Research on Peer Review.' Philosophy of Science, 79: 859–70. Lee, C. J. and Schunn, C. D. 2010. 'Philosophy Journal Practices and Opportunities for Bias.' APA Newsletters: Newsletter on Feminism and Philosophy, 10(1): 5–10. ---. 2011. 'Social Biases and Solutions for Procedural Objectivity.' Hypatia: A Journal of Feminist Philosophy, 26: 352–73. Lee, C. J., Sugimoto, C. R., Zhang, G. and Cronin, B. 2013. 'Bias in Peer Review.' Journal of the American Society for Information Science, 64: 2–17. Lexchin, J., Bero, L. A., Djulbegovic, B. and Clark, O. 2003. 'Pharmaceutical Industry Sponsorship and Research Outcome and Quality: A Systematic Review.' British Medical Journal, 326: 1167–76. Longino, H. E. 1990. Science as Social Knowledge. Princeton, NJ: Princeton University Press. Lundh, A., Sismondo, S., Busuioc, O. A. and Bero, L. 2012. 'Industry Sponsorship and Research Outcome (Review).' Cochrane Database of Systematic Reviews (12). doi: 10.1002/14651858. MR000033.pub2. Martin II, R. C. G., Brennan, M. F. and Jaques, D. P. 2002. 'Quality of Complication Reporting in the Surgical Literature.' Annals of Surgery, 235: 803–13. Mathieu, S., Boutron, I., Moher, D., Altman, D. G. and Ravaud, P. 2009. 'Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials.' Journal of the American Medical Association, 302: 977–84. Melander, H., Ahlqvist-Rastad, J., Meijer, G. and Beermann, B. 2003. 'Evidence B(i)ased Medicine – Selective Reporting from Studies Sponsored by Pharmaceutical Industry: Review of Studies in New Drug Applications.' British Medical Journal, 326: 1171–5. Merton, R. K. 1973. 'The Ambivalence of Scientists (1963).' In N. W. Storer (ed.), The Sociology of Science: Theoretical and Empirical Investigations, pp. 383–412. Chicago, IL: The University of Chicago Press. Merton, R. K. and Zuckerman, H. 1971. 'Institutional Patterns of Evaluation in Science (1971).' In N. W. Storer (ed.), The Sociology of Science: Theoretical and Empirical Investigations, pp. 460–96. Chicago, IL: The University of Chicago Press. Moher, D., Jones, A. and Lepage, L. 2001. 'Use of the CONSORT Statement and Quality of Reports of Randomized Trials: A Comparative Before-and-After Evaluation.' Journal of the American Medical Association, 285: 1992–5. Neuliep, J. W. and Crandall, R. 1990. 'Editorial Bias against Replication Research.' Journal of Social Behavior and Personality, 5(4): 85–90. ---. 1993. 'Reviewer Bias against Replication Research.' Journal of Social Behavior and Personality, 8(6): 21–9. Oldenberg, H. 1966. Correspondence of Henry Oldenberg. Trans. by A. Rupert Hall and Marie Boas Hall. Vol. 2. Madison, WI: University of Wisconsin Press. Olson, C. M., Rennie, D., Cook, D. et al. 2002. 'Publication Bias in Editorial Decision Making.' Journal of the American Medical Association, 287: 2825–8. Palmer, A. R. 2000. 'Quasireplication and the Contract of Error: Lessons from Sex Ratios, Heritabilities and Fluctuating Asymmetry.' Annual Review of Ecology and Systematics, 31: 441–80. Parsons, M. A., Duerr, R. and Minster, J.-B. 2010. 'Data Citation and Peer Review.' Eos, Transactions, American Geophysical Union, 91 (34): 297–8. Pipkin, F. B., Crowther, C., de Swiet, M. et al. 1996. 'Where Next for Prophylaxis against Pre-Eclampsia?' British Journal of Obstetrics and Gynaecology, 103: 603–7. carole j . lee 400 episteme volume 10–4 Pitrou, I., Boutron, I., Ahmad, N. and Ravaud, P. 2009. 'Reporting of Safety Results in Published Reports of Randomized Controlled Trials.' Archives of Internal Medicine, 169: 1756–61. Plint, A. C., Moher, D., Morrison, A. et al. 2006. 'Does the CONSORT Checklist Improve the Quality of Reports of Randomised Controlled Trials? A Systematic Review.' Medical Journal of Australia, 185: 263–7. Prayle, A. P., Hurley, M. N. and Smyth, A. R. 2012. 'Compliance with Mandatory Reporting of Clinical Trial Results on ClinicalTrials.gov: Cross sectional Study.' British Medical Journal, 344: d7373. Psaty, B. M. and Kornmal, R. A. 2008. 'Reporting Mortality Findings in Trials of Rofecoxib for Alzheimer Disease or Cognitive Impairment: A Case Study Based on Documents from Rofecoxib Litigation.' Journal of the American Medical Association, 299: 1813–7. Reiss, J. 2010. 'In favour of a Millian Proposal to Reform Biomedical Research.' Synthese, 177: 427– 47. Reiss, J. and Kitcher, P. 2009. 'Biomedical Research, Neglected Diseases, and Well-Ordered Science.' Theoria, 66: 263–82. Rochon, P. A., Gurwitz, J. H., Simms, R. W. et al. 1994. 'A Study of Manufacturer-Supported Trials of Nonsteroidal Anti-Inammatory Drugs in the Treatment of Arthritis.' Archives of Internal Medicine, 154: 157–63. Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E. and Krumholz, H. M. 2009. 'Trial Publication after Registration in ClinicalTrials.Gov: A cross-sectional analysis.' PLoS Medicine, 6(9): 1–9. Rothman, K. J. 1986. 'Signicance Questing.' Annals of Internal Medicine, 105: 445–7. Rudner, R. 1953. 'The Scientist Qua Scientist Makes Value Judgments.' Philosophy of Science, 20(1): 1–6. Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W. and Haynes, R. B. 2000. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd edn. Oxford: Churchill Livingstone. Safer, D. J. 2002. 'Design and Reporting Modications in Industry-Sponsored Comparative Psychopharmacology Trials.' Journal of Nervous and Mental Disease, 190: 583–92. Schulz, K. F., Altman, D. G., Moher, D. and Fergusson, D. 2010. 'CONSORT 2010 Changes and Testing Blindness in RCTs.' Lancet, 375: 1144–6. Sekeres, M., Gold, J. L., Chan, A.-W. et al. 2008. 'Poor Reporting of Scientic Leadership Information in Clinical Trial Registers.' PLoS One, 3(2): 1–6. Simes, R. J. 1986. 'Publication Bias: The Case for an International Registry of Clinical Trials.' Journal of Clinical Oncology, 4: 1529–41. Simpson, G. M., and Lindenmayer, J.-P. 1997. 'Extrapyramidal Symptoms in Patients Treated with Risperidone.' Journal of Clinical Psychopharmacology, 17(3): 194–201. Smith, R. 2003. 'Medical Journals and Pharmaceutical Companies: Uneasy Bedfellows.' British Medical Journal, 326: 1202–5. Snapinn, S. M. 2000. 'Noninferiority Trials.' Current Controlled Trials in Cardiovascular Medicine 1: 19–21. Solomon, M. and Richardson, A. 2005. 'A Critical Context for Longino's Critical Contextual Empiricism.' Studies in the History and Philosophy of Science, 36: 211–22. Stern, J. M. 1997. 'Publication Bias: Evidence of Delayed Publication in a Cohort Study of Clinical Research Projects.' British Medical Journal, 315 (7109): 640–5. The Lancet. 2011. Types of Article and Manuscript Requirements. http://www.thelancet.com/ lancet-information-for-authors/article-types-manuscript-requirements. Thompson, J., Baird, P. and Downie, J. 2001. The Olivieri Report: The Complete Text of the Report of the Independent Inquiry Commissioned by the Canadian Association of University Teachers. Toronto: James Lorimer & Company. Tramèr, M. R., Reynolds, D. J., Moore, R. A. and McQuay, H. J. 1997. 'Impact of Covert Duplicate Publication on Meta-Analysis: A Case Study.' British Medical Journal, 315(7109): 635–40. Turner, E. H., Matthews, A. M., Linardators, E., Tell, R. A. and Rosenthal, R. 2008. 'Selective Publication of Antidepressant Trials and Its Inuence on Apparent Efcacy.' New England Journal of Medicine, 358: 252–60. Viergever, R. F. and Ghersi, D. 2011. 'The Quality of Registration of Clinical Trials.' PLoS One, 6 (2): 1–8. the limited effectiveness of prest ige episteme volume 10–4 401 Weber, E. J., Callaham, M. L., Wears, R. L., Barton, C. and Young, G. 1998. 'Unpublished Research From a Medical Specialty Meeting: Why Investigators Fail to Publish.' Journal of the American Medical Association, 280: 257–9. Weller, A. C. 2001. Editorial Peer Review: Its Strengths and Weaknesses. Medford, NJ: American Society for Information Science and Technology. Zarin, D. A., Ide, N. C., Tse, T. et al. 2007. 'Issues in the Registration of Clinical Trials.' Journal of the American Medical Association, 297: 2112–20. Ziman, J. 1969. 'Information, Communication, Knowledge.' Nature 224: 318–24. Zuckerman, H. and Merton, R. K. 1971. 'Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System.' Minerva 9: 66–100. CAROLE J. LEE is an Assistant Professor at the University of Washington, Seattle. Her research focuses on methodological assumptions and normative implications of empirical research on judgment bias, with a recent focus on bias in peer review. Her work is currently supported by a Career Enhancement Fellowship, which is funded by the Mellon Foundation and administered by the Woodrow Wilson Foundation. carole j . lee 402 episteme volume 10–