The International Journal of Person Centered Medicine 295 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 ARTICLE The Real World Failure of Evidence-Based Medicine Clifford G. Miller BSc ARCSa and Donald W. Miller, Jr., MDb a Solicitor, Supreme Court of England & Wales and former Lecturer in Law, Imperial College, London, UK b Professor of Surgery, Division of Cardiothoracic Surgery, University of Washington School of Medicine, Seattle, Washington, USA. Abstract As a way to make medical decisions, Evidence-Based Medicine (EBM) has failed. EBM's failure arises from not being founded on real-world decision-making. EBM aspires to a scientific standard for the best way to treat a disease and determine its cause, but it fails to recognise that the scientific method is inapplicable to medical and other real-world decision-making. EBM also wrongly assumes that evidence can be marshaled and applied according to an hierarchy that is determined in an argument by authority to the method by which it has been obtained. If EBM had valid theoretical, practical or empirical foundations, there would be no hierarchy of evidence. In all real-world decision-making, evidence stands or falls on its inherent reliability. This has to be and can only be assessed on a case-by-case basis applying understanding and wisdom against the background of all available facts-the "factual matrix." EBM's failure is structural and was inevitable from its inception. EBM confuses the inherent reliability and probative value of evidence with the means by which it is obtained. EBM is therefore an ad hoc construct and is not a valid basis for medical decision-making. This is further demonstrated by its exclusion of relevant scientific and probative real-world decision-making evidence and processes. It draws upon a narrow evidence base that is itself inherently unreliable. It fails to take adequate account of the nature of causation, the full range of evidence relevant to its determination, and differing approaches to determining cause and effect in real-world decision-making. EBM also makes a muddled attempt to emulate the scientific method and it does not acknowledge the role of experience, understanding and wisdom in making medical decisions. Keywords Anecdote, causation, evidence-based medicine, hierarchy of evidence, individual patients law, medical decision-making, randomised controlled trials, science, scientific theory Correspondence Address Mr. Clifford G. Miller, Burnhill House, 50 Burnhill Road, Beckenham, Kent BR3 3LA, England. UK. E-mail: cgmiller@cliffordmiller.com Accepted for publication: 9 June 2011 Introduction Evidence-Based Medicine aspires to a scientific standard for the best way to treat a disease and determine its cause. An outgrowth of clinical epidemiology, it utilizes statistical trials to guide individual patient care [1]. Advocates of statistics-based research coined the term "Evidence-Based Medicine" in 1992. We examine here the shortcomings of EBM and the limited role it plays in answering this 'real world' question: "In deciding which treatment modality I, the physician, should adopt for the patient before me today, what evidence should I use?" Medical decision-making EBM is proposed to remedy what are considered failings in medicine, such as the wide variations in clinical practice, the use of unproven treatments and failures to utilise practice guidelines. EBM strives to use the "best available evidence" in making medical decisions and especially evidence that is obtained through epidemiological and biostatistical ways of thinking. It considers randomized controlled trials (RCTs) and systematic reviews of multiple RCTs to be the most reliable evidence. Its advocates argue that this is the kind of evidence physicians need to use to make decisions, in a scientific way, about the care of individual patients. Proponents accord such evidence an "A", because they say it is scientifically obtained and of a kind least susceptible to research and other biases. Miller and Miller The Real World Failure of Evidence-Based Medicine 296 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 The Oxford Centre for Evidence-based Medicine defines ten Levels of Evidence, which it grades "A" (the least biased) through "D" (the most biased). It gives an "A" to RCTs and systematic reviews of those trials (metaanalyses). The Levels of Evidence table downplays cohort and case-control studies, giving them a "B." Clinical case series and "poor quality cohort and case-control studies" are consigned a "C." [2]. It gives a "D" to "Expert opinion without explicit critical appraisal, or based on physiology, bench research or 'first principles' ". In so doing, EBM diminishes important evidence that physicians use to make medical decisions. Notable are (unquantifiable) clinical experience and an (also unquantifiable) understanding of pathophysiologic mechanisms of disease. EBM calls evidence based on 30-40 years of clinical experience that a physician uses to make a medical decision "anecdotal" and not sufficiently reliable for inclusion. In a comprehensive, well researched and detailed critique of EBM's reliance on RCTs and observational statistical studies reviewed within this issue of the Journal, James Penston, in Stats.con: How we've been fooled by statistics-based research in medicine, documents how statistics-based research is fundamentally flawed, resistant to independent testing and lacks sound foundations. Penston's main objection to statistics-based medicine is that modern large-scale statistical studies claiming treatment effects in small sub-groups are inherently unreliable, for a wide variety of fundamental reasons. Criticisms of the internal validity of RCTs and their basis for causal inference remain unresolved. External validity, the reliability of generalization to the wider population, is always open to question and deliberate concealment of the size of treatment effects by some researchers can occur [3]. Noting that statistical data apply to groups, not to individuals, Penston cites R.A. Fisher, who early in the 20th century wrote, "Statistics is the study of populations, or aggregates of individuals, rather than of individuals." For 'Mrs Smith', with her specific set of problems, EBM will not directly enable the doctor to make the correct medical decision for her treatment needs. Strikingly, it has been estimated that 127 RCTs would have to be done in 63,500 patients over a 286-year period to determine the optimal combination of agents to treat Alzheimer's disease [4]. Cohen and colleagues [1] address five widely cited criticisms of EBM. They are: 1) its reliance on empiricism, 2) its narrow definition of evidence, 3) the lack of evidence for EBM's efficacy, 4) its limited usefulness for individual patients, and 5) the threats EBM poses to the autonomy of the doctor/patient relationship. EBM tries to apply principles of epidemiology to individual patient care, basing that care on statistical trials. This narrow definition of evidence excludes other information important to clinicians. And EBM fails to provide a means to integrate non-statistical forms of medical information, like clinical experience and patient-specific factors. Furthermore, the authors of this critique note that there is no evidence that EBM is indeed effective in providing higher quality health care. Ackoff posits a now well-known model in information science, proceeding from data to information to knowledge to understanding and finally wisdom [5]. Data consists of raw facts and figures. It has no significance beyond its existence, in and of itself, absent of any relation to other things. Information is data that acquires meaning, useful or not, from its connection to other things. Knowledge is what you know. Understanding is the process by which new knowledge can be derived from existing knowledge. Wisdom is the ability to perceive outcomes and determine their value and to increase effectiveness of existing knowledge and understanding. EBM provides information on statistical relationships. Its data and information offers knowledge, but EBM does not deliver understanding of the particular disease or its pathophysiology, nor does it provide wisdom. As Penston shows, the knowledge that EBM offers is often wrong and even if right, frequently inconsequential in everyday clinical practice. Real-world decision-making evaluates and utilises the available factual matrix, which includes clinical experience, case reports and case series and understanding the pathophysiology of the disease. Causation People marshal evidence to support current beliefs and to establish causation. EBM endorses the hypothesis that evidence of a statistically significant outcome in a randomised controlled trial justifies causal inference. The transition from statistical significance to causation in clinical epidemiology began in the late 1950s and was codified with the publication of the U.S. Surgeon General's report on smoking and health in 1964 [6]. Cause, as the report defines it, expresses "the notion of a significant, effectual, relationship between an agent and an associated disorder or disease in the host." Recognizing that "the causal significance of an association is a matter of judgment which goes beyond any statement of statistical probability," the writers of the report list five epidemiological criteria with which to judge causation. In examining the association of smoking with lung cancer, they judged that smoking causes lung cancer because of the association's Consistency – the two occur together in different populations and locations, and at different times, favouring a causal relationship; its Strength – there is an exceptionally high relative risk linking smoking to lung cancer; Specificity – the accuracy with which the occurrence of the cause (smoking) predicts the occurrence of the effect (lung cancer); Temporal relationship – the cause precedes the effect; and Coherence, which refers to the degree a causal relationship (smoking and lung cancer) may be accommodated with pre-existing knowledge and theory. Bradford Hill [7] and others [8,9] have put forth additional epidemiological criteria of disease causation, which include a Biological gradient – a dose-response The International Journal of Person Centered Medicine 297 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 curve, where, with smoking, the death rate from lung cancer rises linearly with the number of cigarettes smoked; Plausibility – the causation is biologically plausible; Experiments that confirm the epidemiological findings; and Analogy, in cases where it would be reasonable to judge the association by analogy. With EBM today, the statistical strength of an association alone has become the sole arbiter of a causal relationship. Researchers pay little heed to other nonquantifiable epidemiological criteria for judging causation and have lost sight of the fact that it is the strength of association taken in conjunction with all the other criteria that justifies a causal inference. In promoting statistics-based research, EBM has divorced itself from real-world common sense and the scientific assessment of causation. Statistical research, as Penston puts it, "has slipped its mooring and floats free, unfettered by the constraints normally imposed on causal judgments." In modern statistics-based medicine, Penston notes that causal inference is based on small differences in the frequency of occurrence and non-concurrence of an effect with and without an intervention. RCTs and observational statistical studies are also applied to heterogeneous, not homogeneous, groups, where there will be considerable variation between participants which is not always subject to proper control during a trial. Assessment of causation cannot therefore be scientific. RCTs wrongly ascribe a causal inference for a specific drug when no treatment effect is demonstrable with that drug for the majority of participants, only perhaps for a small subgroup in the heterogeneous group of trial participants [3]. A particular strain of bacteria can cause a specific disease; but a single factor is rarely the cause of chronic diseases like obesity, diabetes, heart disease, and cancer. Several factors, temporally related, cause these diseases. Researchers implicated the bacteria Helicobacter pylori as "causing" gastritis and peptic ulcer disease. As much as 70 to 85 percent of some populations have these bacteria in their stomachs and they remain asymptomatic [10]. Epidemiology played no part in the establishing the causal association which was previously long denied. Additionally such studies cannot establish the absence of a causal association. In papers that claim their negative results "do not support" a causal association between factor X and effect Y, one must conclude that the authors do not understand the fundamental issues involved in the establishment of cause and effect within biological systems. Penston tacitly avoids attempting to demark a boundary for the reliability of statististical studies between those where relative risks/benefits and odds ratios are low and where they are higher, (say 30 or more such as for an association between heavy smoking and lung cancer). Aside from the simplistic approach of statistics-based medicine to assessment of causation, one justification is that, as Penston demonstrates, internal validity of RCTs is rarely met by randomization, allocation concealment, double-blinding, accounting for withdrawals and dropouts. This thus renders questionable the reliability of claimed results and tests of statistical confidence and significance [3]. An overarching reason for not doing so reflects a principal failing of the foundation of EvidenceBased Medicine. EBM wrongly posits that evidence can be ascribed a level of inherent reliability according to an hierarchy established by appeal to the assumed authority of the method by which the evidence is obtained. In real world decision-making, items of evidence stand or fall on their own inherent reliability. Establishing reliability of evidence requires examination and assessment of it on a case-by-case basis against the background of the available factual matrix for the decision under consideration [11]. EBM also advises medical practitioners to turn to pertinent journal papers when considering treatment options. But in a significant contradiction of their position, EBM proponents acknowledge that the overall quality of the modern journal published evidence base is poor. Leading journal editors also hold the same view [12,13]. It has been suggested that "we should abandon pre publication review and paying excessive attention to 'top journals'" [14]. EBM compounds its 'real world' failing in judging causation by excluding and relegating to a lower level other probative forms of evidence. Its advocates talk down evidence from case series and case reports, contending that such evidence is observational and not collected in a "scientific" manner. They label it "anecdotal." Anecdotal evidence covers all human observations, scientific and otherwise. It is the personal observation of the observer. Witness testimony in court is "anecdotal." But when subjected to scrutiny and in the courtroom to crossexamination, anecdotal evidence can be as reliable or unreliable as any other form of evidence. Anecdotal eyewitness testimony can be tested and weighed according to its inherent reliability and cross-correlated and corroborated by other evidence in the available "factual matrix". In order to prove that A causes B, as is done in determining guilt in a Court of Law, the medical practitioner must address the available factual matrix bearing on the case. In medicine, this matrix might encompass, in addition to the patient's account and clinical history, RCTs and systematic reviews, cohort and casecontrol studies, case series and case reports, clinical experience, expert opinion, in vitro (test tube) research, animal "bench" research and deductive reasoning based on pathophysiologic mechanisms of disease. Causation, established by evidence collected from an often wideranging factual matrix is tied to individual events in the natural world, not to small statistical differences in heterogeneous groups. Scientific method Humans apply different standards of "proof" depending upon the degree of certainty a decision requires [11]. The Miller and Miller The Real World Failure of Evidence-Based Medicine 298 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 highest standard, requiring the best evidence, is the scientific standard of proof. It must be Irrefutable. Successively lower criminal and civil standards of proof are Beyond a reasonable doubt – the criminal standard for life and liberty cases; Clear and convincing (in U.S. courts) – a civil judgment that implies, "If that's what it is, I'd better believe it," for settling disputes involving child custody, involuntary commitment, withdrawal of life support in comatose patients, and determination of a "punishable frame of mind" driven by malice, oppression, or fraud; and More likely than not – for a civil judgment requiring that the balance of probability be greater than 50 percent to support, or undermine, a disputed proposition. In order to obtain a licence for public use, the U.S. Food and Drug Administration (FDA) subjected the pharmaceutical Vioxx (rofecoxib), as it does to all new prescription drugs, to rigorous RCTs to prove that the drug is both safe and effective [15]. The subsequent debacle of Vioxx, with its belatedly-found increased risk of cardiovascular complications, is just one example among many demonstrating that EBM's RCT-derived evidence is not reliable. RCT evidence sometimes does not even meet the civil More likely than not standard of proof, particularly where multiple RCTs on a given subject provide conflicting evidence. Physicians cannot rest assured that RCT evidence demonstrating that a newly licensed drug is safe and effective is in fact true. There is another aspect of the scientific standard of proof that particularly impacts medicine. Science's quest for objective certainty admits only a narrow range of evidence. The "pure" sciences such as physics and chemistry have made the word "science" mean 'creating reliable knowledge and understanding to a high degree of confidence'. Other disciplines have attempted to emulate that success and have borrowed the same word to describe what they do. But the "soft" sciences, such as economics and sociology, and the practice of medicine, are inherently incapable of achieving the same scientific success. A more fundamental issue is that the scientific method is inapplicable for addressing broadly based questions of immediate relevance in day-to-day medical decisionmaking. "Science" and "scientific" are terms used commonly but inappropriately in medicine to imply, by association, an unjustified imprimatur of "scientific" validity or authority and particularly by reference to papers published in peer-reviewed journals. The success of the scientific method in the pure sciences is contingent on formulating narrow, closely defined questions, or hypotheses, which are tested by experiments designed to eliminate and/or control all variables other than that being tested. The hypotheses posed are those that are capable of yielding precise, often numerical, answers. In medicine especially, precise and narrow questions cannot usually be formulated against which single variables can be tested, so the ability to obtain knowledge to a high degree of precision is substantially diminished. One form of medical evidence that does approach the scientific standard of proof is a special kind of case report, a source of evidence that EBM demeans. In pharmacology, the most powerful proof of causation is a single welldocumented challenge-dechallenge-rechallenge case report that shows event A caused adverse event B. Unlike evidence from RCTs and systematic reviews, the evidence from a single well documented spontaneous report of a Challenge-Dechallenge-Rechallenge (CDR) case is often the strongest form of proof of causation if not irrefutable [11] . A drug is administered (the challenge) and adverse symptoms arise. Then the drug is withdrawn and the symptoms abate (dechallenge). If the offending drug is readministered (rechallenge) after the adverse symptoms have subsided and the same symptoms recur, this is normally proof that the drug caused the adverse symptoms [2]. Lacking a rechallenge to test the hypothesis, one cannot be sure that withdrawing the drug is related to the drop in symptoms in a single patient; but the drug is highly likely to be the cause of the symptoms if, on withdrawing it, symptoms abate in three patients. Three well documented case series reports of challenge-dechallenge evidence is of equal power to CDR in proving causation. Scientific evidence, then, plays a very small role, other than with CDR case reports, in the practice of medicine. CDR case reports are particular examples of the general evidential value of case series and the power of crosscorrelation and corroboration. The probative value of an isolated spontaneous report may seem weak. Multiple independent such reports of the same effect cross-correlate and corroborate, providing potentially powerful evidence of cause and effect. Under EBM, the potentially strong probative value of case series is significantly undervalued. Medical failure and political success EBM has failed in the real world of medicine, in terms of its use in making medical decisions and in proving causality. It has been successful politically. Health service managers, public health professionals, biostatisticians, health economists and politicians continue to prosper using the statistics that EBM provides, acting to rein in the purported dangers of passive smoking and low-dose radiation and to promote the claimed benefits of low fat diets, statins and influenza immunization. As Charlton and Miles have previously observed, "EBM stands revealed as statistical rather than scientific; its success more to do with managerial dominance than medical desirability." [16]. The word 'statistics' originates from the Latin statisticus, meaning "matter of state." There is a close relationship between statistics and politics. Politicians have learned how to massage and rework statistics to their advantage, as they often do when reporting unemployment figures and inflation rates. George Orwell notes that notions of truth and falsity do not apply to the realm of statistics. They are an instrument the State uses to control The International Journal of Person Centered Medicine 299 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 our thoughts and actions – and to dictate health care. In the real world of 21st century medicine, physicians would do well to reconsider and reassign the position of statisticsbased EBM in medical decision-making to a lesser prominence. Should it even be consigned to the wastebasket of medical mistakes, alongside the bleeding of people in the 17th and 18th centuries and treating syphilis with mercury in the 19th and early 20th centuries? Real-World Evidence-Based Medical Decision Making (EBMDM) As Osler so well put it, "Medicine is art of probabilities, or at best, a science of uncertainty" [17]. Medicine is a pragmatic, probabilistic activity, where decisions must be made on the basis of available evidence and within a limited time frame. In the practice of medicine, attempting to employ only a narrowly focussed scientific standard of proof is inappropriate and impossible [11]. The kind of evidence that the current model of "Evidence-Based Medicine" provides is knowledge focussed on statistical knowledge. EBM does not provide understanding, nor promotes wisdom. To attain understanding of a disease and its treatment and possible prevention a physician needs to take an Oslerian approach and embrace the available factual matrix pertaining to it. Health care providers have been led to believe that the best evidence for making medical decisions comes from randomised controlled trials. But in the real world, one must consider RCTs, including those published in leading peer reviewed medical journals, with a healthy scepticism. One must also firstly determine who funded the study. The pharmaceutical and biotechnology industries fund more than 60 percent of the RCTs that medical journals publish, which raises the concern that supposedly objective science is being turned into a marketing tool [18]. As Penston and we show, RCTs are not reliable and tainted by bias. One observer puts it this way, "Epidemiological analysis is notoriously susceptible to misinterpretation, and even manipulation. Two sets of researchers can extract diametrically opposed results from the same data." [19]. In making 'real world' medical decisions, even if an RCT is scientifically conducted, accurate, and unbiased, relative risk reduction percentages can be misleading. Physicians must look for the absolute risk reduction and the number-needed-to-treat (NNT). An RCT on immunisation against influenza during pregnancy to prevent influenza in infants published in the New England Journal of Medicine found that the incidence of influenza in infants whose mothers were immunized during their pregnancy was 4% (6/159). The incidence of influenza in infants whose mothers were not immunized was 10% (16/157). In this study (done in Bangladesh and funded by the Bill and Melinda Gates Foundation, Wyeth Pharmaceuticals and others), influenza immunization reduced the relative risk of influenza in infants by a seemingly impressive 63%. But only 6 out of 100 infants benefited from the immunization. One would have to subject 100 infants in utero to the immunization, the "number-needed-to-treat", to benefit just 6 of them [20]. The other 94 received no benefit, 4 contracted influenza and all were placed at risk from being harmed by the vaccine, particularly from the adjuvants in it, which include mercury, aluminium, and formaldehyde. Strikingly, in a recent Cochrane Collaboration systematic review of influenza vaccines, it was concluded that influenza vaccines in adults have only a modest effect. There was "no evidence that they affect complications, such as pneumonia, or transmission." Reliable evidence was "thin" and there was evidence of widespread manipulation of conclusions and spurious interpretation of the studies reviewed. These included 15 out of 36 trials funded by Industry and 4 that had no source of funding declaration [21]. In the five major studies of statin drugs, the Scandinavian Simvastatin Survival Study (4S), the Cholesterol and Recurrent Events Trial (CARE), the Longterm Intervention with Pravastatin in Ischemic Disease (LIPID), the West of Scotland Coronary Prevention Study (WOSCOPS), and the Air Force Coronary Atherosclerosis Prevention Study (AFCAPS), the end points of total mortality and non-fatal myocardial infarction had a relative risk reduction of 19% and 29% respectively. But the absolute risk reduction of total mortality was a trivial 1.6% and that of non-fatal myocardial infarct, a not-much-better 3% [22]. Less than 5% of patients placed on long-term treatment with statins to lower cholesterol and prevent coronary artery disease, will derive any benefit whatsoever from them. The same principle applies to other drug trials. When 'Mrs. Smith' comes to the office complaining of leg cramps and lapses in memory, the best real-world decision for her is to stop her statins and prescribe Coenzyme Q 10. Researchers know that the best way to get their study published is to avoid dissent from orthodoxy. Peer reviewers, whose expertise and status can too often be tied to the prevailing view on a given subject, are unlikely to welcome challenge to it. It is a system inherently biased to supporting prevailing views. As one observer puts it, "peer review outlaws paradigm change." State-sanctioned paradigms in the biomedical sciences that have become unquestionable dogma include: a) cholesterol and saturated fats cause coronary artery disease; b) mutations in genes cause cancer; c) a virus called HIV (human immunodeficiency virus) causes AIDS (acquired immune deficiency syndrome); d) the damaging effects of toxins are dose-dependent in a linear fashion down to zero, where even a tiny amount of a toxin, such as radiation or cigarette smoke, will harm some people (the linear no-threshold hypothesis); and e) vaccines are safe and effective. In the real-world of medical decision making, each one of these dogmas deserves scrutiny [23]. Relying only on PubMedlisted journals to examine them and to frame medical decisions will not suffice. The U.S. government's PubMed, for example, excludes Fluoride, the Quarterly Journal of Miller and Miller The Real World Failure of Evidence-Based Medicine 300 The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300 the International Society for Fluoride Research. It publishes papers that question government-approved and sponsored fluoridation of public water supplies. PubMed also excludes the Journal of American Physicians and Surgeons. This journal publishes papers that question the accepted wisdom on cholesterol, HIV/AIDS, low-dose radiation and vaccines. Conclusion Real-world EBMDM needs to embrace an inclusive factual matrix. It should consider alternative viewpoints and question majority consensus, accepted biomedical paradigms. Indeed, it should include various types of observational studies, the clinical experience of a 70-year old physician, bench research, (critically considered) expert opinion, and deductive reasoning grounded on pathophysiologic mechanisms of disease and the patient's account and clinical history. This is real world evidencebased medicine. References [1]. Cohen,A.M., Stavri, P.Z., Hersh, W.R. (2004). A categorization and analysis of the criticisms of EvidenceBased Medicine. International Journal of Medical Informatics 73, 35-43. [2]. Oxford Centre for Evidence-Based Medicine – Levels of Evidence (2009, March) Available at: http://www.cebm.net/index.aspx?o=1025 Accessed 7 February 2011. [3]. Penston, J. (2010). Stats.con: How we've been fooled by statistics-based research in medicine. UK: The London Press. [4]. Saver, J.L. and Kalafut, M. (2001). Combination therapies and the theoretical limits of evidence-based medicine. Neuroepidemiology 20, 57-64. [5]. Ackoff, R.L. (1989). From Data to Wisdom. Journal of Applied Systems Analysis 16, 3-9. [6]. Surgeon General's Advisory Committee on Smoking and Health: Smoking and Health (1964). United States Department of Health, Education and Welfare, Public Health Service Publication No. 1103. [7]. Hill, A.B. (1965). The environment and disease: association or causation? Proceedings of the Royal Society of Medicine 58, 295-300. [8]. Feinstein, A.R. (1979). Clinical biostatistics. Scientific standards vs. statistical association and biologic logic in the analysis of causation. Clinical Pharmacology and Therapeutics 25, 481-492. [9]. Susser, M. (1986). Rules of inference in epidemiology. Regulatory Toxicology and Pharmacology 6, 116-128. [10]. Brown, L.M. (2000). Heliobacter pylori: epidemiology and routes of transmission. Epidemiology Reviews, 22,283-297. Available at: http://epirev.oxfordjournals.org/content/22/2/283.long Accessed June 7, 2001. [11]. Miller, D.W. and Miller, C.G. (2005). On evidence medical and legal. Journal of American Physicians and Surgeons 10, 70-75. [12]. Smith, R. (2009). In Search Of an Optimal Peer Review System. Journal of Participatory Medicine 1, (1) e13. [13]. Godlee, F. (2006). Editor's Choice. Of measles and flu. British Medical Journal 333, doi: 10.1136/bmj.333.7574.0-f (published 26 October 2006). [14]. Smith, R. (2010). Scrap peer review and beware of "top journals". British Medical Journal group blogs. Available at: http://blogs.bmj.com/bmj/2010/03/22/richard-smith-scrappeer-review-and-beware-of-%e2%80%9ctopjournals%e2%80%9d Accessed June 7, 2011. [15]. Rout, M. (2009). Doctors signed Merck's Vioxx studies The Australian, April 2009. Available at: http://www.theaustralian.com.au/news/doctors-signedvioxx-studies/story-e6frg6no-1225696993365 Accessed June 7, 2011. [16]. Charlton, B.G. and Miles, A. (1998). The Rise and Fall of EBM. Quarterly Journal of Medicine 91, 371-374. [17]. Silverman, M.E., Murray, T.J. and Bryan, C.S., eds. (2002). The Quotable Osler. Philadelphia: American College of Physicians. [18]. Brownlee, S. (2004). Doctors without borders: why you can't trust medical journals anymore. Washington Monthly (April). Available at: http://www.washingtonmonthly.com/features/2004/0404.br ownlee.html Accessed May 31, 2011. [19]. Kirby, D. (2005). Evidence of Harm: Mercury in Vaccines and the Autism Epidemic: A Medical Controversy. New York: St. Martin's Press. [20]. Zaman, K., Roy, E., Arifeen, S.E., Rahman, M., Raqib, R., Wilson, E., Omer, S.B., Shahid, N.S., Breiman, R.F. and Steinhoff, M.C. (2008). Effectiveness of maternal influenza immunization in mothers and infants. New England Journal of Medicine 359(15), 1555-1564. [21]. Jefferson, T., Di Pietrantonj, C., Rivetti, A., Bawazeer, G.A., Al-Ansary, L.A.and Ferroni, E. (2010). Vaccines for preventing influenza in healthy adults (Review). The Cochrane Library July 7 issue, 1-104. [22]. Curtis, N. (2010). The Cholesterol Delusion. Indianapolis, IN: Dog Ear Publishing. [23]. Miller, D.W. (2008). Questioning HIV/AIDS, Human-Caused Global Warming and other Orthodoxies in the Biomedical Sciences. Lew Rockwell.com (March 4). Available at: http://www.lewrockwell.com/miller/miller26.html Accessed June 1,2011.