You Will Respect My Authoritah!? A Reply to Botting Moti Mizrahi Florida Institute of Technology Forthcoming in Informal Logic Abstract: In Mizrahi (2013a) and (2016a), I argue that arguments from expert opinion are weak. To appeal to expert opinion is to take an expert's judgment that p is the case as (defeasible) evidence for p. Appeals to expert opinion are weak because the fact that an expert judges that p doesn't make it significantly more likely that p is true. Unlike other critics, Botting (2018) says he wants to take issue with the premise that reliability is a necessary condition for strong appeals to expert opinion. I respond to Botting's objections and argue that they miss their intended target. Keywords: appeals to expert opinion, expert performance, inductive support, weak arguments 1. Introduction In Mizrahi (2013a) and (2016a), I argue that arguments from expert opinion are weak. To appeal to expert opinion is to take an expert's judgment that p is the case as (defeasible) evidence for p. Appeals to expert opinion are weak because the fact that an expert judges that p doesn't make it significantly more likely that p is true, as empirical evidence from studies on expert performance suggests (Mizrahi 2016a: 246-247). I sum up my argument as follows (Mizrahi 2013a: 58-59): (1) Arguments from expert opinion are weak unless the fact that expert E judges that p makes it significantly more likely that p is true. (2) [As empirical evidence on expertise shows] the fact that E judges that p does not make it significantly more likely that p is true. (3) Therefore, arguments from expert opinion are weak arguments (Mizrahi 2016a: 238). Unlike other critics (e.g., Seidel 2014; Walton 2014), who take issue with the empirical evidence supporting premise (2), Botting (2018: 503) says he wants to take issue with premise (1): "I will argue that the problematic premise is actually (1)." In what follows, I argue that Botting fails to show that premise (1) is false and that his attempt to show that arguments from expert opinion are strong is unsuccessful. Before I do so, a terminological note is in order. Botting criticizes the arguments in Mizrahi (2013a) and (2016a), so I assume he aims to engage with my arguments concerning appeals to expert opinion. But he uses the phrase "arguments from position to know" when arguing that appeals to expert opinion are strong. These are not the same. Walton et al. (2008: 13-14) distinguish between "Argument from Position to Know" and "Argument from Expert Opinion." In the former, the source doesn't have to be an expert, whereas in the latter there is an explicit appeal to expertise.1 For present purposes, I assume that Botting is talking about arguments from expert opinion, not arguments from position to know. For if he is talking about the latter, then we may be talking past each other in more ways than one. 1 The critical questions of each argumentation scheme are also different (Walton et al. 2008: 13-15). 2 2. What are weak arguments? If Botting wants his objections to hit their intended target, i.e., premise (1), he must use the terms I use in the same ways I use them; otherwise, we would be talking past each other. The aforementioned quote from Botting (2018: 503), however, suggests that he doesn't use "weak argument" as I do. "A weak argument is an argument in which the premises, even if true, provide weak support-or no support at all-for the conclusion" (Mizrahi 2013a: 61). Now, "inferences from 'Expert E says that p' to 'p', where the truth value of p is unknown, are weak in the sense that 'Expert E says that p' does not make p significantly more likely to be true or probable" (Mizrahi 2016a: 238).2 In logic textbooks, this is the standard use of "weak" when applied to non-deductive or inductive arguments. For example, "Because inductive arguments are supposed to provide reasons, and reasons vary in strength, inductive arguments can be evaluated as strong or weak, depending on the strength of the reasons that they provide for their conclusions" (SinnottArmstrong and Fogelin 2015: 180). Accordingly, the following is an example of a weak argument: 1. Few Americans are billionaires. 2. Jeff Bezos is an American. 3. Therefore, Jeff Bezos is a billionaire. The premises, although true, don't provide strong inductive support for the conclusion, for they don't make the conclusion significantly more likely to be true. If only few Americans are billionaires, then the likelihood that any given American is a billionaire is low. As Govier (2010: 65) says, "[i]f the sample is lower than 50 [percent], then, unless the population is extremely uniform or itself very small, the argument is weak."3 On the other hand, the following is an example of a strong argument: 1. Most Americans live in poverty. 2. Jeff Bezos is an American. 3. Therefore, Jeff Bezos lives in poverty. The conclusion, although false, is more likely to be true than false, given the truth of the premises. For, if most Americans live in poverty, then the likelihood that any given American lives in poverty is high. In other words, the premises provide strong inductive support for the conclusion. Of course, Jeff Bezos is an outlier; he is unlike most Americans when it comes to wealth, which is why this sort of inductive reasoning is defeasible. Contrary to my use of "weak" and "strong" as referring to the reasoning or inference from premises to conclusion, Botting uses these terms to refer to degrees of belief (or credences). Textual evidence for this includes the following (emphasis added): 2 "To say that an argument from expert opinion is weak is to say that an expert's judgment that p does not make it significantly more likely that p is the case" (Mizrahi 2018: 176). 3 Cf. Salmon (2013: 158). 3 it is not only the expert's saying so but further facts that together provide a reason to believe (whether strongly or weakly) that what the expert says is true (Botting 2018: 506). outright belief is not justified, tentative belief is, and so that argument from expert opinion is a strong argument (albeit for a weakly held belief) (Botting 2018: 521). Further textual evidence includes expressions used by Botting (2018: 510), such as "[an opinion] asserted in a strong way," etc.4 Insofar as Botting uses "weak" and "strong" to talk about degrees of belief, whereas I use them to talk about inductive support, we are talking past each other. For this reason, to the extent that his objections to premise (1) depend on him using "weak" and "strong" in ways that differ from the ways I use these terms, it's no surprise that his objections miss their intended target. Inductive support has to do with the probability of a conclusion being true given the truth of the premises. Whether one believes the conclusion, and to what degree, is a different matter, which is separate from how well the premises, if true, support the conclusion. For example, whether, and to what degree, I believe the conclusion that Jeff Bezos lives in poverty, the premises "Most Americans live in poverty" and "Jeff Bezos is an American" provide strong inductive support for this conclusion. Likewise, whether, and to what degree, I believe the conclusion that Jeff Bezos is a billionaire, the premises "Few Americans are billionaires" and "Jeff Bezos is an American" provide weak inductive support for this conclusion. Ideally, degree of belief and inductive support should covary; one should believe that p in accordance with the inductive support for p (Mizrahi 2013a: 65).5 Even if two variables do (or should) covary, it doesn't mean that they are one and the same. Degree of belief and inductive support are distinct. The former has to do with one's confidence in p being true, whereas the latter has to do with how well p is supported by evidence. Even if some evidence e provides strong inductive support for p, one might still not believe that p, even if one should believe that p on the basis of e. For example, 34% of Americans don't believe that the theory of evolution applies to humans (Masci 2017), despite the strong evidence for the theory. Likewise, one might believe that p quite strongly, even if e doesn't provide strong inductive support for one's belief that p. For example, 6% of Americans believe that vaccines cause autism (Newport 2015), despite the fact that there is no strong evidence for a causal link between vaccines and autism. The point, then, is that degree of belief and inductive support are distinct. To the extent that Botting conflates the two, his objections miss their intended target, which is premise (1), where "weak" is used to describe the inductive support that an expert's judgment that p allegedly gives to p, not the expert's degree of belief. 4 Botting (2018) misconstrues the phrase "under uncertainty" as well. In the literature on judgment and decisionmaking under uncertainty (Tversky and Kahneman 1974), it doesn't refer to psychological (un)certainty or (lack of) confidence, as Botting (2018: 529) seems to think, but to a state of incomplete information (Kochenderfer 2015: 11). Insofar as appeals to expert opinion go, one is in a state of incomplete information when "the truth-value of p is unknown and there is no reason to believe that p is the case other than the fact that an expert judges that p is the case" (Mizrahi 2018: 175-176). 5 "A wise man [...] proportions his beliefs to the evidence" (Hume 1999: 170). 4 3. What does the evidence on expert performance show? Although he says he wants to take issue with premise (1), Botting (2018: 503) devotes a significant portion of his paper to casting doubt on the empirical evidence supporting premise (2). He divides the studies I cite into three groups. Botting (2018: 519) puts the studies discussed in Stewart (2009) and Freedman (2010) in "Group 1" and claims it "does not prove what Mizrahi wants." He puts the FBI's claim that "U.S. businesses lose $200-$250 billion to counterfeiting on an annual basis" (Jacobson 2010) in "Group 2" and claims it "does not prove what Mizrahi wants either" (Botting 2018: 519).6 Finally, he puts the cold fusion case, Tetlock (2005), Dawes (1994), Camerer and Johnson (1991), Yates and Tschirhart (2006), and Wilson et al. (1997) in "Group 3" and claims that these studies "fail for a quite different reason" (Botting 2018: 520). I emphasize the word "prove" because I'm not sure if Botting uses it literally. Since the evidence in question is empirical, proof is hardly a reasonable standard of evidence. These studies don't prove that experts are unreliable. Rather, they provide strong inductive support for the conclusion that expert opinion is generally unreliable, i.e., not "more likely to be true than false" (Johnson and Blair 1983: 144). My argument for premise (2) is an argument from empirical evidence. That is to say: Experiments show that the opinions of experts are not significantly more likely to be true than the opinions of novices. Therefore, whether an opinion is that of an expert or a novice has no significant effect on whether or not it is true (Mizrahi 2016a: 248-249). Accordingly, the response to Botting's charge that these studies don't prove that expert judgments are unreliable is "of course, they don't." They are not supposed to prove that; rather, they are supposed to provide strong inductive support for the conclusion that expert judgments are generally unreliable. Botting (2018: 519) claims that Group 1 "does not prove what Mizrahi wants" because the evidence shows that expert opinion is "not much better than (and roughly equal to) nonexpert opinion," and "[b]eing not much better (or even worse) than non-expert opinion does not equate with being not much better than chance" (Botting 2018: 518). This is incorrect. If expert performance is no better than (or equal to) novice performance, then it is no better than (or equal to) chance. To see why, consider that "A subject, who is merely guessing, has an even chance (50 percent) of being either right or wrong" (Silverman 1978: 255).7 Now, since a novice with respect to a subject matter S is unfamiliar with S in virtue of being a novice, it's safe to assume that a novice will be guessing when asked to express opinions or answer questions about S. From the fact that a novice will be guessing, it follows that a novice has an even chance of being right or wrong. For this reason, any empirical study that shows that expert performance is no better 6 I will not discuss Botting's "Group 2" because it only contains the FBI case, which doesn't play an evidential role in my (2013a). The same applies to the cold fusion case (Mizrahi 2016a: 248-250). 7 "If we guess randomly, say by flipping a coin to decide which case will be higher, then we should be wrong 50% of the time" (Handel 1978: 153). 5 than (or equal to) novice performance is, ipso facto, an empirical study that shows that expert performance is no better than (or equal to) chance. As for "Group 3," Botting (2018: 520) claims that these studies "ignore doubtpreservation." For Botting (2018: 520), "if E asserts that p, or performs some other act indicating that she takes herself to have justified belief that p, then the arguer likewise would (other things being equal) be justified in believing that p." This is a variation on Walton's Conditional Premise: "If source E is an expert in a subject domain S containing proposition A, and E asserts that proposition A is true (false), then A may plausibly be taken to be true (false)" (Walton et al. 2008: 20). My argument challenges this premise. By asserting this premise without argument, Botting is begging the question. I would not accept that, "if E asserts that p, or performs some other act indicating that she takes herself to have justified belief that p, then the arguer likewise would (other things being equal) be justified in believing that p" (Botting 2018: 520), for I take studies on expert performance to show that expert opinion is unreliable. If expert opinion is unreliable, then the fact that E asserts that p doesn't make it significantly more likely that p is true. Again, Botting is conflating degree of belief with inductive support. He claims that "the cognitive attitude that I take to be justified on the basis of the expert opinion matters, and the argument is to be evaluated according to whether it provides strong reasons for that attitude" (Botting 2018: 520). However, appeals to expert opinion are supposed to be arguments for propositions, not cognitive attitudes. After all, the conclusion of appeals to expert opinion is "A may plausibly be taken to be true (false)" (Walton et al. 2008: 20) or "A is true (false)" (Walton 2014: 148), not "E suspects that A" or "E hopes that A." The reasons stated in the premises of arguments from expert opinion are supposed to be reasons for taking the proposition E asserts to be true, not reasons for taking E to have a particular cognitive attitude with respect to that proposition. On Botting's view, the conclusion of arguments from expert opinion wouldn't be "A may plausibly be taken to be true (false)," or "A is true (false)," but rather "E believes that A" or "E suspects that A," etc. Insofar as appeals to expert opinion go, however, the question is not whether E believes that A. After all, E has asserted that A, i.e., it's E's opinion that A is the case (Walton et al. 2008: 19). Rather, the question is whether the fact that E judges that A provides a strong reason to believe that "A is true (false)" or to take A "to be true (false)." I don't think it does. The fact that E asserts that A doesn't make A significantly more likely to be true than false. This means that expert opinion is unreliable, which, in turn, means that expert opinion is an untrustworthy source of evidence (Mizrahi 2018: 177-179). There is another reason to think that Botting's talk of cognitive attitudes is beside the point. Belief is taken to be a propositional attitude one has when one takes something to be the case or regard it as true (Mizrahi 2012). As Moser (1989: 16) writes, "a state of believing is formed with respect to P for one only if one assents to P." Since to assent to p is to express approval or agreement with p, i.e., to assert p with approval or agreement, it follows that an expert who asserts that p also believes that p. In other words, if the Minor Premise is true, i.e., if E asserts that proposition A (in domain S) is true, then E also assents to A, and thereby believes that A is true. 6 4. Are arguments from position to know strong? Although he wants to "argue that the problematic premise is actually (1)," Botting (2018: 503) makes claims suggesting he accepts premise (1) after all. For example, against my argument that the Conditional Premise of his scheme for appeals to expert opinion is implausible given the empirical evidence on expert performance, Walton (2014) argues that I only think this because I take "the traditional view that such a conditional can only be deductive in nature, like the strict material conditional of classical deductive logic, or an inductive conditional that is statistical in nature." Botting (2018: 510) weighs in: I do not see how Walton can deny the requirement for an expert's saying something (in their domain) to indicate that what is said is significantly more likely to be true than not [Mizrahi's premise (1)]. This looks like Botting accepts reliability as a necessary requirement for appeals to expert opinion being strong. If he accepts that, however, then he accepts premise (1). For premise (1) "states that reliability is a necessary condition for being a trustworthy source of evidence" (Mizrahi 2018: 178). If there is evidence that expert opinion is unreliable, then expert opinion would be an untrustworthy source of evidence, for "[b]eing unreliable disqualifies a source from being a trustworthy source of evidence" (Mizrahi 2018: 178). Since there is evidence that expert opinion is unreliable, it follows that expert opinion is an untrustworthy source of evidence. For any given appeal to expert opinion, we have good reasons to believe that the Conditional Premise is probably false.8 Botting might reply that expert opinion is reliable by definition. That is, an expert is someone whose opinions are more likely to be true than false. If one's opinions in a domain one claims to be an expert in are not more likely to be true than false, then one is not an expert. I address this attempt to establish trust in expert opinion by definition in Mizrahi (2016a).9 Making expert opinion reliable by definition looks like "an instance of the dubious 'no true Scotsman' maneuver," which "would, in effect, make expertise immune to empirical investigation" (Mizrahi 2016a: 240). But we do want to study expertise empirically. I think that Botting (2018: 512) wants that, too, for he wants to be able to "establish [an expert's] trackrecord of speaking truths over falsehoods." As Botting (2018: 522) says, "[t]he issue here is not whether we ought to call them 'experts' who have turned out to be wrong. [...] The issue [...] is whether we would be justified in believing some [...] claim on [an expert's] authority."10 So, if 8 Botting (2018: 511) continues: "an expert is simply one such that appealing to their opinion gives you reasons to think that what they say is true, and I fail to see how this belief can be justified if Mizrahi's premise (1) is false." Again, this looks like Botting accepts that "only reliable sources can be trustworthy sources of evidence" (Mizrahi 2018: 178). 9 Seidel (2014: 198) acknowledges that making expert opinion reliable by definition amounts to "some kind of immunization strategy." 10 After saying that "[t]he issue [...] is whether we would be justified in believing some [...] claim on [an expert's] authority," Botting (2018: 522) goes on to say that "[t]he real issue can be put this way: are we justified now in believing the claims of our best science, given that we know that they may well turn out to be false?" I'm not sure how Botting gets from expert opinion to the pessimistic induction. Scientific results are typically established by 7 we find that expert opinion is often wrong, then we shouldn't say "well, these people aren't really experts." Instead, we should rethink how we appeal to expertise as a source of evidence and/or knowledge (Mizrahi 2018: 188). Accordingly, Botting's attempt to rethink arguments from expert opinion is commendable. But I'm not sure that he succeeds in showing that "arguments from expert opinion are strong" (Botting 2018: 504). Botting distinguishes between what he calls "the inductive argument from position to know" and "the non-inductive argument from position to know." He doesn't provide schemes for these "arguments from position to know," but I think he may have the following scheme in mind for his "inductive argument from position to know"11: Major Premise: Most of what expert E asserts on subject matter S is correct. Minor Premise: E asserts that p about S. Conclusion: p is correct.12 This scheme captures the necessary condition of reliability for appeals to expert opinion. That is, if most of E's opinions on S have turned out to be true, then we have strong inductive evidence for believing that E's new opinion on S will probably be true as well. Again, if he accepts that, then I'm not sure why Botting (2018: 503) says that "the problematic premise is actually (1)." At any rate, the problem with this inductive argument from expert opinion is that we cannot "establish [E's] track-record of speaking truths over falsehoods" (Botting 2018: 512). Studies on expert performance don't establish a track-record of successful expert opinions. As a matter of fact (Mizrahi 2013a: 63-65), they give us reasons to believe that the major premise is probably false for any given expert (Mizrahi 2016a: 246-247). Granted, if the premises of such an inductive argument were true, the conclusion would probably be true as well. For this reason, the argument could be strong in principle. But, of course, any inductive argument can be made strong, just as any deductive argument can be made valid, rather trivially by the addition of suitable premises. The question would then be whether those premises are true. For instance, the following weak argument can be made strong by changing the first premise from 1. Few Americans are billionaires. 2. Jeff Bezos is an American. experimentation and observation, not by appeal to expert opinion (Mizrahi 2016a: 242-243). Seidel (2014: 204) agrees: "no scientist should establish conclusions just by pointing out that he says so but-as Mizrahi correctly notes-by 'appeal to observations and experiments' (Mizrahi 2013a: 67)." As far as I can tell, then, the question about the strength of appeals to expert opinion has nothing to do with whether we should be realists about our best scientific theories. I should note that I'm not a scientific realist (Mizrahi 2016b), as Botting (2018: 523) seems to think. I should also note that the data I present in Mizrahi (2013b) do not show that "many more theories have been found to be true than have been found to be false," as Botting (2018: 523) erroneously says. Whether our best scientific theories are (approximately) true is what is at stake in the scientific realism debate. My data show that scientific theories and laws that are still around and have not been abandoned significantly outnumber those that have been abandoned (Mizrahi 2016b). 11 "My conclusion is that the inductive argument from position to know reduces to an ordinary inductive argument" (Botting 2018: 512-513). I take "ordinary inductive argument" to mean a statistical syllogism. 12 Cf. Salmon (2013: 119). 8 3. Therefore, Jeff Bezos is a billionaire. to 1. Most Americans are billionaires. Now the argument is strong but at the cost of rendering the first premise false. In other words, the argument is technically strong, but trivially so, given that the revised premise that made it strong is clearly false. As for Botting's "inductive argument from position to know," we have no reasons to believe that the major premise is true for any given expert. In fact, we have good reasons to believe that it's probably false (Mizrahi 2016a: 246-247). Botting doesn't provide a scheme for his "non-inductive argument from position to know," either. He invokes speech-act theory, but he doesn't cite any speech-act theorists. Based on the following passage, I think he is referring to Grice's cooperative principle (Botting 2018: 525): Given that you have a normative reason for interpreting my utterance that the cat is on the mat as a successful, though not necessarily truthful, act of asserting that the cat is on the mat, and a normative reason also to believe that most such acts succeed at what the speaker is attempting to do--namely, be truthful--then you have a normative reason to believe that the act is on the mat, and this for the very reason that I said it (emphasis added). The "normative reason," I take it, is Grice's cooperative principle. If so, Botting may have the following scheme in mind for his "non-inductive argument from position to know": Major Premise: One should "make [one's] conversational contribution such as is required, at the stage at which it occurs, by the accepted purpose or direction of the talk exchange in which [one is] engaged" (Grice 1975: 45). Minor Premise: a asserts that A is true (false) with the purpose of being truthful. Conclusion: A may plausibly be taken to be true (false). There are several problems with this scheme. First, attempting to do something is different from doing something. To try to swim is not the same as swimming. Similarly, trying to say something true is not the same as saying something true. One could try to say something true but fail. For this reason, the success conditions for each are different as well. That is, the success conditions for trying to swim are not the same as those for swimming. One can be successful in the former but fail in the latter (as when one is taking swimming lessons). Similarly, one could be successful at trying to say something true, and yet fail in saying something true. Accordingly, even if a asserts that A is true, and a is trying to be truthful, a could still fail in asserting a true proposition. Trying not only doesn't guarantee success but also doesn't even make success more likely. No matter how hard I will try to swim across the Pacific Ocean, I will not succeed. But I can still be successful at trying to swim across the Pacific Ocean. 9 For this reason, Botting (2018: 526) is mistaken when he says, "if the cat is not on the mat but I believe that it is, then I am still successfully asserting and it is still normative for you to interpret me as having successfully asserted" (emphasis added). One may succeed in asserting something, but not in asserting a true proposition. The success conditions for the former are not the same as those for the latter. To successfully assert a true proposition, the proposition asserted must be true. To successfully attempt to assert a true proposition, the proposition asserted need not be true. Second, the major premise is an "ought" (i.e., that one should...), whereas the conclusion is an "is" (i.e., that A is true, probably true, or plausibly true). If there is an "is-ought gap," then, from the fact that one ought to (try to) be truthful, it doesn't follow that one is truthful. This is why Botting (2018: 526) is mistaken when he says, "if you know that the cat is not on the mat, then there does not seem to be a normative reason for believing that I have told you something true since you know that I have not done what I was attempting to do" (emphasis added). Just as an "is" doesn't follow from an "ought," a "not ought" doesn't follow from an "isn't." For example, even if I didn't hold the door to a person behind me, it doesn't mean that I shouldn't have. I ought to have held the door for that person, even if I didn't. Similarly, if what you have asserted is not the case, it doesn't follow that it's not the case that you ought to (try to) be truthful. You should still (try to) be truthful even if you failed. Finally, I don't see how there can be normative reasons "to believe that most such acts [of asserting] succeed at what the speaker is attempting to do--namely, be truthful" (Botting 2018: 525). The way to determine whether one has been successful at φ-ing or trying to φ is to examine one's track record of φ-ing or trying to φ and count the successes and failures. If the successes significantly outnumber the failures, then the conclusion that most of one's acts of φing trying to φ have been successful would be warranted. In each case, however, we would have empirical, not normative, reasons to believe that most of one's acts of φ-ing or trying to φ have been successful. For this reason, Botting's "non-inductive argument from position to know" collapses into his "inductive argument from position to know." Such an inductive argument could be strong in principle, but at the cost of having false premises. Given that expert opinion is generally unreliable, "there simply are no good instantiations of the inductive argument from position to know" (Botting 2018: 528). 5. Conclusion According to Geach (1976: 66), "a physical chain may break either because it has a weak link or because it is too long: a logical chain can break only for the first reason--if (but only if) each link is sound, the whole chain is sound." Insofar as arguments from expert opinion go, the weakest link is the assumption that an expert's judgment that p provides inductive support for p. Goodwin (2011) calls this assumption "the background norm of respect for expertise." I have argued that, rather than take it for granted, we need a justification for it. A justification for "the background norm of respect for expertise" is especially urgent in light of evidence from studies on expertise suggesting that expert opinion is not as reliable as we might think (Mizrahi 2013a; 2016a) and 10 that expert judgments are susceptible to pretty much the same cognitive biases that novice judgments are susceptible to (Mizrahi 2018). References Botting, D. 2018. Two Types of Argument from Position to Know. Informal Logic 38(4): 502530. Camerer, C.F. and Johnson, E.J. 1991. The Process-Performance Paradox in Expert Judgment: How Can the Experts Know So Much and Predict So Badly? In K.A. Ericsson and J. Smith (eds.), Towards a General Theory of Expertise: Prospects and Limits, 47-76. New York: Plenum Press. Dawes, R.M. 1994. House of Cards: Psychology and Psychiatry Built on Myth. New York: Free Press. Freedman, D.H. 2010. Wrong: Why Experts Keep Failing Us. New York: Little, Brown and Company. Geach, P.T. 1976. Reason and Argument. Berkeley: University of California Press. Goodwin, J. 2011. Accounting for the Appeal to the Authority of Experts. Argumentation 25: 285-296. Govier, T. 2010. A Practical Study of Argument. 7th Ed. Belmont: Wadsworth. Grice, H.P. 1975. Logic and Conversation. In P. Cole, & J.L. Morgan (eds.), Syntax and Semantics, Vol. 3, Speech Acts, 41-58. New York: Academic Press. Handel, J.D. 1978. Introductory Statistics for Sociology. Upper Saddle River: Prentice Hall. Hume, D. 1999. An Enquiry Concerning Human Understanding. Edited by T.L. Beauchamp. New York: Oxford University Press. Jacobson, S. 2010. Government Says Data Estimating Piracy Losses is Unsubstantiated. PCWorld, April 14, 2010. Accessed on January 29, 2019. https://www.pcworld.com/article/194203/Government_Piracy.html. Johnson, R.H. and Blair, J.A. 1983. Logical Self-defense. McGraw Hill. Kochenderfer, M.J. 2015. Decision Making Under Uncertainty: Theory and Application. Cambridge: MIT Press. Masci, D. 2017. For Darwin Day, 6 Facts about the Evolution Debate. Pew Research Center. February 10, 2017. Accessed on January 22, 2019. http://www.pewresearch.org/facttank/2017/02/10/darwin-day/ . 11 Mizrahi, M. 2012. Does 'Ought' Imply 'Can' from an Epistemic Point of View? Philosophia 40(4): 829-840. Mizrahi, M. 2013a. Why Arguments from Expert Opinion are Weak Arguments. Informal Logic 33(1): 57-79. Mizrahi, M. 2013b. The Pessimistic Induction: A Bad Argument Gone Too Far. Synthese 190(15): 3209-3226. Mizrahi, M. 2016a. Why Arguments from Expert Opinion are Still Weak Arguments: A Reply to Seidel. Informal Logic 36(2): 238-252. Mizrahi, M. 2016b. The History of Science as a Graveyard of Theories: A Philosophers' Myth? International Studies in the Philosophy of Science 30(3): 263-278. Mizrahi, M. 2018. Arguments from Expert Opinion and Persistent Bias. Argumentation 32(2): 175-195. Moser, P.K. 1989. Knowledge and Evidence. New York: Cambridge University Press. Newport, F. 2015. In U.S., Percentage Saying Vaccines Are Vital Dips Slightly. Gallup. March 6, 2015. Accessed on January 22, 2019. https://news.gallup.com/poll/181844/percentage-sayingvaccines-vital-dips-slightly.aspx . Salmon, M.H. 2013. Introduction to Logic and Critical Thinking. 6th Ed. Boston: Wadsworth. Seidel, M. 2014. Throwing the Baby Out with the Water: From Reasonably Scrutinizing Authorities to Rampant Scepticism about Expertise. Informal Logic 34(2): 192-218. Silverman, R.E. 1978. Psychology. Upper Saddle River: Prentice Hall. Sinnott-Armstrong, W. and Fogelin, R.J. 2015. Understanding Arguments: An Introduction to Informal Logic. 9th Ed. Stamford: Cengage. Stewart, M. 2009. The Management Myth: Why the Experts Keep Getting It Wrong. W. W. Norton & Co. Tetlock, P.E. 2005. Expert Political Judgment. Princeton: Princeton University Press. Tversky, A. and Kahneman, D. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185(4157): 1124-1131. Wilson, I.B., Green, M.L., Goldman, L., Tsevat, J., Cook, E.F., and Phillips, R.S. 1997. Is Experience a Good Teacher? How Interns and Attending Physicians Understand Patients' Choices for End-of-Life Care. Medical Decision Making 17(2): 217-227. 12 Walton, D., Reed, C., and Macagno, F. 2008. Argumentation Schemes. New York: Cambridge University Press. Walton, D. 2014. On a Razor's Edge: Evaluating Arguments from Expert Opinion. Argument and Computation 5(2-3): 139-159. Yates, J.F. and Tschirhart, M.D. 2006. Decision-Making Expertise. In Ericsson et al. (eds.), The Cambridge Handbook of Expertise and Expert Performance, 421-438. New York: Cambridge University Press.