W Is K N OW L E D G E of S C I E N C E Associated with Higher Skepticism of Pseudoscientific Claims? M AT T H E W J O H N S O N M A S S I M O P I G L I U CC I e live in a world that is increasingly shaped by and bathed in science, with most scientific progress occurring in the past century, and much of it in the past few decades. Yet, several authors have puzzled over the observation that modern societies are also characterized by a high degree of belief in a variety of pseudoscientific claims that have been thoroughly debunked or otherwise discarded by scientists (Anonymous, 2001; Ede, 2000). Some people argue that belief in pseudoscientific claims is the result of insufficient science education (references in Goode, 2002; Walker et al., 2002). However, several polls have shown that, at least for some areas of pseudoscience, education does not seem to correlate with skepticism (Goode, 2002). For example, in the United States, the education category with the highest belief in extraterrestrial visits aboard UFOs is that of people with some college experience (51%), although post-graduate education is associated with more skepticism (but still numbers 39% believers). Indeed, a study by Walker et al. (2002) conducted at three undergraduate universities in the U.S. has shown no correlation at all between knowledge of science and belief in an array of pseudoscientific claims. A partial explanation for this state of affairs may be that scientific factual knowledge has little bearing on people's understanding the evidence in favor or against pseudoscientific claims (Walker et al., 2002). It is well known that science education, especially (but not exclusively) at the pre-college level, focuses on the teaching of facts at the detriment of explicit treatment of methodological and conceptual issues surrounding the practice of science (Walker et al., 2002). It is not at all clear why educators expect that massive factual knowledge of science should translate into conceptual understanding of the nature of science and into improved critical thinking skills, allegedly the true targets of science education. This study addresses issues associated with the relationships among science factual knowledge, conceptual understanding of science, and belief in pseudoscience by means of a 30-question survey. The survey consists of three types of questions asked of students enrolled in a MATTHEW JOHNSON is in the Department of Philosophy at the University of Tennessee, Knoxville, TN 37996-1100. MASSIMO PIGLIUCCI is Professor of Ecology & Evolutionary Biology at State University of New York, Stony Brook, NY, as well as Executive Vice-President of the Society for the Study of Evolution; e-mail: massimo@genotypebyenvironment.org. 536 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER 2004 538 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER 2004 science major (mostly biology) and compares the responses to those obtained by groups of non-science majors (business students taking a philosophy class). The first set of questions consists of 10 multiple-choice (five choices) questions intended to assess the students' general knowledge of science (periodic table, the nature of photons, etc.). The second set includes 10 true/false questions testing a respondent's understanding of important scientific concepts, such as the difference between theories and laws. The third set of questions quantifies the respondents' degree of belief (on a scale of one to five, with five signifying highest belief) in paranormal phenomena, such as telepathy, astrology, or the existence of the Loch Ness monster. By surveying science and non-science majors, we wished to test the following hypotheses of association among our measures of scientific knowledge, understanding, and paranormal belief: • Science majors have more factual knowledge of science than non-science majors (since that is what they are primarily taught). • Science majors have more understanding of conceptual issues in science (possibly because they are able to somehow derive it from factual knowledge to which they are mostly exposed). • Science majors express lower degrees of pseudoscientific belief than non-science majors (presumably because their knowledge of science makes them more skeptical of such claims). • There are no differences between genders for belief in pseudoscience, knowledge of science facts, or understanding of conceptual issues in science. (Note: Recent surveys have found a higher degree of pseudoscientific belief in women than men, though the trend is reversed for specific pseudoscientific claims, such as UFOs and unusual life forms like the Loch Ness monster. [Anonymous, 2001]). We also tested the following expectations concerning the pair-wise relationships between the different types of questions we administered: • There is either a positive or no correlation between knowledge of science facts and understanding of science concepts (because factual knowledge somehow translates into conceptual understanding, or because the two are in fact uncorrelated). The only option that is not expected under any educational theory is that of a negative correlation between the two. • There is either a negative or no correlation between knowledge of science facts and degree of pseudoscientific belief (because factual knowledge of science does, in fact, indirectly foster critical thinking), or the two (science knowledge and critical thinking skills) are unrelated to each other. One would not expect the third outcome, that of a positive relationship between conceptual understanding of science and pseudoscientific belief. • There is either a negative or no relationship between understanding of science concepts and pseudoscientific belief (because conceptual understanding of science increases critical thinking, which leads to reduced belief in pseudoscience). Alternatively, conceptual understanding of science does not translate into critical thinking skills, and hence the two are uncorrelated. The third option, of a negative relationship between science concepts and pseudoscience beliefs, is not expected under any scenario. Materials & Methods We assembled our 30-question set by examining two previously published surveys. The first one (Walker et al., 2002) compared knowledge of scientific facts to pseudoscientific beliefs. Nine of the ten science fact multiple-choice questions used by those authors were kept, and we wrote an additional question to replace the one removed. The one question was removed because it required specific knowledge of genetics, immunology, and reproduction, which we felt went beyond what could reasonably be expected at the level of introductory classes. The question we added tests a student's knowledge of the properties of a photon. We picked ten pseudoscientific questions out of the original 14 to place in the survey. We reduced the number of questions to eliminate overlapping topics (e.g., in the original questionnaire there were two questions about ghosts) and to focus on pseudoscientific beliefs that are most common. We also reduced the range of the belief scale from the original 1-7 to 1-5, with five indicating the highest level of belief. For the scientific concepts portion of our survey, we selected ten true/false questions from Richard Carrier's (2001) Test of Scientific Literacy. Again, as in the previous case, we eliminated questions due to overlapping topics. We also eliminated questions that seemed highly technical or could be easily misinterpreted by the students. We randomized the order of the 30 questions in our survey, so that students would be less likely to try to second-guess the answers, which might be the case if a series of pseudoscientific questions was presented after a series of factor concept-based questions. We presented our survey to four classes: two secondyear biology and two second-year philosophy classes at the University of Tennessee in Knoxville. We assumed in our original experimental design that philosophy majors would attend the philosophy classes, however due to class scheduling, the only philosophy classes we had access to were, in fact, ethics classes attended by business majors. Overall, there were 170 respondents. Students were asked during class to volunteer to take the survey. The survey administrator had no relationship with the class. Instructors were asked to offer no extra credit to students taking the survey. Students who responded to the survey were asked to provide only four pieces of personal information: age, gender, school year, and major. After instructions were given to the class, the administrator left the room for 15 minutes to ensure students did not feel pressured to take the survey. Students placed the surveys in a box left at the front of the room for the administrator to pick up after 15 minutes. After we collected responses from all classes, we entered the results onto a spreadsheet that was imported into the statistical software Jump (SAS for Macintosh, v.5.01). We first calculated an average coefficient of scientific fact literacy, one of scientific concepts literacy, and one of pseudoscientific belief, by averaging the responses of each student to all questions within each of the three sets. We then ran an analysis of variance on each of the three summary indices with major, gender, and the major-by-gender interaction as factors. This provided us with an overview of the association between major or gender and science literacy (both factual and conceptual) or pseudoscientific belief. Similar results were also obtained by running one-way non-parametric Kruskall-Wallis* analyses of variance. In order to obtain a more in-depth view of the same relationships, we also ran a series of contingency analyses relating major and gender to the responses to each question within each set. We noted both the overall statistical significance of major and gender effects for each question, and the percentage of correct responses (in the case of science fact or concept questions) or the degree of pseudoscientific belief relative to the total. We then considered the possible relationships among the three sets of measures, which were the major goal of this study. In order to quantify them, we calculated both non-parametric correlation coefficients (Spearman's and Kendall's*) as well as parametric Pearson* correlation coefficients between each pair of overall indices of science factual knowledge, science conceptual understanding, and pseudoscientific belief. Results were very similar regardless of the specific correlation coefficient used. Finally, we wished to quantify and visualize the similarities in students' responses to all 30 questions, which we accomplished by calculating an index of pairwise similarity between responses and subjecting the resulting matrix to a clustering algorithm, which produced a dendrogram (tree-like structure). Results were comparable when we used different indices of similarity Non-Science Majors Science Majors Sc ie n ce F ac t K n o w le d g e 1.0 0.0 Table 1. Analyses of Variance of the Relationship between Major, Gender and Major-by-Gender Interaction and the Overall Students' Scores in Science Facts, Science Concepts, and Pseudoscience Belief. R2 indicates the amount of variance explained by the model, numbers in parentheses on the top row indicate degrees of freedom for each effect. In the main body of the table, mean square values are reported, together with their associated level of statistical significance (in parentheses).The graph illustrates the mean differences between groups in the only case in which significant differences were detected. Notice that similar results were obtained by running non-parametric one-way ANOVAs (Kruskal-Wallis) on major and gender. Variable R2 Major (1) Gender (1) Major x Gender (1) Error (142) Overall Science Fact Score 24.7% 0.8419 (0.0001) 0.0019 (0.7570) 0.0326 (0.2026) 0.0199 Overall Science Concepts Score 1.4% 0.0315 (0.3090) 0.0146 (0.4878) 0.0001 (0.9474) 0.0302 Overall Pseudoscience Belief 1.0% 0.1840 (0.4713) 0.4450 (0.2631) 0.0011 (0.9552) 0.3526 PSEUDOSCIENTIFIC CLAIMS 539 * See Sokal & Rohlf in References. (Gower's* general similarity coefficient, Jaccard's* coefficient, and the simple matching coefficient) suitable for categorical data such as ours (Sneath & Sokal, 1973, pp. 129-137). Tree topology was also stable to the use of different clustering algorithms, such as unweighted arithmetic average (UPGMA), weighted arithmetic average (WPGMA), unweighted centroid (UPGMC), weighted centroid (WPGMC), and Ward's* method (Sneath & Sokal, 1973, pp. 214-244). All calculations of similarity indices and cluster analyses were conducted using the "R package" by Casgrain and Legendre (2002). On the resulting dendrogram, questions that tended to elicit similar responses across all classes of students were grouped together. Results Parametric analyses of variance of the relationship between major, gender, and major-by-gender interaction and the overall students' scores in science facts, science methods, and pseudoscientific belief (Table 1), found only a significant association between majors and their overall science fact score. The graph to the right of the Table illustrates that science majors scored (predictably) better than non-science majors on factual questions regarding a broad range of scientific fields, although the difference between the two groups was certainly not overwhelming. Similar results were obtained using a series of non-parametric one-way ANOVAs (Kruskall-Wallis) on major and gender (details not shown). The general results reported in Table 1 are consistent with the question-by-question analyses detailed in Tables 2-4 and based on a series of contingency tests. For example, note that while there are scattered significant effects of gender on science factual knowledge (Table 2), of major on conceptual understanding of science (Table 3), and of gender on pseudoscientific belief (Table 4), the majority of individual significant effects were found for major on science factual knowledge. Interestingly, questions concerning factual knowledge of the physics of energy, the nature of photons, the difference between organic and inorganic matter, the metric system, the litmus test, and the relationship between earth-sun distance and the seasons all received low scores. Less than 50% of (even) the science majors answered correctly (boldface in Table 2). Perhaps even more discouraging was the fact that no science method question received even 50% of correct answers, regardless of major or gender. Indeed, the difference between theory and laws was understood by less than 5% of the respondents in any category! More encouraging was the fact that the modal degree of belief in pseudoscientific claims was never Table 2. Contingency Analyses of the Responses to Questions on Science Facts, by Major (Non-Science vs. Science) and Gender. Questions highlighted in boldface were characterized by a particularly poor response (i.e., no category reached 50% of correct answers). Percentages refer to total correct answers (to determine how many people responded correctly or incorrectly overall), which means that they do not add up to 100% within factors. Boldfaced p-values highlight particularly striking differences between majors or genders (i.e., p<0.01). MAJOR GENDER QUESTION NONSCIENCE LIKELIHOOD P-VALUE FEMALE MALE LIKELIHOOD P-VALUE SCIENCE RATIO RATIO Dominant source of energy on earth 34.8% 52.8% 11.919 0.0006 45.5% 41.3% 7.105 0.0077 Physics of energy 34.6% 49.4% 5.177 0.0229 41.9% 42.5% 0.475 0.4906 Nature of photons 14.8% 14.2% 0.961 0.3270 13.7% 14.9% 0.002 0.9611 Nature of infectious disease 27.2% 51.9% 26.154 0.0001 42.9% 35.7% 8.363 0.0038 Organic vs. inorganic 22.1% 43.6% 15.775 0.0001 34.9% 31.4% 2.310 0.1286 Periodic table 35.2% 54.3% 21.057 0.0001 43.5% 45.8% 0.011 0.9149 Metric system 29.0% 38.9% 0.753 0.3856 32.1% 36.9% 0.765 0.3818 Litmus test 20.6% 46.3% 24.411 0.0001 34.3% 31.9% 0.765 0.3816 Genetic disorders 40.5% 54.0% 4.319 0.0377 47.0% 47.0% 1.550 0.2131 Earth-sun distance and seasons 8.6% 9.3% 0.147 0.7014 6.6% 12.5% 3.348 0.0673 540 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER 2004 * See Sokal & Rohlf in References. higher than 3 (out of 5), and it was often lower than that (most frequently 1, the most skeptical response) (Table 4). Nevertheless, a low degree of skepticism was found for claims concerning the healing power of magnets, the presence of aliens in a government facility known as Area 51, and the existence of telepathy or clairvoyance (boldface in Table 4). On the positive side, students seemed to be particularly skeptical of the good or bad luck brought by chain letters and broken mirrors. In order to determine the degree of correlation between the pairwise overall scores of students in pseudoscience, science facts, and science concepts, we calculated a series of Spearman rank correlation coefficients (Table 5). They indicated that there was a weak positive correlation between knowledge of science facts and understanding of science concepts. We also found a weak negative correlation between pseudoscientific beliefs and science facts, but apparently no relationship between pseudoscience belief and understanding of scientific concepts. None of these correlation coefficients exceeded 0.27, however, indicating a large amount of unexplained variation in each indicator. Similar results were obtained using Kendall's rank or Pearson's parametric correlation coefficients. Finally, a cluster analysis on the responses to all questions was performed using several measures of similarity and methods of hierarchical clustering (see Materials & Methods). The results reported here (Figure 1) were obtained by subjecting a matrix of Gower's general similarity coefficients to unweighted arithmetic average (UPGMA) clustering (though similar results were obtained with the other methods). Three measures of cophenetic correlation (Sneath & Sokal, 1973, pp. 278-280) between the output of the clustering algorithm and the original similarity matrix were satisfactory, indicating that the dendrogram reliably reproduced the degree of similarity among responses to the various questions (the cophenetic coefficients were as follows: Kendall = +0.77; Pearson = +0.82; Gower = 2.98; notice that the first two vary between 0 and 1, where Table 3. Contingency Analyses of the Responses to Questions on Science Concepts, by Major (Non-Science vs. Science) and Gender. Questions highlighted in boldface were characterized by a particularly poor response (i.e., no category reached 50% of correct answers). Percentages refer to total correct answers (to determine how many people responded correctly or incorrectly overall), which means that they do not add up to 100% within factors. Boldfaced p-values highlight particularly striking differences between Majors or Genders (i.e., p<0.01). Notice that all questions received very low overall percentages of correct answers, and that there were few significant differences between levels of the factors. MAJOR GENDER QUESTION NONSCIENCE LIKELIHOOD P-VALUE FEMALE MALE LIKELIHOOD P-VALUE SCIENCE RATIO RATIO Science produces tentative conclusions 27.6% 39.9% 2.050 0.1522 32.5% 33.1% 0.137 0.7111 Is there only one scientific method? 26.1% 35.0% 0.789 0.3744 34.7% 28.1% 4.289 0.0384 Theories are explanations, not facts 36.4% 44.4% 0.001 0.9901 36.9% 43.5% 2.298 0.1295 Is science just about facts or about interpretations? 35.0% 47.9% 2.077 0.1495 42.0% 40.8% 1.587 0.2077 Does science require to conduct experiments? 19.8% 27.2% 0.506 0.4769 21.4% 25.6% 0.627 0.4285 Can experiments prove theories? 11.8% 26.1% 8.131 0.0044 19.8% 18.0% 0.435 0.5094 Science includes beliefs, assumptions & non-observables 25.9% 27.8% 0.785 0.3755 26.2% 27.4% 0.035 0.8509 Are laws exceedingly well confirmed theories? 3.1% 5.0% 0.273 0.6013 4.8% 3.0% 1.055 0.3044 A theory is a hypothesis that has been amply confirmed 38.9% 49.4% 0.074 0.7852 43.5% 44.6% 0.132 0.7162 Science uses theoretical entities that have never been observed 30.3% 34.6% 0.311 0.5769 31.7% 33.5% 0.029 0.8655 542 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER 2004 higher values indicate better fit, while the third one varies between 0 and infinity, and low values indicate better fit). The results show two major clusters, with several distinct sub-clusters. Most of the pseudoscience questions clustered together (bottom of diagram in Figure 1), with the exception of those concerning luck brought by chain letters and broken mirrors (the same two for which students showed a high degree of skepticism), which clustered with a large number of science factual and conceptual questions (top portion of Figure 1). The second major cluster was made up of several sub-clusters, mostly with a mixture of science factual and conceptual questions, some of which are perhaps suggestive of interesting associations. For example, one tight cluster grouped together answers related to the ideas that scientific conclusions are tentative, that science is based on assumptions and postulates, and that theoretical entities are often featured in scientific conclusions. Other clusters, however, do not seem to hint at any simple relationship within or between the science facts and concepts questions. Discussion Belief in all sorts of paranormal claims is very high in the United States, with recent surveys (Anonymous 2001) indicating, for example, that 36% of Americans think astrology is "very" or "sort of" scientific, 17% report having contacted a fortune teller, and a whopping 1/3 to half of Americans believing in UFOs. The Table 4. Contingency Analyses of the Responses to Questions on Pseudoscientific Beliefs, by Major (Non-Science vs. Science) and Gender. Questions highlighted in boldface were characterized by a particularly non-skeptical response (i.e., not even 50% of students in any category expressed complete disbelief). Entries under the levels of each factor indicate the modal response (from 1 to 5, with 5 as the highest degree of belief), and the percentage of students (within each level of each factor) answering in that fashion (in parentheses.) Boldfaced p-values highlight particularly strikingly (i.e., p<0.01) differences between majors or genders. Notice that there were few significant differences between levels of the factors. MAJOR GENDER QUESTION NONSCIENCE LIKELIHOOD P-VALUE FEMALE MALE LIKELIHOOD P-VALUE LEAST SCIENCE RATIO RATIO SKEPTICAL Magnets can heal 3 3 3.587 0.4648 3 3 2.950 0.5661 (39.7%) (42.2%) (42.7%) (39.1%) There are aliens in Area 51 2-3 3 0.999 0.9100 3 2 0.636 0.9589 (30.1%) (30.0%) (29.3%) (29.9%) Telepathy or clairvoyance is real 2 2 4.237 0.3748 2 1 6.862 0.1434 (38.4%) (32.2%) (35.4%) (40.2%) Astrology predicts personality & future 1 1 2.311 0.6787 1 1 10.441 0.0336 (52.1%) (48.9%) (40.2%) (60.9%) Bigfoot exists 1 1 9.385 0.0522 1 1 4.081 0.3952 (58.9%) (50.0%) (56.1%) (50.6%) The Loch Ness monster exists 1 1 7.339 0.1190 1 1 18.508 0.0010 Males (52.1%) (41.1%) (47.6%) (44.8%) Sending chain letters brings good luck 1 1 8.264 0.0409 1 1 3.785 0.2856 (80.8%) (87.8%) (81.7%) (88.5%) Animals can sense ghosts 1 1 1.256 0.8688 1 1 15.191 0.0043 Females (48%) (40%) (31.7%) (53.5%) Voodoo kills 1 1 0.972 0.9141 1 1 3.796 0.4343 (58.3%) (55.6%) (51.2%) (61.6%) Broken mirrors bring bad luck 1 1 3.641 0.3029 1 1 7.977 0.0465 (72.6%) (82.2%) (72.0%) (85.1%) PSEUDOSCIENTIFIC CLAIMS 543 causes of such widespread belief in irrational or unsubstantiated claims are difficult to pinpoint, as are potential trends (increasing or decreasing), due to the complexity of cultural forces involved and the lack of standardization across surveys. Walker et al. (2002) have suggested that science education is no guarantee of skepticism: These authors found no significant correlation between scores on a test of science literacy and degree of belief in an array of pseudoscientific claims when they surveyed three samples of undergraduate students at small universities in the United States. The scope of our study was such that we could test some specific hypotheses concerning the expected association between indicators of science knowledge (both factual and conceptual) and of pseudoscientific belief. Of course, we were in no position to directly address the causal links between education and belief, although later in this article we suggest some follow-up studies that might come closer to that goal. First, we hypothesized that science majors should display more knowledge of science facts than non-science majors, a minimalistic prediction if, in fact, science education is expected to have any effect whatsoever. Indeed, our results did confirm this expectation, although the difference between the scores of the two groups was not nearly as impressive as one might have hoped. We also made the somewhat more risky prediction that science majors would display more conceptual understanding of science, allegedly the true goal of science education, than their non-science counterparts. No such difference was found, which leads to questioning one of the most cherished assumptions of science educators: If we wish our students to understand how science works, confronting them with a lot of factual knowledge does not seem to help. Moreover, the general degree of conceptual understanding of science on the part of our students was abysmally low, especially in crucial areas such as the distinction between laws of nature (i.e., observations of regular patterns with no exceptions) and well-substantiated scientific theories (i.e., human interpretations of how the world works, which withstood repeated empirical tests). The third prediction was even bolder: We speculated that science majors would display lower degrees of pseudoscientific belief, at least in part as a result of their science training (though, of course, effects due to selfselection are also possible). Again, we were disappointed: While students in our samples did show generally low degrees of pseudoscientific belief (with the notable exceptions of the healing powers of magnets, the existence of aliens being held at "Area 51," and the existence of telepathy or clairvoyance), no difference was found between the majors. We also investigated the possibility of the existence of differences in our indicators between genders, given the repeated observation of such differences in previous surveys. For example, work by Vitulli et al. (1999) found that belief in the paranormal is stronger in young males attending college as well as in elderly women, although they did find a possible positive effect of education: Elderly people attending continuing education courses score significantly lower in their belief in the paranormal (though, of course, this may have been due to a selfselecting effect). Belief in several (though not all) paranormal phenomena was found to be higher in women than in men by a survey conducted by the National Science Foundation (Anonymous, 2001), and a survey by Irwin (1985) found that belief in the paranormal is stronger in women than men. Our overall results did not show any difference between genders when an average indicator of pseudoscientific belief was considered, nor were gender differences significant for overall science factual or conceptual knowledge. However, more detailed analyses did reveal a hint of some differences between genders. For example, female students knew slightly better than their male counterparts about the dominant source of energy on Earth and about the nature of infectious disease, though it is difficult to speculate on the causes of this difference, and we are inclined to attribute it to statistical fluctuations. Significantly, we found no differences between genders even upon a more in-depth analysis in the area of conceptual understanding of science. Curiously, men were less likely to believe in the existence of the Loch Ness monster and more likely to think that animals can sense ghosts. Again, however, it is possible that these findings were due to statistical fluctuations and carry no general meaning. Table 5. Pairwise Spearman's Rank Correlation Coefficients Relating Overall Scores of Students in Pseudoscience, Science Facts, and Science Concepts Categories. Significance levels of the statistical tests are in parentheses. Similar results were obtained using either Kendall's rank or Pearson parametric correlation coefficients. Pseudoscience Science Facts Science Facts -0.18 (0.0228) Science Concepts -0.06 +0.27 (0.4383) (0.0007) PSEUDOSCIENTIFIC CLAIMS 545 Figure 1. Cluster Analysis of the Similarities in Students' Responses to the 30 Questions on Science Facts, Science Concepts, and Pseudoscience. Numbers on the right quantify similarities between objects within each cluster.The dendrogram is based on unweighted arithmetic average (UPGMA) of a matrix of similarities calculated using Gower's general similarity coefficient. Measures of cophenetic relationships between the derived and original matrix (Kendall's tau = 0.77, Pearson's r = 0.82, and Gower's distance = 2.98), indicated a good fit between the dendrogram and the similarity matrix.The same topology was obtained by subjecting the same coefficients to other clustering methods, and the major features of the topology were retained when using different coefficients of similarity such as Jaccard's and the simple matching coefficient.These other methods, however, yielded a lower fit between tree topology and similarity matrix when measured by the above-mentioned cophenetic coefficients, and do not resolve the differences among the responses to the questions on pseudoscientific beliefs. Earth's energy Energy physics Periodic table Genetic disorders Hypothesis vs. theory Theories as explanations Infectious diseases Not just facts Chain letters Broken mirrors Organic vs. inorganic Metric system Litmus test One scientific method Tentative conclusions Theoretical entities Science makes assumptions Nature of photons Laws vs. theories Earth-sun & seasons Experiments prove theories Experiments required Magnets can heal Aliens in Area 51 Telepathy/clairvoyance Astrology Voodoo Bigfoot Loch Ness Animals sense ghosts L E V E L 0.07649 0.08855 0.05768 0.06474 0.04321 0.11243 0.07445 0.08224 0.12402 0.05542 0.12884 0.12192 0.14542 0.14981 0.21221 0.12086 0.13707 0.39355 0.11552 0.12720 0.08163 0.15780 0.30824 0.23521 0.26717 0.20000 0.21454 0.09517 0.21285 Conclusions One of the major goals of our research was to investigate the possible relationships among our three indices of knowledge of science facts, understanding of how science works, and pseudoscientific belief. Under the most optimistic scenario, we had predicted a positive association between knowledge of science facts and conceptual understanding of the nature of science (if the standard educational assumption holds), and a negative association between either measure of science literacy and pseudoscientific belief (under the assumption that more knowledge of science makes for better critical thinking, and therefore more skepticism about pseudoscience). 546 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER 2004 The first prediction turned out to be correct, although the strength of the association between knowledge of science facts and conceptual understanding of science methods was very weak. This is consistent with the idea that there is some detectable seepage from learning many facts about science to a higher-level understanding of how science works. However, the weakness of the relationship strongly suggests that there must be better ways of achieving this, consistent with recent literature on science teaching and critical thinking (Wandersee, 1990; Sundberg et al., 1994; Belzer et al., 2003). On the other hand, neither knowledge of science facts nor understanding of how science works seemed to be associated with the degree of belief in pseudoscience (though both correlation coefficients were in the right direction; that is, negative). This, of course, is subject to several interpretations, and does not necessarily mean that a better understanding of science does not foster critical thinking. However, it does mean that whatever association there may be between knowledge of science and skepticism about pseudoscience, it is not very strong or particularly evident. Indeed, even at the more sophisticated level of graduate studies, Lehman et al. (1988) found that training in the hard sciences (chemistry) did not result in an increased level of transferability of critical thinking skills to everyday problems. On the other hand, graduate students in the social sciences (psychology), who are continuously exposed to complex problems characterized by probabilistic answers, seem to be much better equipped to apply their critical thinking skills to other domains than academic research. This is particularly interesting because it argues that another assumption commonly made by science educators, that science training makes for better critical thinkers, may not be true even at the level of graduate studies, let alone undergraduate. Several caveats and possible future directions in regards to this study need to be addressed. One obvious limitation of our research is that it did not include a longitudinal component to help discriminate between the actual effect of teaching science and the possibility of self-selection of more critically thinking students into scientific disciplines. However, since we did not find significant differences in this respect between science and non-science majors, our results can hardly be attributed to self-selection processes. Nevertheless, it would be interesting to compare, for example, freshmen and seniors in science vs. non-science majors, with the idea that any difference between groups that increases with time would likely be due to training rather than self-selection. It is of course possible that both effects contribute, which would translate into a significant year-by-major interaction in an analysis of variance. It would be interesting to examine the possible differences between actual philosophy students and science majors, as opposed to business students taking ethics classes in philosophy, as happened in our case. The reason for this is that philosophers are among the few majors who actually receive formal training in critical thinking, through courses explicitly designed for that purpose, as well as through rigorous training in logical and conceptual analyses of any course material to which they are exposed. It would also be interesting to expand the study to include graduate students, comparing them between disciplines (a la Lehman et al, 1988), as well as with beginning and advanced undergraduates. One would expect that graduate students might be more skeptical of pseudoscientific claims than undergraduates, regardless of their discipline, because of maturity and education. However, we also predict differences in critical thinking abilities between philosophy and science graduate students (to the advantage of the former), and among different kinds of graduate students (to the advantage of people working on complex problems characterized by probabilistic approaches, such as psychology and organismal biology). Overall, much more needs to be understood about the relationship among factual knowledge of science, its conceptual understanding, critical thinking, and belief in pseudoscience (the latter, incidentally, in itself not representing a homogeneous category, with surveys showing distinctions between different kinds of pseudoscientific belief: Anonymous, 2001; Goode, 2002). Certainly, we cannot assume that all we need to do to improve critical thinking and reasonable skepticism is to teach more science facts. References Anonymous. (2001). Science indicators 2000: Belief in the paranormal or pseudoscience. Skeptical Inquirer, 12-15. Belzer, S., Miller, M. & Shoemake, S. (2003). Concepts in biology: A supplemental study skills course designed to improve introductory students' skills for learning biology. The American Biology Teacher, 65(1) 30-39. Carrier, R. (2001). Test Your Scientific Literacy! Available online at: http//www.infidels.org/library/modern/ richard_carrier/SciLit.html. Casgrain, P. & Legendre, P. (2002). Permute! Version 4.0. Available online at http://www.fas.umontreal.ca/biol/ casgrain/en/labo/permute. Ede, A. (2000). Has science education become an enemy of scientific rationality? Skeptical Inquirer, 24(4) 48-51. Goode, E. (2002). Education, scientific knowledge, and belief in the paranormal. Skeptical Inquirer, 26(1), 24-27. PSEUDOSCIENTIFIC CLAIMS 547 Irwin, H. J. (1985). A study of measurement and correlates of paranormal belief. Journal of the American Society for Psychical Research, 79, 301-326. Lehman, D. R., Lempter, R. O. & Nisbett, R. E. (1988). The effects of graduate training on reasoning: Formal discipline and thinking about everyday-life events. American Psychologist 43, 431-442. Sneath, P. H. A. & Sokal, R. R. (1973). Numerical Taxonomy. San Francisco: W.H. Freeman & Co. Sokal, R. R. & Rohlf, F. J. (2000). Biometry. New York, NY: W.H. Freeman. Sundberg, M. D., Dini, M. L. & Li, E. (1994). Decreasing course content improves student comprehension of science and attitudes towards science in freshman biology. Journal of Research in Science Teaching, 31, 679693. Vitulli, W. F., Tipton, S. M. & Rowe, J. L. (1999). Beliefs in the paranormal: age and sex differences among elderly persons and undergraduate students. Psychological Reports, 85, 847-855. Walker, W. R. & Hoekstra, S. J. (2002). Science education is no guarantee of skepticism. Skeptic, 9(3), 24-27. Wandersee, J. H. (1990). Concept mapping and the cartography of cognition. Journal of Research in Science Teaching, 27, 923-936. 548 THE AMERICAN BIOLOGY TEACHER, VOLUME 66, NO. 8, OCTOBER