Teaching Philosophy, 21:4, December 1998 373 © Teaching Philosophy, 1998. All rights reserved. 0145-5788/98/2132-0237 Philosophers' Views on the Use of Non-Essay Assessment Methods: Discussion of an E-Mail Survey STEPHEN PALMQUIST Hong Kong Baptist University The Survey: Its Background and Limitations How can conventional testing methods best be utilized to assess student achievement in introductory philosophy classes? Are essay exams the only option? Or can so-called "objective" questions also play a proper role? What is the purpose of administering philosophy examinations? Questions such as these loomed large in my mind during the Spring semester of 1996 at the prospect of participating in a departmental discussion on the suitability of a particular way of setting final examinations for introductory philosophy classes. Having studied and/or taught philosophy on three continents over the past 20 years, I knew that different approaches to philosophical education exist in various parts of the world, each tending to be regarded as the only legitimate way by those who have never experienced other approaches. In hopes of collecting some empirical evidence of such variety, I composed a simple survey and sent it to five electronic mailing lists (Philos-l, Philosop, Bridge-L, Kant-L, and the NAKS E-Mail Directory). This paper reports and discusses the responses sent from 145 people over a period of approximately one week. Although the survey was originally intended only for my own private use, the results are worth publishing for the following reasons. Firstly, the unexpectedly large number of responses indicates that the problems raised by the survey are far from being unique to my situation. Secondly, many respondents added extensive comments,1 some revealing that the survey obviously touched a sore spot. Several comments included specific PB STEPHEN PALMQUIST374 requests that the results be made public. Thirdly, the responses ended up not only confirming my fundamental hypothesis, that a wide variety of views exists on the relevant issues, but also suggesting a number of surprisingly definite conclusions. The latter are likely to be of interest to anyone who teaches introductory philosophy classes, and will, I hope, inspire the whole community of philosophy teachers to be somewhat more open-minded about methods of assessment. Finally, the results are worth publishing simply as an example of what can be accomplished by means of an e-mail survey. After a brief introduction, defining as "objective" any question of the "multiple choice, fill in the blank, matching" variety,2 the questionnaire requested yes-or-no responses to four statements: 1. I have been a student in an introductory (or other) philosophy class in which the teacher included objective questions on the final examination. 2. As a student, I believed that the only legitimate kind of final examination for any philosophy class (other than logic) would be an essay exam. 3. I have included objective questions on the final examination of an introductory (or other) philosophy class taught by me. 4. I believe it would be proper for a philosophy department to ban the use of objective questions on all final examinations other than those dealing exclusively with logic. Respondents were encouraged to supplement their yes/no responses by explaining/defending their views, either immediately after each question or in a special section for comments at the end.3 Respondents raised two significant points concerning the "idea" of the survey itself: one noting the irony of using a yes/no style questionnaire to gather views about the use of "objective" questions,4 and the other drawing attention to ambiguities in my use of the word "objective." Concerning the first point, a short discussion arose on one of the e-lists as to whether or not the questionnaire "begs the question" by itself being written in an "objective" (yes-or-no) format. The consensus was that this impression is only apparent, since the purpose of the survey was only to collect data, not to administer a philosophy examination. Moreover, respondents were free to add comments if they so desired, thus providing ample opportunity for supplementing the "objective" answer with a "subjective" explanation. (Here, as throughout this paper, these terms distinguish questions with definite, black-and-white answers from those requiring some value judgment in order to be assessed.) On the second point, numerous respondents insisted that all types of examination are bound to include both objective and subjective aspects, and that an exam's level NON-ESSAY ASSESSMENT METHODS 375 of objectivity depends much more on how the questions are set and on how the teacher grades them than on exactly what format is used.5 Naturally, I fully agree with this important point. My use of "objective" was adopted merely for the sake of convenience, following one of its conventional senses. In retrospect, that was probably a mistake. For this reason, this article uses the more precise term "non-essay" in place of "objective"-though this still leaves a small gray area, when it comes to identifying the status of short answer questions. Two other "background" issues raised by various respondents concern the relativity of assessment methods to the general educational system and to specific class objectives. The first issue arose implicitly as a result of an interesting tendency: respondents from universities where the educational system is of the "continuous assessment" type (as in most U.S. universities, where teachers give individual grades for each class) were more likely to support the idea of using non-essay assessment methods, whereas those from universities using a "cumulative assessment" system (as in most U.K. universities, where students attend "lectures," then take comprehensive exams at the end of the entire degree program) were more likely to regard essay exams as the only viable option. There are very good reasons for this difference; but this is not the place to discuss them. Suffice it to say that in the questionnaire I intended the phrase "philosophy class" to imply a continuous assessment educational system. Had this been specified more clearly, many of the respondents who answered "no" to the odd-numbered questions and "yes" to the evennumbered questions may well have answered differently (or perhaps not replied to the questionnaire at all). The more significant of these two issues is the relativity of assessment method to the class objectives. Many respondents voiced the opinion that the whole issue of the "legitimacy" of non-essay examination methods cannot be answered in the abstract, but must be tied to the stated objectives of the class in question. Thus, one person rejects the use of non-essay questions because they go "completely against my philosophy of what philosophy is; i.e. not factual data which can be thus memorized and regurgitated, but rather understanding ideas and becoming capable of thoughtful articulately [sic] these in well-constructed sentences which reflect the thinking process itself." Nine others likewise cite the role of "reasoning" or "critical"/ rigorous "thinking" in philosophy as ruling out the use of non-essay examination methods. This raises questions such as: Is the improvement of reasoning skills the only objective in teaching philosophy? or Is it true that "Only lazy and incompetent professors would ever dream of giving multiple choice finals?" Before discussing respondents' views on such important questions, let us examine the statistics. PB STEPHEN PALMQUIST376 The Respondents, the Statistics, and the Student-oriented Responses The survey ended with an "optional" section, asking respondents to name any universities where they have studied and/or taught philosophy. The statistics from this section are not very useful, because some respondents gave no answer, while others did not specify whether they taught or studied at the universities named. Likewise, exact statistics cannot be reported as to how many responses were submitted by philosophy students as opposed to teachers. Nevertheless, enough information was provided, either in this section or in respondents' other comments, to reach a close approximation. At least 89 respondents were (or had been) teachers, while at least 28 were still students; the other 28 respondents could have been either (or both). Of the many universities listed in this section, the following countries were represented: multiple responses (in order of frequency) from the U.S.A., Canada, England, Hong Kong, Australia, and Scotland; single responses from Bulgaria, Germany, Iceland, Norway, South Africa, Switzerland, Taiwan, United Arab Emirates, and Wales. Table 1 provides a statistical summary of the 141 usable responses (i.e., responses containing a clear answer to at least one of the four questions):6 Table 1: Statistics on Philosophy Exam Survey Table 1 splits the answers for questions 2 and 4 into different categories, according to how respondents answered Questions 1 and 3, respectively. These more specific statistics will prove to be significant in the ensuing discussion. The first and most obvious observation to make about these statistics is that they must indeed reflect the views of philosophers, because there is widespread disagreement on nearly every point! Seriously, certain interesting trends do emerge in spite of the fairly even distribution of NON-ESSAY ASSESSMENT METHODS 377 answers to the first three questions. The remainder of this section will illustrate several of these as they relate to Questions 1 and 2, leaving a more detailed examination of the supporting comments for another occasion. In the remaining two sections I shall then examine the statistics and comments relating to Questions 3 and 4, respectively. Of the 115 respondents with yes/no answers for Question 1, 47.8% report having taken final exams that included non-essay questions. One expresses strong appreciation ("I probably learnt more in that one introductory course than in any other"), while another conveys obvious distaste ("It was a joke"). One report of an exam "consist[ing] of about 200 T/F questions" being used for a major university's "basic course in metaphysics for first year graduate students" stands in stark contrast to the view echoed by several others, that non-essay questions should be used "only as an otherwise small part of the exam" (see note 14). Along these lines, another response states that "a 200 item multiple choice final testing ability to make picky distinctions between similar ideas would be an abomination in any course whether philosophy or not." Two responses specifically identify the form of non-essay question as multiple choice, while five others explicitly exclude such questions, three citing short answer formats instead. One response defends short answer questions on the grounds that they "are almost as easy to mark [as] True/False and Multiple Choice, are more revealing of the student's knowledge, and the exam can be made to cover more of a course's material than an essay exam can." Four of those who answered "no" to Question 1 add the proviso that non-essay questions were used in logic classes, and one that they were used during the semester on a quiz. Two add that short answer questions were used, so they (unlike some who answered "yes") apparently consider these to be closer in kind to essay than to non-essay questions. And one person notes that "most classes . . . didn't have final exams." Adjusting the statistics in line with these comments would render the numbers for affirmative and negative answers to Question 1 virtually identical. A very interesting inference can be drawn from the statistic relating Question 2 to Question 1. Most of the 52 affirmative responses to Question 2 (amounting to 48.1% of all 108 yes/no answers) came from those who claimed never to have taken a non-essay philosophy exam as students. To be precise, 80.8% (42) of the 52 who rejected the legitimacy of non-essay exams as students did so without a firsthand acquaintance with the full range of possibilities. Two responses make this position of ignorance quite explicit: "I did not realize that other kinds of examination were possible"; "I took it for granted that one had to write . . . essays." Others defend their "no" answers with weightier reasons, but also inadvertently suggest a certain narrowness PB STEPHEN PALMQUIST378 of thinking: "I can't think of any circumstances in which non-essay exams would seem appropriate"; "[w]hen I was [a] student objective questions were unthinkable in philosophy." (Note that 73.2% [41] of the 56 respondents who had experienced non-essay exam questions as students believe them to be appropriate for an introductory philosophy class.) Others suggest that explanations should be required even on logic exams and that "vivas" (oral exams) or "take-home" exams are also acceptable. One respondent, after answering affirmatively, backs off slightly by admitting there could be "an argument for including perhaps a few" non-essay questions "in addition to essay questions." Three of the "no" answers to Question 2 are negative only in order to make room for oral exams, while three others allow for short answers. Several responses, here as elsewhere in the survey, note that essay questions are better, "most important," or even indispensable ("to have no essay questions would be unacceptable" [see notes 9 and 14]), though "take-home essays . . . are even better." Two point out that non-essay questions can be a useful means of checking whether students are able to identify quotes from philosophers, case studies, etc. Strong disapproval of Question 2's universal condemnation of non-essay exams was expressed by only three respondents. As we shall see, subsequent questions elicited a much higher number of strongly worded condemnations. This difference is probably due to the fact that Questions 1 and 2 relate to the respondents' experience/belief as students, whereas Questions 3 and 4 ask for their (in most cases) more seasoned views, as teachers. Teachers' Views on Non-Essay Assessment Methods Some of the most interesting comments relating to the legitimacy of using non-essay questions on introductory philosophy exams came in response to Question 3. Here teachers reflect many of the same attitudes mentioned in the comments to Questions 1 and 2, but in more detail. That logic and/or critical thinking classes are a special case is again generally accepted (mentioned in at least 10 responses), though not without some uncertainty as to how logic teachers should answer Question 3: "If logic counts, then almost everyone who has taught logic will answer 'yes.'" Eight respondents specifically name multiple choice as a format they have used, while one other confesses: "All of the faculty used 'objective' type tests in [large] introductory level philosophy courses." Four express dissatisfaction with the results of using non-essay questions, and apparently no longer use them. Of those teachers who have not used non-essay questions on final exams, seven note that they do use them during the semester-e.g., NON-ESSAY ASSESSMENT METHODS 379 "for checking students' reading progress." Four see no objection "in principle" to their use.7 And four do use "short answer questions" on final exams. Four other respondents answer "no" on Question 3 even though they are or have been teacher's aides in a class whose final exam used non-essay questions. Taking these qualifications into consideration, together with the fact that only a few responses offer strong objections at this point, we can see that the general tone of the responses is weighted more towards the "yes" side than the statistics alone (with their even split on Question 3) would indicate. Teachers who reject non-essay assessment methods often do so because they believe "essays are the only way both for the students to discover just how they do comprehend the ideas and their connections and for the teacher to find out the same." Of course, nobody disputes that essay questions can indeed serve this purpose; the dispute is whether they are the only way, or even always the best way, to assess a student's comprehension of "the ideas and their connections." In what follows I shall first look at some of the limitations of essay questions, as pointed out by various respondents. I shall then examine what situations are generally regarded as appropriate for using non-essay examination questions. Finally, I shall note some creative strategies that were suggested for improving the way nonessay questions are administered. Essay examination questions, despite being viewed almost universally as the best (if not the only legitimate) method of assessing students' philosophical abilities, are not without their limitations. Firstly, the testing environment as such is not conducive to doing philosophy, so an assessment method based entirely on philosophizing may be unfair. Along these lines, one respondent contrasts "taking a test" and "doing philosophy": "in an examination hall, tired and under pressure of time and worries about other exams, a student is not in the best position for demonstrating his ability to do philosophy. A good candidate may do badly so, in a certain sense, the exam is unfair." And another respondent doesn't "think that test environments are the correct place to have students try to develop arguments for the first time." A second limitation relates to the potential for ambiguity and misuse in setting essay questions. Just before recommending the use of short answer questions, one respondent opines: "Often, essay questions permit equivocal answers. This leaves a certain amount of discretion on the part of the instructor but it also means that the student's opinion of his/her work and the instructor's opinion have a significant chance of divergence." This danger, no doubt, is what gave rise to the convention of referring to essay questions as "subjective" and non-essay questions as "objective." Recognizing that PB STEPHEN PALMQUIST380 "essay tests can be totally arbitrary and are called subjective tests for good reason" enables us to see that such questions can be used as an excuse for "lazy and incompetent" grading just as easily as nonessay questions can-if not more so. Perhaps this is why one respondent suggests that in a department with "a high proportion of complaints about marks," using non-essay questions "may serve to offset any further complaints." A third limitation relates to the subjectivity inevitably involved in grading essays. Non-essay questions are often criticized for testing nothing but the students' ability to memorize. However, essay questions are sometimes graded in much the same (quasi-objective) way: teachers may look for evidence that students can repeat key words and phrases taught in class, giving little or no weight to students' original insights.8 One reason for this is that making a good subjective distinction between a genuine insight and an idea the student has simply borrowed (memorized) from elsewhere can be very difficult. Similarly, as one respondent puts it: "It is often hard in [grading] essay questions to tell the difference between students who are vague and confused and those who are just guessing." For instance, my own students tend to study in groups. As a result, their grades often fall into clusters, with several students writing very similar (sometimes poor!) essays. How to assign accurate grades in such situations is a source of considerable concern: some students have obviously just memorized the "model answer" constructed by their study group, while others have participated actively in constructing the model. Sometimes it is simply impossible to tell the difference. In such cases I would agree that "objective [i.e., non-essay] questions give the most reliable measure of student competence." One response points out that "if the instructor cannot be objective [i.e., impartial] on the subjective part [i.e., essays], there should be an objective [i.e., non-essay] component." This is good advice, for the benefit of both teachers and students. When the inability to make good subjective judgments is due to the teacher's incompetence, then of course, heeding such advice is unlikely to solve the problem, since non-essay formats also leave "lots of room for less than objective structuring." But when the "inability" is unavoidable (as when students all tend to memorize the same few "model" answers), then a non-essay component may be just what is needed to double-check or compensate for the (in)accuracy of the grade assigned for the essay portion of the exam. Thus one respondent uses non-essay questions in order "to avoid having students' philosophical agreements/ disagreements with me affect their grade"-a confession I believe implies not incompetence, but a realistic appreciation of the role prejudices inevitably play in making rational evaluations. NON-ESSAY ASSESSMENT METHODS 381 What other situations can give rise to such a necessity? The rationale most frequently cited by respondents is the factor of class size. Few (if any) responses support the use of non-essay questions for small classes.9 But at least five relate details of huge, "overcrowded" classes that necessitate the use of such assessment methods. Some would willingly admit that such "dubious pedagogical methods" are used merely "for expediency" of grading, or blame "the state" for inadequate funding. One argues that being "relieved" from the "tedium" of "grading countless poorly written essays and papers" proves to be a "morale" booster that benefits everyone, including the students. The main reason why large classes legitimize the use of non-essay questions is that even the most conscientious teacher cannot avoid some degree of bias in grading essays: the teacher cannot get to know most of the students (a factor that would assist in interpreting an essay more accurately), is unlikely to have adequate time to read each essay with care (given the often short time frame between the day of the examination and the day teachers must submit the grades), and may be more inclined to ignore individual strong points in a quick search for a particular "model answer." Assigning non-essay questions that can be graded quickly without sacrificing the "objectivity" of the results (provided the questions are carefully written [see notes 8 and 13]) may be a good way of overcoming such obstacles. Another factor to consider in deciding whether or not non-essay questions are appropriate is the expected capability of the students being taught. Support for such assessment methods tends to be less common or weaker in responses from those who study or teach at top universities, than in responses from those at universities that admit students with a lower level of average "raw intelligence." A plausible explanation for this is that less intelligent students are likely to have more difficulty expressing their actual learning level in essay form. One respondent expresses this rather crassly from the teacher's point of view: "The dumber the students, the less inclined one is to wade through pages of nonsense"-which could explain why another teacher "would like to see a system where some set level of success (60% or more) at objective questions is a kind of qualifier to being permitted to write essay questions, or sit an oral exam at all." A similar (though more judiciously worded) rationale adopts the students' point of view: "The important thing seems to be that the examination, however it is construed, allows students of all abilities to fairly demonstrate their skills. Some one or two answer essay questions are 'hit or miss' in that if a student misconstrues the question or is unable to parse good argumentative answers, s/he will fail utterly. 'Objective' questions allow a more even gradient of PB STEPHEN PALMQUIST382 ability/understand[ing] to be demonstrated, though the sort of ability demonstrated isn't that which is most highly prized in philosophy." A similar issue is the likely motivation level of the students. I have rarely given a non-essay examination to students taking a class in their major discipline. In addition to being smaller than most general education classes, major classes normally have students with a comparatively high level of motivation. But when teaching non-majors, particularly in a university where most students are taking the class only to fulfill a general education requirement, the situation is totally different. Students usually regard essay exams as far easier than nonessay exams, and will study harder if they know they will be tested on their "knowledge base" as well as on their thinking ability. This "coercive" aspect of non-essay questions is acknowledged by at least eight respondents. One notes that, by motivating students to do the readings, the presence of non-essay questions can ironically help improve the quality of the essay answer(s) written for the same exam. The general principle here (considered "obvious" by at least one respondent) is that the more advanced the students, the more self-motivated they should be to secure an adequate knowledge base independently, so the less emphasis should be put on non-essay assessment methods. A rather different problem that sometimes arises in grading essay exams is that all the answers begin to look alike. Grades often tend to be clumped together, making a normal grading curve difficult to work out. If a class is being taught at an institution that monitors final grades (e.g., at my university teachers must justify a set of grades if it deviates from the established norm of 5–15% As and 40– 60% Bs), then non-essay questions can be a very effective way of separating the very good and very poor students from the mediocre mass that hovers in between. Whereas "essay tests tend to yield a fair amount of vague, fogged-up answers that might 'get them by,'" as well as those written by students who are able to "fake" a "thoughtful" answer, the genuinely "poor students cannot choose the correct option" on a non-essay exam; "good students," by contrast, "should know" how to answer the non-essay questions as well as writing a thoughtful essay (cf. note 8). This is an especially important point to keep in mind in situations where "the exam must be refereed": "It is easier to defend the mark given for an objective exam! On the other hand, it is easier to fail an objective exam. [Giving] objective exams . . . [serves] to separate the sheep and the goats." (Another respondent employs exactly the same metaphor: "definitions and fill-in-the-blanks [can be used] to separate the sheep from the goats: those who know something will find such sections a boost to their grade, and those who really know nothing will bomb.") In NON-ESSAY ASSESSMENT METHODS 383 light of statements such as these, we can understand why one respondent "would be surprised if final exams didn't have at least some 'objective' questions to see if students have mastered vocabulary and elementary facts." Having now described several types of situation wherein many philosophy teachers would regard non-essay questions as appropriate, I should add that very few respondents would approve of employing them as the sole method of assessment. Although three or four respondents admit having done so, at least 22 maintain (explicitly or implicitly) that non-essay questions should be used only together with one or more essay(s).10 Most respondents would agree that "an exam that was primarily objective [would be] odd." The oddity comes from the disparity between the relatively mechanical nature of many nonessay exams (sometimes even graded by a machine, as two respondents note!) and the notion that the main purpose for studying philosophy is to enhance one's thinking and reasoning skills. This leads us to the final topic of this section. Can non-essay questions be used to assess reasoning skills? Many respondents assume they can do little or nothing in this respect: "T/F questions do not test [a student's reasoning] ability"; "Multiple choice questions cannot test a student's philosophical ability"; "to have a multiple choice question about philosophical issues seems to me to be a travesty. It does nothing to show what the student has learned or how much she has thought about it."11 While everyone would agree that "ill-conceived true/false or multiple choice questions can be disastrous," some reject the "usually undefended" view that they merely test memory: "There is no good reason to presume that multiple choice questions examine only a student's ability to memorize. . . . [Such] questions could easily be devised which tested a student's understanding." One respondent claims "a well-constructed MC question can test conceptual distinctions quite accurately." Likewise, despite answering "yes" to Question 2, another respondent admits: "It may be possible to improve the testing value of [non-essay] tests . . . meaning that the format has perhaps not been exploited to its max. What I am thinking of is that the questions would involve such [a] thought process that in order to get it right, the student would have to traverse down quite a path of reasoning." And a teacher who does not actually use non-essay questions nevertheless emphasizes that "demonstrating the ability to just understand difficult philosophical ideas is a considerable achievement, not to be belittled. And it would be possible, through a subtly composed multiple-choice test, to determine whether a candidate had achieved such understanding" (see note 13). Six respondents share at considerable length examples of how they have constructed multiple choice questions in hopes of avoiding PB STEPHEN PALMQUIST384 some of their well-known drawbacks. To compensate for the possibility of ambiguous questions allowing alternative interpretations, one respondent encourages students to complain if they disagree with the teacher's choice of a "right" answer, while another gives up to full credit to those who "write their comments/challenges . . . on a separate sheet during the exam." Another respondent suggests an interesting "variation": students are asked to choose the best of several right answers, with "the answers [being] given different values" (i.e., a different number of points) as "determined by the accuracy and/or completeness of the answer." This avoids the charge that students do not need to connect and evaluate different ideas when answering nonessay questions. One teacher is reported to have required students to write sample exam questions in a variety of formats as one of their semester projects, with the best questions being used for the actual final exam. Finally, a potential benefit of using non-essay questions that is not often taken very seriously is mentioned by a respondent who admits to using multiple choice questions that were "for the most part . . . trivial pursuits, more of busy work to get the students working on the exam, and break the stress that attends exams. As such, the objective part of the exam was worth very little, had only 20 questions, and for the most part was humorous. . . . I hope its inclusion pointed out the ridiculousness of multiple-guess exams in philosophy." Humor and other diversionary tactics can be a very effective way of establishing rapport with students, even in an examination. It can remind the students that this is, after all, just an examination, and that it is therefore bound to be rather far removed from the real, day-to-day application of what the class has really been about-namely, learning to do philosophy. By relaxing the students and encouraging them to put the examination in its proper perspective, such questions can actually help them perform better on the essay part of the exam. Views on "Banning" Non-Essay Examination Questions Originally the main purpose of the survey was to collect opinions on the issue of whether or not philosophy departments ought to "ban" all use of non-essay questions on final examinations (excluding logic). Not surprisingly, a comfortable majority of respondents with definite yes/no answers (75.8%) oppose such a ban-though a fairly significant number (30) do express a willingness to see such a ban imposed. At least 10 of those with "yes" answers to Question 4 (all those with "n/a" for Question 3), not being teachers, may not have appreciated all the issues at stake. But 10–12 respondents support their "yes" answers with reasons or significant explanations. NON-ESSAY ASSESSMENT METHODS 385 Many of the peripheral issues raised in the comments on Question 4 have already been discussed in the foregoing sections. Of the 30 "yes" respondents, only three express emphatic support for such a ban, though two others have reservations about allowing non-essay questions even on logic exams. One reports actually being in a department that has imposed such a ban. Another (teaching, significantly, at a university with a cumulative assessment system) goes so far as to recommend "hiring committees to ask candidates for their views on examination methodology"-presumably in order to weed out those who might have in mind polluting the system with non-essay questions! But most of the "yes" answers are surprisingly non-committal- e.g., one adds "I could make an argument for the contrary too," while three others reveal an openness to using multiple choice questions in contexts other than the final exam. No affirmative response is supported by any argument we have not yet discussed. The reasons cited in favor of banning are: "The only possible 'justification' for [using non-essay questions] . . . is that it would make our job easier"; using "objective questions . . . goes against the very essence of philosophy," whose purpose is to improve "reading, writing and reasoning skills, rather than come to know [superficial facts]"; their use "would be self defeating to the philosophic and pedagogic exercise." In the previous sections, we have found ample reasons to doubt such sweeping and poorly grounded claims in favor of a more balanced recognition of the subtleties involved in teaching philosophy. In contrast to such extreme claims, one respondent offers the more cautious view that there are "some circumstances in which it would be appropriate" to ban non-essay questions on philosophy exams. Indeed, the arguments discussed in §3 could be extended to support imposing certain restrictions in an educational system that uses only cumulative assessment, or in a continuous assessment system that is so well endowed that all philosophy classes are very small. Along these lines, two respondents limit their approval of a ban to the hypothetical situation in which "infrastructure modifications" would allow for a more personal approach to education. Such qualifications implicitly support the use of non-essay questions in situations that involve mass education. Thus, thinking more realistically, one person cites "growing student numbers and ever-shrinking resources" as a reason for opposing such a ban. As we saw in §3, such opposition need not be based merely on expedience (i.e., on large classes making it too time-consuming to grade all-essay exams); rather, it is more likely a response to the problem of subjectivity (i.e., mass education making it too difficult to form an accurate judgment of individual achievement without using some non-essay questions, the latter being something like a "necessary evil"). PB STEPHEN PALMQUIST386 The problems with actually implementing such a ban are very forcefully articulated by many of the respondents who answer "no" to Question 4. The most obvious and frequently cited point (mentioned by at least 12 respondents) is that such a ban could constitute a very significant "violation" or "infringement" of "academic freedom." Because of its close connection to this sensitive issue, Question 4 evoked by far the greatest number of emphatic (negative) responses (at least 18): "This is just silly"; "Absolutely not. . . . Frankly the notion . . . is just silly"; "highly improper"; "Absolutely not!"; "To even consider such a 'banning' is outrageous!"; "Absolutely not"; "That is silly and mindless"; "Of course not!"; "Absolutely not!"; "Ban? Ugh"; "Yikes!"; "The idea . . . is patently absurd!"; "this is a ridiculous idea"; "No way!"; "Ban??!! Censure the prof??!! no." Such strong expressions are especially striking in light of the fact that many philosophers believe "there is no absolutely right answer" to philosophical questions! Numerous respondents argue, often in connection with academic freedom, that the very idea of a "ban" implies a "lack of confidence in one's colleagues, which seems rather odd," is not befitting to "professionals," and lacks a clear purpose, "especially in a philosophy department!" To resort to such a "draconian" policy is to beg "a substantial metaphilosophical question" by merely assuming "that philosophy can impart no valuable objective knowledge." Yet this is totally contrary to the spirit of philosophy: It would be to concede that there really are no persuasive arguments against the use of objective questions, and so the department must resort to force and coercion. When philosophers resort to bans rather than reasons, they betray philosophy! Indeed, such a banning would clearly be a banning. . . . This would be [a] shameful practice for a legitimate philosophy department. It would be quite consistent with small minded ideological purists who would force their ideology on the[ir] fellows. Despite such strong opposition to banning, some "no" respondents do acknowledge that less forceful measures could be appropriate. The most that would be "proper for those who dislike objective questions to do is to pass a general recommendation" discouraging their use, or "to lay down informal (but strongly encouraged) guidelines about what is and is not appropriate for 'objective questions.'" Otherwise, the resulting lack of "respect for the autonomy of the individual instructor" would be likely to stifle creativity: "An outright ban might well have a chilling effect on pedagogical creativity and diversity, which would be a shame." Instead, by "allow[ing] others to explore the implications of different pedagogical decisions," we should "learn NON-ESSAY ASSESSMENT METHODS 387 from them." "Teachers need to be allowed to experiment." "Banning objective questions outright simply diminishes the tools with which an educator may gauge the progress and abilities of his/her students." As one award-winning teacher says, a department imposing such a ban would be "blind to the possibilities of improving the quality of teaching. . . . Preventing teachers from innovation in teaching will guarantee a mediocre teaching environment." And the chair of a department wherein "roughly half of the profs use objective questions on final exams" confides that "there'd be a revolt here if the department tried to dictate the form of the final examination to the tenured professor running the course." The only way to insure impartiality in a department that imposed a "subjective stance" in such a "heavyhanded" way, claims one respondent, would be for each member "to read the examinations written by the students of all the other members of the department." The "no" answers to Question 4 also contain many comments that are very similar to those we have already discussed. Non-essay questions can be "thought-provoking for many students, stimulating a desire to acquire further knowledge." They are especially suitable for introductory and undergraduate level classes, where many students may "not [be] able to rise to the occasion of [writing] a long answer in an exam setting," and where it is important for an exam "to cover more material" in order to assess whether the students have learned the "basic facts" of the subject (sentiments echoed by nine respondents). Thus, one respondent asks: "How can students even know what the philosophical points are if they are unable to answer objective questions?" Using non-essay questions can help prevent students from adopting the all-too-common belief that "philosophy is easy" because it's all based on personal opinions that simply "can't . . . [be] wrong" (though an exclusive use of such an assessment method, by contrast, "is likely to breed in students a false conception of philosophy" as focused on "facts about philosophers," rather than on "themselves engaging in the activity of philosophizing"). Moreover, such questions "reward those students who have really been paying attention," while "reduc[ing] the classes' anxiety over an 'all essay' exam."12 On a more cautious note, some respondents urge that such questions are better suited for use in tests or quizzes during the semester, that their use "should be subject to rigorous scrutiny and testing, and should be done with a clearly stated set of goals,"13 that an exclusive use or "overuse of objective questions" must be avoided,14 and that they should be mixed with essay questions (a view stated by 11 respondents). One respondent expresses a preference for "short answer" and/or "fill-in-the-blank" formats over "multiple choice or matching," PB STEPHEN PALMQUIST388 while another discounts the use of "'fill in the blank' strategies" as "gloomy signs of the times, along with other attempts to entertain students." Nine others, despite opposing a ban, state a preference against the use of non-essay questions, even to the point of being "angered" by their improper use. Two respondents note that in the U.S.A., "the philosophy GRE" (a multiple choice entrance exam for graduate school) no longer exists, because "it couldn't possibly test anything [graduate schools] wanted to know about" the applicants. And two suggest looking at assessment methods to help evaluate "tenure and promotion cases," and withholding tenure from those who make "life easy for themselves by giving such tests"-as if there could be no rationale other than laziness or incompetence! Some responses include lengthy comments at either the beginning or end of the questionnaire. Short excerpts from most of these have been incorporated into the foregoing discussion at some point, though in several cases I have had to pass over extended accounts of interesting personal opinions and/or experiences. Let it suffice to quote from just one more of these comments, in hopes of putting this whole nest of issues into its proper context: "There are serious doubts as to whether there should be examinations of any kind in philosophy." As we have seen, the exigencies of teaching philosophy, not the nature of philosophy itself, are what give rise to the need for flexibility in designing assessment methods. If there is any consensus among philosophers on the issues covered in this survey, it is that any ruling that disallows a teacher from using a widely accepted assessment strategy in a university that runs on the continuous assessment system would be highly inappropriate. Rather than attempting the impossible-i.e., identifying a more comprehensive set of "consensus" views that would be upheld by all or most respondents-I shall end on a personal note. Compiling the results of this survey has proved to be a far more interesting exercise than I originally anticipated. All too often we teach year-in and year-out without paying much attention to what our colleagues think about our own biases concerning examination assessment methods. The opportunity to be told firsthand by 145 philosophy teachers and students exactly what they think about this issue has made me appreciate how Woody Allen must have felt when he "cheated on a metaphysics exam by peeking into the soul of his neighbor."15 Notes 1. This paper will present short excerpts from many of these comments. The complete set of comments received is available on my web site (see note 3). NON-ESSAY ASSESSMENT METHODS 389 2. As quite a few respondents pointed out, this partial list does not clearly specify what other forms of question were also to be regarded as "objective." My intention was to include true/false questions, but not essays, short answers, or oral (viva voca) examinations. Some of the statistics are slightly inaccurate due to the fact that some respondents interpreted "objective" in somewhat different ways. Using the term "non-essay" in place of "objective" largely avoids this ambiguity. 3. E-mail messages were received at two separate addresses. Responses sent to one address were numbered 1–82; those sent to the other address were numbered 101–163. After numbering all 145 replies in this way, I compiled the comments into four files and placed them on my web site. These files (with names, addresses and other personal remarks deleted) can be accessed from the web page located at: http://www.hkbu.edu.hk/~ppp/srp/examquest.html. Comments are listed in order of the response number and grouped according to the question number and the type of answer given-i.e., either a "y" ("yes"), "n" ("no"), "x" ("blank, no answer, or unclear answer"), or "C" (indicating that the comment was added separately-usually at the end-rather than as part of the answer to a specific question). For the sake of readability, I have omitted all such references throughout this essay, though I have preserved the quotation marks to indicate when I am borrowing the views of respondents verbatim. Readers who are interested in reading the quotations in their original context may consult the version of this paper that appears on my web site (accessible from a link provided on the web page cited above); it contains precise references to the source of each quote, which can then be looked up in the four web files containing the comments themselves. 4. One response playfully suggests the questionnaire "is a bit self-referential, isn't it now?" Another begins by declaring: "I won't fill in your questionnaire (preferring to write an essay on the subject)." 5. One respondent writes: "I'm sure many of your respondents will agree with me that the 'objective/non-objective' distinction is relatively inappropriate here." Another agrees: "I would not call multiple choice exams 'objective.' They are no more objective than written exams." Along the same lines, a third respondent suggests that "it would in fact be more philosophically objective to test for the construction . . . of an argument" in an essay. 6. Four of the 145 respondents did not give clear and definite answers to any of the questions. Their responses were therefore excluded from the overall statistics, though their comments were assigned a number and included in the files placed on my web site (see note 3). Incidentally, my own answers to the four questions (not included in the statistics) would be: (1) I don't remember; (2) No; (3) Yes; (4) No. That the wording of the survey was not biased in favor of these answers is evident, because several responses took for granted that my motive in doing the survey was to gain support for the idea of banning all non-essay questions, whereas only one assumed the opposite (which is in fact the case). 7. Thus, one respondent does not regard non-essay questions as "somehow illegitimate or otherwise pedagogically subpar as such," while another "could conceive of doing it, and of making the experience a challenging and learning event for students." In spite of opposing the use of non-essay questions "at any level" because of the "serious danger" they present of misleading students as to the nature of philosophy, one respondent admits that "it wouldn't be so bad to PB STEPHEN PALMQUIST390 spring" such questions on students "at the end"; they "could have a place in the assessment process," provided "students had done philosophy throughout the year, and it was clear to them from the way their essays were marked that what is valued is rigorous argument, clear presentation, and originality." 8. For instance, the essay exams given by one respondent can be regarded as "objective in the sense that I am looking for a correct rendition of an interpretation of the philosopher in question and not the students' opinions on whether the philosopher is right or not." Employing an essay format that discourages students from expressing their own insights and appreciating the immense diversity of potential interpretations strikes me as being a far greater "travesty" of genuine philosophy than employing assessment methods that are designed to be objective and less thought-provoking. The other extreme is taken, however, by "French institutions of national education": utilizing the cumulative assessment method, they require teachers "to avoid in our final exams" any questions relating too closely to "the lesson taught," because such questions are "more favorable to the memory process than to real reflection." 9. One staunch defender of non-essay formats as not implying laziness or incompetence may be an exception: "My students are probably more informed about philosophy than many others and are as able to engage in critical thinking as any in which essay tests are used. In fact, when I have used both kinds of tests in the same class, I discover that the performances and grades are about the same. There is one big difference, however: students do not have the luxury of [trying to pass off nonsense as genuine knowledge] in objective-format testing. If one of the aims of your program is to improve writing skills, an exam is not the place for it. That should be done as an ongoing (daily, weekly) set of assignments. But for large classes especially, objective format tests are my preferred means of testing. It means, however, that they be formulated with the utmost care. As much time goes into making good questions as one would devote to grading essays." One of the several respondents who comment on the issue of how difficult and time-consuming it is to write good non-essay questions (see also note 13) writes: "It requires a good bit of work to construct penetrating [non-essay] questions." 10. The most succinct of these opines "a mixture is the best," while another states: "The emphasis is on included! I would never base grades on objective examinations exclusively but find that some areas of knowledge can be tested effectively via objective questions, especially logic, terminology, and some purely (and boringly) factual information." 11. One respondent uses non-essay questions even though they "cannot develop anything but the most minimal intellectual and critical skills," utilizing instead "skills that have more to do with test-taking itself than philosophy"; their main purpose is more pragmatic, "to test whether or not a student has attended class regularly, paid attention to the lectures, and done the reading." Another uses them for very similar reasons, on the grounds that "there are objective issues" in teaching philosophy: "definitions of technical terms, short characterizations of arguments . . . even identifying quotations in which some trademark idea is set forth, etc. Lots of room for objective questions, and it helps determine the amount and depth of reading, and to focus the memory." 12. The danger, as one student points out, is that such questions can actually "penalize" students "who have done more than [just "read the material"], but have also actually thought about it." Thus, a teacher laments at having "too NON-ESSAY ASSESSMENT METHODS 391 often seen good students over-think questions with even a hint of ambiguity and foul them up." As we saw in section 3, however, such dangers also apply to essays, and call attention more to the need for care in composing the questions, and flexibility in grading them, than to the impropriety of non-essay questions as such. 13. An example of how to carry out such a test is cited by a respondent who then opines "that so-called 'objective' formats, especially multiple-choice formats, can test for quite sophisticated forms of thinking." Another person who differs from the common view (i.e., that "objective exams test only how well a student has memorized a certain number of facts") writes non-essay questions that "place a strong emphasis on reasoning skills" and "little value in memorizing names or terms." Along these lines, one person tells the story of a class that requested a non-essay exam format only to discover "that 'objective' does not equal 'easy'" if the questions are written "in an adequately challenging . . . way" by "a professionally-trained philosopher." Another tells of a friend who designs non-essay questions in the form of "mini case studies" that require students to identify "some inference": "No one without the critical skills derived from careful reading and writing could have done well on his tests." 14. Thirteen respondents express views of this sort. One suggests it "might" be acceptable to "forbid professors from making the entire exam an objective question exam." The point is "that objective questions only become dangerous when misused, not when used at all." One such danger is highlighted by a respondent who recounts a story of students who "felt cheated when comparing their [all non-essay] exams with their colleagues in different sections of the course-they could not express themselves, so they said." 15. This quip was contributed by a member of one of the e-lists during a brief online discussion of ideas arising from the survey. Stephen Palmquist, Department of Religion and Philosophy, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China, StevePq@hkbu.edu.hk