Abstract
This paper considers ways that experimental design can affect judgments about informally presented context shifting experiments. Reasons are given to think that judgments about informal context shifting experiments are affected by an exclusive reliance on binary truth value judgments and by experimenter bias. Exclusive reliance on binary truth value judgments may produce experimental artifacts by obscuring important differences of degree between the phenomena being investigated. Experimenter bias is an effect generated when, for example, experimenters disclose (even unconsciously) their own beliefs about the outcome of an experiment. Eliminating experimenter bias from context shifting experiments makes it far less obvious what the “intuitive” responses to those experiments are. After it is shown how those different kinds of bias can affect judgments about informal context shifting experiments, those experiments are revised to control for those forms of bias. The upshot of these investigations is that participants in the contextualist debate who employ informal experiments should pay just as much attention to the design of their experiments as those who employ more formal experimental techniques if they want to avoid obscuring the phenomena they aim to uncover.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The “Basic Set” includes indexicals like “I”, “here”, and “today”, demonstratives like “you”, “she”, and “that”, and adjectives like “actual” and “present”.
(Stern (2009), p. 2) draws the helpful distinction between descriptive and explanatory methods of classifying positions in the contextualist debate.
For a representative statement of the role of intuitions of competent speakers as evidence for semantic theory, see (Larson and Segal (1995), p. 9). (Cappelen and Lepore (2005), p. 87) make the connection between speakers’ intuitions and contextualism explicit: “There is a sense...in which [contextualism] is an empirical thesis, based as it is on a variety of contingent features about human psychology, in particular, based on the contingent fact that we happen to have certain intuitions”. Other sources of evidence for linguistic theories include corpus data and non-meta-linguistic performace, though these sources (so far) only play a small role in philosophical reflection on language. See Devitt (2012), Krifka (2011), Ludlow (2005) and (2011, Chap. 3), Martí (2009), and Schütze (2011), for discussions of sources of linguistic evidence beyond the intuitions of competent speakers.
A complementary use of context shifting experiments is to disconfirm existing theories of the meaning of target expressions that would predict no change in judgments about the use of the target expression in two contexts.
An anonymous reviewer pointed out that there is much less uncertainty about how to respond to Travis’s milk case, and observed that one difference between the milk and bank cases is that “the bank case requests of us a verdict concerning a loaded philosophical term”. There is some experimental evidence that backs up the observation that judgments about the bank case are less clear-cut than judgments about the milk case: Hansen and Chemla (forthcoming) found significantly greater contrasts in judgments about “miscellaneous” cases (which include the milk case) than in judgments about “knowledge” cases (which included a revised version of the bank case).
Jonathan (Weinberg (2007), p. 335) claims that “gradation [in intuitions] is largely unexplored—and unexploited—by current philosophical practice”. But, as pointed out in (Schütze (1996), p. 62), linguists have acknowledged that judgments of linguistic acceptability come in various degrees of strength at least since Chomsky (1985) (Chomsky’s thesis from 1955). Chomsky says that “there is little doubt that speakers can fairly consistently order new utterances, never previously heard, with respect to their degree of ‘belongingness’ to the language” (p. 132). See (Schütze (1996), pp. 62–81) for discussion of the gradability of intuitions in linguistics.
One example of a philosopher making use of gradation in responses to thought experiments is Peter Unger (1982), who discusses what he calls dominant and dominated responses to thought experiments. A dominant response is felt more strongly than a dominated response, but both responses are felt to be plausible to some degree. Unger considers Putnam’s (1979, pp. 238–239) robot cat thought experiment, which asks us to imagine a world in which “there never have been any cats”, and in which everything we have thought to be a cat is in fact an artifact,
every movement [of which], every twitch of a muscle, every meow, every flicker of an eyelid is thought out by a man in a control center on Mars and is then executed by the cat’s body as the result of signals that emanate not from the cat’s ‘brain’ but from a highly miniaturized radio receiver located, let us say, in the cat’s pineal gland.
In the scenario Putnam imagines, the following three individually plausible claims cannot all be true:
-
1.
There are cats.
-
2.
Cats are animals.
-
3.
No animals are robots.
Putnam considers the different possible reactions that one might have to the thought experiment, each of which involves giving up one of the three claims (p. 239):
My own feeling is that to say that cats turned out not to be animals is to keep the meaning of both words unchanged. Someone else may feel that the correct thing is to say is [sic], “It’s turned out that there aren’t and never were any cats”. Someone else may feel that the correct thing to say is, “It’s turned out that some animals are robots”.
Unger agrees with Putnam that his “dominant” response to the thought experiment is to reject the claim that cats are animals, but he notes a “dominated” response that aligns with J.J. Katz’s intuition (cited in Putnam 1975, pp. 243–244) that in the scenario Putnam imagines, the correct thing to say is that “it’s turned out that there aren’t and never were any cats”. Unger feels the pull of both the dominant and the dominated responses at the same time. That complex intuition can be represented by ranking the possible responses to the robot cat thought experiment, rather than an all-or-nothing choice of one response. The same “quest for the elusive correct intuition” that Unger resists in the case of Putnam’s thought experiment can be resisted in context shifting experiments as well.
-
1.
This is a modified version of a control scenario employed in Hansen and Chemla (forthcoming).
There is disagreement about the advantages of magnitude estimation over interval scales in measuring responses to linguistic phenomena. (Schütze (2011), pp. 213–214 ) summarizes arguments for and against the superiority of magnitude estimation over interval scales. On one hand, magnitude estimation “allows participants to distinguish as many levels of acceptability as they can perceive, unlike tasks in which they are limited to five or seven choices”, and “the intervals between levels of acceptability are measured in uniform units calibrated by the acceptability of the [standard]”, while Likert scales may be interpreted unevenly by participants. On the other hand, participants employing magnitude estimation often do not seem to be actually guided in their responses by the value assigned to the standard, it has been found that there is more unexplained variance in magnitude estimation response tasks than in standard Likert scale tasks, and magnitude estimation tasks are more cognitively demanding for participants.
In this remark, Schaffer is commenting on the fact that characters in the context shifting experiments make assertions that align with the predictions of SSI.
In conversation, Aidan Gray and Sören Häggqvist pointed out that there may a subtle form of order of presentation bias hidden in Austin’s donkey scenario. Austin switches the order in which the options are given so that the preferred response appears second in each case. Emmanuel Chemla and I conducted a formal experiment that tested this claim, but we found no evidence that the order in which response options are presented for Austin’s donkey stories affects responses (Hansen and Chemla, Ms.).
The original version of the Thelma and Louise case is accompanied with the following gloss (DeRose 2009, p. 5):
When thinking about the case set-up and Thelma’s conversation at the tavern, and especially when not considering those in connection with Louise’ss discussion with the police, it can seem that
(4) When Thelma says at the tavern that Lena knows that John was at work, her claim is true.
But when considering the case set-up and Louise’s discussion with the police, especially when not worrying about how that discussion relates to Thelma’s talk at the tavern, it can seem that
(5) When Louise tells the police that Lena doesn’t know that John was at work, her claim is true.
See Sprouse and Almeida (Ms., p. 32).
For arguments in favor of the superiority of certain kinds of “expert” judgments in contrast with the judgments of naïve subjects in certain cases, see, e.g., Ludwig (2007) and Kornblith (2007). But Sprouse and Almeida, Ms., provides compelling evidence that the informal acceptability judgments of linguists align with the judgments of “ordinary” speakers made in controlled circumstances.
(DeRose (2009), p. 49) says that he thinks that contextualist intuitions about the bank scenario will be strongest when one considers the high and low standards contexts independently of one another: “[W]here the contextualist’s cases are well chosen, those are the fairly strong intuitions about the cases, at least where each case is considered individually”. DeRose says that he thinks that if the high and low standards contexts are presented together, “the pressure to give the same verdict about whether the subject knows in the two cases would be great”. But empirical studies have found that contrast effects actually enhance, rather than detract from, contextualist effects. See Hansen Ms. and Phelan forthcoming for discussion of contrast effects.
Thanks to the editors for emphasizing this point.
References
Alexander, J. (2012). Experimental philosophy: An introduction. Cambridge: Polity.
Austin, J. L. (1979). A plea for excuses. In Philosophical papers. Oxford: Oxford University Press.
Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72(1), 32–68.
Berg, J. (2002). Is semantics still possible? Journal of Pragmatics, 34(4), 349–359.
Boakes, R. A. (1984). From Darwin to behaviorism. Cambridge: Cambridge University Press.
Buckwalter, W. (2010). Knowledge isn’t closed on saturdays: A study in ordinary language. Review of Philosophy and Psychology, 1(3), 1–19.
Cappelen, H., & Lepore, E. (2005). Insensitive semantics: A defense of semantic minimalism and speech-act pluralism. Oxford: Blackwell.
Chemla, E. (2009a). Presuppositions of quantified sentences: Experimental data. Natural Language Semantics, 17(4), 299–340.
Chemla, E. (2009b). Universal implicatures and free choice effects: Experimental effects. Semantics and Pragmatics, 2(2), 1–33.
Chemla, E., & Spector, B. (2011). Experimental evidence for embedded implicatures. Journal of Semantics, 28(3), 359–400.
Chomsky, N. (1985). The logical structure of linguistic theory. Chicago: University of Chicago Press.
Cohen, S. (1999). Contextualism, skepticism, and the structure of reasons. Noûs, 33(13), 57–89.
Cullen, S. (2010). Survey-driven romanticism. Review of Philosophy and Psychology, 1(3), 275–296.
DeRose, K. (1992). Contextualism and knowledge attributions. Philosophy and Phenomenological Research, LII(4), 913–929.
DeRose, K. (2005). The ordinary language basis for contextualism. Philosophical Quarterly, 55(219), 172–198.
DeRose, K. (2009). The case for contextualism. Oxford: Oxford University Press.
DeRose, K. (2011). Contextualism, contrastivism, and X-Phi surveys. Philosophical studies, 156(1), 81–110.
Devitt, M. (2012). Whither experimental semantics? Theoria, 27(1), 5–36.
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PLoS ONE, 7(1), 1–7.
Geurts, B. (2002). Donkey business. Linguistics and Philosophy, 25(2), 129–156.
Fantl, J., & McGrath, M. (2002). Evidence, pragmatics, and justification. The Philosophical Review, 111(1), 67–94.
Hansen, N. (2012). On an alleged truth/falsity asymmetry in context shifting experiments. Philosophical Quarterly, 62(248), 530–545.
Hansen, N. (forthcoming). Contrasting cases. In J. Beebe (Ed.), Advances in experimental epistemology. New York: Continuum.
Hansen, N., & Chemla, E. (forthcoming). Experimenting on contextualism. Mind & Language.
Hansen, N., & Chemla, E. (Ms.). Beating a dead donkey: Linguistic experiments and ordinary language philosophy. Ms., University of Reading and LSCP.
Horn, L. R. (2001). A natural history of negation. Stanford: CSLI.
Intons-Peterson, M. J. (1983). Imagery paradigms: how vulnerable are they to experimenters’ expectations. Journal of Experimental Psychology: Human Perception and Performance, 9(3), 394–412.
Kornblith, H. (2007). Naturalism and intuitions. Grazer Philosophische Studien, 74, 27–49.
Krifka, M. (2011). Varieties of semantic evidence. In C. Maienborn, K. von Heusinger, & P. Portner (Eds.), Semantics: An international handbook of natural language and meaning. Berlin: de Gruyter.
Larson, S., & Segal, M. (1995). Knowledge of meaning: An introduction to semantic theory. Cambridge: MIT Press.
Ludlow, P. (2005). Contextualism and the new linguistic turn in epistemology. In G. Preyer & G. Peter (Eds.), Contextualism in philosophy: Knowledge, meaning and truth. Oxford: Oxford University Press.
Ludlow, P. (2011). The philosophy of generative linguistics. Oxford: Oxford University Press.
Ludwig, K. (2007). The epistemology of thought experiments. Midwest Studies in Philosophy, 31, 128–159.
Martí, G. (2009). Against semantic multi-culturalism. Analysis, 69(1), 42–48.
May, J., Sinnott-Armstrong, W., Hull, J. G., & Zimmerman, A. (2010). Practical interests, relevant alternatives, and knowledge attributions: An empirical study. Review of Philosophy and Psychology, 1(3), 265–273.
Myers, J. (2009). Syntactic judgment experiments. Language and Linguistics Compass, 3(1), 406–423.
Phelan, M. (forthcoming). Evidence that stakes don’t matter for evidence. Philosophical Psychology.
Pfungst, O. (1911). Clever hans. New York, NY: Holt, Reinhart and Winston.
Putnam, H. (1979). It ain’t necessarily so. In Mathematics, matter and method: philosophical papers (Vol. 1). Cambridge: Cambridge University Press. Cambridge University Press.
Putnam, H. (1975). The meaning of meaning. In Mind, language and reality: philosophical papers (Vol. 2). Cambridge: Cambridge University Press. University Press.
Recanati, F. (2004). Literal meaning. Cambridge: Cambridge University Press.
Rosenthal, R. (1976). Experimenter effects in behavioral research. New York: Irvington Publishers.
Pinillos, A. (2011). Some recent work in experimental epistemology. Philosophy Compass, 6(10), 675–688.
Schaffer, J. (2006). The irrelevance of the subject: Against subject-sensitive invariantism. Philosophical Studies, 127(1), 87–107.
Schaffer, J., & Knobe, J. (2012). Contrastive knowledge surveyed. Noûs, 46(4), 675–708.
Schütze, C. T. (1996). The empirical base of linguistics. Chicago: University of Chicago Press.
Schütze, C. T. (2011). Linguistic evidence and grammatical theory. WIREs Cognitive Science, 2(2), 206–221.
Sorensen, R. (1992). Thought experiments. Oxford: Oxford University Press.
Sprouse, J., & Almeida, D. (Ms.). A quantitative defense of linguistic, methodology. lingBuzz/001075.
Stanley, J. (2005). Knowledge and practical interests. Oxford: Oxford University Press.
Stern, J. (2009). Metaphor and minimalism. Philosophical Studies, 153(2), 273–298.
Szabó, S. Z. (2006). Sensitivity training. Mind & Language, 21(1), 31–38.
Travis, C. (1985). On what is strictly speaking true. Canadian Journal of Philosophy, 15(2), 345–366.
Travis, C. (1994). On constraints of generality. Proceedings of the Aristotelian Society, 94, 165–188.
Travis, C. (1989). The uses of sense: Wittgenstein’s philosophy of language. Oxford: Oxford University Press.
Unger, P. (1982). Towards a psychology of common sense. American Philosophical Quarterly, 19(2), 117–129.
Wason, P. C. (1961). Response to affirmative and negative binary statements. British Journal of Psychology, 52(2), 133–142.
Weinberg, J. M. (2007). How to challenge intuitions empirically without risking skepticism. Midwest Studies in Philosophy, 31(1), 318–343.
Acknowledgments
This paper has benefited from comments from audiences at the University of Western Ontario, the University of Wisconsin Madison, the University of Helsinki, Åbo Akademi, the University of Chicago, the New School for Social Research, Northwestern University, the University of Lisbon, the University of Aberdeen, the University of Barcelona, Institut Jean-Nicod, and Stockholm University. Thanks to Zed Adams, Kent Bach, Jonathan Berg, Jason Bridges, Emmanuel Chemla, Jim Conant, Jay Elliott, David Finkelstein, Aidan Gray, Sören Häggqvist, Chris Kennedy, Ernest Lepore, Chauncey Maher, Eliot Michaelson, François Recanati, Josef Stern, two anonymous reviewers, and the editors of this volume for comments and conversations about previous versions of this paper. The research leading to these results has received funding from the European Research Council under the European Community’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no. 229 441–CCC. The phrase “slugfest of intuitions” comes from Berg (2002).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hansen, N. A slugfest of intuitions: contextualism and experimental design. Synthese 190, 1771–1792 (2013). https://doi.org/10.1007/s11229-013-0261-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-013-0261-9