Skip to main content
Log in

Survey-Driven Romanticism

  • Published:
Review of Philosophy and Psychology Aims and scope Submit manuscript

Abstract

Despite well-established results in survey methodology, many experimental philosophers have not asked whether and in what way conclusions about folk intuitions follow from people’s responses to their surveys. Rather, they appear to have proceeded on the assumption that intuitions can be simply read off from survey responses. Survey research, however, is fraught with difficulties. I review some of the relevant literature—particularly focusing on the conversational pragmatic aspects of survey research—and consider its application to common experimental philosophy surveys. I argue for two claims. First, that experimental philosophers’ survey methodology leaves the facts about folk intuitions massively underdetermined; and second, that what has been regarded as evidence for the instability of philosophical intuitions is, at least in some cases, better accounted for in terms of subjects’ reactions to subtle pragmatic cues contained in the surveys.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Chart 1
Chart 2
Chart 3
Chart 4
Chart 5
Chart 6

Similar content being viewed by others

Notes

  1. Searching all of the electronic scholarly resources accessible from Melbourne University—which include all major philosophy, psychology, social and cognitive science journals—for the terms [“experimental philosophy” “survey methodology”] turned up only two results, neither of which contained a discussion of survey methodology. Searching for [Schwarz “Experimental philosophy”] yielded two additional results: Goldman and Pust (1998) who cite Schwarz (1995) in their defence of intuitions; and Doris et al. (2007) who cite Schwarz (1996).

  2. For an overview, see Krosnick (1999).

  3. Nichols et al. use the same survey in their (2003), as do Swain et al. (2008).

  4. By far the most significant effect Swain et al. uncovered is Dave’s case on Charles’ (p = 0.043). It therefore makes sense to test this case first.

  5. Responses are coded from 1 (strongly disagree) to 5 (strongly agree). Prospective subjects were approached at two Melbourne cafés. The experimenters introduced themselves and read the lead-in which was also printed at the top of the page. No demographic data was collected.

  6. In my survey of students at Melbourne University, a number of subjects made their guesses about the purpose of the research explicit in the open feedback question. One example is this student, evidently puzzled by the Gettier case she considered:

    I had to think about this for a long time. Its about accepting that you can know anything. [The person in the vignette] bases their belief on previous experience [...]. But the situation that is described makes me realise that there is no way to be sure you know anything. ...

    This student guessed that the experiment was about “accepting that you can know anything”. The experiment’s conversational context raised her standard for answering “really knows” to truly Cartesian heights: she did produce the philosophers’ response, viz., that the agent only believes that p, but for an entirely different reason to philosophers’. This was a very common theme in the students’ feedback (which is collected in the online appendix). Many who answered “only believes” claimed to have done so because, to quote another student’s response, “nobody can ever truly KNOW anything”.

  7. This result replicates Weinberg et al.’s finding: 32% of their Western subjects (N = 189) answered “really knows”.

  8. Responses were sorted by two people, one blind to the hypothesis of the study, into four categories: “internalist”, “externalist”, “sceptical”, and “other”. Agreement was high (>90%) about which responses indicated the internalist or externalist intuitions.

References

  • Alexander, Joshua and Jonathan M. Weinberg. 2007. Analytic epistemology and experimental philosophy. Philosophy Compass 2(1):56–80.

    Article  Google Scholar 

  • Ceci, Stephen J. 1991. How much does schooling influence general intelligence and its cognitive components?: A reassessment of the evidence. Developmental Psychology 27:703–722.

    Article  Google Scholar 

  • Doris, John M., Joshua Knobe, and Robert L. Woolfolk. 2007. Variantism about responsibility. Philosophical Perspectives 21(1):183–214.

    Google Scholar 

  • Goldman, Alvin and Joel Pust. 1998. Philosophical theory and intuitional evidence. In Rethinking intuition, eds. William Ramsey and Michael DePau. Totowa: Rowman & Littlefield.

    Google Scholar 

  • Haberstroh, Susanne, Daphna Oyserman, Norbert Schwarz, Ulrich Kuhnen, and Li Jun Ji. 2002. Is the interdependent self more sensitive to question context than the independent self? Self-construal and the observation of conversational norms. Journal of Experimental Social Psychology 38:323–329.

    Article  Google Scholar 

  • Higgins, Edward Tory and Arie Kruglanski, eds. 1996. Social psychology: Handbook of basic principles. Newyord: Guilford.

    Google Scholar 

  • Hofstede, Geert H. 2001. Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations. London: Sage.

    Google Scholar 

  • Holbrook, Allyson L., Jon A. Krosnick, David Moore, and Roger Tourangeau. 2007. Response order effects in dichotomous categorical questions presented orally: The impact of question and respondent attributes. Public Opinion Quarterly 71(3):325–348.

    Article  Google Scholar 

  • Jobe, Jared B. and David J. Mingay. 1991. Cognition and survey measurement: History and review. Applied Cognitive Psychology 5:175–192.

    Article  Google Scholar 

  • Johnson, Timothy, Patrick Kulesa, Isr Llc, Young Ik Cho, and Sharon Shavitt. 2005. The relation between culture and response styles. Journal of Cross-Cultural Psychology 36:264–277.

    Article  Google Scholar 

  • Kauppinen, Antti. 2007. The rise and fall of experimental philosophy. Philosophical Explorations 10(2):95–118.

    Article  Google Scholar 

  • Knobe, Joshua. 2006. The concept of intentional action: A case study in the uses of folk psychology. Philosophical Studies 130:203–231.

    Article  Google Scholar 

  • Knobe, Joshua. 2007. Experimental philosophy. Philosophy Compass 2:81–92.

    Article  Google Scholar 

  • Krauss, Robert M. and Susan R. Fussell. 1996. Social psychological models of interpersonal communication. In Social psychology: Handbook of basic principles, eds. Edward Tory Higgins and Arie Kruglanski. Newyord: Guilford.

    Google Scholar 

  • Krosnick, Jon A. 1999. Survey research. Annual Review of Psychology 67:537–567.

    Article  Google Scholar 

  • Krosnick, Jon A. and Michael A. Milburn. 1990. Conversational conventions, order of information acquisition, and the effect of base rates and individuating information on social judgements. Journal of Social Psychology 54:940–952.

    Google Scholar 

  • Lehrer, Keith. 1990. Theory of knowledge. Boulder: Westview.

    Google Scholar 

  • Machery, Edourd, Ron Mallon, Shaun Nichols, and Stephen P. Stich. 2004. Semantics, cross-cultural style. Cognition 92(3):B1–B12.

    Article  Google Scholar 

  • Nadelhoffer, Thomas and Eddy Nahmias. 2007. The past and future of experimental philosophy. Philosophical Explorations 10(2):123–149.

    Article  Google Scholar 

  • Nahmias, Eddy, Stephen Morris, Thomas Nadelhoffer, and J. Turner. 2005. Surveying freedom: Folk intuitions about free will and moral responsibility. Philosophical Psychology 18(5):561–584.

    Article  Google Scholar 

  • Nichols, Shaun. 2004. Folk concepts and intuitions: from philosophy to cognitive science. Trends in Cognitive Science 8(11):514–518.

    Article  Google Scholar 

  • Nichols, Shaun, Stephen P. Stich, and Jonathan M. Weinberg. 2003. Metaskepticism: Meditations in ethno-epistemology. In The skeptics: Contemporary essays. Ashgate epistemology and mind series, ed. Steven Luper. Aldershot: Ashgate.

    Google Scholar 

  • Norenzayan, Ara and Norbert Schwarz. 1999. Telling what they want to know: participants tailor causal attributions to researchers’ interests. European Journal of Social Psychology 29:1011–1020.

    Article  Google Scholar 

  • Oyserman, Daphna, Heather M. Coon, and Markus Kemmelmeier. 2002. Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin 128(1):3–72.

    Article  Google Scholar 

  • Schaeffer, Nora Cate and Stanley Presser. 2003. The science of asking questions. Annual Review of Sociology 29:65–88.

    Article  Google Scholar 

  • Schwarz, Norbert. 1995. What respondents learn from questionnaires: The survey interview and the logic of conversation. International Statistical Review/Revue Internationale de Statistique 63(2):153–168.

    Article  Google Scholar 

  • Schwarz, Norbert. 1996. Cognition and communication: Judgmental biases, research methods, and the logic of conversation. Mahwah: Erlbaum.

    Google Scholar 

  • Schwarz, Norbert, Fritz Strack, and Hans-Peter Mai. 1991. Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. The Public Opinion Quarterly 55:3–23.

    Article  Google Scholar 

  • Strack, Fritz, Leonard L. Martin, and Norbert Schwarz. 1988. Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology 18:429–442.

    Article  Google Scholar 

  • Sudman, Seymour, Norman M. Bradburn, and Norbert Schwarz. 1995. Thinking about answers: The application of cognitive process to survey methodology. Jossey-Bass.

  • Swain, Stacey, Joshua Alexander, and Jonathan M. Weinberg. 2008. The instability of philosophical intuitions: Running hot and cold on truetemp. Philosophy and Phenomenological Research 76(1):138–155.

    Google Scholar 

  • Tourangeau, Roger, Lance J. Rips, and Kenneth Rasinski. 2000. The psychology of survey response. Cambridge: Cambridge University Press.

    Google Scholar 

  • Weinberg, Jonathan M., Shaun Nichols, and Stephan Stich. 2006. Normativity and epistemic intuitions. In Biological and cultural bases of human inference, eds. Riccardo Viale, Daniel Andler, and Lawrence A. Hirschfeld. Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Welkenhuysen-Gybels, Jerry, Jaak Billiet, and Bart Cambré. 2003. Adjustment for acquiescence in the assessment of the construct equivalence of likert-type score items. Journal of Cross-Cultural Psychology 34(6):702–722.

    Article  Google Scholar 

Download references

Acknowledgements

For their discussion and criticism, I am grateful to Joshua Alexander, Sue Finch, Frank Jackson, Tania Lombrozo, Edouard Machery, Thomas Nadelhoffer, Norbert Schwarz, Gerard Vong, Jonathan Weinberg, and especially to Neil Thomason and Ilana Payes.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Simon Cullen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cullen, S. Survey-Driven Romanticism. Rev.Phil.Psych. 1, 275–296 (2010). https://doi.org/10.1007/s13164-009-0016-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13164-009-0016-1

Keywords

Navigation