Abstract
In 2014, widespread reports in the popular media that a chatbot named Eugene Goostman had passed the Turing test became further grist for those who argue that the diversionary tactics of chatbots like Goostman and others, such as those who participate in the Loebner competition, are enabled by the open-ended dialog of the Turing test. Some claim a new kind of test of machine intelligence is needed, and one community has advanced the Winograd schema competition to address this gap. We argue to the contrary that implicit in the Turing test is the cooperative challenge of using language to build a practical working understanding, necessitating a human interrogator to monitor and direct the conversation. We give examples which show that, because ambiguity in language is ubiquitous, open-ended conversation is not a flaw but rather the core challenge of the Turing test. We outline a statistical notion of practical working understanding that permits a reasonable amount of ambiguity, but nevertheless requires that ambiguity be resolved sufficiently for the agents to make progress.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
As is evident, we accept the standard, gender-neutral interpretation of the Turing test, whereby the interrogator must decide which conversation partner is human and which is a machine. Our acceptance of the non-gendered version of the test is based on evidence internal to Turing’s mind paper (1950) as well as some later remarks (Turing et al. 1952). This issue is thoroughly discussed by Copeland and Proudfoot (2008), Moor (2001), and Piccinini (2000).
The reader can see selected transcripts, with commentary, in Warwick and Shah (2016).
References
Aaronson S (2014) My conversation with “Eugene Goostman,” the chatbot that’s all over the news for allegedly passing the Turing test. https://www.scottaaronson.com/blog/?p=1858. Accessed 15 Jan 2019
Alchourrón CE, Gärdenfors P, Makinson D (1985) On the logic of theory change: partial meet contraction and revision functions. J Symb Log 50:510–530
Bacchus F (1990) Representing and reasoning with probabilistic knowledge: a logical approach to probabilities. MIT Press, Cambridge
Commonsense reasoning (2019) Winograd schema challenge. https://www.commonsensereasoning.org/winograd.html. Accessed 9 Mar 2019
Copeland J, Proudfoot D (2008) Turing’s test: a philosophical and historical guide. In: Epstein R, Roberts G, Beber G (eds) Parsing the Turing test: philosophical and methodological issues in the quest for the thinking computer. Springer, Netherlands, pp 119–138
Davis E, Morgenstern L, Ortiz C (2019) The Winograd schema challenge. https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html. Accessed 9 Mar 2019
de Kleer J (1986) An assumption-based TMS. Artif Intell 28(2):127–162
Dennett DC (2012) Turing’s gradualist vision: making minds from proto-minds Turing in context II, Brussels Invited talk
Gärdenfors P (1992) Belief revision: Cambridge tracts in theoretical computer science. Cambridge University Press, Cambridge
Harkaway N (2018) Will computers be able to think? Five books to help us understand AI. The Guardian. https://www.theguardian.com. Accessed 12 Mar 2019
Huang X, McCalla GI, Greer JE, Neufeld E (1991) Revising deductive knowledge and stereotypical knowledge in a student model. User Model User-Adap Inter 1(1):87–115
Kyburg HE Jr (1974) The logical foundations of statistical inference, vol 65. Springer Science and Business Media, New York
Levesque HJ (2011) The Winograd schema challenge. In: Logical formalizations of commonsense reasoning: papers from the 2011 AAAI Spring Symposium. Technical Report SS-11-06. AAAI Press, Palo Alto
Levesque HJ (2014) On our best behaviour. Artif Intell 212:27–35
Levesque HJ (2017) Common sense, the Turing test, and the quest for real AI: reflections on natural and artificial intelligence. MIT Press, Cambridge
Leibniz G (1996) New essays on human understanding. In: Remnant P, Bennett J (eds & trans) Cambridge texts in the history of philosophy, 2nd edn. Cambridge University Press (Original work published 1765)
Leviathan Y, Matias Y (2018) Google duplex: an AI system for accomplishing real-world tasks over the phone. Google AI Blog. https://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html. Accessed 11 Mar 2019
Lopatto E (2014) The AI that wasn’t: why ‘Eugene Goostman’ didn’t pass the turing test. Daily beast. https://www.thedailybeast.com. Accessed 12 Mar 2019
Luger GF, Chakrabarti C (2017) From Alan Turing to modern AI: practical solutions and an implicit epistemic stance. AI Soc 32(3):321–338
Marcus G (2014) Why Can’t my computer understand me? The New Yorker. https://newyorker.com. Accessed 12 Mar 2019
Martins JP, Shapiro SC (1988) A model for belief revision. Artif Intell 35(1):25–79
Moor JH (2001) The status and future of the Turing test. Mind Mach 11(1):77–93
Piccinini G (2000) Turing’s rules for the imitation game. Mind Mach 10(4):573–582
Poole D, Goebel R, Aleliunas R (1987) Theorist: A logical reasoning system for defaults and diagnosis. In: Cercone N, McCalla G (eds) The knowledge frontier. Springer, New York, pp 331–352
Shieber SM (2004) The Turing test’s evidentiary value. In: Shieber SM (ed) The Turing test: verbal behavior as the hallmark of intelligence. MIT Press, Cambridge, pp 293–295
Shotter J (2019) Why being dialogical must come before being logical: the need for a hermeneutical–dialogical approach to robotic activities. AI Soc 34(1):29–35
The Society for the Study of Artificial Intelligence and Simulation of Behaviour (2019) Loebner Prize (n.d.) https://www.aisb.org.uk/events/loebner-prize. Accessed 12 Mar 2019
Trausan-Matu S (2019) Is it possible to grow an I-Thou relation with an artificial agent? A dialogistic perspective. AI Soc 34(1):9–17
Turing AM (1950) Computing machinery and intelligence. Mind Lix 236:433–460
Turing AM, Braithwaite R, Jefferson G, Newman M (2004) Can automatic calculating machines be said to think? In: Copeland BJ (ed) The essential Turing. Clarendon, Oxford, pp 487–506
University of Reading (2014) Turing test success marks milestone in computing history. https://www.reading.ac.uk/news-and-events/releases/PR583836.aspx. Accessed 12 Mar 2019
Warwick K, Shah H (2016) The importance of a human viewpoint on computer natural language capabilities: a Turing test perspective. AI Soc 31(2):207–221
Acknowledgements
Thanks to David Mould for reminding us of the role of the interrogator and to Wlodek Zadrozny for suggesting the idea of language as collaborative planning. Thanks to the University of Saskatchewan for funding this research, and thanks to the numerous reviewers and commentators on earlier presentations of this work.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Neufeld, E., Finnestad, S. In defense of the Turing test. AI & Soc 35, 819–827 (2020). https://doi.org/10.1007/s00146-020-00946-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00146-020-00946-8