Taking iPhone Seriously Epistemic Technologies and the Extended Mind Isaac Record Lyman Briggs College, Michigan State University irecord@msu.edu Boaz Miller Community Information Systems, Zefat Academic College boaz.m@zefat.ac.il (Forthcoming in Extended Epistemology, edited by Duncan Pritchard, Andy Clark, Jesper Kallestrup, Orestis Palermos & J. Adam Carter. Oxford: Oxford University Press.) Abstract David Chalmers thinks his iPhone exemplifies the extended mind thesis by meeting the criteria that he and Andy Clark established in their well-known 1998 paper. Andy Clark agrees. We take this proposal seriously, evaluating the case of the GPS-enabled smartphone as a potential mind extender. We argue that the "trust and glue" criteria enumerated by Clark and Chalmers are incompatible with both the epistemic responsibilities that accompany everyday activities and the practices of trust that enable users to discharge them. Prospects for revision of the original criteria are dim. We therefore call for a rejection of the trust criterion and a reevaluation of the extended mind thesis. Keywords extended mind; knowledge; epistemic responsibility; smartphones; GPS The extended mind thesis challenges traditional conceptual boundaries between the human agent and her environment. It states that some tools are, under some circumstances, literally a part of our mental apparatus, and may realize a cognizer's mental states, such as her beliefs, outside the physical boundaries of her body. It has also been claimed that mass-produced smartphones, such as Apple's iPhone or the Samsung Galaxy series, exemplify the extended mind thesis (Chalmers 2011; Clark 2010). Other experimental devices, such as Google Glass, constitute serious candidates as well. It is therefore high time to examine more carefully whether such devices indeed realize a new cognitive paradigm of humans' interaction with their surroundings. Our analysis will focus on a specific combined activity: using the GPS feature of a smartphone for navigating while driving. Drawing on our examination of the GPS-enabled smartphone (GES), which extended mind proponents should regard as a quintessential 2 illustration of their thesis, we argue that the extended mind thesis does not correctly adjudicate everyday tasks that involve epistemic responsibilities and the practices of trust that enable users to discharge those responsibilities. We further argue that prospects for modifying the extended mind thesis to explicitly incorporate epistemic responsibilities and practices of trust are dim, and the extended mind thesis should therefore be rejected. The plan of this paper is to briefly recapitulate the extended mind thesis, with an emphasis on the four "trust and glue" criteria introduced in Clark and Chalmers (1998). We then show that Chalmers' (2011) claim at first appears quite plausible: in some use cases, his iPhone appears to satisfy the criteria. As we argue, however, the criteria wrongly adjudicate many everyday cases in which epistemic responsibilities demand active checking by users. In three parts, we pose three independent but partially overlapping arguments against the theory of extended mind. First, we argue that while Clark's account of the automatic trust criterion appears to correctly handle responsibilities to passively monitor epistemic artifacts, it mishandles responsibilities to actively check them. We argue, contra Clark, that "checking" responsibilities are pervasive; moreover, such responsibilities are not limited to the digital devices we treat here. The second argument is a stronger version of the first, but from a contextualist stance. We argue that the trust and glue criteria mishandle cases in which responsibilities vary with stakes. The third argument stands on its own, but reinforces the other two. The criterion of prior endorsement mishandles shared artifacts, and attempts to revise or relax the prior endorsement criterion have the effect of introducing further "checking" responsibilities-which are mishandled by the trust criterion, as argued in the first and second parts. We therefore reject the trust criterion for candidate mind extenders. The Extended Mind Thesis and its Discontents Let us briefly recount Clark and Chalmers' notions of extended cognition and extended mind. The premise is straightforward: "If, as we confront some task, a part of the world functions 3 as a process which, were it done in the head, we would have no hesitation in recognizing as part of the cognitive process, then that part of the world is ... part of the cognitive process" (Clark and Chalmers 1998, p. 8). Based on this "Parity Principle", Clark and Chalmers make two claims. First, a subject's cognitive system extends beyond her bodily boundaries into her environment-call this extended cognition. Second, information stored outside a subject's bodily boundaries may constitute the subject's mental states, such as beliefs and desires. Call this extended mind. The second claim is more controversial than the first, and there are scholars who accept the first but deny the second (e.g., Giere 2010; Goldberg 2012). Our arguments are aimed at the extended mind thesis, though they also have implications for how the extended cognition thesis should be understood. The classic example of a person with an extended mind is Otto, whose brain is ravaged by Alzheimer's disease and who relies on an ever-present notebook to help him structure his life. Clark and Chalmers compare the activities of Otto and Inga, a neuro-typical person, as they make their way to MOMA. Inga considers for a moment, remembering that MOMA is on 53rd street, and sets off. Otto consults his notebook, finds the address, and sets off. "It seems reasonable to say that Otto believed the museum was on 53rd Street even before consulting his notebook" (Clark and Chalmers 1998, p. 13). The key to the theory of extended mind is that, since Otto's activity with his notebook is indistinguishable from Inga's activity with her memory in all relevant ways, they should be accorded the same status: that of recalling a standing belief. Clark and Chalmers describe four "trust and glue" criteria for objects that are candidate mind extenders: 1. Constancy. For example, Otto's notebook is almost always readily available. 2. Facility. It requires only trivial effort or time for Otto to recover information from the notebook. 3. Trust. Otto automatically trusts information he finds written in his notebook. 4 4 . Prior endorsement. Otto has, at some past time, endorsed each bit of information – indeed, this is why he wrote it there in the first place. (Adapted from Clark and Chalmers 1998: 17). These criteria are not meant to hold unconditionally. For example, "constancy" does not require that the extender (Otto's notebook) be available under every circumstance, only those in which it might be called upon. A wristwatch, for example, does not cause its wearer to constantly know the time. It makes the wearer constantly able to know the time – what Clark calls "poise" for easy use (2003, p. 41). Likewise, "facility" does not necessarily mean that a device is easy to use, only that it is so closely coupled to its user that "as far as our conscious awareness is concerned, the tool itself fades into the background, becoming transparent in skilled use" (Clark 2003, p. 45). In addition, Clark and Chalmers (1998, p. 11) caution that candidate mind extenders cannot be impugned simply on the basis of the danger of discrete damage, loss, or malfunction, or because of any occasional decoupling: the biological brain is in similar danger, and occasionally loses capacities temporarily in episodes of sleep, intoxication, and emotion. If the relevant capacities are generally there when they are required, this is coupling enough. Palermos (2014) advocates replacing the glue, trust, and prior endorsement criteria with a new, more fundamental condition he calls "continuous reciprocal causation" (CRC) that is supposed to more accurately capture the relationship that must hold between a mind and a cognition extender. Although aimed at revising the account of extended cognition (not mind), we mention CRC because it has been suggested as a basis for revising the account of extended mind (in addition to cognition). Palermos himself is undecided whether CRC can successfully serve as a criterion for judging whether a system satisfies the extended mind thesis, in part because it appears to exclude the canonical case of Otto and his notebook. Even if such challenges could be overcome, a CRC-based account of extended mind would also have to successfully adjudicate the sorts of epistemically responsible behaviors we discuss later in 5 this paper. Working out these details for CRC is beyond the scope of the present paper, which remains fixed upon the original trust and glue criteria. The extended mind thesis has been controversial from its outset.i Here we focus on the implications of the thesis for the role of technology in our normative concepts and conduct. In this context, Preston (2010) argues that the thesis is superfluous and metaphysically extravagant. First, the common-sense understanding of Otto's notebook as a tool, rather than as part of an extended cognitive system, adequately explains Otto's relationship with his notebook and captures its active and externalist dimensions. Second, because the concept of belief, which Clark and Chalmers wish to revise, is part of a wide web of normative concepts, the extended reconceptualization of belief has far-reaching implications for other normative concepts, such as health and disease. For example, medical science tells us that Alzheimer's disease causes one to lose one's memory. Is Otto cured because his notebook restores his memory? Clark and Chalmers may bite the bullet and argue that if their thesis entails changes to our normative concepts, then all the worse for those concepts. They may observe that scientific and ordinary concepts often develop in tandem. Psychological concepts have shaped and been shaped by ordinary concepts, including legal ones (Danziger 1997; Valverde 2009; Pettit 2013). We want to block this possible response by arguing that the extended mind thesis is deeply incompatible not only with our normative concepts, but also with normal responsible epistemic conduct, which Clark and Chalmers give us no good reason to revise. An exchange between Clark and Sterelny is revealing of how Clark wants us to apply the trust criterion, which will be our point of departure. Let us begin, then, with Sterelny's (2004) argument against Clark.ii Sterelny observes that epistemic artifacts, such as Otto's notebook, are typically jointly produced, exist in a common and contested space, and are accessible to more than one user. Many organisms use epistemic artifacts to store and 6 retrieve information. In the evolutionary struggle for survival, organisms have an interest to sabotage each other's sources of information. Organisms therefore need to safeguard their epistemic artifacts against sabotage and deception by others. By contrast, they are typically not required to safeguard their biologically innate information channels, which can be assumed to be reliable due to natural selection.iii For Sterelny, this crucial difference explains why external epistemic artifacts are not part of an organism's extended cognitive system. In his reply to Sterelny, Clark reveals how he wishes to apply the trust criterion. He acknowledges that if, for example, Otto becomes suspicious of his notebook and starts double-checking it, "the notebook would at that point cease to unproblematically count as a proper part of his individual cognitive economy" (2011, p. 104). This is in striking contrast to our treatment of dubious brain-bound recollection, whose status as mental does not waver. That is, Clark concedes that there is, in principle, a difference between externally and internally stored information. Externally stored information is part of a subject's cognitive system and may thus realize a subject's mental states only insofar as it is implicitly trusted, and ceases to be so when it is not, while internal information sources, such as memory or perception, remain part of the subject's cognition regardless. Clark claims that Sterelny's objection is largely academic because it rarely manifests itself in practice. That is, epistemic artifacts usually do not require checking, and people rarely waver between trust and mistrust of artifacts that meet the other criteria. He argues that internal information channels are also vulnerable to deception and must be safeguarded, but in normal day-to-day circumstances, they are sufficiently trustworthy and do not need safeguarding. Similarly, only in isolated, abnormal cases are epistemic artifacts treated with caution, and Clark is willing to concede that under those circumstances, artifacts are not mind extenders. Under usual circumstances, Clark stresses, epistemic artifacts can be as reliable as internal information channels; thus they can unproblematically constitute part of an agent's extended cognitive system (Clark 2011, pp. 102-104). 7 Let us be as clear as possible here: Clark says that Otto's notebook is mental only when trusted. He acknowledges that this is distinct from brain-bound memories, which count as mental whether trusted or not. Clark argues that this principled distinction is acceptable because it makes no difference in practice, because occasions of doubt are rare and responsibilities for checking are rare. To the contrary, we argue that doubt is commonplace and often warranted-we will make this case in detail in the following sections. Note that an alternative response was possible. Clark might have leaned on the Parity Principle to argue that just as we sometimes doubt our memory (as in hazy recollection) or our senses (as in mirage or optical illusion), we sometimes doubt our external minds too. This need not necessarily undercut the extended mind thesis. For example, Clark might have argued that Otto could be unsure of his notebook's contents if the handwriting seems smudged or unfamiliar without necessarily severing the tie to his mind. Yet severing the connection is precisely what Clark explicitly advocates: "the notebook would ... cease to ... count" (2011, p. 104). In this, Clark appears to value the trust criterion above the Parity Principle. Clark's reply to Sterelny is inadequate because it rests on an unwarranted assumption. Clark seems to assume, first, that trust maps onto trustworthiness, and second, that the trustworthiness of an epistemic artifact is a function only of its reliability. Thus, he thinks that if a coupled artifact is as reliable as biologically innate information channels, it is trustworthy (and trusted) and thus constitutes part of the subject's cognition. This assumption is wrong. Instrument trustworthiness does not depend merely on its reliability, but also on the subject's normative circumstances, particularly her epistemic responsibilities. Safeguarding an epistemic artifact against sabotage and deception is just one responsibility subjects normally and routinely have about the instruments they use to obtain knowledge. In the next three sections, we show that the theory of extended mind is incompatible with the responsibilist view of knowledge. First, we argue that Clark's account of the 8 automatic trust criterion mishandles commonplace responsibilities to actively check on the performance of epistemic artifacts by counting as extenders cases it should exclude and excluding cases it should allow. Second, we show that, for those persuaded by the contextualist argument that epistemic responsibilities vary with stakes, the trust criterion performs even worse in a variety of cases. Third, we show that the criterion of prior endorsement rules out many of the shared epistemic artifacts that Clark and Chalmers wish to consider as potential mind extenders. Furthermore, plausible attempts to revise or relax the prior endorsement criterion have the effect of introducing further "checking" responsibilities, which are mishandled by the trust criterion, as argued in the first and second parts. We therefore reject the trust criterion for candidate mind extenders. A responsibilist analysis of the trust criterion Clark contends that requirements of checking occur only rarely. To the contrary, we hold that such responsibilities are pervasive. We analyze the case of navigation with a GPS enabled smartphone (GES) to illustrate a variety of epistemic responsibilities, their ubiquity, and the variety of practices of trust users employ to discharge their responsibilities. In many ordinary circumstances, skilled users do not and should not automatically trust GES results; that trust must be gained through practices of trust that discharge epistemic responsibilities. Clark's account of the automatic trust criterion appears to correctly handle responsibilities to monitor epistemic artifacts, but mishandles responsibilities to actively check them. Therefore, if Clark or his advocates can establish that most everyday responsibilities are monitoring responsibilities, then the force of our objection largely disappears. We argue that the case of GESs shows that in many everyday cases, more than monitoring is required. Furthermore, GESs are not a special case; rather, they provide a clear example of a very common phenomenon. Most prospective mind extenders will be surrounded by epistemic 9 responsibilities and corresponding practices of trust, and their incompatibility with the trust condition is fully general. Before we proceed to our central argument that the extended mind thesis mishandles epistemic responsibility in the GES case, let us clarify and justify the role of the GES case study in our argument. One could reconstruct our argument as follows: (1) if the extended mind thesis is true, then under the trust and glue conditions, GESs constitute mind extenders for normal responsible users; (2) under such conditions, GESs do not constitute mind extenders for normal responsible users; (3) therefore, the extended mind thesis is false. One may immediately object to our argument by denying (1), without which (3) does not follow. One would argue that we, at most, establish that current GES technology does not yet fully satisfy the trust and glue conditions. This is merely a contingent fact about the present state of technology, which does not show that the extended mind thesis is false, or so this objection goes. Our reply is twofold. First, we show that proponents of the extended mind thesis accept (1). Second, we argue that they rightly accept it, because without (1), the extended mind thesis becomes trivial and uninteresting. First, extended mind thesis sympathizers and proponents enthusiastically embrace smartphones and their like, and deem them genuine mind extenders. Here are a few examples. As mentioned, Chalmers (2011) refers to his iPhone as a vindication of his and Clark's original 1998 argument. Clark, Pritchard, and Vaesen (2012, p. 87) write that "external aids (notebooks, watches, and smartphones) [...] may (under certain conditions) become a proper part of the cognizing mind." Clark (2010) adds "iPhones, BlackBerrys, laptops and organizers" to this list. With respect to GPS, Halpin, Clark, and Wheeler (2010, p. 4) argue that a real-time peer-produced navigation application takes the extended mind thesis one step further by showing that people may collectively share a belief on the Web.iv We are on solid ground committing extended mind proponents and sympathizers to (1). We thus note that even if our argument only establishes that GESs are not mind extenders, it still 10 makes a substantial philosophical point, because it calls extended mind proponents' view of current technologies into question, and suggests that there is an error either in the extended mind thesis or in how its proponents apply it to particular cases. Second, we think that extended mind proponents are right to insist on (1), because giving up on (1) would take the wind out of the extended mind thesis, making it trivial and uninteresting. If the extended mind thesis only amounted to the claim that in some farfetched science fiction scenarios, it is possible for artifacts to extend the mind, or that biological organisms could in principle evolve such that they are integrated with external devices for cognitive processing and storage, it would hardly be disputed. But it would not entail any interesting implications for human life either. Douglas Adams' Babel fish, a fictitious symbiote that lives in the ear canal and translates between alien races, is a wonderful illustration of the extended mind thesis, but a standalone argument from it for the extended mind thesis would be as dubious as Adams' playful argument from the existence of the Babel fish to the nonexistence of God.v Similarly, extended mind proponents could just stipulate that there exists a perfectly reliable information storing technology that smoothly and transparently interfaces with our cognition such that its user does not feel any difference whatsoever between it and his biological memory; but this would be entirely question begging. For the extended mind thesis to be philosophically relevant to the human condition, it must not lose sight of real-life interactions between people and their devices, like GESs. Indeed, Clark and Chalmers' arguments are generally well immersed in the current state of technology and recent developments in cognitive science, but they occasionally indulge in wild scenarios about futuristic technology or imaginary aliens with different cognitive makeup. For example, Clark and Chalmers (1998, p. 16) mention the fact that the Terminator's memories are displayed in his visual field, and Clark (2011, pp. 77-78) compares a near-future human who has implanted a 3D-rotating graphics chip that interfaces 11 directly with her brain with a Martian for whom the same ability developed naturally. On their own, such examples have limited value for learning anything significant about the boundaries of human cognition. This is because future technology and future evolution are constrained by the present, and they are hard to predict. Human evolution is path-dependent; its future course is constrained by its past (Beatty and Desjardins 2009), and so is the evolution of technology (Basalla 1988). Thus, to argue for a non-trivial version of the extended mind thesis, it is not enough to show that in principle, certain technologies or cognitive abilities can develop, or could have developed, but rather, it is necessary to show that they have developed or are likely to develop. We therefore think that extended mind proponents' wild, far-fetched scenarios should charitably be viewed as intuition pumps, rather than arguments that are supposed to stand on their own. Extended mind proponents are right to insist that the extended mind thesis should adequately characterize the relations of actual human beings with current or near-future technologies, such as smartphones. Our argument from GES does not miss its target. Let us, then, go back to GES and see to what extent it is compatible with the extended mind thesis and then see if the result generalizes. Many drivers use GESs for everyday travel. For those familiar with GESs, navigating may seem trivial, in just the sense required by the criterion of facility. But in fact, GESs require significant experience before any level of facility is possible. Phone-based GPSs show a video-game-like "third-person" view of a path. It is heavily iconic, for example, with roads in yellow, the route in blue, the car indicated by a blue arrow, and a voice prompting upcoming maneuvers. The user must be able to translate the map, icons, and voice instructions into real actions with the steering wheel and pedals in the world seen in the windscreen and mirrors of the car. This process is cognitively demanding, especially in cases when GPS information is incomplete or inaccurate. Experience hides the challenges beginners face behind skilled use. We will argue that skilled use typically includes 12 practices of active checking that are contrary to the criterion of automatic trust for mind extenders. Brown and Laurier (2012) have conducted an empirical study of the day-to-day troubles with using GPS while driving. Although these devices have improved dramatically in recent years, many locations are still hundreds of meters away from GPS-directed results. When activated during a trip-by a passenger, of course!-GPSs can be confused about which direction you are heading on a highway. They are usually unaware of recent roadwork, sometimes including traffic pattern changes. Even when GPS data are accurate and complete, "instructions are [...] open to re-analysis, re-interpretation and re-use as the car moves through the environment [...] There is always the need for the active skillful construction of action at the point of use" (Brown and Laurier 2012, p. 1629). Skilled users of GESs do not trust the devices implicitly, nor should they. Only when they have ensured that certain conditions have been met, and only in the absence of confounders like construction detours, fuel stops, or other commonplace interruptions between origin and destination, should users trust GESs. Experienced drivers usually trust GESs to help them complete some portions of the navigational task, but not all of them. For all of the other tasks, drivers employ what we call practices of trust to detect, avoid, overcome, or correct for the known failure modes of a device.vi We can draw on Brown and Laurier for a catalog of what they call the "normal natural troubles" of GPS use (2012, p. 1628) and the associated practices of trust for coping with them. GPS navigation often goes wrong for the mundane reason that the wrong destination was programmed. Sometimes, GPSs provide incorrect map information, including road names or traffic patterns. Another normal, natural trouble with GPSs is the timing of instructions: at highway speeds, instructions may come with too little warning for the maneuver to be completed safely; similarly, without foreknowledge of the next instruction, a driver may enter a lane that makes the next maneuver difficult or impossible. GPSs also 13 sometimes provide ambiguous instructions such as "continue" that must be evaluated to determine whether a maneuver is required. Finally, GPSs usually conform very strictly to the law, which can result in instructions to circle the block rather than turning across a solid line into a car park. "Part of the skilled use of GPS is overcoming these problems" (Brown and Laurier 2012, p. 1628). This translates to straightforward practices of trust like reviewing a route before setting off or checking road signs for confirmation that the GPS directions make sense. As simple as these practices are, they demonstrate that GES users often do not automatically trust GPS results, nor should they. Let us catalog a few such practices of trust, to demonstrate just how encompassing they are of the entire process of using a GES. Some practices of trust, like using a GES on a familiar route, can help a driver accustom herself to the peculiarities of an unfamiliar GES, such as a new smartphone application or a significant interface upgrade. Programming a familiar destination can reassure her that the GES is trustworthy if it reproduces her usual route or may alarm her if it does not. Using a GES in a familiar setting can be helpful in learning to interpret its iconography and translate instructions into actions. Only once these or other familiarizing exercises are complete and the user gains facility with the GES can we count the device as a candidate mind extender. But practices of trust do more than buttress our facility with GESs. Another set of practices of trust clearly relate to the trust condition for mind extenders. Drivers may elect to confirm the GES results by attending to road signs, which helps to coordinate GES results with the world, thereby confirming its accuracy. Drivers sometimes voluntarily abandon a GES-programmed route in order to run a quick errand, fill the gas tank, or follow a detour. Once a driver abandons the programmed route, she faces the potential distraction of the GES indefatigably and implacably recalculating the route and announcing the results. She must ignore these commands to continue the detour-an explicit reversal of the "automatic trust" 14 condition for a mind extender. Likewise, drivers have to be aware that their destination is approaching so that they can find parking. In such cases, the driver can use the GES to provide supplementary information, but the primary guide comes from road signs. In these cases, the GES may not fulfill the constancy condition of being "ready-to-hand" because programming a GES to take a detour is not worth the effort for a quick detour-and in any case may be too dangerous to complete while driving. Finally, there are cases in which even skilled users face considerable challenges in navigating with a GES rather than a mental map. In the case of a lack of look-ahead data, a driver has to consult the GES and perhaps interact with it via the touch interface in order to determine the details of the prompted maneuver – e.g., which exit lane to take in order to be able to make the turn at the bottom of the ramp. This lack of guidance coincides with one of the most demanding driving situations: signaling and merging to one side into the off-ramp while paying attention to the cars ahead, which might come to a sudden stop as they encounter heavier traffic on the off-ramp; avoiding collisions with accelerating cars that are entering the highway; and-at the very same time-interacting with a GES to figure out which of two or more exit lanes to merge into. Some of the practices of trust that surround GES use, such as occasionally glancing at the screen to be sure it is still turned on, might be construed as "monitoring" rather than "active checking."vii Monitoring is more a passive posture than an active distrust, and as such is perfectly compatible with both Clark's account of trust and our responsibilist account of practices of trust. As long as a driver's responsibility is fulfilled through the maintenance of an appropriate level of attention to the task (rather than a requirement to distrust or actively check the GES results), the criterion of trust is secure. However, many of the practices of trust just described clearly do require "active checking," even for experienced GES users. Ignoring instructions during a fuel excursion and making lane decisions without needed look-ahead data are just two examples. Furthermore, we think that common activities like corroborating 15 GES instructions against street signs should count as checking rather than monitoring, since it requires a comparison between GES information and environmental information. The evaluation of the use of everyday technologies according to the trust and glue criteria gives us the wrong results: irresponsible uses of purported mind extenders are admitted, while responsible uses are rejected. In practice, trust is not universally given. Responsible users of technologies carefully embed them within practices of trust that help identify when and how much to trust instruments. Implicit trust is given only by the most credulous users, and they are often punished for their credulity. Indeed, the media is full of examples of people who get themselves into serious danger by blindly following their GPS instructions and driving onto train tracks, into a wrong lane, into the ocean, or off a cliff. Others wind up hundreds of miles from their intended destination because they do not perform these practices of trust. In several cases in Brazil, drivers followed their GPS instructions into violent neighborhoods, apparently after having entered a wrong address in the device. Some of these cases tragically ended with their death or injury (Hansen 2013; DeLong 2014; BBC News 2015; Phillips 2015; Milner 2016). In 2016, two Israeli soldiers were nearly lynched after they followed their Waze navigation app into a Palestinian refugee camp. One Palestinian was killed in their rescue operation. Israeli Defense Minister Moshe Ya'alon said, "I learned ages ago the importance of navigating with the aid of a real map, and mainly to know the surrounding area and not to rely too heavily on technology which can lead the user astray" (Reed 2016).viii Notwithstanding developers' responsibilities to make their apps safe, we suggest that these people are not skilled users of GESs and that their troubles are evidence of the stakes of ignoring one's epistemic responsibilities and being too trusting of an epistemic artifact. To put the point in a stronger form, these are cases in which the GES conforms much more closely to the original trust and glue criteria-users implicitly trust the equipment-and that is precisely what gets them into trouble. Facility with the device, including knowing when to 16 trust it, works against the criterion of automatic trust. It would be a strange theory that counts GPS as a mind extender for overly credulous users, but not for responsible ones. Clark has recently put forward a new argument for defending the extended mind thesis against such criticism. Pritchard (2010) argues (similarly to us, though on different grounds) that in order for externally stored information to count as a subject's knowledge, the subject must be able "to take cognitive responsibility" for it (2010, p. 138). Clark takes Pritchard's claim as posing the following dilemma to the extended mind thesis: Otto must either consciously encounter the notebook as an object for epistemically hygienic practice, or not. If he does, this makes the notebook look, at that moment, more like external equipment (it may then be a source of knowledge while failing to be part of Otto). If he doesn't, it looks unable [...] to act as a source of knowledge (Clark 2015, p. 7). Clark argues, however, that Pritchard's dilemma is avoidable. Drawing on current research in cognitive science, Clark distinguishes personal cognitive processes, which are conscious, from sub-personal processes, which are non-conscious. Clark argues that because subpersonal cognitive processes for assessing information reliability are adaptive and responsive to changing conditions or contexts, they can successfully fulfil the cognizer's epistemic responsibilities. As Clark puts this: large and crucial second order apparatus of precision-weighting of prediction error constitutes a potent means of responding, without need for awareness, to the contextvarying reliability of our (inner and outer) information source (2015, p. 13). In the context of our argument, Clark's reply would amount to claiming that a GES user need not consciously deploy practices of trust in order to satisfy her relevant epistemic responsibilities, but may rely on her non-conscious sub-personal information-assessing cognitive processes instead. Clark's reply, however, is inadequate. Sub-personal processes lack features that are necessary for successfully fulfilling a cognizer's epistemic responsibilities. Many cognitive psychologists believe that humans possess two distinct reasoning systems, known as System 1 and System 2. System 1 is a collection of autonomous subsystems that are fast, automatic, 17 effortless , non-conscious, parallel, shaped by biology and personal experience, and independent of working memory and general intelligence. System 2, which is more evolutionally recent, is slow, controlled, effortful, conscious, serial, shaped by culture and formal tuition, demanding of working memory, and related to general intelligence (Kahneman 2011; Evans 2010). As Frankish (2009, p. 96) notes, the features of subpersonal reasoning and personal reasoning coincide closely with the core features of the two putative systems. Subpersonal reasoning is typically fast, automatic, effortless, and nonconscious, whereas personal reasoning is typically slow, controlled, effortful, and conscious. Personal reasoning is also serial, shaped by culture and formal tuition, and, since it typically requires attention, demanding of working memory. If sub-personal processes indeed map onto, or significantly overlap with System 1 processes, ix they lack features needed for meeting the requirements of higher normative systems, such as epistemic norms. This is because "System 1 generates an intuitive response that is adaptive but non-normative, whereas System 2 generates a more considered response that is in line with one's normative theories" (Frankish 2009, p. 98). Suppose, hypothetically, that a cognizer could reliably perform a complex mathematical calculation using subpersonal reasoning. She could quickly provide the right answers to complex math questions, but she would not be able to explain or justify them. By contrast, a cognizer performing the same calculation with a pen and a paper, using personal reasoning, would arrive at the answer more slowly, but could explain and justify it (Frankish 2009, p. 92). Tracking one's reasoning and giving explicit explanations is often normatively required for meeting one's epistemic responsibilities. For instance, in some circumstances, a responsible GES user need not merely have a reliable sense that there is something wrong with the route her GES suggests, but she must consciously wonder and try to explain what is wrong with it. Pace Clark, mere reliability and adaptability to changing circumstances or contexts are often insufficient for meeting a subject's epistemic responsibilities. A similar way, perhaps, to put Clark's objection, without appealing to the 18 personal /subpersonal distinction, is to propose that practices of trust can themselves become transparent to skilled users, automatically incorporated into the use of a potential mind extender. Then, the automatic trust criterion refers to the action loop that includes both the original activity and the practices of trust that discharge the user's epistemic responsibilities, thereby saving the extended mind thesis. For example, in the GES case, we may not automatically trust a GES result by itself, but perhaps we do automatically trust the result of a GES instruction in combination with appropriate checking of street signs to corroborate the instructions. Our reply to this objection is twofold. First, there are still circumstances in which people are required to de-automatize their actions. For example, during takeoff and landing, pilots of heavy aircraft are required to follows an explicit ritualized procedure, in which one pilot goes over a checklist and another pilot makes the necessary checks, verbally verifying the results. The aim of this practice is to make the pilots explicitly aware of any potential problems. Second, revising the theory in this way would constitute a dramatic reinterpretation of "automatic trust," but if Clark and Chalmers or their adherents wish to take this view, we will not object. The theory of extended mind wrongly adjudicates cases of varying stakes According to a family of subject-centered contextualist positions in epistemology, the epistemic standards that a subject's belief that p needs to meet in order to be responsible, justified, or to constitute knowledge, are partly determined by facts about the subject's pragmatic standing vis-à-vis p.x Elsewhere (Miller and Record 2013, and forthcoming; Record 2013) we side with such views, and argue that epistemic responsibilities vary with circumstances; namely, the degree of justification a subject needs to have for her belief that p to be responsible inter alia depends on her stakes regarding p. To give an intuitive example, the belief that a surgical practice is safe requires more evidence to be justified if the believing 19 subject is a doctor about to perform the practice rather than a person merely watching a medical drama on television. Within the context if this paper, such contextualist views entail that, all else equal, in a high-stakes situation, subjects may have an epistemic responsibility to establish the trustworthiness of an instrument, while in a low-stakes scenario, the same instrument would not require such checks. Because driving is a risky activity, this means that driving-related beliefs typically require the subject to perform more checks and verifications if they are to be justified, in addition to other good reasons people have to arrive, rather than not arrive, at their destination. We emphasize that this is an epistemic responsibility rather than ethical or other responsibility.xi If we accept this, then according to Clark's own account, a reliable instrument coupled to a subject would normally constitute part of her extended cognition only in some circumstances, e.g., low-stakes circumstances, but not others. This is an unhappy result for the extended mind thesis. Instead of the isolated and therefore inconsequential mismatches between extended and in-the-head cognition Clark predicts, this concession opens the door to broad and counterintuitive mismatches. Taking epistemic responsibilities into account means that our internal channels and external instruments function differently in day-to-day life after all. Again, GESs provide an apt illustration of the mismatch between everyday practices and the criterion of trust. Drivers intending to travel from point A to point B take on an epistemic responsibility to inspect some aspects of any proposed way to get there. The required degree of checking will depend on the consequences of failing to get there, or failing to get there safely. Even a casual trip to meet friends carries with it some responsibility. Driving is probably the most common dangerous activity in which most people are involved, barring extreme sports or specific occupational hazards. Not knowing the way tends to put people in dangerous situations, such as unexpectedly turning at the last moment or risking 20 an illegal U-turn. While we tend to treat driving casually, it literally puts lives at risk. It might be objected, on the basis of the Parity Principle, that because epistemic responsibilities sometimes require a subject to check her internal channels, e.g., to question her memory, the externalized phenomenon is not a challenge to the extended mind thesis. However, as was noted earlier, Clark explicitly rejects this reply. Memory's being trusted (or trustworthy) is not a condition for constituting part of the subject's cognition, while it is a condition for an instrument, as Clark himself insists. As we will show, trying to do away with the trust condition leaves the extension thesis crippled. Prior Endorsement and GESs As noted, Clark and Chalmers specify four conditions for an object to be part of a subject's extended cognitive system: constancy, facility, trust, and prior endorsement. The first three are generally referred to as the glue and trust criteria. In this section, we assess the remaining condition, prior-endorsement, arguing that something like it is necessary to prevent the "cognitive bloat" that ensues when the other three criteria are too easily met. But GES does not satisfy prior-endorsement, or at best satisfies a relaxed version of it only with difficulty. This fact alone does not seriously threaten the extended mind thesis, because Clark and Chalmers (1998, p. 17) are not fully committed to the prior-endorsement condition, and anyway it is not clear whether they intend that it hold for any belief-storing mind-extender or only memory extenders. However, as we argue in the following, because GESs do not satisfy the prior-endorsement condition, they impose on their users the epistemic responsibility of establishing the device's trustworthiness in a way that often enough makes the device violate the trust condition, which is cardinal for extended mind. It might be objected at the outset that this argument is a red herring: in later works, Clark and Chalmers have dispensed with prior endorsement altogether. However, without prior endorsement or something like it, the glue and trust conditions are too permissive, 21 producing a "cognitive bloat" of multifarious, often short-lived mind extenders. Rupert (2004), for example, argues that a person who accesses a phone book or similar directory typically satisfies the glue and trust criteria, yet few extension proponents would wish to count lookup sources as extensions. These proponents need some criteria by which to deal with cognitive bloat. To briefly review, Chalmers (2011) writes that his iPhone seems to meet the criteria that he and Clark established a decade earlier. Indeed, the smartphone in skilled, familiar hands appears to fare well against three of the four criteria. For many, smartphones are within reach at every waking moment; barring low batteries or signal failures, smartphones fulfill the criterion of constancy. Likewise, many users have considerable facility with smartphones. While the interfaces are constantly evolving, with practice, users typically regain facility quite quickly. Additionally, many smartphone users tend to implicitly trust their devices. We would note that while trust may be a true description of typical attitudes toward smartphone results, that attitude may not always be warranted (see Miller and Record 2013, and forthcoming). The prior-endorsement criterion, however, accords less well with typical smartphone use. Phones seem no worse off than Otto's notebook regarding personally generated content, but popular smartphone tools, including search engines, recommendation services, and maps draw from databases that are created by large institutions or are algorithmically produced from user-generated data. Using such tools may have our prior endorsement, but the specific content delivered by maps and search engines is typically not endorsed by us personally.xii It is endorsed by someone else, a collective, or, in the case of algorithmic services, no one at all. Let us see how this unfolds in the case of GES.xiii Edgar, Otto's twin, afflicted by the same memory problems, uses a smartphone instead of a notebook. Deciding that he does not want to be left out of the MOMA outing, he activates his smartphone and queries the map application for guidance. After a moment, a map and directions appear. Edgar sets off. As in 22 the original case, Edgar's actions seem indistinguishable from Otto's or Inga's in any relevant way. We might not want to say that Edgar "remembers" MOMA's location, but instead perhaps he, together with his GES, has an innate sense of direction.xiv But this does not detract from the central point: it looks like Edgar, Otto, and Inga are all in the same boat so far as satisfying Clark and Chalmers criteria and intuition pumping example go. Unfortunately, the setup of the Otto/Inga example has obscured a part of the process that we think is crucial: the everyday norm of responsible checking. If Edgar is a responsible smartphone user, there is an evaluative step between receiving the result and setting off. As we discuss elsewhere (Miller and Record 2013), subjects often have a responsibility to evaluate their information sources when they generate beliefs, and these responsibilities are determined by the circumstances of those beliefs. This practice of responsible checking puts the extended mind theory in a difficult spot. To accommodate typical smartphone results like GPS data, the prior-endorsement condition has to be relaxed. This threatens the criterion of automatic trust, putting the extended mind theory in serious jeopardy. Some authors (e.g., Halpin 2008) relax the prior-endorsement criterion to allow for "extended endorsement," which means that it may be enough for the content of the putative extended belief to have undergone some external, possibly collective, verifying process. Such relaxation would allow in crowd-sourced or algorithmically produced results. Although this argument would seem to provide a warrant for automatically trusting results, there are still common enough circumstances in which a subject's epistemic responsibilities demand that she personally perform checks on the results, for example, to personally vouch for them (Miller and Record 2013). One plausible response to our argument about prior endorsement is that all we have shown is that a particular formulation of a relaxed prior endorsement criterion is wrong. The capacity of smartphones to access shared resources led us to reconsider Clark and Chalmers' original prior-endorsement requirement. We noted two alternatives: either allow for the 23 endorsement of resources rather than individual points of data, or allow for endorsement by someone who is not the using subject. So far, we have discussed only the latter. One problem with the approach of simply allowing for endorsement by others is that it appears incapable of handling algorithmically provided results, because no one will have endorsed them. Resources like Google's maps are increasingly "personalized" by algorithms, emphasizing locations deemed similar to those users have previously searched out or mentioned. This has additional worrying implications for the trust and endorsement conditions: the businesses that appear on the map differ from person to person – and the process of selection and omission is opaque to users.xv The other reason to resist relaxing the prior endorsement criterion in such a general way was mentioned before: it is too easy to satisfy. Rupert poses a modified Otto case in which a "directory assistance operator is a constant in Otto's life, easily reached; when the information would be relevant, it guides Otto's behavior; and Otto automatically endorses whatever the operator tells him, about phone numbers, anyway" (2004, pp. 402-403). Without the full-strength prior-endorsement criterion, or something like it, we would seem to be forced into accepting that Otto knows everyone's phone number – a conclusion we should be inclined to reject. In response to this objection, we note that if the prior-endorsement condition is not relaxed, it can be abandoned, kept stringent, or replaced. In the first case, we may bite the bullet and accept that we all know lots of things we did not without smartphones, directory assistance, and other extended services. If we wish to adhere to our normal understanding of 'belief' and 'memory', we can paraphrase: a rolodex or directory assistance does not extend Otto's memory of phone numbers; instead, they add a new faculty to his mind of perceiving phone numbers. This approach seems quite distinct from the original extended mind conception. But we need not consider it here because abandoning the prior-endorsement condition would be grist to our mill-it puts even more strain on the trust condition, making 24 it that much harder to satisfy. The second option is preserving the original prior-endorsement criterion, or some stringent modified version of it. Unfortunately, even with the original prior-endorsement criterion, epistemic responsibilities and practices of trust disrupt the trust criterion. First, even in our everyday activities, trust must be gained through discharging epistemic responsibilities. We are so familiar with our innate informational and perceptual systems that we might easily miss the significance of our abilities to identify when they fail, and when they do, to cope with or repair their input. For example, it is our responsibility to distrust our sense of sight when we encounter a known mirage or an optical illusion. Second, as we will shortly see, for extended mind proponents, sticking to a stringent version of the priorendorsement condition is too high a price to pay. This would leave out cutting-edge devices that provide third-party content, such as iPhone and Google Glass, which mind extension proponents celebrate as vindications of their theory. In this section, we argued that the prior-endorsement condition needs to be relaxed for GESs and other shared epistemic devices that provide public, third-party, or algorithmically produced information, to qualify as mind extenders. Yet the relaxation of the prior-endorsement condition makes it much harder for putative mind extenders to meet the trust condition, which is crucial to the theory of extended mind. Revising the Extended Mind Thesis? We have presented three arguments against the classic rendering of the theory of extended mind. First, we used the case of GESs to illustrate how Clark's account of the trust criterion results in a mishandling of everyday responsibilities. Next, we showed that this mishandling is even worse if we allow that responsibilities vary with stakes. Third, we showed that the criterion of prior endorsement would rule out GESs and other shared epistemic artifacts as potential mind extenders, and attempts to revise or relax this criterion have the effect of 25 introducing further "checking" responsibilities, which are mishandled by the trust criterion, as argued in the first and second parts. We therefore reject the trust criterion for potential mind extenders. A possible reply to our argument is that the extended mind thesis can be saved by explicitly incorporating a condition of personal epistemic responsibility into it. Roberts (2012) makes such a suggestion. According to virtue epistemology, knowledge is, roughly, true belief that results from a subject successfully exercising her epistemic virtues. Virtue epistemology appeals to normative standards for evaluating whether a believing subject has successfully exercised her cognitive abilities. Roberts argues that these standards are intimately tied to standards of epistemic responsibility. From this, he concludes that a subject's belief may amount to knowledge only if she can assume epistemic responsibility for it. Consequently, information stored in external devices may qualify as her beliefs only if she can assume epistemic responsibility for it. Roberts argues that subjects cannot assume epistemic responsibility for externally stored information that they do not fully grasp or know how it is produced. Many cases discussed in the extended mind literature, including of reliance on information in smartphones, are of this sort. Therefore, so Roberts argues, the extended mind thesis has a very limited scope. Nevertheless, Roberts thinks that some cases, such as a mathematician who skillfully uses a calculating device, qualify under the extended mind criteria because the subject can assume epistemic responsibility in them. We are sympathetic to Roberts' analysis, but we think it entails more severe consequence to the extended mind thesis. Roberts' analysis may, at most, support the possibility that a subject's cognitive processes extend beyond her bodily boundaries, but it cannot support the more radical possibility that cognitive states are realized outside her mind. As we argued, in sufficiently frequent normal circumstances, such as high-stakes scenarios, epistemic responsibility requires that subjects treat externally stored information with skepticism-a violation of the trust condition. Thus, there is an inherent conflict 26 between Roberts' suggested responsibility condition and Clark and Chalmers' trust condition. A possible solution is adopting Roberts' responsibility condition and discarding the trust condition, but this would constitute a significant revision of, and departure from the extended mind thesis, which would arguably make it less attractive to its current proponents. The implications of such a move need to be further studied and assessed. Conclusion We considered Chalmers' (2011) claim that his iPhone is an extension of his mind. We evaluated the GPS-enabled smartphone against the "trust and glue" criteria enumerated by Clark and Chalmers and argued that they are incompatible with the epistemic responsibilities that accompany everyday activities and the practices of trust that enable users to discharge them. Prospects for revision of the original criteria are dim. We therefore call for a rejection of the trust criterion and a reevaluation of the extended mind thesis. References Adams, Douglas. 2002. The Ultimate Hitchhiker's Guide to the Galaxy (New York: Del Rey). Adams, Fred, and Ken Aizawa. 2008. The Bounds of Cognition (Oxford: Blackwell). Adams, Fred, and Ken Aizawa. 2010. "Defending the Bounds of Cognition," in The Extended Mind, ed. Richard Menary (Cambridge, MA: MIT Press), pp. 67-80. Annis, David. B. 1978. "A Contextualist Theory of Epistemic Justification," American Philosophical Quarterly, vol. 15, no. 3, pp. 213–19. Bassalla, George. 1988. The Evolution of Technology (Cambridge: Cambridge University Press). BBC News. 2015. "GPS Fail on Bus Sends Belgian Tourists on 1,200km Detour," March 10, http://www.bbc.com/news/world-europe-31814083 Beatty, John, and Eric C. Desjardins. 2009. "Natural Selection and History," Biology & Philosophy, vol. 24, no. 2, pp. 231-246. Brown, Barry, and Eric Laurier. 2012. "The Normal Natural Troubles of Driving with GPS," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12) (New York: ACM), pp. 1621-1630. Chalmers, David. 2011. "Forward," in Clark (2011), pp. ix-xvi. Clark, Andy, and David Chalmers. 1998. "The Extended Mind," Analysis, vol. 58, no. 1: 7-19. Clark, Andy, Duncan Pritchard, and Krist Vaesen. 2012. "Extended Cognition and Epistemology," Philosophical Explorations, vol. 15, no. 2, pp. 87-90. 27 Clark , Andy. 2003. Natural-Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence (Oxford: Oxford University Press). Clark, Andy. 2010. "Out of Our Brains," New York Times, December 12, http://opinionator.blogs.nytimes.com/2010/12/12/out-of-our-brains/ Clark, Andy. 2011. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Oxford: Oxford University Press). Clark, Andy. 2015. "What 'Extended Me' Knows", Synthese, in press, http://dx.doi.org/10.1007/s11229-015-0719-z Cohen, Stewart. 1999. "Contextualism, Skepticism, and the Structure of Reasons," Philosophical Perspectives, vol. 13, pp. 57-89. Collins, Harry, and Martin Kusch. 1998. The Shape of Actions: What Humans and Machines Can Do (Cambridge, MA: MIT Press). Danziger, Kurt. 1997. Naming the Mind: How Psychology Found Its Language (London: Sage). DeLong, Katie. 2014. "Sheriff's Office: GPS Gets Semi Driver Stuck on Foot Bridge near North Point Lighthouse," FOX6 News, November 11, http://fox6now.com/2014/11/11/ developing-semi-trailer-truck-stuck-on-foot-bridge-near-north-point-lighthouse/ Evans, Jonathan St.B.T. 2010. Thinking Twice: Two Minds in One Brain (New York: Oxford University Press). Fantl, Jeremy, and Matthew McGrath. 2009. Knowledge in an Uncertain World (Oxford: Oxford University Press). Foley, Richard. 2005. "Justified Belief as Responsible Belief,' in Contemporary Debates in Epistemology, eds. Martin Steup and Ernest Sosa (Malden, MA: Blackwell), pp. 313326. Frankish, Keith. 2009. "Systems and Levels: Dual-System Theories and the Personal-Subpersonal Distinction," in In Two Minds: Dual Processes and Beyond, eds. Jonathan Evans and Keith Frankish (Oxford: Oxford University Press), pp. 90-108. Giere, Ronald. 2010. Scientific Perspectivism (Chicago: University of Chicago Press). Goldberg, Sanford C. 2012. "Epistemic Extendedness, Testimony, and the Epistemology of Instrument-Based Belief," Philosophical Explorations, vol. 15, no. 2, pp. 181-197. Google Maps. 2013. "Meet the New Google Maps: A Map for Every Person and Place," May 15, http://google-latlong.blogspot.ca/2013/05/meet-new-google-maps-map-forevery.html Halpin, Harry, Andy Clark, and Michael Wheeler. 2010. "Towards a Philosophy of the Web: Representation, Enaction, Collective Intelligence," in Proceedings of the WebSci10: Extending the Frontiers of Society On-Line (Raleigh, NC: Web of Science), pp. 1-5. Halpin, Harry. 2008. "Philosophical Engineering: Towards a Philosophy of the Web," APA Newsletter on Philosophy and Computers, vol. 7, no. 2, pp. 5-11. Hansen, Lauren. 2013. "8 Drivers Who Blindly Followed Their GPS into Disaster," The Week, May 7, http://theweek.com/article/index/243813/8-drivers-who-blindly-followed-their-gps-into-disaster Hutchins, Edwin. 1995. Cognition in the Wild (Cambridge, MA: MIT press). Kahneman, Daniel. 2011. Thinking: Fast and Slow (Toronto: Doubleday Canada). 28 Kornblith , Hilary. 1983. "Justified Belief and Epistemically Responsible Action," Philosophical Review, vol. 92, no. 1, pp. 33-48. MacKenzie, Donald. 2001. Mechanizing Proof: Computing, Risk, and Trust (Cambridge, MA: MIT Press). Miller, Boaz, and Isaac Record. 2013. "Justified Belief in a Digital Age: On the Epistemic Implications of Secret Internet Technologies," Episteme, vol. 10, no. 2, pp. 101-118. Miller, Boaz, and Isaac Record. Forthcoming. "Responsible Epistemic Technologies: A SocialEpistemological Analysis of Autocompleted Web Search." Milner, Greg. 2016. "Ignore the GPS. That Ocean Is Not a Road," The New York Times, February 11, http://www.nytimes.com/2016/02/14/opinion/sunday/ignore-the-gps-thatocean-is-not-a-road.html Morozov, Evgeny. 2013. "My Map or Yours? Google's Plan to Personalize Maps Could End Public Space as We Know It," Slate, May 28, http://www.slate.com/articles/technology/future_tense/2013/05/google_maps_personalization_will_hurt_public_space_and_engagement.html Palermos, S. Orestis. 2014. "Loops, Constitution, and Cognitive Extension," Cognitive Systems Research, vol. 27, no. 1, pp. 25-41. Pettit, Michael J. 2013. The Science of Deception: Psychology and Commerce in America (Chicago: University of Chicago Press). Phillips, Dom. 2015. "How Directions on the Waze App Led to Death in Brazil's Favelas," The Washington Post, October 5, http://www.washingtonpost.com/news/ worldviews/wp/2015/10/05/how-directions-on-the-waze-app-led-to-death-inbrazils-favelas/ Preston, John. 2010. "The Extended Mind, the Concept of Belief, and Epistemic Credit," in The Extended Mind, ed. Richard Menary (Cambridge, MA: MIT Press), pp. 355-369. Pritchard, Duncan. 2010. "Cognitive Ability and the Extended Cognition Thesis," Synthese, vol. 175, no. S1, pp. 133-151. Record, Isaac. 2013. "Technology and Epistemic Possibility," Journal for General Philosophy of Science, vol. 2, no. 2, pp. 1-18. Reed, John. 2016. "Google App Led Israeli Troops into Clash," The Financial Times, March 1, http://www.ft.com/cms/s/0/21966fea-dfa4-11e5-b072-006d8d362ba3.html Roberts, Tom. 2012. "You Do the Maths: Rules, Extension, and Cognitive Responsibility," Philosophical Explorations, vol. 15, no. 2, pp. 133-145. Rupert, Robert D. 2009. Cognitive Systems and the Extended Mind (Oxford: Oxford University Press). Stanley, Jason. 2005. Knowledge and Practical Interests (Oxford: Oxford University Press). Sterelny, Kim. 2004. "Externalism, Epistemic Artifacts, and the Extended Mind," in The Externalist Challenge, ed. Richard Schantz (Berlin: Walter de Gruyter), pp. 239-254. Valverde, Mariana. 2009. Law's Dream of a Common Knowledge (Princeton: Princeton University Press). Yaron, Oded. 2016. "Israeli Soldiers' GPS Blunder Proves Waze, Google Don't Really Care About Your Privacy," Haaretz, March 2, http://www.haaretz.com/israel-news/.premium-1.706638?utm_source=dlvr.it&utm_medium=twitter 29 - Notes i See Adams & Aizawa (2008; 2010) and Rupert (2009). For replies, see Clark (2011, chap. 5). ii Strictly speaking, Sterelny's claims pertain to cognitive systems, a technical term, rather than the mind per se (our thanks to John Preston for highlighting this point). iii It is worth noting that the criterion does not require that a candidate extender be trustworthy, only that subjects treat it as though it is. Neither Sterelny nor Clark is very careful about this distinction during this exchange. iv The popular app Waze, which is now a Google property, provides such functionality. v Douglas Adams' (2002, p. 42) flawless proof goes as follows: "I refuse to prove that I exist," says God, "for proof denies faith, and without faith I am nothing." "But," says Man, "the Babel fish is a dead giveaway, isn't it? It could not have evolved by chance. It proves you exist, and so therefore, by your own arguments, you don't. QED." "Oh dear," says God, "I hadn't thought of that," and promptly vanishes in a puff of logic. "Oh, that was easy," says Man, and for an encore goes on to prove that black is white, and gets killed on the next zebra crossing. vi See MacKenzie (2001) and Collins and Kusch (1998) for related accounts of "repair" practices and see Hutchins (1995) for a canonical account of the cognitive processes involved in navigation. vii Thanks to an anonymous reviewer for pressing us to explicitly address the distinction between monitoring and checking. viii Attempting to deflect responsibility, a Waze official claimed that the soldiers disabled the setting of avoiding dangerous or prohibited areas for Israelis and deviated from the plotted route. He also blamed them for ignoring roadside danger signs (Reed 2016). As Yaron (2106) notes, however, it is disturbing how easily Waze, which is owned by Google, disclosed its users' private information and violates its own terms of services when it has an interest to do so. ix Whether there is a complete or partial overlap between sub-personal and personal reasoning, and System 1 and System 2 processes, respectively is in dispute, and the two systems hypotheses is also contested (Evans 2010). Our argument only requires that the sub-personal processes which Clark mentions lack the features required for a reasoned appropriate response to the epistemic responsibilities associated with proper use of external information devices. x Within this family of subject-centered views, we may distinguish between contextualism about justified belief (Annis 1978; Foley 2005) and contextualism about knowledge, also known as "pragmatic encroach- 30 - ment" (Stanley 2005; Fantl & McGrath 2009). Such views should not be confused with attributer's contextualism, according to which standards for knowledge partly depend on semantic facts, rather than pragmatic, about the a knowledge-attributer's conversational context (e.g., Cohen 1999) xi We do not have room to make the full case for epistemic responsibilities here. However, for readers who do not accept our explanation of people's corrective behavior in terms of epistemic responsibilities, cases in which subjects check up on their epistemic artifacts out of caution or distrust still fail to conform to the original four criteria, and the extend mind thesis is in danger of being irrelevant. For more discussion, see Miller and Record (2013). xii Goldberg (2012, pp. 190-194) distinguishes between knowledge from testimony and knowledge from instruments, particularly between cases in which an instrument is used to store and transfer testimony from one subject to another. He argues that in testimony cases, there is an epistemic subject who can be held responsible for the specific content of the testimony, while in mere-instrument cases, there is no such subject. This is, in Goldberg's view, a difference that justifies disqualifying instruments, as opposed to cognitive systems of other human subjects, from constituting part of an extended cognitive process. xiii One may question our focus on GES, rather than GPS or mobile phones alone. The reason is that standalone GPS devices usually do not satisfy the constancy condition for constituting part of a subject's extended cognition, because people do not usually carry them around at all times. On the other hand, many people do carry their smartphones everywhere they go, but mere mobile phones are not primarily epistemic artifacts. Though they can be used for storing and obtaining information, this is not usually their chief function; thus mere mobile phones do not impose on their users the epistemic responsibilities with which our argument is concerned. GES devices combine these two desiderata, hence our argument focuses on them. xiv Recall that Clark and Chalmers suggest a "Parity Principle" by which potential extenders may be identified: if an external process is on a par with certain internal processes, then it should count as an extended version of that internal process. Others have taken the slightly more radical view that external processes may be considered to be extenders if they complement cognitive processes, even if they are not strictly on a par with internal processes. See, e.g., Halpin (2008). xv See Google Maps (2013), for an announcement of this development. For some speculative worries about the implications of personalized maps, see Morozov (2013).