Morality Play: A Model for Developing Games of Moral Expertise Dan Staines1, Paul Formosa2 and Malcolm Ryan3 Abstract According to cognitive psychologists, moral decision-making is a dual-process phenomenon involving two types of cognitive processes: explicit reasoning and implicit intuition. Moral development involves training and integrating both types of cognitive processes through a mix of instruction, practice, and reflection. Serious games are an ideal platform for this kind of moral training, as they provide safe spaces for exploring difficult moral problems and practicing the skills necessary to resolve them. In this article, we present Morality Play, a model for the design of serious games for ethical expertise development based on the Integrative Ethical Education framework from moral psychology and the Lens of the Toy model for serious game design. Keywords ethics, serious games, design, moral psychology, moral choices, dual-process theory, game-based learning The history of moral psychology has largely been a contest between two broad conceptions of morality. First, there is the virtueor character-based approach, which draws on the philosophy of Aristotle and "focuses on the agent and the 1 Department of Communication Studies, Concordia University, Montreal, Quebec, Canada 2 Department of Philosophy, Macquarie University, North Ryde, New South Wales, Australia 3 Department of Computing, Macquarie University, North Ryde, New South Wales, Australia Corresponding Author: Dan Staines, Department of Communication Studies, Concordia University, CJ 3.230, 7141 Sherbrooke St W. Montreal, Quebec, Canada, H4B 1R6. Email: dan.staines@concordia.ca Games and Culture 1-20 a The Author(s) 2017 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1555412017729596 journals.sagepub.com/home/gac deliberate cultivation of virtue or excellences" (Narvaez, 2008a, p. 310). Virtuebased approaches understand moral development in terms of "the importance of early habit formation" and the influence of social groups (Nucci & Narvaez, 2008). Second, there is the rationalist approach, which traces its lineage back to Immanuel Kant and emphasizes "universalist claims regarding justice and reasoning" (Narvaez & Vaydich, 2008). For rationalists, moral conduct "accords with applicable principles, derived from reasoning" (Narvaez, 2008a, p. 310), and moral development is therefore understood primarily in terms of refining one's abilities to rationally judge moral issues.1 In recent years, cognitive psychologists have established a theory that synthesizes these apparently antithetical ideas. Dual-process theory (for a review, see Evans, 2008) makes the claim that our cognitive processes take two different forms.2 Type 1 processes are implicit, intuitive, and automatic, whereas Type 2 processes are explicit, rational, and deliberative. Moral decision-making, like other kinds of cognitive processes, can involve both types. Some of our moral decisions are deliberate, conscious choices based on rules and principles, but others are intuitive and automatic, and reflect dispositions and habits. Although Type 2 processes may be consciously rational, they are effortful, slow, and often rely on implicit Type 1 processes to rapidly recognize important situational features. This interplay between Type 1 and Type 2 processes is especially relevant to designers of serious games, that is, games designed for a specific educational purpose, rather than for enjoyment (Abt, 1987, p. 9).3 Traditionally, serious games (in the form of edutainment) were designed using a didactic approach in which the learner was incentivized to absorb knowledge with game-based rewards and punishments. This approach promotes Type 2 processes but doesn't offer opportunities to employ them in a concrete setting. Gee's (2007) argument for learning to be active, situated, and material is a call for learners to have more opportunity to develop their intuitive (Type 1) decision-making skills, without neglecting the value of explicit knowledge (Type 2) delivered "just in time" to have the maximum impact. Reflection and social sharing are also emphasized by Gee as tools for making implicit knowledge explicit and inviting discussion and critique. In this article, we present Morality Play, our model for developing moral expertise games, based on principles of moral psychology and serious game design. We draw on the social cognitive theory of moral functioning and development to analyze morality as a kind of expertise comprised of implicit and explicit cognitive and emotional skills. In the following section, we describe the psychological foundations of our work, including the four component model of moral behavior and the integrative ethical education (IEE) framework for moral development. In the section on serious game design and the Lens of the Toy approach, we present the Lens of the Toy as a general-purpose theory of serious game design based on Gee's principles for deep learning. Finally, in the section on morality play, we combine these two ideas into a model of serious game design for moral expertise development. 2 Games and Culture XX(X) Background-Moral Psychology Moral psychology is a blanket term encompassing "diverse literatures and fields of study" (Narvaez & Lapsley, 2009, p. 2) focused on two interrelated spheres of inquiry: moral functioning and moral development. Moral functioning looks at the psychological processes involved in making moral decisions and behaving morally. Rather than asking what is moral, it asks how moral choices are made. Moral development then asks how these moral decision-making faculties can be acquired and trained. There are several competing theories of moral psychology in the literature. These include social intuitionism (Haidt & Bjorklund, 2007), moral grammar (Hauser, 2006), "coprimary" cognitive developmentalism (Gibbs, 2003), and social cognitive domain theory (Nucci, 2008; Smetana, 2006). These are all, at present, plausible alternatives with some degree of empirical support. Rather than try to adjudicate between these competing theories, we shall instead demonstrate the usefulness of one of these models, social cognitivism, in the context of thinking about moral expertise development through gaming. Moral Functioning The work of Lawrence Kohlberg dominated much of 20th-century moral psychology (Vozzola, 2014). His ideas defined a paradigm-cognitive developmentalism-that would eventually give rise to a multitude of diverse research traditions. Kohlberg's (1981) approach is strongly rooted in the idea of morality as rational thought, stressing the "need for moral cognition to be conscious and explicit, and for moral action to be chosen for moral reasons" (Lapsley & Hill, 2008, p. 315). One of the more serious limitations of this approach is that it is irreconcilable with the large body of research on the unconscious, implicit roots of human decisionmaking, both within the moral domain and elsewhere. Summarizing more than a decade of research, Bargh and Ferguson (2000, p. 926) note that: higher mental processes that have traditionally served as quintessential examples of choice and free will-such as goal pursuit, judgment, and interpersonal behaviour- have been shown to occur in the absence of conscious choice or guidance. Evidence of this in the moral domain includes the phenomenon of moral dumbfounding: the inability of ordinary people to justify strongly held moral convictions with coherent reasons (Haidt & Bjorklund, 2007; Lapsley, 2005). Moral dumbfounding strongly implies that deliberation does not always precede moral judgments; deliberation may be something that we engage in (if at all) only after we've already arrived at a judgment.4 Developed in part to address the limitations of the Kohlbergian paradigm, the social cognitive approach to moral personality integrates ideas from dual-process Staines et al. 3 theory to formulate models of moral functioning and development that are consistent with current psychological research (Lapsley & Hill, 2008; Lapsley & Narvaez, 2004; Narvaez, 2005). This approach argues that morality is a form of learnt expertise that engages Type 1 and Type 2 cognitive and affective processes. Reason is important because it "guides the individual in determining action" and provides "objective rationale that can be challenged, revised, reputed, or accepted" (Narvaez, 2006, p. 718–719). But moral action is not always the result of conscious decisionmaking. Sometimes morality requires that we act straight away; to stop and think what to do is sometimes to "have one thought too many" (Williams, 1981, p. 18). As such, our ethical principles often need to be automated into schemas and scripts. Schemas are "organized knowledge structures that channel and filter social perceptions and memory" (Lapsley & Narvaez, 2004, p. 195) based on our life experience. Scripts are "highly routinized action sequences" (Lapsley & Hill, 2008) which provide ready plans of action in familiar circumstances. Both scripts and schemas are Type 1 processes which are implicit, automatic and require little conscious effort. For example, a moral schema might allow the automatic recognition of gender-charged language and latent sexism in a piece of writing. A related script would habituate the practice of gender-neutral expression. These are processes that can initially require explicit effort but can become automatic over time, as appropriate schemas and scripts are developed. Type 1 moral processes are particularly important as ethical problems often arise when our limited attentional resources are engaged elsewhere. Without a welldeveloped sensitivity to moral issues, relevant ethical questions may never consciously arise, leading to what Bazerman and Tenbrusel (2011) refer to as ethical fading. However, although Type 1 moral cognitive processes are crucial, we also need to practice explicit Type 2 ethical reasoning and reflection. Decisions triggered by Type 1 processes may be driven by sexual attraction, racial stereotyping, or other irrelevant factors, which is why we sometimes need to introspect and reflect on our choices by comparing them to accepted moral standards and scrutinizing them for common cognitive biases (Gilovich, Griffin, & Kahneman, 2002). Moral Development This social cognitive account of moral functioning informs the way that we think about moral development. If morality is a dual-process phenomenon, then we need to consider how to develop and integrate explicit Type 2 and implicit Type 1 processes. To this end, Narvaez (2006) proposes the IEE framework for promoting moral expertise (p. 716). This framework is based on the four component model (4CM) of moral expertise, which identifies four kinds of skills necessary for expert moral behavior: moral focus, moral sensitivity, moral judgment, and moral action. Experts in moral focus prioritize moral concerns, cultivating self-regulation and the will to act on moral judgments. Experts in moral sensitivity are better at "quickly and accurately reading a moral situation and determining what role they might play." 4 Games and Culture XX(X) Experts in moral judgment possess "many tools for solving complex moral problems" and are capable reasoners who reflect frequently on moral issues. Finally, experts in moral action "stay on task and take the necessary steps to get the job done" (Narvaez, 2006, p. 716). In the IEE program, activities for skill development are organized into four levels of gradually ascending complexity, each intended to promote a particular kind of knowledge (Narvaez & Vaydich, 2008). At the first level, learners develop identification knowledge through being "immers[ed] in examples and opportunities" and by learning to "identify basic aspects of the domain" (Narvaez & Lapsley, 2005, p. 159). At the second level, learners develop elaboration knowledge by focusing on "facts and skills" as the teacher draws attention to the details of problems and patterns, helping learners elaborate their initial intuitions about the domain. At the third level, learners develop procedural knowledge by gaining "extensive practice solving problems in the domain." At the fourth and final level, learners develop execution knowledge by fine-tuning their procedural and conditional knowledge in collaboration with other experts and by applying their skills to different kinds of domain-relevant problems. For example, one of the more important skills associated with moral judgment is the ability to understand and analyze ethical problems. To cultivate this skill, Narvaez and Bock (2009) recommend that students develop identification knowledge (Level 1) by learning to identify ethical and nonethical problems in various media. From there, learners develop elaboration knowledge (Level 2) by examining illustrative examples of conflicting values, ethical failure, and crosscultural variation in how values are expressed. At Level 3, learners develop procedural knowledge by finding examples of national, international, and cultural conflicts, analyzing embedded dilemmas in readings, and identifying biases in social problems. Finally, learners develop execution knowledge (Level 4) by tracking ethical problems in a journal and writing reports on ethical dilemmas that interest them. These reports can be shared and discussed with others, allowing students to fine-tune their moral judgment skills by applying them in a variety of contexts and by critically testing their judgments against others. An important element of the IEE model is the development of expertise through both practice and theory. Practice facilitates automaticity, as Lapsley and Hill (2008) explain: Experts-in-training learn to make decisions in an explicit, deliberate way in the context of explicit theory and explanation. It is System 2 instruction that emphasises rulelearning and controlled processing . . . . Thus, along with the implicit (System 1) learning that comes from immersion in a situation experts-in-training are given (System 2) theoretical tools with which to "see" the domain. (p. 326) The IEE model thus sees moral development as resulting from the cultivation of moral expertise in both relevant Type 1 and Type 2 processes. Staines et al. 5 Given, as noted above, that there are several plausible alternatives in moral psychology, why focus on social cognitivism and the IEE model as the theoretical basis for moral game design? What warrants our focus here on these specific theories is that their view of moral education as a participatory, empowering, and customizable process of developing expertise is, as we shall see, entirely consistent with current understandings of effective serious game design. The IEE framework can therefore function as an effective conceptual tool for the design of moral content in serious games. Further, the 4CM provides a useful set of lenses for game designers thinking about the different aspects of ethical expertise (Ryan, Staines, & Formosa, 2017). The IEE is therefore directly usable for our purposes. Not all the alternatives share this quality.5 Serious Game Design and the Lens of the Toy Approach There are, according to Bogost (2007, pp. 237–247), two main ways to view the educational potential of video games. These are the behaviorist approach that views learning as a process in which the learner is conditioned by punishments and rewards into internalizing external knowledge, and the constructivist approach that views learning as a process in which the learner is motivated to construct knowledge internally through discovery and problem-solving. Championed by Gee (2007), constructivism has become one of the more popular paradigms for the design of serious games in the last decade. The twin engines of constructivist learning are discovery and problem-solving. Guided by an instructor or expert, learners are encouraged to observe relationships and regularities, generalize between cases, and test and refine skills and knowledge, thereby building expertise in both Type 1 and Type 2 processes. Social constructivism extends this framework by further emphasizing the role played by the social settings in which the learning takes place (Powell & Kalina, 2009). According to this view, learning is ideally a meaning-making process in which the individual participates in a "community of practice" characterized by mutual engagement, joint enterprise, and a shared repertoire of skills and knowledge (Wenger, 2011). As a member of a community of practice, the learner becomes part of a "cognitive apprenticeship" (Collins, Brown, & Newman, 1988) in which their developing expertise is guided by an instructor whose role is to prompt reflection, correct misconceptions, and promote the varied application of knowledge and skills. However, while there are many big-picture descriptions of constructivist design, there is less in the way of explicit design advice for developers wanting to create serious games using this approach. In this article, we focus on one model that attempts to do that, the Lens of the Toy by Ryan, Costello, and Stapleton (2012). We select this model because it deliberately addresses both Type 1 and Type 2 cognitive processes and integrates well with the social cognitivist approach to moral development outlined above. 6 Games and Culture XX(X) The Lens of the Toy is an approach to the social constructivist design of serious games for learning, inspired by the toy/game distinction made in Schell's (2014) Book of Lenses: To use this lens, stop thinking about whether your game is fun to play, and start thinking about whether it is fun to play with. Ask yourself: . . . When people see my game, do they want to start interacting with it, even before they know what to do? (p. 107) An engaging educational game is based around an interesting toy, which is something that is already fun to play with before goals, challenges, and narratives are added. A good toy is a complex system with many affordances that engage cognitive abilities of pattern recognition, strategic reasoning, and problem-solving. A toy need not be a physical thing: the city simulation of SimCity (Maxis, 1989) and the circuit simulation of SpaceChem (Zachtronics Industries, 2011) are both rich, playful toys. A game is then created by adding directions, goals, and structured levels to scaffold the player's interaction with the toy and guide them around the possibilities that it offers. The toy at the heart of an educational game should be a concrete model of the system that governs the learning topic. Abstract concepts that are to be taught should arise intrinsically through guided play rather than being extrinsically enforced on the game. The ideas taught by the game are not mere abstractions but are embedded in a concrete simulation. As such, the player's interaction with the game is a process of discovery and mastery. Through a combination of structured experience and advice, the player builds an explicit, abstract model of the game systems, which can be used for planning and problem-solving. But implementing these solutions requires the development of implicit schemas and scripts that turn abstract plans into concrete skills. The Lens of the Toy recommends a five-step process for the development of educational games: 1. Identify a fine-grained model of the mechanics and dynamics of the relevant real-world system. 2. Present the system to the player to facilitate the recognition of patterns. 3. Provide a tool for embodied, playful control by the player. 4. Add goals to stage the player's exposure to the system. 5. Provide support for the social sharing of player expertise. We can think of Steps 1–3 as designing the toy, an interactive simulation of the learning domain which exhibits the important concepts to be taught as patterns of play, that is, dynamics. Step 4 then describes turning this toy into a game by incrementally exposing the player to the important ideas embodied in the toy and by developing their understanding and skills through a schedule of increasingly Staines et al. 7 complex tasks. Step 5 then addresses the community around the game as a resource for encouraging the player's reflection and communication of expertise. Without repeating these steps in detail here, we emphasize that this model of design addresses both Type 1 and Type 2 processes. The model calls for the toy to be a fine-grained simulation of the learning domain, in which the important concepts are implicitly represented as dynamics of play (the patterns of interaction that arise when the game is played), rather than explicitly as mechanics (the immediate rules of the game). For example, in a game teaching introductory microeconomics, the concept of "supply and demand" should be represented as an equilibrium that emerges from the trades of many virtual agents, rather than as a rule that directly enforces this equivalence. This allows the same concept (the supply–demand equilibrium) to be presented in many concrete instances and requires that the player develops the necessary Type 1 schemas and scripts needed to recognize this pattern and skillfully control the toy. Type 2 cognitive processes are also important to analyze unfamiliar problems and suggest new solutions. As the player progresses, tutorials, mentors, and social networks can provide explicit knowledge. The level design process scaffolds this: later levels should problematize simplistic ideas learned earlier by creating anomalous situations in which automatic Type 1 skills fail and the player must consciously draw on Type 2 processes to rethink what they are doing. The game can assist players on such occasions by providing just-in-time prompting with appropriate ideas. Finally, the game should encourage players to reflect on their implicitly learned knowledge and skills, by requiring them to verbalize the schemas and scripts they are using in order to convert these to explicit knowledge. This back-and-forth between Type 1 and Type 2 knowledge is understood to be a crucial ingredient in transfer, defined as the learner's ability to apply knowledge and skills across similar but distinct scenarios (Baartman & de Bruijn, 2011). Transfer is contingent on the ability to recognize cross-situational similarities and differences, and this requires consciously adapting existing knowledge structures to fit the current context. Reflection is therefore needed to recognize the transferability of the skills and expertise developed in the game world to relevantly similar real-world scenarios (Sun, Slusarz, & Terry, 2005). Morality Play We can now consider how to design an educational game that encourages the development of moral expertise by combining the 4CM and IEE from moral psychology with the Lens of the Toy model for serious game design. To do this, we first need to design a "moral toy"-a simulation of a morally significant domain which allows for Morality Play, involving capacities from all four components of moral functioning. Next, we need to design a game around this toy, following IEE to scaffold learning through the stages of identification, elaboration, procedural knowledge, and execution (including the creation of a community of learning). 8 Games and Culture XX(X) Designing a Moral Toy Our starting point is to consider what moral skills we want the player to develop and then design a "toy"-a simulation of a morally significant domain-which engages these skills. As in the economic example above, this toy should not be a direct simulation of moral precepts, but rather a low-level simulation of a world in which moral issues can be recognized as patterns and addressed with strategies. The player is thereby not forced to choose explicit "good" and "evil" actions to gain points of "karma"; rather they need to make moment-to-moment material choices that have moral significance in the narrative context of the game. For example, the game Papers, Please (Pope, 2013) contains no morality at the mechanical level, only a representation of document processing and household economics. However, in the context of the wider narrative, these minor material choices have important moral significance, raising issues of bureaucratic dehumanization, privacy invasion, and obedience to a corrupt state (Formosa, Ryan, & Staines, 2016). When dealing with the topic of morality, it is important to focus on all four components of moral expertise. By drawing on the 4CM, we demonstrate in detail how designers can do this by asking themselves the following questions (Ryan, Staines, & Formosa, 2017; Staines, 2010):  Moral focus: Why is morality a priority in the game? What motivates the player to treat moral decisions as moral decisions and not as instrumental choices?  Moral sensitivity: How is moral content presented to the player? Are morally salient features clearly signposted or is the player expected to recognize them without prompting?  Moral judgment: What are the various moral issues at stake? What are the different moral norms or values that might drive the player's choice? What factors complicate their choices?  Moral action: Is a moral problem solved once a choice is made? Are persistence, leadership and communication skills needed to implement moral choices? The most common form of moral "toy" in video games is the scripted dialogue tree found in games such as Mass Effect (BioWare, 2007) or The Walking Dead (Telltale Games, 2012). This design can be effective at engaging players' moral focus by using well-written narratives that force players to recognize, through the responses of other characters they care about, that they are in a game world in which morality matters. The extent to which the other components of moral expertise are engaged by games designed in this way depends on the designer's commitment to long-term moral play. For example, the brief one-off moral choice encounters found in a game such as Knights of the Old Republic (BioWare, 2003) tend to be over in a couple of dialogue beats. As such, the game signposts its moral choices very clearly, Staines et al. 9 provides no more than two or three alternative solutions, and matters are resolved more or less immediately without the need for persistent moral action. The Walking Dead also contains such one-off encounters but contextualizes them in the longer term moral problem of leading the group and playing a surrogate father to Clementine. These problems require greater sensitivity to the underlying moral themes of the game, while presenting complex challenges to moral judgment, focus, and action. This again points to one of the key criteria of the Lens of the Toy model: the toy ought to be a fine-grained model in which the important learning concepts are realized as dynamics of play rather than as mechanics. We break this criterion when we reduce moral judgment and action to the mechanic of a single dialogue choice. A one-off choice between an explicit set of predetermined moral options does little to develop the player's ability to recognize, judge, and act upon moral problems. The solution to this problem is what Sicart (2013) calls the aggregation of choices (p. 105): Branching narratives are often retraceable . . . . The aggregation of choices is a better fit for designing ethical gameplay because it places players in a narrative or world context in which many choices are offered all the time, and the consequence of each is not easily traceable to a particular choice. The Walking Dead does this effectively by authoring a long story with many individual decisions that aggregate to affect the long-term narrative of the game. This, however, involves a significant authorial burden and still imposes strong limits on the player's freedom to find their own solutions to the game's moral problems. Rather than rely on prescripted dialogue, an alternative is to take a systemic approach to the design of the toy by modeling morally relevant scenarios with a generic set of rules and parameters that evolve over time (Formosa, Ryan, & Staines, 2016). Fate of the World (Red Redemption, 2011) is an example of such a game. It is a game about climate change in which the player manages social, environment, and technological policies for the world. It is based on a realistic mathematical climate prediction model, and the player's decisions feed into that model. The player is not asked directly to make moral choices, but to set policies that have significant moral implications. As such, the game calls on the player to exercise the skill of moral sensitivity by recognizing moral issues without signposting. The game also requires significant skill from the player to put their moral decisions into effect. However, impersonal simulations such as this can cause problems for moral focus, as players may concentrate on practical problem-solving and ignore moral concerns in the absence of strong characters embedded in prescripted narratives. Due to the lack of a convincing "social physics"-a systemic simulation of interpersonal social interaction (McCoy, Treanor, Samuel, Mateas, & WardripFruin, 2011)-the systemic approach to designing a moral toy seems more fitting for issues of macromorality which involve large-scale social themes that can be simulated economically, such as climate change. In contrast, issues of micromorality 10 Games and Culture XX(X) which involve complex interpersonal relationships, such as those found in The Walking Dead, seem better suited to the scripted dialogue approach. The most morally sophisticated games-such as Papers, Please-mix the two, using micromoral interactions to contextualize macromoral issues (Formosa, Ryan, & Staines, 2016). Staging a Moral Game Once we have designed the toy at the heart of the game, Step 4 of the Lens of the Toy calls for us to stage the player's exposure to the game's important moral challenges. The tools for this are level design, goal setting, and explicit instruction. The aim here is to modulate difficulty, keeping the player in the zone of proximal development (Vygotsky, 1978) where they are operating at the outer limits of their ability while avoiding prolonged frustration. Following Gee's advice, we want to stimulate discovery and provide just-in-time advice to facilitate the integration of Type 1 and Type 2 processes. The four stages of the IEE provide a structure for this process. In the following, we look at each stage in turn.6 Identification. Level 1: Immersion in examples and opportunities. In this initial phase, attention is drawn to the big picture and to the recognition of basic patterns in the skill domain. Accordingly, the teacher plunges students into multiple, engaging activities. Students learn to recognize broad patterns in the domain and begin to develop gradual awareness and recognition of elements in the domain (comprising identification knowledge). (Narvaez & Lapsley, 2005, p. 159) Consider this the "tutorial" phase of moral expertise development. At this stage, we should primarily aim to trigger the development of Type 2 knowledge in players by explicitly sharing information with them about the morally important features of the game. We need to make it clear to the player what is being asked of them in terms of focus, sensitivity, judgment, and action by providing practice tasks that exemplify these demands. For example, if we want to train the player's moral sensitivity, we should provide tasks that exhibit the kinds of morally sensitive behavior that will be required of the player (e.g., paying attention to nonplayer character's emotional reactions). Practice tasks should be clearly signposted, yielding to more ambiguous and complex problems in later stages. It is critical, at this stage, to engage the player's moral focus. As Sicart (2010, p. 5) points out, not all players will automatically play a game reflectively, considering "the meaning of the content . . . in terms of wider cultural concerns." Many will simply play the game instrumentally, treating the game as a ludic system to be "solved," and ignoring the meaning it represents. For this reason, it is important to encourage players to take a moral stance toward their decisions early in the game, Staines et al. 11 so that they recognize the game world as one in which morality matters. One way to encourage this is through role-play. By giving players a role of moral authority in the game (such as the father figure role Lee has in The Walking Dead), players are encouraged to take on a moral identity which can influence their in-game decisions.7 During this stage, players need to be made aware that their choices have moral implications, and moral sensitivity needs to be developed by drawing players' attention to moral cues as well as pragmatic ones. Explicit instruction, either diegetic through an in-game mentor or extra-diegetically through direct communication from the game, can be used to alert players to issues that should affect their moral choices. The judgment problems the player faces at this stage should be simple and obvious. Players of moral expertise games should not be expected to confront complex moral issues straightaway or to make world-changing decisions from the word go. Rather they should be introduced to the kinds of decisions they are expected to make and made aware of the values or principles that can inform those decisions. Finally, the player should be made aware that moral judgment requires moral action. The player should be required to put effort into enacting solutions to their moral problems. The risks of long-term failure, however, should be minimal at this level. The point of this stage is to provide a safe space in which the player can practice the processes of moral decision-making without fear of the long-term consequences. This doesn't mean that short-term consequences, such as an NPC expressing their hurt at the player's thoughtless behavior, can't be significant. But, at this stage, such behavior shouldn't have any long-term impacts on the relationship. For example, the NPC could make it clear that, while they are hurt by the player's behavior, they will forgive and forget about it this time. Postapocalyptic role-playing game Fallout 3 (Bethesda Game Studios, 2007) makes for an illuminating example of this first stage. Taking place in an underground "vault" full of friendly NPCs, the game's opening sequences introduce players to the karma-based morality system with illustrative, but otherwise inconsequential, dialogue. In one early scenario, the player can resolve a confrontation with a bully either diplomatically to receive "positive" karma or aggressively to receive "negative" karma. Decisions that accrue or decrease karma are accompanied by a notification at the top of the screen, alerting the player to the moral dimension of their choices. It's only after the system is introduced that the gravity of karmically charged choices escalates, with the player eventually making decisions that drastically affect entire communities. To ensure that player's comprehend moral content and the implications of their actions, explicit verification techniques, such as having an NPC interrogate the player's choices, can prove useful. An example of this occurs at the end of the first episode of The Walking Dead. The player has just saved the life of one character (Carlie or Doug) at the cost of another's. The surviving character approaches Lee and asks him to explain his decision, inviting the player to rationalize a choice made in the heat of the moment. This kind of verification mechanism reminds players that 12 Games and Culture XX(X) other characters will hold them to account in terms of relevant moral and social norms, thereby encouraging players to play the game reflectively rather than merely instrumentally. Elaboration. Level 2: Attention to facts and skills. In this phase of development, knowledge is built through a focus on detail and prototypical examples. The teacher focuses the student's attention on the elemental concepts in the domain in order to build more elaborate concepts. Skills are gradually acquired through motivated, focused attention (comprising elaboration knowledge). (Narvaez & Lapsley, 2005, p. 159) At this point, the training wheels begin to come off, allowing the player to take actions that generate genuine but limited consequences, both positive and negative. This is the stage at which prototypical problems are established and the player begins to practice the skills necessary to address them. But the environment should still be structured to give rapid, unambiguous feedback to alert the player to their mistakes. Explicit moral goals can be set for the player at this stage to encourage moral focus. This can be done, for example, by a mentor figure in the game who sets objectives for the player to achieve and provides feedback on their performance. Nevertheless, to encourage what Sicart (2013) calls "complicity" with the game's moral systems, the rewards for moral play should be intrinsic (e.g., the satisfaction of a job well done and praise from an admired role model) rather than extrinsic (points, money, etc.). The player should pursue morality for its own sake and not in the service of some external goal. In terms of sensitivity, the cues that signpost moral content should become less frequent and less obvious, allowing the player to identify opportunities for moral behavior on their own. In some cases, moral cues can be presented after a moral decision has been made, alerting the player to opportunities that they may have overlooked. The types of moral problems that players face at this stage should be simple and prototypical. These problems should illustrate basic moral principles that will be applied later. Problems can be cast as temptation scenarios where players choose between a clearly moral action and clearly self-interested action. These simple judgment scenarios invite the player to practice making moral judgments without complicating the question of what the right moral choice might be in that scenario. Moral action should begin to require skill, developed through practice. The player should be made aware that failure is a possibility that can be costly. The introduction of moral "quests" which require multiple steps to resolve is a good way of facilitating this goal. To continue the above example, in Fallout 3, the transition to Level 2 is represented in the player leaving the safe confines of the vault and venturing into a hostile world where karmic decisions stop being trivial and evoke more pronounced Staines et al. 13 reactions from NPCs and allies. Inside the vault, the moral choices concern things such as a fight with a bully over a sweet roll. In the wasteland, the player faces issues such as the euthanasia of a long-suffering mutant who holds the key to rejuvenating plant life in the wasteland. The symbolism of leaving the vault is important because it indicates to the player that the game is playing "for real" now and is a compelling example of how aesthetic and narrative devices can be used to frame the player's own developing skills and knowledge. Procedural. Level 3: Practice procedures. At this level, one sets goals, plans the steps of problem solving, and practices skills. The teacher coaches the student and allows the student to try out many skills and ideas throughout the domain to build an understanding of how skills relate and how best to solve problems in the domain. Skills are developed through practice and exploration (comprising planning knowledge). (Narvaez & Lapsley, 2005, p. 159) This level is where most gameplay typically takes place. Equipped with a bag of scripts and schemas, the player is required to combine them in various ways to solve a variety of problems. The problems become more difficult and the stakes get higher, pushing the player's limits. In terms of moral focus, the player is expected to take on more autonomy. The mentor is no longer setting explicit goals, and the player is now expected to adopt their own moral identity and to develop their own standards about what issues matter. For moral sensitivity, the important moral consequences of action and the relevant moral issues are no longer spelled out for the player. Players are also expected to be able to take up the perspectives of others who may not see things the way they do. The feedback loop between action and consequence may grow longer and ambiguous and become harder to predict. The judgment problems faced at this stage should combine the prototypical elements from Level 2 in a variety of complex situations which involve multiple moral considerations. Real moral dilemmas should be introduced by playing different moral values off against one another; for example, by having to choose between loyalty to a friend and fairness to a stranger. Moral action should become more challenging and the consequences more severe. Putting decisions into effect should now require real effort, perseverance, and even courage. The player may need to take an active role in leadership and resolve interpersonal conflicts. The Walking Dead contains numerous examples of Level 3 moral play. In terms of sensitivity, the player is required to recognize subtle and unflagged moral issues (such as the emotional impacts their choices will have on others, including those from different backgrounds). The player is encouraged to adopt a moral focus and play reflectively by being held morally accountable by others. Moral judgments are often difficult and involve conflicting moral values. Although moral action is 14 Games and Culture XX(X) usually very simple, involving little more than choosing a pregiven dialogue option under time pressure, this is not always the case. For example, one task in Episode 3 involves Lee trying to get Kenny, who is upset about his son and feeling guilty about his past actions, to stop a speeding train. The challenge is to do this without resorting to anger and violence by making a series of interconnected dialogue choices. This difficult task requires persistence, emotional intelligence, empathy, and conflict resolution skills. Without these advanced skills in moral action, you cannot get Kenny to stop the train without using force. Execution. Level 4: Integrate knowledge and procedures. At this level, one executes plans and solves problems. Deliberate practice at this level over a long period of time can lead to expertise. The student finds numerous mentors and/or seeks out information to continue building concepts and skills. There is a gradual systematic integration and application of skills and knowledge across many situations. The student learns how to take the steps in solving complex domain problems (comprising execution knowledge). (Narvaez & Lapsley, 2005, p. 159) The final level of Morality Play sees the player develop execution knowledge, by fine-tuning and integrating knowledge and skills in novel ways to solve complex, multidimensional challenges in a wide variety of scenarios. At this level, designers are encouraged to make players coordinate complex moral skills to achieve objectives that are driven by their own moral identities. The goal of this level is to encourage players to reflect on their Type 2 knowledge and have them make the implicit Type 1 skills that they have acquired explicit, thereby allowing them to be tested and refined. To encourage this, problems should be designed to present anomalous cases where the obvious solutions contain hidden flaws, and players are forced to fall back on explicit reasoning to diagnose their failures and to question their intuitive responses. Through an iterative process of accommodation and assimilation, players should advance and integrate both their explicit (Type 2) and implicit (Type 1) decision-making abilities. For example, Papers, Please trains the player to develop quick and largely automatic Type 1 skills at checking the documents of travelers in order to implement the legalistic morality of their position as inspector. This must be done quickly if the player is to progress. However, the game later throws up complex moral cases that encourage the player to stop and consciously question their obedience to the law in terms of values and principles (Type 2). This gives the player pause to reconsider the appropriateness of automatically applying their Type 1 document-checking skills in the case of, for example, a woman fleeing persecution or seeking lifesaving medical treatment. Such cases can be hard to spot, thereby requiring excellent moral sensitivity; morality can be hard to prioritize in such cases given the financial and time costs involved, thereby requiring strong moral focus; it can be hard to decide what to Staines et al. 15 do in such cases given the competing values at stake (such as security, fairness, and compassion), thereby requiring strong moral judgment skills; and it can take persistence and skill to implement moral judgments, thereby requiring abilities in moral action.8 At this stage, both IEE and the Lens of the Toy emphasize the importance of community. To aid the transition to further learning, players should be encouraged to seek advanced instruction from experts and to foster a community of learning to facilitate the social sharing of expertise. One effective way of encouraging the player to engage with the broader playing community is to record key metrics and compare them to other players. For example, The Walking Dead records how the player responds to the game's various moral dilemmas and then compares these to the responses of other players. Comparisons like these are useful because they imply the existence of a community which players can engage with and learn from. With this final level complete, the player will (ideally) have mastered the concepts and skills needed to successfully navigate the game's moral scenarios. Where a serious game is designed to develop moral expertise, it will cultivate moral skills and knowledge across all four components including sensitivity, focus, judgment, and action. The contention that underwrites Morality Play is that if the skills and knowledge that define in-game mastery are moral skills, then the player's real-world moral capacities will benefit. Although this assertion requires empirical verification, the Morality Play model sets the groundwork for such work to commence. However, the Morality Play model does not claim that a single game in isolation can develop robust ethical expertise. Rather, our contention is that games made using this model may contribute to moral expertise development and that institutional support may be required to fully facilitate integration and transfer of moral skills and knowledge. Conclusion Morality Play is a model for the design of educational moral expertise video games. Drawing on social cognitive theory and dual-process accounts of moral functioning and development, we argue for a design that exercises both implicit moral intuition and explicit moral reasoning and encourages their integration. To this end, we combine the Lens of the Toy, a constructivist approach to serious game design, and IEE, an approach to moral education derived from the social cognitivist account of moral development. The toy described by our combined model should provide a rich and subtle space for moral play. To promote learning, the player's experience of this toy should be staged through a sequence of challenges of increasing difficulty, aided by appropriate feedback and just-in-time advice. Finally, the game should be embedded in a community of players, with mechanics to encourage sharing and discussion. Although there are still many challenges involved in the design of sophisticated moral gameplay, we strongly believe this approach, which integrates understandings from across disciplines, will prove successful and therefore warrants further investigation. 16 Games and Culture XX(X) Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding The author(s) received no financial support for the research, authorship, and/or publication of this article. Notes 1. Although this accurately portrays the general reception of these two theorists within moral psychology, it ignores much detail and nuance. For example, Aristotle focused on the importance not just of habits but also of "practical wisdom" or phronesis, and Kant focused not just on moral judgment but also on virtues and good habits. 2. These are also called "System 1/System 2," "implicit/explicit," "experiential/rational,", "deliberative/adaptive," and a variety of other names. We will refer to them here as Type 1 and Type 2 processes. 3. Although the terms usage has broadened since Abt's writing to include games with other serious purposes, including health and research, in this article we only focus on serious games for learning. 4. For Haidt (2013), Type 2 processes-or "affectively laden intuitions" (p. 288)-form the basis of most moral judgments, with Type 1 processes generally serving to correct, justify, and communicate moral intuitions. Haidt acknowledges that people may occasionally "reason their way to a judgment by sheer force of logic," but he insists that such cases are "rare" and contingent on weak intuitions and high "processing capacity" (Haidt & Bjorklund, 2007, p. 193). However, Narvaez challenges this claim, arguing that "people wrestle with moral decisions . . . in a more complex fashion than Haidt and Bjorklund allow" (2008b, p. 234). 5. For example, Lapsley and Hill (2008, pp. 322–325) criticize Haidt's Social Intuitionist Model because it has "little by way of development theory and gives scant attention to educational implications," unlike "accessibility and expertise accounts of moral cognition." Whether other theories avoid this problem is beyond the scope of our focus here. 6. We have illustrated the stages with examples from familiar commercial games, while recognizing that these are not serious moral games and thereby fall short of our final vision. For a worked example of what a complete serious moral game following these principles might look like, see Staines (2016). This detailed example is not included here for reasons of space. 7. Initial data on how people engage with moral content in video games suggests that reflective play is more common than unreflective or immoral play. For example, in a small study of college-age gamers, Consalvo, Busch, and Jong (2016) found that players do not typically view games as "judgment-free zones" for unreflective play but often see avatars as heightened proxies for enacting their own moral identities. This is consistent with a similar study finding that players are "unwilling to play as evil characters" and usually "prefer to be good or heroic" (Lange, 2014), suggesting some degree of reflectiveness. 8. Given the link between moral text comprehension and moral judgment development (Narvaez, 2001), we recognize that some players will misunderstand or fail to notice Staines et al. 17 complex moral themes like the ones embedded in Papers, Please. Verification mechanisms (such as having an NPC interrogate the player's decisions) can go some way to mitigating this concern, but ultimately it is incumbent upon designers of serious moral games to consider the developmental age range of their intended audience and design appropriately. For instance, children in the "preconventional" (Rest, Narvaez, Bebeau, & Thomas, 1999) phase of moral judgment development should not be confronted with sophisticated "postconventional" dilemmas. References Abt, C. C. (1987). Serious games. Boston, MA: University Press of America. Baartman, L. K., & De Bruijn, E. (2011). Integrating knowledge, skills and attitudes. Educational Research Review, 6, 125–134. Bargh, J. A., & Ferguson, M. J. (2000). Beyond behaviorism. Psychological Bulletin, 126, 925–945. Bazerman, M. H., & Tenbrunsel, A. E. (2011). Blind spots: Why we fail to do what's right and what to do about it. Princeton, NJ: Princeton University Press. Bethesda Game Studios. (2007). Fallout 3 [PC game]. Rockville, MD: Bethesda Softworks. BioWare. (2003). Star Wars: Knights of the Old Republic [PC game]. San Francisco, CA: LucasArts. BioWare. (2007). Mass Effect 1 [PC game]. Redwood, CA: Electronic Arts. Bogost, I. (2007). Persuasive games. Boston, MA: MIT Press. Collins, A., Brown, J. S., & Newman, S. E. (1988). Cognitive apprenticeship. Thinking: The Journal of Philosophy for Children, 8, 2–10. Consalvo, M., Busch, T., & Jong, C. (2016). Playing a better me. Games and Culture, 1–20. http://journals.sagepub.com/doi/abs/10.1177/1555412016677449 Evans, J. S. (2008). Dual-processing accounts of reasoning, judgement, and social cognition. Annual Review of Psychology, 59, 255–278. Formosa, P., Ryan, M., & Staines, D. (2016). Papers, Please and the systemic approach to engaging ethical expertise in videogames. Ethics and Information Technology, 18(3), 211–225. Gee, J. P. (2007). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan. Gibbs, J. C. (2003). Moral development and reality. Thousand Oaks, CA: Sage. Gilovich, T., Griffin, D., & Kahneman, D. (Eds.). (2002). Heuristics and biases. Cambridge, MA: Cambridge University Press. Haidt, J. (2013). Moral psychology for the twenty-first century. Journal of Moral Education, 42, 281–297. Haidt, J., & Bjorklund, F. (2007). Social intuitionist answer six questions about morality. In W. Sinnott-Armstrong (Ed.), Moral psychology: The cognitive science of morality: Intuition and diversity (Vol. 2, pp. 181–218). Cambridge, MA: MIT Press. Hauser, M. D. (2006). Moral minds. New York, NY: HarperCollins. Kohlberg, L. (1981). Essays on moral development: Vol. 1-The philosophy of moral development: Moral stages and the idea of justice. San Francisco, CA: Harper & Row. 18 Games and Culture XX(X) Lange, A. (2014). 'You're just gonna be nice': How players engage with moral choice systems. Journal of Games Criticism, 1, 1–16. Lapsley, D. K., & Hill, P. L. (2008). On dual processing and heuristic approaches to moral cognition. Journal of Moral Education, 37, 313–332. Lapsley, D. K., & Narvaez, D. (2004). A social-cognitive approach to the moral personality. In D. K. Lapsley & D. Narvaez (Eds.), Moral development, self and identity (pp. 189–212). Mahwah, NJ: Erlbaum. Lapsley, D. K., & Narvaez, D. (2005). Moral psychology at the crossroads. In D. Lapsley & C. Power (Eds.), Character psychology and character education (pp. 18–35). Notre Dame, IN: University of Notre Dame Press. Maxis. (1989). SimCity [PC game]. McCoy, J., Treanor, M., Samuel, B., Mateas, M., & Wardrip-Fruin, N. (2011, June). Prom week: social physics as gameplay. In Proceedings of the the 6th International Conference on Foundations of Digital Games (pp. 319–321). Narvaez, D. (2001). Moral text comprehension. Journal of Moral Education, 30, 43–54. Narvaez, D. (2005). The Neo-Kohlbergian tradition and beyond: Schemas, expertise, and character. Paper presented at the Nebraska symposium on motivation. Retrieved from http://www3.nd.edu/*dnarvaez/documents/NarvaezNebraska03.15.04.pdf Narvaez, D. (2006). Integrative ethical education. In M. Killen & J. Smetana (Eds.), Handbook of moral development (pp. 703–732). Mahwah, NJ: Erlbaum. Narvaez, D. (2008a). Human flourishing and moral development. In L. Nucci, T. Krettenauer, & D. Narvaez (Eds.), Handbook of moral and character education (pp. 310–327). New York, NY: Routledge. Narvaez, D. (2008b). The social-intuitionist model: Some counter-intuitions. In W. SinnottArmstrong (Ed.), Moral psychology: The cognitive science of morality: Intuition and diversity (Vol. 2, pp. 233–240). Cambridge, MA: MIT Press. Narvaez, D., & Bock, T. (2009). EthEx series book 2: Ethical judgement. Notre Dame: The University of Minnesota. Narvaez, D., & Lapsley, D. K. (2005). The psychological foundations of everyday morality and moral expertise. In D. Lapsley & C. Power (Eds.), Character psychology and character education (pp. 140–165). Notre Dame, IN: University of Notre Dame Press. Narvaez, D., & Lapsley, D. K. (2009). Personality, identity, and character. Cambridge, MA: Cambridge University Press. Narvaez, D., & Vaydich, J. L. (2008). Moral development and behaviour under the spotlight of the neurological sciences. Journal of Moral Education, 37, 289–312. Nucci, L. (2008). Social cognitive domain theory and moral education. In L. Nucci & D. Narvaez (Eds.), Handbook of moral and character education (pp. 121–139). New York, NY: Routledge. Nucci, L., & Narvaez, D. (Eds.). (2008). Handbook of moral and character education. New York, NY: Routledge. Pope, L. (2013). Papers, please [PC game]: 3909. Powell, K. C., & Kalina, C. J. (2009). Cognitive and social constructivism. Education, 130, 241–250. Staines et al. 19 Red Redemption. (2011). Fate of the World [PC game]. Rest, J. R., Narvaez, D., Bebeau, M., & Thomas, S. (1999). Postconventional moral thinking: A Neo-Kohlbergian approach. Mahwah, NJ: Erlbaum. Ryan, M., Costello, B., & Stapleton, A. (2012). Deep learning games through the lens of the toy. Paper presented at the Meaningful Play, Michigan. Retrieved July 2017, from http:// meaningfulplay.msu.edu/proceedings2012/ Ryan, M., Staines, D., & Formosa, P. (2017). Focus, Sensitivity, Judgement, Action: Four Lenses for Designing Morally Engaging Games. Transactions of the Digital Games Research Association, 3(2), 143–173. Schell, J. (2014). The art of game design. Boca Raton, FL: CRC Press. Sicart, M. (2010). Values between systems: Designing ethical gameplay. In K. Schrier (Ed.), Ethics and game design (pp. 1–15). Hershey, PA: IGI Global. Sicart, M. (2013). Beyond choices: The design of ethical gameplay. Cambridge, MA: MIT Press. Smetana, J. (2006). Social-cognitive domain theory. In M. Killen & J. Smetana (Eds.), Handbook of moral development (pp. 119–153). Mahwah, NJ: Erlbaum. Staines, D. (2010). 'Video games and moral pedagogy; A neo-Kohlbergian approach.' In Ethics and Game Design, 35–51. Hershey, PA: IGI Global. Staines, D. (2016). 'Morality Play: Designing Videogames for Moral Expertise Development. ' PhD, Faculty of Arts and Social Sciences, University of New South Wales. Sun, R., Slusarz, P., & Terry, C. (2005). The interaction of the explicit and the implicit in skill learning. Psychological Review, 112, 159–192. Telltale Games. (2012). The walking dead [PC game]. San Rafael, CA: Telltale Games. Vozzola, E. (2014). Moral development. New York, NY: Routledge. Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press. Wenger, E. (2011). Communities of practice: A brief introduction. Paper presented at the STEP Leadership Workshop, University of Oregon. Retrieved July 2017, from https:// scholarsbank.uoregon.edu/xmlui/handle/1794/11736 Williams, B. (1981). Moral luck. Cambridge: Cambridge University Press. Zachtronics Industries. (2011). SpaceChem [PC game]. Author Biographies Dan Staines is a postdoctoral research fellow in the Department of Communication Studies at Concordia University. His areas of research include moral expertise development and game design. Paul Formosa is a senior lecturer in the Department of Philosophy at Macquarie University. He has published widely on topics in moral philosophy and moral psychology. He has particular research interests in Kantian ethics, moral evil, and the ethics of video games. Malcolm Ryan is a senior lecturer in the Department of Computing at Macquarie University, where he leads the game design and development program. His research area is the development of practical methods for serious game design in health and education. He also works as an independent game designer/developer through his studio WordsOnPlay. 20 Games and Culture XX(X)