Full Length ArticleAction co-representation and the sense of agency during a joint Simon task: Comparing human and machine co-agents
Introduction
The sense of agency refers to the experience of controlling one’s own actions, and through them, events in the outside world (Haggard & Chambon, 2012). This form of self-awareness is the basis of the legal and ethical structure of modern societies and forms the basis on which humans experience responsibility (Caspar et al., 2016, Frith, 2014, Moretto et al., 2011). Interestingly, studies have shown the emergence of a form of shared control when engaged in a cooperative task: we develop an experience of agency for actions and outcomes generated by our partner, a phenomenon called we-agency. This experience of agency could support our engagement in the cooperative task. Given that engagement with automated systems is essential in the context of human-machine cooperative activities such as piloting an airplane, understanding the challenges involved in developing a sense of shared agency with automata becomes a critical concern. The present study aims at addressing this issue and in particular at investigating the factors contributing to our experiences of vicarious agency during joint tasks with humans and artificial agents.
During the last decades, the sense of agency has largely been investigated using both explicit and implicit measures (for a recent review see Haggard, 2017). For instance, one of the implicit measures most widely used to quantify people’s sense of agency is the Intentional Binding phenomenon (IB; see Moore & Obhi, 2012): when a voluntary movement generates a sensory outcome, people experience an illusory temporal compression between the movement and the outcome. Importantly, this temporal binding seems to reliably occur in situations in which the participant is the agent, but not with passive movements (Haggard, Clark, & Kalogeras, 2002).
A large body of studies have used IB to shed light on how the sense of agency emerges during joint actions (Capozzi et al., 2016, Obhi and Hall, 2011a, Strother et al., 2010). On the one hand, it has been shown that people can exhibit IB for actions and outcomes generated by another human agent when they are engaged in a cooperative task (Obhi and Hall, 2011a, Strother et al., 2010). For instance, in Strother and colleagues’ (2010) study, pairs of participants were seated in front of a monitor and each member of the pair had his/her right finger on one side of the spacebar. The fingers of the participants were hidden so that no visual information about who was pressing the key was available. Participants alternately pressed the spacebar. During the operant trials, the key press triggered a tone 200 ms later. Participants had to estimate the perceived moment of the key press or the tone occurrence. By computing the action estimate and the tone estimate into one interval, the authors observed intentional binding for the other-generated shared action in a nearly similar fashion as for the self-generated action (Strother et al., 2010). In addition, in Obhi and Hall (2011a) study, participants were divided in pairs and were asked to press a spacebar. In a first experiment, the two participants could initiate the press at the time of their own choosing, but whenever one participant pressed the spacebar first, the other was to join in and press the spacebar as soon as possible. In a second experiment, the roles of the participants were pre-defined: one of the participants was to press the spacebar first while the second was always following her/his move. In both experiments, participants had to report their feeling of causal responsibility (i.e., their experience of agency) using a percentage scale and also to judge the onset time of the first key press or the onset of the tone. Interestingly, in both experiments, although only the initiator reported a reliable feeling of causal responsibility, both individuals showed IB (Obhi & Hall, 2011a). Notably, participants experienced IB for their own actions but also for the actions executed by the partner, suggesting that during joint tasks individuals can develop a vicarious sense of agency. It must be emphasized that it is the mere belief that another human agent caused an outcome (and not the fact of seeing or not the human-generated action) that allows individuals to develop a vicarious sense of agency.
On the other hand, there are studies that showed that intentional binding was not socially shared during cooperative tasks (Capozzi et al., 2016, Pfister et al., 2014). For example, Capozzi and colleagues (2016) asked participants to perform a key press that triggered after 259 ms (T1) an auditory tone that served as a start signal for a (non-naïve) co-agent key press. This co-agent key press triggered a different auditory tone which occurred at a fixed delay of 629 ms after T1 (T2). In the cooperative condition, they were told to coordinate their actions as if they wanted to create a melody with the two tones. In the competitive condition, the participants were told that the co-agent had to perform his key press as fast as possible as if he wanted to “wipe out” T1. Participants had to verbally report their perceived duration of either T1 or T2. The authors found that during sequential joint actions with a co-agent, individuals demonstrated IB for their self-generated actions but not for their partner’s action, independently of the context of the performed action (Capozzi et al., 2016). In addition, in Pfister et al. (2014)'s study, participants were paired. One assumed a leader role and the other a follower role. The leader had to press a key at the moment of their own choice and the key press triggered a tone after a certain delay (interval 1). This tone served as a go-signal for the follower to press his or her own key (interval 2). The follower's key press could trigger a tone after a random delay (interval 3) or no tone at all. Both the leader and the follower had to verbally judge the interval lengths. The results showed that the leader's interval estimations were always shorter than the follower's interval estimations, meaning that the leaders always made more IB than the followers. In addition, as the initiator of the action, the leader made IB while the follower, as the observer, did not (interval 1). For the interval 2 estimation, the leader experienced IB for the follower's action but the follower did not. Hence, leaders' sense of agency does not only concern their own actions and their adjacent effect but also predictable actions of other agents. Finally, when the follower's key press generated a sensory consequence, the temporal interval estimations (interval 3) were similar for the leader and the follower: none of them exhibited IB. That is, the follower never experienced a sense of agency whether it be over their own action and its effect or over those of the leader (Pfister et al., 2014).
Interestingly, with respect to artificial systems, there are studies that have consistently found that individuals’ sense of self-agency and vicarious agency were degraded when interacting with a non-human-like artificial agent (e.g. a computer). For example, Obhi and Hall (2011b) showed that in a joint task participants exhibited IB for their own actions and those generated by another human co-agent, whereas, when acting jointly with a machine, IB for their own actions and those generated by the machine disappeared. In a similar vein, Berberian and colleagues (Berberian, Sarrazin, Le Blaye, & Haggard, 2012) showed that human operators experienced a very strong decrease of their sense of agency when interacting with highly automated autopilot interfaces. Moreover, they reported a lack of control over the outcomes generated by the automated system. Conversely, it has been shown that observing an action performed by a human-like automaton (e.g. an anthropomorphic hand with servo-actuated tendons rotating each finger joint) induced a temporal binding effect in a similar way as during the observation of another person performing the same action (Khalighinejad, Bahrami, Caspar, & Haggard, 2016).
Taken together these studies suggest that the ability to build a sense of vicarious agency during a joint task with another agent is highly influenced by the nature of the agent one interacts with. During joint tasks with non-human-like automated machines people fail not only to experience the agency of the artificial co-actor but also their own agency. This decrease of agency during human-machine interactions might lead in turn to a reduction of involvement with the artificial system since a felt lack of control would directly impact the operator’s engagement and responsibility in the task (Caspar et al., 2016, Frith, 2014, Moretto et al., 2011).
This study aims at exploring this decrease in individuals’ sense of agency during their interactions with automated agents compared to human-human interactions. Past research suggested that the experience of agency for the actions performed by another individual is based on the ability to simulate or co-represent that individual’s motor plans and intentions (Obhi and Hall, 2011b, Wohlschläger et al., 2003; see also Kunde, Weller, & Pfister, 2017). Based on this hypothesis, people would not experience a sense of vicarious agency when interacting with computers because they fail to simulate computer-generated actions (Obhi & Hall, 2011b). In line with this assumption, the present study investigated the link between action co-representation and individuals’ experience of agency during actions performed by another human versus an automated artificial agent in a joint task. The ability to co-represent the actions of the other agent was investigated with the Social Simon task (Sebanz, Knoblich, & Prinz, 2003)
Action co-representation of human-generated actions has been classically demonstrated by the Social Simon Effect (SSE; Sebanz et al., 2003). In the classical Simon effect people’s reaction time in response to the presentation of a target stimulus decreases when the target is presented in the same relative spatial location as the response. For instance, in one version of this paradigm participants are required to execute a left hand action as soon as a red target is presented and a right hand action as soon as a green target it presented. Red/green targets are displayed in either the same relative location as the participant’s response (e.g., the target is presented to the left of a central fixation and participants have to perform a left key press) or in the opposite location (e.g., participants have to press the right key but the target is presented on the left). It has been shown that people are faster and more accurate when the target is on the same side as the response compared to when it is presented on the opposite side. In the go-nogo version of the task, that is to say when participants had to respond to only one target (e.g., respond only to the green target with the right hand), the interference disappeared. This result suggested that the Stimulus-Response congruency effect observed in the classical Simon task derives from the cognitive interference caused by two different action representations that are concurrently activated (Simon & Wolf, 1963). However, interestingly, Sebanz et al. (2003) showed that when the participant performed the same go-nogo task with another agent (joint Simon task), and each of them was required to respond to only one target (i.e., the participant responded only to the green target, and the partner only to the red target), the interference effect for the incongruent Stimulus-Response key mapping reappeared. Hence, it has been proposed that during a joint task, individuals integrate the behavior of their partner into their own motor scheme even though it is not task relevant (Sebanz et al., 2003), but see the reference coding hypothesis (Dolk et al., 2011, Dolk et al., 2013) or the task representation hypothesis (Yamaguchi, Wall, & Hommel, 2018). Consistently, neuroimaging data showed that when someone observes or imagines a conspecific performing an action, their premotor cortex is activated – though to a lesser extent than during action execution – allowing action understanding and prediction with the help of the observer’s own motor expertise (Calvo-Merino et al., 2004, Kilner et al., 2007, Pineda et al., 2000). Moreover, it has been shown that during a joint Simon task, stimuli that referred to the partner’s action evoked a similar frontal cerebral signal as the stimuli that were addressed to the participants themselves (Sebanz, Knoblich, Prinz, & Wascher, 2006).
Some recent studies have started investigating whether action co-representation also occurs during cooperative tasks with artificial automated systems. For example, a study showed a congruency effect during a joint Simon task when participants thought they were interacting with an unseen computer (Wen & Hsieh, 2015). However, other studies that used a similar paradigm found no congruency effect, whether at the behavioral or at the cerebral level, during a joint Simon task with an unseen computer (Tsai, Kuo, Hung, & Tzeng, 2008) or with a robot with high level of human-like appearance (i.e. a full humanoid robot with a torso, an anthropomorphic head, eyes, and two arms with fingers) when it was described as an unintentional robot (Stenzel, Chinellato, Tirado Bou, & del Pobil, 2012). Similarly, cerebral motor resonance has been shown not to arise during the observation of non-human-like machine-generated actions (Mann et al., 2007, Perani et al., 2001; but see Gazzola et al., 2007, Kuz et al., 2015) while action recognition mechanisms has been shown to be sensitive to artificial humanoid avatars’ actions (Ferstl, Bülthoff, & de la Rosa, 2017).
In sum, these studies suggest that during joint actions people tend to spontaneously simulate the action of another human partner but not the actions of an artificial agent (excepted when people believe that the artificial partner is an intentional human-like agent). Moreover as we mentioned above, a separate line of evidence indicates that people can experience agency for other human’s actions but fail to experience it for actions performed by artificial systems. In line with these elements of evidence, the present study sought to investigate whether the ability to experience agency for the actions performed by another agent is linked to the ability to co-represent the agent’s actions. To address this issue we ran a behavioral study combining a Social Simon reaction time task and an intentional binding task. The reaction times observed in the Simon task served as an index of action co-representation while IB was used as an implicit measure of the sense of agency. More specifically, participants performed the Simon task alone (standard Simon condition), or with another agent (joint go-nogo conditions) that could be another human or a computer (we used a desktop computer considering that it is the artificial system that people most commonly work with). Participants were required to estimate the time interval between their action or the action of the co-agent and a subsequent auditory outcome. This allowed us to investigate two issues. Firstly, it permitted us to assess whether participants co-represent the actions performed by a human and an automated artificial co-actor. For instance, if people do not co-represent the action performed by an automated system, we should observe the classical stimulus-response congruency effect (see description of the Simon task above) when participants perform a joint task with another human partner but not when they perform it with an artificial agent. Secondly, it allowed us to evaluate whether participants’ experience of self-agency and vicarious agency during joint actions is linked to their ability to co-represent the action of their partner.
Section snippets
Ethic statement
This study was approved by the CERES (recommendation n°201726, institutional ethical research committee of the Paris Descartes University, France). The investigation was carried out in accordance with the Declaration of Helsinki and each participant provided his or her written informed consent before starting the experiment. All participants were assigned a number in order to ensure the anonymity of the data.
Participants
Sixteen healthy adults volunteered to take part in the experiment (6 women, mean age
Data analyses
Our dependent measures were the mean target detection Response Times (RTs) and the mean perceived action-tone temporal interval. Statistical analyses were performed with R software (3.3.1 version). Extreme values (the values that were below or above 2 standard deviations from the mean) of the participants’ response times and perceived intervals were excluded from further analyses in order to eliminate outliers and allow for robust statistical analyses. The significance level was set at α = .05.
Social Simon effect
We first examined the SSE during joint-actions with another human and with a desktop computer (Fig. 3). We assessed the normality of the RTs distributions of the differences between the congruent trials and the incongruent trials in the standard Simon, individual go-nogo, joint HH self and joint HM self conditions using the Shapiro-Wilk test. The analyses showed that none of the RTs distribution deviated from normality (all W > 0.90 and all p > .10). We then computed a within-subjects 4 × 2
Discussion
In this study, our aim was to investigate the sense of agency in joint action with another human versus a desktop computer. To assess action co-representation, we used a typical Social Simon task where participants had to detect a target that could appear on the same side as their response key or on the opposite side. Accurate target detection triggered an auditory tone after a randomized delay. Participants had to estimate the temporal delay between the target detection and the onset of the
Conclusion
We raised the question of the relationship between action co-representation and the sense of we-agency. Given the convergence of the SSE and the vicarious sense of agency, we hypothesized that as soon as individuals are able to use their own motor system to simulate their partner’s action into their own motor system, they can understand their partner’s intention with the help of their previous experiences, supporting the construction of a vicarious sense of agency. During human-machine
Author contribution
All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding source
This work was funded by a French National Research Agency grant (Young researcher program – ANR-15-CE26-0010-01). Elisabeth Pacherie’s work was supported by ANR-10-LABX-0087 IEC and ANR-10-IDEX-0001-02 PSL*.
Conflict of interest statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References (47)
- et al.
Temporal perception in joint action: This is MY action
Consciousness and Cognition
(2016) - et al.
Coercion changes the sense of agency in the human brain
Current Biology
(2016) - et al.
Sense of control depends on fluency of action selection, not motor performance
Cognition
(2012) - et al.
Action recognition is sensitive to the identity of the actor
Cognition
(2017) Action, agency and responsibility
Neuropsychologia
(2014)- et al.
The anthropomorphic brain: The mirror neuron system responds to human and robotic actions
Neuroimage
(2007) - et al.
Sense of agency
Current Biology
(2012) - et al.
Mirror neuronsand human-robot interaction in assembly cells
Procedia Manufacturing
(2015) - et al.
Intentional binding and the sense of agency: A review
Consciousness and Cognition
(2012) - et al.
Experience of agency and sense of responsibility
Consciousness and Cognition
(2011)