1 Introduction

Education constitutes a major pillar to our modern society. Particularly self-directed and lifelong learning became paramount for people to keep up with the requirements of everyday work in a global information society. Learning and knowledge acquisition are prerequisites for many activities and jobs, but the way to learn often varies along the subject. Previous knowledge and already available skills of learners play an important role in the learning context. Learning is an essential part of our everyday life and accompanies us daily. In general, learning describes the process of psychological change based on experience (Anderson 2000; Kiesel and Koch 2012). Optimal learning requires high motivation to solve and repeat tasks that require prior knowledge and provide immediate feedback (Ericsson et al. 1993). Direct feedback shows the learner that every step towards the overall learning goal counts, which in turn gives the learner a sense of competence (Dichev et al. 2015). Moreover, an effective learning process requires the active participation of the learner (Beer et al. 2010), often referred to as engagement. Consequently, the motivation and engagement of learners can be seen as critical for a successful learning process. To this end, there is high demand and interest for the integration of new technologies into the context of learning. Oliver (2018) states that the primary goal of technology-based learning is to improve the quality of the learning process. If implemented, learners ideally experience not only positive effects on their learning success, but also a more pleasant and flexible learning process (Hassan Taj et al. 2017). The technological extension of a learning environment should, therefore, result in positive motivational and interest-based effects. Several technical solutions have been proposed to scaffold learners to stay motivated in their self-directed learning process. A common type of technology-based learning environments are Massive Open Online Courses (MOOCs). However, MOOCs suffer from a high dropout rate due to various factors (Aldowah et al. 2020). This might potentially be avoided by integrating new technologies in the learning process and implementing alternative learning concepts, supporting and extending traditional educational settings. One approach can be the supplementation of common instructional methods by a social robot.

Recently, social robots have started to emerge as a new technology in language learning with a focus on educating children. In an exhaustive review, van den Berghe et al. (2019) conclude that robot-assisted language learning has potential, but robots cannot conclusively be considered effective language tutors. Van den Berghe et al. (2019) stress the importance of not only considering learning gains but also motivation of participants, as well as the problematic influence of novelty effects. The empirical basis is even more unequivocal in the area of gamification, which is in itself no new technology, but an approach to use game elements in non gaming contexts to enhance motivation (Deterding et al. 2011). A review by Seaborn and Fels (2015) states that even though there appear to be benefits to use gamification in learning, the field is unstructured, lacks controlled studies and theory. An even more recent meta-analysis on gamification by Sailer and Homner (2020) confirms a generally positive effect, but also stresses the problem of insufficient empirical basis and the need for more methodological rigorous studies. We belief that social robots and gamification are tools that hold the potential to benefit learning environments and can thus potentially help individuals of our modern society to improve their learning process.

The present study addresses the extension and supplementation of a digital learning environment for the acquisition of a foreign language in respect to the heterogeneous situation in social robotics and gamification research. The aim was to explore the integration of a social robot and gamification in adult learning, investigating their separate as well as combined effect. As for the integration of the social robot, we want to emphasize that we do not seek to replace teachers in the learning context, but improve the learning process and provide social support for self-directed learning situations in which no teacher is available. The integration of the gamification elements or a social robot might have benefits in terms of engagement, motivation and learning success. A symbiosis of gamification elements and a social robot could also improve the success of a learning environment. Under consideration of various determining factors discussed in Sects. 2.3.2 and 2.4.3, the integration of a social robot and gamification in an educational context should be examined with regard to the effectiveness of these concepts. Our work provides insights into how well and reliable a comprehensive implementation of both approaches and their combination in a learning environment works under standardized conditions.

We focus on a highly controlled setting regarding the learning context, as well as measurements for learning success and retention of knowledge gain in respect to motivation and engagement, which are critical in self-directed learning. Although both, the playful processing of knowledge and technology-supported learning, are not a clean slate, the present work is intended to provide new insights, particularly with regard to the linking of gamification with social robots in the learning context. In this contribution, we first provide an overview and critical view on gamification and social robots in the learning context. Second, we describe the theory-based development of a technology-based learning environment integrating both gamification elements and a social robot. In Sects. 4, 5 and 6, we present the results of an evaluation of the learning application and discuss them with a view to the research context.

2 Related work and theoretical background

We first establish a short overview of our respective theoretical approaches to learning, motivation and engagement and subsequently present a more in depth view on social robots and gamification in the context of learning. Both areas of research are evaluated in terms of their influence on learning, motivation and engagement based on concurrent research. As indicated at the outset, we consider both approaches critically in terms of the potential benefit of standardized integration of social robots and gamification into education.

2.1 Learning in the context of social constructivism

Assuming that learners are individuals who actively participate in their knowledge acquisition process allows a consideration of the concept of learning under the theory of social constructivism as proposed by Vygotsky (Vygotsky 1962; Schreiber and Valle 2013). One of the core components of social constructivism is the zone of proximal development (ZPD), highlighting the role of the teacher in the process of knowledge acquisition. ZPD describes the range between what learners can do with and cannot do without help and encompasses all problem-solving activities that a learner can perform in the presence of a more knowledgeable other (Vygotsky 1981). Hence, within the ZPD, learning might occur when interacting and cooperating with peers that are more capable than the learner (Vygotsky 1981; Eun 2019).

Wass and Golding (2014) provide a conceptual analysis on the ZPD in the context of higher education, deriving central implications for its application to higher education concepts. In that regard, the authors further examined how learners can be supported when solving a task for which they need assistance. This can be done by scaffolding the learning subject in several supportive activities that students are able to complete. Learning material could be structured in separate tasks, hence reducing its complexity, providing learning directions or setting focus on the most important contents (Wass and Golding 2014; Reiser 2004).

Wass and Golding (2014) also indicate that the range of the ZPD might vary depending on the learning environment, indicating that an environment that permits and constructively addresses errors via direct feedback and further provides a safe learning setting without time pressure could increase the student’s ZPD and therefore potentially benefit learning.

2.2 Motivation and engagement

Motivation plays a central role in ensuring learning success for learners (Linnenbrink and Pintrich 2002). According to Ryan and Deci (2000b), motivation includes all aspects of human intentions and activities. Especially in the educational context, situational motivation is an important prerequisite because individuals engage in learning activities. Situational motivation particularly refers to the motivation experienced while performing educational tasks (Vallerand 1997).

The construct motivation can be explained within the framework of self-determination theory. The theory is an approach to explain the human striving for self-improvement and the fulfillment of psychological needs as the basis for one’s own motivation and personality development (Ryan and Deci 2000b). According to self-determination theory, human beings have three central needs inherent: the need to experience competence, the need for autonomy and the need for relatedness. The intrinsic motivation is directly related to human needs for autonomy and competence experience and may also be linked to the desire for relatedness (Ryan and Deci 2000b). In the learning context, this means that learning environments that satisfy the three central human needs, with special focus on autonomy and competence, are expected to induce intrinsic motivation in learners (Ryan and Deci 2000a). Therefore, learning environments must have a basic motivational potential. Aspects such as novelty, challenge or aesthetic demands can create such motivational potential (Ryan and Deci 2000a). The perceived self-determination and, therefore, the perceived autonomy of the learners varies depending on the potential of the learning environment to satisfy the three central needs. Because the support of motivation is especially critical in self-directed learning, which requires to continuously re-engage with the learning material, we consider motivation as central indicator for the adaption of the learning environment.

In addition to the motivational perspective, the engagement of the learner is a relevant factor for the efficient and sustainable acquisition of knowledge (Finn and Zimmer 2012; Rodgers 2008). Engagement in the context of a social interaction is to be seen as the combination of the factors attention and understanding (Heath et al. 2017), as the basis of an interaction process within which people establish, maintain and end a connection with each other (Sidner et al. 2005). In the learning context, engagement describes the active and emotional involvement as well as the workload of the learner as quality criteria of the user experience (Beer et al. 2010; O’Brien et al. 2018). Engagement is directly related to a potential learning outcome (Beer et al. 2010). The design of the learning environment might also be critical to learner engagement (Beer et al. 2010). Therefore, learning environments should generally be exiting, new, esthetically pleasing and offer attractive and interesting content, for example in the form of a technology-supported presentation of learning content. Furthermore, the integration of challenges as well as permanent feedback to the user serves as a holistically motivating concept that induces engagement (O’Brien and Toms 2008).

2.3 Social robots in education

The way in which knowledge is imparted and acquired varies from individual to individual, so different methodologies should always be used in learning environments to reach as many learners as possible (Meghdari et al. 2013). A social robot can provide a useful alternative approach. Li et al. (2016) describe social robots as “devices with mechanical moving parts that interact in socially appropriate ways” (p. 1224). Social robots are therefore able to participate in social interactions and react to human interaction partners (Breazeal 2003), including in learning situations.

The integration of social robots into learning environments is one of their largest and most relevant fields of application. In educational contexts, social robots can support learners in the form of Pedagogical Agents (PAs). PAs are generally described as a computer-based system in the role of a teacher, tutor or learner who communicates and interacts with learners in everyday language. A PA can embody a personality of its own and adapt its mode of interaction to the behavior of the learner (Johnson et al. 2000; Pérez-Marín and Pascual-Nieto 2013).

Accordingly, the implementation of a robot in its role as a PA in both real and virtual learning environments can be beneficial. In general, the robot is actively involved in the learning environment and can react responsively to the learner, which can also result in a direct interaction of both parties. The inclusion of physical activities is also conceivable (Han 2010). Especially the use of gestures and voice variation generates attention and can motivate learners to participate (Chang et al. 2010). In summary, the integration of robots in an educational context can be useful and beneficial to learners (Chang et al. 2010).

A social robot in the role of a PA complements the learning environment through its physical presence. This presence of the robot can play a relevant role with regard to the first impression and the ideas that participants of a learning environment have about the robot. Thus, the targeted educational field should be taken into account when designing the social behavior of the robot (Leite et al. 2013). Using a robot as a PA therefore might be a meaningful addition to existing learning environments in order to provide a stimulating, interesting and instructive learning experience (Mubin et al. 2013). In its function as a tutor, the robot can also significantly increase the concentration, interest and performance of learners compared to a learning environment without a robot (Han et al. 2008).

Along with the concept of social constructivism and the ZPD (Vygotsky 1981; Eun 2019), a social robot in the role of a PA, which is perceived as more capable might benefit the learning process by assisting the learner through the course of the learning environment. The social robot could provide learning directions by guiding the learner, giving direct feedback and ensuring a safe learning environment, therefore increasing the learner’s ZPD and facilitating learning.

In summary, a social robot employed in an educational setting might provide a useful addition for the learning process by enriching the learning experience with the use of gestures and voice variation (Chang et al. 2010) as well as its physical presence (Leite et al. 2013) which in turn might benefit the affective state of learners and their learning process.

2.3.1 Benefits for learning, engagement and motivation

The behavior of the robot and active integration into the learning environment consequently may lead to a higher learning effect for learners (e.g. Han et al. 2008; Mazzoni and Benvenuti 2015; Mwangi et al. 2017; Pfeifer and Lugrin 2018). Unlike virtual PAs, a social robot’s physical presence in the learning environment might also be beneficial for the learning process and result in greater learning gains (e.g. Alemi et al. 2015; Kennedy et al. 2015b; Leyzberg et al. 2012; Ramachandran et al. 2018). Critical for increased learning success is the positive and confirmative feedback from the robot (Alemi et al. 2015; Deublein et al. 2018) as well as the possibility for social interaction of the learners with the robot (Belpaeme et al. 2018; Saerbeck et al. 2010).

A review of van den Berghe et al. (2019) found that working with a social robot in a learning environment especially benefits learning-related variables, such as engagement or motivation. A social robot is able to induce engagement and thus enhance the learning experience for learners (Chang et al. 2010; Kennedy et al. 2015a; Ramachandran et al. 2018; Sidner et al. 2004, 2005). It is possible for the robot to influence learner engagement through speech and conversation as well as non-verbal behavior, such as eye contact or gestures (Anzalone et al. 2015; Hall et al. 2014; Sidner et al. 2004). Therefore, the robot is able to continuously maintain the attention of its interaction partner, which might result in a more attractive learning environment and higher user investment regarding the interaction process (O’Brien et al. 2018).

Additionally, the social robot can have positive effects on the motivational state of learners (Alemi et al. 2014; Chang et al. 2010; Kanda and Ishiguro 2005; Saerbeck et al. 2010). Learners also have the feeling of learning more relaxed and effective together with the robot, which can maintain the motivation to learn in the long term (Alemi et al. 2014). Motivation is critical for the successful acquisition of knowledge (Linnenbrink and Pintrich 2002), which is why robot-supported learning and the associated increased motivation of those involved might therefore be a meaningful continuation of developments in self-directed learning. According to self-determination theory, a positive motivational effect of integrating social robots into the learning context can also be assumed. During the interaction in the learning environment, the robot is able to acknowledge the learner’s feelings, thus fulfilling their need for autonomy (Deci and Ryan 2000). In addition, the integration of a social robot allows for visualization of progress and positive feedback and can thus improve learners’ competence experience (Alemi et al. 2015). In its role as tutor, the robot is also able to convey relatedness through interactive behavior (Han 2010). Furthermore, the recognition of the learner’s performance by the robot, for example in the form of verbal comments, can affect the need for relatedness in a positive way (Ryan and Deci 2000a). It can therefore be assumed that a social robot is able to address all three central human needs and benefit intrinsic motivation accordingly.

The integration of a robot as a tutor can result in positive effects for the learner, especially with a view to increasing the learning effect (Alemi et al. 2015). Lee et al. (2010) demonstrated significantly improved language skills of learners interacting with a robot in a comparison between pre- and post-testing. Nevertheless, it should be emphasized that the integration of the robot into the learning context, especially when learning a foreign language, should support rather than replace teachers (Meghdari et al. 2013). This refers especially to situations with no teacher or tutor available, i.e. in self-directed learning environments.

2.3.2 Critics on social robots and deriving central design applications

The integration of a social robot in an educational setting has been demonstrated to have unexpected side effects. In a study by Kennedy et al. (2015b), two robots were presented to children, one of them showing social, the other antisocial behavior. The children’s gaze behavior indicated that the children increasingly focused their attention and eye contact on the gestures of the social robot instead of the learning content. Excessive social behavior of the robot might not only have attracted the children’s attention, but might also have impaired their concentration and receptiveness (Kennedy et al. 2015b) and negatively affected the interaction experience (Belpaeme et al. 2018). Thus, the novelty of the robot is both an incentive and a potential for shifting the attention of the interaction partner away from the subject of the interaction towards the way the robot behaves (Leyzberg et al. 2012).

Furthermore, it is questionable to what extent the robot can maintain interest and motivation over a longer period of time. Unfulfilled expectations of the robot as an interaction partner or a habituation to the robot as well as the associated loss of the novelty effect can lead to a reduced interest (Leite et al. 2013). Critical for maintaining interest in the robot over a longer period of time could therefore be the use of different behavioral patterns and varying but continuous social interaction of the robot. In this context, the addition of new behaviors also seems to be beneficial (Kanda and Ishiguro 2005; Leite et al. 2013), especially with a view to maintaining attention and active involvement of learners and thus retaining their engagement level (O’Brien et al. 2018).

Even under the assumption that the inclusion of a social robot in an educational setting benefits learners in terms of engagement and motivation, this does not automatically result in an increased learning effect. Regarding potential learning outcomes, integrating a social robot in a learning environment often yields mixed results (van den Berghe et al. 2019; Zhong and Xia 2020). Because learning gains are often related to academic constructs such as motivation (Pekrun et al. 2002), the learning effect might vanish if the robot has been interacting with the user for a long time and if the initial increase in motivation was based on a novelty effect (van den Berghe et al. 2019). Additionally, van den Berghe et al. (2019) indicate that different methodological approaches limit the comparability of results regarding learning gains. This refers to varying roles of the social robot in the learning environment as well as the degree of human control required during the interaction scenario (van den Berghe et al. 2019).

The educational context in which social robots are employed is often unilateral. Most studies target (pre-)school children and young pupils (van den Berghe et al. 2019; Belpaeme et al. 2018). Yet, there is comparatively little research regarding the use of a robot in higher education and adult learning. It is, therefore, unclear whether potential positive effects of integrating a social robot in elementary school settings are equally transferable to its use in a university context or similar learning activities for adults (Belpaeme et al. 2018).

Physical integrity is also a critical factor that must be taken into account. The safety of the participating persons during an interaction with a social robot must be ensured, especially if this interaction takes place without additional supervision by an involved third party (Leite et al. 2013). Chang et al. (2010) also mention the cost of purchasing and maintaining a robot as a significant limiting factor. Furthermore, the operation as well as the handling of a social robot requires a certain level of knowledge, which can potentially complicate the integration of the robot into everyday contexts and the resulting operation by inexperienced laypersons (Chang et al. 2010). Finally, a social robot is also susceptible to technical malfunctions such as network problems or general technical difficulties (Han 2012). This circumstance can also have a negative impact on the perceived reliability of the robot (Sidner et al. 2005). Consequently, it is important to carefully consider whether and in what way the integration of a social robot will add value in the respective context, taking into account the intended application situation and the resulting requirements for the robot.

2.4 Gamification in education

Technology-based learning can be extended by various components. One form is the integration of so-called gamified elements into the learning environment. Dicheva et al. (2015) define gamification as the use of “game thinking and game design elements to improve learners engagement and motivation” (p. 75). Gamification has the potential not only to impart knowledge and skills, but also to make the process of teaching interesting for learners without having to fear making mistakes (Gladun 2016; Iosup and Epema 2014). Gamification can be seen as a multi-stage process of gamifying an application, within which different building blocks extracted from video games are implemented in other environments. This intervention is often based on the goal of provoking or motivating certain behaviors in the gamified environment in order to achieve a desired effect (Sailer et al. 2017).

Specifically with a focus on the learning context, enriching a learning environment with game elements can lead to learners spending significantly more time in it, which can have a positive effect on their performance in the long run (Hakulinen et al. 2015; Todor and Pitica 2013). Gamified applications encourage active participation in the learning environment in addition to increased usage time. By providing various opportunities to improve one’s skills, interest and motivation can be maintained (Lee and Hammer 2011; Su and Cheng 2015). The implementation of gamification elements may encourage learners to set meaningful goals, redefine failure, and provide direct, fair and incremental feedback (Dichev et al. 2015). The primary goal of integrating game elements into non-playful contexts should, therefore, be to create activities that users are happy to engage in voluntarily (Werbach 2014).

Gamification serves as an umbrella term for numerous game mechanics. In a gamified environment, central motivational aspects can be addressed via so-called rewards or achievements (Darejeh and Salim 2016). The primary goal of integrating reward systems in an application is to create positive experiences for the user (Wang and Sun 2011). Reward systems set realistic goals for the user, which increases motivation and improves the overall user experience. Rewards represent the status of the user within the application, attract attention, enable social interaction with other users and encourage them to collaborate. In addition, they can arouse curiosity and fun in discovery (Fitz-Walter et al. 2011; Wang and Sun 2011). Implementing rewards in applications can, therefore, increase the users’ enjoyment of the environment before they even receive them. The pursuit of the reward in itself is already having a positive effect (Wang and Sun 2011). A well-designed application should, therefore, reward any type of user, regardless of their skills and experience, by maintaining a balance between the user’s increasing abilities and the challenge (Fitz-Walter et al. 2014; Hakulinen et al. 2015).

2.4.1 Reward concept and the identification of relevant forms of rewards

Iosup and Epema (2014) structure gamification into three basic mechanics and four basic dynamics. The mechanics outlined in this section include point systems, levels and leaderboards. The basic dynamics include badges, tutorial functions, unlocking content and so-called social engagement loops. We will add three more game elements in this context, which are also frequently used and are characteristic components of games: Setting clear goals, background information and stories and feedback (Dichev et al. 2015; Kapp 2012).

Points are a basic element of many games and gamification applications and are among the five gamification components that induce the highest level of engagement in users (Chang and Wei 2015). The user usually receives points for completing certain tasks within the gamified application. Points represent and measure the individual progress of the user (Sailer et al. 2017) and serve as a virtual reward and direct positive feedback on the user’s behavior (Kapp 2012; Sailer et al. 2013). In general, points are added up over the course of the learning environment (de Byl 2012; Sailer et al. 2013). Levels serve as divisions of the game world. The difficulty of the levels often increases with the progress of the player (de Byl 2012). Users can thus gradually unlock new levels and experience a sense of progress within the application. In the learning context, dividing the learning content into small units and grouping-related content can be beneficial (Dichev et al. 2015; Steinhaeusser et al. 2019). New content can reward and motivate learners to continue to interact with the learning environment (Iosup and Epema 2014). Leaderboards are the third element of the three basic gamification mechanics. Due to contradicting effects regarding the comparison of performance in the learning context such as demotivation (Falkner and Falkner 2014; Fotaris et al. 2016), leaderboards were excluded in the present study.

Visualization of achievements in the form of images as a reward for certain actions in the game is a common gamification dynamic. So-called badges are one of the five game elements that evoke great engagement among users (Chang and Wei 2015) and can form an additional component of the learning environment that offers optional goals and rewards (Hakulinen et al. 2015). They are intended to motivate the user to continue the user experience and provide a visual representation of individual performance (e.g. Nah et al. 2014; Sailer et al. 2013, 2017). In addition to reward-oriented goals, such as badges, users should also have a clear substantive goal to pursue, which they approach through completing certain tasks in order to achieve a specific goal (Darejeh and Salim 2016). In general, a goal can be subdivided into small goals, which are clearly defined and directly related to rewards. Apart from badges and the step-by-step processing of learning units, learners might also get access to new application features by unlocking new content, provided that they meet certain requirements (Wang and Sun 2011). The iterative unlocking of content can serve as a tool to control the learner's progress (Iosup and Epema 2014). Social engagement loops additionally integrate a social component into a learning environment which creates social pressure that encourages learners to be present in the learning environment and can motivate them to regularly return to an application to engage with it again (Iosup and Epema 2014).

To connect tasks and rewards in a learning environment, they can be embedded in a simple background story. Such a story can provide a context for potential rewards (O’Donovan et al. 2013; Villagrasa and Duran 2013), while the storyline can be the scenario or theme of the gamified environment (Darejeh and Salim 2016). The story should span the learning content in such a way that the plot is perceived as significant for the user and allows for transfer of the learning content into a real environment (Nah et al. 2014). To ensure that users are informed about the correct operation and functions of a particular application, it is also important to introduce them to the environment in advance by using tutorials.

Feedback is a central component of many video games. Games provide the user with direct feedback about their potential success and improvement possibilities. Players thus receive constant feedback on their progress in the game and on the tasks that they have to complete to move forward (Dichev et al. 2015; Todor and Pitica 2013; Wang and Sun 2011). Of particular relevance here is direct, timely feedback on user interaction, both positive and negative (Darejeh and Salim 2016; Nah et al. 2014; Villagrasa and Duran 2013). Gamified environments ideally provide continuous feedback in response to user behavior (Eckardt et al. 2015). The goal of feedback is to create positive emotions in the user, for example through images, sound effects or videos (Wang and Sun 2011).

2.4.2 Benefits for learning, engagement and motivation

The meaningful and mindful integration of gamification in a learning application can have a beneficial influence on the learning success of the users (O’Donovan et al. 2013; Su and Cheng 2015). Positive effects on the engagement and motivation of the learners are also possible. Existing research explicitly argues for the positive impact of gamification on learner engagement (Akpolat and Slany 2014; Barata et al. 2013; da Rocha Seixas et al. 2016; Darejeh and Salim 2016; Ding et al. 2018; O’Donovan et al. 2013; Sun and Hsieh 2018), including indicators of active participation and a corresponding workload for learners (Fotaris et al. 2016). Emanating recent research demonstrates that in more than half of the reviewed gamified learning applications, the integration of game elements such as scores, levels, or progress indicators can improve learner engagement (Darejeh and Salim 2016; de Byl and Hooper 2013; Korkealehto and Siklander 2018).

In addition to a beneficial effect on learner engagement, it can also be assumed that the integration of gamification elements into the learning environment can motivate learners to learn better and invest more time in academic success (Lee and Hammer 2011; Sousa Borges et al. 2014). In particular, meaningful gamified learning environments increase intrinsic motivation because users perceive their actions as significant (Mekler et al. 2013; Sailer et al. 2017; Sakamoto et al. 2012). Even if the integration of gamification in practice is often not linked to theoretical constructs as summarized in a review by Seaborn and Fels (2015), several studies demonstrated that gamification elements can address all three basic needs as outlined by self-determination theory. The need for autonomy can be fulfilled by granting users freedom of choice regarding the order and type of task processing within the learning environment (Barata et al. 2013). A significant contextual embedding of the learning environment, for example by means of background stories, also allows users to perceive their actions as meaningful and relevant (Sailer et al. 2017). Learners experience their behavior as competent by receiving positive visual and verbal feedback about their progress, for example in the form of points or badges (Barata et al. 2013; Dichev et al. 2015; Sailer et al. 2017). However, gamification elements only influence the competence experience of the users if they perceive tasks as challenging. Visually appealingly designed and audibly clearly presented gamification elements can also increase their potential to positively influence the users' experience of competence (Mekler et al. 2017). By enabling competition, cooperation and interaction with other learners, the need for relatedness can be fulfilled (Barata et al. 2013; Sailer et al. 2017). A narrative adapted to the learning environment also gives the user a relevant role within the application and provides meaning to the use (Sailer et al. 2017).

2.4.3 Critics on the gamification concept

However, the integration of gamification does not guarantee an optimal and universally applicable template for improving user environments and applications. If learners receive a reward for all actions, there is a risk that they will focus on the reward rather than on the learning process (Gladun 2016). The type and time of awarding a reward, therefore, determines whether the reward yields the desired effect. While verbal rewards might affect intrinsic motivation in a positive way, the effect for material rewards can be contradictory. If the user is aware of the material rewards and expects to receive more, the learner’s intrinsic motivation might decline (Deci et al. 2016). Learners may also feel that learning is always linked to external rewards and is, therefore, reduced to contexts with rewards (Hakulinen et al. 2013; Lee and Hammer 2011). Furthermore, if gamification is firmly integrated into the curriculum, the associated commitment can eliminate the positive effects, because learners might perceive the gamification elements as an obligation rather than an enrichment (Lee and Hammer 2011). In addition, developing meaningful gamified applications costs resources, especially teachers’ resources, which are not always available (Lee and Hammer 2011). The integration of gamification in an educational context therefore requires careful consideration of the relationship between potential costs and expected benefits to provide a useful addition.

The greatest point of criticism is the accumulation of external incentives in gamified applications. The mere integration of game elements without consideration of the context or system-specific adaptation to the learning environment should be viewed critically. Users might work in a rewarding way without having any opportunities to develop intrinsic motivation (Dichev et al. 2015; Hakulinen et al. 2015). As a result, they do not develop intrinsic motivation in gamified applications, but believe their external motivation to be intrinsic (Seaborn and Fels 2015). This circumstance is diametrically opposed to the goals of the learner, especially in the long term, and might even inhibit learning. Supplementing a learning environment with externally motivated game elements could also give learners the impression that the application in its original form is not intrinsically motivating (Hakulinen et al. 2013, 2015). Thus, the integration of gamification elements might not necessarily fulfill the hoped-for intrinsically motivating function. However, Mekler et al. (2017) clearly point out that gamification elements can act as external motivators, but the intrinsic motivation of the users remains unchanged.

Therefore, gamification should be implemented mindfully and based on theory. Research in the application of gamification in educational environments often lacks a theoretical basis (Nacke and Deterding 2017; Seaborn and Fels 2015) or applies insufficient methodologies (Dicheva et al. 2015) and uses small sample sizes (Lumsden et al. 2016). The way in which gamification is integrated into the learning context determines what effects and influences it has on learners and their performance (Lee and Hammer 2011). This requires individual examination of potential application areas and whether gamification can provide a meaningful addition in the specific context (Dicheva et al. 2015). It should be noted that not all applications can be gamified and that the use of gamification is not meaningful or possible in all contexts (Akpolat and Slany 2014; Kuo and Chuang 2016; Nacke and Deterding 2017). Gamification must, therefore, implement its own learning approaches, for example by means of an effective, satisfying reward system and re-establish unfamiliar learning patterns in order to be effective and conducive to learning (Akpolat and Slany 2014; Dichev et al. 2015). Additionally, framing the concept of gamification by providing a theoretical context, such as self-determination theory, can be a fruitful approach to link research to theory.

2.5 Expectations from related work

To systematically investigate the impact of the discussed concepts, we integrated a social robot and various gamification elements into a technology-enhanced learning environment and examined both separately and for a combined effect, compared to a control group working on the same learning environment without gamification elements or a social robot. Assuming that the integration of a social robot and gamification elements in a learning environment might potentially benefit the learning process and knowledge acquisition, we aim to address and critically examine this premise by formulating eight hypotheses.

Regarding the social robot, beneficial effects on the engagement of the participants are expected (Anzalone et al. 2015; Ramachandran et al. 2018). Consequently, learning together with the robot should result in higher engagement values relative to using only a tablet (H1). Based on research on the motivational effect of integrating a social robot into the learning context (Alemi et al. 2014; Chang et al. 2010; Kanda and Ishiguro 2005), the intrinsic motivation of the participants should be higher when using the learning environment with the robot relative to the learners in the control condition who learn on their own (H2).

According to the assumed positive impact of gamification elements on the engagement of users (Darejeh and Salim 2016; de Byl and Hooper 2013; Korkealehto and Siklander 2018), users of the gamified application should be more engaged than users of the non-gamified learning environment in the control condition (H3). Furthermore, a positive effect on the intrinsic motivation of the participants might be expected (Mekler et al. 2013; Sailer et al. 2017; Sakamoto et al. 2012). Therefore, we postulate a higher intrinsic motivation of learners when using a gamified learning environment compared to a version without gamification elements in the control condition (H4).

Learners who use the robot are expected to show higher learning success (e.g. Alemi et al. 2015; Han et al. 2008; Kennedy et al. 2015b; Leyzberg et al. 2012) than learners who only use a tablet (H5). The integration of gamification into a learning environment might also positively influence learning success (O’Donovan et al. 2013; Su and Cheng 2015). We, therefore, assume that the integration of gamification elements in a learning environment leads to higher learning success compared to learning with a non-gamified application (H6).

So far, the addition of a social robot and gamification elements in an educational setting is expected to potentially benefit intrinsic motivation and engagement. Consequently, we would also anticipate a positive effect on intrinsic motivation (H7) and engagement (H8) for the combined approach. Very few research has been conducted regarding the combined integration of gamification and a social robot in a learning environment, yielding mixed results (cf. Donnermann et al. 2021). Due to the lack of comprehensive knowledge about the effect of both gamification and a social robot on the learning effect, research question 1 aims to find out whether users benefit in terms of their learning success from a combination of gamification elements and a social robot in the learning environment (RQ1). Because excessive social behavior might be perceived as distracting (Kennedy et al. 2015b), the combined integration of a social robot and gamification might also result in adverse effects for learners.

3 Learning environment

For our endeavor, we developed an interactive, technology-based learning environment for introductory Spanish in adult learning. The setup includes a tablet-based stand-alone educational system that allows for a separate and combined extension by gamification or a social robot. Depending on whether gamification elements are integrated into the learning environment or not, and depending on the type of processing on the tablet with or without a social robot, four conditions are realized. In the control condition (C), participants work on the learning environment using a tablet. In the gamification condition (G), the learning environment is additionally supplemented by gamification elements. The robot condition (R) contains no gamification elements; however, participants learn together with a social robot. Finally, in a combined condition (GR) both elements are brought together, resulting in a gamified learning environment including a social robot.

3.1 Learning material

The learning material is presented in the form of a tablet application. The language Spanish was found to be an appropriate subject for the learning situation, as it is popular among learners as a second foreign language and has been used in robot-enhanced learning environments before (Deublein et al. 2018). Users can acquire knowledge about basic manners, numbers, food and beverages as well as leisure activities (see Fig. 1). The learning environment is structured in four theoretical blocks, covering four different topics, and subsequent exercises. The difficulty of the blocks increases slightly over the course of the learning environment, increasing the amount and complexity of the content. In order to be able to deepen acquired knowledge directly, learners work on five exercises after each theory block, in which Spanish terms must be perceived audibly and assigned in the context of single choice tasks. Once the learner has completed the learning environment, a test on the previously worked on theory blocks consisting of a total of 14 tasks follows immediately afterwards. In the order of Spanish food and beverages, numbers and leisure activities participants are successively tested using single-choice tasks (see Fig. 2).

Fig. 1
figure 1

Menu with the food and beverages covered in the learning environment. Learners can listen to specific foods and beverages in Spanish and German by touching the respective name

Full size image
Fig. 2
figure 2

Exemplary exercise for learners while working on the topic ‘numbers’. Pressing the speaker symbol starts the playback of a sequence of numbers in Spanish that learners have to recognize

Full size image

Before running the learning application, users completed a tutorial of the application’s general functionalities in order to facilitate a successful learning process. Therefore, users received precise operating instructions both in the theoretical part and in the subsequent test as well as assistance within the application in the event of ambiguities. They also had the opportunity to consolidate their acquired knowledge and become familiar with the task structure by means of exercises for the individual lessons. Consequently, learners gained an understanding of the mechanics of the learning environment. These tutorial functions were implemented in all conditions.

After each exercise or test assignment, learners received direct, constructive feedback whereas negative feedback was avoided. If a task had been solved correctly, learners received approving confirmation (e.g. “Great, you got that right. You can be proud of yourself.”); if the answer was wrong, they were provided with motivating and constructive feedback (e.g. “Unfortunately, this is not correct. It would have been answer D. Next time you will surely remember it!”). This approach aims to prevent demotivation of the learner and is based on the assumption that positive feedback has a motivating effect, while negative feedback can undermine motivation (Ryan and Deci 2000b). In general, the design of instructions and tasks within the learning environment is based on the premise of avoiding unnecessary disciplinary formulations, such as ‘must’ or ‘should’. In this way, negative effects on the intrinsic motivation of participants can be avoided (Mekler et al. 2017).

In order to test the feasibility of our learning material and to be able to classify the results and interpret the demands of the learning environment, we conducted a test-run with ten participants who took the knowledge test on the basics of the Spanish language without having worked on the theoretical part of the learning environment before. With a total of 14 attainable points, the participants achieved an average of M = 8.60 points (SD = 2.76). However, we refrained from an increase in difficulty of the knowledge test to maintain the participant’s motivation.

3.2 Integration of the social robot

A social robot acted as a tutor within the learning environment in both conditions involving a social robot (R and GR). The robot ‘Pepper’ from SoftBank Robotics was considered suitable for the learning environment because it has a tablet, which is attached in its chest area (see Fig. 3). Learners can interact comfortably with the robot in a sitting position (see Fig. 4). Due to the need for an additional presentation facility for teaching material and an option for user input (Belpaeme et al. 2018; Han 2010), the ‘Pepper’ robot is ideally suited for this purpose due to its tablet, which was used for the learning scenario in these conditions. The behavior and reactions of the robot have been defined and adapted using the associated program Choregraphe Suite, version 2.5.10.7.

Fig. 3
figure 3

The social robot ‘Pepper’. On the tablet, an exercise of the learning environment on the topic of leisure activities can be seen without implemented gamification elements

Full size image
Fig. 4
figure 4

Working on the learning environment with the social robot ‘Pepper’

Full size image

In its role as a tutor, the robot ‘Pepper’ introduced itself as ‘Paola’ at the beginning of the interaction and guided the learner through the application. Its primary task was to introduce and explain the tasks and to give support if needed. To emphasize the collaborative aspect of the learning environment, the robot's wording focused on the cooperative aspect of the learning environment. Consequently, ‘Pepper’ used numerous collective formulations (‘we’, ‘us’). In addition, it integrated occasional personal statements into the interaction, such as comments on the country Spain or preferred food and beverages. This form of interaction creates a persona around the robot to keep the learner interested in the robot and the learning environment (Kanda and Ishiguro 2005; Pérez-Marín and Pascual-Nieto 2013).

The robot was also set up to display varying behavior to the learner. Although the reaction is always predetermined and follows a fixed sequence, several options were possible at many points within the learning environment. The robot always gave constructive feedback about which contents were not solved correctly and supplements this with an explanation. Voice output was additionally supported by gestures to attract the attention of learners and motivate them to participate (Chang et al. 2010). The robot emphasized the actual spoken word by means of randomly generated or special, complex gestures adapted to specific interaction situations.

3.3 Integration of gamification elements

We implemented several gamification elements within the learning environment in both conditions involving gamification (G and GR). An appropriate background story was embedded to enable learners to relate to real-world situations (Nah et al. 2014) and to provide meaningful learning activities (Darejeh and Salim 2016; O’Donovan et al. 2013; Villagrasa and Duran 2013). The storyline revolved around a fictional character named ‘Sabina’, visiting Germany and unable to speak German, who the learner gets to know better by undertaking some activities together. The individual sections of the story represent the different content blocks of the learning environment, which are explicitly communicated as ‘levels’ in both conditions involving gamification (G and GR). The test that followed the theoretical part of the learning environment uses a point system to visualize the learner’s progress. For each correct answer, the learner received one point. This was communicated in the form of direct feedback after each task. With a total of 14 tasks, a maximum of 14 points can be achieved. The current score was also displayed in the middle of the screen. Additionally, learners could achieve badges as they progress within the learning environment. Altogether, eight badges were rewarded for either completing a specific part in the learning environment or reaching a certain score. Users also had the opportunity to check an overview of all potentially achievable badges at the beginning of the learning environment and immediately before the knowledge test. Because both the points and badges have no material value and no connection to physical rewards, no negative effects are to be expected with regard to intrinsic motivation (Deci et al. 2016).

Based on an overview of badges, the user obtained clear goals for the entire learning environment, which can also be continuously differentiated and gradually achieved by smaller sub-goals, such as the levels of the learning environment. The clear specification of tasks with regard to a desired goal makes it easier for the learner to use the application (Darejeh and Salim 2016). Additionally, users received a certificate of participation displaying their name, score and all obtained badges after completing the learning environment as a form of a physical reward for their effort. As such a certificate represents a form of unpredictable material reward, no negative effects regarding intrinsic motivation are to be expected (Deci et al. 2016).

We refrained from the possibility of unlocking new content in order not to overburden users with an overload of content. Additionally, the content and formal structure of the learning environment developed in the context of the present study are designed to be worked on by only one user at a time. With additional accompaniment by a robot, it is necessary to limit the number of active users to one person for a standardized approach. We therefore omit the integration of social engagement loops.

3.4 Combined integration

To combine the social robot and gamification elements, both aspects described in Sects. 3.2 and 3.3 were integrated simultaneously in the learning environment. In the combined condition, the robot also addressed some of the gamification elements verbally, e.g. emphasizing the achievement of a new badge or informing the learner about their current score. Additionally, the robot was integrated into the background story as a character considering itself as part of the plot and actively referring to itself when giving feedback to the learner. On completion of the learning environment, the robot acknowledged the learner’s performance by highlighting the final score.

4 User study

We conducted a user study to examine the integration of a social robot and various gamification elements into a technology-enhanced learning environment both separately and for a combined effect, using a 2 × 2 between-subject design. Additionally, long-term effects are assessed by a repeated assessment for all conditions.

4.1 Participants

A total of N = 130 participants took part in the study. Four persons had to be excluded due to technical problems, resulting in a total number of participants of N = 126. Participants were rewarded with partial course credit for participation. The age of the participants ranged from 18 to 35 years, with an average age of M = 20.40 (SD = 2.27). 83.33% of the participants were female (n = 105), 16.67% male (n = 21). Thirty-four participants (54.84%) in one of the two conditions with a social robot (R and GR) already knew the robot ‘Pepper’ or had already interacted with it. 53.97% of the participants (n = 68) had no knowledge of the Spanish language. Forty participants (31.75%), on the other hand, already had little prior knowledge and 13 (10.32%) had basic knowledge of the Spanish language. Only 3.97% stated that they had above average knowledge of the Spanish language (n = 5). Due to the small number of participants with above average prior knowledge, no participants were excluded from the study because no effect on the results was discernible. The participants were randomly assigned to one of the four conditions (C, G, R and GR), resulting in NC = 34, NG = 30, NR = 31 and NGR = 31.

4.2 Dependent variables

Engagement was measured by the User Engagement Scale Short Form (UESSF; O’Brien et al. 2018). The UESSF is a valid and reliable scale consisting of four subscales, each with three items on ‘Perceived Usability’, ‘Aesthetic Appeal’, ‘Reward’ and ‘Focused Attention’. The scale measures the user’s commitment when interacting with a digital system using these four factors on a five-point Likert scale. The dimension ‘Reward’ was excluded due to its low relevance for the present study. According to O’Brien et al. (2018), this procedure is unproblematic.

The participant’s motivation was assessed by the Situational Motivation Scale (SIMS; Guay et al. 2000). The SIMS measures the factors ‘Intrinsic Motivation’, ‘Identified Regulation’, ‘External Regulation’ and ‘Amotivation’ on the basis of 16 items on a seven-point Likert scale. These are based on the theoretical construct of the theory of self-determination (Guay et al. 2000). The scale covers the situational motivation of a person and focuses on the underlying intentions of their behavior. SIMS and UESSF were translated into German. Furthermore, the wording was adapted to fit the learning context.

In order to measure learning success, the number of points of the participants in the Spanish test was measured at two points in time. The first measurement was taken as the knowledge test of the learning environment (t1). The second measurement took place in an interval between 24 and 48 h after the first measurement (t2). The time period was considered appropriate to measure how much knowledge learners retained after one or two days, because motivation is a critical influencing factor for long-term memory (e.g. Murayama et al. 2013; Naceur and Schiefele 2005).

Additionally, the perceived intelligence of the social robot Pepper was assessed using the ‘Godspeed’-questionnaire as part of the ‘Godspeed Questionnaire Series’ (Bartneck et al. 2009) to check whether effective communication with the robot had been possible (Meghdari et al. 2013). The questionnaire consists of five items measuring the perceived intelligence of a robot on a five-point Likert scale. Finally, a query of the obvious implemented gamification elements regarding their perception and appeal to the learners was performed. Participants were asked to report whether they liked the background story, badges and score system on a five-point Likert scale.

4.3 Procedure

The study was conducted with one participant at a time in a separate room in the presence of the instructor. In the non-robot conditions (G and C), the participant was seated at a table with a tablet, while participants who worked on the learning environment together with the social robot (R and GR) sat down facing the social robot to ensure optimal interaction conditions. After a short briefing by the experimenter on the procedure of the study and clarification of potential questions from the participants, they were asked to start working on the learning environment. In order to avoid a feeling of observation for the participants, the experimenter remained behind a partition wall for the duration of the experiment. No further interaction by the experimenter occurred until the participants finished the learning scenario. After completion of the interaction, users of the gamified application received a certificate.

Afterwards, all participants were directed to a separate computer with the request to fill out SIMS, UESSF and demographic data. Participants in the conditions R and GR additionally completed the ‘Godspeed’-questionnaire whereas those who had been part of a condition involving gamification (G and GR) were asked to fill out a short survey about the gamification elements. The duration of the entire examination averaged 40 min. Twenty-four hours after the survey, participants received another questionnaire, containing the same knowledge test as in the learning environment. This additional test contained the same questions in identical order as the knowledge test in the experimental setting, yet participants were not able to repeat the learning environment in advance.

5 Results

Quantitative analysis was performed using IBM SPSS Statistics, version 25, with an alpha level of α = 0.05 for all statistical tests.

5.1 Motivation and engagement

To test the influence of the social robot (present or absent) and the gamification elements (present or absent) on the intrinsic motivation and engagement of the participants as predicted by H1 through H4, a multifactorial, multivariate analysis of variance was used. The equality of the covariance matrices can be assumed according to the Box test (p = 0.219).

Using Wilk’s statistics, we found a significant main effect on engagement and intrinsic motivation for the integration of a social robot into the learning environment (Λ = 0.95, F(2, 121) = 3.31, p = 0.040). A subsequent univariate analysis of variance showed a significant effect of the robot on engagement (F(1, 122) = 6.48, p = 0.012, η2 = 0.05), with significantly lower engagement with a robot (M = 3.88, SD = 0.59) compared to the conditions with only the tablet (M = 4.12, SD = 0.48), but no significant effect on intrinsic motivation (F(1, 122) = 3.01, p = 0.085), contrary to as predicted by H1 and H2.

For the implementation of gamification elements, no significant main effect on engagement and intrinsic motivation as predicted by H3 and H4 was found (Λ = 0.97, F(2, 121) = 1.72, p = 0.183). Contrary to H7 and H8 for the combined approach, no significant interaction of gamification and social robot on engagement and intrinsic motivation was found, Λ = 0.98, F(2, 121) = 1.17, p = 0.315.

5.2 Learning success

In order to test the effect of the integration of gamification elements and the social robot on learning success, a 2 (G, noG) × 2 (R, noR) × 2 (t1, t2) mixed ANOVA showed a significant main effect of time on the score (F(1, 122) = 5.60, p = 0.020, η2 = 0.04). Participants significantly improved their score from the first measurement point (M = 12.74, SD = 1.39) to the second measurement point (M = 13.01, SD = 1.31), see Table 1 for a detailed overview of the scores in all conditions.

Table 1 Means and standard deviations for test scores at first measurement point (t1) and second measurement point (t2) (scale 1–14)
Full size table

Contrary to H5, the integration of a social robot into the learning environment had no significant effect (F(1, 122) = 0.18, p = 0.668). The implementation of gamification elements did also not show a significant effect on test performance (F(1, 122) = 0.11, p = 0.741) contrary to H6. The three-way interaction as addressed by RQ1 was also not significant (F(1, 122) = 0.11, p = 0.741).

5.3 Additional measures

The social robot ‘Pepper’ received a mean score of M = 4.17 (SD = 0.65) on the intelligence scale of the Godspeed questionnaire by the learners who worked together with a robot in the learning environment (condition R and GR; n = 62). A positive correlation of the perceived intelligence of the robot with the engagement of the learners (r = 0.62, p < 0.001) and their motivation (r = 0.32, p = 0.012) was observed.

The attitude of the participants towards the gamification elements correlated positively with the engagement (r = 0.56, p < 0.001), but the motivation of the learners did not (r = 0.19, p = 0.142).

The implemented gamification elements were generally well received. Both the badges (M = 4.26, SD = 0.81) and the point system (M = 4.61, SD = 0.64) achieved a high rating on the five-point Likert scale. The background story was rated slightly worse than the other two elements with M = 3.43 (SD = 1.07).

6 Discussion and future work

The present study demonstrates an approach for the integration of gamification elements, selected based on theory, and a social robot in an interactive, technology-based learning environment for adult learning, aiming to investigate the effect on engagement, motivation and the performance of learners. We did not find higher engagement neither for the condition with the social robot (H1), the gamified condition (H3) nor for the combined integration of both elements (H8). In addition, we were not able to demonstrate higher intrinsic motivation when working on a learning environment with a social robot (H2), with gamification elements (H4) or with the combined integration of both elements (H7). We thus reject hypotheses 1 through 4 as well as hypotheses 7 and 8.

The rather high engagement and intrinsic motivation of the participants across all conditions is worthy of note. This is particularly relevant in the control condition without a social robot and implemented gamification elements. Consequently, it can be assumed that the content itself induced engagement and motivation in the learners. This could be due to the great popularity of Spanish as a second foreign language. Additionally, participants received positive feedback on their actions in all conditions, either written or verbally expressed by the robot, which may have had an impact on the general attitude of users towards the learning environment, especially in terms of motivation (Ryan and Deci 2000b). It is, therefore, unclear to what extent the results are related to the integration of the social robot and the gamification elements.

Our results also suggest that the learning environment in all four tested conditions might have helped in the acquisition and internalization of the Spanish language regarding the high number of points in both the test and the post-test. In general, participants significantly improved their score on the post-test compared to their score on the test immediately after completing the theoretical learning part. With regard to the descriptive data, a nearly identical increase in the number of points achieved is discernible for all conditions. This might be due to the general workflow of the learning environment. Participants could already complete small exercises on the respective topic in the theoretical part of the learning environment whose format was similar to the final test. The content of the test was also comparable. Accordingly, the design of the exercises may have anticipated the function of the actual test. Another reason for this improvement across all conditions might be the opportunity for participants to learn from their mistakes in the first knowledge test and the concomitant positive feedback. This constructive feedback in the first test might have led to an increased performance in the post-test, as learners benefited from wrong answers and deepened their knowledge.

However, we were not able to demonstrate higher learning success for the integration of neither a social robot nor the gamification elements compared to the control condition, rejecting H5 and H6. We also did not observe a beneficial effect of the combined integration of gamification and a social robot on learning success as addressed by our research question 1. The generally high performance in test and post-test might be an indication of a ceiling effect with the potential problem to allow no sufficient differentiation between individual learners (Garin 2014). These effects might be due to previous knowledge of the participants and the overall low difficulty of the learning content. Because a major part of the learners had at least little knowledge of the Spanish language, it can be assumed that the learning material may not have been sophisticated enough to show actual differences between the conditions. Participants who comparatively passed the knowledge test without working on the theoretical part of the learning environment achieved on average more than half of the maximum achievable score. As already explained in Sect. 3.1, we refrained from increasing the difficulty of the knowledge test to prevent a loss of motivation in participants.

Participants’ feedback was throughout positive, although some perceived the tasks as too easy, especially with existing, albeit limited, prior knowledge of Spanish or languages from the same language family, which also supports a ceiling effect. Regardless of the condition, many participants emphasized that they liked the learning environment and had fun. Thus, it can be concluded that the learning environment in all four forms was interesting and entertaining, but that the content may have been too easy and that the gamification elements and the integrated robot did not result in any measurable benefit for the learning environment. It is important to keep in mind that technology-enhanced learning can generally support and promote self-regulated learning (Fahnoe and Mishra 2013; Rashid and Asghar 2016). Because all four conditions included a technology-based learning environment, this may explain the lack of differences in the learning effect between groups. The learning environment has helped learners to acquire and retain knowledge of the Spanish language, but it remains open whether and to what extent the gamification elements and the social robot ‘Pepper’ have contributed to this.

6.1 Reflecting social robots in the learning context

The integration of a social robot resulted in a significant difference regarding the engagement of the participants, which, however, was diametrically opposed to the assumed effect. Participants who worked on the learning environment without the robot showed significantly higher engagement than those who learned together with the robot. In the analysis of engagement, the teacher–learner relationship is a relevant factor (Sagayadevan and Jeyaraj 2012). Consequently, a positive relationship should also have a positive effect on learner engagement. It is possible that the social robot ‘Pepper’ had not been able to establish a sufficiently positive relationship with all learners, which in turn could have a potential impact on learner engagement. This is also supported by the fact that participants gave little positive feedback on the integration of the robot, but primarily criticized technical problems or the unnatural pronunciation. In addition, the non-verbal behavior of the robot might have been distracting. In particular, intense and strenuous social interaction can have a negative impact on the learning experience (Belpaeme et al. 2018). We have taken care to minimize this risk by the utilized gestures and avoidance of excessive movement, yet it cannot be completely ruled out.

The robot may induce engagement, but to a smaller extent than in the tablet-supported conditions, as indicated by the above average engagement values in all conditions. A positive correlation between the perceived intelligence of the robot and the engagement of the learners supports this interpretation. Because the robot was perceived as intelligent in the context of the present study, effective communication between robot and learner can be assumed (Meghdari et al. 2013), although no strong effect on the engagement was found.

Also, integrating the social robot ‘Pepper’ into the learning environment did not yield a significant advantage in terms of the learners’ motivation. With regard to the motivational aspect, participants across all groups showed average to slightly above average motivation. Thus, the integration of a social robot did not have the desired effect. Framed in the context of self-determination theory, the robot’s confirmative verbal feedback did not seem to positively affect the participants’ experience of competence in comparison to the control condition. This might be due to the content of the constructive feedback that participants received in all conditions rather than its modality, which would also be in line with self-determination theory and the reason why participants in the control condition also experienced motivational relevant competence. Furthermore, the possibility of social interaction with the robot ‘Pepper’ may not have been sufficient to meet the need for relatedness, so there is no additional positive effect on motivation. Finally, the robot was only able to respond to user input and the user’s emotional state individually to a limited extent. A more active involvement of the learner could potentially better satisfy the need for autonomy.

The robot represents a new form of learning support. Consequently, it must be taken into account that potential positive effects of a social robot on engagement and motivation could primarily be attributed to the novelty effect (Deublein et al. 2018; van den Berghe et al. 2019). Because the target group in the present study was partly familiar with the robot before, this could be the reason why we did not find significant differences between the conditions where learners worked on the learning environment with and without a social robot. Nevertheless, Alemi et al. (2015) were able to demonstrate in a long-term study that a social robot can motivate and interest learners in the long run. However, this requires the carefully designed interaction and subsequent integration of the robot into the learning context. Further research addressing comprehensive design approaches such as long-term interactions with a social robot in a learning environment is clearly needed (van den Berghe et al. 2019; Zhong and Xia 2020).

Due to the absence of desired effects, it is questionable to what extent the social robot was suitable for the implemented learning scenario. Even though robots have been shown to improve learning experiences (Ramachandran et al. 2018) and participants in the present study even reported the robot as an interesting and fun learning partner, there was no discernible benefit. Therefore, it might be important to reflect whether the integration of a social robot is adequate in basic higher education learning scenarios, especially because social robots do not necessarily offer a direct enrichment for the learning context without a respectively high-quality learning content. Appropriate development and preparation of the learning environment requires knowledge about the functioning and capabilities of the robot (Chang et al. 2010), meaning they should not be developed separately.

With a view to the implementation context and target group, it should be noted that many studies demonstrating positive effects of the integration of robots in the learning context had children as their primary target group (van den Berghe et al. 2019; Belpaeme et al. 2018). However, the present study has been conducted in the context of adult learning and did not indicate that the anticipated beneficial effects from research in children are tenable regarding the different target group of the study. Still, Donnermann et al. (2020) suggest the potential of integrating social robots in higher education, highlighting the positive attitude of students towards robot-supported learning environments that might affect learning outcomes in a positive way.

6.2 Reflecting gamification in the learning context

For the implemented gamification elements, there were no significant differences in the engagement or motivation of the learners. The integration of gamification elements did also not bring any advantage in terms of learning success compared to a non-gamified learning environment. Nevertheless, it is noticeable that learners achieved above average scores for engagement and motivation as well as for the number of points in the test and post-test. With regard to the obvious game elements (badges, points and background story), the participants’ interest is above average. This is also supported by the positive correlation of the participants’ assessment of the three gamification elements with engagement.

In the context of the present work, the implemented gamification elements were tutorial functions, levels, clearly defined content goals and feedback. In addition, badges, a background story and a point system were integrated. The respective elements were selected on the basis of a theoretical analysis and considered suitable (cf. Dichev et al. 2015; Iosup and Epema 2014; Kapp 2012). However, in the application area of gamification, the same game elements are often used and examined (Nacke and Deterding 2017), often with questionable success as noted by Seaborn and Fels (2015); differentiation and expansion could possibly bring advantages here.

Although the implemented gamification elements were perceived positively, they did not bring the expected improvement in terms of learner engagement or motivation. Therefore, the extent to which the developed learning environment actually benefits from the integration of not adequately integrated gamification elements should be questioned (Aparicio et al. 2012). The duration of the learning environment could play a relevant role in this respect. The present learning environment took no longer than 45 min to complete and did not require participants to self-motivate to continue working on it in contrast to an applied setting. The relevance of the gamification elements might, therefore, increase if the application is used over a longer period of time.

In the context of self-determination theory, gamification elements can be useful to enhance the user’s experienced competence as they serve as a form of constructive feedback. Since participants in the control condition also received continuous feedback on their input, this could have satisfied their need for competence, which is why there is no distinct motivational difference between the groups. Embedding a background story provided a context for the learning environment, which benefits the experience of autonomy by the user. Nevertheless, the learner’s freedom of choice is also critical for their need for autonomy and could not be fulfilled due to the fixed task structure. In summary, the integration of gamification might positively influence motivation, yet a meaningful and comprehensible technology-supported learning environment, as introduced in the present study, can as well provide a motivating framework for learning. These findings are consistent with the generally mixed results regarding the integration of gamification elements in education. While gamification might work in some approaches (e.g. Hakulinen et al. 2013; Mekler et al. 2013; Sailer et al. 2017; Schaper et al. 2021), it might also have negative effects, such as reduced performance and other undesired effects in the learning context (Toda et al. 2018).

6.3 Combining a social robot and gamification in one approach

With regard to the combination of the social robot and gamification elements, no significant effect could be found, neither on the engagement of the learners nor on their motivational state. The combination of both factors showed no positive, but also no diametrical effect on learning success. Unlike the results found by Donnermann et al. (2021) where the combined integration seemed to have a detrimental effect on engagement, no disturbing influence of the combined factors on the learning environment and the users can be assumed in the context of the present study.

It can be concluded that users neither benefit nor suffer any disadvantages from a combination of a social robot and gamification elements in the learning environment in terms of engagement, motivation and learning gains. Though the combined integration of both aspects in the context of adult learning has provided new insights into an innovative way of enriching a technology-enhanced learning environment, it remains unclear whether the additional conceptual effort is worthwhile. The appropriate selection and comprehensive integration of both a social robot and adequate gamification elements is time-consuming and requires an elaborated concept, yet seems to bring no significant benefits for learners and their learning process. Integrating a social robot in a learning environment requires in-depth knowledge of the capabilities and operation of the robot (Chang et al. 2010), while gamifying an application involves a lot of time and effort, but depending on the subject matter and content, the application is often not reusable (O’Donovan et al. 2013). Combing a social robot and a gamified application requires even more effort, e.g. verbalization of achievements by the social robot, still seems to yield no additional benefit. Further research and alternative approaches could provide new insights.

6.4 Implications for future work

The present study offers valuable implications for the design and development of engaging and motivating learning content. Future work could improve the learning environment by allowing for an adaptive and individually challenging learning content while retaining the engaging and highly motivating concept. The design of the learning content should also provide users with previous knowledge a chance for improvement. This requirement can be met by adjusting the difficulty and scope of the learning material and adding new content. With regard to the general design of a learning environment, Deci et al. (2016) recommend the adoption of the learner’s perspective in order to effectively promote engagement and motivation. Learning activities should be developed and designed from the perspective of the user in order to provide more freedom of choice and appropriate challenge. The mere integration of scoring systems and badges might also not be appropriate in every context. Depending on the learning content and environment, the implemented gamification elements should be varied accordingly (Kuo and Chuang 2016).

In the general context of adult education, our study demonstrated that the integration of a robot and/or gamification elements seems to be neither beneficial nor disadvantageous. However, because the appropriate implementation of a social robot in a learning environment is often time- and resource consuming, it should be carefully considered for each learning scenario whether the robot might benefit the learner. A sensibly constructed and motivating E-learning environment may possibly be sufficient for many learning scenarios. As already mentioned in Sect. 2.3.2, a robot often entails high costs in acquisition and maintenance, which further limits its accessibility in the learning context.

Our study was conducted before the Covid19 pandemic, which additionally demonstrated the importance of the social role of the classroom and peers. As proposed by Vygotsky (1981), in the context of social constructivism and within the ZPD, assuming that learning occurs when interacting and cooperating with peers that are more capable than the learner (Vygotsky 1981; Eun 2019), a social robot that incorporates the role of a more knowledgeable other in the learning context might possibly provide a low-risk alternative. Depending on the context, it could embody the role of a peer or tutor and hence incorporate social aspects in the learning process, even in a pandemic situation where schools are closed and traditional instruction methods are often not feasible. Still, it is notable that we do not primarily aim to deploy a social robot in homeschooling settings, but focus on potential beneficial effects of integrating robots as a social component in a self-directed learning process or a setting where no human tutor is available, which makes learning at home a relevant use case. Further, the integration of a social robot in the learning context could allow educators to adapt their teaching process in a flexible way, e.g. by delegating tasks und thus freeing up time for individual support during lessons.

Technology-supported learning through the use of tablets or corresponding technical devices has become increasingly present in the learning context in recent years and will continue to gain relevance in the future. Alemi et al. (2015) postulate a comparable development for social robots. Similar to the development of the computer social robots will sooner or later find their way into the personal everyday life of many individuals and thus also into education, just as gamification elements are now widespread and used in many areas. Our results demonstrate that this area still requires more research, especially with regard to integrating social robots in the learning context as efficient as possible. Therefore, future work should further investigate the interaction of these aspects on the learning effect, engagement and motivation of the learner in order to increase the effectiveness of learning environments with technological development in the future.

7 Conclusion

The present work investigates a social robot and gamification in the context of a learning environment for adult learning as well as their impact on engagement, motivation and the learning performance. Four conditions were implemented to investigate both the separate effects of the social robot and gamification as well as their interaction. Our study demonstrates an exemplary and controlled integration of the social robot ‘Pepper’ together with theoretically framed gamification elements reflected within the scope of a learning context for the acquisition of the Spanish language. For the first time, to our knowledge, the robot ‘Pepper’ was integrated and evaluated in a learning environment in combination with gamification elements. The implemented game elements were extracted and selected based on a comprehensive theoretical analysis. Thus, the present study design can be used to derive valuable implications and design hints for future investigations in this context. Similar approaches might take a comprehensive view of potential advantages and disadvantages while examining social robots and gamification in the learning context.

The present study shows that the integration of a social robot and gamification elements should be carried out deliberately and does not necessarily lead to higher motivation and engagement in learners or greater learning performance. Our exemplary and theory-based implementation of the combined integration of a social robot and gamification elements in adult learning aimed for an increase of engagement, motivation and learning success and thus provides a basis for future research. Still, our approach demonstrates that a comprehensive integration of both elements in a learning environment seems not to be sufficient, especially in higher education, thus requiring different approaches and further development in this area.