Affective interaction: How emotional agents affect users

https://doi.org/10.1016/j.ijhcs.2009.05.001Get rights and content

Abstract

Embodied agents have received large amounts of interest in recent years. They are often equipped with the ability to express emotion, but without understanding the impact this can have on the user. Given the amount of research studies that are utilising agent technology with affective capabilities, now is an important time to review the influence of synthetic agent emotion on user attitudes, perceptions and behaviour. We therefore present a structured overview of the research into emotional simulation in agents, providing a summary of the main studies, re-formulating appropriate results in terms of the emotional effects demonstrated, and an in-depth analysis illustrating the similarities and inconsistencies between different experiments across a variety of different domains. We highlight important lessons, future areas for research, and provide a set of guidelines for conducting further research.

Introduction

An increasing number of researchers are examining the potential for embodied agents (screen-based anthropomorphic entities) to enhance interactions with computers. Experiments with embodied agents have been conducted in a number of different fields from health (Bickmore and Picard, 2005; Mazzotta and de Rosis, 2006), learning (Burleson and Picard, 2007; Maldonado et al., 2005) and commerce (Gong, 2007), to therapy (Grolleman et al., 2006), video games (Isbister, 2006), and military systems (Gratch and Marsella, 2004). Such entities are also widely used in massive multiplayer online games (MMOGs) like Second Life and World of Warcraft (e.g. computer-controlled enemies). Despite the large amount of interest in this research space, and the vast number of empirical investigations, the potential for embodied agents to enhance human–computer interaction (HCI) remains unclear. Dehn and Van Mulken's (2000) widely cited review of animated interface agents highlighted a number of conflicting findings regarding the subjective, behavioural, and performance outcomes of interacting with such entities. A more recent meta-analysis conducted by Yee et al. (2007) found that the use of an embodied agent in an interface produced more positive social interactions. However, there remain varying opinions about the use of such agents in interfaces with arguments both for (e.g. more natural interaction that enhances engagement with a system) and against their use (e.g. distraction when completing important tasks).

One of the primary goals for many embodied agent researchers is to create agents capable of having natural and effective interactions with users that can produce some desirable or beneficial outcome. This has resulted in a number of designers exploring how to equip such entities with the ability to express synthetic human emotions through textual content, speech (synthetic and/or recorded), body language, and facial expressions. This increased focus on emotion is motivated by relatively recent findings related to the important role that emotions play in our everyday lives. For example, research has found that emotions play essential roles in attention (Evans, 2001), memory (Christianson and Loftus, 1991), judgement and decision making (Baron, 1987), creative problem solving (Isen et al., 1987), and persuasion (Mackie and Worth, 1989; Cialdini, 2003). Emotional expression is also one of the primary ways in which we can communicate to others how we are feeling (Ekman, 2004), which is of great importance in building and maintaining social relationships with others. Therefore, for embodied agents to have productive and engaging interactions with users, many researchers believe it is essential that they have “emotional engines” that help drive their behaviour and actions.

Despite the increased focus on emotion in embodied agent research, it is still largely unclear how people respond to simulated displays of agent emotion. For example, how do we respond to synthetic displays of happiness, sadness, frustration and fear? Can we catch emotions from computers? What do we think of agents that act “caring” towards us when we are feeling down—does it help improve our mood, or do we find it tedious and annoying? These types of questions are particularly important given the number of research studies and commercial applications that are now utilising emotionally expressive agents to interact with users. Given the important role that emotions play in human–human interaction, it is essential that we understand in detail the impact they have in user–agent interactions. A number of recent studies have been examining the effects of simulated emotion on users, but the results from many of these empirical investigations have produced conflicting results. It can become problematic when attempting to analyse and compare the results of different emotion simulation studies as each study often measures different aspects of the interaction and a number of different experimental approaches can be used. Dehn and Van Mulken (2000) highlight in their review of animated agents the number of different experimental approaches that can be used when running experiments with such entities:

  • The representation of an agent: Different forms of embodiment in agents are often used (e.g. 2D/3D faces, virtual humans, caricatures, photographic images of humans).

  • Comparison made: Some studies compare agents which convey emotional expressions against those that do not. Others keep the inclusion of emotional expressions constant and change the embodiment of the agent.

  • Measurement of effects: The effects of emotion simulation in computer interfaces can be measured in a number of ways such as obtaining user perceptions through questionnaires and interviews, observing user behaviour, or analysing user performance data.

  • Form of interaction: Different studies often ask the subject to interact with the computer in a particular way. For instance, subjects may be asked to complete a well-defined goal or simply to pay attention to information that is provided by the computer.

In addition to these generic issues with embodied agent studies, there are also others that are unique to the comparison of emotion simulation studies:

  • Types of emotions expressed: Different agents express different types of emotion. Some only express basic emotions such as happiness and sadness, while others display a variety of basic and more cognitive emotions such as concern, satisfaction, excitement and empathy. Basic emotions are those that are innate and shared across all cultures—these emotions typically have specific facial and vocal characteristics associated with them that make it easier to detect and recognise them (Ekman, 2004). Cognitive emotions, however, are those that require extra levels of processing in the cortex of the brain (Griffiths, 1997). Similar to basic emotions, cognitive emotions tend to be universal, although there are cultural variations in how they are expressed and experienced. Cognitive emotions also take longer to develop and fade away in comparison to basic emotions (e.g. romantic love (a cognitive emotional state) normally develops over weeks and months whilst surprise (a basic emotion) is a fast reactive response to some stimuli).

  • Representation of emotional expression: The way in which the emotions are expressed differs between agents. Some may primarily use textual content to express emotion, while others may use a combination of animated facial expressions, gestures and other body language coupled with recorded human speech.

  • Method for expression of emotion: Some agents express emotions one at a time (e.g. an agent may jump from being happy to being sad with no intermediary change in emotion) while other agents move smoothly between emotions dynamically displaying intermediary states (e.g. happy–warm–neutral–sad).

These differences make it increasingly difficult to compare different studies and thus to understand the impact that simulated agent emotion has on computer users. This paper provides a structured overview of emotion simulation studies to help understand more clearly the impact that emotion in embodied agents has on users. We review, collate and structure the previous work on emotional agents into a coherent framework and highlight the differences in experimental approaches (particularly when studies provide conflicting results). Our aim is to understand more fully the effects of simulated emotion and the reasons why different studies report conflicting results. We start by providing an overview of the framework we have used to categorise the studies and highlight how we have grouped the studies based on the domain within which their agent was deployed. Then, for each domain, we provide an overview of all the studies that have examined the use of embodied agents in that space, along with discussion of the main findings. We conclude with a number of suggestions for future research and general guidelines on conducting empirical investigations into the effects of emotionally expressive embodied agents.

Section snippets

Overview of the framework

One of the primary suggestions that Dehn and Van Mulken (2000) make about future research with embodied agents is that a fine-grained approach should be taken to understand the impact of embodied agents more fully:

… the simple question as to whether an animated interface agent improves human–computer interaction does not appear to be the appropriate question to ask. Rather, the question to ask is: what kind of animated agent used in what kind of domain influence what aspects of the user's

Analysis of emotion simulation studies

A comprehensive review of the literature provided the studies included in this review; electronic tools such as Google Scholar and Citeseer were complemented by manual searches of journals and conference proceedings related to human–computer interaction, psychology, and virtual reality, to locate any recently published articles that may have not been indexed by these search tools. The first phase of the search involved collecting together all articles that appeared to provide details about an

Suggestions for future research

It is clear from the analysis that a number of different areas related to emotional embodied agents require further investigation. In this section, we provide a number of suggestions of where we feel future research studies need to concentrate in order to deepen our understanding of the role of simulated emotion in human–agent interactions.

  • Longitudinal studies: One area where future studies need to concentrate is that of conducting longitudinal studies with emotional embodied agents.

General guidelines

Based on the discussion throughout this review, we have compiled the following guidelines for conducting research into the effects of simulated emotion.

  • Always validate emotional expressions: One major issue with many of these studies is that they do not validate the emotional expressions of their agent prior to conducting the experiment. This is unsurprising in studies where emotion is not the main focus of the experiment, but the majority of studies with a high to medium relevance, where the

Conclusion

The studies in this review have started to investigate the impact of emotion in embodied agents on user's attitudes and behaviour. The results are often inconclusive and contradictory though several studies have highlighted the potential for simulated emotion to both enhance and hinder interactions with agents. In particular, Maldonado et al. (2005) found that an emotional co-learning agent generally enhanced an interaction, while other studies in the education and learning domain generally

References (45)

  • R.A. Baron

    Interviewer's mood and reaction to job applicants

    Journal of Applied Social Psychology

    (1987)
  • Bartneck, C., 2002. eMuu—an embodied emotional character for the ambient intelligent home. Ph.D. Thesis, Eindhoven...
  • Bartneck, C., 2003. Interacting with an embodied emotional character. In: Proceedings of the 2003 International...
  • Bickmore, T., 2003. Relational agents: effecting change through human–computer relationships. Ph.D. Thesis, Department...
  • T. Bickmore et al.

    Establishing and maintaining long-term human–computer relationships

    ACM Transactions on Computer–Human Interaction (TOCHI)

    (2005)
  • W. Burleson et al.

    Gender-specific approaches to developing emotionally intelligent learning companions

    Intelligent Systems

    (2007)
  • J. Cassell et al.

    The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents

    Applied Artificial Intelligence

    (1999)
  • S.A. Christianson et al.

    Remembering emotional events: the fate of detailed information

    Cognition and Emotion

    (1991)
  • R. Cialdini

    Influence: Science and Practice

    (2003)
  • P. Ekman

    Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life

    (2004)
  • P. Ekman et al.

    Facial Action Coding System

    (1977)
  • D. Evans

    Emotion: The Science of Sentiment

    (2001)
  • Cited by (189)

    • How gender is intertwined with robots and affective technologies: A short review

      2022, Sex and Gender Bias in Technology and Artificial Intelligence: Biomedicine and Healthcare Applications
    • Brand co-creation in tourism industry: The role of guide-tourist interaction

      2021, Journal of Hospitality and Tourism Management
      Citation Excerpt :

      Good service providers will express emotions with care, warmth, and likeability at appropriate time. In return, these behaviors eventually influence customers' subjective experience perception (Beale & Creed, 2009; Teng & Tsai, 2020). Reichenberger (2014) suggested that positively perceived social interactions are essential components of tours, which contribute to a satisfying tourist experience.

    View all citing articles on Scopus
    View full text