Deep learning for affective computing: Text-based emotion recognition in decision support

Highlights

• Affective computing infers the emotional state of humans from text.

• We propose the use of deep learning: recurrent neural networks & transfer learning.

• This yields considerable improvements in predictive accuracy.

• Holistic evaluation and implications for decision support are derived.

Abstract

Emotions widely affect human decision-making. This fact is taken into account by affective computing with the goal of tailoring decision support to the emotional states of individuals. However, the accurate recognition of emotions within narrative documents presents a challenging undertaking due to the complexity and ambiguity of language. Performance improvements can be achieved through deep learning; yet, as demonstrated in this paper, the specific nature of this task requires the customization of recurrent neural networks with regard to bidirectional processing, dropout layers as a means of regularization, and weighted loss functions. In addition, we propose sent2affect, a tailored form of transfer learning for affective computing: here the network is pre-trained for a different task (i.e. sentiment analysis), while the output layer is subsequently tuned to the task of emotion recognition. The resulting performance is evaluated in a holistic setting across 6 benchmark datasets, where we find that both recurrent neural networks and transfer learning consistently outperform traditional machine learning. Altogether, the findings have considerable implications for the use of affective computing.

Introduction

Emotions drive the ubiquitous decision-making of humans in their everyday lives [33,57,73]. Furthermore, emotional states can implicitly affect human communication, attention, and the personal ability to memorize information [20,22]. While the recognition and interpretation of emotional states often come naturally to humans, these tasks pose severe challenges to computational routines (e.g., [64,81]). As such, the term affective computing refers to techniques for detecting, recognizing, and predicting human emotions (e.g., joy, anger, sadness, trust, surprise, anticipation) with the goal of adapting computational systems to these states [62]. The resulting computer systems are not only capable of exhibiting empathy [61] but can also provide decision support tailored to the emotional state of individuals.

Emotional information is conveyed through a multiplicity of physical and physiological characteristics. Examples of such indicators include vital signs such as heart rate, muscle activity or sweat production on the surface of the skin (e.g., [46,80]). A different stream of research tries to infer emotions from the content and its mode of communication. These approaches to affective computing are primarily categorized by the modality of the message, i.e., whether it takes the form of speech, gesture, or written information [11]. In this terminology, affective computing can comprise both unimodal and multimodal analyses. For instance, videos allow for the recognition of facial expressions and vocal tone [15,26,74].

The focus of this work is on the unimodal analysis of written materials in English. This choice reflects the prominence of textual materials as a widespread basis for decision-making [35]. Illustrative examples are as follows (a detailed review is given later in Section 5.3). For instance, the use of affective language as a proxy for emotional closeness can be used to measure the strength of interpersonal ties in social networks [49]. Similarly, marketing utilizes the recognition of emotional states in order to predict the purchase intentions of customers [6], satisfaction with services [32], and even to measure the overall brand reputation [2]. In a related context, decision support can leverage affective signals in financial materials in order to suggest trading decisions [30] or forecast the economic climate [58]. Furthermore, affect can also improve processes and decision-making in the provision of healthcare [76] or education [70].

Previous research on affective computing has merely utilized methods from traditional machine learning, while recent advances from the field of deep learning – namely, recurrent neural networks and transfer learning – have been widely overlooked. However, their use promises further improvements. In fact, techniques from deep learning have become prominent in various decision support activities involving sequential data (e.g., [27]) and especially linguistic materials (e.g., [41,47]), where deep learning was able to enhance the performance when deriving decisions from unstructured data. One of the inherent advantages of deep learning is that it can successfully model highly non-linear relationships.

This work draws upon existing solution techniques from the realm of deep learning [41] and applies them to a problem domain different from that of our research objective. First and foremost, we extend existing techniques from the discipline of deep learning to the task of text-based emotion recognition in order to expand the body of knowledge. Following Kraus & Feuerriegel [41], we also utilize long short-term memory networks (LSTMs) that can make predictions based on running texts of varying lengths. However, affective computing differs substantially from related tasks due to the high number of often imbalanced target labels. Thus, this task requires both customized network architectures and procedures. Hence, its applicability is only made possible through the several methodological innovations that we summarize in the following.

In order to handle class imbalances in affective computing, we propose the following modifications beyond Kraus & Feuerriegel [41]: (i) bidirectional processing of the text, (ii) dropout layers as a means of regularization, and (iii) a weighted loss function. The latter becomes especially critical due to the imbalanced distribution of labels. In fact, without the weighted loss function, the network ends up resembling merely a majority class vote.

We further propose an extension of transfer learning called sent2affect. That is, the network is first trained on the basis of sentiment analysis and, after exchanging the output layer, is then tuned to the task of emotion recognition. To the best of our knowledge, this presents a novel strategy for better affective computing as the inductive knowledge transfer is not merely based on a different dataset, but a different task.

Even though affective computing has gained great traction over the past several years [69], there is a scarcity of widely-accepted datasets for text-based emotion recognition that can be used for benchmarking and that facilitate fair comparisons. A relatively small, but more common, dataset was provided by SemEval-2007 and consists of annotated news headlines [78]. A significantly larger, but underutilized, corpus is composed of affect-labeled literary tales [4]. Our literature review notes considerable differences across datasets that vary in their linguistic style, domain, affective dimensions, and the structure of the outcome variable. With regard to the latter, the majority of datasets involve a classification task in which exactly one affective category is assigned to a document, while others request a numerical score across multiple dimensions, i.e., a regression task. Hence, it is a by-product of this research to contribute a holistic comparison that benchmarks different methods across datasets used in prior research. For this purpose, we conducted an extensive search for affect-labeled datasets that serves as the foundation for our computational experiments. As a result, we find that deep learning consistently outperforms the baselines from traditional machine learning. In fact, we observe performance improvements of up to 23.2% in F1-score as part of classification tasks and 11.6% in mean squared error as part of regression tasks.

The findings of this work have direct implications for management, practice, and research. As such, various application areas of decision support – such as customer support, marketing, or recommender systems – can be improved considerably through the use of affective computing. Similarly, all systems with human-computer interactions (e.g. chatbots and personal assistants) could further benefit from emotion recognition and a deeper understanding of empathy. In fact, emotion detection could significantly impact and refine all use cases in which sentiment analysis (i.e., only positive/negative polarity) has already proved to be a valuable approach, since these lend themselves to a more fine-grained analysis and decision-making beyond only one dimension. In academia, text-based emotion recognition supports the cognitive and social sciences as a new approach to measuring and interpreting individual and collective emotional states.

The rest of this paper is structured as follows. Section 2 reviews earlier works on text-based emotion recognition, including the underlying affect theories, datasets used for benchmarking, and computational approaches. This reveals a research gap with regard to both deep neural networks and transfer learning within the field of affective computing. As a remedy, Section 3 introduces our methods rooted in deep learning, which are then evaluated in Section 4. Based on our findings, we detail implications for both research and management in Section 5, while Section 6 concludes.