Portrayed emotions in the movie "Forrest Gump"

Possible applications

In combination with the already publicly available brain imaging data, these annotations form a two-hour high resolution fMRI measurement for auditory emotion cue processing from 20 participants. With the addition of a future publication of brain imaging data recorded from a stimulation with the audio-visual movie, the full dataset will enable comparative studies investigating the processing of rich emotional stimuli via different sensory pathways. Moreover, these new annotations of portrayed emotions are another step towards a comprehensive description of this reproducible movie stimulus¹ and improve its general utility for independent studies on social cognition with a focus on the perception of emotions in real-life situations.

Stimulus

The annotated stimulus was a slightly shortened version of the movie “Forrest Gump” (R. Zemeckis, Paramount Pictures, 1994). Two different variants of this movie were annotated separately. The first was a German audio-description as broadcast as an additional audio track for visually impaired listeners on Swiss public television (Koop, Michalski, Beckmann, Meinhardt & Benecke produced by Bayrischer Rundfunk, 2009). This audio-only stimulus was the same as the one used in Hanke M et al.¹, and is largely identical to the dubbed German soundtrack of the movie except for interspersed narrations by a male speaker who describes the visual content of a scene. These descriptions take place when there is no dialog, off-screen speech, or other relevant audio-content in the movie. The second variant is the audio-visual movie with the original dubbed German soundtrack (without additional narrations) and the exact same timing (and cut scenes) as the audio-only version (1, contains instructions on how to reproduce the stimulus from the DVD release).

Observers

Annotations were created by the authors themselves as part of a practical course on scientific observation. A total of 12 students at the Otto-von-Guericke-University in Magdeburg, Germany participated in this effort and received course credit. Based on personal preference, students assigned themselves to one of two groups tasked with the annotation of portrayed emotion in either the audio-visual movie, or the audio-only version. The audiovisual group consisted of 9 students (all female), the audio-only group comprised 3 students (all male). This gender bias could have had an impact on the perceived emotions. None of the observers participated in the previous brain imaging study¹.

Procedure

Annotations were performed with the help of Advene¹², a dedicated open-source video annotation tool that offers convenient navigation of structural units of the stimulus and provided uniform position information with subsecond precision. Each observer annotated all of 205 previously identified movie scenes in an individual random order to minimize “carry-over” effects and help observers focus on current indicators of portrayed emotions in contrast to more slowly changing characteristics, such as mood, or other biases introduced by the movie plot. Scenes were used as units for annotation, because transitions between them typically involve a change of location or fast progression of time that make persistence of the nature of portrayed emotions across scenes less likely. Observers used a scene annotation of the movie imported into Advene to aid navigation within the movie. They only had to click on the visual representation of a scene (a numbered box) in a time line display to start the playback at the start of a scene. Advene could also be configured to play a selected scene in a continuous loop. For composing the annotation content, observers used a separate spreadsheet application and not the data input features of Advene.

Observers were instructed to work alone in a setting free of distractions and to use headphones for optimal saliency of the stimulus. There was no strict guideline regarding the length of an annotation session, but observers were informed to stop working for any number of breaks whenever they could no longer guarantee a high level of attention for the task. Sessions were typically spread over multiple days over the course of three weeks. On average, observers reported that it required around 30 hours each to complete the annotation.

Annotation content

Each observer collected annotations in a spreadsheet with columns for start time and end time of an emotion; a label of the movie character portraying the emotion; variables for valence, arousal, and direction of the emotion as well as an optional emotion category label; and a list of identified indicators for the start and end of an emotion. Each of these variables are described in more detail in the remainder of this section. In addition, observers also recorded the index of the scene containing a respective emotion. This variable was only used for error detection (mismatch of emotion duration with the start and end time of a scene) and is not included in the published dataset.

Start and end time  The start time identifies the onset of the first observed indicator of an emotion. Likewise, the end time corresponds to the offset of an emotion — defined as the time where no evidence for an emotion is present anymore. Observers were instructed to aggregate short segments of uniform emotional episodes into a single annotation. For example, if a character was happy and smiling throughout a long period in a scene, the episode would span all the time from the beginning to the end of the smile even if the character’s face was not visible the entire period, as long as no evidence of a change of emotion was found. All times are reported in seconds. Although Advene offers time stamps with the precision of the movie frame rate (25 frames per second), pre-tests revealed that the temporal accuracy of human observers for complex emotion indicators, like a developing facial expression, is much coarser. In order to reduce the chance of unrecoverable typos in the time stamps, observers were instructed to record timing information with second-precision only.

Onset and offset cues  Onset indicators describe what kind of evidence for an emotion was detected in the movie stimulus. We distinguished facial expressions, gestures or body language, context information, verbal, and non-verbal audio cues. Observers could record multiple onset indicators. Despite the term “onset”, these indicators did not all have to be present at the very beginning of an emotional episode. For example, an extended period of sadness could start with a facial expression and later on add a congruent body language cue. In that case, the respective labels were aggregated into a list. In the audio-only stimulus, the narrator was frequently the only source of information regarding the emotional state of a character; thus, we included a dedicated category for this scenario.

In addition to onset indicators, observers also recorded the type of evidence for the end of an emotional episode. Four conditions were distinguished: 1) changing from one emotion to another 2) entering a neutral emotional state 3) a character leaving an ongoing scene with no further evidence for its emotional state 4) the end of a scene. Table 1 describes all distinguished cue categories.

Character label  This label identifies which movie character is portraying an emotion. Observers were provided with a list of character labels that was derived from an annotation of the movie’s dialog in order to achieve a consistent labeling. However, some characters portray emotions but have no lines of dialog, and, in other instances, observers chose inconsistent labels. During data curation, character labels were therefore manually consolidated into a set of 35 categories that uniformize labeling across observers. The full list of character labels is shown in table 2 and was derived using the following rules: a) main characters and famous/familiar characters receive individual labels as do the two narrator voices b) relatives of main characters are labeled according to their relation c) all other characters are consolidated into generic categories that preserve gender, age (children, adults, or seniors), and number (individual or group).

It is important to note that only emotions portrayed by one or more movie characters were recorded. This excludes other potentially emotion-inducing stimuli (e.g, music in the soundtrack) or inanimate stimuli (e.g., the depiction of a beautiful sunset or chirping birds).

Valence, arousal, and direction  Binary variables were used to record the valence of an emotion (positive vs. negative), the state of arousal of the character portraying it (high vs. low), and whether the emotion is directed towards the character itself or another one (e.g. feeling pity for somebody). Together, these three indicator variables offer a coarse categorization of observed emotions in the spirit of dimensional models of emotion.

Arousal was considered a global indicator of the emotional status of a character, i.e., for any given time in the movie annotations, indicate either a high or low level of arousal for a character — but never both at the same time. Unlike arousal, valence and direction indicators were considered non-exclusive, i.e., a character could show signs of both positive and negative emotions at the same time. Likewise, a character could simultaneously exhibit self-directed emotions and emotions directed towards others. In order to annotate such situations, observers had to create records for two or more portrayed emotions with overlapping timing.

Emotion labels  Observers were instructed to assign one of 22 labels for discrete emotion categories. These categories were derived from a schema proposed by Ortony¹³ and concrete definitions were developed based on the Specific Affect Coding System¹¹. The complete list of emotions is shown in table 3. Pre-tests using a larger and more detailed set of emotion categories revealed weak agreement between observers for a forced categorization of emotional episodes. Based on these results, we decided to make this part of the annotation optional, and observers were instructed to assign an emotion label only if they saw a “perfect” match with any of the categories.

Quality control

We used an automated procedure to check annotation records of individual observer for errors or potential problems. Observers submitted their annotations in tabular form to a script that generated a list of error and warning messages. Using this feedback, observers double-checked their annotations as often as necessary until no objective errors were found and all warning messages were confirmed to be false positives. The following annotation properties were tested:

Dataset content

The released data comprises four components: 1) raw annotations 2) aggregate inter-observer agreement (IOA) time series 3) emotion episode segmentations 4) the source code to generate all data derived from the raw annotations and all summary statistics presented in the Data Note.

Raw annotations  In order to maximize the utility of the annotations, they are released in full rather than only as aggregate information. The annotations from each observer are available in an individual plain text file with comma-separated value (CSV) markup in the raw directory. This file format facilitates import and further processing in virtually any software application that supports data in tabular form. The file names indicate the annotated stimulus type (prefix av for the audio-visual movie and ao for the audio-only movie) as well as the observer identity.

Each file contains nine columns that correspond to the annotation properties described above. start, end) start and end time of an emotion episode reported as seconds from the start of the movie; character) a single character label from the set shown in table 2 indicating the character portraying the emotion; arousal) state of arousal indicated with the labels LOW or HIGH; valence) emotional valence indicated with the label POS or NEG; direction) direction of the emotion indicated with the labels SELF or OTHER; emotion) a space-separated list of emotion labels from the set shown in table 3; oncue) a space-separated list of onset indicators for a portrayed emotion from the set shown in table 1; and offcue) a space-separated list of offset indicators from the set also shown in table 1. With the exception of the columns emotion and offcue, all variables are considered mandatory and are present for all annotations. The remaining columns contain optional attributes and can be empty.

Inter-observer agreement (IOA) time series  IOA time series reflect the consistency of observations at any particular time in the movie. The time series amplitude corresponds to the fraction of observers indicating the presence of a particular attribute of an emotion episode (interval [0, 1]). Time series for the three bipolar attributes arousal, valence, and direction were computed by subtracting the IOA with respect to the presence of the both extremes from each other. For example, the time series for arousal was computed by subtracting the time series of the IOA for low-arousal episodes from the time series of high-arousal episodes. This results in values from an interval of [−1, 1] — where −1 corresponds to perfect agreement with respect to the presence of a low-arousal episode and +1 indicates perfect agreement regarding high arousal.

IOA time series were computed for three different segment durations (i.e., agreement is considered when observers indicate the presence of a particular attribute anywhere in a segment): 1 s (temporal precision of the annotations), 2 s (corresponding sampling rate of the available brain imaging data), and the actual location and duration of individual shots in the movie (median duration ≈5 s). Individual time series for the three bipolar attributes, all 22 emotion categories, and the six emotion onset cues are available in the timeseries directory. The file names are encoded as follows: prefix ioats, followed by a label for the segment duration (e.g. 1s), followed by the stimulus label (av or ao), and a character label (e.g. forrest, see table 2) or allchar for an aggregate time series across all characters. All files are in plain text format with CSV markup. The column headers indicate the corresponding emotion attribute.

Emotion episode segmentation  Based on IOA time series, a segmentation of the movie into episodes of portrayed emotion was performed for each of the 35 character categories individually. IOA time series for 33 attributes (arousal (bipolar), valence and direction (separate for each state), 22 emotion category labels, and six onset cues) were binarized by thresholding with a minimum absolute IOA of 50%. An episode of portrayed emotion was defined as a time window with at least one super-threshold emotion attribute. The duration of an episode was determined by the number of consecutive movie segment without a change in the unique combination of super threshold attributes.

Emotion episodes for all characters are available in the segmentation directory in plain-text files with tab-separated value (TSV) markup. The file format follows the conventions required by the Advene software for “simple-structured content”, but are also compatible with generic spreadsheet applications. Each file contains three columns (without a header). The two columns contain the start and end time of an episode in seconds. The third column contains a space-separated list of key/value pairs. Five such properties are present for each episode: char) the label of the character portraying the emotion; tags) a comma-separated list of tags corresponding to all super-threshold emotion attributes; arousal, val_pos, and val_neg) these three properties each correspond to the median IOA value across the entire episode for arousal (bipolar) and positive and negative valence (each unipolar). The names of all tags typically correspond to the lower-case name of the attribute, with the exception of the three bipolar attributes where each extreme is coded with a dedicated label (arousal: ha/la (high/low), valence: pos/neg, direction: self/other).

Emotion episode segmentation was performed separately for the audio-visual and the audio-only movie—once using IOA time series sampled with a 1 s segment size and a second time sampled with the actual duration and location of shots in the audio-visual movie. File names encode these conditions as follows: prefix emotions followed by the stimulus label (av or ao) followed by the segment size label (1s or shots).

Source code  The full source code for all descriptive statistics and figures included in this paper is available in descr_stats.py (Python script). Moreover, this script also contains all functionality to generate the IOA time series as well as the emotion episode segmentation from the raw annotations in the data release. The required information on the timing of movie scenes and shots is contained in two additional CSV files: movie_scenes.csv and movie_shots.csv.

Time stamp conversion

To be able to use the provided emotion annotations for an analysis of functional data (fMRI, cardiac trace, etc.) acquired using the previously described procedure¹, timing information has to be converted because data were recorded in eight sessions using partially overlapping segments of the movie. Table 4 specifies the location of these segments with respect to the timing of the unsegmented movie used for emotion annotation. Subtracting movie segment start times from annotation time stamps yields valid relative time stamps with respect to the time series of the available functional data.

Observer agreement as a probabilistic indicator

It can be assumed that observers in this study have used individual intensity thresholds for determining the start, end, or quality of an emotional episode. Therefore, the IOA with respect to a particular aspect of portrayed emotions at any given time in the movie could be seen as a measure of the relative probability of perceiving an emotion feature. In order to assess the reliability of such a measure, we generated time series for the IOA with a sampling rate of 1 Hz, as described above, and evaluated their reliability and validity in terms of their correlation across observers, stimulus types, and with each other.

Intra-stimulus inter-observer consistency  The reliability of IOA time courses was estimated by repeatedly splitting the set of observers into two groups of approximately equal size. Within each group, IOA time courses where computed and correlated across the observer subgroups. Split-half correlations were computed for all possible ways of combining observers into two groups (without replacement). Table 5 shows the average time series correlation across all splits and the width of the 95% confidence interval of the mean.

In general, IOA reliability is larger for the AV movie annotations than for the AO movie. This finding is likely a result of the lower number of observers for the latter. Intra-stimulus IOA reliability estimates suggest that a subset of emotion attributes or categories is not contained in the AO movie stimulus. The most prominent example is the direction of an emotion. Emotion categories like hate, shame, or relief also show large reliability differences between stimulus types. However, in these cases the small number of emotion episodes may multiply the negative effect of the small number of observers on the interpretability of this estimate.

Uniformly high reliability estimates were found for the two main dimensional emotion attributes arousal and valence, as well as for the five most frequently annotated emotion categories anger/rage, fear, happiness, love, and sadness.

Intra-stimulus inter-variable correlations  In order to assess possible confounds in the annotations, we computed the correlation of intra-stimulus IOA time courses (across all observers) for all pairwise combinations of emotion attributes and categories. The correlation matrices for the AV and the AO movie stimulus are depicted in figure 4. The results reveal an expected correlation pattern between individual emotion categories and the two main dimensional attributes arousal and valence — with stronger correlations being observed for more primary and more frequently portrayed emotions. Interestingly, audio and verbal cues in the AV stimulus seems to be more associated with a high state of arousal and emotions directed towards other people than in the AO stimulus. In the AV stimulus, facial expressions seem to have a tendency to communicate negative emotions rather than positive ones.

Figure 4. Intra-stimulus indicator correlation.

The lower triangular matrix depicts the correlations between the IOA time courses for the three primary bipolar emotion attributes (arousal [high/low], valence [pos/neg], and direction [self/other]), the 22 emotion categories, and the six emotion onset indicators for the audio-visual movie. The upper triangular matrix shows the corresponding correlations for the audio-only movie. There were no observations for the emotion “fears confirmed”, or the facial expression onset cue in the audio-only movie. Likewise, there were no observations for the “narrator” onset cue in the audio-visual movie. The corresponding undefined correlations are depicted as zeros.

Inter-stimulus consistency  To estimate the similarity of the two stimulus types, we computed the correlation of IOA time courses for the AV and the AO movie with respective to emotion attributes and categories. Table 6 shows the respective correlations and their 95% confidence intervals. Again, the three dimensional attributes as well as the five most frequent emotion categories show significant correlations. Notably, verbal cues are the most consistent indicator of emotions across both types of stimuli. This is evidence for an expected substantial semantic similarity with respect to the emotional content in the two variants of the “Forrest Gump” movie.