Keywords
semantic, cognition, brain imaging, frequency statistics
This article is included in the INCF gateway.
This article is included in the Real-life cognition collection.
This article is included in the Data: Use and Reuse collection.
semantic, cognition, brain imaging, frequency statistics
Detection of semantic conflict is an important cognitive skill for human social interaction. It is required to identify lies (false statements made with the intention to deceive) but also to correctly interpret stylistic devices — such as sarcasm and irony (statements with direct meaning that is the opposite1 or contrary2 to the implied semantic content). As the interpretation of such events is highly context dependent, it is difficult to study how the brain processes these in the context of real-life like interactions in complex natural environments.
In this study we explored occurrences of semantic conflict in the core stimulus of the studyforrest project (http://studyforrest.org) — the motion picture “Forrest Gump” — in order to evaluate whether the available brain imaging data3,4 can be readily used to study this aspect of cognition. We annotated the presence of contradictory statements, including lies and ironic statements, as well as the portrayal of cues, such as exaggeration or raised eyebrows, that are often associated with making ironic statements. Additionally, we recorded the context that allowed observers to classify an event as contradictory.
Depending on the exact criterion used for identifying events across observers, we found only between 64 and 36 occurrences of semantic conflict or portrayal of irony cues in the entire movie stimulus. These are likely insufficient numbers for an investigation based on these data alone. However, these new annotations nevertheless contribute to a more comprehensive description of this complex movie stimulus5,6 and may be useful as confound variables in subsequent studies.
The annotated stimulus was a slightly shortened (≈2 h) version of the movie Forrest Gump (R. Zemeckis, Paramount Pictures, 1994), with a dubbed German soundtrack, and is identical to the audio-visual movie annotated in 5,6. Further details on this particular movie cut, and how to reproduce it from commercially available sources, are available in 4.
Three observers (all female, age 19–20) independently annotated the movie. They were also involved in the development of the concept for this annotation.
Observers were instructed to watch the movie from beginning to end, replaying scenes as often as required, and to detect two types of events: 1) whenever a verbal statement is made that contradicts with either the immediate context or with the viewer’s body-of-knowledge at this point in the movie, or 2) whenever one or more cues associated with irony (predefined list, see below) are portrayed. In either case, observers had to describe the event by specifying its properties via a number of variable settings in a spreadsheet. The software video player VLC (http://www.videolan.org/vlc) was used to watch and navigate through the movie.
For each annotated event, a total of 10 properties were recorded, each of which are described in the following sections.
Start and end The duration of each event is recorded in start and end as the number of seconds from movie start (no sub-second precision, due to limitations of the video player time display). The time-points correspond to the onset and offset of the respective evidence. Both times can be identical in the case of events with less than one second duration. For contradictory statements, the duration covers the time from the onset of evidence of a contradiction until the end of the statement.
Sender and receiver The identity of a character making a contradictory statement or portraying an irony cue is encoded in sender using character labels listed in 5. In the case that the respective statement is directed to another present movie character, its identity is encoded in receiver.
Evidence of a contradiction The contradiction flag indicates the presence of a contradiction in an event (1: present, 0: absent). The variable proof qualifies if the current or previous events provide the viewer with information to allow the detection of this contradiction (see Table 1). If proof is empty, the movie itself does not contain such information (e.g. a common sense contradiction).
Irony cues The variable cues contains a space-separated list of labels for all irony cues present in a particular events. See Table 1 for a description of all possible labels.
Event category The category variable classifies events into lies, ironic statements, and other events (value empty).
Intention Two more variables encode whether a contradiction was used deliberately and whether this was noticed by the receiver. The variable intended encodes the presence of evidence for deliberate use (1: yes, 0: no). The variable is empty if there is no evidence for either case. The second variable intention_decoded encodes, in the same way, whether a potential receiver noticed a deliberate ironic statement or lie.
The released annotation are three, text-based, comma-separated-value (CSV) formatted tables (data/o??.csv), one for each observer.
The source code for all descriptive statistics included in this paper is available in code/descriptive_stats.py (Python script).
We used an automated procedure to check the annotation records of individual observers for errors or potential problems. Observers submitted their annotations in tabular form to a script that generated a list of error and warning messages. Using this feedback, observers double-checked their annotations as often as necessary until no objective errors were found and all warning messages were confirmed to be false positives. The tests included, for example, plausibility of timing information (no end time before the respective start time) or the presence of unknown condition labels.
In order to assess inter-observer agreement of annotations, we used a two-step approach. First, the temporal location of events depicting any relevant property were determined by comparing annotation timing across observers. The columns in Table 2 report agreement statistics for events defined by at least one, two, or all three observers recording an annotation for the same sender at the same time. In the case that individual observers reported events of different length, or with only partially overlapping duration, only the time-windows with the minimum number of observers reporting an event were considered.
In the second step, we computed Fleiss’ Kappa7 for each individual property of an annotation separately with respect to being consistently assigned or non-assigned to the identified events (Table 2). We observe increasing inter-observer agreement of all annotated properties with increasing agreement of annotation timing, approaching “substantial” or “almost perfect” agreement — according to the conventions put forth by 8.
F1000Research: Dataset 1. Events of semantic conflict and occurrences of irony cues in the motion picture "Forrest Gump", 10.5256/f1000research.9635.d1362039
F1000Research: Dataset 2. Python script, 10.5256/f1000research.9635.d13620410
In addition, released data, code, and manuscript sources are also available on Github (https://github.com/psychoinformatics-studyforrest-paper-ironyannotation).
MH contributed to the design of the annotation effort, performed the dataset validation, and wrote the paper; PI contributed to the design, coordinated the annotation effort, and wrote the paper. Both authors agreed to the final content of the paper.
Michael Hanke was supported by funds from the German federal state of Saxony-Anhalt and the European Regional Development Fund (ERDF), Project: Center for Behavioral Brain Sciences.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
We are grateful to Denise Naumann, Marisela Markarian, Jasmin Billowie, and Susann Bergmann for their contributions to the design and the execution of the annotation effort. We also appreciate Alex Waite for his seemingly unending willingness to edit papers.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
References
1. Attardo S., Eisterhold J., Hay J., Poggi I.: Multimodal markers of irony and sarcasm. Humor. 2003; 16 (2): 243-260.Competing Interests: No competing interests were disclosed.
Competing Interests: No competing interests were disclosed.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 1 26 Sep 16 |
read | read |
Click here to access the data.
Spreadsheet data files may not format correctly if your computer is using different default delimiters (symbols used to separate values into separate cells) - a spreadsheet created in one region is sometimes misinterpreted by computers in other regions. You can change the regional settings on your computer so that the spreadsheet can be interpreted correctly.
Click here to access the data.
Spreadsheet data files may not format correctly if your computer is using different default delimiters (symbols used to separate values into separate cells) - a spreadsheet created in one region is sometimes misinterpreted by computers in other regions. You can change the regional settings on your computer so that the spreadsheet can be interpreted correctly.
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)