Skip to main content
SpringerLink
Log in

Annotating Expressions of Opinions and Emotions in Language

  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

This paper describes a corpus annotation project to study issues in the manual annotation of opinions, emotions, sentiments, speculations, evaluations and other private states in language. The resulting corpus annotation scheme is described, as well as examples of its use. In addition, the manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • C. Anderson G. McMaster (1982) ArticleTitleComputer Assisted Modeling of Affective Tone in Written Documents Computers and the Humanities 16 1–9 Occurrence Handle10.1007/BF02259727

    Article  Google Scholar 

  • N. Asher (1986) ArticleTitleBelief in Discourse Representation Theory Journal of Philosophical Logic 15 127–189 Occurrence Handle10.1007/BF00305490

    Article  Google Scholar 

  • A. Banfield (1982) Unspeakable Sentences Routledge and Kegan Paul Boston

    Google Scholar 

  • Bergler S. (1992) Evidential Analysis of Reported Speech. Ph.D. thesis, Brandeis University.

  • D. Biber E. Finegan (1989) ArticleTitleStyles of Stance in English: Lexical and Grammatical Marking of Evidentiality and Affect Text 9 IssueID1 93–124

    Google Scholar 

  • Breck E., Cardie C. (2004) Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions. In Proceedings of the Twentieth International Conference on Computational Linguistics (COLING 2004). Geneva, Switzerland, pp. 120–126.

  • R. Bruce J. Wiebe (1999) ArticleTitleRecognizing Subjectivity: A Case Study of Manual Tagging Natural Language Engineering 5 IssueID2 187–205 Occurrence Handle10.1017/S1351324999002181

    Article  Google Scholar 

  • Cardie C., Wiebe J., Wilson T., Litman D. (2003) Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering. In Working Notes – New Directions in Question Answering (AAAI Spring Symposium Series).

  • H.-N. Castañeda (1977) ArticleTitleOn the Philosophical Foundations of the Theory of Communication: Reference Midwest Studies in Philosophy 2 165–186

    Google Scholar 

  • S. Chatman (1978) Story and Discourse: Narrative Structure in Fiction and Film Cornell University Press Ithaca, New York

    Google Scholar 

  • D. Cohn (1978) Transparent Minds: Narrative Modes for Representing Consciousness in Fiction Princeton University Press Princeton, NJ

    Google Scholar 

  • Cunningham H., Maynard D., Bontcheva K., Tablan V. (2002) GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (A CL-02). Philadelphia, Pennsylvania, pp. 168–175.

  • Dave K., Lawrence S., Pennock D.M. (2003) Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. In Proceedings of the 12th International World Wide Web Conference (WWW2003). Budapest, Hungary. Available at http://www2003.org.

  • J. Rivera Particlede (1977) A Structural Theory of Emotions International Universities Press New York

    Google Scholar 

  • L. Doležel (1973) Narrative Modes in Czech Literature University of Toronto Press Toronto, Canada

    Google Scholar 

  • K. Donnellan (1966) ArticleTitleReference and Definite Descriptions Philosophical Review 60 281–304

    Google Scholar 

  • G. Fauconnier (1985) Mental Spaces: Aspects of Meaning Construction in Natural Language MIT Press Cambridge, Massachusetts

    Google Scholar 

  • Fodor J.D. (1979) The Linguistic Description of Opaque Contexts, Outstanding dissertations in linguistics 13. Garland, New York & London.

  • Gordon A., Kazemzadeh A., Nair A., Petrova M. (2003) Recognizing Expressions of Commonsense Psychology in English Text. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (A CL-03). Sapporo, Japan, pp. 208–215.

  • M. Halliday (1985/1994) An Introduction to Functional Grammar Edward Arnold London

    Google Scholar 

  • R.P. Hart (1984) Systematic Analysis of Political Discourse: The Development of DICTION K.S (Eds) et al. Political Communication Yearbook: 1984 Southern Illinois University Press Carbondale, Illinois 97–134

    Google Scholar 

  • Hatzivassiloglou V., McKeown K. (1997) Predicting the Semantic Orientation of Adjectives. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-97). Madrid, Spain, pp. 174–181.

  • Heise D. (2001) Project Magellan: Collecting Cross-Cultural Affective Meanings via The Internet. Electronic Journal of Sociology 5(3). Available at http://www.sociology.org.

  • Heise D.R. (1965) Semantic Differential Profiles for 1000 Most Frequent English Words. Psychological Monographs 79(601).

  • P. Johnson-Laird K. Oatley (1989) ArticleTitleThe Language of Emotions: An Analysis of a Semantic Field Cognition and Emotion 3 IssueID2 81–123

    Google Scholar 

  • Kaufer D. (2000) Flaming: A White Paper. http://www.eudora.com.

  • D. Kaufer S. Ishizaki B. Butler J. Collins (2004) The Power of Words; Unveiling the Speaker and Writer’s Hidden Craft Lawrence Erlbaum Mahwah, New Jersey

    Google Scholar 

  • K. Krippendorf (1980) Content Analysis: An Introduction to its Methodology Sage Publications Beverly Hills

    Google Scholar 

  • S.-Y. Kuroda (1973) Where Epistemology, Style and Grammar Meet: A Case Study from the Japanese P. Kiparsky S. Anderson (Eds) A Festschrift for Morris Halle Holt Rinehart & Winston New York, NY 377–391

    Google Scholar 

  • S.-Y. Kuroda (1976) Reflections on the Foundations of Narrative Theory—From a Linguistic Point of View T. Dijk Particlevan (Eds) Pragmatics of Language and Literature North-Holland Amsterdam 107–140

    Google Scholar 

  • W. Labov (1984) Intensity D. Schiffrin (Eds) Meaning, Form, and Use in Context: Linguistic Applications Georgetown University Press Washington, DC 43–70

    Google Scholar 

  • J. Martin (1992) English Text: System and Structure John Benjamins Philadelphia/Amsterdam

    Google Scholar 

  • J. Martin (2000) Beyond Exchange: APPRAISAL Systems in English S. Hunston G. Thompson (Eds) Evaluation in Text: Authorial Stance and the Construction of Discourse Oxford University Press Oxford 142–175

    Google Scholar 

  • Onega S., Landes J.A.G. (1996) Narratology: An Introduction. Longman.

  • A. Ortony G.L. Clore M.A. Foss (1987) ArticleTitleThe Referential Structure of the Affective Lexicon Cognitive Science 11 341–364 Occurrence Handle10.1016/S0364-0213(87)80010-1

    Article  Google Scholar 

  • C.E. Osgood G. Suci P. Tannenbaum (1957) The Measurement of Meaning University of Illinois Press Urbana

    Google Scholar 

  • Pang B., Lee L., Vaithyanathan S. (2002) Thumbs up? Sentiment Classification Using Machine Learning Techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2002). Philadelphia, Pennsylvania, pp. 79–86.

  • W.V. Quine (1956/1976) The Ways of Paradox and Other Essays, Revised and Enlarged Edition Harvard University Press Cambridge, Massachusetts

    Google Scholar 

  • R. Quirk S. Greenbaum G. Leech J. Svartvik (1985) A Comprehensive Grammar of the English Language Longman New York

    Google Scholar 

  • W. Rapaport (1986) ArticleTitleLogical Foundations for Belief Representation Cognitive Science 10 371–422 Occurrence Handle10.1016/S0364-0213(86)80011-8

    Article  Google Scholar 

  • Riloff E., Wiebe J. (2003) Learning Extraction Patterns for Subjective Expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 105–112.

  • Riloff E., Wiebe J., Wilson T. (2003) Learning Subjective Nouns Using Extraction Pattern Bootstrapping. In Proceedings of the 7th Conference on Natural Language Learning (CoNLL-2003). Edmonton, Canada, pp. 25–32.

  • J. Russell (1980) ArticleTitleA Circumplex Model of Affect Journal of Personality and Social Psychology 39 1161–1178 Occurrence Handle10.1037//0022-3514.39.1.29

    Article  Google Scholar 

  • Spertus E. (1997) Smokey: Automatic Recognition of Hostile Messages. In Proceedings of the Eighth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-97). Providence, Rhode Island, pp. 1058–1065.

  • P. Subasic A. Huettner (2001) ArticleTitleAffect Analysis of Text Using Fuzzy Semantic Typing IEEE Transactions on Fuzzy Systems 9 483–496 Occurrence Handle10.1109/91.940962

    Article  Google Scholar 

  • Tong R. (2001) An Operational System for Detecting and Tracking Opinions in Online Discussions. In Working Notes of the SIGIR Workshop on Operational Text Classification. New Orleans, Louisianna, pp. 1–6.

  • Turney P. (2002) Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, Pennsylvania, pp. 417–424.

  • P. Turney M.L. Littman (2003) ArticleTitleMeasuring Praise and Criticism: Inference of Semantic Orientation from Association ACM Transactions on Information Systems (TOTS) 21 IssueID4 315–346

    Google Scholar 

  • B. Uspensky (1973) A Poetics of Composition University of California Press Berkeley, CA

    Google Scholar 

  • D. Watson A. Tellegen (1985) ArticleTitleToward a Consensual Structure of Mood Psychological Bulletin 98 219–235 Occurrence Handle10.1037/0033-2909.98.2.219

    Article  Google Scholar 

  • White P. (2002) Appraisal: The Language of Attitudinal Evaluation and Intersubjective Stance. In Verschueren, Ostman, blommaert, Bulcaen (eds.), The Handbook of Pragmatics. Amsterdam/Philadelphia, John Benjamins Publishing Company, pp. 1–27.

  • J. Wiebe (1991) ArticleTitleReferences in Narrative Text Noûs 25 IssueID4 457–486

    Google Scholar 

  • J. Wiebe (1994) ArticleTitleTracking Point of View in Narrative Computational Linguistics 20 IssueID2 233–287

    Google Scholar 

  • Wiebe J. (2000) Learning Subjective Adjectives from Corpora. In Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI-2000). Austin, Texas, pp. 735–740.

  • Wiebe J. (2002) Instructions for Annotating Opinions in Newspaper Articles. Department of Computer Science Technical Report TR-02-101, University of Pittsburgh.

  • Wiebe J., Bruce R., Bell M., Martin M., Wilson T. (2001a) A Corpus Study of Evaluative and Speculative Language. In Proceedings of the 2nd ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-2001). Aalborg, Denmark, pp. 186–195.

  • Wiebe J., Wilson T., Bell M. (2001b) Identifying Collocations for Recognizing Opinions. In Proceedings of the ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation. Toulouse, France, pp. 24–31.

  • J. Wiebe T. Wilson R. Bruce M. Bell M. Martin (2004) ArticleTitleLearning Subjective Language Computational Linguistics 30 IssueID3 277–308 Occurrence Handle10.1162/0891201041850885

    Article  Google Scholar 

  • Y. Wilks J. Bien (1983) ArticleTitleBeliefs, Points of View and Multiple Environments Cognitive Science 7 95–119 Occurrence Handle10.1016/S0364-0213(83)80007-X

    Article  Google Scholar 

  • Wilson T., Wiebe J. (2003) Annotating Opinions in the World Press. In Proceedings of the 4th ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-03). Sapporo, Japan, pp. 13–22.

  • Wilson T., Wiebe J., Hwa R. (2004) Just How Mad Are You? Finding Strong and Weak Opinion Clauses. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI, 2004). San Jose, California, pp. 761–766.

  • Yi J., Nasukawa T., Bunescu R., Niblack W. (2003) Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques. In Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM-2003). Melbourne, Florida, pp. 427–434.

  • Yu H., Hatzivassiloglou V. (2003) Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 129–136.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, 15260, USA

    Janyce Wiebe

  2. Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, 15260, USA

    Theresa Wilson

  3. Department of Computer Science, Cornell University, Ithaca, NY, 14853, USA

    Claire Cardie

Authors
  1. Janyce Wiebe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theresa Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claire Cardie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Janyce Wiebe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wiebe, J., Wilson, T. & Cardie, C. Annotating Expressions of Opinions and Emotions in Language. Language Res Eval 39, 165–210 (2005). https://doi.org/10.1007/s10579-005-7880-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-005-7880-9

Keywords

Search

Navigation