Skip to main content
SpringerLink
Log in

Assessment of Interjudge Reliability in the Open-Ended Questions Coding Process

  • Published:
Quality and Quantity Aims and scope Submit manuscript

Abstract

In the process of coding open-ended questions, the evaluation of interjudge reliability is a critical issue. In this paper, using real data, the behavior of three coefficients of reliability among coders, Cohen’s K, Krippendorff’s α and Perreault and Leigh’s I r are patterned, in terms of the number of judges involved and the categories of answer defined. The outcome underlines the importance of both variables in the valuations of interjudge reliability, as well as the higher adequacy of Perreault and Leigh’s I r and Krippendorff’s α for marketing and opinion research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • A.M. Abernethy G.R. Franke (1996) ArticleTitleThe information content of advertising: A meta-analysis Journal of Advertising 25 IssueID2 1–17

    Google Scholar 

  • Bardin, L. (1977). L’analyse de contenu. (Paris: PUF).

  • E.M. Bennett A.C. Goldstein (1954) ArticleTitleCommunications through limited response questioning Public Opinion Quaterly 18 303–308 Occurrence Handle10.1086/266520

    Article  Google Scholar 

  • Berelson, B. (1952). Content Analysis in Communications Research. Glencoe (Ill.): Free Press.

  • H. Brenner U. Kliebsch (1996) ArticleTitleDependence of weighted kappa coefficients on the number of categories Epidemiology 7 IssueID2 199–202

    Google Scholar 

  • J. Cohen (1960) ArticleTitleA coefficient of agreement for nominal scales Educational and Psychological Measurement 20 IssueIDWinter 37–46

    Google Scholar 

  • N.K. Denzin Y.S. Lincoln (1994) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • ESOMAR (2003). Annual Study of the Market Research Industry 2002. (Amsterdam: ESOMAR).

  • J.L. Fleiss (1971) ArticleTitleMeasuring nominal scale agreement among many raters Psychological Bulletin 76 378–382 Occurrence Handle10.1037/h0031619

    Article  Google Scholar 

  • A. Fontana J.H. Frey (1994) Interviewing. The art of science N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • B.G. Glaser A.L. Strauss (1999) The Discovery of Grounded Theory: Strategies for Qualitative Research Aldine de Gruyter New York

    Google Scholar 

  • K. Grayson R. Rust (2001) ArticleTitleInterrater reliability Journal of Consumer Psychology 10 IssueID1&2 71–73 Occurrence Handle10.1207/S15327663JCP1001&2_06

    Article  Google Scholar 

  • L. Herbert (1977) ArticleTitleKappa revisited Psychological Bulletin 84 IssueID2 289–297 Occurrence Handle10.1037/0033-2909.84.2.289

    Article  Google Scholar 

  • M.B. Holbrook (1977) ArticleTitleMore on content analysis in consumer research Journal of Consumer Research 4 IssueID3 176–177 Occurrence Handle10.1086/208694

    Article  Google Scholar 

  • O.R. Holsti (1969) Content Analysis for the Social Sciences and Humanities Addison-Wesley Reading (MA)

    Google Scholar 

  • Hsu Fied (2003) ArticleTitleInterrater agreement measures: Comments on Kappa, Cohen’s Kappa, Scott’s π and Aicking α Understanding statistics 2 IssueID3 205–219 Occurrence Handle10.1207/S15328031US0203_03

    Article  Google Scholar 

  • M.A. Hughes D.E. Garret (1990) ArticleTitleIntercoder reliability estimation approaches in marketing: A Generability Theory framework for quantitative data Journal of Marketing Research 27 IssueID2 185–95 Occurrence Handle10.2307/3172845

    Article  Google Scholar 

  • N. Kang A. Kara H.A. Laskey F.B. Seaton (1993) ArticleTitleA SAS MACRO for calculating intercoder agreement in content analysis Journal of Advertising 22 IssueID2 17–28

    Google Scholar 

  • H.H. Kassarjian (1977) ArticleTitleContent analysis in consumer research Journal of Consumer Research 4 IssueID2 8–18 Occurrence Handle10.1086/208674

    Article  Google Scholar 

  • R.H. Kolbe M.S. Burnett (1991) ArticleTitleContent-analysis research: An examination of applications with directives for improving research reliability and objectivity Journal of Consumer Research 18 IssueIDSeptember 243–250 Occurrence Handle10.1086/209256

    Article  Google Scholar 

  • K. Krippendorff (1971) ArticleTitleReliability of recording instructions: Multivariate agreement for nominal data Behavioral Science 16 IssueID3 228–235

    Google Scholar 

  • K. Krippendorff (1980) Content Analysis, an Introduction to Its Methodology Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • J.R. Landis G.G. Koch (1977) ArticleTitleThe measurement of observer agreement for categorical data Biometrics 33 159–174 Occurrence Handle10.2307/2529310

    Article  Google Scholar 

  • R.L. Lehmann S. Gupta J. H. Steckel (1998) Marketing Research Addison-Wesley Educational Publishers Inc. Reading (MA)

    Google Scholar 

  • R.J. Light (1971) ArticleTitleMeasures of response agreement for qualitative data: some generalizations and alternatives Psychological Bulleting 76 IssueID5 365–377 Occurrence Handle10.1037/h0031643

    Article  Google Scholar 

  • Y.S. Lincoln E.G. Guba (1985) Naturalistic Inquiry Sage Publications Beverly Hills (CA)

    Google Scholar 

  • M. Lombard J. ZINDER-duch C. Campanela (2002) ArticleTitleContent analysis in mass communication: Assessment and reporting of intercoder reliability Human Communication Research 28 IssueID4 587–604 Occurrence Handle10.1093/hcr/28.4.587

    Article  Google Scholar 

  • López-Aranguren (1989). El análisis de contenido. In: M. García et al. (Coord.): El Análisis de la Realidad Social: Métodos y Técnicas de Investigación. Madrid: Alianza Editorial.

  • T. Luque (1997) Investigación de Marketing Ed. Ariel Barcelona

    Google Scholar 

  • K. Mackensen U. Wille (1999) ArticleTitleQualitative text analysis supported by conceptual data systems Quality & Quantity 33 135–156 Occurrence Handle10.1023/A:1004305723553

    Article  Google Scholar 

  • M.B. Miles A.M. Huberman (1994) Quality Data Analysis An expanded Sourcebook. Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • W.L. Miller B.F. Crabtree (1994) Clinical research N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • W.D. Perreault E.L. LEIGH (1989) ArticleTitleReliability of nominal data based on quantitative judgments Journal of Marketing Research 23 IssueIDMay 130–43

    Google Scholar 

  • Richards Richards (1994) Using computers in qualitative research N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)

    Google Scholar 

  • D. Riffe A.A. Freitag (1997) ArticleTitleA content analysis of content analysis: Twenty-five years of Journalism Quarterly Journalism & Mass Communication Quarterly 74 873–882

    Google Scholar 

  • C.W. Roberts (2000) ArticleTitleA conceptual framework for quantitative text analysis: On joining probabilities and substantive inferences about text Quality & Quantity 34 259–274 Occurrence Handle10.1023/A:1004780007748

    Article  Google Scholar 

  • R.T. Rust B. Cooil (1994) ArticleTitleReliability measures for qualitative data: Theory and implications Journal of Marketing Research 31 1–14 Occurrence Handle10.2307/3151942

    Article  Google Scholar 

  • S. Spiggle (1994) ArticleTitleAnalysis and interpretation of qualitative data in consumer research Journal of Marketing Research 21 IssueIDDecember 491–503

    Google Scholar 

  • E.L. Spitznagel J.E. Helzer (1985) ArticleTitleA proposed solution to the base rate problem in the kappa statistic Archives of General Psychiatry 42 IssueID7 725–28

    Google Scholar 

  • A. Strauss J. Corbin (1990) Basic of qualitative research: Grounded theory procedures and techniques Sage Publications Newbury Park (CA)

    Google Scholar 

  • R.P. Weber (1985) Basic Content Analysis Ed. Sage Publications Newbury Park

    Google Scholar 

  • E.A. Weitzman M.B. Miles (1995) Computer Programs for Qualitative Data Analysis: A Software Sourcebook Sage Publications Thousand Oaks (CA)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Marketing and Market Research, Faculty of Economics and Business Sciences of the University of Granada, Campus Universitario La Cartuja s/n, 18071, Granada, Spain

    Francisco Muñoz Leiva, Francisco Javier Montoro Ríos & Teodoro Luque Martínez

Authors
  1. Francisco Muñoz Leiva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francisco Javier Montoro Ríos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Teodoro Luque Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco Muñoz Leiva.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Leiva, F.M., Ríos, F.J.M. & Martínez, T.L. Assessment of Interjudge Reliability in the Open-Ended Questions Coding Process. Qual Quant 40, 519–537 (2006). https://doi.org/10.1007/s11135-005-1093-6

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-005-1093-6

Keywords

Search

Navigation