Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice

Authors:
Nora McDonald

Drexel University, Philadelphia, PA, USA

Drexel University, Philadelphia, PA, USA
View Profile

,
Sarita Schoenebeck

University of Michigan, Ann Arbor, MI, USA

University of Michigan, Ann Arbor, MI, USA
View Profile

,
Andrea Forte

Drexel University, Philadelphia, PA, USA

Drexel University, Philadelphia, PA, USA
View Profile

Proceedings of the ACM on Human-Computer Interaction Volume 3 Issue CSCWArticle No.: 72pp 1–23https://doi.org/10.1145/3359174

Published:07 November 2019Publication History

Proceedings of the ACM on Human-Computer Interaction

Abstract

What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.

References

Ali Abdolrahmani, William Easley, Michele Williams, Stacy Branham, and Amy Hurst. 2017. Embracing Errors: Examining How Context of Use Impacts Blind Individuals' Acceptance of Navigation Aid Errors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 4158--4169. DOI:https://doi.org/10.1145/3025453.3025528Google ScholarDigital Library
Tony E Adams, Carolyn Ellis, and Stacy Holman Jones. 2017. Autoethnography. The international encyclopedia of communication research methods (2017), 1--11.Google Scholar
David Armstrong, Ann Gosling, John Weinman, and Theresa Marteau. 1997. The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study. Sociology 31, 3 (August 1997), 597--606.Google ScholarCross Ref
Paul Anthony Atkinson, Amanda Coffey, Sara Delamont, John Lofland, and Lyn H. Lofland (Eds.). 2001. Handbook of Ethnography (1 edition ed.). SAGE Publications Ltd, London?; Thousand Oaks, Calif.Google Scholar
Mara Balestrini, Paul Marshall, Raymundo Cornejo, Monica Tentori, Jon Bird, and Yvonne Rogers. 2016. Jokebox: Coordinating Shared Encounters in Public Spaces. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 38--49. DOI:https://doi.org/10.1145/2818048.2835203Google ScholarDigital Library
Shaowen Bardzell. 2010. Feminist HCI: Taking Stock and Outlining an Agenda for Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 1301--1310. DOI:https://doi.org/10.1145/1753326.1753521Google ScholarDigital Library
Louise Barkhuus and Chiara Rossitto. 2016. Acting with Technology: Rehearsing for Mixed-Media Live Performances. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 864--875. DOI:https://doi.org/10.1145/2858036.2858344Google ScholarDigital Library
Rosanna Bellini, Angelika Strohmayer, Ebtisam Alabdulqader, Alex A. Ahmed, Katta Spiel, Shaowen Bardzell, and Madeline Balaam. 2018. Feminist HCI: Taking Stock, Moving Forward, and Engaging Community. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18), SIG02:1--SIG02:4. DOI:https://doi.org/10.1145/3170427.3185370Google ScholarDigital Library
Kirsten Boehner and Carl DiSalvo. 2016. Data, Design and Civics: An Exploratory Study of Civic Tech. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 2970--2981. DOI:https://doi.org/10.1145/2858036.2858326Google ScholarDigital Library
Gerd Bohner. 2001. Writing about rape: Use of the passive voice and other distancing text features as an expression of perceived responsibility of the victim. British Journal of Social Psychology 40, 4 (December 2001), 515--529. DOI:https://doi.org/10.1348/014466601164957Google ScholarCross Ref
Adrien Bousseau, Theophanis Tsandilas, Lora Oehlberg, and Wendy E. Mackay. 2016. How Novices Sketch and Prototype Hand-Fabricated Objects. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 397--408. DOI:https://doi.org/10.1145/2858036.2858159Google Scholar
Geoffrey C. Bowker and Susan Leigh Star. 2000. Sorting Things Out: Classification and Its Consequences (Revised edition ed.). The MIT Press, Cambridge, Massachusetts London, England.Google ScholarDigital Library
LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. 2016. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4872--4883. DOI:https://doi.org/10.1145/2858036.2858215Google Scholar
LouAnne E. Boyd, Kyle Rector, Halley Profita, Abigale J. Stangl, Annuska Zolyomi, Shaun K. Kane, and Gillian R. Hayes. 2017. Understanding the Role Fluidity of Stakeholders During Assistive Technology Research ?In the Wild." In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 6147--6158. DOI:https://doi.org/10.1145/3025453.3025493Google ScholarDigital Library
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77--101.Google ScholarCross Ref
Alan Bryman and Robert G. Burgess. 1994. Analyzing Qualitative Data. Routledge, London.Google Scholar
G. Burrell and G. Morgan. 1979. Sociological Paradigms and Organizational Analysis: Elements of the Sociology of Corporate Life. Ashgate Publishing.Google Scholar
Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 981--992. DOI:https://doi.org/10.1145/2858036.2858498Google ScholarDigital Library
John L. Campbell, Charles Quincy, Jordan Osserman, and Ove K. Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research 42, 3 (2013), 294--320.Google ScholarCross Ref
Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis (1 edition ed.). SAGE Publications Ltd, London?; Thousand Oaks, Calif.Google Scholar
Ana Paula Chaves and Marco Aurelio Gerosa. 2018. Single or Multiple Conversational Agents?: An Interactional Coherence Comparison. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 191:1--191:13. DOI:https://doi.org/10.1145/3173574.3173765Google ScholarDigital Library
Chia-Fang Chung, Elena Agapie, Jessica Schroeder, Sonali Mishra, James Fogarty, and Sean A. Munson. 2017. When Personal Tracking Becomes Social: Examining the Use of Instagram for Healthy Eating. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1674--1687. DOI:https://doi.org/10.1145/3025453.3025747Google ScholarDigital Library
Patricia Hill Collins. 2015. Intersectionality's Definitional Dilemmas. Annual Review of Sociology 41, 1 (2015), 1--20.Google ScholarCross Ref
Juliet Corbin and Anselm Strauss. 2007. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3rd ed.). SAGE Publications, Inc.Google Scholar
John W. Creswell. 1998. Qualitative Inquiry and Research Design?: Choosing Among Five Traditions. Sage Publications Inc, Thousand Oaks, Calif.Google Scholar
Dharma Dailey and Kate Starbird. 2017. Social Media Seamsters: Stitching Platforms & Audiences into Local Crisis Infrastructure. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 1277--1289. DOI:https://doi.org/10.1145/2998181.2998290Google ScholarDigital Library
Paul Dourish. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), 541--550. DOI:https://doi.org/10.1145/1124772.1124855Google ScholarDigital Library
Paul Dourish. 2014. Reading and Interpreting Ethnography. In Ways of Knowing in HCI (Judith S Olson and Wendy A. Kellogg (Eds)). Springer-Verlag, New York, 1--23.Google Scholar
Ellen A. Drost. 2011. Validity and Reliability in Social Science Research. Education Research and Perspectives 38, 1 (June 2011), 105--123.Google Scholar
Robert Elliott, Constance T. Fischer, and David L. Rennie. 1999. Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology 38, 3 (September 1999), 215--229.Google ScholarCross Ref
Robert M. Emerson, Rachel I. Fretz, and Linda L. Shaw. 2011. Writing Ethnographic Fieldnotes, Second Edition (Second edition ed.). University of Chicago Press, Chicago.Google Scholar
Casey Fiesler, Shannon Morrison, and Amy S Bruckman. 2016. An archive of their own: a case study of feminist HCI and values in design. 2574--2585.Google Scholar
Megan Finn and Elisa Oreglia. 2016. A Fundamentally Confused Document: Situation Reports and the Work of Producing Humanitarian Information. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 1349--1362. DOI:https://doi.org/10.1145/2818048.2820031Google ScholarDigital Library
Katie Z. Gach, Casey Fiesler, and Jed R. Brubaker. 2017. "Control Your Emotions, Potter": An Analysis of Grief Policing on Facebook in Response to Celebrity Death. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 47:1--47:18. DOI:https://doi.org/10.1145/3134682Google Scholar
C. Geertz. 1983. Thick Description: toward an interpretive theory of culture. In The Interpretation of Cultures. Basic Books, New York, 3--32.Google Scholar
Kathrin Gerling, Kieran Hicks, Michael Kalyn, Adam Evans, and Conor Linehan. 2016. Designing Movement-based Play With Young People Using Powered Wheelchairs. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4447--4458. DOI:https://doi.org/10.1145/2858036.2858070Google ScholarDigital Library
Lisa Given. 2008. Natural Setting. In The Sage encyclopedia of qualitative research methods. SAGE, London.Google Scholar
Barney Glaser and Anselm Strauss. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Transaction.Google Scholar
Daniel Gooch, Asimina Vasalou, Laura Benton, and Rilla Khaled. 2016. Using Gamification to Motivate Students with Dyslexia. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 969--980. DOI:https://doi.org/10.1145/2858036.2858231Google ScholarDigital Library
Beki Grinter. 2010. Inter-Rater Reliability. Beki's Blog. Retrieved January 29, 2019 from https://beki70.wordpress.com/2010/09/09/inter-rater-reliability-apply-with-care/Google Scholar
Egon G. Guba. 1981. Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communication and Technology Journal 29, (1981), 75--91.Google ScholarCross Ref
Kevin A. Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 8, 1 (2012), 23--34.Google ScholarCross Ref
David Hammer and Leema K. Berland. 2014. Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning. Journal of the Learning Sciences 23, 1 (January 2014), 37--46.Google ScholarCross Ref
Hanna Hasselqvist, Mia Hesselgren, and Cristian Bogdan. 2016. Challenging the Car Norm: Opportunities for ICT to Support Sustainable Transportation Practices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1300--1311. DOI:https://doi.org/10.1145/2858036.2858468Google ScholarDigital Library
John Hughes, Tom Rodden, and Hans Andersen. 1994. Moving out from the control room: ethnography in system design. ACM Conference on Computer-Supported Cooperative Work (1994), 429--439.Google ScholarDigital Library
Azra Ismail, Naveena Karusala, and Neha Kumar. 2018. Bridging Disconnected Knowledges for Community Health. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 75:1--75:27. DOI:https://doi.org/10.1145/3274344Google ScholarDigital Library
Jialun "Aaron" Jiang, Casey Fiesler, and Jed R. Brubaker. 2018. "The Perfect One": Understanding Communication Practices and Challenges with Animated GIFs. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 80:1--80:20. DOI:https://doi.org/10.1145/3274349Google Scholar
Eunice Jun, Blue A. Jo, Nigini Oliveira, and Katharina Reinecke. 2018. Digestif: Promoting Science Communication in Online Experiments. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 84:1--84:26. DOI:https://doi.org/10.1145/3274353Google ScholarDigital Library
Vaishnav Kameswaran, Jatin Gupta, Joyojeet Pal, Sile O'Modhrain, Tiffany C. Veinot, Robin Brewer, Aakanksha Parameshwar, Vidhya Y, and Jacki O'Neill. 2018. "We Can Go Anywhere": Understanding Independence Through a Case Study of Ride-hailing Use by People with Visual Impairments in Metropolitan India. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 85:1--85:24. DOI:https://doi.org/10.1145/3274354Google ScholarDigital Library
Matthew Kay, Steve Haroz, Shion Guha, Pierre Dragicevic, and Chat Wacharamanotham. 2017. Moving Transparent Statistics Forward at CHI. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17), 534--541. DOI:https://doi.org/10.1145/3027063.3027084Google ScholarDigital Library
Christina Kelley, Bongshin Lee, and Lauren Wilcox. 2017. Self-tracking for Mental Wellness: Understanding Expert Perspectives and Student Experiences. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 629--641. DOI:https://doi.org/10.1145/3025453.3025750Google ScholarDigital Library
Ryan Kelly, Daniel Gooch, Bhagyashree Patil, and Leon Watts. 2017. Demanding by Design: Supporting Effortful Communication Practices in Close Personal Relationships. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 70--83. DOI:https://doi.org/10.1145/2998181.2998184Google ScholarDigital Library
Jerome Kirk and Marc L. Miller. 1986. Reliability and validity in qualitative research. Sage Publications, Beverly Hills.Google Scholar
Maximilian Klein, Jinhao Zhao, Jiajun Ni, Isaac Johnson, Benjamin Mako Hill, and Haiyi Zhu. 2017. Quality Standards, Service Orientation, and Power in Airbnb and Couchsurfing. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 58:1--58:21. DOI:https://doi.org/10.1145/3134693Google ScholarDigital Library
Klaus H. Krippendorff. 2003. Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage Publications, Inc.Google Scholar
Karen S. Kurasaki. 2000. Intercoder Reliability for Validating Conclusions Drawn from Open-Ended Interview Data. Field Methods 12, 3 (August 2000), 179--194.Google ScholarCross Ref
J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174. DOI:https://doi.org/10.2307/2529310Google ScholarCross Ref
Simone Lanette, Phoebe K. Chua, Gillian Hayes, and Melissa Mazmanian. 2018. How Much is "Too Much"?: The Role of a Smartphone Addiction Narrative in Individuals' Experience of Use. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 101:1--101:22. DOI:https://doi.org/10.1145/3274370Google ScholarDigital Library
Amanda Lazar, Hilaire J. Thompson, Shih-Yin Lin, and George Demiris. 2018. Negotiating Relation Work with Telehealth Home Care Companionship Technologies That Support Aging in Place. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 103:1--103:19. DOI:https://doi.org/10.1145/3274372Google ScholarDigital Library
Pierre Le Bras, David A. Robb, Thomas S. Methven, Stefano Padilla, and Mike J. Chantler. 2018. Improving User Confidence in Concept Maps: Exploring Data Driven Explanations. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 404:1--404:13. DOI:https://doi.org/10.1145/3173574.3173978Google Scholar
Margaret LeCompte and Judith Goetz. 1982. Problems of Reliability and Validity in Ethnographic Research. Review of Educational Research 52, 1 (1982), 31--60.Google ScholarCross Ref
Yvonna S. Lincoln and Egon G. Guba. 1986. But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation 1986, 30 (1986), 73--84. DOI:https://doi.org/10.1002/ev.1427Google ScholarCross Ref
Matthew Lombard, Jennifer Snyder? Duch, and Cheryl Campanella Bracken. Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability. Human Communication Research 28, 4 , 587--604. DOI:https://doi.org/10.1111/j.1468--2958.2002.tb00826.xGoogle Scholar
Catherine MacPhail, Nomhle Khoza, Laurie Abler, and Meghna Ranganathan. 2016. Process guidelines for establishing Intercoder Reliability in qualitative studies. Qualitative Research 16, 2 (April 2016), 198--212.Google ScholarCross Ref
Megh Marathe and Kentaro Toyama. 2018. Semi-Automated Coding for Qualitative Research: A User-Centered Inquiry and Initial Prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 348:1--348:12. DOI:https://doi.org/10.1145/3173574.3173922Google ScholarDigital Library
Joe Marshall, Conor Linehan, and Adrian Hazzard. 2016. Designing Brutal Multiplayer Video Games. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 2669--2680. DOI:https://doi.org/10.1145/2858036.2858080Google ScholarDigital Library
Roberto Martinez-Maldonado, Lucila Carvalho, and Peter Goodyear. 2018. Collaborative Design-in-use: An Instrumental Genesis Lens in Multi-device Environments. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 118:1--118:24. DOI:https://doi.org/10.1145/3274387Google ScholarDigital Library
Alice Marwick, Claire Fontaine, and danah boyd. 2017. "Nobody Sees It, Nobody Gets Mad": Social Media, Privacy, and Personal Responsibility Among Low-SES Youth. Social Media + Society 3, 2 (April 2017), 2056305117710455.Google Scholar
Marius Mikalsen and Eric Monteiro. 2018. Data Handling in Knowledge Infrastructures: A Case Study from Oil Exploration. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 123:1--123:16. DOI:https://doi.org/10.1145/3274392Google ScholarDigital Library
D. R. Millen. 2000. Rapid ethnography: time deepening strategies for HCI field research. 280--286.Google Scholar
Trine Møller. 2018. Presenting The Accessory Approach: A Start-up's Journey Towards Designing An Engaging Fall Detection Device. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 559:1--559:10. DOI:https://doi.org/10.1145/3173574.3174133Google ScholarDigital Library
M. J. Muller and S. Kogan. 2010. Grounded theory method in hci and cscw. In Technical Report. IBM Watson Research Center.Google Scholar
Aisling Ann O'Kane, Yvonne Rogers, and Ann E Blandford. 2014. Gaining empathy for non-routine mobile device use through autoethnography. 987--990.Google Scholar
Judith S. Olson and Wendy A. Kellogg (Eds.). 2014. Ways of Knowing in HCI. Springer-Verlag, New York.Google Scholar
Janne Paavilainen, Hannu Korhonen, Kati Alha, Jaakko Stenros, Elina Koskinen, and Frans Mayra. 2017. The PokéMon GO Experience: A Location-Based Augmented Reality Mobile Game Goes Mainstream. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 2493--2498. DOI:https://doi.org/10.1145/3025453.3025871Google ScholarDigital Library
Chanda Phelan, Cliff Lampe, and Paul Resnick. 2016. It's Creepy, But It Doesn't Bother Me. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 5240--5251. DOI:https://doi.org/10.1145/2858036.2858381Google ScholarDigital Library
Laura R. Pina, Carmen Gonzalez, Carolina Nieto, Wendy Roldan, Edgar Onofre, and Jason C. Yip. 2018. How Latino Children in the U.S. Engage in Collaborative Online Information Problem Solving with Their Families. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 140:1--140:26. DOI:https://doi.org/10.1145/3274409Google Scholar
David Pinelle and Carl Gutwin. 2000. A Review of Groupware Evaluations. In Proceedings of the 9th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE '00), 86--91.Google ScholarDigital Library
MD A. Rahman. 2008. Some Trends in the Praxis of Participatory Action Research. In The SAGE Handbook of Action Research (P. Reason and H. Bradbury (eds)). Sage, London, 49--62.Google Scholar
Shriti Raj, Mark W. Newman, Joyce M. Lee, and Mark S. Ackerman. 2017. Understanding Individual and Collaborative Problem-Solving with Patient-Generated Data: Challenges and Opportunities. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 88:1--88:18. DOI:https://doi.org/10.1145/3134723Google ScholarDigital Library
Amon Rapp. 2018. Gamification for Self-Tracking: From World of Warcraft to the Design of Personal Informatics Systems. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 80:1--80:15. DOI:https://doi.org/10.1145/3173574.3173654Google ScholarDigital Library
Marén Schorch, Lin Wan, David William Randall, and Volker Wulf. 2016. Designing for Those Who Are Overlooked: Insider Perspectives on Care Practices and Cooperative Work of Elderly Informal Caregivers. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 787--799. DOI:https://doi.org/10.1145/2818048.2819999Google ScholarDigital Library
Alfred Schutz. 1967. The Phenomenology of the Social World. Northwestern University Press.Google Scholar
Kate Starbird, Emma Spiro, Isabelle Edwards, Kaitlyn Zhou, Jim Maddock, and Sindhuja Narasimhan. 2016. Could This Be True?: I Think So! Expressed Uncertainty in Online Rumoring. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 360--371. DOI:https://doi.org/10.1145/2858036.2858551Google ScholarDigital Library
Anselm Strauss. 1987. Qualitative Analysis for Social Scientists. Cambridge University Press, Cambridge.Google Scholar
Aaron Tabor, Scott Bateman, Erik Scheme, David R. Flatla, and Kathrin Gerling. 2017. Designing Game-Based Myoelectric Prosthesis Training. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1352--1363. DOI:https://doi.org/10.1145/3025453.3025676Google ScholarDigital Library
Matthieu Tixier and Myriam Lewkowicz. 2016. "Counting on the Group": Reconciling Online and Offline Social Support Among Older Informal Caregivers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 3545--3558. DOI:https://doi.org/10.1145/2858036.2858477Google ScholarDigital Library
James R. Wallace, Saba Oji, and Craig Anslow. 2017. Technologies, Methods, and Values: Changes in Empirical Research at CSCW 1990 - 2015. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 106:1--106:18. DOI:https://doi.org/10.1145/3134741Google ScholarDigital Library
April Y. Wang, Ryan Mitts, Philip J. Guo, and Parmit K. Chilana. 2018. Mismatch of Expectations: How Modern Learning Resources Fail Conversational Programmers. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 511:1--511:13. DOI:https://doi.org/10.1145/3173574.3174085Google Scholar
G. Cochran William. 1977. Sampling Techniques, 3rd Edition (3rd edition ed.). John Wiley & Sons, New York.Google Scholar
Lillian Yang and Carman Neustaedter. 2018. Our House: Living Long Distance with a Telepresence Robot. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 190:1--190:18. DOI:https://doi.org/10.1145/3274459Google ScholarDigital Library
Svetlana Yarosh, Elizabeth Bonsignore, Sarah McRoberts, and Tamara Peyton. 2016. YouthTube: Youth Video Authorship on YouTube and Vine. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 1423--1437. DOI:https://doi.org/10.1145/2818048.2819961Google ScholarDigital Library
Svetlana Yarosh, Sarita Schoenebeck, Shreya Kothaneth, and Elizabeth Bales. 2016. "Best of Both Worlds": Opportunities for Technology in Cross-Cultural Parenting. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 635--647. DOI:https://doi.org/10.1145/2858036.2858210Google ScholarDigital Library
Svetlana Yarosh and Pamela Zave. 2017. Locked or Not?: Mental Models of IoT Feature Interaction. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 2993--2997. DOI:https://doi.org/10.1145/3025453.3025617Google ScholarDigital Library
Jason C. Yip, Tamara Clegg, June Ahn, Judith Odili Uchidiuno, Elizabeth Bonsignore, Austin Beck, Daniel Pauw, and Kelly Mills. 2016. The Evolution of Engagements and Social Bonds During Child-Parent Co-design. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 3607--3619.Google ScholarDigital Library

Index Terms

Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. HCI theory, concepts and models

Recommendations

Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering
Abstract Context:
The qualitative research on empirical software engineering that uses Grounded Theory is increasing (GT). The trustworthiness, rigor, and transparency of GT qualitative data analysis can benefit, among others, when ...
Highlights
- Qualitative research on empirical SE that uses Grounded Theory (GT) is increasing.
Read More
Testing the reliability of inter-rater reliability
LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge

Analyses of learning often rely on coded data. One important aspect of coding is establishing reliability. Previous research has shown that the common approach for establishing coding reliability is seriously flawed in that it produces unacceptably high ...
Read More
Inter-rater reliability and validity of peer reviews in an interdisciplinary field

Peer review is an integral part of science. Devised to ensure and enhance the quality of scientific work, it is a crucial step that influences the publication of papers, the provision of grants and, as a consequence, the career of scientists. In order ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
Proceedings of the ACM on Human-Computer Interaction Volume 3, Issue CSCW
November 2019
5026 pages
EISSN:2573-0142
DOI:10.1145/3371885
Editors:
Airi Lampinen
Stockholm University, Sweden
,
Darren Gergle
Northwestern University, USA
,
David A. Shamma
FXPAL, USA
Issue’s Table of Contents
Copyright © 2019 ACM
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 7 November 2019
Published in pacmhci Volume 3, Issue CSCW

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
IRR
content analysis
inter-rater reliability
interviews
qualitative methods
Qualifiers
- research-article
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 487
  Total Citations
  View Citations
- 18,361
  Total Downloads
- Downloads (Last 12 months)6,892
- Downloads (Last 6 weeks)889
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice

Proceedings of the ACM on Human-Computer Interaction

Abstract

References

Cited By

Index Terms

Recommendations

Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering

Testing the reliability of inter-rater reliability

Inter-rater reliability and validity of peer reviews in an interdisciplinary field