Revisiting methodological issues in transcript analysis: Negotiated coding and reliability

https://doi.org/10.1016/j.iheduc.2005.11.001Get rights and content

Abstract

Transcript analysis is an important methodology to study asynchronous online educational discourse. The purpose of this study is to revisit reliability and validity issues associated with transcript analysis. The goal is to provide researchers with guidance in coding transcripts. For validity reasons, it is suggested that the first step is to select a sound theoretical model and coding scheme. Particular focus is placed on exploring the advantages of the option of a negotiated approach to coding the transcript. It is concluded that researchers need to consider the advantages of negotiation when coders and researchers are not familiar with the coding scheme.

Introduction

Collaborative asynchronous online communication is becoming a common and valuable technique to engage and sustain discourse in higher education. However, online (text-based) communication is a rigorous and not fully understood educational medium, especially when combined with spontaneous face-to-face verbal dialogue (i.e., blended learning). To date, there is barely beyond an intuitive understanding of the properties and potentiality for online learning. Much work remains in understanding the adoption of communications technology for educational purposes. Due to the complexity of educational discourse, it is imperative to have the techniques and tools to study this educational communication medium.

In this regard, researchers need methodologies to systematically and rigorously study online communication in order to guide educators in the adoption and design of learning in this new environment. The permanent nature of online asynchronous text-based transcripts provides an accessible source to study the complexities of the teaching and learning transaction in this environment. Educators are just beginning a systematic study of how to facilitate an effective and efficient online educational experience. One of the methodologies with considerable potential to explore the complexities of online learning is transcript analysis.

Transcript analysis provides a relatively new observational technique to understanding educational discourse in an online text-based environment. The theoretical frameworks and coding schemes/protocols of transcript analysis need to be critically evaluated. Gerbic and Stacey (2005) suggest that it is time that researchers evaluate analytic frameworks in transcript analysis as they “have not been substantively discussed in the literature” (p. 46).

The purpose of this research is to revisit transcript content analysis. In particular, the authors focus on the methodological procedures and coding experiences of a study that used a negotiated approach to transcript analysis. The goal is to provide a guide to researchers in training coders, maintain opportunity for insights, and achieving acceptable reliability measures. The paper begins with a review of the methodological issues (theoretical frameworks and analytical techniques). Reliability results of a recent study are provided next. Finally, a discussion of the advantages and limitations of a negotiated approach to transcript analysis is provided.

Section snippets

Methodological issues

Reliability and validity issues have their source in the theoretical frameworks, models and coding schemes designed to guide the analysis of transcripts. A sound theoretical framework and model is essential to address validity issues. Coding schemes must be both effective from a reliability perspective and efficient from a resource perspective. A relatively straightforward coding scheme can be of great assistance in the training of coders and the consistent application of the protocol. The

Theoretical framework

The Community of Inquiry (CoI) framework was developed by Garrison, Anderson, and Archer (2000) to guide transcript analysis. The framework had its genesis in the higher education literature, particularly Lipman's (1991) work on communities of inquiry. The framework is composed of three elements — social, cognitive and teaching presence. Each of the three elements was also grounded in the higher education literature. Categories of social presence were generated from an extensive review of the

Methodological procedures

At its source, transcript analysis is an exploratory, qualitative methodology. As suggested previously, transcript analysis falls into the qualitative research tradition. The data (text) and analysis (selective coding with constant comparison) are clearly consistent with qualitative analytic practices. However, this is not inductive theory building research. The distinguishing feature of this approach to transcript analysis is that we begin with a credible theoretical framework. As such, the

Transcript analysis

At the beginning of the coding process, the coders received a copy of the Communities of Inquiry (CoI) coding scheme and a set of transcripts from the first classes sampled in the study (The coding scheme is presented in Table 1 and has been described previously under theoretical framework.). At a preliminary meeting with a graduate student who served as a coding trainer, the coders began to code and reconcile a series of online course transcripts. This process helped them to further understand

Results of negotiated transcript analysis

Reliability in the data collection and analysis process is a cornerstone of sound research. Analysis of qualitative, textual data produced by research respondents requires multiple layers of what Giddens (1984) refers to as double hermeneutics: the subjective interpretation of expressed responses relating to concrete experience in the physical world. Raters or coders apply conceptual codes to these expressions, based on their interpretation of that expression. Inter-rater reliability is the

Conclusion

The purpose of this paper was to review, clarify and offer suggestions to enhance the methodology associated with transcript analysis. The approach described here was to frame the study and analysis within a credible theoretical framework. It is argued that this approach provides increased rigor in coding the transcript. In particular, the paper focused on the challenges of reliably coding transcripts and the advantage that a negotiated approach provides. Practical suggestions were provided to

References (26)

  • P. Gerbic et al.

    A purposive approach to content analysis: Designing analytical frameworks

    Internet and Higher Education

    (2005)
  • N. Vaughan et al.

    Creating cognitive presence in a blended faculty development community

    Internet and Higher Education

    (2005)
  • T. Anderson et al.

    Assessing teaching presence in a computer conferencing context

    Journal of Asynchronous Learning Networks

    (2001)
  • Celetin, P. (unpublished manuscript). Online training: Analysis of interaction and knowldge building patterns among...
  • D. Cook et al.

    Sharpening the focus: Methodological issues in analyzing on-line conferences

    Technology, Pedagogy and Education

    (2003)
  • J. Dewey

    How we think

    (1933)
  • M.A. Eisenhart et al.

    Validity in educational research

  • P.J. Fahy

    Addressing some common problems in transcript analysis

    International Review of research in Open and Distance Learning

    (2001)
  • P.J. Fahy et al.

    Patterns of interaction in a computer conference transcript

    International Review of research in Open and Distance Learning

    (2001)
  • Flynn, T., & Polin, L. (2003). Making sense of online learning: Frames, rubrics, tools and coding systems for analyzing...
  • D.R. Garrison et al.

    E-Learning in the 21st century: A framework for research and practice

    (2003)
  • D.R. Garrison et al.

    Critical inquiry in a text-based environment: Computer conferencing in higher education

    The Internet and Higher Education

    (2000)
  • D.R. Garrison et al.

    Critical thinking, cognitive presence and computer conferencing in distance education

    American Journal of Distance Education

    (2001)
  • Cited by (334)

    • IT managers’ perspective on Technical Debt Management

      2023, Journal of Systems and Software
    • Automated Discourse Analysis via Generative Artificial Intelligence

      2024, ACM International Conference Proceeding Series
    View all citing articles on Scopus
    View full text