Revisiting methodological issues in transcript analysis: Negotiated coding and reliability
Introduction
Collaborative asynchronous online communication is becoming a common and valuable technique to engage and sustain discourse in higher education. However, online (text-based) communication is a rigorous and not fully understood educational medium, especially when combined with spontaneous face-to-face verbal dialogue (i.e., blended learning). To date, there is barely beyond an intuitive understanding of the properties and potentiality for online learning. Much work remains in understanding the adoption of communications technology for educational purposes. Due to the complexity of educational discourse, it is imperative to have the techniques and tools to study this educational communication medium.
In this regard, researchers need methodologies to systematically and rigorously study online communication in order to guide educators in the adoption and design of learning in this new environment. The permanent nature of online asynchronous text-based transcripts provides an accessible source to study the complexities of the teaching and learning transaction in this environment. Educators are just beginning a systematic study of how to facilitate an effective and efficient online educational experience. One of the methodologies with considerable potential to explore the complexities of online learning is transcript analysis.
Transcript analysis provides a relatively new observational technique to understanding educational discourse in an online text-based environment. The theoretical frameworks and coding schemes/protocols of transcript analysis need to be critically evaluated. Gerbic and Stacey (2005) suggest that it is time that researchers evaluate analytic frameworks in transcript analysis as they “have not been substantively discussed in the literature” (p. 46).
The purpose of this research is to revisit transcript content analysis. In particular, the authors focus on the methodological procedures and coding experiences of a study that used a negotiated approach to transcript analysis. The goal is to provide a guide to researchers in training coders, maintain opportunity for insights, and achieving acceptable reliability measures. The paper begins with a review of the methodological issues (theoretical frameworks and analytical techniques). Reliability results of a recent study are provided next. Finally, a discussion of the advantages and limitations of a negotiated approach to transcript analysis is provided.
Section snippets
Methodological issues
Reliability and validity issues have their source in the theoretical frameworks, models and coding schemes designed to guide the analysis of transcripts. A sound theoretical framework and model is essential to address validity issues. Coding schemes must be both effective from a reliability perspective and efficient from a resource perspective. A relatively straightforward coding scheme can be of great assistance in the training of coders and the consistent application of the protocol. The
Theoretical framework
The Community of Inquiry (CoI) framework was developed by Garrison, Anderson, and Archer (2000) to guide transcript analysis. The framework had its genesis in the higher education literature, particularly Lipman's (1991) work on communities of inquiry. The framework is composed of three elements — social, cognitive and teaching presence. Each of the three elements was also grounded in the higher education literature. Categories of social presence were generated from an extensive review of the
Methodological procedures
At its source, transcript analysis is an exploratory, qualitative methodology. As suggested previously, transcript analysis falls into the qualitative research tradition. The data (text) and analysis (selective coding with constant comparison) are clearly consistent with qualitative analytic practices. However, this is not inductive theory building research. The distinguishing feature of this approach to transcript analysis is that we begin with a credible theoretical framework. As such, the
Transcript analysis
At the beginning of the coding process, the coders received a copy of the Communities of Inquiry (CoI) coding scheme and a set of transcripts from the first classes sampled in the study (The coding scheme is presented in Table 1 and has been described previously under theoretical framework.). At a preliminary meeting with a graduate student who served as a coding trainer, the coders began to code and reconcile a series of online course transcripts. This process helped them to further understand
Results of negotiated transcript analysis
Reliability in the data collection and analysis process is a cornerstone of sound research. Analysis of qualitative, textual data produced by research respondents requires multiple layers of what Giddens (1984) refers to as double hermeneutics: the subjective interpretation of expressed responses relating to concrete experience in the physical world. Raters or coders apply conceptual codes to these expressions, based on their interpretation of that expression. Inter-rater reliability is the
Conclusion
The purpose of this paper was to review, clarify and offer suggestions to enhance the methodology associated with transcript analysis. The approach described here was to frame the study and analysis within a credible theoretical framework. It is argued that this approach provides increased rigor in coding the transcript. In particular, the paper focused on the challenges of reliably coding transcripts and the advantage that a negotiated approach provides. Practical suggestions were provided to
References (26)
- et al.
A purposive approach to content analysis: Designing analytical frameworks
Internet and Higher Education
(2005) - et al.
Creating cognitive presence in a blended faculty development community
Internet and Higher Education
(2005) - et al.
Assessing teaching presence in a computer conferencing context
Journal of Asynchronous Learning Networks
(2001) - Celetin, P. (unpublished manuscript). Online training: Analysis of interaction and knowldge building patterns among...
- et al.
Sharpening the focus: Methodological issues in analyzing on-line conferences
Technology, Pedagogy and Education
(2003) How we think
(1933)- et al.
Validity in educational research
Addressing some common problems in transcript analysis
International Review of research in Open and Distance Learning
(2001)- et al.
Patterns of interaction in a computer conference transcript
International Review of research in Open and Distance Learning
(2001) - Flynn, T., & Polin, L. (2003). Making sense of online learning: Frames, rubrics, tools and coding systems for analyzing...
E-Learning in the 21st century: A framework for research and practice
Critical inquiry in a text-based environment: Computer conferencing in higher education
The Internet and Higher Education
Critical thinking, cognitive presence and computer conferencing in distance education
American Journal of Distance Education
Cited by (334)
IT managers’ perspective on Technical Debt Management
2023, Journal of Systems and SoftwareAutomated Discourse Analysis via Generative Artificial Intelligence
2024, ACM International Conference Proceeding SeriesCharacterizing Software Maintenance Meetings: Information Shared, Discussion Outcomes, and Information Captured
2024, Proceedings - International Conference on Software EngineeringTeachers’ and students’ perceptions of a sense of community in blended education
2024, Education and Information Technologies