Introduction

Computerized data collection methods are increasingly utilized in clinical and population-based research [1, 2], as they provide a potential means of improving accuracy and efficiency of data collection. Computer-adapted measures overcome several limitations associated with paper-and-pencil alternatives. Importantly, computer adaptation may address possible issues of assessment fidelity, allowing administration by a non-specialist workforce and improving standardization of test administration. Direct data entry and data validation can also be implemented, reducing the amount of time and resources invested in data entry, checking and cleaning [3]. The potential advantages of computer-adapted measures are greater for large-scale studies where data entry and checking processes can become highly resource-intensive. With the development of portable technology, computerized assessments can likewise be conducted on tablet devices—a cheaper and more portable alternative to computers [4].

The portability and reduced costs associated with tablet-based data collection methods are particularly beneficial for studies where assessments are conducted remotely and over dispersed geographical regions [5]. Tablet-based data collection also enhances data security through password-protection and immediate data upload, thus minimizing instances of misplaced or damaged raw data from hard-copy data entry forms. This is particularly important for field studies that require researchers to travel frequently and conduct decentralized data collection. Accordingly, as portable tablet devices become increasingly commonplace, the up-front research costs of tablet-based test administration (i.e., cost of tablets, optimization for tablet format) may be offset by reduced printing and study personnel costs associated with secondary data entry and checking procedures. Whilst some assessments require complex adaptation by tablet or Web application developers with considerable programming expertise, other paper-and-pencil based measures may be more readily adapted with the use of existing electronic data capture technology [6].

Tests of reading, in particular, are frequently text-based and therefore may be more easily adapted to a computerized format compared to other cognitive and developmental assessments that may require more advanced design features, such as complex graphics and reaction time (RT) recording (e.g., tests of visuo-spatial reasoning or psychomotor skills). Yet, the issue of mode equivalence between computer-adapted tests and their paper-and-pencil counterparts should still be considered for any newly adapted measure. One index that may be used to evaluate similarity across testing modalities is quantitative equivalence. Quantitative equivalence is defined as the extent to which means and distributions are similar between test administration modes, and whether the norms associated with one mode of a test may be applied to another mode [7, 8]. This may also be considered a test of criterion validity, i.e., the extent to which outcomes from one method of administration are comparable to those from another method [9].

Research examining mode equivalence of paper-and-pencil and computerized assessments of reading in early childhood is limited and has produced mixed results. For instance, a meta-analysis in kindergarten through 12^th grade populations indicated that computer-based versions of reading assessments were generally comparable to their paper-and-pencil counterparts [10]. However, this synthesis of the literature included a broad age range and the vast majority of samples included were of 4^th grade students and older. Evidence is mixed with younger children. In a sample of four- to five-year-olds, Carson, Gillon and Boustead [11] similarly found that scores on a computerized version of a phonological awareness assessment were comparable to the paper-based version. By contrast, in a sample of two- to six-year-olds, Neumann and Neumann [12] found that despite a high level of agreement between test modalities assessing several aspects of early literacy (ICCs = .81 –.94), some tablet-based mean scores were higher than their paper-based counterparts. Finally, in a group of first to sixth-graders, Lenhard, Schroeders and Lenhard [13] found main effects of administration modality at the word-reading level, but not at the sentence level or text level when comparing paper-and-pencil and computerized versions of the ELFE II reading comprehension task. The authors also found a very small interaction effect with child grade across all three reading levels. These findings suggest that factors leading to non-equivalence may be measure- and to some extent age-dependent, highlighting the need for validation of any computer- or tablet-adapted measure.

In the present study we assessed children’s word reading using a tablet adaptation of a well-validated paper-based assessment of reading—the Castles and Coltheart Reading Test 2 [CC2; 14]—within a large-scale longitudinal study. The CC2 is a measure of children’s visual word recognition and sounding-out abilities for children between the ages of 6 and 12 years, and includes 40 regular words (e.g., need, plant), 40 irregular words (e.g., island, friend), and 40 non-words (e.g., pleech, framp). Items are presented to the child by an assessor in a fixed order from easiest to hardest, with items from the regular, irregular, and non-word lists presented in a mixed fashion. The assessor records whether the child’s pronunciation of the word is “correct” or “incorrect”, without providing feedback to the child. Stopping rules are applied for each list separately, such that 5 consecutive errors on any list results in discontinuation of that particular list. Items on the other lists continue to be presented until further discontinue criteria are met or until all trials are completed. The CC2 is often used for clinical purposes, administered by clinicians and teachers.

For research purposes, the CC2 can be administered and scored in paper-and-pencil format. The paper-and-pencil form is administered using a set of cards featuring the 120 test items printed in large 36pt font which are presented to the child sequentially. The reverse side of each card is color-coded to assist assessors when applying the stopping rules. The CC2 can also be administered using an online program, and both versions of the test are freely available from the developers’ website: the Macquarie Online Test Interface (MOTIF: www.motif.org.au). Although the MOTIF online version of the task is well-suited to clinicians and teachers working with individual children, there are limitations to its use in a research setting. Firstly, administration of the online task requires internet connectivity, which may be impractical for certain types of field work research. Secondly, the output of the results can only be exported in portable document format (PDF) and not in a more convenient tabular format such as a comma-separated file (CSV). Researchers are therefore still required to manually enter data following test administration.

In the current study, we sought to address these limitations by developing a computerized version of the CC2 that was suitable for large-scale data collection and could be administered offline using a tablet device. Advantages of the tablet-based version included circumventing the need for manual data entry, and the ability to use branching logic (i.e., automated skipping that customizes the sequence of questions based on participants’ previous responses) to apply discontinue rules. This administration approach reduced the burden on assessors to track errors across separate word lists, thereby ensuring compliance of examiners with testing instructions. Nonetheless, it is important to ensure that any changes in presentation modality of existing measures do not compromise the reliability of the measure or introduce systematic biases to test administration. This paper therefore aims to (i) describe the method by which the CC2 was adapted for administration via tablet, (ii) examine criterion validity (i.e., equivalence of administration method) and reliability of the tablet administered data compared to normative data from the paper-and-pencil based version of the CC2, and (iii) detail the process for assessing and maintaining inter-rater reliability of the CC2 on a large-scale study with non-clinician field staff.

Method

Measure adaptation

With the permission of the authors [14], the CC2 was adapted for use with REDCap [17] hosted at La Trobe University, and administered on an iPad. REDCap is a secure web application for surveys and databases that supports offline and online data entry. To ensure that presentation closely resembled the paper-and-pencil version of the measure, word lists were coded in an 80-pt font (HTML code) so that each word was singularly presented on the tablet screen with participants unable to see prior or subsequent items (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

(a) CC2 card presentation, and (b) tablet-based CC2. Each stimulus occupies the entire screen, and the assessor scrolls down to access the subsequent stimulus.

https://doi.org/10.1371/journal.pone.0239420.g001

Results

Means comparisons with normative data

Summary statistics (means, standard deviations) are presented in Table 1 by age band (7;0–7;5 and 7;6–7;11) for regular, irregular, and non-words for the current sample (tablet-based CC2; n = 569) and a previously published normative sample (paper-based CC2; n = 177) from Castles and colleagues [14]. For both age bands, there were no significant differences in mean scores for our sample receiving the tablet-based administration compared to the normative sample who received the paper-based administration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. T-test results comparing normative with tablet-administered data.

https://doi.org/10.1371/journal.pone.0239420.t001

Discussion

The aims of this paper were to describe the tablet-based adaptation and administration of a standardized developmental measure of reading, report preliminary psychometric findings, and describe the methodological approach to maintaining inter-rater reliability across a large-scale study. Tablet-based administration was adopted to economize on time and monetary costs associated with paper-and-pencil based tasks, and with the additional aim of mitigating the risk for data entry errors. We developed the assessment within existing data capture technology (REDCap), enabling us to automate the required discontinue rules.

To assess quantitative equivalence between the tablet-adapted CC2 and the paper-based measure, we examined z-scores and conducted means comparisons with published norms for children in our sample aged 7;0 to 7;11. Mean z-scores were within the expected range for age band, while raw score means and standard deviations were similar to the normative sample for children. Together, these findings suggest quantitative equivalence between the tablet-based version of the test and the paper-and-pencil version of the CC2, with no evidence of age effects between test modalities for the two age bands we examined. Nonetheless, it is important to note that, although small, age effects on equivalence have been observed for some tests of reading [e.g., 13] and as such these results do not conclusively demonstrate equivalence for age bands not tested.

Internal consistency for our tablet-based version of the CC2 was good to excellent for the regular words and non-words subscales, but poor for irregular words. Only one other study has reported internal consistency for the CC2, with a sample of 8 to 12-year-old children (n = 30) with reading difficulties. Moore and colleagues [20] reported Cronbach’s alphas for the regular (.85) and non-words lists (.85) that were similar to those achieved in our tablet-based administration. However, their Cronbach’s alpha of .94 for the irregular words list was considerably higher than the alpha of .59 we found based on the subsample of 33 children who completed the irregular words list.

While it is possible that the tablet-based administration adversely affected internal consistency for the irregular words list, several other factors may account for this finding. Irregular words are words that lack consistent grapheme-phoneme correspondence rules and can vary greatly in terms of their spelling-to-pronunciation transparency [25]. In contrast to the regular words and non-words which require the child to apply knowledge of spelling-to-sound correspondence rules (a non-lexical process), the correct reading of an irregular word requires the child to be familiar with that word (i.e., stored in a mental lexicon) [26]. In our relatively young sample compared to Moore and colleagues [20], greater internal inconsistency in the irregular word list may reflect a greater degree of guessing by children with less well-developed lexicons.

Differences in sample diversity may also contribute to the differences between the two studies on this word list. For example, children in Moore and colleagues’ study all had poor reading ability, while our participants scored at or slightly higher than the state average for age on the nation-wide reading assessment (NAPLAN), with a distribution around this mean that was typical for age. Around a third of our sample were from households with non-English speaking backgrounds, which would further increase variability in the sample. Young children from multi-language backgrounds may have relatively less developed lexicons as a result of less frequent exposure to English language words at home, compared with their monolingual peers [27, 28]. It is therefore possible that the combination of greater diversity and younger age in our study contributed to poorer internal consistency compared to Moore and colleagues, due to a wider distribution of lower scores, whereas the more representative nature of our sample with respect to reading ability may have resulted in a wider distribution of higher scores. Further research involving larger samples and greater age diversity is required to determine whether the tablet-based administration of the irregular words component is internally valid in its current form.

Finally, inter-rater reliability was examined for the tablet-based format of the CC2, and found to be excellent across the duration of the study, with no evidence of significant rater drift over 31 months of assessments. Inter-rater reliability has not previously been reported for this measure. Our findings suggest that the CC2 can be reliably administered by non-clinicians for research purposes within the defined context in which training was completed prior to data collection, reliability was assessed repeatedly and “refresher” training with feedback was conducted on a regular basis. While this model may be less feasible in clinical practice settings, the benefits are substantial for large-scale research endeavors and justify the resource costs of using this rigorous process.

Our findings should be considered within the context of several limitations. This research was conducted within the context of a larger study that was not primarily established to validate the tablet-based CC2, and as such did not employ the gold standard design for assessing equivalence of test modality (i.e., allocation of children to complete both test modalities in a randomized order). While our findings support the criterion validity of all CC2 subscales and the internal consistency of the regular words and non-words subscales, the narrow age range of our participants precludes generalizability to the full population for whom the CC2 has been designed (i.e., ages 6–12 years). Additionally, while our findings demonstrate support for tablet-based administration of this measure of word reading, this may not apply to other types of reading assessment (e.g., text reading, comprehension test).

Computerized assessment is not without limitations. For instance, potential data loss may occur due to equipment failure, or in some cases, testers may incorrectly press a button and lose the ability to immediately correct the error. Familiarity with tablets and computers and the degree of interaction required should also be considered when assessing young children. Accordingly, while the potential advantages of computer-adapted measures are many for large-scale studies including costs and time savings, they may equally not be appropriate for all studies.

Finally, a notable advantage of electronic test administration is the potential to collect time-related data such as reaction time and total completion time. The CC2 has not previously assessed these features of children’s responding which may shed light on difficulties in processing and articulating written words. The tablet version employed for this study offers a platform for further development and exploration of the clinical utility of time-related data.

In the current study, adaptation of the CC2 was accomplished using existing data capture technology (REDCap) and demonstrated good reliability and similar score distributions compared with normative data, suggesting that successful adaptation of a measure which is psychometrically similar to the paper-and-pencil version can be achieved. We also demonstrated that quality assurance processes can ensure excellent inter-rater reliability despite the use of multiple raters over a lengthy period when rater drift could normally be expected. Frequent training and continued monitoring were essential to this. Our findings support the feasibility and utility of tablet-based assessment as a potential means of improving efficiency and accuracy of data collection for large-scale developmental studies.

Acknowledgments

We thank all participating parents, children and teachers; EHLS at School staff; and the many staff involved with the original EHLS, including the Maternal and Child Health nurses, site coordinators, group facilitators, and home coaches, and staff from the Parenting Research Centre and the Victorian Government. We thank Professor Anne Castles, the developers of the CC2, and Macquarie University (Australia) for granting permission to adapt the CC2 for tablet-based administration in this study. Use of the tablet-adapted CC2 is subject to explicit prior permission from the CC2’s authors.