Using Response Times to Model Not-Reached Items due to Time Limits

Abstract

Missing values at the end of a test typically are the result of test takers running out of time and can as such be understood by studying test takers’ working speed. As testing moves to computer-based assessment, response times become available allowing to simulatenously model speed and ability. Integrating research on response time modeling with research on modeling missing responses, we propose using response times to model missing values due to time limits. We identify similarities between approaches used to account for not-reached items (Rose et al. in ETS Res Rep Ser 2010:i–53, 2010) and the speed-accuracy (SA) model for joint modeling of effective speed and effective ability as proposed by van der Linden (Psychometrika 72(3):287–308, 2007). In a simulation, we show (a) that the SA model can recover parameters in the presence of missing values due to time limits and (b) that the response time model, using item-level timing information rather than a count of not-reached items, results in person parameter estimates that differ from missing data IRT models applied to not-reached items. We propose using the SA model to model the missing data process and to use both, ability and speed, to describe the performance of test takers. We illustrate the application of the model in an empirical analysis.

Appendices

Appendix A

Prior settings
Speed-accuracy model	$\varSigma _{P} \sim \textit{IW}_{2+1} \left( {{I}_2 } \right)$
	$\varSigma _{I} \sim \textit{IW}_{2+1} \left( {{I}_2 } \right)$
	$\mu _{b} \sim {N}\left( {0,1000^{2}} \right)$
	$\mu _{\beta } \sim {N}\left( {1,1000^{2}} \right)$
	$\alpha ^{2}\sim \varGamma \left( {0.01,0.001} \right)$
Manifest missing response model	$\sigma _{\theta }^2 \sim \textit{IG}\left( {0.01,0.001} \right)$
	$\gamma \sim {N}\left( {0,1000^{2}} \right)$
	$\mu _{\beta } \sim {N}\left( {1,1000^{2}} \right)$
	$\sigma _{\beta }^2 \sim \textit{IG}\left( {0.01,0.001} \right)$

1. $IW_{2+1}(.):$ inverse Wishart prior with 2+1 degrees of freedom; N(., .) :  normal prior; IG(.,.): inverse gamma prior; $\Gamma \left( {.,.} \right)$: gamma prior; ${I}_2$ represents and identity matrix of size 2.

Appendix B

1.1 Difference in Speed Estimates

(see Figs. 14, 15, 16, 17.)

Fig. 14

Difference in speed estimates using the SA model for complete data compared to the true speed values as a function of true ability. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color, with darker colors denoting a higher number of not-reached items.

Fig. 15

Difference in speed estimates using the SA model for complete data compared to the true speed values as a function of true speed. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color, with darker colors denoting a higher number of not-reached items.

Fig. 16

Difference in speed estimates between the SA model for incomplete data (SAinc) and the SA model for complete data (SAcomp) as a function of true ability. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color, with darker colors denoting a higher number of not-reached items.

Fig. 17

Difference in speed estimates between the SA model for incomplete data (SAinc) and the SA model for complete data (SAcomp) as a function of true speed. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color, with darker colors denoting a higher number of not-reached items.

Appendix C

1.1 Subsequent Analyses

(see Figs. 18, 19.)

Fig. 18

Difference in ability estimates using the SA model for complete data (SAcomp) compared to the true ability values as a function of true ability. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color, with darker colors denoting a higher number of not-reached items.

Fig. 19

Difference in ability estimates between using the SA model for incomplete data (SAinc) and using the SA model for complete data (SAcomp) as a function of true ability. White circles represent simulees without missing values and filled circles persons with missing values. The number of missing values is given by the circles’ color with darker colors denoting a higher number of not-reached items.