Skip to main content
Original Article

Measurement Invariance of a Universal Behavioral Screener Across Samples From the USA and Germany

Published Online:https://doi.org/10.1027/1015-5759/a000447

Abstract. The current study examines the item and scalar equivalence of an abbreviated school-based universal screener that was cross-culturally translated and adapted from English into German. The instrument was designed to assess student behavior problems that impact classroom learning. Participants were 1,346 K-6 grade students from the US (n = 390, Mage = 9.23, 38.5% female) and Germany (n = 956, Mage = 8.04, 40.1% female). Measurement invariance was tested by multigroup confirmatory factor analysis (CFA) across students from the US and Germany. Results support full scalar invariance between students from the US and Germany (df = 266, χ2 = 790.141, Δχ2 = 6.9, p < .001, CFI = 0.976, ΔCFI = 0.000, RMSEA = 0.052, ΔRMSEA = −0.003) indicating that the factor structure, the factor loadings, and the item thresholds are comparable across samples. This finding implies that a full cross-cultural comparison including latent factor means and structural coefficients between the US and the German version of the abbreviated screener is possible. Therefore, the tool can be used in German schools as well as for cross-cultural research purposes between the US and Germany.

References

  • Achenbach, T. M., McConaughy, S. H., Ivanova, M. Y. & Rescorla, L. A. (2011). Manual for the ASEBA brief problem monitor (BPM). Burlington, VT: University of Vermont. Retrieved from http://www.aseba.org/ASEBA%20Brief%20Problem%20Monitor%20Manual.pdf First citation in articleGoogle Scholar

  • Beauducel, A. & Yorck Herzberg, P. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203. https://doi.org/10.1207/s15328007sem1302 First citation in articleCrossrefGoogle Scholar

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. https://doi.org/10.1037/0033-2909.107.2.238 First citation in articleCrossrefGoogle Scholar

  • Björn, P. M., Aro, M. T., Koponen, T. K., Fuchs, L. S. & Fuchs, D. H. (2016). The many faces of special education within RTI frameworks in the United States and Finland. Learning Disability Quarterly, 39, 58–66. https://doi.org/10.1177/073194871559478 First citation in articleCrossrefGoogle Scholar

  • Bowen, N. K. & Masa, R. D. (2015). Conducting measurement invariance tests with ordinal data: A guide for social work researchers. Journal of the Society for Social Work and Research, 6, 229–249. https://doi.org/10.1086/681607 First citation in articleCrossrefGoogle Scholar

  • Brown-Chidsey, R. & Steege, M. W. (2010). Response to intervention: Principles and strategies for effective practice. Practical intervention in the schools series (2nd ed.). New York, NY: Guilford Press. First citation in articleGoogle Scholar

  • Chen, F. F. (2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95, 1005–1018. https://doi.org/10.1037/a0013193 First citation in articleCrossrefGoogle Scholar

  • Cheung, G. W. & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233–255. https://doi.org/10.1207/S15328007SEM0902_5 First citation in articleCrossrefGoogle Scholar

  • Christ, T. J., Riley-Tillman, T. C. & Chafouleas, S. M. (2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. https://doi.org/10.1177/1534508409340390 First citation in articleCrossrefGoogle Scholar

  • Cochran, W. G. (1952). The chi-square test of goodness of fit. The Annals of Mathematical Statistics, 23, 315–345. https://doi.org/10.1214/aoms/1177729380 First citation in articleCrossrefGoogle Scholar

  • Conley, L., Marchant, M. & Caldarella, P. (2014). A comparison of teacher perceptions and research-based categories of student behavior difficulties. Education, 134, 439–451. First citation in articleGoogle Scholar

  • Daniels, B., Volpe, R. J., Briesch, A. M. & Fabiano, G. A. (2014). Development of a problem-focused behavioral screener linked to evidence-based intervention. School Psychology Quarterly, 29, 438–451. https://doi.org/10.1037/spq0000100 First citation in articleCrossrefGoogle Scholar

  • Daniels, B., Volpe, R. J., Fabiano, G. A. & Briesch, A. M. (2017). Classification accuracy and acceptability of the integrated screening and intervention system teacher rating form. School Psychology Quarterly, 32, 212–225. https://doi.org/10.1037/spq0000147 First citation in articleCrossrefGoogle Scholar

  • Dimitrov, D. M. (2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43, 121–149. https://doi.org/10.1177/0748175610373459 First citation in articleCrossrefGoogle Scholar

  • Drennan, J. (2003). Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. Journal of Advanced Nursing, 42, 57–63. https://doi.org/10.1046/j.1365-2648.2003.02579.x First citation in articleCrossrefGoogle Scholar

  • Eagle, J. W., Dowd-Eagle, S. E., Snyder, A. & Holtzman, E. G. (2015). Implementing a multi-tiered system of support (MTSS): Collaboration between school psychologists and administrators to promote systems-level change. Journal of Educational and Psychological Consultation, 25, 160–177. https://doi.org/10.1080/10474412.2014.929960 First citation in articleCrossrefGoogle Scholar

  • Eklund, K., Renshaw, T. L., Dowdy, E., Jimerson, S. R., Hart, S. R., Jones, C. N. & Earhart, J. (2009). Early identification of behavioral and emotional problems in youth: Universal screening versus teacher-referral identification. California School Psychologist, 14, 89–95. https://doi.org/10.1007/BF03340954 First citation in articleCrossrefGoogle Scholar

  • Elliott, S. N., Huai, N. & Roach, A. T. (2007). Universal and early screening for educational difficulties: Current and future approaches. Journal of School Psychology, 45, 137–161. https://doi.org/0.1016/j.jsp.2006.11.002 First citation in articleCrossrefGoogle Scholar

  • Feindler, E. & Liebman, M. (2015). Behavioral assessment in school settings. In R. FlanaganK. AllenE. LevineEds., Cognitive and behavioral interventions in the schools: Integrating theory and research into practice (pp. 15–41). New York, NY: Springer Science + Business Media. First citation in articleGoogle Scholar

  • Fletcher, J. M., Coulter, W. A., Reschly, D. J. & Vaughn, S. (2004). Alternative approaches to the definition and identification of learning disabilities: Some questions and answers. Annals of Dyslexia, 54, 304–331. https://doi.org/10.1007/s11881-004-0015-y First citation in articleCrossrefGoogle Scholar

  • Forness, S. R., Kim, J. & Walker, H. M. (2012). Prevalence of students with EBD: Impact on general education. Beyond Behavior, 21, 3–10. First citation in articleGoogle Scholar

  • Gage, N. A. (2013). Characteristics of students with emotional disturbance manifesting internalizing behaviors: A latent class analysis. Education and Treatment of Children, 36, 127–145. https://doi.org/10.1353/etc.2013.0038 First citation in articleCrossrefGoogle Scholar

  • Glover, T. A. & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. https://doi.org/10.1016/j.jsp.2006.05.005 First citation in articleCrossrefGoogle Scholar

  • Gresham, F. M. (2004). Current status and future directions of school-based behavioral interventions. School Psychology Review, 33, 326–343. First citation in articleGoogle Scholar

  • Gresham, F. M. (2007). Evolution of the response-to-intervention concept: Empirical foundations and recent developments. In S. R. JimersonM. K. BurnsA. M. VanDerHeydenEds., Handbook of response to intervention (pp. 10–24). Boston, MA: Springer US. First citation in articleGoogle Scholar

  • Grosche, M. & Volpe, R. J. (2013). Response-to-intervention (RTI) as a model to facilitate inclusion for students with learning and behaviour problems. European Journal of Special Needs Education, 28, 254–269. https://doi.org/10.1080/08856257.2013.768452 First citation in articleCrossrefGoogle Scholar

  • Hambleton, R. K. & Li, S. (2005). Translation and adaptation issues and methods for educational and psychological tests. In C. L. FrisbyC. R. ReynoldsEds., Comprehensive handbook of multicultural school psychology (pp. 881–903). Hoboken, NJ: Wiley. First citation in articleGoogle Scholar

  • Hawken, L. S., Vincent, C. G. & Schumann, J. (2008). Response to intervention for social behavior: Challenges and opportunities. Journal of Emotional and Behavioral Disorders, 16, 213–225. https://doi.org/10.1177/1063426608316018 First citation in articleCrossrefGoogle Scholar

  • Howard, G. S. & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. The Journal of Applied Psychology, 64, 144–150. https://doi.org/10.1037/0021-9010.64.2.144 First citation in articleCrossrefGoogle Scholar

  • Hoyt, W. T. & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403–424. https://doi.org/10.1037/1082-989X.4.4.403 First citation in articleCrossrefGoogle Scholar

  • Hui, C. H. & Triandis, H. C. (1985). Measurement in cross-cultural psychology: A review and comparison of strategies. Journal of Cross-Cultural Psychology, 16, 131–152. https://doi.org/10.1177/0022002185016002001 First citation in articleCrossrefGoogle Scholar

  • Jensen, P. S., Goldman, E., Offord, D., Costello, E. J., Friedman, R., Huff, B., … Roberts, R. (2011). Overlooked and underserved: “Action signs” for identifying children with unmet mental health needs. Pediatrics, 128, 970–979. https://doi.org/10.1542/peds.2009-0367 First citation in articleCrossrefGoogle Scholar

  • Jimerson, S. R., Burns, M. K. & VanDerHeyden, A. (2016). Handbook of response to intervention: The science and practice of multi-tiered systems of support (2nd ed.). New York, NY: Springer US. First citation in articleCrossrefGoogle Scholar

  • Kieling, C., Baker-Henningham, H., Belfer, M., Conti, G., Ertem, I., Omigbodun, O., … Rahman, A. (2011). Child and adolescent mental health worldwide: Evidence for action. The Lancet, 378, 1515–1525. https://doi.org/10.1016/S0140-6736(11)60827-1 First citation in articleCrossrefGoogle Scholar

  • Kim, E. S. & Willson, V. L. (2014). Testing measurement invariance across groups in longitudinal data: Multigroup second-order latent growth model. Structural Equation Modeling: A Multidisciplinary Journal, 21, 566–576. https://doi.org/10.1080/10705511.2014.919821 First citation in articleCrossrefGoogle Scholar

  • KMK, S. der S. K. der K. der L. in der B. D (2016). Sonderpädagogische Förderung in Schulen 2005 bis 2014 [Special Education in Schools 2005 to 2014]. Retrieved from https://www.kmk.org/fileadmin/Dateien/pdf/Statistik/Dokumentationen/Dok_210_SoPae_2014.pdf First citation in articleGoogle Scholar

  • Lane, K. L., Robertson Kalberg, J., Lambert, E. W., Crnobori, M. & Bruhn, A. L. (2010). A comparison of systematic screening tools for emotional and behavioral disorders: A replication. Journal of Emotional and Behavioral Disorders, 18, 100–112. https://doi.org/10.1177/1063426609341069 First citation in articleCrossrefGoogle Scholar

  • Lane, K. L. & Walker, H. M. (2015). The connection between assessment and intervention: How can screening lead to better intervention. In B. BatemanJ. W. LloydM. TankersleyEds., Enduring issues in special education. Personal perspectives (pp. 285–301). New York, NY: Routledge. First citation in articleGoogle Scholar

  • Lüdtke, O., Robitzsch, A., Trautwein, U. & Köller, O. (2007). Umgang mit fehlenden Werten in der psychologischen Forschung [Handling of missing data in psychological research: Problems and solutions]. Psychologische Rundschau, 58, 103–117. https://doi.org/10.1026/0033-3042.58.2.103 First citation in articleLinkGoogle Scholar

  • Mealli, F. & Rubin, D. B. (2015). Clarifying missing at random and related definitions, and implications when coupled with exchangeability: Table 1. Biometrika, 102, 995–1000. https://doi.org/10.1093/biomet/asv035 First citation in articleCrossrefGoogle Scholar

  • Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. https://doi.org/10.1007/BF02294825 First citation in articleCrossrefGoogle Scholar

  • Muthén, B. & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus Web Notes, Nr. 4, version 5. Retrieved from http://www.statmodel.com/download/webnotes/CatMGLong.pdf First citation in articleGoogle Scholar

  • Muthén, B. & Muthén, L. (1998–2015). Mplus version (version 7) (7th ed.). Los Angeles, CA: Muthén & Muthén. First citation in articleGoogle Scholar

  • Osborne, J. W. & Overbay, A. (2004). The power of outliers (and why researchers should always check for them). Practical Assessment, Research & Evaluation, 9, 1. First citation in articleGoogle Scholar

  • Pitts, S. C., West, S. G. & Tein, J.-Y. (1996). Longitudinal measurement models in evaluation research: Examining stability and change. Evaluation and Program Planning, 19, 333–350. https://doi.org/10.1016/S0149-7189(96)00027-4 First citation in articleCrossrefGoogle Scholar

  • Ravens-Sieberer, U., Klasen, F. & Petermann, F. (2016). Psychische Kindergesundheit: Ergebnisse der BELLA-Kohortenstudie [Mental Health in Children and Adolescents: Results of the BELLA Cohort Study]. Kindheit und Entwicklung, 25, 4–9. https://doi.org/10.1026/0942-5403/a000183 First citation in articleLinkGoogle Scholar

  • Ravens-Sieberer, U., Otto, C., Kriston, L., Rothenberger, A., Döpfner, M., Herpertz-Dahlmann, B., … Klasen, F. (2015). The longitudinal BELLA study: Design, methods and first results on the course of mental health problems. European Child & Adolescent Psychiatry, 24, 651–663. https://doi.org/10.1007/s00787-014-0638-4 First citation in articleCrossrefGoogle Scholar

  • Reinke, W. M., Herman, K. C., Petras, H. & Ialongo, N. S. (2008). Empirically derived subtypes of child academic and behavior problems: Co-occurrence and distal outcomes. Journal of Abnormal Child Psychology, 36, 759–770. https://doi.org/10.1007/s10802-007-9208-2 First citation in articleCrossrefGoogle Scholar

  • Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347–363. https://doi.org/10.1177/0734282911406661 First citation in articleCrossrefGoogle Scholar

  • Sass, D. A., Schmitt, T. A. & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21, 167–180. https://doi.org/10.1080/10705511.2014.882658 First citation in articleCrossrefGoogle Scholar

  • Satorra, A. & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75, 243–248. https://doi.org/10.1007/s11336-009-9135-y First citation in articleCrossrefGoogle Scholar

  • Sayal, K., Owen, V., White, K., Merrell, C., Tymms, P. & Taylor, E. (2010). Impact of early school-based screening and intervention programs for ADHD on children’s outcomes and access to services: Follow-up of a school-based trial at age 10 years. Archives of Pediatrics & Adolescent Medicine, 164, 462–469. https://doi.org/10.1001/archpediatrics.2010.40 First citation in articleCrossrefGoogle Scholar

  • Schroeders, U. & Wilhelm, O. (2011). Equivalence of reading and listening comprehension across test media. Educational and Psychological Measurement, 71, 849–869. https://doi.org/10.1177/0013164410391468 First citation in articleCrossrefGoogle Scholar

  • Schwab, S. & Helm, C. (2015). Überprüfung von Messinvarianz mittels CFA und DIF-Analysen [Testing for Measurement Invariance in Students with and without Special Educational Needs – A case example using the Short Form of the Illinois Loneliness and Social Satisfaction Scale]. Empirische Sonderpädagogik, 7, 175–193. First citation in articleGoogle Scholar

  • Schwab, S., Hessels, M. G. P., Obendrauf, T., Polanig, M. C. & Wölflingseder, L. (2015). Assessing special educational needs in Austria: Description of labeling practices and their evolution from 1996 to 2013. Journal of Cognitive Education and Psychology, 14, 329–342. https://doi.org/10.1891/1945-8959.14.3.329 First citation in articleCrossrefGoogle Scholar

  • Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R. & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193–223. https://doi.org/10.1016/j.jsp.2006.11.003 First citation in articleCrossrefGoogle Scholar

  • Stanovich, K. E. (2005). The future of a mistake: Will discrepancy measurement continue to make the learning disabilities field a pseudoscience? Learning Disability Quarterly, 28, 103. https://doi.org/10.2307/1593604 First citation in articleCrossrefGoogle Scholar

  • Steinmetz, H., Schmidt, P., Tina-Booh, A., Wieczorek, S. & Schwartz, S. H. (2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality & Quantity, 43, 599–616. https://doi.org/10.1007/s11135-007-9143-x First citation in articleCrossrefGoogle Scholar

  • Sugai, G. & Horner, R. R. (2006). A promising approach for expanding and sustaining school-wide positive behavior support. School Psychology Review, 35, 245–259. First citation in articleGoogle Scholar

  • Tabachnick, B. G. & Fidell, L. S. (2007). Using multivariate statistics (3rd ed.). New York, NY: Pearson. First citation in articleGoogle Scholar

  • The KIDSCREEN Group. (2004). Translation & validation procedure. Guidelines and documentation form, EC Grant Number: QLG-CT-2000-00751 First citation in articleGoogle Scholar

  • Tobin, R. M. & House, A. E. (2016). DSM-5® diagnosis in the schools. New York, NY: Guilford Press. First citation in articleGoogle Scholar

  • Tucker, L. R. & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. https://doi.org/10.1007/BF02291170 First citation in articleCrossrefGoogle Scholar

  • US Department of Education, & National Center for Education Statistics. (2016). Digest of education statistics: 2014. Retrieved from https://nces.ed.gov/programs/digest/d14/index.asp First citation in articleGoogle Scholar

  • Van De Vijver, F. J. R. & Poortinga, Y. H. (1982). Cross-cultural generalization and universality. Journal of Cross-Cultural Psychology, 13, 387–408. https://doi.org/10.1177/0022002182013004001 First citation in articleCrossrefGoogle Scholar

  • van Geel, M., Keuning, T., Visscher, A. J. & Fox, J.-P. (2016). Assessing the effects of a school-wide data-based decision-making intervention on student achievement growth in primary schools. American Educational Research Journal, 53, 360–394. https://doi.org/10.3102/0002831216637346 First citation in articleCrossrefGoogle Scholar

  • Volpe, R. J., Briesch, A. M. & Chafouleas, S. M. (2010). Linking screening for emotional and behavioral problems to problem-solving efforts: An adaptive model of behavioral assessment. Assessment for Effective Intervention, 35, 240–244. https://doi.org/10.1177/1534508410377194 First citation in articleCrossrefGoogle Scholar

  • Volpe, R. J., Casale, G., Mohiyeddini, C., Grosche, M., Hennemann, T., Briesch, A. M. & Daniels, B. (in press). A universal screener linked to personalized classroom interventions: Psychometric characteristics in a large sample of German schoolchildren. Journal of School Psychology. First citation in articleGoogle Scholar

  • Volpe, R. J. & Fabiano, G. A. (2013). Daily behavior report cards: An evidence-based system of assessment and intervention. New York, NY: Guilford Press. First citation in articleGoogle Scholar

  • Walker, H. M., Colvin, G. & Ramsey, E. (1995). Antisocial behavior in school: Strategies and best practices. Belmont, CA: Thomson Brooks/Cole Publishing. First citation in articleGoogle Scholar

  • Wu, W., Jia, F. & Enders, C. (2015). A comparison of imputation strategies for ordinal missing data on Likert Scale variables. Multivariate Behavioral Research, 50, 484–503. https://doi.org/10.1080/00273171.2015.1022644 First citation in articleCrossrefGoogle Scholar

  • Ziegler, M. & Bensch, D. (2013). Lost in translation: Thoughts regarding the translation of existing psychological measures into other languages. European Journal of Psychological Assessment, 29, 81–83. https://doi.org/10.1027/1015-5759/a000167 First citation in articleLinkGoogle Scholar