Measurement Invariance of a Universal Behavioral Screener Across Samples From the USA and Germany
Abstract
Abstract. The current study examines the item and scalar equivalence of an abbreviated school-based universal screener that was cross-culturally translated and adapted from English into German. The instrument was designed to assess student behavior problems that impact classroom learning. Participants were 1,346 K-6 grade students from the US (n = 390, Mage = 9.23, 38.5% female) and Germany (n = 956, Mage = 8.04, 40.1% female). Measurement invariance was tested by multigroup confirmatory factor analysis (CFA) across students from the US and Germany. Results support full scalar invariance between students from the US and Germany (df = 266, χ2 = 790.141, Δχ2 = 6.9, p < .001, CFI = 0.976, ΔCFI = 0.000, RMSEA = 0.052, ΔRMSEA = −0.003) indicating that the factor structure, the factor loadings, and the item thresholds are comparable across samples. This finding implies that a full cross-cultural comparison including latent factor means and structural coefficients between the US and the German version of the abbreviated screener is possible. Therefore, the tool can be used in German schools as well as for cross-cultural research purposes between the US and Germany.
References
2011). Manual for the ASEBA brief problem monitor (BPM). Burlington, VT: University of Vermont. Retrieved from http://www.aseba.org/ASEBA%20Brief%20Problem%20Monitor%20Manual.pdf
(2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203. https://doi.org/10.1207/s15328007sem1302
(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. https://doi.org/10.1037/0033-2909.107.2.238
(2016). The many faces of special education within RTI frameworks in the United States and Finland. Learning Disability Quarterly, 39, 58–66. https://doi.org/10.1177/073194871559478
(2015). Conducting measurement invariance tests with ordinal data: A guide for social work researchers. Journal of the Society for Social Work and Research, 6, 229–249. https://doi.org/10.1086/681607
(2010). Response to intervention: Principles and strategies for effective practice. Practical intervention in the schools series (2nd ed.). New York, NY: Guilford Press.
(2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95, 1005–1018. https://doi.org/10.1037/a0013193
(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233–255. https://doi.org/10.1207/S15328007SEM0902_5
(2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. https://doi.org/10.1177/1534508409340390
(1952). The chi-square test of goodness of fit. The Annals of Mathematical Statistics, 23, 315–345. https://doi.org/10.1214/aoms/1177729380
(2014). A comparison of teacher perceptions and research-based categories of student behavior difficulties. Education, 134, 439–451.
(2014). Development of a problem-focused behavioral screener linked to evidence-based intervention. School Psychology Quarterly, 29, 438–451. https://doi.org/10.1037/spq0000100
(2017). Classification accuracy and acceptability of the integrated screening and intervention system teacher rating form. School Psychology Quarterly, 32, 212–225. https://doi.org/10.1037/spq0000147
(2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43, 121–149. https://doi.org/10.1177/0748175610373459
(2003). Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. Journal of Advanced Nursing, 42, 57–63. https://doi.org/10.1046/j.1365-2648.2003.02579.x
(2015). Implementing a multi-tiered system of support (MTSS): Collaboration between school psychologists and administrators to promote systems-level change. Journal of Educational and Psychological Consultation, 25, 160–177. https://doi.org/10.1080/10474412.2014.929960
(2009). Early identification of behavioral and emotional problems in youth: Universal screening versus teacher-referral identification. California School Psychologist, 14, 89–95. https://doi.org/10.1007/BF03340954
(2007). Universal and early screening for educational difficulties: Current and future approaches. Journal of School Psychology, 45, 137–161. https://doi.org/0.1016/j.jsp.2006.11.002
(2015).
(Behavioral assessment in school settings . In R. FlanaganK. AllenE. LevineEds., Cognitive and behavioral interventions in the schools: Integrating theory and research into practice (pp. 15–41). New York, NY: Springer Science + Business Media.2004). Alternative approaches to the definition and identification of learning disabilities: Some questions and answers. Annals of Dyslexia, 54, 304–331. https://doi.org/10.1007/s11881-004-0015-y
(2012). Prevalence of students with EBD: Impact on general education. Beyond Behavior, 21, 3–10.
(2013). Characteristics of students with emotional disturbance manifesting internalizing behaviors: A latent class analysis. Education and Treatment of Children, 36, 127–145. https://doi.org/10.1353/etc.2013.0038
(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. https://doi.org/10.1016/j.jsp.2006.05.005
(2004). Current status and future directions of school-based behavioral interventions. School Psychology Review, 33, 326–343.
(2007).
(Evolution of the response-to-intervention concept: Empirical foundations and recent developments . In S. R. JimersonM. K. BurnsA. M. VanDerHeydenEds., Handbook of response to intervention (pp. 10–24). Boston, MA: Springer US.2013). Response-to-intervention (RTI) as a model to facilitate inclusion for students with learning and behaviour problems. European Journal of Special Needs Education, 28, 254–269. https://doi.org/10.1080/08856257.2013.768452
(2005).
(Translation and adaptation issues and methods for educational and psychological tests . In C. L. FrisbyC. R. ReynoldsEds., Comprehensive handbook of multicultural school psychology (pp. 881–903). Hoboken, NJ: Wiley.2008). Response to intervention for social behavior: Challenges and opportunities. Journal of Emotional and Behavioral Disorders, 16, 213–225. https://doi.org/10.1177/1063426608316018
(1979). Response-shift bias: A source of contamination of self-report measures. The Journal of Applied Psychology, 64, 144–150. https://doi.org/10.1037/0021-9010.64.2.144
(1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403–424. https://doi.org/10.1037/1082-989X.4.4.403
(1985). Measurement in cross-cultural psychology: A review and comparison of strategies. Journal of Cross-Cultural Psychology, 16, 131–152. https://doi.org/10.1177/0022002185016002001
(2011). Overlooked and underserved: “Action signs” for identifying children with unmet mental health needs. Pediatrics, 128, 970–979. https://doi.org/10.1542/peds.2009-0367
(2016). Handbook of response to intervention: The science and practice of multi-tiered systems of support (2nd ed.). New York, NY: Springer US.
(2011). Child and adolescent mental health worldwide: Evidence for action. The Lancet, 378, 1515–1525. https://doi.org/10.1016/S0140-6736(11)60827-1
(2014). Testing measurement invariance across groups in longitudinal data: Multigroup second-order latent growth model. Structural Equation Modeling: A Multidisciplinary Journal, 21, 566–576. https://doi.org/10.1080/10705511.2014.919821
(2016). Sonderpädagogische Förderung in Schulen 2005 bis 2014
([Special Education in Schools 2005 to 2014] . Retrieved from https://www.kmk.org/fileadmin/Dateien/pdf/Statistik/Dokumentationen/Dok_210_SoPae_2014.pdf2010). A comparison of systematic screening tools for emotional and behavioral disorders: A replication. Journal of Emotional and Behavioral Disorders, 18, 100–112. https://doi.org/10.1177/1063426609341069
(2015).
(The connection between assessment and intervention: How can screening lead to better intervention . In B. BatemanJ. W. LloydM. TankersleyEds., Enduring issues in special education. Personal perspectives (pp. 285–301). New York, NY: Routledge.2007). Umgang mit fehlenden Werten in der psychologischen Forschung
([Handling of missing data in psychological research: Problems and solutions] . Psychologische Rundschau, 58, 103–117. https://doi.org/10.1026/0033-3042.58.2.1032015). Clarifying missing at random and related definitions, and implications when coupled with exchangeability: Table 1. Biometrika, 102, 995–1000. https://doi.org/10.1093/biomet/asv035
(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. https://doi.org/10.1007/BF02294825
(2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus Web Notes, Nr. 4, version 5. Retrieved from http://www.statmodel.com/download/webnotes/CatMGLong.pdf
(1998–2015). Mplus version (version 7) (7th ed.). Los Angeles, CA: Muthén & Muthén.
(2004). The power of outliers (and why researchers should always check for them). Practical Assessment, Research & Evaluation, 9, 1.
(1996). Longitudinal measurement models in evaluation research: Examining stability and change. Evaluation and Program Planning, 19, 333–350. https://doi.org/10.1016/S0149-7189(96)00027-4
(2016). Psychische Kindergesundheit: Ergebnisse der BELLA-Kohortenstudie
([Mental Health in Children and Adolescents: Results of the BELLA Cohort Study] . Kindheit und Entwicklung, 25, 4–9. https://doi.org/10.1026/0942-5403/a0001832015). The longitudinal BELLA study: Design, methods and first results on the course of mental health problems. European Child & Adolescent Psychiatry, 24, 651–663. https://doi.org/10.1007/s00787-014-0638-4
(2008). Empirically derived subtypes of child academic and behavior problems: Co-occurrence and distal outcomes. Journal of Abnormal Child Psychology, 36, 759–770. https://doi.org/10.1007/s10802-007-9208-2
(2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347–363. https://doi.org/10.1177/0734282911406661
(2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21, 167–180. https://doi.org/10.1080/10705511.2014.882658
(2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75, 243–248. https://doi.org/10.1007/s11336-009-9135-y
(2010). Impact of early school-based screening and intervention programs for ADHD on children’s outcomes and access to services: Follow-up of a school-based trial at age 10 years. Archives of Pediatrics & Adolescent Medicine, 164, 462–469. https://doi.org/10.1001/archpediatrics.2010.40
(2011). Equivalence of reading and listening comprehension across test media. Educational and Psychological Measurement, 71, 849–869. https://doi.org/10.1177/0013164410391468
(2015). Überprüfung von Messinvarianz mittels CFA und DIF-Analysen
([Testing for Measurement Invariance in Students with and without Special Educational Needs – A case example using the Short Form of the Illinois Loneliness and Social Satisfaction Scale] . Empirische Sonderpädagogik, 7, 175–193.2015). Assessing special educational needs in Austria: Description of labeling practices and their evolution from 1996 to 2013. Journal of Cognitive Education and Psychology, 14, 329–342. https://doi.org/10.1891/1945-8959.14.3.329
(2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193–223. https://doi.org/10.1016/j.jsp.2006.11.003
(2005). The future of a mistake: Will discrepancy measurement continue to make the learning disabilities field a pseudoscience? Learning Disability Quarterly, 28, 103. https://doi.org/10.2307/1593604
(2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality & Quantity, 43, 599–616. https://doi.org/10.1007/s11135-007-9143-x
(2006). A promising approach for expanding and sustaining school-wide positive behavior support. School Psychology Review, 35, 245–259.
(2007). Using multivariate statistics (3rd ed.). New York, NY: Pearson.
(2004). Translation & validation procedure. Guidelines and documentation form, EC Grant Number: QLG-CT-2000-00751
. (2016). DSM-5® diagnosis in the schools. New York, NY: Guilford Press.
(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. https://doi.org/10.1007/BF02291170
(2016). Digest of education statistics: 2014. Retrieved from https://nces.ed.gov/programs/digest/d14/index.asp
. (1982). Cross-cultural generalization and universality. Journal of Cross-Cultural Psychology, 13, 387–408. https://doi.org/10.1177/0022002182013004001
(2016). Assessing the effects of a school-wide data-based decision-making intervention on student achievement growth in primary schools. American Educational Research Journal, 53, 360–394. https://doi.org/10.3102/0002831216637346
(2010). Linking screening for emotional and behavioral problems to problem-solving efforts: An adaptive model of behavioral assessment. Assessment for Effective Intervention, 35, 240–244. https://doi.org/10.1177/1534508410377194
(A universal screener linked to personalized classroom interventions: Psychometric characteristics in a large sample of German schoolchildren. Journal of School Psychology.
(in press).2013). Daily behavior report cards: An evidence-based system of assessment and intervention. New York, NY: Guilford Press.
(1995). Antisocial behavior in school: Strategies and best practices. Belmont, CA: Thomson Brooks/Cole Publishing.
(2015). A comparison of imputation strategies for ordinal missing data on Likert Scale variables. Multivariate Behavioral Research, 50, 484–503. https://doi.org/10.1080/00273171.2015.1022644
(2013). Lost in translation: Thoughts regarding the translation of existing psychological measures into other languages. European Journal of Psychological Assessment, 29, 81–83. https://doi.org/10.1027/1015-5759/a000167
(