Validating Virginia's quality rating and improvement system among state-funded pre-kindergarten programs

https://doi.org/10.1016/j.ecresq.2014.03.004Get rights and content

Highlights

  • We tested the relation among pre-kindergarten program quality ratings from Virginia's Quality Rating and Improvement System (QRIS) and children's growth in pre-literacy skills.

  • Higher quality pre-k program ratings were associated with sharper growth in children's pre-literacy skills in the preschool year compared to lower pre-k program ratings.

  • Pre-k programs’ quality ratings were not associated with children's pre-literacy performance by the time children were in kindergarten.

Abstract

Early childhood education programs’ quality ratings are the key output of Quality Rating and Improvement Systems (QRIS), yet there is little empirical evidence as to whether these ratings relate to child outcomes. The present study examines associations between state-funded targeted pre-kindergarten programs’ ratings in Virginia's QRIS and children's growth in pre-literacy skills through kindergarten (n = 2448 children in 71 pre-k programs). Children in higher-rated pre-kindergarten programs had sharper literacy growth in the preschool year compared to children in lower-rated pre-kindergarten programs. There was no difference in children's growth from the spring of preschool to the fall of kindergarten or during the kindergarten year as a function of pre-kindergarten programs’ ratings. Implications of these findings are discussed toward understanding the potential of QRIS to assess quality in pre-kindergarten programs.

Section snippets

Background on Virginia's QRIS: The Virginia Star Quality Initiative (VSQI)

Virginia's QRIS, the Virginia Star Quality Initiative, offers a useful setting for QRIS validation work as it defines ratings using quality standards with strong empirical support, uses a unique rating structure to create star ratings, and is a relative newcomer to the QRIS landscape. Piloted in 2007 and officially implemented in 2008, the main goal of the VSQI is to provide a consistent way to distinguish the level of quality in early education programs within the Commonwealth of Virginia.

Validation of QRIS rating system and quality standards

There are many potential outcomes of interest of QRIS, such as providing a quality framework, unifying early childhood education programs under one system, or promoting professionalization of the early educator workforce. This paper focuses on the ways in which QRIS ratings relate to children's development, building off the assumption that child outcomes are a key outcome of QRIS.

Past research provides little evidence on whether we may expect that the aggregate star ratings relate to child

Using quality standards in a QRIS context

Although there is a general consensus that certain quality standards relate to child outcomes, albeit often modestly, there is little understanding of whether there are certain points in which the magnitude of associations are greater, or points in which associations begin to level off either at the low or high end of the quality distribution (Burchinal, Vandergrift, Pianta, & Mashburn, 2010). This lack of evidence is concerning given that QRIS apply cut-points to individual quality standards

A focus on pre-kindergarten programs

In the current study, we only focus on children who attended Virginia's state-funded targeted pre-kindergarten program, the Virginia Preschool Initiative. Virginia's targeted pre-kindergarten program provides education to at-risk children who are not served by another program (e.g., Head Start). A child is eligible to participate in the pre-kindergarten if they have at least one demographic risk, including if the child lives in poverty, is homeless, comes from a family with low levels of

The current study

The rapid proliferation of QRIS has led to a need for empirical evaluation of how QRIS operate within a potentially high stakes policy context. The QRIS ratings are the key output of QRIS, yet there is very little empirical evidence on whether states’ rating structures capture unique aspects of quality that relate to children's outcomes. The present study validates Virginia's pilot QRIS among children who attended state-funded, targeted pre-kindergarten programs. We capitalize on data collected

Sample description

Analyses draw upon a database constructed from the following datasets: (1) child care facility information from the Virginia Department of Social Services; (2) block group census data from the 2000 U.S. Census; (3) quality standard scores and star ratings on the VSQI from 2007 to 2009; and (4) pre-kindergarten and kindergarten performance on the Phonological Awareness Literacy Screening (PALS), and center and child characteristics from the Virginia Department of Education and the University of

star quality ratings

The VSQI assessed center quality on four performance standards: (1) staff education and qualifications; (2) teacher-child interactions; (3) structure (i.e., staff-to-child ratio); and (4) environment and instruction. Star Quality Raters assessed classrooms through documentation and observations. Star Quality Raters were extensively trained and were tested for inter-rater reliability once every seven visits. In 2007–08 (VSQI Year 1), Star Raters observed one out of every three classrooms. In

Analytic plan

A central tenet to developmental theory is that individual children's development is strikingly heterogeneous across time. Examining the association between child care quality and children's performance using two time points (e.g., predicting fall kindergarten performance while controlling for fall pre-kindergarten performance) does not account for the variability in growth within individual children. Indeed, there is substantial evidence that accounting for within-child variability yields a

Differences between center characteristics

Table 1 presents comparisons of center characteristics among 2-star, 3-star, and 4-star pre-k programs. One-way between-subjects ANOVAs indicated a number of differences between 2-star, 3-star, and 4-star pre-k programs with regard to pre-k programs’ neighborhood characteristics. Two-star programs tended to be located in areas with more African American residents and more single-mother households compared to 3-star and 4-star programs. Four-star programs, and to a lesser extent 3-star programs,

Discussion

Quality Rating and Improvement Systems are an increasingly popular policy approach to assess and improve the quality of early childhood education within a state context. Quality ratings are the central tenant of the QRIS; parental selection, improvement supports and financial incentives are posited to have strong ties to programs’ performance in the rating system. Yet, there is little empirical evidence on whether the complex weighting and aggregation structure used to derive programs’ ratings

Future directions

Although we find some traction that pre-k star ratings relate to growth in the pre-kindergarten year, these relations fade out relatively quickly. The central question remains whether there is a better way to combine and weight the quality standards that predict development across early and later schooling. The aggregation technique in the VSQI, where multiple indicators of quality are summed together, may explain part of the reason that relation among VSQI ratings and outcomes fades over time.

Conclusion

Quality Rating and Improvement Systems have dramatically expanded over the last decade in response to the growing need to improve the quality of early childhood education programs and outcomes for children. Despite the strong theoretical model for QRIS, there is little descriptive understanding of the issues related to QRIS or the alignment of QRIS ratings to empirical evidence of their effects. The evidence that higher star ratings are related to stronger growth in children's pre-literacy

Acknowledgments

The authors are grateful for the guidance and support of this study provided by the Virginia Early Childhood Foundation, the Virginia Office of Early Childhood Development, and the Virginia Department of Education. We would like to acknowledge Deborah Jonas, Zelda Boyd, Maria Brown, Kathy Glazer, Jason Downer, Andrew Mashburn, Amanda Williford, and Yu Cao for their contributions to our work. The research reported here is supported by the Child Care Research Scholars Grant, Administration for

References (59)

  • W.S. Barnett et al.

    The effects of state pre-kindergarten programs on young children's school readiness in five states

    (2005)
  • W.S. Barnett et al.

    The state of preschool 2012: State preschool yearbook

    (2012)
  • W.S. Barnett et al.

    The state of preschool 2007: State preschool yearbook

    (2008)
  • D. Bassok

    Do Black and Hispanic children benefit more from preschool? Understanding differences in preschool effects across racial groups

    Child Development

    (2010)
  • M. Burchinal et al.

    How well do our measures of quality predict child outcomes? A meta-analysis and coordinated analysis of data from large-scale studies of early childhood settings

  • M.R. Burchinal et al.

    Threshold analysis of association between child care quality and child outcomes for low-income children in pre-kindergarten programs

    Early Childhood Research Quarterly

    (2010)
  • R. Breen

    Regression models: Censored, sample selected or truncated data

    (1996)
  • D.M. Bryant

    Validating North Carolina's 5-Star child care licensing system

    (2001)
  • G. Camilli et al.

    Meta-analysis of the effects of early education interventions on cognitive and social development

    Teachers College Record

    (2010)
  • J.B. Carlin et al.

    Tools for analyzing multiple imputed datasets

    Stata Journal

    (2003)
  • D.C. Castro et al.

    Promoting language and literacy in young dual language learners: Research, practice and policy

    Child Development Perspectives

    (2011)
  • R. Chetty et al.

    How does your kindergarten classroom affect your earnings? Evidence from Project STAR

    Quarterly Journal of Economics

    (2011)
  • M.M. Chingos

    Class size and student outcomes: Research and policy implications

    Journal of Policy Analysis and Management

    (2013)
  • H. Cooper et al.

    The effects of summer vacation on achievement test scores: A narrative and meta-analytic review

    Review of Educational Research

    (1996)
  • S. Dynarski et al.

    Experimental evidence on the effect of childhood investments on postsecondary attainment and degree completion (No. w17533)

    (2011)
  • C.K. Enders

    Analyzing longitudinal data with missing values

    Rehabilitation Psychology

    (2011)
  • D.R. Entwisle et al.

    Children, schools, and inequality

    (1997)
  • P.A. Gautier et al.

    Selection in a field experiment with voluntary participation

    Journal of Applied Econometrics

    (2012)
  • W.T. Gormley et al.

    The effects of universal pre-k on cognitive development

    Developmental Psychology

    (2005)
  • Cited by (0)

    Research contained herein was conducted using data provided under a contract with the Virginia Department of Education. The content does not necessarily reflect the views or polices of the Virginia Department of Education, the Virginia Board of Education, or the Commonwealth of Virginia. Consequently, the Virginia Department of Education, the Virginia Board of Education, and the Commonwealth of Virginia are not responsible for the research brief's content or any loss suffered due to the use of such content. Moreover, the mention of any trade names, commercial products or organizations in this research brief is not an endorsement of any of these entities by the Virginia Department of Education, the Virginia Board of Education, or the Commonwealth of Virginia.

    View full text