Observers' reports of self-regulation: Measurement invariance across sex, low-income status, and race/ethnicity

https://doi.org/10.1016/j.appdev.2017.02.001Get rights and content

Highlights

  • The PSRA-AR is a fast, low-cost assessment of self-regulation.

  • Confirmatory factor analyses show a three-factor structure fit the data well.

  • The PSRA-AR shows scalar invariance across sex, low-income status, and race/ethnicity.

  • PSRA-AR factors are correlated with executive function and academic ability.

Abstract

Few assessments of self-regulation for preschool-aged children have been validated for use in socioeconomically and racially/ethnically diverse populations. We address this gap by exploring the validity of the Preschool Self-Regulation Assessment – Assessor Report (PSRA-AR), an observer report of children's global self-regulation ability, using data from a population-based sample drawn from a large and diverse school district. We found that a three factor structure - representing attention regulation, impulse control, and positive engagement - fit the data well for the entire sample. Then, using multigroup confirmatory factor analyses, we found full scalar invariance across sex and low-income status. Scalar invariance held for all but one item across Black, White, and Hispanic children. These findings suggest that the PSRA-AR is valid for use with children of diverse socioeconomic and racial/ethnic backgrounds. Moreover, the PSRA-AR was moderately correlated with direct assessments of executive function and academic ability, including early math and reading.

Introduction

Research on school readiness suggests that self-regulation—which encompasses the ability to control thoughts, behaviors, and emotions—enables young children to successfully transition to formal schooling (Blair, 2002). Better self-regulation ability in preschool is associated with better socioemotional outcomes and academic competence (Blair and Razza, 2007, Diamond, 2016). Given the important role that self-regulation plays in promoting academic competence and rising interest among policy and research communities in measuring it, a critical next step is to ensure that measures of self-regulation demonstrate robust psychometric properties. In this study we aim to take this important empirical step by establishing the psychometric validity for the Preschool Self-Regulation Assessment - Assessor Report (PSRA-AR), an observer rating instrument that measures various aspects of children's self-regulation. Specifically, we will examine the factor structure of the PSRA-AR; test measurement invariance across sex, low-income status, and race/ethnicity; and examine associations with direct assessments of cognitive regulation and academic competence using a large, socioeconomically and racially/ethnically diverse population-based sample.

Recent research suggests that successful self-regulation is influenced both by a volitional system and by reactive, automated systems (Vohs & Baumeister, 2004). The more volitional and “top down” system involving brain regions such as the prefrontal cortex and anterior cingulate cortex is referred to by many investigators as effortful control (Eisenberg et al., 2004). Effortful control has been defined as “the efficiency of executive attention - including the ability to inhibit a dominant response and/or to activate a subdominant response, to plan, and to detect errors” (Rothbart & Bates, 2007, p. 129). Measurement of effortful control frequently focuses on the ability to control attention and to consciously inhibit behavior.

Recent work by Eisenberg et al. (2013) has also highlighted the importance of individual differences in the way that children experience the “pull” of distracting or rewarding stimuli as a related but distinct process from effortful control. Those individual differences in impulsivity, which are more automated than those subserving effortful control, may make it more difficult for some children to exert effortful control as compared to their less impulsive, more placable peers - previous research suggests that while this “reactive control” dimension of behavioral self-regulation is moderately correlated with children's executive attention and inhibitory control it is distinct from those abilities (Carver, 2005).

Children's self-regulation is also influenced by an emotionally-based system of regulation. This system, anchored in limbic as well as cortical brain regions, focuses on the extent to which children are able to engage positively with their learning environments on the one hand and to modulate anxiety, distress, and frustration in those environments, on the other (Fantuzzo et al., 2005, Raver et al., 2007). Three decades of research on child temperament clearly demonstrate that children enter the classroom setting with different psychobiologically-anchored profiles of emotional response (or negative reactivity) to new and challenging situations, task-demands, and people (Calkins, 1994, Rothbart et al., 1994). Those individual differences in children's reactivity and regulation of negative and positive emotion are readily perceived and interpreted by teachers and play an important role in shaping children's educational outcomes (Lerner et al., 1985, O'Connor et al., 2014). For example, children's positive emotion regulation in the context of learning (including sociability with their adult caregivers as well as motivation and pride in completing tasks) is viewed by teachers as key to students' classroom engagement and foundational to their school readiness (Berhenke et al., 2011, Ryan and Deci, 2000). In addition, children's capacity to manage their emotions has been found to be clearly associated with their learning-related behaviors and with teachers' perceptions of classroom adjustment (Denham, Bassett, Zinsser, & Wyatt, 2014). In one recent study, for example, Head Start-enrolled preschoolers' observed displays of pride, persistence, and anxiety each played a role in predicting teachers' reports of work-related learning skills, in addition to serving as significant predictors of social competence (Berhenke et al., 2011). Given the importance of young children's socioemotional competence for their everyday classroom experiences as well as their learning, many measures of school readiness include at least a few items that tap this dimension of regulation (Gresham & Elliott, 1990).

Self-regulation is increasingly assessed through the use of direct assessments; those assessments are generally administered to young children in lab-based settings where researchers can control the environment. It is generally preferred to administer several direct assessments that capture various aspects of self-regulation rather than a single task in order to capture the breadth of self-regulation skills (Sulik et al., 2010). Although direct assessments offer many methodological strengths and are objective, they can be costly and time-consuming to administer and code. In addition, though direct assessments provide microanalytic precision regarding moment-by-moment changes in children's accuracy and latency to respond within a given task, they may be less informative regarding children's more molar or global profiles of self-regulation that are likely to align with children's behavior in the classroom. That is, children's performance on carefully controlled direct assessments may not fully capture their ability to self-regulate in noisy and stimulating classrooms. In short, researchers interested in individual differences in children's self-regulation in educational settings face a significant measurement challenge that cannot be readily or completely solved by the use of direct assessments.

One solution to this empirical challenge has been to develop a battery of direct assessments that can be administered in field contexts, where children's profiles of self-regulation can scored or coded in vivo by an assessor. One example is the Preschool Self-Regulation Assessment (PSRA). Smith-Donald, Raver, Hayes, and Richardson (2007) developed the PSRA battery as one of the first portable direct assessment batteries of young children's self-regulation. The PSRA was created to be an ecologically valid and low-cost measure of self-regulation for preschool-age children (Smith-Donald et al., 2007) and consists of 10 structured tasks that assess effortful control, conflict resolution, and compliance. Following the assessments, the assessor completes the PSRA-AR, a report of children's attentional, behavioral, and emotional regulation during the battery. The PSRA-AR is a 28-item report adapted from the Leiter-R social-emotional ratings scale (Roid & Miller, 1997) and the Disruptive Behavioral-Diagnostic Observation Schedule coding system (DB-DOS; Wakschlag et al., 2005) that captures more global aspects of self-regulation.

Since its publication, the PSRA battery has been widely used, particularly in large-scale data collection efforts conducted in the field (Bassett et al., 2012, Raver et al., 2011). A recent factor analysis of the 10 PSRA structured tasks found support for a three-factor model of self-regulation composed of cool executive control, which focuses on children's ability to focus attention; hot executive control, which focuses on children's ability to delay gratification; and compliance, which focuses on children's ability to comply with directions (Denham, Warren-Khot, Bassett, Wyatt, & Perna, 2012). Moreover, this three-factor structure was invariant across educational setting (Head Start vs. private child care), but some differences emerged when comparing the equivalency of model fit between boys and girls as well as between 3- and 4-year-old children. Overall, these analyses provide evidence that the PSRA structured activities can be used with confidence across early child care and education settings, though important differences should be taken into consideration when comparing performance on specific tasks across gender and age.

In tandem with the PSRA structured tasks, the PSRA-AR has also been used widely as it provides information about children's self-regulatory abilities that is complementary to, but different from, the information gained through direct assessments (Bailey et al., 2016, Ferrier et al., 2014). Past research supports the use of observer ratings of children's self-regulation as valid assessments that demonstrate predictive validity to later outcomes. For example, observer ratings of children's self-regulation during a battery of tasks at ages 3 and 5 have been found to predict wealth, health, and criminal outcomes in adulthood (Moffitt et al., 2011) - suggesting that scores from measures such as the PSRA-AR may have important implications for a variety of later outcomes.

The PSRA-AR, which can be completed in conjunction with any battery of direct assessments, is particularly useful to researchers working in schools and homes due to the ease and speed with which assessors can complete it to gain a more global understanding of children's self-regulation. Moreover, the PSRA-AR is valuable in instances when time with participants is limited and several constructs need to be assessed through direct measures such that there is not enough one-on-one time to thoroughly assess self-regulation. In the current study, we extend previous factor analytic work on the PSRA battery to the PSRA-AR using a large, population-based sample from one of the largest school districts in the United States.

The original publication of the PSRA-AR included analyses examining the measure's structure. The original authors conducted principal components analysis (PCA) on 19 of the 28 items; the other 9 items, which largely related to displays of negative emotion, were too skewed to include in factor analyses (Smith-Donald et al., 2007). This analysis revealed three factors: attention control (reflecting individual differences in children's executive attention, described earlier), impulse control (reflecting individual differences in children's inhibitory control as well as their proneness to demonstrate impulsivity), and positive versus negative emotion. Because many items had substantial cross-loadings on both the attention control and impulse control factors, for their final model the authors forced the PCA into a two factor structure that yielded an attention/impulse control factor (10 items) and a positive emotion factor (7 items) out of 17 items; the remaining 2 items did not load onto either factor. Although appropriate for the sample size of that study, PCA is limited by the fact that it does not allow for a comparison of model fit across different factor structures. In the present study, we aim to overcome this shortcoming by revealing the underlying factor structure of the PSRA-AR using confirmatory factor analysis (CFA). Moreover, the original study used a small convenience sample on which the PSRA-AR was piloted. Given the feasibility of the PSRA-AR as a tool to measure self-regulation in large data collection efforts, we modeled the factor structure of this measure with a large, population-based sample of preschoolers.

Although self-regulation assessments such as the PSRA-AR have shown promise in smaller scale efficacy trials, it is not clear whether those measures can be effectively used in large-scale population-based research. One major methodological hurdle is that the validity of these measures for use with socioeconomically and racially/ethnically diverse samples has not been thoroughly examined, nor have researchers thoroughly investigated whether these measures work equally well for boys and girls. This is a common issue in the self-regulation literature, where many measures were initially developed and validated using data from predominantly European American, middle-income samples (Rothbart, Ahadi, Hershey, & Fisher, 2001) or clinical samples (Gioia, Isquith, Guy, & Kenworthy, 2000). Yet, it is imperative to establish whether our measures assess the same construct in the same way (meaning they are invariant) across groups of children who differ by sociocultural group membership as a key benchmark of psychometric validity (Raver et al., 2007, Sulik et al., 2010).

Our field has already made some headway in establishing the validity of the use of several self-regulation measures with diverse samples. The most common way to assess this is through testing measurement invariance, which refers to a set of statistical conditions that, when satisfied, provides evidence that a measure does not systematically differ across groups (Vandenberg & Lance, 2000). A few studies have examined invariance across groups of children from low and higher income households and found that self-regulation measures tend to work well for children of diverse socioeconomic backgrounds. One study found invariance for two teacher reports of self-regulation, the Barratt Impulsiveness Scale (BIS-11) and the Behavior Rating Inventory of Executive Function (BRIEF), across children from poor versus non-poor households (Charles McCoy, Raver, Lowenstein, & Tirado-Strayer, 2011). A study of direct assessments of self-regulation in young children found invariance across children of low/middle socioeconomic status homes and children of high socioeconomic status households (Mulder, Hoofs, Verhagen, van der Veen, & Leseman, 2014). The direct assessments examined in that study included one inhibitory control task, two working memory tasks, and two delay of gratification tasks.

Some studies have also examined the use of self-regulation measures with children of diverse racial/ethnic backgrounds, and have mostly found that measures work well across children of various different backgrounds. For example, one study found that measurement invariance was supported in African American and Hispanic Head Start participants (Charles McCoy et al., 2011). Another study that investigated the measurement invariance of direct behavioral assessments of self-regulation and teachers' reports of inhibitory control in low-income African American, Hispanic, and European American preschoolers found many similarities across these groups, though some differences did emerge (Sulik et al., 2010).

All of the studies described above also assessed invariance across males and females, with most finding full invariance across these two groups (Charles McCoy et al., 2011, Mulder et al., 2014). Sulik et al. (2010), however, found some small differences across sex – suggesting that researchers should examine whether measures work equally well for male and female participants.

One major limitation in the studies presented above, however, is that most relied on samples of convenience where the range of diversity in either socioeconomic status or race/ethnicity was limited. For example, the study that examined invariance in a Head Start sample was by design limited in socioeconomic diversity and did not include White preschoolers (Charles McCoy et al., 2011). Sulik et al. (2010) were also reliant on a largely high-risk sample that was homogenously low-income. And even though the Mulder et al. (2014) sample was taken from a national cohort, those data are from the Netherlands, a country that has a starkly different racial/ethnic composition than the United States. Here, we aim to expand this growing literature by testing measurement invariance of the underlying structure of the PSRA-AR using a population-based sample from a large city in the United States to ensure that scores on the measure have the same interpretation across boys and girls, children of different income levels, and children of different racial/ethnic backgrounds.

There are three levels of measurement invariance that are frequently tested by researchers who are interested in establishing the validity of a measure in different groups. These levels are hierarchically organized, such that testing each subsequent level of measurement invariance depends on the previous level being satisfied. The first level is configural invariance, which ensures that the same number of factors is present across groups and that each item loads on the same factor in each group. The second level is metric invariance (also known as weak factorial invariance), which refers to equality of the factor loadings across groups. The third level is scalar invariance (also known as strong factorial invariance), which refers to equality of the intercepts across groups. Scalar invariance is crucial because it allows researchers to make valid and meaningful comparisons of scores in different groups, including comparing group means.

These different aspects of measurement invariance are typically tested using CFA. Although there is no formal statistical test for configural invariance, this aspect of measurement invariance is supported if a CFA model with the same number of factors and same pattern of factor loadings fits well in each group. Testing metric and scalar invariance is typically done using chi-square difference tests in which a model with additional constraints is compared to less constrained model. If adding the set of constraints does not significantly worsen model fit, then measurement invariance is supported. It is possible to obtain partial measurement invariance if there is support for some, but not all, of the relevant constraints (Vandenberg & Lance, 2000). In the current study, we will use this framework to test for measurement invariance across sex; across low-income status, indicated by children's receipt of free or reduced-price lunch; and across racial/ethnic groups - specifically Black, Hispanic, and White preschoolers.

In addition to establishing the factor structure of the PSRA-AR and testing the invariance of that structure across low-income status and race/ethnicity, in the current study we examine whether the subscales of the PSRA-AR are associated with children's scores on direct assessments of two abilities known to be associated with self-regulation: Children's executive function and academic ability. Executive function refers to the processes that enable one to hold information in mind, ignore irrelevant distractors, and resolve conflict among competing stimuli – that is, cognitive aspects of self-regulation (Blair & Ursache, 2011). Though in theory assessing very similar constructs, previous research shows that adult ratings of children's behavioral self-regulation are weakly to moderately correlated with direct assessments of executive function (Toplak, West, & Stanovich, 2013). Therefore, we hypothesize that scores on items relating to attention and impulse control in the PSRA-AR will be positively but moderately correlated with direct assessments of executive function in the current sample.

Self-regulation has been hypothesized to be an important mechanism that supports children's learning and processing of academic information (Blair, 2002, McClelland et al., 2007). The ability to focus attention helps children focus on academic content in the classroom and ignore distracting stimuli. Behavioral and emotional self-regulation support children's behavior in the classroom - enabling them to maintain positive relationships with their teacher and peers (Blair, 2002). Previous research shows that ratings of children's self-regulation are predictive of academic ability even after statistically accounting for children's IQ (Blair and Razza, 2007, Fuchs et al., 2006). Therefore, we hypothesize that children's scores on the PSRA-AR will be positively correlated with direct assessments of academic ability.

The goal of the present study was to establish the validity of the use of the PSRA-AR, an observer report of self-regulation, with a very large and diverse population-based sample of preschool-aged children in one of the nation's largest school districts. We had three aims. Our first aim was to test the underlying factor structure of the PSRA-AR. Our second aim was to test invariance of the measure across sex, across low-income and non-low-income children, and across Black, White, and Hispanic children. Our third aim was to establish the concurrent construct validity of the PSRA-AR by examining the extent to which individual differences on the PSRA-AR were associated with children's performance on direct measures of children's executive function and academic ability.

Section snippets

Participants

The sample was drawn from a study of children attending public pre-K in a large, urban setting. A total of 1023 preschool children attending 75 preschool programs were assessed, a sample that was representative of the entire public pre-K population of this large and diverse school district. Of those, 994 had PSRA-AR data (some data were lost due to computer error). There were no demographic differences between children with or without valid PSRA-AR data. Because one of our aims was to explore

Confirmatory factor analysis

The 25 items of the PSRA-AR rated on a 1–4 scale were considered for CFA analyses. Descriptive analyses of the current sample revealed that > 90% of respondents received the same score for five of the 25 items. Those five items represented negative behaviors that assessors rarely reported observing, such as intense angry feelings and frequent feelings of sadness; descriptive analyses suggested a corresponding lack of variability and extreme skewness. Due to their lack of variability, which makes

Discussion

Currently, few assessments of self-regulation have been validated for use with large socioeconomically and racially/ethnically diverse populations that are typical of many urban school districts. Here, we aimed to begin closing this gap in the literature by examining the validity of the PSRA-AR, a short observer report that captures children's profiles of self-regulation across any direct assessment at a molar level, using a large and diverse population-based sample of preschoolers. First, we

Conclusion

Overall, our findings suggest that the PSRA-AR is a valid, quick, and cost-effective measure of self-regulation for use with large and diverse preschool populations. The PSRA-AR works equally well with male and female students, low-income and non-low-income preschoolers, as well as with Black, Hispanic, and White children to assess attention regulation, impulse control, and positive engagement. As such, the PSRA-AR is a valuable tool to the field that can easily and quickly be used in a variety

Acknowledgements

This work was supported by the Institute of Educational Sciences [grant number R305U140002] and the Spencer Foundation, Chicago, IL [grant number 201500074].

References (57)

  • C. Blair

    School readiness: Integrating cognition and emotion in a neurobiological conceptualization of child functioning at school entry

    American Psychologist

    (2002)
  • C. Blair et al.

    Closing the achievement gap through modification of neurocognitive and neuroendocrine function: Results from a cluster randomized controlled trial of an innovative approach to the education of children in kindergarten

    PloS One

    (2014)
  • C. Blair et al.

    Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten

    Child Development

    (2007)
  • C. Blair et al.

    A bidirectional model of executive functions and self-regulation

  • C. Blair et al.

    Multiple aspects of self-regulation uniquely predict mathematics but not letter–word knowledge in the early elementary grades

    Developmental Psychology

    (2015)
  • S.D. Calkins

    Origins and outcomes of individual differences in emotion regulation

    Monographs of the Society for Research in Child Development

    (1994)
  • C.S. Carver

    Impulse and constraint: Perspectives from personality psychology, convergence with theory in other areas, and potential for integration

    Personality and Social Psychology Review

    (2005)
  • D.L. Charles McCoy et al.

    Assessing self-regulation in the classroom: Validation of the BIS-11 and the BRIEF in low-income, ethnic minority school-age children

    Early Education & Development

    (2011)
  • S.A. Denham et al.

    How preschoolers' social–emotional learning predicts their early school success: Developing theory-promoting, competency-based assessments

    Infant and Child Development

    (2014)
  • A. Diamond

    Why improving and assessing executive functions early in life is critical

  • A. Diamond et al.

    Development of an aspect of executive control: Development of the abilities to remember what I said and to “do as I say, not as I do”

    Developmental Psychobiology

    (1996)
  • J.O. Edgin et al.

    Development and validation of the Arizona cognitive test battery for down syndrome

    Journal of Neurodevelopmental Disorders

    (2010)
  • N. Eisenberg et al.

    The relations of effortful control and impulsivity to children's resiliency and adjustment

    Child Development

    (2004)
  • N. Eisenberg et al.

    Are effortful and reactive control unique constructs in young children?

    Developmental Psychology

    (2013)
  • D.E. Ferrier et al.

    Relations between executive function and emotionality in preschoolers: Exploring a transitive cognition-emotion linkage

    Frontiers in Psychology

    (2014)
  • S.J. Finney et al.

    Non-normal and categorical data in structural equation modeling

  • L.S. Fuchs et al.

    The cognitive correlates of third-grade skill in arithmetic, algorithmic computation, and arithmetic word problems

    Journal of Educational Psychology

    (2006)
  • G.A. Gioia et al.

    Behavior rating inventory of executive function: BRIEF

    (2000)
  • Cited by (21)

    • The intersection of teacher-child language & ethnic match for Hispanic/Latine dual language learners in early elementary school

      2023, Early Childhood Research Quarterly
      Citation Excerpt :

      The PSRA-AR is an adaptation of the Leiter-R social-emotional rating scale, examiner version (Roid & Miller, 1997), a tool used to assess attention skills in children and adolescents ages 2-21. The PSRA-AR has been well-validated in diverse samples of school age children, including Hispanic/Latine children and those from economically-disadvantaged backgrounds (Daneri et al., 2018), and is correlated with several other executive functioning measures (r = .28-.62), as well as math and literacy (r = 0.23–0.38; Smith-Donald et al., 2007). After completing the battery of child cognitive assessments, assessors completed the PSRA-AR Attention/Impulse Control subscale to capture children's attention throughout the assessment period, rating behaviors on a four-point Likert scale from least (=0) to most (=3) regulated attention.

    • Time spent playing predicts early reading and math skills through associations with self-regulation

      2022, Journal of Applied Developmental Psychology
      Citation Excerpt :

      The PSRA assessor report has been found to be reliable for the evaluation of preschool-aged children's self-regulation skills (Smith-Donald et al., 2007; Tanribuyurdu & Yildiz, 2014). We collapsed two subscales tapping children's attention and impulse/inhibitory control based on prior studies using this measure (Daneri, Sulik, Raver, & Morris, 2018). For this study, scores on HTKS and PSRA measures were standardized and averaged (r = 0.52).

    • Classroom-level peer self-regulation as a predictor of individual self-regulatory and social-emotional development in Brazil

      2021, Journal of Applied Developmental Psychology
      Citation Excerpt :

      We used CFA to determine the factor structure of the classroom-level PSRA-AR in Brazil. Specifically, we compared the fit of the original one-factor model established by Smith-Donald et al. (2007) to a two-factor model introduced by Daneri et al. (2018). The two-factor model separated the 16 PSRA-AR indicators into two covarying latent variables: (1) attention regulation, which was comprised of eight indicators, and (2) impulse control, which was comprised of the other eight indicators.

    • Null relations between CLASS scores and gains in children's language, math, and executive function skills: A replication and extension study

      2021, Early Childhood Research Quarterly
      Citation Excerpt :

      We used the incongruent trial of Hearts and Flowers to measure inhibitory control, whereas the prior Boston study used the Pencil Tap. The correlation between these two measures when tested in preschool age children is relatively small (r = 0.28, p < 0.001; Daneri, Sulik, Raver, & Morris, 2018) and there is a paucity of empirical evidence on the extent to which the measures and their scoring rules are comparable. In other work, the Pencil Tap and Hearts and Flowers task have been considered equivalent direct measures of executive function skills (McCoy, 2019).

    View all citing articles on Scopus
    View full text