http://orcid.org/0000-0003-4015-7435
Figures
Abstract
Little is known about the underlying relationships between self-reported mental health items measuring both positive and negative emotional and behavioural symptoms at the population level in young people. Improved measurement of the full range of mental well-being and mental illness may aid in understanding the aetiological substrates underlying the development of both mental wellness as well as specific psychiatric diagnoses. A general population sample aged 14 to 24 years completed self-report questionnaires on anxiety, depression, psychotic-like symptoms, obsessionality and well-being. Exploratory and confirmatory factor models for categorical data and latent profile analyses were used to evaluate the structure of both mental wellness and illness items. First order, second order and bifactor structures were evaluated on 118 self-reported items obtained from 2228 participants. A bifactor solution was the best fitting latent variable model with one general latent factor termed ‘distress’ and five ‘distress independent’ specific factors defined as self-confidence, antisocial behaviour, worry, aberrant thinking, and mood. Next, six distinct subgroups were derived from a person-centred latent profile analysis of the factor scores. Finally, concurrent validity was assessed using information on hazardous behaviours (alcohol use, substance misuse, self-harm) and treatment for mental ill health: both discriminated between the latent traits and latent profile subgroups. The findings suggest a complex, multidimensional mental health structure in the youth population rather than the previously assumed first or second order factor structure. Additionally, the analysis revealed a low hazardous behaviour/low mental illness risk subgroup not previously described. Population sub-groups show greater validity over single variable factors in revealing mental illness risks. In conclusion, our findings indicate that the structure of self reported mental health is multidimensional in nature and uniquely finds improved prediction to mental illness risk within person-centred subgroups derived from the multidimensional latent traits.
Citation: St Clair MC, Neufeld S, Jones PB, Fonagy P, Bullmore ET, Dolan RJ, et al. (2017) Characterising the latent structure and organisation of self-reported thoughts, feelings and behaviours in adolescents and young adults. PLoS ONE 12(4): e0175381. https://doi.org/10.1371/journal.pone.0175381
Editor: Gilles van Luijtelaar, Radboud Universiteit, NETHERLANDS
Received: August 1, 2016; Accepted: March 24, 2017; Published: April 12, 2017
Copyright: © 2017 St Clair et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data for this specific paper has been uploaded to the Cambridge Data Repository (https://doi.org/10.17863/CAM.8685) and is password protected. Our participants did not give informed consent for their questionnaire measures to be made publicly available, and it is possible that they could be identified from this data set. Access to the data supporting the analyses presented in this paper will be made available to researchers with a reasonable request to openNSPN@medschl.cam.ac.uk.
Funding: This work has been supported by a Wellcome Trust Strategic Award (reference: 095844) to Ian Goodyer, Ray Dolan, Ed Bullmore, Peter Jones and Peter Fonagy which provides core funding for the Neuroscience in Psychiatry Network (NSPN.org.uk). The Wellcome trust had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. Michael Moutoussis is funded by the UCLH Biomedical Research Council. Additional support was provided by the NIHR Cambridge Biomedical Research Centre. Peter Fonagy is in receipt of a National Institute for Health Research (NIHR) Senior Investigator Award (NF-SI-0514-10157). Peter Fonagy was in part supported by the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Barts Health NHS Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Very few studies have concurrently investigated the underlying structure of mental ill health and mental well-being in young people. The psychopathology literature on mental illness symptom structure has, on the whole, remained distinct from the wider concepts of the structure of mental well-being. Indeed, the more common approach is to characterise mental illness alone through well-established symptom combinations [1], which rely on the assumption that the absence of symptoms is equivalent to no mental illness. However, the absence of mental illness symptoms cannot be taken as either necessary or sufficient evidence of positive mental health. Mental health or well-being has been defined as a state of positive feelings and functioning, which can either be present or absent without necessarily having any implications on an individual’s mental illness status [2]. Failing to account for mental well-being when examining the structure of thoughts, feelings and behaviours in young people, is likely to provide a skewed view of mental health in the population at large [3].
Currently the natural relationships of item level thoughts, feelings and behaviours in the youth community are largely unknown. Adolescence and young adulthood is both a time of substantial maturational changes within physiological and neurological systems [4,5] as well as correspondingly large social changes, both of which are reflected in a rise in the incidence of mental illness diagnoses and mental distress symptoms. Thus, this is a key developmental time to investigate the emerging relationships between thoughts, feelings and behaviours that may discriminate well-being from emerging abnormal mental states. To the authors’ knowledge, no study has investigated the structure of a wide range of mental health difficulties alongside mental well-being items, which is the focus of the current study.
In those studies that have investigated the structure of both positive and negative mental states together the results are rather mixed: some find separate factors for positive mental well-being and mental illness while others report a single higher order mental state factor and lower level factors (e.g, [6–9]). One study found a single general factor from all items, with specific factors for positive and negative mental health [10]. Despite variations in item pool and method of item selection, these prior studies report substantial correlations between the positive and negative mental state factors suggesting a close relationship between the constructs of positive mental well-being and mental illness.
With regards to negative mental states, the higher order structure of psychiatric disorders as well as mental illness symptoms that explain item relationships and diagnostic comorbidity have been extensively reported in adults and younger populations (e.g., [11–15]). Within child and adolescent psychiatry much research has evaluated higher order externalising (i.e., Conduct Disorder, Oppositional Defiant Disorder) and internalising (i.e., Depression, Anxiety) domains (e.g, [13,16]) as a way of accounting for the substantial comorbidity between psychiatric diagnoses. Recent research has also indicated higher order factors relating to thought disorders and pathological introversion, composed of social anxiety, dependence and unassertive traits [14]. Much previous literature (in both mental wellness and illness) has assumed that first order factors (such as externalising and internalising) are orthogonal or uncorrelated factors of equal potential value. However, high correlations between these factors led to investigations of a single general mental illness factor that could be common to all indicators in addition to the lower level domains [17,18]. There is a growing research base that has re-examined the structure of symptoms using diagnoses and/or item level data and identified a single common “psychopathology” factor [17–26]. Initial work has been conducted which combined mental well-being and illness items within a similar analytic framework and also found evidence for a single overarching factor [limitations as described above; 10,see also [27] where prosocial items were included alongside mental illness items, in a younger population (7–14 years of age)]. Therefore, there is evidence to suggest that utilising a larger range of items encompassing mental health (well-being and illness) may generate a general distress or “psychopathology” factor together with more detailed specific factors implying a more multi-dimensional nature to thoughts feelings and behaviours in the youth population than considered hitherto.
The pathogenesis viewpoint on measuring mental illness has, understandably, focused on case-control studies of the mentally ill as compared to the not mentally ill, with the latter defined by the absence of predefined illness criteria rather than by the presence of a positive mental state or well-being. Recently, however, there has been a shift to focus research away from clinical types and towards quantitative measures of behaviour, cognition, physiology, and genetic factors [28,29]. This is driven in part by the low validity inherent in the descriptive diagnostic system and the need to develop a greater understanding of the aetiologies and pathophysiologies of mental illnesses [30]. Redefining how we conceptualise the ‘surface’ characteristics of mental health in the population at large using both mental illness and mental wellness items may create a more accurate behavioural phenotype of mental states in the natural world [31]. This may in turn translate to greater precision in pinpointing the relevant genetic, neural and cognitive factors that could underlie mentally well together with ill states. Just as psychopathology extends beyond diagnoses into the general population so, too, may the biological processes revealed by this approach, revealing a richer pattern of variance than exposed through a binary approach. A broader approach appears to be warranted: Recent research indicates that increasing precision in behavioural phenotypes using modern psychometric methods can increase signal within molecular genetics [32] as well within analysis of personality and cognitive correlates of psychopathology [25]. Such findings demonstrate that mathematical modelling can aid in validity studies of mental illnesses (see Forbes and colleagues [33] for a theoretical argument as well). In summary, the structure of mental illness and well-being in the general population is likely to be a multi-dimensional complex structure. Evaluating the structure of mental illness and well-being across a wide range of items may increase our understanding of mental health as well as increase the validity of behavioural signals identifying the aetiological components specifically underlying mental illness and wellness.
The current study
Our analysis was conducted in three sections: evaluation of the structure of mental health using latent-trait methods, person-centred analysis of individuals with differing profiles of the mental health traits, and external validation of both the latent traits as well as the person-centred subgroups.
Many previous investigations that have indexed clinical symptom structure using item level data alone have only used measurements of anxiety and depression (e.g., [20,23]), with some studies additionally measuring psychotic experiences (e.g., [21,24]) and others a wider range of self and parent report domains (but not well-being; [22,27]). The evaluation of item level data combining mental well-being with items indexing clinical symptoms has often included only those relevant to depression and anxiety (e.g., [6–10]). Our aim was to characterise the common behavioural phenotypes of mental health across a wide-ranging spectrum of items denoting both mental well-being and illness within a community sample of 2228 individuals aged 14–24 years, which is an age group whose mental health structure has been understudied [34]. Measurement of both mental well-being and mental health involves a mixture of positively and negatively worded items. This carries potential confounding effects on factor structures [35,36] so we were careful to control for these wording effects within all of our modelling approaches in a manner previously suggested in the literature [37].
Although we evaluated a range of models, including single factor, correlated factor and second order solutions, we hypothesised that we would find a multidimensional behavioural structure in the youth population. We expected this would consist of a single general factor loading onto all items encompassing a construct ranging from positive well-being to negative illness items and denoting a mental perturbation expressed as distress (low through high). We also predicted there would be ‘distress independent’ specific factors that would not simply be left over variance of no consequence, but index specific unique characteristics of mental wellness and illness and also contribute to valid multidimensional behavioural patterning in the youth community at large. Therefore, our overall prediction was that there would be a bifactor structure, consisting of one general distress factor and distress independent factors. Our objective was to create a data driven behavioural phenotype that could be more closely associated with the underlying intermediate neurocognitive systems than single measurement constructs. This approach resonates with the RDoC initiative to more closely align neuroscience, cognition and behavioural phenotypes [31]. Several lines of evidence support the prediction of a bifactor structure. Previous findings have indicated a best fitting bifactor structure within mental illness [17–26], and a recent paper showed a similar structure when combining mental illness and mental well-being items in a sample of adults [10]. Additionally, a bifactor structure is suggested by literature showing high correlations between mental well-being and mental illness factors [6–9] and between externalising and internalising conditions (e.g., [14]). Finally, the vast literature detailing the high rates of comorbidity of mental illness in the general population supports the hypothesis of a higher order distress factor underlying many psychological diagnoses [38,39].
A second aim was to define subgroups of individuals with distinct combinations of both the general distress and specific distress independent factors. Since behaviour is multidimensional within as well as between individuals we hypothesised that deriving subgroups based on the aforementioned computed quantitative factorial levels would provide added value and greater validity for more mechanistic studies of mental health and illness going forward, as highlighted by recent reviews [34,40].
Finally, to provide a first test of validity of any new data driven factors and (factor based) subgroups we hypothesised that hazardous behaviours (self-harm, drinking and drug misuse) and self-reported experiences of treatment for mental ill health would discriminate between factors and derived classes. We predicted that those who engage in these hazardous behaviours or who have a history of treatment for mental ill health may have either altered factor scores or be more prevalent in certain subgroups. Specifically, hazardous behaviours would be associated with those factors relating to mental illness and subgroups defined by increased rates of general distress and/or specific factors related to mental illness.
Methods
Participants
The Neuroscience in Psychiatry Network (NSPN; nspn.org.uk) cohort consist of 2,257 volunteer participants aged between 14 and 25 recruited from Cambridgeshire (through the University of Cambridge) and London and surrounding areas (through University College London [UCL]). The recruitment of the sample included in this paper was from November 2012 to May 2014. Individuals who were currently taking part in a clinical trial of a medicinal or other therapeutic intervention were excluded. The main sampling framework comprised age-sex registers of 41 general practices (primary health care practices) recruited through the NIHR Clinical Research Network, and through 19 secondary schools/colleges/universities. Letters from the study team were sent in batches by the practices and schools to individuals in order to recruit into age and sex strata, as described below. Thus, we estimate that nearly all individuals who were eligible to participate were informed about the study opportunity. Additionally, our recruitment team made direct visits to secondary schools and colleges to support recruitments from this source. A minority (18.3%) were recruited directly through our website (www.nspn.org.uk). Written, informed consent was obtained for all participants over the age of 16 years. For participants under the age of 16, written informed assent was obtained for the participant and written informed consent from their parent/legal guardian.
Recruitment was stratified according to age and gender to give at least 200 males and females within each of the following five age categories [years;months]– 14;0 to 15;12, 16;0–17;12, 18;0–19;12, 20;0–21;12 and 22;0–24;12. One individual returned the study materials at 25;1 and was included. Consistent with previous studies [20], participants were included in our analysis if they filled in at least 85% of the total items (118 total items, minimum valid item count was 100) as well 85% of each individual measurement to ensure adequate data on all domains. There were a total of 29 individuals excluded based on this criterion. We next tested whether these individuals differed on the main measures investigated in the multivariate analysis. Due to the number of tests used, the significance value was lowered to p = .01. Significance levels between .01 and .05 were treated as marginally significant. The 29 individuals who did not meet the 85% criteria did not differ in age, gender, or total score of the MFQ (depressive symptoms), r-LOI (OCD symptoms), RSE (self-esteem), ABQ (antisocial behaviour), SPQ (psychotic symptoms), or WEMWBS (mental well-being). They had marginally higher total scores on the RCMAS (anxiety symptoms; t(2249) = -2.33, p = .02; M = 19.56, SD = 14.62 included; M = 26.54, SD = 11.85 excluded).
Thus, the study sample for this article is 2,228 individuals (54% girls). The mean age was 19 years (sd = 3 years). Sixty-one percent of the sample was recruited from Cambridgeshire. Differences across recruitment location in the final latent traits were evaluated. Several traits showed significant differences across location but most of these differences became non-significant when controlling for indices of multiple deprivation. This indicated that the majority of location differences were due to different distributions of multiple deprivation across Cambridgeshire and London. One latent trait (Specific Factor 5 –Mood) retained a significant location difference. With all subsequent analysis with this latent trait, recruitment location was included as a covariate. See the S1 Materials for a demographic summary of the sample.
Ethical approval was granted for this study by the NHS NRES Committee East of England—Cambridge Central (project ID 97546). The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Procedure
Members of the public were sent an invitation along with an expression of interest form via GP surgeries and local schools or colleges. Some participants responded to an advert or completed an expression of interest on the study website. Participants returned the completed expression of interest directly to the study team and were subsequently sent consent/assent forms (along with guardian consent forms for those under 16) as well as the study materials. Of those who expressed interest in possibly participating in the study and were sent study materials, 66% completed the completed consent forms and study materials; they received £10 in recompense for their time.
When compared on basic demographics, non-responders were more likely to be male (χ2(1) = 25.87, p < .001, V = .08), recruited from London (χ2(1) = 37.40, p < .001, V = .10) and slightly younger (t(2993.49) = 4.03, p < .001, d = .15; responders: M = 19;0 [years;months]; SD = 3;0; non-responders: M = 18;8; SD = 2;7). Responders were more likely to be white (χ2(1) = 63.08, p < .001, V = .13). When evaluated further, this difference was due to an over-representation in those who did not return the study materials of Asian, Black, and individuals who preferred not to indicate their ethnicity, and an over-representation of white individuals in those who did return the materials. There were no differences in those of mixed race or who ticked the “other” option. As more ethnic diversity was found in the London sample, we repeated the ethnicity comparison just within individuals recruited from London. Results were replicated with the exception that Asian participants were equally represented across responders and non-responders.
Measures
All questionnaire measures used their original coding unless more than half of the items within each instrument had very sparse endorsement (< 1% of entire sample). In the latter instance, the sparse response categories were collapsed into adjacent categories.
Mental health measures.
We wished to measure the common mental illnesses that emerge in the decade from 14 to 24 years where emotional and behavioural symptoms are the constituent components of the most commonly diagnosed disorders. The mental illness domains of interest to us were therefore anxiety and depressive disorders, conduct disorders, and obsessive compulsive and psychoses that emerge with discernible incident risk in adolescence and into the second decade of life [41]. In order to measure the whole of mental health, we included measures of positive mental health (self-esteem and well-being measures).
Thus, the following mental health domains were measured using self-reported measures covering the previous two weeks: Depression (33-item Moods and Feelings Questionnaire; MFQ; [42]); generalised anxiety that included measurements of physiological change, worry/oversensitivity, and social concerns (28-item Revised Children’s Manifest Anxiety Scale; RCMAS; [43]); obsessive compulsive items ([11-item Revised Leyton Obsessional Inventory; r-LOI; [44]); antisocial behaviours consisting of violating social norms, destructive behaviours, violence to people, lying and stealing (11-item Antisocial Behaviour Questionnaire; ABQ); self-esteem (10-item Rosenberg Self-Esteem Questionnaire; RSE; [45]); and mental well-being measuring positive evidence of current happiness, personal activity and personal achievement (14-item Warwick-Edinburgh Mental Well-Being Scale; WEMWBS; [46]).
We used a four category response version (never, mostly, sometimes, and always) for the RCMAS, r-LOI, ABQ, and RSE measurements where participants endorsed each item on how frequently they felt or behaved that way in the previous two weeks. Responses on the ABQ were converted to a “never” vs. “sometimes/mostly/always” binary response due to sparse endorsement of the mostly and always categories. The WEMWBS had five response categories (none of the time, rarely, some of the time, often, all of the time) with higher scores indicating higher mental well-being. All WEMWBS items were positively worded. Five RSE items were positive worded while the remainder were negative worded. No items were reversed for this analysis.
Participants completed the Schizotypal Personality Questionnaire (SPQ), a 74 item self-report questionnaire, measuring the items of the DSM-III diagnosis of Schizotypal Personality Disorder, including items related to psychotic-like experiences [47]. Response choices for all items were “present” or “absent”. As this study focused on measuring psychotic-like experiences, endorsement on the SPQ items was compared with positive ratings of psychotic-like experiences on the semi-structured PLIKS interview (PLIKSi) using a subsample of the total sample. PLIKSi measures 12 ‘core’ psychotic-like experiences [48]. Associations between SPQ items and the PLIKSi total symptoms as well as the hallucinations, delusions, and unusual perceptual experiences subscales were evaluated. Highly significant relationships with medium to large effect size were the criteria for inclusion. Additionally, consensus meetings between MCSC, IMG and PBJ agreed the retained items had face validity as psychotic items. Full details of this analysis can be found in the S1 Materials. The final items retained were SPQ 4, 9, 13, 28, 31, 40, 55, 60, 61, 63 and 64. See supplementary Table C in S1 Materials for item content for all measurements.
Discriminant validity measures.
A self-report questionnaire measured past month alcohol, cannabis, and other UK illegal drugs (e.g., cocaine, heroin, MDMA/ecstasy, ketamine) use as well as non-suicidal self-injury (NSSI). Alcohol use was defined as never, sometimes or often/every day. Cannabis and other illegal drug use was defined as never or some use. NSSI was defined as never or once vs. twice or more, distinguishing those who had multiple occurrences of past month NSSI. Alcohol and drug use as well as NSSI were considered to be indicators of hazardous behaviours in the community; likely outcomes or consequences, rather than symptoms of these mental states directly. None of the items in the structural analysis overlapped with the hazardous behaviours used as discriminating variables. Current or past treatment for any emotional, behavioural or mental health problem was derived from parental reported (under 18) or self-reported (18 and over) difficulties. This was further broken down to measure specific treatment for depression.
Analysis procedure
The latent trait analysis was based on the assumption that items would relate to each other in this population to reveal an underlying latent construct of mental health, across a positive and negative spectrum. This expectation is based on our prior work using item based analyses to elicit underlying latent structures, but to date we have only focused on characterising the illness or negative end of the latent construct [20,21]. There is, however, no clear-cut a priori theory to direct a specific analytic perspective using a combination of narrow band instruments. For this reason we chose to conduct analysis in a bottom up, data driven, and theory free manner in order to generate the latent structure underlying the interrelationships between the selected items. Additionally, we aimed to derive person-centred classes from the results of the structural factor analysis, which has previously been successful in identifying population based subgroups with distinctive probabilities for emotional and behavioural mental illnesses [49,50].
All data were processed through the Research Electronic Data Capture (REDCap) system [51]. Analysis was conducted in two stages, exploratory and confirmatory, with all indicators defined as categorical. The sample was split for these stages of analyses: half the data was used for the exploratory analysis while the second half used for confirmatory models. When fully confirmed, model fit was checked on the exploratory half (consistent with suggestions by Brown [52]) and was found to be similar to the confirmation half in all instances. Final confirmatory analysis was therefore conducted on the full sample. All items were scaled from 0 to the highest category (as specified in the materials section). The WLSMV (Weighed Least Square for categorical data Mean and Variance adjusted) estimator was used. All descriptive statistics, gender and age analysis and discriminate validity analyses were conducted within STATA MP 13.1 [53]. All multivariate analyses, unless otherwise specified, were conducted in MPlus version 7.2 [54].
Exploratory analysis.
The exploratory factor analysis (EFA) for the first order models was conducted using the oblique promax rotation method, which is a methodology for evaluating correlated factor solutions. Previous work in our lab [20] investigating a subset the of the current indicators within a different sample evaluated all possible rotation methods and concluded that the promax rotation method produced the most interpretable solutions. We therefore chose this rotation method based on this previous work. An ESEM exploratory bifactor analysis was conducted using the bi-geomin rotation method. In line with the recommendation from Brown [52], the number of factors was determined through evaluation of the scree plot and eigenvalues as well as the model fit statistics for each potential factor solution. A more objective method, such as parallel analysis or minimal average partial tests, to determine factor number would have been preferable, but these options were not available in MPlus, nor was parallel analysis computationally possible in an alternative program (e.g., FACTOR; [55]).
Confirmatory analysis.
All confirmatory analyses (first order and bifactor) allowed each item to be freely estimated with the factor variance set to 1. Thus, all factors had a mean of 0 and SD of 1. For the confirmatory factor analysis, the factor structure from the exploratory stage was implemented within a confirmatory framework. All loadings above .20 from the EFA (or .15 for the ESEM Exploratory Bifactor solution) were included on the factors, with the exception that any cross loading items below .25 were excluded unless both cross loading items were below .30. We utilised a relatively liberal cut off point for the EFA as we wanted to ensure we had as much information on each CFA factor as possible, with the aim of further refinement of the structure within the CFA phase. A methods factor was also included in all confirmatory analyses to account for the positive items within the entire item pool (five RSE and 14 WEMWBS items), as suggested by Wang and Lin [35] to control for spurious factors related to differences in the valence of item wording.
Items were added to factors as suggested by the modification indices, which estimate the improvement in model fit dependent on adding a specific item to one of the factors. Items were deleted from the factors if they were non-significant (p<0.05) or loaded below .25 for the correlated factor solutions and .20 for the Bifactor models. We note that others performing factor analyses of mental health data have applied equally stringent or even less stringent cut-offs [10,20,37]. Any item with cross loadings on multiple factors was evaluated. The lower loading was removed where one loading was below .30 (unless both were below .30). Cross loadings were also removed if there was a large discrepancy between the loadings (difference of .30). The final step was to consult the modification indices once more for the presence of any correlated errors between individual indicators. Correlated errors between similar items were either included or one item deleted, depending on the degree of similarity in what the items were measuring. Consensus meetings between authors MCSC, IMG and PBJ decided whether the correlated items indicated redundant items, where one item should be deleted, or related, but distinct items, where a correlated residual should be included. Further details of the confirmatory procedure can be found in the S1 Materials along with details of items which were deleted during the confirmation procedures.
In addition to an ESEM exploratory bifactor solutions (see S1 Materials for a summary of the final solution), we also investigated the Schmid-Leiman (SL) transformation of the CFA factor structures on the full sample. The ESEM exploratory bifactor explores the specific factors while estimating a general factor across all items. The SL transformation takes the original CFA factor structure as the specific factors and adds an additional general factor to the model across all items included in the final factor analysis. The SL transformation is computationally equivalent to adding one second order factor over all first order factors [52].
For all bifactor models, all loadings on the general factor had to be at least .30 on the general factor or they were dropped from the model [20]. The general and all specific factors were set to be uncorrelated with each other, creating an orthogonal factor structure.
Age and gender analysis.
Regression analyses were conducted to determine how the latent traits differed by gender as well as developmental differences between the five different age categories. Analysis was done sequentially: firstly, with only main effects for gender and categorical age effects, then with a categorical age by gender interaction term. If the overall main effect of age category was significant, each comparison between the different age categories was conducted. For consistency, results were reported separately for males and females, even if the interaction was non-significant, indicating that the overall pattern was similar across males and females. A robust estimator was used when appropriate to control for unequal variances over the categories.
Latent profile analysis.
The final model from the variable-centred latent trait analysis was evaluated in a person-centred latent profile analysis (LPA) using the whole sample. This analysis determined subgroups of individuals with similar patterning of profiles across all latent traits. The maximum likelihood robust (MLR) estimator was used within Mplus Version 7.2 [54]. As the goal was to combine the investigative strength of the bifactor approach with the pattern recognition of LPA, we opted for a two-stage procedure. Combining both the bifactor measurement model and the LPA analysis in a ML approach was computationally too intensive, and would require a change in the estimator in the bifactor model. Therefore, the latent traits were saved and then re-entered into the LPA analysis. Two to seven subgroups were evaluated through comparison of the AIC, BIC and BIC-SSA values as well as direct testing of lower order solutions using likelihood ratio tests (Vuong-Lo-Mendell-Rubin LRT, Lo-Mendell-Rubin adjusted LRT, and parametric bootstrapping LRT). Additionally, entropy was consulted, with solutions with higher entropy (and thus lower variability in category membership) being preferred. Gender and age were included as covariates on class membership in all models.
Discriminant validity.
Ordered logistic or logistic regression analyses were used to predict the categorical variables (alcohol, cannabis, other illegal drug use, NSSI, MH treatment) jointly from all latent traits while also controlling for gender and age (as well as recruitment location when appropriate). A robust estimator was used when appropriate to control for unequal variances over the categories. For the LPA subgroups, the subgroup prevalence for the categorical variables was evaluated against the whole group distribution with a chi-square analysis. Due to the number of tests used, the significance value was lowered to p = .01. Significance levels between .01 and .05 were treated as marginally significant.
Results
Following exploratory analysis, the scree plot and eigenvalues indicated that a four or five factor model could be perused. Therefore, these two models (alongside a one factor model for comparative purposes) were evaluated. Additionally, one second order and three bifactor models were fully investigated, meaning seven latent variable models were fully confirmed and evaluated, with model fit information presented in Table 1. In evaluating the seven solutions, careful consideration of the theoretical plausibility of all models was combined with evaluation of the absolute statistical model fit of each model to the data. The one factor model was not considered due to unacceptably low model fit, ruling out the possibility that a single factor is the best fit to mental health and wellness symptoms. The four and five factor models were considered and the absolute model fits were comparable, although the five factor model had a better absolute fit. However, the five factor model was also thought more theoretically plausible as the four factor model combined antisocial items with OCD and psychotic-like symptoms. No previous research to the authors’ knowledge has found that psychotic and OCD symptoms load on the same factor as antisocial items, except a general psychopathology factor [14,15,18]. Therefore, the five factor model, which split the “thought” items encapsulated in the OCD and psychotic-like items from the antisocial items, was considered more theoretically plausible.
Similar rationale was applied in choosing the five factor SL bifactor model over the four factor SL bifactor model, as the inclusion of the general factor did not alter the structure of the specific factor in the four factor model which loaded antisocial, obsessive/compulsive as well as psychotic-like items on one factor. Additionally, the absolute model fit was stronger for the five factor SL model. Although the model fit differences between the four and five factor CFA and the related SL bifactor models did not differ substantially, there were unacceptably high correlations (r > .85) between the factors for the four and five factor CFA solutions, which is indicative of a general factor underlying all factors. We also tested whether these high correlations could be accounted for by a second order internalisation factor (accounting for the high correlations between conceptually relevant “internalising” first order factors) but this was not a good model fit and was not further evaluated. Thus, both CFA solutions were not preferred due to unacceptably high correlations and the four factor SL model was not preferred due to theoretically implausible specific factor structures. Finally, the bifactor model confirmed from the ESEM exploratory bifactor solution was not chosen as the model fit was lower than all other models (excepting the one factor model), with the CFI and TLI being below acceptable limits, although the factor structure was theoretically plausible. Therefore, the best fitting and most theoretically plausible model was the Schmid-Leiman bifactor transformation of the five-factor CFA solution, which is fully reported below. See Table 1 for detail of all model fits.
The final model consisted of one general factor and five independent specific factors. The general latent factor corresponded well to the notion of a general distress factor [20,21]. The general factor loaded negative items positively and positive items negatively, such that a higher score in the general factor is equivalent to higher distress and lower scores are related to lower distress and higher positive self-image. This supported our hypothesis that there would be a general extended latent trait factor across well-being as well as mental illness constructs. The specific factors are independent of this general distress latent construct. We tested the validity of these specific traits (see below) to determine if they contributed additional psychological information that aid in the discrimination of novel constructs or are merely redundant variance from the analysis of no clear cut value to detecting mentally well or ill individuals.
See Table C in the S1 Materials for the general and specific factor loadings (as well as the positive item methods factor) on each item as well as thresholds detailing how each item relates to the general distress and specific factors, (see footnote on Table C in S1 Materials for more details). Evaluation of the thresholds demonstrated that negative items generally provided good discrimination at only the high end of the distress and specific factors. This contrasted with positive items, which showed equal discrimination at the high (indicating low agreement with positive traits) as well as low end of the distress and specific factors (indicating high agreement with positive traits). These findings support our assertion that the inclusion of positive well-being items increases precision of the underlying construct of mental health.
Evaluating the severity thresholds also indicated that psychotic-like experience items, particularly those unrelated to social awareness, loaded very highly on the distress continuum, supporting Stochl et al.’s [21] contention that psychotic-like experiences are on the extreme end of a common mental distress continuum.
Specific factor 1 consisted of all 13 items from the well-being scale; we refer to this as a self-confidence factor. All items loaded positively, indicating that higher scores relate to higher self-confidence. Revealing this independent factor supports our speculation that there are independent positive constructs in the population that may aid identification of low risk sub populations of youth.
Specific factor 2 had nine items consisting of the antisocial behaviour items plus feeling like a bad person; we refer to this as an antisocial factor. All nine items loaded positively, indicating that higher scores related to increased antisocial behaviour.
Specific factor 3 had seven items measuring worry and being afraid all the time; a worry factor. All items loaded positively, indicating that higher scores related to increased levels of worry.
Specific factor 4 had 17 items measuring aberrant thinking, including obsessional/compulsive and psychotic-like experience items; an aberrant thought factor. All items loaded positively, indicating that higher scores related to increased levels of aberrant thoughts.
Specific factor 5 contained 30 indicators measuring unhappiness and anhedonia (absence of feelings) items, suicidality, negative self-comparison to others, as well as items relating to positive well-being; a mood factor. All negative items loaded positively, while positive self-esteem and well-being items loaded negatively, indicating that higher scores indicated increased mood problems.
Fig 1A shows the standard error of measurement across all of the traits and demonstrates that all factors have high measurement precision across both high and low ends of each trait. However, there was a general pattern for lower measurement precision for the antisocial, worry and aberrant thought factors on the negative side of the trait. This pattern corresponds to traits with only negative items, whereas the self-confidence, mood and the general factor have consistent measurement precision across the entire trait, likely due to a mixture of both positive and negative mental health items. This supports our previous findings regarding the equal discrimination of positive items along the general distress continuum according to the severity thresholds.
A. Standard Error of Measurement for the general factor and specific factors. B. Standard Error of Measurement for each Narrow Band measurement* Factor Score. *MFQ = Moods and Feelings Questionnaire; RCMAS = Revised Children’s Manifest Anxiety Scale; r-LOI = revised Leyton Obsessional Inventory; ABQ = Antisocial Behaviour Questionnaire; RSE = Rosenberg Self Esteem Scale; SPQ = Schizotypal Personality Questionnaire; WB = Warwick-Edinburgh Mental Well-Being Scale.
For comparison purposes, we investigated the standard error of measurement of each of the individual narrow band instruments. Each instrument was turned into a factor with identical parameters as used above and showed a much more variable standard error of measurements across the latent traits (see Fig 1B). This result clearly indicates that similar levels of precision cannot be obtained across the entire range of a trait by simply using the narrow band measurements.
Age and gender differences
Fig 2 shows that there are both linear and non-linear age trends as well as gender effects, which vary between factors.
* Females: N = 224 for 14–15 year olds; N = 257 for 16–17 year olds; N = 234 for 18–19 year olds; N = 228 for 20–21 year olds; N = 261 for 22–24 year olds. Males: N = 199 for 14–15 year olds; N = 222 for 16–17 year olds; N = 200 for 18–19 year olds; N = 199 for 20–21 year olds; N = 204 for 22–24 year olds.
For the general factor (distress) girls had significantly higher overall scores than boys (β = .26, 95%CI(.18,.34,), p < .001, η2 = .02). This gender effect was qualified by an age grouping by gender interaction (F(4,2218) = 3.74; p < .005), indicating a different pattern throughout adolescence for girls and boys. For girls, there was a significant increase in general distress from early (age 14–15) to middle adolescence (16–17; p < .05), but no further significant differences to girls aged 18–19, 20–21 or 22–24 (all p’s > .10). General distress was elevated in middle adolescence (16–17) compared to early adulthood (20–21 and 22–24, ps < .01). Girls in late adolescence (18–19) also had higher general distress than in adulthood (22–24; p < .05). For boys there was a different pattern, with reduced general distress in early (14–15) and middle adolescence (16–17) compared to late adolescence (18–19; ps < .05). There was also significantly reduced distress in early adolescence (age 14–15) when compared to adulthood (20–21 and 22–24; ps < .05).
For the self-confidence specific factor there was no overall difference between girls and boys nor any interaction with age category, however there were differences across the age categories (F(4,2222) = 6.69; p < .001). For both girls and boys, there were higher levels of self-confidence in early adolescence (14–15) than adulthood (20–21 and 22–24; ps < .05). For both genders, there were also increased self-confidence in late adolescence (18–19) compared to early adulthood (20–21; ps < .05). For girls only, there was increased self-confidence in early (14–15) compared to middle adolescence (15–16; p < .05). For boys only, there was increased self-confidence in middle adolescence (15–16) compared to early adulthood (20–21; p < .05).
For the antisocial specific factor boys had significantly higher overall scores than girls (β = .27, 95%CI(.21,.33), p < .001, η2 = .04). This gender effect was qualified by an age grouping by gender interaction (F(4,2218) = 2.63; p < .05), indicating a different pattern throughout adolescence for girls and boys. For girls, there were no differences between early, middle and late adolescence, but each of these ages had higher antisocial traits than was found at age 20–21 (all p < .001). For boys, there were lower antisocial traits in adulthood (22–24) than in early (14–15) and middle (16–17) adolescence (ps < .05). There were also increased antisocial traits in middle (16–17) compared to late (18–19) adolescence (p < .05).
For the worry specific factor girls had significantly higher overall scores than boys, β = .24, 95%CI(.18,.31), p < .001, η2 = .02, but there was no interaction with age category, however there were differences across the age categories (F(4,2222) = 15.80; p < .001). For both girls and boys, there was a significant increase in worry traits from ages 14–15 and 16–17 to ages 20–21 and 22–24 (all ps < .005). Additionally, for girls only there was a significant increase from age 18–19 to age 20–21 (p < .005) and 22–24 (p < .001). For boys only, there was additionally significant increases in worry traits from ages 14–15 and 16–17 to age 18–19 (ps < .005).
For the aberrant thoughts specific factor there was no overall difference between girls and boys nor any interaction with age category, however there were differences across the age categories (F(4,2222) = 10.64; p < .001). For both girls and boys, there was a decrease in aberrant thought traits from all age periods when compared to age 22–24 (all ps < .05). For girls, there was an increased level of aberrant thoughts at age 14–15 when compared to ages 16–17, 18–19, and 20–21 (all ps < .05). For boys, there was significantly increased aberrant thoughts at age 16–17 when compared to age 20–21 (p < .05).
The mood specific factor did not differ overall between girls and boys, but there was a significant interaction between gender and age grouping (F(4,2217) = 2.59; p < .05). There was overall no main effect of age groupings for girls (F(4,1198) = 2.05; p = .09). For boys, there was a significant increased level of mood scores from early adolescence (14–15) when compared to ages 16–17, 18–19, and 22–24 (all ps < .05).
Full regression results of the specific age comparison can be found in Table D in the S1 Material. In addition to this analysis, the final model of all latent structures considered were run with just males, females and within the age groupings separately. A good fit the data from the SL 5 specific bifactor model was found for both genders and all age bands. See Supplementary Table B in S1 Materials for all fit indices.
Latent profile analysis
Although individuals are frequently selected for investigation on the basis of high or low scores on single behavioural measures, multidimensional behavioural patterns are present in the majority of the population. Using a cut-off on any one measure alone will not capture the multidimensional nature of mental and behavioural states. We therefore investigated whether all the computed latent traits can be used to derive person-centred distinct subgroups. We used Latent Profile Analysis in order to reveal the best fitting number of subgroups with distinct patterns across all factors. We viewed this aspect of our analysis as proof of principle and hypothesis generating, aiming to reveal distinct sub populations based on the latent factor trait patterns in the sample.
The best fitting ‘class’ structure is summarised in Fig 3. See S1 Materials (p.35-36) for details on this analysis and Table 2 for the fit indices.
The general distress factor contributed to defining four subgroups while the (distress independent) specific worry factor helped define three population subgroups. The specific antisocial factor helped define five subgroups, but the self-confidence and aberrant thinking factors help define only one subgroup each. The specific mood factor did not vary above or below .25 SD in any subgroup, therefore only nominally contributed to discrimination of any subgroup.
Subgroup 1 (High Antisocial and Low Worry Subgroup) is defined by scores on two latent traits: a moderately high score (+1.25sd) on the specific antisocial factor with a lower score (-0.25sd) on the specific worry factor and consists of 7% of the sample, two thirds of which are boys.
Subgroup 2 (High Distress and High Worry Subgroup) is defined by a set of differential scores on three latent traits: high general distress (+.75sd) together with marginally higher worry (+0.25sd) and moderately lower (-0.5sd) antisocial factor scores. This subgroup consists of 16% of the sample of which three-quarters are girls.
Subgroup 3 (Low Distress and Reduced Antisocial Subgroup) is also defined by two latent trait scores: low general distress levels (-.5sd) and reduced antisocial traits (-.25sd) and no quantitative differences in any other latent trait. This subgroup consists of 40% of the sample with equal proportions of boys and girls.
Subgroup 4 (High Distress Only Subgroup) is defined by only one latent trait score: somewhat higher (+0.5sd) general distress levels. This subgroup consists of 19% of the sample, of whom three-fifths are girls.
Subgroup 5 (High Antisocial Only Subgroup) is also defined by only one latent trait score: a higher (+0.75sd) antisocial traits. This subgroup consists of 17% of the sample, of whom two-thirds are boys.
Finally, subgroup 6 (High Antisocial and Aberrant Thinking Subgroup) is defined by four latent trait scores: slightly higher on general distress and self-confidence factors (+0.25sd), very high antisocial (+2sd), moderately low worry (-0.5sd) and moderately high aberrant thinking (+0.5sd). This subgroup consists of 2% of the sample and four fifths are boys.
Age effects on class membership.
The LPA analysis was repeated splitting by the sample at age 19 to evaluate whether there are age sensitive alterations in how the behavioural factors delineate subgroups of individuals. Age 19 was chosen due to the central location in the age distribution. Overall, there was a strikingly high correspondence between the solutions for those 18 and under with those 19 and over, which suggested no marked age effects in the latent profile structure. The main age difference was that the latent profile subgroups for those 19 and over showed much more differentiation in the self-confidence and aberrant thoughts specific factors and slightly less differentiation in the antisocial specific factors. In particular, the antisocial factor was reduced in Subgroup 5. This indicates that by adulthood the self-confidence and aberrant thoughts specific factors may become increasingly important in differentiating subgroups of individuals, while the influence of the antisocial specific factor reduces. See supplemental materials for details of the age split analysis.
Discriminant validity
A summary of the findings from the initial discriminant validity analysis of the general distress and specific factors and LPA subgroups are shown in Tables 3 and 4.
Hazardous behaviours.
Elevations in both the general distress factor and the specific antisocial factor were independently associated with all four hazardous behaviours (increased alcohol, cannabis, illegal drug use and non-suicidal self-injury [NSSI]). Elevations in the specific mood factor were only associated with the occurrence of NSSI. In contrast, the specific factors worry, aberrant thoughts and self-confidence were not related to any hazardous behaviours.
Subgroups with increased levels of antisocial traits (High Antisocial and Low Worry, High Antisocial Only, and High Antisocial and Aberrant Thinking Subgroups) were the only subgroups to have increased rates of both cannabis and other illegal drug use. Alcohol consumption was also increased in the High Antisocial and Aberrant Thinking Subgroup.
Interestingly, increased NSSI was associated with the High Distress and High Worry subgroup while the Low Distress and Reduced Antisocial subgroup was associated with lower reporting of this behaviour. In addition, the Low Distress and Reduced Antisocial subgroup was associated with reduced use of cannabis and illegal drug use. Furthermore, the High Distress Only Subgroup showed normative levels on all hazardous behaviours.
Treatment for mental health difficulties.
Higher scores on general distress were associated with treatment for mental ill health. This factor and the specific mood factor were both associated with treatment specifically for clinical depression. There were no other associates with the other specific factors.
Amongst the subgroups, members of the High Distress and High Worry Subgroup were twice as likely to have experienced any treatment for mental ill health as well as twice as likely to have experienced treatment for depression.
Conversely, members of the Low Distress and Reduced Antisocial Subgroup were less likely to have experienced any type of treatment for mental ill health. It is of note that the High Distress Only Subgroup showed normative levels (average for the entire sample) of treatment for mental ill health, even though this subgroup was characterised by high general distress only. The remaining subgroups also showed normative levels.
Overall, there is a greater association signal with treatment for mental ill health as well as hazardous behaviours for subgroups containing patterns of traits rather than single traits alone. This confirms the value of delineating the multidimensionality of behavioural repertoire in the population at large to appreciate the nature of associated risks.
Discussion
To our knowledge this is the first report to use both multivariate variable and person-centred strategies to evaluate the organisation of current mental health characteristics, encompassing a wide range of mental well-being and mental illness domains, in an adolescent and young adult population-based volunteer sample. This data-driven approach has revealed a novel latent structure of thoughts, feelings and behaviours, as well as characterising new population subgroups based on the patterning of their factor profiles.
Consistent with previous research finding a separation of internalising (usually measuring anxiety and depression symptoms) and externalising (usually measuring antisocial behaviours) symptoms in the structure of mental illness was revealed: internalising (general distress and specific mood and worry factors) and externalising (the specific antisocial factor) variable-centred factors emerged. However, the findings also indicate a different organisation to that reported in prior population based item/symptom and diagnosis factor studies (e.g., [13]). Previous studies had suggested that the behavioural structure consisted of only these two components (internalising/externalising), which were orthogonal to each other. Our results reiterate the importance of internalising and externalising dimensions, while replicating previous findings of a general distress or psychopathology factor underpinning both [18–26].
Many of the factors delineating the commonly measured depression and anxiety traits were subsumed entirely under the general distress psychopathology factor. Indeed, 33 of the 37 items loading only on the general distress factor came from the depression and anxiety measurements, and nearly all the items relating to somatic features (e.g., moved and walked more slowly than usual, didn’t sleep as well as usual) loaded only on the general distress factor. From this we draw several conclusions. First, somatic features may be a common feature across many psychiatric conditions, which indeed are reflected in the existing diagnostic criteria. Second, many items included in common measurements of anxiety and depression seem to be tapping into the general distress underlying both conditions. The items loading both on specific factors as well as the general factor were more personal, self-comparative to others, and related directly to mood, anhedonia and worry symptoms, indicating that these items may be more specific to distinct mental health conditions. These findings may be of relevance for the interpretation, construction and usability of these single domain measurements.
The idea that mental health and well-being might be best characterised by one general factor with specific factors relating to more unique aspects of either mental well-being or mental illness has important consequences for the overall measurement of mental health, with possible implications for the diagnosis of mental illness. The multidimensional nature of the bifactor solution need not necessarily indicate that we must measure all domains or items in order to accurately measure the general distress factor [56]. Based on the current results, we suggest a good measure of general distress should include a combination of positive and negative mental health items, but need not include all dimensions measured in this paper. As Reise argues as well, the bifactor model also allows for the measurement of subscales independent of the general distress factor, which is more difficult to model in the related hierarchical factor structure (with first and second order factors [56]). Thus, the bifactor nature of the structure endorsed here, as well as in numerous other papers [20–24], has important implications for the recognition of both mental well-being and mental illness. Summary of subscales or of the general factor total score are possible for both future research studies as well potentially within clinical populations where the computationally intensive methods utilised here are not feasible.
The person-centred analysis reveals for the first time a definable patterning of factor scores resulting in distinctive subgroups based on multidimensional behavioural characteristics in youth. This population heterogeneity may help to explain marked individual differences in symptom profiles and clinical outcome within a psychiatric syndrome. For example, clinical syndromes, such as major depression, are derived from 12 symptoms in differing patterns and therefore may well arise from distinctive population subgroups with putatively different aetiologies, as we have previously suggested [49].
While many of the subgroups were differentiated on many different factors, the specific factor measuring antisocial behaviour seemed to have a distinctive level in each subgroup. This trait had the largest mean variation in the subgroups, with the highest subgroup’s mean 2 SD above the sample average (although this subgroup only accounted for 2% of the sample). This perhaps reflects the distinctive pattern of externalising behaviour in adolescence and how individual propensities towards antisocial behaviour vary widely on different factors [57]. It may also be that differing levels of antisocial behaviour are highly predictive of other, more internalising traits. It is of note when the LPA analysis was replicated separately within the adolescent and young adult subsamples, the antisocial specific factor remained highly differential in the adolescent subgroups, but less so in the young adult subgroups. Further research evaluating the LPA structure in this sample is needed to further illuminate why the antisocial specific factors discriminates between the subgroups to such a high degree.
Added value of person-centred analyses
Revealing a common or general distress factor provides an explanation for covariance and trans-diagnostic effects found for self-report measures of ostensibly different symptom profiles as well as the commonly reported comorbidity across differing psychiatric diagnoses [18,19,22,49,58,59]. Such a factor subserving numerous clinical phenotypes may also explain why cognitive vulnerabilities, such as those revealed using computerised tasks, are associated with higher scores on multiple narrow band behavioural scales when completed concurrently [58]. However, it seems that the person-centred approach may improve the precision and validity of individual latent factors. For example, examining the general distress factor scores suggests higher scoring individuals are more likely to report increased hazardous behaviours and mental health service use. However, the High Distress Only subgroup, with median levels for all other factors, was not associated with either hazardous behaviours or treatment for mental ill health. These findings may help to explain why screening for general distress in either primary healthcare or within specific medical conditions is an inefficient way of finding and treating individuals who are in need of psychiatric care as it is likely that many higher scorers on such scales have low prevalence for mental illness [60–62]. Of course, it may be that increased distress alone may act as a generic risk factor for future mental health issues for a subset of individuals, although it is not indicative of a current or past disorder. Longitudinal studies based on this multidimensional structure and distinct subgroups are required to answer these important prognostic questions.
The discovery of a potential low mental illness risk subgroup
The inclusion of positive well-being items is likely to have made two important contributions to our results. First, there was increased precision of factor measurement across the entire trait in both the general distress and specific factors, indicating that positive items not only predict low distress but the absence of endorsement predicts high distress. Negative items only predict high distress, as their absence was not predictive of low distress. Second, we have revealed for the first time a subpopulation with very low risk of mental illness: the Low Distress and Reduced Antisocial subgroup, comprising 40% of our sample. Revealing ‘wellness’ subgroups to date have mainly been based on the absence of a positive response to illness items, which, as mentioned above, is empirically less satisfactory and valid than looking at positive endorsement on positive items. This subgroup with low mental illness risk is characterised by low general distress together with reduced antisocial tendencies and the population mean of all other factors. As preliminary evidence we note that this subgroup shows reduced levels of cannabis use, reduced history of engaging in non-suicidal self-harm and reduced history of treatment for mental ill health. The identification of this potential low mental illness risk subgroup might also provide new opportunities for investigating conceptually complex notions, such as the underpinning of a resilient human brain and mind [63]. This subgroup also provides new opportunities to reveal distinctions in the brain architecture between those at low risk and the mentally ill [64].
Age and sex effects and their implications for behavioural theory
We found gender and age differences within the general distress factor that replicates the common pattern in the depression literature of a mid-adolescent apex followed by a slow decline for girls and stability for boys [65]. This finding, while consistent with depression literature, does not replicate the absence of gender differences in previous studies of general distress or psychopathology factors observed in childhood and adulthood samples [18,23,24]. This may be due to the age range of this sample being 14 to 24 years when gender differences are most likely to occur in depression. However, the multidimensional structure does reveal some novel age and sex effects that may have a bearing on the incident risk of mental illnesses into the third decade.
Reflective worry and cognitive maturation.
The specific distress independent worry factor indexes an age increase in factor scores for both sexes consistent with the emergence of a more reflective top down cognitive process. For example, this could be a goal directed or model based system, available to allow flexible adaptation to environmental challenges [66].
The added value of the subgroup analysis is further illustrated when considering the potential adaptive role of (reflective) worry. Low worry in the High Antisocial and Low Worry and High Antisocial and Aberrant Thinking subgroups in combination with high antisocial and/or general distress factor scores may index insufficient deployment of top down cognitive abilities, which may help explain the elevated magnitude of association with hazardous behaviours in these subgroups. This is supported by evaluation of the High Distress and Worry subgroup, which was indexed by high worry and distress and lowered antisocial traits. This subgroup showed reduced levels of alcohol and cannabis use, but increased rates of NSSI and mental illness treatment. These findings indicate that increased reflective worry may either be an additive risk factor (indicating higher levels of negative or maladaptive worries) or an insufficient reflective ability to offset very high general distress scores. In all, the specific worry factor may index, within the subgroups, either insufficient cognitive control when decreased or maladaptive responses when increased, particularly in combination with increased general distress. While these musings are speculative and need further evaluation, we can be clear regarding the association between antisocial traits and this worry factor—higher antisocial traits are related to lower worry across all subgroups, while decreased antisocial traits relate to higher levels of worry.
Specific mood, self-confidence and aberrant thought factors.
The distress independent specific mood factor shows a further distinctive age and sex pattern not previously revealed in prior studies. This factor shows an increase in age for boys not revealed for girls. Taking the analyses as a whole and noting the associations between elevated factor scores on this mood factor with self-harm and treatment for mental ill health we conclude that elevated mood scores indexes maladaptive mood change.
It is striking that this mood factor does not contribute to defining any subgroups. This suggests that elevated mood factor scores alone are not sufficient to lead to the emergence of a clinical syndrome. However, the specific mood factor was striking in that it contained measurements of mood related to depression, yet we did not find the normal gender differences and only evidenced developmental changes for boys and not girls. Thus, this specific component may reflect more a stable “depressogenic” characteristic which is emerging in boys, but consistently present at the same level throughout our developmental window for girls. Given the lack of subgroup differentiation, it may be that the more state-like components of depression are expressed through the general distress factor. The more stable component of depression may be measured in the specific mood factor. This may relate to cognitive biasing mechanisms associated with depressive disorders, potentially promoting negative evaluations of self, as these items featured heavily on this specific factor. Future research investigating longitudinal differences within this multivariate structure will seek to illuminate the psychological implications of this specific factor further [67].
Finally, the specific factors for self-confidence and aberrant thoughts both showed a decline with age, which was not sex differentiated. Furthermore, these factors alone are not associated with any increases in hazardous behaviours or treatment for mental ill health. The precise psychological implications of these traits are, however, not clear-cut. Their role in identifying factor based sub-groups is confined to the smallest and most deviant group where both are increased. It is not readily apparent what their function may be in this subgroup. More work on the self-confidence and aberrant thought factor is required to understand their potential contribution to mental health and illness in the population.
Limitations
This study is not without limitations. First, the measurement of mental illness is restricted to most, but not all, common domains likely to encompass the emergence of psychiatric disorders in the second decade of life. For example, we did not have our sample report specifically on mania or specific eating disorder items, such as body image distortion. Unfortunately, practical considerations limited the domains we were able to measure, as each participant filled in the questionnaires used in the study in addition to other questionnaires measuring different domains of interest (personality and environmental influences). Second, we cannot be sure these factors would be correct for younger or older populations. Indeed, our split age LPA indicates that the self-confidence and aberrant thoughts factors may be increasingly important in early adulthood, indicating a slight age shift in the patterning of the factors. Furthermore, natural variations in our sample (age split and gender) were only tested for measurement invariance in the most basic manner, by evaluating the model in each individual subgroup. While these results indicated a good model fit across both genders and all age bands, full measurement invariance was not conducted due to concerns about adding additional complexities to an already complex paper. Future research could investigate and validate the measurement invariance across the genders and age categories within this sample.
Additionally, there has been debate in the literature regarding the meaning of measurements and overarching constructs (such as mental wellness and illness) when loading on an overarching factor, such as our general distress factor. While some authors have concluded that numerous measurements that load strongly on a general distress factor indicate that the individual measurements are overlapping [10], we take a different approach. While we acknowledge that while many psychological well-being and mental illness questionnaires do tap into a common underlying construct (which we term general distress), we note that the validity and utility of the questionnaires themselves should not be necessarily questioned or regarded as “straw men” measuring the same underlying construct. Indeed, the variety of questionnaires used in the current analysis, as well as the specificity found within our specific factors, argues against any explanation that the general distress factor is simply reflecting a lack of differentiation within the questionnaires evaluated. Rather, we believe that we are capturing the common variance across both mental wellness and illness (and the associated questionnaires) while allowing specific, more unique variance to be captured by the specific factors, which often (but not always) reflect the underlying questionnaires used. No single measure was entirely subsumed under the general distress factor, although we have already discussed that the larger proportion of anxiety and depression items only related to the general distress factor. This indicates that these common mental states inherently have a larger component of cross-diagnostic general distress. Additionally, recent research within a large twin sample has found that the general factor (with mental illness indicators) was moderately heritable, which the authors indicate argues against the idea that the general factor is a methodological artefact [68].
There is a wider debate on the usefulness and interpretation of these specific factors. As reported by Chen et al. [37] specific factors can provide additional and useful information above and beyond the general factor, which was seen in our discriminate validity findings. Indeed, Waldman and colleagues [68] found varying contributions from genetics and the environment across specific factors, indicating good signal within specific factors. In the current study, all specific factors, with the exception of the self-confidence and worry specific factors, showed some associations with hazardous behaviours or treatment for mental ill health (or both). The self-confidence and worry specific factors instead showed discrimination within the subgroups as well as clear development trends. Thus, our five specific factors appear to be useful constructs above and apart from the general factor. We are currently further validating and extending this work, specifically within longitudinal analyses, which will help clarify the precise meaning and information contained within these specific factors.
Another limitation of this study is the possibility that some of our participants may have been biologically related. This was investigated and we found 84 known sibling pairs in the sample and four households with three siblings, so in total there were at least 180 individuals with some relatedness to other participants. However, our sampling framework allowed us to identify relatedness only on the basis of participant name and address. We conducted a control analyses that indicated the results were replicated when controlling for the known relatedness within the sample. Therefore, this limitation in our sampling framework does not appear to have had a substantial influence on our results.
Both a strength and limitation of the study is the sample size. The sample size is more than adequate for the latent trait and LPA analysis, but it remains an empirical question whether at the population level there are even more unique subgroups of individuals that this study’s sample size (and recruitment parameters) do not allow us to delineate. A strength of this study, however, was the inclusion of self-esteem and well-being items, which added a positive dimension to the consideration of the structure of mental health and allowed for the measurement of both well and ill health. Finally, our study is confined to cross sectional analysis and therefore requires replication and an examination of prognostic and predictive validity of the traits and sub groups described. In this regard, the findings represent a proof of principle investigation of mental health structure in the youth population at large.
Conclusion
In summary, the use of two formal multivariate mathematical models reveals a unique structural organisation and patterning of self-reported mood, feelings, thoughts and behaviours in a youth population at large. The multidimensional bifactor solution both replicates previous findings [e.g., 10,20,23,24,27] regarding the structure of mental health as well as extends the literature by using a wide range of indicators of both mental illness and mental well-being. Further to evaluating the structure of these symptoms, we have established differing subgroups of individuals across the variable-centred factors. The combination of both methodologies allows a better context for understanding and evaluating these factors, and in particular has allowed an identification of a low risk subgroup with characteristics that may index resilience and a potential differential role of the specific worry factor to hazardous behaviours and mental health treatment, dependent on the levels of the general distress and other specific factors. The combination of variable-centred and person-centred approaches is suggested for future research.
Supporting information
S1 Materials. Provides additional information regarding alternative factor structures evaluated but not retained, further information regarding analytic procedure, and more detailed statistical analysis.
https://doi.org/10.1371/journal.pone.0175381.s001
(DOCX)
Acknowledgments
This research is supported by a Wellcome Trust Strategic Award (reference: 095844) to Ian Goodyer, Ray Dolan, Ed Bullmore, Peter Jones and Peter Fonagy which provides core funding for the Neuroscience in Psychiatry Network (NSPN.org.uk). Michael Moutoussis is funded by the UCLH Biomedical Research Council. Additional support was provided by the NIHR Cambridge Biomedical Research Centre. Peter Fonagy is in receipt of a National Institute for Health Research (NIHR) Senior Investigator Award (NF-SI-0514-10157). Peter Fonagy was in part supported by the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Barts Health NHS Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The authors acknowledge the whole of the NSPN team for ongoing feedback on this work from the early tentative stages, to the presentation of the final results. Particular thanks are due to Elzbieta Polek for help in thinking through psychometric problems and being a general sounding board and to Janet Bultitude for a useful perspective on the clarity of the manuscript. A huge amount of thanks goes to Cinly Ooi for collating the database, to Becky Inkster and Gita Prabhu for coordinating the study and to Laura Villis and Junaid Bhatti for providing invaluable administrative assistance in sending out the packs and entering the packs back in. Finally, we are grateful to all participants in the Neuroscience in Psychiatry Network without whom there would be no research programme.
Author Contributions
- Conceptualization: MCS PBJ IMG PF ETB RJD.
- Data curation: MCS SN.
- Formal analysis: MCS UT.
- Funding acquisition: IMG PBJ PF ETB RJD.
- Investigation: UT IMG PBJ.
- Methodology: MCS SN UT.
- Project administration: IMG.
- Software: MCS UT.
- Supervision: IMG PBJ.
- Validation: MCS SN.
- Visualization: MCS.
- Writing – original draft: MCS IMG PBJ.
- Writing – review & editing: MCS SN PBJ PF ETB RJD MM UT IMG.
References
- 1.
American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorder. Arlington, VA: American Psychiatric Publishing.
- 2. Keyes CLM (2002) The mental health continuum: From languishing to flourishing in life. Journal of Health and Social Behavior 43: 207–222. pmid:12096700
- 3. Keyes CLM (2007) Promoting and protecting mental health as flourishing—A complementary strategy for improving national mental health. American Psychologist 62: 95–108. pmid:17324035
- 4. Blakemore SJ, Choudhury S (2006) Development of the adolescent brain: implications for executive function and social cognition. Journal of Child Psychology and Psychiatry 47: 296–312. pmid:16492261
- 5. Sisk CL, Zehr JL (2005) Pubertal hormones organize the adolescent brain and behavior. Frontiers in Neuroendocrinology 26: 163–174. pmid:16309736
- 6. Veit CT, Ware JE (1983) The Structure of Psychological Distress and Well-Being in General Populations. Journal of Consulting and Clinical Psychology 51: 730–742. pmid:6630688
- 7. Masse R, Poulin C, Dassa C, Lambert J, Belair S, Battaglini A (1998) The structure of mental health: Higher-order confirmatory factor analyses of psychological distress and well-being measures. Social Indicators Research 45: 475–504.
- 8. Headey B, Kelley J, Wearing A (1993) Dimensions of Mental-Health—Life Satisfaction, Positive Affect, Anxiety and Depression. Social Indicators Research 29: 63–82.
- 9. Keyes CLM (2005) Mental illness and/or mental health? Investigating axioms of the complete state model of health. Journal of Consulting and Clinical Psychology 73: 539–548. pmid:15982151
- 10. Bohnke JR, Croudace TJ (2015) Calibrating well-being, quality of life and common mental disorder items: psychometric epidemiology in public mental health research. Br J Psychiatry.
- 11. Liddle PF (1987) The Symptoms of Chronic-Schizophrenia—a Reexamination of the Positive-Negative Dichotomy. British Journal of Psychiatry 151: 145–151. pmid:3690102
- 12. Mellers JDC, Sham P, Jones PB, Toone BK, Murray RM (1996) A factor analytic study of symptoms in acute schizophrenia. Acta Psychiatrica Scandinavica 93: 92–98. pmid:8686489
- 13. Krueger RE, Markon KE (2006) Reinterpreting comorbidity: A model-based approach to understanding and classifying psychopathology. Annual Review of Clinical Psychology 2: 111–133. pmid:17716066
- 14. Markon KE (2010) Modeling psychopathology structure: a symptom-level analysis of Axis I and II disorders. Psychological Medicine 40: 273–288. pmid:19515267
- 15. Wright AGC, Krueger RF, Hobbs MJ, Markon KE, Eaton NR, Slade T (2013) The Structure of Psychopathology: Toward an Expanded Quantitative Empirical Model. Journal of Abnormal Psychology 122: 281–294. pmid:23067258
- 16. Conway CC, Starr LR, Espejo EP, Brennan PA, Hammen C (2016) Stress Responsivity and the Structure of Common Mental Disorders: Transdiagnostic Internalizing and Externalizing Dimensions Are Associated With Contrasting Stress Appraisal Biases. Journal of Abnormal Psychology 125: 1079–1089. pmid:27819469
- 17. Lahey BB, Van Hulle CA, Singh AL, Waldman ID, Rathouz PJ (2011) Higher-order genetic and environmental structure of prevalent forms of child and adolescent psychopathology. Arch Gen Psychiatry 68: 181–189. pmid:21300945
- 18. Caspi A, Houts RM, Belsky DW, Goldman-Mellor SJ, Harrington H, Israel S, et al. (2014) The p Factor: One General Psychopathology Factor in the Structure of Psychiatric Disorders? Clin Psychol Sci 2: 119–137. pmid:25360393
- 19. Lahey BB, Applegate B, Hakes JK, Zald DH, Hariri AR, Rathouz PJ (2012) Is there a general factor of prevalent psychopathology during adulthood? J Abnorm Psychol 121: 971–977. pmid:22845652
- 20. Brodbeck J, Abbott RA, Goodyer IM, Croudace TJ (2011) General and specific components of depression and anxiety in an adolescent population. BMC Psychiatry 11: 191. pmid:22151586
- 21. Stochl J, Khandaker GM, Lewis G, Perez J, Goodyer IM, Zammit S, et al. (2015) Mood, anxiety and psychotic phenomena measure a common psychopathological factor. Psychol Med 45: 1483–1493. pmid:25394403
- 22. Laceulle OM, Vollebergh WAM, Ormel J (2015) The Structure of Psychopathology in Adolescence: Replication of a General Psychopathology Factor in the TRAILS Study. Clinical Psychological Science 3: 850–860.
- 23. Patalay P, Fonagy P, Deighton J, Belsky J, Vostanis P, Wolpert M (2015) A general psychopathology factor in early adolescence. Br J Psychiatry.
- 24. Carragher N, Teesson M, Sunderland M, Newton NC, Krueger RF, Conrod PJ, et al. (2016) The structure of adolescent psychopathology: a symptom-level analysis. Psychological Medicine 46: 981–994. pmid:26620582
- 25. Castellanos-Ryan N, Briere FN, O'Leary-Barrett M, Banaschewski T, Bokde A, Bromberg U, et al. (2016) The Structure of Psychopathology in Adolescence and Its Common Personality and Cognitive Correlates. Journal of Abnormal Psychology 125: 1039–1052. pmid:27819466
- 26. Reininghaus U, Bohnke JR, Hosang G, Farmer A, Burns T, McGuffin P, et al. (2016) Evaluation of the validity and utility of a transdiagnostic psychosis dimension encompassing schizophrenia and bipolar disorder. British Journal of Psychiatry 209: 107–113. pmid:26989099
- 27. Murray AL, Eisner M, Ribeaud D (2016) The Development of the General Factor of Psychopathology 'p Factor' Through Childhood and Adolescence. Journal of Abnormal Child Psychology 44: 1573–1586. pmid:26846993
- 28. Insel TR, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K, et al. (2010) Research Domain Criteria (RDoC): Toward a New Classification Framework for Research on Mental Disorders. American Journal of Psychiatry 167: 748–751. pmid:20595427
- 29. Cuthbert BN, Insel TR (2013) Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med 11: 126. pmid:23672542
- 30. Hyman SE (2010) The Diagnosis of Mental Disorders: The Problem of Reification. Annual Review of Clinical Psychology, Vol 6 6: 155–179.
- 31. Sanislow CA (2016) Connecting Psychopathology Meta-Structure and Mechanisms. Journal of Abnormal Psychology 125: 1158–1165. pmid:27819475
- 32. Xu MK, Gaysina D, Barnett JH, Scoriels L, van de Lagemaat LN, Wong A, et al. (2015) Psychometric precision in phenotype definition is a useful step in molecular genetic investigation of psychiatric disorders. Transl Psychiatry 5: e593. pmid:26125156
- 33. Forbes MK, Tackett JL, Markon KE, Krueger RF (2016) Beyond comorbidity: Toward a dimensional and hierarchical approach to understanding psychopathology across the life span. Development and Psychopathology 28: 971–986. pmid:27739384
- 34. Carragher N, Krueger RF, Eaton NR, Slade T (2015) Disorders without borders: current and future directions in the meta-structure of mental disorders. Social Psychiatry and Psychiatric Epidemiology 50: 339–350. pmid:25557024
- 35. Wang L, Lin WP (2011) Wording effects and the dimensionality of the General Health Questionnaire (GHQ-12). Personality and Individual Differences 50: 1056–1061.
- 36. Hankins M (2008) The factor structure of the twelve item General Health Questionnaire (GHQ-12): the result of negative phrasing? Clin Pract Epidemiol Ment Health 4: 10. pmid:18435842
- 37. Chen FF, West SG, Sousa KH (2006) A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research 41: 189–225. pmid:26782910
- 38. Kessler RC, Mcgonagle KA, Zhao SY, Nelson CB, Hughes M, Eshleman S, et al. (1994) Lifetime and 12-Month Prevalence of DSM-III-R Psychiatric-Disorders in the United-States—Results from the National-Comorbidity-Survey. Archives of General Psychiatry 51: 8–19. pmid:8279933
- 39. Kessler RC, Chiu WT, Demler O, Walters EE (2005) Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry 62: 617–627. pmid:15939839
- 40. Bohnke JR, Croudace TJ (2015) Factors of psychological distress: clinical value, measurement substance, and methodological artefacts. Social Psychiatry and Psychiatric Epidemiology 50: 515–524. pmid:25682108
- 41. Jones PB (2013) Adult mental health disorders and their age at onset. Br J Psychiatry Suppl 54: s5–10.
- 42. Costello EJ, Angold A (1988) Scales to assess child and adolescent depression: checklists, screens, and nets. J Am Acad Child Adolesc Psychiatry 27: 726–737. pmid:3058677
- 43. Reynolds CR, Richmond BO (1978) What I Think and Feel—Revised Measure of Childrens Manifest Anxiety. Journal of Abnormal Child Psychology 6: 271–280. pmid:670592
- 44. Bamber D, Tamplin A, Park RJ, Kyte ZA, Goodyer IM (2002) Development of a Short Leyton Obsessional Inventory for children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry 41: 1246–1252. pmid:12364847
- 45.
Rosenberg M (1965) Society and the adolescent self-image. Princeton, N.J.,: Princeton University Press. xi, 326 p. p.
- 46. Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S, et al. (2007) The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes 5: 63. pmid:18042300
- 47. Raine A (1991) The SPQ—a Scale for the Assessment of Schizotypal Personality Based on DSM-III-R Criteria. Schizophrenia Bulletin 17: 555–564. pmid:1805349
- 48. Horwood J, Salvi G, Thomas K, Duffy L, Gunnell D, Hollis C, et al. (2008) IQ and non-clinical psychotic symptoms in 12-year-olds: results from the ALSPAC birth cohort. Br J Psychiatry 193: 185–191. pmid:18757973
- 49. Owens M, Herbert J, Jones PB, Sahakian BJ, Wilkinson PO, Dunn VJ, et al. (2014) Elevated morning cortisol is a stratified population-level biomarker for major depression in boys only with high depressive symptoms. Proc Natl Acad Sci U S A 111: 3638–3643. pmid:24550453
- 50. Dunn V, Abbot RA, Croudace T, Wilkinson P, Jones PB, Herbert J, et al. (2011) Profiles of Family-focused Adverse Experiences Through Childhood and Early Adolescence: The ROOTS Project a community investigation of adolescent mental health. BMC Psychiatry 11: 109. pmid:21736727
- 51. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG (2009) Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics 42: 377–381. pmid:18929686
- 52.
Brown TA (2006) Confirmatory Factor Analysis for Applied Research. New York, NY: The Guilford Press.
- 53.
StataCorp. (2013) Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.
- 54.
Muthén B, Muthén L (1998–2014) Mplus User's Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.
- 55. Lorenzo-Seva U, Ferrando PJ (2006) FACTOR: A computer program to fit the exploratory factor analysis model. Behavior Research Methods 38: 88–91. pmid:16817517
- 56. Reise SP (2013) The Rediscovery of Bifactor Measurement Models (vol 47, pg 667, 2012). Multivariate Behavioral Research 48: 461–461.
- 57. Mishra S, Lalumiere ML (2011) Individual differences in risk-propensity: Associations between personality and behavioral measures of risk. Personality and Individual Differences 50: 869–873.
- 58. Owens M, Goodyer IM, Wilkinson P, Bhardwaj A, Abbott R, Croudace T, et al. (2012) 5-HTTLPR and early childhood adversities moderate cognitive and emotional processing in adolescence. PLoS One 7: e48482. pmid:23209555
- 59. Lahey BB, Rathouz PJ, Keenan K, Stepp SD, Loeber R, Hipwell AE (2015) Criterion validity of the general factor of psychopathology in a prospective study of girls. Journal of Child Psychology and Psychiatry 56: 415–422. pmid:25052460
- 60. Meijer A, Roseman M, Delisle VC, Milette K, Levis B, Syamchandra A, et al. (2013) Effects of screening for psychological distress on patient outcomes in cancer: a systematic review. J Psychosom Res 75: 1–17. pmid:23751231
- 61. Fleer J, Tovote KA, Keers JC, Links TP, Sanderman R, Coyne JC, et al. (2013) Screening for depression and diabetes-related distress in a diabetes outpatient clinic. Diabet Med 30: 88–94. pmid:22924587
- 62. Thombs BD, Coyne JC, Cuijpers P, de Jonge P, Gilbody S, Ioannidis JP, et al. (2012) Rethinking recommendations for screening for depression in primary care. CMAJ 184: 413–418. pmid:21930744
- 63. Russo SJ, Murroughl JW, Han MH, Charney DS, Nestler EJ (2012) Neurobiology of resilience. Nature Neuroscience 15: 1475–1484. pmid:23064380
- 64. Goodkind M, Eickhoff SB, Oathes DJ, Jiang Y, Chang A, Jones-Hagata LB, et al. (2015) Identification of a Common Neurobiological Substrate for Mental Illness. Jama Psychiatry 72: 305–315. pmid:25651064
- 65. St Clair MC, Goodyer IM, Dunn V, Herbert J, Jones PB, Croudace T (2012) Depressive symptoms during adolescence: comparison between epidemiological and high risk sampling. Soc Psychiatry Psychiatr Epidemiol 47: 1333–1341. pmid:22037558
- 66. Dolan RJ, Dayan P (2013) Goals and Habits in the Brain. Neuron 80: 312–325. pmid:24139036
- 67. Neufeld S, St Clair MC, Brodbeck J, Jones PB, Fonagy P, Bullmore E, et al. (In Preparation) Longitudinal structure of thoughts and behaviours in young adults: stability, predictive ability, and change. Psychological Medicine.
- 68. Waldman ID, Poore HE, van Hulle C, Rathouz PJ, Lahey BB (2016) External Validity of a Hierarchical Dimensional Model of Child and Adolescent Psychopathology: Tests Using Confirmatory Factor Analyses and Multivariate Behavior Genetic Analyses. Journal of Abnormal Psychology 125: 1053–1066. pmid:27819467