http://orcid.org/0000-0002-9491-3665
Figures
Abstract
Introduction
Infancy is an important period of life; adverse experiences during this stage can have both immediate and lifelong impacts on the child’s mental health and well-being. This study evaluates the effects of offering the Incredible Years Parents and Babies (IYPB) program as a universal intervention.
Method
We conducted a pragmatic, two-arm, parallel pilot randomized controlled trial; 112 families with newborns were randomized to the IYPB program (76) or usual care (36) with a 2:1 allocation ratio. The primary outcome was parenting confidence at 20 weeks(Karitane Parenting Confidence Scale and Parental Stress Scale). Secondary outcomes include measures of parent health, parent-child relationship, infant development, parent-child activities, and network. Interviewers and data analysts were blind to allocation status. Multiple linear-regression analyses were used for evaluating the effects of the intervention.
Results
There were no intervention effects on the primary outcomes. Only one effect was detected for secondary outcomes, intervention mothers reported a significantly smaller network than control mothers (β = -0.15 [-1.85,-0.28]). When examining the lowest-functioning mothers in moderator analyses, we found that intervention mothers reported significantly higher parent stress (β = 5.33 [0.27,10.38]), lower parenting confidence (β = -2.37 [-4.45,-0.29]), and worse mental health than control mothers (β = -18.62 [-32.40,-4.84]). In contrast, the highest functioning intervention mothers reported significantly lower parent stress post-intervention (β = -6.11 [-11.07,-1.14]).
Conclusion
Overall, we found no effects of the IYPB as a universal intervention for parents with infants. The intervention was developed to be used with groups of low functioning families and may need to be adapted to be effective with universal parent groups. The differential outcomes for the lowest and highest functioning families suggest that future research should evaluate the effects of delivering the IYPB intervention to groups of parents who have similar experiences with parenting and mental health.
Citation: Pontoppidan M, Klest SK, Sandoy TM (2016) The Incredible Years Parents and Babies Program: A Pilot Randomized Controlled Trial. PLoS ONE 11(12): e0167592. https://doi.org/10.1371/journal.pone.0167592
Editor: Jacobus P. van Wouwe, TNO, NETHERLANDS
Received: July 25, 2016; Accepted: November 14, 2016; Published: December 14, 2016
Copyright: © 2016 Pontoppidan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be made publicly available to protect patient privacy. Data are available upon request from the corresponding authors
Funding: This work was supported by TrygFonden grant nr 7-12-0195 MP. https://www.trygfonden.dk/english. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Increasing numbers of studies with small children suggest that the youngest are at the highest risk of serious developmental harm. During infancy, while children are at their most malleable, they are also exposed to more potentially damaging experiences than older children. For example, the highest rates of child neglect and violent abuse occur when children are under five years old [1,2], with the most serious injuries and death caused by parental violence against children when infants are under one year old [3].
Infants are also more sensitive to disruptions in parental care than older children. Infants with mothers who suffer from severe stress or depression can show physiological, biochemical, and behavioral dysregulations shortly after birth, and have been found to be at increased risk of behavioral and mental problems [4]. These infants often show avoidance of caretakers and display high levels of distress and negative emotion [4]. For many of these children, these effects are the beginning of a trajectory of negative developmental and mental health outcomes throughout childhood and adulthood [5,6].
Early parent-child interactions have been shown as the key predictive factor for many early and late developmental outcomes [7]. Supporting parents in developing and applying positive parenting skills can prevent future problems and encourage healthy child development [8–16]. This study evaluates a program of parental support based on these child development goals.
The new program, specifically designed for parents with infants, has been added to the Incredible Years Series (IY) of treatment and prevention programs for children and families. Extensive Scandinavian and international research, from both the program developer and independent research groups, has demonstrated the effectiveness of the IY programs for older children [9,17–34].
A recent meta-analysis of IY parenting programs for three- to nine-year-old children shows that effect sizes for parent-reported outcomes in treatment studies (families who sought help) were higher (d = 0.50) than for indicated prevention studies (families with children with minimal symptoms) (d = 0.20) and selective prevention studies (families with high risk) (d = 0.13) [28]. A shortened version of the universal Basic IY parent program for children aged 3–8 was evaluated in an RCT, which showed reductions in harsh parenting, parent stress, and parent depression, and increases in positive parenting, and parent sense of competence; improved parenting was maintained four years after the program ended [16,35]. The IY Toddler BASIC program for children aged 1–3 has been evaluated in three trials with positive results [22,36,37], however, we still know less about the effects of the IY programs for children under three than we do for older children [29].
In 2010, the developer of the IY program extended the series to include the Incredible Years Parents and Babies program (IYPB) for families with infants from birth to one year. IYPB has not been evaluated for effectiveness in a randomized controlled trial (RCT), either in Scandinavia or internationally. However, a pre-post evaluation of IYPB in Wales found that both parenting competence and mental health significantly improved over time for those who participated in the program[38]. A second evaluation conducted in Wales with a control group, also found that mothers in both treatment and control groups significantly improved over time. A significant positive effect on observed mother sensitivity was identified, while no differences were found between the two groups on child development, parenting confidence, or parental mental well-being [39].
During the eight-week course of treatment in IYPB, mothers and fathers attend weekly group sessions with their infants. Group leaders (therapists) trained in IYPB conduct the sessions with approximately eight families. During the sessions, parents learn to observe, read, and respond in a sensitive manner to their babies’ cues and signals. They also learn how to understand babies as intelligent learners, how to provide physical and tactile stimulation for babies, how to take care of their own needs as parents, how to understand babies’ developmental processes and needs, and the importance of good care for infant brain development.
In Scandinavia, no programs with strong evidence of effectiveness are available for the treatment and prevention of developmental problems in infants (e.g., insecure attachment). The Ungsinn database (ungsinn.no), which is used for cataloging and rating programs available for children in Norway, lists only one parenting program for infants, and at present not enough evidence exists to support its effectiveness. In Denmark, the Circle of Security program is currently being evaluated in an RCT. Even fewer interventions, either in Scandinavia or internationally, are aimed at a universal sample of parents with infants [40–47]. Three of these programs are delivered postnatally in a group format: (1) Toddlers Without Tears (Australia), with three sessions from age eight months [45,48]; (2) Face to Face (Australia), with five sessions from three months [49]; and (3) an American trial of parent training, with eight sessions from age eight months [50]. None of the studies, however, finds any effects on child development or the parent-child relationship. When compared to these three interventions IYPB starts at an earlier age and offers significantly more sessions than Toddlers Without Tears and Face to Face. In this pilot RCT, we evaluate a more intensive intervention, offered as a universal intervention aimed at a community sample of parents with newborns.
The aim of this pilot trial was to evaluate the effects of the program offered as a universal intervention in Denmark on the parent-infant relationship, parent and infant well-being, infant development, and to establish parameters for a future full-scale trial. The secondary aims were to provide information on the usability of parent and infant measures, to test recruitment procedures and determine rates of recruitment and parent consent, to investigate the implementation and parents’ acceptance of IYPB in a universal setting, and to provide information on the cost of offering IYPB as a universal preventive program.
Methods
Study Design
The trial was a pragmatic, two-armed, parallel pilot RCT. Institutional review board approval was obtained from SFI–the Danish National Center for Social Research. Parents provided informed consent before participation. The trial was carried out according to CONSORT guidelines [51,52] and registered at ClinicalTrials.gov (reference number NCT01931917).
Participants and Recruitment
The eligible participants were mothers with infants living in the Ikast-Brande or Herning local authority area in Denmark. Where fathers were present, they were also invited to participate. All mothers in Denmark are entitled to 46 weeks of maternity leave following the birth of a child; the last 32 weeks can be split with the father. Virtually all families receive five or six free home visits from health visitors and three child-health visits to the general practitioner within the child’s first year [53,54]. Health visitors recruited families between August 2013 and August 2015. After the local contact person received initial consent from the family, the interviewer arranged a home visit, during which written consent was obtained from the mother (and the father, if applicable).
Inclusion Criteria
The trial included mothers who are able to read and write Danish and had infants aged 0–4 months. Ikast-Brande recruited only first-time mothers, whereas Herning included all mothers. Although families could have been excluded in the case of severe physical or mental disability in the parent or the child, or if the child was placed in out-of-home care, no families were excluded from participation.
IYPB Intervention
The IYPB program was developed to promote babies’ physical, emotional, and language development. The intervention aims at promoting a warm, nurturing parent-child relationship, and enhancing parent competencies [55]. Groups consisted of 6–8 parents and were led by two trained group leaders. Mothers brought their babies to the sessions, and partners were strongly encouraged to participate. The purpose of the group format is to stimulate shared learning and peer support networks. Parents practiced new skills with babies within the group, and were encouraged to try out new solutions at home as part of their weekly assignments [55].
The program consisted of eight two-hour sessions. To support the training and foster discussion in the group, during each session, group leaders showed video vignettes of real-life situations with parents and babies. The original American video vignettes were used with Danish subtitles. Group leaders used the IY baby brain poster to explain the importance of infant brain development and demonstrate how parents strengthen neural connections and can support their child’s healthy brain development. Parents also received a Danish translation of the book The Incredible Babies [56], which describes how to promote positive child development and includes a journal section for parents to use. The six parts covered during the course are titled: (1) "Getting to Know Your Baby"; (2) "Babies as Intelligent Learners"; (3) "Providing Physical, Tactile and Visual Stimulation"; (4) "Parents Learning to Read Babies’ Minds"; (5) "Gaining Support", and (6) "Babies’ Emerging Sense of Self" [55]. The group leaders followed a manual to ensure that the intervention was performed with fidelity.
Two group leaders were certified IYPB group leaders and two were in the process of gaining IYPB certification. The remaining group leaders were all experienced IY group leaders who were certified in the IY BASIC Parent Group and who attended three days of training sessions in IYPB. Group leaders attended supervision twice a year with an IYPB mentor.
Control
The families randomized to the control group received usual care (UC), consisting of four or five home visits from health visitors, open consultation hours at a local well-child clinic, voluntary participation in a social group of six local mothers, and extra support if needed (e.g., extra home visits, family therapy, or video-feedback intervention). Intervention families received IYPB in addition to UC.
Measures
Outcomes were collected through home visits at baseline (T1) and post-intervention (T2). The mean number of days between T1 and T2 was 146 (SD 28), with a range of 78–207 days. Given the difference in the number of weeks that the IYPB program ran, the range was large. For some groups half of the IYPB sessions were offered before the summer or the Christmas break, with the other half offered afterwards. Other groups finished within a continuous 12-week period. T2 data was collected between one and three weeks after the IYPB program. Baseline measures and the timing of measures are described in the trial protocol [57]. While both mothers and fathers could complete the questionnaire, only a few fathers did so (50 at T1, 14 at T2). Families received a 200 DKK (≈$30) gift card at each visit.
The primary outcomes were the 15-item Karitane Parenting Confidence Scale [58,59] and the 18-item Parenting Stress Scale [60]. Secondary parental outcomes included the 10-item Major Depression Inventory (MDI10) [61], the five item WHO-5 Well-Being Index [62,63], the 10-item Rosenberg Self-esteem Scale [64], and single items on parent health, and parent life satisfaction, support, and network. Secondary child outcomes included the 26-item Ages and Stages Questionnaire: Social-Emotional-2 experimental version (ASQ:SE-2e) [65,66] and single items on child health, temperament, height, and weight. Secondary relationship outcomes included the 10-item Mother and Baby Interaction Scale (MABISC) [67,68], three single items measuring parent and child interaction through singing and reading, and a 15-minute video of the mother and baby. The videos have not yet been coded; therefore, the results for this outcome will be presented in a separate paper. In addition, we collected demographic characteristics such as parent age, education, occupation, primary language spoken in the home, number of children, household budget, substance abuse, birth weight, gestational age, and child health, and whether parents cohabit or the mother lives alone, and whether housing is rented or owned. All trial outcomes are described in greater detail in the study protocol [57].
Randomization and Blinding
An independent researcher computed a random allocation list, stratified by the local authority, with a block size of three. A 2:1 (IYPB:UC) allocation ratio was used as a practical consideration as the providing health centers needed to have enough families allocated to the IY intervention to be able to start a group before the infants grew to old. Following completion of the baseline assessment, the interviewer informed a designated research administrator that the interview had concluded. The research administrator then randomized the families by adding their names to the randomization list in the order that the interviewer had provided. The health visitor informed each family which arm of the study they were allocated to. In cases where consent to treatment was withdrawn but the participant agreed to remain in the research study, the participant was followed to completion. Given the nature of the trial, participants and group leaders could not be blinded to condition. Interviewers, coders, and data analysts were blind to group allocation status.
Parent and Group Leader Satisfaction
Parents completed questionnaires after each session and a final evaluation after the last session. For Ikast-Brande municipality, final evaluation data from 82 parents who participated in the IYPB between January 2014 and July 2015 was analyzed [69]. Furthermore, qualitative interviews with group leaders and focus groups with parents were held in Ikast-Brande [70].
Statistical Analyses
We aimed to recruit 128 mothers based on power calculations for a clinical sample; however, before the trial began, local political authorities mandated that all families be offered a place in the program. We did not have further resources to collect additional data for a larger sample or the required information to calculate power for a universal population, we therefore changed the focus to a pilot trial for a universal sample.
Categorical data is presented as numbers and percentages, and continuous data as means and standard deviations. For demographic data, we applied independent t-tests for testing differences for continuous variables, while Chi-square tests were applied for categorical variables. The trial included two sets of twins, both in the IYPB group. To account for the lack of independence between twins, we selected the first twin and ignored the other for parent outcomes but kept both twins for child outcomes. We performed intention to treat analyses for primary and secondary outcomes with multiple regression analysis, including controls for site and baseline score. No further covariates were included, as variables such as the mother’s age, education, and parity had no effect on the results.
Of the 112 mothers assessed at T1, eight were lost due to attrition at T2. Depending on the mechanism by which data went missing, attrition could lead to bias in the estimates. To deal with attrition, we first tested the assumption of data being missing completely at random (MCAR) by creating an indicator variable for observations missing at T2 and fitting it with a logit model. All baseline measures of the outcome variables and other covariates, along with the treatment dummy, were included as predictors. Although we found no significant predictors at the 5% level, five out of 18 predictors were significant at the 10% level, indicating that the data was missing at random (MAR). Although the assumption of MAR data is not testable, imputation is still shown to produce less biased results than listwise deletion [71]. Furthermore, should the data really be MCAR instead of MAR, listwise deletion would lead to unbiased, albeit possibly inefficient, estimates. As a result, we generated 200 datasets with multiple imputation [72] using Stata’s multiple imputation (mi) impute procedure on chained equations.
Although not all variables were normally distributed, we applied the ordinary least squares (OLS) regression, as the OLS is consistent even without normality. A two-tailed test α = 0.05 was implemented for all analyses. We calculated effect sizes by dividing the adjusted mean difference between the trial arms by the pooled standard deviation. We used robust standard errors to account for group effects. We applied paired sample t-tests for testing total group change over time. As described in the protocol, we examined differential effects by comparing IYPB to UC in three subsamples: (1) parents with baseline scores in the clinical range of the measures, (2) parents scoring within the lowest 25% of the distribution at baseline, and (3) parents scoring within the lowest 50% of the distribution at baseline. For subsamples (2) and (3) we performed a moderator analysis, including an interaction term between a dummy for being in the 25% or 50% lowest/highest scoring group and a dummy for intervention allocation. Analyses were performed with Stata version 14.
Results
Fig 1 presents the flow diagram of participants in the trial. Of the 125 families who had given their initial consent, 13 withdrew consent when the interviewer contacted them to schedule a home visit. Of the 112 families that were randomized, 76 were allocated to IYPB and 36 to UC. Eight families dropped out before T2 assessment. A larger dropout was observed in the intervention group (seven) than in the control group (one). Four of the seven mothers who dropped out of the study intervention group, did not start or dropped out of the IYPB group. Some statistically significant differences were found between the families who dropped out and those who did not; dropout mothers were older (5.76 years older on average, p = 0.0009), had more children (0.80 more children on average, p = 0.002), reported having attended more open house sessions with health visitors, and had a larger network at baseline for help with practical issues in the home.
Table 1 presents descriptive statistics for the two study conditions. We found no significant differences in demographic characteristics between IYPB and UC.
The mothers who participated in the trial appear to be relatively representative of the general population in relation to most demographic characteristics, with the exception of education level. When comparing the education level of the mothers in the trial with data from Statistics Denmark for all mothers in this age group (20–39), both in Denmark as a whole and for the two specific local authorities, Table 2 shows that the participating mothers generally had higher levels of education than the overall population.
That the families who elect to participate in research studies often have more education than the population mean is a well-known tendency, one that has implications for the generalizability of trial results. Although the education level was higher than that of the general population, the mental health of the mothers appears to be representative of the general population. The total mean of mental health, as measured by WHO5 at T1, is 62.6 at baseline, a number lower than the mean of 67 for Danish women in general [73], but rose to 69.9 at T2. The level of clinical depression in the recruited mothers is 6% at T1, in line with the reported 7–8% for the Danish population when the child is five weeks old [74].
Recruitment
Health visitors had different experiences with the recruitment process. Some found it challenging to ask families to participate in the trial, while others felt confident. During the trial, we asked the more confident recruiters at a meeting to share their strategies and experiences with the others. One of the two sites adopted a more insistent recruitment strategy, recruiting 62 mothers into seven IYPB groups (8.9 mothers per group). In contrast, the less insistent site recruited 50 mothers to 18 groups (2.8 mothers per group). Dropout rates were 10% for the insistent site and 4% for the less insistent site. The two-minute YouTube recruitment video was viewed 88 times, but we have no information on who viewed it or whether it was influential in the decision to participate in the study.
Participation
While both sites registered parent participation, only one site provided full information on the number of completed sessions. Ikast-Brande recruited 50 families, 33 of which were allocated to intervention. Thirty mothers participated in the IYPB group, as three did not show up. On average, the participating mothers completed 6.3 sessions. Fathers participated less often than mothers;—a total of 17 fathers participated, and they completed four sessions on average. We found no difference between IYPB and UC in terms of either the number of extra visits families received from health visitors or the proportion of families who received extra visits in Ikast-Brande. IYPB families received on average 1.48 extra visits; UC families received on average 1.82 extra visits. A total of 33% of IYPB families and 29% of UC families received no extra visits. For Ikast-Brande, IYPB did not reduce the number of extra home visits families received. The protocol describes Complier Average Causal Effects (CACE) analyses for two levels of participation: (1) mothers who participated in at least three of the eight sessions, and (2) mothers who participated in at least six of the eight sessions. However, as we were able to obtain detailed participation data from only one site, we could not perform these analyses.
Effect Evaluation
Table 3 shows the means for the full trial sample at T1 and T2. All outcome measures based on rating scales significantly improve from T1 to T2, except for self-esteem (RSS). Most single items (loneliness, confidants, parent health, child health, overall life satisfaction) do not change over time. One of the single items(parent report of child temperament)improves significantly over time, whereas mother reports of both family budget and network for help with practical issues decline significantly from T1 to T2. Because some of the measures were not strictly normally distributed we performed a sensitivity analysis by applying a non-parametric test (Wilcoxon signed rank test) to the non-imputed data. With the exception of self-esteem, which significantly improves over time (p = 0.02), results are consistent with the t-test.
Change for the full sample from T1 to T2 assessed with t-test p-value and Cohen’s d.
Table 4 shows means and regression output comparing IYPB mothers and UC mothers. The baseline scores for intervention and control mothers do not differ for any of the outcomes apart from one item (child health), for which intervention mothers report significantly better child health than control mothers. However, this difference is not clinically significant, as both groups report a mean above nine on a scale 0–10. A comparison of IYPB and UC mothers at program completion shows no significant difference in any of the outcomes except one: “Mother report of a network that can help with practical issues” is significantly lower for mothers in the IYPB group than in the UC group at T2 (β = -1.07 [-1.85,-0.28], d = -0.18).
Differential Effects
We divided the sample into quarters and halves to look for moderating effects. We only performed the moderator analyses on the rating scales (KPCS, PSS, MDI, WHO5, RSS, MABISC, and ASQ:SE-2e), as the single items had very low variation and could not meaningfully be divided. Table 5 shows regression outputs for post-intervention outcomes for the following groups: mothers scoring within the lowest 25% and 50% at baseline, and mothers scoring within the highest 25% and 50% at baseline. For the lowest-performing 25%, parent confidence (KPCS) and mental health (WHO5) are significantly worse for the intervention group immediately following the intervention. For the lowest-performing 50%, parent stress (PSS) is significantly worse for the intervention group post-intervention. For both the lowest 50% and 25%, all other outcomes also trend in this direction, except for self-esteem (RSS) for the lowest 25% and depression (MDI) for the lowest 50%. In contrast, for mothers in the highest performing groups, all outcomes favor the IYPB group, except for parent confidence (KPCS) for the 25% highest, and depression MDI for the 25% and 50% highest; these results are not statistically significant apart from parent stress. For those who scored in the top 50% at baseline, parent stress (PSS) scores are significantly better for IYPB mothers than mothers who received UC
Results compare mothers immediately following the intervention who received the IYPB program to those who received usual care within the groups scoring in the lowest 25th and 50th percentiles and the highest 50th and 75th percentiles.
Clinical Levels
Table 6 shows the proportion of mothers within IYPB and UC with clinical levels for outcomes with thresholds for clinical levels. There is no difference in the proportions between IYPB and UC for any of the outcomes. For all outcomes, the proportion of mothers with clinical levels falls between T1 and T2.
Sensitivity analyses
Sensitivity analyses included OLS regression without imputation, random effects modeling, and difference-in-differences estimation on the imputed data, as presented in Table 7. The results presented here differ from those in Table 4 only in that, for the OLS regression without imputed data, mothers who participated in the IYPB group reported reading significantly more days per week to their infants than control mothers.
Implementation and Treatment Fidelity
After each session, the group leaders completed the checklists provided. Although most items were applied during the sessions, sometimes elements were postponed due to time constraints. The group leaders usually included two to four vignettes in each session, but one session included four to six. As a result of parent feedback, the group leaders changed some elements of the program to fit the Danish context (i.e., some vignettes and exercises were dropped because parents felt that they were not relevant to their situations). The group leaders also spaced the sessions so that the infants were old enough for the last sessions to be age-relevant.
Parent and Group Leader Satisfaction
An evaluation of user satisfaction with the program was completed for families who took part in the IYPB intervention in Ikast-Brande; not all families who were evaluated in the satisfaction study participated in the trial Almost half of the parents stated that attachment to their child was not influenced by participating in IYPB, while 36% stated that it was improved [69]. Only 1% reported low parenting confidence at the end of the intervention. The elements of the program that parents liked the most were group discussions, talking with others, and sharing experiences. The video vignettes were the least favoured element (25% felt that they were appropriate or very appropriate). Less than half of the parents (42%) felt that issues around babyproofing the home were appropriate or very appropriate. Overall, parents preferred the first sessions. Parents also reported that because their name appeared on the evaluation form, they were not always as critical as they would have been on an anonymous evaluation.
Common reasons for participating in the IYPB program were that parents had a general wish to be a good parent and that they were afraid that they might miss out on important information if they did not participate [70]. Parents’ expectations were not always fulfilled, as some had expected the sessions to involve more direct teaching and advice from the health visitors. Some parents statedthat the program structure was a bit too rigid, and they would have sometimes preferred to discuss other issues that they felt were important. Some parents stated that the information given was too basic for their level of knowledge, these reports may have come from mothers who had higher levels of functioning as the program was developed for lower functioning families and families of all levels participated in the groups. Some suggested that the sessions would have been satisfactory had they only involved the mothers, without group leaders. About half of the parents said they would recommend or strongly recommend the program to other parents.
The group leaders generally felt satisfied with the running of the IYPB groups, although some mentioned that they would have appreciated more acknowledgment from their leaders. Group leaders reported that group dynamics differed according to the specific participants. If parents shared insecurities or worries during the first few sessions, group leaders reported that their doing so opened up the dialogue for the remaining sessions. In groups where parents did not share such feelings, the group leaders reported that the more insecure parents stopped coming. Group leaders also reported that the group dynamics were also affected by the participation of fathers and that it was generally difficult to getfathers to participate.
Cost
We were able to obtain cost data for only one site. Ikast Brande reports that the cost of IYPB per family is approximately DKK 7000 (≈$1045) when groups consist of eight families. The price includes training for group leaders, preparation time, housing, food and drink, supervision, and time for group sessions. No information was available on the cost of UC. As IYPB was offered in addition to UC, all mothers received UC. Compared to the cost analyses reported in the Wales evaluation [75], the price of running a group in Ikast-Brande is lower than the price for the initial Welsh group, but higher than that for subsequent Welsh groups.
Discussion
In this pilot trial of the IYPB program as a universal intervention for parents with infants, there where no statistically significant differences on the measured outcomes between the IYPB and UC groups immediately following the intervention, apart from mothers’ social netvork. Mothers in the intervention group reported a smaller network to help them with practical issues than mothers in the control group. It seems unlikely that during the 10–20 week participation period in the IYPB program from pre to post intervention mothers’ available network would be diminished. An alternative explanation is that working to establish a support network for parents is a central focus of the IYPB program; as a result, intervention mothers may become more aware of the network that they currently have available to them and feel that they need to establish more support while control mothers may not have considered the importance of developing their network to the same degree. In addition, the mothers’ support network was a secondary outcome measured by a single item, and as all 20 outcomes at post-intervention were tested using a 5% significance level, the finding may be a spurious effect. This result should be re-examined at follow-up to determine if it is sustained.
Our findings are consistent with the results of the pre-post evaluation of IYPB in Wales, where Jones et al. did not find effects on parent report measures, but did find improved parent sensitivity in the IYPB group on parent-child observations. The baseline means of our primary outcome, parenting confidence (KPCS), and the change from T1 to T2 are nearly identical to those in the Jones et al. study [39]. Therefore, there is reason to believe that observed parent sensitivity may be improved in the present sample and those findings should be evaluated. The results are also similar to those of Evans et al. IYPB evaluation, which found statistically significant improvement from pre to post for parenting confidence (KPCS) and mental health. The KPCS baseline means in the Evans study are lower than in ours (38.66 compared to 41.09), and the change from T1 to T2 is larger (2.26-point improvement compared to 1.76), this is likely because the sample in the Evans study consisted of a disadvantaged group of mothers.
The lack of effect of IYPB as a universal intervention in the present pilot trial is consistent with some of the previously discussed disadvantages of universal interventions. For instance, it is more likely that the families with the greatest needs will decline to participate [76] and it is difficult to detect overall effects of universal interventions without very large populations due to the heterogeneity of the sample [77]. A recent systematic review of the effects of universal interventions for parents with infants on child development and the parent-child relationship found mixed results for the effect of universal interventions [78]. The systematic review includes three universal interventions discussed in the introduction: the Australian Toddlers Without Tears program [45,48], the Australian Face to Face program [49], and the US trial of parent training [50]. None of the three RCTs found intervention effects. For example, the authors of the Toddlers Without Tears program conclude, “A brief universal parenting programme in primary care is insufficient to prevent development of preschool externalising problems” [45]. Our results indicate that even the more intensive IYPB intervention, which starts when the infant is younger, has no effects immediately following the intervention for a universal population where the quality of usual care is high. It is important to note that the IYPB program was developed for disadvantaged families who showed observable difficulties in caring for their infants, not for a universal population.
There are several possibilities for the results of our findings. The sample is a universal group of mothers with a relatively high education level. These mothers would be expected to do well caring for their infants without the intervention, and their baseline scores were higher than those of disadvantaged groups.
A common issue for evaluations conducted with universal samples is the ceiling effect [79]. This effect applies to some of the outcomes in this trial, including one of the primary outcomes (KPCS). A marked ceiling effect means that measuring improvement over time is difficult. However, when looking at all mothers in the trial as a whole, we find that almost all outcomes (including the KPCS) significantly improve from T1 to T2. The significant negative effects that we find are related to measures that do not have marked ceiling or floor effects (PSS and WHO-5).
It is important to consider when interpreting the results that the control group received usual care in Danish health centers, which is a relatively extensive intervention. Almost all Danish families take advantage of five or six free home visits from a health visitor in addition to three free scheduled child-health visits to their general practitioner within the child’s first year of life [53,54]. For most outcomes, the sample as a whole significantly improved over time, suggesting that the universal intervention already offered to families in Denmark is beneficial or that the parents in this population adjust well to their new roles over time. From the results of this trial, we cannot conclude whether or not IYPB would be superior to no intervention and we do not know what the effects of the intervention will be for the infants or the parents longer term.
The results indicate that there may be differential effects of the IYPB intervention. Contrary to our hypothesis, we find that mothers with the lowest scores at baseline may experience negative effects from IYPB. When the sample is divided into two groups using pre-intervention scores (lowest 50%- and highest 50%)- most outcomes favor the IYPB group for the highest functioning halfof the sample, whereas the opposite applies to all of the outcomes from the lowest functioning half. In the moderator analyses we find interaction effects for the parent stress (PSS) measure; for the highest-functioning half of the mothers, we find that the IYPB group show significantly lower parent stress (PSS). However, for those in the lowest functioning half of the sample at baseline, the mothers who received IYPB scores significantly worse on parent stress (PSS)immediately following the intervention. In addition, for those in the lowest-scoring quarter, parent confidence (KPCS) and general well-being (WHO5) are significantly worse for IYPB mothers than for UC mothers at post treatment. These results are consistent whether or not we conduct the analyses using the imputed data.
Group interventions are based on the assumption that participants benefit from being in a safe environment where they feel supported by others who share their concerns and difficulties [80,81]. If a group is comprised of parents with widely varying experiences of becoming a parent, these differences may reduce the level of cohesion within the group.Previous research has shown that the lowest-functioning members of a group can experience negative effects of group participation if the other members of the group function at a much higher level [82]. In this case, the mothers who reported finding parenting more challenging before the IYPB program began, may have felt even more insecure and less competent in their abilities after meeting with parents who felt confident, self-efficacious, and less stressed in their parenting roles. If the differences in parents functioning reduced group cohesion and participants feelings of acceptance, the intervention may actually contribute to increasing inequality, a point highlighted in discussions of the disadvantages of implementing universal rather than targeted interventions [77].
Two separate studies indicate that the parent and group leader satisfaction with the program could have also influenced our results[69,70]. Only about half the parents reported at the end of the intervention that they would recommend it to other families Both the parents and group leaders indicated in an evaluation of the implementation that the program may need to be modified somewhat to meet the needs of a universal group of parents.
This is a pilot trial and the results should therefore be interpreted with prudence. In addition, it will be important to consider the results of the parent-child interactions (from observational data) and 18-month follow-up data when these analyses become available. The outcomes could differ from the present study for the parent-child relationship quality when rated by objective observers and current outcomes may change over time.
Conclusion
This study is the first RCT of the IYPB program as a universal intervention for parents with infants. There was relatively little difficulty rrecruiting families to the effectiveness l evaluation, and the majority of mothers randomized to the IYPB group participated in the intervention, even though not all reportedfeeling satisfied with the program. Apart from one item, which may be a spurious effect (IYPB mothers reported having a smaller network than UC mothers post-intervention), we found no difference between the groups on any outcomes immediately following the intervention. There were differential effects hen the sample was divided into the lowest- and highest-scoring halves on baseline scores; we found significantly improved parental stress scores for the highest functioning mothers and significant negative effects on parental stress and mental health at post-intervention for the IYPB mothers who were in the lowest-functioning groups at baseline. All other outcomes appear to be trending in this direction. Given these results and the feedback from parents and group leaders, it is possible the IYPB program may be more effective when compared with high quality usual care if some cultural adaptations are made for specific populations and if it is used with more homogenous parent groups (i.e., level of parental function). The IYPB intervention was developed for use with groups of lower functioning families rather than a universal population; future research should investigate the effects of the program with groups of parents (both high and low functioning) who have similar mental health and parenting experiences.
Acknowledgments
The first author would like to thank all the families that participated in the trial, and the Ikast-Brande and Herning local authorities for working together with her in designing the study and recruiting the families. The trial is supported by a grant from the Danish foundation TrygFonden.
Author Contributions
- Conceptualization: MP SKK.
- Data curation: MP TMS.
- Formal analysis: MP TMS.
- Funding acquisition: MP.
- Investigation: MP.
- Methodology: MP SKK TMS.
- Project administration: MP.
- Software: MP TMS.
- Supervision: MP.
- Validation: MP SKK TMS.
- Visualization: MP TMS.
- Writing – original draft: MP SKK TMS.
- Writing – review & editing: MP SKK TMS.
References
- 1.
Corby B. Child abuse: towards a knowledge base (3rd ed). 3rd ed. Berkshire: Open University Press; 2006.
- 2.
Geffner R, Igelman RS, Zellner J. The effects of intimate partner violence on children (Vol. 2). New York: Haworth Maltreatment & Trauma Press; 2003.
- 3.
Grøgaard J. Dype spor. Dvd. Oslo: Barne og familiedepartementet; 2007.
- 4. Van Doesum KTM, Riksen-Walraven JM, Hosman CMH, Hoefnagels C. A randomized controlled trial of a home-visiting intervention aimed at preventing relationship problems in depressed mothers and their infants. Child Dev. 2008;79: 547–561. pmid:18489412
- 5. Cicchetti D, Rogosch FA, Toth SL. Maternal depressive disorder and contextual risk: Contributions to the development of attachment insecurity and behavior problems in toddlerhood. Dev Psychopathol. Cambridge Univ Press; 1998;10: 283–300. pmid:9635225
- 6. Weissman MM, Warner V, Wickramaratne P, Moreau D, Olfson M. Offspring of depressed parents. Arch Gen Psychiatry. 1997;54: 932–940. pmid:9337774
- 7. Field T. Maternal Depression Effects on Infants and Early Interventions. Prev Med (Baltim). 1998;27: 200–203.
- 8. Olds DL, Sadler L, Kitzman H. Programs for parents of infants and toddlers: recent evidence from randomized trials. J Child Psychol Psychiatry. 2007;48: 355–91. pmid:17355402
- 9.
Furlong M, McGilloway S, Bywater T, Hutchings J, Smith SM, Donnelly M. Behavioural and cognitive-behavioural group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years’. Evidence-Based Child Health. Wiley Online Library; 2013. pp. 693–694.
- 10. Piquero AR, Farrington DP, Welsh BC, Tremblay R, Jennings WG. Effects of early family/parent training programs on antisocial behavior and delinquency. Campbell Syst Rev. 2008;11.
- 11. Welsh BC, Farrington DP. Scientific Support for Early Prevention of Delinquency and Later Offending. Vict Offender. 2007;2: 125–140.
- 12. Barlow J, Bennett C, Midgley N. Parent-infant psychotherapy for improving parental and infant mental health. Cochrane Database of Systematic Reviews. 2013.
- 13. Barlow J, Smailagic N, Bennett C, Huband N, Jones H, Coren E. Individual and group based parenting programmes for improving psychosocial outcomes for teenage parents and their children. Cochrane Database Syst Rev. 2011; CD002964. pmid:21412881
- 14. Heckman JJ, Masterov D V. The Productivity Argument for Investing in Young Children. Appl Econ Perspect Policy. 2007;29: 446–493.
- 15.
Heckman JJ. The Case for Investing in Disadvantaged Young Children. In: Darling-Hammond L, Grunewald R, Heckman JJ, Isaacs JB, Kirp DL, Rolnick AJ, et al., editors. Big Ideas for Children: Investing in Our Nation’s Future. Washington, DC: First Focus; 2008. pp. 49–58.
- 16. Reedtz C, Handegård BH, Mørch W-T. Promoting positive parenting practices in primary pare: Outcomes and mechanisms of change in a randomized controlled risk reduction trial. Scand J Psychol. 2011;52: 131–137. pmid:21121925
- 17. Axberg U, Hansson K, Broberg AG. Evaluation of the Incredible Years Series—an open study of its effects when first introduced in Sweden. Nord J Psychiatry. 2007;61: 143–51. pmid:17454729
- 18. Baker-Henningham H, Scott S, Jones K, Walker S. Reducing child conduct problems and promoting social skills in a middle-income country: cluster randomised controlled trial. Br J Psychiatry. 2012;201: 101–8. pmid:22500015
- 19. Bywater T, Hutchings J, Daley D, Whitaker C, Yeo ST, Jones K, et al. Long-term effectiveness of a parenting intervention for children at risk of developing conduct disorder. Br J Psychiatry. 2009;195: 318–24. pmid:19794200
- 20. Drugli MB, Larsson B, Fossum S, Mørch W-T. Five- to six-year outcome and its prediction for children with ODD/CD treated with parent training. J Child Psychol Psychiatry Allied Discip. 2010;51: 559–566.
- 21. Gardner F, Burton J, Klimes I. Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: Outcomes and mechanisms of change. J Child Psychol Psychiatry Allied Discip. 2006;47: 1123–1132.
- 22. Gridley N, Hutchings J, Baker-Henningham H. The Incredible Years Parent-Toddler Programme and parental language: a randomised controlled trial. Child Care Health Dev. 2015;41: 103–111. pmid:24841947
- 23.
Griffith N. Evaluating the Incredible Years Toddler programme in disadvantaged areas of Wales. Bangor University. 2011.
- 24. Hurlburt MS, Nguyen K, Reid J, Webster-Stratton CH, Zhang J. Efficacy of the Incredible Years group parent program with families in Head Start who self-reported a history of child maltreatment. Child Abuse Negl. Elsevier Ltd; 2013;37: 531–43. pmid:23306144
- 25. Hutchings J, Bywater T, Daley D, Gardner F, Whitaker C, Jones K, et al. Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial. BMJ. 2007;334: 678–678. pmid:17350966
- 26. Little M, Social T, Kingdom U. The Impact of Three Evidence-Based Programmes Delivered in Public Systems in Birmingham, UK. Int J Conf Violence. 2012;6: 260–272.
- 27. Marcynyszyn L a., Maher EJ, Corwin TW. Getting with the (evidence-based) program: An evaluation of the Incredible Years Parenting Training Program in child welfare. Child Youth Serv Rev. Elsevier Ltd; 2011;33: 747–757.
- 28. Menting ATA, Orobio de Castro B, Matthys W. Effectiveness of the Incredible Years parent training to modify disruptive and prosocial child behavior: a meta-analytic review. Clin Psychol Rev. Elsevier Ltd; 2013;33: 901–13. pmid:23994367
- 29. Pidano AE, Allen AR. The Incredible Years Series: A Review of the Independent Research Base. J Child Fam Stud. Springer US; 2014; 1898–1916.
- 30. Presnall N, Webster-Stratton CH, Constantino JN. Parent training: equivalent improvement in externalizing behavior for children with and without familial risk. J Am Acad Child Adolesc Psychiatry. Elsevier Inc; 2014;53: 879–87, 887–2. pmid:25062595
- 31. Scott S, Briskman J, O’Connor TG. Early prevention of antisocial personality: long-term follow-up of two randomized controlled trials comparing indicated and selective approaches. Am J Psychiatry. 2014; 649–657. pmid:24626738
- 32. Trillingsgaard T, Trillingsgaard A, Webster-Stratton CH. Assessing the effectiveness of the “Incredible Years ® parent training” to parents of young children with ADHD symptoms—a preliminary report. Scand J Psychol. 2014;55: 538–545. pmid:25130208
- 33. Hutchings J, Bywater T, Gridley N, Whitaker CJ, Martin-Forbes P, Gruffydd S. The incredible years therapeutic social and emotional skills programme: A pilot study. Sch Psychol Int. 2012;33: 285–293.
- 34. Dishion TJ, Shaw D, Connell A, Gardner F, Weaver C, Wilson M. The family check-up with high-risk indigent families: preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child Dev. 2008;79: 1395–414. pmid:18826532
- 35.
Reedtz C, Klest S. Increased Positive Parenting and Decreased Harsh Discipline Maintained Four Years after a Shortened Parent Training Intervention in a Universal Sample. Rev.
- 36. Perrin EC, Sheldrick R, McMenamy JM, Henson BS, Carter AS. Improving parenting skills for families of young children in pediatric settings: A randomized clinical trial. JAMA Pediatr. 2014;168: 16–24. pmid:24190691
- 37. Gross D, Fogg L, Webster-Stratton CH, Garvey C, Julion W, Grady J. Parent training of toddlers in day care in low-income urban communities. J Consult Clin Psychol. 2003;71: 261–278. pmid:12699021
- 38. Evans S, Davies S, Williams M, Hutchings J. Short-term benefits from the Incredible Years Parents and Babies Programme in Powys. Community Pract. 2015; 46–48.
- 39.
Jones CH, Erjavec M, Viktor S, Hutchings J. Outcomes of a comparison study into a group-based infant parenting programme.
- 40. Kan ML, Feinberg ME. Can a Family-Focused, Transition-to-Parenthood Program Prevent Parent and Partner Aggression Among Couples With Young Children? Violence Vict. 2014;29: 967–980. pmid:25905139
- 41. Aronen ET, Kurkela SA. Long-term effects of an early home-based intervention. J Am Acad Child Adolesc Psychiatry. 1996;35: 1665–72. pmid:8973074
- 42. Dodge KA, Goodman WB, Murphy RA, O’Donnell K, Sato J, Guptill S. Implementation and randomized controlled trial evaluation of universal postnatal nurse home visiting. Am J Public Health. 2014;104: 136–144.
- 43. Doherty WJ, Erickson MF, LaRossa R. An intervention to increase father involvement and skills with infants during the transition to parenthood. J Fam Psychol. 2006;20: 438–47. pmid:16938002
- 44. Shapiro AF, Nahm EY, Gottman JM, Content K. Bringing baby home together: examining the impact of a couple-focused intervention on the dynamics within family play. Am J Orthopsychiatry. 2011;81: 337–50. pmid:21729014
- 45. Bayer JK, Hiscock H, Ukoumunne OC, Scalzo K, Wake M. Three-year-old outcomes of a brief universal parenting intervention to prevent behaviour problems: randomised controlled trial. Archives of disease in childhood. 2010. pp. 187–92. pmid:19847011
- 46. Roia A, Paviotti E, Ferluga V, Montico M, Monasta L, Ronfani L, et al. Promoting effective child development practices in the first year of life: does timing make a difference? BMC Pediatr. 2014;14: 222. pmid:25193490
- 47. Santelices MP, Guzmán G M, Aracena M, Farkas C, Armijo I, Pérez-Salas CP, et al. Promoting secure attachment: evaluation of the effectiveness of an early intervention pilot programme with mother-infant dyads in Santiago, Chile. Child Care Health Dev. 2011;37: 203–10. pmid:21083685
- 48. Hiscock H, Bayer JK, Price A, Ukoumunne OC, Rogers S, Wake M. Universal parenting programme to prevent early childhood behavioural problems: cluster randomised trial. BMJ. 2008;336: 318–21. pmid:18244958
- 49. Vlismas W, Malloch S, Burnham D. The effects of music and movement on mother–infant interactions. Early Child Dev Care. 2013;183: 1669–1688.
- 50. Dickie JR, Gerber SC. Training in Social Competence: The Effect on Mothers, Fathers, and Infants. Child Dev. 1980;51: 1248–1251. pmid:7471925
- 51. Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials. Ann Intern Med. 2010;152: 726. pmid:20335313
- 52. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337: a2390. pmid:19001484
- 53.
Sundhedsstyrelsen. Vejledning om forebyggende sundhedsydelser til børn og unge. Copenhagen: Sundhedsstyrelsen; 2011.
- 54.
Foreningen for ledere af sundhedsordninger for børn og unge i Danmark. Ledelsesforhold og kapacitet–et øjebliksbillede. Sundhedsordningerne for børn og unge 1.1.2013. Copenhagen; 2013.
- 55.
Webster-Stratton CH, Reid MJ. The Incredible Years Program for children from infancy to pre-adolescence: Prevention and treatment of behavior problems. Clinical handbook of assessing and treating conduct problems in youth. Springer; 2010. pp. 117–138.
- 56.
Webster-Stratton CH. Incredible Babies: A Guide and Journal of Your Baby’s First Year. 2011.
- 57. Pontoppidan M. The effectiveness of the Incredible YearsTM Parents and Babies Program as a universal prevention intervention for parents of infants in Denmark: study protocol for a pilot randomized controlled trial. Trials. Trials; 2015;16: 386. pmid:26329163
- 58.
Črnčec R, Barnett B, Matthey S. Karitane Parenting Confidence Scale: Manual. Sydney: Australia: Sydney South West Area Health Service; 2008.
- 59. Črnčec R, Barnett B, Matthey S. Development of an instrument to assess perceived self-efficacy in the parents of infants. Res Nurs Heal. 2008;31: 442–453.
- 60. Berry JO, Jones WH. The Parental Stress Scale: Initial Psychometric Evidence. J Soc Pers Relat. 1995;12: 463–472.
- 61. Olsen LR, Jensen D V., Noerholm V, Martiny K, Bech P. The internal and external validity of the Major Depression Inventory in measuring severity of depressive states. Psychol Med. 2003;33: 351–356. pmid:12622314
- 62. Bech P. Measuring the Dimension of Psychological General Well-Being by the WHO-5. Qual Life Newsl. 2004; 16–16.
- 63.
Bech P. Klinisk psykometri. Copenhagen: Munksgaard Danmark; 2011.
- 64.
Rosenberg M. Society and the adolescent self-image. New York; 1965.
- 65.
Squires JK, Bricker D, Twombly E. The ASQ-SE user’s guide: for the ages & stages questionnaires, social-emotional. A Parent-Completed, Child-Monitoring System for Social-Emotional Behaviors. Baltimore, MD: Paul H. Brookes Pub. Co.; 2002.
- 66.
Squires JK, Bricker DD, Twombly E. ASQ:SE-2 user’s guide: Ages & Stages Questionnaires social-emotional. Brookes; 2015.
- 67. Høivik MS, Burkeland N a, Linaker OM, Berg-Nielsen TS. The Mother and Baby Interaction Scale: a valid broadband instrument for efficient screening of postpartum interaction? A preliminary validation in a Norwegian community sample. Scand J Caring Sci. 2013;27: 733–9. pmid:22892011
- 68. Hackney M, Braithwaite S, Radcliff G. Postnatal depression: the development of a self-report scale. Health Visit. 1996;69: 103–104.
- 69.
Rasmussen A, Laursen KK, Nørgaard U, Stenbæksgaard E, Cederstrøm L, Kanschat C, et al. Evaluering af Projekt: “De Utrolige År Baby til alle førstegangsfødende.” 2015.
- 70.
Hjælmhof-Larsen SF, Nielsen M. Fra politik til praksis—en kommunal implementeringsproces. Roskilde Universitet. 2015.
- 71. Schafer JL, Olsen MK. Multiple imputation for multivariate missing-data problems: A data analyst’s perspective. Multivariate Behav Res. Taylor & Francis; 1998;33: 545–571. pmid:26753828
- 72.
Rubin DB. Multiple Imputation for Nonresponse in Surveys. Rubin DB, editor. Hoboken, NJ, USA: John Wiley & Sons, Inc.; 1987.
- 73. Bech P, Olsen LR, Kjoller M, Rasmussen NK. Measuring well-being rather than the absence of distress symptoms: a comparison of the SF-36 Mental Health subscale and the WHO-Five Well-Being Scale. Int J Methods Psychiatr Res. 2003;12: 85–91. pmid:12830302
- 74. Maimburg RD, Væth M. Postpartum depression among first-time mothers—results from a parallel randomised trial. Sex Reprod Healthc. Elsevier B.V.; 2015;6: 95–100. pmid:25998877
- 75. Jones CH, Hutchings J, Erjavec M, Hughes JC. Parent and group leader reflections on a group-based programme for parents and babies. Community Pract. Community Practitioner; 2012;85: 26–29.
- 76. Fonagy P. Prevention, the appropriate target of infant psychotherapy. Infant Ment Health J. 1998;19: 124–150.
- 77. Offord DR, Kraemer HC, Kazdin AE, Jensen PS, Harrington R. Lowering the Burden of Suffering From Child Psychiatric Disorder: Trade-Offs Among Clinical, Targeted, and Universal Interventions. J Am Acad Child Adolesc Psychiatry. 1998;37: 686–694. pmid:9666623
- 78. Pontoppidan M, Klest SK, Patras J, Rayce SB. Effects of universally offered parenting interventions for parents with infants: a systematic review. BMJ Open. 2016;6: e011706. pmid:27683513
- 79. Črnčec R, Barnett B, Matthey S. Review of Scales of Parenting Confidence. J Nurs Meas. Springer Publishing; 2010;18: 210–240. pmid:21290926
- 80.
Yalom ID, Leszcz M. The theory and practice of group psychotherapy. New York: BasicBooks; 2005.
- 81. Joyce AS, Piper WE, Ogrodniczuk JS. Therapeutic Alliance and Cohesion Variables as Predictors of Outcome in Short–Term Group Psychotherapy. Int J. 2007;57: 269–296.
- 82. Carrell SE, Sacerdote B, West J. From Natural Variation to Optimal Policy? The Importance of Endogeneous Peer Group Formation. Econometrica. 2013;81: 855–882.