Control condition.

In comparison, control arm schools were not given any specific guidance or advice and were assumed to carry on with their usual practice of physical activity, PE and sport provision.

Micro-costing methods.

Micro-costing methodology [10,12] was applied to calculate the costs of delivering the programme over a whole school year for the intervention schools taking part in the trial, to provide a mean cost per year, per school and per pupil. Per pupil costs were based on 467 pupils per school, in accordance with the average number of female pupils per school, taken from census data from all schools taking part in the trial (n = 20). In this micro-costing, we fully costed the delivery of the ‘Girls Active’ programme and its associated costs such as teacher time, travel and materials used. Costs were collected from a Local Education Authority perspective, accounting for on-costs (e.g. national insurance and pension costs), and using the cost year 2015–16.

A bespoke cost diary was designed and administered by one member of the research team to lead teachers responsible for delivering ‘Girls Active’. The diary asked lead teachers to complete a record of the additional time, or displaced time taken to offer the ‘Girls Active’ programme, and described activities undertaken and items purchased (e.g. sports equipment or clothing such as hoodies). The diary was developed with input from the wider ‘Girls Active’ research team and the YST who were responsible for training and supporting teachers during programme delivery.

Three versions of the diary were used throughout the course of the trial. The diary was originally sent to teachers via e-mail as a Microsoft Excel file, with sections for categories of programme activity (e.g. training, peer review day and programme delivery). Each section had a pre-written description of programme activity per row (e.g. time spent reviewing current PE and sport culture and practice (hours)), based on key programme components and delivery. The diary also had an ‘other’ section, which provided space for teachers to record programme activities and costs not covered by the sections/rows above. The diary was sent with written guidance for completion and presented to the lead teachers at the initial training event with a rationale and guidance for completion. An example of a completed diary sheet to provide the respondents with the information and level of detail required when completing the diary was also provided. We asked the teachers to complete the diary weekly. A member of the research team requested the Excel diary at two-month intervals whilst ‘Girls Active’ was being implemented (April 2015 until June 2016). Any queries regarding information provided were e-mailed to staff, and researcher contact details were provided in e-mails. After receiving the final Excel diary, any final queries were sent from the health economics researcher (JC) to the lead teacher who sent the diary via e-mail.

Over the first three months of the trial, the research team received feedback that around half of the teachers were not comfortable using the computer-based (Excel) format of recording costs. Teachers reported not being familiar or able to use Excel, and difficulties finding time to log into their e-mail or files to retrieve and add to the Excel. In response, a paper-based survey-style questionnaire was produced in Microsoft Word by two of the research team members (JC and DH). The survey used fields from the Excel diary, and the research team followed up surveys with a telephone call, from the researcher (JC) responsible for the micro-costing, to provide context to the responses or clarify. The survey was presented to the lead teachers who attended the peer leader event (January 2016). The event was organised by the hub school and the YST as an opportunity for the peer leaders to meet and share ideas. As well as commenting on the survey, the teachers stated that a simple logbook would be their preference as a simpler, easier-to-access format for recording time and costs. At the event, the teachers and the research team member (DH) in attendance co-produced the logbook with the lead teachers stating what headings they wanted in the logbook and how they would like it presented. The logbooks contained headings and row for activities and their associated time or cost, so teachers could record the information as it happened. To ensure we had cost information from as many schools as possible, the logbooks were printed and distributed to teachers by a member of the YST. It was requested that teachers keep these logbooks with them in their own diaries, and noted activity and costs related to ‘Girls Active’ as and when it happened. Each of the three methods requested demographic information about the teacher completing the diary including the school name, job title and salary band.

The salary band information was used to calculate teacher costs in the micro-costing, using national sources of salary costs. Teacher costs were sourced from the National Union of Teachers (NUT) pay structure for qualified classroom teachers in England and Wales (from 1 September 2015, for cost year 2015–2016) [20]. A school year consisting of 39 weeks was used to calculate the cost per hour for teacher costs, taking sickness, continuing professional development (CPD) and annual leave into account. Salary calculations were inclusive of employers’ on-costs. On-costs include National Insurance (NI) and Pension charges, as well as costs for annual increments and allowances. This mean hourly rate was applied to information supplied by teachers completing cost diaries. A school year of 39 weeks was chosen despite the variable time-periods of programme delivery in intervention schools in the trial to provide consistency in estimates of costs, and provide those interested in delivering the programme in the future a clear time horizon of costing. ‘Girls Active’ aimed to encourage a culture shift in schools, thus the principles of the programme would be expected to be undertaken annually and for school leads to be constantly ensuring their PE, physical activity and sport provision was appealing and of interest to pupils. Costs relating to research (e.g. time to complete diary, measures or undertake interviews) were not included in the final micro-costing calculations. This decision was made in order to provide local authorities with information pertinent for future rollout (training and delivery costs), rather than costs specific to conducting a research trial. Given the range of methods used to gather cost information, the following assumption was made during micro-costing in order to standardise the costing: Programme activities reported such as curriculum sessions, lunchtime sessions, and after school sessions were assumed to take place every week, throughout the academic year consisting of 39 weeks. These assumptions were made to provide consistency in estimates of costs and standardise a delivery approach in order to calculate costs. We note that intervention schools did not necessarily implement activities for this length of time.

Bottom-up micro-costing was chosen instead of top-down micro-costing, as the intervention was not prescriptive; it did not dictate the amount of time or money schools should spend to adapt their PE curriculum, it only stated core activities that should be conducted such as training, reviewing current activity and setting up a peer leader group. Given this intervention format, discrete information about resources and time taken to deliver ‘Girls Active’ was required from each of the intervention schools.

Cost-consequence methods.

The cost-consequence analysis was conducted from a public sector, multi-agency perspective (community care, GP and local authority, school). Minutes of moderate-to-vigorous (MVPA) and health-related quality of life (CHU-9D) [21] were used as the measures of effect, and primary care (GP) and school-based services (school nurse and school counsellor) was the measure of costs. These health professionals were chosen in line with the perspective of the analysis and to reduce the burden on participants by enquiring about key health professionals pupils would be likely to come into contact within school and in the community.

Data was collected at baseline, approximately 7 months post-baseline and approximately 14 months post-baseline. The outcomes of interest for the cost-consequence analysis were mean minutes of MVPA per day, health-related quality of life measured by the CHU-9D [21] and use of primary care and school-based services. Mean minutes of MVPA per day were taken from data collected by the GENEActiv Original accelerometer (Activinsights Ltd, Kimbolton, UK) for worn 24 hours a day for 7 days by all pupils in the trial on their non-dominant wrist. In order to test uncertainty and as the follow-up period was more than 1 year, costs at 14-month post-baseline were discounted at the base rate of 3.5% [22] as part of sensitivity analyses. Outcomes were not discounted as part of the sensitivity analysis, given only an additional two months fell outside of the 1-year time horizon.

Pupils also completed two further questionnaires for the purposes of the economic evaluation. The CHU-9D [21] is a paediatric generic preference-based measure of health-related quality of life and consists of nine dimensions (worried, sad, pain, tired, annoyed, school work/homework, sleep, daily routine and ability to join in activities). The scores from each domain have a weighting applied and all domain weightings are summed together to produce a utility index. The CHU-9D has been validated with children aged 11 to 17 years as a self-report measure [21]. Participants were given specific guidance on how to complete this questionnaire. A list of common questions were collated and given to team members administering the questionnaires to participants to ensure a standardised response to any participant query. Documentation related to this study can be found at www.leicesterdiabetescentre.org.uk/girls-active-evaluation including a standardised operating procedure for administering the ‘Girls Active’ questionnaires.

The Client Service Receipt Inventory (CSRI) [23] is a questionnaire for collecting retrospective information about study participants’ use of health and social care services. The CSRI was administered at all time points, each time asking the participant to recall service use over the previous 7 months. This information was combined with national sources of reference unit costs [24] in order to calculate a mean cost of service costs per participant per arm for the cost consequences analysis. A supporting information file [S1 Table] shows a unit cost table outlining the published unit costs used in this cost-consequence analysis and their sources. The cost year of 2015–2016 was applied for all costs in UK Pounds Sterling.

Analysis.

In order to analyse clustered data, we aligned the health economics analysis to the statistical analysis plan of the main trial, and used Generalised Estimating Equations (GEE), xtgee command in Stata. The GEE model was used to determine the difference between groups in change in mean MVPA minutes/day, CHU-9D [21] utility index scores and service use frequencies and costs between pupils from schools allocated to the programme and those allocated to control whilst taking account of clustering amongst pupils from the same school. The analysis included: two levels of clustering (pupil level and school level); a binary indicator for randomisation group as the explanatory variable; stratification categories based on school size of small vs. large (pupil median value: <850, ≥850), and percentage BME pupils low vs. high (pupil median percentage: non-white <20%, ≥20%) as potential confounders, adjustments for wear time (14-month measurement subtracted by the baseline) and final adjustments for the baseline measure of the outcome (MVPA).

The GEE model specification also included: an identity link function relating the mean response to the regression equation; Gaussian family distribution assumed for the response; an exchangeable correlation structure, which specifies the within-group correlation structure, and robust standard errors (SE) to provide consistent (i.e. asymptotically unbiased) parameter and SE estimates.

For service use frequency and cost models, GLM diagnostic tests were conducted as the models failed to converge using the above structure. From the results of the diagnostic tests, the family was amended from Gaussian to gamma and the link amended from identity to power -1. After changing these specific parameters, the models achieved convergence and these models were used for the variables of service use frequencies and costs in the xtgee models. For all three outcomes of interest the differences in marginal means the groups was calculated, and 95% Confidence Intervals (CIs) around these differences were produced with 1,000 bootstrapped replications.

Micro-costing results.

We received nine diaries/adequate information to generate costs for the schools in the intervention group. Out of the nine schools, four completed the Excel diary weekly. The remaining five schools provided their information through the survey. Three of the five teachers who completed a survey provided data in logbooks when returning the survey.

From the descriptions of the activity in the diaries and when contacting teachers, it was clear that the extent and methods of implementation varied across the nine schools as well as the length of time activities were delivered as part of the programme. Due to the nature of the programme, some of which was not prescriptive and gave schools the responsibility to implement the programme as appropriate within their own setting, ‘Girls Active’ was delivered in different ways, different extents and over different time-periods by the intervention schools taking part. In order to reflect the diversity of delivery, the results of the micro-costing are presented as three costings:

1. Within curriculum delivery–which entailed creating new development plans and delivering the programme within the curriculum after engaging with female pupils (base case).
2. Within curriculum and after-school delivery–which entailed creating new development plans and delivering the programme within the curriculum after engaging with female pupils, and included additional activity such as after-school clubs and taster sessions (approximately 3 hours per week).
3. Within curriculum, after-school delivery and day trips and events–which entailed creating new development plans and delivering the programme within the curriculum after engaging with female pupils, and included additional activity such as after-school clubs and day trips and events (approximately 4 hours per week and 4 trips per year, respectively).

The nine sets of diaries received were classified into the categories as thus: three schools were categorised as delivering within curriculum; four as delivering within curriculum and after-school, and finally two as delivering within curriculum, after-school and day trips and events. The mean time based on the schools in each category is presented in Table 1. A mean hourly rate of £26.00 (rounded to nearest pound) was applied in the micro-costing, which is based on the mean hourly rate of the salary information provided by the teachers completing the diaries.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Costs of the programme based on the three delivery models reported by teachers in the trial.

https://doi.org/10.1371/journal.pone.0221276.t001

Table 1 shows a range of costing models that could be employed to deliver the programme. Costs ranged from £1,054 (£2 per pupil, per school year) to £3,498 (£7 per pupil, per school year), with the least costly option being to absorb ‘Girls Active’ strictly in curriculum hours. Staff time for intervention delivery accounted for the largest costs in the micro-costing due to the nature of the intervention, which was not prescriptive. Teachers were not told the amount of time or money schools should spend to adapt their PE curriculum, which led to teachers deciding what resources and effort they should give to the intervention. Schools also received £1,000 of funding from the YST, though the majority of schools did not spend any of these funds during the trial. Those who spent some of the funding generally used the funding to hire external coaches or purchase new equipment in order to deliver activities requested by the pupils, for example, dodgeballs and Zumba kits. Some schools received further funding from their school’s own budget and public sector funding to further support the purchase of equipment and provision of new activities.

Opportunity costs were considered in the micro-costing. Opportunity cost is defined as the value of benefits foregone by not using resources in their next best alternative use. As ‘Girls Active’ was delivered in usual school hours replacing previous PE activities with those specified by the pupils, this resulted in minimal opportunity costs for the schools taking part in the trial. However, it is worth stating that in order to deliver ‘Girls Active’ teachers’ usual activities were sometimes displaced, this included Planning, Preparation and Assessment (PPA) time, overseeing non-‘Girls Active’ clubs and free periods as well as personal time outside of working hours. On average across the nine intervention schools included in the micro-costing analysis, teachers experienced 6 hours of displaced activity over the course of the year consisting of 39 weeks.

The three costings provide local authorities and those interested in funding the delivery of ‘Girls Active’ in the future an estimate of the time and resources used when the delivery models witnessed in this trial are extrapolated out to a full academic year (39 weeks). The costings do not make any claims on the quality of the delivery; a process evaluation of the programme is presented in a separate paper [27].

Health economics sample.

In the trial, 1,752 participants were recruited. For the economics analysis, we excluded participants who had missing data for both costs and outcomes, as economic evaluations required complete cases from both costs and outcomes [14]. Of these 1,752 participants, 1,211 had primary outcome (mean MVPA minutes/day) data at both baseline and 14 months post-baseline. 1,163 had CHU-9D [21] data at both baseline and 14 months post-baseline, and 1,157 participants had service use data at both baseline and 14 months post-baseline. After missing data was removed, and participant IDs were matched across time-points, the final sample for the economic complete case analysis was N = 997 (Intervention n = 570, Control n = 427).

Table 2 summarises the characteristics of the complete case economic sample at baseline (N = 997), demonstrating the two groups were similar in terms of ethnicity, age in years, indices of multiple deprivation and proportion in each year group. Of the total sample, 21.5% were of non-white European background, the mean age was 13 years, with a low proportion of free school meal eligibility and classed as having mid-levels of deprivation as measured by the Department for Communities and Local Government English indices of deprivation deciles [28].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Baseline characteristics of the economic sample (N = 997) and whole sample (N = 1,752), separated by randomisation arm^†.

https://doi.org/10.1371/journal.pone.0221276.t002

Table 2 also shows the baseline characteristics of the whole sample (N = 1,752). It shows the complete case economic sample (N = 997) and the whole sample (N = 1,752) were similar in terms of ethnicity, age in years, indices of multiple deprivation and proportion in each year group.

Marginal means are presented throughout, as these report the mean following the xtgee model and take account of clustering. Table 3 show the results from the xtgee models, demonstrating the programme did not have an effect on the main outcomes of MVPA/day, CHU-9D [21] utility index score or total service use for the economic sample. However, factors such as baseline value, school size and percentage of ethnic minority pupils did have an effect on the results, adding weight to the trial design and therefore their inclusion in the xtgee model. Table 4 shows no significant differences between the means of the intervention and control group for the three outcome measures and costs as indicated by the bootstrapped CIs (bootstrapping was used to create 1,000 valid bootstrap replications). A supporting information file [S2 Table] shows the results of the xtgee model and marginal means of service use when broken down into individual services of GPs, school nurses and school counsellors at each time-point. The analysis found no statistically significant effect for the programme when service use was divided into individual services.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results from the xtgee models for the three economic outcomes^{*, ±}.

https://doi.org/10.1371/journal.pone.0221276.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Marginal mean minutes/day of MVPA, CHU-9D [21] utility index scores and total service use following 14-month post baseline xtgee models, and mean differences between groups (1,000 bootstrapped 95% CI)^±.

https://doi.org/10.1371/journal.pone.0221276.t004

After conducting the xtgee models, inter cluster correlations (ICCs) were calculated for the three outcomes. An ICC of 0.198 was found for MVPA/day; -0.000 for CHU-9D [21] utility index score; -0.008 for frequencies of total service use; and -0.005 for costs of total service use, respectively.

Sensitivity analysis.

As the follow-up period was more than one year, as part of sensitivity analysis, service use costs at 14-month post-baseline were discounted at 3.5%, the base rate recommended by NICE [16], and the model was re-run with these discounted costs. Table 5 demonstrates the sensitivity analyses replicated the main analysis results, showing no effect for the programme, but an effect of baseline costs and constant. Discounted marginal means reported in Table 5 are similar to the undiscounted marginal means reported in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results from the xtgee models using discounted costs as part of sensitivity analyses^{*, ±}.

https://doi.org/10.1371/journal.pone.0221276.t005

A supporting information file [S3 Table] shows the results of the exploratory sub-group analyses, testing the effects of year group and programme delivery model (based upon the levels described in the micro-costing). In this analysis, year group was found to have a statistically significant effect on minutes of MVPA per day, but did not affect any other outcomes. Programme delivery model did not affect the main outcomes of MVPA/per day, CHU9D [21] utility index scores and frequencies and costs of service use. No other statistically significant differences were found.

Strengths and limitations.

Schools are viewed as an important setting for public health programmes [30]. This micro-costing and cost-consequence analysis adds to the limited literature and provides results from a UK context. By gathering individual information on programme delivery, we demonstrated the programme could be delivered using a range of delivery methods with a range of associated costs, and we also increased the likelihood of receiving data from each intervention school.

The study was limited by the amount of missing data, which reduced the sample for the economic evaluation, due to the need for complete cases. The main area of missing data came from service use. We endeavoured to focus on key professionals, to increase relevance and reduce burden. Future trials of school-based interventions including a health economics component could shorten the recall period, ask parents to complete the measure or explore the use of electronic records.

We acknowledge the limitations in collecting costs using three different methods, and note that by assuming a school year of 39 weeks and consistent delivery of activities related to the ‘Girls Active’ programme over this year, our costings might have overestimated the costs of the programme [27]. We tried to reduce over-estimation as much as possible by not costing activities that would have taken place in contractual hours and using the information provided by intervention schools to give a reflection of the length of time teachers spent on activities such as specific lessons, after school clubs and day trips. These assumptions were made to standardise costing across schools and provide clear costing for those interested in delivering the programme in future.

Novel aspects of applying health economics to a school-based physical activity trial.

Conducting economic evaluations alongside public health interventions has been described as challenging, often requiring a pragmatic approach [31, 17]. The difficulties faced by teachers in completing the Excel version of the cost diary, which was essential for the economics analysis to quantify the resources required to the deliver the ‘Girls Active’ programme, raised a particular challenge in this trial early on. A pragmatic and iterative approach to co-designing measures to collect this information was taken, adapting the Excel cost diary to a survey and listening to feedback by creating a logbook to increase response rates. Teachers were followed up with telephone calls to gain all information required, and used assumptions to provide consistency across the three different methods employed. This adaptive approach to the economic data collection yielded further responses from the teachers, and resulted in 9 out of 10 intervention schools providing data for the micro-costing analysis.

The ‘Girls Active’ programme is not prescriptive in nature. The programme consisted of a number of core steps that all lead teachers were asked to attend or do. This included attending training and peer review events, completing a school self-review and submitting two action plans, and setting up a peer leader group. However, it did not dictate how schools should implement activities within their own school, how the student voices should be used or what the capacity building funding should be spent on; although, examples and suggestions were given by the YST at each of the training events and through the development coach. The programme did not state the number of hours teachers should spend recruiting peer leaders, reviewing current activity and developing new lesson plans. This presented the challenge of displaying the costs of the programme in a meaningful way that reflected what happened for the intervention schools taking part in the trial, but could also be used by future local authorities to indicate the cost of ‘Girls Active’ if it is to be rolled out in their locality. Due to the adaptive approach taken to collecting cost information, we increased the amount of data available to draw from. Furthermore, by undertaking follow-up telephone calls with the teachers who completed the cost diaries, we had first-hand accounts of their experiences, which provided context to the information received. This context was instrumental in helping the researchers to make the decision to present the data as three costings, using the different delivery methods, which we believe provides useful information for those interested in delivering ‘Girls Active’ in a real-world setting. However, assumptions had to be made in the micro-costing to standardise delivery across the intervention schools.

Lessons learned—Implications for practice and/or future research.

The cost-consequence analysis quantified the resources used in the trial to undertake differing levels of implementation to deliver the programme. However, in trials conducted in the community there is a balance to be achieved between quantity and quality of data. Given our previous experiences [32–33], weekly Excel diaries were considered the preferred method to capture programme delivery information as they use a shorter recall period and could be used to track activity as and when it happened. However, the trial demonstrated this approach does not seem to fit with the school context or how teachers operate during a school day; teachers in the trial most preferred the surveys. In future trials, it would be beneficial to undertake consultations with the teachers prior to the start of a trial to find a method to capture costs that works for both the teachers and researchers. It would also negate the need to adapt and develop methods through the course of the trial, which could be burdensome and confusing for those completing them. However, given the experiences of the researchers in the trial we would advise others undertaking research in school settings to consider alternatives if prechosen methods prove unsuccessful. There is also a need to align questioning with other components of the research, for example, qualitative research components. The micro-costing and process evaluation of the trial found disparity in information provided by teachers [27]. This could be attributed to the line of questioning used by each component and the timing of the interviews compared to the follow-up telephone calls.

Service use information was difficult to obtain in the trial from participants. Though the measure was kept brief, the seven-month period of recall seemed to be the main source of difficulty. The difficulty in remembering contacts could also be attributed to the relative levels of independence and autonomy in the sample, whose ages ranged from 11 to 14 years old. Though adolescence is considered a time when individuals are experiencing greater independence and autonomy, it is likely that for many of the sample appointments such as those to see a GP would be arranged by a parent or guardian. It is also likely that the parent/guardian would remind their child of the appointment and take them to the surgery, thus perhaps making it harder for participants to recall contacts with the healthcare professionals when asked. Future research could use parental recall rather than pupil recall or explore the use of electronic records.

The trial used an objective measure of physical activity. Though this is a strength of the trial as previous research has criticised the reliance of self-report measures in physical activity research [34–36], the trial could have been further strengthened by quantifying attendance rates, numbers of sick notes and participation of pupils who usually have no interest or enjoyment in PE, sport and physical activity as part of the main outcomes. This additional information could have been used to assess the culture shift in schools, which is a key driver of the ‘Girls Active’ programme. Given the approach of the ‘Girls Active’ programme, which focuses upon empowerment and encouragment in physical activity, sport and PE, inclusion of this information could help ascertain if pupils who would have previously avoided PE and sport using absence, sick notes and non-participation begin to attend lessons, join in and take more of an active role in games and sports. These outcomes would align more with the ethos of the programme, and would provide useful additional data alongside validated objective measures such as accelorometers.