Article Text
PDF
Abstract
Aim In recent years, considerable effort has been devoted to the development and revisions to an international consensus statement on concussion in sport (ICSCS). The aim of this study was to obtain expert views on the methodological rigour and transparency with which the 2008 ICSCS was developed, as a precursor to the development of the 2012 update.
Methods Delegates registered for the 2012 fourth International Conference on Concussion in Sport, selected local concussion researchers not involved in any prior international consensus meetings, and all authors of the 2008 ICSCS published paper were invited to assess the methodological rigour and transparency with which the 2008 ICSCS was developed. The online Appraisal of Guidelines for Research and Evaluation (AGREE) II assessment tool, with six quality domains, was used and domain scores were expressed as a percentage of the maximum possible score for that domain.
Results 18 appraisers completed the online AGREE II assessment. Ten appraisers said they would recommend the 2008 ICSCS for use (without modification) and seven said they would recommend its use with some modification. The ‘scope and purpose’ and ‘clarity of presentation’ were rated highest, both scoring 78%. The lowest scoring domain was ‘applicability’ with a score of 55%.
Conclusions The quality of the ICSCS is important because it is used to guide return-to-play decisions and the management of sport-related concussions. This appraisal of the 2008 ICSCS suggests that a greater focus is needed on the actual implementation of future ICSCS and the relationship between implementation and desired health outcomes.
- Implementation
- Injury Prevention
- Concussion
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
Introduction
Sports concussion is a growing public health concern in most collision sports, particularly the football codes. Together with the need for evidence-based assessment and management strategies for sport-related concussion, this has resulted in a number of major international consensus statements for concussion in sport (ICSCS) being developed and widely disseminated.1–4 One of the key international initiatives has been the formation of the ‘Concussion in Sport’ (CIS) group, which consists of worldwide experts working in the field of sports concussion.
Since 2001, in conjunction with several leading sports organisations including the International Ice Hockey Federation, International Olympic Committee, Fédération Internationale de Football Association and the International Rugby Board, the CIS group has held conferences to revise and update key aspects of sports concussion knowledge. These conferences have resulted in consensus statements on key definitions, assessment and management strategies for sports concussion. The CIS group has moved from providing only position and agreement statements to using a recognised methodology to obtain consensus at its conferences leading to the publication of international consensus statements. In addition, the conferences prior to this study have led to the development of evaluation tools, the Sports Concussion Assessment Tool (SCAT) and SCAT2 aimed at improving and standardising the sideline evaluation of sports-related concussion. The concussion statements and concussion evaluation tools are freely available and have been ‘developed for use by doctors, therapists, health professionals, coaches and other people involved in the care of injured athletes, whether at the recreational, elite or professional level’.3
The international statements on sports concussion have been widely cited in the research literature5 and their recommendations have been used to guide diagnosis and treatment of sports concussion around the world. Given the intended impact of these statements on sports medicine practice, it is vital that a rigorous methodology be used to develop them to ensure that their recommendations are sound. Until now, the quality of the various ICSCS in terms of the methodological rigour and transparency with which they were developed has not been independently assessed.
Defining the quality of guidelines and consensus statements can be complicated and previous studies have shown that the quality of development of clinical guidelines is variable.6–8 The Institute of Medicine defines clinical guidelines as ‘systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances’.9 In principle, a ‘good’ clinical guideline acts as a link between clinical research and its translation into practice, leading to improved health outcomes. One of the key assumptions behind clinical guidelines is that the greater the strength of the evidence used, the higher the quality of the guideline and the greater potential for use in actual healthcare practice. Governments, professional associations and healthcare organisations have often developed guidelines based on the consensus of expert groups. Several limitations of consensus/expert guidelines/statements have previously been identified10: they may not consider all of the published evidence available; the quality of the evidence may not be considered rigorously; there may be a lack of transparency in how recommendations are derived; and the recommendations generated may not be reproducible by other expert groups. Because clinical guidelines can vary in quality, it is important to assess the methods used to develop practice guidelines in order to be confident of the resulting recommendations.
One tool for assessing guideline quality, the Appraisal of Guidelines for Research and Evaluation (AGREE),11 was developed by a collaboration between academics and healthcare practitioners in an attempt to coordinate guideline development internationally. The most recent version, AGREE II,12 ,13 is a valid, reliable and internationally recognised tool that assesses the methodological rigour and transparency with which a practice guideline is developed. Several studies have used AGREE II to assess the quality of practice guidelines,7 ,14 ,15 inform practice guideline development and to guide what specific information should be reported in practice guidelines.15 ,16
This paper reports the findings of an evaluation conducted in 2012 using AGREE II of the development of the 2008 ICSCS. It was intended that this assessment would help to identify the independently rated methodological strengths in the development of the 2008 ICSCS to inform the development of the planned 2012 Zurich ICSCS.
Methods
This study was conducted as part of a larger research project concerned with improving the awareness, understanding and management of concussion in community sport. Ethics approval for the study was obtained from the relevant University Human Research Ethics Committee.
The AGREE II guidelines17 recommend that at least two appraisers are used, preferably four, to increase the reliability of the assessment. A variety of strategies were used to recruit appraisers from a broad range of relevant interest groups. These groups were: delegates registered for the Fourth International Conference on Concussion in Sport (n=205); targeted Australian/New Zealand concussion researchers not involved in any of the international consensus statements (n=4); and all authors of the paper ‘Consensus Statement on Concussion in Sport 3rd International Conference on Concussion in Sport held in Zurich, November 2008’ (n=7).
The recruited appraisers used the online AGREE II assessment tool to rate the methodological rigour and transparency with which the 2008 ICSCS was developed. AGREE II comprises 23 items organised into six quality domains: (1) scope and purpose; (2) stakeholder involvement; (3) rigour of development; (4) clarity of presentation; (5) applicability; and (6) editorial independence (table 1). The 23 items (table 2) are rated on a 7-point scale (1, strongly disagree; 7, strongly agree). The internal consistency of the AGREE II domains, previously reported in the AGREE II literature, ranges between 0.64 and 0.89.12 Each item also includes a section for comments. There are two final overall assessment items that require the appraiser to make a judgement regarding the overall quality of the guideline on a 7-point scale (1, lowest possible quality; 7, highest possible quality) and decide whether they would recommend the guidelines for use in practice (yes, as is; yes, with modifications; no). In accordance with the AGREE II guidelines,17 all appraisers were encouraged first to undertake a 10 min online training tutorial about how to complete the AGREE II assessment tool prior to completing the appraisal (http://www.agreetrust.org). However, appraiser training was optional and not monitored. All responses were anonymous and completion of the assessment was taken to indicate implied consent.
The AGREE II domains against which the 2008 International Consensus Statement on Concussion in Sport was assessed
Overall ratings of AGREE II items
A score was calculated for each of the six domains by summing all the scores of the individual items within a given domain and scaling the total as a percentage of the maximum possible score for that domain. The formula is provided in the AGREE II manual as follows:
-
Maximum possible score=7(strongly agree)×number of items in that domain×the number of appraisers
-
Minimum possible score=1 (strongly disagree)×number of items in that domain×the number of appraisers
Domain score formula:
Descriptive statistics of the domain scores, across all appraisers, were used to analyse the overall assessments of the guideline.
Thematic analysis of the feedback survey qualitative comments was conducted independently by two researchers and there was agreement in the identified themes.
Results
Eighteen appraisers completed the online AGREE II assessment: three researchers, 13 delegates registered for the Fourth International Conference on Concussion in Sport, and one author of the ‘Consensus Statement on Concussion in Sport 3rd International Conference on Concussion in Sport Held in Zurich, November 2008’ paper. One person did not identify the group to which they belonged, but their views are still included below.
Ten appraisers said that they would recommend the 2008 ICSCS for use (without modification) and seven appraisers said that they would recommend the 2008 ICSCS for use with some modifications; one appraiser said that they would not recommend its use. The median overall rating of the quality of the 2008 ICSCS was 5.5 (range: 3–7). The mean domain scores for all appraisers and domain percentage scores are shown in table 1. The mean scores for each of the 23 items are shown in table 2.
Appraiser feedback
Some appraisers provided comments in relation to specific domains and their overall assessment of the 2008 ICSCS. Several themes related to the target population, methodology and applicability emerged from these comments.
Target population
There were several comments that indicated a need for more information on the intended target population. There were also comments concerning the participation of end-users in the 2008 ICSCS development. For example:
The target population is not clearly described.
As far as can be ascertained there is no evidence that any members of the target population have been included in the development of the consensus statement. The document was developed for use by physicians, therapists, certified athletic trainers, health professionals, coaches and other people involved in the care of injured athletes, whether at the recreational, elite or professional level. However, it does not seem that anyone other than researchers were involved in developing the guidelines.
This was not an objective: the objective was to present ‘empirically based evidence guidelines’ not subjective opinions of stakeholders.
Methodology
The appraisers identified the need for documentation of the methods used to search for evidence and to develop the recommendations. Comments included:
Consensus process #4 indicates that systematic lit reviews were done prior to the consensus meeting, however the guideline does not include specifics regarding search terms, strategy, data extraction, time periods, etc. A full search strategy is not included.
It is not clear that an external review occurred prior to publication (of the consensus statement on concussion in sport).
A lack of, or insufficient, information on the presence of conflicts of interest in the 2008 ICSCS developers was also commented on:
No information is provided about who funded the consensus statement development although there is a statement about no competing interest.
Applicability
Several appraisers indicated that the inclusion of the SCAT2 tool and the presentation of different options for concussion management were useful.
As mentioned, before the SCAT 2 sideline questions need to be sport specific which is what we have done for the (name of sporting organisation removed) and what I use on the sideline with my son's (name of sporting organisation removed) team. It's a fantastic initiative and an exceptionally valuable tool.
Additional comments highlighted the need for greater discussion on the facilitators, barriers and potential resource implications associated with implementing the 2008 ICSCS recommendations.
There is a section on knowledge transfer but this is very brief and does not address any challenges in any systematic way.
This is an area that needs addressing. For example, many sports organizations serving recreational youth athletes are attempting to undertake mass NP testing of their members at substantial cost (if not actual financial cost certainly in terms of limited time and personnel resources). The added benefit of this testing at this level compared to resource allocation needs to be evaluated and appropriate guidance provided.
Overall assessment
Comments accompanying appraisers’ overall assessment of the guidelines included:
The consensus statement seems to be an excellent summary of the evidence related to concussion in sport. However the usefulness of the guidelines and the lack of consideration about how to successfully implement then is a concern.
The application of the AGREE II instrument to this Consensus document needs to be carefully considered as it was probably not the intent of the document to be developed as a “formal” Guideline. The scope is far ranging and it is thus difficult to carefully analyse the evidence and present it in a way that might be expected in a more focused Guideline. This impacts on the scoring of some questions in this assessment.
Discussion
In this study, the AGREE II online guideline assessment tool was used to evaluate the quality of the 2008 ICSCS. The overall ratings provided by the appraisers indicate that the majority believe the guidelines are of quite a high standard. However, while all but one of the appraisers indicated that they would recommend that the 2008 ICSCS be used, more than one-third also wanted to see some modifications made to it.
Among the six domains evaluated, the rated strengths of the 2008 ICSCS appear to lie in the areas of scope and purpose (Domain 1), clarity of presentation (Domain 4) and editorial independence (Domain 6). The appraisers identified applicability (Domain 5) as the area where the most improvement could be made in the next version of the ICSCS. This area pertains to the likely barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guidelines. The involvement of stakeholders (Domain 2) and rigour of development (Domain 3) also received lower quality scores compared with domains 1, 4 and 6. The AGREE II Consortium has not set minimum domain scores or patterns of scores across domains to differentiate between high quality and poor quality guidelines, so the scores of the AGREE II evaluation require careful interpretation.
Previous studies using AGREE or AGREE II to evaluate the quality of clinical guidelines have reported a similar pattern of strengths and weaknesses in domain scores.14 ,15 One study,15 using the AGREE tool, evaluated the quality of methodological rigour of WHO guidelines on maternal and perinatal health, published between 2007 and 2011. For the evaluation of the 2011 maternal and perinatal health guideline, ‘scope and purpose’ and ‘clarity of presentation’ were the highest scoring domains (89% and 93%, respectively) and ‘applicability’ was the lowest scoring domain (58%). A systematic review of the methodological quality of clinical guideline development for the management of chronic disease in Europe included nine studies, analysing 28 European guidelines from eight countries.14 That study reported that the lowest scoring domains, using AGREE II, were editorial independence (41%) and applicability (44%), while scope of purpose (84%) rated more highly. Consistent with the new results from our study reported here, the findings of these studies indicate that the scope of purpose is generally a high scoring domain across guidelines, while applicability is consistently among the lowest scoring domains.
Applicability (Domain 5) was the lowest scoring domain (55%) in the current study. The low scores for applicability may be due to a view that guideline/statement development and implementation are separate activities. It is well known that the existence of evidence-based guidelines or consensus statements does not guarantee their implementation.18 ,19 Although the 2008 ICSCS recognises the need to optimise knowledge transfer and education strategies, there has been limited work towards this aspect until now.20 The various ICSCS have been widely disseminated in peer-reviewed journals and it has been suggested that this demonstrates ‘implementation’/knowledge exchange.21 While this method may be considered effective from a clinical perspective, it does not take into account factors that may affect the adoption of ICSCS by the end-users, such as other health professionals and those involved in the delivery of safe sport. Many of the clinical experts in the CIS group are engaged with professional sport and peak sport bodies, and this no doubt contributed to the adoption of ICSCS by such organisations. Discouragingly, where sports bodies have not implemented the ICSCS18 ,22 concussion and return-to-play knowledge are likely to be poor. Getting the information on concussion assessment and management to the target audience—health professionals, coaches, players and parents—remains a challenge. Recommendations for improving the applicability of guidelines include the inclusion of relevant professionals with the relevant expertise early in the guideline development process to identify and address potential barriers, cost implications and criteria for guideline monitoring.20 ,21 ,23
The domain for rigour of development (Domain 3) received had a relatively low score in this study. This may be due to a perceived lack of methodological expertise in the 2008 ICSCS developing team or a lack of resources to employ a systematic review of the evidence. This domain can be considered to have the most impact on guideline quality as it ensures that a rigorous, systematic approach is used for identifying and reviewing the evidence on which guidelines are developed. The assumption is that the stronger the evidence, the higher the quality of ICSCS and the increased likelihood of their uptake. The low score could also be due to the poor reporting of the methods used in the development of the 2008 ICSCS. This could be improved by including search terms and strategies, data extraction or evidence tables in future statements.
The low rating for the domain for the involvement of stakeholders (Domain 2) indicates a need for improvement in this area. Low scores could reflect the lack of multidisciplinary involvement and input from the end-users. Stakeholder involvement is critical to the relevance, uptake and effectiveness of clinical guidelines and other interventions.24 Recommendations to increase stakeholder involvement include the inclusion of end-user representatives during guideline development; qualitative research to determine the views and preferences of the target population; and the involvement of all relevant disciplines in future ICSCS development groups.15 ,23
While the overall quality ratings for the scope and purpose (Domain 1) and editorial independence (Domain 6) were relatively high, appraiser comments indicated that there were still opportunities for improvement within these domains. Specific information on the target population and expected health outcomes would contribute to the strength of the domain for scope and purpose. Information about funding sources and conflict of interests would help to clearly establish editorial independence. Some comments, where appraisers questioned whether ICSCS should be considered as formal guidelines and whether feedback from stakeholders was relevant, raise issues around the intended use of the 2008 ICSCS and clear articulation of this. The demonstrated impact of the published ICSCS,5 and the fact that they are designed for use by those acting in the field caring for injured athletes,3 suggests that the quality of the statements and stakeholder input are relevant and essential.
There are several limitations to the analysis in this study. While AGREE II has been tested for reliability and validity, the scores of the AGREE II evaluation should be interpreted with caution, as there is no published threshold for discriminating between ‘‘high quality’’ and ‘‘low quality’’ guidelines. AGREE II was developed to assess clinical guidelines and the 2008 ICSCS is not a clinical guideline, as such, even though it may have been interpreted to be used in this way. It may be that AGREE II is not the best instrument for assessing the methodological rigour and transparency in the 2008 ICSCS development, but the range of pertinent comments provided by the appraisers suggests that it could be applied in this context and is relevant.
This study did not aim for the appraiser group to be a representative sample of the interest groups approached for this study. It is possible that the participating appraisers could have been relatively inexperienced in guideline development and evaluation. This may have affected the quality of the results and provided ratings and may have been a potential source of bias. However, the use of multiple appraisers from a range of backgrounds is quite likely to have reduced this potential bias. The AGREE II guidelines17 recommend that each guideline is assessed by at least two appraisers; however, four appraisers are preferable, as this increases the reliability of the assessment.
The 2008 ICSCS has been important for guiding return-to-play decisions and the management of sport-related concussion. This appraisal of the 2008 ICSCS suggests that the domains of applicability, rigour of development and stakeholder involvement are key areas for improvement in subsequent ICSCS. The findings from this study were provided to the organisers of the 2012 Zurich consensus meeting ahead of time, to inform deliberations at that meeting and the development of the planned 2012 ICSCS. It is recommended that further research is conducted to evaluate the actual implementation and adoption of future ICSCS and, importantly, the relationship between implementation and health outcomes.
What are the new findings?
-
This is the first independent assessment of the methodological rigour and transparency of the development of any of the international consensus statements on concussion in sport (ICSCS).
-
The majority of appraisers would recommend the 2008 ICSCS for use; however, over a third would like to see some modifications to them.
-
Applicability, which refers to the likely barriers and facilitators to implementation, strategies to improve uptake and resource implications of applying the guidelines, was identified as the area where the most improvement could be made in the next version of ICSCS.
How might it impact on clinical practice in the near future?
-
Users of concussion guidelines and associated documents need to be assured that they are evidence-based. This paper provides assurance that the international consensus statements on concussion in sport (ICSCS) meet this requirement.
-
Successful implementation of guidelines remains a challenge for practitioners and ongoing updates to ICSCS will provide enhancements to make this process easier.
-
There will still be a need to further disseminate the guidelines more widely across all sports and practitioners and clinicians have a clear role to play in this.
References
Footnotes
-
Contributors PW led the conduct and design of this study and wrote the Results and Methods sections. AWS led the writing of the rest of the paper and its placement in the context of international literature. CFF contributed to the design and concept of the study. All authors contributed significantly by providing editorial input into the manuscript.
-
Funding This work was supported by a Victorian Sports Injury Prevention Research Grant from the Department of Planning and Community Development, Victoria, Australia. CFF was supported by a National Health and Medical Research Council Principal Research Fellowship (ID: 565900).
-
Competing interests None.
-
Ethics approval Monash University Human Ethics Committee.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
▸ References to this paper are available online at http://bjsm.bmj.com