The framework can be broadly applied to many pathogens; however, components of the evaluation may vary according to the specific pathogen and / or disease. Using three case studies, we provide examples of how the evaluation framework can be applied to a range of public health pathogens (Table 2 and Supplementary table 3).
Case study 1: Listeria monocytogenes: Pre/post design
Listeriosis is a notifiable disease in many countries, including Australia and the United States [28–31]. WGS has emerged as a valuable tool for investigation of listeriosis outbreaks and is now routinely used for genomic surveillance in several countries [32, 33]. Previous research has shown that the higher discriminatory power of WGS can identify distinct nested clusters within groups of L. monocytogenes isolates that were otherwise indistinguishable using other typing methods [34] and utility in identifying contamination sources [35].
When there is availability of data prior to the introduction of WGS, a pre / post design can be used to examine data in defined time periods before (based on previously used typing methods) and after transitioning to WGS. Relevant variables to assess may include the number, size and geographical spread of identified clusters, the percent of isolates linked to a cluster, numbers of isolates/clusters traced to a common source, number of ‘solved’ isolates/clusters, and the time taken to identify and resolve outbreaks. Further, the evaluation could encompass the use of and contribution to international genomics databases. Changes in costs relating to epidemiological investigation could be also be analysed, given the utility of WGS in ruling out transmission links. The evaluation could additionally examine trends in food recalls due to L. monocytogenes, including the frequency and magnitude of recalls. This approach allows for a comprehensive understanding of how the use of pathogen genomic data has affected the identification and characterisation of clusters across the surveillance system, as well as resulting effects on public health outcomes and use of public health resources.
Case study 2: Mycobacterium tuberculosis: Retrospective analysis
Tuberculosis (TB) is the leading cause of death from a single infectious agent, with drug-resistant tuberculosis identified as a global health crisis [36]. The long incubation period and relatively high rates of asymptomatic and undiagnosed infection mean that it can be difficult to confirm transmission through epidemiological links alone. WGS has been shown to provide superior discrimination compared to other typing methods and may be more cost-effective [37, 38]. Sequencing has been used to understand TB transmission dynamics, including identifying super-spreader events [39] The use of WGS may therefore support more efficient and effective contact tracing, earlier and more appropriate treatment and the initiation of focused public health interventions.
If historical isolates are available at the time of WGS, retrospective sequencing allows for examination of the question: If WGS had been available during this time, what may have been done differently? Here, a retrospective design may allow evaluators to examine practice and outcomes prior to transition to WGS and compare against what would have been expected to occur if WGS had been available at that time.
Retrospective sequence data in combination with epidemiological data can be used to determine which TB may have been identified earlier, allowing for interventions to disrupt further transmissions. Estimates can then be made regarding the number of possible cases averted along with attendant costs to the health care system, including costs relating to epidemiological investigations that may not have been needed. Given the retrospective nature of this approach, it would be important to incorporate a strong understanding of how public health practice is informed by the use of pathogen genomic data, supported by Phases 2 and 3 of the evaluation framework. Undertaking the evaluation in this way utilises a whole-of-system approach to draw links between how TB genomic data is used and eventual public health outcomes, enabling further refinement of pathogen genomics-informed public health practice. Further, this type of evaluation would provide evidence towards assessing the cost-effectiveness of WGS, which may assist in decision-making regarding resource allocation and investment in pathogen genomics.
Case study 3: SARS-CoV-2: Mathematical modelling
From the first instance of genomic sequencing of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [40], WGS has been integrated into the global public health response to COVID-19. In settings that use WGS, evidence of transmission and identification of clusters using genomic data have been highly influential in public health decision-making [13, 41–43]. If WGS was introduced alongside a significant shift in practice, or surveillance of the pathogen under consideration was recently introduced, there may be a lack of appropriate ‘pre’ data. In the case of COVID-19, a reasonable evaluation question would therefore be: How has WGS contributed (i) to identification of cases and outbreaks, and (ii) to public health interventions? Public health data from similar contexts where WGS has not been used, in conjunction with existing epidemiological data from the setting under consideration, could form the basis of mathematical models to estimate differences in case numbers and characterised outbreaks. While confounders would need to be accounted for in such a model, the aim would not be to arrive at absolute numbers, but rather relative results, such as the proportion of unidentified cases or the probability of being able to detect outbreaks of a certain size with and without the use of genomics [44]. In the absence of suitable comparative data, available epidemiological data may be examined to determine where identification of transmission events would have been uncertain, or where distinct transmission networks may have been merged without the use of genomic data. Interviews with public health authorities may strengthen understanding of how genomic data has been used in public-health decision-making and the links between genomic data and eventual health outcomes. For laboratories that have used or contributed to international sequence databases, this could also be captured as part of the evaluation.