Abstract
A novel real-world validation of two tools (PICADAR and NA-CDCF) that use routinely collected clinical data and help triage the need for primary ciliary dyskinesia diagnostic testing is performed, showing both tools perform well with equivalent performance https://bit.ly/2YgzxlY
To the Editor:
Primary ciliary dyskinesia (PCD) is an important cause of suppurative airway disease and other comorbidities [1]. Due to nonspecific signs and symptoms, the diagnosis of PCD is challenging, requiring specific expertise [2, 3]. PCD testing should be conducted at specialised PCD centres; however, in many places testing is not readily available. Predictive tools based on clinical presentation would help clinicians triage referral for specialised testing. Two such predictive tools have been developed: the PICADAR score [4] and North American Criteria Defined Clinical Features (NA-CDCF) [5]. There has been little validation of these tools outside the settings where they were developed. The original PICADAR publication contained a constructed validation cohort of 187 patients, half of whom had PCD. A modified version of the score has been shown to have good clinical performance in a real-world adult bronchiectasis cohort [6]. Validation in external cohorts is vital, especially as there is variation in the methods used to diagnose PCD. Investigations used to diagnose PCD include genetic testing [2], and assessment of structural and functional ciliary defects using techniques such as high-speed videography (HSV), transmission electron microscopy (TEM) and immunofluorescence (IF) [3]. The PICADAR score was developed in a setting that primarily uses these latter techniques, whereas the NA-CDCF were derived from patients investigated primarily with genetic testing and TEM [4, 5]. We aimed to assess the performance of both predictive tools in an external cohort.
We undertook an ethically approved (HREC 3842) review of the PCD diagnostic clinic at Royal Children's Hospital (Melbourne, Australia). The clinic is a specialised service that primarily uses HSV and TEM when assessing patients for PCD. Selected patients have genetic testing and IF, with these two tests introduced to the clinic in 2019. Some patients also undergo nasal nitric oxide (nNO) testing; however, the results do not influence whether a patient proceeds to further testing. It accepts referrals from an area with a population of 8.5 million. The records of patients seen in the clinic between April 2016 and 2019 were reviewed. Based on clinical documentation, information was extracted into a template. The PICADAR score (calculated out of 14) and NA-CDCF (based on four criteria) was recorded for each patient, as well as whether they were diagnosed with PCD or not. Where either score or PCD status could not be accurately ascertained patients were excluded. The diagnosis of PCD was based on one or more of the following: 1) abnormal ciliary pattern on HSV [7]; 2) TEM or IF findings [8]; 3) identification of PCD causing genetic mutations [2]. Statistical analysis was performed using Graphpad Prism 7 (GraphPad Software, La Jolla, CA, USA), and sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) and receiver operating characteristic (ROC) curves were calculated. Area under ROC were compared using publically available software for correlated [9] and independent [10] datasets comparison. The performance of the PICADAR score and NA-CDCF was compared within our cohort, and the performance of each tool within our cohort was compared to the performance in the original studies.
We analysed the records of 222 patients; however, 11 were excluded as complete results of PCD testing were not available, leaving 211 patients for analysis. The median age (range) of patients was 6 years (0.25–71 years) and 42.2% were female. The majority of patients had moist cough (200/211, 94.8%); 104/211 (49.3%) had persistent rhinitis, 83/211 (39.3%) had ear disease, 71/211 (33.6%) had neonatal respiratory disease and 20/211 (9.5%) had laterality defects. HSV was performed on 208/211 (98.6%), TEM on 164/211 (77.7%), genetics on 3/211 (1.4%) and IF on 4/211 (1.9%). 56/211 (26.5%) had nNO measured. 25 (11.8%) patients had PCD. 22 patients were diagnosed based on abnormal HSV and TEM, two on abnormal HSV and genetic mutations, and one had abnormal HSV and IF. The mean±sd PICADAR score for those with PCD was 7.28±2.88 and those without 3.99±2.46. The area under the ROC was 0.82 (95% CI 0.73–0.90, p<0.001) (figure 1a). The mean±sd NA-CDCF for those with PCD was 2.68±0.99 and those without 1.57±0.89. The area under the ROC was 0.80 (95% CI 0.70–0.90, p<0.001) (figure 1a). The performance of both tools (including sensitivity, specificity, PPV and NPV) for all different cut-off values are shown in figure 1b. In this cohort there were no significant difference between the area under the PICADAR ROC and NA-CDCF ROC (area under the curve (AUC) difference 0.0199, p=0.60). There was also no significant difference when the area under the ROC derived from this cohort was compared to the original studies for both the PICADAR score (AUC difference 0.05, p=0.40) or the NA-CDCF (AUC difference 0.04, p=0.49).
This is the first study to directly compare the performance of PICADAR and NA-CDCF in the same cohort and shows there was no significant difference in the performance of either tool. Both tools are simple and use information collected as part of routine assessment. The advantage of NA-CDCF is it needs collection of fewer variables (four versus eight), while the advantage of PICADAR is as it is scored out of 14 it gives clinicians greater choice in determining which sensitivity and specificity cut-off values they would like to employ with regards to their clinical practice. Clinicians can review the performance of either tool, for all possible cut-off values, and decide which tool and cut-off value is most appropriate for their setting. For instance, using a PICADAR cut-off value of 5 (suggested in the original publication) as a threshold for whether patients proceed to diagnostic testing would result in a sensitivity of 0.76 and specificity of 0.69, which is lower than the 0.86 and 0.73 in the original publication. If that cut-off was applied to the current cohort of 211 there would have been six patients with PCD not referred for testing and 58 patients who would undergo testing unnecessarily. Using a NA-CDCF cut-off value of 2 (recommended in American Thoracic Society Guidelines [2]) as a threshold for whether patients proceed to diagnostic testing would result in a sensitivity of 0.92 and specificity of 0.46, which compares to 0.80 and 0.72 in the original publication. If that cut-off was applied to the current cohort of 211 there would have been two patients with PCD not referred for testing and 100 patients who would undergo testing unnecessarily. If the previously proposed cut-off values for each test are used on the current cohort the PICADAR score limits unnecessary testing, however, it results in more missed diagnoses; whereas the NA-CDCF results in fewer missed diagnoses, but more unnecessary testing.
The PICADAR score was derived from 641 patients assessed at a single centre in the UK, and then validated in 187 patients from a different UK centre chosen so there were equal numbers of patients with and without PCD (93 with PCD, 94 without) [4]. In this study the mean PICADAR score for those with and without PCD was 7.28 and 3.99, respectively, and this compares to 7.9 and 3.5 in the original study. The area under the ROC for the PICADAR score was 0.82 (95% CI 0.73–0.90) in the current study, which is equivalent to the validation cohort of the original study (0.87, 95% CI 0.81–0.94). To the authors knowledge this is the first time the PICADAR score has been validated in a true external “real-world” cohort. The PICADAR score was derived from a mixed paediatric and adult population, but the validation cohort only included patients under 18 years of age. The inclusion of some adult patients in the current cohort did not affect the performance of the score.
The NA-CDCF were defined based on assessment of 205 patients with definite PCD and 187 negative patients, who were assessed at seven North American centres [5]. The area under the ROC curve for the NA-CDCF was 0.80 (95% CI 0.70–0.90), which was not significantly different to the original study (0.84, 95% CI 0.81–0.88). This is the first validation of the NA-CDCF to occur. The equivalent performance of the NA-CDCF in a cohort which uses HSV and TEM and rarely genetic testing is of note. Also, the NA-CDCF were derived from a cohort of paediatric patients only, yet it performed well in the current cohort that included 24 (11.4%) patients who were older than 18 years.
There are several limitations of the current study. The single centre nature of the study is a limitation. In particular, PCD diagnostic services may differ in how strongly they pursue a PCD diagnosis. The fact our centre tests referrals irrespective of nNO result, on occasion uses additional tests on top of HSV and TEM, and has made challenging diagnoses, such as PCD related to DNA11 mutation, indicates that we are relatively “aggressive” in making a PCD diagnosis and this may limit the applicability of our findings to other centres. The performance of the two clinical tools may also be different in centres who utilise the diagnostic tests differently. There is a small number of positive cases when compared to the original studies; however, this reflects the reality of a well-established clinic over 3 years. This is particularly relevant to the PPV and NPV reported in the current cohort, as they are dependent on prevalence, and hence in a clinic with a different prevalence these values will be altered. The retrospective nature of data collection and reliance on documentation on medical records to calculate the scores is a limitation. This limitation is minimised by the exclusion of patients where either the predictive tools or PCD status could not be accurately determined. Lastly, we did not assess the performance of the weighted models proposed in both of the original studies as they are unlikely to be widely used in clinical practice.
This study contributes novel findings to the literature. First, there is no significant difference in the performance of either the PICADAR score or NA-CDCF when applied to an external cohort. Secondly, when applied to the same cohort both scores have equivalent performance. This suggests that either test could be used to aid clinicians in deciding when to refer patients for PCD testing and to also help PCD centres triage referrals for diagnostic testing.
Shareable PDF
Supplementary Material
This one-page PDF can be shared freely online.
Shareable PDF ERJ-01169-2020.Shareable
Footnotes
Conflict of interest: K. Palmas has nothing to disclose.
Conflict of interest: S. Shanthikumar has nothing to disclose.
Conflict of interest: P. Robinson has nothing to disclose.
- Received April 14, 2020.
- Accepted June 5, 2020.
- Copyright ©ERS 2020