Accuracy of blinded clinician interpretation of single-lead smartphone electrocardiograms and a proposed clinical workflow

doi:10.1016/j.ahj.2018.08.001

American Heart Journal

Volume 205, November 2018, Pages 149-153

https://doi.org/10.1016/j.ahj.2018.08.001 Get rights and content

Despite the appeal of smartphone-based electrocardiograms (ECGs) for arrhythmia screening, a paucity of data exists on the accuracy of primary care physicians' and cardiologists' interpretation of tracings compared with the device's automated diagnosis. Using 408 ECGs in 51 patients, we demonstrate a variable accuracy in clinician interpretation of smartphone-based ECGs, with only cardiologists demonstrating satisfactory agreement when referenced against a 12-lead ECG. Combining the device automated diagnostic algorithm with cardiologist interpretation of only uninterpretable traces yielded excellent results and provides an efficient, cost-effective workflow for the utilization of a smartphone-based ECG in clinical practice.

Section snippets

Methods

This prospective, blinded, observational cohort study was performed at a tertiary university hospital in Australia. Consecutive patients 18 years and older undergoing electrical cardioversions for AF and atrial flutter were recruited over 12 months. Patients with cardiac implantable electronic devices and those unable to hold the device correctly were excluded. The institutional ethics review board approved the study, and written informed consent was obtained from all subjects (ACTRN:

Results

Data from 102 patient encounters for 51 consecutive patients that underwent elective cardioversions were included (mean age 64 ± 15, 65% male). Clinical characteristics are summarized in Online Table I. There were 408 ECG tracings. This included 306 iECGs that were recorded simultaneously with 102 12-lead ECGs. All of the 12-lead ECGs were interpretable, and only 9 iECG tracings (2.9%) were deemed noninterpretable by both EPs and PCPs. These were subsequently marked as incorrectly identified.

Discussion

This study demonstrated 3 key findings: (1) Accuracy of clinician interpretation was variable, with only EPs demonstrating satisfactory agreement with 12-lead ECG. (2) When offered by the AHM, an automated diagnosis yielded comparable diagnostic accuracy to clinician interpretation of tracings. (3) Incorporation of the AHM autodiagnosis with EP interpretation of only “unclassified” tracings resulted in satisfactory diagnostic accuracy which was comparable to EP interpretation of all iECGs.

The

Conclusion

In this cohort of patients, EP interpretation of iECG tracings demonstrated satisfactory diagnostic accuracy when compared with 12-lead ECG. The automated device algorithm was comparable to this only when uninterpretable traces were excluded. However, combining the device automated diagnostic algorithm with EP interpretation of only uninterpretable traces yielded excellent results and provides an efficient, cost-effective workflow for the utilization of a smartphone-based ECG in clinical

References (10)

J.K. Lau et al.
iPhone ECG application for community screening to detect silent atrial fibrillation: a novel technology to prevent stroke
Int J Cardiol
(2013)
I. AliveCor
User manual for Kardia by AliveCor
J.P.J. Halcox et al.
Assessment of remote heart rhythm sampling using the AliveCor Heart Monitor to screen for atrial fibrillation: the REHEARSE-AF study
Circulation
(2017)
J. Orchard et al.
iPhone ECG screening by practice nurses and receptionists for atrial fibrillation in general practice: the GP-SEARCH qualitative pilot study
Aust Fam Physician
(2014)
J. Orchard et al.
Screening for atrial fibrillation during influenza vaccinations by primary care nurses using a smartphone electrocardiograph (iECG): a feasibility study
Eur J Prev Cardiol
(2016)

There are more references available in the full text version of this article.

Cited by (26)

Accuracy and variability of cardiologist interpretation of single lead electrocardiograms for atrial fibrillation: The VITAL-AF trial
2023, American Heart Journal
Screening for atrial fibrillation (AF) using consumer-based devices capable of producing a single lead electrocardiogram (1L ECG) is increasing. There are limited data on the accuracy of physician interpretation of these tracings. The goal of this study is to assess the sensitivity, specificity, confidence, and variability of cardiologist interpretation of point-of-care 1L ECGs.
Fifteen cardiologists reviewed point-of-care handheld 1L ECGs collected from patients aged 65 years or older enrolled in the VITAL-AF clinical trial [NCT035115057] who underwent cardiac rhythm assessments with a 1L ECG using an AliveCor KardiaMobile device. Random sampling of 1L ECGs for cardiologist review was stratified by the AliveCor algorithm interpretation. A 12L ECG performed on the same day for clinical purposes was used as the gold standard. Cardiologists each reviewed a common sample of 200 1L ECG tracings and completed a survey associated with each tracing. Cardiologists were blinded to both the AliveCor algorithm and same day 12L ECG interpretation. For each tracing, study cardiologists were asked to assess the rhythm (sinus rhythm, AF, unclassifiable), report their assessment of the quality of the tracing, and rate their confidence in rhythm interpretation. The outcomes included the sensitivity, specificity, variability, and confidence in physician interpretation. Variables associated with each measure were identified using multivariable regression.
The average sensitivity for AF was 77.4% (range 50%-90.6%, standard deviation [SD]=11.4%) and the average specificity was 73.0% (range 41.3%-94.6%, SD = 15.4%). The mean variability was 30.8% (range 0%-76.2%, SD = 23.2%). The average reviewer confidence of 1L ECG rhythm assessment was 3.6 out of 5 (range 2.5-4.2, SD = 0.6). Patient and tracing factors associated with sensitivity, specificity, variability, and confidence were identified and included age, body mass index, and presence of artifact.
Cardiologist interpretation of point-of-care handheld 1L ECGs has modest diagnostic sensitivity and specificity with substantial variability for AF classification despite high confidence. Variability in cardiologist interpretation of 1L ECGs highlights the importance of confirmatory testing for diagnosing AF.
Smart Devices in Detecting AF: Excellent Signal Quality, But AI Can Still Learn From Clinicians
2023, JACC: Clinical Electrophysiology
Comparison of 2 Smart Watch Algorithms for Detection of Atrial Fibrillation and the Benefit of Clinician Interpretation: SMART WARS Study
2022, JACC: Clinical Electrophysiology
Smart watches and wearable technology capable of heart rhythm assessment have increased in use in the general population. The Apple Watch Series 4 (AW4) and KardiaBand (KB) are devices capable of obtaining single-lead electrocardiographic recordings, presenting a novel opportunity for the detection of paroxysmal arrhythmias.
The aim of this study was to assess the diagnostic utility of the AW4 and KB in an elderly outpatient population.
Consecutive recordings were taken from patients attending cardiology outpatient clinic from the AW4 and KB concurrently with 12-lead electrocardiography. Automated diagnoses and blinded single-lead electrocardiographic tracing interpretations by 2 cardiologists were analyzed. Analysis was also conducted to assess the effect of combined device and clinician interpretation.
One hundred twenty-five patients were prospectively recruited (mean age 76 ± 7 years, 62% men). The accuracy of the automated rhythm assessment was higher with the KB than the AW4 (74% vs 65%). For the detection of atrial fibrillation, the sensitivity and negative predictive value of the KB were 89% and 97%, respectively, and of the AW4 were 19% and 82%, respectively. Using hybrid automated and clinician interpretation, the overall accuracy of the KB and AW4 was 91% and 87%, respectively.
The KB automated algorithm outperformed the AW4 in its accuracy and sensitivity for detecting atrial fibrillation in the outpatient setting. Clinician assessment of the single-lead electrocardiogram improved accuracy. These findings suggest that although these devices’ tracings are of sufficient quality, automated diagnosis alone is not sufficient for making clinical decisions about atrial fibrillation diagnosis and management.
Use of Smartphones and Wearables for Arrhythmia Monitoring
2021, Cardiac Electrophysiology Clinics
Citation Excerpt :
For interpretable tracings, the AliveCor algorithm has been found highly accurate with sensitivity and specificity approaching or exceeding 95% in multiple studies.32,47 The single-lead ECG has been shown to be interpretable by both cardiologists and primary care physicians.47 One meta-analysis of 14 studies examining 7 different handheld devices found that physicians in both hospital and community settings can interpret the single-lead ECGs accurately compared with standard 12-lead ECGs or Holter monitors.48
Survey of current perspectives on consumer-available digital health devices for detecting atrial fibrillation
2020, Cardiovascular Digital Health Journal
Citation Excerpt :
Most commercially available digital health technologies for AF detection record brief (often 30-second) windows of pulse or electrocardiographic data and then employ algorithms that determine the patient’s rhythm status. These algorithms have been shown to be highly accurate when compared to expert review of clinical-grade electrocardiograms (ECGs).7–9 While the European Heart Rhythm Association recommends systematic AF screening for individuals at highest risk, clinical guidelines set by many other professional and government entities, such as the US Preventative Services Task Force,10 do not presently recommend widespread use, since the aggregate benefit of this approach has not been established.
Many digital health technologies capable of atrial fibrillation (AF) detection are directly available to patients. However, adaptation into clinical practice by heart rhythm healthcare practitioners (HCPs) is unclear.
To examine HCP perspectives on use of commercial technologies for AF detection and management.
We created an electronic survey for HCPs assessing practice demographics and perspectives on digital devices for AF detection and management. The survey was distributed electronically to all members of 3 heart rhythm professional societies.
We received 1601 responses out of 73,563 e-mails sent, with 43.6% from cardiac electrophysiologists, 12.8% from fellows, and 11.6% from advanced practice practitioners. Most respondents (62.3%) reported having recommended patient use of a digital device for AF detection. Those who did not had concerns about their accuracy (29.6%), clinical utility of results (22.8%), and integration into electronic health records (19.8%). Results from a 30-second single-lead electrocardiogram were sufficient for 42.7% of HCPs to recommend oral anticoagulation for patients at high risk for stroke. Respondents wanted more data comparing the accuracy of digital devices to conventional devices for AF monitoring (64.9%). A quarter (27.3%) of HCPs had no reservations recommending digital devices for AF detection, and most (53.4%) wanted guidelines from their professional societies providing guidance on their optimal use.
Many HCPs have already integrated digital devices into their clinical practice. However, HCPs reported facing challenges when using digital technologies for AF detection, and professional society recommendations on their use are needed.
Accuracy of wrist-worn heart rate monitors for rate control assessment in atrial fibrillation
2020, International Journal of Cardiology
Citation Excerpt :
To overcome the limitations of the PPG technology in determining heart rate and rhythm, smartphone and SW technologies utilising single-lead ECG has been developed to assess heart rate and rhythm. Despite their accuracy in detecting AF [9–11], their role in population screening for subclinical AF is of considerable debate [12]. Nevertheless, wearable devices with ECG capability could have a role in assessing rate control amongst patients with known persistent AF and represents an opportunity for future clinical studies.
Wrist-worn heart rate (HR) monitors are increasingly popular. A paucity of data exists on their accuracy in atrial fibrillation (AF) in ambulatory patients. We sought to assess the HR accuracy of two commercially available smart watches [SW] (Fitbit Charge HR [FB] and Apple Watch Series 3 [AW]) compared with Holter monitoring in an ambulant patient cohort.
Thirty-two participants ≥18 years referred for 24-hour Holter monitoring were prospectively recruited. Each participant was randomly allocated to wear either a FB or AW along with their Holter monitor.
Across all devices, 53,288 heart rate values were analysed from 32 participants. Twenty wore the AW (17 had persistent AF and 3 had sinus rhythm [SR]) while 12 participants wore the FB (9 in persistent AF and 3 in SR). Participants in SR demonstrated strong agreement compared to Holter monitoring (bias <1 beat, limits of agreement [LoA] −11 to 11 beats). In AF, both devices underestimated HR measurements (bias −9 beats, LoA −41 to 23). The degree of underestimation was more pronounced when HR > 100 bpm (bias of −28 beats for HR range 100–120 bpm, −48 for 120–140 bpm, and −69 for >140 bpm) compared to a slower HR (bias of −6 for HR range 80–100 bpm, <1 for 60–80 bpm, and −1 for <60 bpm).
In ambulatory patients, smartwatches underestimated HR in AF particularly at HR ranges >100 bpm. Further improvements in device technology are needed before integrating them into the clinical management of rate control in AF.

View all citing articles on Scopus

: Clinical trial registration: Australian & New Zealand Clinical Trials Registry (ACTRN:12616991374459).

: Funding: Dr Koshy is supported by the National Health and Medical Research Council of Australia and National Heart Foundation Scholarship. Dr Negishi is supported by the National Heart Foundation Future Leader Fellow Scholarship. Dr Teh is supported by an Early Career Fellowship from the National Health and Medical Research Council of Australia.

: This work was supported by the Eastern Health Foundation Research Grant (EHFRG2017_029). The sponsor had no role in study design, collection, analysis, and interpretation of data and in the decision to submit the article for publication.

: Declarations of interest: none.

¹: Authors A. N. K. and J. K. S. are co-first authors.

View full text

Research LetterAccuracy of blinded clinician interpretation of single-lead smartphone electrocardiograms and a proposed clinical workflow

Section snippets

Methods

Results

Discussion

Conclusion

Int J Cardiol

User manual for Kardia by AliveCor

Assessment of remote heart rhythm sampling using the AliveCor Heart Monitor to screen for atrial fibrillation: the REHEARSE-AF study

Circulation

iPhone ECG screening by practice nurses and receptionists for atrial fibrillation in general practice: the GP-SEARCH qualitative pilot study

Aust Fam Physician

Screening for atrial fibrillation during influenza vaccinations by primary care nurses using a smartphone electrocardiograph (iECG): a feasibility study

Eur J Prev Cardiol

Research Letter
Accuracy of blinded clinician interpretation of single-lead smartphone electrocardiograms and a proposed clinical workflow