Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Translational Therapeutics

Development of an AI system for accurately diagnose hepatocellular carcinoma from computed tomography imaging data

Abstract

Background and aims

Computed tomography (CT) scan is frequently used to detect hepatocellular carcinoma (HCC) in routine clinical practice. The aim of this study is to develop a deep-learning AI system to improve the diagnostic accuracy of HCC by analysing liver CT imaging data.

Methods

We developed a deep-learning AI system by training on CT images from 7512 patients at Henan Provincial Peoples’ Hospital. Its performance was validated on one internal test set (Henan Provincial Peoples’ Hospital, n = 385) and one external test set (Henan Provincial Cancer Hospital, n = 556). The area under the receiver-operating characteristic curve (AUROC) was used as the primary classification metric. Accuracy, sensitivity, specificity, precision, negative predictive value and F1 metric were used to measure the performance of AI systems and radiologists.

Results

AI system achieved high performance in identifying HCC patients, with AUROC of 0.887 (95% CI 0.855–0.919) on the internal test set and 0.883 (95% CI 0.855–0.911) on the external test set. For internal test set, accuracy was 81.0% (76.8–84.8%), sensitivity was 78.4% (72.4–83.7%), specificity was 84.4% (78.0–89.6%) and F1 (harmonic average of precision and recall rate) was 0.824. For external test set, accuracy was 81.3% (77.8–84.5%), sensitivity was 89.4% (85.0–92.8%), specificity was 74.0% (68.5–78.9%) and F1 was 0.819. Compared with radiologists, AI system achieved comparable accuracy and F1 metric on internal test set (0.853 versus 0.818, P = 0.107; 0.863 vs. 0.824, P = 0.082) and external test set (0.805 vs. 0.793, P = 0.663; 0.810 vs. 0.814, P = 0.866). The predicted HCC risk scores by AI system in HCC patients with multiple tumours and high fibrosis stage were higher than those with solitary tumour and low fibrosis stage (tumour number: 0.197 vs. 0.138, P = 0.006; fibrosis stage: 0.183 vs. 0.127, P < 0.001). Radiologists’ review showed that the accuracy of saliency heatmaps predicted by algorithms was 92.1% (95% CI: 89.2–95.0%).

Conclusions

AI system achieved high performance in the detection of HCC compared with a group of specialised radiologists. Further investigation by prospective clinical trials was necessitated to verify this model.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A flowchart depicting the procedures to develop and evaluate HCCNet.
Fig. 2: Performance of the AI model HCCNet and radiologists on two test sets.
Fig. 3: The predicted HCC risk score by AI model HCCNet on test sets stratified by HCC tumour size, AJCC TNM stage, tumour number and METAVIR fibrosis stage.
Fig. 4: Comparisons of diagnostic performance by radiologists with and without AI assistance.
Fig. 5: Exemplified images of HCC tumours.

Similar content being viewed by others

Data availability

The original CT data that consist of the training sets and test sets are available on request from the corresponding author. They are not publicly available due to the privacy of research participants.

Code availability

The code used in this study to train AI models is freely accessed and uploaded in supplementary files.

References

  1. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68:394–424. https://doi.org/10.3322/caac.21492.

    Article  PubMed  Google Scholar 

  2. El-Serag HB. Hepatocellular carcinoma. N Engl J Med. 2011;365:1118–27. https://doi.org/10.1056/NEJMra1001683.

    Article  PubMed  CAS  Google Scholar 

  3. Lim JH, Kim MJ, Park CK, Kang SS, Lee WJ, Lim HK. Dysplastic nodules in liver cirrhosis: detection with triple phase helical dynamic CT. Br. J Radiol. 2004;77:911–6. https://doi.org/10.1259/bjr/56713551.

    Article  PubMed  CAS  Google Scholar 

  4. Choi JY, Lee JM, Sirlin CB. CT and MR imaging diagnosis and staging of hepatocellular carcinoma: part I. Development, growth, and spread: key pathologic and imaging aspects. Radiology. 2014;272:635–54. https://doi.org/10.1148/radiol.14132361.

    Article  PubMed  Google Scholar 

  5. Laghi A, Iannaccone R, Rossi P, Carbone L, Ferrari R, Mangiapane F, et al. Hepatocellular carcinoma: detection with triple-phase multi-detector row helical CT in patients with chronic hepatitis. Radiology. 2003;226:543–9. https://doi.org/10.1148/radiol.2262012043.

    Article  PubMed  Google Scholar 

  6. Ippolito D, Sironi S, Pozzi M, Antolini L, Invernizzi F, Ratti L, et al. Perfusion CT in cirrhotic patients with early stage hepatocellular carcinoma: assessment of tumor-related vascularization. Eur J Radiol. 2010;73:148–52. https://doi.org/10.1016/j.ejrad.2008.10.014.

    Article  PubMed  Google Scholar 

  7. Sahani DV, Holalkere NS, Mueller PR, Zhu AX. Advanced hepatocellular carcinoma: CT perfusion of liver and tumor tissue-initial experience. Radiology. 2007;243:736–43. https://doi.org/10.1148/radiol.2433052020.

    Article  PubMed  Google Scholar 

  8. Khalili K, Kim TK, Jang HJ, Haider MA, Khan L, Guindi M, et al. Optimization of imaging diagnosis of 1-2 cm hepatocellular carcinoma: an analysis of diagnostic performance and resource utilization. J Hepatol. 2011;54:723–8. https://doi.org/10.1016/j.jhep.2010.07.025.

    Article  PubMed  Google Scholar 

  9. Bruix J, Sherman M. Management of hepatocellular carcinoma: an update. American Association for the Study of Liver Diseases. Hepatology. 2011;53:1020–2. https://doi.org/10.1002/hep.24199.

    Article  PubMed  Google Scholar 

  10. European Association For The Study Of The Liver; European Organisation For Research And Treatment Of Cancer. EASL-EORTC clinical practice guidelines: management of hepatocellular carcinoma. J Hepatol. 2012;56:908–43. https://doi.org/10.1016/j.jhep.2011.12.001.

    Article  Google Scholar 

  11. Omata M, Lesmana LA, Tateishi R, Chen PJ, Lin SM, Yoshida H, et al. Asian Pacific Association for the Study of the Liver consensus recommendations on hepatocellular carcinoma. Hepatol Int. 2010;4:439–74. https://doi.org/10.1007/s12072-010-9165-7.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Tang A, Bashir MR, Corwin MT, Cruite I, Dietrich CF, Do RKG, et al. Evidence supporting LI-RADS major features for CT- and MR imaging-based diagnosis of hepatocellular carcinoma: a systematic review. Radiology. 2018;286:29–48. https://doi.org/10.1148/radiol.2017170554.

    Article  PubMed  Google Scholar 

  13. Bruix J, Sherman M. Management of hepatocellular carcinoma. Practice Guidelines Committee, American Association for the Study of Liver Diseases. Hepatology. 2005;42:1208–36. https://doi.org/10.1002/hep.20933.

    Article  PubMed  Google Scholar 

  14. Heimbach JK, Kulik LM, Finn RS, Sirlin CB, Abecassis MM, Roberts LR, et al. AASLD guidelines for the treatment of hepatocellular carcinoma. Hepatology. 2018;67:358–80. https://doi.org/10.1002/hep.29086.

    Article  PubMed  Google Scholar 

  15. Russo FP, Imondi A, Lynch EN, Farinati F. When and how should we perform a biopsy for HCC in patients with liver cirrhosis in 2018? A review. Dig Liver Dis. 2018;50:640–6. https://doi.org/10.1016/j.dld.2018.03.014.

    Article  PubMed  Google Scholar 

  16. Roberts LR, Sirlin CB, Zaiem F, Almasri J, Prokop LJ, Heimbach JK, et al. Imaging for the diagnosis of hepatocellular carcinoma: a systematic review and meta-analysis. Hepatology. 2018;67:401–21. https://doi.org/10.1002/hep.29487.

    Article  PubMed  Google Scholar 

  17. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18:500–10. https://doi.org/10.1038/s41568-018-0016-5.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  18. Bi WL, Hosny A, Schabath MB, Giger ML, Birkbak NJ, Mehrtash A, et al. Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin. 2019;69:127–57. https://doi.org/10.3322/caac.21552.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. 2019;380:1347–58. https://doi.org/10.1056/NEJMra1814259.

    Article  PubMed  Google Scholar 

  20. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J Am Med Assoc. 2016;316:2402–10. https://doi.org/10.1001/jama.2016.17216.

    Article  Google Scholar 

  21. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8. https://doi.org/10.1038/nature21056.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  22. Li XC, Zhang S, Zhang Q, Wei X, Pan Y, Zhao J, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol. 2019;20:193–201. https://doi.org/10.1016/S1470-2045(18)30762-9.

    Article  PubMed  Google Scholar 

  23. Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet. 2018;392:2388–96. https://doi.org/10.1016/S0140-6736(18)31645-3.

    Article  PubMed  Google Scholar 

  24. Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med. 2019;25:954–61. https://doi.org/10.1038/s41591-019-0447-x.

    Article  PubMed  CAS  Google Scholar 

  25. He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. IEEE conference on computer vision and pattern recognition, 2016. Las Vegas, NV, USA; 2016. p. 770–8.  https://doi.org/10.1109/CVPR.2016.90

  26. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis. 2015;115:211–52.

    Article  Google Scholar 

  27. Zhang HY, Cisse M, Dauphin YN, Lopez-Paz D. Mixup: beyond empirical risk minimization. International Conference on Learning Representations, 2018. arXiv:1710.09412v2. 2018. Available from: https://arxiv.org/abs/1710.09412

  28. Mundhenk TN, Chen B, Friedland G. Efficient saliency maps for explainable AI. International Conference on Learning Representations, 2020.  arXiv:1911.11293v2. 2019. Available from: https://arxiv.org/abs/1911.11293

  29. Bedossa P, Poynard T. An algorithm for the grading of activity in chronic hepatitis C. The METAVIR Cooperative Study Group. Hepatology. 1996;24:289–93. https://doi.org/10.1002/hep.510240201.

    Article  PubMed  CAS  Google Scholar 

  30. Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med. 2019;25:65–69. https://doi.org/10.1038/s41591-018-0268-3.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  31. Zhou DJ, Tian F, Tian XD, Sun L, Huang XH, Zhao F, et al. Diagnostic evaluation of a deep learning model for optical diagnosis of colorectal cancer. Nat Commun. 2020;11:2961 https://doi.org/10.1038/s41467-020-16777-6.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  32. Ahmad OF, Soares AS, Mazomenos E, Brandao P, Vega R, Seward E, et al. Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions. Lancet Gastroenterol Hepatol. 2019;4:71–80. https://doi.org/10.1016/S2468-1253(18)30282-6.

    Article  PubMed  Google Scholar 

  33. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, Ginneken BV, Karssemeijer N, Litjens G, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. J Am Med Assoc. 2017;318:2199–210.

    Article  Google Scholar 

  34. Byrne MF, Chapados N, Soudan F, Oertel C, Pérez ML, Kelly R, et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut. 2019;68:94–100. https://doi.org/10.1136/gutjnl-2017-314547.

    Article  PubMed  Google Scholar 

  35. Sun R, Limkin EJ, Vakalopoulou M, Dercle L, Champiat S, Han SR, et al. A radiomics approach to assess tumor-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol. 2018;19:1180–91. https://doi.org/10.1016/S1470-2045(18)30413-3.

    Article  PubMed  CAS  Google Scholar 

  36. Silva MA, Hegab B, Hyde C, Guo B, Buckels JAC, Mirza DF. Needle track seeding following biopsy of liver lesions in the diagnosis of hepatocellular cancer: a systematic review and meta-analysis. Gut. 2008;57:1592–6. https://doi.org/10.1136/gut.2008.149062.

    Article  PubMed  CAS  Google Scholar 

  37. Khalili K, Kim TK, Jang HJ, Yazdi LK, Guindi M, Sherman M. Indeterminate 1-2-cm nodules found on hepatocellular carcinoma surveillance: biopsy for all, some, or none? Hepatology. 2011;54:2048–54. https://doi.org/10.1002/hep.24638.

    Article  PubMed  Google Scholar 

  38. Chernyak V, Fowler KJ, Kamaya A, Kielar AZ, Elsayes KM, Bashir MR, et al. Liver imaging reporting and data system (LI-RADS) version 2018: imaging of hepatocellular carcinoma in at-risk patients. Radiology. 2018;289:816–30. https://doi.org/10.1148/radiol.2018181494.

    Article  PubMed  Google Scholar 

Download references

Funding

This work was supported by the natural science foundation of Tianjin Education Committee (Project No. 2018KJ072), Tianjin Science and Technology Committee (Project No. 18JCQNJC80800), Henan Provincial Science and Technology Research Projects (Project No. 212102310689) and National Natural Science Foundation of China (Project No. 31801117).

Author information

Authors and Affiliations

Authors

Contributions

MW, XL and FT take full responsibility for the integrity of the data and the accuracy of the analysis. Concept and design were contributed by FT, XL and DK. Drafting of the manuscript is attributed to FT, XL and MW. Clinical record data were obtained and reviewed by FF, BZ, YB and XM. Statistical analysis is attributed to YY and HS. FF, BZ, YB, QW, JW and XM read and interpreted CT images. LS, QL and ML extracted and reviewed pathological data. Technical and material support is attributed to PY, XL and YY. We authors thank Genevieve Nemeth of Harvard Medical School for editing the language of this manuscript.

Corresponding authors

Correspondence to Xiangchun Li or Fei Tian.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent to publish

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Fu, F., Zheng, B. et al. Development of an AI system for accurately diagnose hepatocellular carcinoma from computed tomography imaging data. Br J Cancer 125, 1111–1121 (2021). https://doi.org/10.1038/s41416-021-01511-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41416-021-01511-w

This article is cited by

Search

Quick links