Skip to main content
SpringerLink
Log in

Performance changes due to differences in training data for cerebral aneurysm detection in head MR angiography images

  • Original Article
  • Published:
Japanese Journal of Radiology Aims and scope Submit manuscript

Abstract

Purpose

The performance of computer-aided detection (CAD) software depends on the quality and quantity of the dataset used for machine learning. If the data characteristics in development and practical use are different, the performance of CAD software degrades. In this study, we investigated changes in detection performance due to differences in training data for cerebral aneurysm detection software in head magnetic resonance angiography images.

Materials and methods

We utilized three types of CAD software for cerebral aneurysm detection in MRA images, which were based on 3D local intensity structure analysis, graph-based features, and convolutional neural network. For each type of CAD software, we compared three types of training pattern, which were two types of training using single-site data and one type of training using multisite data. We also carried out internal and external evaluations.

Results

In training using single-site data, the performance of CAD software largely and unpredictably fluctuated when the training dataset was changed. Training using multisite data did not show the lowest performance among the three training patterns for any CAD software and dataset.

Conclusion

The training of cerebral aneurysm detection software using data collected from multiple sites is desirable to ensure the stable performance of the software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Giger ML, Chan HP, Boone J. Anniversary paper: history and status of CAD and quantitative image analysis: the role of medical physics and AAPM. Med Phys. 2008;35(12):5799–820.

    Article  Google Scholar 

  2. van Ginneken B, Schaefer-Prokop CM, Prokop M. Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology. 2011;261(3):719–32.

    Article  Google Scholar 

  3. Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Med Image Anal. 2017;42:60–88.

    Article  Google Scholar 

  4. Yasaka K, Akai H, Kunimatsu A, Kiryu S, Abe O. Deep learning with convolutional neural network in radiology. Jpn J Radiol. 2018;36(4):257–72.

    Article  Google Scholar 

  5. Hu Z, Tang J, Wang Z, Zhang K, Zhang L, Sun Q. Deep learning for image-based cancer detection and diagnosis—a survey. Pattern Recogn. 2018;83:134–49.

    Article  Google Scholar 

  6. Weikert T, Cyriac J, Yang S, Nesic I, Parmar V, Stieltjes B. A practical guide to artificial intelligence-based image analysis in radiology. Invest Radiol. 2020;55(1):1–7.

    Article  Google Scholar 

  7. Bluemke DA, Moy L, Bredella MA, et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers–from the Radiology editorial board. Radiology. 2020;294(3):487–9.

    Article  Google Scholar 

  8. Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018;286(3):800–9.

    Article  Google Scholar 

  9. Gibson E, Hu Y, Ghavami N, et al. Inter-site variability in prostate segmentation accuracy using deep learning. MICCAI 2018, LNCS. 2018;11073:506–14.

  10. Onofrey JA, Casetti-Dinescu DI, Lauritzen AD, et al. Generalizable multi-site training and testing of deep neural networks using image normalization. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, pp. 348–351.

  11. Remedios SW, Roy S, Bermudez C, et al. Distributed deep learning across multisite datasets for generalized CT hemorrhage segmentation. Med Phys. 2020;47(1):89–98.

    Article  Google Scholar 

  12. Therrien R, Doyle S. Role of training data variability on classifier performance and generalizability. Proc SPIE. 2018;10581:1058109.

    Google Scholar 

  13. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11):e1002683.

    Article  Google Scholar 

  14. Arimura H, Li Q, Korogi Y, et al. Automated computerized scheme for detection of unruptured intracranial aneurysms in three-dimensional magnetic resonance angiography. Acad Radiol. 2004;11(10):1093–104.

    Article  Google Scholar 

  15. Kobashi SK, Konda K, Hata Y. Computer-aided diagnosis of intracranial aneurysms in MRA images with case-based reasoning. IEICE Trans Inf Syst. 2006;E89-D(1):340–50.

    Article  Google Scholar 

  16. Yang X, Blezek DJ, Cheng LT, Ryan WJ, Kallmes DF, Erickson BJ. Computer-aided detection of intracranial aneurysms in MR angiography. J Digit Imaging. 2011;24(1):86–95.

    Article  Google Scholar 

  17. Nomura Y, Masutani Y, Miki S, et al. Performance improvement in computerized detection of cerebral aneurysms by retraining classifier using feedback data collected in routine reading environment. J Biomed Graph Comput. 2014;4(4):12–21.

    Google Scholar 

  18. Hanaoka S, Nomura Y, Nemoto M, et al. HoTPiG: a novel geometrical feature for vessel morphometry and its application to cerebral aneurysm detection. MICCAI 2015, LNCS. 2015;9350:103–10.

  19. Jin Z, Arimura H, Kakeda S, Yamashita F, Sasaki M, Korogi Y. An ellipsoid convex enhancement filter for detection of asymptomatic intracranial aneurysm candidates in CAD frameworks. Med Phys. 2016;43(2):951–60.

    Article  Google Scholar 

  20. Nakao T, Hanaoka S, Nomura Y, et al. Deep neural network-based computer-assisted detection of cerebral aneurysms in MR angiography. J Magn Reson Imaging. 2018;47(4):948–53.

    Article  Google Scholar 

  21. Sichtermann T, Faron A, Sijben R, Teichert N, Freiherr J, Wiesmann M. Deep learning-based detection of intracranial aneurysms in 3D TOF-MRA. Am J Neuroradiol. 2019;40(1):25–32.

    Article  CAS  Google Scholar 

  22. Ueda D, Yamamoto A, Nishimori M, et al. Deep learning for MR angiography: automated detection of cerebral aneurysms. Radiology. 2019;290(1):187–94.

    Article  Google Scholar 

  23. Hanaoka S, Nomura Y, Takenaga T, et al. HoTPiG: a novel graph-based 3-D image feature set and its applications to computer-assisted detection of cerebral aneurysms and lung nodules. Int J Comput Assist Radiol Surg. 2019;14(12):2095–107.

    Article  Google Scholar 

  24. Joo B, Ahn SS, Yoon PH, et al. A deep learning algorithm may automate intracranial aneurysm detection on MR angiography with high diagnostic performance. Eur Radiol. 2020;30(11):5785–93.

    Article  Google Scholar 

  25. Chen G, Wei X, Lei H, et al. Automated computer-assisted detection system for cerebral aneurysms in time-of-flight magnetic resonance angiography using fully convolutional network. Biomed Eng Online. 2020;19(1):38.

    Article  Google Scholar 

  26. Hou W, Mei S, Gui Q, et al. 1D CNN-based intracranial aneurysms detection in 3D TOF-MRA. Complexity. 2020;2020:7023754.

    Google Scholar 

  27. Nyul LG, Udupa JK. On standardizing the MR image intensity scale. Magn Reson Med. 1999;42(6):1072–81.

    Article  CAS  Google Scholar 

  28. Jiang X, Zhou Z, Ding X, Deng X, Zou L, Li B. Level set based hippocampus segmentation in MR images with improved initialization using region growing. Comput Math Methods Med. 2017;2017:5256346.

    Article  Google Scholar 

  29. Koenderink JJ. Solid shape. Cambridge, MA: MIT Press; 1990.

    Google Scholar 

  30. Dorai C, Jain AK. COSMOS-a representation scheme for 3D free-form objects. IEEE Trans Pattern Anal Mach Intell. 1997;19(10):1115–30.

    Article  Google Scholar 

  31. Nemoto M, Shimizu A, Hagihara Y, Kobatake H, Nawano S. Improvement of tumor detection performance in mammograms by feature selection from a large number of features and proposal of fast feature selection method. Syst Comput Jpn. 2006;37(12):56–68.

    Article  Google Scholar 

  32. Li Q, Sone S, Doi K. Selective enhancement filters for nodules, vessels, and airway walls in two- and three-dimensional CT scans. Med Phys. 2003;30(8):2040–51.

    Article  Google Scholar 

  33. Schapire RE, Freund Y, Bartlett P, Lee WS. Boosting the margin: a new explanation for the effectiveness of voting methods. Ann Statist. 1998;26(5):1651–86.

    Google Scholar 

  34. Kim K, Choi HI, Oh K. Object detection using ensemble of linear classifiers with fuzzy adaptive boosting. J Image Video Proc. 2017. https://doi.org/10.1186/s13640-017-0189-y.

    Article  Google Scholar 

  35. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 2004;31(8):2313–30.

    Article  Google Scholar 

  36. Metz CE. Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. J Am Coll Radiol. 2006;3(6):413–22.

    Article  Google Scholar 

  37. Setio AAA, Traverso A, de Bel T, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med Image Anal. 2017;42:1–13.

    Article  Google Scholar 

  38. Chang K, Balachandar N, Lam C, et al. Distributed deep learning networks among institutions for medical imaging. J Am Med Inform Assoc. 2018;25(8):945–54.

    Article  Google Scholar 

  39. Nomura Y, Miki S, Hayashi N, et al. Novel platform for development, training, and validation of computer-assisted detection/diagnosis software. Int J Comput Assist Radiol Surg. 2020;15(4):661–72.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Dr. Shogo Nishiyama (Fuchinobe General Hospital, Kanagawa, Japan) for providing Dataset C. The Department of Computational Radiology and Preventive Medicine, The University of Tokyo Hospital, is sponsored by HIMEDIC Inc. and Siemens Healthcare K.K. This study was supported by the Japan Agency for Medical Research and Development (AMED) under Grant Number 19lk1010038h0001.

Author information

Authors and Affiliations

  1. Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan

    Yukihiro Nomura, Takahiro Nakao, Naoto Hayashi, Takeharu Yoshikawa & Soichiro Miki

  2. Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan

    Shouhei Hanaoka, Takeyuki Watadani & Osamu Abe

Authors
  1. Yukihiro Nomura
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shouhei Hanaoka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Takahiro Nakao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Naoto Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Takeharu Yoshikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Soichiro Miki
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Takeyuki Watadani
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Osamu Abe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yukihiro Nomura.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest with regard to the present study.

Ethical approval

This study was approved by the ethical review boards of our institutions.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix: Methods used in CAD software

Shape index

The shape index [29, 30] represents a local feature of the surface. The value ranges from 0 to 1. The values of the voxels of a typical cerebral aneurysm range from 0.8 to 1.0, showing a peak or dome.

Selective enhancement filters derived from Hessian matrix

The Hessian matrix is a 3 × 3 square matrix composed of local second-order derivatives of the image. The selective enhancement filters for dot, line, and plane structures were derived from the eigenvalues of the Hessian matrix [32].

HoTPiG feature set

HoTPiG [18, 23] is a voxel-based feature set derived from a graph structure extracted from a binary image of the target structure (e.g., vessel system). The HoTPiG feature set is defined at each node (i.e., each voxel) in a given graph based on a 3D histogram of shortest path distances between the node of interest and each of its neighboring node pairs.

Mahalanobis distance ratio

The Mahalanobis distance is the distance between a point and a distribution. In the classification, first, the Mahalanobis distance dj (j = aneurysm, normal) between a feature vector at the target voxel and the multivariate distributions of two classes, which are estimated during training, are calculated. After that, the Mahalanobis distance ratio γ is calculated as follows:

$$\gamma = \frac{{d_{{normal}} }}{{d_{{aneurysm}} }}$$
(1)

If γ is above a certain threshold, the target voxel is classified as an aneurysm.

AdaBoost algorithm

AdaBoost is an adaptive algorithm to boost a sequence of classifiers [33, 34]. The AdaBoost algorithm chooses a good set of weak classifiers in rounds. On each round, it chooses the optimal classifier, which consists of feature values and the threshold, so that some misclassified data in the previous round would be correctly classified.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nomura, Y., Hanaoka, S., Nakao, T. et al. Performance changes due to differences in training data for cerebral aneurysm detection in head MR angiography images. Jpn J Radiol 39, 1039–1048 (2021). https://doi.org/10.1007/s11604-021-01153-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11604-021-01153-1

Keywords

Search

Navigation