Abstract
This paper presents an overview of the first HEad and neCK TumOR (HECKTOR) challenge, organized as a satellite event of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2020. The task of the challenge is the automatic segmentation of head and neck primary Gross Tumor Volume in FDG-PET/CT images, focusing on the oropharynx region. The data were collected from five centers for a total of 254 images, split into 201 training and 53 testing cases. The interest in the task was shown by the important participation with 64 teams registered and 18 team submissions. The best method obtained a Dice Similarity Coefficient (DSC) of 0.7591, showing a large improvement over our proposed baseline method with a DSC of 0.6610 as well as inter-observer DSC agreement reported in the literature (0.69).
V. Andrearczyk and V. Oreiller—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
www.aicrowd.com/challenges/hecktor, as of October 2020.
- 2.
https://portal.fli-iam.irisa.fr/petseg-challenge/overview#_ftn1, as of October 2020.
- 3.
The target cohort refers to the subjects from whom the data would be acquired in the final biomedical application. It is mentioned for additional information as suggested in BIAS, although all data provided for the challenge are part of the challenge cohort.
- 4.
The challenge cohort refers to the subjects from whom the challenge data were acquired.
- 5.
For simplicity, these centers were renamed CHGJ and CHMR during the challenge.
- 6.
github.com/voreille/hecktor, as of October 2020.
- 7.
github.com/voreille/hecktor/tree/master/src/evaluation, as of October 2020.
- 8.
These values are reported only to give an idea of inter-observer variability on a similar task reported in the literature. The datasets are different and the comparison is limited. In future work, we will compute the inter-observer agreement on the challenge data.
- 9.
- 10.
- 11.
- 12.
References
Andrearczyk, V., Oreiller, V., Depeursinge, A.: Oropharynx detection in PET-CT for tumor segmentation. In: Irish Machine Vision and Image Processing (2020)
Andrearczyk, V., et al.: Automatic segmentation of head and neck tumors and nodal metastases in PET-CT scans. In: International Conference on Medical Imaging with Deep Learning (MIDL) (2020)
Bogowicz, M., Tanadini-Lang, S., Guckenberger, M., Riesterer, O.: Combined CT radiomics of primary tumor and metastatic lymph nodes improves prediction of loco-regional control in head and neck cancer. Sci. Rep. 9(1), 1–7 (2019)
Chen, H., Chen, H., Wang, L.: Iteratively refine the segmentation of head and neck tumor in FDG-PET and CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 53–58. Springer, Cham (2021)
Foster, B., Bagci, U., Mansoor, A., Xu, Z., Mollura, D.J.: A review on segmentation of positron emission tomography images. Comput. Biol. Med. 50, 76–96 (2014)
Ghimire, K., Chen, Q., Feng, X.: Patch-based 3D UNet for head and neck tumor segmentation with an ensemble of conventional and dilated convolutions. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 78–84. Springer, Cham (2021)
Gudi, S., et al.: Interobserver variability in the delineation of gross tumour volume and specified organs-at-risk during IMRT for head and neck cancers and the impact of FDG-PET/CT on such variability at the primary site. J. Med. Imaging Radiat. Sci. 48(2), 184–192 (2017)
Hatt, M., et al.: The first MICCAI challenge on PET tumor segmentation. Med. Image Anal. 44, 177–195 (2018)
Hatt, M., et al.: Classification and evaluation strategies of auto-segmentation approaches for PET: report of AAPM task group no. 211. Med. Phys. 44(6), e1–e42 (2017)
Iantsen, A., Visvikis, D., Hatt, M.: Squeeze-and-excitation normalization for automated delineation of head and neck primary tumors in combined PET and CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 37–43. Springer, Cham (2021)
Kumar, A., Fulham, M., Feng, D., Kim, J.: Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans. Med. Imaging 39(1), 204–217 (2019)
Li, L., Zhao, X., Lu, W., Tan, S.: Deep learning for variational multimodality tumor segmentation in PET/CT. Neurocomputing 392, 277–295 (2019)
Ma, J., Yang, X.: Combining CNN and hybrid active contours for head and neck tumor segmentation in CT and PET Images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 59–64. Springer, Cham (2021)
Maier-Hein, L., et al.: Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun. 9(1), 1–13 (2018)
Maier-Hein, L., et al.: BIAS: transparent reporting of biomedical image analysis challenges. Med. Image Anal. 66, 101796 (2020)
Moe, Y.M., et al.: Deep learning for automatic tumour segmentation in PET/CT images of patients with head and neck cancers. In: Medical Imaging with Deep Learning (2019)
Naser, M.A., van Dijk, L.V., He, R., Wahid, K.A., Fuller, C.D.: Tumor segmentation in patients with head and neck cancers using deep learning based-on multi-modality PET/CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 85–98. Springer, Cham (2021)
Rao, C., et al.: Oropharyngeal Tumour Segmentation using Ensemble 3D PET-CT Fusion Networks for the HECKTOR Challenge. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 65–77. Springer, Cham (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Vallieres, M., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 7(1), 1–14 (2017)
Xie, J., Peng, Y.: The head and neck tumor segmentation using nnU-Net with spatial and channel ‘squeeze & excitation’ blocks. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 28–36. Springer, Cham (2021)
Xu, L., et al.: Automated whole-body bone lesion detection for multiple myeloma on 68Ga-pentixafor PET/CT imaging using deep learning methods. Contrast Media Mol. Imaging 2018, 11 (2018). https://doi.org/10.1155/2018/2391925
Yousefirizi, F., Rahmim, A.: GAN-based bi-modal segmentation using mumford-shah loss: Application to head and neck tumors in PET-CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 99–108. Springer, Cham (2021)
Yuan, Y.: Automatic head and neck tumor segmentation in PET/CT with scale attention network. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 44–52. Springer, Cham (2021)
Zhao, X., Li, L., Lu, W., Tan, S.: Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys. Med. Biol. 64(1), 015011 (2018)
Zhong, Z., et al.: 3D fully convolutional networks for co-segmentation of tumors on PET-CT images. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 228–231. IEEE (2018)
Zhu, S., Dai, Z., Ning, W.: Two-stage approach for segmenting gross tumor volume in head and neck cancer with CT and PET imaging. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 22–27. Springer, Cham (2021)
Acknowledgments
The organizers thank all the teams for their participation and valuable work. This challenge and the winner prize are sponsored by Siemens Healthineers Switzerland. This work was also partially supported by the Swiss National Science Foundation (SNSF, grant 205320_179069) and the Swiss Personalized Health Network (SPHN, via the IMAGINE and QA4IQI projects).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix 1: Challenge Information
In this appendix, we list important information about the challenge as suggested in the BIAS guidelines [15].
Challenge Name
HEad and neCK TumOR segmentation challenge (HECKTOR) 2020.
Organizing Team
(Authors of this paper) Vincent Andrearczyk, Valentin Oreiller, Martin Vallières, Joel Castelli, Mario Jreige, John O. Prior and Adrien Depeursinge.
Life Cycle Type
A fixed submission deadline was set for the challenge results. Open online leaderboard following the conference.
Challenge Venue and Platform
The challenge is associated with MICCAI 2020. Information on the challenge is available on the website, together with the link to download the data, the submission platform and the leaderboardFootnote 10.
Participation Policies
-
(a)
Algorithms producing fully-automatic segmentation of the test cases were allowed.
-
(b)
The data used to train algorithms was not restricted. If using external data (private or public), participants were asked to also report results using only the HECKTOR data.
-
(c)
Members of the organizers’ institutes could participate in the challenge but were not eligible for awards.
-
(d)
The award was 500 euros, sponsored by Siemens Healthineers Switzerland.
-
(e)
Policy for results announcement: The results were made available on the AIcrowd leaderboard and the best three results were announced publicly. Once participants submitted their results on the test set to the challenge organizers via the challenge website, they were considered fully vested in the challenge, so that their performance results (without identifying the participant unless permission is granted) became part of any presentations, publications, or subsequent analyses derived from the challenge at the discretion of the organizers.
-
(f)
Publication policy: This overview paper was written by the organizing team’s members. The participating teams were encouraged to submit a paper describing their method. The participants can publish their results separately elsewhere when citing the overview paper, and (if so) no embargo will be applied.
Submission Method
Submission instructions are available on the websiteFootnote 11 and are reported in the following. Results should be provided as a single binary mask (1 in the predicted GTVt) .nii.gz file per patient in the CT original resolution and cropped using the provided bounding boxes. The participants should pay attention to saving NIfTI volumes with the correct pixel spacing and origin with respect to the original reference frame. The .nii files should be named [PatientID].nii.gz, matching the patients’ file names, e.g.. CHUV001.nii.gz and placed in a folder. This folder should be zipped before submission. If results were submitted without cropping and/or resampling, we employed nearest-neighbor interpolation given that the coordinate system is provided. Participants were allowed five valid submissions. The best result was reported for each team.
Challenge Schedule
The schedule of the challenge, including modifications, is reported in the following.
-
the release date of the training cases:
June 01 2020June 10 2020 -
the release date of the test cases: Aug. 01 2020
-
the results submission date(s): opens Sept. 01 2020 closes Sept. 10 2020
-
paper submission deadline:
Sept. 18 2020Sept. 15 2020 -
the release date of the results: Sept. 15 2020
-
associated workshop days: Oct. 04 2020, 9:00-13:00 UTC
Ethics Approval
Training dataset: The ethics approval was granted by the Research Ethics Committee of McGill University Health Center (Protocol Number: MM-JGH-CR15-50). Test dataset: The ethics approval was obtained from the Commission cantonale (VD) d’éthique de la recherche sur l’étre humain (CER-VD) with protocol number: 2018-01513.
Data Usage Agreement
The participants had to fill out and sign an end-user-agreement in order to be granted access to the data. The form can be found under the Resources tab of the HECKTOR website.
Code Availability
The evaluation software was made available on our github pageFootnote 12. The participating teams decided whether they wanted to disclose their code (they were encouraged to do so).
Conflict of Interest
No conflict of interest applies. Fundings are specified in the acknowledgments. Only the organizers had access to the test cases ground truth contours.
Author Contributions
Vincent Andrearczyk
Design of the task and of the challenge, writing of the proposal, development of baseline algorithms, development of the AIcrowd website, writing of the overview paper, organization of the challenge event, organization of the submission and reviewing process of the participants’ papers.
Valentin Oreiller
Design of the task and of the challenge, writing of the proposal, development of the AIcrowd website, development of the evaluation code, writing of the overview paper, organization of the challenge event, organization of the submission and reviewing process of the papers.
Mario Jreige
Design of the task and of the challenge, quality control/annotations, annotations for inter-annotator agreement, revision of the paper and accepted the last version of the submitted paper.
Martin Vallières
Design of the task and of the challenge, provided the initial data and annotations for the training set [20], revision of the paper and accepted the last version of the submitted paper.
Joel Castelli
Design of the task and of the challenge, annotations for inter-annotator agreement.
Hesham Elhalawani
Design of the task and of the challenge, annotations for inter-annotator agreement.
Sarah Boughdad
Design of the task and of the challenge, annotations for inter-annotator agreement.
John O. Prior
Design of the task and of the challenge, revision of the paper and accepted the last version of the submitted paper.
Adrien Depeursinge
Design of the task and of the challenge, writing of the proposal, writing of the overview paper, organization of the challenge event.
Appendix 2: Image Acquisition Details
HGJ: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery ST, GE Healthcare) within 37 days before treatment (median: 14 days). For the PET portion of the FDG-PET/CT scan, a median of 584 MBq (range: 368–715) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 300 s (range: 180–420) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a span (axial mash) of 5. The FDG-PET slice thickness resolution was 3.27 mm for all patients and the median in-plane resolution was 3.52 \(\times \) 3.52 mm\(^2\) (range: 3.52–4.69). For the CT portion of the FDG-PET/CT scan, an energy of 140 kVp with an exposure of 12 mAs was used. The CT slice thickness resolution was 3.75 mm and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) for all patients.
CHUS: All 102 eligible patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (GeminiGXL 16, Philips) within 54 days before treatment (median: 19 days). For the PET portion of the FDG-PET/CT scan, a median of 325 MBq (range: 165–517) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 150 s (range: 120–151) per bed position. Attenuation corrected images were reconstructed using a LOR-RAMLA iterative algorithm. The FDG-PET slice thickness resolution was 4 mm and the median in-plane resolution was 4 \(\times \) 4 mm\(^2\) for all patients. For the CT portion of the FDG-PET/CT scan, a median energy of 140 kVp (range: 12–140) with a median exposure of 210 mAs (range: 43–250) was used. The median CT slice thickness resolution was 3 mm (range: 2–5) and the median in-plane resolution was 1.17 \(\times \) 1.17 mm\(^2\) (range: 0.68–1.17).
HMR: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery STE, GE Healthcare) within 60 days before treatment (median: 34 days). For the PET portion of the FDG-PET/CT scan, a median of 475 MBq (range: 227–859) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 360 s (range: 120–360) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a median span (axial mash) of 5 (range: 3–5). The FDG-PET slice thickness resolution was 3.27 mm for all patients and the median in-plane resolution was 3.52 \(\times \) 3.52 mm\(^2\) (range: 3.52–5.47). For the CT portion of the FDG-PET/CT scan, a median energy of 140 kVp (range: 120–140) with a median exposure of 11 mAs (range: 5–16) was used. The CT slice thickness resolution was 3.75 mm for all patients and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) (range: 0.98–1.37).
CHUM: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery STE, GE Healthcare) within 66 days before treatment (median: 12 days). For the PET portion of the FDG-PET/CT scan, a median of 315 MBq (range: 199–3182) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 300 s (range: 120–420) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a medianspan (axial mash) of 3 (range: 3–5). The median FDG-PET slice thickness resolution was 4 mm (range: 3.27–4) and the median in-plane resolution was 4 \(\times \) 4 mm\(^2\) (range: 3.52–5.47). For the CT portion of the FDG-PET/CT scan, a median energy of 120 kVp (range: 120–140) with a median exposure of 350 mAs (range: 5–350) was used. The median CT slice thickness resolution was 1.5 mm (range: 1.5–3.75) and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) (range: 0.98–1.37). All patients received their FDG-PET/CT scan dedicated to the head and neck area right before their planning CT scan, in the same position with the immobilization device.
CHUV (test): All patients underwent FDG PET/CT for staging before treatment. Blood glucose levels were checked before the injection of (18F)-FDG. After a 60-min uptake period of rest, patients were imaged with the Discovery D690 TOF PET/CT (General Electric Healthcare, Milwaukee, WI, USA). First, a CT (120 kV, 80 mA, 0.8-s rotation time, slice thickness 3.75 mm) was performed from the base of the skull to the mid-thigh. PET scanning was performed immediately after acquisition of the CT. Images were acquired from the base of the skull to the mid-thigh (2 min/bed position). PET images were reconstructed after time-of-flight and point-spread-function recovery corrections by using an ordered-subset expectation maximization iterative reconstruction (OSEM) (two iterations, 28 subsets) and an iterative fully 3D (Discovery ST). CT data were used for attenuation calculation.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Andrearczyk, V. et al. (2021). Overview of the HECKTOR Challenge at MICCAI 2020: Automatic Head and Neck Tumor Segmentation in PET/CT. In: Andrearczyk, V., Oreiller, V., Depeursinge, A. (eds) Head and Neck Tumor Segmentation. HECKTOR 2020. Lecture Notes in Computer Science(), vol 12603. Springer, Cham. https://doi.org/10.1007/978-3-030-67194-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-67194-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67193-8
Online ISBN: 978-3-030-67194-5
eBook Packages: Computer ScienceComputer Science (R0)