Abstract
We present a novel method for proposal free instance segmentation that can handle sophisticated object shapes which span large parts of an image and form dense object clusters with crossovers. Our method is based on predicting dense local shape descriptors, which we assemble to form instances. All instances are assembled simultaneously in one go. To our knowledge, our method is the first non-iterative method that yields instances that are composed of learnt shape patches. We evaluate our method on a diverse range of data domains, where it defines the new state of the art on four benchmarks, namely the ISBI 2012 EM segmentation benchmark, the BBBC010 C. elegans dataset, and 2d as well as 3d fluorescence microscopy data of cell nuclei. We show furthermore that our method also applies to 3d light microscopy data of Drosophila neurons, which exhibit extreme cases of complex shape clusters.
L. Mais and P. Hirsch—Contributed equally, listed in random order.
Code available: https://github.com/Kainmueller-Lab/PatchPerPix.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
BBBC010v1: C.elegans infection live/dead image set version 1 provided by Fred Ausubel.
- 2.
- 3.
- 4.
BBBC038v1: available from the Broad Bioimage Benchmark Collection [21].
References
Arganda-Carreras, I., et al.: Crowdsourcing the creation of image segmentation algorithms for connectomics. Front. Neuroanat. 9, 142 (2015)
Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Bai, M., Urtasun, R.: Deep watershed transform for instance segmentation. CoRR abs/1611.08303 (2016)
Chen, L., Strauch, M., Merhof, D.: Instance segmentation of biomedical images with an object-aware embedding learned with local constraints. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 451–459. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_50
Chen, X., Girshick, R.B., He, K., Dollár, P.: TensorMask: a foundation for dense object segmentation. CoRR abs/1903.12174 (2019). http://arxiv.org/abs/1903.12174
Dai, J., He, K., Li, Y., Ren, S., Sun, J.: Instance-sensitive fully convolutional networks. CoRR abs/1603.08678 (2016), http://arxiv.org/abs/1603.08678
De Brabandere, B., Neven, D., Van Gool, L.: Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551 (2017)
Funke, J., et al.: Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://doi.org/10.1109/TPAMI.2018.2835450
Gao, N., et al.: SSAP: single-shot instance segmentation with affinity pyramid. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 642–651 (2019)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2014, pp. 580–587. IEEE Computer Society, Washington, DC (2014). https://doi.org/10.1109/CVPR.2014.81
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN (2017). http://arxiv.org/abs/1703.06870. Comment: open source; appendix on more results
Hirsch, P., Kainmueller, D.: An auxiliary task for learning nuclei segmentation in 3d microscopy images. In: Medical Imaging with Deep Learning (MIDL), July 2020
Januszewski, M., Maitin-Shepard, J., Li, P., Kornfeld, J., Denk, W., Jain, V.: Flood-filling networks. CoRR abs/1611.00421 (2016). http://arxiv.org/abs/1611.00421
Jenett, A., et al.: A gal4-driver line resource for drosophila neurobiology. Cell reports 2(4), 991–1001 (2012). https://doi.org/10.1016/j.celrep.2012.09.011, https://www.ncbi.nlm.nih.gov/pubmed/23063364, 23063364[pmid]
Jetley, S., Sapienza, M., Golodetz, S., Torr, P.H.S.: Straight to shapes: real-time detection of encoded shapes. CoRR abs/1611.07932 (2016), http://arxiv.org/abs/1611.07932
Keuper, M., Levinkov, E., Bonneel, N., Lavoué, G., Brox, T., Andres, B.: Efficient decomposition of image and mesh graphs by lifted multicuts. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1751–1759 (2015)
Kulikov, V., Lempitsky, V.: Instance segmentation of biological images using harmonic embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
Lee, K., Lu, R., Luther, K., Seung, H.S.: Learning dense voxel embeddings for 3D neuron reconstruction (2019)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, Y., et al.: Affinity derivation and graph merge for instance segmentation. CoRR abs/1811.10870 (2018). http://arxiv.org/abs/1811.10870
Ljosa, V., Sokolnicki, K.L., Carpenter, A.E.: Annotated high-throughput microscopy image sets for validation. Nat. Methods 9, 637 EP (2012). https://doi.org/10.1038/nmeth.2083
Long, F., Peng, H., Liu, X., Kim, S.K., Myers, E.: A 3D digital atlas of C. elegans and its application to single-cell analyses. Nat. Methods 6(9), 667–672 (2009). https://doi.org/10.1038/nmeth.1366
Maitin-Shepard, J.B., Jain, V., Januszewski, M., Li, P., Abbeel, P.: Combinatorial energy learning for image segmentation. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 1966–1974. Curran Associates, Inc. (2016). http://papers.nips.cc/paper/6595-combinatorial-energy-learning-for-image-segmentation.pdf
Nern, A., Pfeiffer, B.D., Rubin, G.M.: Optimized tools for multicolor stochastic labeling reveal diverse stereotyped cell arrangements in the fly visual system. Proc. Natl. Acad. Sci. 112(22), E2967–E2976 (2015). https://doi.org/10.1073/pnas.1506763112, https://www.pnas.org/content/112/22/E2967
Novotny, D., Albanie, S., Larlus, D., Vedaldi, A.: Semi-convolutional operators for instance segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_6
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Schmidt, U., Weigert, M., Broaddus, C., Myers, G.: Cell detection with star-convex polygons. CoRR abs/1806.03535 (2018)
Wählby, C., et al.: An image analysis toolbox for high-throughput c. elegans assays. Nat. Methods 9(7), 714–716 (2012). https://doi.org/10.1038/nmeth.1984
Weigert, M., Schmidt, U., Haase, R., Sugawara, K., Myers, G.: Star-convex polyhedra for 3D object detection and segmentation in microscopy. arXiv:1908.03636 (2019)
Wolf, S., et al.: The mutex watershed: efficient, parameter-free image partitioning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 571–587. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_34
Wolf, S., Schott, L., Kothe, U., Hamprecht, F.: Learned watershed: end-to-end learning of seeded segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2011–2019 (2017)
Yurchenko, V., Lempitsky, V.S.: Parsing images of overlapping organisms with deep singling-out networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 4752–4760 (2017). https://doi.org/10.1109/CVPR.2017.505
Acknowledgments
We wish to thank Constantin Pape for his invaluable help in reproducing the training- and prediction setup from [30], Carolina Waehlby for help with the BBBC010 data, Stephan Saalfeld and Carsten Rother for inspiring discussions, the FlyLight Project Team (https://www.janelia.org/project-team/flylight) at Janelia Research Campus for providing unpublished data, and Claire Managan and Ramya Kappagantula (Janelia Project Technical Resources) for their conscientious manual neuron segmentations. P.H., L.M. and D.K. were funded by the Berlin Institute of Health and the Max Delbrueck Center for Molecular Medicine. P.H. was funded by HFSP grant RGP0021/2018-102. P.H., L.M. and D.K. were supported by the HHMI Janelia Visiting Scientist Program. VVD Viewer (https://github.com/takashi310/VVD_Viewer) is an open-source software funded by NIH grant R01-GM098151-01.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Mais, L., Hirsch, P., Kainmueller, D. (2020). PatchPerPix for Instance Segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12370. Springer, Cham. https://doi.org/10.1007/978-3-030-58595-2_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-58595-2_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58594-5
Online ISBN: 978-3-030-58595-2
eBook Packages: Computer ScienceComputer Science (R0)