Skip to main content
SpringerLink
Log in

Unsupervised Domain Adaptation for Semantic Segmentation of NIR Images Through Generative Latent Search

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12351))

Included in the following conference series:

Abstract

Segmentation of the pixels corresponding to human skin is an essential first step in multiple applications ranging from surveillance to heart-rate estimation from remote-photoplethysmography. However, the existing literature considers the problem only in the visible-range of the EM-spectrum which limits their utility in low or no light settings where the criticality of the application is higher. To alleviate this problem, we consider the problem of skin segmentation from the Near-infrared images. However, Deep learning based state-of-the-art segmentation techniques demands large amounts of labelled data that is unavailable for the current problem. Therefore we cast the skin segmentation problem as that of target-independent Unsupervised Domain Adaptation (UDA) where we use the data from the Red-channel of the visible-range to develop skin segmentation algorithm on NIR images. We propose a method for target-independent segmentation where the ‘nearest-clone’ of a target image in the source domain is searched and used as a proxy in the segmentation network trained only on the source domain. We prove the existence of ‘nearest-clone’ and propose a method to find it through an optimization algorithm over the latent space of a Deep generative model based on variational inference. We demonstrate the efficacy of the proposed method for NIR skin segmentation over the state-of-the-art UDA segmentation methods on the two newly created skin segmentation datasets in NIR domain despite not having access to the target NIR data. Additionally, we report state-of-the-art results for adaption from Synthia to Cityscapes which is a popular setting in Unsupervised Domain Adaptation for semantic segmentation. The code and datasets are available at https://github.com/ambekarsameer96/GLSS.

P. Pandey and A. K. Tyagi—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.gti.ssr.upm.es/data/MultiModalHandGesture_dataset.

References

  1. CyCADA: cycle consistent adversarial domain adaptation. In: International Conference on Machine Learning (ICML) (2018)

    Google Scholar 

  2. Al-Mohair, H.K., Saleh, J., Saundi, S.: Impact of color space on human skin color detection using an intelligent system. In: 1st WSEAS International Conference on Image Processing and Pattern Recognition (IPPR 2013), vol. 2 (2013)

    Google Scholar 

  3. Brancati, N., De Pietro, G., Frucci, M., Gallo, L.: Human skin detection through correlation rules between the YCb and YCr subspaces based on dynamic color clustering. Comput. Vis. Image Underst. 155, 33–42 (2017)

    Article  Google Scholar 

  4. Chang, W.L., Wang, H.P., Peng, W.H., Chiu, W.C.: All about structure: adapting structural information across domains for boosting semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1900–1909 (2019)

    Google Scholar 

  5. Chaurasia, A., Culurciello, E.: LinkNet: exploiting encoder representations for efficient semantic segmentation. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2017)

    Google Scholar 

  6. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49

    Chapter  Google Scholar 

  7. Chen, W., Wang, K., Jiang, H., Li, M.: Skin color modeling for face detection and segmentation: a review and a new approach. Multimedia Tools Appl. 75(2), 839–862 (2016). https://doi.org/10.1007/s11042-014-2328-0

    Article  Google Scholar 

  8. Chen, W.C., Wang, M.S.: Region-based and content adaptive skin detection in color images. Int. J. Pattern Recogn. Artif. Intell. 21(05), 831–853 (2007)

    Article  Google Scholar 

  9. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

    Google Scholar 

  10. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

    Google Scholar 

  11. Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13(1), 21–27 (1967)

    Article  MATH  Google Scholar 

  12. Dourado, A., Guth, F., de Campos, T.E., Weigang, L.: Domain adaptation for holistic skin detection. arXiv preprint arXiv:1903.06969 (2019)

  13. Erdem, C., Ulukaya, S., Karaali, A., Erdem, A.T.: Combining Haar feature and skin color based classifiers for face detection. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1497–1500. IEEE (2011)

    Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  15. He, Y., et al.: Semi-supervised skin detection by network with mutual guidance. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2111–2120 (2019)

    Google Scholar 

  16. Hoffman, J., Wang, D., Yu, F., Darrell, T.: FCNs in the wild: pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649 (2016)

  17. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)

    Google Scholar 

  18. Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)

    Article  Google Scholar 

  19. Huynh-Thu, Q., Meguro, M., Kaneko, M.: Skin-color-based image segmentation and its application in face detection. In: MVA, pp. 48–51 (2002)

    Google Scholar 

  20. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  21. Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002). https://doi.org/10.1023/A:1013200319198

    Article  MATH  Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  23. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)

  24. Kong, S.G., Heo, J., Abidi, B.R., Paik, J., Abidi, M.A.: Recent advances in visual and infrared face recognition-a review. Comput. Vis. Image Underst. 97(1), 103–135 (2005)

    Article  Google Scholar 

  25. Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection, vol. 2. IEEE (2003)

    Google Scholar 

  26. Li, Y., Yuan, L., Vasconcelos, N.: Bidirectional learning for domain adaptation of semantic segmentation. arXiv preprint arXiv:1904.10620 (2019)

  27. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

    Google Scholar 

  28. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

    Google Scholar 

  29. Liu, Q., Peng, G.: A robust skin color based face detection algorithm. In: 2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010), vol. 2, pp. 525–528. IEEE (2010)

    Google Scholar 

  30. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  31. Luo, Y., Zheng, L., Guan, T., Yu, J., Yang, Y.: Taking a closer look at domain shift: category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507–2516 (2019)

    Google Scholar 

  32. Mahmoodi, M.R.: High performance novel skin segmentation algorithm for images with complex background. arXiv preprint arXiv:1701.05588 (2017)

  33. Mahmoodi, M.R., Sayedi, S.M.: A comprehensive survey on human skin detection. Int. J. Image Graph. Sig. Process. 8(5), 1–35 (2016)

    Article  Google Scholar 

  34. Moallem, P., Mousavi, B.S., Monadjemi, S.A.: A novel fuzzy rule base system for pose independent faces detection. Appl. Soft Comput. 11(2), 1801–1810 (2011)

    Article  Google Scholar 

  35. Pan, Z., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003)

    Article  Google Scholar 

  36. Pandey, P., Prathosh, A.P., Kyatham, V., Mishra, D., Dastidar, T.R.: Target-independent domain adaptation for WBC classification using generative latent search. arXiv preprint arXiv:2005.05432 (2020)

  37. Pandey, P., Prathosh, A., Kohli, M., Pritchard, J.: Guided weak supervision for action recognition with scarce data to assess skills of children with autism. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 463–470 (2020)

    Google Scholar 

  38. Prathosh, A., Praveena, P., Mestha, L.K., Bharadwaj, S.: Estimation of respiratory pattern from video using selective ensemble aggregation. IEEE Trans. Sig. Process. 65(11), 2902–2916 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  39. Qiang-rong, J., Hua-lan, L.: Robust human face detection in complicated color images. In: 2010 2nd IEEE International Conference on Information Management and Engineering, pp. 218–221. IEEE (2010)

    Google Scholar 

  40. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  41. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3234–3243 (2016)

    Google Scholar 

  42. Seow, M.J., Valaparla, D., Asari, V.K.: Neural network based skin color model for face detection. In: 2003 Proceedings of the 32nd Applied Imagery Pattern Recognition Workshop, pp. 141–145. IEEE (2003)

    Google Scholar 

  43. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  44. Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019)

  45. Taqa, A.Y., Jalab, H.A.: Increasing the reliability of skin detectors. Sci. Res. Essays 5(17), 2480–2490 (2010)

    Google Scholar 

  46. Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7472–7481 (2018)

    Google Scholar 

  47. Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation. In: CVPR (2019)

    Google Scholar 

  48. Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: DADA: depth-aware domain adaptation in semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7364–7373 (2019)

    Google Scholar 

  49. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  50. Wu, Q., Cai, R., Fan, L., Ruan, C., Leng, G.: Skin detection using color processing mechanism inspired by the visual system (2012)

    Google Scholar 

  51. Zaidan, A., Ahmad, N.N., Karim, H.A., Larbani, M., Zaidan, B., Sali, A.: On the multi-agent learning neural and Bayesian methods in skin detector and pornography classifier: an automated anti-pornography system. Neurocomputing 131, 397–418 (2014)

    Article  Google Scholar 

  52. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Delhi, New Delhi, India

    Prashant Pandey, Aayush Kumar Tyagi, Sameer Ambekar & A. P. Prathosh

Authors
  1. Prashant Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aayush Kumar Tyagi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sameer Ambekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. P. Prathosh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prashant Pandey .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1500 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pandey, P., Tyagi, A.K., Ambekar, S., Prathosh, A.P. (2020). Unsupervised Domain Adaptation for Semantic Segmentation of NIR Images Through Generative Latent Search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12351. Springer, Cham. https://doi.org/10.1007/978-3-030-58539-6_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58539-6_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58538-9

  • Online ISBN: 978-3-030-58539-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation