Skip to main content
SpringerLink
Log in

SESF-Fuse: an unsupervised deep model for multi-focus image fusion

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Muti-focus image fusion is the extraction of focused regions from different images to create one all-in-focus fused image. The key point is that only objects within the depth-of-field have a sharp appearance in the photograph, while other objects are likely to be blurred. We propose an unsupervised deep learning model for multi-focus image fusion. We train an encoder–decoder network in an unsupervised manner to acquire deep features of input images. Then, we utilize spatial frequency, a gradient-based method to measure sharp variation from these deep features, to reflect activity levels. We apply some consistency verification methods to adjust the decision map and draw out the fused result. Our method analyzes sharp appearances in deep features instead of original images, which can be seen as another success story of unsupervised learning in image processing. Experimental results demonstrate that the proposed method achieves state-of-the-art fusion performance compared to 16 fusion methods in objective and subjective assessments, especially in gradient-based fusion metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Experimental data and code can be found at https://github.com/Keep-Passion/SESF-Fuse.

References

  1. Aslantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Syst Appl 37(12):8861–8870. https://doi.org/10.1016/j.eswa.2010.06.011

    Article  Google Scholar 

  2. Burt P, Adelson E (1983) The laplacian pyramid as a compact image code. IEEE Trans Commun 31(4):532–540. https://doi.org/10.1109/TCOM.1983.1095851

    Article  Google Scholar 

  3. Chen Y, Blum RS (2009) A new automated quality assessment algorithm for image fusion. Image Vis Comput 27(10):1421–1432. https://doi.org/10.1016/j.imavis.2007.12.002(Special Section: Computer Vision Methods for Ambient Intelligence)

    Article  Google Scholar 

  4. De I, Chanda B, Chattopadhyay B (2006) Enhancing effective depth-of-field by image fusion using mathematical morphology. Image Vision Comput 24(12):1278–1287. https://doi.org/10.1016/j.imavis.2006.04.005

    Article  Google Scholar 

  5. Facebook: Pytorch. https://pytorch.org (2019)

  6. Haghighat M, Aghagolzadeh A, Seyedarabi H (2011) Multi-focus image fusion for visual sensor networks in DCT domain. Comput Electr Eng 37(5):789–797

    Article  Google Scholar 

  7. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409. https://doi.org/10.1109/TPAMI.2012.213

    Article  Google Scholar 

  8. Huang J, Le Z, Ma Y, Mei X, Fan F (2020) A generative adversarial network with adaptive constraints for multi-focus image fusion. Neural Comput Appl. https://doi.org/10.1007/s00521-020-04863-1.pdf

    Article  Google Scholar 

  9. Huang G, Liu Z, Van Der Maaten L, Weinberger K.Q (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  10. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  11. Itzcovich E, Riani M, Sannita WG (2017) Stochastic resonance improves vision in the severely impaired. Sci Rep 7(1):1–8

    Article  Google Scholar 

  12. Jung H, Kim Y, Jang H, Ha N, Sohn K (2020) Unsupervised deep image fusion with structure tensor representations. IEEE Trans Image Process 29:3845–3858

    Article  Google Scholar 

  13. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations

  14. Lewis JJ, O'Callaghan RJ, Nikolov SG, Bull DR, Canagarajah N, (2007) Pixel- and region-based image fusion with complex wavelets. Inf Fusion 8(2):119–130. https://doi.org/10.1016/j.inffus.2005.09.006 Special Issue on Image Fusion: Advances in the State of the Art

    Article  Google Scholar 

  15. Li H, Wu X (2019) Densefuse: A fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623. https://doi.org/10.1109/TIP.2018.2887342

    Article  MathSciNet  Google Scholar 

  16. Li H, Manjunath B, Mitra S (1995) Multisensor image fusion using the wavelet transform. Graph Models Image Process 57(3):235–245. https://doi.org/10.1006/gmip.1995.1022

    Article  Google Scholar 

  17. Li S, Kwok JT, Wang Y (2001) Combination of images with diverse focuses using the spatial frequency. Inf Fusion 2(3):169–176. https://doi.org/10.1016/S1566-2535(01)00038-0

    Article  Google Scholar 

  18. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans Image Process 22(7):2864–2875. https://doi.org/10.1109/TIP.2013.2244222

    Article  Google Scholar 

  19. Li S, Kang X, Hu J, Yang B (2013) Image matting for fusion of multi-focus images in dynamic scenes. Inf Fusion 14(2):147–162. https://doi.org/10.1016/j.inffus.2011.07.001

    Article  Google Scholar 

  20. Li S, Kang X, Fang L, Hu J, Yin H (2017) Pixel-level image fusion: a survey of the state of the art. Inf Fusion 33:100–112. https://doi.org/10.1016/j.inffus.2016.05.004

    Article  Google Scholar 

  21. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision—ECCV 2014. Springer, Cham, pp 740–755

    Chapter  Google Scholar 

  22. Liu Y (2019) Image fusion. http://www.escience.cn/people/liuyu1/Codes.html

  23. Liu Z (2012) Image fusion metrics. https://github.com/zhengliu6699/imageFusionMetrics

  24. Liu Z, Blasch E, Xue Z, Zhao J, Laganiere R, Wu W (2012) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans Pattern Anal Mach Intell 34(1):94–109. https://doi.org/10.1109/TPAMI.2011.109

    Article  Google Scholar 

  25. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense sift. Inf Fusion 23:139–155. https://doi.org/10.1016/j.inffus.2014.05.004

    Article  Google Scholar 

  26. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Inf Fusion 36:191–207. https://doi.org/10.1016/j.inffus.2016.12.001

    Article  Google Scholar 

  27. Ma H, Liao Q, Zhang J, Liu S, Xue JH (2019) An \(\alpha \) matte boundary defocus model based cascaded network for multi-focus image fusion

  28. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Inf Fusion 25:72–84. https://doi.org/10.1016/j.inffus.2014.10.004

    Article  Google Scholar 

  29. Nencini F, Garzelli A, Baronti S, Alparone L (2007) Remote sensing image fusion using the curvelet transform. Inf Fusion 8(2):143–156. https://doi.org/10.1016/j.inffus.2006.02.001(Special Issue on Image Fusion: Advances in the State of the Art)

    Article  Google Scholar 

  30. Paul S, Sevcenco IS, Agathoklis P (2016) Multi-exposure and multi-focus image fusion in gradient domain. J Circuits Syst Comput 25:1650123

    Article  Google Scholar 

  31. Peng-wei Wang, Bo Liu (2008) A novel image fusion metric based on multi-scale analysis. In: 2008 9th international conference on signal processing, pp 965–968 . https://doi.org/10.1109/ICOSP.2008.4697288

  32. Prabhakar R (2017) Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: The IEEE international conference on computer vision (ICCV)

  33. Riani M, Simonotto E (1994) Stochastic resonance in the perceptual interpretation of ambiguous figures: a neural network model. Phys Rev Lett 72(19):3120

    Article  Google Scholar 

  34. Roy AG, Navab N, Wachinger C (2018) Concurrent spatial and channel squeeze and excitation in fully convolutional networks. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 421–429

  35. Savić S, Babić Z (2012) Multifocus image fusion based on empirical mode decomposition. In: 19th IEEE international conference on systems, signals and image processing (IWSSIP)

  36. Simonotto E, Riani M, Seife C, Roberts M, Twitty J, Moss F (1997) Visual perception of stochastic resonance. Phys Rev Lett 78(6):1186

    Article  Google Scholar 

  37. Spagnolo B, Valenti D, Guarcello C, Carollo A, Adorno DP, Spezia S, Pizzolato N, Di Paola B (2015) Noise-induced effects in nonlinear relaxation of condensed matter systems. Chaos Solitons Fractals 81:412–424

    Article  MathSciNet  Google Scholar 

  38. Spagnolo B, Guarcello C, Magazzù L, Carollo A, Persano Adorno D, Valenti D (2017) Nonlinear relaxation phenomena in metastable condensed matter systems. Entropy 19(1):20

    Article  Google Scholar 

  39. Stathaki T (2011) Image fusion: algorithms and applications. Elsevier, Amsterdam

    Google Scholar 

  40. Tang H, Xiao B, Li W, Wang G (2017) Pixel convolutional neural network for multi-focus image fusion. Inf Sci. https://doi.org/10.1016/j.ins.2017.12.043

    Article  Google Scholar 

  41. Toet A (1989) Image fusion by a ratio of low-pass pyramid. Pattern Recogn Lett 9(4):245–253. https://doi.org/10.1016/0167-8655(89)90003-2

    Article  MATH  Google Scholar 

  42. Valenti D, Magazzù L, Caldara P, Spagnolo B (2015) Stabilization of quantum metastable states by dissipation. Phys Rev B 91(23):235412

    Article  Google Scholar 

  43. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  44. Wikipedia: Focus stacking. https://github.com/cmcguinness/focusstack (2019)

  45. Xu K (2019) Image fusion. http://xudongkang.weebly.com/index.html

  46. Xu H, Fan F, Zhang H, Le Z, Huang J (2020) A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access 8:26316–26327

    Article  Google Scholar 

  47. Xu S, Wei X, Zhang C, Liu J, Zhang J (2020) Mffw: A new dataset for multi-focus image fusion. arXiv preprint arXiv:2002.04780

  48. Xydeas CS, Petrovic V (2000) Objective image fusion performance measure. Electron Lett 36(4):308–309. https://doi.org/10.1049/el:20000267

    Article  Google Scholar 

  49. Yang B, Li S (2010) Multifocus image fusion and restoration with sparse representation. IEEE Trans Instrum Meas 59(4):884–892. https://doi.org/10.1109/TIM.2009.2026612

    Article  Google Scholar 

  50. Zhang Q, Long Guo B (2009) Multifocus image fusion using the nonsubsampled contourlet transform. Signal Process 89(7):1334–1346. https://doi.org/10.1016/j.sigpro.2009.01.012

    Article  MATH  Google Scholar 

  51. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) IFCNN: a general image fusion framework based on convolutional neural network. Inf Fusion 54:99–118

    Article  Google Scholar 

  52. Zhou Z, Li S, Wang B (2014) Multi-scale weighted gradient-based fusion for multi-focus images. Inf Fusion 20:60–72. https://doi.org/10.1016/j.inffus.2013.11.005

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the support of the National Key Research and Development Program of China (No. 2016YFB0700500), National Science Foundation of China (No. 6170203, No. 61873299), Key Research Plan of Hainan Province (No. ZDYF2019009), Guangdong Province Key Area R and D Program (No. 2019B010940001), Scientific and Technological Innovation Foundation of Shunde Graduate School, USTB (No. BK19BE030), and Fundamental Research Funds for the University of Science and Technology Beijing (No. FRF-BD-19-012A, No. FRF-TP-19-043A2). The computing work was supported by USTB MatCom of Beijing Advanced Innovation Center for Materials Genome Engineering.

Author information

Authors and Affiliations

  1. Beijing Advanced Innovation Center for Materials Genome Engineering, University of Science and Technology Beijing, Beijing, China

    Boyuan Ma, Yu Zhu, Xiang Yin, Xiaojuan Ban & Haiyou Huang

  2. Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing, China

    Boyuan Ma, Yu Zhu, Xiang Yin & Xiaojuan Ban

  3. Institute of Artificial Intelligence, School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China

    Boyuan Ma, Yu Zhu, Xiang Yin & Xiaojuan Ban

  4. Institute for Advanced Materials and Technology, University of Science and Technology Beijing, Beijing, China

    Haiyou Huang

  5. Faculty of Engineering Sciences, University of Burundi, Bujumbura, Burundi

    Michele Mukeshimana

Authors
  1. Boyuan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaojuan Ban
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haiyou Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michele Mukeshimana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xiaojuan Ban or Haiyou Huang.

Ethics declarations

Conflicts of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, B., Zhu, Y., Yin, X. et al. SESF-Fuse: an unsupervised deep model for multi-focus image fusion. Neural Comput & Applic 33, 5793–5804 (2021). https://doi.org/10.1007/s00521-020-05358-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05358-9

Keywords

Search

Navigation