Skip to main content
SpringerLink
Log in

Hypergraph attentional convolutional neural network for salient object detection

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Learning discriminative features and mining salient visual patterns play an important role in salient object detection (SOD) task. Existing SOD methods suffer from limited receptive field and insufficient cross-level feature mining. To this end, we propose a hypergraph attentional convolutional neural network for SOD task. Specifically, our method consists of (1) an attention based feature fusion module, which efficiently fuses lower layer as well as higher layer features, (2) a hypergraph-based long-range dependency encoder, which enhances the receptive field and global context for detection model, (3) a feature refinement layer, which highlights discriminative features and fuses attentional inputs, and (4) a dual iterative feature propagation decoder, which propagates features and upscale lower level feature maps to higher resolution. Both qualitative and quantitively experiments on public datasets verify the effectiveness of our proposed method. Compared with previous works, our model plays favorably against the state-of-the-arts methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Li, Z., Liu, G., Zhang, D., Xu, Y.: Robust single-object image segmentation based on salient transition region. Pattern Recogn. 52, 317–331 (2016)

    Article  Google Scholar 

  2. Zhi, X.-H., Shen, H.-B.: Saliency driven region-edge-based top-down level set evolution reveals the asynchronous focus in image segmentation. Pattern Recogn. 80, 241–255 (2018)

    Article  Google Scholar 

  3. Cai, Q., Liu, H., Qian, Y., Zhou, S., Duan, X., Yang, Y.-H.: Saliency-guided level set model for automatic object segmentation. Pattern Recogn. 93, 147–163 (2019)

    Article  Google Scholar 

  4. Gavrila, D., Philomin, V.: Real-time object detection for “smart” vehicles. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 87–93 (1998)

  5. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  6. Qian, Q., Wu, X.-J., Kittler, J., Xu, T.-Y.: Correlation tracking with implicitly extending search region. Vis. Comput. 37(5), 1029–1043 (2021)

    Article  Google Scholar 

  7. Wang, Y., Wei, X., Ding, L., Tang, X., Zhang, H.: A robust visual tracking method via local feature extraction and saliency detection. Vis. Comput. 36(4), 683–700 (2020)

    Article  Google Scholar 

  8. Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  9. Liu, J.-J., Hou, Q., Cheng, M.-M., Feng, J., Jiang, J.: A simple pooling based design for real-time salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  10. Zhao, X., Pang, Y., Zhang, L., Lu, H., Zhang, L.: Suppress and balance: A simple gated network for salient object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision—ECCV 2020, pp. 35–51. Springer International Publishing, Cham (2020)

    Chapter  Google Scholar 

  11. Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 212–221 (2017)

  12. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1741–1750 (2018)

  13. Mohammadi, S., Noori, M., Bahri, A., Ghofrani Majelan, S., Havaei, M.: Cagnet: content-aware guidance for salient object detection. Pattern Recogn. 103, 107303 (2020)

    Article  Google Scholar 

  14. Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  15. Feng, M., Lu, H., Ding, E.: Attentive feedback network for boundary-aware salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1623–1632 (2019)

  16. Liang, J., Zhou, J., Tong, L., Bai, X., Wang, B.: Material based salient object detection from hyperspectral images. Pattern Recogn. 76, 476–490 (2018)

    Article  Google Scholar 

  17. Huo, L., Jiao, L., Wang, S., Yang, S.: Object-level saliency detection with color attributes. Pattern Recogn. 49, 162–173 (2016)

    Article  Google Scholar 

  18. Li, G., Yu, Y.: Visual saliency detection based on multiscale deep cnn features. IEEE Trans. Image Process. 25(11), 5012–5024 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  19. Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multilevel convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

  20. Wang, T., Zhang, L., Wang, S., Lu, H., Yang, G., Ruan, X., Borji, A.: Detect globally, refine locally: A novel approach to saliency detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3127–3135 (2018)

  21. Zeng, Y., Zhuge, Y., Lu, H., Zhang, L., Qian, M., Yu, Y.: Multi-source weak supervision for saliency detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6067–6076 (2019)

  22. Zhang, L., Zhang, J., Lin, Z., Lu, H., He, Y.: Capsal: leveraging captioning to boost semantics for salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6017–6026 (2019)

  23. Chen, S., Tan, X., Wang, B., Lu, H., Hu, X., Fu, Y.: Reverse attention-based residual network for salient object detection. IEEE Trans. Image Process. 29, 3763–3776 (2020)

    Article  MATH  Google Scholar 

  24. Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 714–722 (2018)

  25. Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.: A stagewise refinement model for detecting salient objects in images. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4039–4048 (2017)

  26. Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902–3911 (2019)

  27. Liu, Y., Cheng, M.-M., Zhang, X.-Y., Nie, G.-Y., Wang, M.: Dna: deeply supervised nonlinear aggregation for salient object detection. IEEE Trans. TYCBetics (2021)

  28. Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, International Joint Conferences on Artificial Intelligence Organization, pp. 684–690 (2018)

  29. Pang, Y., Zhao, X., Zhang, L., Lu, H.: Multi-scale interactive network for salient object detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9410–9419 (2020)

  30. Luo, Z., Mishra, A., Achkar, A., Eichel, J.,Li, S., Jodoin, P.-M.: Non-local deep features for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  31. Liu, N., Han, J., Yang, M.-H.: Picanet: Pixel-wise contextual attention learning for accurate saliency detection. IEEE Trans. Image Process. 29, 6438–6451 (2020)

    Article  MATH  Google Scholar 

  32. Li, X., Song, D., Dong, Y.: Hierarchical feature fusion network for salient object detection. IEEE Trans. Image Process. 29, 9165–9175 (2020)

    Article  MATH  Google Scholar 

  33. Tu, Z., Ma, Y., Li, C., Tang, J., Luo, B.: Edge-guided non-local fully convolutional network for salient object detection. IEEE Trans. Circ. Syst. Video Technol. 31(2), 582–593 (2021)

    Article  Google Scholar 

  34. Wu, R., Feng, M., Guan, W., Wang, D., Lu, H., Ding, E.: A mutual learning method for salient object detection with intertwined multi-supervision. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8142–8151 (2019)

  35. Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. 35(11), 1683–1699 (2019)

    Article  Google Scholar 

  36. Li, L., Qing, L., Wang, Y., Su, J., Cheng, Y., Peng, Y.: Hf-srgr: a new hybrid feature-driven social relation graph reasoning model. Vis. Comput. 1432–2315 (2021)

  37. Harada, S., Akita, H., Tsubaki, M., Baba, Y., Takigawa, I., Yamanishi, Y., Kashima, H.: Dual graph convolutional neural network for predicting chemical networks. BMC Bioinform. 21(3), 94 (2020)

    Article  Google Scholar 

  38. Sanchez-Gonzalez, A., Heess, N., Springenberg, J.T., Merel, J., Riedmiller, M., Hadsell, R., Battaglia, P.: Graph networks as learnable physics engines for inference and control. In: Dy J., Krause A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 4470–4479 (2018)

  39. Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. In: Bengio Y., LeCun Y. (eds.) 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14–16 Apr 2014, Conference Track Proceedings (2014)

  40. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 Apr 2017, Conference Track Proceedings. https://openreview.net/ (2017)

  41. Li, G., Muller, M., Thabet, A., Ghanem, B.: Deepgcns: can gcns go as deep as cnns?. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

  42. Shen, Y., Li, H., Yi, S., Chen, D., Wang, X.: Person re-identification with deep similarity-guided graph neural network. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

  43. Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., Kalantidis, Y.: Graph-based global reasoning networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 433–442 (2019)

  44. Zhang, Q., Wang, S., Wang, X., Sun, Z., Kwong, S., Jiang, J.: Geometry auxiliary salient object detection for light fields via graph neural networks. IEEE Trans. Image Process. 30, 7578–7592 (2021)

    Article  MathSciNet  Google Scholar 

  45. Luo, A., Li, X., Yang, F., Jiao, Z., Cheng, H., Lyu, S.: Cascade graph neural networks for rgb-d salient object detection. In: In 16th European Conference on Computer Vision (ECCV) (2020)

  46. Zhou, D., Huang, J., Schölkopf, B.: Learning with hypergraphs: Clustering, classification, and embedding. In: Scholkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems. MIT Press, Cambridge (2007)

    Google Scholar 

  47. Yan, Y., Qin, J., Chen, J., Liu, L., Zhu, F., Tai, Y., Shao, L.: Learning multigranular hypergraphs for video-based person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  48. Feng, Y., You, H., Zhang, Z., Ji, R., Gao, Y.: Hypergraph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 3558–3565 (2019)

  49. Kim, E.-S., Kang, W.Y., On, K.W., Heo, Y.-J., Zhang, B.-T.: Hypergraph attention networks for multimodal learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14569–14578 (2020)

  50. Bai, S., Zhang, F., Torr, P.H.: Hypergraph convolution and hypergraph attention. Pattern Recogn. 110, 107637 (2021)

    Article  Google Scholar 

  51. Jiang, J., Wei, Y., Feng, Y., Cao, J., Gao, Y.: Dynamic hypergraph neural networks. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, International Joint Conferences on Artificial Intelligence Organization, pp. 2635–2641 (2019)

  52. Liang, Z., Chi, Z., Fu, H., Feng, D.D.: Salient object detection using content sensitive hypergraph representation and partitioning. Pattern Recogn. 45(11), 3886–3901 (2012)

    Article  Google Scholar 

  53. Li, X., Li, Y., Shen, C., Dick, A., Hengel, A.V.D.: Contextual hypergraph modeling for salient object detection. In: 2013 IEEE International Conference on Computer Vision, pp. 3328–3335 (2013)

  54. Zhang, J., Fang, S., Ehinger, K.A., Guo, W., Yang, W., Wei, H.: Probabilistic 1065 hypergraph optimization for salient object detection. In: Sun, Y., Lu, H., Zhang, L., Yang, J., Huang, H. (eds.) Intelligence Science and Big Data Engineering, pp. 368–378. Springer International Publishing, Cham (2017)

    Chapter  Google Scholar 

  55. Ma, M., Xia, C., Li, J.: Pyramidal feature shrinking for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, pp. 2311–2318 (2021)

  56. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  57. Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. In: Bengio Y., LeCun Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May, Conference Track Proceedings (2015)

  58. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  59. Park, J., Woo, S., Lee, J.-Y., Kweon, I.S.: Bam: bottleneck attention module. arXiv preprint arXiv:1807.06514

  60. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3796–3805 (2017)

  61. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)

  62. Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 280–287 (2014)

  63. Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5455–5463 (2015)

  64. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)

  65. Movahedi, V., Elder, J.H.: Design and perceptual validation of performance measures for salient object segmentation. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition—Workshops, pp. 49–56 (2010)

  66. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  67. Powers, D.M.W.: Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation, CoRR abs/2010.16061

  68. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)

  69. Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–740 (2012)

  70. Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.: Structure-measure: a new way to evaluate foreground maps. In: IEEE International Conference on Computer Vision (ICCV), pp. 4558–4567 (2017)

  71. Liu, J.-J., Hou, Q., Cheng, M.-M.: Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Trans. Image Process. 29, 8652–8667 (2020)

    Article  MATH  Google Scholar 

  72. Ehrig, M. and Euzenat, J: Relaxed precision and recall for ontology matching. In: Ashpole B., Ehrig M.,, Euzenat J., Stuckenschmidt H. (eds.) Integrating Ontologies ’05, Proceedings of the K-CAP 2005 Workshop on Integrating Ontologies, Banff, Canada, 2 Oct 2005, vol. 156 of CEUR Workshop Proceedings, CEUR-WS.org (2005)

  73. Ji, G.-P., Zhu, L., Zhuge, M., Fu, K.: Fast camouflaged object detection via edge-based reversible re-calibration network. Pattern Recogn. 123, 108414 (2022)

    Article  Google Scholar 

  74. Zhuge, M., Fan, D., Liu, N., Zhang, D., Xu, D., Shao, L.: Salient object detection via integrity learning. arXiv preprint arXiv:2101.07663

  75. Li, G., Yu, Y.: Deep contrast learning for salient object detection. arXiv preprint arXiv:1603.01976

  76. Liu, Y., Zhang, Q., Zhang, D. and Han, J.: Employing deep part-object relationships for salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1232–1241 (2019)

  77. Gupta, A.K., Seal, A., Khanna, P., Krejcar, O., Yazidi, A.: Awks: adaptive, weighted k-means-based superpixels for improved saliency detection. Pattern Anal. Appl. 24(2), 625–639 (2021)

    Article  Google Scholar 

  78. Gupta, A.K., Seal, A., Prasad, M., Khanna, P.: Salient object detection techniques in computer vision—a survey. Entropy 22(10), 1174 (2020)

    Article  MathSciNet  Google Scholar 

  79. Gupta, A.K., Seal, A., Khanna, P., Herrera-Viedma, E., Krejcar, O.: Almnet: adjacent layer driven multiscale features for salient object detection. IEEE Trans. Instrum. Meas. 70, 1–14 (2021)

    Google Scholar 

  80. Fan, D., Liu, J., Gao, S., Hou, Q., Borji, A., Chen, M.: Salient objects in clutter: bringing salient object detection to the foreground. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

  81. Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H., Yang, R.: Salient object detection in the deep learning era: an in-depth survey. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3051099

    Article  Google Scholar 

  82. Wu, Z., Su, L., Huang, Q.: Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7263–7272 (2019)

  83. Margolin, R., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2014)

  84. Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, International Joint Conferences on Artificial Intelligence Organization, pp. 698–704 (2018)

  85. Fu, K., Zhao, Q., Yu-Hua Gu, I., Yang, J.: Deepside: a general deep framework for salient object detection. Neurocomputing 356, 69–82 (2019)

    Article  Google Scholar 

  86. Liu, N., Zhang, N., Wan, K., Shao, L., Han, J.: Visual saliency transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4702–4712 (2021)

  87. Wei, J., Wang, S., Huang, Q.: F3net: fusion, feedback and focus for salient object detection. arXiv preprint arXiv:1911.11445

  88. Cheng, M.-M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.-M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  89. Wang, B., Chen, S., Wang, J., Hu, X.: Residual feature pyramid networks for salient object detection. Vis. Comput. 36(9), 1897–1908 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Automation, College of Information Science and Engineering, China University of Petroleum, Beijing, China

    Ze-yu Liu & Jian-wei Liu

Authors
  1. Ze-yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian-wei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian-wei Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Zy., Liu, Jw. Hypergraph attentional convolutional neural network for salient object detection. Vis Comput 39, 2881–2907 (2023). https://doi.org/10.1007/s00371-022-02499-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02499-x

Keywords

Search

Navigation