Skip to main content
SpringerLink
Log in

Hierarchical Cellular Automata for Visual Saliency

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Saliency detection, finding the most important parts of an image, has become increasingly popular in computer vision. In this paper, we introduce Hierarchical Cellular Automata (HCA)—a temporally evolving model to intelligently detect salient objects. HCA consists of two main components: Single-layer Cellular Automata (SCA) and Cuboid Cellular Automata (CCA). As an unsupervised propagation mechanism, Single-layer Cellular Automata can exploit the intrinsic relevance of similar regions through interactions with neighbors. Low-level image features as well as high-level semantic information extracted from deep neural networks are incorporated into the SCA to measure the correlation between different image patches. With these hierarchical deep features, an impact factor matrix and a coherence matrix are constructed to balance the influences on each cell’s next state. The saliency values of all cells are iteratively updated according to a well-defined update rule. Furthermore, we propose CCA to integrate multiple saliency maps generated by SCA at different scales in a Bayesian framework. Therefore, single-layer propagation and multi-scale integration are jointly modeled in our unified HCA. Surprisingly, we find that the SCA can improve all existing methods that we applied it to, resulting in a similar precision level regardless of the original results. The CCA can act as an efficient pixel-wise aggregation algorithm that can integrate state-of-the-art methods, resulting in even better results. Extensive experiments on four challenging datasets demonstrate that the proposed algorithm outperforms state-of-the-art conventional methods and is competitive with deep learning based approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  • Achanta, R., Hemami, S., Estrada, F., & Susstrunk, S. (2009). Frequency-tuned salient region detection. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1597–1604).

  • Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunkm, S. (2010). Slic superpixels. Technical report.

  • Alexe, B., Deselaers, T & Ferrari, V. (2010). What is an object? In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 73–80).

  • Batty, M. (2007). Cities and complexity: Understanding cities with cellular automata, agent-based models, and fractals. Cambridge: The MIT press.

    Google Scholar 

  • Borji, A., Cheng, M. M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.

    Article  MathSciNet  Google Scholar 

  • Bruce, N., & Tsotsos, J. (2005). Saliency based on information maximization. In Advances in neural information processing systems (pp. 155–162).

  • Cheng, M., Mitra, N. J., Huang, X., Torr, P. H., & Hu, S. (2015). Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 569–582.

    Article  Google Scholar 

  • Cheng, M. M., Warrell, J., Lin, W. Y., Zheng, S., Vineet, V., & Crook, N. (2013). Efficient salient region detection with soft image abstraction. In Proceedings of the IEEE international conference on computer vision (pp. 1529–1536).

  • Chopard, B., & Droz, M. (2005). Cellular automata modeling of physical systems (Vol. 6). Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Cowburn, R., & Welland, M. (2000). Room temperature magnetic quantum cellular automata. Science, 287(5457), 1466.

    Article  Google Scholar 

  • de Almeida, C. M., Batty, M., Monteiro, A. M. V., Câmara, G., Soares-Filho, B. S., Cerqueira, G. C., et al. (2003). Stochastic cellular automata modeling of urban land use dynamics: Empirical development and estimation. Computers, Environment and Urban Systems, 27(5), 481–509.

    Article  Google Scholar 

  • Ding, Y., Xiao, J., & Yu, J. (2011). Importance filtering for image retargeting. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 89–96).

  • Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of international conference on machine learning (pp. 647–655).

  • Everingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2), 303–338.

    Article  Google Scholar 

  • Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013). Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1915–1929.

    Article  Google Scholar 

  • Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580–587).

  • Goferman, S., Zelnik-manor, L., & Tal, A. (2010). Context-aware saliency detection. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Gong, C., Tao, D., Liu, W., Maybank, S. J., Fang, M., Fu, K., & Yang, J. (2015). Saliency propagation from simple to difficult. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2531–2539).

  • Hariharan, B., Arbeláez, P., Girshick, R., & Malik, J. (2014). Simultaneous detection and segmentation. In Proceedings of European conference on computer vision (pp. 297–312). Springer.

  • Hariharan, B., Arbeláez, P., Girshick, R., & Malik, J. (2015). Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 447–456).

  • Hou, X., & Zhang, L. (2007). Saliency detection: A spectral residual approach. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1–8)

  • Itti, L., & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3), 194–203.

    Article  Google Scholar 

  • Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11, 1254–1259.

    Article  Google Scholar 

  • Jiang, B., Zhang, L., Lu, H., Yang, C., & Yang, MH. (2013a). Saliency detection via absorbing markov chain. In Proceedings of the IEEE international conference on computer vision (pp. 1665–1672).

  • Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. In Proceedings of British machine vision conference (Vol. 6, p. 9).

  • Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., & Li, S. (2013b). Salient object detection: A discriminative regional feature integration approach. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2083–2090).

  • Jiang, P., Ling, H., Yu, J., & Peng, J. (2013c). Salient region detection by ufo: Uniqueness, focusness and objectness. In Proceedings of the IEEE international conference on computer vision (pp. 1976–1983).

  • Jiang, Z., & Davis, L. (2013). Submodular salient region detection. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2043–2050).

  • Kanan, C., & Cottrell, G. W. (2010). Robust classification of objects, faces, and flowers using natural image statistics. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2472–2479).

  • Kim, J., & Pavlovic, V. (2016). A shape-based approach for salient object detection using deep learning. In Proceedings of European conference on computer vision (pp. 455–470).

  • Kim, J., Han, D., Tai, Y. W., & Kim, J. (2014). Salient region detection via high-dimensional color transform. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 883–890).

  • Klein, D. A., & Frintrop, S. (2011). Center-surround divergence of feature statistics for salient object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2214–2219).

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).

  • Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 5455–5463).

  • Li, N., Ye, J., Ji, Y., Ling, H., & Yu, J. (2014a). Saliency detection on light field. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Li, X., Lu, H., Zhang, L., Ruan, X., & Yang, M. H. (2013). Saliency detection via dense and sparse reconstruction. In Proceedings of the IEEE international conference on computer vision (pp. 2976–2983).

  • Li, X., Zhao, L., Wei, L., Yang, M. H., Wu, F., Zhuang, Y., et al. (2016). Deepsaliency: Multi-task deep neural network model for salient object detection. IEEE Transactions on Image Processing, 25(8), 3919–3930.

    Article  MathSciNet  Google Scholar 

  • Li, Y., Hou, X., Koch, C., Rehg, J., & Yuille, A. (2014b). The secrets of salient object segmentation. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 280–287).

  • Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., et al. (2011). Learning to detect a salient object. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(2), 353–367.

    Article  Google Scholar 

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 3431–3440).

  • Ma, C., Huang, JB., Yang, X., & Yang, M. H. (2015). Hierarchical convolutional features for visual tracking. In Proceedings of the IEEE international conference on computer vision (pp. 3074–3082).

  • Mahadevan, V., & Vasconcelos, N. (2009). Saliency-based discriminant tracking. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1007–1013).

  • Marchesotti, L., Cifarelli, C., & Csurka, G. (2009). A framework for visual saliency detection with applications to image thumbnailing. In Proceedings of the IEEE international conference on computer vision (pp. 2232–2239).

  • Martins, A. C. (2008). Continuous opinions and discrete actions in opinion dynamics problems. International Journal of Modern Physics C, 19(04), 617–624.

    Article  MATH  Google Scholar 

  • Ng, A. Y., Jordan, M. I., Weiss, Y., et al. (2002). On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 2, 849–856.

    Google Scholar 

  • Otsu, N. (1975). A threshold selection method from gray-level histograms. Automatica, 11(285–296), 23–27.

    Google Scholar 

  • Pan, Q., Qin, Y., Xu, Y., Tong, M., & He, M. (2016). Opinion evolution in open community. International Journal of Modern Physics C, 28, 1750003.

    Article  MathSciNet  Google Scholar 

  • Perazzi, F., Krähenbühl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 733–740). IEEE.

  • Pinheiro, P. H., & Collobert, R. (2014). Recurrent convolutional neural networks for scene labeling. In Proceedings of international conference on machine learning (pp. 82–90).

  • Qin, Y., Lu, H., Xu, Y., & Wang, H. (2015). Saliency detection via cellular automata. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Rahtu, E., Kannala, J., Salo, M., & Heikkilä, J. (2010). Segmenting salient objects from images and videos. In Proceedings of European conference on computer vision (pp. 366–379).

  • Reinagel, P., & Zador, A. M. (1999). Natural scene statistics at the centre of gaze. Network: Computation in Neural Systems, 10(4), 341–350.

    Article  MATH  Google Scholar 

  • Scharfenberger, C., Wong, A., Fergani, K., Zelek ,J. S., & Clausi, D. A. (2013). Statistical textural distinctiveness for salient region detection in natural images. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 979–986).

  • Shen, X., & Wu, Y. (2012). A unified approach to salient object detection via low rank matrix recovery. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 853–860).

  • Shi, K., Wang, K., Lu, J., & Lin, L. (2013). Pisa: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2115–2122).

  • Siagian, C., & Itti, L. (2007). Rapid biologically-inspired scene classification using features shared with visual attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(2), 300–312.

    Article  Google Scholar 

  • Smith, A. R. (1972). Real-time language recognition by one-dimensional cellular automata. Journal of Computer and System Sciences, 6(3), 233–253.

    Article  MathSciNet  MATH  Google Scholar 

  • Sun, J., Ling, H. (2011). Scale and object aware image retargeting for thumbnail browsing. In Proceedings of the IEEE international conference on computer vision (pp. 1511–1518).

  • Sun, J., Lu, H., Li, S. (2012). Saliency detection based on integration of boundary and soft-segmentation. In Proceedings of IEEE international conference on image processing (pp. 1085–1088).

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9).

  • Szegedy, C., Toshev, A., & Erhan, D. (2013). Deep neural networks for object detection. In Advances in neural information processing systems (pp. 2553–2561).

  • Tong, N., Lu, H., Ruan, X., & Yang, M. H. (2015a). Salient object detection via bootstrap learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1884–1892).

  • Tong, N., Lu, H., Zhang, Y., & Ruan, X. (2015b). Salient object detection via global and local cues. Pattern Recognition, 48(10), 3258–3267.

    Article  Google Scholar 

  • Von Neumann, J. (1951). The general and logical theory of automata. Cerebral Mechanisms in Behavior, 1(41), 1–2.

    MathSciNet  Google Scholar 

  • Von Neumann, J., Burks, A. W., et al. (1966). Theory of self-reproducing automata. IEEE Transactions on Neural Networks, 5(1), 3–14.

    Google Scholar 

  • Wang, L., Lu, H., Ruan, X., Yang, M. H. (2015). Deep networks for saliency detection via local estimation and global search. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 3183–3192).

  • Wang, L., Xue, J., Zheng, N., & Hua, G. (2011). Automatic salient object extraction with contextual cue. In Proceedings of the IEEE international conference on computer vision (pp. 105–112).

  • Wang, Q., Zheng, W., & Piramuthu, R. (2016). Grab: Visual saliency via novel graph model and background priors. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 535–543).

  • Wei, Y., Wen, F., Zhu, W., & Sun, J. (2012). Geodesic saliency using background priors. In Proceedings of European conference on computer vision (pp. 29–42).

  • Wolfram, S. (1983). Statistical mechanics of cellular automata. Reviews of Modern Physics, 55(3), 601.

    Article  MathSciNet  MATH  Google Scholar 

  • Xie, Y., & Lu, H. (2011). Visual saliency detection based on bayesian model. In Proceedings of IEEE international conference on image processing (pp. 645–648).

  • Xie, Y., Lu, H., & Yang, M. H. (2013). Bayesian saliency via low and mid level cues. IEEE Transactions on Image Processing, 22(5), 1689–1698.

    Article  MathSciNet  MATH  Google Scholar 

  • Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1155–1162).

  • Yang, C., Zhang, L., Lu, H., Ruan, X., & Yang, MH. (2013). Saliency detection via graph-based manifold ranking. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 3166–3173).

  • Yang, J., & Yang, M. H. (2012). Top-down visual saliency via joint crf and dictionary learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2296–2303). IEEE.

  • Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015). Saliency detection by multi-context deep learning. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1265–1274).

  • Zhou, F., Bing Kang, S., & Cohen, M. F. (2014). Time-mapping using space-time saliency. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Zhu, W., Liang, S., Wei, Y., & Sun, J. (2014). Saliency optimization from robust background detection. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2814–2821).

  • Zou, W., & Komodakis, N. (2015). Harf: Hierarchy-associated rich features for salient object detection. In Proceedings of the IEEE international conference on computer vision (pp. 406–414).

Download references

Acknowledgements

MY Feng and HC Lu were supported in part by the National Natural Science Foundation of China under Grants 61725202, 61528101, 61472060. YQ and GWC were also partially supported by Guangzhou Science and Technology Planning Project (Grant No. 201704030051).

Author information

Author notes

  1. Yao Qin and Mengyang Feng have equally contributed.

Authors and Affiliations

  1. University of California, San Diego, CA, USA

    Yao Qin & Garrison W. Cottrell

  2. Dalian University of Technology, Dalian, China

    Mengyang Feng & Huchuan Lu

Authors
  1. Yao Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengyang Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huchuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Garrison W. Cottrell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huchuan Lu.

Additional information

Communicated by Florent Perronnin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, Y., Feng, M., Lu, H. et al. Hierarchical Cellular Automata for Visual Saliency. Int J Comput Vis 126, 751–770 (2018). https://doi.org/10.1007/s11263-017-1062-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-017-1062-2

Keywords

Search

Navigation