Skip to main content
SpringerLink
Log in

Multi-feature-based crowd video modeling for visual event detection

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

We propose a novel method for modeling crowd video dynamics by adopting a two-stream convolutional architecture which incorporates spatial and temporal networks. Our proposed method cope with the key challenge of capturing the complementary information on appearance from still frames and motion between frames. In our proposed method, a motion flow field is obtained from the video through dense optical flow. We demonstrate that the proposed method trained on multi-frame dense optical flow achieves significant improvement in performance in spite of limited training data. We train and evaluate our proposed method on a benchmark crowd video dataset. The experimental results of our method show that it outperforms five reference methods. We have chosen these reference methods since they are the most relevant to our work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Dávila, J.D.: Chapter three cities as innovation towards a new understanding of population growth, social inequality and urban sustainability. Cities in the 21st Century, Routledge p. 26 (2016)

  2. Ullah, H., Altamimi, A.B., Uzair, M., Ullah, M.: Anomalous entities detection and localization in pedestrian flows. J. Neurocomput. 290, 74 (2018)

    Article  Google Scholar 

  3. Saqib, M., Khan, S.D., Sharma, N., Blumenstein, M.: In: International conference on image and vision computing, New Zealand, IEEE IVCNZ, pp. 1–6 (2017)

  4. Muñoz-Salinas, R., Marín-Jimenez, M.J., Medina-Carnicer, R.: Spm-slam: simultaneous localization and mapping with squared planar markers. J. Pattern Recognit. 86, 156 (2019)

    Article  Google Scholar 

  5. Salmerón-García, J.J., van den Dries, S., Díaz-del Río, F., Morgado-Estevez, A., Sevillano-Ramos, J.L., van de Molengraft, M.: Towards a cloud-based automated surveillance system using wireless technologies. Multimed. Syst. 25(5), 535 (2019)

    Article  Google Scholar 

  6. Hassan, M.M., Alam, M.G.R., Uddin, M.Z., Huda, S., Almogren, A., Fortino, G.: Human emotion recognition using deep belief network architecture. J. Inf. Fusion. 51, 10 (2019)

    Article  Google Scholar 

  7. Xia, D.X., Su, S.Z., Geng, L.C., Wu, G.X., Li, S.Z.: Learning rich features from objectness estimation for human lying-pose detection. Multimed. Syst. 23(4), 515 (2017)

    Article  Google Scholar 

  8. Shih, H.C.: A survey of content-aware video analysis for sports. Trans. Circ. Syst. Video Technol. IEEE 28(5), 1212 (2018)

    Article  Google Scholar 

  9. Wang, J., Xu, C., Chng, E., Lu, H., Tian, Q.: Automatic composition of broadcast sports video. Multimed. Syst. 14(4), 179 (2008)

    Article  Google Scholar 

  10. Hsu, C.C., Chen, H.T., Chou, C.L., Lee, S.Y.: 2D histogram-based player localization in broadcast volleyball videos. Multimed. Syst. 22(3), 325 (2016)

    Article  Google Scholar 

  11. Krähenbühl, P.: In: Conference on computer vision and pattern recognition. IEEE CVPR, pp. 2955–2964 (2018)

  12. Ji, Z., Ma, Y., Pang, Y., Li, X.: Query-aware sparse coding for web multi-video summarization. J. Inf. Sci. 478, 152 (2019)

    Article  Google Scholar 

  13. Wu, S., Yang, H., Zheng, S., Su, H., Fan, Y., Yang, M.H.: Crowd behavior analysis via curl and divergence of motion trajectories. Int. J. Comput. Vis. 123(3), 499 (2017)

    Article  MathSciNet  Google Scholar 

  14. Ren, W., Kang, D., Tang, Y., Chan, A.B.: In: Conference on computer vision and pattern recognition. IEEE CVPR, pp. 5353–5362 (2018)

  15. Ullah, M., Cheikh, F.A.: In: International conference on image processing. IEEE ICIP, pp. 3738–3742 (2018)

  16. Ullah, M., Alaya Cheikh, F.: In: International conference on computer vision and pattern recognition workshops. IEEE CVPRw, pp. 1816–1823 (2018)

  17. Liu, J., Gao, C., Meng, D., Hauptmann, A.G.: In: Conference on computer vision and pattern recognition. IEEE CVPR, pp. 5197–5206 (2018)

  18. Ullah, H., Ullah, M., Uzair, M.: A hybrid social influence model for pedestrian motion segmentation. J. Neural Comput. Appl. pp. 1–17 (2018)

  19. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. Trans. Pattern Anal. Machine Intell. IEEE 40(4), 834 (2018)

    Article  Google Scholar 

  20. Cheng, G., Yang, C., Yao, X., Guo, L., Han, J.: When deep learning meets metric learning: remote sensing image scene classification via learning discriminative cnns. Trans. Geosci. Remote Sens. IEEE 56(5), 2811 (2018)

    Article  Google Scholar 

  21. Zhang, Z., Geiger, J., Pohjalainen, J., Mousa, A.E.D., Jin, W., Schuller, B.: Deep learning for environmentally robust speech recognition: an overview of recent developments. Trans. Intell. Syst. Technol. ACM 9(5), 49 (2018)

    Article  Google Scholar 

  22. Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N., Peters, M., Schmitz, M., Zettlemoyer, L.: Allennlp: a deep semantic natural language processing platform (2018). arXiv:1803.07640

  23. Ullah, H., Khan, S.D., Ullah, M., Cheikh, F.A., Uzair, M.: In: 2019 8th European workshop on visual information processing (EUVIP) (IEEE, 2019), pp. 93–98 (2019)

  24. Shao, J., Change Loy, C., Wang, X.: In: Conference on computer vision and pattern recognition. IEEE CVPR, pp. 2219–2226 (2014)

  25. Shao, J., Loy, C.C., Wang, X.: Learning scene-independent group descriptors for crowd understanding. Trans. Circ. Syst. Video Technol. IEEE 27(6), 1290 (2017)

    Article  Google Scholar 

  26. Zhang, X., Zhang, Q., Hu, S., Guo, C., Yu, H.: Energy level-based abnormal crowd behavior detection. J. Sens. 18(2), 423 (2018)

    Article  Google Scholar 

  27. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: In: International conference on artificial neural networks. Spinger ICANN, pp. 154–159 (2010)

  28. Burney, A., Syed, T.Q.: In: International conference on frontiers of information technology. IEEE FIT, pp. 247–251 (2016)

  29. Tripathi, G., Singh, K., Vishwakarma, D.K.: Convolutional neural networks for crowd behaviour analysis: a survey. Vis. Comput. 35(5), 753 (2019)

    Article  Google Scholar 

  30. Gahr, B., Ryder, B., Dahlinger, A., Wortmann, F.: In: International conference on machine learning and data mining in pattern recognition. Springer ICMLDM, pp. 183–197 (2018)

  31. Zhao, W., Zhang, Z., Huang, K.: Gestalt laws based tracklets analysis for human crowd understanding. J. Pattern Recognit. 75, 112 (2018)

    Article  Google Scholar 

  32. Cao, L., Zhang, X., Ren, W., Huang, K.: Large scale crowd analysis based on convolutional neural network. J. Pattern Recognit. 48(10), 3016 (2015)

    Article  Google Scholar 

  33. Senst, T., Eiselein, V., Kuhn, A., Sikora, T.: Crowd violence detection using global motion-compensated lagrangian features and scale-sensitive video-level representation. Trans. Inf. Forensics Secur. 12(12), 2945 (2017)

    Article  Google Scholar 

  34. Liu, W., Lau, R.W., Manocha, D.: Robust individual and holistic features for crowd scene classification. Pattern Recognit. 58, 110 (2016)

    Article  Google Scholar 

  35. Wu, S., Su, H., Yang, H., Zheng, S., Fan, Y., Zhou, Q.: Bilinear dynamics for crowd video analysis. J. Vis. Commun. Image Rep. 48, 461 (2017)

    Article  Google Scholar 

  36. Fradi, H., Luvison, B., Pham, Q.C.: Crowd behavior analysis using local mid-level visual descriptors. Trans. Circ. Syst. Video Technol. 27(3), 589 (2017)

    Article  Google Scholar 

  37. Xu, M., Ge, Z., Jiang, X., Cui, G., Lv, P., Zhou, B., Xu, C.: Depth information guided crowd counting for complex crowd scenes. Pattern Recogn. Lett. 125, 563 (2019)

    Article  Google Scholar 

  38. Karamouzas, I., Sohre, N., Hu, R., Guy, S.J.: Crowd space: a predictive crowd analysis technique. ACM Trans. Graph. (TOG) 37(6), 1 (2018)

    Article  Google Scholar 

  39. Wang, Q., Chen, M., Nie, F., Li, X.: Detecting coherent groups in crowd scenes by multiview clustering. IEEE Trans. Pattern Anal. Mach. Intell. 42(1), 46 (2018)

    Article  Google Scholar 

  40. Zhao, W., Zhang, Z., Huang, K.: Gestalt laws based tracklets analysis for human crowd understanding. Pattern Recogn. 75, 112 (2018)

    Article  Google Scholar 

  41. Feichtenhofer, C., Pinz, A., Zisserman, A.: In Conference on computer vision and pattern recognition. IEEE CVPR, pp. 1933–1941 (2016)

  42. Simonyan, K., Zisserman, A.: Advances in neural information processing systems. NIPS, pp. 568–576 (2014)

  43. Wang, H., Kläser, A., Schmid, C., Liu, C.L.: In: Conference on computer vision and pattern recognition. IEEE CVPR, pp. 3169–3176 (2011)

  44. Dalal, N., Triggs, B., Schmid, C.: In: European conference on computer vision. Springer ECCV, pp. 428–441 (2006)

Download references

Acknowledgements

This research work is jointly funded by the Deanship of Resarch at the University of Ha’il and the Ministry of Education, KSA under the Grant Number GR58954656.

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, University of Hail, Hail, Saudi Arabia

    Habib Ullah & Sultan Daud Khan

  2. Sarhad University of Science and IT, Peshawar, 25000, Pakistan

    Ihtesham Ul Islam & Javed Iqbal

  3. Norwegian University of Science and Technology, Gjovik, Norway

    Mohib Ullah

  4. Jeju National University, Jeju-Si, South Korea

    Muhammad Afaq

Authors
  1. Habib Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ihtesham Ul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohib Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Afaq
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sultan Daud Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Javed Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ihtesham Ul Islam.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ullah, H., Islam, I.U., Ullah, M. et al. Multi-feature-based crowd video modeling for visual event detection. Multimedia Systems 27, 589–597 (2021). https://doi.org/10.1007/s00530-020-00652-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-020-00652-x

Keywords

Search

Navigation