Skip to main content
SpringerLink
Log in

An innovative algorithm for key frame extraction in video summarization

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

An Erratum to this article was published on 11 October 2012

Abstract

Video summarization, aimed at reducing the amount of data that must be examined in order to retrieve the information desired from information in a video, is an essential task in video analysis and indexing applications. We propose an innovative approach for the selection of representative (key) frames of a video sequence for video summarization. By analyzing the differences between two consecutive frames of a video sequence, the algorithm determines the complexity of the sequence in terms of changes in the visual content expressed by different frame descriptors. The algorithm, which escapes the complexity of existing methods based, for example, on clustering or optimization strategies, dynamically and rapidly selects a variable number of key frames within each sequence. The key frames are extracted by detecting curvature points within the curve of the cumulative frame differences. Another advantage is that it can extract the key frames on the fly: curvature points can be determined while computing the frame differences and the key frames can be extracted as soon as a second high curvature point has been detected. We compare the performance of this algorithm with that of other key frame extraction algorithms based on different approaches. The summaries obtained have been objectively evaluated by three quality measures: the Fidelity measure, the Shot Reconstruction Degree measure and the Compression Ratio measure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Agraim, P., Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: a state of the art review. Multimed. Tools Appl. 3, 179–202 (1996)

    Article  Google Scholar 

  2. Dimitrova, N., Zhang, H., Shahraray, B., Sezan, M., Huang, T., Zakhor, A.: Applications of video-content analysis and retrieval. IEEE MultiMed. 9(3), 44–55 (2002)

    Article  Google Scholar 

  3. Antani, S., Kasturi, R., Jain, R.: A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video. Pattern Recognit. 35, 945–965 (2002)

    Article  MATH  Google Scholar 

  4. Schettini, R., Brambilla, C., Cusano, C., Ciocca, G.: Automatic classification of digital photographs based on decision forests. Int. J. Pattern Recognit. Artif. Intell. 18(5), 819–846 (2004)

    Article  Google Scholar 

  5. Fredembach, C., Schröder, M., Süsstrunk, S.: Eigenregions for image classification. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 26(12), 1645–1649 (2004)

    Article  Google Scholar 

  6. Hauptmann, A.G., Jin, R., Tobun, D.N.: Video retrieval using speech and image information. In: Proceedings of Electronic Imaging Conference (EI’03), Storage Retrieval for Multimedia Databases, Santa Clara, CA, USA, vol. 5021, pp. 148–159 (2003)

  7. Tonomura, Y., Akutsu, A., Otsugi, K., Sadakata, T.: VideoMAP and VideoSpaceIcon: tools for automatizing video content. In: Proceedings of ACM INTERCHI ’93 Conference, pp. 131–141 (1993)

  8. Ueda, H., Miyatake, T., Yoshizawa, S.: IMPACT: an interactive natural-motion-picture dedicated multimedia authoring system. In: Proceedings of ACM CHI ’91 Conference, pp. 343–350 (1991)

  9. Rui, Y., Huang, T.S., Mehrotra, S.: Exploring video structure beyond the shots. In: Proceedings of IEEE International Conference on Multimedia Computing and Systems (ICMCS), Texas, USA, pp. 237–240 (1998)

  10. Pentland, A., Picard, R., Davenport, G., Haase, K.: Video and image semantics: advanced tools for telecommunications. IEEE MultiMed. 1(2), 73–75 (1994)

    Google Scholar 

  11. Sun, Z., Ping, F.: Combination of color and object outline based method in video segmentation. Proc. SPIE Storage Retr. Methods Appl. Multimed. 5307, 61–69 (2004)

    Google Scholar 

  12. Arman, F., Hsu, A., Chiu, M.Y.: Image processing on compressed data for large video databases. In: Proceedings of ACM Multimedia ’93, Annaheim, CA, USA, pp. 267–272 (1993)

  13. Zhuang, Y., Rui, Y., Huang, T.S., Mehrotra, S.: Key frame extraction using unsupervised clustering. In: Proceedings of ICIP’98, Chicago, USA, vol. 1, pp. 866–870 (1998)

  14. Girgensohn, A., Boreczky, J.: Time-constrained keyframe selection technique. Multimed. Tools Appl. 11, 347–358 (2000)

    Article  MATH  Google Scholar 

  15. Gong, Y., Liu, X.: Generating optimal video summaries. In: Proceedings of IEEE International Conference on Multimedia and Expo, vol. 3, pp. 1559–1562 (2000)

  16. Zhao, L., Qi, W., Li, S.Z., Yang, S.Q., Zhang, H.J.: Key-frame extraction and shot retrieval using nearest feature line (NFL). In: Proceedings of ACM International Workshops on Multimedia Information Retrieval, pp. 217–220 (2000)

  17. Hanjalic, A., Lagendijk, R.L., Biemond, J.: A new method for key frame based video content representation. In: Image Databases and Multimedia Search. World Scientific, Singapore (1998)

  18. Hoon, S.H., Yoon, K., Kweon, I.: A new technique for shot detection and key frames selection in histogram space. In: Proceedings of the 12th Workshop on Image Processing and Image Understanding, pp. 475–479 (2000)

  19. Narasimha, R., Savakis, A., Rao, R.M., De Queiroz, R.: A neural network approach to key frame extraction. In: Proceedings of SPIE-IS & T Electronic Imaging Storage and Retrieval Methods and Applications for Multimedia, vol. 5307, pp. 439–447 (2004)

  20. Calic, J., Izquierdo, E.: Efficient key-frame extraction and video analysis. In: Proceedings of IEEE ITCC2002, Multimedia Web Retrieval Section, pp. 28–33 (2002)

  21. Liu Tianming, M., Zhang, H.J., Qi, F.H.: A novel video key-frame-extraction algorithm based on perceived motion energy model. IEEE Trans. Circuits Syst. Video Technol. 13(10), 1006–1013 (2003)

    Article  Google Scholar 

  22. Zhang, H.J., Wu, J., Zhong, D., Smoliar, S.W.: An integrated system for contentbased video retrieval and browsing. Pattern Recognit. 30(4), 643–658 (1997)

    Article  Google Scholar 

  23. Fayzullin, M., Subrahmanian, V.S., Picarello, A., Sapino, M.L.: The CPR model for summarizing video. In: Proceedings of the 1st ACM International Workshop on Multimedia Databases, New Orleans, LA, USA, pp. 2–9 (2002)

  24. Lagendijk, R.L., Hanjalic, A., Ceccarelli, M.P., Soletic, M., Persoon, E.H.: Visual search in a SMASH system. In: Proceedings of ICIP’96, pp. 671–674 (1995)

  25. Ngo, C.-W., Ma, Y.-F., Zhang, H.-J.: Video summarization and scene detection by graph modeling. IEEE Trans. Circuits Syst. Video Technol. 15(2), 196–305 (2005)

    Google Scholar 

  26. Chang, H.S., Sull, S., Lee, S.U.: Efficient Video Indexing Scheme for Content-Based Retrieval. IEEE Trans. Circuits Syst. Video Technol. 9(8), 1269–1279 (1999)

    Article  Google Scholar 

  27. Tieyan, L., Zhang, X., Feng, J., Lo, K.T.: Shot reconstruction degree: a novel criterion for key frame selection. Pattern Recognit. Lett. 25, 1451–1457 (2004)

    Article  Google Scholar 

  28. Fernando, A.C., Canaharajah, C.N., Bull, D.R.: Fade-in and fade-out detection in video sequences using histograms. In: Proceedings of ISCAS 2000—IEEE International Symposium on Circuits and System, vol. IV, pp. 709–712 (2000)

  29. Swain, M., Ballard, D.: Color indexing. Int. J. Comput. Vis. 7(1), 11–32 (1991)

    Article  Google Scholar 

  30. Smith, J.R., Chang, S.F.: Tools and techniques for color image retrieval. In: IST/SPIE Storage and Retrieval for Image and Video Databases IV, vol. 2670, pp. 426–437 (1996)

  31. Ciocca, G., Gagliardi, I., Schettini, R.: Quicklook2 : an integrated multimedia system. Int. J. Vis. Lang. Comput. 12, 81–103 (Special issue on querying multiple data sources) (2001)

    Google Scholar 

  32. Gonzalez, R., Woods, R.: Digital image processing. Addison Wesley, Reading, pp. 414–428 (1992)

    Google Scholar 

  33. Idris, F., Panchanathan, S.: Storage and retrieval of compressed images using wavelet vector quantization. J. Vis. Lang. Comput. 8, 289–301 (1997)

    Article  Google Scholar 

  34. Scheunders, P., Livens, S., Van de Wouwer, G., Vautrot, P., Van Dyck, D.: Wavelet-based texture analysis. Int. J. Comput. Sci. Inf. Manage. 1(2), 22–34 (1998)

    Google Scholar 

  35. Chetverikov, D., Szabo, Zs.: A simple and efficient algorithm for detection of high curvature points in planar curves. In: Proceedings of the 23rd Workshop of the Austrian Pattern Recognition Group, pp. 175–184 (1999)

  36. Latecki, L., DeMenthon, D., Rosenfeld, A.: Extraction of key frames from videos by polygon simplification. In: International Symposium on Signal Processing and its Applications, pp. 643–646 (2001)

  37. Lefevre, S., Holler, J., Vincent, N.: A review of real time segmentation of uncompressed video sequences for content-based search and retrieval. Real Time Imaging 9, 73–98 (2003)

    Article  Google Scholar 

  38. Daubechies, I., Sweldens, W.: Factoring wavelet transforms into lifting steps. J. Fourier Anal. Appl. 4(3), 247–269 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  39. AESS Archive. http://aess.itc.cnr.it/index.htm

  40. MAIS Consortium, Mais: Multichannel Adaptive Information Systems. http://black.elet.polimi.it/mais/

Download references

Acknowledgements

The video indexing and analysis presented here was supported by the Italian MURST FIRB Project MAIS (Multi-channel Adaptive Information Systems) [40] and by the Regione Lombardia (Italy) within the INTERNUM project aimed at facilitating access to the cultural video documentaries of the AESS (Archivio di Etnografia e Storia Sociale).

Author information

Authors and Affiliations

  1. Dipartimento di Informatica Sistemistica e Comunicazione (DISCo), Università degli studi di Milano-Bicocca, Via Bicocca degli Arcimboldi 8, 20126, Milano, Italy

    Ciocca Gianluigi & Schettini Raimondo

Authors
  1. Ciocca Gianluigi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Schettini Raimondo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ciocca Gianluigi.

Additional information

An erratum to this article can be found online at http://dx.doi.org/10.1007/s11554-012-0278-1.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gianluigi, C., Raimondo, S. An innovative algorithm for key frame extraction in video summarization. J Real-Time Image Proc 1, 69–88 (2006). https://doi.org/10.1007/s11554-006-0001-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-006-0001-1

Keywords

Search

Navigation