Skip to main content
SpringerLink
Log in

Neural network solution for a real-time no-reference video quality assessment of H.264/AVC video bitstreams

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The ever-growing video streaming services require accurate quality assessment with often no reference to the original media. One primary challenge in developing no-reference (NR) video quality metrics is achieving real-timeliness while retaining the accuracy. A real-time no-reference video quality assessment (VQA) method is proposed for videos encoded by H.264/AVC codec. Temporal and spatial features are extracted from the encoded bit-stream and pixel values to train and validate a fully connected neural network. The hand-crafted features and network dynamics are designed in a manner to ensure a high correlation with human judgment of quality as well as minimizing the computational complexities. Proof-of-concept experiments are conducted via comparison with: 1) video sequences rated by a full-reference quality metric, and 2) H.264-encoded sequences from the LIVE video dataset which are subjectively evaluated through differential mean opinion scores (DMOS). The performance of the proposed method is verified by correlation measurements with the aforementioned objective and subjective scores. The framework achieves real-time execution while outperforming state-of-art full-reference and no-reference video quality assessment methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/

  2. Ahn S, Lee S (2018) No-reference video quality assessment based on convolutional neural network and human temporal behavior. In: Asia-Pacific signal and information processing association annual summit and conference (APSIPA ASC), pp 1513–1517

  3. Born RT, Bradley DC (2005) Structure and function of visual area MT. Ann Rev Neurosci 28(1):157–189. https://doi.org/10.1146/annurev.neuro.26.041002.131052

    Article  Google Scholar 

  4. Caruana R, Lawrence S, Giles CL (2001) Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping. In: Advances in neural information processing systems, pp 402–408

  5. Fang R, Al-Bayaty R, Wu D (2017) BNB Method for no-reference image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology 27(7):1381–1391

    Article  Google Scholar 

  6. Fazliani Y, Andrade E, Shirani S (2019) Learning based hybrid no-reference video quality assessment of compressed videos. In: 2019 IEEE international symposium on circuits and systems (ISCAS), pp 1–5

  7. Gu J, Meng G, Redi JA, Xiang S, Pan C (2018) Blind image quality assessment via vector regression and object oriented pooling. IEEE Transactions on Multimedia 20(5):1140–1153

    Article  Google Scholar 

  8. Gunawan IP, Ghanbari M (2008) Reduced-reference video quality assessment using discriminative local harmonic strength with motion consideration. IEEE Transactions on Circuits and Systems for Video Technology 18(1):71–83

    Article  Google Scholar 

  9. Hou W, Gao X, Tao D, Li X (2015) Blind image quality assessment via deep learning. IEEE Transactions on Neural Networks and Learning Systems 26(6):1275–1286

    Article  MathSciNet  Google Scholar 

  10. Huynh-Thu Q, Ghanbari M (2008) Temporal aspect of perceived quality in mobile video broadcasting. IEEE Trans Broadcast 54(3):641–651

    Article  Google Scholar 

  11. Keimel C, Klimpke M, Habigt J, Diepold K (2011) No-reference video quality metric for hdtv based on h.264/avc bitstream features. In: IEEE international conference on image processing, pp 3325–3328

  12. Kim J, Lee S (2017) Fully deep blind image quality predictor. IEEE Journal of Selected Topics in Signal Processing 11(1):206–220

    Article  Google Scholar 

  13. Kim J, Zeng H, Ghadiyaram D, Lee S, Zhang L, Bovik AC (2017) Deep convolutional neural models for picture-quality prediction: challenges and solutions to data-driven image quality assessment. IEEE Signal Proc Mag 34(6):130–141

    Article  Google Scholar 

  14. Larson EC, Chandler DM (2010) Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging 19(1):011006:1–011006:21

    Google Scholar 

  15. Lay D (2012) Linear algebra and its applications. Addison-Wesley

  16. Liu W, Duanmu Z, Wang Z (2018) End-to-end blind quality assessment of compressed videos using deep neural networks. In: ACM international conference on multimedia, ACM, MM ’18, pp 546–554

  17. Mittal A, Moorthy AK, Bovik AC (2012) No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 21(12):4695–4708

    Article  MathSciNet  Google Scholar 

  18. Mittal A, Soundararajan R, Bovik AC (2013) Making a completely blind image quality analyzer. IEEE Signal Processing Letters 20(3):209–212

    Article  Google Scholar 

  19. Mocanu DC, Liotta A, Ricci A, Vega MT, Exarchakos G (2014) When does lower bitrate give higher quality in modern video services?. In: IEEE network operations and management symposium (NOMS), pp 1–5

  20. Ninassi A, Meur OL, Callet PL, Barba D (2009) Considering temporal variations of spatial visual distortions in video quality assessment. IEEE Journal of Selected Topics in Signal Processing 3(2):253– 265

    Article  Google Scholar 

  21. Péchard S, Carnec M, Callet PL, Barba D (2006) From sd to hd television: effects of h.264 distortions versus display size on quality of experience. International Conference on Image Processing: 409–412

  22. Pinson MH, Wolf S (2004) A new standardized method for objectively measuring video quality. IEEE Trans Broadcast 50(3):312–322

    Article  Google Scholar 

  23. Pitrey Y, Barkowsky M, Callet PL, Pépion R (2010) Subjective quality evaluation of h.264 high-definition video coding versus spatial up-scaling and interlacing. Euro ITV

  24. Pitrey Y, Barkowsky M, Pépion R, Callet PL, Hlavacs H (2012) Influence of the source content and encoding configuration on the perceived quality for scalable video coding. SPIE Proc 8291:1–6

    Google Scholar 

  25. Richardson IE (2010) The H.264 advanced video compression standard. Wiley

  26. Saad MA, Bovik AC, Charrier C (2014) Blind prediction of natural video quality. IEEE Trans Image Process 23(3):1352–1365

    Article  MathSciNet  Google Scholar 

  27. Seshadrinathan K, Bovik AC (2010) Motion tuned spatio-temporal quality assessment of natural videos. IEEE Trans Image Process 19(2):335–350

    Article  MathSciNet  Google Scholar 

  28. Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Process 19(6):1427–1441

    Article  MathSciNet  Google Scholar 

  29. Shahid M, Panasiuk J, Wallendael GV, Barkowsky M, Lövström B (2015) Predicting full-reference video quality measures using hevc bitstream-based no-reference features. In: International workshop on quality of multimedia experience (QoMEX), pp 1–2

  30. Soundararajan R, Bovik AC (2012) RRED indices: reduced reference entropic differencing for image quality assessment. IEEE Trans Image Process 21 (2):517–526

    Article  MathSciNet  Google Scholar 

  31. Søgaard J, Forchhammer S, Korhonen J (2015) No-reference video quality assessment using codec analysis. IEEE Transactions on Circuits and Systems for Video Technology 25(10):1637–1650

    Article  Google Scholar 

  32. Usman MA, Shin SY, Shahid M, Lövström B (2016) A no reference video quality metric based on jerkiness estimation focusing on multiple frame freezing in video streaming. IETE Tech Rev 34(3):309–320

    Article  Google Scholar 

  33. Vega MT, Liotta A (2016) LIMP video quality database. https://www.tue.nl/index.php?id=53688

  34. Vega MT, Mocanu DC, Famaey J, Stavrou S, Liotta A (2017) Deep learning for quality assessment in live video streaming. IEEE Signal Processing Letters 24(6):736–740

    Article  Google Scholar 

  35. Video Quality Experts Group (VQEG) (2000) Final report from the video quality experts group on the validation of objective models of video quality assessment, phase I. https://www.tue.nl/index.php?id=53688, http://www.its.bldrdoc.gov/vqeg/projects/frtv_phaseI

  36. Vink JP, de Haan G (2011) No-reference metric design with machine learning for local video compression artifact level. IEEE Journal of Selected Topics in Signal Processing 5(2):297–308

    Article  Google Scholar 

  37. Vu PV, Vu CT, Chandler DM (2011) A spatiotemporal most-apparent-distortion model for video quality assessment. In: IEEE international conference on image processing, pp 2505–2508

  38. Wang Z, Bovik AC, Evan BL (2000) Blind measurement of blocking artifacts in images. In: International conference on image processing, vol 3, pp 981–984

  39. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  40. Wang Z, Sheikh HR, Bovik AC (2002) No-reference perceptual quality assessment of jpeg compressed images. In: International conference on image processing, vol 1, pp I–477–I–480

  41. Weiss Y, Freeman WT (2007) What makes a good model of natural images?. In: IEEE conference on computer vision and pattern recognition, pp 1–8

  42. Xu B, Pan X, Zhou Y, Li Y, Yang D, Chen Z (2017) CNN-based rate-distortion modeling for H.265/HEVC. In: 2017 IEEE visual communications and image processing (VCIP), pp 1–4

  43. Xu J, Ye P, Liu Y, Doermann D (2014) No-reference video quality assessment via feature learning. In: IEEE international conference on image processing (ICIP), pp 491–495

  44. Ye P, Kumar J, Kang L, Doermann D (2012) Unsupervised feature learning framework for no-reference image quality assessment. In: IEEE conference on computer vision and pattern recognition, pp 1098–1105

  45. Zeng K, Wang Z (2010) Temporal motion smoothness measurement for reduced-reference video quality assessment. In: IEEE international conference on acoustics, speech and signal processing, pp 1010–1013

  46. Zhu K, Hirakawa K, Asari V, Saupe D (2013) A no-reference video quality assessment based on laplacian pyramids. In: IEEE international conference on image processing, pp 49–53

Download references

Author information

Authors and Affiliations

  1. McMaster University, Hamilton, Canada

    Yasamin Fazliani & Shahram Shirani

  2. Evertz Microsystems, Burlington, Canada

    Yasamin Fazliani, Ernesto Andrade & Shahram Shirani

Authors
  1. Yasamin Fazliani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ernesto Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shahram Shirani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yasamin Fazliani.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fazliani, Y., Andrade, E. & Shirani, S. Neural network solution for a real-time no-reference video quality assessment of H.264/AVC video bitstreams. Multimed Tools Appl 81, 2409–2427 (2022). https://doi.org/10.1007/s11042-021-10654-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10654-0

Keywords

Search

Navigation