Lea Skorin-Kapov
Martín Varela
Tobias Hoßfeld
Kuan-Ta Chen
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Quality of Experience (QoE) has received much attention over the past years and has become a prominent issue for delivering services and applications. A significant amount of research has been devoted to understanding, measuring, and modelling QoE for a variety of media services. The next logical step is to actively exploit that accumulated knowledge to improve and manage the quality of multimedia services, while at the same time ensuring efficient and cost-effective network operations. Moreover, with many different players involved in the end-to-end service delivery chain, identifying the root causes of QoE impairments and finding effective solutions for meeting the end users’ requirements and expectations in terms of service quality is a challenging and complex problem. In this article, we survey state-of-the-art findings and present emerging concepts and challenges related to managing QoE for networked multimedia services. Going beyond a number of previously published survey articles addressing the topic of QoE management, we address QoE management in the context of ongoing developments, such as the move to softwarized networks, the exploitation of big data analytics and machine learning, and the steady rise of new and immersive services (e.g., augmented and virtual reality). We address the implications of such paradigm shifts in terms of new approaches in QoE modeling and the need for novel QoE monitoring and management infrastructures.
Supplemental Material
Available for Download
zip
skorin-kapov.zip (135.1 KB)
Supplemental movie, appendix, image and software files for, A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services
- 3GPP. 2017. Study on improved streaming Quality of Experience (QoE) reporting in 3GPP services and networks, 3GPP TR 26.909, Release 14, v14.0.0.Google Scholar
- V. Aggarwal, E. Halepovic, J. Pang, S. Venkataraman, and H. Yan. 2014. Prometheus: Toward quality of experience estimation for mobile apps from passive network measurements. In Proceedings of the 15th Workshop on Mobile Computing Systems and Applications. ACM, 18. Google ScholarDigital Library
- A. Ahmad, L. Atzori, and M. G. Martini. 2017. Qualia: A multilayer solution for QoE passive monitoring at the user terminal. In Proceedings of the IEEE International Conference on Communications (ICC’17).Google Scholar
- A. Ahmad, A. Floris, and L. Atzori. 2016. QoE-centric service delivery: A collaborative approach among OTTs and ISPs. Comput. Networks 110 (2016), 168--179. Google ScholarDigital Library
- A. Aijaz, M. Dohler, A. H. Aghvami, V. Friderikos, and M. Frodigh. 2017. Realizing the tactile internet: Haptic communications over next generation 5G cellular networks. IEEE Wireless Commun. 24, 2 (2017), 82--89. Google ScholarDigital Library
- M. Alreshoodi and J. Woods. 2013. Survey on QoE QoS correlation models for multimedia services. arXiv preprint arXiv:1306.0221 (2013).Google Scholar
- S. Aroussi and A. Mellouk. 2014. Survey on machine-learning-based QoE-QoS correlation models. In Proceedinsg of the International Conference on Computing, Management and Telecommunications (ComManTel’14). IEEE, 200--204.Google Scholar
- AT8T. 2016. ECOMP (Enhanced Control, Orchestration, Management 8 Policy) Architecture White Paper.Google Scholar
- O. Awobuluyi, J. Nightingale, Q. Wang, and J. M. Alcaraz-Calero. 2015. Video quality in 5G networks: Context-aware QoE management in the SDN control plane. In Proceedings of the 2015 International Conference on CIT/IUCC/DASC/PICOM. IEEE, 1657--1662.Google Scholar
- A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, and H. Zhang. 2013. Developing a predictive model of quality of experience for internet video. In ACM SIGCOMM Computer Communication Review, Vol. 43. ACM, 339--350. Google ScholarDigital Library
- Y. Bao, H. Wu, and X. Liu. 2017. From prediction to action: Improving user experience with data-driven resource allocation. IEEE J. Select. Areas Commun. 35, 5 (May 2017), 1062--1075.Google ScholarDigital Library
- A. A. Barakabitze, L. Sun, I.-H. Mkwawa, and E. Ifeachor. 2016. A novel QoE-aware SDN-enabled, NFV-based management architecture for future multimedia applications on 5G systems. In Proceedings of the 8th International Conference on Quality of Multimedia Experience (QoMEX’16). Lisbon, Portugal.Google Scholar
- S. Baraković and L. Skorin-Kapov. 2013. Survey and challenges of QoE management issues in wireless networks. J. Comput. Networks Commun. 2013 (2013).Google Scholar
- A. Benoit, P. Le Callet, P. Campisi, and R. Cousseau. 2009. Quality assessment of stereoscopic images. EURASIP J. Image Video Process. 2008, 1 (2009), 659024.Google Scholar
- A. Bentaleb, A. C. Begen, and R. Zimmermann. 2016. SDNDASH: Improving QoE of HTTP adaptive streaming using software defined networking. In Proceedings of the ACM Conference on Multimedia. ACM, 1296--1305. Google ScholarDigital Library
- N. Bouten, S. Latré, J. Famaey, W. Van Leekwijck, and F. De Turck. 2014. In-network quality optimization for adaptive video streaming services. IEEE Trans. Multimedia 16, 8 (Dec. 2014), 2281--2293.Google ScholarCross Ref
- A. Breznick. Feb. 2017. How to Deliver Higher Mobile Video QoE, a Heavy Reading white paper produced for Akamai Technologies Inc. (Feb. 2017).Google Scholar
- F. E. Bustamante, D. Clark, and N. Feamster. 2017. Workshop on tracking quality of experience in the internet: Summary and outcomes. ACM SIGCOMM Comput. Commun. Rev. 47, 1 (2017), 55--60. Google ScholarDigital Library
- P. Casas, B. Gardlo, R. Schatz, and M. Mellia. 2016. An educated guess on QoE in operational networks through large-scale measurements. In Proceedings of the 2016 Workshop on QoE-based Analysis and Management of Data Communication Networks. ACM, 1--6. Google ScholarCross Ref
- P. Casas and R. Schatz. 2014. Quality of experience in cloud services: Survey and measurements. Computer Networks 68 (2014), 149--165.Google ScholarCross Ref
- P. Casas, M. Seufert, F. Wamser, B. Gardlo, A. Sackl, and R. Schatz. 2016. Next to you: Monitoring quality of experience in cellular networks from the end-devices. IEEE Trans. Network Service Manage. 13, 2 (2016), 181--196.Google ScholarCross Ref
- C. Cavaro-Ménard, L. Zhang-Ge, and P. Le Callet. 2013. QoE for telemedicine: Challenges and trends. In SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, 88561A--88561A.Google Scholar
- K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei. 2010. Quadrant of Euphoria: A crowdsourcing platform for QoE assessment. IEEE Network 24, 2 (2010). Google ScholarDigital Library
- Q. A. Chen and others. 2014. QoE doctor: Diagnosing mobile app QoE with automated UI control and cross-layer analysis. In Proceedings of the 2014 Conference on Internet Measurement Conference. ACM, 151--164. Google ScholarDigital Library
- W. Chen, F. Jérôme, M. Barkowsky, and P. Le Callet. 2012. Exploration of quality of experience of stereoscopic images: Binocular depth. In Proceedings of the 6th International Workshop on Video Processing and Quality Metrics for Consumer Electronics. 1--6.Google Scholar
- S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam. 2011. Objective video quality assessment methods: A classification, review, and performance comparison. IEEE Trans. Broadcast. 57, 2 (2011), 165--182.Google ScholarCross Ref
- B.-H. Chu, M.-S. Tsai, and C.-S. Ho. 2007. Toward a hybrid data mining model for customer retention. Knowl.-Based Syst. 20, 8 (2007), 703--718. Google ScholarDigital Library
- L. A. Clark and D. Watson. 1995. Constructing validity: Basic issues in objective scale development. Psychol. Assess. 7, 3 (1995), 309.Google ScholarCross Ref
- L. Cui, F. R. Yu, and Q. Yan. 2016. When big data meets software-defined networking: SDN for big data and big data for SDN. IEEE Network 30, 1 (2016), 58--65.Google ScholarDigital Library
- G. Dimopoulos, I. Leontiadis, P. Barlet-Ros, and K. Papagiannaki. 2016. Measuring video QoE from encrypted traffic. In Proceedings of the 2016 ACM on Internet Measurement Conference. ACM, 513--526. Google ScholarDigital Library
- L. Dinh-Xuan, M. Seufert, F. Wamser, and P. Tran-Gia. 2017. Study on the accuracy of QoE monitoring for HTTP adaptive video streaming using VNF. In Proceedings of the 1st IFIP/IEEE International Workshop on QoE Management. IEEE, 1--6.Google Scholar
- F. Dobrian et al. 2011. Understanding the impact of video quality on user engagement. In ACM SIGCOMM Computer Communication Review, Vol. 41. ACM, 362--373. Google ScholarDigital Library
- O. Dobrijevic, M. Santl, and M. Matijasevic. 2015. Ant colony optimization for QoE-centric flow routing in software-defined networks. In Proceedings of the 11th International Conference on Network and Service Management (CNSM’15). IEEE, 274--278. Google ScholarDigital Library
- L. Duan, L. Huang, C. Langbort, A. Pozdnukhov, J. Walrand, and L. Zhang. 2017. Human-in-the-loop mobile networks: A survey of recent advancements. IEEE JSAC 35, 4 (2017).Google Scholar
- U. Engelke, D. P. Darcy, G. H. Mulliken, S. Bosse, M. G. Martini, S. Arndt, J.-N. Antons, K. Y. Chan, N. Ramzan, and K. Brunnström. 2017. Psychophysiology-based QoE assessment: A survey. IEEE J. Select. Topics Signal Process. 11, 1 (2017), 6--21.Google ScholarCross Ref
- U. Engelke and H.-J. Zepernick. 2007. Perceptual-based quality metrics for image and video services: A survey. In Proceedings of the 3rd EuroNGI Conference on Next Generation Internet Networks. IEEE, 190--197.Google ScholarCross Ref
- J. B. Ernst, S. C. Kremer, and J. J. Rodrigues. 2014. A survey of QoS/QoE mechanisms in heterogeneous wireless networks. Phys. Commun. 13, PB (Dec. 2014), 61--72. Google ScholarDigital Library
- ETSI. 2014. NFV MANO—Network Functions Virtualisation (NFV); Management and Orchestration, v1.1.1. (2014).Google Scholar
- ETSI. 2014. Speech and multimedia Transmission Quality (STQ); Quality of Experience: A Monitoring Architecture, ETSI TS 103 294, V1.1.1. (Dec. 2014).Google Scholar
- M. Fiedler and T. Hoßfeld. 2010. Quality of experience-related differential equations and provisioning-delivery hysteresis. In Proceedings of the 21st ITC Specialist Seminar on Multimedia Applications-Traffic, Performance and QoE. IEICE.Google Scholar
- M. Fiedler, T. Hoßfeld, and P. Tran-Gia. 2010. A generic quantitative relationship between quality of experience and quality of service. IEEE Network 24, 2 (2010). Google ScholarDigital Library
- A. Floris, A. Ahmad, and L. Atzori. 2017. OTT-ISP joint service management: A customer lifetime value-based approach. In Proceedings of the 1st IFIP/IEEE International Workshop on QoE-Management.Google Scholar
- M. Gandy, R. Catrambone, B. MacIntyre, C. Alvarez, E. Eiriksdottir, M. Hilimire, B. Davidson, and A. C. McLaughlin. 2010. Experiences with an AR evaluation test bed: Presence, performance, and physiological measurement. In Proceedings of the 2010 IEEE International Symposium on Mixed and Augmented Reality. 127--136.Google Scholar
- A. Ganjam, V. Sekar, and H. Zhang. 2016. In-situ quality of experience monitoring: The case for prioritizing coverage over fidelity. White-paper submission, CMU/Conviva (2016), 2.Google Scholar
- M.-N. Garcia, F. De Simone, S. Tavakoli, N. Staelens, S. Egger, K. Brunnström, and A. Raake. 2014. Quality of experience and HTTP adaptive streaming: A review of subjective studies. In Proceedings of the 2014 6th International Workshop on Quality of Multimedia Experience (QoMEX’14). IEEE, 141--146.Google Scholar
- B. Gardlo, S. Egger, and T. Hoßfeld. 2015. Do scale-design and training matter for video QoE assessments through crowdsourcing? In Proceedings of the 4th International Workshop on Crowdsourcing for Multimedia. ACM, 15--20. Google ScholarDigital Library
- C. Ge, N. Wang, S. Skillman, G. Foster, and Y. Cao. 2016. QoE-driven DASH video caching and adaptation at 5G mobile edge. In Proceedings of the 3rd ACM Conference on Information-Centric Networking. ACM, 237--242. Google ScholarDigital Library
- G. Ghinea and O. A. Ademoye. 2010. Perceived synchronization of olfactory multimedia. IEEE Transactions on Syst. Man Cybernet.—Part A: Syst. Hum. 40, 4 (July 2010), 657--663. Google ScholarDigital Library
- G. Ghinea, C. Timmerer, W. Lin, and S. R. Gulliver. 2014. Mulsemedia: State of the art, perspectives, and challenges. ACM TOMM 11, 1s, Article 17 (Oct. 2014), 23 pages. Google ScholarDigital Library
- U. Goel, M. P. Wittie, K. C. Claffy, and A. Le. 2016. Survey of end-to-end mobile network measurement testbeds, tools, and services. IEEE Commun. Surveys Tutor. 18, 1 (2016), 105--123.Google ScholarDigital Library
- J. Gupchup, Y. Hosseinkashi, M. Ellis, S. Johnson, and R. Cutler. 2017. Analysis of problem tokens to rank factors impacting quality in VoIP applications. In Proceedings of the 9th International Workshop on Quality of Multimedia Experience (QoMEX’17). IEEE, 1--6.Google Scholar
- R. Gupta, S. Arndt, J.-N. Antons, R. Schleicher, S. Möller, and T. H. Falk. 2013. Neurophysiological experimental facility for quality of experience (QoE) assessment. In Proceedings of the International Symposium on Integrated Network Management (IM’13). IEEE, 1300--1305.Google Scholar
- R. Gupta, H. J. Banville, and T. H. Falk. 2017. Multimodal physiological quality-of-experience assessment of text-to-speech systems. IEEE J. Select. Top. Signal Process. 11, 1 (2017), 22--36.Google ScholarCross Ref
- J. Hadden, A. Tiwari, R. Roy, and D. Ruta. 2007. Computer assisted customer churn management: State-of-the-art and future trends. Comput. Oper. Res. 34, 10 (2007), 2902--2917. Google ScholarDigital Library
- A. Hamam, A. E. Saddik, and J. Alja’am. 2014. A quality of experience model for haptic virtual environments. ACM Trans. Multimedia Comput. Commun. Appl. 10, 3, Article 28 (April 2014), 23 pages. Google ScholarDigital Library
- B. Han, V. Gopalakrishnan, L. Ji, and S. Lee. 2015. Network function virtualization: Challenges and opportunities for innovations. IEEE Commun. Mag. 53, 2 (2015), 90--97.Google ScholarDigital Library
- Heavy Reading, S. O’Boyle, and Scott Sumner. Nov. 2016. Webinar: The Future of QoE Monitoring in Virtualized Networks. https://www.youtube.com/watch?v=6MFSlEOiqvo.Google Scholar
- P. E. Heegaard, G. Biczok, and L. Toka. 2016. Sharing is power: Incentives for information exchange in multi-operator service delivery. In Proceedings of the 2016 IEEE Global Communications Conference (GLOBECOM). 1--7.Google Scholar
- C. T. Hewage and M. G. Martini. 2013. Quality of experience for 3d video streaming. IEEE Commun. Mag. 51, 5 (2013), 101--107.Google ScholarCross Ref
- T. Hoßfeld, S. Biedermann, R. Schatz, A. Platzer, S. Egger, and M. Fiedler. 2011. The memory effect and its implications on web QoE modeling. In Proceedings of the 23rd International Teletraffic Congress (ITC’11). IEEE, 103--110. Google ScholarDigital Library
- T. Hoßfeld, M. Fiedler, and T. Zinner. 2011. The QoE provisioning-delivery-hysteresis and its importance for service provisioning in the future internet. In Proceedings of the 7th EURO-NGI Conference on Next Generation Internet (NGI’11). IEEE, 1--8.Google Scholar
- T. Hoßfeld, M. Hirth, P. Korshunov, P. Hanhart, B. Gardlo, C. Keimel, and C. Timmerer. 2014. Survey of web-based crowdsourcing frameworks for subjective quality assessment. In Proceedings of the 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP’14). 1--6.Google Scholar
- T. Hoßfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia. 2014. Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans. Multimedia 16, 2 (2014), 541--558. Google ScholarDigital Library
- T. Hoßfeld, C. Moldovan, and C. Schwartz. 2015. To each according to his needs: Dimensioning video buffer for specific user profiles and behavior. In Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM’15).Google Scholar
- T. Hoßfeld, R. Schatz, M. Varela, and C. Timmerer. 2012. Challenges of QoE management for cloud applications. IEEE Commun. Mag. 50, 4 (2012).Google Scholar
- T. Hoßfeld, L. Skorin-Kapov, Y. Haddad, P. Pocta, V. A. Siris, A. Zgank, and H. Melvin. 2015. Can context monitoring improve QoE? A case study of video flash crowds in the internet of services. In Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM’15). 1274--1277.Google Scholar
- Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young. 2015. Mobile edge computing—A key technology toward 5G. ETSI White Paper 11, 11 (2015), 1--16.Google Scholar
- International Telecommunication Union. 2016. ITU-T Recommendation P.1203: Parametric Bitstream-based Quality Assessment of Progressive Download and Adaptive Audiovisual Streaming Services Over Reliable Transport. Technical Report.Google Scholar
- K. Ivesic, L. Skorin-Kapov, and M. Matijasevic. 2014. Cross-layer QoE-driven admission control and resource allocation for adaptive multimedia services in LTE. J. Netw. Comput. Appl. 46 (2014), 336--351. Google ScholarDigital Library
- L. Jalal and M. Murroni. 2017. A nonlinear quality of experience model for high dynamic spatio-temporal mulsemedia. In Proceedings of the 2017 9th International Conference on Quality of Multimedia Experience (QoMEX’17). IEEE, 1--6.Google Scholar
- M. Jarschel, F. Wamser, T. Hohn, T. Zinner, and P. Tran-Gia. 2013. SDN-based application-aware networking on the example of youtube video streaming. In Proceedings of the 2013 2nd European Workshop on SDN (EWSDN’13). IEEE, 87--92. Google ScholarDigital Library
- J. Jiang, S. Sun, V. Sekar, and H. Zhang. 2017. Pytheas: Enabling data-driven quality of experience optimization using group-based exploration-exploitation. In Proceedings of the14th USENIX Symposium on Networked Systems Design and Implementation. Google ScholarDigital Library
- A. Kassler, L. Skorin-Kapov, O. Dobrijevic, M. Matijasevic, and P. Dely. 2012. Towards QoE-driven multimedia service negotiation and path optimization with software defined networking. In Proceedings of the 20th International Conference on Software, Telecommunications and Computer Networks (SoftCOM’12). IEEE, 1--5.Google Scholar
- M. Katsarakis, R. Teixeira, M. Papadopouli, and V. Christophides. 2016. Towards a causal analysis of video QoE from network and application QoS. In Proceedings of ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE’16). ACM, 1--6. Google ScholarCross Ref
- N. Khan and M. G. Martini. 2016. QoE-driven multi-user scheduling and rate adaptation with reduced cross-layer signaling for scalable video streaming over LTE wireless systems. EURASIP J. Wireless Commun. Network. 2016, 1 (2016), 93.Google ScholarCross Ref
- C. Kim, A. Sivaraman, N. Katta, A. Bas, A. Dixit, and L. J. Wobker. 2015. In-band network telemetry via programmable dataplanes. In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM’15).Google Scholar
- E. Kroupi, P. Hanhart, J.-S. Lee, M. Rerabek, and T. Ebrahimi. 2014. EEG correlates during video quality perception. In Proceedings of the 22nd European Signal Processing Conference (EUSIPCO’14). IEEE, 2135--2139.Google Scholar
- P. Le Callet, S. Möller, and A. Perkis. 2013. Qualinet White Paper on Definitions of Quality of Experience. Technical Report Version 1.2. European Network on Quality of Experience in Multimedia Systems and Services.Google Scholar
- P. R. Lebreton, T. Mäki, E. Skodras, I. Hupont, and M. Hirth. 2015. Bridging the gap between eye tracking and crowdsourcing. In SPIE/IS8T Electronic Imaging. International Society for Optics and Photonics, 93940W--93940W.Google Scholar
- J.-S. Lee. 2014. On designing paired comparison experiments for subjective multimedia quality assessment. IEEE Trans. Multimedia 16, 2 (2014), 564--571. Google ScholarDigital Library
- E. Liotou, G. Tseliou, K. Samdanis, D. Tsolkas, F. Adelantado, and C. Verikoukis. 2015. An SDN QoE-service for dynamically enhancing the performance of OTT applications. In Proceedings of the 2015 7th International Workshop on Quality of Multimedia Experience (QoMEX’15). 1--2.Google Scholar
- E. Liotou, D. Tsolkas, and N. Passas. 2016. A roadmap on QoE metrics and models. In Proceedings of the 2016 23rd International Conference on Telecommunications (ICT’16). IEEE, 1--5.Google Scholar
- C. Z. Liu and M. Kavakli. 2016. Data-aware QoE-QoS management. In Proceedings of the 2016 IEEE 11th Conference on Industrial Electronics and Applications (ICIEA’16). IEEE, 1818--1823.Google Scholar
- E. Maggioni, E. Agostinelli, and M. Obrist. 2017. Measuring the added value of haptic feedback. In Proceedings of the 2017 9th International Conference on Quality of Multimedia Experience (QoMEX’17). IEEE, 1--6.Google Scholar
- T. Mäki, P. Zwickl, and M. Varela. 2016. Network quality differentiation: Regional effects, market entrance, and empirical testability. In Proceedings of the 2016 International Federation for Information Processing Networking Conference and Workshops. 476--484.Google Scholar
- M. G. Martini, C. W. Chen, Z. Chen, T. Dagiuklas, L. Sun, and X. Zhu. 2012. Guest editorial QoE-aware wireless multimedia systems. IEEE J. Select. Areas Commun. 30, 7 (2012), 1153--1156.Google ScholarCross Ref
- A. M. Medhat, T. Taleb, A. Elmangoush, G. A. Carella, S. Covaci, and T. Magedanz. 2017. Service function chaining in next generation networks: State of the art and research challenges. IEEE Commun. Mag. 55, 2 (2017), 216--223. Google ScholarDigital Library
- V. Menkovski, G. Exarchakos, and A. Liotta. 2011. Tackling the sheer scale of subjective QoE. In Proceedings of the International Conference on Mobile Multimedia Communications. Springer, 1--15.Google Scholar
- V. Menkovski and A. Liotta. 2012. Adaptive psychometric scaling for video quality assessment. Signal Process.: Image Commun. 27, 8 (2012), 788--799. Google ScholarDigital Library
- K. Mitra, A. Zaslavsky, and C. Åhlund. 2015. Context-aware QoE modelling, measurement, and prediction in mobile computing systems. IEEE Trans. Mobile Comput. 14, 5 (2015), 920--936.Google ScholarDigital Library
- H. Moens and F. De Turck. 2014. VNF-P: A model for efficient placement of virtualized network functions. In Proceedings of the 2014 10th International. Conference on Network and Service Management (CNSM’14). IEEE, 418--423.Google Scholar
- S. Möller, M. Wältermann, and M.-N. Garcia. 2014. Features of quality of experience. In Quality of Experience. Springer, 73--84.Google Scholar
- N. Murray, O. A. Ademoye, G. Ghinea, Y. Qiao, G.-M. Muntean, and B. Lee. 2017. Olfactory-enhanced multimedia video clips datasets. In Proceedings of the 2017 9th International Conference on Quality of Multimedia Experience (QoMEX’17). IEEE, 1--5.Google Scholar
- N. Murray, Y. Qiao, B. Lee, A. K. Karunakar, and G.-M. Muntean. 2013. Subjective evaluation of olfactory and visual media synchronization. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys’13). ACM, 162--171. Google ScholarDigital Library
- H. Nam, K.-H. Kim, D. Calin, and H. Schulzrinne. 2014. Youslow: A performance analysis tool for adaptive bitrate video streaming. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 111--112. Google ScholarDigital Library
- H. Nam, K.-H. Kim, J. Y. Kim, and H. Schulzrinne. 2014. Towards QoE-aware video streaming using SDN. In Proceedings of the Global Communications Conference (GLOBECOM’14). IEEE, 1317--1322.Google Scholar
- I. Orsolic, D. Pevec, M. Suznjevic, and L. Skorin-Kapov. 2017. A machine-learning approach to classifying YouTube QoE based on encrypted network traffic. Multimedia Tools Appl. 1--35. Google ScholarDigital Library
- B. Osting, J. Xiong, Q. Xu, and Y. Yao. 2016. Analysis of crowdsourced sampling strategies for hodgerank with sparse random graphs. Appl. Comput. Harmon. Anal. 41, 2 (2016), 540--560.Google ScholarCross Ref
- M. Pallot, R. Eynard, B. Poussard, O. Christmann, and S. Richir. 2013. Augmented sport : Exploring collective user experience. In Proceedings of the Virtual Reality International Conference (VRIC’13). Laval, France, 1--10. Google ScholarDigital Library
- S. Peng and others. 2017. QoE-oriented mobile edge service management leveraging SDN and NFV. Mobile Info. Syst. 2017.Google Scholar
- D. Perritaz, C. Salzmann, and D. Gillet. 2009. Quality of experience for adaptation in augmented reality. In Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics. 888--893. Google ScholarDigital Library
- J. Puig, A. Perkis, F. Lindseth, and T. Ebrahimi. 2012. Towards an efficient methodology for evaluation of quality of experience in augmented reality. In Proceedings of the 2012 4th International Workshop on Quality of Multimedia Experience. 188--193.Google Scholar
- Q. M. Qadir, A. A. Kist, and Z. Zhang. 2015. Mechanisms for QoE optimisation of video traffic: A review paper. Australasian J. Info., Commun. Technol. Appl. 1 (2015). Issue 1.Google Scholar
- R. Rawassizadeh, B. A. Price, and M. Petre. 2015. Wearables: Has the age of smartwatches finally arrived? Commun. ACM 58, 1 (2015), 45--47. Google ScholarDigital Library
- J. Redi, E. Siahaan, P. Korshunov, J. Habigt, and T. Hoßfeld. 2015. When the crowd challenges the lab: Lessons learnt from subjective studies on image aesthetic appeal. In Proceedings of the 4th International Workshop on Crowdsourcing for Multimedia. ACM, 33--38. Google ScholarDigital Library
- P. Reichl, S. Egger, S. Möller, K. Kilkki, M. Fiedler, T. Hoßfeld, C. Tsiaras, and A. Asrese. 2015. Towards a comprehensive framework for QoE and user behavior modelling. In Proceedings of the 2015 7th International Workshop on Quality of Multimedia Experience (QoMEX’15). IEEE, 1--6.Google Scholar
- P. Reichl, S. Egger, R. Schatz, and A. D’Alconzo. 2010. The logarithmic nature of QoE and the role of the Weber-Fechner law in QoE assessment. In Proceedings of the 2010 IEEE International Conference on Communications (ICC’10). IEEE, 1--5.Google Scholar
- P. Reichl, P. Maillé, P. Zwickl, and A. Sackl. 2013. A fixed-point model for QoE-based charging. In Proceedings of the 2013 ACM SIGCOMM Workshop on Future Human-centric Multimedia Networking. ACM, 33--38. Google ScholarDigital Library
- U. Reiter, K. Brunnström, K. De Moor, M.-C. Larabi, M. Pereira, A. Pinheiro, J. You, and A. Zgank. 2014. Factors influencing quality of experience. In Quality of Experience. Springer, 55--72.Google Scholar
- W. Robitza, A. Ahmad, P. A. Kara, L. Atzori, M. G. Martini, A. Raake, and L. Sun. 2017. Challenges of future multimedia QoE monitoring for internet service providers. Multimedia Tools Appl. (2017), 1--24. Google ScholarDigital Library
- W. Robitza, S. Schönfellner, and A. Raake. 2016. A theoretical approach to the formation of quality of experience and user behavior in multimedia services. In Proceedings of the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems. 39--43.Google Scholar
- D. Z. Rodríguez, R. L. Rosa, E. A. Costa, J. Abrahão, and G. Bressan. 2014. Video quality assessment in video streaming services considering user preference for video content. IEEE Trans. Consum. Electron. 60, 3 (2014), 436--444.Google ScholarCross Ref
- G. Rubino. 2005. Quantifying the quality of audio and video transmissions over the internet: The PSQA approach. Design and Operations of Communication Networks: A Review of Wired and Wireless Modeling and Management Challenges. Imperial College Press, London (2005).Google Scholar
- G. Rubino, P. Tirilly, and M. Varela. 2006. Evaluating users’ satisfaction in packet networks using random neural networks. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 303--312. Google ScholarDigital Library
- R. Schatz, M. Fiedler, and L. Skorin-Kapov. 2014. QoE-Based Network and Application Management, in Quality of Experience: Advanced Concepts, Applications and Methods. Springer International Publishing, 411--426.Google Scholar
- R. Schatz, T. Hoßfeld, L. Janowski, and S. Egger. 2013. From packets to people: Quality of experience as a new measurement challenge. In Data Traffic Monitoring and Analysis. Springer, 219--263. Google ScholarDigital Library
- R. Schatz, A. Sackl, C. Timmerer, and B. Gardlo. 2017. Towards subjective quality of experience assessment for omnidirectional video streaming. In 2017 9th International Conference on Quality of Multimedia Experience (QoMEX’17). IEEE, 1--6.Google Scholar
- S. Schwarzmann, T. Zinner, and O. Dobrijevic. 2016. Towards a framework for comparing application-network interaction mechanisms. In Proceedings of the 28th International Teletraffic Congress (ITC’16), Vol. 3. IEEE, 13--18.Google Scholar
- S. Schwarzmann, T. Zinner, and O. Dobrijevic. 2017. Quantitative comparison of application--network interaction: A case study of adaptive video streaming. Qual. User Exper. 2, 1 (2017), 7.Google ScholarCross Ref
- B. Settles. 2010. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.Google Scholar
- M. Seufert, S. Egger, M. Slanina, T. Zinner, T. Hoßfeld, and P. Tran-Gia. 2015. A survey on quality of experience of HTTP adaptive streaming. IEEE Commun. Surveys Tutor. 17, 1 (2015), 469--492.Google ScholarDigital Library
- M. Seufert and T. Hoßfeld. 2016. One shot crowdtesting: Approaching the extremes of crowdsourced subjective quality testing. In Proceedings of the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS’16). 122--126.Google Scholar
- M. Seufert, N. Wehner, F. Wamser, P. Casas, A. D’Alconzo, and P. Tran-Gia. 2017. Unsupervised QoE field study for mobile YouTube video streaming with YoMoApp. In Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX’17).Google Scholar
- M. Seufert, O. Zach, T. Hoßfeld, M. Slanina, and P. Tran-Gia. 2016. Impact of test condition selection in adaptive crowdsourcing studies on subjective quality. In Proceedings of the 8th International Conference on Quality of Multimedia Experience (QoMEX’16). IEEE.Google Scholar
- M. Z. Shafiq. 2015. Tracking mobile video QoE in the encrypted internet. White-paper, University of Iowa (2015).Google Scholar
- M. Z. Shafiq, J. Erman, L. Ji, A. X. Liu, J. Pang, and J. Wang. 2014. Understanding the impact of network dynamics on mobile video user engagement. In ACM SIGMETRICS Performance Evaluation Review, Vol. 42. ACM, 367--379. Google ScholarDigital Library
- L. Skorin-Kapov and M. Varela. 2012. A multi-dimensional view of QoE: The ARCU model. In Proceedings of the 35th International MIPRO Convention. IEEE, 662--666.Google Scholar
- I. Sousa, M. P. Queluz, and A. Rodrigues. 2017. A survey on QoE-oriented wireless resources scheduling. CoRR abs/1705.07839 (2017).Google Scholar
- W. Stallings. 2015. Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud. Addison-Wesley Professional.Google Scholar
- Y. Sulema. 2016. Mulsemedia vs. multimedia: State of the art and future trends. In Proceedings of the 2016 International Conference on Systems, Signals and Image Processing (IWSSIP’16). IEEE, 1--5.Google ScholarCross Ref
- The 5G Infrastructure Public Private Partnership. 5G Vision Brochure (Feb. 2015).Google Scholar
- C. Timmerer and B. Rainer. 2014. The social multimedia experience. Computer 47, 3 (2014), 67--69. Google ScholarDigital Library
- C. Timmerer, M. Waltl, B. Rainer, and N. Murray. 2014. Sensory Experience: Quality of Experience Beyond Audio-Visual. Springer International Publishing, Cham, 351--365.Google Scholar
- C. Tselios and G. Tsolis. 2016. On QoE-awareness through virtualized probes in 5G networks. In Proceedings of the IEEE 21st International Workshop on Computer Aided Modelling and Design of Communication Links and Networks. IEEE, 159--164.Google Scholar
- D. Tsolkas, E. Liotou, N. Passas, and L. Merakos. 2017. A survey on parametric QoE estimation for popular services. J. Netw. Comput. Appl. 77 (2017), 1--17. Google ScholarDigital Library
- M. Varela, T. Mäki, L. Skorin-Kapov, and T. Hoßfeld. 2013. Increasing payments in crowdsourcing: Don’t look a gift horse in the mouth! In Proceedings of the 4th International Workshop on Perceptual Quality of Systems (PQS’13). Vienna, Austria.Google Scholar
- M. Varela, P. Zwickl, P. Reichl, M. Xie, and H. Schulzrinne. 2015. From service level agreements (SLA) to experience level agreements (ELA): The challenges of selling QoE to the user. In Proceedings of the 2015 IEEE International Conference on Communication Workshop (ICCW’15). IEEE, 1741--1746.Google Scholar
- Verizon, Cisco, Ericsson, HPE, Nokia, Intel, R. Hat, and Samsung. 2016. SDN-NFV Reference Architecture v1.0. (2016).Google Scholar
- F. Wamser, M. Seufert, P. Casas, R. Irmer, P. Tran-Gia, and R. Schatz. 2015. YoMoApp: A tool for analyzing QoE of YouTube HTTP adaptive streaming in mobile networks. In Proceedings of the European Conference on Networks and Communications (EuCNC’15). IEEE, 239--243.Google Scholar
- Y. Wang, P. Li, L. Jiao, Z. Su, N. Cheng, X. S. Shen, and P. Zhang. 2017. A data-driven architecture for personalized QoE management in 5G wireless networks. IEEE Wireless Commun. 24, 1 (2017), 102--110. Google ScholarDigital Library
- M. K. Weldon. 2016. The Future X Network: A Bell Labs Perspective. CRC Press. Google ScholarDigital Library
- C.-C. Wu, K.-T. Chen, Y.-C. Chang, and C.-L. Lei. 2013. Crowdsourcing multimedia QoE evaluation: A trusted framework. IEEE Trans. Multimedia 15, 5 (2013), 1121--1137. Google ScholarDigital Library
- W. Wu, A. Arefin, Z. Huang, P. Agarwal, S. Shi, R. Rivas, and K. Nahrstedt. 2010. “I’m the Jedi&excel;” - A case study of user experience in 3d tele-immersive gaming. In Proceedings of the 2010 IEEE International Symposium on Multimedia. 220--227. Google ScholarDigital Library
- L. Xing, J. You, T. Ebrahimi, and A. Perkis. 2010. Estimating quality of experience on stereoscopic images. In Proceedings of the 2010 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS’10). IEEE, 1--4.Google Scholar
- P. Ye and D. Doermann. 2014. Active sampling for subjective image quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4249--4256. Google ScholarDigital Library
- Y. B. Youssef, A. Mellouk, M. Afif, and S. Tabbane. 2016. Video quality assessment based on statistical selection approach for QoE factors dependency. In Proceedings of the Global Communications Conference (GLOBECOM’16). IEEE, 1--6.Google Scholar
- Z. Yuan, S. Chen, G. Ghinea, and G.-M. Muntean. 2014. User quality of experience of multimedia applications. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 15 (Oct. 2014), 19 pages. Google ScholarDigital Library
- Z. Yuan, G. Ghinea, and G.-M. Muntean. 2014. Quality of experience study for multiple sensorial media delivery. In Proceedings of the 2014 International Wireless Communications and Mobile Computing Conference (IWCMC’14). 1142--1146.Google ScholarCross Ref
- T. Zhao, Q. Liu, and C. W. Chen. 2017. QoE in video transmission: A user experience-driven strategy. IEEE Commun. Surveys Tutor. (2017), 285--302.Google Scholar
- K. Zheng, Z. Yang, K. Zhang, P. Chatzimisios, K. Yang, and W. Xiang. 2016. Big data-driven optimization for mobile networks toward 5G. IEEE Network 30, 1 (2016), 44--51.Google ScholarDigital Library
- T. Zinner, M. Jarschel, A. Blenk, F. Wamser, and W. Kellerer. 2014. Dynamic application-aware resource management using software-defined networking: Implementation prospects and challenges. In Proceedings of the IEEE Network Operations and Management Symposium (NOMS’14). 1--6.Google Scholar
Index Terms
- A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services
Recommendations
QoE-aware QoS management
MoMM '08: Proceedings of the 6th International Conference on Advances in Mobile Computing and MultimediaThe streaming of multimedia contents (e.g., Mobile TV) is a bandwidth intensive service. The network operator's aim is to provide an acceptable user experience at minimal network resource usage. It is important from the network operator's perspective to ...
SDN and NFV for QoE-driven multimedia services delivery: The road towards 6G and beyond networks
AbstractThe concept for developing the future mobile communication systems has been explored in recent years with a vision towards 5G and 6G systems. Meeting performance targets such as higher data rate transmission, lower end-to-end latency, ...
QoE-Aware OTT-ISP Collaboration in Service Management: Architecture and Approaches
It is a matter of fact that quality of experience (QoE) has become one of the key factors determining whether a new multimedia service will be successfully accepted by the final users. Accordingly, several QoE models have been developed with the aim of ...
Comments