Gheorghita Ghinea
Christian Timmerer
Weisi Lin
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Mulsemedia—multiple sensorial media—captures a wide variety of research efforts and applications. This article presents a historic perspective on mulsemedia work and reviews current developments in the area. These take place across the traditional multimedia spectrum—from virtual reality applications to computer games—as well as efforts in the arts, gastronomy, and therapy, to mention a few. We also describe standardization efforts, via the MPEG-V standard, and identify future developments and exciting challenges the community needs to overcome.
Supplemental Material
Available for Download
zip
ghinea.zip (40.6 KB)
Supplemental movie, appendix, image and software files for, Mulsemedia: State of the Art, Perspectives, and Challenges
- E. Aarts and B. de Ruyter. 2009. New research perspectives on ambient intelligence. J. Ambient Intell. Smart Environ. 1, 1, 5--14. Google Scholar
Digital Library
- O. Ademoye and G. Ghinea. 2009. Synchronization of olfaction-enhanced multimedia. IEEE Trans. Multimedia 11, 3, 561--565. Google ScholarDigital Library
- J. R. Anderson. 2004. Cognitive Psychology and Its Implications 6th Ed. Worth Publishers.Google Scholar
- J. G. Apostolopoulos, P. A. Chou, B. Culbertson, T. Kalker, M. D. Trott, and S. Wee. 2012. The road to immersive communication. Proc. IEEE 100, 4, 974--990.Google ScholarCross Ref
- S. Ayabe--Kanamura, I. Schicker, M. Laska, R. Hudson, H. Distel, T. Koboyakawa, and S. Saito. 1998. A Japanese-German cross-cultural study. Chem. Sens., 23, 1, 31--38.Google ScholarCross Ref
- A. Bodnar, R. Corbett, and D. Nekrasovski. 2004. AROMA: Ambient awareness through olfaction in a messaging application: Does olfactory notification make ‘scents’? In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI'04). 183--190. Google ScholarDigital Library
- S. Boyd-Davis, G. Davies, R. Haddad, and M. Lai. 2006. Smell Me: Engaging with an interactive olfactory game. In Proceedings 25th Annual Meeting of the Human Factors and Ergonomics Society, 25--40.Google Scholar
- S. A. Brewster, D. K. Mcgookin, and C. A. Miller. 2006. Olfoto: Designing a smell-based interaction. In Proceedings of the Conference on Human Factors in Computing Systems. 653--662. Google ScholarDigital Library
- D. Campbell, E. Jones, and M. Glavin. 2009. Audio quality assessment techniques—A review, and recent developments. Signal Process., 89, 8, 1489--1500. Google ScholarDigital Library
- C.-C. Carbon and M. Jakesch. 2013. A model for haptic aesthetic processing and its implications for design. Proc. IEEE. 101, 9, 2123--2133.Google ScholarCross Ref
- J. P. Cater. 1992. The nose have it! Letters to the editor. Presence 1, 4, 493--494.Google Scholar
- A. Chang and C. O'Sullivan. 2005. Audio-haptic feedback in mobile phones. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI EA'05). ACM, New York, NY, 2005, 1264--1267. Google ScholarDigital Library
- A. D. Craig. 2003. Interoception: The sense of the physiological condition of the body. Curr. Opinion Neurobiol. 13, 4, 500--505.Google ScholarCross Ref
- A. R. Damasio. 1989. Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition 33, 1--2, 25--62.Google ScholarCross Ref
- B. de Ruyter and E. Aarts. 2004. Ambient intelligence: Visualizing the future. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI'04). ACM Press, New York, NY, 203--208. Google ScholarDigital Library
- P. Dimaggio. 1997. Culture and cognition. Ann. Rev. Sociol. 23, 263--287.Google ScholarCross Ref
- H. Q. Dinh, N. Walker, L. F. Hodges, C. Song, and A. Kobayashi. 1999. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of the Virtual Reality Annual International Symposium. 222--228. Google ScholarDigital Library
- C. Fadel and C. Lemke. 2008. Multimodal learning through media: What the research says. CISCO Systems Report.Google Scholar
- G. Ghinea and O. Ademoye. 2010a. Perceived synchronization of olfactory multimedia. IEEE Trans. Syst. Man Cybernet.-Part A 40, 4, 657--663. Google ScholarDigital Library
- G. Ghinea and O. Ademoye. 2010b. A user perspective of olfaction-enhanced mulsemedia. In Proceedings of the International Conference on Management of Emergent Digital EcoSystems (MEDES'10). 277--280. Google ScholarDigital Library
- G. Ghinea and O. Ademoye. 2011. Olfaction-enhanced multimedia: Perspectives and challenges. Multimedia Tools Appl. 55, 3, 601--626. Google ScholarDigital Library
- G. Ghinea and O. Ademoye. 2012. The sweet smell of success: Enhancing multimedia applications with olfaction. ACM Trans. Multimedia Comput. Commun. Appl. 8, 1, 2. Google ScholarDigital Library
- E. B. Goldstein. 2013. Sensation and Perception. Cengage Learning.Google Scholar
- R. Gray, C. Spence, C. Ho, and H. Z. Tan. 2013. Efficient multimodal cuing of spatial attention Proc. IEEE 101, 9, 2113--2121.Google ScholarCross Ref
- M. Grega, L. Janowski, M. Leszczuk, P. Romaniak, and Z. Papir. 2008. Quality of experience evaluation for multimedia services - Szacowanie postrzeganej jako sci uslug (QoE) komunikacji multimedialnej. Przegla d Telekomunika- cyjny 81, 4, 142--153.Google Scholar
- S. Gumtau. 2011. Affordances of touch in multisensory embodied interface design. Ph.D., Dissertation University of Portsmouth, U.K.Google Scholar
- D. Hands. 2004. A basic multimedia quality model. IEEE Trans. Multimedia 6, 6, 806--816. Google ScholarDigital Library
- M. L. Heilig. 1962. Sensorama simulator. United States Patent 3,050,870. Filed January 10, 1961, Patented August 28, 1962.Google Scholar
- P. Hinterseer and E. Steinbach. 2006. A psychophysically motivated compression approach for 3D haptic data. In Proceedings of the 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 35--41. Google ScholarDigital Library
- C. Ho and C. Spence. 2005. Olfactory facilitation of dual-task performance. Neurosci. Lett. 389, 1, 35--40.Google Scholar
- T. Hoß feld, D. Hock, P. Tran-Gia, K. Tutschku, and M. Fiedler. 2008. Testing the IQX hypothesis for exponential interdependency between QoS and QoE of voice codecs iLBC and G.711. In Proceedings of the 18th ITC Specialist Seminar on Quality of Experience.Google Scholar
- Y. Ishibashi, T. Kanbara, and S. Tasaka. 2004. Inter-stream synchronization between haptic media and voice in collaborative virtual environments. In Proceedings of the 12th Annual ACM International Conference on Multimedia. ACM, New York, NY, 604--611. Google ScholarDigital Library
- L. Itti, C. Koch, and E. Niebur. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Patt. Anal. Mach. Intell. 20, 11, 1254-9. Google ScholarDigital Library
- ITU. 2008a. Subjective video quality assessment methods for multimedia applications. ITU-T Rec. P.910.Google Scholar
- ITU. 2008b. Subjective audiovisual quality assessment methods for multimedia applications. ITU-T Rec. P.911.Google Scholar
- R. Jain. 2003. Experiential computing. Commun. ACM 46, 7, 48--55. Google ScholarDigital Library
- N. Jayant, J. Johnston, and R. Safranek. 1993. Signal compression based on models of human perception. Proc. IEEE 81, 10, 1385--1422.Google ScholarCross Ref
- L. Jones, C. A. Bowers, D. Washburn, A. Cortes, and R. V. Satya. 2004. The effect of olfaction on immersion into virtual environments. In Human Performance, Situation Awareness and Automation: Issues and Considerations for the 21st Century. Lawrence Erlbaum Associates, 282--285.Google Scholar
- K. Kahol, P. Tripathi, T. Mcdaniel, L. Bratton, and S. Panchanathan. 2006. Modeling context in haptic perception, rendering, and visualization. ACM Trans. Multimedia Comput. Commun. Appl. 2, 3, 219--240. Google ScholarDigital Library
- J. Kammerl, I. Vittorias, V. Nitsch, B. Faerber, E. Steinbach, and S. Hirche. 2010. Perception-based data reduction for haptic force-feedback signals using adaptive deadbands. Presence, Teleoper. Virtual Environ. 19, 5, 450--462. Google ScholarDigital Library
- D. Kahneman. 2003. A perspective on judgement and choice. Am. Psychol. 58, 9, 697--720.Google ScholarCross Ref
- J. N. Kaye. 2001. Symbolic olfactory display. Master's Thesis, Massachusetts Institute of Technology, MA. http://www.media.mit.edu/∼jofish/thesis/.Google Scholar
- H. Kim, H.-J. Kwon, and K.-S. Hong. 2010. Location awareness-based intelligent multi-agent technology. Multimedia Syst. 16, (4--5), 275--292.Google Scholar
- R. L. Klatzky, D. Pawluk, and A. Peer. 2013. Haptic perception of material properties and implications for applications. Proc. IEEE 101, 9, 2081--2092.Google Scholar
- P. Le Callet, S. Möller, and A. Perkis. (Eds). 2013. Qualinet white paper on definitions of quality of experience. White paper, European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). Lausanne, Switzerland, Version 1.2.Google Scholar
- W. Lin. 2006. Computational models for just-noticeable difference. In Digital Video Image Quality and Perceptual Coding, H. R. Wu and K. R. Rao, (Eds.), CRC Press, Chapter 9.Google Scholar
- W. Lin and C.-C. Jay Kuo. 2011. Perceptual visual quality metrics: A survey. J. Visual Commun. Image Represent. 22, 4, 297--312. Google ScholarDigital Library
- K. Liu and S. R. Gulliver. 2013. Semiotics in building space for working and living. In Intelligent Building: Design, Management and Operation, D. Clements-Croome, (Ed.), ICE Publishing.Google Scholar
- Z. Lu, W. Lin, X. Yang, E. Ong, and S. Yao. 2005. Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation. IEEE Trans. Image Process., 14, 11, 1928--1942. Google ScholarDigital Library
- Y.-F. Ma, X.-S. Hua, L. Lu, and H.-J. Zhang. 2005. A generic framework of user attention model and its application in video summarization. IEEE Trans. Multimedia. 7, 5, 907--919. Google ScholarDigital Library
- R. E. Mayer. 2003. Elements of a science of e-learning. J. Edu. Comput. Res. 29, 3, 297--313.Google ScholarCross Ref
- B. S. Manjunath, P. Salembier, and T. Sikora. 2002. Introduction to MPEG-7: Multimedia Content Description Interface. John Wiley 8. Sons Inc. Google ScholarDigital Library
- R. Marois and J. Ivanoff. 2005. Capacity limits of information processing in the brain. Trends Cognitive Sci., 9, 6, 296--305.Google Scholar
- T. Metzinger. 1995. Faster than thought: Holism, homogeneity and temporal coding. In Conscious Experience T. Metzinger (Ed.), Imprint Academic, 425--461.Google Scholar
- A. Mochizuki, T. Amada, S. Sawa, T. Takeda, S. Motoyashiki, K. Kohyama, M. Imura, and K. Chihara. 2004. Fragra: A visual-olfactory VR game. In Proceedings of the ACM SIGGRAPH (SIGGRAPH'04). ACM Press, New York, NY, 123. Google ScholarDigital Library
- S. Möller, W.-Y. Chan, N. Côté, T. H. Falk, A. Raake, and M. Wältermann. 2011. Speech quality estimation: Models and trends. IEEE Signal Process. Mag. 28, 6, 18--28.Google ScholarCross Ref
- G. Morrot, F. Brochet, and D. Dubourdieu. 2001. The color of odors. Brain Lang. 79, 2, 309--320.Google ScholarCross Ref
- M. Narwaria and W. Lin. 2012. SVD-based quality metric for image and video using machine learning. IEEE Trans. Syst. Man Cybernet. Part B 42, 2, 347--364. Google ScholarDigital Library
- M. Narwaria, W. Lin, I. Mcloughlin, S. Emmanue, and L. T. Chia. 2012. Nonintrusive quality assessment of noise suppressed speech with mel-filtered energies and support vector regression. IEEE Trans. Audio Speech Lang. Process. 20, 4, 1217--1232. Google ScholarDigital Library
- T. Nakamoto, S. Otaguro, M. Kinoshita, M. Nagahama, K. Ohinishi, and T. Ishida. 2008. Cooking up an interactive olfactory game display. IEEE Comput. Graph. Appl. 28, 1, 75--78. Google ScholarDigital Library
- H.-C. Nothdurft. 2000. Salience from feature contrast: Additivity across dimensions. Vis. Res. 40, 10--12, 1183--1201.Google Scholar
- M. A. Otaduy, C. Garre, and M. C. Lin. 2013. Representations and algorithms for force-feedback display. Proc. IEEE 101, 9, 2068--2080.Google ScholarCross Ref
- F. Pereira. 2005. A triple user characterization model for video adaptation and quality of experience evaluation. In Proceedings of the 7th IEEE Workshop on Multimedia Signal Processing. 1--4.Google Scholar
- S. Pyo, S. Joo, B. Choi, M. Kim, and J. Kim. 2008. A metadata schema design on representation of sensory effect information for sensible media and its service framework using UPnP. In Proceedings of the 10th International Conference on Advanced Communication Technology (ICACT'08). 1129--1134.Google Scholar
- B. Rainer, M. Waltl, E. Cheng, M. Shujau, C. Timmerer, S. Davis, I. Burnett, C. Ritz, and H. Hellwagner. 2012. Investigating the impact of sensory effects on the quality of experience and emotional response in Web videos. In Proceedings of the 4th International Workshop on Quality of Multimedia Experience (QoMEX'12). IEEE, 278--283.Google Scholar
- E. Reinhard, A. A. Efros, J. Kautz, and H.-P. Seidel. 2013. On visual realism of synthesized imagery. Proc. IEEE 101, 9, 1998--2007.Google ScholarCross Ref
- A. Revonsuo. 1999. Binding and the phenomenal unity of consciousness. Consciou. Cognition 8, 2, 173--185.Google Scholar
- G. Richard, S. Sundaram, and S. Narayanan. 2013. An overview on perceptually motivated audio indexing and classification. Proc. IEEE 101, 9, 1939--1954.Google ScholarCross Ref
- L. A. Rowe and R. Jain. 2005. ACM SIGMM retreat report on future directions in multimedia research. ACM Trans. Multimedia Comput. Commun. Appl. 1, 1, 3--13. Google ScholarDigital Library
- J. S. Rubinstein, D. E. Meyer, and J. E. Evans. 2001. Executive control of cognitive processes in task switching. J. Exp. Psychol. Human Percep. Perform. 27, 4, 763.Google ScholarCross Ref
- N. Sarter. 2013. Multimodal support for interruption management: Models, empirical findings, and design recommendations. Proc. IEEE 101, 9, 2105--2112.Google ScholarCross Ref
- P. H. Schiller. 1986. The central visual system. Vision Res. 26, 9, 1351--1386.Google ScholarCross Ref
- C. Seungmoon and K. J. Kuchenbecker. 2013. Vibrotactile display: Perception, technology, and applications. Proc. IEEE 101, 9, 2093--2104.Google ScholarCross Ref
- J. R. Smythies. 1994a. The Walls of Plato's cave. Avebury.Google Scholar
- J. R. Smythies. 1994b. Requiem for the identity theory. Inquiry 37, 311--329.Google ScholarCross Ref
- R. K. Stamper. 1973. Information in Business and Administrative Systems. John Wiley & Sons, New York. Google ScholarDigital Library
- E. Steinbach, S. Hirche, M. Ernst, F. Brandi, R. Chaudhari, J. Kammerl, and I. Vittorias. 2012. Haptic communications. Proc. IEEE 100, 4, 937--956.Google ScholarCross Ref
- C. B. Suk, J. S. Hyun, and L. H. Yong. 2009. Sensory effect metadata for SMMD media service. In Proceedings of the 4th International Conference on Internet and Web Applications and Services. IEEE Computer Society, 649--654. Google ScholarDigital Library
- ISO. 2011. ISO/IEC 23005-3 FDIS Information technology—Media context and control—Part 3: Sensory information. ISO.Google Scholar
- C. Timmerer, M. Waltl, B. Rainer, and H. Hellwagner. 2012. Assessing the quality of sensory experience for multimedia presentations. Signal Process. Image Commun. 27, 8, 909--916. Google ScholarDigital Library
- R. Tortell, D. P. Luigi, A. Dozois, S. Bouchard, J. F. Morie, and D. Ilan. 2007. The effects of scent and game play experience on memory of a virtual environment. VirtualReality 11, 1, 61--68. Google ScholarDigital Library
- A. Vetro and C. Timmerer. 2005. Digital item adaptation: Overview of standardization and research activities. IEEE Trans. Multimedia 7, 3, 418--426. Google ScholarDigital Library
- M. Waltl, C. Timmerer, B. Rainer, and H. Hellwagner. 2012. Sensory effect dataset and test setups. In Proceedings of the 4th International Workshop on Quality of Multimedia Experience (QoMEX'12). IEEE, 115--120.Google Scholar
- M. Waltl, B. Rainer, C. Timmerer, and H. Hellwagner. 2013. An end-to-end tool chain for sensory experience based on MPEG-V Signal Process. Image Commun. 28, 2, 136--150. Google ScholarDigital Library
- A. Williams, S. Langron, and A. Noble. 1984. Influence of appearance on the assessment of aroma in Bordeaux wines by trained assessors. J. Institute Brew. 90, 250--253.Google ScholarCross Ref
- H. R. Wu, A. Reibman, W. Lin, F. Pereira, and S. S. Hemami. 2013. Perceptual visual signal compression and transmission. Proc. IEEE 101, 9, 2025--2043.Google ScholarCross Ref
- X. Yang, W. Lin, Z. Lu, E. Ong, and S. Yao. 2005. Just noticeable distortion model and its applications in video coding. Signal Process. Image Commun. 20, 7, 662--680.Google ScholarCross Ref
- A. L. Yarbus. 1967. Eye movements during perception of complex objects. In Eye Movements and Vision. Springer, 171--211.Google Scholar
- A. Yazdani, E. Kroupi, J. Vesin, and T. Ebrahimi. 2012. Electroencephalogram alterations during perception of pleasant and unpleasant odors. In Proceedings of the 4th International Workshop on Quality of Multimedia Experience (QoMEX'12). IEEE, 272--277.Google Scholar
- K. Yoon, B. Choi, E.-S. Lee, and T.-B. Lim. 2010. 4-D Broadcasting with MPEG-V. In Proceedings of the IEEE International Workshop on Multimedia Signal Processing (MMSP). 257--262.Google ScholarCross Ref
- J. You, U. Reiter, M. M. Hannuksela, M. Gabbouj, and A. Perkis. 2010. Perceptual-based quality assessment for audio-visual services: A survey. Signal Process. Image Commun. 25, 7, 482--501. Google ScholarDigital Library
- J. You, G. Liu, L. Sun, and H. Li. 2007. A multiple visual models based perceptive analysis framework for multilevel video summarization. IEEE Trans. Circuits Syst. Video Technol. 17, 3, 273--285. Google ScholarDigital Library
- W. A. Yost and D. W. Nielsen. 1985. Fundamentals of Hearing. Holt, Rinehart and Winston, New York.Google Scholar
- L. M. Zhang and W. Lin. 2013. Selective Visual Attention: Computational Models and Applications. John Wiley & Sons. Google ScholarDigital Library
Index Terms
- Mulsemedia: State of the Art, Perspectives, and Challenges
Recommendations
Is Multimedia Multisensorial? - A Review of Mulsemedia Systems
Mulsemedia—multiple sensorial media—makes possible the inclusion of layered sensory stimulation and interaction through multiple sensory channels. The recent upsurge in technology and wearables provides mulsemedia researchers a vehicle for potentially ...
Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia Environment
Multisensory experiences have been increasingly applied in Human-Computer Interaction (HCI). In recent years, it is commonplace to notice the development of haptic, olfactory, and even gustatory displays to create more immersive experiences. Companies ...
User Quality of Experience of Mulsemedia Applications
Special Issue on Multiple Sensorial (MulSeMedia) Multimodal Media : Advances and ApplicationsUser Quality of Experience (QoE) is of fundamental importance in multimedia applications and has been extensively studied for decades. However, user QoE in the context of the emerging multiple-sensorial media (mulsemedia) services, which involve ...
Comments