Abstract
This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which treats sound and musical structures using these models, when mapping data to sound. However, there are currently very few theoretical frameworks for applying embodied cognition principles in a sonic information design context. This article describes one such framework, the embodied sonification listening model, which provides a theoretical description of sonification listening in terms of conceptual metaphor theory.
Similar content being viewed by others
References
Faste T, Faste H (2012) Demystifying “design research”: design is not research, research is design. Presented at Industrial Designers Society of America Education Symposium, Boston, Massachusetts
Flowers JH (2005) Thirteen years of reflection on auditory graphing: promises, pitfalls, and potential new directions. Faculty Publications, Department of Psychology, 430
Worrall D (2019) Intelligible sonifications. In: Sonification design: from data to intelligible soundfields. Springer, Berlin
Truax B (1984) Acoustic communication. Ablex, Norwood
O’Callaghan C (2007) Sounds: a philosophical theory. Oxford University Press, Oxford
Kahn D (1999) Noise, water, meat: a history of sound in the arts. MIT press, Cambridge
LaBelle B (2010) Acoustic territories: sound culture and everyday life. A&C Black, London
Cox C (2011) Beyond representation and signification: toward a sonic materialism. J Vis Cult 10(2):145–161
Roddy S, Bridges B (2016) Sounding human with data: the role of embodied conceptual metaphors and aesthetics in representing and exploring data sets. In: The proceedings of the MusTWork 2016 the music technology workshop
Roddy S, Bridges B (2018) Addressing the mapping problem in sonic information design through embodied image schemata, conceptual metaphors and conceptual blending. J Sonic Stud 17
Worrall D (2013) Understanding the need for micro-gestural inflections in parameter-mapping sonification. Georgia Institute of Technology, Atlanta
Worrall D (2014) Can micro-gestural inflections be used to improve the soniculatory effectiveness of parameter mapping sonifications? Organ Sound 19(1):52–59
Grond F, Berger J (2011) Parameter mapping sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook. Logos Publishing House, Berlin, pp 363–397
Ryle G (1949) The concept of mind. Hutchinson, London
Searle JR (1980) Minds, brains, and programs. Behav Brain Sci 3(03):417–424
Harnad S (1990) The symbol grounding problem. Phys D Nonlinear Phenom 42(1):335–346
Dreyfus HL (1965) Alchemy and artificial intelligence. The Rand Corporation, Santa Monica, Research Report
Polanyi M (2012) Personal knowledge. Routledge, London
Varela FJ, Thompson E, Rosch E (1991) The embodied mind: cognitive science and human experience. MIT Press, Cambridge
Johnson M (1987) The body in the mind: the bodily basis of meaning, imagination, and reason. University of Chicago Press, Chicago
Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, Chicago
Fauconnier G, Turner M (2002) The way we think. Conceptual blending and the mind’s hidden complexities. Basic Books, New York
Imaz M, Benyon D (2007) Designing with blends: conceptual foundations of human–computer interaction and software engineering methods. MIT Press, Cambridge
Hurtienne J (2009) Image schemas and design for intuitive use. Exploring new guidance for user interface design. (Doctoral Thesis). Retrieved from Deposit Once TU Berlin Reposityory for Research and Data Publications. TU Berlin identifier opus3 2970
Waterworth J, Riva G (2014) Feeling present in the physical world and in computer-mediated environments. Palgrave Macmillan, London
Bødker S, Klokmose CN (2016). Dynamics, multiplicity and conceptual blends in HCI. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 2538–2548
Dourish P (2004) Where the action is: the foundations of embodied interaction. MIT press, Cambridge
Serafin S, Franinovic K, Hermann T, Lemaitre G, Rinott M, Rocchesso D (2011) Sonic interaction design. In: Hermann T, Hunt A, Neuhoff JG (eds) The Sonification Handbook. Logos Publishing. House, Berlin, pp 87–110
Diniz N, Deweppe A, Demey M, Leman M (2010) A framework for music-based interactive sonification. In: 16th International conference on auditory display (ICAD-2010)
Diniz N, Coussement P, Deweppe A, Demey M, Leman M (2012) An embodied music cognition approach to multilevel interactive sonification. J Multimodal User Interfaces 5(3–4):211–219
Dyer J, Stapleton P, Rodger MW (2015) Sonification as concurrent augmented feedback for motor skill learning and the importance of mapping design. Open Psychol J 8(3):1–11
Dyer J, Stapleton P, Rodger M (2017) Transposing musical skill: sonification of movement as concurrent augmented feedback enhances learning in a bimanual task. Psychol Res 81(4):850–862
Peres SC, Verona D, Nisar T, Ritchey P (2017) Towards a systematic approach to real-time sonification design for surface electromyography. Displays 47:25–31
Cox A (2001) The mimetic hypothesis and embodied musical meaning. Musicae Scientiae 5(2):195–212
Brower C (2000) A cognitive theory of musical meaning. J Music Theory 44(2):323–379
Adlington R (2003) Moving beyond motion: metaphors for changing sound. J R Music Assoc 128(2):297–318
Zbikowski LM (2002) Conceptualizing music: cognitive structure, theory, and analysis. Oxford University Press, Oxford
Larson S (2012) Musical forces: motion, metaphor, and meaning in music. Indiana University Press, Bloomington
Kendall GS (2010) Meaning in electroacoustic music and the everyday mind. Organ Sound 15(1):63–74
Kendall GS (2014) The feeling blend: feeling and emotion in electroacoustic art. Organ Sound 19(2):192
Graham R, Bridges B (2014) Gesture and embodied metaphor in spatial music performance systems design. In: NIME, pp 581–584
Graham R, Bridges B (2014) Strategies for spatial music performance: the practicalities and aesthetics of responsive systems design. Divergence Press
Smalley D (1996) The listening imagination: listening in the electroacoustic era. Contemp Music Rev 13(2):77–107
Smalley D (1997) Spectromorphology: explaining sound-shapes. Organ Sound 2(02):107–126
Godøy RI (2006) Gestural-sonorous objects: embodied extensions of Schaeffer’s conceptual apparatus. Organ Sound 11(02):149–157
Graham R, Manzione C, Bridges B, Brent W (2017) Exploring pitch and timbre through 3d spaces: embodied models in virtual reality as a basis for performance systems design. In: New interfaces for musical expression proceedings
Grey JM, Gordon JW (1978) Perceptual effects of spectral modifications on musical timbres. J Acoust Soc Am 63(5):1493–1500
Vickers P, Hogg B (2006) Sonification ab-straite/sonification concr`ete: An ‘æsthetic perspectivespace’ for classifying auditory displays in the ars musica domain. In: 12th International conference on auditory display (ICAD-2010), pp 210–216
Schaeffer P, Reibel G, Ferreyra B, Chiarucci H, Bayle F, Tanguy A et al (1967) Solfège de l’objet sonore. In: INA/GRM
Chion M, Gorbman C, Murch W (1994) Audio-vision
Gaver WW (1989) The SonicFinder: an interface that uses auditory icons. Hum Comput Interact 4:67–94
Roddy S (2015) Embodied sonification. Doctoral Dissertation. Trinity College Dublin. Ireland
Walker BN, Nees MA (2011) Theory of Sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The Sonification Handbook. Logos Publishing House, Berlin, pp 9–39
Koestler A (1967) The ghost in the machine. Hutchinson & Co., London
Talmy L (2008) The fundamentals of spatial systems. In: Hampe B (ed) From perception to meaning: image schemas in cognitive linguistics. Walter de Greuyter, Berlin
White M (2003) Metaphor and economics: the case of growth. Engl Specif Purp 22(2):131–151
Walker BN (2002) Magnitude estimation of conceptual data dimensions for use in sonification. J Exp Psychol Appl 8(4):211
Kövecses Z (2010) Metaphor and culture. Acta Universitatis Sapientiae Philologica 2(2):197–220
Chung SF (2005). MARKET metaphors: Chinese, English and Malay. In: Proceedings of the 19th Pacific Asia conference on language, information and computation, pp 71–81
Charteris-Black J, Ennis T (2001) A comparative study of metaphor in Spanish and English financial reporting. Engl Specif Purp 20(3):249–266
Polli A (2012) Soundscape, sonification, and sound activism. AI Soc 27(2):257–268
Jeon M, Lee JH, Sterkenburg J, Plummer C (2015) Cultural differences in preference of auditory emoticons: USA and South Korea. Georgia Institute of Technology, Atlanta
Acknowledgements
This publication has been funded by an Irish Research Council Government of Ireland Postdoctoral Fellowship Award (Grant no. 14887). This publication has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) and is co- funded under the European Regional Development Fund under Grant No. 13/RC/2077.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Roddy, S., Bridges, B. Mapping for meaning: the embodied sonification listening model and its implications for the mapping problem in sonic information design. J Multimodal User Interfaces 14, 143–151 (2020). https://doi.org/10.1007/s12193-020-00318-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-020-00318-y