Abstract
Large Language Models such as GPT-3 exhibit generative language capabilities with multiple potential applications in creative practice. In this paper, we present a method for data sonification that employs the GPT-3 model to create semantically relevant mappings between artificial intelligence-generated natural language descriptions of data, and human-generated descriptions of sounds. We implemented this method in a public art installation to generate a soundscape based on data from different systems. While common sonification approaches rely on arbitrary mappings between data values and sonic values, our approach explores the use of language models to achieve a mapping not via values but via meaning. We find our approach is a useful tool for musification practice and demonstrates a new application of generative language models in creative new media arts practice. We show how different prompts influence data to sound mappings, and highlight that matching the embeddings of texts of different lengths produces undesired behavior.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Andres, J.: Adaptive human bodies & adaptive built environments for enriching futures. In: Frontiers in Computer Science, Special Issue Inbodied Interaction (2022)
Brand, S.: Pace layering: how complex systems learn and keep learning. J. Design Sci. (Jan 2018)
Floridi, L., Chiriatti, M.: GPT-3: Its nature, scope, limits, and consequences. Minds Mach. 30(4), 681–694 (2020)
Flowers, J.H., Whitwer, L.E., Grafel, D.C., Kotan, C.A.: Sonification of daily weather records: issues of perception, attention and memory in design choices. Faculty Publications, Department of Psychology, p. 432 (2001)
Hermann, T., Drees, J.M., Ritter, H.: Broadcasting auditory weather reports-a pilot project (2003)
Hermann, T., Hunt, A., Neuhoff, J.G.: The sonification handbook. Logos Verlag Berlin (2011)
Andres, J., et al.: Cybernetic lenses for designing and living in a complex world. In: In Extended Abstracts of the 2022 OzCHI Conference on Human Factors in Computing Systems (OZCHI EA 2022). Association for Computing Machinery, New York, NY, USA. (2022)
Kalonaris, S.: Tokyo Kion-On: query-Based generative sonification of atmospheric data (Aug 2022)
Krol, S.J., Llano, M.T., McCormack, J.: Towards the generation of musical explanations with GPT-3. In: Artificial Intelligence in Music, Sound, Art and Design: 11th International Conference, EvoMUSART 2022, Held as Part of EvoStar 2022, Madrid, Spain, April 20–22, 2022, Proceedings, pp. 131–147. Springer-Verlag, Berlin, Heidelberg (Apr 2022)
Mardakheh, M.K., Wilson, S.: A strata-based approach to discussing artistic data sonification. Leonardo 55(5), 516–520 (2022)
Neelakantan, A., et al.: Text and code embeddings by contrastive Pre-Training (Jan 2022)
OpenAI: OpenAI API. https://beta.openai.com/docs/introduction (2022). Accessed 17 Nov 2022
OpenAI: OpenAI API. https://beta.openai.com/docs/guides/embeddings (2022). Accessed 21 Aug 2022
Polli, A.: Atmospherics/weather works: a multi-channel storm sonification project (2004)
Quinn, M.: Research set to music: the climate symphony and other sonifications of ice core, radar, DNA, seismic and solar wind data (2001)
Ramesh, A., et al.: Zero-shot text-to-image generation (Feb 2021)
Rocchesso, D., et al.: Sonic interaction design: sound, information and experience. In: CHI’08 Extended Abstracts on Human Factors in Computing Systems, pp. 3969–3972 (2008)
Roddy, S.: Signal to noise loops: a cybernetic approach to musical performance with smart city data and generative music techniques. Leonardo, pp. 525–532 (2022)
Singhal, A.: Modern information retrieval: a brief overview. http://160592857366.free.fr/joe/ebooks/ShareData/Modern%20Information%20Retrieval%20-%20A%20Brief%20Overview.pdf (2001). Accessed 17 Nov 2022
Acknowledgements
This research was made possible by a commission from the School of Cybernetics at the Australian National University for music studio Uncanny Valley (UV). The development of the novel concept for a semantically relevant sonification using Large Language Models is an original contribution from Rodolfo Ocampo, who also led the technical development of the system, in collaboration with members of the UV team. The artwork uses UV’s MEMU generative music system, developed by Justin Shave and Brendan Wright. The design and development of the visual user interface were led by Adrian Schmidt and Josh Andres. Oliver Bown and Rodolfo Ocampo’s research is supported by an Australian Research Council Discovery Project (DP200101059).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ocampo, R. et al. (2023). Using GPT-3 to Achieve Semantically Relevant Data Sonificiation for an Art Installation. In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M. (eds) Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2023. Lecture Notes in Computer Science, vol 13988. Springer, Cham. https://doi.org/10.1007/978-3-031-29956-8_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-29956-8_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29955-1
Online ISBN: 978-3-031-29956-8
eBook Packages: Computer ScienceComputer Science (R0)