Digitizing the chemical senses: Possibilities & pitfalls

https://doi.org/10.1016/j.ijhcs.2017.06.003Get rights and content

Highlights

  • Review paper assesses the possibilities and pitfalls around the digitization of the chemical senses.

  • Possibilities include delivery of ambient fragrance, digital flavour experiences.

  • We highlight how the majority of the attempts at successful commercialization have failed, often in the face of consumer ambivalence over the perceived benefits/utility.

  • Ultimately, we suggest that mixed reality solutions are currently the most plausible as far as delivering (or rather modulating) flavour experiences digitally is concerned.

  • The identify key problems with digital fragrance delivery related to attention and attribution (i.e., being aware of stimulation and believing that it is doing the work).

Abstract

Many people are understandably excited by the suggestion that the chemical senses can be digitized; be it to deliver ambient fragrances (e.g., in virtual reality or health-related applications), or else to transmit flavour experiences via the internet. However, to date, progress in this area has been surprisingly slow. Furthermore, the majority of the attempts at successful commercialization have failed, often in the face of consumer ambivalence over the perceived benefits/utility. In this review, with the focus squarely on the domain of Human-Computer Interaction (HCI), we summarize the state-of-the-art in the area. We highlight the key possibilities and pitfalls as far as stimulating the so-called ‘lower’ senses of taste, smell, and the trigeminal system are concerned. Ultimately, we suggest that mixed reality solutions are currently the most plausible as far as delivering (or rather modulating) flavour experiences digitally is concerned. The key problems with digital fragrance delivery are related to attention and attribution. People often fail to detect fragrances when they are concentrating on something else; And even when they detect that their chemical senses have been stimulated, there is always a danger that they attribute their experience (e.g., pleasure) to one of the other senses – this is what we call ‘the fundamental attribution error’. We conclude with an outlook on digitizing the chemical senses and summarize a set of open-ended questions that the HCI community has to address in future explorations of smell and taste as interaction modalities

Introduction

Both the popular press and the general public are fascinated by the possibilities associated with the digitization of the chemical senses (e.g., Berenstein, 2015, Cuthbertson, 2015, Lant and Norman, 2017, Marks, 2013, Obrist et al., 2014b, Platt, 1999). And indeed, much research has been conducted in this area in recent decades (as documented by the many papers and sessions at the technology conferences in HCI such as ACM CHI, UIST, and TEI, not to mention, by the various papers referenced in this review). That said, there are still a number of key questions with regard to the digitization of the chemical senses that will need to be addressed before any real progress can be made in delivering plausible (i.e., commercially viable and appealing) solutions to market. These include: What do you want to digitize? Why do you want to digitize it? How do you plan to digitize it? What are the limitations, both technical and psychological, to digital transmission/delivery that are relevant to the chemical senses? It is only by addressing such questions that the various pitfalls that have been highlighted by a number of the high-profile failures in this area in recent years can be avoided (e.g., Dusi, 2014, Twilley, 2016, Velasco et al., 2016).

At the outset, when thinking about the digitization of the chemical senses, it is important to note that there are at least three senses that may be the target of any digital intervention: (1) Stimulation of the sense of taste (gustation); (2) Stimulation of the sense of smell (olfaction via either the orthonasal or retronasal route; e.g., Rozin, 1982, Small et al., 2005); and (3) trigeminal stimulation (responsible for detecting sensations such as heat and cold along with various food textures that are related to biting and chewing actions; e.g., Burdach et al., 1984, Dodd and Kelly, 1991, Lundström et al., 2011, Spence and Piqueras-Fiszman, 2016, Viana, 2011).

The delivery of ambient scent is the simplest application of digitizing the chemical senses, since it requires only orthonasal olfactory stimulation (e.g., as when we inhale/sniff). Such scents might or might not be food (i.e., flavour) related. To date, digitally-controlled scent delivery1 have been used to augment the immersion in audio-visual entertainment/training applications (Cole, 2017; see Ischer et al., 2014, for a review). More generally, ambient scents have been used to trigger specific moods, emotions (Herz, 2002, Leenders et al., in press, Moss et al., 2003, Rétiveau et al., 2004), nostalgia/memories (Chu and Downes, 2000, Chu and Downes, 2002, Doop et al., 2006, Tortell et al., 2007), induce hunger, and even bias our everyday behaviours (Holland et al., 2005).

By contrast, stimulation of the sense of taste, retronasal olfaction,2 and possibly also the trigeminal sense are needed in order to deliver an authentic-tasting flavour experiences (e.g., Bult et al., 2007, Piqueras-Fiszman and Spence, 2016). Just think, for example, about simulating the minty sensation associated with compounds such as 1-methol (the principal flavour in mint). All three of these sensory systems are needed if one is to recreate the characteristic minty aroma, the slightly bitter taste, and the cooling mouth-feel (involving the tactile thermal nociceptors) associated with the experience of this particular stimulus (Nagata et al., 2005). Of course, it is not enough simply to stimulate these senses; The relative intensity of these digital stimuli also needs to be right, as does the time-course of increasing and decreasing sensation (see Obrist et al., 2014a, Stuckey, 2012), if one wants to simulate a genuinely-compelling (i.e., authentic) minty sensation.

Taste (strictly-speaking, gustation) and flavour (the latter referring to the combined input of gustatory, olfactory, and possibly also trigeminal stimulation) are undoubtedly complex/confusing concepts to try and disentangle, both at the theoretical and at the empirical levels (see Spence et al., 2015, for a review). Matters are made more confusing by the existence of phenomenon such as oral referral (of odours to the oral cavity; see Spence, 2016a, for a review), and the fact that different terms are sometimes used in different languages to refer to these two percepts (e.g., Rozin, 1982, Spence, 2017a). Here it is perhaps helpful to bear in mind that stimulation of the taste-buds on the human tongue may only give rise to the sensation of sweet, bitter, salty, sour, and umami.3 Everything else that we enjoy while tasting – the meaty, the fruity, the floral, the herbaceous, and the roasted etc. – are all delivered by the sense of smell instead.4 That is, by volatile molecules hitting the olfactory receptors embedded in the nasal mucosa. It is one of the tricks of the mind that so much of this information, transduced by the olfactory receptors in the nose is referred to the mouth, giving us all the illusion that we are tasting (this is what it is referred to as ‘oral referral’). So, when talking about the digitization of the chemical senses, one needs to keep taste distinct from tasting (the latter normally used to refer to the flavour perceived; see Spence et al., 2015). It is worth bearing in mind that it has widely been estimated that 75–95% of what we think we taste really reflects information delivered by the sense of smell (see Spence, 2015a, for a review). Finally, if one wants to deliver the trigeminal hit of chilli, cinnamon, or ginger, say, then you also need to stimulate the trigeminal sense as well (Cometto-Muñiz and Cain, 1995).

Apart from the senses of taste and smell (or aroma, i.e., food-related smells), simulating the texture of food can also be very important. The trigeminal sense detects heat and cold sensations (e.g., the cool sensation associated with mint, or the burning heat of a good chilli) and is also responsible for detecting the texture of food (e.g., think crunchiness and creaminess; see Spence and Piqueras-Fiszman, 2016, for a review of the literature on oral-somatosensation). Intriguingly here, a number of researchers working in the field of HCI have investigated the consequences for perception of either warming the receptacle in which a drink is held, say, or else warming the air around the nostrils (see Suzuki et al., 2014b).5 The texture and oral-somatosensory mouthfeel that is such a distinctive feature of many foods, while little studied to date (at least relative to the amount of research on the other flavour senses), is undoubtedly an important component of our everyday experience of food. After all, it is a key part of what makes chocolate and ice-cream so desirable. Food textures are also a key feature driving people's food dislikes (see Prescott, 2012, for an overview; and Iwata et al., 2004, Niijima and Ogawa, 2016, for some of the first attempts to simulate the experience of food texture digitally). Hence, there are grounds for thinking that unless any digital delivery system can replicate real food textures/oralsomatosensory experiences they will be ‘thin’ – that is, lacking in substance. Note here only the research showing that people are mostly unable to identify many everyday foods in the absence of the appropriate food texture (Stuckey, 2012).

In the context of exploring the digitalization of the chemical senses, another key distinction needs to be made between flavour expectations and flavour experiences. So far, we have been mostly focused on the senses that directly contribute to flavour perception while eating/drinking. However, we rarely put something in our mouth without having an idea of what it is first. These flavour expectations then anchor our subsequent flavour experience when we actually come to taste (see Piqueras-Fiszman and Spence, 2015, for a review). Vision, orthonasal olfaction, and perhaps, to a lesser extent, sound are the key senses in terms of setting such expectations (see Spence, 2015c, Spence, 2015d, for reviews). As we will see later, given the powerful role of flavour expectations in modulating our flavour experiences, one potential route to digitally modifying our experience of the chemical senses, is by directly targeting the expectation rather than, or in addition to, the experience. However, again, these approaches may or may not stimulate a similar experience in the mind of the user, and hence further experimentation is definitely still needed in order to evaluate their effectiveness in a digital context.

Section snippets

What do you want to digitize?

There are at least two principle suggestions here: (1) Ambient scent delivery (of either food-related aromas or food-unrelated scents); and (2) Tasting experiences. Over the last couple of decades, many researchers have turned their attention to question the opportunities inherent in terms of enhancing the sense of presence afforded by the introduction of virtual olfactory displays (e.g., see Barfield and Danas, 1996, Cater, 1992, Jones et al., 2004, Lombard and Ditton, 1997, Matsukura et al.,

Why digitize the chemical senses?

It is at this point in the discussion that it is important to distinguish between two routes to the ‘digital’ stimulation of the chemical senses: (1) Chemical stimulation (substances) can be released under computerized/digital control; and (2) The taste buds can be stimulated electrically and thermally without any need for chemical stimuli. Notably, whilst the latter might well be especially interesting, it has proved to be extremely difficult to deliver without the aid of additional sensory

How to digitize the chemical senses?

In this section, we will take a look at pure digital approaches to stimulating the chemical senses, starting with taste, then trigeminal, and finally olfactory. In recent years, progress has been made in terms of delivering electric taste sensations in a practical and increasingly well-designed manner (e.g., Murer et al., 2013, Ranasinghe and Do, 2016a, Ranasinghe et al., 2012b, Ranasinghe et al., 2014) (see Fig. 2). This contrasts, then, with the much more modest advance in the world of purely

What are the limitations?

It is at this point that it becomes crucial to highlight some of the key challenges, a number of which have cropped up already, in order to avoid the pitfalls that have beset a number of many previous attempts to digitize the chemical senses.

Modifying taste/flavour using digital stimulation of the other senses

Ultimately, given the limitations associated with digitally stimulating the chemical senses directly, one other solution that is worth considering here is to modify people's experience of actual food/beverage stimuli by more appropriately stimulating the other (more dominant) senses (one can think of this as a kind of mixed, or augmented, reality solution). So, for example, Zampini and Spence (2004) demonstrated that they could modify people's impression of the freshness and crispness of potato

Conclusions

As noted earlier (e.g., Kortum, 2008, Obrist et al., 2016), there has, to date at least, been little relatively interest in the digitization of the chemical senses (at least when compared to the digitization of the other senses). On the one hand, this likely reflects the not inconsiderable technical challenges associated with the effective digital stimulation of the chemical senses. However, it is also consistent with a more general neglect of the chemical (what are sometimes described as the

Acknowledgements

Funding CS would like to acknowledge the AHRC Rethinking the Senses grant (AH/L007053/1). MO wishes to thank the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, grant agreement No 638605.

References (248)

  • E. Iannilli et al.

    Trigeminal activation using chemical, electrical, and mechanical stimuli

    Pain

    (2008)
  • Tube scent machine breaks down

    BBC News Online

    (2001)
  • A. Ashkenazi et al.

    Effect of endogenous attention on detection of weak gustatory and olfactory flavors

    Percept. Psychophys.

    (2004)
  • J.A. Ballas

    Delivery of information through sound

  • W. Barfield et al.

    Comments on the use of olfactory displays for virtual environments

    Presence

    (1996)
  • R.A. Baron et al.

    Effects of a pleasant ambient fragrance on simulated driving performance: the sweet smell of... safety?

    Environ Behav

    (1998)
  • O. Baus et al.

    Exposure to an unpleasant odour increases the sense of presence in virtual reality

    Virtual Reality

    (2016)
  • N. Berenstein

    This smell synthesizer lets you sniff and play flavors like music: a peek behind the scenes at the Museum of Food and Drink's new Flavor exhibition

    Popular Science

    (2015)
  • A. Bodnar et al.

    AROMA: ambient awareness through olfaction in a messaging application

  • A. Bolton

    No salt, no problem! Japanese Electro Fork zaps flavour into your mouth

    CNet

    (2016)
  • M. Bordegoni et al.

    Investigating the use of smell in vehicle-driver interaction

  • W. Bounds

    Sounds and scents to jolt drowsy drivers

    Wall Street J.

    (1996)
  • M.H. Braun et al.

    Emotional priming of digital images through mobile tele-smell and virtual food

    Int. J. Food Des.

    (2016)
  • S. Brewster et al.

    Olfoto: designing a smellbased interaction

  • J.A. Brillat-Savarin

    Physiologie du goût [The philosopher in the kitchen/the physiology of taste]

  • Z. Bujas

    Electrical taste

  • Z. Bujas et al.

    Adaptation effects on evoked electrical taste

    Percept. Psychophys.

    (1974)
  • K.J. Burdach et al.

    Nasal, retronasal, and gustatory perception: an experimental comparison

    Percept. Psychophys.

    (1984)
  • M. Burgess

    We got sprayed in the face by a 9D television

    Wired

    (2016)
  • A.V. Cardello

    Comparison of taste qualities elicited by tactile, electrical and chemical stimulation of single human taste papillae

    Percept. Psychophys.

    (1981)
  • M.A. Carskadon et al.

    Minimal olfactory perception during sleep: why odor alarms will not work for humans

    Sleep

    (2004)
  • J.P. Cater

    The noses have it!

    Presence

    (1992)
  • Chang, L. (2017). Smells of rose and sandalwood lull you to sleep with the Oria from Sensorwake....
  • S. Chu et al.

    Odour-evoked autobiographical memories: psychological investigations of Proustian phenomena

    Chem. Senses.

    (2000)
  • S. Chu et al.

    Proust nose best: odours are better cues of autobiographical memory

    Memory Cognit.

    (2002)
  • Clark, J. (2013, March 26). 2014 Mercedes-Benz S-Class interior is “the essence of luxury”....
  • C. Classen et al.

    Artificial flavours

  • S. Cole

    Do you want Smell-O-Vision for your VR porn? Neither do we but here it is

    Motherboard

    (2017)
  • R. Comber et al.

    Food and interaction design: designing for food in everyday life

    CHI '12 Extended Abstracts on Human Factors in Computing Systems

    (2012)
  • J.E. Cometto-Mufiiz et al.

    Relative sensitivity of the ocular trigeminal, nasal trigeminal and olfactory systems to airborne chemicals

    Chem. Senses.

    (1995)
  • B. Crowther

    How does it smell? “Scent of Mystery” intrudes another question of quality in films

    The New York Times

    (1960)
  • A. Cruz et al.

    Thermal stimulation of taste

    Nature

    (2000)
  • A. Cuthbertson

    Taste+ smart spoon and cup virtually enhance food flavours and restore taste to the elderly

    International Business Times

    (2015)
  • P. Dalton

    Odor perception and beliefs about risk

    Chem. Senses.

    (1996)
  • F. Davide et al.

    Virtual olfactory interfaces: electronic noses and olfactory displays

    Communications Through Virtual Technology: Identity Community and Technology in the Internet Age

    (2001)
  • N. Davis

    Welcome to the Tate Sensorium, where the paintings come with chocolates

    The Guardian

    (2015)
  • D. Dmitrenko et al.

    A comparison of scent-delivery devices and their meaningful use for in-car olfactory interaction

  • J. Dodd et al.

    Trigeminal system

  • M. Doop et al.

    Olfaction and memory

  • M. Duell

    Smell-O-Vision for the 21st century: Phones able to send scented messages are among ten emerging technologies for 2015

    DailyMail Online

    (2014)
  • Cited by (0)

    View full text