Digitizing the chemical senses: Possibilities & pitfalls
Introduction
Both the popular press and the general public are fascinated by the possibilities associated with the digitization of the chemical senses (e.g., Berenstein, 2015, Cuthbertson, 2015, Lant and Norman, 2017, Marks, 2013, Obrist et al., 2014b, Platt, 1999). And indeed, much research has been conducted in this area in recent decades (as documented by the many papers and sessions at the technology conferences in HCI such as ACM CHI, UIST, and TEI, not to mention, by the various papers referenced in this review). That said, there are still a number of key questions with regard to the digitization of the chemical senses that will need to be addressed before any real progress can be made in delivering plausible (i.e., commercially viable and appealing) solutions to market. These include: What do you want to digitize? Why do you want to digitize it? How do you plan to digitize it? What are the limitations, both technical and psychological, to digital transmission/delivery that are relevant to the chemical senses? It is only by addressing such questions that the various pitfalls that have been highlighted by a number of the high-profile failures in this area in recent years can be avoided (e.g., Dusi, 2014, Twilley, 2016, Velasco et al., 2016).
At the outset, when thinking about the digitization of the chemical senses, it is important to note that there are at least three senses that may be the target of any digital intervention: (1) Stimulation of the sense of taste (gustation); (2) Stimulation of the sense of smell (olfaction via either the orthonasal or retronasal route; e.g., Rozin, 1982, Small et al., 2005); and (3) trigeminal stimulation (responsible for detecting sensations such as heat and cold along with various food textures that are related to biting and chewing actions; e.g., Burdach et al., 1984, Dodd and Kelly, 1991, Lundström et al., 2011, Spence and Piqueras-Fiszman, 2016, Viana, 2011).
The delivery of ambient scent is the simplest application of digitizing the chemical senses, since it requires only orthonasal olfactory stimulation (e.g., as when we inhale/sniff). Such scents might or might not be food (i.e., flavour) related. To date, digitally-controlled scent delivery1 have been used to augment the immersion in audio-visual entertainment/training applications (Cole, 2017; see Ischer et al., 2014, for a review). More generally, ambient scents have been used to trigger specific moods, emotions (Herz, 2002, Leenders et al., in press, Moss et al., 2003, Rétiveau et al., 2004), nostalgia/memories (Chu and Downes, 2000, Chu and Downes, 2002, Doop et al., 2006, Tortell et al., 2007), induce hunger, and even bias our everyday behaviours (Holland et al., 2005).
By contrast, stimulation of the sense of taste, retronasal olfaction,2 and possibly also the trigeminal sense are needed in order to deliver an authentic-tasting flavour experiences (e.g., Bult et al., 2007, Piqueras-Fiszman and Spence, 2016). Just think, for example, about simulating the minty sensation associated with compounds such as 1-methol (the principal flavour in mint). All three of these sensory systems are needed if one is to recreate the characteristic minty aroma, the slightly bitter taste, and the cooling mouth-feel (involving the tactile thermal nociceptors) associated with the experience of this particular stimulus (Nagata et al., 2005). Of course, it is not enough simply to stimulate these senses; The relative intensity of these digital stimuli also needs to be right, as does the time-course of increasing and decreasing sensation (see Obrist et al., 2014a, Stuckey, 2012), if one wants to simulate a genuinely-compelling (i.e., authentic) minty sensation.
Taste (strictly-speaking, gustation) and flavour (the latter referring to the combined input of gustatory, olfactory, and possibly also trigeminal stimulation) are undoubtedly complex/confusing concepts to try and disentangle, both at the theoretical and at the empirical levels (see Spence et al., 2015, for a review). Matters are made more confusing by the existence of phenomenon such as oral referral (of odours to the oral cavity; see Spence, 2016a, for a review), and the fact that different terms are sometimes used in different languages to refer to these two percepts (e.g., Rozin, 1982, Spence, 2017a). Here it is perhaps helpful to bear in mind that stimulation of the taste-buds on the human tongue may only give rise to the sensation of sweet, bitter, salty, sour, and umami.3 Everything else that we enjoy while tasting – the meaty, the fruity, the floral, the herbaceous, and the roasted etc. – are all delivered by the sense of smell instead.4 That is, by volatile molecules hitting the olfactory receptors embedded in the nasal mucosa. It is one of the tricks of the mind that so much of this information, transduced by the olfactory receptors in the nose is referred to the mouth, giving us all the illusion that we are tasting (this is what it is referred to as ‘oral referral’). So, when talking about the digitization of the chemical senses, one needs to keep taste distinct from tasting (the latter normally used to refer to the flavour perceived; see Spence et al., 2015). It is worth bearing in mind that it has widely been estimated that 75–95% of what we think we taste really reflects information delivered by the sense of smell (see Spence, 2015a, for a review). Finally, if one wants to deliver the trigeminal hit of chilli, cinnamon, or ginger, say, then you also need to stimulate the trigeminal sense as well (Cometto-Muñiz and Cain, 1995).
Apart from the senses of taste and smell (or aroma, i.e., food-related smells), simulating the texture of food can also be very important. The trigeminal sense detects heat and cold sensations (e.g., the cool sensation associated with mint, or the burning heat of a good chilli) and is also responsible for detecting the texture of food (e.g., think crunchiness and creaminess; see Spence and Piqueras-Fiszman, 2016, for a review of the literature on oral-somatosensation). Intriguingly here, a number of researchers working in the field of HCI have investigated the consequences for perception of either warming the receptacle in which a drink is held, say, or else warming the air around the nostrils (see Suzuki et al., 2014b).5 The texture and oral-somatosensory mouthfeel that is such a distinctive feature of many foods, while little studied to date (at least relative to the amount of research on the other flavour senses), is undoubtedly an important component of our everyday experience of food. After all, it is a key part of what makes chocolate and ice-cream so desirable. Food textures are also a key feature driving people's food dislikes (see Prescott, 2012, for an overview; and Iwata et al., 2004, Niijima and Ogawa, 2016, for some of the first attempts to simulate the experience of food texture digitally). Hence, there are grounds for thinking that unless any digital delivery system can replicate real food textures/oralsomatosensory experiences they will be ‘thin’ – that is, lacking in substance. Note here only the research showing that people are mostly unable to identify many everyday foods in the absence of the appropriate food texture (Stuckey, 2012).
In the context of exploring the digitalization of the chemical senses, another key distinction needs to be made between flavour expectations and flavour experiences. So far, we have been mostly focused on the senses that directly contribute to flavour perception while eating/drinking. However, we rarely put something in our mouth without having an idea of what it is first. These flavour expectations then anchor our subsequent flavour experience when we actually come to taste (see Piqueras-Fiszman and Spence, 2015, for a review). Vision, orthonasal olfaction, and perhaps, to a lesser extent, sound are the key senses in terms of setting such expectations (see Spence, 2015c, Spence, 2015d, for reviews). As we will see later, given the powerful role of flavour expectations in modulating our flavour experiences, one potential route to digitally modifying our experience of the chemical senses, is by directly targeting the expectation rather than, or in addition to, the experience. However, again, these approaches may or may not stimulate a similar experience in the mind of the user, and hence further experimentation is definitely still needed in order to evaluate their effectiveness in a digital context.
Section snippets
What do you want to digitize?
There are at least two principle suggestions here: (1) Ambient scent delivery (of either food-related aromas or food-unrelated scents); and (2) Tasting experiences. Over the last couple of decades, many researchers have turned their attention to question the opportunities inherent in terms of enhancing the sense of presence afforded by the introduction of virtual olfactory displays (e.g., see Barfield and Danas, 1996, Cater, 1992, Jones et al., 2004, Lombard and Ditton, 1997, Matsukura et al.,
Why digitize the chemical senses?
It is at this point in the discussion that it is important to distinguish between two routes to the ‘digital’ stimulation of the chemical senses: (1) Chemical stimulation (substances) can be released under computerized/digital control; and (2) The taste buds can be stimulated electrically and thermally without any need for chemical stimuli. Notably, whilst the latter might well be especially interesting, it has proved to be extremely difficult to deliver without the aid of additional sensory
How to digitize the chemical senses?
In this section, we will take a look at pure digital approaches to stimulating the chemical senses, starting with taste, then trigeminal, and finally olfactory. In recent years, progress has been made in terms of delivering electric taste sensations in a practical and increasingly well-designed manner (e.g., Murer et al., 2013, Ranasinghe and Do, 2016a, Ranasinghe et al., 2012b, Ranasinghe et al., 2014) (see Fig. 2). This contrasts, then, with the much more modest advance in the world of purely
What are the limitations?
It is at this point that it becomes crucial to highlight some of the key challenges, a number of which have cropped up already, in order to avoid the pitfalls that have beset a number of many previous attempts to digitize the chemical senses.
Modifying taste/flavour using digital stimulation of the other senses
Ultimately, given the limitations associated with digitally stimulating the chemical senses directly, one other solution that is worth considering here is to modify people's experience of actual food/beverage stimuli by more appropriately stimulating the other (more dominant) senses (one can think of this as a kind of mixed, or augmented, reality solution). So, for example, Zampini and Spence (2004) demonstrated that they could modify people's impression of the freshness and crispness of potato
Conclusions
As noted earlier (e.g., Kortum, 2008, Obrist et al., 2016), there has, to date at least, been little relatively interest in the digitization of the chemical senses (at least when compared to the digitization of the other senses). On the one hand, this likely reflects the not inconsiderable technical challenges associated with the effective digital stimulation of the chemical senses. However, it is also consistent with a more general neglect of the chemical (what are sometimes described as the
Acknowledgements
Funding CS would like to acknowledge the AHRC Rethinking the Senses grant (AH/L007053/1). MO wishes to thank the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, grant agreement No 638605.
References (248)
- et al.
Thermal taste, PROP responsiveness, and perception of oral sensations
Physiol. Behav.
(2008) On the idea of novelty in cuisine: a brief historical insight
Int. J. Gastronomy Food Sci
(2012)- et al.
The fish is bad: negative food odors elicit faster and more accurate reactions than other odors
Biol. Psychol.
(2010) - et al.
Retronasal perception of odors
Physiol. Behav.
(2012) - et al.
Investigations on multimodal sensory integration: texture, taste, and ortho- and retronasal olfactory stimuli in concert
Neurosci. Lett.
(2007) Electronic wearable using personalizable sound and fragrance for personal branding
- et al.
A bittersweet symphony: systematically modulating the taste of food by changing the sonic properties of the soundtrack playing in the background
Food Qual. Preference
(2012) - et al.
Effects of the sound of the bite on apple perceived crispness and hardness
Food Qual. Preference
(2014) - et al.
The effect of a crunchy pseudo-chewing sound on perceived texture of softened foods
Physiol. Behav.
(2016) - et al.
Olfactory facilitation of dual-task performance
Neurosci. Lett.
(2005)