Linking Haptic Parameters to the Emotional Space for Mediated Social Touch

1. Introduction

A social touch is a physical interaction that expresses an intent between two or more social agents. Typical examples of social touch include shaking hands with colleagues for greetings, hugging family members for comfort and affection, or patting a friend's shoulder for support and congratulation. Social touch is observed in a wide variety of contexts, not only among humans but also between mammals in general (Harlow and Zimmermann, 1959; Van Erp and Toet, 2015). Such physical interactions give a feeling of mutual awareness and enable to build and strengthen social bonds with other social agents such as other humans, animals, or even artificial intelligence.

Certain emotions such as comfort, love, and sympathy are hard to express in words, such as in written text or with oral speech (Field, 2010; Van Erp and Toet, 2015; Huisman, 2017). Touch is our primary non-verbal communication channel for conveying deeper intimate emotions (Jones and Yarbrough, 1985; Hertenstein et al., 2006; Van Erp and Toet, 2015), and preferred over body gestures and facial expressions for conveying both love and sympathy (App et al., 2011). People want to communicate whilst being physically separated, and although current media such as text messages and video-calls can enable social interactions; they are unable to provide any physical interactions. As a result, these current communication technologies help bring users closer, but the lack of tactile interactions leads to impoverished social interactions between the distant users. To maintain the physical connection, social touch can be conveyed using an intermediate haptic feedback device placed on distant interlocutors known as Mediated Social Touch. Several wearable devices have been investigated for the purposes of social haptic communication, including, shared physical spaces (Dodge, 1997) and objects (Brave and Dahley, 1997), handheld vibrotactile arrays (Chang et al., 2002; Borst and Cavanaugh, 2004), gloves (Singhal et al., 2017), sleeves (Huisman et al., 2013; Cang and Israr, 2020; Simons et al., 2020; Salvato et al., 2021), wristbands (Pezent et al. (2019), HeyBracelet, BondTouch), jackets (Chung et al., 2009; Vaucelle et al., 2009; Teh et al., 2012), and belts (Tsetserukou, 2010). These mediated social touch devices either render canned haptic patterns or directly map the sender's activities to real-time spatiotemporal haptic patterns on the receiver's body, in order to convey expressive touch features associated with user intents and emotions. In the present study, we investigate parametric compositions of haptic patterns to render expressive touch gestures on the hand, and how these parameters vary the affective content of the intended tactile message.

Within literature, there is a need to develop an understanding of the characteristics required to communicate social touch using a shared vocabulary between a sender and a receiver (Gallace and Spence, 2010; Van Erp and Toet, 2015). Recent research has investigated the construction of social touch messages, and if the receiver could interpret the sender's intention and embedded emotions from associated touch gestures on the body. Kirsch et al. (2018) examined touch characteristics to communicate emotions and showed that slow, gentle strokes on the forearm were likely to convey arousal and desire, however, love and supportive intentions were reliably elicited by gentle touch only. McIntyre et al. (2021) investigated social touch gestures associated with six common messages (attention, love, happiness, calming, sadness, and gratitude) conveyed between close relatives on the forearm. They examined primitive elements in touch gestures and developed a standardized set of touch expressions. These expressions were intuitive to their participants, even when the touch was delivered by a stranger with minimal context and training. These studies showed the universality of touch gestures and suggested physical features in interpersonal touch communication between users. Salvato et al. (2021) developed an algorithm to map touch features recorded on a discrete sensor array and rendered on a low degree-of-freedom haptic device on the forearm, and demonstrated above-chance success in communicating six social messages.

In the present study, we construct parametric models for haptic messages associated with high five, handshake, caress, asking for attention, and the pulse of a heartbeat, and render them on a user's hand using a haptic glove. We define two scenarios for each haptic message with different levels of emotional context and examine how model parameters vary with the affective content embedded in these messages. Within this study, we aim to look at building blocks of social touch and how users can tune them to haptically represent emotional content. In addition, we aim to determine how well these parameters can be generalized across participants.

The organization of the article is as follows: the details of the glove, control strategy, social scenarios and haptic parameters are described in Section 2. Section 3 will outline the first user study where participants tune the parameters for 10 different interaction scenarios, and Section 4 will detail the follow up user study where naïve participants attempt to recognize the correct interaction scenario using the tuned parameters from the first user study. Lastly, Section 5 will discuss the results from these two experiments and how modifying haptic parameters can alter the perceived emotional content in social interactions, outline limitations of this study, and paths for future work.

2. Experimental Setup

2.1. Haptic Glove

The haptic glove is pneumatically actuated as shown in Figure 1. It embeds three types of actuators; 15 rounded inflatable bubbles that give normal pressure, four kinesthetic impedance actuators that prevent fingers bending, and three multichannel actuators at the thumb, index and middle fingertips. Depending on the actuation, the multichannel actuator can give shear forces along the lateral plane in any of the four directions or normal pressure when all the channels are inflated simultaneously. The pressures in the pneumatic actuators are controlled through a multichannel pneumatic analog control system (Stephens-Fripp et al., 2021) as this allows for controlling the amplitude of the pressure and the attack and release profiles of the pneumatic waveforms, which are sent to the actuators as time-varying pressure envelopes. Shown in Figure 1B, these envelopes have ASR (Attack-Sustain-Release) profiles, consisting of a duration to linearly ramp from zero to a desired maximum pressure level, a duration to sustain that pressure, and a duration to ramp back down to zero.

Figure 1. Haptic glove description. (A) Description and placement of the haptic actuators and (B) illustration of a typical ASR profile for actuation.

2.2. Social Touches and Interaction Scenarios

Based on the results of a recent study (Rognon et al., 2021) and on the feasibility of transmitting a social touch via a glove, we implemented four social touches, each belonging to a different social touch category according to Jones and Yarbrough (1985). For the social touches, we selected a handshake, which is ritualistic, caress which is a positive affection, asking for attention to represent control, and high five as a playful social touch. We also implemented the physiological signal heartbeat as it is currently the state-of-the-art of personal tactile message that one can send to someone (available on the Apple Watch). In our previous survey (Rognon et al., 2021), we have seen that a social touch can express very different emotions depending on the relationship with the other social agent and the interaction context. Therefore, we designed two scenarios for each of the social touches, which aim at triggering contrasting emotions. To design these scenarios, we built on the circumplex model of emotions (Russell, 1980) and more specifically on its representation proposed by Alexandros and Michalis (2013). This model suggests that emotions are distributed in a two-dimensional circular space, containing arousal and valence dimensions. We have designed the scenarios to be as far apart on the arousal and valence scales as possible, while remaining meaningful. For example, one of the high five scenarios is an “enthusiastic” situation, prompted with “You just won a game where the score was tight, you are thrilled and you celebrate it with your game partner with a high five. Good job!”, and the second scenario is a “contented” situation prompted as “You are feeling down and your good friend wants to cheer you up with a high five”. The placement of the scenarios in the circumplex model is shown in Figure 2 and the complete list of the scenario prompts are in Table 1. Each scenario prompt includes the relationship with the other social agent and the context of the interaction. As can be seen in Figure 2, most scenarios are situated in the positive region of the valence axis and are spread across the arousal axis. This distribution of context scenarios was motivated by the desire to keep the interaction scenarios realistic but also engaging and not distressing the participants.

Figure 2. Circumplex model of emotion (adopted from Alexandros and Michalis, 2013) and position of each of the social touch scenario prompts (N.B. valence and arousal placement of the scenarios are speculated by the authors).

Table 1. Scenario prompt for each social touch.

2.3. Social Touch Haptic Signals

For each of the social touch patterns, we implemented a set of parameters, such as the excitement level and the duration of the haptic cues. These parameters can be varied with sliders to determine the emotional content of the tactile message. For example, a high excitement level and short duration expresses an “enthusiastic” high five. The actuators used to construct the haptic patterns, the haptic signal and the parameters are shown on Table 2.

Table 2. Description of the social touch haptic signals.

An example of an ASR signal is given in Figure 1B showing the amplitude, duration, and attack and release characteristics of the signal. To ensure haptic sensations remained perceivable, the minimum pressure for the amplitude setting was 1.2 psi. The maximum pressure was 15 psi to avoid any potential damage to actuators whilst still ensuring a strong force. During pilot studies, the 15 psi was shown to be stronger than any participant required for any of the interactions; this was confirmed in the study with all participants having settings below the maximum pressure level for all scenarios.

2.3.1. High Five

For the high five, all the actuators trigger synchronously following a trapezoidal signal with symmetrical attack and release duration (see Table 2). The multichannel actuators give normal force.

Excitement: when increasing the slider, the user increases the amplitude and decreases the attack and release durations (from 5% of the signal duration to 0%).

Duration: corresponds to the length of the high five and is proportional to the slider position. The duration can be varied between 0.08 and 0.5 s.

2.3.2. Asking for Attention

For asking for attention, five bubble actuators located on the upper palm trigger successive squared signals mimicking pokes (Baumann et al., 2010; McIntyre et al., 2019).

Excitement: this slider modifies the amplitude of a poke, and its length (between 0.04 and 1.2 s). The time between pokes is set to be the same length as the poke itself.

Duration: this slider changes the number of pokes, between one and eight.

2.3.3. Caress

For the caress, only the bubble actuators are triggered, which are divided into four groups along the hand. The group sizing was chosen to minimize complexity whilst maintaining authentic sensation based on initial trials.

Stroke rate: This slider changes the duration of one pulse from 0.2 to 1.5 s. The delay between pulses is set to 10% of the pulse duration and therefore also changes proportionally to the stroke rate. This ratio was chosen following our initial testing on both this glove and other haptic devices, and Stephens-Fripp et al. (2021) demonstrated an enhanced continuity sensation. We set the boundary of the stroke rate to be within the range of pleasant touch, 1 to 10 cm/s (McGlone et al., 2014). The attack and release duration are fixed each to 40% of the pulse duration as our initial testing on both this glove and other haptic devices demonstrated it to be the most pleasant signal (Stephens-Fripp et al., 2021).

Strength: changes the amplitude of the signal and is proportional to the slider position, which can vary between 1.2 and 15 psi.

2.3.4. Handshake

For the handshake, the actuators are triggered with two different signals: the bubble and the kinesthetic impedance actuators receive the black squared signal shown in Table 2, mimicking the grip force between hands (Knoop et al., 2017; Orefice et al., 2018). The multichannel actuators alternatively inflate and deflate as shown with the dark and light pink signals, mimicking the up and down movement of the handshake.

Strength: changes the amplitude of the signal and is proportional to the slider position, which can vary between 1.2 and 15 psi.

Excitement: sets the frequency of oscillation, between 1 and 3.33 Hz.

Duration: corresponds to the length of the handshake and is proportional to the slider position. The duration can be varied between 0.4 and 5 s.

2.3.5. Heartbeat

Number of beats: to convey heartbeats, participants could choose either “single beats” or “double beats”. “Single beats” trigger all the bubbles and the multichannel actuators at the same time, while the “double beats” alternate between two groups (the palm vs. the finger actuators).

Heart rate: this slider sets the heartbeat frequency between 50 and 220 bpm by changing the delay between the ventricles (Benson and Connolly, 2019). One beat length is inversely proportional to the heart rate slider and ranges between 0.6 and 0.1 s. As with the caress signal, the symmetrical attack and release duration are set to 40% of the beat duration. The delay between the two “double beats” is set to 40% of a beat length.

Intensity: changes the amplitude of the signal and is proportional to the slider position, which can vary between 1.2 psi and 15 psi. In the case of “double beats”, the second beat, on the finger actuator, is given at half the amplitude for a more realistic sensation.

3. Study 1: Parameter Definition for Social Haptic Patterns

3.1. User Study Description

The first user study investigated what should be the parameters of each social touch to convey a tactile message carrying a specific emotional content. 14 participants took part in this first user study (five women, eight men, and one unknown). They have been recruited within our organization and the demographics of 13 participants (one chose not to answer the background questionnaire) can be found in the Appendix 2.1 (Supplementary Material). Their task was to tune the parameters using sliders until the haptic patterns fit what they would expect in the presented scenario. Participants were seated, wearing an Oculus head mounted display, with the haptic glove on their right hand, holding the Oculus controller in their left hand to interact with the user interface (UI), and wearing noise canceling headphones playing white noise. We conducted the experiment in a VR environment to control the participants' visual feedback and prevent distractions by the real environment. Figure 3 displays the VR environment that represents a living room. In this environment, participants were also sitting at a table facing a 2D panel with which they could interact using a glove-like left hand pointing finger. The participants' task was to first select the social touch and scenario to work on. The order was pseudo-randomized and dictated by the experiment facilitator. Then, they tuned the parameter values to fit the scenario prompts (see Table 1). The UI in Figure 3 gives the example of the excitement and duration sliders for a high five. Participants had no time limit and could try the haptic pattern as many times as necessary. When they were satisfied with the resulting haptic pattern, they used the sliders shown below to rate their confidence level in the tuning and how close they thought their tuning was to a real social touch.

Figure 3. VR environment with the UI presented to the participants for the first user study. The upper panel is used to choose the social touch, and the lower panel to design the haptic patterns and rate them for each scenario. The insert on the lower left shows the glove-like left hand pointing finger used to interact with the UI.

Participants also filled out a questionnaire about their demographics, and we measured their extroversion and agreeableness with a personality test (Goldberg, 1990). We also assessed their comfort with physical interaction using the CIT scale (Webb and Peck, 2015). The full background questionnaire can be found in Appendix 1 (Supplementary Material).

3.2. Results

To understand how the parametric signal space maps to each contextual social touch communication, we ran a silhouette analysis with a maximum of four clusters, as more clusters would not be meaningful on our 14 data points. Any cluster with less than two data points was defined as outliers according to the silhouette coefficients using the squared Euclidean distance criteria. We reduced the number of clusters until all the clusters (except one that can be treated as an outlier) were composed of at least four participants. We plot the results in the parametric space for each tactile message as shown in Figure 4. Each data point represents the data from a single participant. Typically, according to the Euclidean distance criteria more than one cluster emerged, except for the first scenario for high five and asking for attention communications. Using a Gaussian mixture model (GMM), we then calculated the probability with which each data point belonged to one cluster or another, defining the centroids as the mean of the Gaussian distribution(s), and the cluster covariance as the non-orthogonal variance, represented by the colored meshing in Figure 4.

Figure 4. Results of the first user study with the data clustering for each scenario represented in the parametric stimulus spaces. Each data point represents one participant, the crosses the clustering centroids, and the meshing the clusters' covariances for the (A) high five, (B) asking for attention, (C) caress, (D) handshake, and (E) heartbeat.

For the high five scenario 1, tuning behavior was highly consistent. We found a single clear cluster composed of 12 participant responses (Figure 4A, left). The two additional data points were classified as outliers, not as forming an additional cluster. For scenario 2, there was greater variability in parameter tuning results across participants. Two clusters emerged (see Figure 4A, right) characterized by opposed duration parameter requirements but similar spread along the excitement axis.

For asking for attention in the context of scenario 3 (Figure 4B, left), parameter tuning behavior was highly consistent. We see a distinct cluster (red) emerge, composed of 12 of the 14 participants defined by a high excitement level and a mid-range duration on average. The same outlier criteria as applied to high five was applied here. Higher across participant variability was observed in the tuning behavior for scenario 4 (right) where responses form two clusters (light blue and purple), both characterized by shorter duration signals and excitement on the lower end of the spectrum, however, the two clusters occupy different regions of excitement in the space. The two remaining participants (dark blue) are considered as outliers.

The distribution of the participant data for caress scenario 5 is highly spread along the parametric space (Figure 4C, left). Indeed, even if data are sorted into two clusters, we can observe that the data don't aggregate in a specific area of the space. One hypothesis is that the scenario prompt “You have just spent a great day with a loved one and they are showing their affection to you” was hard for the participants to identify with, as some participants reported. A second hypothesis is that there is no universal haptic pattern to express such feeling. To validate a hypothesis, more investigation is required. Scenario 6 (right) has a more defined clustering, with the light blue cluster being defined by a slow stroke rate and high strength and the dark blue cluster by mid-range stroke rate and strength. The caress distributions show quite high variance or spread for these clusters as indicated by their covariance c = [386.57 52.67] for scenario 5 and c = [83.79 39.18] for scenario 6.

As we can see from the size of the ellipses, the handshake clustering (Figure 4D) has quite a large covariance. Both clusters of scenario 7 (left) have high excitement and mid-duration, with the red cluster having mid-strength and the orange one high strength. Scenario 8 (right) has one cluster at low excitement and high duration (light blue) while the second one (purple) has high excitement and low duration. Both clusters are spread along the strength axis.

Finally, for the heartbeat (Figure 4E), we observe two distinct clusters for each scenario. Scenario 9 (left) is defined either as single beats, low rate and high intensity heartbeat or as double beats, also with low rate, but with low intensity. Scenario 10 is defined with double beats and high rate either at low or high intensity.

To understand if there was an influence of participants' background (see Appendix 2.1 in Supplementary Material) on the parameter settings selected, confidence and realism ratings, we ran a Spearman's rank correlation analysis between each of these datasets. For none of these 200+ analyses was the participants' background significantly correlated with any parameters of interest, r₍₁₁₎ = < 0.65, p > 0.05, see Appendix 2.3 in Supplementary Material for detailed values.

We also computed the confidence and realism mean and standard deviation per cluster. The visual representation of these can be found in Appendix 2.2 (Supplementary Material). We ran t-tests that showed no significant difference between the clusters' ratings, except for the realism of the two clusters for the first heartbeat scenario, t₍₁₃₎ = 2.5363, p = 0.026.

Results of this first user study found specific data clustering, where the centroids are considered the typical parameters to convey the emotion of the scenario prompted. No rating nor participants' background enables us to determine ideal clusters between the ones found.

4. Study 2: Validation of Social Haptic Patterns

4.1. User Study Description

The aim of this second user study is to investigate how the haptic patterns generalize along message types and between users. Using the centroids of the clusters found in the first user study (see Table 3), we implemented these 18 haptic patterns and 10 naïve participants (four women, five men, one prefer not to answer, see Appendix 3.1 in Supplementary Material for more demographics data) of the second user study had to recognize them as one of the 10 possible tactile messages. One additional participant did not complete the experiment and is not included in the data analysis. Participants were recruited from the same organization pool as in user study 1. The participants used the same setup and environment as in the first user study. As shown in Figure 5, the participants tried the haptic pattern, selected the matching social touch (over five choices), and then selected the corresponding scenario (two choices). On their left, a panel reminded them of the social touch scenarios. For each trial, they also rated their confidence in both the social touch selection and the scenario selection, and rated how close they thought this haptic pattern was to a real one. Each of the 18 haptic patterns were presented 3x pseudo-randomly to the participant, and they were able to try the haptic pattern as many times as they wanted. The participants could refer to the panel with the list with the 10 scenario prompts at any time. They did not receive any feedback on their performance. Participants of the second user study also filled the same participant background questionnaire as for the first user study (see Appendix 1 in Supplementary Material).

Table 3. Results of the centroids for each social touch.

Figure 5. VR environment with the UI presented to the participants for the second user study, where they selected the corresponding social touch and scenario, and they rated the haptic signal. The left panel reminded the social touch scenarios.

4.2. Results

4.2.1. Haptic Pattern Recognition Rate

Results are represented in confusion tables (see Figures 6, 7). On the y-axis is the social touch presented to the participant, also called the true class, and on the x-axis the participant answer, or predicted class. The diagonals are the cases where the participant correctly recognized the type of social touch. We can see that all social touches were recognized well above chance level (20%).

Figure 6. Confusion table presenting the recognition rate of the social touch with the true class, the five social touch, in the y-axis and the predicted class, also the five social touch in the x-axis.

Figure 7. Confusion table presenting the recognition rate of the social touch emotional content. The y-axis displays only the cluster of the haptic patterns (true class) that had the highest correct recognition rate per scenario, and the x-axis displays the 10 possible answers (predicted class). The full dataset can be found in Appendix 3.2 (Supplementary Material). The cells outlined in gray highlight the correct social touch selection and the one in black, the correct social touch and scenario selection.

Caress was the most distinct social touch with 91.7% correct recognition, followed by high five with 86.7%. Handshake had a recognition rate of 51.7% and was often mistaken for heartbeat, which also has a “pulsation” pattern. Heartbeat was the least distinctive haptic pattern, often mistaken for asking for attention.

Figure 7 shows the results of the emotional content recognition of the tactile messages. This confusion table shows only the highest correct recognition rate per scenario on the y-axis. On the x-axis, we have the 10 tactile messages, or possible answers. A table representing the full dataset can be found in Appendix 3.2 (Supplementary Material). Each of the 18 haptic patterns has been presented three times to the 10 participants. Therefore, each of them has been rated 30x. Accordingly, 3.33% represents one selection of one participant. The cells outlined in gray highlight the correct social touch selection and the one in black, the correct social touch and scenario selection.

We can observe that the “enthusiastic” high five and the “alert” asking for attention patterns have a high recognition rate (see Table 1 for the scenario prompts), consistent with the human-human communication in McIntyre et al. (2021), who showed higher recognition rates for “happiness” and “attention”. Caress had a high recognition rate for the social touch type, but the emotional content is harder to identify. We can see it with the small rating difference between the two scenarios (between the right and left columns outlined in gray). The emotional content of both scenarios of the handshake is also difficult to recognize, and we can also observe that some haptic patterns are often selected as representing the opposite tactile message such as handshake 2a vs. handshake 2b (see Appendix 3.2 in Supplementary Material). As shown in Figure 6, some haptic patterns are mistaken for another social touch. In Figure 7, we can see more in detail which scenarios are more or less distinct. For example, handshake scenario 7 (conveying “happiness”) is often mistaken for the heartbeat scenario 10 (conveying “stress”).

4.2.2. Confidence and Realism Levels

Results for the confidence and realism levels are presented similarly as for the recognition rate in the confusion tables of Figures 8, 9, respectively. To understand whether the users confidence or realism levels could illuminate the recognition rate results (Figure 7), we computed the correlation between the recognition rates, confidence, and realism level. There is a strong correlation between the three ratings (confidence in the social touch selection, confidence in the scenario selection, and realism), and the recognition rate (Spearman's rank correlation analysis, r₍₈₎ > 0.98, p < 0.001 for all six tests, see Appendix 3.3 in Supplementary Material for detailed statistics). This indicated that in the case of an incorrect selection, the participants were relatively confident in their answer and did not choose it randomly. For example, a participant confusing the handshake for the heartbeat was quite confident in their choice and rated realism relatively high.

Figure 8. Confusion table presenting the participants' confidence rating in (A) their social touch selection and (B) their scenario selection. Only the cluster with the highest correct recognition rate per scenario is represented. The full dataset can be found in Appendix 3.2 (Supplementary Material). The cells outlined in black highlight the correct selections.

Figure 9. Results of the realism rating. (A) Confusion table representing how close to a real social touch participants rated the haptic patterns. Only the clusters with the highest correct recognition rate per scenario are represented. The full dataset can be found in Appendix 3.2 (Supplementary Material). The cells outlined in gray highlight the correct social touch selection and the one in black, the correct social touch and scenario selection. (B) Realism rating for each social touch. Each colored cross represents one participant, the black circles the mean rating per social touch and the black error bar the standard deviation.

Figure 9B displays each participant's realism rating (colored crosses), the mean rating (black circles) and the standard deviation (black error bar) of the correct answers for each social touch. We can observe that the average realism is between 54 and 61% of being considered as a real social touch, with very few differences between the social touch type. However, there is a large standard deviation between the participants, with the cyan participant rating the realism to every social touch above 67%, while the dark red participant never rated a social touch above 21%.

4.2.3. Interparticipant Analysis

There were no significant correlation between the participants' background and personality types, and their social touch and scenario selection correctness r₍₈₎ < 0.6, p > 0.05, see Appendix 3.3 in Supplementary Material.

To check if results were biased due to the novelty of the task, we analyzed whether the participants were learning along the trial and therefore whether their performance was increasing over the trials. We used a general linear mixed model (GLMM) to identify learning at the individual participant and group levels. Our logit link regression function was used to determine whether our binomial data showed a learning effect or not. We performed a single sample test to identify whether the slopes were significantly different than zero, where we hypothesized that a slope greater than zero indicates learning has occurred. We ran the analysis for both the social touch recognition rate and recognition of the scenario to which the social touch belonged. The social touch recognition rate shows no significant effect, t₍₉₎ = 1.4263, p = 0.19. However, the social touch and scenario recognition rate is increasing significantly over time, t₍₉₎ = 2.3940, p = 0.04. We ran a Bayes Factor analysis to understand the effect of this learning, which showed that the evidence is weak/anecdotal (2.08, BF10 < 3) (Jeffreys, 1998).

With this second user study, we demonstrated that the patterns derived from the first user study are generalizable to naïve users. Some of the haptic patterns were easier to recognize than others. Based on the data analysis, we could select the 10 best social touch haptic patterns to represent the scenarios with their emotional content.

5. Discussion

The first user study defined the parameter levels for both scenarios of each social touch (see Figure 4) and the second user study validated the results and pointed out which clusters were the most recognizable (see Figure 7). When the best haptic pattern is selected and the data of both scenarios are plotted in the same parametric space, we can observe that specific emotions belong to a specific area of the parametric space.

Figure 10 displays the results of the most recognizable cluster of each scenario. For the high five (a), we observe that the clusters representing both scenarios are distinct and belonging to specific areas of the parametric space, with a high excitement and low duration representing an “enthusiastic” high five (scenario 1, red cluster), while a long duration is more representative of cheering someone up (scenario 2, blue cluster). Asking for attention (b) also has distinct clusters, with high excitement and mid-range duration representing “alert” (scenario 3, red cluster) and a low excitement and short duration representing a “bored” touch (scenario 4, purple cluster). The emotional content of the caress (c) is harder to interpret as the clusters are spread and the recognition rate is low between both scenarios (see Figure 7). We surmise a high strength may convey “serenity” (scenario 5, orange cluster) while a lower strength represents “calming” (scenario 6, blue cluster). The results of the second user study helped to discern the cluster of the handshake (d). It determined that high strength and high excitement convey a “happy” handshake (scenario 7, orange cluster), while lower strength and excitement convey a more “nervous” one (scenario 8, blue cluster). However, the recognition rate differences were quite low between the clusters (see Figure 7) so further investigation is required to build stronger claims. For the heartbeat (e), we observe distinct clustering belonging to a specific area of the parametric space. A low heart rate with single beats conveys a “relaxed” state (scenario 9, orange cluster), while a high heart rate with double beats conveys “stress” (scenario 10, purple cluster).

Figure 10. Data of both scenarios represented in the same parametric space for the (A) high five, (B) asking for attention, (C) caress, (D) handshake, and (E) heartbeat. The dots represent one participant, the crosses the clustering centroids, and the meshing the clusters' covariances.

We see in the previous paragraph that there appears to be a relationship between the parameter levels and the emotional content within each social touch. We can also observe some parallels between the common parameters and our speculated location of each interaction on the circumplex model of emotion. For example, in our scenarios, there is a relationship between the excitement level and the arousal level.The higher excitement levels were often observed for the scenarios that correspond to high-arousal emotions. We can notice, however, that the handshake excitement level does not follow this trend. This may be due to the used interaction scenario, where people may want to project self-confidence and empowerment and therefore give a low excitement level in their handshake despite being nervous, however, we require cognitive interviews to validate this hypothesis. Similarly we observed the scenarios that were representing a higher valence level were tuned by the participants to have a higher strength level. These results are preliminary as we only tested two scenarios for each social touch and further investigations are required looking at multiple points across the emotional space for each touch in order to generalize these relationships for the parameters across the full emotional quadrant. In addition, the mapping of each of these scenarios on the circumplex model of emotion was chosen by the authors with internal piloting, and differences in trends may have arisen from a different interpretation of the anticipated valence and arousal of each interaction scenario. Follow up studies will require participants to map their perceived valence and arousal from the various versions of the social touch received.

Overall, the accuracy to identify correct social touch by naïve participants was 67.6% on the hand using the glove, which is comparable to the human-human communication scores of the standardized touch gestures in McIntyre et al. (2021) on the forearm (73% in experiment 3 and 65% in experiment 4). It is worth noting that these are haptic only cues, without the contextual visual information that comes with interacting with another person which is hypothesized to add to the overall realism experience.

6. Conclusion

With this research, we demonstrated that social touch with their specific emotional content can be conveyed using a pneumatic haptic glove. For the four social touches and the physiological signal, we were able to change the emotional mapping with differing valence and arousal levels (represented by a different interaction scenario) by changing the chosen haptic based parameters. To the best of the authors knowledge, this is the first study to link changing haptic based parameters to change the emotional space for mediated social touch. The link between strength and excitement with valence and arousal space respectively was consistent across the different types of social touch (with the exception of handshake's excitement). These results demonstrate the potential of creating haptic building blocks to map a social touch to the emotional spaces. However, further experimentations with more scenarios across the emotional space and run on a larger pool of participants is required to determine generalizability of these parameters. The second user study demonstrated that all the haptic patterns were recognizable by a naïve person well above chance level. Although, it appears that personalization may be required to optimize mediated social touch haptic patterns, our results indicate a level of commonality in different people's social touch language. In addition, since we only speculated the arousal/valence mapping of each scenario, future work should include the receiver's interpretation of arousal/valence mapping.

The results indicated that the emotional content of the caress and the handshake were harder to recognize in some of the trials, shown by a higher spread in the clusters. These social touches may benefit from further context and/or personalization such as tunable haptic patterns or gesture recordings on the sender side. It would be also interesting to investigate if training, or simply more familiarity with the system would further improve the recognition rate of the haptic patterns. In addition, although the studies were performed in a virtual reality environment, there was no visual and auditory information for the person to interact with. Future studies need to be developed to study how a multisensory environment and/or additional context impacts on the interaction realism.

During these two user studies, we limited the experiment to five social interactions in two different scenarios. This gave us an indication of how we can alter the different parameters to change the emotional mapping of the same social interaction in the context of haptic glove. Further studies are needed to determine how well this approach generalizes to other social touches not explored here. In addition, in future work we will explore the development of a model to enable the prediction of the required parameter levels for new interaction scenarios based on its anticipated valence and arousal mapping.

Data Availability Statement

The datasets presented in this article are not readily available because Meta Platform Inc., has a strict policy about sharing dataset. Requests to access the datasets should be directed to carinerognon@fb.com.

Ethics Statement

The studies involving human participants were reviewed and approved by Western Institutional Review Board, Inc. (WIRB), Washington, USA. The participants provided their written informed consent to participate in this study.

Author Contributions

CR, BS-F, and AI designed the study. BR made the software for the user studies. CR and BS-F mounted the user studies. CR conducted the data collection. CR and JH-O'B were responsible for data processing and statistical analyses. All authors contributed to the initial piloting study. They also all participated to the manuscript and approved the submitted version.

Funding

All authors are funded by Meta Platforms Inc.

Conflict of Interest

CR, BS-F, AI, JH-O'B and BR are employed by Meta Platforms Inc. This study was funded by Meta Platforms Inc., and others in the relevant organization participated in the related study design, collection, analysis, interpretation of data, and the decision to submit it for publication.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomp.2022.826545/full#supplementary-material

References