Keywords

1 Introduction

Human-computer interaction has been experiencing a trend of personal data analytics in recent years: fitness trackers, smart watches, and wearables allow consumers to monitor themselves during sports, to track sleep patterns, to watch their calories, and much more. This phenomenon is widely known as the quantified self movement [11]. The technology behind this trend, namely pattern recognition through machine learning, can also be applied to other data sources. In automotive research, this has been happening for some time in the form of fatigue detection from steering and video data [13, 33] or the deduction of cognitive load from pupil diameters [37]. Driver states on an emotional level have also been extracted from physiological data, speech recordings, and facial images [23, 38]. Detecting driver states in real-time might be one of the big upcoming topics in future cars, as it enables personalized and affective interfaces [17]. At the same time, it is yet unclear whether this state should be communicated to the driver, and if personalization can help promoting acceptance of such new features. In our vision, user-aware automotive UIs can inform the driver if they show signs of hazardous affective states and ultimately increase road safety by raising awareness for the impacts of the driver state on driving performance.

We designed three approaches for the graphical representation of driver states based on successful examples from previous work in the automotive and other domains. They were evaluated in a public driving simulator experiment (N = 328). We report insights on the participants’ user experience and emotional facial expressions, analyzed by age groups, personality traits, and driving experience.

According to our core findings, we can see that older and inexperienced drivers are in favor of reduced graphical representations as they fear distraction. Users with high scores in neuroticism like a continuous display of their driver state, while drivers with high scores in extraversion rather prefer notifications in case their state could threaten safe driving. We also show that a constant display of the driver state in a GUI works more efficiently in reducing negative states than visual notifications. This work is meant as a foundation for practitioners and researchers working on user state displays in the automotive context and other multi-task environments.

2 Background and Related Work

The idea of visualizing information on user states has been long established by computer gaming and in recent years also found its way into everyday life through step counters or fitness trackers. We introduce this concept into an automotive environment. In this process we have to consider the challenges and opportunities automotive user interfaces bring to the table.

2.1 Affective Automotive User Interfaces

Cars are the perfect environment to detect the user’s state, as everything happens in a confined, easily observable space. In-vehicle applications can also consult information collected from their environment sensing systems to understand a situation’s larger context. An example of this was shown in an early approach for an integrated driver safety and wellness system presented by Coughlin et al. [13]. They propose a sequence framework built on the three steps: “detect, display, alert”. Especially while driving, the user should not be distracted or strained with visual load, which can be achieved through a mix of continuous display and proactive interventions [22, 27].

A detection of the driver state, for example on an emotional level, can be realized using image-video processing techniques and speech analysis [26]. This approach can also be combined with physiological data [21]. Thirunavukkarasu et al. have shown a prototype for an in-car HMI which detects emotions and synchronizes the data with information from an on-board diagnosis (OBD) interface in order to trigger interventions [48]. We take up from their work with an approach of classification with camera-based detection of facial action coding units [16, 30] as previously demonstrated by, for example, Abdic et al. [1].

Emotional states can be related to critical behavior behind the wheel: not only anger, but also sadness has been connected to poorer driving performance [6, 24]. Several approaches for automotive UIs have worked with this knowledge, enhancing driving safety by adapting the UI to the driver’s emotional state [34] or increasing interactivity by mirroring the driver’s emotions in natural language interfaces [35]. Apart from these reactive approaches, prophylaxis of negative states could also be interesting to investigate. We can learn from experiments in other domains, for example, Tsujita and Rekimoto’s smart home appliance which achieved positive influences on emotions by making users smile [49], or we consider gamification approaches in automotive environments, which try to bring joy and driver safety together by focusing the driver’s attention to the driving task in a playful manner [43, 46]. One current challenge in automotive research is bringing user experience and safety together, as measures to make the driver more aware of their current state might raise reactance when users feel patronized by the system [9].

2.2 Visualizing the User State

Visual interfaces are the most regulated information channels in the automotive domain as they can demand visual resources and thus distract the driver from events on the road [39]. To avoid dangerous interactions in situations with increased workload, in-car information systems should make use of mostly static visualizations and ideally include the driving context into the timing of notifications [47]. In current in-car systems, a limited number of notifications are used to communicate the car’s states (low fuel level, speed limits, etc.), which we need to adapt to when designing interventions based on the driver sate.

Related work on driver state visualization is scarce, which is why we also looked at concepts from other domains. We especially searched for ways to make drivers aware of their current state and, if possible, induce positive change. This is based on work by Choe et al., who describe the promotion of self-reflection and the support of self-experimentation as the primary goals for the design of supporting systems [11]. Fan et al. provide guidelines for quantified-self visualizations, which suggest the use of graphs or abstract geometry, preferably placed within an interaction space of daily usage (like the car) [18]. This approach was used for our concept Quantified Self. A second concept (Gamification) was inspired by the idea of gamification, meaning the introduction of game elements to motivate users towards a specific behavior, as demonstrated by Steinberger et al. with a gamified speedometer to encourage safer driving [45]. Our third concept Notification is based on notification systems, which are used to nudge users into a change of behavior [14] or improve wellbeing through text interventions [25].

2.3 Personalization of User Interfaces

Automotive UIs are currently developed with little personalization features, although recent work suggests that an adaptation to user-specific characteristics, can increase personal reference and with it improve user experience and trust [4, 5, 41]. One widespread framework to classify such personality features is the big five inventory, also called OCEAN model due to the initials of its categories openness, conscientiousness, extraversion, agreeableness, and neuroticism [20, 31]. Chen and Donmez analyzed personality and driving data from 525 drivers and found that young drivers are rather influenced by normative behavior, while drivers over the age of 30 show interactions with their associated big five classifications [10]. They see higher risk-taking attitudes in drivers with high extraversion and low conscientiousness ratings, and increased fearfulness in drivers with high neuroticism. Arthur and Graziano looked at traffic accidents and found that high conscientiousness predicted less crashes [2]. Benfield et al. found aggressive driving to correlate with low scores in openness, agreeableness, and conscientiousness [3]. If we consider this framework for the evaluation of driver state visualizations, we might experience preferences for different systems by different personalities, which may also prove to positively modulate driving behavior.

3 Focus Group: Attitudes Towards Driver State Detection

Despite the possibilities we see in driver state detection, we experienced mixed feedback and skepticism towards such concepts when talking to potential users in preparation for this study. We had the impression that users were afraid of being spied on by their cars, and similar fears have been identified in a survey by Schmidt and Braunger [40]. To understand where these notions come from, we conducted two focus groups with experienced drivers (\({>}10.000\) km per year). The groups consisted of 24 participants from the US and 21 from Germany, as these are the two main vehicle markets we had access to. 25 of the participants were women, 20 men. All described themselves as generally open towards digital innovations. All participants used a smartphone daily. 8 owned a smartwatch, 14 regularly used a fitness tracker of whom 12 belong to the US sample.

The goal of the focus groups was to get an impression of how users receive the idea of user state detection in the car. Generally speaking, one third of the participants stated that they would agree to share personal data to use digital services, another third would agree only under certain conditions. US participants were more open towards data usage than Germans.

When confronted with the idea of an in-vehicle information system which can detect and act upon the driver’s or passengers’ emotions through sensors, both groups expressed curiosity. However, German users often described it as an“invasion of privacy” while US participants stated things like “I love this technology” and it could be a “good and handy offer”. General privacy concerns and the fear of being patronized were stated as negative implications. On the other hand, an improvement of driver safety and enhanced smart suggestions were perceived positively.

Participants were then shown a prototype of an ordinary automotive UI with a speedometer and navigation, which we enhanced with visualizations of physiological features and emotion detection (Fig. 1). This design is well known from fitness tracking apps and was proposed as a potential UI to indicate driver state. The visual prototype received more positive feedback than the verbalized ideas, as it was perceived as simple and intuitive to interpret.

Fig. 1.
figure 1

Our prototype of an affective interface showed driver state data in the instrument cluster. Focus group participants (US and Germany) expressed general curiosity and potential for improving driver safety, but also concerns regarding paternalism.

Full size image

3.1 Focus Group Insights

The feedback from this focus group was condensed into four key learnings using thematic analysis [8]. We incorporated these take-aways into the proposed concepts and used them to guide our main study.

Show, Don’t Tell. When asked to evaluate an idea, the visual concept received more positive feedback than a textual description. Since a participant’s understanding of a concept profits from seeing a concrete application, we strive to keep theoretical input at a minimum.

Respect Cultural Diversity. Participants from different cultures showed diverging concerns. Germans are more aware of data privacy, Americans were immediately keen to use the system. Different demographics might bring new insights and participants’ needs may adapt (cf. GDPR in the EU/CSL in China).

Beware of Patronism. User concerns about a feeling of being patronized should be taken seriously. 35 users stated to only want information or suggestions. Only 5 would accept a system trying to influence the driver subconsciously.

Emphasize Safety Features. Participants often mentioned the advantages of driver state detection in case of a medical emergency and on long, tiring rides. Users are aware of the dangers on the road and might be more open to new technologies when they are seen in connection to safety improvements.

4 System Design

Based on the focus group, we designed three interface prototypes to be tested with a larger sample of participants. As a main requirement, each concept should be easy to use from the start, without further explanations. This way we make sure to gather feedback on the idea itself and not the technology behind it.

Participants should experience the prototype in a driving context, thus we have to embed the driver state visualizations into a realistic dashboard design and adhere to standards for in-car UIs, meaning the display should only require minimal active attention while driving. The concepts should also avoid appearing manipulative or patronizing. The ideas for each concept come from related work, which we have illuminated closer in Sect. 2.2.

4.1 Concept Notification

Concept Notification implements interventions to positively influence the driver without constant driver state display. In case the lower threshold for detected driver valence is reached, a text message meant to cheer up the driver is displayed (see Fig. 2). This message stays on the screen for at least 3 s to prevent fidgetiness and only fades out as the detected valence levels rise. The main idea of concept Notification is to reduce visual load and to only interact with the driver in form of interventions when they are in a negative state.

Fig. 2.
figure 2

Concept Notification: a minimalistic background animation combined with text notifications in case of negative user valence.

Full size image

4.2 Concept Gamification

Concept Gamification consists of four scales, representing car status, fuel consumption, driver health, and driver fun levels (Fig. 3). The inspiration was taken from computer games, where players have health levels they can directly influence through power-ups. In our prototype, users could keep consumption low by driving responsibly or increase their personal fun factor by smiling. Fun levels decreased over time if no positive expressions were detected. The general idea stems from approaches of increasing driver engagement through gamification [45] and the notion that the act of smiling itself makes people feel happier [49]. In our prototype the values for driver health was simulated with random positive values as we did not collect health data during the experiment.

Fig. 3.
figure 3

Concept Gamification displays current stats of car and driver in an interactive way. The general idea is to involve the driver and make their ride more fun.

Full size image
Fig. 4.
figure 4

Concept Quantified Self visualizes user vitality and emotion data in circular diagrams. Users can see at one glance whether their condition is classified as positive (green), neutral (blue), or negative (red). (Color figure online)

Full size image

4.3 Concept Quantified Self

Concept Quantified Self is borrowed from the visualization style know from fitness tracking apps. This display variant follows the idea of a quantified self, allowing the user to experiment on themselves through behavioral changes with an intended outcome of self-improvement [11, 42]. Two circular diagrams display the vital and emotional states of the drivers, which are color coded to allow quick information extraction (Fig. 4). When a dangerous driver state is detected, the outline turns red, which we refer to as intervention in the experiment. Here, again, we simulated driver health values, as we did not collect health data.

4.4 Prototype Implementation

The system was built using the Affdex SDK [32] which analyzes facial images to estimate an emotional user state. We process a live video feed of the driver’s face and then classify the driver state conditional to the resulting metrics: measurements exceeding anger values of 60 (out of 100) or falling below a valence of \(-50\) (from \(-100\) to 100) triggered interventions in concepts Notification and Quantified Self. Users were classified as happy when joy was above 60 or when a smile was detected. Additionally, in concept Quantified Self the valence values were transferred one-to-one to the fill percentage of the ring. In each session only one, single, visualization concept was experienced by each user.

5 Deployment Study: Driver State Visualization

The described concepts were tested in a publicly accessible driving simulator deployed close to a busy shopping street in Nuremberg, Germany. We chose Germany over the US as German participants in our focus group were more critical towards the concept of driver state detection. Nuremberg was selected as the city has no direct connection to the automotive industry, so we expected a more generalizable sample than in other cities. In order to minimize bias effects, all company affiliations and characteristic design features were also concealed.

5.1 Methodology

The unique setting in a public space allowed us to open participation for all passers-by, providing us with a much more diverse participant sample than a standard lab test would allow for. The test ran for three months, Monday through Saturday, from 10 am to 6 pm, during which close to 800 user sessions were logged. We modeled the public simulator setup according to best practices from related work, e.g. sturdy hardware, easy setup for the user, and short, concise explanations for every single step within the process [7].

Each participant experienced one visualization approach assigned randomly, resulting in a between-subject design. As additional independent variables we queried user demographics (gender, age, driving experience) and personality traits (BFI-20 Big Five Inventory [15]). The dependent variables were attractiveness, design quality, and usability, assessed using the user experience questionnaire (UEQ [29]), and emotion recognition data based on facial features from a frontal video feed.

5.2 Demographics

During a period of 12 weeks, 798 sessions were logged, of which 328 contained complete and valid measurements and no exclusion criteria (e.g. too young to drive, intoxication, unserious behavior). For an overview of age and gender distributions, see Fig. 5. When asked about driving experience, 6.4% of included participants stated they never drive themselves, 12.5% drive once a month, 13.1% once a week, 26.1% every other day and 41.8% identified themselves as daily drivers. Of the final subset, 33.8% of participants experienced the concepts Notification and Gamification and 32.4% experienced the concept Quantified Self.

Fig. 5.
figure 5

A total of 109 women and 219 men with a mean age of 36 years finished the experiment with evaluable data.

Full size image

5.3 Apparatus

The study was conducted in a white label seating box situated at a research booth at Fraunhofer JOSEPH’S in Nuremberg [19]. The concepts were shown on a curved wide screen display above the steering wheel. (Figure 6) A camera on top of the screen provided a live video stream for facial expression analysis. The simulation was controlled by the driver using a steering wheel and pedals. Participants were provided step by step instructions on a tablet placed in the center console. We provided general information on our research and explained the technology behind the prototype on a nearby information screen. Staff assured smooth operations by answering questions. Participation was possible without personal on-boarding.

Fig. 6.
figure 6

Publicly deployed medium fidelity driving simulation showing a participant experiencing the concept Gamification, displayed above the steering wheel.

Full size image

5.4 Procedure

Participants were welcomed by a starting screen and introduced to the setting and tasks. They were presented questions on their demographics, driving experience and exclusion criteria (for example, alcohol consumption, legal driving age). The main task consisted of following a white car on a highway with moderate traffic, the secondary task was assessing the UI on the dashboard screen.

After completing the ride, which took about 5 min, users were presented two questionnaires; a UEQ questionnaire to rate the user experience of the driver state visualization [29] and the BFI-20 short version of the Big Five Inventory to classify the participant’s personality [15]. We also logged the output of the emotion detection system to see whether interventions had an effect on participants. After the ride, participants were asked to rate the idea of driver state visualizations as negative, neutral, or positive and explain their decisions.

5.5 Limitations

The driving simulator experiment was conducted without further restrictions on participation and a rather short experiment duration. This is attributed to the public study design, which in return allowed for a larger sample than traditional driving studies. Furthermore, although we take into account cultural particularities, our focus group does not reflect input from China as the third big market. This is simply due to a lack of access. We can, however, expect an open attitude from Chinese users, as they are generally appreciative of new technology [28].

6 Results

We collected UEQ ratings on the tested visualization concepts, emotion recognition data from the Affdex SDK and an estimation of participants’ personality characteristics provided by the BFI-20 questionnaire. The personality characteristics and demographic data were combined and used to create participant subsets in order to understand if they experienced and evaluated concepts differently. Results with \(p < 0.05\) are reported as statistically significant.

6.1 Subjective Feedback

Participants gave us an estimation on how they perceived the concept of driver state detection in the car. 12% said they would not like to use a system like the one exhibited. They voiced concerns regarding paternalism or data protection issues. 28% reacted indifferent, one of them said they liked the idea for special groups like novice drivers or to test for intoxication but they would not accept it for themselves. A majority of 60% rated the idea as positive. Feedback included the wish for the detection of fatigue or health-related conditions and an optional annexation of control through the car if the driver was found incapable. Other ideas were music selection according to the driver’s mood or windshield filters when increased pupil sensitivity is detected.

Looking back at our initial focus groups, we expected the feedback to be more narrow-minded. The hands-on presentation might have convinced participants who would have disliked theoretical concepts. We also see that younger users within this group were more open to the driver state detection than older users. There were no differences between genders.

6.2 Concept Ratings

We first analyzed the UEQ data without differentiation between user subsets with an analysis of variance and pairwise comparisons for each concept rating (see Table 1). The data distribution fulfils requirements for parametric tests. Concept Quantified Self was generally liked by the entirety of participants. Concept Gamification and Quantified Self were rated significantly more attractive than concept Notification \((p < 0.01)\), while concept Quantified Self also had significantly higher ratings in novelty than Gamification \((p < 0.05)\). Feedback on the design quality shows a tendency towards a slight disregard for concept Gamification.

This general look at the data suggests that the quantified-self approach of concepts Quantified Self could be a good first step towards displaying driver states in the car. If we however look at sub-groups within the user sample, we can generate further insights on the opportunities of personalization.

Table 1. When we look at the entirety of participants, concept Notification is perceived significantly less attractive than the other tested concepts and concept Gamification received the noticeably worst ratings in Design Quality. Only concept Quantified Self scored well in all dimensions of the UEQ.
Full size table

Demographic Influences. Our dataset contains demographic information on the subjects, which we used to create subsets based on gender, age, and driving experience. This deep dive into demographics was only possible due to our public setup, as it allowed us to include a high number of participants with diverse backgrounds. We discovered no gender differences but age played a role in the UEQ ratings: young drivers (aged 17–24) preferred the gamified concept, while older drivers (age groups 55–64 and 65+) favored concept Notification. This can be explained with preferences of younger drivers for gamification due to under-stimulation [44] and an increased sensitivity for visual load in older drivers [36].

Driving experience also had an effect on ratings as inactive drivers preferred the visually less demanding concept Notification, while experienced drivers have balanced ratings for all three concepts.

Fig. 7.
figure 7

UEQ ratings visualized by corresponding user personality traits.

Full size image

Personality Preferences. Each user took a personality test which led to a personality placement in the five dimensions extraversion, conscientiousness, openness, agreeableness, and neuroticism. We did an analysis of the interplay between UEQ ratings and personality traits for each concept and found high variances in answers on attractiveness and dependability from participants with high values in neuroticism and openness. High openness also led to higher variances in perspicuity. Other groups were generally comparable in ratings. It is noticeable that extremes in neuroticism, extraversion, and openness show greater variances than other categories which makes us think that these are the most important traits when it comes to the personalization of driver state visualizations. We can also see that users with high agreeableness tend to rate all concepts rather well and might therefore be a less ideal demographic to chose from when conducting evaluations.

It is interesting to see that concept Notification, which gives proactive feedback and behaves thus rather extroverted, was rated higher by extroverted users, while concept Quantified Self achieved better ratings from participants with high neuroticism. This makes sense since an emotionally reactive person may also be more receptive to the constant display of their state (see Fig. 7).

Fig. 8.
figure 8

An exemplary view of emotion recognition data for detected valence values of one user: moving mean smoothing is used to visualize trends, linear regression shows the overall slope to categorize positive or negative development. Intervention frames display how this user was twice alerted due to low valence and e.g. in the second instance instantly started smiling.

Full size image

6.3 Emotion Detection

We analyzed the data provided by the Affdex SDK for emotion detection with a focus on time intervals after interventions performed by the system. Concepts Quantified Self and Notification performed different actions when the detected valence of the user fell below a threshold of \(-50\) (from \(-100\) to 100). In concept Notification, a written message appeared which was designed to bring the driver into a more positive state and only disappeared when an improvement was measured. Concept Quantified Self constantly displayed the valence values and switched to an alerting red color scheme when the value was less than the threshold. Figure 8 shows exemplary valence data of a user performing a ride with concept Notification: they experienced two interventions of 6 and 7 s duration and both led to a brief improvement. In the second intervention they actively reacted with a smile as we can see from the high valence values.

We analyzed the development of valence values after each of the total of 937 interventions in our dataset and see significantly better performance in concept Quantified Self than in concept Notification (see Fig. 9). Interventions in concept Notification also took significantly longer (\(t^{(611.43)} = -16.527, p < 0.001\)), supposedly because the written text was harder to process than the abstract representation of concept Quantified Self.

Fig. 9.
figure 9

Interventions during the concepts Quantified Self (495 instances) and Notification (442 instances). Concept Quantified Self worked more effectively in inducing positive valence through interventions.

Full size image

7 Discussion

In this paper we present three different approaches to visualizing driver states. Participants in our study were rather open towards the idea of user state detection in the car (only 12% opposed) with seniors being more sceptical than young drivers. Some users also expressed issues concerning privacy or data protection.

Concept Quantified Self, which incorporates quantified-self visualizations, is generally more preferred than Notification and Gamification. The gamified approach Gamification, however, scores well with the subset of young drivers (cf. related literature [45]). Older and unpracticed drivers prefer the more subtle concept Notification. This shows that users are actually aware of the dangers visual distraction poses on driving. We think a system with real-time cognitive load measurement could recognize available resources and select the appropriate means to visualize information, e.g. an interaction-free visualization in demanding situations and a more involving approach when the driver is bored [44].

Participants with extreme ratings on the big five scales of openness, extraversion, and neuroticism showed the most diverging UEQ ratings. Extroverted users preferred proactive behavior, which resembles an extroverted agent. Neurotic users liked a constant display of their state, possibly because they see this as a way to get feedback on their condition, motivated through higher emotional reactiveness. Users with high agreeableness scores liked all approaches, which was to be expected in a way, and suggests that we do not need to focus our future work on this character trait. Conscientiousness scores also did not have any influence on UX ratings. We however know from related work that this dimension is connected with driving performance, hence still important.

According to the analyzed valence data from image-based emotion detection we found that concept Quantified Self works faster and is more effective in decreasing negative emotions than concept Notification (concept Gamification did not utilize interventions). We see this as an important finding of our study, as current fatigue warning systems work with notifications. We see potential in defining a dedicated area within the in-vehicle UI to display the driver state to keep the driver always in the loop.

8 Conclusion and Future Work

We investigated user preferences on driver state visualization in the car. We conducted a focus group and evaluated three proposed GUIs in a public driving simulator study. The findings lead us to believe there is no one-fits-all solution to driver state visualizations. We propose that interfaces should adapt to age, driving experience, and personality. When testing with users, the personality dimensions openness, extraversion, and neuroticism should be at the focus of attention. We are also aware that personality is a highly complex field to explore.

The main contribution of our focus group and study are a high acceptance of driver state detection within our broad user sample, as well as the positive influences a constant display can have on the driver’s affective state. We hope this affirms other researchers to keep working on user-aware interfaces.

This field offers plenty of directions for future work. One interesting question that emerged while we were working on this project, is whether users will accept such a system in long-term usage or if we will see an abandonment after habitual usage, as was reported for fitness trackers [12]. Furthermore, cultural differences for personalization might play an important role and should be investigated. Therefore, we would like to motivate future research on the long-term usage of user-aware in-vehicle UIs and try to transfer these research questions into culturally different markets.