Kapotaksha Das
Skip Abstract Section
Abstract
Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we performed experiments with visual, physiological, and thermal information to explore potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual, physiological, and thermal groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the three modalities on identifying different types of driving distractions.
- [1] . 2017. Detecting deceptive behavior via integration of discriminative features from multiple modalities. IEEE Transactions on Information Forensics and Security 12, 5 (2017), 1042–1055.Google Scholar
Digital Library
- [2] . 2017. Real-time distracted driver posture classification. arXiv preprint arXiv:1706.09498 (2017).Google Scholar
- [3] . 2010. A novel method to monitor driver’s distractions. In Proceedings of the 28th International Conference Extended Abstracts on Human Factors in Computing Systems, Atlanta, Georgia, USA.Google Scholar
- [4] . 2016. Openface: An open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1–10.Google ScholarCross Ref
- [5] . 2013. Intelligent driver monitoring systems based on physiological sensor signals: A review. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC’13). IEEE, 282–289.Google ScholarCross Ref
- [6] . 2020. Driver stress state evaluation by means of thermal imaging: A supervised machine learning approach based on ECG signal. Applied Sciences 10, 16 (
Aug. 2020), 5673.DOI: Google ScholarCross Ref - [7] . 2015. Automatic detection of alertness/drowsiness from physiological signals using wavelet-based nonlinear features and machine learning. Expert Systems with Applications 42, 21 (2015), 7344–7355.Google ScholarDigital Library
- [8] . 2020. Development of a chest-belt-type biosignal-monitoring wearable platform system. Journal of Electrical Engineering & Technology 15, 4 (2020), 1847–1855.Google ScholarCross Ref
- [9] . 2018. Detecting negative emotions during real-life driving via dynamically labelled physiological data. In 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE, 830–835.Google ScholarCross Ref
- [10] . 2015. Estimated lifetime medical and work-loss costs of fatal injuries-United States, 2013. Morbidity and Mortality Weekly Report 64, 38 (2015), 1074–1077.Google ScholarCross Ref
- [11] . 2021. Supporting driver physical state estimation by means of thermal image processing. In International Conference on Computational Science. Springer, 149–163.Google ScholarDigital Library
- [12] . 2002. Stochastic gradient boosting. Computational Statistics & Data Analysis 38, 4 (2002), 367–378.Google ScholarDigital Library
- [13] . 1998. Heart rate as marker of sympathetic activity. Journal of Hypertension 16, 11 (1998), 1635–1639.Google ScholarCross Ref
- [14] . 2020. Monitoring of steering wheel engagement for autonomous vehicles. (
Sept. 2020).US Patent App. 16/294,541. Google Scholar - [15] . 2007. Working memory, attention control, and the N-back task: A question of construct validity.Journal of Experimental Psychology: Learning, Memory, and Cognition 33, 3 (2007), 615.Google ScholarCross Ref
- [16] . 2019. Driver’s fatigue recognition based on yawn detection in thermal images. Neurocomputing 338 (2019), 274–292.Google ScholarDigital Library
- [17] . 2011. Non-intrusive car driver’s emotion recognition using thermal camera. In Proceedings of the Joint INDS’11 & ISTET’11. IEEE, 1–5.Google ScholarCross Ref
- [18] . 2019. Real-time driver state monitoring using a CNN based spatio-temporal approach. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 3236–3242.Google ScholarDigital Library
- [19] . 2002. Collision warning timing, driver distraction, and driver response to imminent rear-end collisions in a high-fidelity driving simulator. Human Factors 44, 2 (2002), 314–334.
DOI: arXiv:PMID: 12452276. Google ScholarCross Ref - [20] . 1971. Brief reviews: Sympathetic-parasympathetic interactions in the heart. Circulation Research 29, 5 (1971), 437–445.Google ScholarCross Ref
- [21] . 2002. Classification and regression by Random Forest. R News 2, 3 (2002), 18–22.Google Scholar
- [22] . 2015. Driver distraction detection using semi-supervised machine learning. IEEE Transactions on Intelligent Transportation Systems 17, 4 (2015), 1108–1120.Google ScholarDigital Library
- [23] . 2017. Detecting exercise-induced fatigue using thermal imaging and deep learning. In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 1–6.Google ScholarCross Ref
- [24] . 2013. Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Transactions on Intelligent Transportation Systems 14, 3 (2013), 1462–1469.Google ScholarDigital Library
- [25] . 2020. Classification of driver distraction: A comprehensive analysis of feature generation, machine learning, and input measures. Human Factors 62, 6 (2020), 1019–1035.
DOI: arXiv:PMID: 31237788. Google ScholarCross Ref - [26] . 2010. ProComp infiniti/biograph infiniti biofeedback system (version 5.1. 2). Montreal, QB: Thought Technology Ltd (2010).Google Scholar
- [27] . 2017. A driver state detection system-combining a capacitive hand detection sensor with physiological sensors. IEEE Transactions on Instrumentation and Measurement 66, 4 (2017), 624–636.Google ScholarCross Ref
- [28] . 2019. Distracted Driving. https://www.cdc.gov/motorvehiclesafety/distracted_driving/index.html. (2019).
[Online; accessed 13-April-2020]. Google Scholar - [29] . 2019. Road Traffic Injuries and Deaths—A Global Problem. https://www.cdc.gov/injury/features/global-road-safety/index.html. (2019).
[Online; accessed 13-April-2020]. Google Scholar - [30] . 2020. Cost of Injury Data. https://www.cdc.gov/injury/wisqars/cost/index.html. (2020).
[Online; accessed 13-April-2020]. Google Scholar - [31] . 2019. Distracted Driving. https://www.nhtsa.gov/risky-driving/distracted-d riving. (2019).
[Online; accessed 13-April-2020]. Google Scholar - [32] . 2020. Development of Electrostatic Capacity Type Steering Sensor Using Conductive Leather.
Technical Report . SAE Technical Paper.Google ScholarCross Ref - [33] . 2019. CogBeacon: A multi-modal dataset and data-collection platform for modeling cognitive fatigue. Technologies 7, 2 (2019), 46.Google ScholarCross Ref
- [34] . 2011. Scikit-learn: Machine learning in Python. The Journal of Machine Learning Research 12 (2011), 2825–2830.Google ScholarDigital Library
- [35] . 2019. Distracted driving recognition method based on deep convolutional neural network. Journal of Ambient Intelligence and Humanized Computing (2019), 1–8.Google Scholar
- [36] . 2019. Automatic driver stress level classification using multimodal deep learning. Expert Systems with Applications 138 (2019), 112793.
DOI: Google Scholar - [37] . 2020. Towards detecting levels of alertness in drivers using multiple modalities. In Proceedings of the 13th ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA’20). Association for Computing Machinery, New York, NY, USA, Article
12 , 9 pages.DOI: Google ScholarDigital Library - [38] . 2015. Av+ ec 2015: The first affect recognition challenge bridging across audio, video, and physiological data. In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge. 3–8.Google ScholarDigital Library
- [39] . 2015. Sleep variability and cardiac autonomic modulation in adolescents–Penn State Child Cohort (PSCC) study. Sleep Medicine 16, 1 (2015), 67–72.Google ScholarCross Ref
- [40] . 2004. Monitoring mouth movement for driver fatigue or distraction with one camera. In Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No. 04TH8749). IEEE, 314–319.Google ScholarCross Ref
- [41] . 2011. Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision 91, 2 (2011), 200–215.Google ScholarDigital Library
- [42] . 2021. Methods for sweat detection in cars based on thermal images of the face. IEEE Sensors Journal (2021).Google ScholarCross Ref
- [43] . 2020. Wearable radio-frequency sensing of respiratory rate, respiratory volume, and heart rate. NPJ Digital Medicine 3, 1 (2020), 1–10.Google ScholarCross Ref
- [44] . 1994. Good features to track. In 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 593–600.Google Scholar
- [45] . 2010. Real time car driver’s condition monitoring system. In Sensors, 2010 IEEE, 951–954.Google Scholar
- [46] . 2006. GPU-based video feature tracking and matching. In EDGE, Workshop on Edge Computing Using New Commodity Architectures, Vol. 278. 4321.Google Scholar
- [47] . 2020. Intelligent wearable heart rate sensor implementation for in-vehicle infotainment and assistance. Internet of Things 12 (2020), 100277.Google ScholarCross Ref
- [48] . 2014. Classifying driver workload using physiological and driving performance data: Two field studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 4057–4066.Google ScholarDigital Library
- [49] . 2017. A multimodal dataset for various forms of distracted driving. Scientific Data 4 (2017), 170110.Google ScholarCross Ref
- [50] . 2001. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 2 (2001), 97–115.Google ScholarDigital Library
- [51] . 2019. Avoiding stress driving: Online trip recommendation from driving behavior prediction. 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom’19), 1–10.Google Scholar
- [52] . 2019. Detection of driver stress in real-world driving environment using physiological signals. In 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Vol. 1. 1807–1814.Google ScholarCross Ref
- [53] . 2019. Personalized driver workload estimation using deep neural network learning from physiological and vehicle signals. IEEE Transactions on Intelligent Vehicles (2019).Google Scholar
- [54] . 2019. Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders. Computers in Biology and Medicine 109 (2019), 159–170.
DOI: Google ScholarDigital Library - [55] . 2020. Studying personalized just-in-time auditory breathing guides and potential safety implications during simulated driving(
UMAP’20 ). Association for Computing Machinery, New York, NY, USA, 275–283.DOI: Google ScholarDigital Library
Index Terms
- Detection and Recognition of Driver Distraction Using Multimodal Signals
Recommendations
Understanding Driving Distractions: A Multimodal Analysis on Distraction Characterization
IUI '21: Proceedings of the 26th International Conference on Intelligent User InterfacesDistracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually ...
Multimodal Detection of Drivers Drowsiness and Distraction
ICMI '21: Proceedings of the 2021 International Conference on Multimodal InteractionConsidering the ever-growing presence of automobiles around the world, ensuring the safety of those on and near roadways is of great importance. From the causes of accidents, drowsiness and distractedness are among the most consequential. In this paper,...
Towards detecting levels of alertness in drivers using multiple modalities
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive EnvironmentsDistracted and drowsy driving are two very common causes of car accidents as they contribute to 2.3% of all the fatalities caused on the US roads. Therefore, in the era of smart driving there is an increased need of technologies able to monitor driver's ...
Comments