Kais Riani
ABSTRACT
Distracted and drowsy driving are two very common causes of car accidents as they contribute to 2.3% of all the fatalities caused on the US roads. Therefore, in the era of smart driving there is an increased need of technologies able to monitor driver's alertness and provide timely alerts to the driver. In this paper, we conduct as pilot study and we present a preliminary, yet novel multimodal dataset, collected from 10 subjects using three different modalities. Our modalities include a thermal camera, an RGB camera, and four physiological indicators. The dataset consists of two recording sessions for each subject, thus, offering in total 20 multimodal driving sessions. We propose a machine learning framework aiming to investigate the hypothesis that multimodal features have higher potential towards driver alertness detection. Our dataset and analysis focus on exploring the differences between alertness and drowsiness as they intersect with the presence of different distractions. The results highlight the validity of our hypothesis and introduce interesting future directions for research.
Supplemental Material
- Mohamed Abouelenien, Mihai Burzo, and Rada Mihalcea. 2015. Cascaded multimodal analysis of alertness related features for drivers safety applications. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, 59.Google Scholar
Digital Library
- Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2016. Detecting deceptive behavior via integration of discriminative features from multiple modalities. IEEE Transactions on Information Forensics and Security 12, 5 (2016), 1042--1055.Google ScholarDigital Library
- Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2017. Multimodal gender detection. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, 302--311.Google ScholarDigital Library
- Tadas Baltrušaitis, Marwa Mahmoud, and Peter Robinson. 2015. Cross-dataset learning and person-specific normalisation for automatic action unit detection. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 6. IEEE, 1--6.Google ScholarDigital Library
- Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10.Google ScholarCross Ref
- Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 59--66.Google ScholarDigital Library
- Mihai Burzo, Mohamed Abouelenien, David Van Alstine, and Kristen Rusinek. 2017. Thermal discomfort detection using thermal imaging. In ASME 2017 International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers Digital Collection.Google ScholarCross Ref
- YouJun Choi, HeeSung Shin, and JaeYeol Lee. 2014. Smart steering wheel system for driver's emergency situation using physiological sensors and smart phone. In 2014 IEEE International Symposium on Innovations in Intelligent Systems and Applications (INISTA) Proceedings. IEEE, 281--286.Google ScholarCross Ref
- Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapatsoulis, George Votsis, Stefanos Kollias, Winfried Fellenz, and John G Taylor. 2001. Emotion recognition in humancomputer interaction. IEEE Signal processing magazine 18, 1 (2001), 32--80.Google Scholar
- Céline Craye, Abdullah Rashwan, Mohamed S Kamel, and Fakhri Karray. 2016. A multi-modal driver fatigue and distraction assessment system. International Journal of Intelligent Transportation Systems Research 14, 3 (2016), 173--194.Google ScholarCross Ref
- Yulun Du, Chirag Raman, Alan W Black, Louis-Philippe Morency, and Maxine Eskenazi. 2018. Multimodal Polynomial Fusion for Detecting Driver Distraction. arXiv preprint arXiv:1810.10565 (2018).Google Scholar
- Paul Ekman and Wallace V Friesen. 1978. Manual for the facial action coding system. Consulting Psychologists Press.Google Scholar
- National Center for Statistics and Analysis. (2017, October). Drowsy Driving 2015 (Crash•Stats Brief Statistical Summary. Report No. DOT HS 812 446). Washington, DC: National Highway Traffic Safety Administration.Google Scholar
- Maurizio Garbarino, Matteo Lai, Dan Bender, Rosalind W Picard, and Simone Tognetti. 2014. Empatica E3---A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In 2014 4th International Conference on Wireless Mobile Communication and Healthcare-Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH). IEEE, 39--42.Google ScholarCross Ref
- Christopher Geyer and Konstantinos Daniilidis. 1999. Catadioptric camera calibration. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 1. IEEE, 398--404.Google ScholarCross Ref
- Amogh Gudi, H Emrah Tasli, Tim M Den Uyl, and Andreas Maroulis. 2015. Deep learning based facs action unit occurrence and intensity estimation. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 6. IEEE, 1--5.Google ScholarCross Ref
- Amjad Hashemi, Valiallah Saba, and Seyed Navid Resalat. 2014. Real time driver's drowsiness detection by processing the EEG signals stimulated with external flickering light. Basic and clinical neuroscience 5, 1 (2014), 22.Google Scholar
- Christian Hessler, Mohamed Abouelenien, and Mihai Burzo. 2018. A Survey on Extracting Physiological Measurements from Thermal Images. In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference. 229--236.Google ScholarDigital Library
- National highway traffic safety administration. 2012. national statistics Available from. http://www-fars.nhtsa.dot.gov/Main/index.aspxGoogle Scholar
- Zhuoni Jie, Marwa Mahmoud, Quentin Stafford-Fraser, Peter Robinson, Eduardo Dias, and Lee Skrypchuk. 2018. Analysis of yawning behaviour in spontaneous expressions of drowsy drivers. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 571--576.Google ScholarCross Ref
- Serajeddin Ebrahimian Hadi Kiashari, Ali Nahvi, Amirhossein Homayounfard, and Hamidreza Bakhoda. 2018. Monitoring the Variation in Driver Respiration Rate from Wakefulness to Drowsiness: A Non-Intrusive Method for Drowsiness Detection Using Thermal Imaging. Journal of Sleep Sciences 3, 1-2 (2018), 1--9.Google Scholar
- Abhiram Kolli, Alireza Fasih, Fadi Al Machot, and Kyandoghere Kyamakya. 2011. Non-intrusive car driver's emotion recognition using thermal camera. In Proceedings of the Joint INDS'11 & ISTET'11. IEEE, 1--5.Google ScholarCross Ref
- Nanxiang Li and Carlos Busso. 2014. Predicting perceived visual and cognitive distractions of drivers with multimodal features. IEEE Transactions on Intelligent Transportation Systems 16, 1 (2014), 51--65.Google ScholarDigital Library
- Chin-Teng Lin, Yu-Chieh Chen, Teng-Yi Huang, Tien-Ting Chiu, Li-Wei Ko, Sheng-Fu Liang, Hung-Yi Hsieh, Shang-Hwa Hsu, and Jeng-Ren Duann. 2008. Development of wireless brain computer interface with embedded multitask scheduling and its application on real-time driver's drowsiness detection and warning. IEEE Transactions on Biomedical Engineering 55, 5 (2008), 1582--1591.Google ScholarCross Ref
- Fu-Chang Lin, Li-Wei Ko, Chun-Hsiang Chuang, Tung-Ping Su, and Chin-Teng Lin. 2012. Generalized EEG-based drowsiness prediction system by using a self-organizing neural fuzzy system. IEEE Transactions on Circuits and Systems I: Regular Papers 59, 9 (2012), 2044--2055.Google ScholarCross Ref
- Miguel Bordallo Lopez, Carlos R del Blanco, and Narciso Garcia. 2017. Detecting exercise-induced fatigue using thermal imaging and deep learning. In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 1--6.Google ScholarCross Ref
- Gary McKeown, Michel F Valstar, Roderick Cowie, and Maja Pantic. 2010. The SEMAINE corpus of emotionally coloured character interactions. In 2010 IEEE International Conference on Multimedia and Expo. IEEE, 1079--1084.Google ScholarCross Ref
- Stephan Mühlbacher-Karrer, Ahmad Haj Mosa, Lisa-Marie Faller, Mouhannad Ali, Raiyan Hamid, Hubert Zangl, and Kyandoghere Kyamakya. 2017. A driver state detection system---Combining a capacitive hand detection sensor with physiological sensors. IEEE Transactions on Instrumentation and Measurement 66, 4 (2017), 624--636.Google ScholarCross Ref
- Adrian M Owen, Kathryn M McMillan, Angela R Laird, and Ed Bullmore. 2005. N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human brain mapping 25, 1 (2005), 46--59.Google Scholar
- Michalis Papakostas, Akilesh Rajavenkatanarayanan, and Fillia Makedon. 2019. CogBeacon: A Multi-Modal Dataset and Data-Collection Platform for Modeling Cognitive Fatigue. Technologies 7, 2 (2019), 46.Google ScholarCross Ref
- Ioannis Pavlidis, James Levine, and Paulette Baukol. 2000. Thermal imaging for anxiety detection. In Proceedings IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications (Cat. No. PR00640). IEEE, 104--109.Google ScholarDigital Library
- Tiantian Qin, Judee Burgoon, and Jay F Nunamaker. 2004. An exploratory study on promising cues in deception detection and application of decision tree. In 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the. IEEE, 23--32.Google Scholar
- Tiantian Qin, Judee K Burgoon, J Pete Blair, and Jay F Nunamaker. 2005. Modality effects in deception detection and applications in automatic-deception-detection. In Proceedings of the 38th annual Hawaii international conference on system sciences. IEEE, 23b--23b.Google Scholar
- Jianbo Shi et al. 1994. Good features to track. In 1994 Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, 593--600.Google Scholar
- Sudipta N Sinha, Jan-Michael Frahm, Marc Pollefeys, and Yakup Genc. 2006. GPU-based video feature tracking and matching. In EDGE, workshop on edge computing using new commodity architectures, Vol. 278. 4321.Google Scholar
- Brian C Tefft et al. 2014. Prevalence of motor vehicle crashes involving drowsy drivers, United States, 2009-2013. Citeseer.Google Scholar
- Michel F Valstar, Timur Almaev, Jeffrey M Girard, Gary McKeown, Marc Mehu, Lijun Yin, Maja Pantic, and Jeffrey F Cohn. 2015. Fera 2015-second facial expression recognition and analysis challenge. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 6. IEEE, 1--8.Google ScholarCross Ref
- Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems 2, 1--3 (1987), 37--52.Google Scholar
Index Terms
- Towards detecting levels of alertness in drivers using multiple modalities
Recommendations
Understanding Driving Distractions: A Multimodal Analysis on Distraction Characterization
IUI '21: Proceedings of the 26th International Conference on Intelligent User InterfacesDistracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually ...
Detection and Recognition of Driver Distraction Using Multimodal Signals
Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable ...
Multimodal Detection of Drivers Drowsiness and Distraction
ICMI '21: Proceedings of the 2021 International Conference on Multimodal InteractionConsidering the ever-growing presence of automobiles around the world, ensuring the safety of those on and near roadways is of great importance. From the causes of accidents, drowsiness and distractedness are among the most consequential. In this paper,...
Comments