Kapotaksha Das
ABSTRACT
Considering the ever-growing presence of automobiles around the world, ensuring the safety of those on and near roadways is of great importance. From the causes of accidents, drowsiness and distractedness are among the most consequential. In this paper, we use a multimodal dataset consisting of 11 recorded channels over 45 subjects to model driver’s drowsiness and distraction. Our work puts forward the application of this dataset by using segmented windows as features, resulting in four main contributions. We explore the performance of each individual modality and specify which signals and features have a better capability of detecting drowsiness and different kinds of distractions. In addition, we analyze the effects of early fusion on the classification of the driver’s state using multiple physiological and thermal channels. Finally, we use cascaded late fusion and test three voting strategies to evaluate the performance of our proposed approach. Our results confirm the effectiveness of utilizing a multimodal approach in detecting both drowsiness and distraction as two separate factors influencing the driver and provide guidelines on which signals are appropriate for detecting different driver’s states.
- [n.d.]. 100-Car Naturalistic Study Fact Sheet. https://www.csg.org/sslfiles/dockets/2011cycle/31B/31Bbills/100-Car_Fact-Sheet.pdf. Accessed: 2021-05-12.Google Scholar
- [n.d.]. Distracted Driving 2018. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812926. Accessed: 2021-05-22.Google Scholar
- [n.d.]. NHTSA Drowsy Driving Research and Program Plan. https://www.nhtsa.gov/sites/nhtsa.gov/files/drowsydriving_strategicplan_030316.pdf. Accessed: 2021-04-28.Google Scholar
- [n.d.]. Teens and Distracted Driving 2018. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812931. Accessed: 2021-05-22.Google Scholar
- [n.d.]. Undercounted is Underinvested: How incomplete crash reports impact efforts to save lives. https://www.nsc.org/getmedia/88c97198-b7f3-4acd-a294-6391e3b8b56c/undercounted-is-underinvested.pdf. Accessed: 2021-05-22.Google Scholar
- Mohamed Abouelenien, Mihai Burzo, and Rada Mihalcea. 2015. Cascaded Multimodal Analysis of Alertness Related Features for Drivers Safety Applications. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments (Corfu, Greece) (PETRA ’15). Association for Computing Machinery, New York, NY, USA, Article 59, 8 pages. https://doi.org/10.1145/2769493.2769505Google ScholarDigital Library
- Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2017. Detecting deceptive behavior via integration of discriminative features from multiple modalities. IEEE Transactions on Information Forensics and Security 12, 5(2017), 1042–1055.Google ScholarDigital Library
- Yehya Abouelnaga, Hesham M Eraqi, and Mohamed N Moustafa. 2017. Real-time distracted driver posture classification. arXiv preprint arXiv:1706.09498(2017).Google Scholar
- J. Todd Arnedt, Gerald Wilde, Peter Munt, and Alistair Maclean. 2001. How do prolonged wakefulness and alcohol compare in the decrements they produce on a simulated driving task?Accident; analysis and prevention 33 (06 2001), 337–44. https://doi.org/10.1016/S0001-4575(00)00047-6Google Scholar
- Yusuf Artan, Orhan Bulan, Robert P Loce, and Peter Paul. 2014. Driver cell phone usage detection from HOV/HOT NIR images. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 225–230.Google ScholarDigital Library
- Muhammad Awais, Nasreen Badruddin, and Micheal Drieberg. 2017. A hybrid approach to detect driver drowsiness utilizing physiological signals to improve system performance and wearability. Sensors 17, 9 (2017), 1991.Google ScholarCross Ref
- Karel A Brookhuis and Dick De Waard. 2010. Monitoring drivers’ mental workload in driving simulators using physiological measures. Accident Analysis & Prevention 42, 3 (2010), 898–903.Google ScholarCross Ref
- Simiao Chen, Michael Kuhn, Klaus Prettner, and David E Bloom. 2019. The global macroeconomic burden of road injuries: estimates and projections for 166 countries. The Lancet Planetary Health 3, 9 (2019), e390–e398.Google ScholarCross Ref
- Tianqi Chen and Carlos Guestrin. 2016. XGBoost. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Aug 2016). https://doi.org/10.1145/2939672.2939785Google ScholarDigital Library
- Anuva Chowdhury, Rajan Shankaran, Manolya Kavakli, and Md Mokammel Haque. 2018. Sensor applications and physiological features in drivers’ drowsiness detection: A review. IEEE Sensors Journal 18, 8 (2018), 3055–3067.Google ScholarCross Ref
- Yulun Du, Chirag Raman, Alan W. Black, Louis-Philippe Morency, and Maxine Eskénazi. 2018. Multimodal Polynomial Fusion for Detecting Driver Distraction. CoRR abs/1810.10565(2018). arxiv:1810.10565http://arxiv.org/abs/1810.10565Google Scholar
- Yulun Du, Chirag Raman, Alan W Black, Louis-Philippe Morency, and Maxine Eskenazi. 2018. Multimodal Polynomial Fusion for Detecting Driver Distraction. arXiv preprint arXiv:1810.10565(2018).Google Scholar
- Tiziana D’Orazio, Marco Leo, Cataldo Guaragnella, and Arcangelo Distante. 2007. A visual approach for driver inattention detection. Pattern recognition 40, 8 (2007), 2341–2355.Google Scholar
- Paul Ekman and Wallace V Friesen. 1978. Manual for the facial action coding system. Consulting Psychologists Press.Google Scholar
- Centers for Disease Control and Prevention. 2020. Cost of Injury Data. https://www.cdc.gov/injury/wisqars/cost/index.htmlGoogle Scholar
- Ishani Janveja, Akshay Nambi, Shruthi Bannur, Sanchit Gupta, and Venkat Padmanabhan. 2020. InSight: Monitoring the State of the Driver in Low-Light Using Smartphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–29.Google ScholarDigital Library
- Michael J Kane, Andrew RA Conway, Timothy K Miura, and Gregory JH Colflesh. 2007. Working memory, attention control, and the N-back task: a question of construct validity.Journal of Experimental Psychology: Learning, Memory, and Cognition 33, 3(2007), 615.Google ScholarCross Ref
- Serajeddin Ebrahimian Hadi Kiashari, Ali Nahvi, Hamidreza Bakhoda, Amirhossein Homayounfard, and Masoumeh Tashakori. 2020. Evaluation of driver drowsiness using respiration analysis by thermal imaging on a driving simulator. Multimedia Tools and Applications(2020), 1–23.Google Scholar
- Neslihan Kose, Okan Kopuklu, Alexander Unnervik, and Gerhard Rigoll. 2019. Real-Time Driver State Monitoring Using a CNN Based Spatio-Temporal Approach. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 3236–3242.Google ScholarDigital Library
- Miguel Bordallo Lopez, Carlos R del Blanco, and Narciso Garcia. 2017. Detecting exercise-induced fatigue using thermal imaging and deep learning. In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 1–6.Google ScholarCross Ref
- Anthony D. McDonald, Thomas K. Ferris, and Tyler A. Wiener. 2020. Classification of Driver Distraction: A Comprehensive Analysis of Feature Generation, Machine Learning, and Input Measures. Human Factors 62, 6 (2020), 1019–1035. https://doi.org/10.1177/0018720819856454 PMID: 31237788.Google ScholarCross Ref
- Rizwan Ali Naqvi, Muhammad Arsalan, Ganbayar Batchuluun, Hyo Sik Yoon, and Kang Ryoung Park. 2018. Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor. Sensors 18, 2 (2018), 456.Google ScholarCross Ref
- World Health Organization. 2020. Road traffic injuries. https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuriesGoogle Scholar
- Michalis Papakostas, Kapotaksha Das, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo. 2021. Distracted and Drowsy Driving Modeling Using Deep Physiological Representations and Multitask Learning. Applied Sciences 11, 1 (2021), 88.Google ScholarCross Ref
- Anna Persson, Hanna Jonasson, Ingemar Fredriksson, Urban Wiklund, and Christer Ahlström. 2019. Heart Rate Variability for Driver Sleepiness Classification in Real Road Driving Conditions. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 6537–6540.Google Scholar
- Xuli Rao, Feng Lin, Zhide Chen, and Jiaxu Zhao. 2019. Distracted driving recognition method based on deep convolutional neural network. Journal of Ambient Intelligence and Humanized Computing (2019), 1–8.Google Scholar
- Aashreen Raorane, Hitanshu Rami, and Pratik Kanani. 2020. Driver Alertness System using Deep Learning, MQ3 and Computer Vision. In 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS). IEEE, 406–411.Google ScholarCross Ref
- Bryan Reimer and Bruce Mehler. 2011. The impact of cognitive workload on physiological arousal in young adult drivers: a field study and simulation validation. Ergonomics 54, 10 (2011), 932–942.Google ScholarCross Ref
- Kais Riani, Michalis Papakostas, Hussein Kokash, Mohamed Abouelenien, Mihai Burzo, and Rada Mihalcea. 2020. Towards detecting levels of alertness in drivers using multiple modalities. In Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments. 1–9.Google ScholarDigital Library
- Anwesha Sengupta, Anirban Dasgupta, Aritra Chaudhuri, Anjith George, Aurobinda Routray, and Rajlakshmi Guha. 2017. A multimodal system for assessing alertness levels due to cognitive loading. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 7(2017), 1037–1046.Google ScholarCross Ref
- Jianbo Shi 1994. Good features to track. In 1994 Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, 593–600.Google Scholar
- Heung-Sub Shin, Sang-Joong Jung, Jong-Jin Kim, and Wan-Young Chung. 2010. Real time car driver’s condition monitoring system. In SENSORS, 2010 IEEE. IEEE, 951–954.Google Scholar
- Sudipta N Sinha, Jan-Michael Frahm, Marc Pollefeys, and Yakup Genc. 2006. GPU-based video feature tracking and matching. In EDGE, workshop on edge computing using new commodity architectures, Vol. 278. 4321.Google Scholar
- Oxford University Press USA. 2018. Sleep deprived people more likely to have car crashes. https://www.sciencedaily.com/releases/2018/09/180918082041.htmGoogle Scholar
- Michel F. Valstar, Timur Almaev, Jeffrey M. Girard, Gary McKeown, Marc Mehu, Lijun Yin, Maja Pantic, and Jeffrey F. Cohn. 2015. FERA 2015 - second Facial Expression Recognition and Analysis challenge. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 06. 1–8. https://doi.org/10.1109/FG.2015.7284874Google Scholar
- R. Verma, B. Mitra, and Sandip Chakraborty. 2019. Avoiding Stress Driving: Online Trip Recommendation from Driving Behavior Prediction. 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom (2019), 1–10.Google ScholarCross Ref
- K. Wang, Y. L. Murphey, Y. Zhou, X. Hu, and X. Zhang. 2019. Detection of driver stress in real-world driving environment using physiological signals. In 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Vol. 1. 1807–1814.Google Scholar
- A M Williamson and Anne-Marie Feyer. 2000. Moderate sleep deprivation produces impairments in cognitive and motor performance equivalent to legally prescribed levels of alcohol intoxication. Occupational and Environmental Medicine 57, 10 (2000), 649–655. https://doi.org/10.1136/oem.57.10.649 arXiv:https://oem.bmj.com/content/57/10/649.full.pdfGoogle ScholarCross Ref
- Amir Zadeh, Yao Chong Lim, Tadas Baltrusaitis, and Louis-Philippe Morency. 2017. Convolutional experts constrained local model for 3d facial landmark detection. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 2519–2528.Google ScholarCross Ref
- Sebastian Zepf, Neska El Haouij, Jinmo Lee, Asma Ghandeharioun, Javier Hernandez, and Rosalind W. Picard. 2020. Studying Personalized Just-in-Time Auditory Breathing Guides and Potential Safety Implications during Simulated Driving(UMAP ’20). Association for Computing Machinery, New York, NY, USA, 275–283. https://doi.org/10.1145/3340631.3394854Google ScholarDigital Library
Recommendations
Detection and Recognition of Driver Distraction Using Multimodal Signals
Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable ...
Towards detecting levels of alertness in drivers using multiple modalities
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive EnvironmentsDistracted and drowsy driving are two very common causes of car accidents as they contribute to 2.3% of all the fatalities caused on the US roads. Therefore, in the era of smart driving there is an increased need of technologies able to monitor driver's ...
Real-Time V2V Communication With a Machine Learning-Based System for Detecting Drowsiness of Drivers
This article applies machine learning to detect whether a driver is drowsy and alert the driver. The drowsiness of a driver can lead to accidents resulting in severe physical injuries, including deaths, and significant economic losses. Driver fatigue ...
Comments