Skip to main content
SpringerLink
Log in

Real-time eye state recognition using dual convolutional neural network ensemble

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Automatic recognition of the eye states is essential for diverse computer vision applications related to drowsiness detection, facial emotion recognition (FER), human–computer interaction (HCI), etc. Existing solutions for eye state detection are either parameter intensive or suffer from a low recognition rate. This paper presents the design and implementation of a vision-based system for real-time eye state recognition on a resource-constrained embedded platform to tackle these issues. The designed system uses an ensemble of two lightweight convolutional neural networks (CNN), each trained to extract relevant information from the eye patches. We adopted transfer-learning-based fine-tuning to overcome the over-fitting issues when training the CNNs on small sample eye state datasets. Once trained, these CNNs are integrated and jointly fine-tuned to achieve enhanced performance. Experimental results manifest the effectiveness of the proposed eye state recognizer that is robust and computationally efficient. On the ZJU dataset, the proposed DCNNE model delivered the state-of-the-art recognition accuracy of 97.99% and surpassed the prior best recognition accuracy of 97.20% by 0.79%. The designed model also achieved competitive results on the CEW and MRL datasets. Finally, the designed CNNs are optimized and ported on two different embedded platforms for real-world applications with real-time performance. The complete system runs at 62 frames per second (FPS) on an Nvidia Xavier device and 11 FPS on a low-cost Intel NCS2 embedded platform using a frame size of 640 \(\times\) 480 pixels resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Zhao, L., Wang, Z., Zhang, G., Qi, Y., Wang, X.: Eye state recognition based on deep integrated neural network and transfer learning. Multimed. Tools Appl. 77(15), 19415–19438 (2018)

    Article  Google Scholar 

  2. Liu, A., Li, Z., Wang, L., Zhao, Y.: A practical driver fatigue detection algorithm based on eye state. In: 2010 Asia Pacific Conference on Postgraduate Research in Microelectronics and Electronics (PrimeAsia), IEEE, pp 235–238 (2010)

  3. Królak, A., Strumiłło, P.: Eye-blink detection system for human-computer interaction. Univ. Access Inf. Soc. 11(4), 409–419 (2012)

    Article  Google Scholar 

  4. Fuangkaew, S., Patanukhom, K.: Eye state detection and eye sequence classification for paralyzed patient interaction. In: 2013 2nd IAPR Asian Conference on Pattern Recognition, IEEE, pp 376–380 (2013)

  5. Liu, Z. T., Jiang, C.S., Li, S.H., Wu, M., Cao, W.H., Hao, M.: Eye state detection based on weight binarization convolution neural network and transfer learning. Applied Soft Computing, p 107565 (2021)

  6. Liu, Z., Ai, H.: Automatic eye state recognition and closed-eye photo correction. In: 2008 19th International Conference on Pattern Recognition, IEEE, pp 1–4 (2008)

  7. Belkacem, A.N., Saetia, S., Zintus-art, K., Shin, D., Kambara, H., Yoshimura, N., Berrached, N., Koike, Y.: Real-time control of a video game using eye movements and two temporal eeg sensors. Computational intelligence and neuroscience, 2015 (2015)

  8. Ahad, M.A.R., Kobashi, S., Tavares, J.M.R.: Advancements of image processing and vision in healthcare, (2018)

  9. Piatek, Ł, Fiedler, P., Haueisen, J., et al.: Eye state classification from electroencephalography recordings using machine learning algorithms. Digit. Med. 4(2), 84 (2018)

    Article  Google Scholar 

  10. Zhou, Z., Li, P., Liu, J., Dong, W.: A novel real-time eeg based eye state recognition system. In: International Conference on Communications and Networking in China, Springer, pp 175–183 (2018)

  11. Song, F., Tan, X., Liu, X., Chen, S.: Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recogn. 47(9), 2825–2838 (2014)

    Article  Google Scholar 

  12. Zhang, B., Wang, W., Cheng, B.: Driver eye state classification based on cooccurrence matrix of oriented gradients. Adv. Mech. Eng. 7(2), 707106 (2015)

    Article  Google Scholar 

  13. Dong, Y., Zhang, Y., Yue, J., Hu, Z.: Comparison of random forest, random ferns and support vector machine for eye state classification. Multimed. Tools Appl. 75(19), 11763–11783 (2016)

    Article  Google Scholar 

  14. Gou, C., Wu, Y., Wang, K., Wang, K., Wang, F.Y., Ji, Q.: A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recognit. 67, 23–31 (2017)

    Article  Google Scholar 

  15. Chowdhury, A.I., Niloy, A.R., Sharmin, N., et al.: A deep learning based approach for real-time driver drowsiness detection. In: 2021 5th International conference on electrical engineering and information & communication technology (ICEEICT), IEEE, pp 1–5 (2021)

  16. Fitriyani, N.L., Yang, C.K., Syafrudin, M.: Real-time eye state detection system using haar cascade classifier and circular hough transform. In: 2016 IEEE 5th Global Conference on Consumer Electronics, IEEE, pp 1–3 (2016)

  17. Li, B., Fu, H.: Real time eye detector with cascaded convolutional neural networks. Applied Computational Intelligence and Soft Computing (2018)

  18. Ahmed, N.Y.: Real-time accurate eye center localization for low-resolution grayscale images. J. Real-Time Image Proc. 18(1), 193–220 (2021)

    Article  Google Scholar 

  19. Yu, M., Tang, X., Lin, Y., Schmidt, D., Wang, X., Guo, Y., Liang, B.: An eye detection method based on convolutional neural networks and support vector machines. Intell. Data Anal. 22(2), 345–362 (2018)

    Article  Google Scholar 

  20. Mandal, B., Li, L., Wang, G.S., Lin, J.: Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans. Intell. Transp. Syst. 18(3), 545–557 (2016)

    Article  Google Scholar 

  21. Ji, Y., Wang, S., Lu, Y., Wei, J., Zhao, Y.: Eye and mouth state detection algorithm based on contour feature extraction. J. Electron. Imaging 27(5), 051205 (2018)

    Article  Google Scholar 

  22. Yang, H.Y., Jiang, X.H., Wang, L., Zhang, Y.H.: Eye statement recognition for driver fatigue detection based on gabor wavelet and hmm. Appl. Mech. Mater. Trans. Tech. Publ. 128, 123–129 (2012)

    Google Scholar 

  23. Zhou, L., Wang, H.: Open/closed eye recognition by local binary increasing intensity patterns. In: 2011 IEEE 5th International Conference on Robotics, pp. 7–11. Automation and Mechatronics (RAM), IEEE (2011)

  24. Yan, P., Yan, D., Du, C.: Design and implementation of a driver’s eye state recognition algorithm based on perclos. Chin. J. Electron. 4, 669–672 (2014)

    Google Scholar 

  25. Sun, C., Li, J.H., Song, Y., Jin, L.: Real-time driver fatigue detection based on eye state recognition. Appl. Mech. Mater. Trans. Tech. Publ. 457, 944–952 (2014)

    Google Scholar 

  26. Wu, Y.S., Lee, T.W., Wu, Q.Z., Liu, H.S.: An eye state recognition method for drowsiness detection. In: 2010 IEEE 71st Vehicular Technology Conference, IEEE, pp 1–5 (2010)

  27. Aing, L., Kondo, T., Nilkhamhang, I., Bunnun, P., Kaneko, H.: Eye state recognition using the hamming distances of eye image intensities. In: 2017 8th International Conference of Information and Communication Technology for Embedded Systems (IC-ICTES), IEEE, pp 1–5 (2017)

  28. Liu, X., Tan, X., Chen, S.: Eyes closeness detection using appearance based methods. In: International Conference on Intelligent Information Processing, Springer, pp 398–408 (2012)

  29. Kim, K.W., Hong, H.G., Nam, G.P., Park, K.R.: A study of deep cnn-based classification of open and closed eyes using a visible light camera sensor. Sensors 17(7), 1534 (2017)

    Article  Google Scholar 

  30. Rahman, M.M., Islam, M.S., Jannat, M.K.A., Rahman, M.H., Arifuzzaman, M., Sassi, R., Aktaruzzaman, M.: Eyenet: An improved eye states classification system using convolutional neural network. In: 2020 22nd International Conference on Advanced Communication Technology (ICACT), IEEE, pp 84–90 (2020)

  31. Geng, L., Yin, H., Xiao, Z., Xi, J.: Eye state recognition method for drivers with glasses. In: Journal of Physics: Conference Series, IOP Publishing, vol 1213, p 052049 (2019)

  32. Dehnavi, M., Eshghi, M.: Design and implementation of a real time and train less eye state recognition system. EURASIP J. Adv. Signal Process. 1, 30 (2012)

    Article  Google Scholar 

  33. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7310–7311 (2017)

  34. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10(Jul), 1755–1758 (2009)

    Google Scholar 

  35. Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2018)

    Article  MathSciNet  Google Scholar 

  36. Fusek, R.: Pupil localization using geodesic distance. In: International Symposium on Visual Computing, Springer, pp 433–444 (2018)

  37. Eddine, B.D., Dos Santos, F.N., Boulebtateche, B., Bensaoula, S.: Eyelsd a robust approach for eye localization and state detection. J. Signal Process. Syst. 90(1), 99–125 (2018)

    Article  Google Scholar 

  38. Gorbachev, Y., Fedorov, M., Slavutin, I., Tugarev, A., Fatekhov, M., Tarkan, Y.: Openvino deep learning workbench: Comprehensive analysis and tuning of neural networks inference. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp 0–0 (2019)

  39. Ditty, M., Karandikar, A., Reed, D.: Nvidia’s xavier soc. In: Hot Chips: A Symposium on High Performance Chips, (2018)

  40. Vanholder, H.: Efficient inference with tensorrt, (2016)

  41. Zhang, J., Liu, Y., Liu, H., Wang, J., Zhang, Y.: Distractor-aware visual tracking using hierarchical correlation filters adaptive selection. Applied Intelligence, pp 1–19 (2021a)

  42. Zhang, J., Sun, J., Wang, J., Yue, X.G.: Visual object tracking based on residual network and cascaded correlation filters. J. Ambient. Intell. Hum. Comput. 12(8), 8427–8440 (2021)

    Article  Google Scholar 

  43. Zhang, J., Jin, X., Sun, J., Wang, J., Sangaiah, A.K.: Spatial and semantic convolutional features for robust visual object tracking. Multimed. Tools Appl. 79(21), 15095–15115 (2020)

    Article  Google Scholar 

  44. Zhang, J., Jin, X., Sun, J., Wang, J., Li, K.: Dual model learning combined with multiple feature selection for accurate visual tracking. IEEE Access 7, 43956–43969 (2019)

    Article  Google Scholar 

  45. Leng, L., Li, M., Kim, C., Bi, X.: Dual-source discrimination power analysis for multi-instance contactless palmprint recognition. Multimed. Tools Appl. 76(1), 333–354 (2017)

    Article  Google Scholar 

  46. Leng, L., Zhang, J.: Palmhash code vs. palmphasor code. Neurocomputing 108, 1–12 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the director, CSIR-CEERI, Pilani for supporting and encouraging research activities at CSIR-CEERI, Pilani. Constant motivation by the group head, Intelligent Systems Group (ISG) at CSIR-CEERI is also acknowledged.

Author information

Authors and Affiliations

  1. Academy of Scientific and Innovative Research, Ghaziabad, Uttar Pradesh, 201002, India

    Sumeet Saurav, Ravi Saini & Sanjay Singh

  2. CSIR-Central Electronics Engineering Research Institute, Pilani, Rajasthan, 333031, India

    Sumeet Saurav, Prashant Gidde, Ravi Saini & Sanjay Singh

Authors
  1. Sumeet Saurav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prashant Gidde
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ravi Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanjay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sumeet Saurav.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saurav, S., Gidde, P., Saini, R. et al. Real-time eye state recognition using dual convolutional neural network ensemble. J Real-Time Image Proc 19, 607–622 (2022). https://doi.org/10.1007/s11554-022-01211-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-022-01211-5

Keywords

Search

Navigation