Skip to main content
Log in

TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

We study gaze estimation on tablets; our key design goal is uncalibrated gaze estimation using the front-facing camera during natural use of tablets, where the posture and method of holding the tablet are not constrained. We collected a large unconstrained gaze dataset of tablet users, labeled Rice TabletGaze dataset. The dataset consists of 51 subjects, each with 4 different postures and 35 gaze locations. Subjects vary in race, gender and in their need for prescription glasses, all of which might impact gaze estimation accuracy. We made three major observations on the collected data and employed a baseline algorithm for analyzing the impact of several factors on gaze estimation accuracy. The baseline algorithm is based on multilevel HoG feature and Random Forests regressor, which achieves a mean error of 3.17 cm. We perform extensive evaluation on the impact of various practical factors such as person dependency, dataset size, race, wearing glasses and user posture on the gaze estimation accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Baluja, S., Pomerleau, D.: Non-intrusive gaze tracking using artificial neural networks. Tech. rep, DTIC Document (1994)

  2. Benfold, B., Reid, I.: Stable multi-target tracking in real-time surveillance video. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 3457–3464. IEEE (2011)

  3. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  MATH  Google Scholar 

  4. Brolly, X.L., Mulligan, J.B.: Implicit calibration of a remote gaze tracker. In: Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04. Conference on, pp. 134–134. IEEE (2004)

  5. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

    Article  Google Scholar 

  6. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, pp. 886–893. IEEE (2005)

  7. Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2012)

    Article  Google Scholar 

  8. Drewes, H., De Luca, A., Schmidt, A.: Eye-gaze interaction for mobile phones. In: Proceedings of the 4th International Conference on Mobile Technology, Applications, and Systems and the 1st International Symposium on Computer Human Interaction in Mobile Technology, pp. 364–371. ACM (2007)

  9. Drucker, H., Burges, C.J., Kaufman, L., Smola, A., Vapnik, V., et al.: Support vector regression machines. Adv. Neural Inf. Process. Syst. 9, 155–161 (1997)

    Google Scholar 

  10. Fanelli, G., Gall, J., Van Gool, L.: Real time head pose estimation with random regression forests. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 617–624. IEEE (2011)

  11. Frischen, A., Bayliss, A.P., Tipper, S.P.: Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychol. Bull. 133(4), 694 (2007)

    Article  Google Scholar 

  12. Hansen, D.W., Ji, Q.: In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 478–500 (2010)

    Article  Google Scholar 

  13. Hennessey, C., Noureddin, B., Lawrence, P.: A single camera eye-gaze tracking system with free head motion. In: Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, pp. 87–94. ACM (2006)

  14. Huang, J., Shao, X., Wechsler, H.: Face pose discrimination using support vector machines (svm). In: Pattern Recognition, 1998. Proceedings. Fourteenth International Conference on, vol. 1, pp. 154–156. IEEE (1998)

  15. Kim, K.N., Ramakrishna, R.: Vision-based eye-gaze tracking for human computer interface. In: Systems, Man, and Cybernetics, 1999. IEEE SMC’99 Conference Proceedings. 1999 IEEE International Conference on, vol. 2, pp. 324–329. IEEE (1999)

  16. Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye tracking for everyone. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2176–2184 (2016)

  17. Kunze, K., Ishimaru, S., Utsumi, Y., Kise, K.: My reading life: towards utilizing eyetracking on unmodified tablets and phones. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 283–286. ACM (2013)

  18. Li, S.Z., Zhu, L., Zhang, Z., Blake, A., Zhang, H., Shum, H.: Statistical learning of multi-view face detection. In: Computer Vision ECCV 2002, pp. 67–81. Springer (2002)

  19. Lu, F., Okabe, T., Sugano, Y., Sato, Y.: Learning gaze biases with head motion for head pose-free gaze estimation. Image Vis. Comput. 32(3), 169–179 (2014)

    Article  Google Scholar 

  20. Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Adaptive linear regression for appearance-based gaze estimation. IEEE Trans. Pattern Anal. Mach. Intell. 36, 2033–2046 (2014)

    Article  Google Scholar 

  21. Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye appearance: a head pose-free method via eye image synthesis. IEEE Trans. Image Process. 24(11), 3680–3693 (2015)

    Article  MathSciNet  Google Scholar 

  22. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, vol. 2, pp. 674–679. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA (1981)

  23. Maeder, A., Fookes, C., Sridharan, S.: Gaze based user authentication for personal computer applications. In: Intelligent Multimedia, Video and Speech Processing, 2004. Proceedings of 2004 International Symposium on, pp. 727–730. IEEE (2004)

  24. Maji, S., Berg, A.C., Malik, J.: Classification using intersection kernel support vector machines is efficient. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. IEEE (2008)

  25. Martinez, F., Carbone, A., Pissaloux, E.: Gaze estimation using local features and non-linear regression. In: Image Processing (ICIP), 2012 19th IEEE International Conference on, pp. 1961–1964. IEEE (2012)

  26. Merchant, J., Morrissette, R., Porterfield, J.L.: Remote measurement of eye direction allowing subject motion over one cubic foot of space. IEEE Trans. Biomed. Eng. 4, 309–317 (1974)

    Article  Google Scholar 

  27. Mora, K.A.F., Monay, F., Odobez, J.M.: Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 255–258. ACM (2014)

  28. Mora, K.A.F., Odobez, J.M.: Person independent 3d gaze estimation from remote rgb-d cameras. In: Image Processing (ICIP), 2013 20th IEEE International Conference on, pp. 2787–2791. IEEE (2013)

  29. Nagamatsu, T., Iwamoto, Y., Kamahara, J., Tanaka, N., Yamamoto, M.: Gaze estimation method based on an aspherical model of the cornea: surface of revolution about the optical axis of the eye. In: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, pp. 255–258. ACM (2010)

  30. Nagamatsu, T., Yamamoto, M., Sato, H.: Mobigaze: development of a gaze interface for handheld mobile devices. In: CHI’10 Extended Abstracts on Human Factors in Computing Systems, pp. 3349–3354. ACM (2010)

  31. Naish-Guzman, A., Holden, S.: The generalized fitc approximation. In: Advances in Neural Information Processing Systems, pp. 1057–1064 (2007)

  32. Ohno, T., Mukawa, N., Yoshikawa, A.: Freegaze: a gaze tracking system for everyday gaze interaction. In: Proceedings of the 2002 symposium on Eye tracking research & applications, pp. 125–132. ACM (2002)

  33. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

    Article  Google Scholar 

  34. Perez, A., Cordoba, M.L., Garcia, A., Mendez, R., Munoz, M.L., Pedraza, J.L., Sanchez, F.: A precise eye-gaze detection and tracking system. In: 11th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, pp. 105–108. UNION Agency, Plzen (2003)

  35. Porikli, F.: Integral histogram: a fast way to extract histograms in cartesian spaces. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, pp. 829–836. IEEE (2005)

  36. Rasmussen, C.E., Christopher, K.W.: Gaussian Processes For Machine Learning, vol. 1. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  37. Raytchev, B., Yoda, I., Sakaue, K.: Head pose estimation by nonlinear manifold learning. In: Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, vol. 4, pp. 462–466. IEEE (2004)

  38. Schneider, T., Schauerte, B., Stiefelhagen, R.: Manifold alignment for person independent appearance-based gaze estimation. In: Pattern Recognition (ICPR), 2014 22nd International Conference on, pp. 1167–1172. IEEE (2014)

  39. Shi, J., et al.: Good features to track. In: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, pp. 593–600. IEEE (1994)

  40. Shih, S.W., Wu, Y.T., Liu, J.: A calibration-free gaze tracking technique. In: Pattern Recognition, 2000. Proceedings. 15th International Conference on, vol. 4, pp. 201–204. IEEE (2000)

  41. Smith, B.A., Yin, Q., Feiner, S.K., Nayar, S.K.: Gaze locking: Passive eye contact detection for human-object interaction. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, pp. 271–280. ACM (2013)

  42. Sugano, Y., Matsushita, Y., Sato, Y.: Learning-by-synthesis for appearance-based 3d gaze estimation. In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 1821–1828. IEEE (2014)

  43. Tan, K.H., Kriegman, D., Ahuja, N.: Appearance-based eye gaze estimation. In: Applications of Computer Vision, 2002.(WACV 2002). Proceedings. Sixth IEEE Workshop on, pp. 191–195. IEEE (2002)

  44. Tomasi, C., Kanade, T.: Detection and tracking of point features. Carnegie Mellon University, Tech. Rep. CMU-CS- 91-132 (1991)

  45. Tsang, I.W., Kwok, J.T., Cheung, P.M.: Core vector machines: fast svm training on very large data sets. J. Mach. Learn. Res. 6, 363–392 (2005)

    MathSciNet  MATH  Google Scholar 

  46. Vatahska, T., Bennewitz, M., Behnke, S.: Feature-based head pose estimation from images. In: Humanoid Robots, 2007 7th IEEE-RAS International Conference on, pp. 330–335. IEEE (2007)

  47. Wang, J., Sung, E., Venkateswarlu, R.: Eye gaze estimation from a single image of one eye. In: Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, pp. 136–143. IEEE (2003)

  48. Williams, C.K., Rasmussen, C.E.: Gaussian processes for regression. In: Advances in neural information processing systems, pp. 514–520 (1996)

  49. Williams, O., Blake, A., Cipolla, R.: Sparse and semi-supervised visual mapping with the s\(^{3}\)gp. In: Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 230–237. IEEE (2006)

  50. Wollaston, W.H.: On the apparent direction of eyes in a portrait. Philos. Trans. R. Soc. Lond. 114, 247–256 (1824)

    Article  Google Scholar 

  51. Wood, E., Bulling, A.: Eyetab: model-based gaze estimation on unmodified tablet computers. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 207–210. ACM (2014)

  52. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg, J.M.: Detecting eye contact using wearable eye-tracking glasses. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 699–704. ACM (2012)

  53. Yu, S.: Harr feature cart-tree based cascade eye detector homepage. http://yushiqi.cn/research/eyedetection

  54. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2015)

  55. Zhang, Z., Hu, Y., Liu, M., Huang, T.: Head pose estimation in seminar room using multi view face detectors. In: Multimodal Technologies for Perception of Humans, pp. 299–304. Springer (2007)

Download references

Acknowledgements

We acknowledge the support from National Science Foundation (NSF) Grants NSF-IIS: 1116718, NSF-CCF:1117939 and NSF-CNS:1429047. We would further like to thank all the participants in the dataset for volunteering and allowing their data to be released.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Qiong Huang.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Q., Veeraraghavan, A. & Sabharwal, A. TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets. Machine Vision and Applications 28, 445–461 (2017). https://doi.org/10.1007/s00138-017-0852-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0852-4

Keywords

Navigation