Abstract
The success of deep learning in computer vision has greatly increased the need for annotated image datasets. We propose an EEG (Electroencephalogram)-based image annotation system. While humans can recognize objects in 20–200 ms, the need to manually label images results in a low annotation throughput. Our system employs brain signals captured via a consumer EEG device to achieve an annotation rate of up to 10 images per second. We exploit the P300 event-related potential (ERP) signature to identify target images during a rapid serial visual presentation (RSVP) task. We further perform unsupervised outlier removal to achieve an F1-score of 0.88 on the test set. The proposed system does not depend on category-specific EEG signatures enabling the annotation of any new image category without any model pre-training.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, S., Huang, J., Huang, Y., Yu, Y., Li, H., Metaxas, D.N.: Automatic image annotation using group sparsity. In: CVPR (2010)
Verma, Y., Jawahar, C.V.: Image annotation using metric learning in semantic neighbourhoods. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 836–849. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_60
Yashaswi, V., Jawahar, C.: Exploring SVM for image annotation in presence of confusing labels. In: BMVC (2013)
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal visual object classes (VOC) challenge. IJCV 88(2), 303–338 (2010)
Fu, H., Zhang, Q., Qiu, G.: Random forest for image annotation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 86–99. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_7
Wang, M., Hua, X.-S.: Active learning in multimedia annotation and retrieval: a survey. In: TIST 2011
Sychay, G., Chang, E., Goh, K.: Effective image annotation via active learning. In: IEEE International Conference on Multimedia and Expo Proceedings, vol. 1, pp. 209–212 (2002)
Bakliwal, P., Jawahar, C.: Active learning based image annotation. In: NCVPRIPG. IEEE (2015)
Katti, H., Subramanian, R., Kankanhalli, M., Sebe, N., Chua, T.-S., Ramakrishnan, K.R.: Making computers look the way we look: exploiting visual attention for image understanding. In: ACM International Conference on Multimedia, pp. 667–670 (2010)
Subramanian, R., Shankar, D., Sebe, N., Melcher, D.: Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. J. Vis. 14(3), 1–18 (2014)
Oliva, A.: Gist of the scene. Neurobiol. Attention 696, 251–256 (2005)
Keysers, C., Xiao, D., Foldiak, P., Perrett, D.: The speed of sight. J. Cogn. Neurosci. 13, 90–101 (2001)
Linden, D.E.: The P300: where in the brain is it produced and what does it tell us? Neuroscientist 11(6), 563–576 (2005)
Mohedano, E., Healy, G., McGuinness, K., Giró-i Nieto, X., OConnor, N.E., Smeaton, A.F.: Improving object segmentation by using EEG signals and rapid serial visual presentation. Multimedia Tools Appl. 74(22), 10137–10159 (2015)
Pohlmeyer, E.A., Wang, J., Jangraw, D.C., Lou, B., Chang, S.-F., Sajda, P.: Closing the loop in cortically-coupled computer vision: a brain-computer interface for searching image databases. J. Neural Eng. 8(3), 036025 (2011)
Koelstra, S., Mühl, C., Patras, I.: EEG analysis for implicit tagging of video data. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009, pp. 1–6. IEEE (2009)
Subramanian, R., Wache, J., Abadi, M., Vieriu, R., Winkler, S., Sebe, N.: ASCERTAIN: emotion and personality recognition using commercial sensors. IEEE Trans. Affect. Comput. PP, 1 (2016)
Shukla, A., Gullapuram, S.S., Katti, H., Yadati, K., Kankanhalli, M., Subramanian, R.: Affect recognition in ads with application to computational advertising. In: ACM International Conference on Multimedia (2017)
Kapoor, A., Shenoy, P., Tan, D.: Combining brain computer interfaces with vision for object categorization. In: CVPR (2008)
Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Shah, M., Souly, N.: Deep learning human mind for automated visual classification (2017)
Bilalpur, M., Kia, S.M., Chawla, M., Chua, T., Subramanian, R.: Gender and emotion recognition with implicit user signals. In: International Conference on Multimodal Interaction (2017)
Picton, T.W., et al.: The p300 wave of the human event-related potential. J. Clin. Neurophysiol. 9, 456–456 (1992)
Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: EEGnet: a compact convolutional network for EEG-based brain-computer interfaces. arXiv preprint arXiv:1611.08024 (2016)
Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUS). arXiv preprint arXiv:1511.07289 (2015)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Paszke, A., Chintala, S., Collobert, R., Kavukcuoglu, K., Farabet, C., Bengio, S., Melvin, I., Weston, J., Mariethoz, J.: Pytorch: tensors and dynamic neural networks in python with strong GPU acceleration, May 2017. https://github.com/pytorch/pytorch
Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., Ball, T.: Deep learning with convolutional neural networks for EEG decoding and visualization. In: Human Brain Mapping, August 2017. https://doi.org/10.1002/hbm.23730
Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems, pp. 3320–3328 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)
Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)
Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106(1), 59–70 (2007)
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2012 (VOC2012) Results
Polich, J.: Updating P300: an integrative theory of p3a and p3b. Clin. Neurophysiol. 118(10), 2128–2148 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Parekh, V., Subramanian, R., Roy, D., Jawahar, C.V. (2018). An EEG-Based Image Annotation System. In: Rameshan, R., Arora, C., Dutta Roy, S. (eds) Computer Vision, Pattern Recognition, Image Processing, and Graphics. NCVPRIPG 2017. Communications in Computer and Information Science, vol 841. Springer, Singapore. https://doi.org/10.1007/978-981-13-0020-2_27
Download citation
DOI: https://doi.org/10.1007/978-981-13-0020-2_27
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0019-6
Online ISBN: 978-981-13-0020-2
eBook Packages: Computer ScienceComputer Science (R0)