Abstract
Recent advancements in technology have made available a variety of image capturing devices, ranging from handheld mobiles to space-grade rovers. This has generated a tremendous visual data, which has made a necessity to organize and understand this visual data. Thus, there is a need to caption thousands of such images. This has resulted in tremendous research in computer vision and deep learning. Inspired by such recent works, we present an image caption generating system that uses convolutional neural network (CNN) for extracting the feature embedding of an image and feed that as an input to long short-term memory cells that generates a caption. We are using two pre-trained CNN models on ImageNet, VGG16 and ResNet-101. Both the models were tested and compared on Flickr8K dataset. Experimental results on this dataset demonstrate that the proposed architecture is as efficient as the state-of-the-art multi-label classification models. Experimental results on public benchmark dataset demonstrate that the proposed architecture performs as efficiently as the state-of-the-art image captioning model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dai, J., He, K., Sun, J.: BoxSup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: IEEE International Conference on Computer Vision ICCV, pp 1635–1643 (2015). arXiv:1503.01640v2
Das, S.: CNN architectures: Lenet, alexnet, vgg, googlenet, resnet and more (2017). https://medium.com/@sidereal/cnns-architectures-lenetalexnet-vgg-googlenet-resnet-and-more666091488df5
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., FeiFei, L.: Imagenet: a large-scale hierarchical image database. IEEE Comput. Vis. Patt Recogn. (CVPR) 248–255 (2009)
Fang, H., Gupta, S., Iandola, F., Srivastava, R., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J.C., Zitnick, C.L., Zweig, G.: From captions to visual concepts and back. In: IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp 1473–1482 (2015). arXiv:1411.4952v3
Frossard, D.: VGG in tensorflow (2016). https://www.cs.toronto.edu/~frossard/post/vgg16/
Guillaumin, M., Verbeek, J., Schmid, C.: Multimodal semi-supervised learning for image classification. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770–778, June 2016. arXiv:1512.03385v1
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Intell. Res. 47, 853–899 (2013)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. ACM international conference on Multimedia, pp. 675–678, Nov 2014. arXiv:1408.5093v1
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Patt. Anal. Mach. Intell. 14(8) (2015)
Kiros, R., Salakhutdinov, R., Zemel, R.: Multimodal neural language models. In International Conference on Machine Learning, pp 595–603 (2014)
Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. In: Computing Research Repository (CoRR) in Machine Learning (2014). arXiv: 1411.2539v1
Lin, T.-Y., Maire, M., Belongie, S.. Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Springer European Conference on Computer Vision ECCV, pp. 740–755. Springer, Berlin (2014)
Olah, C.: Understanding LSTM networks (2015). http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Redmon, J., Farhadi, A.: YOLO 9000: better, faster, stronger. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2007. arXiv:1612.08242v1
Simonyan, M., Zisserman, A.: Very deep convolutional networks for large scale image recognition. In: International Conference on Learning Representations (ICLR) (2015). arXiv:1409.1556v6
Soh, M.: Learning CNN-LSTM architectures for image caption generation. In: Stanford University, Proceeding (2016)
Szegedy, C., Toshev, A., Erhan, D.: Deep neural network for object detection. In: Proceeding of Neural Information Processing Systems (NIPS 2013)
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.: Translating videos to natural language using deep recurrent neural networks. In: Annual Conference of the North American Chapter of the Association for Computational Linguistics (ACL) (2015). arXiv:1412.4729v3
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and Tell: A Neural Image Caption Generator (2015). arXiv:1411.4555v2
Wang, X., Zhu, Z., Yao, C., Bai, X.: Relaxed multiple-instance SVM with application to object discovery. In: IEEE International Conference on Computer Vision (2015)
Xu, K., Ba, J.L., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2016). arXiv:1502.03044v3
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Bhalekar, M., Sureka, S., Joshi, S., Bedekar, M. (2020). Generation of Image Captions Using VGG and ResNet CNN Models Cascaded with RNN Approach. In: Agarwal, S., Verma, S., Agrawal, D. (eds) Machine Intelligence and Signal Processing. MISP 2019. Advances in Intelligent Systems and Computing, vol 1085. Springer, Singapore. https://doi.org/10.1007/978-981-15-1366-4_3
Download citation
DOI: https://doi.org/10.1007/978-981-15-1366-4_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1365-7
Online ISBN: 978-981-15-1366-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)