Abstract
In this article, we propose a mobile food recognition system that uses the picture of the food, taken by the user’s mobile device, to recognize multiple food items in the same meal, such as steak and potatoes on the same plate, to estimate the calorie and nutrition of the meal. To speed up and make the process more accurate, the user is asked to quickly identify the general area of the food by drawing a bounding circle on the food picture by touching the screen. The system then uses image processing and computational intelligence for food item recognition. The advantage of recognizing items, instead of the whole meal, is that the system can be trained with only single item food images. At the training stage, we first use region proposal algorithms to generate candidate regions and extract the convolutional neural network (CNN) features of all regions. Second, we perform region mining to select positive regions for each food category using maximum cover by our proposed submodular optimization method. At the testing stage, we first generate a set of candidate regions. For each region, a classification score is computed based on its extracted CNN features and predicted food names of the selected regions. Since fast response is one of the important parameters for the user who wants to eat the meal, certain heavy computational parts of the application are offloaded to the cloud. Hence, the processes of food recognition and calorie estimation are performed in cloud server. Our experiments, conducted with the FooDD dataset, show an average recall rate of 90.98%, precision rate of 93.05%, and accuracy of 94.11% compared to 50.8% to 88% accuracy of other existing food recognition systems.
- www.obesitynetwork.ca.Google Scholar
- http://www.who.int.Google Scholar
- Parisa Pouladzadeh, Shervin Shirmohammadi, and Abdulsalam Yassine. 2016. You are what you eat: So, measure what you eat. IEEE Instrum. Meas. Mag. 19, 1, 9--15. Google ScholarCross Ref
- Parisa Pouladzadeh, Pallavi Kuhad, Sri Vijay Bharat Peddi, Abdulsalam Yassine, and Shervin Shirmohammadi. 2016. Calorie measurement and food classification using deep learning neural network In Proceedings of the IEEE International Conference on Instrumentation and Measurement Technology (I2MTC’16).Google Scholar
- P. Pouladzadeh, A. Yassine, and S. Shirmohammadi. 2015 FooDD: Food detection dataset for calorie measurement using food images, in new trends in image analysis and processing. In Proceedings of the ICIAP 2015 Workshops, V. Murino, E. Puppo, D. Sona, M. Cristani, and C. Sansone (eds.). Lecture Notes in Computer Science, Springer, Vol. 9281, 441--448. DOI:10.1007/978-3-319-23222-5_54 Google ScholarDigital Library
- Sri Vijay Bharat Peddi, Abdulsalam Yassine, and Shervin Shirmohammadi. 2015. Cloud based virtualization for a calorie measurement e-health mobile application. In Proceedings of the 2015 International Conference on Multimedia and Expo Workshops (ICME’15). 1--6Google ScholarCross Ref
- Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101--mining discriminative components with random forests. In Proceeding of the European Conference on Computer Vision--(ECCV’14). 446--461.Google ScholarCross Ref
- Yuji Matsuda, Hajime Hoashi, and Keiji Yanai. 2012. Recognition of multiple-food images by detecting candidate regions. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’12). 25--30. Google ScholarDigital Library
- Yoshihiro Kawano and Katsuki Yanai. 2013. Real-time mobile food recognition system. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’13). 1--7. Google ScholarDigital Library
- Weiyu Zhang, Qian Yu, Behjat Siddiquie, Ajay Divakaran, and Harpreet Sawhney. 2015. Food recognition and nutrition estimation on a smartphone. J. Diabetes Sci. Technol. 9, 3, 525--533. Google ScholarCross Ref
- Satoru Fujishige. 2005. Submodular Functions and Optimization, Vol. 58. Elsevier.Google Scholar
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Cafe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia. 675--678.Google ScholarDigital Library
- Laurence Awolsey. 1982. An analysis of the greedy algorithm for the submodular set covering problem. Combinatorica 2, 4, 385--393. Google ScholarCross Ref
- Fengqing Zhu, Marc Bosch, Nitin Khanna, and Carol J. Boushey. 2015. Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE J. Biomed. Health Informat. 19, 1, 377--389. Google ScholarCross Ref
- Hokuto Kagaya, Kiyoharu Aizawa, and Makoto Ogawa. 2014. Food detection and recognition using convolutional neural network. In Proceedings of the ACM International Conference on Multimedia, 1085--1088. Google ScholarDigital Library
- Kiyoharu Aizawa and Makoto Ogawa. 2015. FoodLog: Multimedia tool for healthcare applications. IEEE MultiMed. 22, 2, 4--9. Google ScholarCross Ref
- Sosuke Amano, Kiyoharu Aizawa, and Makoto Ogawa. 2015. Food category representatives: Extracting categories from meal names in food recordings and recipe data. In Proceedings of the IEEE International Conference on Multimedia Big Data. 48--55. DOI:10.1109/BigMM.2015.54 Google ScholarDigital Library
- Colin Ware. 2008. Toward a perceptual theory of flow visualization. IEEE Comput. Graph. Appl. 28, 2 (2008), 6--11. DOI:http://dx.doi.org/10.1109/MCG.2008.39 Google ScholarDigital Library
- Xi-Jin Zhang, Yi-Fan Lu, and Song-Hai Zhang. 2016. Multi-task learning for food identification and analysis with deep convolutional neural networks. J. Comput. Sci. Technol. 31, 3, 489--500. Google ScholarCross Ref
- Morteza Akbari Fard, Hamed Haddadi, and Alireza Tavakoli Targhi. 2016. Fruits and vegetables calorie counter using, convolutional neural networks. In ACM Dig. Health, 121--122.Google Scholar
- Ashutosh Singla, Lin Yuan, and Touradj Ebrahimi. 2016. Food/non-food image classification and food categorization using pre-trained googlenet model. In Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management. 3--11. Google ScholarDigital Library
- Wataru Shimoda Keiji Yanai. 2016. Foodness proposal for multiple food detection by training of single food images. In Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management. 13--21.Google Scholar
- Joachim Dehais, Marios Anthimopoulos, and Stavroula Mougiakakou. 2016. Food image segmentation for dietary assessment. In Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management. 23--28. Google ScholarDigital Library
- A. Krizhevsky, I. Sutskever, and G. Hinton, 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of Neural Information Processing Systems (NIPS’12).Google Scholar
- M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean and G. E. Hinton. 2013. Onrectied linear units for speech processing. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP’13).Google Scholar
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998. “Gradientbased learning applied to document recognition. Proc. IEEE, 86, 11: 2278--2324. Google ScholarCross Ref
- Parisa Pouladzadeh, Shervin Shirmohammadi, and Rana Almaghrabi. 2014. Measuring calorie and nutrition from food image. IEEE Trans. Instrument. Measure. 63, 8, 1947--1956. Google ScholarCross Ref
- Z. Su, Q. XU, M. Fei, and M. Dong. 2016. Game theoretic resource allocation in media cloud with mobile social users. IEEE Trans. Multimed. (TMM) 18, 8, 1650--1660.Google ScholarDigital Library
Index Terms
- Mobile Multi-Food Recognition Using Deep Learning
Recommendations
Food Detection and Recognition Using Convolutional Neural Network
MM '14: Proceedings of the 22nd ACM international conference on MultimediaIn this paper, we apply a convolutional neural network (CNN) to the tasks of detecting and recognizing food images. Because of the wide diversity of types of food, image recognition of food items is generally very difficult. However, deep learning has ...
Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model
MADiMa '16: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary ManagementRecent past has seen a lot of developments in the field of image-based dietary assessment. Food image classification and recognition are crucial steps for dietary assessment. In the last couple of years, advancements in the deep learning and ...
Smartphone-based food recognition system using multiple deep CNN models
AbstractPeople with blindness or low vision utilize mobile assistive tools for various applications such as object recognition, text recognition, etc. Most of the available applications are focused on recognizing generic objects. And they have not ...
Comments