Xiao Zeng
Mi Zhang
ABSTRACT
Correct identification of prescription pills based on their visual appearance is a key step required to assure patient safety and facilitate more effective patient care. With the availability of high-quality cameras and computational power on smartphones, it is possible and helpful to identify unknown prescription pills using smartphones. Towards this goal, in 2016, the U.S. National Library of Medicine (NLM) of the National Institutes of Health (NIH) announced a nationwide competition, calling for the creation of a mobile vision system that can recognize pills automatically from a mobile phone picture under unconstrained real-world settings. In this paper, we present the design and evaluation of such mobile pill image recognition system called MobileDeepPill. The development of MobileDeepPill involves three key innovations: a triplet loss function which attains invariances to real-world noisiness that deteriorates the quality of pill images taken by mobile phones; a multi-CNNs model that collectively captures the shape, color and imprints characteristics of the pills; and a Knowledge Distillation-based deep model compression framework that significantly reduces the size of the multi-CNNs model without deteriorating its recognition performance. Our deep learning-based pill image recognition algorithm wins the First Prize (champion) of the NIH NLM Pill Image Recognition Challenge. Given its promising performance, we believe MobileDeepPill helps NIH tackle a critical problem with significant societal impact and will benefit millions of healthcare personnel and the general public.
- Amazon Shopping Mobile App. https://itunes.apple.com/us/app/amazon-app-shop-scan-compare/id297606951?mt=8.Google Scholar
- Google Translate. https://play.google.com/store/apps/details?id=com.google.android.apps.translate&hl=en.Google Scholar
- Monsoon Power Monitor. https://www.msoon.com/LabEquipment/PowerMonitor/.Google Scholar
- NIH NLM Pill Image Recognition Challenge. https://pir.nlm.nih.gov/challenge/.Google Scholar
- PWRcheck DC power analyzer. http://www.westmountainradio.com/product_info.php?products_id=pwrcheck.Google Scholar
- Your Prescriptions and Your Privacy. https://www.privacyrights.org/consumer-guides/your-prescriptions-and-your-privacy-california-medical-privacy-series.Google Scholar
- S. Bhattacharya and N. D. Lane. Sparsification and separation of deep learning layers for constrained resource inference on wearables. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, pages 176--189. ACM, 2016. Google ScholarDigital Library
- J. J. Caban, A. Rosebrock, and T. S. Yoo. Automatic identification of prescription drugs using shape distribution models. In 2012 19th IEEE International Conference on Image Processing, pages 1005--1008. IEEE, 2012.Google ScholarCross Ref
- T. Y.-H. Chen, L. Ravindranath, S. Deng, P. Bahl, and H. Balakrishnan. Glimpse: Continuous, real-time object recognition on mobile devices. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, pages 155--168. ACM, 2015. Google ScholarDigital Library
- E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pages 1269--1277, 2014. Google ScholarDigital Library
- S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135--1143, 2015. Google ScholarDigital Library
- S. Han, H. Shen, M. Philipose, S. Agarwal, A. Wolman, and A. Krishnamurthy. Mcdnn: An approximation-based execution framework for deep stream processing under resource constraints. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '16, pages 123--136, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- K. He and J. Sun. Convolutional neural networks at constrained time cost. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5353--5360, 2015.Google ScholarCross Ref
- K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770--778, 2016.Google ScholarCross Ref
- G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.Google Scholar
- F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 1mb model size. arXiv preprint arXiv:1602.07360, 2016.Google Scholar
- A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725--1732, 2014. Google ScholarDigital Library
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012. Google ScholarDigital Library
- N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao, L. Qendro, and F. Kawsar. Deepx: A software accelerator for low-power deep learning inference on mobile devices. In 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pages 1--12. IEEE, 2016. Google ScholarDigital Library
- Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436--444, 2015.Google ScholarCross Ref
- Y.-B. Lee, U. Park, A. K. Jain, and S.-W. Lee. Pill-id: Matching and retrieval of drug pill images. Pattern Recognition Letters, 33(7):904--910, 2012. Google ScholarDigital Library
- M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.Google Scholar
- M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279, 2016.Google Scholar
- F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815--823, 2015.Google ScholarCross Ref
- K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.Google Scholar
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1--9, 2015.Google ScholarCross Ref
- Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701--1708, 2014. Google ScholarDigital Library
- B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages 487--495, 2014. Google ScholarDigital Library
Index Terms
- MobileDeepPill: A Small-Footprint Mobile Deep Learning System for Recognizing Unconstrained Pill Images
Recommendations
Experiences in mHealth for chronic disease management in 4 countries
ISABEL '11: Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication TechnologiesThis paper describes mHealth applications to deal with Non Communicable Diseases in North and Latin America: In Chile, a project focused on Diabetes Mellitus type 2; In the United States, Honduras, and Mexico, projects focused in diabetes, heart failure,...
Development of a Point-of-Care HIV/Aids Medication Dosing Support System Using the Android Mobile Platform
Medication dosing errors can greatly reduce HIV treatment effectiveness as incorrect dosing leads to drug resistance and non-adherence. In order to dose correctly, HIV therapy providers must balance several patient characteristics such as renal ...
Wireless networks, physician handhelds use, and medical devices in U.S. hospitals
WH '15: Proceedings of the conference on Wireless HealthAs wireless technologies such as smartphones, tablets, and other handheld devices continue to permeate into every aspect of everyday life, hospitals have attempted to integrate them into the clinical workflow. Similarly, wireless sensor devices are ...
Comments