Youngjun Cho
Nicolai Marquardt
ABSTRACT
We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognize 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.
Supplemental Material
- Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2015 34, 4: 110--122. Google Scholar
Digital Library
- Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild With the Materials in Context Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3479--3487.Google ScholarCross Ref
- LouAnne E. Boyd, Xinlong Jiang, and Gillian R. Hayes. 2017. ProCom: Designing and Evaluating a Mobile and Wearable System to Support Proximity Awareness for People with Autism. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 2865--2877. Google ScholarDigital Library
- William D. Callister and David G. Rethwisch. 2011. Materials science and engineering: an introduction. John Wiley & Sons NY.Google Scholar
- Victor C. Chen and Hao Ling. 2002. Time-frequency transforms for radar imaging and signal analysis. Artech House.Google Scholar
- E. Cheung and V. J. Lumelsky. 1989. Proximity sensing in robot manipulator motion planning: system and implementation issues. IEEE Transactions on Robotics and Automation 5, 6: 740--751.Google ScholarCross Ref
- Youngjun Cho, Andrea Bianchi, Nicolai Marquardt, and Nadia Bianchi-Berthouze. 2016. RealPen: Providing Realism in Handwriting Tasks on Touch Surfaces Using Auditory-Tactile Feedback. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 195-- 205. Google ScholarDigital Library
- Youngjun Cho, Nadia Bianchi-Berthouze, and Simon J. Julier. 2017. DeepBreath: Deep Learning of Breathing Patterns for Automatic Stress Recognition using LowCost Thermal Imaging in Unconstrained Settings. In the 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017, 456--463.Google Scholar
- Youngjun Cho, Munchae Joung, and Sunuk Kim. 2015. Electronic device having proximity touch function and control method thereof, US Patent, Publication Number: US2015/0062087. https://patents.google.com/patent/US20150062087A1Google Scholar
- Youngjun Cho, Simon J. Julier, Nicolai Marquardt, and Nadia Bianchi-Berthouze. 2017. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical Optics Express 8, 10: 4480--4503.Google ScholarCross Ref
- Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi. 2015. Deep Filter Banks for Texture Recognition and Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3828-- 3836.Google ScholarCross Ref
- Mike Crang and Nigel Thrift. 2002. Thinking Space. Routledge.Google Scholar
- Anind K. Dey and Jonna Häkkilä. 2008. ContextAwareness and Mobile Devices. Handbook of research on user interface design and evaluation for mobile technology: 205--217.Google Scholar
- Andrey Dimitrov and Mani Golparvar-Fard. 2014. Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections. Advanced Engineering Informatics 28, 1: 37--49. Google ScholarDigital Library
- Paul Dourish. 2004. What we talk about when we talk about context. Personal and Ubiquitous Computing 8, 1: 19--30.Google ScholarDigital Library
- Marks Eric and Teizer Jochen. Proximity Sensing and Warning Technology for Heavy Construction Equipment Operation. Construction Research Congress 2012: 981--990.Google Scholar
- Jakob Eriksson, Lewis Girod, Bret Hull, Ryan Newton, Samuel Madden, and Hari Balakrishnan. 2008. The pothole patrol: using a mobile sensor network for road surface monitoring. In Proceedings of the 6th international conference on Mobile systems, applications, and services, 29--39. Google ScholarDigital Library
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, 1440--1448. Google ScholarDigital Library
- Kotaro Hara, Vicki Le, and Jon Froehlich. 2013. Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 631--640. Google ScholarDigital Library
- Chris Harrison and Scott E. Hudson. 2008. Lightweight Material Detection for Placement-aware Mobile Computing. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08), 279--282. Google ScholarDigital Library
- Antonius Hendriks, Damian M. Lyons, and Frank Guida. 2001. Vacuum cleaner with obstacle avoidance, US Patent, Publication Number: US6226830 B1. https://patents.google.com/patent/US6226830B1Google Scholar
- Ken Hinckley and Mike Sinclair. 1999. Touch-sensing Input Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99), 223--230. Google ScholarDigital Library
- Catherine Holloway and Nick Tyler. 2013. A microlevel approach to measuring the accessibility of footways for wheelchair users using the Capability Model. Transportation planning and technology 36, 7: 636--649.Google Scholar
- H. Holone, G. Misund, and H. Holmstedt. 2007. Users Are Doing It For Themselves: Pedestrian Navigation With User Generated Content. In The 2007 International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST 2007), 91--99. Google ScholarDigital Library
- Harald Holone, Gunnar Misund, H\a akon Tolsby, and Steinar Kristoffersen. 2008. Aspects of Personal Navigation with Collaborative User Feedback. In Proceedings of the 5th Nordic Conference on Humancomputer Interaction: Building Bridges (NordiCHI '08), 182--191. Google ScholarDigital Library
- Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. 2015. Spatial Transformer Networks. In Proceedings of Advances in Neural Information Processing Systems 28 (NIPS), 2017-- 2025. Google ScholarDigital Library
- Piyawan Kasemsuppakorn and Hassan A. Karimi. 2009. Personalised routing for wheelchair navigation. Journal of Location Based Services 3, 1: 24--54. Google ScholarDigital Library
- J. Kölzer, E. Oesterschulze, and G. Deboy. 1996. Thermal imaging and measurement techniques for electronic materials and devices. Microelectronic Engineering 31, 1: 251--270.Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of Advances in Neural Information Processing Systems 25 (NIPS), 1097--1105. Google ScholarDigital Library
- Daniel Kurz. 2014. Thermal touch: Thermographyenabled everywhere touch interfaces for mobile augmented reality applications. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on, 9--16.Google Scholar
- Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 321-- 333. Google ScholarDigital Library
- Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: Thermal Imaging for Surface User Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2565--2574. Google ScholarDigital Library
- John J. Leonard and Hugh F. Durrant-Whyte. 2012. Directed Sonar Sensing for Mobile Robot Navigation. Springer Science & Business Media.Google Scholar
- C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. 2010. Exploring features in a Bayesian framework for material recognition. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 239--246.Google Scholar
- H. Liu, X. Song, J. Bimbo, L. Seneviratne, and K. Althoefer. 2012. Surface material recognition through haptic exploration using an intelligent contact sensing finger. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 52--57.Google Scholar
- J. M. Lloyd. 2013. Thermal Imaging Systems. Springer Science & Business Media.Google Scholar
- Sapan Naik and Bankim Patel. 2017. Thermal imaging with fuzzy classifier for maturity and size based nondestructive mango (Mangifera Indica L.) grading. In Emerging Trends & Innovation in ICT (ICEI), 2017 International Conference on, 15--20.Google ScholarCross Ref
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), 807--814. Retrieved from http://machinelearning.wustl.edu/mlpapers/paper_files/ icml2010_NairH10.pdf Google ScholarDigital Library
- I. Pavlidis, Peter Symosek, B. Fritz, Mike Bazakos, and Nikolaos Papanikolopoulos. 2000. Automatic detection of vehicle occupants: the imaging problemand its solution. Machine Vision and Applications 11, 6: 313-- 320. Google ScholarDigital Library
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of Advances in Neural Information Processing Systems 28 (NIPS), 91--99. Google ScholarDigital Library
- E. F. J. Ring and K. Ammer. 2012. Infrared thermal imaging in medicine. Physiological Measurement 33, 3: R33.Google ScholarCross Ref
- Marc Rioux. 1984. Laser range finder based on synchronized scanners. Applied Optics 23, 21: 3837-- 3844.Google ScholarCross Ref
- Yvonne Rogers. 2011. Interaction design gone wild: striving for wild theory. Interactions 18, 4: 58--62. Google ScholarDigital Library
- Bernardino Romera-Paredes and Philip Torr. 2015. An embarrassingly simple approach to zero-shot learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), 2152--2161. Google ScholarDigital Library
- Chandra Roychoudhuri, Al F. Kracklauer, and Kathy Creath. 2008. The nature of light: what is a photon? CRC Press.Google Scholar
- Fauzia Saeed, Siti Qamariatul, Mujib Rahman, and Alan Woodside. 2015. The state of pothole management in UK local authority. Bituminous Mixtures and Pavements VI: 153--159.Google Scholar
- Alireza Sahami Shirazi, Yomna Abdelrahman, Niels Henze, Stefan Schneegass, Mohammadreza Khalilbeigi, and Albrecht Schmidt. 2014. Exploiting Thermal Reflection for Interactive Systems. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 3483--3492. Google ScholarDigital Library
- Munehiko Sato, Shigeo Yoshida, Alex Olwal, Boxin Shi, Atsushi Hiyama, Tomohiro Tanikawa, Michitaka Hirose, and Ramesh Raskar. 2015. SpecTrans: Versatile Material Classification for Interaction with Textureless, Specular and Transparent Surfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2191--2200. Google ScholarDigital Library
- Albrecht Schmidt, Michael Beigl, and Hans-W. Gellersen. 1999. There is more to context than location. Computers & Graphics 23, 6: 893--901.Google ScholarCross Ref
- Igor V. Tetko, David J. Livingstone, and Alexander I. Luik. 1995. Neural network studies. 1. Comparison of overfitting and overtraining. Journal of Chemical Information and Computer Sciences 35, 5: 826--833.Google ScholarCross Ref
- Andrea Vedaldi and Karel Lenc. 2015. MatConvNet: Convolutional Neural Networks for MATLAB. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15), 689--692. Google ScholarDigital Library
- Michael Vollmer and MÃ Klaus-Peter. 2017. Infrared thermal imaging: fundamentals, research and applications. John Wiley & Sons.Google Scholar
- Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi. 2016. A 4D Light-Field Dataset and CNN Architectures for Material Recognition. In Computer Vision -- ECCV 2016 (Lecture Notes in Computer Science), 121--138.Google Scholar
- Jason Wiese, T. Scott Saponas, and A.J. Bernheim Brush. 2013. Phoneprioception: Enabling Mobile Phones to Infer Where They Are Kept. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 2157--2166. Google ScholarDigital Library
- Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. RadarCat: Radar Categorization for Input & Interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 833--841. Google ScholarDigital Library
- Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. SpeCam: Sensing Surface Color and Material with the Frontfacing Camera of a Mobile Device. In Proceedings of the 19th International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '17), 25:1--25:9. Google ScholarDigital Library
- Xiangxin Zhu, Carl Vondrick, Deva Ramanan, and Charless C. Fowlkes. 2012. Do we need more training data or better models for object detection. In Proceedings of the British Machine Vision Conference (BMVC '12).Google Scholar
- Seeing AI. Retrieved from https://www.microsoft.com/en-us/seeing-ai/Google Scholar
- Encoded Reality. Retrieved from http://viral.media.mit.edu/projects/encoded_reality/Google Scholar
- Porsche Panamera 2017. Retrieved from http://www.motorauthority.com/news/1105162_2017- porsche-panamera-deep-dive/page-3#Google Scholar
- CAT S60: The World's First Thermal Imaging Smartphone. Retrieved from https://www.catphones.com/?product=cat-s60- smartphoneGoogle Scholar
- JETSON TK1. Retrieved from http://www.nvidia.com/object/jetson-tk1-embeddeddev-kit.htmlGoogle Scholar
- Caffe deep learning framework. Retrieved from http://caffe.berkeleyvision.org/Google Scholar
- TensorFlow Mobile. Retrieved from https://www.tensorflow.org/mobile/Google Scholar
- Dyson 360 Eye robot. Retrieved from https://www.dyson.co.uk/robot-vacuums/dyson-360- eye-overview.htmlGoogle Scholar
- LG HOM-BOT Square. Retrieved from http://www.lg.com/uk/hom-botGoogle Scholar
- Google Maps now lets users add wheelchair accessibility details for locations. Retrieved from https://techcrunch.com/2017/07/08/google-maps-nowlets-users-add-wheelchair-accessibility-details-forlocations/Google Scholar
Index Terms
- Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns
Recommendations
Smile detection in the wild with deep convolutional neural networks
Smile or happiness is one of the most universal facial expressions in our daily life. Smile detection in the wild is an important and challenging problem, which has attracted a growing attention from affective computing community. In this paper, we ...
Skin Temperature Extraction Using Facial Landmark Detection and Thermal Imaging for Comfort Assessment
BuildSys '19: Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and TransportationDespite the large share of energy consumption, current HVAC systems in buildings fail to meet their primary purpose of maintaining comfortable indoor conditions. Current "one size fits all" approach to control the thermal conditions in an environment ...
Face Recognition via Thermal Imaging: A Comparative Study of Traditional and CNN-Based Approaches
ICMLSC '24: Proceedings of the 2024 8th International Conference on Machine Learning and Soft ComputingIn this article, a face recognition via thermal imaging: a comparative study of traditional and CNN-based approaches is proposed. The methodology comprises two distinct components: traditional face recognition and CNN-based face recognition. In the ...
Comments