Chris Xiaoxuan Lu
ABSTRACT
This paper presents the design, implementation and evaluation of milliMap, a single-chip millimetre wave (mmWave) radar based indoor mapping system targetted towards low-visibility environments to assist in emergency response. A unique feature of milliMap is that it only leverages a low-cost, off-the-shelf mmWave radar, but can reconstruct a dense grid map with accuracy comparable to lidar, as well as providing semantic annotations of objects on the map. milliMap makes two key technical contributions. First, it autonomously overcomes the sparsity and multi-path noise of mmWave signals by combining cross-modal supervision from a co-located lidar during training and the strong geometric priors of indoor spaces. Second, it takes the spectral response of mmWave reflections as features to robustly identify different types of objects e.g. doors, walls etc. Extensive experiments in different indoor environments show that milliMap can achieve a map reconstruction error less than 0.2m and classify key semantics with an accuracy of ~ 90%, whilst operating through dense smoke.
- Markus Achtelik, Michael Achtelik, Yorick Brunet, Margarita Chli, Savvas Chatzichristofis, Jean-Dominique Decotignie, Klaus-Michael Doth, Friedrich Fraundorfer, Laurent Kneip, Daniel Gurdan, et al. 2012. Sfly: Swarm of micro flying robots. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2649--2650.Google Scholar
Cross Ref
- Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Transactions on Graphics 34, 6 (2015), 219.Google ScholarDigital Library
- Fadel Adib, Zachary Kabelac, and Dina Katabi. 2015. Multi-Person Localization via RF Body Reflections. In NSDI.Google Scholar
- Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In NSDI.Google Scholar
- Fadel Adib and Dina Katabi. 2013. See through walls with WiFi! ACM SIGCOMM.Google Scholar
- Federal Emergency Management Agency. [n. d.]. Fire in the United States (1989 - 1998). file:///home/chris/Desktop/10409.pdfGoogle Scholar
- Oscar Alsing. 2018. Mobile Object Detection using TensorFlow Lite and Transfer Learning.Google Scholar
- Kok Seng Chong and Lindsay Kleeman. 1999. Feature-based mapping in real, large scale environments using an ultrasonic array. The International Journal of Robotics Research 18, 1 (1999), 3--19.Google Scholar
- Jeffrey Delmerico, Stefano Mintchev, Alessandro Giusti, Boris Gromov, Kamilo Melo, Tomislav Horvat, Cesar Cadena, Marco Hutter, Auke Ijspeert, Dario Floreano, et al. 2019. The current state and future outlook of rescue robotics. Journal of Field Robotics 36, 7 (2019), 1171--1191.Google ScholarCross Ref
- Saandeep Depatla, Lucas Buckland, and Yasamin Mostofi. 2015. X-ray vision with only wifi power measurements using rytov wave models. IEEE Transactions on Vehicular Technology 64, 4 (2015), 1376--1387.Google ScholarCross Ref
- Ashutosh Dhekne, Ayon Chakraborty, Karthikeyan Sundaresan, and Sampath Rangarajan. 2019. TrackIO: tracking first responders inside-out. In 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19).Google Scholar
- Ashutosh Dhekne, Mahanth Gowda, Yixuan Zhao, Haitham Hassanieh, and Romit Roy Choudhury. 2018. Liquid: A wireless liquid identifier. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 442--454.Google ScholarDigital Library
- Jiang Dong, Yu Xiao, Marius Noreikis, Zhonghong Ou, and Antti Ylä-Jääski. 2015. imoon: Using smartphones for image-based indoor navigation. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. ACM, 85--97.Google ScholarDigital Library
- Alexey Dosovitskiy and Thomas Brox. 2016. Generating images with perceptual similarity metrics based on deep networks. In NIPS.Google Scholar
- Chao Feng, Jie Xiong, Liqiong Chang, Ju Wang, Xiaojiang Chen, Dingyi Fang, and Zhanyong Tang. 2019. WiMi: Target Material Identification with Commodity Wi-Fi Devices. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).Google Scholar
- Ruipeng Gao, Mingmin Zhao, Tao Ye, Fan Ye, Yizhou Wang, Kaigui Bian, Tao Wang, and Xiaoming Li. 2014. Jigsaw: Indoor floor plan reconstruction via mobile crowdsensing. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 249--260.Google ScholarDigital Library
- Andrea Garulli, Antonio Giannitrapani, Andrea Rossi, and Antonio Vicino. 2005. Mobile robot SLAM for line-based environment representation. In CDC.Google Scholar
- Aaron Gokaslan, Vivek Ramanujan, Daniel Ritchie, Kwang In Kim, and James Tompkin. 2018. Improving shape deformation in unsupervised image-to-image translation. In ECCV.Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS.Google Scholar
- Angelos A Goulianos, Alberto L Freire, Tom Barratt, Evangelos Mellios, Peter Cain, Moray Rumney, Andrew Nix, and Mark Beach. 2017. Measurements and characterisation of surface scattering at 60 GHz. In IEEE 86th Vehicular Technology Conference (VTC-Fall).Google ScholarCross Ref
- Junfeng Guan, Sohrab Madani, Suraj Jog, and Haitham Hassanieh. 2020. High Resolution Millimeter Wave Imaging For Self-Driving Cars. IEEE CVPR (2020).Google Scholar
- Simon Haykin, John Litva, and Terence J Shepherd. 1993. Radar array processing. Springer.Google Scholar
- Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. 2014. RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In Experimental robotics. Springer, 477--491.Google Scholar
- Christopher L Holloway, Patrick L Perini, Ronald R DeLyser, and Kenneth C Allen. 1997. Analysis of composite walls and their effects on short-path propagation modeling. IEEE Transactions on Vehicular Technology 46, 3 (1997), 730--738.Google ScholarCross Ref
- Armin Hornung, Kai M. Wurm, Maren Bennewitz, Cyrill Stachniss, and Wolfram Burgard. 2013. OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees. Autonomous Robots (2013). Software available at http://octomap.github.com. Google ScholarDigital Library
- Donny Huang, Rajalakshmi Nandakumar, and Shyamnath Gollakota. 2014. Feasibility and limits of wi-fi imaging. In SenSys.Google Scholar
- Texas Instruments. [n. d.]. Automotive mmWave sensors. http://www.ti.com/sensors/mmwave/overview.htmlGoogle Scholar
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In CVPR.Google Scholar
- Yifei Jiang, Yun Xiang, Xin Pan, Kun Li, Qin Lv, Robert P Dick, Li Shang, and Michael Hannigan. 2013. Hallway based automatic indoor floorplan construction using room fingerprints. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 315--324.Google ScholarDigital Library
- Antonio R Jimenez, Fernando Seco, Carlos Prieto, and Jorge Guevara. 2009. A comparison of pedestrian dead-reckoning algorithms using a low-cost MEMS IMU. In WISP.Google Scholar
- Suraj Jog, Jiaming Wang, Junfeng Guan, Thomas Moon, Haitham Hassanieh, and Romit Roy Choudhury. 2019. Many-to-many beam alignment in millimeter wave networks. In 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19).Google Scholar
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV.Google Scholar
- Y Kuga and P Phu. 1996. Experimental studies of millimeter-wave scattering in discrete random media and from rough surfaces. Progress In Electromagnetics Research 14 (1996), 37--88.Google Scholar
- KUKA. [n. d.]. Mobile robots from KUKA. https://www.kuka.com/en-de/products/mobility/mobile-robotsGoogle Scholar
- Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4681--4690.Google ScholarCross Ref
- Zhengxiong Li, Fenglong Ma, Aditya Singh Rathore, Zhuolin Yang, Baicheng Chen, Lu Su, and Wenyao Xu. 2020. Wavespy: Remote and through-wall screen attack via mmwave sensing. In 2020 IEEE Symposium on Security and Privacy (SP).Google ScholarCross Ref
- Zhengxiong Li, Aditya Singh Rathore, Chen Song, Sheng Wei, Yanzhi Wang, and Wenyao Xu. 2018. PrinTracker: Fingerprinting 3D printers using commodity scanners. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security.Google ScholarDigital Library
- Zhengxiong Li, Zhuolin Yang, Chen Song, Changzhi Li, Zhengyu Peng, and Wenyao Xu. 2018. E-Eye: Hidden electronics recognition through mmwave nonlinear effects. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems.Google ScholarDigital Library
- Hongbo Liu, Yu Gan, Jie Yang, Simon Sidhom, Yan Wang, Yingying Chen, and Fan Ye.2012. Push the limit of WiFi based localization for smartphones. In Proceedings of the 18th annual international conference on Mobile computing and networking. ACM, 305--316.Google Scholar
- Babak Mamandipoor, Greg Malysa, Amin Arbabian, Upamanyu Madhow, and Karam Noujeim. 2014. 60 ghz synthetic aperture radar for short-range imaging: Theory and experiments. In ACSSC.Google Scholar
- Wenguang Mao, Mei Wang, and Lili Qiu. 2018. Aim: acoustic imaging on a mobile. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 468--481.Google ScholarDigital Library
- Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. In arXiv preprint arXiv:1411.1784.Google Scholar
- JW Odendaal, E Barnard, and CWI Pistorius. 1994. Two-dimensional superresolution radar imaging using the MUSIC algorithm. IEEE Transactions on Antennas and Propagation 42, 10 (1994), 1386--1391.Google ScholarCross Ref
- Joan Palacios, Paolo Casari, and Joerg Widmer. 2017. JADE: Zero-knowledge device localization and environment mapping for millimeter wave systems. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.Google ScholarCross Ref
- Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M Álvarez. 2016. Invertible Conditional GANs for image editing. In NIPS Workshop on Adversarial Training.Google Scholar
- Samuel TPfister, Stergios I Roumeliotis, and Joel W Burdick. 2003. Weighted line fitting algorithms for mobile robot map building and efficient data representation. In ICRA.Google Scholar
- Swadhin Pradhan, Ghufran Baig, Wenguang Mao, Lili Qiu, Guohai Chen, and Bo Yang. 2018. Smartphone-based Acoustic Indoor Space Mapping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 75.Google ScholarDigital Library
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In MobiCom.Google Scholar
- Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. 5.Google Scholar
- Peng Rong and Mihail L Sichitiu. 2006. Angle of arrival localization for wireless sensor networks. In SECON.Google Scholar
- Olaf Ronneberger, Philipp Fischer, and et al. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI.Google Scholar
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In MobiCom.Google ScholarDigital Library
- Hartmut Surmann, Andreas Nüchter, and Joachim Hertzberg. 2003. An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalization of indoor environments. Robotics and Autonomous Systems 45, 3-4 (2003), 181--198.Google ScholarCross Ref
- H Tashakkori, A Rajabifard, and M Kalantari. 2016. Facilitating the 3D Indoor Search and Rescue Problem: An Overview of the Problem and an Ant Colony Solution Approach. ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences 4 (2016).Google Scholar
- Seyedeh Hosna Tashakkori Hashemi. 2017. Indoor search and rescue using a 3D indoor emergency spatial model. Ph.D. Dissertation.Google Scholar
- Lucas Theis, Aäron van den Oord, and Matthias Bethge. 2015. A note on the evaluation of generative models. In ICLR.Google Scholar
- Sebastian Thrun. 2002. Probabilistic Robotics. Commun. ACM 45, 3 (March 2002), 52âĂŞ57. Google ScholarDigital Library
- Sebastian Thrun, Wolfram Burgard, and Dieter Fox. 2005. Probabilistic robotics. MIT press.Google ScholarDigital Library
- Tiberius Tomoiagă, Cristian Predoi, and Liviu Coşereanu. 2016. Indoor mapping using low cost LIDAR based systems. In Applied Mechanics and Materials, Vol. 841. Trans Tech Publ, 198--205.Google Scholar
- Deepak Uttam and B Culshaw. 1985. Precision time domain reflectometry in optical fiber systems using a frequency modulated continuous wave ranging technique. Journal of Lightwave Technology (1985).Google Scholar
- Ju Wang, Jie Xiong, Xiaojiang Chen, Hongbo Jiang, Rajesh Krishna Balan, and Dingyi Fang. 2017. TagScan: Simultaneous target imaging and material identification with commodity RFID devices. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. ACM, 288--300.Google ScholarDigital Library
- Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR.Google Scholar
- DK Barton HR Ward. 1969. Handbook of radar measurement.Google Scholar
- Teng Wei, Anfu Zhou, and Xinyu Zhang. 2017. Facilitating robust 60 ghz network deployment by sensing ambient reflectors. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17). 213--226.Google Scholar
- Rob Weston, Sarah Cen, Paul Newman, and Ingmar Posner. 2018. Probably unknown: Deep inverse sensor modelling in radar. In ICRA.Google Scholar
- Chenhan Xu, Zhengxiong Li, Hanbin Zhang, Aditya Singh Rathore, Huining Li, Chen Song, Kun Wang, and Wenyao Xu. 2019. WaveEar: Exploring a mmWave-based Noise-resistant Speech Sensing for Voice-User Interface. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. ACM.Google ScholarDigital Library
- Qiaojing Yan and Wei Wang. 2017. DCGANs for image super-resolution, denoising and debluring. Advances in Neural Information Processing Systems (2017), 487--495.Google Scholar
- Yan Yan, Long Li, Guodong Xie, Changjing Bao, Peicheng Liao, Hao Huang, Yongxiong Ren, Nisar Ahmed, Zhe Wang, et al. 2016. Multipath effects in millimetre-wave wireless communication using orbital angular momentum multiplexing. Scientific reports 6 (2016), 33482.Google Scholar
- Bo Yang, Stefano Rosa, Andrew Markham, Niki Trigoni, and Hongkai Wen. 2018. Dense 3D object reconstruction from a single depth view. IEEE transactions on pattern analysis and machine intelligence (2018).Google Scholar
- Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. Radarcat: Radar categorization for input & interaction. In UIST. 833--841.Google ScholarDigital Library
- Ji Zhang and Sanjiv Singh. 2014. LOAM: Lidar Odometry and Mapping in Real-time.. In Robotics: Science and Systems, Vol. 2.Google ScholarCross Ref
- Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging 3, 1 (2016), 47--57.Google ScholarCross Ref
- Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In CVPR.Google Scholar
- Mingmin Zhao, Yingcheng Liu, Aniruddh Raghu, Tianhong Li, Hang Zhao, Antonio Torralba, and Dina Katabi. 2019. Through-wall human mesh recovery using radio signals. In Proceedings of the IEEE International Conference on Computer Vision. 10113--10122.Google ScholarCross Ref
- Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, and et al. 2018. RF-based 3D skeletons. In SIGCOMM.Google Scholar
- Anfu Zhou, Shaoyuan Yang, Yi Yang, Yuhang Fan, and Huadong Ma. 2019. Autonomous Environment Mapping Using Commodity Millimeter-wave Network Device. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 1126--1134.Google Scholar
- Bing Zhou, Mohammed Elbadry, Ruipeng Gao, and Fan Ye. 2017. BatMapper: Acoustic sensing based indoor floor plan construction using smartphones. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 42--55.Google ScholarDigital Library
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV.Google Scholar
- Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017. Toward multimodal image-to-image translation. In NIPS.Google Scholar
- Yanzi Zhu, Yuanshun Yao, Ben Y Zhao, and Haitao Zheng. 2017. Object recognition and navigation using a single networking device. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 265--277.Google ScholarDigital Library
- Yibo Zhu, Yanzi Zhu, Zengbin Zhang, Ben Y Zhao, and Haitao Zheng. 2015. 60GHz mobile imaging radar. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 75--80.Google ScholarDigital Library
- Yanzi Zhu, Yibo Zhu, Ben Y Zhao, and Haitao Zheng. 2015. Reusing 60ghz radios for mobile radar imaging. In MobiCom.Google Scholar
Index Terms
- See through smoke: robust indoor mapping with low-cost mmWave radar
Recommendations
Constructing Floor Plan through Smoke Using Ultra Wideband Radar
Floor plan construction has been one of the key techniques in many important applications such as indoor navigation, location-based services, and emergency rescue. Existing floor plan construction methods require expensive dedicated hardware (e.g., Lidar ...
Research on 5G Signal Interference Test for 24GHz Vehicle Millimeter Wave Radar
CNIOT '20: Proceedings of the 2020 International Conference on Computing, Networks and Internet of ThingsNowadays, millimeter-wave radars have gradually moved into people's sight due to their high frequency band, large signal bandwidth, and ability to distinguish targets. 24GHz vehicle millimeter-wave radar is researched as the target in this paper. The ...
mmBody Benchmark: 3D Body Reconstruction Dataset and Analysis for Millimeter Wave Radar
MM '22: Proceedings of the 30th ACM International Conference on MultimediaMillimeter Ware (mmWave) Radar is gaining popularity as it can work in adverse environments like smoke, rain, snow, poor lighting, etc. Prior work has explored the possibility of reconstructing 3D skeletons or meshes from the noisy and sparse mmWare ...
Comments