Ke Huo
Sang Ho Yoon
ABSTRACT
The emerging simultaneous localizing and mapping (SLAM) based tracking technique allows the mobile AR device spatial awareness of the physical world. Still, smart things are not fully supported with the spatial awareness in AR. Therefore, we present Scenariot, a method that enables instant discovery and localization of the surrounding smart things while also spatially registering them with a SLAM based mobile AR system. By exploiting the spatial relationships between mobile AR systems and smart things, Scenariot fosters in-situ interactions with connected devices. We embed Ultra-Wide Band (UWB) RF units into the AR device and the controllers of the smart things, which allows for measuring the distances between them. With a one-time initial calibration, users localize multiple IoT devices and map them within the AR scenes. Through a series of experiments and evaluations, we validate the localization accuracy as well as the performance of the enabled spatial aware interactions. Further, we demonstrate various use cases through Scenariot.
Supplemental Material
- Abdulrahman Alarifi, AbdulMalik Al-Salman, Mansour Alsaleh, Ahmad Alnafessah, Suheer Al-Hadhrami, Mai A Al-Ammar, and Hend S Al-Khalifa. 2016. Ultra wideband indoor positioning technologies: Analysis and recent advances. Sensors 16, 5 (2016), 707.Google Scholar
Cross Ref
- Isaac Amundson and Xenofon Koutsoukos. 2009. A survey on localization for mobile wireless sensor networks. Mobile entity localization and tracking in GPS-less environnments (2009), 235--254. Google ScholarDigital Library
- Ferran Argelaguet and Carlos Andujar. 2009. Visual feedback techniques for virtual pointing on stereoscopic displays. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology. ACM, 163--170. Google ScholarDigital Library
- ASUS. 2017. ZenFone-AR. (2017). Retrieved September 1, 2017 from = https://www.asus.com/us/Phone/ZenFoneAR-ZS571KL/.Google Scholar
- Shuanghua Bai and Houduo Qi. 2016. Tackling the flip ambiguity in wireless sensor network localization and beyond. Digital Signal Processing 55 (2016), 85--97. Google ScholarDigital Library
- Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic interaction: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces. ACM, 121--130. Google ScholarDigital Library
- Ingwer Borg and Patrick JF Groenen. 2005. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media.Google Scholar
- Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, and Patrick Baudisch. 2010. Touch projector: mobile interaction through video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2287--2296. Google ScholarDigital Library
- Barry Brumitt, John Krumm, Brian Meyers, and Steven Shafer. 2000. Ubiquitous computing and the role of geometry. IEEE Personal Communications 7, 5 (2000), 41--43.Google ScholarCross Ref
- Yu-Hsiang Chen, Ben Zhang, Claire Tuna, Yang Li, Edward A Lee, and Bjorn Hartmann. 2013. A context menu for the real world: Controlling physical appliances through head-worn infrared targeting. Technical Report. CALIFORNIA UNIV BERKELEY DEPT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCES.Google Scholar
- Jose A Costa, Neal Patwari, and Alfred O Hero III. 2006. Distributed weighted-multidimensional scaling for node localization in sensor networks. ACM Transactions on Sensor Networks (TOSN) 2, 1 (2006), 39--64. Google ScholarDigital Library
- Adrian A de Freitas, Michael Nebeling, Xiang'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K Dey. 2016. Snap-to-it: A user-inspired platform for opportunistic device interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5909--5920. Google ScholarDigital Library
- Jan De Leeuw and Patrick Mair. 2011. Multidimensional scaling using majorization: SMACOF in R. Department of Statistics, UCLA (2011).Google Scholar
- Carmelo Di Franco, Enrico Bini, Mauro Marinoni, and Giorgio C Buttazzo. 2017a. Multidimensional scaling localization with anchors. In Autonomous Robot Systems and Competitions (ICARSC), 2017 IEEE International Conference on. IEEE, 49--54.Google ScholarCross Ref
- Carmelo Di Franco, Amanda Prorok, Nikolay Atanasov, Benjamin P Kempke, Prabal Dutta, Vijay Kumar, and George J Pappas. 2017b. Calibration-free network localization using non-line-of-sight ultra-wideband measurements.. In IPSN. 235--246. Google ScholarDigital Library
- Ivan Dokmanic, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. 2015. Euclidean distance matrices: Essential theory, algorithms, and applications. IEEE Signal Processing Magazine 32, 6 (2015), 12--30.Google ScholarCross Ref
- Hans Gellersen, Carl Fischer, Dominique Guinard, Roswitha Gostner, Gerd Kortuem, Christian Kray, Enrico Rukzio, and Sara Streng. 2009. Supporting device discovery and spontaneous interaction with spatial references. Personal and Ubiquitous Computing 13, 4 (2009), 255--264. Google ScholarDigital Library
- Google. 2017. ARCore. (2017). Retrieved September 1, 2017 from = https://developers.google.com/ar/.Google Scholar
- Uwe Gruenefeld, Abdallah El Ali, Wilko Heuten, and Susanne Boll. 2017. Visualizing out-of-view objects in head-mounted augmented reality. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 81. Google ScholarDigital Library
- Anhong Guo, Jeeeun Kim, Xiang'Anthony' Chen, Tom Yeh, Scott E Hudson, Jennifer Mankoff, and Jeffrey P Bigham. 2017. Facade: Auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5826--5838. Google ScholarDigital Library
- Dirk Hahnel, Wolfram Burgard, Dieter Fox, Ken Fishkin, and Matthai Philipose. 2004. Mapping and localization with RFID technology. In Robotics and Automation, 2004. Proceedings. ICRA'04. 2004 IEEE International Conference on, Vol. 1. IEEE, 1015--1020.Google ScholarCross Ref
- Valentin Heun, James Hobin, and Pattie Maes. 2013. Reality editor: programming smarter objects. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 307--310. Google ScholarDigital Library
- Yi Jiang and Victor CM Leung. 2007. An asymmetric double sided two-way ranging for crystal offset. In Signals, Systems and Electronics, 2007. ISSSE'07. International Symposium on. IEEE, 525--528.Google ScholarCross Ref
- Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. Em-sense: Touch recognition of uninstrumented, electrical and electromechanical objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 157--166. Google ScholarDigital Library
- David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-aware controls: Designing remote controls for ubiquitous computing ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 187--198. Google ScholarDigital Library
- Sikun Lin, Hao Fei Cheng, Weikai Li, Zhanpeng Huang, Pan Hui, and Christoph Peylo. 2017a. Ubii: Physical World Interaction Through Augmented Reality. IEEE Transactions on Mobile Computing 16, 3 (2017), 872--885. Google ScholarDigital Library
- Yung-Ta Lin, Yi-Chi Liao, Shan-Yuan Teng, Yi-Ju Chung, Liwei Chan, and Bing-Yu Chen. 2017b. Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360 Video Using Spatial Picture-in-Picture Previews. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology. ACM, --. Google ScholarDigital Library
- Simon Mayer, Markus Schalch, Marian George, and Gábor Sörös. 2013. Device recognition for intuitive interaction with the web of things. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 239--242. Google ScholarDigital Library
- Microsoft. 2017. Hololens. (2017). Retrieved September 1, 2017 from = https://www.microsoft.com/en-us/hololens.Google Scholar
- Gang Pan, Jiahui Wu, Daqing Zhang, Zhaohui Wu, Yingchun Yang, and Shijian Li. 2010. GeeAir: a universal multimodal remote control device for home appliances. Personal and Ubiquitous Computing 14, 8 (2010), 723--735. Google ScholarDigital Library
- Charith Perera, Arkady Zaslavsky, Peter Christen, and Dimitrios Georgakopoulos. 2014. Context aware computing for the internet of things: A survey. IEEE Communications Surveys & Tutorials 16, 1 (2014), 414--454.Google ScholarCross Ref
- Sudeep Pillai and John Leonard. 2015. Monocular slam supported object recognition. arXiv preprint arXiv:1506.01732 (2015).Google Scholar
- Math.Net Project. 2017. Math.Net Numerics. (2017). Retrieved September 1, 2017 from = https://numerics.mathdotnet.com/.Google Scholar
- Kun Qian, Chenshu Wu, Zimu Zhou, Yue Zheng, Zheng Yang, and Yunhao Liu. 2017. Inferring motion direction using commodity wi-fi for interactive exergames. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1961--1972. Google ScholarDigital Library
- Jun Rekimoto, Yuji Ayatsuka, Michimune Kohno, and Haruo Oba. 2003. Proximal Interactions: A Direct Manipulation Technique for Wireless Networking.. In Interact, Vol. 3. 511--518.Google Scholar
- Enrico Rukzio, Karin Leichtenstern, Vic Callaghan, Paul Holleis, Albrecht Schmidt, and Jeannette Chin. 2006. An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. UbiComp 2006: Ubiquitous Computing (2006), 87--104. Google ScholarDigital Library
- Spencer Russell, Gershon Dublon, and Joseph A Paradiso. 2016. Hearthere: Networked sensory prosthetics through auditory augmented reality. In Proceedings of the 7th Augmented Human International Conference 2016. ACM, 20. Google ScholarDigital Library
- Renato F Salas-Moreno, Richard A Newcombe, Hauke Strasdat, Paul HJ Kelly, and Andrew J Davison. 2013. Slam++: Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1352--1359. Google ScholarDigital Library
- Dominik Schmidt, David Molyneaux, and Xiang Cao. 2012. PICOntrol: using a handheld projector for direct control of physical devices through visible light. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 379--388. Google ScholarDigital Library
- Eldon Schoop, Michelle Nguyen, Daniel Lim, Valkyrie Savage, Sean Follmer, and Björn Hartmann. 2016. Drill Sergeant: Supporting physical construction projects through an ecosystem of augmented tools. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1607--1614. Google ScholarDigital Library
- Unity3D. 2017. Unity3D. (2017). Retrieved September 1, 2017 from = https://unity3d.com/.Google Scholar
- Pasi Välkkynen and Timo Tuomisto. 2005. Physical Browsing Research. PERMID 2005 (2005), 35--38.Google Scholar
- S Venkatesh and RM Buehrer. 2007. Non-line-of-sight identification in ultra-wideband systems based on received signal statistics. IET Microwaves, Antennas & Propagation 1, 6 (2007), 1120--1130.Google ScholarCross Ref
- Edward J Wang, Tien-Jui Lee, Alex Mariakakis, Mayank Goel, Sidhant Gupta, and Shwetak N Patel. 2015. Magnifisense: Inferring device interaction using wrist-worn passive magneto-inductive sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 15--26. Google ScholarDigital Library
- Roy Want, Kenneth P Fishkin, Anuj Gujar, and Beverly L Harrison. 1999. Bridging physical and virtual worlds with electronic tags. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 370--377. Google ScholarDigital Library
- wikitude. 2017. wikitude. (2017). Retrieved September 1, 2017 from = https://www.wikitude.com/.Google Scholar
- Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4000--4008. Google ScholarDigital Library
- Sang Ho Yoon, Yunbo Zhang, Ke Huo, and Karthik Ramani. 2016. TRing: Instant and Customizable Interactions with Objects Using an Embedded Magnet and a Finger-Worn Device. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 169--181. Google ScholarDigital Library
Index Terms
- Scenariot: Spatially Mapping Smart Things Within Augmented Reality Scenes
Recommendations
A magic lens for revealing device interactions in smart environments
SA '14: SIGGRAPH Asia 2014 Mobile Graphics and Interactive ApplicationsKeeping track of device interactions in smart environments is a challenging task for everyday users. Given the expected high number of communicating devices in future smart homes, it will become increasingly important to put users more in control of ...
Ubiquitous home simulation using augmented reality
CEA'07: Proceedings of the 2007 annual Conference on International Conference on Computer Engineering and ApplicationsComputing paradigm is moving toward ubiquitous computing in which devices, software agents, and services are all expected to seamlessly integrate and cooperate in support of human objectives. Augmented reality (AR) can naturally complement ubiquitous ...
A smart environment for augmented learning through physical books
ICME '11: Proceedings of the 2011 IEEE International Conference on Multimedia and ExpoToday, augmented reality is evolving towards sophisticated approaches exploiting the opportunities offered by immersive environments and high quality 3D graphics. Such systems can prove to be very useful in the context of education, especially when ...
Comments