Wolfgang Büschel
Annett Mitschick
Raimund Dachselt
ABSTRACT
Today, the widespread use of mobile devices allows users to search information "on the go»», whenever and wherever they want, no longer confining Information Retrieval to classic desktop interfaces. We believe that technical advances in Augmented Reality will allow Information Retrieval to go even further, making use of both the users» surroundings and their abilities to interact with the physical world. In this paper, we present the fundamental concept of Reality-Based Information Retrieval, which combines the classic Information Retrieval process with Augmented Reality technologies to provide context-dependent search cues and situated visualizations of the query and the results. With information needs often stemming from real-world experiences, this novel combination has the potential to better support both Just-in-time Information Retrieval and serendipity. Based on extensive literature research, we propose a conceptual framework for Reality-Based Information Retrieval. We illustrate and discuss this framework and present two prototypical implementations, which we tested in small user studies. They demonstrate the feasibility of our concepts and inspired our discussion of notable challenges for further research in this novel and promising area.
- Antti Ajanki, Mark Billinghurst, Toni Järvenpää, Melih Kandemir, Samuel Kaski, Markus Koskela, Mikko Kurimo, Jorma Laaksonen, Kai Puolamäki, Teemu Ruokolainen, et al. 2010. Contextual information access with augmented reality. In Proc. 2010 IEEE Int. Workshop MLSP. IEEE, 95--100.Google Scholar
Cross Ref
- James Allan, Bruce Croft, Alistair Moffat, and Mark Sanderson. 2012. Frontiers, Challenges, and Opportunities for Information Retrieval: Report from SWIRL 2012 the Second Strategic Workshop on Information Retrieval in Lorne. SIGIR Forum 46, 1 (May 2012), 2--32. Google ScholarDigital Library
- Jürgen Assfalg, Alberto Del Bimbo, and Pietro Pala. 2002. Three-Dimensional Interfaces for Querying by Example in Content-Based Image Retrieval. IEEE Trans. Vis. Comput. Graphics 8, 4 (Oct. 2002), 305--318. Google ScholarDigital Library
- Juhee Bae, Vidya Setlur, and Benjamin Watson. 2015. GraphTiles: A Visual Interface Supporting Browsing and Imprecise Mobile Search. In Proc. MobileHCI '15. ACM, New York, NY, USA, 63--70. Google ScholarDigital Library
- Robert Ball, Chris North, and Doug A. Bowman. 2007. Move to Improve: Promoting Physical Navigation to Increase User Performance with Large Displays. In Proc. CHI '07. ACM, New York, NY, USA, 191--200. Google ScholarDigital Library
- Daniel Belcher, Mark Billinghurst, SE Hayes, and Randy Stiles. 2003. Using Augmented Reality for Visualizing Complex Graphs in Three Dimensions. In Proc. ISMAR '03. IEEE CS, Washington, DC, USA, 84--93. Google ScholarDigital Library
- Mark Billinghurst, Adrian Clark, and Gun Lee. 2015. A Survey of Augmented Reality. Found. Trends Hum.-Comput. Interact. 8, 2--3 (March 2015), 73--272. Google ScholarDigital Library
- Ourdia Bouidghaghen, Lynda Tamine-Lechani, and Mohand Boughanem. 2009. Dynamically Personalizing Search Results for Mobile Users. In Proc. FQAS '09. Springer-Verlag, Berlin, Heidelberg, 99--110. Google ScholarDigital Library
- John Brooke. 1996. SUS: a "quick and dirty" usability scale. In Usability Evaluation in Industry. Taylor and Francis.Google Scholar
- Michael K. Buckland and Christian Plaunt. 2000. On the Construction of Selection Systems. Library Hi Tech 12 (06 2000).Google Scholar
- Wolfgang Büschel, Patrick Reipschläger, Ricardo Langner, and Raimund Dachselt. 2017. Investigating the Use of Spatial Interaction for 3D Data Visualization on Mobile Devices. In Proc. ISS '17. ACM, New York, NY, USA, 62--71. Google ScholarDigital Library
- Juan Pablo Carrascal and Karen Church. 2015. An In-Situ Study of Mobile App&Mobile Search Interactions. In Proc. CHI '15. ACM, New York, NY, USA, 2739--2748. Google ScholarDigital Library
- Patrick Chiu, Andreas Girgensohn, Surapong Lertsithichai, Wolf Polak, and Frank Shipman. 2005. MediaMetro: Browsing Multimedia Document Collections with a 3D City Metaphor. In Proc. MULTIMEDIA '05. ACM, New York, NY, USA, 213--214. Google ScholarDigital Library
- Karen Church, Mauro Cherubini, and Nuria Oliver. 2014. A Large-scale Study of Daily Information Needs Captured in Situ. ACM Trans. Comput.-Hum. Interact. 21, 2, Article 10 (Feb. 2014), 46 pages. Google ScholarDigital Library
- Karen Church and Barry Smyth. 2009. Understanding the Intent Behind Mobile Information Needs. In Proc. IUI '09. ACM, New York, NY, USA, 247--256. Google ScholarDigital Library
- John Eakins, Margaret Graham, and Tom Franklin. 1999. Content-based Image Retrieval. In Library and Information Briefings.Google Scholar
- Neven ElSayed, Bruce Thomas, Kim Marriott, Julia Piantadosi, and Ross Smith. 2015. Situated Analytics. In 2015 Big Data Visual Analytics (BDVA). 1--8. https: //doi.org/10.1109/BDVA.2015.7314302{18} Barrett Ens, Ahmad Byagowi, Teng Han, Juan David Hincapié-Ramos, and Pourang Irani. 2016. Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces. In Proc. SUI '16. ACM, New York, NY, USA, 99--102. Google ScholarDigital Library
- Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani. 2014. Ethereal Planes: A Design Framework for 2D Information Space in 3D Mixed Reality Environments. In Proc. SUI '14. ACM, New York, NY, USA, 2--12. Google ScholarDigital Library
- Barrett Ens and Pourang Irani. 2017. Spatial Analytic Interfaces: Spatial User Interfaces for In Situ Visual Analytics. IEEE Comp. Graph. Appl. 37, 2 (2017), 66--79. Google ScholarDigital Library
- Xin Fan, Xing Xie, Zhiwei Li, Mingjing Li, and Wei-Ying Ma. 2005. Photo-tosearch: Using Multimodal Queries to Search the Web from Mobile Devices. In Proc. MIR '05. ACM, New York, NY, USA, 143--150. Google ScholarDigital Library
- Bernd Girod, Vijay Chandrasekhar, David M Chen, Ngai-Man Cheung, Radek Grzeszczuk, Yuriy Reznik, Gabriel Takacs, Sam S Tsai, and Ramakrishna Vedantham. 2011. Mobile visual search. IEEE Signal Process. Mag. 28, 4 (2011), 61--76.Google ScholarCross Ref
- Raphael Grasset, Tobias Langlotz, Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. 2012. Image-driven View Management for Augmented Reality Browsers. In Proc. ISMAR '12. IEEE Computer Society, Washington, DC, USA, 177--186. Google ScholarDigital Library
- Yulan Guo, Mohammed Bennamoun, Ferdous Sohel, Min Lu, and Jianwei Wan. 2014. 3D Object Recognition in Cluttered Scenes with Local Surface Features: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 36, 11 (Nov 2014), 2270--2287.Google ScholarCross Ref
- Djoerd Hiemstra. 2009. Information Retrieval Models. In Information Retrieval. John Wiley&Sons, Ltd, 1--19.Google Scholar
- Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proc. CVPR '16. 4555-- 4564.Google ScholarCross Ref
- Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. 1985. Direct Manipulation Interfaces. Hum.-Comput. Interact. 1, 4 (Dec. 1985), 311--338. Google ScholarDigital Library
- Peter Ingwersen and Kalervo Järvelin. 2005. The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series). Springer-Verlag New York, Inc., Secaucus, NJ, USA. Google ScholarDigital Library
- Daisuke Iwai, Tatsunori Yabiki, and Kosuke Sato. 2013. View management of projected labels on nonplanar and textured surfaces. IEEE Trans. Vis. Comput. Graphics 19, 8 (2013), 1415--1424. Google ScholarDigital Library
- Masakazu Iwamura, Kai Kunze, Yuya Kato, Yuzuko Utsumi, and Koichi Kise. 2014. Haven'T We Met Before?: A Realistic Memory Assistance System to Remind You of the Person in Front of You. In Proc. AH '14. ACM, New York, NY, USA, Article 32, 4 pages. Google ScholarDigital Library
- Matt Jones, George Buchanan, Richard Harper, and Pierre-Louis Xech. 2007. Questions Not Answers: A Novel Mobile Search Technique. In Proc. CHI '07. ACM, New York, NY, USA, 155--158. Google ScholarDigital Library
- Amy K. Karlson, George G. Robertson, Daniel C. Robbins, Mary P. Czerwinski, and Greg R. Smith. 2006. FaThumb: A Facet-based Interface for Mobile Search. In Proc. CHI '06. ACM, New York, NY, USA, 711--720. Google ScholarDigital Library
- Jens Keil, Michael Zoellner, Timo Engelke, Folker Wientapper, and Michael Schmitt. 2013. Controlling and Filtering Information Density with Spatial Interaction Techniques via Handheld Augmented Reality. Springer, Berlin, Heidelberg, 49--57.Google Scholar
- Konstantin Klamka and Raimund Dachselt. 2015. Elasticcon: Elastic Controllers for Casual Interaction. In Proc. MobileHCI '15. ACM, NY, USA, 410--419. Google ScholarDigital Library
- Jarrod Knibbe, Sue Ann Seah, and Mike Fraser. 2014. VideoHandles: Replicating Gestures to Search Through Action-camera Video. In Proc. SUI '14. ACM, New York, NY, USA, 50--53. Google ScholarDigital Library
- ByoungChul Ko and Hyeran Byun. 2002. Query-by-Gesture: An Alternative Content-Based Image Retrieval Query Scheme. J. Vis. Lang. Comput. 13 (2002), 375--390.Google ScholarCross Ref
- Jihoon Ko, Sangjin Shin, Sungkwang Eom, Minjae Song, Dong-Hoon Shin, KyongHo Lee, and Yongil Jang. 2014. Semantically Enhanced Keyword Search for Smartphones. In Proc. WWW '14 Companion. ACM, New York, NY, USA, 327--328. Google ScholarDigital Library
- Ricardo Langner, Ulrich von Zadow, Tom Horak, Annett Mitschick, and Raimund Dachselt. 2016. Content Sharing Between Spatially-Aware Mobile Phones and Large Vertical Displays Supporting Collaborative Work. Springer International Publishing, 75--96.Google Scholar
- Gang Li, Yue Liu, and Yongtian Wang. 2017. Evaluation of Labelling Layout Methods in Augmented Reality. In Proc. VR '17. 351--352. 00000Google ScholarCross Ref
- Jacob Boesen Madsen, Markus Tatzqern, Claus B Madsen, Dieter Schmalstieg, and Denis Kalkofen. 2016. Temporal Coherence Strategies for Augmented Reality Labeling. IEEE Trans. Vis. Comput. Graphics 22, 4 (April 2016), 1415--1423. Google ScholarDigital Library
- Stephann Makri, Ann Blandford, Mel Woods, Sarah Sharples, and Deborah Maxwell. 2014. "Making my own luck": Serendipity strategies and how to support them in digital information environments. J. Assn. Inf. Sci. Tec. 65, 11 (2014), 2179--2194. Google ScholarDigital Library
- Tamás Matuszka. 2015. The Design and Implementation of Semantic WebBased Architecture for Augmented Reality Browser. In The Semantic Web. Latest Advances and New Domains. Springer, 731--739. Google ScholarDigital Library
- Lori McCay-Peet and Elaine G. Toms. 2015. Investigating serendipity: How it unfolds and what may influence it. J. Assn. Inf. Sci. Tec. 66, 7 (2015), 1463--1476. Google ScholarDigital Library
- Lori McCay-Peet, Elaine G. Toms, and E. Kevin Kelloway. 2015. Examination of relationships among serendipity, the environment, and individual differences. Information Processing&Management 51, 4 (2015), 391 -- 412. Google ScholarDigital Library
- Taniya Mishra and Srinivas Bangalore. 2010. Qme!: A Speech-based Questionanswering System on Mobile Devices. In Proc. HLT '10. Association for Computational Linguistics, Stroudsburg, PA, USA, 55--63. http://dl.acm.org/citation.cfm' id=1857999.1858006 Google ScholarDigital Library
- Munehiro Nakazato and Thomas S Huang. 2001. 3D MARS: immersive virtual reality for content-based image retrieval. In Proc. ICME '01. 44--47.Google ScholarCross Ref
- Jason Orlosky, Kiyoshi Kiyokawa, Takumi Toyama, and Daniel Sonntag. 2015. Halo Content: Context-aware Viewspace Management for Non-invasive Augmented Reality. In Proc. IUI '15. ACM, New York, NY, USA, 369--373. Google ScholarDigital Library
- Bradley J Rhodes and Pattie Maes. 2000. Just-in-time Information Retrieval Agents. IBM Syst. J. 39, 3--4 (July 2000), 685--704. Google ScholarDigital Library
- Cornelis J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.). ButterworthHeinemann, Newton, MA, USA. Google ScholarDigital Library
- Dieter Schmalstieg and Tobias Hollerer. 2016. Augmented reality: principles and practice. Addison-Wesley Professional.Google Scholar
- Kimiaki Shirahama and Kuniaki Uehara. 2011. Query by Virtual Example: Video Retrieval Using Example Shots Created by Virtual Reality Techniques. In Int. Conf. on Image and Graphics '11. 829--834. Google ScholarDigital Library
- Sophie Stellmach, Sebastian Stober, Andreas Nürnberger, and Raimund Dachselt. 2011. Designing Gaze-supported Multimodal Interactions for the Exploration of Large Image Collections. In Proc. NGCA '11. ACM, New York, NY, USA, Article 1, 8 pages. Google ScholarDigital Library
- Zsolt Szalavári and Michael Gervautz. 1997. The Personal Interaction Panel -- a Two-Handed Interface for Augmented Reality. Computer Graphics Forum 16, 3 (1997), C335--C346.Google ScholarCross Ref
- Norimichi Ukita, Tomohisa Ono, and Masatsugu Kidode. 2005. Region Extraction of a Gaze Object Using the Gaze Point and View Image Sequences. In Proc. ICMI '05. ACM, NY, USA, 129--136. 00005 Google ScholarDigital Library
- Nayuko Watanabe, Masayuki Okamoto, Masaaki Kikuchi, Takayuki Iida, Kenta Sasaki, Kensuke Horiuchi, and Masanori Hattori. 2011. Designing Mobile Search Interface with Query Term Extraction. Springer, Berlin, Heidelberg, 67--76.Google Scholar
- Sean White and Steven Feiner. 2009. SiteLens: Situated Visualization Techniques for Urban Site Visits. In Proc. CHI '09. ACM, New York, NY, USA, 1117--1120. Google ScholarDigital Library
- Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2017. Embedded Data Representations. IEEE Trans. Vis. Comput. Graphics 23, 1 (Jan 2017), 461--470. Google ScholarDigital Library
- Xing Xie, Lie Lu, Menglei Jia, Hua Li, Frank Seide, and Wei-Ying Ma. 2008. Mobile Search With Multimodal Queries. Proc. of the IEEE 96, 4 (April 2008), 589--601.Google Scholar
- Tom Yeh, John J. Lee, and Trevor Darrell. 2008. Photo-based Question Answering. In Proc. MM '08. ACM, New York, NY, USA, 389--398. Google ScholarDigital Library
- Degi Young and Ben Shneiderman. 1993. A graphical filter/flow representation of Boolean queries: A prototype implementation and evaluation. J. Am. Soc. Inf. Sci. 44, 6 (1993), 327--339. Google ScholarDigital Library
- Stefan Zander, Chris Chiu, and Gerhard Sageder. 2012. A Computational Model for the Integration of Linked Data in Mobile Augmented Reality Applications. In Proc. I-SEMANTICS '12. ACM, New York, USA, 133--140. Google ScholarDigital Library
- Qi Zhang, Simon Zaaijer, Song Wu, and Michael S. Lew. 2014. 3D Image Browsing: The Planets. In Proc. ICMR '14. ACM, New York, USA, Article 511, 3 pages. Google ScholarDigital Library
- Yun Zhang, Hong Fu, Zhen Liang, Zheru Chi, and Dagan Feng. 2010. Eye Movement As an Interaction Mechanism for Relevance Feedback in a Content-based Image Retrieval System. In Proc. ETRA '10. ACM, New York, NY, USA, 37--40. Google ScholarDigital Library
Index Terms
- Here and Now: Reality-Based Information Retrieval: Perspective Paper
Recommendations
Demonstrating Reality-Based Information Retrieval
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsWith this work, we demonstrate our concept of Reality-Based Information Retrieval. Our principal idea is to bring Information Retrieval closer to the real world, for a new class of future, immersive IR interfaces. Technological advances in computer ...
Gaze-directed Immersive Visualization of Scientific Ensembles
ISS '16 Companion: Proceedings of the 2016 ACM Companion on Interactive Surfaces and SpacesThe latest advances in head-mounted displays (HMDs) for augmented reality (AR) and mixed reality (MR) have produced commercialized devices that are gradually accepted by the public. These HMDs are generally equipped with head tracking, which provides an ...
A geoscience perspective on immersive 3D gridded data visualization
We describe visualization software, Visualizer, that was developed specifically for interactive, visual exploration in immersive virtual reality (VR) environments. Visualizer uses carefully optimized algorithms and data structures to support the high ...
Comments