Accurate overlaying for mobile augmented reality

Abstract

Mobile augmented reality requires accurate alignment of virtual information with objects visible in the real world. We describe a system for mobile communications to be developed to meet these strict alignment criteria using a combination of computer vision, inertial tracking and low-latency rendering techniques. A prototype low-power and low-latency renderer using an off-the-shelf 3D card is discussed.

Introduction

Mobile augmented reality [1], [2] is a relatively new and intriguing concept. The ability of augmented reality [3] to present information superimposed on our view of the world opens up many interesting opportunities for graphical interaction with our direct environment. Combining this with mobility further increases the potential usage of this technology.

However, the technical problems with mobile augmented reality are just as great. First, it is necessary to acquire very accurate viewpoint measurements, because errors cause virtual objects to be merged at a wrong position in the real world, and a mismatch between virtual and real objects will be directly visible. Second, even more than other head-mounted display systems, augmented reality displays require an extremely low latency to keep the virtual objects at a stable position. Third, mobile augmented reality requires a low-power approach, limiting the amount of hardware that can be used to solve these problems.

An error in the measurement of the viewpoint of the user causes a deviation of the location of the displayed virtual objects from their expected position. The effect of a measurement error depends on its direction and the distance to the virtual and real objects. For example, positional errors will cause big changes in the virtual image of nearby objects, while it hardly affects objects that are far away.

Very little research has been done on the requirements for augmented reality. An accuracy corresponding to the visual acuity of the human, which is half an arcminute and sub-millimeter, seems to be the ultimate goal. But such an accuracy is far more than required for most applications. In general, the required accuracy depends on the consequences of an error, and the consequences of errors will depend on the task at hand. For example for medical purposes a misalignment of a millimeter may already be fatal [4], while for tourist information an error of a few meters and a few degrees will be acceptable.

Although literature [5], [6], [7], [9], [10] gives no conclusive numbers we think that many tasks can be supported with an AR system having a positional accuracy of a few millimeters within about 2 m from the observer, and a rotational accuracy of a fraction of a degree. For comparison, the pixels of a 640×480 display projected in a 20°×35° field of view at a distance of 1 m have a size of about 0.8 mm. Here we think of applications such as remote maintenance and repair of complex machines, assembly of complex systems, architectural support, tourist information systems, military applications and entertainment (see [5], [6]). Only a few, especially medical AR applications, really require submillimeter accuracy [4].

An important source of alignment errors is the time difference between the moment the observer moves and the moment the image corresponding to his new position is displayed. This time difference is called end-to-end latency. End-to-end latency is important because head rotations can be extremely fast and cause significant changes in the visible scene (Fig. 1).

Padmos and Milders [7] indicate that for immersive reality (where the observer cannot see the normal world), the end-to-end latency should be below 40 ms. For augmented reality the requirements will be even higher. They suggest that the displacement of objects between two frames should not exceed 15 arcmin (0.25°), which would require a maximal latency of 5 ms even when the observer rotates his head with a moderate speed of 50°/s. Several other authors use a similar approach [5], [6], [8], [9], [10] and come to similar maximal latency times. Actually, during typical head motions speeds of up to 370°/s may occur [11], and for fighter pilots speeds of up to 2000°/s have been reported [12]. But it is not likely that observers rotating their head that fast will notice slight object displacements. Many authors suggest that 10 ms will be acceptable for AR [8], [13], [14].

For indoor tracking, the required positional and rotational precision are realisable, as current trackers are capable of a rotational accuracy in the order of 0.25° and a positional accuracy of 0.25 in [15]. For outdoors, systems are still under development [16]. The latency requirement of 10 ms is extreme: it is an order of magnitude smaller than the latency of current systems.

In this paper we describe how the requirements could be met with a combination of several levels of position and orientation tracking with different relative and absolute accuracies, and several levels of rendering to reduce the complex 3D data to simple scenes that can be rendered just-in-time. In Section 2 we first describe the context of our research, the Ubicom project, a multi-disciplinary project carried out at Delft University of Technology, which aims at the development of a system for Ubiquitous Communication. In 3 Tracking, 4 Low-latency rendering, 5 First result we focus on the problem of image stabilisation and discuss latency issues related to position tracking and display. We summarize our system set-up in Section 6 and conclude with describing the current state in Section 7.