MVE—An image-based reconstruction environment

Highlights

• End-to-end multi-view geometry reconstruction and texturing pipeline.

• Multi-scale reconstruction approach.

Abstract

We present an image-based reconstruction system, the Multi-View Environment. MVE is an end-to-end multi-view geometry reconstruction software which takes photos of a scene as input and produces a textured surface mesh as result. The system covers a structure-from-motion algorithm, multi-view stereo reconstruction, generation of extremely dense point clouds, reconstruction of surfaces from point clouds, and surface texturing. In contrast to most image-based geometry reconstruction approaches, our system is focused on reconstruction of multi-scale scenes, an important aspect in many areas such as cultural heritage. It allows to reconstruct large datasets containing some detailed regions with much higher resolution than the rest of the scene. Our system provides a graphical user interface for visual inspection of the individual steps of the pipeline, i.e., the structure-from-motion result, multi-view stereo depth maps, and rendering of scenes and meshes.

Introduction

Acquiring geometric data from natural and man-made objects or scenes is a fundamental field of research in computer vision and graphics. 3D digitization is relevant for designers, the entertainment industry, and for the preservation as well as digital distribution of cultural heritage objects and sites. In this paper, we introduce MVE, the Multi-View Environment, a free software solution for low-cost geometry acquisition from images. The system takes as input a set of photos and provides the algorithmic steps necessary to obtain a high-quality surface mesh of the captured object as final output. This includes structure-from-motion, multi-view stereo, surface reconstruction and texturing.

Geometric acquisition approaches are broadly classified into active and passive scanning. Active scanning technologies for 3D data acquisition exist in various flavors. Time of flight and structured light scanners are known to produce geometry with remarkable detail and accuracy. But these systems require expensive hardware and elaborate capture planning and execution. Real-time stereo systems such as the Kinect primarily exist for the purpose of gaming, but are often used for real-time geometry acquisition. These systems are based on structured infra-red light which is emitted into the scene. They are often of moderate quality and limited to indoor settings because of inference with sunlight׳s infrared component. Finally, there is some concern that active systems may damage objects of cultural value due to intense light emission.

Passive scanning systems do not emit light, are purely based on the existing illumination, and will not physically affect the subject matter. The main advantage of these systems is the cheap capture setup which does not require special hardware: A consumer-grade camera (or just a smartphone) is enough to capture datasets. The reconstruction process is based on finding visual correspondences in the input images, which, compared to active systems, usually leads to less complete geometry, and limits the scenes to static, well-textured surfaces. The inexpensive demands on the capture setup, however, come at the cost of much more elaborate computer software to process the unstructured input. The standard pipeline for geometry reconstruction from images involves four major algorithmic steps (see Fig. 1):

• Structure-from-Motion (SfM) infers the extrinsic camera parameters (position and orientation) and the camera calibration (focal length and radial distortion) by finding sparse but stable correspondences between images. A sparse point-based 3D representation of the subject is created as a by-product of camera reconstruction.

• Multi-View Stereo (MVS) reconstructs dense 3D geometry by finding visual correspondences in the images using the estimated camera parameters. These correspondences are triangulated yielding dense 3D information.

• Surface Reconstruction takes as input a dense point cloud or individual depth maps and produces a globally consistent surface mesh.

• Surface Texturing computes a consistent texture for the surface mesh using the input images.

It is not surprising that software solutions for end-to-end passive geometry reconstruction are rare. The reason lies in the technical complexity and the effort required to create such tools. Many projects cover parts of the pipeline, such as Bundler [1], VisualSfM [2], or OpenMVG [3] for structure-from-motion reconstruction, PMVS [4] for multi-view stereo, and Poisson Surface Reconstruction [5] for mesh reconstruction. A few commercial software projects offer complete end-to-end pipelines covering SfM, MVS, Surface Reconstruction and Texturing. This includes Arc3D, Agisoft Photoscan and Acute3D Smart3DCapture. All of them are, however, closed source and do not facilitate research. In contrast, we offer a complete pipeline as a free, open source software system, which was introduced in an earlier version of this paper [6].

Our system handles many kinds of scenes, such as compact objects, open outdoor scenes, and controlled studio datasets. It avoids to fill holes in regions with insufficient data for a reliable reconstruction. This may leave holes in the surfaces but does not introduce artificial geometry, common to many global reconstruction approaches. Our software puts a special emphasis on multi-resolution datasets which can contain very detailed regions in otherwise less detailed datasets. It has been shown that inferior results are produced if the multi-resolution nature of the input data is not considered properly [7], [8], [9].

In the paper׳s remainder we first give a technical overview of our system and introduce its individual components in Section 2. A few practical aspects and limitations of our system are discussed in Section 3. We then show reconstruction results on several datasets with different characteristics and demonstrate the versatility of our pipeline in Section 4. We briefly describe our software framework and conclude in Section 5.