3DPD: A photogrammetric pipeline for a PUSH frame stereo cameras

https://doi.org/10.1016/j.pss.2021.105165Get rights and content

Highlights

  • 3-dimensional reconstruction software for planetary surfaces.

  • High resolution DTMs of targets on the Mars surface provided by CaSSIS Stereo Camera images.

  • Photogrammetric pipeline for push frame instruments from CaSSIS to Bepicolombo stereo camera STC/SIMBIOSYS.

  • Archiving of the Digital Terrain Models obtained by the CaSSIS stereo pairs.

Abstract

An innovative photogrammetric pipeline has been developed by INAF-Padova for the processing of the stereo images from the CaSSIS (Colour and Stereo Imaging System) (Thomas et al., 2014). CaSSIS is the multispectral stereo push frame camera on board ExoMars TGO (Trace Gas Orbiter) which will image 1.5% of the Mars surface in stereo mode with a spatial resolution of 4.6 ​m/pixel: the highest resolution single pass stereo capability currently operating at Mars. Data acquisition started in April 2018. The camera is able to provide two images of the same target from two different points of view along the same orbit and within one minute. The telescope is mounted on a rotational stage and its boresight is oriented to 10° with respect to nadir direction. After the acquisition of the first set of images looking forward along track, the rotational stage is rotated by 180° and a second set of images (looking backward) is acquired. The stereo pairs can then be processed to provide the 3D topography of specific targets.

The suite of photogrammetry and imaging tools, named 3DPD (3Dimensional reconstruction of Planetary Data) (Simioni et al. 2017), is designed for processing stereo push frame data and producing the three-dimensional data for geomorphological analysis of planetary surfaces.

The workflow involves a MATLAB tool for the preparation of the inputs (the mosaicked images and the projection matrices) to be ingested into the 3DPD matching core software. The pipeline is in continuous development and routinely ingests a large number of images that CaSSIS is presently acquiring and will continue to acquire in the future. CaSSIS 3DPD products are the unique DTMs available nowadays and the stereo products have been considered in some scientific work (as described in Section 6.2). The same pipeline faces the need of a dedicated pipeline for the Mercury Global Mapping with the Spectrometers and Imagers for the Mercury Planetary Orbiter (MPO) BepiColombo Integrated Observatory SYStem (SIMBIO-SYS) (Cremonese et al., 2020).

Introduction

The aim of this article is to introduce the 3D reconstruction software for planetary surfaces developed (Simioni et al., 2017) by the research group of the Astronomical Observatory of Padova (OAPD-INAF). The group is involved in different imaging systems on board space missions, such as BepiColombo, ExoMars 2016 and JUICE, that are acquiring, or will acquire, stereo pairs of planetary surfaces.

BepiColombo has been launched on 20 October 2018 towards planet Mercury. SIMBIO-SYS, on board the spacecraft, is equipped with a stereo imaging channel (STC) designed to obtain a global coverage of the planet surface in stereo mode.

The ExoMars Trace Gas Orbiter (EM-TGO) mission, which has been launched on 16 March 2016, is equipped with the stereo camera CaSSIS (Colour and Stereo Imaging System) (Thomas et al., 2014) which has provided until 29th MTP (Medium Term Planning corresponding to 1 month of operation) 1613 stereo-pairs (August 2020) with different acquisition strategies and different filters modes.

The JUICE mission, which will be launched in 2022 with the camera JANUS on board will acquire stereo pairs of the surfaces of the icy satellites of Jupiter, primarily Ganymede.

The INAF-OAPD institute, as part of the CaSSIS team, has the responsibility for generating and archiving the Digital Terrain Models (DTMs). The DTMs archiving group (Cremonese and Re, 2018) implemented a Repository Dynamic Interface for the management of the requests of DTMs reconstruction and the delivery of the stereo products (https://cassis.oapd.inaf.it/archive/).

STC and CaSSIS use the same type of detector and are both based on the push-frame stereo acquisition mode. The photogrammetric procedures applied for the topographic reconstruction of the planetary surfaces in three dimensions are the same and most of the efforts devoted to the design and integration of the processes will be shared.

In the first Section, an introduction describing the instrumental context is proposed. Section 3 describes the developed software, the structure and all the implemented methods. Section 4 is dedicated to the user interface while the details of the base pipeline are illustrated in Section 5.

Section snippets

Pipeline context

Stereo photogrammetry is a digital image processing technique that allows to metrically extract information about the target observed; providing important support for geomorphological analysis and for the quantitative investigation of the features examined by the camera. Combining a variety of techniques, such as camera calibration, image matching and bundle adjustment, photogrammetry is able to derive 3D information from a pair or a set of overlapping images (Wu, 2017).

The result of this

3DPD pipeline

Fig. 1 shows the block diagram of the photogrammetric pipeline.

It consists of several steps:

  • -

    the creation of the mosaiced images (see Section 3.1) starting from the framelets (single shot acquisitions),

  • -

    the computation of the integer disparity map (see Section 3.2.2) (which define the parallax between the images),

  • -

    the disparity refinement (see 3.2.3) at the sub-pixel level and

  • -

    finally, the triangulation phase.

All these steps will be described in detail in the following sections.

GUI (Graphical user interface)

Even though the software has been designed to be as much automatic as possible, a user-friendly interactive processing is possible through the software’s Graphical User Interface (GUI). The interactive approach offers more control over specific and critical steps of the photogrammetric process.

Through the GUI it is possible to choose different steps of the workflow which have to be executed as well as to define related parameters, such as window template dimension, search area dimension, number

Technical description and architecture of the software

The software is developed in C SHARP language and it takes advantage of pre-processing tools developed in MATLAB.

The code is multithreaded and works on the Windows Operating System (Windows 7 SP 1 or later, 64 bit). It is recommended a configuration Intel Core i7 multi-processor with at least 16 ​GB of RAM.

The input framelets/images can be accepted in EDR (Experiment Data Records) format and/or in other more common image formats (JPG, TIFF, PNG) or even as MATLAB binary file. Camera parameters

Validation and performance

At the base of the validation procedure is the estimation of the performance and of the vertical precision by collecting statistics on the differences between the stereo DTM and a “ground truth” (another topographic data of higher accuracy that can be used as reference) (Da Deppo et al., 2006), in the following tests:

  • (i)

    HERSCHEL_CRATER: based on a pair of synthetic images generated by using the HiRISE Texturized DTM.

  • (ii)

    CASSIS_IMAGES: status of the 3D reconstruction with the CaSSIS images

  • (iii)

    STC_SVS:

Conclusions

A photogrammetric stereo processing workflow has been developed for the 3D reconstruction from CaSSIS images. The pipeline includes a procedure for the image pre-processing management and different approaches for the stereo image matching. Two different tests based on synthetic and real images are presented to validate the pipeline. At the same time, the first stereo images acquired by the CaSSIS instrument have been processed to achieve the best possible accuracy relying on a preliminary

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This activity has been realized under the Italian Space Agency ASI-INAF agreement no. 2017-03-17. Under the contract the authors have made the pipeline available for the general users on proper workstation settled in their laboratories. We are currently considering to distribute a completely open release of the 3DPD at the end of the contract. A first SUM (Software User Manual) can be found as technical note (Simioni et al., 2020).

The authors wish to thank the spacecraft and instrument

References (48)

  • W.S. Ching

    Normalized cross-correlation: the contrast-dependent problem and its solution

    J. Electron. Imag.

    (1995)
  • G. Cremonese et al.

    The CaSSIS Digital Terrain Model generation and Archiving at OAPD

    European Planetary Science Congress

    (2018)
  • G. Cremonese et al.

    March). First Mars surface stereo reconstruction with the CaSSIS stereo camera

  • G. Cremonese

    The stereo camera on the BepiColombo ESA/JAXA mission: a novel approach

    Adv. Geosci.

    (2009)
  • G. Cremonese et al.

    SIMBIO-SYS: scientific cameras and spectrometer for the BepiColombo mission

    Space Sci. Rev.

    (2020)
  • V. Da Deppo et al.

    June). A novel optical design for planetary surface stereo-imaging: preliminary design of the Stereoscopic Imaging Channel of SIMBIOSYS for the BepiColombo ESA mission

  • V. Da Deppo et al.

    Optical design of the single-detector planetary stereo camera for the BepiColombo European Space Agency mission to Mercury

    Appl. Optic.

    (2010)
  • A. Gruen

    Adaptive least squares correlation: a powerful image matching technique

    South Afr. J. Photogrammetry, Remote Sens. Cartogr.

    (1985)
  • A. Gruen

    Development and status of image matching in photogrammetry

    Photogramm. Rec.

    (2012)
  • H.U. Keller

    OSIRIS–The scientific camera system onboard Rosetta

    Space Sci. Rev.

    (2007)
  • B.R. Kent

    3D Scientific Visualization with Blender

    (2014)
  • R.L. Kirk

    High- resolution topomapping of candidate MER landing sites with Mars Orbiter Camera narrowangle images

    J. Geophys. Res.

    (2003)
  • R.L. Kirk et al.

    The effect of illumination on stereo dtm quality:simulations in support of europa exploration. , XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic 10.5194/isprs-annals-III-4-103-2016[ES1] [ES1]Corrected a quote at line 671

    ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences

    (2016)
  • J.P. Lewis

    Fast Template Matching, Vision Interface 95, Canadian Image Processing and Pattern Recognition Society, Quebec City, Canada, May 15-19

    (1995)
  • Cited by (16)

    • In-flight radiometric calibration of the ExoMars TGO Colour and Stereo Surface Imaging System

      2022, Planetary and Space Science
      Citation Excerpt :

      An imaging mode is the combination of parameters that defines a particular observation, these include the number of colour channels (up to four), image compression settings, and final swath size. In addition, CaSSIS is capable of acquiring nearly-simultaneous stereo images that can be combined to produce Digital Terrain Models of the surface and anaglyphs (Simioni et al., 2021, Re et al. this issue). Common to all possible imaging modes is the general scheme for colour imaging with CaSSIS which combines a “push-frame” technique with colour filters deposited on a glass window fixed in front of the detector (Fig. 1).

    • CaSSIS-based stereo products for Mars after three years in orbit

      2022, Planetary and Space Science
      Citation Excerpt :

      The DTM spatial resolution is strictly related to the image quality and to the accuracy of the orientation data so the grid spacing depends on the acquisition conditions. For the time being, a nominal stereo pair acquired at 4.6 m/pix provides a DTM with 15 m/pix post sampling as for the case presented in (Simioni et al., 2021). The 3DPD software also permits the generation of DTMs “at every pixel” (on a grid with post spacing equal to the image resolution: 4.6 m/pix).

    • Geology, in-situ resource-identification and engineering analysis of the Vernal crater area (Arabia Terra): A suitable mars human landing site candidate

      2022, Planetary and Space Science
      Citation Excerpt :

      On a specific area located inside the Vernal crater (Fig. 5D), we planned and acquired a CaSSIS stereo pair characterised by a spatial scale of 4.0 ​m. In order to prepare the CaSSIS DTM, following the procedure described in Simioni et al. (2021), the undistorted panchromatic images have been first mosaicked after a bundle adjustment process for both intrinsic and extrinsic parameters with an on-ground grid size equal to the image pixel resolution. The matching workflow then focused on a feature-based extraction of the seed points.

    View all citing articles on Scopus
    View full text