Large-scale urban point cloud labeling and reconstruction

https://doi.org/10.1016/j.isprsjprs.2018.02.008Get rights and content

Abstract

The large number of object categories and many overlapping or closely neighboring objects in large-scale urban scenes pose great challenges in point cloud classification. In this paper, a novel framework is proposed for classification and reconstruction of airborne laser scanning point cloud data. To label point clouds, we present a rectified linear units neural network named ReLu-NN where the rectified linear units (ReLu) instead of the traditional sigmoid are taken as the activation function in order to speed up the convergence. Since the features of the point cloud are sparse, we reduce the number of neurons by the dropout to avoid over-fitting of the training process. The set of feature descriptors for each 3D point is encoded through self-taught learning, and forms a discriminative feature representation which is taken as the input of the ReLu-NN. The segmented building points are consolidated through an edge-aware point set resampling algorithm, and then they are reconstructed into 3D lightweight models using the 2.5D contouring method (Zhou and Neumann, 2010). Compared with deep learning approaches, the ReLu-NN introduced can easily classify unorganized point clouds without rasterizing the data, and it does not need a large number of training samples. Most of the parameters in the network are learned, and thus the intensive parameter tuning cost is significantly reduced. Experimental results on various datasets demonstrate that the proposed framework achieves better performance than other related algorithms in terms of classification accuracy and reconstruction quality.

Introduction

Three dimensional (3D) modeling of urban buildings from point clouds has long been an active research topic in photogrammetry, GIS and remote sensing communities. Many methods have been developed from airborne laser scanning (ALS) point cloud-based building reconstruction (Lafarge and Mallet, 2012, Lin et al., 2013) to terrestrial laser scanning (TLS) or mobile laser scanning (MLS) point cloud-based street-side modeling (Frueh et al., 2005, Nan et al., 2010). Due to the large number of object categories and many overlapping or closely neighboring objects in large-scale urban scenes, automatic 3D urban reconstruction remains an open problem (Musialski et al., 2013).

In this paper, we propose a novel framework for ALS point cloud classification and building reconstruction. Fig. 1 illustrates the main process of labeling point clouds and modeling the segmented building points. A Rectified Linear Units Neural Network (ReLu-NN) is introduced to classify the point clouds. Before the training process, we compute the point-based feature descriptors and encode them by using the sparse autoencoder for obtaining discriminative features. The features are then taken as the input layer of the ReLu-NN. In the training process, we utilize the ReLu instead of the traditional sigmoid as the activation function of the network. The segmentation method based on the Euclidean distance (Rusu, 2009) divides building points from the classification results. Since the segmented building point clouds are often noisy, a structure-aware repairing method is introduced to consolidate them. Finally, lightweight buildings are reconstructed from the point cloud by using the contouring method (Zhou and Neumann, 2010).

The main contributions of our approach are threefold:

  • (i)

    An urban point cloud labeling and building modeling approach is presented. This approach provides a complete description of the process from the input of the ALS point clouds to the building reconstruction.

  • (ii)

    The ReLu-NN introduced can easily classify unorganized point clouds without rasterizing the data. Most of the parameters in the network are learned, and thus the intensive parameter tuning cost is significantly reduced. The training does not need a large number of training samples, and it achieves high labeling performances even when the scenes are complex.

  • (iii)

    The data consolidation and modeling algorithms generate a high generalization and compact representations of urban buildings with complex roof components.

Section snippets

Point cloud classification

Feature representation is a key step for point cloud classification. Much work focuses on point-based features and local neighbourhood features (Vosselman, 2013, Xu et al., 2014, Martinovic et al., 2015, Li et al., 2017a, Li et al., 2017b). Li et al. (2016) present a three-step point cloud parsing framework to formulate and solve a joint classification and segmentation problem from TLS point clouds. They first compute the feature per 3D point, then a linear SVM classifier is trained, and the

Construction of feature descriptors for each point

The architectures of tree crowns are generally complex while structures of building facades and roofs are smooth. In this paper, we use a method similar to our previous one (Li et al., 2017b) to obtain the features from different datasets.

For the TLS point cloud, the full descriptor per point pi is:

FiTLS119×1=RGBiT3×1LABiT3×1niT3×1FspinT108×1heightiT1×1heighti-1T1×1, where FiTLS is a 119-dimensional vector; RGBi is the mean RGB colors of pi as seen in the camera images; LABi is the LAB values

Building reconstruction

After the class building is determined, we use the distance-based segmentation approach (Rusu, 2009) to divide the building points from the point cloud. Reconstruction of 3D buildings from the segmented building point cloud is a challenge due to the poor-quality data, contamination with missing data, noise and outliers. To overcome this problem, we first consolidate the segmented point clouds, and enhance the quality of the data through removing the noise, outliers and recovering the regions

Experimental results

To validate the performance of our method, we carry out both qualitative and quantitative evaluations on various large-scale point clouds.

Discussions

The presented ReLu-NN still utilizes hand-crafted features for each modality independently and combine them in a heuristic manner. It often fails to consider the consistency and complementary information among features adequately. Currently, the features learned by the deep learning approaches can obtain high-quality image classification results. However, most of the state-of-the-art deep learning-based methods like Qi et al. (2017) and Liu et al. (2017) are hard to apply to parse large-scale

Conclusions

In this paper, we have presented a framework for urban scene labeling and building modeling from outdoor point clouds. The ReLu-NN is applied to parse each point in the point cloud into one semantic class. It can accurately label the points from the point clouds of complex urban environments. Experimental results show that the ReLu-NN parses the scenes efficiently and effectively, and it is capable of recognizing different class objects with similar shapes. Compared with other methods, our

References (43)

  • C. Frueh et al.

    Data processing algorithms for generating textured 3D building facade meshes from laser scans and camera images

    Int. J. Comput. Vis.

    (2005)
  • X. Glorot et al.

    Deep sparse rectifier neural networks

    Aistats

    (2011)
  • H. Guan et al.

    Deep learning-based tree classification using mobile LiDAR data

    Rem. Sens. Lett.

    (2015)
  • N. Haala et al.

    Digital photogrammetric camera evaluation–generation of digital elevation models

    Photogrammetrie-Fernerkundung-Geoinformation

    (2010)
  • Hinton, G., et al., 2012. Improving Neural Networks by Preventing Co-adaptation of Feature Detectors. Available from:...
  • H. Huang et al.

    Edge-aware point set resampling

    ACM Trans. Graph.

    (2013)
  • H. Huang et al.

    Consolidation of unorganized point clouds for surface reconstruction

    ACM Trans. Graph.

    (2009)
  • C. Haene et al.

    Dense semantic 3D reconstruction

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • F. Lafarge et al.

    Creating large-scale city models from 3D-point clouds: a robust approach with hybrid representation

    Int. J. Comput. Vis.

    (2012)
  • Y. LeCun et al.

    Deep learning

    Nature

    (2015)
  • Z. Li et al.

    Classification of urban point clouds: a robust supervised approach with automatically generating training data

    IEEE J. Sel. Top. Appl. Earth Obs. Rem. Sens.

    (2017)
  • Cited by (76)

    • Conservation and management of Chinese classical royal garden heritages based on 3D digitalization - A case study of Jianxin courtyard in Jingyi garden in fragrant hills

      2022, Journal of Cultural Heritage
      Citation Excerpt :

      of these cultural sites. [18,19]. The digital collection and archiving of Chinese traditional gardens has made progress; however, there are many important elements of artificial and natural surface features in Chinese royal gardens, amongst which there are many forms and complex classification objects, including water bodies, rocks, plants, buildings, topography, historical information, etc., which causes the scene to have diverse targets, complex morphological structures, target occlusion and overlapping, big spatial density differences, etc., resulting in common problems with the fine classification of 3D point clouds of this kind of cultural heritage [5,20-23]. Fine classification of 3D point clouds is the process of identifying and extracting artificial and natural surface features from disordered point clouds, and it is the basis of subsequent applications, such as digital ground model generation and 3D reconstruction of complex scenes [1,24,25].

    • DCPLD-Net: A diffusion coupled convolution neural network for real-time power transmission lines detection from UAV-Borne LiDAR data

      2022, International Journal of Applied Earth Observation and Geoinformation
      Citation Excerpt :

      Recent methods, such as the PCCN (Zhang et al., 2019) and Superpoint Graph (Landrieu and Simonovsky, 2018) try to improve the global and local context integration by extending the convolution operator to high-dimensional lattice or graph to handle the unordered raw point clouds. Zhang et al.(2018) encode the point-based feature descriptors with a sparse autoencoder to obtain features and process those features using the proposed ReLu-NN for point cloud classification. Computing the features of non-object points will bring more computation costs and increase the difficulty of detection.

    • A new weakly supervised approach for ALS point cloud semantic segmentation

      2022, ISPRS Journal of Photogrammetry and Remote Sensing
      Citation Excerpt :

      The 3D coordinates and associated attributes (e.g., laser reflectance and return count information) are usually contained in ALS point clouds. To fully interpret a complex geographical scene, the key step is to acquire semantic information as a valuable cue utilized in various remote sensing applications, such as land cover surveys (Yan et al., 2015), forest monitoring (Yao et al., 2012; Polewski et al., 2015), change detection (Okyay et al., 2019), and 3D mapping (Fan et al., 2014; Zhang et al., 2018). Semantic segmentation, or classification, which usually assigns a label to each point, is an indispensable solution for point cloud parsing.

    • Data acquisition for urban building energy modeling: A review

      2022, Building and Environment
      Citation Excerpt :

      As LiDAR is efficient in object scanning and capable of providing sufficient point clouds, its data completeness is high, reaching 94%–100% [67]. In addition, its data accuracy is also good, which could be more than 85%, as advanced algorithms have been proposed to classify point clouds [64,67,109,110]. Similar to LiDAR, oblique photogrammetry data also have high completeness due to the sufficient point clouds provided.

    View all citing articles on Scopus
    View full text