skip to main content
research-article

BigSUR: large-scale structured urban reconstruction

Published:20 November 2017Publication History
Skip Abstract Section

Abstract

The creation of high-quality semantically parsed 3D models for dense metropolitan areas is a fundamental urban modeling problem. Although recent advances in acquisition techniques and processing algorithms have resulted in large-scale imagery or 3D polygonal reconstructions, such data-sources are typically noisy, and incomplete, with no semantic structure. In this paper, we present an automatic data fusion technique that produces high-quality structured models of city blocks. From coarse polygonal meshes, street-level imagery, and GIS footprints, we formulate a binary integer program that globally balances sources of error to produce semantically parsed mass models with associated facade elements. We demonstrate our system on four city regions of varying complexity; our examples typically contain densely built urban blocks spanning hundreds of buildings. In our largest example, we produce a structured model of 37 city blocks spanning a total of 1, 011 buildings at a scale and quality previously impossible to achieve automatically.

Skip Supplemental Material Section

Supplemental Material

a204-kelly.mp4

References

  1. Lama Affara, Liangliang Nan, Bernard Ghanem, and Peter Wonka. 2016. Large Scale Asset Extraction for Urban Images. ECCV (2016), 437--452.Google ScholarGoogle Scholar
  2. Oswin Aichholzer, Franz Aurenhammer, David Alberts, and Bernd Gärtner. 1996. A novel type of skeleton for polygons. In The Journal of Universal Computer Science. Springer, 752--761.Google ScholarGoogle Scholar
  3. Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. 2010. Google street view: Capturing the world at street level. Computer 43, 6 (2010), 32--38. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE TPAMI (2017).Google ScholarGoogle Scholar
  5. Therese Biedl, Stefan Huber, and Peter Palfrader. 2016. Planar matchings for weighted straight skeletons. International Journal of Computational Geometry & Applications 26, 03n04 (2016), 211--229. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Claus Brenner. 2005. Building reconstruction from images and laser scanning. International Journal of Applied Earth Observation and Geoinformation 6, 3 (2005), 187--198.Google ScholarGoogle ScholarCross RefCross Ref
  7. Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. 2008. Segmentation and Recognition Using Structure from Motion Point Clouds. ECCV (2008), 44--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Duygu Ceylan, Minh Dang, Niloy J. Mitra, Boris Neubert, and Mark Pauly. 2016. Discovering Structured Variations Via Template Matching. CGF (01 2016). Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Duygu Ceylan, Niloy J. Mitra, Youyi Zheng, and Mark Pauly. 2013. Coupled Structure-from-Motion and 3D Symmetry Detection for Urban Facades. ACM TOG (2013), 15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. John W. Chinneck. 2008. Feasibility and Infeasibility in Optimization: Algorithms and Computational Methods. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Andrea Cohen, Alexander G Schwing, and Marc Pollefeys. 2014. Efficient structured parsing of facades using dynamic programming. IEEE CVPR (2014), 3206--3213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The Cityscapes Dataset for Semantic Urban Scene Understanding. In IEEE CVPR.Google ScholarGoogle Scholar
  13. Dengxin Dai, Mukta Prasad, Gerhard Schmitt, and Luc Van Gool. 2012. Learning domain knowledge for facade labelling. ECCV (2012), 710--723. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. ACM SIGGRAPH (1996), 11--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Anthony R Dick, Philip HS Torr, and Roberto Cipolla. 2004. Modelling and interpretation of architecture from several images. IJCV 60, 2 (2004), 111--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. David Eigen and Rob Fergus. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. IEEE ICCV (2015), 2650--2658. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. David Eppstein and Jeff Erickson. 1999. Raising roofs, crashing cycles, and playing pool: Applications of a data structure for finding pairwise interactions. Discrete & Computational Geometry 22, 4 (1999), 569--592.Google ScholarGoogle ScholarCross RefCross Ref
  18. Tian Fang, Zhexi Wang, Honghui Zhang, and Long Quan. 2013. Image-based modeling of unwrappable facades. IEEE TVCG 19, 10 (2013), 1720--1731.Google ScholarGoogle Scholar
  19. K. Fukunaga and L. Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE TIT 21, 1 (January 1975), 32--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32, 8 (2010), 1362--1376. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ignacio Garcia-Dorado, Ilke Demir, and Daniel G Aliaga. 2013. Automatic urban modeling using volumetric reconstruction with surface graph cuts. Computers & Graphics 37, 7 (2013), 896--910. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Aleksey Golovinskiy, Vladimir G Kim, and Thomas Funkhouser. 2009. Shape-based recognition of 3D point clouds in urban environments. IEEE ICCV (2009), 2154--2161.Google ScholarGoogle Scholar
  23. Gurobi. 2016. Gurobi Optimizer Reference Manual. (2016). http://www.gurobi.comGoogle ScholarGoogle Scholar
  24. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).Google ScholarGoogle Scholar
  25. Haiyong Jiang, Liangliang Nan, Dong-Ming Yan, Weiming Dong, Xiaopeng Zhang, and Peter Wonka. 2016. Automatic constraint detection for 2D layout regularization. IEEE TVCG 22, 8 (2016), 1933--1944. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. SGP (2006), 61--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM TOG 32, 3 (2013), 29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Tom Kelly and Peter Wonka. 2011. Interactive architectural modeling with procedural extrusions. ACM TOG 30, 2 (2011), 14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. 2015. Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding. arXiv preprint arXiv.1511.02680 (2015).Google ScholarGoogle Scholar
  30. Mateusz Kozinski, Raghudeep Gadde, Sergey Zagoruyko, Guillaume Obozinski, and Renaud Marlet. 2015. A MRF shape prior for facade parsing with occlusions. IEEE CVPR (2015), 2820--2828.Google ScholarGoogle Scholar
  31. Florent Lafarge, Xavier Descombes, Josiane Zerubia, and Marc Pierrot-Deseilligny. 2010. Structural approach for building reconstruction from a single DSM. IEEE TPAMI 32, 1 (2010), 135--147. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Robert G Laycock and AM Day. 2003. Automatically generating large urban environments based on the footprint data of buildings. ACM SMA (2003), 346--351. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Minglei Li, Peter Wonka, and Liangliang Nan. 2016. Manhattan-world urban reconstruction from point clouds. ECCV (2016), 54--69.Google ScholarGoogle Scholar
  34. Yangyan Li, Xiaokun Wu, Yiorgos Chrysathou, Andrei Sharf, Daniel Cohen-Or, and Niloy J Mitra. 2011a. Globfit: Consistently fitting primitives by discovering global relations. ACM SIGGRAPH 30, 4 (2011), 52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Yangyan Li, Qian Zheng, Andrei Sharf, Daniel Cohen-Or, Baoquan Chen, and Niloy J Mitra. 2011b. 2D-3D fusion for layer decomposition of urban facades. IEEE ICCV (2011), 882--889. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Hui Lin, Jizhou Gao, Yu Zhou, Guiliang Lu, Mao Ye, Chenxi Zhang, Ligang Liu, and Ruigang Yang. 2013. Semantic decomposition and reconstruction of residential scenes from LiDAR data. ACM SIGGRAPH 32, 4 (2013), 66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Andelo Martinović, Markus Mathias, Julien Weissenberg, and Luc Van Gool. 2012. A three-layered approach to facade parsing. ECCV (2012), 416--429. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Bogdan C Matei, Harpreet S Sawhney, Supun Samarasekera, Janet Kim, and Rakesh Kumar. 2008. Building segmentation for densely built urban regions using aerial lidar data. IEEE CVPR (2008), 1--8.Google ScholarGoogle Scholar
  39. Peter Miller et al. 2017. Buildings - OpenStreetMap Wiki. (2017). Retrieved August 8, 2017 from http://wiki.openstreetmap.org/wiki/BuildingsGoogle ScholarGoogle Scholar
  40. Aron Monszpart, Nicolas Mellado, Gabriel J Brostow, and Niloy J Mitra. 2015. RAPter: rebuilding man-made scenes with regular arrangements of planes. ACM SIGGRAPH 34, 4 (2015), 103--1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Pascal Müller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Luc Van Gool. 2006. Procedural modeling of buildings. ACM SIGGRAPH 25, 3 (2006), 614--623. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Pascal Müller, Gang Zeng, Peter Wonka, and Luc Van Gool. 2007. Image-based procedural modeling of facades. ACM SIGGRAPH 26, 3 (2007), 85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, L v Gool, and Werner Purgathofer. 2013. A survey of urban reconstruction. CGF 32, 6 (2013), 146--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Liangliang Nan, Caigui Jiang, Bernard Ghanem, and Peter Wonka. 2015. Template assembly for detailed urban reconstruction. CGF Eurographics 34, 2 (2015), 217--228. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Liangliang Nan, Andrei Sharf, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. 2010. Smartboxes for interactive urban reconstruction. ACM SIGGRAPH 29, 4 (2010), 93. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Gen Nishida, Ignacio Garcia-Dorado, Daniel G Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive sketching of urban procedural models. ACM SIGGRAPH 35, 4 (2016), 130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Charalambos Poullis and Suya You. 2009. Automatic reconstruction of cities from remote sensor data. IEEE CVPR (2009), 2775--2782.Google ScholarGoogle Scholar
  48. Hayko Riemenschneider, Ulrich Krispel, Wolfgang Thaller, Michael Donoser, Sven Havemann, Dieter Fellner, and Horst Bischof. 2012. Irregular lattices for complex shape grammar facade parsing. IEEE CVPR (2012), 1640--1647. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. David Salinas, Florent Lafarge, and Pierre Alliez. 2015. Structure-Aware Mesh Decimation. CGF 34, 6 (2015), 211--227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Chao-Hui Shen, Shi-Sheng Huang, Hongbo Fu, and Shi-Min Hu. 2011. Adaptive partitioning of urban facades. ACM SIGGRAPH Asia 30, 6 (2011), 184. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006. Photo tourism: exploring photo collections in 3D. ACM SIGGRAPH 25, 3 (2006), 835--846. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Olivier Teboul, Iasonas Kokkinos, Loic Simon, Panagiotis Koutsourakis, and Nikos Paragios. 2013. Parsing facades with shape grammars and reinforcement learning. IEEE TPAMI 35, 7 (2013), 1744--1756. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Radim Tylecek. 2012. The CMP facade database. Technical Report. Tech. rep., CTU-CMP-2012-24, Czech Technical University.Google ScholarGoogle Scholar
  54. Carlos A Vanegas, Daniel G Aliaga, and Bedřich Beneš. 2010. Building reconstruction using manhattan-world grammars. IEEE CVPR (2010), 358--365.Google ScholarGoogle Scholar
  55. Yannick Verdie, Florent Lafarge, and Pierre Alliez. 2015. LOD generation for urban scenes. Technical Report 3. 30:1--30:14 pages.Google ScholarGoogle Scholar
  56. Jianxiong Xiao, Tian Fang, Ping Tan, Peng Zhao, Eyal Ofek, and Long Quan. 2008. Image-based façade modeling. ACM SIGGRAPH Asia 27, 5 (2008), 161.Google ScholarGoogle Scholar
  57. Chao Yang, Tian Han, Long Quan, and Chiew-Lan Tai. 2012. Parsing façade with rank-one approximation. IEEE CVPR (2012), 1720--1727. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Qian Zheng, Andrei Sharf, Guowei Wan, Yangyan Li, Niloy J Mitra, Daniel Cohen-Or, and Baoquan Chen. 2010. Non-local scan consolidation for 3D urban scenes. ACM SIGGRAPH 29, 4 (2010), 94. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Qian-Yi Zhou and Ulrich Neumann. 2010. 2.5D dual contouring: a robust approach to creating building models from aerial lidar point clouds. ECCV (2010), 115--128. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. BigSUR: large-scale structured urban reconstruction

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Graphics
            ACM Transactions on Graphics  Volume 36, Issue 6
            December 2017
            973 pages
            ISSN:0730-0301
            EISSN:1557-7368
            DOI:10.1145/3130800
            Issue’s Table of Contents

            Copyright © 2017 ACM

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 20 November 2017
            Published in tog Volume 36, Issue 6

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader