Skip to main content
SpringerLink
Log in

Fully Automatic Registration of Image Sets on Approximate Geometry

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The photorealistic acquisition of 3D objects often requires color information from digital photography to be mapped on the acquired geometry, in order to obtain a textured 3D model. This paper presents a novel fully automatic 2D/3D global registration pipeline consisting of several stages that simultaneously register the input image set on the corresponding 3D object. The first stage exploits Structure From Motion (SFM) on the image set in order to generate a sparse point cloud. During the second stage, this point cloud is aligned to the 3D object using an extension of the 4 Point Congruent Set (4PCS) algorithm for the alignment of range maps. The extension accounts for models with different scales and unknown regions of overlap. In the last processing stage a global refinement algorithm based on mutual information optimizes the color projection of the aligned photos on the 3D object, in order to obtain high quality textures. The proposed registration pipeline is general, capable of dealing with small and big objects of any shape, and robust. We present results from six real cases, evaluating the quality of the final colors mapped onto the 3D object. A comparison with a ground truth dataset is also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Listing 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Aiger, D., Mitra, N. J., & Cohen-Or, D. (2008). 4-Points congruent sets for robust pairwise surface registration. ACM Transactions on Graphics, 27, 85:1–85:10.

    Article  Google Scholar 

  • Arya, S., Mount, D. M., Netanyahu, N. S., Silverman, R., & Wu, A. Y. (1998). An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM, 45, 891–923.

    Article  MathSciNet  MATH  Google Scholar 

  • Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Bonarrigo, F., & Signoroni, A. (2011). An enhanced ‘optimization-on-a-manifold’ framework for global registration of 3D range data. In Proc. of 3DIMPVT ’11 (pp. 350–357). Washington: IEEE Computer Society.

    Google Scholar 

  • Bradley, D., Boubekeur, T., & Heidrich, W. (2008). Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In IEEE conference on computer vision and pattern recognition (CVPR2008) (pp. 1–8).

    Chapter  Google Scholar 

  • Brown, M., & Lowe, D. (2003). Recognising panoramas. In Proc. int. conf. computer vision (Vol. 2, pp. 1218–1225).

    Chapter  Google Scholar 

  • Brown, M., & Lowe, D. G. (2005). Unsupervised 3D object recognition and reconstruction in unordered datasets. In Proc. int. conf. on 3D digital imaging and modeling.

    Google Scholar 

  • Brunie, L., Lavallée, S., & Szeliski, R. (1992). Using force fields derived from 3D distance maps for inferring the attitude of a 3D rigid object. In Proc. of the second European conference on computer vision (ECCV’92) (pp. 670–675). Berlin: Springer.

    Google Scholar 

  • Callieri, M., Cignoni, P., Corsini, M., & Scopigno, R. (2008). Masked photo blending: mapping dense photographic dataset on high-resolution 3D models. Computers & Graphics, 32(4), 464–473.

    Article  Google Scholar 

  • Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., & Ranzuglia, G. (2008). Meshlab: an open-source mesh processing tool. In Sixth eurographics Italian chapter conference (pp. 129–136). Eurographics.

  • Cleju, I., & Saupe, D. (2007). Stochastic optimization of multiple texture registration using mutual information. In Proceedings of the 29th DAGM conference on pattern recognition (pp. 517–526). Berlin: Springer.

    Google Scholar 

  • Cohen-Steiner, D., Alliez, P., & Desbrun, M. (2004). Variational shape approximation. ACM Transactions on Graphics, 23, 905–914.

    Article  Google Scholar 

  • Corsini, M., Dellepiane, M., Ponchio, F., & Scopigno, R. (2009). Image-to-geometry registration: a mutual information method exploiting illumination-related geometric properties. Computer Graphics Forum, 28(7), 1755–1764.

    Article  Google Scholar 

  • Dellepiane, M., Callieri, M., Ponchio, F., & Scopigno, R. (2008). Mapping highly detailed colour information on extremely dense 3d models: the case of David’s restoration. Computer Graphics Forum, 27(8), 2178–2187.

    Article  Google Scholar 

  • Dellepiane, M., Marroquim, R., Callieri, M., Cignoni, P., & Scopigno, R. (2012). Flow-based local optimization for image-to-geometry projection. IEEE Transactions on Visualization and Computer Graphics, 18(3), 463–474.

    Article  Google Scholar 

  • Eisemann, M., De Decker, B., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., Theobalt, C., & Sellent, A. (2008). Floating textures. Computer Graphics Forum (Proc. of Eurographics), 27(2), 409–418.

    Article  Google Scholar 

  • Farenzena, M., Fusiello, A., & Gherardi, R. (2009). Structure-and-motion pipeline on a hierarchical cluster tree. In IEEE int. workshop on 3-D digital imaging and modeling, Kyoto, Japan.

    Google Scholar 

  • Fitzgibbon, A. W., & Zisserman, A. (1998). Automatic camera recovery for closed and open image sequences. In Proc. Europ. conf. computer vision (ECCV1998) (pp. 311–326).

    Google Scholar 

  • Franken, T., Dellepiane, M., Ganovelli, F., Cignoni, P., Montani, C., & Scopigno, R. (2005). Minimizing user intervention in registering 2D images to 3D models. The Visual Computer, 21(8–10), 619–628.

    Article  Google Scholar 

  • Früh, C., & Zakhor, A. (2003). Constructing 3D city models by merging aerial and ground views. IEEE Computer Graphics and Applications, 23, 52–61.

    Article  Google Scholar 

  • Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1362–1376.

    Article  Google Scholar 

  • Gehua Yang, G. Y., Becker, J., & Stewart, C. V. (2007). Estimating the location of a camera with respect to a 3d model. In Proc. of the sixth international conference on 3-D digital imaging and modeling (3DIM2007) (pp. 159–166). Los Alamitos: IEEE Computer Society.

    Chapter  Google Scholar 

  • Gherardi, R., Farenzena, M., & Fusiello, A. (2010). Improving the efficiency of hierarchical structure-and-motion. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR 2010) (pp. 1594–1600).

    Google Scholar 

  • Gibson, S., Cook, J., Howard, T., Hubbold, R., & Oram, D. (2002). Accurate camera calibration for off-line, video-based augmented reality. In IEEE/ACM int. symp. on mixed and augmented reality.

    Google Scholar 

  • Goesele, M., Curless, B., & Seitz, S. M. (2006). Multi-view stereo revisited. In Proc. of CVPR ’06 (Vol. 2, pp. 2402–2409). Los Alamitos: IEEE Computer Society.

    Google Scholar 

  • Horn, B. K. P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, A, 4(4), 629–642.

    Article  MathSciNet  Google Scholar 

  • Ikeuchi, K., Nakazawa, A., Hasegawa, K., & Ohishi, T. (2003). The great Buddha project: modeling cultural heritage for vr systems through observation. In Proc. of the 2nd IEEE/ACM international symposium on mixed and augmented reality (ISMAR’03) (p. 7). Los Alamitos: IEEE Computer Society.

    Chapter  Google Scholar 

  • Irschara, A., Zach, C., & Bischof, H. (2007). Towards wiki-based dense city modeling. In Proc. int. conf. computer vision (ICCV2007) (pp. 1–8).

    Google Scholar 

  • Johnson, A. (1997). Spin-images: a representation for 3-d surface matching. Ph.D. thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA.

  • Kalogerakis, E., Nowrouzezahrai, D., Simari, P., & Singh, K. (2009). Extracting lines of curvature from noisy point clouds. Computer Aided Design, 41(4), 282–292.

    Article  Google Scholar 

  • Kamberov, G., Kamberova, G., Chum, O., Obdrzalek, S., Martinec, D., Kostkova, J., Pajdla, T., Matas, J., & Sara, R. (2006). 3D geometry from uncalibrated images. In Proc. 2nd int. symp. on visual computing.

    Google Scholar 

  • Kazhdan, M., Bolitho, M., & Hoppe, H. (2006). Poisson surface reconstruction. In A. Sheffer & K. Polthier (Eds.), Eurographics symposium on geometry processing (SGP2006) (pp. 61–70). Cagliari: Eurographics Association.

    Google Scholar 

  • Krishnan, S., Lee, P. Y., Moore, J. B., & Venkatasubramanian, S. (2005). Global registration of multiple 3D point sets via optimization-on-a-manifold. In Proc. of the 3rd eurographics symposium on geometry processing (SGP2005). Cagliari: Eurographics Association.

    Google Scholar 

  • Krishnan, S., Lee, P. Y., Moore, J. B., & Venkatasubramanian, S. (2007). Optimisation-on-a-manifold for global registration of multiple 3D point sets. International Journal of Intelligent Systems Technologies and Applications, 3(3/4), 319–340.

    Article  Google Scholar 

  • Lensch, H. P. A., Heidrich, W., & Seidel, H. P. (2000). Automated texture registration and stitching for real world models. In PG ’00: proceedings of the 8th pacific conference on computer graphics and applications (p. 317). Los Alamitos: IEEE Computer Society.

    Chapter  Google Scholar 

  • Li, X., & Guskov, I. (2005). Multi-scale features for approximate alignment of point-based surfaces. In Proc. of the 3rd eurographics symposium on geometry processing (SGP2005). Cagliari: Eurographics Association.

    Google Scholar 

  • Liu, L., & Stamos, I. (2005). Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In CVPR (Vol. 2, pp. 137–143). Los Alamitos: IEEE Computer Society.

    Google Scholar 

  • Liu, L., Stamos, I., Yu, G., Wolberg, G., & Zokai, S. (2006). Multiview geometry for texture mapping 2D images onto 3D range data. In Proc. of the 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06) (Vol. 2, pp. 2293–2300). Los Alamitos: IEEE Computer Society.

    Google Scholar 

  • Lowe, D. G. (1991). Fitting parameterized three-dimensional models to images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13, 441–450.

    Article  Google Scholar 

  • Maes, F., Collignon, A., Vandeermeulen, D., Marchal, G., & Suetens, P. (1997). Multimodality image registration by maximization of mutual information. IEEE Transactions on Medical Imaging, 16, 187–198.

    Article  Google Scholar 

  • Makadia, A., Patterson, A., & Daniilidis, K. (2006). Fully automatic registration of 3d point clouds. In IEEE computer society conference on computer vision and pattern recognition (Vol. 1, pp. 1297–1304).

    Google Scholar 

  • Malzbender, T., Gelb, D., & Wolters, H. (2001). Polynomial texture maps. In SIGGRAPH’01 (pp. 519–528). New York: ACM.

    Google Scholar 

  • Matsushita, K., & Kaneko, T. (1999). Efficient and handy texture mapping on 3D surfaces. Computer Graphics Forum, 18(3), 349–358.

    Article  Google Scholar 

  • Neugebauer, P. J., & Klein, K. (1999). Texturing 3D models of real world objects from multiple unregistered photographic views. Computer Graphics Forum, 18(3), 245–256.

    Article  Google Scholar 

  • Ni, K., Steedly, D., & Dellaert, F. (2007). Out-of-core bundle adjustment for large-scale 3D reconstruction. In Proc. int. conf. computer vision (pp. 1–8).

    Google Scholar 

  • Nistér, D. (2000). Reconstruction from uncalibrated sequences with a hierarchy of trifocal tensors. In Proc. Europ. conf. computer vision (ECCV2000) (pp. 649–663).

    Google Scholar 

  • Pintus, R., Gobbetti, E., & Combet, R. (2011). Fast and robust semi-automatic registration of photographs to 3D geometry. In The 12th international symposium on virtual reality, archaeology and cultural heritage. To appear.

    Google Scholar 

  • Pluim, J., Maintz, J., & Viergever, M. (2003). Mutual-information-based registration of medical images: a survey. IEEE Transactions on Medical Imaging, 22(8), 986–1004.

    Article  Google Scholar 

  • Pons, J. P., Keriven, R., & Faugeras, O. (2007). Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision, 72(2), 179–193.

    Article  Google Scholar 

  • Portelli, D., Ganovelli, F., Tarini, M., Cignoni, P., Dellepiane, M., & Scopigno, R. (2010). A framework for user-assisted sketch-based fitting of geometric primitives. In Proceedings of WSCG, the 18th int. conference on computer graphics, visualization and computer vision.

    Google Scholar 

  • Pottmann, H., Huang, Q. X., Yang, Y. L., & Hu, S. M. (2006). Geometry and convergence analysis of algorithms for registration of 3d shapes. International Journal of Computer Vision, 67(3), 277–296.

    Article  Google Scholar 

  • Powell, M. J. D. (2008). Developments of NEWUOA for minimization without derivatives. IMA Journal of Numerical Analysis, 28(4), 649–664.

    Article  MathSciNet  MATH  Google Scholar 

  • Pulli, K. (1999). Multiview registration for large data sets. In Proc. of the 2nd international conference on 3-D digital imaging and modeling (3DIM’99) (pp. 160–168). Washington, DC: IEEE Computer Society.

    Chapter  Google Scholar 

  • Pulli, K., Abi-Rached, H., Duchamp, T., Shapiro, L. G., & Stuetzle, W. (1998). Acquisition and visualization of colored 3D objects. In Proc. of the 14th international conference on pattern recognition (ICPR’98) (Vol. 1, p. 11). Los Alamitos: IEEE Computer Society.

    Google Scholar 

  • Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In Proc. of the third international conference on 3-D digital imaging and modeling (pp. 145–152).

    Chapter  Google Scholar 

  • Sequeira, V., & Goncalves, J. G. (2002). 3D reality modelling: Photo-realistic 3d models of real world scenes. In International symposium on 3D data processing visualization and transmission (p. 776).

    Chapter  Google Scholar 

  • Shum, H. Y., Ke, Q., & Zhang, Z. (1999). Efficient bundle adjustment with virtual key frames: a hierarchical approach to multi-frame structure from motion. In Proc. int. conf. computer vision and pattern rec.

    Google Scholar 

  • Skelly, L., & Sclaroff, S. (2007). Improved feature descriptors for 3-D surface matching. In Proc. SPIE conf. on two- and three-dimensional methods for inspection and metrology V (Vol. 6762).

    Google Scholar 

  • Snavely, N., Seitz, S. M., & Szeliski, R. (2006). Photo tourism: exploring photo collections in 3D. In SIGGRAPH’06 (pp. 835–846).

    Google Scholar 

  • Sottile, M., Dellepiane, M., Cignoni, P., & Scopigno, R. (2010). Mutual correspondences: an hybrid method for image-to-geometry registration. In Eurographics Italian chapter conference 2010 (pp. 81–88). EG

    Google Scholar 

  • Stamos, I., Liu, L., Chen, C., Wolberg, G., Yu, G., & Zokai, S. (2008). Integrating automated range registration with multiview geometry for the photorealistic modeling of large-scale scenes. International Journal of Computer Vision, 78, 237–260.

    Article  Google Scholar 

  • Steedly, D., Essa, I., & Dellaert, F. (2003). Spectral partitioning for structure from motion. In Proc. int. conf. computer vision (ICCV2003) (pp. 649–663).

    Google Scholar 

  • Strecha, C., von Hansen, W., Van Gool, L., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In CVPR’08 (pp. 1–8).

    Google Scholar 

  • Vergauwen, M., & Gool, L. V. (2006). Web-based 3D reconstruction service. Machine Vision and Applications, 17(6), 411–426.

    Article  Google Scholar 

  • Viola, P., William, M., & Wells, I. (1997). Alignment by maximization of mutual information. International Journal of Computer Vision, 24(2), 137–154.

    Article  Google Scholar 

  • Wu, C., Clipp, B., Li, X., Frahm, J. M., & Pollefeys, M. (2008). 3d model matching with viewpoint-invariant patches (vip). In CVPR’08. Los Alamitos: IEEE Computer Society.

    Google Scholar 

  • Zhao, W., Nister, D., & Hsu, S. (2005). Alignment of continuous video onto 3d point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1305–1318.

    Article  Google Scholar 

  • Zheng, H., Cleju, I., & Saupe, D. (2009). Highly-automatic mi based multiple 2D/3D image registration using self-initialized geodesic feature correspondences. In H. Zha, R. I. Taniguchi, & S. J. Maybank (Eds.), Lecture notes in computer science: Vol. 3. ACCV (Vol. 5996, pp. 426–435). Berlin: Springer.

    Google Scholar 

Download references

Acknowledgements

This research work was partly funded by the EU Community FP7 ICT under the V-MUST.net project (Grant Agreement 270404). We would also like to thank the anonymous reviewers for their feedback which enabled us to improve the final version of the paper.

Author information

Authors and Affiliations

  1. Visual Computing Lab (ISTI-CNR), Pisa, Italy

    M. Corsini, M. Dellepiane, F. Ganovelli & R. Scopigno

  2. Toshiba Cambridge Research Laboratory, Cambridge, UK

    R. Gherardi

  3. University of Verona, Verona, Italy

    A. Fusiello

Authors
  1. M. Corsini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Dellepiane
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. F. Ganovelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Gherardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. A. Fusiello
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. R. Scopigno
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Corsini.

Appendix

Appendix

A coplanar quadruple (a,b,c,d) is expressed as the combination of the two segments s 1=(a,b) and s 2=(c,d) and characterized by their length d 1=∥a−b∥ and d 2=∥c−d∥ (see Fig. 16). Since the segments are coplanar, they will meet at an intermediate point e, so the quadruple is further characterized by the points along the two segments where they meet, i.e. ratios:

(10)
(11)
Fig. 16
figure 17

Quadruple characterization in 4PCS algorithm

Full size image

Aiger et al. (2008) use this simple characterization of a quadruple to prune the number of possibly congruent quadruples on Q and hence to speed up the process. Consider a couple in Q as the segment (a′,b′) of a candidate congruent quadruple (a′,b′,c′,d′). If this quadruple is congruent to (a,b,c,d) it means that if we generate the intermediate points on (a′,b′) and (c′,d′) with the ratios r 1 and r 2 they will coincide, because affine transformations preserve ratio of the distances. Their approach consists in generating the intermediate points for all the segments in Q and inserting them into a range-query data structure (Arya et al. 1998). This can be built in O(klogk) time and accessed in O(logk) (where k is the number of the intermediate points). Then, they only test the quadruples made of two segments which intermediate points coincide. If we consider all the couples in Q as potential segments of a congruent quadruple k is O(n 2), but if we restrict the choice to a couple of points at a distance d 1 or d 2 from each other and assume a uniform distribution of points k will be O(n). Therefore they have O(n 2) for finding O(n) segments/intermediate points plus O(n logn) for building and accessing the search data structure, and hence the global complexity is O(n 2).

In principle, we could almost apply it to our case without any change, simply by including scaling in computing the transformation between candidate congruent quadruples. The problem with this is that we cannot limit the list of couples in Q to those within a distance d 1 or d 2 because there is a unknown scale factor between Q and P. Therefore the order of magnitude of k is O(n 2) and the global complexity becomes O(n 2logn). Although the total asymptotic complexity raised “only” by a factor logn the actual time for computing the result becomes very large. This is because the number of quadruples to be tested for affinity with the base in P raised by O(n) to O(n 2) and the cost of evaluating the transformation between two quadruples is high.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Corsini, M., Dellepiane, M., Ganovelli, F. et al. Fully Automatic Registration of Image Sets on Approximate Geometry. Int J Comput Vis 102, 91–111 (2013). https://doi.org/10.1007/s11263-012-0552-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-012-0552-5

Keywords

Search

Navigation