Abstract
Gradient-domain compositing is an essential tool in computer vision and its applications, e.g., seamless cloning, panorama stitching, shadow removal, scene completion and reshuffling. While easy to implement, these gradient-domain techniques often generate bleeding artifacts where the composited image regions do not match. One option is to modify the region boundary to minimize such mismatches. However, this option may not always be sufficient or applicable, e.g., the user or algorithm may not allow the selection to be altered. We propose a new approach to gradient-domain compositing that is robust to inaccuracies and prevents color bleeding without changing the boundary location. Our approach improves standard gradient-domain compositing in two ways. First, we define the boundary gradients such that the produced gradient field is nearly integrable. Second, we control the integration process to concentrate residuals where they are less conspicuous. We show that our approach can be formulated as a standard least-squares problem that can be solved with a sparse linear system akin to the classical Poisson equation. We demonstrate results on a variety of scenes. The visual quality and run-time complexity compares favorably to other approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that for a 2D vector field u=(u x ,u y ), the curl is a scalar value that corresponds to the z component of the 3D curl applied to the 3D vector field (u x ,u y ,0).
References
Agarwala, A. (2007). Efficient gradient-domain compositing using quadtrees. In ACM transactions on graphics: Vol. 26. Proceedings of the ACM SIGGRAPH conference.
Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D. H., & Cohen, M. F. (2004). Interactive digital photomontage. In ACM transactions on graphics: Vol. 23. Proceedings of the ACM SIGGRAPH conference (pp. 294–302).
Agrawal, A., Raskar, R., & Chellappa, R. (2006). What is the range of surface reconstructions from a gradient field? In Proceedings of the European conference on computer vision.
Aubert, G., & Kornprobst, P. (2002). Applied mathematical sciences: Vol. 147. Mathematical problems in image processing: partial differential equations and the calculus of variations. Berlin: Springer.
Bae, S., Paris, S., & Durand, F. (2006). Two-scale tone management for photographic look. In ACM transactions on graphics: Vol. 25. Proceedings of the ACM SIGGRAPH conference (pp. 637–645).
Bhat, P., Zitnick, C. L., Cohen, M., & Curless, B. (2009). Gradientshop: a gradient-domain optimization framework for image and video filtering. In ACM transactions on graphics.
Cho, T. S., Avidan, S., & Freeman, W. T. (2010). The patch transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1489–1501.
Drettakis, G., Bonneel, N., Dachsbacher, C., Lefebvre, S., Schwarz, M., & Viaud-Delmon, I. (2007). An interactive perceptual rendering pipeline using contrast and spatial masking. Rendering Techniques.
Farbman, Z., Fattal, R., Lischinski, D., & Szeliski, R. (2008). Edge-preserving decompositions for multi-scale tone and detail manipulation. In ACM transactions on graphics: Vol. 27. Proceedings of the ACM SIGGRAPH conference.
Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Fattal, R., & Lischinski, D. (2009). Coordinates for instant image cloning. In ACM transactions on graphics: Vol. 28. Proceedings of the ACM SIGGRAPH conference.
Finlayson, G. D., Hordley, S. D., Lu, C., & Drew, M. S. (2006). On the removal of shadows from images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 59–68.
Finlayson, G. D., Drew, M. S., & Lu, C. (2009). Entropy minimization for shadow removal. International Journal of Computer Vision, 85(1), 35–57.
Georgiev, T. (2006). Covariant derivatives and vision. In Proceedings of the European conference on computer vision.
Hays, J., & Efros, A. A. (2007). Scene completion using millions of photographs. In ACM transactions on graphics: Vol. 26. Proceedings of the ACM SIGGRAPH conference.
Jia, J., Sun, J., Tang, C. K., & Shum, H. Y. (2006). Drag-and-drop pasting. In ACM transactions on graphics: Vol. 25. Proceedings of the ACM SIGGRAPH conference.
Lalonde, J. F., Hoiem, D., Efros, A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. In ACM transactions on graphics: Vol. 26. Proceedings of the ACM SIGGRAPH conference.
Levin, A., Zomet, A., Peleg, S., & Weiss, Y. (2006). Seamless image stitching in the gradient domain. In Proceedings of the European conference on computer vision.
Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 629–639.
Prez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. In ACM transactions on graphics: Vol. 22. Proceedings of the ACM SIGGRAPH conference.
Ramanarayanan, G., Ferwerda, J., Walter, B., & Bala, K. (2007). Visual equivalence: towards a new standard for image fidelity. In ACM transactions on graphics: Vol. 26. Proceedings of the ACM SIGGRAPH conference.
Ramanarayanan, G., Bala, K., & Ferwerda, J. (2008). Perception of complex aggregates. In ACM transactions on graphics: Vol. 27. Proceedings of the ACM SIGGRAPH conference.
Reddy, D., Agrawal, A., & Chellappa, R. (2009). Enforcing integrability by error correction using L-1 minimization. In Proceedings of the conference on computer vision and pattern recognition.
Sivic, J., Kaneva, B., Torralba, A., Avidan, S., & Freeman, W. T. (2008). Creating and exploring a large photorealistic virtual space. In Proceedings of the IEEE workshop on internet vision.
Su, S., Durand, F., & Agrawala, M. (2005). De-emphasis of distracting image regions using texture power maps. In Proceedings of the ICCV workshop on texture analysis and synthesis.
Tappen, M. F., Adelson, E. H., & Freeman, W. T. (2005). Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 1459–1472.
Vangorp, P., Laurijssen, J., & Dutr, P. (2007). The influence of shape on the perception of material reflectance. In ACM transactions on graphics: Vol. 26. Proceedings of the ACM SIGGRAPH conference.
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(14), 600–612.
Whyte, O., Sivic, J., & Zisserman, A. (2009). Get out of my picture! Internet-based inpainting. In Proceedings of the British machine vision conference.
Acknowledgements
The authors thank Todor Georgiev for the link with the Poisson equation, Kavita Bala and George Drettakis for their discussion about visual masking, Aseem Agarwala and Bill Freeman for their help with the paper, Tim Cho and Biliana Kaneva for helping with the validation, Medhat H. Ibrahim for the image of the Egyption pyramids, Adobe Systems, Inc. for supporting Micah K. Johnson’s research, and Ravi Ramamoorthi for supporting Michael Tao’s work. This material is based upon work supported by the National Science Foundation under Grant Nos. 0739255 and 0924968.
Author information
Authors and Affiliations
Corresponding author
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Tao, M.W., Johnson, M.K. & Paris, S. Error-Tolerant Image Compositing. Int J Comput Vis 103, 178–189 (2013). https://doi.org/10.1007/s11263-012-0579-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-012-0579-7