Neel Joshi
Wojciech Matusik
Abstract
We present an algorithm and a system for high-quality natural video matting using a camera array. The system uses high frequencies present in natural scenes to compute mattes by creating a synthetic aperture image that is focused on the foreground object, which reduces the variance of pixels reprojected from the foreground while increasing the variance of pixels reprojected from the background. We modify the standard matting equation to work directly with variance measurements and show how these statistics can be used to construct a trimap that is later upgraded to an alpha matte. The entire process is completely automatic, including an automatic method for focusing the synthetic aperture image on the foreground object and an automatic method to compute the trimap and the alpha matte. The proposed algorithm is very efficient and has a per-pixel running time that is linear in the number of cameras. Our current system runs at several frames per second, and we believe that it is the first system capable of computing high-quality alpha mattes at near real-time rates without the use of active illumination or special backgrounds.
Supplemental Material
- Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 2001, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 425--432. Google Scholar
Digital Library
- Chuang, Y.-Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D. H., and Szeliski, R. 2000. Environment matting extensions: towards higher accuracy and real-time capture. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 121--130. Google ScholarDigital Library
- Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2001), IEEE Computer Society, vol. 2, 264--271.Google Scholar
- Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243--248. Google ScholarDigital Library
- Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 297--306. Google ScholarDigital Library
- Kolomogrov, V., Criminisi, A., Blake, A., Cross, G., and Rother, C. 2005. Bi-layer segmentation of binocular stereo video. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, 407--414. Google ScholarDigital Library
- Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303--308. Google ScholarDigital Library
- Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595--600. Google ScholarDigital Library
- McGuire, M., Matusik, W., Pfister, H., Durand, F., and Hughes, J. 2005. Defocus Video Matting. ACM Transactions on Graphics 24, 3, 567--576. Google ScholarDigital Library
- Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309--314. Google ScholarDigital Library
- Ruzon, M. A., and Tomasi, C. 2000. Alpha estimation in natural images. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2000), vol. 1, 18--25.Google Scholar
- Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH 1996. ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 259--268. Google ScholarDigital Library
- Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3 (August), 315--321. Google ScholarDigital Library
- Vlahos, P., 1958. Composite photography utilizing sodium vapor illumination (u.s. patent 3,095,304), May.Google Scholar
- Vlahos, P., 1971. Electronic composite photography (u.s. patent 3,595,987, July.Google Scholar
- Vlahos, P., 1978. Comprehensive electronic compositing system (u.s. patent 4,100,569), July.Google Scholar
- Wang, J., Bhat, P., Colburn, A., Agrawala, M., and Cohen, M. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585--594. Google ScholarDigital Library
- Wexler, Y., Fitzgibbon, A., and Zisserman., A. 2002. Bayesian estimation of layers from multiple images. In Proceedings of 7th European Conference on Computer Vision (ECCV), vol. III, 487--501. Google ScholarDigital Library
- Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3, 765--776. Google ScholarDigital Library
- Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 23, 3, 600--608. Google ScholarDigital Library
- Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In Proceedings of ACM SIGGRAPH 1999, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 205--214. Google ScholarDigital Library
Index Terms
- Natural video matting using camera arrays
Recommendations
Video matting of complex scenes
SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniquesThis paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background ...
Natural video matting using camera arrays
SIGGRAPH '06: ACM SIGGRAPH 2006 PapersWe present an algorithm and a system for high-quality natural video matting using a camera array. The system uses high frequencies present in natural scenes to compute mattes by creating a synthetic aperture image that is focused on the foreground ...
Exploring Defocus Matting: Nonparametric Acceleration, Super-Resolution, and Off-Center Matting
Defocus matting is a fully automatic and passive method for pulling mattes from video captured with coaxial cameras that have different depths of field and planes of focus. Nonparametric sampling can accelerate the video-matting process from minutes to ...
Comments