Yağiz Aksoy
Tunç Ozan Aydin
Marc Pollefeys
Aljoša Smolić
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Due to the widespread use of compositing in contemporary feature films, green-screen keying has become an essential part of postproduction workflows. To comply with the ever-increasing quality requirements of the industry, specialized compositing artists spend countless hours using multiple commercial software tools, while eventually having to resort to manual painting because of the many shortcomings of these tools. Due to the sheer amount of manual labor involved in the process, new green-screen keying approaches that produce better keying results with less user interaction are welcome additions to the compositing artist’s arsenal. We found that—contrary to the common belief in the research community—production-quality green-screen keying is still an unresolved problem with its unique challenges. In this article, we propose a novel green-screen keying method utilizing a new energy minimization-based color unmixing algorithm. We present comprehensive comparisons with commercial software packages and relevant methods in literature, which show that the quality of our results is superior to any other currently available green-screen keying solution. It is important to note that, using the proposed method, these high-quality results can be generated using only one-tenth of the manual editing time that a professional compositing artist requires to process the same content having all previous state-of-the-art tools at one’s disposal.
Supplemental Material
Available for Download
zip
aksoy.zip (238.8 MB)
Supplemental movie, appendix, image and software files for, Interactive High-Quality Green-Screen Keying via Color Unmixing
- R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 11, 2274--2282. Google Scholar
Digital Library
- Xue Bai, Jue Wang, and David Simons. 2011. Towards temporally-coherent video matting. In Proceedings of MIRAGE. Google ScholarDigital Library
- Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. 2009. Video SnapCut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics 28, 3, 70:1--70:11. Google ScholarDigital Library
- Dimitri P. Bertsekas. 1982. The method of multipliers for equality constrained problems. In Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York, NY, 96--157.Google Scholar
- Y. Boykov and V. Kolmogorov. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 9, 1124--1137. Google ScholarDigital Library
- Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11, 1222--1239. Google ScholarDigital Library
- Robert Carroll, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Transactions on Graphics 30, 4, 43:1--43:10. Google ScholarDigital Library
- Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Transactions on Graphics 34, 4, 139:1--139:11. Google ScholarDigital Library
- Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013a. KNN matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9, 2175--2188. Google ScholarDigital Library
- Xiaowu Chen, Dongqing Zou, S. Z. Zhou, Qinping Zhao, and Ping Tan. 2013b. Image matting with local and nonlocal smooth priors. In Proceedings of CVPR. Google ScholarDigital Library
- Inchang Choi, Minhaeng Lee, and Yu-Wing Tai. 2012. Video matting using multi-frame nonlocal matting Laplacian. In Proceedings of ECCV. Google ScholarDigital Library
- Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. 2002. Video matting of complex scenes. Transactions on Graphics 243--248. Google ScholarDigital Library
- Yung-Yu Chuang, Brian Curless, David H. Salesin, and Richard Szeliski. 2001. A Bayesian approach to digital matting. In Proceedings of CVPR.Google Scholar
- Jialue Fan, Xiaohui Shen, and Ying Wu. 2012. Scribble tracker: A matting-based approach for robust tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 8, 1633--1644. Google ScholarDigital Library
- Eduardo S. L. Gastal and Manuel M. Oliveira. 2010. Shared sampling for real-time alpha matting. Computer Graphics Forum 29, 2, 575--584.Google ScholarCross Ref
- Anselm Grundhöfer and Oliver Bimber. 2008. VirtualStudio2Go: Digital video composition for real environments. ACM Transactions on Graphics 27, 5, 151:1--151:8. Google ScholarDigital Library
- Anselm Grundhöfer, Daniel Kurz, Sebastian Thiele, and Oliver Bimber. 2010. Color invariant chroma keying and color spill neutralization for dynamic scenes and cameras. The Visual Computer 26, 9, 1167--1176. Google ScholarDigital Library
- Bei He, Guijin Wang, Chenbo Shi, Xuanwu Yin, Bo Liu, and Xinggang Lin. 2013. Iterative transductive learning for alpha matting. In Proceedings of ICIP.Google ScholarCross Ref
- Kaiming He, C. Rhemann, C. Rother, Xiaoou Tang, and Jian Sun. 2011. A global sampling method for alpha matting. In Proceedings of CVPR. Google ScholarDigital Library
- Meiguang Jin, Byoung-Kwang Kim, and Woo-Jin Song. 2014. Adaptive propagation-based color-sampling for alpha matting. IEEE Transactions on Circuits and Systems for Video Technology 24, 7, 1101--1110.Google ScholarCross Ref
- Jubin Johnson, Deepu Rajan, and Hisham Cholakkal. 2014. Sparse codes as alpha matte. In Proceedings of BMVC.Google ScholarCross Ref
- V. Kolmogorov and R. Zabih. 2004. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 2, 147--159. Google ScholarDigital Library
- P. Lee and Ying Wu. 2011. Nonlocal matting. In Proceedings of CVPR. Google ScholarDigital Library
- Anat Levin, Dani Lischinski, and Yair Weiss. 2008a. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 228--242. Google ScholarDigital Library
- Anat Levin, Alex Rav-Acha, and Dani Lischinski. 2008b. Spectral matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 10, 1699--1712. Google ScholarDigital Library
- Dingzeyu Li, Qifeng Chen, and Chi-Keung Tang. 2013. Motion-aware KNN Laplacian for video matting. In Proceedings of ICCV. Google ScholarDigital Library
- Yin Li, Jian Sun, and Heung-Yeung Shum. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595--600. Google ScholarDigital Library
- Hai Ting Lin, Yu-Wing Tai, and M. S. Brown. 2011. Motion regularization for matting motion blurred objects. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 11, 2329--2336. Google ScholarDigital Library
- Christoph Rhemann, Carsten Rother, and Margrit Gelautz. 2008. Improving color modeling for alpha matting. In Proceedings of BMVC.Google ScholarCross Ref
- Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A perceptually motivated online benchmark for image matting. In Proceedings of CVPR.Google ScholarCross Ref
- Mark A. Ruzon and Carlo Tomasi. 2000. Alpha estimation in natural images. In Proceedings of CVPR.Google Scholar
- E. Shahrian, B. Price, S. Cohen, and D. Rajan. 2014. Temporally coherent and spatially accurate video matting. Computer Graphics Forum 33, 2, 381--390. Google ScholarDigital Library
- E. Shahrian and D. Rajan. 2012. Weighted color and texture sample selection for image matting. In Proceedings of CVPR. Google ScholarDigital Library
- E. Shahrian, D. Rajan, B. Price, and S. Cohen. 2013. Improving image matting using comprehensive sampling sets. In Proceedings of CVPR. Google ScholarDigital Library
- Yongfang Shi, O. C. Au, Jiahao Pang, K. Tang, Wenxiu Sun, Hong Zhang, Wenjing Zhu, and Luheng Jia. 2013. Color clustering matting. In Proceedings of ICME.Google Scholar
- YiChang Shih, Dilip Krishnan, Fredo Durand, and William T. Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of CVPR.Google Scholar
- D. Singaraju, C. Rother, and C. Rhemann. 2009. New appearance models for natural image matting. In Proceedings of CVPR.Google Scholar
- D. Singaraju and R. Vidal. 2011. Estimation of alpha mattes for multiple image layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 7, 1295--1309. Google ScholarDigital Library
- Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics 31, 4, 100:1--100:10. Google ScholarDigital Library
- Alvy Ray Smith and James F. Blinn. 1996. Blue screen matting. ACM Transactions on Graphics (1996), 259--268. Google ScholarDigital Library
- Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315--321. Google ScholarDigital Library
- Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proceedings of CVPR. Google ScholarDigital Library
- Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2007. Soft color segmentation and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9, 1520--1537. Google ScholarDigital Library
- Zhen Tang, Zhenjiang Miao, Yanli Wan, and Dianyong Zhang. 2012. Video matting via opacity propagation. The Visual Computer 28, 1, 47--61. Google ScholarDigital Library
- Jue Wang, Pravin Bhat, R. Alex Colburn, Maneesh Agrawala, and Michael F. Cohen. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585--594. Google ScholarDigital Library
- Jue Wang and M. F. Cohen. 2005. An iterative optimization approach for unified image segmentation and matting. In Proceedings of ICCV. Google ScholarDigital Library
- Jue Wang and M. F. Cohen. 2007. Optimized color sampling for robust matting. In Proceedings of CVPR.Google Scholar
- Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2. Google ScholarDigital Library
- Sai-Kit Yeung, Tai-Pang Wu, and Chi-Keung Tang. 2008. Extracting smooth and transparent layers from a single image. In Proceedings of CVPR.Google Scholar
- Fan Zhong, Xueying Qin, Qunsheng Peng, and Xiangxu Meng. 2012. Discontinuity-aware video object cutout. ACM Transactions on Graphics 31, 6, 175:1--175:10. Google ScholarDigital Library
Index Terms
- Interactive High-Quality Green-Screen Keying via Color Unmixing
Recommendations
Unmixing-Based Soft Color Segmentation for Image Manipulation
We present a new method for decomposing an image into a set of soft color segments that are analogous to color layers with alpha channels that have been commonly utilized in modern image manipulation software. We show that the resulting decomposition ...
Interactive High-Quality Green-Screen Keying via Color Unmixing
Due to the widespread use of compositing in contemporary feature films, green-screen keying has become an essential part of postproduction workflows. To comply with the ever-increasing quality requirements of the industry, specialized compositing ...
Improved chromakey of hair strands via orientation filter convolution
SIGGRAPH '17: ACM SIGGRAPH 2017 PostersWe present a technique for improving the alpha matting of challenging green-screen video sequences involving hair strands. As hair strands are thin and can be semi-translucent, they are especially hard to separate from a background. However, they appear ...
Comments