Jakub Fišer
Paul Asente
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We introduce a novel approach to example-based stylization of portrait videos that preserves both the subject's identity and the visual richness of the input style exemplar. Unlike the current state-of-the-art based on neural style transfer [Selim et al. 2016], our method performs non-parametric texture synthesis that retains more of the local textural details of the artistic exemplar and does not suffer from image warping artifacts caused by aligning the style exemplar with the target face. Our method allows the creation of videos with less than full temporal coherence [Ruder et al. 2016]. By introducing a controllable amount of temporal dynamics, it more closely approximates the appearance of real hand-painted animation in which every frame was created independently. We demonstrate the practical utility of the proposed solution on a variety of style exemplars and target videos.
Supplemental Material
Available for Download
zip
a155-fiser.zip (91.2 MB)
Supplemental files.
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.Google Scholar
Digital Library
- Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation By Example. ACM Transactions on Graphics 32, 4 (2013), 119.Google ScholarDigital Library
- Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth J. Carter, and Jessica K. Hodgins. 2013. Style and abstraction in portrait sketching. ACM Transactions on Graphics 32, 4 (2013), 55.Google ScholarDigital Library
- Hong Chen, Lin Liang, Ying-Qing Xu, Heung-Yeung Shum, and Nan-Ning Zheng. 2002. Example-Based Automatic Portraiture. In Proceedings of Asian Conference on Computer Vision. 171--178.Google Scholar
- Hong Chen, Ziqiang Liu, Chuck Rose, Yingqing Xu, Heung-Yeung Shum, and David Salesin. 2004. Example-Based Composite Sketching of Human Portraits. In Proceedings of International Symposium on Non-photorealistic Animation and Rendering. 95--102. Google ScholarDigital Library
- Hong Chen, Nanning Zheng, Lin Liang, Yan Li, Ying-Qing Xu, and Heung-Yeung Shum. 2002. PicToon: A Personalized Image-Based Cartoon System. In Proceedings of ACM International Conference on Multimedia. 171--178. Google ScholarDigital Library
- Doug DeCarlo and Anthony Santella. 2002. Stylization and Abstraction of Photographs. ACM Transactions on Graphics 21, 3 (2002), 769--776. Google ScholarDigital Library
- Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of Complex Image Appearance from Limited Exemplars. ACM Transactions on Graphics 34, 2 (2015), 22.Google ScholarDigital Library
- Steve DiPaola. 2007. Painterly rendered portraits from photographs using a knowledge-based approach. In Proceedings of SPIE Human Vision and Electronic Imaging, Vol. 6492. 33--43. Google ScholarCross Ref
- Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH Conference Proceedings. 341--346. Google ScholarDigital Library
- Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.Google ScholarDigital Library
- Jakub Fišer, Michal Lukáč, Ondřej Jamriška, Martin Čadík, Yotam Gingold, Paul Asente, and Daniel Sýkora. 2014. Color Me Noisy: Example-based Rendering of Hand-colored Animations with Temporal Noise Control. Computer Graphics Forum 33, 4 (2014), 1--10. Google ScholarDigital Library
- Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2414--2423. Google ScholarCross Ref
- Yihong Gong and Masao Sakauchi. 1995. Detection of Regions Matching Specified Chromatic Features. Computer Vision and Image Understanding 61, 2 (1995), 263--269. Google ScholarDigital Library
- Bruce Gooch, Erik Reinhard, and Amy Gooch. 2004. Human Facial Illustrations: Creation and Psychophysical Evaluation. ACM Transactions on Graphics 23, 1 (2004), 27--44. Google ScholarDigital Library
- James Hays and Irfan A. Essa. 2004. Image and Video Based Painterly Animation. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 113--120. Google ScholarDigital Library
- Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH Conference Proceedings. 327--340. Google ScholarDigital Library
- Aaron Jackson, Michel Valstar, and Georgios Tzimiropoulos. 2016. A CNN Cascade for Landmark Guided Semantic Part Segmentation. In Proceedings of ECCV 2016 Workshops, Geometry meets Deep Learning.Google ScholarCross Ref
- Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM Transactions on Graphics 34, 4 (2015), 92.Google ScholarDigital Library
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proceedings of European Conference on Computer Vision. 694--711. Google ScholarCross Ref
- Scott F. Johnston. 2002. Lumo: Illumination for Cel Animation. In Proceedings of the 2Nd International Symposium on Non-photorealistic Animation and Rendering. 45-ff.Google ScholarDigital Library
- Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2 (2015), 349--360. Google ScholarDigital Library
- Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1867--1874. Google ScholarDigital Library
- Ira Kemelmacher-Shlizerman. 2016. Transfiguring Portraits. ACM Transactions on Graphics 35, 4 (2016), 94.Google ScholarDigital Library
- Vivek Kwatra, Irfan A. Essa, Aaron F. Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3 (2005), 795--802. Google ScholarDigital Library
- Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. 2013. State of the "Art": A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE Transactions on Visualization and Computer Graphics 19, 5 (2013), 866--885. Google ScholarDigital Library
- Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 228--242. Google ScholarDigital Library
- Hongliang Li, Guanghui Liu, and King Ngi Ngan. 2011. Guided Face Cartoon Synthesis. IEEE Transactions on Multimedia 13, 6 (2011), 1230--1239. Google ScholarDigital Library
- Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence across Scenes and Its Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5 (2011), 978--994. Google ScholarDigital Library
- Sifei Liu, Jimei Yang, Chang Huang, and Ming-Hsuan Yang. 2015. Multi-Objective Convolutional Learning for Face Labeling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Michal Lukáč, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. Brushables: Example-based Edge-aware Directional Texture Painting. Computer Graphics Forum 34, 7 (2015), 257--268. Google ScholarDigital Library
- Michal Lukáč, Jakub Fišer, Jean-Charles Bazin, Ondřej Jamriška, Alexander Sorkine-Hornung, and Daniel Sýkora. 2013. Painting by Feature: Texture Boundaries for Example-based Image Creation. ACM Transaction on Graphics 32, 4 (2013), 116.Google ScholarDigital Library
- Meng Meng, Mingtian Zhao, and Song Chun Zhu. 2010. Artistic paper-cut of human portraits. In Proceedings of ACM Multimedia. 931--934. Google ScholarDigital Library
- Gioacchino Noris, Daniel Sýkora, Stelian Coros, Brian Whited, Maryann Simmons, Alexander Hornung, Marcus Gross, and Robert Sumner. 2011. Temporal Noise Control for Sketchy Animation. In Proceedings of International Symposium on Non-photorealistic Animation and Rendering. 93--98. Google ScholarDigital Library
- Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, JoËlle Thollot, and David Salesin. 2008. Diffusion Curves: A Vector Representation for Smooth-Shaded Images. ACM Transactions on Graphics 27, 3 (2008), 92.Google ScholarDigital Library
- Javier Portilla and Eero P. Simoncelli. 2000. A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients. International Journal of Computer Vision 40, 1 (2000), 49--70. Google ScholarDigital Library
- Konstantinos Rematas, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars. 2014. Image-Based Synthesis and Re-synthesis of Viewpoints Guided by 3D Models.. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 3898--3905. Google ScholarDigital Library
- Lincoln Ritter, Wilmot Li, Brian Curless, Maneesh Agrawala, and David Salesin. 2006. Painting With Texture. In Proceedings of Eurographics Symposium on Rendering. 371--376.Google Scholar
- Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2016. Artistic Style Transfer for Videos. In Proceedings of German Conference Pattern Recognition. 26--36. Google ScholarCross Ref
- Scott Schaefer, Travis McPhail, and Joe Warren. 2006. Image Deformation Using Moving Least Squares. ACM Transactions on Graphics 25, 3 (2006), 533--540. Google ScholarDigital Library
- Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting Style Transfer for Head Portraits Using Convolutional Neural Networks. ACM Transactions on Graphics 35, 4 (2016), 129.Google ScholarDigital Library
- Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian L. Price, Eli Shechtman, and Ian Sachs. 2016. Automatic Portrait Segmentation for Image Stylization. Computer Graphics Forum 35, 2 (2016), 93--102. Google ScholarDigital Library
- Yi-Chang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style Transfer for Headshot Portraits. ACM Transactions on Graphics 33, 4 (2014), 148.Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).Google Scholar
- Pawan Sinha, Benjamin Balas, Yuri Ostrovsky, and Richard Russell. 2006. Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About. Proc. IEEE 94, 11 (2006), 1948--1962. Google ScholarCross Ref
- Patrick Tresset and Frédéric F. Leymarie. 2005. Generative Portrait Sketching. In Proceedings of International Conference on Virtual Systems and Multimedia. 739--748.Google Scholar
- Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2013b. Transductive Face Sketch-Photo Synthesis. IEEE Transactions on Neural Networks and Learning Systems 24, 9 (2013), 1364--1376. Google ScholarCross Ref
- Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2014. A Comprehensive Survey to Face Hallucination. International Journal of Computer Vision 106, 1 (2014), 9--30. Google ScholarDigital Library
- Tinghuai Wang, John P. Collomosse, Andrew Hunter, and Darryl Greig. 2013a. Learnable Stroke Models for Example-based Portrait Painting. In Proceedings of British Machine Vision Conference. Google ScholarCross Ref
- Xiaogang Wang and Xiaoou Tang. 2009. Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 11 (2009), 1955--1967. Google ScholarDigital Library
- Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463--476. Google ScholarDigital Library
- Holger Winnemöller, Sven C. Olsen, and Bruce Gooch. 2006. Real-time video abstraction. ACM Transactions on Graphics 25, 3 (2006), 1221--1226. Google ScholarDigital Library
- Ming Yang, Shu Lin, Ping Luo, Liang Lin, and Hongyang Chao. 2010. Semantics-driven portrait cartoon stylization. In Proceedings of International Conference on Image Processing. 1805--1808. Google ScholarCross Ref
- Yue Yang, Hanli Zhao, Lihua You, Renlong Tu, Xueyi Wu, and Xiaogang Jin. 2015. Semantic portrait color transfer with internet images. Multimedia Tools and Applications (2015), 1--19.Google Scholar
- Kun Zeng, Mingtian Zhao, Caiming Xiong, and Song-Chun Zhu. 2009. From image parsing to painterly rendering. ACM Transactions on Graphics 29, 1 (2009), 2.Google ScholarDigital Library
- Yong Zhang, Weiming Dong, Oliver Deussen, Feiyue Huang, Ke Li, and Bao-Gang Hu. 2014. Data-driven Face Cartoon Stylization. In SIGGRAPH Asia Technical Briefs. 14. Google ScholarDigital Library
- Mingtian Zhao and Song-Chun Zhu. 2011. Portrait Painting Using Active Templates. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 117--124. Google ScholarDigital Library
- Hao Zhou, Zhanghui Kuang, and Kwan-Yee Kenneth Wong. 2012. Markov Weight Fields for face sketch synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1091--1097.Google Scholar
- Yang Zhou, Huajie Shi, Dani Lischinski, Minglun Gong, Johannes Kopf, and Hui Huang. 2017. Analysis and Controlled Synthesis of Inhomogeneous Textures. Computer Graphics Forum 36, 2 (2017). Google ScholarDigital Library
Index Terms
- Example-based synthesis of stylized facial animations
Recommendations
Example-based facial rigging
SIGGRAPH '10: ACM SIGGRAPH 2010 papersWe introduce a method for generating facial blendshape rigs from a set of example poses of a CG character. Our system transfers controller semantics and expression dynamics from a generic template to the target blendshape model, while solving for an ...
Fast and reliable example-based mesh IK for stylized deformations
Example-based shape deformation allows a mesh to be easily manipulated or animated with simple inputs. As the user pulls parts of the shape, the rest of the mesh automatically changes in an intuitive way by drawing from a set of exemplars. This provides ...
Stylized lighting for cartoon shader
CASA' 2009 Special IssueIn the context of non-photorealistic imaging, such as digital cel animation, lighting is symbolic and stylized to depict the scene's mood and the geometric or physical features of the objects in the scene. Stylized light and shade should therefore be ...
Comments