Hugo Bertiche
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present a methodology to automatically obtain Pose Space Deformation (PSD) basis for rigged garments through deep learning. Classical approaches rely on Physically Based Simulations (PBS) to animate clothes. These are general solutions that, given a sufficiently fine-grained discretization of space and time, can achieve highly realistic results. However, they are computationally expensive and any scene modification prompts the need of re-simulation. Linear Blend Skinning (LBS) with PSD offers a lightweight alternative to PBS, though, it needs huge volumes of data to learn proper PSD. We propose using deep learning, formulated as an implicit PBS, to un-supervisedly learn realistic cloth Pose Space Deformations in a constrained scenario: dressed humans. Furthermore, we show it is possible to train these models in an amount of time comparable to a PBS of a few sequences. To the best of our knowledge, we are the first to propose a neural simulator for cloth. While deep-based approaches in the domain are becoming a trend, these are data-hungry models. Moreover, authors often propose complex formulations to better learn wrinkles from PBS data. Supervised learning leads to physically inconsistent predictions that require collision solving to be used. Also, dependency on PBS data limits the scalability of these solutions, while their formulation hinders its applicability and compatibility. By proposing an unsupervised methodology to learn PSD for LBS models (3D animation standard), we overcome both of these drawbacks. Results obtained show cloth-consistency in the animated garments and meaningful pose-dependant folds and wrinkles. Our solution is extremely efficient, handles multiple layers of cloth, allows unsupervised outfit resizing and can be easily applied to any custom 3D avatar.
Supplemental Material
- Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8387--8397.Google Scholar
Cross Ref
- Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. 2019. Tex2shape: Detailed full human body geometry from a single image. In Proceedings of the IEEE International Conference on Computer Vision. 2293--2303.Google ScholarCross Ref
- Brett Allen, Brian Curless, and Zoran Popović. 2002. Articulated body deformation from range scan data. ACM Transactions on Graphics (TOG) 21, 3 (2002), 612--619. Google ScholarDigital Library
- Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers. 408--416. Google ScholarDigital Library
- Amir Arsalan Soltani, Haibin Huang, Jiajun Wu, Tejas D Kulkarni, and Joshua B Tenenbaum. 2017. Synthesizing 3d shapes via modeling multi-view depth maps and silhouettes with deep generative networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1511--1519.Google ScholarCross Ref
- Stephen W Bailey, Dalton Omens, Paul Dilorenzo, and James F O'Brien. 2020. Fast and deep facial deformations. ACM Transactions on Graphics (TOG) 39, 4 (2020), 94--1.Google ScholarDigital Library
- Stephen W Bailey, Dave Otte, Paul Dilorenzo, and James F O'Brien. 2018. Fast and deep deformation approximations. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--12. Google ScholarDigital Library
- David Baraff and Andrew Witkin. 1998. Large steps in cloth simulation. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 43--54. Google ScholarDigital Library
- Hugo Bertiche, Meysam Madadi, and Sergio Escalera. 2019. CLOTH3D: Clothed 3D Humans. arXiv preprint arXiv:1912.02792 (2019).Google Scholar
- Hugo Bertiche, Meysam Madadi, Emilio Tylson, and Sergio Escalera. 2020. DeePSD: Automatic deep skinning and pose space deformation for 3D garment animation. arXiv preprint arXiv:2009.02715 (2020).Google Scholar
- Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. 2019. Multi-garment net: Learning to dress 3d people from images. In Proceedings of the IEEE International Conference on Computer Vision. 5420--5430.Google ScholarCross Ref
- Nuttapong Chentanez, Miles Macklin, Matthias Müller, Stefan Jeschke, and Tae-Yong Kim. 2020. Cloth and skin deformation with a triangle mesh based convolutional neural network. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 123--134. Google ScholarDigital Library
- Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. 2020. Coercing machine learning to output physically accurate results. J. Comput. Phys. 406 (2020), 109099.Google ScholarCross Ref
- Peng Guan, Loretta Reiss, David A Hirshberg, Alexander Weiss, and Michael J Black. 2012. Drape: Dressing any person. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1--10. Google ScholarDigital Library
- Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, and Pascal Fua. 2019. GarNet: A two-stream network for fast and accurate 3D cloth draping. In Proceedings of the IEEE International Conference on Computer Vision. 8739--8748.Google ScholarCross Ref
- Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace clothing simulation using adaptive bases. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--9. Google ScholarDigital Library
- Xiaoguang Han, Chang Gao, and Yizhou Yu. 2017. DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling. ACM Transactions on graphics (TOG) 36, 4 (2017), 1--12. Google ScholarDigital Library
- Daniel Holden, Bang Chi Duong, Sayantan Datta, and Derek Nowrouzezahrai. 2019. Subspace neural physics: Fast data-driven interactive simulation. In Proceedings of the 18th annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 1--12. Google ScholarDigital Library
- Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. 2020. BCNet: Learning Body and Cloth Shape from A Single Image. arXiv preprint arXiv:2004.00214 (2020).Google Scholar
- Ning Jin, Yilin Zhu, Zhenglin Geng, and Ronald Fedkiw. 2020. A Pixel-Based Framework for Data-Driven Clothing. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 135--144. Google ScholarDigital Library
- Ladislav Kavan, Steven Collins, Jiří Žára, and Carol O'Sullivan. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG) 27, 4 (2008), 1--23. Google ScholarDigital Library
- Ladislav Kavan and Jiří Žára. 2005. Spherical blend skinning: a real-time deformation of articulated models. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 9--16. Google ScholarDigital Library
- Doyub Kim, Woojong Koh, Rahul Narain, Kayvon Fatahalian, Adrien Treuille, and James F O'Brien. 2013. Near-exhaustive precomputation of secondary cloth effects. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--8. Google ScholarDigital Library
- Zorah Lahner, Daniel Cremers, and Tony Tung. 2018. Deepwrinkles: Accurate and realistic clothing modeling. In Proceedings of the European Conference on Computer Vision (ECCV). 667--684.Google ScholarDigital Library
- Binh Huy Le and Zhigang Deng. 2012. Smooth skinning decomposition with rigid bones. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--10. Google ScholarDigital Library
- John P Lewis, Matt Cordner, and Nickson Fong. 2000. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 165--172. Google ScholarDigital Library
- Tiantian Liu, Sofien Bouaziz, and Ladislav Kavan. 2017. Quasi-newton methods for real-time simulation of hyperelastic materials. ACM Transactions on Graphics (TOG) 36, 3 (2017), 1--16. Google ScholarDigital Library
- Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34, 6 (2015), 1--16. Google ScholarDigital Library
- Meysam Madadi, Hugo Bertiche, and Sergio Escalera. 2020. SMPLR: Deep learning based SMPL reverse for 3D human pose and shape recovery. Pattern Recognition (2020), 107472.Google Scholar
- Nadia Magnenat-thalmann, Richard Laperrire, Daniel Thalmann, and Université De Montréal. 1988. Joint-Dependent Local Deformations for Hand Animation and Object Grasping. In In Proceedings on Graphics interface '88. 26--33. Google ScholarDigital Library
- Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. 2019. AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE International Conference on Computer Vision. 5442--5451.Google ScholarCross Ref
- Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. 2018. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In 2018 international conference on 3D vision (3DV). IEEE, 484--494.Google ScholarCross Ref
- Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7365--7375.Google ScholarCross Ref
- Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. 2020. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409 (2020).Google Scholar
- Xavier Provot. 1997. Collision and self-collision handling in cloth model dedicated to design garments. In Computer Animation and Simulation'97. Springer, 177--189.Google Scholar
- Xavier Provot et al. 1995. Deformation constraints in a mass-spring model to describe rigid cloth behaviour. In Graphics interface. Canadian Information Processing Society, 147--147.Google Scholar
- Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652--660.Google Scholar
- Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. 2019. On the spectral bias of neural networks. In International Conference on Machine Learning. PMLR, 5301--5310.Google Scholar
- Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In 2016 fourth international conference on 3D vision (3DV). IEEE, 460--469.Google Scholar
- Igor Santesteban, Miguel A Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 355--366.Google Scholar
- Richard Socher, Brody Huval, Bharath Bath, Christopher D Manning, and Andrew Y Ng. 2012. Convolutional-recursive deep learning for 3d object classification. In Advances in neural information processing systems. 656--664. Google ScholarDigital Library
- Min Tang, Ruofeng Tong, Rahul Narain, Chang Meng, and Dinesh Manocha. 2013. A GPU-based streaming algorithm for high-resolution cloth simulation. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 21--30.Google Scholar
- Min Tang, Tongtong Wang, Zhongyuan Liu, Ruofeng Tong, and Dinesh Manocha. 2018. I-Cloth: Incremental collision handling for GPU-based interactive cloth simulation. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1--10. Google ScholarDigital Library
- Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. 2020. Sizer: A dataset and model for parsing 3d clothing and learning size sensitive 3d clothing. arXiv preprint arXiv:2007.11610 (2020).Google Scholar
- Tzvetomir Vassilev, Bernhard Spanlang, and Yiorgos Chrysanthou. 2001. Fast cloth animation on walking avatars. In Computer Graphics Forum, Vol. 20. Wiley Online Library, 260--267.Google Scholar
- Raquel Vidaurre, Igor Santesteban, Elena Garces, and Dan Casas. 2020. Fully Convolutional Graph Neural Networks for Parametric Virtual Try-On. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 145--156. Google ScholarDigital Library
- Robert Y Wang, Kari Pulli, and Jovan Popović. 2007. Real-time enveloping with rotational regression. In ACM SIGGRAPH 2007 papers. 73--es. Google ScholarDigital Library
- Xiaohuan Corina Wang and Cary Phillips. 2002. Multi-weight enveloping: least-squares approximation techniques for skin animation. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 129--138. Google ScholarDigital Library
- Hongyi Xu and Jernej Barbič. 2016. Pose-space subspace dynamics. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--14. Google ScholarDigital Library
- Cyril Zeller. 2005. Cloth simulation on the GPU. In ACM SIGGRAPH 2005 Sketches. 39--es. Google ScholarDigital Library
- Javier S Zurdo, Juan P Brito, and Miguel A Otaduy. 2012. Animating wrinkles by example on non-skinned cloth. IEEE Transactions on Visualization and Computer Graphics 19, 1 (2012), 149--158. Google ScholarDigital Library
Index Terms
- PBNS: physically based neural simulation for unsupervised garment pose space deformation
Recommendations
Coco animsim: increasing quality and efficiency
SIGGRAPH '18: ACM SIGGRAPH 2018 TalksCoco's clothing design and story presented challenges for the Simulation and Animation departments, requiring a new approach to create appealing clothed silhouettes and believable motion on characters. Many of the characters are skeletons, whose ...
An art-directed wrinkle system for CG character clothing and skin
We present a kinematic system for creating art-directed clothing and skin wrinkles on CG characters used in the production of computer-animated feature films. This system employs a curve-based method for generating wrinkles on reference poses, which are ...
Physically based rigging for deformable characters
In this paper, we introduce a framework for instrumenting (rigging) characters that are modeled as dynamic elastic bodies, so that their shapes can be controlled by an animator. Because the shape of such a character is determined by physical dynamics, ...
Comments