Alexandr Kuznetsov
Abstract
We propose NeuMIP, a neural method for representing and rendering a variety of material appearances at different scales. Classical prefiltering (mipmapping) methods work well on simple material properties such as diffuse color, but fail to generalize to normals, self-shadowing, fibers or more complex microstructures and reflectances. In this work, we generalize traditional mipmap pyramids to pyramids of neural textures, combined with a fully connected network. We also introduce neural offsets, a novel method which enables rendering materials with intricate parallax effects without any tessellation. This generalizes classical parallax mapping, but is trained without supervision by any explicit heightfield. Neural materials within our system support a 7-dimensional query, including position, incoming and outgoing direction, and the desired filter kernel size. The materials have small storage (on the order of standard mipmapping except with more texture channels), and can be integrated within common Monte-Carlo path tracing systems. We demonstrate our method on a variety of materials, resulting in complex appearance across levels of detail, with accurate parallax, self-shadowing, and other effects.
Supplemental Material
- Adobe. 2021. Substance Source library. https://source.substance3d.com.Google Scholar
- Barry G Becker and Nelson L Max. 1993. Smooth transitions between bump rendering algorithms. In SIGGRAPH 93. 183--190.Google Scholar
- Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020).Google Scholar
- Eric Bruneton and Fabrice Neyret. 2011. A survey of nonlinear prefiltering methods for efficient and accurate surface shading. IEEE Transactions on Visualization and Computer Graphics 18, 2 (2011), 242--260.Google ScholarDigital Library
- Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum 35, 2 (2016), 275--283.Google ScholarCross Ref
- Kristin J Dana, Bram Van Ginneken, Shree K Nayar, and Jan J Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Transactions on Graphics (TOG) 18, 1 (1999), 1--34.Google ScholarDigital Library
- Jonathan Dupuy, Eric Heitz, Jean-Claude Iehl, Pierre Poulin, Fabrice Neyret, and Victor Ostromoukhov. 2013. Linear efficient antialiased displacement and reflectance mapping. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1--11.Google ScholarDigital Library
- Jiři Filip and Michal Haindl. 2008. Bidirectional texture function modeling: A state of the art survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 11 (2008), 1921--1940.Google ScholarDigital Library
- Charles Han, Bo Sun, Ravi Ramamoorthi, and Eitan Grinspun. 2007. Frequency Domain Normal Map Filtering. ACM Transactions on Graphics (TOG) 26, 3, 28.Google ScholarDigital Library
- Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
- Wenzel Jakob, Miloš Hašan, Ling-Qi Yan, Jason Lawrence, Ravi Ramamoorthi, and Steve Marschner. 2014. Discrete stochastic microfacet models. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--10.Google ScholarDigital Library
- Tomomichi Kaneko, Toshiyuki Takahei, Masahiko Inami, Naoki Kawakami, Yasuyuki Yanagida, Taro Maeda, and Susumu Tachi. 2001. Detailed shape representation with parallax mapping. In Proceedings of the ICAT 2001 (01 2001).Google Scholar
- Anton S Kaplanyan, Stephan Hill, Anjul Patney, and Aaron E Lefohn. 2016. Filtering distributions of normals for shading antialiasing.. In High Performance Graphics. 151--162.Google Scholar
- Melissa L Koudelka, Sebastian Magda, Peter N Belhumeur, and David J Kriegman. 2003. Acquisition, compression, and synthesis of bidirectional texture functions. In 3rd International Workshop on Texture Analysis and Synthesis (Texture 2003). 59--64.Google Scholar
- Alexandr Kuznetsov, Milos Hasan, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning generative models for rendering specular microgeometry. ACM Transactions on Graphics (TOG) 38, 6 (2019), 225.Google ScholarDigital Library
- Maxim Maximov, Laura Leal-Taixé, Mario Fritz, and Tobias Ritschel. 2019. Deep appearance maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8729--8738.Google ScholarCross Ref
- Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020.Google ScholarDigital Library
- Gero Müller, Jan Meseth, and Reinhard Klein. 2003. Compression and Real-Time Rendering of Measured BTFs Using Local PCA.. In VMV. 271--279.Google Scholar
- Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM Transactions on Graphics (TOG) 38, 5, Article 145 (2019), 19 pages.Google ScholarDigital Library
- Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural Control Variates. ACM Transactions on Graphics (TOG) 39, 6, Article 243 (2020), 19 pages.Google ScholarDigital Library
- Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. Computer Graphics Forum (Proc. EGSR 2017) 36, 4 (2017), 65--78.Google Scholar
- Marc Olano and Dan Baker. 2010. Lean mapping. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 181--188.Google ScholarDigital Library
- Manuel M Oliveira and Fabio Policarpo. 2005. An efficient representation for surface details. UFRGS Technical Report (2005).Google Scholar
- Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2018. PBRT v3. https://github.com/mmp/pbrt-v3/.Google Scholar
- Gilles Rainer, Abhijeet Ghosh, Wenzel Jakob, and Tim Weyrich. 2020. Unified Neural Encoding of BTFs. Computer Graphics Forum (Proceedings of Eurographics) 39, 2 (2020), 167--178.Google ScholarCross Ref
- Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (2019), 235--244.Google ScholarCross Ref
- Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep reflectance maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4508--4516.Google ScholarCross Ref
- Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google ScholarDigital Library
- Jiaping Wang, Xin Tong, John Snyder, Yanyun Chen, Baining Guo, and Heung-Yeung Shum. 2005. Capturing and rendering geometry details for BTF-mapped surfaces. The Visual Computer 21, 8 (2005), 559--568.Google ScholarCross Ref
- Michael Weinmann, Juergen Gall, and Reinhard Klein. 2014. Material Classification Based on Training Data Synthesized Using a BTF Database. In ECCV 2014. 156--171.Google ScholarCross Ref
- Lance Williams. 1983. Pyramidal parametrics. In SIGGRAPH 83. 1--11.Google Scholar
- Lifan Wu, Shuang Zhao, Ling-Qi Yan, and Ravi Ramamoorthi. 2019. Accurate appearance preserving prefiltering for rendering displacement-mapped surfaces. ACM Transactions on Graphics (TOG) 38, 4 (2019), 137.Google ScholarDigital Library
- Ling-Qi Yan, Weilun Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2017. A BSSRDF Model for Efficient Rendering of Fur with Global Illumination. ACM Transactions on Graphics (TOG) 36, 6 (2017), 208.Google ScholarDigital Library
- Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM Transactions on Graphics (TOG) 35, 6 (2016), 166.Google ScholarDigital Library
Index Terms
- NeuMIP: multi-resolution neural materials
Recommendations
Rendering Neural Materials on Curved Surfaces
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference ProceedingsNeural material reflectance representations address some limitations of traditional analytic BRDFs with parameter textures; they can theoretically represent any material data, whether a complex synthetic microgeometry with displacements, shadows and ...
Real-time rendering of realistic-looking grass
GRAPHITE '05: Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East AsiaThe absence of accurately rendered grass in real-time applications such as games and simulation systems can be directly attributed to the massive amounts of geometry required to model grass patches. This in turn is responsible for the drastic increase ...
Shell texture functions
We propose a texture function for realistic modeling and efficient rendering of materials that exhibit surface mesostructures, translucency and volumetric texture variations. The appearance of such complex materials for dynamic lighting and viewing ...
Comments