Gen Nishida
Ignacio Garcia-Dorado
Daniel G. Aliaga
Bedrich Benes
Skip Abstract Section
Skip Supplemental Material SectionAbstract
3D modeling remains a notoriously difficult task for novices despite significant research effort to provide intuitive and automated systems. We tackle this problem by combining the strengths of two popular domains: sketch-based modeling and procedural modeling. On the one hand, sketch-based modeling exploits our ability to draw but requires detailed, unambiguous drawings to achieve complex models. On the other hand, procedural modeling automates the creation of precise and detailed geometry but requires the tedious definition and parameterization of procedural models. Our system uses a collection of simple procedural grammars, called snippets, as building blocks to turn sketches into realistic 3D models. We use a machine learning approach to solve the inverse problem of finding the procedural model that best explains a user sketch. We use non-photorealistic rendering to generate artificial data for training convolutional neural networks capable of quickly recognizing the procedural rule intended by a sketch and estimating its parameters. We integrate our algorithm in a coarse-to-fine urban modeling system that allows users to create rich buildings by successively sketching the building mass, roof, facades, windows, and ornaments. A user study shows that by using our approach non-expert users can generate complex buildings in just a few minutes.
Supplemental Material
Available for Download
zip
a130-nishida-supp.zip (113.3 MB)
Supplemental files.
- Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., and Yu, D. 2014. Convolutional neural networks for speech recognition. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 22, 10, 1533--1545. Google Scholar
Digital Library
- Akinlar, C., and Topal, C. 2011. Edlines: Real-time line segment detection by edge drawing (ed). In ICIP, 2837--2840.Google Scholar
- Anastacio, F., Prusinkiewicz, P., and Sousa, M. C. 2009. Sketch-based parameterization of l-systems using illustration-inspired construction lines and depth modulation. Comp. & Graph. 33, 4, 440--451. Google ScholarDigital Library
- Applegate, C. S., Laycock, S. D., and Day, A. 2012. A sketch-based system for highway design with user-specified regions of influence. Comp. & Graph. 36, 6, 685--695. Google ScholarDigital Library
- Bae, S., Balakrishnan, R., and Singh, K. 2008. Ilovesketch: as-natural-as-possible sketching system for creating 3d curve models. In User Interface Software and Technology. Google ScholarDigital Library
- Bell, S., and Bala, K. 2015. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34, 4, 98:1--98:10. Google ScholarDigital Library
- Chen, G., Esch, G., Wonka, P., Müller, P., and Zhang, E. 2008. Interactive procedural street modeling. ACM Trans. Graph. 27, 3, 103:1--103:10. Google ScholarDigital Library
- Chen, X., Kang, S. B., Xu, Y.-Q., Dorsey, J., and Shum, H.-Y. 2008. Sketching reality: Realistic interpretation of architectural designs. ACM Trans. Graph. 27, 2, 11:1--11:15. Google ScholarDigital Library
- Eigen, D., Puhrsch, C., and Fergus, R. 2014. Depth map prediction from a single image using a multi-scale deep network. In Neural Information Processing Systems (NIPS). Google ScholarDigital Library
- Eissen, K., and Steur, R. 2009. Sketching: Drawing Techniques for Product Designers. BIS Publishers.Google Scholar
- Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4, 31:1--31:10. Google ScholarDigital Library
- Emilien, A., Vimont, U., Cani, M.-P., Poulin, P., and Benes, B. 2015. Worldbrush: Interactive example-based synthesis of procedural virtual worlds. ACM Trans. Graph. 34, 4, 106:1--106:11. Google ScholarDigital Library
- Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In Proc. of Siggraph, 409--416. Google ScholarDigital Library
- Ijiri, T., Owada, S., and Igarashi, T. 2006. The sketch l-system: Global control of tree modeling using free-form strokes. In Smart Graphics, Springer, 138--146.Google ScholarCross Ref
- Jampani, V., Nowozin, S., Loper, M., and Gehler, P. V. 2015. The informed sampler: A discriminative approach to bayesian inference in generative computer vision models. Computer Vision and Image Understanding 136, 32--44. Google ScholarDigital Library
- Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093.Google Scholar
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097--1105.Google Scholar
- Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278--2324.Google ScholarCross Ref
- Lipp, M., Wonka, P., and Wimmer, M. 2008. Interactive visual editing of grammars for procedural architecture. ACM Trans. Graph. 27, 3, 102:1--102:10. Google ScholarDigital Library
- Lipson, H., and Shpitalni, M. 1996. Optimization-based reconstruction of a 3d object from a single freehand line drawing. Computer-Aided Design 28, 651--663.Google ScholarCross Ref
- Lipson, H., and Shpitalni, M. 2000. Conceptual design and analysis by sketching. Artif. Intell. Eng. Des. Anal. Manuf. 14, 5, 391--401. Google ScholarDigital Library
- Longay, S., Runions, A., Boudon, F., and Prusinkiewicz, P. 2012. Treesketch: Interactive procedural modeling of trees on a tablet. In SBIM, Eurographics Association, 107--120. Google ScholarDigital Library
- Loomis, A., and Ross, A. 2014. I'd Love to Draw. Titan Books.Google Scholar
- Müller, P., Wonka, P., Haegler, S., Ulmer, A., and Van Gool, L. 2006. Procedural modeling of buildings. ACM Trans. Graph. 25, 3, 614--623. Google ScholarDigital Library
- Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge, J. A. 2009. Sketch-based modeling: A survey. Comp. & Graph. 33, 1, 85--103. Google ScholarDigital Library
- Parish, Y. I., and Müller, P. 2001. Procedural modeling of cities. In Comp. graphics and interactive techniques, ACM, 301--308. Google ScholarDigital Library
- Pfister, T., Charles, J., and Zisserman, A. 2015. Flowing convnets for human pose estimation in videos. In ICCV. Google ScholarDigital Library
- Prusinkiewicz, P., and Lindenmayer, A. 2012. The algorithmic beauty of plants. Springer Science & Business Media.Google Scholar
- Ritchie, D., Mildenhall, B., Goodman, N. D., and Hanrahan, P. 2015. Controlling procedural modeling programs with stochastically-ordered sequential monte carlo. ACM Trans. Graph. 34, 4, 105:1--105:11. Google ScholarDigital Library
- Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV 115, 3, 211--252. Google ScholarDigital Library
- Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. 2009. Analytic drawing of 3d scaffolds. ACM Trans. Graph. 28, 5, 149:1--149:10. Google ScholarDigital Library
- Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. 2013. Geosemantic snapping for sketch-based modeling. Comp. Graph. Forum 32, 2, 245--253.Google ScholarCross Ref
- Smelik, R., Tutenel, T., de Kraker, K. J., and Bidarra, R. 2010. Interactive creation of virtual worlds using procedural sketching. In Proceedings of eurographics.Google Scholar
- Smelik, R. M., Tutenel, T., Bidarra, R., and Benes, B. 2014. A survey on procedural modelling for virtual worlds. In Comp. Graph. Forum, vol. 33, 31--50.Google ScholarDigital Library
- Smith, A. R. 1984. Plants, fractals, and formal languages. SIGGRAPH Comput. Graph. 18, 3, 1--10. Google ScholarDigital Library
- Stava, O., Pirk, S., Kratt, J., Chen, B., Mch, R., Deussen, O., and Benes, B. 2014. Inverse procedural modelling of trees. Comp. Graph. Forum 33, 6, 118--131.Google ScholarDigital Library
- Su, H., Qi, C. R., Li, Y., and Guibas, L. J. 2015. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In ICCV. Google ScholarDigital Library
- Talton, J. O., Lou, Y., Lesser, S., Duke, J., Měch, R., and Koltun, V. 2011. Metropolis procedural modeling. ACM Trans. Graph. 30, 2, 11:1--11:14. Google ScholarDigital Library
- Vanegas, C. A., Aliaga, D. G., Wonka, P., Müller, P., Waddell, P., and Watson, B. 2010. Modelling the appearance and behaviour of urban spaces. Comp. Graph. Forum 29, 1, 25--42.Google ScholarCross Ref
- Vanegas, C. A., Garcia-Dorado, I., Aliaga, D. G., Benes, B., and Waddell, P. 2012. Inverse design of urban procedural models. ACM Trans. Graph. 31, 6, 168:1--168:11. Google ScholarDigital Library
- Wang, F., Kang, L., and Li, Y. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. arXiv preprint arXiv:1504.03504.Google Scholar
- Wonka, P., Wimmer, M., Sillion, F., and Ribarsky, W. 2003. Instant architecture. ACM Trans. Graph. 22, 3, 669--677. Google ScholarDigital Library
- Xie, X., Xu, K., Mitra, N. J., Cohen-Or, D., Gong, W., Su, Q., and Chen, B. 2013. Sketch-to-design: Context-based part assembly. Comp. Graph. Forum 32, 8, 233--245.Google ScholarCross Ref
- Xu, B., Chang, W., Sheffer, A., Bousseau, A., Mccrae, J., and Singh, K. 2014. True2form: 3d curve networks from 2d sketches via selective regularization. ACM Trans. on Graph. 33, 4, 131:1--131:13. Google ScholarDigital Library
- Xue, T., Liu, J., and Tang, X. 2012. Example-based 3d object reconstruction from line drawings. In CVPR, IEEE, 302--309. Google ScholarDigital Library
- Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. 1996. Sketch: An interface for sketching 3d scenes. In Computer Graphics, Proceedings of Siggraph 1996, 163--170. Google ScholarDigital Library
- Zheng, Y., Liu, H., Dorsey, J., and Mitra, N. J. 2016. Smartcanvas; context-inferred interpretation of sketches for preparatory design studies. Comp. Graph. Forum.Google Scholar
Index Terms
- Interactive sketching of urban procedural models
Recommendations
Inverse design of urban procedural models
We propose a framework that enables adding intuitive high level control to an existing urban procedural model. In particular, we provide a mechanism to interactively edit urban models, a task which is important to stakeholders in gaming, urban planning, ...
3D Hair sketching for real-time dynamic & key frame animations
Physically based simulation of human hair is a well studied and well known problem. But the “pure” physically based representation of hair (and other animation elements) is not the only concern of the animators, who want to “control” the creation and ...
Sketching interactive systems with sketchify
INTERACT'11: Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction - Volume Part IVSketching is at the heart of design and creativity, an omnipresent element of any disciplined activity of design. In this tutorial we will summarize many of the existing studies of sketching, and emphasize its role in supporting creativity. We will look ...
Comments