Matthew O'Toole
Abstract
We analyze light propagation in an unknown scene using projectors and cameras that operate at transient timescales. In this new photography regime, the projector emits a spatio-temporal 3D signal and the camera receives a transformed version of it, determined by the set of all light transport paths through the scene and the time delays they induce. The underlying 3D-to-3D transformation encodes scene geometry and global transport in great detail, but individual transport components (e.g., direct reflections, inter-reflections, caustics, etc.) are coupled nontrivially in both space and time.
To overcome this complexity, we observe that transient light transport is always separable in the temporal frequency domain. This makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport. We use this idea in a prototype that offers three never-seen-before abilities: (1) acquiring time-of-flight depth images that are robust to general indirect transport, such as interreflections and caustics; (2) distinguishing between direct views of objects and their mirror reflection; and (3) using a photonic mixer device to capture sharp, evolving wavefronts of "light-in-flight".
Supplemental Material
Available for Download
Supplemental material.
- Bai, J., Chandraker, M., Ng, T.-T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In Proc. ECCV, 294--307. Google Scholar
Digital Library
- Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. ACM SIGGRAPH, 145--156. Google ScholarDigital Library
- Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. SPIE, vol. 7864.Google Scholar
- Fuchs, S. 2010. Multipath interference compensation in time-of-flight camera images. In Proc. ICPR, 3583--3586. Google ScholarDigital Library
- Garg, G., Talvala, E.-V., Levoy, M., and Lensch, H. P. 2006. Symmetric photography: exploiting data-sparseness in reflectance fields. In Proc. EGSR, 251--262. Google ScholarDigital Library
- Godbaz, J. P., Cree, M. J., and Dorrington, A. A. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in amcw lidar. In Proc. SPIE, vol. 8296.Google Scholar
- Goral, C. M., Torrance, K. E., Greenberg, D. P., and Battaile, B. 1984. Modeling the interaction of light between diffuse surfaces. ACM SIGGRAPH, 213--222. Google ScholarDigital Library
- Gupta, M., and Nayar, S. K. 2012. Micro phase shifting. In Proc. ICCV, 813--820. Google ScholarDigital Library
- Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM SIGGRAPH, 45:1--45:10. Google ScholarDigital Library
- Hlawitschka, M., Ebling, J., and Scheuermann, G. 2004. Convolution and fourier transform of second order tensor fields. In Proc. IASTED VIIP, 78--83.Google Scholar
- Jiménez, D., Pizarro, D., Mazo, M., and Palazuelos, S. 2014. Modeling and correction of multipath interference in time of flight cameras. Image and Vision Computing 32, 1, 1--13. Google ScholarDigital Library
- Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Coded time of flight cameras: sparse deconvolution to address multi-path interference and recover time profiles. ACM Trans. Graph. 32, 6, 167:1--167:10. Google ScholarDigital Library
- Kajiya, J. T. 1986. The rendering equation. ACM SIGGRAPH, 143--150. Google ScholarDigital Library
- Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2011. Looking around the corner using ultrafast transient imaging. Int. J. of Computer Vision 95, 13--28. Google ScholarDigital Library
- Kirmani, A., Colaco, A., Wong, F. N. C., and Goyal, V. K. 2012. Codac: a compressive depth acquisition camera framework. In Proc. ICASSP, 5425--5428.Google Scholar
- Kirmani, A., Venkatraman, D., Shin, D., Colaco, A., Wong, F. N. C., Shapiro, J. H., and Goyal, V. K. 2013. First-photon imaging. Science.Google Scholar
- Mei, J., Kirmani, A., Colaco, A., and Goyal, V. 2013. Phase unwrapping and denoising for time-of-flight imaging using generalized approximate message passing. In Proc. ICIP, 364--368.Google Scholar
- Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM SIGGRAPH, 935--944. Google ScholarDigital Library
- Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM SIGGRAPH, 376--381. Google ScholarDigital Library
- O'Toole, M., and Kutulakos, K. N. 2010. Optical computing for fast light transport analysis. ACM Trans. Graph., 164:1--164:12. Google ScholarDigital Library
- O'Toole, M., Raskar, R., and Kutulakos, K. N. 2012. Primal-dual coding to probe light transport.Google Scholar
- O'Toole, M., Mather, J., and Kutulakos, K. N. 2014. 3D shape and indirect appearance by structured light transport. In Proc. CVPR.Google Scholar
- Peers, P., Mahajan, D. K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R., and Debevec, P. 2009. Compressive light transport sensing. ACM Trans. Graph. 28, 1. Google ScholarDigital Library
- Reddy, D., Ramamoorthi, R., and Curless, B. 2012. Frequency-space decomposition and acquisition of light transport under spatially varying illumination. In Proc. ECCV, 596--610. Google ScholarDigital Library
- Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. ICCV, 1440--1447. Google ScholarDigital Library
- Sen, P., and Darabi, S. 2009. Compressive dual photography. Computer Graphics Forum 28, 2, 609--618.Google ScholarCross Ref
- Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. ACM SIGGRAPH, 745--755. Google ScholarDigital Library
- Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M. G., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications.Google Scholar
- Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4. Google ScholarDigital Library
- Wetzstein, G., and Bimber, O. 2007. Radiometric compensation through inverse light transport. 391--399. Google ScholarDigital Library
- Wu, D., O'Toole, M., Velten, A., Agrawal, A., and Raskar, R. 2012. Decomposing global light transport using time of flight imaging. In Proc. CVPR, 366--373. Google ScholarDigital Library
- Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., Dai, Q., and Raskar, R. 2013. Ultra-fast lensless computational imaging through 5d frequency analysis of time-resolved light transport. Int. J. of Computer Vision, 1--13.Google Scholar
Index Terms
- Temporal frequency probing for 5D transient analysis of global light transport
Recommendations
Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging
In correlation-based time-of-flight (C-ToF) imaging systems, light sources with temporally varying intensities illuminate the scene. Due to global illumination, the temporally varying radiance received at the sensor is a combination of light received ...
Primal-dual coding to probe light transport
We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the ...
A practical model for subsurface light transport
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniquesThis paper introduces a simple model for subsurface light transport in translucent materials. The model enables efficient simulation of effects that BRDF models cannot capture, such as color bleeding within materials and diffusion of light across shadow ...
Comments