Felix Heide
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Conventional pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, each responsible for addressing a particular problem. While this divide-and-conquer approach offers many benefits, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the original sensor data. We propose an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation (e.g., YUV, DCT). Our system is flexible and we demonstrate it on regular Bayer images as well as images from custom sensors. In all cases, we achieve large improvements in image quality and signal reconstruction compared to state-of-the-art techniques. Finally, we show that our approach is capable of very efficiently handling high-resolution images, making even mobile implementations feasible.
Supplemental Material
Available for Download
zip
a231.zip (360.2 MB)
Supplemental material.
- Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D., Ajdin, B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J., Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K., Horowitz, M., and Levoy, M. 2010. The Frankencamera: an experimental platform for computational photography. ACM TOG 29, 4. Google Scholar
Digital Library
- Afonso, M. V., Bioucas-Dias, J. M., and Figueiredo, M. A. 2010. Fast image recovery using variable splitting and constrained optimization. IEEE TIP 19, 9. Google ScholarDigital Library
- Ajdin, B., Hullin, M. B., Fuchs, C., Seidel, H.-P., and Lensch, H. 2008. Demosaicing by smoothing along 1d features. In CVPR.Google Scholar
- Bennett, E. P., Uyttendaele, M., Zitnick, C., Szeliski, R., and Kang, S. 2006. Video and image bayesian demosaicing with a two color image prior. In ECCV. Google ScholarDigital Library
- Brauers, J., Seiler, C., and Aach, T. 2010. Direct psf estimation using a random noise target. In Electronic Imaging.Google Scholar
- Buades, A., Coll, B., and Morel, J. M. 2005. A non-local algorithm for image denoising. In CVPR. Google ScholarDigital Library
- Buades, T., Lou, Y., Morel, J. M., and Tang, Z. 2009. A note on multi-image denoising. In Int. Workshop on Local and Non-Local Approximation in Image Processing.Google Scholar
- Buades, A., Coll, B., and Morel, J.-M. 2011. Non-Local Means Denoising. Image Processing On Line.Google Scholar
- Candès, E., and Donoho, D. 1999. Curvelets: A surprisingly effective nonadaptive representation of objects with edges. In Curves and Surfaces. Vanderbilt University Press.Google Scholar
- Chambolle, A., and Pock, T. 2011. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40, 1. Google ScholarDigital Library
- Chatterjee, P., Joshi, N., Kang, S. B., and Matsushita, Y. 2011. Noise suppression in low-light images through joint denoising and demosaicing. In CVPR. Google ScholarDigital Library
- Coifman, R. R., and Donoho, D. L. 1995. Translation-invariant denoising. Tech. Rep. 475, Stanford University.Google Scholar
- Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE TIP 16, 8. Google ScholarDigital Library
- Dabov, K., Foi, A., and Egiazarian, K. 2007. Video de-noising by sparse 3d transform-domain collaborative filtering. In European Signal Processing Conference.Google Scholar
- Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K. 2008. Image restoration by sparse 3d transform-domain collaborative filtering. In Electronic Imaging.Google Scholar
- Danielyan, A., Katkovnik, V., and Egiazarian, K. 2012. BM3D frames and variational image deblurring. IEEE TIP 21, 4.Google Scholar
- Egiazarian, K. O., Astola, J., Helsingius, M., and Kuosmanen, P. 1999. Adaptive denoising and lossy compression of images in transform domain. J. Electronic Imaging 8, 3.Google ScholarCross Ref
- Farsiu, S., Elad, M., and Milanfar, P. 2006. Multiframe demosaicing and superresolution of color image. IEEE TIP 15, 1. Google ScholarDigital Library
- Foi, A., Katkovnik, V., and Egiazarian, K. 2007. Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE TIP 16, 5. Google ScholarDigital Library
- Gu, J., Hitomi, Y., Mitsunaga, T., and Nayar, S. 2010. Coded rolling shutter photography: Flexible space-time sampling. In ICCP.Google Scholar
- Hajisharif, S., Kronander, J., and Unger, J. 2014. HDR reconstruction for alternating gain (ISO) sensor readout. In Proc. of Eurographics (short paper).Google Scholar
- Heide, F., Rouf, M., Hullin, M. B., Labitzke, B., Heidrich, W., and Kolb, A. 2013. High-quality computational imaging through simple lenses. ACM TOG 32, 5. Google ScholarDigital Library
- Irani, M., and Peleg, S. 1991. Improving resolution by image registration. Graphical Models and Image Processing 53, 3. Google ScholarDigital Library
- Jeon, G., and Dubois, E. 2013. Demosaicking of noisy Bayer-sampled color images with least-squares luma-chroma demultiplexing and noise level estimation. IEEE TIP 22, 1.Google Scholar
- Joulin, A., and Kang, S. B. 2013. Recovering stereo pairs from anaglyphs. In CVPR. Google ScholarDigital Library
- Krishnan, D., and Fergus, R. 2009. Fast image deconvolution using hyper-laplacian priors. In NIPS.Google Scholar
- Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Deconvolution using natural image priors. ACM Transactions on Graphics (TOG) 26, 3.Google ScholarDigital Library
- Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In ECCV. Google ScholarDigital Library
- Liu, C. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. PhD thesis, MIT. Google ScholarDigital Library
- Mitra, K., Cossairt, O., and Veeraraghavan, A. 2014. A framework for analysis of computational imaging systems: Role of signal prior, sensor noise and multiplexing. ieee PAMI 36, 10.Google Scholar
- Mosseri, I., Zontak, M., and Irani, M. 2013. Combining the power of internal and external denoising. In ICCP.Google Scholar
- Narasimhan, S. G., and Nayar, S. K. 2005. Enhancing resolution along multiple imaging dimensions using assorted pixels. IEEE PAMI 27, 4. Google ScholarDigital Library
- Nayar, S., and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In ICCV. Google ScholarDigital Library
- Oymak, S., and Hassibi, B. 2013. Sharp MSE bounds for proximal denoising. arXiv preprint arXiv:1305.2714.Google Scholar
- Parikh, N., and Boyd, S. 2013. Proximal algorithms. Foundations and Trends in Optimization 1, 3.Google Scholar
- Ramanath, R., Snyder, W. E., Yoo, Y., and Drew, M. S. 2005. Color image processing pipeline in digital still cameras. IEEE Signal Processing Magazine 22, 1.Google ScholarCross Ref
- Reinhard, E., Ward, G., Pattanaik, S., Debevec, P., Heidrich, W., and Myszkowski, K. 2010. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann.Google Scholar
- Roth, S., and Black, M. J. 2009. Fields of experts. International Journal of Computer Vision 82, 2. Google ScholarDigital Library
- Rudin, L. I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60, 1. Google ScholarDigital Library
- Schuler, C. J., Hirsch, M., Harmeling, S., and Schölkopf, B. 2011. Non-stationary correction of optical aberrations. In ICCV. Google ScholarDigital Library
- Schuler, C. J., Burger, H. C., Harmeling, S., and Schölkopf, B. 2013. A machine learning approach for non-blind image deconvolution. In CVPR. Google ScholarDigital Library
- Shao, L., Yan, R., Li, X., and Liu, Y. 2013. From heuristic optimization to dictionary learning: A review and comprehensive comparison of image denoising algorithms. IEEE Transactions on Cybernetics, 99.Google Scholar
- Starck, J.-L., Murtagh, F., and Bijaoui, A. 1998. Image Processing and Data Analysis: The Multiscale Approach. Cambridge University Press, Cambridge. Google ScholarDigital Library
- Tico, M., and Pulli, K. 2009. Image enhancement method via blur and noisy image fusion. In ICIP. Google ScholarDigital Library
- Tico, M. 2008. Multiframe image denoising and stabilization. In European Signal Processing Conference.Google Scholar
- Tsai, Y.-T., Steinberger, M., Pająk, D., and Pulli, K. 2014. Fast ANN for high-quality collaborative filtering. In High-Performance Graphics.Google Scholar
- Venkatakrishnan, S. V., Bouman, C. A., and Wohlberg, B. 2013. Plug-and-play priors for model based reconstruction. In IEEE GlobalSIP.Google Scholar
- Venkataraman, K., Lelescu, D., Duparré, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., and Nayar, S. 2013. Picam: an ultra-thin high performance monolithic camera array. ACM TOG 32, 6. Google ScholarDigital Library
- Wallace, G. K. 1991. The JPEG still picture compression standard. Communications of the ACM 34, 4. Google ScholarDigital Library
- Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM TOG 24, 3. Google ScholarDigital Library
- Wu, X., and Zhang, D. 2006. Improvement of color video demosaicking in temporal domain. IEEE TIP 15, 10. Google ScholarDigital Library
- Xu, L., and Jia, J. 2010. Two-phase kernel estimation for robust motion deblurring. In ECCV. Google ScholarDigital Library
- Zhang, L., Wu, X., Buades, A., and Li, X. 2011. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging 20, 2.Google Scholar
- Zoran, D., and Weiss, Y. 2011. From learning models of natural image patches to whole image restoration. In ICCV. Google ScholarDigital Library
Index Terms
- FlexISP: a flexible camera image processing framework
Recommendations
An Improved Algorithm Based on Exemplar Based Image Inpainting
AIAM 2019: Proceedings of the 2019 International Conference on Artificial Intelligence and Advanced ManufacturingImage restoration technology is an important image processing technology. In image restoration, it not only can reconstruct the loss part of impaired image caused by external factors, but also can remove unwanted parts from the specified image. ...
Image Representation Using Block Pattern Models and Its Image Processing Applications
An image representation scheme using a set of block pattern models (BPMs) consisting of three categories (constant, oriented, and irregular) is introduced. Algorithms for model classification, model parameter estimation, and image reconstruction from ...
Bayesian Image Restoration: An Application to Edge-Preserving Surface Recovery
Bayesian methods for recovering a 2-D surface are discussed. It is assumed that there is a textural image that can be modeled by a Markov random field and that the original surface is composed of different surfaces, each of which is associated with one ...
Comments