Abstract
Deep learning-based image denoising approaches have been extensively studied in recent years, prevailing in many public benchmark datasets. However, the stat-of-the-art networks are computationally too expensive to be directly applied on mobile devices. In this work, we propose a light-weight, efficient neural network-based raw image denoiser that runs smoothly on mainstream mobile devices, and produces high quality denoising results. Our key insights are twofold: (1) by measuring and estimating sensor noise level, a smaller network trained on synthetic sensor-specific data can out-perform larger ones trained on general data; (2) the large noise level variation under different ISO settings can be removed by a novel k-Sigma Transform, allowing a small network to efficiently handle a wide range of noise levels. We conduct extensive experiments to demonstrate the efficiency and accuracy of our approach. Our proposed mobile-friendly denoising model runs at \(\sim \)70 ms per megapixel on Qualcomm Snapdragon 855 chipset, and it is the basis of the night shot feature of several flagship smartphones released in 2019.
This work is supported by The National Key Research and Development Program of China under Grant 2018YFC0831700.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Aharon, M., Elad, M., Bruckstein, A., et al.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Sig. Process. 54(11), 4311 (2006)
Anaya, J., Barbu, A.: RENOIR-a dataset for real low-light image noise reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)
Anscombe, F.J.: The transformation of Poisson, binomial and negative-binomial data. Biometrika 35(3/4), 246–254 (1948)
Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11036–11045 (2019)
Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 60–65. IEEE (2005)
Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: CVPR (2012)
Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3291–3300 (2018)
Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: CVPR (2018)
Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2017)
Chollet, F.: Xception: deep learning with depthwise separable convolutions, October 2016. http://arxiv.org/abs/1610.02357
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image restoration by sparse 3D transform-domain collaborative filtering. In: Image Processing: Algorithms and Systems VI, vol. 6812, p. 681207. International Society for Optics and Photonics (2008)
Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)
European Machine Vision Association.: Standard for Characterization of Image Sensors and Cameras (2010). https://doi.org/10.1063/1.1518010
Foi, A., Alenius, S., Katkovnik, V., Egiazarian, K.: Noise measurement for raw-data of digital imaging sensors by automatic segmentation of nonuniform targets. IEEE Sens. J. 7(10), 1456–1461 (2007)
Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. 17(10), 1737–1754 (2008)
Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (TOG) 35(6), 191 (2016)
Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: CVPR (2014)
Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6), 1–12 (2016). https://doi.org/10.1145/2980179.2980254. http://dl.acm.org/citation.cfm?doid=2980179.2980254
Hirakawa, K., Parks, T.W.: Joint demosaicing and denoising. IEEE Trans. Image Process. 15(8), 2146–2157 (2006)
Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in neural information processing systems, pp. 769–776 (2009)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018)
Liba, O., et al.: Handheld mobile photography in very low light. ACM Trans. Graph. 38(6) (2019). https://doi.org/10.1145/3355089.3356508
Liu, C., Szeliski, R., Kang, S.B., Zitnick, C.L., Freeman, W.T.: Automatic estimation and removal of noise from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 299–314 (2008)
Liu, J., et al.: Learning raw image denoising with Bayer pattern unification and Bayer preserving augmentation, April 2019. http://arxiv.org/abs/1904.12945
Liu, X., Tanaka, M., Okutomi, M.: Practical signal-dependent noise parameter estimation from a single noisy image. IEEE Trans. Image Process. 23(10), 4361–4371 (2014)
Mairal, J., Bach, F.R., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: ICCV, vol. 29, pp. 54–62. Citeseer (2009)
Makitalo, M., Foi, A.: Optimal inversion of the Anscombe transformation in low-count Poisson image denoising. IEEE Trans. Image Process. 20(1), 99–109 (2010)
Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: NeurIPS (2016)
Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks, December 2017. https://arxiv.org/abs/1712.02327
Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shi, G., Zifei, Y., Kai, Z., Wangmeng, Z., Lei, Z.: Toward convolutional blind denoising of real photographs. arXiv preprint arXiv:1807.04686 (2018)
Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)
Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: Proceedings of the IEEE international Conference on Computer Vision, pp. 4539–4547 (2017)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)
Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, pp. 341–349 (2012)
Xu, J., Zhang, L., Zhang, D., Feng, X.: Multi-channel weighted nuclear norm minimization for real color image denoising. In: ICCV (2017)
Yair, N., Michaeli, T.: Multi-scale weighted nuclear norm image restoration. In: CVPR (2018)
Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)
Zhou, Y., et al.: When AWGN-based denoiser meets real noises. arXiv preprint arXiv:1904.03485 (2019)
Zhou, Y., Liu, D., Huang, T.: Survey of face detection on low-quality images. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), pp. 769–773. IEEE (2018)
Zhu, F., Chen, G., Heng, P.A.: From noise modeling to blind image denoising. In: CVPR (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Huang, H., Xu, Q., Liu, J., Liu, Y., Wang, J. (2020). Practical Deep Raw Image Denoising on Mobile Devices. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12351. Springer, Cham. https://doi.org/10.1007/978-3-030-58539-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-58539-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58538-9
Online ISBN: 978-3-030-58539-6
eBook Packages: Computer ScienceComputer Science (R0)