Defocus map estimation from a single image
Research highlights
► We address the challenging problem of defocus estimation from a single image. ► The defocus blur is estimated at edge locations from the gradient ratio between tlie original and re-blurred input image. ► A hill defocus map is obtained by propagating the blur amount at edges to the entire image using soft matting. ► We also discuss the ambiguities in defocus estimation and the relationship between defocus map and depth.
Introduction
Defocus estimation plays an important role in many computer vision and computer graphics applications including depth estimation, image quality assessment, image deblurring and refocusing. Conventional methods for defocus estimation have relied on multiple images [1], [2], [3], [4]. A set of images of the same scene are captured using multiple focus settings. Then the defocus is measured during a implicit or explicit deblurring process. Recently, image pairs captured using coded aperture cameras [5] are used for better defocus blur measure and all-focused image recovery. However, these methods suffer from the occlusion problem and require the scene to be static, which limits their applications in practice.
In very specific settings, several methods have been proposed to recover defocus map from a single image. Active illumination methods [6] project sparse grid dots onto the scene and the defocus blur of those dots is measured by comparing them with calibrated images. Then the defocus measure can be used to estimate the depth of a scene. The coded aperture method [7] changes the shape of camera aperture to make defocus deblurring more reliable. A defocus map and an all-focused image can be obtained after deconvolution using calibrated blur kernels. These methods require additional illumination or camera modification to obtain a defocus map from a single image.
In this paper, we focus on a more challenging problem of recovering the defocus map from a single image captured by an uncalibrated conventional camera. Elder and Zucker [8] used the first- and second-order derivatives of the input image to find the locations and the blur amount of edges. The defocus map obtained is sparse. Bae et al. [9] extend this work and obtain a full defocus map from the sparse map using an interpolation method. Zhang and Cham [10] estimate the defocus map by fitting a well-parameterized model to edges and use the defocus map to perform single image refocusing. The inverse diffusion method [11] models the defocus blur as a heat diffusion process and uses the inhomogeneous inversion heat diffusion to estimate defocus blur at edge locations. Tai and Brown [12] use local contrast prior to measure the defocus at each pixel and then apply MRF propagation to refine the defocus map. In contrast, we estimate the defocus map in a different but effective way. The input image is re-blurred using a known Gaussian blur kernel and the ratio between the gradients of input and re-blurred images is calculated. We show that the blur amount at edge locations can be derived from the ratio. We then formulate the blur propagation as an optimization problem. By solving the optimization problem, we finally obtain a full defocus map.
We propose an efficient blur estimation method based on the Gaussian gradient ratio, and show that it is robust to noise, inaccurate edge location and interference from neighboring edges. Without any modification to cameras or using additional illumination, our method is able to obtain the defocus map of a single image captured by conventional camera. As shown in Fig. 1, our method can estimate the defocus map of the scene with fairly good extent of accuracy.
Section snippets
Defocus model
We estimate the defocus blur at edge locations. As step edge is the main edge type in natural images, we consider only step edges in this paper. An ideal step edge can be modeled aswhere u(x) is the step function. A and B are the amplitude and offset of the edge, respectively. Note that the edge is located at x=0.
We assume that focus and defocus obey the thin lens model [13]. When an object is placed at the focus distance df, all the rays from a point of the object will converge to
Defocus blur estimation
Fig. 3 shows the overview of our blur estimation method. An edge is re-blurred using a known Gaussian kernel. Then the ratio between the gradient magnitude of the step edge and its re-blurred version is calculated. The ratio is maximum at the edge location. Using the maximum value, we can compute the amount of the defocus blur at the edge location.
For convenience, we describe our blur estimation method for 1D case first and then extend it to 2D image. The gradient of the re-blurred edge is
Defocus map interpolation
Our defocus blur estimation method described in previous step produces a sparse defocus map . In this section, we provided a way to propagate the defocus blur estimates from edge locations to the entire image and obtain a full depth map d(x). To achieve this, we want to seek a defocus map d(x) which is close to the sparse defocus map at each edge location. Furthermore, we prefer the defocus blur discontinuities to be aligned with image edges. Edge-aware interpolation methods [16],
Experiments
We first test the robustness of our method on synthetic images. We synthesize a set of bar images, one of which is shown in Fig. 5(a). The blur amount of the edge increases linearly from 0 to 5. Under noise conditions, as shown in Fig. 5(d), our method can achieve a reliable estimation. And we also find that our blur estimation result of edges with smaller blur amounts is less affected by noise compared with those with larger blur amounts.
We also test our blur estimation on bar images with
Limitations and discussions
Blur texture ambiguity. One limitation of our blur estimation is that it cannot tell whether a blur edge is caused by defocus or blur texture (soft shadows or blur patterns) of the input image. For the latter case, the defocus value we obtained is a measurement of the sharpness of the edge. It is not the actual defocus value of the edge. This ambiguity may cause some artifacts in our result. One example is shown in Fig. 10. The region indicated by the white rectangle is actually blur texture of
Conclusion
In this paper, we show that the defocus map can be recovered from a single image. A new method is presented to estimate the blur amount at edge locations based on the Gaussian gradient ratio. A full defocus map is then produced using the matting interpolation. We show that our method is robust to noise, inaccurate edge location and interferences of neighboring edges and is able to generate more accurate defocus maps compared with existing methods. We also discuss the blur texture ambiguity
Acknowledgments
We thank the reviewers for helping to improve this paper. We thank Xiaopeng Zhang, Dong Guo and Ning Ye for their discussion and useful suggestion. This work is supported by NUS Research Grant #R-252-000-383-112.
Shaojie Zhuo received his B.S. degree in Computer Science from Fudan University, China, in 2005. He is now a Ph.D. candidate in Computer Science at National University of Singapore. His research interests include image processing, computational photography and computer vision.
References (20)
- et al.
A geometric approach to shape from defocus
IEEE Trans. Pattern Anal. Mach. Intell.
(2005) - et al.
Shape from defocus via diffusion
IEEE Trans. Pattern Anal. Mach. Intell.
(2008) A new sense for depth of field
IEEE Trans. Pattern Anal. Mach. Intell.
(1987)- et al.
Depth from diffusion
- et al.
Coded aperture pairs for depth from defocus
- et al.
Active refocusing of images and videos
ACM Trans. Graphics
(2007) - et al.
Image and depth from a conventional camera with a coded aperture
ACM Trans. Graphics
(2007) - et al.
Local scale control for edge detection and blur estimation
IEEE Trans. Pattern Anal. Mach. Intell.
(1998) - et al.
Defocus magnification
- et al.
Single image focus editing
Cited by (384)
An efficient autofocus method for microscope based on the improved first-order derivative Gaussian filtering operator
2024, Optics and Laser TechnologyAn edge detail enhancement strategy based on Fourier single-pixel imaging
2024, Optics and Lasers in EngineeringAn image defocus deblurring method based on gradient difference of boundary neighborhood
2023, Virtual Reality and Intelligent HardwarePerception-guided defocus blur detection based on SVD feature
2023, Image and Vision ComputingImproving defocus blur detection via adaptive supervision prior-tokens
2023, Image and Vision ComputingImproved shape-from-focus reconstruction for high dynamic range freedom surface
2023, Optics and Lasers in Engineering
Shaojie Zhuo received his B.S. degree in Computer Science from Fudan University, China, in 2005. He is now a Ph.D. candidate in Computer Science at National University of Singapore. His research interests include image processing, computational photography and computer vision.
Terence Sim obtained his Ph.D. from Carnegie Mellon University, M.Sc. from Stanford University, and S.B. from the Massachusetts Institute of Technology. He is currently an Assistant Professor at the School of Computing, National University of Singapore. His research interests include face recognition, biometrics, pattern recognition and computation photography.