Elsevier

Pattern Recognition

Volume 44, Issue 9, September 2011, Pages 1852-1858
Pattern Recognition

Defocus map estimation from a single image

https://doi.org/10.1016/j.patcog.2011.03.009Get rights and content

Abstract

In this paper, we address the challenging problem of recovering the defocus map from a single image. We present a simple yet effective approach to estimate the amount of spatially varying defocus blur at edge locations. The input defocused image is re-blurred using a Gaussian kernel and the defocus blur amount can be obtained from the ratio between the gradients of input and re-blurred images. By propagating the blur amount at edge locations to the entire image, a full defocus map can be obtained. Experimental results on synthetic and real images demonstrate the effectiveness of our method in providing a reliable estimation of the defocus map.

Research highlights

► We address the challenging problem of defocus estimation from a single image. ► The defocus blur is estimated at edge locations from the gradient ratio between tlie original and re-blurred input image. ► A hill defocus map is obtained by propagating the blur amount at edges to the entire image using soft matting. ► We also discuss the ambiguities in defocus estimation and the relationship between defocus map and depth.

Introduction

Defocus estimation plays an important role in many computer vision and computer graphics applications including depth estimation, image quality assessment, image deblurring and refocusing. Conventional methods for defocus estimation have relied on multiple images [1], [2], [3], [4]. A set of images of the same scene are captured using multiple focus settings. Then the defocus is measured during a implicit or explicit deblurring process. Recently, image pairs captured using coded aperture cameras [5] are used for better defocus blur measure and all-focused image recovery. However, these methods suffer from the occlusion problem and require the scene to be static, which limits their applications in practice.

In very specific settings, several methods have been proposed to recover defocus map from a single image. Active illumination methods [6] project sparse grid dots onto the scene and the defocus blur of those dots is measured by comparing them with calibrated images. Then the defocus measure can be used to estimate the depth of a scene. The coded aperture method [7] changes the shape of camera aperture to make defocus deblurring more reliable. A defocus map and an all-focused image can be obtained after deconvolution using calibrated blur kernels. These methods require additional illumination or camera modification to obtain a defocus map from a single image.

In this paper, we focus on a more challenging problem of recovering the defocus map from a single image captured by an uncalibrated conventional camera. Elder and Zucker [8] used the first- and second-order derivatives of the input image to find the locations and the blur amount of edges. The defocus map obtained is sparse. Bae et al. [9] extend this work and obtain a full defocus map from the sparse map using an interpolation method. Zhang and Cham [10] estimate the defocus map by fitting a well-parameterized model to edges and use the defocus map to perform single image refocusing. The inverse diffusion method [11] models the defocus blur as a heat diffusion process and uses the inhomogeneous inversion heat diffusion to estimate defocus blur at edge locations. Tai and Brown [12] use local contrast prior to measure the defocus at each pixel and then apply MRF propagation to refine the defocus map. In contrast, we estimate the defocus map in a different but effective way. The input image is re-blurred using a known Gaussian blur kernel and the ratio between the gradients of input and re-blurred images is calculated. We show that the blur amount at edge locations can be derived from the ratio. We then formulate the blur propagation as an optimization problem. By solving the optimization problem, we finally obtain a full defocus map.

We propose an efficient blur estimation method based on the Gaussian gradient ratio, and show that it is robust to noise, inaccurate edge location and interference from neighboring edges. Without any modification to cameras or using additional illumination, our method is able to obtain the defocus map of a single image captured by conventional camera. As shown in Fig. 1, our method can estimate the defocus map of the scene with fairly good extent of accuracy.

Section snippets

Defocus model

We estimate the defocus blur at edge locations. As step edge is the main edge type in natural images, we consider only step edges in this paper. An ideal step edge can be modeled asf(x)=Au(x)+B,where u(x) is the step function. A and B are the amplitude and offset of the edge, respectively. Note that the edge is located at x=0.

We assume that focus and defocus obey the thin lens model [13]. When an object is placed at the focus distance df, all the rays from a point of the object will converge to

Defocus blur estimation

Fig. 3 shows the overview of our blur estimation method. An edge is re-blurred using a known Gaussian kernel. Then the ratio between the gradient magnitude of the step edge and its re-blurred version is calculated. The ratio is maximum at the edge location. Using the maximum value, we can compute the amount of the defocus blur at the edge location.

For convenience, we describe our blur estimation method for 1D case first and then extend it to 2D image. The gradient of the re-blurred edge isi1(x)

Defocus map interpolation

Our defocus blur estimation method described in previous step produces a sparse defocus map d^(x). In this section, we provided a way to propagate the defocus blur estimates from edge locations to the entire image and obtain a full depth map d(x). To achieve this, we want to seek a defocus map d(x) which is close to the sparse defocus map d^(x) at each edge location. Furthermore, we prefer the defocus blur discontinuities to be aligned with image edges. Edge-aware interpolation methods [16],

Experiments

We first test the robustness of our method on synthetic images. We synthesize a set of bar images, one of which is shown in Fig. 5(a). The blur amount of the edge increases linearly from 0 to 5. Under noise conditions, as shown in Fig. 5(d), our method can achieve a reliable estimation. And we also find that our blur estimation result of edges with smaller blur amounts is less affected by noise compared with those with larger blur amounts.

We also test our blur estimation on bar images with

Limitations and discussions

Blur texture ambiguity. One limitation of our blur estimation is that it cannot tell whether a blur edge is caused by defocus or blur texture (soft shadows or blur patterns) of the input image. For the latter case, the defocus value we obtained is a measurement of the sharpness of the edge. It is not the actual defocus value of the edge. This ambiguity may cause some artifacts in our result. One example is shown in Fig. 10. The region indicated by the white rectangle is actually blur texture of

Conclusion

In this paper, we show that the defocus map can be recovered from a single image. A new method is presented to estimate the blur amount at edge locations based on the Gaussian gradient ratio. A full defocus map is then produced using the matting interpolation. We show that our method is robust to noise, inaccurate edge location and interferences of neighboring edges and is able to generate more accurate defocus maps compared with existing methods. We also discuss the blur texture ambiguity

Acknowledgments

We thank the reviewers for helping to improve this paper. We thank Xiaopeng Zhang, Dong Guo and Ning Ye for their discussion and useful suggestion. This work is supported by NUS Research Grant #R-252-000-383-112.

Shaojie Zhuo received his B.S. degree in Computer Science from Fudan University, China, in 2005. He is now a Ph.D. candidate in Computer Science at National University of Singapore. His research interests include image processing, computational photography and computer vision.

References (20)

  • P. Favaro et al.

    A geometric approach to shape from defocus

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2005)
  • P. Favaro et al.

    Shape from defocus via diffusion

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2008)
  • A.P. Pentland

    A new sense for depth of field

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1987)
  • C. Zhou et al.

    Depth from diffusion

  • C. Zhou et al.

    Coded aperture pairs for depth from defocus

  • F. Moreno-Noguer et al.

    Active refocusing of images and videos

    ACM Trans. Graphics

    (2007)
  • A. Levin et al.

    Image and depth from a conventional camera with a coded aperture

    ACM Trans. Graphics

    (2007)
  • J. Elder et al.

    Local scale control for edge detection and blur estimation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1998)
  • S. Bae et al.

    Defocus magnification

  • W. Zhang et al.

    Single image focus editing

There are more references available in the full text version of this article.

Cited by (384)

View all citing articles on Scopus

Shaojie Zhuo received his B.S. degree in Computer Science from Fudan University, China, in 2005. He is now a Ph.D. candidate in Computer Science at National University of Singapore. His research interests include image processing, computational photography and computer vision.

Terence Sim obtained his Ph.D. from Carnegie Mellon University, M.Sc. from Stanford University, and S.B. from the Massachusetts Institute of Technology. He is currently an Assistant Professor at the School of Computing, National University of Singapore. His research interests include face recognition, biometrics, pattern recognition and computation photography.

View full text