This paper presents a novel inpainting method based on structure estimation. The method first estimates an initial image that captures the rough structure and colors in the missing region. This image is generated by probabilistically estimating the gradient within the missing region based on edge segments intersecting its boundary, and then by flooding the colors on the boundary into the missing region. The color flooding is formulated as an energy minimization problem, and is efficiently optimized by the conjugate gradient method. Finally, by locally replacing the missing region with local patches similar to both the adjacent patches and the initial image, the inpainted image is synthesized. The initial image not only serves as a guide to ensure the underlying structure is preserved, but also allows the patch selection process to be carried out in a greedy manner, which leads to substantial speedup. Experimental results show the proposed method is capable of preserving the underlying structure in the missing region, while achieving more than 5 times faster computational speed than the state-of-the-art inpainting method. Subjective evaluation of image quality also shows the proposed method outperforms the previous methods.