Skip to main content
SpringerLink
Log in

Learning to restore deteriorated line drawing

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We propose a fully automatic approach to restore aged old line drawings. We decompose the task into two subtasks: the line extraction subtask, which aims to extract line fragments and remove the paper texture background, and the restoration subtask, which fills in possible gaps and deterioration of the lines to produce a clean line drawing. Our approach is based on a convolutional neural network that consists of two sub-networks corresponding to the two subtasks. They are trained as part of a single framework in an end-to-end fashion. We also introduce a new dataset consisting of manually annotated sketches by Leonardo da Vinci which, in combination with a synthetic data generation approach, allows training the network to restore deteriorated line drawings. We evaluate our method on challenging 500-year-old sketches and compare with existing approaches with a user study, in which it is found that our approach is preferred 72.7% of the time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. (Proc. of SIGGRAPH 2009) 28(3), 24:1–24:11 (2009)

    Google Scholar 

  2. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)

  3. Chiang, J.Y., Tue, S., Leu, Y.: A new algorithm for line image vectorization. Pattern Recognit. 31(10), 1541–1549 (1998)

    Article  Google Scholar 

  4. Cugini, U., Ferri, G., Mussio, P., Protti, M.: Pattern-directed restoration and vectorization of digitized engineering drawings. Comput. Graph. 8(4), 337–350 (1984)

    Article  Google Scholar 

  5. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  6. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (Proc. SIGGRAPH 2012) 31(4), 82:1–82:10 (2012)

    Google Scholar 

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199. Springer, Berlin (2014)

  8. Favreau, J.D., Lafarge, F., Bousseau, A.: Fidelity vs. simplicity: a global approach to line drawing vectorization. ACM Trans. Graph. (TOG) 35(4), 120 (2016)

    Article  Google Scholar 

  9. Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1823–1831 (2015)

  10. Han, C.C., Fan, K.C.: Skeleton generation of engineering drawings via contour matching. Pattern Recognit. 27(2), 261–275 (1994)

    Article  Google Scholar 

  11. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 107:1–107:14 (2017)

    Google Scholar 

  12. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (2015)

  13. Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)

  14. Janssen, R.D., Vossepoel, A.M.: Adaptive vectorization of line drawing images. Comput. Vis. Image Underst. 65(1), 38–56 (1997)

    Article  Google Scholar 

  15. Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. Image Process. 16(11), 2649–2661 (2007)

    Article  MathSciNet  Google Scholar 

  16. Li, C., Liu, X., Wong, T.T.: Deep extraction of manga structural lines. ACM Trans. Graph. 36(4), 117:1–117:12 (2017)

    Google Scholar 

  17. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  18. Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder–decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)

  19. Milanfar, P.: A tour of modern image filtering: new insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30(1), 106–128 (2013)

    Article  Google Scholar 

  20. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4(2), 460–489 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  21. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting. In: Computer Vision and Pattern Recognition (2016)

  22. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1–4), 259–268 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  23. Sasaki, K., Iizuka, S., Simo-Serra, E., Ishikawa, H.: Joint gap detection and inpainting of line drawings. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  24. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  25. Simo-Serra, E., Iizuka, S., Ishikawa, H.: Mastering sketching: adversarial augmentation for structured prediction. Trans. Graph. (TOG) 37(1), 11 (2018)

    Google Scholar 

  26. Simo-Serra, E., Iizuka, S., Sasaki, K., Ishikawa, H.: Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. (in Proc. of SIGGRAPH 2016) 35(4), 121 (2016)

    Google Scholar 

  27. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)

  28. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)

  29. Wang, Z., Yang, Y., Wang, Z., Chang, S., Yang, J., Huang, T.S.: Learning super-resolution jointly from external and internal examples. IEEE Trans. Image Process. 24(11), 4359–4371 (2015)

    Article  MathSciNet  Google Scholar 

  30. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  31. Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, pp. 341–349 (2012)

  32. Zeiler, M.D.: Adadelta: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 (2012)

Download references

Funding

This work was partially supported by JST ACT-I Grant No. JPMJPR16U3, JST ACT-I, Grant No. JPMJPR16UD, and JST CREST Grant No. JPMJCR14D1.

Author information

Authors and Affiliations

  1. Waseda University, Tokyo, Japan

    Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra & Hiroshi Ishikawa

Authors
  1. Kazuma Sasaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Satoshi Iizuka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edgar Simo-Serra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hiroshi Ishikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kazuma Sasaki.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sasaki, K., Iizuka, S., Simo-Serra, E. et al. Learning to restore deteriorated line drawing. Vis Comput 34, 1077–1085 (2018). https://doi.org/10.1007/s00371-018-1528-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1528-4

Keywords

Search

Navigation