Abstract
Sketch Image Style Transfer aims to stylize the sketch images from given art images to make them look like the same artistic styles while still persevering the original sketch contents. Previous methods generally disentangle the content and style of reference image and transfer the style to sketch image. However, the textures or the painting strokes of the reference art image could be a part of content as well as style. It is difficult to decide how much it should be involved for sketch image style transferring. In this paper, we propose a novel sketch image style transfer method to use the sketch density to control it, reaching the common sense that simpler sketch images own richer textures of reference style images after stylization and otherwise the opposite. The proposed model is built upon the general content and style encoding and fused decoding architecture, but adds sketch density extraction to obtain the density level of input sketch image, and uses it to control texture information during transferring. With these special designing, our model is more flexible for inputting multi-density sketch images and reasonable for deciding the degree to transfer the texture styles of reference image. Experimental results on different datasets demonstrate the effectiveness and superiority of our method for sketch image style transfer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
An, J., Huang, S., et al.: ArtFlow: unbiased image style transfer via reversible neural flows. In: CVPR (2021)
Liu, B., Zhu, Y., et al.: A self-supervised sketch-to-image synthesis. In: AAAI (2021)
Chen, H., Wang, Z., et al.: Artistic style transfer with internal-external learning and contrastive learning. In: NIPS (2021)
Chen, S.Y., Su, W., et al.: Deep generation of face images from sketches. ACMTOG 39(4) (2020)
Chen, W., Hays, J.: SketchyGAN: towards diverse and realistic sketch to image synthesis. In: CVPR (2018)
Deng, Y., Tang, F., et al.: Stytr\(\hat{\,}\)2: unbiased image style transfer with transformers. In: CVPR (2022)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR (2016)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: NIPS (2017)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR (2020)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)
Lee, C.H., Liu, Z., Wu, L., Luo, P.: MaskGAN: towards diverse and interactive facial image manipulation. In: CVPR (2020)
Liao, J., Yao, Y., et al.: Visual attribute transfer through deep image analogy. ACMTOG 36(4), 120 (2017)
Liu, B., Song, K., Zhu, Y., Elgammal, A.: Sketch-to-art: synthesizing stylized art images from sketches. In: ACCV (2020)
Qin, X., Fan, D.P., et al.: Boundary-aware segmentation network for mobile and web applications. arXiv preprint arXiv:2101.04704 (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Simo-Serra, E., Iizuka, S., Ishikawa, H.: Mastering sketching: adversarial augmentation for structured prediction. ACMTOG 37(1), 1–13 (2018)
Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: NIPS (2015)
Sun, J., Xu, Z., Shum, H.Y.: Gradient profile prior and its applications in image super-resolution and enhancement. TIP 20(6), 1529–1542 (2010)
Wang, S.Y., Bau, D., Zhu, J.Y.: Sketch your own GAN. In: CVPR (2021)
Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)
Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.H.: Object contour detection with a fully convolutional encoder-decoder network. In: CVPR (2016)
Acknowledgement
This work was partly supported by the special project of “Tibet Economic and Social Development and Plateau Scientific Research Co-construction Innovation Foundation” of Wuhan University of Technology & Tibet University (No. lzt2021008).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yu, S., Wang, H., Zhu, A. (2023). Sketch Image Style Transfer Based on Sketch Density Controlling. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Communications in Computer and Information Science, vol 1794. Springer, Singapore. https://doi.org/10.1007/978-981-99-1648-1_10
Download citation
DOI: https://doi.org/10.1007/978-981-99-1648-1_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-1647-4
Online ISBN: 978-981-99-1648-1
eBook Packages: Computer ScienceComputer Science (R0)