Skip to main content
SpringerLink
Log in

SemiPainter: Learning to Draw Semi-realistic Paintings from the Manga Line Drawings and Flat Shadow

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13443))

Included in the following conference series:

Abstract

Semi-realistic paintings are widely used in game concept art, film posters and animation, etc. However, none of existing work can reproduce this type of art. We propose the SemiPainter to generate semi-realistic paintings from line drawings based on a deep learning network. We divide the complex work into two parts, generating the shadow from a line drawing and a simple flat shadow image at first, then generating flat color from line drawings. We merge these two paintings and get the final result. We use two networks with U-Net structure to realize these two stages of work. In order to ensure the global consistency of picture illumination, we add the lighting direction to the shadow network. The color of each part of the painting is uncertain, so color hints from users is provided when coloring, so that the network can realize controllable colorization. To the best of our knowledge, this is the first framework for reproducing semi-realistic paintings.

This work was partly supported by the Natural Science Foundation of China under grant no. 62072328.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bahng, H., et al.: Coloring with words: guiding image colorization through text-based palette generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 443–459. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01258-8_27

    Chapter  Google Scholar 

  2. Chen, S.Y., et al.: Active colorization for cartoon line drawings. IEEE Trans. Vis. Comput. Graph. 28(2), 1198–1208 (2022)

    Article  Google Scholar 

  3. Chen, Y., Lai, Y.K., Liu, Y.J.: CartoonGAN: generative adversarial networks for photo cartoonization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9465–9474 (2018)

    Google Scholar 

  4. Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision (ICCV), pp. 415–423 (2015)

    Google Scholar 

  5. Deshpande, A., Lu, J., Yeh, M.C., Chong, M.J., Forsyth, D.: Learning diverse image colorization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2877–2885 (2017)

    Google Scholar 

  6. Fang, F., Wang, T., Zeng, T., Zhang, G.: A superpixel-based variational model for image colorization. IEEE Trans. Vis. Comput. Graph. 26(10), 2931–2943 (2020)

    Article  Google Scholar 

  7. Furusawa, C., Hiroshiba, K., Ogaki, K., Odagiri, Y.: Comicolorization: semi-automatic manga colorization. In: SIGGRAPH Asia Technical Briefs, pp. 1–4 (2017)

    Google Scholar 

  8. Gao, Z., onetsuji, T.Y., Takamura, T., Matsuoka, T., Naradowsky, J.: Automatic illumination effects for 2D characters. In: NIPS Workshop on Machine Learning for Creativity and Design (2018)

    Google Scholar 

  9. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016)

    Google Scholar 

  10. He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37(4), 1–16 (2018)

    Google Scholar 

  11. Hensman, P., Aizawa, K.: cGAN-based manga colorization using a single training image. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 03, pp. 72–77 (2017)

    Google Scholar 

  12. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510–1519 (2017)

    Google Scholar 

  13. Huang, Z., Zhou, S., Heng, W.: Learning to paint with model-based deep reinforcement learning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8708–8717 (2019)

    Google Scholar 

  14. Hudon, M., Grogan, M., Pagés, R., Smolić, A.: Deep normal estimation for automatic shading of hand-drawn characters. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 246–262. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_20

    Chapter  Google Scholar 

  15. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017)

    Google Scholar 

  16. Jin, Y., Li, P., Sheng, B., Nie, Y., Kim, J., Wu, E.: SRNPD: spatial rendering network for pencil drawing stylization. Comput. Animation Virtual Worlds 30(3–4), e1890 (2019)

    Google Scholar 

  17. Jin, Y., Sheng, B., Li, P., Chen, C.L.P.: Broad colorization. IEEE Trans. Neural Netw. Learn. Syst. 32(6), 2330–2343 (2021)

    Article  Google Scholar 

  18. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  19. Kolkin, N., Salavon, J., Shakhnarovich, G.: Style transfer by relaxed optimal transport and self-similarity. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10043–10052 (2019)

    Google Scholar 

  20. Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_43

    Chapter  Google Scholar 

  21. Li, J., Gao, N., Shen, T., Zhang, W., Mei, T., Ren, H.: SketchMan: learning to create professional sketches, pp. 3237–3245. Association for Computing Machinery, New York (2020)

    Google Scholar 

  22. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. ACM Trans. Graph. 36(4), 1–15 (2017)

    Article  Google Scholar 

  23. Lin, D., Wang, Y., Xu, G., Li, J., Fu, K.: Transform a simple sketch to a Chinese painting by a multiscale deep neural network. Algorithms 11(1), 1–18 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  24. Liu, S.: Two decades of colorization and decolorization for images and videos. arXiv: 2204.13322 (2022)

  25. Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recogn. Lett. 33(13), 1673–1681 (2012)

    Article  Google Scholar 

  26. Liu, S., Zhang, X.: Image colorization based on texture map. J. Electron. Imaging 22(1), 1–9 (2013)

    Article  Google Scholar 

  27. Liu, S., Zhu, T.: Structure-guided arbitrary style transfer for artistic image and video. IEEE Trans. Multimed. 24, 1299–1312 (2022)

    Article  Google Scholar 

  28. Liu, S., et al.: Paint transformer: feed forward neural painting with stroke prediction. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6578–6587 (2021)

    Google Scholar 

  29. Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)

    Article  Google Scholar 

  30. Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5873–5881 (2019)

    Google Scholar 

  31. Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6836–6845 (2017)

    Google Scholar 

  32. Sheng, B., Li, P., Gao, C., Ma, K.L.: Deep neural representation guided face sketch synthesis. IEEE Trans. Vis. Comput. Graph. 25(12), 3216–3230 (2019)

    Article  Google Scholar 

  33. Su, W., Du, D., Yang, X., Zhou, S., Fu, H.: Interactive sketch-based normal map generation with deep neural networks. Proc. ACM Comput. Graph. Interact. Tech. 1(1), 1–17 (2018)

    Article  Google Scholar 

  34. Xiao, C., et al.: Example-based colourization via dense encoding pyramids. Comput. Graph. Forum 39(1), 20–33 (2020)

    Article  Google Scholar 

  35. Xue, A.: End-to-end Chinese landscape painting creation using generative adversarial networks. arXiv: 2011.05552 (2020)

  36. Ye, R.T., Wang, W.L., Chen, J.C., Lin, K.W.: Interactive anime sketch colorization with style consistency via a deep residual neural network. In: 2019 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp. 1–5 (2019)

    Google Scholar 

  37. Zan, T.: Paintschainer (2017). https://petalica-paint.pixiv.dev/index_zh

  38. Zhang, L., Jiang, J., Ji, Y., Liu, C.: SmartShadow: artistic shadow drawing tool for line drawings. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5371–5380 (2021)

    Google Scholar 

  39. Zhang, L., Li, C., Wong, T.T., Ji, Y., Liu, C.: Two-stage sketch colorization. ACM Trans. Graph. 37(6), 1–14 (2018)

    Google Scholar 

  40. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40

    Chapter  Google Scholar 

  41. Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36(4), 1–11 (2017)

    Google Scholar 

  42. Zheng, Q., Li, Z., Bargteil, A.: Learning to shadow hand-drawn sketches. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7434–7443 (2020)

    Google Scholar 

  43. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)

    Google Scholar 

  44. Zou, Z., Shi, T., Qiu, S., Yuan, Y., Shi, Z.: Stylized neural painting. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15684–15693 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China

    Keyue Fan, Shiguang Liu & Wenhuan Lu

Authors
  1. Keyue Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiguang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenhuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shiguang Liu or Wenhuan Lu .

Editor information

Editors and Affiliations

  1. University of Geneva, Geneva, Switzerland

    Nadia Magnenat-Thalmann

  2. Bournemouth University, Poole, UK

    Jian Zhang

  3. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  4. University of Crete, Heraklion, Greece

    George Papagiannakis

  5. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  6. Swiss Federal Institute of Technology, Lausanne, Switzerland

    Daniel Thalmann

  7. University of Calgary, Calgary, AB, Canada

    Marina Gavrilova

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fan, K., Liu, S., Lu, W. (2022). SemiPainter: Learning to Draw Semi-realistic Paintings from the Manga Line Drawings and Flat Shadow. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2022. Lecture Notes in Computer Science, vol 13443. Springer, Cham. https://doi.org/10.1007/978-3-031-23473-6_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23473-6_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23472-9

  • Online ISBN: 978-3-031-23473-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation