Skip to main content
SpringerLink
Log in

Image palette: painting style transfer via brushstroke control synthesis

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As one kind of technology of style transfer, painting style transfer can be used to render the sample images with a specific art style. With this technology, we can render the target images in the same style as the samples after some computation. We present a new approach of painting style transfer in which such a style transfer artwork is done by simulating the process of creation. We take the sample as a palette where users can select arbitrary contours or textures as the input brush strokes. We then analyze the style feature of the brush strokes and use this feature for synthesis and style transfer along the stroke curves learned from the specified area in target images or target 3D models to get the same painting style as the samples. Based on this approach, we also design and realize the corresponding painting system. The results show that the users can get a style-transferred personalized target image just by using the given sample images with the least interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Ashikhmin M (2001) Synthesizing natural textures. In: Proceedings of the 2001 symposium on interactive 3D graphics, ser. I3D ’01. New York, NY, USA: ACM, pp 217–226. doi:10.1145/364338.364405

  2. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3):24:1–24:11. doi:10.1145/1531326.1531330

    Article  Google Scholar 

  3. Bellman R (1956) Dynamic programming and lagrange multipliers. Proc Natl Acad Sci USA 42(10):767

    Article  MathSciNet  MATH  Google Scholar 

  4. Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11):1222–1239. doi:10.1109/34.969114

    Article  Google Scholar 

  5. Brito A (2008) Blender 3D: architecture, buildings, and scenery: create photorealistic 3D architectural visualizations of buildings, interiors, and environmental scenery. Packt Publishing Ltd

  6. DiVerdi S, Krishnaswamy A, Mech R, Ito D (2013) Painting with polygons: a procedural watercolor engine. IEEE Trans Vis Comput Graph 19(5):723–735

    Article  Google Scholar 

  7. Gooch B, Coombe G, Shirley P (2002) Artistic vision: painterly rendering using computer vision techniques. In: Proceedings of the 2Nd international symposium on non-photorealistic animation and rendering, ser. NPAR ’02. New York, NY, USA: ACM, pp 83–ff. doi:10.1145/508530.508545

  8. Hays J, Essa I (2004) Image and video based painterly animation. In: Proceedings of the 3rd international symposium on non-photorealistic animation and rendering, ser. NPAR ’04. New York, NY, USA: ACM, pp 113–120. doi:10.1145/987657.987676

  9. He K, Sun J (2012) Statistics of patch offsets for image completion. In: Proceedings of the 12th European Conference on Computer Vision - Volume Part II, ser. ECCV’12. Springer, Berlin, Heidelberg, pp 16–29. doi:10.1007/978-3-642-33709-3_2

  10. Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’98. New York, NY, USA: ACM, pp 453–460. doi:10.1145/280814.280951

  11. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH Image analogies. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’01. New York, NY, USA: ACM, pp 327–340. doi:10.1145/383259.3832952001

  12. Huang H, Zang Y, Li C-F (2010) Example-based painting guided by color features. Vis. Comput. 26(6-8):933–942. doi:10.1007/s00371-010-0498-y

    Article  Google Scholar 

  13. Kang H, Lee S, Chui CK (2007) Coherent line drawing. In: Proceedings of the 5th international symposium on non-photorealistic animation and rendering. ACM, pp 43–50

  14. Lasram A, Lefebvre S (2012) Parallel patch-based texture synthesis. In: Proceedings of the Fourth ACM SIGGRAPH / Eurographics Conference on High-Performance Graphics, ser. EGGH-HPG’12, pp 115–124. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association. doi:10.2312/EGGH/HPG12/115-124

  15. Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, ser. NPAR ’10. New York, NY, USA: ACM, pp 43–48. doi:10.1145/1809939.1809945

  16. Lefebvre S, Hoppe H (2006) Appearance-space texture synthesis. In: ACM SIGGRAPH 2006 Papers, ser. SIGGRAPH ’06. New York, NY, USA: ACM, pp 541–548. doi:10.1145/1179352.1141921

  17. Lim JJ, Zitnick CL, Dollr P (2013) Sketch tokens: a learned mid-level representation for contour and object detection. In: CVPR. IEEE, pp 3158–3165. http://dblp.uni-trier.de/db/conf/cvpr/cvpr2013.html

  18. Lu J, Barnes C, DiVerdi S, Finkelstein A (2013) Realbrush: Painting with examples of physical media. ACM Trans. Graph. 32(4):117:1–117:12. doi:10.1145/2461912.2461998

    Article  Google Scholar 

  19. Lu J, Sander PV, Finkelstein A (2010) Interactive painterly stylization of images, videos and 3D animations. In: Proceedings of i3d 2010

  20. Lukáč M, Fišer J, Bazin J-C, Jamriška O, Sorkine-Hornung A, Sýkora D (2013) Painting by feature: texture boundaries for example-based image creation. ACM Trans. Graph. 32(4):116:1–116:8. doi:10.1145/2461912.2461956

    Google Scholar 

  21. Miao Z, Zhang Y, Zheng Z, Sun Z Image palette: brushstroke synthesis-based style transfer, pp 55–60. http://diglib.eg.org/EG/DL/PE/PG/PG2014short/055-060.pdf

  22. Nehab D, Velho L (2002) Multiscale moment-based painterly rendering. In: Proceedings of the 15th Brazilian symposium on computer graphics and image processing, ser. SIBGRAPI ’02. Washington, DC, USA: IEEE Computer Society, pp 244–251. http://dl.acm.org/citation.cfm?id=646016.677978

  23. NPAR ’02 (2002) Proceedings of the 2Nd international symposium on non-photorealistic animation and rendering. ACM, New York, pp 434–020

    Google Scholar 

  24. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7):971–987. doi:10.1109/TPAMI.2002.1017623

    Article  MATH  Google Scholar 

  25. Ojala T, Pietikainen M, Harwood D (1994) Performance evaluation of texture measures with classification based on kullback discrimination of distributions, pp A:582–585

  26. Olsen SC, Maxwell BA, Gooch B (2005) Interactive vector fields for painterly rendering. In: GI ’05: Proceedings of the 2005 conference on graphics interface. School of computer science, University of Waterloo, Waterloo, Ontario, Canada: Canadian Human-Computer Communications Society, pp 241–247

  27. Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput. Graph. Appl. 21(5):34–41. doi:10.1109/38.946629

    Article  Google Scholar 

  28. Ritter L, Li W, Curless B, Agrawala M, Salesin D (2006) Painting with texture. In: Proceedings of the 17th eurographics conference on rendering techniques, ser. EGSR’06. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association, pp 371–376. doi:10.2312/EGWR/EGSR06/371-376

  29. Simakov D, Caspi Y, Shechtman E, Irani M (2008) Summarizing visual data using bidirectional similarity. In: CVPR. IEEE Computer Society. http://dblp.uni-trier.de/db/conf/cvpr/cvpr2008.html

  30. Sun J, Yuan L, Jia J, Shum H-Y (2005) Image completion with structure propagation. ACM Trans. Graph. 24(3):861–868. doi:10.1145/1073204.1073274

    Article  Google Scholar 

  31. Wang B, Wang W, Yang H, Sun J-G (2004) Efficient example-based painting and synthesis of 2d directional texture. IEEE Trans. Vis. Comput. Graph. 10 (3):266–277. http://dblp.uni-trier.de/db/journals/tvcg/tvcg10.html

    Article  Google Scholar 

  32. Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3):463–476. doi:10.1109/TPAMI.2007.60

    Article  Google Scholar 

  33. Zang Y, Huang H, Li C-F (2013) Stroke style analysis for painterly rendering. J. Comput. Sci. Technol.:762–775

  34. Zeng K, Zhao M, Xiong C, Zhu S-C (2009) From image parsing to painterly rendering. ACM Trans. Graph. 29(1):2:1–2:11. doi:10.1145/1640443.1640445

    Article  Google Scholar 

  35. Zhang E, Hays J, Turk G (2007) Interactive tensor field design and visualization on surfaces. IEEE Trans Vis Comput Graph 13(1):94–107

    Article  Google Scholar 

  36. Zhao M, Zhu S-C (2011) Customizing painterly rendering styles using stroke processes. In: NPAR ’11: Proceedings Of the ACM SIGGRAPH/eurographics symposium on non-photorealistic animation and rendering. New York, NY, USA: ACM, pp 137–146

Download references

Acknowledgments

We would like to thank al l anonymous reviewers for their constructive comments. This research has been supported by the National Science Foundation of China (61321491, 61100110, 61272219) and the Science and Technology Program of Jiangsu Province (BY2012190, BY2013072-04).

Author information

Authors and Affiliations

  1. Nanjing University, Nanjing, China

    Zheng Miao, Yan Zhang, Zhibin Zheng & Zhengxing Sun

Authors
  1. Zheng Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhibin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhengxing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheng Miao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miao, Z., Zhang, Y., Zheng, Z. et al. Image palette: painting style transfer via brushstroke control synthesis. Multimed Tools Appl 76, 7989–8010 (2017). https://doi.org/10.1007/s11042-016-3408-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3408-0

Keywords

Search

Navigation