Skip to main content
SpringerLink
Log in

Learning Reaction-Diffusion Models for Image Inpainting

  • Conference paper
  • First Online:
Pattern Recognition (DAGM 2015)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9358))

Included in the following conference series:

Abstract

In this paper we present a trained diffusion model for image inpainting based on the structural similarity measure. The proposed diffusion model uses several parametrized linear filters and influence functions. Those parameters are learned in a loss based approach, where we first perform a greedy training before conducting a joint training to further improve the inpainting performance. We provide a detailed comparison to state-of-the-art inpainting algorithms based on the TUM-image inpainting database. The experimental results show that the proposed diffusion model is efficient and achieves superior performance. Moreover, we also demonstrate that the proposed method has a texture preserving property, that makes it stand out from previous PDE based methods.

This research was supported by the FWF-START project Bilevel optimization for Computer Vision, No. Y729 and the Vision\(+\) project Integrating visual information with independent knowledge, No. 836630.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    \( {\mathbf {K}_i^{t}}^\top \phi _i^t(\mathbf {K}_i^t\mathbf {u}_{t-1})\) can be rewritten as \(\mathbf {\overline{k}}_i^t*\phi _i^t(\mathbf {K}_i^t\mathbf {u}_{t-1}) \).

  2. 2.

    Used solver: http://www.cs.toronto.edu/~liam/software.shtml.

References

  1. Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. In: proceedings of CVPR, vol. 2, pp. II-707-12 (2003)

    Google Scholar 

  2. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Conference on Computer graphics and interactive techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)

    Google Scholar 

  3. Bugeau, A., Bertalmio, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. Image Process. 19(10), 2634–2645 (2010)

    Article  MathSciNet  Google Scholar 

  4. Cao, F., Gousseau, Y., Masnou, S., Pérez, P.: Geometrically guided exemplar-based inpainting. SIAM J. Img. Sci. 4(4), 1143–1179 (2011)

    Article  MathSciNet  Google Scholar 

  5. Caselles, V.: Exemplar-based image inpainting and applications. SIAM News 44(10), 1–3 (2011)

    Google Scholar 

  6. Chan, T.F., Shen, J.: Local inpainting models and TV inpainting. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)

    MathSciNet  Google Scholar 

  7. Chan, T.F., Shen, J.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)

    Article  Google Scholar 

  8. Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression (2014). arXiv preprint arXiv:1401.4112

  9. Chen, Y., Ranftl, R., Pock, T.: Insights into analysis operator learning: from patch-based sparse models to higher order MRFs. Image Process. 23(3), 1060–1072 (2014)

    Article  MathSciNet  Google Scholar 

  10. Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: Proceeding of CVPR (2015)

    Google Scholar 

  11. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. Trans. Img. Proc. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  12. Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: proceeding of ICCV, vol. 2, pp. 1033–1038 (1999)

    Google Scholar 

  13. Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford-Shah-Euler image model. Eur. J. Appl. Math. 13(04), 353–370 (2002)

    Article  MathSciNet  Google Scholar 

  14. Facciolo, G., Arias, P., Caselles, V., Sapiro, G.: Exemplar-based interpolation of sparsely sampled images. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 331–344. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  15. Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. JMIV 31(2–3), 255–269 (2008)

    Article  MathSciNet  Google Scholar 

  16. Getreuer, P.: Total variation inpainting using split bregman. Image Process. On Line 2, 147–157 (2012)

    Article  Google Scholar 

  17. Grossauer, H.: A combined PDE and texture synthesis approach to inpainting. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3022, pp. 214–224. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  18. Hel-Or, Y., Shaked, D.: A discriminative approach for wavelet denoising. IEEE Trans. Image Process. 17(4), 443–457 (2008)

    Article  MathSciNet  Google Scholar 

  19. Herling, J., Broll, W.: Pixmix: A real-time approach to high-quality diminished reality. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp. 141–150. IEEE (2012)

    Google Scholar 

  20. Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)

    Google Scholar 

  21. Kokaram, A.C., Morris, R.D., Fitzgerald, W.J., Rayner, P.J.: Interpolation of missing data in image sequences. Image Process. 4(11), 1509–1519 (1995)

    Article  Google Scholar 

  22. Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. Trans. Img. Proc. 16(11), 2649–2661 (2007)

    Article  MathSciNet  Google Scholar 

  23. Kong, X., Li, K., Yang, Q., Wenyin, L., Yang, M.H.: A new image quality metric for image auto-denoising. In: Proceeding of ICCV, pp. 2888–2895. IEEE (2013)

    Google Scholar 

  24. Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: poceeding of ICCV, pp. 305–312 (2003)

    Google Scholar 

  25. Liu, A., Lin, W., Narwaria, M.: Image quality assessment based on gradient similarity. Image Process. 21(4), 1500–1512 (2012)

    Article  MathSciNet  Google Scholar 

  26. Liu, D., Sun, X., Wu, F., Li, S., Zhang, Y.Q.: Image compression with edge-based inpainting. Circuits Syst. Video Technol. 17(10), 1273–1287 (2007)

    Article  Google Scholar 

  27. Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989)

    Article  MathSciNet  Google Scholar 

  28. Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceeding of ICIP, vol. 3, pp. 259–263 (1998)

    Google Scholar 

  29. Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42(5), 577–685 (1989)

    Article  MathSciNet  Google Scholar 

  30. Nikolaidis, N., Pitas, I.: Digital image processing in painting restoration and archiving. In: proceeding of ICCV, vol. 1, pp. 586–589. IEEE (2001)

    Google Scholar 

  31. Peter, P., Weickert, J.: Compressing images with diffusion- and exemplar-based inpainting. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 154–165. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  32. Rehman, A., Wang, Z.: Ssim-based non-local means image denoising. In: proceeding of ICIP, pp. 217–220. IEEE (2011)

    Google Scholar 

  33. Roth, S., Black, M.: Fields of experts: a framework for learning image priors. In: proceeding of CVPR, vol. 2, pp. 860–867 (2005)

    Google Scholar 

  34. Roth, S., Black, M.: Steerable random fields. In: proceeding of ICCV, pp. 1–8 (2007)

    Google Scholar 

  35. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60(1–4), 259–268 (1992)

    Article  MathSciNet  Google Scholar 

  36. Schmidt, U., Gao, Q., Roth, S.: A generative perspective on MRFs in low-level vision. In: proceeding of CVPR, pp. 1751–1758. IEEE (2010)

    Google Scholar 

  37. Tiefenbacher, P., Bogischef, V., Merget, D., Rigoll, G.: Subjective and objective evaluation of image inpainting quality. In: proceeding of ICIP. IEEE (2015)

    Google Scholar 

  38. TUM-image inpainting database. http://www.mmk.ei.tum.de/tumiid/

  39. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  40. Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. Image Process. 19(5), 1153–1165 (2010)

    Article  MathSciNet  Google Scholar 

  41. Zhu, S.C., Mumford, D.: Prior learning and gibbs reaction-diffusion. Pattern Anal. Mach. Intell. 19(11), 1236–1250 (1997)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria

    Wei Yu, Stefan Heber & Thomas Pock

  2. Safety & Security Department, AIT Austrian Institute of Technology, Graz, Austria

    Thomas Pock

Authors
  1. Wei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Heber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Pock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Yu .

Editor information

Editors and Affiliations

  1. Institute of Computer Science III, University of Bonn, Bonn, Germany

    Juergen Gall

  2. MPI for Intelligent Systems, University of Tübingen, Tübingen, Germany

    Peter Gehler

  3. Computer Vision Group, Visual Computing Institute, RWTH Aachen, Aachen, Germany

    Bastian Leibe

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Yu, W., Heber, S., Pock, T. (2015). Learning Reaction-Diffusion Models for Image Inpainting. In: Gall, J., Gehler, P., Leibe, B. (eds) Pattern Recognition. DAGM 2015. Lecture Notes in Computer Science(), vol 9358. Springer, Cham. https://doi.org/10.1007/978-3-319-24947-6_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24947-6_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24946-9

  • Online ISBN: 978-3-319-24947-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation