Abstract
Recent image inpainting works have shown promising results thanks to great advances of generative adversarial networks (GANs). However, these methods would still generate distorted structures or blurry textures for the situation of large missing area, which is mainly due to the inherent difficulty to train GANs. In this paper, we propose a novel multi-level discriminator (MLD) and wavelet loss (WT) to improve the learning of image inpainting generators. Our method does not change the structure of generator and only works in the training phase, which thus can be easily embedded into sophisticated inpainting networks and would not increase the inference time. Specifically, MLD divides the mask into multiple subregions and then imposes an independent discriminator to each subregion. It essentially increases the distribution overlap between the real images and generated images. Consequently, MLD improves the optimization of GANs by providing more effective gradients to generators. In addition, WT builds a reconstruction loss in the frequency domain, which can facilitate the training of image inpainting networks as a regularization term. Consequently, WT can enforce the generated contents to be more consistent and sharper than the traditional pixel-wise reconstruction loss. We integrate WLD and WT into off-the-shelf image inpainting networks, and conduct extensive experiments on CelebA-HQ, Paris StreetView, and Places2. The results well demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance and generates higher-quality images than the baselines.
J. Li—Student.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Antonini, M., Barlaud, M., Mathieu, P., Daubechies, I.: Image coding using wavelet transform. TIP (1992)
Arjovsky, M., Bottou, L.: Towards principled methods for training generative adversarial networks. In: ICLR (2017)
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. TOG (2009)
Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes paris look like paris? TOG (2012)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH (2001)
Ghorai, M., Samanta, S., Mandal, S., Chanda, B.: Multiple pyramids based image inpainting using local patch statistics and steering kernel feature. TIP (2019)
Goodfellow, I., et al.: Generative adversarial nets. In: NeurIPS (2014)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017)
Hu, J., Shen, L., Sun, G.: Edgeconnect: Generative image inpainting with adversarial edge learning. In: ICCV Workshop (2019)
Huang, H., He, R., Sun, Z., Tan, T.: Wavelet-srnet: a wavelet-based cnn for multi-scale face super resolution. In: ICCV (2017)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. TOG (2017)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: ICCV (2017)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: ICLR (2018)
Kwatra, V., Essa, I., Bobick, A., Kwatra, N.: Texture optimization for example-based synthesis. In: TOG (2005)
Lewis, A.S., Knowles, G.: Image compression using the 2-d wavelet transform. TIP (1992)
Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: ECCV (2018)
Liu, J., Yang, S., Fang, Y., Guo, Z.: Structure-guided image inpainting using homography transformation. TMM (2018)
Odena, A., Olah, C., Shlens, J.: Conditional image synthesis with auxiliary classifier gans. In: ICML (2017)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016)
Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: NeurIPS (2018)
Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: NeurIPS (2012)
Yang, Y., Guo, X.: Generative landmark guided face inpainting. In: PRCV (2020)
Yi, Z., Tang, Q., Azizi, S., Jang, D., Xu, Z.: Contextual residual aggregation for ultra high-resolution image inpainting. In: CVPR (2020)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: CVPR (2018)
Zhang, H., Hu, Z., Luo, C., Zuo, W., Wang, M.: Semantic image inpainting with progressive generative networks. In: ACM MM (2018)
Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: CVPR (2019)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. TPAMI (2018)
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant 61836008 and 61673362, Youth Innovation Promotion Association CAS (2017496).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, J., Wang, Z. (2021). Multi-level Discriminator and Wavelet Loss for Image Inpainting with Large Missing Area. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13021. Springer, Cham. https://doi.org/10.1007/978-3-030-88010-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-88010-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88009-5
Online ISBN: 978-3-030-88010-1
eBook Packages: Computer ScienceComputer Science (R0)