Skip to main content
SpringerLink
Log in

A neural network approach to remove rain using reconstruction and feature losses

  • Green and Human Information Technology 2019
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Rain streaks can eclipse some information of an image taken during rainfall which can degrade the performance of a vision system. While existing rain removing methods can recover the semantic structure, they lack natural texture recovery. The aim of this work is to recover the hidden structure and texture under the rain streaks with fine details. We propose a novel generative adversarial network with two discriminators to remove rain called rain removal generative adversarial network, where a combination of reconstruction, feature and adversarial losses is used for low level, structural and natural recovery, respectively. We have found that exploiting low-level \({l_1}\) loss with high-level structural similarity loss as a reconstruction loss is quite effective in attaining visually plausible and consistent texture. Qualitative and quantitative evaluations on our synthetically created dataset and a benchmark dataset show substantial performance gain than state-of-the-art rain removing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Barnum PC, Narasimhan S, Kanade T (2010) Analysis of rain and snow in frequency space. Int J Comput Vis 86(2–3):256

    Article  Google Scholar 

  2. Bossu J, Hautière N, Tarel JP (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93(3):348–367

    Article  Google Scholar 

  3. Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587

  4. Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision, pp 1968–1975

  5. Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8789–8797

  6. Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision, pp 633–640

  7. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Tran Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  8. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3855–3863

  9. Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, CVPR 2004, vol 1. IEEE, pp I–I

  10. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 2:2672–2680

    Google Scholar 

  11. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. Adv Neural Inf Process Syst 27:5767–5777

    Google Scholar 

  12. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  13. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: IEEE CVPR, pp 5967–5976

  14. Jetchev N, Bergmann U, Vollgraf R (2016) Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611.08207

  15. Kim JH, Lee C, Sim JY, Kim CS (2013) Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE international conference on image processing. IEEE, pp 914–917

  16. Kingma DP, Ba JL (2014) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations

  17. Li S, Ren W, Zhang J, Yu J, Guo X (2019) Single image rain removal via a deep decomposition-composition network. Computer Vis Image Underst 186:8–57

    Google Scholar 

  18. Mao X, Li Q, Xie H, Lau RY, Wang Z, Smolley SP (2017) Least squares generative adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV). IEEE, pp 2813–2821

  19. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  20. Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Phys D Nonlinear Phenom 60(1–4):259–268

    Article  MathSciNet  Google Scholar 

  21. Santhaseelan V, Asari VK (2015) Utilizing local phase information to remove rain from video. Int J Comput Vis 112(1):71–89

    Article  Google Scholar 

  22. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR

  23. Wang Y, Gong D, Yang J, Shi Q, Hengel Avd, Xie D, Zeng B (2019) An effective two-branch model-based deep network for single image deraining. arXiv preprint arXiv:1905.05404

  24. Wang YT, Zhao XL, Jiang TX, Deng LJ, Chang Y, Huang TZ (2018) Rain streak removal for single image via kernel guided cnn. arXiv preprint arXiv:1808.08545

  25. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  26. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  27. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514

  28. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. In: IEEE transactions on circuits and systems for video technology

  29. Zhang H, Xu T, Li H, Zhang S, Huang X, Wang X, Metaxas D (2017) Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE international conference on computer vision (ICCV), pp 5907–5915

  30. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: a 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452–1464

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (No. 2017R1A2B4001801).

Author information

Authors and Affiliations

  1. Faculty of Electronics and Electrical Engineering, University of Engineering and Technology, Taxila, 47050, Pakistan

    Kamran Javed & Furqan Shaukat

  2. College of Information and Communication Engineering, Sungkyunkwan University, Suwon, 16419, Korea

    Kamran Javed

  3. Quaid-e-Awam University of Engineering, Science and Technology, Larkana, Pakistan

    Ghulam Hussain

  4. Department of Software and Communications Engineering, Hongik University, Sejong, Korea

    Seong Oun Hwang

Authors
  1. Kamran Javed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ghulam Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Furqan Shaukat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seong Oun Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seong Oun Hwang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Javed, K., Hussain, G., Shaukat, F. et al. A neural network approach to remove rain using reconstruction and feature losses. Neural Comput & Applic 32, 13129–13138 (2020). https://doi.org/10.1007/s00521-019-04558-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04558-2

Keywords

Search

Navigation