Skip to main content
SpringerLink
Log in

Super-resolution of PROBA-V images using convolutional neural networks

  • Research Article
  • Published:
Astrodynamics Aims and scope Submit manuscript

A Correction to this article was published on 11 February 2022

This article has been updated

Abstract

European Space Aqency (ESA)’s PROBA-V Earth observation (EO) satellite enables us to monitor our planet at a large scale to study the interaction between vegetation and climate, and provides guidance for important decisions on our common global future. However, the interval at which high-resolution images are recorded spans over several days, in contrast to the availability of lower-resolution images which is often daily. We collect an extensive dataset of both high- and low-resolution images taken by PROBA-V instruments during monthly periods to investigate Multi Image Super-resolution, a technique to merge several low-resolution images into one image of higher quality. We propose a convolutional neural network (CNN) that is able to cope with changes in illumination, cloud coverage, and landscape features which are introduced by the fact that the different images are taken over successive satellite passages at the same region. Given a bicubic upscaling of low resolution images taken under optimal conditions, we find the Peak Signal to Noise Ratio of the reconstructed image of the network to be higher for a large majority of different scenes. This shows that applied machine learning has the potential to enhance large amounts of previously collected EO data during multiple satellite passes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Change history

References

  1. Park, S. C., Park, M. K., Kang, M. G. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21–36.

    Article  Google Scholar 

  2. Nasrollahi, K., Moeslund, T. B. Super-resolution: a comprehensive survey. Machine Vision and Applications, 2014, 25(6): 1423–1468.

    Article  Google Scholar 

  3. Latry, C., Delvit, J. M. Staggered arrays for high resolution earth observing systems. In: Proceedings of SPIE 7452, Earth Observing Systems XIV, 2009: 74520O.

    Chapter  Google Scholar 

  4. Li, L., Wang, W., Luo, H., Ying, S. Super-resolution reconstruction of high-resolution satellite ZY-3 TLC images. Sensors, 2017, 17(5): 1062.

    Article  Google Scholar 

  5. Dierckx, W., Sterckx, S., Benhadj, I., Livens, S., Duhoux, G., van Achteren, T., Francois, M., Mellab, K., Saint, G. PROBA-V mission for global vegetation monitoring: Standard products and image quality. International Journal of Remote Sensing, 2014, 35(7): 2589–2614.

    Article  Google Scholar 

  6. Yang, C. Y., Ma, C., Yang, M. H. Single-image super-resolution: a benchmark. In: Computer Vision — ECCV 2014. Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds. Cham: Springer International Publishing, 2014: 372–386.

    Chapter  Google Scholar 

  7. Yang, J. C., Wright, J., Huang, T. S., Ma, Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861–2873.

    Article  MathSciNet  MATH  Google Scholar 

  8. Dong, C., Loy, C. C., He, K. M., Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307.

    Article  Google Scholar 

  9. Kim, J., Lee, J. K., Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646–1654.

    Google Scholar 

  10. Dong, C., Loy, C. C., Tang, X. O. Accelerating the super-resolution convolutional neural network. In: Computer Vision — ECCV 2016. Leibe, B., Matas, J., Sebe, N., Welling, M., Eds. Cham: Springer International Publishing, 2016: 391–407.

    Chapter  Google Scholar 

  11. Shi, W. Z., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., Wang, Z. H. Realtime single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.

  12. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. H. et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017.

  13. Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., Do, M. N. Semantic image inpainting with deep generative models. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5485–5493.

    Google Scholar 

  14. Timofte, R., Gu, S. H., van Gool, L., Zhang, L., Yang, M. H. NTIRE 2018 challenge on single image super-resolution: Methods and results. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018: 852–863.

    Google Scholar 

  15. Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M., et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of Computer Vision and Pattern Recognition Workshops, 2017: 1110–1121.

    Google Scholar 

  16. He, K. M., Zhang, X. Y., Ren, S. Q., Sun, J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778.

    Google Scholar 

  17. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700–4708.

    Google Scholar 

  18. Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612.

    Article  Google Scholar 

  19. Schultz, R. R., Stevenson, R. L. Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing, 1996, 5(6): 996–1011.

    Article  Google Scholar 

  20. Faramarzi, E., Rajan, D., Fernandes, F. C. A., Christensen, M. P. Blind super resolution of real-life video sequences. IEEE Transactions on Image Processing, 2016, 25(4): 1544–1555.

    Article  MathSciNet  MATH  Google Scholar 

  21. Takeda, H., Milanfar, P., Protter, M., Elad, M. Superresolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958–1975.

    Article  MathSciNet  MATH  Google Scholar 

  22. Mitzel, D., Pock, T., Schoenemann, T., Cremers, D. Video super resolution using duality based TV-L1 optical flow. In: Lecture Notes in Computer Science. Denzler, J., Notni, G., Süße, H., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009: 432–441.

    Google Scholar 

  23. Brodu, N. Super-resolving multiresolution images with band-independent geometry of multispectral pixels. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8): 4610–4617.

    Article  Google Scholar 

  24. Gillespie, A. R., Kahle, A. B., Walker, R. E. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment, 1987, 22(3): 343–365.

    Article  Google Scholar 

  25. Thomas, C., Ranchin, T., Wald, L., Chanussot, J. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1301–1312.

    Article  Google Scholar 

  26. Murthy, K., Shearn, M., Smiley, B. D., Chau, A. H., Levine, J., Robinson, M. D. SkySat-1: very high-resolution imagery from a small satellite. In: Proceedings of SPIE 9241,Sensors, Systems, and Next-Generation Satellites XVIII, 2014: 92411E.

    Google Scholar 

  27. Wolters, E., Dierckx, W., Iordache, M. D., Swinnen, E. PROBA-V Products User Manual, 2014.

  28. Carlson, T. N., Ripley, D. A. On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sensing of Environment, 1997, 62(3): 241–252.

    Article  Google Scholar 

  29. Pettorelli, N., Vik, J. O., Mysterud, A., Gaillard, J. M., Tucker, C. J., Stenseth, N. C. Using the satellite-derived NDVI to assess ecological responses to environmental change. Trends in Ecology & Evolution, 2005, 20(9): 503–510.

    Article  Google Scholar 

  30. Wilson, A. M., Jetz, W. Remotely sensed high-resolution global cloud dynamics for predicting ecosystem and biodiversity distributions. PLoS Biology, 2016, 14(3): e1002415.

    Article  Google Scholar 

  31. Sheikh, H. R., Bovik, A. C. Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2):430–444.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their extremely detailed comments and critical questions, which helped to make the presentation of these results clearer and increased the overall quality of this work.

Author information

Authors and Affiliations

  1. European Space Agency, Noordwijk, 2001 AZ, the Netherlands

    Marcus Märtens, Dario Izzo, Andrej Krzic & Daniël Cox

Authors
  1. Marcus Märtens
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dario Izzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrej Krzic
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniël Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcus Märtens.

Additional information

Marcus Märtens graduated from the University of Paderborn (Germany) with a master degree in computer science. He joined ESA as a young graduate trainee in artificial intelligence where he worked on multi-objective optimization of spacecraft trajectories. He was part of the winning team of the 8th edition of the Global Trajectory Optimization Competition (GTOC) and received a HUMIES gold medal for developing algorithms achieving human competitive results in trajectory design. The Delft University of Technology awarded him a Ph.D. degree for his thesis on information propagation in complex networks. After working at the network architectures and services group in Delft (the Netherlands), Marcus rejoined ESA, where he now works as a research follow in the Advanced Concepts Team. While his main focus is on applied artificial intelligence and evolutionary optimization, Marcus has been working with experts from different fields and authored works related to neuroscience, cyber-security and gaming.

Dario Izzo graduated as a doctor of aeronautical engineering from the University Sapienza of Rome (Italy). He then took his second master degree in satellite platforms at the University of Cranfield in the United Kingdom, and received his Ph.D. degree in mathematical modeling at the University Sapienza of Rome where he lectured classical mechanics and space flight mechanics. He later joined the ESA and became the scientific coordinator of its Advanced Concepts Team. He devised and managed the Global Trajectory Optimization Competitions events, the ESA’s Summer of Code in Space and the Kelvins innovation and competition platform for space problems. He published more than 170 papers in international journals and conferences, making key contributions to the understanding of flight mechanics and spacecraft control and pioneering techniques based on evolutionary and machine learning approaches. He received the HUMIES Gold Medal and led the team winning the 8th edition of the Global Trajectory Optimization Competition.

Andrej Krzic graduated in physics from the University of Ljubljana (Slovenia), and received his M.Sc. degree in optics from the Friedrich Schiller University Jena (Germany). While in Jena, he was working at Carl Zeiss Microscopy GmbH, where he investigated the applications of adaptive optics in microscopic imaging. After studying, Andrej spent two years as a young graduate trainee at the ESA in the Netherlands. He was working at the Advanced Concepts Team, where he was active in several areas of research, including super-resolution imaging, quantum metrology, and optical communication. Andrej later returned to Jena, where he is now working on adaptive optics assisted quantum communication at the Fraunhofer Institute for Applied Optics and Precision Engineering.

Daniël Cox graduated as a Bachelor of science in applied physics at the University of Twente (the Netherlands) and is currently working on his master degree in applied physics with a focus on optics and image processing. He was briefly working at the Advanced Concepts Team of ESA during his internship.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Märtens, M., Izzo, D., Krzic, A. et al. Super-resolution of PROBA-V images using convolutional neural networks. Astrodyn 3, 387–402 (2019). https://doi.org/10.1007/s42064-019-0059-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42064-019-0059-8

Keywords

Search

Navigation