Skip to main content
SpringerLink
Log in

Video enhancement using reference photographs

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Digital video cameras are becoming commonplace in many households, but they still leave something to be desired in terms of image quality. Their poor light sensitivity make images noisy and blurry, and internal storage bandwidth limits the frame resolution. We present a technique for enhancing a low quality video sequence, using a set of high quality reference photographs, taken of the same scene. Our technique generates a high quality frame by copying information from the photographs in a patch-wise fashion. The copying is guided by a sparse set of reliable correspondences between the video frames and photographs. Our technique is purely image-based, and does not require depth estimation. A robust descriptor is employed for establishing valid matches between the video frames and the photographs. Then, the geometric transformation is estimated between every corresponding patch. With only a few reference photographs, we are able to reduce noise and motion blur, and more important, increase resolution by a factor of 6 (see Fig. 1).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ancuti, C., Haber, T., Mertens, T., Bekaert, P.: Video enhancement using reference photographs. In: Conference Abstracts and Applications of ACM SIGGRAPH 2007, (Sketch session), San Diego. ACM, New York (2007)

    Google Scholar 

  2. Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of Symposium on Interactive 3D graphics, pp. 217–226. ACM, New York (2001)

    Chapter  Google Scholar 

  3. Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 24(9), 1167–1182 (2002)

    Article  Google Scholar 

  4. Bhat, P., Zitnick, C.L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., Kang, S.B.: Using photographs to enhance videos of a static scene. In: Proceedings of Eurographics Symposium on Rendering (EGSR), pp. 327–338. ACM, New York (2007)

    Google Scholar 

  5. Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: Proceedings of International Conference on Pattern Recognition, pp. 600–605. IEEE Press, Washington, DC (2000)

    Google Scholar 

  6. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of ACM SIGGRAPH, pp. 341–346. ACM, New York (2001)

    Google Scholar 

  7. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 1033–1038. IEEE Press, Washington, DC (1999)

    Chapter  Google Scholar 

  8. Fattal, R.: Upsampling via imposed edges statistics. ACM Trans. Graph. (SIGGRAPH) 26(3), (2007)

  9. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  10. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of 4th Alvey Vision Conference, vol. 18, pp. 147–151. ACM, New York (1988)

    Google Scholar 

  11. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  12. Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of ACM SIGGRAPH 2001, pp. 327–340. ACM, New York (2001)

    Google Scholar 

  13. Irani, M., Peleg, S.: Improving resolution by image registration. J. Comput. Vis. Graph. Image Process. 55(3), 231–239 (1991)

    Google Scholar 

  14. Jiang, Z., Wong, T.-T., Bao, H.: Practical super-resolution from dynamic video sequences. In: Proceedings of IEEE Computer Video and Pattern Recognition. IEEE Press, Washington, DC (2003)

    Google Scholar 

  15. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoustics Speech Signal Process 26(9), 1153–1160 (1981)

    Article  MathSciNet  Google Scholar 

  16. Kopf, J., Cohen, M., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. Proc. ACM Trans. Graph. (SIGGRAPH) 26(3), 96–100 (2007)

    Article  Google Scholar 

  17. Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. Proc. ACM SIGGRAPH 2003 22(3), 277–286 (2003)

    Article  Google Scholar 

  18. Liang, L., Liu, C., Xu, Y.-Q., Guo, B., Shum, H.-Y.: Real-time texture synthesis by patch-based sampling. In: Proceedings of ACM SIGGRAPH 2001. ACM, New York (2001)

    Google Scholar 

  19. Lindeberg, T.: Feature detection with automatic scale selection. Int. J. Comput. Vis. 30(2), 77–116 (1999)

    Google Scholar 

  20. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  21. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Comput. Vis. Pattern Recogn. (CVPR) 30(2), 257–263 (2004)

    Google Scholar 

  22. Moreels, P., Perona, P.: Evaluation of features detectors and descriptors based on 3D objects. Int. J. Comput. Vis. 73(3), 263–284 (2007)

    Article  Google Scholar 

  23. Schultz, R., Stevenson, R.: Extraction of high-resolution frames from video sequences. J. IEEE Trans. Image Process. 5(6), 996–1011 (1996)

    Article  Google Scholar 

  24. Shechtman, E., Caspi, Y., Irani, M.: Space-time super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 27(4), 531–545 (2005)

    Article  Google Scholar 

  25. Shi, J., Tomasi, C.: Good features to track. IEEE Comput. Vis. Pattern Recogn. (CVPR), 593–600 (1994)

  26. Sun, J., Zheng, N.-N., Tao, H., Shum, H.-Y.: Image hallucination with primal sketch priors. IEEE Comput. Vis. Pattern Recogn. (CVPR), 729 (2003)

  27. Wei, L.-Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of ACM SIGGRAPH 2000, pp. 479–488. ACM, New York (2000)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Expertise Centre for Digital Media, Hasselt University – tUL – IBBT, Wetenschapspark 2, Diepenbeek, 3590, Belgium

    Cosmin Ancuti, Tom Haber, Tom Mertens & Philippe Bekaert

Authors
  1. Cosmin Ancuti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tom Haber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tom Mertens
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Philippe Bekaert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cosmin Ancuti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ancuti, C., Haber, T., Mertens, T. et al. Video enhancement using reference photographs. Visual Comput 24, 709–717 (2008). https://doi.org/10.1007/s00371-008-0251-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-008-0251-y

Keywords

Search

Navigation