Abstract
Occlusion is visible in only one frame and cannot be seen in the other frame which is a vital challenge in video stitching. Occlusion always brings ghost artifacts in the blended area. Meanwhile, the traditional image stitching approaches ignore temporal consistency and cannot avoid flicking problem. To solve these challenges, we propose a unified framework in which the stitching quality and stabilization both perform well. Specifically, we explicitly detect the potential occlusion regions to indicate blending information. Then, based on the occlusion maps, we choose a proper strip in the overlapped region as the blending area. With spatial–temporal Bayesian view synthesis, spatial ghost-like artifacts can be significantly eliminated and the output videos can be kept stable. The experimental results show the out performance of the proposed approach compared to state-of-the-art approaches.
Similar content being viewed by others
References
Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)
Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vision 74(1), 59–73 (2007)
El-Saban, M., Izz, M., Kaheel, A., Refaat, M.: Improved optimal seam selection blending for fast video stitching of videos captured from freely moving devices. In: ICIP, IEEE (2011)
Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-based rendering using image-based priors. In: ICCV (2003)
Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-based rendering using image-based priors. Int. J. Comput. Vision 63(2), 141–151 (2005)
Gao, J.H., Li, Y., Chin, T.J., Brown, M.S.: Seam-driven image stitching. In: Eurographics Short Papers. IEEE (2013)
Hsiao, E., Hebert, M.: Occlusion reasoning for object detectionunder arbitrary viewpoint. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1803–1815 (2014)
Hu, J., Zhang, D.Q., Yu, H., Chen, C.W.: High resolution free-view interpolation of planar structure. In: ICME (2014)
Huang, K.C., Chien, P.Y., Chien, C.A., Chang, H.C., Guo, J.I.: A 360-degree panoramic video system design. In: International Symposium on VLSI Design. IEEE (2014)
Jang, W.S., Ho, Y.S.: Discontinuity preserving disparity estimation with occlusion handling. J. Vis. Commun. Image Represent. 25(7), 1595–1603 (2014)
Jiang, W., Gu, J.: Video stitching with spatial-temporal content-preserving warping. In: CVPR (2015)
Kang, S.B., Szeliski, R., Uyttendaele, M.: Seamless stitching using multi-perspective plane sweep. Technical Report. Microsoft Research, Redmond (2004)
Kauff, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Smolic, A., Tanger, R.: Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability. Sig. Process. Image Commun. 22(2), 217–234 (2007)
Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: ICCV. IEEE (2013)
Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: CVPR. IEEE (2015)
McCloskey, S.: Masking light fields to remove partial occlusion. In: ICPR. IEEE (2014)
McGuire, M., Stone, H.S.: Techniques for multiresolution image registration in the presence of occlusions. IEEE Trans. Geosci. Remote Sens. 38(3), 1476–1479 (2000)
Oh, J.D., Kuo, C.C.J.: Robust stereo matching with improved graph and surface models and occlusion handling. J. Vis. Commun. Image Represent. 21(5), 404–415 (2010)
Okumura, K., Raut, S., Gu, Q., Aoyama, T., Takaki, T., Ishii, I.: Real-time feature-based video mosaicing at 500 fps. In: International Conference on Intelligent Robots and Systems. IEEE (2013)
Owens, A., Barnes, C., Flint, A., Singh, H., Freeman, W.: Camouflaging an object from many viewpoints. In: CVPR. IEEE (2014)
Pujades, S., Devernay, F., Goldluecke, B.: Bayesian view synthesis and image-based rendering principles. In: CVPR (2014)
Srinivasan, A., Balamurugan, V.: Occlusion detection and image restoration in 3d face image. In: TENCON. IEEE (2014)
Su, J., Luo, A., Yang, L., Cheng, H.: Video stitching with spatial-temporal content-preserving warping. In: ICME Workshop (2016)
Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2010)
Thomas, B., Jitendra, M.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)
Uyttendaele, M., Eden, A., Skeliski, R.: Eliminating ghosting and exposure artifacts in image mosaics. In: CVPR. IEEE (2001)
Vladimir, K., Ramin, Z.: Computing visual correspondence with occlusions using graph cuts. In: ICCV. IEEE (2001)
Xiao, X., Daneshpanah, M., Javidi, B.: Occlusion removal using depth mapping in three-dimensional integral imaging. J. Disp. Technol. 8(8), 483–490 (2012)
Xiong, H., Wang, Z., He, R., Feng, D.D.: Video object segmentation with occlusion map. In: DICTA. IEEE (2012)
Yang, L., Cheng, H., Su, J.A.: Pixel-to-model background modeling in crowded scenes. In: ICME (2014)
Yang, L., Yendo, T., Tehrani, M.P.: Artifact reduction using reliability reasoning for image generation of FTV. J. Vis. Commun. Image Represent. 21, 542–560 (2010)
Yang, L., Yendo, T., Tehranil, M.P., Fujii, T.: Probabilistic reliability based view synthesis for FTV. In: ICIP (2010)
Yoo, J.C., Ahn, C.W.: Image matching using peak signal-to-noise ratio-based occlusion detection. Image Process. 6(5), 483–495 (2012)
Yuan, Z., Kebin, J., Peng-Yu, L.: Video stitch algorithm based on dynamic foreground extraction. In: CISP. IEEE (2009)
Zagrouba, E., Barhoumi, W., Amri, S.: An efficient image-mosaicing method based on multifeature matching. Mach. Vis. Appl. 20(3), 139–162 (2009)
Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: Asprojective-as-possible image stitching with moving DLT. In: CVPR. IEEE (2013)
Zeng, L., Zhang, S., Zhang, J., Zhang, Y.: Dynamic image mosaic via sift and dynamic programming. Mach. Vis. Appl. 25(5), 1271–1282 (2014)
Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: CVPR. IEEE (2014)
Zhang, F., Liu, F.: Casual stereoscopic panorama stitching. In: CVPR. IEEE (2015)
Zhang, H.M., Zeng, W., Chen, X.: Foreground based borderline adjusting for real time multi-camera video stitching. In: ICIG. IEEE (2009)
Zhang, T., Jia, K., Xu, C., Ma, Y., Ahuja, N.: Partial occlusion handling for visual tracking via robust part matching. In: CVPR. IEEE (2014)
Zhao, Y.J., Lu, Z.Q., Liu, Y.K.: Video image stitching based on moving object detection and motion prediction compensation. In: CISP. IEEE (2010)
Acknowledgements
The authors would like to thank all the reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (Grant Nos. U1613223, 61673088 and 61603078), National Key Research and Development Plan: New Energy Vehicles focus on special projects (No. 2017YFB0102500).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Su, J., Cheng, H., Yang, L. et al. Robust spatial–temporal Bayesian view synthesis for video stitching with occlusion handling. Machine Vision and Applications 29, 219–232 (2018). https://doi.org/10.1007/s00138-017-0888-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00138-017-0888-5