Skip to main content
SpringerLink
Log in

SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

State-of-the-art scene flow algorithms pursue the conflicting targets of accuracy, run time, and robustness. With the successful concept of pixel-wise matching and sparse-to-dense interpolation, we shift the operating point in this field of conflicts towards universality and speed. Avoiding strong assumptions on the domain or the problem yields a more robust algorithm. This algorithm is fast because we avoid explicit regularization during matching, which allows an efficient computation. Using image information from multiple time steps and explicit visibility prediction based on previous results, we achieve competitive performances on different data sets. Our contributions and results are evaluated in comparative experiments. Overall, we present an accurate scene flow algorithm that is faster and more generic than any individual benchmark leader.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In International conference on computer vision (ICCV).

  • Bailer, C., Taetz, B., & Stricker, D. (2019). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 41, 1879–1892.

    Article  Google Scholar 

  • Basha, T., Moses, Y., & Kiryati, N. (2013). Multi-view scene flow estimation: A view centered variational approach. International Journal of Computer Vision (IJCV), 101, 6–21.

    Article  MathSciNet  Google Scholar 

  • Behl, A., Jafari, O. H., Mustikovela, S. K., Alhaija, H. A., Rother, C., & Geiger, A. (2017). Bounding boxes, segmentations and object coordinates: How important is recognition for 3D scene flow estimation in autonomous driving scenarios? In Conference on computer vision and pattern recognition (CVPR).

  • Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision (ECCV).

  • Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In European conference on computer vision (ECCV).

  • Chen, Q., & Koltun, V. (2016). Full flow: Optical flow estimation by global optimization over regular grids. In Conference on computer vision and pattern recognition (CVPR).

  • Dollár, P., & Zitnick, C. L. (2013). Structured forests for fast edge detection. In International conference on computer vision (ICCV).

  • Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In Conference on computer vision and pattern recognition (CVPR).

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In Conference on computer vision and pattern recognition (CVPR).

  • He, K., & Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kD-trees. In Conference on computer vision and pattern recognition (CVPR).

  • Herbst, E., Ren, X., & Fox, D. (2013). RGB-D flow: Dense 3-D motion estimation using color and depth. In International conference on robotics and automation (ICRA).

  • Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 30, 328–341.

    Article  Google Scholar 

  • Hu, Y., Li, Y., & Song, R. (2017). Robust interpolation of correspondences for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).

  • Hu, Y., Song, R., & Li, Y. (2016). Efficient coarse-to-fine patchmatch for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).

  • Huguet, F., & Devernay, F. (2007). A variational method for scene flow estimation from stereo sequences. In International conference on computer vision (ICCV).

  • Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., & Cremers, D. (2015). A primal-dual framework for real-time dense RGB-D scene flow. In International conference on robotics and automation (ICRA).

  • Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 33, 978–994.

    Article  Google Scholar 

  • Lv, Z., Beall, C., Alcantarilla, P. F., Li, F., Kira, Z., & Dellaert, F. (2016). A continuous optimization approach for efficient and accurate scene flow. In European conference on computer vision (ECCV).

  • Lv, F., Lian, Q., Yang, G., Lin, G., Jialin Pan, S., & Duan, L. (2018). Domain adaptive semantic segmentation through structure enhancement. In European conference on computer vision (ECCV).

  • Ma, W. C., Wang, S., Hu, R., Xiong, Y., & Urtasun, R. (2019). Deep Rigid Instance Scene Flow. In Conference on computer vision and pattern recognition (CVPR).

  • Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International conference on computer vision (ICCV).

  • Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016). A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Conference on computer vision and pattern recognition (CVPR).

  • Menze, M., & Geiger, A. (2015). Object scene flow for autonomous vehicles. In Conference on computer vision and pattern recognition (CVPR).

  • Menze, M., Heipke, C., & Geiger, A. (2018). Object scene flow. Journal of Photogrammetry and Remote Sensing (JPRS), 140, 60–76.

    Article  Google Scholar 

  • Neoral, M., & Šochman, J. (2017). Object scene flow with temporal consistency. In Computer vision winter workshop (CVWW).

  • Ošep, A., Hermans, A., Engelmann, F., Klostermann, D., Mathias, M., & Leibe, B. (2016). Multi-scale object candidates for generic object tracking in street scenes. In International conference on robotics and automation (ICRA).

  • Ren, Z., Sun, D., Kautz, J., & Sudderth, E. B. (2017). Cascaded scene flow prediction using semantic segmentation. In International conference on 3D vision (3DV).

  • Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: edge-preserving interpolation of correspondences for optical flow. In Conference on computer vision and pattern recognition (CVPR).

  • Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vazquez, D., & Lopez, A. M. (2015). Vision-based offline–online perception paradigm for autonomous driving. In Winter conference on applications of computer vision (WACV).

  • Saxena, R., Schuster, R., Wasenmüller O, & Stricker D (2019) PWOC-3D: Deep occlusion-aware end-to-end scene flow estimation. In Intelligent vehicles symposium (IV).

  • Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018a) Combining stereo disparity and optical flow for basic scene flow. In Commercial vehicle technology symposium (CVT).

  • Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018b). FlowFields++: Accurate optical flow correspondences meet robust interpolation. In International conference on image processing (ICIP).

  • Schuster, R., Wasenmüller, O., Kuschk, G., Bailer, C., & Stricker, D. (2018c). SceneFlowFields: Dense interpolation of sparse scene flow correspondences. In Winter conference on applications of computer vision (WACV).

  • Taniai, T., Sinha, S. N., & Sato, Y. (2017). Fast multi-frame stereo scene flow with motion segmentation. In Conference on computer vision and pattern recognition (CVPR).

  • Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (1999). Three-dimensional scene flow. In International conference on computer vision (ICCV).

  • Vogel, C., Roth, S., & Schindler, K. (2014). View-consistent 3d scene flow estimation over multiple frames. In European conference on computer vision (ECCV).

  • Vogel, C., Schindler, K., & Roth, S. (2013). Piecewise rigid scene flow. In International conference on computer vision (ICCV).

  • Vogel, C., Schindler, K., & Roth, S. (2015). 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision (IJCV), 115, 1–28.

    Article  MathSciNet  Google Scholar 

  • Wang, M., & Deng, W. (2018). Deep visual domain adaptation: A survey. Neurocomputing, 312, 135–153.

    Article  Google Scholar 

  • Wannenwetsch, A. S., Keuper, M., & Roth, S. (2017). ProbFlow: Joint optical flow and uncertainty estimation. In International conference on computer vision (ICCV).

  • Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers. D. (2008). Efficient dense scene flow from sparse or dense stereo data. In European conference on computer vision (ECCV).

  • Xu, P., Davoine, F., Bordes, J. B., Zhao, H., & Denœux, T. (2016). Multimodal information fusion for urban scene understanding. Machine Vision and Applications (MVA), 27, 331–349.

    Article  Google Scholar 

  • Yamaguchi, K., McAllester, D., & Urtasun, R. (2014). Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In European conference on computer vision (ECCV).

  • Yoshida, T., Wasenmüller, O., & Stricker, D. (2017). Time-of-flight sensor depth enhancement for automotive exhaust gas. In International conference on image processing (ICIP).

  • Zweig, S., & Wolf, L. (2017). InterpoNet, a brain inspired neural network for optical flow dense interpolation. In Conference on computer vision and pattern recognition (CVPR).

Download references

Author information

Authors and Affiliations

  1. DFKI - German Research Center for Artificial Intelligence, Kaiserslautern, Germany

    René Schuster, Oliver Wasenmüller & Didier Stricker

  2. BMW Group, Munich, Germany

    Christian Unger & Georg Kuschk

Authors
  1. René Schuster
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver Wasenmüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Unger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Georg Kuschk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Didier Stricker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to René Schuster.

Additional information

Communicated by Andreas Geiger.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schuster, R., Wasenmüller, O., Unger, C. et al. SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation. Int J Comput Vis 128, 527–546 (2020). https://doi.org/10.1007/s11263-019-01258-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-019-01258-1

Keywords

Search

Navigation