Abstract
Real-time tracking of anatomical landmarks in 3D medical images is of great importance, ranging from live quantification to optimal visualization.Existing deep network models have shown promising performance but typically require a large amount of annotated data for training. However, obtaining accurate and consistent annotations on sequences of 3D medical images can be very challenging even for skilled clinicians. In this paper, we propose a semi-supervised spatial-temporal modeling framework for real-time anatomical landmark tracking in 3D transesophageal echocardiography (TEE) images, which requires annotations on only a small fraction of frames in a sequence. Specifically, a spatial discriminative feature encoder is first trained via deep Q-learning on static images across all patients. Then we introduce a Cycle Ynet framework that integrates the encoded spatial features and learns temporal landmark correspondence over a sequence using a generative model by enforcing both cycle-consistency and accurate prediction on a couple of annotated frames. We validate the proposed model using 738 TEE sequences with around 15,000 frames and demonstrate that by combining a discriminative feature extractor with a generative tracking model, we could achieve superior performance using a small number of annotated data compared to state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Such criterion is only for generative tracking methods but is not applicable to TDD, which uses no bounding box in tracking process.
References
Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., Torr, P.H.S.: Fully-convolutional siamese networks for object tracking. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 850–865. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_56
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: Temporal cycle-consistency learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1801–1810 (2019)
Ghesu, F.C., Georgescu, B., Mansi, T., Neumann, D., Hornegger, J., Comaniciu, D.: An artificial agent for anatomical landmark detection in medical images. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 229–237. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46726-9_27
Ghesu, F.C., et al.: Multi-scale deep reinforcement learning for real-time 3d-landmark detection in ct scans. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 176–189 (2017)
Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2014)
Lukezic, A., Vojir, T., Cehovin Zajc, L., Matas, J., Kristan, M.: Discriminative correlation filter with channel and spatial reliability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6309–6318 (2017)
Misra, I., Zitnick, C.L., Hebert, M.: Shuffle and learn: unsupervised learning using temporal order verification. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 527–544. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_32
Voigt, I., et al.: Robust live tracking of mitral valve annulus for minimally-invasive intervention guidance. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 439–446. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24553-9_54
Wang, G., Luo, C., Sun, X., Xiong, Z., Zeng, W.: Tracking by instance detection: a meta-learning approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2020)
Wang, N., Song, Y., Ma, C., Zhou, W., Liu, W., Li, H.: Unsupervised deep tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1308–1317 (2019)
Wang, W., Shen, J., Shao, L.: Video salient object detection via fully convolutional networks. IEEE Trans. Image Process. 27(1), 38–49 (2017)
Wang, X., Jabri, A., Efros, A.A.: Learning correspondence from the cycle-consistency of time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2566–2576 (2019)
Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP), pp. 3645–3649. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, J., Zhang, Y., Amadou, Aa., Voigt, I., Mansi, T., Liao, R. (2020). Cycle Ynet: Semi-supervised Tracking of 3D Anatomical Landmarks. In: Liu, M., Yan, P., Lian, C., Cao, X. (eds) Machine Learning in Medical Imaging. MLMI 2020. Lecture Notes in Computer Science(), vol 12436. Springer, Cham. https://doi.org/10.1007/978-3-030-59861-7_60
Download citation
DOI: https://doi.org/10.1007/978-3-030-59861-7_60
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59860-0
Online ISBN: 978-3-030-59861-7
eBook Packages: Computer ScienceComputer Science (R0)
