Skip to main content
SpringerLink
Log in

Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The automated segmentation of regions of interest (ROIs) in medical imaging is the fundamental requirement for the derivation of high-level semantics for image analysis in clinical decision support systems. Traditional segmentation approaches such as region-based depend heavily upon hand-crafted features and a priori knowledge of the user. As such, these methods are difficult to adopt within a clinical environment. Recently, methods based on fully convolutional networks (FCN) have achieved great success in the segmentation of general images. FCNs leverage a large labeled dataset to hierarchically learn the features that best correspond to the shallow appearance as well as the deep semantics of the images. However, when applied to medical images, FCNs usually produce coarse ROI detection and poor boundary definitions primarily due to the limited number of labeled training data and limited constraints of label agreement among neighboring similar pixels. In this paper, we propose a new stacked FCN architecture with multi-channel learning (SFCN-ML). We embed the FCN in a stacked architecture to learn the foreground ROI features and background non-ROI features separately and then integrate these different channels to produce the final segmentation result. In contrast to traditional FCN methods, our SFCN-ML architecture enables the visual attributes and semantics derived from both the fore- and background channels to be iteratively learned and inferred. We conducted extensive experiments on three public datasets with a variety of visual challenges. Our results show that our SFCN-ML is more effective and robust than a routine FCN and its variants, and other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Musen, M.A., et al.: Clinical decision-support systems. Biomedical informatics. Springer, London (2014)

    Book  Google Scholar 

  2. Doi, K.: Current status and future potential of computer-aided diagnosis in medical imaging. Br. J. Radiol. 78, s3–s19 (2005)

  3. Bi, L., et al.: Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies, Comput. Med. Imaging Graph. (2016). doi:10.1016/j.compmedimag.2016.11.008

  4. Chen, X., et al.: Medical image segmentation by combining graph cuts and oriented active appearance models. IEEE Trans. Image Process. 21, 2035–2046 (2012)

    Article  MathSciNet  Google Scholar 

  5. Ahn, E., et al.: Saliency-based Lesion Segmentation via Background Detection in Dermoscopic Images. IEEE J. Biomed. Health Inform. (2017). doi:10.1109/JBHI.2017.2653179

  6. Li, B.N., et al.: Integrating spatial fuzzy clustering with level set methods for automated medical image segmentation. Comput. Biol. Med. 41, 1–10 (2011)

    Article  Google Scholar 

  7. Long, J., et al.: Fully convolutional networks for semantic segmentation, In: Proc. IEEE CVPR, 3431–3440 (2015)

  8. Chen, L.-C., et al.: Semantic image segmentation with deep convolutional nets and fully connected crfs, In: Proc. ICLR (2015)

  9. Zheng, S., et al.: Conditional random fields as recurrent neural networks, In: Proc. IEEE ICCV, 1529–1537 (2015)

  10. Chen, H., et al.: DCAN: Deep contour-aware networks for object instance segmentation from histology images. Med. Image Anal. 36, 135–146 (2017)

    Article  Google Scholar 

  11. Dou, Q., et al.: 3d deeply supervised network for automatic liver segmentation from ct volumes, In: Proc. MICCAI, 149–157 (2016)

  12. Xu, Y., et al.: Gland instance segmentation by deep multichannel side supervision. In: Proc. MICCAI, 496–504 (2016)

  13. BenTaieb, A., et al.: Topology Aware Fully Convolutional Networks for Histology Gland Segmentation. In: Proc. MICCAI, 460–468 (2016)

  14. Paulano, F., et al.: 3D segmentation and labeling of fractured bone from CT images. Vis. Comput. 30, 939–948 (2014)

    Article  Google Scholar 

  15. Lin, L., et al.: Inference with collaborative model for interactive tumor segmentation in medical image sequences. IEEE Trans. Cybern. 46, 2796–2809 (2016)

    Article  Google Scholar 

  16. Bi, L., et al.: Cellular automata and anisotropic diffusion filter based interactive tumor segmentation for positron emission tomography, IEEE EMBC, 5453–5456 (2013)

  17. ONeill, G.T., et al.: Segmentation of cam-type femurs from CT scans. Vis. Comput. 28, 205–218 (2012)

    Article  Google Scholar 

  18. Silveira, M., et al.: Comparison of segmentation methods for melanoma diagnosis in dermoscopy images. IEEE J. Sel. Top. Signal Process. 3, 35–45 (2009)

    Article  Google Scholar 

  19. Emre Celebi, M., et al.: Lesion border detection in dermoscopy images using ensembles of thresholding methods. Skin Res. Technol. 19, e252–e258 (2013)

    Article  Google Scholar 

  20. Li, C., et al.: Minimization of region-scalable fitting energy for image segmentation. IEEE Trans. Image Process. 17, 1940–1949 (2008)

    Article  MathSciNet  Google Scholar 

  21. Li, C., et al.: Distance regularized level set evolution and its application to image segmentation. IEEE Trans. Image Process. 19, 3243–3254 (2010)

    Article  MathSciNet  Google Scholar 

  22. Wong, A., et al.: Automatic skin lesion segmentation via iterative stochastic region merging. IEEE Trans. Inf. Technol. Biomed. 15, 929–936 (2011)

    Article  Google Scholar 

  23. Somkantha, K., et al.: Boundary detection in medical images using edge following algorithm based on intensity gradient and texture gradient features. IEEE Trans. Biomed. Eng. 58, 567–573 (2011)

    Article  Google Scholar 

  24. Roy, A., et al.: JCLMM: A finite mixture model for clustering of circular–linear data and its application to psoriatic plaque segmentation, Pattern Recognit. (2017)

  25. Nock, R., et al.: Statistical region merging. IEEE Trans. Pattern Anal. Mach. Intel. 26, 1452–1458 (2004)

    Article  Google Scholar 

  26. van Rikxoort, E.M., et al.: Adaptive local multi-atlas segmentation: application to the heart and the caudate nucleus. Med. Image Anal. 14, 39 (2010)

    Article  Google Scholar 

  27. Song, Y., et al.: Similarity guided feature labeling for lesion detection, In Proc. MICCAI, 284–291 (2013)

  28. Li, C., et al.: Automated PET-guided liver segmentation from low-contrast CT volumes using probabilistic atlas. Comput. Method Program Biomed. 107, 164–174 (2012)

    Article  Google Scholar 

  29. Bi, L., et al.: Automatic Descending Aorta Segmentation in Whole-Body PET-CT Studies for PERCIST-Based Thresholding, In: Proc. DICTA, 1–6 (2012)

  30. Li, C., et al.: Joint probabilistic model of shape and intensity for multiple abdominal organ segmentation from volumetric CT images. IEEE J. Biomed. Health Inform. 17, 92–102 (2013)

    Article  Google Scholar 

  31. Isgum, I., et al.: Multi-atlas-based segmentation with local decision fusion: application to cardiac and aortic segmentation in CT scans. IEEE Trans. Med. Imaging 28, 1000–1010 (2009)

    Article  Google Scholar 

  32. Bi, L., et al.: Multi-stage thresholded region classification for whole-body PET–CT lymphoma studies, In: Proc. MICCAI, 569–576 (2014)

  33. Song, Y., et al.: A multistage discriminative model for tumor and lymph node detection in thoracic images. IEEE Trans. Med. Imaging 31, 1061–1075 (2012)

    Article  Google Scholar 

  34. Lartizien, C., et al.: Computer-aided staging of lymphoma patients with FDG PET/CT imaging based on textural information. IEEE J. Biomed. Health Inform. 18, 946–955 (2014)

    Article  Google Scholar 

  35. Ciresan, D., et al.: Deep neural networks segment neuronal membranes in electron microscopy images, In: Proc. NIPS, 2843–2851 (2012)

  36. Cha, K.H., et al.: Urinary bladder segmentation in CT urography using deeplearning convolutional neural network and level sets. Med. Phys. 43, 1882–1896 (2016)

    Article  Google Scholar 

  37. Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)

    Article  Google Scholar 

  38. Roth, H.R., et al.: Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation, In: Proc. MICCAI, 556–564 (2015)

  39. Farag, A., et al.: A bottom-up approach for pancreas segmentation using cascaded superpixels and (deep) image patch labeling. IEEE Trans. Image Process. 26, 386–399 (2017)

    Article  MathSciNet  Google Scholar 

  40. Ronneberger, O., et al.: U-net: convolutional networks for biomedical image segmentation. In: Proc. MICCAI, 234–241 (2015)

  41. Fu, H., et al.: Retinal vessel segmentation via deep learning network and fully-connected conditional random fields, In: Proc. IEEE ISBI, 698–701 (2016)

  42. Lu, F., et al.: Automatic 3D liver location and segmentation via convolutional neural network and graph cut. Int. J. Comput. Assist. Radiol. Surg. 12, 171–182 (2017)

  43. Hu, P., et al.: Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets, Int. J. Comput. Assist. Radiol. Surg. 1–13, (2016)

  44. Chatfield, K., et al.: Return of the devil in the details: delving deep into convolutional nets, BMVC (2014)

  45. Wang, L., et al.: Saliency detection with recurrent fully convolutional networks, In: Proc. ECCV, 825–841 (2016)

  46. Dean, J., et al.: Large scale distributed deep networks, In: NIPS, 1223–1231 (2012)

  47. Hogeweg, L., et al.: Clavicle segmentation in chest radiographs. Med. Image Anal. 16, 1490–1502 (2012)

    Article  Google Scholar 

  48. Sirinukunwattana, K., et al.: A stochastic polygons model for glandular structures in colon histology images. IEEE Trans. Med. Imaging 34, 2366–2378 (2015)

    Article  Google Scholar 

  49. Shin, H.-C., et al.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016)

    Article  Google Scholar 

  50. Deng, J., et al.: Imagenet: a large-scale hierarchical image database, In: Proc. CVPR, 248–255, (2009)

  51. Kumar, A., et al.: An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE J. Biomed. Health Inform. 21, 31–40 (2017)

    Article  Google Scholar 

  52. Vedaldi, A., et al.: Matconvnet: Convolutional neural networks for matlab, In: Proc. ACM MM, 689–692 (2015)

  53. Krizhevsky, A., et al.: Imagenet classification with deep convolutional neural networks, In: NIPS, 1097–1105 (2012)

  54. Li, X., et al.: Contextual hypergraph modeling for salient object detection, In: Proc. ICCV, 3328–3335 (2013)

  55. Chan, T.F., Vese, L.A.: Active contour and segmentation models using geometric PDEs for medical imaging. Geometric methods in bio-medical image processing. Springer, Berlin (2002)

    MATH  Google Scholar 

  56. Zhang, Y., et al.: Medical image segmentation using new hybrid level-set method, In: Proc. MEDIVIS, 71–76 (2008)

  57. Shotton, J., et al.: Semantic texton forests for image categorization and segmentation, In: Proc. CVPR, 1–8 (2008)

  58. Bai, X., et al.: Video snapcut: robust video object cutout using localized classifiers, In: ACM Trans. Graph. (2009)

Download references

Author information

Authors and Affiliations

  1. School of Information Technologies, The University of Sydney, Sydney, Australia

    Lei Bi, Jinman Kim, Ashnil Kumar, Michael Fulham & Dagan Feng

  2. Department of Molecular Imaging, Royal Prince Alfred Hospital, Sydney, Australia

    Michael Fulham

  3. Sydney Medical School, The University of Sydney, Sydney, Australia

    Michael Fulham

  4. Med-X Research Institute, Shanghai Jiao Tong University, Shanghai, China

    Dagan Feng

Authors
  1. Lei Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinman Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashnil Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Fulham
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dagan Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinman Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, L., Kim, J., Kumar, A. et al. Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation. Vis Comput 33, 1061–1071 (2017). https://doi.org/10.1007/s00371-017-1379-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-017-1379-4

Keywords

Search

Navigation