Abstract
Feature learning refers to techniques that learn to transform raw data input into an effective representation for further higher-level processing in many computer vision tasks. This chapter presents the investigations and the results of feature learning using convolutional neural networks to automatically assess knee osteoarthritis (OA) severity and the associated clinical and diagnostic features of knee OA from radiographs (X-ray images). Also, this chapter demonstrates that feature learning in a supervised manner is more effective than using conventional handcrafted features for automatic detection of knee joints and fine-grained knee OA image classification. In the general machine learning approach to automatically assess knee OA severity, the first step is to localize the region of interest that is to detect and extract the knee joint regions from the radiographs, and the next step is to classify the localized knee joints based on a radiographic classification scheme such as Kellgren and Lawrence grades. First, the existing approaches for detecting (or localizing) the knee joint regions based on handcrafted features are reviewed and outlined in this chapter. Next, three new approaches are introduced: (1) to automatically detect the knee joint region using a fully convolutional network, (2) to automatically assess the radiographic knee OA using CNNs trained from scratch for classification and regression of knee joint images to predict KL grades in ordinal and continuous scales, and (3) to quantify the knee OA severity optimizing a weighted ratio of two loss functions: categorical cross entropy and mean-squared error using multi-objective convolutional learning. The results from these methods show progressive improvement in the overall quantification of the knee OA severity. Two public datasets: the OAI and the MOST are used to evaluate the approaches with promising results that outperform existing approaches. In summary, this work primarily contributes to the field of automated methods for localization (automatic detection) and quantification (image classification) of radiographic knee OA.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Weighted Neighbor Distance using Compound Hierarchy of Algorithms Representing Morphology.
- 2.
ImageNet Large Scale Visual Recognition Challenge.
- 3.
Osteoarthritis Initiative.
- 4.
Multicenter Osteoarthritis Study.
- 5.
Osteoarthritis Research Society International.
- 6.
Western Ontario and McMaster Universities Osteoarthritis Index.
- 7.
Fisher score is one of the widely used methods for determining the most relevant features for classification.
- 8.
References
Shamir, L., Ling, S.M., Scott Jr., W.W., Bos, A., Orlov, N., Macura, T.J., Eckley, D.M., Ferrucci, L., Goldberg, I.G.: Knee X-ray image analysis method for automated detection of osteoarthritis. IEEE Trans. Biomed. Eng. 56(2) (2009)
Thomson, J., ONeill, T., Felson, D., Cootes, T.: Automated shape and texture analysis for detection of osteoarthritis from radiographs of the knee. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 127–134. Springer (2015)
Shamir, L., Ling, S.M., Scott, W., Hochberg, M., Ferrucci, L., Goldberg, I.G.: Early detection of radiographic knee osteoarthritis using computer-aided analysis. Osteoarthr. Cartil. 17(10), 1307–1312 (2009)
Shamir, L., Orlov, N., Eckley, D.M., Macura, T., Johnston, J., Goldberg, I.G.: WNDCHARM—an open source utility for biological image analysis. Source Code Biol. Med. 3(1), 13 (2008)
Orlov, N., Shamir, L., Macura, T., Johnston, J., Eckley, D.M., Goldberg, I.G.: WND-CHARM: multi-purpose image classification using compound image transforms. Pattern Recognit. Lett. 29(11), 1684–1693 (2008)
Oka, H., Muraki, S., Akune, T., Mabuchi, A., Suzuki, T., Yoshida, H., Yamamoto, S., Nakamura, K., Yoshimura, N., Kawaguchi, H.: Fully automatic quantification of knee osteoarthritis severity on plain radiographs. Osteoarthr. Cartil. 16(11), 1300–1306 (2008)
Park, H.J., Kim, S.S., Lee, S.Y., Park, N.H., Park, J.Y., Choi, Y.J., Jeon, H.J.: A practical MRI grading system for osteoarthritis of the knee: association with Kellgren-Lawrence radiographic scores. Eur. J. Radiol. 82(1), 112–117 (2013)
Lee, H.: Unsupervised feature learning via sparse hierarchical representations. Ph.D. thesis, Stanford University (2010)
Le, Q.V.: Scalable feature learning. Ph.D. thesis, Stanford University (2013)
Yang, S.: Feature engineering in fine-grained image classification. Ph.D. thesis, University of Washington (2013)
Donoghue, C.R.: Analysis of MRI for knee osteoarthritis using machine learning. Ph.D. thesis, Imperial College London (2013)
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Osteoarthr. Cartil. 42(1), 60–88 (2017)
Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Ann. Rev. Biomed. Eng. (0) (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Tajbakhsh, N., Shin, J.Y., Gurudu, S.R., Hurst, R.T., Kendall, C.B., Gotway, M.B., Liang, J.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35(5), 1299–1312 (2016)
Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., Nielsen, M.: Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. In: International Conference on Medical Image Computing and Computer-assisted Intervention, 246–253. Springer (2013)
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., Shen, D.: Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage 108, 214–224 (2015)
Roth, H.R., Farag, A., Lu, L., Turkbey, E.B., Summers, R.M.: Deep convolutional networks for pancreas segmentation in CT imaging. In: SPIE Medical Imaging, International Society for Optics and Photonics, pp. 94131G–94131G (2015)
Ciresan, D., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. In: Advances in Neural Information Processing Systems, pp. 2843–2851 (2012)
Marijnissen, A.C., Vincken, K.L., Vos, P.A., Saris, D., Viergever, M., Bijlsma, J., Bartels, L., Lafeber, F.: Knee images digital analysis (kida): a novel method to quantify individual radiographic features of knee osteoarthritis in detail. Osteoarthr. Cartil. 16(2), 234–243 (2008)
Felson, D.T., McAlindon, T.E., Anderson, J.J., Weissman, B.W., Aliabadi, P., Evans, S., Levy, D., LaValley, M.P.: Defining radiographic osteoarthritis for the whole knee. Osteoarthr. Cartil. 5(4), 241–250 (1997)
Braun, H.J., Gold, G.E.: Diagnosis of osteoarthritis: imaging. Bone 51(2), 278–288 (2012)
Emrani, P.S., Katz, J.N., Kessler, C.L., Reichmann, W.M., Wright, E.A., McAlindon, T.E., Losina, E.: Joint space narrowing and kellgren-lawrence progression in knee osteoarthritis: an analytic literature synthesis. Osteoarthr. Cartil. 16(8), 873–882 (2008)
Hart, D., Spector, T.: Kellgren & Lawrence grade 1 osteophytes in the kneedoubtful or definite? Osteoarthr. Cartil. 11(2), 149–150 (2003)
Shaikh, H., Panbude, J., Joshi, A.: Image segmentation techniques and its applications for knee joints: a survey. IOSR J. Electron. Commun. Eng. (IOSR-JECE) 9(5), 23–28 (2014)
Sun, Y., Teo, E., Zhang, Q.: Discussions of knee joint segmentation. In: International Conference on Biomedical and Pharmaceutical Engineering, 2006. ICBPE 2006. IEEE (2006)
Gornale, S.S., Patravali, P.U., Manza, R.R.: Detection of osteoarthritis using knee x-ray image analyses: a machine vision based approach. Int. J. Comput. Appl. 145(1) (2016)
Tiulpin, A., Thevenot, J., Rahtu, E., Saarakkala, S.: A novel method for automatic localization of joint area on knee plain radiographs. In: Scandinavian Conference on Image Analysis, pp. 290–301. Springer (2017)
Stammberger, T., Eckstein, F., Michaelis, M., Englmeier, K.H., Reiser, M.: Interobserver reproducibility of quantitative cartilage measurements: comparison of b-spline snakes and manual segmentation. Magn. Reson. Imaging 17(7), 1033–1042 (1999)
Cohen, Z.A., Mccarthy, D.M., Kwak, S.D., Legrand, P., Fogarasi, F., Ciaccio, E.J., Ateshian, G.A.: Knee cartilage topography, thickness, and contact areas from mri: in-vitro calibration and in-vivo measurements. Osteoarthr. Cartil. 7(1), 95–109 (1999)
Hirvasniemi, J., Thevenot, J., Immonen, V., Liikavainio, T., Pulkkinen, P., Jämsä, T., Arokoski, J., Saarakkala, S.: Quantification of differences in bone texture from plain radiographs in knees with and without osteoarthritis. Osteoarthr. Cartil. 22(10), 1724–1731 (2014)
Woloszynski, T., Podsiadlo, P., Stachowiak, G., Kurzynski, M.: A signature dissimilarity measure for trabecular bone texture in knee radiographs. Med. Phys. 37(5), 2030–2042 (2010)
Zhao, F., Xie, X.: An overview of interactive medical image segmentation. Ann. BMVA 2013(7), 1–22 (2013)
Pirnog, C.D.: Articular cartilage segmentation and tracking in sequential MR images of the knee. Ph.D. thesis, ETH Zurich (2005)
Duryea, J., Li, J., Peterfy, C., Gordon, C., Genant, H.: Trainable rule-based algorithm for the measurement of joint space width in digital radiographic images of the knee. Med. Phys. 27(3), 580–591 (2000)
Podsiadlo, P., Wolski, M., Stachowiak, G.: Automated selection of trabecular bone regions in knee radiographs. Med. Phys. 35(5), 1870–1883 (2008)
Anifah, L., Purnama, I.K.E., Hariadi, M., Purnomo, M.H.: Automatic segmentation of impaired joint space area for osteoarthritis knee on x-ray image using gabor filter based morphology process. IPTEK J. Technol. Sci. 22(3) (2011)
Lee, H.C., Lee, J.S., Lin, M.C.J., Wu, C.H., Sun, Y.N.: Automatic assessment of knee osteoarthritis parameters from two-dimensional x-ray image. In: First International Conference on Innovative Computing, Information and Control, 2006. ICICIC’06, vol. 2, pp. 673–676. IEEE (2006)
Subramoniam, M., Rajini, V.: Local binary pattern approach to the classification of osteoarthritis in knee x-ray images. Asian J. Sci. Res. 6(4), 805 (2013)
Subramoniam, B., et al.: A non-invasive computer aided diagnosis of osteoarthritis from digital x-ray images. Biomed. Res. (2015)
Deokar, D.D., Patil, C.G.: Effective feature extraction based automatic knee osteoarthritis detection and classification using neural network. Int. J. Eng. Tech. 1(3) (2015)
Yoo, T.K., Kim, D.W., Choi, S.B., Park, J.S.: Simple scoring system and artificial neural network for knee osteoarthritis risk prediction: a cross-sectional study. PLoS One 11(2), e0148724 (2016)
Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P., Saarakkala, S.: Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8(1), 1727 (2018)
Antony, J., McGuinness, K., O’Connor, N.E., Moran, K.: Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 1195–1200. IEEE (2016)
Sobel, I.: An isotropic 3\(\times \) 3 image gradient operator. In: Machine Vision for Three-dimensional Sciences (1990)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, vol. 1, pp. 886–893. CVPR 2005. IEEE (2005)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. IEEE (2016)
Kayalibay, B., Jensen, G., van der Smagt, P.: CNN-based segmentation of medical imaging data. arXiv:1701.03056 (2017)
Christ, P.F., Ettlinger, F., Grün, F., Elshaera, M.E.A., Lipkova, J., Schlecht, S., Ahmaddy, F., Tatavarty, S., Bickel, M., Bilic, P., et al.: Automatic liver and tumor segmentation of ct and mri volumes using cascaded fully convolutional neural networks. arXiv:1702.05970 (2017)
Li, F.F., Karpathy, A., Johnson, J.: CS231n: convolutional neural networks for visual recognition. http://cs231n.github.io/convolutional-networks/ (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
Kilian, J.: Simple image analysis by moments. OpenCV library documentation (2001)
Bressan, M., Dance, C.R., Poirier, H., Arregui, D.: Local contrast enhancement. In: Color Imaging: Processing, Hardcopy, and Applications, p. 64930Y (2007)
Pizer, S.M., Amburn, E.P., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny, B., Zimmerman, J.B., Zuiderveld, K.: Adaptive histogram equalization and its variations. Comput. Vis. Graph. Image Process 39(3), 355–368 (1987)
Erhan, D., Szegedy, C., Toshev, A., Anguelov, D.: Scalable object detection using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2147–2154 (2014)
Antony, J., McGuinness, K., Moran, K., OConnor, N.E.: Automatic detection of knee joints and quantification of knee osteoarthritis severity using convolutional neural networks. In: International Conference on Machine Learning and Data Mining in Pattern Recognition, pp. 376–390. Springer (2017)
Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)
Shamir, L., Orlov, N., Eckley, D.M., Macura, T., Johnston, J., Goldberg, I.: Wnd-charm: multi-purpose image classifier. Astrophysics Source Code Library (2013)
Tamura, H., Mori, S., Yamawaki, T.: Textural features corresponding to visual perception. IEEE Trans. Syst. Man Cybern. 8(6), 460–473 (1978)
Haralick, R.M., Shanmugam, K., et al.: Textural features for image classification. IEEE Trans. Syst. Man Cybern. 3(6), 610–621 (1973)
Grigorescu, S.E., Petkov, N., Kruizinga, P.: Comparison of texture features based on gabor filters. IEEE Trans. Image Proc. 11(10), 1160–1167 (2002)
Teague, M.R.: Image analysis via the general theory of moments. JOSA 70(8), 920–930 (1980)
Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: Proceedings of British Machine Vision Conference (2014)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678 (2014)
Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A., Winnemoeller, H.: Recognizing image style. In: Proceedings of the British Machine Vision Conference. BMVA Press (2014)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems, pp. 3320–3328 (2014)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)
Miao, S., Wang, Z.J., Zheng, Y., Liao, R.: Real-time 2d/3d registration via CNN regression. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 1430–1434. IEEE (2016)
Spampinato, C., Palazzo, S., Giordano, D., Aldinucci, M., Leonardi, R.: Deep learning for automated skeletal bone age assessment in x-ray images. Med. Image Anal. 36, 41–51 (2017)
Roth, H.R., Wang, Y., Yao, J., Lu, L., Burns, J.E., Summers, R.M.: Deep convolutional networks for automated detection of posterior-element fractures on spine CT. In: Proceedings Volume 9785, Medical Imaging 2016: Computer-Aided Diagnosis, SPIE Medical Imaging (2016)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR09 (2009)
Liu, S., Yang, J., Huang, C., Yang, M.H.: Multi-objective convolutional learning for face labeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3451–3459 (2015)
Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint training of a convolutional network and a graphical model for human pose estimation. In: Advances in Neural Information Processing Systems, pp. 1799–1807 (2014)
Ranftl, R., Pock, T.: A deep variational model for image segmentation. In: German Conference on Pattern Recognition, pp. 107–118. Springer (2014)
Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., Barbano, P.E.: Toward automatic phenotyping of developing embryos from videos. IEEE Trans. Image Proc. 14(9), 1360–1371 (2005)
Rudd, E.M., Günther, M., Boult, T.E.: Moon: a mixed objective optimization network for the recognition of facial attributes. In: European Conference on Computer Vision, pp. 19–35. Springer (2016)
Schiphof, D., Boers, M., Bierma-Zeinstra, S.M.: Differences in descriptions of kellgren and lawrence grades of knee osteoarthritis. Ann. Rheum. Dis. 67(7), 1034–1036 (2008)
Shamir, L., Felson, D.T., Ferrucci, L., Goldberg, I.G.: Assessment of osteoarthritis initiative-kellgren and lawrence scoring projects quality using computer analysis. J. Musculoskelet. Res. 13(04), 197–201 (2010)
Gutiérrez, P.A., Perez-Ortiz, M., Sanchez-Monedero, J., Fernandez-Navarro, F., Hervas-Martinez, C.: Ordinal regression methods: survey and experimental study. IEEE Trans. Knowl. Data Eng. 28(1), 127–146 (2016)
Pedregosa, F., Bach, F., Gramfort, A.: On the consistency of ordinal regression methods. J. Mach. Learn. Res. 18(55), 1–35 (2017)
Beckham, C., Pal, C.: A simple squared-error reformulation for ordinal classification. arXiv:1612.00775 (2016)
Górriz, M., Antony, J., McGuinness, K., Giró-i Nieto, X., OConnor, N.E.: Assessing knee OA severity with CNN attention-based end-to-end architectures. In: International Conference on Medical Imaging with Deep Learning, pp. 197–214 (2019)
Williams, D.A., Farrell, M.J., Cunningham, J., Gracely, R.H., Ambrose, K., Cupps, T., Mohan, N., Clauw, D.J.: Knee pain and radiographic osteoarthritis interact in the prediction of levels of self-reported disability. Arthritis Care Res. 51(4), 558–561 (2004)
Abedin, J., Antony, J., McGuinness, K., Moran, K., OConnor, N.E., Rebholz-Schuhmann, D., Newell, J.: Predicting knee osteoarthritis severity: comparative modeling based on patients data and plain x-ray images. Sci. Rep. 9(1), 5761 (2019)
Acknowledgements
This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant numbers SFI/12/RC/2289 and 15/SIRG/3283.
The OAI is a public-private partnership comprised of five contracts (N01-AR-2-2258; N01-AR-2-2259; N01-AR-2-2260; N01-AR-2-2261; N01-AR-2-2262) funded by the National Institutes of Health, a branch of the Department of Health and Human Services, and conducted by the OAI Study Investigators. Private funding partners include Merck Research Laboratories; Novartis Pharmaceuticals Corporation, GlaxoSmithKline; and Pfizer, Inc. Private sector funding for the OAI is managed by the Foundation for the National Institutes of Health.
MOST is comprised of four cooperative grants (Felson—AG18820; Torner—AG18832; Lewis—AG18947; and Nevitt—AG19069) funded by the National Institutes of Health, a branch of the Department of Health and Human Services, and conducted by MOST study investigators. This manuscript was prepared using MOST data and does not necessarily reflect the opinions or views of MOST investigators.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Antony, J., McGuinness, K., Moran, K., O’Connor, N.E. (2020). Feature Learning to Automatically Assess Radiographic Knee Osteoarthritis Severity. In: Nanni, L., Brahnam, S., Brattin, R., Ghidoni, S., Jain, L. (eds) Deep Learners and Deep Learner Descriptors for Medical Applications. Intelligent Systems Reference Library, vol 186. Springer, Cham. https://doi.org/10.1007/978-3-030-42750-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-42750-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-42748-1
Online ISBN: 978-3-030-42750-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)