Skip to main content
SpringerLink
Log in

Determining the Relevance of Features for Deep Neural Networks

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12371))

Included in the following conference series:

Abstract

Deep neural networks are tremendously successful in many applications, but end-to-end trained networks often result in hard to understand black-box classifiers or predictors. In this work, we present a novel method to identify whether a specific feature is relevant to a classifier’s decision or not. This relevance is determined at the level of the learned mapping, instead of for a single example. The approach does neither need retraining of the network nor information on intermediate results or gradients. The key idea of our approach builds upon concepts from causal inference. We interpret machine learning in a structural causal model and use Reichenbach’s common cause principle to infer whether a feature is relevant. We demonstrate empirically that the method is able to successfully evaluate the relevance of given features on three real-life data sets, namely MS COCO, CUB200 and HAM10000.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adebayo, J., Gilmer, J., Goodfellow, I., Kim, B.: Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv:1810.03307 (2018)

  2. Barry-Jester, A.M., Casselman, B., Goldstein, D.: The new science of sentencing. In: The Marshall Project, vol. 4 (2015)

    Google Scholar 

  3. Berkeley, G.: A Treatise Concerning the Principles of Human Knowledge. JB Lippincott & Company, Philadelphia (1881)

    Google Scholar 

  4. Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5177–5186 (2019)

    Google Scholar 

  5. Cui, Y., Song, Y., Sun, C., Howard, A., Belongie, S.: Large scale fine-grained categorization and domain-specific transfer learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4109–4118 (2018)

    Google Scholar 

  6. Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3429–3437 (2017)

    Google Scholar 

  7. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

  8. Goyal, Y., Shalit, U., Kim, B.: Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv:1907.07165 (2019)

  9. Granger, C.W.: Investigating causal relations by econometric models and cross-spectral methods. Econometr. J. Econometr. Soc. 37, 424–438 (1969)

    Article  MATH  Google Scholar 

  10. Gretton, A., Bousquet, O., Smola, A., Schölkopf, B.: Measuring statistical dependence with Hilbert-Schmidt norms. In: Jain, S., Simon, H.U., Tomita, E. (eds.) ALT 2005. LNCS (LNAI), vol. 3734, pp. 63–77. Springer, Heidelberg (2005). https://doi.org/10.1007/11564089_7

    Chapter  Google Scholar 

  11. Jarisa, W., Henze, R., Kücükay, F., Schneider, F., Denzler, J., Hartmann, B.: Fusionskonzept zur reibwertschätzung auf basis von wetter- und fahrbahnzustandsinformationen. In: VDI-Fachtagung Reifen - Fahrwerk - Fahrbahn, pp. 169–188 (2019)

    Google Scholar 

  12. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., Sayres, R.: Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). arXiv preprint arXiv:1711.11279 (2017)

  13. Kindermans, P.-J., et al.: The (Un)reliability of saliency methods. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 267–280. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_14

    Chapter  Google Scholar 

  14. Kretschmer, M., Runge, J., Coumou, D.: Early prediction of extreme stratospheric polar vortex states based on causal precursors. Geophys. Res. Lett. 44(16), 8592–8600 (2017)

    Article  Google Scholar 

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  16. Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever HANs predictors and assessing what machines really learn. Nat. Commun. 10(1), 1–8 (2019)

    Article  Google Scholar 

  17. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  18. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  19. Mendonça, T., Ferreira, P.M., Marques, J.S., Marcal, A.R., Rozeira, J.: PH2-a dermoscopic image database for research and benchmarking. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5437–5440. IEEE (2013)

    Google Scholar 

  20. Montavon, G., Samek, W., Müller, K.R.: Methods for interpreting and understanding deep neural networks. Digit. Sig. Process. 73, 1–15 (2018)

    Article  MathSciNet  Google Scholar 

  21. Mopuri, K.R., Garg, U., Babu, R.V.: CNN fixations: an unraveling approach to visualize the discriminative image regions. IEEE Trans. Image Process. 28(5), 2116–2125 (2018)

    Article  Google Scholar 

  22. Pearl, J.: Causality. Cambridge University Press, New York (2009)

    Book  MATH  Google Scholar 

  23. Perez, F., Vasconcelos, C., Avila, S., Valle, E.: Data augmentation for skin lesion analysis. In: Stoyanov, D., et al. (eds.) CARE/CLIP/OR 2.0/ISIC -2018. LNCS, vol. 11041, pp. 303–311. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01201-4_33

    Chapter  Google Scholar 

  24. Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  25. Reichenbach, H.: The Direction of Time. Dover Publications, New York (1956)

    Book  Google Scholar 

  26. Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N.: Prabhat: deep learning and process understanding for data-driven earth system science. Nature 566(7743), 195–204 (2019)

    Article  Google Scholar 

  27. Reimers, C., Requena-Mesa, C.: Deep learning - an opportunity and a challenge for geo- and astrophysics. In: Skoda, P., Adam, F. (eds.) Knowledge Discovery in Big Data from Astronomy and Earth Observation, chap. 13, pp. 251–266. Elsevier (2020)

    Google Scholar 

  28. Simon, M., Rodner, E.: Neural activation constellations: unsupervised part model discovery with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1143–1151 (2015)

    Google Scholar 

  29. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)

  30. Stolz, W., et al.: Multivariate analysis of criteria given by dermatoscopy for the recognition of melanocytic lesions. In: Book of Abstracts, Fiftieth Meeting of the American Academy of Dermatology, Dallas, Tex: Dec, pp. 7–12 (1991)

    Google Scholar 

  31. Strobl, E.V., Zhang, K., Visweswaran, S.: Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. J. Causal Infer. 7(1), 1–24 (2019)

    Google Scholar 

  32. Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)

  33. The Cornell Lab of Ornithology: All about birds. https://www.allaboutbirds.org/guide/White-crowned_Sparrow/species-compare/64980371. Accessed 01 Mar 2020

  34. Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5, 180161 (2018)

    Article  Google Scholar 

  35. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Technical report CNS-TR-2010-001, California Institute of Technology (2010)

    Google Scholar 

  36. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Visualizing and understanding convolutional networks. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

  37. Zhu, F., Li, H., Ouyang, W., Yu, N., Wang, X.: Learning spatial regularization with image-level supervisions for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5513–5522 (2017)

    Google Scholar 

  38. Zintgraf, L.M., Cohen, T.S., Adel, T., Welling, M.: Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595 (2017)

Download references

Author information

Authors and Affiliations

  1. Computer Vision Group, Friedrich Schiller University Jena, 07743, Jena, Germany

    Christian Reimers & Joachim Denzler

  2. Institute of Data Science, German Aerospace Center, 07745, Jena, Germany

    Christian Reimers & Jakob Runge

Authors
  1. Christian Reimers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jakob Runge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joachim Denzler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Reimers .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 157 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Reimers, C., Runge, J., Denzler, J. (2020). Determining the Relevance of Features for Deep Neural Networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12371. Springer, Cham. https://doi.org/10.1007/978-3-030-58574-7_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58574-7_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58573-0

  • Online ISBN: 978-3-030-58574-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation