Skip to main content
SpringerLink
Log in

CBR Confidence as a Basis for Confidence in Black Box Systems

  • Conference paper
  • First Online:
Case-Based Reasoning Research and Development (ICCBR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11680))

Included in the following conference series:

Abstract

Determining when to trust black box systems is a well-known challenge. An important factor affecting users’ trust is confidence in system solutions. Previous case-based reasoning (CBR) research has developed criteria for assigning confidence to the solutions of a CBR system. This paper investigates whether such analysis, coupled with factors such as CBR system competence, can be used to predict confidence in the outputs of a black box system, when the black box and CBR systems are provided with the same training data. The paper presents initial strategies for using CBR confidence to predict black box system confidence. An evaluation explores the ability of the strategies to provide useful information and suggests future questions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Using SciPy [13].

  2. 2.

    Plotted using the Matplotlib package [12]).

References

  1. Aha, D., Agudo, B.D., Garcia, J.R. (eds.): Proceedings of XCBR-2018: First Workshop on Case-Based Reasoning for the Explanation of Intelligent Systems (2018)

    Google Scholar 

  2. Aha, D., Darrell, T., Pazzani, M., Reid, D., Sammut, C., Stone, P. (eds.): Proceedings of the IJCAI-17: Workshop on Explainable AI (XAI) (2017)

    Google Scholar 

  3. Akoglu, H.: User’s guide to correlation coefficients. Turkish J. Emerg. Med. 18(3), 91–93 (2018)

    Article  Google Scholar 

  4. Carney, J.G., Cunningham, P., Bhagwan, U.: Confidence and prediction intervals for neural network ensembles. In: International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339), IJCNN 1999, vol. 2, pp. 1215–1218, July 1999

    Google Scholar 

  5. Cheetham, W.: Case-based reasoning with confidence. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 15–25. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-44527-7_3

    Chapter  Google Scholar 

  6. Cheetham, W., Price, J.: Measures of solution accuracy in case-based reasoning systems. In: Funk, P., González Calero, P.A. (eds.) ECCBR 2004. LNCS (LNAI), vol. 3155, pp. 106–118. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28631-8_9

    Chapter  Google Scholar 

  7. Cheetham, W.E.: Case-based reasoning with confidence. Ph.D. thesis, Rensselaer Polytechnic Institute (1996)

    Google Scholar 

  8. Cortez, P., Silva, A.: Using data mining to predict secondary school student performance. In: Brito, A., Teixeira, J. (eds.) Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008), pp. 5–12. EUROSIS (2008)

    Google Scholar 

  9. Dodge, Y.: The Concise Encyclopedia of Statistics, chap. 379, pp. 502–505. Springer, New York (2008). https://doi.org/10.1007/978-0-387-32833-1_379

  10. Domingos, P.: Knowledge discovery via multiple models. Intell. Data Anal. 2(3), 187–202 (1998)

    Article  Google Scholar 

  11. Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

  12. Hunter, J.D.: Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9(3), 90–95 (2007)

    Article  Google Scholar 

  13. Jones, E., Oliphant, T., Peterson, P., et al.: SciPy: Open source scientific tools for Python (2001). http://www.scipy.org/

  14. Keane, M., Kenny, E.: How case based reasoning explained neural networks: An XAI survey of post-hoc explanation-by-example in ANN-CBR twins. arXiv preprint arXiv:1905.07186 (2019)

  15. Leake, D.: CBR in context: the present and future. In: Leake, D. (ed.) Case-Based Reasoning: Experiences, Lessons, and Future Directions, pp. 3–30. AAAI Press, Menlo Park (1996). http://www.cs.indiana.edu/~leake/papers/a-96-01.html

  16. Leake, D., Birnbaum, L., Hammond, K., Marlow, C., Yang, H.: An integrated interface for proactive, experience-based design support. In: Proceedings of the 2001 International Conference on Intelligent User Interfaces, pp. 101–108 (2001)

    Google Scholar 

  17. Madsen, M., Gregor, S.: Measuring human-computer trust. In: Proceedings of the Eleventh Australasian Conference on Information Systems, pp. 6–8 (2000)

    Google Scholar 

  18. Marling, C., Sqalli, M., Rissland, E., Munoz-Avila, H., Aha, D.: Case-based reasoning integrations. AI Mag. 23(1), 69–86 (2002)

    Google Scholar 

  19. Massie, S., Craw, S., Wiratunga, N.: A visualisation tool to explain case-base reasoning solutions for tablet formulation. In: Macintosh, A., Ellis, R., Allen, T. (eds.) SGAI 2004. Springer, Berlin (2004). https://doi.org/10.1007/1-84628-103-2_16

  20. Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2574–2582 (2016)

    Google Scholar 

  21. Nasiri, S., Helsper, J., Jung, M., Fathi, M.: Enriching a CBR recommender system by classification of skin lesions using deep neural networks. In: ICCBR 2018, p. 86, July 2018

    Google Scholar 

  22. Nugent, C., Cunningham, P.: A case-based recommender for black-box systems. Artif. Intell. Rev. 24(2), 163–178 (2005)

    Article  Google Scholar 

  23. Oliphant, T.: NumPy: A Guide to NumPy. Trelgol Publishing, USA (2006). http://www.numpy.org/

  24. Papadopoulos, G., Edwards, P.J., Murray, A.F.: Confidence estimation methods for neural networks: a practical comparison. IEEE Trans. Neural Netw. 12(6), 1278–1287 (2001)

    Article  Google Scholar 

  25. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  26. Reilly, J., Smyth, B., McGinty, L., McCarthy, K.: Critiquing with confidence. In: ICCBR, pp. 436–450 (2005)

    Google Scholar 

  27. Shin, C., Yun, U.T., Kim, H.K., Park, S.: A hybrid approach of neural network and memory-based learning to data mining. IEEE Trans. Neural Netw. Learning Syst. 11(3), 637–646 (2000)

    Article  Google Scholar 

  28. Smyth, B., McKenna, E.: Modelling the competence of case-bases. In: Smyth, B., Cunningham, P. (eds.) EWCBR 1998. LNCS, vol. 1488, pp. 208–220. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0056334

    Chapter  Google Scholar 

  29. Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)

  30. Zaragoza, H., Buc, D.: Confidence measures for neural network classifiers. In: Proceedings of the Seventh Int. Conf. Information Processing and Management of Uncertainty in Knowlegde-Based Systems (IPMU), vol. 1, pp. 886–893. Editions EDK (Paris), January 1998

    Google Scholar 

Download references

Acknowledgment

Testing was performed on a machine supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute.

Author information

Authors and Affiliations

  1. School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, 47408, USA

    Lawrence Gates, Caleb Kisby & David Leake

Authors
  1. Lawrence Gates
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caleb Kisby
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Leake
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lawrence Gates .

Editor information

Editors and Affiliations

  1. Norwegian University of Science and Technology, Trondheim, Norway

    Kerstin Bach

  2. Ohio University, Athens, OH, USA

    Cindy Marling

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gates, L., Kisby, C., Leake, D. (2019). CBR Confidence as a Basis for Confidence in Black Box Systems. In: Bach, K., Marling, C. (eds) Case-Based Reasoning Research and Development. ICCBR 2019. Lecture Notes in Computer Science(), vol 11680. Springer, Cham. https://doi.org/10.1007/978-3-030-29249-2_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-29249-2_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-29248-5

  • Online ISBN: 978-3-030-29249-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation