Skip to main content

Advertisement

SpringerLink
Log in

Knowledge visualization for evaluation tasks

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Although various methods for the evaluation of intelligent systems have been proposed in the past, almost no techniques are present that support the manual inspection of knowledge bases by the domain specialist. Manual knowledge base inspection is an important and frequently applied method in knowledge engineering. Since it can hardly be performed in an automated manner, it is a time-consuming and costly task. In this paper, we discuss a collection of appropriate visualization techniques that help developers to interactively browse and analyze the knowledge base in order to find deficiencies and semantic errors in their implementation. We describe standard visualization methods adapted to specifically support the analysis of the static knowledge base structure, but also of the usage of knowledge base objects such as questions or solutions. Additionally, we introduce a novel visualization technique that supports the validation of the derivation and interview behavior of a knowledge system in a semi-automatic manner. The application of the presented methods was motivated by the daily practice of knowledge base development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aamodt A, Plaza E (1994) Case-based reasoning: foundational issues, methodological variations, and system approaches. AI Commun 7(1): 39–59

    Google Scholar 

  2. Adeli H (1990) Knowledge engineering, vol I: fundamentals. McGraw-Hill Inc., New York

    Google Scholar 

  3. Akers S (1978) Binary decision diagrams. IEEE Trans Comput 27(6): 509–516

    Article  MATH  Google Scholar 

  4. Ayel M, Laurent JP (1991) Validation, verification and test of knowledge-based systems. Wiley, London

    Google Scholar 

  5. Baumeister J (2009) Advanced measures for empirical testing. In: FLAIRS’09: proceedings of the 22th international Florida artificial intelligence research society conference. AAAI Press, pp 378–383

  6. Baumeister J, Bregenzer J, Puppe F (2006) Gray box robustness testing of rule systems. In: KI’06: proceedings of the 29th annual German conference on artificial intelligence. LNAI 4314. Springer, pp 346–360

  7. Baumeister J, Menge M, Puppe F (2008) Visualization techniques for the evaluation of knowledge systems. In: FLAIRS’08: proceedings of the 21th international Florida artificial intelligence research society conference. AAAI Press, pp 329–334

  8. Baumeister J, Reutelshoefer J, Puppe F (2010) KnowWE: a semantic wiki for knowledge engineering. Appl Intell. doi:10.1007/s10489-010-0224-5

  9. Baumeister J, Seipel D (2005) Smelly owls—design anomalies in ontologies. In: FLAIRS’05: proceedings of the 18th international Florida artificial intelligence research society conference. AAAI Press, pp 215–220

  10. Baumeister J, Seipel D (2006) Verification and refactoring of ontologies with rules. In: EKAW’06: proceedings of the 15th international conference on knowledge engineering and knowledge management, LNAI 4248. Springer, Berlin, pp 82–95

  11. Baumeister J, Seipel D (2010) Anomalies in ontologies with rules. Web Semant Sci Serv Agents World Wide Web 8(1): 55–68

    Article  Google Scholar 

  12. Baumeister J, Seipel D, Puppe F (2003) Incremental development of diagnostic set-covering models with therapy effects. Int J Uncertain Fuzziness Knowl Based Syst 11(Suppl Issue 2): 25–49

    Article  MathSciNet  Google Scholar 

  13. Baumeister J et al (2008) The knowledge modeling environment d3web.KnowME, open-source at: http://d3web.sourceforge.net

  14. Bederson BB, Shneiderman B, Wattenberg M (2002) Ordered and quantum treemaps: making effective use of 2D space to display hierarchies. ACM Trans Graph (TOG) 21(4): 833–854

    Article  Google Scholar 

  15. Blanchard J, Guillet F, Briand H (2007) Interactive visual exploration of association rules with rule- focusing methodology. Knowl Inf Syst 13(1): 43–75

    Article  Google Scholar 

  16. Bresciani S, Eppler MJ (2008) Do visualizations foster experience sharing and retention in groups? towards an experimental validation. In: Proceedings of I-KNOW ’08 and I-MEDIA ’08

  17. Cai Y, Stumpf R, Wynne T, Tomlinson M, Chung DSH, Boutonnier X, Ihmig M, Franco R, Bauernfeind N (2007) Visual transformation for interactive spatiotemporal data mining. Knowl Inf Syst 13(2): 119–142

    Article  Google Scholar 

  18. Clancey WJ (1985) Heuristic classification. Artif Intell 27(3): 289–350

    Article  Google Scholar 

  19. Ensan F, Du W (2010) A knowledge encapsulation approach to ontology modularization. Knowl Inf Syst. doi:10.1007/s10115-009-0279-y

  20. Fluit C, Sabou M, van Harmelen F (2006) Ontology-based information visualization. In: Visualizing the semantic web. Springer, pp 36–48

  21. Groot P, ten Teije A, van Harmelen F (2003) A quantitative analysis of the robustness of knowledge-based systems through degradation studies. Knowl Inf Syst 7(2): 224–245

    Article  Google Scholar 

  22. Hartung RL, Håkansson A (2007) Automated testing for knowledge based systems. In: KES 2007: proceedings of the 11th international conference on knowledge-based intelligent information and engineering systems, part I, LNCS 4693. Springer, pp 270–278

  23. Helfman R, Dumer J, Hanratty T (1995) TED-turbine engine diagnostics: an expert diagnostic system for the M1 Abrams AGT1500 turbine engine. In: 11th conference on artificial intelligence for applications. IEEE Computer Society, Los Alamitos, CA, USA, pp 1032–1038

  24. Horiyama T, Ibaraki T (2002) Ordered binary decision diagrams as knowledge-bases. Artif Intell 136(2): 189–213

    Article  MathSciNet  MATH  Google Scholar 

  25. Hu B, Shadbolt N (2003) Visualising a DL knowledge base with delogviz. In: DL2003: proceedings of the international workshop on description logics (CEUR 81)

  26. Hüttig M, Buscher G, Menzel T, Scheppach W, Puppe F, Buscher H-P (2004) A diagnostic expert system for structured reports, quality assessment, and training of residents in sonography. Med Klin 3: 117–122

    Article  Google Scholar 

  27. Iglezakis I, Reinartz T, Roth-Berghofer T (2004) Maintenance memories: beyond concepts and techniques for case base maintenance. In: Advances in case-based reasoning. Springer, LNCS 3155, pp 227–241

  28. Ji S, Matsumura Y, Kuwata S, Nakano H, Chen Y, Teratani T, Zhang Q, Mineno T, Takeda H (2004) Creation of a master table for checking indication and contraindication of medicine from a knowledge base linked with a thesaurus. J Med Syst 28(6): 561–573

    Article  Google Scholar 

  29. JSPWiki (n.d.) The open-source wiki clone JSPWiki. http://www.jspwiki.org

  30. Jurisica I, Mylopoulos J, Yu E (2004) Ontologies for knowledge management: an information systems perspective. Knowl Inf Syst 6(4): 380–401

    Article  Google Scholar 

  31. Kimura M, Sakamoto M, Adachi T, Sagara H (2005) Diagnosis of febrile illnesses in returned travelers using the PC software GIDEON. Travel Med Infect Dis 3(3): 157–160

    Article  Google Scholar 

  32. Knauf R, Gonzalez AJ, Abel T (2002) A framework for validation of rule-based systems. IEEE Trans Syst Man Cybern B Cybern 32(3): 281–295

    Article  Google Scholar 

  33. Kotis K, Vouros A (2006) Human-centered ontology engineering: the hcome methodology. Knowl Inf Syst 10(1): 109–131

    Article  Google Scholar 

  34. Lancaster FW, Ulvila JW, Humphrey SM, Smith LC, Allen B, Herner S (1996) Evaluation of interactive knowledge-based systems: overview and design for empirical testing. J Am Soc Inf Sci 47(1): 57–69

    Article  Google Scholar 

  35. Ligeza A (2006) Logical foundations for rule-based systems, studies in computational intelligence, vol 11. Springer, Berlin

    Google Scholar 

  36. Mazza R (2009) Introduction to information visualization. Springer, Berlin

    Google Scholar 

  37. Mersmann S, Dojat M (2004) SmartCaretm—automated clinical guidelines in critical care. In: ECAI’04/PAIS’04: proceedings of the 16th European conference on artificial intelligence, including prestigious applications of intelligent systems. IOS Press, Valencia, Spain, pp 745–749

  38. Miah SJ, Kerr D, Gammack J, Cowan T (2008) A generic design environment for the rural industry knowledge acquisition. Knowl Based Syst 21(8): 892–899

    Article  Google Scholar 

  39. Middelfart M, Pedersen TB (2009) Discovering sentinel rules for business intelligence. In: DEXA ’09: proceedings of the 20th international conference on database and expert systems applications. pp 592–602

  40. Milne R, Nicol C (2000) TIGER: continuous diagnosis of gas turbines. In: ECAI’00: proceedings of the 14th European conference on artificial intelligence. Berlin, Germany

  41. Nalepa GJ (2007) Proposal of business process and rules modeling with the XTT method. In: SYNASC ’07: proceedings of the ninth international symposium on symbolic and numeric algorithms for scientific computing. IEEE Computer Society, Washington, DC, USA, pp 500–506

  42. Nalepa GJ, Ligeza A (2005) A graphical tabular model for rule-based logic programming and verification. Syst Sci 31(2): 89–95

    Google Scholar 

  43. Nghia DD, Puppe F (2009) Hybrides, skalierbares Diagnosesystem für freie Kfz-Werkstätten[Hybrid and scalable diagnosis system for car garages]. KI Ger AI Mag 23(2): 31–37

    Google Scholar 

  44. Nilsson NJ (1982) Principles of artificial intelligence. Springer, Berlin

    MATH  Google Scholar 

  45. Noy NF, Klein M (2004) Ontology evolution: not the same as schema evolution. Knowl Inf Syst 6(4): 428–440

    Article  Google Scholar 

  46. O’Leary DE (1991) Design, development and validation of expert systems: a survey of developers. In: Ayel M, Laurent JP (eds) Validation, verification and test of knowledge-based systems. John Wiley & Sons Inc., London, pp 3–19

    Google Scholar 

  47. Ou MH, West GAW, Lazarescu M, Clay C (2007) Dynamic knowledge validation and verification for CBR teledermatology system. Art Intell Med 39(1): 79–96

    Article  Google Scholar 

  48. Padma T, Balasubramanie P (2009) Knowledge based decision support system to assist work-related risk analysis in musculoskeletal disorder. Knowle Based Syst 22(1): 72–78

    Article  Google Scholar 

  49. Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann Publisher, San Mateo

    Google Scholar 

  50. Pinto HS, Martins JP (2004) Ontologies: how can they be built?. Knowl Inf Syst 6(4): 441–464

    Article  Google Scholar 

  51. Plaisant C, Grosjean J, Bederson BB (2002) Spacetree: supporting exploration in large node link tree, design evolution and empirical evaluation. In: INFOVIS ’02: proceedings of the IEEE symposium on information visualization (InfoVis’02). pp 57–64

  52. Preece A (1998) Building the right system right. In: Evaluating V&V methods in knowledge engineering, verification and validation of knowledge-based systems: papers from the AAAI-98 Workshop. AAAI Press, pp 38–45

  53. Preece A (2001) Evaluating verification and validation methods in knowledge engineering. In: Micro-level knowledge management. Morgan-Kaufman, pp 123–145

  54. Preece A, Shinghal R (1994) Foundation and application of knowledge base verification. Int J Intell Syst 9: 683–702

    Article  Google Scholar 

  55. Puppe F (2000) Knowledge formalization patterns. In: PKAW 2000: proceedings of the Pacific rim knowledge acquisition workshop. Sydney, Australia

  56. Puppe F, Atzmueller M, Buscher G, Hüttig M, Luehrs H, Buscher H-P (2008) Application and evaluation of a medical knowledge system in sonography (SONOCONSULT). In: ECAI’08/PAIS’08: proceedings of the 18th European conference on artificial intelligence, including prestigious applications of intelligent systems. IOS Press, Amsterdam, The Netherlands, pp 683–687

  57. Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1): 81–106

    Google Scholar 

  58. Reggia JA, Nau DS, Wang PY (1983) Diagnostic expert systems based on a set covering model. J Man Mach Stud 19(5): 437–460

    Article  Google Scholar 

  59. Richards D (2004) Addressing the ontology acquisition bottleneck through reverse ontological engineering. Knowl Inf Syst 6(4): 402–427

    Article  MathSciNet  Google Scholar 

  60. Roth-Berghofer T (2003) Developing maintainable case-based reasoning systems: applying SIAM to empolis orenge. In: GWEM’03: German workshop on experience management, CEUR vol 67

  61. Seipel D, Hopfner M, Baumeister J (2005) Declarative querying and visualizing knowledge bases in XML. In: INAP/WLP’04: applications of declarative programming and knowledge management (selected papers), LNAI 3392. Springer, Berlin, pp 16–31

  62. Shneiderman B (1991) Tree visualization with tree-maps: a 2-d space-filling approach. In: ACM transactions on graphics, vol 11. pp 92–99

  63. Soman KP, Diwakar S, Ajay V (2006) Insight into data mining: theory and practice. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  64. Spence R (2007) Information visualization. Design for interaction. Prentice-Hall, Upper Saddle River

    Google Scholar 

  65. Subasic I, Berendt B (2009) Discovery of interactive graphs for understanding and searching time-indexed corpora. Knowl Inf Syst 23(3): 293–319

    Article  Google Scholar 

  66. Vermesan A, Coenen F (1999) Validation and verification of knowledge based systems. Theory, tools and practice. Kluwer Academic Publisher, Dordrecht

    Google Scholar 

  67. Wang S, Wang H (2002) Knowledge discovery through self-organizing maps: data visualization and query processing. Knowl Inf Syst 4(1): 31–45

    Article  MATH  Google Scholar 

  68. Yi JS, Kang Ya, Stasko J, Jacko J (2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans Vis Comput Graph 13(6): 1224–1231

    Article  Google Scholar 

  69. Zacharias V (2007) Visualization of rule bases—the overall structure. In: I-KNOW ’07: proceedings of international conference on knowledge management

  70. Zacharias V (2008) Development and verification of rule based systems—a survey of developers. In: RuleML ’08: proceedings of the international symposium on rule representation, interchange and reasoning on the Web, LNCS 5321. Springer-Verlag, Berlin, Heidelberg, pp 6–16

  71. Zlatareva NP (1997) Verification of non-monotonic knowledge bases. Decis Support Syst 21(4): 253–261

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Informatics and Artificial Intelligence, University of Würzburg, 97074, Würzburg, Germany

    Joachim Baumeister & Martina Freiberg

Authors
  1. Joachim Baumeister
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martina Freiberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joachim Baumeister.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baumeister, J., Freiberg, M. Knowledge visualization for evaluation tasks. Knowl Inf Syst 29, 349–378 (2011). https://doi.org/10.1007/s10115-010-0350-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-010-0350-8

Keywords

Search

Navigation