Skip to main content
SpringerLink
Log in

Testing System Accuracy

  • Chapter
Clinical Decision Support Systems

Part of the book series: Health Informatics ((HI))

Abstract

Evaluation is a crucial component in the development of any clinical diagnostic decision support system (CDDSS). Much of it takes place informally as part of the development process and is used by the CDDSS developers for system improvement. Once a system is sufficiently mature, more formal evaluation studies should be done, initially of system accuracy and later, of system impact. A wide range of study design choices can be appropriate for assessing accuracy, but once the CDDSS appears to be ready for use in practice, there is a need for more rigorous evaluation. Most published evaluation studies have focused on the issue of system accuracy, with few studies evaluating the impact of using a CDDSS on clinical care. This chapter will address issues involved in assessing the accuracy of CDDSS. Key results from research or evaluation studies of system accuracy will be summarized and discussed. The reader who is interested in the details of individual studies should read the references at the end of this chapter.

Portions of this chapter have been taken Verbatim, with permission of the Journal of the Irish Colleges of Physicians and Surgeons, which owns the copyrights, from: Berner, ES. Computer-assisted diagnosis—consensus and controversies. JICPS 1996; 25:43–47.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 74.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bleich HL. Computer evaluation of acid-base disorders. J Clin Invest 1969; 48:1689–1696.

    Article  PubMed  CAS  Google Scholar 

  2. Willems JL, Abreu-Lima C, Arnaud P et al. The diagnostic performance of computer programs for the interpretation of electrocardiograms. N Engl J Med 1991; 325:1767–1773.

    Article  PubMed  CAS  Google Scholar 

  3. de Dombal, FT. The diagnosis of acute abdominal pain with computer assistance: worldwide perspective. Ann Chir 1991; 45:273–277.

    PubMed  Google Scholar 

  4. Adams ID, Chan M, Clifford PC et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. Br Med J (Clin Res Ed) 1986; 293:800–804.

    Article  CAS  Google Scholar 

  5. Sutton GC. How accurate is computer-aided diagnosis? Lancet 1989; 2:905–908.

    Article  PubMed  CAS  Google Scholar 

  6. Gough IR. Computer assisted diagnosis of the acute abdomen. Aust N Z J Surg 1993; 63:699–702.

    Article  PubMed  CAS  Google Scholar 

  7. Barnett GO, Cimino JJ, Hupp JA et al. DXplain—an evolving diagnostic decision-support system. JAMA 1987; 258:67–74.

    Article  PubMed  CAS  Google Scholar 

  8. Warner HR Jr. Iliad: moving medical decision-making into new frontiers. Methods Inf Med 1989; 28:370–372.

    PubMed  Google Scholar 

  9. Waxman HS, Worley WE. Computer-assisted adult medical diagnosis: subject review and evaluation of a new microcomputer-based system. Medicine 1990; 69:125–136.

    Article  PubMed  CAS  Google Scholar 

  10. Weed LL. Knowledge Coupling: New Premises and New Tools for Medical Care and Education. New York, NY: Springer-Verlag, 1991.

    Book  Google Scholar 

  11. Miller R, Masarie FE, Myers J. Quick Medical Reference (QMR) for diagnostic assistance. MD Comput 1986; 3:34–48.

    PubMed  CAS  Google Scholar 

  12. Miller RA. Medical diagnostic decision support systems—past, present, and future: a threaded bibliography and commentary. JAMIA 1994; 1:8–27.

    Article  PubMed  CAS  Google Scholar 

  13. Georgakis DC, Trace DA, Naeymi-Rad F et al. A statistical evaluation of the diagnostic performance of MEDAS—the medical emergency decision assistance system. Proc Annu Symp Comput Appl Med Care 1990:815–819.

    Google Scholar 

  14. Nelson SJ, Blois MS, Tuttle MS et al. Evaluating RECONSIDER: a computer program for diagnostic prompting. J Med Sys 1985; 9:379–388.

    Article  CAS  Google Scholar 

  15. Hammersley JR, Cooney K. Evaluating the utility of available differential diagnosis systems. Proc Annu Symp Comput Appl Med Care 1988:229–231.

    Google Scholar 

  16. Feldman MJ, Barnett GO. An approach to evaluating the accuracy of DXplain. Comput Methods Programs Biomed 1991; 35:261–266.

    Article  PubMed  CAS  Google Scholar 

  17. Heckerling PS, Elstein AS, Terzian CG et al. The effect of incomplete knowledge on the diagnosis of a computer consultant system. Med Inf 1991; 16:363–370.

    Article  CAS  Google Scholar 

  18. Lau LM, Warner HR. Performance of a diagnostic system (Iliad) as a tool for quality assurance. Comput Biomed Res 1992; 25:314–323.

    Article  PubMed  CAS  Google Scholar 

  19. Barness LA, Tunnessen WW Jr, Worley WE et al. Computer-assisted diagnosis in pediatrics. Am J Dis Child 1974; 127:852–858.

    PubMed  CAS  Google Scholar 

  20. O’Shea JS. Computer-assisted pediatric diagnosis. Am J Dis Child 1975; 129:199–202.

    PubMed  Google Scholar 

  21. Swender PT, Tunnessen WW Jr, Oski FA. Computer-assisted diagnosis. Am J Dis Child 1974; 127:859–861.

    PubMed  CAS  Google Scholar 

  22. Wexler JR, Swender PT, Tunnessen WW Jr et al. Impact of a system of computer-assisted diagnosis. Initial evaluation of the hospitalized patient. Am J Dis Child 1975; 129:203–205.

    PubMed  CAS  Google Scholar 

  23. Bankowitz RA, Lave JR, McNeil MA. A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med 1992; 31:3–11.

    PubMed  CAS  Google Scholar 

  24. Bankowitz RA, McNeil MA, Challinor SM et al. A computer-assisted medical diagnostic consultation service: implementation and prospective evaluation of a prototype. Ann Intern Med 1989; 110:824–832.

    Article  PubMed  CAS  Google Scholar 

  25. Bankowitz RA, McNeil MA, Challinor SM et al. Effect of a computer-assisted general medicine diagnostic consultation service on housestaff diagnostic strategy. Methods Inf Med 1989; 28:352–356.

    PubMed  CAS  Google Scholar 

  26. Berman L, Miller RA. Problem area formation as an element of computer aided diagnosis: a comparison of two strategies within quick medical reference (QMR). Methods Inf Med 1991; 30:90–95.

    PubMed  CAS  Google Scholar 

  27. Middleton B, Shwe MA, Heckerman DE et al. Probabilistic diagnosis using a reformulation of the Internist-1/QMR knowledge base. II. Evaluation of diagnostic performance. Methods Inf Med 1991; 30:256–267.

    PubMed  CAS  Google Scholar 

  28. Miller RA, Pople HE Jr, Myers J. Internist-I, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 1982; 307:468–476.

    Article  PubMed  CAS  Google Scholar 

  29. Miller RA, Masarie FE Jr. The quick medical reference (QMR) relationships function: description and evaluation of a simple, efficient “multiple diagnoses” algorithm. Medinfo 1992:512–518.

    Google Scholar 

  30. Miller RA, McNeil MA, Challinor S et al. Status Report: The Internist-1 / Quick Medical Reference project. West J Med 1986; 145:816–822.

    PubMed  CAS  Google Scholar 

  31. Berner ES, Webster GD, Shugerman AA et al. Performance of four computer-based diagnostic systems. N Engl J Med 1994; 330: 1792–1796.

    Article  PubMed  CAS  Google Scholar 

  32. Berner ES, Jackson JR, Algina J. Relationships among performance scores of four diagnostic decision support systems. JAMIA 1996; 3:208–215.

    Article  PubMed  CAS  Google Scholar 

  33. Bankowitz, RA. The effectiveness of QMR in medical decision support. Executive summary and final report. Springfield, VA: U.S. Department of Commerce, National Technical Information Service, 1994.

    Google Scholar 

  34. Murphy GC, Friedman CP, Elstein AS. The influence of a decision support system on the differential diagnosis of medical practitioners at three levels of training. Proc AMIA Fall Symp Comput 1996: 219–223.

    Google Scholar 

  35. Elstein AS, Friedman CP, Wolf FM et al. Effects of a decision support system on the diagnostic accuracy of users: a preliminary report. JAMIA 1996; 3:422–428.

    Article  PubMed  CAS  Google Scholar 

  36. Innis MD. Medisets. Computer-assisted diagnosis using a modified set theory. Proc Second Natl Health Conf, Health Informatics Conf, ’94, Gold Coast Australia, 1994, 286–291.

    Google Scholar 

  37. Bacchus CM, Quinton C, O’Rourke K et al. A ramdomized crossover trial of quick medical reference (QMR) as a teaching tool for medical interns. J Gen Intern Med 1994; 9:616–621.

    Article  PubMed  CAS  Google Scholar 

  38. Schoolman HM. Obligations of the expert system builder: meeting the needs of the user. MD Comput 1991; 8:316–321.

    PubMed  CAS  Google Scholar 

  39. Diamond LW. A different view of ILIAD. MD Comput 1992; 9:76–77.

    PubMed  CAS  Google Scholar 

  40. Waxman HS, Worley WE. Computer-assisted diagnosis. Ann Intern Med 1990; 113:561.

    Article  Google Scholar 

  41. Berner ES. The problem with software reviews of decision support systems. MD Comput 1993; 10:8–12.

    PubMed  CAS  Google Scholar 

  42. Kassirer JP. A report card on computer-assisted diagnosis—the grade: C. N Engl J Med 1994; 330:1824–1825.

    Article  CAS  Google Scholar 

  43. de Dombal FT. Computer-assisted diagnosis in Europe. N Engl J Med 1994; 331:1238.

    Article  PubMed  Google Scholar 

  44. Anderson RE, Hill RB, Key CR. The sensitivity and specificity of clinical diagnostics during five decades. JAMA 1989; 261:1610–1617.

    Article  PubMed  CAS  Google Scholar 

  45. Chimowitz MI, Logigian EL, Caplan LR. The accuracy of bedside neurological diagnoses. Ann Neurol 1990; 28:78–87.

    Article  PubMed  CAS  Google Scholar 

  46. Elmore JG, Wells CK, Lee CH et al. Variability in radiologists’ interpretation of mammograms. N Engl J Med 1994; 331:1493–1499.

    Article  PubMed  CAS  Google Scholar 

  47. Mazoue JG. Diagnosis without doctors. J Med Philos 1990; 15: 559–579.

    Article  PubMed  CAS  Google Scholar 

  48. Yolton RL. Computer-based diagnostic systems. N Engl J Med 1994; 331:1023.

    PubMed  CAS  Google Scholar 

  49. Miller RA, Masarie FE Jr. The demise of the “Greek Oracle” model for medical diagnosis systems. Methods Inf Med 1990; 29:1–2.

    PubMed  CAS  Google Scholar 

  50. Miller RA. Why the standard view is standard: people, not machines, understand patients’ problems. J Med Philos 1990; 15:581–591.

    Article  PubMed  CAS  Google Scholar 

  51. Weed LL. Problem-knowledge couplers: Philosophy, use and interpretation. PKC Corporation 1982; pgs. 2–22.

    Google Scholar 

  52. Weed LL. Reengineering medicine: questions and answers. Federation Bull 1995; 82:24–36.

    Google Scholar 

  53. Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993; 32:1–8.

    PubMed  CAS  Google Scholar 

  54. Johnston ME, Langton KB, Haynes RB et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome:a critical appraisal of research. Ann Intern Med 1994; 120:135–142.

    Article  PubMed  CAS  Google Scholar 

  55. Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Proc Annu Symp Comput Appl Med Care 1991:3–7.

    Google Scholar 

  56. Miller PL. The evaluation of artificial intelligence systems in medicine. Comput Methods Programs Biomed 1986; 22:5–11.

    PubMed  CAS  Google Scholar 

  57. McAdam WA, Brock BM, Armitage T et al. Twelve years experience of computer-aided diagnosis in a district general hospital. Ann R Coll Surg Engl 1990; 72:140–146.

    PubMed  CAS  Google Scholar 

  58. de Dombal FT. Computer-aided decision support:in praise of level playing fields. Methods Inf Med 1994; 33:161–163.

    PubMed  Google Scholar 

  59. Berner ES, Maisiak RS. Physician use of interactive functions in diagnostic decision support systems. Proc AMIA Fall Symp 1997; 842.

    Google Scholar 

  60. Miller RA. Evaluating evaluations of medical diagnostic systems. JAMIA 1996; 3:429–431.

    Article  PubMed  CAS  Google Scholar 

  61. Welford CR. A comprehensive computerized patient record with automated linkage to QMR. Proc Annu Symp Comput Appl Med Care 1994: 814–818.

    Google Scholar 

Download references

Authors
  1. Eta S. Berner
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Informatics Program, Department of Health Services Administration, School of Health Related Professions, 1675 University Boulevard, Room 544, Birmingham, AL, 35294-8219, USA

    Eta S. Berner EdD (Professor, Masters of Science in Health) (Professor, Masters of Science in Health)

  2. Section of Medical Informatics, Division of General Medicine, Department of Medicine, School of Medicine, University of Alabama, 1675 University Boulevard, Room 544, Birmingham, AL, 35294-8219, USA

    Eta S. Berner EdD (Professor, Masters of Science in Health) (Professor, Masters of Science in Health)

Rights and permissions

Reprints and permissions

Copyright information

© 1999 Springer Science+Business Media New York

About this chapter

Cite this chapter

Berner, E.S. (1999). Testing System Accuracy. In: Berner, E.S. (eds) Clinical Decision Support Systems. Health Informatics. Springer, New York, NY. https://doi.org/10.1007/978-1-4757-3903-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-1-4757-3903-9_3

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4757-3905-3

  • Online ISBN: 978-1-4757-3903-9

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation