Skip to main content
SpringerLink
Log in

Minding the AI: Ethical Challenges and Practice for AI Mental Health Care Tools

  • Chapter
  • First Online:
Artificial Intelligence in Brain and Mental Health: Philosophical, Ethical & Policy Issues

Part of the book series: Advances in Neuroethics ((AIN))

Abstract

The use of artificial intelligence (AI) for mental health raises ethical challenges regarding bias, privacy, and potential impact on fiduciary obligations in the therapeutic relationship. Health tools utilizing AI present particular challenges for regulation, both in terms of the technical aspects of evaluating the algorithms and because many of the applications may be used outside of healthcare settings, and thus are outside of the traditional frameworks for regulation of health issues. Bias can enter into AI tools at different stages, such as the data collection and preparation stages, as well as into the way that issues are framed and presented for AI. Addressing bias and fairness in mental health applications of AI is important to avoid results that reflect and reinforce existing social problems and inequality in mental health care. At the same time, AI can present opportunities for addressing existing inequities in mental health care. Protection of patients and other users of mental health AI tools from potential misuse of their health information and potential negative repercussions from sharing their data is another critical area of concern. Finally, AI tools present challenges for the fiduciary obligations generally expected in the therapeutic relationship. It will be necessary to carefully consider likely areas of concern in order to formulate processes for integrating these tools appropriately into mental health care.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Yu K-H, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018;2(10):719. https://doi.org/10.1038/s41551-018-0305-z.

    Article  PubMed  Google Scholar 

  2. Price N. Artificial intelligence in health care: applications and legal issues. The Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics at Harvard Law School. https://petrieflom.law.harvard.edu/resources/article/artificial-intelligence-in-health-care-applications-and-legal-issues. Accessed 2 Mar 2019.

  3. Libbrecht MW, Noble WS. Machine learning applications in genetics and genomics. Nat Rev Genet. 2015;16(6):321–32. https://doi.org/10.1038/nrg3920.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Bzdok D, Meyer-Lindenberg A. Machine learning for precision psychiatry. ArXiv:1705.10553 [Stat]. 2017. http://arxiv.org/abs/1705.10553.

  5. Rose S. Machine learning for prediction in electronic health data. JAMA Netw Open. 2018;1(4):e181404. https://doi.org/10.1001/jamanetworkopen.2018.1404.

    Article  PubMed  Google Scholar 

  6. Scalable and accurate deep learning with electronic health records. npj Digital Medicine. n.d. https://www.nature.com/articles/s41746-018-0029-1. Accessed 29 Aug 2019.

  7. Hao B, Li L, Li A, Zhu T. Predicting mental health status on social media. In: Rau PLP, editor. Cross-cultural design. Cultural differences in everyday life. Berlin Heidelberg: Springer; 2013. p. 101–10.

    Chapter  Google Scholar 

  8. Ngiam KY, Khor IW. Big data and machine learning algorithms for health-care delivery. Lancet Oncol. 2019;20(5):e262–73. https://doi.org/10.1016/S1470-2045(19)30149-4.

    Article  PubMed  Google Scholar 

  9. Mols B. In black box algorithms we trust (or do we?). https://cacm.acm.org/news/214618-in-black-box-algorithms-we-trust-or-do-we/fulltext. Accessed 31 Aug 2019.

  10. Price WN. Regulating black-box medicine. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network; 2017. https://papers.ssrn.com/abstract=2938391.

  11. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform. 2009;42(5):760–72. https://doi.org/10.1016/j.jbi.2009.08.007.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Jackson RG, Patel R, Jayatilleke N, Kolliakou A, Ball M, Gorrell G, Roberts A, Dobson RJ, Stewart R. Natural language processing to extract symptoms of severe mental illness from clinical text: the clinical record interactive search comprehensive data extraction (CRIS-CODE) project. BMJ Open. 2017;7(1):e012012. https://doi.org/10.1136/bmjopen-2016-012012.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Cook BL, Progovac AM, Chen P, Mullin B, Hou S, Baca-Garcia E. Novel use of natural language processing (NLP) to predict suicidal ideation and psychiatric symptoms in a text-based mental health intervention in Madrid [Research article]. 2016. https://doi.org/10.1155/2016/8708434.

  14. Althoff T, Clark K, Leskovec J. Large-scale analysis of counseling conversations: an application of natural language processing to mental health. Trans Assoc Comput Linguist. 2016;4:463–76. https://doi.org/10.1162/tacl_a_00111.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Denecke K, May R, Deng Y. Towards emotion-sensitive conversational user interfaces in healthcare applications. Stud Health Technol Inform. 2019;264:1164–8. https://doi.org/10.3233/SHTI190409.

    Article  PubMed  Google Scholar 

  16. Miner A, Chow A, Adler S, Zaitsev I, Tero P, Darcy A, Paepcke A. Conversational agents and mental health: theory-informed assessment of language and affect. In: Proceedings of the fourth international conference on human agent interaction, 123–130. HAI ‘16. New York, NY: ACM; 2016. https://doi.org/10.1145/2974804.2974820.

  17. Luxton DD. Chapter 1—An introduction to artificial intelligence in behavioral and mental health care. In: Luxton DD, editor. Artificial intelligence in behavioral and mental health care; 2016. p. 1–26. https://doi.org/10.1016/B978-0-12-420248-1.00001-5.

    Chapter  Google Scholar 

  18. Patel UK, Anwar A, Saleem S, Malik P, Rasul B, Patel K, et al. Artificial intelligence as an emerging technology in the current care of neurological disorders. J Neurol. 2019; https://doi.org/10.1007/s00415-019-09518-3.

  19. Rothstein MA. Health privacy in the electronic age. J Leg Med. 2007;28(4):487–501. https://doi.org/10.1080/01947640701732148.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Martinez-Martin N. What are important ethical implications of using facial recognition technology in health care? AMA J Ethics. 2019;21(2):180–7. https://doi.org/10.1001/amajethics.2019.180.

    Article  Google Scholar 

  21. Bennett CC, Doub TW. Chapter 2—Expert systems in mental health care: AI applications in decision-making and consultation. In: Luxton DD, editor. Artificial intelligence in behavioral and mental health care; 2016. p. 27–51. https://doi.org/10.1016/B978-0-12-420248-1.00002-7.

    Chapter  Google Scholar 

  22. Masri RY, Jani HM. Employing artificial intelligence techniques in Mental Health Diagnostic Expert System. In: 2012 international conference on computer information science (ICCIS), vol. 1. 2012. p. 495–99. https://doi.org/10.1109/ICCISci.2012.6297296.

  23. Singh VK, Shrivastava U, Bouayad L, Padmanabhan B, Ialynytchev A, Schultz SK. Machine learning for psychiatric patient triaging: an investigation of cascading classifiers. J Am Med Inform Assoc JAMIA. 2018;25(11):1481–7. https://doi.org/10.1093/jamia/ocy109.

    Article  PubMed  Google Scholar 

  24. Koh HC, Tan G. Data mining applications in healthcare. J Healthcare Inform Manag JHIM. 2005;19(2):64–72.

    Google Scholar 

  25. Vayena E, Blasimme A, Cohen IG. Machine learning in medicine: addressing ethical challenges. PLoS Med. 2018;15(11):e1002689. https://doi.org/10.1371/journal.pmed.1002689.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Char DS, Shah NH, Magnus D. Implementing machine learning in health care—addressing ethical challenges. N Engl J Med. 2018;378(11):981–3. https://doi.org/10.1056/NEJMp1714229.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Laranjo L, Dunn AG, Tong HL, Kocaballi AB, Chen J, Bashir R, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc. 2018;25(9):1248–58. https://doi.org/10.1093/jamia/ocy072.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Mental Health. 2017;4(2):e19.

    Article  Google Scholar 

  29. Riek LD. Chapter 8—Robotics technology in mental health care. In: Luxton DD, editor. Artificial intelligence in behavioral and mental health care. San Diego: Academic Press; 2016. p. 185–203. https://doi.org/10.1016/B978-0-12-420248-1.00008-8.

    Chapter  Google Scholar 

  30. Robins B, Dautenhahn K. Tactile interactions with a humanoid robot: novel play scenario implementations with children with autism. Int J Soc Robot. 2014;6(3):397–415. https://doi.org/10.1007/s12369-014-0228-0.

    Article  Google Scholar 

  31. Vanderborght B, Simut R, Saldien J, Pop C, Rusu AS, Pintea S, Lefeber D, David DO. Using the social robot Probo as a social story telling agent for children with ASD. Interact Stud. 2012;13(3):348–72. https://doi.org/10.1075/is.13.3.02van.

    Article  Google Scholar 

  32. Miner AS, Milstein A, Hancock JT. Talking to machines about personal mental health problems. JAMA. 2017; https://doi.org/10.1001/jama.2017.14151.

  33. Lányi CS. Virtual reality in healthcare. In: Ichalkaranje N, Ichalkaranje A, Jain LC, editors. Intelligent paradigms for assistive and preventive healthcare; 2006. p. 87–116. https://doi.org/10.1007/11418337_3.

    Chapter  Google Scholar 

  34. Virtual reality might be the next big thing for mental health. n.d. Scientific American Blog Network website: https://blogs.scientificamerican.com/observations/virtual-reality-might-be-the-next-big-thing-for-mental-health/. Accessed 20 Aug 2019.

  35. Anderson PL, Price M, Edwards SM, Obasaju MA, Schmertz SK, Zimand E, Calamaras MR. Virtual reality exposure therapy for social anxiety disorder: a randomized controlled trial. J Consult Clin Psychol. 2013;81(5):751–60. https://doi.org/10.1037/a0033559.

    Article  PubMed  Google Scholar 

  36. Insel TR. Digital phenotyping: technology for a new science of behavior. JAMA. 2017;318(13):1215–6. https://doi.org/10.1001/jama.2017.11295.

    Article  PubMed  Google Scholar 

  37. Onnela J-P, Rauch SL. Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology. 2016;41(7):1691–6. https://doi.org/10.1038/npp.2016.7.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Torous J, Staples P, Barnett I, Sandoval LR, Keshavan M, Onnela J-P. Characterizing the clinical relevance of digital phenotyping data quality with applications to a cohort with schizophrenia. Npj Digit Med. 2018;1(1):15. https://doi.org/10.1038/s41746-018-0022-8.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Jain SH, Powers BW, Hawkins JB, Brownstein JS. The digital phenotype. Nat Biotechnol. 2015;33(5):462–3. https://doi.org/10.1038/nbt.3223.

    Article  CAS  PubMed  Google Scholar 

  40. Kantrowitz L. When Facebook and Instagram think you’re depressed. 2017. Vice website: https://www.vice.com/en_us/article/pg7d59/when-facebook-and-instagram-thinks-youre-depressed. Accessed 26 Oct 2017.

  41. Dans E. The rise of real-time, context-based insurance. n.d. Forbes website: https://www.forbes.com/sites/enriquedans/2017/03/12/the-rise-of-real-time-context-based-insurance/. Accessed 29 Sept 2018.

  42. Martinez-Martin N, Insel TR, Dagum P, Greely HT, Cho MK. Data mining for health: staking out the ethical territory of digital phenotyping. Npj Digit Med. 2018;1(1):68. https://doi.org/10.1038/s41746-018-0075-8.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Cortez NG, Cohen IG, Kesselheim AS. FDA regulation of mobile health technologies. N Engl J Med. 2014;371(4):372–9. https://doi.org/10.1056/NEJMhle1403384.

    Article  CAS  PubMed  Google Scholar 

  44. Center for Devices and Radiological Health. Digital Health [WebContent]. n.d. FDA.gov website: https://www.fda.gov/medicaldevices/digitalhealth/. Accessed 20 Feb 2018.

  45. Glenn T, Monteith S. Privacy in the digital world: medical and health data outside of HIPAA protections. Curr Psychiatry Rep. 2014;16(11):494. https://doi.org/10.1007/s11920-014-0494-4.

    Article  PubMed  Google Scholar 

  46. Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open. 2019;2(4):e192542. https://doi.org/10.1001/jamanetworkopen.2019.2542.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Bloss C, Nebeker C, Bietz M, Bae D, Bigby B, Devereaux M, et al. Reimagining human research protections for 21st century science. J Med Internet Res. 2016;18(12):e329. https://doi.org/10.2196/jmir.6634.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Danks D, London AJ. Algorithmic bias in autonomous systems. In: Proceedings of the 26th international joint conference on artificial intelligence. 2017. p. 4691–7. http://dl.acm.org/citation.cfm?id=3171837.3171944.

  49. Mittelstadt BD, Floridi L. The ethics of big data: current and foreseeable issues in biomedical contexts. Sci Eng Ethics. 2016;22(2):303–41. https://doi.org/10.1007/s11948-015-9652-2.

    Article  PubMed  Google Scholar 

  50. Jha S, Topol EJ. Adapting to artificial intelligence: radiologists and pathologists as information specialists. JAMA. 2016;316(22):2353–4. https://doi.org/10.1001/jama.2016.17438.

    Article  PubMed  Google Scholar 

  51. Luxton DD. Artificial intelligence in psychological practice: current and future applications and implications. Prof Psychol Res Pract. 2014;45(5):332–9. https://doi.org/10.1037/a0034559.

    Article  Google Scholar 

  52. Sucala M, Schnur JB, Constantino MJ, Miller SJ, Brackman EH, Montgomery GH. The therapeutic relationship in e-therapy for mental health: a systematic review. Journal of Medical Internet Research. 2012;14(4). https://doi.org/10.2196/jmir.2084.

  53. Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. J Nerv Ment Dis. 2017;205(1):4–8. https://doi.org/10.1097/NMD.0000000000000596.

    Article  PubMed  Google Scholar 

  54. Rendina HJ, Mustanski B. Privacy, trust, and data sharing in web-based and mobile research: participant perspectives in a large nationwide sample of men who have sex with men in the United States. J Med Internet Res. 2018;20(7):e233. https://doi.org/10.2196/jmir.9019.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Nebeker C, Lagare T, Takemoto M, et al. Engaging research participants to inform the ethical conduct of mobile imaging, pervasive sensing, and location tracking research. Transl Behav Med. 2016;6(4):577–86. https://doi.org/10.1007/s13142-016-0426-4.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Martinez-Martin N, Kreitmair K. Ethical issues for direct-to-consumer digital psychotherapy apps: addressing accountability, data protection, and consent. JMIR Mental Health. 2018;5(2). https://doi.org/10.2196/mental.9423.

  57. Chan S, Torous J, Hinton L, Yellowlees P. Towards a framework for evaluating mobile mental health apps. Telemed J E-Health: Offic J Am Telemed Assoc. 2015;21(12):1038–41. https://doi.org/10.1089/tmj.2015.0002.

    Article  Google Scholar 

  58. Center for Devices and Radiological Health. Digital health—digital health software precertification (Pre-Cert) program [WebContent]. n.d. https://www.fda.gov/MedicalDevices/DigitalHealth/UCM567265. Accessed 2 Aug 2018.

  59. Koene A. Algorithmic bias: addressing growing concerns [leading edge]. IEEE Technol Soc Mag. 2017;36(2):31–2. https://doi.org/10.1109/MTS.2017.2697080.

    Article  Google Scholar 

  60. Cohen IG, Amarasingham R, Shah A, Xie B, Lo B. The legal and ethical concerns that arise from using complex predictive analytics in health care. Health Aff. 2014;33(7):1139–47. https://doi.org/10.1377/hlthaff.2014.0048.

    Article  Google Scholar 

  61. Miner AS, Milstein A, Schueller S, Hegde R, Mangurian C, Linos E. Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Intern Med. 2016;176(5):619–25. https://doi.org/10.1001/jamainternmed.2016.0400.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Torous J, Onnela J-P, Keshavan M. New dimensions and new tools to realize the potential of RDoC: digital phenotyping via smartphones and connected devices. Transl Psychiatry. 2017;7(3):e1053. https://doi.org/10.1038/tp.2017.25.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Glymour B, Herington J. Measuring the biases that matter: the ethical and casual foundations for measures of fairness in algorithms. In: Proceedings of the conference on fairness, accountability, and transparency. FAT* ‘19. Atlanta, GA: Association for Computing Machinery; 2019. p. 269–78. https://doi.org/10.1145/3287560.3287573.

  64. Towards trustable machine learning. Nat Biomed Eng. 2018;2(10):709. https://doi.org/10.1038/s41551-018-0315-x.

  65. Tunkelang D. Ten things everyone should know about machine learning. n.d. Forbes website: https://www.forbes.com/sites/quora/2017/09/06/ten-things-everyone-should-know-about-machine-learning/. Accessed 13 Jan 2018.

  66. Dressel J, Farid H. The accuracy, fairness, and limits of predicting recidivism. Sci Adv. 2018;4(1):eaao5580. https://doi.org/10.1126/sciadv.aao5580.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Winfield A, Halverson M. Artificial intelligence and autonomous systems: why principles matter. n.d. IEEE Future Directions website: http://sites.ieee.org/futuredirections/tech-policy-ethics/september-2017/artificial-intelligence-and-autonomous-systems-why-principles-matter/. Accessed 28 Aug 2019.

  68. Policy recommendations: control and responsible innovation of artificial intelligence. 2018. The Hastings Center website: https://www.thehastingscenter.org/news/policy-recommendations-control-responsible-innovation-artificial-intelligence/. Accessed 5 Dec 2018.

  69. Institute AN. Algorithmic impact assessments: toward accountable automation in public agencies. 2018. Medium website: https://medium.com/@AINowInstitute/algorithmic-impact-assessments-toward-accountable-automation-in-public-agencies-bd9856e6fdde. Accessed 31 Aug 2019.

  70. Kleinberg J, Ludwig J, Mullainathan S, Sunstein CR. Discrimination in the age of algorithms. Journal of Legal Analysis. 2018;10. https://doi.org/10.1093/jla/laz001.

  71. EU General Data Protection Regulation (GDPR): Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016.

    Google Scholar 

  72. California Consumer Privacy Act of 2018.

    Google Scholar 

  73. Wachter S, Mittelstadt B. A right to reasonable inferences: re-thinking data protection law in the age of big data and AI. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network; 2019. https://papers.ssrn.com/abstract=3248829.

  74. Costanza-Chock S. Design justice: towards an intersectional feminist framework for design theory and practice. SSRN Scholarly Paper. Rochester, NY: Social Science Research Network; 2018. https://papers.ssrn.com/abstract=3189696.

  75. Martinez-Martin N, Char D. Surveillance and digital health. Am J Bioeth AJOB. 2018; 18(9):67–8. https://doi.org/10.1080/15265161.2018.1498954.

    Article  PubMed  Google Scholar 

  76. Wachter S, Mittelstadt B. A right to reasonable inferences: re-thinking data protection law in the age of big data and AI (SSRN Scholarly Paper No. ID 3248829). 2019. Social Science Research Network website: https://papers.ssrn.com/abstract=3248829.

  77. Feng E. How China is using facial recognition technology. NPR.Org. n.d. https://www.npr.org/2019/12/16/788597818/how-china-is-using-facial-recognition-technology. Accessed 11 Mar 2020.

  78. China uses DNA to map faces, with help from the west. The New York Times. n.d. https://www.nytimes.com/2019/12/03/business/china-dna-uighurs-xinjiang.html. Accessed 11 Mar 2020.

  79. Conger K, Fausset R, Kovaleski SF. San Francisco bans facial recognition technology. The New York Times. 2019, May 14. https://www.nytimes.com/2019/05/14/us/facial-recognition-ban-san-francisco.html.

  80. Big other: surveillance capitalism and the prospects of an information civilization—Shoshana Zuboff, 2015. n.d. https://journals.sagepub.com/doi/10.1057/jit.2015.5. Accessed 11 Mar 2020.

Download references

Author information

Authors and Affiliations

  1. Stanford Center for Biomedical Ethics, Stanford, CA, USA

    Nicole Martinez-Martin

Authors
  1. Nicole Martinez-Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicole Martinez-Martin .

Editor information

Editors and Affiliations

  1. Medical College of Wisconsin, Center for Bioethics and Medical Humanities, Milwaukee, WI, USA

    Fabrice Jotterand

  2. Department of Health Sciences and Technology, EPFL CDH-DIR, ERANET-NEURON Group Leader, Zürich, Switzerland

    Marcello Ienca

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Martinez-Martin, N. (2021). Minding the AI: Ethical Challenges and Practice for AI Mental Health Care Tools. In: Jotterand, F., Ienca, M. (eds) Artificial Intelligence in Brain and Mental Health: Philosophical, Ethical & Policy Issues. Advances in Neuroethics. Springer, Cham. https://doi.org/10.1007/978-3-030-74188-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74188-4_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74187-7

  • Online ISBN: 978-3-030-74188-4

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation