Abstract
Despite the recognized value of machine learning (ML) techniques and high expectation of applying ML techniques within various applications, significant barriers to widespread adoption and local implementation of ML approaches still exist in the areas of trust (of ML results), comprehension (of ML processes) and related workload, as well as confidence (in decision making based on ML results) by users. This paper argues that the revealing of human cognition states with a multimodal interface during ML-based data analytics-driven decision making could provide a rich view for both ML researchers and domain experts to learn the effectiveness of ML technologies in applications. On the one hand, human cognition states could help understand to what degree users accept innovative technologies. On the other hand, through understanding human cognition states during data analytics-driven decision making, ML-based decision attributes and even ML models can be adaptively refined in order to make ML transparent. The paper also identifies examples of impact challenges and obstacles, as well as high-demand research directions in making ML transparent.
Similar content being viewed by others
References
Zhou J, Khawaja MA, Li Z et al (2016) Making machine learning useable by revealing internal states update—a transparent approach. Int J Comput Sci Eng 13:378–389
Zhou J, Bridon C, Chen F et al (2015) Be informed and be involved: effects of uncertainty and correlation on user’s confidence in decision making. In: Proceedings of ACM SIGCHI conference on human factors in computing systems (CHI2015) works-in-progress
Zhou J, Sun J, Chen F et al (2015) Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans Comput Hum Interact 21:33
Helgee EA (2010) Improving drug discovery decision making using machine learning and graph theory in QSAR modeling. Ph.D. thesis, University of Gothenburg
Wang W, Li Z, Wang Y, Chen F (2013) Indexing cognitive workload based on pupillary response under luminance and emotional changes. In: Proceedings of the 2013 international conference on intelligent user interfaces. ACM, New York, NY, USA, pp 247–256
Holzinger A (2013) Human–computer interaction and knowledge discovery (HCI-KDD): What is the benefit of bringing those two fields to work together? In: Cuzzocrea A, Kittl C, Simos DE et al (eds) Availability, reliability, and security in information systems and HCI. Springer, Berlin, pp 319–328
Khawaji A, Chen F, Marcus N, Zhou J (2013) Trust and cooperation in text-based computer-mediated communication. In: Proceedings of the 25th Australian computer–human interaction conference: augmentation, application, innovation, collaboration. ACM, New York, NY, USA, pp 37–40
Zhou J, Chen F (2015) Making machine learning useable. Int J Intell Syst Technol Appl 14:91. doi:10.1504/IJISTA.2015.074069
Moll J, De Oliveira-Souza R, Zahn R (2008) The neural basis of moral cognition: sentiments, concepts, and values. Ann N Y Acad Sci 1124:161–180. doi:10.1196/annals.1440.005
van Gog T, Paas F, Marcus N et al (2008) The mirror neuron system and observational learning: implications for the effectiveness of dynamic visualizations. Educ Psychol Rev 21:21–30. doi:10.1007/s10648-008-9094-3
Ayres P, Paas F (2009) Interdisciplinary perspectives inspiring a new generation of cognitive load research. Educ Psychol Rev 21:1–9. doi:10.1007/s10648-008-9090-7
Chen F, Zhou J, Wang Y et al (2016) Robust multimodal cognitive load measurement. Springer, Berlin
Krueger F, McCabe K, Moll J et al (2007) Neural correlates of trust. PNAS 104:20084–20089. doi:10.1073/pnas.0710103104
Aimone JA, Houser D, Weber B (2014) Neural signatures of betrayal aversion: an fMRI study of trust. Proc Biol Sci 281:20132127. doi:10.1098/rspb.2013.2127
Hahn T, Notebaert K, Anderl C et al (2015) How to trust a perfect stranger: predicting initial trust behavior from resting-state brain-electrical connectivity. Soc Cogn Affect Neurosci 10:809–813. doi:10.1093/scan/nsu122
Heekeren HR, Marrett S, Ungerleider LG (2008) The neural systems that mediate human perceptual decision making. Nat Rev Neurosci 9:467–479. doi:10.1038/nrn2374
Smith LJ, Anand P, Benattayallah A, Hodgson TL (2015) An fMRI investigation of moral cognition in healthcare decision making. J Neurosci Psychol Econ 8:116–133. doi:10.1037/npe0000038
White TP, Engen NH, Sørensen S et al (2014) Uncertainty and confidence from the triple-network perspective: voxel-based meta-analyses. Brain Cogn 85:191–200. doi:10.1016/j.bandc.2013.12.002
Franco-Watkins AM, Johnson JG (2011) Applying the decision moving window to risky choice: comparison of eye-tracking and mouse-tracing methods. Judgm Decis Mak 6:740–749
Schall JD (2001) Neural basis of deciding, choosing and acting. Nat Rev Neurosci 2:33–42
Guetl C, Pivec M, Trummer C et al (2005) ADELE (adaptive e-learning with eye-tracking): theoretical background, system architecture and application scenarios. Eur J Open Distance E Learn 2: 1–8
Arshad S, Wang Y, Chen F (2015) Interactive mouse stream as real-time indicator of user’s cognitive load. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems, pp 1025–1030
Khawaja MA, Chen F, Marcus N (2014) Measuring cognitive load using linguistic features: implications for usability evaluation and adaptive interaction design. Int J Hum Comput Interact 30:343–368. doi:10.1080/10447318.2013.860579
Khawaji A, Chen F, Zhou J, Marcus N (2014) Trust and cognitive load in the text-chat environment: the role of mouse movement. In: Proceedings of the 26th Australian computer–human interaction conference on designing futures: the future of design, pp 324–327
Jenkins Q, Jiang X (2010) Measuring trust and application of eye tracking in human robotic interaction. In: Proceedings of the 2010 industrial engineering research conference
Preuschoff K, Hart BM, Einhäuser W (2011) Pupil dilation signals surprise: evidence for noradrenaline’s role in decision making. Front Neurosci 5:115. doi:10.3389/fnins.2011.00115
Fiedler S, Glockner A (2012) The dynamics of decision making in risky choice: an eye-tracking analysis. Front Psychol 3:1–18
Li Z, Zhang B, Wang Y et al (2014) Water pipe condition assessment: a hierarchical beta process approach for sparse incident data. Mach Learn 95:11–26
Peper E, Harvey R, Lin I-M et al (2007) Is there more to blood volume pulse than heart rate variability respiratory sinus arrhythmia, and cardiorespiratory synchrony? Biofeedback 35:54–61
Oppenheim AV, Schafer RW (2010) Discrete-time signal processing. Pearson, Upper Saddle River
Healey J, Picard R (2000) SmartCar: detecting driver stress. In: Proceedings of 15th international conference on pattern recognition 2000. pp 218–221
Zhai J, Barreto AB, Chin C, Li C (2005) Realization of stress detection using psychophysiological signals for improvement of human–computer interactions. Proc IEEE SoutheastCon 2005:415–420
Kristal-Boneh E, Raifel M, Froom P, Ribak J (1995) Heart rate variability in health and disease. Scand J Work Environ Health 21:85–95
Karvonen MJ, Kentala E, Mustala O (1957) The effects of training on heart rate: a longitudinal study. Ann Med Exp Biol Fenn 35:307–315
Tanaka H, Monahan KD, Seals DR (2001) Age-predicted maximal heart rate revisited. J Am Coll Cardiol 37:153–156
Acknowledgements
Authors thank all volunteer participants for the experiment. This work was supported in part by the Asian Office of Aerospace Research & Development (AOARD) under grant No. FA2386-14-1-0022 AOARD 134131.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, J., Chen, F. DecisionMind: revealing human cognition states in data analytics-driven decision making with a multimodal interface. J Multimodal User Interfaces 12, 67–76 (2018). https://doi.org/10.1007/s12193-017-0249-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-017-0249-8