Abstract
In typical human interactions emotional states are communicated via a variety of modalities such as auditory (through speech), visual (through facial expressions) and kinesthetic (through gestures). However, one or more modalities might be compromised in some situations, as in the case of facial masking in Parkinson’s disease (PD). In these cases, we need to focus the communication and detection of emotions on the reliable modalities, by inferring emotions from what is being said, and compensate for the modalities that are problematic, by having another agent (e.g., a robot) provide the missing facial expressions. We describe the initial development stage of a robot companion that can assist the communication and detection of emotions in interactions where some modalities are totally or partially compromised. Such is the case for people living with Parkinson’s disease.
Our approach is based on a Latent Dirichlet Allocation topic model as a principled way to extract features from speech based on a trained classifier that can be linked to measures of emotion. The trained model is integrated into a robotic cognitive architecture to perform real-time, continuous speech detection of positive, negative, or neutral emotional valence that is expressed through the facial features of a humanoid robot. To evaluate the integrated system, we conducted a human-robot interaction experiment in which the robot credibly detected and displayed emotions as it listened to utterances spoken by a confederate. The utterance were directly extracted from interviews with people with Parkinson’s Disease. The encouraging results will form the basis for further developments of finer prediction models to be employed in a companion robot for persons with PD.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Balahur, A.: Sentiment analysis in social media texts. In: Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 120–128 (2013)
Balahur, A., Hermida, J.M., Montoyo, A.: Building and exploiting emotinet, a knowledge base for emotion detection based on the appraisal theory model. IEEE Trans. Affect. Comput. 3(1), 88–101 (2011)
Calvo, R.A., D’Mello, S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1(1), 18–37 (2010). https://doi.org/10.1109/T-AFFC.2010.1
Cambria, E.: Affective computing and sentiment analysis. IEEE Intell. Syst. 31(2), 102–107 (2016). https://doi.org/10.1109/MIS.2016.31
Chuang, Z.J., Wu, C.H.: Multi-modal emotion recognition from speech and text. Int. J. Comput. Linguist. Chin. Lang. Proces. 9(2), 45–62 (2004). Special Issue on New Trends of Speech and Language Processing
DeGroat, E., Lyons, K.D., Tickle-Degnen, L.: Verbal content during favorite activity interview as a window into the identity of people with Parkinson’s disease. Occup. Ther. J. Res. Occup. Participation Health 26(2) (2006)
Hutto, C., Gilbert, E.: VADER: a parsimonious rule-based model for sentiment analysis of social media text. In: International AAAI Conference on Web and Social Media (2014)
Mower, E., Mataric, M.J., Narayanan, S.: A framework for automatic human emotion classification using emotion profiles. IEEE Trans. Audio Speech Lang. Process. 19(5), 1057–1070 (2011). https://doi.org/10.1109/TASL.2010.2076804
Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up?: Sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP 2002, pp. 79–86. Association for Computational Linguistics, Stroudsburg (2002). https://doi.org/10.3115/1118693.1118704
Pang, B., Lee, L., et al.: Opinion mining and sentiment analysis. Found. Trends® Inf. Retrieval 2(1–2), 1–135 (2008)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Shah, M., Miao, L., Chakrabarti, C., Spanias, A.: A speech emotion recognition framework based on latent dirichlet allocation: algorithm and FPGA implementation. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2553–2557, May 2013. https://doi.org/10.1109/ICASSP.2013.6638116
Shah, M., Chakrabarti, C., Spanias, A.: Within and cross-corpus speech emotion recognition using latent topic model-based features. EURASIP J. Audio Speech Music Process. 2015(1), 4 (2015). https://doi.org/10.1186/s13636-014-0049-y
Svetlana, K., Xiaodan, Z., Saif, M.: Sentiment analysis of short informal texts. J. Artif. Intell. Res. 50, 723–762 (2014)
Takahashi, K., Tickle-Degnen, L., Coster, W.J., Latham, N.K.: Expressive behavior in Parkinson’s disease as a function of interview context. Am. J. Occup. Ther. 64(3), 484–495 (2010)
Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010). https://doi.org/10.1177/0261927X09351676
Tickle-Degnen, L., Ellis, T., Saint-Hilaire, M., Thomas, C., Wagenaar, R.C.: Self-management rehabilitation and health-related quality of life in Parkinson’s disease: a randomized controlled trial. Mov. Disord. 25, 194–204 (2010)
Valenti, A.P., Chita-Tegmark, M., Law, T., Bock, A.W., Oosterveld, B., Scheutz, M.: Using topic modeling to infer emotional state: when your face and tone of voice don’t say it all: inferring emotional state from word semantics and conversational topics. In: Workshop on Cognitive Architectures for HRI: Embodied Models of Situated Natural Language Interactions at AAMAS 2019, Montreal, Canada, May 2019
Valenti, A.P., Chita-Tegmark, M., Tickle-Degnen, L., Bock, A.W., Scheutz, M.J.: Using topic modeling to infer the emotional state of people living with Parkinson’s disease. Assistive Technol., 1–10 (2019). https://doi.org/10.1080/10400435.2019.1623342
Acknowledgements
The work is supported by the Center for Scientific Review under Grant No.: R01-NAG21152 and the National Science Foundation under Grant No.: IIS-1316809.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Valenti, A., Chita-Tegmark, M., Gold, M., Law, T., Scheutz, M. (2019). In Their Own Words: A Companion Robot for Detecting the Emotional State of Persons with Parkinson’s Disease. In: Salichs, M., et al. Social Robotics. ICSR 2019. Lecture Notes in Computer Science(), vol 11876. Springer, Cham. https://doi.org/10.1007/978-3-030-35888-4_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-35888-4_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-35887-7
Online ISBN: 978-3-030-35888-4
eBook Packages: Computer ScienceComputer Science (R0)