Abstract
Recently there has been arising interest in automatically recognizing nonverbal behaviors that are linked with psychological conditions. Work in this direction has shown great potential for cases such as depression and post-traumatic stress disorder (PTSD), however most of the times gender differences have not been explored. In this paper, we show that gender plays an important role in the automatic assessment of psychological conditions such as depression and PTSD. We identify a directly interpretable and intuitive set of predictive indicators, selected from three general categories of nonverbal behaviors: affect, expression variability and motor variability. For the analysis, we employ a semi-structured virtual human interview dataset which includes 53 video recorded interactions. Our experiments on automatic classification of psychological conditions show that a gender-dependent approach significantly improves the performance over a gender agnostic one.
Similar content being viewed by others
Notes
Sample interaction between the virtual agent and a human actor can be seen here: http://www.youtube.com/watch?v=ejczMs6b1Q4.
References
Baltrusaitis T, Robinson P, Morency L (2012) 3d constrained local model for rigid and non-rigid facial tracking. In: CVPR, 2012 IEEE Conference
Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on FG 2011
Ekman P, Matsumoto D, Friesen WV (1997) Facial expression in affective disorders. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS) 2:331–342
Ellgring H (1989) Nonverbal communication in depression. Cambridge University Press, Cambridge
Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, De la Torre F (2009) Detecting depression from facial actions and vocal prosody. In: 3rd international conference on ACII 2009, pp 1–7
McIntyre G, Gocke R, Hyett M, Green M, Breakspear M (2009) An approach for automatically measuring facial activity in depressed subjects. In: 3rd international conference on ACII 2009
Maddage MC, Senaratne R, Low L-SA, Lech M, Allen N (2009) Video-based detection of the clinical depression in adolescents. In: EMBC 2009. Annual International Conference of the IEEE, pp 3723–3726
Diagnostic and Statistical Manual of Mental Disorders DSM-IV-TR Fourth Edition (Text Revision). Amer Psychiatric Pub, 4th edition, July 2000
Diagnostic and Statistical Manual of Mental Disorders DSM-V Fifth Edition. American Psychiatric Association, 5th edition, June 2013
Russell JA, Barrett LF (1999) Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J Personal Soc Psychol 76(5):805–819
Ian Reed L, Sayette MA, Cohn JF (2007) Impact of depression on response to comedy: a dynamic facial coding analysis. J Abnorm Psychol 116:804–809
Kirsch A, Brunnhuber S (2007) Facial expression and experience of emotions in psychodynamic interviews with patients with ptsd in comparison to healthy subjects. Psychopathology 40(5):296–302
Troisi A, Moles A (1999) Gender differences in depression: an ethological study of nonverbal behavior during interviews. J Psychiat Res 33(3):243–250
Wolfsdorf BA, Mattia JI, Zlotnick C, Zimmerman M (2001) Gender differences in patients with posttraumatic stress disorder in a general psychiatric practice. Am J Psychiatry 158(11):1923–1925
Draijer N, Gersons BP, Olff M, Langeland W (2007) Gender differences in posttraumatic stress disorder. Psychol Bull 133(2):183–204
Green B (2003) Post-traumatic stress disorder: symptom profiles in men and women. Curr Med Res Opin 19(3):200–204
Cummins N, Joshi J, Dhall A, Sethu V, Goecke R, Epps J (2013) Diagnosis of depression by behavioural signals: a multimodal approach. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge, AVEC ’13, pp 11–20, New York, NY, USA, 2013. ACM
Meng H, Huang D, Wang H, Yang H, AI-Shuraifi M, Wang Y (2013) Depression recognition based on dynamic facial and vocal expression features using partial least square regression. In: Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge, AVEC ’13, pp 21–30, New York, NY, USA, 2013. ACM
Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP Nonverbal social withdrawal in depression: evidence from manual and automatic analyses. Image and Vision Computing
D’Mello S, Graesser A (2012) Dynamics of affective states during complex learning. Learn Instr 22(2):145–157
Joshi J, Dhall A, Goecke R, Cohn JF (2013) Relative body parts movement for automatic depression analysis. In: Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, ACII ’13, pp 492–497, Washington, DC, USA, 2013. IEEE Computer Society
Scherer S, Stratou G, Mahmoud M, Boberg J, Gratch J, Rizzo A, Morency L-P (2013) Automatic behavior descriptors for psychological disorder analysis. In: IEEE conference on automatic face and gesture recognition, 2013
Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA (1996) Psychometric properties of the ptsd checklist (pcl). Behav Res Therapy 34(8):669–673
Kroenke K, Spitzer RL (2002) The phq-9: a new depression and diagnostic severity measure. Psychiatr Ann 32:509–521
Kroenke K, Spitzer RL, Williams JBW (2001) The phq-9. J Gen Intern Med 16(9):606–613
Bolton EE, Gray MJ, Litz BT (2006) A cross-lagged analysis of the relationship between symptoms of ptsd and retrospective reports of exposure. J Anxiety Disord 20(7):877–895
Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, Koffman RL (2004) Combat duty in iraq and afghanistan, mental health problems, and barriers to care. N Engl J Med 351(1):13–22
Andrykowski MA, Cordova MJ, Studts JL, Miller TW (1998) Posttraumatic stress disorder after treatment for breast cancer: prevalence of diagnosis and use of the ptsd checklistcivilian version (pclc) as a screening instrument. J Consult Clin Psychol 66(3):586–590
Stefan S, Giota S, Jonathan G, Morency L-P (2013) Investigating voice quality as a speaker-independent indicator of depression and ptsd. In: Proceedings of interspeech
DeVault D, Georgila K, Artstein R, Morbini F, Traum D, Scherer S, Rizzo A, Morency L-P (2013) Verbal indicators of psychological distress in interactive dialogue with a virtual human. In SIGDIAL, Metz, France, August
Ekman P, Rosenberg EL (2005) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS) (series in affective Science). Oxford UP, USA, New York
Bonanno GA, Keltner D (1997) Facial expressions of emotion and the course of conjugal bereavement. J Abnorm Psychol 106:126–137
Morency L, Whitehill J, Movellan J (2008) Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In: 8th IEEE International Conference on FG ’08
Kublbeck C, Ernst A (2006) Face detection and tracking in video sequences using the modifiedcensus transformation. Image Vis Comput 24(6):564–572
Hedges LV (1981) Distribution theory for glass’s estimator of effect size and related estimators. J Educ Behav Stat 6(2):107–128
Deutsch FM, LeBaron D, Fryer MM (1987) What is in a smile? Psychol Women Q 11(3):341–352
Hess U, Adams RB, Kleck RE Jr (2004) Facial appearance, gender, and emotion expression. Emotion 4(4):378
Adams R Jr, Kleck R, Hess U (2005) Who may frown and who should smile? dominance, affiliation, and the display of happiness and anger
Krämer NC, Hoffmann L, Kopp S (2010) Know your users! empirical results for tailoring an agent’s nonverbal behavior to different user groups. In: Proceedings of IVA, IVA’10, pp 468–474
Kulms P, Kramer NC, Gratch J, Kang S-H (2011) It’s in their eyes: a study on female and male virtual humans gaze. In: IVA, volume 6895 of, Lecture Notes in Computer Science, pp 80–92
Rief W, Nanke A, Klaiberg A, Braehler E (2004) Base rates for panic and depression according to the brief patient health questionnaire: a population-based study. J Affect Disord 82(2):271–276
Campbell DG, Felker BL, Liu C-F, Yano EM, Kirchner JE, Chan D, Rubenstein LV, Chaney EF (2007) Prevalence of depressionptsd comorbidity: implications for clinical practice guidelines and primary care-based interventions. J Gen Intern Med 22(6):711–718
Acknowledgments
This work is supported by DARPA under contract (W911NF-04-D-0005) and U.S. Army Research, Development, and Engineering Command. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Stratou, G., Scherer, S., Gratch, J. et al. Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender. J Multimodal User Interfaces 9, 17–29 (2015). https://doi.org/10.1007/s12193-014-0161-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-014-0161-4