Skip to main content
SpringerLink
Log in

Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals

  • Conference paper
Biomedical Engineering Systems and Technologies (BIOSTEC 2009)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 52))

Abstract

As is known for centuries, humans exhibit an electrical profile. This profile  is  altered  through various  psychological  and  physiological proce-sses, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Picard, R.W.: Affective Computing. MIT Press, Boston (1997)

    Google Scholar 

  2. van Tulder, M., Malmivaara, A., Koes, B.: Repetitive strain injury. The Lancet 369(9575), 1815–1822 (2007)

    Article  Google Scholar 

  3. Schuler, J.L.H., O’Brien, W.H.: Cardiovascular recovery from stress and hypertension factors: A meta-analytic view. Psychophysiology 34(6), 649–659 (1997)

    Article  Google Scholar 

  4. Frederickson, B.L., Manusco, R.A., Branigan, C., Tugade, M.M.: The undoing effect of positive emotions. Motivation and Emotion 24(4), 237–257 (2000)

    Article  Google Scholar 

  5. Ader, R., Cohen, N., Felten, D.: Psychoneuroimmunology: Interactions between the nervous system and the immune system. The Lancet 345(8942), 99–103 (1995)

    Article  Google Scholar 

  6. Solomon, G.F., Amkraut, A.A., Kasper, P.: Immunity, emotions, and stress with special reference to the mechanisms of stress effects on the immune system. Psychotherapy and Psychosomatics 23(1-6), 209–217 (1974)

    Article  Google Scholar 

  7. Fairclough, S.H.: Fundamentals of physiological computing. Interacting with Computers 21(1-2), 133–145 (2009)

    Article  Google Scholar 

  8. Mauss, I.B., Robinson, M.D.: Measures of emotion: A review. Cognition and Emotion 23(2), 209–237 (2009)

    Article  Google Scholar 

  9. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)

    Article  Google Scholar 

  10. van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisits for Affective Signal Processing (ASP). In: Encarnação, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, Porto – Portugal, pp. 426–433 (2009)

    Google Scholar 

  11. Critchley, H.D., Elliott, R., Mathias, C.J., Dolan, R.J.: Neural activity relating to generation and representation of galvanic skin conductance responses: A functional magnetic resonance imaging study. The Journal of Neuroscience 20(8), 3033–3040 (2000)

    Google Scholar 

  12. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)

    Article  Google Scholar 

  13. James, W.: Review: La pathologie des emotions by Ch. Féré. The Philosophical Review 2(3), 333–336 (1893)

    Article  Google Scholar 

  14. Marwitz, M., Stemmler, G.: On the status of individual response specificity. Psychophysiology 35(1), 1–15 (1998)

    Article  Google Scholar 

  15. Gunes, H., Piccardi, M.: Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics 39(1), 64–84 (2009)

    Article  Google Scholar 

  16. Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Towards practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11), 2106–2111 (2009)

    Article  Google Scholar 

  17. Daly, A.: Movement analysis: Piecing together the puzzle. TDR – The Drama Review: A Journal of Performance Studies 32(4), 40–52 (1988)

    MathSciNet  Google Scholar 

  18. Ververidis, D., Kotropoulos, C.: Emotional speech recognition: Resources, features, and methods. Speech Communication 48(9), 1162–1181 (2006)

    Article  Google Scholar 

  19. Van den Broek, E.L.: Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics) 103, 118–125 (2004)

    Google Scholar 

  20. van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., Tuinenbreijer, K.: Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments 1(3), 287–299 (2009)

    Google Scholar 

  21. Gamboa, H., Silva, F., Silva, H., Falcão, R.: PLUX – Biosignals Acquisition and Processing (2010), http://www.plux.info (Last accessed January 30, 2010)

  22. van den Broek, E.L., Westerink, J.H.D.M.: Considerations for emotion-aware consumer products. Applied Ergonomics 40(6), 1055–1064 (2009)

    Article  Google Scholar 

  23. Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufmann, P.G., Malik, M., Nagaraja, H.N., Porges, S.W., Saul, J.P., Stone, P.H., van der Molen, M.W.: Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology 34(6), 623–648 (1997)

    Article  Google Scholar 

  24. Boucsein, W.: Electrodermal activity. Plenum Press, New York (1992)

    Google Scholar 

  25. Grossman, P., Taylor, E.W.: Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology 74(2), 263–285 (2007)

    Article  Google Scholar 

  26. Fridlund, A.J., Cacioppo, J.T.: Guidelines for human electromyographic research. Psychophysiology 23(5), 567–589 (1986)

    Article  Google Scholar 

  27. Reaz, M.B.I., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online 8(1), 11–35 (2006)

    Article  Google Scholar 

  28. Grandjean, D., Scherer, K.R.: Unpacking the cognitive architecture of emotion processes. Emotion 8(3), 341–351 (2008)

    Article  Google Scholar 

  29. Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., Arnaldi, B.: A review of classification algorithms for EEG-based brain-computer interfaces. Journal of Neural Engineering 4(2), R1–R13 (2007)

    Google Scholar 

  30. Bimber, O.: Brain-Computer Interfaces. IEEE Computer 41(10) (2008); [special issue]

    Google Scholar 

  31. Minsky, M.: The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. Simon & Schuster, New York (2006)

    Google Scholar 

  32. Aarts, E.: Ambient intelligence: Vision of our future. IEEE Multimedia 11(1), 12–19 (2004)

    Article  Google Scholar 

  33. Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)

    Article  Google Scholar 

  34. Liu, C., Rani, P., Sarkar, N.: Human-robot interaction using affective cues. In: Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2006), Hatfield, UK, pp. 285–290. IEEE Computer Society, Los Alamitos (2006)

    Chapter  Google Scholar 

  35. Rani, P., Sims, J., Brackin, R., Sarkar, N.: Online stress detection using psychophysiological signals for implicit human-robot cooperation. Robotica 20(6), 673–685 (2002)

    Article  Google Scholar 

  36. Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology, 3rd edn. Cambridge University Press, New York (2007)

    Book  Google Scholar 

  37. Sinha, R., Parsons, O.A.: Multivariate response patterning of fear. Cognition and Emotion 10(2), 173–198 (1996)

    Article  Google Scholar 

  38. Scheirer, J., Fernandez, R., Klein, J., Picard, R.W.: Frustrating the user on purpose: A step toward building an affective computer. Interacting with Computers 14(2), 93–118 (2002)

    Google Scholar 

  39. Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work 6(1), 4–14 (2003)

    Article  Google Scholar 

  40. Takahashi, K.: Remarks on emotion recognition from bio-potential signals. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Palmerston North, New Zealand, October 5-8, vol. 2, pp. 1655–1659 (2003)

    Google Scholar 

  41. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 36–48. Springer, Heidelberg (2004)

    Google Scholar 

  42. Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing 42(3), 419–427 (2004)

    Article  Google Scholar 

  43. Lisetti, C.L., Nasoz, F.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing 2004(11), 1672–1687 (2004)

    Article  Google Scholar 

  44. Wagner, J., Kim, J., André, E.: From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: Proceedings of the IEEE International Conference on Multimedia and Expo. (ICME), Amsterdam, The Netherlands, July 6-8, pp. 940–943 (2005)

    Google Scholar 

  45. Yoo, S.K., Lee, C.K., Park, J.Y., Kim, N.H., Lee, B.C., Jeong, K.S.: Neural network based emotion estimation using heart rate variability and skin resistance. In: Wang, L., Chen, K., S. Ong, Y. (eds.) ICNC 2005. LNCS, vol. 3610, pp. 818–824. Springer, Heidelberg (2005)

    Google Scholar 

  46. Choi, A., Woo, W.: Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 590–597. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  47. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6(2), 156–166 (2005)

    Article  Google Scholar 

  48. Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis & Applications 9(1), 58–69 (2006)

    Article  Google Scholar 

  49. Zhai, J., Barreto, A.: Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Science Instrumentation 42, 495–500 (2006)

    Google Scholar 

  50. Jones, C.M., Troen, T.: Biometric valence and arousal recognition. In: Thomas, B.H. (ed.) Proceedings of the Australasian Computer-Human Interaction Conference (OzCHI), Adelaide, Australia, pp. 191–194 (2007)

    Google Scholar 

  51. Leon, E., Clarke, G., Callaghan, V., Sepulveda, F.: A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence 20(3), 337–345 (2007)

    Article  Google Scholar 

  52. Liu, C., Conn, K., Sarkar, N., Stone, W.: Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. International Journal of Human-Computer Studies 66(9), 662–677 (2008)

    Article  Google Scholar 

  53. Katsis, C.D., Katertsidis, N., Ganiatsas, G., Fotiadis, D.I.: Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics–Part A: Systems and Humans 38(3), 502–512 (2008)

    Article  Google Scholar 

  54. Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies 66(10), 741–755 (2008)

    Article  Google Scholar 

  55. Task Force: Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 17(3), 354–381 (1996)

    Google Scholar 

  56. Ravenswaaij-Arts, C.M.A.V., Kollee, L.A.A., Hopman, J.C.W., Stoelinga, G.B.A., Geijn, H.P.: Heart rate variability. Annals of Internal Medicine 118(6), 436–447 (1993)

    Google Scholar 

  57. Butler, E.A., Wilhelm, F.H., Gross, J.J.: Respiratory sinus arrhythmia, emotion, and emotion regulation during social interaction. Psychophysiology 43(6), 612–622 (2006)

    Article  Google Scholar 

  58. van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., van Herk, J., Tuinenbreijer, K.: Computing emotion awareness through facial electromyography. In: Huang, T.S., Sebe, N., Lew, M., Pavlović, V., Kölsch, M., Galata, A., Kisačanin, B. (eds.) ECCV 2006 Workshop on HCI. LNCS, vol. 3979, pp. 52–63. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  59. Westerink, J.H.D.M., van den Broek, E.L., Schut, M.H., van Herk, J., Tuinenbreijer, K.: 14. In: Computing emotion awareness through galvanic skin response and facial electromyography. Philips Research Book Series, vol. 8, pp. 137–150. Springer, Dordrecht (2008)

    Google Scholar 

  60. Cacioppo, J., Tassinary, L.: Inferring psychological significance from physiological signals. American Psychologist 45(1), 16–28 (1990)

    Article  Google Scholar 

  61. Mitchell, T.M.: Machine Learning. The McGraw-Hill Companies, Inc., Columbus (1997)

    MATH  Google Scholar 

  62. Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer Science+Business Media, LLC, New York (2006)

    Book  MATH  Google Scholar 

  63. Schölkopf, B., Smola, A.J.: Learning with kernels: Support Vector Machines, Regularization, Optimization, and Beyond. In: Adaptive Computation and Machine Learning. The MIT Press, Cambridge (2002)

    Google Scholar 

  64. Rencher, A.C.: Methods of Multivariate Analysis, 2nd edn. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., New York (2002)

    MATH  Google Scholar 

  65. Rottenberg, J., Ray, R.R., Gross, J.J.: 1. In: Emotion elicitation using films, pp. 9–28. Oxford University Press, New York (2007)

    Google Scholar 

  66. Kreibig, S.D., Wilhelm, F.H., Roth, W.T., Gross, J.J.: Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films. Psychophysiology 44(5), 787–806 (2007)

    Article  Google Scholar 

  67. Kring, A.M., Gordon, A.H.: Sex differences in emotion: Expression, experience, and physiology. Journal of Personality and Social Psychology 74(3), 686–703 (1998)

    Article  Google Scholar 

  68. Carrera, P., Oceja, L.: Drawing mixed emotions: Sequential or simultaneous experiences? Cognition & Emotion 21(2), 422–441 (2007)

    Article  Google Scholar 

  69. Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980)

    Article  Google Scholar 

  70. Cover, T.M., van Campenhout, J.M.: On the possible orderings in the measurement selection problem. IEEE Transactions on Systems, Man, and Cybernetics SMC-7(9), 657–661 (1977)

    Google Scholar 

  71. Lawrence, S., Giles, C.L., Tsoi, A.: What size neural network gives optimal generalization? Convergence properties of backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617 (April 1996)

    Google Scholar 

  72. Barrett, L.F.: Valence as a basic building block of emotional life. Journal of Research in Personality 40, 35–55 (2006)

    Article  MathSciNet  Google Scholar 

  73. Russel, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology 26(5), 805–819 (1999)

    Article  Google Scholar 

  74. Gendolla, G.H.E.: On the impact of mood on behavior: An integrative theory and a review. Review of General Psychology 4(4), 378–408 (2000)

    Article  Google Scholar 

  75. Cooper, C.L., Pervin, L.A.: Personality: Critical concepts in psychology, 1st edn. Critical concepts in psychology. Routledge, New York (1998)

    Google Scholar 

  76. Lukowicz, P.: Wearable computing and artificial intelligence for healthcare applications. Artificial Intelligence in Medicine 42(2), 95–98 (2008)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Telematics and Information Technology (CTIT), University of Twente, P.O. Box 217, 7500, AE Enschede, The Netherlands

    Egon L. van den Broek

  2. Agent Technology Center, Dept. of Cybernetics, FEE, Czech Technical University, Technická 2, 16627, Praha 6, Czech Republic

    Viliam Lisý

  3. User Experience Group, Philips Research, High Tech Campus 34, 5656, AE Eindhoven, The Netherlands

    Joris H. Janssen & Joyce H. D. M. Westerink

  4. Dept. of Human Technology Interaction, Eindhoven University of Technology, P.O. Box 513, 5600 MB, Eindhoven, The Netherlands

    Joris H. Janssen

  5. Philips Consumer Lifestyle Advanced Technology, High Tech Campus 37, 5656, AE Eindhoven, The Netherlands

    Marleen H. Schut & Kees Tuinenbreijer

Authors
  1. Egon L. van den Broek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Viliam Lisý
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joris H. Janssen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joyce H. D. M. Westerink
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marleen H. Schut
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kees Tuinenbreijer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Instituto de Telecomunicaçöes, IST - Instituto Superior Técnico, Av. Rovisco Pais, 1, 1049-001, Portugal, Lisbon

    Ana Fred

  2. Departament of Systems and Informatics, Polytechnic Institute of Setúbal – INSTICC, Rua do Vale de Chaves - Estefanilha, 2910-761, Setúbal, Portugal

    Joaquim Filipe

  3. Institute of Telecommunications, Av. Rovisco Pais, 1, 1049-001, Lisboa, Portugal

    Hugo Gamboa

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

van den Broek, E.L., Lisý, V., Janssen, J.H., Westerink, J.H.D.M., Schut, M.H., Tuinenbreijer, K. (2010). Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds) Biomedical Engineering Systems and Technologies. BIOSTEC 2009. Communications in Computer and Information Science, vol 52. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11721-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-11721-3_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-11720-6

  • Online ISBN: 978-3-642-11721-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation