Skip to main content
SpringerLink
Log in

How Convincing is Mr. Data's Smile: Affective Expressions of Machines

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Emotions should play an important role in the design of interfaces because people interact with machines as if they were social actors. This paper presents a literature review on affective expressions through speech, music and body language. It summarizes the quality and quantity of their parameters and successful examples of synthesis. Moreover, a model for the convincingness of affective expressions, based on Fogg and Hsiang Tseng (1999), was developed and tested. The empirical data did not support the original model and therefore this paper proposes a new model, which is based on appropriateness and intensity of the expressions. Furthermore, the experiment investigated if the type of emotion (happiness, sadness, anger, surprise, fear and disgust), knowledge about the source (human or machine), the level of abstraction (natural face, computer rendered face and matrix face) and medium of presentation (visual, audio/visual, audio) of an affective expression influences its convincingness and distinctness. Only the type of emotion and multimedia presentations had an effect on convincingness. The distinctness of an expression depends on the abstraction and the media through which it is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bartneck, C.: 2000, Affective Expressions of Machines, Technical University of Eindhoven, The Netherlands. Also available at http://www.bartneck.de/work/aem.pdf

    Google Scholar 

  • Barnes, A. and Thagard, P.: 1996, Emotional decisions. Proceedings of the Eighteen's Annual Conference of the Cognitive Science Society, University of California, pp. 426-429.

  • Brennan, S. E.: 1985, The caricature generator. Leonardo 18, 170-178.

    Google Scholar 

  • Bresin, R. and Friberg, A.: 1999, Synthesis and decoding of emotionally expressive music performance. Proceedings of the IEEE 1999 Systems, Man and Cybernetics Conference (SMC'99), Tokyo, pp. 317-322.

  • Cahn, J. E.: 1990, The generation of affect in synthesized speech. Journal of American Voice I/O Society, 8, 1-19.

    Google Scholar 

  • Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, W., Douville, B., Prevost, S. and Stone, M.: 1998, Animated conversation: rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In: M. Maybury and W. Wahlster (eds.), Readings in Intelligent User Interfaces, Morgan Kaufmann Publishers.

  • Crowder, R. G.: 1984, Perception of the major/minor distinction: I. Historical and theoretical foundations. Psychomusicology, 4, 3-10.

    Google Scholar 

  • CSLU, 1999, CSLU Toolkit. http://cslu.cse.ogi.edu/toolkit/

  • Cunningham, J. G. and Sterling, R. S.: 1988, Development change in the understanding of affective meaning in music. Motivation and Emotion, 12, 399-413.

    Google Scholar 

  • Davies, J. B. (ed.): 1978, The Psychology of Music. Stanford, Stanford University Press.

    Google Scholar 

  • Ekman, P. (ed.): 1973, Darwin and Facial Expression: A Century of Research in Review. New York: Academic Press.

    Google Scholar 

  • Ekman, P., Friesen, W. V. and Ellsworth, P. (eds.): 1972, Emotion in the Human Face: Guidelines for Research and an Integration of Findings. New York: Pergamon Press.

    Google Scholar 

  • Ekman, P., Friesen, W.V., O'Sullivan, M. and Scherer, K.: 1980, Relative importance of face, body, and speech in judgments of personality and affect. Journal of Personality and Social Psychology, 38, 270-277.

    Google Scholar 

  • Ekman, P. and Frieser, W. (eds.): 1976, Pictures of Facial Affects. Palo Alto: Consulting Psychologist Press.

    Google Scholar 

  • Etcoff, N. L. and Magee, J. J.: 1992, Categorical perception of facial expressions. Cognition, 44, 227-240.

    Google Scholar 

  • Feist, G.: 1994, The affective cpmsequences of artistic and scientific problem solving. Cognition and Emotion, 8, 489-502.

    Google Scholar 

  • Fogg, B.J. and Hsiang Tseng: 1999, The elements of computer credibility. Proceedings of CHI 99, Pittsburgh, pp. 80-87.

  • Frijda, N. H. (ed.): 1986, The Emotions. Cambridge, Cambridge University Press.

    Google Scholar 

  • Frijda, N. H.: 1969, Recognition of emotion. Advances in Experimental Social Psychology, 4, 167-223.

    Google Scholar 

  • Gerardi, G. M. and Gerken, L.: 1995, The development of affective responses to modality and melodic contour. Music Perception, 12, 279-290.

    Google Scholar 

  • Izard, C. E. (ed.): 1977, Human Emotions. New York: Plenum Press.

    Google Scholar 

  • Juslin, P. N.: 1997a, Emotional communication in music performance: a functionalist perspective and some data. Music Perception, 14, 383-418.

    Google Scholar 

  • Juslin, P. N.: 1997b, Perceived emotional expression in synthesized performances of a short melody: capturing the listener's judgment policy. Musicae Scientiae, 2, 225-256.

    Google Scholar 

  • Kastner, M. P. and Crowder, R. G.: 1990, Perception of major/minor: IV. Emotional connotation in young children. Music Perception, 8, 189-202.

    Google Scholar 

  • Krauss, R. M., Morrel-Samuels, P. and Colasante, C.: 1991, Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 743-754.

  • Lazarus, R. S. (ed.): 1991, Emotion and Adaptation. Oxford, Oxford University Press.

    Google Scholar 

  • Massaro, D. W. (ed.): 1998, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle. MIT Press.

  • McNeill, D. (ed.): 1992, Hand and Mind: What Gestures Reveal about Thought. Chicago, University of Chicago.

    Google Scholar 

  • Miller, G.A.: 1956, The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 63, 81-97.

    Google Scholar 

  • Mozziconacci, S. (ed.): 1998, Speech Variability and Emotions: Production and Perception. Ph.D. thesis, IPO, Technical University of Eindhoven.

  • Murray, I. R. and Arnott, J. L.: 1992, Towards the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal of the Acoustic Society of America, 93, 1097-1108.

    Google Scholar 

  • Nass, C. and Reeves, B. (eds.): 1996, The Media Equation. Cambridge, SLI Publications, Cambridge University Press.

    Google Scholar 

  • Nielsen, J. (ed.): 1994, Usability Engineering. San Diego: AP Professional.

    Google Scholar 

  • Nobe, S., Hayamizu, S., Hasegawa, O. and Takahashi, H.: 1997, Are listeners paying attention to the hand gestures of an anthropomorphic agent? An evaluation using a gaze tracking method'. In: I. Wachsmuth and M. Fröhlich (eds.), Gesture and Sign Language in Human-Computer Interaction, Springer.

  • Norman, D. A.: 1981, Twelve issues for cognitive science. In: D. Norman (ed.), Perspectives on Cognitive Science, Norwood: Ablex Publishing, pp. 265-295.

    Google Scholar 

  • Nowlis, V.: 1966, Research with the mood adjective check list. In: S. S. Tomkin and C. E. Izard (eds.), Affect, Cognition and Personality, New York, Springer, pp. 352-389.

    Google Scholar 

  • Oatley, K. and Jenkins, J. M. (eds.): 1996, Understanding Emotions. Oxford: Blackwell.

    Google Scholar 

  • Osgood, C. E., Suci, G. J. and Tannenbaum, P. H. (eds.): 1957, The Measurements of Meaning. Champaign: University of Illinois Press.

    Google Scholar 

  • Picard, R. (ed.): 1997a, Affective Computing. Cambridge: MIT Press.

    Google Scholar 

  • Picard, R.: 1997b, Does HAL cry digital tears? Emotion and computers. In: D. Stork (ed.): Hal's Legacy: 2001's Computer as Dream and Reality, Cambridge: MIT Press.

    Google Scholar 

  • Pixar, 1998, Geri's game. http://www.pixar.com/shorts/shorts.html

  • Plutchik, R. (ed.): 1980, Emotion: A Psycho Evolutionary Synthesis. New York, Harper and Row.

    Google Scholar 

  • Russel, J. A.: 1979, Affective space is bipolar. Journal of Personality and Social Psychology, 37, 345-356.

    Google Scholar 

  • Scherer, K. R.: 1979, Nonlinguistic vocal indicators of emotion and psychopathology. In: C. E. Izard (eds.), Emotions in Personality and Psychopathology, New York, Plenum.

    Google Scholar 

  • Scherer, K. R. and Oshinsky, J. S.: 1977, Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion, 4, 331-346.

    Google Scholar 

  • Schiano, D. J., Ehrlich, S. M., Rahardja, K. and Sheridan, K.: 2000, Face to InterFace: Facial affect in (Hu)Man and Machine. CHI 2000 Conference Proceedings, The Hague, pp. 193-200.

  • Schlossberg, H.: 1954, Three dimensions of emotion. Psychological Review, 61, 81-88.

    Google Scholar 

  • Smith, C. A. and Ellsworth, P. C.: 1985, Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48, 813-838.

    Google Scholar 

  • Sony, 1999, Aibo. http://www.world.sony.com/aibo/frame13.html

Download references

Author information

Authors and Affiliations

  1. Eindhoven University of Technology, Den Dolech 2, P.O. Box 513, 5600 MB, Eindhoven, The Netherlands.

    Christoph Bartneck

Authors
  1. Christoph Bartneck
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bartneck, C. How Convincing is Mr. Data's Smile: Affective Expressions of Machines. User Modeling and User-Adapted Interaction 11, 279–295 (2001). https://doi.org/10.1023/A:1011811315582

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1011811315582

Search

Navigation