Abstract
This is a pilot study of multimodal behaviours in manually annotated comparable video recordings of Danish and Polish triadic naturally occurring conversations. The data are comparable with respect to the conversational settings, the familiarity degree, age and gender of the participants. Furthermore, they have been annotated according to the same annotation scheme following common coding strategies. The analysis of the annotations indicates that although the conversations in the two languages differ in content, Danes and Poles use the same type of head movements and with the same frequency. In both datasets the most common facial expressions are laughter and smile, however, facial expressions are much more frequent in the Polish data than in the Danish data. Furthermore, the facial expressions in the Polish data are often used as feedback signals to the interlocutors while the Danes use facial expression to comment their own spoken contribution. Finally, the Danes use more frequently hand gestures than the Poles and their hand gestures have a deictic function while the hand gestures of the Poles are iconic. The differences in the behaviours in the two corpora can partly depend on the language, but is also due to the type of relationship between the participants and the content of the conversations.
Chapter PDF
Similar content being viewed by others
References
McClave, E.: Linguistic functions of head movements in the context of speech. Journal of Pragmatics 32, 855–878 (2000)
Cerrato, L.: Investigating Communicative Feedback Phenomena across Languages and Modalities. PhD thesis, Stockholm, KTH, Speech and Music Communication (2007)
Paggio, P., Navarretta, C.: Head movements, facial expressions and feedback in Danish first encounters interactions: A culture-specific analysis. In: Stephanidis, C. (ed.) Universal Access in HCI, Part II, HCII 2011. LNCS, vol. 6766, pp. 583–590. Springer, Heidelberg (2011)
Maynard, S.: Interactional functions of a nonverbal sign: Head movement in Japanese dyadic casual conversation. Journal of Pragmatics 11, 589–606 (1987)
Jokinen, K., Navarretta, C., Paggio, P.: Distinguishing the communicative functions of gestures. In: Popescu-Belis, A., Stiefelhagen, R. (eds.) MLMI 2008. LNCS, vol. 5237, pp. 38–49. Springer, Heidelberg (2008)
Rehm, M., Nakano, Y., André, E., Nishida, T.: Culture-Specific First Meeting Encounters between Virtual Agents. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 223–236. Springer, Heidelberg (2008)
Lu, J., Allwood, J., Ahlsén, E.: A study on cultural variations of smile based on empirical recordings of Chinese and Swedish first encounters. In: Heylen, D., Kipp, M., Paggio, P. (eds.) Proceedings of the Workshop on Multimodal Corpora at ICMI-MLMI 2011, Alicante, Spain (2011)
Navarretta, C., Ahlsn, E., Allwood, J., Jokinen, K., Paggio, P.: Feedback in nordic first-encounters: a comparative study. In: Proceedings of LREC 2012, Istanbul, Turkey, pp. 2494–2499 (May 2012)
Jacobs, N., Garnham, A.: The role of conversational hand gestures in a narrative task. Journal of Memory and Language 56(2), 291–303 (2007)
Kita, S., Özyurek, A.: How does spoken language shape iconic gestures? In: Duncan, S., Cassel, J., Levy, E. (eds.) Gesture and the Dynamic Dimension of Language, pp. 67–74. Benjamins, Amsterdam (2007)
Gullberg, M.: Thinking, speaking and gesturing about motion in more than one language. In: Pavlenko, A. (ed.) Thinking and Speaking in two Languages, pp. 143–169 (2011)
Lis, M., Parrill, F.: Referent type and its verbal and gestural representation: A test on English multimodal corpus and WordNet 3.1. In: Extended Abstract in Proceedings of the 1st European Symposium on Multimodal Communication, MMSym 2013 (2013)
McNeill, D.: Hand and mind: What gestures reveal about thought. University of Chicago Press, Chicago (2000)
Goldin-Meadow, S., Chee So, W., Ozyurek, A., Mylander, C.: The natural order of events: How speakers of different languages represent events nonverbally. Proceedings of the National Academy of Sciences of the USA 105(27), 9163–9168 (2008)
Navarretta, C., Paggio, P.: Verbal and non-verbal feedback in different types of interactions. In: Proceedings of LREC 2012, Istanbul Turkey, pp. 2338–2342 (2012)
Campbell, N.: Individual Traits of Speaking Style and Speech Rhythm in a Spoken Discourse. In: Proceedings of the COST 2007 Workshop, pp. 107–120 (2007)
Navarretta, C., Lis, M.: Multimodal feedback expressions in Danish and Polish spontaneous conversations. In: Allwood, J., et al. (eds.) NEALT Proceedings. Northern European Association for Language and Technology, Proceedings of the Fourth Nordic Symposium of Multimodal Communication, Gothenburg, Sweden, November 2012. Linköping Electronic Conference Proceedings, pp. 55–62 (2013)
VYngve, V.: On getting a word in edgewise. Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, pp. 567–578 (1970)
Rehm, M., et al.: Creating Standardized Video Recordings of Multimodal Interactions across Cultures. In: Kipp, M., Martin, J.-C., Paggio, P., Heylen, D., et al. (eds.) Multimodal Corpora. LNCS (LNAI), vol. 5509, pp. 138–159. Springer, Heidelberg (2009)
MacWhinney, B., Wagner, J.: Transcribing, searching and data sharing: The CLAN software and the TalkBank data repository. Gespraechsforschung 11, 154–173 (2010)
Navarretta, C.: Annotating Behaviours in Informal Interactions. In: Esposito, A. (ed.) Communication and Enactment 2010. LNCS, vol. 6800, pp. 309–315. Springer, Heidelberg (2011)
Navarretta, C.: Anaphora and gestures in multimodal communication. In: Hendrickx, B., Devi, L., Mitkov (eds.) Proceedings of the 8th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2011), Edicoes Colibri, Faro, Portugal, pp. 171–181 (2011)
Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The mumin coding scheme for the annotation of feedback, turn management and sequencing. Multimodal Corpora for Modelling Human Multimodal Behaviour. Special Issue of the International Journal of Language Resources and Evaluation 41(3-4), 273–287 (2007)
Boersma, P., Weenink, D.: Praat: Doing phonetics by computer (2013), retrieved from http://www.praat.org/
Kipp, M.: Gesture generation by imitation - from human behavior to computer character animation. Ph.D. thesis, Saarland University, Saarbruecken, Germany, Boca Raton, Florida, dissertation.com (2004)
Peirce, C.S.: Collected Papers of Charles Sanders Peirce, 8 vols. Harvard University Press, Cambridge (1931, 1958)
Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46 (1960)
Campbell, N.: An audio-visual approach to measuring discourse synchrony in multimodal conversation data. In: Proceedings of Interspeech 2009, pp. 12–14 (2009)
Jokinen, K., Pärkson, S.: Synchrony and copying in conversational interactions. In: Paggio, P., Ahlsén, E., Allwood, J., Jokinen, K., Navarretta, C. (eds.) Proceedings of the 3rd Nordic Symposium on Multimodal Communication. NEALT Proceedings Series, vol. 15, pp. 18–24 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Navarretta, C., Lis, M. (2014). Multimodal Behaviours in Comparable Danish and Polish Human-Human Triadic Spontaneous Interactions. In: Kurosu, M. (eds) Human-Computer Interaction. Advanced Interaction Modalities and Techniques. HCI 2014. Lecture Notes in Computer Science, vol 8511. Springer, Cham. https://doi.org/10.1007/978-3-319-07230-2_44
Download citation
DOI: https://doi.org/10.1007/978-3-319-07230-2_44
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07229-6
Online ISBN: 978-3-319-07230-2
eBook Packages: Computer ScienceComputer Science (R0)