Abstract
With the increasing pervasiveness of smart phones and smart devices, dialogue systems are gaining ever growing attention from both academic and industry. These systems can be broadly classified into two categories, one that is aimed at helping user to gain new knowledge and one that can chat with users without completing any specific tasks. Although dialogue systems are improving substantially, the user experience of such systems are still unsatisfactory as there are no specific rules covering all possible situations of real human–machine dialogue, resulting in breakdowns. There are two technical issues affecting the detection of dialogue breakdown in an open domain conversation: human resources to prepare and annotate a large chunk of conversation data and dialogue histories containing words that don’t appear directly in training data. To tackle these issues, we propose a novel encoding method for temporal utterances with memory attention based on end-to-end dialogue breakdown detection. Specifically, long short-term memory (LSTM) is employed to encode each word of all previous user and system utterances. Encoded vectors from LSTM (user and system utterances), along with system and user utterances from sentence embedding, are then stored in memory wherein an attention mechanism is applied to select the most relevant piece of words from system and user utterances for breakdown detection. An evaluation of the proposed approach on a breakdown detection task (DBDC3) showed that the model for single-labeled breakdown detection outperforms other state-of-the-art methods in a classification task. In conclusion, a more effective knowledge gain and management can be achieved by integration of our proposed breakdown detection into dialogue systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We use the evaluation code provided at https://github.com/dbd-challenge/dbdc3/tree/master/prog.
References
Nevo D, Furneaux B, Wand Y (2008) Towards an evaluation framework for knowledge management systems. Inf Technol Manag 9(4):233–249
Piccoli G, Ahmad R, Ives B (2000) Knowledge management in academia: a proposed framework. Inf Technol Manag 1(4):229–245
Martinovsky B, Traum D (2006) The error is the clue: breakdown in human–machine interaction. Technical report, University of Southern California Marina Del Reyca Inst for Creative Technologies
Bickmore T, Cassell J (2001) Relational agents: a model and implementation of building user trust. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 396–403
Williams J, Young S (2007) Partially observable markov decision processes for spoken dialog systems. Comput Speech Lang 21(2):393–422
Zhou X, Dong D, Wu H, Zhao S, Yu D, Tian H, Liu X, Yan R (2016) Multi-view response selection for human–computer conversation. In: Proceedings of the 2016 conference on empirical methods in natural language processing, pp 372–381
Saito A, Iki T (2017) End-to-end character-level dialogue breakdown detection with external memory models. In: Proceedings of the dialog system technology challenges workshop (DSTC6)
Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. Preprint arXiv:1409.0473
Serban IV, Sordoni A, Bengio Y, Courville AC, Pineau J (2016) Building end-to-end dialogue systems using generative hierarchical neural network models. AAAI 16:3776–3784
Hochreite S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Walker MA, Geary IL, Hastie HW, Wright J, Gorin A (2002) Automatically training a problematic dialogue predictor for a spoken dialogue system. J Artif Intell Res 16:293–319
Higashinaka R, Funakoshi K, Kobayashi Y, Inaba M (2016) The dialogue breakdown detection challenge: task description, datasets, and evaluation metrics. In: LREC
Higashinaka R, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y, Mizukami M (2015) Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of the 16th annual meeting of the special interest group on discourse and dialogue, pp 87–95
Bulyko I, Kirchhoff K, Ostendorf M, Goldberg J (2005) Error-correction detection and response generation in a spoken dialogue system. Speech Commun 45(3):271–288
Black AW, Eskenazi M (2009) The spoken dialogue challenge. In: Proceedings of the SIGDIAL 2009 conference: the 10th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, pp 337–340
Kobayashi S, Unno Y, Fukuda M (2015) Multitask learning of recurrent neural network for detecting breakdowns of dialog and language modeling. JSAI technical report (SIG-SLUD-75-B502), pp 41–46
Taniguchi R, Kano Y (2015) Construction of automatic detector for dialogue breakdowns based on rules with keywords extraction. JSAI technical report (SIG-SLUD-75-B502), pp 37–40
Sugiyama H (2017) Dialogue breakdown detection based on estimating appropriateness of topic transition. In: Dialog system technology challenges (DSTC6)
Piramuthu S (2005) Feature selection for reduction of tabular knowledge-based systems. Inf Technol Manag 6(4):351–362
Park C, Kim K, Kim S (2017) Attention-based dialog embedding for dialog breakdown detection. In: Proceedings of the dialog system technology challenges workshop (DSTC6)
Al-Rfou R, Perozzi B, Skiena S (2013) Polyglot: distributed word representations for multilingual NLP. Preprint arXiv:1307.1662
Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: EMNLP. The Association for Computational Linguistics, pp 1412–1421
Higashinaka R, Funakoshi K, Inaba M, Tsunomori Y, Takahashi T, Kaji N (2017) Overview of dialogue breakdown detection challenge 3. In: Proceedings of dialog system technology challenge, p 6
Zeiler MD (2012) ADADELTA: an adaptive learning rate method. Preprint arXiv:1212.5701
Lopes J (2017) “How generic can dialogue breakdown detection be?” the kth entry to DBDC3. In: Proceedings of dialog system technology challenge, vol 6
Kato S, Sakai T (2017) RSL17BD at DBDC3: computing utterance similarities based on term frequency and word embedding vectors. In: Proceedings of DSTC6. http://workshop.colips. org/dstc6/papers/track3_paper13_kato.pdf
Lee S, Hooshyar D, Ji H, Nam K, Lim H (2018) Mining biometric data to predict programmer expertise and task difficulty. Clust Comput 21(1):1097–1107
Hooshyar D, Yousefi M, Lim H (2018) Data-driven approaches to game player modeling: a systematic literature review. ACM Comput Surv (CSUR) 50(6):90
Acknowledgements
This work was supported by a Ministry of Culture, Sports and Tourism (No. R2017030045).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lee, S., Lee, D., Hooshyar, D. et al. Integrating breakdown detection into dialogue systems to improve knowledge management: encoding temporal utterances with memory attention. Inf Technol Manag 21, 51–59 (2020). https://doi.org/10.1007/s10799-019-00308-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10799-019-00308-x