Abstract
Task-oriented dialogue-based spatial reasoning systems need to maintain history of the world/discourse states in order to convey that the dialogue agent is mentally present and engaged with the task, as well as to be able to refer to earlier states, which may be crucial in collaborative planning (e.g., for diagnosing a past misstep). We approach the problem of spatial memory in a multi-modal spoken dialogue system capable of answering questions about interaction history in a physical blocks world setting. We employ a pipeline consisting of a vision system, speech I/O mediated by an animated avatar, a dialogue system that robustly interprets queries, and a constraint solver that derives answers based on 3D spatial modelling. The contributions of this work include a semantic parser competent in this domain and a symbolic dialogue context allowing for interpreting and answering free-form historical questions using world and discourse history.
This work was supported by DARPA grant W911NF-15-1-0542, NSF NRT Graduate Training Grant 2019–2020, and NSF EAGER Award IIS-1940981.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Intended actions obviated by earlier events may be deleted.
References
Allen, J.F., Ferguson, G.: Actions and events in interval temporal logic. J. Logic Comput. 4(5), 531–579 (1994)
Bisk, Y., Shih, K.J., Choi, Y., Marcu, D.: Learning interpretable spatial operations in a rich 3D blocks world. In: 32nd AAAI Conference on Artificial Intelligence (2018)
Fahlman, S.E.: A planning system for robot construction tasks. Arti. Intell. 5(1), 1–49 (1974)
Ferguson, G., Allen, J.: Trips: an integrated intelligent problem-solving assistant. In: AAAI (1998)
Ferguson, G., Allen, J., Miller, B.: Trains-95: towards a mixed-initiative planning assistant. In: Proceedings of 3rd International Conference on Artificial Intelligence Planning Systems, pp. 70–77 (1996)
Franklin, N.T., Norman, K.A., Ranganath, C., Zacks, J.M., Gershman, S.J.: Structured event memory: a neuro-symbolic model of event cognition. bioRxiv (2019)
Hwang, C.H., Schubert, L.K.: Interpreting tense, aspect and time adverbials: a compositional, unified approach. In: Gabbay, D.M., Ohlbach, H.J. (eds.) ICTL 1994. LNCS, vol. 827, pp. 238–264. Springer, Heidelberg (1994). https://doi.org/10.1007/BFb0013992
Kim, G., et al.: Generating discourse inferences from unscoped episodic logical formulas. In: Proceedings of 1st International Workshop on Designing Meaning Representations, pp. 56–65. ACL (2019)
Kim, G.L., Schubert, L.: A type-coherent, expressive representation as an initial step to language understanding. In: Proceedings of 13th International Conference on Computational Semantics-Long Papers, pp. 13–30 (2019)
Kottur, S., Moura, J.M., Parikh, D., Batra, D., Rohrbach, M.: Clevr-dialog: a diagnostic dataset for multi-round reasoning in visual dialog (2019). arXiv:1903.03166
Madl, T., Franklin, S., Chen, K., Trappl, R.: Spatial working memory in the LIDA cognitive architecture. In: Proceedings of International Conference on Cognitive Modelling, pp. 384–390 (2013)
Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J.: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision (2019). arXiv:1904.12584
Perera, I., Allen, J., Teng, C.M., Galescu, L.: Building and learning structures in a situated blocks world through deep language understanding. In: Proceedings of 1st International Workshop on Spatial Language Understanding. pp. 12–20 (2018)
Razavi, S., Schubert, L., Ali, M., Hoque, H.: Managing casual spoken dialogue using flexible schemas, pattern transduction trees, and gist clauses. In: 5th Annual Conference on Advances in Cognitive Systems (2017)
Rensink, R.: Scene Perception, pp. 151–155. Oxford University Press, Oxford (2001)
Rothfuss, J., Ferreira, F., Aksoy, E., You, Z., Asfour, T.: Deep episodic memory: encoding, recalling, and predicting episodic experiences for robot action execution. IEEE Rob. Autom. Lett. 3, 4007–4014 (2018)
Schubert, L., Hwang, C.: Episodic logic meets little red riding hood: a comprehensive, natural representation for language understanding. In: Natural Language Processing and Knowledge Representation: Language for Knowledge and Knowledge for Language (2000)
Winograd, T.: Understanding natural language. Cogn. Psychol. 3(1), 1–191 (1972)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kane, B., Platonov, G., Schubert, L. (2020). Registering Historical Context for Question Answering in a Blocks World Dialogue System. In: Sojka, P., Kopeček, I., Pala, K., Horák, A. (eds) Text, Speech, and Dialogue. TSD 2020. Lecture Notes in Computer Science(), vol 12284. Springer, Cham. https://doi.org/10.1007/978-3-030-58323-1_52
Download citation
DOI: https://doi.org/10.1007/978-3-030-58323-1_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58322-4
Online ISBN: 978-3-030-58323-1
eBook Packages: Computer ScienceComputer Science (R0)