Abstract
Advancement of deep learning has improved performances on a wide variety of tasks. However, language reasoning and understanding remain difficult tasks in Natural Language Processing (NLP). In this work, we consider this problem and propose a novel Knowledge-Infused Document Embedding Representation (KIDER) for text categorization. We use knowledge patterns to generate high quality document representation. These patterns preserve categorical-distinctive semantic information, provide interpretability, and achieve superior performances at the same time. Experiments show that the KIDER model outperforms state-of-the-art methods on two important NLP tasks, i.e., emotion analysis and news topic detection, by 7% and 20%. In addition, we also demonstrate the potential of highlighting important information for each category and news using these patterns. These results show the value of knowledge-infused patterns in terms of interpretability and performance enhancement.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We empirically set the window size to 3, dimension of vector to 50, and epochs to 100.
References
Aggarwal, C.C., Zhai, C.: Mining Text Data. Springer Science & Business Media, Heidelberg (2012)
Wu, H., Salton, G.: A comparison of search term weighting: term relevance vs. inverse document frequency. In: ACM SIGIR Forum, vol. 16, pp. 30–39. ACM (1981)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Lee, J.Y., Dernoncourt, F.: Sequential short-text classification with recurrent and convolutional neural networks. arXiv preprint arXiv:1603.03827 (2016)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Alexander, J.A.: Template-based procedures for neural network interpretation. Ph.D. thesis, University of Colorado (1994)
Seidenberg, M.: Language at the Speed of Sight How We Read, Why So Many Can’t, and What Can Be Done About It, 1st edn. Basic Books, New York (2017)
Landauer, T.K., Foltz, P.W., Laham, D.: An introduction to latent semantic analysis. Discourse Process. 25(2–3), 259–284 (1998)
Finkel, J.R., Grenager, T., Manning, C.: Incorporating non-local information into information extraction systems by Gibbs sampling. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 363–370. Association for Computational Linguistic (2005)
Chen, K.J., Huang, S.L., Shih, Y.Y., Chen, Y.J.: Extended-Hownet: a representational framework for concepts. In: Proceedings of OntoLex 2005-Ontologies and Lexical Resources (2005)
Dunning, T.: Accurate methods for the statistics of surprise and coincidence. Comput. Linguist. 19(1), 61–74 (1993)
Chang, Y.C., Chu, C.H., Su, Y.C., Chen, C.C., Hsu, W.L.: PIPE: a BIOC module for protein-protein interaction passage extraction. Database (Oxford) 2016, baw101 (2016)
Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, pp. 1188–1196 (2014)
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML, vol. 30, p. 3 (2013)
Chang, Y.C., Chen, C.C., Hsieh, Y.L., Chen, C.C., Hsu, W.L.: Linguistic template extraction for recognizing reader-emotion and emotional resonance writing assistance. In: ACL, pp. 775–780 (2015)
Chang, Y.-C., Hsieh, Y.-L., Chen, C.-C., Hsu, W.-L.: A semantic frame-based intelligent agent for topic detection. Soft. Comput. 21(2), 391–401 (2015). https://doi.org/10.1007/s00500-015-1695-4
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Yogatama, D., Dyer, C., Ling, W., Blunsom, P.: Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898 (2017)
McCallum, A., Nigam, K.: A comparison of event models for Naïve Bayes text classification. In: AAAI/ICML-1998 Workshop on Learning for Text Categorization, pp. 41–48 (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, YT., Lin, ZW., Chang, YC., Hsu, WL. (2020). KIDER: Knowledge-Infused Document Embedding Representation for Text Categorization. In: Fujita, H., Fournier-Viger, P., Ali, M., Sasaki, J. (eds) Trends in Artificial Intelligence Theory and Applications. Artificial Intelligence Practices. IEA/AIE 2020. Lecture Notes in Computer Science(), vol 12144. Springer, Cham. https://doi.org/10.1007/978-3-030-55789-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-55789-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55788-1
Online ISBN: 978-3-030-55789-8
eBook Packages: Computer ScienceComputer Science (R0)