Abstract
Sandhi splitting is a primary and an important step for any natural language processing (NLP) application for languages which have agglutinative morphology. This paper presents a statistical approach to build a sandhi splitter for agglutinative languages. The input to the model is a valid string in the language and the output is a split of that string into meaningful word/s. The approach adopted comprises of two stages namely Segmentation and Word generation, both of which use conditional random fields (CRFs). Our approach is robust and language independent. The results for two Dravidian languages viz. Telugu and Malayalam show an accuracy of 89.07% and 90.50% respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kolachina, S., Sharma, D.M., Gadde, P., Vijay, M., Sangal, R., Bharati, A.: External sandhi and its relevance to syntactic treebanking. Polibits (43), 67–74 (2011)
Bharati, A., et al.: Natural language processing: a Paninian perspective, ch. 3. Prentice-Hall of India, New Delhi (1995)
Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: Probabilistic models for segmenting and labelling sequence data (2001)
Nguyen, C.-T., Nguyen, T.-K., Phan, X.-H., Nguyen L.-M., and Ha, Q.-T.: Vietnamese word segmentation with crfs and svms: An investigation. In: Proceedings of the 20th Pacific Asia Conference on Language, Information and Computation, PACLIC 2006 (2006)
Peng, F., Feng, F., McCallum, A.: Chinese segmentation and new word detection using conditional random fields. In: Proceedings of the 20th International Conference on Computational Linguistics, p. 562. Association for Computational Linguistics (2004)
Xue, N., et al.: Chinese word segmentation as character tagging. Computational Linguistics and Chinese Language Processing 8(1), 29–48 (2003)
Nair, L.R., Peter, S.D.: Development of a rule based learning system for splitting compound words in malayalam language. In: 2011 IEEE Recent Advances in Intelligent Computational Systems (RAICS), pp. 751–755. IEEE (2011)
Joshi Shripad, S.: Sandhi splitting of Marathi compound words. Int. J. on Adv. Computer Theory and Engg. 2(2) (2012)
Vempaty, P.C., Nagalla, S.C.P.: Automatic sandhi spliting method for telugu, an indian language. Procedia-Social and Behavioral Sciences 27, 218–225 (2011)
Natarajan, A., Charniak, E.: S3-statistical sam. dhi splitting (2011)
Devadath, V.V., Kurisinkel, L.J., Sharma, D.M., Varma, V.: A sandhi splitter for malayalam (accepted but yet to be published in proceedings of ICON 2014)
Krishnamurti, B.: A grammar of modern Telugu. Oxford University Press, New York (1985)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Kuncham, P., Nelakuditi, K., Nallani, S., Mamidi, R. (2015). Statistical Sandhi Splitter for Agglutinative Languages. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2015. Lecture Notes in Computer Science(), vol 9041. Springer, Cham. https://doi.org/10.1007/978-3-319-18111-0_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-18111-0_13
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-18110-3
Online ISBN: 978-3-319-18111-0
eBook Packages: Computer ScienceComputer Science (R0)