Abstract
Simple Recurrent Networks (Srns) have been widely used in natural language processing tasks. However, their ability to handle long-term dependencies between sentence constituents is rather limited. Narx networks have recently been shown to outperform Srns by preserving past information in explicit delays from the network’s prior output. Determining the number of delays, however, is problematic in itself. In this study on a shift-reduce parsing task, we demonstrate a hybrid localist-distributed approach that yields comparable performance in a more concise manner. A SardNet self-organizing map is used to represent the details of the input sequence in addition to the recurrent distributed representations of the Srn and Narx networks. The resulting architectures can represent arbitrarily long sequences and are cognitively more plausible.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Allen, R.B.: Several studies on natural language and back-propagation. In: Proceedings of the IEEE First International Conference on Neural Networks, San Diego, CA, vol. II, pp. 335–341. IEEE, Piscataway (1987)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (1994)
Berg, G.: A connectionist parser with recursive sentence structure and lexical disam biguation. In: Swartout, W. (ed.) Proceedings of the Tenth National Conference on Artificial Intelligence, pp. 32–37. MIT Press, Cambridge (1992)
Chalmers, D.J.: Syntactic transformations on distributed representations. Connection Science 2, 53–62 (1990)
Chen, S., Billings, S., Grant, P.: Non-linear system identification using neural networks. International Journal of Control, 1191–1214 (1990)
Connor, J., Atlas, L., Martin, D.: Recurrent networks and narma modeling. Advances in Neural Information Processing Systems 4, 301–308 (1992)
Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)
Elman, J.L.: Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7, 195–225 (1991)
Hermjakob, U.: Learning Parse and Translation Decisions from Examples with Rich Context. PhD thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX, Technical Report UT-AI97-261 (1997)
Horne, B., Giles, C.: An experimental comparison of recurrent neural networks. Advances in Neural Information Processing Systems 7, 697–704 (1995)
James, D.L., Miikkulainen, R.: SARDNET: A self-organizing feature map for sequences. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems 7, pp. 577–584. MIT Press, Cambridge (1995)
Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78, 1464–1480 (1990)
Kohonen, T.: Self-Organizing Maps. Springer, Berlin (1995)
Lin, T., Horne, B.G., Giles, C.L.: How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies. Neural Networks 11(5), 861–868 (1998)
Lin, T., Horne, B.G., Giles, C.L.: Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks 7(6), 1329–1338 (1996)
Lin, T., Giles, C.L., Horne, B.G., Kung, S.Y.: A Delay Damage Model Selection Algorithm for NARX Neural Networks. IEEE Transactions on Signal Processing 45(11), 2719–2730 (1997)
McClelland, J.L., Kawamoto, A.H.: Mechanisms of sentence processing: As-signing roles to constituents. In: McClelland, J.L., Rumelhart, D.E. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Psychological and Biological Models, vol. 2, pp. 272–325. MIT Press, Cambridge (1986)
Miikkulainen, R.: Subsymbolic Natural Language Processing: An Integrated Model of Scripts, Lexicon, and Memory. MIT Press, Cambridge (1993)
Miikkulainen, R.: Subsymbolic case-role analysis of sentences with embedded clauses. Cognitive Science 20, 47–73 (1996)
Miikkulainen, R.: Dyslexic and category-specific impairments in a self-organizing feature map model of the lexicon. Brain and Language 59, 334–366 (1997)
Munro, P., Cosic, C., Tabasko, M.: A network for encoding, decoding and translating locative prepositions. Connection Science 3, 225–240 (1991)
Narendra, K.S., Parthasarathy, K.: Identification and control of dynamical systems using neural networks. IEEE Transactions on Neural Networks 1, 4–27 (1990)
Plaut, D.C.: Connectionist Neuropsychology: The Breakdown and Recovery of Behavior in Lesioned Attractor Networks. PhD thesis, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, Technical Report CMU-CS-91- 185 (1991)
Plaut, D.C., Shallice, T.: Perseverative and semantic influences on visual object naming errors in optic aphasia: A connectionist account. Technical Report PDP.CNS.92.1, Parallel Distributed Processing and Cognitive Neuroscience, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA (1992)
Pollack, J.B.: Recursivedistributed representations. Artificial Intelligence 46, 77–105 (1990)
Pollard, C., Sag, I.A.: Head-Driven Phrase Structure Grammar. University of Chicago Press, Chicago (1994)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, vol. 1, pp. 318–362. MIT Press, Cambridge (1986)
Sharkey, N.E., Sharkey, A.J.C.: A modular design for connectionist parsing. In: Drossaers, M.F.J., Nijholt, A. (eds.) Twente Workshop on Language Technology 3: Connectionism and Natural Language Processing, Department of Computer Science, University of Twente, Enschede, the Netherlands, pp. 87–96 (1992)
Simmons, R.F., Yu, Y.-H.: The acquisition and application of context sensitive grammar for English. In: Proceedings of the 29th Annual Meeting of the ACL. Association for Computational Linguistics, Morristown (1991)
Simmons, R.F., Yu, Y.-H.: The acquisition and use of context dependent grammars for English. Computational Linguistics 18, 391–418 (1992)
St. John, M.F., McClelland, J.L.: Learning and applying contextual constraints in sentence comprehension. Artificial Intelligence 46, 217–258 (1990)
Stolcke, A.: Learning feature-based semantics with simple recurrent networks. Technical Report TR-90-015, International Computer Science Institute, Berkeley, CA (1990)
Tomita, M.: Efficient Parsing for Natural Language. Kluwer, Dordrecht (1986)
Touretzky, D.S.: Connectionism and compositional semantics. In: Barnden, J.A., Pollack, J.B. (eds.) High-Level Connectionist Models, Advances in Connectionist and Neural Computation Theory, Ablex, Norwood, vol. 1, pp. 17–31 (1991)
Zelle, J.M., Mooney, R.J.: Comparative results on using inductive logic programming for corpus-based parser construction. In: Wermter, S., Riloff, E., Scheler, G. (eds.) Connectionist, Statistical, and Symbolic Approaches to Learning for Natural Language Processing, pp. 355–369. Springer, Berlin (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mayberry, M.R., Miikkulainen, R. (2000). Combining Maps and Distributed Representations for Shift-Reduce Parsing. In: Wermter, S., Sun, R. (eds) Hybrid Neural Systems. Hybrid Neural Systems 1998. Lecture Notes in Computer Science(), vol 1778. Springer, Berlin, Heidelberg. https://doi.org/10.1007/10719871_10
Download citation
DOI: https://doi.org/10.1007/10719871_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67305-7
Online ISBN: 978-3-540-46417-4
eBook Packages: Springer Book Archive