Connectionist Models of Language Processing

Jeff Elman

University of California, San Diego

Over the past 15 years, a number of approaches to the study of language have emerged are quite different than those which derive from Chomsky's generative theory. Many of these new frameworks bear a family resemblance to each other; and many are also highly compatible with connectionist models. In this course I will present a historical overview of connectionism (with particular emphasis on language), from the early 1980's up to the present. I will be particularly concerned with the questions of (a) whether and how connectionist models differ fundamentally from symbolic approaches, and (b) which approach provides greater insight into various language phenomena.

• Introduction to course
• Historical background: the roots of digital vs. neural computation; issues in linguistic theory and psycholinguistics
• Motivations for the PDP approach

McClelland, J.L. & Rumelhart, D.E. (1991). An interactive activation model of context effects in letter perception: Part 1. Psychological Review, 5, 375-407.

McClelland, J.L., Rumelhart, D.E., & Hinton, G.E. (1986). The appeal of parallel distributed processing. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. I. Cambridge, MA: MIT Press.

McClelland, J.L. & Elman, J.L. (1986). Interactive processes in speech perception: The TRACE Model. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. II. Cambridge, MA: MIT Press.

• Hebbian learning; the Perceptron; backpropagation of error
• NETtalk; other early language models

Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986). Learning internal representations by error propagation. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. I. Cambridge, MA: MIT Press.

• the English past tense

Rumelhart, D.E., & McClelland, J.L. (1986). Learning the past tense. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. II. Cambridge, MA: MIT Press.

Hare, M., & Elman, J.L. (1995). Learning and morphological change. Cognition, 56, 61-98.

Pinker, S., & Prince, A. (1988). On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28, 73-193.

Plunkett, K., & Marchman, V. (1993) From rote learning to system building. i, 48, 21-69.

Janisse, M, & Seidenberg, M.

Plaut, D.C., McClelland, J.L., Seidenberg, M.S., & Patterson, K.E. (1996). Understanding normal and impaired reading: Computational principles in quasi-regular domains. Psychological Review, 103, 56-115

• infant behavior
• the critical period
• induction and generalization in the face of sparse evidence
• the challenge: representing constituent structure: Fodor & Pylyshyn; Elman
• language and dynamics

Gomez, R.L., & Gerken, L. (1999). Artificial grammar learning by 1-year-olds leads to specific and abstract knowledge. Cognition, 70, 109-135.

Elman, J.L. (1990). Finding structure in time. Cognitive Science, 14, 179-211.

Marchman, V.A. (1993). Constraints on plasticity in a connectionist model of the past tense. Journal of Cognitive Neuroscience, 5, 215-234.

Marcus, G.F., Vijayan, S., Bandi Rao, S., Vishton, P.M. (1999). Rule learning by seven-month-old infants. Science, 283, 77-80.

Elman, J.L. (1998). Generalization, simple recurrent networks, and the emergence of structure. In M.A. Gernsbacher & S. Derry (Eds.)., Proceedings of the 20th Annual Conference of the Cognitive Science Society. Mahway, NJ: Lawrence Erlbaum Associates.

• ambiguity resolution and thematic role assignment

Elman, J.L. (1995). Language as a dynamical system. In R.F. Port & T. van Gelder (Eds.) Mind as Motion. Cambridge, MA: MIT Press.

Seidenberg, M.S. (1997). Language acquisition and use: Learning and applying probabilistic constraints. Science, 275, 1599-1602.

St. John, M.F., & McClelland, J.L. Learning and applying contextual constraints in sentence comprehension. Artificial Intelligence, 46, 217-257.

MacDonald, M.C. (1994) Probabilistic constraints and syntactic ambiguity resolution. Language and Cognitive Processes, 9, 157-201.

McRae, K., Spivey-Knowlton, M.J., & Tanenhaus, M.K. (1998). Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38, 283-312.

Small groups

Participants will get hands-on experience with various connectionist simulation models. Discussions on topics raised by the participants.

Assessment

Students who desire credit write a paper on critical issues in modeling language processing.

Jeff Elman is Professor of Cognitive Science at the University of California, San Diego. He received his Ph.D. (Linguistics) from the University of Texas at Austin and has been at UCSD since 1977. A founding member of the Cognitive Science department and recent chair, Elman's primary research interests are in connectionist models of development and language processing. With Jay McClelland, he developed the TRACE model of speech perception. Subsequently, he developed the Simple Recurrent Network architecture, which has been widely used to model sequential phenomena in language. He is co-author of 'Rethinking Innateness: A Connectionist Perspective on Development' (MIT Press, 1996). He is the President of the Cognitive Science Society.