Language Processing

Jeff Elman

University of California, San Diego

 

Language has been one of the most fruitful domains for studying human cognition. Not only does it play a central role in human activity, making possible cultural, social,  and intellectual activity that is unparalleled among the animal kingdom; but it is also a domain that is richly complex and attractive as an arena for formal modeling. Not surprisingly, the history of modern computation, cognitive theory, and language reveal tight and important interconnections.

 

The major focus of this course will be on language, from a connectionist perspective. We will begin with a brief historical review of the intellectual roots of modern cognitive science (from the mid-1800s to present). The bulk of the course will then a set of phenomena in psycholinguistics and language acquisition, viewed from a connectionist perspective, and concluding with a discussion of implications for linguistic theory.

 

 

Day 1: Historical roots of modern cognitive science

        Psychology in the 19th century

        Behaviorism

        Cybernetics, computation, AI

        The cognitive revolution

        Cognition revised: Connectionism, Artificial Life, Situated & Embodied Cognition, Dynamical Systems

 

Required Readings

 

A short history of AI

 

Von Neumann, J. (1948/1963). The general and logical theory of automata. In A.H. Taub, Ed., John von Neumann, Collected Works, Vol. 5. Oxford: Pergamon Press.

 

Turing, A.M. (1950). Computing Machinery and Intelligence. In Readings in Cognitive Science: A Perspective from Psychology and Artificial Intelligence, (1988). A. Collins and E. E. Smith (Eds). Kaufmann, San Mateo, CA.

        (Only available on CD)

 

 

Optional Readings

 

Chomsky, N. (1959). On certain formal properties of grammars. Information and Control, 2, 137-167.

Chomsky, N. (1959). A review of B. F. Skinner's Verbal Behavior. Language, 35, 26-58.

 

 

Day 2: Early connectionist language models

 

        The word superiority effect model

        The TRACE model of speech perception

        The problem of learning, and a solution

        Rules or networks:  The past tense debate

 

Required Readings

Rumelhart, D.E., & McClelland, J.L. (1986). On learning the past tenses of English verbs. In J.L. McClelland and D.E. Rumelhart (Eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 2. Cambridge, MA: MIT Press. Ch. 18.

Rumelhart, D.E., Hinton, G.E., & Williams, R. (1986). Learning internal representations by error propagation. In D.E. Rumelhart and J.L. McClelland (Eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. Cambridge, MA: MIT Press. Ch. 9.

Annotated reading list of past tense, infant learning, and generalization

 

 

Optional Readings

 

McClelland, J.L. & Rumelhart, D.E. (1991). An interactive activation model of context effects in letter perception: Part 1. Psychological Review, 5, 375-407.

McClelland, J.L. & Elman, J.L. (1986). Interactive processes in speech perception: The TRACE Model. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. II. Cambridge, MA: MIT Press.

Hare, M., & Elman, J.L. (1995). Learning and morphological change. Cognition, 56, 61-98.

 

 

 

 

Day 3: Language acquisition

 

        The poverty of the stimulus; lack of negative feedback

        Generalization vs. conservatism: Empirical findings

        What do infants and children know, and when do they know it?

speech perception; language identification; word segmentation; grammatical categories

        Infants and grammar learning

        (The past tense)

 

Required Readings

Bates, E., & Goodman, J. C. (1997). On the inseparability of grammar and the lexicon: Evidence from acquisition, aphasia, and real-time processing. Language and Cognitive Processes, 12, 507-584.

 

Marcus, G. F., Vijayan, S., Rao, S. B., & Vishton, P. M. (1999). Rule learning by seven-month-old infants. Science, 283(5398), 77-80.

 

Seidenberg, M. S., & Elman, J. L. (1999a). Do infants learn grammar with algebra or statistics. Science, 284, 434-435.

 

Seidenberg, M. S., & Elman, J. L. (1999b). Networks are not 'hidden rules'. Trends in Cognitive Sciences, 3(8), 288-289.

 

Optional Readings

Lewis, J., & Elman, J. (2001). A connectionist investigation of linguistic arguments from the poverty of the stimulus: Learning the unlearnable.

 

Gomez, R. L., & Gerken, L. (1999). Artificial grammar learning by 1-year-olds leads to specific and abstract knowledge. Cognition, 70(2), 109-135.

 

Plunkett, K., & Marchman, V. (1993). From Rote Learning to System Building - Acquiring Verb Morphology in Children and Connectionist Nets. Cognition, 48(1), 21-69.

 

Plunkett, K., & Juola, P. (1999). A connectionist model of english past tense and plural morphology, Cognitive Science, 23, 463-490.

 

 

 

 

Day 4: Sentence processing: Experimental results and modeling

 

        Basic issues and phenomena in sentence processing

        The sausage machine and two-stage processor

        Constraint satisfaction / probabilistic / expectation generation models

 

Required Readings

 

Tanenhaus, M.K., & Trueswell, J.C., (1995). Sentence comprehension. In J.L. Miller and P.D. Eimas (Eds.) Handbook of Perception and Cognition, 2nd edition. Vol. 11: Speech, Language, and Communication. NY: Academic Press. Pp. 217-262.

 

McRae, K., Spivey-Knowlton, M.J., & Tanenhaus, M.K. (1998). Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38, 283-312.

 

McRae, K., Hare, M., Elman, J.L., & Ferretti, T. (2006). A basis for generating expectancies for verbs from nouns. Memory and Cognition, 33, 1174-1184.

 

 

 

Optional Readings

 

Seidenberg, M.S., & MacDonald, M.C. (1999). A probabilistic constraints approach to language acquisition and processing. Cognitive Science, 23, 569-588.

 

Christiansen, M.H., & Chater, N. (1999). Connectionist natural language processing: the state of the art, Cognitive Science, 23, 417-437.

 

 

Day 5: Dynamical approaches & linguistic theory

 

        Sequential processing

        Simple (and other) recurrent networks

        The lexicon and grammar, rethought

        Dynamical analyses and usage-based grammars

        The role of event knowledge in language

 

Required Readings

 

Elman, J.L. (1990). Finding structure in time. Cognitive Science, 14, 179-211.

 

Elman, J.L. (draft). On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. (in draft format; please do not distribute or cite)\

 

 

Optional Readings

 

Elman, J.L. (1995). Language as a dynamical system. In R. Port and T. van Gelder (Eds.), Mind as Motion: Explorations in the Dynamics of Cognition. Cambridge, MA: MIT Press. Pp. 195-223.

 

Rodriguez, P., Wiles, J., & Elman, J.L. (1999). A recurrent neural network that learns to count. Connection Science, 11, 5-40.

 

 

 

 

Afternoon section meetings

Afternoon sessions will be used to explore specific topics in depth and to present additional material.

 

 

Assessment

Students who take the course for credit will be asked to write a brief (5-7 page) paper that critical reviews one or more of the articles read in class, or to comment on other work that is related to the issues discussed in the class.

 

Jeff Elman

Jeff Elman received his Ph.D. in 1977 from the University of Texas at Austin. He joined the Department of Linguistics at the University of California, San Diego in that year. In 1986, he participated in the founding of the Department of Cognitive Science at UC San Diego, serving as Chair of that department from 1990 to 1994. He was Chair of the Governing Board of the Cognitive Science Society in 1999-2000. In 2007, Jeff Elman won the Rumelhart Prize. His research interests include language processing, connectionist models, human development, and the evolution of cognition.