Speech Science Forum: Prof Gabriella Vigliocco

Ecological Language: a multimodal approach to the study of language learning and processing

The brain has evolved the ability to support communication in complex and dynamic environments. In such environments, language is learned, and mostly used in face-to-face contexts in which processing and learning are based on multiple cues both linguistic and non-linguistic (such as gestures, eye gaze, mouth patterns and prosody). Yet, our understanding of how language is learnt and processed comes mostly from reductionist approaches in which the multimodal signal is reduced to speech or text. I will introduce our current programme of research that investigates language in real-world settings in which the listener/learner has access to -- and therefore can take advantage of -- the multiple cues provided by the speaker. I will then describe studies that aim at characterising the distribution of the multimodal cues in the language used by caregivers when interacting with their children (mostly 2-4 years old) and provide data concerning how these cues are differentially distributed depending upon whether the child knows the objects being talked about (allowing us to more clearly isolate learning episodes), and whether objects being talked about are present. I will then move to a study using EEG addressing the question of how discourse but crucially also the non-linguistic cues modulate predictions about the next word in a sentence.