Co-speech gestures facilitate language comprehension

Abstract

How do co-speech gestures aid in language comprehension? It is proposed that observed gestures are associated with processing in brain regions involved in the production of hand and arm movements (Skipper et al., 2009) because of the the predictive ability of motor cortex (Skipper et al., 2006). The brain predicts, e.g., forthcoming words associated with observed gestures. These predictions constrain interpretation and allow the brain to expend less time processing incoming sensory information, allowing resources to be allocated elsewhere. This model was tested using functional magnetic resonance imaging (fMRI) while participants watched a television game show containing natural co-speech gestures. It was hypothesized that 1) Gestures would be associated with additional processing in brain regions involved in the production of movement and areas involved in semantic aspects of spoken language comprehension (e.g., anterior superior temporal cortex; STa); 2) Processing in production areas associated with gestures would temporally precede activity in language comprehension areas if the motor system is engaged in a predictive capacity; and 3) There would be a concomitant decrease in processing in “early” sensory cortices, i.e., areas of cortex that are a relatively few number of synapses away from peripheral sensory receptors. These hypotheses were tested with a new fMRI analysis method called “turnpoints analysis”. Turnpoints analysis allows researchers to analyze timeseries data resulting from naturalistic stimuli rather than use the subtractive technique typically employed in fMRI studies that must make use of tightly controlled stimuli. This method was developed because of the difficulty of creating stimuli containing artificial co-speech gestures that maintain the intricate timing relationship between speech and gesture. Results indicate that, relative to speech alone, co-speech gestures were more associated with processing in pre- and primary motor (PM) and STa cortex. Hand and arm movements not associated with speech (e.g., self-adaptor movements and buzzer presses) were not associated with processing in PM or STa cortex but, rather, with processing in visual and parietal areas. Activity in PM cortex associated with gesture preceded activity in STa cortex. Finally, relative to speech without gesture, there was a marked decrease in processing in early sensory areas when gestures were observed (e.g., in Heschel’s gyrus and the calcarine fissure, usually associated with primary auditory and visual cortex respectively). Ancillary analyses show that the reduction of processing in sensory areas is a function of the amount of semantic content contained in gestures and a more general property of language comprehension when preceding discourse content highly semantically constrained the predictability of forthcoming words. In conclusion, gestures seem to be used by the brain to predict forthcoming words associated with those gestures and this results in a decrease in processing in early sensory areas. This is likely because less evidence is required from early sensory areas to confirm what the talker is attempting to say. Thus, gestures would seem to aid in language comprehension by freeing up brain resources to perform other functions, e.g., processing associated with further elaboration of a talker’s intended meaning.

Acknowledgements

Research supported by NIH-NICHD K99/R00 HD060307 – “Neurobiology of Speech Perception in Real-World Contexts”

References

Skipper, J. I., Goldin-Meadow, S., & Small, S. L. (2009). Gestures orchestrate brain networks for language understanding. Current Biology, 19(8). 661-667

Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2006). Lending a helping hand to hearing: Another motor theory of speech perception. Cambridge, MA: Cambridge University Press