Listening to audio can help language learners process illustrated stories

A UCL Institute of Education (IOE) study used eye-tracking to look at how second language learners process written text and images, known as multimodal input, with and without audio and what impact audio has on reading comprehension.

Theories of multimedia learning suggest that the simultaneous presentation of text and pictures supports learning. Also, adding audio in reading-while-listening conditions benefits performance on a variety of linguistic tasks.

Despite this, little research has been done on how people process multimedia input with and without accompanying audio.

Dr Ana Pellicer-Sánchez, Associate Professor in the IOE’s Centre for Applied Linguistics, recorded young learners’ eye movements when they were reading a multimedia story on its own as well as when they were reading while listening. Through doing this, the team investigated people’s looking patterns and their relationship with comprehension using a multiple-choice test.

Analysis of the eye-movement data showed that the presence of audio in reading-while-listening conditions allowed learners to look at the image more often, suggesting a better integration of text and pictures. Longer time looking at the pictures seemed to support comprehension of the story. However, longer time processing the text was indicative of processing difficulties.

Dr Ana Pellicer-Sánchez said: “This study shows that when we add auditory input to a multimodal text, it affects the way young learners process text and pictures. Adding auditory input in this type of multimodal materials supports the integration of text and pictures, which is key for comprehension and learning.

“Findings of the study provide further evidence for the positive role of images in reading comprehension. It also shows the benefits of using eye-tracking to investigate the relationship between processing patterns and comprehension.”

Links

• Read the paper: ‘Young Learners’ Processing of Multimodal Input and its Impact on Reading Comprehension: an Eye-Tracking Study’
• View Dr Ana Pellicer-Sánchez’s research profile
• Centre for Applied Linguistics
• Department of Culture, Communication and Media