What We Learned

We ran a number of different studies...

Watch my lips!

The study that we have analysed most extensively was the one where we explored how the brain of the native BSL signer takes account of mouth actions. Many signs use speechlike mouth actions. We were interested to see if the brain detected the difference between signs that may use such actions (like the sign BLUE) and signs that use mouth actions that are not speechlike (like the sign TRUE). Upper parts of the temporal lobe liked signs with speechlike mouth actions, and lower parts preferred signs where there was no speechlike action. This might be important in helping people with brain damage recover their BSL skills – and their speechreading. These findings have been published in Neuropsychologia and the Journal of Cognitive Neuroscience (see Publications page).

Fingerspelling

We also completed the first detailed study of how the brain perceives fingerspelled words. It turns out that fingerspelling activated the same regions as signed language – but more so. One region which was especially active for fingerspelling was a small region in the lower part of the temporal lobe which had already been identified as active when hearing people read written words (and which is less active in people with dyslexia). Because this region is involved in word reading, some researchers have called it the ‘visual word form area'. Until our work, it was thought that this region was specialised to process the visual characteristics of letter groups that make up words – letter groups like ‘BA' or ‘CH'. Since fingerspelled words activated this region too, and fingerspelled letters do not look very much like printed letters, this cannot be the whole story of the function of this region. Dafydd Waters and Mairéad MacSweeney are following up on this study. This study was published in NeuroImage (see Publications page).

How low can you go?

Another study with deaf native signers explored how the brain managed signs where you could not see the signer easily, but only small points of light on their body. You cannot identify who someone is or what they are signing when you see just a photograph of this: all you see is a jumble of points of light on a dark background. But when the person moves, you see the patterns of movement they make as they move their hands and body. In the first frame, below, the signer is standing still. In the second and third frames, you see the beginning and end points for the sign DRESS.

It's quite easy to read signs when they are degraded like this. The ‘visual movement' pathways of the brain allow these movement patterns to be interpreted as signs, and to identify aspects of the signer too. We found that the brain deals with these displays just as it deals with real signs, but the right side of the brain, especially the frontal lobes, were used more to try to identify the signer compared with identifying the sign. This finding might help us understand more about the visual processes that contribute to understanding sign language. Engineers want to know about this as it could provide a way to compress videoimages naturalistically, and to develop better signing robots and avatars (artificial signers). These studies are now being written up for publication.

Usher syndrome

We worked with people with Usher syndrome who were BSL signers, but had experience receiving sign language by touch when they were in environments that were difficult to see in. In the scanner, words were fingerspelled on volunteers' hands using the deafblind manual alphabet. Which brain regions are active? Naturally the ‘touch' regions (upper part of the brain, in the middle) were activated, but visual regions at the back of the brain were active too – suggesting that people were using visual imagery to support the hand readings. More work with Usher participants is going on at DCAL, with the help of Sarah Reed (SENSE). These studies are now being written up for publication.