Scientists have found our understanding of language may depend more heavily on vision than previously thought: under the right conditions, what you see can override what you hear.
The brain considers both sight and sound when processing speech. However, if the two are slightly different, visual cues dominate sound.
This phenomenon is named the McGurk effect for Scottish cognitive psychologist Harry McGurk, who pioneered studies on the link between hearing and vision in speech perception in the 1970s. The McGurk effect has been observed for decades. However, its origin has been elusive.
In the new study, published in the journal PLOS ONE, the University of Utah team pinpointed the source of the McGurk effect by recording and analysing brain signals in the temporal cortex, the region of the brain that typically processes sound.
"For the first time, we were able to link the auditory signal in the brain to what a person said they heard when what they actually heard was something different," said study's first author, Elliot Smith, a bioengineering and neuroscience graduate student at the university.
"We found vision is influencing the hearing part of the brain to change your perception of reality - and you can't turn off the illusion," Smith said.
Researchers recorded electrical signals from the brain surfaces of four severely epileptic adults (two male, two female) from Utah and Idaho.
These four test subjects were then asked to watch and listen to videos focused on a person's mouth as they said the syllables "ba," "va," "ga" and "tha."
Depending on which of three different videos were being watched, the patients had one of three possible experiences as they watched the syllables being mouthed.
In one scenario, the motion of the mouth matched the sound. For example, the video showed "ba" and the audio sound also was "ba," so the patients saw and heard "ba."
In another experience, the motion of the mouth did not match the corresponding sound, like a badly dubbed movie.
In the third one, the motion of the mouth only was mismatched slightly with the corresponding sound. For example, the video showed "ba" but the audio was "va," and patients heard "ba" even though the sound really was "va."
This demonstrates the McGurk effect - vision overriding hearing.
By measuring the electrical signals in the brain while each video was being watched, Smith and Greger could pinpoint whether auditory or visual brain signals were being used to identify the syllable in each video.
When the syllable being mouthed matched the sound or didn't match at all, brain activity increased in correlation to the sound being watched.
However, when the McGurk effect video was viewed, the activity pattern changed to resemble what the person saw, not what they heard, researchers found.
