Dr Guoxing Yu receives British Academy research grant
28 April 2016
Audio-visual Lexical Tone Perception: Evidence from Eye-tracking Studies
PI: Dr Biao ZENG, Senior Research Fellow, Psychology, (Bournemouth University)
Co-Applicant: Dr Guoxing Yu, Reader in Language Education & Assessment, GSoE
Lip movement gives an indication of how consonants and vowels are being pronounced. Lexical tone is the distinctive pitch level carried by the syllable of a word and is an essential feature of the meaning of the word in some Asian and African languages, however lexical tones offer hardly any visual cues in lip movements because they are produced by vocal cords. Lexical tone is acoustically dominated by fundamental frequency (F0); changing of the pitch results in the change of meaning of the word. For example, Mandarin Chinese has five lexical tones of ma: mā (媽/妈) 'mom/mum', má (麻/麻) 'hemp', mǎ (馬/马) 'horse', mà (罵/骂) 'scold', ma (嗎/吗) (an interrogative particle). Perception of lexical tones in Mandarin Chinese presents one of the most significant challenges to learners of Chinese as a second or foreign language.
This one-year project funded by British Academy/Leverhulme Trust Small Research Grant will investigate the extent to which Chinese and English native speakers use any dynamic facial information, especially lip movement, to perceive and recognise Mandarin lexical tones, as well as the differences between the two groups of participants. They will be presented with Chinese monosyllables in three modes: audio-only, audio-visual and visual-only. The participants’ eye-movements will be recorded to measure both the point of gaze (where one is looking) and saccadic pattern amongst different facial regions of the speaker. The eye-tracking studies will be supplemented with two behavioural paradigms: (1) lexical tone identification and (2) lexical tone discrimination. In the lexical tone identification tasks in both clear and noisy conditions, the participants are asked to decide which lexical tone they hear. With auditory information degraded in a noisy condition, we expect that the participants turn their eyes to the width, height or duration of the lip movements in order to enhance their lexical tone perceptions. In the lexical discrimination tasks, the participants are asked to decide whether the two lexical tones are the same or not.
We expect the findings of the study will be useful for a number of stakeholders, e.g., (1) for learners of Chinese as a foreign/second language to better understand what visual cues that they need to take into consideration when learning lexical tones, (2) for language institutes and publishers to design visual materials to support student learning, and (3) for test providers to improve their video-based listening assessment tasks.