Limitations of audiovisual speech on robots for second language pronunciation learning
- Author
- Saya Amioka, Ruben Janssens (UGent) , Pieter Wolfert, Qiaoqiao Ren (UGent) , Maria Jose Pinto Bernal (UGent) and Tony Belpaeme (UGent)
- Organization
- Project
- Abstract
- The perception of audiovisual speech plays an important role in infants' first language acquisition and continues to be important for language understanding beyond infancy. Beyond that, the perception of speech and congruent lip motion supports language understanding for adults, and it has been suggested that second language learning benefits from audiovisual speech, as it helps learners distinguish speech sounds in the target language. In this paper, we study whether congruent audiovisual speech on a robot facilitates the learning of Japanese pronunciation. 27 native-Dutch speaking participants were trained in Japanese pronunciation by a social robot. The robot demonstrated 30 Japanese words of varying complexity using either congruent audiovisual speech, incongruent visual speech, or computer-generated audiovisual speech. Participants were asked to imitate the robot's pronunciation, recordings of which were rated by native Japanese speakers. Against expectation, the results showed that congruent audiovisual speech resulted in lower pronunciation performance than low-fidelity or incongruent speech. We show that our learners, being native Dutch speakers, are only very weakly sensitive to audiovisual Japanese speech which possibly explains why learning performance does not seem to benefit from audiovisual speech.
- Keywords
- human-robot interaction, robot-assisted language learning, multimodal interaction, audiovisual speech, orofacial animations, HEARING LIPS, PERCEPTION
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 2.95 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01GVZ7K3P8TAE3E3E0VR8ERAAS
- MLA
- Amioka, Saya, et al. “Limitations of Audiovisual Speech on Robots for Second Language Pronunciation Learning.” PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023, Association for Computing Machinery (ACM), 2023, pp. 359–67, doi:10.1145/3568162.3578633.
- APA
- Amioka, S., Janssens, R., Wolfert, P., Ren, Q., Pinto Bernal, M. J., & Belpaeme, T. (2023). Limitations of audiovisual speech on robots for second language pronunciation learning. PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023, 359–367. https://doi.org/10.1145/3568162.3578633
- Chicago author-date
- Amioka, Saya, Ruben Janssens, Pieter Wolfert, Qiaoqiao Ren, Maria Jose Pinto Bernal, and Tony Belpaeme. 2023. “Limitations of Audiovisual Speech on Robots for Second Language Pronunciation Learning.” In PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023, 359–67. New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/3568162.3578633.
- Chicago author-date (all authors)
- Amioka, Saya, Ruben Janssens, Pieter Wolfert, Qiaoqiao Ren, Maria Jose Pinto Bernal, and Tony Belpaeme. 2023. “Limitations of Audiovisual Speech on Robots for Second Language Pronunciation Learning.” In PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023, 359–367. New York: Association for Computing Machinery (ACM). doi:10.1145/3568162.3578633.
- Vancouver
- 1.Amioka S, Janssens R, Wolfert P, Ren Q, Pinto Bernal MJ, Belpaeme T. Limitations of audiovisual speech on robots for second language pronunciation learning. In: PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023. New York: Association for Computing Machinery (ACM); 2023. p. 359–67.
- IEEE
- [1]S. Amioka, R. Janssens, P. Wolfert, Q. Ren, M. J. Pinto Bernal, and T. Belpaeme, “Limitations of audiovisual speech on robots for second language pronunciation learning,” in PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023, Stockholm, Sweden, 2023, pp. 359–367.
@inproceedings{01GVZ7K3P8TAE3E3E0VR8ERAAS,
abstract = {{The perception of audiovisual speech plays an important role in infants' first language acquisition and continues to be important for language understanding beyond infancy. Beyond that, the perception of speech and congruent lip motion supports language understanding for adults, and it has been suggested that second language learning benefits from audiovisual speech, as it helps learners distinguish speech sounds in the target language. In this paper, we study whether congruent audiovisual speech on a robot facilitates the learning of Japanese pronunciation. 27 native-Dutch speaking participants were trained in Japanese pronunciation by a social robot. The robot demonstrated 30 Japanese words of varying complexity using either congruent audiovisual speech, incongruent visual speech, or computer-generated audiovisual speech. Participants were asked to imitate the robot's pronunciation, recordings of which were rated by native Japanese speakers. Against expectation, the results showed that congruent audiovisual speech resulted in lower pronunciation performance than low-fidelity or incongruent speech. We show that our learners, being native Dutch speakers, are only very weakly sensitive to audiovisual Japanese speech which possibly explains why learning performance does not seem to benefit from audiovisual speech.}},
author = {{Amioka, Saya and Janssens, Ruben and Wolfert, Pieter and Ren, Qiaoqiao and Pinto Bernal, Maria Jose and Belpaeme, Tony}},
booktitle = {{PROCEEDINGS OF THE 2023 ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI 2023}},
isbn = {{9781450399647}},
issn = {{2167-2121}},
keywords = {{human-robot interaction,robot-assisted language learning,multimodal interaction,audiovisual speech,orofacial animations,HEARING LIPS,PERCEPTION}},
language = {{eng}},
location = {{Stockholm, Sweden}},
pages = {{359--367}},
publisher = {{Association for Computing Machinery (ACM)}},
title = {{Limitations of audiovisual speech on robots for second language pronunciation learning}},
url = {{http://doi.org/10.1145/3568162.3578633}},
year = {{2023}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: