Large language models cover for speech recognition mistakes : evaluating conversational AI for second language learners
- Author
- Eva Verhelst (UGent) and Tony Belpaeme (UGent)
- Organization
- Project
- Abstract
- Automatic Speech Recognition (ASR) technology has been reported to reach near-human performance in recent years, yet it continues to struggle with atypical speakers, particularly second language learners. This limitation has hindered progress in leveraging social robots for second language education, a field with significant promise. Recent advancements in Large Language Models (LLMs), which demonstrate capabilities in context understanding, common sense reasoning, and pragmatics, offer a potential solution by compensating for transcription errors introduced by ASR. This study examines whether ASR combined with an LLM can produce flowing conversation. Particularly, we look at its application in learning French as a second language by Dutch-speaking students. Through task-based interactions, where successful task completion depends on the accurate interpretation of user speech, the study evaluates the impact of LLMs on conversational outcomes. Results confirm that the performance of ASR degrades significantly for both speakers with limited proficiency and a non-English language. Nonetheless, LLMs demonstrate the ability to interpret context and sustain meaningful conversations despite suboptimal ASR outputs, highlighting a promising path forward for the integration of these technologies in second-language education.
- Keywords
- l2 speakers, large language models, pragmatics, speech recognition
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 4.10 MB
-
DS878 acc.pdf
- full text (Accepted manuscript)
- |
- open access
- |
- |
- 1.28 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01JPMTYSFGD77QJ5V9J6FSANNF
- MLA
- Verhelst, Eva, and Tony Belpaeme. “Large Language Models Cover for Speech Recognition Mistakes : Evaluating Conversational AI for Second Language Learners.” 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI, IEEE, 2025, pp. 1705–09, doi:10.1109/HRI61500.2025.10974188.
- APA
- Verhelst, E., & Belpaeme, T. (2025). Large language models cover for speech recognition mistakes : evaluating conversational AI for second language learners. 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI, 1705–1709. https://doi.org/10.1109/HRI61500.2025.10974188
- Chicago author-date
- Verhelst, Eva, and Tony Belpaeme. 2025. “Large Language Models Cover for Speech Recognition Mistakes : Evaluating Conversational AI for Second Language Learners.” In 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI, 1705–9. IEEE. https://doi.org/10.1109/HRI61500.2025.10974188.
- Chicago author-date (all authors)
- Verhelst, Eva, and Tony Belpaeme. 2025. “Large Language Models Cover for Speech Recognition Mistakes : Evaluating Conversational AI for Second Language Learners.” In 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI, 1705–1709. IEEE. doi:10.1109/HRI61500.2025.10974188.
- Vancouver
- 1.Verhelst E, Belpaeme T. Large language models cover for speech recognition mistakes : evaluating conversational AI for second language learners. In: 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI. IEEE; 2025. p. 1705–9.
- IEEE
- [1]E. Verhelst and T. Belpaeme, “Large language models cover for speech recognition mistakes : evaluating conversational AI for second language learners,” in 2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI, Melbourne, Australia, 2025, pp. 1705–1709.
@inproceedings{01JPMTYSFGD77QJ5V9J6FSANNF,
abstract = {{Automatic Speech Recognition (ASR) technology has been reported to reach near-human performance in recent years, yet it continues to struggle with atypical speakers, particularly second language learners. This limitation has hindered progress in leveraging social robots for second language education, a field with significant promise. Recent advancements in Large Language Models (LLMs), which demonstrate capabilities in context understanding, common sense reasoning, and pragmatics, offer a potential solution by compensating for transcription errors introduced by ASR. This study examines whether ASR combined with an LLM can produce flowing conversation. Particularly, we look at its application in learning French as a second language by Dutch-speaking students. Through task-based interactions, where successful task completion depends on the accurate interpretation of user speech, the study evaluates the impact of LLMs on conversational outcomes. Results confirm that the performance of ASR degrades significantly for both speakers with limited proficiency and a non-English language. Nonetheless, LLMs demonstrate the ability to interpret context and sustain meaningful conversations despite suboptimal ASR outputs, highlighting a promising path forward for the integration of these technologies in second-language education.}},
author = {{Verhelst, Eva and Belpaeme, Tony}},
booktitle = {{2025 20TH ACM/IEEE INTERNATIONAL CONFERENCE ON HUMAN-ROBOT INTERACTION, HRI}},
isbn = {{9798350378948}},
issn = {{2167-2121}},
keywords = {{l2 speakers,large language models,pragmatics,speech recognition}},
language = {{eng}},
location = {{Melbourne, Australia}},
pages = {{1705--1709}},
publisher = {{IEEE}},
title = {{Large language models cover for speech recognition mistakes : evaluating conversational AI for second language learners}},
url = {{http://doi.org/10.1109/HRI61500.2025.10974188}},
year = {{2025}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: