
Machine translation from signed to spoken languages : state of the art and challenges
- Author
- Mathieu De Coster (UGent) , Dimitar Shterionov, Mieke Van Herreweghe (UGent) and Joni Dambre (UGent)
- Organization
- Project
- Abstract
- Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.
- Keywords
- SYSTEM, RECOGNITION, KINECT, Sign language, Computer vision, Machine translation, Deep learning, Literature review
Downloads
-
s10209-023-00992-1.pdf
- full text (Published version)
- |
- open access
- |
- |
- 1.52 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01GYS3X6Z60V1EVKG1C5CF5AVP
- MLA
- De Coster, Mathieu, et al. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, vol. 23, no. 3, 2024, pp. 1305–31, doi:10.1007/s10209-023-00992-1.
- APA
- De Coster, M., Shterionov, D., Van Herreweghe, M., & Dambre, J. (2024). Machine translation from signed to spoken languages : state of the art and challenges. UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, 23(3), 1305–1331. https://doi.org/10.1007/s10209-023-00992-1
- Chicago author-date
- De Coster, Mathieu, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2024. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 23 (3): 1305–31. https://doi.org/10.1007/s10209-023-00992-1.
- Chicago author-date (all authors)
- De Coster, Mathieu, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2024. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 23 (3): 1305–1331. doi:10.1007/s10209-023-00992-1.
- Vancouver
- 1.De Coster M, Shterionov D, Van Herreweghe M, Dambre J. Machine translation from signed to spoken languages : state of the art and challenges. UNIVERSAL ACCESS IN THE INFORMATION SOCIETY. 2024;23(3):1305–31.
- IEEE
- [1]M. De Coster, D. Shterionov, M. Van Herreweghe, and J. Dambre, “Machine translation from signed to spoken languages : state of the art and challenges,” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, vol. 23, no. 3, pp. 1305–1331, 2024.
@article{01GYS3X6Z60V1EVKG1C5CF5AVP, abstract = {{Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.}}, author = {{De Coster, Mathieu and Shterionov, Dimitar and Van Herreweghe, Mieke and Dambre, Joni}}, issn = {{1615-5289}}, journal = {{UNIVERSAL ACCESS IN THE INFORMATION SOCIETY}}, keywords = {{SYSTEM,RECOGNITION,KINECT,Sign language,Computer vision,Machine translation,Deep learning,Literature review}}, language = {{eng}}, number = {{3}}, pages = {{1305--1331}}, title = {{Machine translation from signed to spoken languages : state of the art and challenges}}, url = {{http://doi.org/10.1007/s10209-023-00992-1}}, volume = {{23}}, year = {{2024}}, }
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: