Advanced search
1 file | 1.26 MB Add to list

Machine translation from signed to spoken languages : state of the art and challenges

Author
Organization
Project
Abstract
Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.
Keywords
SYSTEM, RECOGNITION, KINECT, Sign language, Computer vision, Machine translation, Deep learning, Literature review

Downloads

  • DS618.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 1.26 MB

Citation

Please use this url to cite or link to this publication:

MLA
De Coster, Mathieu, et al. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, 2023, pp. 1–27, doi:10.1007/s10209-023-00992-1.
APA
De Coster, M., Shterionov, D., Van Herreweghe, M., & Dambre, J. (2023). Machine translation from signed to spoken languages : state of the art and challenges. UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, 1–27. https://doi.org/10.1007/s10209-023-00992-1
Chicago author-date
De Coster, Mathieu, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2023. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, 1–27. https://doi.org/10.1007/s10209-023-00992-1.
Chicago author-date (all authors)
De Coster, Mathieu, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2023. “Machine Translation from Signed to Spoken Languages : State of the Art and Challenges.” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY: 1–27. doi:10.1007/s10209-023-00992-1.
Vancouver
1.
De Coster M, Shterionov D, Van Herreweghe M, Dambre J. Machine translation from signed to spoken languages : state of the art and challenges. UNIVERSAL ACCESS IN THE INFORMATION SOCIETY. 2023;1–27.
IEEE
[1]
M. De Coster, D. Shterionov, M. Van Herreweghe, and J. Dambre, “Machine translation from signed to spoken languages : state of the art and challenges,” UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, pp. 1–27, 2023.
@article{01GYS3X6Z60V1EVKG1C5CF5AVP,
  abstract     = {{Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.}},
  author       = {{De Coster, Mathieu and Shterionov, Dimitar and Van Herreweghe, Mieke and Dambre, Joni}},
  issn         = {{1615-5289}},
  journal      = {{UNIVERSAL ACCESS IN THE INFORMATION SOCIETY}},
  keywords     = {{SYSTEM,RECOGNITION,KINECT,Sign language,Computer vision,Machine translation,Deep learning,Literature review}},
  language     = {{eng}},
  pages        = {{1--27}},
  title        = {{Machine translation from signed to spoken languages : state of the art and challenges}},
  url          = {{http://doi.org/10.1007/s10209-023-00992-1}},
  year         = {{2023}},
}

Altmetric
View in Altmetric
Web of Science
Times cited: