Advanced search
1 file | 751.24 KB Add to list

Leveraging frozen pretrained written language models for neural sign language translation

Mathieu De Coster (UGent) and Joni Dambre (UGent)
(2022) INFORMATION. 13(5).
Author
Organization
Project
Abstract
We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.
Keywords
Information Systems, sign language translation, machine translation, transfer learning

Downloads

  • information-13-00220-v2.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 751.24 KB

Citation

Please use this url to cite or link to this publication:

MLA
De Coster, Mathieu, and Joni Dambre. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION, vol. 13, no. 5, 2022, doi:10.3390/info13050220.
APA
De Coster, M., & Dambre, J. (2022). Leveraging frozen pretrained written language models for neural sign language translation. INFORMATION, 13(5). https://doi.org/10.3390/info13050220
Chicago author-date
De Coster, Mathieu, and Joni Dambre. 2022. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION 13 (5). https://doi.org/10.3390/info13050220.
Chicago author-date (all authors)
De Coster, Mathieu, and Joni Dambre. 2022. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION 13 (5). doi:10.3390/info13050220.
Vancouver
1.
De Coster M, Dambre J. Leveraging frozen pretrained written language models for neural sign language translation. INFORMATION. 2022;13(5).
IEEE
[1]
M. De Coster and J. Dambre, “Leveraging frozen pretrained written language models for neural sign language translation,” INFORMATION, vol. 13, no. 5, 2022.
@article{8751070,
  abstract     = {{We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.}},
  articleno    = {{220}},
  author       = {{De Coster, Mathieu and Dambre, Joni}},
  issn         = {{2078-2489}},
  journal      = {{INFORMATION}},
  keywords     = {{Information Systems,sign language translation,machine translation,transfer learning}},
  language     = {{eng}},
  number       = {{5}},
  pages        = {{17}},
  title        = {{Leveraging frozen pretrained written language models for neural sign language translation}},
  url          = {{http://doi.org/10.3390/info13050220}},
  volume       = {{13}},
  year         = {{2022}},
}

Altmetric
View in Altmetric