
Leveraging frozen pretrained written language models for neural sign language translation
- Author
- Mathieu De Coster (UGent) and Joni Dambre (UGent)
- Organization
- Project
- Abstract
- We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.
- Keywords
- Information Systems, sign language translation, machine translation, transfer learning
Downloads
-
information-13-00220-v2.pdf
- full text (Published version)
- |
- open access
- |
- |
- 751.24 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-8751070
- MLA
- De Coster, Mathieu, and Joni Dambre. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION, vol. 13, no. 5, 2022, doi:10.3390/info13050220.
- APA
- De Coster, M., & Dambre, J. (2022). Leveraging frozen pretrained written language models for neural sign language translation. INFORMATION, 13(5). https://doi.org/10.3390/info13050220
- Chicago author-date
- De Coster, Mathieu, and Joni Dambre. 2022. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION 13 (5). https://doi.org/10.3390/info13050220.
- Chicago author-date (all authors)
- De Coster, Mathieu, and Joni Dambre. 2022. “Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation.” INFORMATION 13 (5). doi:10.3390/info13050220.
- Vancouver
- 1.De Coster M, Dambre J. Leveraging frozen pretrained written language models for neural sign language translation. INFORMATION. 2022;13(5).
- IEEE
- [1]M. De Coster and J. Dambre, “Leveraging frozen pretrained written language models for neural sign language translation,” INFORMATION, vol. 13, no. 5, 2022.
@article{8751070, abstract = {{We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.}}, articleno = {{220}}, author = {{De Coster, Mathieu and Dambre, Joni}}, issn = {{2078-2489}}, journal = {{INFORMATION}}, keywords = {{Information Systems,sign language translation,machine translation,transfer learning}}, language = {{eng}}, number = {{5}}, pages = {{17}}, title = {{Leveraging frozen pretrained written language models for neural sign language translation}}, url = {{http://doi.org/10.3390/info13050220}}, volume = {{13}}, year = {{2022}}, }
- Altmetric
- View in Altmetric