Academic Bibliography

 Search 200 years of publications by Ghent University researchers.
Advanced search
1 file | 365.17 KB
Add to list
  1. Search results
  2. Frozen pretrained transformers for ne...

Frozen pretrained transformers for neural sign language translation

Mathieu De Coster (UGent) , Karel D'Oosterlinck (UGent) , Marija Pizurica (UGent) , Paloma Rabaey (UGent) , Severine Verlinden, Mieke Van Herreweghe (UGent) and Joni Dambre (UGent)
(2021) Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL). p.88-97
Author
Organization
Project
Abstract
One of the major challenges in sign language translation from a sign language to a spoken language is the lack of parallel corpora. Recent works have achieved promising results on the RWTH-PHOENIX-Weather 2014T dataset, which consists of over eight thousand parallel sentences between German sign language and German. However, from the perspective of neural machine translation, this is still a tiny dataset. To improve the performance of models trained on small datasets, transfer learning can be used. While this has been previously applied in sign language translation for feature extraction, to the best of our knowledge, pretrained language models have not yet been investigated. We use pretrained BERT-base and mBART-50 models to initialize our sign language video to spoken language text translation model. To mitigate overfitting, we apply the frozen pretrained transformer technique: we freeze the majority of parameters during training. Using a pretrained BERT model, we outperform a baseline trained from scratch by 1 to 2 BLEU-4. Our results show that pretrained language models can be used to improve sign language translation performance and that the self-attention patterns in BERT transfer in zero-shot to the encoder and decoder of sign language translation models.

Downloads

  • Download
    DS445.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 365.17 KB

Citation

Please use this url to cite or link to this publication:

MLA
De Coster, Mathieu, et al. “Frozen Pretrained Transformers for Neural Sign Language Translation.” Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Association for Machine Translation in the Americas, 2021, pp. 88–97.
APA
De Coster, M., D’Oosterlinck, K., Pizurica, M., Rabaey, P., Verlinden, S., Van Herreweghe, M., & Dambre, J. (2021). Frozen pretrained transformers for neural sign language translation. Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 88–97. Association for Machine Translation in the Americas.
Chicago author-date
De Coster, Mathieu, Karel D’Oosterlinck, Marija Pizurica, Paloma Rabaey, Severine Verlinden, Mieke Van Herreweghe, and Joni Dambre. 2021. “Frozen Pretrained Transformers for Neural Sign Language Translation.” In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 88–97. Association for Machine Translation in the Americas.
Chicago author-date (all authors)
De Coster, Mathieu, Karel D’Oosterlinck, Marija Pizurica, Paloma Rabaey, Severine Verlinden, Mieke Van Herreweghe, and Joni Dambre. 2021. “Frozen Pretrained Transformers for Neural Sign Language Translation.” In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 88–97. Association for Machine Translation in the Americas.
Vancouver
1.
De Coster M, D’Oosterlinck K, Pizurica M, Rabaey P, Verlinden S, Van Herreweghe M, et al. Frozen pretrained transformers for neural sign language translation. In: Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL). Association for Machine Translation in the Americas; 2021. p. 88–97.
IEEE
[1]
M. De Coster et al., “Frozen pretrained transformers for neural sign language translation,” in Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Online, 2021, pp. 88–97.
@inproceedings{8719287,
  abstract     = {{One of the major challenges in sign language translation from a sign language to a spoken language is the lack of parallel corpora. Recent works have achieved promising results on the RWTH-PHOENIX-Weather 2014T dataset, which consists of over eight thousand parallel sentences between German sign language and German. However, from the perspective of neural machine translation, this is still a tiny dataset. To improve the performance of models trained on small datasets, transfer learning can be used. While this has been previously applied in sign language translation for feature extraction, to the best of our knowledge, pretrained language models have not yet been investigated. We use pretrained BERT-base and mBART-50 models to initialize our sign language video to spoken language text translation model. To mitigate overfitting, we apply the frozen pretrained transformer technique: we freeze the majority of parameters during training. Using a pretrained BERT model, we outperform a baseline trained from scratch by 1 to 2 BLEU-4. Our results show that pretrained language models can be used to improve sign language translation performance and that the self-attention patterns in BERT transfer in zero-shot to the encoder and decoder of sign language translation models.}},
  author       = {{De Coster, Mathieu and D'Oosterlinck, Karel and Pizurica, Marija and Rabaey, Paloma and Verlinden, Severine and Van Herreweghe, Mieke and Dambre, Joni}},
  booktitle    = {{Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)}},
  language     = {{eng}},
  location     = {{Online}},
  pages        = {{88--97}},
  publisher    = {{Association for Machine Translation in the Americas}},
  title        = {{Frozen pretrained transformers for neural sign language translation}},
  url          = {{https://aclanthology.org/2021.mtsummit-at4ssl.10.pdf}},
  year         = {{2021}},
}