Advanced search
2 files | 940.17 KB Add to list

Sign language recognition with transformer networks

Mathieu De Coster (UGent) , Mieke Van Herreweghe (UGent) and Joni Dambre (UGent)
Author
Organization
Abstract
Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.
Keywords
sign language recognition, deep learning, corpus annotation

Downloads

  • DeCoster Sign 2020.pdf
    • full text
    • |
    • open access
    • |
    • PDF
    • |
    • 434.76 KB
  • published.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 505.41 KB

Citation

Please use this url to cite or link to this publication:

MLA
De Coster, Mathieu, et al. “Sign Language Recognition with Transformer Networks.” 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings, European Language Resources Association (ELRA), 2020, pp. 6018–24.
APA
De Coster, M., Van Herreweghe, M., & Dambre, J. (2020). Sign language recognition with transformer networks. In 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings (pp. 6018–6024). Marseille, France: European Language Resources Association (ELRA).
Chicago author-date
De Coster, Mathieu, Mieke Van Herreweghe, and Joni Dambre. 2020. “Sign Language Recognition with Transformer Networks.” In 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings, 6018–24. European Language Resources Association (ELRA).
Chicago author-date (all authors)
De Coster, Mathieu, Mieke Van Herreweghe, and Joni Dambre. 2020. “Sign Language Recognition with Transformer Networks.” In 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings, 6018–6024. European Language Resources Association (ELRA).
Vancouver
1.
De Coster M, Van Herreweghe M, Dambre J. Sign language recognition with transformer networks. In: 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings. European Language Resources Association (ELRA); 2020. p. 6018–24.
IEEE
[1]
M. De Coster, M. Van Herreweghe, and J. Dambre, “Sign language recognition with transformer networks,” in 12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings, Marseille, France, 2020, pp. 6018–6024.
@inproceedings{8660743,
  abstract     = {{Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.}},
  author       = {{De Coster, Mathieu and Van Herreweghe, Mieke and Dambre, Joni}},
  booktitle    = {{12th International Conference on Language Resources and Evaluation (LREC 2020), Proceedings}},
  isbn         = {{9791095546344}},
  issn         = {{2522-2686}},
  keywords     = {{sign language recognition,deep learning,corpus annotation}},
  language     = {{eng}},
  location     = {{Marseille, France}},
  pages        = {{6018--6024}},
  publisher    = {{European Language Resources Association (ELRA)}},
  title        = {{Sign language recognition with transformer networks}},
  url          = {{http://www.lrec-conf.org/proceedings/lrec2020/LREC-2020.pdf}},
  year         = {{2020}},
}