When the student becomes the master : learning better and smaller monolingual models from mBERT
- Author
- Pranaydeep Singh (UGent) and Els Lefever (UGent)
- Organization
- Abstract
- In this research, we present pilot experiments to distil monolingual models from a jointly trained model for 102 languages (mBERT). We demonstrate that it is possible for the target language to outperform the original model, even with a basic distillation setup. We evaluate our methodology for 6 languages with varying amounts of resources and belonging to different language families.
- Keywords
- LT3
Downloads
-
2022.coling-1.391.pdf
- full text (Published version)
- |
- open access
- |
- |
- 629.88 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-8769631
- MLA
- Singh, Pranaydeep, and Els Lefever. “When the Student Becomes the Master : Learning Better and Smaller Monolingual Models from MBERT.” Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022), International Committee on Computational Linguistics, 2022, pp. 4434–41.
- APA
- Singh, P., & Lefever, E. (2022). When the student becomes the master : learning better and smaller monolingual models from mBERT. Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022), 4434–4441. International Committee on Computational Linguistics.
- Chicago author-date
- Singh, Pranaydeep, and Els Lefever. 2022. “When the Student Becomes the Master : Learning Better and Smaller Monolingual Models from MBERT.” In Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022), 4434–41. International Committee on Computational Linguistics.
- Chicago author-date (all authors)
- Singh, Pranaydeep, and Els Lefever. 2022. “When the Student Becomes the Master : Learning Better and Smaller Monolingual Models from MBERT.” In Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022), 4434–4441. International Committee on Computational Linguistics.
- Vancouver
- 1.Singh P, Lefever E. When the student becomes the master : learning better and smaller monolingual models from mBERT. In: Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022). International Committee on Computational Linguistics; 2022. p. 4434–41.
- IEEE
- [1]P. Singh and E. Lefever, “When the student becomes the master : learning better and smaller monolingual models from mBERT,” in Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022), Gyeongju, Republic of Korea, 2022, pp. 4434–4441.
@inproceedings{8769631,
abstract = {{In this research, we present pilot experiments to distil monolingual models from a jointly trained model for 102 languages (mBERT). We demonstrate that it is possible for the target language to outperform the original model, even with a basic distillation setup. We evaluate our methodology for 6 languages with varying amounts of resources and belonging to different language families.}},
author = {{Singh, Pranaydeep and Lefever, Els}},
booktitle = {{Proceedings of the 29th International Conference on Computational Linguistic (COLING 2022)}},
issn = {{2951-2093}},
keywords = {{LT3}},
language = {{eng}},
location = {{Gyeongju, Republic of Korea}},
pages = {{4434--4441}},
publisher = {{International Committee on Computational Linguistics}},
title = {{When the student becomes the master : learning better and smaller monolingual models from mBERT}},
url = {{https://aclanthology.org/2022.coling-1.391}},
year = {{2022}},
}