Advanced search
1 file | 1.48 MB Add to list

Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level

Margot Fonteyne (UGent) , Arda Tezcan (UGent) and Lieve Macken (UGent)
Author
Organization
Project
Abstract
Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enormously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie's novel The Mysterious Affair at Styles with Google's NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel.
Keywords
literary machine translation, machine translation, quality assessment, document-level evaluation, translation technology, LT3

Downloads

  • 2020.lrec-1.467.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 1.48 MB

Citation

Please use this url to cite or link to this publication:

MLA
Fonteyne, Margot, et al. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings, edited by Nicoletta Calzolari et al., European Language Resources Association (ELRA), 2020, pp. 3783–91.
APA
Fonteyne, M., Tezcan, A., & Macken, L. (2020). Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, … S. Piperidis (Eds.), 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings (pp. 3783–3791). Paris, France: European Language Resources Association (ELRA).
Chicago author-date
Fonteyne, Margot, Arda Tezcan, and Lieve Macken. 2020. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” In 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings, edited by Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, et al., 3783–91. Paris, France: European Language Resources Association (ELRA).
Chicago author-date (all authors)
Fonteyne, Margot, Arda Tezcan, and Lieve Macken. 2020. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” In 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings, ed by. Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, 3783–3791. Paris, France: European Language Resources Association (ELRA).
Vancouver
1.
Fonteyne M, Tezcan A, Macken L. Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level. In: Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, et al., editors. 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings. Paris, France: European Language Resources Association (ELRA); 2020. p. 3783–91.
IEEE
[1]
M. Fonteyne, A. Tezcan, and L. Macken, “Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level,” in 12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings, Marseille, France, 2020, pp. 3783–3791.
@inproceedings{8662553,
  abstract     = {Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enormously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie's novel The Mysterious Affair at Styles with Google's NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel.},
  author       = {Fonteyne, Margot and Tezcan, Arda and Macken, Lieve},
  booktitle    = {12th International Conference on Language Resources and Evaluation Conference (LREC 2020), Proceedings},
  editor       = {Calzolari, Nicoletta and Béchet, Frédéric and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, Hélène and Moreno, Asuncion and Odijk, Jan and Piperidis, Stelios},
  isbn         = {9791095546344},
  issn         = {2522-2686},
  keywords     = {literary machine translation,machine translation,quality assessment,document-level evaluation,translation technology,LT3},
  language     = {eng},
  location     = {Marseille, France},
  pages        = {3783--3791},
  publisher    = {European Language Resources Association (ELRA)},
  title        = {Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level},
  url          = {https://www.aclweb.org/anthology/2020.lrec-1.467},
  year         = {2020},
}