Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level
- Author
- Margot Fonteyne (UGent) , Arda Tezcan (UGent) and Lieve Macken (UGent)
- Organization
- Project
- Abstract
- Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enormously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie's novel The Mysterious Affair at Styles with Google's NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel.
- Keywords
- literary machine translation, machine translation, quality assessment, document-level evaluation, translation technology, LT3, AGREEMENT, ENGLISH
Downloads
-
2020.lrec-1.467.pdf
- full text (Published version)
- |
- open access
- |
- |
- 1.48 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-8662553
- MLA
- Fonteyne, Margot, et al. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), European Language Resources Association (ELRA), 2020, pp. 3783–91.
- APA
- Fonteyne, M., Tezcan, A., & Macken, L. (2020). Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level. PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), 3783–3791. European Language Resources Association (ELRA).
- Chicago author-date
- Fonteyne, Margot, Arda Tezcan, and Lieve Macken. 2020. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), 3783–91. European Language Resources Association (ELRA).
- Chicago author-date (all authors)
- Fonteyne, Margot, Arda Tezcan, and Lieve Macken. 2020. “Literary Machine Translation under the Magnifying Glass : Assessing the Quality of an NMT-Translated Detective Novel on Document Level.” In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), 3783–3791. European Language Resources Association (ELRA).
- Vancouver
- 1.Fonteyne M, Tezcan A, Macken L. Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level. In: PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020). European Language Resources Association (ELRA); 2020. p. 3783–91.
- IEEE
- [1]M. Fonteyne, A. Tezcan, and L. Macken, “Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level,” in PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), Marseille, France, 2020, pp. 3783–3791.
@inproceedings{8662553,
abstract = {{Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enormously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie's novel The Mysterious Affair at Styles with Google's NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel.}},
author = {{Fonteyne, Margot and Tezcan, Arda and Macken, Lieve}},
booktitle = {{PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020)}},
isbn = {{9791095546344}},
issn = {{2522-2686}},
keywords = {{literary machine translation,machine translation,quality assessment,document-level evaluation,translation technology,LT3,AGREEMENT,ENGLISH}},
language = {{eng}},
location = {{Marseille, France}},
pages = {{3783--3791}},
publisher = {{European Language Resources Association (ELRA)}},
title = {{Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level}},
url = {{https://www.aclweb.org/anthology/2020.lrec-1.467}},
year = {{2020}},
}