Human and system perspectives on the expression of irony : an analysis of likelihood labels and rationales
- Author
- Aaron Maladry (UGent) , Alessandra Teresa Cignarella (UGent) , Els Lefever (UGent) , Cynthia Van Hee (UGent) and Veronique Hoste (UGent)
- Organization
- Project
- Abstract
- In this paper, we examine the recognition of irony by both humans and automatic systems. We achieve this by enhancing the annotations of an English benchmark data set for irony detection. This enhancement involves a layer of human-annotated irony likelihood using a 7-point Likert scale that combines binary annotation with a confidence measure. Additionally, the annotators indicated the trigger words that led them to perceive the text as ironic, which leveraged necessary theoretical insights into the definition of irony and its various forms. By comparing these trigger word spans across annotators, we determine the extent to which humans agree on the source of irony in a text. Finally, we compare the human-annotated spans with sub-token importance attributions for fine-tuned transformers using Layer Integrated Gradients, a state-of-the-art interpretability metric. Our results indicate that our model achieves better performance on tweets that were annotated with high confidence and high agreement. Although automatic systems can identify trigger words with relative success, they still attribute a significant amount of their importance to the wrong tokens.
- Keywords
- Irony Detection, Sarcasm, Explainability, Social Media, Annotation
Downloads
-
Human and System Perspectives on the Expression of Irony An Analysis of Likelihood Labels and Rationales.pdf
- full text (Published version)
- |
- open access
- |
- |
- 489.40 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HYWPM1A6V764Y867BXPFXADG
- MLA
- Maladry, Aaron, et al. “Human and System Perspectives on the Expression of Irony : An Analysis of Likelihood Labels and Rationales.” Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), edited by Nicoletta Calzolari et al., ELRA, 2024, pp. 8372–82.
- APA
- Maladry, A., Cignarella, A. T., Lefever, E., Van Hee, C., & Hoste, V. (2024). Human and system perspectives on the expression of irony : an analysis of likelihood labels and rationales. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, & N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 8372–8382). ELRA.
- Chicago author-date
- Maladry, Aaron, Alessandra Teresa Cignarella, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2024. “Human and System Perspectives on the Expression of Irony : An Analysis of Likelihood Labels and Rationales.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), edited by Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 8372–82. ELRA.
- Chicago author-date (all authors)
- Maladry, Aaron, Alessandra Teresa Cignarella, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2024. “Human and System Perspectives on the Expression of Irony : An Analysis of Likelihood Labels and Rationales.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ed by. Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 8372–8382. ELRA.
- Vancouver
- 1.Maladry A, Cignarella AT, Lefever E, Van Hee C, Hoste V. Human and system perspectives on the expression of irony : an analysis of likelihood labels and rationales. In: Calzolari N, Kan M-Y, Hoste V, Lenci A, Sakti S, Xue N, editors. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). ELRA; 2024. p. 8372–82.
- IEEE
- [1]A. Maladry, A. T. Cignarella, E. Lefever, C. Van Hee, and V. Hoste, “Human and system perspectives on the expression of irony : an analysis of likelihood labels and rationales,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Turin, Italy, 2024, pp. 8372–8382.
@inproceedings{01HYWPM1A6V764Y867BXPFXADG,
abstract = {{In this paper, we examine the recognition of irony by both humans and automatic systems. We achieve this by enhancing the annotations of an English benchmark data set for irony detection. This enhancement involves a layer of human-annotated irony likelihood using a 7-point Likert scale that combines binary annotation with a confidence measure. Additionally, the annotators indicated the trigger words that led them to perceive the text as ironic, which leveraged necessary theoretical insights into the definition of irony and its various forms. By comparing these trigger word spans across annotators, we determine the extent to which humans agree on the source of irony in a text. Finally, we compare the human-annotated spans with sub-token importance attributions for fine-tuned transformers using Layer Integrated Gradients, a state-of-the-art interpretability metric. Our results indicate that our model achieves better performance on tweets that were annotated with high confidence and high agreement. Although automatic systems can identify trigger words with relative success, they still attribute a significant amount of their importance to the wrong tokens.
}},
author = {{Maladry, Aaron and Cignarella, Alessandra Teresa and Lefever, Els and Van Hee, Cynthia and Hoste, Veronique}},
booktitle = {{Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}},
editor = {{Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen}},
isbn = {{9782493814104}},
issn = {{2951-2093}},
keywords = {{Irony Detection,Sarcasm,Explainability,Social Media,Annotation}},
language = {{eng}},
location = {{Turin, Italy}},
pages = {{8372--8382}},
publisher = {{ELRA}},
title = {{Human and system perspectives on the expression of irony : an analysis of likelihood labels and rationales}},
url = {{https://aclanthology.org/2024.lrec-main.734}},
year = {{2024}},
}