Advanced search
1 file | 225.01 KB Add to list

A fine line between irony and sincerity : identifying bias in transformer models for irony detection

Aaron Maladry (UGent) , Els Lefever (UGent) , Cynthia Van Hee (UGent) and Veronique Hoste (UGent)
Author
Organization
Project
Abstract
In this paper we investigate potential bias in fine-tuned transformer models for irony detection. Bias is defined in this research as spurious associations between word n-grams and class labels, that can cause the system to rely too much on superficial cues and miss the essence of the irony. For this purpose, we looked for correlations between class labels and words that are prone to trigger irony, such as positive adjectives, intensifiers and topical nouns. Additionally, we investigate our irony model’s predictions before and after manipulating the data set through irony trigger replacements. We further support these insights with state-of-the-art explainability techniques (Layer Integrated Gradients, Discretized Integrated Gradients and Layer-wise Relevance Propagation). Both approaches confirm the hypothesis that transformer models generally encode correlations between positive sentiments and ironic texts, with even higher correlations between vividly expressed sentiment and irony. Based on these insights, we implemented a number of modification strategies to enhance the robustness of our irony classifier.
Keywords
LT3

Downloads

  • FineLine Maladry WASSA2023.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 225.01 KB

Citation

Please use this url to cite or link to this publication:

MLA
Maladry, Aaron, et al. “A Fine Line between Irony and Sincerity : Identifying Bias in Transformer Models for Irony Detection.” Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis, edited by Jeremy Barnes et al., Association for Computational Linguistics (ACL), 2023, pp. 315–24, doi:10.18653/v1/2023.wassa-1.28.
APA
Maladry, A., Lefever, E., Van Hee, C., & Hoste, V. (2023). A fine line between irony and sincerity : identifying bias in transformer models for irony detection. In J. Barnes, O. De Clercq, & R. Klinger (Eds.), Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis (pp. 315–324). https://doi.org/10.18653/v1/2023.wassa-1.28
Chicago author-date
Maladry, Aaron, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2023. “A Fine Line between Irony and Sincerity : Identifying Bias in Transformer Models for Irony Detection.” In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis, edited by Jeremy Barnes, Orphée De Clercq, and Roman Klinger, 315–24. Toronto, Canada: Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.wassa-1.28.
Chicago author-date (all authors)
Maladry, Aaron, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2023. “A Fine Line between Irony and Sincerity : Identifying Bias in Transformer Models for Irony Detection.” In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis, ed by. Jeremy Barnes, Orphée De Clercq, and Roman Klinger, 315–324. Toronto, Canada: Association for Computational Linguistics (ACL). doi:10.18653/v1/2023.wassa-1.28.
Vancouver
1.
Maladry A, Lefever E, Van Hee C, Hoste V. A fine line between irony and sincerity : identifying bias in transformer models for irony detection. In: Barnes J, De Clercq O, Klinger R, editors. Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis. Toronto, Canada: Association for Computational Linguistics (ACL); 2023. p. 315–24.
IEEE
[1]
A. Maladry, E. Lefever, C. Van Hee, and V. Hoste, “A fine line between irony and sincerity : identifying bias in transformer models for irony detection,” in Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis, Toronto, Canada, 2023, pp. 315–324.
@inproceedings{01H68Q8FTBXREBZPDT9T5ZY5F1,
  abstract     = {{In this paper we investigate potential bias in fine-tuned transformer models for irony detection. Bias is defined in this research as spurious associations between word n-grams and class labels, that can cause the system to rely too much on superficial cues and miss the essence of the irony. For this purpose, we looked for correlations between class labels and words that are prone to trigger irony, such as positive adjectives, intensifiers and topical nouns. Additionally, we investigate our irony model’s predictions before and after manipulating the data set through irony trigger replacements. We further support these insights with state-of-the-art explainability techniques (Layer Integrated Gradients, Discretized Integrated Gradients and Layer-wise Relevance Propagation). Both approaches confirm the hypothesis that transformer models generally encode correlations between positive sentiments and ironic texts, with even higher correlations between vividly expressed sentiment and irony. Based on these insights, we implemented a number of modification strategies to enhance the robustness of our irony classifier.}},
  author       = {{Maladry, Aaron and Lefever, Els and Van Hee, Cynthia and Hoste, Veronique}},
  booktitle    = {{Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis}},
  editor       = {{Barnes, Jeremy and De Clercq, Orphée and Klinger, Roman}},
  isbn         = {{9781959429876}},
  keywords     = {{LT3}},
  language     = {{eng}},
  location     = {{Toronto, Canada}},
  pages        = {{315--324}},
  publisher    = {{Association for Computational Linguistics (ACL)}},
  title        = {{A fine line between irony and sincerity : identifying bias in transformer models for irony detection}},
  url          = {{http://doi.org/10.18653/v1/2023.wassa-1.28}},
  year         = {{2023}},
}

Altmetric
View in Altmetric