Advanced search
1 file | 257.43 KB Add to list
Author
Organization
Abstract
Detecting event mentions is the first step in event extraction from text and annotating them is a notoriously difficult task. Evaluating annotator consistency is crucial when building datasets for mention detection. When event mentions are allowed to cover many tokens, annotators may disagree on their span, which means that overlapping annotations may then refer to the same event or to different events. This paper explores different fuzzy matching functions which aim to resolve this ambiguity. The functions extract the sets of syntactic heads present in the annotations, use the Dice coefficient to measure the similarity between sets and return a judgment based on a given threshold. The functions are tested against the judgments of a human evaluator and a comparison is made between sets of tokens and sets of syntactic heads. The best-performing function is a head-based function that is found to agree with the human evaluator in 89% of cases.
Keywords
LT3

Downloads

  • D19-5903.pdf
    • full text
    • |
    • open access
    • |
    • PDF
    • |
    • 257.43 KB

Citation

Please use this url to cite or link to this publication:

MLA
Colruyt, Camiel, et al. “Leveraging Syntactic Parsing to Improve Event Annotation Matching.” Aggregating and Analysing Crowdsourced Annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, Association for Computational Linguistics (ACL), 2019, pp. 15–23, doi:10.18653/v1/d19-5903.
APA
Colruyt, C., De Clercq, O., & Hoste, V. (2019). Leveraging syntactic parsing to improve event annotation matching. In Aggregating and analysing crowdsourced annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (pp. 15–23). Hong Kong: Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/d19-5903
Chicago author-date
Colruyt, Camiel, Orphée De Clercq, and Veronique Hoste. 2019. “Leveraging Syntactic Parsing to Improve Event Annotation Matching.” In Aggregating and Analysing Crowdsourced Annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, 15–23. Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/d19-5903.
Chicago author-date (all authors)
Colruyt, Camiel, Orphée De Clercq, and Veronique Hoste. 2019. “Leveraging Syntactic Parsing to Improve Event Annotation Matching.” In Aggregating and Analysing Crowdsourced Annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, 15–23. Association for Computational Linguistics (ACL). doi:10.18653/v1/d19-5903.
Vancouver
1.
Colruyt C, De Clercq O, Hoste V. Leveraging syntactic parsing to improve event annotation matching. In: Aggregating and analysing crowdsourced annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. Association for Computational Linguistics (ACL); 2019. p. 15–23.
IEEE
[1]
C. Colruyt, O. De Clercq, and V. Hoste, “Leveraging syntactic parsing to improve event annotation matching,” in Aggregating and analysing crowdsourced annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, Hong Kong, 2019, pp. 15–23.
@inproceedings{8636394,
  abstract     = {{Detecting event mentions is the first step in event extraction from text and annotating them is a notoriously difficult task. Evaluating annotator consistency is crucial when building datasets for mention detection. When event mentions are allowed to cover many tokens, annotators may disagree on their span, which means that overlapping annotations may then refer to the same event or to different events.
This paper explores different fuzzy matching functions which aim to resolve this ambiguity. The functions extract the sets of syntactic heads present in the annotations, use the Dice coefficient to measure the similarity between sets and return a judgment based on a given threshold. The functions are tested against the judgments of a human evaluator and a comparison is made between sets of tokens and sets of syntactic heads. The best-performing function is a head-based function that is found to agree with the human evaluator in 89% of cases.}},
  author       = {{Colruyt, Camiel and De Clercq, Orphée and Hoste, Veronique}},
  booktitle    = {{Aggregating and analysing crowdsourced annotations for NLP : Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP}},
  isbn         = {{9781950737802}},
  keywords     = {{LT3}},
  language     = {{eng}},
  location     = {{Hong Kong}},
  pages        = {{15--23}},
  publisher    = {{Association for Computational Linguistics (ACL)}},
  title        = {{Leveraging syntactic parsing to improve event annotation matching}},
  url          = {{http://dx.doi.org/10.18653/v1/d19-5903}},
  year         = {{2019}},
}

Altmetric
View in Altmetric