Advanced search
1 file | 202.08 KB Add to list

How 'open' are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation

Author
Organization
Abstract
Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain conversations are not clear. To clarify the boundaries of “openness”, we conduct two studies: First, we classify the types of “speech events” encountered in a chatbot evaluation data set (i.e., Meena by Google) and find that these conversations mainly cover the “small talk” category and exclude the other speech event categories encountered in real life human-human communication. Second, we conduct a small-scale pilot study to generate online conversations covering a wider range of speech event categories between two humans vs. a human and a state-of-the-art chatbot (i.e., Blender by Facebook). A human evaluation of these generated conversations indicates a preference for human-human conversations, since the human-chatbot conversations lack coherence in most speech event categories. Based on these results, we suggest (a) using the term “small talk” instead of “open-domain” for the current chatbots which are not that “open” in terms of conversational abilities yet, and (b) revising the evaluation methods to test the chatbot conversations against other speech events.
Keywords
SIGDIAL, open-domain chatbots, speech events, LT3, small-talk chatbots, chatbot evaluation, Turing Test

Downloads

  • Dogruoz Skantze 2021.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 202.08 KB

Citation

Please use this url to cite or link to this publication:

MLA
Doğruöz, A. Seza, and Gabriel Skantze. “How ‘open’ Are the Conversations with Open-Domain Chatbots? A Proposal for Speech Event Based Evaluation.” SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021), edited by Haizhou Li et al., Association for Computational Linguistics (ACL), 2021, pp. 392–402.
APA
Doğruöz, A. S., & Skantze, G. (2021). How “open” are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation. In H. Li, G.-A. Levow, Z. Yu, C. Gupta, B. Sisman, S. Cai, … J. J. Li (Eds.), SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021) (pp. 392–402). Singapore, SINGAPORE: Association for Computational Linguistics (ACL).
Chicago author-date
Doğruöz, A. Seza, and Gabriel Skantze. 2021. “How ‘open’ Are the Conversations with Open-Domain Chatbots? A Proposal for Speech Event Based Evaluation.” In SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021), edited by Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefts, Yan Wu, and Junji Jessy Li, 392–402. Association for Computational Linguistics (ACL).
Chicago author-date (all authors)
Doğruöz, A. Seza, and Gabriel Skantze. 2021. “How ‘open’ Are the Conversations with Open-Domain Chatbots? A Proposal for Speech Event Based Evaluation.” In SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021), ed by. Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefts, Yan Wu, and Junji Jessy Li, 392–402. Association for Computational Linguistics (ACL).
Vancouver
1.
Doğruöz AS, Skantze G. How “open” are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation. In: Li H, Levow G-A, Yu Z, Gupta C, Sisman B, Cai S, et al., editors. SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021). Association for Computational Linguistics (ACL); 2021. p. 392–402.
IEEE
[1]
A. S. Doğruöz and G. Skantze, “How ‘open’ are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation,” in SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021), Singapore, SINGAPORE, 2021, pp. 392–402.
@inproceedings{8712379,
  abstract     = {{Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain conversations are not clear. To clarify the boundaries of “openness”, we conduct two studies: First, we classify the types of “speech events” encountered in a chatbot evaluation data set (i.e., Meena by Google) and find that these conversations mainly cover the “small talk” category and exclude the other speech event categories encountered in real life human-human communication. Second, we conduct a small-scale pilot study to generate online conversations covering a wider range of speech event categories between two humans vs. a human and a state-of-the-art chatbot (i.e., Blender by Facebook). A human evaluation of these generated conversations indicates a preference for human-human conversations, since the human-chatbot conversations lack coherence in most speech event categories. Based on these results, we suggest (a) using the term “small talk” instead of “open-domain” for the current chatbots which are not that “open” in terms of conversational abilities yet, and (b) revising the evaluation methods to test the chatbot conversations against other speech events.}},
  author       = {{Doğruöz, A. Seza and Skantze, Gabriel}},
  booktitle    = {{SIGDIAL 2021 : 22ND ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE (SIGDIAL 2021)}},
  editor       = {{Li, Haizhou and Levow, Gina-Anne and Yu, Zhou and Gupta, Chitralekha and Sisman, Berrak and Cai, Siqi and Vandyke, David and Dethlefts, Nina and Wu, Yan and Li, Junji Jessy}},
  isbn         = {{9781954085817}},
  keywords     = {{SIGDIAL,open-domain chatbots,speech events,LT3,small-talk chatbots,chatbot evaluation,Turing Test}},
  language     = {{eng}},
  location     = {{Singapore, SINGAPORE}},
  pages        = {{392--402}},
  publisher    = {{Association for Computational Linguistics (ACL)}},
  title        = {{How 'open' are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation}},
  url          = {{https://aclanthology.org/2021.sigdial-1.0}},
  year         = {{2021}},
}

Web of Science
Times cited: