Advanced search
2 files | 11.74 MB Add to list

A review of evaluation practices of gesture generation in embodied conversational agents

Author
Organization
Project
Abstract
Embodied conversational agents (ECAs) are often designed to produce nonverbal behavior to complement or enhance their verbal communication. One such form of the nonverbal behavior is co-speech gesturing, which involves movements that the agent makes with its arms and hands that are paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, divided into rule-based and data-driven processes, with the latter, gaining traction because of the increasing interest from the applied machine learning community. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we present a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic, and beat gestures, including reported evaluation methods. We review 22 studies that have an ECA with a human-like upper body that uses co-speech gesturing in social human-agent interaction. This includes studies that use human participants to evaluate performance. We found most studies use a within-subject design and rely on a form of subjective evaluation, but without a systematic approach. We argue that the field requires more rigorous and uniform tools for co-speech gesture evaluation, and formulate recommendations for empirical evaluation, including standardized phrases and example scenarios to help systematically test generative models across studies. Furthermore, we also propose a checklist that can be used to report relevant information for the evaluation of generative models, as well as to evaluate co-speech gesture use.
Keywords
Artificial Intelligence, Computer Networks and Communications, Computer Science Applications, Human-Computer Interaction, Signal Processing, Control and Systems Engineering, Human Factors and Ergonomics, Measurement, Systematics, Databases, Data mining, Avatars, Protocols, Neural networks, Human–computer interface, human–robot interaction, social robotics, virtual interaction, PSYCHOLOGICAL REACTANCE, SOCIAL CUES, ROBOT, SPEECH, COMMUNICATION

Downloads

  • camera ready.pdf
    • full text (Accepted manuscript)
    • |
    • open access
    • |
    • PDF
    • |
    • 9.95 MB
  • (...).pdf
    • full text (Published version)
    • |
    • UGent only
    • |
    • PDF
    • |
    • 1.78 MB

Citation

Please use this url to cite or link to this publication:

MLA
Wolfert, Pieter, et al. “A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents.” IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, vol. 52, no. 3, 2022, pp. 379–89, doi:10.1109/thms.2022.3149173.
APA
Wolfert, P., Robinson, N., & Belpaeme, T. (2022). A review of evaluation practices of gesture generation in embodied conversational agents. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, 52(3), 379–389. https://doi.org/10.1109/thms.2022.3149173
Chicago author-date
Wolfert, Pieter, Nicole Robinson, and Tony Belpaeme. 2022. “A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents.” IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS 52 (3): 379–89. https://doi.org/10.1109/thms.2022.3149173.
Chicago author-date (all authors)
Wolfert, Pieter, Nicole Robinson, and Tony Belpaeme. 2022. “A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents.” IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS 52 (3): 379–389. doi:10.1109/thms.2022.3149173.
Vancouver
1.
Wolfert P, Robinson N, Belpaeme T. A review of evaluation practices of gesture generation in embodied conversational agents. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS. 2022;52(3):379–89.
IEEE
[1]
P. Wolfert, N. Robinson, and T. Belpaeme, “A review of evaluation practices of gesture generation in embodied conversational agents,” IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, vol. 52, no. 3, pp. 379–389, 2022.
@article{8743014,
  abstract     = {{Embodied conversational agents (ECAs) are often designed to produce nonverbal behavior to complement or enhance their verbal communication. One such form of the nonverbal behavior is co-speech gesturing, which involves movements that the agent makes with its arms and hands that are paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, divided into rule-based and data-driven processes, with the latter, gaining traction because of the increasing interest from the applied machine learning community. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we present a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic, and beat gestures, including reported evaluation methods. We review 22 studies that have an ECA with a human-like upper body that uses co-speech gesturing in social human-agent interaction. This includes studies that use human participants to evaluate performance. We found most studies use a within-subject design and rely on a form of subjective evaluation, but without a systematic approach. We argue that the field requires more rigorous and uniform tools for co-speech gesture evaluation, and formulate recommendations for empirical evaluation, including standardized phrases and example scenarios to help systematically test generative models across studies. Furthermore, we also propose a checklist that can be used to report relevant information for the evaluation of generative models, as well as to evaluate co-speech gesture use.}},
  author       = {{Wolfert, Pieter and Robinson, Nicole and Belpaeme, Tony}},
  issn         = {{2168-2291}},
  journal      = {{IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS}},
  keywords     = {{Artificial Intelligence,Computer Networks and Communications,Computer Science Applications,Human-Computer Interaction,Signal Processing,Control and Systems Engineering,Human Factors and Ergonomics,Measurement,Systematics,Databases,Data mining,Avatars,Protocols,Neural networks,Human–computer interface,human–robot interaction,social robotics,virtual interaction,PSYCHOLOGICAL REACTANCE,SOCIAL CUES,ROBOT,SPEECH,COMMUNICATION}},
  language     = {{eng}},
  number       = {{3}},
  pages        = {{379--389}},
  title        = {{A review of evaluation practices of gesture generation in embodied conversational agents}},
  url          = {{http://doi.org/10.1109/thms.2022.3149173}},
  volume       = {{52}},
  year         = {{2022}},
}

Altmetric
View in Altmetric
Web of Science
Times cited: