Distractor generation for multiple-choice questions with predictive prompting and large language models
- Author
- Semere Kiros Bitew (UGent) , Johannes Deleu (UGent) , Chris Develder (UGent) and Thomas Demeester (UGent)
- Organization
- Abstract
- Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable performance across various tasks and have garnered significant attention from both researchers and practitioners. However, in an educational context, we still observe a performance gap in generating distractors-i.e., plausible yet incorrect answers-with LLMs for multiple-choice questions (MCQs). In this study, we propose a strategy for guiding LLMs such as ChatGPT, in generating relevant distractors by prompting them with question items automatically retrieved from a question bank as well-chosen in-context examples. We evaluate our LLM-based solutions using a quantitative assessment on an existing test set, as well as through quality annotations by human experts, i.e., teachers. We found that on average 53% of the generated distractors presented to the teachers were rated as high-quality, i.e., suitable for immediate use as is, outperforming the state-of-the-art model. We also show the gains of our approach (https://github.com/semerekiros/distractGPT/) in generating high-quality distractors by comparing it with a zero-shot ChatGPT and a few-shot ChatGPT prompted with static examples.
- Keywords
- Distractor generation, natural language processing, large language models, predictive prompting, language learning, neural networks, TESTS
Downloads
-
(...).pdf
- full text (Accepted manuscript)
- |
- UGent only
- |
- |
- 315.43 KB
-
(...).pdf
- full text (Accepted manuscript)
- |
- UGent only
- |
- |
- 29.97 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HAKWXY2CJ9GW5T5SN59FX3TN
- MLA
- Bitew, Semere Kiros, et al. “Distractor Generation for Multiple-Choice Questions with Predictive Prompting and Large Language Models.” MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II, vol. 2134, 2025, pp. 48–63, doi:10.1007/978-3-031-74627-7_4.
- APA
- Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2025). Distractor generation for multiple-choice questions with predictive prompting and large language models. MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II, 2134, 48–63. https://doi.org/10.1007/978-3-031-74627-7_4
- Chicago author-date
- Bitew, Semere Kiros, Johannes Deleu, Chris Develder, and Thomas Demeester. 2025. “Distractor Generation for Multiple-Choice Questions with Predictive Prompting and Large Language Models.” In MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II, 2134:48–63. https://doi.org/10.1007/978-3-031-74627-7_4.
- Chicago author-date (all authors)
- Bitew, Semere Kiros, Johannes Deleu, Chris Develder, and Thomas Demeester. 2025. “Distractor Generation for Multiple-Choice Questions with Predictive Prompting and Large Language Models.” In MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II, 2134:48–63. doi:10.1007/978-3-031-74627-7_4.
- Vancouver
- 1.Bitew SK, Deleu J, Develder C, Demeester T. Distractor generation for multiple-choice questions with predictive prompting and large language models. In: MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II. 2025. p. 48–63.
- IEEE
- [1]S. K. Bitew, J. Deleu, C. Develder, and T. Demeester, “Distractor generation for multiple-choice questions with predictive prompting and large language models,” in MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II, Turin, Italy, 2025, vol. 2134, pp. 48–63.
@inproceedings{01HAKWXY2CJ9GW5T5SN59FX3TN,
abstract = {{Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable performance across various tasks and have garnered significant attention from both researchers and practitioners. However, in an educational context, we still observe a performance gap in generating distractors-i.e., plausible yet incorrect answers-with LLMs for multiple-choice questions (MCQs). In this study, we propose a strategy for guiding LLMs such as ChatGPT, in generating relevant distractors by prompting them with question items automatically retrieved from a question bank as well-chosen in-context examples. We evaluate our LLM-based solutions using a quantitative assessment on an existing test set, as well as through quality annotations by human experts, i.e., teachers. We found that on average 53% of the generated distractors presented to the teachers were rated as high-quality, i.e., suitable for immediate use as is, outperforming the state-of-the-art model. We also show the gains of our approach (https://github.com/semerekiros/distractGPT/) in generating high-quality distractors by comparing it with a zero-shot ChatGPT and a few-shot ChatGPT prompted with static examples.}},
author = {{Bitew, Semere Kiros and Deleu, Johannes and Develder, Chris and Demeester, Thomas}},
booktitle = {{MACHINE LEARNING AND PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2023, PT II}},
isbn = {{9783031746260}},
issn = {{1865-0929}},
keywords = {{Distractor generation,natural language processing,large language models,predictive prompting,language learning,neural networks,TESTS}},
language = {{eng}},
location = {{Turin, Italy}},
pages = {{48--63}},
title = {{Distractor generation for multiple-choice questions with predictive prompting and large language models}},
url = {{http://doi.org/10.1007/978-3-031-74627-7_4}},
volume = {{2134}},
year = {{2025}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: