Counter-hypothesis generation : towards evaluating how LLMs reason about alternatives
- Author
- Marzieh Abdolmaleki (UGent) , Aaron Maladry (UGent) , Veronique Hoste (UGent) and Els Lefever (UGent)
- Organization
- Project
- Abstract
- Reasoning about alternatives is a fundamental component of human cognition and argumentation, yet it remains unclear whether large language models (LLMs) can coherently generate and assess them. This paper introduces Counter-Hypothesis Generation (CHG), a novel task for evaluating how LLMs construct plausible hypotheses when contextual information changes. Inspired by open-domain commonsense reasoning, where models infer and compare multiple explanations, CHG bridges commonsense and counterfactual reasoning by requiring models to generate hypotheses that remain logically consistent with modified premises. We present a test set annotated by a human expert and complemented with counter-hypotheses generated by OpenAI-o3 and DeepSeek-r1. Experimental results reveal that even advanced reasoning models exhibit notable limitations in counter-hypothesis generation.
- Keywords
- Counterfactual Reasoning, Hypothesis Generation, Large Language Models
Downloads
-
Counter-Hypothesis Generation- Towards Evaluating How LLMs Reason About Alternatives.pdf
- full text (Published version)
- |
- open access
- |
- |
- 622.54 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01M1BMB4HQHVGSV4BKQ62A9EV2
- MLA
- Abdolmaleki, Marzieh, et al. “Counter-Hypothesis Generation : Towards Evaluating How LLMs Reason about Alternatives.” Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), edited by Stelios Piperidis et al., European Language Resources Association (ELRA), 2026, pp. 5445–49, doi:10.63317/336pssnozxaw.
- APA
- Abdolmaleki, M., Maladry, A., Hoste, V., & Lefever, E. (2026). Counter-hypothesis generation : towards evaluating how LLMs reason about alternatives. In S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, & A. Toral (Eds.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) (pp. 5445–5449). https://doi.org/10.63317/336pssnozxaw
- Chicago author-date
- Abdolmaleki, Marzieh, Aaron Maladry, Veronique Hoste, and Els Lefever. 2026. “Counter-Hypothesis Generation : Towards Evaluating How LLMs Reason about Alternatives.” In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), edited by Stelios Piperidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, and Antonio Toral, 5445–49. European Language Resources Association (ELRA). https://doi.org/10.63317/336pssnozxaw.
- Chicago author-date (all authors)
- Abdolmaleki, Marzieh, Aaron Maladry, Veronique Hoste, and Els Lefever. 2026. “Counter-Hypothesis Generation : Towards Evaluating How LLMs Reason about Alternatives.” In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), ed by. Stelios Piperidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, and Antonio Toral, 5445–5449. European Language Resources Association (ELRA). doi:10.63317/336pssnozxaw.
- Vancouver
- 1.Abdolmaleki M, Maladry A, Hoste V, Lefever E. Counter-hypothesis generation : towards evaluating how LLMs reason about alternatives. In: Piperidis S, Bel N, van den Heuvel H, Ide N, Krek S, Toral A, editors. Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association (ELRA); 2026. p. 5445–9.
- IEEE
- [1]M. Abdolmaleki, A. Maladry, V. Hoste, and E. Lefever, “Counter-hypothesis generation : towards evaluating how LLMs reason about alternatives,” in Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), Palma de Mallorca, Spain, 2026, pp. 5445–5449.
@inproceedings{01M1BMB4HQHVGSV4BKQ62A9EV2,
abstract = {{Reasoning about alternatives is a fundamental component of human cognition and argumentation, yet it remains unclear whether large language models (LLMs) can coherently generate and assess them. This paper introduces Counter-Hypothesis Generation (CHG), a novel task for evaluating how LLMs construct plausible hypotheses when contextual information changes. Inspired by open-domain commonsense reasoning, where models infer and compare multiple explanations, CHG bridges commonsense and counterfactual reasoning by requiring models to generate hypotheses that remain logically consistent with modified premises. We present a test set annotated by a human expert and complemented with counter-hypotheses generated by OpenAI-o3 and DeepSeek-r1. Experimental results reveal that even advanced reasoning models exhibit notable limitations in counter-hypothesis generation.}},
author = {{Abdolmaleki, Marzieh and Maladry, Aaron and Hoste, Veronique and Lefever, Els}},
booktitle = {{Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)}},
editor = {{Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio}},
isbn = {{9782493814494}},
issn = {{2522-2686}},
keywords = {{Counterfactual Reasoning,Hypothesis Generation,Large Language Models}},
language = {{eng}},
location = {{Palma de Mallorca, Spain}},
pages = {{5445--5449}},
publisher = {{European Language Resources Association (ELRA)}},
title = {{Counter-hypothesis generation : towards evaluating how LLMs reason about alternatives}},
url = {{http://doi.org/10.63317/336pssnozxaw}},
year = {{2026}},
}
- Altmetric
- View in Altmetric