A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning
(2024)
- Author
- Nicolas Dewolf (UGent)
- Promoter
- Bernard De Baets (UGent) and Willem Waegeman (UGent)
- Organization
- Abstract
- In the past decades, most work in the area of data analysis and machine learning was focused on optimizing predictive models and getting better results than what was possible with existing models. To what extent the metrics with which such improvements were measured were accurately capturing the intended goal, whether the numerical differences in the resulting values were significant, or whether uncertainty played a role in this study and if it should have been taken into account, was of secondary importance. Whereas probability theory, be it frequentist or Bayesian, used to be the gold standard in science before the advent of the supercomputer, it was quickly replaced in favor of black box models and sheer computing power because of their ability to handle large data sets. This sadly happened at the expense of interpretability and trustworthiness. However, while people are still trying to improve the predictive power of their models, the community is starting to realize that for many applications it is not so much only the exact prediction that is of importance, but rather the variability or uncertainty. The work in this dissertation tries to further the quest for a world where everyone is aware of uncertainty, of how important it is and how to embrace it instead of fearing it. A specific, though general, framework that allows anyone to obtain accurate uncertainty estimates is singled out and analyzed. Certain aspects and applications of the framework — dubbed `conformal prediction' — are studied in detail. Whereas many uncertainty quantification frameworks make strong assumptions about the data, conformal prediction is at the time of writing the only framework that deserves the title distribution-free.
- Gedurende de afgelopen decennia was het leeuwendeel van het onderzoek binnen de data-analyse en machine learning gefocust op het verbeteren van predictieve modellen en het bekomen van betere voorspellingen dan wat mogelijk was met bestaande modellen. Of de metrieken die hiervoor gebruikt werden representatief waren voor de beoogde doelstellingen, de numerieke verschillen in de bekomen resultaten significant waren, of onzekerheid in beschouwing genomen moest worden, was van beperkt belang. Waar kansrekening, zij het frequentistisch of Bayesiaans, de gouden standaard vormde voor de wetenschap voor de komst van de supercomputer, werd deze snel vervangen door zwarte dozen en pure rekenkracht om de alsmaar groter wordende datasets aan te kunnen. Deze evolutie ging echter ten koste van de interpreteerbaarheid en de betrouwbaarheid. Gelukkig begint men te beseffen dat, ook al blijft het verbeteren van voorspellingen een belangrijke drijfveer, voor veel toepassingen het niet zozeer de exacte voorspelling is die er toe doet, maar eerder de onzekerheid of variabiliteit. Het werk in dit proefschrift probeert iets bij te brengen aan de strijd voor een wereld waarin iedereen zich bewust is van de onzekerheid in data, hoe belangrijk het is om deze correct in te schatten en hoe deze te leren gebruiken in plaats van er bang van te zijn. Een specifiek, maar breed toepasbaar raamwerk voor het bekomen van nauwkeurige onzekerheidsschattingen wordt uitgelicht en geanalyseerd. Enkele aspecten en toepassingen van dit framework -- dat de naam `conform voorspellen' gekregen heeft -- worden in detail bestudeerd. Waar veel andere paradigma's voor het schatten van onzekerheid sterke veronderstellingen over de data maken, is conform voorspellen voorlopig de enige kandidaat die de titel `assumptievrij' verdient. Geen enkele parametrische veronderstelling is vereist en de niet-parametrische resultaten kunnen bekomen worden zonder gebruik te maken van asymptotische stellingen zoals de wet van de grote aantallen. Nadat de algemene theorie van onzekerheid en conform voorspellen geïntroduceerd is, wordt een van de prototypische probleemstellingen uit de statistiek beschouwd, dat van (univariate) regressie. Zelfs in deze doorsneesituatie kunnen reeds enkele interessante eigenschappen bestudeerd worden. Hierop verderbouwend wordt vervolgens het probleem van conditionele onzekerheidsschatting aangepakt. Waar de eerste situatie eerder op een globaal niveau plaatsvindt, waarbij alle aspecten van de data evenwaardig geacht worden, ligt de focus bij het tweede probleem eerder op specifieke deelverzamelingen van de data. In het derde luik van dit werk wordt een mogelijke middenweg onderzocht, waarbij conditionele garanties worden afgewogen tegen het gebruik van zo veel mogelijk beschikbare data. Dit zal ook de ideale kans bieden om extreme classificatieproblemen en problemen met meerdere variabelen te behandelen.
- Keywords
- computational statistics, machine learning, statistics, probability theory, conformal prediction, uncertainty, prediction regions, confidence regions
Downloads
-
BiblioVersion.pdf
- full text (Published version)
- |
- open access
- |
- |
- 14.27 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HVK8CNHPMDFGQ8BGF86VMVTY
- MLA
- Dewolf, Nicolas. A Comparative Study of Conformal Prediction Methods for Valid Uncertainty Quantification in Machine Learning. Ghent University. Faculty of Bioscience Engineering, 2024.
- APA
- Dewolf, N. (2024). A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning. Ghent University. Faculty of Bioscience Engineering, Ghent, Belgium.
- Chicago author-date
- Dewolf, Nicolas. 2024. “A Comparative Study of Conformal Prediction Methods for Valid Uncertainty Quantification in Machine Learning.” Ghent, Belgium: Ghent University. Faculty of Bioscience Engineering.
- Chicago author-date (all authors)
- Dewolf, Nicolas. 2024. “A Comparative Study of Conformal Prediction Methods for Valid Uncertainty Quantification in Machine Learning.” Ghent, Belgium: Ghent University. Faculty of Bioscience Engineering.
- Vancouver
- 1.Dewolf N. A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning. [Ghent, Belgium]: Ghent University. Faculty of Bioscience Engineering; 2024.
- IEEE
- [1]N. Dewolf, “A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning,” Ghent University. Faculty of Bioscience Engineering, Ghent, Belgium, 2024.
@phdthesis{01HVK8CNHPMDFGQ8BGF86VMVTY,
abstract = {{In the past decades, most work in the area of data analysis and machine learning was focused on optimizing predictive models and getting better results than what was possible with existing models. To what extent the metrics with which such improvements were measured were accurately capturing the intended goal, whether the numerical differences in the resulting values were significant, or whether uncertainty played a role in this study and if it should have been taken into account, was of secondary importance. Whereas probability theory, be it frequentist or Bayesian, used to be the gold standard in science before the advent of the supercomputer, it was quickly replaced in favor of black box models and sheer computing power because of their ability to handle large data sets. This sadly happened at the expense of interpretability and trustworthiness. However, while people are still trying to improve the predictive power of their models, the community is starting to realize that for many applications it is not so much only the exact prediction that is of importance, but rather the variability or uncertainty.
The work in this dissertation tries to further the quest for a world where everyone is aware of uncertainty, of how important it is and how to embrace it instead of fearing it. A specific, though general, framework that allows anyone to obtain accurate uncertainty estimates is singled out and analyzed. Certain aspects and applications of the framework — dubbed `conformal prediction' — are studied in detail. Whereas many uncertainty quantification frameworks make strong assumptions about the data, conformal prediction is at the time of writing the only framework that deserves the title distribution-free.}},
author = {{Dewolf, Nicolas}},
isbn = {{9789463577342}},
keywords = {{computational statistics,machine learning,statistics,probability theory,conformal prediction,uncertainty,prediction regions,confidence regions}},
language = {{eng}},
pages = {{XXII, 317}},
publisher = {{Ghent University. Faculty of Bioscience Engineering}},
school = {{Ghent University}},
title = {{A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning}},
year = {{2024}},
}