Advanced search
1 file | 288.84 KB Add to list

Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression

Pieter Buteneers (UGent) , Ken Caluwaerts (UGent) , David Verstraeten (UGent) , Joni Dambre (UGent) and Benjamin Schrauwen (UGent)
(2013) NEURAL PROCESSING LETTERS. 38(3). p.403-416
Author
Organization
Abstract
In this paper we propose mathematical optimizations to select the optimal regularization parameter for ridge regression using cross-validation. The resulting algorithm is suited for large datasets and the computational cost does not depend on the size of the training set. We extend this algorithm to forward or backward feature selection in which the optimal regularization parameter is selected for each possible feature set. These feature selection algorithms yield solutions with a sparse weight matrix using a quadratic cost on the norm of the weights. A naive approach to optimizing the ridge regression parameter has a computational complexity of the order with the number of applied regularization parameters, the number of folds in the validation set, the number of input features and the number of data samples in the training set. Our implementation has a computational complexity of the order . This computational cost is smaller than that of regression without regularization for large datasets and is independent of the number of applied regularization parameters and the size of the training set. Combined with a feature selection algorithm the algorithm is of complexity and for forward and backward feature selection respectively, with the number of selected features and the number of removed features. This is an order faster than and for the naive implementation, with for large datasets. To show the performance and reduction in computational cost, we apply this technique to train recurrent neural networks using the reservoir computing approach, windowed ridge regression, least-squares support vector machines (LS-SVMs) in primal space using the fixed-size LS-SVM approximation and extreme learning machines.
Keywords
VALIDATION, VARIABLE SELECTION, Cross-validation, Feature selection, Ridge regression, Regularization parameter optimization, Computationally efficient, Model selection

Downloads

  • paper.pdf
    • full text
    • |
    • open access
    • |
    • PDF
    • |
    • 288.84 KB

Citation

Please use this url to cite or link to this publication:

MLA
Buteneers, Pieter, et al. “Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression.” NEURAL PROCESSING LETTERS, vol. 38, no. 3, 2013, pp. 403–16, doi:10.1007/s11063-013-9279-8.
APA
Buteneers, P., Caluwaerts, K., Verstraeten, D., Dambre, J., & Schrauwen, B. (2013). Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression. NEURAL PROCESSING LETTERS, 38(3), 403–416. https://doi.org/10.1007/s11063-013-9279-8
Chicago author-date
Buteneers, Pieter, Ken Caluwaerts, David Verstraeten, Joni Dambre, and Benjamin Schrauwen. 2013. “Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression.” NEURAL PROCESSING LETTERS 38 (3): 403–16. https://doi.org/10.1007/s11063-013-9279-8.
Chicago author-date (all authors)
Buteneers, Pieter, Ken Caluwaerts, David Verstraeten, Joni Dambre, and Benjamin Schrauwen. 2013. “Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression.” NEURAL PROCESSING LETTERS 38 (3): 403–416. doi:10.1007/s11063-013-9279-8.
Vancouver
1.
Buteneers P, Caluwaerts K, Verstraeten D, Dambre J, Schrauwen B. Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression. NEURAL PROCESSING LETTERS. 2013;38(3):403–16.
IEEE
[1]
P. Buteneers, K. Caluwaerts, D. Verstraeten, J. Dambre, and B. Schrauwen, “Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression,” NEURAL PROCESSING LETTERS, vol. 38, no. 3, pp. 403–416, 2013.
@article{3102381,
  abstract     = {{In this paper we propose mathematical optimizations to select the optimal regularization parameter for ridge regression using cross-validation. The resulting algorithm is suited for large datasets and the computational cost does not depend on the size of the training set. We extend this algorithm to forward or backward feature selection in which the optimal regularization parameter is selected for each possible feature set. These feature selection algorithms yield solutions with a sparse weight matrix using a quadratic cost on the norm of the weights. A naive approach to optimizing the ridge regression parameter has a computational complexity of the order with the number of applied regularization parameters, the number of folds in the validation set, the number of input features and the number of data samples in the training set. Our implementation has a computational complexity of the order . This computational cost is smaller than that of regression without regularization for large datasets and is independent of the number of applied regularization parameters and the size of the training set. Combined with a feature selection algorithm the algorithm is of complexity and for forward and backward feature selection respectively, with the number of selected features and the number of removed features. This is an order faster than and for the naive implementation, with for large datasets. To show the performance and reduction in computational cost, we apply this technique to train recurrent neural networks using the reservoir computing approach, windowed ridge regression, least-squares support vector machines (LS-SVMs) in primal space using the fixed-size LS-SVM approximation and extreme learning machines.}},
  author       = {{Buteneers, Pieter and Caluwaerts, Ken and Verstraeten, David and Dambre, Joni and Schrauwen, Benjamin}},
  issn         = {{1370-4621}},
  journal      = {{NEURAL PROCESSING LETTERS}},
  keywords     = {{VALIDATION,VARIABLE SELECTION,Cross-validation,Feature selection,Ridge regression,Regularization parameter optimization,Computationally efficient,Model selection}},
  language     = {{eng}},
  number       = {{3}},
  pages        = {{403--416}},
  title        = {{Optimized parameter search for large datasets of the regularization parameter and feature selection for ridge regression}},
  url          = {{http://doi.org/10.1007/s11063-013-9279-8}},
  volume       = {{38}},
  year         = {{2013}},
}

Altmetric
View in Altmetric
Web of Science
Times cited: