Advanced search
1 file | 7.74 MB Add to list

Improved performance measures for video quality assessment algorithms using training and validation sets

(2019) IEEE TRANSACTIONS ON MULTIMEDIA. 21(8). p.2026-2041
Author
Organization
Abstract
The training and performance analysis of objective video quality assessment algorithms is complex due to the huge variety of possible content classes and transmission distortions. Several secondary issues such as free parameters in machine learning algorithms and alignment of subjective datasets put an additional burden on the developer. In this paper, three subsequent steps are presented to address such issues. First, the content and coding parameter space of a large-scale database is used to select dedicated subsets for training objective algorithms. This aims at providing a method for selecting the most significant contents and coding parameters from all imaginable combinations. In the practical case where only a limited set is available, it also helps us to avoid redundancy in the training subset selection. The second step is a discussion on performance measures for algorithms that employ machine-learning methods. The particularity of the performance measures is that the quality of the training and verification datasets is taken into consideration. Common issues that often use existing measures are presented, and improved or complementary methods are proposed. The measures are applied to two examples of no-reference objective assessment algorithms using the aforementioned subsets of the large-scale database. While limited in terms of practical applications, this sandbox approach of objectively predicting an objectively evaluated video sequences allows for eliminating additional influence factors from subjective studies. In the third step, the proposed performance measures are applied to the practical case of training and analyzing assessment algorithms on readily available subjectively annotated image datasets. The presentation method in this part of the paper can also be used as an exemplified recommendation for reporting in-depth information on the performance. Using this presentation method, future publications presenting newly developed quality assessment algorithms may be significantly improved.
Keywords
INDEX, Video quality, no-reference VQA, HRC selection, datasets evaluation, content features

Downloads

  • (...).pdf
    • full text (Published version)
    • |
    • UGent only
    • |
    • PDF
    • |
    • 7.74 MB

Citation

Please use this url to cite or link to this publication:

MLA
Aldahdooh, Ahmed, et al. “Improved Performance Measures for Video Quality Assessment Algorithms Using Training and Validation Sets.” IEEE TRANSACTIONS ON MULTIMEDIA, vol. 21, no. 8, 2019, pp. 2026–41.
APA
Aldahdooh, A., Masala, E., Janssens, O., Van Wallendael, G., Barkowsky, M., & Le Callet, P. (2019). Improved performance measures for video quality assessment algorithms using training and validation sets. IEEE TRANSACTIONS ON MULTIMEDIA, 21(8), 2026–2041.
Chicago author-date
Aldahdooh, Ahmed, Enrico Masala, Olivier Janssens, Glenn Van Wallendael, Marcus Barkowsky, and Patrick Le Callet. 2019. “Improved Performance Measures for Video Quality Assessment Algorithms Using Training and Validation Sets.” IEEE TRANSACTIONS ON MULTIMEDIA 21 (8): 2026–41.
Chicago author-date (all authors)
Aldahdooh, Ahmed, Enrico Masala, Olivier Janssens, Glenn Van Wallendael, Marcus Barkowsky, and Patrick Le Callet. 2019. “Improved Performance Measures for Video Quality Assessment Algorithms Using Training and Validation Sets.” IEEE TRANSACTIONS ON MULTIMEDIA 21 (8): 2026–2041.
Vancouver
1.
Aldahdooh A, Masala E, Janssens O, Van Wallendael G, Barkowsky M, Le Callet P. Improved performance measures for video quality assessment algorithms using training and validation sets. IEEE TRANSACTIONS ON MULTIMEDIA. 2019;21(8):2026–41.
IEEE
[1]
A. Aldahdooh, E. Masala, O. Janssens, G. Van Wallendael, M. Barkowsky, and P. Le Callet, “Improved performance measures for video quality assessment algorithms using training and validation sets,” IEEE TRANSACTIONS ON MULTIMEDIA, vol. 21, no. 8, pp. 2026–2041, 2019.
@article{8641407,
  abstract     = {The training and performance analysis of objective video quality assessment algorithms is complex due to the huge variety of possible content classes and transmission distortions. Several secondary issues such as free parameters in machine learning algorithms and alignment of subjective datasets put an additional burden on the developer. In this paper, three subsequent steps are presented to address such issues. First, the content and coding parameter space of a large-scale database is used to select dedicated subsets for training objective algorithms. This aims at providing a method for selecting the most significant contents and coding parameters from all imaginable combinations. In the practical case where only a limited set is available, it also helps us to avoid redundancy in the training subset selection. The second step is a discussion on performance measures for algorithms that employ machine-learning methods. The particularity of the performance measures is that the quality of the training and verification datasets is taken into consideration. Common issues that often use existing measures are presented, and improved or complementary methods are proposed. The measures are applied to two examples of no-reference objective assessment algorithms using the aforementioned subsets of the large-scale database. While limited in terms of practical applications, this sandbox approach of objectively predicting an objectively evaluated video sequences allows for eliminating additional influence factors from subjective studies. In the third step, the proposed performance measures are applied to the practical case of training and analyzing assessment algorithms on readily available subjectively annotated image datasets. The presentation method in this part of the paper can also be used as an exemplified recommendation for reporting in-depth information on the performance. Using this presentation method, future publications presenting newly developed quality assessment algorithms may be significantly improved.},
  author       = {Aldahdooh, Ahmed and Masala, Enrico and Janssens, Olivier and Van Wallendael, Glenn and Barkowsky, Marcus and Le Callet, Patrick},
  issn         = {1520-9210},
  journal      = {IEEE TRANSACTIONS ON MULTIMEDIA},
  keywords     = {INDEX,Video quality,no-reference VQA,HRC selection,datasets evaluation,content features},
  language     = {eng},
  number       = {8},
  pages        = {2026--2041},
  title        = {Improved performance measures for video quality assessment algorithms using training and validation sets},
  url          = {http://dx.doi.org/10.1109/TMM.2018.2882091},
  volume       = {21},
  year         = {2019},
}

Altmetric
View in Altmetric
Web of Science
Times cited: