Speaker
Description
Ranking objects is one of the most commonly applied computational tasks. In machine learning, query results are ranked by search engines, features are ranked during feature selection, algorithms are ranked by their performance. Often there is a reference through which the solutions are compared to each other. In content recommendation, this can be the user for which we would like to generate suggestions, in image search the queried image itself serves as a reference, in machine learning the test dataset is used as a reference for algorithms refined on the training dataset. Despite the widespread use of rankings, there are still unresolved tasks regarding their statistical treatment. This research investigates how to determine whether two rankings come from the same distribution without assumptions on the data generating process but assuming the knowledge of some reference ranking. For this purpose, we evaluate three hybrid tests: Wilcoxon’s, Dietterich’s, and Alpaydin’s statistical tests combined with cross-validation (CV), each operating with folds ranging from 5 to 10, thus altogether 18 variants. We have applied these tests in the framework of a popular comparative statistical test, the Sum of Ranking Differences which builds upon the Manhattan distance between the rankings. The optimal CV test method depends on the preferences regarding the minimization of errors in type I or II cases, the size of the input, and expected patterns in the data. The Wilcoxon method with eight folds proved to be the best for all three investigated input sizes.
