WorksComputer Science
Unsupervised Feature Selection Using Local Intrinsic Dimensionality – Extended Version
Abstract
High dimensionality degrades both the efficiency and the effectiveness of learning algorithms, and in many applications no class labels are available to guide dimensionality reduction. This paper proposes LIDFS, an unsupervised feature selection method based on Local Intrinsic Dimensionality (LID). The method builds on the formal equivalence between LID and the indiscriminability of distance measures. The data matrix is transposed so that each feature is treated as a point, and the LID of each feature is estimated by maximum likelihood from the distances to its nearest neighboring features. Features with lower LID exhibit greater local discriminability and receive higher importance scores. A refinement step then retains one representative per feature neighborhood and demotes locally redundant features. The result is a complete ranking from which any number of features can be selected without rerunning the procedure. LIDFS does not use any information about labels or the number of classes, and runs in time for instances and features. LIDFS is compared with classic and state-of-the-art methods and a random baseline on 23 high-dimensional datasets, the classic benchmark collection. The evaluation uses classification accuracy and AUC with five downstream classifiers, as well as clustering accuracy and normalized mutual information, over both full and aggressive selection ranges. LIDFS ranks first in the supervised evaluations and first or tied second in the clustering-based evaluations. It belongs to the top statistical group together with methods that require the number of classes, and it runs faster than its closest competitor. Several recent methods, by contrast, fail to outperform the random baseline. A sensitivity analysis shows that small neighborhood sizes yield the best results.
Introduction
Advances in computer technology are opening the way for the handling of increasingly complex data. The scale of data can be measured with respect to volume (the number of data instances) and dimensionality (the number of features that describe these instances). When dealing with colossal data volume on high-capacity computing platforms, volume reduction techniques such as sampling, together with parallelization, can allow the exploration and analysis of data with millions or even billions of data entries.
Feature reduction, the dimensional analogue of volume reduction, has several motivations [1]: reducing the costs of data collection, storage, and processing; improving the model generalizability and interpretability; and improving the discriminative power of feature-based distance measures. In general, high dimensionality is associated with a degradation in the efficiency and effectiveness of fundamental data mining and machine learning tasks such as clustering and anomaly detection [2], an effect often referred to as the “curse of dimensionality”. As the dimensionality rises, distance values between points tend to concentrate around their mean values due to the effects of noise and other sources of error, which drastically diminishes the discriminative power of the distance measure.
Feature reduction methods fall into two main categories: selection and extraction. Feature selection, as the name suggests, retains a subset of the original features, chosen with the aim of improving the performance and efficiency of data mining and machine learning tasks [3]. In feature extraction, the attributes are transformed to a smaller set within an artificial feature space, with each new feature value depending on the values of many (or even all) of the original features. Many feature extraction algorithms are based on Principal Component Analysis (PCA) [4] or Linear Discriminant Analysis [5].
For applications in which extraction methods are appropriate, extracted features are very likely to outperform any equal number of selected features, due to the greater flexibility in determining a data transformation that maximizes the quality criteria used to guide the feature generation process. However, in some contexts where feature semantics are of major importance and where artificial features are meaningless, feature extraction cannot be used. Such applications are common in bioinformatics [6] (gene annotation, micro-array analysis) and chemistry [6, 7] (mass spectra analysis, interpretability of molecular signatures), among others. In such contexts, however, feature selection can still be applied to reduce the dimensionality of the data.
Even when feature extraction methods are applicable, they are generally not suitable for very large datasets, as the transformation of the full dataset to a new full-dimensional basis can incur a prohibitively high computational cost. This computational complexity, together with a lack of robustness in the presence of noise [8], makes feature extraction less useful when the instances outnumber the features. Furthermore, feature extraction has limited use when the goal is to remove irrelevant features, since all of the original features must be retained in order to compute values within the transformed feature space. Feature extraction also reduces the explainability and transparency of the final downstream task as it becomes difficult to interpret the results which rely on a transformed space.
In this paper, we consider the case of unsupervised feature selection, where class label information is unavailable. Unsupervised feature selection is a more difficult problem than supervised feature selection, due to the difficulty in evaluating the quality of features. In the absence of ground truth information, there are two main types of quality criteria that are commonly used to guide the selection process: redundancy elimination, where an attribute is discarded whenever it can be entirely or partially inferred from other attributes, and similarity conservation, in which a feature is selected according to the degree to which it helps to conserve the similarity between data points. Of the two, similarity conservation has received relatively less attention in the research literature.
Depending on their evaluation strategy, feature selection frameworks can be categorized as either filters or wrappers (or embedded). Wrapper methods execute a predictive learning task over the data for many choices of feature subsets, and retain candidate features for which the predictive model achieves high performance [1, 9]. In practical contexts involving data of even moderate volume or dimension, the cost of wrapper methods quickly becomes prohibitively high. A more affordable alternative is that of filter methods, which evaluate feature quality without resorting to an external learning machine. Although filters generally cannot match the quality of wrappers on small-scale datasets, on datasets of larger scale filters generally require far less computation time, and are less susceptible to overfitting. Many unsupervised feature selection algorithms (such as LapAOFS / LapDOFS [10]) require that a target number of features be supplied as an input parameter, while others (such as GLFS [11]) automatically determine the number of selected features. With algorithms of the former type, the user is free to increase the number of features if needed, although the entire feature selection task usually must be reiterated. On the other hand, with most algorithms of the latter type, the user has no control over the number of features selected. When dealing with large datasets, a third and more convenient alternative is to produce a ranked list of all features, so as to allow any downstream processes to select the number of features that are most appropriate to the task at hand, given the amount of computational resources available. The most widely known unsupervised feature ranking framework is that of spectral feature selection [12], which includes Laplacian Score for Feature Selection (LS) [13]. LS attempts to produce a reduced feature set that preserves the original neighborhood information to the greatest possible extent. More recent work on filters for unsupervised feature ranking includes Multi-Cluster Feature Selection (MCFS) [14], which attempts to preserve the cluster structure of the data.
Almost all existing filter methods for feature selection are suitable only for datasets of relatively small volume and dimensionality, such as those found in the context of bioinformatics. In this work, we target the problem of unsupervised feature ranking at higher scales, where the data volume and data dimension together preclude the use of wrapper methods and computationally expensive filters. Such feature selection could be of particular benefit for database and data mining operations such as similarity search, neighborhood-based classification, and clustering.
As a criterion for guiding the selection process, we propose that features be assessed according to their ability to discriminate between the distances encountered in the neighborhoods of data points, in an attempt to alleviate the effects of the curse of dimensionality. Accordingly, we develop a feature ranking method that assesses the discriminative power of individual features according to a model of the local intrinsic dimensionality (LID) of continuous distance distributions [15, 16, 17]. While most feature selection methods employ indirect strategies in order to (implicitly) improve the discriminability of distance measures, the use of LID as guiding criterion allows for a more explicit approach, due to a formal equivalence [16] between the discriminability of distance measures and low intrinsic dimensionality.
Taking into account the performance issues due to larger data scales (in terms of both volume and dimensionality), we develop a forward filter algorithm generally applicable to continuous-valued numerical datasets.
The specific contributions of this paper include:
a greedy unsupervised feature selection algorithm based on LID2;
a theoretical analysis and discussion of the proposed method;
an experimental framework setting random feature selection as a baseline for feature selection, and showing the advantage of using our method with high-dimensional feature selection benchmark datasets.
The paper is organized as follows. In the next section we present a survey of existing methods for feature selection. In § 3, we introduce the LID model upon which our feature ranking method is based, as well as its relationship to the discriminability of distance measures. In § 4, we present the details of the proposed unsupervised feature selection method. The experimental framework is presented in § 5, in which our method is compared against state-of-the-art unsupervised feature selection algorithms. The experimental results are discussed in § 6. Finally, in § 7 we conclude by summarizing the advantages and the limitations of our feature selection strategy, and propose some possible extensions of our algorithm as future work.
Feature selection
Feature selection consists of choosing a subset of the original features so as to best satisfy a quality criterion. The ultimate goal is to improve model generalizability and interpretability, as well as to lower the computational costs of learning algorithms that use the data.
Formally, let us assume that we are given a set of points in a space of dimension represented by the matrix ,
where are points and are features. Feature selection consists of finding , a subset of with , that best satisfies an optimality condition.
Feature selection algorithms can be differentiated in several ways. They can be
wrappers or filters, according to whether or not they employ a learning task as part of their feature evaluation strategy (with embedded methods combining the characteristics of both);
forward or backward depending on their search direction, whether they build a feature set incrementally starting from an empty set, or cull features incrementally starting from the full set;
univariate or multivariate, according to whether they assess features individually or in groups.
Wrapper methods apply a predictive learning machine to the data using the candidate feature subsets; the objective function is defined by the performance of this predictive model on a task for which the feature selection is targeted [9]. Wrappers can achieve high performance with the learning machine used in the selection process. However, they are model-specific [1], as features selected using a given learning model are often inadequate for others. Wrapper methods are also computationally expensive [1]: in a wrapper selection process, assessing the quality of each candidate subset requires one run of the learning machine, and one measurement of the quality of its output. Moreover, wrappers present a risk of overfitting not only to the predictive model, but also to the data. In contrast, filter methods use properties of the data to compare the candidate subsets: they evaluate candidate feature subsets in terms of some quality criterion, often information-theoretic (MRMR [18], Information Gain [19]), correlation-based (Correlation-based Feature Selection [20]), statistical (t-test and chi-squared [21]), or based on interclass distances (Fisher Score [22], supervised Spectral Feature Selection [12]). Although they cannot match the performance of wrappers on the learning tasks used to guide the feature selection process, filters have the advantage of not being specific to any particular learning model. Furthermore, filters generally require far less computation time, and are less susceptible to overfitting. In the context of complex data, the advantages of filters are crucial. Feature selection algorithms can also be categorized according to the order in which candidate feature subsets are considered. With backward elimination, the first feature subset candidate assessed is the full dataset ; new candidate subsets are constructed by incrementally removing features that underperform with respect to some criterion. The opposite approach, forward selection, consists of starting with an empty set, and building up candidate feature subsets through the incremental introduction of individual features. When the number of features is very high, examining candidate subsets of smaller cardinality is more affordable. In the context of high-dimensional data, this argument is largely in favor of forward selection.
Feature selection methods also differ in the way they group features for assessment: univariate methods assess features individually and independently, while multivariate methods consider sets of features during the selection process. Since combinations of features are not considered, the univariate approach cannot account for redundancy of features. However, univariate methods have the general advantage of being faster: since they consider attributes separately, their complexity is usually linear in the number of features. Although the multivariate approach is usually more computationally expensive, in the context of high-dimensional data, multivariate methods that strictly limit the numbers of candidates assessed can sometimes be more affordable than their univariate competitors.
In some cases, a vector of class labels associated with may also be available. Feature selection methods that make use of such label information as a ground truth are said to be ‘supervised’; if label information is not used, the method is said to be ‘unsupervised’. In general, wrapper methods tend to be supervised (due to the need to assess the result on a specific learning task).
In the context of big data, where ground truth label information is rarely available, the complexity argument favors unsupervised feature selection filters which operate in a forward direction. Among all existing candidates that fit these criteria, scalability concerns prevent the use of computationally expensive algorithms such as LapDOFS and LapAOFS [10], both of which require time more than quadratic in the number of points and number of features. Among all algorithms with reasonable scalability characteristics, the unsupervised general-purpose forward-filter Laplacian Score (LS) feature selection method [13] is perhaps the most popular. It belongs to the general framework of spectral feature selection [12], and has a computational complexity of . The spectral framework offers three different approaches to scoring features, with the objective always being to preserve the graph structure of the dataset described by a spectral matrix . Of the three scoring functions available, the second is used by LS:
where
is the variance of the feature ; denotes the Euclidean distance between and ; is 1 when or is among the -nearest neighbors of the other, and 0 otherwise; and represents the variance within a set of distance values, but is not explicitly defined in the original work. Features are ranked in ascending order of .
Feature selection can also be guided according to how well the new features preserve the cluster structure of the data. Given a similarity graph represented as a matrix with , and a target number of clusters , Multi-Cluster Feature Selection (MCFS) [14] computes the eigenvectors with the smallest eigenvalues of the generalized eigenproblem , where is the diagonal matrix , and . Least Angle Regression (LAR) is then used to solve the regression problems
Finally, features are scored by
Features are ranked in descending order of . MCFS has limited use in practice due to its high computational complexity: .
Structured Optimal Graph Feature Selection (SOGFS) [23] jointly learns a probabilistic-neighbor similarity graph and a row-sparse projection matrix, rather than fixing the graph from the raw data beforehand. To keep the graph well-structured, it constrains the Laplacian to have exactly connected components, relaxed via Ky Fan’s theorem into a trace-minimization term,
solved by alternating updates of the projection, , and the graph, at complexity , where is the projection dimensionality.
Local structure Learning and exponentially weighted Sparse Regression Feature Selection (LLSRFS) [24] counters the tendency of sparse mapping matrices to overweight already-high-scoring features by reweighting each feature’s contribution to the pairwise distance by ,
where a smaller exponent flattens the weight distribution across more features. The authors show this is equivalent to -norm sparse regression under , generalizing earlier rescaled-least-squares selectors, at complexity .
Subspace Clustering unsupervised Feature Selection (SCFS) [25] implicitly learns sample similarity through a self-expressive representation rather than an explicit graph,
where the nonnegative doubles as a soft clustering assignment, so encodes pairwise similarity implicitly. This is linked to the features by a sparse regression , optimized jointly with the first objective, at complexity when .
Variance–Covariance Subspace Distance Feature Selection (VCSDFS) [26] selects features by minimizing a proper metric distance between the full feature matrix and a candidate sub-matrix,
which the authors show equals the Frobenius norm of the covariance matrix of the discarded features, so minimizing it discards the least-variance features. This avoids the degeneracy of principal-angle-based distances, which are provably zero between any matrix and its own sub-matrices. The feature weights are learned via a matrix-factorization objective with an inner-product regularizer, at complexity , where is the number of iterations.
Feature selection with Minkowski weighted -means++ (FS-MWK++) [27] selects centroids probabilistically using feature relevance estimates derived from the data, and then aggregates feature weights across a range of Minkowski exponents to identify informative features. A scalable variant of the method, namely SFS-MWK++, is based on subsampling. Robust, Adaptive and Flexible Graph (RAFG) [28] is a graph-learning framework for unsupervised feature selection. The -norm is imposed on the flexible regression term to alleviate the adverse effects of both noisy features and outliers, and an -norm regularization term is incorporated to ensure that the selected transformation matrix is sufficiently sparse. Several other methods proposed in recent years focus on the unsupervised multi-view feature selection problem [29, 30].
Indiscriminability and Local ID
The success of learning algorithms on high-dimensional data is often attributed to a substantial gap between the high representational dimension of the data and its comparatively low intrinsic dimensionality (ID) [16, 17]. Many feature extraction and feature selection methods implicitly exploit this gap in order to improve discriminability. However, intrinsic dimensionality may vary considerably across different regions or clusters of a dataset. Such local variation in ID is seldom exploited explicitly in feature selection.
Expansion Dimension and Local ID
Many models of ID have been proposed over the last few decades: for the theoretical analysis and characterization of datasets, the most popular of these have been topological models, such as the well-known PCA [4]; fractal models [31]; covering models [32]; and expansion models [33, 34]. Most of the measures of ID associated with these models are very expensive to compute, and are thus of limited use in practice. These models can be used ‘globally’ across the entire dataset, or adapted ‘locally’ so as to characterize the local dimensionality in the vicinity of reference points.
Global estimates of ID are typically higher than local estimates, since the number of degrees of freedom required to describe a dataset is (as a rule) higher than that required to describe its components. This gap between global and local ID measurements indicates an opportunity for the latter in guiding feature selection to improve the performance of similarity search. Improvements in the performance of search would have a direct impact on applications of machine learning, data mining, and databases. Among the models of ID mentioned above, expansion models have seen use in the design and analysis of similarity search algorithms [34, 35, 36, 37], as well as outlier detection [38, 39].
The intrinsic dimensionality in a Euclidean vector space can be determined using the ratios of two volume and two distance measurements. Consider the concentric balls and centered at with radii . Let be a (Lebesgue) volume measure. The ratio of volumes leads to a simple closed-form expression for the dimensionality of Euclidean spaces:
An oracle that accepts two concentric balls as a query, and returns the dimension of the space, is not immediately useful. However, if volume is estimated by the number of points within distance from a reference point , then the returned value can serve as a measure of intrinsic dimensionality, the local expansion dimension (ED) [33, 34].
The model of intrinsic dimensionality considered here is a variant of ED, in which the notion of volume is generalized to a probability measure, and distances are modeled as a random variable [16, 17]. Let be a reference point, and be a random variable expressing a distance measurement from . Let be the cumulative distribution function associated with . For any such that and is continuously differentiable in an interval containing , the Local Intrinsic Dimensionality (LID) of at distance is defined, wherever the limit exists, as
Indiscriminability
A distance measure is ‘discriminative’ when an expansion in distance results in a relatively small increase in the number of observations. Discriminability of points improves the efficiency of data mining and machine learning tasks, minimizes error and ensures robustness. Capturing a huge number of points by a small increase in distance measure greatly increases the computational cost of applications, and has a negative impact on effectiveness. Moreover, the relative ranking of points with reference to an indiscriminative distance measure can easily be affected by noise. When selecting features, it is strongly desirable that they be chosen so as to ensure better discriminability.
Formally, let be a reference point and let be a random distance variable with respect to that reference point, as defined in § 3.1. For any distance from such that , the indiscriminability of at is given by the following limit wherever it exists [16, 17]:
Note that this definition of indiscriminability is unitless, does not require knowledge of the statistical parameters of the underlying distance distribution, and coincides precisely with local ID (Eq. 10).
Estimation of Local Intrinsic Dimensionality
As the number of data elements increases, the distance from a query to its -th nearest neighbor tends to decrease. Asymptotically speaking, in the context of continuous distance distributions, the radius of the local vicinity within which a query result is determined tends to zero. Accordingly, the discriminability of distances to the query should be assessed as the limit of as tends to zero, where is the Cumulative Distribution Function (CDF) of the random distance variable , representing distances from the query point to other data objects. As it turns out, this limit can be estimated using Extreme Value Theory (EVT), which states that the tail of the distribution of the random distance variable can be modeled as a Weibull distribution. The Weibull distribution is specified by two parameters: the scale and the shape . For a sufficiently small neighborhood distance threshold, the random distance variable representing the threshold excesses is Weibull-distributed [15].
Concretely, the distances to the -nearest neighbors of a reference point are instances of the random variable . These instances can be used to estimate the Weibull shape parameter . A maximum likelihood estimator (MLE) can be used to estimate and therefore [15]:
Figure 1 illustrates the model on three planar point sets whose distance distributions from the reference point satisfy exactly, for . Starting from neighborhoods of equal size, an expansion of the radius by a common factor admits comparatively few additional points when the local ID is low, and a great many when it is high; the corresponding expansion dimension recovers in each case, as does the MLE. The rightmost distance measure is thus the least discriminative, even though every case is embedded in the plane.
Figure 1. LID as neighborhood growth. Top: three point sets sampled so that holds exactly for , which is possible in the plane since LID depends on the distance distribution alone. Each disc starts from a neighborhood of the same size, , and expands the radius by the same factor : the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension. (a) On log-log axes, the slope of is the dimension. (b) The ratio across distances, with the MLE from neighbors.
Method description
For a given reference point, LID quantifies the local rate of neighborhood growth, or equivalently, the difficulty of discriminating among nearby objects, with lower LID indicating greater local discriminability. Because higher local discriminability benefits distance-based learning, our selection strategy applies this principle in the transposed feature space, treating each feature as a point and evaluating it by the discriminability of its distances to neighboring features. A feature’s overall quality is then given by the inverse of the LID estimate computed from its -nearest-neighbor distances.
We detail our algorithm in the following; a sketch is provided in Alg. 1. Let denote the input data matrix, where is the number of samples, is the number of features, and each column vector contains the values of the -th feature across all samples. Since the goal is to evaluate relationships among features rather than among samples, LID-based feature selection first transposes the data matrix, yielding . In this representation, each row corresponds to a feature described by its values across the dataset. The algorithm can therefore construct neighborhoods in the transposed space, such that features with similar value profiles across the samples are considered close to one another.
Given a user-defined neighborhood size and a chosen distance measure, the algorithm computes the nearest neighboring features of each feature . The resulting neighbor distances are stored in a matrix , where denotes the distance from to its -th nearest neighbor. The corresponding neighbor indices are stored in a matrix , where identifies the index of the -th nearest neighbor of . The distances in the -th row of are then used to estimate the Local Intrinsic Dimensionality of , denoted by . Before the neighbor search, duplicate features are removed, since they would result in zero neighbor distances. Moreover, the LID estimate of a feature is undefined if all of its neighbor distances are equal; in this case, is increased until every feature has at least two distinct neighbor distances.
A lower LID estimate indicates a lower relative rate of neighborhood growth and, according to the connection between LID and indiscriminability, greater local discriminability among neighboring features. In the transposed space, this suggests that the feature has a comparatively distinctive value profile across the samples and is less locally redundant with nearby features. Such a feature may therefore have greater potential to preserve distinctions among the points of , although this implication is heuristic rather than guaranteed by the LID estimate alone. The initial importance score of each feature is therefore defined as the inverse of its estimated LID:
Accordingly, features with lower estimated LID values receive higher initial importance scores.
After computing the initial scores, the features are sorted in descending order of importance. The resulting ordered list is denoted by
where is the feature at rank , and for all . The LID-based feature selection algorithm then performs a refinement step to reduce redundancy among highly ranked features. Two sets are initialized: the selected-feature set , which stores representative features, and the reserve set , which stores features considered locally redundant with respect to previously selected features.
The algorithm scans the ranked list from the highest-scoring feature to the lowest-scoring feature. For each candidate feature , it determines whether belongs to the -neighborhood of any feature already included in . If it does not belong to the neighborhood of any selected feature, it is added to and treated as non-redundant under the neighborhood criterion. Otherwise, it is added to , since it is locally similar to at least one previously selected feature.
Finally, the scores of the features stored in the reserve set are adjusted. If , let
@page 8/120 starts_mid=0 ends_mid=1
Figure 1 (PDF p. 8). LID as neighborhood growth. Top: three point sets sampled so that holds exactly for , which is possible in the plane since LID depends on the distance distribution alone. Each disc starts from a neighborhood of the same size, , and expands the radius by the same factor : the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension. (a) On log-log axes, the slope of is the dimension. (b) The ratio across distances, with the MLE from neighbors.
importance score of each feature is therefore defined as the inverse of its estimated LID:
Accordingly, features with lower estimated LID values receive higher initial importance scores.
After computing the initial scores, the features are sorted in descending order of importance. The resulting ordered list is denoted by
where is the feature at rank , and for all . The LID-based feature selection algorithm then performs a refinement step to reduce redundancy among highly ranked features. Two sets are initialized: the selected-feature set , which stores representative features, and the reserve set , which stores features considered locally redundant with respect to previously selected features.
The algorithm scans the ranked list from the highest-scoring feature to the lowest-scoring feature. For each candidate feature , it determines whether belongs to the -neighborhood of any feature already included in . If it does not belong to the neighborhood of any selected feature, it is added to and treated as non-redundant under the neighborhood criterion. Otherwise, it is added to , since it is locally similar to at least one previously selected feature.
Finally, the scores of the features stored in the reserve set are adjusted. If , let
Algorithm 1 LID Unsupervised Feature Selection
Input: Dataset and feature-neighborhood size .
Output: Feature-importance score vector .
Remove duplicate rows of .
For each , compute the nearest neighboring features of in , excluding itself. Store the neighbor distances in , where is the distance from to its -th nearest neighbor, and store the corresponding feature indices in , where is the index of the -th nearest neighbor of .
while do
Extend and with the -th nearest neighbor of each feature.
end while
Estimate the LID of each feature and compute its initial score: , .
Let be a permutation of such that for all .
Initialize the selected-index set and the reserve-index set .
for to do
for do
if then
break
end if
end for
if then
else
end if
end for
if then
for do
end for
end if
return .
denote the largest initial importance score among the reserved features. For each feature , the score is updated according to
Since the initial scores are positive and , this transformation assigns a score in the interval to every reserved feature while preserving their relative ordering. In particular, the highest-scoring feature in receives a refined score of 0, whereas the remaining reserved features receive negative scores. Features in the selected set retain their positive initial scores and therefore precede all features in in the final ranking.
As a result of this adjustment, a complete ranking is produced in which locally non-redundant features are prioritized, while the relative ordering of the reserved features is retained. Consequently, a user may select any desired number of top-ranked features without rerunning the feature-selection procedure. If , no score adjustment is required. The final output of LIDFS is the feature-importance vector
which can be used to rank the features for unsupervised feature selection.
LID-based feature-importance scores quantify the local discriminability of features in the transposed feature space. A feature with a lower estimated LID has a more distinctive value profile relative to its neighboring features and is therefore treated as a potentially informative representative feature. The refinement step further prioritizes features from different neighborhoods, reducing the selection of locally redundant features. The neighborhood size controls the locality of both the LID estimates and the redundancy criterion: smaller values of emphasize fine-scale neighborhood structure but yield less stable estimates, whereas larger values yield smoother estimates but may merge distinct local structures. The choice of thus involves a trade-off between locality and estimation stability.
Computational complexity
The time complexity of LIDFS is analyzed for a dataset with samples and features, assuming a distance measure that is computed in time for a pair of features, such as the Euclidean distance. Since the removal of duplicate features can only reduce the number of features, also bounds the number of features that remain after this step. Transposing the data matrix requires time, and duplicate features are removed in expected time by hashing the rows of . The dominant cost is the -nearest-neighbor search in the transposed space. An exact search computes the distances between all pairs of features in time, after which the nearest neighbors of each feature are obtained by partial selection in time and sorted in time, resulting in time in total. The loop that increases is only entered if all neighbor distances of a feature are equal, in which case the neighbor lists are extended from the already computed distances; even if grows to , this requires at most time for sorting all rows of the distance matrix. The maximum-likelihood estimate of LID (Eq. (12)) requires a single pass over the neighbor distances of a feature, and therefore the initial scores of all features are computed in time. Sorting the features by their scores requires time. In the refinement step, the membership test is implemented with a Boolean array that marks the neighbors of all selected features, such that each candidate is checked in time and each insertion into requires time, resulting in time for the entire scan.3 Finally, the adjustment of the scores of the reserved features requires time. Consequently, the overall time complexity of LIDFS is
where the simplification holds for , which is satisfied by any practical dataset. The running time of LIDFS is therefore quadratic in the number of features and linear in the number of samples, while the neighborhood size only affects lower-order terms. The space complexity is if the full distance matrix is stored, which reduces to if the distances are computed in blocks and only the nearest neighbors of each feature are retained. For datasets with a very large number of features, the exact neighbor search can be replaced by an approximate nearest-neighbor search to alleviate the quadratic dependence on .
Experimental framework
In this section, we describe the experimental setup used to compare the feature selection performance of LID-based Feature Selection (LIDFS) with various unsupervised feature selection methods. Competitors include traditional methods (Variance-based [40], Correlation-based [41, 42], Laplacian Score [13], and MCFS [14]) as well as recent methods (SCFS [25], SOGFS [23], LLSRFS [24], and VCSDFS [26]).
Evaluation Metrics
To evaluate the feature selection methods across both supervised and unsupervised downstream tasks, four key metrics are employed: Accuracy (ACC), Area Under the Curve (AUC), Clustering Accuracy (CLSACC), and Normalized Mutual Information (NMI).
Accuracy (ACC) Accuracy is a fundamental metric for supervised classification tasks, representing the proportion of correct predictions. For a set of instances with true labels and predicted labels , it is calculated as:
where is the indicator function.
Area Under the Curve (AUC) AUC measures the ability of a classifier to distinguish between classes. It is the area under the Receiver Operating Characteristic (ROC) curve, which plots the True Positive Rate (Sensitivity) against the False Positive Rate (1 - Specificity) at various threshold settings. AUC ranges from 0 to 1, with 0.5 indicating random performance and a higher value indicating better performance. For a given threshold, with TP, FP, TN, and FN denoting the numbers of true positives, false positives, true negatives, and false negatives, the True Positive Rate (TPR) and False Positive Rate (FPR) are defined as:
Clustering Accuracy (CLSACC) Clustering Accuracy assesses how well the clusters found by the algorithm match the true labels of the data. It requires finding the best one-to-one mapping (permutation ) between the cluster assignments () and the true labels ().
where is the number of data points. The optimal mapping is typically found using the Hungarian algorithm [43].
Normalized Mutual Information (NMI) NMI quantifies the degree of dependence between the clustering result () and the true labels (). It is a normalization of the Mutual Information (MI), making it comparable across different cluster numbers. NMI is calculated as:
where is the Mutual Information between and , and and are the entropies of and , respectively. The Mutual Information is given by:
Feature Selection Dynamic Evaluation Metric (FSDEM) In order to evaluate the overall process of feature selection, we use FSDEM combined with the aforementioned metrics. FSDEM is calculated as:
where indicates the approximated function based on different observations of an arbitrary evaluation metric, and and indicate the specific range of values (percentage of selected features) in which the performance is evaluated.
Datasets
We conduct experiments on 23 high-dimensional datasets from the scikit-feature repository [44]. Table 1 presents an overview of the characteristics of the datasets. The datasets vary in the number of instances, features, and domains. This repository is the standard source for feature selection benchmarks.
| ID | Dataset | Instances | Features |
| D1 | ALLAML | 72 | 7,129 |
| D2 | arcene | 200 | 10,000 |
| D3 | Carcinom | 174 | 9,182 |
| D4 | CLL_SUB_111 | 111 | 11,340 |
| D5 | COIL20 | 1,440 | 1,024 |
| D6 | colon | 62 | 2,000 |
| D7 | gisette | 7,000 | 5,000 |
| D8 | GLI_85 | 85 | 22,283 |
| D9 | GLIOMA | 50 | 4,434 |
| D10 | Isolet | 1,560 | 617 |
| D11 | leukemia | 72 | 7,070 |
| D12 | lung | 203 | 3,312 |
| D13 | lung_discrete | 73 | 325 |
| D14 | madelon | 2,600 | 500 |
| D15 | ORL | 400 | 1,024 |
| D16 | orlraws10P | 100 | 10,304 |
| D17 | pixraw10P | 100 | 10,000 |
| D18 | Prostate_GE | 102 | 5,966 |
| D19 | SMK_CAN_187 | 187 | 19,993 |
| D20 | TOX_171 | 171 | 5,748 |
| D21 | warpAR10P | 130 | 2,400 |
| D22 | warpPIE10P | 210 | 2,420 |
| D23 | Yale | 165 | 1,024 |
Table 1. Characteristics of the high-dimensional benchmark datasets.
Evaluation Configuration
We perform our experiments using the FSEVAL benchmarking suite [45], a specialized framework designed for the extensive and comprehensive evaluation of feature selection algorithms. The supervised evaluation is conducted with five downstream classifiers, namely KNN [46], Decision Tree [47], SVM [48], Random Forest [49], and XGBoost [50] with a 5-fold stratified cross-validation to provide a robust evaluation. For brevity, we present the detailed results for the Random Forest classifier in § 6. The results for the other classifiers (reported in the Appendix) lead to the same conclusions. Multiclass AUC is macro-averaged. For CLSACC and NMI, we use the average of 10 runs of the -means algorithm. -means is always given the true number of classes. For the random baseline, the evaluation is repeated 100 times to clearly reflect the standard deviation. The average value is used for further analysis. For LIDFS, we use as a default setting to calculate LID values for each feature. We favor smaller values for , as fine-grained neighborhoods are deemed more important than the approximation accuracy of the LID. We use Euclidean distance as the chosen distance measure and we also include Random feature selection [51] as a baseline for our evaluation. Runtime tests ran on Ubuntu with an AMD Ryzen 9 9950X (128 GB RAM).
We conduct two different experiments to evaluate the performance of the feature selection process. First, we evaluate the overall feature selection process by selecting from 5% to 100% of the features with a step size of 5%. As the datasets are very high-dimensional, we also conduct a different experiment by selecting from 0.5% to 10% of features with a step size of 0.5%. This allows us to have a more realistic view of the performance of feature selection algorithms to make sure the high number of selected features relative to the number of instances does not obscure the results.
Results
In this section, we present the experimental results based on the described evaluation framework. We aim to evaluate different aspects of unsupervised feature selection algorithms, including efficiency and performance, to comprehensively assess the characteristics of LIDFS against the competitors.
Runtime Analysis
Our first experiment evaluates the runtime of the unsupervised feature selection algorithms. Demonstrating computational efficiency is essential to understanding the practical significance of the proposed method’s performance. This experiment considers two variables: the number of instances and the number of features. We report two settings, in each of which one variable is fixed at 100 while the other varies from 1000 to 20000 in steps of 500. Runtime is capped at one hour per method; the first time a method exceeds this cap, the elapsed time is recorded and no further settings are evaluated for that method. Figure 2 summarizes the results, with the -axis on a logarithmic scale for readability.
Figure 2. Runtime analysis of the feature selection methods. The -axis is presented in logarithmic scale.
As expected, the random baseline is by far the most efficient method, as it involves no significant computation. It is followed by traditional methods such as variance-based and correlation-based feature selection. Recent state-of-the-art methods are computationally more expensive. LIDFS shows competitive results in the experiment with varying numbers of features and highly efficient performance when the number of instances is varied. The only recent method that consistently achieves better efficiency is VCSDFS, which outperforms LIDFS by a small margin. SOGFS and LLSRFS are the most expensive methods, and their runtime is capped at an early stage. As we aim to provide consistent and comprehensive experiments on large, high-dimensional datasets, we omit SOGFS and LLSRFS from the remainder of the experiments.
Performance Analysis
We analyze the performance of the proposed method in comparison with traditional and state-of-the-art unsupervised feature selection algorithms based on the 23 benchmark datasets and the four different evaluation metrics.
Figure 3 presents an example of the four evaluation metrics on the Isolet dataset for the full-range experiment. LIDFS achieves the top performance while also maintaining the efficiency demonstrated earlier, and it generally outperforms the random baseline. Many feature selection methods, including recent state-of-the-art algorithms, fail to outperform the random baseline. This might seem surprising, but is in fact consistent with previous observations [51]. The same pattern holds for the second case, in which the range of the number of selected features is limited to 10%, as presented in Figure 4.
Figure 3. Comparison of the feature selection performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the full range of the number of selected features.
Figure 4. Comparison of the performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the first 10% of the features.
An interesting observation in Figure 3 for the full-range experiment is that, for the top-performing methods, the performance curve flattens after roughly the top 15% of features (as ranked by each method) have been selected. Naïve, low-performing methods such as correlation-based feature selection continue to improve up to around the 80% selection mark, or beyond. This is consistent with our assumption in LIDFS: within each feature neighborhood, one feature is designated as the representative and the rest are moved to the reserve set, reflecting their possible redundancy. The results suggest that, with an appropriate choice of for the neighborhood search, LIDFS identifies this redundancy, assigning positive importance values only to features whose selection is expected to increase the informational value of the lower-dimensional representation based on LIDFS’s perspective. Figure 4 confirms this, as the performance curves remain almost monotonically increasing throughout the 10% selection experiment.
Feature selection performance varies across datasets; we selected one exhibiting a commonly observed pattern for illustration. Individual results for all datasets are available in the Appendix. The FSDEM score, which shows the overall performance over a range, is presented for both experiments in Figure 5 and Figure 7. We observe that LIDFS constitutes the frontier almost consistently, providing a reliable and efficient solution for unsupervised feature selection. To analyze the overall performance across all benchmark datasets, we use critical difference diagrams, which rank methods by performance from right to left, following the methodology of Demšar [52].
Figure 5. Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the full (100%) selection scenarios. LIDFS constitutes the frontier almost consistently.
Figure 7. Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the aggressive (10%) selection scenarios. LIDFS constitutes the frontier almost consistently.
The critical difference diagrams for the different metrics, over the full range of feature numbers and the 10% range, are shown in Figure 6 and Figure 8, respectively. This analysis uses the FSDEM score [3], computed as the average performance over the range for each of the four metrics.
Figure 6. Critical difference diagram over the full range of number of features for the four different metrics based on the average performance measured by the FSDEM score.
Figure 8. Critical difference diagram over the first 10% of number of features for the four different metrics based on the average performance measured by the FSDEM score.
LIDFS is in most cases the highest-performing method for unsupervised feature selection. For supervised downstream tasks, LIDFS, SCFS [25], and MCFS [14] form the top statistical group, with LIDFS ranked first and a clear margin over the remaining methods. Other state-of-the-art methods, such as VCSDFS [26], perform even worse than the random baseline. For the unsupervised evaluation, LIDFS is either the best or the second-best (in a tie), without a statistically significant difference or a large margin relative to the best method. On the efficiency frontier, LIDFS shows a lower computational complexity and runtime compared to the closest competitor, SCFS [25]. It is noteworthy that LIDFS works in a truly unsupervised setting, while some of the competitors, including SCFS [25] and MCFS [14], use information extracted from the labels, such as the number of unique classes. The results for the other classifiers are almost identical to those for the Random Forest classifier and lead to the same conclusions.
Sensitivity Analysis
The neighborhood size used to estimate the LID of each point is the central hyperparameter of the proposed feature selection procedure. As discussed earlier, theoretically, larger values of are expected to produce more accurate LID estimates, since the estimator averages over a larger sample. At the same time, a larger enlarges the neighborhood over which LID is computed possibly beyond local characteristics, reducing the granularity of the discrimination between features. These two effects act in opposite directions, so the net effect of on downstream feature selection quality is not obvious a priori and must be assessed empirically. We evaluate on all 23 benchmark datasets under the two feature-selection scenarios, retaining up to 10% of the original features, and full selection up to 100%.
Downstream performance Figure 9 summarizes different evaluation metrics as a function of , averaged over datasets and over the feature-selection fraction within each scenario. This figure shows the overall trend of different performance metrics when increases, which is generally decreasing. This illustrates that the granularity of neighborhoods in the feature selection process is more important than the smoothness of the LID estimation.
Figure 9. Mean AUC, ACC, NMI, and CLSACC as a function of the neighborhood size , averaged over datasets and over the feature-selection fraction, for the aggressive (10%) and full (100%) selection scenarios. Error bars show standard error of the mean over the 23 datasets. For all metrics higher values are better.
Per-dataset heterogeneity and practical guidance The FSDEM score based on each metric on different datasets is presented, for the full selection and aggressive selection scenarios, in Figure 10 and Figure 11, respectively. These figures highlight that smaller values of , which yield finer neighborhoods, generally outperform larger values, which yield more stable LID estimates. However, it is evident that no single value of is optimal on every dataset. The rank heatmap in Figure 12 shows that the preference for small is strongest for AUC and ACC, while NMI and CLSACC are comparatively indifferent. In the full (100%) selection scenario, the best FSDEM score averaged over the four metrics is attained at on 7 of the 23 datasets, at on 6, at on 1, at on 5, and at on 4. Small-to-moderate neighborhoods ( or ) are therefore individually optimal on 13 of 23 datasets (57%), consistent with the aggregate trend, but the remaining 43% of datasets favor , so dataset-specific tuning of remains worthwhile when computational budget allows. In the absence of tuning, our results support using a small ( as used and discussed in the main experimental results and discussions) rather than a large neighborhood.
Figure 10. The performance of LIDFS with different choices of on different datasets and metrics for the full (100%) selection scenario. Smaller values of show a better overall performance.
Figure 11. The performance of LIDFS with different choices of on different datasets and metrics for the aggressive (10%) selection scenario. Smaller values of show a better overall performance.
Figure 12. Average ranks across the metrics for both scenarios. The preference for small neighborhoods is strongest for AUC and ACC; NMI and CLSACC are comparatively flat.
Summary The sensitivity analysis was conducted with all classifiers and shows the same trend as for the Random Forest classifier, where small values of , particularly and , perform comparably, and larger values lead to a slight degradation in performance. Taken together, the results confirm the theoretical trade-off motivating this analysis: larger is expected to improve the fidelity of the LID estimate itself, yet this improvement does not transfer into better downstream performance; on average the coarser local granularity mildly degrades the four metrics we track. A small-to-moderate neighborhood size is therefore the recommended default, with per-dataset tuning of available as an option when the observed heterogeneity across datasets justifies the additional cost.
Conclusion
We proposed a novel unsupervised feature selection algorithm based on LID to compute feature importance values and select informative features without requiring labeled data. We theoretically motivated LID and its suitability for the unsupervised feature selection task, and provided a brief but comprehensive review of progress in the field. We selected a large number of datasets to comprehensively and thoroughly evaluate the proposed method against classic and state-of-the-art algorithms, including a random baseline to further guide the evaluation. Finally, we conducted an in-depth runtime analysis to compare the computational efficiency of the different unsupervised feature selection algorithms considered, and a sensitivity analysis of the neighborhood size .
The experimental results show that the proposed unsupervised feature selection method, LIDFS, offers the best overall observed performance against all competitors, outperforming competitors in most cases and remaining highly competitive in the rest. The superiority of LIDFS, while not statistically significant, is remarkable in particular as LIDFS shows a better computational complexity in comparison with the closest competitors (e.g., SCFS). These observations highlight the effectiveness and efficiency of LIDFS in conducting a truly unsupervised feature selection with high performance figures and low computational complexity, without using limited or partial information from the labels like some competitors (e.g., SCFS, MCFS) do.
Discussion
LIDFS differs from most other methods in the literature in the sense that it uses neither information-theoretic nor spectral measures to select features. Instead, it directly assesses the discriminability of features via the estimation of LID. One major advantage of LID is that, aside from continuity, it requires no knowledge or assumptions concerning the distribution of the data. The main drawback is that LIDFS requires a relatively smooth distribution of distance values, which is more likely to occur when the data are dense, high-dimensional, or both. Nevertheless, such settings are those in which classical feature selection methods often fail, either due to high computational complexity or unstable performance, sometimes falling even below the random baseline.
The underlying logic of LIDFS is to assign greater importance, via an inverse relationship with the LID estimate, to features that reflect a lower-dimensional local structure relative to others, and therefore exhibit higher discriminability. It also includes a pruning step that prevents the selection of too many mutually close features, maintaining low redundancy in the selection process. As shown in the experimental results, this discrimination between features provides a systematic approach to identifying redundancy, an ability tied to the neighborhood size used when computing LID. This orientation toward low-dimensional local structure is reflected in LIDFS’s strong performance relative to the other algorithms in the experiments.
Future work
As data volume increases, the cost of building an exact index becomes prohibitive, motivating the use of approximate nearest-neighbor algorithms such as -NNS [2] or NN-Descent [53] in place of exact search. Using approximate nearest-neighbor distances may adversely affect LID estimation and, consequently, alter the feature ranking; however, trading off feature quality for computation time can be advantageous when the dataset is large or high-dimensional.
One possible extension of this approach is to consider other models of intrinsic dimensionality as an alternative to LID for assessing feature quality. Although such models may lack the theoretical guarantees regarding the discriminability of distance measures, or may have higher computational complexity or greater sensitivity to noise [15], they may still merit consideration, particularly if they prove more reliable for non-smooth distance distributions or for smaller data samples.
There are several other possible directions for future work on ID-based feature selection. First, it would be interesting to consider the problem of unsupervised tuning of the parameters of ID-based feature selection. Also, our approach to unsupervised feature ranking could be extended to supervised applications by restricting the ID estimation to training examples with common labels. Finally, the algorithmic design of LIDFS could be revisited to yield a more granular feature-importance calculation, either through a refinement step that avoids assigning similar importance values within the same neighborhood, or through other approaches independent of . This, however, must be done with careful consideration to maintain the low computational overhead of the current unsupervised feature selection process.
Acknowledgments
This study was funded by the Innovation Fund Denmark project “PREPARE: Personalized Risk Estimation and Prevention of Cardiovascular Disease”.
Supplementary results
This appendix presents the results that are not included in the main text. It consists of two parts. Section A.1 presents the critical difference diagrams of the supervised metrics (ACC and AUC) for the KNN, decision tree, SVM, and XGBoost classifiers, and Section A.2 presents the performance plots on all 23 benchmark datasets. All supervised results are reported for each of the five downstream classifiers separately, whereas the unsupervised metrics, namely clustering accuracy (CLSACC) and NMI, do not depend on the classifier and are therefore reported once. The results for the random forest classifier and for the unsupervised metrics are presented in the main text, and they are not repeated here.
Critical difference diagrams
The critical difference diagrams are computed in the same way as in the main text, i.e., over all 23 benchmark datasets based on the average performance measured by the FSDEM score, using Wilcoxon-Holm with a significance level of 0.2. The diagrams of the random forest classifier and of the unsupervised metrics for the first 10% and for the entire range of the number of selected features are presented in Figure 8 and Figure 6 of the main text, respectively. Each figure below contains the diagrams of ACC (left) and AUC (right) for one classifier and one range, and Table A1 provides an index of all figures, where each entry links to the corresponding figure.
| Classifier / metrics | 0.5% to 10% | 5% to 100% |
| KNN (ACC, AUC) | A1 | A2 |
| Decision tree (ACC, AUC) | A3 | A4 |
| SVM (ACC, AUC) | A5 | A6 |
| Random forest (ACC, AUC) | 8{sup}†` | 6{sup}†` |
| XGBoost (ACC, AUC) | A7 | A8 |
| CLSACC, NMI (classifier independent) | 8{sup}†` | 6{sup}†` |
Table A1. Index of the supplementary critical difference diagrams. Each entry gives the number of the figure with the diagrams of ACC and AUC for the corresponding classifier and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.
Figure A1. Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (0.5% to 10% of the features).
↑ Back to Table A1
Figure A2. Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (5% to 100% of the features).
↑ Back to Table A1
Figure A3. Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (0.5% to 10% of the features).
↑ Back to Table A1
Figure A4. Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (5% to 100% of the features).
↑ Back to Table A1
Figure A5. Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (0.5% to 10% of the features).
↑ Back to Table A1
Figure A6. Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (5% to 100% of the features).
↑ Back to Table A1
Figure A7. Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (0.5% to 10% of the features).
↑ Back to Table A1
Figure A8. Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (5% to 100% of the features).
↑ Back to Table A1
Performance plots
This part presents the performance plots on all 23 benchmark datasets and for both experimental settings, in which the number of selected features ranges from 0.5% to 10% and from 5% to 100% of the features, respectively. The unsupervised metrics (CLSACC and NMI) are reported once for each dataset, whereas the supervised metrics (ACC and AUC) are reported separately for each of the five classifiers, i.e., KNN, decision tree, SVM, random forest, and XGBoost. In all plots, the random baseline is the average of 100 independent runs, and the shaded area denotes one standard deviation. The results on the Isolet dataset for the random forest classifier and for the unsupervised metrics are presented in Figure 4 and Figure 3 of the main text for the first 10% and for the entire range of the number of selected features, respectively, and are not repeated here. Table A2 provides an index of all plots, where each entry links to the corresponding figure.
| Dataset | CLSACC, NMI 10% | CLSACC, NMI 100% | KNN 10% | KNN 100% | Decision tree 10% | Decision tree 100% | SVM 10% | SVM 100% | Random forest 10% | Random forest 100% | XGBoost 10% | XGBoost 100% |
| ALLAML | A9 | A31 | A53 | A76 | A99 | A122 | A145 | A168 | A191 | A213 | A235 | A258 |
| arcene | A10 | A32 | A54 | A77 | A100 | A123 | A146 | A169 | A192 | A214 | A236 | A259 |
| Carcinom | A11 | A33 | A55 | A78 | A101 | A124 | A147 | A170 | A193 | A215 | A237 | A260 |
| CLL_SUB_111 | A12 | A34 | A56 | A79 | A102 | A125 | A148 | A171 | A194 | A216 | A238 | A261 |
| COIL20 | A13 | A35 | A57 | A80 | A103 | A126 | A149 | A172 | A195 | A217 | A239 | A262 |
| colon | A14 | A36 | A58 | A81 | A104 | A127 | A150 | A173 | A196 | A218 | A240 | A263 |
| gisette | A15 | A37 | A59 | A82 | A105 | A128 | A151 | A174 | A197 | A219 | A241 | A264 |
| GLI_85 | A16 | A38 | A60 | A83 | A106 | A129 | A152 | A175 | A198 | A220 | A242 | A265 |
| GLIOMA | A17 | A39 | A61 | A84 | A107 | A130 | A153 | A176 | A199 | A221 | A243 | A266 |
| Isolet | A62 | A85 | A108 | A131 | A154 | A177 | A244 | A267 | ||||
| leukemia | A18 | A40 | A63 | A86 | A109 | A132 | A155 | A178 | A200 | A222 | A245 | A268 |
| lung | A19 | A41 | A64 | A87 | A110 | A133 | A156 | A179 | A201 | A223 | A246 | A269 |
| lung_discrete | A20 | A42 | A65 | A88 | A111 | A134 | A157 | A180 | A202 | A224 | A247 | A270 |
| madelon | A21 | A43 | A66 | A89 | A112 | A135 | A158 | A181 | A203 | A225 | A248 | A271 |
| ORL | A22 | A44 | A67 | A90 | A113 | A136 | A159 | A182 | A204 | A226 | A249 | A272 |
| orlraws10P | A23 | A45 | A68 | A91 | A114 | A137 | A160 | A183 | A205 | A227 | A250 | A273 |
| pixraw10P | A24 | A46 | A69 | A92 | A115 | A138 | A161 | A184 | A206 | A228 | A251 | A274 |
| Prostate_GE | A25 | A47 | A70 | A93 | A116 | A139 | A162 | A185 | A207 | A229 | A252 | A275 |
| SMK_CAN_187 | A26 | A48 | A71 | A94 | A117 | A140 | A163 | A186 | A208 | A230 | A253 | A276 |
| TOX_171 | A27 | A49 | A72 | A95 | A118 | A141 | A164 | A187 | A209 | A231 | A254 | A277 |
| warpAR10P | A28 | A50 | A73 | A96 | A119 | A142 | A165 | A188 | A210 | A232 | A255 | A278 |
| warpPIE10P | A29 | A51 | A74 | A97 | A120 | A143 | A166 | A189 | A211 | A233 | A256 | A279 |
| Yale | A30 | A52 | A75 | A98 | A121 | A144 | A167 | A190 | A212 | A234 | A257 | A280 |
Table A2. Index of the supplementary performance plots. Each entry gives the number of the figure for the corresponding dataset, metric group or classifier, and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.
Figure A9. CLSACC and NMI on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A10. CLSACC and NMI on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A11. CLSACC and NMI on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A12. CLSACC and NMI on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A13. CLSACC and NMI on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A14. CLSACC and NMI on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A15. CLSACC and NMI on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A16. CLSACC and NMI on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A17. CLSACC and NMI on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A18. CLSACC and NMI on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A19. CLSACC and NMI on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A20. CLSACC and NMI on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A21. CLSACC and NMI on the madelon dataset (0.5% to 10% of the features).
↑ Back to tbl:2
Figure A22. CLSACC and NMI on the ORL dataset (0.5% to 10% of the features).
↑ Back to tbl:2
Figure A23. CLSACC and NMI on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to tbl:2
Figure A24. CLSACC and NMI on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A25. CLSACC and NMI on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A26. CLSACC and NMI on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A27. CLSACC and NMI on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A28. CLSACC and NMI on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A29. CLSACC and NMI on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A30. CLSACC and NMI on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A31. CLSACC and NMI on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A32. CLSACC and NMI on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A33. CLSACC and NMI on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A34. CLSACC and NMI on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A35. CLSACC and NMI on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A36. CLSACC and NMI on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A37. CLSACC and NMI on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A38. CLSACC and NMI on the GLI_85 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A39. CLSACC and NMI on the GLIOMA dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A40. CLSACC and NMI on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A41. CLSACC and NMI on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A42. CLSACC and NMI on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A43. CLSACC and NMI on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A44. CLSACC and NMI on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A45. CLSACC and NMI on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A46. CLSACC and NMI on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A47. CLSACC and NMI on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A48. CLSACC and NMI on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A49. CLSACC and NMI on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A50. CLSACC and NMI on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A51. CLSACC and NMI on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A52. CLSACC and NMI on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A53. ACC and AUC of the KNN classifier on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A54. ACC and AUC of the KNN classifier on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A55. ACC and AUC of the KNN classifier on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A56. ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A57. ACC and AUC of the KNN classifier on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A58. ACC and AUC of the KNN classifier on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A59. ACC and AUC of the KNN classifier on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A60. ACC and AUC of the KNN classifier on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A61. ACC and AUC of the KNN classifier on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A62. ACC and AUC of the KNN classifier on the Isolet dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A63. ACC and AUC of the KNN classifier on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A64. ACC and AUC of the KNN classifier on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A65. ACC and AUC of the KNN classifier on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A66. ACC and AUC of the KNN classifier on the madelon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A67. ACC and AUC of the KNN classifier on the ORL dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A68. ACC and AUC of the KNN classifier on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A69. ACC and AUC of the KNN classifier on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A70. ACC and AUC of the KNN classifier on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A71. ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A72. ACC and AUC of the KNN classifier on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A73. ACC and AUC of the KNN classifier on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A74. ACC and AUC of the KNN classifier on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A75. ACC and AUC of the KNN classifier on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A76. ACC and AUC of the KNN classifier on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A77. ACC and AUC of the KNN classifier on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A78. ACC and AUC of the KNN classifier on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A79. ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A80. ACC and AUC of the KNN classifier on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A81. ACC and AUC of the KNN classifier on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A82. ACC and AUC of the KNN classifier on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A83. ACC and AUC of the KNN classifier on the GLI_85 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A84. ACC and AUC of the KNN classifier on the GLIOMA dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A85. ACC and AUC of the KNN classifier on the Isolet dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A86. ACC and AUC of the KNN classifier on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A87. ACC and AUC of the KNN classifier on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A88. ACC and AUC of the KNN classifier on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A89. ACC and AUC of the KNN classifier on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A90. ACC and AUC of the KNN classifier on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A91. ACC and AUC of the KNN classifier on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A92. ACC and AUC of the KNN classifier on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A93. ACC and AUC of the KNN classifier on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A94. ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A95. ACC and AUC of the KNN classifier on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A96. ACC and AUC of the KNN classifier on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A97. ACC and AUC of the KNN classifier on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A98. ACC and AUC of the KNN classifier on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A99. ACC and AUC of the decision tree classifier on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A100. ACC and AUC of the decision tree classifier on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A101. ACC and AUC of the decision tree classifier on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A102. ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A103. ACC and AUC of the decision tree classifier on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A104. ACC and AUC of the decision tree classifier on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A105. ACC and AUC of the decision tree classifier on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A106. ACC and AUC of the decision tree classifier on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A107. ACC and AUC of the decision tree classifier on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A108. ACC and AUC of the decision tree classifier on the Isolet dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A109. ACC and AUC of the decision tree classifier on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A110. ACC and AUC of the decision tree classifier on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A111. ACC and AUC of the decision tree classifier on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A112. ACC and AUC of the decision tree classifier on the madelon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A113. ACC and AUC of the decision tree classifier on the ORL dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A114. ACC and AUC of the decision tree classifier on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A115. ACC and AUC of the decision tree classifier on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A116. ACC and AUC of the decision tree classifier on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A117. ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A118. ACC and AUC of the decision tree classifier on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A119. ACC and AUC of the decision tree classifier on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A120. ACC and AUC of the decision tree classifier on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A121. ACC and AUC of the decision tree classifier on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A122. ACC and AUC of the decision tree classifier on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A123. ACC and AUC of the decision tree classifier on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A124. ACC and AUC of the decision tree classifier on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A125. ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A126. ACC and AUC of the decision tree classifier on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A127. ACC and AUC of the decision tree classifier on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A128. ACC and AUC of the decision tree classifier on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A129. ACC and AUC of the decision tree classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A130. ACC and AUC of the decision tree classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A131. ACC and AUC of the decision tree classifier on the Isolet dataset (5% to 100% of the features).
Figure A132. ACC and AUC of the decision tree classifier on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A133. ACC and AUC of the decision tree classifier on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A134. ACC and AUC of the decision tree classifier on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A135. ACC and AUC of the decision tree classifier on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A136. ACC and AUC of the decision tree classifier on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A137. ACC and AUC of the decision tree classifier on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A138. ACC and AUC of the decision tree classifier on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A139. ACC and AUC of the decision tree classifier on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A140. ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A141. ACC and AUC of the decision tree classifier on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A142. ACC and AUC of the decision tree classifier on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A143. ACC and AUC of the decision tree classifier on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A144. ACC and AUC of the decision tree classifier on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A145. ACC and AUC of the SVM classifier on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A146. ACC and AUC of the SVM classifier on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A147. ACC and AUC of the SVM classifier on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A148. ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A149. ACC and AUC of the SVM classifier on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A150. ACC and AUC of the SVM classifier on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A151. ACC and AUC of the SVM classifier on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A152. ACC and AUC of the SVM classifier on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A153. ACC and AUC of the SVM classifier on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A154. ACC and AUC of the SVM classifier on the Isolet dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A155. ACC and AUC of the SVM classifier on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A156. ACC and AUC of the SVM classifier on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A157. ACC and AUC of the SVM classifier on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A158. ACC and AUC of the SVM classifier on the madelon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A159. ACC and AUC of the SVM classifier on the ORL dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A160. ACC and AUC of the SVM classifier on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A161. ACC and AUC of the SVM classifier on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A162. ACC and AUC of the SVM classifier on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A163. ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A164. ACC and AUC of the SVM classifier on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A165. ACC and AUC of the SVM classifier on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A166. ACC and AUC of the SVM classifier on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A167. ACC and AUC of the SVM classifier on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A168. ACC and AUC of the SVM classifier on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A169. ACC and AUC of the SVM classifier on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A170. ACC and AUC of the SVM classifier on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A171. ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A172. ACC and AUC of the SVM classifier on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A173. ACC and AUC of the SVM classifier on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A174. ACC and AUC of the SVM classifier on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A175. ACC and AUC of the SVM classifier on the GLI_85 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A176. ACC and AUC of the SVM classifier on the GLIOMA dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A177. ACC and AUC of the SVM classifier on the Isolet dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A178. ACC and AUC of the SVM classifier on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A179. ACC and AUC of the SVM classifier on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A180. ACC and AUC of the SVM classifier on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A181. ACC and AUC of the SVM classifier on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A182. ACC and AUC of the SVM classifier on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A183. ACC and AUC of the SVM classifier on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A184. ACC and AUC of the SVM classifier on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A185. ACC and AUC of the SVM classifier on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A186. ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A187. ACC and AUC of the SVM classifier on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A188. ACC and AUC of the SVM classifier on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A189. ACC and AUC of the SVM classifier on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A190. ACC and AUC of the SVM classifier on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A191. ACC and AUC of the random forest classifier on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A192. ACC and AUC of the random forest classifier on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A193. ACC and AUC of the random forest classifier on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A194. ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A195. ACC and AUC of the random forest classifier on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A196. ACC and AUC of the random forest classifier on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A197. ACC and AUC of the random forest classifier on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A198. ACC and AUC of the random forest classifier on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A199. ACC and AUC of the random forest classifier on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A200. ACC and AUC of the random forest classifier on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A201. ACC and AUC of the random forest classifier on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A202. ACC and AUC of the random forest classifier on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A203. ACC and AUC of the random forest classifier on the madelon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A204. ACC and AUC of the random forest classifier on the ORL dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A205. ACC and AUC of the random forest classifier on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A206. ACC and AUC of the random forest classifier on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A207. ACC and AUC of the random forest classifier on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A208. ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A209. ACC and AUC of the random forest classifier on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A210. ACC and AUC of the random forest classifier on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A211. ACC and AUC of the random forest classifier on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A212. ACC and AUC of the random forest classifier on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A213. ACC and AUC of the random forest classifier on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A214. ACC and AUC of the random forest classifier on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A215. ACC and AUC of the random forest classifier on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A216. ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A217. ACC and AUC of the random forest classifier on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A218. ACC and AUC of the random forest classifier on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A219. ACC and AUC of the random forest classifier on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A220. ACC and AUC of the random forest classifier on the GLI_85 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A221. ACC and AUC of the random forest classifier on the GLIOMA dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A222. ACC and AUC of the random forest classifier on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A223. ACC and AUC of the random forest classifier on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A224. ACC and AUC of the random forest classifier on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A225. ACC and AUC of the random forest classifier on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A226. ACC and AUC of the random forest classifier on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A227. ACC and AUC of the random forest classifier on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A228. ACC and AUC of the random forest classifier on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A229. ACC and AUC of the random forest classifier on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A230. ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A231. ACC and AUC of the random forest classifier on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A232. ACC and AUC of the random forest classifier on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A233. ACC and AUC of the random forest classifier on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A234. ACC and AUC of the random forest classifier on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A235. ACC and AUC of the XGBoost classifier on the ALLAML dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A236. ACC and AUC of the XGBoost classifier on the arcene dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A237. ACC and AUC of the XGBoost classifier on the Carcinom dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A238. ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A239. ACC and AUC of the XGBoost classifier on the COIL20 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A240. ACC and AUC of the XGBoost classifier on the colon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A241. ACC and AUC of the XGBoost classifier on the gisette dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A242. ACC and AUC of the XGBoost classifier on the GLI_85 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A243. ACC and AUC of the XGBoost classifier on the GLIOMA dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A244. ACC and AUC of the XGBoost classifier on the Isolet dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A245. ACC and AUC of the XGBoost classifier on the leukemia dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A246. ACC and AUC of the XGBoost classifier on the lung dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A247. ACC and AUC of the XGBoost classifier on the lung_discrete dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A248. ACC and AUC of the XGBoost classifier on the madelon dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A249. ACC and AUC of the XGBoost classifier on the ORL dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A250. ACC and AUC of the XGBoost classifier on the orlraws10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A251. ACC and AUC of the XGBoost classifier on the pixraw10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A252. ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A253. ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A254. ACC and AUC of the XGBoost classifier on the TOX_171 dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A255. ACC and AUC of the XGBoost classifier on the warpAR10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A256. ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A257. ACC and AUC of the XGBoost classifier on the Yale dataset (0.5% to 10% of the features).
↑ Back to Table A2
Figure A258. ACC and AUC of the XGBoost classifier on the ALLAML dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A259. ACC and AUC of the XGBoost classifier on the arcene dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A260. ACC and AUC of the XGBoost classifier on the Carcinom dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A261. ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A262. ACC and AUC of the XGBoost classifier on the COIL20 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A263. ACC and AUC of the XGBoost classifier on the colon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A264. ACC and AUC of the XGBoost classifier on the gisette dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A265. Figure A265: ACC and AUC of the XGBoost classifier on the GLI_85 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A266. Figure A266: ACC and AUC of the XGBoost classifier on the GLIOMA dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A267. Figure A267: ACC and AUC of the XGBoost classifier on the Isolet dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A268. ACC and AUC of the XGBoost classifier on the leukemia dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A269. ACC and AUC of the XGBoost classifier on the lung dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A270. ACC and AUC of the XGBoost classifier on the lung_discrete dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A271. ACC and AUC of the XGBoost classifier on the madelon dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A272. ACC and AUC of the XGBoost classifier on the ORL dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A273. ACC and AUC of the XGBoost classifier on the orlraws10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A274. ACC and AUC of the XGBoost classifier on the pixraw10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A275. ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A276. ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A277. ACC and AUC of the XGBoost classifier on the TOX_171 dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A278. ACC and AUC of the XGBoost classifier on the warpAR10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A279. ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (5% to 100% of the features).
↑ Back to Table A2
Figure A280. ACC and AUC of the XGBoost classifier on the Yale dataset (5% to 100% of the features).
↑ Back to Table A2
References
- [1]I. Guyon, A. Elisseeff, An introduction to variable and feature selection, JMLR 3 (2003).
- [2]P. Indyk, R. Motwani, Approximate nearest neighbors: towards removing the curse of dimensionality, in: STOC, ACM, 1998.
- [3]M. Rajabinasab, A. D. Lautrup, T. Hyrup, A. Zimek, A dynamic evaluation metric for feature selection, in: Similarity Search and Applications, Springer Nature Switzerland, Cham, 2025.
- [4]F. Camastra, Data dimensionality estimation methods: a survey, Pattern Recognition 36 (12) (2003).DOI
- [5]S. Niijima, Y. Okuno, Laplacian linear discriminant analysis approach to unsupervised feature selection, TCBB 6 (4) (2009).DOI
- [6]Y. Saeys, I. Inza, P. Larrañaga, A review of feature selection techniques in bioinformatics, Bioinformatics 23 (19) (2007).pubmed.ncbi.nlm.nih.gov/17720704
- [7]A.-C. Hayur, P. Gestraud, J.-P. Vert, The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures, PloS One 6 (12) (2011).
- [8]J. Feng, H. Xu, S. Yan, Robust PCA in high-dimension: A deterministic approach, in: ICML, 2012.arxiv.org/abs/1206.4628
- [9]R. Wald, T. Khoshgoftaar, A. Napatoliano, The importance of performance metrics within wrapper feature selection, in: IRI, IEEE, 2013.DOI
- [10]X. He, M. Ji, C. Zhang, H. Bao, A variance minimization criterion for feature selection using laplacian regularization, PAMI 33 (10) (2011).DOI
- [11]B. Wang, A. Golderberg, Gradient-based laplacian feature selection, CoRR abs/1404.2948 (2014).arxiv.org/abs/1404.2948
- [12]Z. Zhao, H. Liu, Spectral feature selection for supervised and unsupervised learning, in: ICML, ACM, 2007.DOI
- [13]X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, in: NIPS, 2005.
- [14]D. Cai, C. Zhang, X. He, Unsupervised feature selection for multi-cluster data, in: KDD, ACM, 2010.DOI
- [15]L. Amsalge, O. Chelly, T. Furon, S. Girard, M. E. Houle, K. Kawarabayshi, M. Nett, Estimating local intrinsic dimensionality, in: KDD, ACM, 2015.
- [16]M. E. Houle, Dimensionality, Discriminability, Density & Distance Distributions, in: ICDMW, IEEE, 2013.DOI
- [17]M. E. Houle, Local intrinsic dimensionality I: An extreme-value-theoretic foundation for similarity applications, in: SISAP, 2017.DOI
- [18]H. Peng, F. Long, C. Ding, Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, PAMI 27 (8) (2005).pubmed.ncbi.nlm.nih.gov/16119262
- [19]T. M. Cover, J. A. Thomas, Elements of Information Theory, 1991.
- [20]M. A. Hall, L. A. Smith, Feature selection for machine learning: Comparing a correlation-based filter approach to the wrapper, in: FLAIRS Conference, Vol. 1999, 1999.
- [21]H. Liu, R. Setiono, Chi2: Feature selection and discretization of numeric attributes, in: TAI, IEEE, 1995.
- [22]R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, 2001.
- [23]F. Nie, W. Zhu, X. Li, Unsupervised feature selection with structured graph optimization, in: AAAI, AAAI Press, 2016, pp. 1302–1308.DOI
- [24]C. Wang, J. Wang, Z. Gu, J.-M. Wei, J. Liu, Unsupervised feature selection by learning exponential weights, Pattern Recognition 148 (2024). doi:https://doi.org/10.1016/j.patcog.2023.110183.
- [25]M. G. Parza, H. Zare, M. Ghatee, Unsupervised feature selection based on adaptive similarity learning and subspace clustering, Eng. Appl. Artif. Intell. 95 (2020). doi:10.1016/J.ENGAPPAI.2020.103855.
- [26]S. Karami, F. Saberi-Movahhed, P. Tiwari, P. Marttinen, S. Vahdati, Unsupervised feature selection based on variance-covariance subspace distance, Neural Networks 166 (2023). doi:10.1016/J.NEUNET.2023.06.018.DOI
- [27]X. Zhang, R. C. de Amorim, Scalable unsupervised feature selection via weight stability (2025). arXiv:2506.06114.DOI
- [28]K. Jiang, T. Cao, L. Zhu, Q. Sun, Adaptive and flexible ℓ₁-norm graph embedding for unsupervised feature selection, Applied Intelligence 54 (22) (2024). doi:10.1007/s10489-024-05760-z.
- [29]Z. Cao, X. Xie, Y. Li, Multi-view unsupervised feature selection with consensus partition and diverse graph, Information Sciences 661 (2024).DOI
- [30]J.-S. Wu, Y. Li, J.-X. Gong, W. Min, Collaborative and discriminative subspace learning for unsupervised multi-view feature selection, Engineering Applications of Artificial Intelligence 133 (2024).DOI
- [31]B. B. Mandelbrot, How long is the coast of britain?, Science 156 (3775) (1967).
- [32]T. W. Gamelin, R. E. Greene, Introduction to Topology: Second Edition, Courier Dover Publications, 2013.
- [33]M. E. Houle, H. Kashima, M. Nett, Generalized expansion dimension, in: ICDMW, IEEE, 2012.DOI
- [34]D. R. Karger, M. Ruhl, Finding nearest neighbors in growth-restricted metrics, in: STOC, ACM, 2002.
- [35]M. E. Houle, M. Nett, Rank cover trees for nearest neighbor search, in: SISAP, Springer, 2013.DOI
- [36]M. E. Houle, X. Ma, M. Nett, V. Oria, Dimensional testing for multi-step similarity search, in: ICDM, IEEE, 2012.DOI
- [37]G. Casanova, E. Englmeier, M. E. Houle, P. Kröger, M. Nett, E. Schubert, A. Zimek, Dimensional testing for reverse k-nearest neighbor search, Proc. VLDB Endow. 10 (7) (2017).DOI
- [38]T. de Vries, S. Chawla, M. E. Houle, Density-preserving projections for large-scale local anomaly detection, Knowledge and Information Systems 32 (6) (2012).DOI
- [39]A. Anderberg, J. Bailey, R. J. G. B. Campello, M. E. Houle, H. O. Marques, M. Radovanovic, A. Zimek, Dimensionality-aware outlier detection, in: SIAM International Conference on Data Mining, 2024.
- [40]X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, Advances in Neural Information Processing Systems 18, section 2 explicitly describes "Variance" as the simplest unsupervised criterion: select features with largest variance (2005).
- [41]M. A. Hall, Correlation-based feature selection for machine learningPhD thesis, University of Waikato. Originally supervised, but the correlation part is frequently adapted to unsupervised scenarios (1999).
- [42]P. Mitra, C. A. Murty, S. K. Pal, Unsupervised feature selection using feature similarity, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (3) (2002). doi:10.1109/34.990132.
- [43]H. W. Kuhn, The Hungarian Method for the assignment problem, Naval Research Logistics Quarterly 2 (1-2) (1955).
- [44]J. Li, K. Cheng, S. Wang, F. Morstatter, R. P. Trevino, J. Tang, H. Liu, Feature selection: A data perspective (2017).DOI
- [45]M. Rajabinasab, A. Zimek, FSEVAL: feature selection evaluation toolbox and dashboard, CoRR abs/2604.18227 (2026). arXiv:2604.18227, doi:10.48550/ARXIV.2604.18227.DOI
- [46]T. M. Cover, P. E. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13 (1) (1967).DOI
- [47]L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification and Regression Trees, Wadsworth, 1984.DOI
- [48]C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (3) (1995).DOI
- [49]L. Breiman, Random Forests, Machine Learning 45 (1) (2001).
- [50]T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2016.DOI
- [51]M. Rajabinasab, M. E. Houle, O. Chelly, A. Zimek, Worse than random: The importance of a baseline for unsupervised feature selection – extended version, CoRR abs/2605.22973 (2026). arXiv:2605.22973, doi:10.48550/ARXIV.2605.22973.DOI
- [52]J. Demšar, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 7 (2006).DOI
- [53]W. Dong, C. Moses, K. Li, Efficient k-nearest neighbor graph construction for generic similarity measures, in: WWW, ACM, 2011.DOI

















































