betaFeedback

WorksComputer Science

Unsupervised Feature Selection Using Local Intrinsic Dimensionality – Extended Version

Abstract

High dimensionality degrades both the efficiency and the effectiveness of learning algorithms, and in many applications no class labels are available to guide dimensionality reduction. This paper proposes LIDFS, an unsupervised feature selection method based on Local Intrinsic Dimensionality (LID). The method builds on the formal equivalence between LID and the indiscriminability of distance measures. The data matrix is transposed so that each feature is treated as a point, and the LID of each feature is estimated by maximum likelihood from the distances to its kk nearest neighboring features. Features with lower LID exhibit greater local discriminability and receive higher importance scores. A refinement step then retains one representative per feature neighborhood and demotes locally redundant features. The result is a complete ranking from which any number of features can be selected without rerunning the procedure. LIDFS does not use any information about labels or the number of classes, and runs in O(d2n)\mathcal{O}(d^2 n) time for nn instances and dd features. LIDFS is compared with classic and state-of-the-art methods and a random baseline on 23 high-dimensional datasets, the classic benchmark collection. The evaluation uses classification accuracy and AUC with five downstream classifiers, as well as clustering accuracy and normalized mutual information, over both full and aggressive selection ranges. LIDFS ranks first in the supervised evaluations and first or tied second in the clustering-based evaluations. It belongs to the top statistical group together with methods that require the number of classes, and it runs faster than its closest competitor. Several recent methods, by contrast, fail to outperform the random baseline. A sensitivity analysis shows that small neighborhood sizes yield the best results.

Introduction

Advances in computer technology are opening the way for the handling of increasingly complex data. The scale of data can be measured with respect to volume (the number of data instances) and dimensionality (the number of features that describe these instances). When dealing with colossal data volume on high-capacity computing platforms, volume reduction techniques such as sampling, together with parallelization, can allow the exploration and analysis of data with millions or even billions of data entries.

Feature reduction, the dimensional analogue of volume reduction, has several motivations [1]: reducing the costs of data collection, storage, and processing; improving the model generalizability and interpretability; and improving the discriminative power of feature-based distance measures. In general, high dimensionality is associated with a degradation in the efficiency and effectiveness of fundamental data mining and machine learning tasks such as clustering and anomaly detection [2], an effect often referred to as the “curse of dimensionality”. As the dimensionality rises, distance values between points tend to concentrate around their mean values due to the effects of noise and other sources of error, which drastically diminishes the discriminative power of the distance measure.

Feature reduction methods fall into two main categories: selection and extraction. Feature selection, as the name suggests, retains a subset of the original features, chosen with the aim of improving the performance and efficiency of data mining and machine learning tasks [3]. In feature extraction, the attributes are transformed to a smaller set within an artificial feature space, with each new feature value depending on the values of many (or even all) of the original features. Many feature extraction algorithms are based on Principal Component Analysis (PCA) [4] or Linear Discriminant Analysis [5].

For applications in which extraction methods are appropriate, extracted features are very likely to outperform any equal number of selected features, due to the greater flexibility in determining a data transformation that maximizes the quality criteria used to guide the feature generation process. However, in some contexts where feature semantics are of major importance and where artificial features are meaningless, feature extraction cannot be used. Such applications are common in bioinformatics [6] (gene annotation, micro-array analysis) and chemistry [6, 7] (mass spectra analysis, interpretability of molecular signatures), among others. In such contexts, however, feature selection can still be applied to reduce the dimensionality of the data.

Even when feature extraction methods are applicable, they are generally not suitable for very large datasets, as the transformation of the full dataset to a new full-dimensional basis can incur a prohibitively high computational cost. This computational complexity, together with a lack of robustness in the presence of noise [8], makes feature extraction less useful when the instances outnumber the features. Furthermore, feature extraction has limited use when the goal is to remove irrelevant features, since all of the original features must be retained in order to compute values within the transformed feature space. Feature extraction also reduces the explainability and transparency of the final downstream task as it becomes difficult to interpret the results which rely on a transformed space.

In this paper, we consider the case of unsupervised feature selection, where class label information is unavailable. Unsupervised feature selection is a more difficult problem than supervised feature selection, due to the difficulty in evaluating the quality of features. In the absence of ground truth information, there are two main types of quality criteria that are commonly used to guide the selection process: redundancy elimination, where an attribute is discarded whenever it can be entirely or partially inferred from other attributes, and similarity conservation, in which a feature is selected according to the degree to which it helps to conserve the similarity between data points. Of the two, similarity conservation has received relatively less attention in the research literature.

Depending on their evaluation strategy, feature selection frameworks can be categorized as either filters or wrappers (or embedded). Wrapper methods execute a predictive learning task over the data for many choices of feature subsets, and retain candidate features for which the predictive model achieves high performance [1, 9]. In practical contexts involving data of even moderate volume or dimension, the cost of wrapper methods quickly becomes prohibitively high. A more affordable alternative is that of filter methods, which evaluate feature quality without resorting to an external learning machine. Although filters generally cannot match the quality of wrappers on small-scale datasets, on datasets of larger scale filters generally require far less computation time, and are less susceptible to overfitting. Many unsupervised feature selection algorithms (such as LapAOFS / LapDOFS [10]) require that a target number of features be supplied as an input parameter, while others (such as GLFS [11]) automatically determine the number of selected features. With algorithms of the former type, the user is free to increase the number of features if needed, although the entire feature selection task usually must be reiterated. On the other hand, with most algorithms of the latter type, the user has no control over the number of features selected. When dealing with large datasets, a third and more convenient alternative is to produce a ranked list of all features, so as to allow any downstream processes to select the number of features that are most appropriate to the task at hand, given the amount of computational resources available. The most widely known unsupervised feature ranking framework is that of spectral feature selection [12], which includes Laplacian Score for Feature Selection (LS) [13]. LS attempts to produce a reduced feature set that preserves the original neighborhood information to the greatest possible extent. More recent work on filters for unsupervised feature ranking includes Multi-Cluster Feature Selection (MCFS) [14], which attempts to preserve the cluster structure of the data.

Almost all existing filter methods for feature selection are suitable only for datasets of relatively small volume and dimensionality, such as those found in the context of bioinformatics. In this work, we target the problem of unsupervised feature ranking at higher scales, where the data volume and data dimension together preclude the use of wrapper methods and computationally expensive filters. Such feature selection could be of particular benefit for database and data mining operations such as similarity search, neighborhood-based classification, and clustering.

As a criterion for guiding the selection process, we propose that features be assessed according to their ability to discriminate between the distances encountered in the neighborhoods of data points, in an attempt to alleviate the effects of the curse of dimensionality. Accordingly, we develop a feature ranking method that assesses the discriminative power of individual features according to a model of the local intrinsic dimensionality (LID) of continuous distance distributions [15, 16, 17]. While most feature selection methods employ indirect strategies in order to (implicitly) improve the discriminability of distance measures, the use of LID as guiding criterion allows for a more explicit approach, due to a formal equivalence [16] between the discriminability of distance measures and low intrinsic dimensionality.

Taking into account the performance issues due to larger data scales (in terms of both volume and dimensionality), we develop a forward filter algorithm generally applicable to continuous-valued numerical datasets.

The specific contributions of this paper include:

  • a greedy unsupervised feature selection algorithm based on LID2;

  • a theoretical analysis and discussion of the proposed method;

  • an experimental framework setting random feature selection as a baseline for feature selection, and showing the advantage of using our method with high-dimensional feature selection benchmark datasets.

The paper is organized as follows. In the next section we present a survey of existing methods for feature selection. In § 3, we introduce the LID model upon which our feature ranking method is based, as well as its relationship to the discriminability of distance measures. In § 4, we present the details of the proposed unsupervised feature selection method. The experimental framework is presented in § 5, in which our method is compared against state-of-the-art unsupervised feature selection algorithms. The experimental results are discussed in § 6. Finally, in § 7 we conclude by summarizing the advantages and the limitations of our feature selection strategy, and propose some possible extensions of our algorithm as future work.

Feature selection

Feature selection consists of choosing a subset of the original features so as to best satisfy a quality criterion. The ultimate goal is to improve model generalizability and interpretability, as well as to lower the computational costs of learning algorithms that use the data.

Formally, let us assume that we are given a set of nn points in a space of dimension dd represented by the matrix X∈Mn,d(R)X \in M_{n,d}(\mathbb{R}),

X=(x1,x2,…,xn)T=(f1,f2,…,fd),X = (x_1, x_2, \ldots, x_n)^{\mathsf{T}} = (f_1, f_2, \ldots, f_d),

where xi∈Rdx_i \in\mathbb{R}^d are points and fi∈Rnf_i \in\mathbb{R}^n are features. Feature selection consists of finding Ω′={f1′,…,fd′′}\Omega' = \{f'_1, \ldots, f'_{d'}\}, a subset of Ω={f1,…,fd}\Omega= \{f_1, \ldots, f_d\} with d′<dd' < d, that best satisfies an optimality condition.

Feature selection algorithms can be differentiated in several ways. They can be

  • wrappers or filters, according to whether or not they employ a learning task as part of their feature evaluation strategy (with embedded methods combining the characteristics of both);

  • forward or backward depending on their search direction, whether they build a feature set incrementally starting from an empty set, or cull features incrementally starting from the full set;

  • univariate or multivariate, according to whether they assess features individually or in groups.

Wrapper methods apply a predictive learning machine to the data using the candidate feature subsets; the objective function is defined by the performance of this predictive model on a task for which the feature selection is targeted [9]. Wrappers can achieve high performance with the learning machine used in the selection process. However, they are model-specific [1], as features selected using a given learning model are often inadequate for others. Wrapper methods are also computationally expensive [1]: in a wrapper selection process, assessing the quality of each candidate subset requires one run of the learning machine, and one measurement of the quality of its output. Moreover, wrappers present a risk of overfitting not only to the predictive model, but also to the data. In contrast, filter methods use properties of the data to compare the candidate subsets: they evaluate candidate feature subsets in terms of some quality criterion, often information-theoretic (MRMR [18], Information Gain [19]), correlation-based (Correlation-based Feature Selection [20]), statistical (t-test and chi-squared [21]), or based on interclass distances (Fisher Score [22], supervised Spectral Feature Selection [12]). Although they cannot match the performance of wrappers on the learning tasks used to guide the feature selection process, filters have the advantage of not being specific to any particular learning model. Furthermore, filters generally require far less computation time, and are less susceptible to overfitting. In the context of complex data, the advantages of filters are crucial. Feature selection algorithms can also be categorized according to the order in which candidate feature subsets are considered. With backward elimination, the first feature subset candidate assessed is the full dataset XX; new candidate subsets are constructed by incrementally removing features that underperform with respect to some criterion. The opposite approach, forward selection, consists of starting with an empty set, and building up candidate feature subsets through the incremental introduction of individual features. When the number of features is very high, examining candidate subsets of smaller cardinality is more affordable. In the context of high-dimensional data, this argument is largely in favor of forward selection.

Feature selection methods also differ in the way they group features for assessment: univariate methods assess features individually and independently, while multivariate methods consider sets of features during the selection process. Since combinations of features are not considered, the univariate approach cannot account for redundancy of features. However, univariate methods have the general advantage of being faster: since they consider attributes separately, their complexity is usually linear in the number of features. Although the multivariate approach is usually more computationally expensive, in the context of high-dimensional data, multivariate methods that strictly limit the numbers of candidates assessed can sometimes be more affordable than their univariate competitors.

In some cases, a vector L=(l1,l2,...,ln)L = (l_1, l_2, ..., l_n) of class labels associated with XX may also be available. Feature selection methods that make use of such label information as a ground truth are said to be ‘supervised’; if label information is not used, the method is said to be ‘unsupervised’. In general, wrapper methods tend to be supervised (due to the need to assess the result on a specific learning task).

In the context of big data, where ground truth label information is rarely available, the complexity argument favors unsupervised feature selection filters which operate in a forward direction. Among all existing candidates that fit these criteria, scalability concerns prevent the use of computationally expensive algorithms such as LapDOFS and LapAOFS [10], both of which require time more than quadratic in the number of points and number of features. Among all algorithms with reasonable scalability characteristics, the unsupervised general-purpose forward-filter Laplacian Score (LS) feature selection method [13] is perhaps the most popular. It belongs to the general framework of spectral feature selection [12], and has a computational complexity of O(dn2)\mathcal{O}(dn^2). The spectral framework offers three different approaches to scoring features, with the objective always being to preserve the graph structure of the dataset described by a spectral matrix SS. Of the three scoring functions available, the second is used by LS:

Score⁡LS(ft)=∑xi,xj∈X(xi,t−xj,t)2Var⁡(ft)Si,j,(1)\operatorname{Score}_{\mathrm{LS}}(f_t) = \sum_{x_i,x_j \in X} \frac{(x_{i,t} - x_{j,t})^2}{\operatorname{Var}(f_t)} S_{i,j}, \tag*{(1)}

where

Si,j=δi,je−di,j2θ,(2)S_{i,j} = \delta_{i,j}e^{-\frac{d_{i,j}^2}{\theta}}, \tag*{(2)}

Var⁡(ft)\operatorname{Var}(f_t) is the variance of the feature ftf_t; di,jd_{i,j} denotes the Euclidean distance between xix_i and xjx_j; δi,j\delta_{i,j} is 1 when xix_i or xjx_j is among the kk-nearest neighbors of the other, and 0 otherwise; and θ\theta represents the variance within a set of distance values, but is not explicitly defined in the original work. Features are ranked in ascending order of Score⁡LS\operatorname{Score}_{\mathrm{LS}}.

Feature selection can also be guided according to how well the new features preserve the cluster structure of the data. Given a similarity graph represented as a matrix WW with (W)i,j=δi,j(W)_{i,j} = \delta_{i,j}, and a target number of clusters cc, Multi-Cluster Feature Selection (MCFS) [14] computes the cc eigenvectors (y1,...,yc)(y_1, ..., y_c) with the smallest eigenvalues of the generalized eigenproblem Ly=λDyLy = \lambda Dy, where DD is the diagonal matrix (D)i,i=∑j(W)i,j(D)_{i,i} = \sum_j(W)_{i,j}, and L=D−WL = D - W. Least Angle Regression (LAR) is then used to solve the cc regression problems

min⁡ai∥yi−Xai∥2,subject to∥ai∥1≤γ.(3)\min_{a_i} \|y_i - Xa_i\|^2,\quad\text{subject to}\quad\|a_i\|_1 \le\gamma. \tag*{(3)}

Finally, features are scored by

Score⁡MCFS(ft)=max⁡i∣ai,t∣.(4)\operatorname{Score}_{\mathrm{MCFS}}(f_t) = \max_i |a_{i,t}|. \tag*{(4)}

Features are ranked in descending order of Score⁡MCFS\operatorname{Score}_{\mathrm{MCFS}}. MCFS has limited use in practice due to its high computational complexity: O(dn2+cd3+cd2)\mathcal{O}(dn^2 + cd^3 + cd^2).

Structured Optimal Graph Feature Selection (SOGFS) [23] jointly learns a probabilistic-neighbor similarity graph and a row-sparse projection matrix, rather than fixing the graph from the raw data beforehand. To keep the graph well-structured, it constrains the Laplacian LS\mathbf{L}_S to have exactly cc connected components, relaxed via Ky Fan’s theorem into a trace-minimization term,

∑i=1cσi(LS)=min⁡F∈Rn×c, FTF=ITr⁡(FTLSF),(5)\sum_{i=1}^{c} \sigma_i(\mathbf{L}_S) = \min_{\mathbf{F}\in\mathbb{R}^{n\times c},\,\mathbf{F}^{\mathsf{T}}\mathbf{F}=\mathbf{I}} \operatorname{Tr}(\mathbf{F}^{\mathsf{T}}\mathbf{L}_S\mathbf{F}), \tag*{(5)}

solved by alternating updates of the projection, F\mathbf{F}, and the graph, at complexity O(d3+n2m)\mathcal{O}(d^3+n^2m), where mm is the projection dimensionality.

Local structure Learning and exponentially weighted Sparse Regression Feature Selection (LLSRFS) [24] counters the tendency of sparse mapping matrices to overweight already-high-scoring features by reweighting each feature’s contribution to the pairwise distance by wlqw_l^q,

∥xi−xj∥ew2=∑l=1dwlq(xil−xjl)2,(6)\lVert\mathbf{x}_i-\mathbf{x}_j\rVert_{ew}^2 = \sum_{l=1}^{d} w_l^q(x_{il}-x_{jl})^2, \tag*{(6)}

where a smaller exponent q∈(0,1]q\in(0,1] flattens the weight distribution across more features. The authors show this is equivalent to ℓ2,pq\ell_{2,p}^{q}-norm sparse regression under q=1/p−1/2q=1/p-1/2, generalizing earlier rescaled-least-squares selectors, at complexity O(d3+n2d+nd2)\mathcal{O}(d^3+n^2d+nd^2).

Subspace Clustering unsupervised Feature Selection (SCFS) [25] implicitly learns sample similarity through a self-expressive representation rather than an explicit graph,

min⁡G∥X−GGTX∥F2,s.t. G≥0, GGT1=1,(7)\min_{\mathbf{G}}\lVert\mathbf{X}-\mathbf{G}\mathbf{G}^{\mathsf{T}}\mathbf{X}\rVert_F^2,\quad\text{s.t. }\mathbf{G}\geq0,\ \mathbf{G}\mathbf{G}^{\mathsf{T}}\mathbf{1}=\mathbf{1}, \tag*{(7)}

where the nonnegative G\mathbf{G} doubles as a soft clustering assignment, so GGT\mathbf{G}\mathbf{G}^{\mathsf{T}} encodes pairwise similarity implicitly. This is linked to the features by a sparse regression min⁡W∥XW−G∥F2+β∥W∥2,1\min_{\mathbf{W}}\lVert\mathbf{X}\mathbf{W}-\mathbf{G}\rVert_F^2+\beta\lVert\mathbf{W}\rVert_{2,1}, optimized jointly with the first objective, at complexity max⁡{O(d3),O(n2d)}\max\{\mathcal{O}(d^3),\mathcal{O}(n^2d)\} when c≪dc\ll d.

Variance–Covariance Subspace Distance Feature Selection (VCSDFS) [26] selects features by minimizing a proper metric distance between the full feature matrix and a candidate sub-matrix,

dist⁡(X,XI)=∥XXT−XIXIT∥F,(8)\operatorname{dist}(\mathbf{X},\mathbf{X}_I)=\lVert\mathbf{X}\mathbf{X}^{\mathsf{T}}-\mathbf{X}_I\mathbf{X}_I^{\mathsf{T}}\rVert_F, \tag*{(8)}

which the authors show equals the Frobenius norm of the covariance matrix of the discarded features, so minimizing it discards the least-variance features. This avoids the degeneracy of principal-angle-based distances, which are provably zero between any matrix and its own sub-matrices. The feature weights are learned via a matrix-factorization objective with an inner-product regularizer, at complexity O(td2)\mathcal{O}(td^2), where tt is the number of iterations.

Feature selection with Minkowski weighted kk-means++ (FS-MWK++) [27] selects centroids probabilistically using feature relevance estimates derived from the data, and then aggregates feature weights across a range of Minkowski exponents to identify informative features. A scalable variant of the method, namely SFS-MWK++, is based on subsampling. Robust, Adaptive and Flexible Graph (RAFG) [28] is a graph-learning framework for unsupervised feature selection. The L2,1L_{2,1}-norm is imposed on the flexible regression term to alleviate the adverse effects of both noisy features and outliers, and an L2,1L_{2,1}-norm regularization term is incorporated to ensure that the selected transformation matrix is sufficiently sparse. Several other methods proposed in recent years focus on the unsupervised multi-view feature selection problem [29, 30].

Indiscriminability and Local ID

The success of learning algorithms on high-dimensional data is often attributed to a substantial gap between the high representational dimension of the data and its comparatively low intrinsic dimensionality (ID) [16, 17]. Many feature extraction and feature selection methods implicitly exploit this gap in order to improve discriminability. However, intrinsic dimensionality may vary considerably across different regions or clusters of a dataset. Such local variation in ID is seldom exploited explicitly in feature selection.

Expansion Dimension and Local ID

Many models of ID have been proposed over the last few decades: for the theoretical analysis and characterization of datasets, the most popular of these have been topological models, such as the well-known PCA [4]; fractal models [31]; covering models [32]; and expansion models [33, 34]. Most of the measures of ID associated with these models are very expensive to compute, and are thus of limited use in practice. These models can be used ‘globally’ across the entire dataset, or adapted ‘locally’ so as to characterize the local dimensionality in the vicinity of reference points.

Global estimates of ID are typically higher than local estimates, since the number of degrees of freedom required to describe a dataset is (as a rule) higher than that required to describe its components. This gap between global and local ID measurements indicates an opportunity for the latter in guiding feature selection to improve the performance of similarity search. Improvements in the performance of search would have a direct impact on applications of machine learning, data mining, and databases. Among the models of ID mentioned above, expansion models have seen use in the design and analysis of similarity search algorithms [34, 35, 36, 37], as well as outlier detection [38, 39].

The intrinsic dimensionality mm in a Euclidean vector space can be determined using the ratios of two volume and two distance measurements. Consider the concentric balls B(x,r1)B(x, r_1) and B(x,r2)B(x, r_2) centered at xx with radii 0<r1<r20 < r_1 < r_2. Let λ\lambda be a (Lebesgue) volume measure. The ratio of volumes leads to a simple closed-form expression for the dimensionality dd of Euclidean spaces:

λ(B(x,r2))λ(B(x,r1))=(r2r1)m⇒m=ln⁡λ(B(x,r2))−ln⁡λ(B(x,r1))ln⁡r2−ln⁡r1.(9)\begin{aligned} \frac{\lambda(B(x,r_2))}{\lambda(B(x,r_1))} = \left(\frac{r_2}{r_1}\right)^m \\ &\Rightarrow\quad m = \frac{\ln\lambda(B(x,r_2))-\ln\lambda(B(x,r_1))}{\ln r_2-\ln r_1}. \tag*{(9)} \end{aligned}

An oracle that accepts two concentric balls as a query, and returns the dimension of the space, is not immediately useful. However, if volume is estimated by the number of points within distance rr from a reference point xx, then the returned value can serve as a measure of intrinsic dimensionality, the local expansion dimension (ED) [33, 34].

The model of intrinsic dimensionality considered here is a variant of ED, in which the notion of volume is generalized to a probability measure, and distances are modeled as a random variable [16, 17]. Let xx be a reference point, and XX be a random variable expressing a distance measurement from XX. Let PXP_X be the cumulative distribution function associated with XX. For any rr such that PX(r)>0P_X(r) > 0 and PXP_X is continuously differentiable in an interval containing rr, the Local Intrinsic Dimensionality (LID) of XX at distance rr is defined, wherever the limit exists, as

LID⁡PX(r)≜lim⁡ε→0+ln⁡(PX((1+ε)r)/PX(r))ln⁡(1+ε)=r⋅PX′(r)PX(r)(10)\operatorname{LID}_{P_X}(r) \triangleq\lim_{\varepsilon\to0^+}\frac{\ln(P_X((1+\varepsilon)r)/P_X(r))}{\ln(1+\varepsilon)} = \frac{r\cdot P_X'(r)}{P_X(r)} \tag*{(10)}

Indiscriminability

A distance measure is ‘discriminative’ when an expansion in distance results in a relatively small increase in the number of observations. Discriminability of points improves the efficiency of data mining and machine learning tasks, minimizes error and ensures robustness. Capturing a huge number of points by a small increase in distance measure greatly increases the computational cost of applications, and has a negative impact on effectiveness. Moreover, the relative ranking of points with reference to an indiscriminative distance measure can easily be affected by noise. When selecting features, it is strongly desirable that they be chosen so as to ensure better discriminability.

Formally, let xx be a reference point and let XX be a random distance variable with respect to that reference point, as defined in § 3.1. For any distance rr from xx such that PX(r)>0P_X(r) > 0, the indiscriminability of XX at rr is given by the following limit wherever it exists [16, 17]:

InDiscr⁡PX(r)≜lim⁡ε→0+(PX((1+ε)r)−PX(r)ε⋅FR(r))=r⋅PX′(r)PX(r)=LID⁡PX(r).(11)\begin{aligned} \operatorname{InDiscr}_{P_X}(r) \triangleq\lim_{\varepsilon\to0^+}\left(\frac{P_X((1+\varepsilon)r)-P_X(r)}{\varepsilon\cdot F_R(r)}\right) \\ &= \frac{r\cdot P_X'(r)}{P_X(r)} = \operatorname{LID}_{P_X}(r). \tag*{(11)} \end{aligned}

Note that this definition of indiscriminability is unitless, does not require knowledge of the statistical parameters of the underlying distance distribution, and coincides precisely with local ID (Eq. 10).

Estimation of Local Intrinsic Dimensionality

As the number of data elements increases, the distance from a query to its kk-th nearest neighbor tends to decrease. Asymptotically speaking, in the context of continuous distance distributions, the radius of the local vicinity within which a query result is determined tends to zero. Accordingly, the discriminability of distances to the query should be assessed as the limit LIDFR∗\mathrm{LID}^{*}_{F_R} of LIDFR(r)\mathrm{LID}_{F_R}(r) as rr tends to zero, where FRF_R is the Cumulative Distribution Function (CDF) of the random distance variable RR, representing distances from the query point to other data objects. As it turns out, this limit can be estimated using Extreme Value Theory (EVT), which states that the tail of the distribution of the random distance variable RR can be modeled as a Weibull distribution. The Weibull distribution is specified by two parameters: the scale σ\sigma and the shape ξ\xi. For a sufficiently small neighborhood distance threshold, the random distance variable representing the threshold excesses is Weibull-distributed [15].

Concretely, the distances (r1,…,rk)(r_1,\ldots,r_k) to the kk-nearest neighbors of a reference point xx are instances of the random variable RR. These instances can be used to estimate the Weibull shape parameter ξ\xi. A maximum likelihood estimator (MLE) can be used to estimate ξ\xi and therefore LIDFR∗\mathrm{LID}^{*}_{F_R} [15]:

LID^FR=−1ξ^=(−1k∑i=1kln⁡rirk)−1.(12)\widehat{\mathrm{LID}}_{F_R}=-\frac{1}{\hat{\xi}}=\left(-\frac{1}{k}\sum_{i=1}^{k}\ln\frac{r_i}{r_k}\right)^{-1}. \tag*{(12)}

Figure 1 illustrates the model on three planar point sets whose distance distributions from the reference point satisfy FR(r)=rmF_R(r)=r^m exactly, for m∈{1,2,4}m\in\{1,2,4\}. Starting from neighborhoods of equal size, an expansion of the radius by a common factor 1+ϵ1+\epsilon admits comparatively few additional points when the local ID is low, and a great many when it is high; the corresponding expansion dimension recovers dd in each case, as does the MLE. The rightmost distance measure is thus the least discriminative, even though every case is embedded in the plane.

Figure 1. LID as neighborhood growth. Top: three point sets sampled so that FR(r)=rmF_R(r)=r^m holds exactly for m=1,2,4m=1,2,4, which is possible in the plane since LID depends on the distance distribution alone. Each disc starts from a neighborhood of the same size, ∣B(x,r)∣≈120|B(x,r)|\approx120, and expands the radius by the same factor 1+ϵ=1.41+\epsilon=1.4: the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension. (a) On log-log axes, the slope of FRF_R is the dimension. (b) The (1+ϵ)(1+\epsilon) ratio across distances, with the MLE LID^FR\widehat{\mathrm{LID}}_{F_R} from k=500k=500 neighbors.

Method description

For a given reference point, LID quantifies the local rate of neighborhood growth, or equivalently, the difficulty of discriminating among nearby objects, with lower LID indicating greater local discriminability. Because higher local discriminability benefits distance-based learning, our selection strategy applies this principle in the transposed feature space, treating each feature as a point and evaluating it by the discriminability of its distances to neighboring features. A feature’s overall quality is then given by the inverse of the LID estimate computed from its kk-nearest-neighbor distances.

We detail our algorithm in the following; a sketch is provided in Alg. 1. Let X=[f1,…,fd]∈Rn×dX=[f_1,\ldots,f_d]\in\mathbb{R}^{n\times d} denote the input data matrix, where nn is the number of samples, dd is the number of features, and each column vector fi∈Rnf_i\in\mathbb{R}^n contains the values of the ii-th feature across all samples. Since the goal is to evaluate relationships among features rather than among samples, LID-based feature selection first transposes the data matrix, yielding XT∈Rd×nX^{\mathsf{T}}\in\mathbb{R}^{d\times n}. In this representation, each row corresponds to a feature described by its values across the dataset. The algorithm can therefore construct neighborhoods in the transposed space, such that features with similar value profiles across the samples are considered close to one another.

Given a user-defined neighborhood size kk and a chosen distance measure, the algorithm computes the kk nearest neighboring features of each feature fif_i. The resulting neighbor distances are stored in a matrix DD, where DiℓD_{i\ell} denotes the distance from fif_i to its ℓ\ell-th nearest neighbor. The corresponding neighbor indices are stored in a matrix II, where IiℓI_{i\ell} identifies the index of the ℓ\ell-th nearest neighbor of fif_i. The distances in the ii-th row of DD are then used to estimate the Local Intrinsic Dimensionality of fif_i, denoted by LID^(fi;Di,:)\widehat{\mathrm{LID}}(f_i;D_{i,:}). Before the neighbor search, duplicate features are removed, since they would result in zero neighbor distances. Moreover, the LID estimate of a feature is undefined if all of its kk neighbor distances are equal; in this case, kk is increased until every feature has at least two distinct neighbor distances.

A lower LID estimate indicates a lower relative rate of neighborhood growth and, according to the connection between LID and indiscriminability, greater local discriminability among neighboring features. In the transposed space, this suggests that the feature has a comparatively distinctive value profile across the samples and is less locally redundant with nearby features. Such a feature may therefore have greater potential to preserve distinctions among the points of XX, although this implication is heuristic rather than guaranteed by the LID estimate alone. The initial importance score of each feature is therefore defined as the inverse of its estimated LID:

S(fi)=1LID^(fi;Di,:)S(f_i)=\frac{1}{\widehat{\mathrm{LID}}(f_i;D_{i,:})}

Accordingly, features with lower estimated LID values receive higher initial importance scores.

After computing the initial scores, the features are sorted in descending order of importance. The resulting ordered list is denoted by

Ω∗=(f1∗,f2∗,…,fd∗),\Omega^*=(f_1^*,f_2^*,\ldots,f_d^*),

where fi∗f_i^* is the feature at rank ii, and S(fi∗)≥S(fj∗)S(f_i^*)\ge S(f_j^*) for all i<ji<j. The LID-based feature selection algorithm then performs a refinement step to reduce redundancy among highly ranked features. Two sets are initialized: the selected-feature set FF, which stores representative features, and the reserve set RR, which stores features considered locally redundant with respect to previously selected features.

The algorithm scans the ranked list Ω∗\Omega^* from the highest-scoring feature to the lowest-scoring feature. For each candidate feature fi∗f_i^*, it determines whether fi∗f_i^* belongs to the kk-neighborhood of any feature already included in FF. If it does not belong to the neighborhood of any selected feature, it is added to FF and treated as non-redundant under the neighborhood criterion. Otherwise, it is added to RR, since it is locally similar to at least one previously selected feature.

Finally, the scores of the features stored in the reserve set RR are adjusted. If R≠∅R\ne\varnothing, let

Smax⁡=max⁡f∈RS(f)S_{\max}=\max_{f\in R}S(f)

@page 8/120 starts_mid=0 ends_mid=1

Figure 1 (PDF p. 8). LID as neighborhood growth. Top: three point sets sampled so that FR(r)=rmF_R(r)=r^m holds exactly for m=1,2,4m=1,2,4, which is possible in the plane since LID depends on the distance distribution alone. Each disc starts from a neighborhood of the same size, ∣B(x,r)∣≈120|B(x,r)|\approx120, and expands the radius by the same factor 1+ϵ=1.41+\epsilon=1.4: the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension. (a) On log-log axes, the slope of FRF_R is the dimension. (b) The (1+ϵ)(1+\epsilon) ratio across distances, with the MLE LID^FR\widehat{\mathrm{LID}}_{F_R} from k=500k=500 neighbors.

importance score of each feature is therefore defined as the inverse of its estimated LID:

S(fi)=1LID^(fi;Di,:)S(f_i)=\frac{1}{\widehat{\mathrm{LID}}(f_i;D_{i,:})}

Accordingly, features with lower estimated LID values receive higher initial importance scores.

After computing the initial scores, the features are sorted in descending order of importance. The resulting ordered list is denoted by

Ω∗=(f1∗,f2∗,…,fd∗),\Omega^*=(f_1^*,f_2^*,\ldots,f_d^*),

where fi∗f_i^* is the feature at rank ii, and S(fi∗)≥S(fj∗)S(f_i^*)\ge S(f_j^*) for all i<ji<j. The LID-based feature selection algorithm then performs a refinement step to reduce redundancy among highly ranked features. Two sets are initialized: the selected-feature set FF, which stores representative features, and the reserve set RR, which stores features considered locally redundant with respect to previously selected features.

The algorithm scans the ranked list Ω∗\Omega^* from the highest-scoring feature to the lowest-scoring feature. For each candidate feature fi∗f_i^*, it determines whether fi∗f_i^* belongs to the kk-neighborhood of any feature already included in FF. If it does not belong to the neighborhood of any selected feature, it is added to FF and treated as non-redundant under the neighborhood criterion. Otherwise, it is added to RR, since it is locally similar to at least one previously selected feature.

Finally, the scores of the features stored in the reserve set RR are adjusted. If R≠∅R\ne\varnothing, let

Smax⁡=max⁡f∈RS(f)S_{\max}=\max_{f\in R}S(f)

Algorithm 1 LID Unsupervised Feature Selection

Input: Dataset X∈Rn×dX \in\mathbb{R}^{n \times d} and feature-neighborhood size k∈[2,d)k \in[2,d).

Output: Feature-importance score vector SS.

  1. XT←transpose⁡(X)X^{\mathsf{T}} \leftarrow\operatorname{transpose}(X)

  2. Remove duplicate rows of XTX^{\mathsf{T}}.

  3. For each i∈{1,…,d}i \in\{1,\ldots,d\}, compute the kk nearest neighboring features of fif_i in XTX^{\mathsf{T}}, excluding fif_i itself. Store the neighbor distances in DD, where DiℓD_{i\ell} is the distance from fif_i to its ℓ\ell-th nearest neighbor, and store the corresponding feature indices in II, where IiℓI_{i\ell} is the index of the ℓ\ell-th nearest neighbor of fif_i.

  4. while ∃i:Di1=Dik\exists i : D_{i1}=D_{ik} do

  5. k←k+1k \leftarrow k+1

  6. Extend DD and II with the kk-th nearest neighbor of each feature.

  7. end while

  8. Estimate the LID of each feature and compute its initial score: S(fi)←1LID⁡^(fi;Di,:)S(f_i) \leftarrow\frac{1}{\widehat{\operatorname{LID}}(f_i;D_{i,:})}, i=1,…,di=1,\ldots,d.

  9. Let π=(π1,π2,…,πd)\pi=(\pi_1,\pi_2,\ldots,\pi_d) be a permutation of {1,…,d}\{1,\ldots,d\} such that S(fπr)≥S(fπs)S(f_{\pi_r}) \ge S(f_{\pi_s}) for all r<sr<s.

  10. Initialize the selected-index set F←∅F \leftarrow\varnothing and the reserve-index set R←∅R \leftarrow\varnothing.

  11. for r←1r \leftarrow1 to dd do

  12. i←πri \leftarrow\pi_r

  13. is_neighbor←falseis\_neighbor \leftarrow\mathrm{false}

  14. for j∈Fj \in F do

  15. if i∈{Ij1,…,Ijk}i \in\{I_{j1},\ldots,I_{jk}\} then

  16. is_neighbor←trueis\_neighbor \leftarrow\mathrm{true}

  17. break

  18. end if

  19. end for

  20. if is_neighboris\_neighbor then

  21. R←R∪{i}R \leftarrow R \cup\{i\}

  22. else

  23. F←F∪{i}F \leftarrow F \cup\{i\}

  24. end if

  25. end for

  26. if R≠∅R \ne\varnothing then

  27. Smax⁡←max⁡i∈RS(fi)S_{\max} \leftarrow\max_{i \in R} S(f_i)

  28. for i∈Ri \in R do

  29. S(fi)←−(1−S(fi)/Smax⁡)S(f_i) \leftarrow-\left(1-S(f_i)/S_{\max}\right)

  30. end for

  31. end if

  32. return S=(S(f1),…,S(fd))S=(S(f_1),\ldots,S(f_d)).

denote the largest initial importance score among the reserved features. For each feature fr∈Rf_r \in R, the score is updated according to

S(fr)=−(1−S(fr)Smax⁡).S(f_r)=-\left(1-\frac{S(f_r)}{S_{\max}}\right).

Since the initial scores are positive and S(fr)≤Smax⁡S(f_r) \le S_{\max}, this transformation assigns a score in the interval (−1,0](-1,0] to every reserved feature while preserving their relative ordering. In particular, the highest-scoring feature in RR receives a refined score of 0, whereas the remaining reserved features receive negative scores. Features in the selected set FF retain their positive initial scores and therefore precede all features in RR in the final ranking.

As a result of this adjustment, a complete ranking is produced in which locally non-redundant features are prioritized, while the relative ordering of the reserved features is retained. Consequently, a user may select any desired number of top-ranked features without rerunning the feature-selection procedure. If R=∅R=\varnothing, no score adjustment is required. The final output of LIDFS is the feature-importance vector

S=(S(f1),S(f2),…,S(fd)),(13)S = (S(f_1), S(f_2), \ldots, S(f_d)), \tag*{(13)}

which can be used to rank the features for unsupervised feature selection.

LID-based feature-importance scores quantify the local discriminability of features in the transposed feature space. A feature with a lower estimated LID has a more distinctive value profile relative to its neighboring features and is therefore treated as a potentially informative representative feature. The refinement step further prioritizes features from different neighborhoods, reducing the selection of locally redundant features. The neighborhood size kk controls the locality of both the LID estimates and the redundancy criterion: smaller values of kk emphasize fine-scale neighborhood structure but yield less stable estimates, whereas larger values yield smoother estimates but may merge distinct local structures. The choice of kk thus involves a trade-off between locality and estimation stability.

Computational complexity

The time complexity of LIDFS is analyzed for a dataset with nn samples and dd features, assuming a distance measure that is computed in O(n)\mathcal{O}(n) time for a pair of features, such as the Euclidean distance. Since the removal of duplicate features can only reduce the number of features, dd also bounds the number of features that remain after this step. Transposing the data matrix requires O(nd)\mathcal{O}(nd) time, and duplicate features are removed in O(nd)\mathcal{O}(nd) expected time by hashing the rows of X⊤X^\top. The dominant cost is the kk-nearest-neighbor search in the transposed space. An exact search computes the distances between all pairs of features in O(d2n)\mathcal{O}(d^2n) time, after which the kk nearest neighbors of each feature are obtained by partial selection in O(d)\mathcal{O}(d) time and sorted in O(klog⁡k)\mathcal{O}(k \log k) time, resulting in O(d2n+dklog⁡k)\mathcal{O}(d^2n + dk \log k) time in total. The loop that increases kk is only entered if all kk neighbor distances of a feature are equal, in which case the neighbor lists are extended from the already computed distances; even if kk grows to d−1d - 1, this requires at most O(d2log⁡d)\mathcal{O}(d^2 \log d) time for sorting all rows of the distance matrix. The maximum-likelihood estimate of LID (Eq. (12)) requires a single pass over the kk neighbor distances of a feature, and therefore the initial scores of all features are computed in O(dk)\mathcal{O}(dk) time. Sorting the features by their scores requires O(dlog⁡d)\mathcal{O}(d \log d) time. In the refinement step, the membership test is implemented with a Boolean array that marks the neighbors of all selected features, such that each candidate is checked in O(1)\mathcal{O}(1) time and each insertion into FF requires O(k)\mathcal{O}(k) time, resulting in O(dk)\mathcal{O}(dk) time for the entire scan.3 Finally, the adjustment of the scores of the reserved features requires O(∣R∣)≤O(d)\mathcal{O}(|R|) \leq\mathcal{O}(d) time. Consequently, the overall time complexity of LIDFS is

O(d2n+d2log⁡d+dklog⁡k)=O(d2n),\mathcal{O}(d^2 n + d^2 \log d + dk \log k) = \mathcal{O}(d^2 n),

where the simplification holds for log⁡d≤n\log d \leq n, which is satisfied by any practical dataset. The running time of LIDFS is therefore quadratic in the number of features and linear in the number of samples, while the neighborhood size kk only affects lower-order terms. The space complexity is O(nd+d2)\mathcal{O}(nd + d^2) if the full distance matrix is stored, which reduces to O(nd+dk)\mathcal{O}(nd + dk) if the distances are computed in blocks and only the nearest neighbors of each feature are retained. For datasets with a very large number of features, the exact neighbor search can be replaced by an approximate nearest-neighbor search to alleviate the quadratic dependence on dd.

Experimental framework

In this section, we describe the experimental setup used to compare the feature selection performance of LID-based Feature Selection (LIDFS) with various unsupervised feature selection methods. Competitors include traditional methods (Variance-based [40], Correlation-based [41, 42], Laplacian Score [13], and MCFS [14]) as well as recent methods (SCFS [25], SOGFS [23], LLSRFS [24], and VCSDFS [26]).

Evaluation Metrics

To evaluate the feature selection methods across both supervised and unsupervised downstream tasks, four key metrics are employed: Accuracy (ACC), Area Under the Curve (AUC), Clustering Accuracy (CL­SACC), and Normalized Mutual Information (NMI).

Accuracy (ACC) Accuracy is a fundamental metric for supervised classification tasks, representing the proportion of correct predictions. For a set of nn instances with true labels lil_i and predicted labels l^i\hat{l}_i, it is calculated as:

ACC=1n∑i=1nI(li=l^i)(14)\mathrm{ACC} = \frac{1}{n}\sum_{i=1}^{n}\mathbb{I}(l_i = \hat{l}_i) \tag*{(14)}

where I(⋅)\mathbb{I}(\cdot) is the indicator function.

Area Under the Curve (AUC) AUC measures the ability of a classifier to distinguish between classes. It is the area under the Receiver Operating Characteristic (ROC) curve, which plots the True Positive Rate (Sensitivity) against the False Positive Rate (1 - Specificity) at various threshold settings. AUC ranges from 0 to 1, with 0.5 indicating random performance and a higher value indicating better performance. For a given threshold, with TP, FP, TN, and FN denoting the numbers of true positives, false positives, true negatives, and false negatives, the True Positive Rate (TPR) and False Positive Rate (FPR) are defined as:

TPR=TPTP+FNandFPR=FPFP+TN(15)\mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} \quad\text{and} \quad\mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}} \tag*{(15)}

Clustering Accuracy (CLSACC) Clustering Accuracy assesses how well the clusters found by the algorithm match the true labels of the data. It requires finding the best one-to-one mapping (permutation π\pi) between the cluster assignments (cic_i) and the true labels (lil_i).

CLSACC=1n∑i=1nI(li=π(ci))(16)\mathrm{CLSACC} = \frac{1}{n}\sum_{i=1}^{n}\mathbb{I}(l_i = \pi(c_i)) \tag*{(16)}

where nn is the number of data points. The optimal mapping π\pi is typically found using the Hungarian algorithm [43].

Normalized Mutual Information (NMI) NMI quantifies the degree of dependence between the clustering result (CC) and the true labels (LL). It is a normalization of the Mutual Information (MI), making it comparable across different cluster numbers. NMI is calculated as:

NMI(C,L)=2MI⁡(C,L)H(C)+H(L)(17)\mathrm{NMI}(C,L) = \frac{2\operatorname{MI}(C,L)}{H(C)+H(L)} \tag*{(17)}

where MI⁡(C,L)\operatorname{MI}(C,L) is the Mutual Information between CC and LL, and H(C)H(C) and H(L)H(L) are the entropies of CC and LL, respectively. The Mutual Information is given by:

MI⁡(C,L)=∑c∈C∑l∈LP(c,l)log⁡(P(c,l)P(c)P(l))(18)\operatorname{MI}(C,L) = \sum_{c \in C}\sum_{l \in L} P(c,l)\log\left(\frac{P(c,l)}{P(c)P(l)}\right) \tag*{(18)}

Feature Selection Dynamic Evaluation Metric (FSDEM) In order to evaluate the overall process of feature selection, we use FSDEM combined with the aforementioned metrics. FSDEM is calculated as:

FSDEM=∫abg(x) dxb−a,(19)\mathrm{FSDEM} = \frac{\int_{a}^{b} g(x)\,\mathrm{d}x}{b-a}, \tag*{(19)}

where g(x)g(x) indicates the approximated function based on different observations of an arbitrary evaluation metric, and aa and bb indicate the specific range of values (percentage of selected features) in which the performance is evaluated.

Datasets

We conduct experiments on 23 high-dimensional datasets from the scikit-feature repository [44]. Table 1 presents an overview of the characteristics of the datasets. The datasets vary in the number of instances, features, and domains. This repository is the standard source for feature selection benchmarks.

IDDatasetInstancesFeatures
D1ALLAML727,129
D2arcene20010,000
D3Carcinom1749,182
D4CLL_SUB_11111111,340
D5COIL201,4401,024
D6colon622,000
D7gisette7,0005,000
D8GLI_858522,283
D9GLIOMA504,434
D10Isolet1,560617
D11leukemia727,070
D12lung2033,312
D13lung_discrete73325
D14madelon2,600500
D15ORL4001,024
D16orlraws10P10010,304
D17pixraw10P10010,000
D18Prostate_GE1025,966
D19SMK_CAN_18718719,993
D20TOX_1711715,748
D21warpAR10P1302,400
D22warpPIE10P2102,420
D23Yale1651,024

Table 1. Characteristics of the high-dimensional benchmark datasets.

Evaluation Configuration

We perform our experiments using the FSEVAL benchmarking suite [45], a specialized framework designed for the extensive and comprehensive evaluation of feature selection algorithms. The supervised evaluation is conducted with five downstream classifiers, namely KNN [46], Decision Tree [47], SVM [48], Random Forest [49], and XGBoost [50] with a 5-fold stratified cross-validation to provide a robust evaluation. For brevity, we present the detailed results for the Random Forest classifier in § 6. The results for the other classifiers (reported in the Appendix) lead to the same conclusions. Multiclass AUC is macro-averaged. For CLSACC and NMI, we use the average of 10 runs of the kk-means algorithm. kk-means is always given the true number of classes. For the random baseline, the evaluation is repeated 100 times to clearly reflect the standard deviation. The average value is used for further analysis. For LIDFS, we use k=5k = 5 as a default setting to calculate LID values for each feature. We favor smaller values for kk, as fine-grained neighborhoods are deemed more important than the approximation accuracy of the LID. We use Euclidean distance as the chosen distance measure and we also include Random feature selection [51] as a baseline for our evaluation. Runtime tests ran on Ubuntu with an AMD Ryzen 9 9950X (128 GB RAM).

We conduct two different experiments to evaluate the performance of the feature selection process. First, we evaluate the overall feature selection process by selecting from 5% to 100% of the features with a step size of 5%. As the datasets are very high-dimensional, we also conduct a different experiment by selecting from 0.5% to 10% of features with a step size of 0.5%. This allows us to have a more realistic view of the performance of feature selection algorithms to make sure the high number of selected features relative to the number of instances does not obscure the results.

Results

In this section, we present the experimental results based on the described evaluation framework. We aim to evaluate different aspects of unsupervised feature selection algorithms, including efficiency and performance, to comprehensively assess the characteristics of LIDFS against the competitors.

Runtime Analysis

Our first experiment evaluates the runtime of the unsupervised feature selection algorithms. Demonstrating computational efficiency is essential to understanding the practical significance of the proposed method’s performance. This experiment considers two variables: the number of instances and the number of features. We report two settings, in each of which one variable is fixed at 100 while the other varies from 1000 to 20000 in steps of 500. Runtime is capped at one hour per method; the first time a method exceeds this cap, the elapsed time is recorded and no further settings are evaluated for that method. Figure 2 summarizes the results, with the yy-axis on a logarithmic scale for readability.

Figure 2. Runtime analysis of the feature selection methods. The yy-axis is presented in logarithmic scale.

As expected, the random baseline is by far the most efficient method, as it involves no significant computation. It is followed by traditional methods such as variance-based and correlation-based feature selection. Recent state-of-the-art methods are computationally more expensive. LIDFS shows competitive results in the experiment with varying numbers of features and highly efficient performance when the number of instances is varied. The only recent method that consistently achieves better efficiency is VCSDFS, which outperforms LIDFS by a small margin. SOGFS and LLSRFS are the most expensive methods, and their runtime is capped at an early stage. As we aim to provide consistent and comprehensive experiments on large, high-dimensional datasets, we omit SOGFS and LLSRFS from the remainder of the experiments.

Performance Analysis

We analyze the performance of the proposed method in comparison with traditional and state-of-the-art unsupervised feature selection algorithms based on the 23 benchmark datasets and the four different evaluation metrics.

Figure 3 presents an example of the four evaluation metrics on the Isolet dataset for the full-range experiment. LIDFS achieves the top performance while also maintaining the efficiency demonstrated earlier, and it generally outperforms the random baseline. Many feature selection methods, including recent state-of-the-art algorithms, fail to outperform the random baseline. This might seem surprising, but is in fact consistent with previous observations [51]. The same pattern holds for the second case, in which the range of the number of selected features is limited to 10%, as presented in Figure 4.

Figure 3. Comparison of the feature selection performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the full range of the number of selected features.

Figure 4. Comparison of the performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the first 10% of the features.

An interesting observation in Figure 3 for the full-range experiment is that, for the top-performing methods, the performance curve flattens after roughly the top 15% of features (as ranked by each method) have been selected. Naïve, low-performing methods such as correlation-based feature selection continue to improve up to around the 80% selection mark, or beyond. This is consistent with our assumption in LIDFS: within each feature neighborhood, one feature is designated as the representative and the rest are moved to the reserve set, reflecting their possible redundancy. The results suggest that, with an appropriate choice of kk for the neighborhood search, LIDFS identifies this redundancy, assigning positive importance values only to features whose selection is expected to increase the informational value of the lower-dimensional representation based on LIDFS’s perspective. Figure 4 confirms this, as the performance curves remain almost monotonically increasing throughout the 10% selection experiment.

Feature selection performance varies across datasets; we selected one exhibiting a commonly observed pattern for illustration. Individual results for all datasets are available in the Appendix. The FSDEM score, which shows the overall performance over a range, is presented for both experiments in Figure 5 and Figure 7. We observe that LIDFS constitutes the frontier almost consistently, providing a reliable and efficient solution for unsupervised feature selection. To analyze the overall performance across all benchmark datasets, we use critical difference diagrams, which rank methods by performance from right to left, following the methodology of Demšar [52].

Figure 5. Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the full (100%) selection scenarios. LIDFS constitutes the frontier almost consistently.

Figure 7. Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the aggressive (10%) selection scenarios. LIDFS constitutes the frontier almost consistently.

The critical difference diagrams for the different metrics, over the full range of feature numbers and the 10% range, are shown in Figure 6 and Figure 8, respectively. This analysis uses the FSDEM score [3], computed as the average performance over the range for each of the four metrics.

Figure 6. Critical difference diagram over the full range of number of features for the four different metrics based on the average performance measured by the FSDEM score.

Figure 8. Critical difference diagram over the first 10% of number of features for the four different metrics based on the average performance measured by the FSDEM score.

LIDFS is in most cases the highest-performing method for unsupervised feature selection. For supervised downstream tasks, LIDFS, SCFS [25], and MCFS [14] form the top statistical group, with LIDFS ranked first and a clear margin over the remaining methods. Other state-of-the-art methods, such as VCSDFS [26], perform even worse than the random baseline. For the unsupervised evaluation, LIDFS is either the best or the second-best (in a tie), without a statistically significant difference or a large margin relative to the best method. On the efficiency frontier, LIDFS shows a lower computational complexity and runtime compared to the closest competitor, SCFS [25]. It is noteworthy that LIDFS works in a truly unsupervised setting, while some of the competitors, including SCFS [25] and MCFS [14], use information extracted from the labels, such as the number of unique classes. The results for the other classifiers are almost identical to those for the Random Forest classifier and lead to the same conclusions.

Sensitivity Analysis

The neighborhood size kk used to estimate the LID of each point is the central hyperparameter of the proposed feature selection procedure. As discussed earlier, theoretically, larger values of kk are expected to produce more accurate LID estimates, since the estimator averages over a larger sample. At the same time, a larger kk enlarges the neighborhood over which LID is computed possibly beyond local characteristics, reducing the granularity of the discrimination between features. These two effects act in opposite directions, so the net effect of kk on downstream feature selection quality is not obvious a priori and must be assessed empirically. We evaluate k∈{5,10,25,50,100}k \in\{5, 10, 25, 50, 100\} on all 23 benchmark datasets under the two feature-selection scenarios, retaining up to 10% of the original features, and full selection up to 100%.

Downstream performance Figure 9 summarizes different evaluation metrics as a function of kk, averaged over datasets and over the feature-selection fraction within each scenario. This figure shows the overall trend of different performance metrics when kk increases, which is generally decreasing. This illustrates that the granularity of neighborhoods in the feature selection process is more important than the smoothness of the LID estimation.

Figure 9. Mean AUC, ACC, NMI, and CLSACC as a function of the neighborhood size kk, averaged over datasets and over the feature-selection fraction, for the aggressive (10%) and full (100%) selection scenarios. Error bars show ±1\pm1 standard error of the mean over the 23 datasets. For all metrics higher values are better.

Per-dataset heterogeneity and practical guidance The FSDEM score based on each metric on different datasets is presented, for the full selection and aggressive selection scenarios, in Figure 10 and Figure 11, respectively. These figures highlight that smaller values of kk, which yield finer neighborhoods, generally outperform larger values, which yield more stable LID estimates. However, it is evident that no single value of kk is optimal on every dataset. The rank heatmap in Figure 12 shows that the preference for small kk is strongest for AUC and ACC, while NMI and CLSACC are comparatively indifferent. In the full (100%) selection scenario, the best FSDEM score averaged over the four metrics is attained at k=5k = 5 on 7 of the 23 datasets, at k=10k = 10 on 6, at k=25k = 25 on 1, at k=50k = 50 on 5, and at k=100k = 100 on 4. Small-to-moderate neighborhoods (k=5k = 5 or k=10k = 10) are therefore individually optimal on 13 of 23 datasets (57%), consistent with the aggregate trend, but the remaining 43% of datasets favor k≥25k \ge25, so dataset-specific tuning of kk remains worthwhile when computational budget allows. In the absence of tuning, our results support using a small kk (k=5k = 5 as used and discussed in the main experimental results and discussions) rather than a large neighborhood.

Figure 10. The performance of LIDFS with different choices of kk on different datasets and metrics for the full (100%) selection scenario. Smaller values of kk show a better overall performance.

Figure 11. The performance of LIDFS with different choices of kk on different datasets and metrics for the aggressive (10%) selection scenario. Smaller values of kk show a better overall performance.

Figure 12. Average ranks across the metrics for both scenarios. The preference for small neighborhoods is strongest for AUC and ACC; NMI and CLSACC are comparatively flat.

Summary The sensitivity analysis was conducted with all classifiers and shows the same trend as for the Random Forest classifier, where small values of kk, particularly k=5k = 5 and k=10k = 10, perform comparably, and larger values lead to a slight degradation in performance. Taken together, the results confirm the theoretical trade-off motivating this analysis: larger kk is expected to improve the fidelity of the LID estimate itself, yet this improvement does not transfer into better downstream performance; on average the coarser local granularity mildly degrades the four metrics we track. A small-to-moderate neighborhood size is therefore the recommended default, with per-dataset tuning of kk available as an option when the observed heterogeneity across datasets justifies the additional cost.

Conclusion

We proposed a novel unsupervised feature selection algorithm based on LID to compute feature importance values and select informative features without requiring labeled data. We theoretically motivated LID and its suitability for the unsupervised feature selection task, and provided a brief but comprehensive review of progress in the field. We selected a large number of datasets to comprehensively and thoroughly evaluate the proposed method against classic and state-of-the-art algorithms, including a random baseline to further guide the evaluation. Finally, we conducted an in-depth runtime analysis to compare the computational efficiency of the different unsupervised feature selection algorithms considered, and a sensitivity analysis of the neighborhood size kk.

The experimental results show that the proposed unsupervised feature selection method, LIDFS, offers the best overall observed performance against all competitors, outperforming competitors in most cases and remaining highly competitive in the rest. The superiority of LIDFS, while not statistically significant, is remarkable in particular as LIDFS shows a better computational complexity in comparison with the closest competitors (e.g., SCFS). These observations highlight the effectiveness and efficiency of LIDFS in conducting a truly unsupervised feature selection with high performance figures and low computational complexity, without using limited or partial information from the labels like some competitors (e.g., SCFS, MCFS) do.

Discussion

LIDFS differs from most other methods in the literature in the sense that it uses neither information-theoretic nor spectral measures to select features. Instead, it directly assesses the discriminability of features via the estimation of LID. One major advantage of LID is that, aside from continuity, it requires no knowledge or assumptions concerning the distribution of the data. The main drawback is that LIDFS requires a relatively smooth distribution of distance values, which is more likely to occur when the data are dense, high-dimensional, or both. Nevertheless, such settings are those in which classical feature selection methods often fail, either due to high computational complexity or unstable performance, sometimes falling even below the random baseline.

The underlying logic of LIDFS is to assign greater importance, via an inverse relationship with the LID estimate, to features that reflect a lower-dimensional local structure relative to others, and therefore exhibit higher discriminability. It also includes a pruning step that prevents the selection of too many mutually close features, maintaining low redundancy in the selection process. As shown in the experimental results, this discrimination between features provides a systematic approach to identifying redundancy, an ability tied to the neighborhood size kk used when computing LID. This orientation toward low-dimensional local structure is reflected in LIDFS’s strong performance relative to the other algorithms in the experiments.

Future work

As data volume increases, the cost of building an exact index becomes prohibitive, motivating the use of approximate nearest-neighbor algorithms such as ϵ\epsilon-NNS [2] or NN-Descent [53] in place of exact search. Using approximate nearest-neighbor distances may adversely affect LID estimation and, consequently, alter the feature ranking; however, trading off feature quality for computation time can be advantageous when the dataset is large or high-dimensional.

One possible extension of this approach is to consider other models of intrinsic dimensionality as an alternative to LID for assessing feature quality. Although such models may lack the theoretical guarantees regarding the discriminability of distance measures, or may have higher computational complexity or greater sensitivity to noise [15], they may still merit consideration, particularly if they prove more reliable for non-smooth distance distributions or for smaller data samples.

There are several other possible directions for future work on ID-based feature selection. First, it would be interesting to consider the problem of unsupervised tuning of the parameters of ID-based feature selection. Also, our approach to unsupervised feature ranking could be extended to supervised applications by restricting the ID estimation to training examples with common labels. Finally, the algorithmic design of LIDFS could be revisited to yield a more granular feature-importance calculation, either through a refinement step that avoids assigning similar importance values within the same neighborhood, or through other approaches independent of kk. This, however, must be done with careful consideration to maintain the low computational overhead of the current unsupervised feature selection process.

Acknowledgments

This study was funded by the Innovation Fund Denmark project “PREPARE: Personalized Risk Estimation and Prevention of Cardiovascular Disease”.

Supplementary results

This appendix presents the results that are not included in the main text. It consists of two parts. Section A.1 presents the critical difference diagrams of the supervised metrics (ACC and AUC) for the KNN, decision tree, SVM, and XGBoost classifiers, and Section A.2 presents the performance plots on all 23 benchmark datasets. All supervised results are reported for each of the five downstream classifiers separately, whereas the unsupervised metrics, namely clustering accuracy (CLSACC) and NMI, do not depend on the classifier and are therefore reported once. The results for the random forest classifier and for the unsupervised metrics are presented in the main text, and they are not repeated here.

Critical difference diagrams

The critical difference diagrams are computed in the same way as in the main text, i.e., over all 23 benchmark datasets based on the average performance measured by the FSDEM score, using Wilcoxon-Holm with a significance level of 0.2. The diagrams of the random forest classifier and of the unsupervised metrics for the first 10% and for the entire range of the number of selected features are presented in Figure 8 and Figure 6 of the main text, respectively. Each figure below contains the diagrams of ACC (left) and AUC (right) for one classifier and one range, and Table A1 provides an index of all figures, where each entry links to the corresponding figure.

Classifier / metrics0.5% to 10%5% to 100%
KNN (ACC, AUC)A1A2
Decision tree (ACC, AUC)A3A4
SVM (ACC, AUC)A5A6
Random forest (ACC, AUC)8{sup}†`6{sup}†`
XGBoost (ACC, AUC)A7A8
CLSACC, NMI (classifier independent)8{sup}†`6{sup}†`

Table A1. Index of the supplementary critical difference diagrams. Each entry gives the number of the figure with the diagrams of ACC and AUC for the corresponding classifier and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.

Figure A1. Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (0.5% to 10% of the features).

↑ Back to Table A1

Figure A2. Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (5% to 100% of the features).

↑ Back to Table A1

Figure A3. Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (0.5% to 10% of the features).

↑ Back to Table A1

Figure A4. Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (5% to 100% of the features).

↑ Back to Table A1

Figure A5. Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (0.5% to 10% of the features).

↑ Back to Table A1

Figure A6. Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (5% to 100% of the features).

↑ Back to Table A1

Figure A7. Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (0.5% to 10% of the features).

↑ Back to Table A1

Figure A8. Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (5% to 100% of the features).

↑ Back to Table A1

Performance plots

This part presents the performance plots on all 23 benchmark datasets and for both experimental settings, in which the number of selected features ranges from 0.5% to 10% and from 5% to 100% of the features, respectively. The unsupervised metrics (CLSACC and NMI) are reported once for each dataset, whereas the supervised metrics (ACC and AUC) are reported separately for each of the five classifiers, i.e., KNN, decision tree, SVM, random forest, and XGBoost. In all plots, the random baseline is the average of 100 independent runs, and the shaded area denotes one standard deviation. The results on the Isolet dataset for the random forest classifier and for the unsupervised metrics are presented in Figure 4 and Figure 3 of the main text for the first 10% and for the entire range of the number of selected features, respectively, and are not repeated here. Table A2 provides an index of all plots, where each entry links to the corresponding figure.

DatasetCLSACC, NMI 10%CLSACC, NMI 100%KNN 10%KNN 100%Decision tree 10%Decision tree 100%SVM 10%SVM 100%Random forest 10%Random forest 100%XGBoost 10%XGBoost 100%
ALLAMLA9A31A53A76A99A122A145A168A191A213A235A258
arceneA10A32A54A77A100A123A146A169A192A214A236A259
CarcinomA11A33A55A78A101A124A147A170A193A215A237A260
CLL_SUB_111A12A34A56A79A102A125A148A171A194A216A238A261
COIL20A13A35A57A80A103A126A149A172A195A217A239A262
colonA14A36A58A81A104A127A150A173A196A218A240A263
gisetteA15A37A59A82A105A128A151A174A197A219A241A264
GLI_85A16A38A60A83A106A129A152A175A198A220A242A265
GLIOMAA17A39A61A84A107A130A153A176A199A221A243A266
Isolet4†4^{\dagger}3†3^{\dagger}A62A85A108A131A154A1774†4^{\dagger}3†3^{\dagger}A244A267
leukemiaA18A40A63A86A109A132A155A178A200A222A245A268
lungA19A41A64A87A110A133A156A179A201A223A246A269
lung_discreteA20A42A65A88A111A134A157A180A202A224A247A270
madelonA21A43A66A89A112A135A158A181A203A225A248A271
ORLA22A44A67A90A113A136A159A182A204A226A249A272
orlraws10PA23A45A68A91A114A137A160A183A205A227A250A273
pixraw10PA24A46A69A92A115A138A161A184A206A228A251A274
Prostate_GEA25A47A70A93A116A139A162A185A207A229A252A275
SMK_CAN_187A26A48A71A94A117A140A163A186A208A230A253A276
TOX_171A27A49A72A95A118A141A164A187A209A231A254A277
warpAR10PA28A50A73A96A119A142A165A188A210A232A255A278
warpPIE10PA29A51A74A97A120A143A166A189A211A233A256A279
YaleA30A52A75A98A121A144A167A190A212A234A257A280

Table A2. Index of the supplementary performance plots. Each entry gives the number of the figure for the corresponding dataset, metric group or classifier, and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.

Figure A9. CLSACC and NMI on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A10. CLSACC and NMI on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A11. CLSACC and NMI on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A12. CLSACC and NMI on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A13. CLSACC and NMI on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A14. CLSACC and NMI on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A15. CLSACC and NMI on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A16. CLSACC and NMI on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A17. CLSACC and NMI on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A18. CLSACC and NMI on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A19. CLSACC and NMI on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A20. CLSACC and NMI on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A21. CLSACC and NMI on the madelon dataset (0.5% to 10% of the features).

↑ Back to tbl:2

Figure A22. CLSACC and NMI on the ORL dataset (0.5% to 10% of the features).

↑ Back to tbl:2

Figure A23. CLSACC and NMI on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to tbl:2

Figure A24. CLSACC and NMI on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A25. CLSACC and NMI on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A26. CLSACC and NMI on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A27. CLSACC and NMI on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A28. CLSACC and NMI on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A29. CLSACC and NMI on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A30. CLSACC and NMI on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Figure A31. CLSACC and NMI on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A32. CLSACC and NMI on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A33. CLSACC and NMI on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A34. CLSACC and NMI on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A35. CLSACC and NMI on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A36. CLSACC and NMI on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A37. CLSACC and NMI on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Figure A38. CLSACC and NMI on the GLI_85 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A39. CLSACC and NMI on the GLIOMA dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A40. CLSACC and NMI on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A41. CLSACC and NMI on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A42. CLSACC and NMI on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A43. CLSACC and NMI on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A44. CLSACC and NMI on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A45. CLSACC and NMI on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A46. CLSACC and NMI on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A47. CLSACC and NMI on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A48. CLSACC and NMI on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A49. CLSACC and NMI on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A50. CLSACC and NMI on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A51. CLSACC and NMI on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A52. CLSACC and NMI on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A53. ACC and AUC of the KNN classifier on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A54. ACC and AUC of the KNN classifier on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A55. ACC and AUC of the KNN classifier on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A56. ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A57. ACC and AUC of the KNN classifier on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A58. ACC and AUC of the KNN classifier on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A59. ACC and AUC of the KNN classifier on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A60. ACC and AUC of the KNN classifier on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A61. ACC and AUC of the KNN classifier on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A62. ACC and AUC of the KNN classifier on the Isolet dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A63. ACC and AUC of the KNN classifier on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A64. ACC and AUC of the KNN classifier on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A65. ACC and AUC of the KNN classifier on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A66. ACC and AUC of the KNN classifier on the madelon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A67. ACC and AUC of the KNN classifier on the ORL dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A68. ACC and AUC of the KNN classifier on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A69. ACC and AUC of the KNN classifier on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A70. ACC and AUC of the KNN classifier on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A71. ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A72. ACC and AUC of the KNN classifier on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A73. ACC and AUC of the KNN classifier on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A74. ACC and AUC of the KNN classifier on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A75. ACC and AUC of the KNN classifier on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A76. ACC and AUC of the KNN classifier on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A77. ACC and AUC of the KNN classifier on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A78. ACC and AUC of the KNN classifier on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A79. ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A80. ACC and AUC of the KNN classifier on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A81. ACC and AUC of the KNN classifier on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A82. ACC and AUC of the KNN classifier on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A83. ACC and AUC of the KNN classifier on the GLI_85 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A84. ACC and AUC of the KNN classifier on the GLIOMA dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A85. ACC and AUC of the KNN classifier on the Isolet dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A86. ACC and AUC of the KNN classifier on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A87. ACC and AUC of the KNN classifier on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A88. ACC and AUC of the KNN classifier on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A89. ACC and AUC of the KNN classifier on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A90. ACC and AUC of the KNN classifier on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A91. ACC and AUC of the KNN classifier on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A92. ACC and AUC of the KNN classifier on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A93. ACC and AUC of the KNN classifier on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A94. ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A95. ACC and AUC of the KNN classifier on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A96. ACC and AUC of the KNN classifier on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A97. ACC and AUC of the KNN classifier on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A98. ACC and AUC of the KNN classifier on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A99. ACC and AUC of the decision tree classifier on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A100. ACC and AUC of the decision tree classifier on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A101. ACC and AUC of the decision tree classifier on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A102. ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A103. ACC and AUC of the decision tree classifier on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A104. ACC and AUC of the decision tree classifier on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A105. ACC and AUC of the decision tree classifier on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A106. ACC and AUC of the decision tree classifier on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A107. ACC and AUC of the decision tree classifier on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A108. ACC and AUC of the decision tree classifier on the Isolet dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A109. ACC and AUC of the decision tree classifier on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A110. ACC and AUC of the decision tree classifier on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A111. ACC and AUC of the decision tree classifier on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A112. ACC and AUC of the decision tree classifier on the madelon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A113. ACC and AUC of the decision tree classifier on the ORL dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A114. ACC and AUC of the decision tree classifier on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A115. ACC and AUC of the decision tree classifier on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A116. ACC and AUC of the decision tree classifier on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A117. ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A118. ACC and AUC of the decision tree classifier on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A119. ACC and AUC of the decision tree classifier on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A120. ACC and AUC of the decision tree classifier on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A121. ACC and AUC of the decision tree classifier on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A122. ACC and AUC of the decision tree classifier on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A123. ACC and AUC of the decision tree classifier on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A124. ACC and AUC of the decision tree classifier on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A125. ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A126. ACC and AUC of the decision tree classifier on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A127. ACC and AUC of the decision tree classifier on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A128. ACC and AUC of the decision tree classifier on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A129. ACC and AUC of the decision tree classifier on the GLI_85 dataset (5% to 100% of the features).

Image not extracted

Figure A130. ACC and AUC of the decision tree classifier on the GLIOMA dataset (5% to 100% of the features).

Image not extracted

Figure A131. ACC and AUC of the decision tree classifier on the Isolet dataset (5% to 100% of the features).

Image not extracted

Figure A132. ACC and AUC of the decision tree classifier on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A133. ACC and AUC of the decision tree classifier on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A134. ACC and AUC of the decision tree classifier on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A135. ACC and AUC of the decision tree classifier on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A136. ACC and AUC of the decision tree classifier on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A137. ACC and AUC of the decision tree classifier on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A138. ACC and AUC of the decision tree classifier on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A139. ACC and AUC of the decision tree classifier on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A140. ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A141. ACC and AUC of the decision tree classifier on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A142. ACC and AUC of the decision tree classifier on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A143. ACC and AUC of the decision tree classifier on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A144. ACC and AUC of the decision tree classifier on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A145. ACC and AUC of the SVM classifier on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A146. ACC and AUC of the SVM classifier on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A147. ACC and AUC of the SVM classifier on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A148. ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A149. ACC and AUC of the SVM classifier on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A150. ACC and AUC of the SVM classifier on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A151. ACC and AUC of the SVM classifier on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A152. ACC and AUC of the SVM classifier on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A153. ACC and AUC of the SVM classifier on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A154. ACC and AUC of the SVM classifier on the Isolet dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A155. ACC and AUC of the SVM classifier on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A156. ACC and AUC of the SVM classifier on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A157. ACC and AUC of the SVM classifier on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A158. ACC and AUC of the SVM classifier on the madelon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A159. ACC and AUC of the SVM classifier on the ORL dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A160. ACC and AUC of the SVM classifier on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A161. ACC and AUC of the SVM classifier on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A162. ACC and AUC of the SVM classifier on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A163. ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A164. ACC and AUC of the SVM classifier on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A165. ACC and AUC of the SVM classifier on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A166. ACC and AUC of the SVM classifier on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A167. ACC and AUC of the SVM classifier on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A168. ACC and AUC of the SVM classifier on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A169. ACC and AUC of the SVM classifier on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A170. ACC and AUC of the SVM classifier on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A171. ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A172. ACC and AUC of the SVM classifier on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A173. ACC and AUC of the SVM classifier on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A174. ACC and AUC of the SVM classifier on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A175. ACC and AUC of the SVM classifier on the GLI_85 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A176. ACC and AUC of the SVM classifier on the GLIOMA dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A177. ACC and AUC of the SVM classifier on the Isolet dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A178. ACC and AUC of the SVM classifier on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A179. ACC and AUC of the SVM classifier on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A180. ACC and AUC of the SVM classifier on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A181. ACC and AUC of the SVM classifier on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A182. ACC and AUC of the SVM classifier on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A183. ACC and AUC of the SVM classifier on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A184. ACC and AUC of the SVM classifier on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A185. ACC and AUC of the SVM classifier on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A186. ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A187. ACC and AUC of the SVM classifier on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A188. ACC and AUC of the SVM classifier on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A189. ACC and AUC of the SVM classifier on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A190. ACC and AUC of the SVM classifier on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A191. ACC and AUC of the random forest classifier on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A192. ACC and AUC of the random forest classifier on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A193. ACC and AUC of the random forest classifier on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A194. ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A195. ACC and AUC of the random forest classifier on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A196. ACC and AUC of the random forest classifier on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A197. ACC and AUC of the random forest classifier on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A198. ACC and AUC of the random forest classifier on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A199. ACC and AUC of the random forest classifier on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A200. ACC and AUC of the random forest classifier on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A201. ACC and AUC of the random forest classifier on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A202. ACC and AUC of the random forest classifier on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A203. ACC and AUC of the random forest classifier on the madelon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A204. ACC and AUC of the random forest classifier on the ORL dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A205. ACC and AUC of the random forest classifier on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A206. ACC and AUC of the random forest classifier on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A207. ACC and AUC of the random forest classifier on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A208. ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A209. ACC and AUC of the random forest classifier on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A210. ACC and AUC of the random forest classifier on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A211. ACC and AUC of the random forest classifier on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A212. ACC and AUC of the random forest classifier on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A213. ACC and AUC of the random forest classifier on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A214. ACC and AUC of the random forest classifier on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A215. ACC and AUC of the random forest classifier on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A216. ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A217. ACC and AUC of the random forest classifier on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A218. ACC and AUC of the random forest classifier on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A219. ACC and AUC of the random forest classifier on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A220. ACC and AUC of the random forest classifier on the GLI_85 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A221. ACC and AUC of the random forest classifier on the GLIOMA dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A222. ACC and AUC of the random forest classifier on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A223. ACC and AUC of the random forest classifier on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A224. ACC and AUC of the random forest classifier on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A225. ACC and AUC of the random forest classifier on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A226. ACC and AUC of the random forest classifier on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A227. ACC and AUC of the random forest classifier on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A228. ACC and AUC of the random forest classifier on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A229. ACC and AUC of the random forest classifier on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A230. ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A231. ACC and AUC of the random forest classifier on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A232. ACC and AUC of the random forest classifier on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A233. ACC and AUC of the random forest classifier on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A234. ACC and AUC of the random forest classifier on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A235. ACC and AUC of the XGBoost classifier on the ALLAML dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A236. ACC and AUC of the XGBoost classifier on the arcene dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A237. ACC and AUC of the XGBoost classifier on the Carcinom dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A238. ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A239. ACC and AUC of the XGBoost classifier on the COIL20 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A240. ACC and AUC of the XGBoost classifier on the colon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A241. ACC and AUC of the XGBoost classifier on the gisette dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A242. ACC and AUC of the XGBoost classifier on the GLI_85 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A243. ACC and AUC of the XGBoost classifier on the GLIOMA dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A244. ACC and AUC of the XGBoost classifier on the Isolet dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A245. ACC and AUC of the XGBoost classifier on the leukemia dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A246. ACC and AUC of the XGBoost classifier on the lung dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A247. ACC and AUC of the XGBoost classifier on the lung_discrete dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A248. ACC and AUC of the XGBoost classifier on the madelon dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A249. ACC and AUC of the XGBoost classifier on the ORL dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A250. ACC and AUC of the XGBoost classifier on the orlraws10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A251. ACC and AUC of the XGBoost classifier on the pixraw10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A252. ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A253. ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A254. ACC and AUC of the XGBoost classifier on the TOX_171 dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A255. ACC and AUC of the XGBoost classifier on the warpAR10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A256. ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A257. ACC and AUC of the XGBoost classifier on the Yale dataset (0.5% to 10% of the features).

↑ Back to Table A2

Image not extracted

Figure A258. ACC and AUC of the XGBoost classifier on the ALLAML dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A259. ACC and AUC of the XGBoost classifier on the arcene dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A260. ACC and AUC of the XGBoost classifier on the Carcinom dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A261. ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A262. ACC and AUC of the XGBoost classifier on the COIL20 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A263. ACC and AUC of the XGBoost classifier on the colon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A264. ACC and AUC of the XGBoost classifier on the gisette dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A265. Figure A265: ACC and AUC of the XGBoost classifier on the GLI_85 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A266. Figure A266: ACC and AUC of the XGBoost classifier on the GLIOMA dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A267. Figure A267: ACC and AUC of the XGBoost classifier on the Isolet dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A268. ACC and AUC of the XGBoost classifier on the leukemia dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A269. ACC and AUC of the XGBoost classifier on the lung dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A270. ACC and AUC of the XGBoost classifier on the lung_discrete dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A271. ACC and AUC of the XGBoost classifier on the madelon dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A272. ACC and AUC of the XGBoost classifier on the ORL dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A273. ACC and AUC of the XGBoost classifier on the orlraws10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A274. ACC and AUC of the XGBoost classifier on the pixraw10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A275. ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A276. ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A277. ACC and AUC of the XGBoost classifier on the TOX_171 dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A278. ACC and AUC of the XGBoost classifier on the warpAR10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A279. ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (5% to 100% of the features).

↑ Back to Table A2

Image not extracted

Figure A280. ACC and AUC of the XGBoost classifier on the Yale dataset (5% to 100% of the features).

↑ Back to Table A2

References

  1. [1]I. Guyon, A. Elisseeff, An introduction to variable and feature selection, JMLR 3 (2003).
  2. [2]P. Indyk, R. Motwani, Approximate nearest neighbors: towards removing the curse of dimensionality, in: STOC, ACM, 1998.
  3. [3]M. Rajabinasab, A. D. Lautrup, T. Hyrup, A. Zimek, A dynamic evaluation metric for feature selection, in: Similarity Search and Applications, Springer Nature Switzerland, Cham, 2025.
  4. [4]F. Camastra, Data dimensionality estimation methods: a survey, Pattern Recognition 36 (12) (2003).DOI
  5. [5]S. Niijima, Y. Okuno, Laplacian linear discriminant analysis approach to unsupervised feature selection, TCBB 6 (4) (2009).DOI
  6. [6]Y. Saeys, I. Inza, P. Larrañaga, A review of feature selection techniques in bioinformatics, Bioinformatics 23 (19) (2007).pubmed.ncbi.nlm.nih.gov/17720704
  7. [7]A.-C. Hayur, P. Gestraud, J.-P. Vert, The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures, PloS One 6 (12) (2011).
  8. [8]J. Feng, H. Xu, S. Yan, Robust PCA in high-dimension: A deterministic approach, in: ICML, 2012.arxiv.org/abs/1206.4628
  9. [9]R. Wald, T. Khoshgoftaar, A. Napatoliano, The importance of performance metrics within wrapper feature selection, in: IRI, IEEE, 2013.DOI
  10. [10]X. He, M. Ji, C. Zhang, H. Bao, A variance minimization criterion for feature selection using laplacian regularization, PAMI 33 (10) (2011).DOI
  11. [11]B. Wang, A. Golderberg, Gradient-based laplacian feature selection, CoRR abs/1404.2948 (2014).arxiv.org/abs/1404.2948
  12. [12]Z. Zhao, H. Liu, Spectral feature selection for supervised and unsupervised learning, in: ICML, ACM, 2007.DOI
  13. [13]X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, in: NIPS, 2005.
  14. [14]D. Cai, C. Zhang, X. He, Unsupervised feature selection for multi-cluster data, in: KDD, ACM, 2010.DOI
  15. [15]L. Amsalge, O. Chelly, T. Furon, S. Girard, M. E. Houle, K. Kawarabayshi, M. Nett, Estimating local intrinsic dimensionality, in: KDD, ACM, 2015.
  16. [16]M. E. Houle, Dimensionality, Discriminability, Density & Distance Distributions, in: ICDMW, IEEE, 2013.DOI
  17. [17]M. E. Houle, Local intrinsic dimensionality I: An extreme-value-theoretic foundation for similarity applications, in: SISAP, 2017.DOI
  18. [18]H. Peng, F. Long, C. Ding, Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, PAMI 27 (8) (2005).pubmed.ncbi.nlm.nih.gov/16119262
  19. [19]T. M. Cover, J. A. Thomas, Elements of Information Theory, 1991.
  20. [20]M. A. Hall, L. A. Smith, Feature selection for machine learning: Comparing a correlation-based filter approach to the wrapper, in: FLAIRS Conference, Vol. 1999, 1999.
  21. [21]H. Liu, R. Setiono, Chi2: Feature selection and discretization of numeric attributes, in: TAI, IEEE, 1995.
  22. [22]R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, 2001.
  23. [23]F. Nie, W. Zhu, X. Li, Unsupervised feature selection with structured graph optimization, in: AAAI, AAAI Press, 2016, pp. 1302–1308.DOI
  24. [24]C. Wang, J. Wang, Z. Gu, J.-M. Wei, J. Liu, Unsupervised feature selection by learning exponential weights, Pattern Recognition 148 (2024). doi:https://doi.org/10.1016/j.patcog.2023.110183.
  25. [25]M. G. Parza, H. Zare, M. Ghatee, Unsupervised feature selection based on adaptive similarity learning and subspace clustering, Eng. Appl. Artif. Intell. 95 (2020). doi:10.1016/J.ENGAPPAI.2020.103855.
  26. [26]S. Karami, F. Saberi-Movahhed, P. Tiwari, P. Marttinen, S. Vahdati, Unsupervised feature selection based on variance-covariance subspace distance, Neural Networks 166 (2023). doi:10.1016/J.NEUNET.2023.06.018.DOI
  27. [27]X. Zhang, R. C. de Amorim, Scalable unsupervised feature selection via weight stability (2025). arXiv:2506.06114.DOI
  28. [28]K. Jiang, T. Cao, L. Zhu, Q. Sun, Adaptive and flexible ℓ₁-norm graph embedding for unsupervised feature selection, Applied Intelligence 54 (22) (2024). doi:10.1007/s10489-024-05760-z.
  29. [29]Z. Cao, X. Xie, Y. Li, Multi-view unsupervised feature selection with consensus partition and diverse graph, Information Sciences 661 (2024).DOI
  30. [30]J.-S. Wu, Y. Li, J.-X. Gong, W. Min, Collaborative and discriminative subspace learning for unsupervised multi-view feature selection, Engineering Applications of Artificial Intelligence 133 (2024).DOI
  31. [31]B. B. Mandelbrot, How long is the coast of britain?, Science 156 (3775) (1967).
  32. [32]T. W. Gamelin, R. E. Greene, Introduction to Topology: Second Edition, Courier Dover Publications, 2013.
  33. [33]M. E. Houle, H. Kashima, M. Nett, Generalized expansion dimension, in: ICDMW, IEEE, 2012.DOI
  34. [34]D. R. Karger, M. Ruhl, Finding nearest neighbors in growth-restricted metrics, in: STOC, ACM, 2002.
  35. [35]M. E. Houle, M. Nett, Rank cover trees for nearest neighbor search, in: SISAP, Springer, 2013.DOI
  36. [36]M. E. Houle, X. Ma, M. Nett, V. Oria, Dimensional testing for multi-step similarity search, in: ICDM, IEEE, 2012.DOI
  37. [37]G. Casanova, E. Englmeier, M. E. Houle, P. Kröger, M. Nett, E. Schubert, A. Zimek, Dimensional testing for reverse k-nearest neighbor search, Proc. VLDB Endow. 10 (7) (2017).DOI
  38. [38]T. de Vries, S. Chawla, M. E. Houle, Density-preserving projections for large-scale local anomaly detection, Knowledge and Information Systems 32 (6) (2012).DOI
  39. [39]A. Anderberg, J. Bailey, R. J. G. B. Campello, M. E. Houle, H. O. Marques, M. Radovanovic, A. Zimek, Dimensionality-aware outlier detection, in: SIAM International Conference on Data Mining, 2024.
  40. [40]X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, Advances in Neural Information Processing Systems 18, section 2 explicitly describes "Variance" as the simplest unsupervised criterion: select features with largest variance (2005).
  41. [41]M. A. Hall, Correlation-based feature selection for machine learningPhD thesis, University of Waikato. Originally supervised, but the correlation part is frequently adapted to unsupervised scenarios (1999).
  42. [42]P. Mitra, C. A. Murty, S. K. Pal, Unsupervised feature selection using feature similarity, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (3) (2002). doi:10.1109/34.990132.
  43. [43]H. W. Kuhn, The Hungarian Method for the assignment problem, Naval Research Logistics Quarterly 2 (1-2) (1955).
  44. [44]J. Li, K. Cheng, S. Wang, F. Morstatter, R. P. Trevino, J. Tang, H. Liu, Feature selection: A data perspective (2017).DOI
  45. [45]M. Rajabinasab, A. Zimek, FSEVAL: feature selection evaluation toolbox and dashboard, CoRR abs/2604.18227 (2026). arXiv:2604.18227, doi:10.48550/ARXIV.2604.18227.DOI
  46. [46]T. M. Cover, P. E. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13 (1) (1967).DOI
  47. [47]L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification and Regression Trees, Wadsworth, 1984.DOI
  48. [48]C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (3) (1995).DOI
  49. [49]L. Breiman, Random Forests, Machine Learning 45 (1) (2001).
  50. [50]T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2016.DOI
  51. [51]M. Rajabinasab, M. E. Houle, O. Chelly, A. Zimek, Worse than random: The importance of a baseline for unsupervised feature selection – extended version, CoRR abs/2605.22973 (2026). arXiv:2605.22973, doi:10.48550/ARXIV.2605.22973.DOI
  52. [52]J. Demšar, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 7 (2006).DOI
  53. [53]W. Dong, C. Moses, K. Li, Efficient k-nearest neighbor graph construction for generic similarity measures, in: WWW, ACM, 2011.DOI

Paper details

vlnc:W_cna68qqn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents