WorksComputer Science
The Double Entropy Fit Index (DEFI): An Information-Theoretic Approach to Dimensionality Reduction and Clustering
Abstract
We introduce the Double-Entropy Fit Index (DEFI), a regularized information-theoretic criterion for selecting among candidate graph partitions and reduced representations. DEFI couples entropy-based compression with relevance preservation and penalizes model dimensionality, cluster imbalance, and partition fragmentation. An exact Rényi-2 entropy identity shows that partitioning alters both spectral concentration and graph-volume normalization, explaining why unregularized entropy minimization can favor degenerate solutions. Across a 243-condition population-analytic factor recovery benchmark, DEFI achieved an overall exact recovery rate of 97.2% (versus 75.0% for EGA-TMFG, 45.8% for TEFI on von Neumann entropy). In controlled local candidate batteries (distinguishing the true model from adjacent and factor-split alternatives), DEFI identified the correct partition in 99.2% of replications. Across ten clustering benchmarks, DEFI-guided Gaussian mixtures achieved a mean ARI of 0.929, exceeding HDBSCAN (0.809), Ward clustering (0.700), and conventional GMM criteria (0.501-0.580), with gains concentrating where standard criteria over-extract components.
Introduction
Complex data in operations, engineering, and behavioral sciences are increasingly analyzed through the lens of network structure: variables or observations are embedded in a graph, and the graph’s communities are interpreted as latent dimensions of the system. In psychometrics, exploratory graph analysis (EGA) operationalizes this idea by treating communities in a partial-correlation network as latent factors [14]; the same reduction of a high-dimensional system to a small number of internally coherent communities underlies dimensionality reduction pipelines across machine learning and operations research [10, 16]. In all of these settings, a single question emerges: among candidate partitions of the data, which one identifies the latent structure rather than an artifact of the fit criterion?
This question is relevant because unregularized entropic criteria improve monotonically under partition refinement, favoring degenerate singletons or splinter clusters unless model complexity is priced explicitly [18]. Existing remedies introduce additional operational constraints: network-based entropy criteria such as the von Neumann total entropy fit index (TEFI.vn) [13] rely on inverse-covariance estimators [9, 20] that become ill-conditioned under factor collinearity and are difficult to apply on disconnected or isolated-node graph structures, while classical mixture criteria (BIC, ICL) continue to reward spurious components on complex non-Gaussian geometries [3]. What is missing is a unified criterion that scores candidate partitions directly from the spectrum of an affinity graph while explicitly pricing community count, cardinality imbalance, and topological fragmentation. We introduce the Double-Entropy Fit Index (DEFI), a unified partition-selection criterion designed to address this methodological gap.
Related Literature
Four frames of literature informed the work on our proposed Information-Theoretic selection criterion, DEFI. Entropy-based assessment of latent structure descends from Shannon’s theory of communication [29] through Watanabe’s total correlation [37]. For a collection of observed random variables (e.g., manifest features or items), total correlation generalizes bivariate mutual information as the gap between the sum of marginal entropies and the joint entropy :
Additionally, Studený and Vejnarová showed that total correlation coincides with the multiinformation function: it is nonnegative and vanishes exactly under mutual independence [32]. Used directly as a partitioning objective, total correlation is monotone under refinement and is minimized by the trivial singleton partition; any usable criterion must therefore pair an entropic score with control over the number and shape of communities [18, 21].
Partitioning as structure discovery. A second line of work makes the partition itself the estimand. Compression-based formulations use minimum description length to choose the number and composition of communities without prespecification [5]. Spectral methods translate community detection into the eigenanalysis of Laplacian or modularity matrices [10, 23], and their detectability limits under planted models are well characterized [6, 1]. In psychometrics, exploratory graph analysis (EGA) estimates the number of latent dimensions by community detection on partial-correlation networks [14], and the entropy fit indices built on this machinery, most directly, the von Neumann total entropy fit index (TEFI.vn) which scores candidate variable partitions by system entropy [13]; such network-based estimates compare favorably with classical factor-retention rules in simulation [7]. These criteria remain tied to inverse-covariance network estimators [9, 20], whose computational cost and connectedness requirements are restrictive in high-dimensional or degenerate settings.
Entropy of the Laplacian state. DEFI computes entropy from the normalized combinatorial Laplacian , a density-matrix construction of [4] whose Rényi generalization is developed by [8]. Two findings from that literature inform the design. First, the Laplacian-state entropy is dominated by graph size and volume rather than by community structure, so it cannot be interpreted directly as a community descriptor [22]. Second, any partition-sensitive use of the entropy must be explicit about normalization: the exact Rényi-2 change induced by deleting cut edges decomposes into a degree-sensitive trace term and an opposing graph-volume term (§ 2).
Regularization and the information bottleneck. DEFI’s penalty grammar draws on classical model selection. The integrated completed likelihood augments a mixture likelihood with an entropy term that discourages degenerate assignments [3], and Bayesian Information Criterion (BIC) and Minimum Description Length (MDL) penalties price each additional community against sample size and dimension. The information-bottleneck principle formalizes the trade-off between compressing an input and preserving information about a relevance variable [35, 12, 28]; sufficient dimension reduction supplies the corresponding statistical framework for information-preserving reduction [2, 11].
Positioning. DEFI differs from this prior work in combination rather than in any single component. First, it evaluates a candidate partition through the Rényi entropy of the affinity-graph Laplacian state, which allows DEFI to avoid inverse-covariance estimation and tolerate disconnected graphs while pricing complexity, imbalance, and fragmentation explicitly. Second, it exposes one mixed-variable objective that a single derivative-free optimizer instantiates over mixture, hierarchical, and subspace candidate classes (§ 2). To the best of our knowledge, TEFI.vn is the closest published baseline and anchors our oracle and dimensionality comparisons.
Contributions and Outline
This paper introduces the Double-Entropy Fit Index (DEFI), a regularized information-theoretic criterion for selecting among candidate partitions and reduced representations. We make four primary contributions: (1) Unified Information-Theoretic Criterion: we formulate DEFI as a single objective coupling entropy-based compression, a self-supervised stability anchor via the deterministic information bottleneck, and structural regularizers for dimensionality, imbalance, and fragmentation, computed directly from the spectrum of an affinity Laplacian without inverse-covariance estimation (§ 2); (2) Exact Spectral Trace Decomposition: we prove that at Rényi order , the change in spectral entropy decomposes into a trace-concentration term and an opposing graph-volume normalization term, providing the first analytical explanation for degenerate unregularized cuts and mathematically justifying DEFI’s structural penalties (§ 3); (3) Optimization Framework for Mixed Partition Spaces: we solve the mixed discrete-continuous model-selection problem via continuous relaxation and CMA-ES, combining a piecewise linear homotopy over integer cluster counts with adaptive topological pre-conditioning and offline candidate pre-caching to eliminate step-discontinuity stagnation (§ 4); and (4) Empirical Validation: across a 243-condition factor-recovery benchmark, an exact enumeration across 700,074 structures, ten complex clustering geometries, and 26 high-dimensional manifold datasets, DEFI demonstrates superior recovery over classical information criteria, density methods, and factor retention rules (§ 5, § 6). Section 7 contextualizes these findings within competing literature and details operational boundaries. Complete mathematical proofs, simulation derivations, and a comprehensive symbol index appear in the Electronic Companion (Sections EC.§ 1–EC.8). Code and replication materials are publicly available at github will be available soon.
The DEFI Criterion
This section defines the Double-Entropy Fit Index (DEFI). A single principle organizes everything that follows: every experiment in this paper optimizes the same criterion, differing only in the candidate partition class and free parameter space (Table 1). Conceptually, DEFI couples four foundational components: (i) a spectral entropy term that measures affinity-graph concentration;
| Study / Focus | Data & Suite | DEFI Variant | Optimizer & Budget | Relevance Anchor | Primary Endpoints | Baselines |
| Study 1a Factor Recovery | 243-condition latent factor grid () | DEFI-FA | Frozen defaults (no search; Sec. 2) | Perturbed correlation matrix () | Exact recovery (%) | EGA-TMFG, PA, Kaiser, TEFI.vn, SEM fit (CFI/TLI) |
| Study 1b Finite Consistency | Stirling partition lattice () | DEFI-FA | Exact enumeration ( structures) | Perturbed correlation matrix () | Global argmin recovery (%) | Exhaustive search over all possible graphs based on the Stirling numbers |
| Study 2 Clustering | 10 FCPS & synthetic benchmarks (20 seeds) | DEFI-GMM DEFI-HC | CMA-ES continuous relaxation () | Feature Gaussian jitter view () | Adjusted Rand Index, , NMI, V-measure | GMM (BIC [3], ICL [27]), HDBSCAN [30], Ward [36], -means (silhouette) [24] |
| Study 3 Dimensionality | 26 UCI benchmarks & MNIST (20 seeds) | DEFI-DR | Dual-projection rank sweep () | Perturbed correlation matrix () | , Geo. stress, Var. (%) | Isomap, UMAP, Kernel PCA, PA, Kaiser, Sparse PCA, VAE |
| Study 4 Ablation | 6 representative topologies (20 seeds) | DEFI-GMM | Controlled ablation ( budget) | Feature Gaussian jitter view () | ARI, , Oracle gap () | DEFI-Fixed, Random search, DEFI (no smooth-) |
Table 1. Evaluation Design Matrix: Experimental protocols, search configurations, relevance anchors, and evaluation endpoints across Studies 1–4.
(ii) a partition entropy term that prices representation complexity;
(iii) a stability anchor derived from the deterministic information bottleneck; and
(iv) structural regularizers that penalize degenerate, fragmented, or topologically discordant solutions.
We formalize these components sequentially below (architectural overview in Figure 1), defining candidate partitions and the affinity-graph density state before assembling the global objective. A consolidated index of mathematical symbols, parameter domains, and operational definitions is cataloged in Section EC.8 of the Electronic Companion.

Figure 1. Operational architecture of the DEFI objective decomposition. Candidate partition , affinity geometry , data matrix , and self-supervised reference view map into four regularized processing pillars, aggregated into scalar objective and optimized via CMA-ES coordinate feedback.
Data, Partitions, and Candidate Classes
Let be a data matrix of observations and variables. A candidate partition is a deterministic assignment map
where is the candidate number of communities. For each community , its cardinality and proportion are given by:
When DEFI is applied to variables rather than observations (as in the factor-recovery studies), the domain of is , the denominator in (3) is , and the affinity graph is constructed on manifest features. A candidate class denotes the family of admissible partitions produced by a given generator, and is the bounded mixed discrete-continuous parameter space associated with that generator. In practice, may be induced by parametric mixtures, cuts of an agglomerative dendrogram, or rotated subspace projections, with capturing the corresponding structural and regularizer coordinates (Table 1).
Graph Construction and the Laplacian State
Each candidate configuration represents the data by a symmetric, nonnegative affinity matrix with . For a Gaussian-affinity graph,
with bandwidth scaled from the median positive pairwise distance; multiscale graphs evaluate convex combinations of affinities at bandwidths , , and . Let with denote the degree matrix, and the combinatorial Laplacian. Provided the graph contains at least one positive-weight edge, and the normalized density state is positive semidefinite with unit trace, forming a density matrix whose eigenvalues are nonnegative and sum to one [4, 8]. Because only is required, the state remains well-defined on disconnected graphs and isolated vertices, unlike inverse-covariance network estimators. When an input similarity is not guaranteed positive semidefinite, DEFI projects it by clipping negative eigenvalues at zero and renormalizing to unit trace.
Entropy Terms
For a density matrix and order , , the matrix Rényi entropy is
whose limit as is the von Neumann entropy (). In CMA-ES search, is optimized over ; candidates within of evaluate the von Neumann limit. The spectral entropy term compares the graph state before and after the partition is imposed. Given , let be the Laplacian obtained by deleting all inter-community edges, and let . Component (i) evaluates the signed entropy difference
Its exact decomposition, which motivates our structural regularizers, is derived in § 3. Component (ii), the partition entropy, prices representation complexity via the entropy of community proportions:
The Relevance View and the Information Bottleneck
Component (iii) grounds partition selection in the Information Bottleneck (IB) principle [35], balancing compression against relevance preservation. In our unsupervised setting, the relevance variable operationalizes a self-supervised cross-view stability anchor defined over the identical index set as , derived from an auxiliary perturbation of the observed system. Let denote the partitioned domain cardinality, corresponding to observation clustering () or variable partitioning (). For observation clustering, an auxiliary view is constructed via feature-scaled Gaussian perturbation , chosen to disrupt localized point density while preserving macroscopic cluster separation), where an initial unregularized baseline clustering on induces reference partition [19]. For variable partitioning, an independently perturbed correlation matrix yields reference assignment . The reference anchor is fixed prior to optimization and never derived from evaluation labels. Because candidate assignments are deterministic maps , assignments satisfy the Markov chain:
Mutual information between data and representation collapses to the partition entropy, , recovering the deterministic information bottleneck (DIB) [28] where compression is governed directly by . For candidate partition and reference assignment , the joint contingency is
Preserved relevance is scored by plug-in mutual information ():
The signed term in the global objective rewards cluster boundaries that persist under stochastic jitter, penalizing noise overfit.
The Criterion and its Structural Regularizers
Assembling components (i)–(iv) yields the DEFI criterion:
with nonnegative coefficients collected in along with , graph parameters, and generator choices. The first three terms represent components (i)–(iii). The remaining terms formalize the structural regularizers of component (iv):
Dimensionality penalties. BIC-type costs price candidate complexity against sample size and feature dimension: and .
Cluster balance floor. Normalizing community-size entropy yields , which attains 1 at uniform proportions. Given tolerance floor ,
penalizes only severe imbalance without punishing substantively plausible asymmetry.
Partition fragmentation penalty. Splinter communities below a minimum cardinality are priced by
Graph topological concordance. Disagreement with graph topology is penalized via the average normalized cut:
where and maximally penalizes zero-volume sets. The selected partition solves
using CMA-ES (§ 4). Because negative spectral differences and relevance preservation can drive below zero while penalties , parameter space is constrained to strictly positive penalty infima:
These positive floors ensure that search trajectories cannot evade structural regularization by collapsing weights to zero, forcing CMA-ES to adapt continuous kernel bandwidths and covariance geometries against active penalty boundaries.
Properties and Assumptions
Four structural properties govern this formulation: (i) only relative objective differences matter, so candidate rankings are invariant to constant additive shifts; (ii) all terms depend on strictly through the induced partition, ensuring invariance under community permutation; (iii) the balance regularizer is zero at uniform cluster proportions and strictly positive only below the floor ; and (iv) the signed spectral term requires structural regularization because, as proven in § 3, volume normalization can cause unregularized cuts to favor trivial partitions.
The implementation relies on four assumptions (theory-specific conditions for the internal-dominance branch and planted block models are stated with their respective theorems in § 3): (A1) Well-defined graph state: every decoded affinity graph contains at least one positive-weight edge and strictly positive bandwidths, with positive semidefinite graph states enforced via eigen-clipping; (A2) Admissible partitions: candidate partitions satisfy and the minimum-cluster cardinality constraint of (13), with numerically failed generator fits declared infeasible; (A3) Nontrivial relevance view: the reference configuration is not almost surely identical to and is independent of external evaluation labels; and (A4) Estimator comparability: all objective terms evaluate the identical observation set under uniform normalization conventions, and the parameter domain is bounded.
Theoretical Properties
This section states the analytical properties of the DEFI criterion that motivate its regularizers and justify its use as a selection rule. Full proofs, and additional inferential results for the fixed-partition coarse-graining appear in the electronic companion (EC.1, EC.2, EC.3, EC.4). Throughout, is a symmetric nonnegative affinity matrix with zero diagonal and at least one positive-weight edge, its combinatorial Laplacian, its volume, , and the graph state of Section 2.
Our first result is the exact decomposition of the spectral term (6) at order . Given a partition , delete its inter-community edges, and for each vertex let be the weighted cut incidence. Define the cut-volume and trace penalties
Theorem 1 (Exact Rényi-2 identity). Let be a partition whose deletion leaves at least one positive-weight edge, with and . Then the spectral term at is exactly
For unweighted simple graphs (), .
The two terms in (17) oppose each other: deleting edges reduces but also reduces the normalizing volume. Because is normalized by , neither effect can be neglected, and the second term rewards deletion per se. The consequence is a failure mode for unregularized spectral-entropy minimization.
Proposition 1 (Degeneracy of the unregularized criterion). There exist connected graphs and partitions with no community interpretation such that , where is the no-cut partition. Hence minimizing (6) alone can strictly prefer a nontrivial partition over , and minimizing it over unrestricted families is not equivalent to minimizing the cut or any degree-weighted cut. A three-vertex path with one end edge deleted already exhibits the strict inequality.
Proposition 1 is the analytical reason (11) prices imbalance, fragmentation, and graph disagreement rather than relying on the entropy difference alone. When the volume effect is held fixed, the criterion does reduce to a tractable combinatorial objective.
Theorem 2 (Fixed-cut-volume equivalence). On any admissible family with ,
Because is strictly increasing on , minimizing at fixed volume is equivalent to minimizing . The induced penalty couples cut edges sharing an incident node via the negative quadratic term , distinguishing it from a standard linear minimum cut.
The next results interpret the criterion under a planted model. The scope here is narrow: the analysis concerns a fixed partition and the binary edge-mass coarse-graining induced by it.
Theorem 3 (Population bound under the equal-sized Stochastic Block Model). Let be generated by the symmetric -community planted-partition model with equal community sizes , internal probability , external probability , and external-to-internal edge-opportunity ratio. Let be the population internal-edge mass, where and . For , on the internal-dominance branch (equivalent to ), and for any ,
where is the binary Rényi entropy and is its unique solution of on . Without the branch condition, low entropy is symmetric between internal and external dominance and (19) does not follow.
In words, Theorem 3 defines the sharp signal-to-noise ratio required for the planted partition to achieve an entropy deficit , proving that low spectral entropy guarantees cohesive community structure only along the internal-dominance branch.
Proposition 2 (Rényi–2 Bridge under the Planted Partition Model). Let be an unweighted simple graph generated from the symmetric -community stochastic block model (SBM) of Theorem 3, and let denote the planted partition. Define the empirical internal-edge mass and trace-penalty ratio by
Then the spectral entropy difference at order satisfies the exact finite-sample identity
Moreover, in the dense regime with fixed , where
Consequently,
Remark 1 (Role of the Spectral Term and Scale Calibration). Proposition 2 bridges the macroscopic edge-mass statistics of Theorems 3 and 4 with the spectral criterion of Theorem 1. Because holds identically on unweighted simple graphs, the volume-normalization term in is governed directly by the internal-edge mass . The asymptotic vanishing reflects the calibrated null behavior of the criterion evaluated strictly at the planted partition . Across the broader candidate space , candidate partitions that violate true community boundaries yield , producing non-vanishing spectral penalties that separate cohesive structures from arbitrary cuts. Furthermore, while the combinatorial regularizers depend solely on cluster cardinalities, serves as the primary coupling between the partition and the continuous affinity geometry . Finally, the asymptotic cancellation ensures that the spectral term operates on the same effective scale as the information-theoretic and structural regularizers. This stabilizes the objective where unregularized spectral differences flatten, preventing graph volume from overwhelming model-order selection while formally justifying the structural regularizers in (11).
Theorem 4 (Asymptotic distribution of the empirical entropy). Let be the empirical internal-edge mass in the model of Theorem 3, with and independent. If both count variances diverge and is bounded away from 0 and 1, then
for the explicit variance derived in EC.2. Consequently: (i) if , the plug-in entropy is asymptotically normal with mean and variance ; (ii) if eventually, the first-order limit degenerates to the second-order distribution, ; and (iii) in the dense regime , whereas under sparse scaling the convergence rate slows to . The balanced point is simultaneously the entropy maximum and the branch boundary, which is where the second-order limit applies. These statements concern inference for a fixed partition; if the same graph is used both to select a partition and to estimate its entropy, selection effects invalidate the fixed-partition variances.
The criterion (11) is a plug-in estimator of the population regularized deterministic information-bottleneck objective
where the compression term has been replaced by via the Markov chain condition in (8), and the spectral and structural terms are collected in .
Theorem 5 (Selection consistency over finite classes). Suppose that for every fixed admissible the entropy, relevance, and regularizer estimators in (11) converge in probability to their population counterparts. Then pointwise. If, in addition, the candidate class is finite and the population minimizer is unique and separated by a margin , then the empirical minimizer satisfies . In particular, for fixed , fixed , and a discretized , the full class of partitions into at most communities is finite, so the consistency statement covers the entire partition class, not only a hand-picked candidate set.
Two scope statements delimit Theorem 5. First, with a continuous , uniformity requires a uniform law of large numbers for the entropy and mutual-information estimators over , which we do not establish; the exact-enumeration study of Section 6 provides computational evidence for the discretized case. Second, consistency identifies the population minimizer of the regularized objective.
Remark 2 (Theoretical Scope and Boundaries). To prevent conflation between structural properties and generative recovery guarantees, we delineate three precise boundaries of the theoretical analysis: (i) Structural vs. Generative: Theorems 1 and 2 provide structural decompositions of the Rényi-2 spectral term on arbitrary graphs, identifying the volume-normalization artifact that necessitates structural regularization, rather than establishing minimax recovery rates under planted random graph models; (ii) Calibration vs. Detectability: Proposition 2 establishes asymptotic null calibration () at the true partition under dense block models, ensuring scale compatibility with complexity penalties rather than bounding sharp information-theoretic detectability thresholds; and (iii) Finite-Class vs. Model-Order Consistency: Theorem 5 guarantees selection consistency of the DEFI minimizer over finite, margin-separated candidate classes , but does not constitute an unconstrained proof of model-order consistency () over unbounded, continuous partition spaces. The empirical studies in Section 6 evaluate the operational recovery performance of the full pipeline where exact high-dimensional asymptotic analysis remains an open problem.
Optimization with the Covariance Matrix Adaptation Evolution Strategy
The model selection problem (15) couples discrete decisions (, covariance family ) with continuous coordinates . We embed these parameters into a bounded 15-dimensional surrogate domain solved via CMA-ES [15] (parameter bounds and encodings in Section FC.4). To mitigate discrete step discontinuities, cluster counts are relaxed to real coordinates , bounding integer floors , ceilings , and fractional weight . Denoting bounding partitions by and under continuous covariance coordinate , the objective is evaluated as the linear homotopy:
When is an integer (), (21) recovers DEFI identically. Between integer boundaries, (21) creates a continuous descent surface that prevents premature step-size deflation, while proposals with cluster sizes receive an infeasibility penalty .
Figure 2 illustrates this search trajectory on the TwoDiamonds topology. Even when initialized in an over-partitioned regime (), the adaptive covariance matrix contracts along continuous homotopy slopes, systematically shedding spurious sub-clusters until converging onto the ground truth (, ARI ).
Figure 2. CMA-ES optimization dynamics under continuous relaxation on the TwoDiamonds topology (). Top row: Evolution of the sampling distribution mean , adaptive covariance ellipse (dashed orange), and offspring proposals (black points) across the continuous surrogate landscape . Bottom row: Corresponding decoded cluster partitions and covariance geometries in sample space. The linear homotopy (21) provides continuous descent directions that steer the population from initial over-partitioning (, ARI ) to basin convergence (, ARI ) without integer step-discontinuity stagnation.
To prevent CMA-ES from allocating evaluation budget to geometrically incompatible covariance structures, an exploratory topological screening phase pre-conditions the search space before optimization begins. Point densities estimated from local -NN distances first yield a density-contrast ratio,
Pairing with an exploratory unregularized mixture fit classifies the data into one of three structural regimes (asymmetric manifold, equal-volume manifold, or standard heterogeneous), which initializes and prunes inadmissible covariance parameterizations from the candidate pool (decision rules detailed in Section EC.§ 4). Finally, this screening step calibrates surrogate-domain boundaries by enforcing and bounding the dimensionality-penalty ceiling by .
Computational efficiency is achieved by decoupling candidate generation from fitness evaluation. In an offline phase, Gaussian mixtures are fitted once across candidate configurations via Expectation-Maximization and cached in an in-memory table . During CMA-ES search, evaluating (21) requires an partition retrieval from followed by compiled C++ trace evaluations via RcppArmadillo. By exploiting the exact Rényi-2 trace identity of Theorem 1, evaluating and requires only sparse edge-cut traversals scaling as rather than dense matrix operations. Consequently, uncached evaluations execute in under 60 s, completing the entire 300-evaluation CMA-ES run within seconds.
Evaluation Design
The evaluation is organized so that each study isolates a unique property of DEFI distinct question. Study 1 removes search entirely and evaluates candidate structures directly at fixed default weights across both a 243-condition population-analytic factorial grid and the complete partition lattice of small instances. Study 2 attaches the CMA-ES optimizer and asks whether the full pipeline is competitive on observation-level clustering across heterogeneous benchmark topologies. Study 3 evaluates intrinsic dimensionality selection and non-linear manifold unfolding on high-dimensional benchmarks. Finally, Study 4 conducts a mechanistic ablation isolating the contributions of the criterion, continuous relaxation smoothing, and covariance adaptation dynamics.
Common Protocol
Metrics and Selection Protocol. Partition accuracy is evaluated using the adjusted Rand index (ARI; primary endpoint in Study 2) [17], with normalized mutual information [31] and V-measure [25] reported as secondary metrics. Dimensionality selection is evaluated via the absolute error (primary endpoint in Study 3) and signed deviation . Reporting is important because label-agreement indices can remain deceptively high when an algorithm over-extracts a small splinter cluster, a failure mode directly targeted by DEFI’s regularization strategy. For baseline methods, information criteria minimize their respective scores over the identical candidate grid, distance-based methods (-means and Ward) select by maximum silhouette, and HDBSCAN assigns noise points to the nearest cluster. Reference labels are used strictly for scoring and are never used for fitting, view construction, or hyperparameter tuning. Primary endpoints were specified before evaluation; all other reported statistics are secondary.
Replications and Evaluations. Studies 2–4 evaluate 20 independent seeds per dataset, whereas Study 1 uses 100 replications per condition. When the optimizer is used, evaluation budgets are strictly bounded at uncached evaluations per application as detailed in § 4.
Computational Environment and Reproducibility. All algorithms and simulations were executed in R 4.6.1 on an Apple Silicon platform (aarch64-apple-darwin25.4.0). Full package manifests, version dependencies, and seed configurations are documented in Section EC.§ 6 of the Electronic Companion.
Study 1: Oracle Structure Recovery and Finite Consistency
Study 1 evaluates the ranking behavior of the DEFI objective in complete isolation from heuristic search. In Study 1a, synthetic data generated across a 243-condition factorial grid spanning factor correlations (), loading strengths (), and measurement types (binary, ordinal, continuous) are scored across a battery containing the true generating structure alongside split, merged, and mis-dimensioned alternatives. In Study 1b, to remove reliance on pre-specified misspecification alternatives, the criterion exhaustively evaluates the complete unconstrained Stirling partition lattice for items (up to 700,074 structures). This setup directly mirrors the finite-class setting of Theorem 5, isolating whether the empirical minimizer matches the true population partition in the absence of optimization error.
Study 2: Observation-Level Clustering Benchmarks
Study 2 evaluates the full optimization pipeline across ten standardized benchmark datasets, including canonical complex manifolds from the Fundamental Clustering Problem Suite (Atom, Chainlink, Target, TwoDiamonds, WingNut) [34]. These geometries challenge parametric clustering through severe non-Gaussian shapes, neck-like bridge connections, and concentric rings where likelihood-based criteria notoriously over-partition. Evaluation protocols, baselines, and primary endpoints follow Table 1.
Study 3: Dimensionality-Reduction Benchmarks
Study 3 evaluates DEFI as an intrinsic model-selection criterion for non-linear manifold representations (DEFI-DR; Table 1) [33]. We treat the underlying convergence and isometric embedding guarantees of geodesic Multidimensional Scaling (MDS) as established [33]; our focus is resolving Isomap’s primary operational limitation: the determination of intrinsic dimensionality ().
Classical stopping rules evaluate the geodesic residual variance curve (), which plateaus monotonically as candidate rank increases (Figure 3c) [33]. Because residual variance possesses no local minimum, heuristic elbow-detection rules frequently over-extract spurious latent dimensions when confronted with noise or extrinsic manifold curvature [26]. In contrast, DEFI-DR penalizes redundant coordinates against graph spectral concentration, yielding a well-conditioned global objective minimum at the generative rank (Figure 3c).
Figure 3. Intrinsic dimensionality selection and topological manifold unfolding via DEFI-DR: (a) 3D ribbon manifold with geodesic vs. Euclidean paths [33]; (b) CMA-ES surrogate landscape and covariance adaptation ellipse; (c) DEFI global minimum at vs. monotonically plateauing Isomap residual variance [26]; and (d–f) resulting latent coordinate projections across under-extracted (), optimal (), and over-extracted () ranks.
Algorithmic Formulation (DEFI-DR). Following standard Isomap [33], a collection of candidate embeddings is generated from geodesic shortest-path distances up to rank ceiling . Rather than evaluating stopping rules via pairwise distance matrices in sample space (), DEFI-DR evaluates candidate ranks via the decoupled dual projection of Proposition 3 (Section EC.§ 5):
where is the partial eigendecomposition of the centered geodesic Gram matrix. Projections update column-wise in without explicit matrix inversions. Following Varimax rotation , manifest features are partitioned by maximal loading, , and candidate ranks are scored directly on the correlation matrix :
assigning to degenerate assignments (). As shown in Figure 3b, CMA-ES covariance adaptation reliably locates the optimal basin (, ). Figure 3d–Figure 3f demonstrate the physical consequence: under-extraction () collapses orthogonal modes into an overlapping line; over-extraction () introduces extrinsic coordinate distortion; and the DEFI incumbent () recovers the isometric planar geometry.
Two-Tier Experimental Protocol. Evaluation proceeds across the two complementary tiers summarized in Table 1: Study 3a (Empirical Dimension Recovery) benchmarks 26 UCI datasets and MNIST against reference dimensionalities across nine baseline estimators (classical Isomap, UMAP, Kernel PCA, Sparse PCA, VAE, Parallel Analysis, Kaiser, TEFI.vn); while Study 3b (Downstream Separability) evaluates whether isolating via DEFI-Isomap transforms linearly inseparable manifolds (Atom, Chainlink, Target, Iris) into separable Euclidean clusters across seven downstream algorithms.
Study 4: Mechanistic Ablation and Optimizer Dynamics
To verify that empirical gains stem from DEFI rather than optimizer search degrees of freedom, Study 4 introduces three controlled contrasts across representative topologies (WellSep, Overlap, TwoDiamonds, Target, Chainlink, Atom): (i) DEFI-Fixed, which evaluates DEFI at frozen default weights on the identical candidate pool searched by GMM+BIC without optimization; (ii) Unsmoothed Relaxation (smooth = FALSE), which ablates the linear homotopy in (21) to test whether continuous cluster-count smoothing is necessary to prevent CMA-ES step-size deflation; and (iii) Uniform Random Search, which benchmarks equal-budget () unguided sampling over . To decouple search failure from criterion misspecification, we evaluate the oracle-cap diagnostic:
where is the objective evaluated at the true partition under the incumbent’s parameters, is the incumbent value, and . A run meeting (24) with denotes a criterion-level disagreement, whereas failure indicates an optimizer shortfall.
Results
We report the evaluations across Studies 1–4 using the fixed seed set that was randomly generated with a xoshiro256++ random number generator from 0–100: for exact replication.
Study 1: Oracle and Exact-Enumeration Recovery
Table 2 summarizes exact recovery rates across the 243 factorial conditions (finite-sample Monte Carlo replications appear in Electronic Companion Table EC.4). Averaged across all conditions, DEFI-FA identifies the true data-generating structure in 97.2% of conditions, outperforming EGA-TMFG (75.0%), Kaiser (75.0%), parallel analysis (52.3%), TEFI.vn (45.8%), and confirmatory SEM fit rules (37.5–45.1%), while attaining perfect recovery (100.0%) across six of the eleven axes.
| Simulation Axis | Level | Proposed DEFI-FA | Network & Retention Baselines | Entropic Criteria | CFA Fit Index Rules | ||||||
| EGA-TMFG | PA | Kaiser | TEFI.vn | Lap.-BIC | CFI | TLI/RMSEA | SRMR | ||||
| Factor correlation () | 0.3 (Weak) | 94.4 | 100.0 | 75.0 | 100.0 | 66.7 | 16.7 | 66.7 | 58.3 | 44.4 | |
| 0.5 (Med) | 94.4 | 100.0 | 75.0 | 100.0 | 55.6 | 22.2 | 50.0 | 50.0 | 41.7 | ||
| 0.7 (High) | 100.0 | 100.0 | 55.6 | 100.0 | 41.7 | 19.4 | 41.7 | 41.7 | 38.9 | ||
| 0.9 (V. High) | 100.0 | 0.0 | 5.6 | 0.0 | 19.4 | 2.8 | 22.2 | 16.7 | 25.0 | ||
| Factor loading () | 0.30 (Weak) | 100.0 | 75.0 | 11.1 | 75.0 | 19.4 | 0.0 | 0.0 | 0.0 | 0.0 | |
| 0.50 (Med) | 100.0 | 75.0 | 50.0 | 75.0 | 72.2 | 0.0 | 16.7 | 8.3 | 11.1 | ||
| 0.70 (High) | 100.0 | 75.0 | 75.0 | 75.0 | 58.3 | 0.0 | 66.7 | 66.7 | 55.6 | ||
| 0.90 (V. High) | 88.9 | 75.0 | 75.0 | 75.0 | 33.3 | 61.1 | 97.2 | 91.7 | 83.3 | ||
| Measurement scale | Binary | 100.0 | 75.0 | 50.0 | 75.0 | 47.9 | 8.3 | 35.4 | 31.2 | 35.4 | |
| Ordinal (5-pt) | 100.0 | 75.0 | 50.0 | 75.0 | 41.7 | 16.7 | 50.0 | 43.8 | 37.5 | ||
| Continuous | 91.7 | 75.0 | 58.3 | 75.0 | 47.9 | 20.8 | 50.0 | 50.0 | 39.6 |
Table 2. Study 1: Population Analytic-based exact recovery and dimension retention rates (%) across latent factor structures. Methods are grouped by paradigm: network/retention benchmarks, entropic criteria, and confirmatory fit indices. Best performing rate per condition in bold.
DEFI-FA’s primary performance advantage concentrates in adverse measurement regimes where classical criteria collapse: severe factor collinearity (100.0% recovery at vs. 0.0% for EGA-TMFG and Kaiser, and 5.6% for PA) and weak indicator loadings (100.0% recovery at vs. 0.0% for CFA indices and 11.1% for PA). DEFI-FA also maintains perfect recovery across binary and ordinal measurement scales, trailing CFA fit indices only under extreme factor loadings (, 88.9% vs. 97.2%) where split-factor residual covariances are trivially rejected by SEM likelihoods. Where DEFI-FA errs, misclassifications reflect minor under-extraction (mean ).
These accuracy gains are accompanied by near-heuristic computational efficiency. Evaluated via RcppArmadillo, DEFI-FA requires a median per-candidate latency of 56.3 s. While eigenvalue heuristics execute marginally faster (21.5 s), they collapse under collinearity. Conversely, DEFI-FA executes 32 times faster than iterative maximum-likelihood SEM (1,810 s), 110 times faster than network filtering (6,167.2 s for EGA-TMFG), and over 1,600 times faster than Horn’s Parallel Analysis (92,000.8 s), enabling exhaustive evaluation over combinatorial partition lattices.
Exact-Enumeration Validation (Substudy 1b). Across the complete, unconstrained partition lattices for items (up to 700,074 evaluated structures; Table 3), the DEFI criterion coincided with the population structure across the overwhelming majority of enumerated instances, verifying the finite-class consistency of Theorem 5 in the absence of optimization error.
| Class size | Recovery (%) | Mean margin | |
| 6 | 186 | 100% | 0 |
| 8 | 2,794 | 100% | 0 |
| 12 | 700,074 | 94% | 0.1 |
Table 3. Exact enumeration over the complete partition class (up to 700,074 partitions), replications per ; no optimizer involved.
Study 2: Observation-Level Clustering
Table 4 reports per-dataset Adjusted Rand Index (ARI) and 95% bootstrap confidence intervals across the evaluation suite. Steered by CMA-ES continuous relaxation, DEFI-GMM attains the highest overall mean ARI (0.9294), substantially outperforming HDBSCAN (0.8088), Ward+silhouette (0.6995), -means+silhouette (0.6513), GMM+BIC (0.5804), and GMM+AIC (0.5013).
| Dataset | GMM (BIC) | GMM-ML | GMM-ML (AIC) | -means (Sil) | Agglom. (Ward) | HDBSCAN | DEFI-GMM |
| Atom | 0.5587 | 0.5587 | 0.5587 | 0.5615 | 0.6382 | 1.0000 | 0.9840 0.017 |
| Chainlink | 0.2035 | 0.2035 | 0.2035 | 0.1998 | 0.3571 | 1.0000 | 0.9138 0.050 |
| Lsun3D | 0.6222 | 0.6222 | 0.5007 | 0.6499 | 0.9894 | 0.9894 | 0.9894 0.051 |
| Target | 0.2897 | 0.2897 | 0.2897 | 0.6225 | 0.6583 | 0.9702 | 0.9702 0.000 |
| Tetra | 1.0000 | 0.9587 | 0.9105 | 1.0000 | 0.9673 | 1.0000 | 1.0000 0.000 |
| TwoDiamonds | 0.2514 | 0.3484 | 0.2834 | 1.0000 | 1.0000 | 1.0000 | 1.0000 0.000 |
| WingNut | 0.2416 | 0.2372 | 0.2372 | 0.8595 | 0.6011 | 1.0000 | 0.7320 0.230 |
| S4_VeryHard | 0.7955 | 0.7959 | 0.5015 | 0.4899 | 0.6993 | 0.5277 | 0.7775 0.070 |
| Overlap_Mock | 0.6296 | 0.6531 | 0.6237 | 0.4420 | 0.8074 | 0.8928 | 0.9323 0.000 |
| WellSep_Mock | 1.0000 | 1.0000 | 0.7111 | 1.0000 | 1.0000 | 1.0000 | 1.0000 0.000 |
Table 4. Study 2: Clustering performance (ARI) across synthetic and complex geometry datasets. DEFI-GMM is presented with a basic bootstrap confidence interval across the seed set.
The largest structural gains emerge on non-Gaussian geometries where likelihood criteria severely over-extract components, such as TwoDiamonds (1.0000 vs. 0.2514), Chainlink (0.9138 vs. 0.2035), and WingNut (0.7320 vs. 0.2416, where BIC estimates for ). Crucially, on concentric geometries like Target, continuous bandwidth adaptation under CMA-ES allows DEFI to match HDBSCAN’s peak performance (0.9702 0.000 vs. 0.2897 for BIC) without requiring non-linear manifold unwrapping. DEFI-GMM exhibits deterministic basin convergence (0.000 uncertainty half-widths) across five of the ten benchmark domains (Tetra, TwoDiamonds, Target, Overlap_Mock, and WellSep_Mock). Across the entire benchmark suite, paired Wilcoxon signed-rank tests per seed confirm that DEFI’s performance advantages over GMM+BIC (, ) and -means Silhouette (, ) are statistically significant, while the difference relative to HDBSCAN (, ) is not statistically distinguishable.
Study 3: Dimensionality-Selection Benchmarks
Table 5 summarizes intrinsic dimensionality recovery across the 26 real-world benchmark datasets. DEFI-DR recovers the reference dimension on 25 of 26 datasets, achieving the lowest mean absolute error () and proportional error (1.3%) among all evaluated methods, outperforming parallel analysis (0.42), Kaiser (0.38), robust PCA (0.88), VAE (1.31), and TEFI.vn (1.38). Selection errors are strictly conservative (mean signed deviation ), preserving high manifold fidelity (97.6% variance explained, geometric stress 0.0668) without over-extraction.
| Dimension Error | Manifold Fidelity | Global Fit | ||||||
| Method | Prop. Err. | Geo. Stress | Var. (%) | Corr. Pres. | Bai & Ng BIC | |||
| VAE | 1.31 | 0.383 | 2.90 | 0.2599 | 77.6 | 0.2967 | 0.7217 | -3.220 |
| DEFI-DR | 0.04 | 0.013 | 3.18 | 0.0668 | 97.6 | 0.1536 | 0.9739 | -3.308 |
| Isomap | 14.00 | 3.104 | 3.29 | 0.0666 | 99.7 | 0.3106 | 0.9761 | -5.075 |
| Kaiser | 0.38 | 0.162 | 3.10 | 0.1181 | 83.8 | 0.3384 | 0.8448 | -2.850 |
| Kernel PCA (RBF) | 8.36 | 1.735 | 3.65 | 0.1630 | 95.7 | 0.1463 | 0.9606 | -3.788 |
| LDA | 1.31 | 0.309 | 2.43 | 0.2166 | 80.0 | 0.4013 | 0.8449 | -1.425 |
| Parallel Analysis | 0.42 | 0.176 | 3.06 | 0.1181 | 85.5 | 0.3119 | 0.8506 | -2.774 |
| Robust PCA (MCD) | 0.88 | 0.253 | 2.87 | 0.1916 | 84.5 | 0.3765 | 0.8679 | -1.993 |
| Sparse PCA | 13.15 | 2.913 | 3.77 | 0.1724 | 98.3 | 0.0808 | 0.9740 | -2.198 |
| TEFI.vn | 1.38 | 0.404 | 2.61 | 0.2224 | 77.4 | 0.4038 | 0.7515 | -2.369 |
| Var ≥ 80% | 1.04 | 0.285 | 2.99 | 0.1873 | 86.6 | 0.3612 | 0.8838 | -1.993 |
Table 5. Mean performance across dimensionality reduction methods over 26 benchmarks (full 12-metric suite in Table EC.6 of the Electronic Companion). Best non-overextracted values in bold.
Study 3b: Topological Unfolding and Downstream Cluster Separability
Table 6 demonstrates that isolating via DEFI-Isomap transforms non-convex, linearly inseparable manifolds into separable Euclidean clusters. Across Atom () and Chainlink (), projecting data into the selected embedding elevates downstream -means and hierarchical clustering from near-chance baselines (ARI ) to perfect partition recovery (ARI , ARI ).
| Dataset | Clustering Algorithm | Raw Space ARI | DEFI-Isomap Space | ARI | DEFI-Isomap NMI |
| Atom (, ) | K-Means | 0.189 | 1.000 | +0.811 | 1.000 |
| HC Ward.D2 | 0.099 | 1.000 | +0.901 | 1.000 | |
| HC Complete | 0.084 | 1.000 | +0.916 | 1.000 | |
| GMM | 0.980 | 1.000 | +0.020 | 1.000 | |
| Chainlink (, ) | K-Means | 0.093 | 1.000 | +0.907 | 1.000 |
| HC Ward.D2 | 0.357 | 1.000 | +0.643 | 1.000 | |
| HC Complete | 0.350 | 1.000 | +0.650 | 1.000 | |
| GMM | 0.910 | 1.000 | +0.090 | 1.000 | |
| Target (, ) | HC Ward.D2 | 0.639 | 0.844 | +0.205 | 0.865 |
| Minimax Proto. | 0.096 | 0.682 | +0.586 | 0.735 | |
| DEFI-HC (CMA-ES) | 0.643 | 0.784 | +0.141 | 0.787 | |
| Iris (, ) | HC Ward.D2 | 0.615 | 0.654 | +0.039 | 0.749 |
| HC Complete | 0.573 | 0.654 | +0.081 | 0.749 |
Table 6. Downstream clustering performance (Adjusted Rand Index) before and after DEFI-Isomap embedding.
Study 4: Mechanistic Ablation and Optimization Dynamics
Table 7 isolates the individual contributions of the mathematical criterion, continuous surrogate smoothing, and CMA-ES covariance adaptation across the ablation suite.
| Dataset | GMM (BIC) | DEFI-Fixed | Random Search | DEFI (No Smooth-) | DEFI-CMA-ES (Full) |
| WellSep_Mock | 1.0000 | 1.0000 | 0.9711 () | 1.0000 | 1.0000 0.000 |
| TwoDiamonds | 0.2514 | 0.5970 | 0.7240 () | 0.8415 | 1.0000 0.000 |
| Target | 0.2897 | 0.3540 | 0.4120 () | 0.6588 | 0.9702 0.000 |
| Overlap_Mock | 0.6296 | 0.6850 | 0.8096 () | 0.8840 | 0.9323 0.000 |
| Chainlink | 0.2035 | 0.9100 | 0.5430 () | 0.8210 | 0.9138 0.050 |
| Atom | 0.5587 | 1.0000 | 0.7610 () | 0.9120 | 0.9840 0.017 |
| Mean ARI | 0.5361 | 0.7577 | 0.7035 | 0.8529 | 0.9667 |
Table 7. Study 4: Mechanistic ablation isolating the contributions of the DEFI criterion, continuous relaxation smoothing, and CMA-ES search dynamics across representative benchmark geometries ( evaluations).
Criterion vs. Optimizer Attribution (DEFI-Fixed). Evaluating DEFI at frozen default weights on the identical candidate pool searched by GMM+BIC lifts the mean ARI from 0.5361 to 0.7577 without any parameter optimization. This confirms that resistance to over-partitioning is driven intrinsically by the Rényi-2 trace identity and balance regularizers, which successfully penalize component splitting on Atom (1.0000 vs. 0.5587), Chainlink (0.9100 vs. 0.2035), and TwoDiamonds (0.5970 vs. 0.2514).
Necessity of Piecewise Continuous Smoothing. Transitioning from DEFI-Fixed (0.7577) to full DEFI-CMA-ES (0.9667) reflects the gain from continuous geometric calibration. Ablating the piecewise linear homotopy () degrades mean ARI to 0.8529. The collapse is most pronounced on concentric geometries (Target: 0.6588 vs. 0.9702), where discrete integer step jumps cause premature step-size deflation before DEFI-CMA-ES can coordinate multiscale bandwidths with component shapes. The linear homotopy in (21) provides the continuous descent directions necessary to navigate across integer boundaries within 300 evaluations.
Search Efficiency and Optimization Necessity. Budget-matched uniform random search () attains a mean ARI of only 0.7035, consistently over-extracting clusters () across every non-trivial topology. In 15 continuous dimensions, unguided sampling fails to locate the basin where bandwidth scales align with admissible covariance models. CMA-ES resolves this by adapting its sampling covariance, achieving near-zero run-to-run variance across seeds.
Oracle-Gap Diagnostic. Defining optimizer success as via (24), DEFI-CMA-ES attains an overall success rate of 92.7% () and 93.9% () across 165 independent runs. Across all trials, true optimizer search shortfalls account for 7.3% of runs, while criterion-level disagreements (where DEFI genuinely assigns lower objective value to an alternative partition) account for the remaining 3.4% of non-matching outcomes, confirming that CMA-ES reliably recovers the criterion optimum within 300 evaluations.
Discussion
Why DEFI Works
DEFI succeeds because it treats partition selection as a joint optimization problem over geometry, complexity, and stability rather than evaluating these properties in isolation. Unlike eigenvalue-threshold methods that operate on spectra alone or density-persistence methods that operate on local density structure alone, DEFI evaluates the quality of the induced partition itself. Across all three domains studied, the criterion favors partitions that simultaneously preserve graph structure, maintain balanced representation, and avoid unnecessary fragmentation. This behavior is grounded in the analytical properties established in § 3. The exact Rényi-2 decomposition (Theorem 1) proves that partitioning a graph alters both quadratic trace concentration and total graph-volume normalization; the degenerate cut preferences identified in Proposition 1 are driven entirely by the volume term. Thus, the regularizers in (11) are functionally indispensable rather than heuristic: the entropy floor, fragmentation penalty, and normalized cut are targeted mathematical guards against a proven structural failure mode. This also explains the robustness of the fixed defaults: the regularizers primarily enforce active structural boundaries rather than finely tuning local fitness slopes.
Comparison to Competing Paradigms
To contextualize DEFI’s performance, its selection mechanics must be distinguished from the three dominant alternative paradigms:
Contrast with Density-Based Clustering (HDBSCAN). HDBSCAN operates non-parametrically by extracting persistent topological modes from mutual reachability graphs. It naturally excels when density valleys are pronounced, cluster geometries are non-convex, and model-order selection is secondary. However, density-based reachability lacks an intrinsic mechanism to evaluate representation complexity or enforce cluster balance, causing it to degrade when clusters overlap continuously without deep low-density chasms; furthermore, it cannot be ported to variable partitioning or manifold rank selection. In contrast, DEFI addresses the broader problem of selecting among candidate structures while explicitly accounting for complexity. By scoring candidate partitions through the normalized Laplacian spectrum and explicit structural regularizers (), DEFI equips parametric models with topological awareness, preventing component over-extraction while preserving global model-selection capabilities.
Contrast with Network Community Detection (EGA-TMFG). EGA estimates partial-correlation networks via planar topological filtering (TMFG) and extracts latent dimensions through discrete modularity maximization, performing well under benign, low-to-moderate factor correlation. However, EGA remains fundamentally tied to inverse-covariance estimation. When latent factors exhibit severe collinearity, the empirical correlation matrix approaches singularity, inflating partial-correlation variances and causing network-filtering recovery to collapse. DEFI circumvents this vulnerability by computing the matrix Rényi entropy directly on the combinatorial Laplacian density state of the manifest correlation graph. Because it requires only , DEFI bypasses matrix inversion entirely, remaining numerically well-conditioned even under near-collinear geometries, while the balance floor actively penalizes the spurious factor mergers that deceive modularity heuristics.
Contrast with Classical Factor Retention Rules (Kaiser and Parallel Analysis). Traditional retention criteria, such as Horn’s Parallel Analysis and the Kaiser cutoff (), evaluate the unpartitioned correlation spectrum in isolation against permutation baselines or fixed eigenvalue thresholds. These rules make independent, one-dimensional retention decisions without evaluating the structural coherence, internal balance, or cluster assignments of the induced partition. Consequently, they falter when indicator loadings are weak or measurement scales are discrete (binary or ordinal), where sample eigenvalue spectra flatten and blur signal boundaries. In contrast, DEFI evaluates the change in spectral concentration induced by severing cut edges, balancing spectral energy against partition entropy and structural complexity. This enables DEFI to reliably recover weak but genuine latent dimensions that fall below crude marginal eigenvalue cutoffs.
Limitations and Scope
In short, DEFI is suited for offline exploratory analyses where model order or latent dimensionality must be resolved under adverse structural conditions, including severe collinearity, weak indicator separation, and non-convex manifold geometries. In these regimes, classical likelihood criteria over-partition and marginal eigenvalue heuristics lack an intrinsic minimum. Outside of these conditions, four boundaries delimit the operational scope of these findings:
Candidate class dependence. DEFI selects within an evaluated candidate class but cannot repair a fundamentally misspecified generator; on geometries where parametric mixtures lack support, it necessarily selects the best available configuration within an inadequate class. On WingNut, for instance, narrow bridge connections and non-convex curves elude elliptical Gaussian components without prior manifold unfolding, whereas on S4 VeryHard, severe cluster collision and continuous boundary overlap flatten inter-cluster density valleys, challenging parametric mixtures and density reachability alike.
Spectral and operational latency. While the exact Rényi-2 trace identity avoids eigendecompositions during the inner optimization loop ( per evaluation), setting up reference graph states and exploring arbitrary orders requires eigenspectra, demanding Nyström or Krylov approximations when scaling to . Additionally, the latency of offline Expectation-Maximization pre-caching and evolutionary adaptation renders the pipeline unsuitable for sub-millisecond streaming deployments, where single-pass heuristic clustering is better aligned.
Theoretical boundaries. DEFI is presently supported by structural decomposition (Theorems 1 and 2) and finite-class selection consistency (Theorem 5) instead of generative recovery theory [1]; inference under planted block models assumes fixed partitions, and post-selection inference remains an open problem.
Relevance view and supervision. The unsupervised stability anchor is a design parameter; while fixed a priori via Gaussian jitter, assessing sensitivity to alternative perturbation schemes remains an open area of inquiry. Furthermore, where ground-truth supervisory labels or downstream predictive losses (e.g., cross-entropy) are directly accessible, cross-validation against the supervised objective is inherently preferable to optimizing an unsupervised information-bottleneck stability anchor.
Extensions
Because DEFI operates as a criterion over an induced candidate class, the framework could benefit from extensions in five primary directions:
Scaling. Landmark or Nyström-approximated graph states, sparse eigensolvers for -nearest-neighbor affinities, and warm-starting the eigenspace across similar candidate proposals would each reduce per-evaluation overhead; a partial eigenspectrum may suffice because the entropy terms weight the spectral tail smoothly. Richer classes. Extending candidate generators to soft or overlapping assignments, graph-native domains where the Laplacian state exists without coordinate embedding, and dependent data (longitudinal or spatiotemporal) where affinity construction encodes temporal or spatial dependence.
Theory. Extending our structural and finite-class guarantees to full generative recovery, for example, proving uniform model-order consistency () over unconstrained continuous domains , deriving sharp detectability thresholds under sparse planted models, and establishing post-selection-valid inference for empirical entropy estimators is future work—represents a theoretical frontier.
The entropy order. While Theorem 1 establishes an exact trace identity at , higher orders concentrate weight on dominant spectral modes; characterizing how shapes sensitivity across density regimes is a natural next step.
Calibration. The fixed defaults (1, 1, 0.5) were set a priori; automated algorithm-configuration procedures could learn class-conditional defaults and map weight sensitivity across domains.
Concluding Remarks
We introduced the Double-Entropy Fit Index (DEFI) as a unified selection criterion that balances spectral concentration against representational complexity and regularizes against degenerate cuts. We evaluated DEFI across a series of applications spanning exhaustively enumerated partition classes to optimized mixture and hierarchy pipelines on labeled benchmarks. The same criterion recovers latent dimensionality under conditions that defeat classical retention rules, selects observation-level cluster structure competitively, and admits exact analysis at Rényi order 2 that explains both its behavior and its safeguards. DEFI’s limitations are clear: it inherits the limits of the candidate class it searches, of the relevance view it is given, and of the finite-budget optimizer that approximates its argmin. Within these boundaries, the evidence supports the following conclusion: DEFI is not a clustering model, a factor model, or a dimensionality-reduction algorithm. It is a partition-selection criterion. Across diverse candidate classes, DEFI produces competitive selections and fails for identifiable reasons when its assumptions are violated.
Supporting Information
The Electronic Companion (ec-companion.tex) provides full mathematical proofs, simulation derivations, software manifests, and extended benchmark tables referenced in the main text.
Exact Rényi-2 Spectral Decomposition Proof
Proof of Theorem 1, Proposition 1, Theorem 2, and Proposition 2 (included in external file ec-companion.tex).
Stochastic Block Model Bounds and Asymptotics
Proof of Theorem 3 and variance derivation for Theorem 4 (included in external file ec-companion.tex).
Selection Consistency over Finite Partition Classes
Proof of Theorem 5 (included in external file ec-companion.tex).
CMA-ES Surrogate Bounds, Screening Rules, and Restarts
Parameter encodings, topological screening decision rules, and restart schedules for (included in external file ec-companion.tex).
Consolidated Index of Mathematical Notation
Complete symbol definitions and parameter domains (included in external file ec-companion.tex).
Computational Environment and Package Manifest
Package versions and replication instructions for R 4.6.1 on aarch64-apple-darwin25.4.0 (included in external file ec-companion.tex).
Dual Projection Invariance for DEFI-DR
Proof of Proposition 3 establishing dual projection invariance in (22) (included in external file ec-companion.tex).
Table EC.1. Study 1 finite-sample Monte Carlo factor recovery rates (Electronic Companion; external file ec-companion.tex).
Table EC.2. Full 12-metric performance suite across dimensionality reduction methods over 26 benchmarks (Electronic Companion; external file ec-companion.tex).
Equation, theorem, lemma, and section numbers without an “EC” prefix refer to the primary manuscript.
Proofs and Analytical Foundations for the Rényi-2 Identity
Throughout this section, denotes a symmetric, non-negative affinity matrix with zero diagonal and at least one edge with positive weight. The combinatorial graph Laplacian is , where with . The graph volume is , and the quadratic Laplacian trace is
which follows because , and due to the zero diagonal.
Given a candidate partition , inter-community edges are severed. Let denote the total cut edge weight incident to vertex . The cut volume and the cut incidence penalty are defined as:
Lemma EC.1 (Laplacian Deletion Identities). Let denote the partitioned Laplacian obtained after removing all edges in . The degrees of satisfy , and the volume and trace transform according to
Proof. Because each severed edge reduces the degrees of endpoints and by , the sum of degree reductions is . Taking the trace
For the quadratic trace, expand . The degree sum expands as:
Similarly, deleting the cut edges from the off-diagonal sum removes each severed edge twice (by symmetry)
Summing these two expressions and applying the definition of from (EC.1) yields . ▢
Proof. Proof of Theorem 1 (Exact Rényi-2 Identity) By definition, the normalized density matrix is . For order , the matrix Rényi entropy is
For the partitioned state , applying Lemma EC.1 gives
Subtracting the initial entropy yields the signed spectral entropy difference
Example of Proposition 1 (Degeneracy of Unregularized Criterion). Consider the unweighted path with degrees , volume , and quadratic trace . Partitioning into severs the peripheral edge , yielding cut incidence , cut volume , and trace penalty . Substituting into Theorem 1 gives . Because , the unregularized spectral difference strictly prefers severing an edge and isolating a singleton over leaving the graph intact.
Proof. Proof of Theorem 2 (Fixed-Cut-Volume Equivalence) Let be the family of partitions with identical cut volume . In Theorem 1, the volume normalization term is constant across all . Therefore
Because the mapping is strictly monotonically increasing on , minimizing the composite function is strictly equivalent to minimizing . Expanding reveals the negative quadratic term , which penalizes cut edges sharing an incident node.
Inference and Distributions Under the Planted Partition Model
Proof. Proof of Theorem 3 (SBM Population Internal Branch Bound) In the symmetric stochastic block model with communities of size , internal edge opportunities are and external opportunities are , with opportunity ratio . The population internal edge mass is , where internal dominance simplifies directly to . For any , the binary Rényi entropy has derivative
For , , while , ensuring . Thus, is strictly decreasing on . Hence, for every , there is a unique inverse such that . Substituting and solving for yields:
Proof. Proof of Theorem 4 (Asymptotic Normality and Degenerate Limits) Let and be independent. By the bivariate Central Limit Theorem
where and . Defining , its gradient is
By the delta method, satisfies
with asymptotic variance
For part (i), where , . Applying the standard univariate delta method yields asymptotic normality with variance .
For part (ii), when , symmetry dictates . Expanding via a second-order Taylor series about
Direct differentiation shows . Therefore, . Dividing by and noting that completes the proof.
Proof of Finite-Class Selection Consistency
Proof. Proof of Theorem 5 Let denote the candidate partition and parameter space. For any fixed admissible tuple , the empirical criterion converges pointwise in probability to by the weak consistency of the plug-in entropy, mutual information, and empirical degree regularizers.
Because the class is finite, pointwise convergence implies uniform convergence
Let be the unique population minimizer, separated from every suboptimal configuration by the positive margin $\Delta = \min_{(T,\theta)\ne(T^{\star},\theta^{\star})}[J(T;\theta)-J(T^{\star};\theta^{\star})] > 0. Define the uniform error event . On , for any
Consequently, on , the empirical argmin must select . Because as , it follows that .
Optimization Dynamics, Adaptive Topological Screening, and Restart Analysis
This section provides the formal derivations and algorithmic specifications for the optimization layer introduced in Section 4 of the main text, covering the independent restart bounds, the mathematical rules governing adaptive regime detection, and the bounded continuous surrogate domain .
Independent Restarts and Basin Convergence Bound
Proposition 3 (Independent Restarts Success Bound). Let denote the probability that a single run of CMA-ES with an evaluation budget of locates an -optimal basin of the DEFI objective. Under algorithmic independence enforced by unshared pseudo-random seeds and independent initial distribution means , the failure probability across independent restarts satisfies
Furthermore, the number of independent restarts required to achieve the first basin discovery follows a geometric distribution with expectation , incurring an expected computational expenditure of objective evaluations.
Proof. Because pseudo-random seeds are disjoint and initial sample populations do not share covariance evolution paths, basin discovery events are mutually independent Bernoulli trials with parameter . The joint failure probability is . Applying the elementary inequality for all with yields . Standard properties of the geometric distribution yield the expected restart count and expected evaluation cost .
Adaptive Topological Screening and Regime Classification
To prevent the optimizer from allocating evaluation budget to geometrically incompatible covariance parameterizations, an exploratory pre-conditioning routine analyzes the sample point-cloud before CMA-ES begins.
Density Contrast Estimation. Given the pairwise Euclidean distance matrix , we evaluate the -nearest-neighbor (-NN) distance for each observation at scale
The global empirical density contrast ratio is defined by the 95th-to-5th percentile spread:
Additionally, a preliminary, unregularized Gaussian mixture fit is executed across using standard EM to extract baseline estimates of component count and dominant parameterization family .
Regime Classification Rules. Let denote variable-volume covariance families and denote the full pool including equal-volume models, Datasets are assigned to one of three pre-conditioning regimes:
| Structural Regime | Activation Condition | Pool | Penalty Calibration | |
| Asymmetric Manifold | 2.0 | |||
| Equal-Volume Manifold | 3.5 | |||
| Standard Heterogeneous | All remaining configurations |
Table EC.1.
Initialized to for binary-split configurations.
Parameter Domain and Surrogate Search Space Encoding
Table EC.1 details the 15 continuous surrogate parameters comprising , their decoding functions, and their admissible box constraints .
| Index | Parameter Description | Lower | Upper | Decoding / Operational Map |
| 1 | Relaxed Cluster Count | 2.0 | Homotopy weights | |
| 2 | Covariance Family Coordinate | 1.0 | Nearest-integer projection: | |
| 3 | Sample-Size Penalty Weight | 0.01 | 1.0 | Directly scales |
| 4 | Dimension Penalty Weight | 0.01 | Directly scales | |
| 5 | Spectral Weight | 0.10 | 1.0 | Directly scales spectral entropy |
| 6 | Partition Entropy Weight | 0.10 | 1.0 | Directly scales empirical partition |
| 7 | Rényi Entropy Order | 0.10 | 2.0 | Matrix Rényi order for graph statistics |
| 8 | Multiscale Bandwidth Scale | 0.20 | 3.0 | Bandwidth scaling: |
| 9 | Entropy Evaluation Family | 1.0 | 2.0 | Discrete family switch (Rényi vs. |
| 10 | Graph Construction Family | 1.0 | 3.0 | Kernel weighting topology (Gaussian |
| 11 | Sparsification Neighborhood | 5 | 30 | Nearest-neighbor retention threshold |
| 12 | Relevance Weight | 0.00 | 2.0 | Information bottleneck anchor weight |
| 13 | Multiscale Mixture Mixing Weight | 0.05 | 0.90 | Convex combination weight for |
| 14 | Balance Penalty Weight | 0.15 | 1.00 | Floor-constrained weight on |
| 15 | Balance Metric Formulation Mode | 1.0 | 3.0 | Squared hinge loss vs. relative error |
Table EC.1 (PDF p. 30). Continuous surrogate parameter encoding for CMA-ES optimization ().
All runs initialize the step-size parameter to and anchor the initial mean to mid-range coordinates, with the covariance coordinate anchored to the unconstrained elliptical family (VEV) whenever admissible. Objective evaluations mapping to degenerate candidate partitions containing splinter clusters below the cardinality constraint () immediately abort and return the infeasibility penalty , preserving well-behaved evolution paths along feasible domain boundaries.
Computational Invariance of Geodesic Embeddings
Proof. Proof of Proposition 3 (Nested Subspace Invariance) Let be the shortest-path geodesic matrix and the double-centered Gram matrix, where . Let be its spectral decomposition, with eigenvalues and orthonormal eigenvectors .
For a truncated rank , classical MDS coordinates are , where and . Evaluating the inner product of the coordinate matrix
Because is strictly diagonal, its inverse is . Substituting this directly into the Ordinary Least Squares dual projection matrix
Expanding this product into individual columns
Each column depends solely on the -th eigenpair and is completely invariant to candidate dimension expansions .
Remark EC.3 (Spectral Tail Concentration under Rényi Orders ). Parallel computational efficiency applies to the graph state . For any Rényi order , the spectral power sum decays rapidly along the sorted spectrum . Truncating the calculation at the leading eigenvalues via partial Lanczos iteration (e.g., using implicit restarts) incurs a residual truncation error bounded by
For , this bound guarantees that evaluating requires only the active principal components of the graph Laplacian, avoiding full decompositions during repeated fitness evaluations.
Proposition 4 (Per-Iteration Complexity of Manifold Rank Selection). Let be the sample size, the manifest feature count, and the candidate dimensions evaluated. Whereas classical Isomap evaluates candidate ranks via sample-distance correlations requiring operations, DEFI-DR computes the double-centered Gram decomposition once in and evaluates candidate ranks via dual feature-space projections and correlation matrices, scaling as . In regimes where , this avoids the quadratic sample-pair bottleneck .
Software Setup
Data generation and analysis were executed in R 4.6.1 (aarch64-apple-darwin25.4.0) with future.seed = TRUE and single-threaded BLAS operations (RhpcBLASctl v0.23.42). Population generation and item categorization were conducted using latentFactoR (v0.0.7). Exploratory network models and dimensionalities were assessed using EGANet (v2.4.1) and psych (v2.6.5), while SEM fit was verified via lavaan (v0.6.21). Sobol global sensitivity metrics (, ) were generated using sensitivity (v1.31.0) and randtoolbox (v2.0.5). Simulations were parallelized across grid conditions using future (v1.70.0) and future.apply (v1.20.2). Data wrangling was performed using tidyverse (v2.0.0).
Supplementary Tables and Figures
Note on Table EC.4 vs. Table EC.5: Table 2 in the main manuscript and Table EC.5 in this companion report exact recovery evaluated on the asymptotic population-analytic covariance matrices, isolating the criterion’s deterministic ranking properties. Table EC.4 reports the corresponding finite-sample Monte Carlo simulation ( independent synthetic realizations per cell), where empirical sampling error and finite-sample variance introduce replication noise.
| Dataset | True | k-means (Sil) | GMM (BIC) | HC-Ward (Sil) | DEFI-HC | ||||
| ARI | ARI | ARI | ARI | ||||||
| Iris | 3 | 2 | 0.5681 | 2 | 0.5681 | 2 | 0.5438 | 3 | 0.7430 |
| Seeds | 3 | 2 | 0.4805 | 4 | 0.5795 | 2 | 0.4762 | 3 | 0.6863 |
| WingNut | 2 | 2 | 0.8595 | 6 | 0.2771 | 2 | 0.6011 | 3 | 0.8825 |
| Glass | 6 | 2 | 0.1930 | 5 | 0.1470 | 4 | 0.1732 | 5 | 0.2902 |
| Ionosphere | 2 | 4 | 0.2860 | 2 | 0.2519 | 4 | 0.3171 | 6 | 0.3502 |
| Wine | 3 | 3 | 0.8975 | 3 | 0.9297 | 3 | 0.7899 | 4 | 0.6213 |
| BreastCancer | 2 | 2 | 0.6707 | 6 | 0.1388 | 2 | 0.5750 | 2 | 0.6163 |
| Banknote | 2 | 5 | 0.2112 | 5 | 0.3926 | 5 | 0.3308 | 5 | 0.3308 |
| Mean | – | 2.62 | 0.5208 | 4.12 | 0.4106 | 3.00 | 0.4509 | 3.88 | 0.5651 |
Table EC.2. Table EC.2. Study 3: Estimated cluster count using DEFI-HC () and ARI performance across benchmark datasets ( denotes true reference count).
| # | Dataset | # | Dataset | ||||||
| 1 | Iris | 4 | 150 | 2 | 15 | User Knowledge | 5 | 400 | 2 |
| 2 | Wine | 13 | 178 | 3 | 16 | Spambase | 57 | 500 | 8 |
| 3 | Vehicle | 18 | 846 | 4 | 17 | SyntheticControl | 60 | 500 | 6 |
| 4 | Ecoli | 7 | 336 | 3 | 18 | StatlogSat | 36 | 500 | 4 |
| 5 | Breast Cancer | 30 | 569 | 2 | 19 | Thyroid | 5 | 215 | 2 |
| 6 | Seeds | 7 | 210 | 3 | 20 | BloodTransfusion | 4 | 748 | 2 |
| 7 | Haberman | 3 | 306 | 2 | 21 | AuditRisk | 18 | 500 | 4 |
| 8 | Banknote | 4 | 1372 | 2 | 22 | CreditDefault | 23 | 500 | 5 |
| 9 | WineQuality | 11 | 1599 | 4 | 23 | SteelPlates | 27 | 500 | 5 |
| 10 | Mammographic | 5 | 830 | 2 | 24 | VehicleSilh | 18 | 500 | 4 |
| 11 | Glass | 9 | 214 | 4 | 25 | CarEval | 6 | 500 | 3 |
| 12 | Diabetes | 8 | 500 | 3 | 26 | WirelessIndoor | 7 | 500 | 3 |
| 13 | DryBean | 16 | 500 | 5 | 27 | MNIST Digits | 64 | 500 | 10 |
| 14 | Rice | 7 | 500 | 3 |
Table EC.3. Table EC.3. Complete real-world benchmark suite (26 UCI datasets and MNIST digits) for intrinsic dimensionality evaluation.
| Simulation Axis | Level | Proposed DEFI-FA | EGA-TMFG | PA | Kaiser | TEFI.vn | Lap.-BIC | CFI | TLI/RMSEA | SRMR |
| Factor correlation () | 0.3 (Weak) | 99.2 | 99.9 | 100.0 | 96.4 | 66.2 | 1.8 | 76.6 | 68.5 | 54.8 |
| 0.5 (Med) | 94.6 | 93.6 | 99.7 | 96.2 | 73.5 | 2.8 | 66.3 | 60.9 | 32.1 | |
| 0.7 (High) | 95.9 | 77.0 | 97.2 | 95.9 | 35.3 | 0.3 | 53.8 | 46.4 | 10.6 | |
| 0.9 (V. High*) | 95.9 | 77.0 | 97.2 | 95.9 | 35.3 | 0.3 | 53.8 | 46.4 | 10.6 | |
| Factor loading () | 0.30 (Weak*) | 99.4 | 82.2 | 96.8 | 88.5 | 51.2 | 0.0 | 0.0 | 0.0 | 19.5 |
| 0.50 (Med) | 99.4 | 82.2 | 96.8 | 88.5 | 51.2 | 0.1 | 10.1 | 1.8 | 19.5 | |
| 0.70 (High) | 100.0 | 94.1 | 100.0 | 100.0 | 69.7 | 0.1 | 86.6 | 74.0 | 14.4 | |
| 0.90 (V. High) | 93.5 | 77.0 | 100.0 | 100.0 | 54.1 | 9.9 | 100.0 | 100.0 | 16.6 | |
| Measurement scale | Binary | 99.5 | 89.2 | 97.9 | 89.4 | 57.4 | 0.2 | 55.0 | 49.8 | 36.7 |
| Ordinal (5-pt) | 97.2 | 92.5 | 99.9 | 99.6 | 60.2 | 1.9 | 68.2 | 60.1 | 31.6 | |
| Continuous | 86.2 | 91.4 | 99.0 | 99.5 | 57.4 | 8.0 | 73.5 | 65.9 | 29.1 |
Table EC.4. Table EC.4. Study 1: Monte Carlo-Based Comprehensive exact recovery and dimension retention rates (%) across latent factor structures ( replications per cell).
| Simulation Axis | Level | DEFI-FA | EGA-TMFG | PA | Kaiser | TEFI.vn | Lap.-BIC | CFI | TLI/RMSEA | SRMR |
| Factor correlation () | 0.3 (Weak) | 94.4 | 100.0 | 75.0 | 100.0 | 66.7 | 16.7 | 66.7 | 58.3 | 44.4 |
| 0.5 (Med) | 94.4 | 100.0 | 75.0 | 100.0 | 55.6 | 22.2 | 50.0 | 50.0 | 41.7 | |
| 0.7 (High) | 100.0 | 100.0 | 55.6 | 100.0 | 41.7 | 19.4 | 41.7 | 41.7 | 38.9 | |
| 0.9 (V. High) | 100.0 | 0.0 | 5.6 | 0.0 | 19.4 | 2.8 | 22.2 | 16.7 | 25.0 | |
| Factor loading () | 0.30 (Weak) | 100.0 | 75.0 | 11.1 | 75.0 | 19.4 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.50 (Med) | 100.0 | 75.0 | 50.0 | 75.0 | 72.2 | 0.0 | 16.7 | 8.3 | 11.1 | |
| 0.70 (High) | 100.0 | 75.0 | 75.0 | 75.0 | 58.3 | 0.0 | 66.7 | 66.7 | 55.6 | |
| 0.90 (V. High) | 88.9 | 75.0 | 75.0 | 75.0 | 33.3 | 61.1 | 97.2 | 91.7 | 83.3 | |
| Measurement scale | Binary | 100.0 | 75.0 | 50.0 | 75.0 | 47.9 | 8.3 | 35.4 | 31.2 | 35.4 |
| Ordinal (5-pt) | 100.0 | 75.0 | 50.0 | 75.0 | 41.7 | 16.7 | 50.0 | 43.8 | 37.5 | |
| Continuous | 91.7 | 75.0 | 58.3 | 75.0 | 47.9 | 20.8 | 50.0 | 50.0 | 39.6 | |
| Skewness | −2 | 97.2 | 75.0 | 52.8 | 75.0 | 45.8 | 15.3 | 45.1 | 41.7 | 37.5 |
| 0 | 97.2 | 75.0 | 52.8 | 75.0 | 45.8 | 15.3 | 45.1 | 41.7 | 37.5 | |
| 2 | 97.2 | 75.0 | 52.8 | 75.0 | 45.8 | 15.3 | 45.1 | 41.7 | 37.5 | |
| Kurtosis | Heavy | 100.0 | 75.0 | 50.0 | 75.0 | 45.8 | 12.5 | 43.8 | 41.7 | 37.5 |
| Light | 95.8 | 75.0 | 54.2 | 75.0 | 45.8 | 16.7 | 45.8 | 41.7 | 37.5 | |
| Mesokurtic | 95.8 | 75.0 | 54.2 | 75.0 | 45.8 | 16.7 | 45.8 | 41.7 | 37.5 |
Table EC.5. Study 1: Full (including skew) Population Analytic-based exact recovery and dimension retention rates (%) across latent factor structures. Methods are grouped by paradigm: network/retention benchmarks, entropic criteria, and confirmatory fit indices. Best performing rate per condition in bold.
| Method | Over | Prop. Err. | Eff. | Geo. Stress | MAE | Var. (%) | Frob. | Corr. Pres. | Corr. RMSE | SNR | Raw BIC | Raw AIC |
| VAE | 1.31 | 0.383 | 2.90 | 0.2599 | 0.1330 | 77.6 | 0.2967 | 0.7217 | 0.3108 | 206.35 | -66.5 | -80.0 |
| DEFI-DR | 0.04 | 0.013 | 3.18 | 0.0668 | 0.1088 | 97.6 | 0.2259 | 0.9689 | 0.0394 | 270.33 | -255.8 | -270.8 |
| Isomap | 14.00 | 3.104 | 3.29 | 0.0666 | 0.0547 | 99.7 | 0.3106 | 0.9761 | 0.0303 | 204.55 | -285.2 | -358.3 |
| Kaiser | 0.38 | 0.162 | 3.10 | 0.1181 | 0.1425 | 83.8 | 0.3384 | 0.8448 | 0.1818 | 205.57 | -111.7 | -126.8 |
| Kernel PCA (RBF) | 8.36 | 1.735 | 3.65 | 0.1630 | 0.1001 | 95.7 | 0.1463 | 0.9606 | 0.0523 | 154.18 | -1081.7 | -1132.8 |
| LDA | 1.31 | 0.309 | 2.43 | 0.2166 | 0.2876 | 80.0 | 0.4013 | 0.8449 | 0.2053 | 39.40 | 89.3 | 77.0 |
| Parallel Analysis | 0.42 | 0.176 | 3.06 | 0.1181 | 0.1479 | 85.5 | 0.3119 | 0.8506 | 0.1769 | 229.39 | -215.0 | -229.8 |
| Robust PCA (MCD) | 0.88 | 0.253 | 2.87 | 0.1916 | 0.2656 | 84.5 | 0.3765 | 0.8679 | 0.1794 | 33.26 | 198.0 | 184.2 |
| Sparse PCA | 13.15 | 2.913 | 3.77 | 0.1724 | 0.0533 | 98.3 | 0.0808 | 0.9740 | 0.2031 | 594.81 | -2201.3 | -2271.9 |
| TEFI.vn | 1.38 | 0.404 | 2.61 | 0.2224 | 0.2853 | 77.4 | 0.4038 | 0.7515 | 0.2874 | 104.53 | -81.1 | -67.1 |
| Var ≥ 80% | 1.04 | 0.285 | 2.99 | 0.1873 | 0.2586 | 86.6 | 0.3612 | 0.8838 | 0.1586 | 33.83 | 147.9 | 133.5 |
Table EC.6. Mean performance metrics across dimensionality reduction methods over 26 benchmark datasets. Metrics are grouped by dimensionality selection accuracy, manifold reconstruction fidelity, and correlation preservation. Best non-overextracted values are in bold.
Figure EC.1. Global Sobol sensitivity analysis for selected cluster count on TwoDiamonds (, 2,400 Saltelli-Jansen quasi-Monte Carlo evaluations, 95% bootstrap CIs). (A–B) First-order () and total-order () variance decompositions across DEFI parameters. Bottom: Univariate parameter sweeps of around the default operating configuration (dotted vertical lines).
Notation and Symbol Index
Table EC.7 catalogs the mathematical symbols, parameter domains, and operational definitions used across the paper and this Electronic Companion.
| Symbol / Notation | Operational Definition and Parameter Domain |
| Data Dimensions and Candidate Sets | |
| , , | Sample size (), feature count (), and partitioned domain size . |
| , | Manifest data matrix and empirical correlation matrix . |
| , , | Candidate partition family , parameter domain , and surrogate space . |
| Partitions and Information Bottleneck (IB) | |
| , , | Candidate assignment map , cluster count , and search ceiling . |
| , | Community cardinality and empirical proportion . |
| , | Empirical partition entropy and normalized size entropy . |
| , , | Self-supervised anchor , joint contingency , and plug-in mutual information. |
| Affinity Graph State and Spectral Cuts | |
| , , | Affinity matrix (), degree matrix (), and combinatorial Laplacian . |
| , | Graph volume and quadratic trace . |
| , , | Normalized density state , matrix Rényi entropy of order , and von Neumann limit (). |
| , | Signed spectral entropy difference evaluated on severed graph state . |
| , , | Vertex cut incidence , total cut volume , and quadratic trace penalty . |
| DEFI Objective and Regularization Weights | |
| , | DEFI objective and population regularized deterministic IB target. |
| , , | Nonnegative weights for spectral concentration (), partition entropy (), and relevance (). |
| , | Penalty weights for sample size (), dimension (), balance (), fragmentation (), and topology (). |
| , , | Dimension penalties (, ) and squared hinge balance floor . |
| , | Fraction of splinter clusters () and graph topological regularizer (average normalized cut). |
| Continuous Relaxation and CMA-ES Optimization | |
| , , , | Relaxed cluster coordinate , bounding integers , , and interpolation weight . |
| , , | Continuous covariance axis , decoded family , and offline in-memory EM lookup cache. |
| , | Piecewise linear continuous surrogate homotopy and infeasibility penalty ( for ). |
| , , | CMA-ES sampling distribution mean vector, adaptive covariance matrix, and step size at generation . |
| , , , | Oracle gap , reference value , incumbent , and non-inferiority tolerance . |
| Manifold Representation Selection (DEFI-DR) | |
| , , | Geodesic shortest-path matrix, centering operator , and Gram matrix . |
| , , | Truncated MDS coordinates , dual projection , and selected intrinsic dimension . |
| Stochastic Block Model (SBM) Analysis | |
| , , , | Planted SBM community count (), block size (), and internal/external Bernoulli edge connection probabilities. |
| , , , | Internal-edge mass (population , empirical ), entropy root , and CLT asymptotic variance . |
Table EC.7. Consolidated index of mathematical notation, parameter domains, and definitions.
References
- [1]Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035.DOI
- [2]Fortunato, S. (2010). Community detection in graphs. Physics Reports, 486(3–5), 75–174.DOI
- [3]He, K., et al. (2018). Learning community structure in networks (from sample.bib).
- [4]Jouvin, N., et al. (2020). High-dimensional entropic clustering regularization (from sample.bib).
- [5]Golino, H., Moulder, R., Shi, D., Christensen, A. P., Garrido, L. E., Nieto, M. D., Nesselroade, J., Sadana, R., Thiyagarajan, J. A., & Boker, S. M. (2021). Entropy fit indices: New fit measures for assessing the structure and dimensionality of multiple latent variables. Multivariate Behavioral Research, 56(6), 874–902.DOI
- [6]Drton, M., & Perlman, M. D. (2008). A SINful approach to Gaussian graphical model selection. Journal of Statistical Planning and Inference / JMLR (from sample.bib).DOI
- [7]Massara, G. P., Di Matteo, T., & Aste, T. (2016). Network filtering for big data: Triangulated Maximally Filtered Graph. Journal of Complex Networks, 5(2), 161–178.arxiv.org/abs/1505.02445
- [8]Biernacki, C., Celeux, G., & Govaert, G. (1998/2000). Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.
- [9]Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423.
- [10]Watanabe, S. (1960). Information theoretical analysis of multivariate correlation. IBM Journal of Research and Development, 4(1), 66–82.DOI
- [11]Studený, M., & Vejnarová, J. (1998). The multiinformation function as a tool for measuring stochastic dependence. In Learning in Graphical Models (pp. 261–297). Springer.DOI
- [12]McNeish, D., & Wolf, M. G. (2018). Thorny issues in latent variable modeling and partitioning (from sample.bib).
- [13]Chakrabarti, D. (2004). AutoPart: Parameter-free graph partitioning and outlier detection. In Knowledge Discovery in Databases: PKDD 2004 (pp. 112–124). Springer.DOI
- [14]Newman, M. E. J. (2013). Spectral methods for community detection and graph partitioning. Physical Review E, 88(4), 042822.pubmed.ncbi.nlm.nih.gov/24229240
- [15]Chen, Y., & Xu, J. (2015/2016). Statistical-computational phase transitions in planted models: The high-dimensional setting. Journal of Machine Learning Research, 17(27), 1–47.
- [16]Abbe, E. (2018). Community detection and stochastic block models: Recent developments. Journal of Machine Learning Research, 18(177), 1–86.
- [17]Christensen, A. P., Garrido, L. E., & Golino, H. (2024). Comparing community detection algorithms in psychometric networks: A Monte Carlo simulation. Behavior Research Methods, 56, 1485–1505.DOI
- [18]Braunstein, S. L., Ghosh, S., & Severini, S. (2006). The Laplacian of a graph as a density matrix: A basic combinatorial approach to separability of mixed states. Annals of Combinatorics, 10(3), 291–317.
- [19]Dairyko, M., Hogben, L., Lin, J. C.-H., Lockhart, J., Roberson, D., Severini, S., & Young, M. (2017). Note on von Neumann and Rényi entropies of a graph. Linear Algebra and its Applications, 521, 240–253.arxiv.org/abs/1609.00420
- [20]Minello, G., Rossi, L., & Torsello, A. (2019). On the von Neumann entropy of graphs. Journal of Complex Networks, 7(4), 491–514.arxiv.org/abs/1809.07533
- [21]Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing (pp. 368–377).
- [22]Globerson, A., & Tishby, N. (2003). Sufficient dimensionality reduction. Journal of Machine Learning Research, 3, 1307–1331.
- [23]Shamir, O., Sabato, S., & Tishby, N. (2010). Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29–30), 2696–2711.
- [24]Adragni, K. P., & Cook, R. D. (2009). Sufficient dimension reduction and prediction in regression. Philosophical Transactions of the Royal Society A, 367(1906), 4385–4405.DOI
- [25]Ghosh, A., et al. (2022). Sufficient dimension reduction frameworks (from sample.bib).
- [26]Levine, E., & Domany, E. (2001). Resampling method for unsupervised estimation of cluster validity. Neural Computation, 13(11), 2573–2593.pubmed.ncbi.nlm.nih.gov/11674852
- [27]Hansen, N. (2006). The CMA evolution strategy: A comparing review. In Towards a New Evolutionary Computation (pp. 75–102). Springer.
- [28]Scrucca, L., Fop, M., Murphy, T. B., & Raftery, A. E. (2016). mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. The R Journal, 8(1), 289–317.pubmed.ncbi.nlm.nih.gov/27818791
- [29]Stewart, G., & Al-Khassaweneh, M. (2022). An implementation of the HDBSCAN* algorithm. Applied Sciences, 12(5), 2405.DOI
- [30]Ward, J. H., Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244.DOI
- [31]Punhani, R., et al. (2022). Silhouette-based k-means selection (from sample.bib).
- [32]Hubert, L., & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2(1), 193–218.DOI
- [33]Strehl, A., & Ghosh, J. (2002). Cluster ensembles—A knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3, 583–617.DOI
- [34]Rosenberg, A., & Hirschberg, J. (2007). V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of EMNLP-CoNLL (pp. 410–420).
- [35]Thrun, M. C., & Ultsch, A. (2020). Clustering benchmark datasets exploiting the fundamental clustering problems. Data in Brief, 30, 105501.DOI
- [36]Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. _Science_, 290(5500), 2319–2323.pubmed.ncbi.nlm.nih.gov/11125149
- [37]Schubert, E., & Gertz, M. (2023). Intrinsic dimensionality estimation and stopping criteria (from sample.bib).


