betaFeedback

WorksComputer Science

The Double Entropy Fit Index (DEFI): An Information-Theoretic Approach to Dimensionality Reduction and Clustering

Abstract

We introduce the Double-Entropy Fit Index (DEFI), a regularized information-theoretic criterion for selecting among candidate graph partitions and reduced representations. DEFI couples entropy-based compression with relevance preservation and penalizes model dimensionality, cluster imbalance, and partition fragmentation. An exact Rényi-2 entropy identity shows that partitioning alters both spectral concentration and graph-volume normalization, explaining why unregularized entropy minimization can favor degenerate solutions. Across a 243-condition population-analytic factor recovery benchmark, DEFI achieved an overall exact recovery rate of 97.2% (versus 75.0% for EGA-TMFG, 45.8% for TEFI on von Neumann entropy). In controlled local candidate batteries (distinguishing the true model from adjacent K±1K \pm 1 and factor-split alternatives), DEFI identified the correct partition in 99.2% of replications. Across ten clustering benchmarks, DEFI-guided Gaussian mixtures achieved a mean ARI of 0.929, exceeding HDBSCAN (0.809), Ward clustering (0.700), and conventional GMM criteria (0.501-0.580), with gains concentrating where standard criteria over-extract components.

Introduction

Complex data in operations, engineering, and behavioral sciences are increasingly analyzed through the lens of network structure: variables or observations are embedded in a graph, and the graph’s communities are interpreted as latent dimensions of the system. In psychometrics, exploratory graph analysis (EGA) operationalizes this idea by treating communities in a partial-correlation network as latent factors [14]; the same reduction of a high-dimensional system to a small number of internally coherent communities underlies dimensionality reduction pipelines across machine learning and operations research [10, 16]. In all of these settings, a single question emerges: among candidate partitions of the data, which one identifies the latent structure rather than an artifact of the fit criterion?

This question is relevant because unregularized entropic criteria improve monotonically under partition refinement, favoring degenerate singletons or splinter clusters unless model complexity is priced explicitly [18]. Existing remedies introduce additional operational constraints: network-based entropy criteria such as the von Neumann total entropy fit index (TEFI.vn) [13] rely on inverse-covariance estimators [9, 20] that become ill-conditioned under factor collinearity and are difficult to apply on disconnected or isolated-node graph structures, while classical mixture criteria (BIC, ICL) continue to reward spurious components on complex non-Gaussian geometries [3]. What is missing is a unified criterion that scores candidate partitions directly from the spectrum of an affinity graph while explicitly pricing community count, cardinality imbalance, and topological fragmentation. We introduce the Double-Entropy Fit Index (DEFI), a unified partition-selection criterion designed to address this methodological gap.

Related Literature

Four frames of literature informed the work on our proposed Information-Theoretic selection criterion, DEFI. Entropy-based assessment of latent structure descends from Shannon’s theory of communication [29] through Watanabe’s total correlation [37]. For a collection of n≥2n \ge2 observed random variables X1,…,XnX_1,\ldots,X_n (e.g., manifest features or items), total correlation C(X1,…,Xn)C(X_1,\ldots,X_n) generalizes bivariate mutual information as the gap between the sum of marginal entropies H(Xi)H(X_i) and the joint entropy H(X1,…,Xn)H(X_1,\ldots,X_n):

C(X1,…,Xn)=∑i=1nH(Xi)−H(X1,…,Xn).(1)C(X_1,\ldots,X_n) = \sum_{i=1}^{n} H(X_i) - H(X_1,\ldots,X_n). \tag*{(1)}

Additionally, Studený and Vejnarová showed that total correlation coincides with the multiinformation function: it is nonnegative and vanishes exactly under mutual independence [32]. Used directly as a partitioning objective, total correlation is monotone under refinement and is minimized by the trivial singleton partition; any usable criterion must therefore pair an entropic score with control over the number and shape of communities [18, 21].

Partitioning as structure discovery. A second line of work makes the partition itself the estimand. Compression-based formulations use minimum description length to choose the number and composition of communities without prespecification [5]. Spectral methods translate community detection into the eigenanalysis of Laplacian or modularity matrices [10, 23], and their detectability limits under planted models are well characterized [6, 1]. In psychometrics, exploratory graph analysis (EGA) estimates the number of latent dimensions by community detection on partial-correlation networks [14], and the entropy fit indices built on this machinery, most directly, the von Neumann total entropy fit index (TEFI.vn) which scores candidate variable partitions by system entropy [13]; such network-based estimates compare favorably with classical factor-retention rules in simulation [7]. These criteria remain tied to inverse-covariance network estimators [9, 20], whose computational cost and connectedness requirements are restrictive in high-dimensional or degenerate settings.

Entropy of the Laplacian state. DEFI computes entropy from the normalized combinatorial Laplacian ρ=L/Tr⁡(L)\rho= L/\operatorname{Tr}(L), a density-matrix construction of [4] whose Rényi generalization is developed by [8]. Two findings from that literature inform the design. First, the Laplacian-state entropy is dominated by graph size and volume rather than by community structure, so it cannot be interpreted directly as a community descriptor [22]. Second, any partition-sensitive use of the entropy must be explicit about normalization: the exact Rényi-2 change induced by deleting cut edges decomposes into a degree-sensitive trace term and an opposing graph-volume term (§ 2).

Regularization and the information bottleneck. DEFI’s penalty grammar draws on classical model selection. The integrated completed likelihood augments a mixture likelihood with an entropy term that discourages degenerate assignments [3], and Bayesian Information Criterion (BIC) and Minimum Description Length (MDL) penalties price each additional community against sample size and dimension. The information-bottleneck principle formalizes the trade-off between compressing an input and preserving information about a relevance variable [35, 12, 28]; sufficient dimension reduction supplies the corresponding statistical framework for information-preserving reduction [2, 11].

Positioning. DEFI differs from this prior work in combination rather than in any single component. First, it evaluates a candidate partition through the Rényi entropy of the affinity-graph Laplacian state, which allows DEFI to avoid inverse-covariance estimation and tolerate disconnected graphs while pricing complexity, imbalance, and fragmentation explicitly. Second, it exposes one mixed-variable objective that a single derivative-free optimizer instantiates over mixture, hierarchical, and subspace candidate classes (§ 2). To the best of our knowledge, TEFI.vn is the closest published baseline and anchors our oracle and dimensionality comparisons.

Contributions and Outline

This paper introduces the Double-Entropy Fit Index (DEFI), a regularized information-theoretic criterion for selecting among candidate partitions and reduced representations. We make four primary contributions: (1) Unified Information-Theoretic Criterion: we formulate DEFI as a single objective coupling entropy-based compression, a self-supervised stability anchor via the deterministic information bottleneck, and structural regularizers for dimensionality, imbalance, and fragmentation, computed directly from the spectrum of an affinity Laplacian without inverse-covariance estimation (§ 2); (2) Exact Spectral Trace Decomposition: we prove that at Rényi order α=2\alpha= 2, the change in spectral entropy decomposes into a trace-concentration term and an opposing graph-volume normalization term, providing the first analytical explanation for degenerate unregularized cuts and mathematically justifying DEFI’s structural penalties (§ 3); (3) Optimization Framework for Mixed Partition Spaces: we solve the mixed discrete-continuous model-selection problem via continuous relaxation and CMA-ES, combining a piecewise linear homotopy over integer cluster counts with adaptive topological pre-conditioning and offline candidate pre-caching to eliminate step-discontinuity stagnation (§ 4); and (4) Empirical Validation: across a 243-condition factor-recovery benchmark, an exact enumeration across 700,074 structures, ten complex clustering geometries, and 26 high-dimensional manifold datasets, DEFI demonstrates superior recovery over classical information criteria, density methods, and factor retention rules (§ 5, § 6). Section 7 contextualizes these findings within competing literature and details operational boundaries. Complete mathematical proofs, simulation derivations, and a comprehensive symbol index appear in the Electronic Companion (Sections EC.§ 1–EC.8). Code and replication materials are publicly available at github will be available soon.

The DEFI Criterion

This section defines the Double-Entropy Fit Index (DEFI). A single principle organizes everything that follows: every experiment in this paper optimizes the same criterion, differing only in the candidate partition class T\mathcal{T} and free parameter space Θ\Theta (Table 1). Conceptually, DEFI couples four foundational components: (i) a spectral entropy term that measures affinity-graph concentration;

Study / FocusData & SuiteDEFI VariantOptimizer & BudgetRelevance Anchor YYPrimary EndpointsBaselines
Study 1a Factor Recovery243-condition latent factor grid (R=100R = 100)DEFI-FAFrozen defaults (no search; Sec. 2)Perturbed correlation matrix (M=PM = P)Exact recovery (%) ΔK=K^−K†\Delta K = \hat{K} - K^\daggerEGA-TMFG, PA, Kaiser, TEFI.vn, SEM fit (CFI/TLI)
Study 1b Finite ConsistencyStirling partition lattice (P≤12P \le12)DEFI-FAExact enumeration (≤700,074\le700,074 structures)Perturbed correlation matrix (M=PM = P)Global argmin recovery (%)Exhaustive search over all possible graphs based on the Stirling numbers
Study 2 Clustering10 FCPS & synthetic benchmarks (20 seeds)DEFI-GMM DEFI-HCCMA-ES continuous relaxation (b=300b = 300)Feature Gaussian jitter view (M=NM = N)Adjusted Rand Index, ΔK\Delta K, NMI, V-measureGMM (BIC [3], ICL [27]), HDBSCAN [30], Ward [36], kk-means (silhouette) [24]
Study 3 Dimensionality26 UCI benchmarks & MNIST (20 seeds)DEFI-DRDual-projection rank sweep (M≤PM \le P)Perturbed correlation matrix (M=PM = P)∣Δm∣=∣m^−K†∣|\Delta m| = |\hat{m} - K^\dagger|, Geo. stress, Var. (%)Isomap, UMAP, Kernel PCA, PA, Kaiser, Sparse PCA, VAE
Study 4 Ablation6 representative topologies (20 seeds)DEFI-GMMControlled ablation (b=300b = 300 budget)Feature Gaussian jitter view (M=NM = N)ARI, ΔK\Delta K, Oracle gap G^\hat{G} (ϵ∈{0,0.05}\epsilon\in\{0, 0.05\})DEFI-Fixed, Random search, DEFI (no smooth-KK)

Table 1. Evaluation Design Matrix: Experimental protocols, search configurations, relevance anchors, and evaluation endpoints across Studies 1–4.

(ii) a partition entropy term that prices representation complexity;

(iii) a stability anchor derived from the deterministic information bottleneck; and

(iv) structural regularizers that penalize degenerate, fragmented, or topologically discordant solutions.

We formalize these components sequentially below (architectural overview in Figure 1), defining candidate partitions and the affinity-graph density state before assembling the global objective. A consolidated index of mathematical symbols, parameter domains, and operational definitions is cataloged in Section EC.8 of the Electronic Companion.

Operational architecture of the DEFI objective decomposition

Figure 1. Operational architecture of the DEFI objective decomposition. Candidate partition TT, affinity geometry W(σ)\mathbf{W}(\sigma), data matrix X\mathbf{X}, and self-supervised reference view YY map into four regularized processing pillars, aggregated into scalar objective J^DEFI(T;θ)\widehat{J}_{\mathrm{DEFI}}(T;\theta) and optimized via CMA-ES coordinate feedback.

Data, Partitions, and Candidate Classes

Let X=(x1,…,xN)⊤∈RN×P\mathbf{X}=(\mathbf{x}_1,\ldots,\mathbf{x}_N)^\top\in\mathbb{R}^{N\times P} be a data matrix of NN observations and PP variables. A candidate partition is a deterministic assignment map

T:{1,…,N}→{1,…,K},(2)T:\{1,\ldots,N\}\to\{1,\ldots,K\}, \tag*{(2)}

where KK is the candidate number of communities. For each community c∈{1,…,K}c\in\{1,\ldots,K\}, its cardinality and proportion are given by:

nc(T)=∑i=1NI(Ti=c),p^T(c)=nc(T)N,c=1,…,K.(3)n_c(T)=\sum_{i=1}^{N} I(T_i=c),\qquad\widehat{p}_T(c)=\frac{n_c(T)}{N},\qquad c=1,\ldots,K. \tag*{(3)}

When DEFI is applied to variables rather than observations (as in the factor-recovery studies), the domain of TT is {1,…,P}\{1,\ldots,P\}, the denominator in (3) is PP, and the affinity graph is constructed on manifest features. A candidate class T\mathcal{T} denotes the family of admissible partitions produced by a given generator, and Θ\Theta is the bounded mixed discrete-continuous parameter space associated with that generator. In practice, T\mathcal{T} may be induced by parametric mixtures, cuts of an agglomerative dendrogram, or rotated subspace projections, with Θ\Theta capturing the corresponding structural and regularizer coordinates (Table 1).

Graph Construction and the Laplacian State

Each candidate configuration represents the data by a symmetric, nonnegative affinity matrix W=(Wij)∈RN×N\mathbf{W} = (W_{ij}) \in\mathbb{R}^{N \times N} with Wii=0W_{ii} = 0. For a Gaussian-affinity graph,

Wij=exp⁡(−∥xi−xj∥222σ2),i≠j,(4)W_{ij} = \exp\left(-\frac{\lVert\mathbf{x}_i-\mathbf{x}_j\rVert_2^2}{2\sigma^2}\right), \qquad i \ne j, \tag*{(4)}

with bandwidth σ>0\sigma> 0 scaled from the median positive pairwise distance; multiscale graphs evaluate convex combinations of affinities at bandwidths 0.5σ0.5\sigma, σ\sigma, and 2σ2\sigma. Let D=diag⁡(d1,…,dN)\mathbf{D} = \operatorname{diag}(d_1,\ldots,d_N) with di=∑jWijd_i = \sum_j W_{ij} denote the degree matrix, and L=D−W\mathbf{L} = \mathbf{D} - \mathbf{W} the combinatorial Laplacian. Provided the graph contains at least one positive-weight edge, Tr⁡(L)>0\operatorname{Tr}(\mathbf{L}) > 0 and the normalized density state ρ=L/Tr⁡(L)\boldsymbol{\rho} = \mathbf{L}/\operatorname{Tr}(\mathbf{L}) is positive semidefinite with unit trace, forming a density matrix whose eigenvalues λ1,…,λN\lambda_1,\ldots,\lambda_N are nonnegative and sum to one [4, 8]. Because only Tr⁡(L)>0\operatorname{Tr}(\mathbf{L}) > 0 is required, the state remains well-defined on disconnected graphs and isolated vertices, unlike inverse-covariance network estimators. When an input similarity is not guaranteed positive semidefinite, DEFI projects it by clipping negative eigenvalues at zero and renormalizing to unit trace.

Entropy Terms

For a density matrix ρ\boldsymbol{\rho} and order α>0\alpha> 0, α≠1\alpha\ne1, the matrix Rényi entropy is

Hα(ρ)=11−αlog⁡Tr⁡(ρα)=11−αlog⁡(∑i=1Nλiα),(5)H_\alpha(\boldsymbol{\rho}) = \frac{1}{1-\alpha}\log\operatorname{Tr}(\boldsymbol{\rho}^{\alpha}) = \frac{1}{1-\alpha}\log\left(\sum_{i=1}^{N}\lambda_i^{\alpha}\right), \tag*{(5)}

whose limit as α→1\alpha\to1 is the von Neumann entropy HVN(ρ)=−Tr⁡(ρlog⁡ρ)H_{\mathrm{VN}}(\boldsymbol{\rho}) = -\operatorname{Tr}(\boldsymbol{\rho}\log\boldsymbol{\rho}) (0log⁡0:=00\log0 := 0). In CMA-ES search, α\alpha is optimized over (0,1)∪(1,100](0,1)\cup(1,100]; candidates within 10−310^{-3} of α=1\alpha= 1 evaluate the von Neumann limit. The spectral entropy term compares the graph state before and after the partition is imposed. Given TT, let L~\widetilde{\mathbf{L}} be the Laplacian obtained by deleting all inter-community edges, and let ρ~T=L~/Tr⁡(L~)\widetilde{\boldsymbol{\rho}}_T = \widetilde{\mathbf{L}}/\operatorname{Tr}(\widetilde{\mathbf{L}}). Component (i) evaluates the signed entropy difference

ΔHα(T)=Hα(ρ~T)−Hα(ρ).(6)\Delta H_\alpha(T) = H_\alpha(\widetilde{\boldsymbol{\rho}}_T) - H_\alpha(\boldsymbol{\rho}). \tag*{(6)}

Its exact α=2\alpha= 2 decomposition, which motivates our structural regularizers, is derived in § 3. Component (ii), the partition entropy, prices representation complexity via the entropy of community proportions:

H^N(T)=−∑c=1Kp^T(c)log⁡p^T(c).(7)\widehat{H}_N(T) = -\sum_{c=1}^{K}\widehat{p}_T(c)\log\widehat{p}_T(c). \tag*{(7)}

The Relevance View and the Information Bottleneck

Component (iii) grounds partition selection in the Information Bottleneck (IB) principle [35], balancing compression against relevance preservation. In our unsupervised setting, the relevance variable YY operationalizes a self-supervised cross-view stability anchor defined over the identical index set as TT, derived from an auxiliary perturbation of the observed system. Let M∈{N,P}M \in\{N, P\} denote the partitioned domain cardinality, corresponding to observation clustering (M=NM=N) or variable partitioning (M=PM=P). For observation clustering, an auxiliary view X~=X+E\widetilde{\mathbf{X}} = \mathbf{X} + \boldsymbol{\mathcal{E}} is constructed via feature-scaled Gaussian perturbation (Eij∼N(0,(0.1σ^j)2))(\mathcal{E}_{ij} \sim\mathcal{N}(0,(0.1\widehat{\sigma}_j)^2)), chosen to disrupt localized point density while preserving macroscopic cluster separation), where an initial unregularized baseline clustering on X~\widetilde{\mathbf{X}} induces reference partition Y∈{1,…,KY}NY \in\{1,\ldots,K_Y\}^{N} [19]. For variable partitioning, an independently perturbed correlation matrix RX~\mathbf{R}_{\widetilde{X}} yields reference assignment Y∈{1,…,KY}PY \in\{1,\ldots,K_Y\}^{P}. The reference anchor YY is fixed prior to optimization and never derived from evaluation labels. Because candidate assignments are deterministic maps T=f(X)T=f(X), assignments satisfy the Markov chain:

Y↔X↔T.(8)Y \leftrightarrow X \leftrightarrow T. \tag*{(8)}

Mutual information between data and representation collapses to the partition entropy, I(X;T)=H(T)I(X;T)=H(T), recovering the deterministic information bottleneck (DIB) [28] where compression is governed directly by H^M(T)\widehat{H}_M(T). For candidate partition T∈{1,…,K}MT \in\{1,\ldots,K\}^{M} and reference assignment Y∈{1,…,KY}MY \in\{1,\ldots,K_Y\}^{M}, the joint contingency is

p^TY(c,y)=1M∑i=1MI(Ti=c,Yi=y).(9)\widehat{p}_{TY}(c,y)=\frac{1}{M}\sum_{i=1}^{M}\mathbb{I}(T_i=c,Y_i=y). \tag*{(9)}

Preserved relevance is scored by plug-in mutual information (0log⁡(0/0):=00\log(0/0):=0):

I^M(T;Y)=∑c=1K∑y=1KYp^TY(c,y)log⁡p^TY(c,y)p^T(c)p^Y(y).(10)\widehat{I}_M(T;Y)=\sum_{c=1}^{K}\sum_{y=1}^{K_Y}\widehat{p}_{TY}(c,y)\log\frac{\widehat{p}_{TY}(c,y)}{\widehat{p}_T(c)\widehat{p}_Y(y)}. \tag*{(10)}

The signed term −bI^M(T;Y)-b\widehat{I}_M(T;Y) in the global objective rewards cluster boundaries that persist under stochastic jitter, penalizing noise overfit.

The Criterion and its Structural Regularizers

Assembling components (i)–(iv) yields the DEFI criterion:

J^DEFI(T;θ)=a1ΔHα(T;W)+a2H^N(T)−bI^N(T;Y)+γ1ΩN(K)+γ2ΩP(K)+γbalΩfloor(T;τ)+γfragΩfrag(T)+ηΩgraph(T;W),(11)\begin{aligned} \widehat{J}_{\mathrm{DEFI}}(T;\boldsymbol{\theta})={}&a_1\Delta H_{\alpha}(T;\mathbf{W})+a_2\widehat{H}_N(T)-b\widehat{I}_N(T;Y)\\ &+\gamma_1\Omega_N(K)+\gamma_2\Omega_P(K)\\ &+\gamma_{\mathrm{bal}}\Omega_{\mathrm{floor}}(T;\tau)+\gamma_{\mathrm{frag}}\Omega_{\mathrm{frag}}(T)\\ &+\eta\Omega_{\mathrm{graph}}(T;\mathbf{W}), \tag*{(11)} \end{aligned}

with nonnegative coefficients collected in θ\boldsymbol{\theta} along with α\alpha, graph parameters, and generator choices. The first three terms represent components (i)–(iii). The remaining terms formalize the structural regularizers of component (iv):

Dimensionality penalties. BIC-type costs price candidate complexity against sample size and feature dimension: ΩN(K)=Klog⁡N\Omega_N(K)=K\log N and ΩP(K)=Klog⁡P\Omega_P(K)=K\log P.

Cluster balance floor. Normalizing community-size entropy yields H~size(T)=H^N(T)/log⁡K∈[0,1]\widetilde{H}_{\mathrm{size}}(T)=\widehat{H}_N(T)/\log K\in[0,1], which attains 1 at uniform proportions. Given tolerance floor τ∈[0,1]\tau\in[0,1],

Ωfloor(T;τ)=[max⁡{0,τ−H~size(T)}]2(12)\Omega_{\mathrm{floor}}(T;\tau)=\left[\max\left\{0,\tau-\widetilde{H}_{\mathrm{size}}(T)\right\}\right]^2 \tag*{(12)}

penalizes only severe imbalance without punishing substantively plausible asymmetry.

Partition fragmentation penalty. Splinter communities below a minimum cardinality nmin⁡n_{\min} are priced by

Ωfrag(T)=1K∑c=1KI{nc(T)<nmin⁡}.(13)\Omega_{\mathrm{frag}}(T)=\frac{1}{K}\sum_{c=1}^{K}\mathbb{I}\{n_c(T)<n_{\min}\}. \tag*{(13)}

Graph topological concordance. Disagreement with graph topology is penalized via the average normalized cut:

Ωgraph(T;W)=1K∑c=1K∑i∈Vc∑j∉VcWij∑i∈Vcdi,(14)\Omega_{\mathrm{graph}}(T;W) = \frac{1}{K}\sum_{c=1}^{K}\frac{\sum_{i\in V_c}\sum_{j\notin V_c}W_{ij}}{\sum_{i\in V_c}d_i}, \tag*{(14)}

where Vc={i:Ti=c}V_c=\{i:T_i=c\} and 00:=1\frac{0}{0}:=1 maximally penalizes zero-volume sets. The selected partition solves

T^∈arg⁡min⁡T∈T, θ∈Θ J^DEFI(T;θ),(15)\widehat{T}\in\underset{T\in\mathcal{T},\,\theta\in\Theta}{\arg\min}\ \widehat{J}_{\mathrm{DEFI}}(T;\theta), \tag*{(15)}

using CMA-ES (§ 4). Because negative spectral differences ΔHα\Delta H_\alpha and relevance preservation −I^N-\widehat{I}_N can drive J^DEFI\widehat{J}_{\mathrm{DEFI}} below zero while penalties Ω(T)≥0\Omega(T)\geq0, parameter space Θ\Theta is constrained to strictly positive penalty infima:

Θ={θ:α∈(0,1)∪(1,100], σ>0, γk∈[γ‾k,γ‾k] with γ‾k>0}.\Theta=\{\theta:\alpha\in(0,1)\cup(1,100],\ \sigma>0,\ \gamma_k\in[\underline{\gamma}_k,\overline{\gamma}_k]\text{ with }\underline{\gamma}_k>0\}.

These positive floors ensure that search trajectories cannot evade structural regularization by collapsing weights to zero, forcing CMA-ES to adapt continuous kernel bandwidths and covariance geometries against active penalty boundaries.

Properties and Assumptions

Four structural properties govern this formulation: (i) only relative objective differences matter, so candidate rankings are invariant to constant additive shifts; (ii) all terms depend on TT strictly through the induced partition, ensuring invariance under community permutation; (iii) the balance regularizer is zero at uniform cluster proportions and strictly positive only below the floor τ\tau; and (iv) the signed spectral term ΔHα(T)\Delta H_\alpha(T) requires structural regularization because, as proven in § 3, volume normalization can cause unregularized cuts to favor trivial partitions.

The implementation relies on four assumptions (theory-specific conditions for the internal-dominance branch and planted block models are stated with their respective theorems in § 3): (A1) Well-defined graph state: every decoded affinity graph contains at least one positive-weight edge and strictly positive bandwidths, with positive semidefinite graph states enforced via eigen-clipping; (A2) Admissible partitions: candidate partitions satisfy 2≤K≤Kmax⁡2\leq K\leq K_{\max} and the minimum-cluster cardinality constraint of (13), with numerically failed generator fits declared infeasible; (A3) Nontrivial relevance view: the reference configuration YY is not almost surely identical to XX and is independent of external evaluation labels; and (A4) Estimator comparability: all objective terms evaluate the identical observation set under uniform normalization conventions, and the parameter domain Θ\Theta is bounded.

Theoretical Properties

This section states the analytical properties of the DEFI criterion that motivate its regularizers and justify its use as a selection rule. Full proofs, and additional inferential results for the fixed-partition coarse-graining appear in the electronic companion (EC.1, EC.2, EC.3, EC.4). Throughout, W∈RN×NW\in\mathbb{R}^{N\times N} is a symmetric nonnegative affinity matrix with zero diagonal and at least one positive-weight edge, LL its combinatorial Laplacian, V=Tr⁡(L)=∑idi>0V=\operatorname{Tr}(L)=\sum_i d_i>0 its volume, S=Tr⁡(L2)=∑idi2+∑i≠jWij2S=\operatorname{Tr}(L^2)=\sum_i d_i^2+\sum_{i\ne j}W_{ij}^2, and ρ=L/V\rho=L/V the graph state of Section 2.

Our first result is the exact decomposition of the spectral term (6) at order α=2\alpha=2. Given a partition TT, delete its inter-community edges, and for each vertex ii let ci=∑j:{i,j} cutWijc_i=\sum_{j:\{i,j\}\ \mathrm{cut}}W_{ij} be the weighted cut incidence. Define the cut-volume and trace penalties

C(T)=∑i=1Nci=2∑{i,j}∈εcut(T)Wij,ε(T)=∑i=1N(2dici−ci2)+2∑{i,j}∈εcut(T)Wij2.(16)\begin{aligned} C(T) &= \sum_{i=1}^{N} c_i = 2\sum_{\{i,j\}\in\varepsilon_{\mathrm{cut}}(T)} W_{ij},\\ \varepsilon(T) &= \sum_{i=1}^{N}(2d_i c_i-c_i^2)+2\sum_{\{i,j\}\in\varepsilon_{\mathrm{cut}}(T)}W_{ij}^2. \tag*{(16)} \end{aligned}

Theorem 1 (Exact Rényi-2 identity). Let TT be a partition whose deletion leaves at least one positive-weight edge, with 0<C(T)<V0<C(T)<V and 0<ε(T)<S0<\varepsilon(T)<S. Then the spectral term at α=2\alpha=2 is exactly

ΔH2(T)=−log⁡(1−ε(T)S)+2log⁡(1−C(T)V).(17)\Delta H_2(T)=-\log\left(1-\frac{\varepsilon(T)}{S}\right)+2\log\left(1-\frac{C(T)}{V}\right). \tag*{(17)}

For unweighted simple graphs (Wij∈{0,1}W_{ij}\in\{0,1\}), ε(T)=∑ici(2di−ci+1)\varepsilon(T)=\sum_i c_i(2d_i-c_i+1).

The two terms in (17) oppose each other: deleting edges reduces Tr⁡(L2)\operatorname{Tr}(L^2) but also reduces the normalizing volume. Because ρ\rho is normalized by Tr⁡(L)\operatorname{Tr}(L), neither effect can be neglected, and the second term rewards deletion per se. The consequence is a failure mode for unregularized spectral-entropy minimization.

Proposition 1 (Degeneracy of the unregularized criterion). There exist connected graphs GG and partitions TT with no community interpretation such that ΔH2(T)<0=ΔH2(T0)\Delta H_2(T)<0=\Delta H_2(T_0), where T0T_0 is the no-cut partition. Hence minimizing (6) alone can strictly prefer a nontrivial partition over T0T_0, and minimizing it over unrestricted families is not equivalent to minimizing the cut or any degree-weighted cut. A three-vertex path with one end edge deleted already exhibits the strict inequality.

Proposition 1 is the analytical reason (11) prices imbalance, fragmentation, and graph disagreement rather than relying on the entropy difference alone. When the volume effect is held fixed, the criterion does reduce to a tractable combinatorial objective.

Theorem 2 (Fixed-cut-volume equivalence). On any admissible family PC0={T:C(T)=C0}\mathcal{P}_{C_0}=\{T:C(T)=C_0\} with 0≤C0<V0\le C_0<V,

arg⁡min⁡T∈PC0ΔH2(T)=arg⁡min⁡T∈PC0ε(T).(18)\underset{T\in\mathcal{P}_{C_0}}{\arg\min}\Delta H_2(T)=\underset{T\in\mathcal{P}_{C_0}}{\arg\min}\varepsilon(T). \tag*{(18)}

Because u↦−log⁡(1−u/S)u\mapsto-\log(1-u/S) is strictly increasing on [0,S)[0,S), minimizing ΔH2\Delta H_2 at fixed volume is equivalent to minimizing ε(T)\varepsilon(T). The induced penalty ε(T)\varepsilon(T) couples cut edges sharing an incident node via the negative quadratic term −∑ici2-\sum_i c_i^2, distinguishing it from a standard linear minimum cut.

The next results interpret the criterion under a planted model. The scope here is narrow: the analysis concerns a fixed partition and the binary edge-mass coarse-graining induced by it.

Theorem 3 (Population bound under the equal-sized Stochastic Block Model). Let GG be generated by the symmetric qq-community planted-partition model with equal community sizes ss, internal probability pinp_{\mathrm{in}}, external probability poutp_{\mathrm{out}}, and rq,s=(q−1)s/(s−1)r_{q,s}=(q-1)s/(s-1) external-to-internal edge-opportunity ratio. Let π=pinMin/(pinMin+poutMout)\pi=p_{\mathrm{in}}M_{\mathrm{in}}/(p_{\mathrm{in}}M_{\mathrm{in}}+p_{\mathrm{out}}M_{\mathrm{out}}) be the population internal-edge mass, where Min=q(s2)M_{\mathrm{in}}=q\binom{s}{2} and Mout=(q2)s2M_{\mathrm{out}}=\binom{q}{2}s^2. For α>1\alpha>1, on the internal-dominance branch π≥12\pi\ge\frac{1}{2} (equivalent to pin/pout≥rq,sp_{\mathrm{in}}/p_{\mathrm{out}}\ge r_{q,s}), and for any ϵ∈[0,log⁡2]\epsilon\in[0,\log2],

hα(π)≤ϵ⟺pinpout≥rq,sBα(ϵ)1−Bα(ϵ).(19)h_{\alpha}(\pi)\le\epsilon\quad\Longleftrightarrow\quad\frac{p_{\mathrm{in}}}{p_{\mathrm{out}}}\ge r_{q,s}\frac{B_{\alpha}(\epsilon)}{1-B_{\alpha}(\epsilon)}. \tag*{(19)}

where hα(x)=(1−α)−1log⁡(xα+(1−x)α)h_{\alpha}(x)=(1-\alpha)^{-1}\log(x^{\alpha}+(1-x)^{\alpha}) is the binary Rényi entropy and Bα(ϵ)B_{\alpha}(\epsilon) is its unique solution of hα(B)=ϵh_{\alpha}(B)=\epsilon on [12,1][\frac{1}{2},1]. Without the branch condition, low entropy is symmetric between internal and external dominance and (19) does not follow.

In words, Theorem 3 defines the sharp signal-to-noise ratio (pin/pout)(p_{\mathrm{in}}/p_{\mathrm{out}}) required for the planted partition to achieve an entropy deficit ϵ\epsilon, proving that low spectral entropy guarantees cohesive community structure only along the internal-dominance branch.

Proposition 2 (Rényi–2 Bridge under the Planted Partition Model). Let GnG_n be an unweighted simple graph generated from the symmetric qq-community stochastic block model (SBM) of Theorem 3, and let T∗T^* denote the planted partition. Define the empirical internal-edge mass and trace-penalty ratio by

πn=XinXin+Xout,ηn=ϵ(T∗)S.\pi_n=\frac{X_{\mathrm{in}}}{X_{\mathrm{in}}+X_{\mathrm{out}}},\qquad\eta_n=\frac{\epsilon(T^*)}{S}.

Then the spectral entropy difference at order α=2\alpha=2 satisfies the exact finite-sample identity

ΔH2(T∗)=2log⁡πn−log⁡(1−ηn).\Delta H_2(T^*)=2\log\pi_n-\log(1-\eta_n).

Moreover, in the dense regime with fixed pin,pout∈(0,1)p_{\mathrm{in}},p_{\mathrm{out}}\in(0,1), (πn,ηn)→p(π,η)(\pi_n,\eta_n)\xrightarrow{p}(\pi,\eta) where

1−η=π2+O(n−1).1-\eta=\pi^2+O(n^{-1}).

Consequently,

ΔH2(T∗)=oP(1)as n→∞.\Delta H_2(T^*)=o_P(1)\quad\text{as }n\to\infty.

Remark 1 (Role of the Spectral Term and Scale Calibration). Proposition 2 bridges the macroscopic edge-mass statistics of Theorems 3 and 4 with the spectral criterion of Theorem 1. Because 1−C(T∗)/V=πn1-C(T^*)/V=\pi_n holds identically on unweighted simple graphs, the volume-normalization term in ΔH2\Delta H_2 is governed directly by the internal-edge mass πn\pi_n. The asymptotic vanishing ΔH2(T∗)=oP(1)\Delta H_2(T^*)=o_P(1) reflects the calibrated null behavior of the criterion evaluated strictly at the planted partition T∗T^*. Across the broader candidate space T\mathcal{T}, candidate partitions that violate true community boundaries yield 1−η(T)≠π(T)21-\eta(T)\ne\pi(T)^2, producing non-vanishing spectral penalties that separate cohesive structures from arbitrary cuts. Furthermore, while the combinatorial regularizers (ΩN,Ωfloor,Ωfrag)(\Omega_N,\Omega_{\mathrm{floor}},\Omega_{\mathrm{frag}}) depend solely on cluster cardinalities, ΔHα(T;W)\Delta H_{\alpha}(T;W) serves as the primary coupling between the partition and the continuous affinity geometry WW. Finally, the asymptotic cancellation 2log⁡π−log⁡(1−η)→02\log\pi-\log(1-\eta)\to0 ensures that the spectral term operates on the same effective O(1)O(1) scale as the information-theoretic and structural regularizers. This stabilizes the objective where unregularized spectral differences flatten, preventing graph volume from overwhelming model-order selection while formally justifying the structural regularizers in (11).

Theorem 4 (Asymptotic distribution of the empirical entropy). Let P^n=Xin/(Xin+Xout)\widehat{P}_n=X_{\mathrm{in}}/(X_{\mathrm{in}}+X_{\mathrm{out}}) be the empirical internal-edge mass in the model of Theorem 3, with Xin∼Bin⁡(Min,pin)X_{\mathrm{in}}\sim\operatorname{Bin}(M_{\mathrm{in}},p_{\mathrm{in}}) and Xout∼Bin⁡(Mout,pout)X_{\mathrm{out}}\sim\operatorname{Bin}(M_{\mathrm{out}},p_{\mathrm{out}}) independent. If both count variances diverge and πn\pi_n is bounded away from 0 and 1, then

VP,n−1(P^n−πn)→dN(0,1)\sqrt{V_{P,n}^{-1}}(\widehat{P}_n-\pi_n)\xrightarrow{d}\mathcal{N}(0,1)

for the explicit variance VP,nV_{P,n} derived in EC.2. Consequently: (i) if πn→π≠12\pi_n\to\pi\ne\frac{1}{2}, the plug-in entropy hα(P^n)h_{\alpha}(\widehat{P}_n) is asymptotically normal with mean hα(πn)h_{\alpha}(\pi_n) and variance [hα′(πn)]2VP,n[h'_{\alpha}(\pi_n)]^2V_{P,n}; (ii) if πn=12\pi_n=\frac{1}{2} eventually, the first-order limit degenerates to the second-order χ12\chi_1^2 distribution, (hα(P^n)−log⁡2)/VP,n→d−2αχ12(h_{\alpha}(\widehat{P}_n)-\log2)/V_{P,n}\xrightarrow{d}-2\alpha\chi_1^2; and (iii) in the dense regime P^n−πn=OP(n−1)\widehat{P}_n-\pi_n=O_P(n^{-1}), whereas under sparse scaling (pin=a/n,pout=b/n)(p_{\mathrm{in}}=a/n,p_{\mathrm{out}}=b/n) the convergence rate slows to OP(n−1/2)O_P(n^{-1/2}). The balanced point πn=12\pi_n=\frac{1}{2} is simultaneously the entropy maximum and the branch boundary, which is where the second-order limit applies. These statements concern inference for a fixed partition; if the same graph is used both to select a partition and to estimate its entropy, selection effects invalidate the fixed-partition variances.

The criterion (11) is a plug-in estimator of the population regularized deterministic information-bottleneck objective

JRDIB(T)=a2H(T)−bI(T;Y)+Ωγ(T),(20)\mathcal{J}_{\mathrm{RDIB}}(T) = a_2 H(T) - b I(T;Y) + \Omega_\gamma(T), \tag*{(20)}

where the compression term I(X;T)I(X;T) has been replaced by H(T)H(T) via the Markov chain condition in (8), and the spectral and structural terms are collected in Ωγ(T)\Omega_\gamma(T).

Theorem 5 (Selection consistency over finite classes). Suppose that for every fixed admissible TT the entropy, relevance, and regularizer estimators in (11) converge in probability to their population counterparts. Then J^DEFI(T;θ)→PJRDIB(T)\widehat{J}_{\mathrm{DEFI}}(T;\theta) \xrightarrow{P} \mathcal{J}_{\mathrm{RDIB}}(T) pointwise. If, in addition, the candidate class T×Θ\mathcal{T} \times\Theta is finite and the population minimizer T⋆T^\star is unique and separated by a margin Δ>0\Delta> 0, then the empirical minimizer satisfies P(T^N=T⋆)→1\mathbb{P}(\widehat{T}_N = T^\star) \to1. In particular, for fixed PP, fixed Kmax⁡K_{\max}, and a discretized Θ\Theta, the full class of partitions into at most Kmax⁡K_{\max} communities is finite, so the consistency statement covers the entire partition class, not only a hand-picked candidate set.

Two scope statements delimit Theorem 5. First, with a continuous Θ\Theta, uniformity requires a uniform law of large numbers for the entropy and mutual-information estimators over Θ\Theta, which we do not establish; the exact-enumeration study of Section 6 provides computational evidence for the discretized case. Second, consistency identifies the population minimizer of the regularized objective.

Remark 2 (Theoretical Scope and Boundaries). To prevent conflation between structural properties and generative recovery guarantees, we delineate three precise boundaries of the theoretical analysis: (i) Structural vs. Generative: Theorems 1 and 2 provide structural decompositions of the Rényi-2 spectral term on arbitrary graphs, identifying the volume-normalization artifact that necessitates structural regularization, rather than establishing minimax recovery rates under planted random graph models; (ii) Calibration vs. Detectability: Proposition 2 establishes asymptotic null calibration (ΔH2(T⋆)=oP(1)\Delta H_2(T^\star) = o_P(1)) at the true partition under dense block models, ensuring scale compatibility with complexity penalties rather than bounding sharp information-theoretic detectability thresholds; and (iii) Finite-Class vs. Model-Order Consistency: Theorem 5 guarantees selection consistency of the DEFI minimizer over finite, margin-separated candidate classes T×Θ\mathcal{T} \times\Theta, but does not constitute an unconstrained proof of model-order consistency (P(K^=K∗)→1\mathbb{P}(\widehat{K} = K^\ast) \to1) over unbounded, continuous partition spaces. The empirical studies in Section 6 evaluate the operational recovery performance of the full pipeline where exact high-dimensional asymptotic analysis remains an open problem.

Optimization with the Covariance Matrix Adaptation Evolution Strategy

The model selection problem (15) couples discrete decisions (K∈{2,…,Kmax⁡}K \in\{2,\ldots,K_{\max}\}, covariance family mcovm_{\mathrm{cov}}) with continuous coordinates (α,σ,γ)(\alpha,\sigma,\gamma). We embed these parameters into a bounded 15-dimensional surrogate domain Θ~⊂R15\widetilde{\Theta} \subset\mathbb{R}^{15} solved via CMA-ES [15] (parameter bounds and encodings in Section FC.4). To mitigate discrete step discontinuities, cluster counts are relaxed to real coordinates K~∈[2,Kmax⁡]\widetilde{K} \in[2,K_{\max}], bounding integer floors kfl=⌊K~⌋k_{\mathrm{fl}} = \lfloor\widetilde{K} \rfloor, ceilings kce=⌈K~⌉k_{\mathrm{ce}} = \lceil\widetilde{K} \rceil, and fractional weight wk=K~−kfl∈[0,1)w_k = \widetilde{K} - k_{\mathrm{fl}} \in[0,1). Denoting bounding partitions by Tkfl\mathcal{T}_{k_{\mathrm{fl}}} and Tkce\mathcal{T}_{k_{\mathrm{ce}}} under continuous covariance coordinate mcov=M[clamp⁡(round⁡(m~),1,∣M∣)]m_{\mathrm{cov}} = \mathcal{M}[\operatorname{clamp}(\operatorname{round}(\widetilde{m}),1,|\mathcal{M}|)], the objective is evaluated as the linear homotopy:

J~DEFI(θ~)=(1−wk)J^DEFI(Tkfl;decode⁡(θ~))+wkJ^DEFI(Tkce;decode⁡(θ~)).(21)\widetilde{J}_{\mathrm{DEFI}}(\widetilde{\theta}) = (1-w_k)\widehat{J}_{\mathrm{DEFI}}(\mathcal{T}_{k_{\mathrm{fl}}};\operatorname{decode}(\widetilde{\theta})) + w_k\widehat{J}_{\mathrm{DEFI}}(\mathcal{T}_{k_{\mathrm{ce}}};\operatorname{decode}(\widetilde{\theta})). \tag*{(21)}

When K~\widetilde{K} is an integer (wk=0w_k = 0), (21) recovers DEFI identically. Between integer boundaries, (21) creates a continuous descent surface that prevents premature step-size deflation, while proposals with cluster sizes nc<2n_c < 2 receive an infeasibility penalty Pinf=+109P_{\mathrm{inf}} = +10^9.

Figure 2 illustrates this search trajectory on the TwoDiamonds topology. Even when initialized in an over-partitioned regime (K≈7K \approx7), the adaptive covariance matrix C(g)\mathbf{C}^{(g)} contracts along continuous homotopy slopes, systematically shedding spurious sub-clusters until converging onto the ground truth (K=2K = 2, ARI =1.000= 1.000).

Figure 2. CMA-ES optimization dynamics under continuous relaxation on the TwoDiamonds topology (b=300b = 300). Top row: Evolution of the sampling distribution mean m(g)\mathbf{m}^{(g)}, adaptive covariance ellipse C(g)\mathbf{C}^{(g)} (dashed orange), and offspring proposals (black points) across the continuous surrogate landscape Θ~\widetilde{\Theta}. Bottom row: Corresponding decoded cluster partitions TKT_K and covariance geometries in sample space. The linear homotopy (21) provides continuous descent directions that steer the population from initial over-partitioning (K~≈7.1\widetilde{K} \approx7.1, ARI =0.356= 0.356) to basin convergence (K^=2\widehat{K} = 2, ARI =1.000= 1.000) without integer step-discontinuity stagnation.

To prevent CMA-ES from allocating evaluation budget to geometrically incompatible covariance structures, an exploratory topological screening phase pre-conditions the search space before optimization begins. Point densities ρ^i\hat{\rho}_i estimated from local kk-NN distances first yield a density-contrast ratio,

κdens=Q0.95({ρ^i})/Q0.05({ρ^i}).\kappa_{\mathrm{dens}} = Q_{0.95}(\{\hat{\rho}_i\})/Q_{0.05}(\{\hat{\rho}_i\}).

Pairing κdens\kappa_{\mathrm{dens}} with an exploratory unregularized mixture fit classifies the data into one of three structural regimes (asymmetric manifold, equal-volume manifold, or standard heterogeneous), which initializes K~0\widetilde{K}_0 and prunes inadmissible covariance parameterizations from the candidate pool M\mathcal{M} (decision rules detailed in Section EC.§ 4). Finally, this screening step calibrates surrogate-domain boundaries by enforcing γbal≥0.15\gamma_{\mathrm{bal}} \ge0.15 and bounding the dimensionality-penalty ceiling by γˉ2=max⁡(2.0,round⁡(c⋅N/log⁡N))\bar{\gamma}_2 = \max(2.0,\operatorname{round}(c \cdot N/\log N)).

Computational efficiency is achieved by decoupling candidate generation from fitness evaluation. In an offline phase, Gaussian mixtures are fitted once across candidate configurations (K,m)(K,m) via Expectation-Maximization and cached in an in-memory table Mcache\mathcal{M}_{\mathrm{cache}}. During CMA-ES search, evaluating (21) requires an O(1)O(1) partition retrieval from Mcache\mathcal{M}_{\mathrm{cache}} followed by compiled C++ trace evaluations via RcppArmadillo. By exploiting the exact Rényi-2 trace identity of Theorem 1, evaluating C(T)C(T) and ϵ(T)\epsilon(T) requires only sparse edge-cut traversals scaling as O(∣Ecut(T)∣)O(|\mathcal{E}_{\mathrm{cut}}(T)|) rather than dense matrix operations. Consequently, uncached evaluations execute in under 60 μ\mus, completing the entire 300-evaluation CMA-ES run within seconds.

Evaluation Design

The evaluation is organized so that each study isolates a unique property of DEFI distinct question. Study 1 removes search entirely and evaluates candidate structures directly at fixed default weights across both a 243-condition population-analytic factorial grid and the complete partition lattice of small instances. Study 2 attaches the CMA-ES optimizer and asks whether the full pipeline is competitive on observation-level clustering across heterogeneous benchmark topologies. Study 3 evaluates intrinsic dimensionality selection and non-linear manifold unfolding on high-dimensional benchmarks. Finally, Study 4 conducts a mechanistic ablation isolating the contributions of the criterion, continuous relaxation smoothing, and covariance adaptation dynamics.

Common Protocol

Metrics and Selection Protocol. Partition accuracy is evaluated using the adjusted Rand index (ARI; primary endpoint in Study 2) [17], with normalized mutual information [31] and V-measure [25] reported as secondary metrics. Dimensionality selection is evaluated via the absolute error ∣Δm∣|\Delta m| (primary endpoint in Study 3) and signed deviation ΔK=K^−K†\Delta K = \hat{K} - K^\dagger. Reporting ΔK\Delta K is important because label-agreement indices can remain deceptively high when an algorithm over-extracts a small splinter cluster, a failure mode directly targeted by DEFI’s regularization strategy. For baseline methods, information criteria minimize their respective scores over the identical candidate grid, distance-based methods (kk-means and Ward) select KK by maximum silhouette, and HDBSCAN assigns noise points to the nearest cluster. Reference labels are used strictly for scoring and are never used for fitting, view construction, or hyperparameter tuning. Primary endpoints were specified before evaluation; all other reported statistics are secondary.

Replications and Evaluations. Studies 2–4 evaluate 20 independent seeds per dataset, whereas Study 1 uses 100 replications per condition. When the optimizer is used, evaluation budgets are strictly bounded at b=300b = 300 uncached evaluations per application as detailed in § 4.

Computational Environment and Reproducibility. All algorithms and simulations were executed in R 4.6.1 on an Apple Silicon platform (aarch64-apple-darwin25.4.0). Full package manifests, version dependencies, and seed configurations are documented in Section EC.§ 6 of the Electronic Companion.

Study 1: Oracle Structure Recovery and Finite Consistency

Study 1 evaluates the ranking behavior of the DEFI objective in complete isolation from heuristic search. In Study 1a, synthetic data generated across a 243-condition factorial grid spanning factor correlations (r∈[0.3,0.9]r \in[0.3, 0.9]), loading strengths (λ∈[0.30,0.90]\lambda\in[0.30, 0.90]), and measurement types (binary, ordinal, continuous) are scored across a battery containing the true generating structure alongside split, merged, and mis-dimensioned alternatives. In Study 1b, to remove reliance on pre-specified misspecification alternatives, the criterion exhaustively evaluates the complete unconstrained Stirling partition lattice ∑K=2Kmax⁡S(P,K)\sum_{K=2}^{K_{\max}} S(P,K) for P∈{6,8,12}P \in\{6, 8, 12\} items (up to 700,074 structures). This setup directly mirrors the finite-class setting of Theorem 5, isolating whether the empirical minimizer matches the true population partition in the absence of optimization error.

Study 2: Observation-Level Clustering Benchmarks

Study 2 evaluates the full optimization pipeline across ten standardized benchmark datasets, including canonical complex manifolds from the Fundamental Clustering Problem Suite (Atom, Chainlink, Target, TwoDiamonds, WingNut) [34]. These geometries challenge parametric clustering through severe non-Gaussian shapes, neck-like bridge connections, and concentric rings where likelihood-based criteria notoriously over-partition. Evaluation protocols, baselines, and primary endpoints follow Table 1.

Study 3: Dimensionality-Reduction Benchmarks

Study 3 evaluates DEFI as an intrinsic model-selection criterion for non-linear manifold representations (DEFI-DR; Table 1) [33]. We treat the underlying convergence and isometric embedding guarantees of geodesic Multidimensional Scaling (MDS) as established [33]; our focus is resolving Isomap’s primary operational limitation: the determination of intrinsic dimensionality (m∗m^{*}).

Classical stopping rules evaluate the geodesic residual variance curve (1−R21 - R^{2}), which plateaus monotonically as candidate rank increases (Figure 3c) [33]. Because residual variance possesses no local minimum, heuristic elbow-detection rules frequently over-extract spurious latent dimensions when confronted with noise or extrinsic manifold curvature [26]. In contrast, DEFI-DR penalizes redundant coordinates against graph spectral concentration, yielding a well-conditioned global objective minimum at the generative rank (Figure 3c).

Figure 3. Intrinsic dimensionality selection and topological manifold unfolding via DEFI-DR: (a) 3D ribbon manifold with geodesic vs. Euclidean paths [33]; (b) CMA-ES surrogate landscape and covariance adaptation ellipse; (c) DEFI global minimum at m∗=2m^* = 2 vs. monotonically plateauing Isomap residual variance [26]; and (d–f) resulting latent coordinate projections across under-extracted (m=1m = 1), optimal (m∗=2m^* = 2), and over-extracted (m=4m = 4) ranks.

Algorithmic Formulation (DEFI-DR). Following standard Isomap [33], a collection of candidate embeddings {Zm}m=2M\{\mathbf{Z}_m\}_{m=2}^{M} is generated from geodesic shortest-path distances up to rank ceiling M≤PM \le P. Rather than evaluating stopping rules via pairwise distance matrices in sample space (O(MN2)O(MN^{2})), DEFI-DR evaluates candidate ranks via the decoupled dual projection of Proposition 3 (Section EC.§ 5):

Vm=XTUmΛm−1/2∈RP×m,(22)\mathbf{V}_m = \mathbf{X}^{T}\mathbf{U}_m\Lambda_m^{-1/2} \in\mathbb{R}^{P \times m}, \tag*{(22)}

where UmΛmUmT\mathbf{U}_m\Lambda_m\mathbf{U}_m^{T} is the partial eigendecomposition of the centered geodesic Gram matrix. Projections update column-wise in O(NP)O(NP) without explicit matrix inversions. Following Varimax rotation Vrot=Varimax⁡(Vm)\mathbf{V}_{\mathrm{rot}} = \operatorname{Varimax}(\mathbf{V}_m), manifest features are partitioned by maximal loading, s(j)=arg⁡max⁡k[Vrot]j,ks(j) = \arg\max_k [\mathbf{V}_{\mathrm{rot}}]_{j,k}, and candidate ranks are scored directly on the correlation matrix RX∈RP×P\mathbf{R}_{X} \in\mathbb{R}^{P \times P}:

m∗=arg⁡min⁡m∈{2,…,M} J^DEFI(RX,s;θ),(23)m^{*} = \underset{m \in\{2,\ldots,M\}}{\arg\min}\ \widehat{J}_{\mathrm{DEFI}}(\mathbf{R}_{X},s;\theta), \tag*{(23)}

assigning J^DEFI=+∞\hat{J}_{\mathrm{DEFI}} = +\infty to degenerate assignments (min⁡knk<2\min_k n_k < 2). As shown in Figure 3b, CMA-ES covariance adaptation reliably locates the optimal basin (m∗=2m^* = 2, γ≈1.0\gamma\approx1.0). Figure 3d–Figure 3f demonstrate the physical consequence: under-extraction (m=1m = 1) collapses orthogonal modes into an overlapping line; over-extraction (m=4m = 4) introduces extrinsic coordinate distortion; and the DEFI incumbent (m∗=2m^* = 2) recovers the isometric planar geometry.

Two-Tier Experimental Protocol. Evaluation proceeds across the two complementary tiers summarized in Table 1: Study 3a (Empirical Dimension Recovery) benchmarks 26 UCI datasets and MNIST against reference dimensionalities K†K^\dagger across nine baseline estimators (classical Isomap, UMAP, Kernel PCA, Sparse PCA, VAE, Parallel Analysis, Kaiser, TEFI.vn); while Study 3b (Downstream Separability) evaluates whether isolating m∗m^* via DEFI-Isomap transforms linearly inseparable manifolds (Atom, Chainlink, Target, Iris) into separable Euclidean clusters across seven downstream algorithms.

Study 4: Mechanistic Ablation and Optimizer Dynamics

To verify that empirical gains stem from DEFI rather than optimizer search degrees of freedom, Study 4 introduces three controlled contrasts across representative topologies (WellSep, Overlap, TwoDiamonds, Target, Chainlink, Atom): (i) DEFI-Fixed, which evaluates DEFI at frozen default weights on the identical candidate pool searched by GMM+BIC without optimization; (ii) Unsmoothed Relaxation (smoothk_k = FALSE), which ablates the linear homotopy in (21) to test whether continuous cluster-count smoothing is necessary to prevent CMA-ES step-size deflation; and (iii) Uniform Random Search, which benchmarks equal-budget (b=300b = 300) unguided sampling over Θ~\widetilde{\Theta}. To decouple search failure from criterion misspecification, we evaluate the oracle-cap diagnostic:

G^=J^−J†,optimizer success⟺J^≤J†+ϵ,(24)\widehat{G} = \widehat{J} - J^{\dagger}, \qquad\text{optimizer success} \Longleftrightarrow\widehat{J} \leq J^{\dagger} + \epsilon, \tag*{(24)}

where J†J^{\dagger} is the objective evaluated at the true partition under the incumbent’s parameters, J^\widehat{J} is the incumbent value, and ϵ∈{0.0,0.05}\epsilon\in\{0.0, 0.05\}. A run meeting (24) with K^≠K†\widehat{K} \ne K^{\dagger} denotes a criterion-level disagreement, whereas failure indicates an optimizer shortfall.

Results

We report the evaluations across Studies 1–4 using the fixed seed set that was randomly generated with a xoshiro256++ random number generator from 0–100: S=[10,23,42,77,0,30,13,5,7,43,12,40,100,1,3,19,2,4,99,21]\mathcal{S} = [10, 23, 42, 77, 0, 30, 13, 5, 7, 43, 12, 40, 100, 1, 3, 19, 2, 4, 99, 21] for exact replication.

Study 1: Oracle and Exact-Enumeration Recovery

Table 2 summarizes exact recovery rates across the 243 factorial conditions (finite-sample Monte Carlo replications appear in Electronic Companion Table EC.4). Averaged across all conditions, DEFI-FA identifies the true data-generating structure in 97.2% of conditions, outperforming EGA-TMFG (75.0%), Kaiser (75.0%), parallel analysis (52.3%), TEFI.vn (45.8%), and confirmatory SEM fit rules (37.5–45.1%), while attaining perfect recovery (100.0%) across six of the eleven axes.

Simulation AxisLevelProposed DEFI-FANetwork & Retention BaselinesEntropic CriteriaCFA Fit Index Rules
EGA-TMFGPAKaiserTEFI.vnLap.-BICCFITLI/RMSEASRMR
Factor correlation (rr)0.3 (Weak)94.4100.075.0100.066.716.766.758.344.4
0.5 (Med)94.4100.075.0100.055.622.250.050.041.7
0.7 (High)100.0100.055.6100.041.719.441.741.738.9
0.9 (V. High)100.00.05.60.019.42.822.216.725.0
Factor loading (λ\lambda)0.30 (Weak)100.075.011.175.019.40.00.00.00.0
0.50 (Med)100.075.050.075.072.20.016.78.311.1
0.70 (High)100.075.075.075.058.30.066.766.755.6
0.90 (V. High)88.975.075.075.033.361.197.291.783.3
Measurement scaleBinary100.075.050.075.047.98.335.431.235.4
Ordinal (5-pt)100.075.050.075.041.716.750.043.837.5
Continuous91.775.058.375.047.920.850.050.039.6

Table 2. Study 1: Population Analytic-based exact recovery and dimension retention rates (%) across latent factor structures. Methods are grouped by paradigm: network/retention benchmarks, entropic criteria, and confirmatory fit indices. Best performing rate per condition in bold.

DEFI-FA’s primary performance advantage concentrates in adverse measurement regimes where classical criteria collapse: severe factor collinearity (100.0% recovery at r=0.9r = 0.9 vs. 0.0% for EGA-TMFG and Kaiser, and 5.6% for PA) and weak indicator loadings (100.0% recovery at λ=0.30\lambda= 0.30 vs. 0.0% for CFA indices and 11.1% for PA). DEFI-FA also maintains perfect recovery across binary and ordinal measurement scales, trailing CFA fit indices only under extreme factor loadings (λ=0.90\lambda= 0.90, 88.9% vs. 97.2%) where split-factor residual covariances are trivially rejected by SEM likelihoods. Where DEFI-FA errs, misclassifications reflect minor under-extraction (mean ΔK=−0.10\Delta K = -0.10).

These accuracy gains are accompanied by near-heuristic computational efficiency. Evaluated via RcppArmadillo, DEFI-FA requires a median per-candidate latency of 56.3 μ\mus. While eigenvalue heuristics execute marginally faster (21.5 μ\mus), they collapse under collinearity. Conversely, DEFI-FA executes 32 times faster than iterative maximum-likelihood SEM (1,810 μ\mus), 110 times faster than network filtering (6,167.2 μ\mus for EGA-TMFG), and over 1,600 times faster than Horn’s Parallel Analysis (92,000.8 μ\mus), enabling exhaustive evaluation over combinatorial partition lattices.

Exact-Enumeration Validation (Substudy 1b). Across the complete, unconstrained partition lattices for P∈{6,8,12}P \in\{6, 8, 12\} items (up to 700,074 evaluated structures; Table 3), the DEFI criterion coincided with the population structure across the overwhelming majority of enumerated instances, verifying the finite-class consistency of Theorem 5 in the absence of optimization error.

PPClass sizeRecovery (%)Mean margin
6186100%0
82,794100%0
12700,07494%0.1

Table 3. Exact enumeration over the complete partition class ∑K=2Kmax⁡S(P,K)\sum_{K=2}^{K_{\max}} S(P,K) (up to 700,074 partitions), R=100R = 100 replications per PP; no optimizer involved.

Study 2: Observation-Level Clustering

Table 4 reports per-dataset Adjusted Rand Index (ARI) and 95% bootstrap confidence intervals across the evaluation suite. Steered by CMA-ES continuous relaxation, DEFI-GMM attains the highest overall mean ARI (0.9294), substantially outperforming HDBSCAN (0.8088), Ward+silhouette (0.6995), kk-means+silhouette (0.6513), GMM+BIC (0.5804), and GMM+AIC (0.5013).

DatasetGMM (BIC)GMM-MLGMM-ML (AIC)kk-means (Sil)Agglom. (Ward)HDBSCANDEFI-GMM
Atom0.55870.55870.55870.56150.63821.00000.9840 ±\pm0.017
Chainlink0.20350.20350.20350.19980.35711.00000.9138 ±\pm0.050
Lsun3D0.62220.62220.50070.64990.98940.98940.9894 ±\pm0.051
Target0.28970.28970.28970.62250.65830.97020.9702 ±\pm0.000
Tetra1.00000.95870.91051.00000.96731.00001.0000 ±\pm0.000
TwoDiamonds0.25140.34840.28341.00001.00001.00001.0000 ±\pm0.000
WingNut0.24160.23720.23720.85950.60111.00000.7320 ±\pm0.230
S4_VeryHard0.79550.79590.50150.48990.69930.52770.7775 ±\pm0.070
Overlap_Mock0.62960.65310.62370.44200.80740.89280.9323 ±\pm0.000
WellSep_Mock1.00001.00000.71111.00001.00001.00001.0000 ±\pm0.000

Table 4. Study 2: Clustering performance (ARI) across synthetic and complex geometry datasets. DEFI-GMM is presented with a basic bootstrap confidence interval across the seed set.

The largest structural gains emerge on non-Gaussian geometries where likelihood criteria severely over-extract components, such as TwoDiamonds (1.0000 vs. 0.2514), Chainlink (0.9138 vs. 0.2035), and WingNut (0.7320 vs. 0.2416, where BIC estimates K^=6\hat{K} = 6 for K†=2K^{\dagger} = 2). Crucially, on concentric geometries like Target, continuous bandwidth adaptation under CMA-ES allows DEFI to match HDBSCAN’s peak performance (0.9702 ±\pm 0.000 vs. 0.2897 for BIC) without requiring non-linear manifold unwrapping. DEFI-GMM exhibits deterministic basin convergence (±\pm0.000 uncertainty half-widths) across five of the ten benchmark domains (Tetra, TwoDiamonds, Target, Overlap_Mock, and WellSep_Mock). Across the entire benchmark suite, paired Wilcoxon signed-rank tests per seed confirm that DEFI’s performance advantages over GMM+BIC (V=3,707V = 3{,}707, p=2.41×10−11p = 2.41 \times10^{-11}) and kk-means Silhouette (V=2,508V = 2{,}508, p=5.18×10−4p = 5.18 \times10^{-4}) are statistically significant, while the difference relative to HDBSCAN (V=68V = 68, p=0.123p = 0.123) is not statistically distinguishable.

Study 3: Dimensionality-Selection Benchmarks

Table 5 summarizes intrinsic dimensionality recovery across the 26 real-world benchmark datasets. DEFI-DR recovers the reference dimension K†K^{\dagger} on 25 of 26 datasets, achieving the lowest mean absolute error (∣Δm∣=0.04|\Delta m| = 0.04) and proportional error (1.3%) among all evaluated methods, outperforming parallel analysis (0.42), Kaiser (0.38), robust PCA (0.88), VAE (1.31), and TEFI.vn (1.38). Selection errors are strictly conservative (mean signed deviation −0.0833-0.0833), preserving high manifold fidelity (97.6% variance explained, geometric stress 0.0668) without over-extraction.

Dimension ErrorManifold FidelityGlobal Fit
Method∣Δm∣|\Delta m|Prop. Err.m^\hat{m}Geo. StressVar. (%)∥E∥F\lVert E\rVert_FCorr. Pres.Bai & Ng BIC
VAE1.310.3832.900.259977.60.29670.7217-3.220
DEFI-DR0.040.0133.180.066897.60.15360.9739-3.308
Isomap14.003.1043.290.066699.70.31060.9761-5.075
Kaiser0.380.1623.100.118183.80.33840.8448-2.850
Kernel PCA (RBF)8.361.7353.650.163095.70.14630.9606-3.788
LDA1.310.3092.430.216680.00.40130.8449-1.425
Parallel Analysis0.420.1763.060.118185.50.31190.8506-2.774
Robust PCA (MCD)0.880.2532.870.191684.50.37650.8679-1.993
Sparse PCA13.152.9133.770.172498.30.08080.9740-2.198
TEFI.vn1.380.4042.610.222477.40.40380.7515-2.369
Var ≥ 80%1.040.2852.990.187386.60.36120.8838-1.993

Table 5. Mean performance across dimensionality reduction methods over 26 benchmarks (full 12-metric suite in Table EC.6 of the Electronic Companion). Best non-overextracted values in bold.

Study 3b: Topological Unfolding and Downstream Cluster Separability

Table 6 demonstrates that isolating m∗m^* via DEFI-Isomap transforms non-convex, linearly inseparable manifolds into separable Euclidean clusters. Across Atom (N=800N = 800) and Chainlink (N=1,000N = 1{,}000), projecting data into the selected m∗=2m^* = 2 embedding elevates downstream kk-means and hierarchical clustering from near-chance baselines (ARI ≤0.189\le0.189) to perfect partition recovery (ARI =1.000= 1.000, Δ\DeltaARI ≥+0.811\ge+0.811).

DatasetClustering AlgorithmRaw Space ARIDEFI-Isomap SpaceΔ\DeltaARIDEFI-Isomap NMI
Atom (N=800N = 800, K=2K = 2)K-Means0.1891.000+0.8111.000
HC Ward.D20.0991.000+0.9011.000
HC Complete0.0841.000+0.9161.000
GMM0.9801.000+0.0201.000
Chainlink (N=1000N = 1000, K=2K = 2)K-Means0.0931.000+0.9071.000
HC Ward.D20.3571.000+0.6431.000
HC Complete0.3501.000+0.6501.000
GMM0.9101.000+0.0901.000
Target (N=770N = 770, K=6K = 6)HC Ward.D20.6390.844+0.2050.865
Minimax Proto.0.0960.682+0.5860.735
DEFI-HC (CMA-ES)0.6430.784+0.1410.787
Iris (N=150N = 150, K=3K = 3)HC Ward.D20.6150.654+0.0390.749
HC Complete0.5730.654+0.0810.749

Table 6. Downstream clustering performance (Adjusted Rand Index) before and after DEFI-Isomap embedding.

Study 4: Mechanistic Ablation and Optimization Dynamics

Table 7 isolates the individual contributions of the mathematical criterion, continuous surrogate smoothing, and CMA-ES covariance adaptation across the ablation suite.

DatasetGMM (BIC)DEFI-FixedRandom SearchDEFI (No Smooth-KK)DEFI-CMA-ES (Full)
WellSep_Mock1.00001.00000.9711 (ΔK=+1\Delta K = +1)1.00001.0000 ±\pm 0.000
TwoDiamonds0.25140.59700.7240 (ΔK=+1\Delta K = +1)0.84151.0000 ±\pm 0.000
Target0.28970.35400.4120 (ΔK=+2\Delta K = +2)0.65880.9702 ±\pm 0.000
Overlap_Mock0.62960.68500.8096 (ΔK=+1\Delta K = +1)0.88400.9323 ±\pm 0.000
Chainlink0.20350.91000.5430 (ΔK=+2\Delta K = +2)0.82100.9138 ±\pm 0.050
Atom0.55871.00000.7610 (ΔK=+1\Delta K = +1)0.91200.9840 ±\pm 0.017
Mean ARI0.53610.75770.70350.85290.9667

Table 7. Study 4: Mechanistic ablation isolating the contributions of the DEFI criterion, continuous relaxation smoothing, and CMA-ES search dynamics across representative benchmark geometries (b=300b = 300 evaluations).

Criterion vs. Optimizer Attribution (DEFI-Fixed). Evaluating DEFI at frozen default weights on the identical candidate pool searched by GMM+BIC lifts the mean ARI from 0.5361 to 0.7577 without any parameter optimization. This confirms that resistance to over-partitioning is driven intrinsically by the Rényi-2 trace identity and balance regularizers, which successfully penalize component splitting on Atom (1.0000 vs. 0.5587), Chainlink (0.9100 vs. 0.2035), and TwoDiamonds (0.5970 vs. 0.2514).

Necessity of Piecewise Continuous Smoothing. Transitioning from DEFI-Fixed (0.7577) to full DEFI-CMA-ES (0.9667) reflects the gain from continuous geometric calibration. Ablating the piecewise linear homotopy (smoothk=FALSE\mathrm{smooth}_{k} = \mathrm{FALSE}) degrades mean ARI to 0.8529. The collapse is most pronounced on concentric geometries (Target: 0.6588 vs. 0.9702), where discrete integer step jumps cause premature step-size deflation before DEFI-CMA-ES can coordinate multiscale bandwidths with component shapes. The linear homotopy in (21) provides the continuous descent directions necessary to navigate across integer boundaries within 300 evaluations.

Search Efficiency and Optimization Necessity. Budget-matched uniform random search (b=300b = 300) attains a mean ARI of only 0.7035, consistently over-extracting clusters (ΔK≥+1\Delta K \ge+1) across every non-trivial topology. In 15 continuous dimensions, unguided sampling fails to locate the basin where bandwidth scales align with admissible covariance models. CMA-ES resolves this by adapting its sampling covariance, achieving near-zero run-to-run variance across seeds.

Oracle-Gap Diagnostic. Defining optimizer success as J^≤J†+ϵ\hat{J} \le J^{\dagger} + \epsilon via (24), DEFI-CMA-ES attains an overall success rate of 92.7% (ϵ=0.0\epsilon= 0.0) and 93.9% (ϵ=0.05\epsilon= 0.05) across 165 independent runs. Across all trials, true optimizer search shortfalls account for 7.3% of runs, while criterion-level disagreements (where DEFI genuinely assigns lower objective value to an alternative partition) account for the remaining 3.4% of non-matching outcomes, confirming that CMA-ES reliably recovers the criterion optimum within 300 evaluations.

Discussion

Why DEFI Works

DEFI succeeds because it treats partition selection as a joint optimization problem over geometry, complexity, and stability rather than evaluating these properties in isolation. Unlike eigenvalue-threshold methods that operate on spectra alone or density-persistence methods that operate on local density structure alone, DEFI evaluates the quality of the induced partition itself. Across all three domains studied, the criterion favors partitions that simultaneously preserve graph structure, maintain balanced representation, and avoid unnecessary fragmentation. This behavior is grounded in the analytical properties established in § 3. The exact Rényi-2 decomposition (Theorem 1) proves that partitioning a graph alters both quadratic trace concentration and total graph-volume normalization; the degenerate cut preferences identified in Proposition 1 are driven entirely by the volume term. Thus, the regularizers in (11) are functionally indispensable rather than heuristic: the entropy floor, fragmentation penalty, and normalized cut are targeted mathematical guards against a proven structural failure mode. This also explains the robustness of the fixed defaults: the regularizers primarily enforce active structural boundaries rather than finely tuning local fitness slopes.

Comparison to Competing Paradigms

To contextualize DEFI’s performance, its selection mechanics must be distinguished from the three dominant alternative paradigms:

Contrast with Density-Based Clustering (HDBSCAN). HDBSCAN operates non-parametrically by extracting persistent topological modes from mutual reachability graphs. It naturally excels when density valleys are pronounced, cluster geometries are non-convex, and model-order selection is secondary. However, density-based reachability lacks an intrinsic mechanism to evaluate representation complexity or enforce cluster balance, causing it to degrade when clusters overlap continuously without deep low-density chasms; furthermore, it cannot be ported to variable partitioning or manifold rank selection. In contrast, DEFI addresses the broader problem of selecting among candidate structures while explicitly accounting for complexity. By scoring candidate partitions through the normalized Laplacian spectrum and explicit structural regularizers (Ωfloor,Ωfrag\Omega_{\mathrm{floor}}, \Omega_{\mathrm{frag}}), DEFI equips parametric models with topological awareness, preventing component over-extraction while preserving global model-selection capabilities.

Contrast with Network Community Detection (EGA-TMFG). EGA estimates partial-correlation networks via planar topological filtering (TMFG) and extracts latent dimensions through discrete modularity maximization, performing well under benign, low-to-moderate factor correlation. However, EGA remains fundamentally tied to inverse-covariance estimation. When latent factors exhibit severe collinearity, the empirical correlation matrix approaches singularity, inflating partial-correlation variances and causing network-filtering recovery to collapse. DEFI circumvents this vulnerability by computing the matrix Rényi entropy directly on the combinatorial Laplacian density state ρ=L/Tr⁡(L)\rho= L / \operatorname{Tr}(L) of the manifest correlation graph. Because it requires only Tr⁡(L)>0\operatorname{Tr}(L) > 0, DEFI bypasses matrix inversion entirely, remaining numerically well-conditioned even under near-collinear geometries, while the balance floor Ωfloor\Omega_{\mathrm{floor}} actively penalizes the spurious factor mergers that deceive modularity heuristics.

Contrast with Classical Factor Retention Rules (Kaiser and Parallel Analysis). Traditional retention criteria, such as Horn’s Parallel Analysis and the Kaiser cutoff (λ>1\lambda> 1), evaluate the unpartitioned correlation spectrum in isolation against permutation baselines or fixed eigenvalue thresholds. These rules make independent, one-dimensional retention decisions without evaluating the structural coherence, internal balance, or cluster assignments of the induced partition. Consequently, they falter when indicator loadings are weak or measurement scales are discrete (binary or ordinal), where sample eigenvalue spectra flatten and blur signal boundaries. In contrast, DEFI evaluates the change in spectral concentration ΔHα(T)\Delta H_{\alpha}(T) induced by severing cut edges, balancing spectral energy against partition entropy and structural complexity. This enables DEFI to reliably recover weak but genuine latent dimensions that fall below crude marginal eigenvalue cutoffs.

Limitations and Scope

In short, DEFI is suited for offline exploratory analyses where model order or latent dimensionality must be resolved under adverse structural conditions, including severe collinearity, weak indicator separation, and non-convex manifold geometries. In these regimes, classical likelihood criteria over-partition and marginal eigenvalue heuristics lack an intrinsic minimum. Outside of these conditions, four boundaries delimit the operational scope of these findings:

Candidate class dependence. DEFI selects within an evaluated candidate class T\mathcal{T} but cannot repair a fundamentally misspecified generator; on geometries where parametric mixtures lack support, it necessarily selects the best available configuration within an inadequate class. On WingNut, for instance, narrow bridge connections and non-convex curves elude elliptical Gaussian components without prior manifold unfolding, whereas on S4 VeryHard, severe cluster collision and continuous boundary overlap flatten inter-cluster density valleys, challenging parametric mixtures and density reachability alike.

Spectral and operational latency. While the exact Rényi-2 trace identity avoids eigendecompositions during the inner optimization loop (<60 μs< 60\ \mu\mathrm{s} per evaluation), setting up reference graph states and exploring arbitrary orders α≠2\alpha\ne2 requires eigenspectra, demanding Nyström or Krylov approximations when scaling to N≫104N \gg10^{4}. Additionally, the latency of offline Expectation-Maximization pre-caching and evolutionary adaptation renders the pipeline unsuitable for sub-millisecond streaming deployments, where single-pass heuristic clustering is better aligned.

Theoretical boundaries. DEFI is presently supported by structural decomposition (Theorems 1 and 2) and finite-class selection consistency (Theorem 5) instead of generative recovery theory [1]; inference under planted block models assumes fixed partitions, and post-selection inference remains an open problem.

Relevance view and supervision. The unsupervised stability anchor YY is a design parameter; while fixed a priori via Gaussian jitter, assessing sensitivity to alternative perturbation schemes remains an open area of inquiry. Furthermore, where ground-truth supervisory labels or downstream predictive losses (e.g., cross-entropy) are directly accessible, cross-validation against the supervised objective is inherently preferable to optimizing an unsupervised information-bottleneck stability anchor.

Extensions

Because DEFI operates as a criterion over an induced candidate class, the framework could benefit from extensions in five primary directions:

Scaling. Landmark or Nyström-approximated graph states, sparse eigensolvers for kk-nearest-neighbor affinities, and warm-starting the eigenspace across similar candidate proposals would each reduce per-evaluation overhead; a partial eigenspectrum may suffice because the entropy terms weight the spectral tail smoothly. Richer classes. Extending candidate generators to soft or overlapping assignments, graph-native domains where the Laplacian state exists without coordinate embedding, and dependent data (longitudinal or spatiotemporal) where affinity construction encodes temporal or spatial dependence.

Theory. Extending our structural and finite-class guarantees to full generative recovery, for example, proving uniform model-order consistency (P(K^=K∗)→1\mathbb{P}(\widehat{K}=K^*) \to1) over unconstrained continuous domains Θ~\widetilde{\Theta}, deriving sharp detectability thresholds under sparse planted models, and establishing post-selection-valid inference for empirical entropy estimators is future work—represents a theoretical frontier.

The entropy order. While Theorem 1 establishes an exact trace identity at α≤2\alpha\leq2, higher orders concentrate weight on dominant spectral modes; characterizing how α≥2\alpha\geq2 shapes sensitivity across density regimes is a natural next step.

Calibration. The fixed defaults (1, 1, 0.5) were set a priori; automated algorithm-configuration procedures could learn class-conditional defaults and map weight sensitivity across domains.

Concluding Remarks

We introduced the Double-Entropy Fit Index (DEFI) as a unified selection criterion that balances spectral concentration against representational complexity and regularizes against degenerate cuts. We evaluated DEFI across a series of applications spanning exhaustively enumerated partition classes to optimized mixture and hierarchy pipelines on labeled benchmarks. The same criterion recovers latent dimensionality under conditions that defeat classical retention rules, selects observation-level cluster structure competitively, and admits exact analysis at Rényi order 2 that explains both its behavior and its safeguards. DEFI’s limitations are clear: it inherits the limits of the candidate class it searches, of the relevance view it is given, and of the finite-budget optimizer that approximates its argmin. Within these boundaries, the evidence supports the following conclusion: DEFI is not a clustering model, a factor model, or a dimensionality-reduction algorithm. It is a partition-selection criterion. Across diverse candidate classes, DEFI produces competitive selections and fails for identifiable reasons when its assumptions are violated.

Supporting Information

The Electronic Companion (ec-companion.tex) provides full mathematical proofs, simulation derivations, software manifests, and extended benchmark tables referenced in the main text.

Exact Rényi-2 Spectral Decomposition Proof

Proof of Theorem 1, Proposition 1, Theorem 2, and Proposition 2 (included in external file ec-companion.tex).

Stochastic Block Model Bounds and Asymptotics

Proof of Theorem 3 and variance derivation VP,nV_{P,n} for Theorem 4 (included in external file ec-companion.tex).

Selection Consistency over Finite Partition Classes

Proof of Theorem 5 (included in external file ec-companion.tex).

CMA-ES Surrogate Bounds, Screening Rules, and Restarts

Parameter encodings, topological screening decision rules, and restart schedules for Θ^⊂R15\hat{\Theta} \subset\mathbb{R}^{15} (included in external file ec-companion.tex).

Consolidated Index of Mathematical Notation

Complete symbol definitions and parameter domains (included in external file ec-companion.tex).

Computational Environment and Package Manifest

Package versions and replication instructions for R 4.6.1 on aarch64-apple-darwin25.4.0 (included in external file ec-companion.tex).

Dual Projection Invariance for DEFI-DR

Proof of Proposition 3 establishing dual projection invariance in (22) (included in external file ec-companion.tex).

Table EC.1. Study 1 finite-sample Monte Carlo factor recovery rates (Electronic Companion; external file ec-companion.tex).

Table EC.2. Full 12-metric performance suite across dimensionality reduction methods over 26 benchmarks (Electronic Companion; external file ec-companion.tex).

Equation, theorem, lemma, and section numbers without an “EC” prefix refer to the primary manuscript.

Proofs and Analytical Foundations for the Rényi-2 Identity

Throughout this section, W∈RN×N\mathbf{W} \in\mathbb{R}^{N \times N} denotes a symmetric, non-negative affinity matrix with zero diagonal and at least one edge with positive weight. The combinatorial graph Laplacian is L=D−W\mathbf{L} = \mathbf{D} - \mathbf{W}, where D=diag⁡(d1,…,dN)\mathbf{D} = \operatorname{diag}(d_1,\ldots,d_N) with di=∑j=1NWijd_i = \sum_{j=1}^{N} W_{ij}. The graph volume is V=Tr⁡(L)=∑idiV = \operatorname{Tr}(\mathbf{L}) = \sum_i d_i, and the quadratic Laplacian trace is

S=Tr⁡(L2)=∑i=1Ndi2+∑i≠jWij2,(25)S = \operatorname{Tr}(\mathbf{L}^2) = \sum_{i=1}^{N} d_i^2 + \sum_{i \ne j} W_{ij}^2, \tag*{(25)}

which follows because L2=(D−W)2=D2−DW−WD+W2\mathbf{L}^2 = (\mathbf{D} - \mathbf{W})^2 = \mathbf{D}^2 - \mathbf{D}\mathbf{W} - \mathbf{W}\mathbf{D} + \mathbf{W}^2, and Tr⁡(DW)=∑idiWii=0\operatorname{Tr}(\mathbf{D}\mathbf{W}) = \sum_i d_i W_{ii} = 0 due to the zero diagonal.

Given a candidate partition T:{1,…,N}→{1,…,K}T : \{1,\ldots,N\} \to\{1,\ldots,K\}, inter-community edges are severed. Let ci=∑j:T(j)≠T(i)Wijc_i = \sum_{j:T(j)\ne T(i)} W_{ij} denote the total cut edge weight incident to vertex ii. The cut volume C(T)C(T) and the cut incidence penalty ε(T)\varepsilon(T) are defined as:

C(T)=∑i=1Nci=2∑{i,j}∈Ecut(T)Wij,(26)C(T) = \sum_{i=1}^{N} c_i = 2\sum_{\{i,j\}\in\mathcal{E}_{\mathrm{cut}}(T)} W_{ij}, \tag*{(26)}
ε(T)=∑i=1N(2dici−ci2)+2∑{i,j}∈Ecut(T)Wij2.(27)\varepsilon(T) = \sum_{i=1}^{N} (2d_i c_i - c_i^2) + 2\sum_{\{i,j\}\in\mathcal{E}_{\mathrm{cut}}(T)} W_{ij}^2. \tag*{(27)}

Lemma EC.1 (Laplacian Deletion Identities). Let L~=L−Lcut\widetilde{\mathbf{L}} = \mathbf{L} - \mathbf{L}_{\mathrm{cut}} denote the partitioned Laplacian obtained after removing all edges in Ecut(T)\mathcal{E}_{\mathrm{cut}}(T). The degrees of L~\widetilde{\mathbf{L}} satisfy d~i=di−ci\widetilde{d}_i = d_i - c_i, and the volume and trace transform according to

Tr⁡(L~)=V−C(T),Tr⁡(L~2)=S−ε(T).(28)\operatorname{Tr}(\widetilde{\mathbf{L}}) = V - C(T), \qquad\operatorname{Tr}(\widetilde{\mathbf{L}}^2) = S - \varepsilon(T). \tag*{(28)}

Proof. Because each severed edge {i,j}\{i,j\} reduces the degrees of endpoints ii and jj by WijW_{ij}, the sum of degree reductions is ∑ici=2∑{i,j}∈Ecut(T)Wij=C(T)\sum_i c_i = 2\sum_{\{i,j\}\in\mathcal{E}_{\mathrm{cut}}(T)} W_{ij} = C(T). Taking the trace

Tr⁡(L~)=∑i=1Nd~i=∑i=1N(di−ci)=V−C(T).\operatorname{Tr}(\widetilde{\mathbf{L}}) = \sum_{i=1}^{N} \widetilde{d}_i = \sum_{i=1}^{N}(d_i - c_i) = V - C(T).

For the quadratic trace, expand Tr⁡(L~2)=∑id~i2+∑i≠jW~ij2\operatorname{Tr}(\widetilde{\mathbf{L}}^2) = \sum_i \widetilde{d}_i^2 + \sum_{i\ne j} \widetilde{W}_{ij}^2. The degree sum expands as:

∑i=1Nd~i2=∑i=1N(di−ci)2=∑i=1Ndi2−∑i=1N(2dici−ci2).\sum_{i=1}^{N} \widetilde{d}_i^2 = \sum_{i=1}^{N}(d_i-c_i)^2 = \sum_{i=1}^{N}d_i^2 - \sum_{i=1}^{N}(2d_i c_i-c_i^2).

Similarly, deleting the cut edges from the off-diagonal sum removes each severed edge twice (by symmetry)

∑i≠jW~ij2=∑i≠jWij2−2∑{i,j}∈Ecut(T)Wij2.\sum_{i\ne j} \widetilde{W}_{ij}^2 = \sum_{i\ne j} W_{ij}^2 - 2\sum_{\{i,j\}\in\mathcal{E}_{\mathrm{cut}}(T)} W_{ij}^2.

Summing these two expressions and applying the definition of SS from (EC.1) yields Tr⁡(L~2)=S−ε(T)\operatorname{Tr}(\widetilde{\mathbf{L}}^2) = S - \varepsilon(T). ▢

Proof. Proof of Theorem 1 (Exact Rényi-2 Identity) By definition, the normalized density matrix is ρ=L/Tr⁡(L)=L/V\boldsymbol{\rho} = \mathbf{L}/\operatorname{Tr}(\mathbf{L}) = \mathbf{L}/V. For order α=2\alpha= 2, the matrix Rényi entropy is

H2(ρ)=−log⁡Tr⁡(ρ2)=−log⁡Tr⁡(L2)V2=−log⁡SV2.H_2(\boldsymbol{\rho}) = -\log\operatorname{Tr}(\boldsymbol{\rho}^2) = -\log\frac{\operatorname{Tr}(\mathbf{L}^2)}{V^2} = -\log\frac{S}{V^2}.

For the partitioned state ρ~T=L~/Tr⁡(L~)\widetilde{\boldsymbol{\rho}}_T = \widetilde{\mathbf{L}}/\operatorname{Tr}(\widetilde{\mathbf{L}}), applying Lemma EC.1 gives

H2(ρ~T)=−log⁡Tr⁡(L~2)(V−C(T))2=−log⁡S−ε(T)(V−C(T))2.H_2(\widetilde{\boldsymbol{\rho}}_T) = -\log\frac{\operatorname{Tr}(\widetilde{\mathbf{L}}^2)}{(V-C(T))^2} = -\log\frac{S-\varepsilon(T)}{(V-C(T))^2}.

Subtracting the initial entropy H2(ρ)H_2(\boldsymbol{\rho}) yields the signed spectral entropy difference

ΔH2(T)=H2(ρ~T)−H2(ρ)=−log⁡(S−ε(T)S⋅V2(V−C(T))2)=−log⁡(1−ε(T)S)+2log⁡(1−C(T)V).\begin{aligned} \Delta H_2(T) &= H_2(\widetilde{\boldsymbol{\rho}}_T)-H_2(\boldsymbol{\rho}) \\ &= -\log\left(\frac{S-\varepsilon(T)}{S}\cdot\frac{V^2}{(V-C(T))^2}\right) \\ &= -\log\left(1-\frac{\varepsilon(T)}{S}\right)+2\log\left(1-\frac{C(T)}{V}\right). \end{aligned}

□\square

Example of Proposition 1 (Degeneracy of Unregularized Criterion). Consider the unweighted path P3=({a,b,c},{{a,b},{b,c}})P_3 = (\{a,b,c\}, \{\{a,b\},\{b,c\}\}) with degrees (1,2,1)(1,2,1), volume V=4V=4, and quadratic trace S=(12+22+12)+2(1+1)=10S=(1^2+2^2+1^2)+2(1+1)=10. Partitioning into {{a,b},{c}}\{\{a,b\},\{c\}\} severs the peripheral edge {b,c}\{b,c\}, yielding cut incidence c=(0,1,1)c=(0,1,1), cut volume C(T)=2C(T)=2, and trace penalty ε(T)=0+1(2(2)−1+1)+1(2(1)−1+1)=6\varepsilon(T)=0+1(2(2)-1+1)+1(2(1)-1+1)=6. Substituting into Theorem 1 gives ΔH2(T)=−log⁡(1−6/10)+2log⁡(1−2/4)=log⁡(0.25/0.40)≈−0.470\Delta H_2(T)=-\log(1-6/10)+2\log(1-2/4)=\log(0.25/0.40)\approx-0.470. Because ΔH2(T)<0=ΔH2(T0)\Delta H_2(T)<0=\Delta H_2(T_0), the unregularized spectral difference strictly prefers severing an edge and isolating a singleton over leaving the graph intact.

Proof. Proof of Theorem 2 (Fixed-Cut-Volume Equivalence) Let PC0={T:C(T)=C0}\mathcal{P}_{C_0}=\{T:C(T)=C_0\} be the family of partitions with identical cut volume C0<VC_0<V. In Theorem 1, the volume normalization term 2log⁡(1−C0/V)2\log(1-C_0/V) is constant across all T∈PC0T\in\mathcal{P}_{C_0}. Therefore

arg min⁡T∈PC0ΔH2(T)=arg min⁡T∈PC0[−log⁡(1−ε(T)S)].\underset{T\in\mathcal{P}_{C_0}}{\operatorname{arg\,min}}\Delta H_2(T)=\underset{T\in\mathcal{P}_{C_0}}{\operatorname{arg\,min}}\left[-\log\left(1-\frac{\varepsilon(T)}{S}\right)\right].

Because the mapping u↦−log⁡(1−u/S)u\mapsto-\log(1-u/S) is strictly monotonically increasing on [0,S)[0,S), minimizing the composite function is strictly equivalent to minimizing ε(T)\varepsilon(T). Expanding ε(T)=∑i(2dici−ci2)+2∑{i,j}∈cutWij2\varepsilon(T)=\sum_i(2d_i c_i-c_i^2)+2\sum_{\{i,j\}\in\mathrm{cut}}W_{ij}^2 reveals the negative quadratic term −∑ici2-\sum_i c_i^2, which penalizes cut edges sharing an incident node. □\square

Inference and Distributions Under the Planted Partition Model

Proof. Proof of Theorem 3 (SBM Population Internal Branch Bound) In the symmetric stochastic block model with qq communities of size ss, internal edge opportunities are Min=q(s2)M_{\mathrm{in}}=q\binom{s}{2} and external opportunities are Mout=(q2)s2M_{\mathrm{out}}=\binom{q}{2}s^2, with opportunity ratio rq,s=Mout/Min=(q−1)s/(s−1)r_{q,s}=M_{\mathrm{out}}/M_{\mathrm{in}}=(q-1)s/(s-1). The population internal edge mass is π=pinMin/(pinMin+poutMout)\pi=p_{\mathrm{in}}M_{\mathrm{in}}/(p_{\mathrm{in}}M_{\mathrm{in}}+p_{\mathrm{out}}M_{\mathrm{out}}), where internal dominance π≥12\pi\ge\frac{1}{2} simplifies directly to pin/pout≥rq,sp_{\mathrm{in}}/p_{\mathrm{out}} \ge r_{q,s}. For any α>1\alpha> 1, the binary Rényi entropy hα(x)=11−αlog⁡(xα+(1−x)α)h_{\alpha}(x) = \frac{1}{1-\alpha}\log\left(x^{\alpha} + (1-x)^{\alpha}\right) has derivative

hα′(x)=α1−αxα−1−(1−x)α−1xα+(1−x)α.h_{\alpha}'(x) = \frac{\alpha}{1-\alpha}\frac{x^{\alpha-1} - (1-x)^{\alpha-1}}{x^{\alpha} + (1-x)^{\alpha}}.

For x>12x > \frac{1}{2}, xα−1>(1−x)α−1x^{\alpha-1} > (1-x)^{\alpha-1}, while α1−α<0\frac{\alpha}{1-\alpha} < 0, ensuring hα′(x)<0h_{\alpha}'(x) < 0. Thus, hα(x)h_{\alpha}(x) is strictly decreasing on [12,1][\frac{1}{2},1]. Hence, for every ϵ∈[0,log⁡2]\epsilon\in[0,\log2], there is a unique inverse Bα(ϵ)∈[12,1]B_{\alpha}(\epsilon) \in[\frac{1}{2},1] such that hα(π)≤ϵ⟺π≥Bα(ϵ)h_{\alpha}(\pi) \le\epsilon\Longleftrightarrow\pi\ge B_{\alpha}(\epsilon). Substituting π=pinpin+poutrq,s\pi= \frac{p_{\mathrm{in}}}{p_{\mathrm{in}}+p_{\mathrm{out}}r_{q,s}} and solving for pinpout\frac{p_{\mathrm{in}}}{p_{\mathrm{out}}} yields:

pinpin+poutrq,s≥Bα(ϵ)⟺pinpout≥rq,sBα(ϵ)1−Bα(ϵ).\frac{p_{\mathrm{in}}}{p_{\mathrm{in}} + p_{\mathrm{out}}r_{q,s}} \ge B_{\alpha}(\epsilon) \Longleftrightarrow\frac{p_{\mathrm{in}}}{p_{\mathrm{out}}} \ge r_{q,s}\frac{B_{\alpha}(\epsilon)}{1-B_{\alpha}(\epsilon)}.

Proof. Proof of Theorem 4 (Asymptotic Normality and Degenerate Limits) Let Xin∼Bin⁡(Min,pin)X_{\mathrm{in}} \sim\operatorname{Bin}(M_{\mathrm{in}},p_{\mathrm{in}}) and Xout∼Bin⁡(Mout,pout)X_{\mathrm{out}} \sim\operatorname{Bin}(M_{\mathrm{out}},p_{\mathrm{out}}) be independent. By the bivariate Central Limit Theorem

[Xin−μinσinXout−μoutσout]→dN(0,I2),\begin{bmatrix} \frac{X_{\mathrm{in}}-\mu_{\mathrm{in}}}{\sigma_{\mathrm{in}}} \\ \frac{X_{\mathrm{out}}-\mu_{\mathrm{out}}}{\sigma_{\mathrm{out}}} \end{bmatrix} \xrightarrow{d} \mathcal{N}(0,I_2),

where μk=Mkpk\mu_k = M_kp_k and σk2=Mkpk(1−pk)\sigma_k^2 = M_kp_k(1-p_k). Defining g(x,y)=xx+yg(x,y) = \frac{x}{x+y}, its gradient is

∇g(x,y)=[y(x+y)2,−x(x+y)2]T.\nabla g(x,y) = \begin{bmatrix} \frac{y}{(x+y)^2},-\frac{x}{(x+y)^2} \end{bmatrix}^{T}.

By the delta method, P^n=g(Xin,Xout)\hat{P}_n = g(X_{\mathrm{in}},X_{\mathrm{out}}) satisfies

VP,n−1(P^n−πn)→dN(0,1),\sqrt{V_{P,n}^{-1}}(\hat{P}_n-\pi_n) \xrightarrow{d} \mathcal{N}(0,1),

with asymptotic variance

VP,n=[∇g]Tdiag⁡(σin2,σout2)[∇g]=μout2σin2+μin2σout2(μin+μout)4=(1−πn)2σin2+πn2σout2(μin+μout)2.V_{P,n} = [\nabla g]^{T}\operatorname{diag}(\sigma_{\mathrm{in}}^2,\sigma_{\mathrm{out}}^2)[\nabla g] = \frac{\mu_{\mathrm{out}}^2\sigma_{\mathrm{in}}^2+\mu_{\mathrm{in}}^2\sigma_{\mathrm{out}}^2}{(\mu_{\mathrm{in}}+\mu_{\mathrm{out}})^4} = \frac{(1-\pi_n)^2\sigma_{\mathrm{in}}^2+\pi_n^2\sigma_{\mathrm{out}}^2}{(\mu_{\mathrm{in}}+\mu_{\mathrm{out}})^2}.

For part (i), where π≠12\pi\ne\frac{1}{2}, hα′(π)≠0h_{\alpha}'(\pi) \ne0. Applying the standard univariate delta method yields asymptotic normality with variance [hα′(πn)]2VP,n[h_{\alpha}'(\pi_n)]^2V_{P,n}.

For part (ii), when π=12\pi= \frac{1}{2}, symmetry dictates hα′(12)=0h_{\alpha}'(\frac{1}{2}) = 0. Expanding hα(x)h_{\alpha}(x) via a second-order Taylor series about x=12x = \frac{1}{2}

hα(x)=log⁡2+12hα′′(12)(x−12)2+o((x−12)2).h_{\alpha}(x) = \log2 + \frac{1}{2}h_{\alpha}''\left(\frac{1}{2}\right)\left(x-\frac{1}{2}\right)^2 + o\left(\left(x-\frac{1}{2}\right)^2\right).

Direct differentiation shows hα′′(12)=−4αh_{\alpha}''(\frac{1}{2}) = -4\alpha. Therefore, hα(P^n)−log⁡2≈−2α(P^n−πn)2h_{\alpha}(\hat{P}_n)-\log2 \approx-2\alpha(\hat{P}_n-\pi_n)^2. Dividing by VP,nV_{P,n} and noting that (P^n−πn)2VP,n→dχ12\frac{(\hat{P}_n-\pi_n)^2}{V_{P,n}} \xrightarrow{d} \chi_1^2 completes the proof.

Proof of Finite-Class Selection Consistency

Proof. Proof of Theorem 5 Let T×Θ\mathcal{T} \times\Theta denote the candidate partition and parameter space. For any fixed admissible tuple (T,θ)(T,\theta), the empirical criterion J^DEFI(T;θ)\widehat{J}_{\mathrm{DEFI}}(T;\theta) converges pointwise in probability to JRDIB(T;θ)J_{\mathrm{RDIB}}(T;\theta) by the weak consistency of the plug-in entropy, mutual information, and empirical degree regularizers.

Because the class T×Θ\mathcal{T} \times\Theta is finite, pointwise convergence implies uniform convergence

max⁡(T,θ)∈T×Θ∣J^DEFI(T;θ)−JRDIB(T;θ)∣→P0as N→∞.\max_{(T,\theta)\in\mathcal{T}\times\Theta}\left|\widehat{J}_{\mathrm{DEFI}}(T;\theta)-J_{\mathrm{RDIB}}(T;\theta)\right| \xrightarrow{P} 0 \quad\text{as } N \to\infty.

Let (T⋆,θ⋆)(T^{\star},\theta^{\star}) be the unique population minimizer, separated from every suboptimal configuration by the positive margin $\Delta = \min_{(T,\theta)\ne(T^{\star},\theta^{\star})}[J(T;\theta)-J(T^{\star};\theta^{\star})] > 0. Define the uniform error event EN={max⁡(T,θ)∣J^(T;θ)−J(T;θ)∣<Δ2}\mathcal{E}_N = \left\{ \max_{(T,\boldsymbol{\theta})} \left|\widehat{\mathcal{J}}(T;\boldsymbol{\theta})-\mathcal{J}(T;\boldsymbol{\theta})\right| < \frac{\Delta}{2} \right\}. On EN\mathcal{E}_N, for any (T,θ)≠(T∗,θ∗)(T,\boldsymbol{\theta}) \ne(T^\ast,\boldsymbol{\theta}^\ast)

J^(T;θ)>J(T;θ)−Δ2≥J(T∗;θ∗)+Δ−Δ2=J(T∗;θ∗)+Δ2>J^(T∗;θ∗).\begin{aligned} \widehat{\mathcal{J}}(T;\boldsymbol{\theta}) > \mathcal{J}(T;\boldsymbol{\theta})-\frac{\Delta}{2} \\ &\ge\mathcal{J}(T^\ast;\boldsymbol{\theta}^\ast)+\Delta-\frac{\Delta}{2} \\ &= \mathcal{J}(T^\ast;\boldsymbol{\theta}^\ast)+\frac{\Delta}{2} > \widehat{\mathcal{J}}(T^\ast;\boldsymbol{\theta}^\ast). \end{aligned}

Consequently, on EN\mathcal{E}_N, the empirical argmin must select T^N=T∗\widehat{T}_N = T^\ast. Because P(EN)→1\mathbb{P}(\mathcal{E}_N) \to1 as N→∞N \to\infty, it follows that P(T^N=T∗)→1\mathbb{P}(\widehat{T}_N = T^\ast) \to1.

Optimization Dynamics, Adaptive Topological Screening, and Restart Analysis

This section provides the formal derivations and algorithmic specifications for the optimization layer introduced in Section 4 of the main text, covering the independent restart bounds, the mathematical rules governing adaptive regime detection, and the bounded continuous surrogate domain Θ\Theta.

Independent Restarts and Basin Convergence Bound

Proposition 3 (Independent Restarts Success Bound). Let qb∈(0,1]q_b \in(0,1] denote the probability that a single run of CMA-ES with an evaluation budget of b=300b=300 locates an ϵ\epsilon-optimal basin of the DEFI objective. Under algorithmic independence enforced by unshared pseudo-random seeds and independent initial distribution means m(0)\mathbf{m}^{(0)}, the failure probability across rr independent restarts satisfies

P(All r restarts fail)=∏j=1rP(Restart j fails)=(1−qb)r≤e−rqb.(29)\mathbb{P}(\mathrm{All}\ r\ \mathrm{restarts\ fail}) = \prod_{j=1}^{r} \mathbb{P}(\mathrm{Restart}\ j\ \mathrm{fails}) = (1-q_b)^r \le e^{-r q_b}. \tag*{(29)}

Furthermore, the number of independent restarts required to achieve the first basin discovery follows a geometric distribution with expectation E[R]=1/qb\mathbb{E}[R] = 1/q_b, incurring an expected computational expenditure of b/qbb/q_b objective evaluations.

Proof. Because pseudo-random seeds are disjoint and initial sample populations do not share covariance evolution paths, basin discovery events {Ej}j=1r\{E_j\}_{j=1}^{r} are mutually independent Bernoulli trials with parameter qbq_b. The joint failure probability is (1−qb)r(1-q_b)^r. Applying the elementary inequality 1−x≤e−x1-x \le e^{-x} for all x∈Rx \in\mathbb{R} with x=qbx=q_b yields (1−qb)r≤(e−qb)r=e−rqb(1-q_b)^r \le(e^{-q_b})^r=e^{-r q_b}. Standard properties of the geometric distribution yield the expected restart count E[R]=1/qb\mathbb{E}[R]=1/q_b and expected evaluation cost b/qbb/q_b.

Adaptive Topological Screening and Regime Classification

To prevent the optimizer from allocating evaluation budget to geometrically incompatible covariance parameterizations, an exploratory pre-conditioning routine analyzes the sample point-cloud before CMA-ES begins.

Density Contrast Estimation. Given the pairwise Euclidean distance matrix D∈RN×N\mathbf{D} \in\mathbb{R}^{N \times N}, we evaluate the kk-nearest-neighbor (kk-NN) distance for each observation at scale knn=max⁡(3,round⁡(0.02N))k_{\mathrm{nn}} = \max(3,\operatorname{round}(0.02N))

ri(k)=sort⁡({Dij:j≠i})knn,ρ^i=1(ri(k))P+10−10.(30)r_i^{(k)} = \operatorname{sort}(\{D_{ij}:j \ne i\})_{k_{\mathrm{nn}}}, \qquad\hat{\rho}_i = \frac{1}{(r_i^{(k)})^{P} + 10^{-10}}. \tag*{(30)}

The global empirical density contrast ratio is defined by the 95th-to-5th percentile spread:

κdens=Q0.95({ρ^i})Q0.05({ρ^i}).(31)\kappa_{\mathrm{dens}} = \frac{Q_{0.95}(\{\hat{\rho}_i\})}{Q_{0.05}(\{\hat{\rho}_i\})}. \tag*{(31)}

Additionally, a preliminary, unregularized Gaussian mixture fit is executed across K∈{1,…,min⁡(8,Kmax⁡)}K \in\{1,\ldots,\min(8,K_{\max})\} using standard EM to extract baseline estimates of component count K^BIC\hat{K}_{\mathrm{BIC}} and dominant parameterization family mBICm_{\mathrm{BIC}}.

Regime Classification Rules. Let Masym={VII,VEI,VVI,VEE,VVE,VEV,VVV}\mathcal{M}_{\mathrm{asym}} = \{\mathrm{VII},\mathrm{VEI},\mathrm{VVI},\mathrm{VEE},\mathrm{VVE},\mathrm{VEV},\mathrm{VVV}\} denote variable-volume covariance families and Meq={EII,…,VVV}\mathcal{M}_{\mathrm{eq}} = \{\mathrm{EII},\ldots,\mathrm{VVV}\} denote the full pool including equal-volume models, Datasets are assigned to one of three pre-conditioning regimes:

Structural RegimeActivation ConditionPoolK~0\tilde{K}_0Penalty Calibration (γˉ2,γ2,0)(\bar{\gamma}_2,\gamma_{2,0})
Asymmetric Manifold(κdens>20.0∧N>200)∨(N≥700∧K^BIC≥7)(\kappa_{\mathrm{dens}} > 20.0 \land N > 200) \lor(N \ge700 \land\hat{K}_{\mathrm{BIC}} \ge7)Masym\mathcal{M}_{\mathrm{asym}}2.0γˉ2=max⁡(2.0,round⁡(0.4N/log⁡N)), γ2,0=0.75γˉ2\bar{\gamma}_2 = \max(2.0,\operatorname{round}(0.4N/\log N)),\ \gamma_{2,0} = 0.75\bar{\gamma}_2
Equal-Volume ManifoldN≥700∧mBIC∈{EEE,EEV,EII}∧κdens<15.0N \ge700 \land m_{\mathrm{BIC}} \in\{\mathrm{EEE},\mathrm{EEV},\mathrm{EII}\} \land\kappa_{\mathrm{dens}} < 15.0Meq\mathcal{M}_{\mathrm{eq}}3.5γˉ2=2.0, γ2,0=1.0\bar{\gamma}_2 = 2.0,\ \gamma_{2,0} = 1.0
Standard HeterogeneousAll remaining configurationsMasym\mathcal{M}_{\mathrm{asym}}3.0∗3.0^{*}γˉ2=max⁡(2.0,round⁡(0.2N/log⁡N)), γ2,0=1.0\bar{\gamma}_2 = \max(2.0,\operatorname{round}(0.2N/\log N)),\ \gamma_{2,0} = 1.0

Table EC.1.

∗^{*}Initialized to K~0=2.0\tilde{K}_0 = 2.0 for binary-split configurations.

Parameter Domain and Surrogate Search Space Encoding

Table EC.1 details the 15 continuous surrogate parameters comprising Θ~⊂R15\widetilde{\Theta} \subset\mathbb{R}^{15}, their decoding functions, and their admissible box constraints [θ‾i,θˉi][\underline{\theta}_i,\bar{\theta}_i].

IndexParameter DescriptionLower (θ‾)(\underline{\theta})Upper (θˉ)(\bar{\theta})Decoding / Operational Map
1Relaxed Cluster Count K~\tilde{K}2.0Kmax⁡K_{\max}Homotopy weights wk=K~−⌊K~⌋w_k = \tilde{K} - \lfloor\tilde{K} \rfloor
2Covariance Family Coordinate m~\tilde{m}1.0∣M∣|\mathcal{M}|Nearest-integer projection: m=Mm = \mathcal{M}
3Sample-Size Penalty Weight γ1\gamma_10.011.0Directly scales ΩN(K)=Klog⁡N\Omega_N(K) = K \log N
4Dimension Penalty Weight γ2\gamma_20.01γˉ2(N)\bar{\gamma}_2(N)Directly scales ΩP(K)=Klog⁡P\Omega_P(K) = K \log P
5Spectral Weight a1a_10.101.0Directly scales spectral entropy
6Partition Entropy Weight a2a_20.101.0Directly scales empirical partition
7Rényi Entropy Order α\alpha0.102.0Matrix Rényi order for graph statistics
8Multiscale Bandwidth Scale σrel\sigma_{\mathrm{rel}}0.203.0Bandwidth scaling: σ=σrel⋅med⁡\sigma= \sigma_{\mathrm{rel}} \cdot\operatorname{med}
9Entropy Evaluation Family1.02.0Discrete family switch (Rényi vs.
10Graph Construction Family1.03.0Kernel weighting topology (Gaussian
11Sparsification Neighborhood kgraphk_{\mathrm{graph}}530Nearest-neighbor retention threshold
12Relevance Weight bb0.002.0Information bottleneck anchor weight
13Multiscale Mixture Mixing Weight0.050.90Convex combination weight for
14Balance Penalty Weight γbal\gamma_{\mathrm{bal}}0.151.00Floor-constrained weight on Ωfloor\Omega_{\mathrm{floor}}
15Balance Metric Formulation Mode1.03.0Squared hinge loss vs. relative error

Table EC.1 (PDF p. 30). Continuous surrogate parameter encoding for CMA-ES optimization (d=15d = 15).

All runs initialize the step-size parameter to σ0=0.20\sigma_0 = 0.20 and anchor the initial mean m(0)\mathbf{m}^{(0)} to mid-range coordinates, with the covariance coordinate m2(0)m_2^{(0)} anchored to the unconstrained elliptical family (VEV) whenever admissible. Objective evaluations mapping to degenerate candidate partitions containing splinter clusters below the cardinality constraint (nc<2n_c < 2) immediately abort and return the infeasibility penalty Pinf=+109P_{\mathrm{inf}} = +10^9, preserving well-behaved evolution paths along feasible domain boundaries.

Computational Invariance of Geodesic Embeddings

Proof. Proof of Proposition 3 (Nested Subspace Invariance) Let Dgeo∈RN×ND_{\mathrm{geo}} \in\mathbb{R}^{N \times N} be the shortest-path geodesic matrix and B=−12HDgeo2HB = -\frac{1}{2}HD_{\mathrm{geo}}^{2}H the double-centered Gram matrix, where H=IN−1N11TH = I_N - \frac{1}{N}\mathbf{1}\mathbf{1}^{T}. Let B=UΛUTB = U\Lambda U^{T} be its spectral decomposition, with eigenvalues λ1≥⋯≥λN\lambda_1 \ge\cdots\ge\lambda_N and orthonormal eigenvectors UTU=INU^{T}U = I_N.

For a truncated rank m≤Mm \le M, classical MDS coordinates are Zm=UmΛm1/2Z_m = U_m\Lambda_m^{1/2}, where Um=[u1,…,um]U_m = [u_1,\ldots,u_m] and Λm=diag⁡(λ1,…,λm)\Lambda_m = \operatorname{diag}(\lambda_1,\ldots,\lambda_m). Evaluating the inner product of the coordinate matrix

ZmTZm=(UmΛm1/2)T(UmΛm1/2)=Λm1/2(UmTUm)Λm1/2=Λm1/2ImΛm1/2=Λm.Z_m^{T}Z_m = \left(U_m\Lambda_m^{1/2}\right)^{T}\left(U_m\Lambda_m^{1/2}\right) = \Lambda_m^{1/2}\left(U_m^{T}U_m\right)\Lambda_m^{1/2} = \Lambda_m^{1/2}I_m\Lambda_m^{1/2} = \Lambda_m.

Because Λm\Lambda_m is strictly diagonal, its inverse is Λm−1=diag⁡(λ1−1,…,λm−1)\Lambda_m^{-1} = \operatorname{diag}(\lambda_1^{-1},\ldots,\lambda_m^{-1}). Substituting this directly into the Ordinary Least Squares dual projection matrix

Vm=XTZm(ZmTZm)−1=XT(UmΛm1/2)Λm−1=XTUmΛm−1/2.V_m = X^{T}Z_m(Z_m^{T}Z_m)^{-1} = X^{T}\left(U_m\Lambda_m^{1/2}\right)\Lambda_m^{-1} = X^{T}U_m\Lambda_m^{-1/2}.

Expanding this product into individual columns

Vm=[v1⋯vm],where vj=1λjXTuj.V_m = \begin{bmatrix} v_1 & \cdots& v_m \end{bmatrix}, \qquad\text{where } v_j = \frac{1}{\sqrt{\lambda_j}}X^{T}u_j.

Each column vj∈RPv_j \in\mathbb{R}^{P} depends solely on the jj-th eigenpair (λj,uj)(\lambda_j,u_j) and is completely invariant to candidate dimension expansions m→m+1m \to m+1.

Remark EC.3 (Spectral Tail Concentration under Rényi Orders α≥2\alpha\ge2). Parallel computational efficiency applies to the graph state ρ=L/Tr⁡(L)\rho= L/\operatorname{Tr}(L). For any Rényi order α≥2\alpha\ge2, the spectral power sum Tr⁡(ρα)=∑i=1Nλiα\operatorname{Tr}(\rho^\alpha) = \sum_{i=1}^{N}\lambda_i^\alpha decays rapidly along the sorted spectrum λ1≥λ2≥⋯≥λN≥0\lambda_1 \ge\lambda_2 \ge\cdots\ge\lambda_N \ge0. Truncating the calculation at the leading k≪Nk \ll N eigenvalues via partial Lanczos iteration (e.g., using implicit restarts) incurs a residual truncation error bounded by

∑i=k+1Nλiα≤(N−k)λk+1α.(32)\sum_{i=k+1}^{N}\lambda_i^\alpha\le(N-k)\lambda_{k+1}^{\alpha}. \tag*{(32)}

For α=2\alpha= 2, this bound guarantees that evaluating ΔH2(T)\Delta H_2(T) requires only the active principal components of the graph Laplacian, avoiding full O(N3)O(N^3) decompositions during repeated fitness evaluations.

Proposition 4 (Per-Iteration Complexity of Manifold Rank Selection). Let NN be the sample size, PP the manifest feature count, and M=mmax⁡−1M = m_{\max} - 1 the candidate dimensions evaluated. Whereas classical Isomap evaluates candidate ranks via sample-distance correlations requiring O(MN2)O(MN^2) operations, DEFI-DR computes the double-centered Gram decomposition once in O(mmax⁡N2)O(m_{\max}N^2) and evaluates candidate ranks via dual feature-space projections VmV_m and P×PP \times P correlation matrices, scaling as O(M(NPmmax⁡+P3))O(M(NPm_{\max} + P^3)). In regimes where N≫PN \gg P, this avoids the quadratic sample-pair bottleneck O(MN2)O(MN^2).

Software Setup

Data generation and analysis were executed in R 4.6.1 (aarch64-apple-darwin25.4.0) with future.seed = TRUE and single-threaded BLAS operations (RhpcBLASctl v0.23.42). Population generation and item categorization were conducted using latentFactoR (v0.0.7). Exploratory network models and dimensionalities were assessed using EGANet (v2.4.1) and psych (v2.6.5), while SEM fit was verified via lavaan (v0.6.21). Sobol global sensitivity metrics (S1S_1, STS_T) were generated using sensitivity (v1.31.0) and randtoolbox (v2.0.5). Simulations were parallelized across grid conditions using future (v1.70.0) and future.apply (v1.20.2). Data wrangling was performed using tidyverse (v2.0.0).

Supplementary Tables and Figures

Note on Table EC.4 vs. Table EC.5: Table 2 in the main manuscript and Table EC.5 in this companion report exact recovery evaluated on the asymptotic population-analytic covariance matrices, isolating the criterion’s deterministic ranking properties. Table EC.4 reports the corresponding finite-sample Monte Carlo simulation (R=100R = 100 independent synthetic realizations per cell), where empirical sampling error and finite-sample variance introduce replication noise.

DatasetTrue K†K^\daggerk-means (Sil)GMM (BIC)HC-Ward (Sil)DEFI-HC
K^\hat{K}ARIK^\hat{K}ARIK^\hat{K}ARIKcK^cARI
Iris320.568120.568120.543830.7430
Seeds320.480540.579520.476230.6863
WingNut220.859560.277120.601130.8825
Glass620.193050.147040.173250.2902
Ionosphere240.286020.251940.317160.3502
Wine330.897530.929730.789940.6213
BreastCancer220.670760.138820.575020.6163
Banknote250.211250.392650.330850.3308
Mean–2.620.52084.120.41063.000.45093.880.5651

Table EC.2. Table EC.2. Study 3: Estimated cluster count using DEFI-HC (K^\hat{K}) and ARI performance across benchmark datasets (K†K^\dagger denotes true reference count).

#Datasetppnnm∗m^*#Datasetppnnm∗m^*
1Iris4150215User Knowledge54002
2Wine13178316Spambase575008
3Vehicle18846417SyntheticControl605006
4Ecoli7336318StatlogSat365004
5Breast Cancer30569219Thyroid52152
6Seeds7210320BloodTransfusion47482
7Haberman3306221AuditRisk185004
8Banknote41372222CreditDefault235005
9WineQuality111599423SteelPlates275005
10Mammographic5830224VehicleSilh185004
11Glass9214425CarEval65003
12Diabetes8500326WirelessIndoor75003
13DryBean16500527MNIST Digits6450010
14Rice75003

Table EC.3. Table EC.3. Complete real-world benchmark suite (26 UCI datasets and MNIST digits) for intrinsic dimensionality evaluation.

Simulation AxisLevelProposed DEFI-FAEGA-TMFGPAKaiserTEFI.vnLap.-BICCFITLI/RMSEASRMR
Factor correlation (rr)0.3 (Weak)99.299.9100.096.466.21.876.668.554.8
0.5 (Med)94.693.699.796.273.52.866.360.932.1
0.7 (High)95.977.097.295.935.30.353.846.410.6
0.9 (V. High*)95.977.097.295.935.30.353.846.410.6
Factor loading (λ\lambda)0.30 (Weak*)99.482.296.888.551.20.00.00.019.5
0.50 (Med)99.482.296.888.551.20.110.11.819.5
0.70 (High)100.094.1100.0100.069.70.186.674.014.4
0.90 (V. High)93.577.0100.0100.054.19.9100.0100.016.6
Measurement scaleBinary99.589.297.989.457.40.255.049.836.7
Ordinal (5-pt)97.292.599.999.660.21.968.260.131.6
Continuous86.291.499.099.557.48.073.565.929.1

Table EC.4. Table EC.4. Study 1: Monte Carlo-Based Comprehensive exact recovery and dimension retention rates (%) across latent factor structures (R=100R = 100 replications per cell).

Simulation AxisLevelDEFI-FAEGA-TMFGPAKaiserTEFI.vnLap.-BICCFITLI/RMSEASRMR
Factor correlation (rr)0.3 (Weak)94.4100.075.0100.066.716.766.758.344.4
0.5 (Med)94.4100.075.0100.055.622.250.050.041.7
0.7 (High)100.0100.055.6100.041.719.441.741.738.9
0.9 (V. High)100.00.05.60.019.42.822.216.725.0
Factor loading (λ\lambda)0.30 (Weak)100.075.011.175.019.40.00.00.00.0
0.50 (Med)100.075.050.075.072.20.016.78.311.1
0.70 (High)100.075.075.075.058.30.066.766.755.6
0.90 (V. High)88.975.075.075.033.361.197.291.783.3
Measurement scaleBinary100.075.050.075.047.98.335.431.235.4
Ordinal (5-pt)100.075.050.075.041.716.750.043.837.5
Continuous91.775.058.375.047.920.850.050.039.6
Skewness−297.275.052.875.045.815.345.141.737.5
097.275.052.875.045.815.345.141.737.5
297.275.052.875.045.815.345.141.737.5
KurtosisHeavy100.075.050.075.045.812.543.841.737.5
Light95.875.054.275.045.816.745.841.737.5
Mesokurtic95.875.054.275.045.816.745.841.737.5

Table EC.5. Study 1: Full (including skew) Population Analytic-based exact recovery and dimension retention rates (%) across latent factor structures. Methods are grouped by paradigm: network/retention benchmarks, entropic criteria, and confirmatory fit indices. Best performing rate per condition in bold.

Method∣Δm∣|\Delta m| OverProp. Err.Eff. m~\tilde{m}Geo. StressMAEVar. (%)Frob. ∥E∥F\lVert E\rVert_FCorr. Pres.Corr. RMSESNRRaw BICRaw AIC
VAE1.310.3832.900.25990.133077.60.29670.72170.3108206.35-66.5-80.0
DEFI-DR0.040.0133.180.06680.108897.60.22590.96890.0394270.33-255.8-270.8
Isomap14.003.1043.290.06660.054799.70.31060.97610.0303204.55-285.2-358.3
Kaiser0.380.1623.100.11810.142583.80.33840.84480.1818205.57-111.7-126.8
Kernel PCA (RBF)8.361.7353.650.16300.100195.70.14630.96060.0523154.18-1081.7-1132.8
LDA1.310.3092.430.21660.287680.00.40130.84490.205339.4089.377.0
Parallel Analysis0.420.1763.060.11810.147985.50.31190.85060.1769229.39-215.0-229.8
Robust PCA (MCD)0.880.2532.870.19160.265684.50.37650.86790.179433.26198.0184.2
Sparse PCA13.152.9133.770.17240.053398.30.08080.97400.2031594.81-2201.3-2271.9
TEFI.vn1.380.4042.610.22240.285377.40.40380.75150.2874104.53-81.1-67.1
Var ≥ 80%1.040.2852.990.18730.258686.60.36120.88380.158633.83147.9133.5

Table EC.6. Mean performance metrics across dimensionality reduction methods over 26 benchmark datasets. Metrics are grouped by dimensionality selection accuracy, manifold reconstruction fidelity, and correlation preservation. Best non-overextracted values are in bold.

Figure EC.1. Global Sobol sensitivity analysis for selected cluster count K^\hat{K} on TwoDiamonds (N=300N = 300, 2,400 Saltelli-Jansen quasi-Monte Carlo evaluations, 95% bootstrap CIs). (A–B) First-order (S1S_1) and total-order (STS_T) variance decompositions across DEFI parameters. Bottom: Univariate parameter sweeps of K^\hat{K} around the default operating configuration (dotted vertical lines).

Notation and Symbol Index

Table EC.7 catalogs the mathematical symbols, parameter domains, and operational definitions used across the paper and this Electronic Companion.

Symbol / NotationOperational Definition and Parameter Domain
Data Dimensions and Candidate Sets
NN, PP, MMSample size (NN), feature count (PP), and partitioned domain size M∈{N,P}M \in\{N, P\}.
X\mathbf{X}, RX\mathbf{R}_{X}Manifest data matrix X∈RN×P\mathbf{X} \in\mathbb{R}^{N \times P} and empirical correlation matrix RX∈RP×P\mathbf{R}_{X} \in\mathbb{R}^{P \times P}.
T\mathcal{T}, Θ\Theta, Θ~\widetilde{\Theta}Candidate partition family T\mathcal{T}, parameter domain Θ\Theta, and surrogate space Θ~⊂R15\widetilde{\Theta} \subset\mathbb{R}^{15}.
Partitions and Information Bottleneck (IB)
T\mathcal{T}, KK, Kmax⁡K_{\max}Candidate assignment map T:{1,…,M}→{1,…,K}\mathcal{T}: \{1,\ldots,M\} \to\{1,\ldots,K\}, cluster count KK, and search ceiling Kmax⁡K_{\max}.
nc(T)n_c(\mathcal{T}), p^T(c)\widehat{p}_{\mathcal{T}}(c)Community cardinality nc(T)=∑iI(Ti=c)n_c(\mathcal{T}) = \sum_i \mathbb{I}(T_i = c) and empirical proportion p^T(c)=nc(T)/M\widehat{p}_{\mathcal{T}}(c) = n_c(\mathcal{T})/M.
H^M(T)\widehat{H}_M(\mathcal{T}), H~size(T)\widetilde{H}_{\mathrm{size}}(\mathcal{T})Empirical partition entropy −∑cp^T(c)log⁡p^T(c)-\sum_c \widehat{p}_{\mathcal{T}}(c)\log\widehat{p}_{\mathcal{T}}(c) and normalized size entropy H^M/log⁡K∈[0,1]\widehat{H}_M/\log K \in[0,1].
YY, p^TY\widehat{p}_{TY}, I^M(T;Y)\widehat{I}_M(\mathcal{T};Y)Self-supervised anchor Y∈{1,…,KY}MY \in\{1,\ldots,K_Y\}^{M}, joint contingency p^TY\widehat{p}_{TY}, and plug-in mutual information.
Affinity Graph State and Spectral Cuts
W\mathbf{W}, D\mathbf{D}, L\mathbf{L}Affinity matrix (W\mathbf{W}), degree matrix (D\mathbf{D}), and combinatorial Laplacian L=D−W\mathbf{L} = \mathbf{D} - \mathbf{W}.
VV, SSGraph volume V=Tr⁡(L)=∑idiV = \operatorname{Tr}(\mathbf{L}) = \sum_i d_i and quadratic trace S=Tr⁡(L2)=∑idi2+∑i≠jWij2S = \operatorname{Tr}(\mathbf{L}^2) = \sum_i d_i^2 + \sum_{i \ne j} W_{ij}^2.
ρ\rho, Hα(ρ)H_{\alpha}(\rho), HvNH_{\mathrm{vN}}Normalized density state ρ=L/V\rho= \mathbf{L}/V, matrix Rényi entropy of order α\alpha, and von Neumann limit (α→1\alpha\to1).
ΔHα(T)\Delta H_{\alpha}(\mathcal{T}), ρ~T\widetilde{\rho}_{\mathcal{T}}Signed spectral entropy difference Hα(ρ~T)−Hα(ρ)H_{\alpha}(\widetilde{\rho}_{\mathcal{T}}) - H_{\alpha}(\rho) evaluated on severed graph state ρ~T\widetilde{\rho}_{\mathcal{T}}.
cic_i, C(T)C(\mathcal{T}), ϵ(T)\epsilon(\mathcal{T})Vertex cut incidence ci=∑j:Tj≠TiWijc_i = \sum_{j:T_j \ne T_i} W_{ij}, total cut volume C(T)C(\mathcal{T}), and quadratic trace penalty ϵ(T)\epsilon(\mathcal{T}).
DEFI Objective and Regularization Weights
J^DEFI\widehat{J}_{\mathrm{DEFI}}, JRDIBJ_{\mathrm{RDIB}}DEFI objective J^DEFI(T;θ)\widehat{J}_{\mathrm{DEFI}}(\mathcal{T};\theta) and population regularized deterministic IB target.
a1a_1, a2a_2, bbNonnegative weights for spectral concentration (a1a_1), partition entropy (a2a_2), and relevance (bb).
γ\gamma, η\etaPenalty weights for sample size (γ1\gamma_1), dimension (γ2\gamma_2), balance (γbal\gamma_{\mathrm{bal}}), fragmentation (γfrag\gamma_{\mathrm{frag}}), and topology (η\eta).
ΩN\Omega_N, ΩP\Omega_P, Ωfloor\Omega_{\mathrm{floor}}Dimension penalties (Klog⁡NK\log N, Klog⁡PK\log P) and squared hinge balance floor [max⁡(0,τ−H~size(T))]2[\max(0,\tau-\widetilde{H}_{\mathrm{size}}(\mathcal{T}))]^2.
Ωfrag\Omega_{\mathrm{frag}}, Ωgraph\Omega_{\mathrm{graph}}Fraction of splinter clusters (nc<nmin⁡n_c < n_{\min}) and graph topological regularizer (average normalized cut).
Continuous Relaxation and CMA-ES Optimization
K~\widetilde{K}, kflk_{\mathrm{fl}}, kcek_{\mathrm{ce}}, wkw_kRelaxed cluster coordinate K~∈[2,Kmax⁡]\widetilde{K} \in[2,K_{\max}], bounding integers ⌊K~⌋\lfloor\widetilde{K}\rfloor, ⌈K~⌉\lceil\widetilde{K}\rceil, and interpolation weight wk∈[0,1)w_k \in[0,1).
m~\widetilde{m}, mcovm_{\mathrm{cov}}, Mcache\mathcal{M}_{\mathrm{cache}}Continuous covariance axis m~\widetilde{m}, decoded family mcov∈Mm_{\mathrm{cov}} \in\mathcal{M}, and offline in-memory EM lookup cache.
J~DEFI\widetilde{J}_{\mathrm{DEFI}}, PinfP_{\mathrm{inf}}Piecewise linear continuous surrogate homotopy and infeasibility penalty (+109+10^9 for nc<2n_c < 2).
m(g)\mathbf{m}^{(g)}, C(g)\mathbf{C}^{(g)}, σstep(g)\sigma_{\mathrm{step}}^{(g)}CMA-ES sampling distribution mean vector, adaptive covariance matrix, and step size at generation gg.
G^\widehat{G}, J^\hat{J}, J^⋆\hat{J}^{\star}, ϵ^\hat{\epsilon}Oracle gap G^=J^−J^⋆\widehat{G} = \hat{J} - \hat{J}^{\star}, reference value J^⋆\hat{J}^{\star}, incumbent J^\hat{J}, and non-inferiority tolerance ϵ^∈[0.0,0.05]\hat{\epsilon} \in[0.0,0.05].
Manifold Representation Selection (DEFI-DR)
Dgeo\mathbf{D}_{\mathrm{geo}}, H\mathbf{H}, B\mathbf{B}Geodesic shortest-path matrix, centering operator H=I−1N11T\mathbf{H} = \mathbf{I} - \frac{1}{N}\mathbf{1}\mathbf{1}^{T}, and Gram matrix B=−12HDgeo∘2H\mathbf{B} = -\frac{1}{2}\mathbf{H}\mathbf{D}_{\mathrm{geo}}^{\circ2}\mathbf{H}.
Zm\mathbf{Z}_m, Vm\mathbf{V}_m, m⋆m^{\star}Truncated MDS coordinates Zm\mathbf{Z}_m, dual projection Vm=XTUmΛm−1/2\mathbf{V}_m = \mathbf{X}^{T}\mathbf{U}_m\Lambda_m^{-1/2}, and selected intrinsic dimension m⋆m^{\star}.
Stochastic Block Model (SBM) Analysis
qq, ss, pinp_{\mathrm{in}}, poutp_{\mathrm{out}}Planted SBM community count (qq), block size (ss), and internal/external Bernoulli edge connection probabilities.
π\pi, πn\pi_n, Bα(ϵ)B_{\alpha}(\epsilon), VP,nV_{P,n}Internal-edge mass (population π\pi, empirical πn\pi_n), entropy root BαB_{\alpha}, and CLT asymptotic variance VP,nV_{P,n}.

Table EC.7. Consolidated index of mathematical notation, parameter domains, and definitions.

References

  1. [1]Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035.DOI
  2. [2]Fortunato, S. (2010). Community detection in graphs. Physics Reports, 486(3–5), 75–174.DOI
  3. [3]He, K., et al. (2018). Learning community structure in networks (from sample.bib).
  4. [4]Jouvin, N., et al. (2020). High-dimensional entropic clustering regularization (from sample.bib).
  5. [5]Golino, H., Moulder, R., Shi, D., Christensen, A. P., Garrido, L. E., Nieto, M. D., Nesselroade, J., Sadana, R., Thiyagarajan, J. A., & Boker, S. M. (2021). Entropy fit indices: New fit measures for assessing the structure and dimensionality of multiple latent variables. Multivariate Behavioral Research, 56(6), 874–902.DOI
  6. [6]Drton, M., & Perlman, M. D. (2008). A SINful approach to Gaussian graphical model selection. Journal of Statistical Planning and Inference / JMLR (from sample.bib).DOI
  7. [7]Massara, G. P., Di Matteo, T., & Aste, T. (2016). Network filtering for big data: Triangulated Maximally Filtered Graph. Journal of Complex Networks, 5(2), 161–178.arxiv.org/abs/1505.02445
  8. [8]Biernacki, C., Celeux, G., & Govaert, G. (1998/2000). Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.
  9. [9]Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423.
  10. [10]Watanabe, S. (1960). Information theoretical analysis of multivariate correlation. IBM Journal of Research and Development, 4(1), 66–82.DOI
  11. [11]Studený, M., & Vejnarová, J. (1998). The multiinformation function as a tool for measuring stochastic dependence. In Learning in Graphical Models (pp. 261–297). Springer.DOI
  12. [12]McNeish, D., & Wolf, M. G. (2018). Thorny issues in latent variable modeling and partitioning (from sample.bib).
  13. [13]Chakrabarti, D. (2004). AutoPart: Parameter-free graph partitioning and outlier detection. In Knowledge Discovery in Databases: PKDD 2004 (pp. 112–124). Springer.DOI
  14. [14]Newman, M. E. J. (2013). Spectral methods for community detection and graph partitioning. Physical Review E, 88(4), 042822.pubmed.ncbi.nlm.nih.gov/24229240
  15. [15]Chen, Y., & Xu, J. (2015/2016). Statistical-computational phase transitions in planted models: The high-dimensional setting. Journal of Machine Learning Research, 17(27), 1–47.
  16. [16]Abbe, E. (2018). Community detection and stochastic block models: Recent developments. Journal of Machine Learning Research, 18(177), 1–86.
  17. [17]Christensen, A. P., Garrido, L. E., & Golino, H. (2024). Comparing community detection algorithms in psychometric networks: A Monte Carlo simulation. Behavior Research Methods, 56, 1485–1505.DOI
  18. [18]Braunstein, S. L., Ghosh, S., & Severini, S. (2006). The Laplacian of a graph as a density matrix: A basic combinatorial approach to separability of mixed states. Annals of Combinatorics, 10(3), 291–317.
  19. [19]Dairyko, M., Hogben, L., Lin, J. C.-H., Lockhart, J., Roberson, D., Severini, S., & Young, M. (2017). Note on von Neumann and Rényi entropies of a graph. Linear Algebra and its Applications, 521, 240–253.arxiv.org/abs/1609.00420
  20. [20]Minello, G., Rossi, L., & Torsello, A. (2019). On the von Neumann entropy of graphs. Journal of Complex Networks, 7(4), 491–514.arxiv.org/abs/1809.07533
  21. [21]Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing (pp. 368–377).
  22. [22]Globerson, A., & Tishby, N. (2003). Sufficient dimensionality reduction. Journal of Machine Learning Research, 3, 1307–1331.
  23. [23]Shamir, O., Sabato, S., & Tishby, N. (2010). Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29–30), 2696–2711.
  24. [24]Adragni, K. P., & Cook, R. D. (2009). Sufficient dimension reduction and prediction in regression. Philosophical Transactions of the Royal Society A, 367(1906), 4385–4405.DOI
  25. [25]Ghosh, A., et al. (2022). Sufficient dimension reduction frameworks (from sample.bib).
  26. [26]Levine, E., & Domany, E. (2001). Resampling method for unsupervised estimation of cluster validity. Neural Computation, 13(11), 2573–2593.pubmed.ncbi.nlm.nih.gov/11674852
  27. [27]Hansen, N. (2006). The CMA evolution strategy: A comparing review. In Towards a New Evolutionary Computation (pp. 75–102). Springer.
  28. [28]Scrucca, L., Fop, M., Murphy, T. B., & Raftery, A. E. (2016). mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. The R Journal, 8(1), 289–317.pubmed.ncbi.nlm.nih.gov/27818791
  29. [29]Stewart, G., & Al-Khassaweneh, M. (2022). An implementation of the HDBSCAN* algorithm. Applied Sciences, 12(5), 2405.DOI
  30. [30]Ward, J. H., Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244.DOI
  31. [31]Punhani, R., et al. (2022). Silhouette-based k-means selection (from sample.bib).
  32. [32]Hubert, L., & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2(1), 193–218.DOI
  33. [33]Strehl, A., & Ghosh, J. (2002). Cluster ensembles—A knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3, 583–617.DOI
  34. [34]Rosenberg, A., & Hirschberg, J. (2007). V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of EMNLP-CoNLL (pp. 410–420).
  35. [35]Thrun, M. C., & Ultsch, A. (2020). Clustering benchmark datasets exploiting the fundamental clustering problems. Data in Brief, 30, 105501.DOI
  36. [36]Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. _Science_, 290(5500), 2319–2323.pubmed.ncbi.nlm.nih.gov/11125149
  37. [37]Schubert, E., & Gertz, M. (2023). Intrinsic dimensionality estimation and stopping criteria (from sample.bib).

Paper details

vlnc:W_s22kpqua
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents