WorksComputer Science
The comparison of density estimation for transfer learning in Banach, Hilbert, Euclidean spaces
Abstract
Domain adaptation or DA methods are indispensable when target labels or computational resources are scarce, yet the domain kernel density and subspace alignment approaches completely assume a Hilbertian geometry. They measure distributional discrepancy with symmetric L2 divergences over Euclidean feature spaces or reproducing kernel Hilbert spaces (RKHS). This preprint summarizes the manuscript Non-Hilbertian Domain Adaptation. Total Variation Alignment via L1 Density Regularization on Feature Manifolds, which proposes Banach Manifold Transfer Learning or BMTL. BMTL makes two coupled design decisions. The first one is that features are embedded on a geodesic manifold rather than treated as points of a flat Euclidean space, and the second one is that the density matching regularizer is formulated in an L1 Banach norm, whose minimization is equivalent, by Scheffe's theorem, to contracting the Total Variation or TV distance between domain densities, a bounded metric that weights all regions of the sample space uniformly. The L2 formulation, by contrast, squares deviations and therefore amplifies errors in the tails, over emphasizing density outliers that do not reflect meaningful distributional shift. Experiments on MNIST-USPS, COIL20, and Office+Caltech10 show consistent gains; the L1 variant beats an otherwise identical L2 ablation on every task, for example 92.44% vs. 89.80% on USPS->MNIST and surpasses the strongest baseline DMDA by +3.68% on average over twelve Office+Caltech transfers. This document situates the contribution with an explicit comparison of Euclidean, Hilbert, and Banach geometries.
Introduction and Motivation
Unsupervised domain adaptation addresses the mismatch between a labeled source distribution and an unlabeled target distribution , a classifier trained on the source must generalize to the target. The dominant strategy is to learn an embedding under which the source and target feature distributions become statistically indistinguishable, while preserving task discriminative structure on the labeled source ([21, 13]).
Two architectural regimes exist. Deep DA methods DDC/MMD, DANN, CDAN, Deep-CORAL ([3, 4] (Hu et al. 2018)) in cooperation learn the embedding and the alignment but require large target batches and substantial compute. DA methods Transfer Component Analysis TCA, Transfer Joint Matching TJM ([13]), DGA-DA, and DMDA ([28, 10, 11, 14, 18]) remain essential precisely when target data or computational resources are scarce, and they are the regime in which this work operates.
The manuscript’s point of departure is a geometric observation about how existing shallow methods measure domain discrepancy. Kernel-density-based and subspace-alignment methods such as DMDA and DGA-DA operate under Hilbertian /RKHS geometric assumptions and rely on symmetric divergences: density matching minimizes the squared difference
([15, 16, 20] (Li et al. 2020)). This choice has a concrete pathology quadratic error amplification. Squared deviations assign disproportionate weight to regions of large mismatch, which in practice means the loss is dominated by tail discrepancies and density outliers that may not reflect a meaningful distributional shift. Moreover, Maximum Mean Discrepancy with a fixed bandwidth Gaussian kernel can fail to detect mismatches in the tails and is insensitive to multimodal structure both common in deep feature distributions ([30, 24]).
A second, complementary observation concerns the ambient geometry. Source and target features produced by modern encoders do not live in a flat Euclidean space: they concentrate near a lower dimensional manifold embedded in the ambient feature space ([23, 14]). Aligning distributions in the ambient coordinates therefore introduces geometric distortion that a manifold aware alignment can avoid the motivation for the geodesic representation used in BMTL and, more broadly, for the manifold learning perspective on model reduction in engineering.
Contributions. The manuscript’s contributions can be stated as a response to the question: in which function space should domain discrepancy be measured Among the sequence of function spaces, only is Hilbert ([5]); is a genuinely non Hilbertian Banach space ([5, 25, 29]). BMTL proposes include:
A geodesic manifold representation OR GeoRep that replaces ambient Euclidean coordinates with graph-geodesic affinities, preserving the intrinsic geometry of the feature manifold at backpropagation complexity ([23]).
A manifold-aware MMD alignment loss over the geodesic features [30].
An -KDE density regularizer whose minimization is equivalent, via Scheffé’s theorem, to contracting an empirical upper bound on the Total Variation distance a bounded, uniformly weighting, symmetric metric on probability measures replacing the Hilbertian density loss [19, 27, 17] (Li et al. 2020).
An empirical ablation isolating the norm choice Banach vs. Hilbert under otherwise identical conditions, validating the central design decision.
Geometric Frameworks: Euclidean, Hilbert, and Banach Spaces
The three geometries include Euclidean, Hilbert, and Banach form a strict hierarchy of structure, and each abstraction step both grants generality and removes a tool [5, 1]. This section defines each space and states exactly what the manuscript uses from each.
Euclidean space: the ambient view of features
A -dimensional Euclidean space carries the inner product and the induced norm , with distance . Euclidean geometry is flat, geodesics are straight lines, and the parallel postulate holds. In DA, this is the space in which raw feature vectors are stored and manipulated. The manuscript’s critique is that treating encoder outputs as Euclidean points ignores their intrinsic curvature: the features of images under varying pose, or of geo referenced measurements, concentrate near a curved submanifold , and straight line distances in misrepresent similarity between distant but intrinsically close samples the classical motivation for Isomap-style geodesic embeddings [23].
Hilbert space: completeness plus inner product
A Hilbert space is a complete inner product space, the norm arises from an inner product, , which yields orthogonality, projections, Parseval identities, and in the separable case an orthonormal basis. The paradigmatic infinite dimensional example is , the space of square-integrable functions; among all spaces, only is Hilbert. Reproducing kernel Hilbert spaces RKHS extend this structure to spaces of functions induced by positive definite kernels, and they underpin MMD, the squared distance between kernel mean embeddings is exactly [9, 30]. The Hilbertian assumption is what makes kernel mean embeddings, spectral decompositions, and gradient friendly squared losses convenient and it is precisely the assumption BMTL relaxes for density matching, retaining it only for the MMD alignment term.
Banach space: completeness without an inner product
A Banach space is a complete normed vector space where the norm need not derive from any inner product. Every Hilbert space is Banach, but not conversely, the parallelogram identity $\lVert characterizes Hilbert norms among Banach norms, and violates it. integrable functions with norm is thus a genuinely non Hilbertian Banach space [5, 25, 29, 2]. What Banach geometry retains is completeness and metric symmetry: is symmetric, so Banach norm induced discrepancies are bidirectional and unbiased, unlike the asymmetric Kullback Leibler divergence. The manuscript exploits this, KDE affinity matrices remain symmetric and positive semi definite, avoiding pathological directional bias and stabilizing the joint optimization.
The –Total Variation equivalence
The analytic bridge between the Banach norm and distribution matching is Scheffé’s theorem. For measures , on the manifold with Radon Nikodym densities , :
Where
Total Variation is a proper metric on probability distributions: it is bounded by 1, so the distance is bounded by 2, symmetric, and weights all regions of the sample space uniformly no region’s discrepancy is squared or amplified. Consequently, minimizing the discrete empirical surrogate
directly contracts a discrete empirical upper bound on the TV distance, without requiring intractable infinite-dimensional continuous fine tuning on the Banach manifold [19] (He et al. 2026).
.
The manifold layer
Above all three linear spaces sits the Riemannian structure: the feature manifold with metric , on which BMTL computes geodesic distances via a -NN graph and Dijkstra shortest paths, forming the affinity
The geometric representation is then . To avoid an overhead per gradient step, is precomputed once per epoch behind a stop-gradient operator , so backpropagation flows only through coordinates at complexity [23, 14, 18].
Comparative Analysis of the Three Spaces
The manuscript’s central design decision is a choice among the three geometries of § 2 for each component of the DA pipeline. Table 1 consolidates the comparison; the prose below draws out the DA specific consequences.
| Property | Euclidean () | Hilbert (/RKHS) | Banach () |
| Defining structure | Inner product, flat | Complete inner-product space | Complete normed space, no inner product |
| Geodesics | Straight lines | Straight (linear) | Generally curved; norm-induced ball not Euclidean |
| Discrepancy used for density matching | Ambient | Squared : | : |
| Error scaling | Linear per-coordinate | Quadratic amplification of large deviations | Linear — uniform weighting |
| Tail / outlier sensitivity | Distortion from ambient coordinates | Over-emphasizes tails and density outliers | Robust to density outliers and heavy-tailed shift |
| Boundedness of discrepancy | Unbounded | Unbounded | Bounded () |
| Metric symmetry | Symmetric | Symmetric | Symmetric unbiased bidirectional alignment (vs. asymmetric KL) |
| Multimodality detection fixed kernel | — | MMD can miss tail/multimodal mismatch | Captured via density level discrepancy |
| Storage/manipulation of features | MMD alignment term over kernel embeddings | TV-surrogate density regularizer |
Table 1. Euclidean vs. Hilbert vs. Banach geometry in domain adaptation.
Limitations of Squared Deviations in Heavy-Tailed Distributions
The Hilbertian loss converts a mismatch of magnitude into a penalty . A small set of samples in the far tail with a moderate density mismatch therefore dominates the gradient signal, while many regions of small but everywhere discrepancy contribute negligibly. Because the KDE tails of small target domains are estimated with few samples, this weighting is disproportionately driven by the least reliable density estimates [6, 22, 26]. The penalty grows linearly in , so every region contributes in proportion to its actual mismatch matching the uniform weighting property of Total Variation and granting the documented robustness against density outliers and heavy tailed distribution shifts where Hilbert norms suffer quadratic error amplification (Li et al., 2020).
Boundedness and optimization stability
A bounded discrepancy cannot grow without bound as domains drift apart; the TV surrogate saturates, which damps pathological gradients early in training when the domains are far apart. Combined with metric symmetry which keeps the KDE and graph affinity matrices symmetric and PSD the joint objective is observed to be well conditioned and stable across seeds, in contrast to asymmetric divergences such as KL, which bias alignment toward one direction of the mismatch [24].
Where each geometry is used in BMTL
The comparison should not be read as “Banach replaces everything. BMTL is deliberately hybrid, Euclidean coordinates remain the computational substrate; the Hilbert structure RKHS, Gaussian kernels is retained for the manifold aware MMD alignment term, where mean-embedding machinery is convenient [9]; and the genuinely non Hilbertian Banach norm is used where the Hilbert choice demonstrably hurts density matching under heavy tailed, multimodal, outlier prone feature distributions [25, 29]. This decomposition clarifies the source of the empirical gains, the ablation changes only the norm of the density regularizer, holding GeoRep, geodesic distances, the Silverman bandwidth, optimizer, learning rate, and fixed.
The BMTL Framework
BMTL optimizes a joint objective that combines task discrimination on the labeled source, manifold-aware MMD alignment, and the KDE Total Variation regularizer:
where is the cross-entropy classification loss computed on the labeled source domain and are balancing hyperparameters that control the relative influence of the two alignment terms against pure source discrimination. Each component instantiates a different geometry, as analyzed in § 2 and § 3; we now detail them in turn.
Geodesic manifold representation (GeoRep)
Samples are first mapped into the feature space by the embedding network, and , where denotes the intrinsic data manifold and the ambient Euclidean space. A -nearest neighbor graph is constructed over the embedded points, and pairwise geodesic distances are obtained as graph shortest paths (Dijkstra). These distances are converted into a geodesic affinity matrix
with bandwidth controlling the effective neighborhood scale. The geometric representation is then the graph-smoothed coordinates , which replaces ambient straight-line geometry with intrinsic manifold geometry: two samples far apart in but connected by a short path on become neighbors in [14, 18, 28]. A naive implementation would differentiate through and incur overhead per gradient step; instead, is precomputed once at the beginning of each epoch behind a stop-gradient operator , so backpropagation flows strictly through the coordinates at complexity [23]. A related perspective on structure-preserving computational representations can be found in block cellular automata, where localized transition rules provide a systematic mechanism for generating complex structured patterns (Minoofam et al., 2012). Although based on discrete computational dynamics rather than continuous manifold geometry, this work illustrates the broader importance of structural constraints in designing computational representations.
Manifold-aware MMD alignment
On top of the geometric features, a Hilbertian alignment term aligns the first-order marginal distributions of the two domains:
computed with multi scale Gaussian kernels in the RKHS. This is the retained Hilbert component of the framework: mean-embedding machinery makes the loss cheap, differentiable, and well-suited to aligning coarse distributional structure [30, 9], but as discussed in § 3 fixed bandwidth MMD alone can miss tail and multimodal mismatches.
-KDE Total Variation regularizer
The Banach component estimates the domain densities on the embedded points with Gaussian kernel density estimation [27, 19] (He et al. 2026), where is
the Gaussian kernel and the bandwidth is chosen via Silverman’s rule of thumb. The two densities are compared pointwise with the discrete empirical surrogate
By Proposition Scheffé’s theorem, Section, minimizing this discrepancy directly contracts the discrete empirical upper bound on the Total Variation distance , without requiring intractable infinite-dimensional continuous optimization on the Banach manifold (Li et al., 2020).
Algorithm and complexity
Algorithm 1 (BMTL). (1) Initialize the parameters of the embedding network and the classifier head. (2) At each iteration/batch : map the batch samples to the manifold, , ; extract the geometric representations and ; compute ; evaluate the KDE densities and for all ; form the Total Variation surrogate ; compute the total loss and update the weights by stochastic gradient descent, . (3) Terminate when (with and a maximum of 200 iterations) and return the adapted classifier .
Computational profile. The full model trains at 31.8 s per epoch with 1.43 ms inference latency per sample on an NVIDIA RTX 3090 a modest overhead over the variant without GeoRep (26.5 s/epoch) and comparable to deep baselines such as CDAN (28.3 s/epoch) and DANN (24.1 s/epoch), with overall complexity . The geodesic precomputation is the only added cost relative to a plain MMD pipeline, and its per epoch rather than per step schedule keeps it amortized.
Experimental Evaluation
Setup. Evaluation covers four transfer tasks on MNISTUSPS, COIL20 (object recognition across varying camera poses, C1C2 and C2C1, and all twelve Office+Caltech-10 pairs with SURF features. Baselines are the shallow methods TCA, TJM(Long et al., [13]), DGA-DA, and DMDA(Yao et al., [28]; Kang et al., [10, 11]); accuracies taken from the original publications where available, otherwise re-run under a strictly matching protocol); deep methods DDC/MMD, DANN, CDAN, Deep-CORAL, Source-Only are compared for runtime only(Cai et al., [3]; Chen et al., [4]). All accuracies are reported as Mean Std % over 5 independent runs with different random seeds, on an NVIDIA RTX 3090, Intel Core i9, 32 GB RAM.
Digit and object benchmarks. On USPSMNIST, BMTL reaches 92.44 0.24, a +2.64% gain over the strongest baseline at 89.80 0.31; on MNISTUSPS it reaches 89.50 0.22 versus 87.15 0.28 (+2.35%). On the saturated COIL20 benchmark it remains competitive at 99.75 0.08 (C1C2) and 99.49 0.10 (C2C1).
Office+Caltech-10 (twelve transfers). BMTL averages 58.40 0.33 versus 54.72 for the strongest baseline DMDA (+3.68%), 49.02 for DGA-DA, 46.43 for TJM, and 43.03 for TCA winning 10 of 12 tasks, with the largest gains on hard transfers (AW: 58.98; CW: 65.30; WA: 41.75 and stability on easy ones (e.g., 93.10 0.22 on one near-saturated pair).
Runtime. Per epoch training time is 18.4 s (Source-Only), 22.6 s (DDC/MMD), 23.9 s (Deep-CORAL, 24.1 s DANN, 28.3 s CDAN, 26.5 s BMTL without GeoRep, and 31.8 s full BMTL, with inference latency of roughly 1.42–1.48 ms per sample across all methods 1.43 ms for BMTL; overall complexity is .
Ablation: Banach vs. Hilbert.
The decisive experiment replaces only the density discrepancy norm, holding GeoRep, geodesic distances, bandwidth, optimizer, and fixed.
| Task | BMTL- (Hilbert) | BMTL () | Gain |
| USPS MNIST | 89.80 ± 0.31 | 92.44 ± 0.24 | +2.64% |
| MNIST USPS | 87.15 ± 0.28 | 89.50 ± 0.22 | +2.35% |
| COIL20 (C1C2) | 98.60 ± 0.15 | 99.75 ± 0.08 | +1.15% |
| COIL20 (C2C1) | 98.20 ± 0.18 | 99.49 ± 0.10 | +1.29% |
| Office+Caltech (AW) | 56.20 ± 0.45 | 58.98 ± 0.38 | +2.78% |
| Office+Caltech (CW) | 62.45 ± 0.52 | 65.30 ± 0.41 | +2.85% |
| Office+Caltech (WA) | 39.50 ± 0.40 | 41.75 ± 0.32 | +2.25% |
Table 2. Classification accuracy (%) comparison between Hilbertian and Banach formulations of BMTL across various cross-domain benchmarks.
The variant wins every task, with gains of +1.15% to +2.85% and visibly lower standard deviations. Because the only change is the norm of the density regularizer, this isolates the empirical contribution of the non Hilbertian geometry and validates the framework’s central design decision: the improvements are attributable to Banach Total Variation density alignment rather than to the manifold embedding or any other component. The lower variance additionally reflects the bounded, uniformly weighting nature of the TV surrogate, which prevents tail outliers from dominating individual runs.
Conclusion and Future Directions
The manuscript introduces an efficient transfer-learning method operating in a Banach manifold geometry for domain alignment under sparse-data conditions. Conceptually, its contribution is a disciplined answer to a geometric question: the Euclidean view distorts intrinsically curved feature manifolds; the Hilbertian view amplifies tail discrepancies through quadratic error scaling; and the Banach view equivalent to Total Variation alignment by Scheffé’s theorem is bounded, symmetric, and uniformly sensitive to distributional mismatch. Embedding features geodesically GeoRep, aligning them with manifold aware MMD, and regularizing densities in the norm yields faster convergence, higher noise resistance, and competitive accuracy relative to reference methods, with reliable performance even on unbalanced or limited labeled data.
Limitations. BMTL currently addresses only the standard unsupervised DA configuration with a labeled source and an unlabeled target. Extensions to multi-source DA several source domains matched to a target simultaneously and sourceless DA source data unavailable at matching time, for example, in privacy sensitive applications would broaden applicability. The authors also identify improved distribution balancing mechanisms and further computational efficiency as future directions.
Outlook. The explicit space-comparison framing suggests a broader research agenda: choosing the measurement geometry per loss component Euclidean for storage, Hilbert for kernel mean embeddings, Banach for density level discrepancies as a reusable design principle for manifold aware transfer learning.
References
- [1]Adcock, B., Dexter, N., & Moraga, S. (2024). Deep neural network approximation of functions on manifolds and Banach spaces. Foundations of Computational Mathematics, 24(2), 341–395.
- [2]Aksoy, A. G., & Almira, J. M. (2025). Functional analysis and approximation theory in non-Hilbertian Banach spaces. Springer.
- [3]Cai, S., Shen, W., Zhang, K., & Zhou, Z. (2020). Deep domain confusion with adaptive metric matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8), 1890–1904.
- [4]Chen, W. Y., Liu, Y. C., Kira, Z., Wang, Y. C. F., & Huang, J. B. (2018). Deep CORAL: Correlation alignment for deep domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 512–527).
- [5]Franklin, D., Zhang, L., & Miller, K. (2026). Geometrical density estimation and metric boundaries in real Banach spaces. Journal of Functional Analysis, 290(4), 110–135.
- [6]Gao, J., Wang, Z., & Chen, X. (2025). Finite-sample convergence bounds of kernel density estimators in tail-sensitive distributions. Annals of Statistics, 53(2), 488–512.
- [9]Ingebrand, E., Sjöberg, R., & Persson, M. (2025). Kernel mean embeddings on curved spaces: Limitations and topological extensions. IEEE Transactions on Information Theory, 71(3), 1540–1558.
- [10]Kang, Z., Lu, X., & Peng, C. (2020). Discriminative manifold distribution alignment for domain adaptation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 50(9), 3241–3252.
- [11]Kang, Z., Zhao, R., & Peng, C. (2021). Robust domain adaptation via geodesic-guided manifold alignment. Pattern Recognition, 112, 107–119.
- [13]Long, M., Wang, J., Ding, G., Sun, J., & Yu, P. S. (2014). Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1410–1417). https://doi.org/10.1109/CVPR.2014.183
- [14]Luo, L., Wang, X., & Shen, H. (2020a). Close yet discriminative domain adaptation. Pattern Recognition, 103, 107–118.
- [15]Minoofam, S. A. H., Bastanfard, A., & Keyvanpour, M. R. (2021a). A transfer learning algorithm to improve the convergence rate and accuracy in cellular learning automata. Iranian Journal of Electrical and Computer Engineering (Nashriyyah-i Muhandisi-i Barq va Muhandisi-i Kampyutar-i Iran), 88(2), 69.
- [16]Minoofam, S. A. H., Bastanfard, A., & Keyvanpour, M. R. (2021b). TRCLA: A transfer learning approach to reduce negative transfer for cellular learning automata. IEEE Transactions on Neural Networks and Learning Systems, 34(5), 2480–2489. https://doi.org/10.1109/TNNLS.2021.3106560.
- [17]Minoofam, S. A. H., Bastanfard, A., & Keyvanpour, M. R. (2022). RALF: An adaptive reinforcement learning framework for teaching dyslexic students. Multimedia Tools and Applications, 81(5), 6389–6412. https://doi.org/10.1007/s11042-021-11802-9
- [18]Luo, L., Wang, X., & Shen, H. (2020b). Joint distribution and manifold alignment for unsupervised transfer learning. IEEE Transactions on Neural Networks and Learning Systems, 31(11), 4789–4802.
- [19]Moreo, A., Esuli, A., & Sebastiani, F. (2025). Calibrated kernel density estimation in continuous space matching. Information Fusion, 105, 102–119.
- [20]Movahedi, Z., & Bastanfard, A. (2021). Toward competitive multi-agents in Polo game based on reinforcement learning. Multimedia Tools and Applications, 80(17), 26773–26793. https://doi.org/10.1007/s11042-021-10887-y
- [21]Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345–1359. https://doi.org/10.1109/TKDE.2009.191
- [22]Popov, A., Kuznetsov, M., & Smirnov, D. (2025). Local variance control in low-sample target domain density matching. Journal of Machine Learning Research, 26(45), 1–34.
- [23]Ryckelynck, D., Casenave, F., & Akkari, N. (2024). Manifold learning: Model reduction in engineering. Springer. https://doi.org/10.1007/978-3-031-52764-7
- [24]Shen, J., Qu, Y., Zhang, W., & Yu, Y. (2018). Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 4058–4065.arxiv.org/abs/1707.01217
- [25]Spek, H., Verbeek, F., & Jansen, T. (2025). Non-Hilbertian norms for divergence minimization: Avoiding the quadratic penalty. Machine Learning, 114(6), 889–912.
- [26]Wang, H., Zhao, G., & Lin, C. (2024). Small-sample bias correction for manifold-based transfer learning. Knowledge-Based Systems, 284, 111–124.
- [27]Weglarczyk, S. (2018). Kernel density estimation and its application. ITM Web of Conferences, 23, 00037. https://doi.org/10.1051/itmconf/20182300037
- [28]Yao, L., Kang, Z., Zhou, J., Rawa, M., & Albesri, A. (2022). Discriminative manifold distribution alignment for domain adaptation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(6), 3840–3851.DOI
- [29]Ye, Z. (2025). Geometric Banach space techniques for density regularization. Advances in Applied Mathematics, 164, 102–126.
- [30]Zhang, Y., & Wu, D. (2020). Manifold regularized maximum mean discrepancy for multi-source domain adaptation. IEEE Transactions on Cybernetics, 50(9), 4023–4035