INTRODUCTION
Empirical studies have demonstrated that reusing pretrained representations improves downstream performance in computer vision [9 , 40 ] and language tasks [16 , 8 ] . Methods such as L2-SP (Li et al., 2018), adapters [15 ] , and LoRA (Hu et al., 2022) adapt models through regularization toward pretrained weights, the addition of task-specific modules, and low-rank parameter updates, respectively. Besides the choice of a specific fine-tuning method, downstream performance also depends on the scale of the source and target model. More precisely, empirical scaling laws describe how transfer performance varies with the scaling of model size and the amounts of pretraining and fine-tuning data [14 , 41 ] . Notably, better pretraining performance on the source task does not necessarily improve results downstream: longer pretraining can improve source performance while worsening downstream performance [34 ] ; conversely, stronger pretraining weight decay can improve downstream performance even when it increases the pretraining validation loss [10 ] . Understanding these effects requires studying when pretraining improves downstream learning, especially in the scaling law, and how the source model should be trained to achieve these gains.
From a theoretical viewpoint, several works have studied regularization with respect to a source estimator, referred to as L2-SP in (Li et al., 2018). In this regard, Dar et al. [6 ] analyze minimum-norm pretraining with isotropic source covariates followed by ridge fine-tuning under model shift, identifying conditions under which transfer outperforms ridge regression only on target data. Kim et al. [21 ] study minimum-norm pretraining followed by the target interpolator closest to the source estimator, deriving non-asymptotic risk bounds under covariate and model shift in benign-overfitting regimes. In functional linear regression, Lin and Reimherr [28 ] establish minimax rates for transfer learning where both the pretraining and fine-tuning objective is L 2 L_2 L 2 -regularized under bounded model shift and covariance compatibility conditions, quantifying its benefits over learning only on target data. In nonparametric regression under model shift, Lin and Reimherr [29 ] develop a method for selecting regularization parameters in pretraining and fine-tuning that achieves minimax rates. However, these works do not explicitly characterize when transfer learning achieves a strictly better rate than learning from a single dataset (only the source, or only the target). Wu et al. (2022) establish sharp risk bounds under covariate shift when both pretraining and fine-tuning use one-pass stochastic gradient descent (SGD), and construct specific examples where this two-stage procedure improves sample complexity. This motivates us to derive a systematic characterization of how covariance spectra and sample sizes determine when transfer improves the decay of the risk. A related question concerns how the source model should be trained to achieve these gains. For ridge regression, Hedges [13 ] shows that, under proportional asymptotics with isotropic data, the optimal source regularization is independent of the target sample size but it still depends on task alignment. This leaves open whether a single source estimator, trained without target information, can remain optimal across a family of target distributions in regimes where transfer improves the scaling exponent over either dataset alone.
In this paper, we study these questions through a tractable model of high-dimensional linear regression. The source and target tasks may differ in their regression coefficients, covariate distributions, and noise levels, with sample sizes growing at different polynomial rates. We obtain the source estimator from ridge regression and fine-tune it on target data using a squared ℓ 2 \ell_2 ℓ 2 penalty centered at the source estimator. We summarize our main contributions as follows:
Scaling laws for transfer learning. We derive a deterministic equivalent for the target excess risk of ridge regression, together with a non-asymptotic approximation error (Theorem 1). Specializing to power-law covariance spectra and a common regressor, we obtain scaling laws for transfer learning (Theorem 3). Comparing them to the joint minimax rate which uses both datasets (Theorem 2), we show that the optimally tuned deterministic equivalent achieves the joint minimax rate whenever the latter is strictly faster than the minimax rates of either dataset alone (Corollary 1).
Universality of optimal source training. Our analysis unveils that the optimal source regularization depends only on source information, allowing a single source estimator to achieve these gains across various target distributions without target-dependent tuning of the source. Furthermore, the same source estimator remains optimal when fine-tuning is performed via one-pass SGD (Lemma 1), indicating universality across fine-tuning methods.
Related Work
Empirical works. Early empirical studies examined how pretrained representations are adapted and how transfer benefits vary across downstream tasks [40 , 22 ] . Common approaches include using frozen features, fine-tuning all pretrained parameters, and adapting a small number of trainable parameters. L2-SP regularizes the target model toward pretrained weights (Li et al., 2018), while adapters and LoRA introduce task-specific modules and low-rank weight updates [15 ] (Hu et al. 2022). The adaptation procedure also affects out-of-distribution performance: Kumar et al. [24 ] show that full fine-tuning can distort pretrained features, and that training a linear head on frozen features before fine-tuning can improve robustness to distribution shift. Empirical scaling laws further quantify the benefits of pretraining in terms of an equivalent amount of additional target data [14 ] , and describe how downstream performance depends on model size, pretraining and fine-tuning data sizes, and the adaptation method [41 ] . Isik et al. [18 ] show that source-target distribution alignment and fine-tuning data size affect downstream scaling: increasing pretraining data can reduce downstream cross-entropy without consistently improving translation quality. These findings motivate theoretical analyses of how task alignments, datasets size, and fine-tuning procedures affect the benefit of transfer learning.
Theoretical works. Beyond the two-stage regression analyses discussed in the introduction, theoretical studies have considered other forms of transfer learning. One related line of work is continual linear regression, which can be considered as a generalization of transfer learning to multiple sequential tasks. Zhao et al. [42 ] identify matrix-valued regularizers that match the estimation rates of learning jointly from all tasks, under shared regression coefficients and commuting covariances. Karpel et al. [20 ] characterize the generalization error and the optimal regularization strength for isotropic data in the proportional asymptotics regime. For classification, classical domain-adaptation bounds relate target error to source error, distribution discrepancy, and the joint error of the best shared classifier [1 ] . Hanneke and Kpotufe [11 ] characterize minimax rates, quantifying how transfer difficulty and label noise determine the relative value of source and target samples. [35 , 5 ] study models in which tasks share an unknown feature representation and show that learning this representation from sufficiently diverse training tasks can reduce the sample complexity of learning a new task. In high-dimensional regression, Li et al. (2022) establish minimax rates under sparse or small source–target coefficient differences and develop methods that adapt to unknown informative sources. He et al. (2024) develop transfer estimators that improve estimation under heterogeneous covariances when source–target coefficient differences are sufficiently small. Complementing these results, we study how covariance spectra and relative sample sizes determine the scaling benefits of transfer, identifying regimes in which ridge pretraining followed by ridge or SGD fine-tuning achieves minimax scaling and outperforms both target-only and source-only learning.
Problem SETUP
Given n ∈ N n \in\mathbb{N} n ∈ N , we use the shorthand [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } . We consider two independent training datasets, corresponding to a source task (sr) and a target task (tr). For the source and target tasks, respectively, we observe n s r n_{\mathrm{sr}} n sr and n t r n_{\mathrm{tr}} n tr independent samples ( x s r ( j ) , y s r ( j ) ) j ∈ [ n s r ] (x_{\mathrm{sr}^{(j)}}, y_{\mathrm{sr}}^{(j)})_{j\in[n_{\mathrm{sr}}]} ( x sr ( j ) , y sr ( j ) ) j ∈ [ n sr ] and ( x t r ( j ) , y t r ( j ) ) j ∈ [ n t r ] (x_{\mathrm{tr}}^{(j)}, y_{\mathrm{tr}}^{(j)})_{j\in[n_{\mathrm{tr}}]} ( x tr ( j ) , y tr ( j ) ) j ∈ [ n tr ] , where x s r ( j ) x_{\mathrm{sr}}^{(j)} x sr ( j ) and x t r ( j ) ∈ R d x_{\mathrm{tr}}^{(j)} \in\mathbb{R}^{d} x tr ( j ) ∈ R d are drawn from distributions P s r P_{sr} P sr and P t r P_{tr} P t r such that E P s r [ x ] = E P t r [ x ] = 0 \mathbb{E}_{P_{sr}}[x] = \mathbb{E}_{P_{tr}}[x] = 0 E P sr [ x ] = E P t r [ x ] = 0 , E P s r [ x x T ] = Σ s r \mathbb{E}_{P_{sr}}[xx^{\mathsf{T}}] = \Sigma_{sr} E P sr [ x x T ] = Σ sr , and E P t r [ x x T ] = Σ t r \mathbb{E}_{P_{tr}}[xx^{\mathsf{T}}] = \Sigma_{tr} E P t r [ x x T ] = Σ t r . We assume the following concentration property on the designs.
Assumption 1. There exist constants C , c > 0 C,c > 0 C , c > 0 such that, for every z ∈ { s r , t r } z \in\{sr,tr\} z ∈ { sr , t r } , every PSD matrix A ∈ R d × d A \in\mathbb{R}^{d \times d} A ∈ R d × d , and every t > 0 t > 0 t > 0 , a sample x ∼ P z x \sim P_z x ∼ P z satisfies
P ( ∣ x T A x − Tr ( Σ z A ) ∣ > t ∥ Σ z 1 / 2 A Σ z 1 / 2 ∥ F ) ≤ C e − c t . (1) \mathbb{P}\left(\left|x^{\mathsf{T}}Ax-\operatorname{Tr}(\Sigma_z A)\right|>t\left\|\Sigma_z^{1/2}A\Sigma_z^{1/2}\right\|_{F}\right)\le Ce^{-ct}. \tag*{(1)} P ( x T A x − Tr ( Σ z A ) > t Σ z 1/2 A Σ z 1/2 F ) ≤ C e − c t . (1)
Assumption 1 is standard in the literature ([31 ] (Wu et al. 2026a)) and covers e.g. independent sub-Gaussian entries. We denote X z = [ x z ( 1 ) , … , x z ( n z ) ] T ∈ R n z × d X_z = [x_z^{(1)}, \ldots, x_z^{(n_z)}]^{\mathsf{T}} \in\mathbb{R}^{n_z \times d} X z = [ x z ( 1 ) , … , x z ( n z ) ] T ∈ R n z × d the data matrix and y z = ( y z ( j ) ) j ∈ [ n z ] ∈ R n z y_z = (y_z^{(j)})_{j \in[n_z]} \in\mathbb{R}^{n_z} y z = ( y z ( j ) ) j ∈ [ n z ] ∈ R n z the label for source (z = s r z = sr z = sr ) and target (z = t r z = tr z = t r ). The labels are generated by
y z = X z β z + ε z , ε z ∼ N ( 0 , σ z 2 I n z ) . (2) y_z = X_z\beta_z+\varepsilon_z,\qquad\varepsilon_z\sim\mathcal{N}(0,\sigma_z^2 I_{n_z}). \tag*{(2)} y z = X z β z + ε z , ε z ∼ N ( 0 , σ z 2 I n z ) . (2)
The noise vectors are independent of the data and of each other. In particular, the source and target tasks may differ in their parameters, covariances, and noise levels. For an estimator β ^ \hat{\beta} β ^ , we consider the excess prediction risk on the target task,
R ( β ^ ) : = E x , ε [ ( x T β ^ − x T β t r ) 2 ] = E ε [ ( β ^ − β t r ) T Σ t r ( β ^ − β t r ) ] , (3) \mathcal{R}(\hat{\beta}) := \mathbb{E}_{x,\varepsilon}\left[\left(x^{\mathsf{T}}\hat{\beta}-x^{\mathsf{T}}\beta_{tr}\right)^2\right]
= \mathbb{E}_{\varepsilon}\left[(\hat{\beta}-\beta_{tr})^{\mathsf{T}}\Sigma_{tr}(\hat{\beta}-\beta_{tr})\right], \tag*{(3)} R ( β ^ ) := E x , ε [ ( x T β ^ − x T β t r ) 2 ] = E ε [ ( β ^ − β t r ) T Σ t r ( β ^ − β t r ) ] , (3)
where the expectation also averages over label noise.
In this paper, we study transfer learning via the following two-step procedure. We first obtain a source estimator via ridge regression on the source dataset,
β ^ s r = arg min β { ∥ X s r β − y s r ∥ 2 2 + λ s r ∥ β ∥ 2 2 } , (4) \hat{\beta}_{sr}=\underset{\beta}{\arg\min}\left\{\left\|X_{sr}\beta-y_{sr}\right\|_2^2+\lambda_{sr}\|\beta\|_2^2\right\}, \tag*{(4)} β ^ sr = β arg min { ∥ X sr β − y sr ∥ 2 2 + λ sr ∥ β ∥ 2 2 } , (4)
and then obtain the target model by fitting a regression model on the target dataset with ridge penalty centered at the source estimator,
β ^ t r = arg min β { ∥ X t r β − y t r ∥ 2 2 + λ t r ∥ β − β ^ s r ∥ 2 2 } . (5) \hat{\beta}_{tr}=\underset{\beta}{\arg\min}\left\{\left\|X_{tr}\beta-y_{tr}\right\|_2^2+\lambda_{tr}\|\beta-\hat{\beta}_{sr}\|_2^2\right\}. \tag*{(5)} β ^ t r = β arg min { ∥ X t r β − y t r ∥ 2 2 + λ t r ∥ β − β ^ sr ∥ 2 2 } . (5)
The above two-step procedure has been used by Kim et al. [21 ] to study properties of transfer learning. The two estimators admit the closed forms
β ^ s r = ( X s r T X s r + λ s r I d ) − 1 X s r T y s r , \hat{\beta}_{sr}=(X_{sr}^{\mathsf{T}}X_{sr}+\lambda_{sr}I_d)^{-1}X_{sr}^{\mathsf{T}}y_{sr}, β ^ sr = ( X sr T X sr + λ sr I d ) − 1 X sr T y sr ,
β ^ t r = ( X t r T X t r + λ t r I d ) − 1 ( X t r T y t r + λ t r β ^ s r ) . (6) \hat{\beta}_{tr}=(X_{tr}^{\mathsf{T}}X_{tr}+\lambda_{tr}I_d)^{-1}\left(X_{tr}^{\mathsf{T}}y_{tr}+\lambda_{tr}\hat{\beta}_{sr}\right). \tag*{(6)} β ^ t r = ( X t r T X t r + λ t r I d ) − 1 ( X t r T y t r + λ t r β ^ sr ) . (6)
Deterministic equivalent for target risk
We start by establishing a deterministic equivalent for the risk (3) of β ^ t r \hat{\beta}_{tr} β ^ t r . First, let the effective regularization parameters τ s r , τ t r > 0 \tau_{sr},\tau_{tr}>0 τ sr , τ t r > 0 be the unique solution of
n z − λ z τ z = Tr [ Σ z ( Σ z + τ z I d ) − 1 ] , z ∈ { s r , t r } . (7) n_z-\frac{\lambda_z}{\tau_z}=\operatorname{Tr}\left[\Sigma_z(\Sigma_z+\tau_z I_d)^{-1}\right],\qquad z\in\{sr,tr\}. \tag*{(7)} n z − τ z λ z = Tr [ Σ z ( Σ z + τ z I d ) − 1 ] , z ∈ { sr , t r } . (7)
Furthermore, for z ∈ { s r , t r } z\in\{sr,tr\} z ∈ { sr , t r } , let us also define
W z : = Σ z ( Σ z + τ z I d ) − 2 , (8) W_z:=\Sigma_z(\Sigma_z+\tau_z I_d)^{-2}, \tag*{(8)} W z := Σ z ( Σ z + τ z I d ) − 2 , (8)
and the two scalar prefactors
c t r : = τ t r 2 n t r n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] , c_{tr}:=\frac{\tau_{tr}^{2}n_{tr}}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}, c t r := n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] τ t r 2 n t r ,
c s r : = 1 n s r − Tr [ Σ s r 2 ( Σ s r + τ s r I d ) − 2 ] . (9) c_{sr}:=\frac{1}{n_{sr}-\operatorname{Tr}\left[\Sigma_{sr}^{2}(\Sigma_{sr}+\tau_{sr}I_d)^{-2}\right]}. \tag*{(9)} c sr := n sr − Tr [ Σ sr 2 ( Σ sr + τ sr I d ) − 2 ] 1 . (9)
where ρ z \rho_z ρ z is defined in (34) . Define
ϵ s r : = ρ s r 6 log 5 / 2 ( n s r ) n s r , ϵ t r : = ρ t r 6 log 5 / 2 ( n t r ) n t r , \epsilon_{sr}:=\frac{\rho_{sr}^{6}\log^{5/2}(n_{sr})}{\sqrt{n_{sr}}},\qquad\epsilon_{tr}:=\frac{\rho_{tr}^{6}\log^{5/2}(n_{tr})}{\sqrt{n_{tr}}}, ϵ sr := n sr ρ sr 6 log 5/2 ( n sr ) , ϵ t r := n t r ρ t r 6 log 5/2 ( n t r ) , and
E s r : = c t r τ s r 2 ( β s r T ( Σ s r + τ s r I d ) − 1 β s r ) ⋅ ∥ W t r 1 / 2 ( Σ s r + τ s r I d ) − 1 W t r 1 / 2 ∥ o p . (10) \mathcal{E}_{sr}:=c_{tr}\tau_{sr}^{2}\left(\beta_{sr}^{\mathsf{T}}(\Sigma_{sr}+\tau_{sr}I_d)^{-1}\beta_{sr}\right)\cdot\left\|W_{tr}^{1/2}(\Sigma_{sr}+\tau_{sr}I_d)^{-1}W_{tr}^{1/2}\right\|_{\mathrm{op}}. \tag*{(10)} E sr := c t r τ sr 2 ( β sr T ( Σ sr + τ sr I d ) − 1 β sr ) ⋅ W t r 1/2 ( Σ sr + τ sr I d ) − 1 W t r 1/2 op . (10) Then for any D > 0 D>0 D > 0 , with probability at least 1 − ( n s r − D + n t r − D ) 1-(n_{sr}^{-D}+n_{tr}^{-D}) 1 − ( n sr − D + n t r − D ) ,
∣ R − R ‾ ∣ ≤ C D , K [ ( ϵ s r + ϵ t r ) R ‾ + ϵ s r E s r ] , (11) \left|\mathcal{R}-\overline{\mathcal{R}}\right|\leq C_{D,K}\left[(\epsilon_{sr}+\epsilon_{tr})\overline{\mathcal{R}}+\epsilon_{sr}\mathcal{E}_{sr}\right], \tag*{(11)} R − R ≤ C D , K [ ( ϵ sr + ϵ t r ) R + ϵ sr E sr ] , (11) where
R ‾ : = B ‾ 1 + B ‾ 2 ⏟ B ‾ + V ‾ s r + V ‾ t r , (12) \overline{\mathcal{R}}:=\underbrace{\overline{B}_1+\overline{B}_2}_{\overline{B}}+\overline{V}_{sr}+\overline{V}_{tr}, \tag*{(12)} R := B B 1 + B 2 + V sr + V t r , (12) B ‾ 1 : = c t r ∥ W t r 1 / 2 ( Σ s r ( Σ s r + τ s r I d ) − 1 β s r − β t r ) ∥ 2 2 , (13) \overline{B}_1:=c_{tr}\left\|W_{tr}^{1/2}\left(\Sigma_{sr}(\Sigma_{sr}+\tau_{sr}I_d)^{-1}\beta_{sr}-\beta_{tr}\right)\right\|_2^2, \tag*{(13)} B 1 := c t r W t r 1/2 ( Σ sr ( Σ sr + τ sr I d ) − 1 β sr − β t r ) 2 2 , (13) B ‾ 2 : = c t r c s r τ s r 2 Tr ( W t r W s r ) β s r T W s r β s r , (14) \overline{B}_2:=c_{tr}c_{sr}\tau_{sr}^{2}\operatorname{Tr}(W_{tr}W_{sr})\beta_{sr}^{\mathsf{T}}W_{sr}\beta_{sr}, \tag*{(14)} B 2 := c t r c sr τ sr 2 Tr ( W t r W sr ) β sr T W sr β sr , (14) V ‾ s r : = σ s r 2 c t r c s r Tr ( W t r W s r ) , (15) \overline{V}_{sr}:=\sigma_{sr}^{2}c_{tr}c_{sr}\operatorname{Tr}(W_{tr}W_{sr}), \tag*{(15)} V sr := σ sr 2 c t r c sr Tr ( W t r W sr ) , (15) V ‾ t r : = σ t r 2 Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] . (16) \overline{V}_{tr}:=\sigma_{tr}^{2}\frac{\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}. \tag*{(16)} V t r := σ t r 2 n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] . (16)
The proof of Theorem 1 is deferred to Appendix § 9 , and it uses the deterministic equivalents for various functionals proved in Wu et al., 2026a, Theorem 6. We note that, while Misiakiewicz and Saeed [31 ] establish multiplicative error bounds, the error bound in (11) is O ( max ( ϵ s r , ϵ t r ) ) O(\max(\epsilon_{sr},\epsilon_{tr})) O ( max ( ϵ sr , ϵ t r )) and it is additive, as in Wu et al., 2026a,b, due to asymmetric terms in the risk expression. We expect that a multiplicative error bound can be established in certain structured settings, such as the power-law model considered in § 5 .
SCALING LAWS for Transfer Learning
We start by describing in § 5.1 the power-law model under which scaling laws are derived. Then, in § 5.2 , we recall minimax results from Wu et al. (2026b) using both source and target data; in § 5.3 , we establish scaling laws for the ridge estimator in (5) obtained via transfer learning; finally, in § 5.4 we give scaling laws for one-pass SGD initialized from the source estimator in (4) .
Power-law model
Throughout, we consider d = ∞ d=\infty d = ∞ and assume that (i) Σ s r , Σ t r \Sigma_{sr},\Sigma_{tr} Σ sr , Σ t r commute, so that w.l.o.g. we can take them to be diagonal, (ii) the source–target tasks share the same regression weights β s r = β t r = β ∗ \beta_{sr}=\beta_{tr}=\beta_* β sr = β t r = β ∗ , and (iii) the noise variances σ ϵ s r 2 , σ ϵ t r 2 = Θ ( 1 ) \sigma_{\epsilon_{sr}}^2,\sigma_{\epsilon_{tr}}^2=\Theta(1) σ ϵ sr 2 , σ ϵ t r 2 = Θ ( 1 ) .
More precisely, for (i), we consider power-law spectra
( Σ s r ) k , k ≍ k − a , ( Σ t r ) k , k ≍ k − b , a , b > 1. (17) (\Sigma_{sr})_{k,k}\asymp k^{-a},\qquad(\Sigma_{tr})_{k,k}\asymp k^{-b},\qquad a,b>1. \tag*{(17)} ( Σ sr ) k , k ≍ k − a , ( Σ t r ) k , k ≍ k − b , a , b > 1. (17)
For (ii), we study two types of signals β ∗ \beta_* β ∗ . For the minimax model in § 5.2 , we assume that the signal β ∗ \beta_* β ∗ belongs to the augmented parameter class Θ ( R ) \Theta(R) Θ ( R ) , defined for fixed r s r , r t r > 0 r_{sr},r_{tr}>0 r sr , r t r > 0 and a radius R > 0 R>0 R > 0 independent of n t r , n s r n_{tr},n_{sr} n t r , n sr by
Θ ( R ) : = { β : ∥ Σ s r 1 / 2 − r s r β ∥ 2 ≤ R , ∥ Σ t r 1 / 2 − r t r β ∥ 2 ≤ R } . (18) \Theta(R):=\left\{\beta:\left\|\Sigma_{sr}^{1/2-r_{sr}}\beta\right\|_2\leq R,\ \left\|\Sigma_{tr}^{1/2-r_{tr}}\beta\right\|_2\leq R\right\}. \tag*{(18)} Θ ( R ) := { β : Σ sr 1/2 − r sr β 2 ≤ R , Σ t r 1/2 − r t r β 2 ≤ R } . (18)
The conditions in (18) correspond to a source condition (Caponnetto and De Vito, [3 ] ). Defining
δ : = b − a , s : = ( b r t r ) ∨ ( a r s r + δ 2 ) , γ c : = 1 − δ 1 + 2 s . (19) \delta:=b-a,\qquad s:=(br_{tr})\vee\left(ar_{sr}+\frac{\delta}{2}\right),\qquad\gamma_c:=1-\frac{\delta}{1+2s}. \tag*{(19)} δ := b − a , s := ( b r t r ) ∨ ( a r sr + 2 δ ) , γ c := 1 − 1 + 2 s δ . (19)
(18) is, up to multiplicative constants in the radius, equivalent to the ellipsoid { β : ∑ k k 2 s − b ( β ) k 2 ≤ R 2 } \{\beta: \sum_k k^{2s-b}(\beta)_k^2 \le R^2\} { β : ∑ k k 2 s − b ( β ) k 2 ≤ R 2 } , see Wu et al. (2026b, Appendix C).
When studying the scaling law of the ridge estimator (§ 5.3 ) and one-pass SGD (§ 5.4 ), we consider β ∗ \beta_* β ∗ satisfying
( Σ s r 1 / 2 β ∗ ) k 2 ≍ k − 1 − 2 a r s r , ( Σ t r 1 / 2 β ∗ ) k 2 ≍ k − 1 − 2 b r t r , (20) \left(\Sigma_{\mathrm{sr}}^{1/2}\beta_*\right)_k^2 \asymp k^{-1-2ar_{\mathrm{sr}}}, \qquad\left(\Sigma_{\mathrm{tr}}^{1/2}\beta_*\right)_k^2 \asymp k^{-1-2br_{\mathrm{tr}}}, \tag*{(20)} ( Σ sr 1/2 β ∗ ) k 2 ≍ k − 1 − 2 a r sr , ( Σ tr 1/2 β ∗ ) k 2 ≍ k − 1 − 2 b r tr , (20)
in the regime where b r t r = a r s r + δ / 2 br_{\mathrm{tr}}=ar_{\mathrm{sr}}+\delta/2 b r tr = a r sr + δ /2 . This also implies that s = b r t r = a r s r + δ / 2 s=br_{\mathrm{tr}}=ar_{\mathrm{sr}}+\delta/2 s = b r tr = a r sr + δ /2 by definition. Equivalently, we can write ( β ∗ ) k 2 ≍ k − 1 + b − 2 s (\beta_*)_k^2 \asymp k^{-1+b-2s} ( β ∗ ) k 2 ≍ k − 1 + b − 2 s . We further restrict to the regime where 2 s > b 2s>b 2 s > b so that β ∗ \beta_* β ∗ is square-integrable.
Note that β ∗ \beta_* β ∗ lies outside of Θ ( R ) \Theta(R) Θ ( R ) . Nevertheless, the scaling law on β ∗ \beta_* β ∗ reflects the worst case in the ellipsoid. To see this, for any β ∈ Θ ( R ) \beta\in\Theta(R) β ∈ Θ ( R ) , letting z = Σ t r 1 / 2 β z=\Sigma_{\mathrm{tr}}^{1/2}\beta z = Σ tr 1/2 β and z ∗ , k = Σ t r 1 / 2 β ∗ z_{*,k}=\Sigma_{\mathrm{tr}}^{1/2}\beta_* z ∗ , k = Σ tr 1/2 β ∗ , we have
∑ k = M 2 M − 1 z ∗ , k 2 ≲ M − 2 s ≲ sup β ∈ Θ ( R ) ∑ k = M 2 M − 1 z k 2 . \sum_{k=M}^{2M-1}z_{*,k}^2\lesssim M^{-2s}\lesssim\sup_{\beta\in\Theta(R)}\sum_{k=M}^{2M-1}z_k^2. k = M ∑ 2 M − 1 z ∗ , k 2 ≲ M − 2 s ≲ β ∈ Θ ( R ) sup k = M ∑ 2 M − 1 z k 2 .
Thus, on each spectral band M ≤ k < 2 M M\le k<2M M ≤ k < 2 M , z ∗ z_* z ∗ has the same energy order as the largest allowed by the ellipsoid. The bias term in the test error sums the contribution of each band and is governed by the dominant band, up to logarithmic factors. Thus, the worst-case bias is obtained by picking a signal in the ellipsoid that concentrates its energy on the worst band, yielding the same bias exponent as β ∗ \beta_* β ∗ . As the variance does not depend on β ∗ \beta_* β ∗ , this gives the same polynomial scaling exponent for the test error.
We set n : = n t r n:=n_{\mathrm{tr}} n := n tr and take
n s r ≍ n γ n , λ s r ≍ n − γ λ s r , λ t r ≍ n − γ λ t r . (21) n_{\mathrm{sr}}\asymp n^{\gamma_n}, \qquad\lambda_{\mathrm{sr}}\asymp n^{-\gamma_{\lambda_{\mathrm{sr}}}}, \qquad\lambda_{\mathrm{tr}}\asymp n^{-\gamma_{\lambda_{\mathrm{tr}}}}. \tag*{(21)} n sr ≍ n γ n , λ sr ≍ n − γ λ sr , λ tr ≍ n − γ λ tr . (21)
We consider γ n > 0 \gamma_n>0 γ n > 0 and ridge penalties satisfying γ λ s r > − γ n \gamma_{\lambda_{\mathrm{sr}}}>-\gamma_n γ λ sr > − γ n and γ λ t r > − 1 \gamma_{\lambda_{\mathrm{tr}}}>-1 γ λ tr > − 1 , as well as the target-only ( λ s r = ∞ ) (\lambda_{\mathrm{sr}}=\infty) ( λ sr = ∞ ) and source-only ( λ t r = ∞ ) (\lambda_{\mathrm{tr}}=\infty) ( λ tr = ∞ ) cases. This condition on ridge penalties simply ensures that n z − 1 λ z = O ( 1 ) n_z^{-1}\lambda_z=O(1) n z − 1 λ z = O ( 1 ) for z ∈ { s r , t r } z\in\{\mathrm{sr},\mathrm{tr}\} z ∈ { sr , tr } , see also the remark on stable regularization in Wu et al. (2026a).
Minimax rates
For a given dataset or pair of datasets D \mathcal{D} D , define the minimax risk as
R ∗ ( n t r , n s r ) = inf β ^ = β ^ ( D ) sup β ∈ Θ ( R ) E β ∥ Σ t r 1 / 2 ( β ^ − β ) ∥ 2 2 , (22) R^*(n_{\mathrm{tr}},n_{\mathrm{sr}})=\inf_{\widehat{\beta}=\widehat{\beta}(\mathcal{D})}\sup_{\beta\in\Theta(R)}\mathbb{E}_{\beta}\left\|\Sigma_{\mathrm{tr}}^{1/2}(\widehat{\beta}-\beta)\right\|_2^2, \tag*{(22)} R ∗ ( n tr , n sr ) = β = β ( D ) inf β ∈ Θ ( R ) sup E β Σ tr 1/2 ( β − β ) 2 2 , (22)
where E β \mathbb{E}_{\beta} E β denotes the expectation w.r.t the data distribution generated by the signal β \beta β .
Theorem 2 (Minimax scaling laws, Theorem 3 in Wu et al. (2026b)). Under the linear model in (2) , the power-law spectra in (17) , and the parameter class $\Theta(R)\ \text{in}\ (18) , the minimax risks for individual datasets satisfy
R ∗ ( n , 0 ) ≍ n − Γ t r , R ∗ ( 0 , n s r ) ≍ { n − Γ s r , δ < 1 , n − γ n log n , δ = 1 , n − γ n , δ > 1 , (23) R^*(n,0)\asymp n^{-\Gamma_{\mathrm{tr}}}, \qquad R^*(0,n_{\mathrm{sr}})\asymp
\begin{cases}
n^{-\Gamma_{\mathrm{sr}}}, & \delta<1,\\
n^{-\gamma_n}\log n, & \delta=1,\\
n^{-\gamma_n}, & \delta>1,
\end{cases} \tag*{(23)} R ∗ ( n , 0 ) ≍ n − Γ tr , R ∗ ( 0 , n sr ) ≍ ⎩ ⎨ ⎧ n − Γ sr , n − γ n log n , n − γ n , δ < 1 , δ = 1 , δ > 1 , (23) and the minimax risk using both datasets satisfies
R ∗ ( n , n s r ) ≍ { R ∗ ( n , 0 ) , γ n ≤ γ c , R ∗ ( 0 , n s r ) , δ ≤ 1 , γ n > γ c , R ∗ ( 0 , n s r ) , δ > 1 , γ n ≥ 1 , n − ( δ − 1 + γ n ) / δ , δ > 1 , γ c < γ n < 1. (24) R^*(n,n_{\mathrm{sr}})\asymp
\begin{cases}
R^*(n,0), & \gamma_n\le\gamma_c,\\
R^*(0,n_{\mathrm{sr}}), & \delta\le1,\ \gamma_n>\gamma_c,\\
R^*(0,n_{\mathrm{sr}}), & \delta>1,\ \gamma_n\ge1,\\
n^{-(\delta-1+\gamma_n)/\delta}, & \delta>1,\ \gamma_c<\gamma_n<1.
\end{cases} \tag*{(24)} R ∗ ( n , n sr ) ≍ ⎩ ⎨ ⎧ R ∗ ( n , 0 ) , R ∗ ( 0 , n sr ) , R ∗ ( 0 , n sr ) , n − ( δ − 1 + γ n ) / δ , γ n ≤ γ c , δ ≤ 1 , γ n > γ c , δ > 1 , γ n ≥ 1 , δ > 1 , γ c < γ n < 1. (24) The implicit constants may depend on constants R , s , a , b R,s,a,b R , s , a , b and noise levels σ s r 2 , σ t r 2 \sigma_{\mathrm{sr}}^2,\sigma_{\mathrm{tr}}^2 σ sr 2 , σ tr 2 .
Theorem 2 shows that, when δ > 1 \delta>1 δ > 1 and γ c < γ n < 1 \gamma_c<\gamma_n<1 γ c < γ n < 1 , using both datasets yields a strictly faster minimax rate than using either dataset alone. Furthermore, Wu et al. (2026b) also show that this improved minimax rate can be achieved by a ridge estimator. However, both the minimax rate of Theorem 2 and the ridge estimator of Wu et al. (2026b) require access to both source and target data at the same time. In contrast, in the transfer learning model we are considering, the target estimator does not have access to the source data per se, but rather only to the pretrained estimator. We will next show that, despite this restriction, minimax rates remain achievable both by ridge regression and one-pass SGD.
Scaling laws for ridge estimator
We now study the estimator in (5) , and derive the scaling law for the deterministic equivalent of its risk. To express the resulting rates, we introduce the effective cutoff exponents
u : = ( γ λ s r + γ n ) ∧ ( a γ n ) a , v : = ( 1 + γ λ t r ) ∧ b b , (25) u := \frac{(\gamma_{\lambda_{\mathrm{sr}}}+\gamma_n)\wedge(a\gamma_n)}{a}, \qquad v := \frac{(1+\gamma_{\lambda_{\mathrm{tr}}})\wedge b}{b}, \tag*{(25)} u := a ( γ λ sr + γ n ) ∧ ( a γ n ) , v := b ( 1 + γ λ tr ) ∧ b , (25)
and
Γ t r : = 2 s 1 + 2 s , Γ s r : = 2 s γ n 1 + 2 s − δ . (26) \Gamma_{\mathrm{tr}} := \frac{2s}{1+2s}, \qquad\Gamma_{\mathrm{sr}} := \frac{2s\gamma_n}{1+2s-\delta}. \tag*{(26)} Γ tr := 1 + 2 s 2 s , Γ sr := 1 + 2 s − δ 2 s γ n . (26)
Theorem 3 (Scaling laws for ridge estimator). Under the assumptions of Theorem 1, for any fixed u , v u,v u , v , define the following quantities
Γ V t r = 1 − v , Γ B 1 = 2 b v + Γ s h , \Gamma_{V_{\mathrm{tr}}}=1-v,\qquad\Gamma_{B_1}=2bv+\Gamma_{\mathrm{sh}}, Γ V tr = 1 − v , Γ B 1 = 2 b v + Γ sh , Γ V s r = γ n + 2 b v + Γ T , Γ B 2 = Γ V s r + 2 a u + Γ S , (27) \Gamma_{V_{\mathrm{sr}}}=\gamma_n+2bv+\Gamma_T,\qquad\Gamma_{B_2}=\Gamma_{V_{\mathrm{sr}}}+2au+\Gamma_S, \tag*{(27)} Γ V sr = γ n + 2 b v + Γ T , Γ B 2 = Γ V sr + 2 a u + Γ S , (27) where Γ S = − ( a + b − 2 b r t r ) + u \Gamma_S=-(a+b-2br_{\mathrm{tr}})+u Γ S = − ( a + b − 2 b r tr ) + u ,
Γ T = { − ( a + b + 1 ) u − ( δ + 1 ) + ( v − u ) , u ≤ v , − ( a + b + 1 ) v − ( 1 − δ ) + ( u − v ) , v ≤ u , \Gamma_T =
\begin{cases}
-(a+b+1)u-(\delta+1)+(v-u), & u\le v,\\
-(a+b+1)v-(1-\delta)+(u-v), & v\le u,
\end{cases} Γ T = { − ( a + b + 1 ) u − ( δ + 1 ) + ( v − u ) , − ( a + b + 1 ) v − ( 1 − δ ) + ( u − v ) , u ≤ v , v ≤ u , and
Γ s h = { 2 a u − 2 b u ( 1 + a b − r t r ) − 2 b ( v − u ) ( 1 − r t r ) , u ≤ v , 2 a u − 2 b v ( 1 + a b − r t r ) − 2 b ( u − v ) ( a b − r t r ) , v ≤ u . \Gamma_{\mathrm{sh}} =
\begin{cases}
2au-2bu\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right)-2b(v-u)(1-r_{\mathrm{tr}}), & u\le v,\\
2au-2bv\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right)-2b(u-v)\left(\frac{a}{b}-r_{\mathrm{tr}}\right), & v\le u.
\end{cases} Γ sh = { 2 a u − 2 b u ( 1 + b a − r tr ) − 2 b ( v − u ) ( 1 − r tr ) , 2 a u − 2 b v ( 1 + b a − r tr ) − 2 b ( u − v ) ( b a − r tr ) , u ≤ v , v ≤ u . Then, the deterministic equivalents of the bias and variance terms satisfy
B ‾ 1 = Θ ~ ( n − Γ B 1 ) , B ‾ 2 = Θ ~ ( n − Γ B 2 ) , \overline{B}_1=\widetilde{\Theta}(n^{-\Gamma_{B_1}}),\qquad\overline{B}_2=\widetilde{\Theta}(n^{-\Gamma_{B_2}}), B 1 = Θ ( n − Γ B 1 ) , B 2 = Θ ( n − Γ B 2 ) , V ‾ s r = Θ ~ ( n − Γ V s r ) , V ‾ t r = Θ ~ ( n − Γ V t r ) , \overline{V}_{\mathrm{sr}}=\widetilde{\Theta}(n^{-\Gamma_{V_{\mathrm{sr}}}}),\qquad\overline{V}_{\mathrm{tr}}=\widetilde{\Theta}(n^{-\Gamma_{V_{\mathrm{tr}}}}), V sr = Θ ( n − Γ V sr ) , V tr = Θ ( n − Γ V tr ) , and therefore R ‾ = Θ ~ ( n − Γ T L ) \overline{R}=\widetilde{\Theta}(n^{-\Gamma_{\mathrm{TL}}}) R = Θ ( n − Γ TL ) , with
Γ T L = min { Γ V t r , Γ B 1 , Γ V s r , Γ B 2 } . (28) \Gamma_{\mathrm{TL}}=\min\{\Gamma_{V_{\mathrm{tr}}},\Gamma_{B_1},\Gamma_{V_{\mathrm{sr}}},\Gamma_{B_2}\}. \tag*{(28)} Γ TL = min { Γ V tr , Γ B 1 , Γ V sr , Γ B 2 } . (28) Theorem 3 establishes the scaling law of the deterministic equivalent R ‾ \overline{R} R , up to a poly-logarithmic factor (included in the notation Θ ~ ( ⋅ ) \widetilde{\Theta}(\cdot) Θ ( ⋅ ) ), and its proof is deferred to Appendix § 10 .
We next choose ridge parameters for which R ‾ \overline{R} R achieves the same exponent in the scaling law as the minimax risk using both datasets given by (24) .
Corollary 1 (Minimax-optimality of ridge deterministic equivalent). Under the assumptions of Theorem 1, in the regime δ > 1 \delta>1 δ > 1 , γ c < γ n < 1 \gamma_c<\gamma_n<1 γ c < γ n < 1 , by picking
λ s r ≍ n s r 1 − a , λ t r ≍ n 1 − b ( 1 − γ n ) / δ , (29) \lambda_{\mathrm{sr}}\asymp n_{\mathrm{sr}}^{1-a},\qquad\lambda_{\mathrm{tr}}\asymp n^{1-b(1-\gamma_n)/\delta}, \tag*{(29)} λ sr ≍ n sr 1 − a , λ tr ≍ n 1 − b ( 1 − γ n ) / δ , (29) or equivalently, u = γ n u=\gamma_n u = γ n and v = ( 1 − γ n ) / δ v=(1-\gamma_n)/\delta v = ( 1 − γ n ) / δ , we have that Γ T L = δ − 1 + γ n δ \Gamma_{\mathrm{TL}}=\frac{\delta-1+\gamma_n}{\delta} Γ TL = δ δ − 1 + γ n .
Corollary 1 shows that the deterministic equivalent of the risk for the ridge estimator obtained via transfer learning matches the minimax rate using jointly source and target data. We note that the regularization is chosen by optimizing the deterministic equivalent R ‾ \overline{R} R from Theorem 1. For this choice, we expect ∣ R − R ‾ ∣ = o ( R ‾ ) |R-\overline{R}|=o(\overline{R}) ∣ R − R ∣ = o ( R ) with high probability, which would imply the same scaling law for the actual test risk R R R . This expectation is motivated by the analogous approximation guarantee for mixed ridge regression in Wu et al. (2026b), Lemma 1 and is supported by our numerical results (Figure 1 ). This gives evidence that the two-step procedure composed of pretraining and fine-tuning is optimal and improves on using either dataset in isolation. We defer the full characterization of optimal rate and optimal regularization to Corollary 2 in Appendix § 11 , from which Corollary 1 directly follows.
Figure 1. Accuracy of theoretical predictions across various regimes for a = 2 a = 2 a = 2 , b = 6 b = 6 b = 6 , r t r = 0.6 r_{\mathrm{tr}} = 0.6 r tr = 0.6 , d = 50,000 d = 50{,}000 d = 50 , 000 . Solid lines show the deterministic equivalent, markers show Gaussian simulations (mean ± \pm ± s.e., 50 seeds), and dotted segments show the predicted slopes. Left: Bias is the bottleneck in the risk, and the optimal scaling law is reached by choosing the correct λ s r \lambda_{\mathrm{sr}} λ sr . Middle: Variance is the bottleneck in the risk, and the optimal λ t r \lambda_{\mathrm{tr}} λ tr balances V ‾ t r = V ‾ s r \overline{V}_{\mathrm{tr}} = \overline{V}_{\mathrm{sr}} V tr = V sr . Right: Comparison between optimally tuned target-only, source-only, and transfer learning risks.
We further note that the optimal source regularization depends only on source sample size and spectral decay, which makes the same tuning of the source estimator optimal across different targets. This supports pretraining a single source model and reusing it across downstream tasks, with only the target regularization chosen according to the specific fine-tuning data.
We finally remark that Corollary 1 shows the existence of a target-independent optimal source estimator rather than its uniqueness. In particular, the optimal ridge pair in (29) may not be unique, and it is possible that there exist other optimal pairs in which λ s r \lambda_{\mathrm{sr}} λ sr depends on the target.
Scaling laws for one-pass SGD
We now replace ridge fine-tuning with one-pass SGD. We first obtain the source estimator β ^ s r \widehat{\beta}_{\mathrm{sr}} β sr by ridge regression (4) , and initialize β ^ 0 = β ^ s r \widehat{\beta}_0 = \widehat{\beta}_{\mathrm{sr}} β 0 = β sr . Using each target sample once, SGD returns β ^ t r = β ^ n \widehat{\beta}_{\mathrm{tr}} = \widehat{\beta}_n β tr = β n , where
β ^ t = β ^ t − 1 − η t x t ( x t ⊤ β ^ t − 1 − y t ) , t = 1 , … , n . (30) \widehat{\beta}_t = \widehat{\beta}_{t-1} - \eta_t x_t(x_t^{\top}\widehat{\beta}_{t-1} - y_t), \qquad t = 1,\ldots,n. \tag*{(30)} β t = β t − 1 − η t x t ( x t ⊤ β t − 1 − y t ) , t = 1 , … , n . (30)
For the target covariates, we additionally impose the following fourth-moment condition.
Assumption 2. There exists a constant C ≥ 1 C \ge1 C ≥ 1 such that, for every bounded positive semidefinite matrix A A A ,
E [ ( x ⊤ A x ) x x ⊤ ] ≤ C Tr ( Σ t r A ) Σ t r , x ∼ P t r . (31) \mathbb{E}\left[(x^{\top}Ax)xx^{\top}\right] \le C\operatorname{Tr}(\Sigma_{\mathrm{tr}}A)\Sigma_{\mathrm{tr}}, \qquad x \sim P_{\mathrm{tr}}. \tag*{(31)} E [ ( x ⊤ A x ) x x ⊤ ] ≤ C Tr ( Σ tr A ) Σ tr , x ∼ P tr . (31)
Lemma 1 (Scaling laws for one-pass SGD). In the setting of Theorem 3 and under Assumption 2, consider the regime δ > 1 \delta> 1 δ > 1 , γ c < γ n < 1 \gamma_c < \gamma_n < 1 γ c < γ n < 1 . Let
λ s r = κ n s r 1 − a , η t = η 0 2 − ⌊ ( t − 1 ) / N ⌋ , (32) \lambda_{\mathrm{sr}} = \kappa n_{\mathrm{sr}}^{1-a}, \qquad\eta_t = \eta_0 2^{-\lfloor(t-1)/N \rfloor}, \tag*{(32)} λ sr = κ n sr 1 − a , η t = η 0 2 − ⌊( t − 1 ) / N ⌋ , (32) where κ > 0 \kappa> 0 κ > 0 is independent of the target task, and
N = ⌊ n log 2 n ⌋ , η 0 = 1 N ( N n s r ) b / δ . (33) N = \left\lfloor\frac{n}{\log_2 n} \right\rfloor, \qquad\eta_0 = \frac{1}{N}\left(\frac{N}{n_{\mathrm{sr}}}\right)^{b/\delta}. \tag*{(33)} N = ⌊ log 2 n n ⌋ , η 0 = N 1 ( n sr N ) b / δ . (33) Then, the last SGD iterate satisfies
E [ ∥ β ^ t r − β ∗ ∥ Σ t r 2 ] = Θ ~ ( n − Γ T L ) , Γ T L = δ − 1 + γ n δ . \mathbb{E}\left[\left\|\widehat{\beta}_{\mathrm{tr}}-\beta_*\right\|_{\Sigma_{\mathrm{tr}}}^2\right] = \widetilde{\Theta}\left(n^{-\Gamma_{\mathrm{TL}}}\right), \qquad\Gamma_{\mathrm{TL}} = \frac{\delta- 1 + \gamma_n}{\delta}. E [ β tr − β ∗ Σ tr 2 ] = Θ ( n − Γ TL ) , Γ TL = δ δ − 1 + γ n .
Lemma 1 shows that one-pass SGD matches the exponent of optimally regularized ridge, which is in turn minimax optimal. We note that both methods use the same rate-optimal source regularization, which does not depend on the target task. Thus, the universality of the pretrained source estimator across target tasks extends to both ridge and SGD fine-tuning.
The proof is given in Appendix D. For the upper bound, we condition on the source estimator and apply the SGD bound of Wu et al. (2022). We then evaluate the deterministic equivalent w.r.t the source estimator. At the selected ridge λ s r ≍ n s r 1 − a \lambda_{\mathrm{sr}} \asymp n_{\mathrm{sr}}^{1-a} λ sr ≍ n sr 1 − a , we show that the related deterministic equivalents have multiplicative approximation error. For the lower bound, we compare the second-moment recursion of the target-noise component with its population counterpart.
Remark 5.1 (Saturation effect of ridge and comparison with one-pass SGD). In linear regression under power-law spectra, ridge regression suffers from saturation, which is instead avoided by one-pass SGD (Wu et al., 2026c). In our transfer setting, ridge and SGD fine-tuning attain the same polynomial scaling exponent in the regime δ > 1 \delta> 1 δ > 1 and γ c < γ n < 1 \gamma_c < \gamma_n < 1 γ c < γ n < 1 since the test error is dominated by the variance terms, while the signal-dependent bias is of lower order. Consequently, ridge saturation does not affect the leading scaling exponent in this regime.
Numerical Results
We first showcase our theoretical predictions for synthetic data, and then examine whether a similar behavior occurs in the training of language models.
Experiments on synthetic data. We consider the power-law model of § 5 , with ( Σ s r ) k , k ≍ k − a (\Sigma_{\mathrm{sr}})_{k,k} \asymp k^{-a} ( Σ sr ) k , k ≍ k − a and ( Σ t r ) k , k ≍ k − b (\Sigma_{\mathrm{tr}})_{k,k} \asymp k^{-b} ( Σ tr ) k , k ≍ k − b . We use a = 2 a = 2 a = 2 , b = 6 b = 6 b = 6 , r t r = 0.6 r_{\mathrm{tr}} = 0.6 r tr = 0.6 , d = 50,000 d = 50{,}000 d = 50 , 000 , with n s r = n t r γ n n_{\mathrm{sr}} = n_{\mathrm{tr}}^{\gamma_n} n sr = n tr γ n . The ridge parameters follow the scaling in (21) .
Figure 1 shows bias-limited, variance-limited, and optimally tuned ridge regression, and how the regularization choice affects the scaling law of the target risk. The empirical results are closely tracked by deterministic-equivalent predictions, and they recover the corresponding theoretical decay rates. In the left panel, the bias is limiting the scaling law, and choosing the correct source regularization λ s r \lambda_{\mathrm{sr}} λ sr mitigates the issue. In the middle panel, the variance is limiting the scaling law, and choosing the correct target regularization λ t r \lambda_{\mathrm{tr}} λ tr mitigates the issue. In the right panel, all estimators (target-only, source-only, and ridge transfer) are optimally tuned, and transfer learning improves the scaling law as compared to using a single dataset.
While Figure 1 fixes γ n \gamma_n γ n and shows the effect of regularization on performance, Figure 2 varies γ n \gamma_n γ n and examines how the optimal exponent Γ \Gamma Γ of ridge regression changes as the size of the source dataset grows. The exponents are obtained from Theorem 3 and highlight the regime γ c < γ n < 1 \gamma_c < \gamma_n < 1 γ c < γ n < 1 in which combining source and target data improves the decay rate of the risk. Additionally, in Appendix § 13 .§ 1 , we show that the empirical risk is tracked by deterministic equivalents for PCA features from MNIST [25 ] and CIFAR-10 [23 ] .
Figure 2. Scaling law exponents of optimally regularized ridge and minimax rates , as a function of pretraining data size. Solid lines show the exponents from Theorem 3. The region γ c < γ n < 1 \gamma_c < \gamma_n < 1 γ c < γ n < 1 is where transfer learning attains the minimax rate obtained by employing both datasets, and it improves over estimators using either target-only or source-only data. The gray band is the minimax rate, i.e., the best achievable worst-case exponent any estimator can reach using both source and target data.
Experiments on language models. We train GPT-2 small (Radford et al., [33 ] ), using FineWeb-Edu (Penedo et al., [32 ] ) as the source distribution and fine-tune it using Python (CodeParrot, [4 ] ) as the target distribution. We design these experiments to see whether fine-tuning (FT) from the source trained on n s r ≍ n γ n n_{\mathrm{sr}} \asymp n^{\gamma_n} n sr ≍ n γ n tokens to a target with n n n tokens can make the target loss decrease faster in n n n than both target-only (TO) and source-only (SO) training. We also check whether this loss decrease depends on γ n \gamma_n γ n .
Figure 3 shows that the improvement in the transfer learning performance depends on both the size of source data and the optimization method used for target data. With a fixed source budget, FT achieves lower loss at small target budgets, but TO gradually closes the gap. When the source budget grows with the target budget, FT can decrease faster over the early scaling range under short target training; this rate advantage becomes smaller after additional target optimization. Thus, beyond the linear setting, we observe a similar dependence of transfer scaling on how the source budget grows with the target budget. Overall, we observe a higher FT rate under target training than under TO and SO separately, but this advantage depends on target optimization. Further details on the experimental setup are in Appendix § 13 .§ 2 .
Figure 3. Transfer learning in language models. Target test loss as a function of target sample size. Different columns correspond to different values of the source sample size (γ = 0 \gamma= 0 γ = 0 in the left-most plot and then γ = 0.75 \gamma= 0.75 γ = 0.75 , γ = 1 \gamma= 1 γ = 1 , and γ = 1.25 \gamma= 1.25 γ = 1.25 ). Different rows correspond to different number of passes on the data (single pass in the first row, and multiple passes with number of epochs chosen from validation in the second row). Access to more source data produces a significant advantage of FT under short target training (one-pass), while the advantage becomes weaker with additional target optimization (multi-pass). We denote the local exponent by Γ ^ = − ln L ( 2 n ) − ln L ( n ) ln 2 \widehat{\Gamma} = -\frac{\ln L(2n)-\ln L(n)}{\ln2} Γ = − l n 2 l n L ( 2 n ) − l n L ( n ) , see (129) for details. Green shading shows the ranges where Γ ^ F T > max ( Γ ^ T O , Γ ^ S O ) \widehat{\Gamma}_{FT} > \max(\widehat{\Gamma}_{TO},\widehat{\Gamma}_{SO}) Γ F T > max ( Γ T O , Γ S O ) , and purple hatching marks where Γ ^ F T S P > max ( Γ ^ T O , Γ ^ S O ) \widehat{\Gamma}_{FTSP} > \max(\widehat{\Gamma}_{TO},\widehat{\Gamma}_{SO}) Γ F T S P > max ( Γ T O , Γ S O ) . FTSP refers to our implementation of L2-SP ((Li et al., 2018)), see Appendix § 13 .§ 2 for details.
Conclusions
We study transfer learning in high-dimensional regression, where the source model is obtained by ridge regression and the target model is regularized toward the source estimator. Under rather general conditions, we establish a deterministic equivalent for the risk of the estimator implementing ridge regression with target data. Specializing to power-law covariance spectra and a shared regression function, we derive explicit scaling laws and compare them with the minimax rate of using both datasets. Notably, whenever combining the datasets yields a strictly better minimax rate than using either dataset alone, transfer learning via ridge regression attains this optimal exponent. Moreover, fine-tuning via one-pass SGD with a suitable geometric learning rate schedule attains the same exponent in the scaling law. Both ridge regression and SGD employ an optimal source estimator which does not depend on the target task and it is, therefore, universal across target distributions. Experiments on language models corroborate the generality of our findings beyond linear models.
Future directions. A natural direction is to extend our analysis beyond regression to classification, building on existing minimax theory ([2 ] ). In particular, it would be interesting to determine whether the improved scaling laws and the universality of the source estimator persist in this setting. Another direction is to incorporate model size explicitly through random feature models ([7 ] ). Allowing the number of random features to grow jointly with the source and target sample sizes would enable us to characterize how model capacity affects the benefits of transfer, and to identify the model size required to retain the improved scaling exponent.
AI use statement
We used generative AI tools to polish the writing, implement straightforward parts of the code, fill in proof details after we had identified the key methods and arguments, check proofs for correctness, and improve the coverage of related work. We did not use generative AI tools to develop the paper’s core ideas and define the problem setting. We take full responsibility for the final content of this work, including all text, claims, code, and other artifacts produced with the assistance of generative AI.
Proof of Theorem 1
We first introduce a few quantities that appear in the approximation error bounds. Following Wu et al. (2026a, Appendix B), given a covariance matrix Σ ∈ R d × d \Sigma\in\mathbb{R}^{d \times d} Σ ∈ R d × d with eigenvalues ξ 1 ≥ ξ 2 ≥ ξ 3 ≥ ⋯ \xi_1 \ge\xi_2 \ge\xi_3 \ge\cdots ξ 1 ≥ ξ 2 ≥ ξ 3 ≥ ⋯ , we define the intrinsic dimension r Σ ( k ) r_{\Sigma}(k) r Σ ( k ) at level k ∈ N k \in\mathbb{N} k ∈ N of Σ \Sigma Σ to be the intrinsic dimension of the covariance matrix Σ ≥ k = diag ( ξ k + 1 2 , ξ k + 1 2 , … ) \Sigma_{\geq k} = \operatorname{diag}(\xi_{k+1}^2,\xi_{k+1}^2,\ldots) Σ ≥ k = diag ( ξ k + 1 2 , ξ k + 1 2 , … ) , which is
r Σ ( k ) : = Tr ( Σ ≥ k ) ∥ Σ ≥ k ∥ op = ∑ j = k + 1 d ξ j 2 ξ k 2 . r_{\Sigma}(k) := \frac{\operatorname{Tr}(\Sigma_{\geq k})}{\lVert\Sigma_{\geq k}\rVert_{\operatorname{op}}} = \frac{\sum_{j=k+1}^{d}\xi_j^2}{\xi_k^2}. r Σ ( k ) := ∥ Σ ≥ k ∥ op Tr ( Σ ≥ k ) = ξ k 2 ∑ j = k + 1 d ξ j 2 .
Given any λ > 0 \lambda> 0 λ > 0 , define
ρ λ ( Σ , n ) : = 1 + n ⋅ ξ ⌊ η ∗ ⋅ n ⌋ 2 λ { 1 + r Σ ( ⌊ η ∗ ⋅ n ⌋ ) ∨ n n log ( r Σ ( ⌊ η ∗ ⋅ n ⌋ ) ∨ n ) } . (34) \rho_{\lambda}(\Sigma,n) := 1 + \frac{n \cdot\xi_{\lfloor\eta_* \cdot n\rfloor}^2}{\lambda}\left\{1 + \frac{r_{\Sigma}(\lfloor\eta_* \cdot n\rfloor)\vee n}{n}\log\left(r_{\Sigma}(\lfloor\eta_* \cdot n\rfloor)\vee n\right)\right\}. \tag*{(34)} ρ λ ( Σ , n ) := 1 + λ n ⋅ ξ ⌊ η ∗ ⋅ n ⌋ 2 { 1 + n r Σ (⌊ η ∗ ⋅ n ⌋) ∨ n log ( r Σ (⌊ η ∗ ⋅ n ⌋) ∨ n ) } . (34)
where η ∗ ∈ ( 0 , 1 / 2 ) \eta_* \in(0,1/2) η ∗ ∈ ( 0 , 1/2 ) is a constant that only depends on the constants in Assumption 1. For fixed λ \lambda λ and for z ∈ { s r , t r } z \in\{\mathrm{sr},\mathrm{tr}\} z ∈ { sr , tr } , we define ρ z ( λ ) : = ρ λ ( Σ z , n z ) \rho_z(\lambda) := \rho_{\lambda}(\Sigma_z,n_z) ρ z ( λ ) := ρ λ ( Σ z , n z ) . We drop the dependence on λ \lambda λ when there is no confusion.
Define the source and target resolvents by
H s r : = ( X s r T X s r + λ s r I d ) − 1 , H t r : = ( X t r T X t r + λ t r I d ) − 1 . (35) H_{\mathrm{sr}} := \left(X_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}+\lambda_{\mathrm{sr}}I_d\right)^{-1}, \qquad H_{\mathrm{tr}} := \left(X_{\mathrm{tr}}^{\mathsf{T}}X_{\mathrm{tr}}+\lambda_{\mathrm{tr}}I_d\right)^{-1}. \tag*{(35)} H sr := ( X sr T X sr + λ sr I d ) − 1 , H tr := ( X tr T X tr + λ tr I d ) − 1 . (35)
We use throughout the effective regularization parameters τ s r , τ t r \tau_{\mathrm{sr}},\tau_{\mathrm{tr}} τ sr , τ tr , the matrices W s r , W t r W_{\mathrm{sr}},W_{\mathrm{tr}} W sr , W tr , and the prefactors c s r , c t r c_{\mathrm{sr}},c_{\mathrm{tr}} c sr , c tr defined in (7) , (8) , and (9) .
The proof of Theorem 1 has two steps. First, we decompose the risk exactly into a bias term and two variance terms. We then control each term by its deterministic counterpart.
Lemma 2 (Exact risk decomposition). The conditional excess risk satisfies
R = B + V s r + V t r , (36) \mathcal{R} = \mathcal{B} + \mathcal{V}_{\mathrm{sr}} + \mathcal{V}_{\mathrm{tr}}, \tag*{(36)} R = B + V sr + V tr , (36) where
B : = λ t r 2 ( β s r − β t r − λ s r H s r β s r ) T H t r Σ t r H t r ( β s r − β t r − λ s r H s r β s r ) , (37) \mathcal{B} := \lambda_{\mathrm{tr}}^2\left(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}\right)^{\mathsf{T}}H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}\left(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}\right), \tag*{(37)} B := λ tr 2 ( β sr − β tr − λ sr H sr β sr ) T H tr Σ tr H tr ( β sr − β tr − λ sr H sr β sr ) , (37) V s r : = λ t r 2 σ s r 2 Tr ( H t r Σ t r H t r H s r X s r T X s r H s r ) , (38) \mathcal{V}_{\mathrm{sr}} := \lambda_{\mathrm{tr}}^2\sigma_{\mathrm{sr}}^2\operatorname{Tr}\left(H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}H_{\mathrm{sr}}\right), \tag*{(38)} V sr := λ tr 2 σ sr 2 Tr ( H tr Σ tr H tr H sr X sr T X sr H sr ) , (38) and
V t r : = σ t r 2 Tr ( H t r Σ t r H t r X t r T X t r ) . (39) \mathcal{V}_{\mathrm{tr}} := \sigma_{\mathrm{tr}}^2\operatorname{Tr}\left(H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}X_{\mathrm{tr}}\right). \tag*{(39)} V tr := σ tr 2 Tr ( H tr Σ tr H tr X tr T X tr ) . (39) Proof. Using H t r X t r T X t r = I d − λ t r H t r H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}X_{\mathrm{tr}}=I_d-\lambda_{\mathrm{tr}}H_{\mathrm{tr}} H tr X tr T X tr = I d − λ tr H tr , the error can be written as
β ^ t r − β t r = λ t r H t r ( β ^ s r − β t r ) + H t r X t r T ϵ t r . (40) \widehat{\beta}_{\mathrm{tr}}-\beta_{\mathrm{tr}}=\lambda_{\mathrm{tr}}H_{\mathrm{tr}}\left(\widehat{\beta}_{\mathrm{sr}}-\beta_{\mathrm{tr}}\right)+H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}\epsilon_{\mathrm{tr}}. \tag*{(40)} β tr − β tr = λ tr H tr ( β sr − β tr ) + H tr X tr T ϵ tr . (40) Similarly, since H s r X s r T X s r = I d − λ s r H s r H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}=I_d-\lambda_{\mathrm{sr}}H_{\mathrm{sr}} H sr X sr T X sr = I d − λ sr H sr ,
β ^ s r = β s r − λ s r H s r β s r + H s r X s r T ϵ s r . (41) \widehat{\beta}_{\mathrm{sr}}=\beta_{\mathrm{sr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}+H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}\epsilon_{\mathrm{sr}}. \tag*{(41)} β sr = β sr − λ sr H sr β sr + H sr X sr T ϵ sr . (41) Substituting this into (40) gives
β ^ t r − β t r = λ t r H t r ( β s r − β t r − λ s r H s r β s r ) + λ t r H t r H s r X s r T ϵ s r + H t r X t r T ϵ t r . (42) \widehat{\beta}_{\mathrm{tr}}-\beta_{\mathrm{tr}}=\lambda_{\mathrm{tr}}H_{\mathrm{tr}}\left(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}\right)+\lambda_{\mathrm{tr}}H_{\mathrm{tr}}H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}\epsilon_{\mathrm{sr}}+H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}\epsilon_{\mathrm{tr}}. \tag*{(42)} β tr − β tr = λ tr H tr ( β sr − β tr − λ sr H sr β sr ) + λ tr H tr H sr X sr T ϵ sr + H tr X tr T ϵ tr . (42) Conditional on X s r X_{\mathrm{sr}} X sr and X t r X_{\mathrm{tr}} X tr , the first term is deterministic, while the two noise terms are independent and zero mean. Hence, all cross terms vanish in the conditional risk. The deterministic term is exactly B \mathcal{B} B by (37) . For the source-noise term, using E [ ϵ s r ϵ s r T ] = σ s r 2 I \mathbb{E}[\epsilon_{\mathrm{sr}}\epsilon_{\mathrm{sr}}^{\mathsf{T}}]=\sigma_{\mathrm{sr}}^2I E [ ϵ sr ϵ sr T ] = σ sr 2 I and E [ ϵ T A ϵ ] = Tr ( A E [ ϵ ϵ T ] ) \mathbb{E}[\epsilon^{\mathsf{T}}A\epsilon]=\operatorname{Tr}(A\mathbb{E}[\epsilon\epsilon^{\mathsf{T}}]) E [ ϵ T A ϵ ] = Tr ( A E [ ϵ ϵ T ]) ,
λ t r 2 E [ ϵ s r T X s r H s r H t r Σ t r H t r H s r X s r T ϵ s r ∣ X s r , X t r ] = λ t r 2 σ s r 2 Tr ( H t r Σ t r H t r H s r X s r T X s r H s r ) = V s r , \lambda_{\mathrm{tr}}^2\mathbb{E}\left[\epsilon_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}H_{\mathrm{sr}}H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}\epsilon_{\mathrm{sr}}\mid X_{\mathrm{sr}},X_{\mathrm{tr}}\right]=\lambda_{\mathrm{tr}}^2\sigma_{\mathrm{sr}}^2\operatorname{Tr}\left(H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}H_{\mathrm{sr}}\right)=\mathcal{V}_{\mathrm{sr}}, λ tr 2 E [ ϵ sr T X sr H sr H tr Σ tr H tr H sr X sr T ϵ sr ∣ X sr , X tr ] = λ tr 2 σ sr 2 Tr ( H tr Σ tr H tr H sr X sr T X sr H sr ) = V sr , where we used the cyclicity of the trace. Likewise, since E [ ϵ t r ϵ t r T ] = σ t r 2 I \mathbb{E}[\epsilon_{\mathrm{tr}}\epsilon_{\mathrm{tr}}^{\mathsf{T}}]=\sigma_{\mathrm{tr}}^2I E [ ϵ tr ϵ tr T ] = σ tr 2 I ,
E [ ϵ t r T X t r H t r Σ t r H t r X t r T ϵ t r ∣ X s r , X t r ] = σ t r 2 Tr ( H t r Σ t r H t r X t r T X t r ) = V t r . \mathbb{E}\left[\epsilon_{\mathrm{tr}}^{\mathsf{T}}X_{\mathrm{tr}}H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}\epsilon_{\mathrm{tr}}\mid X_{\mathrm{sr}},X_{\mathrm{tr}}\right]=\sigma_{\mathrm{tr}}^2\operatorname{Tr}\left(H_{\mathrm{tr}}\Sigma_{\mathrm{tr}}H_{\mathrm{tr}}X_{\mathrm{tr}}^{\mathsf{T}}X_{\mathrm{tr}}\right)=\mathcal{V}_{\mathrm{tr}}. E [ ϵ tr T X tr H tr Σ tr H tr X tr T ϵ tr ∣ X sr , X tr ] = σ tr 2 Tr ( H tr Σ tr H tr X tr T X tr ) = V tr . Combining the three terms gives (36) .
Proposition 1 (Fine-tuning variance). Let ρ t r : = ρ λ t r ( n t r ) \rho_{tr}:=\rho_{\lambda_{tr}}(n_{tr}) ρ t r := ρ λ t r ( n t r ) be defined as in (Wu et al., 2026a, Appendix B). Fix D , K > 0 D,K>0 D , K > 0 and assume
λ t r ρ t r ≥ ∥ Σ t r ∥ op n t r − K , ρ t r 5 / 2 log 3 / 2 ( n t r ) ≤ K n t r . \lambda_{tr}\rho_{tr}\ge\lVert\Sigma_{tr}\rVert_{\operatorname{op}}n_{tr}^{-K}, \qquad\rho_{tr}^{5/2}\log^{3/2}(n_{tr})\le K\sqrt{n_{tr}}. λ t r ρ t r ≥ ∥ Σ t r ∥ op n t r − K , ρ t r 5/2 log 3/2 ( n t r ) ≤ K n t r . Define
V ‾ t r : = σ t r 2 Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] . (43) \overline{V}_{tr}:=\sigma_{tr}^{2}\frac{\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}. \tag*{(43)} V t r := σ t r 2 n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] . (43) Then, for sufficiently large n t r n_{tr} n t r , with probability at least 1 − n t r − D 1-n_{tr}^{-D} 1 − n t r − D ,
∣ V t r − V ‾ t r ∣ ≤ C D , K ρ t r 6 log 3 / 2 ( n t r ) n t r V ‾ t r . \left|V_{tr}-\overline{V}_{tr}\right|\le C_{D,K}\frac{\rho_{tr}^{6}\log^{3/2}(n_{tr})}{\sqrt{n_{tr}}}\overline{V}_{tr}. V t r − V t r ≤ C D , K n t r ρ t r 6 log 3/2 ( n t r ) V t r .
Proof. By (39) ,
V t r = σ t r 2 Tr ( H t r Σ t r H t r X t r ⊤ X t r ) . V_{tr}=\sigma_{tr}^{2}\operatorname{Tr}\left(H_{tr}\Sigma_{tr}H_{tr}X_{tr}^{\top}X_{tr}\right). V t r = σ t r 2 Tr ( H t r Σ t r H t r X t r ⊤ X t r ) .
The trace above is exactly the functional in (Wu et al., 2026a, Theorem 5, equation (37)) with A = I d A=I_d A = I d . Therefore, that equation gives
σ t r 2 Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] = V ‾ t r . \sigma_{tr}^{2}\frac{\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}=\overline{V}_{tr}. σ t r 2 n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] = V t r .
More precisely, the finite-sample bound in the same theorem yields
∣ V t r − V ‾ t r ∣ ≤ C D , K ρ t r 6 log 3 / 2 ( n t r ) n t r V ‾ t r . \left|V_{tr}-\overline{V}_{tr}\right|\le C_{D,K}\frac{\rho_{tr}^{6}\log^{3/2}(n_{tr})}{\sqrt{n_{tr}}}\overline{V}_{tr}. V t r − V t r ≤ C D , K n t r ρ t r 6 log 3/2 ( n t r ) V t r .
with probability at least 1 − n t r − D 1-n_{tr}^{-D} 1 − n t r − D .
Proposition 2 (Pretraining variance). Assume the two design matrices are independent and satisfy the regularization conditions of Proposition 1. Set
ε t r : = ρ t r 6 log 5 / 2 ( n t r ) n t r , ε s r : = ρ s r 6 log 3 / 2 ( n s r ) n s r , \varepsilon_{tr}:=\frac{\rho_{tr}^{6}\log^{5/2}(n_{tr})}{\sqrt{n_{tr}}},\qquad\varepsilon_{sr}:=\frac{\rho_{sr}^{6}\log^{3/2}(n_{sr})}{\sqrt{n_{sr}}}, ε t r := n t r ρ t r 6 log 5/2 ( n t r ) , ε sr := n sr ρ sr 6 log 3/2 ( n sr ) , and assume ε s r , ε t r ≤ 1 \varepsilon_{sr},\varepsilon_{tr}\le1 ε sr , ε t r ≤ 1 . Define
V ‾ s r : = σ s r 2 c t r c s r Tr ( W t r W s r ) . (44) \overline{V}_{sr}:=\sigma_{sr}^{2}c_{tr}c_{sr}\operatorname{Tr}(W_{tr}W_{sr}). \tag*{(44)} V sr := σ sr 2 c t r c sr Tr ( W t r W sr ) . (44) Then, for sufficiently large sample sizes, with probability at least 1 − n s r − D − n t r − D 1-n_{sr}^{-D}-n_{tr}^{-D} 1 − n sr − D − n t r − D ,
∣ V s r − V ‾ s r ∣ ≤ C D , K ( ε s r + ε t r ) V ‾ s r . \left|V_{sr}-\overline{V}_{sr}\right|\le C_{D,K}(\varepsilon_{sr}+\varepsilon_{tr})\overline{V}_{sr}. V sr − V sr ≤ C D , K ( ε sr + ε t r ) V sr .
Proof. By (38) ,
V s r = λ t r 2 σ s r 2 Tr ( H t r Σ t r H t r H s r X s r ⊤ X s r H s r ) . V_{sr}=\lambda_{tr}^{2}\sigma_{sr}^{2}\operatorname{Tr}\left(H_{tr}\Sigma_{tr}H_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right). V sr = λ t r 2 σ sr 2 Tr ( H t r Σ t r H t r H sr X sr ⊤ X sr H sr ) .
We remove the two random designs one at a time. First, we condition on X s r X_{sr} X sr , so that H s r X s r ⊤ X s r H s r H_{sr}X_{sr}^{\top}X_{sr}H_{sr} H sr X sr ⊤ X sr H sr is fixed and positive semidefinite. The trace in V s r V_{sr} V sr is the functional in (Wu et al., 2026a, Theorem 5, equation (36)) with
A t r = Σ t r − 1 / 2 H s r X s r ⊤ X s r H s r Σ t r − 1 / 2 , B t r = I d . A_{tr}=\Sigma_{tr}^{-1/2}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\Sigma_{tr}^{-1/2},\qquad B_{tr}=I_d. A t r = Σ t r − 1/2 H sr X sr ⊤ X sr H sr Σ t r − 1/2 , B t r = I d .
For B t r = I d B_{tr}=I_d B t r = I d , the deterministic term in (Wu et al., 2026a, equation (36)) is
τ t r 2 λ t r 2 n t r Tr [ A t r Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] . \frac{\tau_{tr}^{2}}{\lambda_{tr}^{2}}\frac{n_{tr}\operatorname{Tr}\left[A_{tr}\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]}. λ t r 2 τ t r 2 n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] n t r Tr [ A t r Σ t r 2 ( Σ t r + τ t r I d ) − 2 ] .
Multiplying by λ t r 2 σ s r 2 \lambda_{tr}^{2}\sigma_{sr}^{2} λ t r 2 σ sr 2 and substituting A t r A_{tr} A t r gives σ s r 2 c t r Tr ( W t r H s r X s r ⊤ X s r H s r ) \sigma_{sr}^{2}c_{tr}\operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right) σ sr 2 c t r Tr ( W t r H sr X sr ⊤ X sr H sr ) . Hence, with probability at least 1 − n t r − D 1-n_{tr}^{-D} 1 − n t r − D ,
∣ V s r − σ s r 2 c t r Tr ( W t r H s r X s r ⊤ X s r H s r ) ∣ ≤ C D , K ε t r σ s r 2 c t r Tr ( W t r H s r X s r ⊤ X s r H s r ) . (45) \left|V_{sr}-\sigma_{sr}^{2}c_{tr}\operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right)\right|\le C_{D,K}\varepsilon_{tr}\sigma_{sr}^{2}c_{tr}\operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right). \tag*{(45)} V sr − σ sr 2 c t r Tr ( W t r H sr X sr ⊤ X sr H sr ) ≤ C D , K ε t r σ sr 2 c t r Tr ( W t r H sr X sr ⊤ X sr H sr ) . (45)
Next, note that Tr ( W t r H s r X s r ⊤ X s r H s r ) \operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right) Tr ( W t r H sr X sr ⊤ X sr H sr ) is the functional in (Wu et al., 2026a, Theorem 5, equation (37)) with A s r = Σ s r − 1 / 2 W t r Σ s r − 1 / 2 A_{sr}=\Sigma_{sr}^{-1/2}W_{tr}\Sigma_{sr}^{-1/2} A sr = Σ sr − 1/2 W t r Σ sr − 1/2 . Its deterministic term is
σ s r 2 c t r Tr [ W t r Σ s r ( Σ s r + τ s r I d ) − 2 ] n s r − Tr [ Σ s r 2 ( Σ s r + τ s r I d ) − 2 ] = σ s r 2 c t r c s r Tr ( W t r W s r ) = V ‾ s r . \sigma_{sr}^{2}c_{tr}\frac{\operatorname{Tr}\left[W_{tr}\Sigma_{sr}(\Sigma_{sr}+\tau_{sr}I_d)^{-2}\right]}{n_{sr}-\operatorname{Tr}\left[\Sigma_{sr}^{2}(\Sigma_{sr}+\tau_{sr}I_d)^{-2}\right]}=\sigma_{sr}^{2}c_{tr}c_{sr}\operatorname{Tr}(W_{tr}W_{sr})=\overline{V}_{sr}. σ sr 2 c t r n sr − Tr [ Σ sr 2 ( Σ sr + τ sr I d ) − 2 ] Tr [ W t r Σ sr ( Σ sr + τ sr I d ) − 2 ] = σ sr 2 c t r c sr Tr ( W t r W sr ) = V sr .
Therefore, with probability at least 1 − n s r − D 1-n_{sr}^{-D} 1 − n sr − D ,
∣ σ s r 2 c t r Tr ( W t r H s r X s r ⊤ X s r H s r ) − V ‾ s r ∣ ≤ C D , K ε s r V ‾ s r . (46) \left|\sigma_{sr}^{2}c_{tr}\operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right)-\overline{V}_{sr}\right|\le C_{D,K}\varepsilon_{sr}\overline{V}_{sr}. \tag*{(46)} σ sr 2 c t r Tr ( W t r H sr X sr ⊤ X sr H sr ) − V sr ≤ C D , K ε sr V sr . (46)
Since ε s r ≤ 1 \varepsilon_{sr} \le1 ε sr ≤ 1 , (46) implies
σ s r 2 c t r Tr ( W t r H s r X s r ⊤ X s r H s r ) ≤ C D , K V ‾ s r . \sigma_{sr}^{2}c_{tr}\operatorname{Tr}\left(W_{tr}H_{sr}X_{sr}^{\top}X_{sr}H_{sr}\right) \le C_{D,K}\overline{V}_{sr}. σ sr 2 c t r Tr ( W t r H sr X sr ⊤ X sr H sr ) ≤ C D , K V sr .
Thus, by the triangle inequality, (45) and (46) give
∣ V s r − V ‾ s r ∣ ≤ C D , K ( ε s r + ε t r ) V ‾ s r . \left|V_{sr}-\overline{V}_{sr}\right| \le C_{D,K}(\varepsilon_{sr}+\varepsilon_{tr})\overline{V}_{sr}. V sr − V sr ≤ C D , K ( ε sr + ε t r ) V sr .
Since the first bound holds conditionally on X s r X_{sr} X sr with failure probability at most n t r − D n_{tr}^{-D} n t r − D , averaging over X s r X_{sr} X sr gives the same unconditional failure probability. A union bound with the source event then gives probability at least 1 − n s r − D − n t r − D 1-n_{sr}^{-D}-n_{tr}^{-D} 1 − n sr − D − n t r − D .
Proposition 3 (Bias). Assume
ε s r : = ρ s r 6 log 5 / 2 ( n s r ) n s r , ε t r : = ρ t r 6 log 5 / 2 ( n t r ) n t r . \varepsilon_{sr}:=\frac{\rho_{sr}^{6}\log^{5/2}(n_{sr})}{\sqrt{n_{sr}}}, \qquad\varepsilon_{tr}:=\frac{\rho_{tr}^{6}\log^{5/2}(n_{tr})}{\sqrt{n_{tr}}}. ε sr := n sr ρ sr 6 log 5/2 ( n sr ) , ε t r := n t r ρ t r 6 log 5/2 ( n t r ) . Under the assumptions of Proposition 1, let β s r , β t r ∈ R d \beta_{sr},\beta_{tr}\in\mathbb{R}^{d} β sr , β t r ∈ R d be fixed. Define
B ‾ : = c t r [ Σ s r ( Σ s r + τ s r I d ) − 1 β s r − β t r ] ⊤ W t r [ Σ s r ( Σ s r + τ s r I d ) − 1 β s r − β t r ] + c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r , (47) \overline{B}:=c_{tr}\left[\Sigma_{sr}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}\beta_{sr}-\beta_{tr}\right]^{\top}W_{tr}\left[\Sigma_{sr}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}\beta_{sr}-\beta_{tr}\right]+c_{tr}c_{sr}\tau_{sr}^{2}\operatorname{Tr}(W_{tr}W_{sr})\beta_{sr}^{\top}W_{sr}\beta_{sr}, \tag*{(47)} B := c t r [ Σ sr ( Σ sr + τ sr I d ) − 1 β sr − β t r ] ⊤ W t r [ Σ sr ( Σ sr + τ sr I d ) − 1 β sr − β t r ] + c t r c sr τ sr 2 Tr ( W t r W sr ) β sr ⊤ W sr β sr , (47) and
E s r : = c t r τ s r 2 ( β s r ⊤ ( Σ s r + τ s r I d ) − 1 β s r ) ∥ W t r 1 / 2 ( Σ s r + τ s r I d ) − 1 W t r 1 / 2 ∥ o p . (48) \mathcal{E}_{sr}:=c_{tr}\tau_{sr}^{2}\left(\beta_{sr}^{\top}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}\beta_{sr}\right)\left\|W_{tr}^{1/2}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}W_{tr}^{1/2}\right\|_{\mathrm{op}}. \tag*{(48)} E sr := c t r τ sr 2 ( β sr ⊤ ( Σ sr + τ sr I d ) − 1 β sr ) W t r 1/2 ( Σ sr + τ sr I d ) − 1 W t r 1/2 op . (48) Then, for sufficiently large sample sizes, with probability at least 1 − n s r − D − n t r − D 1-n_{sr}^{-D}-n_{tr}^{-D} 1 − n sr − D − n t r − D ,
∣ B − B ‾ ∣ ≤ C D , K [ ( ε s r + ε t r ) B ‾ + ε s r E s r ] . (49) \left|B-\overline{B}\right|\le C_{D,K}\left[(\varepsilon_{sr}+\varepsilon_{tr})\overline{B}+\varepsilon_{sr}\mathcal{E}_{sr}\right]. \tag*{(49)} B − B ≤ C D , K [ ( ε sr + ε t r ) B + ε sr E sr ] . (49)
Proof. From (37) , we have
B = λ t r 2 ( β s r − β t r − λ s r H s r β s r ) ⊤ H t r Σ t r H t r ( β s r − β t r − λ s r H s r β s r ) . B=\lambda_{tr}^{2}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}H_{tr}\Sigma_{tr}H_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right). B = λ t r 2 ( β sr − β t r − λ sr H sr β sr ) ⊤ H t r Σ t r H t r ( β sr − β t r − λ sr H sr β sr ) .
We first remove the target design and condition on X s r X_{sr} X sr , so that β s r − β t r − λ s r H s r β s r \beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr} β sr − β t r − λ sr H sr β sr is fixed. By cyclic property of trace, the quadratic form above is the functional in Wu et al. (2026a) Theorem 5, equation (36) with
A t r = Σ t r − 1 / 2 ( β s r − β t r − λ s r H s r β s r ) ( β s r − β t r − λ s r H s r β s r ) ⊤ Σ t r − 1 / 2 , B t r = I d . A_{tr}=\Sigma_{tr}^{-1/2}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}\Sigma_{tr}^{-1/2}, \qquad B_{tr}=I_{d}. A t r = Σ t r − 1/2 ( β sr − β t r − λ sr H sr β sr ) ( β sr − β t r − λ sr H sr β sr ) ⊤ Σ t r − 1/2 , B t r = I d .
The deterministic term in Wu et al. (2026a), equation (36), after multiplying by λ t r 2 \lambda_{tr}^{2} λ t r 2 , is
c t r ( β s r − β t r − λ s r H s r β s r ) ⊤ W t r ( β s r − β t r − λ s r H s r β s r ) . c_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}W_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right). c t r ( β sr − β t r − λ sr H sr β sr ) ⊤ W t r ( β sr − β t r − λ sr H sr β sr ) .
Hence, with probability at least 1 − n t r − D 1-n_{tr}^{-D} 1 − n t r − D ,
∣ B − c t r ( β s r − β t r − λ s r H s r β s r ) ⊤ W t r ( β s r − β t r − λ s r H s r β s r ) ∣ ≤ C D , K ε t r c t r ( β s r − β t r − λ s r H s r β s r ) ⊤ W t r ( β s r − β t r − λ s r H s r β s r ) . (50) \left|B-c_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}W_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)\right|
\le C_{D,K}\varepsilon_{tr}c_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}W_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right). \tag*{(50)} B − c t r ( β sr − β t r − λ sr H sr β sr ) ⊤ W t r ( β sr − β t r − λ sr H sr β sr ) ≤ C D , K ε t r c t r ( β sr − β t r − λ sr H sr β sr ) ⊤ W t r ( β sr − β t r − λ sr H sr β sr ) . (50)
We next remove the source design. It remains to analyze
c t r ( β s r − β t r − λ s r H s r β s r ) ⊤ W t r ( β s r − β t r − λ s r H s r β s r ) . c_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right)^{\top}W_{tr}\left(\beta_{sr}-\beta_{tr}-\lambda_{sr}H_{sr}\beta_{sr}\right). c t r ( β sr − β t r − λ sr H sr β sr ) ⊤ W t r ( β sr − β t r − λ sr H sr β sr ) .
Expanding the remaining quadratic form gives
c t r [ ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) − 2 λ s r ( β s r − β t r ) ⊤ W t r H s r β s r + λ s r 2 β s r ⊤ H s r W t r H s r β s r ] . c_{tr}\left[(\beta_{sr}-\beta_{tr})^{\top}W_{tr}(\beta_{sr}-\beta_{tr})-2\lambda_{sr}(\beta_{sr}-\beta_{tr})^{\top}W_{tr}H_{sr}\beta_{sr}+\lambda_{sr}^{2}\beta_{sr}^{\top}H_{sr}W_{tr}H_{sr}\beta_{sr}\right]. c t r [ ( β sr − β t r ) ⊤ W t r ( β sr − β t r ) − 2 λ sr ( β sr − β t r ) ⊤ W t r H sr β sr + λ sr 2 β sr ⊤ H sr W t r H sr β sr ] .
The first term is already deterministic, so only the cross and quadratic terms remain. For the cross term, we apply Wu et al. (2026a) Theorem 6, equation (38) with
A = Σ s r − 1 / 2 β s r ( β s r − β t r ) ⊤ W t r Σ s r − 1 / 2 . A=\Sigma_{sr}^{-1/2}\beta_{sr}(\beta_{sr}-\beta_{tr})^{\top}W_{tr}\Sigma_{sr}^{-1/2}. A = Σ sr − 1/2 β sr ( β sr − β t r ) ⊤ W t r Σ sr − 1/2 .
Its deterministic term is
2 c t r τ s r ( β s r − β t r ) ⊤ W t r ( Σ s r + τ s r I d ) − 1 β s r . 2c_{tr}\tau_{sr}(\beta_{sr}-\beta_{tr})^{\top}W_{tr}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}\beta_{sr}. 2 c t r τ sr ( β sr − β t r ) ⊤ W t r ( Σ sr + τ sr I d ) − 1 β sr .
Substituting the above choice of A A A into the error term in Wu et al. (2026a), Theorem 6, equation (38) yields a quadratic form ( Σ s r + τ s r I d ) − 1 (\Sigma_{sr}+\tau_{sr}I_{d})^{-1} ( Σ sr + τ sr I d ) − 1 . To reduce it to ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) (\beta_{sr}-\beta_{tr})^{\top}W_{tr}(\beta_{sr}-\beta_{tr}) ( β sr − β t r ) ⊤ W t r ( β sr − β t r ) , we use
W t r ( Σ s r + τ s r I d ) − 1 W t r ⪯ ∥ W t r 1 / 2 ( Σ s r + τ s r I d ) − 1 W t r 1 / 2 ∥ o p W t r , W_{tr}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}W_{tr}\preceq\left\|W_{tr}^{1/2}(\Sigma_{sr}+\tau_{sr}I_{d})^{-1}W_{tr}^{1/2}\right\|_{\mathrm{op}}W_{tr}, W t r ( Σ sr + τ sr I d ) − 1 W t r ⪯ W t r 1/2 ( Σ sr + τ sr I d ) − 1 W t r 1/2 op W t r ,
@references#+#+#+#+ which gives
∣ 2 c t r λ s r ( β s r − β t r ) ⊤ W t r H s r β s r − 2 c t r τ s r ( β s r − β t r ) ⊤ W t r ( Σ s r + τ s r I d ) − 1 β s r ∣ ≤ C D , K ε s r E s r c t r ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) . (51) \left|2c_{\mathrm{tr}}\lambda_{\mathrm{sr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}-2c_{\mathrm{tr}}\tau_{\mathrm{sr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}\right|
\le C_{D,K}\varepsilon_{\mathrm{sr}}\sqrt{\mathcal{E}_{\mathrm{sr}}c_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})}. \tag*{(51)} 2 c tr λ sr ( β sr − β tr ) ⊤ W tr H sr β sr − 2 c tr τ sr ( β sr − β tr ) ⊤ W tr ( Σ sr + τ sr I d ) − 1 β sr ≤ C D , K ε sr E sr c tr ( β sr − β tr ) ⊤ W tr ( β sr − β tr ) . (51)
For the quadratic term, we apply (Wu et al., 2026a, Theorem 6, equation (40)) with
A = Σ s r − 1 / 2 β s r β s r ⊤ Σ s r − 1 / 2 , B = Σ s r − 1 / 2 W t r Σ s r − 1 / 2 . A=\Sigma_{\mathrm{sr}}^{-1/2}\beta_{\mathrm{sr}}\beta_{\mathrm{sr}}^{\top}\Sigma_{\mathrm{sr}}^{-1/2},\qquad B=\Sigma_{\mathrm{sr}}^{-1/2}W_{\mathrm{tr}}\Sigma_{\mathrm{sr}}^{-1/2}. A = Σ sr − 1/2 β sr β sr ⊤ Σ sr − 1/2 , B = Σ sr − 1/2 W tr Σ sr − 1/2 .
Its deterministic term is
c t r τ s r 2 β s r ⊤ ( Σ s r + τ s r I d ) − 1 W t r ( Σ s r + τ s r I d ) − 1 β s r + c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r . c_{\mathrm{tr}}\tau_{\mathrm{sr}}^{2}\beta_{\mathrm{sr}}^{\top}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}+c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^{2}\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta_{\mathrm{sr}}^{\top}W_{\mathrm{sr}}\beta_{\mathrm{sr}}. c tr τ sr 2 β sr ⊤ ( Σ sr + τ sr I d ) − 1 W tr ( Σ sr + τ sr I d ) − 1 β sr + c tr c sr τ sr 2 Tr ( W tr W sr ) β sr ⊤ W sr β sr .
Therefore,
∣ c t r λ s r 2 β s r ⊤ H s r W t r H s r β s r − c t r τ s r 2 β s r ⊤ ( Σ s r + τ s r I d ) − 1 W t r ( Σ s r + τ s r I d ) − 1 β s r − c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r ∣ ≤ C D , K ε s r ( E s r + c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r ) . (52) \left|c_{\mathrm{tr}}\lambda_{\mathrm{sr}}^{2}\beta_{\mathrm{sr}}^{\top}H_{\mathrm{sr}}W_{\mathrm{tr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}}-c_{\mathrm{tr}}\tau_{\mathrm{sr}}^{2}\beta_{\mathrm{sr}}^{\top}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}-c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^{2}\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta_{\mathrm{sr}}^{\top}W_{\mathrm{sr}}\beta_{\mathrm{sr}}\right|
\le C_{D,K}\varepsilon_{\mathrm{sr}}\left(\mathcal{E}_{\mathrm{sr}}+c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^{2}\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta_{\mathrm{sr}}^{\top}W_{\mathrm{sr}}\beta_{\mathrm{sr}}\right). \tag*{(52)} c tr λ sr 2 β sr ⊤ H sr W tr H sr β sr − c tr τ sr 2 β sr ⊤ ( Σ sr + τ sr I d ) − 1 W tr ( Σ sr + τ sr I d ) − 1 β sr − c tr c sr τ sr 2 Tr ( W tr W sr ) β sr ⊤ W sr β sr ≤ C D , K ε sr ( E sr + c tr c sr τ sr 2 Tr ( W tr W sr ) β sr ⊤ W sr β sr ) . (52)
We now identify the deterministic terms produced by these two calculations. Combining them with the deterministic first term in the expansion gives
c t r ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) − 2 c t r τ s r ( β s r − β t r ) ⊤ W t r ( Σ s r + τ s r I d ) − 1 β s r + c t r τ s r 2 β s r ⊤ ( Σ s r + τ s r I d ) − 1 W t r ( Σ s r + τ s r I d ) − 1 β s r + c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r = B ‾ . c_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})-2c_{\mathrm{tr}}\tau_{\mathrm{sr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}
+c_{\mathrm{tr}}\tau_{\mathrm{sr}}^{2}\beta_{\mathrm{sr}}^{\top}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}+c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^{2}\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta_{\mathrm{sr}}^{\top}W_{\mathrm{sr}}\beta_{\mathrm{sr}}=\overline{B}. c tr ( β sr − β tr ) ⊤ W tr ( β sr − β tr ) − 2 c tr τ sr ( β sr − β tr ) ⊤ W tr ( Σ sr + τ sr I d ) − 1 β sr + c tr τ sr 2 β sr ⊤ ( Σ sr + τ sr I d ) − 1 W tr ( Σ sr + τ sr I d ) − 1 β sr + c tr c sr τ sr 2 Tr ( W tr W sr ) β sr ⊤ W sr β sr = B .
The first three terms form the quadratic
c t r ( β s r − β t r − τ s r ( Σ s r + τ s r I d ) − 1 β s r ) ⊤ W t r ( β s r − β t r − τ s r ( Σ s r + τ s r I d ) − 1 β s r ) . c_{\mathrm{tr}}\left(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\tau_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}\right)^{\top}W_{\mathrm{tr}}\left(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\tau_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}\right). c tr ( β sr − β tr − τ sr ( Σ sr + τ sr I d ) − 1 β sr ) ⊤ W tr ( β sr − β tr − τ sr ( Σ sr + τ sr I d ) − 1 β sr ) .
Using the resolvent identity
I d − τ s r ( Σ s r + τ s r I d ) − 1 = Σ s r ( Σ s r + τ s r I d ) − 1 , I_d-\tau_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}=\Sigma_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}, I d − τ sr ( Σ sr + τ sr I d ) − 1 = Σ sr ( Σ sr + τ sr I d ) − 1 ,
this equals
c t r [ Σ s r ( Σ s r + τ s r I d ) − 1 β s r − β t r ] ⊤ W t r [ Σ s r ( Σ s r + τ s r I d ) − 1 β s r − β t r ] = B ‾ 1 . c_{\mathrm{tr}}\left[\Sigma_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}\right]^{\top}W_{\mathrm{tr}}\left[\Sigma_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}\right]=\overline{B}_1. c tr [ Σ sr ( Σ sr + τ sr I d ) − 1 β sr − β tr ] ⊤ W tr [ Σ sr ( Σ sr + τ sr I d ) − 1 β sr − β tr ] = B 1 .
Together with the remaining trace term ( c t r c s r τ s r 2 Tr ( W t r W s r ) β s r ⊤ W s r β s r = B ‾ 2 ) (c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^{2}\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta_{\mathrm{sr}}^{\top}W_{\mathrm{sr}}\beta_{\mathrm{sr}}=\overline{B}_2) ( c tr c sr τ sr 2 Tr ( W tr W sr ) β sr ⊤ W sr β sr = B 2 ) , this is exactly B ‾ \overline{B} B in (47) .
It remains to control the errors. By the definition of E s r \mathcal{E}_{\mathrm{sr}} E sr ,
c t r τ s r 2 β s r ⊤ ( Σ s r + τ s r I d ) − 1 W t r ( Σ s r + τ s r I d ) − 1 β s r ≤ E s r . c_{\mathrm{tr}}\tau_{\mathrm{sr}}^{2}\beta_{\mathrm{sr}}^{\top}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}W_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I_d)^{-1}\beta_{\mathrm{sr}}\le\mathcal{E}_{\mathrm{sr}}. c tr τ sr 2 β sr ⊤ ( Σ sr + τ sr I d ) − 1 W tr ( Σ sr + τ sr I d ) − 1 β sr ≤ E sr .
Moreover,
c t r ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) ≤ 2 B ‾ + 2 E s r . c_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})\le2\overline{B}+2\mathcal{E}_{\mathrm{sr}}. c tr ( β sr − β tr ) ⊤ W tr ( β sr − β tr ) ≤ 2 B + 2 E sr .
Hence Young’s inequality gives
E s r c t r ( β s r − β t r ) ⊤ W t r ( β s r − β t r ) ≤ C ( B ‾ + E s r ) . \sqrt{\mathcal{E}_{\mathrm{sr}}c_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})^{\top}W_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}})}\le C(\overline{B}+\mathcal{E}_{\mathrm{sr}}). E sr c tr ( β sr − β tr ) ⊤ W tr ( β sr − β tr ) ≤ C ( B + E sr ) .
Since the trace correction in (52) is a nonnegative term in B ‾ \overline{B} B , combining (51) and (52) gives
∣ c t r ( β s r − β t r − λ s r H s r β s r ) ⊤ W t r ( β s r − β t r − λ s r H s r β s r ) − B ‾ ∣ ≤ C D , K ε s r ( B ‾ + E s r ) . \left|c_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}})^{\top}W_{\mathrm{tr}}(\beta_{\mathrm{sr}}-\beta_{\mathrm{tr}}-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_{\mathrm{sr}})-\overline{B}\right|\le C_{D,K}\varepsilon_{\mathrm{sr}}(\overline{B}+\mathcal{E}_{\mathrm{sr}}). c tr ( β sr − β tr − λ sr H sr β sr ) ⊤ W tr ( β sr − β tr − λ sr H sr β sr ) − B ≤ C D , K ε sr ( B + E sr ) .
Applying both source bounds with exponent D + 1 D+1 D + 1 makes their joint failure probability at most n s r − D n_{\mathrm{sr}}^{-D} n sr − D , after adjusting C D , K C_{D,K} C D , K . On this source event and the target event in (50) ,
∣ B − B ‾ ∣ ≤ C D , K [ ( ε s r + ε t r ) B ‾ + ε s r E s r ] , \left|B-\overline{B}\right|\le C_{D,K}\left[(\varepsilon_{\mathrm{sr}}+\varepsilon_{\mathrm{tr}})\overline{B}+\varepsilon_{\mathrm{sr}}\mathcal{E}_{\mathrm{sr}}\right], B − B ≤ C D , K [ ( ε sr + ε tr ) B + ε sr E sr ] ,
where ε t r ≤ 1 \varepsilon_{\mathrm{tr}}\le1 ε tr ≤ 1 is used to absorb the product of the target error with the source error. Finally, the target bound holds conditionally on X s r X_{\mathrm{sr}} X sr with failure probability at most n t r − D n_{\mathrm{tr}}^{-D} n tr − D , so averaging over X s r X_{\mathrm{sr}} X sr gives the same unconditional failure probability. A union bound with the source event gives probability at least 1 − n s r − D − n t r − D 1-n_{\mathrm{sr}}^{-D}-n_{\mathrm{tr}}^{-D} 1 − n sr − D − n tr − D .
Proof of Theorem 1. By Lemma 2 and the definition of R ‾ \overline{R} R ,
R = B + V s r + V t r , R ‾ = B ‾ + V ‾ s r + V ‾ t r . R=B+V_{\mathrm{sr}}+V_{\mathrm{tr}},\qquad\overline{R}=\overline{B}+\overline{V}_{\mathrm{sr}}+\overline{V}_{\mathrm{tr}}. R = B + V sr + V tr , R = B + V sr + V tr .
Define
ε s r : = ρ s r 6 log 5 / 2 ( n s r ) n s r , ε t r : = ρ t r 6 log 5 / 2 ( n t r ) n t r . \varepsilon_{\mathrm{sr}} := \frac{\rho_{\mathrm{sr}}^{6}\log^{5/2}(n_{\mathrm{sr}})}{\sqrt{n_{\mathrm{sr}}}}, \qquad\varepsilon_{\mathrm{tr}} := \frac{\rho_{\mathrm{tr}}^{6}\log^{5/2}(n_{\mathrm{tr}})}{\sqrt{n_{\mathrm{tr}}}}. ε sr := n sr ρ sr 6 log 5/2 ( n sr ) , ε tr := n tr ρ tr 6 log 5/2 ( n tr ) .
Since the above quantities dominate all error factors appearing in Propositions 1 to 3, by combining them and taking a union bound we obtain, with probability at least 1 − C D ( n s r − D + n t r − D ) 1-C_D(n_{\mathrm{sr}}^{-D}+n_{\mathrm{tr}}^{-D}) 1 − C D ( n sr − D + n tr − D ) ,
∣ R − R ‾ ∣ ≤ ∣ B − B ‾ ∣ + ∣ V s r − V ‾ s r ∣ + ∣ V t r − V ‾ t r ∣ ≤ C D , K [ ( ε s r + ε t r ) R ‾ + ε s r E s r ] . |\mathcal{R}-\overline{\mathcal{R}}| \leq|\mathcal{B}-\overline{\mathcal{B}}|+|\mathcal{V}_{\mathrm{sr}}-\overline{\mathcal{V}}_{\mathrm{sr}}|+|\mathcal{V}_{\mathrm{tr}}-\overline{\mathcal{V}}_{\mathrm{tr}}| \leq C_{D,K}\left[(\varepsilon_{\mathrm{sr}}+\varepsilon_{\mathrm{tr}})\overline{\mathcal{R}}+\varepsilon_{\mathrm{sr}}\mathcal{E}_{\mathrm{sr}}\right]. ∣ R − R ∣ ≤ ∣ B − B ∣ + ∣ V sr − V sr ∣ + ∣ V tr − V tr ∣ ≤ C D , K [ ( ε sr + ε tr ) R + ε sr E sr ] .
Since ε s r , ε t r ≤ 1 \varepsilon_{\mathrm{sr}},\varepsilon_{\mathrm{tr}}\leq1 ε sr , ε tr ≤ 1 , this also implies
R ≤ C D , K ( R ‾ + ε s r E s r ) . \mathcal{R}\leq C_{D,K}\left(\overline{\mathcal{R}}+\varepsilon_{\mathrm{sr}}\mathcal{E}_{\mathrm{sr}}\right). R ≤ C D , K ( R + ε sr E sr ) .
Proof of Theorem 3
Throughout this section, we use f n ≍ g n f_n\asymp g_n f n ≍ g n for two-sided constant-order equivalence, f n = Θ ~ ( g n ) f_n=\widetilde{\Theta}(g_n) f n = Θ ( g n ) for equivalence up to fixed powers of log n \log n log n , and x + : = max { x , 0 } x_+:=\max\{x,0\} x + := max { x , 0 } . The proof has three main steps. First, we determine the scaling of the effective source and target regularization parameters. Next, we derive the scaling of the spectral quantities appearing in the bias and variance terms. Finally, we combine these results to obtain the scaling of the total risk. The required power-law sum estimates are given at the end of the section.
We first derive the scaling of the effective regularizers. For eigenvalues ξ k ≍ k − α \xi_k\asymp k^{-\alpha} ξ k ≍ k − α , define
S 1 , α ( λ ) : = ∑ k ≥ 1 ξ k ξ k + λ , S 2 , α ( λ ) : = ∑ k ≥ 1 ξ k 2 ( ξ k + λ ) 2 . (53) S_{1,\alpha}(\lambda):=\sum_{k\geq1}\frac{\xi_k}{\xi_k+\lambda}, \qquad S_{2,\alpha}(\lambda):=\sum_{k\geq1}\frac{\xi_k^2}{(\xi_k+\lambda)^2}. \tag*{(53)} S 1 , α ( λ ) := k ≥ 1 ∑ ξ k + λ ξ k , S 2 , α ( λ ) := k ≥ 1 ∑ ( ξ k + λ ) 2 ξ k 2 . (53)
Lemma 3 (Effective-dimension traces). Let α > 1 \alpha>1 α > 1 . As λ → 0 + \lambda\to0^+ λ → 0 + ,
S 1 , α ( λ ) ≍ S 2 , α ( λ ) ≍ S 1 , α ( λ ) − S 2 , α ( λ ) ≍ λ − 1 / α . (54) S_{1,\alpha}(\lambda)\asymp S_{2,\alpha}(\lambda)\asymp S_{1,\alpha}(\lambda)-S_{2,\alpha}(\lambda)\asymp\lambda^{-1/\alpha}. \tag*{(54)} S 1 , α ( λ ) ≍ S 2 , α ( λ ) ≍ S 1 , α ( λ ) − S 2 , α ( λ ) ≍ λ − 1/ α . (54)
Proof. Since ξ k ≍ k − α \xi_k\asymp k^{-\alpha} ξ k ≍ k − α ,
ξ k ξ k + λ ≍ 1 1 + λ k α , ξ k 2 ( ξ k + λ ) 2 ≍ 1 ( 1 + λ k α ) 2 . \frac{\xi_k}{\xi_k+\lambda}\asymp\frac{1}{1+\lambda k^\alpha}, \qquad\frac{\xi_k^2}{(\xi_k+\lambda)^2}\asymp\frac{1}{(1+\lambda k^\alpha)^2}. ξ k + λ ξ k ≍ 1 + λ k α 1 , ( ξ k + λ ) 2 ξ k 2 ≍ ( 1 + λ k α ) 2 1 .
Therefore, Lemma 4 gives
S 1 , α ( λ ) ≍ S 2 , α ( λ ) ≍ λ − 1 / α . S_{1,\alpha}(\lambda)\asymp S_{2,\alpha}(\lambda)\asymp\lambda^{-1/\alpha}. S 1 , α ( λ ) ≍ S 2 , α ( λ ) ≍ λ − 1/ α .
It remains to control their difference. Let K : = λ − 1 / α K:=\lambda^{-1/\alpha} K := λ − 1/ α , so that λ K α = 1 \lambda K^\alpha=1 λ K α = 1 , and split the sum at K K K . Then,
S 1 , α ( λ ) − S 2 , α ( λ ) = ∑ k ≥ 1 λ ξ k ( ξ k + λ ) 2 S_{1,\alpha}(\lambda)-S_{2,\alpha}(\lambda)=\sum_{k\geq1}\frac{\lambda\xi_k}{(\xi_k+\lambda)^2} S 1 , α ( λ ) − S 2 , α ( λ ) = k ≥ 1 ∑ ( ξ k + λ ) 2 λ ξ k
≍ ∑ k ≥ 1 λ k α ( 1 + λ k α ) 2 = ∑ k ≤ K λ k α ( 1 + λ k α ) 2 + ∑ k > K λ k α ( 1 + λ k α ) 2 . \begin{aligned}
\asymp\sum_{k\geq1}\frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2} \\
&=\sum_{k\leq K}\frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}+\sum_{k>K}\frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}.
\end{aligned} ≍ k ≥ 1 ∑ ( 1 + λ k α ) 2 λ k α = k ≤ K ∑ ( 1 + λ k α ) 2 λ k α + k > K ∑ ( 1 + λ k α ) 2 λ k α .
For k ≤ K k\leq K k ≤ K , we have λ k α ≤ 1 \lambda k^\alpha\leq1 λ k α ≤ 1 , and hence
λ k α ( 1 + λ k α ) 2 ≍ λ k α . \frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}\asymp\lambda k^\alpha. ( 1 + λ k α ) 2 λ k α ≍ λ k α .
Thus
∑ k ≤ K λ k α ( 1 + λ k α ) 2 ≍ ∑ k ≤ K λ k α ≍ λ K α + 1 = λ − 1 / α . \sum_{k\leq K}\frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}\asymp\sum_{k\leq K}\lambda k^\alpha\asymp\lambda K^{\alpha+1}=\lambda^{-1/\alpha}. k ≤ K ∑ ( 1 + λ k α ) 2 λ k α ≍ k ≤ K ∑ λ k α ≍ λ K α + 1 = λ − 1/ α .
For k > K k>K k > K , we have λ k α > 1 \lambda k^\alpha>1 λ k α > 1 , so
λ k α ( 1 + λ k α ) 2 ≍ 1 λ k α . \frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}\asymp\frac{1}{\lambda k^\alpha}. ( 1 + λ k α ) 2 λ k α ≍ λ k α 1 .
Since α > 1 \alpha>1 α > 1 ,
∑ k > K λ k α ( 1 + λ k α ) 2 ≍ λ − 1 ∑ k > K k − α ≍ λ − 1 K 1 − α = λ − 1 / α . \sum_{k>K}\frac{\lambda k^\alpha}{(1+\lambda k^\alpha)^2}\asymp\lambda^{-1}\sum_{k>K}k^{-\alpha}\asymp\lambda^{-1}K^{1-\alpha}=\lambda^{-1/\alpha}. k > K ∑ ( 1 + λ k α ) 2 λ k α ≍ λ − 1 k > K ∑ k − α ≍ λ − 1 K 1 − α = λ − 1/ α .
Both parts are therefore of order λ − 1 / α \lambda^{-1/\alpha} λ − 1/ α , which proves (54) .
We next use these trace estimates to determine the scaling of the effective regularization parameter defined by the self-consistency equation.
Proposition 4 (Generic effective-ridge scaling). Let α > 1 \alpha> 1 α > 1 , N → ∞ N \to\infty N → ∞ , q > − 1 q > -1 q > − 1 , λ N ≍ N − q \lambda_N \asymp N^{-q} λ N ≍ N − q , and let μ N > 0 \mu_N > 0 μ N > 0 solve
μ N = N 1 + Tr ( Σ ( μ N Σ + λ N I ) − 1 ) (55) \mu_N = \frac{N}{1 + \operatorname{Tr}\left(\boldsymbol{\Sigma}(\mu_N\boldsymbol{\Sigma}+\lambda_N I)^{-1}\right)} \tag*{(55)} μ N = 1 + Tr ( Σ ( μ N Σ + λ N I ) − 1 ) N (55) where ξ k ( Σ ) ≍ k − α \xi_k(\boldsymbol{\Sigma}) \asymp k^{-\alpha} ξ k ( Σ ) ≍ k − α . Define τ N : = λ N / μ N \tau_N := \lambda_N/\mu_N τ N := λ N / μ N . Then the solution is unique and
μ N + S 1 , α ( τ N ) = N . (56) \mu_N + S_{1,\alpha}(\tau_N) = N. \tag*{(56)} μ N + S 1 , α ( τ N ) = N . (56) Furthermore,
τ N ≍ N − ( ( 1 + q ) ∧ α ) , (57) \tau_N \asymp N^{-((1+q)\wedge\alpha)}, \tag*{(57)} τ N ≍ N − (( 1 + q ) ∧ α ) , (57) and
μ N ≍ { N , q ≤ α − 1 , N α − q , q > α − 1. (58) \mu_N \asymp
\begin{cases}
N, & q \le\alpha- 1,\\
N^{\alpha-q}, & q > \alpha- 1.
\end{cases} \tag*{(58)} μ N ≍ { N , N α − q , q ≤ α − 1 , q > α − 1. (58) More precisely,
N − μ N ≍ { N ( 1 + q ) / α , q < α − 1 , N , q ≥ α − 1. N-\mu_N \asymp
\begin{cases}
N^{(1+q)/\alpha}, & q < \alpha- 1,\\
N, & q \ge\alpha- 1.
\end{cases} N − μ N ≍ { N ( 1 + q ) / α , N , q < α − 1 , q ≥ α − 1.
Proof. Factoring μ N \mu_N μ N in (55) gives (56) . The map μ ↦ μ + S 1 , α ( λ N / μ ) \mu\mapsto\mu+ S_{1,\alpha}(\lambda_N/\mu) μ ↦ μ + S 1 , α ( λ N / μ ) is continuous and strictly increasing from 0 0 0 to ∞ \infty ∞ , so the solution is unique.
We first show τ N → 0 \tau_N \to0 τ N → 0 . Otherwise, along some subsequence, τ N \tau_N τ N is bounded away from zero, so S 1 , α ( τ N ) = O ( 1 ) S_{1,\alpha}(\tau_N)=O(1) S 1 , α ( τ N ) = O ( 1 ) and μ N = N − O ( 1 ) \mu_N=N-O(1) μ N = N − O ( 1 ) . Hence τ N = λ N / μ N ≍ N − q − 1 → 0 \tau_N=\lambda_N/\mu_N\asymp N^{-q-1}\to0 τ N = λ N / μ N ≍ N − q − 1 → 0 , since q > − 1 q>-1 q > − 1 , a contradiction.
By Lemma 3,
N − μ N = S 1 , α ( τ N ) ≍ τ N − 1 / α = ( μ N λ N ) 1 / α . N-\mu_N=S_{1,\alpha}(\tau_N)\asymp\tau_N^{-1/\alpha}=\left(\frac{\mu_N}{\lambda_N}\right)^{1/\alpha}. N − μ N = S 1 , α ( τ N ) ≍ τ N − 1/ α = ( λ N μ N ) 1/ α .
Using λ N ≍ N − q \lambda_N\asymp N^{-q} λ N ≍ N − q and dividing by N N N gives
1 − μ N N ≍ N ( 1 + q − α ) / α ( μ N N ) 1 / α . (59) 1-\frac{\mu_N}{N}\asymp N^{(1+q-\alpha)/\alpha}\left(\frac{\mu_N}{N}\right)^{1/\alpha}. \tag*{(59)} 1 − N μ N ≍ N ( 1 + q − α ) / α ( N μ N ) 1/ α . (59)
If q < α − 1 q<\alpha-1 q < α − 1 , then N ( 1 + q − α ) / α → 0 N^{(1+q-\alpha)/\alpha}\to0 N ( 1 + q − α ) / α → 0 , so (59) gives μ N / N → 1 \mu_N/N\to1 μ N / N → 1 . Thus μ N ≍ N \mu_N\asymp N μ N ≍ N , and
N − μ N ≍ ( N N − q ) 1 / α = N ( 1 + q ) / α , τ N = λ N μ N ≍ N − ( 1 + q ) . N-\mu_N\asymp\left(\frac{N}{N^{-q}}\right)^{1/\alpha}=N^{(1+q)/\alpha},\qquad
\tau_N=\frac{\lambda_N}{\mu_N}\asymp N^{-(1+q)}. N − μ N ≍ ( N − q N ) 1/ α = N ( 1 + q ) / α , τ N = μ N λ N ≍ N − ( 1 + q ) .
If q = α − 1 q=\alpha-1 q = α − 1 , then (59) becomes
1 − μ N N ≍ ( μ N N ) 1 / α , 1-\frac{\mu_N}{N}\asymp\left(\frac{\mu_N}{N}\right)^{1/\alpha}, 1 − N μ N ≍ ( N μ N ) 1/ α ,
so μ N / N ≍ 1 \mu_N/N\asymp1 μ N / N ≍ 1 and 1 − μ N / N ≍ 1 1-\mu_N/N\asymp1 1 − μ N / N ≍ 1 . Hence μ N ≍ N \mu_N\asymp N μ N ≍ N , N − μ N ≍ N N-\mu_N\asymp N N − μ N ≍ N , and τ N ≍ N − α \tau_N\asymp N^{-\alpha} τ N ≍ N − α .
If q > α − 1 q>\alpha-1 q > α − 1 , then N ( 1 + q − α ) / α → ∞ N^{(1+q-\alpha)/\alpha}\to\infty N ( 1 + q − α ) / α → ∞ , so (59) gives μ N / N → 0 \mu_N/N\to0 μ N / N → 0 . Hence 1 − μ N / N ≍ 1 1-\mu_N/N\asymp1 1 − μ N / N ≍ 1 and
1 ≍ N ( 1 + q − α ) / α ( μ N N ) 1 / α , μ N N ≍ N − ( 1 + q − α ) . 1\asymp N^{(1+q-\alpha)/\alpha}\left(\frac{\mu_N}{N}\right)^{1/\alpha},\qquad
\frac{\mu_N}{N}\asymp N^{-(1+q-\alpha)}. 1 ≍ N ( 1 + q − α ) / α ( N μ N ) 1/ α , N μ N ≍ N − ( 1 + q − α ) .
Therefore μ N ≍ N α − q \mu_N\asymp N^{\alpha-q} μ N ≍ N α − q and τ N = λ N / μ N ≍ N − α \tau_N=\lambda_N/\mu_N\asymp N^{-\alpha} τ N = λ N / μ N ≍ N − α .
Combining the three regimes gives
τ N ≍ N − ( ( 1 + q ) ∧ α ) , μ N ≍ { N , q ≤ α − 1 , N α − q , q > α − 1 , \tau_N\asymp N^{-((1+q)\wedge\alpha)},\qquad
\mu_N\asymp
\begin{cases}
N, & q\le\alpha-1,\\
N^{\alpha-q}, & q>\alpha-1,
\end{cases} τ N ≍ N − (( 1 + q ) ∧ α ) , μ N ≍ { N , N α − q , q ≤ α − 1 , q > α − 1 ,
which proves (57) and (58) .
We now specialize Proposition 4 to the source and target problems. This gives the effective regularization scales that will be used throughout the remainder of the proof.
We apply Proposition 4 with ( N , α , q ) = ( n , b , γ λ t r ) (N,\alpha,q)=(n,b,\gamma_{\lambda_{\mathrm{tr}}}) ( N , α , q ) = ( n , b , γ λ tr ) for the source object:
τ t r ≍ n − γ τ t r , γ τ t r : = ( 1 + γ λ t r ) ∧ b . (60) \tau_{\mathrm{tr}}\asymp n^{-\gamma_{\tau_{\mathrm{tr}}}},\qquad
\gamma_{\tau_{\mathrm{tr}}}:=(1+\gamma_{\lambda_{\mathrm{tr}}})\wedge b. \tag*{(60)} τ tr ≍ n − γ τ tr , γ τ tr := ( 1 + γ λ tr ) ∧ b . (60)
Moreover,
μ t r ≍ { n , γ λ t r ≤ b − 1 , n b − γ λ t r , γ λ t r > b − 1. (61) \mu_{\mathrm{tr}} \asymp
\begin{cases}
n, & \gamma_{\lambda_{\mathrm{tr}}} \le b-1,\\
n^{b-\gamma_{\lambda_{\mathrm{tr}}}}, & \gamma_{\lambda_{\mathrm{tr}}} > b-1.
\end{cases} \tag*{(61)} μ tr ≍ { n , n b − γ λ tr , γ λ tr ≤ b − 1 , γ λ tr > b − 1. (61)
For the target objects, we use N = n s r ≍ n γ n N=n_{\mathrm{sr}}\asymp n^{\gamma_n} N = n sr ≍ n γ n , α = a \alpha=a α = a , and q = γ λ s r / γ n q=\gamma_{\lambda_{\mathrm{sr}}}/\gamma_n q = γ λ sr / γ n :
τ s r ≍ n − γ τ s r , γ τ s r : = ( γ λ s r + γ n ) ∧ ( a γ n ) . (62) \tau_{\mathrm{sr}}\asymp n^{-\gamma_{\tau_{\mathrm{sr}}}}, \qquad\gamma_{\tau_{\mathrm{sr}}}:=(\gamma_{\lambda_{\mathrm{sr}}}+\gamma_n)\wedge(a\gamma_n). \tag*{(62)} τ sr ≍ n − γ τ sr , γ τ sr := ( γ λ sr + γ n ) ∧ ( a γ n ) . (62)
Moreover,
μ s r ≍ { n γ n , γ λ s r ≤ ( a − 1 ) γ n , n a γ n − γ λ s r , γ λ s r > ( a − 1 ) γ n . (63) \mu_{\mathrm{sr}}\asymp
\begin{cases}
n^{\gamma_n}, & \gamma_{\lambda_{\mathrm{sr}}}\le(a-1)\gamma_n,\\
n^{a\gamma_n-\gamma_{\lambda_{\mathrm{sr}}}}, & \gamma_{\lambda_{\mathrm{sr}}}>(a-1)\gamma_n.
\end{cases} \tag*{(63)} μ sr ≍ { n γ n , n a γ n − γ λ sr , γ λ sr ≤ ( a − 1 ) γ n , γ λ sr > ( a − 1 ) γ n . (63)
At either threshold, both μ \mu μ and the corresponding gap N − μ N-\mu N − μ are of order N N N . Now for the next step, we define the corresponding effective cutoffs
K s r : = τ s r − 1 / a , K t r : = τ t r − 1 / b . (64) K_{\mathrm{sr}}:=\tau_{\mathrm{sr}}^{-1/a}, \qquad K_{\mathrm{tr}}:=\tau_{\mathrm{tr}}^{-1/b}. \tag*{(64)} K sr := τ sr − 1/ a , K tr := τ tr − 1/ b . (64)
Proposition 5 (Uniform denominator scaling). In the setting of Proposition 4 , we have
N − S 2 , α ( τ N ) ≍ N . (65) N-S_{2,\alpha}(\tau_N)\asymp N. \tag*{(65)} N − S 2 , α ( τ N ) ≍ N . (65)
Proof. By (56) ,
N − S 2 , α ( τ N ) = μ N + S 1 , α ( τ N ) − S 2 , α ( τ N ) . N-S_{2,\alpha}(\tau_N)=\mu_N+S_{1,\alpha}(\tau_N)-S_{2,\alpha}(\tau_N). N − S 2 , α ( τ N ) = μ N + S 1 , α ( τ N ) − S 2 , α ( τ N ) .
If q ≤ α − 1 q\le\alpha-1 q ≤ α − 1 , then μ N ≍ N \mu_N\asymp N μ N ≍ N . If q > α − 1 q>\alpha-1 q > α − 1 , then S 1 , α = N − μ N ≍ N S_{1,\alpha}=N-\mu_N\asymp N S 1 , α = N − μ N ≍ N , and Lemma 3 gives S 1 , α − S 2 , α ≍ S 1 , α ≍ N S_{1,\alpha}-S_{2,\alpha}\asymp S_{1,\alpha}\asymp N S 1 , α − S 2 , α ≍ S 1 , α ≍ N . The upper bound is immediate because S 2 , α ≥ 0 S_{2,\alpha}\ge0 S 2 , α ≥ 0 .
Recall the prefactors c s r c_{\mathrm{sr}} c sr and c t r c_{\mathrm{tr}} c tr in (9) . By Proposition 5,
n s r − Tr [ Σ s r 2 ( Σ s r + τ s r I ) − 2 ] ≍ n s r , n t r − Tr [ Σ t r 2 ( Σ t r + τ t r I ) − 2 ] ≍ n t r . n_{\mathrm{sr}}-\operatorname{Tr}\left[\Sigma_{\mathrm{sr}}^2(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I)^{-2}\right]\asymp n_{\mathrm{sr}}, \qquad
n_{\mathrm{tr}}-\operatorname{Tr}\left[\Sigma_{\mathrm{tr}}^2(\Sigma_{\mathrm{tr}}+\tau_{\mathrm{tr}}I)^{-2}\right]\asymp n_{\mathrm{tr}}. n sr − Tr [ Σ sr 2 ( Σ sr + τ sr I ) − 2 ] ≍ n sr , n tr − Tr [ Σ tr 2 ( Σ tr + τ tr I ) − 2 ] ≍ n tr .
Substituting this into the definitions of c s r c_{\mathrm{sr}} c sr and c t r c_{\mathrm{tr}} c tr gives
c s r ≍ n s r − 1 ≍ n − γ n , c t r ≍ τ t r 2 ≍ n − 2 γ τ t r . (66) c_{\mathrm{sr}}\asymp n_{\mathrm{sr}}^{-1}\asymp n^{-\gamma_n}, \qquad c_{\mathrm{tr}}\asymp\tau_{\mathrm{tr}}^2\asymp n^{-2\gamma_{\tau_{\mathrm{tr}}}}. \tag*{(66)} c sr ≍ n sr − 1 ≍ n − γ n , c tr ≍ τ tr 2 ≍ n − 2 γ τ tr . (66)
Now, with the effective regularization parameters and prefactors established, we determine the scaling of the two variance terms.
For V t r V_{\mathrm{tr}} V tr , recall that σ s r 2 ≍ σ t r 2 ≍ 1 \sigma_{\mathrm{sr}}^2\asymp\sigma_{\mathrm{tr}}^2\asymp1 σ sr 2 ≍ σ tr 2 ≍ 1 . Lemma 3, applied with α = b \alpha=b α = b , gives
Tr [ Σ t r 2 ( Σ t r + τ t r I ) − 2 ] = S 2 , b ( τ t r ) ≍ τ t r − 1 / b ≍ n γ τ t r / b . \operatorname{Tr}\left[\Sigma_{\mathrm{tr}}^2(\Sigma_{\mathrm{tr}}+\tau_{\mathrm{tr}}I)^{-2}\right]=S_{2,b}(\tau_{\mathrm{tr}})\asymp\tau_{\mathrm{tr}}^{-1/b}\asymp n^{\gamma_{\tau_{\mathrm{tr}}}/b}. Tr [ Σ tr 2 ( Σ tr + τ tr I ) − 2 ] = S 2 , b ( τ tr ) ≍ τ tr − 1/ b ≍ n γ τ tr / b .
Proposition 5, applied with N = n N=n N = n and α = b \alpha=b α = b , gives n − S 2 , b ( τ t r ) ≍ n n-S_{2,b}(\tau_{\mathrm{tr}})\asymp n n − S 2 , b ( τ tr ) ≍ n . Hence the fine-tuning variance satisfies
V ‾ t r ≍ S 2 , b ( τ t r ) n − S 2 , b ( τ t r ) ≍ n − 1 + γ τ t r / b . \overline{V}_{\mathrm{tr}}\asymp\frac{S_{2,b}(\tau_{\mathrm{tr}})}{n-S_{2,b}(\tau_{\mathrm{tr}})}\asymp n^{-1+\gamma_{\tau_{\mathrm{tr}}}/b}. V tr ≍ n − S 2 , b ( τ tr ) S 2 , b ( τ tr ) ≍ n − 1 + γ τ tr / b .
Finally, using γ τ t r = ( 1 + γ λ t r ) ∧ b \gamma_{\tau_{\mathrm{tr}}}=(1+\gamma_{\lambda_{\mathrm{tr}}})\wedge b γ τ tr = ( 1 + γ λ tr ) ∧ b gives
Γ V t r = 1 − γ τ t r b = 1 − ( 1 + γ λ t r ) ∧ b b . (67) \Gamma_{V_{\mathrm{tr}}}=1-\frac{\gamma_{\tau_{\mathrm{tr}}}}{b}
=1-\frac{(1+\gamma_{\lambda_{\mathrm{tr}}})\wedge b}{b}. \tag*{(67)} Γ V tr = 1 − b γ τ tr = 1 − b ( 1 + γ λ tr ) ∧ b . (67)
The fine-tuning variance depends only on the target spectrum. The pretraining variance and the second bias term also involve the interaction between the source and target spectra, through the cross-trace Tr ( W s r W t r ) \operatorname{Tr}(W_{\mathrm{sr}}W_{\mathrm{tr}}) Tr ( W sr W tr ) . We therefore determine its scaling next.
Set u : = γ τ s r a u:=\frac{\gamma_{\tau_{\mathrm{sr}}}}{a} u := a γ τ sr and v : = γ τ t r b v:=\frac{\gamma_{\tau_{\mathrm{tr}}}}{b} v := b γ τ tr . Recalling the matrices W s r W_{\mathrm{sr}} W sr and W t r W_{\mathrm{tr}} W tr from (8) and using ( Σ s r ) k , k ≍ k − a (\Sigma_{\mathrm{sr}})_{k,k}\asymp k^{-a} ( Σ sr ) k , k ≍ k − a , ( Σ t r ) k , k ≍ k − b (\Sigma_{\mathrm{tr}})_{k,k}\asymp k^{-b} ( Σ tr ) k , k ≍ k − b , we have
Tr ( W s r W t r ) ≍ ∑ k ≥ 1 k − a − b ( k − a + τ s r ) 2 ( k − b + τ t r ) 2 . \operatorname{Tr}(W_{\mathrm{sr}}W_{\mathrm{tr}})
\asymp\sum_{k\ge1}\frac{k^{-a-b}}{(k^{-a}+\tau_{\mathrm{sr}})^2(k^{-b}+\tau_{\mathrm{tr}})^2}. Tr ( W sr W tr ) ≍ k ≥ 1 ∑ ( k − a + τ sr ) 2 ( k − b + τ tr ) 2 k − a − b .
Applying Lemma 5 with μ = τ s r \mu=\tau_{\mathrm{sr}} μ = τ sr , ν = τ t r \nu=\tau_{\mathrm{tr}} ν = τ tr , γ 1 = γ 2 = 2 \gamma_1=\gamma_2=2 γ 1 = γ 2 = 2 , s = ( a + b ) / 2 s=(a+b)/2 s = ( a + b ) /2 , and δ a = δ b = 1 / 2 \delta_a=\delta_b=1/2 δ a = δ b = 1/2 . Here note that s + a δ a + b δ b = a + b s+a\delta_a+b\delta_b=a+b s + a δ a + b δ b = a + b , and the exponent in Lemma 5 is
η = 1 − ( a + b ) + 2 a + 2 b = a + b + 1. \eta=1-(a+b)+2a+2b=a+b+1. η = 1 − ( a + b ) + 2 a + 2 b = a + b + 1.
Since τ s r ≍ n − a u \tau_{\mathrm{sr}}\asymp n^{-au} τ sr ≍ n − a u and τ t r ≍ n − b v \tau_{\mathrm{tr}}\asymp n^{-bv} τ tr ≍ n − b v , we have τ s r 1 / a ≍ n − u \tau_{\mathrm{sr}}^{1/a}\asymp n^{-u} τ sr 1/ a ≍ n − u and τ t r 1 / b ≍ n − v \tau_{\mathrm{tr}}^{1/b}\asymp n^{-v} τ tr 1/ b ≍ n − v . So, the relative size of the two effective scales is determined by the ordering of u u u and v v v .
Suppose first that u ≤ v u \le v u ≤ v . If η ≤ 2 a \eta\le2a η ≤ 2 a , equivalently b − a + 1 ≤ 0 b-a+1 \le0 b − a + 1 ≤ 0 , case (1) of Lemma 5 gives
Tr ( W s r W t r ) = Θ ~ ( τ s r − η / a ) = Θ ~ ( n ( a + b + 1 ) u ) . \operatorname{Tr}(W_{sr}W_{tr})=\widetilde{\Theta}\left(\tau_{sr}^{-\eta/a}\right)=\widetilde{\Theta}\left(n^{(a+b+1)u}\right). Tr ( W sr W t r ) = Θ ( τ sr − η / a ) = Θ ( n ( a + b + 1 ) u ) .
If instead η > 2 a \eta>2a η > 2 a , equivalently b − a + 1 > 0 b-a+1>0 b − a + 1 > 0 , then
Tr ( W s r W t r ) = Θ ( τ s r − 2 τ t r − ( η − 2 a ) / b ) = Θ ( n ( a + b + 1 ) u + ( b − a + 1 ) ( v − u ) ) . \operatorname{Tr}(W_{sr}W_{tr})=\Theta\left(\tau_{sr}^{-2}\tau_{tr}^{-(\eta-2a)/b}\right)=\Theta\left(n^{(a+b+1)u+(b-a+1)(v-u)}\right). Tr ( W sr W t r ) = Θ ( τ sr − 2 τ t r − ( η − 2 a ) / b ) = Θ ( n ( a + b + 1 ) u + ( b − a + 1 ) ( v − u ) ) .
where we used η − 2 a = b − a + 1 \eta-2a=b-a+1 η − 2 a = b − a + 1 . Therefore, for u ≤ v u\le v u ≤ v ,
− Γ T = ( a + b + 1 ) u + ( b − a + 1 ) + ( v − u ) . -\Gamma_T=(a+b+1)u+(b-a+1)+(v-u). − Γ T = ( a + b + 1 ) u + ( b − a + 1 ) + ( v − u ) .
The case v ≤ u v\le u v ≤ u follows symmetrically from case (2) of Lemma 5. Since η − 2 b = a − b + 1 \eta-2b=a-b+1 η − 2 b = a − b + 1 ,
− Γ T = ( a + b + 1 ) v + ( a − b + 1 ) + ( u − v ) . -\Gamma_T=(a+b+1)v+(a-b+1)+(u-v). − Γ T = ( a + b + 1 ) v + ( a − b + 1 ) + ( u − v ) .
Combining the two cases gives
Tr ( W s r W t r ) = Θ ~ ( n − Γ T ) , (68) \operatorname{Tr}(W_{sr}W_{tr})=\widetilde{\Theta}\left(n^{-\Gamma_T}\right), \tag*{(68)} Tr ( W sr W t r ) = Θ ( n − Γ T ) , (68)
where
Γ T = { − ( a + b + 1 ) u − ( δ + 1 ) + ( v − u ) , u ≤ v , − ( a + b + 1 ) v − ( 1 − δ ) + ( u − v ) , v ≤ u . (69) \Gamma_T=
\begin{cases}
-(a+b+1)u-(\delta+1)+(v-u), & u\le v,\\
-(a+b+1)v-(1-\delta)+(u-v), & v\le u.
\end{cases} \tag*{(69)} Γ T = { − ( a + b + 1 ) u − ( δ + 1 ) + ( v − u ) , − ( a + b + 1 ) v − ( 1 − δ ) + ( u − v ) , u ≤ v , v ≤ u . (69)
For V ‾ s r \overline{\mathcal{V}}_{sr} V sr , combining (66) with (68) gives
V ‾ s r = σ s r 2 c s r c t r Tr ( W s r W t r ) = Θ ~ ( n − γ n − 2 γ τ t r − Γ T ) , (70) \overline{\mathcal{V}}_{sr}=\sigma_{sr}^{2}c_{sr}c_{tr}\operatorname{Tr}(W_{sr}W_{tr})=\widetilde{\Theta}\left(n^{-\gamma_n-2\gamma\tau_{tr}-\Gamma_T}\right), \tag*{(70)} V sr = σ sr 2 c sr c t r Tr ( W sr W t r ) = Θ ( n − γ n − 2 γ τ t r − Γ T ) , (70)
so
Γ V s r = γ n + 2 γ τ t r + Γ T . (71) \Gamma_{\mathcal{V}_{sr}}=\gamma_n+2\gamma\tau_{tr}+\Gamma_T. \tag*{(71)} Γ V sr = γ n + 2 γ τ t r + Γ T . (71)
We now start with the bias terms. The cross-trace also appears in B ‾ 2 \overline{B}_2 B 2 , while the remaining bias scaling is determined by the source quadratic form β T W s r β \beta^{\mathsf T}W_{sr}\beta β T W sr β and the shrinkage contribution in B ‾ 1 \overline{B}_1 B 1 . Recall that
( β s r ) k = ( β t r ) k = : ( β ) k , ( β ) k 2 ≍ k b − 1 − 2 b r t r . (72) (\beta_{sr})_k=(\beta_{tr})_k=:(\beta)_k,\qquad(\beta)_k^2\asymp k^{b-1-2b r_{tr}}. \tag*{(72)} ( β sr ) k = ( β t r ) k =: ( β ) k , ( β ) k 2 ≍ k b − 1 − 2 b r t r . (72)
Under this signal scaling, we first determine the source quadratic form appearing in B ‾ 2 \overline{B}_2 B 2 , and then the shrinkage contribution appearing in B ‾ 1 \overline{B}_1 B 1 . Recall that in this case
a − 1 − 2 a r s r = b − 1 − 2 b r t r . (73) a-1-2a r_{sr}=b-1-2b r_{tr}. \tag*{(73)} a − 1 − 2 a r sr = b − 1 − 2 b r t r . (73)
We now establish the scaling law for the bias terms. Define
B s h : = ∑ k ≥ 1 ( Σ t r ) k , k ( ( Σ t r ) k , k + τ t r ) 2 ( τ s r ( Σ s r ) k , k + τ s r ) 2 ( β ) k 2 , (74) B_{\mathrm{sh}}:=\sum_{k\ge1}\frac{(\Sigma_{tr})_{k,k}}{\left((\Sigma_{tr})_{k,k}+\tau_{tr}\right)^2}\left(\frac{\tau_{sr}}{(\Sigma_{sr})_{k,k}+\tau_{sr}}\right)^2(\beta)_k^2, \tag*{(74)} B sh := k ≥ 1 ∑ ( ( Σ t r ) k , k + τ t r ) 2 ( Σ t r ) k , k ( ( Σ sr ) k , k + τ sr τ sr ) 2 ( β ) k 2 , (74)
since under (17) and (20) the two covariances are both diagonal, (13) factors as B ‾ 1 = c t r B s h \overline{B}_1=c_{tr}B_{\mathrm{sh}} B 1 = c t r B sh . Further, by (14) , we remain to characterize the scaling of β T W s r β \beta^{\mathsf T}W_{sr}\beta β T W sr β . We compute these scalings in the Proposition below.
Proposition 6. Under the same assumptions in Theorem 3,
β T W s r β = Θ ~ ( n − Γ S ) , B s h = Θ ~ ( n − Γ s h ) . (75) \beta^{\mathsf T}W_{sr}\beta=\widetilde{\Theta}\left(n^{-\Gamma_S}\right),\qquad B_{\mathrm{sh}}=\widetilde{\Theta}\left(n^{-\Gamma_{\mathrm{sh}}}\right). \tag*{(75)} β T W sr β = Θ ( n − Γ S ) , B sh = Θ ( n − Γ sh ) . (75) where
Γ S = − ( a + b − 2 b r t r ) + u , (76) \Gamma_S=-(a+b-2b r_{tr})+u, \tag*{(76)} Γ S = − ( a + b − 2 b r t r ) + u , (76) and
Γ s h = { 2 a u − 2 b u ( 1 + a b − r t r ) + − 2 b ( v − u ) ( 1 − r t r ) + , u ≤ v , 2 a u − 2 b v ( 1 + a b − r t r ) + − 2 b ( u − v ) ( a b − r t r ) + , v ≤ u . (77) \Gamma_{\mathrm{sh}}=
\begin{cases}
2au-2bu\left(1+\frac{a}{b}-r_{tr}\right)_{+}-2b(v-u)(1-r_{tr})_{+}, & u\le v,\\
2au-2bv\left(1+\frac{a}{b}-r_{tr}\right)_{+}-2b(u-v)\left(\frac{a}{b}-r_{tr}\right)_{+}, & v\le u.
\end{cases} \tag*{(77)} Γ sh = { 2 a u − 2 b u ( 1 + b a − r t r ) + − 2 b ( v − u ) ( 1 − r t r ) + , 2 a u − 2 b v ( 1 + b a − r t r ) + − 2 b ( u − v ) ( b a − r t r ) + , u ≤ v , v ≤ u . (77)
Proof. We already established cross-scaling. Now for the other two, we establish them separately. Using ( Σ s r ) k , k ≍ k − a (\Sigma_{sr})_{k,k}\asymp k^{-a} ( Σ sr ) k , k ≍ k − a and (72) ,
( Σ s r ) k , k ( β ) k 2 ( ( Σ s r ) k , k + τ s r ) 2 ≳ k a + b − 2 b r t r − 1 ( 1 + τ s r k a ) 2 . \frac{(\Sigma_{sr})_{k,k}(\beta)_k^2}{\left((\Sigma_{sr})_{k,k}+\tau_{sr}\right)^2}\gtrsim\frac{k^{a+b-2b r_{tr}-1}}{\left(1+\tau_{sr}k^a\right)^2}. ( ( Σ sr ) k , k + τ sr ) 2 ( Σ sr ) k , k ( β ) k 2 ≳ ( 1 + τ sr k a ) 2 k a + b − 2 b r t r − 1 .
Hence
β T W s r β ≲ ∑ k ≥ 1 k a + b − 2 b r t r − 1 ( 1 + τ s r k a ) 2 . (78) \beta^{\mathsf T}W_{sr}\beta\lesssim\sum_{k\geq1}\frac{k^{a+b-2br_{tr}-1}}{(1+\tau_{sr}k^a)^2}. \tag*{(78)} β T W sr β ≲ k ≥ 1 ∑ ( 1 + τ sr k a ) 2 k a + b − 2 b r t r − 1 . (78)
If a + b − 2 b r t r < 2 a a+b-2br_{tr}<2a a + b − 2 b r t r < 2 a , Lemma 4 with ( α , γ , η ) = ( a , 2 , a + b − 2 b r t r ) (\alpha,\gamma,\eta)=(a,2,a+b-2br_{tr}) ( α , γ , η ) = ( a , 2 , a + b − 2 b r t r ) gives
β T W s r β ≍ { 1 , a + b − 2 b r t r < 0 , log ( 1 / τ s r ) , a + b − 2 b r t r = 0 , τ s r − ( a + b − 2 b r t r ) / a , 0 < a + b − 2 b r t r < 2 a . (79) \beta^{\mathsf T}W_{sr}\beta\asymp
\begin{cases}
1, & a+b-2br_{tr}<0,\\
\log(1/\tau_{sr}), & a+b-2br_{tr}=0,\\
\tau_{sr}^{-(a+b-2br_{tr})/a}, & 0<a+b-2br_{tr}<2a.
\end{cases} \tag*{(79)} β T W sr β ≍ ⎩ ⎨ ⎧ 1 , log ( 1/ τ sr ) , τ sr − ( a + b − 2 b r t r ) / a , a + b − 2 b r t r < 0 , a + b − 2 b r t r = 0 , 0 < a + b − 2 b r t r < 2 a . (79)
If a + b − 2 b r t r ≥ 2 a a+b-2br_{tr}\geq2a a + b − 2 b r t r ≥ 2 a , then for large k k k ,
k a + b − 2 b r t r − 1 ( 1 + τ s r k a ) 2 ≲ τ s r − 2 k b − a − 2 b r t r − 1 . \frac{k^{a+b-2br_{tr}-1}}{(1+\tau_{sr}k^a)^2}\lesssim\tau_{sr}^{-2}k^{b-a-2br_{tr}-1}. ( 1 + τ sr k a ) 2 k a + b − 2 b r t r − 1 ≲ τ sr − 2 k b − a − 2 b r t r − 1 .
The tail is summable if and only if b − a − 2 b r t r < 0 b-a-2br_{tr}<0 b − a − 2 b r t r < 0 . Under the exact alignment condition (73) ,
a − b + 2 b r t r = 2 a r s r > 0 , a-b+2br_{tr}=2ar_{sr}>0, a − b + 2 b r t r = 2 a r sr > 0 ,
since r s r > 0 r_{sr}>0 r sr > 0 . Hence, the quadratic form is always finite under our assumptions.
Using τ s r ≍ n − a u \tau_{sr}\asymp n^{-au} τ sr ≍ n − a u , we therefore obtain
β T W s r β = Θ ~ ( n ( a + b − 2 b r t r ) + u ) . \beta^{\mathsf T}W_{sr}\beta=\widetilde{\Theta}\left(n^{(a+b-2br_{tr})+u}\right). β T W sr β = Θ ( n ( a + b − 2 b r t r ) + u ) .
Equivalently,
β T W s r β = Θ ~ ( n − Γ S ) , Γ S : = − ( a + b − 2 b r t r ) + u , (80) \beta^{\mathsf T}W_{sr}\beta=\widetilde{\Theta}(n^{-\Gamma_S}),\qquad\Gamma_S:=-(a+b-2br_{tr})+u, \tag*{(80)} β T W sr β = Θ ( n − Γ S ) , Γ S := − ( a + b − 2 b r t r ) + u , (80)
where the logarithmic boundary case is absorbed by Θ ~ \widetilde{\Theta} Θ .
Recall B s h B_{sh} B s h in (74) . Under (72) ,
B s h ≍ τ s r 2 ∑ k ≥ 1 k 2 a + 2 b − 2 b r t r − 1 ( 1 + τ s r k a ) 2 ( 1 + τ t r k b ) 2 . (81) B_{sh}\asymp\tau_{sr}^2\sum_{k\geq1}\frac{k^{2a+2b-2br_{tr}-1}}{(1+\tau_{sr}k^a)^2(1+\tau_{tr}k^b)^2}. \tag*{(81)} B s h ≍ τ sr 2 k ≥ 1 ∑ ( 1 + τ sr k a ) 2 ( 1 + τ t r k b ) 2 k 2 a + 2 b − 2 b r t r − 1 . (81)
Rewriting ( 1 + τ s r k a ) 2 = k 2 a ( k − a + τ s r ) 2 (1+\tau_{sr}k^a)^2=k^{2a}(k^{-a}+\tau_{sr})^2 ( 1 + τ sr k a ) 2 = k 2 a ( k − a + τ sr ) 2 and ( 1 + τ t r k b ) 2 = k 2 b ( k − b + τ t r ) 2 (1+\tau_{tr}k^b)^2=k^{2b}(k^{-b}+\tau_{tr})^2 ( 1 + τ t r k b ) 2 = k 2 b ( k − b + τ t r ) 2 , we obtain
B s h ≍ τ s r 2 ∑ k ≥ 1 k − 2 b r t r − 1 ( k − a + τ s r ) 2 ( k − b + τ t r ) 2 . (82) B_{sh}\asymp\tau_{sr}^2\sum_{k\geq1}\frac{k^{-2br_{tr}-1}}{(k^{-a}+\tau_{sr})^2(k^{-b}+\tau_{tr})^2}. \tag*{(82)} B s h ≍ τ sr 2 k ≥ 1 ∑ ( k − a + τ sr ) 2 ( k − b + τ t r ) 2 k − 2 b r t r − 1 . (82)
Lemma 5 applies with μ = τ s r \mu=\tau_{sr} μ = τ sr , ν = τ t r \nu=\tau_{tr} ν = τ t r , γ 1 = γ 2 = 2 \gamma_1=\gamma_2=2 γ 1 = γ 2 = 2 , s = 1 s=1 s = 1 , δ a = b r t r / a \delta_a=br_{tr}/a δ a = b r t r / a , and δ b = r t r \delta_b=r_{tr} δ b = r t r , so its relevant exponent is 2 a + 2 b − 2 b r t r 2a+2b-2br_{tr} 2 a + 2 b − 2 b r t r .
Recall the effective cutoffs K s r K_{sr} K sr and K t r K_{tr} K t r from (64) . Suppose first that K s r ≤ K t r K_{sr}\leq K_{tr} K sr ≤ K t r , equivalently τ s r 1 / a ≥ τ t r 1 / b \tau_{sr}^{1/a}\geq\tau_{tr}^{1/b} τ sr 1/ a ≥ τ t r 1/ b , so case (1) of Lemma 5 applies.
If 2 a + 2 b − 2 b r t r ≤ 0 2a+2b-2br_{tr}\leq0 2 a + 2 b − 2 b r t r ≤ 0 , equivalently r t r ≥ 1 + a / b r_{tr}\geq1+a/b r t r ≥ 1 + a / b , the sum in (82) is Θ ~ ( 1 ) \widetilde{\Theta}(1) Θ ( 1 ) , hence B s h = Θ ~ ( τ s r 2 ) B_{sh}=\widetilde{\Theta}(\tau_{sr}^2) B s h = Θ ( τ sr 2 ) .
If 0 < 2 a + 2 b − 2 b r t r ≤ 2 a 0<2a+2b-2br_{tr}\leq2a 0 < 2 a + 2 b − 2 b r t r ≤ 2 a , equivalently 1 ≤ r t r < 1 + a / b 1\leq r_{tr}<1+a/b 1 ≤ r t r < 1 + a / b , then
B s h = Θ ~ ( τ s r 2 − ( 2 a + 2 b − 2 b r t r ) / a ) = Θ ~ ( τ s r 2 b a ( r t r − 1 ) ) . B_{sh}=\widetilde{\Theta}\left(\tau_{sr}^{2-(2a+2b-2br_{tr})/a}\right)=\widetilde{\Theta}\left(\tau_{sr}^{\frac{2b}{a}(r_{tr}-1)}\right). B s h = Θ ( τ sr 2 − ( 2 a + 2 b − 2 b r t r ) / a ) = Θ ( τ sr a 2 b ( r t r − 1 ) ) .
If 2 a + 2 b − 2 b r t r > 2 a 2a+2b-2br_{tr}>2a 2 a + 2 b − 2 b r t r > 2 a , equivalently r t r < 1 r_{tr}<1 r t r < 1 , then
B s h = Θ ~ ( τ t r − ( 2 b − 2 b r t r ) / b ) = Θ ~ ( τ t r 2 ( r t r − 1 ) ) . B_{sh}=\widetilde{\Theta}\left(\tau_{tr}^{-(2b-2br_{tr})/b}\right)=\widetilde{\Theta}\left(\tau_{tr}^{2(r_{tr}-1)}\right). B s h = Θ ( τ t r − ( 2 b − 2 b r t r ) / b ) = Θ ( τ t r 2 ( r t r − 1 ) ) .
Thus, if K s r ≤ K t r K_{sr}\leq K_{tr} K sr ≤ K t r ,
B s h = { Θ ~ ( τ s r 2 ) , r t r ≥ 1 + a b , Θ ~ ( τ s r 2 b a ( r t r − 1 ) ) , 1 ≤ r t r < 1 + a b , Θ ~ ( τ t r 2 ( r t r − 1 ) ) , r t r < 1. (83) B_{sh}=
\begin{cases}
\widetilde{\Theta}(\tau_{sr}^2), & r_{tr}\geq1+\dfrac{a}{b},\\
\widetilde{\Theta}\left(\tau_{sr}^{\frac{2b}{a}(r_{tr}-1)}\right), & 1\leq r_{tr}<1+\dfrac{a}{b},\\
\widetilde{\Theta}\left(\tau_{tr}^{2(r_{tr}-1)}\right), & r_{tr}<1.
\end{cases} \tag*{(83)} B s h = ⎩ ⎨ ⎧ Θ ( τ sr 2 ) , Θ ( τ sr a 2 b ( r t r − 1 ) ) , Θ ( τ t r 2 ( r t r − 1 ) ) , r t r ≥ 1 + b a , 1 ≤ r t r < 1 + b a , r t r < 1. (83)
Now suppose K t r ≤ K s r K_{tr}\leq K_{sr} K t r ≤ K sr , equivalently τ s r 1 / a ≤ τ t r 1 / b \tau_{sr}^{1/a}\leq\tau_{tr}^{1/b} τ sr 1/ a ≤ τ t r 1/ b , so case (2) of Lemma 5 applies.
If 2 a + 2 b − 2 b r t r ≤ 0 2a+2b-2br_{tr}\leq0 2 a + 2 b − 2 b r t r ≤ 0 , equivalently r t r ≥ 1 + a / b r_{tr}\geq1+a/b r t r ≥ 1 + a / b , we again obtain B s h = Θ ~ ( τ s r 2 ) B_{sh}=\widetilde{\Theta}(\tau_{sr}^2) B s h = Θ ( τ sr 2 ) .
If 0 < 2 a + 2 b − 2 b r t r ≤ 2 b 0<2a+2b-2br_{tr}\leq2b 0 < 2 a + 2 b − 2 b r t r ≤ 2 b , equivalently a / b ≤ r t r < 1 + a / b a/b\leq r_{tr}<1+a/b a / b ≤ r t r < 1 + a / b , then
B s h = Θ ~ ( τ s r 2 τ t r − ( 2 a + 2 b − 2 b r t r ) / b ) = Θ ~ ( τ s r 2 τ t r − 2 ( 1 + a / b − r t r ) ) . B_{sh}=\widetilde{\Theta}\left(\tau_{sr}^2\tau_{tr}^{-(2a+2b-2br_{tr})/b}\right)=\widetilde{\Theta}\left(\tau_{sr}^2\tau_{tr}^{-2(1+a/b-r_{tr})}\right). B s h = Θ ( τ sr 2 τ t r − ( 2 a + 2 b − 2 b r t r ) / b ) = Θ ( τ sr 2 τ t r − 2 ( 1 + a / b − r t r ) ) .
Finally, if 2 a + 2 b − 2 b r t r > 2 b 2a+2b-2br_{\mathrm{tr}}>2b 2 a + 2 b − 2 b r tr > 2 b , equivalently r t r < a / b r_{\mathrm{tr}}<a/b r tr < a / b , then
B s h = Θ ( τ s r 2 − ( 2 a − 2 b r t r ) / a τ t r − 2 ) = Θ ( τ s r 2 b r t r / a τ t r − 2 ) . B_{\mathrm{sh}}=\Theta\left(\tau_{\mathrm{sr}}^{2-(2a-2br_{\mathrm{tr}})/a}\tau_{\mathrm{tr}}^{-2}\right)=\Theta\left(\tau_{\mathrm{sr}}^{2br_{\mathrm{tr}}/a}\tau_{\mathrm{tr}}^{-2}\right). B sh = Θ ( τ sr 2 − ( 2 a − 2 b r tr ) / a τ tr − 2 ) = Θ ( τ sr 2 b r tr / a τ tr − 2 ) .
Thus, if K t r ≤ K s r K_{\mathrm{tr}}\le K_{\mathrm{sr}} K tr ≤ K sr ,
B s h = { Θ ~ ( τ s r 2 ) , r t r ≥ 1 + a b , Θ ~ ( τ s r − 2 ( 1 + a / b − r t r ) ) , a b ≤ r t r < 1 + a b , Θ ( τ s r 2 b r t r / a τ t r − 2 ) , r t r < a b . (84) B_{\mathrm{sh}}=
\begin{cases}
\widetilde{\Theta}\left(\tau_{\mathrm{sr}}^2\right), & r_{\mathrm{tr}}\ge1+\frac{a}{b},\\
\widetilde{\Theta}\left(\tau_{\mathrm{sr}}^{-2(1+a/b-r_{\mathrm{tr}})}\right), & \frac{a}{b}\le r_{\mathrm{tr}}<1+\frac{a}{b},\\
\Theta\left(\tau_{\mathrm{sr}}^{2br_{\mathrm{tr}}/a}\tau_{\mathrm{tr}}^{-2}\right), & r_{\mathrm{tr}}<\frac{a}{b}.
\end{cases} \tag*{(84)} B sh = ⎩ ⎨ ⎧ Θ ( τ sr 2 ) , Θ ( τ sr − 2 ( 1 + a / b − r tr ) ) , Θ ( τ sr 2 b r tr / a τ tr − 2 ) , r tr ≥ 1 + b a , b a ≤ r tr < 1 + b a , r tr < b a . (84)
At K s r = K t r K_{\mathrm{sr}}=K_{\mathrm{tr}} K sr = K tr , either representation gives the same order.
Now suppose
τ s r ≍ n − a u , τ t r ≍ n − b v . \tau_{\mathrm{sr}}\asymp n^{-au},\qquad\tau_{\mathrm{tr}}\asymp n^{-bv}. τ sr ≍ n − a u , τ tr ≍ n − b v .
Writing
B s h = Θ ~ ( n − Γ s h ) , B_{\mathrm{sh}}=\widetilde{\Theta}\left(n^{-\Gamma_{\mathrm{sh}}}\right), B sh = Θ ( n − Γ sh ) ,
we can directly read off the shrinkage exponent.
If u ≤ v u\le v u ≤ v , equivalently K s r ≤ K t r K_{\mathrm{sr}}\le K_{\mathrm{tr}} K sr ≤ K tr , then
Γ s h = { 2 a u , r t r ≥ 1 + a b , 2 b u ( r t r − 1 ) , 1 ≤ r t r < 1 + a b , 2 b v ( r t r − 1 ) , r t r < 1. (85) \Gamma_{\mathrm{sh}}=
\begin{cases}
2au, & r_{\mathrm{tr}}\ge1+\frac{a}{b},\\
2b u(r_{\mathrm{tr}}-1), & 1\le r_{\mathrm{tr}}<1+\frac{a}{b},\\
2bv(r_{\mathrm{tr}}-1), & r_{\mathrm{tr}}<1.
\end{cases} \tag*{(85)} Γ sh = ⎩ ⎨ ⎧ 2 a u , 2 b u ( r tr − 1 ) , 2 b v ( r tr − 1 ) , r tr ≥ 1 + b a , 1 ≤ r tr < 1 + b a , r tr < 1. (85)
Equivalently,
Γ s h = 2 a u − 2 b u ( 1 + a b − r t r ) + − 2 b ( v − u ) ( 1 − r t r ) + . (86) \Gamma_{\mathrm{sh}}=2au-2bu\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right)_{+}-2b(v-u)(1-r_{\mathrm{tr}})_{+}. \tag*{(86)} Γ sh = 2 a u − 2 b u ( 1 + b a − r tr ) + − 2 b ( v − u ) ( 1 − r tr ) + . (86)
If v ≤ u v\le u v ≤ u , equivalently K t r ≤ K s r K_{\mathrm{tr}}\le K_{\mathrm{sr}} K tr ≤ K sr , then
Γ s h = { 2 a u , r t r ≥ 1 + a b , 2 a u − 2 b v ( 1 + a b − r t r ) , a b ≤ r t r < 1 + a b , 2 b r t r u − 2 b v , r t r < a b . (87) \Gamma_{\mathrm{sh}}=
\begin{cases}
2au, & r_{\mathrm{tr}}\ge1+\frac{a}{b},\\
2au-2bv\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right), & \frac{a}{b}\le r_{\mathrm{tr}}<1+\frac{a}{b},\\
2br_{\mathrm{tr}}u-2bv, & r_{\mathrm{tr}}<\frac{a}{b}.
\end{cases} \tag*{(87)} Γ sh = ⎩ ⎨ ⎧ 2 a u , 2 a u − 2 b v ( 1 + b a − r tr ) , 2 b r tr u − 2 b v , r tr ≥ 1 + b a , b a ≤ r tr < 1 + b a , r tr < b a . (87)
Equivalently,
Γ s h = 2 a u − 2 b v ( 1 + a b − r t r ) + − 2 b ( u − v ) ( a b − r t r ) + . (88) \Gamma_{\mathrm{sh}}=2au-2bv\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right)_{+}-2b(u-v)\left(\frac{a}{b}-r_{\mathrm{tr}}\right)_{+}. \tag*{(88)} Γ sh = 2 a u − 2 b v ( 1 + b a − r tr ) + − 2 b ( u − v ) ( b a − r tr ) + . (88)
When u = v u=v u = v , both expressions reduce to
Γ s h = 2 a u − 2 b u ( 1 + a b − r t r ) + , \Gamma_{\mathrm{sh}}=2au-2bu\left(1+\frac{a}{b}-r_{\mathrm{tr}}\right)_{+}, Γ sh = 2 a u − 2 b u ( 1 + b a − r tr ) + ,
so the two cutoff orderings agree. □
Proof of Theorem 3. For the first bias term, the resolvent identity
Σ s r ( Σ s r + τ s r I ) − 1 − I = − τ s r ( Σ s r + τ s r I ) − 1 \Sigma_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I)^{-1}-I=-\tau_{\mathrm{sr}}(\Sigma_{\mathrm{sr}}+\tau_{\mathrm{sr}}I)^{-1} Σ sr ( Σ sr + τ sr I ) − 1 − I = − τ sr ( Σ sr + τ sr I ) − 1
gives B ‾ 1 = c t r B s h \overline{B}_1=c_{\mathrm{tr}}B_{\mathrm{sh}} B 1 = c tr B sh . Since c t r ≍ n − 2 γ t r c_{\mathrm{tr}}\asymp n^{-2\gamma_{\mathrm{tr}}} c tr ≍ n − 2 γ tr and B s h = Θ ~ ( n − Γ s h ) B_{\mathrm{sh}}=\widetilde{\Theta}(n^{-\Gamma_{\mathrm{sh}}}) B sh = Θ ( n − Γ sh ) ,
B ‾ 1 = Θ ~ ( n − Γ B 1 ) , Γ B 1 = 2 γ t r + Γ s h . (89) \overline{B}_1=\widetilde{\Theta}\left(n^{-\Gamma_{B_1}}\right),\qquad\Gamma_{B_1}=2\gamma_{\mathrm{tr}}+\Gamma_{\mathrm{sh}}. \tag*{(89)} B 1 = Θ ( n − Γ B 1 ) , Γ B 1 = 2 γ tr + Γ sh . (89)
For the second bias term,
B ‾ 2 = c t r c s r τ s r 2 Tr ( W t r W s r ) β T W s r β . \overline{B}_2=c_{\mathrm{tr}}c_{\mathrm{sr}}\tau_{\mathrm{sr}}^2\operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})\beta^{\mathsf T}W_{\mathrm{sr}}\beta. B 2 = c tr c sr τ sr 2 Tr ( W tr W sr ) β T W sr β .
Using c t r ≍ n − 2 γ t r c_{\mathrm{tr}}\asymp n^{-2\gamma_{\mathrm{tr}}} c tr ≍ n − 2 γ tr , c s r ≍ n − γ n c_{\mathrm{sr}}\asymp n^{-\gamma_n} c sr ≍ n − γ n , τ s r 2 ≍ n − 2 a u \tau_{\mathrm{sr}}^2\asymp n^{-2au} τ sr 2 ≍ n − 2 a u , Tr ( W t r W s r ) = Θ ~ ( n − Γ T ) \operatorname{Tr}(W_{\mathrm{tr}}W_{\mathrm{sr}})=\widetilde{\Theta}(n^{-\Gamma_T}) Tr ( W tr W sr ) = Θ ( n − Γ T ) , and β T W s r β = Θ ~ ( n − Γ S ) \beta^{\mathsf T}W_{\mathrm{sr}}\beta=\widetilde{\Theta}(n^{-\Gamma_S}) β T W sr β = Θ ( n − Γ S ) gives
B ‾ 2 = Θ ~ ( n − Γ B 2 ) , Γ B 2 = 2 γ t r + γ n + 2 a u + Γ T + Γ S . (90) \overline{B}_2=\widetilde{\Theta}\left(n^{-\Gamma_{B_2}}\right),\qquad\Gamma_{B_2}=2\gamma_{\mathrm{tr}}+\gamma_n+2au+\Gamma_T+\Gamma_S. \tag*{(90)} B 2 = Θ ( n − Γ B 2 ) , Γ B 2 = 2 γ tr + γ n + 2 a u + Γ T + Γ S . (90)
Hence
B ‾ = Θ ~ ( n − Γ B ) , Γ B = min { Γ B 1 , Γ B 2 } . (91) \overline{B}=\widetilde{\Theta}\left(n^{-\Gamma_B}\right),\qquad\Gamma_B=\min\{\Gamma_{B_1},\Gamma_{B_2}\}. \tag*{(91)} B = Θ ( n − Γ B ) , Γ B = min { Γ B 1 , Γ B 2 } . (91)
For the variance terms, (67) gives
V ‾ t r = Θ ~ ( n − Γ V t r ) , Γ V t r = 1 − v , \overline{V}_{\mathrm{tr}}=\widetilde{\Theta}\left(n^{-\Gamma_{V_{\mathrm{tr}}}}\right),\qquad\Gamma_{V_{\mathrm{tr}}}=1-v, V tr = Θ ( n − Γ V tr ) , Γ V tr = 1 − v ,
while (71) gives
V ‾ s r = Θ ~ ( n − Γ V s r ) , Γ V s r = γ n + 2 b v + Γ T . \overline{V}_{\mathrm{sr}}=\widetilde{\Theta}\left(n^{-\Gamma_{V_{\mathrm{sr}}}}\right),\qquad\Gamma_{V_{\mathrm{sr}}}=\gamma_n+2bv+\Gamma_T. V sr = Θ ( n − Γ V sr ) , Γ V sr = γ n + 2 b v + Γ T .
□
Technical lemmas
Lemma 4. Let α , γ > 0 \alpha,\gamma>0 α , γ > 0 , η < α γ \eta<\alpha\gamma η < α γ , and
S α , γ , η ( λ ) : = ∑ k ≥ 1 k η − 1 ( 1 + λ k α ) γ . S_{\alpha,\gamma,\eta}(\lambda):=\sum_{k\geq1}\frac{k^{\eta-1}}{(1+\lambda k^\alpha)^\gamma}. S α , γ , η ( λ ) := k ≥ 1 ∑ ( 1 + λ k α ) γ k η − 1 .
As λ → 0 + \lambda\to0^+ λ → 0 + ,
S α , γ , η ( λ ) ≲ { 1 , η < 0 , log ( 1 / λ ) , η = 0 , λ − η / α , 0 < η < α γ . (92) S_{\alpha,\gamma,\eta}(\lambda)\lesssim
\begin{cases}
1, & \eta<0,\\
\log(1/\lambda), & \eta=0,\\
\lambda^{-\eta/\alpha}, & 0<\eta<\alpha\gamma.
\end{cases} \tag*{(92)} S α , γ , η ( λ ) ≲ ⎩ ⎨ ⎧ 1 , log ( 1/ λ ) , λ − η / α , η < 0 , η = 0 , 0 < η < α γ . (92)
Proof. Set K = λ − 1 / α K=\lambda^{-1/\alpha} K = λ − 1/ α . For k ≤ K k\leq K k ≤ K , the denominator is ≍ 1 \asymp1 ≍ 1 , so the contribution is ≍ ∑ k ≤ K k η − 1 \asymp\sum_{k\leq K}k^{\eta-1} ≍ ∑ k ≤ K k η − 1 , which is respectively 1 1 1 , log K \log K log K , or K η K^\eta K η . For k > K k>K k > K ,
∑ k > K k η − 1 ( 1 + λ k α ) γ ≲ λ − γ ∑ k > K k η − α γ − 1 ≲ K η , \sum_{k>K}\frac{k^{\eta-1}}{(1+\lambda k^\alpha)^\gamma}\lesssim\lambda^{-\gamma}\sum_{k>K}k^{\eta-\alpha\gamma-1}\lesssim K^\eta, k > K ∑ ( 1 + λ k α ) γ k η − 1 ≲ λ − γ k > K ∑ k η − α γ − 1 ≲ K η ,
where η < α γ \eta<\alpha\gamma η < α γ ensures summability. Substituting K = λ − 1 / α K=\lambda^{-1/\alpha} K = λ − 1/ α proves the claim. □ \square □
Lemma 5. ((Wu et al., 2026a, Appendix F.1)) For any a , b > 1 a,b>1 a , b > 1 and s , δ a , δ b , γ 1 , γ 2 > 0 s,\delta_a,\delta_b,\gamma_1,\gamma_2>0 s , δ a , δ b , γ 1 , γ 2 > 0 , such that s + a δ a + b δ b > 1 s+a\delta_a+b\delta_b>1 s + a δ a + b δ b > 1 , let
T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = ∑ k = 1 ∞ k − s − δ a a − δ b b ( k − a + μ ) γ 1 ( k − b + ν ) γ 2 . T^{s}_{(\delta_a,\delta_b),(\gamma_1,\gamma_2)}(\mu,\nu;a,b)=\sum_{k=1}^{\infty}\frac{k^{-s-\delta_a a-\delta_b b}}{(k^{-a}+\mu)^{\gamma_1}(k^{-b}+\nu)^{\gamma_2}}. T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = k = 1 ∑ ∞ ( k − a + μ ) γ 1 ( k − b + ν ) γ 2 k − s − δ a a − δ b b .
We define η = 1 − s − a δ a − b δ b + a γ 1 + b γ 2 \eta=1-s-a\delta_a-b\delta_b+a\gamma_1+b\gamma_2 η = 1 − s − a δ a − b δ b + a γ 1 + b γ 2 . Then, the order of the summation above is as follows:
If μ 1 / a ≥ ν 1 / b \mu^{1/a}\geq\nu^{1/b} μ 1/ a ≥ ν 1/ b , then
T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = { Θ ~ ( 1 ) , η ≤ 0 , Θ ~ ( μ − η / a ) , 0 < η ≤ a γ 1 , Θ ~ ( μ − γ 1 ν − ( η − a γ 1 ) / b ) , η > a γ 1 . T^{s}_{(\delta_a,\delta_b),(\gamma_1,\gamma_2)}(\mu,\nu;a,b)=
\begin{cases}
\widetilde{\Theta}(1), & \eta\leq0,\\
\widetilde{\Theta}(\mu^{-\eta/a}), & 0<\eta\leq a\gamma_1,\\
\widetilde{\Theta}(\mu^{-\gamma_1}\nu^{-(\eta-a\gamma_1)/b}), & \eta>a\gamma_1.
\end{cases} T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = ⎩ ⎨ ⎧ Θ ( 1 ) , Θ ( μ − η / a ) , Θ ( μ − γ 1 ν − ( η − a γ 1 ) / b ) , η ≤ 0 , 0 < η ≤ a γ 1 , η > a γ 1 .
If μ 1 / a < ν 1 / b \mu^{1/a}<\nu^{1/b} μ 1/ a < ν 1/ b , then
T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = { Θ ~ ( 1 ) , η ≤ 0 , Θ ~ ( ν − η / b ) , 0 < η ≤ b γ 2 , Θ ~ ( ν − γ 2 μ − ( η − b γ 2 ) / a ) , η > b γ 2 . T^{s}_{(\delta_a,\delta_b),(\gamma_1,\gamma_2)}(\mu,\nu;a,b)=
\begin{cases}
\widetilde{\Theta}(1), & \eta\leq0,\\
\widetilde{\Theta}(\nu^{-\eta/b}), & 0<\eta\leq b\gamma_2,\\
\widetilde{\Theta}(\nu^{-\gamma_2}\mu^{-(\eta-b\gamma_2)/a}), & \eta>b\gamma_2.
\end{cases} T ( δ a , δ b ) , ( γ 1 , γ 2 ) s ( μ , ν ; a , b ) = ⎩ ⎨ ⎧ Θ ( 1 ) , Θ ( ν − η / b ) , Θ ( ν − γ 2 μ − ( η − b γ 2 ) / a ) , η ≤ 0 , 0 < η ≤ b γ 2 , η > b γ 2 .
Proof of Corollary 1
Corollary 2 (Optimal scaling of the deterministic equivalent). Under the assumptions of Theorem 3, define
Γ ≤ : = 1 − a + b + 1 − ( δ + 1 ) + + 2 ( ( s − b ) + ∧ a ) ( 1 − γ n ) [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( ( s − b ) + ∧ a ) . (93) \Gamma_{\leq}:=1-\frac{a+b+1-(\delta+1)_++2((s-b)_+\wedge a)(1-\gamma_n)}{[a+b+1-(\delta+1)_+][1+2(s\wedge(a+b))]+2\delta((s-b)_+\wedge a)}. \tag*{(93)} Γ ≤ := 1 − [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ))] + 2 δ (( s − b ) + ∧ a ) a + b + 1 − ( δ + 1 ) + + 2 (( s − b ) + ∧ a ) ( 1 − γ n ) . (93)
Γ ≥ : = 1 − ( 1 − δ ) + + 2 ( s ∧ a ) ( 1 − γ n ) ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( s ∧ a ) . (94) \Gamma_{\geq}:=1-\frac{(1-\delta)_++2(s\wedge a)(1-\gamma_n)}{(1-\delta)_+[1+2(s\wedge(a+b))]+2\delta(s\wedge a)}. \tag*{(94)} Γ ≥ := 1 − ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b ))] + 2 δ ( s ∧ a ) ( 1 − δ ) + + 2 ( s ∧ a ) ( 1 − γ n ) . (94)
Then, the optimal exponent of the deterministic equivalent is given by
Γ r i d g e ∗ = max { 2 ( s ∧ b ) 1 + 2 ( s ∧ b ) , 2 ( s ∧ a ) γ n 2 ( s ∧ a ) + ( 1 − δ ) + , Γ ≤ , Γ ≥ } . (95) \Gamma_{\mathrm{ridge}}^*=\max\left\{\frac{2(s\wedge b)}{1+2(s\wedge b)},\frac{2(s\wedge a)\gamma_n}{2(s\wedge a)+(1-\delta)_+},\Gamma_{\leq},\Gamma_{\geq}\right\}. \tag*{(95)} Γ ridge ∗ = max { 1 + 2 ( s ∧ b ) 2 ( s ∧ b ) , 2 ( s ∧ a ) + ( 1 − δ ) + 2 ( s ∧ a ) γ n , Γ ≤ , Γ ≥ } . (95)
The following choices attain
R ‾ = Θ ~ ( n − Γ r i d g e ∗ ) . (96) \overline{R}=\widetilde{\Theta}\left(n^{-\Gamma_{\mathrm{ridge}}^*}\right). \tag*{(96)} R = Θ ( n − Γ ridge ∗ ) . (96)
If 2 ( s ∧ b ) 1 + 2 ( s ∧ b ) \frac{2(s\wedge b)}{1+2(s\wedge b)} 1 + 2 ( s ∧ b ) 2 ( s ∧ b ) in (95) attains the maximum, using only target data is optimal, and the corresponding optimal ridge is
λ s r = ∞ , λ t r ≍ n ( 1 − b ) / [ 1 + 2 ( s ∧ b ) ] . \lambda_{sr}=\infty,\qquad\lambda_{tr}\asymp n^{(1-b)/[1+2(s\wedge b)]}. λ sr = ∞ , λ t r ≍ n ( 1 − b ) / [ 1 + 2 ( s ∧ b )] .
If 2 ( s ∧ a ) γ n 2 ( s ∧ a ) + ( 1 − δ ) + \frac{2(s\wedge a)\gamma_n}{2(s\wedge a)+(1-\delta)_+} 2 ( s ∧ a ) + ( 1 − δ ) + 2 ( s ∧ a ) γ n in (95) attains the maximum, using only source data is optimal, and the corresponding optimal ridge is
λ t r = ∞ , λ s r ≍ n ( γ n − a γ n ) / [ 2 ( s ∧ a ) + ( 1 − δ ) + ] . \lambda_{tr}=\infty,\qquad\lambda_{sr}\asymp n^{(\gamma_n-a\gamma_n)/[2(s\wedge a)+(1-\delta)_+]}. λ t r = ∞ , λ sr ≍ n ( γ n − a γ n ) / [ 2 ( s ∧ a ) + ( 1 − δ ) + ] .
Otherwise, the optimal ridge is chosen by
v ⋆ = 1 − Γ r i d g e ⋆ , v^\star= 1 - \Gamma^\star_{\mathrm{ridge}}, v ⋆ = 1 − Γ ridge ⋆ ,
and
u ⋆ = { Γ r i d g e ⋆ − 2 b v ⋆ 2 ( ( s − b ) ∧ a ) , Γ r i d g e ⋆ = Γ ≤ , Γ r i d g e ⋆ − 2 ( ( s − a ) + ∧ b ) v ⋆ 2 ( s ∧ a ) , Γ r i d g e ⋆ = Γ ≥ , u^\star=
\begin{cases}
\dfrac{\Gamma^\star_{\mathrm{ridge}} - 2bv^\star}{2((s-b)\wedge a)}, & \Gamma^\star_{\mathrm{ridge}} = \Gamma_{\le},\\[6pt]
\dfrac{\Gamma^\star_{\mathrm{ridge}} - 2((s-a)_+\wedge b)v^\star}{2(s\wedge a)}, & \Gamma^\star_{\mathrm{ridge}} = \Gamma_{\ge},
\end{cases} u ⋆ = ⎩ ⎨ ⎧ 2 (( s − b ) ∧ a ) Γ ridge ⋆ − 2 b v ⋆ , 2 ( s ∧ a ) Γ ridge ⋆ − 2 (( s − a ) + ∧ b ) v ⋆ , Γ ridge ⋆ = Γ ≤ , Γ ridge ⋆ = Γ ≥ ,
which implies
λ s r ≍ n γ n − a u ⋆ , λ t r ≍ n 1 − b v ⋆ . \lambda_{\mathrm{sr}} \asymp n^{\gamma_n-au^\star}, \qquad\lambda_{\mathrm{tr}} \asymp n^{1-bv^\star}. λ sr ≍ n γ n − a u ⋆ , λ tr ≍ n 1 − b v ⋆ .
Proof of Corollary 2. Note that λ s r = ∞ , λ t r = ∞ \lambda_{\mathrm{sr}}=\infty,\lambda_{\mathrm{tr}}=\infty λ sr = ∞ , λ tr = ∞ exactly correspond to the target-only and auxiliary-only rate in Wu et al., 2026b, Theorem 4. Thus, it remains to consider the case where non-trivial transfer attains the maximum, i.e., Γ r i d g e ⋆ = Γ ≥ \Gamma^\star_{\mathrm{ridge}}=\Gamma_{\ge} Γ ridge ⋆ = Γ ≥ or Γ r i d g e ⋆ = Γ ≤ \Gamma^\star_{\mathrm{ridge}}=\Gamma_{\le} Γ ridge ⋆ = Γ ≤ .
We first show that the exponents in Theorem 3 satisfy Γ B 2 > Γ V s r \Gamma_{B2}>\Gamma_{V_{\mathrm{sr}}} Γ B 2 > Γ V sr for every admissible u > 0 u>0 u > 0 , so this term does not determine the leading order of the deterministic equivalent. Indeed, by (27) and (76) ,
Γ B 2 − Γ V s r = 2 a u + Γ S = [ 2 a − ( a + b − 2 s ) + ] u = [ 2 a ∧ ( 2 s − δ ) ] u > 0. \Gamma_{B2}-\Gamma_{V_{\mathrm{sr}}}=2au+\Gamma_S=\left[2a-(a+b-2s)_+\right]u=\left[2a\wedge(2s-\delta)\right]u>0. Γ B 2 − Γ V sr = 2 a u + Γ S = [ 2 a − ( a + b − 2 s ) + ] u = [ 2 a ∧ ( 2 s − δ ) ] u > 0.
Thus, the exponent of the deterministic equivalent reduces to
min { Γ B 1 , Γ V s r , Γ V t r } . \min\{\Gamma_{B1},\Gamma_{V_{\mathrm{sr}}},\Gamma_{V_{\mathrm{tr}}}\}. min { Γ B 1 , Γ V sr , Γ V tr } .
Next, we express the scaling as a function of u , v u,v u , v . Substituting s = b r t r s=br_{\mathrm{tr}} s = b r tr into Theorem 3 gives
Γ B 1 = 2 b v + Γ s h = { 2 ( ( s − b ) + ∧ a ) u + 2 ( s ∧ b ) v , u ≤ v , 2 ( s ∧ a ) u + 2 ( ( s − a ) + ∧ b ) v , v ≤ u , (97) \Gamma_{B1}=2bv+\Gamma_{\mathrm{sh}}=
\begin{cases}
2((s-b)_+\wedge a)u+2(s\wedge b)v, & u\le v,\\
2(s\wedge a)u+2((s-a)_+\wedge b)v, & v\le u,
\end{cases} \tag*{(97)} Γ B 1 = 2 b v + Γ sh = { 2 (( s − b ) + ∧ a ) u + 2 ( s ∧ b ) v , 2 ( s ∧ a ) u + 2 (( s − a ) + ∧ b ) v , u ≤ v , v ≤ u , (97)
and
Γ V s r = { γ n − [ a + b + 1 − ( δ + 1 ) + ] u + [ 2 b − ( δ + 1 ) + ] v , u ≤ v , γ n − ( 1 − δ ) + u + ( δ − 1 ) + v , v ≤ u , Γ V t r = 1 − v . (98) \Gamma_{V_{\mathrm{sr}}}=
\begin{cases}
\gamma_n-[a+b+1-(\delta+1)_+]u+[2b-(\delta+1)_+]v, & u\le v,\\
\gamma_n-(1-\delta)_+u+(\delta-1)_+v, & v\le u,
\end{cases}
\qquad\Gamma_{V_{\mathrm{tr}}}=1-v. \tag*{(98)} Γ V sr = { γ n − [ a + b + 1 − ( δ + 1 ) + ] u + [ 2 b − ( δ + 1 ) + ] v , γ n − ( 1 − δ ) + u + ( δ − 1 ) + v , u ≤ v , v ≤ u , Γ V tr = 1 − v . (98)
Combining the above bounds yields
R ‾ = Θ ~ ( n − min { Γ B 1 , Γ V s r , Γ V t r } ) . (99) \overline{R}=\widetilde{\Theta}\left(n^{-\min\{\Gamma_{B1},\Gamma_{V_{\mathrm{sr}}},\Gamma_{V_{\mathrm{tr}}}\}}\right). \tag*{(99)} R = Θ ( n − m i n { Γ B 1 , Γ V sr , Γ V tr } ) . (99)
For simplicity, define F ( u , v ) = min { Γ B 1 , Γ V s r , Γ V t r } F(u,v)=\min\{\Gamma_{B1},\Gamma_{V_{\mathrm{sr}}},\Gamma_{V_{\mathrm{tr}}}\} F ( u , v ) = min { Γ B 1 , Γ V sr , Γ V tr } . Thus, optimizing the polynomial decay exponent over the two-stage ridge scalings reduces to maximizing F ( u , v ) F(u,v) F ( u , v ) over 0 < u ≤ γ n 0<u\le\gamma_n 0 < u ≤ γ n and 0 < v ≤ 1 0<v\le1 0 < v ≤ 1 . Next, we show that max u , v F ( u , v ) = Γ r i d g e ⋆ \max_{u,v}F(u,v)=\Gamma^\star_{\mathrm{ridge}} max u , v F ( u , v ) = Γ ridge ⋆ . To do so, we first show that F ( u , v ) ≤ Γ r i d g e ⋆ F(u,v)\le\Gamma^\star_{\mathrm{ridge}} F ( u , v ) ≤ Γ ridge ⋆ for any u , v u,v u , v , and then that F ( u ⋆ , v ⋆ ) = Γ r i d g e ⋆ F(u^\star,v^\star)=\Gamma^\star_{\mathrm{ridge}} F ( u ⋆ , v ⋆ ) = Γ ridge ⋆ , with u ⋆ , v ⋆ u^\star,v^\star u ⋆ , v ⋆ in Corollary 2.
Upper bound. Recall that
F ( u , v ) ≤ 1 − v , F ( u , v ) ≤ Γ V s r , F ( u , v ) ≤ Γ B 1 . F(u,v)\le1-v,\qquad F(u,v)\le\Gamma_{V_{\mathrm{sr}}},\qquad F(u,v)\le\Gamma_{B1}. F ( u , v ) ≤ 1 − v , F ( u , v ) ≤ Γ V sr , F ( u , v ) ≤ Γ B 1 .
Thus, F ( u , v ) F(u,v) F ( u , v ) is bounded above by any nonnegative weighted average of these three quantities.
If u ≤ v u\le v u ≤ v , take the weighted average with weights
2 ( s ∧ b ) [ a + b + 1 − ( δ + 1 ) + ] + 2 ( ( s − b ) + ∧ a ) [ 2 b − ( δ + 1 ) + ] , 2(s\wedge b)[a+b+1-(\delta+1)_+]+2((s-b)_+\wedge a)[2b-(\delta+1)_+], 2 ( s ∧ b ) [ a + b + 1 − ( δ + 1 ) + ] + 2 (( s − b ) + ∧ a ) [ 2 b − ( δ + 1 ) + ] ,
2 ( ( s − b ) + ∧ a ) , a + b + 1 − ( δ + 1 ) + , 2((s-b)_+\wedge a),\qquad a+b+1-(\delta+1)_+, 2 (( s − b ) + ∧ a ) , a + b + 1 − ( δ + 1 ) + ,
respectively. These weights are nonnegative. Substituting from (97) and (98) , the coefficients of both u u u and v v v cancel. Dividing by the sum of the weights and simplifying gives
F ( u , v ) ≤ 1 − a + b + 1 − ( δ + 1 ) + + 2 ( ( s − b ) + ∧ a ) ( 1 − γ n ) [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( ( s − b ) + ∧ a ) = Γ ≤ . (100) F(u,v)\le1-\frac{a+b+1-(\delta+1)_++2((s-b)_+\wedge a)(1-\gamma_n)}{[a+b+1-(\delta+1)_+][1+2(s\wedge(a+b))]+2\delta((s-b)_+\wedge a)}=\Gamma_{\le}. \tag*{(100)} F ( u , v ) ≤ 1 − [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ))] + 2 δ (( s − b ) + ∧ a ) a + b + 1 − ( δ + 1 ) + + 2 (( s − b ) + ∧ a ) ( 1 − γ n ) = Γ ≤ . (100)
If v ≤ u v\le u v ≤ u and δ < 1 \delta<1 δ < 1 , use instead the nonnegative weights
2 ( 1 − δ ) ( ( s − a ) + ∧ b ) , 2 ( s ∧ a ) , 1 − δ , 2(1-\delta)((s-a)_+\wedge b),\qquad2(s\wedge a),\qquad1-\delta, 2 ( 1 − δ ) (( s − a ) + ∧ b ) , 2 ( s ∧ a ) , 1 − δ ,
respectively. Again, the coefficients of u u u and v v v cancel, yielding
F ( u , v ) ≤ 2 ( s ∧ a ) γ n + 2 ( 1 − δ ) ( ( s − a ) + ∧ b ) 2 ( s ∧ a ) + ( 1 − δ ) [ 1 + 2 ( ( s − a ) + ∧ b ) ] = Γ ≥ . (101) F(u,v)\le\frac{2(s\wedge a)\gamma_n+2(1-\delta)((s-a)_+\wedge b)}{2(s\wedge a)+(1-\delta)[1+2((s-a)_+\wedge b)]}=\Gamma_{\ge}. \tag*{(101)} F ( u , v ) ≤ 2 ( s ∧ a ) + ( 1 − δ ) [ 1 + 2 (( s − a ) + ∧ b )] 2 ( s ∧ a ) γ n + 2 ( 1 − δ ) (( s − a ) + ∧ b ) = Γ ≥ . (101)
Finally, if v ≤ u v \le u v ≤ u and δ ≥ 1 \delta\ge1 δ ≥ 1 , then Γ V s r = γ n + ( δ − 1 ) v \Gamma_{\mathrm{Vsr}}=\gamma_n+(\delta-1)v Γ Vsr = γ n + ( δ − 1 ) v . The two variance constraints alone give
F ( u , v ) ≤ ( δ − 1 ) ( 1 − v ) + Γ V s r δ = δ − 1 + γ n δ = Γ ≥ . F(u,v)\le\frac{(\delta-1)(1-v)+\Gamma_{\mathrm{Vsr}}}{\delta}=\frac{\delta-1+\gamma_n}{\delta}=\Gamma_{\ge}. F ( u , v ) ≤ δ ( δ − 1 ) ( 1 − v ) + Γ Vsr = δ δ − 1 + γ n = Γ ≥ .
Therefore, every admissible two-stage choice satisfies
F ( u , v ) ≤ max { Γ ≤ , Γ ≥ } ≤ Γ r i d g e ⋆ . F(u,v)\le\max\{\Gamma_{\le},\Gamma_{\ge}\}\le\Gamma_{\mathrm{ridge}}^{\star}. F ( u , v ) ≤ max { Γ ≤ , Γ ≥ } ≤ Γ ridge ⋆ .
Together with the previously established single-dataset exponents, this proves that no admissible choice has a polynomial decay exponent exceeding (95) .
Lower bound. We show that u ⋆ , v ⋆ u_{\star},v_{\star} u ⋆ , v ⋆ are admissible and in fact
F ( u ⋆ , v ⋆ ) = Γ r i d g e ⋆ . F(u^{\star},v^{\star})=\Gamma_{\mathrm{ridge}}^{\star}. F ( u ⋆ , v ⋆ ) = Γ ridge ⋆ .
We first discuss properties of Γ ≥ , Γ ≤ \Gamma_{\ge},\Gamma_{\le} Γ ≥ , Γ ≤ . As a function of γ n \gamma_n γ n , Γ ≤ \Gamma_{\le} Γ ≤ and Γ ≥ \Gamma_{\ge} Γ ≥ are affine and intersect at
γ n = 1 − δ 1 + 2 ( s ∧ ( a + b ) ) , \gamma_n=1-\frac{\delta}{1+2(s\wedge(a+b))}, γ n = 1 − 1 + 2 ( s ∧ ( a + b )) δ ,
where their common value is 2 ( s ∧ ( a + b ) ) / ( 1 + 2 ( s ∧ ( a + b ) ) ) 2(s\wedge(a+b))/(1+2(s\wedge(a+b))) 2 ( s ∧ ( a + b )) / ( 1 + 2 ( s ∧ ( a + b ))) . Moreover, Γ ≥ \Gamma_{\ge} Γ ≥ has strictly larger slope as
2 ( s ∧ a ) ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( s ∧ a ) > 2 ( ( s − b ) + ∧ a ) [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( ( s − b ) + ∧ a ) \frac{2(s\wedge a)}{(1-\delta)_{+}[1+2(s\wedge(a+b))]+2\delta(s\wedge a)}
>
\frac{2((s-b)_{+}\wedge a)}{[a+b+1-(\delta+1)_{+}][1+2(s\wedge(a+b))]+2\delta((s-b)_{+}\wedge a)} ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b ))] + 2 δ ( s ∧ a ) 2 ( s ∧ a ) > [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ))] + 2 δ (( s − b ) + ∧ a ) 2 (( s − b ) + ∧ a )
where both denominators are positive, and we use
0 ≤ ( ( s − b ) + ∧ a ) ≤ s ∧ a , s ∧ a > 0 , 0\le((s-b)_{+}\wedge a)\le s\wedge a,\qquad s\wedge a>0, 0 ≤ (( s − b ) + ∧ a ) ≤ s ∧ a , s ∧ a > 0 ,
together with
0 < a + b + 1 − ( δ + 1 ) + − ( 1 − δ ) + = { 2 b , δ < − 1 , a + b − 1 , − 1 ≤ δ ≤ 1 , 2 a , δ > 1 , 0<a+b+1-(\delta+1)_{+}-(1-\delta)_{+}=
\begin{cases}
2b,&\delta<-1,\\
a+b-1,&-1\le\delta\le1,\\
2a,&\delta>1,
\end{cases} 0 < a + b + 1 − ( δ + 1 ) + − ( 1 − δ ) + = ⎩ ⎨ ⎧ 2 b , a + b − 1 , 2 a , δ < − 1 , − 1 ≤ δ ≤ 1 , δ > 1 ,
which follows from δ = b − a \delta=b-a δ = b − a and a , b > 1 a,b>1 a , b > 1 . Consequently,
Γ ≤ ≥ Γ ≥ ⟹ Γ ≤ ≤ 2 ( s ∧ ( a + b ) ) 1 + 2 ( s ∧ ( a + b ) ) , \Gamma_{\le}\ge\Gamma_{\ge}\Longrightarrow\Gamma_{\le}\le\frac{2(s\wedge(a+b))}{1+2(s\wedge(a+b))}, Γ ≤ ≥ Γ ≥ ⟹ Γ ≤ ≤ 1 + 2 ( s ∧ ( a + b )) 2 ( s ∧ ( a + b )) ,
Γ ≥ ≥ Γ ≤ ⟹ Γ ≥ ≥ 2 ( s ∧ ( a + b ) ) 1 + 2 ( s ∧ ( a + b ) ) . (102) \Gamma_{\ge}\ge\Gamma_{\le}\Longrightarrow\Gamma_{\ge}\ge\frac{2(s\wedge(a+b))}{1+2(s\wedge(a+b))}. \tag*{(102)} Γ ≥ ≥ Γ ≤ ⟹ Γ ≥ ≥ 1 + 2 ( s ∧ ( a + b )) 2 ( s ∧ ( a + b )) . (102)
Next, we show that by choosing
v ⋆ = 1 − Γ r i d g e ⋆ , u ⋆ = { Γ r i d g e ⋆ − 2 b v ⋆ 2 ( ( s − b ) ∧ a ) , Γ r i d g e ⋆ = Γ ≤ , Γ r i d g e ⋆ − 2 ( ( s − a ) + ∧ b ) v ⋆ 2 ( s ∧ a ) , Γ r i d g e ⋆ = Γ ≥ , v^{\star}=1-\Gamma_{\mathrm{ridge}}^{\star},\qquad
u^{\star}=
\begin{cases}
\frac{\Gamma_{\mathrm{ridge}}^{\star}-2bv^{\star}}{2((s-b)\wedge a)},&\Gamma_{\mathrm{ridge}}^{\star}=\Gamma_{\le},\\[1.2ex]
\frac{\Gamma_{\mathrm{ridge}}^{\star}-2((s-a)_{+}\wedge b)v^{\star}}{2(s\wedge a)},&\Gamma_{\mathrm{ridge}}^{\star}=\Gamma_{\ge},
\end{cases} v ⋆ = 1 − Γ ridge ⋆ , u ⋆ = ⎩ ⎨ ⎧ 2 (( s − b ) ∧ a ) Γ ridge ⋆ − 2 b v ⋆ , 2 ( s ∧ a ) Γ ridge ⋆ − 2 (( s − a ) + ∧ b ) v ⋆ , Γ ridge ⋆ = Γ ≤ , Γ ridge ⋆ = Γ ≥ ,
we have
Γ B 1 ( u ⋆ , v ⋆ ) , Γ V s r ( u ⋆ , v ⋆ ) , Γ V t r ( u ⋆ , v ⋆ ) ≥ Γ r i d g e ⋆ . \Gamma_{\mathrm{B1}}(u^{\star},v^{\star}),\Gamma_{\mathrm{Vsr}}(u^{\star},v^{\star}),\Gamma_{\mathrm{Vtr}}(u^{\star},v^{\star})\ge\Gamma_{\mathrm{ridge}}^{\star}. Γ B1 ( u ⋆ , v ⋆ ) , Γ Vsr ( u ⋆ , v ⋆ ) , Γ Vtr ( u ⋆ , v ⋆ ) ≥ Γ ridge ⋆ .
First note that, by definition, Γ V t r ( u ⋆ , v ⋆ ) = 1 − v ⋆ = Γ r i d g e ⋆ \Gamma_{\mathrm{Vtr}}(u^{\star},v^{\star})=1-v^{\star}=\Gamma_{\mathrm{ridge}}^{\star} Γ Vtr ( u ⋆ , v ⋆ ) = 1 − v ⋆ = Γ ridge ⋆ . We then show that
Γ V s r ( u ⋆ , v ⋆ ) ≥ Γ r i d g e ⋆ , Γ B 1 ( u ⋆ , v ⋆ ) ≥ Γ r i d g e ⋆ , \Gamma_{\mathrm{Vsr}}(u^{\star},v^{\star})\ge\Gamma_{\mathrm{ridge}}^{\star},\qquad
\Gamma_{\mathrm{B1}}(u^{\star},v^{\star})\ge\Gamma_{\mathrm{ridge}}^{\star}, Γ Vsr ( u ⋆ , v ⋆ ) ≥ Γ ridge ⋆ , Γ B1 ( u ⋆ , v ⋆ ) ≥ Γ ridge ⋆ ,
and that u ⋆ , v ⋆ u^{\star},v^{\star} u ⋆ , v ⋆ are admissible, namely
0 < u ⋆ ≤ γ n , 0 < v ⋆ ≤ 1. 0<u^{\star}\le\gamma_n,\qquad0<v^{\star}\le1. 0 < u ⋆ ≤ γ n , 0 < v ⋆ ≤ 1.
Recall that Γ r i d g e ⋆ \Gamma_{\mathrm{ridge}}^{\star} Γ ridge ⋆ strictly exceeds both single-dataset exponents in the case under consideration. Case Γ r i d g e ⋆ = Γ ≥ \Gamma_{\mathrm{ridge}}^{\star}=\Gamma_{\ge} Γ ridge ⋆ = Γ ≥ . We first verify ( u ⋆ , v ⋆ ) (u^{\star},v^{\star}) ( u ⋆ , v ⋆ ) are admissible. Using (94) gives
0 < Γ r i d g e ⋆ − 2 ( s ∧ a ) γ n 2 ( s ∧ a ) + ( 1 − δ ) + = 2 ( 1 − δ ) + ( ( s − a ) + ∧ b ) + 2 ( s ∧ a ) ( δ − 1 ) + 2 ( s ∧ a ) + ( 1 − δ ) + + v ⋆ . 0<\Gamma_{\mathrm{ridge}}^{\star}-\frac{2(s\wedge a)\gamma_n}{2(s\wedge a)+(1-\delta)_{+}}
=\frac{2(1-\delta)_{+}((s-a)_{+}\wedge b)+2(s\wedge a)(\delta-1)_{+}}{2(s\wedge a)+(1-\delta)_{+}}+v^{\star}. 0 < Γ ridge ⋆ − 2 ( s ∧ a ) + ( 1 − δ ) + 2 ( s ∧ a ) γ n = 2 ( s ∧ a ) + ( 1 − δ ) + 2 ( 1 − δ ) + (( s − a ) + ∧ b ) + 2 ( s ∧ a ) ( δ − 1 ) + + v ⋆ .
Since the coefficient of v ⋆ v^\star v ⋆ is nonnegative, this implies v ⋆ > 0 v^\star> 0 v ⋆ > 0 . Also v ⋆ < 1 v^\star< 1 v ⋆ < 1 , since Γ r i d g e ⋆ > 0 \Gamma^\star_{\mathrm{ridge}} > 0 Γ ridge ⋆ > 0 . By (102) ,
u ⋆ − v ⋆ = [ 1 + 2 ( s ∧ ( a + b ) ) ] Γ r i d g e ⋆ − 2 ( s ∧ ( a + b ) ) 2 ( s ∧ a ) ≥ 0. u^\star-v^\star=\frac{\left[1+2(s\wedge(a+b))\right]\Gamma^\star_{\mathrm{ridge}}-2(s\wedge(a+b))}{2(s\wedge a)}\geq0. u ⋆ − v ⋆ = 2 ( s ∧ a ) [ 1 + 2 ( s ∧ ( a + b )) ] Γ ridge ⋆ − 2 ( s ∧ ( a + b )) ≥ 0.
Thus 0 < v ⋆ ≤ u ⋆ 0<v^\star\leq u^\star 0 < v ⋆ ≤ u ⋆ . The definitions and (94) also give
γ n − u ⋆ = [ 2 ( s ∧ a ) + ( 1 − δ ) + − 1 ] ( u ⋆ − v ⋆ ) + [ 2 ( s ∧ ( a + b ) ) − δ ] v ⋆ > 0. \gamma_n-u^\star=\left[2(s\wedge a)+(1-\delta)_+-1\right](u^\star-v^\star)+\left[2(s\wedge(a+b))-\delta\right]v^\star>0. γ n − u ⋆ = [ 2 ( s ∧ a ) + ( 1 − δ ) + − 1 ] ( u ⋆ − v ⋆ ) + [ 2 ( s ∧ ( a + b )) − δ ] v ⋆ > 0.
Here both coefficients are positive under a , b > 1 a,b>1 a , b > 1 , s > 0 s>0 s > 0 , and 2 s > δ 2s>\delta 2 s > δ . In this case ( u ⋆ , v ⋆ ) (u^\star,v^\star) ( u ⋆ , v ⋆ ) are admissible and satisfy v ⋆ ≤ u ⋆ v^\star\leq u^\star v ⋆ ≤ u ⋆ .
For B 1 B_1 B 1 , we therefore have
Γ B 1 ( u ⋆ , v ⋆ ) = 2 ( s ∧ a ) u ⋆ + 2 ( ( s − a ) + ∧ b ) v ⋆ = Γ r i d g e ⋆ − 2 ( ( s − a ) + ∧ b ) v ⋆ + 2 ( ( s − a ) + ∧ b ) v ⋆ = Γ r i d g e ⋆ . \Gamma_{B_1}(u^\star,v^\star)=2(s\wedge a)u^\star+2((s-a)_+\wedge b)v^\star=\Gamma^\star_{\mathrm{ridge}}-2((s-a)_+\wedge b)v^\star+2((s-a)_+\wedge b)v^\star=\Gamma^\star_{\mathrm{ridge}}. Γ B 1 ( u ⋆ , v ⋆ ) = 2 ( s ∧ a ) u ⋆ + 2 (( s − a ) + ∧ b ) v ⋆ = Γ ridge ⋆ − 2 (( s − a ) + ∧ b ) v ⋆ + 2 (( s − a ) + ∧ b ) v ⋆ = Γ ridge ⋆ .
For the source variance, using
s ∧ ( a + b ) = ( s ∧ a ) + ( ( s − a ) + ∧ b ) , ( 1 − δ ) + + δ = 1 + ( δ − 1 ) + , s\wedge(a+b)=(s\wedge a)+((s-a)_+\wedge b),\qquad(1-\delta)_++\delta=1+(\delta-1)_+, s ∧ ( a + b ) = ( s ∧ a ) + (( s − a ) + ∧ b ) , ( 1 − δ ) + + δ = 1 + ( δ − 1 ) + ,
we obtain
Γ v s r ( u ⋆ , v ⋆ ) = γ n − ( 1 − δ ) + u ⋆ + ( δ − 1 ) + v ⋆ = γ n − ( 1 − δ ) + 2 ( s ∧ a ) + [ ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( s ∧ a ) 2 ( s ∧ a ) − 1 ] v ⋆ = γ n − ( 1 − δ ) + 2 ( s ∧ a ) + ( 1 − δ ) + + 2 ( s ∧ a ) ( 1 − γ n ) 2 ( s ∧ a ) − v ⋆ = 1 − v ⋆ = Γ r i d g e ⋆ . \begin{aligned}
\Gamma_{\mathrm{vsr}}(u^\star,v^\star)=\gamma_n-(1-\delta)_+u^\star+(\delta-1)_+v^\star\\
&=\gamma_n-\frac{(1-\delta)_+}{2(s\wedge a)}+\left[\frac{(1-\delta)_+\left[1+2(s\wedge(a+b))\right]+2\delta(s\wedge a)}{2(s\wedge a)}-1\right]v^\star\\
&=\gamma_n-\frac{(1-\delta)_+}{2(s\wedge a)}+\frac{(1-\delta)_++2(s\wedge a)(1-\gamma_n)}{2(s\wedge a)}-v^\star\\
&=1-v^\star=\Gamma^\star_{\mathrm{ridge}}.
\end{aligned} Γ vsr ( u ⋆ , v ⋆ ) = γ n − ( 1 − δ ) + u ⋆ + ( δ − 1 ) + v ⋆ = γ n − 2 ( s ∧ a ) ( 1 − δ ) + + [ 2 ( s ∧ a ) ( 1 − δ ) + [ 1 + 2 ( s ∧ ( a + b )) ] + 2 δ ( s ∧ a ) − 1 ] v ⋆ = γ n − 2 ( s ∧ a ) ( 1 − δ ) + + 2 ( s ∧ a ) ( 1 − δ ) + + 2 ( s ∧ a ) ( 1 − γ n ) − v ⋆ = 1 − v ⋆ = Γ ridge ⋆ .
Case Γ r i d g e ⋆ = Γ ≤ \Gamma^\star_{\mathrm{ridge}}=\Gamma_{\leq} Γ ridge ⋆ = Γ ≤ . In this case, we must have s > b s>b s > b ; otherwise Γ ≤ = 2 s / ( 1 + 2 s ) \Gamma_{\leq}=2s/(1+2s) Γ ≤ = 2 s / ( 1 + 2 s ) would equal the target-only exponent, contrary to the strict comparison. Thus the denominator in the definition of u ⋆ u^\star u ⋆ is positive. Moreover, Γ r i d g e ⋆ > 2 b / ( 1 + 2 b ) \Gamma^\star_{\mathrm{ridge}}>2b/(1+2b) Γ ridge ⋆ > 2 b / ( 1 + 2 b ) implies
u ⋆ = ( 1 + 2 b ) Γ r i d g e ⋆ − 2 b 2 ( ( s − b ) ∧ a ) > 0. u^\star=\frac{(1+2b)\Gamma^\star_{\mathrm{ridge}}-2b}{2((s-b)\wedge a)}>0. u ⋆ = 2 (( s − b ) ∧ a ) ( 1 + 2 b ) Γ ridge ⋆ − 2 b > 0.
By (102) ,
u ⋆ − v ⋆ = [ 1 + 2 ( s ∧ ( a + b ) ) ] Γ r i d g e ⋆ − 2 ( s ∧ ( a + b ) ) 2 ( ( s − b ) ∧ a ) ≤ 0. u^\star-v^\star=\frac{\left[1+2(s\wedge(a+b))\right]\Gamma^\star_{\mathrm{ridge}}-2(s\wedge(a+b))}{2((s-b)\wedge a)}\leq0. u ⋆ − v ⋆ = 2 (( s − b ) ∧ a ) [ 1 + 2 ( s ∧ ( a + b )) ] Γ ridge ⋆ − 2 ( s ∧ ( a + b )) ≤ 0.
Hence 0 < u ⋆ ≤ v ⋆ 0<u^\star\leq v^\star 0 < u ⋆ ≤ v ⋆ . Also v ⋆ < 1 v^\star<1 v ⋆ < 1 , since Γ r i d g e ⋆ > 0 \Gamma^\star_{\mathrm{ridge}}>0 Γ ridge ⋆ > 0 . The definitions and (93) give
γ n − u ⋆ = [ a + b − ( δ + 1 ) + + 2 ( ( s − b ) ∧ a ) ] u ⋆ + ( δ + 1 ) + v ⋆ > 0. \gamma_n-u^\star=\left[a+b-(\delta+1)_++2((s-b)\wedge a)\right]u^\star+(\delta+1)_+v^\star>0. γ n − u ⋆ = [ a + b − ( δ + 1 ) + + 2 (( s − b ) ∧ a ) ] u ⋆ + ( δ + 1 ) + v ⋆ > 0.
Indeed, the coefficient of u ⋆ u^\star u ⋆ is positive because
a + b − ( δ + 1 ) + = min { a + b , 2 a − 1 } > 0. a+b-(\delta+1)_+=\min\{a+b,2a-1\}>0. a + b − ( δ + 1 ) + = min { a + b , 2 a − 1 } > 0.
Thus ( u ⋆ , v ⋆ ) (u^\star,v^\star) ( u ⋆ , v ⋆ ) are admissible and satisfy u ⋆ ≤ v ⋆ u^\star\leq v^\star u ⋆ ≤ v ⋆ .
For the leading bias, using s > b s>b s > b , we obtain
Γ B 1 ( u ⋆ , v ⋆ ) = 2 ( ( s − b ) ∧ a ) u ⋆ + 2 b v ⋆ = Γ r i d g e ⋆ − 2 b v ⋆ + 2 b v ⋆ = Γ r i d g e ⋆ . \Gamma_{B_1}(u^\star,v^\star)=2((s-b)\wedge a)u^\star+2bv^\star=\Gamma^\star_{\mathrm{ridge}}-2bv^\star+2bv^\star=\Gamma^\star_{\mathrm{ridge}}. Γ B 1 ( u ⋆ , v ⋆ ) = 2 (( s − b ) ∧ a ) u ⋆ + 2 b v ⋆ = Γ ridge ⋆ − 2 b v ⋆ + 2 b v ⋆ = Γ ridge ⋆ .
For the source variance, using
s ∧ ( a + b ) = b + ( ( s − b ) ∧ a ) , δ = b − a , s\wedge(a+b)=b+((s-b)\wedge a),\qquad\delta=b-a, s ∧ ( a + b ) = b + (( s − b ) ∧ a ) , δ = b − a ,
we have
Γ v s r ( u ⋆ , v ⋆ ) = γ n − [ a + b + 1 − ( δ + 1 ) + ] u ⋆ + [ 2 b − ( δ + 1 ) + ] v ⋆ = γ n − a + b + 1 − ( δ + 1 ) + 2 ( ( s − b ) ∧ a ) + [ [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b ) ) ] + 2 δ ( ( s − b ) ∧ a ) 2 ( ( s − b ) ∧ a ) − 1 ] v ⋆ = γ n − a + b + 1 − ( δ + 1 ) + 2 ( ( s − b ) ∧ a ) + a + b + 1 − ( δ + 1 ) + + 2 ( ( s − b ) ∧ a ) ( 1 − γ n ) 2 ( ( s − b ) ∧ a ) − v ⋆ = 1 − v ⋆ = Γ r i d g e ⋆ . \begin{aligned}
\Gamma_{\mathrm{vsr}}(u^\star,v^\star)=\gamma_n-\left[a+b+1-(\delta+1)_+\right]u^\star+\left[2b-(\delta+1)_+\right]v^\star\\
&=\gamma_n-\frac{a+b+1-(\delta+1)_+}{2((s-b)\wedge a)}+\left[\frac{\left[a+b+1-(\delta+1)_+\right]\left[1+2(s\wedge(a+b))\right]+2\delta((s-b)\wedge a)}{2((s-b)\wedge a)}-1\right]v^\star\\
&=\gamma_n-\frac{a+b+1-(\delta+1)_+}{2((s-b)\wedge a)}+\frac{a+b+1-(\delta+1)_++2((s-b)\wedge a)(1-\gamma_n)}{2((s-b)\wedge a)}-v^\star\\
&=1-v^\star=\Gamma^\star_{\mathrm{ridge}}.
\end{aligned} Γ vsr ( u ⋆ , v ⋆ ) = γ n − [ a + b + 1 − ( δ + 1 ) + ] u ⋆ + [ 2 b − ( δ + 1 ) + ] v ⋆ = γ n − 2 (( s − b ) ∧ a ) a + b + 1 − ( δ + 1 ) + + [ 2 (( s − b ) ∧ a ) [ a + b + 1 − ( δ + 1 ) + ] [ 1 + 2 ( s ∧ ( a + b )) ] + 2 δ (( s − b ) ∧ a ) − 1 ] v ⋆ = γ n − 2 (( s − b ) ∧ a ) a + b + 1 − ( δ + 1 ) + + 2 (( s − b ) ∧ a ) a + b + 1 − ( δ + 1 ) + + 2 (( s − b ) ∧ a ) ( 1 − γ n ) − v ⋆ = 1 − v ⋆ = Γ ridge ⋆ .
The last substitution uses v ⋆ = 1 − Γ ≤ v^\star=1-\Gamma_{\leq} v ⋆ = 1 − Γ ≤ and (93) .
In both cases, the proposed cutoffs are admissible and satisfy
Γ B 1 ( u ⋆ , v ⋆ ) = Γ v s r ( u ⋆ , v ⋆ ) = Γ v t r ( u ⋆ , v ⋆ ) = Γ r i d g e ⋆ . \Gamma_{B_1}(u^\star,v^\star)=\Gamma_{\mathrm{vsr}}(u^\star,v^\star)=\Gamma_{\mathrm{vtr}}(u^\star,v^\star)=\Gamma^\star_{\mathrm{ridge}}. Γ B 1 ( u ⋆ , v ⋆ ) = Γ vsr ( u ⋆ , v ⋆ ) = Γ vtr ( u ⋆ , v ⋆ ) = Γ ridge ⋆ .
Consequently,
F ( u ⋆ , v ⋆ ) = Γ r i d g e ⋆ , F(u^\star,v^\star)=\Gamma^\star_{\mathrm{ridge}}, F ( u ⋆ , v ⋆ ) = Γ ridge ⋆ ,
which establishes that picking ( u ⋆ , v ⋆ ) (u^\star,v^\star) ( u ⋆ , v ⋆ ) achieves the optimal scaling exponent.
Proof of cor:minimax-optimal-ridge-transfer
Suppose δ > 1 \delta> 1 δ > 1 and γ c < γ n < 1 \gamma_c < \gamma_n < 1 γ c < γ n < 1 . Since
γ n > γ c = 1 − δ 1 + 2 s ≥ 1 − δ 1 + 2 ( s ∧ ( a + b ) ) , \gamma_n > \gamma_c = 1 - \frac{\delta}{1 + 2s} \ge1 - \frac{\delta}{1 + 2(s \wedge(a+b))}, γ n > γ c = 1 − 1 + 2 s δ ≥ 1 − 1 + 2 ( s ∧ ( a + b )) δ ,
we have Γ ≥ > Γ ≤ \Gamma_{\ge} > \Gamma_{\le} Γ ≥ > Γ ≤ . Indeed, δ > 1 \delta> 1 δ > 1 and δ = b − a \delta= b-a δ = b − a imply
( 1 − δ ) + = 0 , a + b + 1 − ( δ + 1 ) + = 2 a . (1-\delta)_+ = 0, \qquad a+b+1-(\delta+1)_+ = 2a. ( 1 − δ ) + = 0 , a + b + 1 − ( δ + 1 ) + = 2 a .
Substituting these identities into (93) and (94) gives
Γ ≥ − Γ ≤ = a + ( ( s − b ) + ∧ a ) ( 1 − γ n ) a [ 1 + 2 ( s ∧ ( a + b ) ) ] + δ ( ( s − b ) + ∧ a ) − 1 − γ n δ = a { δ − [ 1 + 2 ( s ∧ ( a + b ) ) ] ( 1 − γ n ) } δ { a [ 1 + 2 ( s ∧ ( a + b ) ) ] + δ ( ( s − b ) + ∧ a ) } > 0. \Gamma_{\ge} - \Gamma_{\le} = \frac{a + ((s-b)_+ \wedge a)(1-\gamma_n)}{a[1+2(s\wedge(a+b))] + \delta((s-b)_+ \wedge a)} - \frac{1-\gamma_n}{\delta}
= \frac{a\{\delta- [1+2(s\wedge(a+b))](1-\gamma_n)\}}{\delta\{a[1+2(s\wedge(a+b))] + \delta((s-b)_+ \wedge a)\}} > 0. Γ ≥ − Γ ≤ = a [ 1 + 2 ( s ∧ ( a + b ))] + δ (( s − b ) + ∧ a ) a + (( s − b ) + ∧ a ) ( 1 − γ n ) − δ 1 − γ n = δ { a [ 1 + 2 ( s ∧ ( a + b ))] + δ (( s − b ) + ∧ a )} a { δ − [ 1 + 2 ( s ∧ ( a + b ))] ( 1 − γ n )} > 0.
Here the denominator is positive, and the numerator is positive because
δ − [ 1 + 2 ( s ∧ ( a + b ) ) ] ( 1 − γ n ) = [ 1 + 2 ( s ∧ ( a + b ) ) ] ( γ n − 1 + δ 1 + 2 ( s ∧ ( a + b ) ) ) > 0. \delta- [1+2(s\wedge(a+b))](1-\gamma_n) = [1+2(s\wedge(a+b))]\left(\gamma_n - 1 + \frac{\delta}{1+2(s\wedge(a+b))}\right) > 0. δ − [ 1 + 2 ( s ∧ ( a + b ))] ( 1 − γ n ) = [ 1 + 2 ( s ∧ ( a + b ))] ( γ n − 1 + 1 + 2 ( s ∧ ( a + b )) δ ) > 0.
Moreover,
Γ ≥ = δ − 1 + γ n δ > max { γ n , 2 s 1 + 2 s } , \Gamma_{\ge} = \frac{\delta-1+\gamma_n}{\delta} > \max\left\{\gamma_n,\frac{2s}{1+2s}\right\}, Γ ≥ = δ δ − 1 + γ n > max { γ n , 1 + 2 s 2 s } ,
so Γ ≥ \Gamma_{\ge} Γ ≥ also strictly exceeds both single-dataset exponents. Thus Corollary 2 yields
Γ r i d g e ∗ = δ − 1 + γ n δ , v ∗ = 1 − γ n δ . \Gamma_{\mathrm{ridge}}^* = \frac{\delta-1+\gamma_n}{\delta}, \qquad v^* = \frac{1-\gamma_n}{\delta}. Γ ridge ∗ = δ δ − 1 + γ n , v ∗ = δ 1 − γ n .
We finally argue that u = γ n u=\gamma_n u = γ n is optimal. Note that the optimal ( u ∗ , v ∗ ) (u^*,v^*) ( u ∗ , v ∗ ) by Corollary 2 satisfies v ∗ ≤ u ∗ ≤ γ n v^* \le u^* \le\gamma_n v ∗ ≤ u ∗ ≤ γ n . For fixed v = v ∗ v=v^* v = v ∗ and v ∗ ≤ u ≤ γ n v^* \le u \le\gamma_n v ∗ ≤ u ≤ γ n , the two variance exponents are
Γ V t r = 1 − v ∗ , Γ V s r = γ n + ( δ − 1 ) v ∗ , \Gamma_{V_{\mathrm{tr}}} = 1-v^*, \qquad\Gamma_{V_{\mathrm{sr}}} = \gamma_n + (\delta-1)v^*, Γ V tr = 1 − v ∗ , Γ V sr = γ n + ( δ − 1 ) v ∗ ,
and hence do not depend on u u u . Meanwhile, the leading-bias exponent
Γ B 1 = 2 ( s ∧ a ) u + 2 ( ( s − a ) + ∧ b ) v ∗ \Gamma_{B_1} = 2(s\wedge a)u + 2((s-a)_+ \wedge b)v^* Γ B 1 = 2 ( s ∧ a ) u + 2 (( s − a ) + ∧ b ) v ∗
is nondecreasing in u u u . Since Γ B 2 > Γ V s r \Gamma_{B_2} > \Gamma_{V_{\mathrm{sr}}} Γ B 2 > Γ V sr , the second bias term does not determine the overall exponent. Consequently, replacing u ∗ u^* u ∗ in Corollary 2 by γ n \gamma_n γ n preserves the optimal exponent, as Γ r i d g e ∗ \Gamma_{\mathrm{ridge}}^* Γ ridge ∗ is the minimal over the three exponents. Thus, the choice u = γ n u=\gamma_n u = γ n , v = 1 − γ n δ v=\frac{1-\gamma_n}{\delta} v = δ 1 − γ n is also optimal. By (25) , this choice is realized by
λ s r ≍ n s r 1 − a , λ t r ≍ n 1 − b ( 1 − γ n ) / δ , \lambda_{\mathrm{sr}} \asymp n_{\mathrm{sr}}^{1-a}, \qquad\lambda_{\mathrm{tr}} \asymp n^{1-b(1-\gamma_n)/\delta}, λ sr ≍ n sr 1 − a , λ tr ≍ n 1 − b ( 1 − γ n ) / δ ,
which proves the Corollary 1.
Proof of Lemma 1
To prove Lemma 1, we first condition on the pretrained source estimator and apply the last-iterate SGD upper bound in Wu et al., 2022, Corollary 3.4. This reduces the upper bound to two terms that depend on the source ridge estimator. We then compute the deterministic equivalent of these two terms, and evaluate the scaling law. Finally, we derive a matching lower bound.
Before going into the proof, we define some shorthand notations. Recall the power-law model in (17) and (20) , with
δ = b − a > 1 , γ c = 1 − δ 1 + 2 s < γ n < 1 , 2 s > b . \delta= b-a > 1, \qquad\gamma_c = 1-\frac{\delta}{1+2s} < \gamma_n < 1, \qquad2s>b. δ = b − a > 1 , γ c = 1 − 1 + 2 s δ < γ n < 1 , 2 s > b .
Also by the condition that 2 s > b 2s>b 2 s > b we have
∥ β ∗ ∥ 2 2 ≍ ∑ k ≥ 1 k b − 1 − 2 s < ∞ . (103) \lVert\beta_*\rVert_2^2 \asymp\sum_{k\ge1} k^{b-1-2s} < \infty. \tag*{(103)} ∥ β ∗ ∥ 2 2 ≍ k ≥ 1 ∑ k b − 1 − 2 s < ∞. (103)
The signal follows a power law, with s = b t r = a r s r + δ / 2 s=b_{\mathrm{tr}}=a r_{\mathrm{sr}}+\delta/2 s = b tr = a r sr + δ /2 , and both noise variances are O ( 1 ) O(1) O ( 1 ) . For simplicity define
m : = n s r , N : = ⌊ n log 2 n ⌋ , K : = ( N m ) 1 / δ , v : = 1 − γ n δ . (104) m:=n_{\mathrm{sr}},\qquad N:=\left\lfloor\frac{n}{\log_2 n}\right\rfloor,\qquad K:=\left(\frac{N}{m}\right)^{1/\delta},\qquad v:=\frac{1-\gamma_n}{\delta}. \tag*{(104)} m := n sr , N := ⌊ log 2 n n ⌋ , K := ( m N ) 1/ δ , v := δ 1 − γ n . (104)
We first derive the order of certain quantities that will be frequently used in the proof. A direct computation gives
K N = 1 m K 1 − δ = Θ ~ ( n − Γ T L ) , Γ T L : = δ − 1 + γ n δ . (105) \frac{K}{N}=\frac{1}{m}K^{1-\delta}=\widetilde{\Theta}(n^{-\Gamma_{\mathrm{TL}}}),\qquad\Gamma_{\mathrm{TL}}:=\frac{\delta-1+\gamma_n}{\delta}. \tag*{(105)} N K = m 1 K 1 − δ = Θ ( n − Γ TL ) , Γ TL := δ δ − 1 + γ n . (105)
Moreover, since δ > 1 \delta>1 δ > 1 and 2 s > δ 2s>\delta 2 s > δ ,
γ c − 1 1 + δ = δ ( 2 s − δ ) ( 1 + 2 s ) ( 1 + δ ) > 0. \gamma_c-\frac{1}{1+\delta}=\frac{\delta(2s-\delta)}{(1+2s)(1+\delta)}>0. γ c − 1 + δ 1 = ( 1 + 2 s ) ( 1 + δ ) δ ( 2 s − δ ) > 0.
Hence γ n > γ c > 1 / ( 1 + δ ) \gamma_n>\gamma_c>1/(1+\delta) γ n > γ c > 1/ ( 1 + δ ) , which implies
1 − γ n < δ γ n , v = 1 − γ n δ < γ n . 1-\gamma_n<\delta\gamma_n,\qquad v=\frac{1-\gamma_n}{\delta}<\gamma_n. 1 − γ n < δ γ n , v = δ 1 − γ n < γ n .
Using N ≍ n / log n N\asymp n/\log n N ≍ n / log n and m ≍ n γ n m\asymp n^{\gamma_n} m ≍ n γ n , we obtain
K m = ( N / m ) 1 / δ m ≍ n v − γ n ( log n ) − 1 / δ ⟶ 0. \frac{K}{m}=\frac{(N/m)^{1/\delta}}{m}\asymp n^{v-\gamma_n}(\log n)^{-1/\delta}\longrightarrow0. m K = m ( N / m ) 1/ δ ≍ n v − γ n ( log n ) − 1/ δ ⟶ 0.
Upper bound
Step 1: Upper bound conditioned on source dataset. Conditional on the source estimator β ^ s r \widehat{\beta}_{\mathrm{sr}} β sr , we write e 0 : = β ^ s r − β ∗ e_0:=\widehat{\beta}_{\mathrm{sr}}-\beta_* e 0 := β sr − β ∗ , which is independent of the target dataset. Recall the geometric schedule in (32) –(33) , use the shorthand ξ z , k : = ( Σ z ) k , k \xi_{z,k}:=(\Sigma_z)_{k,k} ξ z , k := ( Σ z ) k , k (z ∈ { s r , t r } z\in\{\mathrm{sr},\mathrm{tr}\} z ∈ { sr , tr } ), and define the following quantities
P : = ∏ t = 1 n ( I − η t Σ t r ) , p k : = ∏ t = 1 n ( 1 − η t ξ t r , k ) , C P : = P Σ t r P , (106) P:=\prod_{t=1}^{n}(I-\eta_t\Sigma_{\mathrm{tr}}),\qquad p_k:=\prod_{t=1}^{n}(1-\eta_t\xi_{\mathrm{tr},k}),\qquad C_P:=P\Sigma_{\mathrm{tr}}P, \tag*{(106)} P := t = 1 ∏ n ( I − η t Σ tr ) , p k := t = 1 ∏ n ( 1 − η t ξ tr , k ) , C P := P Σ tr P , (106)
as well as
J : = { k : ξ t r , k ≥ ( N η 0 ) − 1 = K − b } , J:=\{k:\xi_{\mathrm{tr},k}\ge(N\eta_0)^{-1}=K^{-b}\}, J := { k : ξ tr , k ≥ ( N η 0 ) − 1 = K − b } ,
Q : = K − b I J + ( Σ t r ) J c , Q:=K^{-b}I_J+(\Sigma_{\mathrm{tr}})_{J^c}, Q := K − b I J + ( Σ tr ) J c ,
D e f f : = ∣ J ∣ + K 2 b ∑ k ∉ J ξ t r , k 2 . (107) D_{\mathrm{eff}}:=|J|+K^{2b}\sum_{k\notin J}\xi_{\mathrm{tr},k}^{2}. \tag*{(107)} D eff := ∣ J ∣ + K 2 b k ∈ / J ∑ ξ tr , k 2 . (107)
where I J I_J I J and ( Σ t r ) J c (\Sigma_{\mathrm{tr}})_{J^c} ( Σ tr ) J c are given by
[ I J ] k k : = 1 { k ∈ J } , [ ( Σ t r ) J c ] k k : = ξ t r , k 1 { k ∉ J } , k ≥ 1. [I_J]_{kk}:=\mathbf{1}_{\{k\in J\}},\qquad[(\Sigma_{\mathrm{tr}})_{J^c}]_{kk}:=\xi_{\mathrm{tr},k}\mathbf{1}_{\{k\notin J\}},\qquad k\ge1. [ I J ] k k := 1 { k ∈ J } , [( Σ tr ) J c ] k k := ξ tr , k 1 { k ∈ / J } , k ≥ 1.
Next, we apply the last-iterate upper bound in Wu et al., 2022, Corollary 3.4. To do so, we check that the relevant assumptions hold. Wu et al., 2022, Assumption 1A and Wu et al., 2022, Assumption 2 trivially follow from Assumption 2, and the way the target labels are generated in our model. It remains to verify η 0 < 1 4 C Tr ( Σ t r ) \eta_0<\frac{1}{4C\operatorname{Tr}(\Sigma_{\mathrm{tr}})} η 0 < 4 C Tr ( Σ tr ) 1 . Note that
γ c − a b = δ ( 1 + 2 s − b ) b ( 1 + 2 s ) > 0 , \gamma_c-\frac{a}{b}=\frac{\delta(1+2s-b)}{b(1+2s)}>0, γ c − b a = b ( 1 + 2 s ) δ ( 1 + 2 s − b ) > 0 ,
and hence
η 0 = K b N = N a / δ m − b / δ ≍ n ( a − b γ n ) / δ ( log n ) − a / δ ⟶ 0. \eta_0=\frac{K^b}{N}=N^{a/\delta}m^{-b/\delta}\asymp n^{(a-b\gamma_n)/\delta}(\log n)^{-a/\delta}\longrightarrow0. η 0 = N K b = N a / δ m − b / δ ≍ n ( a − b γ n ) / δ ( log n ) − a / δ ⟶ 0.
Since Tr ( Σ t r ) = O ( 1 ) \operatorname{Tr}(\Sigma_{\mathrm{tr}})=O(1) Tr ( Σ tr ) = O ( 1 ) , there exists N 0 N_0 N 0 that only depends on C , a , b , γ n C,a,b,\gamma_n C , a , b , γ n such that for large enough n > N 0 n>N_0 n > N 0 , η 0 < min ( 1 / 2 , [ 4 C Tr ( Σ t r ) ] − 1 ) \eta_0<\min(1/2,[4C\operatorname{Tr}(\Sigma_{\mathrm{tr}})]^{-1}) η 0 < min ( 1/2 , [ 4 C Tr ( Σ tr ) ] − 1 ) , where C C C is the constant in Assumption 2.
Applying (Wu et al., 2022, Corollary 3.4, upper bound) to the initialization β ^ s r \widehat{\beta}_{\mathrm{sr}} β sr yields
E [ ∥ β ^ n − β ∗ ∥ Σ t r 2 ∣ D s r ] ≲ ∥ P e 0 ∥ Σ t r 2 + D e f f N ( σ t r 2 + ∥ e 0 ∥ Q 2 ) , (108) \mathbb{E}\left[\left\|\widehat{\beta}_n-\beta_*\right\|_{\Sigma_{\mathrm{tr}}}^{2}\mid D_{\mathrm{sr}}\right]\lesssim\|Pe_0\|_{\Sigma_{\mathrm{tr}}}^{2}+\frac{D_{\mathrm{eff}}}{N}\left(\sigma_{\mathrm{tr}}^{2}+\|e_0\|_{Q}^{2}\right), \tag*{(108)} E [ β n − β ∗ Σ tr 2 ∣ D sr ] ≲ ∥ P e 0 ∥ Σ tr 2 + N D eff ( σ tr 2 + ∥ e 0 ∥ Q 2 ) , (108)
where we denote D s r = { X s r , y s r } D_{\mathrm{sr}}=\{X_{\mathrm{sr}},y_{\mathrm{sr}}\} D sr = { X sr , y sr } , and ≲ \lesssim ≲ absorbs universal constants.
The power-law spectrum gives
D e f f = ∑ k ≥ 1 min { 1 , K 2 b ξ t r , k 2 } ≍ ∑ k ≤ K 1 + K 2 b ∑ k > K k − 2 b ≍ K + K 2 b K 1 − 2 b ≍ K . D_{\mathrm{eff}}=\sum_{k\ge1}\min\{1,K^{2b}\xi_{\mathrm{tr},k}^{2}\}\asymp\sum_{k\le K}1+K^{2b}\sum_{k>K}k^{-2b}\asymp K+K^{2b}K^{1-2b}\asymp K. D eff = k ≥ 1 ∑ min { 1 , K 2 b ξ tr , k 2 } ≍ k ≤ K ∑ 1 + K 2 b k > K ∑ k − 2 b ≍ K + K 2 b K 1 − 2 b ≍ K .
Moreover, for every k k k ,
0 ≤ [ Q ] k k = min { K − b , ξ t r , k } ≤ ξ t r , k . 0 \le[Q]_{kk} = \min\{K^{-b}, \xi_{\mathrm{tr},k}\} \le\xi_{\mathrm{tr},k}. 0 ≤ [ Q ] k k = min { K − b , ξ tr , k } ≤ ξ tr , k .
Since 0 ≤ η t ξ t r , k ≤ η 0 ∥ Σ t r ∥ o p < 1 / 2 0 \le\eta_t \xi_{\mathrm{tr},k} \le\eta_0 \lVert\boldsymbol{\Sigma}_{\mathrm{tr}} \rVert_{\mathrm{op}} < 1/2 0 ≤ η t ξ tr , k ≤ η 0 ∥ Σ tr ∥ op < 1/2 , we also have
0 ≤ [ C P ] k k = ξ t r , k p k 2 = ξ t r , k ∏ t = 1 n ( 1 − η t ξ t r , k ) 2 ≤ ξ t r , k . 0 \le[C_P]_{kk} = \xi_{\mathrm{tr},k}p_k^2 = \xi_{\mathrm{tr},k}\prod_{t=1}^{n}(1-\eta_t\xi_{\mathrm{tr},k})^2 \le\xi_{\mathrm{tr},k}. 0 ≤ [ C P ] k k = ξ tr , k p k 2 = ξ tr , k t = 1 ∏ n ( 1 − η t ξ tr , k ) 2 ≤ ξ tr , k .
Thus,
D e f f ≍ K , 0 ⪯ Q ⪯ Σ t r , 0 ⪯ C P ⪯ Σ t r . (109) D_{\mathrm{eff}} \asymp K,\qquad0 \preceq Q \preceq\boldsymbol{\Sigma}_{\mathrm{tr}},\qquad0 \preceq C_P \preceq\boldsymbol{\Sigma}_{\mathrm{tr}}. \tag*{(109)} D eff ≍ K , 0 ⪯ Q ⪯ Σ tr , 0 ⪯ C P ⪯ Σ tr . (109)
We now average over the source label noise while keeping the source design fixed. Let
H s r : = ( X s r T X s r + λ s r I ) − 1 . H_{\mathrm{sr}} := (X_{\mathrm{sr}}^{\mathsf T}X_{\mathrm{sr}}+\lambda_{\mathrm{sr}}I)^{-1}. H sr := ( X sr T X sr + λ sr I ) − 1 .
We compute
e 0 = H s r X s r T y s r − β ∗ = ( H s r X s r T X s r − I ) β ∗ + H s r X s r T ε s r = − λ s r H s r β ∗ + H s r X s r T ε s r . e_0 = H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}y_{\mathrm{sr}}-\beta_* = (H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}X_{\mathrm{sr}}-I)\beta_*+H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}\varepsilon_{\mathrm{sr}}=-\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\beta_*+H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}\varepsilon_{\mathrm{sr}}. e 0 = H sr X sr T y sr − β ∗ = ( H sr X sr T X sr − I ) β ∗ + H sr X sr T ε sr = − λ sr H sr β ∗ + H sr X sr T ε sr .
Averaging over the source noise gives
S 0 : = E ε s r [ e 0 e 0 T ∣ X s r ] = λ s r 2 H s r β ∗ β ∗ T H s r + H s r X s r T E [ ε s r ε s r T ∣ X s r ] X s r H s r = λ s r 2 H s r β ∗ β ∗ T H s r + σ s r 2 H s r X s r T X s r H s r . (110) \begin{aligned}
S_0 &:= \mathbb{E}_{\varepsilon_{\mathrm{sr}}}[e_0e_0^{\mathsf T}\mid X_{\mathrm{sr}}]\\
&= \lambda_{\mathrm{sr}}^2H_{\mathrm{sr}}\beta_*\beta_*^{\mathsf T}H_{\mathrm{sr}}+H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}\mathbb{E}[\varepsilon_{\mathrm{sr}}\varepsilon_{\mathrm{sr}}^{\mathsf T}\mid X_{\mathrm{sr}}]X_{\mathrm{sr}}H_{\mathrm{sr}}\\
&= \lambda_{\mathrm{sr}}^2H_{\mathrm{sr}}\beta_*\beta_*^{\mathsf T}H_{\mathrm{sr}}+\sigma_{\mathrm{sr}}^2H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf T}X_{\mathrm{sr}}H_{\mathrm{sr}}. \tag*{(110)}
\end{aligned} S 0 := E ε sr [ e 0 e 0 T ∣ X sr ] = λ sr 2 H sr β ∗ β ∗ T H sr + H sr X sr T E [ ε sr ε sr T ∣ X sr ] X sr H sr = λ sr 2 H sr β ∗ β ∗ T H sr + σ sr 2 H sr X sr T X sr H sr . (110)
Thus,
E ε s r [ ∥ β ^ n − β ∗ ∥ Σ t r 2 ∣ X s r ] ≲ E ε s r [ ∥ P e 0 ∥ Σ t r 2 ∣ X s r ] + D e f f N ( σ t r 2 + E ε s r [ ∥ e 0 ∥ Q 2 ∣ X s r ] ) = Tr ( C P S 0 ) + D e f f N ( σ t r 2 + Tr ( Q S 0 ) ) ≲ Tr ( C P S 0 ) + K N ( σ t r 2 + Tr ( Σ t r S 0 ) ) . (111) \begin{aligned}
\mathbb{E}_{\varepsilon_{\mathrm{sr}}}\left[\left\|\widehat{\beta}_n-\beta_*\right\|_{\boldsymbol{\Sigma}_{\mathrm{tr}}}^2\mid X_{\mathrm{sr}}\right]
&\lesssim\mathbb{E}_{\varepsilon_{\mathrm{sr}}}\left[\|Pe_0\|_{\boldsymbol{\Sigma}_{\mathrm{tr}}}^2\mid X_{\mathrm{sr}}\right]+\frac{D_{\mathrm{eff}}}{N}\left(\sigma_{\mathrm{tr}}^2+\mathbb{E}_{\varepsilon_{\mathrm{sr}}}\left[\|e_0\|_Q^2\mid X_{\mathrm{sr}}\right]\right)\\
&=\operatorname{Tr}(C_PS_0)+\frac{D_{\mathrm{eff}}}{N}\left(\sigma_{\mathrm{tr}}^2+\operatorname{Tr}(QS_0)\right)\\
&\lesssim\operatorname{Tr}(C_PS_0)+\frac{K}{N}\left(\sigma_{\mathrm{tr}}^2+\operatorname{Tr}(\boldsymbol{\Sigma}_{\mathrm{tr}}S_0)\right). \tag*{(111)}
\end{aligned} E ε sr [ β n − β ∗ Σ tr 2 ∣ X sr ] ≲ E ε sr [ ∥ P e 0 ∥ Σ tr 2 ∣ X sr ] + N D eff ( σ tr 2 + E ε sr [ ∥ e 0 ∥ Q 2 ∣ X sr ] ) = Tr ( C P S 0 ) + N D eff ( σ tr 2 + Tr ( Q S 0 ) ) ≲ Tr ( C P S 0 ) + N K ( σ tr 2 + Tr ( Σ tr S 0 ) ) . (111)
The last inequality uses D e f f ≍ K D_{\mathrm{eff}}\asymp K D eff ≍ K and
Tr ( Σ t r S 0 ) − Tr ( Q S 0 ) = Tr ( S 0 1 / 2 ( Σ t r − Q ) S 0 1 / 2 ) ≥ 0 , \operatorname{Tr}(\boldsymbol{\Sigma}_{\mathrm{tr}}S_0)-\operatorname{Tr}(QS_0)=\operatorname{Tr}\left(S_0^{1/2}(\boldsymbol{\Sigma}_{\mathrm{tr}}-Q)S_0^{1/2}\right)\ge0, Tr ( Σ tr S 0 ) − Tr ( Q S 0 ) = Tr ( S 0 1/2 ( Σ tr − Q ) S 0 1/2 ) ≥ 0 ,
since S 0 ⪰ 0 S_0\succeq0 S 0 ⪰ 0 and Q ⪯ Σ t r Q\preceq\boldsymbol{\Sigma}_{\mathrm{tr}} Q ⪯ Σ tr . The remaining task for the upper bound is therefore to compute the two terms: Tr ( C P S 0 ) \operatorname{Tr}(C_PS_0) Tr ( C P S 0 ) and Tr ( Σ t r S 0 ) \operatorname{Tr}(\boldsymbol{\Sigma}_{\mathrm{tr}}S_0) Tr ( Σ tr S 0 ) .
Step 2: Deterministic equivalents for the source estimator. Choose λ s r = κ m 1 − a \lambda_{\mathrm{sr}}=\kappa m^{1-a} λ sr = κ m 1 − a and let τ : = τ s r \tau:=\tau_{\mathrm{sr}} τ := τ sr be its effective regularization. By Propositions 4 and 5,
τ ≍ m − a , c s r : = { m − Tr [ Σ s r 2 ( Σ s r + τ I ) − 2 ] } − 1 ≍ m − 1 . (112) \tau\asymp m^{-a},\qquad c_{\mathrm{sr}}:=\left\{m-\operatorname{Tr}\left[\boldsymbol{\Sigma}_{\mathrm{sr}}^2(\boldsymbol{\Sigma}_{\mathrm{sr}}+\tau I)^{-2}\right]\right\}^{-1}\asymp m^{-1}. \tag*{(112)} τ ≍ m − a , c sr := { m − Tr [ Σ sr 2 ( Σ sr + τ I ) − 2 ] } − 1 ≍ m − 1 . (112)
For a deterministic diagonal positive semidefinite C C C , define
W s r : = Σ s r ( Σ s r + τ I ) − 2 , W_{\mathrm{sr}}:=\boldsymbol{\Sigma}_{\mathrm{sr}}(\boldsymbol{\Sigma}_{\mathrm{sr}}+\tau I)^{-2}, W sr := Σ sr ( Σ sr + τ I ) − 2 ,
A ( C ) : = τ 2 β ∗ T ( Σ s r + τ I ) − 1 C ( Σ s r + τ I ) − 1 β ∗ , A(C):=\tau^2\beta_*^{\mathsf T}(\boldsymbol{\Sigma}_{\mathrm{sr}}+\tau I)^{-1}C(\boldsymbol{\Sigma}_{\mathrm{sr}}+\tau I)^{-1}\beta_*, A ( C ) := τ 2 β ∗ T ( Σ sr + τ I ) − 1 C ( Σ sr + τ I ) − 1 β ∗ ,
L ( C ) : = A ( C ) + c s r ( σ s r 2 + τ 2 β ∗ T W s r β ∗ ) Tr ( C W s r ) . (113) L(C):=A(C)+c_{\mathrm{sr}}\left(\sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf T}W_{\mathrm{sr}}\beta_*\right)\operatorname{Tr}(CW_{\mathrm{sr}}). \tag*{(113)} L ( C ) := A ( C ) + c sr ( σ sr 2 + τ 2 β ∗ T W sr β ∗ ) Tr ( C W sr ) . (113)
We further define the following quantities that will appear in the approximation error
χ m : = τ 2 β ∗ T ( Σ s r + τ I ) − 1 β ∗ , ε m : = ρ λ s r ( m ) 6 log 5 / 2 m m . (114) \chi_m:=\tau^2\beta_*^{\mathsf T}(\boldsymbol{\Sigma}_{\mathrm{sr}}+\tau I)^{-1}\beta_*,\qquad\varepsilon_m:=\frac{\rho_{\lambda_{\mathrm{sr}}}(m)^6\log^{5/2}m}{\sqrt{m}}. \tag*{(114)} χ m := τ 2 β ∗ T ( Σ sr + τ I ) − 1 β ∗ , ε m := m ρ λ sr ( m ) 6 log 5/2 m . (114)
Next we apply Wu et al., 2026a's Theorem 6, equation (40), and Theorem 5, equation (37) to obtain the deterministic equivalence. We verify assumptions in Wu et al., 2026a's Theorem 6, and Theorem 5 at the chosen source regularization. By definition in (34) ,
ρ λ s r ( m ) = 1 + O ( m 1 − a log m λ s r ) = O ( log m ) . \rho_{\lambda_{\mathrm{sr}}}(m)=1+O\left(\frac{m^{1-a}\log m}{\lambda_{\mathrm{sr}}}\right)=O(\log m). ρ λ sr ( m ) = 1 + O ( λ sr m 1 − a log m ) = O ( log m ) .
To see this, take j = ⌊ ϑ m ⌋ j=\lfloor\vartheta m\rfloor j = ⌊ ϑ m ⌋ , with the fixed ϑ ∈ ( 0 , 1 / 2 ) \vartheta\in(0,1/2) ϑ ∈ ( 0 , 1/2 ) appearing in the definition of ρ \rho ρ , and use
∑ k ≥ j ξ s r , k ≍ m 1 − a , ∑ k ≥ j ξ s r , k ξ s r , j ≍ m . \sum_{k\ge j}\xi_{\mathrm{sr},k}\asymp m^{1-a},\qquad\frac{\sum_{k\ge j}\xi_{\mathrm{sr},k}}{\xi_{\mathrm{sr},j}}\asymp m. k ≥ j ∑ ξ sr , k ≍ m 1 − a , ξ sr , j ∑ k ≥ j ξ sr , k ≍ m .
Consequently,
ε m = O ( m − 1 / 2 log 17 / 2 m ) = o ( 1 ) , χ m ≤ τ ∥ β ∗ ∥ 2 2 = O ( m − a ) . (115) \varepsilon_m=O(m^{-1/2}\log^{17/2}m)=o(1),\qquad\chi_m\leq\tau\lVert\beta_*\rVert_2^2=O(m^{-a}). \tag*{(115)} ε m = O ( m − 1/2 log 17/2 m ) = o ( 1 ) , χ m ≤ τ ∥ β ∗ ∥ 2 2 = O ( m − a ) . (115)
Thus, for large enough m m m , we have that λ s r ρ λ s r ( m ) ≳ m 1 − a \lambda_{\mathrm{sr}}\rho_{\lambda_{\mathrm{sr}}}(m)\gtrsim m^{1-a} λ sr ρ λ sr ( m ) ≳ m 1 − a dominates ∥ Σ s r ∥ o p m − K 0 \lVert\Sigma_{\mathrm{sr}}\rVert_{\mathrm{op}}m^{-K_0} ∥ Σ sr ∥ op m − K 0 for fixed K 0 > a − 1 K_0>a-1 K 0 > a − 1 , and ρ λ s r ( m ) 5 / 2 log 3 / 2 m = o ( m ) \rho_{\lambda_{\mathrm{sr}}}(m)^{5/2}\log^{3/2}m=o(\sqrt{m}) ρ λ sr ( m ) 5/2 log 3/2 m = o ( m ) , which implies that the assumptions holds.
Applying (Wu et al., 2026a, Theorem 6, equation (40), and Theorem 5, equation (37)) gives
∣ Tr ( C S 0 ) − L ( C ) ∣ ≤ C D ε m [ L ( C ) + χ m ∥ C ( Σ s r + τ I ) − 1 ∥ o p ] , (116) \left|\operatorname{Tr}(CS_0)-L(C)\right|\leq C_D\varepsilon_m\left[L(C)+\chi_m\left\lVert C(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\rVert_{\mathrm{op}}\right], \tag*{(116)} ∣ Tr ( C S 0 ) − L ( C ) ∣ ≤ C D ε m [ L ( C ) + χ m C ( Σ sr + τ I ) − 1 op ] , (116)
with probability at least 1 − C D m − D 1-C_Dm^{-D} 1 − C D m − D over the source design, for every fixed D > 0 D>0 D > 0 .
Step 3: Evaluation of the deterministic equivalents and approximation errors. We first control Tr ( C P S 0 ) \operatorname{Tr}(C_PS_0) Tr ( C P S 0 ) through its deterministic equivalent
L ( C P ) = A ( C P ) + c s r ( σ s r 2 + τ 2 β ∗ T W s r β ∗ ) Tr ( C P W s r ) . L(C_P)=A(C_P)+c_{\mathrm{sr}}\left(\sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf T}W_{\mathrm{sr}}\beta_*\right)\operatorname{Tr}(C_PW_{\mathrm{sr}}). L ( C P ) = A ( C P ) + c sr ( σ sr 2 + τ 2 β ∗ T W sr β ∗ ) Tr ( C P W sr ) .
Since c s r ≍ m − 1 c_{\mathrm{sr}}\asymp m^{-1} c sr ≍ m − 1 and K / N = m − 1 K 1 − δ K/N=m^{-1}K^{1-\delta} K / N = m − 1 K 1 − δ , we will show that
Tr ( C P W s r ) ≍ K 1 − δ , A ( C P ) = o ( K / N ) , τ 2 β ∗ T W s r β ∗ = O ( m − a ) , \operatorname{Tr}(C_PW_{\mathrm{sr}})\asymp K^{1-\delta},\qquad A(C_P)=o(K/N),\qquad\tau^2\beta_*^{\mathsf T}W_{\mathrm{sr}}\beta_*=O(m^{-a}), Tr ( C P W sr ) ≍ K 1 − δ , A ( C P ) = o ( K / N ) , τ 2 β ∗ T W sr β ∗ = O ( m − a ) ,
∥ C P ( Σ s r + τ I ) − 1 ∥ o p ≲ K − δ , χ m ≲ m − a . \left\lVert C_P(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\rVert_{\mathrm{op}}\lesssim K^{-\delta},\qquad\chi_m\lesssim m^{-a}. C P ( Σ sr + τ I ) − 1 op ≲ K − δ , χ m ≲ m − a .
For the term Tr ( C P W s r ) \operatorname{Tr}(C_PW_{\mathrm{sr}}) Tr ( C P W sr ) , we first estimate the order of p k p_k p k . The geometric schedule satisfies
K b = N η 0 ≤ ∑ t = 1 n η t ≤ 2 N η 0 = 2 K b . K^b=N\eta_0\leq\sum_{t=1}^{n}\eta_t\leq2N\eta_0=2K^b. K b = N η 0 ≤ t = 1 ∑ n η t ≤ 2 N η 0 = 2 K b .
Since 0 ≤ η t ξ t r , k < 1 / 2 0\leq\eta_t\xi_{\mathrm{tr},k}<1/2 0 ≤ η t ξ tr , k < 1/2 , the inequalities − 2 x ≤ log ( 1 − x ) ≤ − x -2x\leq\log(1-x)\leq-x − 2 x ≤ log ( 1 − x ) ≤ − x give
− 2 ξ t r , k ∑ t = 1 n η t ≤ log p k ≤ − ξ t r , k ∑ t = 1 n η t . -2\xi_{\mathrm{tr},k}\sum_{t=1}^{n}\eta_t\leq\log p_k\leq-\xi_{\mathrm{tr},k}\sum_{t=1}^{n}\eta_t. − 2 ξ tr , k t = 1 ∑ n η t ≤ log p k ≤ − ξ tr , k t = 1 ∑ n η t .
Using ξ t r , k ≍ k − b \xi_{\mathrm{tr},k}\asymp k^{-b} ξ tr , k ≍ k − b , we obtain
e − C 1 ( K / k ) b ≤ p k ≤ e − c 1 ( K / k ) b . (117) e^{-C_1(K/k)^b}\leq p_k\leq e^{-c_1(K/k)^b}. \tag*{(117)} e − C 1 ( K / k ) b ≤ p k ≤ e − c 1 ( K / k ) b . (117)
for fixed positive constants c 1 , C 1 c_1,C_1 c 1 , C 1 . Combining this estimate with ξ s r , k ≍ k − a \xi_{\mathrm{sr},k}\asymp k^{-a} ξ sr , k ≍ k − a and τ ≍ m − a \tau\asymp m^{-a} τ ≍ m − a , we have
Tr ( C P W s r ) = ∑ k ≥ 1 p k 2 ξ t r , k ξ s r , k ( ξ s r , k + τ ) 2 \operatorname{Tr}(C_PW_{\mathrm{sr}})=\sum_{k\geq1}\frac{p_k^2\xi_{\mathrm{tr},k}\xi_{\mathrm{sr},k}}{(\xi_{\mathrm{sr},k}+\tau)^2} Tr ( C P W sr ) = k ≥ 1 ∑ ( ξ sr , k + τ ) 2 p k 2 ξ tr , k ξ sr , k
≲ ∑ k ≤ m e − c ( K / k ) b k − δ + m 2 a ∑ k > m k − a − b \lesssim\sum_{k\leq m}e^{-c(K/k)^b}k^{-\delta}+m^{2a}\sum_{k>m}k^{-a-b} ≲ k ≤ m ∑ e − c ( K / k ) b k − δ + m 2 a k > m ∑ k − a − b
≲ K 1 − δ + m 1 − δ ≲ K 1 − δ . \lesssim K^{1-\delta}+m^{1-\delta}\lesssim K^{1-\delta}. ≲ K 1 − δ + m 1 − δ ≲ K 1 − δ .
Indeed, boundedness of x δ e − c x b x^\delta e^{-cx^b} x δ e − c x b and δ > 1 \delta>1 δ > 1 give
∑ k ≤ m e − c ( K / k ) b k − δ ≲ K − δ ∑ k ≤ K 1 + ∑ k > K k − δ ≲ K 1 − δ . \sum_{k\leq m}e^{-c(K/k)^b}k^{-\delta}\lesssim K^{-\delta}\sum_{k\leq K}1+\sum_{k>K}k^{-\delta}\lesssim K^{1-\delta}. k ≤ m ∑ e − c ( K / k ) b k − δ ≲ K − δ k ≤ K ∑ 1 + k > K ∑ k − δ ≲ K 1 − δ .
Also, m 2 a ∑ k > m k − a − b ≍ m 1 − δ m^{2a}\sum_{k>m}k^{-a-b}\asymp m^{1-\delta} m 2 a ∑ k > m k − a − b ≍ m 1 − δ , and m 1 − δ = o ( K 1 − δ ) m^{1-\delta}=o(K^{1-\delta}) m 1 − δ = o ( K 1 − δ ) because K = o ( m ) K=o(m) K = o ( m ) and δ > 1 \delta>1 δ > 1 .
For the matching lower bound, restrict to ⌈ K ⌉ ≤ k ≤ ⌊ 2 K ⌋ \lceil K\rceil\leq k\leq\lfloor2K\rfloor ⌈ K ⌉ ≤ k ≤ ⌊ 2 K ⌋ . On this range, p k ≳ 1 p_k\gtrsim1 p k ≳ 1 and τ / ξ s r , k ≍ ( K / m ) a = o ( 1 ) \tau/\xi_{\mathrm{sr},k}\asymp(K/m)^a=o(1) τ / ξ sr , k ≍ ( K / m ) a = o ( 1 ) . Consequently,
Tr ( C P W s r ) ≳ ∑ k = ⌈ K ⌉ ⌊ 2 K ⌋ ξ t r , k ξ s r , k ≲ ∑ k = ⌈ K ⌉ ⌊ 2 K ⌋ k − δ ≲ K 1 − δ . \operatorname{Tr}(C_PW_{\mathrm{sr}})\gtrsim\sum_{k=\lceil K\rceil}^{\lfloor2K\rfloor}\frac{\xi_{\mathrm{tr},k}}{\xi_{\mathrm{sr},k}}\lesssim\sum_{k=\lceil K\rceil}^{\lfloor2K\rfloor}k^{-\delta}\lesssim K^{1-\delta}. Tr ( C P W sr ) ≳ k = ⌈ K ⌉ ∑ ⌊ 2 K ⌋ ξ sr , k ξ tr , k ≲ k = ⌈ K ⌉ ∑ ⌊ 2 K ⌋ k − δ ≲ K 1 − δ .
For the term ∥ C P ( Σ s r + τ I ) − 1 ∥ o p \left\lVert C_P(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\rVert_{\mathrm{op}} C P ( Σ sr + τ I ) − 1 op , a direct computation gives
∥ C P ( Σ s r + τ I ) − 1 ∥ o p = sup k p k 2 ξ t r , k ξ s r , k + τ ≤ sup k p k 2 ξ t r , k ξ s r , k ≲ sup k e − c ( K / k ) b k − δ = K − δ sup k ( K / k ) δ e − c ( K / k ) b ≲ K − δ . \left\lVert C_P(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\rVert_{\mathrm{op}}
=\sup_k\frac{p_k^2\xi_{\mathrm{tr},k}}{\xi_{\mathrm{sr},k}+\tau}
\leq\sup_k\frac{p_k^2\xi_{\mathrm{tr},k}}{\xi_{\mathrm{sr},k}}
\lesssim\sup_k e^{-c(K/k)^b}k^{-\delta}
=K^{-\delta}\sup_k(K/k)^\delta e^{-c(K/k)^b}\lesssim K^{-\delta}. C P ( Σ sr + τ I ) − 1 op = k sup ξ sr , k + τ p k 2 ξ tr , k ≤ k sup ξ sr , k p k 2 ξ tr , k ≲ k sup e − c ( K / k ) b k − δ = K − δ k sup ( K / k ) δ e − c ( K / k ) b ≲ K − δ .
Together with the trace estimate, this gives
1 m Tr ( C P W s r ) ≍ 1 m K 1 − δ , ∥ C P ( Σ s r + τ I ) − 1 ∥ o p ≲ K − δ . (118) \frac{1}{m}\operatorname{Tr}(C_P W_{\mathrm{sr}}) \asymp\frac{1}{m}K^{1-\delta}, \qquad\left\|C_P(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\|_{\mathrm{op}} \lesssim K^{-\delta}. \tag*{(118)} m 1 Tr ( C P W sr ) ≍ m 1 K 1 − δ , C P ( Σ sr + τ I ) − 1 op ≲ K − δ . (118)
For the remaining terms that depend on β ∗ \beta_* β ∗ , by (103) ,
χ m = τ 2 β ∗ T ( Σ s r + τ I ) − 1 β ∗ ≤ τ ∥ β ∗ ∥ 2 2 = O ( m − a ) . \chi_m=\tau^2\beta_*^{\mathsf{T}}(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\beta_* \leq\tau\|\beta_*\|_2^2=O(m^{-a}). χ m = τ 2 β ∗ T ( Σ sr + τ I ) − 1 β ∗ ≤ τ ∥ β ∗ ∥ 2 2 = O ( m − a ) .
For any diagonal positive semidefinite C C C ,
A ( C ) = τ 2 ∑ k ≥ 1 β ∗ [ k ] 2 [ C ] k k ( ξ s r , k + τ ) 2 ≤ ( τ 2 ∑ k ≥ 1 β ∗ [ k ] 2 ξ s r , k + τ ) sup k [ C ] k k ξ s r , k + τ = χ m ∥ C ( Σ s r + τ I ) − 1 ∥ o p . (119) A(C)=\tau^2\sum_{k\geq1}\frac{\beta_*[k]^2[C]_{kk}}{(\xi_{\mathrm{sr},k}+\tau)^2}\leq\left(\tau^2\sum_{k\geq1}\frac{\beta_*[k]^2}{\xi_{\mathrm{sr},k}+\tau}\right)\sup_k\frac{[C]_{kk}}{\xi_{\mathrm{sr},k}+\tau}=\chi_m\left\|C(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\|_{\mathrm{op}}. \tag*{(119)} A ( C ) = τ 2 k ≥ 1 ∑ ( ξ sr , k + τ ) 2 β ∗ [ k ] 2 [ C ] k k ≤ ( τ 2 k ≥ 1 ∑ ξ sr , k + τ β ∗ [ k ] 2 ) k sup ξ sr , k + τ [ C ] k k = χ m C ( Σ sr + τ I ) − 1 op . (119)
Applying this inequality to C P C_P C P yields
A ( C P ) ≲ m − a K − δ , A ( C P ) K / N ≲ m − a K − δ m − 1 K 1 − δ = m 1 − a K ⟶ 0. A(C_P)\lesssim m^{-a}K^{-\delta},\qquad\frac{A(C_P)}{K/N}\lesssim\frac{m^{-a}K^{-\delta}}{m^{-1}K^{1-\delta}}=\frac{m^{1-a}}{K}\longrightarrow0. A ( C P ) ≲ m − a K − δ , K / N A ( C P ) ≲ m − 1 K 1 − δ m − a K − δ = K m 1 − a ⟶ 0.
Finally, the inequality ( x + τ ) 2 ≥ 4 x τ (x+\tau)^2\geq4x\tau ( x + τ ) 2 ≥ 4 xτ gives
0 ≤ τ 2 β ∗ T W s r β ∗ = ∑ k ≥ 1 τ 2 ξ s r , k ( ξ s r , k + τ ) 2 β ∗ [ k ] 2 ≤ τ 4 ∥ β ∗ ∥ 2 2 = O ( m − a ) . 0\leq\tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*=\sum_{k\geq1}\frac{\tau^2\xi_{\mathrm{sr},k}}{(\xi_{\mathrm{sr},k}+\tau)^2}\beta_*[k]^2\leq\frac{\tau}{4}\|\beta_*\|_2^2=O(m^{-a}). 0 ≤ τ 2 β ∗ T W sr β ∗ = k ≥ 1 ∑ ( ξ sr , k + τ ) 2 τ 2 ξ sr , k β ∗ [ k ] 2 ≤ 4 τ ∥ β ∗ ∥ 2 2 = O ( m − a ) .
Since σ s r 2 \sigma_{\mathrm{sr}}^2 σ sr 2 is fixed and strictly positive,
σ s r 2 + τ 2 β ∗ T W s r β ∗ = σ s r 2 + O ( m − a ) ≍ 1. \sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*=\sigma_{\mathrm{sr}}^2+O(m^{-a})\asymp1. σ sr 2 + τ 2 β ∗ T W sr β ∗ = σ sr 2 + O ( m − a ) ≍ 1.
Substituting the above estimates into L ( C P ) L(C_P) L ( C P ) gives
L ( C P ) = A ( C P ) + c s r ( σ s r 2 + τ 2 β ∗ T W s r β ∗ ) Tr ( C P W s r ) = o ( K / N ) + Θ ( m − 1 K 1 − δ ) ≍ K / N . L(C_P)=A(C_P)+c_{\mathrm{sr}}\left(\sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*\right)\operatorname{Tr}(C_PW_{\mathrm{sr}})=o(K/N)+\Theta(m^{-1}K^{1-\delta})\asymp K/N. L ( C P ) = A ( C P ) + c sr ( σ sr 2 + τ 2 β ∗ T W sr β ∗ ) Tr ( C P W sr ) = o ( K / N ) + Θ ( m − 1 K 1 − δ ) ≍ K / N .
For the approximation error, we have
∣ Tr ( C P S 0 ) − L ( C P ) ∣ K / N ≲ ϵ m [ L ( C P ) K / N + χ m K − δ m − 1 K 1 − δ ] ≲ ϵ m ( 1 + m χ m K ) ≲ ϵ m ( 1 + m 1 − a K ) = o ( 1 ) . (120) \frac{\left|\operatorname{Tr}(C_PS_0)-L(C_P)\right|}{K/N}\lesssim\epsilon_m\left[\frac{L(C_P)}{K/N}+\frac{\chi_mK^{-\delta}}{m^{-1}K^{1-\delta}}\right]\lesssim\epsilon_m\left(1+\frac{m\chi_m}{K}\right)\lesssim\epsilon_m\left(1+\frac{m^{1-a}}{K}\right)=o(1). \tag*{(120)} K / N ∣ Tr ( C P S 0 ) − L ( C P ) ∣ ≲ ϵ m [ K / N L ( C P ) + m − 1 K 1 − δ χ m K − δ ] ≲ ϵ m ( 1 + K m χ m ) ≲ ϵ m ( 1 + K m 1 − a ) = o ( 1 ) . (120)
Here we used ϵ m = o ( 1 ) \epsilon_m=o(1) ϵ m = o ( 1 ) and m 1 − a / K → 0 m^{1-a}/K\to0 m 1 − a / K → 0 . Since L ( C P ) ≍ K / N L(C_P)\asymp K/N L ( C P ) ≍ K / N , it follows that
Tr ( C P S 0 ) = L ( C P ) ( 1 + o ( 1 ) ) ≍ K N . (121) \operatorname{Tr}(C_PS_0)=L(C_P)(1+o(1))\asymp\frac{K}{N}. \tag*{(121)} Tr ( C P S 0 ) = L ( C P ) ( 1 + o ( 1 )) ≍ N K . (121)
Next, we control Tr ( Σ t r S 0 ) \operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0) Tr ( Σ tr S 0 ) . Recall that
L ( Σ t r ) = A ( Σ t r ) + c s r ( σ s r 2 + τ 2 β ∗ T W s r β ∗ ) Tr ( Σ t r W s r ) . L(\Sigma_{\mathrm{tr}})=A(\Sigma_{\mathrm{tr}})+c_{\mathrm{sr}}\left(\sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*\right)\operatorname{Tr}(\Sigma_{\mathrm{tr}}W_{\mathrm{sr}}). L ( Σ tr ) = A ( Σ tr ) + c sr ( σ sr 2 + τ 2 β ∗ T W sr β ∗ ) Tr ( Σ tr W sr ) .
We already know that c s r ≍ m − 1 c_{\mathrm{sr}}\asymp m^{-1} c sr ≍ m − 1 , τ 2 β ∗ T W s r β ∗ = O ( m − a ) \tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*=O(m^{-a}) τ 2 β ∗ T W sr β ∗ = O ( m − a ) , and χ m = O ( m − a ) \chi_m=O(m^{-a}) χ m = O ( m − a ) . It therefore suffices to establish
Tr ( Σ t r W s r ) = O ( 1 ) , ∥ Σ t r ( Σ s r + τ I ) − 1 ∥ o p = O ( 1 ) , A ( Σ t r ) = O ( 1 ) , \operatorname{Tr}(\Sigma_{\mathrm{tr}}W_{\mathrm{sr}})=O(1),\qquad\left\|\Sigma_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\|_{\mathrm{op}}=O(1),\qquad A(\Sigma_{\mathrm{tr}})=O(1), Tr ( Σ tr W sr ) = O ( 1 ) , Σ tr ( Σ sr + τ I ) − 1 op = O ( 1 ) , A ( Σ tr ) = O ( 1 ) ,
and then apply (116) .
A direct computation gives
Tr ( Σ t r W s r ) = ∑ k ≥ 1 ξ t r , k ξ s r , k ( ξ s r , k + τ ) 2 ≤ ∑ k ≥ 1 ξ t r , k ξ s r , k ≲ ∑ k ≥ 1 k − δ = O ( 1 ) , \operatorname{Tr}(\Sigma_{\mathrm{tr}}W_{\mathrm{sr}})=\sum_{k\geq1}\frac{\xi_{\mathrm{tr},k}\xi_{\mathrm{sr},k}}{(\xi_{\mathrm{sr},k}+\tau)^2}\leq\sum_{k\geq1}\frac{\xi_{\mathrm{tr},k}}{\xi_{\mathrm{sr},k}}\lesssim\sum_{k\geq1}k^{-\delta}=O(1), Tr ( Σ tr W sr ) = k ≥ 1 ∑ ( ξ sr , k + τ ) 2 ξ tr , k ξ sr , k ≤ k ≥ 1 ∑ ξ sr , k ξ tr , k ≲ k ≥ 1 ∑ k − δ = O ( 1 ) ,
and
A ( Σ t r ) = ∑ k ≥ 1 ( τ ξ s r , k + τ ) 2 ξ t r , k β ∗ [ k ] 2 ≤ ∑ k ≥ 1 ξ t r , k β ∗ [ k ] 2 = ∥ β ∗ ∥ Σ t r 2 ≤ ∥ Σ t r ∥ o p ∥ β ∗ ∥ 2 2 = O ( 1 ) . A(\Sigma_{\mathrm{tr}})=\sum_{k\geq1}\left(\frac{\tau}{\xi_{\mathrm{sr},k}+\tau}\right)^2\xi_{\mathrm{tr},k}\beta_*[k]^2\leq\sum_{k\geq1}\xi_{\mathrm{tr},k}\beta_*[k]^2=\|\beta_*\|_{\Sigma_{\mathrm{tr}}}^2\leq\|\Sigma_{\mathrm{tr}}\|_{\mathrm{op}}\|\beta_*\|_2^2=O(1). A ( Σ tr ) = k ≥ 1 ∑ ( ξ sr , k + τ τ ) 2 ξ tr , k β ∗ [ k ] 2 ≤ k ≥ 1 ∑ ξ tr , k β ∗ [ k ] 2 = ∥ β ∗ ∥ Σ tr 2 ≤ ∥ Σ tr ∥ op ∥ β ∗ ∥ 2 2 = O ( 1 ) .
Substituting these bounds into the deterministic equivalent, we obtain
L ( Σ t r ) = A ( Σ t r ) + c s r ( σ s r 2 + τ 2 β ∗ T W s r β ∗ ) Tr ( Σ t r W s r ) = A ( Σ t r ) + O ( m − 1 ) = O ( 1 ) . L(\Sigma_{\mathrm{tr}})=A(\Sigma_{\mathrm{tr}})+c_{\mathrm{sr}}\left(\sigma_{\mathrm{sr}}^2+\tau^2\beta_*^{\mathsf{T}}W_{\mathrm{sr}}\beta_*\right)\operatorname{Tr}(\Sigma_{\mathrm{tr}}W_{\mathrm{sr}})=A(\Sigma_{\mathrm{tr}})+O(m^{-1})=O(1). L ( Σ tr ) = A ( Σ tr ) + c sr ( σ sr 2 + τ 2 β ∗ T W sr β ∗ ) Tr ( Σ tr W sr ) = A ( Σ tr ) + O ( m − 1 ) = O ( 1 ) .
Moreover,
χ m ∥ Σ t r ( Σ s r + τ I ) − 1 ∥ o p = O ( m − a ) . \chi_m\left\|\Sigma_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\|_{\mathrm{op}}=O(m^{-a}). χ m Σ tr ( Σ sr + τ I ) − 1 op = O ( m − a ) .
Hence, on the same event, (116) yields
∣ Tr ( Σ t r S 0 ) − L ( Σ t r ) ∣ ≤ C D ϵ m [ L ( Σ t r ) + χ m ∥ Σ t r ( Σ s r + τ I ) − 1 ∥ o p ] = O ( ϵ m ) = o ( 1 ) . \left|\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)-L(\Sigma_{\mathrm{tr}})\right|\leq C_D\epsilon_m\left[L(\Sigma_{\mathrm{tr}})+\chi_m\left\|\Sigma_{\mathrm{tr}}(\Sigma_{\mathrm{sr}}+\tau I)^{-1}\right\|_{\mathrm{op}}\right]=O(\epsilon_m)=o(1). ∣ Tr ( Σ tr S 0 ) − L ( Σ tr ) ∣ ≤ C D ϵ m [ L ( Σ tr ) + χ m Σ tr ( Σ sr + τ I ) − 1 op ] = O ( ϵ m ) = o ( 1 ) .
Consequently,
Tr ( Σ t r S 0 ) = L ( Σ t r ) + O ( ϵ m ) = O ( 1 ) . (122) \operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)=L(\Sigma_{\mathrm{tr}})+O(\epsilon_m)=O(1). \tag*{(122)} Tr ( Σ tr S 0 ) = L ( Σ tr ) + O ( ϵ m ) = O ( 1 ) . (122)
Step 4: Compute the expectation. For every source design X s r X_{\mathrm{sr}} X sr ,
∥ λ s r H s r ∥ o p ≤ 1 , H s r X s r T X s r H s r ⪯ 1 4 λ s r I . \lVert\lambda_{\mathrm{sr}}H_{\mathrm{sr}}\rVert_{\mathrm{op}} \le1,\qquad H_{\mathrm{sr}}X_{\mathrm{sr}}^{\mathsf{T}}X_{\mathrm{sr}}H_{\mathrm{sr}} \preceq\frac{1}{4\lambda_{\mathrm{sr}}}I. ∥ λ sr H sr ∥ op ≤ 1 , H sr X sr T X sr H sr ⪯ 4 λ sr 1 I .
Therefore, for either C = C P C=C_P C = C P or C = Σ t r C=\Sigma_{\mathrm{tr}} C = Σ tr ,
0 ≤ Tr ( C S 0 ) ≤ ∥ Σ t r ∥ o p ∥ β ∗ ∥ 2 2 + σ s r 2 Tr ( Σ t r ) 4 λ s r = O ( 1 + m a − 1 ) . 0\leq\operatorname{Tr}(CS_0)\leq\lVert\Sigma_{\mathrm{tr}}\rVert_{\mathrm{op}}\lVert\beta_*\rVert_2^2+\frac{\sigma_{\mathrm{sr}}^2\operatorname{Tr}(\Sigma_{\mathrm{tr}})}{4\lambda_{\mathrm{sr}}}=O(1+m^{a-1}). 0 ≤ Tr ( C S 0 ) ≤ ∥ Σ tr ∥ op ∥ β ∗ ∥ 2 2 + 4 λ sr σ sr 2 Tr ( Σ tr ) = O ( 1 + m a − 1 ) .
Choose the exponent D > a − 1 + Γ T L / γ n D>a-1+\Gamma_{\mathrm{TL}}/\gamma_n D > a − 1 + Γ TL / γ n in (116), and recall that K N = Θ ~ ( n − Γ T L ) \frac{K}{N}=\widetilde{\Theta}(n^{-\Gamma_{\mathrm{TL}}}) N K = Θ ( n − Γ TL ) . Let E \mathcal{E} E denote the common source-design event on which (121) and (122) hold, so that P ( E c ) ≤ C D m − D \mathbb{P}(\mathcal{E}^{c})\leq C_Dm^{-D} P ( E c ) ≤ C D m − D . For either C = C P C=C_P C = C P or C = Σ t r C=\Sigma_{\mathrm{tr}} C = Σ tr , the uniform bound above gives
E X s r [ Tr ( C S 0 ) 1 E c ] ≲ ( 1 + m a − 1 ) P ( E c ) ≲ m a − 1 − D = o ( K / N ) . \mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(CS_0)\mathbf{1}_{\mathcal{E}^{c}}\right]\lesssim(1+m^{a-1})\mathbb{P}(\mathcal{E}^{c})\lesssim m^{a-1-D}=o(K/N). E X sr [ Tr ( C S 0 ) 1 E c ] ≲ ( 1 + m a − 1 ) P ( E c ) ≲ m a − 1 − D = o ( K / N ) .
Indeed, by the choice D > a − 1 + Γ T L / γ n D>a-1+\Gamma_{\mathrm{TL}}/\gamma_n D > a − 1 + Γ TL / γ n ,
m a − 1 − D K / N = O ~ ( n γ n ( a − 1 − D ) + Γ T L ) ⟶ 0. \frac{m^{a-1-D}}{K/N}=\widetilde{O}\left(n^{\gamma_n(a-1-D)+\Gamma_{\mathrm{TL}}}\right)\longrightarrow0. K / N m a − 1 − D = O ( n γ n ( a − 1 − D ) + Γ TL ) ⟶ 0.
We now split the expectation over E \mathcal{E} E and E c \mathcal{E}^{c} E c . The upper bound on E \mathcal{E} E yields
E X s r Tr ( C P S 0 ) = E X s r [ Tr ( C P S 0 ) 1 E ] + E X s r [ Tr ( C P S 0 ) 1 E c ] ≲ K N + o ( K / N ) ≲ K N . \mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(C_PS_0)=\mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(C_PS_0)\mathbf{1}_{\mathcal{E}}\right]+\mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(C_PS_0)\mathbf{1}_{\mathcal{E}^{c}}\right]\lesssim\frac{K}{N}+o(K/N)\lesssim\frac{K}{N}. E X sr Tr ( C P S 0 ) = E X sr [ Tr ( C P S 0 ) 1 E ] + E X sr [ Tr ( C P S 0 ) 1 E c ] ≲ N K + o ( K / N ) ≲ N K .
Conversely, nonnegativity and the lower bound on E \mathcal{E} E give
E X s r Tr ( C P S 0 ) ≥ E X s r [ Tr ( C P S 0 ) 1 E ] ≳ K N P ( E ) ≍ K N . \mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(C_PS_0)\geq\mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(C_PS_0)\mathbf{1}_{\mathcal{E}}\right]\gtrsim\frac{K}{N}\mathbb{P}(\mathcal{E})\asymp\frac{K}{N}. E X sr Tr ( C P S 0 ) ≥ E X sr [ Tr ( C P S 0 ) 1 E ] ≳ N K P ( E ) ≍ N K .
Thus E X s r Tr ( C P S 0 ) ≍ K / N \mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(C_PS_0)\asymp K/N E X sr Tr ( C P S 0 ) ≍ K / N . Similarly,
E X s r Tr ( Σ t r S 0 ) = E X s r [ Tr ( Σ t r S 0 ) 1 E ] + E X s r [ Tr ( Σ t r S 0 ) 1 E c ] ≤ O ( 1 ) + o ( K / N ) = O ( 1 ) . \mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)=\mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)\mathbf{1}_{\mathcal{E}}\right]+\mathbb{E}_{X_{\mathrm{sr}}}\left[\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)\mathbf{1}_{\mathcal{E}^{c}}\right]\leq O(1)+o(K/N)=O(1). E X sr Tr ( Σ tr S 0 ) = E X sr [ Tr ( Σ tr S 0 ) 1 E ] + E X sr [ Tr ( Σ tr S 0 ) 1 E c ] ≤ O ( 1 ) + o ( K / N ) = O ( 1 ) .
Finally, by the definition S 0 = E [ e 0 e 0 T ∣ X s r ] S_0=\mathbb{E}[e_0e_0^{\mathsf{T}}\mid X_{\mathrm{sr}}] S 0 = E [ e 0 e 0 T ∣ X sr ] ,
E ∥ P e 0 ∥ Σ t r 2 = E X s r [ E [ e 0 T C P e 0 ∣ X s r ] ] = E X s r Tr ( C P S 0 ) ≲ K / N , \mathbb{E}\lVert Pe_0\rVert_{\Sigma_{\mathrm{tr}}}^2=\mathbb{E}_{X_{\mathrm{sr}}}\left[\mathbb{E}[e_0^{\mathsf{T}}C_Pe_0\mid X_{\mathrm{sr}}]\right]=\mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(C_PS_0)\lesssim K/N, E ∥ P e 0 ∥ Σ tr 2 = E X sr [ E [ e 0 T C P e 0 ∣ X sr ] ] = E X sr Tr ( C P S 0 ) ≲ K / N ,
E ∥ e 0 ∥ Σ t r 2 = E X s r [ E [ e 0 T Σ t r e 0 ∣ X s r ] ] = E X s r Tr ( Σ t r S 0 ) = O ( 1 ) . (123) \mathbb{E}\lVert e_0\rVert_{\Sigma_{\mathrm{tr}}}^2=\mathbb{E}_{X_{\mathrm{sr}}}\left[\mathbb{E}[e_0^{\mathsf{T}}\Sigma_{\mathrm{tr}}e_0\mid X_{\mathrm{sr}}]\right]=\mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)=O(1). \tag*{(123)} E ∥ e 0 ∥ Σ tr 2 = E X sr [ E [ e 0 T Σ tr e 0 ∣ X sr ] ] = E X sr Tr ( Σ tr S 0 ) = O ( 1 ) . (123)
Taking the expectation of (111) over the source design therefore yields
E ∥ β ^ n − β ∗ ∥ Σ t r 2 = E X s r [ E [ ∥ β ^ n − β ∗ ∥ Σ t r 2 ∣ X s r ] ] \mathbb{E}\lVert\widehat{\beta}_n-\beta_*\rVert_{\Sigma_{\mathrm{tr}}}^2=\mathbb{E}_{X_{\mathrm{sr}}}\left[\mathbb{E}\left[\lVert\widehat{\beta}_n-\beta_*\rVert_{\Sigma_{\mathrm{tr}}}^2\mid X_{\mathrm{sr}}\right]\right] E ∥ β n − β ∗ ∥ Σ tr 2 = E X sr [ E [ ∥ β n − β ∗ ∥ Σ tr 2 ∣ X sr ] ]
≲ E X s r Tr ( C P S 0 ) + K N ( σ t r 2 + E X s r Tr ( Σ t r S 0 ) ) \lesssim\mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(C_PS_0)+\frac{K}{N}\left(\sigma_{\mathrm{tr}}^2+\mathbb{E}_{X_{\mathrm{sr}}}\operatorname{Tr}(\Sigma_{\mathrm{tr}}S_0)\right) ≲ E X sr Tr ( C P S 0 ) + N K ( σ tr 2 + E X sr Tr ( Σ tr S 0 ) )
≲ K N + K N ( σ t r 2 + O ( 1 ) ) ≲ K N . (124) \lesssim\frac{K}{N}+\frac{K}{N}\left(\sigma_{\mathrm{tr}}^2+O(1)\right)\lesssim\frac{K}{N}. \tag*{(124)} ≲ N K + N K ( σ tr 2 + O ( 1 ) ) ≲ N K . (124)
This completes the proof of the upper bound.
Lower bound
We derive a matching lower bound from the target label noise alone. Let us condition on the source sample D s r \mathcal{D}_{\mathrm{sr}} D sr , so that the initialization error e 0 = β ^ s r − β ∗ e_0=\widehat{\beta}_{\mathrm{sr}}-\beta_* e 0 = β sr − β ∗ is fixed. Writing e t : = β ^ t − β ∗ e_t:=\widehat{\beta}_t-\beta_* e t := β t − β ∗ and using y t = x t T β ∗ + ϵ t y_t=x_t^{\mathsf{T}}\beta_*+\epsilon_t y t = x t T β ∗ + ϵ t , the SGD recursion gives
e t = ( I − η t x t x t T ) e t − 1 + η t ϵ t x t . e_t=(I-\eta_tx_tx_t^{\mathsf{T}})e_{t-1}+\eta_t\epsilon_tx_t. e t = ( I − η t x t x t T ) e t − 1 + η t ϵ t x t .
We decompose this error as
e t = r t + z t , e_t=r_t+z_t, e t = r t + z t ,
where r t r_t r t is the initialization error that propagates during training, which satisfies
r 0 = e 0 , r t = ( I − η t x t x t T ) r t − 1 , r_0=e_0,\qquad r_t=(I-\eta_tx_tx_t^{\mathsf{T}})r_{t-1}, r 0 = e 0 , r t = ( I − η t x t x t T ) r t − 1 ,
and z t z_t z t is the error due to the noisy labels,
z 0 = 0 , z t = ( I − η t x t x t T ) z t − 1 + η t ϵ t x t . z_0=0,\qquad z_t=(I-\eta_tx_tx_t^{\mathsf{T}})z_{t-1}+\eta_t\epsilon_tx_t. z 0 = 0 , z t = ( I − η t x t x t T ) z t − 1 + η t ϵ t x t .
The identity e t = r t + z t e_t = r_t + z_t e t = r t + z t follows by induction from these recursions.
Conditional on D s r D_{\mathrm{sr}} D sr and all target covariates x 1 , … , x n x_1,\ldots,x_n x 1 , … , x n , the vector
r n = ( I − η n x n x n T ) ⋯ ( I − η 1 x 1 x 1 T ) e 0 r_n = (I-\eta_n x_n x_n^{\mathsf T})\cdots(I-\eta_1 x_1 x_1^{\mathsf T})e_0 r n = ( I − η n x n x n T ) ⋯ ( I − η 1 x 1 x 1 T ) e 0
is deterministic. Furthermore, expanding the noise recursion gives
z n = ∑ t = 1 n η t ϵ t ( I − η n x n x n T ) ⋯ ( I − η t + 1 x t + 1 x t + 1 T ) x t , z_n = \sum_{t=1}^{n}\eta_t\epsilon_t(I-\eta_n x_n x_n^{\mathsf T})\cdots(I-\eta_{t+1}x_{t+1}x_{t+1}^{\mathsf T})x_t, z n = t = 1 ∑ n η t ϵ t ( I − η n x n x n T ) ⋯ ( I − η t + 1 x t + 1 x t + 1 T ) x t ,
where the matrix product is interpreted as I I I when t = n t=n t = n . Thus, z n z_n z n is a linear combination of the target noises with coefficients fixed under this conditioning.
Averaging over the conditioned variables yields the lower bounds
E ∥ e n ∥ Σ t r 2 = E ∥ r n ∥ Σ t r 2 + E ∥ z n ∥ Σ t r 2 ≥ E ∥ z n ∥ Σ t r 2 , \mathbb{E}\lVert e_n\rVert_{\Sigma_{\mathrm{tr}}}^{2}
=
\mathbb{E}\lVert r_n\rVert_{\Sigma_{\mathrm{tr}}}^{2}
+
\mathbb{E}\lVert z_n\rVert_{\Sigma_{\mathrm{tr}}}^{2}
\ge
\mathbb{E}\lVert z_n\rVert_{\Sigma_{\mathrm{tr}}}^{2}, E ∥ e n ∥ Σ tr 2 = E ∥ r n ∥ Σ tr 2 + E ∥ z n ∥ Σ tr 2 ≥ E ∥ z n ∥ Σ tr 2 ,
where we use the fact that the target noises are centered and independent of the source sample and all target covariates, so that the cross term is zero.
It remains to control E ∥ z n ∥ Σ t r 2 \mathbb{E}\lVert z_n\rVert_{\Sigma_{\mathrm{tr}}}^{2} E ∥ z n ∥ Σ tr 2 . Defining V t : = E [ z t z t T ] V_t := \mathbb{E}[z_tz_t^{\mathsf T}] V t := E [ z t z t T ] , we have E ∥ β ^ n − β ∗ ∥ Σ t r 2 ≥ Tr ( Σ t r V n ) \mathbb{E}\lVert\widehat{\beta}_n-\beta_*\rVert_{\Sigma_{\mathrm{tr}}}^{2} \ge\operatorname{Tr}(\Sigma_{\mathrm{tr}}V_n) E ∥ β n − β ∗ ∥ Σ tr 2 ≥ Tr ( Σ tr V n ) , where V t V_t V t satisfies the following recursion
V t = E [ ( I − η t x t x t T ) V t − 1 ( I − η t x t x t T ) ] + η t 2 σ t r 2 Σ t r . V_t = \mathbb{E}\left[(I-\eta_t x_t x_t^{\mathsf T})V_{t-1}(I-\eta_t x_t x_t^{\mathsf T})\right]+\eta_t^2\sigma_{\mathrm{tr}}^2\Sigma_{\mathrm{tr}}. V t = E [ ( I − η t x t x t T ) V t − 1 ( I − η t x t x t T ) ] + η t 2 σ tr 2 Σ tr .
For every positive semidefinite A A A , we compute
E [ ( I − η t x t x t T ) A ( I − η t x t x t T ) − ( I − η t Σ t r ) A ( I − η t Σ t r ) ] = η t 2 E [ ( x t x t T − Σ t r ) A ( x t x t T − Σ t r ) ] ⪰ 0. \mathbb{E}\left[(I-\eta_t x_t x_t^{\mathsf T})A(I-\eta_t x_t x_t^{\mathsf T})-(I-\eta_t\Sigma_{\mathrm{tr}})A(I-\eta_t\Sigma_{\mathrm{tr}})\right]
=\eta_t^2\mathbb{E}\left[(x_t x_t^{\mathsf T}-\Sigma_{\mathrm{tr}})A(x_t x_t^{\mathsf T}-\Sigma_{\mathrm{tr}})\right]\succeq0. E [ ( I − η t x t x t T ) A ( I − η t x t x t T ) − ( I − η t Σ tr ) A ( I − η t Σ tr ) ] = η t 2 E [ ( x t x t T − Σ tr ) A ( x t x t T − Σ tr ) ] ⪰ 0.
Iterating this positive-semidefinite comparison and taking the target trace yields
Tr ( Σ t r V n ) ≥ σ t r 2 ∑ k ∑ t = 1 n η t 2 ξ t r , k 2 ∏ j = t + 1 n ( 1 − η j ξ t r , k ) 2 (125) \operatorname{Tr}(\Sigma_{\mathrm{tr}}V_n)
\ge
\sigma_{\mathrm{tr}}^2
\sum_{k}\sum_{t=1}^{n}\eta_t^2\xi_{\mathrm{tr},k}^2
\prod_{j=t+1}^{n}(1-\eta_j\xi_{\mathrm{tr},k})^2 \tag*{(125)} Tr ( Σ tr V n ) ≥ σ tr 2 k ∑ t = 1 ∑ n η t 2 ξ tr , k 2 j = t + 1 ∏ n ( 1 − η j ξ tr , k ) 2 (125)
We now restrict (125) to 1 ≤ t ≤ N 1\le t\le N 1 ≤ t ≤ N and ⌈ K ⌉ ≤ k ≤ ⌊ 2 K ⌋ \lceil K\rceil\le k\le\lfloor2K\rfloor ⌈ K ⌉ ≤ k ≤ ⌊ 2 K ⌋ . On this range,
η t = η 0 = K b N , ξ t r , k ≍ K − b , ∑ j = 1 n η j ≤ 2 K b . \eta_t=\eta_0=\frac{K^b}{N},\qquad\xi_{\mathrm{tr},k}\asymp K^{-b},\qquad\sum_{j=1}^{n}\eta_j\le2K^b. η t = η 0 = N K b , ξ tr , k ≍ K − b , j = 1 ∑ n η j ≤ 2 K b .
For sufficiently large n n n , we have 0 ≤ η j ξ t r , k ≤ 1 / 2 0\le\eta_j\xi_{\mathrm{tr},k}\le1/2 0 ≤ η j ξ tr , k ≤ 1/2 . Using log ( 1 − x ) ≥ − 2 x \log(1-x)\ge-2x log ( 1 − x ) ≥ − 2 x on this interval, we obtain uniformly over these t , k t,k t , k
∏ j = t + 1 n ( 1 − η j ξ t r , k ) 2 ≥ exp ( − 4 ξ t r , k ∑ j = t + 1 n η j ) ≥ c > 0. \prod_{j=t+1}^{n}(1-\eta_j\xi_{\mathrm{tr},k})^2
\ge
\exp\left(-4\xi_{\mathrm{tr},k}\sum_{j=t+1}^{n}\eta_j\right)
\ge c>0. j = t + 1 ∏ n ( 1 − η j ξ tr , k ) 2 ≥ exp ( − 4 ξ tr , k j = t + 1 ∑ n η j ) ≥ c > 0.
Computing the summation gives
E ∥ β ^ n − β ∗ ∥ Σ t r 2 ≥ c σ t r 2 ∑ k = ⌈ K ⌉ ⌊ 2 K ⌋ ∑ t = 1 N η 0 2 ξ t r , k 2 ≳ K N η 0 2 K − 2 b = K N . (126) \mathbb{E}\lVert\widehat{\beta}_n-\beta_*\rVert_{\Sigma_{\mathrm{tr}}}^{2}
\ge
c\sigma_{\mathrm{tr}}^2
\sum_{k=\lceil K\rceil}^{\lfloor2K\rfloor}
\sum_{t=1}^{N}\eta_0^2\xi_{\mathrm{tr},k}^2
\gtrsim
KN\eta_0^2K^{-2b}
=
\frac{K}{N}. \tag*{(126)} E ∥ β n − β ∗ ∥ Σ tr 2 ≥ c σ tr 2 k = ⌈ K ⌉ ∑ ⌊ 2 K ⌋ t = 1 ∑ N η 0 2 ξ tr , k 2 ≳ K N η 0 2 K − 2 b = N K . (126)
Combining with the upper bound gives
E ∥ β ^ n − β ∗ ∥ Σ t r 2 ≍ K N = Θ ~ ( n − Γ T L ) , \mathbb{E}\lVert\widehat{\beta}_n-\beta_*\rVert_{\Sigma_{\mathrm{tr}}}^{2}
\asymp
\frac{K}{N}
=
\widetilde{\Theta}(n^{-\Gamma_{\mathrm{TL}}}), E ∥ β n − β ∗ ∥ Σ tr 2 ≍ N K = Θ ( n − Γ TL ) ,
which proves Lemma 1.
Additional numerical results and experimental details
Deterministic equivalent predictions for empirical spectra
We test the accuracy of the deterministic equivalent predictions when we replace the Gaussian inputs with PCA features from MNIST [25 ] and CIFAR-10 [23 ] , while keeping the outputs generated by the same linear model as in the theory. Figure 4 and Figure 5 compare predictions from the deterministic equivalent with the empirical values of the transfer learning risk as the feature dimension, regularization, and sample size vary. We also include a Gaussian control with the same covariance spectrum. The Gaussian control follows the deterministic equivalent, while the image features show slightly larger deviations, especially under weaker regularization. In both datasets, the deterministic prediction captures the main shape of the risk curve, including the peak near p = n p=n p = n and its shift with n n n .
Figure 4. Deterministic equivalent on MNIST PCA features. Inputs are the top p p p principal components of the centered training images, and outputs follow a linear model with noise standard deviation 0.2 and normalized correlation between β s r \beta_{\rm sr} β sr and β t r \beta_{\rm tr} β tr equal to 0.8. Source and target inputs are drawn with replacement from the training images. We use λ s r = 0.5 λ t r \lambda_{\rm sr}=0.5\lambda_{\rm tr} λ sr = 0.5 λ tr . The left column fixes n s r = n t r = 300 n_{\rm sr}=n_{\rm tr}=300 n sr = n tr = 300 and varies λ t r ∈ { 10 − 1 , 5 × 10 − 1 , 1 } \lambda_{\rm tr}\in\{10^{-1}, 5\times10^{-1}, 1\} λ tr ∈ { 1 0 − 1 , 5 × 1 0 − 1 , 1 } ; the right column fixes λ t r = 10 − 1 \lambda_{\rm tr}=10^{-1} λ tr = 1 0 − 1 and varies n s r = n t r ∈ { 300 , 500 , 700 } n_{\rm sr}=n_{\rm tr}\in\{300,500,700\} n sr = n tr ∈ { 300 , 500 , 700 } . Top: solid curves show the deterministic equivalent and markers show the mean target excess risk over 100 training draws with ± 1 \pm1 ± 1 s.e. bars. The risk increases up to p = n p=n p = n and then flattens. For n = 300 n=300 n = 300 and λ t r = 10 − 1 \lambda_{\rm tr}=10^{-1} λ tr = 1 0 − 1 , there is a peak at p = n p=n p = n . Discrepancies are larger at weaker regularization. Bottom: ratios of empirical risk to the deterministic equivalent for image features and Gaussian features with the same covariance spectrum. The Gaussian control stays within 3.4% of the prediction, while the image features deviate more.
Figure 5. Deterministic equivalent on CIFAR-10 PCA features. We use the same setup as in Figure 4 , using the same image pool for PCA and for drawing 20,000 training samples. Source and target inputs are drawn with replacement from this pool. The left column fixes n s r = n t r = 300 n_{\mathrm{sr}} = n_{\mathrm{tr}} = 300 n sr = n tr = 300 and varies λ t r ∈ { 10 − 1 , 5 × 10 − 1 , 1 } \lambda_{\mathrm{tr}} \in\{10^{-1}, 5 \times10^{-1}, 1\} λ tr ∈ { 1 0 − 1 , 5 × 1 0 − 1 , 1 } ; the right column fixes λ t r = 10 − 1 \lambda_{\mathrm{tr}} = 10^{-1} λ tr = 1 0 − 1 and varies n s r = n t r ∈ { 300 , 500 , 700 } n_{\mathrm{sr}} = n_{\mathrm{tr}} \in\{300, 500, 700\} n sr = n tr ∈ { 300 , 500 , 700 } . Top: solid curves show the deterministic equivalent and markers show the mean target excess risk over 100 training draws with ± 1 \pm1 ± 1 s.e. bars. The curves show a peak near p = n p = n p = n , which shifts with n n n , with larger errors under weaker regularization. Bottom: ratios of empirical risk to the deterministic equivalent for image features and Gaussian features with the same covariance spectrum. The Gaussian control stays within 3.5% of the prediction, while the image features deviate more.
Details for experiments on language models
Model and data. We use GPT-2 small [33 ] in the nanoGPT implementation [19 ] , with 12 layers, 12 attention heads, width 768, context length 1024, and 123.6M parameters. FineWeb-Edu [32 ] is the source distribution, and Python [4 ] is the target distribution. We tokenize all data with the GPT-2 tokenizer. We ensure that the target validation and test sets do not overlap with the source training pool; for Python, we additionally split the train and held-out data by repository and remove near-duplicate training files.
Setup. We refer to the number of unique target and source tokens as the target budget and source budget , respectively. The target budget is n = 2 k n=2^k n = 2 k for k ∈ { 21 , … , 26 } k\in\{21,\ldots,26\} k ∈ { 21 , … , 26 } . Within each seed, the budgets are nested, so different methods see the same target examples in the same order. The source budget follows
m γ ( n ) = m ∗ ( n n ∗ ) γ , n ∗ = 2 21 , m ∗ = 7 ⋅ 2 21 , (127) m_{\gamma}(n)=m^*\left(\frac{n}{n^*}\right)^\gamma,\qquad n^*=2^{21},\qquad m^*=7\cdot2^{21}, \tag*{(127)} m γ ( n ) = m ∗ ( n ∗ n ) γ , n ∗ = 2 21 , m ∗ = 7 ⋅ 2 21 , (127)
with γ ∈ { 0 , 0.75 , 1 , 1.25 } \gamma\in\{0,0.75,1,1.25\} γ ∈ { 0 , 0.75 , 1 , 1.25 } . Thus, γ = 0 \gamma=0 γ = 0 corresponds to a fixed source budget, while γ > 0 \gamma>0 γ > 0 makes the source budget grow with the target budget.
Training. We train all models with AdamW [30 ] , using β = ( 0.9 , 0.95 ) \beta=(0.9,0.95) β = ( 0.9 , 0.95 ) , gradient clipping at 1, and batches of 65,536 tokens. Each source model makes one pass over FineWeb-Edu. The main source learning rate, 6 × 10 − 4 6\times10^{-4} 6 × 1 0 − 4 , is selected on seed 0 from { 3 , 6 , 12 } × 10 − 4 \{3,6,12\}\times10^{-4} { 3 , 6 , 12 } × 1 0 − 4 using source validation loss at m ∗ m^* m ∗ .
We compare three procedures for training with target data. Target only (TO) trains from random initialization. Fine-tuning (FT) starts from the corresponding source checkpoint with a fresh optimizer state. L2-SP (Li et al., 2018) replaces ordinary weight decay with a pull toward the source parameters. In our implementation (FTSP), we apply the decoupled update
θ ← θ − min ( η t λ , 1 ) ( θ − θ s r ) , (128) \theta\leftarrow\theta- \min(\eta_t\lambda, 1)(\theta- \theta_{\mathrm{sr}}), \tag*{(128)} θ ← θ − min ( η t λ , 1 ) ( θ − θ sr ) , (128)
after every AdamW step, so it is decoupled from the gradient in the same way AdamW’s weight decay is. The original L2-SP adds a penalty to the loss instead. Also, here η t \eta_t η t is the scheduled learning rate, which can be during the warm-up phase, constant phase, or cool-down phase.
Target-training length. We train for E ∈ { 1 , 2 , 4 , … } E \in\{1, 2, 4, \ldots\} E ∈ { 1 , 2 , 4 , … } epochs using cool-down branches from the same run. Hyperparameters are tuned with at most 16 epochs. We choose the number of epochs based on validation, and we increase E E E until doubling the number of epochs improves validation loss by less than 0.01, and then we report the branch with the lowest validation loss. The E = 1 E = 1 E = 1 branch of the same run gives the one-pass result.
Hyperparameters and seeds. Learning rate, regularization strength, and dropout are tuned on seed 0, separately for each target and training method, at n ∈ { 2 22 , 2 26 } n \in\{2^{22}, 2^{26}\} n ∈ { 2 22 , 2 26 } . The experiments use seeds 0, 4, 5. Each seed fixes the initialization, dropout noise, and data order, allowing paired comparisons across methods.
Evaluation. The test loss is measured as the average cross-entropy loss per token. Since the irreducible loss floor is unknown, we summarize scaling directly from the observed test loss. Between neighboring target budgets n i n_i n i and n i + 1 n_{i+1} n i + 1 , we define the local exponent
Γ ^ i , i + 1 : = − log L i + 1 − log L i log n i + 1 − log n i (129) \widehat{\Gamma}_{i,i+1} := -\frac{\log L_{i+1} - \log L_i}{\log n_{i+1} - \log n_i} \tag*{(129)} Γ i , i + 1 := − log n i + 1 − log n i log L i + 1 − log L i (129)
For windows containing more than two target budgets n i n_i n i , we fit a single exponent by least squares. We compute ratios and differences within each seed and then average them, reporting standard errors across seeds.