betaFeedback

WorksComputer Science

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Abstract

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the L2L_2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

INTRODUCTION

Empirical studies have demonstrated that reusing pretrained representations improves downstream performance in computer vision [9, 40] and language tasks [16, 8]. Methods such as L2-SP (Li et al., 2018), adapters [15], and LoRA (Hu et al., 2022) adapt models through regularization toward pretrained weights, the addition of task-specific modules, and low-rank parameter updates, respectively. Besides the choice of a specific fine-tuning method, downstream performance also depends on the scale of the source and target model. More precisely, empirical scaling laws describe how transfer performance varies with the scaling of model size and the amounts of pretraining and fine-tuning data [14, 41]. Notably, better pretraining performance on the source task does not necessarily improve results downstream: longer pretraining can improve source performance while worsening downstream performance [34]; conversely, stronger pretraining weight decay can improve downstream performance even when it increases the pretraining validation loss [10]. Understanding these effects requires studying when pretraining improves downstream learning, especially in the scaling law, and how the source model should be trained to achieve these gains.

From a theoretical viewpoint, several works have studied regularization with respect to a source estimator, referred to as L2-SP in (Li et al., 2018). In this regard, Dar et al. [6] analyze minimum-norm pretraining with isotropic source covariates followed by ridge fine-tuning under model shift, identifying conditions under which transfer outperforms ridge regression only on target data. Kim et al. [21] study minimum-norm pretraining followed by the target interpolator closest to the source estimator, deriving non-asymptotic risk bounds under covariate and model shift in benign-overfitting regimes. In functional linear regression, Lin and Reimherr [28] establish minimax rates for transfer learning where both the pretraining and fine-tuning objective is L2L_2-regularized under bounded model shift and covariance compatibility conditions, quantifying its benefits over learning only on target data. In nonparametric regression under model shift, Lin and Reimherr [29] develop a method for selecting regularization parameters in pretraining and fine-tuning that achieves minimax rates. However, these works do not explicitly characterize when transfer learning achieves a strictly better rate than learning from a single dataset (only the source, or only the target). Wu et al. (2022) establish sharp risk bounds under covariate shift when both pretraining and fine-tuning use one-pass stochastic gradient descent (SGD), and construct specific examples where this two-stage procedure improves sample complexity. This motivates us to derive a systematic characterization of how covariance spectra and sample sizes determine when transfer improves the decay of the risk. A related question concerns how the source model should be trained to achieve these gains. For ridge regression, Hedges [13] shows that, under proportional asymptotics with isotropic data, the optimal source regularization is independent of the target sample size but it still depends on task alignment. This leaves open whether a single source estimator, trained without target information, can remain optimal across a family of target distributions in regimes where transfer improves the scaling exponent over either dataset alone.

In this paper, we study these questions through a tractable model of high-dimensional linear regression. The source and target tasks may differ in their regression coefficients, covariate distributions, and noise levels, with sample sizes growing at different polynomial rates. We obtain the source estimator from ridge regression and fine-tune it on target data using a squared ℓ2\ell_2 penalty centered at the source estimator. We summarize our main contributions as follows:

  • Scaling laws for transfer learning. We derive a deterministic equivalent for the target excess risk of ridge regression, together with a non-asymptotic approximation error (Theorem 1). Specializing to power-law covariance spectra and a common regressor, we obtain scaling laws for transfer learning (Theorem 3). Comparing them to the joint minimax rate which uses both datasets (Theorem 2), we show that the optimally tuned deterministic equivalent achieves the joint minimax rate whenever the latter is strictly faster than the minimax rates of either dataset alone (Corollary 1).

  • Universality of optimal source training. Our analysis unveils that the optimal source regularization depends only on source information, allowing a single source estimator to achieve these gains across various target distributions without target-dependent tuning of the source. Furthermore, the same source estimator remains optimal when fine-tuning is performed via one-pass SGD (Lemma 1), indicating universality across fine-tuning methods.

  • Numerical evidence. We evaluate the accuracy of the predictions based on deterministic equivalents in synthetic data and PCA features from MNIST and CIFAR-10. We further conduct language-modeling experiments, giving evidence that similar transfer phenomena may arise beyond linear models.

Related Work

Empirical works. Early empirical studies examined how pretrained representations are adapted and how transfer benefits vary across downstream tasks [40, 22]. Common approaches include using frozen features, fine-tuning all pretrained parameters, and adapting a small number of trainable parameters. L2-SP regularizes the target model toward pretrained weights (Li et al., 2018), while adapters and LoRA introduce task-specific modules and low-rank weight updates [15] (Hu et al. 2022). The adaptation procedure also affects out-of-distribution performance: Kumar et al. [24] show that full fine-tuning can distort pretrained features, and that training a linear head on frozen features before fine-tuning can improve robustness to distribution shift. Empirical scaling laws further quantify the benefits of pretraining in terms of an equivalent amount of additional target data [14], and describe how downstream performance depends on model size, pretraining and fine-tuning data sizes, and the adaptation method [41]. Isik et al. [18] show that source-target distribution alignment and fine-tuning data size affect downstream scaling: increasing pretraining data can reduce downstream cross-entropy without consistently improving translation quality. These findings motivate theoretical analyses of how task alignments, datasets size, and fine-tuning procedures affect the benefit of transfer learning.

Theoretical works. Beyond the two-stage regression analyses discussed in the introduction, theoretical studies have considered other forms of transfer learning. One related line of work is continual linear regression, which can be considered as a generalization of transfer learning to multiple sequential tasks. Zhao et al. [42] identify matrix-valued regularizers that match the estimation rates of learning jointly from all tasks, under shared regression coefficients and commuting covariances. Karpel et al. [20] characterize the generalization error and the optimal regularization strength for isotropic data in the proportional asymptotics regime. For classification, classical domain-adaptation bounds relate target error to source error, distribution discrepancy, and the joint error of the best shared classifier [1]. Hanneke and Kpotufe [11] characterize minimax rates, quantifying how transfer difficulty and label noise determine the relative value of source and target samples. [35, 5] study models in which tasks share an unknown feature representation and show that learning this representation from sufficiently diverse training tasks can reduce the sample complexity of learning a new task. In high-dimensional regression, Li et al. (2022) establish minimax rates under sparse or small source–target coefficient differences and develop methods that adapt to unknown informative sources. He et al. (2024) develop transfer estimators that improve estimation under heterogeneous covariances when source–target coefficient differences are sufficiently small. Complementing these results, we study how covariance spectra and relative sample sizes determine the scaling benefits of transfer, identifying regimes in which ridge pretraining followed by ridge or SGD fine-tuning achieves minimax scaling and outperforms both target-only and source-only learning.

Problem SETUP

Given n∈Nn \in\mathbb{N}, we use the shorthand [n]:={1,…,n}[n] := \{1,\ldots,n\}. We consider two independent training datasets, corresponding to a source task (sr) and a target task (tr). For the source and target tasks, respectively, we observe nsrn_{\mathrm{sr}} and ntrn_{\mathrm{tr}} independent samples (xsr(j),ysr(j))j∈[nsr](x_{\mathrm{sr}^{(j)}}, y_{\mathrm{sr}}^{(j)})_{j\in[n_{\mathrm{sr}}]} and (xtr(j),ytr(j))j∈[ntr](x_{\mathrm{tr}}^{(j)}, y_{\mathrm{tr}}^{(j)})_{j\in[n_{\mathrm{tr}}]}, where xsr(j)x_{\mathrm{sr}}^{(j)} and xtr(j)∈Rdx_{\mathrm{tr}}^{(j)} \in\mathbb{R}^{d} are drawn from distributions PsrP_{sr} and PtrP_{tr} such that EPsr[x]=EPtr[x]=0\mathbb{E}_{P_{sr}}[x] = \mathbb{E}_{P_{tr}}[x] = 0, EPsr[xxT]=Σsr\mathbb{E}_{P_{sr}}[xx^{\mathsf{T}}] = \Sigma_{sr}, and EPtr[xxT]=Σtr\mathbb{E}_{P_{tr}}[xx^{\mathsf{T}}] = \Sigma_{tr}. We assume the following concentration property on the designs.

Assumption 1. There exist constants C,c>0C,c > 0 such that, for every z∈{sr,tr}z \in\{sr,tr\}, every PSD matrix A∈Rd×dA \in\mathbb{R}^{d \times d}, and every t>0t > 0, a sample x∼Pzx \sim P_z satisfies

P(∣xTAx−Tr⁡(ΣzA)∣>t∥Σz1/2AΣz1/2∥F)≤Ce−ct.(1)\mathbb{P}\left(\left|x^{\mathsf{T}}Ax-\operatorname{Tr}(\Sigma_z A)\right|>t\left\|\Sigma_z^{1/2}A\Sigma_z^{1/2}\right\|_{F}\right)\le Ce^{-ct}. \tag*{(1)}

Assumption 1 is standard in the literature ([31] (Wu et al. 2026a)) and covers e.g. independent sub-Gaussian entries. We denote Xz=[xz(1),…,xz(nz)]T∈Rnz×dX_z = [x_z^{(1)}, \ldots, x_z^{(n_z)}]^{\mathsf{T}} \in\mathbb{R}^{n_z \times d} the data matrix and yz=(yz(j))j∈[nz]∈Rnzy_z = (y_z^{(j)})_{j \in[n_z]} \in\mathbb{R}^{n_z} the label for source (z=srz = sr) and target (z=trz = tr). The labels are generated by

yz=Xzβz+εz,εz∼N(0,σz2Inz).(2)y_z = X_z\beta_z+\varepsilon_z,\qquad\varepsilon_z\sim\mathcal{N}(0,\sigma_z^2 I_{n_z}). \tag*{(2)}

The noise vectors are independent of the data and of each other. In particular, the source and target tasks may differ in their parameters, covariances, and noise levels. For an estimator β^\hat{\beta}, we consider the excess prediction risk on the target task,

R(β^):=Ex,ε[(xTβ^−xTβtr)2]=Eε[(β^−βtr)TΣtr(β^−βtr)],(3)\mathcal{R}(\hat{\beta}) := \mathbb{E}_{x,\varepsilon}\left[\left(x^{\mathsf{T}}\hat{\beta}-x^{\mathsf{T}}\beta_{tr}\right)^2\right] = \mathbb{E}_{\varepsilon}\left[(\hat{\beta}-\beta_{tr})^{\mathsf{T}}\Sigma_{tr}(\hat{\beta}-\beta_{tr})\right], \tag*{(3)}

where the expectation also averages over label noise.

In this paper, we study transfer learning via the following two-step procedure. We first obtain a source estimator via ridge regression on the source dataset,

β^sr=arg⁡min⁡β{∥Xsrβ−ysr∥22+λsr∥β∥22},(4)\hat{\beta}_{sr}=\underset{\beta}{\arg\min}\left\{\left\|X_{sr}\beta-y_{sr}\right\|_2^2+\lambda_{sr}\|\beta\|_2^2\right\}, \tag*{(4)}

and then obtain the target model by fitting a regression model on the target dataset with ridge penalty centered at the source estimator,

β^tr=arg⁡min⁡β{∥Xtrβ−ytr∥22+λtr∥β−β^sr∥22}.(5)\hat{\beta}_{tr}=\underset{\beta}{\arg\min}\left\{\left\|X_{tr}\beta-y_{tr}\right\|_2^2+\lambda_{tr}\|\beta-\hat{\beta}_{sr}\|_2^2\right\}. \tag*{(5)}

The above two-step procedure has been used by Kim et al. [21] to study properties of transfer learning. The two estimators admit the closed forms

β^sr=(XsrTXsr+λsrId)−1XsrTysr,\hat{\beta}_{sr}=(X_{sr}^{\mathsf{T}}X_{sr}+\lambda_{sr}I_d)^{-1}X_{sr}^{\mathsf{T}}y_{sr},
β^tr=(XtrTXtr+λtrId)−1(XtrTytr+λtrβ^sr).(6)\hat{\beta}_{tr}=(X_{tr}^{\mathsf{T}}X_{tr}+\lambda_{tr}I_d)^{-1}\left(X_{tr}^{\mathsf{T}}y_{tr}+\lambda_{tr}\hat{\beta}_{sr}\right). \tag*{(6)}

Remark 3.1 (Infinite-dimensional formulation). For convenience, we use matrix notation throughout. In the infinite-dimensional setting, covariance matrices Σz\Sigma_z are interpreted as positive semidefinite trace-class operators on a separable Hilbert space, and transposes should be understood as adjoints. Our arguments apply to both settings under the stated assumptions.

Deterministic equivalent for target risk

We start by establishing a deterministic equivalent for the risk (3) of β^tr\hat{\beta}_{tr}. First, let the effective regularization parameters τsr,τtr>0\tau_{sr},\tau_{tr}>0 be the unique solution of

nz−λzτz=Tr⁡[Σz(Σz+τzId)−1],z∈{sr,tr}.(7)n_z-\frac{\lambda_z}{\tau_z}=\operatorname{Tr}\left[\Sigma_z(\Sigma_z+\tau_z I_d)^{-1}\right],\qquad z\in\{sr,tr\}. \tag*{(7)}

Furthermore, for z∈{sr,tr}z\in\{sr,tr\}, let us also define

Wz:=Σz(Σz+τzId)−2,(8)W_z:=\Sigma_z(\Sigma_z+\tau_z I_d)^{-2}, \tag*{(8)}

and the two scalar prefactors

ctr:=τtr2ntrntr−Tr⁡[Σtr2(Σtr+τtrId)−2],c_{tr}:=\frac{\tau_{tr}^{2}n_{tr}}{n_{tr}-\operatorname{Tr}\left[\Sigma_{tr}^{2}(\Sigma_{tr}+\tau_{tr}I_d)^{-2}\right]},

References

  1. [1]Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. (2010). A theory of learning from different domains. Machine Learning, 79(1):151–175.
  2. [2]Cai, T. T. and Wei, H. (2021). Transfer learning for nonparametric classification: Minimax rate and adaptive classifier. The Annals of Statistics, 49(1):100–128.arxiv.org/abs/1906.02903
  3. [3]Caponnetto, A. and De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368.DOI
  4. [4]CodeParrot (2021). CodeParrot-Clean dataset. https://huggingface.co/datasets/codeparrot/codeparrot-clean.
  5. [5]Collins, L., Hassani, H., Soltanolkotabi, M., Mokhtari, A., and Shakkottai, S. (2024). Provable multi-task representation learning by two-layer ReLU neural networks. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9292–9345. PMLR.arxiv.org/abs/2307.06887
  6. [6]Dar, Y., LeJeune, D., and Baraniuk, R. G. (2024). The common intuition to transfer learning can win or lose: Case studies for linear regression. SIAM Journal on Mathematics of Data Science, 6(2):454–480.arxiv.org/abs/2103.05621
  7. [7]Defilippis, L., Loureiro, B., and Misiakiewicz, T. (2024). Dimension-free deterministic equivalents and scaling laws for random feature regression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2405.15699
  8. [8]Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1810.04805
  9. [9]Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In Xing, E. P. and Jebara, T., editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 647–655, Beijing, China. PMLR.arxiv.org/abs/1310.1531
  10. [10]Han, T., Bordt, S., Zhang, H., and Kakade, S. M. (2026). Weight decay improves language model plasticity. In Zhang, T., Dudik, M., Jaggi, M., Agarwal, A., Li, S., Schuurmans, D., Zhu, J., Berkenkamp, F., Dong, H., and Bietti, A., editors, Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 39571–39602. PMLR.arxiv.org/abs/2602.11137
  11. [11]Hanneke, S. and Kpotufe, S. (2019). On the value of target data in transfer learning. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.arxiv.org/abs/2002.04747
  12. [13]Hedges, C. E. (2026). Source-optimal training is transfer-suboptimal. Transactions on Machine Learning Research.arxiv.org/abs/2511.08401
  13. [14]Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). Scaling laws for transfer. arXiv preprint arXiv:2102.01293.arxiv.org/abs/2102.01293
  14. [15]Houlsby, N., Giu rgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attarian, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR.arxiv.org/abs/1902.00751
  15. [16]Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Gurevych, I. and Miyao, Y., editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.arxiv.org/abs/1801.06146
  16. [18]Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., and Koyejо, S. (2025). Scaling laws for downstream task performance in machine translation. In International Conference on Learning Representations.
  17. [19]Karpathy, A. (2022). nanoGPT. https://github.com/karpathy/nanoGPT.
  18. [20]Karpel, G., Moroshko, E., Lev̄instein, R., Meir, R., Soudry, D., and Evron, I. (2026). Optimal l2 regularization in high-dimensional continual linear regression. In Telgarsky, M. and Ullman, J., editors, Proceedings of The 37th International Conference on Algorithmic Learning Theory, volume 313 of Proceedings of Machine Learning Research, pages 1–62. PMLR.arxiv.org/abs/2601.13844
  19. [21]Kim, Y., Kim, I., and Park, S. (2025). Transfer learning for benign overfitting in high-dimensional linear regression. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2510.15337
  20. [22]Kornblith, S., Shlens, J., and Le, Q. V. (2019). Do better ImageNet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661–2671.arxiv.org/abs/1805.08974
  21. [23]Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto.
  22. [24]Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. (2022). Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations.arxiv.org/abs/2202.10054
  23. [25]Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
  24. [28]Lin, H. and Reimherr, M. (2024a). On hypothesis transfer learning of functional linear models. In Forty-first International Conference on Machine Learning.
  25. [29]Lin, H. and Reimherr, M. (2024b). Smoothness adaptive hypothesis transfer learning. In Forty-first International Conference on Machine Learning.arxiv.org/abs/2402.14966
  26. [30]Loschchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
  27. [31]Misiakiewicz, T. and Saeed, B. (2024). A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator. arXiv preprint arXiv:2403.08938.arxiv.org/abs/2403.08938
  28. [32]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. (2024). The FineWeb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849.arxiv.org/abs/2406.17557
  29. [33]Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. Technical report, OpenAI.
  30. [34]Springer, J. M., Goyal, S., Wen, K., Kumar, T., Yue, X., Malladi, S., Neubig, G., and Raghunathan, A. (2025). Overtrained language models are harder to fine-tune. In Forty-second International Conference on Machine Learning.arxiv.org/abs/2503.19206
  31. [35]Tripuraneni, N., Jordan, M. I., and Jin, C. (2020). On the theory of transfer learning: The importance of task diversity. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 7852–7862. Curran Associates, Inc.arxiv.org/abs/2006.11650
  32. [40]Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? In Gharamani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.arxiv.org/abs/1411.1792
  33. [41]Zhang, B., Liu, Z., Cherry, C., and Firat, O. (2024). When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2402.17193
  34. [42]Zhao, X., Wang, H., Huang, W., and Lin, W. (2024). A statistical theory of regularization-based continual learning. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61021–61039. PMLR.arxiv.org/abs/2406.06213

Paper details

vlnc:W_dzj952w6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents