betaFeedback

WorksMathematics

Semiparametric regression with stagewise empirical risk minimization

Abstract

Maintaining both interpretability and accuracy remains a fundamental challenge in regression analysis. This paper addresses these two objectives through a semiparametric approach that combines a parametric crude guess with its nonparametric adjustment. In particular, the nonparametric adjustment is implemented via a machine learning technique called ``stagewise empirical risk minimization algorithm.'' Once a set (dictionary) of functions (words) is prepared, this algorithm constructs a convex combination of words selected iteratively from the dictionary. This selection is based on the empirical risk derived from the Bregman divergence, which enables the proposed estimators to be applied in broad settings, including generalized linear models. This paper examines these proposed semiparametric estimators theoretically and numerically. In the theoretical study, we derive a non-asymptotic error bound for the estimators constructed from a general data-dependent dictionary. Among the concrete dictionaries proposed in this paper, some can be utilized to correct variable selection in LASSO-based parametric regression, which is investigated in the numerical study. Moreover, simulation studies investigate practical performance of the proposed estimators, while real data applications further confirm their predictive performance. These studies indicate that the proposed semiparametric estimators can address the above challenge in several settings.

Introduction

Regression analysis is one of the most widely used statistical methods and has extensive applications across many fields. It investigates functional relationships between output (response) and input (explanatory) variables, so that one can model and describe these relationships or can predict new response values, see Chatterjee and Hadi [5], Klemelä [19], James et al. [16].

In this paper, we propose a semiparametric regression method that seeks both interpretability and flexibility by incorporating a machine learning technique. To outline the proposed method, Figure 1 illustrates its application to a simple estimation problem in a one-dimensional normal regression setting. Our regression estimators are based on a “parametric” crude guess and its “nonparametric” adjustment: panel (a) displays a scatterplot of the data points and the least squares regression line m(x,θ^)=θ^0+θ^1xm\left(x,\widehat{\theta}\right)=\widehat{\theta}_0+\widehat{\theta}_1x with θ^=[θ^0 θ^1]T\widehat{\theta}=\left[\widehat{\theta}_0\ \widehat{\theta}_1\right]^{\mathsf{T}}, an example of a parametric regression estimator, represented by a dotted line. Here the parameters θ=[θ0 θ1]T\theta=\left[\theta_0\ \theta_1\right]^{\mathsf{T}} in m(x,θ)=θ0+θ1xm\left(x,\theta\right)=\theta_0+\theta_1x are estimated to be θ^0≈0.05\widehat{\theta}_0\approx0.05 and θ^1≈1.08>0\widehat{\theta}_1\approx1.08>0, suggesting that the data points globally increase with xx in a sense. By using parametric estimators as in linear models, we can interpret such global behavior of the regression function based only on a model function m(x,θ)m(x,\boldsymbol{\theta}) and estimated parameters θ^\hat{\boldsymbol{\theta}}. However, as observed in panel (a), the linear model above does not fit the data points well locally. Then, allowing for model misspecification, we adjust this initial “parametric” regression with a “nonparametric” approach, which enables estimators to capture broad classes of regression functions flexibly. After this combination of parametric and nonparametric approaches, the proposed method is referred to as “semiparametric.” In this paper, we implement the adjustment via a machine learning technique called “stagewise empirical risk minimization algorithm.” Briefly, from a user-prepared set (dictionary) of functions (words), this algorithm selects one word in each iteration, and then forms a convex combination of the selected words for estimation or prediction. Based on this algorithm and using a dictionary introduced in Section 5, we now construct a term ξ^I(x)\hat{\xi}_{I}(x) for additive adjustment with the number of iterations I=5,40,150I = 5, 40, 150, as presented in panel (b) of Figure 1. Then we compute a proposed semiparametric estimator m(x,θ^)+ξ^I(x)m(x,\hat{\boldsymbol{\theta}}) + \hat{\xi}_{I}(x) shown in panel (a), demonstrating that the regression line is adjusted and that the proposed estimator seems to fit the data points better as II increases.

Figure 1. An example of regression with our proposed method. “Parametric” regression is conducted by using the least squares regression line m(x,θ^)=θ^0+θ^1xm\left(x,\widehat{\theta}\right)=\widehat{\theta}_0+\widehat{\theta}_1x. Its “nonparametric” adjustment function ξ^I(x)\widehat{\xi}_I(x) is implemented via the “stagewise empirical risk minimization algorithm” with the number of iterations I=5,40,150I=5,40,150. Then we obtain a proposed semiparametric estimator m(x,θ^)+ξ^I(x)m\left(x,\widehat{\theta}\right)+\widehat{\xi}_I(x) with additive adjustment.

Now we describe each of the parametric, nonparametric and semiparametric approaches. The parametric approach is fundamental in regression analysis. It models the target regression function as a function determined by a finite-dimensional parameter, as in linear models. This approach reduces the problem of estimating a “function” to that of estimating a “parameter vector” contained in the model, and parametric regression estimators are constructed by plugging an estimator for the parameters (such as the maximum likelihood estimator (MLE) in McCullagh and Nelder [21], Dunn and Smyth [8]) into the model function. Its simple structure facilitates interpretation of the underlying regression function: for example, whether it increases, decreases or hardly changes globally with some explanatory variables. Particularly in linear regression, the LASSO estimator is known to estimate several coefficients for explanatory variables to be exactly 0, so that this estimator can be utilized for variable selection, see Tibshirani [27], Hastie et al. [14], Hastie et al. [13]. Although this parametric approach has the merits of simplicity and interpretability and is widely used in statistics, this approach has a drawback: the assumption that the underlying structure can be captured by a finite-dimensional parameter vector is sometimes very restrictive. It is well known that parametric estimators generally lack consistency under model misspecification.

On the other hand, the nonparametric approach offers flexibility in terms of consistency. This approach includes several methods for regression analysis, such as kernel smoothing, spline smoothing, gradient boosted regression trees (GBRTs). In general, by appropriately tuning smoothing parameters, these methods can achieve consistency for broad classes of regression functions, which makes the nonparametric approach advantageous. However, the structure of nonparametric estimators is sometimes more difficult to interpret than that of parametric models. In addition, several nonparametric methods suffer from data sparsity for high-dimensional explanatory variables, a phenomenon referred to as “curse of dimensionality,” see, for example, Chapter 3.2.6 of Klemelä [19].

From the above observations, the parametric and nonparametric approaches each have their own advantages and disadvantages. This motivates us to consider a fine combination of these two approaches, in order to make regression estimators both easily interpretable and flexible. Specifically, we consider a semiparametric approach in which an initial crude estimation with the parametric approach is subsequently adjusted through the nonparametric approach. The underlying idea can be found in several pioneering works. Focusing on the additive adjustment above, this approach coincides with performing an additional regression on the residuals resulting from the initial regression. This strategy appeared, for example, in Section 5C of Tukey [28], which fits lines to an original data set and then to the resulting residuals. Boosting also applies this strategy by iteratively fitting the residuals, see Friedman [9] and Bühlmann and Yu [4]. The semiparametric approach based on this idea has been explored in several studies, including [23, 11, 32 ) and references therein]. In particular, Fan et al. [11] unified broad classes of semiparametric estimators in kernel smoothing: they incorporated a quasi-likelihood method including Bernoulli (logistic) and Poisson regression; they addressed both additive and multiplicative adjustment; and, through local polynomial smoothing, they proposed semiparametric variants of the Nadaraya–Watson estimator, local linear fitting and so on. Fan et al. [11] also noted on p.4154 that “Here we focus on local polynomial techniques although the idea can be extended to other nonparametric methods,” thereby indicating that the nonparametric part of the semiparametric approach can be implemented with some techniques other than local polynomial ones.

As the nonparametric adjustment in the above semiparametric approach, we utilize a stagewise empirical risk minimization algorithm studied by Klemelä [17], Klemelä [18], Naito and Eguchi [24], Nishida and Naito [26] in density estimation and by Yoshida and Naito [33] in regression analysis. This machine learning technique has several advantages, as pointed out in these studies: (i) Yoshida and Naito [33] utilized the empirical risk derived from the Bregman divergence, which allows their proposed estimators to be applied widely in various settings, such as Bernoulli and Poisson regression. (ii) There are almost no constraints on preparing the dictionary. (ii-1) For instance, dictionaries can be designed to automate smoothing parameter selection in conventional estimators, through the algorithmic selection of words. In density estimation, Nishida and Naito [26] utilized a dictionary composed of scaled kernel functions with several bandwidths and they conducted bandwidth selection analogous to that in kernel density estimation. (ii-2) Another utilization of dictionaries is for variable selection. Yoshida and Naito [33] constructed a dictionary of univariate BB-spline bases corresponding to each explanatory variable and demonstrated variable selection with this dictionary for sparse additive models.

In addition to proposing such a semiparametric method, we aim to investigate its performance both theoretically and numerically. In the theoretical study, we develop a non-asymptotic error bound for the proposed estimators as in Klemelä [18], Naito and Eguchi [24], Nishida and Naito [26] and Yoshida and Naito [33]. At the same time, we examine the asymptotic behavior of the bound, even under general settings where the dictionary and words may depend on the data and sample size. The numerical study explores accuracy of the proposed estimators with concrete dictionaries, by comparing them with several other regression estimators. It also addresses behavior of our estimators in terms of variable selection, when utilizing a dictionary composed of univariate words.

This paper is organized as follows. Section 2 describes the Bregman divergence. Using this divergence, the proposed semiparametric estimators are formulated in Section 3. Their theoretical results are developed in Section 4. Section 5 provides some concrete dictionaries for implementing the estimators. Section 6 discusses simulations investigating the finite-sample performance of our estimators in several settings. Applications to real data sets are presented in Section 7. Section 8 offers concluding remarks. All proofs and calculations for the theoretical results are provided in the Appendix.

Risk functions based on the Bregman divergence

Let Y∈RY \in\mathbb{R} be a response variable and X∈Rd\mathbf{X} \in\mathbb{R}^{d} be a vector of explanatory variables. We are given i.i.d. pairs of observations X={(Y1,X1),…,(Yn,Xn)}⊂R×Rd\mathbf{X} = \{(Y_{1}, \mathbf{X}_{1}), \ldots, (Y_{n}, \mathbf{X}_{n})\} \subset\mathbb{R} \times\mathbb{R}^{d} drawn from a joint distribution with the density function f(y,x)=p(y∣x)q(x)f(y,\mathbf{x}) = p(y \mid\mathbf{x})q(\mathbf{x}), where p(⋅∣x)p(\cdot\mid\mathbf{x}) denotes the conditional density of YY given X=x\mathbf{X} = \mathbf{x}, and qq is the marginal density of X\mathbf{X} with the support D⊂Rd\mathbb{D} \subset\mathbb{R}^{d}. Our goal is to estimate the regression function μ(x)=E[Y∣X=x]\mu(\mathbf{x}) = \mathbb{E}[Y \mid\mathbf{X} = \mathbf{x}] at points x∈D\mathbf{x} \in\mathbb{D}.

To formulate the proposed estimator, we introduce a divergence measure between functions. For a strictly convex real-valued function U:R→RU : \mathbb{R} \to\mathbb{R}, the functional Bregman divergence between the target μ\mu and a candidate estimator μˉ\bar{\mu} is defined as

DU(μ,μˉ)=∫RddU(μ(x),μˉ(x))q(x) dxD_{U}(\mu,\bar{\mu}) = \int_{\mathbb{R}^{d}} d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x}))q(\mathbf{x})\,d\mathbf{x}

where

dU(μ(x),μˉ(x))=U(μ(x))−U(μˉ(x))−u(μˉ(x)){μ(x)−μˉ(x)}d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) = U(\mu(\mathbf{x})) - U(\bar{\mu}(\mathbf{x})) - u(\bar{\mu}(\mathbf{x}))\{\mu(\mathbf{x})-\bar{\mu}(\mathbf{x})\}

and u=U′u = U', see Bregman [2], Basu et al. [1], Yoshida and Naito [33], Naito and Penev [25]. It is known that the inequality dU(μ(x),μˉ(x))≥0d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) \ge0 always holds, and that dU(μ(x),μˉ(x))=0d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) = 0 holds if and only if μ(x)=μˉ(x)\mu(\mathbf{x}) = \bar{\mu}(\mathbf{x}), see Bregman [2] and Basu et al. [1]. These two properties of dUd_{U} are also satisfied by the functional Bregman divergence DUD_{U}. Therefore, DUD_{U} can be regarded as a measure of discrepancy between functions, although it is not a metric in the mathematical sense. As discussed below, different strictly convex functions UU yield different forms of DUD_U, such as the squared error and the loss function used in Bernoulli regression.

We next derive an empirical risk function based on DUD_U. To do this, we use an alternative expression for dUd_U:

dU(μ(x),μˉ(x))=dU∗(u(μˉ(x)),u(μ(x)))=U∗(u(μˉ(x)))−U∗(u(μ(x)))−μ(x){u(μˉ(x))−u(μ(x))}={U∗(u(μˉ(x)))−μ(x)u(μˉ(x))}−{U∗(u(μ(x)))−μ(x)u(μ(x))},(1)\begin{aligned} d_U\left(\mu(\boldsymbol{x}),\bar{\mu}(\boldsymbol{x})\right) &=d_{U^*}\left(u(\bar{\mu}(\boldsymbol{x})),u(\mu(\boldsymbol{x}))\right)\\ &=U^*\left(u(\bar{\mu}(\boldsymbol{x}))\right)-U^*\left(u(\mu(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})\left\{u(\bar{\mu}(\boldsymbol{x}))-u(\mu(\boldsymbol{x}))\right\}\\ &=\left\{U^*\left(u(\bar{\mu}(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})u(\bar{\mu}(\boldsymbol{x}))\right\}-\left\{U^*\left(u(\mu(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})u(\mu(\boldsymbol{x}))\right\}, \tag*{(1)} \end{aligned}

where U∗(s)=sup⁡z∈R{zs−U(z)}U^*(s)=\sup_{z\in\mathbb{R}}\{zs-U(z)\} is the convex conjugate function of UU, with useful properties such as (U∗)′=u−1(U^*)'=u^{-1}, see [15]. By using (2.1) and defining ℓ(y,x∣μˉ)=U∗(u(μˉ(x)))−y⋅u(μˉ(x))\ell(y,\boldsymbol{x}\mid\bar{\mu})=U^*(u(\bar{\mu}(\boldsymbol{x})))-y\cdot u(\bar{\mu}(\boldsymbol{x})), DUD_U can be calculated as

DU(μ,μˉ)=DU∗(u(μˉ),u(μ))=∫Rd{ℓ(μ(x),x∣μˉ)−ℓ(μ(x),x∣μ)}q(x) dx=∫R×Rd{ℓ(y,x∣μˉ)−ℓ(y,x∣μ)}f(y,x) dy dx.\begin{aligned} D_U(\mu,\bar{\mu})&=D_{U^*}(u(\bar{\mu}),u(\mu))\\ &=\int_{\mathbb{R}^d}\{\ell(\mu(\boldsymbol{x}),\boldsymbol{x}\mid\bar{\mu})-\ell(\mu(\boldsymbol{x}),\boldsymbol{x}\mid\mu)\}q(\boldsymbol{x})\,d\boldsymbol{x}\\ &=\int_{\mathbb{R}\times\mathbb{R}^d}\{\ell(y,\boldsymbol{x}\mid\bar{\mu})-\ell(y,\boldsymbol{x}\mid\mu)\}f(y,\boldsymbol{x})\,dy\,d\boldsymbol{x}. \end{aligned}

Then the empirical version of DUD_U based on the observations X\mathcal{X} is given by

References

  1. [1]Basu, A., Shioya, H. and Park, C. (2011). Statistical Inference: The Minimum Distance Approach. CRC Press.
  2. [2]Bregman, L. M. (1967). A relaxation method of finding a common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7:200–217.DOI
  3. [3]Bühlmann, P., Kalisch, M., and Meier, L. (2014). High-dimensional statistics with a view toward applications in biology. Annual Review of Statistics and Its Applications, 1:255–278.DOI
  4. [4]Bühlmann, P. and Yu, B. (2003). Boosting with the L₂ loss: regression and classification. J. Amer. Statist. Assoc., 98(462):324–339.
  5. [5]Chatterjee, S. and Hadi, A.S. (2006). Regression Analysis by Example. 5th Edition. John Wiley & Sons.
  6. [6]Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794.DOI
  7. [7]Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., Li, M., Xie, J., Lin, M., Geng, Y., Li, Y., Yuan, J., Cortes, D. and XGBoost contributors (2026). xgboost: Extreme Gradient Boosting. https://cran.r-project.org/web/packages/xgboost/index.html.
  8. [8]Dunn, P. K. and Smyth, G. K. (2018). Generalized Linear Models with Examples in R. Springer-Verlag, New York.DOI
  9. [9]Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Ann. Statist., 29(5):1189–1232.DOI
  10. [10]Fan, J., Heckman, N. E., and Wand, M. P. (1995). Local polynomial kernel regression for generalized linear models and quasi-likelihood functions. J. Amer. Statist. Assoc., 90(429):141–150.
  11. [11]Fan, J., Wu, Y. and Feng, Y. (2009). Local quasi-likelihood with a parametric guide. Ann. Statist., 37(6B):4153–4183.DOI
  12. [12]Guo, Y., Hastie, T., and Tibshirani, R. (2007). Regularized linear discriminant analysis and its application in microarrays. Biostatistics, 8(1):86–100.pubmed.ncbi.nlm.nih.gov/16603682
  13. [13]Hastie, T., Qian, J. and Tay, K. (2026). An Introduction to ‘glmnet’ https://glmnet.stanford.edu/articles/glmnet.html
  14. [14]Hastie, T., Tibshirani, R., and Wainwright, M. (2015). Statistical Learning with Sparsity: The Lasso and Generalizations. CRC Press, Boca Raton, FL.
  15. [15]Hiriart-Urruty, J.-B. and Lemaréchal, C. (1993). Convex Analysis and Minimization Algorithms II: Advanced Theory and Bundle Methods. Springer-Verlag, Berlin.
  16. [16]James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning: with Applications in R. 2nd Edition. Springer-Verlag, New York.
  17. [17]Klemelä, J. (2007). Density estimation with stagewise optimization of the empirical risk. Mach. Learn., 67(3):169–195.DOI
  18. [18]Klemelä, J. (2009). Smoothing of Multivariate Data: Density Estimation and Visualization. John Wiley & Sons.
  19. [19]Klemelä, J. (2014). Multivariate Nonparametric Regression and Visualization: With R and Applications to Finance. John Wiley & Sons.
  20. [20]Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference. Springer-Verlag, New York.DOI
  21. [21]McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. 2nd Edition. Chapman & Hall.
  22. [23]Naito, K. (2002). Semiparametric regression with multiplicative adjustment. Comm. Statist. Theory Methods, 31(12):2289–2309.DOI
  23. [24]Naito, K. and Eguchi, S. (2013). Density estimation with minimization of U-divergence. Mach. Learn., 90(1):29–57.DOI
  24. [25]Naito, K. and Penev, S. (2021). Regression using localised functional Bregman divergence. Electron. J. Stat., 15(2):6544–6585.DOI
  25. [26]Nishida, K. and Naito, K. (2024). Kernel density estimation by stagewise algorithm with a simple dictionary. Comput. Statist., 39(2):523–560.
  26. [27]Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B, 58(1):267–288.
  27. [28]Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
  28. [29]van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer-Verlag, New York.
  29. [30]Wainwright, M. J. (2019). High-dimensional Statistics: A Non-asymptotic Viewpoint. Cambridge University Press.
  30. [31]Yeh, I.-C. (1998). Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete Research, 28(12):1797–1808.
  31. [32]Yoshida, T. (2018). Semiparametric method for model structure discovery in additive regression models. Econom. Stat., 5:124–136.DOI
  32. [33]Yoshida, T. and Naito, K. (2019). Regression with stagewise minimization on risk function. Comput. Statist. Data Anal., 134:123–143.DOI
  33. [34]Zhang, C., Jiang, Y., and Shang, Z. (2009). New aspects of Bregman divergence in regression and classification with parametric and nonparametric estimation. Canad. J. Statist., 37(1):119–139.DOI

Paper details

vlnc:W_wpba5hpr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents