WorksMathematics
Second-Order Least Squares Method for Statistical Inference in Linear Models with Asymmetric Errors
Abstract
In a linear regression model with normal random errors, the ordinary least squares estimator is the most efficient estimator. However, when the random errors have an asymmetric distribution, a more efficient estimator can be constructed using the higher moments of the data. In this paper, we use the second-order least squares method to obtain more accurate asymptotic confidence intervals and more powerful hypothesis tests in linear models. We also develop an efficient and stable numerical algorithm to calculate the proposed estimator. Our extensive simulation studies demonstrate the advantages of the proposed method. Two real data examples are presented.
Introduction
Asymmetrically distributed data arise in many fields, including environmental science, finance and economics, engineering, life-time and extreme data analysis, see e.g., Beguería and Vicente-Serrano [2]; Castro-Camilo et al. [4]; Green [8]; Wright et al. [20]; Yousof et al. [23]; Nandy et al. [12]. It is well-known that, in linear regression models, the ordinary least squares estimator (OLSE) is the best unbiased estimator and is most efficient if the model error has a normal distribution. However, if the model error has an asymmetric distribution, then the higher moments of the data can be used to construct more efficient estimators. One such example is the second-order least squares (SLS) estimator (SLSE) that uses the first two conditional moments of the response variable given the predictors. It was shown that this estimator is asymptotically more efficient than the OLSE if the model error has a non-zero third moment and that both estimators are asymptotically equivalent otherwise [19]. Additionally, the SLSE attains the optimal semiparametric efficiency bound [11]. In this paper, we demonstrate, among other things, that the efficiency gain of the SLSE may lead to conclusions different from when the OLSE is used in real data analysis (§ 6).
While the asymptotic theory of the SLSE has been established for general nonlinear regression models [19], its more in-depth properties for statistical inferences in linear models have not been explored. In this paper, we fill in these gaps by investigating more properties of the SLSE in linear models and studying its performance in asymptotic confidence intervals and hypothesis testing problems. Moreover, the theoretical results of Wang and Leblanc [19] were derived for general nonlinear models under some restrictive regularity conditions. Here we show that most of their conditions can be relaxed or removed in linear models, thus making the method more widely applicable. Another challenge with the SLSE is numerical computation, which has not been systematically explored in the literature. In § 4, we develop an iteratively reweighted quasi-Newton method that is relatively easy to implement and efficient in practice, as well as numerically stable (§ 5.2).
The SLS method has been used for parameter estimation problems, including in measurement error models [16, 17], mixed effects models [18], censored linear models [1], and dynamic models [13 b]. More recently, this method has also been used in optimal design problems [7, 3, 22, 21] (He and Yue 2022; He et al. 2024).
The SLS Estimator
Consider the linear model , where and . Given the i.i.d. random sample , , the OLSE minimizes the “first-order” distance
The OLSE for is defined as
which is not obtained through an optimization process.
The SLSE is based on the first two (conditional) moments of given : and , where and are the true parameter values. Specifically, the SLSE for the vector of the unknown parameters is defined as
where and is a nonnegative definite weight matrix that may depend on . Wang and Leblanc [19] showed that the most efficient SLSE is obtained by using the optimal weight matrix , with , which has the following form for linear models:
where , and . In this paper, we will use the term SLSE to refer to the most efficient version of the estimator.
Further, Wang and Leblanc [19] established consistency and asymptotic normality of the SLSE in nonlinear models under general conditions. Their conditions can be substantially simplified for linear models as follows.
A1 The error has finite constant moments , , and .
A2 The predictor has a finite fourth moment .
A3 The true parameter value and the parameter space is compact.
Let and . Then we have the following results, the proofs of which are given in Appendix.
Theorem 1. Under assumptions A1-A3, as , the SLSE satisfies
;
, where
and
From Theorem 1 we see that when and hence , the SLSEs and have the same asymptotic variances as the respective OLSEs: and . However, the following theorem shows that in general the SLSE is asymptotically more efficient than the OLSE when .
Theorem 2. Suppose . Then, under the conditions of Theorem 1,
The matrix is positive definite if and only if ; and is nonnegative definite with rank if .
with equality holding if and only if .
From Theorem 2 we can see that the efficiency gain of the SLSE depends on the value of (see Appendix § 11). For example, the maximum gain is achieved when the data on are centered, so that , in which case and .
On the other hand, if the model has an intercept term, i.e., for the first predictor variable , then and therefore , where is the unit vector with the first element one and others zero. In this case, the matrix have all positive but one zero eigenvalues. In particular, using the block matrix inversion formula we obtain an alternative expression
However, in the case where the model does not have intercept, then and, therefore, all elements of are asymptotically more efficient than the corresponding those of .
Statistical Inference
Although the SLS method has been used in point estimation and experimental design problems in the literature, it has not been used in confidence intervals and hypothesis testing problems. In this section, we use the asymptotic theory of the previous section to construct Wald statistics for statistical inferences. Specifically, by Theorem 1, for each , the -th element of satisfies , where is the -th diagonal element of the estimated version of the matrix of (4), where all moments and true parameter values are replaced with the sample moments and SLSEs, respectively.
Therefore, for any , a level asymptotic confidence interval for is
where is the upper quantile of the standard normal distribution.
Similarly, for testing the hypotheses
where is known, the Wald test statistic is
Hence, for a given significance level , will be rejected if .
Reweighted Quasi-Newton Method
Although the SLSE has desirable statistical properties under skewed error distributions, its numerical computation is not straightforward, as it does not have a closed-form formula as the OLSE. In this section, we develop a numerical algorithm to calculate the SLSE in general linear models, based on the Newton method for minimization of the objective function in (1).
First, the standard Newton procedure is based on the iterative equation
where and are the gradient and Hessian matrix of , respectively:
and
However, the weight matrix of (2) in the gradient and Hessian depends on the unknown parameters , and . To overcome this complication, we propose a numerically simpler and more efficient quasi-Newton method, where these parameters are updated in each iteration by using the current value and the corresponding residuals . This so-called reweighted Newton update equation is given by (9) with
and , . The derivation of (12) and (13) are given in Appendix § 12.
Another challenge is that in the case of small samples, the so-called “bad samples” may cause the Hessian matrix to be near singular. To address this issue, we implemented adaptive ridge regularization and Armijo backtracking line search, which assure the stability of the Hessian and prevent divergence of the algorithm.
First, we initialize and increase it over iteration until becomes positive definite. This way, the descent direction is computed as . Further, to introduce stronger stability and to increase convergence speed, Armijo backtracking line search is utilized as
where is the step size. We start with and decrease it until two conditions (Armijo Conditions) are met:
Boundary constraint: ;
Sufficient decrease: , where c is a small constant ensuring a sufficient reduction.
Finally, we use the OLSE as the starting point of the iteration process. The algorithm of this reweighted quasi-Newton procedure is given below. The numerical stability of the algorithm will be illustrated for two simulation models in § 5.2.
Algorithm: Reweighted Quasi-Newton Method with Line Search
Input: Data , Initial estimate , Convergence tolerance , Max iterations .
Initialize: , .
while and do
Compute Gradient and Hessian using estimated moments .
▷ Adaptive Ridge Regularization
Set .
Repeat
.
If is Positive Definite (via Cholesky): Break
.
Compute Descent Direction .
▷ Armijo Backtracking Line Search
Set .
Repeat
Candidate .
If (variance constraint violated): ; Continue
If : Break (Sufficient Decrease met)
.
Until reaches minimum tolerance.
▷ Update and Iterate
.
.
end while
Output: .
Simulation Studies
In this section, we investigate numerically the finite-sample performance of the SLSE vs. the OLSE for each of the components of , the asymptotic CIs for based on the asymptotic normality of from Theorem 1 vs. the well-known CIs based on , as well as the respective asymptotic tests of the hypotheses in (8) using the corresponding Wald test statistics. In view of that converges to much slower compared to the convergence of to , comparison of the vs. based inference results is omitted in this study.
Simulation models
For our simulation studies, we consider the following two models:
Model 1: with , where , , , and .
Model 2: with , where , , and .
For each of the models, we consider the following three error distributions with the means of zero and the variances of one: ; , called std-Exp(1) hereafter, with the skewness of 2; and , abbreviated as std-Weibull(0.6, 1), with the skewness of 4.59.
In our simulation studies, we use the sample sizes , 300, 500, 1000 and, for some tables, , 5000, 10000, 20000, 50000, and Monte Carlo replications.
We use the large-sample standard errors (ASE) and empirical standard errors (SE) to compare and for each component of . The performance of the respective CIs is studied based on their empirical finite-sample coverage probabilities (CP) and the ratio of their empirical expected lengths, namely,
The performance of the and based Wald tests for the hypotheses in (8) is evaluated through their empirical power functions.
Numerical stability of the SLSE
Before studying the performance of the SLSE for in statistical inference, we address numerical stability of our proposed reweighted quasi-Newton algorithm for computing .
Under the standard reweighted Newton–Raphson algorithm (without regularization or line search), a large proportion of simulation runs resulted in failure due to instability in Hessian inversion. This issue was particularly severe in Model 2 for smaller sample sizes, where the rate of such “bad samples” reached as high as 37.14% under the std-Weibull(0.6, 1) error distribution for and was at 5.96% for this skewed distribution even for , as shown in Table 1.
| Reweighted Newton | Reweighted quasi–Newton | ||||||||||
| Model | Error distribution | ||||||||||
| Model 1 | N(0,1) | 13.76% | 3.47% | < 0.01% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| Model 1 | std-Exp(1) | 8.22% | 0.73% | < 0.01% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| Model 1 | std-Weibull(0.6, 1) | 2.81% | < 0.01% | < 0.01% | 0% | 0% | < 0.01% | 0% | 0% | 0% | 0% |
| Model 2 | N(0,1) | 16.54% | 7.15% | 0.02% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| Model 2 | std-Exp(1) | 28.41% | 13.22% | 2.64% | 0.52% | 0.02% | 0% | 0% | 0% | 0% | 0% |
| Model 2 | std-Weibull(0.6, 1) | 37.14% | 30.37% | 20.43% | 14.43% | 5.96% | < 0.01% | < 0.01% | 0% | 0% | 0% |
Table 1. Bad sample rates when computing the SLSE using the standard reweighted Newton method and § 4 reweighted quasi–Newton method.
By introducing the reweighted quasi-Newton method with an Armijo-type line search, these failures were effectively eliminated. As seen from Table 1, across all simulation settings for both Models 1 and 2, the bad sample rate was reduced dramatically, dropping to essentially 0%. This highlights the importance of regularization and adaptive step-size control in ensuring reliable implementation of the SLSE, particularly in smaller samples and under asymmetric error distributions.
Comparison of the OLS and SLS based statistical inference
Table 2 and Table 3 below for Models 1 and 2, respectively, with three different error distributions, contain the values of the OLSE and SLSE, their respective ASEs and SEs, the CPs of the corresponding 95% CIs, and the ratio of (14), for each .
| Error distribution | Parameter | OLS | SLS | ||||||||
| OLSE | ASE | SE | CP | SLSE | ASE | SE | CP | ||||
| N(0,1) | 100 | 0.5041 | 0.3238 | 0.3236 | 0.9432 | 0.5044 | 0.3147 | 0.3287 | 0.9320 | 1.0292 | |
| 1.0001 | 0.0175 | 0.0178 | 0.9400 | 1.0001 | 0.0170 | 0.0182 | 0.9294 | 1.0307 | |||
| -1.5006 | 0.0508 | 0.0505 | 0.9430 | -1.5007 | 0.0493 | 0.0513 | 0.9318 | 1.0307 | |||
| 300 | 0.4999 | 0.1854 | 0.1852 | 0.9481 | 0.4998 | 0.1836 | 0.1857 | 0.9448 | 1.0095 | ||
| 0.9999 | 0.0100 | 0.0101 | 0.9467 | 0.9999 | 0.0099 | 0.0101 | 0.9430 | 1.0100 | |||
| -1.4999 | 0.0290 | 0.0290 | 0.9515 | -1.4999 | 0.0287 | 0.0291 | 0.9471 | 1.0100 | |||
| 500 | 0.5005 | 0.1435 | 0.1456 | 0.9457 | 0.5005 | 0.1427 | 0.1460 | 0.9435 | 1.0057 | ||
| 1.0000 | 0.0078 | 0.0077 | 0.9505 | 1.0000 | 0.0077 | 0.0077 | 0.9477 | 1.0060 | |||
| -1.4999 | 0.0224 | 0.0228 | 0.9447 | -1.4999 | 0.0223 | 0.0229 | 0.9428 | 1.0060 | |||
| 1000 | 0.5014 | 0.1014 | 0.1019 | 0.9503 | 0.5014 | 0.1011 | 0.1020 | 0.9499 | 1.0028 | ||
| 0.9999 | 0.0055 | 0.0055 | 0.9529 | 0.9999 | 0.0055 | 0.0055 | 0.9519 | 1.0030 | |||
| -1.5001 | 0.0158 | 0.0159 | 0.9488 | -1.5001 | 0.0158 | 0.0159 | 0.9474 | 1.0030 | |||
| std-Exp(1) | 100 | 0.4957 | 0.3216 | 0.3278 | 0.9483 | 0.4972 | 0.2245 | 0.2282 | 0.9448 | 1.4329 | |
| 1.0000 | 0.0174 | 0.0175 | 0.9497 | 0.9999 | 0.0115 | 0.0117 | 0.9480 | 1.5165 | |||
| -1.4994 | 0.0504 | 0.0518 | 0.9432 | -1.4995 | 0.0332 | 0.0342 | 0.9430 | 1.5165 | |||
| 300 | 0.5012 | 0.1851 | 0.1858 | 0.9467 | 0.4994 | 0.1327 | 0.1336 | 0.9495 | 1.3945 | ||
| 0.9999 | 0.0100 | 0.0100 | 0.9486 | 1.0000 | 0.0068 | 0.0069 | 0.9489 | 1.4702 | |||
| -1.5000 | 0.0290 | 0.0292 | 0.9466 | -1.4999 | 0.0197 | 0.0199 | 0.9504 | 1.4701 | |||
| 500 | 0.4976 | 0.1432 | 0.1450 | 0.9473 | 0.4992 | 0.1036 | 0.1038 | 0.9484 | 1.3822 | ||
| 1.0001 | 0.0077 | 0.0078 | 0.9487 | 1.0000 | 0.0053 | 0.0054 | 0.9475 | 1.4548 | |||
| -1.4998 | 0.0224 | 0.0225 | 0.9479 | -1.5000 | 0.0154 | 0.0154 | 0.9502 | 1.4548 | |||
| 1000 | 0.4992 | 0.1013 | 0.1004 | 0.9501 | 0.4995 | 0.0739 | 0.0742 | 0.9508 | 1.3710 | ||
| 1.0000 | 0.0055 | 0.0055 | 0.9488 | 1.0000 | 0.0038 | 0.0038 | 0.9476 | 1.4408 | |||
| -1.4999 | 0.0158 | 0.0158 | 0.9473 | -1.5000 | 0.0110 | 0.0110 | 0.9512 | 1.4408 | |||
| std-Weibull(0.6,1) | 100 | 0.5014 | 0.3130 | 0.3245 | 0.9435 | 0.4994 | 0.1801 | 0.1869 | 0.9357 | 1.7374 | |
| 0.9998 | 0.0169 | 0.0177 | 0.9454 | 1.0000 | 0.0086 | 0.0089 | 0.9499 | 1.9546 | |||
| -1.4999 | 0.0490 | 0.0510 | 0.9478 | -1.4999 | 0.0251 | 0.0264 | 0.9425 | 1.9548 | |||
| 300 | 0.4965 | 0.1827 | 0.1871 | 0.9448 | 0.4979 | 0.1126 | 0.1149 | 0.9429 | 1.6217 | ||
| 1.0002 | 0.0099 | 0.0099 | 0.9505 | 1.0001 | 0.0055 | 0.0055 | 0.9548 | 1.7843 | |||
| -1.4999 | 0.0285 | 0.0292 | 0.9478 | -1.5000 | 0.0160 | 0.0163 | 0.9496 | 1.7843 | |||
| 500 | 0.5010 | 0.1424 | 0.1443 | 0.9478 | 0.5011 | 0.0904 | 0.0906 | 0.9534 | 1.5752 | ||
| 1.0000 | 0.0077 | 0.0078 | 0.9464 | 1.0000 | 0.0045 | 0.0046 | 0.9493 | 1.7178 | |||
| -1.5000 | 0.0223 | 0.0225 | 0.9500 | -1.5001 | 0.0130 | 0.0130 | 0.9521 | 1.7178 | |||
| 1000 | 0.5000 | 0.1010 | 0.1010 | 0.9469 | 0.5008 | 0.0662 | 0.0666 | 0.9489 | 1.5269 | ||
| 1.0000 | 0.0055 | 0.0055 | 0.9515 | 1.0000 | 0.0033 | 0.0033 | 0.9534 | 1.6501 | |||
| -1.5000 | 0.0158 | 0.0159 | 0.9495 | -1.5001 | 0.0096 | 0.0097 | 0.9483 | 1.6502 |
Table 2. OLSE vs. SLSE, and 95% OLS CI vs. 95% SLS CI for Model 1.
| Error distribution | Parameter | OLS OLSE | OLS ASE | OLS SE | OLS CP | SLS SLSE | SLS ASE | SLS SE | SLS CP | ||
| 100 | 0.4916 | 0.5185 | 0.5271 | 0.9442 | 0.4902 | 0.5023 | 0.5329 | 0.9328 | 1.0321 | ||
| 1.0008 | 0.0778 | 0.0785 | 0.9464 | 1.0010 | 0.0755 | 0.0794 | 0.9356 | 1.0307 | |||
| N(0,1) | -1.4972 | 0.1088 | 0.1107 | 0.9402 | -1.4974 | 0.1053 | 0.1120 | 0.9304 | 1.0328 | ||
| 300 | 0.4996 | 0.2961 | 0.2960 | 0.9462 | 0.4991 | 0.2928 | 0.2980 | 0.9424 | 1.0112 | ||
| 0.9999 | 0.0444 | 0.0444 | 0.9489 | 1.0000 | 0.0439 | 0.0446 | 0.9445 | 1.0107 | |||
| -1.4999 | 0.0621 | 0.0628 | 0.9477 | -1.4999 | 0.0614 | 0.0631 | 0.9444 | 1.0114 | |||
| 500 | 0.4992 | 0.2290 | 0.2300 | 0.9461 | 0.4989 | 0.2275 | 0.2309 | 0.9449 | 1.0068 | ||
| 1.0002 | 0.0344 | 0.0343 | 0.9522 | 1.0002 | 0.0342 | 0.0344 | 0.9498 | 1.0065 | |||
| -1.4996 | 0.0480 | 0.0481 | 0.9511 | -1.4996 | 0.0477 | 0.0483 | 0.9477 | 1.0069 | |||
| 1000 | 0.4991 | 0.1617 | 0.1633 | 0.9495 | 0.4991 | 0.1612 | 0.1637 | 0.9477 | 1.0033 | ||
| 1.0000 | 0.0243 | 0.0243 | 0.9496 | 1.0000 | 0.0242 | 0.0243 | 0.9481 | 1.0032 | |||
| -1.5001 | 0.0339 | 0.0340 | 0.9503 | -1.5001 | 0.0338 | 0.0340 | 0.9488 | 1.0034 | |||
| std-Exp(1) | 100 | 0.4938 | 0.5149 | 0.5180 | 0.9479 | 0.4989 | 0.3489 | 0.3591 | 0.9424 | 1.4760 | |
| 1.0001 | 0.0771 | 0.0779 | 0.9438 | 0.9991 | 0.0544 | 0.0560 | 0.9399 | 1.4180 | |||
| -1.4996 | 0.1078 | 0.1113 | 0.9436 | -1.4997 | 0.0717 | 0.0751 | 0.9391 | 1.5040 | |||
| 300 | 0.4980 | 0.2940 | 0.2966 | 0.9456 | 0.5007 | 0.2044 | 0.2095 | 0.9416 | 1.4390 | ||
| 0.9997 | 0.0441 | 0.0445 | 0.9449 | 0.9989 | 0.0319 | 0.0328 | 0.9428 | 1.3860 | |||
| -1.5006 | 0.0617 | 0.0622 | 0.9517 | -1.4997 | 0.0422 | 0.0426 | 0.9472 | 1.4650 | |||
| 500 | 0.4987 | 0.2286 | 0.2291 | 0.9512 | 0.4990 | 0.1602 | 0.1622 | 0.9460 | 1.4270 | ||
| 1.0003 | 0.0343 | 0.0343 | 0.9469 | 0.9997 | 0.0249 | 0.0250 | 0.9500 | 1.3760 | |||
| -1.4998 | 0.0480 | 0.0481 | 0.9481 | -1.4997 | 0.0330 | 0.0333 | 0.9482 | 1.4520 | |||
| 1000 | 0.5000 | 0.1615 | 0.1619 | 0.9490 | 0.4997 | 0.1146 | 0.1164 | 0.9440 | 1.4090 | ||
| 1.0001 | 0.0243 | 0.0241 | 0.9513 | 0.9998 | 0.0178 | 0.0177 | 0.9504 | 1.3600 | |||
| -1.4999 | 0.0339 | 0.0336 | 0.9471 | -1.4998 | 0.0236 | 0.0237 | 0.9500 | 1.4330 | |||
| std-Weibull(0.6,1) | 100 | 0.5047 | 0.5016 | 0.5191 | 0.9491 | 0.5112 | 0.2728 | 0.3222 | 0.9251 | 1.8390 | |
| 0.9992 | 0.0752 | 0.0776 | 0.9408 | 1.0000 | 0.0438 | 0.0506 | 0.9228 | 1.7160 | |||
| -1.5004 | 0.1051 | 0.1104 | 0.9435 | -1.5001 | 0.0553 | 0.0655 | 0.9260 | 1.9020 | |||
| 300 | 0.4946 | 0.2918 | 0.2973 | 0.9456 | 0.4981 | 0.1715 | 0.1790 | 0.9438 | 1.7010 | ||
| 1.0004 | 0.0438 | 0.0442 | 0.9496 | 1.0001 | 0.0274 | 0.0283 | 0.9412 | 1.6020 | |||
| -1.4993 | 0.0612 | 0.0610 | 0.9517 | -1.5001 | 0.0350 | 0.0361 | 0.9479 | 1.7520 | |||
| 500 | 0.4992 | 0.2262 | 0.2271 | 0.9502 | 0.4981 | 0.1372 | 0.1395 | 0.9499 | 1.6490 | ||
| 0.9997 | 0.0340 | 0.0341 | 0.9495 | 0.9996 | 0.0218 | 0.0220 | 0.9518 | 1.5600 | |||
| -1.4999 | 0.0475 | 0.0485 | 0.9473 | -1.4997 | 0.0280 | 0.0288 | 0.9457 | 1.6950 | |||
| 1000 | 0.5019 | 0.1611 | 0.1634 | 0.9461 | 0.5010 | 0.1014 | 0.1022 | 0.9480 | 1.5880 | ||
| 0.9997 | 0.0242 | 0.0244 | 0.9468 | 0.9998 | 0.0160 | 0.0162 | 0.9496 | 1.5090 | |||
| -1.4997 | 0.0338 | 0.0341 | 0.9468 | -1.4996 | 0.0208 | 0.0212 | 0.9489 | 1.6280 |
Table 3. OLSE vs. SLSE, and 95% OLS CI vs. 95% SLS CI for Model 2.
Notably, across all simulation settings for both models, is greater than 1, indicating that the SLSE based CI is shorter on average than the OLSE based CI. For the N(0, 1) distributed errors, is very close to one, which agrees with the fact that the OLSE and SLSE estimators for are equally efficient for symmetrically distributed errors, as noted right after Theorem 1. However, for the std-Exp(1) and more skewed std-Weibull(0.6, 1) errors, the SLSE based CI is significantly shorter than the OLSE CI, more so for the std-Weibull(0.6, 1) errors. Moreover, as illustrated in Table 4, due to the very slow rate of convergence of for these two error distributions, this advantage of the SLSE based CI continues to hold at least up to the sample size of 50000. Furthermore, we can also see that the efficiency gain of the SLSE is greater in Model 2 without the intercept, which is consistent with our earlier discussions after Theorem 2 in § 2.
| Model, error distribution | Parameter | 50 | 100 | 300 | 500 | 1000 | 5000 | 10000 | 20000 | 50000 |
| Model 1, Exp(1) | 1.456 | 1.424 | 1.401 | 1.384 | 1.369 | 1.352 | 1.354 | 1.353 | 1.351 | |
| 1.542 | 1.505 | 1.477 | 1.457 | 1.438 | 1.417 | 1.419 | 1.418 | 1.415 | ||
| 1.542 | 1.505 | 1.477 | 1.457 | 1.438 | 1.417 | 1.419 | 1.418 | 1.415 | ||
| Model 1, Weibull(0.6,1) | 1.826 | 1.742 | 1.629 | 1.573 | 1.529 | 1.453 | 1.429 | 1.421 | 1.397 | |
| 2.088 | 1.959 | 1.796 | 1.715 | 1.653 | 1.550 | 1.517 | 1.507 | 1.475 | ||
| 2.089 | 1.958 | 1.795 | 1.715 | 1.653 | 1.550 | 1.517 | 1.507 | 1.475 | ||
| Model 2, Exp(1) | 1.503 | 1.475 | 1.434 | 1.414 | 1.402 | 1.390 | 1.387 | 1.382 | 1.382 | |
| 1.442 | 1.415 | 1.382 | 1.365 | 1.355 | 1.343 | 1.341 | 1.337 | 1.337 | ||
| 1.532 | 1.501 | 1.460 | 1.439 | 1.426 | 1.412 | 1.409 | 1.404 | 1.404 | ||
| Model 2, Weibull(0.6,1) | 1.947 | 1.829 | 1.708 | 1.654 | 1.590 | 1.497 | 1.480 | 1.454 | 1.443 | |
| 1.808 | 1.711 | 1.608 | 1.563 | 1.511 | 1.433 | 1.419 | 1.397 | 1.388 | ||
| 2.016 | 1.895 | 1.757 | 1.700 | 1.630 | 1.528 | 1.509 | 1.482 | 1.470 |
Table 4. Empirical expected length ratio of (14) of the and based CIs across sample sizes.
As for the empirical coverages of the OLSE and SLSE based CIs, they appear to converge fast for both models. Moreover, they are essentially on par and are both close enough to 95% for in Table 2 and Table 3. For , the coverage of the SLSE based CI is somewhat lower than that of the OLSE based CI, by at most 1.12% in Table 2 and 2.4% in Table 3, while not getting less than 92.94% for Model 1 and 92.28% for Model 2.
The superior asymptotic efficiency of the SLSE for skewed error distributions can also be seen through its lower ASEs for the std-Exp(1) and std-Weibull(0.6,1) errors in Table 2 and Table 3. The empirical SEs of the SLSE, which align closely with its ASEs, are also lower than the SEs of the OLSE for these error distributions, which indicates a higher precision of the SLSE in finite samples.
Table 5 and Table 6 contain the empirical power function comparison for the OLSE and SLSE based tests of the hypotheses in (8) for Models 1 and 2, respectively. Both tests have well-controlled empirical type I errors that are close enough to the nominal 5% for , except for a few cases for the std-Weibull(0.6, 1) errors and and 500 for Model 2, where one or both errors are between 6.2% and 6.8%. For and all three parameters , , and for both models, the type I errors of the SLSE based test are somewhat higher than those of the OLSE based test, by at most 1.6% in Table 5 and 2.04% in Table 6, where the type I errors of the SLSE based test do not exceed 6.9% and 7.86%, respectively.
| Parameter | Test value | OLS | SLS | OLS | SLS | OLS | SLS | OLS | SLS |
| std-Exp(1) | |||||||||
| 0.50 | 0.0538 | 0.0628 | 0.0488 | 0.0472 | 0.0483 | 0.0509 | 0.0554 | 0.0530 | |
| 0.72 | 0.1194 | 0.2000 | 0.2385 | 0.3981 | 0.3579 | 0.5852 | 0.5884 | 0.8366 | |
| 0.94 | 0.3104 | 0.5162 | 0.6665 | 0.8975 | 0.8668 | 0.9827 | 0.9880 | 0.9999 | |
| 1.00 | 0.0544 | 0.0564 | 0.0515 | 0.0453 | 0.0499 | 0.0499 | 0.0528 | 0.0507 | |
| 1.02 | 0.2238 | 0.4272 | 0.5160 | 0.8396 | 0.7381 | 0.9637 | 0.9576 | 0.9993 | |
| 1.04 | 0.6434 | 0.9264 | 0.9752 | 0.9998 | 0.9990 | 1.0000 | 1.0000 | 1.0000 | |
| -1.50 | 0.0530 | 0.0690 | 0.0521 | 0.0518 | 0.0526 | 0.0505 | 0.0546 | 0.0510 | |
| -1.45 | 0.1704 | 0.3376 | 0.4165 | 0.7239 | 0.6011 | 0.8971 | 0.8817 | 0.9950 | |
| -1.40 | 0.5188 | 0.8328 | 0.9270 | 0.9978 | 0.9924 | 0.9999 | 1.0000 | 1.0000 | |
| std-Weibull(0.6, 1) | |||||||||
| 0.50 | 0.0576 | 0.0604 | 0.0544 | 0.0500 | 0.0530 | 0.0500 | 0.0492 | 0.0512 | |
| 0.72 | 0.1544 | 0.3066 | 0.2900 | 0.5040 | 0.4040 | 0.6920 | 0.5940 | 0.8940 | |
| 0.94 | 0.3636 | 0.6712 | 0.6680 | 0.9440 | 0.8800 | 0.9840 | 0.9900 | 1.0000 | |
| 1.00 | 0.0480 | 0.0514 | 0.0510 | 0.0506 | 0.0570 | 0.0496 | 0.0484 | 0.0504 | |
| 1.02 | 0.2580 | 0.6516 | 0.5400 | 0.9350 | 0.7610 | 0.9890 | 0.9550 | 0.9990 | |
| 1.04 | 0.6862 | 0.9820 | 0.9740 | 0.9980 | 0.9960 | 1.0000 | 1.0000 | 1.0000 | |
| -1.50 | 0.0504 | 0.0588 | 0.0452 | 0.0486 | 0.0496 | 0.0498 | 0.0540 | 0.0554 | |
| -1.45 | 0.2150 | 0.5516 | 0.4600 | 0.8920 | 0.6000 | 0.9520 | 0.9020 | 1.0000 | |
| -1.40 | 0.5660 | 0.9510 | 0.9260 | 0.9980 | 0.9920 | 1.0000 | 1.0000 | 1.0000 |
Table 5. Power function comparison for the OLSE and SLSE based tests of the hypotheses in (8) for Model 1.
| Parameter | Test value | ||||||||
| OLS | SLS | OLS | SLS | OLS | SLS | OLS | SLS | ||
| 0.50 | 0.0580 | 0.0640 | 0.0478 | 0.0529 | 0.0508 | 0.0549 | 0.0490 | 0.0512 | |
| 0.72 | 0.0854 | 0.1182 | 0.1239 | 0.2078 | 0.1651 | 0.2960 | 0.2873 | 0.5025 | |
| 0.94 | 0.1586 | 0.2742 | 0.3407 | 0.5911 | 0.4944 | 0.7880 | 0.7823 | 0.9662 | |
| 1.00 | 0.0504 | 0.0612 | 0.0512 | 0.0564 | 0.0497 | 0.0554 | 0.0494 | 0.0523 | |
| 1.02 | 0.0642 | 0.0846 | 0.0801 | 0.1124 | 0.0916 | 0.1399 | 0.1345 | 0.2167 | |
| 1.04 | 0.0978 | 0.1446 | 0.1586 | 0.2614 | 0.2234 | 0.3771 | 0.3832 | 0.6170 | |
| -1.50 | 0.0508 | 0.0564 | 0.0531 | 0.0543 | 0.0485 | 0.0546 | 0.0538 | 0.0483 | |
| -1.45 | 0.0768 | 0.1148 | 0.1251 | 0.2196 | 0.1767 | 0.3226 | 0.3155 | 0.5648 | |
| -1.40 | 0.1572 | 0.2890 | 0.3637 | 0.6592 | 0.5437 | 0.8559 | 0.8429 | 0.9893 | |
| 0.50 | 0.0562 | 0.0728 | 0.0580 | 0.0520 | 0.0480 | 0.0660 | 0.0520 | 0.0460 | |
| 0.72 | 0.0962 | 0.1850 | 0.1640 | 0.3260 | 0.1880 | 0.4160 | 0.3200 | 0.5880 | |
| 0.94 | 0.1860 | 0.4370 | 0.4140 | 0.7560 | 0.5300 | 0.8600 | 0.7920 | 0.9920 | |
| 1.00 | 0.0582 | 0.0786 | 0.0680 | 0.0680 | 0.0540 | 0.0520 | 0.0580 | 0.0400 | |
| 1.02 | 0.0822 | 0.1272 | 0.0820 | 0.1470 | 0.1080 | 0.1720 | 0.1530 | 0.2690 | |
| 1.04 | 0.1238 | 0.2198 | 0.1940 | 0.3780 | 0.2400 | 0.4720 | 0.3640 | 0.7020 | |
| -1.50 | 0.0508 | 0.0684 | 0.0480 | 0.0540 | 0.0640 | 0.0620 | 0.0460 | 0.0540 | |
| -1.45 | 0.0760 | 0.1616 | 0.1300 | 0.3080 | 0.1660 | 0.4520 | 0.3480 | 0.6940 | |
| -1.40 | 0.1712 | 0.4680 | 0.3960 | 0.8120 | 0.5340 | 0.9340 | 0.8580 | 0.9920 |
Table 6. Power function comparison for the OLSE and SLSE based tests of the hypotheses in (8) for Model 2.
In order to compare the powers of the OLSE and SLSE based tests under the two-sided alternative hypotheses in (8), we consider two values of for each of . Notably, the SLSE based test is significantly more powerful than the OLSE based one across all settings in Table 5 and Table 6. Both test powers increase with sample size and are higher for the values of that are further away from the respective true parameter values. For for Model 1, both powers are essentially at 100% for that are further away from the true , while for the other value of , Table 5 shows that the power of the OLSE based test is still behind that of the SLSE based one, by as much as 30% for the std-Weibull(0.6,1) errors and . Compared to Model 1, both the OLSE and SLSE based tests are less powerful across all settings for Model 2, where the power convergence for both tests appears to be slower and only the SLSE based test power reaches almost 100% at and only for and . Just like for Model 1, the power superiority of the SLSE based test is still considerable across all sample sizes, where the latter test is as much as 34.6% more powerful for the std-Weibull(0.6, 1) errors and .
Real data application
In this section, we apply the SLS method to two real data examples that have been used in the literature.
Australian Institute of Sports (AIS) – Biomedicine
The Australian Institute of Sports (AIS) dataset contains biomedical measurements of 706 athletes from 12 sports, collected in 1991 [15, 6]. A subset of 202 observations, available in the R sn package, has been widely used for modeling with skewed error distributions. Following the approach of Nandy et al. [12], we model plasma ferritin concentration (Fe) using body mass index (BMI) and lean body mass (LBM) as covariates:
We first fitted the model using the OLS method and obtained the residuals. The histogram and Q-Q plot of the residuals in Figure 1 clearly indicate the right-skewness of the error distribution. Subsequently, we fitted the model again using the SLS method.
Figure 1. The OLS residual distribution of the AIS dataset.
Tables 7 and 8 present the estimates, standard errors, and 95% asymptotic confidence intervals for the intercept and coefficients and using the OLSEs and SLSEs. In all cases, the SLSE produces shorter CIs, indicating improved estimation precision. For instance, the interval length for the intercept decreases from 101.34 (OLSE) to 74.35 (SLSE), for (BMI) from 6.20 to 4.52, and for (LBM) from 1.36 to 0.99. These reductions reflect the variance-reducing nature of the SLSE. Although the SLSE point estimates differ from their OLSE counterparts, all estimates are within the corresponding CIs. However, a striking point is that the OLSE of the coefficient of BMI is insignificant at 5% level because the corresponding 95% CI covers zero, while the coefficient is significant by the SLSE. This means that based on the OLSE one would conclude that BMI has no impact on plasma ferritin concentration, while it does have impact based on the SLSE.
| Parameter | Estimate | SE | 95% CI | CI Length |
| (Intercept) | -30.8571 | 25.8511 | [-81.5253, 19.8110] | 101.3363 |
| (BMI) | 2.5653 | 1.5816 | [-0.5347, 5.6652] | 6.1999 |
| (LBM) | 0.7529 | 0.3466 | [0.0737, 1.4322] | 1.3585 |
Table 7. The OLSE estimates, standard errors, 95% asymptotic CIs, and CI lengths for the ASI dataset.
| Parameter | Estimate | SE | 95% CI | CI Length |
| (Intercept) | -12.6490 | 18.9672 | [-49.8247, 24.5267] | 74.3514 |
| (BMI) | 2.3743 | 1.1530 | [0.1144, 4.6342] | 4.5198 |
| (LBM) | 0.5398 | 0.2526 | [0.0447, 1.0350] | 0.9904 |
Table 8. The SLSE estimates, standard errors, 95% asymptotic CIs, and CI lengths for the ASI dataset.
Health insurance
A publicly available dataset containing information on 1,338 individuals from the USA was used to compare the performance of the OLSE and SLSE for statistical inference [5]. The dataset includes variables such as age, sex, body mass index (BMI), number of children, smoking status, region of residence, and medical charges. In this analysis, medical charges were treated as the response variable. All predictor variables were standardized to improve computational efficiency. The main goal was to estimate the regression coefficients () and their variances using both OLS and SLS, and to compare the resulting confidence intervals.
Since the dataset includes categorical variables, appropriate encoding was applied before fitting the models. Specifically, sex was encoded as 0 for female and 1 for male; smoker was coded as 0 for yes and 1 for no; and region was numerically encoded as 0 for northwest, 1 for northeast, 2 for southwest, and 3 for southeast.
| Variable | Description | Type | Values |
| age | Age of primary beneficiary | quantitative | – |
| sex | Gender of insurance contractor | qualitative | female, male |
| bmi | Body Mass Index (BMI) | quantitative | – |
| children | Number of children covered by insurance | qualitative | 0, 1, 2, ... |
| smoker | Smoking status | qualitative | yes, no |
| region | Residential area in the U.S. | qualitative | north/south-east/west |
| charges | Individual medical costs billed by insurance | quantitative | – |
Table 9. Description of the variables in the health insurance dataset.
Similarly to the previous example, a linear model was first fitted using the OLS method. Again, the OLS residual plots in Figure 2 clearly indicate the right-skewness of the error distribution. Hence, the model was fitted again using the SLS method.
Figure 2. The OLS residual distribution for the health insurance data.
Table 10 and Table 11 show the OLSE and SLSE based estimates of the regression coefficients, standard errors, 95% asymptotic confidence intervals, and interval lengths across the six covariates. In all cases, the SLSE produced narrower CIs than the OLSE, indicating greater precision. For instance, from Table 10, the CI for the age coefficient under the OLSE is [0.2711, 0.3252] with a length of 0.0541, while, as shown in Table 11, the SLSE yields a tighter interval of [0.2878, 0.3190], with a length of just 0.0312.
| Covariate | Coef. Estimate | SE | 95% CI | CI Length |
| age | 0.2981 | 0.0138 | [0.2711, 0.3252] | 0.0541 |
| sex | -0.0053 | 0.0137 | [-0.0323, 0.0216] | 0.0539 |
| bmi | 0.1697 | 0.0143 | [0.1417, 0.1977] | 0.0560 |
| children | 0.0466 | 0.0137 | [0.0198, 0.0735] | 0.0537 |
| smoker | -0.7956 | 0.0138 | [-0.8226, -0.7687] | 0.0539 |
| region | -0.0278 | 0.0142 | [-0.0557, 0.0000] | 0.0557 |
Table 10. The OLSE estimates, standard errors, 95% asymptotic CIs, and CI lengths for the regression coefficients for the health insurance dataset.
| Covariate | Coef. Estimate | SE | 95% CI | CI Length |
| age | 0.3034 | 0.0080 | [0.2878, 0.3190] | 0.0312 |
| sex | -0.0041 | 0.0079 | [-0.0196, 0.0115] | 0.0311 |
| bmi | 0.0722 | 0.0082 | [0.0561, 0.0884] | 0.0323 |
| children | 0.0425 | 0.0079 | [0.0270, 0.0580] | 0.0310 |
| smoker | -0.5950 | 0.0079 | [-0.6105, -0.5794] | 0.0311 |
| region | -0.0177 | 0.0082 | [-0.0337, -0.0016] | 0.0321 |
Table 11. The SLSE estimates, standard errors, 95% asymptotic CIs, and CI lengths for the regression coefficients for the health insurance dataset.
This trend is consistent across other predictors, including sex, BMI, number of children, smoker status, and region. Notably, the estimated effect of BMI (correlation of 0.2 with charges) under the SLSE is substantially smaller (0.0722) compared to the OLSE (0.1697), yet the SLSE based CI remains narrower. This suggests that the SLSE may be less influenced by multicollinearity. A similar pattern is observed with the smoker variable (correlation of 0.79 with charges), where both methods detect a strong negative effect, but the SLSE estimate is more moderate (-0.5950 vs. -0.7956). Similarly to the previous example, the OLSE indicates that regional variation has weak or no impact on medical charges, while it has a clearly negative impact by the SLSE.
Overall, these results support the use of the SLSE for inference in this context. The tighter confidence intervals and more stable estimates suggest that the SLSE may offer improved robustness, particularly when the classical OLS assumptions, such as normality or homoscedasticity, are potentially violated.
Acknowledgments
The authors gratefully acknowledge financial support by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the University of Manitoba Graduate Fellowship (UMGF).
Disclosure Statement
No potential conflict of interest was reported by the author(s).
Preliminary
Wang and Leblanc [19] proved the consistency and asymptotic normality of the SLSE in general nonlinear models. In the case of linear models, many of their conditions and derivations are not required and, therefore, can be substantially simplified. The following lemmas are from Amemiya (1973) or Wang and Leblanc [19]. Let be a measurable function of with respect to Lebesgue measure and continuous in , where and is compact.
Lemma 1. If, as , , where is a non-stochastic function and has a unique minimum at , then
Lemma 2. If, as , and is an interior point of , then for any sequence of estimators it holds .
Proof of Theorem 1
(1) First, denote and
where , , . Then it is easy to see that the quadratic form
can be expanded in terms of the powers of and and the highest orders in this expansion are , , and . Further, since appears only in the coefficients of these terms and the parameter space is compact, by assumption A2 it holds . It follows from the uniform law of large numbers (ULLN) that , where (Amemiya, 1985; Jennrich, 1969). In order to show that has a unique minimizer at the true value , note that, for any ,
It follows that , with equality holding if and only if
Since is positive definite with probability one, the above equality holds if and only if
with probability one, implying and . Finally, it follows from Lemma 1 that .
(2) First, by Taylor expansion of the score function, we have
where ,
and
Similar to the proof (1), we can show that , where , and hence, by the ULLN and Lemma 2, it holds
where the last equality holds because the second order derivative does not depend on and hence the second term has expectation zero. It follows from (16) that
Further, since , by the Central Limit Theorem, , where
Thus, it follows from (17) that . Finally, using the expression
we have
Therefore the asymptotic covariance matrix can be obtained by using the block matrix inversion formula.
Proof of Theorem 2
(1). First, since , by A1 we have . Further, since
and , we have
which is nonnegative definite, where is the covariance matrix of . It follows that is nonnegative definite. Further, since
and has the same nonzero eigenvalue as , , and therefore , is positive definite if and only if .
(2). First, since
we have . It follows that
with equality holding if and only if $\mu_x'M_{xx}^{-1}\mu_x=1.
Derivation of the gradient and Hessian matrix of
We derive the gradient and Hessian for the given data . To evaluate the gradient, we first have
and therefore
Further, to evaluate the Hessian, we have
Since
we have
and therefore
It follows that
and
References
- [1]Abarin, T. and L. Wang (2009). Second-order least squares estimation of censored regression models. Journal of Statistical Planning and Inference 139(2), 125–135.DOI
- [2]Beguería, S. and S. M. Vicente-Serrano (2006). Mapping the hazard of extreme rainfall by peaks over threshold extreme value analysis and spatial regression techniques. Journal of applied meteorology and climatology 45(1), 108–124.DOI
- [3]Bose, M. and R. Mukerjee (2015). Optimal design measures under asymmetric errors, with application to binary design points. Journal of Statistical Planning and Inference 159, 28–36.arxiv.org/abs/1405.3126
- [4]Castro-Camilo, D., R. Huser, and H. Rue (2022). Practical strategies for generalized extreme value-based regression models for extremes. Environmetrics 33(6), e2742.
- [5]Choi, M. (2018). Medical cost personal datasets. Kaggle dataset. Accessed 2026-06-16.
- [6]Cook, R. D. and S. Weisberg (1994). An Introduction to Regression Graphics. New York: John Wiley & Sons.
- [7]Gao, L. L. and J. Zhou (2014). New optimal design criteria for regression models with asymmetric errors. Journal of Statistical Planning and Inference 149, 140–151.DOI
- [8]Green, P. J. (1984). Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. Journal of the Royal Statistical Society: Series B (Methodological) 46(2), 149–170.
- [11]Kim, M. and Y. Ma (2012). The efficiency of the second-order nonlinear least squares estimator and its extension. Annals of the Institute of Statistical Mathematics 64(4), 751–764.pubmed.ncbi.nlm.nih.gov/23794748
- [12]Nandy, A., A. Basu, and A. Ghosh (2025). Robust inference for linear regression models with possibly skewed error distribution. Journal of Computational and Applied Mathematics 463, 116502.arxiv.org/abs/2404.03404
- [13]Salamh, M. and L. Wang (2021a). Second-order least squares estimation in nonlinear time series models with arch errors. Econometrics 9(4), 41.DOI
- [15]Telford, R. D. and R. B. Cunningham (1991). Sex, sport, and body-size dependency of hematology in highly trained athletes. Medicine and Science in Sports and Exercise 23(7), 788–794.pubmed.ncbi.nlm.nih.gov/1921671
- [16]Wang, L. (2003). Estimation of nonlinear berkson-type measurement error models. Statistica Sinica 13(4), 1201–1210.
- [17]Wang, L. (2004). Estimation of nonlinear models with berkson measurement errors. The Annals of Statistics 32(6), 2559–2579.
- [18]Wang, L. (2007). A unified approach to estimation of nonlinear mixed effects and berkson measurement error models. Canadian Journal of Statistics 35(2), 233–248.DOI
- [19]Wang, L. and A. Leblanc (2008). Second-order nonlinear least squares estimation. Annals of the Institute of Statistical Mathematics 60(4), 883–900.DOI
- [20]Wright, B., P. Green, and P. Braiden (1982). Quantitative analysis of delayed fracture observed in stress rate tests on brittle materials. Journal of Materials Science 17(11), 3227–3234.
- [21]Yeh, C.-K. and J. Zhou (2021). Properties of optimal regression designs under the second-order least squares estimator. Statistical Papers 62(1), 75–92.
- [22]Yin, Y. and J. Zhou (2017). Optimal designs for regression models using the second-order least squares estimator. Statistica Sinica, 1841–1856.DOI
- [23]Yousof, H., S. Jahanshahi, T. Ramires, G. Aryal, and G. Hamedani (2018). A new distribution for extreme values: regression model, characterizations and applications. Journal of Data Science 16(4), 677–706.

