WorksMathematics
Global Convergence of Multilevel Gradient-Regularized Newton Methods
Abstract
We develop and analyze multilevel gradient-regularized Newton methods for unconstrained convex optimization with Lipschitz-continuous Hessians. By computing Newton corrections in lower-dimensional subspaces, the proposed methods aim to reduce computational costs while preserving the global convergence rate in function values of the full-space method. To establish this rate for multilevel methods, we introduce a class of row-nested restriction operators that ensures monotonicity of the reduced-gradient norms across levels. Numerical experiments with row-nested operators demonstrate convergence behavior comparable to full-space gradient-regularized Newton, while offering substantial computational savings on large-scale instances.
Introduction
Cubic regularization of Newton’s method [22, 20] is a fundamental second-order approach for smooth, possibly nonconvex optimization, achieving an complexity for first-order stationarity in nonconvex problems and a global function-value rate for convex objectives with Lipschitz-continuous Hessians. Subsequent developments include accelerated cubic Newton methods attaining an rate [9] and Adaptive Regularization with Cubics (ARC) [18, 21, 24]. However, cubic regularization requires solving a potentially expensive subproblem at each iteration.
Gradient-regularized Newton methods provide a simpler alternative, with iterations of the form
where . Following the early work of Polyak [23], Mishchenko [13] and Doikov and Nesterov [14] established that choosing proportional to yields the global function-value rate for convex objectives with Lipschitz-continuous Hessians, improving upon the rate of standard gradient descent. Nevertheless, computing full-dimensional Hessians and solving the corresponding linear systems can be prohibitively expensive for large-scale problems.
Inspired by classical multigrid methods [26], multilevel optimization reduces these costs by computing corrections in lower-dimensional spaces, replacing full-dimensional Newton systems with smaller ones. Multilevel strategies have been successfully developed for Newton and regularized Newton methods [1, 6, 3, 10, 5], higher-order regularization methods, including ARC [15], as well as line-search, accelerated gradient, and proximal methods [16, 4, 25].
In a foundational work, Gratton, Sartenaer, and Toint [1] introduced recursive multilevel trust-region (RMTR) methods, establishing an worst-case complexity bound for first-order stationarity in possibly nonconvex problems. Their framework permits recursion to a coarser level when
where is the gradient of the current level-dependent model, is a prescribed parameter, and restricts information to the next coarser level. Importantly, this eligibility condition is evaluated before computing a coarse correction, avoiding additional trial steps for level selection.
In general, second- or higher-order multilevel methods need not inherit the fast convergence rates of their full-space counterparts, since coarse corrections exploit only partial second- or higher-order information. Notable results preserving full-space convergence guarantees include RMTR [1] and the high-order multilevel methods of Calandra et al. [15], which attain the first-order stationarity complexity of cubic regularization for possibly nonconvex problems. More recently, for convex objectives, the multilevel gradient-regularized Newton method [12] was shown to achieve the global function-value rate, while the adaptive multilevel Newton method [19] achieves local quadratic convergence for strongly convex objectives.
However, the deterministic formulations of these last two methods rely on a stronger level-selection criterion, namely
The second condition requires computing a trial coarse step to evaluate the gradient at the prospective next iterate, potentially increasing the computational cost. In contrast, the classical eligibility criterion (2) uses only current-iterate information. To the best of our knowledge, a direct global convergence analysis for deterministic multilevel gradient-regularized Newton methods employing such a pre-step level-selection criterion has remained unavailable. Establishing this rate will provide theoretical justification for employing multilevel methods in convex optimization, combining the fast convergence guarantees of full-space Newton methods with the potential computational savings of lower-dimensional corrections.
Contributions. We develop multilevel gradient-regularized Newton methods for convex objectives with Lipschitz-continuous Hessians, exploiting lower-dimensional subspaces and the classical level-selection criterion (2) (cf. [1]). We show that these methods preserve the global function-value rate of their full-space counterpart. For two levels, Theorem 2.5 establishes this rate under arbitrary coarse/fine alternation. For multiple levels with general restriction operators, Theorem 2.5 requires an explicit fine-iteration frequency condition, which is eventually satisfied by the algorithm (§ 3.1) but may be restrictive during coarse-dominated phases. To eliminate this condition, we introduce row-nested restriction operators (§ 3.2), whose structure guarantees monotonicity of reduced-gradient norms across levels. Exploiting this property, we establish the global rate without imposing any fine-iteration frequency condition (Theorem 3.8).
Our analysis builds on the ideas of [13], but addresses a fundamental difficulty absent from full-space methods: coarse and fine corrections control different gradient norms, whose convergence properties cannot be directly transferred across levels. To overcome this, we introduce a level-dependent weighted gradient measure that provides uniform control across all possible coarse-to-fine, coarse-to-coarse, and fine-to-coarse transitions. This, together with the proposed class of row-nested restriction operators (def:nested-restrictions), constitutes the main theoretical novelty of our analysis, enabling us to preserve the convergence rate of the full-space method without imposing any condition on the frequency of fine-level iterations.
We support our theoretical analysis with numerical experiments comparing the proposed methods against the full-space gradient-regularized Newton method. The results demonstrate that the two-level and row-nested multilevel methods exhibit convergence behavior comparable to that of their full-dimensional counterpart, while achieving substantial reductions in computational time on large-scale instances. Furthermore, the experiments highlight the importance of the restriction operator structure, with row-nested hierarchies generally exhibiting more favorable convergence behavior than their non-nested counterparts. These findings suggest that row-nested restriction operators can offer an attractive choice from both theoretical and computational perspectives.
Related work
Early multilevel optimization frameworks were developed by Nash [2] and Lewis and Nash [29], including implementations based on truncated-Newton methods. A common feature of subsequent multilevel methods [1, 11, 3, 15] is the use of coarse-level eligibility conditions of the form (2). Wen and Goldfarb [4] developed a line-search multigrid method establishing R-linear convergence under strong convexity and suitable conditions on the search directions. Calandra et al. [15] extended higher-order regularization to multilevel optimization. Other multilevel Newton-type methods [3, 5, 6] establish global convergence or fast local rates under specific assumptions. However, unlike our approach, these Newton-type methods do not employ gradient-dependent Hessian regularization to handle potentially singular Hessians, nor do they establish the global function-value rate for general convex objectives.
Tsipinakis and Parpas [12] developed the multilevel gradient-regularized Newton method achieving the global function-value rate for convex objectives. However, unlike the methods proposed in this paper, their method requires the look-ahead condition (3), introducing additional computational costs. This condition avoids directly analyzing transitions between coarse and fine gradient measures and thus their proof cannot be applied to our setting. In contrast, our analysis preserves the global rate under classical level selection without requiring look-ahead trial steps.
Nested subspaces are widely used in Krylov-based trust-region methods, reduced-basis approximation, and multilevel optimization [27, 28, 11]. The row-nested restriction operators considered here additionally guarantee monotonicity of the restricted-gradient norms across levels. Moreover, the weighted-gradient analysis is closely related to multiple-Lyapunov-function techniques for switched systems [30, 31]. In our setting, we combine a suitably weighted gradient measure with regularized Newton decrease bounds to establish a global convergence rate in function values under arbitrary coarse/fine alternation.
Main assumptions and notation
We consider the unconstrained optimization problem
where is a twice continuously differentiable convex function. Given an initial point , our goal is to develop iterative methods achieving the global convergence rate in function values. Throughout, we impose the following assumptions.
Assumption 1.1. The initial sublevel set is bounded.
Since is continuous, is compact, ensuring the existence of a global minimizer . We denote .
Assumption. There exists an such that
These assumptions are standard in the analysis of second-order optimization methods; see, e.g., [7]. Throughout, denotes the Euclidean norm for vectors and the induced spectral norm for matrices. For convenience, we write
Moreover, by compactness of , the constants
are finite.
Organization. In § 2, we introduce the two-level gradient-regularized Newton method and establish its convergence rate. In § 3, we extend the analysis to multiple coarse levels, establishing convergence guarantees under a fine-iteration frequency condition and, alternatively, without this condition using row-nested restriction operators introduced in § 3.2. Numerical experiments comparing the proposed methods with full-space gradient-regularized Newton are presented in § 4. Finally, § 5 concludes the paper.
The two-level Newton method with gradient regularization
We consider the Newton method with gradient regularization that alternates between the fine and one coarse level. For the two-level algorithm we consider a single restriction operator that transfers the information from fine to coarse space, i.e., , where are the coarse space dimensions. Moreover, the prolongation operator is defined in a standard way, i.e., . For the operators we take the standard assumptions (see for instance [1, 3]).
Assumption. Let be a full-row-rank restriction operator, with , and let be its corresponding prolongation operator. We define
Let and . We define the sets of coarse and fine iterations
We can consider as the set of effective coarse iterations and the set of fine iterations. Thus, if then the algorithm will perform a coarse iteration otherwise a fine iteration. The two-level gradient-regularized Newton algorithm is presented in alg:two-level.
\caption{Two-Level Regularized Newton (TLRN)}
\label{alg:two-level}
\begin{algorithmic}
\REQUIRE $x_0 \in \mathbb{R}^n, R : \mathbb{R}^n \to \mathbb{R}^d, \sigma \in (0, \min\{1, \omega \}), \varepsilon > 0$.
\ENSURE Sequence of iterates $(x_0,x_1,x_2,\ldots)$.
\FOR{$k = 0, 1, \ldots$}
\IF{$\nabla f_k = 0$} \STATE \textbf{stop} \ENDIF
\STATE Set {\begin{equation*}
x_{k+1} \coloneqq
\begin{cases}
x_k - P \left(R \nabla^2 f_kP + \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}} I_d \right)^{-1} R \nabla f_k, & k \in \mathcal{C}, \\
x_k - \left( \nabla^2 f_k+ \sqrt{\frac{L \left\lVert \nabla f_k \right\rVert}{2}} I_n \right)^{-1} \nabla f_k, & k \in \mathcal{F}.
\end{cases}
\end{equation*}}
\ENDFOR
\end{algorithmic}
The following result will be needed for the analysis of alg:two-level; it has been proved in [12 Lemma 3.3] and [13 Lemma 3].
The following stability estimates for the coarse and fine corrections were also established in [12] and [13], and hold for all coarse and fine updates, respectively:
Convergence analysis
When , then alg:two-level coincides with the gradient regularization of Newton’s method (see [14, 13]) which converges with the fast rate. In what follows we show that this rate can be attained when the sequence is always generated from the coarse space, i.e.,
For the purposes of this result, the condition is therefore not imposed.
Proof. Fix and set . We distinguish two cases.
Case I: . By the definition of and (5),
Therefore,
Using convexity and the coarse-level condition,
Thus, in this case the function value gap decreases geometrically and, in particular, is .
Case II: . Let , enumerate and set . By ineq:reduction-coarse and the definition of , for ,
Since , it follows that for Moreover, by ineq:reduction-coarse, the sequence is non-increasing. Hence,
Combining this with convexity, i.e., , we obtain
where
Hence, prop:recurrence-rate yields
Combining the two cases proves the result. ◻
prop:rate-coarse-steps analyzes the limiting case in which only coarse corrections are performed and shows that the coarse-only iteration retains the rate of the full-space regularized Newton method. Its proof adapts the strategy of [13 Theorem 1] to the reduced-gradient setting. A key feature of this analysis is that every iteration controls the same quantity, namely , so the corresponding estimates can be propagated consistently along the sequence.
The situation is fundamentally different for alg:two-level. Coarse iterations naturally control , whereas fine iterations control , and neither of these quantities is necessarily contractive when the algorithm switches to the other level. Thus, the fact that both the coarse-only and fine-only schemes achieve an rate does not by itself imply convergence, or the same rate, for an arbitrary mixture of the two iteration types. The main difficulty is therefore not the behavior within either level, but the transitions between them.
The following lemma provides the cross-level stability estimate needed to bridge this mismatch: although a coarse correction need not decrease the full gradient norm, it can increase it by at most a uniform factor.
Proof. Denote
Next, we have
By cor:f-nonincreasing, . Since is convex, is convex, and hence for every . Therefore,
Then, by using the definitions of and as well as and the symmetric positive semi-definiteness of , we obtain
◻
lemma:bounded-gradient-in-coarse-level shows that a coarse correction is stable with respect to the full gradient, in the sense that the gradient norm can increase by at most a uniformly bounded factor. Together with the level-specific stability estimates ineq:reduction-coarse and ineq:reduction-fine, this bound will allow us to construct a level-dependent weighted gradient measure that controls transitions between the two levels.
Theorem 2.5. Let be convex and suppose that Assumptions 1.1, (?), and (?) hold. Then the sequence generated by Algorithm alg:two-level satisfies
Proof. To compare the reduced and full gradient norms across changes of level, we introduce a level-dependent weighted gradient measure. The weights are chosen so that all four possible coarse/fine transitions admit uniform contraction or stability bounds. Choose
and set . Since implies , we get
Moreover, define the scalar
and the sets
We set
Now, choose and fix a . We distinguish two cases.
Case I: . Define
For , we show that . Let . Then, for and we have four possibilities:
. Then,
. Then, with similar arguments we obtain
\begin{align*} V_{m+1} &\overset{m+1 \in \mathcal{F}}{=} b \left\lVert \nabla f_{m+1} \right\rVert \overset{\text{Lem.}~\ref{lemma:bounded-gradient-in-coarse-level}}{\leq} b M_\mathcal{C}\left\lVert \nabla f_{m} \right\rVert \\ &\quad \ {\leq} \frac{b M_\mathcal{C}}{\sigma} \left\lVert R\nabla f_{m} \right\rVert \overset{m \in \mathcal{C}}{\leq} \frac{t M_\mathcal{C}}{\sigma} V_m. \end{align*}. Then,
. Then,
Hence,
Now let . We again consider four cases as above:
. Then,
\begin{equation*} V_{m+1} = a \left\lVert R \nabla f_{m+1} \right\rVert \overset{\eqref{ineq:reduced-grad-bound}}{\le} 2 V_m. \end{equation*}. Then,
. Then,
\begin{equation*} V_{m+1} = b \left\lVert \nabla f_{m+1} \right\rVert \overset{\eqref{ineq:full-grad-bound}}{\le} 2 V_m. \end{equation*}. Then,
As a result,
Let . Thus, we have
and hence, by the definition of , . Combining now (8) and (9) we obtain
Furthermore, for any , we have that
Then, by convexity and Assumption 1.1, we obtain
and thus the desired rate of convergence is established when .
Case II: . Let and enumerate , with . Let . Thus, . If , then
Using similar arguments, if , then
and thus, since , we get
for all . Since , it follows that for Following the same argument as in Case II of prop:rate-coarse-steps, monotonicity of the function values and convexity yield
where
Hence, prop:recurrence-rate and give
which completes the proof of Case II.
Combining now the results of the two cases we have
as required. ◻
Theorem 2.5 shows that the use of coarse corrections does not deteriorate the global convergence rate of the regularized Newton method. In particular, Algorithm alg:two-level achieves the rate in function values while allowing arbitrary switching between coarse and fine iterations according to the level-selection criterion. A key feature of the analysis is that coarse and fine iterations naturally control different quantities, namely and , respectively. The proof resolves this mismatch through a level-dependent weighted gradient measure that provides uniform control across all possible coarse/fine transitions: it contracts on iterations outside , while its growth on the remaining iterations is uniformly bounded. Consequently, the analysis does not require the full gradient norm to decrease at every coarse iteration; uniform stability of the coarse corrections is sufficient. This provides a mechanism for preserving convergence rates in multilevel schemes even when different iteration types control different measures of stationarity.
The multilevel Newton method with gradient regularization
In this section we consider the gradient regularization of Newton’s method that alternates between coarse levels (plus the fine level) when generating the updates. In particular, we consider restriction operators as in [1], but unlike their recursive framework, our operators map directly between each coarse level and the full space, allowing coarse corrections to be transferred directly to the fine level. More precisely, for each , let be a restriction operator, where . Further, is the prolongation operator at level . We will see that this nonrecursive formulation simplifies both the algorithm and its convergence analysis while retaining the same computational cost at the th coarse level.
Accordingly, (?) is now replaced by the following assumption.
Assumption. For each , let be a full-row-rank restriction operator, with , and let be its corresponding prolongation operator. We define
Let and . For each , the sets of coarse and fine iterations are now defined as follows. First define the eligible sets
The set contains the iterations for which level is eligible. For the multilevel method to become efficient, we need to select the lowest-dimensional eligible level
Then the sets are pairwise disjoint and, whenever ,
Moreover, we set
The multilevel gradient-regularized Newton method is presented in alg:multilevel.
\caption{MultiLevel Regularized Newton (MLRN)}
\label{alg:multilevel}
\begin{algorithmic}
\REQUIRE $x_0 \in \mathbb{R}^n, \ell \in \mathbb{N}, R_i : \mathbb{R}^n \to \mathbb{R}^{n_i}$ for $i \in \mathcal{J}$, $\sigma\in \left(0, \min\left\{1,\min_{i\in\mathcal J}\omega_i\right\}\right), \varepsilon > 0$.
\ENSURE Sequence of iterates $(x_0,x_1,x_2,\ldots)$.
\FOR{$k = 0, 1, \ldots$}
\IF{$\nabla f_k = 0$} \STATE \textbf{stop} \ENDIF
\STATE Set \begin{equation*}
x_{k+1} \coloneqq
\begin{cases}
x_k - P_i
\left(
R_i \nabla^2 f_kP_i
+
\sqrt{\dfrac{\omega_i^3 L \left\lVert R_i \nabla f_k \right\rVert}{2}}\, I_{n_i}
\right)^{-1}
R_i \nabla f_k,
& k \in \mathcal{C}_i, i\in\mathcal{J}, \\[2ex]
x_k -
\left(
\nabla^2 f_k
+
\sqrt{\dfrac{L\left\lVert \nabla f_k \right\rVert}{2}}\, I_n
\right)^{-1}
\nabla f_k,
& k \in \mathcal{F}^{(\ell)}.
\end{cases}
\end{equation*}
\ENDFOR
\end{algorithmic}
The next lemma shows that alg:multilevel generates a non-increasing sequence of function values. It also shows that it guarantees the reduced-gradient stability of a coarse correction step. These results arise by [12].
Proof. The proof is outlined in § 6.1. ◻
lemma:multilevel-coarse-estimates guarantees that , and by (6), is non-increasing. In particular, this means that the multilevel iterates stay in . Finally, in lemma:finite-coarse we show that alg:multilevel performs only finitely many coarse iterations.
Conditional rate of convergence
Let . Similar to the analysis of the two-level method, we will need the following definitions,
where . We further set
Hence, the disjointness of the sets , , and gives
The following lemma shows the rate of alg:multilevel when is sufficiently large.
Proof. The proof is given in § 6.2. ◻
Further, the coarse correction is also stable with respect to the full gradient. In particular, by lemma:bounded-gradient-in-coarse-level there exist , , such that
In what follows we show that alg:multilevel converges with when it performs a sufficiently large numbersufficiently of fine steps.
Theorem 2.5. Suppose that Assumptions 1.1, (?), and (?) hold. Let Let and choose Suppose also that, for all sufficiently large , Then the sequence generated by alg:multilevel satisfies
Proof. Fix a sufficiently large such that , and set , If , then lemma:case-II yields
It remains to consider the case For every , exactly one of the following possibilities occurs:
If , then
If , then, by (6),
If , then
Since iterating these bounds gives
Since we obtain
where
Moreover,
by the choice of , and hence .
Therefore, by convexity and Assumption 1.1,
Combining the two cases proves the result. ◻
Theorem 2.5 provides a quantitative condition under which the multilevel scheme retains the convergence rate of the underlying full-space method. Although this rate need not be inherited by an arbitrary multilevel realization, the theorem shows that the rate is preserved whenever a sufficiently large fraction of the iterations is performed on the fine level. Such preservation of complexity has also been established for other multilevel high-order regularization frameworks; see, e.g., [15] for the multilevel ARC method and possibly non-convex functions. To the best of our knowledge, Theorem 2.5 is the first result showing that a classical multilevel gradient-regularized Newton method can retain the global function value rate of the full-space method. This is possible to achieve under an explicit quantitative condition on the proportion of fine-level iterations.
We emphasize that the fine-iteration frequency condition is not restrictive. Indeed, the eligibility condition implies that only finitely many coarse iterations can occur, see lemma:finite-coarse for details. Consequently, the algorithm eventually performs exclusively fine-level iterations, and the required frequency condition is automatically satisfied for all sufficiently large . Nevertheless, this condition is not computationally irrelevant. For instance, for large and , and yield approximately and , respectively. This indicates that arbitrarily long sequences of coarse corrections cannot not occur before the algorithm enters the fine-only regime. In § 3.3 we develop analysis establishing an explicit convergence bound valid at every iteration, without relying on the eventual transition to the fine-only regime, and which also allows the method to perform arbitrarily long sequences of coarse corrections.
Row-nested restriction operators
In the previous subsection, we established an convergence rate under the condition that a sufficiently large fraction of the iterations is performed on the fine level. Such a condition may lead to a large number of computationally expensive fine-level iterations before alg:multilevel enters its fine-level regime. Here we introduce an additional structure on the restriction operators and discuss its most important properties. This new structure will, later, allow us to show a global rate for alg:multilevel without this condition.
Nested hierarchies of coarse spaces are standard in multigrid and have also been employed in multilevel optimization; see, e.g., [11, 17]. Here we impose a more specific algebraic structure: all restriction operators act on the same fine space, and the rows defining a lower-dimensional restriction are retained at every subsequent level. This row-nested construction yields the monotonicity for , which will play a central role in our convergence analysis.
For every there exists such that
Consequently, for every ,
and therefore,
Further, def:nested-restrictions implies nestedness in multilevel spaces. By the definition of we have
Therefore, since ,
and thus, as in classical multilevel methods with nested subspaces, each successive level enriches the previous coarse correction space with additional directions. However, the stronger row-nested structure additionally implies the norm monotonicity (13). There also is a corresponding reverse nesting of null spaces, namely
where the inclusion is strict since . Thus, larger coarse levels miss fewer directions of the fine-space gradient.
def:nested-restrictions yields properties that significantly simplify the multilevel algorithm and provides us with further insights. In particular,
and the admissible range for simplifies to Moreover, (13) implies that the eligible sets satisfy
Hence,
and,
Therefore, means that level is the lowest-dimensional level satisfying the eligibility criterion at iteration . Every level is then automatically eligible, whereas no level is.
Convergence analysis using the row-nested restriction operators
We now establish an convergence rate for alg:multilevel under the row-nested structure of def:nested-restrictions, without imposing any condition on the frequency of fine-level iterations. The analysis extends the weighted-gradient argument developed for the two-level method. An additional difficulty arises from transitions between different coarse levels, since the corresponding updates control distinct reduced-gradient norms . For general restriction operators, these quantities need not be comparable across levels. The row-nested structure provides the necessary norm comparisons, which, together with suitable level-dependent weights, allow us to control transitions among all coarse and fine levels.
Theorem 3.8. Let f be convex and suppose that Assumptions 1.1 and (?) hold. If the sequence is generated by alg:multilevel with row-nested operators for which (?) hold, then
Proof. The proof makes use of the set and as previously we split the analysis into two cases, according to the magnitude of .
Recall that, due to the row-nestedness, , and define . We choose
where
and note that and . Moreover, for , we define
therefore , and note also that , if , and , if .
Let us fix a . If , then lemma:case-II provides the desired rate of convergence. Assume now that . Define the weight gradient measure for coarse levels
Thus, by the definition of and ,
Next, it holds
We first examine the case , where, in particular, we will show that is a contraction. To achieve this, in comparison to the proof of Theorem 2.5, we need to examine an extra case which concerns the transitions between the coarse levels:
, . We analyze this case by splitting into additional three subcases according to the ordering of the coarse levels:
. Then, by the row-nestedness (13), we have
where the last inequality uses the fact that is non-increasing for .
. Then,
where we have use the uniform stability of the coarse corrections, i.e., with defined in Theorem 2.5.
. Then we immediately have
Combining the above subcases yields
where the last equality hold by the choice on . This concludes the first case of arbitrary transitions between coarse spaces.
Then
. Then,
, then we directly obtain
Therefore, by the definition of ,
which shows that is a contraction for elements of .
Assume now that . Similarly, we split our arguments into four cases, to obtain a recurrent relation for . However, since contains a few elements only, is not required to be a contraction.
. Then,
where the last inequality uses the monotonicity of , .
. Then,
. Then, by (6),
. Then,
As a result, the above four cases imply
Let us denote
By the choices of and it easy to see that and since we have . Hence, by (16) and (17), we obtain
which shows that is a contraction. Now the desired result follows from the last inequality, convexity, and (15) as follows. Let
Then
which shows exponential rate for the case . Hence the two cases imply
which concludes the proof. ◻
Theorem 3.8 establishes a direct convergence bound for the multilevel method with row-nested restriction operators, without imposing any condition on the frequency of fine-level iterations. Although the asymptotic rate also follows for general restriction operators from the eventual fine-only behavior, the present analysis controls the convergence rate throughout the entire sequence, including coarse-dominated phases. In particular, the resulting bound does not depend on the iteration at which the method enters the fine-only regime.
The main difficulty for achieving this result is that a coarse correction at level controls , but not necessarily the reduced gradient at the subsequently selected level. Row-nestedness resolves this mismatch through monotonicity of the reduced-gradient norms, which, together with level-dependent weights and uniform full-gradient stability, allows us to control arbitrary transitions between levels. The resulting weighted gradient measure contracts outside a set of non--contracting iterations, while its growth remains uniformly bounded otherwise. When non--contracting iterations are sufficiently frequent, the function-value decrease estimates instead yield the rate. Thus, the analysis accommodates arbitrarily long sequences of coarse corrections without requiring a prescribed frequency of fine iterations or a decrease of the full gradient at every coarse step.
Experiments
\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=483,
ymin=3.114e-11, ymax=9.808e+00,
ytickten={-10,-8,-6,-4,-2,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$},
legend to name=leg:logsum]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 2.94392 \\ 1 2.47479 \\ 2 2.19928 \\ 3 1.98376 \\ 4 1.79968 \\ 5 1.62039 \\
6 1.44752 \\ 7 1.26373 \\ 8 1.15339 \\ 9 1.00203 \\ 10 0.912407 \\ 11 0.830229 \\
12 0.780814 \\ 13 0.694941 \\ 14 0.648902 \\ 15 0.602882 \\ 16 0.577572 \\ 17 0.519541 \\
18 0.495368 \\ 19 0.460214 \\ 20 0.444787 \\ 21 0.416234 \\ 22 0.403257 \\ 23 0.385474 \\
24 0.379241 \\ 25 0.370261 \\ 26 0.353854 \\ 27 0.346997 \\ 28 0.33437 \\ 29 0.328108 \\
30 0.316451 \\ 31 0.312097 \\ 32 0.305135 \\ 33 0.297026 \\ 34 0.29301 \\ 35 0.287836 \\
36 0.280994 \\ 37 0.277909 \\ 38 0.273738 \\ 39 0.268201 \\ 40 0.264626 \\ 41 0.260734 \\
42 0.258182 \\ 43 0.254717 \\ 44 0.250655 \\ 45 0.247916 \\ 46 0.245113 \\ 47 0.242819 \\
48 0.240288 \\ 49 0.237976 \\ 50 0.235205 \\ 51 0.232127 \\ 52 0.230043 \\ 53 0.228007 \\
54 0.207965 \\ 55 0.205501 \\ 56 0.201717 \\ 57 0.200045 \\ 58 0.198764 \\ 59 0.196233 \\
60 0.194468 \\ 61 0.192817 \\ 62 0.191417 \\ 63 0.190035 \\ 64 0.174654 \\ 65 0.172679 \\
66 0.169392 \\ 67 0.168424 \\ 68 0.166709 \\ 69 0.164867 \\ 70 0.163728 \\ 71 0.162761 \\
72 0.161016 \\ 73 0.159865 \\ 74 0.15875 \\ 75 0.157827 \\ 76 0.156896 \\ 77 0.155412 \\
78 0.154409 \\ 79 0.153468 \\ 80 0.141548 \\ 81 0.140002 \\ 82 0.137064 \\ 83 0.136318 \\
84 0.134977 \\ 85 0.13359 \\ 86 0.132705 \\ 87 0.132028 \\ 88 0.130804 \\ 89 0.129946 \\
90 0.12916 \\ 91 0.128506 \\ 92 0.127859 \\ 93 0.126852 \\ 94 0.126144 \\ 95 0.125497 \\
96 0.117108 \\ 97 0.116067 \\ 98 0.113885 \\ 99 0.113086 \\ 100 0.112536 \\ 101 0.111542 \\
102 0.110854 \\ 103 0.110237 \\ 104 0.109709 \\ 105 0.109241 \\ 106 0.108465 \\ 107 0.10789 \\
108 0.107378 \\ 109 0.106932 \\ 110 0.10651 \\ 111 0.106034 \\ 112 0.105452 \\ 113 0.104894 \\
114 0.0982914 \\ 115 0.0973747 \\ 116 0.095532 \\ 117 0.0949204 \\ 118 0.0944805 \\ 119 0.0937065 \\
120 0.0931906 \\ 121 0.0927493 \\ 122 0.0923735 \\ 123 0.0919879 \\ 124 0.0914189 \\ 125 0.0910079 \\
126 0.0906601 \\ 127 0.0858601 \\ 128 0.0851369 \\ 129 0.0835472 \\ 130 0.083206 \\ 131 0.082597 \\
132 0.0820216 \\ 133 0.0816437 \\ 134 0.081373 \\ 135 0.0808834 \\ 136 0.0805168 \\ 137 0.0802061 \\
138 0.0799318 \\ 139 0.0796579 \\ 140 0.0792521 \\ 141 0.0789488 \\ 142 0.0786921 \\ 143 0.0750417 \\
144 0.0744425 \\ 145 0.0730551 \\ 146 0.072768 \\ 147 0.0722212 \\ 148 0.0717229 \\ 149 0.0713873 \\
150 0.0711528 \\ 151 0.0707287 \\ 152 0.0704119 \\ 153 0.0701432 \\ 154 0.0699064 \\ 155 0.0696657 \\
156 0.0693151 \\ 157 0.0690542 \\ 158 0.0688336 \\ 159 0.0659578 \\ 160 0.0654489 \\ 161 0.0643576 \\
162 0.0641027 \\ 163 0.0636522 \\ 164 0.0632351 \\ 165 0.0629552 \\ 166 0.0627503 \\ 167 0.0623944 \\
168 0.0621294 \\ 169 0.0619074 \\ 170 0.0617107 \\ 171 0.0614992 \\ 172 0.0612061 \\ 173 0.0609884 \\
174 0.0608058 \\ 175 0.0606402 \\ 176 0.0604377 \\ 177 0.0601745 \\ 178 0.0599808 \\ 179 0.0598182 \\
180 0.057461 \\ 181 0.0570283 \\ 182 0.0561403 \\ 183 0.0559303 \\ 184 0.0555632 \\ 185 0.0552332 \\
186 0.0550864 \\ 187 0.0548166 \\ 188 0.0545559 \\ 189 0.0543675 \\ 190 0.0542064 \\ 191 0.0540492 \\
192 0.053836 \\ 193 0.0536315 \\ 194 0.0534839 \\ 195 0.0533537 \\ 196 0.0531988 \\ 197 0.0530113 \\
198 0.0528289 \\ 199 0.0512099 \\ 200 0.0508777 \\ 201 0.0502265 \\ 202 0.0500613 \\ 203 0.049776 \\
204 0.049518 \\ 205 0.0493422 \\ 206 0.0492081 \\ 207 0.048983 \\ 208 0.0488127 \\ 209 0.0486703 \\
210 0.0485435 \\ 211 0.0484059 \\ 212 0.0482159 \\ 213 0.0480725 \\ 214 0.0479513 \\ 215 0.0478414 \\
216 0.0477102 \\ 217 0.0475368 \\ 218 0.0474072 \\ 219 0.0472973 \\ 220 0.0460273 \\ 221 0.0457478 \\
222 0.0452269 \\ 223 0.045087 \\ 224 0.044847 \\ 225 0.0446333 \\ 226 0.0444855 \\ 227 0.0443695 \\
228 0.0441794 \\ 229 0.044033 \\ 230 0.0439106 \\ 231 0.0438016 \\ 232 0.0436826 \\ 233 0.0435194 \\
234 0.0433951 \\ 235 0.0432901 \\ 236 0.0431953 \\ 237 0.0430814 \\ 238 0.0429321 \\ 239 0.0428195 \\
240 0.0427245 \\ 241 0.0416482 \\ 242 0.0413876 \\ 243 0.0409363 \\ 244 0.0408141 \\ 245 0.0406019 \\
246 0.0404157 \\ 247 0.0402869 \\ 248 0.0401844 \\ 249 0.04002 \\ 250 0.0398927 \\ 251 0.0397869 \\
252 0.039693 \\ 253 0.0395889 \\ 254 0.0394483 \\ 255 0.039341 \\ 256 0.0392511 \\ 257 0.0391703 \\
258 0.0390712 \\ 259 0.0389432 \\ 260 0.0388471 \\ 261 0.0387667 \\ 262 0.0379002 \\ 263 0.0376777 \\
264 0.0373165 \\ 265 0.0372169 \\ 266 0.0370386 \\ 267 0.0368846 \\ 268 0.036778 \\ 269 0.0366907 \\
270 0.036554 \\ 271 0.0364476 \\ 272 0.0363599 \\ 273 0.0362824 \\ 274 0.0361934 \\ 275 0.0360755 \\
276 0.0359862 \\ 277 0.035912 \\ 278 0.0352463 \\ 279 0.0350498 \\ 280 0.0347437 \\ 281 0.0346556 \\
282 0.0344951 \\ 283 0.0343575 \\ 284 0.0342622 \\ 285 0.0341829 \\ 286 0.0340596 \\ 287 0.0339638 \\
288 0.0338849 \\ 289 0.0338155 \\ 290 0.0337345 \\ 291 0.0336275 \\ 292 0.0335469 \\ 293 0.0334802 \\
294 0.0328925 \\ 295 0.0327146 \\ 296 0.03244 \\ 297 0.0323594 \\ 298 0.0322117 \\ 299 0.0320856 \\
300 0.0319985 \\ 301 0.0319248 \\ 302 0.0318113 \\ 303 0.0317234 \\ 304 0.0316636 \\ 305 0.0315619 \\
306 0.0314758 \\ 307 0.0314059 \\ 308 0.0313448 \\ 309 0.0312766 \\ 310 0.0311838 \\ 311 0.0311125 \\
312 0.0310536 \\ 313 0.0304766 \\ 314 0.0303118 \\ 315 0.030054 \\ 316 0.0299818 \\ 317 0.0298472 \\
318 0.0297323 \\ 319 0.0296534 \\ 320 0.0295869 \\ 321 0.0294848 \\ 322 0.0294054 \\ 323 0.0293404 \\
324 0.0292836 \\ 325 0.0292154 \\ 326 0.0291265 \\ 327 0.02906 \\ 328 0.0290053 \\ 329 0.0284955 \\
330 0.028343 \\ 331 0.0281047 \\ 332 0.0280365 \\ 333 0.0279122 \\ 334 0.0278044 \\ 335 0.0277309 \\
336 0.0276693 \\ 337 0.027574 \\ 338 0.0275002 \\ 339 0.0274398 \\ 340 0.0273873 \\ 341 0.0273241 \\
342 0.0272413 \\ 343 0.0271797 \\ 344 0.0271292 \\ 345 0.026667 \\ 346 0.0265283 \\ 347 0.0263118 \\
348 0.0262488 \\ 349 0.026136 \\ 350 0.0260357 \\ 351 0.0259678 \\ 352 0.0259115 \\ 353 0.0258234 \\
354 0.0257554 \\ 355 0.0257097 \\ 356 0.025631 \\ 357 0.0255649 \\ 358 0.0255116 \\ 359 0.0254656 \\
360 0.025413 \\ 361 0.0253418 \\ 362 0.0252878 \\ 363 0.0252436 \\ 364 0.0248023 \\ 365 0.0246768 \\
366 0.0244786 \\ 367 0.0244232 \\ 368 0.0243238 \\ 369 0.0242336 \\ 370 0.024173 \\ 371 0.0241236 \\
372 0.0240457 \\ 373 0.0239855 \\ 374 0.0239452 \\ 375 0.0238757 \\ 376 0.0238172 \\ 377 0.0237702 \\
378 0.0237297 \\ 379 0.0236831 \\ 380 0.0236206 \\ 381 0.023573 \\ 382 0.023534 \\ 383 0.0231362 \\
384 0.0230243 \\ 385 0.0228459 \\ 386 0.0227964 \\ 387 0.0227092 \\ 388 0.0226287 \\ 389 0.0225751 \\
390 0.0225314 \\ 391 0.0224625 \\ 392 0.0224095 \\ 393 0.0223736 \\ 394 0.0223124 \\ 395 0.0222608 \\
396 0.0222193 \\ 397 0.0221837 \\ 398 0.0221423 \\ 399 0.0220874 \\ 400 0.0220454 \\ 401 0.022011 \\
402 0.0216523 \\ 403 0.0215525 \\ 404 0.0213914 \\ 405 0.0213463 \\ 406 0.0212694 \\ 407 0.0211971 \\
408 0.0211495 \\ 409 0.0211104 \\ 410 0.0210492 \\ 411 0.0210019 \\ 412 0.0209699 \\ 413 0.0209154 \\
414 0.0208695 \\ 415 0.0208325 \\ 416 0.0208006 \\ 417 0.0207637 \\ 418 0.020715 \\ 419 0.0206773 \\
420 0.0206465 \\ 421 0.0203185 \\ 422 0.0202281 \\ 423 0.0200812 \\ 424 0.0200394 \\ 425 0.0199704 \\
426 0.0199048 \\ 427 0.0198617 \\ 428 0.0198264 \\ 429 0.019771 \\ 430 0.0197283 \\ 431 0.0196993 \\
432 0.0196501 \\ 433 0.0196085 \\ 434 0.0195749 \\ 435 0.0195459 \\ 436 0.0195127 \\ 437 0.0194687 \\
438 0.0194344 \\ 439 0.0194063 \\ 440 0.0191033 \\ 441 0.0190194 \\ 442 0.018883 \\ 443 0.0188437 \\
444 0.0187805 \\ 445 0.01872 \\ 446 0.0186804 \\ 447 0.018648 \\ 448 0.0185971 \\ 449 0.0185579 \\
450 0.0185315 \\ 451 0.0184863 \\ 452 0.0184482 \\ 453 0.0184173 \\ 454 0.0183907 \\ 455 0.0183605 \\
456 0.0183202 \\ 457 0.0182887 \\ 458 0.0182629 \\ 459 0.017984 \\ 460 0.0179067 \\ 461 0.017782 \\
462 0.0177454 \\ 463 0.0176875 \\ 464 0.017632 \\ 465 0.0176062 \\ 466 0.017562 \\ 467 0.0175159 \\
468 0.0174842 \\ 469 0.0174587 \\
};
\addlegendentry{MLRN}
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 2.94392 \\ 1 2.46526 \\ 2 2.1985 \\ 3 1.97428 \\ 4 1.73697 \\ 5 1.53124 \\
6 1.38611 \\ 7 1.2029 \\ 8 1.06726 \\ 9 0.994471 \\ 10 0.874889 \\ 11 0.834332 \\
12 0.733917 \\ 13 0.709088 \\ 14 0.659855 \\ 15 0.603971 \\ 16 0.57969 \\ 17 0.538684 \\
18 0.509148 \\ 19 0.44321 \\ 20 0.4296 \\ 21 0.399961 \\ 22 0.340424 \\ 23 0.335883 \\
24 0.3112 \\ 25 0.290646 \\ 26 0.250232 \\ 27 0.247406 \\ 28 0.238011 \\ 29 0.209505 \\
30 0.205976 \\ 31 0.185121 \\ 32 0.182899 \\ 33 0.166412 \\ 34 0.147913 \\ 35 0.145957 \\
36 0.134107 \\ 37 0.115648 \\ 38 0.115086 \\ 39 0.11155 \\ 40 0.1009 \\ 41 0.0985478 \\
42 0.0844256 \\ 43 0.083977 \\ 44 0.0834993 \\ 45 0.0819649 \\ 46 0.0762798 \\ 47 0.0644385 \\
48 0.0641516 \\ 49 0.0629924 \\ 50 0.0593333 \\ 51 0.0508897 \\ 52 0.0507937 \\ 53 0.0501947 \\
54 0.0469941 \\ 55 0.037968 \\ 56 0.0378513 \\ 57 0.0377134 \\ 58 0.0373469 \\ 59 0.035375 \\
60 0.0286797 \\ 61 0.0286326 \\ 62 0.0283249 \\ 63 0.0267126 \\ 64 0.0210442 \\ 65 0.0209662 \\
66 0.0208736 \\ 67 0.0206686 \\ 68 0.0197053 \\ 69 0.014662 \\ 70 0.0146256 \\ 71 0.0144432 \\
72 0.0137748 \\ 73 0.0107971 \\ 74 0.010786 \\ 75 0.010691 \\ 76 0.0101635 \\ 77 0.0079016 \\
78 0.00788349 \\ 79 0.00785899 \\ 80 0.00778959 \\ 81 0.00740045 \\ 82 0.00538395 \\ 83 0.00537666 \\
84 0.00536279 \\ 85 0.00532486 \\ 86 0.00510866 \\ 87 0.00384158 \\ 88 0.00383467 \\ 89 0.00380387 \\
90 0.00367851 \\ 91 0.00268108 \\ 92 0.00267917 \\ 93 0.0026759 \\ 94 0.00266464 \\ 95 0.00259045 \\
96 0.00148949 \\ 97 0.00148712 \\ 98 0.00148401 \\ 99 0.00147489 \\ 100 0.00141753 \\ 101 0.000852665 \\
102 0.000851499 \\ 103 0.000850173 \\ 104 0.000845003 \\ 105 0.000814938 \\ 106 0.000478148 \\ 107 0.000477811 \\
108 0.000477075 \\ 109 0.000474219 \\ 110 0.000458419 \\ 111 0.000211634 \\ 112 0.000211361 \\ 113 0.000210817 \\
114 0.000209296 \\ 115 0.000202466 \\ 116 9.82836e-05 \\ 117 9.82288e-05 \\ 118 9.81058e-05 \\ 119 9.77538e-05 \\
120 9.5332e-05 \\ 121 2.89025e-05 \\ 122 2.88794e-05 \\ 123 2.88289e-05 \\ 124 2.87075e-05 \\ 125 2.80048e-05 \\
126 4.87929e-06 \\ 127 4.87506e-06 \\ 128 4.86781e-06 \\ 129 4.84936e-06 \\ 130 4.73983e-06 \\ 131 4.4787e-07 \\
132 4.47372e-07 \\ 133 4.4653e-07 \\ 134 4.44172e-07 \\ 135 4.30307e-07 \\ 136 1.61916e-09 \\
};
\addlegendentry{MLRN nested}
\addplot[rn] table[row sep=\\] {
x y \\
0 2.94392 \\ 1 2.44985 \\ 2 2.05165 \\ 3 1.77977 \\ 4 1.56477 \\ 5 1.33745 \\
6 1.16218 \\ 7 0.997177 \\ 8 0.857844 \\ 9 0.753039 \\ 10 0.647931 \\ 11 0.576747 \\
12 0.495326 \\ 13 0.427393 \\ 14 0.369959 \\ 15 0.323661 \\ 16 0.27943 \\ 17 0.240469 \\
18 0.207472 \\ 19 0.17288 \\ 20 0.146722 \\ 21 0.123346 \\ 22 0.108173 \\ 23 0.096226 \\
24 0.0878709 \\ 25 0.0783986 \\ 26 0.0664209 \\ 27 0.0555386 \\ 28 0.0500955 \\ 29 0.0438219 \\
30 0.0365067 \\ 31 0.0318277 \\ 32 0.0261896 \\ 33 0.0212425 \\ 34 0.0165991 \\ 35 0.0138685 \\
36 0.0109732 \\ 37 0.00829399 \\ 38 0.00622675 \\ 39 0.00466276 \\ 40 0.0038435 \\ 41 0.00303281 \\
42 0.00209377 \\ 43 0.00146101 \\ 44 0.000982211 \\ 45 0.000569945 \\ 46 0.000341941 \\ 47 0.00017826 \\
48 8.27995e-05 \\ 49 3.43078e-05 \\ 50 6.88103e-06 \\ 51 7.97022e-07 \\ 52 2.26362e-08 \\ 53 1.03754e-10 \\
};
\addlegendentry{RN}
\addplot[tlrn] table[row sep=\\] {
x y \\
0 2.94392 \\ 1 2.47303 \\ 2 2.14783 \\ 3 1.88345 \\ 4 1.63515 \\ 5 1.42657 \\
6 1.25229 \\ 7 1.122 \\ 8 0.999022 \\ 9 0.898615 \\ 10 0.824901 \\ 11 0.763266 \\
12 0.718185 \\ 13 0.625943 \\ 14 0.60037 \\ 15 0.529623 \\ 16 0.511027 \\ 17 0.450868 \\
18 0.435377 \\ 19 0.373664 \\ 20 0.361335 \\ 21 0.303631 \\ 22 0.293115 \\ 23 0.252309 \\
24 0.245907 \\ 25 0.213493 \\ 26 0.209782 \\ 27 0.177384 \\ 28 0.17418 \\ 29 0.148079 \\
30 0.145967 \\ 31 0.126371 \\ 32 0.125065 \\ 33 0.105038 \\ 34 0.103879 \\ 35 0.0885395 \\
36 0.087781 \\ 37 0.0757871 \\ 38 0.0752292 \\ 39 0.0658458 \\ 40 0.0653782 \\ 41 0.0549365 \\
42 0.0546214 \\ 43 0.0486912 \\ 44 0.0484931 \\ 45 0.0416366 \\ 46 0.041484 \\ 47 0.0359402 \\
48 0.0357737 \\ 49 0.0295853 \\ 50 0.0294481 \\ 51 0.0227541 \\ 52 0.0226243 \\ 53 0.019725 \\
54 0.0196709 \\ 55 0.0150221 \\ 56 0.0149674 \\ 57 0.012443 \\ 58 0.0124046 \\ 59 0.0103272 \\
60 0.0103015 \\ 61 0.00811095 \\ 62 0.0080831 \\ 63 0.00575928 \\ 64 0.00573436 \\ 65 0.00418805 \\
66 0.00416435 \\ 67 0.00298684 \\ 68 0.00298008 \\ 69 0.00181965 \\ 70 0.00181395 \\ 71 0.00121712 \\
72 0.00121431 \\ 73 0.000781969 \\ 74 0.000780655 \\ 75 0.000378816 \\ 76 0.000377358 \\ 77 0.000184322 \\
78 0.000183912 \\ 79 8.93031e-05 \\ 80 8.91374e-05 \\ 81 2.87246e-05 \\ 82 2.86859e-05 \\ 83 4.68364e-06 \\
84 4.6685e-06 \\ 85 3.26239e-07 \\ 86 3.25308e-07 \\ 87 2.69317e-09 \\
};
\addlegendentry{TLRN}
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(a) $\gamma=0.05$, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=4.11,
ymin=3.114e-11, ymax=9.808e+00,
ytickten={-10,-8,-6,-4,-2,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 2.94392 \\ 0.00788021 2.47479 \\ 0.0137869 2.19928 \\ 0.0195872 1.98376 \\ 0.0258607 1.79968 \\ 0.0316682 1.62039 \\
0.0378602 1.44752 \\ 0.047111 1.26373 \\ 0.054443 1.15339 \\ 0.0633059 1.00203 \\ 0.0697139 0.912407 \\ 0.080536 0.830229 \\
0.0877426 0.780814 \\ 0.097041 0.694941 \\ 0.104484 0.648902 \\ 0.117546 0.602882 \\ 0.12484 0.577572 \\ 0.134012 0.519541 \\
0.141512 0.495368 \\ 0.154973 0.460214 \\ 0.16221 0.444787 \\ 0.172183 0.416234 \\ 0.17855 0.403257 \\ 0.189805 0.385474 \\
0.193602 0.379241 \\ 0.199791 0.370261 \\ 0.210114 0.353854 \\ 0.215652 0.346997 \\ 0.224116 0.33437 \\ 0.230774 0.328108 \\
0.239647 0.316451 \\ 0.243273 0.312097 \\ 0.252134 0.305135 \\ 0.264321 0.297026 \\ 0.268882 0.29301 \\ 0.27636 0.287836 \\
0.289167 0.280994 \\ 0.293544 0.277909 \\ 0.300047 0.273738 \\ 0.312399 0.268201 \\ 0.319564 0.264626 \\ 0.328591 0.260734 \\
0.332829 0.258182 \\ 0.341563 0.254717 \\ 0.35778 0.250655 \\ 0.365549 0.247916 \\ 0.374643 0.245113 \\ 0.381627 0.242819 \\
0.39091 0.240288 \\ 0.394246 0.237976 \\ 0.401038 0.235205 \\ 0.414042 0.232127 \\ 0.420582 0.230043 \\ 0.428931 0.228007 \\
0.451561 0.207965 \\ 0.457535 0.205501 \\ 0.467088 0.201717 \\ 0.474002 0.200045 \\ 0.478011 0.198764 \\ 0.486246 0.196233 \\
0.493935 0.194468 \\ 0.504035 0.192817 \\ 0.511305 0.191417 \\ 0.521469 0.190035 \\ 0.545069 0.174654 \\ 0.552459 0.172679 \\
0.560833 0.169392 \\ 0.564031 0.168424 \\ 0.571924 0.166709 \\ 0.580486 0.164867 \\ 0.587757 0.163728 \\ 0.591087 0.162761 \\
0.601334 0.161016 \\ 0.608168 0.159865 \\ 0.616845 0.15875 \\ 0.623522 0.157827 \\ 0.627636 0.156896 \\ 0.639327 0.155412 \\
0.645948 0.154409 \\ 0.654548 0.153468 \\ 0.67461 0.141548 \\ 0.68135 0.140002 \\ 0.690456 0.137064 \\ 0.694459 0.136318 \\
0.700913 0.134977 \\ 0.710009 0.13359 \\ 0.717574 0.132705 \\ 0.721461 0.132028 \\ 0.733385 0.130804 \\ 0.74059 0.129946 \\
0.750973 0.12916 \\ 0.758009 0.128506 \\ 0.762443 0.127859 \\ 0.773647 0.126852 \\ 0.782229 0.126144 \\ 0.792609 0.125497 \\
0.815125 0.117108 \\ 0.822275 0.116067 \\ 0.82987 0.113885 \\ 0.836022 0.113086 \\ 0.839606 0.112536 \\ 0.847454 0.111542 \\
0.85349 0.110854 \\ 0.861427 0.110237 \\ 0.867924 0.109709 \\ 0.871215 0.109241 \\ 0.881863 0.108465 \\ 0.888284 0.10789 \\
0.897796 0.107378 \\ 0.904273 0.106932 \\ 0.9154 0.10651 \\ 0.919946 0.106034 \\ 0.927629 0.105452 \\ 0.938502 0.104894 \\
0.967948 0.0982914 \\ 0.974349 0.0973747 \\ 0.983312 0.095532 \\ 0.990883 0.0949204 \\ 0.994788 0.0944805 \\ 1.00333 0.0937065 \\
1.01022 0.0931906 \\ 1.0186 0.0927493 \\ 1.02598 0.0923735 \\ 1.03016 0.0919879 \\ 1.04141 0.0914189 \\ 1.04803 0.0910079 \\
1.05845 0.0906601 \\ 1.07939 0.0858601 \\ 1.08644 0.0851369 \\ 1.09544 0.0835472 \\ 1.09869 0.083206 \\ 1.10684 0.082597 \\
1.11577 0.0820216 \\ 1.12539 0.0816437 \\ 1.13211 0.081373 \\ 1.14954 0.0808834 \\ 1.1566 0.0805168 \\ 1.16893 0.0802061 \\
1.1766 0.0799318 \\ 1.17975 0.0796579 \\ 1.19275 0.0792521 \\ 1.19985 0.0789488 \\ 1.20985 0.0786921 \\ 1.22885 0.0750417 \\
1.23589 0.0744425 \\ 1.24382 0.0730551 \\ 1.2473 0.072768 \\ 1.25399 0.0722212 \\ 1.26216 0.0717229 \\ 1.268 0.0713873 \\
1.27164 0.0711528 \\ 1.28199 0.0707287 \\ 1.28886 0.0704119 \\ 1.29998 0.0701432 \\ 1.30682 0.0699064 \\ 1.31061 0.0696657 \\
1.32174 0.0693151 \\ 1.32863 0.0690542 \\ 1.33947 0.0688336 \\ 1.36275 0.0659578 \\ 1.37087 0.0654489 \\ 1.37955 0.0643576 \\
1.38421 0.0641027 \\ 1.39152 0.0636522 \\ 1.39985 0.0632351 \\ 1.40612 0.0629552 \\ 1.40997 0.0627503 \\ 1.42062 0.0623944 \\
1.42635 0.0621294 \\ 1.43607 0.0619074 \\ 1.44232 0.0617107 \\ 1.44578 0.0614992 \\ 1.45706 0.0612061 \\ 1.46356 0.0609884 \\
1.47377 0.0608058 \\ 1.48023 0.0606402 \\ 1.48418 0.0604377 \\ 1.495 0.0601745 \\ 1.50133 0.0599808 \\ 1.51118 0.0598182 \\
1.53456 0.057461 \\ 1.54234 0.0570283 \\ 1.55103 0.0561403 \\ 1.55557 0.0559303 \\ 1.56224 0.0555632 \\ 1.5714 0.0552332 \\
1.57583 0.0550864 \\ 1.58344 0.0548166 \\ 1.59366 0.0545559 \\ 1.6006 0.0543675 \\ 1.60998 0.0542064 \\ 1.61345 0.0540492 \\
1.6239 0.053836 \\ 1.63317 0.0536315 \\ 1.63946 0.0534839 \\ 1.64887 0.0533537 \\ 1.65327 0.0531988 \\ 1.66044 0.0530113 \\
1.66956 0.0528289 \\ 1.69223 0.0512099 \\ 1.69926 0.0508777 \\ 1.70869 0.0502265 \\ 1.7129 0.0500613 \\ 1.71988 0.049776 \\
1.72889 0.049518 \\ 1.73686 0.0493422 \\ 1.74194 0.0492081 \\ 1.75528 0.048983 \\ 1.76646 0.0488127 \\ 1.77919 0.0486703 \\
1.78648 0.0485435 \\ 1.79066 0.0484059 \\ 1.80346 0.0482159 \\ 1.81051 0.0480725 \\ 1.82091 0.0479513 \\ 1.8275 0.0478414 \\
1.83113 0.0477102 \\ 1.84385 0.0475368 \\ 1.85075 0.0474072 \\ 1.86073 0.0472973 \\ 1.88477 0.0460273 \\ 1.89146 0.0457478 \\
1.89934 0.0452269 \\ 1.90431 0.045087 \\ 1.91149 0.044847 \\ 1.9204 0.0446333 \\ 1.92787 0.0444855 \\ 1.93234 0.0443695 \\
1.94383 0.0441794 \\ 1.95071 0.044033 \\ 1.96077 0.0439106 \\ 1.96748 0.0438016 \\ 1.97177 0.0436826 \\ 1.98487 0.0435194 \\
1.99168 0.0433951 \\ 2.00229 0.0432901 \\ 2.01025 0.0431953 \\ 2.01367 0.0430814 \\ 2.02639 0.0429321 \\ 2.03245 0.0428195 \\
2.0441 0.0427245 \\ 2.06784 0.0416482 \\ 2.07561 0.0413876 \\ 2.08478 0.0409363 \\ 2.08844 0.0408141 \\ 2.09475 0.0406019 \\
2.10336 0.0404157 \\ 2.11006 0.0402869 \\ 2.11433 0.0401844 \\ 2.12654 0.04002 \\ 2.13305 0.0398927 \\ 2.14332 0.0397869 \\
2.15076 0.039693 \\ 2.15493 0.0395889 \\ 2.16715 0.0394483 \\ 2.1752 0.039341 \\ 2.18653 0.0392511 \\ 2.19458 0.0391703 \\
2.19783 0.0390712 \\ 2.21029 0.0389432 \\ 2.21687 0.0388471 \\ 2.22685 0.0387667 \\ 2.25174 0.0379002 \\ 2.25834 0.0376777 \\
2.26659 0.0373165 \\ 2.27035 0.0372169 \\ 2.27746 0.0370386 \\ 2.28626 0.0368846 \\ 2.2935 0.036778 \\ 2.29714 0.0366907 \\
2.30858 0.036554 \\ 2.31577 0.0364476 \\ 2.32549 0.0363599 \\ 2.33258 0.0362824 \\ 2.33794 0.0361934 \\ 2.35136 0.0360755 \\
2.35932 0.0359862 \\ 2.36962 0.035912 \\ 2.39361 0.0352463 \\ 2.40129 0.0350498 \\ 2.41018 0.0347437 \\ 2.41409 0.0346556 \\
2.42209 0.0344951 \\ 2.4301 0.0343575 \\ 2.43652 0.0342622 \\ 2.44089 0.0341829 \\ 2.45226 0.0340596 \\ 2.458 0.0339638 \\
2.4685 0.0338849 \\ 2.47445 0.0338155 \\ 2.47846 0.0337345 \\ 2.49035 0.0336275 \\ 2.49629 0.0335469 \\ 2.50525 0.0334802 \\
2.52702 0.0328925 \\ 2.53474 0.0327146 \\ 2.54414 0.03244 \\ 2.54817 0.0323594 \\ 2.55603 0.0322117 \\ 2.56441 0.0320856 \\
2.57277 0.0319985 \\ 2.57652 0.0319248 \\ 2.58946 0.0318113 \\ 2.59659 0.0317234 \\ 2.60088 0.0316636 \\ 2.61202 0.0315619 \\
2.61998 0.0314758 \\ 2.62918 0.0314059 \\ 2.63567 0.0313448 \\ 2.63964 0.0312766 \\ 2.65038 0.0311838 \\ 2.65694 0.0311125 \\
2.6671 0.0310536 \\ 2.68785 0.0304766 \\ 2.69447 0.0303118 \\ 2.70189 0.030054 \\ 2.70488 0.0299818 \\ 2.71117 0.0298472 \\
2.71984 0.0297323 \\ 2.72579 0.0296534 \\ 2.72944 0.0295869 \\ 2.7409 0.0294848 \\ 2.74793 0.0294054 \\ 2.75749 0.0293404 \\
2.76386 0.0292836 \\ 2.76751 0.0292154 \\ 2.7795 0.0291265 \\ 2.78701 0.02906 \\ 2.79777 0.0290053 \\ 2.81968 0.0284955 \\
2.82657 0.028343 \\ 2.83543 0.0281047 \\ 2.8392 0.0280365 \\ 2.84518 0.0279122 \\ 2.8529 0.0278044 \\ 2.85921 0.0277309 \\
2.86284 0.0276693 \\ 2.87397 0.027574 \\ 2.87994 0.0275002 \\ 2.88964 0.0274398 \\ 2.8959 0.0273873 \\ 2.89981 0.0273241 \\
2.91057 0.0272413 \\ 2.91723 0.0271797 \\ 2.92606 0.0271292 \\ 2.94758 0.026667 \\ 2.95594 0.0265283 \\ 2.96412 0.0263118 \\
2.9674 0.0262488 \\ 2.9748 0.026136 \\ 2.98357 0.0260357 \\ 2.99035 0.0259678 \\ 2.99397 0.0259115 \\ 3.00641 0.0258234 \\
3.01367 0.0257554 \\ 3.01796 0.0257097 \\ 3.02944 0.025631 \\ 3.03702 0.0255649 \\ 3.04713 0.0255116 \\ 3.05378 0.0254656 \\
3.05789 0.025413 \\ 3.06854 0.0253418 \\ 3.07531 0.0252878 \\ 3.08444 0.0252436 \\ 3.10475 0.0248023 \\ 3.11094 0.0246768 \\
3.11889 0.0244786 \\ 3.12267 0.0244232 \\ 3.12817 0.0243238 \\ 3.13605 0.0242336 \\ 3.14224 0.024173 \\ 3.14506 0.0241236 \\
3.15694 0.0240457 \\ 3.16423 0.0239855 \\ 3.16806 0.0239452 \\ 3.17863 0.0238757 \\ 3.1867 0.0238172 \\ 3.19643 0.0237702 \\
3.20344 0.0237297 \\ 3.2072 0.0236831 \\ 3.21841 0.0236206 \\ 3.22516 0.023573 \\ 3.23598 0.023534 \\ 3.25856 0.0231362 \\
3.26539 0.0230243 \\ 3.27405 0.0228459 \\ 3.27768 0.0227964 \\ 3.28346 0.0227092 \\ 3.29151 0.0226287 \\ 3.29811 0.0225751 \\
3.30163 0.0225314 \\ 3.3114 0.0224625 \\ 3.31776 0.0224095 \\ 3.32064 0.0223736 \\ 3.33113 0.0223124 \\ 3.33691 0.0222608 \\
3.3479 0.0222193 \\ 3.35408 0.0221837 \\ 3.35724 0.0221423 \\ 3.36891 0.0220874 \\ 3.37579 0.0220454 \\ 3.3855 0.022011 \\
3.41112 0.0216523 \\ 3.41942 0.0215525 \\ 3.42782 0.0213914 \\ 3.43302 0.0213463 \\ 3.43969 0.0212694 \\ 3.44762 0.0211971 \\
3.45488 0.0211495 \\ 3.45884 0.0211104 \\ 3.46946 0.0210492 \\ 3.47623 0.0210019 \\ 3.47961 0.0209699 \\ 3.49104 0.0209154 \\
3.49763 0.0208695 \\ 3.50813 0.0208325 \\ 3.5145 0.0208006 \\ 3.51823 0.0207637 \\ 3.53014 0.020715 \\ 3.53676 0.0206773 \\
3.54636 0.0206465 \\ 3.57044 0.0203185 \\ 3.57823 0.0202281 \\ 3.587 0.0200812 \\ 3.59205 0.0200394 \\ 3.59964 0.0199704 \\
3.6101 0.0199048 \\ 3.61766 0.0198617 \\ 3.62179 0.0198264 \\ 3.63376 0.019771 \\ 3.64168 0.0197283 \\ 3.64564 0.0196993 \\
3.65926 0.0196501 \\ 3.66671 0.0196085 \\ 3.67901 0.0195749 \\ 3.6858 0.0195459 \\ 3.68971 0.0195127 \\ 3.70223 0.0194687 \\
3.70947 0.0194344 \\ 3.72027 0.0194063 \\ 3.74533 0.0191033 \\ 3.75269 0.0190194 \\ 3.76271 0.018883 \\ 3.76737 0.0188437 \\
3.77547 0.0187805 \\ 3.78534 0.01872 \\ 3.7971 0.0186804 \\ 3.80447 0.018648 \\ 3.81739 0.0185971 \\ 3.82475 0.0185579 \\
3.82857 0.0185315 \\ 3.84159 0.0184863 \\ 3.84956 0.0184482 \\ 3.85995 0.0184173 \\ 3.86728 0.0183907 \\ 3.87169 0.0183605 \\
3.88364 0.0183202 \\ 3.89019 0.0182887 \\ 3.90093 0.0182629 \\ 3.92358 0.017984 \\ 3.93042 0.0179067 \\ 3.93903 0.017782 \\
3.94229 0.0177454 \\ 3.94902 0.0176875 \\ 3.95787 0.017632 \\ 3.96173 0.0176062 \\ 3.96901 0.017562 \\ 3.97943 0.0175159 \\
3.9865 0.0174842 \\ 3.99037 0.0174587 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 2.94392 \\ 0.00835242 2.46526 \\ 0.0155068 2.1985 \\ 0.0226113 1.97428 \\ 0.0286566 1.73697 \\ 0.0365999 1.53124 \\
0.0439183 1.38611 \\ 0.0526926 1.2029 \\ 0.061252 1.06726 \\ 0.0691174 0.994471 \\ 0.0796323 0.874889 \\ 0.0870235 0.834332 \\
0.102337 0.733917 \\ 0.109538 0.709088 \\ 0.120172 0.659855 \\ 0.135674 0.603971 \\ 0.14452 0.57969 \\ 0.159772 0.538684 \\
0.173156 0.509148 \\ 0.190281 0.44321 \\ 0.200575 0.4296 \\ 0.214944 0.399961 \\ 0.232096 0.340424 \\ 0.24042 0.335883 \\
0.251755 0.3112 \\ 0.263552 0.290646 \\ 0.284863 0.250232 \\ 0.292623 0.247406 \\ 0.308109 0.238011 \\ 0.322387 0.209505 \\
0.330582 0.205976 \\ 0.346754 0.185121 \\ 0.355899 0.182899 \\ 0.371765 0.166412 \\ 0.383629 0.147913 \\ 0.392091 0.145957 \\
0.409003 0.134107 \\ 0.427523 0.115648 \\ 0.43582 0.115086 \\ 0.447592 0.11155 \\ 0.46213 0.1009 \\ 0.475434 0.0985478 \\
0.501234 0.0844256 \\ 0.508565 0.083977 \\ 0.517205 0.0834993 \\ 0.52742 0.0819649 \\ 0.544768 0.0762798 \\ 0.561921 0.0644385 \\
0.568415 0.0641516 \\ 0.578573 0.0629924 \\ 0.59478 0.0593333 \\ 0.612104 0.0508897 \\ 0.619497 0.0507937 \\ 0.63094 0.0501947 \\
0.647195 0.0469941 \\ 0.666947 0.037968 \\ 0.675081 0.0378513 \\ 0.685075 0.0377134 \\ 0.696043 0.0373469 \\ 0.717135 0.035375 \\
0.736171 0.0286797 \\ 0.742599 0.0286326 \\ 0.753092 0.0283249 \\ 0.765097 0.0267126 \\ 0.785178 0.0210442 \\ 0.791129 0.0209662 \\
0.79968 0.0208736 \\ 0.80989 0.0206686 \\ 0.824483 0.0197053 \\ 0.846555 0.014662 \\ 0.855213 0.0146256 \\ 0.86548 0.0144432 \\
0.884643 0.0137748 \\ 0.907273 0.0107971 \\ 0.91481 0.010786 \\ 0.925878 0.010691 \\ 0.939661 0.0101635 \\ 0.962535 0.0079016 \\
0.968387 0.00788349 \\ 0.976982 0.00785899 \\ 0.987803 0.00778959 \\ 1.00009 0.00740045 \\ 1.02285 0.00538395 \\ 1.02958 0.00537666 \\
1.03852 0.00536279 \\ 1.04946 0.00532486 \\ 1.06332 0.00510866 \\ 1.0919 0.00384158 \\ 1.09906 0.00383467 \\ 1.11145 0.00380387 \\
1.12522 0.00367851 \\ 1.14825 0.00268108 \\ 1.15585 0.00267917 \\ 1.16486 0.0026759 \\ 1.1759 0.00266464 \\ 1.19016 0.00259045 \\
1.21246 0.00148949 \\ 1.21982 0.00148712 \\ 1.2293 0.00148401 \\ 1.24089 0.00147489 \\ 1.25972 0.00141753 \\ 1.2866 0.000852665 \\
1.29431 0.000851499 \\ 1.30387 0.000850173 \\ 1.31612 0.000845003 \\ 1.32977 0.000814938 \\ 1.3581 0.000478148 \\ 1.36589 0.000477811 \\
1.3757 0.000477075 \\ 1.38679 0.000474219 \\ 1.40025 0.000458419 \\ 1.42412 0.000211634 \\ 1.43162 0.000211361 \\ 1.44058 0.000210817 \\
1.4524 0.000209296 \\ 1.46702 0.000202466 \\ 1.49576 9.82836e-05 \\ 1.50476 9.82288e-05 \\ 1.51424 9.81058e-05 \\ 1.52602 9.77538e-05 \\
1.54059 9.5332e-05 \\ 1.56267 2.89025e-05 \\ 1.57105 2.88794e-05 \\ 1.5801 2.88289e-05 \\ 1.58964 2.87075e-05 \\ 1.60158 2.80048e-05 \\
1.62419 4.87929e-06 \\ 1.63108 4.87506e-06 \\ 1.63912 4.86781e-06 \\ 1.64864 4.84936e-06 \\ 1.66141 4.73983e-06 \\ 1.68782 4.4787e-07 \\
1.69581 4.47372e-07 \\ 1.7046 4.4653e-07 \\ 1.71586 4.44172e-07 \\ 1.72917 4.30307e-07 \\ 1.74607 1.61916e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 2.94392 \\ 0.0128552 2.44985 \\ 0.0253005 2.05165 \\ 0.0418782 1.77977 \\ 0.0543513 1.56477 \\ 0.0684135 1.33745 \\
0.0804738 1.16218 \\ 0.0927511 0.997177 \\ 0.104744 0.857844 \\ 0.118716 0.753039 \\ 0.130911 0.647931 \\ 0.145077 0.576747 \\
0.157128 0.495326 \\ 0.173511 0.427393 \\ 0.189717 0.369959 \\ 0.205691 0.323661 \\ 0.218561 0.27943 \\ 0.23477 0.240469 \\
0.249192 0.207472 \\ 0.261102 0.17288 \\ 0.273458 0.146722 \\ 0.287744 0.123346 \\ 0.304302 0.108173 \\ 0.31863 0.096226 \\
0.335393 0.0878709 \\ 0.348912 0.0783986 \\ 0.360738 0.0664209 \\ 0.372768 0.0555386 \\ 0.396817 0.0500955 \\ 0.413126 0.0438219 \\
0.42801 0.0365067 \\ 0.445361 0.0318277 \\ 0.458083 0.0261896 \\ 0.474477 0.0212425 \\ 0.487615 0.0165991 \\ 0.505471 0.0138685 \\
0.517963 0.0109732 \\ 0.532961 0.00829399 \\ 0.548299 0.00622675 \\ 0.563337 0.00466276 \\ 0.579612 0.0038435 \\ 0.591986 0.00303281 \\
0.605062 0.00209377 \\ 0.621808 0.00146101 \\ 0.637876 0.000982211 \\ 0.651468 0.000569945 \\ 0.670331 0.000341941 \\ 0.684414 0.00017826 \\
0.697888 8.27995e-05 \\ 0.710936 3.43078e-05 \\ 0.723926 6.88103e-06 \\ 0.736536 7.97022e-07 \\ 0.748407 2.26362e-08 \\ 0.760551 1.03754e-10 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 2.94392 \\ 0.010446 2.47303 \\ 0.0193987 2.14783 \\ 0.02844 1.88345 \\ 0.0362753 1.63515 \\ 0.0440239 1.42657 \\
0.0525061 1.25229 \\ 0.0612749 1.122 \\ 0.0701493 0.999022 \\ 0.0794605 0.898615 \\ 0.0892558 0.824901 \\ 0.0986297 0.763266 \\
0.10856 0.718185 \\ 0.129628 0.625943 \\ 0.139307 0.60037 \\ 0.159598 0.529623 \\ 0.16942 0.511027 \\ 0.188993 0.450868 \\
0.198357 0.435377 \\ 0.21547 0.373664 \\ 0.225575 0.361335 \\ 0.24151 0.303631 \\ 0.250285 0.293115 \\ 0.267668 0.252309 \\
0.276769 0.245907 \\ 0.293604 0.213493 \\ 0.300804 0.209782 \\ 0.315463 0.177384 \\ 0.323952 0.17418 \\ 0.342059 0.148079 \\
0.351107 0.145967 \\ 0.36896 0.126371 \\ 0.378108 0.125065 \\ 0.393899 0.105038 \\ 0.402944 0.103879 \\ 0.421914 0.0885395 \\
0.429856 0.087781 \\ 0.447729 0.0757871 \\ 0.456431 0.0752292 \\ 0.472989 0.0658458 \\ 0.480262 0.0653782 \\ 0.494121 0.0549365 \\
0.502034 0.0546214 \\ 0.520421 0.0486912 \\ 0.528124 0.0484931 \\ 0.541628 0.0416366 \\ 0.550008 0.041484 \\ 0.56769 0.0359402 \\
0.576261 0.0357737 \\ 0.592633 0.0295853 \\ 0.601124 0.0294481 \\ 0.6173 0.0227541 \\ 0.625536 0.0226243 \\ 0.648673 0.019725 \\
0.656784 0.0196709 \\ 0.669527 0.0150221 \\ 0.677147 0.0149674 \\ 0.696139 0.012443 \\ 0.703491 0.0124046 \\ 0.720203 0.0103272 \\
0.728317 0.0103015 \\ 0.742533 0.00811095 \\ 0.749479 0.0080831 \\ 0.764621 0.00575928 \\ 0.773354 0.00573436 \\ 0.792205 0.00418805 \\
0.799831 0.00416435 \\ 0.816035 0.00298684 \\ 0.82654 0.00298008 \\ 0.842655 0.00181965 \\ 0.85131 0.00181395 \\ 0.872222 0.00121712 \\
0.880138 0.00121431 \\ 0.896855 0.000781969 \\ 0.904536 0.000780655 \\ 0.919359 0.000378816 \\ 0.926573 0.000377358 \\ 0.942735 0.000184322 \\
0.950033 0.000183912 \\ 0.966253 8.93031e-05 \\ 0.974607 8.91374e-05 \\ 0.988128 2.87246e-05 \\ 0.996834 2.86859e-05 \\ 1.01311 4.68364e-06 \\
1.0227 4.6685e-06 \\ 1.04038 3.26239e-07 \\ 1.04871 3.25308e-07 \\ 1.06218 2.69317e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(b) $\gamma=0.05$, runtime};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=502,
ymin=2.397e-13, ymax=1.094e+01,
ytickten={-11,-8,-5,-2,1},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 2.62004 \\ 1 2.17423 \\ 2 1.92613 \\ 3 1.73542 \\ 4 1.57236 \\ 5 1.34215 \\
6 1.21211 \\ 7 1.01608 \\ 8 0.914591 \\ 9 0.778692 \\ 10 0.704122 \\ 11 0.592775 \\
12 0.540933 \\ 13 0.462196 \\ 14 0.430572 \\ 15 0.37599 \\ 16 0.353238 \\ 17 0.323733 \\
18 0.314269 \\ 19 0.300633 \\ 20 0.279947 \\ 21 0.273014 \\ 22 0.262865 \\ 23 0.250858 \\
24 0.244174 \\ 25 0.237712 \\ 26 0.233227 \\ 27 0.227116 \\ 28 0.220772 \\ 29 0.216452 \\
30 0.212663 \\ 31 0.175018 \\ 32 0.169675 \\ 33 0.161696 \\ 34 0.15874 \\ 35 0.156509 \\
36 0.152448 \\ 37 0.149875 \\ 38 0.148146 \\ 39 0.144926 \\ 40 0.14271 \\ 41 0.140731 \\
42 0.139106 \\ 43 0.137528 \\ 44 0.13508 \\ 45 0.133414 \\ 46 0.131942 \\ 47 0.113769 \\
48 0.11158 \\ 49 0.106981 \\ 50 0.105362 \\ 51 0.104214 \\ 52 0.102325 \\ 53 0.10102 \\
54 0.0999046 \\ 55 0.0989823 \\ 56 0.0980815 \\ 57 0.0967624 \\ 58 0.0957967 \\ 59 0.0949761 \\
60 0.0847803 \\ 61 0.083257 \\ 62 0.0795231 \\ 63 0.0783471 \\ 64 0.0775894 \\ 65 0.0762915 \\
66 0.0754201 \\ 67 0.0749219 \\ 68 0.0739573 \\ 69 0.0731927 \\ 70 0.0725696 \\ 71 0.0720396 \\
72 0.0715326 \\ 73 0.0707925 \\ 74 0.070228 \\ 75 0.0697577 \\ 76 0.0640049 \\ 77 0.063038 \\
78 0.0607521 \\ 79 0.0602579 \\ 80 0.0593697 \\ 81 0.058571 \\ 82 0.0580479 \\ 83 0.0576784 \\
84 0.0570366 \\ 85 0.0565568 \\ 86 0.0561633 \\ 87 0.0558204 \\ 88 0.0554498 \\ 89 0.0549486 \\
90 0.0545707 \\ 91 0.0542586 \\ 92 0.0507378 \\ 93 0.0500783 \\ 94 0.0486459 \\ 95 0.0482795 \\
96 0.0476478 \\ 97 0.0470944 \\ 98 0.0468518 \\ 99 0.0463911 \\ 100 0.0459539 \\ 101 0.0456388 \\
102 0.0454002 \\ 103 0.0450034 \\ 104 0.0446892 \\ 105 0.0444298 \\ 106 0.0441995 \\ 107 0.043947 \\
108 0.0436067 \\ 109 0.0433432 \\ 110 0.0431248 \\ 111 0.0408231 \\ 112 0.0403205 \\ 113 0.0393591 \\
114 0.0391008 \\ 115 0.0386411 \\ 116 0.0382449 \\ 117 0.0380685 \\ 118 0.0377238 \\ 119 0.037403 \\
120 0.0371703 \\ 121 0.0369912 \\ 122 0.0366962 \\ 123 0.0364611 \\ 124 0.0362678 \\ 125 0.0360974 \\
126 0.0359077 \\ 127 0.035653 \\ 128 0.0354563 \\ 129 0.0352942 \\ 130 0.0337087 \\ 131 0.0333227 \\
132 0.0326317 \\ 133 0.0324504 \\ 134 0.0321047 \\ 135 0.0318105 \\ 136 0.0316075 \\ 137 0.0314467 \\
138 0.0311948 \\ 139 0.0309946 \\ 140 0.0308327 \\ 141 0.0307037 \\ 142 0.0304964 \\ 143 0.030307 \\
144 0.03017 \\ 145 0.0300423 \\ 146 0.0298453 \\ 147 0.0296947 \\ 148 0.029571 \\ 149 0.0284124 \\
150 0.0281246 \\ 151 0.0276086 \\ 152 0.0274741 \\ 153 0.0272178 \\ 154 0.0269899 \\ 155 0.0268331 \\
156 0.026715 \\ 157 0.0265194 \\ 158 0.0263651 \\ 159 0.02624 \\ 160 0.0261383 \\ 161 0.0259829 \\
162 0.0258336 \\ 163 0.0257273 \\ 164 0.0256303 \\ 165 0.025476 \\ 166 0.0253591 \\ 167 0.0252632 \\
168 0.0243647 \\ 169 0.024135 \\ 170 0.0237234 \\ 171 0.0236152 \\ 172 0.0234153 \\ 173 0.0232304 \\
174 0.023105 \\ 175 0.0230119 \\ 176 0.0228549 \\ 177 0.0227319 \\ 178 0.0226571 \\ 179 0.022518 \\
180 0.0224007 \\ 181 0.0223066 \\ 182 0.022226 \\ 183 0.0221384 \\ 184 0.0220164 \\ 185 0.0219224 \\
186 0.0218462 \\ 187 0.021113 \\ 188 0.0209232 \\ 189 0.0205868 \\ 190 0.0204958 \\ 191 0.0203356 \\
192 0.0201843 \\ 193 0.0200835 \\ 194 0.0200064 \\ 195 0.0198791 \\ 196 0.01978 \\ 197 0.0197178 \\
198 0.0196053 \\ 199 0.0195108 \\ 200 0.0194355 \\ 201 0.0193714 \\ 202 0.0192987 \\ 203 0.0192005 \\
204 0.019125 \\ 205 0.0190641 \\ 206 0.0184424 \\ 207 0.0182829 \\ 208 0.0180017 \\ 209 0.0179219 \\
210 0.0177905 \\ 211 0.0176648 \\ 212 0.0176073 \\ 213 0.0175073 \\ 214 0.0174033 \\ 215 0.0173324 \\
216 0.0172761 \\ 217 0.0171827 \\ 218 0.0171103 \\ 219 0.0170516 \\ 220 0.0169984 \\ 221 0.0169267 \\
222 0.016852 \\ 223 0.0163359 \\ 224 0.0161986 \\ 225 0.0159598 \\ 226 0.0158886 \\ 227 0.0157766 \\
228 0.015667 \\ 229 0.0156166 \\ 230 0.015531 \\ 231 0.0154405 \\ 232 0.0153792 \\ 233 0.0153313 \\
234 0.0152506 \\ 235 0.0151887 \\ 236 0.0151386 \\ 237 0.0150926 \\ 238 0.015032 \\ 239 0.014968 \\
240 0.0145269 \\ 241 0.0144096 \\ 242 0.0142078 \\ 243 0.0141439 \\ 244 0.0140492 \\ 245 0.0139552 \\
246 0.0139106 \\ 247 0.0138383 \\ 248 0.0137607 \\ 249 0.0137086 \\ 250 0.0136675 \\ 251 0.0135982 \\
252 0.0135458 \\ 253 0.0135036 \\ 254 0.0134636 \\ 255 0.0134126 \\ 256 0.0133582 \\ 257 0.0129804 \\
258 0.0128801 \\ 259 0.0127089 \\ 260 0.0126519 \\ 261 0.0125718 \\ 262 0.0124917 \\ 263 0.0124523 \\
264 0.0123914 \\ 265 0.0123252 \\ 266 0.0122921 \\ 267 0.0122394 \\ 268 0.0121809 \\ 269 0.0121414 \\
270 0.0121106 \\ 271 0.0120567 \\ 272 0.0120165 \\ 273 0.0119839 \\ 274 0.0119525 \\ 275 0.0119133 \\
276 0.0118707 \\ 277 0.011529 \\ 278 0.0114447 \\ 279 0.0113002 \\ 280 0.0112522 \\ 281 0.011187 \\
282 0.0111214 \\ 283 0.0110884 \\ 284 0.0110393 \\ 285 0.0109852 \\ 286 0.0109576 \\ 287 0.0109152 \\
288 0.0108675 \\ 289 0.0108353 \\ 290 0.0108078 \\ 291 0.010776 \\ 292 0.0107422 \\ 293 0.0107035 \\
294 0.0104267 \\ 295 0.0103573 \\ 296 0.0102348 \\ 297 0.010193 \\ 298 0.0101376 \\ 299 0.0100815 \\
300 0.0100525 \\ 301 0.0100107 \\ 302 0.00996438 \\ 303 0.00994016 \\ 304 0.00990399 \\ 305 0.00986297 \\
306 0.00983548 \\ 307 0.009812 \\ 308 0.00978422 \\ 309 0.00975545 \\ 310 0.00972218 \\ 311 0.00948969 \\
312 0.00943203 \\ 313 0.0093266 \\ 314 0.00929067 \\ 315 0.00924345 \\ 316 0.00919548 \\ 317 0.00917044 \\
318 0.00913471 \\ 319 0.00909501 \\ 320 0.00907401 \\ 321 0.00904303 \\ 322 0.00900782 \\ 323 0.00898427 \\
324 0.00896419 \\ 325 0.00894002 \\ 326 0.0089153 \\ 327 0.00888669 \\ 328 0.00868803 \\ 329 0.0086391 \\
330 0.0085467 \\ 331 0.00851581 \\ 332 0.00847496 \\ 333 0.00843356 \\ 334 0.00841198 \\ 335 0.00838109 \\
336 0.00834686 \\ 337 0.00832873 \\ 338 0.00830192 \\ 339 0.00827154 \\ 340 0.00825122 \\ 341 0.00823394 \\
342 0.00821291 \\ 343 0.0081915 \\ 344 0.00816679 \\ 345 0.00799295 \\ 346 0.0079502 \\ 347 0.00786738 \\
348 0.00784046 \\ 349 0.00780443 \\ 350 0.00776818 \\ 351 0.00774942 \\ 352 0.00772233 \\ 353 0.0076925 \\
354 0.00767673 \\ 355 0.00765326 \\ 356 0.00762681 \\ 357 0.00760911 \\ 358 0.00759412 \\ 359 0.00757568 \\
360 0.00755697 \\ 361 0.00753548 \\ 362 0.00737948 \\ 363 0.0073414 \\ 364 0.00726606 \\ 365 0.00724225 \\
366 0.00721002 \\ 367 0.00717788 \\ 368 0.00716137 \\ 369 0.00713736 \\ 370 0.00711108 \\ 371 0.00709394 \\
372 0.00707967 \\ 373 0.00706145 \\ 374 0.00704344 \\ 375 0.00702303 \\ 376 0.00689083 \\ 377 0.00685689 \\
378 0.0067882 \\ 379 0.00676592 \\ 380 0.00673619 \\ 381 0.00670667 \\ 382 0.00669131 \\ 383 0.00666918 \\
384 0.00664508 \\ 385 0.00662947 \\ 386 0.00661653 \\ 387 0.00659955 \\ 388 0.00658302 \\ 389 0.00656436 \\
390 0.00644341 \\ 391 0.00641251 \\ 392 0.00634897 \\ 393 0.00632832 \\ 394 0.00630089 \\ 395 0.00627381 \\
396 0.00625963 \\ 397 0.00623932 \\ 398 0.00621728 \\ 399 0.00620308 \\ 400 0.00619136 \\ 401 0.00617566 \\
402 0.00616055 \\ 403 0.00614354 \\ 404 0.00603177 \\ 405 0.00600351 \\ 406 0.00594458 \\ 407 0.00592549 \\
408 0.0059002 \\ 409 0.00587537 \\ 410 0.0058623 \\ 411 0.00584368 \\ 412 0.00582354 \\ 413 0.00581063 \\
414 0.00580001 \\ 415 0.00578553 \\ 416 0.00577174 \\ 417 0.00575624 \\ 418 0.00565256 \\ 419 0.00562665 \\
420 0.00557191 \\ 421 0.00555429 \\ 422 0.00553098 \\ 423 0.00550819 \\ 424 0.00549614 \\ 425 0.00547906 \\
426 0.00546064 \\ 427 0.00544889 \\ 428 0.00536766 \\ 429 0.00534689 \\ 430 0.00529488 \\ 431 0.00527756 \\
432 0.0052555 \\ 433 0.00523379 \\ 434 0.00522208 \\ 435 0.00520583 \\ 436 0.00518822 \\ 437 0.00517702 \\
438 0.00516786 \\ 439 0.00515498 \\ 440 0.00514295 \\ 441 0.00512945 \\ 442 0.00504039 \\ 443 0.00501895 \\
444 0.00497129 \\ 445 0.00495573 \\ 446 0.00493547 \\ 447 0.00491568 \\ 448 0.00490505 \\ 449 0.00489023 \\
450 0.00487424 \\ 451 0.0048641 \\ 452 0.00479398 \\ 453 0.00477644 \\ 454 0.004731 \\ 455 0.00471582 \\
456 0.00469663 \\ 457 0.00467778 \\ 458 0.0046675 \\ 459 0.0046534 \\ 460 0.00463812 \\ 461 0.00462845 \\
462 0.00462056 \\ 463 0.00460927 \\ 464 0.00459884 \\ 465 0.00458715 \\ 466 0.00450955 \\ 467 0.00449124 \\
468 0.00444941 \\ 469 0.00443579 \\ 470 0.00441812 \\ 471 0.00440089 \\ 472 0.00439157 \\ 473 0.00437867 \\
474 0.00436476 \\ 475 0.00435596 \\ 476 0.0042948 \\ 477 0.00427975 \\ 478 0.0042398 \\ 479 0.00422654 \\
480 0.00420976 \\ 481 0.00419333 \\ 482 0.00418432 \\ 483 0.00417202 \\ 484 0.0041587 \\ 485 0.00415028 \\
486 0.00414344 \\ 487 0.00413354 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 2.62004 \\ 1 2.15617 \\ 2 1.90829 \\ 3 1.70543 \\ 4 1.50875 \\ 5 1.34768 \\
6 1.22357 \\ 7 1.04786 \\ 8 0.903195 \\ 9 0.785781 \\ 10 0.645765 \\ 11 0.586898 \\
12 0.500471 \\ 13 0.46886 \\ 14 0.365819 \\ 15 0.353375 \\ 16 0.33605 \\ 17 0.296576 \\
18 0.27033 \\ 19 0.203517 \\ 20 0.19879 \\ 21 0.183014 \\ 22 0.143885 \\ 23 0.14012 \\
24 0.118807 \\ 25 0.114181 \\ 26 0.0874135 \\ 27 0.0864458 \\ 28 0.0854621 \\ 29 0.0733016 \\
30 0.0529782 \\ 31 0.052438 \\ 32 0.0507236 \\ 33 0.0447546 \\ 34 0.0296785 \\ 35 0.0294101 \\
36 0.0291486 \\ 37 0.02864 \\ 38 0.0258711 \\ 39 0.0176964 \\ 40 0.0176265 \\ 41 0.0175191 \\
42 0.0172687 \\ 43 0.0160677 \\ 44 0.00966453 \\ 45 0.00962143 \\ 46 0.00939913 \\ 47 0.00861624 \\
48 0.0051708 \\ 49 0.00514865 \\ 50 0.00511592 \\ 51 0.00504399 \\ 52 0.00466851 \\ 53 0.00251996 \\
54 0.00251086 \\ 55 0.00249927 \\ 56 0.00246938 \\ 57 0.00233811 \\ 58 0.000883204 \\ 59 0.000880258 \\
60 0.000875852 \\ 61 0.000861695 \\ 62 0.000811354 \\ 63 0.000295298 \\ 64 0.000294649 \\ 65 0.000293459 \\
66 0.000289499 \\ 67 0.000276848 \\ 68 5.82265e-05 \\ 69 5.80942e-05 \\ 70 5.782e-05 \\ 71 5.70698e-05 \\
72 5.37935e-05 \\ 73 5.25335e-06 \\ 74 5.24328e-06 \\ 75 5.22402e-06 \\ 76 5.16688e-06 \\ 77 4.83653e-06 \\
78 2.71006e-08 \\ 79 2.68984e-08 \\ 80 2.65219e-08 \\ 81 2.5549e-08 \\ 82 2.16667e-08 \\ 83 1.00131e-12 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 2.62004 \\ 1 2.04261 \\ 2 1.66719 \\ 3 1.41893 \\ 4 1.21069 \\ 5 1.03654 \\
6 0.882149 \\ 7 0.735464 \\ 8 0.602441 \\ 9 0.483466 \\ 10 0.378142 \\ 11 0.287996 \\
12 0.22183 \\ 13 0.168629 \\ 14 0.1266 \\ 15 0.100262 \\ 16 0.079239 \\ 17 0.0594962 \\
18 0.0447528 \\ 19 0.0341128 \\ 20 0.0247474 \\ 21 0.0182319 \\ 22 0.013138 \\ 23 0.00852754 \\
24 0.00497867 \\ 25 0.00301797 \\ 26 0.00164924 \\ 27 0.000772019 \\ 28 0.000326264 \\ 29 0.000122966 \\
30 2.73494e-05 \\ 31 3.30271e-06 \\ 32 1.10255e-07 \\ 33 4.80382e-10 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 2.62004 \\ 1 2.11368 \\ 2 1.82008 \\ 3 1.57698 \\ 4 1.38079 \\ 5 1.2104 \\
6 1.07496 \\ 7 0.95755 \\ 8 0.843897 \\ 9 0.752349 \\ 10 0.594379 \\ 11 0.548336 \\
12 0.426255 \\ 13 0.397026 \\ 14 0.299201 \\ 15 0.279723 \\ 16 0.205973 \\ 17 0.197038 \\
18 0.142244 \\ 19 0.136916 \\ 20 0.0967403 \\ 21 0.0937064 \\ 22 0.0751158 \\ 23 0.0741583 \\
24 0.0605005 \\ 25 0.0598373 \\ 26 0.0462955 \\ 27 0.0458972 \\ 28 0.0338068 \\ 29 0.0334532 \\
30 0.0227015 \\ 31 0.0223391 \\ 32 0.0168823 \\ 33 0.0167651 \\ 34 0.0126302 \\ 35 0.0125609 \\
36 0.0085827 \\ 37 0.00853239 \\ 38 0.00419159 \\ 39 0.00413693 \\ 40 0.00215738 \\ 41 0.00214286 \\
42 0.000932538 \\ 43 0.000924419 \\ 44 0.000382321 \\ 45 0.000380452 \\ 46 0.000117205 \\ 47 0.00011661 \\
48 1.7294e-05 \\ 49 1.72194e-05 \\ 50 5.58444e-07 \\ 51 5.54315e-07 \\ 52 1.67421e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(c) $\gamma=0.1$, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=4.12,
ymin=2.397e-13, ymax=1.094e+01,
ytickten={-11,-8,-5,-2,1},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 2.62004 \\ 0.00847604 2.17423 \\ 0.0169285 1.92613 \\ 0.0236262 1.73542 \\ 0.0297475 1.57236 \\ 0.0393675 1.34215 \\
0.0451263 1.21211 \\ 0.0555089 1.01608 \\ 0.0624937 0.914591 \\ 0.0716204 0.778692 \\ 0.0780206 0.704122 \\ 0.0874056 0.592775 \\
0.0937725 0.540933 \\ 0.103916 0.462196 \\ 0.11027 0.430572 \\ 0.119224 0.37599 \\ 0.125943 0.353238 \\ 0.13633 0.323733 \\
0.14051 0.314269 \\ 0.14721 0.300633 \\ 0.158637 0.279947 \\ 0.162335 0.273014 \\ 0.16941 0.262865 \\ 0.180967 0.250858 \\
0.187542 0.244174 \\ 0.195687 0.237712 \\ 0.200054 0.233227 \\ 0.208618 0.227116 \\ 0.222872 0.220772 \\ 0.229357 0.216452 \\
0.238132 0.212663 \\ 0.257964 0.175018 \\ 0.264846 0.169675 \\ 0.274113 0.161696 \\ 0.28142 0.15874 \\ 0.285613 0.156509 \\
0.294116 0.152448 \\ 0.301053 0.149875 \\ 0.305445 0.148146 \\ 0.316594 0.144926 \\ 0.323687 0.14271 \\ 0.331984 0.140731 \\
0.339654 0.139106 \\ 0.343161 0.137528 \\ 0.356875 0.13508 \\ 0.363679 0.133414 \\ 0.373583 0.131942 \\ 0.396527 0.113769 \\
0.404225 0.11158 \\ 0.412175 0.106981 \\ 0.419898 0.105362 \\ 0.42395 0.104214 \\ 0.432793 0.102325 \\ 0.439962 0.10102 \\
0.448494 0.0999046 \\ 0.454667 0.0989823 \\ 0.458475 0.0980815 \\ 0.469988 0.0967624 \\ 0.476221 0.0957967 \\ 0.485267 0.0949761 \\
0.506404 0.0847803 \\ 0.513121 0.083257 \\ 0.520711 0.0795231 \\ 0.527603 0.0783471 \\ 0.530888 0.0775894 \\ 0.539294 0.0762915 \\
0.546051 0.0754201 \\ 0.550194 0.0749219 \\ 0.561166 0.0739573 \\ 0.568271 0.0731927 \\ 0.579866 0.0725696 \\ 0.58726 0.0720396 \\
0.591569 0.0715326 \\ 0.604465 0.0707925 \\ 0.611529 0.070228 \\ 0.622744 0.0697577 \\ 0.643771 0.0640049 \\ 0.65095 0.063038 \\
0.659864 0.0607521 \\ 0.663114 0.0602579 \\ 0.668592 0.0593697 \\ 0.676324 0.058571 \\ 0.682217 0.0580479 \\ 0.686082 0.0576784 \\
0.696135 0.0570366 \\ 0.702676 0.0565568 \\ 0.71257 0.0561633 \\ 0.718288 0.0558204 \\ 0.722013 0.0554498 \\ 0.731945 0.0549486 \\
0.737554 0.0545707 \\ 0.747546 0.0542586 \\ 0.7766 0.0507378 \\ 0.783976 0.0500783 \\ 0.792778 0.0486459 \\ 0.797503 0.0482795 \\
0.804136 0.0476478 \\ 0.812459 0.0470944 \\ 0.816492 0.0468518 \\ 0.823898 0.0463911 \\ 0.833409 0.0459539 \\ 0.840164 0.0456388 \\
0.844155 0.0454002 \\ 0.855235 0.0450034 \\ 0.861911 0.0446892 \\ 0.871465 0.0444298 \\ 0.877506 0.0441995 \\ 0.881136 0.043947 \\
0.893923 0.0436067 \\ 0.900045 0.0433432 \\ 0.908844 0.0431248 \\ 0.929252 0.0408231 \\ 0.935205 0.0403205 \\ 0.943552 0.0393591 \\
0.947234 0.0391008 \\ 0.953385 0.0386411 \\ 0.962271 0.0382449 \\ 0.966247 0.0380685 \\ 0.97378 0.0377238 \\ 0.98288 0.037403 \\
0.990272 0.0371703 \\ 0.994354 0.0369912 \\ 1.00642 0.0366962 \\ 1.0138 0.0364611 \\ 1.02478 0.0362678 \\ 1.03174 0.0360974 \\
1.03585 0.0359077 \\ 1.04675 0.035653 \\ 1.05372 0.0354563 \\ 1.06237 0.0352942 \\ 1.08383 0.0337087 \\ 1.09052 0.0333227 \\
1.09877 0.0326317 \\ 1.10188 0.0324504 \\ 1.10841 0.0321047 \\ 1.11666 0.0318105 \\ 1.12279 0.0316075 \\ 1.12587 0.0314467 \\
1.13651 0.0311948 \\ 1.14283 0.0309946 \\ 1.15185 0.0308327 \\ 1.15564 0.0307037 \\ 1.16375 0.0304964 \\ 1.17326 0.030307 \\
1.17893 0.03017 \\ 1.18262 0.0300423 \\ 1.19425 0.0298453 \\ 1.20077 0.0296947 \\ 1.21185 0.029571 \\ 1.23453 0.0284124 \\
1.24235 0.0281246 \\ 1.25141 0.0276086 \\ 1.25537 0.0274741 \\ 1.2621 0.0272178 \\ 1.27125 0.0269899 \\ 1.27777 0.0268331 \\
1.28143 0.026715 \\ 1.29124 0.0265194 \\ 1.29789 0.0263651 \\ 1.30771 0.02624 \\ 1.31157 0.0261383 \\ 1.31874 0.0259829 \\
1.32684 0.0258336 \\ 1.33266 0.0257273 \\ 1.33678 0.0256303 \\ 1.34852 0.025476 \\ 1.35448 0.0253591 \\ 1.36367 0.0252632 \\
1.38608 0.0243647 \\ 1.39387 0.024135 \\ 1.40241 0.0237234 \\ 1.40705 0.0236152 \\ 1.41412 0.0234153 \\ 1.42304 0.0232304 \\
1.42966 0.023105 \\ 1.43381 0.0230119 \\ 1.44509 0.0228549 \\ 1.45233 0.0227319 \\ 1.45599 0.0226571 \\ 1.46738 0.022518 \\
1.47418 0.0224007 \\ 1.4839 0.0223066 \\ 1.49063 0.022226 \\ 1.49426 0.0221384 \\ 1.50498 0.0220164 \\ 1.51102 0.0219224 \\
1.52113 0.0218462 \\ 1.54168 0.021113 \\ 1.54851 0.0209232 \\ 1.55572 0.0205868 \\ 1.55954 0.0204958 \\ 1.56597 0.0203356 \\
1.57456 0.0201843 \\ 1.58138 0.0200835 \\ 1.5845 0.0200064 \\ 1.59473 0.0198791 \\ 1.60211 0.01978 \\ 1.60559 0.0197178 \\
1.6176 0.0196053 \\ 1.62479 0.0195108 \\ 1.6363 0.0194355 \\ 1.64258 0.0193714 \\ 1.64734 0.0192987 \\ 1.65994 0.0192005 \\
1.66782 0.019125 \\ 1.67721 0.0190641 \\ 1.70056 0.0184424 \\ 1.70762 0.0182829 \\ 1.71681 0.0180017 \\ 1.72084 0.0179219 \\
1.72737 0.0177905 \\ 1.73558 0.0176648 \\ 1.73948 0.0176073 \\ 1.74534 0.0175073 \\ 1.75643 0.0174033 \\ 1.7632 0.0173324 \\
1.76818 0.0172761 \\ 1.78067 0.0171827 \\ 1.78811 0.0171103 \\ 1.79851 0.0170516 \\ 1.80274 0.0169984 \\ 1.81117 0.0169267 \\
1.82146 0.016852 \\ 1.84518 0.0163359 \\ 1.85262 0.0161986 \\ 1.86114 0.0159598 \\ 1.86526 0.0158886 \\ 1.87291 0.0157766 \\
1.88233 0.015667 \\ 1.88627 0.0156166 \\ 1.89301 0.015531 \\ 1.9028 0.0154405 \\ 1.9093 0.0153792 \\ 1.91265 0.0153313 \\
1.92581 0.0152506 \\ 1.93209 0.0151887 \\ 1.94166 0.0151386 \\ 1.94609 0.0150926 \\ 1.95506 0.015032 \\ 1.96446 0.014968 \\
1.98721 0.0145269 \\ 1.99468 0.0144096 \\ 2.00335 0.0142078 \\ 2.00731 0.0141439 \\ 2.01367 0.0140492 \\ 2.02358 0.0139552 \\
2.0271 0.0139106 \\ 2.03498 0.0138383 \\ 2.04456 0.0137607 \\ 2.05141 0.0137086 \\ 2.0561 0.0136675 \\ 2.06836 0.0135982 \\
2.07531 0.0135458 \\ 2.08478 0.0135036 \\ 2.08882 0.0134636 \\ 2.09678 0.0134126 \\ 2.10683 0.0133582 \\ 2.12783 0.0129804 \\
2.13476 0.0128801 \\ 2.14417 0.0127089 \\ 2.14719 0.0126519 \\ 2.15482 0.0125718 \\ 2.16344 0.0124917 \\ 2.16722 0.0124523 \\
2.17494 0.0123914 \\ 2.18475 0.0123252 \\ 2.1882 0.0122921 \\ 2.19711 0.0122394 \\ 2.20755 0.0121809 \\ 2.2142 0.0121414 \\
2.21832 0.0121106 \\ 2.23039 0.0120567 \\ 2.23748 0.0120165 \\ 2.24684 0.0119839 \\ 2.2506 0.0119525 \\ 2.25937 0.0119133 \\
2.26899 0.0118707 \\ 2.29268 0.011529 \\ 2.29917 0.0114447 \\ 2.3081 0.0113002 \\ 2.31207 0.0112522 \\ 2.3188 0.011187 \\
2.32674 0.0111214 \\ 2.33068 0.0110884 \\ 2.3364 0.0110393 \\ 2.34569 0.0109852 \\ 2.34874 0.0109576 \\ 2.35496 0.0109152 \\
2.36549 0.0108675 \\ 2.3715 0.0108353 \\ 2.38036 0.0108078 \\ 2.38357 0.010776 \\ 2.39123 0.0107422 \\ 2.40135 0.0107035 \\
2.42276 0.0104267 \\ 2.4298 0.0103573 \\ 2.43953 0.0102348 \\ 2.44362 0.010193 \\ 2.45122 0.0101376 \\ 2.46126 0.0100815 \\
2.46561 0.0100525 \\ 2.47313 0.0100107 \\ 2.48429 0.00996438 \\ 2.48873 0.00994016 \\ 2.49616 0.00990399 \\ 2.50731 0.00986297 \\
2.51344 0.00983548 \\ 2.52327 0.009812 \\ 2.52695 0.00978422 \\ 2.53467 0.00975545 \\ 2.54399 0.00972218 \\ 2.56751 0.00948969 \\
2.57477 0.00943203 \\ 2.58373 0.0093266 \\ 2.58738 0.00929067 \\ 2.59347 0.00924345 \\ 2.60235 0.00919548 \\ 2.60642 0.00917044 \\
2.61371 0.00913471 \\ 2.62559 0.00909501 \\ 2.62978 0.00907401 \\ 2.63607 0.00904303 \\ 2.64707 0.00900782 \\ 2.65349 0.00898427 \\
2.6645 0.00896419 \\ 2.66917 0.00894002 \\ 2.67797 0.0089153 \\ 2.68818 0.00888669 \\ 2.71171 0.00868803 \\ 2.71972 0.0086391 \\
2.72806 0.0085467 \\ 2.73114 0.00851581 \\ 2.73766 0.00847496 \\ 2.74549 0.00843356 \\ 2.74868 0.00841198 \\ 2.7548 0.00838109 \\
2.76522 0.00834686 \\ 2.76937 0.00832873 \\ 2.77573 0.00830192 \\ 2.78502 0.00827154 \\ 2.79085 0.00825122 \\ 2.79994 0.00823394 \\
2.80354 0.00821291 \\ 2.81121 0.0081915 \\ 2.82177 0.00816679 \\ 2.84307 0.00799295 \\ 2.85038 0.0079502 \\ 2.85856 0.00786738 \\
2.86286 0.00784046 \\ 2.87017 0.00780443 \\ 2.8806 0.00776818 \\ 2.88482 0.00774942 \\ 2.89112 0.00772233 \\ 2.90114 0.0076925 \\
2.90495 0.00767673 \\ 2.91358 0.00765326 \\ 2.9243 0.00762681 \\ 2.931 0.00760911 \\ 2.94063 0.00759412 \\ 2.9445 0.00757568 \\
2.95289 0.00755697 \\ 2.96203 0.00753548 \\ 2.98293 0.00737948 \\ 2.99005 0.0073414 \\ 2.99799 0.00726606 \\ 3.00096 0.00724225 \\
3.00707 0.00721002 \\ 3.01495 0.00717788 \\ 3.01833 0.00716137 \\ 3.02575 0.00713736 \\ 3.03634 0.00711108 \\ 3.04316 0.00709394 \\
3.05321 0.00707967 \\ 3.05761 0.00706145 \\ 3.06561 0.00704344 \\ 3.0753 0.00702303 \\ 3.09735 0.00689083 \\ 3.10478 0.00685689 \\
3.11377 0.0067882 \\ 3.11752 0.00676592 \\ 3.12388 0.00673619 \\ 3.13307 0.00670667 \\ 3.13695 0.00669131 \\ 3.14255 0.00666918 \\
3.15328 0.00664508 \\ 3.15971 0.00662947 \\ 3.16758 0.00661653 \\ 3.17136 0.00659955 \\ 3.17939 0.00658302 \\ 3.18919 0.00656436 \\
3.20953 0.00644341 \\ 3.21674 0.00641251 \\ 3.22503 0.00634897 \\ 3.22908 0.00632832 \\ 3.23546 0.00630089 \\ 3.24519 0.00627381 \\
3.24938 0.00625963 \\ 3.25641 0.00623932 \\ 3.26754 0.00621728 \\ 3.27468 0.00620308 \\ 3.28378 0.00619136 \\ 3.28848 0.00617566 \\
3.29624 0.00616055 \\ 3.30736 0.00614354 \\ 3.32776 0.00603177 \\ 3.33532 0.00600351 \\ 3.34361 0.00594458 \\ 3.34682 0.00592549 \\
3.35293 0.0059002 \\ 3.36189 0.00587537 \\ 3.36479 0.0058623 \\ 3.37056 0.00584368 \\ 3.38011 0.00582354 \\ 3.38632 0.00581063 \\
3.39477 0.00580001 \\ 3.39845 0.00578553 \\ 3.40551 0.00577174 \\ 3.41559 0.00575624 \\ 3.4363 0.00565256 \\ 3.44384 0.00562665 \\
3.45236 0.00557191 \\ 3.45631 0.00555429 \\ 3.46366 0.00553098 \\ 3.47374 0.00550819 \\ 3.47768 0.00549614 \\ 3.48429 0.00547906 \\
3.49451 0.00546064 \\ 3.50237 0.00544889 \\ 3.52421 0.00536766 \\ 3.53151 0.00534689 \\ 3.54018 0.00529488 \\ 3.5437 0.00527756 \\
3.54954 0.0052555 \\ 3.5585 0.00523379 \\ 3.56198 0.00522208 \\ 3.56841 0.00520583 \\ 3.57843 0.00518822 \\ 3.58414 0.00517702 \\
3.59363 0.00516786 \\ 3.597 0.00515498 \\ 3.60404 0.00514295 \\ 3.61282 0.00512945 \\ 3.63349 0.00504039 \\ 3.64062 0.00501895 \\
3.64911 0.00497129 \\ 3.65301 0.00495573 \\ 3.65951 0.00493547 \\ 3.66813 0.00491568 \\ 3.67237 0.00490505 \\ 3.67851 0.00489023 \\
3.6907 0.00487424 \\ 3.69801 0.0048641 \\ 3.71953 0.00479398 \\ 3.72664 0.00477644 \\ 3.73589 0.004731 \\ 3.7398 0.00471582 \\
3.74593 0.00469663 \\ 3.75539 0.00467778 \\ 3.75906 0.0046675 \\ 3.76488 0.0046534 \\ 3.77622 0.00463812 \\ 3.78256 0.00462845 \\
3.79104 0.00462056 \\ 3.79486 0.00460927 \\ 3.80306 0.00459884 \\ 3.81331 0.00458715 \\ 3.83221 0.00450955 \\ 3.83895 0.00449124 \\
3.8475 0.00444941 \\ 3.85084 0.00443579 \\ 3.85721 0.00441812 \\ 3.86764 0.00440089 \\ 3.8725 0.00439157 \\ 3.879 0.00437867 \\
3.89039 0.00436476 \\ 3.89747 0.00435596 \\ 3.92039 0.0042948 \\ 3.92818 0.00427975 \\ 3.93658 0.0042398 \\ 3.9411 0.00422654 \\
3.94779 0.00420976 \\ 3.95681 0.00419333 \\ 3.96077 0.00418432 \\ 3.96702 0.00417202 \\ 3.97746 0.0041587 \\ 3.98402 0.00415028 \\
3.99277 0.00414344 \\ 3.9968 0.00413354 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 2.62004 \\ 0.00657158 2.15617 \\ 0.013018 1.90829 \\ 0.0193127 1.70543 \\ 0.0257121 1.50875 \\ 0.0324574 1.34768 \\
0.0385604 1.22357 \\ 0.0474699 1.04786 \\ 0.0568907 0.903195 \\ 0.066352 0.785781 \\ 0.0785238 0.645765 \\ 0.0883262 0.586898 \\
0.101875 0.500471 \\ 0.110973 0.46886 \\ 0.130684 0.365819 \\ 0.138663 0.353375 \\ 0.148232 0.33605 \\ 0.160858 0.296576 \\
0.171716 0.27033 \\ 0.189032 0.203517 \\ 0.195792 0.19879 \\ 0.206971 0.183014 \\ 0.223253 0.143885 \\ 0.231267 0.14012 \\
0.247357 0.118807 \\ 0.259219 0.114181 \\ 0.283974 0.0874135 \\ 0.291195 0.0864458 \\ 0.301306 0.0854621 \\ 0.314582 0.0733016 \\
0.334941 0.0529782 \\ 0.343173 0.052438 \\ 0.355645 0.0507236 \\ 0.371189 0.0447546 \\ 0.38903 0.0296785 \\ 0.395755 0.0294101 \\
0.404563 0.0291486 \\ 0.414519 0.02864 \\ 0.428561 0.0258711 \\ 0.450828 0.0176964 \\ 0.457723 0.0176265 \\ 0.467184 0.0175191 \\
0.479153 0.0172687 \\ 0.492621 0.0160677 \\ 0.519148 0.00966453 \\ 0.526507 0.00962143 \\ 0.538475 0.00939913 \\ 0.551971 0.00861624 \\
0.576944 0.0051708 \\ 0.583879 0.00514865 \\ 0.592874 0.00511592 \\ 0.603933 0.00504399 \\ 0.616388 0.00466851 \\ 0.641031 0.00251996 \\
0.648256 0.00251086 \\ 0.657361 0.00249927 \\ 0.669159 0.00246938 \\ 0.682309 0.00233811 \\ 0.705699 0.000883204 \\ 0.71323 0.000880258 \\
0.722211 0.000875852 \\ 0.734016 0.000861695 \\ 0.747904 0.000811354 \\ 0.775695 0.000295298 \\ 0.783619 0.000294649 \\ 0.792911 0.000293459 \\
0.804322 0.000289499 \\ 0.817154 0.000276848 \\ 0.838208 5.82265e-05 \\ 0.844976 5.80942e-05 \\ 0.853279 5.782e-05 \\ 0.863909 5.70698e-05 \\
0.876468 5.37935e-05 \\ 0.901033 5.25335e-06 \\ 0.90831 5.24328e-06 \\ 0.917486 5.22402e-06 \\ 0.92881 5.16688e-06 \\ 0.941927 4.83653e-06 \\
0.958593 2.71006e-08 \\ 0.966444 2.68984e-08 \\ 0.975576 2.65219e-08 \\ 0.98708 2.5549e-08 \\ 1.0008 2.16667e-08 \\ 1.01422 1.00131e-12 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 2.62004 \\ 0.014207 2.04261 \\ 0.0270202 1.66719 \\ 0.0394301 1.41893 \\ 0.0520846 1.21069 \\ 0.0654954 1.03654 \\
0.0783087 0.882149 \\ 0.091297 0.735464 \\ 0.105715 0.602441 \\ 0.118535 0.483466 \\ 0.13249 0.378142 \\ 0.144595 0.287996 \\
0.156914 0.22183 \\ 0.172681 0.168629 \\ 0.18736 0.1266 \\ 0.204624 0.100262 \\ 0.221444 0.079239 \\ 0.235408 0.0594962 \\
0.249108 0.0447528 \\ 0.265357 0.0341128 \\ 0.281078 0.0247474 \\ 0.296953 0.0182319 \\ 0.312847 0.013138 \\ 0.325479 0.00852754 \\
0.337947 0.00497867 \\ 0.356079 0.00301797 \\ 0.369241 0.00164924 \\ 0.381868 0.000772019 \\ 0.398489 0.000326264 \\ 0.414184 0.000122966 \\
0.429753 2.73494e-05 \\ 0.444529 3.30271e-06 \\ 0.459579 1.10255e-07 \\ 0.474395 4.80382e-10 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 2.62004 \\ 0.00902808 2.11368 \\ 0.0183005 1.82008 \\ 0.0268793 1.57698 \\ 0.0356993 1.38079 \\ 0.0437612 1.2104 \\
0.0525115 1.07496 \\ 0.0599446 0.95755 \\ 0.0689149 0.843897 \\ 0.076837 0.752349 \\ 0.0934599 0.594379 \\ 0.103623 0.548336 \\
0.119916 0.426255 \\ 0.128278 0.397026 \\ 0.144268 0.299201 \\ 0.153688 0.279723 \\ 0.170085 0.205973 \\ 0.179252 0.197038 \\
0.19424 0.142244 \\ 0.202199 0.136916 \\ 0.216641 0.0967403 \\ 0.225333 0.0937064 \\ 0.244861 0.0751158 \\ 0.25285 0.0741583 \\
0.270975 0.0605005 \\ 0.280624 0.0598373 \\ 0.296568 0.0462955 \\ 0.307202 0.0458972 \\ 0.324134 0.0338068 \\ 0.333862 0.0334532 \\
0.349599 0.0227015 \\ 0.359401 0.0223391 \\ 0.38175 0.0168823 \\ 0.390644 0.0167651 \\ 0.409263 0.0126302 \\ 0.418587 0.0125609 \\
0.434269 0.0085827 \\ 0.442771 0.00853239 \\ 0.456716 0.00419159 \\ 0.465684 0.00413693 \\ 0.485535 0.00215738 \\ 0.495083 0.00214286 \\
0.511404 0.000932538 \\ 0.521221 0.000924419 \\ 0.540785 0.000382321 \\ 0.550788 0.000380452 \\ 0.567929 0.000117205 \\ 0.57805 0.00011661 \\
0.590997 1.7294e-05 \\ 0.600742 1.72194e-05 \\ 0.614204 5.58444e-07 \\ 0.623971 5.54315e-07 \\ 0.636831 1.67421e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(d) $\gamma=0.1$, runtime};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=558,
ymin=1.479e-13, ymax=3.623e+00,
ytickten={-12,-9,-6,-3,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.892289 \\ 1 0.856823 \\ 2 0.819789 \\ 3 0.780921 \\ 4 0.739686 \\ 5 0.695712 \\
6 0.620709 \\ 7 0.572333 \\ 8 0.492742 \\ 9 0.443235 \\ 10 0.367583 \\ 11 0.325908 \\
12 0.263216 \\ 13 0.241106 \\ 14 0.214556 \\ 15 0.167784 \\ 16 0.15446 \\ 17 0.134533 \\
18 0.12568 \\ 19 0.115778 \\ 20 0.104512 \\ 21 0.0633304 \\ 22 0.059111 \\ 23 0.0487491 \\
24 0.0466881 \\ 25 0.0432648 \\ 26 0.0404797 \\ 27 0.0394271 \\ 28 0.0375916 \\ 29 0.0358542 \\
30 0.0347966 \\ 31 0.0286345 \\ 32 0.0276409 \\ 33 0.0245935 \\ 34 0.0238228 \\ 35 0.022628 \\
36 0.0215658 \\ 37 0.0211398 \\ 38 0.0204106 \\ 39 0.0196774 \\ 40 0.0192116 \\ 41 0.0188427 \\
42 0.0184542 \\ 43 0.0180517 \\ 44 0.0176137 \\ 45 0.015076 \\ 46 0.0146481 \\ 47 0.0134842 \\
48 0.0131769 \\ 49 0.012713 \\ 50 0.0122857 \\ 51 0.0120988 \\ 52 0.0117958 \\ 53 0.0114852 \\
54 0.0112898 \\ 55 0.0102106 \\ 56 0.00960499 \\ 57 0.00941754 \\ 58 0.00903999 \\ 59 0.00874299 \\
60 0.0086191 \\ 61 0.00840986 \\ 62 0.00820374 \\ 63 0.00810824 \\ 64 0.0079505 \\ 65 0.00778722 \\
66 0.00706414 \\ 67 0.00690508 \\ 68 0.00651254 \\ 69 0.00638921 \\ 70 0.00622224 \\ 71 0.00606069 \\
72 0.00598265 \\ 73 0.00586628 \\ 74 0.00574418 \\ 75 0.00566932 \\ 76 0.00560544 \\ 77 0.00550461 \\
78 0.005435 \\ 79 0.00497253 \\ 80 0.00488538 \\ 81 0.00482592 \\ 82 0.00453822 \\ 83 0.00443128 \\
84 0.00437624 \\ 85 0.00427362 \\ 86 0.0042011 \\ 87 0.00415939 \\ 88 0.00408374 \\ 89 0.00402639 \\
90 0.00372687 \\ 91 0.00366729 \\ 92 0.00348034 \\ 93 0.0034243 \\ 94 0.00334921 \\ 95 0.00327563 \\
96 0.00324086 \\ 97 0.00318877 \\ 98 0.00313353 \\ 99 0.00291071 \\ 100 0.00285849 \\ 101 0.00272596 \\
102 0.00268267 \\ 103 0.00262536 \\ 104 0.00256873 \\ 105 0.00254198 \\ 106 0.00250152 \\ 107 0.00245878 \\
108 0.00229299 \\ 109 0.00225092 \\ 110 0.0021515 \\ 111 0.00211816 \\ 112 0.00207391 \\ 113 0.00203025 \\
114 0.00200978 \\ 115 0.00197839 \\ 116 0.0019454 \\ 117 0.0018186 \\ 118 0.00178511 \\ 119 0.00176624 \\
120 0.00168191 \\ 121 0.00164643 \\ 122 0.00163032 \\ 123 0.00159656 \\ 124 0.00157238 \\ 125 0.00155504 \\
126 0.00154007 \\ 127 0.0015215 \\ 128 0.00150198 \\ 129 0.00139525 \\ 130 0.00137001 \\ 131 0.00131202 \\
132 0.001294 \\ 133 0.0012689 \\ 134 0.00124446 \\ 135 0.00123354 \\ 136 0.00121595 \\ 137 0.00119762 \\
138 0.0011213 \\ 139 0.00110079 \\ 140 0.00105565 \\ 141 0.00104141 \\ 142 0.00102118 \\ 143 0.00100153 \\
144 0.000989682 \\ 145 0.000980363 \\ 146 0.000965365 \\ 147 0.000954945 \\ 148 0.000897352 \\ 149 0.000884243 \\
150 0.000847687 \\ 151 0.000836048 \\ 152 0.000820304 \\ 153 0.000805024 \\ 154 0.000795886 \\ 155 0.000788503 \\
156 0.000776782 \\ 157 0.00076869 \\ 158 0.000723012 \\ 159 0.000712918 \\ 160 0.000705699 \\ 161 0.000673859 \\
162 0.000660848 \\ 163 0.000655239 \\ 164 0.00064311 \\ 165 0.000634379 \\ 166 0.000628081 \\ 167 0.000590437 \\
168 0.000580994 \\ 169 0.000558363 \\ 170 0.000550999 \\ 171 0.000540822 \\ 172 0.000530962 \\ 173 0.000524975 \\
174 0.00052037 \\ 175 0.000512877 \\ 176 0.000507625 \\ 177 0.000478218 \\ 178 0.000471571 \\ 179 0.000452818 \\
180 0.000446927 \\ 181 0.000438845 \\ 182 0.000431013 \\ 183 0.000426271 \\ 184 0.000422635 \\ 185 0.000416712 \\
186 0.000412562 \\ 187 0.000388735 \\ 188 0.000383454 \\ 189 0.000368272 \\ 190 0.000363561 \\ 191 0.00035707 \\
192 0.000350789 \\ 193 0.000346979 \\ 194 0.000344097 \\ 195 0.000339372 \\ 196 0.000336052 \\ 197 0.000316734 \\
198 0.000312468 \\ 199 0.000300183 \\ 200 0.000296409 \\ 201 0.000291176 \\ 202 0.00028612 \\ 203 0.000283045 \\
204 0.000280752 \\ 205 0.000276967 \\ 206 0.00026103 \\ 207 0.000256719 \\ 208 0.000247112 \\ 209 0.000244221 \\
210 0.00023988 \\ 211 0.000235734 \\ 212 0.000233166 \\ 213 0.000231332 \\ 214 0.000228251 \\ 215 0.000226033 \\
216 0.000213203 \\ 217 0.000210233 \\ 218 0.000202154 \\ 219 0.000199729 \\ 220 0.000196261 \\ 221 0.000192928 \\
222 0.000190883 \\ 223 0.000189404 \\ 224 0.00018693 \\ 225 0.000176345 \\ 226 0.000173392 \\ 227 0.000167019 \\
228 0.000165133 \\ 229 0.000162249 \\ 230 0.000159504 \\ 231 0.000157794 \\ 232 0.000156473 \\ 233 0.000155047 \\
234 0.000146183 \\ 235 0.000143696 \\ 236 0.000138369 \\ 237 0.000136878 \\ 238 0.00013448 \\ 239 0.000132214 \\
240 0.000130799 \\ 241 0.000129706 \\ 242 0.000128536 \\ 243 0.000121311 \\ 244 0.000119211 \\ 245 0.000114805 \\
246 0.0001136 \\ 247 0.000111613 \\ 248 0.000109746 \\ 249 0.000108578 \\ 250 0.000107675 \\ 251 0.000106717 \\
252 0.000100774 \\ 253 9.90096e-05 \\ 254 9.53707e-05 \\ 255 9.4387e-05 \\ 256 9.27422e-05 \\ 257 9.12045e-05 \\
258 9.02407e-05 \\ 259 8.94953e-05 \\ 260 8.87088e-05 \\ 261 8.78468e-05 \\ 262 8.69436e-05 \\ 263 8.146e-05 \\
264 8.00353e-05 \\ 265 7.7159e-05 \\ 266 7.63088e-05 \\ 267 7.50287e-05 \\ 268 7.38314e-05 \\ 269 7.30962e-05 \\
270 7.25291e-05 \\ 271 7.18918e-05 \\ 272 6.7834e-05 \\ 273 6.66315e-05 \\ 274 6.42307e-05 \\ 275 6.3564e-05 \\
276 6.24844e-05 \\ 277 6.14801e-05 \\ 278 6.08567e-05 \\ 279 6.03756e-05 \\ 280 5.9855e-05 \\ 281 5.65403e-05 \\
282 5.55314e-05 \\ 283 5.3537e-05 \\ 284 5.29936e-05 \\ 285 5.20937e-05 \\ 286 5.12581e-05 \\ 287 5.07378e-05 \\
288 5.03358e-05 \\ 289 4.99068e-05 \\ 290 4.71686e-05 \\ 291 4.6323e-05 \\ 292 4.46677e-05 \\ 293 4.42202e-05 \\
294 4.34718e-05 \\ 295 4.27779e-05 \\ 296 4.23453e-05 \\ 297 4.20109e-05 \\ 298 4.16561e-05 \\ 299 3.93849e-05 \\
300 3.86761e-05 \\ 301 3.73018e-05 \\ 302 3.69313e-05 \\ 303 3.63091e-05 \\ 304 3.5733e-05 \\ 305 3.53736e-05 \\
306 3.50959e-05 \\ 307 3.48017e-05 \\ 308 3.29135e-05 \\ 309 3.23191e-05 \\ 310 3.11774e-05 \\ 311 3.08697e-05 \\
312 3.03522e-05 \\ 313 2.98734e-05 \\ 314 2.9575e-05 \\ 315 2.93444e-05 \\ 316 2.78022e-05 \\ 317 2.73316e-05 \\
318 2.64252e-05 \\ 319 2.61008e-05 \\ 320 2.56665e-05 \\ 321 2.52586e-05 \\ 322 2.50153e-05 \\ 323 2.48163e-05 \\
324 2.45113e-05 \\ 325 2.42988e-05 \\ 326 2.29849e-05 \\ 327 2.26462e-05 \\ 328 2.18632e-05 \\ 329 2.16087e-05 \\
330 2.12584e-05 \\ 331 2.09282e-05 \\ 332 2.07305e-05 \\ 333 1.97946e-05 \\ 334 1.95357e-05 \\ 335 1.93377e-05 \\
336 1.85876e-05 \\ 337 1.82818e-05 \\ 338 1.81409e-05 \\ 339 1.78516e-05 \\ 340 1.76442e-05 \\ 341 1.74932e-05 \\
342 1.65795e-05 \\ 343 1.63168e-05 \\ 344 1.57822e-05 \\ 345 1.55909e-05 \\ 346 1.5338e-05 \\ 347 1.50957e-05 \\
348 1.49509e-05 \\ 349 1.48369e-05 \\ 350 1.46575e-05 \\ 351 1.4532e-05 \\ 352 1.37581e-05 \\ 353 1.35625e-05 \\
354 1.30999e-05 \\ 355 1.29461e-05 \\ 356 1.27402e-05 \\ 357 1.25432e-05 \\ 358 1.24258e-05 \\ 359 1.23358e-05 \\
360 1.22275e-05 \\ 361 1.15645e-05 \\ 362 1.1372e-05 \\ 363 1.09887e-05 \\ 364 1.08715e-05 \\ 365 1.06968e-05 \\
366 1.05318e-05 \\ 367 1.04317e-05 \\ 368 1.03549e-05 \\ 369 1.02666e-05 \\ 370 9.71912e-06 \\ 371 9.55021e-06 \\
372 9.2287e-06 \\ 373 9.13352e-06 \\ 374 8.9867e-06 \\ 375 8.84879e-06 \\ 376 8.76478e-06 \\ 377 8.70022e-06 \\
378 8.62701e-06 \\ 379 8.1698e-06 \\ 380 8.02488e-06 \\ 381 7.75536e-06 \\ 382 7.67678e-06 \\ 383 7.55364e-06 \\
384 7.43839e-06 \\ 385 7.36807e-06 \\ 386 7.31401e-06 \\ 387 6.94386e-06 \\ 388 6.82802e-06 \\ 389 6.76199e-06 \\
390 6.51443e-06 \\ 391 6.41307e-06 \\ 392 6.36407e-06 \\ 393 6.26745e-06 \\ 394 6.1978e-06 \\ 395 6.14728e-06 \\
396 5.83322e-06 \\ 397 5.73401e-06 \\ 398 5.55387e-06 \\ 399 5.48707e-06 \\ 400 5.40096e-06 \\ 401 5.3185e-06 \\
402 5.26996e-06 \\ 403 5.03885e-06 \\ 404 4.97005e-06 \\ 405 4.91869e-06 \\ 406 4.73518e-06 \\ 407 4.66077e-06 \\
408 4.62419e-06 \\ 409 4.55244e-06 \\ 410 4.50152e-06 \\ 411 4.46468e-06 \\ 412 4.2393e-06 \\ 413 4.17092e-06 \\
414 4.04134e-06 \\ 415 3.99088e-06 \\ 416 3.9285e-06 \\ 417 3.86792e-06 \\ 418 3.83262e-06 \\ 419 3.80295e-06 \\
420 3.75788e-06 \\ 421 3.72701e-06 \\ 422 3.53245e-06 \\ 423 3.4812e-06 \\ 424 3.36806e-06 \\ 425 3.32755e-06 \\
426 3.27649e-06 \\ 427 3.22705e-06 \\ 428 3.19829e-06 \\ 429 3.05955e-06 \\ 430 3.01972e-06 \\ 431 2.98805e-06 \\
432 2.87651e-06 \\ 433 2.83174e-06 \\ 434 2.80961e-06 \\ 435 2.76631e-06 \\ 436 2.73581e-06 \\ 437 2.7138e-06 \\
438 2.57717e-06 \\ 439 2.53656e-06 \\ 440 2.4585e-06 \\ 441 2.42741e-06 \\ 442 2.39001e-06 \\ 443 2.35334e-06 \\
444 2.33217e-06 \\ 445 2.3141e-06 \\ 446 2.28683e-06 \\ 447 2.26831e-06 \\ 448 2.15077e-06 \\ 449 2.11989e-06 \\
450 2.05178e-06 \\ 451 2.0268e-06 \\ 452 1.9961e-06 \\ 453 1.96613e-06 \\ 454 1.94884e-06 \\ 455 1.86504e-06 \\
456 1.84093e-06 \\ 457 1.82139e-06 \\ 458 1.75382e-06 \\ 459 1.72686e-06 \\ 460 1.71327e-06 \\ 461 1.68703e-06 \\
462 1.66864e-06 \\ 463 1.65541e-06 \\ 464 1.57265e-06 \\ 465 1.54804e-06 \\ 466 1.5331e-06 \\ 467 1.47804e-06 \\
468 1.45551e-06 \\ 469 1.44024e-06 \\ 470 1.42745e-06 \\ 471 1.41233e-06 \\ 472 1.39667e-06 \\ 473 1.32667e-06 \\
474 1.30458e-06 \\ 475 1.26478e-06 \\ 476 1.24948e-06 \\ 477 1.23057e-06 \\ 478 1.21196e-06 \\ 479 1.2012e-06 \\
480 1.14985e-06 \\ 481 1.13435e-06 \\ 482 1.12245e-06 \\ 483 1.08145e-06 \\ 484 1.065e-06 \\ 485 1.05658e-06 \\
486 1.04045e-06 \\ 487 1.02916e-06 \\ 488 1.02105e-06 \\ 489 9.70702e-07 \\ 490 9.55258e-07 \\ 491 9.46003e-07 \\
492 9.12377e-07 \\ 493 8.98623e-07 \\ 494 8.89303e-07 \\ 495 8.81346e-07 \\ 496 8.72062e-07 \\ 497 8.62414e-07 \\
498 8.19669e-07 \\ 499 8.05873e-07 \\ 500 7.81657e-07 \\ 501 7.72113e-07 \\ 502 7.60534e-07 \\ 503 7.4908e-07 \\
504 7.42502e-07 \\ 505 7.10935e-07 \\ 506 7.01268e-07 \\ 507 6.93837e-07 \\ 508 6.68692e-07 \\ 509 6.58639e-07 \\
510 6.53379e-07 \\ 511 6.43454e-07 \\ 512 6.36532e-07 \\ 513 6.3157e-07 \\ 514 6.00712e-07 \\ 515 5.91107e-07 \\
516 5.85311e-07 \\ 517 5.64662e-07 \\ 518 5.56245e-07 \\ 519 5.52028e-07 \\ 520 5.43841e-07 \\ 521 5.3804e-07 \\
522 5.33873e-07 \\ 523 5.07537e-07 \\ 524 4.99114e-07 \\ 525 4.84399e-07 \\ 526 4.78222e-07 \\ 527 4.71141e-07 \\
528 4.64079e-07 \\ 529 4.60081e-07 \\ 530 4.56511e-07 \\ 531 4.5124e-07 \\ 532 4.4771e-07 \\ 533 4.24757e-07 \\
534 4.18471e-07 \\ 535 4.05559e-07 \\ 536 4.00533e-07 \\ 537 3.94687e-07 \\ 538 3.88876e-07 \\ 539 3.85599e-07 \\
540 3.69265e-07 \\ 541 3.65018e-07 \\ 542 3.598e-07 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.892289 \\ 1 0.85324 \\ 2 0.812681 \\ 3 0.770475 \\ 4 0.726512 \\ 5 0.681145 \\
6 0.635411 \\ 7 0.566084 \\ 8 0.502384 \\ 9 0.446006 \\ 10 0.362573 \\ 11 0.296648 \\
12 0.24667 \\ 13 0.157121 \\ 14 0.133603 \\ 15 0.0855638 \\ 16 0.0613181 \\ 17 0.0270768 \\
18 0.0266708 \\ 19 0.0245764 \\ 20 0.0193805 \\ 21 0.00375514 \\ 22 0.00368098 \\ 23 0.00357297 \\
24 0.00335934 \\ 25 0.00282382 \\ 26 5.70838e-05 \\ 27 5.29865e-05 \\ 28 4.84e-05 \\ 29 4.27866e-05 \\
30 3.35564e-05 \\ 31 1.00816e-08 \\ 32 9.26111e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.892289 \\ 1 0.82317 \\ 2 0.759037 \\ 3 0.695303 \\ 4 0.629444 \\ 5 0.560673 \\
6 0.489551 \\ 7 0.417474 \\ 8 0.346436 \\ 9 0.27886 \\ 10 0.217275 \\ 11 0.16391 \\
12 0.119085 \\ 13 0.0825614 \\ 14 0.0541362 \\ 15 0.0329191 \\ 16 0.0182497 \\ 17 0.00904316 \\
18 0.00371144 \\ 19 0.00107568 \\ 20 0.000175038 \\ 21 1.08363e-05 \\ 22 1.29045e-07 \\ 23 1.01554e-10 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.892289 \\ 1 0.842268 \\ 2 0.792419 \\ 3 0.741548 \\ 4 0.689009 \\ 5 0.634915 \\
6 0.580158 \\ 7 0.526202 \\ 8 0.47473 \\ 9 0.427613 \\ 10 0.309416 \\ 11 0.277269 \\
12 0.180364 \\ 13 0.163865 \\ 14 0.0958351 \\ 15 0.0898533 \\ 16 0.0450625 \\ 17 0.0431453 \\
18 0.0171994 \\ 19 0.0166538 \\ 20 0.00467334 \\ 21 0.0045521 \\ 22 0.000567494 \\ 23 0.00055295 \\
24 1.49448e-05 \\ 25 1.44797e-05 \\ 26 2.79147e-08 \\ 27 2.71205e-08 \\ 28 6.00409e-13 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(e) $\gamma=0.5$, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=4.12,
ymin=1.479e-13, ymax=3.623e+00,
ytickten={-12,-9,-6,-3,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.892289 \\ 0.00697496 0.856823 \\ 0.0131778 0.819789 \\ 0.0192875 0.780921 \\ 0.0254993 0.739686 \\ 0.0311688 0.695712 \\
0.0389777 0.620709 \\ 0.04533 0.572333 \\ 0.0529931 0.492742 \\ 0.0590774 0.443235 \\ 0.0668218 0.367583 \\ 0.0732781 0.325908 \\
0.0813902 0.263216 \\ 0.0847307 0.241106 \\ 0.0919553 0.214556 \\ 0.0999307 0.167784 \\ 0.106964 0.15446 \\ 0.115429 0.134533 \\
0.119012 0.12568 \\ 0.125831 0.115778 \\ 0.133979 0.104512 \\ 0.144641 0.0633304 \\ 0.151134 0.059111 \\ 0.160334 0.0487491 \\
0.163846 0.0466881 \\ 0.170481 0.0432648 \\ 0.179433 0.0404797 \\ 0.182846 0.0394271 \\ 0.189172 0.0375916 \\ 0.197599 0.0358542 \\
0.204009 0.0347966 \\ 0.222083 0.0286345 \\ 0.228785 0.0276409 \\ 0.236385 0.0245935 \\ 0.239965 0.0238228 \\ 0.247049 0.022628 \\
0.255813 0.0215658 \\ 0.259018 0.0211398 \\ 0.265007 0.0204106 \\ 0.275486 0.0196774 \\ 0.281737 0.0192116 \\ 0.290766 0.0188427 \\
0.294161 0.0184542 \\ 0.301624 0.0180517 \\ 0.311941 0.0176137 \\ 0.331209 0.015076 \\ 0.337947 0.0146481 \\ 0.347236 0.0134842 \\
0.350785 0.0131769 \\ 0.357303 0.012713 \\ 0.365333 0.0122857 \\ 0.369328 0.0120988 \\ 0.377402 0.0117958 \\ 0.386612 0.0114852 \\
0.394443 0.0112898 \\ 0.412037 0.0102106 \\ 0.420277 0.00960499 \\ 0.424229 0.00941754 \\ 0.430685 0.00903999 \\ 0.440707 0.00874299 \\
0.444526 0.0086191 \\ 0.450773 0.00840986 \\ 0.461136 0.00820374 \\ 0.464725 0.00810824 \\ 0.471318 0.0079505 \\ 0.480412 0.00778722 \\
0.498797 0.00706414 \\ 0.506876 0.00690508 \\ 0.514625 0.00651254 \\ 0.518305 0.00638921 \\ 0.525455 0.00622224 \\ 0.535767 0.00606069 \\
0.539575 0.00598265 \\ 0.548076 0.00586628 \\ 0.558609 0.00574418 \\ 0.564621 0.00566932 \\ 0.568746 0.00560544 \\ 0.580814 0.00550461 \\
0.587867 0.005435 \\ 0.605813 0.00497253 \\ 0.612709 0.00488538 \\ 0.616085 0.00482592 \\ 0.624823 0.00453822 \\ 0.629944 0.00443128 \\
0.633812 0.00437624 \\ 0.642612 0.00427362 \\ 0.648604 0.0042011 \\ 0.651864 0.00415939 \\ 0.663588 0.00408374 \\ 0.670272 0.00402639 \\
0.687868 0.00372687 \\ 0.69415 0.00366729 \\ 0.702047 0.00348034 \\ 0.706303 0.0034243 \\ 0.712609 0.00334921 \\ 0.722473 0.00327563 \\
0.726455 0.00324086 \\ 0.733182 0.00318877 \\ 0.743842 0.00313353 \\ 0.763036 0.00291071 \\ 0.770059 0.00285849 \\ 0.77849 0.00272596 \\
0.781921 0.00268267 \\ 0.788329 0.00262536 \\ 0.797781 0.00256873 \\ 0.801843 0.00254198 \\ 0.807558 0.00250152 \\ 0.817351 0.00245878 \\
0.835131 0.00229299 \\ 0.841651 0.00225092 \\ 0.84971 0.0021515 \\ 0.853428 0.00211816 \\ 0.858905 0.00207391 \\ 0.867273 0.00203025 \\
0.870622 0.00200978 \\ 0.876817 0.00197839 \\ 0.887377 0.0019454 \\ 0.904332 0.0018186 \\ 0.911611 0.00178511 \\ 0.914987 0.00176624 \\
0.925061 0.00168191 \\ 0.931142 0.00164643 \\ 0.935101 0.00163032 \\ 0.94404 0.00159656 \\ 0.949586 0.00157238 \\ 0.957961 0.00155504 \\
0.96139 0.00154007 \\ 0.967667 0.0015215 \\ 0.975554 0.00150198 \\ 0.994982 0.00139525 \\ 1.00171 0.00137001 \\ 1.01013 0.00131202 \\
1.01372 0.001294 \\ 1.01986 0.0012689 \\ 1.02946 0.00124446 \\ 1.03237 0.00123354 \\ 1.03846 0.00121595 \\ 1.04623 0.00119762 \\
1.06284 0.0011213 \\ 1.0688 0.00110079 \\ 1.07703 0.00105565 \\ 1.08071 0.00104141 \\ 1.0872 0.00102118 \\ 1.09521 0.00100153 \\
1.10188 0.000989682 \\ 1.10526 0.000980363 \\ 1.11305 0.000965365 \\ 1.1197 0.000954945 \\ 1.13755 0.000897352 \\ 1.14419 0.000884243 \\
1.15208 0.000847687 \\ 1.15603 0.000836048 \\ 1.16276 0.000820304 \\ 1.17099 0.000805024 \\ 1.17724 0.000795886 \\ 1.18075 0.000788503 \\
1.18981 0.000776782 \\ 1.19712 0.00076869 \\ 1.21281 0.000723012 \\ 1.22049 0.000712918 \\ 1.22448 0.000705699 \\ 1.23479 0.000673859 \\
1.24151 0.000660848 \\ 1.24491 0.000655239 \\ 1.25272 0.00064311 \\ 1.25878 0.000634379 \\ 1.26745 0.000628081 \\ 1.28215 0.000590437 \\
1.28882 0.000580994 \\ 1.29685 0.000558363 \\ 1.3004 0.000550999 \\ 1.30723 0.000540822 \\ 1.31491 0.000530962 \\ 1.32156 0.000524975 \\
1.32569 0.00052037 \\ 1.33345 0.000512877 \\ 1.34015 0.000507625 \\ 1.35422 0.000478218 \\ 1.36206 0.000471571 \\ 1.37043 0.000452818 \\
1.37492 0.000446927 \\ 1.38112 0.000438845 \\ 1.39073 0.000431013 \\ 1.39752 0.000426271 \\ 1.4016 0.000422635 \\ 1.41042 0.000416712 \\
1.41707 0.000412562 \\ 1.43053 0.000388735 \\ 1.43752 0.000383454 \\ 1.44514 0.000368272 \\ 1.44899 0.000363561 \\ 1.45538 0.00035707 \\
1.46352 0.000350789 \\ 1.47033 0.000346979 \\ 1.47373 0.000344097 \\ 1.4813 0.000339372 \\ 1.48773 0.000336052 \\ 1.50042 0.000316734 \\
1.50701 0.000312468 \\ 1.51542 0.000300183 \\ 1.51957 0.000296409 \\ 1.52667 0.000291176 \\ 1.53485 0.00028612 \\ 1.54208 0.000283045 \\
1.54634 0.000280752 \\ 1.55431 0.000276967 \\ 1.56683 0.00026103 \\ 1.57388 0.000256719 \\ 1.58225 0.000247112 \\ 1.58572 0.000244221 \\
1.59314 0.00023988 \\ 1.60118 0.000235734 \\ 1.60813 0.000233166 \\ 1.61182 0.000231332 \\ 1.61952 0.000228251 \\ 1.6257 0.000226033 \\
1.63569 0.000213203 \\ 1.64344 0.000210233 \\ 1.65097 0.000202154 \\ 1.6549 0.000199729 \\ 1.66229 0.000196261 \\ 1.67061 0.000192928 \\
1.67666 0.000190883 \\ 1.67975 0.000189404 \\ 1.68725 0.00018693 \\ 1.69721 0.000176345 \\ 1.70405 0.000173392 \\ 1.71325 0.000167019 \\
1.71647 0.000165133 \\ 1.72369 0.000162249 \\ 1.73201 0.000159504 \\ 1.73911 0.000157794 \\ 1.74803 0.000156473 \\ 1.75203 0.000155047 \\
1.76271 0.000146183 \\ 1.77013 0.000143696 \\ 1.77918 0.000138369 \\ 1.78319 0.000136878 \\ 1.78939 0.00013448 \\ 1.79852 0.000132214 \\
1.80507 0.000130799 \\ 1.81394 0.000129706 \\ 1.81739 0.000128536 \\ 1.82833 0.000121311 \\ 1.83481 0.000119211 \\ 1.84349 0.000114805 \\
1.84752 0.0001136 \\ 1.85389 0.000111613 \\ 1.86233 0.000109746 \\ 1.86901 0.000108578 \\ 1.87759 0.000107675 \\ 1.88058 0.000106717 \\
1.8911 0.000100774 \\ 1.89702 9.90096e-05 \\ 1.90523 9.53707e-05 \\ 1.90886 9.4387e-05 \\ 1.91453 9.27422e-05 \\ 1.9231 9.12045e-05 \\
1.92933 9.02407e-05 \\ 1.93905 8.94953e-05 \\ 1.94279 8.87088e-05 \\ 1.94978 8.78468e-05 \\ 1.95876 8.69436e-05 \\ 1.96979 8.146e-05 \\
1.97625 8.00353e-05 \\ 1.98512 7.7159e-05 \\ 1.98885 7.63088e-05 \\ 1.99513 7.50287e-05 \\ 2.00325 7.38314e-05 \\ 2.01053 7.30962e-05 \\
2.01895 7.25291e-05 \\ 2.0223 7.18918e-05 \\ 2.03212 6.7834e-05 \\ 2.03942 6.66315e-05 \\ 2.0481 6.42307e-05 \\ 2.0513 6.3564e-05 \\
2.0578 6.24844e-05 \\ 2.06588 6.14801e-05 \\ 2.07248 6.08567e-05 \\ 2.08077 6.03756e-05 \\ 2.084 5.9855e-05 \\ 2.09412 5.65403e-05 \\
2.10053 5.55314e-05 \\ 2.10877 5.3537e-05 \\ 2.11192 5.29936e-05 \\ 2.1184 5.20937e-05 \\ 2.1271 5.12581e-05 \\ 2.13319 5.07378e-05 \\
2.14177 5.03358e-05 \\ 2.14574 4.99068e-05 \\ 2.15633 4.71686e-05 \\ 2.1625 4.6323e-05 \\ 2.17263 4.46677e-05 \\ 2.17631 4.42202e-05 \\
2.18283 4.34718e-05 \\ 2.19135 4.27779e-05 \\ 2.19815 4.23453e-05 \\ 2.20694 4.20109e-05 \\ 2.21051 4.16561e-05 \\ 2.22075 3.93849e-05 \\
2.22859 3.86761e-05 \\ 2.23662 3.73018e-05 \\ 2.24055 3.69313e-05 \\ 2.24654 3.63091e-05 \\ 2.25499 3.5733e-05 \\ 2.26086 3.53736e-05 \\
2.26833 3.50959e-05 \\ 2.27202 3.48017e-05 \\ 2.28141 3.29135e-05 \\ 2.28822 3.23191e-05 \\ 2.2968 3.11774e-05 \\ 2.30044 3.08697e-05 \\
2.30696 3.03522e-05 \\ 2.31497 2.98734e-05 \\ 2.32121 2.9575e-05 \\ 2.32931 2.93444e-05 \\ 2.34109 2.78022e-05 \\ 2.34835 2.73316e-05 \\
2.3567 2.64252e-05 \\ 2.36013 2.61008e-05 \\ 2.36693 2.56665e-05 \\ 2.37594 2.52586e-05 \\ 2.38246 2.50153e-05 \\ 2.3863 2.48163e-05 \\
2.39426 2.45113e-05 \\ 2.4005 2.42988e-05 \\ 2.41222 2.29849e-05 \\ 2.4189 2.26462e-05 \\ 2.42811 2.18632e-05 \\ 2.43165 2.16087e-05 \\
2.43884 2.12584e-05 \\ 2.50016 2.09282e-05 \\ 2.50705 2.07305e-05 \\ 2.51752 1.97946e-05 \\ 2.52374 1.95357e-05 \\ 2.52675 1.93377e-05 \\
2.53466 1.85876e-05 \\ 2.54159 1.82818e-05 \\ 2.54591 1.81409e-05 \\ 2.55394 1.78516e-05 \\ 2.56087 1.76442e-05 \\ 2.56932 1.74932e-05 \\
2.58031 1.65795e-05 \\ 2.5871 1.63168e-05 \\ 2.59633 1.57822e-05 \\ 2.60012 1.55909e-05 \\ 2.60677 1.5338e-05 \\ 2.61431 1.50957e-05 \\
2.62133 1.49509e-05 \\ 2.62578 1.48369e-05 \\ 2.63372 1.46575e-05 \\ 2.6401 1.4532e-05 \\ 2.65002 1.37581e-05 \\ 2.6571 1.35625e-05 \\
2.66462 1.30999e-05 \\ 2.66851 1.29461e-05 \\ 2.67511 1.27402e-05 \\ 2.68409 1.25432e-05 \\ 2.69042 1.24258e-05 \\ 2.69804 1.23358e-05 \\
2.70217 1.22275e-05 \\ 2.71192 1.15645e-05 \\ 2.71876 1.1372e-05 \\ 2.72682 1.09887e-05 \\ 2.73036 1.08715e-05 \\ 2.73653 1.06968e-05 \\
2.74505 1.05318e-05 \\ 2.75175 1.04317e-05 \\ 2.76131 1.03549e-05 \\ 2.7656 1.02666e-05 \\ 2.77743 9.71912e-06 \\ 2.78461 9.55021e-06 \\
2.79388 9.2287e-06 \\ 2.79775 9.13352e-06 \\ 2.80512 8.9867e-06 \\ 2.81393 8.84879e-06 \\ 2.82 8.76478e-06 \\ 2.82964 8.70022e-06 \\
2.83354 8.62701e-06 \\ 2.84353 8.1698e-06 \\ 2.85029 8.02488e-06 \\ 2.85899 7.75536e-06 \\ 2.86222 7.67678e-06 \\ 2.86858 7.55364e-06 \\
2.87778 7.43839e-06 \\ 2.88309 7.36807e-06 \\ 2.89191 7.31401e-06 \\ 2.90256 6.94386e-06 \\ 2.90951 6.82802e-06 \\ 2.91257 6.76199e-06 \\
2.91941 6.51443e-06 \\ 2.92585 6.41307e-06 \\ 2.92898 6.36407e-06 \\ 2.93697 6.26745e-06 \\ 2.94337 6.1978e-06 \\ 2.95209 6.14728e-06 \\
2.9627 5.83322e-06 \\ 2.97034 5.73401e-06 \\ 2.97916 5.55387e-06 \\ 2.98311 5.48707e-06 \\ 2.99 5.40096e-06 \\ 2.99879 5.3185e-06 \\
3.00537 5.26996e-06 \\ 3.01573 5.03885e-06 \\ 3.02318 4.97005e-06 \\ 3.02733 4.91869e-06 \\ 3.03602 4.73518e-06 \\ 3.04313 4.66077e-06 \\
3.04666 4.62419e-06 \\ 3.05427 4.55244e-06 \\ 3.06048 4.50152e-06 \\ 3.06809 4.46468e-06 \\ 3.07828 4.2393e-06 \\ 3.08474 4.17092e-06 \\
3.09337 4.04134e-06 \\ 3.09659 3.99088e-06 \\ 3.10348 3.9285e-06 \\ 3.11191 3.86792e-06 \\ 3.11853 3.83262e-06 \\ 3.1216 3.80295e-06 \\
3.12929 3.75788e-06 \\ 3.13639 3.72701e-06 \\ 3.146 3.53245e-06 \\ 3.15348 3.4812e-06 \\ 3.16178 3.36806e-06 \\ 3.16512 3.32755e-06 \\
3.17246 3.27649e-06 \\ 3.18089 3.22705e-06 \\ 3.18809 3.19829e-06 \\ 3.19847 3.05955e-06 \\ 3.20518 3.01972e-06 \\ 3.2092 2.98805e-06 \\
3.21693 2.87651e-06 \\ 3.22455 2.83174e-06 \\ 3.22821 2.80961e-06 \\ 3.23642 2.76631e-06 \\ 3.24337 2.73581e-06 \\ 3.25265 2.7138e-06 \\
3.26282 2.57717e-06 \\ 3.2695 2.53656e-06 \\ 3.27836 2.4585e-06 \\ 3.28185 2.42741e-06 \\ 3.28823 2.39001e-06 \\ 3.29642 2.35334e-06 \\
3.30307 2.33217e-06 \\ 3.30657 2.3141e-06 \\ 3.31432 2.28683e-06 \\ 3.32154 2.26831e-06 \\ 3.33344 2.15077e-06 \\ 3.34108 2.11989e-06 \\
3.34971 2.05178e-06 \\ 3.35432 2.0268e-06 \\ 3.36146 1.9961e-06 \\ 3.36974 1.96613e-06 \\ 3.37655 1.94884e-06 \\ 3.38758 1.86504e-06 \\
3.39523 1.84093e-06 \\ 3.39844 1.82139e-06 \\ 3.40753 1.75382e-06 \\ 3.41519 1.72686e-06 \\ 3.4189 1.71327e-06 \\ 3.42768 1.68703e-06 \\
3.43388 1.66864e-06 \\ 3.44263 1.65541e-06 \\ 3.4535 1.57265e-06 \\ 3.46135 1.54804e-06 \\ 3.46488 1.5331e-06 \\ 3.47293 1.47804e-06 \\
3.47956 1.45551e-06 \\ 3.48842 1.44024e-06 \\ 3.49338 1.42745e-06 \\ 3.49908 1.41233e-06 \\ 3.50799 1.39667e-06 \\ 3.51867 1.32667e-06 \\
3.5254 1.30458e-06 \\ 3.53376 1.26478e-06 \\ 3.53781 1.24948e-06 \\ 3.54423 1.23057e-06 \\ 3.55242 1.21196e-06 \\ 3.55954 1.2012e-06 \\
3.57042 1.14985e-06 \\ 3.57749 1.13435e-06 \\ 3.58094 1.12245e-06 \\ 3.58982 1.08145e-06 \\ 3.59638 1.065e-06 \\ 3.60067 1.05658e-06 \\
3.61028 1.04045e-06 \\ 3.61678 1.02916e-06 \\ 3.62654 1.02105e-06 \\ 3.63732 9.70702e-07 \\ 3.64406 9.55258e-07 \\ 3.64782 9.46003e-07 \\
3.65648 9.12377e-07 \\ 3.66322 8.98623e-07 \\ 3.67162 8.89303e-07 \\ 3.67575 8.81346e-07 \\ 3.68162 8.72062e-07 \\ 3.69007 8.62414e-07 \\
3.70095 8.19669e-07 \\ 3.70736 8.05873e-07 \\ 3.71463 7.81657e-07 \\ 3.71836 7.72113e-07 \\ 3.7246 7.60534e-07 \\ 3.73302 7.4908e-07 \\
3.73958 7.42502e-07 \\ 3.75033 7.10935e-07 \\ 3.75694 7.01268e-07 \\ 3.76072 6.93837e-07 \\ 3.76875 6.68692e-07 \\ 3.77536 6.58639e-07 \\
3.77889 6.53379e-07 \\ 3.78689 6.43454e-07 \\ 3.79329 6.36532e-07 \\ 3.8016 6.3157e-07 \\ 3.81288 6.00712e-07 \\ 3.82034 5.91107e-07 \\
3.82416 5.85311e-07 \\ 3.83142 5.64662e-07 \\ 3.83719 5.56245e-07 \\ 3.84108 5.52028e-07 \\ 3.84915 5.43841e-07 \\ 3.85587 5.3804e-07 \\
3.86378 5.33873e-07 \\ 3.87418 5.07537e-07 \\ 3.88059 4.99114e-07 \\ 3.88892 4.84399e-07 \\ 3.89256 4.78222e-07 \\ 3.89844 4.71141e-07 \\
3.90593 4.64079e-07 \\ 3.91237 4.60081e-07 \\ 3.91568 4.56511e-07 \\ 3.9236 4.5124e-07 \\ 3.92975 4.4771e-07 \\ 3.93968 4.24757e-07 \\
3.94616 4.18471e-07 \\ 3.95424 4.05559e-07 \\ 3.95772 4.00533e-07 \\ 3.96348 3.94687e-07 \\ 3.9712 3.88876e-07 \\ 3.97829 3.85599e-07 \\
3.98822 3.69265e-07 \\ 3.99173 3.65018e-07 \\ 3.99823 3.598e-07 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.892289 \\ 0.00688712 0.85324 \\ 0.0136891 0.812681 \\ 0.0199933 0.770475 \\ 0.0261077 0.726512 \\ 0.0323998 0.681145 \\
0.0385642 0.635411 \\ 0.0477402 0.566084 \\ 0.0558909 0.502384 \\ 0.064093 0.446006 \\ 0.0744898 0.362573 \\ 0.0845359 0.296648 \\
0.0951073 0.24667 \\ 0.108733 0.157121 \\ 0.119554 0.133603 \\ 0.132151 0.0855638 \\ 0.14559 0.0613181 \\ 0.15921 0.0270768 \\
0.167067 0.0266708 \\ 0.176746 0.0245764 \\ 0.190156 0.0193805 \\ 0.203518 0.00375514 \\ 0.210432 0.00368098 \\ 0.219042 0.00357297 \\
0.229273 0.00335934 \\ 0.241444 0.00282382 \\ 0.254436 5.70838e-05 \\ 0.261698 5.29865e-05 \\ 0.270089 4.84e-05 \\ 0.279931 4.27866e-05 \\
0.292338 3.35564e-05 \\ 0.305607 1.00816e-08 \\ 0.312259 9.26111e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.892289 \\ 0.0139959 0.82317 \\ 0.0265816 0.759037 \\ 0.0392865 0.695303 \\ 0.0524117 0.629444 \\ 0.0650371 0.560673 \\
0.077461 0.489551 \\ 0.0895497 0.417474 \\ 0.101996 0.346436 \\ 0.114094 0.27886 \\ 0.126216 0.217275 \\ 0.138471 0.16391 \\
0.151115 0.119085 \\ 0.163475 0.0825614 \\ 0.175163 0.0541362 \\ 0.188523 0.0329191 \\ 0.202135 0.0182497 \\ 0.21453 0.00904316 \\
0.226495 0.00371144 \\ 0.238372 0.00107568 \\ 0.250852 0.000175038 \\ 0.264323 1.08363e-05 \\ 0.276787 1.29045e-07 \\ 0.288616 1.01554e-10 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.892289 \\ 0.00787263 0.842268 \\ 0.0150669 0.792419 \\ 0.0223075 0.741548 \\ 0.0294364 0.689009 \\ 0.0364162 0.634915 \\
0.043769 0.580158 \\ 0.0510798 0.526202 \\ 0.0586214 0.47473 \\ 0.0659071 0.427613 \\ 0.0774533 0.309416 \\ 0.0856454 0.277269 \\
0.0984259 0.180364 \\ 0.106964 0.163865 \\ 0.119165 0.0958351 \\ 0.126895 0.0898533 \\ 0.139548 0.0450625 \\ 0.147037 0.0431453 \\
0.160002 0.0171994 \\ 0.167781 0.0166538 \\ 0.179731 0.00467334 \\ 0.188008 0.0045521 \\ 0.199702 0.000567494 \\ 0.207242 0.00055295 \\
0.218906 1.49448e-05 \\ 0.226653 1.44797e-05 \\ 0.238127 2.79147e-08 \\ 0.245397 2.71205e-08 \\ 0.257856 6.00409e-13 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(f) $\gamma=0.5$, runtime};Figure 1. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) for the log-sum-exp objective with . The left column shows the objective gap versus iterations, while the right column shows the objective gap versus wall-clock time. Row-nested MLRN and TLRN methods demonstrate comparable convergence behavior to RN.
In this section, we present numerical experiments to support our theoretical findings. In particular, we compare the performance of the two-level regularized Newton (TLRN), multilevel regularized Newton (MLRN), and row-nested multilevel regularized Newton (MLRN nested) methods with that of the full-space gradient-regularized Newton method (RN) [13]. Although our methods retain the convergence rate of RN, the constants in their complexity bounds depend on , see lemma:case-II, potentially leading to less favorable theoretical bounds. Nevertheless, since the multilevel methods compute some of their updates in lower-dimensional spaces, they may offer significant computational savings. Our experiments aim to illustrate that the multilevel methods exhibit convergence behavior comparable to RN while potentially reducing the overall computational time.
The restriction operators for MLRN are constructed by selecting rows of the identity matrix as in [19], whereas for the row-nested method we use the construction described in rem:nested-orthonormal. We consider coarse levels with dimensions , , , , and , whereas for TLRN we select .
Since the Hessian’s Lipschitz constant is generally unknown, we employ an adaptive line-search procedure, following the approach of [7 Section 4.1.4.2] and [13]. At each iteration, the estimate is increased until the corresponding sufficient-decrease and gradient-stability conditions are satisfied. We set and terminate the algorithms when or , where and . We use L-BFGS to compute a numerical approximation of the optimal objective value . We set , balancing the frequency of coarse corrections against their effectiveness (note that larger values of yield more favorable convergence constants but impose a stricter coarse-level eligibility condition, potentially resulting in more fine-level iterations). All experiments are initialized at the origin.
Log-sum-exp
Given data and , we consider minimizing the log-sum-exp function
The parameter controls the smoothing of the log-sum-exp objective. We generate a random dataset with observations and variables and consider three values of this parameter, namely , to investigate the sensitivity of the algorithms to different degrees of smoothing.
The results are presented in Figure 1. For all three values of , row-nested MLRN and TLRN exhibit convergence behavior comparable to that of RN, both in terms of iterations and computational time. In contrast, the non-nested MLRN method converges substantially more slowly. We suspect that this difference may be related to the fine-iteration frequency condition in Theorem 2.5, which need not be satisfied during the observed iterations. Nevertheless, these findings suggest that row-nested restriction operators can be an attractive choice from both theoretical and computational perspectives.
\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=35,
ymin=2.968e-09, ymax=1.735e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$},
legend to name=leg:gisette-duke]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.341234 \\ 2 0.221978 \\ 3 0.156725 \\ 4 0.114743 \\ 5 0.0712025 \\
6 0.0448428 \\ 7 0.0312058 \\ 8 0.0182202 \\ 9 0.0121134 \\ 10 0.00679738 \\ 11 0.00437192 \\
12 0.00236882 \\ 13 0.00128142 \\ 14 0.000787644 \\ 15 0.000413953 \\ 16 0.000249913 \\ 17 0.000128266 \\
18 6.57309e-05 \\ 19 3.86296e-05 \\ 20 1.94499e-05 \\ 21 1.13001e-05 \\ 22 5.6e-06 \\ 23 2.77471e-06 \\
24 1.58442e-06 \\ 25 7.74963e-07 \\ 26 4.38665e-07 \\ 27 2.12032e-07 \\ 28 1.03326e-07 \\ 29 5.91656e-08 \\
30 3.07516e-08 \\ 31 1.83143e-08 \\ 32 1.36845e-08 \\ 33 1.0955e-08 \\ 34 9.77847e-09 \\
};
\addlegendentry{MLRN}
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.351924 \\ 2 0.22864 \\ 3 0.160374 \\ 4 0.106423 \\ 5 0.071223 \\
6 0.0469322 \\ 7 0.0299318 \\ 8 0.0184503 \\ 9 0.00978718 \\ 10 0.00582648 \\ 11 0.00304582 \\
12 0.00176339 \\ 13 0.000906382 \\ 14 0.000514131 \\ 15 0.000260414 \\ 16 0.000131736 \\ 17 7.33761e-05 \\
18 3.6682e-05 \\ 19 2.01809e-05 \\ 20 9.99131e-06 \\ 21 4.94788e-06 \\ 22 2.69538e-06 \\ 23 1.32472e-06 \\
24 6.52368e-07 \\ 25 3.54835e-07 \\ 26 1.76152e-07 \\ 27 9.00848e-08 \\ 28 4.92358e-08 \\ 29 3.04536e-08 \\
30 2.23469e-08 \\ 31 1.91902e-08 \\ 32 1.51284e-08 \\ 33 1.13527e-08 \\ 34 7.85223e-09 \\
};
\addlegendentry{MLRN nested}
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.291616 \\ 2 0.169193 \\ 3 0.104108 \\ 4 0.0639569 \\ 5 0.0379529 \\
6 0.0217411 \\ 7 0.0122087 \\ 8 0.00675044 \\ 9 0.00368603 \\ 10 0.00199245 \\ 11 0.00106787 \\
12 0.000568102 \\ 13 0.000300249 \\ 14 0.000157754 \\ 15 8.24476e-05 \\ 16 4.2884e-05 \\ 17 2.22081e-05 \\
18 1.14541e-05 \\ 19 5.88436e-06 \\ 20 3.01082e-06 \\ 21 1.53353e-06 \\ 22 7.76618e-07 \\ 23 3.90147e-07 \\
24 1.93615e-07 \\ 25 9.42246e-08 \\ 26 4.44096e-08 \\ 27 1.98449e-08 \\ 28 8.10823e-09 \\
};
\addlegendentry{RN}
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.323456 \\ 2 0.200904 \\ 3 0.134309 \\ 4 0.0921445 \\ 5 0.0629467 \\
6 0.0420071 \\ 7 0.0270752 \\ 8 0.0168813 \\ 9 0.00809604 \\ 10 0.00492521 \\ 11 0.00292718 \\
12 0.00139256 \\ 13 0.000815136 \\ 14 0.000385658 \\ 15 0.000223043 \\ 16 0.000126941 \\ 17 5.89582e-05 \\
18 3.33341e-05 \\ 19 1.54134e-05 \\ 20 8.66531e-06 \\ 21 3.98938e-06 \\ 22 2.23132e-06 \\ 23 1.23211e-06 \\
24 5.58869e-07 \\ 25 3.0866e-07 \\ 26 1.3972e-07 \\ 27 7.88348e-08 \\ 28 3.70373e-08 \\ 29 1.88081e-08 \\
30 1.09698e-08 \\ 31 7.42987e-09 \\
};
\addlegendentry{TLRN}
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(a) Gisette, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=30.6,
ymin=2.968e-09, ymax=1.735e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.289378 0.341234 \\ 0.557436 0.221978 \\ 0.821544 0.156725 \\ 1.08214 0.114743 \\ 1.58452 0.0712025 \\
2.04477 0.0448428 \\ 2.30896 0.0312058 \\ 2.77607 0.0182202 \\ 3.07122 0.0121134 \\ 3.53687 0.00679738 \\ 3.80257 0.00437192 \\
4.26728 0.00236882 \\ 4.7345 0.00128142 \\ 5.00719 0.000787644 \\ 5.46636 0.000413953 \\ 5.73065 0.000249913 \\ 6.21407 0.000128266 \\
6.67125 6.57309e-05 \\ 6.93631 3.86296e-05 \\ 7.39711 1.94499e-05 \\ 7.66253 1.13001e-05 \\ 8.12234 5.6e-06 \\ 8.58261 2.77471e-06 \\
8.86199 1.58442e-06 \\ 9.31813 7.74963e-07 \\ 9.58241 4.38665e-07 \\ 10.042 2.12032e-07 \\ 10.5024 1.03326e-07 \\ 10.7664 5.91656e-08 \\
11.2271 3.07516e-08 \\ 11.6939 1.83143e-08 \\ 11.7873 1.36845e-08 \\ 12.0539 1.0955e-08 \\ 12.5185 9.77847e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.267127 0.351924 \\ 0.529526 0.22864 \\ 0.790137 0.160374 \\ 1.24833 0.106423 \\ 1.71108 0.071223 \\
2.17185 0.0469322 \\ 2.63325 0.0299318 \\ 3.09394 0.0184503 \\ 3.82027 0.00978718 \\ 4.3024 0.00582648 \\ 5.00095 0.00304582 \\
5.46795 0.00176339 \\ 6.16167 0.000906382 \\ 6.62809 0.000514131 \\ 7.32517 0.000260414 \\ 8.02063 0.000131736 \\ 8.48979 7.33761e-05 \\
9.18555 3.6682e-05 \\ 9.65402 2.01809e-05 \\ 10.3486 9.99131e-06 \\ 11.0309 4.94788e-06 \\ 11.5176 2.69538e-06 \\ 12.234 1.32472e-06 \\
12.9144 6.52368e-07 \\ 13.4224 3.54835e-07 \\ 14.1379 1.76152e-07 \\ 14.8347 9.00848e-08 \\ 15.5462 4.92358e-08 \\ 16.2597 3.04536e-08 \\
16.946 2.23469e-08 \\ 17.6668 1.91902e-08 \\ 18.6288 1.51284e-08 \\ 19.5943 1.13527e-08 \\ 20.5477 7.85223e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1.08027 0.291616 \\ 2.21109 0.169193 \\ 3.29773 0.104108 \\ 4.33974 0.0639569 \\ 5.38297 0.0379529 \\
6.426 0.0217411 \\ 7.47015 0.0122087 \\ 8.5293 0.00675044 \\ 9.5925 0.00368603 \\ 10.6371 0.00199245 \\ 11.6978 0.00106787 \\
12.7447 0.000568102 \\ 13.7868 0.000300249 \\ 14.8456 0.000157754 \\ 15.9064 8.24476e-05 \\ 16.9845 4.2884e-05 \\ 18.0453 2.22081e-05 \\
19.1044 1.14541e-05 \\ 20.1633 5.88436e-06 \\ 21.223 3.01082e-06 \\ 22.287 1.53353e-06 \\ 23.3288 7.76618e-07 \\ 24.3738 3.90147e-07 \\
25.418 1.93615e-07 \\ 26.4628 9.42246e-08 \\ 27.5762 4.44096e-08 \\ 28.6168 1.98449e-08 \\ 29.6604 8.10823e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.472023 0.323456 \\ 0.940126 0.200904 \\ 1.40833 0.134309 \\ 1.87575 0.0921445 \\ 2.34369 0.0629467 \\
2.81126 0.0420071 \\ 3.27368 0.0270752 \\ 3.73204 0.0168813 \\ 4.78667 0.00809604 \\ 5.25437 0.00492521 \\ 5.71603 0.00292718 \\
6.76159 0.00139256 \\ 7.23062 0.000815136 \\ 8.32499 0.000385658 \\ 8.79135 0.000223043 \\ 9.25218 0.000126941 \\ 10.299 5.89582e-05 \\
10.7667 3.33341e-05 \\ 11.8089 1.54134e-05 \\ 12.2749 8.66531e-06 \\ 13.3293 3.98938e-06 \\ 13.805 2.23132e-06 \\ 14.3121 1.23211e-06 \\
15.3669 5.58869e-07 \\ 15.864 3.0866e-07 \\ 16.9224 1.3972e-07 \\ 17.389 7.88348e-08 \\ 18.4801 3.70373e-08 \\ 19.5553 1.88081e-08 \\
20.6 1.09698e-08 \\ 21.6455 7.42987e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(b) Gisette, runtime};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=21.6,
ymin=1.826e-09, ymax=1.776e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.15956 \\ 2 0.0581917 \\ 3 0.0227258 \\ 4 0.00912504 \\ 5 0.00372137 \\
6 0.0015348 \\ 7 0.000639153 \\ 8 0.000268621 \\ 9 0.000113924 \\ 10 4.87567e-05 \\ 11 2.10567e-05 \\
12 9.17541e-06 \\ 13 4.03382e-06 \\ 14 1.78946e-06 \\ 15 8.01012e-07 \\ 16 3.61722e-07 \\ 17 1.64745e-07 \\
18 7.56655e-08 \\ 19 3.50577e-08 \\ 20 1.64112e-08 \\ 21 7.7959e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.159741 \\ 2 0.0582728 \\ 3 0.0227713 \\ 4 0.00915124 \\ 5 0.0037363 \\
6 0.00154315 \\ 7 0.000643727 \\ 8 0.000271074 \\ 9 0.000115212 \\ 10 4.94217e-05 \\ 11 2.13962e-05 \\
12 9.34789e-06 \\ 13 4.12087e-06 \\ 14 1.83279e-06 \\ 15 8.22337e-07 \\ 16 3.72166e-07 \\ 17 1.69853e-07 \\
18 7.81665e-08 \\ 19 3.6286e-08 \\ 20 1.70185e-08 \\ 21 8.10038e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.141749 \\ 2 0.0504893 \\ 3 0.0192 \\ 4 0.00749097 \\ 5 0.00296374 \\
6 0.00118443 \\ 7 0.000477524 \\ 8 0.000194157 \\ 9 7.96153e-05 \\ 10 3.29305e-05 \\ 11 1.3742e-05 \\
12 5.78676e-06 \\ 13 2.45928e-06 \\ 14 1.05477e-06 \\ 15 4.56483e-07 \\ 16 1.99339e-07 \\ 17 8.78302e-08 \\
18 3.90267e-08 \\ 19 1.74686e-08 \\ 20 7.86086e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.150709 \\ 2 0.05382 \\ 3 0.0205865 \\ 4 0.0080899 \\ 5 0.00322643 \\
6 0.00130055 \\ 7 0.000529112 \\ 8 0.00021718 \\ 9 8.99358e-05 \\ 10 3.75784e-05 \\ 11 1.58451e-05 \\
12 6.74293e-06 \\ 13 2.8961e-06 \\ 14 1.25535e-06 \\ 15 5.49125e-07 \\ 16 2.42413e-07 \\ 17 1.07997e-07 \\
18 4.85483e-08 \\ 19 2.20212e-08 \\ 20 1.00858e-08 \\ 21 4.67671e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(c) Duke, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=16.1,
ymin=1.826e-09, ymax=1.776e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.107316 0.15956 \\ 0.189849 0.0581917 \\ 0.269627 0.0227258 \\ 0.348134 0.00912504 \\ 0.426704 0.00372137 \\
0.510825 0.0015348 \\ 0.587515 0.000639153 \\ 0.664879 0.000268621 \\ 0.7443 0.000113924 \\ 0.823644 4.87567e-05 \\ 0.902933 2.10567e-05 \\
0.984368 9.17541e-06 \\ 1.0645 4.03382e-06 \\ 1.14271 1.78946e-06 \\ 1.22532 8.01012e-07 \\ 1.30544 3.61722e-07 \\ 1.382 1.64745e-07 \\
1.4615 7.56655e-08 \\ 1.53983 3.50577e-08 \\ 1.61899 1.64112e-08 \\ 1.69816 7.7959e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.075281 0.159741 \\ 0.161511 0.0582728 \\ 0.241713 0.0227713 \\ 0.324653 0.00915124 \\ 0.407831 0.0037363 \\
0.489157 0.00154315 \\ 0.572311 0.000643727 \\ 0.653658 0.000271074 \\ 0.732206 0.000115212 \\ 0.810311 4.94217e-05 \\ 0.895539 2.13962e-05 \\
0.974757 9.34789e-06 \\ 1.05202 4.12087e-06 \\ 1.13764 1.83279e-06 \\ 1.21614 8.22337e-07 \\ 1.29337 3.72166e-07 \\ 1.37782 1.69853e-07 \\
1.45812 7.81665e-08 \\ 1.54099 3.6286e-08 \\ 1.62539 1.70185e-08 \\ 1.707 8.10038e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.741092 0.141749 \\ 2.16882 0.0504893 \\ 2.9132 0.0192 \\ 3.66182 0.00749097 \\ 4.40453 0.00296374 \\
5.14713 0.00118443 \\ 5.89295 0.000477524 \\ 6.63739 0.000194157 \\ 7.38009 7.96153e-05 \\ 8.12174 3.29305e-05 \\ 8.86554 1.3742e-05 \\
9.61025 5.78676e-06 \\ 10.3585 2.45928e-06 \\ 11.1026 1.05477e-06 \\ 11.8472 4.56483e-07 \\ 12.5905 1.99339e-07 \\ 13.3354 8.78302e-08 \\
14.1053 3.90267e-08 \\ 14.8898 1.74686e-08 \\ 15.6721 7.86086e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.259394 0.150709 \\ 0.463393 0.05382 \\ 0.651306 0.0205865 \\ 0.830558 0.0080899 \\ 1.01122 0.00322643 \\
1.19159 0.00130055 \\ 1.36947 0.000529112 \\ 1.55353 0.00021718 \\ 1.73696 8.99358e-05 \\ 1.96636 3.75784e-05 \\ 2.18405 1.58451e-05 \\
2.38181 6.74293e-06 \\ 2.57252 2.8961e-06 \\ 2.75113 1.25535e-06 \\ 2.93384 5.49125e-07 \\ 3.11331 2.42413e-07 \\ 3.29476 1.07997e-07 \\
3.47367 4.85483e-08 \\ 3.65467 2.20212e-08 \\ 3.83513 1.00858e-08 \\ 4.01907 4.67671e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(d) Duke, runtime};Figure 2. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) using the large Gisette and Duke datasets for logistic regression. The left column reports the objective gap versus iterations, while the right column reports the objective gap versus wall-clock time. The multilevel gradient-regularized Newton methods offer substantial computational savings compared to their full-space counterpart.
\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=308,
ymin=1.533e-09, ymax=9.290e-01,
ytickten={-7,-5,-3,-1},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$},
legend to name=leg:a9a-mushrooms-w8t]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.370526 \\ 1 0.220844 \\ 2 0.160654 \\ 3 0.125031 \\ 4 0.0785861 \\ 5 0.0624869 \\
6 0.0435495 \\ 7 0.0311968 \\ 8 0.022553 \\ 9 0.0169181 \\ 10 0.0126976 \\ 11 0.00835148 \\
12 0.00638134 \\ 13 0.00441419 \\ 14 0.00367332 \\ 15 0.00288761 \\ 16 0.00224696 \\ 17 0.00181493 \\
18 0.00154746 \\ 19 0.00135312 \\ 20 0.00123353 \\ 21 0.00113809 \\ 22 0.00106878 \\ 23 0.00101229 \\
24 0.000967882 \\ 25 0.000930739 \\ 26 0.00090097 \\ 27 0.000875672 \\ 28 0.000855072 \\ 29 0.000837271 \\
30 0.000822621 \\ 31 0.000809794 \\ 32 0.00079896 \\ 33 0.000789373 \\ 34 0.000781162 \\ 35 0.000773838 \\
36 0.000767419 \\ 37 0.000761655 \\ 38 0.000203549 \\ 39 0.000199489 \\ 40 0.000191282 \\ 41 0.000153031 \\
42 0.000149533 \\ 43 0.000144525 \\ 44 0.000140593 \\ 45 0.00013727 \\ 46 0.000134464 \\ 47 0.000132051 \\
48 0.00012569 \\ 49 0.00012358 \\ 50 0.000120437 \\ 51 0.000117863 \\ 52 0.000115635 \\ 53 0.000113742 \\
54 0.000112086 \\ 55 0.000109894 \\ 56 0.000108417 \\ 57 0.000106168 \\ 58 0.000104288 \\ 59 0.000102641 \\
60 0.00010123 \\ 61 9.99924e-05 \\ 62 9.84197e-05 \\ 63 9.72928e-05 \\ 64 9.55848e-05 \\ 65 9.41379e-05 \\
66 9.28675e-05 \\ 67 9.17764e-05 \\ 68 9.08155e-05 \\ 69 8.99872e-05 \\ 70 8.92585e-05 \\ 71 8.75101e-05 \\
72 8.67453e-05 \\ 73 8.55176e-05 \\ 74 8.44727e-05 \\ 75 8.3555e-05 \\ 76 8.27647e-05 \\ 77 8.20699e-05 \\
78 8.14691e-05 \\ 79 8.09405e-05 \\ 80 7.95864e-05 \\ 81 7.90162e-05 \\ 82 7.81046e-05 \\ 83 7.73263e-05 \\
84 7.66434e-05 \\ 85 7.60537e-05 \\ 86 7.55357e-05 \\ 87 7.50876e-05 \\ 88 7.46935e-05 \\ 89 7.36761e-05 \\
90 7.32463e-05 \\ 91 7.25652e-05 \\ 92 7.19824e-05 \\ 93 7.14714e-05 \\ 94 7.10298e-05 \\ 95 7.0642e-05 \\
96 7.03062e-05 \\ 97 7.00114e-05 \\ 98 6.92491e-05 \\ 99 6.89254e-05 \\ 100 6.84156e-05 \\ 101 6.79791e-05 \\
102 6.75966e-05 \\ 103 6.72658e-05 \\ 104 6.69757e-05 \\ 105 6.6724e-05 \\ 106 6.65032e-05 \\ 107 6.63115e-05 \\
108 6.61433e-05 \\ 109 6.54004e-05 \\ 110 6.51775e-05 \\ 111 6.4822e-05 \\ 112 6.45189e-05 \\ 113 6.42538e-05 \\
114 6.40241e-05 \\ 115 6.38229e-05 \\ 116 6.36482e-05 \\ 117 6.3495e-05 \\ 118 6.33619e-05 \\ 119 6.32452e-05 \\
120 6.27019e-05 \\ 121 6.25434e-05 \\ 122 6.22916e-05 \\ 123 6.20771e-05 \\ 124 6.18896e-05 \\ 125 6.1727e-05 \\
126 6.15847e-05 \\ 127 6.1461e-05 \\ 128 6.13527e-05 \\ 129 6.12585e-05 \\ 130 6.11759e-05 \\ 131 2.0168e-05 \\
132 1.98087e-05 \\ 133 1.95027e-05 \\ 134 1.92555e-05 \\ 135 1.90977e-05 \\ 136 1.8967e-05 \\ 137 1.88537e-05 \\
138 1.86977e-05 \\ 139 1.85904e-05 \\ 140 1.84314e-05 \\ 141 1.82968e-05 \\ 142 1.81792e-05 \\ 143 1.80771e-05 \\
144 1.79877e-05 \\ 145 1.791e-05 \\ 146 1.7842e-05 \\ 147 1.76785e-05 \\ 148 1.76063e-05 \\ 149 1.74925e-05 \\
150 1.73952e-05 \\ 151 1.73102e-05 \\ 152 1.72364e-05 \\ 153 1.71719e-05 \\ 154 1.71158e-05 \\ 155 1.70667e-05 \\
156 1.69418e-05 \\ 157 1.68887e-05 \\ 158 1.6805e-05 \\ 159 1.67331e-05 \\ 160 1.66703e-05 \\ 161 1.66158e-05 \\
162 1.65682e-05 \\ 163 1.65267e-05 \\ 164 1.64905e-05 \\ 165 1.63976e-05 \\ 166 1.63581e-05 \\ 167 1.62961e-05 \\
168 1.62428e-05 \\ 169 1.61963e-05 \\ 170 1.61559e-05 \\ 171 1.61206e-05 \\ 172 1.60898e-05 \\ 173 1.6063e-05 \\
174 1.5994e-05 \\ 175 1.59645e-05 \\ 176 1.59185e-05 \\ 177 1.5879e-05 \\ 178 1.58445e-05 \\ 179 1.58145e-05 \\
180 1.57883e-05 \\ 181 1.57654e-05 \\ 182 1.57455e-05 \\ 183 1.56943e-05 \\ 184 1.56723e-05 \\ 185 1.56382e-05 \\
186 1.56088e-05 \\ 187 1.55832e-05 \\ 188 1.55609e-05 \\ 189 1.55415e-05 \\ 190 1.55245e-05 \\ 191 1.55097e-05 \\
192 1.54968e-05 \\ 193 1.54855e-05 \\ 194 1.5436e-05 \\ 195 1.54211e-05 \\ 196 1.53975e-05 \\ 197 1.53774e-05 \\
198 1.53598e-05 \\ 199 1.53445e-05 \\ 200 1.53311e-05 \\ 201 1.53195e-05 \\ 202 1.53093e-05 \\ 203 1.53004e-05 \\
204 1.52927e-05 \\ 205 5.85371e-06 \\ 206 5.81938e-06 \\ 207 5.78911e-06 \\ 208 5.76124e-06 \\ 209 5.74598e-06 \\
210 5.73318e-06 \\ 211 5.72214e-06 \\ 212 5.71258e-06 \\ 213 5.70425e-06 \\ 214 5.68388e-06 \\ 215 5.67499e-06 \\
216 5.66107e-06 \\ 217 5.64939e-06 \\ 218 5.63922e-06 \\ 219 5.63036e-06 \\ 220 5.62263e-06 \\ 221 5.6159e-06 \\
222 5.61002e-06 \\ 223 5.59503e-06 \\ 224 5.58869e-06 \\ 225 5.5787e-06 \\ 226 5.57016e-06 \\ 227 5.56272e-06 \\
228 5.55624e-06 \\ 229 5.55058e-06 \\ 230 5.54566e-06 \\ 231 5.54136e-06 \\ 232 5.53037e-06 \\ 233 5.52571e-06 \\
234 5.51838e-06 \\ 235 5.51209e-06 \\ 236 5.5066e-06 \\ 237 5.50183e-06 \\ 238 5.49766e-06 \\ 239 5.49403e-06 \\
240 5.49086e-06 \\ 241 5.48275e-06 \\ 242 5.4793e-06 \\ 243 5.4739e-06 \\ 244 5.46924e-06 \\ 245 5.46519e-06 \\
246 5.46166e-06 \\ 247 5.45858e-06 \\ 248 5.45589e-06 \\ 249 5.45355e-06 \\ 250 5.45151e-06 \\ 251 5.44972e-06 \\
252 5.44192e-06 \\ 253 5.43958e-06 \\ 254 5.43587e-06 \\ 255 5.43268e-06 \\ 256 5.42991e-06 \\ 257 5.4275e-06 \\
258 5.42539e-06 \\ 259 5.42355e-06 \\ 260 5.42195e-06 \\ 261 5.42055e-06 \\ 262 5.41933e-06 \\ 263 5.41367e-06 \\
264 5.41202e-06 \\ 265 5.40942e-06 \\ 266 5.40719e-06 \\ 267 5.40525e-06 \\ 268 5.40355e-06 \\ 269 5.40208e-06 \\
270 5.40079e-06 \\ 271 5.39967e-06 \\ 272 5.39869e-06 \\ 273 5.39783e-06 \\ 274 2.15022e-06 \\ 275 2.14635e-06 \\
276 2.14316e-06 \\ 277 2.13876e-06 \\ 278 2.13706e-06 \\ 279 2.13563e-06 \\ 280 2.13439e-06 \\ 281 2.13331e-06 \\
282 2.13237e-06 \\ 283 2.13016e-06 \\ 284 2.12917e-06 \\ 285 2.12763e-06 \\ 286 2.12634e-06 \\ 287 2.12523e-06 \\
288 2.12425e-06 \\ 289 2.1234e-06 \\ 290 2.12266e-06 \\ 291 2.12201e-06 \\ 292 2.12038e-06 \\ 293 2.11968e-06 \\
294 2.11859e-06 \\ 295 2.11766e-06 \\ 296 2.11684e-06 \\ 297 2.11614e-06 \\ 298 2.11552e-06 \\ 299 2.11498e-06 \\
};
\addlegendentry{MLRN}
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.370526 \\ 1 0.245464 \\ 2 0.17626 \\ 3 0.119014 \\ 4 0.0824586 \\ 5 0.0592795 \\
6 0.0304271 \\ 7 0.0215966 \\ 8 0.011448 \\ 9 0.00775859 \\ 10 0.00287514 \\ 11 0.00214857 \\
12 0.000998239 \\ 13 0.000846347 \\ 14 0.000648285 \\ 15 0.00030038 \\ 16 0.000113784 \\ 17 0.000101073 \\
18 6.18168e-05 \\ 19 2.83696e-05 \\ 20 1.57979e-05 \\ 21 9.08908e-06 \\ 22 5.99197e-06 \\ 23 3.06418e-06 \\
24 1.72343e-06 \\ 25 9.7451e-07 \\ 26 6.72319e-07 \\ 27 3.19381e-07 \\ 28 1.5412e-07 \\ 29 8.15245e-08 \\
30 4.38879e-08 \\ 31 2.93518e-08 \\ 32 1.05093e-08 \\ 33 3.84338e-09 \\
};
\addlegendentry{MLRN nested}
\addplot[rn] table[row sep=\\] {
x y \\
0 0.370526 \\ 1 0.194305 \\ 2 0.119113 \\ 3 0.0713522 \\ 4 0.0423861 \\ 5 0.0244413 \\
6 0.0133646 \\ 7 0.00694675 \\ 8 0.00356043 \\ 9 0.00196368 \\ 10 0.00123244 \\ 11 0.000808864 \\
12 0.000502406 \\ 13 0.000285909 \\ 14 0.000154609 \\ 15 8.66915e-05 \\ 16 5.17797e-05 \\ 17 3.15256e-05 \\
18 1.90743e-05 \\ 19 1.14358e-05 \\ 20 6.80782e-06 \\ 21 4.02e-06 \\ 22 2.3503e-06 \\ 23 1.35805e-06 \\
24 7.73439e-07 \\ 25 4.3267e-07 \\ 26 2.36667e-07 \\ 27 1.25711e-07 \\ 28 6.41038e-08 \\ 29 3.07788e-08 \\
30 1.34621e-08 \\ 31 5.06505e-09 \\
};
\addlegendentry{RN}
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.370526 \\ 1 0.245464 \\ 2 0.17626 \\ 3 0.0999392 \\ 4 0.0732158 \\ 5 0.0387032 \\
6 0.0279254 \\ 7 0.012742 \\ 8 0.00918266 \\ 9 0.00391495 \\ 10 0.00196513 \\ 11 0.00165493 \\
12 0.000938637 \\ 13 0.000807706 \\ 14 0.000374031 \\ 15 0.000315199 \\ 16 0.000111387 \\ 17 9.04894e-05 \\
18 3.8996e-05 \\ 19 3.0339e-05 \\ 20 1.52633e-05 \\ 21 1.20985e-05 \\ 22 5.95702e-06 \\ 23 4.848e-06 \\
24 2.48231e-06 \\ 25 2.0914e-06 \\ 26 1.09324e-06 \\ 27 9.40484e-07 \\ 28 4.94303e-07 \\ 29 4.30823e-07 \\
30 2.24572e-07 \\ 31 1.97364e-07 \\ 32 1.00573e-07 \\ 33 5.211e-08 \\ 34 2.50597e-08 \\ 35 1.96207e-08 \\
36 8.06526e-09 \\
};
\addlegendentry{TLRN}
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(a) a9a, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=3.08,
ymin=1.533e-09, ymax=9.290e-01,
ytickten={-7,-5,-3,-1},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.370526 \\ 0.0115412 0.220844 \\ 0.0203368 0.160654 \\ 0.0280939 0.125031 \\ 0.0391305 0.0785861 \\ 0.0467821 0.0624869 \\
0.0572532 0.0435495 \\ 0.0633852 0.0311968 \\ 0.0708939 0.022553 \\ 0.0806933 0.0169181 \\ 0.0880545 0.0126976 \\ 0.0939562 0.00835148 \\
0.103897 0.00638134 \\ 0.111321 0.00441419 \\ 0.121158 0.00367332 \\ 0.128294 0.00288761 \\ 0.134309 0.00224696 \\ 0.145315 0.00181493 \\
0.153767 0.00154746 \\ 0.16459 0.00135312 \\ 0.172979 0.00123353 \\ 0.184575 0.00113809 \\ 0.192661 0.00106878 \\ 0.204133 0.00101229 \\
0.212091 0.000967882 \\ 0.223273 0.000930739 \\ 0.231644 0.00090097 \\ 0.242634 0.000875672 \\ 0.250447 0.000855072 \\ 0.260493 0.000837271 \\
0.269641 0.000822621 \\ 0.279376 0.000809794 \\ 0.288388 0.00079896 \\ 0.29804 0.000789373 \\ 0.308625 0.000781162 \\ 0.318436 0.000773838 \\
0.32734 0.000767419 \\ 0.336857 0.000761655 \\ 0.371336 0.000203549 \\ 0.379466 0.000199489 \\ 0.390746 0.000191282 \\ 0.397143 0.000153031 \\
0.405942 0.000149533 \\ 0.417094 0.000144525 \\ 0.425363 0.000140593 \\ 0.436532 0.00013727 \\ 0.444116 0.000134464 \\ 0.45491 0.000132051 \\
0.462495 0.00012569 \\ 0.470732 0.00012358 \\ 0.483371 0.000120437 \\ 0.492945 0.000117863 \\ 0.504715 0.000115635 \\ 0.513685 0.000113742 \\
0.523827 0.000112086 \\ 0.529581 0.000109894 \\ 0.542375 0.000108417 \\ 0.555713 0.000106168 \\ 0.566069 0.000104288 \\ 0.577076 0.000102641 \\
0.587585 0.00010123 \\ 0.600792 9.99924e-05 \\ 0.607428 9.84197e-05 \\ 0.619449 9.72928e-05 \\ 0.633984 9.55848e-05 \\ 0.642125 9.41379e-05 \\
0.654812 9.28675e-05 \\ 0.664456 9.17764e-05 \\ 0.67485 9.08155e-05 \\ 0.683617 8.99872e-05 \\ 0.695188 8.92585e-05 \\ 0.702361 8.75101e-05 \\
0.711421 8.67453e-05 \\ 0.724146 8.55176e-05 \\ 0.731586 8.44727e-05 \\ 0.74316 8.3555e-05 \\ 0.753078 8.27647e-05 \\ 0.766111 8.20699e-05 \\
0.774497 8.14691e-05 \\ 0.787869 8.09405e-05 \\ 0.798339 7.95864e-05 \\ 0.80856 7.90162e-05 \\ 0.821589 7.81046e-05 \\ 0.831779 7.73263e-05 \\
0.844835 7.66434e-05 \\ 0.853338 7.60537e-05 \\ 0.865363 7.55357e-05 \\ 0.874571 7.50876e-05 \\ 0.884563 7.46935e-05 \\ 0.893233 7.36761e-05 \\
0.901909 7.32463e-05 \\ 0.915008 7.25652e-05 \\ 0.922425 7.19824e-05 \\ 0.933905 7.14714e-05 \\ 0.943073 7.10298e-05 \\ 0.953132 7.0642e-05 \\
0.962744 7.03062e-05 \\ 0.975425 7.00114e-05 \\ 0.986306 6.92491e-05 \\ 0.994456 6.89254e-05 \\ 1.00931 6.84156e-05 \\ 1.01782 6.79791e-05 \\
1.03075 6.75966e-05 \\ 1.04109 6.72658e-05 \\ 1.05444 6.69757e-05 \\ 1.06234 6.6724e-05 \\ 1.07456 6.65032e-05 \\ 1.08445 6.63115e-05 \\
1.09594 6.61433e-05 \\ 1.10616 6.54004e-05 \\ 1.11376 6.51775e-05 \\ 1.12544 6.4822e-05 \\ 1.13469 6.45189e-05 \\ 1.14614 6.42538e-05 \\
1.15362 6.40241e-05 \\ 1.1652 6.38229e-05 \\ 1.18122 6.36482e-05 \\ 1.19417 6.3495e-05 \\ 1.20486 6.33619e-05 \\ 1.21789 6.32452e-05 \\
1.23007 6.27019e-05 \\ 1.23858 6.25434e-05 \\ 1.25201 6.22916e-05 \\ 1.262 6.20771e-05 \\ 1.27482 6.18896e-05 \\ 1.28289 6.1727e-05 \\
1.2951 6.15847e-05 \\ 1.30484 6.1461e-05 \\ 1.31634 6.13527e-05 \\ 1.32375 6.12585e-05 \\ 1.33523 6.11759e-05 \\ 1.3762 2.0168e-05 \\
1.38223 1.98087e-05 \\ 1.3931 1.95027e-05 \\ 1.40152 1.92555e-05 \\ 1.41274 1.90977e-05 \\ 1.42126 1.8967e-05 \\ 1.43235 1.88537e-05 \\
1.43879 1.86977e-05 \\ 1.44691 1.85904e-05 \\ 1.4585 1.84314e-05 \\ 1.46712 1.82968e-05 \\ 1.47768 1.81792e-05 \\ 1.48572 1.80771e-05 \\
1.49577 1.79877e-05 \\ 1.50342 1.791e-05 \\ 1.51313 1.7842e-05 \\ 1.51875 1.76785e-05 \\ 1.5287 1.76063e-05 \\ 1.54325 1.74925e-05 \\
1.55156 1.73952e-05 \\ 1.5634 1.73102e-05 \\ 1.57256 1.72364e-05 \\ 1.58287 1.71719e-05 \\ 1.59327 1.71158e-05 \\ 1.60623 1.70667e-05 \\
1.61709 1.69418e-05 \\ 1.62562 1.68887e-05 \\ 1.64066 1.6805e-05 \\ 1.64891 1.67331e-05 \\ 1.66202 1.66703e-05 \\ 1.67286 1.66158e-05 \\
1.68615 1.65682e-05 \\ 1.69365 1.65267e-05 \\ 1.70397 1.64905e-05 \\ 1.71109 1.63976e-05 \\ 1.72004 1.63581e-05 \\ 1.73152 1.62961e-05 \\
1.74058 1.62428e-05 \\ 1.75211 1.61963e-05 \\ 1.75949 1.61559e-05 \\ 1.76937 1.61206e-05 \\ 1.77677 1.60898e-05 \\ 1.78671 1.6063e-05 \\
1.79251 1.5994e-05 \\ 1.80082 1.59645e-05 \\ 1.81368 1.59185e-05 \\ 1.82214 1.5879e-05 \\ 1.83336 1.58445e-05 \\ 1.8417 1.58145e-05 \\
1.85302 1.57883e-05 \\ 1.86118 1.57654e-05 \\ 1.87268 1.57455e-05 \\ 1.87913 1.56943e-05 \\ 1.88766 1.56723e-05 \\ 1.89811 1.56382e-05 \\
1.90619 1.56088e-05 \\ 1.91671 1.55832e-05 \\ 1.92431 1.55609e-05 \\ 1.93443 1.55415e-05 \\ 1.94178 1.55245e-05 \\ 1.95178 1.55097e-05 \\
1.9593 1.54968e-05 \\ 1.96919 1.54855e-05 \\ 1.97476 1.5436e-05 \\ 1.98207 1.54211e-05 \\ 1.99224 1.53975e-05 \\ 2.00023 1.53774e-05 \\
2.01159 1.53598e-05 \\ 2.02051 1.53445e-05 \\ 2.0314 1.53311e-05 \\ 2.03983 1.53195e-05 \\ 2.05103 1.53093e-05 \\ 2.05958 1.53004e-05 \\
2.0709 1.52927e-05 \\ 2.11563 5.85371e-06 \\ 2.12156 5.81938e-06 \\ 2.13169 5.78911e-06 \\ 2.13923 5.76124e-06 \\ 2.14936 5.74598e-06 \\
2.15672 5.73318e-06 \\ 2.16646 5.72214e-06 \\ 2.1738 5.71258e-06 \\ 2.18379 5.70425e-06 \\ 2.18941 5.68388e-06 \\ 2.19706 5.67499e-06 \\
2.20808 5.66107e-06 \\ 2.21654 5.64939e-06 \\ 2.22768 5.63922e-06 \\ 2.2363 5.63036e-06 \\ 2.24725 5.62263e-06 \\ 2.25593 5.6159e-06 \\
2.26714 5.61002e-06 \\ 2.27372 5.59503e-06 \\ 2.28227 5.58869e-06 \\ 2.29377 5.5787e-06 \\ 2.30222 5.57016e-06 \\ 2.31259 5.56272e-06 \\
2.32089 5.55624e-06 \\ 2.33082 5.55058e-06 \\ 2.33838 5.54566e-06 \\ 2.34831 5.54136e-06 \\ 2.35396 5.53037e-06 \\ 2.36155 5.52571e-06 \\
2.37164 5.51838e-06 \\ 2.37904 5.51209e-06 \\ 2.38892 5.5066e-06 \\ 2.39636 5.50183e-06 \\ 2.40716 5.49766e-06 \\ 2.4158 5.49403e-06 \\
2.42684 5.49086e-06 \\ 2.43351 5.48275e-06 \\ 2.44215 5.4793e-06 \\ 2.45369 5.4739e-06 \\ 2.46195 5.46924e-06 \\ 2.47339 5.46519e-06 \\
2.4815 5.46166e-06 \\ 2.49257 5.45858e-06 \\ 2.50116 5.45589e-06 \\ 2.51144 5.45355e-06 \\ 2.5196 5.45151e-06 \\ 2.53011 5.44972e-06 \\
2.53609 5.44192e-06 \\ 2.5437 5.43958e-06 \\ 2.55433 5.43587e-06 \\ 2.56185 5.43268e-06 \\ 2.57171 5.42991e-06 \\ 2.57951 5.4275e-06 \\
2.59055 5.42539e-06 \\ 2.59809 5.42355e-06 \\ 2.60827 5.42195e-06 \\ 2.616 5.42055e-06 \\ 2.6271 5.41933e-06 \\ 2.63372 5.41367e-06 \\
2.64215 5.41202e-06 \\ 2.65363 5.40942e-06 \\ 2.66164 5.40719e-06 \\ 2.67314 5.40525e-06 \\ 2.68147 5.40355e-06 \\ 2.69277 5.40208e-06 \\
2.70153 5.40079e-06 \\ 2.71246 5.39967e-06 \\ 2.72098 5.39869e-06 \\ 2.73141 5.39783e-06 \\ 2.77068 2.15022e-06 \\ 2.77614 2.14635e-06 \\
2.78603 2.14316e-06 \\ 2.79359 2.13876e-06 \\ 2.80383 2.13706e-06 \\ 2.8113 2.13563e-06 \\ 2.82149 2.13439e-06 \\ 2.82946 2.13331e-06 \\
2.84053 2.13237e-06 \\ 2.84679 2.13016e-06 \\ 2.85566 2.12917e-06 \\ 2.86716 2.12763e-06 \\ 2.87557 2.12634e-06 \\ 2.88695 2.12523e-06 \\
2.89515 2.12425e-06 \\ 2.9065 2.1234e-06 \\ 2.91465 2.12266e-06 \\ 2.92549 2.12201e-06 \\ 2.93184 2.12038e-06 \\ 2.9399 2.11968e-06 \\
2.95054 2.11859e-06 \\ 2.95802 2.11766e-06 \\ 2.96793 2.11684e-06 \\ 2.97513 2.11614e-06 \\ 2.9851 2.11552e-06 \\ 2.99259 2.11498e-06 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.370526 \\ 0.0115394 0.245464 \\ 0.0215122 0.17626 \\ 0.0351462 0.119014 \\ 0.0494607 0.0824586 \\ 0.063812 0.0592795 \\
0.0801981 0.0304271 \\ 0.0941092 0.0215966 \\ 0.10991 0.011448 \\ 0.126056 0.00775859 \\ 0.142686 0.00287514 \\ 0.157922 0.00214857 \\
0.173969 0.000998239 \\ 0.183963 0.000846347 \\ 0.195857 0.000648285 \\ 0.209516 0.00030038 \\ 0.225594 0.000113784 \\ 0.233608 0.000101073 \\
0.24678 6.18168e-05 \\ 0.263213 2.83696e-05 \\ 0.279646 1.57979e-05 \\ 0.297047 9.08908e-06 \\ 0.308514 5.99197e-06 \\ 0.32206 3.06418e-06 \\
0.338552 1.72343e-06 \\ 0.355321 9.7451e-07 \\ 0.365918 6.72319e-07 \\ 0.379003 3.19381e-07 \\ 0.390952 1.5412e-07 \\ 0.406135 8.15245e-08 \\
0.420678 4.38879e-08 \\ 0.4305 2.93518e-08 \\ 0.443248 1.05093e-08 \\ 0.458801 3.84338e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.370526 \\ 0.0178052 0.194305 \\ 0.0340228 0.119113 \\ 0.0484172 0.0713522 \\ 0.0627713 0.0423861 \\ 0.0780268 0.0244413 \\
0.0923394 0.0133646 \\ 0.108325 0.00694675 \\ 0.123613 0.00356043 \\ 0.13894 0.00196368 \\ 0.15426 0.00123244 \\ 0.168464 0.000808864 \\
0.183726 0.000502406 \\ 0.199015 0.000285909 \\ 0.214142 0.000154609 \\ 0.231832 8.66915e-05 \\ 0.249281 5.17797e-05 \\ 0.266276 3.15256e-05 \\
0.283534 1.90743e-05 \\ 0.299903 1.14358e-05 \\ 0.315326 6.80782e-06 \\ 0.330217 4.02e-06 \\ 0.345712 2.3503e-06 \\ 0.361232 1.35805e-06 \\
0.376598 7.73439e-07 \\ 0.392102 4.3267e-07 \\ 0.408409 2.36667e-07 \\ 0.425438 1.25711e-07 \\ 0.442824 6.41038e-08 \\ 0.45921 3.07788e-08 \\
0.476792 1.34621e-08 \\ 0.493983 5.06505e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.370526 \\ 0.012615 0.245464 \\ 0.0239598 0.17626 \\ 0.0404104 0.0999392 \\ 0.0512433 0.0732158 \\ 0.0686374 0.0387032 \\
0.0801329 0.0279254 \\ 0.0975579 0.012742 \\ 0.108253 0.00918266 \\ 0.123155 0.00391495 \\ 0.138853 0.00196513 \\ 0.148898 0.00165493 \\
0.163416 0.000938637 \\ 0.174368 0.000807706 \\ 0.190745 0.000374031 \\ 0.200681 0.000315199 \\ 0.218082 0.000111387 \\ 0.229409 9.04894e-05 \\
0.245702 3.8996e-05 \\ 0.256848 3.0339e-05 \\ 0.27359 1.52633e-05 \\ 0.284598 1.20985e-05 \\ 0.301452 5.95702e-06 \\ 0.312522 4.848e-06 \\
0.330713 2.48231e-06 \\ 0.340767 2.0914e-06 \\ 0.357419 1.09324e-06 \\ 0.367262 9.40484e-07 \\ 0.384914 4.94303e-07 \\ 0.394927 4.30823e-07 \\
0.412141 2.24572e-07 \\ 0.422151 1.97364e-07 \\ 0.441895 1.00573e-07 \\ 0.45949 5.211e-08 \\ 0.476626 2.50597e-08 \\ 0.487659 1.96207e-08 \\
0.506292 8.06526e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(b) a9a, runtime};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=85.5,
ymin=2.237e-09, ymax=1.759e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.545341 \\ 2 0.43393 \\ 3 0.335126 \\ 4 0.269546 \\ 5 0.172444 \\
6 0.118233 \\ 7 0.0833317 \\ 8 0.0677165 \\ 9 0.0545676 \\ 10 0.0416062 \\ 11 0.0337825 \\
12 0.0185656 \\ 13 0.0132214 \\ 14 0.0105894 \\ 15 0.00719845 \\ 16 0.00587158 \\ 17 0.00382227 \\
18 0.00315565 \\ 19 0.00198005 \\ 20 0.00164922 \\ 21 0.00100297 \\ 22 0.000854502 \\ 23 0.000554198 \\
24 0.000470355 \\ 25 0.000307366 \\ 26 0.000260156 \\ 27 0.000170825 \\ 28 0.000144181 \\ 29 9.49728e-05 \\
30 8.05667e-05 \\ 31 5.56683e-05 \\ 32 4.68785e-05 \\ 33 3.22474e-05 \\ 34 2.70617e-05 \\ 35 1.20885e-05 \\
36 8.26427e-06 \\ 37 6.85102e-06 \\ 38 4.68655e-06 \\ 39 3.90975e-06 \\ 40 2.67279e-06 \\ 41 2.25053e-06 \\
42 1.61275e-06 \\ 43 1.35448e-06 \\ 44 9.76483e-07 \\ 45 8.23972e-07 \\ 46 6.0661e-07 \\ 47 5.18093e-07 \\
48 2.62543e-07 \\ 49 2.20622e-07 \\ 50 1.98813e-07 \\ 51 1.79266e-07 \\ 52 1.67256e-07 \\ 53 1.16853e-07 \\
54 1.0838e-07 \\ 55 7.37312e-08 \\ 56 6.61626e-08 \\ 57 4.11142e-08 \\ 58 3.8803e-08 \\ 59 3.51083e-08 \\
60 3.28428e-08 \\ 61 3.15616e-08 \\ 62 3.0502e-08 \\ 63 2.73174e-08 \\ 64 2.5754e-08 \\ 65 2.5091e-08 \\
66 2.44342e-08 \\ 67 2.17863e-08 \\ 68 2.05239e-08 \\ 69 2.00135e-08 \\ 70 1.95522e-08 \\ 71 1.7774e-08 \\
72 1.69198e-08 \\ 73 1.65541e-08 \\ 74 1.6254e-08 \\ 75 1.49375e-08 \\ 76 1.4278e-08 \\ 77 1.39814e-08 \\
78 1.37723e-08 \\ 79 1.36871e-08 \\ 80 1.27964e-08 \\ 81 1.23106e-08 \\ 82 1.20622e-08 \\ 83 5.67549e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.536883 \\ 2 0.421246 \\ 3 0.340824 \\ 4 0.275528 \\ 5 0.210846 \\
6 0.162096 \\ 7 0.0999726 \\ 8 0.0761073 \\ 9 0.046244 \\ 10 0.0349206 \\ 11 0.0202517 \\
12 0.0145337 \\ 13 0.00803647 \\ 14 0.00556781 \\ 15 0.00295004 \\ 16 0.00186592 \\ 17 0.000954641 \\
18 0.000593362 \\ 19 0.000296123 \\ 20 0.000181345 \\ 21 8.88166e-05 \\ 22 5.36655e-05 \\ 23 2.58613e-05 \\
24 1.24581e-05 \\ 25 7.84227e-06 \\ 26 3.69761e-06 \\ 27 1.73294e-06 \\ 28 1.08077e-06 \\ 29 4.90161e-07 \\
30 2.19728e-07 \\ 31 1.31945e-07 \\ 32 6.40227e-08 \\ 33 3.67988e-08 \\ 34 2.55849e-08 \\ 35 1.91521e-08 \\
36 7.93124e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.431157 \\ 2 0.288745 \\ 3 0.201799 \\ 4 0.144547 \\ 5 0.105459 \\
6 0.0769296 \\ 7 0.0548534 \\ 8 0.0379864 \\ 9 0.0255232 \\ 10 0.0165972 \\ 11 0.0104031 \\
12 0.00633232 \\ 13 0.00377324 \\ 14 0.00220642 \\ 15 0.00126914 \\ 16 0.000719564 \\ 17 0.000402804 \\
18 0.000222927 \\ 19 0.000122108 \\ 20 6.62528e-05 \\ 21 3.56294e-05 \\ 22 1.89963e-05 \\ 23 1.00389e-05 \\
24 5.25314e-06 \\ 25 2.71575e-06 \\ 26 1.38111e-06 \\ 27 6.857e-07 \\ 28 3.28059e-07 \\ 29 1.47979e-07 \\
30 6.06975e-08 \\ 31 2.13675e-08 \\ 32 5.98003e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 1 0.509233 \\ 2 0.396147 \\ 3 0.321033 \\ 4 0.264128 \\ 5 0.163973 \\
6 0.107948 \\ 7 0.0726923 \\ 8 0.0579912 \\ 9 0.0366024 \\ 10 0.0279694 \\ 11 0.0164888 \\
12 0.0119683 \\ 13 0.00663298 \\ 14 0.00465468 \\ 15 0.002467 \\ 16 0.00169334 \\ 17 0.000866022 \\
18 0.000440597 \\ 19 0.000293594 \\ 20 0.000145472 \\ 21 7.198e-05 \\ 22 4.70487e-05 \\ 23 2.2772e-05 \\
24 1.10081e-05 \\ 25 7.08416e-06 \\ 26 3.33875e-06 \\ 27 1.55924e-06 \\ 28 9.83611e-07 \\ 29 4.37564e-07 \\
30 1.87534e-07 \\ 31 7.63759e-08 \\ 32 4.96572e-08 \\ 33 2.09909e-08 \\ 34 9.91787e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(c) Mushrooms, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=0.356,
ymin=2.237e-09, ymax=1.759e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.00408658 0.545341 \\ 0.00735092 0.43393 \\ 0.011082 0.335126 \\ 0.0147091 0.269546 \\ 0.0190524 0.172444 \\
0.0232187 0.118233 \\ 0.0276763 0.0833317 \\ 0.0313986 0.0677165 \\ 0.0345645 0.0545676 \\ 0.0381675 0.0416062 \\ 0.0418025 0.0337825 \\
0.0466598 0.0185656 \\ 0.050676 0.0132214 \\ 0.0540299 0.0105894 \\ 0.0586222 0.00719845 \\ 0.0627108 0.00587158 \\ 0.0669422 0.00382227 \\
0.0704386 0.00315565 \\ 0.074295 0.00198005 \\ 0.078443 0.00164922 \\ 0.0830586 0.00100297 \\ 0.0886173 0.000854502 \\ 0.092882 0.000554198 \\
0.0974655 0.000470355 \\ 0.101652 0.000307366 \\ 0.106918 0.000260156 \\ 0.11164 0.000170825 \\ 0.116247 0.000144181 \\ 0.120684 9.49728e-05 \\
0.126333 8.05667e-05 \\ 0.130519 5.56683e-05 \\ 0.135005 4.68785e-05 \\ 0.139323 3.22474e-05 \\ 0.143609 2.70617e-05 \\ 0.148151 1.20885e-05 \\
0.152279 8.26427e-06 \\ 0.15748 6.85102e-06 \\ 0.161608 4.68655e-06 \\ 0.165653 3.90975e-06 \\ 0.169356 2.67279e-06 \\ 0.174187 2.25053e-06 \\
0.178098 1.61275e-06 \\ 0.181875 1.35448e-06 \\ 0.185978 9.76483e-07 \\ 0.189573 8.23972e-07 \\ 0.193582 6.0661e-07 \\ 0.197489 5.18093e-07 \\
0.201651 2.62543e-07 \\ 0.205175 2.20622e-07 \\ 0.209514 1.98813e-07 \\ 0.213217 1.79266e-07 \\ 0.217116 1.67256e-07 \\ 0.221136 1.16853e-07 \\
0.224988 1.0838e-07 \\ 0.229481 7.37312e-08 \\ 0.232881 6.61626e-08 \\ 0.237257 4.11142e-08 \\ 0.24046 3.8803e-08 \\ 0.248668 3.51083e-08 \\
0.251407 3.28428e-08 \\ 0.254675 3.15616e-08 \\ 0.257417 3.0502e-08 \\ 0.264929 2.73174e-08 \\ 0.267721 2.5754e-08 \\ 0.271212 2.5091e-08 \\
0.273871 2.44342e-08 \\ 0.280087 2.17863e-08 \\ 0.282862 2.05239e-08 \\ 0.285991 2.00135e-08 \\ 0.28895 1.95522e-08 \\ 0.296679 1.7774e-08 \\
0.299478 1.69198e-08 \\ 0.302694 1.65541e-08 \\ 0.306381 1.6254e-08 \\ 0.313245 1.49375e-08 \\ 0.316443 1.4278e-08 \\ 0.319742 1.39814e-08 \\
0.323453 1.37723e-08 \\ 0.32727 1.36871e-08 \\ 0.333139 1.27964e-08 \\ 0.336433 1.23106e-08 \\ 0.340138 1.20622e-08 \\ 0.345409 5.67549e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.00408704 0.536883 \\ 0.00754121 0.421246 \\ 0.0115202 0.340824 \\ 0.0160289 0.275528 \\ 0.0203637 0.210846 \\
0.0254362 0.162096 \\ 0.0304781 0.0999726 \\ 0.034875 0.0761073 \\ 0.040075 0.046244 \\ 0.0441938 0.0349206 \\ 0.0489762 0.0202517 \\
0.0529205 0.0145337 \\ 0.0578235 0.00803647 \\ 0.06206 0.00556781 \\ 0.0671782 0.00295004 \\ 0.0714023 0.00186592 \\ 0.0760384 0.000954641 \\
0.0802573 0.000593362 \\ 0.0854147 0.000296123 \\ 0.0899539 0.000181345 \\ 0.0948024 8.88166e-05 \\ 0.0994938 5.36655e-05 \\ 0.104053 2.58613e-05 \\
0.109679 1.24581e-05 \\ 0.114145 7.84227e-06 \\ 0.120098 3.69761e-06 \\ 0.125357 1.73294e-06 \\ 0.1303 1.08077e-06 \\ 0.135878 4.90161e-07 \\
0.141484 2.19728e-07 \\ 0.14636 1.31945e-07 \\ 0.151923 6.40227e-08 \\ 0.157568 3.67988e-08 \\ 0.163267 2.55849e-08 \\ 0.168935 1.91521e-08 \\
0.174809 7.93124e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.00592042 0.431157 \\ 0.0113538 0.288745 \\ 0.0168111 0.201799 \\ 0.0217997 0.144547 \\ 0.0273106 0.105459 \\
0.0324927 0.0769296 \\ 0.0376236 0.0548534 \\ 0.0424066 0.0379864 \\ 0.0473991 0.0255232 \\ 0.0524854 0.0165972 \\ 0.0576825 0.0104031 \\
0.0627549 0.00633232 \\ 0.0678747 0.00377324 \\ 0.072809 0.00220642 \\ 0.0778794 0.00126914 \\ 0.0829668 0.000719564 \\ 0.0878128 0.000402804 \\
0.0928164 0.000222927 \\ 0.098061 0.000122108 \\ 0.102743 6.62528e-05 \\ 0.107963 3.56294e-05 \\ 0.112884 1.89963e-05 \\ 0.117561 1.00389e-05 \\
0.122489 5.25314e-06 \\ 0.12761 2.71575e-06 \\ 0.132671 1.38111e-06 \\ 0.137637 6.857e-07 \\ 0.142832 3.28059e-07 \\ 0.147933 1.47979e-07 \\
0.153241 6.06975e-08 \\ 0.158213 2.13675e-08 \\ 0.162984 5.98003e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.693147 \\ 0.005493 0.509233 \\ 0.00962942 0.396147 \\ 0.0139168 0.321033 \\ 0.018858 0.264128 \\ 0.0246076 0.163973 \\
0.0307511 0.107948 \\ 0.0366311 0.0726923 \\ 0.041213 0.0579912 \\ 0.0462771 0.0366024 \\ 0.0502911 0.0279694 \\ 0.05582 0.0164888 \\
0.059893 0.0119683 \\ 0.0650789 0.00663298 \\ 0.0689845 0.00465468 \\ 0.0741242 0.002467 \\ 0.0780105 0.00169334 \\ 0.0834156 0.000866022 \\
0.0881778 0.000440597 \\ 0.0921121 0.000293594 \\ 0.0971826 0.000145472 \\ 0.102224 7.198e-05 \\ 0.105959 4.70487e-05 \\ 0.111309 2.2772e-05 \\
0.116578 1.10081e-05 \\ 0.121185 7.08416e-06 \\ 0.126142 3.33875e-06 \\ 0.131007 1.55924e-06 \\ 0.134805 9.83611e-07 \\ 0.140519 4.37564e-07 \\
0.145549 1.87534e-07 \\ 0.151461 7.63759e-08 \\ 0.156449 4.96572e-08 \\ 0.161866 2.09909e-08 \\ 0.167814 9.91787e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(d) Mushrooms, runtime};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=184,
ymin=1.319e-09, ymax=1.503e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Iterations},
ylabel={$f(x_k)-f^*$}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.582335 \\ 1 0.4337 \\ 2 0.33647 \\ 3 0.267829 \\ 4 0.218 \\ 5 0.163284 \\
6 0.138639 \\ 7 0.109985 \\ 8 0.0911602 \\ 9 0.0786289 \\ 10 0.0660601 \\ 11 0.0571492 \\
12 0.047937 \\ 13 0.0367164 \\ 14 0.0335445 \\ 15 0.028686 \\ 16 0.0222888 \\ 17 0.0167878 \\
18 0.0149408 \\ 19 0.0129415 \\ 20 0.0088422 \\ 21 0.00790385 \\ 22 0.00711905 \\ 23 0.00435323 \\
24 0.00395944 \\ 25 0.00339657 \\ 26 0.00231424 \\ 27 0.00118693 \\ 28 0.000947434 \\ 29 0.000829448 \\
30 0.000562658 \\ 31 0.0004685 \\ 32 0.000413601 \\ 33 0.000327178 \\ 34 0.000159744 \\ 35 0.000140826 \\
36 0.000126295 \\ 37 9.67604e-05 \\ 38 9.23581e-05 \\ 39 8.63374e-05 \\ 40 7.55509e-05 \\ 41 7.2893e-05 \\
42 6.81414e-05 \\ 43 3.13889e-05 \\ 44 3.02759e-05 \\ 45 2.82147e-05 \\ 46 2.70396e-05 \\ 47 2.5801e-05 \\
48 2.52873e-05 \\ 49 2.44842e-05 \\ 50 2.41719e-05 \\ 51 2.36779e-05 \\ 52 1.11925e-05 \\ 53 1.08024e-05 \\
54 1.05145e-05 \\ 55 1.036e-05 \\ 56 1.02138e-05 \\ 57 1.0127e-05 \\ 58 1.00069e-05 \\ 59 5.17811e-06 \\
60 5.0127e-06 \\ 61 4.85964e-06 \\ 62 4.80262e-06 \\ 63 4.74811e-06 \\ 64 2.8197e-06 \\ 65 2.67014e-06 \\
66 2.63777e-06 \\ 67 2.6088e-06 \\ 68 1.784e-06 \\ 69 1.66696e-06 \\ 70 1.64516e-06 \\ 71 1.2906e-06 \\
72 1.19255e-06 \\ 73 1.17857e-06 \\ 74 1.00888e-06 \\ 75 9.3084e-07 \\ 76 9.21717e-07 \\ 77 8.28977e-07 \\
78 7.65878e-07 \\ 79 7.58939e-07 \\ 80 6.98798e-07 \\ 81 6.46936e-07 \\ 82 6.41388e-07 \\ 83 5.9688e-07 \\
84 5.53893e-07 \\ 85 5.49367e-07 \\ 86 5.13825e-07 \\ 87 4.7802e-07 \\ 88 4.7429e-07 \\ 89 4.44964e-07 \\
90 4.15048e-07 \\ 91 4.11957e-07 \\ 92 3.8748e-07 \\ 93 3.62431e-07 \\ 94 3.59859e-07 \\ 95 3.39349e-07 \\
96 3.18342e-07 \\ 97 3.16195e-07 \\ 98 2.98982e-07 \\ 99 2.81344e-07 \\ 100 2.79548e-07 \\ 101 2.65087e-07 \\
102 2.06796e-07 \\ 103 1.96355e-07 \\ 104 1.94682e-07 \\ 105 1.82166e-07 \\ 106 1.69397e-07 \\ 107 1.68092e-07 \\
108 1.57649e-07 \\ 109 1.4692e-07 \\ 110 1.4583e-07 \\ 111 1.37038e-07 \\ 112 1.28007e-07 \\ 113 1.27092e-07 \\
114 1.19688e-07 \\ 115 1.12082e-07 \\ 116 1.11313e-07 \\ 117 1.05077e-07 \\ 118 9.8669e-08 \\ 119 9.80215e-08 \\
120 9.27674e-08 \\ 121 8.73665e-08 \\ 122 8.68211e-08 \\ 123 8.23931e-08 \\ 124 7.784e-08 \\ 125 7.73802e-08 \\
126 7.36475e-08 \\ 127 6.9808e-08 \\ 128 6.94204e-08 \\ 129 6.62729e-08 \\ 130 6.30345e-08 \\ 131 6.27076e-08 \\
132 6.0053e-08 \\ 133 5.7321e-08 \\ 134 5.70452e-08 \\ 135 5.48058e-08 \\ 136 5.25007e-08 \\ 137 5.22679e-08 \\
138 5.03784e-08 \\ 139 4.84331e-08 \\ 140 4.82366e-08 \\ 141 4.66421e-08 \\ 142 4.50001e-08 \\ 143 4.48343e-08 \\
144 4.34885e-08 \\ 145 4.21023e-08 \\ 146 4.19623e-08 \\ 147 4.08262e-08 \\ 148 3.96559e-08 \\ 149 3.95377e-08 \\
150 3.85786e-08 \\ 151 3.75904e-08 \\ 152 3.74906e-08 \\ 153 3.66807e-08 \\ 154 3.58461e-08 \\ 155 3.57618e-08 \\
156 3.50779e-08 \\ 157 3.43731e-08 \\ 158 3.43019e-08 \\ 159 3.37242e-08 \\ 160 3.31289e-08 \\ 161 3.30687e-08 \\
162 3.25808e-08 \\ 163 3.20779e-08 \\ 164 3.20271e-08 \\ 165 3.1615e-08 \\ 166 3.11901e-08 \\ 167 3.11472e-08 \\
168 1.23478e-08 \\ 169 1.19925e-08 \\ 170 1.16233e-08 \\ 171 1.15849e-08 \\ 172 1.12871e-08 \\ 173 1.09804e-08 \\
174 1.09493e-08 \\ 175 1.06982e-08 \\ 176 1.04392e-08 \\ 177 1.0413e-08 \\ 178 1.02008e-08 \\ 179 9.98206e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.582335 \\ 1 0.451223 \\ 2 0.358614 \\ 3 0.26243 \\ 4 0.199731 \\ 5 0.157979 \\
6 0.129642 \\ 7 0.109689 \\ 8 0.0948846 \\ 9 0.0764026 \\ 10 0.0630313 \\ 11 0.0528272 \\
12 0.0448301 \\ 13 0.032912 \\ 14 0.0278534 \\ 15 0.0208812 \\ 16 0.0159524 \\ 17 0.0100414 \\
18 0.00860714 \\ 19 0.00694714 \\ 20 0.00458545 \\ 21 0.00221401 \\ 22 0.00190006 \\ 23 0.00164359 \\
24 0.000956062 \\ 25 0.000281592 \\ 26 0.000257799 \\ 27 0.000168989 \\ 28 5.51274e-05 \\ 29 4.85022e-05 \\
30 2.08523e-05 \\ 31 1.86481e-05 \\ 32 7.83686e-06 \\ 33 7.20842e-06 \\ 34 5.1481e-06 \\ 35 2.1734e-06 \\
36 1.83339e-06 \\ 37 7.48888e-07 \\ 38 6.68299e-07 \\ 39 4.54069e-07 \\ 40 3.45584e-07 \\ 41 1.32935e-07 \\
42 4.57895e-08 \\ 43 1.35236e-08 \\ 44 3.40308e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.582335 \\ 1 0.374003 \\ 2 0.263252 \\ 3 0.192927 \\ 4 0.146944 \\ 5 0.116017 \\
6 0.0940388 \\ 7 0.0771577 \\ 8 0.0635129 \\ 9 0.0523406 \\ 10 0.0430097 \\ 11 0.0349751 \\
12 0.028007 \\ 13 0.0220558 \\ 14 0.0170483 \\ 15 0.0128889 \\ 16 0.00947547 \\ 17 0.00669358 \\
18 0.00445622 \\ 19 0.00273116 \\ 20 0.00153769 \\ 21 0.000800379 \\ 22 0.000377149 \\ 23 0.000164906 \\
24 7.4716e-05 \\ 25 3.83748e-05 \\ 26 2.03407e-05 \\ 27 1.04754e-05 \\ 28 5.31126e-06 \\ 29 2.63769e-06 \\
30 1.25998e-06 \\ 31 5.68508e-07 \\ 32 2.33498e-07 \\ 33 8.18241e-08 \\ 34 2.18315e-08 \\ 35 3.65207e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.582335 \\ 1 0.408896 \\ 2 0.303376 \\ 3 0.231964 \\ 4 0.182626 \\ 5 0.148348 \\
6 0.12419 \\ 7 0.106546 \\ 8 0.0811697 \\ 9 0.0715511 \\ 10 0.0544982 \\ 11 0.0485349 \\
12 0.0364045 \\ 13 0.0273557 \\ 14 0.0241469 \\ 15 0.0171112 \\ 16 0.0151133 \\ 17 0.00991592 \\
18 0.00630655 \\ 19 0.00523701 \\ 20 0.00285294 \\ 21 0.00242302 \\ 22 0.00107215 \\ 23 0.000428186 \\
24 0.000360487 \\ 25 0.000124262 \\ 26 0.000110197 \\ 27 4.80827e-05 \\ 28 4.42553e-05 \\ 29 2.28192e-05 \\
30 2.12956e-05 \\ 31 1.00711e-05 \\ 32 9.22107e-06 \\ 33 4.38657e-06 \\ 34 4.01477e-06 \\ 35 1.91873e-06 \\
36 1.75372e-06 \\ 37 8.16963e-07 \\ 38 7.4686e-07 \\ 39 3.2861e-07 \\ 40 3.02027e-07 \\ 41 1.07704e-07 \\
42 1.00524e-07 \\ 43 2.90731e-08 \\ 44 2.80957e-08 \\ 45 4.92121e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(e) w8t, iterations};\begin{axis}[name=ax,
mlrnaxis, ymode=log,
xmin=0, xmax=6.73,
ymin=1.319e-09, ymax=1.503e+00,
ytickten={-8,-6,-4,-2,0},
xlabel={Runtime (s)}]
\addplot[mlrn] table[row sep=\\] {
x y \\
0 0.582335 \\ 0.0362168 0.4337 \\ 0.0636937 0.33647 \\ 0.090266 0.267829 \\ 0.115878 0.218 \\ 0.162783 0.163284 \\
0.187926 0.138639 \\ 0.22459 0.109985 \\ 0.263127 0.0911602 \\ 0.290982 0.0786289 \\ 0.328775 0.0660601 \\ 0.354793 0.0571492 \\
0.391761 0.047937 \\ 0.442286 0.0367164 \\ 0.45883 0.0335445 \\ 0.485384 0.028686 \\ 0.525437 0.0222888 \\ 0.57675 0.0167878 \\
0.591198 0.0149408 \\ 0.616995 0.0129415 \\ 0.653441 0.0088422 \\ 0.667518 0.00790385 \\ 0.692655 0.00711905 \\ 0.731601 0.00435323 \\
0.758611 0.00395944 \\ 0.773411 0.00339657 \\ 0.809988 0.00231424 \\ 0.860987 0.00118693 \\ 0.87536 0.000947434 \\ 0.901577 0.000829448 \\
0.941766 0.000562658 \\ 0.957516 0.0004685 \\ 0.983942 0.000413601 \\ 1.03656 0.000327178 \\ 1.10021 0.000159744 \\ 1.11632 0.000140826 \\
1.14355 0.000126295 \\ 1.1819 9.67604e-05 \\ 1.19675 9.23581e-05 \\ 1.22212 8.63374e-05 \\ 1.26966 7.55509e-05 \\ 1.29466 7.2893e-05 \\
1.33709 6.81414e-05 \\ 1.41193 3.13889e-05 \\ 1.42603 3.02759e-05 \\ 1.4626 2.82147e-05 \\ 1.4878 2.70396e-05 \\ 1.52467 2.5801e-05 \\
1.55017 2.52873e-05 \\ 1.58985 2.44842e-05 \\ 1.6162 2.41719e-05 \\ 1.65361 2.36779e-05 \\ 1.73308 1.11925e-05 \\ 1.74725 1.08024e-05 \\
1.78726 1.05145e-05 \\ 1.81399 1.036e-05 \\ 1.85131 1.02138e-05 \\ 1.87643 1.0127e-05 \\ 1.91293 1.00069e-05 \\ 1.98129 5.17811e-06 \\
1.99612 5.0127e-06 \\ 2.03539 4.85964e-06 \\ 2.06187 4.80262e-06 \\ 2.09879 4.74811e-06 \\ 2.16042 2.8197e-06 \\ 2.19691 2.67014e-06 \\
2.22375 2.63777e-06 \\ 2.26289 2.6088e-06 \\ 2.31742 1.784e-06 \\ 2.35395 1.66696e-06 \\ 2.37926 1.64516e-06 \\ 2.43023 1.2906e-06 \\
2.47106 1.19255e-06 \\ 2.49686 1.17857e-06 \\ 2.54739 1.00888e-06 \\ 2.58419 9.3084e-07 \\ 2.60961 9.21717e-07 \\ 2.6631 8.28977e-07 \\
2.70263 7.65878e-07 \\ 2.72825 7.58939e-07 \\ 2.77856 6.98798e-07 \\ 2.81519 6.46936e-07 \\ 2.84057 6.41388e-07 \\ 2.89345 5.9688e-07 \\
2.93225 5.53893e-07 \\ 2.95777 5.49367e-07 \\ 3.00792 5.13825e-07 \\ 3.04474 4.7802e-07 \\ 3.06983 4.7429e-07 \\ 3.1222 4.44964e-07 \\
3.16098 4.15048e-07 \\ 3.18615 4.11957e-07 \\ 3.23643 3.8748e-07 \\ 3.27345 3.62431e-07 \\ 3.29861 3.59859e-07 \\ 3.3509 3.39349e-07 \\
3.39083 3.18342e-07 \\ 3.41648 3.16195e-07 \\ 3.46622 2.98982e-07 \\ 3.50435 2.81344e-07 \\ 3.52962 2.79548e-07 \\ 3.58133 2.65087e-07 \\
3.6007 2.06796e-07 \\ 3.63933 1.96355e-07 \\ 3.66448 1.94682e-07 \\ 3.7145 1.82166e-07 \\ 3.75168 1.69397e-07 \\ 3.77709 1.68092e-07 \\
3.83151 1.57649e-07 \\ 3.87098 1.4692e-07 \\ 3.89616 1.4583e-07 \\ 3.94674 1.37038e-07 \\ 3.98356 1.28007e-07 \\ 4.00875 1.27092e-07 \\
4.06119 1.19688e-07 \\ 4.10045 1.12082e-07 \\ 4.12653 1.11313e-07 \\ 4.17636 1.05077e-07 \\ 4.213 9.8669e-08 \\ 4.23815 9.80215e-08 \\
4.29062 9.27674e-08 \\ 4.32956 8.73665e-08 \\ 4.35505 8.68211e-08 \\ 4.40614 8.23931e-08 \\ 4.44299 7.784e-08 \\ 4.46796 7.73802e-08 \\
4.52008 7.36475e-08 \\ 4.56128 6.9808e-08 \\ 4.58702 6.94204e-08 \\ 4.63867 6.62729e-08 \\ 4.67558 6.30345e-08 \\ 4.70088 6.27076e-08 \\
4.75394 6.0053e-08 \\ 4.79283 5.7321e-08 \\ 4.81829 5.70452e-08 \\ 4.86867 5.48058e-08 \\ 4.90561 5.25007e-08 \\ 4.93092 5.22679e-08 \\
4.98308 5.03784e-08 \\ 5.02284 4.84331e-08 \\ 5.04869 4.82366e-08 \\ 5.0989 4.66421e-08 \\ 5.13663 4.50001e-08 \\ 5.16195 4.48343e-08 \\
5.2146 4.34885e-08 \\ 5.25429 4.21023e-08 \\ 5.28009 4.19623e-08 \\ 5.33052 4.08262e-08 \\ 5.36691 3.96559e-08 \\ 5.3921 3.95377e-08 \\
5.44455 3.85786e-08 \\ 5.48401 3.75904e-08 \\ 5.50967 3.74906e-08 \\ 5.55998 3.66807e-08 \\ 5.59743 3.58461e-08 \\ 5.62353 3.57618e-08 \\
5.67626 3.50779e-08 \\ 5.71505 3.43731e-08 \\ 5.74089 3.43019e-08 \\ 5.79116 3.37242e-08 \\ 5.82798 3.31289e-08 \\ 5.85317 3.30687e-08 \\
5.90586 3.25808e-08 \\ 5.94467 3.20779e-08 \\ 5.96994 3.20271e-08 \\ 6.02035 3.1615e-08 \\ 6.05702 3.11901e-08 \\ 6.08216 3.11472e-08 \\
6.10455 1.23478e-08 \\ 6.15806 1.19925e-08 \\ 6.19601 1.16233e-08 \\ 6.22118 1.15849e-08 \\ 6.27133 1.12871e-08 \\ 6.30793 1.09804e-08 \\
6.33308 1.09493e-08 \\ 6.38453 1.06982e-08 \\ 6.42447 1.04392e-08 \\ 6.45018 1.0413e-08 \\ 6.50023 1.02008e-08 \\ 6.53732 9.98206e-09 \\
};
\addplot[mlrnnested] table[row sep=\\] {
x y \\
0 0.582335 \\ 0.028616 0.451223 \\ 0.0535636 0.358614 \\ 0.0927941 0.26243 \\ 0.134111 0.199731 \\ 0.171692 0.157979 \\
0.208458 0.129642 \\ 0.245271 0.109689 \\ 0.284643 0.0948846 \\ 0.338262 0.0764026 \\ 0.392724 0.0630313 \\ 0.446426 0.0528272 \\
0.502853 0.0448301 \\ 0.569327 0.032912 \\ 0.628772 0.0278534 \\ 0.688632 0.0208812 \\ 0.752435 0.0159524 \\ 0.816952 0.0100414 \\
0.853688 0.00860714 \\ 0.904211 0.00694714 \\ 0.964218 0.00458545 \\ 1.03152 0.00221401 \\ 1.06904 0.00190006 \\ 1.12007 0.00164359 \\
1.18157 0.000956062 \\ 1.24889 0.000281592 \\ 1.28591 0.000257799 \\ 1.34624 0.000168989 \\ 1.41759 5.51274e-05 \\ 1.46897 4.85022e-05 \\
1.54101 2.08523e-05 \\ 1.59149 1.86481e-05 \\ 1.66312 7.83686e-06 \\ 1.70105 7.20842e-06 \\ 1.76849 5.1481e-06 \\ 1.83267 2.1734e-06 \\
1.88483 1.83339e-06 \\ 1.95024 7.48888e-07 \\ 1.98705 6.68299e-07 \\ 2.05488 4.54069e-07 \\ 2.11691 3.45584e-07 \\ 2.18261 1.32935e-07 \\
2.24881 4.57895e-08 \\ 2.31313 1.35236e-08 \\ 2.38038 3.40308e-09 \\
};
\addplot[rn] table[row sep=\\] {
x y \\
0 0.582335 \\ 0.0702865 0.374003 \\ 0.135164 0.263252 \\ 0.199737 0.192927 \\ 0.267834 0.146944 \\ 0.332501 0.116017 \\
0.400306 0.0940388 \\ 0.466096 0.0771577 \\ 0.536093 0.0635129 \\ 0.600554 0.0523406 \\ 0.664672 0.0430097 \\ 0.728891 0.0349751 \\
0.795964 0.028007 \\ 0.860421 0.0220558 \\ 0.924539 0.0170483 \\ 0.988188 0.0128889 \\ 1.05518 0.00947547 \\ 1.1192 0.00669358 \\
1.18327 0.00445622 \\ 1.24784 0.00273116 \\ 1.31502 0.00153769 \\ 1.37926 0.000800379 \\ 1.4434 0.000377149 \\ 1.50735 0.000164906 \\
1.57525 7.4716e-05 \\ 1.63985 3.83748e-05 \\ 1.70509 2.03407e-05 \\ 1.76914 1.04754e-05 \\ 1.83877 5.31126e-06 \\ 1.90527 2.63769e-06 \\
1.96963 1.25998e-06 \\ 2.03365 5.68508e-07 \\ 2.1011 2.33498e-07 \\ 2.16519 8.18241e-08 \\ 2.23124 2.18315e-08 \\ 2.29507 3.65207e-09 \\
};
\addplot[tlrn] table[row sep=\\] {
x y \\
0 0.582335 \\ 0.0412766 0.408896 \\ 0.0780677 0.303376 \\ 0.114693 0.231964 \\ 0.152118 0.182626 \\ 0.188808 0.148348 \\
0.227908 0.12419 \\ 0.267764 0.106546 \\ 0.333339 0.0811697 \\ 0.370537 0.0715511 \\ 0.435259 0.0544982 \\ 0.474775 0.0485349 \\
0.539207 0.0364045 \\ 0.602915 0.0273557 \\ 0.639684 0.0241469 \\ 0.707371 0.0171112 \\ 0.744888 0.0151133 \\ 0.808943 0.00991592 \\
0.876341 0.00630655 \\ 0.91578 0.00523701 \\ 0.979897 0.00285294 \\ 1.01708 0.00242302 \\ 1.08118 0.00107215 \\ 1.14791 0.000428186 \\
1.18491 0.000360487 \\ 1.25092 0.000124262 \\ 1.28803 0.000110197 \\ 1.36208 4.80827e-05 \\ 1.3988 4.42553e-05 \\ 1.47082 2.28192e-05 \\
1.50862 2.12956e-05 \\ 1.57454 1.00711e-05 \\ 1.61109 9.22107e-06 \\ 1.67923 4.38657e-06 \\ 1.71646 4.01477e-06 \\ 1.78659 1.91873e-06 \\
1.82329 1.75372e-06 \\ 1.89107 8.16963e-07 \\ 1.92806 7.4686e-07 \\ 1.99959 3.2861e-07 \\ 2.03662 3.02027e-07 \\ 2.10083 1.07704e-07 \\
2.13846 1.00524e-07 \\ 2.20999 2.90731e-08 \\ 2.2472 2.80957e-08 \\ 2.31214 4.92121e-09 \\
};
\end{axis}
\node[anchor=north,font=\small] at (ax.below south) {(f) w8t, runtime};Figure 3. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) on the a9a, Mushrooms, and w8t datasets for logistic regression. The left column reports the objective gap versus iterations, while the right column reports the objective gap versus wall-clock time. Row-nested restriction operators significantly improve the convergence behavior of MLRN.
Logistic regression
In the second experiment, we consider logistic regression problems using real-world datasets. The objective function is given by
where and we set to make the problem ill-conditioned. The results are presented in Figure 2 and Figure 3.
Across the considered datasets, nested MLRN and TLRN exhibit convergence behavior comparable to that of the full-space RN method. Moreover, as shown in Figure 2, when applied to large datasets, both methods achieve substantial reductions in computational time, illustrating the benefits of performing corrections in lower-dimensional spaces. In contrast, non-row-nested MLRN exhibits slower convergence on some instances, particularly those shown in Figure 3, which may prevent the method from converging faster than RN in computational time. These observations suggest that the structure of the restriction operators, particularly row-nestedness, can play an important role in the practical performance of multilevel gradient-regularized Newton methods. Overall, the results demonstrate that row-nested restriction operators can yield computationally efficient multilevel methods, complementing the theoretical convergence guarantees established in this paper.
Conclusions
We developed and analyzed multilevel gradient-regularized Newton methods for unconstrained convex optimization, establishing a global convergence rate in function values. Among our results, the proof of Theorem 3.8 is of particular interest as it combines a weighted gradient measure with row-nestedness, a special structure of the restriction operators introduced in § 3.2 for the purposes of this analysis. These ideas are new in multilevel optimization theory and we believe that they could prove useful for establishing fast convergence rates for other multilevel methods, such as cubic-regularized Newton and higher-order methods for convex optimization.
Code and data availability
The code and data for reproducing the numerical results is available at
https://github.com/Ntsip/Multilevel-Gradient-Regularized-Newton-Method.
It is authored by Nick Tsipinakis and published under the MIT license.
CRediT author statement
Nick Tsipinakis: Conceptualization, Methodology, Software, Formal Analysis, Investigation, Writing — Original Draft, Visualization; Matthias Voigt: Writing — Review & Editing, Visualization, Supervision.
Acknowledgments
The authors used AI tools to assist with language and figure editing, mathematical proof verification, and the development and debugging of numerical implementations. All mathematical results, algorithms, and computational experiments were reviewed and validated by the authors. The authors assume responsibility for all content.
Background and proofs of auxiliary results
Proof. If for some , the recurrence and nonnegativity imply for all , so the result follows immediately. Otherwise, nonnegativity implies , and hence
where we used for . Summing over , exploiting the telescoping property, and using gives
which proves the claim. ◻
We also refer to [7 Theorem 4.1.4] for a related recurrence estimate.
Proof. Let for some , and define
By the coarse-level eligibility condition,
Moreover, since the iterates remain in the compact set , there exists
Consequently,
Using the definition of , we obtain
Hence,
By the descent estimate in [12 Lemma 3.3], the coarse update satisfies
Combining the preceding estimates yields
Since the number of coarse levels is finite, . Therefore, every coarse iteration decreases the objective function by at least . By monotonicity of the function values and their lower boundedness by ,
Thus, , proving the result. ◻
Proof of lemma:multilevel-coarse-estimates
Proof. The result follows from the analysis in [12 Lemma 3.2 and Corollary 3.4], which allows the restriction operator to vary with the iteration. Indeed, at an iteration , applying the corresponding estimates with and yields
where
These inequalities yield ineq:multilevel-reduced-gradient-stability, while the proof of [12 Corollary 3.4] gives ineq:multilevel-reduction. ◻
Proof of lemma:case-II
Proof. The proof of Case II in Theorem 2.5 extends naturally to coarse levels. Thus, below, only the main steps of the proof will be discussed. Let and enumerate increasingly, with . If for some , then, using the coarse decrease estimate and , we obtain
The same bound holds for by the fine-level decrease estimate. Hence, arguing as in Case II of Theorem 2.5,
Since , Proposition prop:recurrence-rate yields
◻
References
- [1]Gratton, S., Sartenaer, A., & Toint, P. L. (2008). Recursive Trust-Region Methods for Multiscale Nonlinear Optimization. SIAM J. Optim., 19(1), 414–444. https://doi.org/10.1137/050623012
- [2]Nash, S. G. (2000). A Multigrid Approach to Discretized Optimization Problems. Optim. Methods Softw., 14(1-2), 99–116. https://doi.org/10.1080/10556780008805795
- [3]Ho, C. P., Kočvara, M., & Parpas, P. (2022). Newton-Type Multilevel Optimization Method. Optim. Methods Softw., 37(1), 45–78. https://doi.org/10.1080/10556788.2019.1700256
- [4]Wen, Z., & Goldfarb, D. (2009). A Line Search Multigrid Method for Large-Scale Nonlinear Optimization. SIAM J. Optim., 20(3), 1478–1503. https://doi.org/10.1137/08071524X
- [5]Tsipinakis, N., & Parpas, P. (2024). A Multilevel Method for Self-Concordant Minimization. J. Optim. Theory Appl., 203(3), 2509–2559. https://doi.org/10.1007/s10957-024-02509-z
- [6]Tsipinakis, N., Tigas, P., & Parpas, P. (2026). A Multilevel Low-Rank Newton Method with Super-Linear Convergence Rate and Its Application to Non-Convex Problems. Trans. Mach. Learn. Res.. https://openreview.net/forum?id=PKakPzVVjaopenreview.net/forum?id=PKakPzVVja
- [7]Nesterov, Y. (2018). Lectures on Convex Optimization (Vol. 137). Springer. https://doi.org/10.1007/978-3-319-91578-4
- [9]Nesterov, Y. (2008). Accelerating the Cubic Regularization of Newton's Method on Convex Problems. Math. Program., 112(1), 159–181. https://doi.org/10.1007/s10107-006-0089-x
- [10]Tsipinakis, N., & Parpas, P. (2025). Convergence Rates of Newton's Method for Strongly Self-Concordant Minimization (Numbers arXiv:2507.23558). https://doi.org/10.48550/arXiv.2507.23558
- [11]Gross, C., & Krause, R. (2009). On the Convergence of Recursive Trust-Region Methods for Multiscale Nonlinear Optimization and Applications to Nonlinear Mechanics. SIAM J. Numer. Anal., 47(4), 3044–3069. https://doi.org/10.1137/08071819X
- [12]Tsipinakis, N., & Parpas, P. (2025). Multilevel Regularized Newton Methods with Fast Convergence Rates. SIAM J. Sci. Comput., S232–S257. https://doi.org/10.1137/24M1677307
- [13]Mishchenko, K. (2023). Regularized Newton Method with Global \( O( 1/k^2)\) Convergence. SIAM J. Optim., 33(3), 1440–1462. https://doi.org/10.1137/22M1488752
- [14]Doikov, N., & Nesterov, Y. (2024). Gradient regularization of Newton method with Bregman distances. Math. Program., 204, 1–25. https://doi.org/10.1007/s10107-023-01943-7
- [15]Calandra, H., Gratton, S., Riccietti, E., & Vasseur, X. (2021). On High-Order Multilevel Optimization Strategies. SIAM J. Optim., 31(1), 307–330. https://doi.org/10.1137/19M1255355
- [16]Ang, A., De Sterck, H., & Vavasis, S. (2024). MGProx: A Nonsmooth Multigrid Proximal Gradient Method with Adaptive Restriction for Strongly Convex Optimization. SIAM J. Optim., 34(3), 2788–2820. https://doi.org/10.1137/23M1552140
- [17]Kočvara, M., & Mohammed, S. (2016). A First-Order Multigrid Method for Bound-Constrained Convex Optimization. Optim. Methods Softw., 31(3), 622–644. https://doi.org/10.1080/10556788.2016.1146267
- [18]Cartis, C., Gould, N. I. M., & Toint, P. L. (2011). Adaptive Cubic Regularisation Methods for Unconstrained Optimization. Part I: Motivation, Convergence and Numerical Results. Math. Program., 127(2), 245–295. https://doi.org/10.1007/s10107-009-0286-5
- [19]Tsipinakis, N., Parpas, P., & Voigt, M. (2025). Adaptive Multilevel Newton: A Quadratically Convergent Optimization Method (Numbers arXiv:2510.24967). https://doi.org/10.48550/arXiv.2510.24967
- [20]Nesterov, Y., & Polyak, B. T. (2006). Cubic regularization of Newton method and its global performance. Math. Program., 108(1), 177–205. https://doi.org/10.1007/s10107-006-0706-8
- [21]Cartis, C., Gould, N. I. M., & Toint, P. L. (2011). Adaptive cubic regularisation methods for unconstrained optimization. Part II: Worst-case function- and derivative-evaluation complexity. Math. Program., 130(2), 295–319. https://doi.org/10.1007/s10107-009-0337-y
- [22]Griewank, A. (1981). The modification of Newton's method for unconstrained optimization by bounding cubic terms. In Department of Applied Mathematics and Theoretical Physics, University of Cambridge (Number NA/12).
- [23]Polyak, R. A. (2009). Regularized Newton Method for Unconstrained Convex Optimization. Math. Program., 120(1), 125–145. https://doi.org/10.1007/s10107-007-0143-3
- [24]Cartis, C., Gould, N. I. M., & Toint, P. L. (2012). Evaluation complexity of adaptive cubic regularization methods for convex unconstrained optimization. Optim. Methods Softw., 27(2), 197–219. https://doi.org/10.1080/10556788.2011.602076
- [25]Hovhannisyan, V., Parpas, P., & Zafeiriou, S. (2016). MAGMA: Multilevel Accelerated Gradient Mirror Descent Algorithm for Large-Scale Convex Composite Minimization. SIAM J. Imaging Sci., 9(4), 1829–1857. https://doi.org/10.1137/15M104013X
- [26]Briggs, W. L., Henson, V. E., & McCormick, S. F. (2000). A Multigrid Tutorial. SIAM. https://doi.org/10.1137/1.9780898719505
- [27]Gould, N. I. M., Lucidi, S., Roma, M., & Toint, P. L. (1999). Solving the Trust-Region Subproblem Using the Lanczos Method. SIAM J. Optim., 9(2), 504–525. https://doi.org/10.1137/S1052623497322735
- [28]Binev, P., Cohen, A., Dahmen, W., DeVore, R., Petrova, G., & Wojtaszczyk, P. (2011). Convergence Rates for Greedy Algorithms in Reduced Basis Methods. SIAM J. Math. Anal., 43(3), 1457–1472. https://doi.org/10.1137/100795772
- [29]Lewis, R. M., & Nash, S. G. (2005). Model Problems for the Multigrid Optimization of Systems Governed by Differential Equations. SIAM J. Sci. Comput., 26(6), 1811–1837. https://doi.org/10.1137/S1064827502407792
- [30]Branicky, M. S. (1998). Multiple Lyapunov Functions and Other Analysis Tools for Switched and Hybrid Systems. IEEE Trans. Automat. Control, 43(4), 475–482. https://doi.org/10.1109/9.664150
- [31]Zhao, X., Shi, P., Yin, Y., & Nguang, S. K. (2017). New Results on Stability of Slowly Switched Systems: A Multiple Discontinuous Lyapunov Function Approach. IEEE Trans. Automat. Control, 62(7), 3502–3509. https://doi.org/10.1109/TAC.2016.2614911