betaFeedback

Reading v1View latest version

WorksMathematics

Global O(k−2)\mathcal{O}\left(k^{-2}\right) Convergence of Multilevel Gradient-Regularized Newton Methods

Abstract

We develop and analyze multilevel gradient-regularized Newton methods for unconstrained convex optimization with Lipschitz-continuous Hessians. By computing Newton corrections in lower-dimensional subspaces, the proposed methods aim to reduce computational costs while preserving the global O(k−2)\mathcal{O}\left(k^{-2}\right) convergence rate in function values of the full-space method. To establish this rate for multilevel methods, we introduce a class of row-nested restriction operators that ensures monotonicity of the reduced-gradient norms across levels. Numerical experiments with row-nested operators demonstrate convergence behavior comparable to full-space gradient-regularized Newton, while offering substantial computational savings on large-scale instances.

Introduction

Cubic regularization of Newton’s method [22, 20] is a fundamental second-order approach for smooth, possibly nonconvex optimization, achieving an O(ε−3/2)\mathcal{O}(\varepsilon^{-3/2}) complexity for first-order stationarity in nonconvex problems and a global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate for convex objectives with Lipschitz-continuous Hessians. Subsequent developments include accelerated cubic Newton methods attaining an O(k−3)\mathcal{O}(k^{-3}) rate [9] and Adaptive Regularization with Cubics (ARC) [18, 21, 24]. However, cubic regularization requires solving a potentially expensive subproblem at each iteration.

Gradient-regularized Newton methods provide a simpler alternative, with iterations of the form

xk+1=xk−(∇2f(xk)+akI)−1∇f(xk),\begin{equation} x_{k+1}=x_k-\left(\nabla^2 f(x_k)+a_k I\right)^{-1}\nabla f(x_k), \tag*{(1)} \end{equation}

where ak>0a_k>0. Following the early work of Polyak [23], Mishchenko [13] and Doikov and Nesterov [14] established that choosing aka_k proportional to L∥∇f(xk)∥\sqrt{L\|\nabla f(x_k)\|} yields the global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate for convex objectives with Lipschitz-continuous Hessians, improving upon the O(k−1)\mathcal{O}(k^{-1}) rate of standard gradient descent. Nevertheless, computing full-dimensional Hessians and solving the corresponding linear systems can be prohibitively expensive for large-scale problems.

Inspired by classical multigrid methods [26], multilevel optimization reduces these costs by computing corrections in lower-dimensional spaces, replacing full-dimensional Newton systems with smaller ones. Multilevel strategies have been successfully developed for Newton and regularized Newton methods [1, 6, 3, 10, 5], higher-order regularization methods, including ARC [15], as well as line-search, accelerated gradient, and proximal methods [16, 4, 25].

In a foundational work, Gratton, Sartenaer, and Toint [1] introduced recursive multilevel trust-region (RMTR) methods, establishing an O(ε−2)\mathcal{O}\big(\varepsilon^{-2}\big) worst-case complexity bound for first-order stationarity in possibly nonconvex problems. Their framework permits recursion to a coarser level when

∥Rigi,k∥≥κg∥gi,k∥,∥Rigi,k∥>εi−1g,\begin{equation} \left\lVert R_i g_{i,k} \right\rVert \geq \kappa_g \left\lVert g_{i,k} \right\rVert, \qquad \left\lVert R_i g_{i,k} \right\rVert > \varepsilon^g_{i-1}, \tag*{(2)} \end{equation}

where gi,kg_{i,k} is the gradient of the current level-dependent model, κg>0\kappa_g>0 is a prescribed parameter, and Ri∈Rni−1×niR_i \in \mathbb{R}^{n_{i-1}\times n_i} restricts information to the next coarser level. Importantly, this eligibility condition is evaluated before computing a coarse correction, avoiding additional trial steps for level selection.

In general, second- or higher-order multilevel methods need not inherit the fast convergence rates of their full-space counterparts, since coarse corrections exploit only partial second- or higher-order information. Notable results preserving full-space convergence guarantees include RMTR [1] and the high-order multilevel methods of Calandra et al. [15], which attain the O(ε−3/2)\mathcal{O}\big(\varepsilon^{-3/2}\big) first-order stationarity complexity of cubic regularization for possibly nonconvex problems. More recently, for convex objectives, the multilevel gradient-regularized Newton method [12] was shown to achieve the global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate, while the adaptive multilevel Newton method [19] achieves local quadratic convergence for strongly convex objectives.

However, the deterministic formulations of these last two methods rely on a stronger level-selection criterion, namely

∥Ri∇f(xk)∥≥σ∥∇f(xk)∥,∥Ri∇f(xk+1)∥≥σ∥∇f(xk+1)∥.\begin{equation} \left\lVert R_i \nabla f(x_k) \right\rVert \geq \sigma \left\lVert \nabla f(x_k) \right\rVert, \qquad \left\lVert R_i \nabla f(x_{k+1}) \right\rVert \geq \sigma \left\lVert \nabla f(x_{k+1}) \right\rVert. \tag*{(3)} \end{equation}

The second condition requires computing a trial coarse step to evaluate the gradient at the prospective next iterate, potentially increasing the computational cost. In contrast, the classical eligibility criterion (2) uses only current-iterate information. To the best of our knowledge, a direct global O(k−2)\mathcal{O}\big(k^{-2}\big) convergence analysis for deterministic multilevel gradient-regularized Newton methods employing such a pre-step level-selection criterion has remained unavailable. Establishing this rate will provide theoretical justification for employing multilevel methods in convex optimization, combining the fast convergence guarantees of full-space Newton methods with the potential computational savings of lower-dimensional corrections.

Contributions. We develop multilevel gradient-regularized Newton methods for convex objectives with Lipschitz-continuous Hessians, exploiting lower-dimensional subspaces and the classical level-selection criterion (2) (cf. [1]). We show that these methods preserve the global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate of their full-space counterpart. For two levels, Theorem 2.5 establishes this rate under arbitrary coarse/fine alternation. For multiple levels with general restriction operators, Theorem 2.5 requires an explicit fine-iteration frequency condition, which is eventually satisfied by the algorithm (§ 3.1) but may be restrictive during coarse-dominated phases. To eliminate this condition, we introduce row-nested restriction operators (§ 3.2), whose structure guarantees monotonicity of reduced-gradient norms across levels. Exploiting this property, we establish the global O(k−2)\mathcal{O}\big(k^{-2}\big) rate without imposing any fine-iteration frequency condition (Theorem 3.8).

Our analysis builds on the ideas of [13], but addresses a fundamental difficulty absent from full-space methods: coarse and fine corrections control different gradient norms, whose convergence properties cannot be directly transferred across levels. To overcome this, we introduce a level-dependent weighted gradient measure that provides uniform control across all possible coarse-to-fine, coarse-to-coarse, and fine-to-coarse transitions. This, together with the proposed class of row-nested restriction operators (def:nested-restrictions), constitutes the main theoretical novelty of our analysis, enabling us to preserve the convergence rate of the full-space method without imposing any condition on the frequency of fine-level iterations.

We support our theoretical analysis with numerical experiments comparing the proposed methods against the full-space gradient-regularized Newton method. The results demonstrate that the two-level and row-nested multilevel methods exhibit convergence behavior comparable to that of their full-dimensional counterpart, while achieving substantial reductions in computational time on large-scale instances. Furthermore, the experiments highlight the importance of the restriction operator structure, with row-nested hierarchies generally exhibiting more favorable convergence behavior than their non-nested counterparts. These findings suggest that row-nested restriction operators can offer an attractive choice from both theoretical and computational perspectives.

Early multilevel optimization frameworks were developed by Nash [2] and Lewis and Nash [29], including implementations based on truncated-Newton methods. A common feature of subsequent multilevel methods [1, 11, 3, 15] is the use of coarse-level eligibility conditions of the form (2). Wen and Goldfarb [4] developed a line-search multigrid method establishing R-linear convergence under strong convexity and suitable conditions on the search directions. Calandra et al. [15] extended higher-order regularization to multilevel optimization. Other multilevel Newton-type methods [3, 5, 6] establish global convergence or fast local rates under specific assumptions. However, unlike our approach, these Newton-type methods do not employ gradient-dependent Hessian regularization to handle potentially singular Hessians, nor do they establish the global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate for general convex objectives.

Tsipinakis and Parpas [12] developed the multilevel gradient-regularized Newton method achieving the global O(k−2)\mathcal{O}\big(k^{-2}\big) function-value rate for convex objectives. However, unlike the methods proposed in this paper, their method requires the look-ahead condition (3), introducing additional computational costs. This condition avoids directly analyzing transitions between coarse and fine gradient measures and thus their proof cannot be applied to our setting. In contrast, our analysis preserves the global O(k−2)\mathcal{O}\big(k^{-2}\big) rate under classical level selection without requiring look-ahead trial steps.

Nested subspaces are widely used in Krylov-based trust-region methods, reduced-basis approximation, and multilevel optimization [27, 28, 11]. The row-nested restriction operators considered here additionally guarantee monotonicity of the restricted-gradient norms across levels. Moreover, the weighted-gradient analysis is closely related to multiple-Lyapunov-function techniques for switched systems [30, 31]. In our setting, we combine a suitably weighted gradient measure with regularized Newton decrease bounds to establish a global O(k−2)\mathcal{O}\big(k^{-2}\big) convergence rate in function values under arbitrary coarse/fine alternation.

Main assumptions and notation

We consider the unconstrained optimization problem

min⁡x∈Rnf(x),\begin{equation*} \min_{x\in\mathbb{R}^n} f(x), \end{equation*}

where f:Rn→Rf:\mathbb{R}^n\to\mathbb{R} is a twice continuously differentiable convex function. Given an initial point x0∈Rnx_0 \in \mathbb{R}^n, our goal is to develop iterative methods achieving the global O(k−2)\mathcal{O}\big(k^{-2}\big) convergence rate in function values. Throughout, we impose the following assumptions.

Assumption 1.1. The initial sublevel set L(x0)≔{x∈Rn : f(x)≤f(x0)}\mathcal L(x_0) \coloneqq \{x\in\mathbb{R}^n \, : \, f(x)\leq f(x_0)\} is bounded.

Since ff is continuous, L(x0)\mathcal L(x_0) is compact, ensuring the existence of a global minimizer x∗∈Rnx^*\in\mathbb{R}^n. We denote f∗≔f(x∗)f^*\coloneqq f(x^*).

Assumption. There exists an L>0L>0 such that ∥∇2f(x)−∇2f(y)∥≤L∥x−y∥∀x,y∈Rn.\left\lVert \nabla^2f(x)-\nabla^2f(y) \right\rVert \leq L\left\lVert x-y \right\rVert \qquad \forall x,y\in\mathbb{R}^n.

These assumptions are standard in the analysis of second-order optimization methods; see, e.g., [7]. Throughout, ∥⋅∥\left\lVert \cdot \right\rVert denotes the Euclidean norm for vectors and the induced spectral norm for matrices. For convenience, we write

∇fk≔∇f(xk),∇2fk≔∇2f(xk).\nabla f_k\coloneqq\nabla f(x_k), \qquad \nabla^2f_k\coloneqq\nabla^2f(x_k).

Moreover, by compactness of L(x0)\mathcal L(x_0), the constants

H≔sup⁡x∈L(x0)∥∇2f(x)∥,D≔sup⁡x∈L(x0)∥x−x∗∥\begin{equation} H\coloneqq\sup_{x\in\mathcal L(x_0)} \left\lVert \nabla^2f(x) \right\rVert, \qquad D\coloneqq\sup_{x\in\mathcal L(x_0)} \left\lVert x-x^* \right\rVert \tag*{(4)} \end{equation}

are finite.

Organization. In § 2, we introduce the two-level gradient-regularized Newton method and establish its convergence rate. In § 3, we extend the analysis to multiple coarse levels, establishing convergence guarantees under a fine-iteration frequency condition and, alternatively, without this condition using row-nested restriction operators introduced in § 3.2. Numerical experiments comparing the proposed methods with full-space gradient-regularized Newton are presented in § 4. Finally, § 5 concludes the paper.

The two-level Newton method with gradient regularization

We consider the Newton method with gradient regularization that alternates between the fine and one coarse level. For the two-level algorithm we consider a single restriction operator that transfers the information from fine to coarse space, i.e., R:Rn→RdR : \mathbb{R}^n \to \mathbb{R}^d, where d<nd < n are the coarse space dimensions. Moreover, the prolongation operator is defined in a standard way, i.e., P≔R⊤P \coloneqq R^\top. For the operators we take the standard assumptions (see for instance [1, 3]).

Assumption. Let R∈Rd×nR\in\mathbb{R}^{d\times n} be a full-row-rank restriction operator, with d≤nd\leq n, and let P=R⊤P=R^\top be its corresponding prolongation operator. We define ω≔∥R∥. \omega\coloneqq\left\lVert R \right\rVert.

Let σ∈(0,min⁡{1,ω})\sigma \in (0, \min\{1, \omega \}) and ε>0\varepsilon > 0. We define the sets of coarse and fine iterations

C:−{k∈N : ∥R∇fk∥≥σ∥∇fk∥, ∥R∇fk∥≥ε},F:−N∖C.\begin{equation*} \mathcal{C}\coloneq \{ k \in \mathbb{N}\, : \, \left\lVert R \nabla f_k \right\rVert \geq \sigma \left\lVert \nabla f_k \right\rVert, \, \left\lVert R \nabla f_k \right\rVert \geq \varepsilon\}, \quad \mathcal{F}\coloneq \mathbb{N}\setminus \mathcal{C}. \end{equation*}

We can consider C\mathcal{C} as the set of effective coarse iterations and F\mathcal{F} the set of fine iterations. Thus, if m∈Cm \in \mathcal{C} then the algorithm will perform a coarse iteration otherwise a fine iteration. The two-level gradient-regularized Newton algorithm is presented in alg:two-level.

\caption{Two-Level Regularized Newton (TLRN)}
\label{alg:two-level}
\begin{algorithmic}
\REQUIRE $x_0 \in \mathbb{R}^n, R : \mathbb{R}^n \to \mathbb{R}^d, \sigma \in (0, \min\{1, \omega \}), \varepsilon > 0$. 
\ENSURE Sequence of iterates $(x_0,x_1,x_2,\ldots)$.
\FOR{$k = 0, 1,  \ldots$}
\IF{$\nabla f_k = 0$} \STATE \textbf{stop} \ENDIF
\STATE Set {\begin{equation*} 
    x_{k+1} \coloneqq
\begin{cases}
x_k - P \left(R \nabla^2 f_kP + \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}} I_d \right)^{-1} R \nabla f_k, & k \in \mathcal{C}, \\
x_k - \left( \nabla^2 f_k+ \sqrt{\frac{L \left\lVert \nabla f_k \right\rVert}{2}} I_n \right)^{-1}  \nabla f_k,  & k \in \mathcal{F}.
\end{cases}
\end{equation*}}
\ENDFOR
\end{algorithmic}

The following result will be needed for the analysis of alg:two-level; it has been proved in [12 Lemma 3.3] and [13 Lemma 3].

The following stability estimates for the coarse and fine corrections were also established in [12] and [13], and hold for all coarse and fine updates, respectively:

∥R∇fk+1∥≤2∥R∇fk∥for a coarse update,∥∇fk+1∥≤2∥∇fk∥for a fine update.\begin{alignat} {2} \left\lVert R \nabla f_{k+1} \right\rVert &\leq 2 \left\lVert R \nabla f_k \right\rVert &\qquad& \text{for a coarse update}, \tag*{(5)} \\ \left\lVert \nabla f_{k+1} \right\rVert &\leq 2 \left\lVert \nabla f_k \right\rVert &\qquad& \text{for a fine update}. \tag*{(6)} \end{alignat}

Convergence analysis

When C=∅\mathcal{C}= \emptyset, then alg:two-level coincides with the gradient regularization of Newton’s method (see [14, 13]) which converges with the fast O(k−2)\mathcal{O}\big(k^{-2}\big) rate. In what follows we show that this rate can be attained when the sequence (xk)k∈N{(x_k)}_{k \in \mathbb{N}} is always generated from the coarse space, i.e.,

xk+1=xk−P(R∇2fkP+ω3L∥R∇fk∥2Id)−1R∇fk.x_{k+1} = x_k - P \left(R \nabla^2 f_kP + \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}} I_d \right)^{-1} R \nabla f_k.

For the purposes of this result, the condition ∥R∇fk∥≥ε\left\lVert R\nabla f_k \right\rVert\geq\varepsilon is therefore not imposed.

Proof. Fix k∈Nk\in\mathbb{N} and set Ik≔{m∈{0,…,k−1} : ∥R∇fm+1∥>14∥R∇fm∥}I^k\coloneqq \left\{ m\in\{0,\ldots,k-1\} \, : \, \|R\nabla f_{m+1}\|>\frac14\|R\nabla f_m\| \right\}. We distinguish two cases.

Case I: ∣Ik∣≤k/2\big|I^k\big| \le k/2. By the definition of IkI^k and (5),

∥R∇fm+1∥≤{14∥R∇fm∥,m∉Ik,2∥R∇fm∥,m∈Ik.\|R\nabla f_{m+1}\| \leq \begin{cases} \frac14\|R\nabla f_m\|, & m\notin I^k,\\[1ex] 2\|R\nabla f_m\|, & m\in I^k. \end{cases}

Therefore,

∥R∇fk∥≤2∣Ik∣4k−∣Ik∣∥R∇f0∥≤2−k/2∥R∇f0∥.\|R\nabla f_k\| \leq \frac{2^{\left|I^k\right|}}{4^{k-\left|I^k\right|}} \|R\nabla f_0\| \leq 2^{-k/2}\|R\nabla f_0\|.

Using convexity and the coarse-level condition,

f(xk)−f∗≤D∥∇fk∥≤Dσ∥R∇fk∥≤ωDσ 2−k/2∥∇f0∥.\begin{aligned} f(x_k)-f^* &\leq D\|\nabla f_k\| \leq \frac{D}{\sigma}\|R\nabla f_k\| \leq \frac{\omega D}{\sigma}\, 2^{-k/2}\|\nabla f_0\|. \end{aligned}

Thus, in this case the function value gap decreases geometrically and, in particular, is O(k−2)\mathcal{O}\big(k^{-2}\big).

Case II: ∣Ik∣>k/2\big|I^k\big| > k/2. Let τ≔∣Ik∣\tau\coloneqq \big|I^k\big|, enumerate Ik={k0,…,kτ−1}, k0<⋯<kτ−1<k,I^k=\{k_0,\ldots,k_{\tau-1}\}, \ k_0<\cdots<k_{\tau-1}<k, and set kτ≔kk_\tau\coloneqq k. By ineq:reduction-coarse and the definition of IkI^k, for ℓ=0,…,τ−1\ell=0,\ldots,\tau-1,

f(xkℓ)−f(xkℓ+1)≥∥R∇fkℓ+1∥28(ω3L∥R∇fkℓ∥/2)1/2≥∥R∇fkℓ∥3/2128(ω3L/2)1/2≥σ3/2∥∇fkℓ∥3/2128(ω3L/2)1/2.\begin{aligned} f(x_{k_\ell})-f(x_{k_\ell+1}) &\geq \frac{\|R\nabla f_{k_{\ell}+1}\|^{2}}{8(\omega^3L\|R\nabla f_{k_{\ell}}\|/2)^{1/2}} \ge \frac{\|R\nabla f_{k_\ell}\|^{3/2}}{128(\omega^3L/2)^{1/2}} \geq \frac{\sigma^{3/2}\|\nabla f_{k_\ell}\|^{3/2}} {128(\omega^3L/2)^{1/2}} . \end{aligned}

Since Ik⊆{0,…,k−1}I^k \subseteq\{0,\ldots,k-1\}, it follows that kℓ+1≤kℓ+1k_\ell+1\le k_{\ell+1} for ℓ=0,…,τ−1.\ell=0,\ldots,\tau-1. Moreover, by ineq:reduction-coarse, the sequence (f(xj))j∈N{(f(x_j))}_{j\in\mathbb{N}} is non-increasing. Hence,

f(xkℓ)−f(xkℓ+1)≥σ3/2128(ω3L/2)1/2∥∇fkℓ∥3/2.\begin{equation*} f(x_{k_{\ell}}) - f(x_{k_{\ell+1}}) \geq \frac{\sigma^{3/2}}{128(\omega^3L/2)^{1/2}}\|\nabla f_{k_\ell}\|^{3/2}. \end{equation*}

Combining this with convexity, i.e., f(xkℓ)−f∗≤⟨∇fkℓ,xkℓ−x∗⟩≤D∥∇fkℓ∥f(x_{k_\ell}) - f^* \leq \langle \nabla f_{k_\ell}, x_{k_\ell} - x^* \rangle \le D \| \nabla f_{k_\ell} \|, we obtain

f(xkℓ+1)−f∗≤f(xkℓ)−f∗−CC(f(xkℓ)−f∗)3/2,f(x_{k_{\ell+1}})-f^* \leq f(x_{k_\ell})-f^* -C_{\mathcal C}\bigl(f(x_{k_\ell})-f^*\bigr)^{3/2},

where

CC≔σ3/2128(D3ω3L/2)1/2.C_{\mathcal C} \coloneqq \frac{\sigma^{3/2}} {128(D^3\omega^3L/2)^{1/2}}.

Hence, prop:recurrence-rate yields

f(xk)−f∗≤1CC2(1+τ/2)2≤1CC2(1+k/4)2=O ⁣(1σ3k2).f(x_k)-f^* \leq \frac{1}{C_{\mathcal C}^2(1+\tau/2)^2} \leq \frac{1}{C_{\mathcal C}^2(1+k/4)^2} = \mathcal{O}\!\left(\frac{1}{\sigma^3k^2}\right).

Combining the two cases proves the result. ◻

prop:rate-coarse-steps analyzes the limiting case in which only coarse corrections are performed and shows that the coarse-only iteration retains the O(k−2)\mathcal{O}\big(k^{-2}\big) rate of the full-space regularized Newton method. Its proof adapts the strategy of [13 Theorem 1] to the reduced-gradient setting. A key feature of this analysis is that every iteration controls the same quantity, namely ∥R∇fk∥\|R\nabla f_k\|, so the corresponding estimates can be propagated consistently along the sequence.

The situation is fundamentally different for alg:two-level. Coarse iterations naturally control ∥R∇fk∥\|R\nabla f_k\|, whereas fine iterations control ∥∇fk∥\|\nabla f_k\|, and neither of these quantities is necessarily contractive when the algorithm switches to the other level. Thus, the fact that both the coarse-only and fine-only schemes achieve an O(k−2)\mathcal{O}\big(k^{-2}\big) rate does not by itself imply convergence, or the same rate, for an arbitrary mixture of the two iteration types. The main difficulty is therefore not the behavior within either level, but the transitions between them.

The following lemma provides the cross-level stability estimate needed to bridge this mismatch: although a coarse correction need not decrease the full gradient norm, it can increase it by at most a uniform factor.

Proof. Denote

αk:−ω3L∥R∇fk∥2.\begin{equation*} \alpha_k \coloneq \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}}. \end{equation*}

Next, we have

∇f(xk+1)−∇f(xk)=∫01∇2f(xk+t(xk+1−xk))(xk+1−xk) dt.\nabla f(x_{k+1})-\nabla f(x_k) = \int_0^1 \nabla^2 f\bigl(x_k+t(x_{k+1}-x_k)\bigr) (x_{k+1}-x_k)\,\mathrm{d}t.

By cor:f-nonincreasing, xk,xk+1∈L(x0)x_k,x_{k+1}\in\mathcal L(x_0). Since ff is convex, L(x0)\mathcal L(x_0) is convex, and hence xk+t(xk+1−xk)∈L(x0)x_k+t(x_{k+1}-x_k)\in\mathcal L(x_0) for every t∈[0,1]t\in[0,1]. Therefore,

∥∇f(xk+1)−∇f(xk)∥≤H∥xk+1−xk∥.\begin{equation} \|\nabla f(x_{k+1})-\nabla f(x_k)\| \le H \|x_{k+1}-x_k\|. \tag*{(7)} \end{equation}

Then, by using the definitions of ω\omega and αk\alpha_k as well as k∈Ck \in \mathcal{C} and the symmetric positive semi-definiteness of R∇2fkPR \nabla^2 f_kP, we obtain

∥∇fk+1∥≤∥∇fk+1−∇fk∥+∥∇fk∥≤H∥P(R∇2fkP+ω3L∥R∇fk∥2Id)−1R∇fk∥+∥∇fk∥≤Hω∥(R∇2fkP+ω3L∥R∇fk∥2Id)−1R∇fk∥+∥∇fk∥≤Hω∥R∇fk∥αk+∥∇fk∥≤Hω2ω3Lε∥R∇fk∥+∥∇fk∥≤(1+H2ωLε)∥∇fk∥.\begin{align*} \left\lVert \nabla f_{k+1} \right\rVert & \leq \left\lVert \nabla f_{k+1}- \nabla f_k \right\rVert + \left\lVert \nabla f_k \right\rVert \\ & \leq H \left\lVert P \left(R \nabla^2 f_kP + \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}} I_d\right)^{-1} R \nabla f_k \right\rVert + \left\lVert \nabla f_k \right\rVert \\ &\leq H \omega \left\lVert \left(R \nabla^2 f_kP + \sqrt{\frac{\omega^3 L \left\lVert R \nabla f_k \right\rVert}{2}} I_d\right)^{-1} R \nabla f_k \right\rVert + \left\lVert \nabla f_k \right\rVert \\ & \leq H \omega \frac{\left\lVert R \nabla f_k \right\rVert}{\alpha_k} + \left\lVert \nabla f_k \right\rVert \\ & \leq H \omega \frac{\sqrt{2}}{\sqrt{\omega^3 L \varepsilon}} \left\lVert R \nabla f_k \right\rVert + \left\lVert \nabla f_k \right\rVert \\ & \leq \left(1 + \frac{H \sqrt{2 \omega}}{\sqrt{L \varepsilon}}\right) \left\lVert \nabla f_k \right\rVert. \end{align*}

◻

lemma:bounded-gradient-in-coarse-level shows that a coarse correction is stable with respect to the full gradient, in the sense that the gradient norm can increase by at most a uniformly bounded factor. Together with the level-specific stability estimates ineq:reduction-coarse and ineq:reduction-fine, this bound will allow us to construct a level-dependent weighted gradient measure that controls transitions between the two levels.

Theorem 2.5. Let ff be convex and suppose that Assumptions 1.1, (?), and (?) hold. Then the sequence generated by Algorithm alg:two-level satisfies f(xk)−f∗=O(1k2).f(x_k)-f^* = \mathcal{O} \left(\frac{1}{k^2}\right).

Proof. To compare the reduced and full gradient norms across changes of level, we introduce a level-dependent weighted gradient measure. The weights are chosen so that all four possible coarse/fine transitions admit uniform contraction or stability bounds. Choose

a>0,ρ∈(0,σωMC),t∈(ρω,σMC),\begin{equation*} a > 0, \qquad \rho\in\left(0,\frac{\sigma}{\omega M_C}\right), \qquad t \in \left(\rho\omega ,\frac{\sigma}{M_C}\right), \end{equation*}

and set b≔atb \coloneqq a t. Since ∥R∥∥∇fk∥≥∥R∇fk∥≥σ∥∇fk∥\|R\|\|\nabla f_k\| \ge \|R\nabla f_k\| \ge \sigma \|\nabla f_k\| implies ω≥σ\omega \ge \sigma, we get

q≔max⁡{ρ,tMCσ,ωρt}<1.\begin{equation*} q \coloneqq \max \left\lbrace \rho, \frac{t M_\mathcal{C}}{\sigma}, \frac{\omega \rho}{t} \right\rbrace < 1. \end{equation*}

Moreover, define the scalar

B≔max⁡{2,2ωt,tMCσ},\begin{equation*} B \coloneqq \max \left\lbrace 2, \frac{2\omega}{t}, \frac{t M_\mathcal{C}}{\sigma} \right\rbrace, \end{equation*}

and the sets

Ck≔C∩{0,…,k−1},Fk≔F∩{0,…,k−1},ICk(ρ)≔{m∈C∩{0,…,k−1} : ∥R∇fm+1∥>ρ∥R∇fm∥},IFk(ρ)≔{m∈F∩{0,…,k−1} : ∥∇fm+1∥>ρ∥∇fm∥}.\begin{align*} \mathcal C^k & \coloneqq\mathcal C\cap\{0,\ldots,k-1\}, \qquad \mathcal F^k\coloneqq\mathcal F\cap\{0,\ldots,k-1\},\\ \mathcal I_{\mathcal{C}}^k(\rho) &\coloneqq \left\{ m\in\mathcal{C}\cap\{0,\ldots,k-1\} \, : \, \left\lVert R\nabla f_{m+1} \right\rVert> \rho\left\lVert R\nabla f_m \right\rVert \right\},\\ \mathcal I_{\mathcal{F}}^k(\rho) &\coloneqq \left\{ m\in\mathcal{F}\cap\{0,\ldots,k-1\} \, : \, \left\lVert \nabla f_{m+1} \right\rVert> \rho\left\lVert \nabla f_m \right\rVert \right\}. \end{align*}

We set

Ik(ρ)≔ICk(ρ)∪IFk(ρ).\mathcal I^k(\rho) \coloneqq \mathcal I_{\mathcal{C}}^k(\rho) \cup \mathcal I_{\mathcal{F}}^k(\rho).

Now, choose θ∈(0,−ln⁡qln⁡B−ln⁡q)\theta\in\left(0,\frac{-\ln q}{\ln B-\ln q}\right) and fix a k∈Nk \in \mathbb{N}. We distinguish two cases.

Case I: ∣Ik(ρ)∣≤θk\big|\mathcal{I}^k(\rho)\big| \leq \theta k. Define

Vk≔{a∥R∇fk∥,k∈C,b∥∇fk∥,k∈F,\begin{equation*} V_k \coloneqq \begin{cases} a\|R \nabla f_k\|, & k \in \mathcal{C},\\ b\|\nabla f_k\|,& k\in \mathcal{F}, \end{cases} \end{equation*}

For m∈(Ik(ρ))c≔{0,…,k−1}∖Ik(ρ)=(Ck∖ICk(ρ))∪(Fk∖IFk(ρ))m\in (\mathcal I^k(\rho))^{\rm c} \coloneqq \{0,\ldots, k-1 \} \setminus \mathcal{I}^k(\rho) = (\mathcal{C}^k \setminus \mathcal{I}_\mathcal{C}^k (\rho)) \cup (\mathcal{F}^k \setminus \mathcal{I}_\mathcal{F}^k(\rho)), we show that Vm+1≤qVmV_{m+1}\le qV_m. Let m∈Ik(ρ)cm \in \mathcal{I}^k(\rho)^{\rm c}. Then, for mm and m+1m+1 we have four possibilities:

  1. m∈Ck∖ICk(ρ), m+1∈Cm \in \mathcal{C}^k \setminus \mathcal{I}_\mathcal{C}^k (\rho), \ m+1 \in \mathcal{C}. Then,

    Vm+1=m+1∈Ca∥R∇fm+1∥≤m∈C∖ICk(ρ)aρ∥R∇fm∥=m∈CρVm.\begin{equation*} V_{m+1} \overset{m+1 \in \mathcal{C}}{=} a \left\lVert R \nabla f_{m+1} \right\rVert \overset{m \in \mathcal{C}\setminus \mathcal{I}_\mathcal{C}^k(\rho)}{\leq} a \rho \left\lVert R \nabla f_m \right\rVert \overset{m \in \mathcal{C}}{=} \rho V_m. \end{equation*}
  2. m∈Ck∖ICk(ρ), m+1∈Fm \in \mathcal{C}^k \setminus \mathcal{I}_\mathcal{C}^k (\rho), \ m+1 \in \mathcal{F}. Then, with similar arguments we obtain

    \begin{align*} V_{m+1} &\overset{m+1 \in \mathcal{F}}{=} b \left\lVert \nabla f_{m+1} \right\rVert \overset{\text{Lem.}~\ref{lemma:bounded-gradient-in-coarse-level}}{\leq} b M_\mathcal{C}\left\lVert \nabla f_{m} \right\rVert \\ &\quad \ {\leq} \frac{b M_\mathcal{C}}{\sigma} \left\lVert R\nabla f_{m} \right\rVert \overset{m \in \mathcal{C}}{\leq} \frac{t M_\mathcal{C}}{\sigma} V_m. \end{align*}
  3. m∈Fk∖IFk(ρ), m+1∈Fm \in \mathcal{F}^k \setminus \mathcal{I}_\mathcal{F}^k (\rho), \ m+1 \in \mathcal{F}. Then,

    Vm+1=m+1∈Fb∥∇fm+1∥≤m∈F∖IFk(ρ)bρ∥∇fm∥=m∈FρVm.\begin{equation*} V_{m+1} \overset{m+1 \in \mathcal{F}}{=} b \left\lVert \nabla f_{m+1} \right\rVert \overset{m \in \mathcal{F}\setminus \mathcal{I}_\mathcal{F}^k (\rho)}{\leq} b \rho \left\lVert \nabla f_{m} \right\rVert \overset{m \in \mathcal{F}}{=} \rho V_m. \end{equation*}
  4. m∈Fk∖IFk(ρ), m+1∈Cm \in \mathcal{F}^k \setminus \mathcal{I}_\mathcal{F}^k (\rho), \ m+1 \in \mathcal{C}. Then,

    Vm+1=m+1∈Ca∥R∇fm+1∥≤aω∥∇fm+1∥≤m∈F∖IFk(ρ)aωρ∥∇fm∥=m∈FωρtVm.\begin{equation*} V_{m+1} \overset{m+1 \in \mathcal{C}}{=} a \left\lVert R \nabla f_{m+1} \right\rVert \leq a \omega \left\lVert \nabla f_{m+1} \right\rVert \overset{m \in \mathcal{F}\setminus \mathcal{I}_\mathcal{F}^k (\rho)}{\leq} a \omega \rho \left\lVert \nabla f_{m} \right\rVert \overset{m \in \mathcal{F}}{=} \frac{\omega \rho}{t} V_m. \end{equation*}

Hence,

Vm+1≤qVm  for any  m∈Ik(ρ)c.\begin{equation} V_{m+1} \leq q V_m \ \ \text{for any} \ \ m \in \mathcal{I}^k(\rho)^{\rm c}. \tag*{(8)} \end{equation}

Now let m∈Ik(ρ)m \in \mathcal{I}^k(\rho). We again consider four cases as above:

  1. m∈ICk(ρ), m+1∈Cm \in \mathcal{I}_\mathcal{C}^k (\rho), \ m+1 \in \mathcal{C}. Then,

    \begin{equation*} V_{m+1} = a \left\lVert R \nabla f_{m+1} \right\rVert \overset{\eqref{ineq:reduced-grad-bound}}{\le} 2 V_m. \end{equation*}
  2. m∈ICk(ρ), m+1∈Fm \in \mathcal{I}_\mathcal{C}^k (\rho), \ m+1 \in \mathcal{F}. Then,

    Vm+1≤tMCσVm.\begin{equation*} V_{m+1} \leq \frac{t M_\mathcal{C}}{\sigma} V_m. \end{equation*}
  3. m∈IFk(ρ), m+1∈Fm \in \mathcal{I}_\mathcal{F}^k (\rho), \ m+1 \in \mathcal{F}. Then,

    \begin{equation*} V_{m+1} = b \left\lVert \nabla f_{m+1} \right\rVert \overset{\eqref{ineq:full-grad-bound}}{\le} 2 V_m. \end{equation*}
  4. m∈IFk(ρ), m+1∈Cm \in \mathcal{I}_\mathcal{F}^k (\rho), \ m+1 \in \mathcal{C}. Then,

    Vm+1≤2ωtVm.\begin{equation*} V_{m+1} \leq \frac{2 \omega}{t} V_m. \end{equation*}

As a result,

Vm+1≤BVm  for any  m∈Ik(ρ).\begin{equation} V_{m+1} \leq B V_m \ \ \text{for any} \ \ m \in \mathcal{I}^k(\rho). \tag*{(9)} \end{equation}

Let η≔q1−θBθ\eta \coloneqq q^{1-\theta} B^\theta. Thus, we have

ln⁡(q1−θBθ)=(1−θ)ln⁡q+θln⁡B=ln⁡q+θ(ln⁡B−ln⁡q),\begin{equation*} \ln\big(q^{1-\theta}B^\theta\big) = (1-\theta) \ln q + \theta \ln B = \ln q + \theta (\ln B- \ln q), \end{equation*}

and hence, by the definition of θ\theta, η<1\eta < 1. Combining now (8) and (9) we obtain

Vk≤qk−∣Ik(ρ)∣B∣Ik(ρ)∣V0≤ηkV0.\begin{align} V_k \leq q^{k - \left|\mathcal{I}^k(\rho)\right|} B^{\left|\mathcal{I}^k(\rho)\right|} V_0 \leq \eta^k V_0. \tag*{(10)} \end{align}

Furthermore, for any k∈Nk \in \mathbb{N}, we have that

∥∇fk∥≤AVk,A≔max⁡{1aσ,1b}.\begin{equation} \left\lVert \nabla f_k \right\rVert \leq A V_k, \quad A \coloneqq \max \left\lbrace \frac{1}{a \sigma}, \frac{1}{b} \right\rbrace. \tag*{(11)} \end{equation}

Then, by convexity and Assumption 1.1, we obtain

\begin{equation} f(x_k) - f^* \leq D \left\lVert \nabla f_k \right\rVert \overset{\eqref{ineq:V_k-final-bound}, \eqref{ineq:grad-V_k-relation}}{\leq} \eta^k A D V_0 %= \mathcal{O} \left(\eta^k\right) = \mathcal{O} \left(\frac{1}{k^2}\right), \tag*{(12)} \end{equation}

and thus the desired rate of convergence is established when ∣Ik(ρ)∣≤θk\big|\mathcal{I}^k(\rho)\big| \leq \theta k.

Case II: ∣Ik(ρ)∣>θk\big|\mathcal{I}^k(\rho)\big| > \theta k. Let τ≔∣Ik(ρ)∣\tau \coloneqq \big|\mathcal{I}^k(\rho)\big| and enumerate Ik(ρ)={k0,…kτ−1}\mathcal{I}^k(\rho) = \{k_0, \ldots k_{\tau-1}\}, with k0<k1<⋯<kτ−1<k, kτ≔kk_0<k_1<\cdots<k_{\tau-1} < k, \ k_\tau \coloneqq k. Let ℓ∈{0,…,τ−1}\ell \in \{0, \ldots, \tau - 1\}. Thus, kℓ∈Ik(ρ)k_\ell \in \mathcal{I}^k(\rho). If kℓ∈ICk(ρ)k_\ell \in \mathcal{I}^k_\mathcal{C}(\rho), then

\begin{align*} f(x_{k_\ell}) - f(x_{k_\ell + 1}) & \overset{\eqref{ineq:reduction-coarse}}{\geq} \frac{\|R \nabla f_{k_\ell + 1}\|^2}{8 \left( \omega^3 L / 2 \|R \nabla f_{k_\ell}\|\right)^{1/2}} \overset{k_\ell \in \mathcal{I}^k_\mathcal{C}(\rho)}{\geq} \frac{ \rho^2 \|R \nabla f_{k_\ell}\|^{3/2}}{8 \left( \omega^3 L / 2 \right)^{1/2}} \\ & \overset{k_\ell \in \mathcal{C}}{\geq} \left(\frac{\sigma}{\omega}\right)^{3/2}\frac{\rho^2 \| \nabla f_{k_\ell}\|^{3/2}}{8 \sqrt{L / 2} }. \end{align*}

Using similar arguments, if kℓ∈IFk(ρ)k_\ell \in \mathcal{I}^k_\mathcal{F}(\rho), then

f(xkℓ)−f(xkℓ+1)≥ρ2∥∇fkℓ∥3/26L/2,\begin{align*} f(x_{k_\ell}) - f(x_{k_\ell + 1}) \geq \frac{\rho^2 \left\lVert \nabla f_{k_\ell} \right\rVert^{3/2}}{6 \sqrt{L / 2}}, \end{align*}

and thus, since σ≤ω\sigma \leq \omega, we get

f(xkℓ)−f(xkℓ+1)≥min⁡{(σω)3/2ρ28L/2,ρ26L/2}∥∇fkℓ∥3/2=(σω)3/2ρ2∥∇fkℓ∥3/28L/2\begin{align*} f(x_{k_\ell}) - f(x_{k_\ell + 1}) & \geq \min \left\lbrace \left(\frac{\sigma}{\omega}\right)^{3/2}\frac{\rho^2 }{8 \sqrt{L / 2} }, \frac{\rho^2 }{6 \sqrt{L / 2}} \right\rbrace \| \nabla f_{k_\ell}\|^{3/2} \\ & = \left(\frac{\sigma}{\omega}\right)^{3/2}\frac{\rho^2 \| \nabla f_{k_\ell}\|^{3/2}}{8 \sqrt{L / 2} } \end{align*}

for all ℓ∈{0,…,τ−1}\ell \in \{0, \ldots, \tau - 1\}. Since Ik(ρ)⊆{0,…,k−1}I^k(\rho)\subseteq\{0,\ldots,k-1\}, it follows that kℓ+1≤kℓ+1k_\ell+1\le k_{\ell+1} for ℓ=0,…,τ−1.\ell=0,\ldots,\tau-1. Following the same argument as in Case II of prop:rate-coarse-steps, monotonicity of the function values and convexity yield

Δℓ+1≤Δℓ−CΔℓ3/2,Δℓ≔f(xkℓ)−f∗,\Delta_{\ell+1} \leq \Delta_\ell-C\Delta_\ell^{3/2}, \qquad \Delta_\ell\coloneqq f(x_{k_\ell})-f^*,

where

C≔2ρ28L(σωD)3/2.C\coloneqq \frac{\sqrt{2}\rho^2}{8\sqrt{L}} \left(\frac{\sigma}{\omega D}\right)^{3/2}.

Hence, prop:recurrence-rate and τ>θk\tau > \theta k give

f(xk)−f∗=Δτ≤1C2(1+τ/2)2≤1C2(1+θk/2)2=O(k−2),f(x_k)-f^* =\Delta_\tau \leq \frac{1}{C^2(1+\tau/2)^2} \leq \frac{1}{C^2(1+\theta k/2)^2} = \mathcal{O}\big(k^{-2}\big),

which completes the proof of Case II.

Combining now the results of the two cases we have

f(xk)−f∗≤max⁡{ADV0ηk,1C2(1+θk/2)2}=O(k−2),f(x_k)-f^* \leq \max \left\lbrace ADV_0 \eta^k, \dfrac{1}{C^2(1+\theta k/2)^2} \right\rbrace = \mathcal{O}\big(k^{-2}\big), % \begin{cases} % ADV_0\eta^k,& |\mathcal I^k(\rho)|\leq\theta k,\\[2mm] % \dfrac{1}{C^2(1+\theta k/2)^2}, % &|\mathcal I^k(\rho)|>\theta k, % \end{cases}

as required. ◻

Theorem 2.5 shows that the use of coarse corrections does not deteriorate the global convergence rate of the regularized Newton method. In particular, Algorithm alg:two-level achieves the O(k−2)\mathcal{O}\big(k^{-2}\big) rate in function values while allowing arbitrary switching between coarse and fine iterations according to the level-selection criterion. A key feature of the analysis is that coarse and fine iterations naturally control different quantities, namely ∥R∇fk∥\|R\nabla f_k\| and ∥∇fk∥\|\nabla f_k\|, respectively. The proof resolves this mismatch through a level-dependent weighted gradient measure that provides uniform control across all possible coarse/fine transitions: it contracts on iterations outside Ik(ρ)\mathcal I^k(\rho), while its growth on the remaining iterations is uniformly bounded. Consequently, the analysis does not require the full gradient norm to decrease at every coarse iteration; uniform stability of the coarse corrections is sufficient. This provides a mechanism for preserving convergence rates in multilevel schemes even when different iteration types control different measures of stationarity.

The multilevel Newton method with gradient regularization

In this section we consider the gradient regularization of Newton’s method that alternates between ℓ\ell coarse levels (plus the fine level) when generating the updates. In particular, we consider ℓ\ell restriction operators as in [1], but unlike their recursive framework, our operators map directly between each coarse level and the full space, allowing coarse corrections to be transferred directly to the fine level. More precisely, for each i∈J≔{1,…,ℓ}i \in \mathcal{J} \coloneqq \{1, \ldots, \ell \}, let Ri:Rn→RniR_i : \mathbb{R}^n \to \mathbb{R}^{n_i} be a restriction operator, where n1<⋯<nℓ<nn_1 < \cdots < n_\ell < n. Further, Pi≔Ri⊤P_i \coloneqq R^\top_i is the prolongation operator at level i∈Ji \in \mathcal{J}. We will see that this nonrecursive formulation simplifies both the algorithm and its convergence analysis while retaining the same O(ni2+ni3)\mathcal{O}\big(n_i^2+n_i^3\big) computational cost at the iith coarse level.

Accordingly, (?) is now replaced by the following assumption.

Assumption. For each i∈Ji\in\mathcal{J}, let Ri∈Rni×nR_i\in\mathbb{R}^{n_i\times n} be a full-row-rank restriction operator, with ni≤nn_i\leq n, and let Pi=Ri⊤P_i=R_i^\top be its corresponding prolongation operator. We define ωi≔∥Ri∥,ζ≔max⁡i∈Jωi.\omega_i\coloneqq\left\lVert R_i \right\rVert, \qquad \zeta\coloneqq\max_{i\in\mathcal{J}}\omega_i.

Let σ∈(0,min⁡{1,min⁡i∈Jωi})\sigma\in \left(0, \min\left\{1,\min_{i\in\mathcal J}\omega_i\right\}\right) and ε>0\varepsilon > 0. For each i∈Ji \in \mathcal{J}, the sets of coarse and fine iterations are now defined as follows. First define the eligible sets

Ei≔{k∈N : ∥Ri∇fk∥≥σ∥∇fk∥, ∥Ri∇fk∥≥ε}.\mathcal E_i \coloneqq \left\{ k\in\mathbb N \, : \, \|R_i\nabla f_k\|\ge\sigma\|\nabla f_k\|, \, \|R_i\nabla f_k\|\ge \varepsilon \right\}.

The set Ei\mathcal E_i contains the iterations for which level ii is eligible. For the multilevel method to become efficient, we need to select the lowest-dimensional eligible level

Ci:−Ei∖⋃j<iEj,i∈J.\begin{align*} \mathcal{C}_i \coloneq \mathcal{E}_i \setminus \bigcup_{j < i} \mathcal{E}_j, \qquad i \in \mathcal{J}. \end{align*}

Then the sets Ci\mathcal C_i are pairwise disjoint and, whenever {i∈J : k∈Ei}≠∅\{i\in\mathcal J \, : \, k\in\mathcal E_i\}\neq\emptyset,

k∈Ci⟺i=i∗(k)≔min⁡{j∈J : k∈Ej}.k\in\mathcal C_i \quad\Longleftrightarrow\quad i=i^*(k)\coloneqq \min\{j\in\mathcal J \, : \, k\in\mathcal E_j\}.

Moreover, we set

F(ℓ)≔N∖C(ℓ),C(ℓ)≔⋃i∈JCi.\begin{equation*} \mathcal{F}^{(\ell)} \coloneqq \mathbb{N}\setminus \mathcal{C}^{(\ell)}, \qquad \mathcal{C}^{(\ell)} \coloneqq \bigcup_{i \in \mathcal{J}} \mathcal{C}_i. \end{equation*}

The multilevel gradient-regularized Newton method is presented in alg:multilevel.

\caption{MultiLevel Regularized Newton (MLRN)}
\label{alg:multilevel}
\begin{algorithmic}
\REQUIRE $x_0 \in \mathbb{R}^n, \ell \in \mathbb{N}, R_i : \mathbb{R}^n \to \mathbb{R}^{n_i}$ for $i \in \mathcal{J}$, $\sigma\in \left(0, \min\left\{1,\min_{i\in\mathcal J}\omega_i\right\}\right), \varepsilon > 0$. 
\ENSURE Sequence of iterates $(x_0,x_1,x_2,\ldots)$.
\FOR{$k = 0, 1,  \ldots$}
\IF{$\nabla f_k = 0$} \STATE \textbf{stop} \ENDIF
\STATE Set \begin{equation*}
x_{k+1} \coloneqq
\begin{cases}
x_k - P_i
\left(
R_i \nabla^2 f_kP_i
+
\sqrt{\dfrac{\omega_i^3 L \left\lVert R_i \nabla f_k \right\rVert}{2}}\, I_{n_i}
\right)^{-1}
R_i \nabla f_k,
& k \in \mathcal{C}_i, i\in\mathcal{J}, \\[2ex]
x_k -
\left(
\nabla^2 f_k
+
\sqrt{\dfrac{L\left\lVert \nabla f_k \right\rVert}{2}}\, I_n
\right)^{-1}
\nabla f_k,
& k \in \mathcal{F}^{(\ell)}.
\end{cases}
\end{equation*}
\ENDFOR
\end{algorithmic}

The next lemma shows that alg:multilevel generates a non-increasing sequence of function values. It also shows that it guarantees the reduced-gradient stability of a coarse correction step. These results arise by [12].

Proof. The proof is outlined in § 6.1. ◻

lemma:multilevel-coarse-estimates guarantees that f(xk+1)≤f(xk), k∈Cif(x_{k+1})\le f(x_k), \ k\in \mathcal C_i, and by (6), (f(xk))k∈N{(f(x_k))}_{k \in \mathbb{N}} is non-increasing. In particular, this means that the multilevel iterates stay in L(x0)\mathcal{L}(x_0). Finally, in lemma:finite-coarse we show that alg:multilevel performs only finitely many coarse iterations.

Conditional rate of convergence

Let ρ∈(0,1)\rho \in (0,1). Similar to the analysis of the two-level method, we will need the following definitions,

Cik≔Ci∩{0,…,k−1},  C(ℓ),k≔⋃i∈JCik,F(ℓ),k≔F(ℓ)∩{0,…,k−1},ICik(ρ)≔{m∈Cik : ∥Ri∇fm+1∥>ρ∥Ri∇fm∥},IF(ℓ)k(ρ)≔{m∈F(ℓ),k : ∥∇fm+1∥>ρ∥∇fm∥},\begin{align*} \mathcal{C}_i^k &\coloneqq \mathcal{C}_i \cap \{0,\ldots ,k-1\}, \ \ \mathcal{C}^{(\ell), k} \coloneqq \bigcup_{i \in \mathcal{J}} \mathcal{C}_i^k, \\ \mathcal F^{(\ell), k} &\coloneqq \mathcal F^{(\ell)} \cap \{0,\ldots,k-1\}, \\ \mathcal I_{\mathcal{C}_i}^k(\rho) & \coloneqq \left\{ m\in\mathcal{C}^k_i \, : \, \left\lVert R_i \nabla f_{m+1} \right\rVert> \rho\left\lVert R_i \nabla f_m \right\rVert \right\}, \\ \mathcal I_{\mathcal{F}^{(\ell)}}^k(\rho) & \coloneqq \left\{ m\in\mathcal{F}^{(\ell), k} \, : \, \left\lVert \nabla f_{m+1} \right\rVert> \rho\left\lVert \nabla f_m \right\rVert \right\}, \end{align*}

where i∈Ji \in \mathcal{J}. We further set

Iℓk(ρ)≔(⋃i∈JICik(ρ))∪IF(ℓ)k(ρ).\mathcal{I}_\ell^k(\rho) \coloneqq \left( \bigcup_{i \in \mathcal{J}} I_{\mathcal{C}_i}^k(\rho) \right) \cup \mathcal{I}_{\mathcal{F}^{(\ell)}}^k(\rho).

Hence, the disjointness of the sets Ci\mathcal{C}_i, i∈Ji \in \mathcal{J}, and F(ℓ)\mathcal{F}^{(\ell)} gives

∣Iℓk(ρ)∣=∑i=1ℓ∣ICik(ρ)∣+∣IF(ℓ)k(ρ)∣.\begin{equation*} \big|\mathcal{I}_\ell^k(\rho)\big| = \sum_{i=1}^\ell \big| I_{\mathcal{C}_i}^k(\rho)\big| + \big|\mathcal{I}_{\mathcal{F}^{(\ell)}}^k(\rho)\big|. \end{equation*}

The following lemma shows the O(k−2)\mathcal{O}\big(k^{-2}\big) rate of alg:multilevel when ∣Iℓk(ρ)∣\big|\mathcal{I}_\ell^k(\rho)\big| is sufficiently large.

Proof. The proof is given in § 6.2. ◻

Further, the coarse correction is also stable with respect to the full gradient. In particular, by lemma:bounded-gradient-in-coarse-level there exist MCi∈(0,∞)M_{\mathcal{C}_i} \in (0, \infty), i∈Ji \in \mathcal{J}, such that

∥∇fk+1∥≤MCi∥∇fk∥,k∈Ci.\begin{equation*} \left\lVert \nabla f_{k+1} \right\rVert \leq M_{\mathcal{C}_i} \left\lVert \nabla f_k \right\rVert, \qquad k \in \mathcal{C}_i. \end{equation*}

In what follows we show that alg:multilevel converges with O(k−2)\mathcal{O}\big(k^{-2}\big) when it performs a sufficiently large numbersufficiently of fine steps.

Theorem 2.5. Suppose that Assumptions 1.1, (?), and (?) hold. Let MC(ℓ)≔max⁡{1,max⁡i∈JMCi},h≔log⁡2MC(ℓ).M_{\mathcal{C}^{(\ell)}} \coloneqq \max\left\{1,\max_{i\in\mathcal{J}}M_{\mathcal{C}_i}\right\}, \qquad h\coloneqq\log_2 M_{\mathcal{C}^{(\ell)}}. Let θ∈(0,2/3)\theta\in(0,2/3) and choose a∈(h+3θh+2,1).a\in \left( \frac{h+3\theta}{h+2},1 \right). Suppose also that, for all sufficiently large kk, ∣F(ℓ),k∣≥ak.\big|\mathcal{F}^{(\ell),k}\big|\geq ak. Then the sequence generated by alg:multilevel satisfies f(xk)−f∗=O(k−2).f(x_k)-f^*=\mathcal O\big(k^{-2}\big).

Proof. Fix a sufficiently large k∈Nk\in\mathbb{N} such that ∣F(ℓ),k∣≥ak\big|\mathcal{F}^{(\ell),k}\big|\geq ak, and set Iℓk≔Iℓk(1/4)\mathcal{I}_\ell^k \coloneqq \mathcal{I}_\ell^k(1/4), IF(ℓ)k≔IF(ℓ)k(1/4).\mathcal{I}_{\mathcal{F}^{(\ell)}}^k \coloneqq \mathcal{I}_{\mathcal{F}^{(\ell)}}^k(1/4). If ∣Iℓk∣>θk\big|\mathcal{I}_\ell^k\big|>\theta k, then lemma:case-II yields

f(xk)−f∗=O(k−2).f(x_k)-f^*=\mathcal O\big(k^{-2}\big).

It remains to consider the case ∣Iℓk∣≤θk.|\mathcal{I}_\ell^k|\leq\theta k. For every m∈{0,…,k−1}m\in\{0,\ldots,k-1\}, exactly one of the following possibilities occurs:

  1. If m∈C(ℓ),km\in\mathcal{C}^{(\ell),k}, then ∥∇fm+1∥≤MC(ℓ)∥∇fm∥.\|\nabla f_{m+1}\| \leq M_{\mathcal{C}^{(\ell)}}\|\nabla f_m\|.

  2. If m∈IF(ℓ)km\in\mathcal{I}_{\mathcal{F}^{(\ell)}}^k, then, by (6), ∥∇fm+1∥≤2∥∇fm∥.\|\nabla f_{m+1}\| \leq 2\|\nabla f_m\|.

  3. If m∈F(ℓ),k∖IF(ℓ)km\in\mathcal{F}^{(\ell),k}\setminus\mathcal{I}_{\mathcal{F}^{(\ell)}}^k, then ∥∇fm+1∥≤14∥∇fm∥.\|\nabla f_{m+1}\| \leq \frac14\|\nabla f_m\|.

Since ∣C(ℓ),k∣+∣F(ℓ),k∣=k,\big|\mathcal{C}^{(\ell),k}\big|+\big|\mathcal{F}^{(\ell),k}\big|=k, iterating these bounds gives

∥∇fk∥≤MC(ℓ) k−∣F(ℓ),k∣2∣IF(ℓ)k∣(14)∣F(ℓ),k∣−∣IF(ℓ)k∣∥∇f0∥=MC(ℓ) k−∣F(ℓ),k∣2−2∣F(ℓ),k∣+3∣IF(ℓ)k∣∥∇f0∥.\begin{aligned} \|\nabla f_k\| &\leq M_{\mathcal{C}^{(\ell)}}^{\,k-\left|\mathcal{F}^{(\ell),k}\right|} 2^{\left|\mathcal{I}_{\mathcal{F}^{(\ell)}}^k\right|} \left(\frac14\right)^{ \left|\mathcal{F}^{(\ell),k}\right|-\left|\mathcal{I}_{\mathcal{F}^{(\ell)}}^k\right|} \|\nabla f_0\|\\ &= M_{\mathcal{C}^{(\ell)}}^{\,k-\left|\mathcal{F}^{(\ell),k}\right|} 2^{-2\left|\mathcal{F}^{(\ell),k}\right| +3\left|\mathcal{I}_{\mathcal{F}^{(\ell)}}^k\right|} \|\nabla f_0\|. \end{aligned}

Since ∣F(ℓ),k∣≥ak and ∣IF(ℓ)k∣≤∣Iℓk∣≤θk,\big|\mathcal{F}^{(\ell),k}\big|\geq ak \ \text{and} \ \big|\mathcal{I}_{\mathcal{F}^{(\ell)}}^k\big| \leq \big|\mathcal{I}_\ell^k\big| \leq\theta k, we obtain

∥∇fk∥≤βk∥∇f0∥,\|\nabla f_k\| \leq \beta^k\|\nabla f_0\|,

where

β≔MC(ℓ)1−a2−2a+3θ.\beta \coloneqq M_{\mathcal{C}^{(\ell)}}^{1-a}2^{-2a+3\theta}.

Moreover,

log⁡2β=h+3θ−a(h+2)<0\log_2\beta = h+3\theta-a(h+2)<0

by the choice of aa, and hence β<1\beta<1.

Therefore, by convexity and Assumption 1.1,

f(xk)−f∗≤D∥∇fk∥≤Dβk∥∇f0∥=O(βk)=O(k−2).f(x_k)-f^* \leq D\|\nabla f_k\| \leq D\beta^k\|\nabla f_0\| = \mathcal O\big(\beta^k\big) = \mathcal O\big(k^{-2}\big).

Combining the two cases proves the result. ◻

Theorem 2.5 provides a quantitative condition under which the multilevel scheme retains the convergence rate of the underlying full-space method. Although this rate need not be inherited by an arbitrary multilevel realization, the theorem shows that the O(k−2)\mathcal O\big(k^{-2}\big) rate is preserved whenever a sufficiently large fraction of the iterations is performed on the fine level. Such preservation of complexity has also been established for other multilevel high-order regularization frameworks; see, e.g., [15] for the multilevel ARC method and possibly non-convex functions. To the best of our knowledge, Theorem 2.5 is the first result showing that a classical multilevel gradient-regularized Newton method can retain the global O(k−2)\mathcal O\big(k^{-2}\big) function value rate of the full-space method. This is possible to achieve under an explicit quantitative condition on the proportion of fine-level iterations.

We emphasize that the fine-iteration frequency condition is not restrictive. Indeed, the eligibility condition ∥Ri∇fk∥≥ε\left\lVert R_i \nabla f_k \right\rVert \ge \varepsilon implies that only finitely many coarse iterations can occur, see lemma:finite-coarse for details. Consequently, the algorithm eventually performs exclusively fine-level iterations, and the required frequency condition is automatically satisfied for all sufficiently large kk. Nevertheless, this condition is not computationally irrelevant. For instance, for large kk and θ=0.1\theta = 0.1, MC(ℓ)=2M_{\mathcal C^{(\ell)}}=2 and MC(ℓ)=2000M_{\mathcal C^{(\ell)}}=2000 yield approximately a>0.44a>0.44 and a>0.87a>0.87, respectively. This indicates that arbitrarily long sequences of coarse corrections cannot not occur before the algorithm enters the fine-only regime. In § 3.3 we develop analysis establishing an explicit convergence bound valid at every iteration, without relying on the eventual transition to the fine-only regime, and which also allows the method to perform arbitrarily long sequences of coarse corrections.

Row-nested restriction operators

In the previous subsection, we established an O(k−2)\mathcal O\big(k^{-2}\big) convergence rate under the condition that a sufficiently large fraction of the iterations is performed on the fine level. Such a condition may lead to a large number of computationally expensive fine-level iterations before alg:multilevel enters its fine-level regime. Here we introduce an additional structure on the restriction operators and discuss its most important properties. This new structure will, later, allow us to show a global O(k−2)\mathcal O\big(k^{-2}\big) rate for alg:multilevel without this condition.

Nested hierarchies of coarse spaces are standard in multigrid and have also been employed in multilevel optimization; see, e.g., [11, 17]. Here we impose a more specific algebraic structure: all restriction operators act on the same fine space, and the rows defining a lower-dimensional restriction are retained at every subsequent level. This row-nested construction yields the monotonicity ∥Riv∥≤∥Rjv∥\|R_i v\|\le \|R_jv\| for i<ji<j, which will play a central role in our convergence analysis.

For every i<ji<j there exists R^ij∈R(nj−ni)×n\widehat R_{ij}\in\mathbb{R}^{(n_j-n_i)\times n} such that

Rj=[RiR^ij].R_j = \begin{bmatrix} R_i\\ \widehat R_{ij} \end{bmatrix}.

Consequently, for every v∈Rnv\in\mathbb{R}^n,

∥Rjv∥2=∥Riv∥2+∥R^ijv∥2,\|R_jv\|^2 = \|R_iv\|^2+\big\|\widehat R_{ij}v\big\|^2,

and therefore,

∥Riv∥≤∥Rjv∥,1≤i<j≤ℓ.\begin{equation} \|R_iv\|\leq\|R_jv\|, \qquad 1\leq i<j\leq\ell. \tag*{(13)} \end{equation}

Further, def:nested-restrictions implies nestedness in multilevel spaces. By the definition of RjR_j we have

Pj=Rj⊤=[PiR^ij⊤].P_j=R_j^\top = \begin{bmatrix} P_i & \widehat R_{ij}^{\top} \end{bmatrix}.

Therefore, since ni<njn_i<n_j,

range⁡(Pi)⊂range⁡(Pj),i<j,\begin{equation} \operatorname{range}(P_i) \subset \operatorname{range}(P_j), \qquad i<j, \tag*{(14)} \end{equation}

and thus, as in classical multilevel methods with nested subspaces, each successive level enriches the previous coarse correction space with additional directions. However, the stronger row-nested structure additionally implies the norm monotonicity (13). There also is a corresponding reverse nesting of null spaces, namely

ker⁡(Rj)⊂ker⁡(Ri),i<j,\ker(R_j)\subset\ker(R_i), \qquad i<j,

where the inclusion is strict since dim⁡ker⁡(Rj)=n−nj<n−ni=dim⁡ker⁡(Ri)\dim\ker(R_j)=n-n_j<n-n_i=\dim\ker(R_i). Thus, larger coarse levels miss fewer directions of the fine-space gradient.

def:nested-restrictions yields properties that significantly simplify the multilevel algorithm and provides us with further insights. In particular,

ω1≤ω2≤⋯≤ωℓ=ζ,\omega_1\leq\omega_2\leq\cdots\leq\omega_\ell=\zeta,

and the admissible range for σ\sigma simplifies to σ∈(0,min⁡{1,ω1}).\sigma\in(0,\min\{1,\omega_1\}). Moreover, (13) implies that the eligible sets satisfy

E1⊆E2⊆⋯⊆Eℓ,⋃j<iEj=Ei−1,i=2,…,ℓ.\mathcal E_1\subseteq\mathcal E_2 \subseteq\cdots\subseteq\mathcal E_\ell, \qquad \bigcup_{j<i}\mathcal E_j=\mathcal E_{i-1}, \quad i = 2, \ldots, \ell.

Hence,

C1=E1,Ci=Ei∖Ei−1,i=2,…,ℓ,\mathcal C_1=\mathcal E_1, \qquad \mathcal C_i = \mathcal E_i\setminus\mathcal E_{i-1}, \quad i=2,\ldots,\ell,

and,

C(ℓ)=⋃i=1ℓCi=Eℓ,F(ℓ)=N∖Eℓ.\mathcal C^{(\ell)} = \bigcup_{i=1}^{\ell}\mathcal C_i = \mathcal E_\ell, \qquad \mathcal F^{(\ell)} = \mathbb N\setminus\mathcal E_\ell.

Therefore, k∈Cik\in\mathcal C_i means that level ii is the lowest-dimensional level satisfying the eligibility criterion at iteration kk. Every level j>ij>i is then automatically eligible, whereas no level j<ij<i is.

Convergence analysis using the row-nested restriction operators

We now establish an O(k−2)\mathcal O\big(k^{-2}\big) convergence rate for alg:multilevel under the row-nested structure of def:nested-restrictions, without imposing any condition on the frequency of fine-level iterations. The analysis extends the weighted-gradient argument developed for the two-level method. An additional difficulty arises from transitions between different coarse levels, since the corresponding updates control distinct reduced-gradient norms ∥Ri∇fk∥\|R_i\nabla f_k\|. For general restriction operators, these quantities need not be comparable across levels. The row-nested structure provides the necessary norm comparisons, which, together with suitable level-dependent weights, allow us to control transitions among all coarse and fine levels.

Theorem 3.8. Let f be convex and suppose that Assumptions 1.1 and (?) hold. If the sequence (xk)k∈N{(x_k)}_{k \in \mathbb{N}} is generated by alg:multilevel with row-nested operators for which (?) hold, then f(xk)−f∗=O(k−2). f(x_k) - f^* = \mathcal{O}\big(k^{-2}\big).

Proof. The proof makes use of the set Iℓk(ρ)\mathcal{I}_\ell^k(\rho) and as previously we split the analysis into two cases, according to the magnitude of Iℓk(ρ)\mathcal{I}_\ell^k(\rho).

Recall that, due to the row-nestedness, ζ=ωℓ\zeta = \omega_\ell, and define ωˉ≔max⁡{1,ωℓ}\bar{\omega} \coloneqq \max\{1, \omega_\ell \}. We choose

θ∈(0,11+ℓ),r∈(0,min⁡{1Λ,B}),ρ∈(0,Λωˉr1+ℓ),\begin{align*} \theta \in \left(0, \frac{1}{1+\ell}\right), \qquad r \in \left(0, \min \left\lbrace \frac{1}{\Lambda}, B \right\rbrace \right), \qquad \rho \in \left(0, \frac{\Lambda}{\bar{\omega}} r^{1+\ell} \right), \end{align*}

where

Λ≔ωˉMC(ℓ)σ,B≔exp⁡(−(1−θ)ln⁡(Λ)+θln⁡(2ωˉ)1−(1+ℓ)θ),\begin{equation*} \Lambda \coloneqq \frac{\bar{\omega} M_{\mathcal{C}^{(\ell)}}}{\sigma}, \qquad B \coloneqq \exp \left( - \frac{(1-\theta) \ln(\Lambda) + \theta \ln(2\bar{\omega})}{1- (1+\ell)\theta}\right), \end{equation*}

and note that Λ≥1\Lambda \geq 1 and B,r,ρ∈(0,1)B, r, \rho \in (0, 1). Moreover, for i∈Ji \in \mathcal{J}, we define

ai≔ri−1,b≔rℓ,a_i \coloneqq r^{i-1}, \qquad b \coloneqq r^\ell,

therefore 1=a1>⋯>aℓ>01=a_1 > \cdots > a_\ell>0, and note also that ajai=rj−i>1\frac{a_j}{a_i} = r^{j-i} > 1, if j<ij<i, and rj−i<1r^{j-i} < 1, if j>ij>i.

Let us fix a k∈Nk \in \mathbb{N}. If ∣Iℓk(ρ)∣>θk\big|\mathcal{I}_\ell^k(\rho)\big| > \theta k, then lemma:case-II provides the desired rate of convergence. Assume now that ∣Iℓk(ρ)∣≤θk\big|\mathcal{I}_\ell^k(\rho)\big| \leq \theta k. Define the weight gradient measure for ℓ\ell coarse levels

Vk≔{ai∥Ri∇fk∥,k∈Ci, i∈J,b∥∇fk∥,k∈F(ℓ).\begin{equation*} V_k \coloneqq \begin{cases} a_i \|R_i \nabla f_k\|, & k \in \mathcal{C}_i, \ i \in \mathcal{J},\\ b\|\nabla f_k\|,& k\in \mathcal{F}^{(\ell)}. \end{cases} \end{equation*}

Thus, by the definition of aia_i and VkV_k,

∥∇fk∥≤max⁡{1a1σ,…,1aℓσ,1b}Vk=max⁡{1aℓσ,1b}Vk.\begin{equation} \left\lVert \nabla f_k \right\rVert \leq \max \left\lbrace \frac{1}{a_1 \sigma}, \ldots, \frac{1}{a_\ell \sigma}, \frac{1}{b} \right\rbrace V_k = \max \left\lbrace \frac{1}{a_\ell \sigma}, \frac{1}{b} \right\rbrace V_k. \tag*{(15)} \end{equation}

Next, it holds

(Iℓk(ρ))c≔{0,1,…,k−1}∖Iℓk(ρ)=(⋃i∈JCik∖ICik(ρ))∪(F(ℓ),k∖IF(ℓ)k(ρ)).\begin{equation*} \big(\mathcal{I}_\ell^k(\rho)\big)^{\rm c} \coloneqq \{0, 1, \ldots, k-1\} \setminus \mathcal{I}_\ell^k(\rho) = \left(\bigcup_{i \in \mathcal{J}} C_i^k \setminus \mathcal{I}_{\mathcal{C}_i}^{k} (\rho) \right) \cup \left(\mathcal{F}^{(\ell), k} \setminus \mathcal{I}_{\mathcal{F}^{(\ell)} }^{k} (\rho) \right). \end{equation*}

We first examine the case m∈(Iℓk(ρ))cm \in (\mathcal{I}_\ell^k(\rho))^{\rm c}, where, in particular, we will show that VkV_k is a contraction. To achieve this, in comparison to the proof of Theorem 2.5, we need to examine an extra case which concerns the transitions between the ℓ\ell coarse levels:

  1. m∈Cik∖ICikm \in \mathcal{C}_i^k \setminus \mathcal{I}_{\mathcal{C}_i}^{k}, m+1∈Cjm+1 \in \mathcal{C}_j. We analyze this case by splitting into additional three subcases according to the ordering of the coarse levels:

    1. j<ij < i. Then, by the row-nestedness (13), we have

      Vm+1=aj∥Rj∇fm+1∥≤aj∥Ri∇fm+1∥  ≤k∉ICikρaj∥Ri∇fm∥=rj−iρVk≤r1−ℓρVk,\begin{align*} & V_{m+1} = a_j\left\lVert R_j \nabla f_{m+1} \right\rVert \leq a_j\left\lVert R_i \nabla f_{m+1} \right\rVert \\ & \quad \ \ \overset{k \notin \mathcal{I}_{\mathcal{C}_i}^{k}}{\leq} \rho a_j \left\lVert R_i \nabla f_{m} \right\rVert = r^{j-i} \rho V_k \leq r^{1-\ell} \rho V_k, \end{align*}

      where the last inequality uses the fact that x↦rxx \mapsto r^x is non-increasing for r∈(0,1)r \in (0,1).

    2. i<ji < j. Then,

      Vm+1=aj∥Rj∇fm+1∥≤ajωˉ∥∇fm+1∥≤ajωˉMC(ℓ)∥∇fm∥≤k∈CikΛrj−iVm≤ΛrVm,\begin{align*} V_{m+1} & = a_j\left\lVert R_j \nabla f_{m+1} \right\rVert \leq a_j \bar{\omega} \left\lVert \nabla f_{m+1} \right\rVert \\ & \leq a_j \bar{\omega} M_{\mathcal{C}^{(\ell)}}\left\lVert \nabla f_{m} \right\rVert \overset{k \in \mathcal{C}_i^k}{\leq} \Lambda r^{j-i} V_m \leq \Lambda r V_m, \end{align*}

      where we have use the uniform stability of the coarse corrections, i.e., ∥∇fk+1∥≤MC(ℓ)∥∇fk∥\left\lVert \nabla f_{k+1} \right\rVert \leq M_{\mathcal{C}^{(\ell)}} \left\lVert \nabla f_k \right\rVert with MC(ℓ)M_{\mathcal{C}^{(\ell)}} defined in Theorem 2.5.

    3. i=ji=j. Then we immediately have

      Vm+1≤ρVm.\begin{equation*} V_{m+1} \leq \rho V_m. \end{equation*}

    Combining the above subcases yields

    Vm+1≤max⁡{ρ,Λr,r1−ℓρ}Vm=ΛrVm,\begin{equation*} V_{m+1} \leq \max \left\lbrace \rho, \Lambda r, r^{1-\ell} \rho \right\rbrace V_m = \Lambda r V_m, \end{equation*}

    where the last equality hold by the choice on ρ\rho. This concludes the first case of arbitrary transitions between coarse spaces.

  2. m∈Cik∖ICik,m+1∈F(ℓ).m \in \mathcal{C}_i^k \setminus \mathcal{I}_{\mathcal{C}_i}^{k}, m+1 \in \mathcal{F}^{(\ell)}. Then

    Vm+1≤b∥∇fm+1∥≤bMC(ℓ)∥∇fm∥≤k∈Cikrℓ−i+1MC(ℓ)σ∥∇fm∥≤ΛrVm.\begin{align*} V_{m+1} \leq b \left\lVert \nabla f_{m+1} \right\rVert \leq b M_{\mathcal{C}^{(\ell)}} \left\lVert \nabla f_{m} \right\rVert %& \quad \ \ \overset{k \in \mathcal{C}_i^k}{\leq} r^{\ell - i + 1} \frac{ M_{\mathcal{C}^{(\ell)}}}{\sigma} \left\lVert \nabla f_{m} \right\rVert \leq \Lambda r V_m. \end{align*}
  3. m∈F(ℓ),k∖IF(ℓ)k(ρ),m+1∈Cim \in \mathcal{F}^{(\ell), k} \setminus \mathcal{I}_{\mathcal{F}^{(\ell)} }^{k} (\rho), m+1 \in \mathcal{C}_i. Then,

    Vm+1=ai∥Ri∇fm+1∥≤aiωˉρ∥∇fm∥=ωˉρrℓ−i+1Vm≤ωˉρrℓVm.\begin{align*} V_{m+1} = a_i \left\lVert R_i \nabla f_{m+1} \right\rVert \leq a_i \bar{\omega} \rho \left\lVert \nabla f_{m} \right\rVert = \frac{\bar{\omega} \rho}{r^{\ell-i+1}} V_m \leq \frac{\bar{\omega} \rho}{r^{\ell}} V_m. \end{align*}
  4. m∈F(ℓ),k∖IF(ℓ)k(ρ),m+1∈F(ℓ)m \in \mathcal{F}^{(\ell), k} \setminus \mathcal{I}_{\mathcal{F}^{(\ell)} }^{k} (\rho), m+1 \in \mathcal{F}^{(\ell)}, then we directly obtain

    Vm+1≤ρVm.\begin{equation*} V_{m+1} \leq \rho V_m. \end{equation*}

Therefore, by the definition of ρ\rho,

Vm+1≤{ρ,Λr,ωˉρrℓ}Vm=ΛrVm<Vm,m∈(Iℓk(ρ))c,\begin{equation} V_{m+1} \leq \left\lbrace \rho, \Lambda r, \frac{\bar{\omega} \rho}{r^{\ell}} \right\rbrace V_m = \Lambda r V_m < V_m, \qquad m \in \big(\mathcal{I}_\ell^k(\rho)\big)^{\rm c}, \tag*{(16)} \end{equation}

which shows that VkV_k is a contraction for elements of (Iℓk(ρ))c(\mathcal{I}_\ell^k(\rho))^{\rm c}.

Assume now that m∈Iℓk(ρ)m \in \mathcal{I}_\ell^k(\rho). Similarly, we split our arguments into four cases, to obtain a recurrent relation for VkV_k. However, since Iℓk(ρ)\mathcal{I}_\ell^k(\rho) contains a few elements only, VkV_k is not required to be a contraction.

  1. m∈Cik,m+1∈Cjm \in C_i^k, m+1 \in C_j. Then,

    Vm+1=aj∥Rj∇fm+1∥≤ajωˉMC(ℓ)∥∇fm∥≤Λrj−iVm≤r−ℓVm,\begin{equation*} V_{m+1} = a_j \left\lVert R_j \nabla f_{m+1} \right\rVert \leq a_j \bar{\omega} M_{\mathcal{C}^{(\ell)}} \left\lVert \nabla f_{m} \right\rVert \leq \Lambda r^{j-i} V_m \leq r^{-\ell} V_m, \end{equation*}

    where the last inequality uses the monotonicity of x↦rxx \mapsto r^x, r∈(0,1)r \in (0,1).

  2. m∈Cik,m+1∈F(ℓ)m \in C_i^k, m+1 \in \mathcal{F}^{(\ell)}. Then,

    Vm+1≤rℓMC(ℓ)∥∇fm∥≤rℓ−i+1ΛVm≤Vm.\begin{align*} V_{m+1} \leq r^{\ell} M_{\mathcal{C}^{(\ell)}} \left\lVert \nabla f_m \right\rVert \leq r^{\ell - i + 1} \Lambda V_m \leq V_m. \end{align*}
  3. m,∈F(ℓ),k,m+1∈F(ℓ)m, \in \mathcal{F}^{(\ell), k}, m+1 \in \mathcal{F}^{(\ell)}. Then, by (6),

    Vm+1≤2Vm.\begin{equation*} V_{m+1} \leq 2 V_m. \end{equation*}
  4. m∈F(ℓ),k,m+1∈Cim \in \mathcal{F}^{(\ell), k}, m+1 \in \mathcal{C}_i. Then,

    Vm+1≤ai∥Ri∇fm+1∥≤2aiωˉ∥∇fm∥≤2r−ℓωˉVm.\begin{equation*} V_{m+1} \leq a_i \left\lVert R_i \nabla f_{m+1} \right\rVert \leq 2 a_i \bar{\omega} \left\lVert \nabla f_{m} \right\rVert \leq 2 r^{-\ell} \bar{\omega} V_m. \end{equation*}

As a result, the above four cases imply

Vm+1≤2ωˉr−ℓVm,m∈Iℓk(ρ).\begin{equation} V_{m+1} \leq 2 \bar{\omega} r^{-\ell} V_m, \qquad m \in \mathcal{I}_\ell^k(\rho). \tag*{(17)} \end{equation}

Let us denote

η≔r1−(1+ℓ)θΛ1−θ(2ωˉ)θ.\begin{equation*} \eta \coloneqq r^{1 - (1+\ell) \theta} \Lambda^{1-\theta} (2\bar{\omega})^\theta. \end{equation*}

By the choices of θ\theta and ρ\rho it easy to see that r1−(1+ℓ)θ<1r^{1 - (1+\ell) \theta} < 1 and since r<Br < B we have η<1\eta < 1. Hence, by (16) and (17), we obtain

Vk≤(Λr)k−∣Iℓk(ρ)∣(2r−ℓωˉ)∣Iℓk(ρ)∣V0≤∣Iℓk(ρ)∣≤θkηkV0,\begin{align*} V_k \leq \left(\Lambda r \right)^{k - \left|\mathcal{I}_\ell^k(\rho)\right|} \left(2 r^{-\ell} \bar{\omega}\right)^{\left|\mathcal{I}_\ell^k(\rho)\right|} V_0 \overset{\left|\mathcal{I}_\ell^k(\rho)\right| \le \theta k}{\leq} \eta^k V_0, \end{align*}

which shows that VkV_k is a contraction. Now the desired result follows from the last inequality, convexity, and (15) as follows. Let

Aℓ=max⁡{1σrℓ−1,1rℓ}.A_\ell= \max\left\{\frac{1}{\sigma r^{\ell-1}},\frac1{r^\ell} \right\}.

Then

f(xk)−f∗≤D∥∇fk∥≤DAℓVk≤DAℓV0ηk\begin{align*} f(x_k) - f^* \leq D \left\lVert \nabla f_k \right\rVert \leq D A_\ell V_k \leq D A_\ell V_0 \eta^k \end{align*}

which shows exponential rate for the case ∣Iℓk(ρ)∣≤θk\big|\mathcal{I}_\ell^k(\rho)\big| \leq \theta k. Hence the two cases imply

f(xk)−f∗≤max⁡{DAℓV0ηk, 1Cℓ2(1+θk/2)2},f(x_k)-f^* \leq \max\left\{ D A_\ell V_0\eta^k,\, \frac{1}{C_\ell^2(1+\theta k/2)^2} \right\},

which concludes the proof. ◻

Theorem 3.8 establishes a direct O(k−2)\mathcal{O}\big(k^{-2}\big) convergence bound for the multilevel method with row-nested restriction operators, without imposing any condition on the frequency of fine-level iterations. Although the asymptotic rate also follows for general restriction operators from the eventual fine-only behavior, the present analysis controls the convergence rate throughout the entire sequence, including coarse-dominated phases. In particular, the resulting bound does not depend on the iteration at which the method enters the fine-only regime.

The main difficulty for achieving this result is that a coarse correction at level ii controls ∥Ri∇fk+1∥\|R_i\nabla f_{k+1}\|, but not necessarily the reduced gradient at the subsequently selected level. Row-nestedness resolves this mismatch through monotonicity of the reduced-gradient norms, which, together with level-dependent weights and uniform full-gradient stability, allows us to control arbitrary transitions between levels. The resulting weighted gradient measure contracts outside a set of non-ρ\rho-contracting iterations, while its growth remains uniformly bounded otherwise. When non-ρ\rho-contracting iterations are sufficiently frequent, the function-value decrease estimates instead yield the O(k−2)\mathcal{O}\big(k^{-2}\big) rate. Thus, the analysis accommodates arbitrarily long sequences of coarse corrections without requiring a prescribed frequency of fine iterations or a decrease of the full gradient at every coarse step.

Experiments

leg:logsum

\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=483,
        ymin=3.114e-11, ymax=9.808e+00,
        ytickten={-10,-8,-6,-4,-2,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$},
        legend to name=leg:logsum]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 1 2.47479 \\ 2 2.19928 \\ 3 1.98376 \\ 4 1.79968 \\ 5 1.62039 \\
      6 1.44752 \\ 7 1.26373 \\ 8 1.15339 \\ 9 1.00203 \\ 10 0.912407 \\ 11 0.830229 \\
      12 0.780814 \\ 13 0.694941 \\ 14 0.648902 \\ 15 0.602882 \\ 16 0.577572 \\ 17 0.519541 \\
      18 0.495368 \\ 19 0.460214 \\ 20 0.444787 \\ 21 0.416234 \\ 22 0.403257 \\ 23 0.385474 \\
      24 0.379241 \\ 25 0.370261 \\ 26 0.353854 \\ 27 0.346997 \\ 28 0.33437 \\ 29 0.328108 \\
      30 0.316451 \\ 31 0.312097 \\ 32 0.305135 \\ 33 0.297026 \\ 34 0.29301 \\ 35 0.287836 \\
      36 0.280994 \\ 37 0.277909 \\ 38 0.273738 \\ 39 0.268201 \\ 40 0.264626 \\ 41 0.260734 \\
      42 0.258182 \\ 43 0.254717 \\ 44 0.250655 \\ 45 0.247916 \\ 46 0.245113 \\ 47 0.242819 \\
      48 0.240288 \\ 49 0.237976 \\ 50 0.235205 \\ 51 0.232127 \\ 52 0.230043 \\ 53 0.228007 \\
      54 0.207965 \\ 55 0.205501 \\ 56 0.201717 \\ 57 0.200045 \\ 58 0.198764 \\ 59 0.196233 \\
      60 0.194468 \\ 61 0.192817 \\ 62 0.191417 \\ 63 0.190035 \\ 64 0.174654 \\ 65 0.172679 \\
      66 0.169392 \\ 67 0.168424 \\ 68 0.166709 \\ 69 0.164867 \\ 70 0.163728 \\ 71 0.162761 \\
      72 0.161016 \\ 73 0.159865 \\ 74 0.15875 \\ 75 0.157827 \\ 76 0.156896 \\ 77 0.155412 \\
      78 0.154409 \\ 79 0.153468 \\ 80 0.141548 \\ 81 0.140002 \\ 82 0.137064 \\ 83 0.136318 \\
      84 0.134977 \\ 85 0.13359 \\ 86 0.132705 \\ 87 0.132028 \\ 88 0.130804 \\ 89 0.129946 \\
      90 0.12916 \\ 91 0.128506 \\ 92 0.127859 \\ 93 0.126852 \\ 94 0.126144 \\ 95 0.125497 \\
      96 0.117108 \\ 97 0.116067 \\ 98 0.113885 \\ 99 0.113086 \\ 100 0.112536 \\ 101 0.111542 \\
      102 0.110854 \\ 103 0.110237 \\ 104 0.109709 \\ 105 0.109241 \\ 106 0.108465 \\ 107 0.10789 \\
      108 0.107378 \\ 109 0.106932 \\ 110 0.10651 \\ 111 0.106034 \\ 112 0.105452 \\ 113 0.104894 \\
      114 0.0982914 \\ 115 0.0973747 \\ 116 0.095532 \\ 117 0.0949204 \\ 118 0.0944805 \\ 119 0.0937065 \\
      120 0.0931906 \\ 121 0.0927493 \\ 122 0.0923735 \\ 123 0.0919879 \\ 124 0.0914189 \\ 125 0.0910079 \\
      126 0.0906601 \\ 127 0.0858601 \\ 128 0.0851369 \\ 129 0.0835472 \\ 130 0.083206 \\ 131 0.082597 \\
      132 0.0820216 \\ 133 0.0816437 \\ 134 0.081373 \\ 135 0.0808834 \\ 136 0.0805168 \\ 137 0.0802061 \\
      138 0.0799318 \\ 139 0.0796579 \\ 140 0.0792521 \\ 141 0.0789488 \\ 142 0.0786921 \\ 143 0.0750417 \\
      144 0.0744425 \\ 145 0.0730551 \\ 146 0.072768 \\ 147 0.0722212 \\ 148 0.0717229 \\ 149 0.0713873 \\
      150 0.0711528 \\ 151 0.0707287 \\ 152 0.0704119 \\ 153 0.0701432 \\ 154 0.0699064 \\ 155 0.0696657 \\
      156 0.0693151 \\ 157 0.0690542 \\ 158 0.0688336 \\ 159 0.0659578 \\ 160 0.0654489 \\ 161 0.0643576 \\
      162 0.0641027 \\ 163 0.0636522 \\ 164 0.0632351 \\ 165 0.0629552 \\ 166 0.0627503 \\ 167 0.0623944 \\
      168 0.0621294 \\ 169 0.0619074 \\ 170 0.0617107 \\ 171 0.0614992 \\ 172 0.0612061 \\ 173 0.0609884 \\
      174 0.0608058 \\ 175 0.0606402 \\ 176 0.0604377 \\ 177 0.0601745 \\ 178 0.0599808 \\ 179 0.0598182 \\
      180 0.057461 \\ 181 0.0570283 \\ 182 0.0561403 \\ 183 0.0559303 \\ 184 0.0555632 \\ 185 0.0552332 \\
      186 0.0550864 \\ 187 0.0548166 \\ 188 0.0545559 \\ 189 0.0543675 \\ 190 0.0542064 \\ 191 0.0540492 \\
      192 0.053836 \\ 193 0.0536315 \\ 194 0.0534839 \\ 195 0.0533537 \\ 196 0.0531988 \\ 197 0.0530113 \\
      198 0.0528289 \\ 199 0.0512099 \\ 200 0.0508777 \\ 201 0.0502265 \\ 202 0.0500613 \\ 203 0.049776 \\
      204 0.049518 \\ 205 0.0493422 \\ 206 0.0492081 \\ 207 0.048983 \\ 208 0.0488127 \\ 209 0.0486703 \\
      210 0.0485435 \\ 211 0.0484059 \\ 212 0.0482159 \\ 213 0.0480725 \\ 214 0.0479513 \\ 215 0.0478414 \\
      216 0.0477102 \\ 217 0.0475368 \\ 218 0.0474072 \\ 219 0.0472973 \\ 220 0.0460273 \\ 221 0.0457478 \\
      222 0.0452269 \\ 223 0.045087 \\ 224 0.044847 \\ 225 0.0446333 \\ 226 0.0444855 \\ 227 0.0443695 \\
      228 0.0441794 \\ 229 0.044033 \\ 230 0.0439106 \\ 231 0.0438016 \\ 232 0.0436826 \\ 233 0.0435194 \\
      234 0.0433951 \\ 235 0.0432901 \\ 236 0.0431953 \\ 237 0.0430814 \\ 238 0.0429321 \\ 239 0.0428195 \\
      240 0.0427245 \\ 241 0.0416482 \\ 242 0.0413876 \\ 243 0.0409363 \\ 244 0.0408141 \\ 245 0.0406019 \\
      246 0.0404157 \\ 247 0.0402869 \\ 248 0.0401844 \\ 249 0.04002 \\ 250 0.0398927 \\ 251 0.0397869 \\
      252 0.039693 \\ 253 0.0395889 \\ 254 0.0394483 \\ 255 0.039341 \\ 256 0.0392511 \\ 257 0.0391703 \\
      258 0.0390712 \\ 259 0.0389432 \\ 260 0.0388471 \\ 261 0.0387667 \\ 262 0.0379002 \\ 263 0.0376777 \\
      264 0.0373165 \\ 265 0.0372169 \\ 266 0.0370386 \\ 267 0.0368846 \\ 268 0.036778 \\ 269 0.0366907 \\
      270 0.036554 \\ 271 0.0364476 \\ 272 0.0363599 \\ 273 0.0362824 \\ 274 0.0361934 \\ 275 0.0360755 \\
      276 0.0359862 \\ 277 0.035912 \\ 278 0.0352463 \\ 279 0.0350498 \\ 280 0.0347437 \\ 281 0.0346556 \\
      282 0.0344951 \\ 283 0.0343575 \\ 284 0.0342622 \\ 285 0.0341829 \\ 286 0.0340596 \\ 287 0.0339638 \\
      288 0.0338849 \\ 289 0.0338155 \\ 290 0.0337345 \\ 291 0.0336275 \\ 292 0.0335469 \\ 293 0.0334802 \\
      294 0.0328925 \\ 295 0.0327146 \\ 296 0.03244 \\ 297 0.0323594 \\ 298 0.0322117 \\ 299 0.0320856 \\
      300 0.0319985 \\ 301 0.0319248 \\ 302 0.0318113 \\ 303 0.0317234 \\ 304 0.0316636 \\ 305 0.0315619 \\
      306 0.0314758 \\ 307 0.0314059 \\ 308 0.0313448 \\ 309 0.0312766 \\ 310 0.0311838 \\ 311 0.0311125 \\
      312 0.0310536 \\ 313 0.0304766 \\ 314 0.0303118 \\ 315 0.030054 \\ 316 0.0299818 \\ 317 0.0298472 \\
      318 0.0297323 \\ 319 0.0296534 \\ 320 0.0295869 \\ 321 0.0294848 \\ 322 0.0294054 \\ 323 0.0293404 \\
      324 0.0292836 \\ 325 0.0292154 \\ 326 0.0291265 \\ 327 0.02906 \\ 328 0.0290053 \\ 329 0.0284955 \\
      330 0.028343 \\ 331 0.0281047 \\ 332 0.0280365 \\ 333 0.0279122 \\ 334 0.0278044 \\ 335 0.0277309 \\
      336 0.0276693 \\ 337 0.027574 \\ 338 0.0275002 \\ 339 0.0274398 \\ 340 0.0273873 \\ 341 0.0273241 \\
      342 0.0272413 \\ 343 0.0271797 \\ 344 0.0271292 \\ 345 0.026667 \\ 346 0.0265283 \\ 347 0.0263118 \\
      348 0.0262488 \\ 349 0.026136 \\ 350 0.0260357 \\ 351 0.0259678 \\ 352 0.0259115 \\ 353 0.0258234 \\
      354 0.0257554 \\ 355 0.0257097 \\ 356 0.025631 \\ 357 0.0255649 \\ 358 0.0255116 \\ 359 0.0254656 \\
      360 0.025413 \\ 361 0.0253418 \\ 362 0.0252878 \\ 363 0.0252436 \\ 364 0.0248023 \\ 365 0.0246768 \\
      366 0.0244786 \\ 367 0.0244232 \\ 368 0.0243238 \\ 369 0.0242336 \\ 370 0.024173 \\ 371 0.0241236 \\
      372 0.0240457 \\ 373 0.0239855 \\ 374 0.0239452 \\ 375 0.0238757 \\ 376 0.0238172 \\ 377 0.0237702 \\
      378 0.0237297 \\ 379 0.0236831 \\ 380 0.0236206 \\ 381 0.023573 \\ 382 0.023534 \\ 383 0.0231362 \\
      384 0.0230243 \\ 385 0.0228459 \\ 386 0.0227964 \\ 387 0.0227092 \\ 388 0.0226287 \\ 389 0.0225751 \\
      390 0.0225314 \\ 391 0.0224625 \\ 392 0.0224095 \\ 393 0.0223736 \\ 394 0.0223124 \\ 395 0.0222608 \\
      396 0.0222193 \\ 397 0.0221837 \\ 398 0.0221423 \\ 399 0.0220874 \\ 400 0.0220454 \\ 401 0.022011 \\
      402 0.0216523 \\ 403 0.0215525 \\ 404 0.0213914 \\ 405 0.0213463 \\ 406 0.0212694 \\ 407 0.0211971 \\
      408 0.0211495 \\ 409 0.0211104 \\ 410 0.0210492 \\ 411 0.0210019 \\ 412 0.0209699 \\ 413 0.0209154 \\
      414 0.0208695 \\ 415 0.0208325 \\ 416 0.0208006 \\ 417 0.0207637 \\ 418 0.020715 \\ 419 0.0206773 \\
      420 0.0206465 \\ 421 0.0203185 \\ 422 0.0202281 \\ 423 0.0200812 \\ 424 0.0200394 \\ 425 0.0199704 \\
      426 0.0199048 \\ 427 0.0198617 \\ 428 0.0198264 \\ 429 0.019771 \\ 430 0.0197283 \\ 431 0.0196993 \\
      432 0.0196501 \\ 433 0.0196085 \\ 434 0.0195749 \\ 435 0.0195459 \\ 436 0.0195127 \\ 437 0.0194687 \\
      438 0.0194344 \\ 439 0.0194063 \\ 440 0.0191033 \\ 441 0.0190194 \\ 442 0.018883 \\ 443 0.0188437 \\
      444 0.0187805 \\ 445 0.01872 \\ 446 0.0186804 \\ 447 0.018648 \\ 448 0.0185971 \\ 449 0.0185579 \\
      450 0.0185315 \\ 451 0.0184863 \\ 452 0.0184482 \\ 453 0.0184173 \\ 454 0.0183907 \\ 455 0.0183605 \\
      456 0.0183202 \\ 457 0.0182887 \\ 458 0.0182629 \\ 459 0.017984 \\ 460 0.0179067 \\ 461 0.017782 \\
      462 0.0177454 \\ 463 0.0176875 \\ 464 0.017632 \\ 465 0.0176062 \\ 466 0.017562 \\ 467 0.0175159 \\
      468 0.0174842 \\ 469 0.0174587 \\
    };
    \addlegendentry{MLRN}
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 1 2.46526 \\ 2 2.1985 \\ 3 1.97428 \\ 4 1.73697 \\ 5 1.53124 \\
      6 1.38611 \\ 7 1.2029 \\ 8 1.06726 \\ 9 0.994471 \\ 10 0.874889 \\ 11 0.834332 \\
      12 0.733917 \\ 13 0.709088 \\ 14 0.659855 \\ 15 0.603971 \\ 16 0.57969 \\ 17 0.538684 \\
      18 0.509148 \\ 19 0.44321 \\ 20 0.4296 \\ 21 0.399961 \\ 22 0.340424 \\ 23 0.335883 \\
      24 0.3112 \\ 25 0.290646 \\ 26 0.250232 \\ 27 0.247406 \\ 28 0.238011 \\ 29 0.209505 \\
      30 0.205976 \\ 31 0.185121 \\ 32 0.182899 \\ 33 0.166412 \\ 34 0.147913 \\ 35 0.145957 \\
      36 0.134107 \\ 37 0.115648 \\ 38 0.115086 \\ 39 0.11155 \\ 40 0.1009 \\ 41 0.0985478 \\
      42 0.0844256 \\ 43 0.083977 \\ 44 0.0834993 \\ 45 0.0819649 \\ 46 0.0762798 \\ 47 0.0644385 \\
      48 0.0641516 \\ 49 0.0629924 \\ 50 0.0593333 \\ 51 0.0508897 \\ 52 0.0507937 \\ 53 0.0501947 \\
      54 0.0469941 \\ 55 0.037968 \\ 56 0.0378513 \\ 57 0.0377134 \\ 58 0.0373469 \\ 59 0.035375 \\
      60 0.0286797 \\ 61 0.0286326 \\ 62 0.0283249 \\ 63 0.0267126 \\ 64 0.0210442 \\ 65 0.0209662 \\
      66 0.0208736 \\ 67 0.0206686 \\ 68 0.0197053 \\ 69 0.014662 \\ 70 0.0146256 \\ 71 0.0144432 \\
      72 0.0137748 \\ 73 0.0107971 \\ 74 0.010786 \\ 75 0.010691 \\ 76 0.0101635 \\ 77 0.0079016 \\
      78 0.00788349 \\ 79 0.00785899 \\ 80 0.00778959 \\ 81 0.00740045 \\ 82 0.00538395 \\ 83 0.00537666 \\
      84 0.00536279 \\ 85 0.00532486 \\ 86 0.00510866 \\ 87 0.00384158 \\ 88 0.00383467 \\ 89 0.00380387 \\
      90 0.00367851 \\ 91 0.00268108 \\ 92 0.00267917 \\ 93 0.0026759 \\ 94 0.00266464 \\ 95 0.00259045 \\
      96 0.00148949 \\ 97 0.00148712 \\ 98 0.00148401 \\ 99 0.00147489 \\ 100 0.00141753 \\ 101 0.000852665 \\
      102 0.000851499 \\ 103 0.000850173 \\ 104 0.000845003 \\ 105 0.000814938 \\ 106 0.000478148 \\ 107 0.000477811 \\
      108 0.000477075 \\ 109 0.000474219 \\ 110 0.000458419 \\ 111 0.000211634 \\ 112 0.000211361 \\ 113 0.000210817 \\
      114 0.000209296 \\ 115 0.000202466 \\ 116 9.82836e-05 \\ 117 9.82288e-05 \\ 118 9.81058e-05 \\ 119 9.77538e-05 \\
      120 9.5332e-05 \\ 121 2.89025e-05 \\ 122 2.88794e-05 \\ 123 2.88289e-05 \\ 124 2.87075e-05 \\ 125 2.80048e-05 \\
      126 4.87929e-06 \\ 127 4.87506e-06 \\ 128 4.86781e-06 \\ 129 4.84936e-06 \\ 130 4.73983e-06 \\ 131 4.4787e-07 \\
      132 4.47372e-07 \\ 133 4.4653e-07 \\ 134 4.44172e-07 \\ 135 4.30307e-07 \\ 136 1.61916e-09 \\
    };
    \addlegendentry{MLRN nested}
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 1 2.44985 \\ 2 2.05165 \\ 3 1.77977 \\ 4 1.56477 \\ 5 1.33745 \\
      6 1.16218 \\ 7 0.997177 \\ 8 0.857844 \\ 9 0.753039 \\ 10 0.647931 \\ 11 0.576747 \\
      12 0.495326 \\ 13 0.427393 \\ 14 0.369959 \\ 15 0.323661 \\ 16 0.27943 \\ 17 0.240469 \\
      18 0.207472 \\ 19 0.17288 \\ 20 0.146722 \\ 21 0.123346 \\ 22 0.108173 \\ 23 0.096226 \\
      24 0.0878709 \\ 25 0.0783986 \\ 26 0.0664209 \\ 27 0.0555386 \\ 28 0.0500955 \\ 29 0.0438219 \\
      30 0.0365067 \\ 31 0.0318277 \\ 32 0.0261896 \\ 33 0.0212425 \\ 34 0.0165991 \\ 35 0.0138685 \\
      36 0.0109732 \\ 37 0.00829399 \\ 38 0.00622675 \\ 39 0.00466276 \\ 40 0.0038435 \\ 41 0.00303281 \\
      42 0.00209377 \\ 43 0.00146101 \\ 44 0.000982211 \\ 45 0.000569945 \\ 46 0.000341941 \\ 47 0.00017826 \\
      48 8.27995e-05 \\ 49 3.43078e-05 \\ 50 6.88103e-06 \\ 51 7.97022e-07 \\ 52 2.26362e-08 \\ 53 1.03754e-10 \\
    };
    \addlegendentry{RN}
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 1 2.47303 \\ 2 2.14783 \\ 3 1.88345 \\ 4 1.63515 \\ 5 1.42657 \\
      6 1.25229 \\ 7 1.122 \\ 8 0.999022 \\ 9 0.898615 \\ 10 0.824901 \\ 11 0.763266 \\
      12 0.718185 \\ 13 0.625943 \\ 14 0.60037 \\ 15 0.529623 \\ 16 0.511027 \\ 17 0.450868 \\
      18 0.435377 \\ 19 0.373664 \\ 20 0.361335 \\ 21 0.303631 \\ 22 0.293115 \\ 23 0.252309 \\
      24 0.245907 \\ 25 0.213493 \\ 26 0.209782 \\ 27 0.177384 \\ 28 0.17418 \\ 29 0.148079 \\
      30 0.145967 \\ 31 0.126371 \\ 32 0.125065 \\ 33 0.105038 \\ 34 0.103879 \\ 35 0.0885395 \\
      36 0.087781 \\ 37 0.0757871 \\ 38 0.0752292 \\ 39 0.0658458 \\ 40 0.0653782 \\ 41 0.0549365 \\
      42 0.0546214 \\ 43 0.0486912 \\ 44 0.0484931 \\ 45 0.0416366 \\ 46 0.041484 \\ 47 0.0359402 \\
      48 0.0357737 \\ 49 0.0295853 \\ 50 0.0294481 \\ 51 0.0227541 \\ 52 0.0226243 \\ 53 0.019725 \\
      54 0.0196709 \\ 55 0.0150221 \\ 56 0.0149674 \\ 57 0.012443 \\ 58 0.0124046 \\ 59 0.0103272 \\
      60 0.0103015 \\ 61 0.00811095 \\ 62 0.0080831 \\ 63 0.00575928 \\ 64 0.00573436 \\ 65 0.00418805 \\
      66 0.00416435 \\ 67 0.00298684 \\ 68 0.00298008 \\ 69 0.00181965 \\ 70 0.00181395 \\ 71 0.00121712 \\
      72 0.00121431 \\ 73 0.000781969 \\ 74 0.000780655 \\ 75 0.000378816 \\ 76 0.000377358 \\ 77 0.000184322 \\
      78 0.000183912 \\ 79 8.93031e-05 \\ 80 8.91374e-05 \\ 81 2.87246e-05 \\ 82 2.86859e-05 \\ 83 4.68364e-06 \\
      84 4.6685e-06 \\ 85 3.26239e-07 \\ 86 3.25308e-07 \\ 87 2.69317e-09 \\
    };
    \addlegendentry{TLRN}
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(a) $\gamma=0.05$, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=4.11,
        ymin=3.114e-11, ymax=9.808e+00,
        ytickten={-10,-8,-6,-4,-2,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 0.00788021 2.47479 \\ 0.0137869 2.19928 \\ 0.0195872 1.98376 \\ 0.0258607 1.79968 \\ 0.0316682 1.62039 \\
      0.0378602 1.44752 \\ 0.047111 1.26373 \\ 0.054443 1.15339 \\ 0.0633059 1.00203 \\ 0.0697139 0.912407 \\ 0.080536 0.830229 \\
      0.0877426 0.780814 \\ 0.097041 0.694941 \\ 0.104484 0.648902 \\ 0.117546 0.602882 \\ 0.12484 0.577572 \\ 0.134012 0.519541 \\
      0.141512 0.495368 \\ 0.154973 0.460214 \\ 0.16221 0.444787 \\ 0.172183 0.416234 \\ 0.17855 0.403257 \\ 0.189805 0.385474 \\
      0.193602 0.379241 \\ 0.199791 0.370261 \\ 0.210114 0.353854 \\ 0.215652 0.346997 \\ 0.224116 0.33437 \\ 0.230774 0.328108 \\
      0.239647 0.316451 \\ 0.243273 0.312097 \\ 0.252134 0.305135 \\ 0.264321 0.297026 \\ 0.268882 0.29301 \\ 0.27636 0.287836 \\
      0.289167 0.280994 \\ 0.293544 0.277909 \\ 0.300047 0.273738 \\ 0.312399 0.268201 \\ 0.319564 0.264626 \\ 0.328591 0.260734 \\
      0.332829 0.258182 \\ 0.341563 0.254717 \\ 0.35778 0.250655 \\ 0.365549 0.247916 \\ 0.374643 0.245113 \\ 0.381627 0.242819 \\
      0.39091 0.240288 \\ 0.394246 0.237976 \\ 0.401038 0.235205 \\ 0.414042 0.232127 \\ 0.420582 0.230043 \\ 0.428931 0.228007 \\
      0.451561 0.207965 \\ 0.457535 0.205501 \\ 0.467088 0.201717 \\ 0.474002 0.200045 \\ 0.478011 0.198764 \\ 0.486246 0.196233 \\
      0.493935 0.194468 \\ 0.504035 0.192817 \\ 0.511305 0.191417 \\ 0.521469 0.190035 \\ 0.545069 0.174654 \\ 0.552459 0.172679 \\
      0.560833 0.169392 \\ 0.564031 0.168424 \\ 0.571924 0.166709 \\ 0.580486 0.164867 \\ 0.587757 0.163728 \\ 0.591087 0.162761 \\
      0.601334 0.161016 \\ 0.608168 0.159865 \\ 0.616845 0.15875 \\ 0.623522 0.157827 \\ 0.627636 0.156896 \\ 0.639327 0.155412 \\
      0.645948 0.154409 \\ 0.654548 0.153468 \\ 0.67461 0.141548 \\ 0.68135 0.140002 \\ 0.690456 0.137064 \\ 0.694459 0.136318 \\
      0.700913 0.134977 \\ 0.710009 0.13359 \\ 0.717574 0.132705 \\ 0.721461 0.132028 \\ 0.733385 0.130804 \\ 0.74059 0.129946 \\
      0.750973 0.12916 \\ 0.758009 0.128506 \\ 0.762443 0.127859 \\ 0.773647 0.126852 \\ 0.782229 0.126144 \\ 0.792609 0.125497 \\
      0.815125 0.117108 \\ 0.822275 0.116067 \\ 0.82987 0.113885 \\ 0.836022 0.113086 \\ 0.839606 0.112536 \\ 0.847454 0.111542 \\
      0.85349 0.110854 \\ 0.861427 0.110237 \\ 0.867924 0.109709 \\ 0.871215 0.109241 \\ 0.881863 0.108465 \\ 0.888284 0.10789 \\
      0.897796 0.107378 \\ 0.904273 0.106932 \\ 0.9154 0.10651 \\ 0.919946 0.106034 \\ 0.927629 0.105452 \\ 0.938502 0.104894 \\
      0.967948 0.0982914 \\ 0.974349 0.0973747 \\ 0.983312 0.095532 \\ 0.990883 0.0949204 \\ 0.994788 0.0944805 \\ 1.00333 0.0937065 \\
      1.01022 0.0931906 \\ 1.0186 0.0927493 \\ 1.02598 0.0923735 \\ 1.03016 0.0919879 \\ 1.04141 0.0914189 \\ 1.04803 0.0910079 \\
      1.05845 0.0906601 \\ 1.07939 0.0858601 \\ 1.08644 0.0851369 \\ 1.09544 0.0835472 \\ 1.09869 0.083206 \\ 1.10684 0.082597 \\
      1.11577 0.0820216 \\ 1.12539 0.0816437 \\ 1.13211 0.081373 \\ 1.14954 0.0808834 \\ 1.1566 0.0805168 \\ 1.16893 0.0802061 \\
      1.1766 0.0799318 \\ 1.17975 0.0796579 \\ 1.19275 0.0792521 \\ 1.19985 0.0789488 \\ 1.20985 0.0786921 \\ 1.22885 0.0750417 \\
      1.23589 0.0744425 \\ 1.24382 0.0730551 \\ 1.2473 0.072768 \\ 1.25399 0.0722212 \\ 1.26216 0.0717229 \\ 1.268 0.0713873 \\
      1.27164 0.0711528 \\ 1.28199 0.0707287 \\ 1.28886 0.0704119 \\ 1.29998 0.0701432 \\ 1.30682 0.0699064 \\ 1.31061 0.0696657 \\
      1.32174 0.0693151 \\ 1.32863 0.0690542 \\ 1.33947 0.0688336 \\ 1.36275 0.0659578 \\ 1.37087 0.0654489 \\ 1.37955 0.0643576 \\
      1.38421 0.0641027 \\ 1.39152 0.0636522 \\ 1.39985 0.0632351 \\ 1.40612 0.0629552 \\ 1.40997 0.0627503 \\ 1.42062 0.0623944 \\
      1.42635 0.0621294 \\ 1.43607 0.0619074 \\ 1.44232 0.0617107 \\ 1.44578 0.0614992 \\ 1.45706 0.0612061 \\ 1.46356 0.0609884 \\
      1.47377 0.0608058 \\ 1.48023 0.0606402 \\ 1.48418 0.0604377 \\ 1.495 0.0601745 \\ 1.50133 0.0599808 \\ 1.51118 0.0598182 \\
      1.53456 0.057461 \\ 1.54234 0.0570283 \\ 1.55103 0.0561403 \\ 1.55557 0.0559303 \\ 1.56224 0.0555632 \\ 1.5714 0.0552332 \\
      1.57583 0.0550864 \\ 1.58344 0.0548166 \\ 1.59366 0.0545559 \\ 1.6006 0.0543675 \\ 1.60998 0.0542064 \\ 1.61345 0.0540492 \\
      1.6239 0.053836 \\ 1.63317 0.0536315 \\ 1.63946 0.0534839 \\ 1.64887 0.0533537 \\ 1.65327 0.0531988 \\ 1.66044 0.0530113 \\
      1.66956 0.0528289 \\ 1.69223 0.0512099 \\ 1.69926 0.0508777 \\ 1.70869 0.0502265 \\ 1.7129 0.0500613 \\ 1.71988 0.049776 \\
      1.72889 0.049518 \\ 1.73686 0.0493422 \\ 1.74194 0.0492081 \\ 1.75528 0.048983 \\ 1.76646 0.0488127 \\ 1.77919 0.0486703 \\
      1.78648 0.0485435 \\ 1.79066 0.0484059 \\ 1.80346 0.0482159 \\ 1.81051 0.0480725 \\ 1.82091 0.0479513 \\ 1.8275 0.0478414 \\
      1.83113 0.0477102 \\ 1.84385 0.0475368 \\ 1.85075 0.0474072 \\ 1.86073 0.0472973 \\ 1.88477 0.0460273 \\ 1.89146 0.0457478 \\
      1.89934 0.0452269 \\ 1.90431 0.045087 \\ 1.91149 0.044847 \\ 1.9204 0.0446333 \\ 1.92787 0.0444855 \\ 1.93234 0.0443695 \\
      1.94383 0.0441794 \\ 1.95071 0.044033 \\ 1.96077 0.0439106 \\ 1.96748 0.0438016 \\ 1.97177 0.0436826 \\ 1.98487 0.0435194 \\
      1.99168 0.0433951 \\ 2.00229 0.0432901 \\ 2.01025 0.0431953 \\ 2.01367 0.0430814 \\ 2.02639 0.0429321 \\ 2.03245 0.0428195 \\
      2.0441 0.0427245 \\ 2.06784 0.0416482 \\ 2.07561 0.0413876 \\ 2.08478 0.0409363 \\ 2.08844 0.0408141 \\ 2.09475 0.0406019 \\
      2.10336 0.0404157 \\ 2.11006 0.0402869 \\ 2.11433 0.0401844 \\ 2.12654 0.04002 \\ 2.13305 0.0398927 \\ 2.14332 0.0397869 \\
      2.15076 0.039693 \\ 2.15493 0.0395889 \\ 2.16715 0.0394483 \\ 2.1752 0.039341 \\ 2.18653 0.0392511 \\ 2.19458 0.0391703 \\
      2.19783 0.0390712 \\ 2.21029 0.0389432 \\ 2.21687 0.0388471 \\ 2.22685 0.0387667 \\ 2.25174 0.0379002 \\ 2.25834 0.0376777 \\
      2.26659 0.0373165 \\ 2.27035 0.0372169 \\ 2.27746 0.0370386 \\ 2.28626 0.0368846 \\ 2.2935 0.036778 \\ 2.29714 0.0366907 \\
      2.30858 0.036554 \\ 2.31577 0.0364476 \\ 2.32549 0.0363599 \\ 2.33258 0.0362824 \\ 2.33794 0.0361934 \\ 2.35136 0.0360755 \\
      2.35932 0.0359862 \\ 2.36962 0.035912 \\ 2.39361 0.0352463 \\ 2.40129 0.0350498 \\ 2.41018 0.0347437 \\ 2.41409 0.0346556 \\
      2.42209 0.0344951 \\ 2.4301 0.0343575 \\ 2.43652 0.0342622 \\ 2.44089 0.0341829 \\ 2.45226 0.0340596 \\ 2.458 0.0339638 \\
      2.4685 0.0338849 \\ 2.47445 0.0338155 \\ 2.47846 0.0337345 \\ 2.49035 0.0336275 \\ 2.49629 0.0335469 \\ 2.50525 0.0334802 \\
      2.52702 0.0328925 \\ 2.53474 0.0327146 \\ 2.54414 0.03244 \\ 2.54817 0.0323594 \\ 2.55603 0.0322117 \\ 2.56441 0.0320856 \\
      2.57277 0.0319985 \\ 2.57652 0.0319248 \\ 2.58946 0.0318113 \\ 2.59659 0.0317234 \\ 2.60088 0.0316636 \\ 2.61202 0.0315619 \\
      2.61998 0.0314758 \\ 2.62918 0.0314059 \\ 2.63567 0.0313448 \\ 2.63964 0.0312766 \\ 2.65038 0.0311838 \\ 2.65694 0.0311125 \\
      2.6671 0.0310536 \\ 2.68785 0.0304766 \\ 2.69447 0.0303118 \\ 2.70189 0.030054 \\ 2.70488 0.0299818 \\ 2.71117 0.0298472 \\
      2.71984 0.0297323 \\ 2.72579 0.0296534 \\ 2.72944 0.0295869 \\ 2.7409 0.0294848 \\ 2.74793 0.0294054 \\ 2.75749 0.0293404 \\
      2.76386 0.0292836 \\ 2.76751 0.0292154 \\ 2.7795 0.0291265 \\ 2.78701 0.02906 \\ 2.79777 0.0290053 \\ 2.81968 0.0284955 \\
      2.82657 0.028343 \\ 2.83543 0.0281047 \\ 2.8392 0.0280365 \\ 2.84518 0.0279122 \\ 2.8529 0.0278044 \\ 2.85921 0.0277309 \\
      2.86284 0.0276693 \\ 2.87397 0.027574 \\ 2.87994 0.0275002 \\ 2.88964 0.0274398 \\ 2.8959 0.0273873 \\ 2.89981 0.0273241 \\
      2.91057 0.0272413 \\ 2.91723 0.0271797 \\ 2.92606 0.0271292 \\ 2.94758 0.026667 \\ 2.95594 0.0265283 \\ 2.96412 0.0263118 \\
      2.9674 0.0262488 \\ 2.9748 0.026136 \\ 2.98357 0.0260357 \\ 2.99035 0.0259678 \\ 2.99397 0.0259115 \\ 3.00641 0.0258234 \\
      3.01367 0.0257554 \\ 3.01796 0.0257097 \\ 3.02944 0.025631 \\ 3.03702 0.0255649 \\ 3.04713 0.0255116 \\ 3.05378 0.0254656 \\
      3.05789 0.025413 \\ 3.06854 0.0253418 \\ 3.07531 0.0252878 \\ 3.08444 0.0252436 \\ 3.10475 0.0248023 \\ 3.11094 0.0246768 \\
      3.11889 0.0244786 \\ 3.12267 0.0244232 \\ 3.12817 0.0243238 \\ 3.13605 0.0242336 \\ 3.14224 0.024173 \\ 3.14506 0.0241236 \\
      3.15694 0.0240457 \\ 3.16423 0.0239855 \\ 3.16806 0.0239452 \\ 3.17863 0.0238757 \\ 3.1867 0.0238172 \\ 3.19643 0.0237702 \\
      3.20344 0.0237297 \\ 3.2072 0.0236831 \\ 3.21841 0.0236206 \\ 3.22516 0.023573 \\ 3.23598 0.023534 \\ 3.25856 0.0231362 \\
      3.26539 0.0230243 \\ 3.27405 0.0228459 \\ 3.27768 0.0227964 \\ 3.28346 0.0227092 \\ 3.29151 0.0226287 \\ 3.29811 0.0225751 \\
      3.30163 0.0225314 \\ 3.3114 0.0224625 \\ 3.31776 0.0224095 \\ 3.32064 0.0223736 \\ 3.33113 0.0223124 \\ 3.33691 0.0222608 \\
      3.3479 0.0222193 \\ 3.35408 0.0221837 \\ 3.35724 0.0221423 \\ 3.36891 0.0220874 \\ 3.37579 0.0220454 \\ 3.3855 0.022011 \\
      3.41112 0.0216523 \\ 3.41942 0.0215525 \\ 3.42782 0.0213914 \\ 3.43302 0.0213463 \\ 3.43969 0.0212694 \\ 3.44762 0.0211971 \\
      3.45488 0.0211495 \\ 3.45884 0.0211104 \\ 3.46946 0.0210492 \\ 3.47623 0.0210019 \\ 3.47961 0.0209699 \\ 3.49104 0.0209154 \\
      3.49763 0.0208695 \\ 3.50813 0.0208325 \\ 3.5145 0.0208006 \\ 3.51823 0.0207637 \\ 3.53014 0.020715 \\ 3.53676 0.0206773 \\
      3.54636 0.0206465 \\ 3.57044 0.0203185 \\ 3.57823 0.0202281 \\ 3.587 0.0200812 \\ 3.59205 0.0200394 \\ 3.59964 0.0199704 \\
      3.6101 0.0199048 \\ 3.61766 0.0198617 \\ 3.62179 0.0198264 \\ 3.63376 0.019771 \\ 3.64168 0.0197283 \\ 3.64564 0.0196993 \\
      3.65926 0.0196501 \\ 3.66671 0.0196085 \\ 3.67901 0.0195749 \\ 3.6858 0.0195459 \\ 3.68971 0.0195127 \\ 3.70223 0.0194687 \\
      3.70947 0.0194344 \\ 3.72027 0.0194063 \\ 3.74533 0.0191033 \\ 3.75269 0.0190194 \\ 3.76271 0.018883 \\ 3.76737 0.0188437 \\
      3.77547 0.0187805 \\ 3.78534 0.01872 \\ 3.7971 0.0186804 \\ 3.80447 0.018648 \\ 3.81739 0.0185971 \\ 3.82475 0.0185579 \\
      3.82857 0.0185315 \\ 3.84159 0.0184863 \\ 3.84956 0.0184482 \\ 3.85995 0.0184173 \\ 3.86728 0.0183907 \\ 3.87169 0.0183605 \\
      3.88364 0.0183202 \\ 3.89019 0.0182887 \\ 3.90093 0.0182629 \\ 3.92358 0.017984 \\ 3.93042 0.0179067 \\ 3.93903 0.017782 \\
      3.94229 0.0177454 \\ 3.94902 0.0176875 \\ 3.95787 0.017632 \\ 3.96173 0.0176062 \\ 3.96901 0.017562 \\ 3.97943 0.0175159 \\
      3.9865 0.0174842 \\ 3.99037 0.0174587 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 0.00835242 2.46526 \\ 0.0155068 2.1985 \\ 0.0226113 1.97428 \\ 0.0286566 1.73697 \\ 0.0365999 1.53124 \\
      0.0439183 1.38611 \\ 0.0526926 1.2029 \\ 0.061252 1.06726 \\ 0.0691174 0.994471 \\ 0.0796323 0.874889 \\ 0.0870235 0.834332 \\
      0.102337 0.733917 \\ 0.109538 0.709088 \\ 0.120172 0.659855 \\ 0.135674 0.603971 \\ 0.14452 0.57969 \\ 0.159772 0.538684 \\
      0.173156 0.509148 \\ 0.190281 0.44321 \\ 0.200575 0.4296 \\ 0.214944 0.399961 \\ 0.232096 0.340424 \\ 0.24042 0.335883 \\
      0.251755 0.3112 \\ 0.263552 0.290646 \\ 0.284863 0.250232 \\ 0.292623 0.247406 \\ 0.308109 0.238011 \\ 0.322387 0.209505 \\
      0.330582 0.205976 \\ 0.346754 0.185121 \\ 0.355899 0.182899 \\ 0.371765 0.166412 \\ 0.383629 0.147913 \\ 0.392091 0.145957 \\
      0.409003 0.134107 \\ 0.427523 0.115648 \\ 0.43582 0.115086 \\ 0.447592 0.11155 \\ 0.46213 0.1009 \\ 0.475434 0.0985478 \\
      0.501234 0.0844256 \\ 0.508565 0.083977 \\ 0.517205 0.0834993 \\ 0.52742 0.0819649 \\ 0.544768 0.0762798 \\ 0.561921 0.0644385 \\
      0.568415 0.0641516 \\ 0.578573 0.0629924 \\ 0.59478 0.0593333 \\ 0.612104 0.0508897 \\ 0.619497 0.0507937 \\ 0.63094 0.0501947 \\
      0.647195 0.0469941 \\ 0.666947 0.037968 \\ 0.675081 0.0378513 \\ 0.685075 0.0377134 \\ 0.696043 0.0373469 \\ 0.717135 0.035375 \\
      0.736171 0.0286797 \\ 0.742599 0.0286326 \\ 0.753092 0.0283249 \\ 0.765097 0.0267126 \\ 0.785178 0.0210442 \\ 0.791129 0.0209662 \\
      0.79968 0.0208736 \\ 0.80989 0.0206686 \\ 0.824483 0.0197053 \\ 0.846555 0.014662 \\ 0.855213 0.0146256 \\ 0.86548 0.0144432 \\
      0.884643 0.0137748 \\ 0.907273 0.0107971 \\ 0.91481 0.010786 \\ 0.925878 0.010691 \\ 0.939661 0.0101635 \\ 0.962535 0.0079016 \\
      0.968387 0.00788349 \\ 0.976982 0.00785899 \\ 0.987803 0.00778959 \\ 1.00009 0.00740045 \\ 1.02285 0.00538395 \\ 1.02958 0.00537666 \\
      1.03852 0.00536279 \\ 1.04946 0.00532486 \\ 1.06332 0.00510866 \\ 1.0919 0.00384158 \\ 1.09906 0.00383467 \\ 1.11145 0.00380387 \\
      1.12522 0.00367851 \\ 1.14825 0.00268108 \\ 1.15585 0.00267917 \\ 1.16486 0.0026759 \\ 1.1759 0.00266464 \\ 1.19016 0.00259045 \\
      1.21246 0.00148949 \\ 1.21982 0.00148712 \\ 1.2293 0.00148401 \\ 1.24089 0.00147489 \\ 1.25972 0.00141753 \\ 1.2866 0.000852665 \\
      1.29431 0.000851499 \\ 1.30387 0.000850173 \\ 1.31612 0.000845003 \\ 1.32977 0.000814938 \\ 1.3581 0.000478148 \\ 1.36589 0.000477811 \\
      1.3757 0.000477075 \\ 1.38679 0.000474219 \\ 1.40025 0.000458419 \\ 1.42412 0.000211634 \\ 1.43162 0.000211361 \\ 1.44058 0.000210817 \\
      1.4524 0.000209296 \\ 1.46702 0.000202466 \\ 1.49576 9.82836e-05 \\ 1.50476 9.82288e-05 \\ 1.51424 9.81058e-05 \\ 1.52602 9.77538e-05 \\
      1.54059 9.5332e-05 \\ 1.56267 2.89025e-05 \\ 1.57105 2.88794e-05 \\ 1.5801 2.88289e-05 \\ 1.58964 2.87075e-05 \\ 1.60158 2.80048e-05 \\
      1.62419 4.87929e-06 \\ 1.63108 4.87506e-06 \\ 1.63912 4.86781e-06 \\ 1.64864 4.84936e-06 \\ 1.66141 4.73983e-06 \\ 1.68782 4.4787e-07 \\
      1.69581 4.47372e-07 \\ 1.7046 4.4653e-07 \\ 1.71586 4.44172e-07 \\ 1.72917 4.30307e-07 \\ 1.74607 1.61916e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 0.0128552 2.44985 \\ 0.0253005 2.05165 \\ 0.0418782 1.77977 \\ 0.0543513 1.56477 \\ 0.0684135 1.33745 \\
      0.0804738 1.16218 \\ 0.0927511 0.997177 \\ 0.104744 0.857844 \\ 0.118716 0.753039 \\ 0.130911 0.647931 \\ 0.145077 0.576747 \\
      0.157128 0.495326 \\ 0.173511 0.427393 \\ 0.189717 0.369959 \\ 0.205691 0.323661 \\ 0.218561 0.27943 \\ 0.23477 0.240469 \\
      0.249192 0.207472 \\ 0.261102 0.17288 \\ 0.273458 0.146722 \\ 0.287744 0.123346 \\ 0.304302 0.108173 \\ 0.31863 0.096226 \\
      0.335393 0.0878709 \\ 0.348912 0.0783986 \\ 0.360738 0.0664209 \\ 0.372768 0.0555386 \\ 0.396817 0.0500955 \\ 0.413126 0.0438219 \\
      0.42801 0.0365067 \\ 0.445361 0.0318277 \\ 0.458083 0.0261896 \\ 0.474477 0.0212425 \\ 0.487615 0.0165991 \\ 0.505471 0.0138685 \\
      0.517963 0.0109732 \\ 0.532961 0.00829399 \\ 0.548299 0.00622675 \\ 0.563337 0.00466276 \\ 0.579612 0.0038435 \\ 0.591986 0.00303281 \\
      0.605062 0.00209377 \\ 0.621808 0.00146101 \\ 0.637876 0.000982211 \\ 0.651468 0.000569945 \\ 0.670331 0.000341941 \\ 0.684414 0.00017826 \\
      0.697888 8.27995e-05 \\ 0.710936 3.43078e-05 \\ 0.723926 6.88103e-06 \\ 0.736536 7.97022e-07 \\ 0.748407 2.26362e-08 \\ 0.760551 1.03754e-10 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 2.94392 \\ 0.010446 2.47303 \\ 0.0193987 2.14783 \\ 0.02844 1.88345 \\ 0.0362753 1.63515 \\ 0.0440239 1.42657 \\
      0.0525061 1.25229 \\ 0.0612749 1.122 \\ 0.0701493 0.999022 \\ 0.0794605 0.898615 \\ 0.0892558 0.824901 \\ 0.0986297 0.763266 \\
      0.10856 0.718185 \\ 0.129628 0.625943 \\ 0.139307 0.60037 \\ 0.159598 0.529623 \\ 0.16942 0.511027 \\ 0.188993 0.450868 \\
      0.198357 0.435377 \\ 0.21547 0.373664 \\ 0.225575 0.361335 \\ 0.24151 0.303631 \\ 0.250285 0.293115 \\ 0.267668 0.252309 \\
      0.276769 0.245907 \\ 0.293604 0.213493 \\ 0.300804 0.209782 \\ 0.315463 0.177384 \\ 0.323952 0.17418 \\ 0.342059 0.148079 \\
      0.351107 0.145967 \\ 0.36896 0.126371 \\ 0.378108 0.125065 \\ 0.393899 0.105038 \\ 0.402944 0.103879 \\ 0.421914 0.0885395 \\
      0.429856 0.087781 \\ 0.447729 0.0757871 \\ 0.456431 0.0752292 \\ 0.472989 0.0658458 \\ 0.480262 0.0653782 \\ 0.494121 0.0549365 \\
      0.502034 0.0546214 \\ 0.520421 0.0486912 \\ 0.528124 0.0484931 \\ 0.541628 0.0416366 \\ 0.550008 0.041484 \\ 0.56769 0.0359402 \\
      0.576261 0.0357737 \\ 0.592633 0.0295853 \\ 0.601124 0.0294481 \\ 0.6173 0.0227541 \\ 0.625536 0.0226243 \\ 0.648673 0.019725 \\
      0.656784 0.0196709 \\ 0.669527 0.0150221 \\ 0.677147 0.0149674 \\ 0.696139 0.012443 \\ 0.703491 0.0124046 \\ 0.720203 0.0103272 \\
      0.728317 0.0103015 \\ 0.742533 0.00811095 \\ 0.749479 0.0080831 \\ 0.764621 0.00575928 \\ 0.773354 0.00573436 \\ 0.792205 0.00418805 \\
      0.799831 0.00416435 \\ 0.816035 0.00298684 \\ 0.82654 0.00298008 \\ 0.842655 0.00181965 \\ 0.85131 0.00181395 \\ 0.872222 0.00121712 \\
      0.880138 0.00121431 \\ 0.896855 0.000781969 \\ 0.904536 0.000780655 \\ 0.919359 0.000378816 \\ 0.926573 0.000377358 \\ 0.942735 0.000184322 \\
      0.950033 0.000183912 \\ 0.966253 8.93031e-05 \\ 0.974607 8.91374e-05 \\ 0.988128 2.87246e-05 \\ 0.996834 2.86859e-05 \\ 1.01311 4.68364e-06 \\
      1.0227 4.6685e-06 \\ 1.04038 3.26239e-07 \\ 1.04871 3.25308e-07 \\ 1.06218 2.69317e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(b) $\gamma=0.05$, runtime};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=502,
        ymin=2.397e-13, ymax=1.094e+01,
        ytickten={-11,-8,-5,-2,1},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 1 2.17423 \\ 2 1.92613 \\ 3 1.73542 \\ 4 1.57236 \\ 5 1.34215 \\
      6 1.21211 \\ 7 1.01608 \\ 8 0.914591 \\ 9 0.778692 \\ 10 0.704122 \\ 11 0.592775 \\
      12 0.540933 \\ 13 0.462196 \\ 14 0.430572 \\ 15 0.37599 \\ 16 0.353238 \\ 17 0.323733 \\
      18 0.314269 \\ 19 0.300633 \\ 20 0.279947 \\ 21 0.273014 \\ 22 0.262865 \\ 23 0.250858 \\
      24 0.244174 \\ 25 0.237712 \\ 26 0.233227 \\ 27 0.227116 \\ 28 0.220772 \\ 29 0.216452 \\
      30 0.212663 \\ 31 0.175018 \\ 32 0.169675 \\ 33 0.161696 \\ 34 0.15874 \\ 35 0.156509 \\
      36 0.152448 \\ 37 0.149875 \\ 38 0.148146 \\ 39 0.144926 \\ 40 0.14271 \\ 41 0.140731 \\
      42 0.139106 \\ 43 0.137528 \\ 44 0.13508 \\ 45 0.133414 \\ 46 0.131942 \\ 47 0.113769 \\
      48 0.11158 \\ 49 0.106981 \\ 50 0.105362 \\ 51 0.104214 \\ 52 0.102325 \\ 53 0.10102 \\
      54 0.0999046 \\ 55 0.0989823 \\ 56 0.0980815 \\ 57 0.0967624 \\ 58 0.0957967 \\ 59 0.0949761 \\
      60 0.0847803 \\ 61 0.083257 \\ 62 0.0795231 \\ 63 0.0783471 \\ 64 0.0775894 \\ 65 0.0762915 \\
      66 0.0754201 \\ 67 0.0749219 \\ 68 0.0739573 \\ 69 0.0731927 \\ 70 0.0725696 \\ 71 0.0720396 \\
      72 0.0715326 \\ 73 0.0707925 \\ 74 0.070228 \\ 75 0.0697577 \\ 76 0.0640049 \\ 77 0.063038 \\
      78 0.0607521 \\ 79 0.0602579 \\ 80 0.0593697 \\ 81 0.058571 \\ 82 0.0580479 \\ 83 0.0576784 \\
      84 0.0570366 \\ 85 0.0565568 \\ 86 0.0561633 \\ 87 0.0558204 \\ 88 0.0554498 \\ 89 0.0549486 \\
      90 0.0545707 \\ 91 0.0542586 \\ 92 0.0507378 \\ 93 0.0500783 \\ 94 0.0486459 \\ 95 0.0482795 \\
      96 0.0476478 \\ 97 0.0470944 \\ 98 0.0468518 \\ 99 0.0463911 \\ 100 0.0459539 \\ 101 0.0456388 \\
      102 0.0454002 \\ 103 0.0450034 \\ 104 0.0446892 \\ 105 0.0444298 \\ 106 0.0441995 \\ 107 0.043947 \\
      108 0.0436067 \\ 109 0.0433432 \\ 110 0.0431248 \\ 111 0.0408231 \\ 112 0.0403205 \\ 113 0.0393591 \\
      114 0.0391008 \\ 115 0.0386411 \\ 116 0.0382449 \\ 117 0.0380685 \\ 118 0.0377238 \\ 119 0.037403 \\
      120 0.0371703 \\ 121 0.0369912 \\ 122 0.0366962 \\ 123 0.0364611 \\ 124 0.0362678 \\ 125 0.0360974 \\
      126 0.0359077 \\ 127 0.035653 \\ 128 0.0354563 \\ 129 0.0352942 \\ 130 0.0337087 \\ 131 0.0333227 \\
      132 0.0326317 \\ 133 0.0324504 \\ 134 0.0321047 \\ 135 0.0318105 \\ 136 0.0316075 \\ 137 0.0314467 \\
      138 0.0311948 \\ 139 0.0309946 \\ 140 0.0308327 \\ 141 0.0307037 \\ 142 0.0304964 \\ 143 0.030307 \\
      144 0.03017 \\ 145 0.0300423 \\ 146 0.0298453 \\ 147 0.0296947 \\ 148 0.029571 \\ 149 0.0284124 \\
      150 0.0281246 \\ 151 0.0276086 \\ 152 0.0274741 \\ 153 0.0272178 \\ 154 0.0269899 \\ 155 0.0268331 \\
      156 0.026715 \\ 157 0.0265194 \\ 158 0.0263651 \\ 159 0.02624 \\ 160 0.0261383 \\ 161 0.0259829 \\
      162 0.0258336 \\ 163 0.0257273 \\ 164 0.0256303 \\ 165 0.025476 \\ 166 0.0253591 \\ 167 0.0252632 \\
      168 0.0243647 \\ 169 0.024135 \\ 170 0.0237234 \\ 171 0.0236152 \\ 172 0.0234153 \\ 173 0.0232304 \\
      174 0.023105 \\ 175 0.0230119 \\ 176 0.0228549 \\ 177 0.0227319 \\ 178 0.0226571 \\ 179 0.022518 \\
      180 0.0224007 \\ 181 0.0223066 \\ 182 0.022226 \\ 183 0.0221384 \\ 184 0.0220164 \\ 185 0.0219224 \\
      186 0.0218462 \\ 187 0.021113 \\ 188 0.0209232 \\ 189 0.0205868 \\ 190 0.0204958 \\ 191 0.0203356 \\
      192 0.0201843 \\ 193 0.0200835 \\ 194 0.0200064 \\ 195 0.0198791 \\ 196 0.01978 \\ 197 0.0197178 \\
      198 0.0196053 \\ 199 0.0195108 \\ 200 0.0194355 \\ 201 0.0193714 \\ 202 0.0192987 \\ 203 0.0192005 \\
      204 0.019125 \\ 205 0.0190641 \\ 206 0.0184424 \\ 207 0.0182829 \\ 208 0.0180017 \\ 209 0.0179219 \\
      210 0.0177905 \\ 211 0.0176648 \\ 212 0.0176073 \\ 213 0.0175073 \\ 214 0.0174033 \\ 215 0.0173324 \\
      216 0.0172761 \\ 217 0.0171827 \\ 218 0.0171103 \\ 219 0.0170516 \\ 220 0.0169984 \\ 221 0.0169267 \\
      222 0.016852 \\ 223 0.0163359 \\ 224 0.0161986 \\ 225 0.0159598 \\ 226 0.0158886 \\ 227 0.0157766 \\
      228 0.015667 \\ 229 0.0156166 \\ 230 0.015531 \\ 231 0.0154405 \\ 232 0.0153792 \\ 233 0.0153313 \\
      234 0.0152506 \\ 235 0.0151887 \\ 236 0.0151386 \\ 237 0.0150926 \\ 238 0.015032 \\ 239 0.014968 \\
      240 0.0145269 \\ 241 0.0144096 \\ 242 0.0142078 \\ 243 0.0141439 \\ 244 0.0140492 \\ 245 0.0139552 \\
      246 0.0139106 \\ 247 0.0138383 \\ 248 0.0137607 \\ 249 0.0137086 \\ 250 0.0136675 \\ 251 0.0135982 \\
      252 0.0135458 \\ 253 0.0135036 \\ 254 0.0134636 \\ 255 0.0134126 \\ 256 0.0133582 \\ 257 0.0129804 \\
      258 0.0128801 \\ 259 0.0127089 \\ 260 0.0126519 \\ 261 0.0125718 \\ 262 0.0124917 \\ 263 0.0124523 \\
      264 0.0123914 \\ 265 0.0123252 \\ 266 0.0122921 \\ 267 0.0122394 \\ 268 0.0121809 \\ 269 0.0121414 \\
      270 0.0121106 \\ 271 0.0120567 \\ 272 0.0120165 \\ 273 0.0119839 \\ 274 0.0119525 \\ 275 0.0119133 \\
      276 0.0118707 \\ 277 0.011529 \\ 278 0.0114447 \\ 279 0.0113002 \\ 280 0.0112522 \\ 281 0.011187 \\
      282 0.0111214 \\ 283 0.0110884 \\ 284 0.0110393 \\ 285 0.0109852 \\ 286 0.0109576 \\ 287 0.0109152 \\
      288 0.0108675 \\ 289 0.0108353 \\ 290 0.0108078 \\ 291 0.010776 \\ 292 0.0107422 \\ 293 0.0107035 \\
      294 0.0104267 \\ 295 0.0103573 \\ 296 0.0102348 \\ 297 0.010193 \\ 298 0.0101376 \\ 299 0.0100815 \\
      300 0.0100525 \\ 301 0.0100107 \\ 302 0.00996438 \\ 303 0.00994016 \\ 304 0.00990399 \\ 305 0.00986297 \\
      306 0.00983548 \\ 307 0.009812 \\ 308 0.00978422 \\ 309 0.00975545 \\ 310 0.00972218 \\ 311 0.00948969 \\
      312 0.00943203 \\ 313 0.0093266 \\ 314 0.00929067 \\ 315 0.00924345 \\ 316 0.00919548 \\ 317 0.00917044 \\
      318 0.00913471 \\ 319 0.00909501 \\ 320 0.00907401 \\ 321 0.00904303 \\ 322 0.00900782 \\ 323 0.00898427 \\
      324 0.00896419 \\ 325 0.00894002 \\ 326 0.0089153 \\ 327 0.00888669 \\ 328 0.00868803 \\ 329 0.0086391 \\
      330 0.0085467 \\ 331 0.00851581 \\ 332 0.00847496 \\ 333 0.00843356 \\ 334 0.00841198 \\ 335 0.00838109 \\
      336 0.00834686 \\ 337 0.00832873 \\ 338 0.00830192 \\ 339 0.00827154 \\ 340 0.00825122 \\ 341 0.00823394 \\
      342 0.00821291 \\ 343 0.0081915 \\ 344 0.00816679 \\ 345 0.00799295 \\ 346 0.0079502 \\ 347 0.00786738 \\
      348 0.00784046 \\ 349 0.00780443 \\ 350 0.00776818 \\ 351 0.00774942 \\ 352 0.00772233 \\ 353 0.0076925 \\
      354 0.00767673 \\ 355 0.00765326 \\ 356 0.00762681 \\ 357 0.00760911 \\ 358 0.00759412 \\ 359 0.00757568 \\
      360 0.00755697 \\ 361 0.00753548 \\ 362 0.00737948 \\ 363 0.0073414 \\ 364 0.00726606 \\ 365 0.00724225 \\
      366 0.00721002 \\ 367 0.00717788 \\ 368 0.00716137 \\ 369 0.00713736 \\ 370 0.00711108 \\ 371 0.00709394 \\
      372 0.00707967 \\ 373 0.00706145 \\ 374 0.00704344 \\ 375 0.00702303 \\ 376 0.00689083 \\ 377 0.00685689 \\
      378 0.0067882 \\ 379 0.00676592 \\ 380 0.00673619 \\ 381 0.00670667 \\ 382 0.00669131 \\ 383 0.00666918 \\
      384 0.00664508 \\ 385 0.00662947 \\ 386 0.00661653 \\ 387 0.00659955 \\ 388 0.00658302 \\ 389 0.00656436 \\
      390 0.00644341 \\ 391 0.00641251 \\ 392 0.00634897 \\ 393 0.00632832 \\ 394 0.00630089 \\ 395 0.00627381 \\
      396 0.00625963 \\ 397 0.00623932 \\ 398 0.00621728 \\ 399 0.00620308 \\ 400 0.00619136 \\ 401 0.00617566 \\
      402 0.00616055 \\ 403 0.00614354 \\ 404 0.00603177 \\ 405 0.00600351 \\ 406 0.00594458 \\ 407 0.00592549 \\
      408 0.0059002 \\ 409 0.00587537 \\ 410 0.0058623 \\ 411 0.00584368 \\ 412 0.00582354 \\ 413 0.00581063 \\
      414 0.00580001 \\ 415 0.00578553 \\ 416 0.00577174 \\ 417 0.00575624 \\ 418 0.00565256 \\ 419 0.00562665 \\
      420 0.00557191 \\ 421 0.00555429 \\ 422 0.00553098 \\ 423 0.00550819 \\ 424 0.00549614 \\ 425 0.00547906 \\
      426 0.00546064 \\ 427 0.00544889 \\ 428 0.00536766 \\ 429 0.00534689 \\ 430 0.00529488 \\ 431 0.00527756 \\
      432 0.0052555 \\ 433 0.00523379 \\ 434 0.00522208 \\ 435 0.00520583 \\ 436 0.00518822 \\ 437 0.00517702 \\
      438 0.00516786 \\ 439 0.00515498 \\ 440 0.00514295 \\ 441 0.00512945 \\ 442 0.00504039 \\ 443 0.00501895 \\
      444 0.00497129 \\ 445 0.00495573 \\ 446 0.00493547 \\ 447 0.00491568 \\ 448 0.00490505 \\ 449 0.00489023 \\
      450 0.00487424 \\ 451 0.0048641 \\ 452 0.00479398 \\ 453 0.00477644 \\ 454 0.004731 \\ 455 0.00471582 \\
      456 0.00469663 \\ 457 0.00467778 \\ 458 0.0046675 \\ 459 0.0046534 \\ 460 0.00463812 \\ 461 0.00462845 \\
      462 0.00462056 \\ 463 0.00460927 \\ 464 0.00459884 \\ 465 0.00458715 \\ 466 0.00450955 \\ 467 0.00449124 \\
      468 0.00444941 \\ 469 0.00443579 \\ 470 0.00441812 \\ 471 0.00440089 \\ 472 0.00439157 \\ 473 0.00437867 \\
      474 0.00436476 \\ 475 0.00435596 \\ 476 0.0042948 \\ 477 0.00427975 \\ 478 0.0042398 \\ 479 0.00422654 \\
      480 0.00420976 \\ 481 0.00419333 \\ 482 0.00418432 \\ 483 0.00417202 \\ 484 0.0041587 \\ 485 0.00415028 \\
      486 0.00414344 \\ 487 0.00413354 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 1 2.15617 \\ 2 1.90829 \\ 3 1.70543 \\ 4 1.50875 \\ 5 1.34768 \\
      6 1.22357 \\ 7 1.04786 \\ 8 0.903195 \\ 9 0.785781 \\ 10 0.645765 \\ 11 0.586898 \\
      12 0.500471 \\ 13 0.46886 \\ 14 0.365819 \\ 15 0.353375 \\ 16 0.33605 \\ 17 0.296576 \\
      18 0.27033 \\ 19 0.203517 \\ 20 0.19879 \\ 21 0.183014 \\ 22 0.143885 \\ 23 0.14012 \\
      24 0.118807 \\ 25 0.114181 \\ 26 0.0874135 \\ 27 0.0864458 \\ 28 0.0854621 \\ 29 0.0733016 \\
      30 0.0529782 \\ 31 0.052438 \\ 32 0.0507236 \\ 33 0.0447546 \\ 34 0.0296785 \\ 35 0.0294101 \\
      36 0.0291486 \\ 37 0.02864 \\ 38 0.0258711 \\ 39 0.0176964 \\ 40 0.0176265 \\ 41 0.0175191 \\
      42 0.0172687 \\ 43 0.0160677 \\ 44 0.00966453 \\ 45 0.00962143 \\ 46 0.00939913 \\ 47 0.00861624 \\
      48 0.0051708 \\ 49 0.00514865 \\ 50 0.00511592 \\ 51 0.00504399 \\ 52 0.00466851 \\ 53 0.00251996 \\
      54 0.00251086 \\ 55 0.00249927 \\ 56 0.00246938 \\ 57 0.00233811 \\ 58 0.000883204 \\ 59 0.000880258 \\
      60 0.000875852 \\ 61 0.000861695 \\ 62 0.000811354 \\ 63 0.000295298 \\ 64 0.000294649 \\ 65 0.000293459 \\
      66 0.000289499 \\ 67 0.000276848 \\ 68 5.82265e-05 \\ 69 5.80942e-05 \\ 70 5.782e-05 \\ 71 5.70698e-05 \\
      72 5.37935e-05 \\ 73 5.25335e-06 \\ 74 5.24328e-06 \\ 75 5.22402e-06 \\ 76 5.16688e-06 \\ 77 4.83653e-06 \\
      78 2.71006e-08 \\ 79 2.68984e-08 \\ 80 2.65219e-08 \\ 81 2.5549e-08 \\ 82 2.16667e-08 \\ 83 1.00131e-12 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 1 2.04261 \\ 2 1.66719 \\ 3 1.41893 \\ 4 1.21069 \\ 5 1.03654 \\
      6 0.882149 \\ 7 0.735464 \\ 8 0.602441 \\ 9 0.483466 \\ 10 0.378142 \\ 11 0.287996 \\
      12 0.22183 \\ 13 0.168629 \\ 14 0.1266 \\ 15 0.100262 \\ 16 0.079239 \\ 17 0.0594962 \\
      18 0.0447528 \\ 19 0.0341128 \\ 20 0.0247474 \\ 21 0.0182319 \\ 22 0.013138 \\ 23 0.00852754 \\
      24 0.00497867 \\ 25 0.00301797 \\ 26 0.00164924 \\ 27 0.000772019 \\ 28 0.000326264 \\ 29 0.000122966 \\
      30 2.73494e-05 \\ 31 3.30271e-06 \\ 32 1.10255e-07 \\ 33 4.80382e-10 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 1 2.11368 \\ 2 1.82008 \\ 3 1.57698 \\ 4 1.38079 \\ 5 1.2104 \\
      6 1.07496 \\ 7 0.95755 \\ 8 0.843897 \\ 9 0.752349 \\ 10 0.594379 \\ 11 0.548336 \\
      12 0.426255 \\ 13 0.397026 \\ 14 0.299201 \\ 15 0.279723 \\ 16 0.205973 \\ 17 0.197038 \\
      18 0.142244 \\ 19 0.136916 \\ 20 0.0967403 \\ 21 0.0937064 \\ 22 0.0751158 \\ 23 0.0741583 \\
      24 0.0605005 \\ 25 0.0598373 \\ 26 0.0462955 \\ 27 0.0458972 \\ 28 0.0338068 \\ 29 0.0334532 \\
      30 0.0227015 \\ 31 0.0223391 \\ 32 0.0168823 \\ 33 0.0167651 \\ 34 0.0126302 \\ 35 0.0125609 \\
      36 0.0085827 \\ 37 0.00853239 \\ 38 0.00419159 \\ 39 0.00413693 \\ 40 0.00215738 \\ 41 0.00214286 \\
      42 0.000932538 \\ 43 0.000924419 \\ 44 0.000382321 \\ 45 0.000380452 \\ 46 0.000117205 \\ 47 0.00011661 \\
      48 1.7294e-05 \\ 49 1.72194e-05 \\ 50 5.58444e-07 \\ 51 5.54315e-07 \\ 52 1.67421e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(c) $\gamma=0.1$, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=4.12,
        ymin=2.397e-13, ymax=1.094e+01,
        ytickten={-11,-8,-5,-2,1},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 0.00847604 2.17423 \\ 0.0169285 1.92613 \\ 0.0236262 1.73542 \\ 0.0297475 1.57236 \\ 0.0393675 1.34215 \\
      0.0451263 1.21211 \\ 0.0555089 1.01608 \\ 0.0624937 0.914591 \\ 0.0716204 0.778692 \\ 0.0780206 0.704122 \\ 0.0874056 0.592775 \\
      0.0937725 0.540933 \\ 0.103916 0.462196 \\ 0.11027 0.430572 \\ 0.119224 0.37599 \\ 0.125943 0.353238 \\ 0.13633 0.323733 \\
      0.14051 0.314269 \\ 0.14721 0.300633 \\ 0.158637 0.279947 \\ 0.162335 0.273014 \\ 0.16941 0.262865 \\ 0.180967 0.250858 \\
      0.187542 0.244174 \\ 0.195687 0.237712 \\ 0.200054 0.233227 \\ 0.208618 0.227116 \\ 0.222872 0.220772 \\ 0.229357 0.216452 \\
      0.238132 0.212663 \\ 0.257964 0.175018 \\ 0.264846 0.169675 \\ 0.274113 0.161696 \\ 0.28142 0.15874 \\ 0.285613 0.156509 \\
      0.294116 0.152448 \\ 0.301053 0.149875 \\ 0.305445 0.148146 \\ 0.316594 0.144926 \\ 0.323687 0.14271 \\ 0.331984 0.140731 \\
      0.339654 0.139106 \\ 0.343161 0.137528 \\ 0.356875 0.13508 \\ 0.363679 0.133414 \\ 0.373583 0.131942 \\ 0.396527 0.113769 \\
      0.404225 0.11158 \\ 0.412175 0.106981 \\ 0.419898 0.105362 \\ 0.42395 0.104214 \\ 0.432793 0.102325 \\ 0.439962 0.10102 \\
      0.448494 0.0999046 \\ 0.454667 0.0989823 \\ 0.458475 0.0980815 \\ 0.469988 0.0967624 \\ 0.476221 0.0957967 \\ 0.485267 0.0949761 \\
      0.506404 0.0847803 \\ 0.513121 0.083257 \\ 0.520711 0.0795231 \\ 0.527603 0.0783471 \\ 0.530888 0.0775894 \\ 0.539294 0.0762915 \\
      0.546051 0.0754201 \\ 0.550194 0.0749219 \\ 0.561166 0.0739573 \\ 0.568271 0.0731927 \\ 0.579866 0.0725696 \\ 0.58726 0.0720396 \\
      0.591569 0.0715326 \\ 0.604465 0.0707925 \\ 0.611529 0.070228 \\ 0.622744 0.0697577 \\ 0.643771 0.0640049 \\ 0.65095 0.063038 \\
      0.659864 0.0607521 \\ 0.663114 0.0602579 \\ 0.668592 0.0593697 \\ 0.676324 0.058571 \\ 0.682217 0.0580479 \\ 0.686082 0.0576784 \\
      0.696135 0.0570366 \\ 0.702676 0.0565568 \\ 0.71257 0.0561633 \\ 0.718288 0.0558204 \\ 0.722013 0.0554498 \\ 0.731945 0.0549486 \\
      0.737554 0.0545707 \\ 0.747546 0.0542586 \\ 0.7766 0.0507378 \\ 0.783976 0.0500783 \\ 0.792778 0.0486459 \\ 0.797503 0.0482795 \\
      0.804136 0.0476478 \\ 0.812459 0.0470944 \\ 0.816492 0.0468518 \\ 0.823898 0.0463911 \\ 0.833409 0.0459539 \\ 0.840164 0.0456388 \\
      0.844155 0.0454002 \\ 0.855235 0.0450034 \\ 0.861911 0.0446892 \\ 0.871465 0.0444298 \\ 0.877506 0.0441995 \\ 0.881136 0.043947 \\
      0.893923 0.0436067 \\ 0.900045 0.0433432 \\ 0.908844 0.0431248 \\ 0.929252 0.0408231 \\ 0.935205 0.0403205 \\ 0.943552 0.0393591 \\
      0.947234 0.0391008 \\ 0.953385 0.0386411 \\ 0.962271 0.0382449 \\ 0.966247 0.0380685 \\ 0.97378 0.0377238 \\ 0.98288 0.037403 \\
      0.990272 0.0371703 \\ 0.994354 0.0369912 \\ 1.00642 0.0366962 \\ 1.0138 0.0364611 \\ 1.02478 0.0362678 \\ 1.03174 0.0360974 \\
      1.03585 0.0359077 \\ 1.04675 0.035653 \\ 1.05372 0.0354563 \\ 1.06237 0.0352942 \\ 1.08383 0.0337087 \\ 1.09052 0.0333227 \\
      1.09877 0.0326317 \\ 1.10188 0.0324504 \\ 1.10841 0.0321047 \\ 1.11666 0.0318105 \\ 1.12279 0.0316075 \\ 1.12587 0.0314467 \\
      1.13651 0.0311948 \\ 1.14283 0.0309946 \\ 1.15185 0.0308327 \\ 1.15564 0.0307037 \\ 1.16375 0.0304964 \\ 1.17326 0.030307 \\
      1.17893 0.03017 \\ 1.18262 0.0300423 \\ 1.19425 0.0298453 \\ 1.20077 0.0296947 \\ 1.21185 0.029571 \\ 1.23453 0.0284124 \\
      1.24235 0.0281246 \\ 1.25141 0.0276086 \\ 1.25537 0.0274741 \\ 1.2621 0.0272178 \\ 1.27125 0.0269899 \\ 1.27777 0.0268331 \\
      1.28143 0.026715 \\ 1.29124 0.0265194 \\ 1.29789 0.0263651 \\ 1.30771 0.02624 \\ 1.31157 0.0261383 \\ 1.31874 0.0259829 \\
      1.32684 0.0258336 \\ 1.33266 0.0257273 \\ 1.33678 0.0256303 \\ 1.34852 0.025476 \\ 1.35448 0.0253591 \\ 1.36367 0.0252632 \\
      1.38608 0.0243647 \\ 1.39387 0.024135 \\ 1.40241 0.0237234 \\ 1.40705 0.0236152 \\ 1.41412 0.0234153 \\ 1.42304 0.0232304 \\
      1.42966 0.023105 \\ 1.43381 0.0230119 \\ 1.44509 0.0228549 \\ 1.45233 0.0227319 \\ 1.45599 0.0226571 \\ 1.46738 0.022518 \\
      1.47418 0.0224007 \\ 1.4839 0.0223066 \\ 1.49063 0.022226 \\ 1.49426 0.0221384 \\ 1.50498 0.0220164 \\ 1.51102 0.0219224 \\
      1.52113 0.0218462 \\ 1.54168 0.021113 \\ 1.54851 0.0209232 \\ 1.55572 0.0205868 \\ 1.55954 0.0204958 \\ 1.56597 0.0203356 \\
      1.57456 0.0201843 \\ 1.58138 0.0200835 \\ 1.5845 0.0200064 \\ 1.59473 0.0198791 \\ 1.60211 0.01978 \\ 1.60559 0.0197178 \\
      1.6176 0.0196053 \\ 1.62479 0.0195108 \\ 1.6363 0.0194355 \\ 1.64258 0.0193714 \\ 1.64734 0.0192987 \\ 1.65994 0.0192005 \\
      1.66782 0.019125 \\ 1.67721 0.0190641 \\ 1.70056 0.0184424 \\ 1.70762 0.0182829 \\ 1.71681 0.0180017 \\ 1.72084 0.0179219 \\
      1.72737 0.0177905 \\ 1.73558 0.0176648 \\ 1.73948 0.0176073 \\ 1.74534 0.0175073 \\ 1.75643 0.0174033 \\ 1.7632 0.0173324 \\
      1.76818 0.0172761 \\ 1.78067 0.0171827 \\ 1.78811 0.0171103 \\ 1.79851 0.0170516 \\ 1.80274 0.0169984 \\ 1.81117 0.0169267 \\
      1.82146 0.016852 \\ 1.84518 0.0163359 \\ 1.85262 0.0161986 \\ 1.86114 0.0159598 \\ 1.86526 0.0158886 \\ 1.87291 0.0157766 \\
      1.88233 0.015667 \\ 1.88627 0.0156166 \\ 1.89301 0.015531 \\ 1.9028 0.0154405 \\ 1.9093 0.0153792 \\ 1.91265 0.0153313 \\
      1.92581 0.0152506 \\ 1.93209 0.0151887 \\ 1.94166 0.0151386 \\ 1.94609 0.0150926 \\ 1.95506 0.015032 \\ 1.96446 0.014968 \\
      1.98721 0.0145269 \\ 1.99468 0.0144096 \\ 2.00335 0.0142078 \\ 2.00731 0.0141439 \\ 2.01367 0.0140492 \\ 2.02358 0.0139552 \\
      2.0271 0.0139106 \\ 2.03498 0.0138383 \\ 2.04456 0.0137607 \\ 2.05141 0.0137086 \\ 2.0561 0.0136675 \\ 2.06836 0.0135982 \\
      2.07531 0.0135458 \\ 2.08478 0.0135036 \\ 2.08882 0.0134636 \\ 2.09678 0.0134126 \\ 2.10683 0.0133582 \\ 2.12783 0.0129804 \\
      2.13476 0.0128801 \\ 2.14417 0.0127089 \\ 2.14719 0.0126519 \\ 2.15482 0.0125718 \\ 2.16344 0.0124917 \\ 2.16722 0.0124523 \\
      2.17494 0.0123914 \\ 2.18475 0.0123252 \\ 2.1882 0.0122921 \\ 2.19711 0.0122394 \\ 2.20755 0.0121809 \\ 2.2142 0.0121414 \\
      2.21832 0.0121106 \\ 2.23039 0.0120567 \\ 2.23748 0.0120165 \\ 2.24684 0.0119839 \\ 2.2506 0.0119525 \\ 2.25937 0.0119133 \\
      2.26899 0.0118707 \\ 2.29268 0.011529 \\ 2.29917 0.0114447 \\ 2.3081 0.0113002 \\ 2.31207 0.0112522 \\ 2.3188 0.011187 \\
      2.32674 0.0111214 \\ 2.33068 0.0110884 \\ 2.3364 0.0110393 \\ 2.34569 0.0109852 \\ 2.34874 0.0109576 \\ 2.35496 0.0109152 \\
      2.36549 0.0108675 \\ 2.3715 0.0108353 \\ 2.38036 0.0108078 \\ 2.38357 0.010776 \\ 2.39123 0.0107422 \\ 2.40135 0.0107035 \\
      2.42276 0.0104267 \\ 2.4298 0.0103573 \\ 2.43953 0.0102348 \\ 2.44362 0.010193 \\ 2.45122 0.0101376 \\ 2.46126 0.0100815 \\
      2.46561 0.0100525 \\ 2.47313 0.0100107 \\ 2.48429 0.00996438 \\ 2.48873 0.00994016 \\ 2.49616 0.00990399 \\ 2.50731 0.00986297 \\
      2.51344 0.00983548 \\ 2.52327 0.009812 \\ 2.52695 0.00978422 \\ 2.53467 0.00975545 \\ 2.54399 0.00972218 \\ 2.56751 0.00948969 \\
      2.57477 0.00943203 \\ 2.58373 0.0093266 \\ 2.58738 0.00929067 \\ 2.59347 0.00924345 \\ 2.60235 0.00919548 \\ 2.60642 0.00917044 \\
      2.61371 0.00913471 \\ 2.62559 0.00909501 \\ 2.62978 0.00907401 \\ 2.63607 0.00904303 \\ 2.64707 0.00900782 \\ 2.65349 0.00898427 \\
      2.6645 0.00896419 \\ 2.66917 0.00894002 \\ 2.67797 0.0089153 \\ 2.68818 0.00888669 \\ 2.71171 0.00868803 \\ 2.71972 0.0086391 \\
      2.72806 0.0085467 \\ 2.73114 0.00851581 \\ 2.73766 0.00847496 \\ 2.74549 0.00843356 \\ 2.74868 0.00841198 \\ 2.7548 0.00838109 \\
      2.76522 0.00834686 \\ 2.76937 0.00832873 \\ 2.77573 0.00830192 \\ 2.78502 0.00827154 \\ 2.79085 0.00825122 \\ 2.79994 0.00823394 \\
      2.80354 0.00821291 \\ 2.81121 0.0081915 \\ 2.82177 0.00816679 \\ 2.84307 0.00799295 \\ 2.85038 0.0079502 \\ 2.85856 0.00786738 \\
      2.86286 0.00784046 \\ 2.87017 0.00780443 \\ 2.8806 0.00776818 \\ 2.88482 0.00774942 \\ 2.89112 0.00772233 \\ 2.90114 0.0076925 \\
      2.90495 0.00767673 \\ 2.91358 0.00765326 \\ 2.9243 0.00762681 \\ 2.931 0.00760911 \\ 2.94063 0.00759412 \\ 2.9445 0.00757568 \\
      2.95289 0.00755697 \\ 2.96203 0.00753548 \\ 2.98293 0.00737948 \\ 2.99005 0.0073414 \\ 2.99799 0.00726606 \\ 3.00096 0.00724225 \\
      3.00707 0.00721002 \\ 3.01495 0.00717788 \\ 3.01833 0.00716137 \\ 3.02575 0.00713736 \\ 3.03634 0.00711108 \\ 3.04316 0.00709394 \\
      3.05321 0.00707967 \\ 3.05761 0.00706145 \\ 3.06561 0.00704344 \\ 3.0753 0.00702303 \\ 3.09735 0.00689083 \\ 3.10478 0.00685689 \\
      3.11377 0.0067882 \\ 3.11752 0.00676592 \\ 3.12388 0.00673619 \\ 3.13307 0.00670667 \\ 3.13695 0.00669131 \\ 3.14255 0.00666918 \\
      3.15328 0.00664508 \\ 3.15971 0.00662947 \\ 3.16758 0.00661653 \\ 3.17136 0.00659955 \\ 3.17939 0.00658302 \\ 3.18919 0.00656436 \\
      3.20953 0.00644341 \\ 3.21674 0.00641251 \\ 3.22503 0.00634897 \\ 3.22908 0.00632832 \\ 3.23546 0.00630089 \\ 3.24519 0.00627381 \\
      3.24938 0.00625963 \\ 3.25641 0.00623932 \\ 3.26754 0.00621728 \\ 3.27468 0.00620308 \\ 3.28378 0.00619136 \\ 3.28848 0.00617566 \\
      3.29624 0.00616055 \\ 3.30736 0.00614354 \\ 3.32776 0.00603177 \\ 3.33532 0.00600351 \\ 3.34361 0.00594458 \\ 3.34682 0.00592549 \\
      3.35293 0.0059002 \\ 3.36189 0.00587537 \\ 3.36479 0.0058623 \\ 3.37056 0.00584368 \\ 3.38011 0.00582354 \\ 3.38632 0.00581063 \\
      3.39477 0.00580001 \\ 3.39845 0.00578553 \\ 3.40551 0.00577174 \\ 3.41559 0.00575624 \\ 3.4363 0.00565256 \\ 3.44384 0.00562665 \\
      3.45236 0.00557191 \\ 3.45631 0.00555429 \\ 3.46366 0.00553098 \\ 3.47374 0.00550819 \\ 3.47768 0.00549614 \\ 3.48429 0.00547906 \\
      3.49451 0.00546064 \\ 3.50237 0.00544889 \\ 3.52421 0.00536766 \\ 3.53151 0.00534689 \\ 3.54018 0.00529488 \\ 3.5437 0.00527756 \\
      3.54954 0.0052555 \\ 3.5585 0.00523379 \\ 3.56198 0.00522208 \\ 3.56841 0.00520583 \\ 3.57843 0.00518822 \\ 3.58414 0.00517702 \\
      3.59363 0.00516786 \\ 3.597 0.00515498 \\ 3.60404 0.00514295 \\ 3.61282 0.00512945 \\ 3.63349 0.00504039 \\ 3.64062 0.00501895 \\
      3.64911 0.00497129 \\ 3.65301 0.00495573 \\ 3.65951 0.00493547 \\ 3.66813 0.00491568 \\ 3.67237 0.00490505 \\ 3.67851 0.00489023 \\
      3.6907 0.00487424 \\ 3.69801 0.0048641 \\ 3.71953 0.00479398 \\ 3.72664 0.00477644 \\ 3.73589 0.004731 \\ 3.7398 0.00471582 \\
      3.74593 0.00469663 \\ 3.75539 0.00467778 \\ 3.75906 0.0046675 \\ 3.76488 0.0046534 \\ 3.77622 0.00463812 \\ 3.78256 0.00462845 \\
      3.79104 0.00462056 \\ 3.79486 0.00460927 \\ 3.80306 0.00459884 \\ 3.81331 0.00458715 \\ 3.83221 0.00450955 \\ 3.83895 0.00449124 \\
      3.8475 0.00444941 \\ 3.85084 0.00443579 \\ 3.85721 0.00441812 \\ 3.86764 0.00440089 \\ 3.8725 0.00439157 \\ 3.879 0.00437867 \\
      3.89039 0.00436476 \\ 3.89747 0.00435596 \\ 3.92039 0.0042948 \\ 3.92818 0.00427975 \\ 3.93658 0.0042398 \\ 3.9411 0.00422654 \\
      3.94779 0.00420976 \\ 3.95681 0.00419333 \\ 3.96077 0.00418432 \\ 3.96702 0.00417202 \\ 3.97746 0.0041587 \\ 3.98402 0.00415028 \\
      3.99277 0.00414344 \\ 3.9968 0.00413354 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 0.00657158 2.15617 \\ 0.013018 1.90829 \\ 0.0193127 1.70543 \\ 0.0257121 1.50875 \\ 0.0324574 1.34768 \\
      0.0385604 1.22357 \\ 0.0474699 1.04786 \\ 0.0568907 0.903195 \\ 0.066352 0.785781 \\ 0.0785238 0.645765 \\ 0.0883262 0.586898 \\
      0.101875 0.500471 \\ 0.110973 0.46886 \\ 0.130684 0.365819 \\ 0.138663 0.353375 \\ 0.148232 0.33605 \\ 0.160858 0.296576 \\
      0.171716 0.27033 \\ 0.189032 0.203517 \\ 0.195792 0.19879 \\ 0.206971 0.183014 \\ 0.223253 0.143885 \\ 0.231267 0.14012 \\
      0.247357 0.118807 \\ 0.259219 0.114181 \\ 0.283974 0.0874135 \\ 0.291195 0.0864458 \\ 0.301306 0.0854621 \\ 0.314582 0.0733016 \\
      0.334941 0.0529782 \\ 0.343173 0.052438 \\ 0.355645 0.0507236 \\ 0.371189 0.0447546 \\ 0.38903 0.0296785 \\ 0.395755 0.0294101 \\
      0.404563 0.0291486 \\ 0.414519 0.02864 \\ 0.428561 0.0258711 \\ 0.450828 0.0176964 \\ 0.457723 0.0176265 \\ 0.467184 0.0175191 \\
      0.479153 0.0172687 \\ 0.492621 0.0160677 \\ 0.519148 0.00966453 \\ 0.526507 0.00962143 \\ 0.538475 0.00939913 \\ 0.551971 0.00861624 \\
      0.576944 0.0051708 \\ 0.583879 0.00514865 \\ 0.592874 0.00511592 \\ 0.603933 0.00504399 \\ 0.616388 0.00466851 \\ 0.641031 0.00251996 \\
      0.648256 0.00251086 \\ 0.657361 0.00249927 \\ 0.669159 0.00246938 \\ 0.682309 0.00233811 \\ 0.705699 0.000883204 \\ 0.71323 0.000880258 \\
      0.722211 0.000875852 \\ 0.734016 0.000861695 \\ 0.747904 0.000811354 \\ 0.775695 0.000295298 \\ 0.783619 0.000294649 \\ 0.792911 0.000293459 \\
      0.804322 0.000289499 \\ 0.817154 0.000276848 \\ 0.838208 5.82265e-05 \\ 0.844976 5.80942e-05 \\ 0.853279 5.782e-05 \\ 0.863909 5.70698e-05 \\
      0.876468 5.37935e-05 \\ 0.901033 5.25335e-06 \\ 0.90831 5.24328e-06 \\ 0.917486 5.22402e-06 \\ 0.92881 5.16688e-06 \\ 0.941927 4.83653e-06 \\
      0.958593 2.71006e-08 \\ 0.966444 2.68984e-08 \\ 0.975576 2.65219e-08 \\ 0.98708 2.5549e-08 \\ 1.0008 2.16667e-08 \\ 1.01422 1.00131e-12 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 0.014207 2.04261 \\ 0.0270202 1.66719 \\ 0.0394301 1.41893 \\ 0.0520846 1.21069 \\ 0.0654954 1.03654 \\
      0.0783087 0.882149 \\ 0.091297 0.735464 \\ 0.105715 0.602441 \\ 0.118535 0.483466 \\ 0.13249 0.378142 \\ 0.144595 0.287996 \\
      0.156914 0.22183 \\ 0.172681 0.168629 \\ 0.18736 0.1266 \\ 0.204624 0.100262 \\ 0.221444 0.079239 \\ 0.235408 0.0594962 \\
      0.249108 0.0447528 \\ 0.265357 0.0341128 \\ 0.281078 0.0247474 \\ 0.296953 0.0182319 \\ 0.312847 0.013138 \\ 0.325479 0.00852754 \\
      0.337947 0.00497867 \\ 0.356079 0.00301797 \\ 0.369241 0.00164924 \\ 0.381868 0.000772019 \\ 0.398489 0.000326264 \\ 0.414184 0.000122966 \\
      0.429753 2.73494e-05 \\ 0.444529 3.30271e-06 \\ 0.459579 1.10255e-07 \\ 0.474395 4.80382e-10 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 2.62004 \\ 0.00902808 2.11368 \\ 0.0183005 1.82008 \\ 0.0268793 1.57698 \\ 0.0356993 1.38079 \\ 0.0437612 1.2104 \\
      0.0525115 1.07496 \\ 0.0599446 0.95755 \\ 0.0689149 0.843897 \\ 0.076837 0.752349 \\ 0.0934599 0.594379 \\ 0.103623 0.548336 \\
      0.119916 0.426255 \\ 0.128278 0.397026 \\ 0.144268 0.299201 \\ 0.153688 0.279723 \\ 0.170085 0.205973 \\ 0.179252 0.197038 \\
      0.19424 0.142244 \\ 0.202199 0.136916 \\ 0.216641 0.0967403 \\ 0.225333 0.0937064 \\ 0.244861 0.0751158 \\ 0.25285 0.0741583 \\
      0.270975 0.0605005 \\ 0.280624 0.0598373 \\ 0.296568 0.0462955 \\ 0.307202 0.0458972 \\ 0.324134 0.0338068 \\ 0.333862 0.0334532 \\
      0.349599 0.0227015 \\ 0.359401 0.0223391 \\ 0.38175 0.0168823 \\ 0.390644 0.0167651 \\ 0.409263 0.0126302 \\ 0.418587 0.0125609 \\
      0.434269 0.0085827 \\ 0.442771 0.00853239 \\ 0.456716 0.00419159 \\ 0.465684 0.00413693 \\ 0.485535 0.00215738 \\ 0.495083 0.00214286 \\
      0.511404 0.000932538 \\ 0.521221 0.000924419 \\ 0.540785 0.000382321 \\ 0.550788 0.000380452 \\ 0.567929 0.000117205 \\ 0.57805 0.00011661 \\
      0.590997 1.7294e-05 \\ 0.600742 1.72194e-05 \\ 0.614204 5.58444e-07 \\ 0.623971 5.54315e-07 \\ 0.636831 1.67421e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(d) $\gamma=0.1$, runtime};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=558,
        ymin=1.479e-13, ymax=3.623e+00,
        ytickten={-12,-9,-6,-3,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 1 0.856823 \\ 2 0.819789 \\ 3 0.780921 \\ 4 0.739686 \\ 5 0.695712 \\
      6 0.620709 \\ 7 0.572333 \\ 8 0.492742 \\ 9 0.443235 \\ 10 0.367583 \\ 11 0.325908 \\
      12 0.263216 \\ 13 0.241106 \\ 14 0.214556 \\ 15 0.167784 \\ 16 0.15446 \\ 17 0.134533 \\
      18 0.12568 \\ 19 0.115778 \\ 20 0.104512 \\ 21 0.0633304 \\ 22 0.059111 \\ 23 0.0487491 \\
      24 0.0466881 \\ 25 0.0432648 \\ 26 0.0404797 \\ 27 0.0394271 \\ 28 0.0375916 \\ 29 0.0358542 \\
      30 0.0347966 \\ 31 0.0286345 \\ 32 0.0276409 \\ 33 0.0245935 \\ 34 0.0238228 \\ 35 0.022628 \\
      36 0.0215658 \\ 37 0.0211398 \\ 38 0.0204106 \\ 39 0.0196774 \\ 40 0.0192116 \\ 41 0.0188427 \\
      42 0.0184542 \\ 43 0.0180517 \\ 44 0.0176137 \\ 45 0.015076 \\ 46 0.0146481 \\ 47 0.0134842 \\
      48 0.0131769 \\ 49 0.012713 \\ 50 0.0122857 \\ 51 0.0120988 \\ 52 0.0117958 \\ 53 0.0114852 \\
      54 0.0112898 \\ 55 0.0102106 \\ 56 0.00960499 \\ 57 0.00941754 \\ 58 0.00903999 \\ 59 0.00874299 \\
      60 0.0086191 \\ 61 0.00840986 \\ 62 0.00820374 \\ 63 0.00810824 \\ 64 0.0079505 \\ 65 0.00778722 \\
      66 0.00706414 \\ 67 0.00690508 \\ 68 0.00651254 \\ 69 0.00638921 \\ 70 0.00622224 \\ 71 0.00606069 \\
      72 0.00598265 \\ 73 0.00586628 \\ 74 0.00574418 \\ 75 0.00566932 \\ 76 0.00560544 \\ 77 0.00550461 \\
      78 0.005435 \\ 79 0.00497253 \\ 80 0.00488538 \\ 81 0.00482592 \\ 82 0.00453822 \\ 83 0.00443128 \\
      84 0.00437624 \\ 85 0.00427362 \\ 86 0.0042011 \\ 87 0.00415939 \\ 88 0.00408374 \\ 89 0.00402639 \\
      90 0.00372687 \\ 91 0.00366729 \\ 92 0.00348034 \\ 93 0.0034243 \\ 94 0.00334921 \\ 95 0.00327563 \\
      96 0.00324086 \\ 97 0.00318877 \\ 98 0.00313353 \\ 99 0.00291071 \\ 100 0.00285849 \\ 101 0.00272596 \\
      102 0.00268267 \\ 103 0.00262536 \\ 104 0.00256873 \\ 105 0.00254198 \\ 106 0.00250152 \\ 107 0.00245878 \\
      108 0.00229299 \\ 109 0.00225092 \\ 110 0.0021515 \\ 111 0.00211816 \\ 112 0.00207391 \\ 113 0.00203025 \\
      114 0.00200978 \\ 115 0.00197839 \\ 116 0.0019454 \\ 117 0.0018186 \\ 118 0.00178511 \\ 119 0.00176624 \\
      120 0.00168191 \\ 121 0.00164643 \\ 122 0.00163032 \\ 123 0.00159656 \\ 124 0.00157238 \\ 125 0.00155504 \\
      126 0.00154007 \\ 127 0.0015215 \\ 128 0.00150198 \\ 129 0.00139525 \\ 130 0.00137001 \\ 131 0.00131202 \\
      132 0.001294 \\ 133 0.0012689 \\ 134 0.00124446 \\ 135 0.00123354 \\ 136 0.00121595 \\ 137 0.00119762 \\
      138 0.0011213 \\ 139 0.00110079 \\ 140 0.00105565 \\ 141 0.00104141 \\ 142 0.00102118 \\ 143 0.00100153 \\
      144 0.000989682 \\ 145 0.000980363 \\ 146 0.000965365 \\ 147 0.000954945 \\ 148 0.000897352 \\ 149 0.000884243 \\
      150 0.000847687 \\ 151 0.000836048 \\ 152 0.000820304 \\ 153 0.000805024 \\ 154 0.000795886 \\ 155 0.000788503 \\
      156 0.000776782 \\ 157 0.00076869 \\ 158 0.000723012 \\ 159 0.000712918 \\ 160 0.000705699 \\ 161 0.000673859 \\
      162 0.000660848 \\ 163 0.000655239 \\ 164 0.00064311 \\ 165 0.000634379 \\ 166 0.000628081 \\ 167 0.000590437 \\
      168 0.000580994 \\ 169 0.000558363 \\ 170 0.000550999 \\ 171 0.000540822 \\ 172 0.000530962 \\ 173 0.000524975 \\
      174 0.00052037 \\ 175 0.000512877 \\ 176 0.000507625 \\ 177 0.000478218 \\ 178 0.000471571 \\ 179 0.000452818 \\
      180 0.000446927 \\ 181 0.000438845 \\ 182 0.000431013 \\ 183 0.000426271 \\ 184 0.000422635 \\ 185 0.000416712 \\
      186 0.000412562 \\ 187 0.000388735 \\ 188 0.000383454 \\ 189 0.000368272 \\ 190 0.000363561 \\ 191 0.00035707 \\
      192 0.000350789 \\ 193 0.000346979 \\ 194 0.000344097 \\ 195 0.000339372 \\ 196 0.000336052 \\ 197 0.000316734 \\
      198 0.000312468 \\ 199 0.000300183 \\ 200 0.000296409 \\ 201 0.000291176 \\ 202 0.00028612 \\ 203 0.000283045 \\
      204 0.000280752 \\ 205 0.000276967 \\ 206 0.00026103 \\ 207 0.000256719 \\ 208 0.000247112 \\ 209 0.000244221 \\
      210 0.00023988 \\ 211 0.000235734 \\ 212 0.000233166 \\ 213 0.000231332 \\ 214 0.000228251 \\ 215 0.000226033 \\
      216 0.000213203 \\ 217 0.000210233 \\ 218 0.000202154 \\ 219 0.000199729 \\ 220 0.000196261 \\ 221 0.000192928 \\
      222 0.000190883 \\ 223 0.000189404 \\ 224 0.00018693 \\ 225 0.000176345 \\ 226 0.000173392 \\ 227 0.000167019 \\
      228 0.000165133 \\ 229 0.000162249 \\ 230 0.000159504 \\ 231 0.000157794 \\ 232 0.000156473 \\ 233 0.000155047 \\
      234 0.000146183 \\ 235 0.000143696 \\ 236 0.000138369 \\ 237 0.000136878 \\ 238 0.00013448 \\ 239 0.000132214 \\
      240 0.000130799 \\ 241 0.000129706 \\ 242 0.000128536 \\ 243 0.000121311 \\ 244 0.000119211 \\ 245 0.000114805 \\
      246 0.0001136 \\ 247 0.000111613 \\ 248 0.000109746 \\ 249 0.000108578 \\ 250 0.000107675 \\ 251 0.000106717 \\
      252 0.000100774 \\ 253 9.90096e-05 \\ 254 9.53707e-05 \\ 255 9.4387e-05 \\ 256 9.27422e-05 \\ 257 9.12045e-05 \\
      258 9.02407e-05 \\ 259 8.94953e-05 \\ 260 8.87088e-05 \\ 261 8.78468e-05 \\ 262 8.69436e-05 \\ 263 8.146e-05 \\
      264 8.00353e-05 \\ 265 7.7159e-05 \\ 266 7.63088e-05 \\ 267 7.50287e-05 \\ 268 7.38314e-05 \\ 269 7.30962e-05 \\
      270 7.25291e-05 \\ 271 7.18918e-05 \\ 272 6.7834e-05 \\ 273 6.66315e-05 \\ 274 6.42307e-05 \\ 275 6.3564e-05 \\
      276 6.24844e-05 \\ 277 6.14801e-05 \\ 278 6.08567e-05 \\ 279 6.03756e-05 \\ 280 5.9855e-05 \\ 281 5.65403e-05 \\
      282 5.55314e-05 \\ 283 5.3537e-05 \\ 284 5.29936e-05 \\ 285 5.20937e-05 \\ 286 5.12581e-05 \\ 287 5.07378e-05 \\
      288 5.03358e-05 \\ 289 4.99068e-05 \\ 290 4.71686e-05 \\ 291 4.6323e-05 \\ 292 4.46677e-05 \\ 293 4.42202e-05 \\
      294 4.34718e-05 \\ 295 4.27779e-05 \\ 296 4.23453e-05 \\ 297 4.20109e-05 \\ 298 4.16561e-05 \\ 299 3.93849e-05 \\
      300 3.86761e-05 \\ 301 3.73018e-05 \\ 302 3.69313e-05 \\ 303 3.63091e-05 \\ 304 3.5733e-05 \\ 305 3.53736e-05 \\
      306 3.50959e-05 \\ 307 3.48017e-05 \\ 308 3.29135e-05 \\ 309 3.23191e-05 \\ 310 3.11774e-05 \\ 311 3.08697e-05 \\
      312 3.03522e-05 \\ 313 2.98734e-05 \\ 314 2.9575e-05 \\ 315 2.93444e-05 \\ 316 2.78022e-05 \\ 317 2.73316e-05 \\
      318 2.64252e-05 \\ 319 2.61008e-05 \\ 320 2.56665e-05 \\ 321 2.52586e-05 \\ 322 2.50153e-05 \\ 323 2.48163e-05 \\
      324 2.45113e-05 \\ 325 2.42988e-05 \\ 326 2.29849e-05 \\ 327 2.26462e-05 \\ 328 2.18632e-05 \\ 329 2.16087e-05 \\
      330 2.12584e-05 \\ 331 2.09282e-05 \\ 332 2.07305e-05 \\ 333 1.97946e-05 \\ 334 1.95357e-05 \\ 335 1.93377e-05 \\
      336 1.85876e-05 \\ 337 1.82818e-05 \\ 338 1.81409e-05 \\ 339 1.78516e-05 \\ 340 1.76442e-05 \\ 341 1.74932e-05 \\
      342 1.65795e-05 \\ 343 1.63168e-05 \\ 344 1.57822e-05 \\ 345 1.55909e-05 \\ 346 1.5338e-05 \\ 347 1.50957e-05 \\
      348 1.49509e-05 \\ 349 1.48369e-05 \\ 350 1.46575e-05 \\ 351 1.4532e-05 \\ 352 1.37581e-05 \\ 353 1.35625e-05 \\
      354 1.30999e-05 \\ 355 1.29461e-05 \\ 356 1.27402e-05 \\ 357 1.25432e-05 \\ 358 1.24258e-05 \\ 359 1.23358e-05 \\
      360 1.22275e-05 \\ 361 1.15645e-05 \\ 362 1.1372e-05 \\ 363 1.09887e-05 \\ 364 1.08715e-05 \\ 365 1.06968e-05 \\
      366 1.05318e-05 \\ 367 1.04317e-05 \\ 368 1.03549e-05 \\ 369 1.02666e-05 \\ 370 9.71912e-06 \\ 371 9.55021e-06 \\
      372 9.2287e-06 \\ 373 9.13352e-06 \\ 374 8.9867e-06 \\ 375 8.84879e-06 \\ 376 8.76478e-06 \\ 377 8.70022e-06 \\
      378 8.62701e-06 \\ 379 8.1698e-06 \\ 380 8.02488e-06 \\ 381 7.75536e-06 \\ 382 7.67678e-06 \\ 383 7.55364e-06 \\
      384 7.43839e-06 \\ 385 7.36807e-06 \\ 386 7.31401e-06 \\ 387 6.94386e-06 \\ 388 6.82802e-06 \\ 389 6.76199e-06 \\
      390 6.51443e-06 \\ 391 6.41307e-06 \\ 392 6.36407e-06 \\ 393 6.26745e-06 \\ 394 6.1978e-06 \\ 395 6.14728e-06 \\
      396 5.83322e-06 \\ 397 5.73401e-06 \\ 398 5.55387e-06 \\ 399 5.48707e-06 \\ 400 5.40096e-06 \\ 401 5.3185e-06 \\
      402 5.26996e-06 \\ 403 5.03885e-06 \\ 404 4.97005e-06 \\ 405 4.91869e-06 \\ 406 4.73518e-06 \\ 407 4.66077e-06 \\
      408 4.62419e-06 \\ 409 4.55244e-06 \\ 410 4.50152e-06 \\ 411 4.46468e-06 \\ 412 4.2393e-06 \\ 413 4.17092e-06 \\
      414 4.04134e-06 \\ 415 3.99088e-06 \\ 416 3.9285e-06 \\ 417 3.86792e-06 \\ 418 3.83262e-06 \\ 419 3.80295e-06 \\
      420 3.75788e-06 \\ 421 3.72701e-06 \\ 422 3.53245e-06 \\ 423 3.4812e-06 \\ 424 3.36806e-06 \\ 425 3.32755e-06 \\
      426 3.27649e-06 \\ 427 3.22705e-06 \\ 428 3.19829e-06 \\ 429 3.05955e-06 \\ 430 3.01972e-06 \\ 431 2.98805e-06 \\
      432 2.87651e-06 \\ 433 2.83174e-06 \\ 434 2.80961e-06 \\ 435 2.76631e-06 \\ 436 2.73581e-06 \\ 437 2.7138e-06 \\
      438 2.57717e-06 \\ 439 2.53656e-06 \\ 440 2.4585e-06 \\ 441 2.42741e-06 \\ 442 2.39001e-06 \\ 443 2.35334e-06 \\
      444 2.33217e-06 \\ 445 2.3141e-06 \\ 446 2.28683e-06 \\ 447 2.26831e-06 \\ 448 2.15077e-06 \\ 449 2.11989e-06 \\
      450 2.05178e-06 \\ 451 2.0268e-06 \\ 452 1.9961e-06 \\ 453 1.96613e-06 \\ 454 1.94884e-06 \\ 455 1.86504e-06 \\
      456 1.84093e-06 \\ 457 1.82139e-06 \\ 458 1.75382e-06 \\ 459 1.72686e-06 \\ 460 1.71327e-06 \\ 461 1.68703e-06 \\
      462 1.66864e-06 \\ 463 1.65541e-06 \\ 464 1.57265e-06 \\ 465 1.54804e-06 \\ 466 1.5331e-06 \\ 467 1.47804e-06 \\
      468 1.45551e-06 \\ 469 1.44024e-06 \\ 470 1.42745e-06 \\ 471 1.41233e-06 \\ 472 1.39667e-06 \\ 473 1.32667e-06 \\
      474 1.30458e-06 \\ 475 1.26478e-06 \\ 476 1.24948e-06 \\ 477 1.23057e-06 \\ 478 1.21196e-06 \\ 479 1.2012e-06 \\
      480 1.14985e-06 \\ 481 1.13435e-06 \\ 482 1.12245e-06 \\ 483 1.08145e-06 \\ 484 1.065e-06 \\ 485 1.05658e-06 \\
      486 1.04045e-06 \\ 487 1.02916e-06 \\ 488 1.02105e-06 \\ 489 9.70702e-07 \\ 490 9.55258e-07 \\ 491 9.46003e-07 \\
      492 9.12377e-07 \\ 493 8.98623e-07 \\ 494 8.89303e-07 \\ 495 8.81346e-07 \\ 496 8.72062e-07 \\ 497 8.62414e-07 \\
      498 8.19669e-07 \\ 499 8.05873e-07 \\ 500 7.81657e-07 \\ 501 7.72113e-07 \\ 502 7.60534e-07 \\ 503 7.4908e-07 \\
      504 7.42502e-07 \\ 505 7.10935e-07 \\ 506 7.01268e-07 \\ 507 6.93837e-07 \\ 508 6.68692e-07 \\ 509 6.58639e-07 \\
      510 6.53379e-07 \\ 511 6.43454e-07 \\ 512 6.36532e-07 \\ 513 6.3157e-07 \\ 514 6.00712e-07 \\ 515 5.91107e-07 \\
      516 5.85311e-07 \\ 517 5.64662e-07 \\ 518 5.56245e-07 \\ 519 5.52028e-07 \\ 520 5.43841e-07 \\ 521 5.3804e-07 \\
      522 5.33873e-07 \\ 523 5.07537e-07 \\ 524 4.99114e-07 \\ 525 4.84399e-07 \\ 526 4.78222e-07 \\ 527 4.71141e-07 \\
      528 4.64079e-07 \\ 529 4.60081e-07 \\ 530 4.56511e-07 \\ 531 4.5124e-07 \\ 532 4.4771e-07 \\ 533 4.24757e-07 \\
      534 4.18471e-07 \\ 535 4.05559e-07 \\ 536 4.00533e-07 \\ 537 3.94687e-07 \\ 538 3.88876e-07 \\ 539 3.85599e-07 \\
      540 3.69265e-07 \\ 541 3.65018e-07 \\ 542 3.598e-07 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 1 0.85324 \\ 2 0.812681 \\ 3 0.770475 \\ 4 0.726512 \\ 5 0.681145 \\
      6 0.635411 \\ 7 0.566084 \\ 8 0.502384 \\ 9 0.446006 \\ 10 0.362573 \\ 11 0.296648 \\
      12 0.24667 \\ 13 0.157121 \\ 14 0.133603 \\ 15 0.0855638 \\ 16 0.0613181 \\ 17 0.0270768 \\
      18 0.0266708 \\ 19 0.0245764 \\ 20 0.0193805 \\ 21 0.00375514 \\ 22 0.00368098 \\ 23 0.00357297 \\
      24 0.00335934 \\ 25 0.00282382 \\ 26 5.70838e-05 \\ 27 5.29865e-05 \\ 28 4.84e-05 \\ 29 4.27866e-05 \\
      30 3.35564e-05 \\ 31 1.00816e-08 \\ 32 9.26111e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 1 0.82317 \\ 2 0.759037 \\ 3 0.695303 \\ 4 0.629444 \\ 5 0.560673 \\
      6 0.489551 \\ 7 0.417474 \\ 8 0.346436 \\ 9 0.27886 \\ 10 0.217275 \\ 11 0.16391 \\
      12 0.119085 \\ 13 0.0825614 \\ 14 0.0541362 \\ 15 0.0329191 \\ 16 0.0182497 \\ 17 0.00904316 \\
      18 0.00371144 \\ 19 0.00107568 \\ 20 0.000175038 \\ 21 1.08363e-05 \\ 22 1.29045e-07 \\ 23 1.01554e-10 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 1 0.842268 \\ 2 0.792419 \\ 3 0.741548 \\ 4 0.689009 \\ 5 0.634915 \\
      6 0.580158 \\ 7 0.526202 \\ 8 0.47473 \\ 9 0.427613 \\ 10 0.309416 \\ 11 0.277269 \\
      12 0.180364 \\ 13 0.163865 \\ 14 0.0958351 \\ 15 0.0898533 \\ 16 0.0450625 \\ 17 0.0431453 \\
      18 0.0171994 \\ 19 0.0166538 \\ 20 0.00467334 \\ 21 0.0045521 \\ 22 0.000567494 \\ 23 0.00055295 \\
      24 1.49448e-05 \\ 25 1.44797e-05 \\ 26 2.79147e-08 \\ 27 2.71205e-08 \\ 28 6.00409e-13 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(e) $\gamma=0.5$, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=4.12,
        ymin=1.479e-13, ymax=3.623e+00,
        ytickten={-12,-9,-6,-3,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 0.00697496 0.856823 \\ 0.0131778 0.819789 \\ 0.0192875 0.780921 \\ 0.0254993 0.739686 \\ 0.0311688 0.695712 \\
      0.0389777 0.620709 \\ 0.04533 0.572333 \\ 0.0529931 0.492742 \\ 0.0590774 0.443235 \\ 0.0668218 0.367583 \\ 0.0732781 0.325908 \\
      0.0813902 0.263216 \\ 0.0847307 0.241106 \\ 0.0919553 0.214556 \\ 0.0999307 0.167784 \\ 0.106964 0.15446 \\ 0.115429 0.134533 \\
      0.119012 0.12568 \\ 0.125831 0.115778 \\ 0.133979 0.104512 \\ 0.144641 0.0633304 \\ 0.151134 0.059111 \\ 0.160334 0.0487491 \\
      0.163846 0.0466881 \\ 0.170481 0.0432648 \\ 0.179433 0.0404797 \\ 0.182846 0.0394271 \\ 0.189172 0.0375916 \\ 0.197599 0.0358542 \\
      0.204009 0.0347966 \\ 0.222083 0.0286345 \\ 0.228785 0.0276409 \\ 0.236385 0.0245935 \\ 0.239965 0.0238228 \\ 0.247049 0.022628 \\
      0.255813 0.0215658 \\ 0.259018 0.0211398 \\ 0.265007 0.0204106 \\ 0.275486 0.0196774 \\ 0.281737 0.0192116 \\ 0.290766 0.0188427 \\
      0.294161 0.0184542 \\ 0.301624 0.0180517 \\ 0.311941 0.0176137 \\ 0.331209 0.015076 \\ 0.337947 0.0146481 \\ 0.347236 0.0134842 \\
      0.350785 0.0131769 \\ 0.357303 0.012713 \\ 0.365333 0.0122857 \\ 0.369328 0.0120988 \\ 0.377402 0.0117958 \\ 0.386612 0.0114852 \\
      0.394443 0.0112898 \\ 0.412037 0.0102106 \\ 0.420277 0.00960499 \\ 0.424229 0.00941754 \\ 0.430685 0.00903999 \\ 0.440707 0.00874299 \\
      0.444526 0.0086191 \\ 0.450773 0.00840986 \\ 0.461136 0.00820374 \\ 0.464725 0.00810824 \\ 0.471318 0.0079505 \\ 0.480412 0.00778722 \\
      0.498797 0.00706414 \\ 0.506876 0.00690508 \\ 0.514625 0.00651254 \\ 0.518305 0.00638921 \\ 0.525455 0.00622224 \\ 0.535767 0.00606069 \\
      0.539575 0.00598265 \\ 0.548076 0.00586628 \\ 0.558609 0.00574418 \\ 0.564621 0.00566932 \\ 0.568746 0.00560544 \\ 0.580814 0.00550461 \\
      0.587867 0.005435 \\ 0.605813 0.00497253 \\ 0.612709 0.00488538 \\ 0.616085 0.00482592 \\ 0.624823 0.00453822 \\ 0.629944 0.00443128 \\
      0.633812 0.00437624 \\ 0.642612 0.00427362 \\ 0.648604 0.0042011 \\ 0.651864 0.00415939 \\ 0.663588 0.00408374 \\ 0.670272 0.00402639 \\
      0.687868 0.00372687 \\ 0.69415 0.00366729 \\ 0.702047 0.00348034 \\ 0.706303 0.0034243 \\ 0.712609 0.00334921 \\ 0.722473 0.00327563 \\
      0.726455 0.00324086 \\ 0.733182 0.00318877 \\ 0.743842 0.00313353 \\ 0.763036 0.00291071 \\ 0.770059 0.00285849 \\ 0.77849 0.00272596 \\
      0.781921 0.00268267 \\ 0.788329 0.00262536 \\ 0.797781 0.00256873 \\ 0.801843 0.00254198 \\ 0.807558 0.00250152 \\ 0.817351 0.00245878 \\
      0.835131 0.00229299 \\ 0.841651 0.00225092 \\ 0.84971 0.0021515 \\ 0.853428 0.00211816 \\ 0.858905 0.00207391 \\ 0.867273 0.00203025 \\
      0.870622 0.00200978 \\ 0.876817 0.00197839 \\ 0.887377 0.0019454 \\ 0.904332 0.0018186 \\ 0.911611 0.00178511 \\ 0.914987 0.00176624 \\
      0.925061 0.00168191 \\ 0.931142 0.00164643 \\ 0.935101 0.00163032 \\ 0.94404 0.00159656 \\ 0.949586 0.00157238 \\ 0.957961 0.00155504 \\
      0.96139 0.00154007 \\ 0.967667 0.0015215 \\ 0.975554 0.00150198 \\ 0.994982 0.00139525 \\ 1.00171 0.00137001 \\ 1.01013 0.00131202 \\
      1.01372 0.001294 \\ 1.01986 0.0012689 \\ 1.02946 0.00124446 \\ 1.03237 0.00123354 \\ 1.03846 0.00121595 \\ 1.04623 0.00119762 \\
      1.06284 0.0011213 \\ 1.0688 0.00110079 \\ 1.07703 0.00105565 \\ 1.08071 0.00104141 \\ 1.0872 0.00102118 \\ 1.09521 0.00100153 \\
      1.10188 0.000989682 \\ 1.10526 0.000980363 \\ 1.11305 0.000965365 \\ 1.1197 0.000954945 \\ 1.13755 0.000897352 \\ 1.14419 0.000884243 \\
      1.15208 0.000847687 \\ 1.15603 0.000836048 \\ 1.16276 0.000820304 \\ 1.17099 0.000805024 \\ 1.17724 0.000795886 \\ 1.18075 0.000788503 \\
      1.18981 0.000776782 \\ 1.19712 0.00076869 \\ 1.21281 0.000723012 \\ 1.22049 0.000712918 \\ 1.22448 0.000705699 \\ 1.23479 0.000673859 \\
      1.24151 0.000660848 \\ 1.24491 0.000655239 \\ 1.25272 0.00064311 \\ 1.25878 0.000634379 \\ 1.26745 0.000628081 \\ 1.28215 0.000590437 \\
      1.28882 0.000580994 \\ 1.29685 0.000558363 \\ 1.3004 0.000550999 \\ 1.30723 0.000540822 \\ 1.31491 0.000530962 \\ 1.32156 0.000524975 \\
      1.32569 0.00052037 \\ 1.33345 0.000512877 \\ 1.34015 0.000507625 \\ 1.35422 0.000478218 \\ 1.36206 0.000471571 \\ 1.37043 0.000452818 \\
      1.37492 0.000446927 \\ 1.38112 0.000438845 \\ 1.39073 0.000431013 \\ 1.39752 0.000426271 \\ 1.4016 0.000422635 \\ 1.41042 0.000416712 \\
      1.41707 0.000412562 \\ 1.43053 0.000388735 \\ 1.43752 0.000383454 \\ 1.44514 0.000368272 \\ 1.44899 0.000363561 \\ 1.45538 0.00035707 \\
      1.46352 0.000350789 \\ 1.47033 0.000346979 \\ 1.47373 0.000344097 \\ 1.4813 0.000339372 \\ 1.48773 0.000336052 \\ 1.50042 0.000316734 \\
      1.50701 0.000312468 \\ 1.51542 0.000300183 \\ 1.51957 0.000296409 \\ 1.52667 0.000291176 \\ 1.53485 0.00028612 \\ 1.54208 0.000283045 \\
      1.54634 0.000280752 \\ 1.55431 0.000276967 \\ 1.56683 0.00026103 \\ 1.57388 0.000256719 \\ 1.58225 0.000247112 \\ 1.58572 0.000244221 \\
      1.59314 0.00023988 \\ 1.60118 0.000235734 \\ 1.60813 0.000233166 \\ 1.61182 0.000231332 \\ 1.61952 0.000228251 \\ 1.6257 0.000226033 \\
      1.63569 0.000213203 \\ 1.64344 0.000210233 \\ 1.65097 0.000202154 \\ 1.6549 0.000199729 \\ 1.66229 0.000196261 \\ 1.67061 0.000192928 \\
      1.67666 0.000190883 \\ 1.67975 0.000189404 \\ 1.68725 0.00018693 \\ 1.69721 0.000176345 \\ 1.70405 0.000173392 \\ 1.71325 0.000167019 \\
      1.71647 0.000165133 \\ 1.72369 0.000162249 \\ 1.73201 0.000159504 \\ 1.73911 0.000157794 \\ 1.74803 0.000156473 \\ 1.75203 0.000155047 \\
      1.76271 0.000146183 \\ 1.77013 0.000143696 \\ 1.77918 0.000138369 \\ 1.78319 0.000136878 \\ 1.78939 0.00013448 \\ 1.79852 0.000132214 \\
      1.80507 0.000130799 \\ 1.81394 0.000129706 \\ 1.81739 0.000128536 \\ 1.82833 0.000121311 \\ 1.83481 0.000119211 \\ 1.84349 0.000114805 \\
      1.84752 0.0001136 \\ 1.85389 0.000111613 \\ 1.86233 0.000109746 \\ 1.86901 0.000108578 \\ 1.87759 0.000107675 \\ 1.88058 0.000106717 \\
      1.8911 0.000100774 \\ 1.89702 9.90096e-05 \\ 1.90523 9.53707e-05 \\ 1.90886 9.4387e-05 \\ 1.91453 9.27422e-05 \\ 1.9231 9.12045e-05 \\
      1.92933 9.02407e-05 \\ 1.93905 8.94953e-05 \\ 1.94279 8.87088e-05 \\ 1.94978 8.78468e-05 \\ 1.95876 8.69436e-05 \\ 1.96979 8.146e-05 \\
      1.97625 8.00353e-05 \\ 1.98512 7.7159e-05 \\ 1.98885 7.63088e-05 \\ 1.99513 7.50287e-05 \\ 2.00325 7.38314e-05 \\ 2.01053 7.30962e-05 \\
      2.01895 7.25291e-05 \\ 2.0223 7.18918e-05 \\ 2.03212 6.7834e-05 \\ 2.03942 6.66315e-05 \\ 2.0481 6.42307e-05 \\ 2.0513 6.3564e-05 \\
      2.0578 6.24844e-05 \\ 2.06588 6.14801e-05 \\ 2.07248 6.08567e-05 \\ 2.08077 6.03756e-05 \\ 2.084 5.9855e-05 \\ 2.09412 5.65403e-05 \\
      2.10053 5.55314e-05 \\ 2.10877 5.3537e-05 \\ 2.11192 5.29936e-05 \\ 2.1184 5.20937e-05 \\ 2.1271 5.12581e-05 \\ 2.13319 5.07378e-05 \\
      2.14177 5.03358e-05 \\ 2.14574 4.99068e-05 \\ 2.15633 4.71686e-05 \\ 2.1625 4.6323e-05 \\ 2.17263 4.46677e-05 \\ 2.17631 4.42202e-05 \\
      2.18283 4.34718e-05 \\ 2.19135 4.27779e-05 \\ 2.19815 4.23453e-05 \\ 2.20694 4.20109e-05 \\ 2.21051 4.16561e-05 \\ 2.22075 3.93849e-05 \\
      2.22859 3.86761e-05 \\ 2.23662 3.73018e-05 \\ 2.24055 3.69313e-05 \\ 2.24654 3.63091e-05 \\ 2.25499 3.5733e-05 \\ 2.26086 3.53736e-05 \\
      2.26833 3.50959e-05 \\ 2.27202 3.48017e-05 \\ 2.28141 3.29135e-05 \\ 2.28822 3.23191e-05 \\ 2.2968 3.11774e-05 \\ 2.30044 3.08697e-05 \\
      2.30696 3.03522e-05 \\ 2.31497 2.98734e-05 \\ 2.32121 2.9575e-05 \\ 2.32931 2.93444e-05 \\ 2.34109 2.78022e-05 \\ 2.34835 2.73316e-05 \\
      2.3567 2.64252e-05 \\ 2.36013 2.61008e-05 \\ 2.36693 2.56665e-05 \\ 2.37594 2.52586e-05 \\ 2.38246 2.50153e-05 \\ 2.3863 2.48163e-05 \\
      2.39426 2.45113e-05 \\ 2.4005 2.42988e-05 \\ 2.41222 2.29849e-05 \\ 2.4189 2.26462e-05 \\ 2.42811 2.18632e-05 \\ 2.43165 2.16087e-05 \\
      2.43884 2.12584e-05 \\ 2.50016 2.09282e-05 \\ 2.50705 2.07305e-05 \\ 2.51752 1.97946e-05 \\ 2.52374 1.95357e-05 \\ 2.52675 1.93377e-05 \\
      2.53466 1.85876e-05 \\ 2.54159 1.82818e-05 \\ 2.54591 1.81409e-05 \\ 2.55394 1.78516e-05 \\ 2.56087 1.76442e-05 \\ 2.56932 1.74932e-05 \\
      2.58031 1.65795e-05 \\ 2.5871 1.63168e-05 \\ 2.59633 1.57822e-05 \\ 2.60012 1.55909e-05 \\ 2.60677 1.5338e-05 \\ 2.61431 1.50957e-05 \\
      2.62133 1.49509e-05 \\ 2.62578 1.48369e-05 \\ 2.63372 1.46575e-05 \\ 2.6401 1.4532e-05 \\ 2.65002 1.37581e-05 \\ 2.6571 1.35625e-05 \\
      2.66462 1.30999e-05 \\ 2.66851 1.29461e-05 \\ 2.67511 1.27402e-05 \\ 2.68409 1.25432e-05 \\ 2.69042 1.24258e-05 \\ 2.69804 1.23358e-05 \\
      2.70217 1.22275e-05 \\ 2.71192 1.15645e-05 \\ 2.71876 1.1372e-05 \\ 2.72682 1.09887e-05 \\ 2.73036 1.08715e-05 \\ 2.73653 1.06968e-05 \\
      2.74505 1.05318e-05 \\ 2.75175 1.04317e-05 \\ 2.76131 1.03549e-05 \\ 2.7656 1.02666e-05 \\ 2.77743 9.71912e-06 \\ 2.78461 9.55021e-06 \\
      2.79388 9.2287e-06 \\ 2.79775 9.13352e-06 \\ 2.80512 8.9867e-06 \\ 2.81393 8.84879e-06 \\ 2.82 8.76478e-06 \\ 2.82964 8.70022e-06 \\
      2.83354 8.62701e-06 \\ 2.84353 8.1698e-06 \\ 2.85029 8.02488e-06 \\ 2.85899 7.75536e-06 \\ 2.86222 7.67678e-06 \\ 2.86858 7.55364e-06 \\
      2.87778 7.43839e-06 \\ 2.88309 7.36807e-06 \\ 2.89191 7.31401e-06 \\ 2.90256 6.94386e-06 \\ 2.90951 6.82802e-06 \\ 2.91257 6.76199e-06 \\
      2.91941 6.51443e-06 \\ 2.92585 6.41307e-06 \\ 2.92898 6.36407e-06 \\ 2.93697 6.26745e-06 \\ 2.94337 6.1978e-06 \\ 2.95209 6.14728e-06 \\
      2.9627 5.83322e-06 \\ 2.97034 5.73401e-06 \\ 2.97916 5.55387e-06 \\ 2.98311 5.48707e-06 \\ 2.99 5.40096e-06 \\ 2.99879 5.3185e-06 \\
      3.00537 5.26996e-06 \\ 3.01573 5.03885e-06 \\ 3.02318 4.97005e-06 \\ 3.02733 4.91869e-06 \\ 3.03602 4.73518e-06 \\ 3.04313 4.66077e-06 \\
      3.04666 4.62419e-06 \\ 3.05427 4.55244e-06 \\ 3.06048 4.50152e-06 \\ 3.06809 4.46468e-06 \\ 3.07828 4.2393e-06 \\ 3.08474 4.17092e-06 \\
      3.09337 4.04134e-06 \\ 3.09659 3.99088e-06 \\ 3.10348 3.9285e-06 \\ 3.11191 3.86792e-06 \\ 3.11853 3.83262e-06 \\ 3.1216 3.80295e-06 \\
      3.12929 3.75788e-06 \\ 3.13639 3.72701e-06 \\ 3.146 3.53245e-06 \\ 3.15348 3.4812e-06 \\ 3.16178 3.36806e-06 \\ 3.16512 3.32755e-06 \\
      3.17246 3.27649e-06 \\ 3.18089 3.22705e-06 \\ 3.18809 3.19829e-06 \\ 3.19847 3.05955e-06 \\ 3.20518 3.01972e-06 \\ 3.2092 2.98805e-06 \\
      3.21693 2.87651e-06 \\ 3.22455 2.83174e-06 \\ 3.22821 2.80961e-06 \\ 3.23642 2.76631e-06 \\ 3.24337 2.73581e-06 \\ 3.25265 2.7138e-06 \\
      3.26282 2.57717e-06 \\ 3.2695 2.53656e-06 \\ 3.27836 2.4585e-06 \\ 3.28185 2.42741e-06 \\ 3.28823 2.39001e-06 \\ 3.29642 2.35334e-06 \\
      3.30307 2.33217e-06 \\ 3.30657 2.3141e-06 \\ 3.31432 2.28683e-06 \\ 3.32154 2.26831e-06 \\ 3.33344 2.15077e-06 \\ 3.34108 2.11989e-06 \\
      3.34971 2.05178e-06 \\ 3.35432 2.0268e-06 \\ 3.36146 1.9961e-06 \\ 3.36974 1.96613e-06 \\ 3.37655 1.94884e-06 \\ 3.38758 1.86504e-06 \\
      3.39523 1.84093e-06 \\ 3.39844 1.82139e-06 \\ 3.40753 1.75382e-06 \\ 3.41519 1.72686e-06 \\ 3.4189 1.71327e-06 \\ 3.42768 1.68703e-06 \\
      3.43388 1.66864e-06 \\ 3.44263 1.65541e-06 \\ 3.4535 1.57265e-06 \\ 3.46135 1.54804e-06 \\ 3.46488 1.5331e-06 \\ 3.47293 1.47804e-06 \\
      3.47956 1.45551e-06 \\ 3.48842 1.44024e-06 \\ 3.49338 1.42745e-06 \\ 3.49908 1.41233e-06 \\ 3.50799 1.39667e-06 \\ 3.51867 1.32667e-06 \\
      3.5254 1.30458e-06 \\ 3.53376 1.26478e-06 \\ 3.53781 1.24948e-06 \\ 3.54423 1.23057e-06 \\ 3.55242 1.21196e-06 \\ 3.55954 1.2012e-06 \\
      3.57042 1.14985e-06 \\ 3.57749 1.13435e-06 \\ 3.58094 1.12245e-06 \\ 3.58982 1.08145e-06 \\ 3.59638 1.065e-06 \\ 3.60067 1.05658e-06 \\
      3.61028 1.04045e-06 \\ 3.61678 1.02916e-06 \\ 3.62654 1.02105e-06 \\ 3.63732 9.70702e-07 \\ 3.64406 9.55258e-07 \\ 3.64782 9.46003e-07 \\
      3.65648 9.12377e-07 \\ 3.66322 8.98623e-07 \\ 3.67162 8.89303e-07 \\ 3.67575 8.81346e-07 \\ 3.68162 8.72062e-07 \\ 3.69007 8.62414e-07 \\
      3.70095 8.19669e-07 \\ 3.70736 8.05873e-07 \\ 3.71463 7.81657e-07 \\ 3.71836 7.72113e-07 \\ 3.7246 7.60534e-07 \\ 3.73302 7.4908e-07 \\
      3.73958 7.42502e-07 \\ 3.75033 7.10935e-07 \\ 3.75694 7.01268e-07 \\ 3.76072 6.93837e-07 \\ 3.76875 6.68692e-07 \\ 3.77536 6.58639e-07 \\
      3.77889 6.53379e-07 \\ 3.78689 6.43454e-07 \\ 3.79329 6.36532e-07 \\ 3.8016 6.3157e-07 \\ 3.81288 6.00712e-07 \\ 3.82034 5.91107e-07 \\
      3.82416 5.85311e-07 \\ 3.83142 5.64662e-07 \\ 3.83719 5.56245e-07 \\ 3.84108 5.52028e-07 \\ 3.84915 5.43841e-07 \\ 3.85587 5.3804e-07 \\
      3.86378 5.33873e-07 \\ 3.87418 5.07537e-07 \\ 3.88059 4.99114e-07 \\ 3.88892 4.84399e-07 \\ 3.89256 4.78222e-07 \\ 3.89844 4.71141e-07 \\
      3.90593 4.64079e-07 \\ 3.91237 4.60081e-07 \\ 3.91568 4.56511e-07 \\ 3.9236 4.5124e-07 \\ 3.92975 4.4771e-07 \\ 3.93968 4.24757e-07 \\
      3.94616 4.18471e-07 \\ 3.95424 4.05559e-07 \\ 3.95772 4.00533e-07 \\ 3.96348 3.94687e-07 \\ 3.9712 3.88876e-07 \\ 3.97829 3.85599e-07 \\
      3.98822 3.69265e-07 \\ 3.99173 3.65018e-07 \\ 3.99823 3.598e-07 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 0.00688712 0.85324 \\ 0.0136891 0.812681 \\ 0.0199933 0.770475 \\ 0.0261077 0.726512 \\ 0.0323998 0.681145 \\
      0.0385642 0.635411 \\ 0.0477402 0.566084 \\ 0.0558909 0.502384 \\ 0.064093 0.446006 \\ 0.0744898 0.362573 \\ 0.0845359 0.296648 \\
      0.0951073 0.24667 \\ 0.108733 0.157121 \\ 0.119554 0.133603 \\ 0.132151 0.0855638 \\ 0.14559 0.0613181 \\ 0.15921 0.0270768 \\
      0.167067 0.0266708 \\ 0.176746 0.0245764 \\ 0.190156 0.0193805 \\ 0.203518 0.00375514 \\ 0.210432 0.00368098 \\ 0.219042 0.00357297 \\
      0.229273 0.00335934 \\ 0.241444 0.00282382 \\ 0.254436 5.70838e-05 \\ 0.261698 5.29865e-05 \\ 0.270089 4.84e-05 \\ 0.279931 4.27866e-05 \\
      0.292338 3.35564e-05 \\ 0.305607 1.00816e-08 \\ 0.312259 9.26111e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 0.0139959 0.82317 \\ 0.0265816 0.759037 \\ 0.0392865 0.695303 \\ 0.0524117 0.629444 \\ 0.0650371 0.560673 \\
      0.077461 0.489551 \\ 0.0895497 0.417474 \\ 0.101996 0.346436 \\ 0.114094 0.27886 \\ 0.126216 0.217275 \\ 0.138471 0.16391 \\
      0.151115 0.119085 \\ 0.163475 0.0825614 \\ 0.175163 0.0541362 \\ 0.188523 0.0329191 \\ 0.202135 0.0182497 \\ 0.21453 0.00904316 \\
      0.226495 0.00371144 \\ 0.238372 0.00107568 \\ 0.250852 0.000175038 \\ 0.264323 1.08363e-05 \\ 0.276787 1.29045e-07 \\ 0.288616 1.01554e-10 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.892289 \\ 0.00787263 0.842268 \\ 0.0150669 0.792419 \\ 0.0223075 0.741548 \\ 0.0294364 0.689009 \\ 0.0364162 0.634915 \\
      0.043769 0.580158 \\ 0.0510798 0.526202 \\ 0.0586214 0.47473 \\ 0.0659071 0.427613 \\ 0.0774533 0.309416 \\ 0.0856454 0.277269 \\
      0.0984259 0.180364 \\ 0.106964 0.163865 \\ 0.119165 0.0958351 \\ 0.126895 0.0898533 \\ 0.139548 0.0450625 \\ 0.147037 0.0431453 \\
      0.160002 0.0171994 \\ 0.167781 0.0166538 \\ 0.179731 0.00467334 \\ 0.188008 0.0045521 \\ 0.199702 0.000567494 \\ 0.207242 0.00055295 \\
      0.218906 1.49448e-05 \\ 0.226653 1.44797e-05 \\ 0.238127 2.79147e-08 \\ 0.245397 2.71205e-08 \\ 0.257856 6.00409e-13 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(f) $\gamma=0.5$, runtime};

Figure 1. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) for the log-sum-exp objective with γ∈{0.05,0.1,0.5}\gamma\in\{0.05,0.1,0.5\}. The left column shows the objective gap f(xk)−f∗f(x_k)-f^* versus iterations, while the right column shows the objective gap versus wall-clock time. Row-nested MLRN and TLRN methods demonstrate comparable convergence behavior to RN.

In this section, we present numerical experiments to support our theoretical findings. In particular, we compare the performance of the two-level regularized Newton (TLRN), multilevel regularized Newton (MLRN), and row-nested multilevel regularized Newton (MLRN nested) methods with that of the full-space gradient-regularized Newton method (RN) [13]. Although our methods retain the O(k−2)\mathcal{O}\big(k^{-2}\big) convergence rate of RN, the constants in their complexity bounds depend on σ<1\sigma<1, see lemma:case-II, potentially leading to less favorable theoretical bounds. Nevertheless, since the multilevel methods compute some of their updates in lower-dimensional spaces, they may offer significant computational savings. Our experiments aim to illustrate that the multilevel methods exhibit convergence behavior comparable to RN while potentially reducing the overall computational time.

The restriction operators for MLRN are constructed by selecting rows of the identity matrix as in [19], whereas for the row-nested method we use the construction described in rem:nested-orthonormal. We consider ℓ=5\ell=5 coarse levels with dimensions n1=0.1nn_1=0.1n, n2=0.3nn_2=0.3n, n3=0.5nn_3=0.5n, n4=0.7nn_4=0.7n, and n5=0.9nn_5=0.9n, whereas for TLRN we select d=0.5nd=0.5n.

Since the Hessian’s Lipschitz constant LL is generally unknown, we employ an adaptive line-search procedure, following the approach of [7 Section 4.1.4.2] and [13]. At each iteration, the estimate LkL_k is increased until the corresponding sufficient-decrease and gradient-stability conditions are satisfied. We set ε=10−10\varepsilon=10^{-10} and terminate the algorithms when f(xk)−f∗≤εff(x_k)-f^* \leq \varepsilon_f or ∥∇f(xk)∥≤εg\left\lVert \nabla f(x_k) \right\rVert\leq\varepsilon_g, where εf=10−8\varepsilon_f=10^{-8} and εg=ε\varepsilon_g=\varepsilon. We use L-BFGS to compute a numerical approximation of the optimal objective value f∗f^*. We set σ=0.5\sigma=0.5, balancing the frequency of coarse corrections against their effectiveness (note that larger values of σ\sigma yield more favorable convergence constants but impose a stricter coarse-level eligibility condition, potentially resulting in more fine-level iterations). All experiments are initialized at the origin.

Log-sum-exp

Given data a1,…,am∈Rna_1,\ldots,a_m\in\mathbb{R}^n and b1,…,bm∈Rb_1,\ldots,b_m\in\mathbb{R}, we consider minimizing the log-sum-exp function

fγ(x)=γln⁡(∑i=1mexp⁡(ai⊤x−biγ)),γ>0.\begin{equation*} f_\gamma(x) = \gamma \ln\left( \sum_{i=1}^{m} \exp\left(\frac{a_i^\top x-b_i}{\gamma}\right) \right), \qquad \gamma>0. \end{equation*}

The parameter γ\gamma controls the smoothing of the log-sum-exp objective. We generate a random dataset with m=1000m=1000 observations and n=800n=800 variables and consider three values of this parameter, namely γ∈{0.05,0.1,0.5}\gamma\in\{0.05,0.1,0.5\}, to investigate the sensitivity of the algorithms to different degrees of smoothing.

The results are presented in Figure 1. For all three values of γ\gamma, row-nested MLRN and TLRN exhibit convergence behavior comparable to that of RN, both in terms of iterations and computational time. In contrast, the non-nested MLRN method converges substantially more slowly. We suspect that this difference may be related to the fine-iteration frequency condition in Theorem 2.5, which need not be satisfied during the observed iterations. Nevertheless, these findings suggest that row-nested restriction operators can be an attractive choice from both theoretical and computational perspectives.

leg:gisette-duke

\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=35,
        ymin=2.968e-09, ymax=1.735e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$},
        legend to name=leg:gisette-duke]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.341234 \\ 2 0.221978 \\ 3 0.156725 \\ 4 0.114743 \\ 5 0.0712025 \\
      6 0.0448428 \\ 7 0.0312058 \\ 8 0.0182202 \\ 9 0.0121134 \\ 10 0.00679738 \\ 11 0.00437192 \\
      12 0.00236882 \\ 13 0.00128142 \\ 14 0.000787644 \\ 15 0.000413953 \\ 16 0.000249913 \\ 17 0.000128266 \\
      18 6.57309e-05 \\ 19 3.86296e-05 \\ 20 1.94499e-05 \\ 21 1.13001e-05 \\ 22 5.6e-06 \\ 23 2.77471e-06 \\
      24 1.58442e-06 \\ 25 7.74963e-07 \\ 26 4.38665e-07 \\ 27 2.12032e-07 \\ 28 1.03326e-07 \\ 29 5.91656e-08 \\
      30 3.07516e-08 \\ 31 1.83143e-08 \\ 32 1.36845e-08 \\ 33 1.0955e-08 \\ 34 9.77847e-09 \\
    };
    \addlegendentry{MLRN}
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.351924 \\ 2 0.22864 \\ 3 0.160374 \\ 4 0.106423 \\ 5 0.071223 \\
      6 0.0469322 \\ 7 0.0299318 \\ 8 0.0184503 \\ 9 0.00978718 \\ 10 0.00582648 \\ 11 0.00304582 \\
      12 0.00176339 \\ 13 0.000906382 \\ 14 0.000514131 \\ 15 0.000260414 \\ 16 0.000131736 \\ 17 7.33761e-05 \\
      18 3.6682e-05 \\ 19 2.01809e-05 \\ 20 9.99131e-06 \\ 21 4.94788e-06 \\ 22 2.69538e-06 \\ 23 1.32472e-06 \\
      24 6.52368e-07 \\ 25 3.54835e-07 \\ 26 1.76152e-07 \\ 27 9.00848e-08 \\ 28 4.92358e-08 \\ 29 3.04536e-08 \\
      30 2.23469e-08 \\ 31 1.91902e-08 \\ 32 1.51284e-08 \\ 33 1.13527e-08 \\ 34 7.85223e-09 \\
    };
    \addlegendentry{MLRN nested}
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.291616 \\ 2 0.169193 \\ 3 0.104108 \\ 4 0.0639569 \\ 5 0.0379529 \\
      6 0.0217411 \\ 7 0.0122087 \\ 8 0.00675044 \\ 9 0.00368603 \\ 10 0.00199245 \\ 11 0.00106787 \\
      12 0.000568102 \\ 13 0.000300249 \\ 14 0.000157754 \\ 15 8.24476e-05 \\ 16 4.2884e-05 \\ 17 2.22081e-05 \\
      18 1.14541e-05 \\ 19 5.88436e-06 \\ 20 3.01082e-06 \\ 21 1.53353e-06 \\ 22 7.76618e-07 \\ 23 3.90147e-07 \\
      24 1.93615e-07 \\ 25 9.42246e-08 \\ 26 4.44096e-08 \\ 27 1.98449e-08 \\ 28 8.10823e-09 \\
    };
    \addlegendentry{RN}
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.323456 \\ 2 0.200904 \\ 3 0.134309 \\ 4 0.0921445 \\ 5 0.0629467 \\
      6 0.0420071 \\ 7 0.0270752 \\ 8 0.0168813 \\ 9 0.00809604 \\ 10 0.00492521 \\ 11 0.00292718 \\
      12 0.00139256 \\ 13 0.000815136 \\ 14 0.000385658 \\ 15 0.000223043 \\ 16 0.000126941 \\ 17 5.89582e-05 \\
      18 3.33341e-05 \\ 19 1.54134e-05 \\ 20 8.66531e-06 \\ 21 3.98938e-06 \\ 22 2.23132e-06 \\ 23 1.23211e-06 \\
      24 5.58869e-07 \\ 25 3.0866e-07 \\ 26 1.3972e-07 \\ 27 7.88348e-08 \\ 28 3.70373e-08 \\ 29 1.88081e-08 \\
      30 1.09698e-08 \\ 31 7.42987e-09 \\
    };
    \addlegendentry{TLRN}
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(a) Gisette, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=30.6,
        ymin=2.968e-09, ymax=1.735e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.289378 0.341234 \\ 0.557436 0.221978 \\ 0.821544 0.156725 \\ 1.08214 0.114743 \\ 1.58452 0.0712025 \\
      2.04477 0.0448428 \\ 2.30896 0.0312058 \\ 2.77607 0.0182202 \\ 3.07122 0.0121134 \\ 3.53687 0.00679738 \\ 3.80257 0.00437192 \\
      4.26728 0.00236882 \\ 4.7345 0.00128142 \\ 5.00719 0.000787644 \\ 5.46636 0.000413953 \\ 5.73065 0.000249913 \\ 6.21407 0.000128266 \\
      6.67125 6.57309e-05 \\ 6.93631 3.86296e-05 \\ 7.39711 1.94499e-05 \\ 7.66253 1.13001e-05 \\ 8.12234 5.6e-06 \\ 8.58261 2.77471e-06 \\
      8.86199 1.58442e-06 \\ 9.31813 7.74963e-07 \\ 9.58241 4.38665e-07 \\ 10.042 2.12032e-07 \\ 10.5024 1.03326e-07 \\ 10.7664 5.91656e-08 \\
      11.2271 3.07516e-08 \\ 11.6939 1.83143e-08 \\ 11.7873 1.36845e-08 \\ 12.0539 1.0955e-08 \\ 12.5185 9.77847e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.267127 0.351924 \\ 0.529526 0.22864 \\ 0.790137 0.160374 \\ 1.24833 0.106423 \\ 1.71108 0.071223 \\
      2.17185 0.0469322 \\ 2.63325 0.0299318 \\ 3.09394 0.0184503 \\ 3.82027 0.00978718 \\ 4.3024 0.00582648 \\ 5.00095 0.00304582 \\
      5.46795 0.00176339 \\ 6.16167 0.000906382 \\ 6.62809 0.000514131 \\ 7.32517 0.000260414 \\ 8.02063 0.000131736 \\ 8.48979 7.33761e-05 \\
      9.18555 3.6682e-05 \\ 9.65402 2.01809e-05 \\ 10.3486 9.99131e-06 \\ 11.0309 4.94788e-06 \\ 11.5176 2.69538e-06 \\ 12.234 1.32472e-06 \\
      12.9144 6.52368e-07 \\ 13.4224 3.54835e-07 \\ 14.1379 1.76152e-07 \\ 14.8347 9.00848e-08 \\ 15.5462 4.92358e-08 \\ 16.2597 3.04536e-08 \\
      16.946 2.23469e-08 \\ 17.6668 1.91902e-08 \\ 18.6288 1.51284e-08 \\ 19.5943 1.13527e-08 \\ 20.5477 7.85223e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1.08027 0.291616 \\ 2.21109 0.169193 \\ 3.29773 0.104108 \\ 4.33974 0.0639569 \\ 5.38297 0.0379529 \\
      6.426 0.0217411 \\ 7.47015 0.0122087 \\ 8.5293 0.00675044 \\ 9.5925 0.00368603 \\ 10.6371 0.00199245 \\ 11.6978 0.00106787 \\
      12.7447 0.000568102 \\ 13.7868 0.000300249 \\ 14.8456 0.000157754 \\ 15.9064 8.24476e-05 \\ 16.9845 4.2884e-05 \\ 18.0453 2.22081e-05 \\
      19.1044 1.14541e-05 \\ 20.1633 5.88436e-06 \\ 21.223 3.01082e-06 \\ 22.287 1.53353e-06 \\ 23.3288 7.76618e-07 \\ 24.3738 3.90147e-07 \\
      25.418 1.93615e-07 \\ 26.4628 9.42246e-08 \\ 27.5762 4.44096e-08 \\ 28.6168 1.98449e-08 \\ 29.6604 8.10823e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.472023 0.323456 \\ 0.940126 0.200904 \\ 1.40833 0.134309 \\ 1.87575 0.0921445 \\ 2.34369 0.0629467 \\
      2.81126 0.0420071 \\ 3.27368 0.0270752 \\ 3.73204 0.0168813 \\ 4.78667 0.00809604 \\ 5.25437 0.00492521 \\ 5.71603 0.00292718 \\
      6.76159 0.00139256 \\ 7.23062 0.000815136 \\ 8.32499 0.000385658 \\ 8.79135 0.000223043 \\ 9.25218 0.000126941 \\ 10.299 5.89582e-05 \\
      10.7667 3.33341e-05 \\ 11.8089 1.54134e-05 \\ 12.2749 8.66531e-06 \\ 13.3293 3.98938e-06 \\ 13.805 2.23132e-06 \\ 14.3121 1.23211e-06 \\
      15.3669 5.58869e-07 \\ 15.864 3.0866e-07 \\ 16.9224 1.3972e-07 \\ 17.389 7.88348e-08 \\ 18.4801 3.70373e-08 \\ 19.5553 1.88081e-08 \\
      20.6 1.09698e-08 \\ 21.6455 7.42987e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(b) Gisette, runtime};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=21.6,
        ymin=1.826e-09, ymax=1.776e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.15956 \\ 2 0.0581917 \\ 3 0.0227258 \\ 4 0.00912504 \\ 5 0.00372137 \\
      6 0.0015348 \\ 7 0.000639153 \\ 8 0.000268621 \\ 9 0.000113924 \\ 10 4.87567e-05 \\ 11 2.10567e-05 \\
      12 9.17541e-06 \\ 13 4.03382e-06 \\ 14 1.78946e-06 \\ 15 8.01012e-07 \\ 16 3.61722e-07 \\ 17 1.64745e-07 \\
      18 7.56655e-08 \\ 19 3.50577e-08 \\ 20 1.64112e-08 \\ 21 7.7959e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.159741 \\ 2 0.0582728 \\ 3 0.0227713 \\ 4 0.00915124 \\ 5 0.0037363 \\
      6 0.00154315 \\ 7 0.000643727 \\ 8 0.000271074 \\ 9 0.000115212 \\ 10 4.94217e-05 \\ 11 2.13962e-05 \\
      12 9.34789e-06 \\ 13 4.12087e-06 \\ 14 1.83279e-06 \\ 15 8.22337e-07 \\ 16 3.72166e-07 \\ 17 1.69853e-07 \\
      18 7.81665e-08 \\ 19 3.6286e-08 \\ 20 1.70185e-08 \\ 21 8.10038e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.141749 \\ 2 0.0504893 \\ 3 0.0192 \\ 4 0.00749097 \\ 5 0.00296374 \\
      6 0.00118443 \\ 7 0.000477524 \\ 8 0.000194157 \\ 9 7.96153e-05 \\ 10 3.29305e-05 \\ 11 1.3742e-05 \\
      12 5.78676e-06 \\ 13 2.45928e-06 \\ 14 1.05477e-06 \\ 15 4.56483e-07 \\ 16 1.99339e-07 \\ 17 8.78302e-08 \\
      18 3.90267e-08 \\ 19 1.74686e-08 \\ 20 7.86086e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.150709 \\ 2 0.05382 \\ 3 0.0205865 \\ 4 0.0080899 \\ 5 0.00322643 \\
      6 0.00130055 \\ 7 0.000529112 \\ 8 0.00021718 \\ 9 8.99358e-05 \\ 10 3.75784e-05 \\ 11 1.58451e-05 \\
      12 6.74293e-06 \\ 13 2.8961e-06 \\ 14 1.25535e-06 \\ 15 5.49125e-07 \\ 16 2.42413e-07 \\ 17 1.07997e-07 \\
      18 4.85483e-08 \\ 19 2.20212e-08 \\ 20 1.00858e-08 \\ 21 4.67671e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(c) Duke, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=16.1,
        ymin=1.826e-09, ymax=1.776e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.107316 0.15956 \\ 0.189849 0.0581917 \\ 0.269627 0.0227258 \\ 0.348134 0.00912504 \\ 0.426704 0.00372137 \\
      0.510825 0.0015348 \\ 0.587515 0.000639153 \\ 0.664879 0.000268621 \\ 0.7443 0.000113924 \\ 0.823644 4.87567e-05 \\ 0.902933 2.10567e-05 \\
      0.984368 9.17541e-06 \\ 1.0645 4.03382e-06 \\ 1.14271 1.78946e-06 \\ 1.22532 8.01012e-07 \\ 1.30544 3.61722e-07 \\ 1.382 1.64745e-07 \\
      1.4615 7.56655e-08 \\ 1.53983 3.50577e-08 \\ 1.61899 1.64112e-08 \\ 1.69816 7.7959e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.075281 0.159741 \\ 0.161511 0.0582728 \\ 0.241713 0.0227713 \\ 0.324653 0.00915124 \\ 0.407831 0.0037363 \\
      0.489157 0.00154315 \\ 0.572311 0.000643727 \\ 0.653658 0.000271074 \\ 0.732206 0.000115212 \\ 0.810311 4.94217e-05 \\ 0.895539 2.13962e-05 \\
      0.974757 9.34789e-06 \\ 1.05202 4.12087e-06 \\ 1.13764 1.83279e-06 \\ 1.21614 8.22337e-07 \\ 1.29337 3.72166e-07 \\ 1.37782 1.69853e-07 \\
      1.45812 7.81665e-08 \\ 1.54099 3.6286e-08 \\ 1.62539 1.70185e-08 \\ 1.707 8.10038e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.741092 0.141749 \\ 2.16882 0.0504893 \\ 2.9132 0.0192 \\ 3.66182 0.00749097 \\ 4.40453 0.00296374 \\
      5.14713 0.00118443 \\ 5.89295 0.000477524 \\ 6.63739 0.000194157 \\ 7.38009 7.96153e-05 \\ 8.12174 3.29305e-05 \\ 8.86554 1.3742e-05 \\
      9.61025 5.78676e-06 \\ 10.3585 2.45928e-06 \\ 11.1026 1.05477e-06 \\ 11.8472 4.56483e-07 \\ 12.5905 1.99339e-07 \\ 13.3354 8.78302e-08 \\
      14.1053 3.90267e-08 \\ 14.8898 1.74686e-08 \\ 15.6721 7.86086e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.259394 0.150709 \\ 0.463393 0.05382 \\ 0.651306 0.0205865 \\ 0.830558 0.0080899 \\ 1.01122 0.00322643 \\
      1.19159 0.00130055 \\ 1.36947 0.000529112 \\ 1.55353 0.00021718 \\ 1.73696 8.99358e-05 \\ 1.96636 3.75784e-05 \\ 2.18405 1.58451e-05 \\
      2.38181 6.74293e-06 \\ 2.57252 2.8961e-06 \\ 2.75113 1.25535e-06 \\ 2.93384 5.49125e-07 \\ 3.11331 2.42413e-07 \\ 3.29476 1.07997e-07 \\
      3.47367 4.85483e-08 \\ 3.65467 2.20212e-08 \\ 3.83513 1.00858e-08 \\ 4.01907 4.67671e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(d) Duke, runtime};

Figure 2. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) using the large Gisette and Duke datasets for logistic regression. The left column reports the objective gap f(xk)−f∗f(x_k)-f^* versus iterations, while the right column reports the objective gap versus wall-clock time. The multilevel gradient-regularized Newton methods offer substantial computational savings compared to their full-space counterpart.

leg:a9a-mushrooms-w8t

\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=308,
        ymin=1.533e-09, ymax=9.290e-01,
        ytickten={-7,-5,-3,-1},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$},
        legend to name=leg:a9a-mushrooms-w8t]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 1 0.220844 \\ 2 0.160654 \\ 3 0.125031 \\ 4 0.0785861 \\ 5 0.0624869 \\
      6 0.0435495 \\ 7 0.0311968 \\ 8 0.022553 \\ 9 0.0169181 \\ 10 0.0126976 \\ 11 0.00835148 \\
      12 0.00638134 \\ 13 0.00441419 \\ 14 0.00367332 \\ 15 0.00288761 \\ 16 0.00224696 \\ 17 0.00181493 \\
      18 0.00154746 \\ 19 0.00135312 \\ 20 0.00123353 \\ 21 0.00113809 \\ 22 0.00106878 \\ 23 0.00101229 \\
      24 0.000967882 \\ 25 0.000930739 \\ 26 0.00090097 \\ 27 0.000875672 \\ 28 0.000855072 \\ 29 0.000837271 \\
      30 0.000822621 \\ 31 0.000809794 \\ 32 0.00079896 \\ 33 0.000789373 \\ 34 0.000781162 \\ 35 0.000773838 \\
      36 0.000767419 \\ 37 0.000761655 \\ 38 0.000203549 \\ 39 0.000199489 \\ 40 0.000191282 \\ 41 0.000153031 \\
      42 0.000149533 \\ 43 0.000144525 \\ 44 0.000140593 \\ 45 0.00013727 \\ 46 0.000134464 \\ 47 0.000132051 \\
      48 0.00012569 \\ 49 0.00012358 \\ 50 0.000120437 \\ 51 0.000117863 \\ 52 0.000115635 \\ 53 0.000113742 \\
      54 0.000112086 \\ 55 0.000109894 \\ 56 0.000108417 \\ 57 0.000106168 \\ 58 0.000104288 \\ 59 0.000102641 \\
      60 0.00010123 \\ 61 9.99924e-05 \\ 62 9.84197e-05 \\ 63 9.72928e-05 \\ 64 9.55848e-05 \\ 65 9.41379e-05 \\
      66 9.28675e-05 \\ 67 9.17764e-05 \\ 68 9.08155e-05 \\ 69 8.99872e-05 \\ 70 8.92585e-05 \\ 71 8.75101e-05 \\
      72 8.67453e-05 \\ 73 8.55176e-05 \\ 74 8.44727e-05 \\ 75 8.3555e-05 \\ 76 8.27647e-05 \\ 77 8.20699e-05 \\
      78 8.14691e-05 \\ 79 8.09405e-05 \\ 80 7.95864e-05 \\ 81 7.90162e-05 \\ 82 7.81046e-05 \\ 83 7.73263e-05 \\
      84 7.66434e-05 \\ 85 7.60537e-05 \\ 86 7.55357e-05 \\ 87 7.50876e-05 \\ 88 7.46935e-05 \\ 89 7.36761e-05 \\
      90 7.32463e-05 \\ 91 7.25652e-05 \\ 92 7.19824e-05 \\ 93 7.14714e-05 \\ 94 7.10298e-05 \\ 95 7.0642e-05 \\
      96 7.03062e-05 \\ 97 7.00114e-05 \\ 98 6.92491e-05 \\ 99 6.89254e-05 \\ 100 6.84156e-05 \\ 101 6.79791e-05 \\
      102 6.75966e-05 \\ 103 6.72658e-05 \\ 104 6.69757e-05 \\ 105 6.6724e-05 \\ 106 6.65032e-05 \\ 107 6.63115e-05 \\
      108 6.61433e-05 \\ 109 6.54004e-05 \\ 110 6.51775e-05 \\ 111 6.4822e-05 \\ 112 6.45189e-05 \\ 113 6.42538e-05 \\
      114 6.40241e-05 \\ 115 6.38229e-05 \\ 116 6.36482e-05 \\ 117 6.3495e-05 \\ 118 6.33619e-05 \\ 119 6.32452e-05 \\
      120 6.27019e-05 \\ 121 6.25434e-05 \\ 122 6.22916e-05 \\ 123 6.20771e-05 \\ 124 6.18896e-05 \\ 125 6.1727e-05 \\
      126 6.15847e-05 \\ 127 6.1461e-05 \\ 128 6.13527e-05 \\ 129 6.12585e-05 \\ 130 6.11759e-05 \\ 131 2.0168e-05 \\
      132 1.98087e-05 \\ 133 1.95027e-05 \\ 134 1.92555e-05 \\ 135 1.90977e-05 \\ 136 1.8967e-05 \\ 137 1.88537e-05 \\
      138 1.86977e-05 \\ 139 1.85904e-05 \\ 140 1.84314e-05 \\ 141 1.82968e-05 \\ 142 1.81792e-05 \\ 143 1.80771e-05 \\
      144 1.79877e-05 \\ 145 1.791e-05 \\ 146 1.7842e-05 \\ 147 1.76785e-05 \\ 148 1.76063e-05 \\ 149 1.74925e-05 \\
      150 1.73952e-05 \\ 151 1.73102e-05 \\ 152 1.72364e-05 \\ 153 1.71719e-05 \\ 154 1.71158e-05 \\ 155 1.70667e-05 \\
      156 1.69418e-05 \\ 157 1.68887e-05 \\ 158 1.6805e-05 \\ 159 1.67331e-05 \\ 160 1.66703e-05 \\ 161 1.66158e-05 \\
      162 1.65682e-05 \\ 163 1.65267e-05 \\ 164 1.64905e-05 \\ 165 1.63976e-05 \\ 166 1.63581e-05 \\ 167 1.62961e-05 \\
      168 1.62428e-05 \\ 169 1.61963e-05 \\ 170 1.61559e-05 \\ 171 1.61206e-05 \\ 172 1.60898e-05 \\ 173 1.6063e-05 \\
      174 1.5994e-05 \\ 175 1.59645e-05 \\ 176 1.59185e-05 \\ 177 1.5879e-05 \\ 178 1.58445e-05 \\ 179 1.58145e-05 \\
      180 1.57883e-05 \\ 181 1.57654e-05 \\ 182 1.57455e-05 \\ 183 1.56943e-05 \\ 184 1.56723e-05 \\ 185 1.56382e-05 \\
      186 1.56088e-05 \\ 187 1.55832e-05 \\ 188 1.55609e-05 \\ 189 1.55415e-05 \\ 190 1.55245e-05 \\ 191 1.55097e-05 \\
      192 1.54968e-05 \\ 193 1.54855e-05 \\ 194 1.5436e-05 \\ 195 1.54211e-05 \\ 196 1.53975e-05 \\ 197 1.53774e-05 \\
      198 1.53598e-05 \\ 199 1.53445e-05 \\ 200 1.53311e-05 \\ 201 1.53195e-05 \\ 202 1.53093e-05 \\ 203 1.53004e-05 \\
      204 1.52927e-05 \\ 205 5.85371e-06 \\ 206 5.81938e-06 \\ 207 5.78911e-06 \\ 208 5.76124e-06 \\ 209 5.74598e-06 \\
      210 5.73318e-06 \\ 211 5.72214e-06 \\ 212 5.71258e-06 \\ 213 5.70425e-06 \\ 214 5.68388e-06 \\ 215 5.67499e-06 \\
      216 5.66107e-06 \\ 217 5.64939e-06 \\ 218 5.63922e-06 \\ 219 5.63036e-06 \\ 220 5.62263e-06 \\ 221 5.6159e-06 \\
      222 5.61002e-06 \\ 223 5.59503e-06 \\ 224 5.58869e-06 \\ 225 5.5787e-06 \\ 226 5.57016e-06 \\ 227 5.56272e-06 \\
      228 5.55624e-06 \\ 229 5.55058e-06 \\ 230 5.54566e-06 \\ 231 5.54136e-06 \\ 232 5.53037e-06 \\ 233 5.52571e-06 \\
      234 5.51838e-06 \\ 235 5.51209e-06 \\ 236 5.5066e-06 \\ 237 5.50183e-06 \\ 238 5.49766e-06 \\ 239 5.49403e-06 \\
      240 5.49086e-06 \\ 241 5.48275e-06 \\ 242 5.4793e-06 \\ 243 5.4739e-06 \\ 244 5.46924e-06 \\ 245 5.46519e-06 \\
      246 5.46166e-06 \\ 247 5.45858e-06 \\ 248 5.45589e-06 \\ 249 5.45355e-06 \\ 250 5.45151e-06 \\ 251 5.44972e-06 \\
      252 5.44192e-06 \\ 253 5.43958e-06 \\ 254 5.43587e-06 \\ 255 5.43268e-06 \\ 256 5.42991e-06 \\ 257 5.4275e-06 \\
      258 5.42539e-06 \\ 259 5.42355e-06 \\ 260 5.42195e-06 \\ 261 5.42055e-06 \\ 262 5.41933e-06 \\ 263 5.41367e-06 \\
      264 5.41202e-06 \\ 265 5.40942e-06 \\ 266 5.40719e-06 \\ 267 5.40525e-06 \\ 268 5.40355e-06 \\ 269 5.40208e-06 \\
      270 5.40079e-06 \\ 271 5.39967e-06 \\ 272 5.39869e-06 \\ 273 5.39783e-06 \\ 274 2.15022e-06 \\ 275 2.14635e-06 \\
      276 2.14316e-06 \\ 277 2.13876e-06 \\ 278 2.13706e-06 \\ 279 2.13563e-06 \\ 280 2.13439e-06 \\ 281 2.13331e-06 \\
      282 2.13237e-06 \\ 283 2.13016e-06 \\ 284 2.12917e-06 \\ 285 2.12763e-06 \\ 286 2.12634e-06 \\ 287 2.12523e-06 \\
      288 2.12425e-06 \\ 289 2.1234e-06 \\ 290 2.12266e-06 \\ 291 2.12201e-06 \\ 292 2.12038e-06 \\ 293 2.11968e-06 \\
      294 2.11859e-06 \\ 295 2.11766e-06 \\ 296 2.11684e-06 \\ 297 2.11614e-06 \\ 298 2.11552e-06 \\ 299 2.11498e-06 \\
    };
    \addlegendentry{MLRN}
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 1 0.245464 \\ 2 0.17626 \\ 3 0.119014 \\ 4 0.0824586 \\ 5 0.0592795 \\
      6 0.0304271 \\ 7 0.0215966 \\ 8 0.011448 \\ 9 0.00775859 \\ 10 0.00287514 \\ 11 0.00214857 \\
      12 0.000998239 \\ 13 0.000846347 \\ 14 0.000648285 \\ 15 0.00030038 \\ 16 0.000113784 \\ 17 0.000101073 \\
      18 6.18168e-05 \\ 19 2.83696e-05 \\ 20 1.57979e-05 \\ 21 9.08908e-06 \\ 22 5.99197e-06 \\ 23 3.06418e-06 \\
      24 1.72343e-06 \\ 25 9.7451e-07 \\ 26 6.72319e-07 \\ 27 3.19381e-07 \\ 28 1.5412e-07 \\ 29 8.15245e-08 \\
      30 4.38879e-08 \\ 31 2.93518e-08 \\ 32 1.05093e-08 \\ 33 3.84338e-09 \\
    };
    \addlegendentry{MLRN nested}
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 1 0.194305 \\ 2 0.119113 \\ 3 0.0713522 \\ 4 0.0423861 \\ 5 0.0244413 \\
      6 0.0133646 \\ 7 0.00694675 \\ 8 0.00356043 \\ 9 0.00196368 \\ 10 0.00123244 \\ 11 0.000808864 \\
      12 0.000502406 \\ 13 0.000285909 \\ 14 0.000154609 \\ 15 8.66915e-05 \\ 16 5.17797e-05 \\ 17 3.15256e-05 \\
      18 1.90743e-05 \\ 19 1.14358e-05 \\ 20 6.80782e-06 \\ 21 4.02e-06 \\ 22 2.3503e-06 \\ 23 1.35805e-06 \\
      24 7.73439e-07 \\ 25 4.3267e-07 \\ 26 2.36667e-07 \\ 27 1.25711e-07 \\ 28 6.41038e-08 \\ 29 3.07788e-08 \\
      30 1.34621e-08 \\ 31 5.06505e-09 \\
    };
    \addlegendentry{RN}
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 1 0.245464 \\ 2 0.17626 \\ 3 0.0999392 \\ 4 0.0732158 \\ 5 0.0387032 \\
      6 0.0279254 \\ 7 0.012742 \\ 8 0.00918266 \\ 9 0.00391495 \\ 10 0.00196513 \\ 11 0.00165493 \\
      12 0.000938637 \\ 13 0.000807706 \\ 14 0.000374031 \\ 15 0.000315199 \\ 16 0.000111387 \\ 17 9.04894e-05 \\
      18 3.8996e-05 \\ 19 3.0339e-05 \\ 20 1.52633e-05 \\ 21 1.20985e-05 \\ 22 5.95702e-06 \\ 23 4.848e-06 \\
      24 2.48231e-06 \\ 25 2.0914e-06 \\ 26 1.09324e-06 \\ 27 9.40484e-07 \\ 28 4.94303e-07 \\ 29 4.30823e-07 \\
      30 2.24572e-07 \\ 31 1.97364e-07 \\ 32 1.00573e-07 \\ 33 5.211e-08 \\ 34 2.50597e-08 \\ 35 1.96207e-08 \\
      36 8.06526e-09 \\
    };
    \addlegendentry{TLRN}
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(a) a9a, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=3.08,
        ymin=1.533e-09, ymax=9.290e-01,
        ytickten={-7,-5,-3,-1},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 0.0115412 0.220844 \\ 0.0203368 0.160654 \\ 0.0280939 0.125031 \\ 0.0391305 0.0785861 \\ 0.0467821 0.0624869 \\
      0.0572532 0.0435495 \\ 0.0633852 0.0311968 \\ 0.0708939 0.022553 \\ 0.0806933 0.0169181 \\ 0.0880545 0.0126976 \\ 0.0939562 0.00835148 \\
      0.103897 0.00638134 \\ 0.111321 0.00441419 \\ 0.121158 0.00367332 \\ 0.128294 0.00288761 \\ 0.134309 0.00224696 \\ 0.145315 0.00181493 \\
      0.153767 0.00154746 \\ 0.16459 0.00135312 \\ 0.172979 0.00123353 \\ 0.184575 0.00113809 \\ 0.192661 0.00106878 \\ 0.204133 0.00101229 \\
      0.212091 0.000967882 \\ 0.223273 0.000930739 \\ 0.231644 0.00090097 \\ 0.242634 0.000875672 \\ 0.250447 0.000855072 \\ 0.260493 0.000837271 \\
      0.269641 0.000822621 \\ 0.279376 0.000809794 \\ 0.288388 0.00079896 \\ 0.29804 0.000789373 \\ 0.308625 0.000781162 \\ 0.318436 0.000773838 \\
      0.32734 0.000767419 \\ 0.336857 0.000761655 \\ 0.371336 0.000203549 \\ 0.379466 0.000199489 \\ 0.390746 0.000191282 \\ 0.397143 0.000153031 \\
      0.405942 0.000149533 \\ 0.417094 0.000144525 \\ 0.425363 0.000140593 \\ 0.436532 0.00013727 \\ 0.444116 0.000134464 \\ 0.45491 0.000132051 \\
      0.462495 0.00012569 \\ 0.470732 0.00012358 \\ 0.483371 0.000120437 \\ 0.492945 0.000117863 \\ 0.504715 0.000115635 \\ 0.513685 0.000113742 \\
      0.523827 0.000112086 \\ 0.529581 0.000109894 \\ 0.542375 0.000108417 \\ 0.555713 0.000106168 \\ 0.566069 0.000104288 \\ 0.577076 0.000102641 \\
      0.587585 0.00010123 \\ 0.600792 9.99924e-05 \\ 0.607428 9.84197e-05 \\ 0.619449 9.72928e-05 \\ 0.633984 9.55848e-05 \\ 0.642125 9.41379e-05 \\
      0.654812 9.28675e-05 \\ 0.664456 9.17764e-05 \\ 0.67485 9.08155e-05 \\ 0.683617 8.99872e-05 \\ 0.695188 8.92585e-05 \\ 0.702361 8.75101e-05 \\
      0.711421 8.67453e-05 \\ 0.724146 8.55176e-05 \\ 0.731586 8.44727e-05 \\ 0.74316 8.3555e-05 \\ 0.753078 8.27647e-05 \\ 0.766111 8.20699e-05 \\
      0.774497 8.14691e-05 \\ 0.787869 8.09405e-05 \\ 0.798339 7.95864e-05 \\ 0.80856 7.90162e-05 \\ 0.821589 7.81046e-05 \\ 0.831779 7.73263e-05 \\
      0.844835 7.66434e-05 \\ 0.853338 7.60537e-05 \\ 0.865363 7.55357e-05 \\ 0.874571 7.50876e-05 \\ 0.884563 7.46935e-05 \\ 0.893233 7.36761e-05 \\
      0.901909 7.32463e-05 \\ 0.915008 7.25652e-05 \\ 0.922425 7.19824e-05 \\ 0.933905 7.14714e-05 \\ 0.943073 7.10298e-05 \\ 0.953132 7.0642e-05 \\
      0.962744 7.03062e-05 \\ 0.975425 7.00114e-05 \\ 0.986306 6.92491e-05 \\ 0.994456 6.89254e-05 \\ 1.00931 6.84156e-05 \\ 1.01782 6.79791e-05 \\
      1.03075 6.75966e-05 \\ 1.04109 6.72658e-05 \\ 1.05444 6.69757e-05 \\ 1.06234 6.6724e-05 \\ 1.07456 6.65032e-05 \\ 1.08445 6.63115e-05 \\
      1.09594 6.61433e-05 \\ 1.10616 6.54004e-05 \\ 1.11376 6.51775e-05 \\ 1.12544 6.4822e-05 \\ 1.13469 6.45189e-05 \\ 1.14614 6.42538e-05 \\
      1.15362 6.40241e-05 \\ 1.1652 6.38229e-05 \\ 1.18122 6.36482e-05 \\ 1.19417 6.3495e-05 \\ 1.20486 6.33619e-05 \\ 1.21789 6.32452e-05 \\
      1.23007 6.27019e-05 \\ 1.23858 6.25434e-05 \\ 1.25201 6.22916e-05 \\ 1.262 6.20771e-05 \\ 1.27482 6.18896e-05 \\ 1.28289 6.1727e-05 \\
      1.2951 6.15847e-05 \\ 1.30484 6.1461e-05 \\ 1.31634 6.13527e-05 \\ 1.32375 6.12585e-05 \\ 1.33523 6.11759e-05 \\ 1.3762 2.0168e-05 \\
      1.38223 1.98087e-05 \\ 1.3931 1.95027e-05 \\ 1.40152 1.92555e-05 \\ 1.41274 1.90977e-05 \\ 1.42126 1.8967e-05 \\ 1.43235 1.88537e-05 \\
      1.43879 1.86977e-05 \\ 1.44691 1.85904e-05 \\ 1.4585 1.84314e-05 \\ 1.46712 1.82968e-05 \\ 1.47768 1.81792e-05 \\ 1.48572 1.80771e-05 \\
      1.49577 1.79877e-05 \\ 1.50342 1.791e-05 \\ 1.51313 1.7842e-05 \\ 1.51875 1.76785e-05 \\ 1.5287 1.76063e-05 \\ 1.54325 1.74925e-05 \\
      1.55156 1.73952e-05 \\ 1.5634 1.73102e-05 \\ 1.57256 1.72364e-05 \\ 1.58287 1.71719e-05 \\ 1.59327 1.71158e-05 \\ 1.60623 1.70667e-05 \\
      1.61709 1.69418e-05 \\ 1.62562 1.68887e-05 \\ 1.64066 1.6805e-05 \\ 1.64891 1.67331e-05 \\ 1.66202 1.66703e-05 \\ 1.67286 1.66158e-05 \\
      1.68615 1.65682e-05 \\ 1.69365 1.65267e-05 \\ 1.70397 1.64905e-05 \\ 1.71109 1.63976e-05 \\ 1.72004 1.63581e-05 \\ 1.73152 1.62961e-05 \\
      1.74058 1.62428e-05 \\ 1.75211 1.61963e-05 \\ 1.75949 1.61559e-05 \\ 1.76937 1.61206e-05 \\ 1.77677 1.60898e-05 \\ 1.78671 1.6063e-05 \\
      1.79251 1.5994e-05 \\ 1.80082 1.59645e-05 \\ 1.81368 1.59185e-05 \\ 1.82214 1.5879e-05 \\ 1.83336 1.58445e-05 \\ 1.8417 1.58145e-05 \\
      1.85302 1.57883e-05 \\ 1.86118 1.57654e-05 \\ 1.87268 1.57455e-05 \\ 1.87913 1.56943e-05 \\ 1.88766 1.56723e-05 \\ 1.89811 1.56382e-05 \\
      1.90619 1.56088e-05 \\ 1.91671 1.55832e-05 \\ 1.92431 1.55609e-05 \\ 1.93443 1.55415e-05 \\ 1.94178 1.55245e-05 \\ 1.95178 1.55097e-05 \\
      1.9593 1.54968e-05 \\ 1.96919 1.54855e-05 \\ 1.97476 1.5436e-05 \\ 1.98207 1.54211e-05 \\ 1.99224 1.53975e-05 \\ 2.00023 1.53774e-05 \\
      2.01159 1.53598e-05 \\ 2.02051 1.53445e-05 \\ 2.0314 1.53311e-05 \\ 2.03983 1.53195e-05 \\ 2.05103 1.53093e-05 \\ 2.05958 1.53004e-05 \\
      2.0709 1.52927e-05 \\ 2.11563 5.85371e-06 \\ 2.12156 5.81938e-06 \\ 2.13169 5.78911e-06 \\ 2.13923 5.76124e-06 \\ 2.14936 5.74598e-06 \\
      2.15672 5.73318e-06 \\ 2.16646 5.72214e-06 \\ 2.1738 5.71258e-06 \\ 2.18379 5.70425e-06 \\ 2.18941 5.68388e-06 \\ 2.19706 5.67499e-06 \\
      2.20808 5.66107e-06 \\ 2.21654 5.64939e-06 \\ 2.22768 5.63922e-06 \\ 2.2363 5.63036e-06 \\ 2.24725 5.62263e-06 \\ 2.25593 5.6159e-06 \\
      2.26714 5.61002e-06 \\ 2.27372 5.59503e-06 \\ 2.28227 5.58869e-06 \\ 2.29377 5.5787e-06 \\ 2.30222 5.57016e-06 \\ 2.31259 5.56272e-06 \\
      2.32089 5.55624e-06 \\ 2.33082 5.55058e-06 \\ 2.33838 5.54566e-06 \\ 2.34831 5.54136e-06 \\ 2.35396 5.53037e-06 \\ 2.36155 5.52571e-06 \\
      2.37164 5.51838e-06 \\ 2.37904 5.51209e-06 \\ 2.38892 5.5066e-06 \\ 2.39636 5.50183e-06 \\ 2.40716 5.49766e-06 \\ 2.4158 5.49403e-06 \\
      2.42684 5.49086e-06 \\ 2.43351 5.48275e-06 \\ 2.44215 5.4793e-06 \\ 2.45369 5.4739e-06 \\ 2.46195 5.46924e-06 \\ 2.47339 5.46519e-06 \\
      2.4815 5.46166e-06 \\ 2.49257 5.45858e-06 \\ 2.50116 5.45589e-06 \\ 2.51144 5.45355e-06 \\ 2.5196 5.45151e-06 \\ 2.53011 5.44972e-06 \\
      2.53609 5.44192e-06 \\ 2.5437 5.43958e-06 \\ 2.55433 5.43587e-06 \\ 2.56185 5.43268e-06 \\ 2.57171 5.42991e-06 \\ 2.57951 5.4275e-06 \\
      2.59055 5.42539e-06 \\ 2.59809 5.42355e-06 \\ 2.60827 5.42195e-06 \\ 2.616 5.42055e-06 \\ 2.6271 5.41933e-06 \\ 2.63372 5.41367e-06 \\
      2.64215 5.41202e-06 \\ 2.65363 5.40942e-06 \\ 2.66164 5.40719e-06 \\ 2.67314 5.40525e-06 \\ 2.68147 5.40355e-06 \\ 2.69277 5.40208e-06 \\
      2.70153 5.40079e-06 \\ 2.71246 5.39967e-06 \\ 2.72098 5.39869e-06 \\ 2.73141 5.39783e-06 \\ 2.77068 2.15022e-06 \\ 2.77614 2.14635e-06 \\
      2.78603 2.14316e-06 \\ 2.79359 2.13876e-06 \\ 2.80383 2.13706e-06 \\ 2.8113 2.13563e-06 \\ 2.82149 2.13439e-06 \\ 2.82946 2.13331e-06 \\
      2.84053 2.13237e-06 \\ 2.84679 2.13016e-06 \\ 2.85566 2.12917e-06 \\ 2.86716 2.12763e-06 \\ 2.87557 2.12634e-06 \\ 2.88695 2.12523e-06 \\
      2.89515 2.12425e-06 \\ 2.9065 2.1234e-06 \\ 2.91465 2.12266e-06 \\ 2.92549 2.12201e-06 \\ 2.93184 2.12038e-06 \\ 2.9399 2.11968e-06 \\
      2.95054 2.11859e-06 \\ 2.95802 2.11766e-06 \\ 2.96793 2.11684e-06 \\ 2.97513 2.11614e-06 \\ 2.9851 2.11552e-06 \\ 2.99259 2.11498e-06 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 0.0115394 0.245464 \\ 0.0215122 0.17626 \\ 0.0351462 0.119014 \\ 0.0494607 0.0824586 \\ 0.063812 0.0592795 \\
      0.0801981 0.0304271 \\ 0.0941092 0.0215966 \\ 0.10991 0.011448 \\ 0.126056 0.00775859 \\ 0.142686 0.00287514 \\ 0.157922 0.00214857 \\
      0.173969 0.000998239 \\ 0.183963 0.000846347 \\ 0.195857 0.000648285 \\ 0.209516 0.00030038 \\ 0.225594 0.000113784 \\ 0.233608 0.000101073 \\
      0.24678 6.18168e-05 \\ 0.263213 2.83696e-05 \\ 0.279646 1.57979e-05 \\ 0.297047 9.08908e-06 \\ 0.308514 5.99197e-06 \\ 0.32206 3.06418e-06 \\
      0.338552 1.72343e-06 \\ 0.355321 9.7451e-07 \\ 0.365918 6.72319e-07 \\ 0.379003 3.19381e-07 \\ 0.390952 1.5412e-07 \\ 0.406135 8.15245e-08 \\
      0.420678 4.38879e-08 \\ 0.4305 2.93518e-08 \\ 0.443248 1.05093e-08 \\ 0.458801 3.84338e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 0.0178052 0.194305 \\ 0.0340228 0.119113 \\ 0.0484172 0.0713522 \\ 0.0627713 0.0423861 \\ 0.0780268 0.0244413 \\
      0.0923394 0.0133646 \\ 0.108325 0.00694675 \\ 0.123613 0.00356043 \\ 0.13894 0.00196368 \\ 0.15426 0.00123244 \\ 0.168464 0.000808864 \\
      0.183726 0.000502406 \\ 0.199015 0.000285909 \\ 0.214142 0.000154609 \\ 0.231832 8.66915e-05 \\ 0.249281 5.17797e-05 \\ 0.266276 3.15256e-05 \\
      0.283534 1.90743e-05 \\ 0.299903 1.14358e-05 \\ 0.315326 6.80782e-06 \\ 0.330217 4.02e-06 \\ 0.345712 2.3503e-06 \\ 0.361232 1.35805e-06 \\
      0.376598 7.73439e-07 \\ 0.392102 4.3267e-07 \\ 0.408409 2.36667e-07 \\ 0.425438 1.25711e-07 \\ 0.442824 6.41038e-08 \\ 0.45921 3.07788e-08 \\
      0.476792 1.34621e-08 \\ 0.493983 5.06505e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.370526 \\ 0.012615 0.245464 \\ 0.0239598 0.17626 \\ 0.0404104 0.0999392 \\ 0.0512433 0.0732158 \\ 0.0686374 0.0387032 \\
      0.0801329 0.0279254 \\ 0.0975579 0.012742 \\ 0.108253 0.00918266 \\ 0.123155 0.00391495 \\ 0.138853 0.00196513 \\ 0.148898 0.00165493 \\
      0.163416 0.000938637 \\ 0.174368 0.000807706 \\ 0.190745 0.000374031 \\ 0.200681 0.000315199 \\ 0.218082 0.000111387 \\ 0.229409 9.04894e-05 \\
      0.245702 3.8996e-05 \\ 0.256848 3.0339e-05 \\ 0.27359 1.52633e-05 \\ 0.284598 1.20985e-05 \\ 0.301452 5.95702e-06 \\ 0.312522 4.848e-06 \\
      0.330713 2.48231e-06 \\ 0.340767 2.0914e-06 \\ 0.357419 1.09324e-06 \\ 0.367262 9.40484e-07 \\ 0.384914 4.94303e-07 \\ 0.394927 4.30823e-07 \\
      0.412141 2.24572e-07 \\ 0.422151 1.97364e-07 \\ 0.441895 1.00573e-07 \\ 0.45949 5.211e-08 \\ 0.476626 2.50597e-08 \\ 0.487659 1.96207e-08 \\
      0.506292 8.06526e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(b) a9a, runtime};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=85.5,
        ymin=2.237e-09, ymax=1.759e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.545341 \\ 2 0.43393 \\ 3 0.335126 \\ 4 0.269546 \\ 5 0.172444 \\
      6 0.118233 \\ 7 0.0833317 \\ 8 0.0677165 \\ 9 0.0545676 \\ 10 0.0416062 \\ 11 0.0337825 \\
      12 0.0185656 \\ 13 0.0132214 \\ 14 0.0105894 \\ 15 0.00719845 \\ 16 0.00587158 \\ 17 0.00382227 \\
      18 0.00315565 \\ 19 0.00198005 \\ 20 0.00164922 \\ 21 0.00100297 \\ 22 0.000854502 \\ 23 0.000554198 \\
      24 0.000470355 \\ 25 0.000307366 \\ 26 0.000260156 \\ 27 0.000170825 \\ 28 0.000144181 \\ 29 9.49728e-05 \\
      30 8.05667e-05 \\ 31 5.56683e-05 \\ 32 4.68785e-05 \\ 33 3.22474e-05 \\ 34 2.70617e-05 \\ 35 1.20885e-05 \\
      36 8.26427e-06 \\ 37 6.85102e-06 \\ 38 4.68655e-06 \\ 39 3.90975e-06 \\ 40 2.67279e-06 \\ 41 2.25053e-06 \\
      42 1.61275e-06 \\ 43 1.35448e-06 \\ 44 9.76483e-07 \\ 45 8.23972e-07 \\ 46 6.0661e-07 \\ 47 5.18093e-07 \\
      48 2.62543e-07 \\ 49 2.20622e-07 \\ 50 1.98813e-07 \\ 51 1.79266e-07 \\ 52 1.67256e-07 \\ 53 1.16853e-07 \\
      54 1.0838e-07 \\ 55 7.37312e-08 \\ 56 6.61626e-08 \\ 57 4.11142e-08 \\ 58 3.8803e-08 \\ 59 3.51083e-08 \\
      60 3.28428e-08 \\ 61 3.15616e-08 \\ 62 3.0502e-08 \\ 63 2.73174e-08 \\ 64 2.5754e-08 \\ 65 2.5091e-08 \\
      66 2.44342e-08 \\ 67 2.17863e-08 \\ 68 2.05239e-08 \\ 69 2.00135e-08 \\ 70 1.95522e-08 \\ 71 1.7774e-08 \\
      72 1.69198e-08 \\ 73 1.65541e-08 \\ 74 1.6254e-08 \\ 75 1.49375e-08 \\ 76 1.4278e-08 \\ 77 1.39814e-08 \\
      78 1.37723e-08 \\ 79 1.36871e-08 \\ 80 1.27964e-08 \\ 81 1.23106e-08 \\ 82 1.20622e-08 \\ 83 5.67549e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.536883 \\ 2 0.421246 \\ 3 0.340824 \\ 4 0.275528 \\ 5 0.210846 \\
      6 0.162096 \\ 7 0.0999726 \\ 8 0.0761073 \\ 9 0.046244 \\ 10 0.0349206 \\ 11 0.0202517 \\
      12 0.0145337 \\ 13 0.00803647 \\ 14 0.00556781 \\ 15 0.00295004 \\ 16 0.00186592 \\ 17 0.000954641 \\
      18 0.000593362 \\ 19 0.000296123 \\ 20 0.000181345 \\ 21 8.88166e-05 \\ 22 5.36655e-05 \\ 23 2.58613e-05 \\
      24 1.24581e-05 \\ 25 7.84227e-06 \\ 26 3.69761e-06 \\ 27 1.73294e-06 \\ 28 1.08077e-06 \\ 29 4.90161e-07 \\
      30 2.19728e-07 \\ 31 1.31945e-07 \\ 32 6.40227e-08 \\ 33 3.67988e-08 \\ 34 2.55849e-08 \\ 35 1.91521e-08 \\
      36 7.93124e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.431157 \\ 2 0.288745 \\ 3 0.201799 \\ 4 0.144547 \\ 5 0.105459 \\
      6 0.0769296 \\ 7 0.0548534 \\ 8 0.0379864 \\ 9 0.0255232 \\ 10 0.0165972 \\ 11 0.0104031 \\
      12 0.00633232 \\ 13 0.00377324 \\ 14 0.00220642 \\ 15 0.00126914 \\ 16 0.000719564 \\ 17 0.000402804 \\
      18 0.000222927 \\ 19 0.000122108 \\ 20 6.62528e-05 \\ 21 3.56294e-05 \\ 22 1.89963e-05 \\ 23 1.00389e-05 \\
      24 5.25314e-06 \\ 25 2.71575e-06 \\ 26 1.38111e-06 \\ 27 6.857e-07 \\ 28 3.28059e-07 \\ 29 1.47979e-07 \\
      30 6.06975e-08 \\ 31 2.13675e-08 \\ 32 5.98003e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 1 0.509233 \\ 2 0.396147 \\ 3 0.321033 \\ 4 0.264128 \\ 5 0.163973 \\
      6 0.107948 \\ 7 0.0726923 \\ 8 0.0579912 \\ 9 0.0366024 \\ 10 0.0279694 \\ 11 0.0164888 \\
      12 0.0119683 \\ 13 0.00663298 \\ 14 0.00465468 \\ 15 0.002467 \\ 16 0.00169334 \\ 17 0.000866022 \\
      18 0.000440597 \\ 19 0.000293594 \\ 20 0.000145472 \\ 21 7.198e-05 \\ 22 4.70487e-05 \\ 23 2.2772e-05 \\
      24 1.10081e-05 \\ 25 7.08416e-06 \\ 26 3.33875e-06 \\ 27 1.55924e-06 \\ 28 9.83611e-07 \\ 29 4.37564e-07 \\
      30 1.87534e-07 \\ 31 7.63759e-08 \\ 32 4.96572e-08 \\ 33 2.09909e-08 \\ 34 9.91787e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(c) Mushrooms, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=0.356,
        ymin=2.237e-09, ymax=1.759e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.00408658 0.545341 \\ 0.00735092 0.43393 \\ 0.011082 0.335126 \\ 0.0147091 0.269546 \\ 0.0190524 0.172444 \\
      0.0232187 0.118233 \\ 0.0276763 0.0833317 \\ 0.0313986 0.0677165 \\ 0.0345645 0.0545676 \\ 0.0381675 0.0416062 \\ 0.0418025 0.0337825 \\
      0.0466598 0.0185656 \\ 0.050676 0.0132214 \\ 0.0540299 0.0105894 \\ 0.0586222 0.00719845 \\ 0.0627108 0.00587158 \\ 0.0669422 0.00382227 \\
      0.0704386 0.00315565 \\ 0.074295 0.00198005 \\ 0.078443 0.00164922 \\ 0.0830586 0.00100297 \\ 0.0886173 0.000854502 \\ 0.092882 0.000554198 \\
      0.0974655 0.000470355 \\ 0.101652 0.000307366 \\ 0.106918 0.000260156 \\ 0.11164 0.000170825 \\ 0.116247 0.000144181 \\ 0.120684 9.49728e-05 \\
      0.126333 8.05667e-05 \\ 0.130519 5.56683e-05 \\ 0.135005 4.68785e-05 \\ 0.139323 3.22474e-05 \\ 0.143609 2.70617e-05 \\ 0.148151 1.20885e-05 \\
      0.152279 8.26427e-06 \\ 0.15748 6.85102e-06 \\ 0.161608 4.68655e-06 \\ 0.165653 3.90975e-06 \\ 0.169356 2.67279e-06 \\ 0.174187 2.25053e-06 \\
      0.178098 1.61275e-06 \\ 0.181875 1.35448e-06 \\ 0.185978 9.76483e-07 \\ 0.189573 8.23972e-07 \\ 0.193582 6.0661e-07 \\ 0.197489 5.18093e-07 \\
      0.201651 2.62543e-07 \\ 0.205175 2.20622e-07 \\ 0.209514 1.98813e-07 \\ 0.213217 1.79266e-07 \\ 0.217116 1.67256e-07 \\ 0.221136 1.16853e-07 \\
      0.224988 1.0838e-07 \\ 0.229481 7.37312e-08 \\ 0.232881 6.61626e-08 \\ 0.237257 4.11142e-08 \\ 0.24046 3.8803e-08 \\ 0.248668 3.51083e-08 \\
      0.251407 3.28428e-08 \\ 0.254675 3.15616e-08 \\ 0.257417 3.0502e-08 \\ 0.264929 2.73174e-08 \\ 0.267721 2.5754e-08 \\ 0.271212 2.5091e-08 \\
      0.273871 2.44342e-08 \\ 0.280087 2.17863e-08 \\ 0.282862 2.05239e-08 \\ 0.285991 2.00135e-08 \\ 0.28895 1.95522e-08 \\ 0.296679 1.7774e-08 \\
      0.299478 1.69198e-08 \\ 0.302694 1.65541e-08 \\ 0.306381 1.6254e-08 \\ 0.313245 1.49375e-08 \\ 0.316443 1.4278e-08 \\ 0.319742 1.39814e-08 \\
      0.323453 1.37723e-08 \\ 0.32727 1.36871e-08 \\ 0.333139 1.27964e-08 \\ 0.336433 1.23106e-08 \\ 0.340138 1.20622e-08 \\ 0.345409 5.67549e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.00408704 0.536883 \\ 0.00754121 0.421246 \\ 0.0115202 0.340824 \\ 0.0160289 0.275528 \\ 0.0203637 0.210846 \\
      0.0254362 0.162096 \\ 0.0304781 0.0999726 \\ 0.034875 0.0761073 \\ 0.040075 0.046244 \\ 0.0441938 0.0349206 \\ 0.0489762 0.0202517 \\
      0.0529205 0.0145337 \\ 0.0578235 0.00803647 \\ 0.06206 0.00556781 \\ 0.0671782 0.00295004 \\ 0.0714023 0.00186592 \\ 0.0760384 0.000954641 \\
      0.0802573 0.000593362 \\ 0.0854147 0.000296123 \\ 0.0899539 0.000181345 \\ 0.0948024 8.88166e-05 \\ 0.0994938 5.36655e-05 \\ 0.104053 2.58613e-05 \\
      0.109679 1.24581e-05 \\ 0.114145 7.84227e-06 \\ 0.120098 3.69761e-06 \\ 0.125357 1.73294e-06 \\ 0.1303 1.08077e-06 \\ 0.135878 4.90161e-07 \\
      0.141484 2.19728e-07 \\ 0.14636 1.31945e-07 \\ 0.151923 6.40227e-08 \\ 0.157568 3.67988e-08 \\ 0.163267 2.55849e-08 \\ 0.168935 1.91521e-08 \\
      0.174809 7.93124e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.00592042 0.431157 \\ 0.0113538 0.288745 \\ 0.0168111 0.201799 \\ 0.0217997 0.144547 \\ 0.0273106 0.105459 \\
      0.0324927 0.0769296 \\ 0.0376236 0.0548534 \\ 0.0424066 0.0379864 \\ 0.0473991 0.0255232 \\ 0.0524854 0.0165972 \\ 0.0576825 0.0104031 \\
      0.0627549 0.00633232 \\ 0.0678747 0.00377324 \\ 0.072809 0.00220642 \\ 0.0778794 0.00126914 \\ 0.0829668 0.000719564 \\ 0.0878128 0.000402804 \\
      0.0928164 0.000222927 \\ 0.098061 0.000122108 \\ 0.102743 6.62528e-05 \\ 0.107963 3.56294e-05 \\ 0.112884 1.89963e-05 \\ 0.117561 1.00389e-05 \\
      0.122489 5.25314e-06 \\ 0.12761 2.71575e-06 \\ 0.132671 1.38111e-06 \\ 0.137637 6.857e-07 \\ 0.142832 3.28059e-07 \\ 0.147933 1.47979e-07 \\
      0.153241 6.06975e-08 \\ 0.158213 2.13675e-08 \\ 0.162984 5.98003e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.693147 \\ 0.005493 0.509233 \\ 0.00962942 0.396147 \\ 0.0139168 0.321033 \\ 0.018858 0.264128 \\ 0.0246076 0.163973 \\
      0.0307511 0.107948 \\ 0.0366311 0.0726923 \\ 0.041213 0.0579912 \\ 0.0462771 0.0366024 \\ 0.0502911 0.0279694 \\ 0.05582 0.0164888 \\
      0.059893 0.0119683 \\ 0.0650789 0.00663298 \\ 0.0689845 0.00465468 \\ 0.0741242 0.002467 \\ 0.0780105 0.00169334 \\ 0.0834156 0.000866022 \\
      0.0881778 0.000440597 \\ 0.0921121 0.000293594 \\ 0.0971826 0.000145472 \\ 0.102224 7.198e-05 \\ 0.105959 4.70487e-05 \\ 0.111309 2.2772e-05 \\
      0.116578 1.10081e-05 \\ 0.121185 7.08416e-06 \\ 0.126142 3.33875e-06 \\ 0.131007 1.55924e-06 \\ 0.134805 9.83611e-07 \\ 0.140519 4.37564e-07 \\
      0.145549 1.87534e-07 \\ 0.151461 7.63759e-08 \\ 0.156449 4.96572e-08 \\ 0.161866 2.09909e-08 \\ 0.167814 9.91787e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(d) Mushrooms, runtime};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=184,
        ymin=1.319e-09, ymax=1.503e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Iterations},
        ylabel={$f(x_k)-f^*$}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 1 0.4337 \\ 2 0.33647 \\ 3 0.267829 \\ 4 0.218 \\ 5 0.163284 \\
      6 0.138639 \\ 7 0.109985 \\ 8 0.0911602 \\ 9 0.0786289 \\ 10 0.0660601 \\ 11 0.0571492 \\
      12 0.047937 \\ 13 0.0367164 \\ 14 0.0335445 \\ 15 0.028686 \\ 16 0.0222888 \\ 17 0.0167878 \\
      18 0.0149408 \\ 19 0.0129415 \\ 20 0.0088422 \\ 21 0.00790385 \\ 22 0.00711905 \\ 23 0.00435323 \\
      24 0.00395944 \\ 25 0.00339657 \\ 26 0.00231424 \\ 27 0.00118693 \\ 28 0.000947434 \\ 29 0.000829448 \\
      30 0.000562658 \\ 31 0.0004685 \\ 32 0.000413601 \\ 33 0.000327178 \\ 34 0.000159744 \\ 35 0.000140826 \\
      36 0.000126295 \\ 37 9.67604e-05 \\ 38 9.23581e-05 \\ 39 8.63374e-05 \\ 40 7.55509e-05 \\ 41 7.2893e-05 \\
      42 6.81414e-05 \\ 43 3.13889e-05 \\ 44 3.02759e-05 \\ 45 2.82147e-05 \\ 46 2.70396e-05 \\ 47 2.5801e-05 \\
      48 2.52873e-05 \\ 49 2.44842e-05 \\ 50 2.41719e-05 \\ 51 2.36779e-05 \\ 52 1.11925e-05 \\ 53 1.08024e-05 \\
      54 1.05145e-05 \\ 55 1.036e-05 \\ 56 1.02138e-05 \\ 57 1.0127e-05 \\ 58 1.00069e-05 \\ 59 5.17811e-06 \\
      60 5.0127e-06 \\ 61 4.85964e-06 \\ 62 4.80262e-06 \\ 63 4.74811e-06 \\ 64 2.8197e-06 \\ 65 2.67014e-06 \\
      66 2.63777e-06 \\ 67 2.6088e-06 \\ 68 1.784e-06 \\ 69 1.66696e-06 \\ 70 1.64516e-06 \\ 71 1.2906e-06 \\
      72 1.19255e-06 \\ 73 1.17857e-06 \\ 74 1.00888e-06 \\ 75 9.3084e-07 \\ 76 9.21717e-07 \\ 77 8.28977e-07 \\
      78 7.65878e-07 \\ 79 7.58939e-07 \\ 80 6.98798e-07 \\ 81 6.46936e-07 \\ 82 6.41388e-07 \\ 83 5.9688e-07 \\
      84 5.53893e-07 \\ 85 5.49367e-07 \\ 86 5.13825e-07 \\ 87 4.7802e-07 \\ 88 4.7429e-07 \\ 89 4.44964e-07 \\
      90 4.15048e-07 \\ 91 4.11957e-07 \\ 92 3.8748e-07 \\ 93 3.62431e-07 \\ 94 3.59859e-07 \\ 95 3.39349e-07 \\
      96 3.18342e-07 \\ 97 3.16195e-07 \\ 98 2.98982e-07 \\ 99 2.81344e-07 \\ 100 2.79548e-07 \\ 101 2.65087e-07 \\
      102 2.06796e-07 \\ 103 1.96355e-07 \\ 104 1.94682e-07 \\ 105 1.82166e-07 \\ 106 1.69397e-07 \\ 107 1.68092e-07 \\
      108 1.57649e-07 \\ 109 1.4692e-07 \\ 110 1.4583e-07 \\ 111 1.37038e-07 \\ 112 1.28007e-07 \\ 113 1.27092e-07 \\
      114 1.19688e-07 \\ 115 1.12082e-07 \\ 116 1.11313e-07 \\ 117 1.05077e-07 \\ 118 9.8669e-08 \\ 119 9.80215e-08 \\
      120 9.27674e-08 \\ 121 8.73665e-08 \\ 122 8.68211e-08 \\ 123 8.23931e-08 \\ 124 7.784e-08 \\ 125 7.73802e-08 \\
      126 7.36475e-08 \\ 127 6.9808e-08 \\ 128 6.94204e-08 \\ 129 6.62729e-08 \\ 130 6.30345e-08 \\ 131 6.27076e-08 \\
      132 6.0053e-08 \\ 133 5.7321e-08 \\ 134 5.70452e-08 \\ 135 5.48058e-08 \\ 136 5.25007e-08 \\ 137 5.22679e-08 \\
      138 5.03784e-08 \\ 139 4.84331e-08 \\ 140 4.82366e-08 \\ 141 4.66421e-08 \\ 142 4.50001e-08 \\ 143 4.48343e-08 \\
      144 4.34885e-08 \\ 145 4.21023e-08 \\ 146 4.19623e-08 \\ 147 4.08262e-08 \\ 148 3.96559e-08 \\ 149 3.95377e-08 \\
      150 3.85786e-08 \\ 151 3.75904e-08 \\ 152 3.74906e-08 \\ 153 3.66807e-08 \\ 154 3.58461e-08 \\ 155 3.57618e-08 \\
      156 3.50779e-08 \\ 157 3.43731e-08 \\ 158 3.43019e-08 \\ 159 3.37242e-08 \\ 160 3.31289e-08 \\ 161 3.30687e-08 \\
      162 3.25808e-08 \\ 163 3.20779e-08 \\ 164 3.20271e-08 \\ 165 3.1615e-08 \\ 166 3.11901e-08 \\ 167 3.11472e-08 \\
      168 1.23478e-08 \\ 169 1.19925e-08 \\ 170 1.16233e-08 \\ 171 1.15849e-08 \\ 172 1.12871e-08 \\ 173 1.09804e-08 \\
      174 1.09493e-08 \\ 175 1.06982e-08 \\ 176 1.04392e-08 \\ 177 1.0413e-08 \\ 178 1.02008e-08 \\ 179 9.98206e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 1 0.451223 \\ 2 0.358614 \\ 3 0.26243 \\ 4 0.199731 \\ 5 0.157979 \\
      6 0.129642 \\ 7 0.109689 \\ 8 0.0948846 \\ 9 0.0764026 \\ 10 0.0630313 \\ 11 0.0528272 \\
      12 0.0448301 \\ 13 0.032912 \\ 14 0.0278534 \\ 15 0.0208812 \\ 16 0.0159524 \\ 17 0.0100414 \\
      18 0.00860714 \\ 19 0.00694714 \\ 20 0.00458545 \\ 21 0.00221401 \\ 22 0.00190006 \\ 23 0.00164359 \\
      24 0.000956062 \\ 25 0.000281592 \\ 26 0.000257799 \\ 27 0.000168989 \\ 28 5.51274e-05 \\ 29 4.85022e-05 \\
      30 2.08523e-05 \\ 31 1.86481e-05 \\ 32 7.83686e-06 \\ 33 7.20842e-06 \\ 34 5.1481e-06 \\ 35 2.1734e-06 \\
      36 1.83339e-06 \\ 37 7.48888e-07 \\ 38 6.68299e-07 \\ 39 4.54069e-07 \\ 40 3.45584e-07 \\ 41 1.32935e-07 \\
      42 4.57895e-08 \\ 43 1.35236e-08 \\ 44 3.40308e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 1 0.374003 \\ 2 0.263252 \\ 3 0.192927 \\ 4 0.146944 \\ 5 0.116017 \\
      6 0.0940388 \\ 7 0.0771577 \\ 8 0.0635129 \\ 9 0.0523406 \\ 10 0.0430097 \\ 11 0.0349751 \\
      12 0.028007 \\ 13 0.0220558 \\ 14 0.0170483 \\ 15 0.0128889 \\ 16 0.00947547 \\ 17 0.00669358 \\
      18 0.00445622 \\ 19 0.00273116 \\ 20 0.00153769 \\ 21 0.000800379 \\ 22 0.000377149 \\ 23 0.000164906 \\
      24 7.4716e-05 \\ 25 3.83748e-05 \\ 26 2.03407e-05 \\ 27 1.04754e-05 \\ 28 5.31126e-06 \\ 29 2.63769e-06 \\
      30 1.25998e-06 \\ 31 5.68508e-07 \\ 32 2.33498e-07 \\ 33 8.18241e-08 \\ 34 2.18315e-08 \\ 35 3.65207e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 1 0.408896 \\ 2 0.303376 \\ 3 0.231964 \\ 4 0.182626 \\ 5 0.148348 \\
      6 0.12419 \\ 7 0.106546 \\ 8 0.0811697 \\ 9 0.0715511 \\ 10 0.0544982 \\ 11 0.0485349 \\
      12 0.0364045 \\ 13 0.0273557 \\ 14 0.0241469 \\ 15 0.0171112 \\ 16 0.0151133 \\ 17 0.00991592 \\
      18 0.00630655 \\ 19 0.00523701 \\ 20 0.00285294 \\ 21 0.00242302 \\ 22 0.00107215 \\ 23 0.000428186 \\
      24 0.000360487 \\ 25 0.000124262 \\ 26 0.000110197 \\ 27 4.80827e-05 \\ 28 4.42553e-05 \\ 29 2.28192e-05 \\
      30 2.12956e-05 \\ 31 1.00711e-05 \\ 32 9.22107e-06 \\ 33 4.38657e-06 \\ 34 4.01477e-06 \\ 35 1.91873e-06 \\
      36 1.75372e-06 \\ 37 8.16963e-07 \\ 38 7.4686e-07 \\ 39 3.2861e-07 \\ 40 3.02027e-07 \\ 41 1.07704e-07 \\
      42 1.00524e-07 \\ 43 2.90731e-08 \\ 44 2.80957e-08 \\ 45 4.92121e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(e) w8t, iterations};
\begin{axis}[name=ax,
        mlrnaxis, ymode=log,
        xmin=0, xmax=6.73,
        ymin=1.319e-09, ymax=1.503e+00,
        ytickten={-8,-6,-4,-2,0},
        xlabel={Runtime (s)}]
    \addplot[mlrn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 0.0362168 0.4337 \\ 0.0636937 0.33647 \\ 0.090266 0.267829 \\ 0.115878 0.218 \\ 0.162783 0.163284 \\
      0.187926 0.138639 \\ 0.22459 0.109985 \\ 0.263127 0.0911602 \\ 0.290982 0.0786289 \\ 0.328775 0.0660601 \\ 0.354793 0.0571492 \\
      0.391761 0.047937 \\ 0.442286 0.0367164 \\ 0.45883 0.0335445 \\ 0.485384 0.028686 \\ 0.525437 0.0222888 \\ 0.57675 0.0167878 \\
      0.591198 0.0149408 \\ 0.616995 0.0129415 \\ 0.653441 0.0088422 \\ 0.667518 0.00790385 \\ 0.692655 0.00711905 \\ 0.731601 0.00435323 \\
      0.758611 0.00395944 \\ 0.773411 0.00339657 \\ 0.809988 0.00231424 \\ 0.860987 0.00118693 \\ 0.87536 0.000947434 \\ 0.901577 0.000829448 \\
      0.941766 0.000562658 \\ 0.957516 0.0004685 \\ 0.983942 0.000413601 \\ 1.03656 0.000327178 \\ 1.10021 0.000159744 \\ 1.11632 0.000140826 \\
      1.14355 0.000126295 \\ 1.1819 9.67604e-05 \\ 1.19675 9.23581e-05 \\ 1.22212 8.63374e-05 \\ 1.26966 7.55509e-05 \\ 1.29466 7.2893e-05 \\
      1.33709 6.81414e-05 \\ 1.41193 3.13889e-05 \\ 1.42603 3.02759e-05 \\ 1.4626 2.82147e-05 \\ 1.4878 2.70396e-05 \\ 1.52467 2.5801e-05 \\
      1.55017 2.52873e-05 \\ 1.58985 2.44842e-05 \\ 1.6162 2.41719e-05 \\ 1.65361 2.36779e-05 \\ 1.73308 1.11925e-05 \\ 1.74725 1.08024e-05 \\
      1.78726 1.05145e-05 \\ 1.81399 1.036e-05 \\ 1.85131 1.02138e-05 \\ 1.87643 1.0127e-05 \\ 1.91293 1.00069e-05 \\ 1.98129 5.17811e-06 \\
      1.99612 5.0127e-06 \\ 2.03539 4.85964e-06 \\ 2.06187 4.80262e-06 \\ 2.09879 4.74811e-06 \\ 2.16042 2.8197e-06 \\ 2.19691 2.67014e-06 \\
      2.22375 2.63777e-06 \\ 2.26289 2.6088e-06 \\ 2.31742 1.784e-06 \\ 2.35395 1.66696e-06 \\ 2.37926 1.64516e-06 \\ 2.43023 1.2906e-06 \\
      2.47106 1.19255e-06 \\ 2.49686 1.17857e-06 \\ 2.54739 1.00888e-06 \\ 2.58419 9.3084e-07 \\ 2.60961 9.21717e-07 \\ 2.6631 8.28977e-07 \\
      2.70263 7.65878e-07 \\ 2.72825 7.58939e-07 \\ 2.77856 6.98798e-07 \\ 2.81519 6.46936e-07 \\ 2.84057 6.41388e-07 \\ 2.89345 5.9688e-07 \\
      2.93225 5.53893e-07 \\ 2.95777 5.49367e-07 \\ 3.00792 5.13825e-07 \\ 3.04474 4.7802e-07 \\ 3.06983 4.7429e-07 \\ 3.1222 4.44964e-07 \\
      3.16098 4.15048e-07 \\ 3.18615 4.11957e-07 \\ 3.23643 3.8748e-07 \\ 3.27345 3.62431e-07 \\ 3.29861 3.59859e-07 \\ 3.3509 3.39349e-07 \\
      3.39083 3.18342e-07 \\ 3.41648 3.16195e-07 \\ 3.46622 2.98982e-07 \\ 3.50435 2.81344e-07 \\ 3.52962 2.79548e-07 \\ 3.58133 2.65087e-07 \\
      3.6007 2.06796e-07 \\ 3.63933 1.96355e-07 \\ 3.66448 1.94682e-07 \\ 3.7145 1.82166e-07 \\ 3.75168 1.69397e-07 \\ 3.77709 1.68092e-07 \\
      3.83151 1.57649e-07 \\ 3.87098 1.4692e-07 \\ 3.89616 1.4583e-07 \\ 3.94674 1.37038e-07 \\ 3.98356 1.28007e-07 \\ 4.00875 1.27092e-07 \\
      4.06119 1.19688e-07 \\ 4.10045 1.12082e-07 \\ 4.12653 1.11313e-07 \\ 4.17636 1.05077e-07 \\ 4.213 9.8669e-08 \\ 4.23815 9.80215e-08 \\
      4.29062 9.27674e-08 \\ 4.32956 8.73665e-08 \\ 4.35505 8.68211e-08 \\ 4.40614 8.23931e-08 \\ 4.44299 7.784e-08 \\ 4.46796 7.73802e-08 \\
      4.52008 7.36475e-08 \\ 4.56128 6.9808e-08 \\ 4.58702 6.94204e-08 \\ 4.63867 6.62729e-08 \\ 4.67558 6.30345e-08 \\ 4.70088 6.27076e-08 \\
      4.75394 6.0053e-08 \\ 4.79283 5.7321e-08 \\ 4.81829 5.70452e-08 \\ 4.86867 5.48058e-08 \\ 4.90561 5.25007e-08 \\ 4.93092 5.22679e-08 \\
      4.98308 5.03784e-08 \\ 5.02284 4.84331e-08 \\ 5.04869 4.82366e-08 \\ 5.0989 4.66421e-08 \\ 5.13663 4.50001e-08 \\ 5.16195 4.48343e-08 \\
      5.2146 4.34885e-08 \\ 5.25429 4.21023e-08 \\ 5.28009 4.19623e-08 \\ 5.33052 4.08262e-08 \\ 5.36691 3.96559e-08 \\ 5.3921 3.95377e-08 \\
      5.44455 3.85786e-08 \\ 5.48401 3.75904e-08 \\ 5.50967 3.74906e-08 \\ 5.55998 3.66807e-08 \\ 5.59743 3.58461e-08 \\ 5.62353 3.57618e-08 \\
      5.67626 3.50779e-08 \\ 5.71505 3.43731e-08 \\ 5.74089 3.43019e-08 \\ 5.79116 3.37242e-08 \\ 5.82798 3.31289e-08 \\ 5.85317 3.30687e-08 \\
      5.90586 3.25808e-08 \\ 5.94467 3.20779e-08 \\ 5.96994 3.20271e-08 \\ 6.02035 3.1615e-08 \\ 6.05702 3.11901e-08 \\ 6.08216 3.11472e-08 \\
      6.10455 1.23478e-08 \\ 6.15806 1.19925e-08 \\ 6.19601 1.16233e-08 \\ 6.22118 1.15849e-08 \\ 6.27133 1.12871e-08 \\ 6.30793 1.09804e-08 \\
      6.33308 1.09493e-08 \\ 6.38453 1.06982e-08 \\ 6.42447 1.04392e-08 \\ 6.45018 1.0413e-08 \\ 6.50023 1.02008e-08 \\ 6.53732 9.98206e-09 \\
    };
    \addplot[mlrnnested] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 0.028616 0.451223 \\ 0.0535636 0.358614 \\ 0.0927941 0.26243 \\ 0.134111 0.199731 \\ 0.171692 0.157979 \\
      0.208458 0.129642 \\ 0.245271 0.109689 \\ 0.284643 0.0948846 \\ 0.338262 0.0764026 \\ 0.392724 0.0630313 \\ 0.446426 0.0528272 \\
      0.502853 0.0448301 \\ 0.569327 0.032912 \\ 0.628772 0.0278534 \\ 0.688632 0.0208812 \\ 0.752435 0.0159524 \\ 0.816952 0.0100414 \\
      0.853688 0.00860714 \\ 0.904211 0.00694714 \\ 0.964218 0.00458545 \\ 1.03152 0.00221401 \\ 1.06904 0.00190006 \\ 1.12007 0.00164359 \\
      1.18157 0.000956062 \\ 1.24889 0.000281592 \\ 1.28591 0.000257799 \\ 1.34624 0.000168989 \\ 1.41759 5.51274e-05 \\ 1.46897 4.85022e-05 \\
      1.54101 2.08523e-05 \\ 1.59149 1.86481e-05 \\ 1.66312 7.83686e-06 \\ 1.70105 7.20842e-06 \\ 1.76849 5.1481e-06 \\ 1.83267 2.1734e-06 \\
      1.88483 1.83339e-06 \\ 1.95024 7.48888e-07 \\ 1.98705 6.68299e-07 \\ 2.05488 4.54069e-07 \\ 2.11691 3.45584e-07 \\ 2.18261 1.32935e-07 \\
      2.24881 4.57895e-08 \\ 2.31313 1.35236e-08 \\ 2.38038 3.40308e-09 \\
    };
    \addplot[rn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 0.0702865 0.374003 \\ 0.135164 0.263252 \\ 0.199737 0.192927 \\ 0.267834 0.146944 \\ 0.332501 0.116017 \\
      0.400306 0.0940388 \\ 0.466096 0.0771577 \\ 0.536093 0.0635129 \\ 0.600554 0.0523406 \\ 0.664672 0.0430097 \\ 0.728891 0.0349751 \\
      0.795964 0.028007 \\ 0.860421 0.0220558 \\ 0.924539 0.0170483 \\ 0.988188 0.0128889 \\ 1.05518 0.00947547 \\ 1.1192 0.00669358 \\
      1.18327 0.00445622 \\ 1.24784 0.00273116 \\ 1.31502 0.00153769 \\ 1.37926 0.000800379 \\ 1.4434 0.000377149 \\ 1.50735 0.000164906 \\
      1.57525 7.4716e-05 \\ 1.63985 3.83748e-05 \\ 1.70509 2.03407e-05 \\ 1.76914 1.04754e-05 \\ 1.83877 5.31126e-06 \\ 1.90527 2.63769e-06 \\
      1.96963 1.25998e-06 \\ 2.03365 5.68508e-07 \\ 2.1011 2.33498e-07 \\ 2.16519 8.18241e-08 \\ 2.23124 2.18315e-08 \\ 2.29507 3.65207e-09 \\
    };
    \addplot[tlrn] table[row sep=\\] {
      x y \\
      0 0.582335 \\ 0.0412766 0.408896 \\ 0.0780677 0.303376 \\ 0.114693 0.231964 \\ 0.152118 0.182626 \\ 0.188808 0.148348 \\
      0.227908 0.12419 \\ 0.267764 0.106546 \\ 0.333339 0.0811697 \\ 0.370537 0.0715511 \\ 0.435259 0.0544982 \\ 0.474775 0.0485349 \\
      0.539207 0.0364045 \\ 0.602915 0.0273557 \\ 0.639684 0.0241469 \\ 0.707371 0.0171112 \\ 0.744888 0.0151133 \\ 0.808943 0.00991592 \\
      0.876341 0.00630655 \\ 0.91578 0.00523701 \\ 0.979897 0.00285294 \\ 1.01708 0.00242302 \\ 1.08118 0.00107215 \\ 1.14791 0.000428186 \\
      1.18491 0.000360487 \\ 1.25092 0.000124262 \\ 1.28803 0.000110197 \\ 1.36208 4.80827e-05 \\ 1.3988 4.42553e-05 \\ 1.47082 2.28192e-05 \\
      1.50862 2.12956e-05 \\ 1.57454 1.00711e-05 \\ 1.61109 9.22107e-06 \\ 1.67923 4.38657e-06 \\ 1.71646 4.01477e-06 \\ 1.78659 1.91873e-06 \\
      1.82329 1.75372e-06 \\ 1.89107 8.16963e-07 \\ 1.92806 7.4686e-07 \\ 1.99959 3.2861e-07 \\ 2.03662 3.02027e-07 \\ 2.10083 1.07704e-07 \\
      2.13846 1.00524e-07 \\ 2.20999 2.90731e-08 \\ 2.2472 2.80957e-08 \\ 2.31214 4.92121e-09 \\
    };
    \end{axis}
    \node[anchor=north,font=\small] at (ax.below south) {(f) w8t, runtime};

Figure 3. Convergence comparison of MLRN, row-nested MLRN, regularized Newton (RN), and two-level regularized Newton (TLRN) on the a9a, Mushrooms, and w8t datasets for logistic regression. The left column reports the objective gap f(xk)−f∗f(x_k)-f^* versus iterations, while the right column reports the objective gap versus wall-clock time. Row-nested restriction operators significantly improve the convergence behavior of MLRN.

Logistic regression

In the second experiment, we consider logistic regression problems using real-world datasets. The objective function is given by

f(x)=1m∑i=1mln⁡(1+exp⁡(−biai⊤x))+λ22∥x∥2,\begin{equation} f(x)=\frac{1}{m}\sum_{i=1}^{m} \ln\left(1+\exp\big(-b_i a_i^\top x\big)\right) +\frac{\lambda_2}{2}\left\lVert x \right\rVert^2, \tag*{(18)} \end{equation}

where bi∈{−1,1}b_i\in\{-1,1\} and we set λ2=10−10\lambda_2=10^{-10} to make the problem ill-conditioned. The results are presented in Figure 2 and Figure 3.

Across the considered datasets, nested MLRN and TLRN exhibit convergence behavior comparable to that of the full-space RN method. Moreover, as shown in Figure 2, when applied to large datasets, both methods achieve substantial reductions in computational time, illustrating the benefits of performing corrections in lower-dimensional spaces. In contrast, non-row-nested MLRN exhibits slower convergence on some instances, particularly those shown in Figure 3, which may prevent the method from converging faster than RN in computational time. These observations suggest that the structure of the restriction operators, particularly row-nestedness, can play an important role in the practical performance of multilevel gradient-regularized Newton methods. Overall, the results demonstrate that row-nested restriction operators can yield computationally efficient multilevel methods, complementing the theoretical convergence guarantees established in this paper.

Conclusions

We developed and analyzed multilevel gradient-regularized Newton methods for unconstrained convex optimization, establishing a global O(k−2)\mathcal{O}\big(k^{-2}\big) convergence rate in function values. Among our results, the proof of Theorem 3.8 is of particular interest as it combines a weighted gradient measure with row-nestedness, a special structure of the restriction operators introduced in § 3.2 for the purposes of this analysis. These ideas are new in multilevel optimization theory and we believe that they could prove useful for establishing fast convergence rates for other multilevel methods, such as cubic-regularized Newton and higher-order methods for convex optimization.

Code and data availability

The code and data for reproducing the numerical results is available at

https://github.com/Ntsip/Multilevel-Gradient-Regularized-Newton-Method.

It is authored by Nick Tsipinakis and published under the MIT license.

CRediT author statement

Nick Tsipinakis: Conceptualization, Methodology, Software, Formal Analysis, Investigation, Writing — Original Draft, Visualization; Matthias Voigt: Writing — Review & Editing, Visualization, Supervision.

Acknowledgments

The authors used AI tools to assist with language and figure editing, mathematical proof verification, and the development and debugging of numerical implementations. All mathematical results, algorithms, and computational experiments were reviewed and validated by the authors. The authors assume responsibility for all content.

Background and proofs of auxiliary results

Proof. If Δℓ=0\Delta_\ell=0 for some ℓ\ell, the recurrence and nonnegativity imply Δj=0\Delta_j=0 for all j≥ℓj\geq\ell, so the result follows immediately. Otherwise, nonnegativity implies CΔℓ<1C\sqrt{\Delta_\ell} < 1, and hence

1Δℓ+1≥1Δℓ11−CΔℓ≥1Δℓ+C2,\frac{1}{\sqrt{\Delta_{\ell+1}}} \geq \frac{1}{\sqrt{\Delta_\ell}} \frac{1}{\sqrt{1-C\sqrt{\Delta_\ell}}} \geq \frac{1}{\sqrt{\Delta_\ell}}+\frac{C}{2},

where we used (1−s)−1/2≥1+s/2(1-s)^{-1/2}\geq1+s/2 for s∈[0,1)s\in[0,1). Summing 1Δℓ+1−1Δℓ\frac{1}{\sqrt{\Delta_{\ell+1}}}- \frac{1}{\sqrt{\Delta_{\ell}}} over ℓ=0,…,τ−1\ell=0,\ldots,\tau-1, exploiting the telescoping property, and using 1/Δ0≥C1/\sqrt{\Delta_0}\geq C gives

1Δτ≥C(1+τ2),\frac{1}{\sqrt{\Delta_\tau}} \geq C\left(1+\frac{\tau}{2}\right),

which proves the claim. ◻

We also refer to [7 Theorem 4.1.4] for a related recurrence estimate.

Proof. Let k∈Cik\in\mathcal C_i for some i∈Ji\in\mathcal J, and define

αi,k≔ωi3L∥Ri∇fk∥2,di,k≔−(Ri∇2fkPi+αi,kIni)−1Ri∇fk.\alpha_{i,k}\coloneqq \sqrt{\frac{\omega_i^3L\left\lVert R_i\nabla f_k \right\rVert}{2}}, \qquad d_{i,k}\coloneqq -\left(R_i\nabla^2 f_kP_i+\alpha_{i,k}I_{n_i}\right)^{-1} R_i\nabla f_k.

By the coarse-level eligibility condition,

αi,k≥ωi3Lε2>0.\alpha_{i,k}\geq \sqrt{\frac{\omega_i^3L\varepsilon}{2}}>0.

Moreover, since the iterates remain in the compact set L(x0)\mathcal L(x_0), there exists

G≔max⁡x∈L(x0)∥∇f(x)∥<∞.G\coloneqq\max_{x\in\mathcal L(x_0)}\left\lVert \nabla f(x) \right\rVert<\infty.

Consequently,

αi,k≤αˉi≔ωi2LG2.\alpha_{i,k}\leq \bar\alpha_i\coloneqq\omega_i^2\sqrt{\frac{LG}{2}}.

Using the definition of di,kd_{i,k}, we obtain

ε≤∥Ri∇fk∥≤(ωi2H+αi,k)∥di,k∥≤(ωi2H+αˉi)∥di,k∥.%\begin{aligned} \varepsilon \leq \left\lVert R_i\nabla f_k \right\rVert \leq \left(\omega_i^2H+\alpha_{i,k}\right) \left\lVert d_{i,k} \right\rVert \leq \left(\omega_i^2H+\bar\alpha_i\right) \left\lVert d_{i,k} \right\rVert. %\end{aligned}

Hence,

∥di,k∥≥εωi2H+αˉi.\left\lVert d_{i,k} \right\rVert\geq \frac{\varepsilon}{\omega_i^2H+\bar\alpha_i}.

By the descent estimate in [12 Lemma 3.3], the coarse update satisfies

f(xk)−f(xk+1)≥αi,k2∥di,k∥2.f(x_k)-f(x_{k+1}) \geq \frac{\alpha_{i,k}}{2}\left\lVert d_{i,k} \right\rVert^2.

Combining the preceding estimates yields

f(xk)−f(xk+1)≥12ωi3Lε2(εωi2H+αˉi)2⏟≕δi>0.f(x_k)-f(x_{k+1}) \geq \underbrace{ \frac12\sqrt{\frac{\omega_i^3L\varepsilon}{2}} \left( \frac{\varepsilon}{\omega_i^2H+\bar\alpha_i} \right)^2 }_{\eqqcolon \delta_i>0}.

Since the number of coarse levels is finite, δ≔min⁡i∈Jδi>0\delta\coloneqq\min_{i\in\mathcal J}\delta_i>0. Therefore, every coarse iteration decreases the objective function by at least δ\delta. By monotonicity of the function values and their lower boundedness by f∗f^*,

∣C(ℓ)∣δ≤f(x0)−f∗.\big|\mathcal C^{(\ell)}\big| \delta \leq f(x_0)-f^*.

Thus, ∣C(ℓ)∣<∞\big|\mathcal C^{(\ell)}\big|<\infty, proving the result. ◻

Proof of lemma:multilevel-coarse-estimates

Proof. The result follows from the analysis in [12 Lemma 3.2 and Corollary 3.4], which allows the restriction operator to vary with the iteration. Indeed, at an iteration k∈Cik\in\mathcal{C}_i, applying the corresponding estimates with Rk=RiR_k=R_i and ωi=∥Ri∥\omega_i=\|R_i\| yields

∥Ri∇fk+1∥≤2αk∥di,k∥,∥di,k∥≤∥Ri∇fk∥αk,\|R_i\nabla f_{k+1}\| \leq 2\alpha_k\|d_{i,k}\|, \qquad \|d_{i,k}\| \leq \frac{\|R_i\nabla f_k\|}{\alpha_k},

where

αk=ωi3L∥Ri∇fk∥2,di,k≔−(Ri∇2fkPi+αkIni)−1Ri∇fk.\alpha_k= \sqrt{\frac{\omega_i^3L\|R_i\nabla f_k\|}{2}}, \qquad d_{i,k}\coloneqq -\left(R_i\nabla^2f_kP_i+\alpha_kI_{n_i}\right)^{-1}R_i\nabla f_k.

These inequalities yield ineq:multilevel-reduced-gradient-stability, while the proof of [12 Corollary 3.4] gives ineq:multilevel-reduction. ◻

Proof of lemma:case-II

Proof. The proof of Case II in Theorem 2.5 extends naturally to ℓ\ell coarse levels. Thus, below, only the main steps of the proof will be discussed. Let τ≔∣Iℓk(ρ)∣\tau\coloneqq\big|\mathcal I_\ell^k(\rho)\big| and enumerate Iℓk(ρ)={k0,…,kτ−1}\mathcal I_\ell^k(\rho)=\{k_0,\ldots,k_{\tau-1}\} increasingly, with kτ≔kk_\tau\coloneqq k. If kj∈ICik(ρ)k_j\in\mathcal I_{\mathcal C_i}^k(\rho) for some i∈Ji\in\mathcal J, then, using the coarse decrease estimate ωi≤ζ\omega_i\leq\zeta and ∥Ri∇fkj∥≥σ∥∇fkj∥\|R_i\nabla f_{k_j}\|\geq\sigma\|\nabla f_{k_j}\|, we obtain

f(xkj)−f(xkj+1)≥2ρ28L(σζ)3/2∥∇fkj∥3/2.f(x_{k_j})-f(x_{k_j+1}) \geq \frac{\sqrt{2}\rho^2}{8\sqrt L} \left(\frac{\sigma}{\zeta}\right)^{3/2} \|\nabla f_{k_j}\|^{3/2}.

The same bound holds for kj∈IF(ℓ)k(ρ)k_j\in\mathcal I_{\mathcal F^{(\ell)}}^k(\rho) by the fine-level decrease estimate. Hence, arguing as in Case II of Theorem 2.5,

Δj+1≤Δj−CℓΔj3/2,Δj≔f(xkj)−f∗.\Delta_{j+1} \leq \Delta_j-C_\ell\Delta_j^{3/2}, \qquad \Delta_j\coloneqq f(x_{k_j})-f^*.

Since τ>θk\tau>\theta k, Proposition prop:recurrence-rate yields

f(xk)−f∗≤1Cℓ2(1+θk/2)2.f(x_k)-f^* \leq \frac{1}{C_\ell^2(1+\theta k/2)^2}.

◻

References

  1. [1]Gratton, S., Sartenaer, A., & Toint, P. L. (2008). Recursive Trust-Region Methods for Multiscale Nonlinear Optimization. SIAM J. Optim., 19(1), 414–444. https://doi.org/10.1137/050623012
  2. [2]Nash, S. G. (2000). A Multigrid Approach to Discretized Optimization Problems. Optim. Methods Softw., 14(1-2), 99–116. https://doi.org/10.1080/10556780008805795
  3. [3]Ho, C. P., Kočvara, M., & Parpas, P. (2022). Newton-Type Multilevel Optimization Method. Optim. Methods Softw., 37(1), 45–78. https://doi.org/10.1080/10556788.2019.1700256
  4. [4]Wen, Z., & Goldfarb, D. (2009). A Line Search Multigrid Method for Large-Scale Nonlinear Optimization. SIAM J. Optim., 20(3), 1478–1503. https://doi.org/10.1137/08071524X
  5. [5]Tsipinakis, N., & Parpas, P. (2024). A Multilevel Method for Self-Concordant Minimization. J. Optim. Theory Appl., 203(3), 2509–2559. https://doi.org/10.1007/s10957-024-02509-z
  6. [6]Tsipinakis, N., Tigas, P., & Parpas, P. (2026). A Multilevel Low-Rank Newton Method with Super-Linear Convergence Rate and Its Application to Non-Convex Problems. Trans. Mach. Learn. Res.. https://openreview.net/forum?id=PKakPzVVjaopenreview.net/forum?id=PKakPzVVja
  7. [7]Nesterov, Y. (2018). Lectures on Convex Optimization (Vol. 137). Springer. https://doi.org/10.1007/978-3-319-91578-4
  8. [9]Nesterov, Y. (2008). Accelerating the Cubic Regularization of Newton's Method on Convex Problems. Math. Program., 112(1), 159–181. https://doi.org/10.1007/s10107-006-0089-x
  9. [10]Tsipinakis, N., & Parpas, P. (2025). Convergence Rates of Newton's Method for Strongly Self-Concordant Minimization (Numbers arXiv:2507.23558). https://doi.org/10.48550/arXiv.2507.23558
  10. [11]Gross, C., & Krause, R. (2009). On the Convergence of Recursive Trust-Region Methods for Multiscale Nonlinear Optimization and Applications to Nonlinear Mechanics. SIAM J. Numer. Anal., 47(4), 3044–3069. https://doi.org/10.1137/08071819X
  11. [12]Tsipinakis, N., & Parpas, P. (2025). Multilevel Regularized Newton Methods with Fast Convergence Rates. SIAM J. Sci. Comput., S232–S257. https://doi.org/10.1137/24M1677307
  12. [13]Mishchenko, K. (2023). Regularized Newton Method with Global \( O( 1/k^2)\) Convergence. SIAM J. Optim., 33(3), 1440–1462. https://doi.org/10.1137/22M1488752
  13. [14]Doikov, N., & Nesterov, Y. (2024). Gradient regularization of Newton method with Bregman distances. Math. Program., 204, 1–25. https://doi.org/10.1007/s10107-023-01943-7
  14. [15]Calandra, H., Gratton, S., Riccietti, E., & Vasseur, X. (2021). On High-Order Multilevel Optimization Strategies. SIAM J. Optim., 31(1), 307–330. https://doi.org/10.1137/19M1255355
  15. [16]Ang, A., De Sterck, H., & Vavasis, S. (2024). MGProx: A Nonsmooth Multigrid Proximal Gradient Method with Adaptive Restriction for Strongly Convex Optimization. SIAM J. Optim., 34(3), 2788–2820. https://doi.org/10.1137/23M1552140
  16. [17]Kočvara, M., & Mohammed, S. (2016). A First-Order Multigrid Method for Bound-Constrained Convex Optimization. Optim. Methods Softw., 31(3), 622–644. https://doi.org/10.1080/10556788.2016.1146267
  17. [18]Cartis, C., Gould, N. I. M., & Toint, P. L. (2011). Adaptive Cubic Regularisation Methods for Unconstrained Optimization. Part I: Motivation, Convergence and Numerical Results. Math. Program., 127(2), 245–295. https://doi.org/10.1007/s10107-009-0286-5
  18. [19]Tsipinakis, N., Parpas, P., & Voigt, M. (2025). Adaptive Multilevel Newton: A Quadratically Convergent Optimization Method (Numbers arXiv:2510.24967). https://doi.org/10.48550/arXiv.2510.24967
  19. [20]Nesterov, Y., & Polyak, B. T. (2006). Cubic regularization of Newton method and its global performance. Math. Program., 108(1), 177–205. https://doi.org/10.1007/s10107-006-0706-8
  20. [21]Cartis, C., Gould, N. I. M., & Toint, P. L. (2011). Adaptive cubic regularisation methods for unconstrained optimization. Part II: Worst-case function- and derivative-evaluation complexity. Math. Program., 130(2), 295–319. https://doi.org/10.1007/s10107-009-0337-y
  21. [22]Griewank, A. (1981). The modification of Newton's method for unconstrained optimization by bounding cubic terms. In Department of Applied Mathematics and Theoretical Physics, University of Cambridge (Number NA/12).
  22. [23]Polyak, R. A. (2009). Regularized Newton Method for Unconstrained Convex Optimization. Math. Program., 120(1), 125–145. https://doi.org/10.1007/s10107-007-0143-3
  23. [24]Cartis, C., Gould, N. I. M., & Toint, P. L. (2012). Evaluation complexity of adaptive cubic regularization methods for convex unconstrained optimization. Optim. Methods Softw., 27(2), 197–219. https://doi.org/10.1080/10556788.2011.602076
  24. [25]Hovhannisyan, V., Parpas, P., & Zafeiriou, S. (2016). MAGMA: Multilevel Accelerated Gradient Mirror Descent Algorithm for Large-Scale Convex Composite Minimization. SIAM J. Imaging Sci., 9(4), 1829–1857. https://doi.org/10.1137/15M104013X
  25. [26]Briggs, W. L., Henson, V. E., & McCormick, S. F. (2000). A Multigrid Tutorial. SIAM. https://doi.org/10.1137/1.9780898719505
  26. [27]Gould, N. I. M., Lucidi, S., Roma, M., & Toint, P. L. (1999). Solving the Trust-Region Subproblem Using the Lanczos Method. SIAM J. Optim., 9(2), 504–525. https://doi.org/10.1137/S1052623497322735
  27. [28]Binev, P., Cohen, A., Dahmen, W., DeVore, R., Petrova, G., & Wojtaszczyk, P. (2011). Convergence Rates for Greedy Algorithms in Reduced Basis Methods. SIAM J. Math. Anal., 43(3), 1457–1472. https://doi.org/10.1137/100795772
  28. [29]Lewis, R. M., & Nash, S. G. (2005). Model Problems for the Multigrid Optimization of Systems Governed by Differential Equations. SIAM J. Sci. Comput., 26(6), 1811–1837. https://doi.org/10.1137/S1064827502407792
  29. [30]Branicky, M. S. (1998). Multiple Lyapunov Functions and Other Analysis Tools for Switched and Hybrid Systems. IEEE Trans. Automat. Control, 43(4), 475–482. https://doi.org/10.1109/9.664150
  30. [31]Zhao, X., Shi, P., Yin, Y., & Nguang, S. K. (2017). New Results on Stability of Slowly Switched Systems: A Multiple Discontinuous Lyapunov Function Approach. IEEE Trans. Automat. Control, 62(7), 3502–3509. https://doi.org/10.1109/TAC.2016.2614911

Paper details

vlnc:W_bzvcc9f2
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

AI tools

View statement
Looking for related papers…
Contents