Part I
Numerical linear algebra for PDEs, machine learning, and data assimilation
Lecture notes
Preface
These notes form the first part of a longer course on advanced numerical linear algebra, taught at master’s level. Their purpose is not only to present the classical algorithms, but to show why the subject has become considerably more central than it was a generation ago.
Numerical linear algebra grew up alongside the numerical solution of partial differential equations, and for a long time that was where its large sparse systems came from. This is no longer the whole story. Ranking the nodes of a network, assimilating observations into a weather forecast, and fitting a model to a large noisy data set all lead to problems of exactly the same kind: too large to factorise, structured, and accessible only through matrix–vector products. Strikingly few ideas are needed for all of them. Sparsity, the spectrum, Krylov subspaces and preconditioning recur throughout, and a method designed for a discretised Laplacian is usually the right method for a graph Laplacian or a design matrix as well.
The chapters that follow are organised around that observation. Each develops a classical topic and then puts it to work outside its original setting: sparse matrices from finite differences, graphs and machine learning; stationary iterations and the smoothing property; conjugate gradient and Lanczos, with spectral clustering and regularisation by early stopping; Arnoldi and GMRES, with PageRank and large least squares; and finally preconditioning, domain decomposition and multigrid.
This first part is deliberately foundational: factorisations and conditioning, iterative methods, Krylov subspaces and preconditioners are what the second part of the course, not reproduced in this volume, takes for granted throughout.
The level is that of a first-year master’s student in mathematics, computer science or a computational discipline. We assume a first course in linear algebra and some familiarity with numerical analysis, and we prove what can be proved compactly, leaving the more technical results to the literature. Every section closes with a summary of what should be retained, and every chapter with exercises, several taken from past examinations.
Reading about an iterative method is, however, a poor substitute for watching one converge. The illustrations in these notes are reproducible, and we encourage readers to run and modify the accompanying Python code, available at https://github.com/vicdolean/scicomp_examples : changing a parameter, a matrix or a stopping criterion is by far the fastest way to develop a feeling for when these methods work and when they do not.
We are grateful to Rafał Wrona (Independent Researcher, Kraków, Poland), whose careful reading and explicit counterexamples led us to correct a number of errors, mainly in Chapter 3.
A note on the use of AI tools. The structure of these notes, the choice and ordering of the topics, and the mathematical content are the authors’ own. Some of the material is standard and presented here in our own way; some of it is drawn from research we ourselves engaged in. In preparing the manuscript we used a generative AI assistant to help identify the seminal references on a given topic, to improve the language, and to improve the graphical content. All statements, proofs and references have been checked by the authors, who take full responsibility for the contents.
Victorita Dolean and Jemima M. Tabeart Eindhoven, October 20, 2026
Solving Linear Systems: A Unified Foundation
Overview
Linear algebra forms the backbone of many modern scientific and engineering computations. From solving systems of equations in structural analysis, optimizing models in machine learning, and forecasting weather through data assimilation, linear algebraic methods are pervasive. However, in practical computations, the mathematical representations of real numbers must confront the limitations of computer arithmetic. Floating-point numbers can only approximate real numbers, leading to small but inevitable rounding errors, and even mathematically exact operations can be extremely sensitive to those tiny errors depending on the structure of A A A .
This chapter lays the numerical foundations used throughout the rest of the course: how to measure the size of vectors, matrices, and errors (norms ); how a problem’s own sensitivity to perturbations is quantified (conditioning ); how floating-point arithmetic and algorithms can amplify or control error (stability ); and the two standard families of solvers for A x = b Ax=b A x = b — direct factorizations (LU, Cholesky) and stationary iterative methods (Jacobi, Gauss–Seidel, SOR, Richardson).
Table 2.
By the end of this chapter, you should be able to:
Define and compute common vector and matrix norms, and explain why all norms are equivalent in finite dimension.
Relate eigenvalues, eigendecompositions, the Schur decomposition, and the SVD to the spectral radius and to matrix norms.
Explain the sources of floating-point error (rounding, truncation, cancellation) and use the condition number κ ( A ) \kappa(A) κ ( A ) to relate backward and forward error.
Compute the L U LU LU and Cholesky factorizations of a matrix, and state why partial pivoting is needed for backward stability.
Derive the Jacobi, Gauss–Seidel, SOR, and Richardson iterations from a matrix splitting A = M − N A=M-N A = M − N , and state the convergence criterion ρ ( G ) < 1 \rho(G)<1 ρ ( G ) < 1 .
Norms and Matrix Decompositions
The first electronic computer, ENIAC (1940s), already faced severe rounding problems. Engineers had to carefully redesign algorithms to avoid divergence even after a few thousand operations.
Vector and Matrix Norms
Definition 1.1.1 (Vector norms). Given a vector x ∈ R n x\in\mathbb{R}^{n} x ∈ R n , the common norms are:
ℓ 1 \ell^{1} ℓ 1 norm: ∥ x ∥ 1 = ∑ i = 1 n ∣ x i ∣ \lVert x\rVert_{1}=\sum_{i=1}^{n}|x_{i}| ∥ x ∥ 1 = ∑ i = 1 n ∣ x i ∣
ℓ 2 \ell^{2} ℓ 2 norm (Euclidean): ∥ x ∥ 2 = ( ∑ i = 1 n ∣ x i ∣ 2 ) 1 / 2 \lVert x\rVert_{2}=\left(\sum_{i=1}^{n}|x_{i}|^{2}\right)^{1/2} ∥ x ∥ 2 = ( ∑ i = 1 n ∣ x i ∣ 2 ) 1/2
ℓ ∞ \ell^{\infty} ℓ ∞ norm: ∥ x ∥ ∞ = max i ∣ x i ∣ \lVert x\rVert_{\infty}=\max_{i}|x_{i}| ∥ x ∥ ∞ = max i ∣ x i ∣
These norms satisfy the following properties:
Non-negativity: ∥ x ∥ ≥ 0 \lVert x\rVert\geq0 ∥ x ∥ ≥ 0 and ∥ x ∥ = 0 \lVert x\rVert=0 ∥ x ∥ = 0 if and only if x = 0 x=0 x = 0
Homogeneity: ∥ α x ∥ = ∣ α ∣ ∥ x ∥ \lVert\alpha x\rVert=|\alpha|\lVert x\rVert ∥ α x ∥ = ∣ α ∣ ∥ x ∥ for any scalar α \alpha α
Triangle inequality: ∥ x + y ∥ ≤ ∥ x ∥ + ∥ y ∥ \lVert x+y\rVert\leq\lVert x\rVert+\lVert y\rVert ∥ x + y ∥ ≤ ∥ x ∥ + ∥ y ∥
Norms give us a way to measure the size (or length) of a vector. They are particularly useful in determining convergence of sequences, measuring distances between vectors, and quantifying approximation errors.
Proposition 1.1.2 (Norm equivalence). All norms on a finite-dimensional vector space are equivalent. This means that for any two norms ∥ ⋅ ∥ a \lVert\cdot\rVert_{a} ∥ ⋅ ∥ a and ∥ ⋅ ∥ b \lVert\cdot\rVert_{b} ∥ ⋅ ∥ b , there exist constants c 1 , c 2 > 0 c_{1},c_{2}>0 c 1 , c 2 > 0 such that:
c 1 ∥ x ∥ a ≤ ∥ x ∥ b ≤ c 2 ∥ x ∥ a for all x ∈ R n . c_{1}\lVert x\rVert_{a}\leq\lVert x\rVert_{b}\leq c_{2}\lVert x\rVert_{a}\qquad\text{for all }x\in\mathbb{R}^{n}. c 1 ∥ x ∥ a ≤ ∥ x ∥ b ≤ c 2 ∥ x ∥ a for all x ∈ R n . Thus, different norms can be used interchangeably when analyzing asymptotic behavior, even though their numerical values may differ. For instance, ∥ x ∥ ∞ ≤ ∥ x ∥ 2 ≤ n ∥ x ∥ ∞ \lVert x\rVert_{\infty}\leq\lVert x\rVert_{2}\leq\sqrt{n}\lVert x\rVert_{\infty} ∥ x ∥ ∞ ≤ ∥ x ∥ 2 ≤ n ∥ x ∥ ∞ and ∥ x ∥ 2 ≤ ∥ x ∥ 1 ≤ n ∥ x ∥ 2 \lVert x\rVert_{2}\leq\lVert x\rVert_{1}\leq\sqrt{n}\lVert x\rVert_{2} ∥ x ∥ 2 ≤ ∥ x ∥ 1 ≤ n ∥ x ∥ 2 , so here c 1 = 1 c_{1}=1 c 1 = 1 and c 2 = n c_{2}=\sqrt{n} c 2 = n (see Problem 2).
Figure 1.1. Unit balls { x : ∥ x ∥ = 1 } \{x : \lVert x\rVert= 1\} { x : ∥ x ∥ = 1 } for the ℓ 1 \ell^1 ℓ 1 , ℓ 2 \ell^2 ℓ 2 , and ℓ ∞ \ell^\infty ℓ ∞ norms in R 2 \mathbb{R}^2 R 2 : the same vector has a different "size" depending on which norm is used.
Eigenvalues and Matrix Decompositions
Before turning to matrix norms, it is useful to recall the key spectral objects that will reappear throughout the course: eigenvalues, and the two canonical matrix factorizations built from them.
Definition 1.1.3 (Eigenvalues, eigendecomposition, spectral radius). A scalar λ ∈ C \lambda\in\mathbb{C} λ ∈ C and nonzero vector x x x satisfying A x = λ x Ax = \lambda x A x = λ x are an eigenvalue and eigenvector of A A A . If A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n is diagonalizable, it admits an eigendecomposition
A = V Λ V − 1 , Λ = diag ( λ 1 , … , λ n ) , A = V\Lambda V^{-1}, \qquad\Lambda= \operatorname{diag}(\lambda_1,\ldots,\lambda_n), A = V Λ V − 1 , Λ = diag ( λ 1 , … , λ n ) , where the columns of V V V are eigenvectors. The spectral radius is ρ ( A ) = max i ∣ λ i ∣ \rho(A) = \max_i |\lambda_i| ρ ( A ) = max i ∣ λ i ∣ .
Definition 1.1.4 (Schur decomposition and SVD). Every A ∈ C n × n A \in\mathbb{C}^{n \times n} A ∈ C n × n admits a Schur decomposition A = Q T Q ∗ A = QTQ^* A = QT Q ∗ with Q Q Q unitary (Q ∗ Q = Q Q ∗ = I Q^*Q = QQ^* = I Q ∗ Q = Q Q ∗ = I ) and T T T upper triangular, with the eigenvalues of A A A on the diagonal of T T T . Every A ∈ R m × n A \in\mathbb{R}^{m \times n} A ∈ R m × n (or C m × n \mathbb{C}^{m \times n} C m × n ) admits a singular value decomposition (SVD) A = U Σ V ∗ A = U\Sigma V^* A = U Σ V ∗ with U , V U,V U , V unitary and Σ = diag ( σ 1 , … ) \Sigma= \operatorname{diag}(\sigma_1,\ldots) Σ = diag ( σ 1 , … ) where σ i ≥ 0 \sigma_i \ge0 σ i ≥ 0 .
Remark 1.1.5. Every symmetric matrix is diagonalizable with real eigenvalues; if it is also symmetric positive definite (SPD), i.e. x T A x ≥ 0 x^\mathsf{T}Ax \ge0 x T A x ≥ 0 with equality only for x = 0 x = 0 x = 0 , its eigenvalues are strictly positive. Unlike the eigendecomposition, the Schur decomposition and the SVD exist for every matrix, which is why they underpin general-purpose numerical algorithms.
Takeaways Eigenvalues and the spectral radius ρ ( A ) \rho(A) ρ ( A ) control the long-run behaviour of iterative processes (§ 2.5 ); the Schur decomposition and the SVD are the two factorizations that always exist and that later chapters (and the induced 2-norm below) build on.
Matrix Norms
Matrix norms are critical for analyzing the stability and conditioning of algorithms and problems involving linear systems, eigenvalue computations, and optimization. Matrix norms are often induced by vector norms.
Definition 1.1.6 (Induced norm (operator norm)). For a matrix A ∈ R m × n A \in\mathbb{R}^{m \times n} A ∈ R m × n the induced norm (operator norm) is defined by
∥ A ∥ = sup x ≠ 0 ∥ A x ∥ ∥ x ∥ = max ∥ x ∥ = 1 ∥ A x ∥ \lVert A\rVert= \sup_{x \ne0} \frac{\lVert Ax\rVert}{\lVert x\rVert} = \max_{\lVert x\rVert=1} \lVert Ax\rVert ∥ A ∥ = x = 0 sup ∥ x ∥ ∥ A x ∥ = ∥ x ∥ = 1 max ∥ A x ∥ The three induced norms used throughout these notes are the following.
Spectral norm (2-norm): ∥ A ∥ 2 = σ max ( A ) \lVert A\rVert_2 = \sigma_{\max}(A) ∥ A ∥ 2 = σ m a x ( A ) , where σ max \sigma_{\max} σ m a x is the largest singular value.
1-norm ∥ A ∥ 1 \lVert A\rVert_1 ∥ A ∥ 1 : maximum absolute column sum
Infinity-norm ∥ A ∥ ∞ \lVert A\rVert_\infty ∥ A ∥ ∞ : maximum absolute row sum
Not every matrix norm is induced by a vector norm. The most important exception is the Frobenius norm ,
∥ A ∥ F = ( ∑ i = 1 m ∑ j = 1 n ∣ a i j ∣ 2 ) 1 / 2 \lVert A\rVert_F = \left(\sum_{i=1}^{m}\sum_{j=1}^{n} \lvert a_{ij}\rvert^2\right)^{1/2} ∥ A ∥ F = ( i = 1 ∑ m j = 1 ∑ n ∣ a ij ∣ 2 ) 1/2
It is exactly the ℓ 2 \ell^2 ℓ 2 norm of the matrix viewed as a vector of its entries.
Proposition 1.1.7 (Note on induced norms). For an induced matrix norm ∥ A ∥ \lVert A\rVert ∥ A ∥ , we have:
This submultiplicative property makes induced norms useful in bounding errors and proving convergence.
Takeaways
On finite-dimensional spaces all norms are equivalent: asymptotically they give the same notion of convergence, but numerically their values can differ significantly.
Floating-Point Arithmetic, Error, Conditioning, and Factorizations
Floating-Point Arithmetic
The Ariane 5 rocket self-destructed (1996) due to a floating-point overflow error when converting a 64-bit number to a 16-bit integer. This incident underscores the risks inherent in floating-point arithmetic and careless type conversions.
Floating-point arithmetic is governed by the IEEE 754 standard [25 ] , which fixes both the set of representable numbers and the way operations on them are rounded.
Definition 1.2.1 (Binary floating-point numbers). Fix a precision p p p and an exponent range [ e min , e max ] [e_{\min}, e_{\max}] [ e m i n , e m a x ] . A normal floating-point number is
x = ( − 1 ) S ( 1. m 1 m 2 ⋯ m p − 1 ) 2 2 e , ( 1. m 1 ⋯ m p − 1 ) 2 = 1 + ∑ k = 1 p − 1 m k 2 − k ∈ [ 1 , 2 ) , x = (-1)^S (1.m_1m_2 \cdots m_{p-1})_2 2^e,\qquad(1.m_1 \cdots m_{p-1})_2 = 1 + \sum_{k=1}^{p-1} m_k 2^{-k} \in[1,2), x = ( − 1 ) S ( 1. m 1 m 2 ⋯ m p − 1 ) 2 2 e , ( 1. m 1 ⋯ m p − 1 ) 2 = 1 + k = 1 ∑ p − 1 m k 2 − k ∈ [ 1 , 2 ) , with sign S ∈ { 0 , 1 } S \in\{0,1\} S ∈ { 0 , 1 } , bits m k ∈ { 0 , 1 } m_k \in\{0,1\} m k ∈ { 0 , 1 } and e min ≤ e ≤ e max e_{\min} \le e \le e_{\max} e m i n ≤ e ≤ e m a x .
The leading 1 is not stored, so p p p bits of precision cost p − 1 p-1 p − 1 bits of storage. The exponent is held as an unsigned E = e + bias E=e+\text{bias} E = e + bias , bias = 2 k − 1 − 1 \text{bias}=2^{k-1}-1 bias = 2 k − 1 − 1 for k k k exponent bits; the two extreme E E E are reserved, one for zero and the subnormals (leading digit 0, filling the gap around zero), the other for ± ∞ \pm\infty ± ∞ and NaN. For instance − 6.5 = − ( 1.101 ) 2 2 -6.5=-(1.101)_2^2 − 6.5 = − ( 1.101 ) 2 2 . S = 1 S=1 S = 1 , m = 1010 … 0 m=1010\ldots0 m = 1010 … 0 , e = 2 e=2 e = 2 .
The first IEEE standard was introduced in 1985. For a long time single (8 exponent bits and 23-bits for the significand) and double precision (11 exponent bits and 52 significand bits) were the most commonly-used formats. However, in recent years there has been a growth in interest in reduced precisions, such as half precision (5 exponent bits and 10 significand bits), non-standard precisions such as bfloat16 (8 exponent bits and 7 significand bits) and proprietary formats like Nvidia’s TensorFloat-32 (8 exponent bits and 10 significand bits). Much of this development has been driven by machine learning applications, and the desire to accelerate performance. In data assimilation, the computational cost is dominated by evaluations of the PDE model, so large gains can be made by applying the model in single precision [20 ] .
One option is to run an entire algorithm in a reduced precision, which will lead to increased computational efficiency, but may result in degraded performance. This has motivated ‘mixed-precision’ approaches, particularly in the numerical linear algebra community, where different steps of an algorithm are run in different precisions. Such approaches aim to combine the computational benefits of reduced precision, while ensuring rapid convergence and computational stability (see e.g. [24 ] [1] for comprehensive reviews).
The Gap Between Neighbouring Floats Floating-point numbers are not spread evenly along the real line. Between 2 e 2^e 2 e and 2 e + 1 2^{e+1} 2 e + 1 , with e e e the exponent of Definition 1.2.1 , they are evenly spaced 2 e − ( p − 1 ) 2^{e-(p-1)} 2 e − ( p − 1 ) apart, since the significand advances one bit at a time. Because ∣ x ∣ \lvert x\rvert ∣ x ∣ lies in that range, the spacing near x x x or its gap , is
gap ( x ) ≈ ∣ x ∣ ⋅ 2 − ( p − 1 ) , \operatorname{gap}(x) \approx\lvert x\rvert\cdot2^{-(p-1)}, gap ( x ) ≈ ∣ x ∣ ⋅ 2 − ( p − 1 ) ,
proportional to the magnitude: doubling x x x doubles it. In double precision the gap just above 1 is 2 − 52 ≈ 2.2 × 10 − 16 2^{-52}\approx2.2\times10^{-16} 2 − 52 ≈ 2.2 × 1 0 − 16 ; near 10 16 10^{16} 1 0 16 it is 2, so the representable numbers there run … , 10 16 , 10 16 + 2 , … \ldots,10^{16},10^{16}+2,\ldots … , 1 0 16 , 1 0 16 + 2 , … with nothing in between, and 10 16 + 1 10^{16}+1 1 0 16 + 1 falls exactly halfway between two neighbours.
Unit Roundoff Rounding to the nearest moves x x x by at most half a gap , that is by at most ∣ x ∣ ⋅ 2 − p \lvert x\rvert\cdot2^{-p} ∣ x ∣ ⋅ 2 − p . Dividing by ∣ x ∣ \lvert x\rvert ∣ x ∣ , the relative error is at most 2 − p 2^{-p} 2 − p regardless of the magnitude of x x x .
Definition 1.2.2 (Unit roundoff and machine precision). For a precision-p p p format, the unit roundoff is u : = 2 − p u:=2^{-p} u := 2 − p and the machine precision is ϵ m a c h : = 2 u = 2 − ( p − 1 ) \epsilon_{\mathrm{mach}}:=2u=2^{-(p-1)} ϵ mach := 2 u = 2 − ( p − 1 ) , the gap between 1 and the next floating-point number. Under round-to-nearest, every real x x x in the normal range and every basic operation ∗ ∈ { + , − , × , / } \ast\in\{+,-,\times,/\} ∗ ∈ { + , − , × , / } satisfy
fl ( x ) = x ( 1 + δ ) , fl ( a ∗ b ) = ( a ∗ b ) ( 1 + δ ) , ∣ δ ∣ ≤ u . \operatorname{fl}(x)=x(1+\delta),\qquad\operatorname{fl}(a\ast b)=(a\ast b)(1+\delta),\qquad\lvert\delta\rvert\leq u. fl ( x ) = x ( 1 + δ ) , fl ( a ∗ b ) = ( a ∗ b ) ( 1 + δ ) , ∣ δ ∣ ≤ u . In double precision u = 2 − 53 ≈ 1.11 × 10 − 16 u=2^{-53}\approx1.11\times10^{-16} u = 2 − 53 ≈ 1.11 × 1 0 − 16 and ϵ m a c h = 2 − 52 ≈ 2.22 × 10 − 16 \epsilon_{\mathrm{mach}}=2^{-52}\approx2.22\times10^{-16} ϵ mach = 2 − 52 ≈ 2.22 × 1 0 − 16 , the value reported by numpy.finfo(float).eps.
Sources of Numerical Errors
Rounding errors: Due to truncating or rounding numbers to fit into finite precision.
Truncation errors: Arise when series are approximated by finite sums.
Cancellation errors: Occur when subtracting two nearly equal numbers, leading to loss of significant digits.
Example: Catastrophic Cancellation Two beams are measured with a ruler marked in centimetres: a = 253 a = 253 a = 253 cm and b = 252 b = 252 b = 252 cm, each correct to within half a centimetre, so each is good to about 0.2%. Their difference is 1 cm, but the true difference could be anything between 0 and 2 cm, a relative error of 100%. Subtracting did not make the measurements worse; it produced an answer so small that the errors they were already carrying are the same size as the answer itself. Floating-point arithmetic behaves the same way, with the unit roundoff u u u in the role of the ruler’s half-centimetre: when two nearly equal numbers are subtracted, the leading digits they agree on cancel and only the uncertain trailing digits remain. Problem 5 shows the standard remedy, rewriting 1 − cos x 1-\cos x 1 − cos x as 2 sin 2 ( x / 2 ) 2\sin^2(x/2) 2 sin 2 ( x /2 ) so that the subtraction never happens.
Takeaways
Rounding is relative : fl ( x ) = x ( 1 + δ ) \operatorname{fl}(x) = x(1+\delta) fl ( x ) = x ( 1 + δ ) with ∣ δ ∣ ≤ u |\delta| \le u ∣ δ ∣ ≤ u , and u = 2 − 53 ≈ 10 − 16 u = 2^{-53} \approx10^{-16} u = 2 − 53 ≈ 1 0 − 16 in double precision.
The gap between neighbouring floats is not constant: it is ≈ ∣ x ∣ ⋅ 2 − 52 \approx|x| \cdot2^{-52} ≈ ∣ x ∣ ⋅ 2 − 52 near x x x .
Cancellation i.e. subtracting two nearly equal numbers, is the most dangerous source of error: it destroys significant digits.
Floating-point addition is not associative, ( x + y ) + z ≠ x + ( y + z ) (x+y)+z \ne x+(y+z) ( x + y ) + z = x + ( y + z ) , so the order of summation matters; see Problem 6.
Lower precisions are faster but more exposed to overflow and rounding.
Further Reading Higham [23 Ch. 2] is the standard reference; Goldberg [11] the readable introduction, Overton [33 ] a short book on IEEE arithmetic, and Muller et al. [30 ] the handbook.
Error, Conditioning and Stability of algorithms
Numerical errors in computations arise from various sources and can significantly affect the reliability of results. To understand and mitigate these effects, we analyze the nature of errors and how sensitive a problem is to perturbations.
During the Gulf War (1991), the Patriot missile system failed to intercept a missile due to a small rounding error in the system’s clock, leading to a timing drift of 0.34 seconds. This small error had devastating consequences, highlighting the importance of careful error analysis.
The Norwegian oil platform Sleipner A collapsed (1991) due to a small error in the finite element analysis stiffness matrix, demonstrating how critical numerical stability and accuracy are in engineering simulations.
Definition 1.2.3 (Forward and backward error). If x x x is the exact solution and x ^ \hat{x} x ^ the computed one:
forward error = ∥ x ^ − x ∥ , backward error = min { ∥ δ A ∥ : ( A + δ A ) x ^ = b } . \text{forward error} = \lVert\hat{x} - x \rVert,\qquad\text{backward error} = \min\{\lVert\delta A\rVert: (A+\delta A)\hat{x}=b\}. forward error = ∥ x ^ − x ∥ , backward error = min {∥ δ A ∥ : ( A + δ A ) x ^ = b } .
Definition 1.2.4 (Condition number). For a nonsingular A A A ,
κ ( A ) = ∥ A ∥ ⋅ ∥ A − 1 ∥ . \kappa(A) = \lVert A\rVert\cdot\lVert A^{-1}\rVert. κ ( A ) = ∥ A ∥ ⋅ ∥ A − 1 ∥ .
Definition 1.2.5 (Condition number in the 2-norm). For a symmetric positive definite A A A ,
κ 2 ( A ) = ∥ A ∥ 2 ⋅ ∥ A − 1 ∥ 2 = λ max ( A ) λ min ( A ) . \kappa_2(A)=\|A\|_2\cdot\|A^{-1}\|_2=\frac{\lambda_{\max}(A)}{\lambda_{\min}(A)}. κ 2 ( A ) = ∥ A ∥ 2 ⋅ ∥ A − 1 ∥ 2 = λ m i n ( A ) λ m a x ( A ) .
Proposition 1.2.6 (Error amplification). A perturbation δ b \delta b δ b yields
∥ δ x ∥ ∥ x ∥ ≤ κ ( A ) ∥ δ b ∥ ∥ b ∥ . \frac{\|\delta x\|}{\|x\|}\leq\kappa(A)\frac{\|\delta b\|}{\|b\|}. ∥ x ∥ ∥ δ x ∥ ≤ κ ( A ) ∥ b ∥ ∥ δ b ∥ .
Example 1.2.7 (Error amplification). Consider
A = [ 1 1 1 1.0001 ] , b = [ 2 2.0001 ] . A=\begin{bmatrix}1 & 1 \\ 1 & 1.0001\end{bmatrix},\qquad b=\begin{bmatrix}2 \\ 2.0001\end{bmatrix}. A = [ 1 1 1 1.0001 ] , b = [ 2 2.0001 ] . This system is very close to being singular, so solving A x = b Ax=b A x = b produces a result that is highly sensitive to small perturbations in A A A or in b b b .
Stable algorithm: Controls the propagation of rounding errors.
Backward stable algorithm: Solves a nearby (perturbed) problem exactly.
Combined view: Stability of the algorithm + conditioning of the problem together determine the final accuracy.
Takeaways
Small κ ( A ) \kappa(A) κ ( A ) (close to 1): well-conditioned, stable predictions.
Large κ ( A ) \kappa(A) κ ( A ) : ill-conditioned, even perfect algorithms may give unreliable results.
An algorithm can be stable (small backward error) yet produce inaccurate results if the problem is ill-conditioned (large condition number).
Conversely, for ill-conditioned problems no algorithm can suppress error amplification.
Gaussian Elimination and LU Factorization
Gaussian elimination, though often attributed to Gauss (1820), was already known in ancient China (Chapter 8 of The Nine Chapters on the Mathematical Art , ∼ \sim ∼ 200 AD). Gauss’ major contribution was to interpret elimination as a matrix factorization. It took him more than twenty years to arrive at that interpretation.
We wish to solve A x = b Ax=b A x = b with A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n , and proceed in three steps.
Apply elimination to zero out entries below the diagonal.
This produces a factorization A = L U A=LU A = LU with L L L unit lower triangular, U U U upper triangular.
Solve L y = b Ly=b L y = b by forward substitution, then U x = y Ux=y U x = y by backward substitution.
Definition 1.2.8 (LU factorization). A matrix A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n admits an LU factorization if it can be written
where L L L is unit lower triangular and U U U is upper triangular.
Example 1.2.9 (Worked 3 × 3 3 \times3 3 × 3 LU factorization, no pivoting). For
A = [ 2 1 1 4 − 6 0 − 2 7 2 ] , A = \begin{bmatrix} 2 & 1 & 1 \\ 4 & -6 & 0 \\ -2 & 7 & 2 \end{bmatrix}, A = 2 4 − 2 1 − 6 7 1 0 2 ,
elimination proceeds as follows:
ℓ 21 = 4 / 2 = 2 , r o w 2 ← r o w 2 − 2 r o w 1 = [ 0 , − 8 , − 2 ] ; \ell_{21} = 4/2 = 2,\qquad\mathrm{row}_2 \leftarrow\mathrm{row}_2 - 2\mathrm{row}_1 = [0,-8,-2]; ℓ 21 = 4/2 = 2 , row 2 ← row 2 − 2 row 1 = [ 0 , − 8 , − 2 ] ;
ℓ 31 = − 2 / 2 = − 1 , r o w 3 ← r o w 3 + r o w 1 = [ 0 , 8 , 3 ] ; \ell_{31} = -2/2 = -1,\qquad\mathrm{row}_3 \leftarrow\mathrm{row}_3 + \mathrm{row}_1 = [0,8,3]; ℓ 31 = − 2/2 = − 1 , row 3 ← row 3 + row 1 = [ 0 , 8 , 3 ] ;
ℓ 32 = 8 / ( − 8 ) = − 1 , r o w 3 ← r o w 3 − ( − 1 ) r o w 2 = [ 0 , 0 , 1 ] , \ell_{32} = 8/(-8) = -1,\qquad\mathrm{row}_3 \leftarrow\mathrm{row}_3 - (-1)\mathrm{row}_2 = [0,0,1], ℓ 32 = 8/ ( − 8 ) = − 1 , row 3 ← row 3 − ( − 1 ) row 2 = [ 0 , 0 , 1 ] ,
the second pivot being U 22 = − 8 U_{22}=-8 U 22 = − 8 . This gives
L = [ 1 0 0 2 1 0 − 1 − 1 1 ] , U = [ 2 1 1 0 − 8 − 2 0 0 1 ] , L = \begin{bmatrix} 1 & 0 & 0 \\ 2 & 1 & 0 \\ -1 & -1 & 1 \end{bmatrix},\qquad U = \begin{bmatrix} 2 & 1 & 1 \\ 0 & -8 & -2 \\ 0 & 0 & 1 \end{bmatrix}, L = 1 2 − 1 0 1 − 1 0 0 1 , U = 2 0 0 1 − 8 0 1 − 2 1 ,
and one checks A = L U A=LU A = LU . To solve A x = b Ax=b A x = b , solve L y = b Ly=b L y = b by forward substitution, then U x = y Ux=y U x = y by backward substitution.
Proposition 1.2.10 (Complexity). The cost of Gaussian elimination (LU factorization) is O ( n 3 ) O(n^3) O ( n 3 ) flops, while forward and backward substitution cost O ( n 2 ) O(n^2) O ( n 2 ) each.
Remark 1.2.11. For large linear systems with n ∼ 10 6 n\sim10^6 n ∼ 1 0 6 , O ( n 3 ) O(n^3) O ( n 3 ) is infeasible. This motivates the use of iterative solvers and preconditioners.
Definition 1.2.12 (Partial pivoting). During elimination, rows are swapped to ensure that pivot elements are large in magnitude. This produces a factorization
P A = L U , P a permutation matrix . PA=LU,\qquad P\text{ a permutation matrix}. P A = LU , P a permutation matrix .
Theorem 1.2.13 (Backward stability of GEPP). Gaussian elimination with partial pivoting (GEPP) is backward stable: the computed x ^ \hat{x} x ^ solves
( A + δ A ) x ^ = b , ∥ δ A ∥ ∥ A ∥ = O ( ε m a c h ) . (A+\delta A)\hat{x}=b,\qquad\frac{\lVert\delta A\rVert}{\lVert A\rVert}=O(\varepsilon_{\mathrm{mach}}). ( A + δ A ) x ^ = b , ∥ A ∥ ∥ δ A ∥ = O ( ε mach ) .
Pitfall. Without pivoting, Gaussian elimination may fail catastrophically even for well-conditioned A A A .
[5] develops bounds on the growth of entries when using full pivoting (not commonly used in practice). The appendix contains a nice overview of computational issues with partial pivoting, including in relation to new hardware such as GPUs.
Gauss used elimination in 1809 to compute planetary orbits from astronomical data. Today, LU is the default “black box” in numerical linear algebra libraries (LAPACK, and hence NumPy/SciPy).
Takeaways
LU factorization provides a direct, stable way to solve A x = b Ax=b A x = b .
Pivoting is essential for stability.
Complexity O ( n 3 ) O(n^3) O ( n 3 ) makes LU unsuitable for very large sparse linear systems.
Iterative solvers (next section) are designed to overcome this scaling barrier.
Cholesky Factorization for SPD Matrices
When A A A is symmetric positive definite, LU factorization can be specialized to exploit the symmetry, halving both storage and computational cost.
Definition 1.2.14 (Cholesky factorization). If A ≻ 0 A \succ0 A ≻ 0 is SPD, then A = L L T A = LL^{\mathsf{T}} A = L L T for a unique lower triangular L L L with positive diagonal entries, computed via
ℓ k k = a k k − ∑ j = 1 k − 1 ℓ k j 2 , ℓ i k = a i k − ∑ j = 1 k − 1 ℓ i j ℓ k j ℓ k k , i > k . \ell_{kk} = \sqrt{a_{kk} - \sum_{j=1}^{k-1} \ell_{kj}^{2}}, \qquad\ell_{ik} = \frac{a_{ik} - \sum_{j=1}^{k-1} \ell_{ij}\ell_{kj}}{\ell_{kk}}, \qquad i > k. ℓ k k = a k k − j = 1 ∑ k − 1 ℓ k j 2 , ℓ ik = ℓ k k a ik − ∑ j = 1 k − 1 ℓ ij ℓ k j , i > k .
Key idea. Cholesky needs no pivoting and is backward stable for SPD matrices, uses half the storage of LU (only L L L is stored), and costs ≈ n 3 / 3 \approx n^{3}/3 ≈ n 3 /3 flops — about half the cost of a general LU factorization.
Takeaways. If A A A is SPD, use Cholesky first: it is the fastest, most memory-efficient, backward-stable direct solver available. If some ℓ k k ≤ 0 \ell_{kk} \le0 ℓ k k ≤ 0 is encountered during the factorization, A A A is not SPD (or is poorly scaled); switch to L D L T LDL^{\mathsf{T}} L D L T with symmetric pivoting, or regularize via A ← A + λ I A \leftarrow A + \lambda I A ← A + λ I .
Stationary Iterative Methods
Direct methods (Gaussian elimination / LU with pivoting) are robust and backward stable, but they become impractical at scale.
Cost. Factorization costs O ( n 3 ) O(n^{3}) O ( n 3 ) flops; each triangular solve costs O ( n 2 ) O(n^{2}) O ( n 2 ) . For sparse banded matrices with half-bandwidth w w w , the costs are O ( n w 2 ) O(nw^{2}) O ( n w 2 ) (factor) and O ( n w ) O(nw) O ( n w ) (solve).
Memory. Fill-in during elimination destroys sparsity; storage can grow from O ( nnz ( A ) ) O(\operatorname{nnz}(A)) O ( nnz ( A )) 1 to orders of magnitude larger.
Key idea. Iterative methods trade exactness for scalability: they approximate the solution via many cheap updates . One step costs O ( nnz ( A ) ) O(\operatorname{nnz}(A)) O ( nnz ( A )) , preserves sparsity, and is parallel-friendly.
In the early 19th century, Gauss and Jacobi independently realized that, instead of eliminating variables, one can successively refine a guess. Jacobi’s 1845 paper on planetary orbits is often cited as the first systematic study of iterative solvers.
Definition 1.3.1 (Splitting method). Let A = M − N A = M - N A = M − N with M M M invertible. Define the stationary iteration
x ( k + 1 ) = M − 1 N x ( k ) + M − 1 b , G : = M − 1 N , c : = M − 1 b . x^{(k+1)} = M^{-1}N x^{(k)} + M^{-1}b, \qquad G := M^{-1}N, \qquad c := M^{-1}b. x ( k + 1 ) = M − 1 N x ( k ) + M − 1 b , G := M − 1 N , c := M − 1 b . If it converges, the limit x ⋆ x^{\star} x ⋆ satisfies ( I − G ) x ⋆ = c (I-G)x^{\star} = c ( I − G ) x ⋆ = c and hence A x ⋆ = b Ax^{\star} = b A x ⋆ = b .
Theorem 1.3.2 (Convergence criterion). The iteration converges for every initial guess x ( 0 ) x^{(0)} x ( 0 ) if and only if ρ ( G ) < 1 \rho(G) < 1 ρ ( G ) < 1 . Moreover, x ( k ) → x ⋆ = ( I − G ) − 1 c x^{(k)} \to x^{\star} = (I-G)^{-1}c x ( k ) → x ⋆ = ( I − G ) − 1 c and the solution is unique.
Sketch. Let e ( k ) : = x ( k ) − x ⋆ e^{(k)} := x^{(k)} - x^\star e ( k ) := x ( k ) − x ⋆ . Subtract the fixed-point relation ( I − G ) x ⋆ = c (I-G)x^\star= c ( I − G ) x ⋆ = c from the iteration to obtain e ( k + 1 ) = G e ( k ) e^{(k+1)} = G e^{(k)} e ( k + 1 ) = G e ( k ) , hence e ( k ) = G k e ( 0 ) e^{(k)} = G^k e^{(0)} e ( k ) = G k e ( 0 ) .
( ⇒ ) (\Rightarrow) ( ⇒ ) If ρ ( G ) ≥ 1 \rho(G) \ge1 ρ ( G ) ≥ 1 , there is an eigenpair ( λ , v ) (\lambda,v) ( λ , v ) with ∣ λ ∣ ≥ 1 |\lambda| \ge1 ∣ λ ∣ ≥ 1 ; choose e ( 0 ) = v e^{(0)} = v e ( 0 ) = v to get e ( k ) = λ k v e^{(k)} = \lambda^k v e ( k ) = λ k v , which does not tend to 0 0 0 . Thus convergence for all x ( 0 ) x^{(0)} x ( 0 ) implies ρ ( G ) < 1 \rho(G) < 1 ρ ( G ) < 1 .
( ⇐ ) (\Leftarrow) ( ⇐ ) If ρ ( G ) < 1 \rho(G) < 1 ρ ( G ) < 1 , then for any ε > 0 \varepsilon> 0 ε > 0 there exists a (subordinate) norm ∥ ⋅ ∥ ε \|\cdot\|_\varepsilon ∥ ⋅ ∥ ε with ∥ G ∥ ε ≤ ρ ( G ) + ε = : q < 1 \|G\|_\varepsilon\le\rho(G) + \varepsilon=: q < 1 ∥ G ∥ ε ≤ ρ ( G ) + ε =: q < 1 .2 Therefore ∥ e ( k ) ∥ ε ≤ ∥ G ∥ ε k ∥ e ( 0 ) ∥ ε ≤ q k ∥ e ( 0 ) ∥ ε → 0 \|e^{(k)}\|_\varepsilon\le\|G\|_\varepsilon^k\|e^{(0)}\|_\varepsilon\le q^k\|e^{(0)}\|_\varepsilon\to0 ∥ e ( k ) ∥ ε ≤ ∥ G ∥ ε k ∥ e ( 0 ) ∥ ε ≤ q k ∥ e ( 0 ) ∥ ε → 0 . Equivalently, the Neumann series ∑ j = 0 ∞ G j \sum_{j=0}^{\infty}G^j ∑ j = 0 ∞ G j converges3 and ( I − G ) − 1 = ∑ j = 0 ∞ G j (I-G)^{-1} = \sum_{j=0}^{\infty}G^j ( I − G ) − 1 = ∑ j = 0 ∞ G j , hence x ⋆ = ( I − G ) − 1 c x^\star= (I-G)^{-1}c x ⋆ = ( I − G ) − 1 c .
Corollary 1.3.3 (A practical sufficient condition). If there exists any subordinate matrix norm with ∥ G ∥ < 1 \|G\| < 1 ∥ G ∥ < 1 , then x ( k ) → x ⋆ x^{(k)} \to x^\star x ( k ) → x ⋆ and
∥ e ( k ) ∥ ≤ ∥ G ∥ k ∥ e ( 0 ) ∥ , ∥ x ⋆ − x ( k ) ∥ ≤ ∥ G ∥ k 1 − ∥ G ∥ ∥ x ( 1 ) − x ( 0 ) ∥ . \|e^{(k)}\| \le\|G\|^k\|e^{(0)}\|,\qquad\|x^\star-x^{(k)}\| \le\frac{\|G\|^k}{1-\|G\|}\|x^{(1)}-x^{(0)}\|. ∥ e ( k ) ∥ ≤ ∥ G ∥ k ∥ e ( 0 ) ∥ , ∥ x ⋆ − x ( k ) ∥ ≤ 1 − ∥ G ∥ ∥ G ∥ k ∥ x ( 1 ) − x ( 0 ) ∥.
Takeaways
To design an effective stationary method:
Choose M M M easy to invert (triangular / block-diagonal / diagonal) so each step is cheap.
Ensure ρ ( M − 1 N ) < 1 \rho(M^{-1}N) < 1 ρ ( M − 1 N ) < 1 (often via diagonal dominance or SPD structure).
Prefer splittings that keep ∥ G ∥ \|G\| ∥ G ∥ comfortably below 1 to avoid roundoff-limited plateaus.
Examples of Stationary Methods and their convergence
Throughout, write
A = D − L − U , A = D - L - U, A = D − L − U ,
with D D D diagonal, − L -L − L the strictly lower and − U -U − U the strictly upper triangular part of A A A (so L L L and U U U carry the minus signs of A A A and have nonnegative entries when A A A has nonpositive off-diagonal entries, as in all our model problems). This convention is used in every chapter of these notes and in the accompanying code; a stray sign here is the single most common source of confusion in this subject.
Jacobi: M = D M = D M = D , G J = D − 1 ( L + U ) G_J = D^{-1}(L+U) G J = D − 1 ( L + U ) . Each component is updated using only values from the previous iterate.
Gauss–Seidel: M = D − L M = D-L M = D − L , G G S = ( D − L ) − 1 U G_{GS} = (D-L)^{-1}U G GS = ( D − L ) − 1 U . New values are used immediately within the same sweep, leading to faster convergence.
Successive Over-Relaxation (SOR):
M = 1 ω D − L , N = 1 − ω ω D + U , G ω = ( 1 ω D − L ) − 1 ( 1 − ω ω D + U ) = ( D − ω L ) − 1 ( ( 1 − ω ) D + ω U ) . M = \frac{1}{\omega}D-L,\qquad N = \frac{1-\omega}{\omega}D+U,\qquad G_\omega= \left(\frac{1}{\omega}D-L\right)^{-1}\left(\frac{1-\omega}{\omega}D+U\right) = (D-\omega L)^{-1}\left((1-\omega)D+\omega U\right). M = ω 1 D − L , N = ω 1 − ω D + U , G ω = ( ω 1 D − L ) − 1 ( ω 1 − ω D + U ) = ( D − ω L ) − 1 ( ( 1 − ω ) D + ω U ) .
Here ω > 0 \omega> 0 ω > 0 is a relaxation parameter; Kahan’s theorem below shows that convergence forces 0 < ω < 2 0 < \omega< 2 0 < ω < 2 . For ω = 1 \omega= 1 ω = 1 this gives M = D − L M = D-L M = D − L and G 1 = ( D − L ) − 1 U G_1 = (D-L)^{-1}U G 1 = ( D − L ) − 1 U : we recover Gauss–Seidel, as we must.
Remark 1.3.4. While Jacobi updates all components simultaneously from the old iterate, Gauss–Seidel reuses fresh values within the same sweep. SOR goes one step further: each update is extrapolated by ω \omega ω , effectively accelerating convergence if ω \omega ω is chosen well.
Lewis Fry Richardson also proposed the first numerical weather forecasting approach, via the solution of differential equations. His paper was published in 1922 and is the foundation of modern numerical weather prediction. He attempted to perform computations by hand while serving with a Quaker ambulance unit on the Western Front in 1917 (his results were not good, but the method was later shown to be sound https://www.americanscientist.org/article/the-weatherman).
The SOR method was proposed by David M. Young in 1950 while working on elliptic PDEs as a PhD student at Harvard. His method was so effective that it immediately became the standard iterative scheme of its era, especially in engineering applications.
Theorem 1.3.5 (Convergence under diagonal dominance). If A A A is strictly diagonally dominant by rows, then both Jacobi and Gauss–Seidel converge.
Sketch. With A = D − L − U A = D - L - U A = D − L − U as above, the Jacobi iteration matrix is G J = D − 1 ( L + U ) G_J = D^{-1}(L + U) G J = D − 1 ( L + U ) . For the ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ norm,
∥ G J ∥ ∞ = max i ∑ j ≠ i ∣ a i j ∣ ∣ a i i ∣ = max i ∑ j ≠ i ∣ a i j ∣ ∣ a i i ∣ . \lVert G_J \rVert_\infty= \max_i \sum_{j \ne i} \frac{\lvert a_{ij}\rvert}{\lvert a_{ii}\rvert} = \max_i \frac{\sum_{j \ne i}\lvert a_{ij}\rvert}{\lvert a_{ii}\rvert}. ∥ G J ∥ ∞ = i max j = i ∑ ∣ a ii ∣ ∣ a ij ∣ = i max ∣ a ii ∣ ∑ j = i ∣ a ij ∣ .
Strict row diagonal dominance (SRDD) means ∣ a i i ∣ > ∑ j ≠ i ∣ a i j ∣ \lvert a_{ii}\rvert> \sum_{j \ne i}\lvert a_{ij}\rvert ∣ a ii ∣ > ∑ j = i ∣ a ij ∣ for all i i i , hence ∥ G J ∥ ∞ < 1 \lVert G_J\rVert_\infty< 1 ∥ G J ∥ ∞ < 1 . Therefore ρ ( G J ) ≤ ∥ G J ∥ ∞ < 1 \rho(G_J) \le\lVert G_J\rVert_\infty< 1 ρ ( G J ) ≤ ∥ G J ∥ ∞ < 1 , and Jacobi converges by the fixed-point criterion. For Gauss–Seidel similar arguments can be employed. An idea would be to use a weighted ∞ \infty ∞ -norm ∥ x ∥ ∞ , w = max i ∣ x i ∣ / w i \lVert x\rVert_{\infty,w} = \max_i \lvert x_i\rvert/w_i ∥ x ∥ ∞ , w = max i ∣ x i ∣ / w i . SRDD implies the existence of w > 0 w > 0 w > 0 with ∥ G G S ∥ ∞ , w < 1. □ \lVert G_{GS}\rVert_{\infty,w} < 1. \square ∥ G GS ∥ ∞ , w < 1. □
Theorem 1.3.6 (Gauss–Seidel for SPD matrices, Theorem 11.2.3 [13]). If A A A is symmetric positive definite, then Gauss–Seidel converges.
Remark 1.3.7. Jacobi and GS are not comparable in general: examples exist where one converges and the other does not.
Theorem 1.3.8 (Convergence SOR (Kahan)). For any A A A and ω ∈ R \omega\in\mathbb{R} ω ∈ R ,
ρ ( G ω ) ≥ ∣ ω − 1 ∣ . \rho(G_\omega) \ge\lvert\omega- 1\rvert. ρ ( G ω ) ≥ ∣ ω − 1 ∣ . Hence a necessary condition for convergence is 0 < ω < 2 0 < \omega< 2 0 < ω < 2 .
Proof. See [3, Theorem 6.32] □ \square □
Theorem 1.3.9 (Convergence SOR (Ostrowski [32 ] ). Let A A A be an SPD matrix, so that in our convention A = D − L − L T A = D - L - L^{\mathsf{T}} A = D − L − L T with D D D diagonal and positive. Then SOR converges for all 0 < ω < 2 0 < \omega< 2 0 < ω < 2 .
Theorem 1.3.10 (Convergence rate SOR (Young [47 ] )). Assume that A A A has property A and is consistently ordered (as the 1D Poisson matrix is, in the natural ordering), that the Jacobi eigenvalues μ ( G J ) \mu(G_J) μ ( G J ) are real, and that ρ ( G J ) < 1 \rho(G_J)<1 ρ ( G J ) < 1 . Then the optimal relaxation parameter in the SOR method is
ω ⋆ = 2 1 + 1 − ρ ( G J ) 2 \omega^\star= \frac{2}{1+\sqrt{1-\rho(G_J)^2}} ω ⋆ = 1 + 1 − ρ ( G J ) 2 2 ( 0 < ω ⋆ < 2 ) (0<\omega^\star<2) ( 0 < ω ⋆ < 2 ) , and the optimized SOR factor is
ρ ( G ω ⋆ ) = ( ρ ( G J ) 1 + 1 − ρ ( G J ) 2 ) 2 . \rho(G_{\omega^\star})=\left(\frac{\rho(G_J)}{1+\sqrt{1-\rho(G_J)^2}}\right)^2. ρ ( G ω ⋆ ) = ( 1 + 1 − ρ ( G J ) 2 ρ ( G J ) ) 2 .
Theorem 1.3.11 (Convergence of Richardson’s method for SPD matrices [46 ] ). Let A A A a SPD matrix. Then:
Convergence iff 0 < α < 2 ρ ( A ) = 2 λ max ( A ) 0<\alpha<\frac{2}{\rho(A)}=\frac{2}{\lambda_{\max}(A)} 0 < α < ρ ( A ) 2 = λ m a x ( A ) 2 .
The optimal step is α ⋆ = 2 λ max ( A ) + λ min ( A ) \alpha^\star=\frac{2}{\lambda_{\max}(A)+\lambda_{\min}(A)} α ⋆ = λ m a x ( A ) + λ m i n ( A ) 2 .
The optimized factor is ρ ( I − α ⋆ A ) = κ ( A ) − 1 κ ( A ) + 1 \rho(I-\alpha^\star A)=\frac{\kappa(A)-1}{\kappa(A)+1} ρ ( I − α ⋆ A ) = κ ( A ) + 1 κ ( A ) − 1 with κ ( A ) = λ max / λ min \kappa(A)=\lambda_{\max}/\lambda_{\min} κ ( A ) = λ m a x / λ m i n .
Example 1.3.12 (Diagonally dominant system). The tridiagonal matrix
A = [ 4 − 1 0 ⋯ − 1 4 − 1 ⋱ 0 − 1 4 ⋱ ⋮ ⋱ ⋱ ⋱ ] A=\begin{bmatrix}
4 & -1 & 0 & \cdots\\
-1 & 4 & -1 & \ddots\\
0 & -1 & 4 & \ddots\\
\vdots& \ddots& \ddots& \ddots
\end{bmatrix} A = 4 − 1 0 ⋮ − 1 4 − 1 ⋱ 0 − 1 4 ⋱ ⋯ ⋱ ⋱ ⋱
is strictly diagonally dominant. Jacobi and Gauss–Seidel are guaranteed to converge.
Example 1.3.13 (Ill-behaved system). The Hilbert matrix H i j = 1 / ( i + j − 1 ) H_{ij}=1/(i+j-1) H ij = 1/ ( i + j − 1 ) is symmetric positive definite, but extremely ill-conditioned. Gauss–Seidel converges in theory, but in finite precision it stagnates quickly due to roundoff.
It is now widely recognised that Jacobi and Gauss–Seidel should not be judged solely by their speed as solvers. Their true importance lies in their role as building blocks for the preconditioned methods widely used in modern scientific computing.
Takeaways
Existence of a good splitting and ρ ( G ) < 1 \rho(G)<1 ρ ( G ) < 1 are the universal convergence keys.
Jacobi/GS: guaranteed under SRDD; GS also for all SPD A A A .
SOR: must take 0 < ω < 2 0<\omega<2 0 < ω < 2 ; for SPD, convergence holds for every such ω \omega ω ; if Jacobi’s ρ ( G J ) \rho(G_J) ρ ( G J ) is known/estimated, Young’s formula gives a near-optimal ω \omega ω .
Richardson: simple baseline; optimal α ⋆ \alpha^\star α ⋆ gives factor ( κ − 1 ) / ( κ + 1 ) (\kappa-1)/(\kappa+1) ( κ − 1 ) / ( κ + 1 ) .
Though obsolete as standalone solvers for large systems, Jacobi, GS, and SOR remain indispensable as smoothers in multigrid and as preconditioners in Krylov methods.
Exercises
Problem 1 (Condition number comparison). Compute the condition number of
Discuss what these condition numbers tell you.
Companion notebook: week1/script_week1.ipynb
Problem 2 (Vector norm equivalence). Prove that for any vector x ∈ R n x \in\mathbb{R}^n x ∈ R n ,
∥ x ∥ ∞ ≤ ∥ x ∥ 2 ≤ n ∥ x ∥ ∞ , \|x\|_\infty\le\|x\|_2 \le\sqrt{n}\|x\|_\infty, ∥ x ∥ ∞ ≤ ∥ x ∥ 2 ≤ n ∥ x ∥ ∞ ,
in other words that these norms are equivalent.
Problem 3 (Properties of the Frobenius norm, exam 2024/2025). Let A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n and let U , V ∈ R n × n U,V \in\mathbb{R}^{n \times n} U , V ∈ R n × n be unitary (orthogonal) matrices, i.e., U T U = U U T = I U^{\mathsf{T}}U = UU^{\mathsf{T}} = I U T U = U U T = I and V T V = V V T = I V^{\mathsf{T}}V = VV^{\mathsf{T}} = I V T V = V V T = I . The Frobenius norm is
∥ A ∥ F : = ( ∑ i = 1 n ∑ j = 1 n a i j 2 ) 1 / 2 . \|A\|_F := \left(\sum_{i=1}^{n}\sum_{j=1}^{n}a_{ij}^2\right)^{1/2}. ∥ A ∥ F := ( i = 1 ∑ n j = 1 ∑ n a ij 2 ) 1/2 .
The trace of A A A is trace ( A ) = ∑ i = 1 n a i i \operatorname{trace}(A) = \sum_{i=1}^{n}a_{ii} trace ( A ) = ∑ i = 1 n a ii .
(a) Show that the Frobenius norm is a matrix norm, but it is not induced by any vector norm.
(b) Show that ∥ A ∥ F 2 = trace ( A T A ) \|A\|_F^2 = \operatorname{trace}(A^{\mathsf{T}}A) ∥ A ∥ F 2 = trace ( A T A ) .
(c) Show that unitary transformations leave the Frobenius norm unchanged: ∥ U A V ∥ F = ∥ A ∥ F \|UAV\|_F = \|A\|_F ∥ U A V ∥ F = ∥ A ∥ F .
Problem 4 (Induced 2-norm, resit exam 2024/2025).
(a) Let A A A be an n × n n \times n n × n matrix. Define the induced 2-norm as
∥ A ∥ 2 = sup ∥ x ∥ 2 = 1 ∥ A x ∥ 2 . \|A\|_2 = \sup_{\|x\|_2=1}\|Ax\|_2. ∥ A ∥ 2 = ∥ x ∥ 2 = 1 sup ∥ A x ∥ 2 .
Prove that the induced 2-norm satisfies the properties of a matrix norm.
(b) Show that the induced 2-norm of a diagonal matrix is equal to the maximum absolute value of its diagonal entries.
(c) Let A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n and U , V U,V U , V be unitary matrices of appropriate dimensions. Prove that ∥ U A V ∥ 2 = ∥ A ∥ 2 \|UAV\|_2 = \|A\|_2 ∥ U A V ∥ 2 = ∥ A ∥ 2 .
Problem 5 (Catastrophic cancellation). Let a = 1.0000000001 a = 1.0000000001 a = 1.0000000001 , b = 1.0000000 b = 1.0000000 b = 1.0000000 and assume we work in double precision.
Companion notebook: week1/script_week1.ipynb
Problem 6 (Floating-point summation order). Design an experiment to show that the order of operations in floating-point summation matters. Sum a list of numbers in increasing and decreasing order and compare the results.
Companion notebook: week1/script_week1.ipynb
Problem 7 (Effect of conditioning). Solve the systems A x = b Ax=b A x = b for two matrices:
A 1 = [ 1 2 3 4 ] , A 2 = [ 1 1 1 1.0001 ] , b = [ 5 11 ] . A_1=\begin{bmatrix}1&2\\3&4\end{bmatrix},\qquad A_2=\begin{bmatrix}1&1\\1&1.0001\end{bmatrix},\qquad b=\begin{bmatrix}5\\11\end{bmatrix}. A 1 = [ 1 3 2 4 ] , A 2 = [ 1 1 1 1.0001 ] , b = [ 5 11 ] .
Compare the computed solutions and relate the error to the condition number of A A A .
Companion notebook: week1/script_week1.ipynb
Problem 8 (Approximating the spectral radius by an induced norm). For any A ∈ C n × n A\in\mathbb{C}^{n\times n} A ∈ C n × n and any ε > 0 \varepsilon>0 ε > 0 , show there exists a (vector) norm on C n \mathbb{C}^{n} C n such that the induced matrix norm satisfies
ρ ( A ) ≤ ∥ A ∥ ≤ ρ ( A ) + ε . \rho(A)\leq\lVert A\rVert\leq\rho(A)+\varepsilon. ρ ( A ) ≤ ∥ A ∥ ≤ ρ ( A ) + ε .
Problem 9 (Gelfand’s formula for induced norms). For any induced matrix norm and any G ∈ R n × n G\in\mathbb{R}^{n\times n} G ∈ R n × n , prove
lim k → ∞ ∥ G k ∥ 1 / k = ρ ( G ) . \lim_{k\to\infty}\lVert G^k\rVert^{1/k}=\rho(G). k → ∞ lim ∥ G k ∥ 1/ k = ρ ( G ) .
Problem 10 (Neumann series). Let B ∈ C n × n B\in\mathbb{C}^{n\times n} B ∈ C n × n with ρ ( B ) < 1 \rho(B)<1 ρ ( B ) < 1 . Prove that I − B I-B I − B is invertible and
( I − B ) − 1 = ∑ j = 0 ∞ B j . (I-B)^{-1}=\sum_{j=0}^{\infty}B^j. ( I − B ) − 1 = j = 0 ∑ ∞ B j .
Problem 11 (Basic iterative methods, resit exam 2024/2025). Consider the linear system A x = b Ax=b A x = b , where
A = [ 4 − 1 0 − 1 4 − 1 0 − 1 4 ] , b = [ 2 6 2 ] . A=\begin{bmatrix}4&-1&0\\-1&4&-1\\0&-1&4\end{bmatrix},\qquad b=\begin{bmatrix}2\\6\\2\end{bmatrix}. A = 4 − 1 0 − 1 4 − 1 0 − 1 4 , b = 2 6 2 .
(a) Derive the iteration formula for the Jacobi method for solving this system.
(b) Perform two iterations of the Jacobi method starting from x ( 0 ) = [ 0 , 0 , 0 ] T x^{(0)}=[0,0,0]^T x ( 0 ) = [ 0 , 0 , 0 ] T .
(c) Prove that the iteration matrix G G G for the Jacobi method satisfies ρ ( G ) < 1 \rho(G)<1 ρ ( G ) < 1 , ensuring convergence of the method.
Problem 12 (Stationary methods applied to a SOAR correlation matrix). A second-order autoregressive (SOAR) correlation matrix, as used for spatial correlations in data assimilation, is defined in Chapter 2. Implement Jacobi, Gauss–Seidel, and SOR for a SOAR matrix with n = 100 n=100 n = 100 , L = 0.4 L=0.4 L = 0.4 , a = 1 a=1 a = 1 . How do the numerical results connect to the theory?
Companion notebook: week1/script_week1.ipynb
Discretization of 1D and 2D PDEs (and Beyond)
Overview
Chapter 1 built the numerical foundations (norms, conditioning, direct and stationary iterative solvers) for A x = b Ax=b A x = b . In this chapter we will see where the matrix A A A actually comes from, and why it is almost always sparse : the discretization of a PDE by finite differences, the Laplacian of a graph, and the normal equations of a machine learning model all produce matrices whose nonzero pattern is dictated by local structure - a stencil, an edge list, or a feature’s support. This chapter covers: (1) from Taylor expansions to finite-difference formulas and the 1D tridiagonal Poisson system; (2) the 2D Poisson matrix, graph Laplacians, and sparsity in machine learning; (3) a toolbox for working with sparse matrices in practice - storage formats, bandwidth/fill-in, reordering, and the spectral properties that govern solver convergence.
Learning objectives
By the end of this chapter, you should be able to:
Derive finite-difference formulas from Taylor expansions and state their consistency order (truncation error).
Assemble the tridiagonal (1D) and block-tridiagonal/Kronecker (2D) linear systems that arise from discretizing the Poisson equation.
Define the graph Laplacian L = D − A L = D - A L = D − A and relate its sparsity pattern and spectrum to those of PDE and machine-learning matrices.
Explain why sparsity appears in ML (normal equations, Hessians) and data assimilation (localized correlation matrices).
Choose an appropriate sparse storage format (CSR/CSC, banded), and explain how half-bandwidth, profile, and reordering affect fill-in and solver cost.
Relate the spectral properties (eigenvalues, condition number) of a sparse matrix to the convergence of iterative solvers and the need for preconditioning.
Finite difference approximation
In Chapter 1 we discussed the importance of numerical solvers for differential equations. Some differential equations can be solved analytically (e.g. separable partial differential equations, or using the method of characteristics), giving solution in closed form. However, many PDEs of interest for applications cannot be solved analytically. In this case, we solve the problem approximately by discretising the continuous problem, which can then be solved on a computer. Three of the most common approaches include the finite element method, finite volume method, and finite difference method. In this course, we focus on finite difference approximations and consider how the properties of the resulting matrices play a role in the convergence of iterative methods.
Figure 2.1. Zoom-in of u ( x ) u(x) u ( x ) around a point with neighbors x ± h x \pm h x ± h
Long before computers, finite differences were already a powerful idea. Euler used them to tabulate functions, and a century later Richardson boldly extended the same spirit to the whole atmosphere. In his 1922 Weather Prediction by Numerical Process, he replaced derivatives by local difference operators to turn the equations of fluid dynamics into linear systems, attempting a six-hour forecast by hand. The calculation took him weeks and produced unstable and incorrect results, but his dream came back to life again with the advent of parallel computers [46 ] .
Let u : R → R u : \mathbb{R} \to\mathbb{R} u : R → R be smooth and h > 0 h > 0 h > 0 . A Taylor expansion around x x x gives the following result.
u ( x ± h ) = u ( x ) ± h u ′ ( x ) + h 2 2 u ′ ′ ( x ) ± h 3 6 u ( 3 ) ( x ) + h 4 24 u ( 4 ) ( ξ ± ) , u(x \pm h) = u(x) \pm h u'(x) + \frac{h^2}{2}u''(x) \pm\frac{h^3}{6}u^{(3)}(x) + \frac{h^4}{24}u^{(4)}(\xi_{\pm}), u ( x ± h ) = u ( x ) ± h u ′ ( x ) + 2 h 2 u ′′ ( x ) ± 6 h 3 u ( 3 ) ( x ) + 24 h 4 u ( 4 ) ( ξ ± ) ,
for some ξ ± \xi_{\pm} ξ ± between x x x and x ± h x \pm h x ± h . From this we obtain the following.
Definition 2.1.1 (Finite difference operators).
Forward: u ( x + h ) − u ( x ) h = u ′ ( x ) + h 2 u ′ ′ ( x ) + O ( h 2 ) , Backward: u ( x ) − u ( x − h ) h = u ′ ( x ) − h 2 u ′ ′ ( x ) + O ( h 2 ) , Central: u ( x + h ) − u ( x − h ) 2 h = u ′ ( x ) + O ( h 2 ) , Second derivative: u ( x + h ) − 2 u ( x ) + u ( x − h ) h 2 = u ′ ′ ( x ) + O ( h 2 ) . \begin{aligned}
\text{Forward:}\quad&\frac{u(x+h)-u(x)}{h} = u'(x) + \frac{h}{2}u''(x) + O(h^2),\\
\text{Backward:}\quad&\frac{u(x)-u(x-h)}{h} = u'(x) - \frac{h}{2}u''(x) + O(h^2),\\
\text{Central:}\quad&\frac{u(x+h)-u(x-h)}{2h} = u'(x) + O(h^2),\\
\text{Second derivative:}\quad&\frac{u(x+h)-2u(x)+u(x-h)}{h^2} = u''(x) + O(h^2).
\end{aligned} Forward: Backward: Central: Second derivative: h u ( x + h ) − u ( x ) = u ′ ( x ) + 2 h u ′′ ( x ) + O ( h 2 ) , h u ( x ) − u ( x − h ) = u ′ ( x ) − 2 h u ′′ ( x ) + O ( h 2 ) , 2 h u ( x + h ) − u ( x − h ) = u ′ ( x ) + O ( h 2 ) , h 2 u ( x + h ) − 2 u ( x ) + u ( x − h ) = u ′′ ( x ) + O ( h 2 ) .
Definition 2.1.2 (Consistency and truncation error). A finite difference operator L h L_h L h is consistent with a differential operator L L L if ( L h u ) ( x ) = ( L u ) ( x ) + τ h ( x ) (L_hu)(x) = (Lu)(x) + \tau_h(x) ( L h u ) ( x ) = ( Lu ) ( x ) + τ h ( x ) with ∥ τ h ∥ → 0 \|\tau_h\| \to0 ∥ τ h ∥ → 0 as h → 0 h \to0 h → 0 . The term τ h \tau_h τ h is the local truncation error; its order (O ( h ) O(h) O ( h ) , O ( h 2 ) O(h^2) O ( h 2 ) , …) is the accuracy.
Proposition 2.1.3 (Consistency and order of the three-point stencil). Let u ∈ C 4 u \in C^4 u ∈ C 4 in a neighborhood of x ∈ R x \in\mathbb{R} x ∈ R . Then
D x x h u ( x ) : = u ( x + h ) − 2 u ( x ) + u ( x − h ) h 2 = u ′ ′ ( x ) + h 2 12 u ( 4 ) ( ξ ) D_{xx}^h u(x) := \frac{u(x+h)-2u(x)+u(x-h)}{h^2} = u''(x) + \frac{h^2}{12}u^{(4)}(\xi) D xx h u ( x ) := h 2 u ( x + h ) − 2 u ( x ) + u ( x − h ) = u ′′ ( x ) + 12 h 2 u ( 4 ) ( ξ ) for some ξ ∈ ( x − h , x + h ) \xi\in(x-h,x+h) ξ ∈ ( x − h , x + h ) .
In particular, the finite difference operator D x x h u ( x ) D_{xx}^h u(x) D xx h u ( x ) is consistent with u ′ ′ ( x ) u''(x) u ′′ ( x ) and has truncation error O ( h 2 ) O(h^2) O ( h 2 ) as h → 0 h \to0 h → 0 .
Figure 2.2. Left: Function values on discretised mesh. Right: Stencil for central differences in 1D
Proof. By Taylor’s theorem with Lagrange remainder, for some ξ + ∈ ( x , x + h ) \xi_{+} \in(x,x+h) ξ + ∈ ( x , x + h ) and ξ − ∈ ( x − h , x ) \xi_{-} \in(x-h,x) ξ − ∈ ( x − h , x ) ,
u ( x ± h ) = u ( x ) ± h u ′ ( x ) + h 2 2 u ′ ′ ( x ) ± h 3 6 u ( 3 ) ( x ) + h 4 24 u ( 4 ) ( ξ ± ) . u(x \pm h) = u(x) \pm h u'(x) + \frac{h^2}{2}u''(x) \pm\frac{h^3}{6}u^{(3)}(x) + \frac{h^4}{24}u^{(4)}(\xi_{\pm}). u ( x ± h ) = u ( x ) ± h u ′ ( x ) + 2 h 2 u ′′ ( x ) ± 6 h 3 u ( 3 ) ( x ) + 24 h 4 u ( 4 ) ( ξ ± ) .
Adding the + + + and − - − expansions and subtracting 2 u ( x ) 2u(x) 2 u ( x ) yields
u ( x + h ) − 2 u ( x ) + u ( x − h ) = h 2 u ′ ′ ( x ) + h 4 24 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) . u(x+h)-2u(x)+u(x-h) = h^2u''(x) + \frac{h^4}{24}\left(u^{(4)}(\xi_{+})+u^{(4)}(\xi_{-})\right). u ( x + h ) − 2 u ( x ) + u ( x − h ) = h 2 u ′′ ( x ) + 24 h 4 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) .
Dividing by h 2 h^2 h 2 gives
u ( x + h ) − 2 u ( x ) + u ( x − h ) h 2 = u ′ ′ ( x ) + h 2 24 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) . \frac{u(x+h)-2u(x)+u(x-h)}{h^2} = u''(x) + \frac{h^2}{24}\left(u^{(4)}(\xi_{+})+u^{(4)}(\xi_{-})\right). h 2 u ( x + h ) − 2 u ( x ) + u ( x − h ) = u ′′ ( x ) + 24 h 2 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) .
By the intermediate value theorem, there exists ξ ∈ ( x − h , x + h ) \xi\in(x-h,x+h) ξ ∈ ( x − h , x + h ) such that 1 2 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) = u ( 4 ) ( ξ ) \frac{1}{2}\left(u^{(4)}(\xi_{+})+u^{(4)}(\xi_{-})\right)=u^{(4)}(\xi) 2 1 ( u ( 4 ) ( ξ + ) + u ( 4 ) ( ξ − ) ) = u ( 4 ) ( ξ ) , hence
u ( x + h ) − 2 u ( x ) + u ( x − h ) h 2 = u ′ ′ ( x ) + h 2 12 u ( 4 ) ( ξ ) . \frac{u(x+h)-2u(x)+u(x-h)}{h^2} = u''(x) + \frac{h^2}{12}u^{(4)}(\xi). h 2 u ( x + h ) − 2 u ( x ) + u ( x − h ) = u ′′ ( x ) + 12 h 2 u ( 4 ) ( ξ ) .
Therefore
∣ D x x h u ( x ) − u ′ ′ ( x ) ∣ ≤ h 2 12 max ∣ y − x ∣ ≤ h ∣ u ( 4 ) ( y ) ∣ = O ( h 2 ) , \left|D^h_{xx}u(x)-u''(x)\right| \le\frac{h^2}{12}\max_{|y-x|\le h}\left|u^{(4)}(y)\right| = O(h^2), D xx h u ( x ) − u ′′ ( x ) ≤ 12 h 2 ∣ y − x ∣ ≤ h max u ( 4 ) ( y ) = O ( h 2 ) ,
which proves consistency with second-order accuracy. □
Remark 2.1.4. Central differences are second order accurate for u ′ u' u ′ and u ′ ′ u'' u ′′ , while forward/backward are first order for u ′ u' u ′ . Higher order formulas follow by eliminating higher derivatives from longer Taylor stencils.
Consider now the boundary value problem (BVP)
− u ′ ′ ( x ) = f ( x ) for x ∈ ( 0 , 1 ) , u ( 0 ) = α , u ( 1 ) = β . -u''(x)=f(x) \quad\text{for } x \in(0,1), \qquad u(0)=\alpha,\ u(1)=\beta. − u ′′ ( x ) = f ( x ) for x ∈ ( 0 , 1 ) , u ( 0 ) = α , u ( 1 ) = β .
Let x i = i h x_i=ih x i = ih with h = 1 / ( n + 1 ) h=1/(n+1) h = 1/ ( n + 1 ) and unknowns u i ≈ u ( x i ) u_i \approx u(x_i) u i ≈ u ( x i ) for i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Approximating u ′ ′ ( x i ) u''(x_i) u ′′ ( x i ) with the central difference gives
− u i − 1 − 2 u i + u i + 1 h 2 = f ( x i ) , i = 1 , … , n , -\frac{u_{i-1}-2u_i+u_{i+1}}{h^2}=f(x_i), \qquad i=1,\ldots,n, − h 2 u i − 1 − 2 u i + u i + 1 = f ( x i ) , i = 1 , … , n ,
where u 0 = α u_0=\alpha u 0 = α and u n + 1 = β u_{n+1}=\beta u n + 1 = β are known from the Dirichlet boundary conditions.
Proposition 2.1.5 (Tridiagonal linear system). Collecting the n n n interior equations yields
A u = g , A = 1 h 2 tridiag ( − 1 , 2 , − 1 ) ∈ R n × n , A\mathbf{u}=\mathbf{g}, \qquad A=\frac{1}{h^2}\operatorname{tridiag}(-1,2,-1)\in\mathbb{R}^{n\times n}, A u = g , A = h 2 1 tridiag ( − 1 , 2 , − 1 ) ∈ R n × n , with g = ( f ( x i ) ) i = 1 n \mathbf{g}=(f(x_i))_{i=1}^{n} g = ( f ( x i ) ) i = 1 n modified at the endpoints: g 1 ← g 1 + α / h 2 , g n ← g n + β / h 2 g_1\leftarrow g_1+\alpha/h^2,\quad g_n\leftarrow g_n+\beta/h^2 g 1 ← g 1 + α / h 2 , g n ← g n + β / h 2 .
Explicit form of the matrix. For n = 5 n=5 n = 5 , the system has the form
1 h 2 [ 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 ] [ u 1 u 2 u 3 u 4 u 5 ] = [ g 1 g 2 g 3 g 4 g 5 ] . \frac{1}{h^2}
\begin{bmatrix}
2 & -1 & 0 & 0 & 0 \\
-1 & 2 & -1 & 0 & 0 \\
0 & -1 & 2 & -1 & 0 \\
0 & 0 & -1 & 2 & -1 \\
0 & 0 & 0 & -1 & 2
\end{bmatrix}
\begin{bmatrix}
u_1\\
u_2\\
u_3\\
u_4\\
u_5
\end{bmatrix}
=
\begin{bmatrix}
g_1\\
g_2\\
g_3\\
g_4\\
g_5
\end{bmatrix}. h 2 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 u 1 u 2 u 3 u 4 u 5 = g 1 g 2 g 3 g 4 g 5 .
The sparsity pattern corresponds exactly to the stencil [ − 1 , 2 , − 1 ] / h 2 \left[-1,\,2,\,-1\right]/h^2 [ − 1 , 2 , − 1 ] / h 2 .
Takeaways
Finite difference discretisation converts differential operators into sparse matrices whose nonzero pattern is determined by the stencil .
The stencil is therefore the object to look at: it fixes the bandwidth, and with it the cost of every direct method applied to the resulting system.
Sparse Matrices: PDEs, Graphs, and Machine Learning
In 1928, Courant, Friedrichs, and Lewy introduced the finite difference method for PDEs in their seminal work. At the same time, scientists like Kirchhoff (1847) were studying electrical networks using adjacency matrices and Laplacians. Nearly a century later, sparse matrices from both PDEs and graphs lie at the core of modern scientific computing and machine learning.
The 2D Poisson equation: structure, stencils and blocks
Consider the Poisson problem on the unit square
− Δ u ( x , y ) = f ( x , y ) , ( x , y ) ∈ ( 0 , 1 ) 2 , u ∣ ∂ Ω = 0. -\Delta u(x,y) = f(x,y), \qquad(x,y) \in(0,1)^2,\quad u|_{\partial\Omega}=0. − Δ u ( x , y ) = f ( x , y ) , ( x , y ) ∈ ( 0 , 1 ) 2 , u ∣ ∂ Ω = 0.
Let x i = i h x_i=ih x i = ih , y j = j h y_j=jh y j = j h , with h = 1 n + 1 h=\frac{1}{n+1} h = n + 1 1 and interior indices i , j = 1 , … , n i,j=1,\ldots,n i , j = 1 , … , n . Denote the grid unknowns by u i , j ≈ u ( x i , y j ) u_{i,j}\approx u(x_i,y_j) u i , j ≈ u ( x i , y j ) . We discretize the two second derivatives separately:
u x x ( x i , y j ) ≈ u i − 1 , j − 2 u i , j + u i + 1 , j h 2 , u y y ( x i , y j ) ≈ u i , j − 1 − 2 u i , j + u i , j + 1 h 2 . u_{xx}(x_i,y_j) \approx\frac{u_{i-1,j}-2u_{i,j}+u_{i+1,j}}{h^2}, \qquad
u_{yy}(x_i,y_j) \approx\frac{u_{i,j-1}-2u_{i,j}+u_{i,j+1}}{h^2}. u xx ( x i , y j ) ≈ h 2 u i − 1 , j − 2 u i , j + u i + 1 , j , u y y ( x i , y j ) ≈ h 2 u i , j − 1 − 2 u i , j + u i , j + 1 .
Thus
− Δ u ( x i , y j ) ≈ 4 u i , j − u i − 1 , j − u i + 1 , j − u i , j − 1 − u i , j + 1 h 2 . -\Delta u(x_i,y_j) \approx\frac{4u_{i,j}-u_{i-1,j}-u_{i+1,j}-u_{i,j-1}-u_{i,j+1}}{h^2}. − Δ u ( x i , y j ) ≈ h 2 4 u i , j − u i − 1 , j − u i + 1 , j − u i , j − 1 − u i , j + 1 .
Definition 2.2.1 (Kronecker product). For matrices A ∈ R m × n A\in\mathbb{R}^{m\times n} A ∈ R m × n and B ∈ R p × q B\in\mathbb{R}^{p\times q} B ∈ R p × q , their Kronecker product is the block matrix
A ⊗ B = [ a 11 B a 12 B ⋯ a 1 n B a 21 B a 22 B ⋯ a 2 n B ⋮ ⋮ ⋱ ⋮ a m 1 B a m 2 B ⋯ a m n B ] ∈ R m p × n q . A\otimes B=
\begin{bmatrix}
a_{11}B & a_{12}B & \cdots& a_{1n}B\\
a_{21}B & a_{22}B & \cdots& a_{2n}B\\
\vdots& \vdots& \ddots& \vdots\\
a_{m1}B & a_{m2}B & \cdots& a_{mn}B
\end{bmatrix}
\in\mathbb{R}^{mp\times nq}. A ⊗ B = a 11 B a 21 B ⋮ a m 1 B a 12 B a 22 B ⋮ a m 2 B ⋯ ⋯ ⋱ ⋯ a 1 n B a 2 n B ⋮ a mn B ∈ R m p × n q .
From grid to linear system. Collect the n 2 n^2 n 2 interior equations into A 2 D u = g A_{2D}u=g A 2 D u = g . Using row-wise (lexicographic) ordering p = ( j − 1 ) n + i p=(j-1)n+i p = ( j − 1 ) n + i , we obtain the block tridiagonal matrix
A 2 D = 1 h 2 T 2 D , T 2 D = I n ⊗ T + T ⊗ I n , T = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n . A_{2D}=\frac{1}{h^2}T_{2D}, \qquad T_{2D}=I_n\otimes T+T\otimes I_n,\qquad T=\operatorname{tridiag}(-1,\,2,\,-1)\in\mathbb{R}^{n\times n}. A 2 D = h 2 1 T 2 D , T 2 D = I n ⊗ T + T ⊗ I n , T = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n .
Figure 2.3. Left: Indexing of grid. Right: stencil to update u ( i , j ) u(i,j) u ( i , j )
Throughout, T T T and T 2 D T_{2D} T 2 D denote the unscaled 1D and 2D matrices, and A = 1 h 2 T A=\frac{1}{h^2}T A = h 2 1 T , A 2 D = 1 h 2 T 2 D A_{2D}=\frac{1}{h^2}T_{2D} A 2 D = h 2 1 T 2 D the matrices that actually discretise the operator. Eigenvalues of the one are h − 2 h^{-2} h − 2 times those of the other; keeping the two apart avoids a factor h − 2 h^{-2} h − 2 later.
Written blockwise, the same matrix has the structure
A 2 D = 1 h 2 [ B − I 0 ⋯ 0 − I B − I ⋱ ⋮ 0 − I B ⋱ 0 ⋮ ⋱ ⋱ ⋱ − I 0 ⋯ 0 − I B ] A_{2D}=\frac{1}{h^2}\begin{bmatrix}
B & -I & 0 & \cdots& 0 \\
-I & B & -I & \ddots& \vdots\\
0 & -I & B & \ddots& 0 \\
\vdots& \ddots& \ddots& \ddots& -I \\
0 & \cdots& 0 & -I & B
\end{bmatrix} A 2 D = h 2 1 B − I 0 ⋮ 0 − I B − I ⋱ ⋯ 0 − I B ⋱ 0 ⋯ ⋱ ⋱ ⋱ − I 0 ⋮ 0 − I B
Each block B B B is itself tridiagonal with stencil [ − 1 , 4 , − 1 ] [-1,4,-1] [ − 1 , 4 , − 1 ] , and each − I -I − I couples neighboring rows of the grid.
Remark 2.2.2. A 2 D A_{2D} A 2 D is sparse (each row has at most 5 nonzeros), symmetric, and positive definite (SPD) under Dirichlet data.
Boundary conditions. For homogeneous Dirichlet (u = 0 u=0 u = 0 on ∂ Ω \partial\Omega ∂ Ω ) the boundary values are known and do not appear as unknowns. For inhomogeneous Dirichlet (u = g u=g u = g ), add boundary contributions to the right hand side near the boundary; sparsity is preserved. One sided differences handle Neumann or Robin conditions, modifying only the first/last block rows.
Graph Laplacians
Graphs give rise to another fundamental family of sparse matrices. Let G = ( V , E ) G=(V,E) G = ( V , E ) be an undirected graph with ∣ V ∣ = n |V|=n ∣ V ∣ = n vertices and edges E E E . The adjacency matrix A A A is defined by
A i j = { 1 if ( i , j ) ∈ E , 0 otherwise. A_{ij}=
\begin{cases}
1 & \text{if }(i,j)\in E,\\
0 & \text{otherwise.}
\end{cases} A ij = { 1 0 if ( i , j ) ∈ E , otherwise.
The degree matrix D D D is diagonal with entries D i i = deg ( i ) = ∑ j A i j D_{ii}=\deg(i)=\sum_j A_{ij} D ii = deg ( i ) = ∑ j A ij .
The graph Laplacian of G G G is L = D − A L = D - A L = D − A . It is symmetric and sparse, with nnz ( L ) = 2 ∣ E ∣ + # { i : deg ( i ) > 0 } \operatorname{nnz}(L) = 2|E| + \#\{i : \operatorname{deg}(i) > 0\} nnz ( L ) = 2∣ E ∣ + # { i : deg ( i ) > 0 } , which is 2 ∣ E ∣ + n 2|E| + n 2∣ E ∣ + n when no vertex is isolated. It is positive semidefinite because
x T L x = ∑ { i , j } ∈ E ( x i − x j ) 2 ≥ 0 , x^{\mathsf{T}}Lx = \sum_{\{i,j\}\in E}(x_i-x_j)^2 \ge0, x T Lx = { i , j } ∈ E ∑ ( x i − x j ) 2 ≥ 0 ,
so L L L measures how much a signal varies along the edges of G G G .
Example. For the path graph on 5 vertices:
Figure 3.
the adjacency matrix is
A = ( 0 1 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0 ) , D = diag ( 1 , 2 , 2 , 2 , 1 ) , A =
\begin{pmatrix}
0 & 1 & 0 & 0 & 0\\
1 & 0 & 1 & 0 & 0\\
0 & 1 & 0 & 1 & 0\\
0 & 0 & 1 & 0 & 1\\
0 & 0 & 0 & 1 & 0
\end{pmatrix},
\qquad D=\operatorname{diag}(1,2,2,2,1), A = 0 1 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0 , D = diag ( 1 , 2 , 2 , 2 , 1 ) ,
so
L = D − A = ( 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 1 ) . L=D-A=
\begin{pmatrix}
1 & -1 & 0 & 0 & 0\\
-1 & 2 & -1 & 0 & 0\\
0 & -1 & 2 & -1 & 0\\
0 & 0 & -1 & 2 & -1\\
0 & 0 & 0 & -1 & 1
\end{pmatrix}. L = D − A = 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 1 .
Remark 2.2.3. This L L L is a rank-2 perturbation of T = tridiag ( − 1 , 2 , − 1 ) T=\operatorname{tridiag}(-1,2,-1) T = tridiag ( − 1 , 2 , − 1 ) , the unscaled matrix of the 1D finite difference discretisation of − u ′ ′ ( x ) -u^{\prime\prime}(x) − u ′′ ( x ) : only the two corner entries differ.
Topology changes the pattern. The path graph is only one extreme; the cycle C n C_n C n (periodic path) and the star K 1 , n − 1 K_{1,n-1} K 1 , n − 1 (one hub, n − 1 n-1 n − 1 leaves) illustrate how differently L L L can look for the same number of vertices.
Figure 4. Cycle graph C 6 C_6 C 6 and Star graph K 1 , 5 K_{1,5} K 1 , 5
For the cycle C n C_n C n , L L L is a circulant tridiagonal matrix with wrap-around corner entries; its eigenvalues are λ k = 2 − 2 cos ( 2 π k / n ) \lambda_k=2-2\cos(2\pi k/n) λ k = 2 − 2 cos ( 2 π k / n ) (see § 4.5 ). For the star K 1 , n − 1 K_{1,n-1} K 1 , n − 1 , the hub has degree n − 1 n-1 n − 1 and every leaf has degree 1, giving the sparse but highly non-tridiagonal spectrum σ ( L ) = { 0 , 1 , … , 1 , n } \sigma(L)=\{0,1,\ldots,1,n\} σ ( L ) = { 0 , 1 , … , 1 , n } , with 1 1 1 repeated n − 2 n-2 n − 2 times: a single dominant eigenvalue reflects the hub’s central role, in sharp contrast to the path and cycle.
On a path graph, the Laplacian matrix “almost” coincides with the finite difference stencil of the PDE Laplacian.4
Graph Laplacians are put to work in several directions.
Spectral clustering: eigenvectors of L L L reveal community structure in networks.
Graph neural networks: message passing relies on L L L or its normalized variants.
Electrical networks: resistor networks are governed by Kirchhoff’s laws, which are encoded in L L L .
Data assimilation: L L L is used to construct diffusion operators, which are commonly used as spatial covariance matrices in meteorological and oceanographic data assimilation.
Takeaways Graph Laplacians are sparse and symmetric matrices that extend the notion of the Laplace operator from continuous domains to arbitrary discrete networks. Their role as “discrete PDEs on graphs” explains their central place in modern data science and network analysis.
Sparse Matrices in Machine Learning
Machine learning problems generate some of the largest sparse matrices encountered today. The source of sparsity is locality : each data point, feature, or parameter interacts only with a small subset of the whole system. In least squares regression , for instance,
min x ∥ A x − b ∥ 2 2 \min_x \lVert Ax-b\rVert_2^2 x min ∥ A x − b ∥ 2 2
the optimality condition yields the normal equations
A T A x = A T b , A^{\mathsf{T}}Ax=A^{\mathsf{T}}b, A T A x = A T b ,
where A ∈ R m × n A \in\mathbb{R}^{m \times n} A ∈ R m × n is the feature matrix and b ∈ R m b \in\mathbb{R}^{m} b ∈ R m the labels.
If each row of A A A contains only k ≪ n k \ll n k ≪ n nonzeros (each sample depends on k k k features), then nnz ( A ) = O ( m k ) \operatorname{nnz}(A)=O(mk) nnz ( A ) = O ( mk ) and A A A is extremely sparse. The matrix A T A A^{\mathsf{T}}A A T A is SPSD and encodes feature correlations. It is usually denser than A A A , but it still preserves significant sparsity compared to a dense n × n n \times n n × n matrix.
Remark 2.2.4. In text classification, A A A is the document-term matrix : rows correspond to documents, columns to words, and entries record word frequencies. Each document contains only a few words compared to the dictionary size ⇒ A \Rightarrow A ⇒ A is sparse. Then A T A A^{\mathsf{T}}A A T A counts word co-occurrences across documents, producing a sparse graph of word relations.
Figure 5 (PDF p. 27). Sparse design matrix A A A (left) vs. denser normal matrix A T A A^{\mathsf{T}}A A T A (right).
Second-order optimisation methods require the Hessian
H i j = ∂ 2 ∂ x i ∂ x j F ( x ) . H_{ij} = \frac{\partial^2}{\partial x_i \partial x_j} F(x). H ij = ∂ x i ∂ x j ∂ 2 F ( x ) .
When the loss F F F couples only nearby features, H H H inherits sparsity. This occurs frequently in PDE-constrained optimization , and deep learning layers with local connectivity .
For logistic regression with data matrix A A A (columns α k \alpha_k α k ) and labels b k ∈ { − 1 , 1 } b_k \in\{-1,1\} b k ∈ { − 1 , 1 } ,
F ( x ) = ∑ k = 1 m log ( 1 + exp ( − b k α k T x ) ) , F(x) = \sum_{k=1}^{m} \log\left(1+\exp(-b_k \alpha_k^{\mathsf{T}}x)\right), F ( x ) = k = 1 ∑ m log ( 1 + exp ( − b k α k T x ) ) ,
the Hessian takes the form H = A T W A H=A^{\mathsf{T}}WA H = A T W A where W W W is a diagonal matrix of weights depending on x x x (w k = exp ( b k α k T x ) ( 1 + exp ( b k α k T x ) ) 2 w_k = \frac{\exp(b_k\alpha_k^{\mathsf{T}}x)}{(1+\exp(b_k\alpha_k^{\mathsf{T}}x))^2} w k = ( 1 + e x p ( b k α k T x ) ) 2 e x p ( b k α k T x ) ). If A A A is sparse, so is H H H , and computations can be done efficiently.
Takeaways
In machine learning, sparsity comes from locality in data or models. Normal equations, Hessians, and Jacobians inherit this structure. Harnessing it is essential: without sparse techniques, modern large-scale ML training would be infeasible.
Sparsity in data assimilation
In the final part of the course we will see that covariance matrices are important for weighting the contribution of different components to a data assimilation system. The sparsity structure of these covariance matrices has important computational and physical consequences.
In ensemble data assimilation methods, underlying ‘physical’ covariances are obtained using sample estimates. This can lead to spurious long-range correlations which destroy sparsity. In a later part of the course we will consider localisation methods, which are used to prescribe a given local sparsity structure on estimated spatial covariance matrices.
Sometimes we wish to estimate the value of the covariance matrix from measurement data. One method is to use the measurements to fit appropriate parameter values for a standard correlation function (e.g. Gaussian, auto-regressive functions) see e.g. [28 ] . These distributions all have long tails and therefore yield dense matrices. In practice, correlation distributions are often truncated when they fall below a threshold value (typically chosen as 0.2) to ensure sparsity/fixed bandwidth of the resulting matrices.
Sparse Matrix Toolbox
We have now seen where sparse matrices come from: discretised PDEs, graphs, and machine learning. Knowing where they arise is not enough to compute with them. We also need to know how to store them, how to operate on them efficiently, and how their spectral properties affect the algorithms of the following chapters. That is the subject of this section.
Sparsity Patterns and storage formats
The sparsity pattern of a matrix A A A is the set of positions ( i , j ) (i,j) ( i , j ) where a i j ≠ 0 a_{ij} \ne0 a ij = 0 .
Example. For the tridiagonal finite difference Poisson matrix:
nnz ( A ) ≈ 3 n , density ( A ) = nnz ( A ) n 2 ≈ 1 / n . \operatorname{nnz}(A) \approx3n,\qquad\operatorname{density}(A) = \frac{\operatorname{nnz}(A)}{n^2} \approx1/n. nnz ( A ) ≈ 3 n , density ( A ) = n 2 nnz ( A ) ≈ 1/ n .
1D Poisson (tridiagonal). For n n n interior points, A = 1 h 2 tridiag ( − 1 , 2 , − 1 ) A = \frac{1}{h^2}\operatorname{tridiag}(-1,2,-1) A = h 2 1 tridiag ( − 1 , 2 , − 1 ) has nonzeros only on the main diagonal and the first sub-/super-diagonals.
Figure 1 (PDF p. 29). Spy pattern for 1D Poisson, n = 8 n = 8 n = 8 . Half-bandwidth b = 1 b = 1 b = 1 , nnz / row ≈ 3 \operatorname{nnz}/\operatorname{row} \approx3 nnz / row ≈ 3 (except at the ends).
2D Poisson with lexicographic ordering. Let the grid be n × n n \times n n × n (N = n 2 N = n^2 N = n 2 unknowns) and use row-wise (lexicographic) ordering. Then A A A has nonzeros at indices j = i j = i j = i (main), j = i ± 1 j = i \pm1 j = i ± 1 within the same row block, and j = i ± n j = i \pm n j = i ± n (coupling to the row above/below). That is exactly the 5-point stencil.
Figure 2 (PDF p. 29). Spy pattern for 2D Poisson, n = 4 n = 4 n = 4 . Half-bandwidth b = 4 b = 4 b = 4 , nnz / row ≈ 5 \operatorname{nnz}/\operatorname{row} \approx5 nnz / row ≈ 5 (except at end blocks).
Graph Laplacians (path vs. star). Two small graphs with n = 8 n = 8 n = 8 nodes to highlight how topology changes the pattern.
Figure 1 (PDF p. 30). Left: path L ⇒ L \Rightarrow L ⇒ tridiagonal. Right: star L ⇒ L \Rightarrow L ⇒ dense first row/column + diagonal.
Matrix structure and storage. Sparse matrices are not stored entry by entry; only the nonzero pattern is kept. Common formats include:
CSC/CSR (Compressed Sparse Column/Row): general-purpose, store indices and values compactly.
Diagonal/banded storage: efficient when nonzeros cluster near the diagonal.
Example 2.3.1 (CSR and CSC formats). Consider the 4 × 4 4 \times4 4 × 4 sparse matrix
A = [ 1 0 0 2 0 3 0 4 5 6 7 0 0 0 8 9 ] . A = \begin{bmatrix} 1 & 0 & 0 & 2 \\ 0 & 3 & 0 & 4 \\ 5 & 6 & 7 & 0 \\ 0 & 0 & 8 & 9 \end{bmatrix}. A = 1 0 5 0 0 3 6 0 0 0 7 8 2 4 0 9 . CSR (Compressed Sparse Row).
val = [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] , col_ind = [ 1 , 4 , 2 , 4 , 1 , 2 , 3 , 3 , 4 ] , row_ptr = [ 1 , 3 , 5 , 8 , 10 ] . \begin{aligned}
\texttt{val} &= [1, 2, 3, 4, 5, 6, 7, 8, 9],\\
\texttt{col\_ind} &= [1, 4, 2, 4, 1, 2, 3, 3, 4],\\
\texttt{row\_ptr} &= [1, 3, 5, 8, 10].
\end{aligned} val col_ind row_ptr = [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] , = [ 1 , 4 , 2 , 4 , 1 , 2 , 3 , 3 , 4 ] , = [ 1 , 3 , 5 , 8 , 10 ] . CSC (Compressed Sparse Column).
val = [ 1 , 5 , 3 , 6 , 7 , 8 , 2 , 4 , 9 ] , row_ind = [ 1 , 3 , 2 , 3 , 3 , 4 , 1 , 2 , 4 ] , col_ptr = [ 1 , 3 , 5 , 7 , 10 ] . \begin{aligned}
\texttt{val} &= [1, 5, 3, 6, 7, 8, 2, 4, 9],\\
\texttt{row\_ind} &= [1, 3, 2, 3, 3, 4, 1, 2, 4],\\
\texttt{col\_ptr} &= [1, 3, 5, 7, 10].
\end{aligned} val row_ind col_ptr = [ 1 , 5 , 3 , 6 , 7 , 8 , 2 , 4 , 9 ] , = [ 1 , 3 , 2 , 3 , 3 , 4 , 1 , 2 , 4 ] , = [ 1 , 3 , 5 , 7 , 10 ] . Here, indices are 1-based for readability (note that Python/NumPy uses 0-based indexing).
In CSR, row_ptr[i] points to the start of row i i i in val.
In CSC, col_ptr[j] points to the start of column j j j in val.
Bandwidth vs. profile. Two measures quantify “how far” nonzeros lie from the diagonal:
b = max { ∣ i − j ∣ : a i j ≠ 0 } , b = \max\{|i-j| : a_{ij} \ne0\}, b = max { ∣ i − j ∣ : a ij = 0 } ,
the worst-case distance from the diagonal; the band is then 2 b + 1 2b + 1 2 b + 1 diagonals wide. A tridiagonal matrix has b = 1 b = 1 b = 1 . Narrow bands allow cheap banded LU; wide bands inflate fill and storage.
Profile (envelope): for each row i i i , the span ℓ i = i − min { j : a i j ≠ 0 } \ell_i = i - \min\{j : a_{ij} \ne0\} ℓ i = i − min { j : a ij = 0 } to the left of the diagonal; the profile is ∑ i ℓ i \sum_i \ell_i ∑ i ℓ i . This reflects actual memory usage more sensitively than bandwidth alone, since one wide row does not inflate it the way it inflates b b b .
Remark 2.3.2. In the 2D Poisson problem with N = n 2 N = n^2 N = n 2 , each row has only 5 nonzeros, yet the half-bandwidth grows like n n n . Naive banded LU then costs O ( N b 2 ) = O ( N 2 ) O(Nb^2) = O(N^2) O ( N b 2 ) = O ( N 2 ) , whereas good reorderings (e.g. nested dissection) reduce the cost to O ( N 3 / 2 ) O(N^{3/2}) O ( N 3/2 ) .
Fill-in. During LU factorization, zeros in A A A can become nonzeros in L L L or U U U .
For banded matrices, fill grows with bandwidth.
In general, the amount of fill depends strongly on the sparsity pattern.
Figure fill-in. Illustration: fill-in introduces new nonzeros (red).
Remark 2.3.3. By permuting rows/columns before factorization, one can reduce bandwidth and hence fill-in. Effective reordering can save orders of magnitude in memory and runtime—critical for large-scale finite element systems.
The invention of compressed storage and reordering heuristics in the 1970s was pivotal: without them, sparse PDE systems of size 10 5 10^5 1 0 5 could not be solved on mainframes.
The inverse of a sparse matrix is not guaranteed to be sparse, and can suffer from more fill-in than an LU or Cholesky decomposition. One example is the given in Figure 2.4 , where A A A is a lower triangular matrix with ones on the diagonal and first subdiagonal, plus a single entry above the diagonal. The LU factorization of A A A is sparse, with some fill-in seen in U U U (lower right panel). This aligns with the element above the diagonal. On the other hand, A − 1 A^{-1} A − 1 is a dense matrix, with almost all entries being non-zero. In general we aim to avoid using inverse matrices, but in the final section of the course we will see that inverse covariance matrices play a very important role in data assimilation and cannot be completely avoided.
Figure 2.4. Visualisation of fill-in with LU vs inverse. Top left: sparsity pattern original matrix, A A A . Top right: sparsity pattern of A − 1 A^{-1} A − 1 . Lower left: sparsity pattern of L L L . Lower right: sparsity pattern of U U U .
Spectral Properties
For some particular sparsity patterns, we can compute the eigenvalues explicitly:
If a tridiagonal matrix is Toeplitz , i.e. with stencil [ p , d , q ] [p,d,q] [ p , d , q ] (p on the subdiagonal, d on the diagonal, q on the superdiagonal) and p q > 0 pq > 0 pq > 0 , then the eigenvalues can be computed explicitly as:
λ j = d − 2 p q cos ( j π n + 1 ) j = 1 , … , n . \lambda_j = d - 2\sqrt{pq}\cos\left(\frac{j\pi}{n+1}\right) \qquad j = 1,\ldots,n. λ j = d − 2 pq cos ( n + 1 j π ) j = 1 , … , n .
Replacing − - − by + + + gives the same set of eigenvalues, since j ↦ n + 1 − j j \mapsto n+1-j j ↦ n + 1 − j flips the sign of the cosine; both forms occur in the literature. In the symmetric case q = p q = p q = p this reads
λ j = d + 2 p cos ( j π n + 1 ) , \lambda_j = d + 2p\cos\left(\frac{j\pi}{n+1}\right), λ j = d + 2 p cos ( n + 1 j π ) ,
which is the form used in Problem 4. If p q < 0 pq < 0 pq < 0 the eigenvalues are complex and the formula does not apply.
Definition. A circulant matrix is a particular form of Toeplitz matrix which has rows composed of all the same elements, with each row being shifted one element to the right relative to the preceding row, i.e.
C = ( c 0 c 1 ⋯ c n − 2 c n − 1 c n − 1 c 0 c 1 ⋯ c n − 2 ⋮ c n − 1 c 0 ⋱ ⋮ c 2 ⋱ ⋱ ⋱ c 1 c 1 c 2 ⋯ c n − 1 c 0 ) . C = \begin{pmatrix}
c_0 & c_1 & \cdots& c_{n-2} & c_{n-1} \\
c_{n-1} & c_0 & c_1 & \cdots& c_{n-2} \\
\vdots& c_{n-1} & c_0 & \ddots& \vdots\\
c_2 & \ddots& \ddots& \ddots& c_1 \\
c_1 & c_2 & \cdots& c_{n-1} & c_0
\end{pmatrix}. C = c 0 c n − 1 ⋮ c 2 c 1 c 1 c 0 c n − 1 ⋱ c 2 ⋯ c 1 c 0 ⋱ ⋯ c n − 2 ⋯ ⋱ ⋱ c n − 1 c n − 1 c n − 2 ⋮ c 1 c 0 . Circulant matrices are diagonalized by a discrete Fourier transform, i.e. the eigenvalues are given by
λ j = c 0 + c n − 1 ω − j + c n − 2 ω − 2 j + ⋯ + c 1 ω − ( n − 1 ) j , j = 0 , … , n − 1 , \lambda_j = c_0 + c_{n-1}\omega^{-j} + c_{n-2}\omega^{-2j} + \cdots+ c_1\omega^{-(n-1)j}, \qquad j=0,\ldots,n-1, λ j = c 0 + c n − 1 ω − j + c n − 2 ω − 2 j + ⋯ + c 1 ω − ( n − 1 ) j , j = 0 , … , n − 1 , where ω = exp ( 2 π i n ) \omega=\exp\left(\frac{2\pi i}{n}\right) ω = exp ( n 2 π i ) is an n n n -th root of unity. The eigenvectors of C C C are given by
v j = 1 n ( 1 , ω j , ω 2 j , … , ω ( n − 1 ) j ) T , j = 0 , 1 , 2 , … , n − 1. v_j=\frac{1}{\sqrt{n}}(1,\omega^j,\omega^{2j},\ldots,\omega^{(n-1)j})^{T}, \qquad j=0,1,2,\ldots,n-1. v j = n 1 ( 1 , ω j , ω 2 j , … , ω ( n − 1 ) j ) T , j = 0 , 1 , 2 , … , n − 1.
Eigenvalues of model problems. For the 1D Poisson matrix,
λ j = 4 h 2 sin 2 ( j π 2 ( n + 1 ) ) , j = 1 , … , n . \lambda_j=\frac{4}{h^2}\sin^2\left(\frac{j\pi}{2(n+1)}\right), \qquad j=1,\ldots,n. λ j = h 2 4 sin 2 ( 2 ( n + 1 ) j π ) , j = 1 , … , n .
Hence
κ ( A ) = λ max λ min = O ( n 2 ) . \kappa(A) = \frac{\lambda_{\max}}{\lambda_{\min}} = O(n^2). κ ( A ) = λ m i n λ m a x = O ( n 2 ) .
Similarly, consider the matrix of an implicit diffusion step on the unit square with Dirichlet boundary conditions,
A ν = I + ν A 2 D ∈ R n 2 × n 2 , A 2 D = 1 h 2 ( I n ⊗ T + T ⊗ I n ) , T = tridiag ( − 1 , 2 , − 1 ) , A_\nu= I + \nu A_{2D} \in\mathbb{R}^{n^2 \times n^2}, \qquad A_{2D} = \frac{1}{h^2}(I_n \otimes T + T \otimes I_n), \qquad T = \operatorname{tridiag}(-1,2,-1), A ν = I + ν A 2 D ∈ R n 2 × n 2 , A 2 D = h 2 1 ( I n ⊗ T + T ⊗ I n ) , T = tridiag ( − 1 , 2 , − 1 ) ,
where ν > 0 \nu> 0 ν > 0 collects the diffusion coefficient and the time step. (The latter A ν A_\nu A ν keeps this matrix apart from the 1D Poisson matrix A A A of the previous paragraph.) Since A 2 D A_{2D} A 2 D has the eigenvalues of Section 2.2.1, the n 2 n^2 n 2 eigenvalues of A ν A_\nu A ν are
μ i , j = 1 + 4 ν h 2 ( sin 2 ( i 2 ( n + 1 ) π ) + sin 2 ( j 2 ( n + 1 ) π ) ) , i , j = 1 , 2 , … , n . \mu_{i,j} = 1 + \frac{4\nu}{h^2}\left(\sin^2\left(\frac{i}{2(n+1)}\pi\right) + \sin^2\left(\frac{j}{2(n+1)}\pi\right)\right), \qquad i,j = 1,2,\ldots,n. μ i , j = 1 + h 2 4 ν ( sin 2 ( 2 ( n + 1 ) i π ) + sin 2 ( 2 ( n + 1 ) j π ) ) , i , j = 1 , 2 , … , n .
These condition numbers grow as h − 2 h^{-2} h − 2 : iterative solvers (CG, GMRES, etc.) slow down as the mesh refines. Spectral clustering of eigenvalues governs convergence rates: if most eigenvalues lie in a tight interval, Krylov methods converge quickly; wide spreads or outliers cause stagnation. Preconditioners are therefore essential to compress the spectrum.
Spectral viewpoint on sparsity. Even though A A A is sparse, its eigenvalues may cluster near zero, leading to ill-conditioning. Sparsity alone does not guarantee fast convergence: what matters is the distribution of eigenvalues . For example, reordering does not change eigenvalues (it is a similarity transformation), but preconditioning actively reshapes the spectrum to improve solver efficiency.
Graph Laplacians. Eigenvalues encode combinatorial structure:
Machine learning. Normal equations A T A A^\mathsf{T}A A T A have eigenvalues σ i 2 \sigma_i^2 σ i 2 , the squares of the singular values of A A A . Ill-conditioning corresponds to nearly collinear features, where small singular values slow down gradient descent and amplify noise. Regularization (ridge regression, Tikhonov) shifts the spectrum, lifting small eigenvalues away from zero.
Takeaways
Sparsity patterns (bandwidth, profile) determine storage and fill-in.
Compressed storage formats (CSR/CSC) make large problems feasible.
Fill-in can destroy sparsity; reordering strategies mitigate this.
Spectral properties are decisive: eigenvalues dictate conditioning, convergence, and the design of effective preconditioners.
Efficient sparse linear algebra requires both algebraic insight (spectrum, conditioning) and practical storage/reordering techniques.
Exercises
Problem 1 (FD accuracy at a point). For u ( x ) = sin ( π x ) u(x)=\sin(\pi x) u ( x ) = sin ( π x ) , approximate u ′ ′ ( 0.5 ) u''(0.5) u ′′ ( 0.5 ) with the 3-point stencil
D x x h u ( 0.5 ) = u ( 0.5 + h ) − 2 u ( 0.5 ) + u ( 0.5 − h ) h 2 D_{xx}^{h}u(0.5)=\frac{u(0.5+h)-2u(0.5)+u(0.5-h)}{h^2} D xx h u ( 0.5 ) = h 2 u ( 0.5 + h ) − 2 u ( 0.5 ) + u ( 0.5 − h )
using h = 0.25 h=0.25 h = 0.25 , h = 0.125 h=0.125 h = 0.125 and h = 0.0625 h=0.0625 h = 0.0625 . Compare with the exact u ′ ′ ( 0.5 ) u''(0.5) u ′′ ( 0.5 ) , and check that the error is divided by roughly 4 each time h h h is halved.
Companion notebook: week2/script_week2_1.ipynb
Problem 2 (1D Poisson: assemble & solve by hand). Discretize − u ′ ′ = 1 -u''=1 − u ′′ = 1 on ( 0 , 1 ) (0,1) ( 0 , 1 ) with u ( 0 ) = u ( 1 ) = 0 u(0)=u(1)=0 u ( 0 ) = u ( 1 ) = 0 and n = 4 n=4 n = 4 interior points (h = 1 / 5 h=1/5 h = 1/5 ).
Write the linear system A u = g Au=g A u = g .
Solve it explicitly for u ∈ R 4 u\in\mathbb{R}^4 u ∈ R 4 .
Companion notebook: week2/script_week2_1.ipynb
Problem 3 (Discretisation of a BVP and application of a basic iterative method, exam 2024/2025). Consider the one-dimensional boundary value problem
{ − u ′ ′ ( x ) + c u ( x ) = 0 , x ∈ ( 0 , 1 ) , u ( 0 ) = 1 , u ( 1 ) = 0 , \begin{cases}
-u''(x)+cu(x)=0, & x\in(0,1),\\
u(0)=1,\quad u(1)=0,
\end{cases} { − u ′′ ( x ) + c u ( x ) = 0 , u ( 0 ) = 1 , u ( 1 ) = 0 , x ∈ ( 0 , 1 ) ,
where c > 0 c>0 c > 0 is a constant. Use a uniform mesh with grid points x j = j h x_j=jh x j = j h , h = 1 n + 1 h=\frac{1}{n+1} h = n + 1 1 , j = 0 , … , n + 1 j=0,\ldots,n+1 j = 0 , … , n + 1 , and denote u j ≈ u ( x j ) u_j\approx u(x_j) u j ≈ u ( x j ) .
(a) Give the finite difference scheme for the discretisation of the BVP.
(b) Let u = ( u 1 , … , u n ) T u=(u_1,\ldots,u_n)^{\mathsf{T}} u = ( u 1 , … , u n ) T be the unknown vector solving A u = b Au=b A u = b . Determine A A A and b b b .
(c) We solve the linear system by the Jacobi method based on the splitting A = D − E − F A=D-E-F A = D − E − F , where D D D is the diagonal part and E E E (resp. F F F ) the strict lower (resp. upper) triangular part. (This is the splitting A = D − L − U A=D-L-U A = D − L − U of Chapter 1, with E = L E=L E = L and F = U F=U F = U ; the exam statement uses the other pair of letters.) Give the exact Jacobi iteration.
(d) If e k = u − u k e^k=u-u^k e k = u − u k denotes the error at iteration k k k , then e k = G k e 0 e^k=G^ke^0 e k = G k e 0 with iteration matrix G G G . Give G G G explicitly.
Problem 4 (Finite Differences for Boundary Value Problems, resit exam 2024/2025). Consider the boundary value problem:
− u ′ ′ ( x ) + c u ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = α , u ( 1 ) = β , -u''(x)+cu(x)=f(x),\quad x\in(0,1),\quad u(0)=\alpha,\quad u(1)=\beta, − u ′′ ( x ) + c u ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = α , u ( 1 ) = β ,
where c c c is a positive constant, and f ( x ) f(x) f ( x ) is a given source term.
(a) Discretize the equation using the finite difference method with a uniform grid x j = j h x_j=jh x j = j h , j = 0 , 1 , … , n + 1 j=0,1,\ldots,n+1 j = 0 , 1 , … , n + 1 , where h = 1 / ( n + 1 ) h=1/(n+1) h = 1/ ( n + 1 ) . Derive the corresponding difference scheme.
(b) Write the scheme in matrix form. Give the expression of the matrix and right hand side.
(c) Compute the spectral condition number of the matrix A A A . Hint: For a symmetric positive definite matrix we have: κ ( A ) = λ max ( A ) λ min ( A ) \kappa(A) = \frac{\lambda_{\max}(A)}{\lambda_{\min}(A)} κ ( A ) = λ m i n ( A ) λ m a x ( A ) , where λ max ( A ) \lambda_{\max}(A) λ m a x ( A ) and λ min ( A ) \lambda_{\min}(A) λ m i n ( A ) are the largest and smallest eigenvalues of A A A , respectively. By the Toeplitz box of § 4.5.2 , a symmetric tridiagonal Toeplitz matrix tridiag ( p , d , p ) \operatorname{tridiag}(p,d,p) tridiag ( p , d , p ) has eigenvalues
λ k = d + 2 p cos ( k π n + 1 ) , k = 1 , 2 , … , n . \lambda_k = d + 2p\cos\left(\frac{k\pi}{n+1}\right),\quad k=1,2,\ldots,n. λ k = d + 2 p cos ( n + 1 k π ) , k = 1 , 2 , … , n .
This formula can be used to compute λ max \lambda_{\max} λ m a x and λ min \lambda_{\min} λ m i n .
(d) Analyze how the spectral condition number κ ( A ) \kappa(A) κ ( A ) behaves when c c c decreases and n n n increases and interpret the result.
Problem 5 (Bandwidth and naive LU cost). For a 2D n × n n \times n n × n grid with natural ordering, take n = 20 n=20 n = 20 (N = n 2 N=n^2 N = n 2 ). Estimate the half-bandwidth b b b and the work of banded LU via O ( N b 2 ) O(Nb^2) O ( N b 2 ) .
Companion notebook: week2/script_week2_2.ipynb
Problem 6 (Graph spectra, small and exact). For the path graph on 4 vertices, compute the eigenvalues of its Laplacian L L L . Then do the same for the complete graph K 4 K_4 K 4 . Compare the spectral gaps.
Problem 7 (2D Laplacian on an n × n n \times n n × n grid: spectrum and conditioning). Let T = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n T=\operatorname{tridiag}(-1,2,-1) \in\mathbb{R}^{n \times n} T = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n (Dirichlet 1D Laplacian without the 1 / h 2 1/h^2 1/ h 2 scaling) and let
T 2 D = I n ⊗ T + T ⊗ I n ∈ R n 2 × n 2 T_{2D}=I_n\otimes T+T\otimes I_n\in\mathbb{R}^{n^2\times n^2} T 2 D = I n ⊗ T + T ⊗ I n ∈ R n 2 × n 2
be the unscaled 2D matrix of Section 2.2.1, so that A 2 D = 1 h 2 T 2 D A_{2D}=\frac{1}{h^2}T_{2D} A 2 D = h 2 1 T 2 D .
(i) Show that the eigenvalues of T 2 D T_{2D} T 2 D are
λ p , q = λ p ( T ) + λ q ( T ) = ( 2 − 2 cos p π n + 1 ) + ( 2 − 2 cos q π n + 1 ) , p , q = 1 , … , n . \lambda_{p,q}=\lambda_p(T)+\lambda_q(T)=\left(2-2\cos\frac{p\pi}{n+1}\right)+\left(2-2\cos\frac{q\pi}{n+1}\right),\quad p,q=1,\ldots,n. λ p , q = λ p ( T ) + λ q ( T ) = ( 2 − 2 cos n + 1 p π ) + ( 2 − 2 cos n + 1 q π ) , p , q = 1 , … , n .
(ii) With the standard spacing h = 1 / ( n + 1 ) h=1/(n+1) h = 1/ ( n + 1 ) , the discretisation matrix is A 2 D = 1 h 2 T 2 D A_{2D}=\frac{1}{h^2}T_{2D} A 2 D = h 2 1 T 2 D . Prove that
λ max ( A 2 D ) ∼ 8 h 2 , λ min ( A 2 D ) → 2 π 2 ( n → ∞ ) , \lambda_{\max}(A_{2D})\sim\frac{8}{h^2},\qquad\lambda_{\min}(A_{2D})\to2\pi^2\quad(n\to\infty), λ m a x ( A 2 D ) ∼ h 2 8 , λ m i n ( A 2 D ) → 2 π 2 ( n → ∞ ) ,
and deduce the asymptotic condition number κ 2 ( A 2 D ) = O ( n 2 ) \kappa_2(A_{2D})=O(n^2) κ 2 ( A 2 D ) = O ( n 2 ) .
(iii) (Optional, d d d D generalization.) For d d d spatial dimensions with n d n^d n d unknowns, show that κ 2 ( A d D ) = O ( n 2 ) \kappa_2(A_{dD})=O(n^2) κ 2 ( A d D ) = O ( n 2 ) still holds, where A d D A_{dD} A d D is the analogous scaled matrix.
Companion notebook: week2/script_week2_2.ipynb
Problem 8 (Graph Laplacians: spectra and expansion, general n n n ). Let G = ( V , E ) G=(V,E) G = ( V , E ) be a simple undirected graph on n n n vertices with (combinatorial) Laplacian L = D − A L=D-A L = D − A .
(i) Show that 0 is an eigenvalue of L L L with multiplicity equal to the number of connected components.
(ii) For the cycle graph C n C_n C n , give the eigenvalues and eigenvectors of its Laplacian in terms of ω = exp ( 2 π i n ) \omega=\exp\left(\frac{2\pi i}{n}\right) ω = exp ( n 2 π i ) . (You may use the properties of circulant matrices.) Give an interval in which these eigenvalues lie.
(iii) For the star K 1 , n − 1 K_{1,n-1} K 1 , n − 1 on n n n vertices (one hub, n − 1 n-1 n − 1 leaves), show that σ ( L ) = { 0 , 1 (mult n − 2 ) , n } \sigma(L)=\{0,1\text{ (mult }n-2),n\} σ ( L ) = { 0 , 1 (mult n − 2 ) , n } .
Problem 9 (1D Poisson eigenvalues and κ ( A ) \kappa(A) κ ( A ) , small n n n ). For n = 5 n=5 n = 5 interior points, use
λ j = 4 h 2 sin 2 ( j π 2 ( n + 1 ) ) , h = 1 n + 1 , \lambda_j = \frac{4}{h^2}\sin^2\left(\frac{j\pi}{2(n+1)}\right), \qquad h=\frac{1}{n+1}, λ j = h 2 4 sin 2 ( 2 ( n + 1 ) j π ) , h = n + 1 1 , to list λ 1 , … , λ 5 \lambda_1,\ldots,\lambda_5 λ 1 , … , λ 5 numerically and compute κ ( A ) \kappa(A) κ ( A ) .
Repeat this, for the same n = 5 n=5 n = 5 , with the implicit diffusion matrix A ν = I + ν A 2 D A_\nu=I+\nu A_{2D} A ν = I + ν A 2 D of § 4.5.2 , whose n 2 = 25 n^2=25 n 2 = 25 eigenvalues are
μ i , j = 1 + 4 ν h 2 ( sin 2 ( i 2 ( n + 1 ) π ) + sin 2 ( j 2 ( n + 1 ) π ) ) , i , j = 1 , 2 , … , n , \mu_{i,j}=1+\frac{4\nu}{h^2}\left(\sin^2\left(\frac{i}{2(n+1)}\pi\right)+\sin^2\left(\frac{j}{2(n+1)}\pi\right)\right), \qquad i,j=1,2,\ldots,n, μ i , j = 1 + h 2 4 ν ( sin 2 ( 2 ( n + 1 ) i π ) + sin 2 ( 2 ( n + 1 ) j π ) ) , i , j = 1 , 2 , … , n , taking ν = 0.0025 \nu=0.0025 ν = 0.0025 . What differences do you notice between the two sets of eigenvalues? What will happen to the largest/smallest eigenvalues as n n n increases?
Problem 10 (Data assimilation spectrum example). We can construct a second order autoregressive (SOAR) correlation function on the unit circle for N N N equally distributed gridpoints, via
C ( i , j ) = ( 1 + ∣ 2 a sin ( θ i , j 2 ) ∣ L ) exp ( − ∣ 2 a sin ( θ i , j 2 ) ∣ L ) , 1 ≤ i , j ≤ N C(i,j)=\left(1+\frac{\left|2a\sin\left(\frac{\theta_{i,j}}{2}\right)\right|}{L}\right)\exp\left(-\frac{\left|2a\sin\left(\frac{\theta_{i,j}}{2}\right)\right|}{L}\right), \qquad1\leq i,j\leq N C ( i , j ) = 1 + L 2 a sin ( 2 θ i , j ) exp − L 2 a sin ( 2 θ i , j ) , 1 ≤ i , j ≤ N where L > 0 L>0 L > 0 is the correlation lengthscale, θ i , j \theta_{i,j} θ i , j denotes the angle between grid points i i i and j j j , and a a a is the radius of the domain [16 ] ; take a = 1 a=1 a = 1 throughout. This yields a dense matrix, which can be ill-conditioned. What happens to the spectrum if we replace all values less than some threshold (e.g. 0.2) by zero? Investigate how this changes as the lengthscale, threshold value and problem dimension increase.
Companion notebook: week2/script_week2_3.ipynb
Problem 11 (Concrete ML sparsity example). Four documents over a 6-word vocabulary with bags:
d 1 = { 1 , 2 } , d 2 = { 2 , 3 } , d 3 = { 3 , 4 } , d 4 = { 1 , 4 } . d_1=\{1,2\},\ d_2=\{2,3\},\ d_3=\{3,4\},\ d_4=\{1,4\}. d 1 = { 1 , 2 } , d 2 = { 2 , 3 } , d 3 = { 3 , 4 } , d 4 = { 1 , 4 } . Build the sparse document-term matrix A ∈ R 4 × 6 A\in\mathbb{R}^{4\times6} A ∈ R 4 × 6 (entries 0/1). Compute A T A A^\mathsf{T}A A T A explicitly and report its sparsity (which off-diagonal entries are nonzero?). Interpret the pattern.
Applications of Basic Iterative Methods
Overview
Chapter 1 introduced Jacobi, Gauss–Seidel, SOR, and Richardson as instances of a single fixed-point iteration x ( k + 1 ) = G x ( k ) + c x^{(k+1)} = Gx^{(k)} + c x ( k + 1 ) = G x ( k ) + c , convergent whenever ρ ( G ) < 1 \rho(G) < 1 ρ ( G ) < 1 . This chapter shows how this works on three model problems where the spectrum of G G G can be computed explicitly, which turns the abstract convergence criterion into concrete, comparable convergence rates. This chapter covers: (1) the 1D Poisson matrix, where explicit eigenpairs give exact convergence factors for Jacobi/GS/SOR; (2) graph Laplacians, where the same iterations become diffusion/random-walk processes and connect to spectral clustering; (3) machine learning, where Richardson iteration is exactly gradient descent and the spectrum of X T X X^{\mathsf{T}}X X T X governs both convergence speed and spectral bias .
By the end of this chapter, you should be able to:
Derive the explicit eigenvalues of the 1D Poisson matrix and use them to compute the exact spectral radius (and hence convergence rate) of Jacobi, Gauss–Seidel, and SOR.
Explain why Gauss–Seidel converges roughly twice as fast as Jacobi, and why SOR with an optimal ω \omega ω improves the rate further.
Interpret Jacobi iteration on a graph Laplacian as a random walk/diffusion process, and relate its slow modes to the Fiedler vector and spectral clustering.
Recognize Richardson iteration as gradient descent on a quadratic loss, and relate the spectrum of A A A (or X T X X^{\mathsf{T}}X X T X ) to the choice of step size and the resulting convergence rate.
Explain the phenomenon of spectral bias in machine learning as the same spectral-filtering mechanism that governs smoothing in PDE solvers, with the roles of low- and high-frequency modes swapped.
Basic Iterative Methods for PDE discretisations
We now go beyond the general fixed point framework of Chapter 1 and analyze in detail the convergence of classical iterative methods for the model problem of the one-dimensional Poisson equation with Dirichlet boundary conditions introduced in Chapter 2. This case is simple enough to allow explicit eigenvalue analysis, yet rich enough to illustrate the key phenomena that will reappear in higher–dimensional PDEs, graphs, and machine learning applications.
Recall the discretization of − u ′ ′ = f -u^{\prime\prime} = f − u ′′ = f on ( 0 , 1 ) (0,1) ( 0 , 1 ) with homogeneous Dirichlet boundary conditions using n n n interior points, mesh width h = 1 n + 1 h = \frac{1}{n+1} h = n + 1 1 . The stiffness matrix is
A = 1 h 2 tridiag ( − 1 , 2 , − 1 ) = 1 h 2 [ 2 − 1 − 1 2 − 1 ⋱ ⋱ ⋱ − 1 2 − 1 − 1 2 ] ∈ R n × n . (1) A = \frac{1}{h^2}\operatorname{tridiag}(-1,2,-1)=\frac{1}{h^2}
\begin{bmatrix}
2 & -1 & & & \\
-1 & 2 & -1 & & \\
& \ddots& \ddots& \ddots& \\
& & -1 & 2 & -1 \\
& & & -1 & 2
\end{bmatrix}\in\mathbb{R}^{n\times n}. \tag*{(1)} A = h 2 1 tridiag ( − 1 , 2 , − 1 ) = h 2 1 2 − 1 − 1 2 ⋱ − 1 ⋱ − 1 ⋱ 2 − 1 − 1 2 ∈ R n × n . (1)
This matrix is symmetric, positive definite, and tridiagonal.
Remark 3.1.1. The eigenvalues of A A A are explicitly known:
λ k ( A ) = 1 h 2 ( 2 − 2 cos θ k ) , θ k = k π n + 1 , k = 1 , … , n . (2) \lambda_k(A)=\frac{1}{h^2}(2-2\cos\theta_k),\qquad\theta_k=\frac{k\pi}{n+1},\quad k=1,\ldots,n. \tag*{(2)} λ k ( A ) = h 2 1 ( 2 − 2 cos θ k ) , θ k = n + 1 k π , k = 1 , … , n . (2)
or alternatively
λ k ( A ) = 4 h 2 sin 2 ( k π 2 ( n + 1 ) ) = 4 h 2 sin 2 ( k π h 2 ) , k = 1 , … , n . \lambda_k(A)=\frac{4}{h^2}\sin^2\left(\frac{k\pi}{2(n+1)}\right)=\frac{4}{h^2}\sin^2\left(\frac{k\pi h}{2}\right),\qquad k=1,\ldots,n. λ k ( A ) = h 2 4 sin 2 ( 2 ( n + 1 ) k π ) = h 2 4 sin 2 ( 2 k π h ) , k = 1 , … , n .
Hence λ min ∼ π 2 \lambda_{\min}\sim\pi^2 λ m i n ∼ π 2 , λ max ∼ 4 / h 2 \lambda_{\max}\sim4/h^2 λ m a x ∼ 4/ h 2 , and κ ( A ) = O ( n 2 ) \kappa(A)=O(n^2) κ ( A ) = O ( n 2 ) .
Figure 1 (PDF p. 38). Low -frequency (k k k small) vs. high -frequency (k k k large) eigenvectors v j ( k ) = sin ( j k π / ( n + 1 ) ) v_j^{(k)}=\sin(jk\pi/(n+1)) v j ( k ) = sin ( j k π / ( n + 1 )) of A A A .
Oscillatory modes are the ones a damped iteration removes first, while smooth modes decay slowly; this is the smoothing property multigrid rests on [7]. Note that undamped Jacobi does not have it: ∣ μ n + 1 − k ∣ = ∣ μ k ∣ |\mu_{n+1-k}|=|\mu_k| ∣ μ n + 1 − k ∣ = ∣ μ k ∣ , so the most oscillatory mode decays exactly as slowly as the smoothest (§ 5.2 and Problem 2(c)}); it is weighted Jacobi, at ω = 2 3 \omega=\frac{2}{3} ω = 3 2 , that smooths.
Convergence of the Jacobi iteration
We split A = D − L − U A=D-L-U A = D − L − U with D = 2 h 2 I D=\frac{2}{h^2}I D = h 2 2 I . The Jacobi iteration matrix is
G J = I − D − 1 A = I − h 2 2 A = I − 1 2 tridiag ( − 1 , 2 , − 1 ) = tridiag ( 1 2 , 0 , 1 2 ) . G_J=I-D^{-1}A=I-\frac{h^2}{2}A=I-\frac{1}{2}\operatorname{tridiag}(-1,2,-1)=\operatorname{tridiag}\left(\frac{1}{2},0,\frac{1}{2}\right). G J = I − D − 1 A = I − 2 h 2 A = I − 2 1 tridiag ( − 1 , 2 , − 1 ) = tridiag ( 2 1 , 0 , 2 1 ) .
Since G J = I − h 2 2 A G_J=I-\frac{h^2}{2}A G J = I − 2 h 2 A and given (2) the Jacobi eigenvalues are
μ k = 1 − h 2 2 λ k ( A ) = 1 − h 2 2 ⋅ 1 h 2 ( 2 − 2 cos θ k ) = cos θ k = cos ( k π n + 1 ) . \mu_k=1-\frac{h^2}{2}\lambda_k(A)=1-\frac{h^2}{2}\cdot\frac{1}{h^2}(2-2\cos\theta_k)=\cos\theta_k=\cos\left(\frac{k\pi}{n+1}\right). μ k = 1 − 2 h 2 λ k ( A ) = 1 − 2 h 2 ⋅ h 2 1 ( 2 − 2 cos θ k ) = cos θ k = cos ( n + 1 k π ) .
Hence the spectral radius is
ρ ( G J ) = max k ∣ μ k ∣ = cos ( π n + 1 ) = 1 − π 2 2 ( n + 1 ) 2 + O ( n − 4 ) , \rho(G_J)=\max_k|\mu_k|=\cos\left(\frac{\pi}{n+1}\right)=1-\frac{\pi^2}{2(n+1)^2}+O(n^{-4}), ρ ( G J ) = k max ∣ μ k ∣ = cos ( n + 1 π ) = 1 − 2 ( n + 1 ) 2 π 2 + O ( n − 4 ) ,
so Jacobi converges very slowly as n n n grows.
Convergence of the Gauss–Seidel iteration
For Gauss–Seidel we split A = ( D − L ) − U A=(D-L)-U A = ( D − L ) − U , with
D = 2 h 2 I , L = 1 h 2 tridiag ( 1 , 0 , 0 ) , U = 1 h 2 tridiag ( 0 , 0 , 1 ) . D=\frac{2}{h^2}I,\qquad L=\frac{1}{h^2}\operatorname{tridiag}(1,0,0),\qquad U=\frac{1}{h^2}\operatorname{tridiag}(0,0,1). D = h 2 2 I , L = h 2 1 tridiag ( 1 , 0 , 0 ) , U = h 2 1 tridiag ( 0 , 0 , 1 ) .
Then the iteration matrix is (see also the first exercise on the list for the exact formula)
G G S = ( D − L ) − 1 U . G_{GS} = (D-L)^{-1}U. G GS = ( D − L ) − 1 U .
Theorem 3.1.2 (Relation between eigenvalues of Gauss–Seidel and Jacobi). Let A = D − L − U A = D-L-U A = D − L − U with D D D diagonal, − L -L − L strictly lower, − U -U − U strictly upper, and all diagonal entries of D D D nonzero. Assume moreover that A A A has property A.4 Define
G J = D − 1 ( L + U ) , G G S = ( D − L ) − 1 U . G_J = D^{-1}(L+U), \qquad G_{GS} = (D-L)^{-1}U. G J = D − 1 ( L + U ) , G GS = ( D − L ) − 1 U . If μ \mu μ is a nonzero eigenvalue of G J G_J G J , then λ = μ 2 \lambda= \mu^2 λ = μ 2 is an eigenvalue of G G S G_{GS} G GS .
Proof. By definition,
G G S = ( I − D − 1 L ) − 1 D − 1 U . G_{GS} = (I-D^{-1}L)^{-1}D^{-1}U. G GS = ( I − D − 1 L ) − 1 D − 1 U .
Hence λ ∈ σ ( G G S ) \lambda\in\sigma(G_{GS}) λ ∈ σ ( G GS ) if and only if
det ( ( I − D − 1 L ) − 1 D − 1 U − λ I ) = 0 , \det\left((I-D^{-1}L)^{-1}D^{-1}U-\lambda I\right)=0, det ( ( I − D − 1 L ) − 1 D − 1 U − λ I ) = 0 ,
which is equivalent to
det ( D − 1 U − λ ( I − D − 1 L ) ) = 0 ⟺ det ( λ I − D − 1 ( λ L + 1 λ U ) ) = 0. (3) \det\left(D^{-1}U-\lambda(I-D^{-1}L)\right)=0
\iff
\det\left(\sqrt{\lambda}I-D^{-1}\left(\sqrt{\lambda}L+\frac{1}{\sqrt{\lambda}}U\right)\right)=0. \tag*{(3)} det ( D − 1 U − λ ( I − D − 1 L ) ) = 0 ⟺ det ( λ I − D − 1 ( λ L + λ 1 U ) ) = 0. (3)
Thus λ \sqrt{\lambda} λ is an eigenvalue of
M ( λ ) : = D − 1 ( 1 λ U + λ L ) . M(\lambda):=D^{-1}\left(\frac{1}{\sqrt{\lambda}}U+\sqrt{\lambda}L\right). M ( λ ) := D − 1 ( λ 1 U + λ L ) .
Now observe that M ( λ ) M(\lambda) M ( λ ) is similar to D − 1 ( L + U ) D^{-1}(L+U) D − 1 ( L + U ) . Therefore
σ ( M ( λ ) ) = σ ( D − 1 ( L + U ) ) = σ ( G J ) . \sigma(M(\lambda))=\sigma(D^{-1}(L+U))=\sigma(G_J). σ ( M ( λ )) = σ ( D − 1 ( L + U )) = σ ( G J ) .
Hence λ ∈ σ ( G J ) \sqrt{\lambda}\in\sigma(G_J) λ ∈ σ ( G J ) . Same holds for − λ -\sqrt{\lambda} − λ .
Conclusion. Every nonzero Jacobi eigenvalue contributes μ k 2 = cos 2 ( k π n + 1 ) \mu_k^2=\cos^2\left(\frac{k\pi}{n+1}\right) μ k 2 = cos 2 ( n + 1 k π ) to the Gauss–Seidel spectrum, and in particular ρ ( G G S ) = ρ ( G J ) 2 \rho(G_{GS})=\rho(G_J)^2 ρ ( G GS ) = ρ ( G J ) 2 , so Gauss–Seidel converges twice as fast as Jacobi. Note that in the proof we divide by λ \sqrt{\lambda} λ , so we don’t consider the case λ = 0 \lambda=0 λ = 0 , which is an eigenvalue of G G S G_{GS} G GS here, of high multiplicity (Remark 3.1.4 ). And it gives an inclusion of spectra, not an equality.
Remark 3.1.3 (Why the property A is important). Take D = I D=I D = I and
G J = [ 0 1 1 1 0 1 1 1 0 ] , that is A = I − G J = [ 1 − 1 − 1 − 1 1 − 1 − 1 − 1 1 ] . G_J =
\begin{bmatrix}
0 & 1 & 1\\
1 & 0 & 1\\
1 & 1 & 0
\end{bmatrix},
\qquad\text{that is}\qquad
A=I-G_J=
\begin{bmatrix}
1 & -1 & -1\\
-1 & 1 & -1\\
-1 & -1 & 1
\end{bmatrix}. G J = 0 1 1 1 0 1 1 1 0 , that is A = I − G J = 1 − 1 − 1 − 1 1 − 1 − 1 − 1 1 .
Here σ ( G J ) = { 2 , − 1 , − 1 } \sigma(G_J)=\{2,-1,-1\} σ ( G J ) = { 2 , − 1 , − 1 } , so the squares are { 4 , 1 , 1 } \{4,1,1\} { 4 , 1 , 1 } , whereas σ ( G G S ) = { 0 , 2 − 5 , 2 + 5 } \sigma(G_{GS})=\{0,2-\sqrt{5},2+\sqrt{5}\} σ ( G GS ) = { 0 , 2 − 5 , 2 + 5 } : no Jacobi eigenvalue’s square is a Gauss–Seidel eigenvalue. The off-diagonal pattern of A A A is the triangle K 3 K_3 K 3 , an odd cycle, hence not bipartite, so A A A does not have property A.
Convergence of the SOR iteration
For SOR with relaxation parameter ω ∈ ( 0 , 2 ) \omega\in(0,2) ω ∈ ( 0 , 2 ) we use the splitting A = 1 ω ( D − ω L ) − ( ( 1 − ω ) D + ω U ) A=\frac{1}{\omega}(D-\omega L)-((1-\omega)D+\omega U) A = ω 1 ( D − ω L ) − (( 1 − ω ) D + ω U ) , so that the iteration matrix is
G ω = ( D − ω L ) − 1 ( ( 1 − ω ) D + ω U ) . G_\omega=(D-\omega L)^{-1}((1-\omega)D+\omega U). G ω = ( D − ω L ) − 1 (( 1 − ω ) D + ω U ) .
Young’s theorem [47 , 42 ] 2 gives for certain matrices a relation between the SOR convergence factor and that of Jacobi and the optimal parameter ω \omega ω :
ω o p t = 2 1 + 1 − ρ ( G J ) 2 , ρ ( G ω o p t ) = 1 − 1 − ρ ( G J ) 2 1 + 1 − ρ ( G J ) 2 . \omega_{\mathrm{opt}}=\frac{2}{1+\sqrt{1-\rho(G_J)^2}},\qquad\rho(G_{\omega_{\mathrm{opt}}})=\frac{1-\sqrt{1-\rho(G_J)^2}}{1+\sqrt{1-\rho(G_J)^2}}. ω opt = 1 + 1 − ρ ( G J ) 2 2 , ρ ( G ω opt ) = 1 + 1 − ρ ( G J ) 2 1 − 1 − ρ ( G J ) 2 .
For our 1D Poisson problem, ρ ( G J ) = cos ( π n + 1 ) \rho(G_J)=\cos\left(\frac{\pi}{n+1}\right) ρ ( G J ) = cos ( n + 1 π ) , which yields
ω o p t = 2 1 + sin ( π n + 1 ) , ρ ( G ω o p t ) = 1 − sin ( π n + 1 ) 1 + sin ( π n + 1 ) . \omega_{\mathrm{opt}}=\frac{2}{1+\sin\left(\frac{\pi}{n+1}\right)},\qquad\rho(G_{\omega_{\mathrm{opt}}})=\frac{1-\sin\left(\frac{\pi}{n+1}\right)}{1+\sin\left(\frac{\pi}{n+1}\right)}. ω opt = 1 + sin ( n + 1 π ) 2 , ρ ( G ω opt ) = 1 + sin ( n + 1 π ) 1 − sin ( n + 1 π ) .
Proposition 3.1.5 (Asymptotic Convergence and Limitations). Based on the previous results we can derive the asymptotic convergence behaviour of the three classical iterative methods. For large n n n , the spectral radii expand as
ρ ( G J ) = 1 − π 2 2 ( n + 1 ) 2 + O ( n − 4 ) , ρ ( G G S ) = 1 − π 2 ( n + 1 ) 2 + O ( n − 4 ) , ρ ( G ω o p t ) = 1 − 2 π n + 1 + O ( n − 2 ) . \begin{aligned}
\rho(G_J)&=1-\frac{\pi^2}{2(n+1)^2}+O(n^{-4}),\\
\rho(G_{GS})&=1-\frac{\pi^2}{(n+1)^2}+O(n^{-4}),\\
\rho(G_{\omega_{\mathrm{opt}}})&=1-\frac{2\pi}{n+1}+O(n^{-2}).
\end{aligned} ρ ( G J ) ρ ( G GS ) ρ ( G ω opt ) = 1 − 2 ( n + 1 ) 2 π 2 + O ( n − 4 ) , = 1 − ( n + 1 ) 2 π 2 + O ( n − 4 ) , = 1 − n + 1 2 π + O ( n − 2 ) .
Example 3.1.6 (Numerical comparison, n = 49 n=49 n = 49 ). Take n = 49 n=49 n = 49 interior points (so h = 1 / 50 h=1/50 h = 1/50 ). The three spectral radii evaluate to
ρ ( G J ) = cos ( π 50 ) ≈ 0.99803 , ρ ( G G S ) = ρ ( G J ) 2 ≈ 0.99606 , ρ ( G ω o p t ) ≈ 0.88184. \rho(G_J)=\cos\left(\frac{\pi}{50}\right)\approx0.99803,\qquad\rho(G_{GS})=\rho(G_J)^2\approx0.99606,\qquad\rho(G_{\omega_{\mathrm{opt}}})\approx0.88184. ρ ( G J ) = cos ( 50 π ) ≈ 0.99803 , ρ ( G GS ) = ρ ( G J ) 2 ≈ 0.99606 , ρ ( G ω opt ) ≈ 0.88184. The asymptotic rate predicts k ≈ − 6 / log 10 ρ k\approx-6/\log_{10}\rho k ≈ − 6/ log 10 ρ iterations to gain six digits, that is
k J ≈ 7000 , k G S ≈ 3500 , k ω o p t ≈ 110. k_J\approx7000,\qquad k_{GS}\approx3500,\qquad k_{\omega_{\mathrm{opt}}}\approx110. k J ≈ 7000 , k GS ≈ 3500 , k ω opt ≈ 110. So for this modest grid, optimally relaxed SOR needs roughly 64 × 64\times 64 × fewer iterations than Jacobi to reach the same accuracy, which is a direct, numerical illustration of the asymptotic rates above. These are asymptotic estimates , not guarantees: G ω o p t G_{\omega_{\mathrm{opt}}} G ω opt is non-normal (∥ G ω o p t ∥ 2 ≈ 2.78 \lVert G_{\omega_{\mathrm{opt}}}\rVert_2\approx2.78 ∥ G ω opt ∥ 2 ≈ 2.78 ), so the error can grow before it decays. Indeed ρ 110 ≈ 9.8 ⋅ 10 − 7 \rho^{110}\approx9.8\cdot10^{-7} ρ 110 ≈ 9.8 ⋅ 1 0 − 7 while ∥ G ω o p t 110 ∥ 2 ≈ 7.2 ⋅ 10 − 5 \lVert G_{\omega_{\mathrm{opt}}}^{110}\rVert_2\approx7.2\cdot10^{-5} ∥ G ω opt 110 ∥ 2 ≈ 7.2 ⋅ 1 0 − 5 , and in practice k = 148 k=148 k = 148 is needed to guarantee 10 − 6 10^{-6} 1 0 − 6 for every initial error.
Takeaways
Jacobi converges very slowly: ρ ( G J ) → 1 \rho(G_J) \to1 ρ ( G J ) → 1 as n → ∞ n \to\infty n → ∞ , with rate O ( 1 / n 2 ) O(1/n^2) O ( 1/ n 2 ) .
Gauss–Seidel roughly doubles the effective rate, but is still O ( 1 / n 2 ) O(1/n^2) O ( 1/ n 2 ) .
SOR with optimal relaxation improves to O ( 1 / n ) O(1/n) O ( 1/ n ) per iteration, but this is still poor compared to modern solvers (e.g. multigrid or Krylov methods).
Thus none of these methods are suitable as standalone solvers for large PDE systems.
Basic Iterative Methods for Graph Laplacians
Graph Laplacians defined in Chapter 2 provide another fundamental class of sparse matrices. We recall that for any graph, the smallest eigenvalue of L L L is given by 0 0 0 , with corresponding eigenvector e = [ 1 , 1 , … , 1 ] T \mathbf{e} = [1,1,\ldots,1]^T e = [ 1 , 1 , … , 1 ] T i.e. the vector with all unit entries. This can be seen as by definition, each row of the graph Laplacian matrix sums to 0 0 0 .
Below a few examples with corresponding spectral properties
Figure 1 (PDF p. 41).
Path graph P n P_n P n . Here L L L is a rank−2 perturbation of T = tridiag ( − 1 , 2 , − 1 ) T = \operatorname{tridiag}(-1,2,-1) T = tridiag ( − 1 , 2 , − 1 ) , the unscaled matrix of (3.1) (Chapter 2). We can bound its eigenvalues with the following result
Theorem 3.2.1 ([45 ] ). Consider two symmetric matrices S 1 , S 2 ∈ R n × n S_1,S_2 \in\mathbb{R}^{n \times n} S 1 , S 2 ∈ R n × n . Let λ 1 ≥ λ 2 ≥ ⋯ ≥ λ n \lambda_1 \ge\lambda_2 \ge\cdots\ge\lambda_n λ 1 ≥ λ 2 ≥ ⋯ ≥ λ n . Then the k k k th eigenvalue of the matrix S 1 + S 2 S_1+S_2 S 1 + S 2 satisfies
λ k ( S 1 ) + λ n ( S 2 ) ≤ λ k ( S 1 + S 2 ) ≤ λ k ( S 1 ) + λ 1 ( S 2 ) \lambda_k(S_1) + \lambda_n(S_2) \le\lambda_k(S_1 + S_2) \le\lambda_k(S_1) + \lambda_1(S_2) λ k ( S 1 ) + λ n ( S 2 ) ≤ λ k ( S 1 + S 2 ) ≤ λ k ( S 1 ) + λ 1 ( S 2 )
Corollary 3.2.2 (Bound on eigenvalues of L L L for P n P_n P n ). Let T = h 2 A = tridiag ( − 1 , 2 , − 1 ) T = h^2 A = \operatorname{tridiag}(-1,2,-1) T = h 2 A = tridiag ( − 1 , 2 , − 1 ) be as above. Then L = T + D L = T + D L = T + D , where D D D is the diagonal matrix with D ( 1 , 1 ) = D ( n , n ) = − 1 D(1,1) = D(n,n) = -1 D ( 1 , 1 ) = D ( n , n ) = − 1 and all other entries zero: the two ends of the path are free, where the boundary value problem has Dirichlet conditions. Order eigenvalues decreasingly, λ 1 ≥ ⋯ ≥ λ n \lambda_1 \ge\cdots\ge\lambda_n λ 1 ≥ ⋯ ≥ λ n , so that λ 1 ( D ) = 0 \lambda_1(D) = 0 λ 1 ( D ) = 0 and λ n ( D ) = − 1 \lambda_n(D) = -1 λ n ( D ) = − 1 . Applying Theorem 3.2.1 to T + D T + D T + D and then to D + T D + T D + T gives
max ( λ k ( T ) − 1 , λ n ( T ) + λ k ( D ) ) ≤ λ k ( L ) ≤ min ( λ k ( T ) , λ 1 ( T ) + λ k ( D ) ) \max\left(\lambda_k(T)-1,\lambda_n(T)+\lambda_k(D)\right) \le\lambda_k(L) \le\min\left(\lambda_k(T),\lambda_1(T)+\lambda_k(D)\right) max ( λ k ( T ) − 1 , λ n ( T ) + λ k ( D ) ) ≤ λ k ( L ) ≤ min ( λ k ( T ) , λ 1 ( T ) + λ k ( D ) ) In particular the extreme eigenvalues are bounded by:
λ min ( T ) − 1 ≤ λ min ( L ) ≤ min ( λ min ( T ) , λ max ( T ) − 1 ) \lambda_{\min}(T)-1 \le\lambda_{\min}(L) \le\min\left(\lambda_{\min}(T),\lambda_{\max}(T)-1\right) λ m i n ( T ) − 1 ≤ λ m i n ( L ) ≤ min ( λ m i n ( T ) , λ m a x ( T ) − 1 ) max ( λ max ( T ) − 1 , λ min ( T ) ) ≤ λ max ( L ) ≤ λ max ( T ) \max\left(\lambda_{\max}(T)-1,\lambda_{\min}(T)\right) \le\lambda_{\max}(L) \le\lambda_{\max}(T) max ( λ m a x ( T ) − 1 , λ m i n ( T ) ) ≤ λ m a x ( L ) ≤ λ m a x ( T )
Cycle graph C n C_n C n . Eigenvectors are Fourier modes; eigenvalues are λ k = 2 − 2 cos ( 2 π k / n ) \lambda_k = 2 - 2\cos(2\pi k/n) λ k = 2 − 2 cos ( 2 π k / n ) . Jacobi eigenvalues are cos ( 2 π k / n ) \cos(2\pi k/n) cos ( 2 π k / n ) .
Star graph K 1 , n − 1 K_{1,n-1} K 1 , n − 1 . The spectrum of the graph Laplacian is { 0 , 1 ( × n − 2 ) , n } \{0,1(\times n-2),n\} { 0 , 1 ( × n − 2 ) , n } .
For real graph algorithms, one typically turns to more advanced methods. Still, the classical iterations remain a useful “microscope” for understanding graph structure. They also have clear interpretations from the perspective of classical graph theory, as we shall see later.
Example 3.2.3 (Jacobi = random walk on a path graph). Consider the path P 4 P_4 P 4 with vertices { 1 , 2 , 3 , 4 } \{1,2,3,4\} { 1 , 2 , 3 , 4 } . The adjacency matrix and degree matrix are
A = [ 0 1 0 0 1 0 1 0 0 1 0 1 0 0 1 0 ] , D = diag ( 1 , 2 , 2 , 1 ) . A = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 0 & 0 & 1 & 0 \end{bmatrix}, \qquad D = \operatorname{diag}(1,2,2,1). A = 0 1 0 0 1 0 1 0 0 1 0 1 0 0 1 0 , D = diag ( 1 , 2 , 2 , 1 ) . The Jacobi iteration matrix is
G J = D − 1 A = [ 0 1 0 0 1 2 0 1 2 0 0 1 2 0 1 2 0 0 1 0 ] . G_J = D^{-1}A = \begin{bmatrix} 0 & 1 & 0 & 0 \\ \frac{1}{2} & 0 & \frac{1}{2} & 0 \\ 0 & \frac{1}{2} & 0 & \frac{1}{2} \\ 0 & 0 & 1 & 0 \end{bmatrix}. G J = D − 1 A = 0 2 1 0 0 1 0 2 1 0 0 2 1 0 1 0 0 2 1 0 . Interpretation. Each row of G J G_J G J gives the transition probabilities of a random walk:
From vertex 1, you move to vertex 2 with probability 1.
From vertex 2, you move to 1 or 3 with equal probability 1 / 2 1/2 1/2 .
From vertex 3, you move to 2 or 4 with probability 1 / 2 1/2 1/2 .
From vertex 4, you move to vertex 3 with probability 1.
A function on the vertices is averaged over neighbours by P : = G J P := G_J P := G J - this is the Jacobi iteration: from x ( 0 ) = [ 1 , 0 , 0 , 0 ] T x^{(0)} = [1,0,0,0]^{\mathsf{T}} x ( 0 ) = [ 1 , 0 , 0 , 0 ] T ,
x ( 1 ) = P x ( 0 ) = [ 0 , 1 2 , 0 , 0 ] T , x ( 2 ) = P x ( 1 ) = [ 1 2 , 0 , 1 4 , 0 ] T , x^{(1)} = P x^{(0)} = [0,\tfrac{1}{2},0,0]^{\mathsf{T}}, \qquad x^{(2)} = P x^{(1)} = [\tfrac{1}{2},0,\tfrac{1}{4},0]^{\mathsf{T}}, x ( 1 ) = P x ( 0 ) = [ 0 , 2 1 , 0 , 0 ] T , x ( 2 ) = P x ( 1 ) = [ 2 1 , 0 , 4 1 , 0 ] T , and so on.
Takeaways
Jacobi iteration on L x = b Lx=b Lx = b is local averaging along the edges of the underlying graph, with the eigenvalues describing the rates. Damping turns it into a diffusion: since
G ω = I − ω D − 1 L = ( 1 − ω ) I + ω P , G_\omega= I - \omega D^{-1}L = (1-\omega)I + \omega P, G ω = I − ω D − 1 L = ( 1 − ω ) I + ω P ,
damped Jacobi with ω = 1 2 \omega=\frac{1}{2} ω = 2 1 is exactly the lazy random walk P 1 / 2 = 1 2 ( I + P ) P_{1/2}=\frac{1}{2}(I+P) P 1/2 = 2 1 ( I + P ) .
Relation to spectral clustering. The slowest-decaying Jacobi modes are precisely the eigenvectors of the Laplacian associated with the smallest nonzero eigenvalues. These eigenvectors are smooth signals on the graph: they vary little across densely connected regions and change more abruptly across weak connections. In spectral graph theory, this property is exploited to detect communities [43 ] . For instance, the Fiedler vector [10] (the eigenvector associated with the second-smallest Laplacian eigenvalue) typically has nearly constant values on different clusters and a sharp transition across the cut between them. Thresholding this vector provides a natural bipartition of the graph.
Example 3.2.4 (Jacobi iteration and community detection). Consider the graph consisting of two triangles connected by a single edge: vertices { 1 , 2 , 3 } \{1,2,3\} { 1 , 2 , 3 } fully connected to each other, vertices { 4 , 5 , 6 } \{4,5,6\} { 4 , 5 , 6 } fully connected to each other, and one edge ( 3 , 4 ) (3,4) ( 3 , 4 ) connecting the two groups.
Figure 1 (PDF p. 43).
Graph Laplacian construction. The adjacency matrix is
A = [ 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 ] , A = \begin{bmatrix} 0 & 1 & 1 & 0 & 0 & 0 \\ 1 & 0 & 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 & 0 & 1 \\ 0 & 0 & 0 & 1 & 1 & 0 \end{bmatrix}, A = 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 , and the degree matrix is
D = diag ( 2 , 2 , 3 , 3 , 2 , 2 ) . D = \operatorname{diag}(2,2,3,3,2,2). D = diag ( 2 , 2 , 3 , 3 , 2 , 2 ) . Thus the Laplacian is
L = D − A = [ 2 − 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 − 1 3 − 1 0 0 0 0 − 1 3 − 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 − 1 2 ] . L = D - A = \begin{bmatrix} 2 & -1 & -1 & 0 & 0 & 0 \\ -1 & 2 & -1 & 0 & 0 & 0 \\ -1 & -1 & 3 & -1 & 0 & 0 \\ 0 & 0 & -1 & 3 & -1 & -1 \\ 0 & 0 & 0 & -1 & 2 & -1 \\ 0 & 0 & 0 & -1 & -1 & 2 \end{bmatrix}. L = D − A = 2 − 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 − 1 3 − 1 0 0 0 0 − 1 3 − 1 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 − 1 2 .
Spectrum and Fiedler vector. For the L L L above, λ ( L ) ≈ { 0 , 0.438447 , 3 , 3 , 3 , 4.561553 } \lambda(L) \approx\{0, 0.438447, 3, 3, 3, 4.561553\} λ ( L ) ≈ { 0 , 0.438447 , 3 , 3 , 3 , 4.561553 } , so λ 2 ≈ 0.438447 \lambda_2 \approx0.438447 λ 2 ≈ 0.438447 is smallest nonzero. An associated Fiedler vector is
v F i e d l e r ≈ [ − 0.465 , − 0.465 , − 0.261 , 0.261 , 0.465 , 0.465 ] T . v_{\mathrm{Fiedler}} \approx[-0.465,-0.465,-0.261,0.261,0.465,0.465]^{\mathsf{T}}. v Fiedler ≈ [ − 0.465 , − 0.465 , − 0.261 , 0.261 , 0.465 , 0.465 ] T .
It has a sign change across the bridge ( 3 , 4 ) (3,4) ( 3 , 4 ) . Thresholding at 0 0 0 (or performing a sweep cut) yields the two clusters.
Two Fiedler vectors, and they are not the same. The vector above belongs to the combinatorial problem L v = λ v Lv=\lambda v Lv = λ v . The iteration of the next section instead solves the generalized problem L v = λ D v Lv=\lambda Dv Lv = λ D v whose second eigenvector is
v F i e d l e r n o r m = D − 1 / 2 v 2 ( L s y m ) ≈ [ − 0.461 , − 0.461 , − 0.273 , 0.273 , 0.461 , 0.461 ] T . v_{\mathrm{Fiedler}}^{\mathrm{norm}} = D^{-1/2}v_2(L_{\mathrm{sym}}) \approx[-0.461,-0.461,-0.273,0.273,0.461,0.461]^{\mathsf{T}}. v Fiedler norm = D − 1/2 v 2 ( L sym ) ≈ [ − 0.461 , − 0.461 , − 0.273 , 0.273 , 0.461 , 0.461 ] T .
On this graph, the two differ and induce the same bipartition, but they are different vectors, and on graphs with a wide spread of degrees they can differ more substantially. We will be careful below to say which of the two an iteration actually produces.
Damped Jacobi as both power iteration and low-pass filter. Let G = ( V , E ) G=(V,E) G = ( V , E ) be an undirected graph with adjacency A A A , degree matrix D D D , and random walk matrix P : = D − 1 A P:=D^{-1}A P := D − 1 A . Define the symmetric normalized adjacency S : = D − 1 / 2 A D − 1 / 2 S:=D^{-1/2}AD^{-1/2} S := D − 1/2 A D − 1/2 and the normalized Laplacians L r w : = I − P \mathcal{L}_{\mathrm{rw}}:=I-P L rw := I − P and L s y m : = I − S \mathcal{L}_{\mathrm{sym}}:=I-S L sym := I − S . We have the similarity P = D − 1 / 2 S D 1 / 2 P=D^{-1/2}SD^{1/2} P = D − 1/2 S D 1/2 , hence P P P and S S S share eigenvalues 1 = μ 1 ≥ μ 2 ≥ ⋯ ≥ μ n ≥ − 1 1=\mu_1\ge\mu_2\ge\cdots\ge\mu_n\ge-1 1 = μ 1 ≥ μ 2 ≥ ⋯ ≥ μ n ≥ − 1 . Let { v i } i = 1 n \{v_i\}_{i=1}^n { v i } i = 1 n be an orthonormal basis of eigenvectors of S S S (S v i = μ i v i Sv_i=\mu_i v_i S v i = μ i v i ), so v 1 = D 1 / 2 1 ∥ D 1 / 2 1 ∥ 2 v_1=\frac{D^{1/2}\mathbf{1}}{\lVert D^{1/2}\mathbf{1}\rVert_2} v 1 = ∥ D 1/2 1 ∥ 2 D 1/2 1 and λ i s y m = 1 − μ i \lambda_i^{\mathrm{sym}}=1-\mu_i λ i sym = 1 − μ i are the eigenvalues of L s y m \mathcal{L}_{\mathrm{sym}} L sym .
Low-pass view. Damped Jacobi with b = 0 b=0 b = 0 is x ( k + 1 ) = P 1 / 2 x ( k ) x^{(k+1)}=P_{1/2}x^{(k)} x ( k + 1 ) = P 1/2 x ( k ) , so with y ( k ) : = D 1 / 2 x ( k ) y^{(k)}:=D^{1/2}x^{(k)} y ( k ) := D 1/2 x ( k ) ,
y ( k ) = ( I + S 2 ) k y ( 0 ) ⟹ x ( k ) = D − 1 / 2 ∑ i = 1 n ( 1 + μ i 2 ) k ( D 1 / 2 x ( 0 ) , v i ) v i . y^{(k)}=\left(\frac{I+S}{2}\right)^k y^{(0)}\quad\Longrightarrow\quad x^{(k)}=D^{-1/2}\sum_{i=1}^{n}\left(\frac{1+\mu_i}{2}\right)^k(D^{1/2}x^{(0)},v_i)v_i. y ( k ) = ( 2 I + S ) k y ( 0 ) ⟹ x ( k ) = D − 1/2 i = 1 ∑ n ( 2 1 + μ i ) k ( D 1/2 x ( 0 ) , v i ) v i .
Each spectral component is therefore multiplied by ( 1 + μ i 2 ) k = ( 1 − λ i s y m 2 ) k \left(\frac{1+\mu_i}{2}\right)^k=\left(1-\frac{\lambda_i^{\mathrm{sym}}}{2}\right)^k ( 2 1 + μ i ) k = ( 1 − 2 λ i sym ) k , a factor lying in [ 0 , 1 ] [0,1] [ 0 , 1 ] and decreasing in λ i s y m \lambda_i^{\mathrm{sym}} λ i sym : high graph frequencies are attenuated fastest and smooth modes persist. Damped Jacobi is thus a low-pass graph filter (a filter which removes high-frequency components and permits low-frequency components to pass through). In the undamped Jacobi the modes are multiplied by μ i k \mu_i^k μ i k instead, and ∣ μ i ∣ → 1 |\mu_i|\to1 ∣ μ i ∣ → 1 as λ i s y m → 2 \lambda_i^{\mathrm{sym}}\to2 λ i sym → 2 , so the most oscillatory ones are not attenuated at all.
Power method view. The low-pass picture has a classical counterpart. Let M ∈ R n × n M\in\mathbb{R}^{n\times n} M ∈ R n × n have a dominant eigenvalue, that is, let its eigenvalues be ordered as ∣ λ 1 ∣ > ∣ λ 2 ∣ ≥ ⋯ ≥ ∣ λ n ∣ |\lambda_1|>|\lambda_2|\ge\cdots\ge|\lambda_n| ∣ λ 1 ∣ > ∣ λ 2 ∣ ≥ ⋯ ≥ ∣ λ n ∣ , so that ρ ( M ) = ∣ λ 1 ∣ \rho(M)=|\lambda_1| ρ ( M ) = ∣ λ 1 ∣ . Multiplying a vector repeatedly by M M M amplifies its component along the dominant eigenvector v 1 v_1 v 1 more than any other, so that after rescaling at each step the iterate lines up with v 1 v_1 v 1 . This is the power method , the oldest of all eigenvalue iterations, and the simplest predecessor of the Lanczos method of Chapter 4.
Algorithm 1 Power method
Require: Matrix M ∈ R n × n M\in\mathbb{R}^{n\times n} M ∈ R n × n with a dominant eigenvalue, starting vector y ( 0 ) y^{(0)} y ( 0 ) with ∥ y ( 0 ) ∥ 2 = 1 \lVert y^{(0)}\rVert_2=1 ∥ y ( 0 ) ∥ 2 = 1 and a nonzero component along v 1 v_1 v 1 , tolerance ε > 0 \varepsilon>0 ε > 0 .
for k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … do
z ← M y ( k ) z\leftarrow My^{(k)} z ← M y ( k ) \hfill {one matrix–vector product per step}
μ k ← ( y ( k ) ) T z \mu_k\leftarrow(y^{(k)})^{\mathsf{T}}z μ k ← ( y ( k ) ) T z \hfill {Rayleigh quotient, since ∥ y ( k ) ∥ 2 = 1 \lVert y^{(k)}\rVert_2=1 ∥ y ( k ) ∥ 2 = 1 }
if ∥ z − μ k y ( k ) ∥ 2 ≤ ε \lVert z-\mu_k y^{(k)}\rVert_2\le\varepsilon ∥ z − μ k y ( k ) ∥ 2 ≤ ε then
stop
end if
y ( k + 1 ) ← z / ∥ z ∥ 2 y^{(k+1)}\leftarrow z/\lVert z\rVert_2 y ( k + 1 ) ← z / ∥ z ∥ 2
end for
Ensure: Approximate dominant eigenpair ( μ k , y ( k ) ) (\mu_k,y^{(k)}) ( μ k , y ( k ) ) .
Provided y ( 0 ) y^{(0)} y ( 0 ) has a nonzero component along v 1 v_1 v 1 , the iterates converge (up to sign) to the dominant eigenvector, and the Rayleigh quotients μ k \mu_k μ k converge to λ 1 \lambda_1 λ 1 . The rate is governed by the ratio of the two largest eigenvalue moduli: the eigenvector error decays like ∣ λ 2 / λ 1 ∣ k |\lambda_2/\lambda_1|^k ∣ λ 2 / λ 1 ∣ k and the eigenvalue error, for symmetric M M M , twice as fast. Slow convergence is therefore expected when the two largest eigenvalues are not well separated. See Problem 8 for details.
Deflating the trivial mode to reveal v 2 v_2 v 2 . The lazy walk P 1 / 2 P_{1/2} P 1/2 has dominant eigenvalue 1 1 1 with right eigenvector 1 \mathbf{1} 1 . To find the first eigenpair different from ( 1 , 1 ) (1,1) ( 1 , 1 ) we project it out after each step, in the D D D -inner product ⟨ u , v ⟩ D : = u T D v \langle u,v\rangle_D:=u^{\mathsf{T}}Dv ⟨ u , v ⟩ D := u T D v , and only then normalize:
x ^ ( k + 1 ) = P 1 / 2 x ( k ) − ⟨ P 1 / 2 x ( k ) , 1 ⟩ D ⟨ 1 , 1 ⟩ D 1 , x ( k + 1 ) = x ^ ( k + 1 ) ∥ x ^ ( k + 1 ) ∥ D . \widehat{x}^{(k+1)}=P_{1/2}x^{(k)}-\frac{\langle P_{1/2}x^{(k)},\mathbf{1}\rangle_D}{\langle\mathbf{1},\mathbf{1}\rangle_D}\mathbf{1},\qquad x^{(k+1)}=\frac{\widehat{x}^{(k+1)}}{\lVert\widehat{x}^{(k+1)}\rVert_D}. x ( k + 1 ) = P 1/2 x ( k ) − ⟨ 1 , 1 ⟩ D ⟨ P 1/2 x ( k ) , 1 ⟩ D 1 , x ( k + 1 ) = ∥ x ( k + 1 ) ∥ D x ( k + 1 ) .
(The projection alone does not normalize anything; the two steps are separate.) Setting y ( k ) : = D 1 / 2 x ( k ) y^{(k)} := D^{1/2}x^{(k)} y ( k ) := D 1/2 x ( k ) and expanding y ( 0 ) = ∑ i ≥ 2 α i v i y^{(0)} = \sum_{i\ge2}\alpha_i v_i y ( 0 ) = ∑ i ≥ 2 α i v i , the iterates satisfy
y ( k ) ∝ ∑ i ≥ 2 α i ( 1 + μ i 2 ) k v i , y^{(k)} \propto\sum_{i\ge2}\alpha_i\left(\frac{1+\mu_i}{2}\right)^k v_i, y ( k ) ∝ i ≥ 2 ∑ α i ( 2 1 + μ i ) k v i ,
the omitted factor being the product of the normalizations. All these factors are nonnegative , so the largest of them is simply the one corresponding to μ 2 \mu_2 μ 2 : if G G G is connected, μ 2 \mu_2 μ 2 is simple, and the starting vector has a nonzero component along v 2 v_2 v 2 (that is, α 2 ≠ 0 \alpha_2 \ne0 α 2 = 0 ), then
x ( k ) ⟶ ± D − 1 / 2 v 2 at rate ( 1 + μ 3 1 + μ 2 ) k . x^{(k)} \longrightarrow\pm D^{-1/2}v_2 \qquad\text{at rate} \qquad\left(\frac{1+\mu_3}{1+\mu_2}\right)^k. x ( k ) ⟶ ± D − 1/2 v 2 at rate ( 1 + μ 2 1 + μ 3 ) k .
The limit is the normalized Fiedler direction of Example 3.2.4, the second eigenvector of L v = λ D v Lv = \lambda Dv Lv = λ D v . The hypothesis is the same one the power method always needs, a dominant eigenvalue, here on the subspace 1 ⊥ D 1^{\perp_D} 1 ⊥ D . Note that this is again a place where the damping is doing real work: for the undamped walk the surviving mode is the one of largest modulus among μ 2 , … , μ n \mu_2,\ldots,\mu_n μ 2 , … , μ n , which need not be μ 2 \mu_2 μ 2 at all (Problem 8(e)). Thus damped Jacobi = power iteration , and with deflation it converges to the normalized Fiedler direction.
Remark 3.2.5 (How much damping? ω = 1 2 \omega= \frac{1}{2} ω = 2 1 is safe, not optimal). Writing P ω = ( 1 − ω ) I + ω P P_\omega= (1-\omega)I+\omega P P ω = ( 1 − ω ) I + ω P , the eigenvalues are 1 − ω + ω μ i 1-\omega+\omega\mu_i 1 − ω + ω μ i and the deflated iteration converges at the rate r ( ω ) = max i ≥ 3 ∣ 1 − ω + ω μ i ∣ 1 − ω + ω μ 2 r(\omega) = \frac{\max_{i\ge3}|1-\omega+\omega\mu_i|}{1-\omega+\omega\mu_2} r ( ω ) = 1 − ω + ω μ 2 m a x i ≥ 3 ∣1 − ω + ω μ i ∣ . For 0 < ω ≤ 1 2 0 < \omega\le\frac{1}{2} 0 < ω ≤ 2 1 every 1 − ω + ω μ i 1-\omega+\omega\mu_i 1 − ω + ω μ i is nonnegative whatever the graph , since μ i ≥ − 1 \mu_i \ge-1 μ i ≥ − 1 ; so ω = 1 2 \omega= \frac{1}{2} ω = 2 1 is the best choice available without knowing the spectrum. If μ 3 \mu_3 μ 3 and μ n \mu_n μ n are known, balancing the two extreme unwanted modes gives ω ⋆ = min { 1 , 2 2 − μ 3 − μ n } \omega_\star= \min\left\{1,\frac{2}{2-\mu_3-\mu_n}\right\} ω ⋆ = min { 1 , 2 − μ 3 − μ n 2 } , which is faster still. The lazy walk is the robust default, not the optimum.
Takeaways
Classical iterations such as Jacobi act as diffusion or averaging processes on graphs.
Their eigenmode analysis shows that, once damped, high-frequency oscillations on the graph are quickly removed while smooth global modes persist.
The slow modes correspond to eigenvectors of the Laplacian with small eigenvalues; these capture global connectivity patterns and reveal community structure.
This perspective connects basic iterative methods to spectral clustering: Jacobi iterations naturally enhance low-frequency information that underlies graph partitioning.
In practice their convergence is too slow to serve as efficient graph solvers, but they remain conceptually valuable as “microscopes” for understanding spectral structure.
Basic Iterative Methods in Machine Learning
The linear algebra perspective on Jacobi, Gauss–Seidel, and SOR extends naturally to optimization and machine learning. At their core, many ML algorithms amount to solving large linear or least-squares problems via iterative schemes. This section shows how the classical iterations appear as special cases of gradient descent, and how their spectral behaviour connects to convergence properties of training.
Richardson iteration and gradient descent
Recall the Richardson scheme for A x = b Ax = b A x = b with step size τ > 0 \tau> 0 τ > 0 :
x ( k + 1 ) = x ( k ) + τ ( b − A x ( k ) ) . x^{(k+1)} = x^{(k)} + \tau\left(b - Ax^{(k)}\right). x ( k + 1 ) = x ( k ) + τ ( b − A x ( k ) ) .
This is exactly gradient descent for the quadratic loss
min x f ( x ) = 1 2 x T A x − b T x \min_x f(x)=\frac{1}{2}x^{\mathsf T}Ax-b^{\mathsf T}x x min f ( x ) = 2 1 x T A x − b T x
since ∇ f ( x ) = A x − b \nabla f(x)=Ax-b ∇ f ( x ) = A x − b . Thus gradient descent on quadratic objectives is nothing but Richardson iteration.
Spectral behaviour. If A A A is symmetric positive definite with eigenpairs ( λ j , v j ) (\lambda_j,v_j) ( λ j , v j ) , then the error evolves as
e ( k ) = ∑ j ( 1 − τ λ j ) k α j v j . e^{(k)}=\sum_j(1-\tau\lambda_j)^k\alpha_jv_j. e ( k ) = j ∑ ( 1 − τ λ j ) k α j v j .
Hence convergence requires 0 < τ < 2 / λ max 0<\tau<2/\lambda_{\max} 0 < τ < 2/ λ m a x , and the rate is controlled by the condition number κ ( A ) = λ max / λ min \kappa(A)=\lambda_{\max}/\lambda_{\min} κ ( A ) = λ m a x / λ m i n [31 ] : the closer eigenvalues are to zero, the slower their modes decay. This is the same “bottleneck” phenomenon seen for smooth modes in PDEs and graph Laplacians.
Figure 1 (PDF p. 46).
Choosing the step size τ \tau τ . Each mode j j j is damped at rate g j ( τ ) = ∣ 1 − τ λ j ∣ g_j(\tau)=|1-\tau\lambda_j| g j ( τ ) = ∣1 − τ λ j ∣ , so the overall (worst-case) rate is q ( τ ) = max j g j ( τ ) q(\tau)=\max_j g_j(\tau) q ( τ ) = max j g j ( τ ) , which for λ j ∈ [ λ min , λ max ] \lambda_j\in[\lambda_{\min},\lambda_{\max}] λ j ∈ [ λ m i n , λ m a x ] reduces to q ( τ ) = max { ∣ 1 − τ λ min ∣ , ∣ 1 − τ λ max ∣ } q(\tau)=\max\{|1-\tau\lambda_{\min}|,|1-\tau\lambda_{\max}|\} q ( τ ) = max { ∣1 − τ λ m i n ∣ , ∣1 − τ λ m a x ∣ } . This is exactly the Richardson-for-SPD-matrices result of Chapter 1: the minimax-optimal step
τ ⋆ = 2 λ min + λ max \tau^\star=\frac{2}{\lambda_{\min}+\lambda_{\max}} τ ⋆ = λ m i n + λ m a x 2
equalizes the two extreme mode gains and gives q ( τ ⋆ ) = κ − 1 κ + 1 q(\tau^\star)=\frac{\kappa-1}{\kappa+1} q ( τ ⋆ ) = κ + 1 κ − 1 , κ = λ max λ min \kappa=\frac{\lambda_{\max}}{\lambda_{\min}} κ = λ m i n λ m a x .
Example 3.3.1 (Optimal step size for a two-eigenvalue toy problem). Suppose A A A (or X T X X^{\mathsf T}X X T X ) has only two distinct eigenvalues, λ min = 1 \lambda_{\min}=1 λ m i n = 1 and λ max = 9 \lambda_{\max}=9 λ m a x = 9 , so κ = 9 \kappa=9 κ = 9 . The optimal step is τ ⋆ = 2 / ( 1 + 9 ) = 0.2 \tau^\star=2/(1+9)=0.2 τ ⋆ = 2/ ( 1 + 9 ) = 0.2 , giving mode gains ∣ 1 − τ ⋆ λ min ∣ = ∣ 1 − 0.2 ∣ = 0.8 |1-\tau^\star\lambda_{\min}|=|1-0.2|=0.8 ∣1 − τ ⋆ λ m i n ∣ = ∣1 − 0.2∣ = 0.8 and ∣ 1 − τ ⋆ λ max ∣ = ∣ 1 − 1.8 ∣ = 0.8 |1-\tau^\star\lambda_{\max}|=|1-1.8|=0.8 ∣1 − τ ⋆ λ m a x ∣ = ∣1 − 1.8∣ = 0.8 : both extreme modes decay at exactly the same rate q ⋆ = ( κ − 1 ) / ( κ + 1 ) = 8 / 10 = 0.8 q^\star=(\kappa-1)/(\kappa+1)=8/10=0.8 q ⋆ = ( κ − 1 ) / ( κ + 1 ) = 8/10 = 0.8 , as predicted. A poorly chosen step, e.g. τ = 2 / λ max = 2 / 9 ≈ 0.222 \tau=2/\lambda_{\max}=2/9\approx0.222 τ = 2/ λ m a x = 2/9 ≈ 0.222 , leaves the λ max \lambda_{\max} λ m a x mode exactly on the boundary of divergence (g = ∣ 1 − 2 ∣ = 1 g=|1-2|=1 g = ∣1 − 2∣ = 1 ) while the λ min \lambda_{\min} λ m i n mode still decays comfortably (g = ∣ 1 − 0.222 ∣ ≈ 0.778 g=|1-0.222|\approx0.778 g = ∣1 − 0.222∣ ≈ 0.778 ) – a reminder that the largest eigenvalue alone does not determine the best step size; both extremes of the spectrum must be balanced.
Connection to Jacobi. Jacobi can be viewed as Richardson with a preconditioner D − 1 D^{-1} D − 1 .
x ( k + 1 ) = x ( k ) + D − 1 ( b − A x ( k ) ) . x^{(k+1)}=x^{(k)}+D^{-1}(b-Ax^{(k)}). x ( k + 1 ) = x ( k ) + D − 1 ( b − A x ( k ) ) .
This interpretation carries over directly to ML: preconditioning accelerates gradient descent by scaling the update direction.
Applications in machine learning
(1) Linear regression. Given data X ∈ R m × n X\in\mathbb{R}^{m\times n} X ∈ R m × n and labels y ∈ R m y\in\mathbb{R}^{m} y ∈ R m , the least squares problem
min w 1 2 ∥ X w − y ∥ 2 \min_w\frac{1}{2}\lVert Xw-y\rVert^2 w min 2 1 ∥ X w − y ∥ 2
leads to the normal equations X T X w = X T y X^{\mathsf T}Xw=X^{\mathsf T}y X T X w = X T y . Richardson/gradient descent updates are
w ( k + 1 ) = w ( k ) − τ ( X T X w ( k ) − X T y ) . w^{(k+1)}=w^{(k)}-\tau(X^{\mathsf T}Xw^{(k)}-X^{\mathsf T}y). w ( k + 1 ) = w ( k ) − τ ( X T X w ( k ) − X T y ) .
The spectrum of X T X X^{\mathsf T}X X T X (squared singular values of X X X ) dictates the convergence speed. If X X X is ill-conditioned, convergence is slow, mirroring the PDE case where low frequencies persist. This motivates preconditioning and methods like stochastic gradient descent (SGD).
(2) Logistic regression and nonlinear losses. For convex but nonlinear losses, e.g.
f ( w ) = ∑ i = 1 m log ( 1 + exp ( − y i x i T w ) ) , f(w) = \sum_{i=1}^{m} \log(1+\exp(-y_i x_i^{\mathsf T}w)), f ( w ) = i = 1 ∑ m log ( 1 + exp ( − y i x i T w )) ,
gradient descent is no longer exactly Richardson, but locally (near the solution) the Hessian H = ∇ 2 f ( w ∗ ) H = \nabla^2 f(w^*) H = ∇ 2 f ( w ∗ ) plays the role of A A A . Convergence is again governed by the spectrum of H H H : flat directions (small eigenvalues) cause slow learning, steep directions (large eigenvalues) require small step sizes for stability.
Smoothing and spectral bias
Just as Jacobi smooths high-frequency error modes in PDEs, gradient descent smooths high-frequency parameter components. In training neural networks this is visible as spectral bias [36 ] : early iterations capture low-frequency/global patterns in the data, while fine details emerge later.
Example 3.3.2 (Smoothing vs. spectral bias). Consider two settings:
(a) Jacobi for the 1D Poisson problem. Let A A A be the Laplacian matrix on n n n grid points. The Jacobi error iteration reads
e ( k + 1 ) = G J e ( k ) , α j ( k ) = μ j k α j ( 0 ) , μ j = cos ( π j n + 1 ) . e^{(k+1)} = G_J e^{(k)}, \qquad\alpha_j^{(k)} = \mu_j^k \alpha_j^{(0)}, \qquad\mu_j = \cos\left(\frac{\pi j}{n+1}\right). e ( k + 1 ) = G J e ( k ) , α j ( k ) = μ j k α j ( 0 ) , μ j = cos ( n + 1 π j ) . The eigenvectors are sine waves with frequency j j j . Here ∣ μ n + 1 − j ∣ = ∣ μ j ∣ \lvert\mu_{n+1-j}\rvert= \lvert\mu_j\rvert ∣ μ n + 1 − j ∣ = ∣ μ j ∣ , so the most oscillatory mode decays exactly as slowly as the smoothest one. Damping repairs this. For G ω = I − ω D − 1 A G_\omega= I-\omega D^{-1}A G ω = I − ω D − 1 A the eigenvalues are
μ j ( ω ) = 1 − 2 ω sin 2 ( π j 2 ( n + 1 ) ) , \mu_j(\omega) = 1 - 2\omega\sin^2\left(\frac{\pi j}{2(n+1)}\right), μ j ( ω ) = 1 − 2 ω sin 2 ( 2 ( n + 1 ) π j ) , and ω = 2 3 \omega=\frac{2}{3} ω = 3 2 gives ∣ μ j ( ω ) ∣ ≤ 1 3 \lvert\mu_j(\omega)\rvert\le\frac{1}{3} ∣ μ j ( ω )∣ ≤ 3 1 for every j ≥ ( n + 1 ) / 2 j \ge(n+1)/2 j ≥ ( n + 1 ) /2 , uniformly in n n n . Thus weighted Jacobi acts as a low-pass smoother : it removes oscillations first and leaves global structure last. This is the property multigrid rests on (Chapter 6).
(b) Gradient descent for least squares regression. Suppose we fit X w ≈ y Xw \approx y X w ≈ y with normal equations X T X w = X T y X^{\mathsf T}Xw = X^{\mathsf T}y X T X w = X T y . Let X T X = V Λ V T X^{\mathsf T}X = V\Lambda V^{\mathsf T} X T X = V Λ V T with eigenpairs ( λ j , v j ) (\lambda_j,v_j) ( λ j , v j ) . Gradient descent with step size τ \tau τ yields
α j ( k ) = ( 1 − τ λ j ) k α j ( 0 ) . \alpha_j^{(k)} = (1-\tau\lambda_j)^k\alpha_j^{(0)}. α j ( k ) = ( 1 − τ λ j ) k α j ( 0 ) . For many data distributions (e.g. Fourier features, random features, kernel regression), it turns out that low-frequency patterns correspond to larger eigenvalues of the kernel or data covariance, while high-frequency patterns correspond to smaller eigenvalues. This phenomenon is known as spectral bias .
Duality. Both Jacobi and gradient descent share the same filtering mechanism: α j ( k ) = g ( λ j ) k α j ( 0 ) \alpha_j^{(k)} = g(\lambda_j)^k\alpha_j^{(0)} α j ( k ) = g ( λ j ) k α j ( 0 ) , where λ j \lambda_j λ j indexes spectral modes. In PDE solvers, large λ j \lambda_j λ j corresponds to high-frequency error and is damped quickly (smoothing). In learning, large λ j \lambda_j λ j corresponds to low-frequency patterns in the data, which are learned quickly. Thus the roles of low vs. high frequency are swapped , but the underlying principle is the same: iterative methods act as frequency filters, with the spectrum of A A A or X T X X^{\mathsf T}X X T X dictating which modes persist.
Exercises
Problem 1 (Closed form of the 1D Gauss–Seidel iteration matrix). Let
A = 1 h 2 tridiag ( − 1 , 2 , − 1 ) ∈ R n × n , A = D − L − U , A = \frac{1}{h^2}\operatorname{tridiag}(-1,2,-1) \in\mathbb{R}^{n \times n}, \qquad A = D-L-U, A = h 2 1 tridiag ( − 1 , 2 , − 1 ) ∈ R n × n , A = D − L − U ,
where the Gauss–Seidel splitting uses nonnegative strict parts
D = 2 h 2 I , L = 1 h 2 tridiag ( 1 , 0 , 0 ) , U = 1 h 2 tridiag ( 0 , 0 , 1 ) . D = \frac{2}{h^2}I, \qquad L = \frac{1}{h^2}\operatorname{tridiag}(1,0,0), \qquad U = \frac{1}{h^2}\operatorname{tridiag}(0,0,1). D = h 2 2 I , L = h 2 1 tridiag ( 1 , 0 , 0 ) , U = h 2 1 tridiag ( 0 , 0 , 1 ) .
The Gauss–Seidel iteration matrix is G G S = ( D − L ) − 1 U G_{GS}=(D-L)^{-1}U G GS = ( D − L ) − 1 U .
Show that G G S G_{GS} G GS is independent of h h h and can be written as
G G S = B − 1 S + , B : = 2 I − S − , G_{GS}=B^{-1}S_{+}, \qquad B:=2I-S_{-}, G GS = B − 1 S + , B := 2 I − S − ,
where S − S_{-} S − has ones on the subdiagonal and S + S_{+} S + has ones on the superdiagonal.
Compute B − 1 B^{-1} B − 1 explicitly and deduce a closed form for the entries of G G S G_{GS} G GS .
Write the first few rows of G G S G_{GS} G GS to display its “plain matrix” pattern.
Problem 2 (Error components for Jacobi). Let A A A be the 1D Poisson matrix with n n n interior points and G J G_J G J the Jacobi iteration matrix.
(a) Show that the error e ( k ) = x ( k ) − x ∗ e^{(k)}=x^{(k)}-x^* e ( k ) = x ( k ) − x ∗ can be expanded in the eigenbasis
e ( k ) = ∑ j = 1 n α j ( k ) v ( j ) , α j ( k + 1 ) = μ j α j ( k ) , e^{(k)}=\sum_{j=1}^{n}\alpha_j^{(k)}v^{(j)}, \qquad\alpha_j^{(k+1)}=\mu_j\alpha_j^{(k)}, e ( k ) = j = 1 ∑ n α j ( k ) v ( j ) , α j ( k + 1 ) = μ j α j ( k ) ,
with eigenvalues μ j = cos ( j π n + 1 ) \mu_j=\cos\left(\frac{j\pi}{n+1}\right) μ j = cos ( n + 1 j π ) .
(b) Which modes (low j j j or high j j j ) have eigenvalues closest to 1?
(c) Deduce that undamped Jacobi is not a smoother, and that ω = 2 3 \omega=\frac{2}{3} ω = 3 2 in G ω = I − ω D − 1 A G_\omega=I-\omega D^{-1}A G ω = I − ω D − 1 A gives
∣ μ j ( ω ) ∣ ≤ 1 3 for all j ≥ ( n + 1 ) / 2. |\mu_j(\omega)|\leq\frac{1}{3}\quad\text{for all }j\geq(n+1)/2. ∣ μ j ( ω ) ∣ ≤ 3 1 for all j ≥ ( n + 1 ) /2.
Preamble (for Problems 3 and 4, exam 2021/2022). For a 0 , a − 1 , a 1 ∈ R a_0,a_{-1},a_1\in\mathbb{R} a 0 , a − 1 , a 1 ∈ R with a − 1 ⋅ a 1 > 0 a_{-1}\cdot a_1>0 a − 1 ⋅ a 1 > 0 the Toeplitz tridiagonal matrix
A = [ a 0 a 1 a − 1 a 0 a 1 ⋱ ⋱ ⋱ a − 1 a 0 a 1 a − 1 a 0 ] ∈ R n × n A=
\begin{bmatrix}
a_0 & a_1 \\
a_{-1} & a_0 & a_1 \\
& \ddots& \ddots& \ddots\\
&& a_{-1} & a_0 & a_1 \\
&&& a_{-1} & a_0
\end{bmatrix}
\in\mathbb{R}^{n\times n} A = a 0 a − 1 a 1 a 0 ⋱ a 1 ⋱ a − 1 ⋱ a 0 a − 1 a 1 a 0 ∈ R n × n
has eigenvalues
λ p = a 0 + 2 sgn ( a − 1 ) a − 1 a 1 cos ( p π n + 1 ) , p = 1 , … , n , \lambda_p=a_0+2\operatorname{sgn}(a_{-1})\sqrt{a_{-1}a_1}\cos\left(\frac{p\pi}{n+1}\right), \qquad p=1,\ldots,n, λ p = a 0 + 2 sgn ( a − 1 ) a − 1 a 1 cos ( n + 1 p π ) , p = 1 , … , n ,
and associated eigenvectors
s p = ( a − 1 a 1 sin ( i p π n + 1 ) ) i = 1 n . s_p=\left(\sqrt{\frac{a_{-1}}{a_1}}\sin\left(\frac{ip\pi}{n+1}\right)\right)_{i=1}^{n}. s p = ( a 1 a − 1 sin ( n + 1 i p π ) ) i = 1 n .
If A A A is symmetric (a − 1 = a 1 a_{-1}=a_1 a − 1 = a 1 ), then the eigenvectors form an orthogonal basis. Moreover, for any matrix B B B ,
lim k → ∞ B k = 0 ⟺ ρ ( B ) < 1 , \lim_{k\to\infty}B^k=0 \qquad\Longleftrightarrow\qquad\rho(B)<1, k → ∞ lim B k = 0 ⟺ ρ ( B ) < 1 ,
where ρ ( B ) \rho(B) ρ ( B ) denotes the spectral radius.
Exercises
Problem 3 (Tridiagonal Laplace matrix, exam 2021/2022). Consider the symmetric tridiagonal matrix
A = [ 2 − 1 − 1 2 − 1 ⋱ ⋱ ⋱ − 1 2 − 1 − 1 2 ] ∈ R n × n . A=\begin{bmatrix}
2 & -1 \\
-1 & 2 & -1 \\
& \ddots& \ddots& \ddots\\
& & -1 & 2 & -1 \\
& & & -1 & 2
\end{bmatrix}\in\mathbb{R}^{n\times n}. A = 2 − 1 − 1 2 ⋱ − 1 ⋱ − 1 ⋱ 2 − 1 − 1 2 ∈ R n × n .
Assume n n n is odd.
(a) Show that the vector b ∈ R n b\in\mathbb{R}^{n} b ∈ R n with entries
b 2 k − 1 = ( − 1 ) k − 1 , b 2 k = 0 , b_{2k-1}=(-1)^{k-1},\qquad b_{2k}=0, b 2 k − 1 = ( − 1 ) k − 1 , b 2 k = 0 ,
that is b = [ 1 , 0 , − 1 , 0 , 1 , 0 , − 1 , … ] T b=[1,0,-1,0,1,0,-1,\ldots]^{\mathsf{T}} b = [ 1 , 0 , − 1 , 0 , 1 , 0 , − 1 , … ] T , is an eigenvector of A A A . (Its last entry is + 1 +1 + 1 only when n ≡ 1 ( m o d 4 ) n\equiv1\pmod{4} n ≡ 1 ( mod 4 ) .)
(b) Show that all eigenvalues of A A A lie in the interval ( 0 , 4 ) (0,4) ( 0 , 4 ) .
Problem 4 (Jacobi’s method and convergence, exam 2021/2022). Let A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n be nonsingular and let D = diag ( A ) D=\operatorname{diag}(A) D = diag ( A ) . Jacobi’s iteration for solving A x = b Ax=b A x = b is
x ( k + 1 ) = ( I − D − 1 A ) x ( k ) + D − 1 b , k = 0 , 1 , 2 , … (4) x^{(k+1)}=(I-D^{-1}A)x^{(k)}+D^{-1}b,\qquad k=0,1,2,\ldots \tag*{(4)} x ( k + 1 ) = ( I − D − 1 A ) x ( k ) + D − 1 b , k = 0 , 1 , 2 , … (4)
Let B = I − D − 1 A B=I-D^{-1}A B = I − D − 1 A .
(a) Show that if x ^ = lim k → ∞ x ( k ) \hat{x}=\lim_{k\to\infty}x^{(k)} x ^ = lim k → ∞ x ( k ) exists, then Jacobi’s method converges if and only if A A A has no zeros on its diagonal and ρ ( B ) < 1 \rho(B)<1 ρ ( B ) < 1 .
(b) Suppose A A A is the Toeplitz tridiagonal matrix above. Give necessary and sufficient conditions on ( a − 1 , a 0 , a 1 ) (a_{-1},a_0,a_1) ( a − 1 , a 0 , a 1 ) independent of n n n for Jacobi’s method to converge.
(c) Consider the special case
A = [ a 0 a 1 a 0 a 0 a 1 ⋱ ⋱ ⋱ a 0 a 0 a 1 a 0 a 0 ] ∈ R n × n . A=\begin{bmatrix}
a_0 & a_1 \\
a_0 & a_0 & a_1 \\
& \ddots& \ddots& \ddots\\
& & a_0 & a_0 & a_1 \\
& & & a_0 & a_0
\end{bmatrix}\in\mathbb{R}^{n\times n}. A = a 0 a 0 a 1 a 0 ⋱ a 1 ⋱ a 0 ⋱ a 0 a 0 a 1 a 0 ∈ R n × n .
What are the necessary and sufficient conditions on a 0 , a 1 a_0,a_1 a 0 , a 1 (independent of n n n ) for ρ ( B ) < 1 \rho(B)<1 ρ ( B ) < 1 ?
(d) Let A A A be as in (c) with a 0 = 1 a_0=1 a 0 = 1 and a 1 > 0 a_1>0 a 1 > 0 (so the preamble’s formula applies to B B B ), b = e n b=e_n b = e n , and initial guess x ( 0 ) = 0 x^{(0)}=0 x ( 0 ) = 0 . Does there exist a Jacobi iterate x ( k ) x^{(k)} x ( k ) that solves A x = b Ax=b A x = b exactly?
Problem 5 (Numerical verification of convergence rates). For the 1D Poisson matrix with n = 19 n=19 n = 19 interior points:
(a) Compute ρ ( G J ) \rho(G_J) ρ ( G J ) , ρ ( G G S ) \rho(G_{GS}) ρ ( G GS ) , and ω o p t \omega_{\mathrm{opt}} ω opt , ρ ( G ω o p t ) \rho(G_{\omega_{\mathrm{opt}}}) ρ ( G ω opt ) using the closed-form expressions derived in this chapter.
(b) Estimate the number of iterations each method needs to reduce the initial error by a factor 10 − 8 10^{-8} 1 0 − 8 .
(c) Implement Jacobi, Gauss–Seidel, and SOR (with ω = ω o p t \omega=\omega_{\mathrm{opt}} ω = ω opt ) for this system and confirm numerically that the observed asymptotic convergence factors match your predictions from (a). (See week3/script_week3_1.ipynb in the course code repository.)
Problem 6 (Spectrum of a small path graph). Let P 5 P_5 P 5 be the path graph on 5 vertices with graph Laplacian L L L (as constructed in Chapter 2).
(a) Write down L L L explicitly and compute its eigenvalues and eigenvectors directly (by hand or numerically).
(b) Verify that they match the general formula λ k ( L ) = 2 − 2 cos ( π k / n ) \lambda_k(L) = 2 - 2\cos(\pi k/n) λ k ( L ) = 2 − 2 cos ( π k / n ) for the path graph.
(c) Identify the Fiedler vector (the eigenvector for the smallest nonzero eigenvalue) and describe the bipartition it induces on the 5 vertices.
Problem 7 (Gradient descent convergence rate from the data spectrum). Consider a least-squares problem min w 1 2 ∥ X w − y ∥ 2 \min_w \frac{1}{2}\lVert Xw-y\rVert^2 min w 2 1 ∥ X w − y ∥ 2 where X ∈ R m × n X \in\mathbb{R}^{m\times n} X ∈ R m × n has singular values σ 1 = 4 \sigma_1 = 4 σ 1 = 4 , σ 2 = 2 \sigma_2 = 2 σ 2 = 2 , σ 3 = 1 \sigma_3 = 1 σ 3 = 1 (so X T X X^\mathsf{T}X X T X has eigenvalues 16 , 4 , 1 16,4,1 16 , 4 , 1 ).
(a) Compute the condition number κ ( X T X ) \kappa(X^\mathsf{T}X) κ ( X T X ) and the largest admissible step size τ max = 2 / λ max \tau_{\max} = 2/\lambda_{\max} τ m a x = 2/ λ m a x for gradient descent on X T X w = X T y X^\mathsf{T}Xw = X^\mathsf{T}y X T X w = X T y .
(b) Compute the minimax-optimal step size τ ⋆ \tau^\star τ ⋆ and the resulting worst-case convergence factor q ⋆ = ( κ − 1 ) / ( κ + 1 ) q^\star= (\kappa- 1)/(\kappa+ 1) q ⋆ = ( κ − 1 ) / ( κ + 1 ) .
(c) Using τ ⋆ \tau^\star τ ⋆ , estimate how many iterations are needed to reduce the worst-case mode’s error by a factor of 100.
(d) Which mode (associated with σ 1 \sigma_1 σ 1 , σ 2 \sigma_2 σ 2 , or σ 3 \sigma_3 σ 3 ) is learned fastest, and which slowest? Relate your answer to the notion of spectral bias .
Problem 8 (Convergence of the power method). Let M ∈ R n × n M \in\mathbb{R}^{n\times n} M ∈ R n × n be symmetric, with orthonormal eigenvectors v 1 , … , v n v_1,\ldots,v_n v 1 , … , v n and eigenvalues ordered by modulus, ∣ λ 1 ∣ > ∣ λ 2 ∣ ≥ ⋯ ≥ ∣ λ n ∣ \lvert\lambda_1\rvert> \lvert\lambda_2\rvert\ge\cdots\ge\lvert\lambda_n\rvert ∣ λ 1 ∣ > ∣ λ 2 ∣ ≥ ⋯ ≥ ∣ λ n ∣ . Run Algorithm 1 from y ( 0 ) = ∑ i = 1 n α i v i y^{(0)} = \sum_{i=1}^{n}\alpha_i v_i y ( 0 ) = ∑ i = 1 n α i v i with ∥ y ( 0 ) ∥ 2 = 1 \lVert y^{(0)}\rVert_2 = 1 ∥ y ( 0 ) ∥ 2 = 1 and α 1 ≠ 0 \alpha_1 \ne0 α 1 = 0 , and let θ k ∈ [ 0 , π / 2 ] \theta_k \in[0,\pi/2] θ k ∈ [ 0 , π /2 ] be the angle between y ( k ) y^{(k)} y ( k ) and the line spanned by v 1 v_1 v 1 .
(a) Show that rescaling at every step is the same as rescaling once at the end, i.e.
y ( k ) = M k y ( 0 ) ∥ M k y ( 0 ) ∥ 2 , so M k y ( 0 ) = ∑ i = 1 n α i λ i k v i . y^{(k)} = \frac{M^k y^{(0)}}{\lVert M^k y^{(0)}\rVert_2},
\qquad\text{so}\qquad
M^k y^{(0)} = \sum_{i=1}^{n}\alpha_i\lambda_i^k v_i. y ( k ) = ∥ M k y ( 0 ) ∥ 2 M k y ( 0 ) , so M k y ( 0 ) = i = 1 ∑ n α i λ i k v i .
(b) Splitting this sum into its component along v 1 v_1 v 1 and the rest, deduce that
tan θ k ≤ ( ∑ i ≥ 2 α i 2 ) 1 / 2 ∣ α 1 ∣ ∣ λ 2 λ 1 ∣ k , \tan\theta_k \le\frac{\left(\sum_{i\ge2}\alpha_i^2\right)^{1/2}}{\lvert\alpha_1\rvert}\left\lvert\frac{\lambda_2}{\lambda_1}\right\rvert^k, tan θ k ≤ ∣ α 1 ∣ ( ∑ i ≥ 2 α i 2 ) 1/2 λ 1 λ 2 k ,
so that y ( k ) → ± v 1 y^{(k)} \to\pm v_1 y ( k ) → ± v 1 geometrically with ratio ∣ λ 2 / λ 1 ∣ \lvert\lambda_2/\lambda_1\rvert ∣ λ 2 / λ 1 ∣ .
(c) Writing c i = ( y ( k ) ) T v i c_i = (y^{(k)})^\mathsf{T}v_i c i = ( y ( k ) ) T v i , show that μ k = ∑ i λ i c i 2 \mu_k = \sum_i \lambda_i c_i^2 μ k = ∑ i λ i c i 2 and hence that
∣ μ k − λ 1 ∣ ≤ ( λ max − λ min ) sin 2 θ k . \lvert\mu_k-\lambda_1\rvert\le(\lambda_{\max}-\lambda_{\min})\sin^2\theta_k. ∣ μ k − λ 1 ∣ ≤ ( λ m a x − λ m i n ) sin 2 θ k .
Conclude that the eigenvalue converges at rate ∣ λ 2 / λ 1 ∣ 2 k \lvert\lambda_2/\lambda_1\rvert^{2k} ∣ λ 2 / λ 1 ∣ 2 k , twice as fast as the eigenvector.
(d) Take the 1D Poisson matrix A = tridiag ( − 1 , 2 , − 1 ) A=\operatorname{tridiag}(-1,2,-1) A = tridiag ( − 1 , 2 , − 1 ) with n = 19 n=19 n = 19 , whose eigenvalues are
λ j = 2 − 2 cos ( j π 20 ) . \lambda_j = 2 - 2\cos\left(\frac{j\pi}{20}\right). λ j = 2 − 2 cos ( 20 j π ) .
Identify the two largest in modulus, compute the ratio ∣ λ 2 / λ 1 ∣ \lvert\lambda_2/\lambda_1\rvert ∣ λ 2 / λ 1 ∣ , and estimate how many power iterations are needed to reduce the eigenvector error by a factor 10 − 2 10^{-2} 1 0 − 2 . Relate the answer to the discussion of smoothing in § 5.2 .
(e) For a connected graph P = D − 1 A P=D^{-1}A P = D − 1 A has dominant eigenvalue 1 with eigenvector 1 \mathbf{1} 1 . Show that deflating 1 \mathbf{1} 1 is not enough: on P 4 P_4 P 4 , q = ( 1 , − 1 , 1 , − 1 ) T q=(1,-1,1,-1)^\mathsf{T} q = ( 1 , − 1 , 1 , − 1 ) T has P q = − q Pq=-q P q = − q and q T D 1 = 0 q^\mathsf{T}D\mathbf{1}=0 q T D 1 = 0 . Show that the lazy walk 1 2 ( I + P ) \frac{1}{2}(I+P) 2 1 ( I + P ) repairs this, and state the assumption on μ 2 \mu_2 μ 2 needed. (See week3/script_week3_3.ipynb.)
Krylov Methods for Symmetric Systems
Overview
Chapters 1 and 3 built iterative solvers from a fixed matrix splitting A = M − N A = M - N A = M − N : every step applies the same operator G = M − 1 N G = M^{-1}N G = M − 1 N , so the error can be reduced by at best a fixed factor ρ ( G ) \rho(G) ρ ( G ) per iteration. For the PDE model matrices of Chapter 3 that factor behaves like 1 − O ( h 2 ) 1 - O(h^2) 1 − O ( h 2 ) , which makes stationary methods unusually slow as the mesh is refined. This chapter replaces that fixed recipe by an adaptive one. At step k k k , a Krylov subspace method relies on the subspace K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } \mathcal{K}_k(A, r^{(0)}) = \operatorname{span}\{r^{(0)}, Ar^{(0)}, \ldots, A^{k-1}r^{(0)}\} K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } and then selects the best approximation in the affine space x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)} + \mathcal{K}_k(A, r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) . For symmetric positive definite (SPD) A A A , this optimal choice can be computed with short recurrences and constant storage: this is the Conjugate Gradient (CG) method [22 ] , and its convergence is governed by κ ( A ) \sqrt{\kappa(A)} κ ( A ) instead of κ ( A ) \kappa(A) κ ( A ) .
This chapter covers: (1) CG, its optimality properties and its Chebyshev convergence bound; (2) the Lanczos algorithm, including the fact that CG and Lanczos are one recurrence seen from two sides, and its use to compute the Fiedler vector of a graph; (3) machine learning, where we show that ridge regression, truncated SVD and CG are all spectral filters of the same data, so that stopping CG early is itself a regularisation.
By the end of this chapter, you should be able to:
Derive the Conjugate Gradient method from the minimisation of the quadratic functional J ( x ) = 1 2 x T A x − b T x J(x) = \frac{1}{2}x^{\mathsf{T}}Ax - b^{\mathsf{T}}x J ( x ) = 2 1 x T A x − b T x , and explain why the steepest descent is not enough.
State and use the optimality property of CG: the iterate x ( k ) x^{(k)} x ( k ) minimises the A A A -norm of the error over x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)} + \mathcal{K}_k(A, r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) .
Use the Chebyshev bound to relate the CG convergence rate to κ ( A ) \sqrt{\kappa(A)} κ ( A ) , and contrast this with the κ ( A ) \kappa(A) κ ( A ) dependence of steepest descent and stationary methods.
Describe Lanczos iteration and explain why Ritz values converge at the ends of the spectrum first, and use deflation to compute the Fiedler vector of a graph Laplacian.
Explain the CG-Lanczos equivalence, and interpret ridge regression and truncated CG as spectral filters - so that early stopping is itself a regularisation.
Explain what semi-convergence is, and why some stopping rule is still needed when the data are noisy.
The Conjugate Gradient method
We now move from stationary iterations such as Jacobi, Gauss–Seidel, and SOR to Krylov subspace methods. These methods generate successively richer approximation spaces and the most famous example is the Conjugate Gradient (CG) method. For symmetric positive definite (SPD) matrices, such as those arising from elliptic PDEs, CG achieves convergence far superior to classical schemes. We wish to solve A x = b Ax=b A x = b with A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n a SPD matrix. In this case, it makes sense to consider the quadratic functional
J ( x ) = 1 2 x T A x − b T x . J(x)=\frac{1}{2}x^{\mathsf{T}}Ax-b^{\mathsf{T}}x. J ( x ) = 2 1 x T A x − b T x .
Minimising J ( x ) J(x) J ( x ) is equivalent to solving ∇ J = A x − b = 0 \nabla J=Ax-b=0 ∇ J = A x − b = 0 hence the linear system A x = b Ax=b A x = b .
Definition 4.1.1 (Steepest descent method). Starting from an initial guess x 0 x^0 x 0 , the steepest descent method generates iterates
x ( k + 1 ) = x ( k ) + τ k r ( k ) , r ( k ) = b − A x ( k ) , x^{(k+1)}=x^{(k)}+\tau_k r^{(k)},\qquad r^{(k)}=b-Ax^{(k)}, x ( k + 1 ) = x ( k ) + τ k r ( k ) , r ( k ) = b − A x ( k ) , where the step size τ k \tau_k τ k is chosen to minimize J J J along the search direction opposite to the gradient of J J J i.e. r ( k ) r^{(k)} r ( k ) .
Let’s compute the formula of the step length in the steepest descent method. With the notations above, let ϕ ( τ ) = J ( x ( k ) + τ r ( k ) ) \phi(\tau)=J\left(x^{(k)}+\tau r^{(k)}\right) ϕ ( τ ) = J ( x ( k ) + τ r ( k ) ) . Then
ϕ ′ ( τ ) = ( r ( k ) ) T ( A x ( k ) − b ) + τ ( r ( k ) ) T A r ( k ) = − ∥ r ( k ) ∥ 2 + τ ( r ( k ) ) T A r ( k ) . \phi'(\tau)=\left(r^{(k)}\right)^{\mathsf{T}}\left(Ax^{(k)}-b\right)+\tau\left(r^{(k)}\right)^{\mathsf{T}}Ar^{(k)}=-\lVert r^{(k)}\rVert^2+\tau\left(r^{(k)}\right)^{\mathsf{T}}Ar^{(k)}. ϕ ′ ( τ ) = ( r ( k ) ) T ( A x ( k ) − b ) + τ ( r ( k ) ) T A r ( k ) = − ∥ r ( k ) ∥ 2 + τ ( r ( k ) ) T A r ( k ) .
Setting ϕ ′ ( τ ) = 0 \phi'(\tau)=0 ϕ ′ ( τ ) = 0 gives
τ k = ( r ( k ) ) T r ( k ) ( r ( k ) ) T A r ( k ) = ∥ r ( k ) ∥ 2 ∥ r ( k ) ∥ A 2 , \tau_k=\frac{\left(r^{(k)}\right)^{\mathsf{T}}r^{(k)}}{\left(r^{(k)}\right)^{\mathsf{T}}Ar^{(k)}}=\frac{\lVert r^{(k)}\rVert^2}{\lVert r^{(k)}\rVert_A^2}, τ k = ( r ( k ) ) T A r ( k ) ( r ( k ) ) T r ( k ) = ∥ r ( k ) ∥ A 2 ∥ r ( k ) ∥ 2 ,
which is equivalent to minimising J J J along the direction r ( k ) r^{(k)} r ( k ) .
Proposition 4.1.2 (Convergence of steepest descent). Let x ∗ x^* x ∗ be the exact solution of A x = b Ax=b A x = b and let κ ( A ) = λ max ( A ) / λ min ( A ) \kappa(A)=\lambda_{\max}(A)/\lambda_{\min}(A) κ ( A ) = λ m a x ( A ) / λ m i n ( A ) denote the condition number of A A A . Then the steepest descent iterates satisfy
∥ x ( k ) − x ∗ ∥ A ≤ ( κ ( A ) − 1 κ ( A ) + 1 ) k ∥ x ( 0 ) − x ∗ ∥ A , \lVert x^{(k)}-x^*\rVert_A\leq\left(\frac{\kappa(A)-1}{\kappa(A)+1}\right)^k\lVert x^{(0)}-x^*\rVert_A, ∥ x ( k ) − x ∗ ∥ A ≤ ( κ ( A ) + 1 κ ( A ) − 1 ) k ∥ x ( 0 ) − x ∗ ∥ A , where ∥ v ∥ A 2 = v T A v \lVert v\rVert_A^2=v^{\mathsf{T}}Av ∥ v ∥ A 2 = v T A v (norm induced by A A A ).
Sketch of proof. Let e ( k ) = x ( k ) − x ∗ e^{(k)}=x^{(k)}-x^* e ( k ) = x ( k ) − x ∗ and r ( k ) = b − A x ( k ) = − A e ( k ) r^{(k)}=b-Ax^{(k)}=-Ae^{(k)} r ( k ) = b − A x ( k ) = − A e ( k ) . A steepest descent step gives
e ( k + 1 ) = e ( k ) + τ k r ( k ) . e^{(k+1)}=e^{(k)}+\tau_k r^{(k)}. e ( k + 1 ) = e ( k ) + τ k r ( k ) .
Expanding the A A A -norm of the error yields
∥ e ( k + 1 ) ∥ A 2 = ∥ e ( k ) ∥ A 2 − τ k ∥ r ( k ) ∥ 2 , \lVert e^{(k+1)}\rVert_A^2=\lVert e^{(k)}\rVert_A^2-\tau_k\lVert r^{(k)}\rVert^2, ∥ e ( k + 1 ) ∥ A 2 = ∥ e ( k ) ∥ A 2 − τ k ∥ r ( k ) ∥ 2 ,
where we used the optimal choice of τ k \tau_k τ k . Dividing by ∥ e ( k ) ∥ A 2 \lVert e^{(k)}\rVert_A^2 ∥ e ( k ) ∥ A 2 and substituting the formula of τ k \tau_k τ k gives
∥ e ( k + 1 ) ∥ A 2 ∥ e ( k ) ∥ A 2 = 1 − ∥ r ( k ) ∥ 4 ( r ( k ) ) T A r ( k ) ( r ( k ) ) T A − 1 r ( k ) . \frac{\lVert e^{(k+1)}\rVert_A^2}{\lVert e^{(k)}\rVert_A^2}=1-\frac{\lVert r^{(k)}\rVert^4}{\left(r^{(k)}\right)^{\mathsf{T}}Ar^{(k)}\left(r^{(k)}\right)^{\mathsf{T}}A^{-1}r^{(k)}}. ∥ e ( k ) ∥ A 2 ∥ e ( k + 1 ) ∥ A 2 = 1 − ( r ( k ) ) T A r ( k ) ( r ( k ) ) T A − 1 r ( k ) ∥ r ( k ) ∥ 4 .
The denominator is bounded below via the Kantorovich inequality 1 , which relates ( v T A v ) ( v T A − 1 v ) (v^{\mathsf{T}}Av)(v^{\mathsf{T}}A^{-1}v) ( v T A v ) ( v T A − 1 v ) to the condition number κ ( A ) \kappa(A) κ ( A ) . Inserting this bound yields
∥ e ( k + 1 ) ∥ A ∥ e ( k ) ∥ A ≤ κ ( A ) − 1 κ ( A ) + 1 . \frac{\lVert e^{(k+1)}\rVert_A}{\lVert e^{(k)}\rVert_A} \le\frac{\kappa(A)-1}{\kappa(A)+1}. ∥ e ( k ) ∥ A ∥ e ( k + 1 ) ∥ A ≤ κ ( A ) + 1 κ ( A ) − 1 .
By induction over k k k , the stated result follows.
Figure 4.1. Steepest descent versus Conjugate Gradient on the level sets of J ( x ) = 1 2 x T A x − b T x J(x)=\frac{1}{2}x^{\mathsf{T}}Ax-b^{\mathsf{T}}x J ( x ) = 2 1 x T A x − b T x with A = diag ( 1 , 9 ) A=\operatorname{diag}(1,9) A = diag ( 1 , 9 ) and the same starting point (iterates computed exactly). Because consecutive steepest-descent residuals are orthogonal, the iterates zig-zag across the valley and creep towards x ∗ x^* x ∗ ; the number of steps grows like κ ( A ) \kappa(A) κ ( A ) , so the picture gets dramatically worse as the contours become more elongated. CG replaces orthogonality of the residuals by A A A -conjugacy of the search directions: no direction is ever revisited, and in exact arithmetic n n n steps suffice.
When Hestenes and Stiefel introduced Conjugate Gradient in 1952, computing was still in its infancy: machines like the SWAC in Los Angeles (then fastest computer in the world) had memory measured in kilobytes and could barely store a few thousand numbers. Steepest descent was well known, but its zig-zagging inefficiency was frustrating. The breakthrough of CG was to design successive directions that “remember” past progress and never waste effort repeating the same zig-zag. This made it one of the first practical Krylov methods and a cornerstone of numerical linear algebra.
The idea of Conjugate Gradient (CG), developed by Hestenes and Stiefel [22 ] , is to construct search directions p ( k ) p^{(k)} p ( k ) that are not only descent directions but are A A A -orthogonal. Using such conjugate directions ensures that each step eliminates error components independently, guaranteeing convergence in at most n n n steps in exact arithmetic.
Definition 4.1.4 (Conjugate directions). Let A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n be SPD. Two vectors p , q ∈ R n p,q \in\mathbb{R}^n p , q ∈ R n are said to be A A A -conjugate if
p T A q = 0. p^{\mathsf{T}}Aq = 0. p T A q = 0. A family of vectors { p ( 0 ) , … , p ( k ) } \{p^{(0)},\ldots,p^{(k)}\} { p ( 0 ) , … , p ( k ) } is called mutually A A A -conjugate if every pair is A A A -conjugate. Such directions play the role of “orthogonal axes” with respect to the A A A -inner product ⟨ u , v ⟩ A = u T A v \langle u,v\rangle_A = u^{\mathsf{T}}Av ⟨ u , v ⟩ A = u T A v .
Definition 4.1.5 (Krylov subspaces). Given an initial residual r ( 0 ) = b − A x ( 0 ) r^{(0)} = b - Ax^{(0)} r ( 0 ) = b − A x ( 0 ) , the k k k -th Krylov subspace is
K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } . \mathcal{K}_k(A,r^{(0)}) = \operatorname{span}\{r^{(0)},Ar^{(0)},\ldots,A^{k-1}r^{(0)}\}. K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } . Equivalently,
K k ( A , r ( 0 ) ) = { q ( A ) r ( 0 ) : q ∈ Π k − 1 } , \mathcal{K}_k(A,r^{(0)}) = \{q(A)r^{(0)} : q \in\Pi_{k-1}\}, K k ( A , r ( 0 ) ) = { q ( A ) r ( 0 ) : q ∈ Π k − 1 } , where Π k − 1 \Pi_{k-1} Π k − 1 denotes the set of polynomials of degree at most k − 1 k-1 k − 1 .
Definition 4.1.6 (Conjugate Gradient method). Let A x = b Ax=b A x = b with A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n be SPD and the right-hand side b ∈ R n b \in\mathbb{R}^n b ∈ R n . Starting from an initial guess x ( 0 ) x^{(0)} x ( 0 ) and the initial descent direction p ( 0 ) = r ( 0 ) p^{(0)}=r^{(0)} p ( 0 ) = r ( 0 ) , the Conjugate Gradient (CG) generates a sequence of approximations { x ( k ) } \{x^{(k)}\} { x ( k ) } by choosing search directions p ( k ) ∈ K k + 1 ( A , r ( 0 ) ) p^{(k)} \in\mathcal{K}_{k+1}(A,r^{(0)}) p ( k ) ∈ K k + 1 ( A , r ( 0 ) ) iteratively as follows:
x ( k + 1 ) = x ( k ) + α k p ( k ) , α k = r ( k ) T r ( k ) p ( k ) T A p ( k ) , r ( k + 1 ) = r ( k ) − α k A p ( k ) , β k = r ( k + 1 ) T r ( k + 1 ) r ( k ) T r ( k ) , p ( k + 1 ) = r ( k + 1 ) + β k p ( k ) . \begin{aligned}
x^{(k+1)} &= x^{(k)}+\alpha_kp^{(k)}, & \alpha_k &= \frac{r^{(k)\mathsf{T}}r^{(k)}}{p^{(k)\mathsf{T}}Ap^{(k)}},\\
r^{(k+1)} &= r^{(k)}-\alpha_kAp^{(k)}, & \beta_k &= \frac{r^{(k+1)\mathsf{T}}r^{(k+1)}}{r^{(k)\mathsf{T}}r^{(k)}},\\
p^{(k+1)} &= r^{(k+1)}+\beta_kp^{(k)}.
\end{aligned} x ( k + 1 ) r ( k + 1 ) p ( k + 1 ) = x ( k ) + α k p ( k ) , = r ( k ) − α k A p ( k ) , = r ( k + 1 ) + β k p ( k ) . α k β k = p ( k ) T A p ( k ) r ( k ) T r ( k ) , = r ( k ) T r ( k ) r ( k + 1 ) T r ( k + 1 ) ,
Proposition 4.1.7 (Properties of CG). Let A A A be SPD. Then the iterates of CG satisfy:
Residuals are mutually orthogonal: r ( i ) T r ( j ) = 0 r^{(i)\mathsf{T}}r^{(j)}=0 r ( i ) T r ( j ) = 0 for i ≠ j i\ne j i = j .
Search directions are A A A -orthogonal: p ( i ) T A p ( j ) = 0 p^{(i)\mathsf{T}}Ap^{(j)}=0 p ( i ) T A p ( j ) = 0 for i ≠ j i\ne j i = j .
x ( k ) x^{(k)} x ( k ) minimizes ∥ x − x ∗ ∥ A \lVert x-x^*\rVert_A ∥ x − x ∗ ∥ A over the affine space x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)}+\mathcal{K}_k(A,r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) .
Figure 4.2. The optimality property of CG. The ellipses are level sets of the energy norm ∥ x − x ∗ ∥ A \lVert x-x^*\rVert_A ∥ x − x ∗ ∥ A . At step k k k the iterate is the point of the affine space x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)}+\mathcal{K}_k(A,r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) (dashed line) closest to x ∗ x^* x ∗ in the A A A -norm, i.e. the A A A -orthogonal projection of x ∗ x^* x ∗ onto it.
Definition 4.1.8 (Minimal polynomial, and minimal polynomial relative to a vector). The minimal polynomial of A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n is the monic polynomial m m m of smallest degree with m ( A ) = 0 m(A)=0 m ( A ) = 0 .
Given also v ≠ 0 v \ne0 v = 0 , the minimal polynomial of A A A relative to v v v is the monic polynomial m v m_v m v of smallest degree with m v ( A ) v = 0 m_v(A)v=0 m v ( A ) v = 0 . It divides m m m , so d v : = deg m v ≤ deg m ≤ n d_v := \deg m_v \le\deg m \le n d v := deg m v ≤ deg m ≤ n , often strictly. Equivalently d v d_v d v is the largest k k k for which v , A v , … , A k − 1 v v, Av, \ldots, A^{k-1}v v , A v , … , A k − 1 v are linearly independent, so dim K k ( A , v ) = min ( k , d v ) \dim\mathcal{K}_k(A,v)=\min(k,d_v) dim K k ( A , v ) = min ( k , d v ) : the Krylov spaces stop growing at k = d v k=d_v k = d v . For symmetric A A A , take an orthonormal basis of eigenvectors v 1 , … , v n v_1,\ldots,v_n v 1 , … , v n , A v j = λ j v j Av_j=\lambda_jv_j A v j = λ j v j , and expand v = ∑ j α j v j v=\sum_j\alpha_jv_j v = ∑ j α j v j with α j = v j T v \alpha_j=v_j^{T}v α j = v j T v : then m v ( λ ) = ∏ μ ∈ Λ v ( λ − μ ) m_v(\lambda)=\prod_{\mu\in\Lambda_v}(\lambda-\mu) m v ( λ ) = ∏ μ ∈ Λ v ( λ − μ ) , where Λ v = { λ j : α j ≠ 0 } \Lambda_v=\{\lambda_j:\alpha_j\ne0\} Λ v = { λ j : α j = 0 } is the set of eigenvalues present in v v v , a repeated eigenvalue counting once however many of its α j \alpha_j α j are nonzero. So d v d_v d v is simply the number of eigenvalues present in v v v , one fewer for every eigenvalue of A A A that v v v misses.
Proposition 4.1.9 (Polynomial interpretation and finite termination of CG). Let A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n be SPD, x ∗ x^* x ∗ denote the exact solution of A x = b Ax=b A x = b , and e ( k ) = x ( k ) − x ∗ e^{(k)}=x^{(k)}-x^* e ( k ) = x ( k ) − x ∗ the CG error at step k k k (with x ( 0 ) x^{(0)} x ( 0 ) arbitrary).
(Polynomial representation) For each k ≥ 0 k \ge0 k ≥ 0 there exists a polynomial q k q_k q k with deg q k ≤ k \deg q_k \le k deg q k ≤ k and q k ( 0 ) = 1 q_k(0)=1 q k ( 0 ) = 1 such that
e ( k ) = q k ( A ) e ( 0 ) . e^{(k)}=q_k(A)e^{(0)}. e ( k ) = q k ( A ) e ( 0 ) . The polynomial q k q_k q k minimizes the A A A norm of the error at step k k k over the set
{ q ( A ) e ( 0 ) : q ∈ Π k , q ( 0 ) = 1 } . \{q(A)e^{(0)}:q\in\Pi_k,\ q(0)=1\}. { q ( A ) e ( 0 ) : q ∈ Π k , q ( 0 ) = 1 } . (Finite termination) Let d 0 = deg m r ( 0 ) d_0=\deg m_{r^{(0)}} d 0 = deg m r ( 0 ) , where m r ( 0 ) m_{r^{(0)}} m r ( 0 ) is the minimal polynomial of A A A relative to the initial residual r ( 0 ) r^{(0)} r ( 0 ) (Definition 4.1.8). Then CG attains the exact solution in at most d 0 d_0 d 0 steps, e ( d 0 ) = 0 e^{(d_0)}=0 e ( d 0 ) = 0 . Always d 0 ≤ d ≤ n d_0\le d\le n d 0 ≤ d ≤ n with d = deg m d=\deg m d = deg m (for symmetric A A A , d d d is the number of distinct eigenvalues), and d 0 < d d_0<d d 0 < d as soon as r ( 0 ) r^{(0)} r ( 0 ) misses part of the spectrum.
Proof. (1) CG produces x ( k ) ∈ x ( 0 ) + K k ( A , r ( 0 ) ) x^{(k)}\in x^{(0)}+\mathcal{K}_k(A,r^{(0)}) x ( k ) ∈ x ( 0 ) + K k ( A , r ( 0 ) ) , hence
e ( k ) = x ( k ) − x ∗ = e ( 0 ) − v k , v k ∈ K k ( A , r ( 0 ) ) . e^{(k)}=x^{(k)}-x^*=e^{(0)}-v_k,\qquad v_k\in\mathcal{K}_k(A,r^{(0)}). e ( k ) = x ( k ) − x ∗ = e ( 0 ) − v k , v k ∈ K k ( A , r ( 0 ) ) .
Write v k = A s k − 1 ( A ) e ( 0 ) v_k=A s_{k-1}(A)e^{(0)} v k = A s k − 1 ( A ) e ( 0 ) with deg s k − 1 ≤ k − 1 \deg s_{k-1}\le k-1 deg s k − 1 ≤ k − 1 (since K k ( A , r ( 0 ) ) = A K k ( A , e ( 0 ) ) \mathcal{K}_k(A,r^{(0)})=A\mathcal{K}_k(A,e^{(0)}) K k ( A , r ( 0 ) ) = A K k ( A , e ( 0 ) ) ). Then
e ( k ) = ( I − A s k − 1 ( A ) ) e ( 0 ) = q k ( A ) e ( 0 ) , e^{(k)}=(I-As_{k-1}(A))e^{(0)}=q_k(A)e^{(0)}, e ( k ) = ( I − A s k − 1 ( A )) e ( 0 ) = q k ( A ) e ( 0 ) ,
where q k ( λ ) = 1 − λ s k − 1 ( λ ) q_k(\lambda)=1-\lambda s_{k-1}(\lambda) q k ( λ ) = 1 − λ s k − 1 ( λ ) has degree ≤ k \le k ≤ k and satisfies q k ( 0 ) = 1 q_k(0)=1 q k ( 0 ) = 1 . This also shows e ( k ) ∈ K k + 1 ( A , e ( 0 ) ) = span { e ( 0 ) , A e ( 0 ) , … , A k e ( 0 ) } e^{(k)}\in\mathcal{K}_{k+1}(A,e^{(0)})=\operatorname{span}\{e^{(0)},Ae^{(0)},\ldots,A^ke^{(0)}\} e ( k ) ∈ K k + 1 ( A , e ( 0 ) ) = span { e ( 0 ) , A e ( 0 ) , … , A k e ( 0 ) } . The minimization follows from Prop. 4.1.7.
(2) A A A is SPD, hence nonsingular, and r ( 0 ) = − A e ( 0 ) r^{(0)}=-Ae^{(0)} r ( 0 ) = − A e ( 0 ) , so q ( A ) r ( 0 ) = 0 q(A)r^{(0)}=0 q ( A ) r ( 0 ) = 0 if and only if q ( A ) e ( 0 ) = 0 q(A)e^{(0)}=0 q ( A ) e ( 0 ) = 0 : the relative minimal polynomial is the same for r ( 0 ) r^{(0)} r ( 0 ) and for e ( 0 ) e^{(0)} e ( 0 ) . Its roots are eigenvalues of A A A , which are positive, so m r ( 0 ) ( 0 ) ≠ 0 m_{r^{(0)}}(0)\ne0 m r ( 0 ) ( 0 ) = 0 and q d 0 = m r ( 0 ) / m r ( 0 ) ( 0 ) q_{d_0}=m_{r^{(0)}}/m_{r^{(0)}}(0) q d 0 = m r ( 0 ) / m r ( 0 ) ( 0 ) has q d 0 ( 0 ) = 1 q_{d_0}(0)=1 q d 0 ( 0 ) = 1 and q d 0 ( A ) e ( 0 ) = 0 q_{d_0}(A)e^{(0)}=0 q d 0 ( A ) e ( 0 ) = 0 . By part (1), the attainable errors after k ≥ d 0 k\ge d_0 k ≥ d 0 steps include 0 0 0 , so ∥ e ( d 0 ) ∥ A = 0 \lVert e^{(d_0)}\rVert_A=0 ∥ e ( d 0 ) ∥ A = 0 . Since m r ( 0 ) m_{r^{(0)}} m r ( 0 ) divides m m m , d 0 ≤ d = deg m d_0\le d=\deg m d 0 ≤ d = deg m ; for symmetric A A A with d d d distinct eigenvalues m ( λ ) = ∏ j = 1 d ( λ − λ j ) m(\lambda)=\prod_{j=1}^{d}(\lambda-\lambda_j) m ( λ ) = ∏ j = 1 d ( λ − λ j ) , so d ≤ n d\le n d ≤ n . The relative count is the sharp one: three distinct eigenvalues with r ( 0 ) r^{(0)} r ( 0 ) orthogonal to one eigenvector give d 0 = 2 d_0=2 d 0 = 2 , so CG is exact after two steps, not three.
Remark: All the above is true in exact arithmetic. In floating-point arithmetic, loss of orthogonality may delay (or prevent) exact termination; the classical analysis of this effect is due to Paige [35 ] .
:::
Proposition 4.1.10 (Convergence of Conjugate Gradient). Let x ∗ = A − 1 b x^* = A^{-1}b x ∗ = A − 1 b denote the exact solution and κ ( A ) = λ max ( A ) / λ min ( A ) \kappa(A) = \lambda_{\max}(A)/\lambda_{\min}(A) κ ( A ) = λ m a x ( A ) / λ m i n ( A ) the condition number of A A A . Then the CG iterates satisfy the error bound
∥ x ( k ) − x ∗ ∥ A ∥ x ( 0 ) − x ∗ ∥ A ≤ 2 ( κ ( A ) − 1 κ ( A ) + 1 ) k , \frac{\lVert x^{(k)} - x^* \rVert_A}{\lVert x^{(0)} - x^* \rVert_A} \le2\left(\frac{\sqrt{\kappa(A)} - 1}{\sqrt{\kappa(A)} + 1}\right)^k, ∥ x ( 0 ) − x ∗ ∥ A ∥ x ( k ) − x ∗ ∥ A ≤ 2 ( κ ( A ) + 1 κ ( A ) − 1 ) k , where ∥ v ∥ A 2 = v T A v \lVert v \rVert_A^2 = v^{\mathsf{T}} A v ∥ v ∥ A 2 = v T A v . In exact arithmetic the method terminates in at most n n n steps.
Figure 4.3. Error decay of steepest descent and CG for the same SPD matrix. Both are linear (geometric) rates, but CG’s contraction factor is governed by κ ( A ) \sqrt{\kappa(A)} κ ( A ) rather than κ ( A ) \kappa(A) κ ( A ) . For the 1D Poisson matrix, where κ ( A ) = O ( n 2 ) \kappa(A) = O(n^2) κ ( A ) = O ( n 2 ) , this is the difference between O ( n 2 ) O(n^2) O ( n 2 ) and O ( n ) O(n) O ( n ) iterations. The bound is only an upper estimate: when the eigenvalues of A A A are clustered, CG typically converges considerably faster than the curve shown.
Example. For the 1D Poisson matrix of size n n n , steepest descent needs on the order of n 2 n^2 n 2 iterations, while CG requires only on the order of n n n (Problem 4 ). This dramatic acceleration is what makes CG the standard choice for large SPD PDE systems. A full treatment of CG and its convergence theory can be found in [14 , 38 ] .
Table 4.
The Lanczos Method and the Symmetric Eigenproblem
So far, the Krylov subspace K m ( A , r ( 0 ) ) K_m(A,r^{(0)}) K m ( A , r ( 0 ) ) has been a useful tool for solving A x = b Ax = b A x = b . The same subspace answers a different question as well: what does the spectrum of A A A look like? There are two different aspects to this question:
Table 4.1. Comparison of classical stationary methods, steepest descent, and Conjugate Gradient for SPD PDE systems.
We need κ ( A ) \kappa(A) κ ( A ) . Every bound in § 6.2 is stated in terms of λ min ( A ) \lambda_{\min}(A) λ m i n ( A ) and λ max ( A ) \lambda_{\max}(A) λ m a x ( A ) , yet for a large sparse A A A these are exactly the quantities we cannot compute directly. Estimating them is not a side task: it is what tells us whether CG will take ten iterations or ten thousand, and, from Chapter 6, whether a preconditioner is doing its job.
The spectrum is needed but also the eigenvectors. In Chapter 3 we saw that for a graph Laplacian L = D − A L = D - A L = D − A the eigenvector belonging to the smallest nonzero eigenvalue, i.e. the Fiedler vector [10], encodes how the graph splits into communities.
In both cases n n n is far too large for a full eigendecomposition, which costs O ( n 3 ) \mathcal{O}(n^3) O ( n 3 ) and destroys sparsity. What we can afford, as always, is the product A v Av A v . Instead, we rely on iterative Krylov subspace methods such as the Lanczos algorithm [26 ] . The Lanczos algorithm is an efficient iterative method for approximating eigenvalues and eigenvectors of a large symmetric matrix A A A . The key idea is:
K m ( A , q 1 ) = span { q 1 , A q 1 , A 2 q 1 , … , A m − 1 q 1 } . \mathcal{K}_m(A,q_1)=\operatorname{span}\{q_1,Aq_1,A^2q_1,\ldots,A^{m-1}q_1\}. K m ( A , q 1 ) = span { q 1 , A q 1 , A 2 q 1 , … , A m − 1 q 1 } .
Iteratively construct an orthonormal basis Q m = [ q 1 , … , q m ] Q_m=[q_1,\ldots,q_m] Q m = [ q 1 , … , q m ] of K m ( A , q 1 ) \mathcal{K}_m(A,q_1) K m ( A , q 1 ) and project:
T m = Q m T A Q m . T_m=Q_m^{\mathsf{T}}AQ_m. T m = Q m T A Q m .
T m T = ( Q m T A Q m ) T = Q m T A T Q m = Q m T A Q m = T m , T_m^{\mathsf{T}}=(Q_m^{\mathsf{T}}AQ_m)^{\mathsf{T}}=Q_m^{\mathsf{T}}A^{\mathsf{T}}Q_m=Q_m^{\mathsf{T}}AQ_m=T_m, T m T = ( Q m T A Q m ) T = Q m T A T Q m = Q m T A Q m = T m ,
so T m T_m T m is symmetric as well. Being simultaneously Hessenberg and symmetric, it must be tridiagonal :
T m = [ α 1 β 1 β 1 α 2 ⋱ ⋱ ⋱ β m − 1 β m − 1 α m ] . T_m=
\begin{bmatrix}
\alpha_1 & \beta_1 & & & \\
\beta_1 & \alpha_2 & \ddots& & \\
& \ddots& \ddots& \beta_{m-1} & \\
& & \beta_{m-1} & \alpha_m
\end{bmatrix}. T m = α 1 β 1 β 1 α 2 ⋱ ⋱ ⋱ β m − 1 β m − 1 α m .
Only the scalars α j \alpha_j α j and β j \beta_j β j therefore need to be computed, and each new basis vector is orthogonal to all but two of its predecessors automatically. This is the Lanczos algorithm, displayed in Algorithm 2.
Algorithm 2 Lanczos Algorithm
Require: Symmetric matrix A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n , starting vector q 1 q_1 q 1 with ∥ q 1 ∥ 2 = 1 \lVert q_1\rVert_2 = 1 ∥ q 1 ∥ 2 = 1 , number of steps m m m .
Set β 0 = 0 \beta_0 = 0 β 0 = 0 , q 0 = 0 q_0 = 0 q 0 = 0 .
for j = 1 , … , m j = 1,\ldots,m j = 1 , … , m do
w ← A q j − β j − 1 q j − 1 w \leftarrow Aq_j - \beta_{j-1}q_{j-1} w ← A q j − β j − 1 q j − 1
α j ← q j T w \alpha_j \leftarrow q_j^{\mathsf{T}}w α j ← q j T w
w ← w − α j q j w \leftarrow w - \alpha_jq_j w ← w − α j q j
β j ← ∥ w ∥ 2 \beta_j \leftarrow\lVert w\rVert_2 β j ← ∥ w ∥ 2
if β j = 0 \beta_j = 0 β j = 0 then
stop
else
q j + 1 ← w / β j q_{j+1} \leftarrow w/\beta_j q j + 1 ← w / β j
end if
end for
Construct the tridiagonal matrix T m T_m T m .
Notation: two different α \alpha α ’s and β \beta β ’s. The symbols α \alpha α and β \beta β are unfortunately standard in both algorithms, with unrelated meanings. In CG (Definition 4.1.6) α k \alpha_k α k is a step length and β k \beta_k β k a direction-update coefficient ; in Lanczos (Algorithm 2) α j \alpha_j α j and β j \beta_j β j are the diagonal and off-diagonal entries of T m T_m T m . Both conventions are too well established to be worth changing, so we keep them. In Section 4.2.1, where the two algorithms are compared side by side, we write α k c g \alpha_k^{\mathrm{cg}} α k cg , β k c g \beta_k^{\mathrm{cg}} β k cg for the CG scalars and reserve the α j \alpha_j α j , β j \beta_j β j for the entries of T m T_m T m .
Figure 4.4. The Lanczos idea in one line. The large sparse symmetric matrix A A A is never factorised; it is only applied to vectors. After m m m steps the problem has been compressed into an m × m m \times m m × m symmetric tridiagonal matrix T m T_m T m whose eigenvalues (the Ritz values ) approximate the extreme eigenvalues of A A A , and whose eigenvectors lift back to approximate eigenvectors of A A A through Q m Q_m Q m .
Interpretation. The matrix T m ∈ R m × m T_m \in\mathbb{R}^{m \times m} T m ∈ R m × m represents the orthogonal projection of A A A onto the Krylov subspace K m ( A , q 1 ) \mathcal{K}_m(A,q_1) K m ( A , q 1 ) , expressed in the orthonormal basis q 1 , … , q m q_1,\ldots,q_m q 1 , … , q m . Its eigenvalues, called Ritz values , provide approximations to the eigenvalues of A A A and, as the next proposition makes precise, to the extremal ones first. Since T m T_m T m is symmetric and tridiagonal, its eigenvalues can be computed efficiently and stably using standard algorithms [13]. Eigenvectors of T m T_m T m , lifted back via Q m = [ q 1 , … , q m ] Q_m = [q_1,\ldots,q_m] Q m = [ q 1 , … , q m ] , yield approximations of the eigenvectors of A A A .
Proposition 4.2.1 (Ritz values converge at the ends of the spectrum). Let A = A T A=A^{\mathsf{T}} A = A T have eigenvalues λ 1 ≤ ⋯ ≤ λ n \lambda_1 \le\cdots\le\lambda_n λ 1 ≤ ⋯ ≤ λ n and let θ 1 ≤ ⋯ ≤ θ m \theta_1 \le\cdots\le\theta_m θ 1 ≤ ⋯ ≤ θ m be the eigenvalues of T m = Q m T A Q m T_m=Q_m^{\mathsf{T}}AQ_m T m = Q m T A Q m . Then
λ 1 ≤ θ 1 , θ m ≤ λ n , \lambda_1 \le\theta_1,\qquad\theta_m \le\lambda_n, λ 1 ≤ θ 1 , θ m ≤ λ n , and more generally the θ i \theta_i θ i interlace the λ i \lambda_i λ i (Problem 9). Moreover θ 1 \theta_1 θ 1 and θ m \theta_m θ m are the smallest and largest values of the Rayleigh quotient R A ( x ) = x T A x / x T x R_A(x)=x^{\mathsf{T}}Ax/x^{\mathsf{T}}x R A ( x ) = x T A x / x T x over K m ( A , q 1 ) \mathcal{K}_m(A,q_1) K m ( A , q 1 ) , so they improve monotonically with m m m .
Two consequences matter in practice. First, the Ritz values approach λ 1 \lambda_1 λ 1 and λ n \lambda_n λ n from the inside, so a handful of Lanczos steps already brackets the spectrum which is precisely the estimate of κ ( A ) \kappa(A) κ ( A ) that the CG bound of § 6.2 needs. Second, interior eigenvalues are reached much more slowly. If the eigenvalue of interest is not extremal, one must deflate the eigenvectors that are in the way. The next example uses the first of these. Sharp convergence estimates for the Ritz values, which show that the rate again improves by a square root over the power method, are given in [37 ] .
Lanczos and CG are the same recurrence
It is no accident that Lanczos and CG appear in the same chapter. Both build an orthonormal basis of the same Krylov subspace by a three-term recurrence, and in exact arithmetic they build the same one. The following proposition makes this precise and, more usefully, shows that the entries of T m T_m T m can be read off from a CG run without any extra work.
Proposition 4.2.2 (CG and Lanczos generate the same basis, [6, Section 3.1, Theorem 3]). Let A A A be SPD and b ≠ 0 b \ne0 b = 0 . Run CG (Definition 4.1.6 ) from x ( 0 ) = 0 x^{(0)}=0 x ( 0 ) = 0 , so that r ( 0 ) = b r^{(0)}=b r ( 0 ) = b , and assume r ( j ) ≠ 0 r^{(j)} \ne0 r ( j ) = 0 for j = 0 , … , m − 1 j=0,\ldots,m-1 j = 0 , … , m − 1 ; denote its scalars by α j c g , β j c g \alpha_j^{\mathrm{cg}},\beta_j^{\mathrm{cg}} α j cg , β j cg . Run Lanczos (Algorithm 2) on A A A with q 1 = b / ∥ b ∥ 2 q_1=b/\lVert b\rVert_2 q 1 = b / ∥ b ∥ 2 , producing q 1 , … , q m q_1,\ldots,q_m q 1 , … , q m and the entries α j , β j \alpha_j,\beta_j α j , β j of T m T_m T m . Then
(i) the two methods produce the same orthonormal basis of K m ( A , b ) \mathcal{K}_m(A,b) K m ( A , b ) , namely
q j + 1 = ( − 1 ) j r ( j ) ∥ r ( j ) ∥ 2 , j = 0 , … , m − 1 ; q_{j+1}=(-1)^j\frac{r^{(j)}}{\lVert r^{(j)}\rVert_2},\qquad j=0,\ldots,m-1; q j + 1 = ( − 1 ) j ∥ r ( j ) ∥ 2 r ( j ) , j = 0 , … , m − 1 ; (ii) the entries of T m T_m T m are determined by the CG scalars,
α 1 = 1 α 0 c g , α j = 1 α j − 1 c g + β j − 2 c g α j − 2 c g ( j ≥ 2 ) , β j = β j − 1 c g α j − 1 c g ( j ≥ 1 ) . \alpha_1=\frac{1}{\alpha_0^{\mathrm{cg}}},\qquad
\alpha_j=\frac{1}{\alpha_{j-1}^{\mathrm{cg}}}+\frac{\beta_{j-2}^{\mathrm{cg}}}{\alpha_{j-2}^{\mathrm{cg}}}\quad(j\ge2),\qquad
\beta_j=\frac{\sqrt{\beta_{j-1}^{\mathrm{cg}}}}{\alpha_{j-1}^{\mathrm{cg}}}\quad(j\ge1). α 1 = α 0 cg 1 , α j = α j − 1 cg 1 + α j − 2 cg β j − 2 cg ( j ≥ 2 ) , β j = α j − 1 cg β j − 1 cg ( j ≥ 1 ) .
Proof. (i) By Proposition 4.1.7 the residuals r ( 0 ) , … , r ( m − 1 ) r^{(0)},\ldots,r^{(m-1)} r ( 0 ) , … , r ( m − 1 ) are mutually orthogonal, and r ( j ) ∈ K j + 1 ( A , b ) ∖ K j ( A , b ) r^{(j)}\in\mathcal{K}_{j+1}(A,b)\setminus\mathcal{K}_j(A,b) r ( j ) ∈ K j + 1 ( A , b ) ∖ K j ( A , b ) . Once normalised they are therefore an orthonormal basis of K m ( A , b ) \mathcal{K}_m(A,b) K m ( A , b ) adapted to the nested sequence K 1 ⊂ K 2 ⊂ ⋯ \mathcal{K}_1\subset\mathcal{K}_2\subset\cdots K 1 ⊂ K 2 ⊂ ⋯ . The Lanczos vectors are, by construction, the Gram–Schmidt basis of that same nested sequence, and such a basis is unique up to the sign of each vector. The signs are pinned down by the convention β j = ∥ w ∥ 2 > 0 \beta_j=\lVert w\rVert_2>0 β j = ∥ w ∥ 2 > 0 in Algorithm 2, which produces the alternating factor ( − 1 ) j (-1)^j ( − 1 ) j .
(ii) Eliminate the search directions. From r ( k + 1 ) = r ( k ) − α k c g A p ( k ) r^{(k+1)}=r^{(k)}-\alpha_k^{\mathrm{cg}}Ap^{(k)} r ( k + 1 ) = r ( k ) − α k cg A p ( k ) we get A p ( k ) = ( r ( k ) − r ( k + 1 ) ) / α k c g Ap^{(k)}=(r^{(k)}-r^{(k+1)})/\alpha_k^{\mathrm{cg}} A p ( k ) = ( r ( k ) − r ( k + 1 ) ) / α k cg , and substituting this into A r ( k ) = A p ( k ) − β k − 1 c g A p ( k − 1 ) Ar^{(k)}=Ap^{(k)}-\beta_{k-1}^{\mathrm{cg}}Ap^{(k-1)} A r ( k ) = A p ( k ) − β k − 1 cg A p ( k − 1 ) , which follows from p ( k ) = r ( k ) + β k − 1 c g p ( k − 1 ) p^{(k)}=r^{(k)}+\beta_{k-1}^{\mathrm{cg}}p^{(k-1)} p ( k ) = r ( k ) + β k − 1 cg p ( k − 1 ) , gives the three-term recurrence
A r ( k ) = − r ( k + 1 ) α k c g + ( 1 α k c g + β k − 1 c g α k − 1 c g ) r ( k ) − β k − 1 c g α k − 1 c g r ( k − 1 ) . (5) Ar^{(k)}=-\frac{r^{(k+1)}}{\alpha_k^{\mathrm{cg}}}+\left(\frac{1}{\alpha_k^{\mathrm{cg}}}+\frac{\beta_{k-1}^{\mathrm{cg}}}{\alpha_{k-1}^{\mathrm{cg}}}\right)r^{(k)}-\frac{\beta_{k-1}^{\mathrm{cg}}}{\alpha_{k-1}^{\mathrm{cg}}}r^{(k-1)}. \tag*{(5)} A r ( k ) = − α k cg r ( k + 1 ) + ( α k cg 1 + α k − 1 cg β k − 1 cg ) r ( k ) − α k − 1 cg β k − 1 cg r ( k − 1 ) . (5)
Now write ρ k = ∥ r ( k ) ∥ 2 \rho_k=\lVert r^{(k)}\rVert_2 ρ k = ∥ r ( k ) ∥ 2 , so that r ( k ) = ( − 1 ) k ρ k q k + 1 r^{(k)}=(-1)^k\rho_k q_{k+1} r ( k ) = ( − 1 ) k ρ k q k + 1 by (i). The definition of β k c g \beta_k^{\mathrm{cg}} β k cg gives ρ k + 1 / ρ k = β k c g \rho_{k+1}/\rho_k=\sqrt{\beta_k^{\mathrm{cg}}} ρ k + 1 / ρ k = β k cg . Dividing the recurrence by ( − 1 ) k ρ k (-1)^k\rho_k ( − 1 ) k ρ k turns every sign positive and yields
A q k + 1 = β k c g α k c g q k + 2 + ( 1 α k c g + β k − 1 c g α k − 1 c g ) q k + 1 + β k − 1 c g α k − 1 c g q k . Aq_{k+1}=\frac{\sqrt{\beta_k^{\mathrm{cg}}}}{\alpha_k^{\mathrm{cg}}}q_{k+2}+\left(\frac{1}{\alpha_k^{\mathrm{cg}}}+\frac{\beta_{k-1}^{\mathrm{cg}}}{\alpha_{k-1}^{\mathrm{cg}}}\right)q_{k+1}+\frac{\sqrt{\beta_{k-1}^{\mathrm{cg}}}}{\alpha_{k-1}^{\mathrm{cg}}}q_k. A q k + 1 = α k cg β k cg q k + 2 + ( α k cg 1 + α k − 1 cg β k − 1 cg ) q k + 1 + α k − 1 cg β k − 1 cg q k .
Comparing with the Lanczos recurrence A q j = β j q j + 1 + α j q j + β j − 1 q j − 1 Aq_j=\beta_jq_{j+1}+\alpha_jq_j+\beta_{j-1}q_{j-1} A q j = β j q j + 1 + α j q j + β j − 1 q j − 1 and setting j = k + 1 j=k+1 j = k + 1 gives the stated formulas.
Remark 4.2.3. The alternating sign in (i) is a normalisation convention: replacing q j + 1 q_{j+1} q j + 1 by − q j + 1 -q_{j+1} − q j + 1 conjugates T m T_m T m by a diagonal matrix of signs, which changes the signs of the off-diagonal entries but leaves the Ritz values, and hence everything we use them for, unchanged.
The practical payoff is that spectral information comes for free from a run one was performing anyway: a CG solve can report an estimate of λ min \lambda_{\min} λ m i n , λ max \lambda_{\max} λ m a x and hence κ ( A ) \kappa(A) κ ( A ) at the cost of a few scalar operations and no extra matrix–vector products. Proposition 4.2.2 also explains the finite-termination property of § 6.2 from the other side: CG stops after d d d steps because Lanczos exhausts the Krylov space after d d d steps, d d d being the number of distinct eigenvalues. A systematic account of this correspondence, together with the consequences it has for error estimation inside a running CG solve, is given in [6].
Application: computing the Fiedler vector
We return to the graph of Chapter 3: two triangles joined by a single edge,
A = ( 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 ) , L = D − A , D = diag ( 2 , 2 , 3 , 3 , 2 , 2 ) . A=\begin{pmatrix}
0 & 1 & 1 & 0 & 0 & 0\\
1 & 0 & 1 & 0 & 0 & 0\\
1 & 1 & 0 & 1 & 0 & 0\\
0 & 0 & 1 & 0 & 1 & 1\\
0 & 0 & 0 & 1 & 0 & 1\\
0 & 0 & 0 & 1 & 1 & 0
\end{pmatrix},\qquad L=D-A,\qquad D=\operatorname{diag}(2,2,3,3,2,2). A = 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 , L = D − A , D = diag ( 2 , 2 , 3 , 3 , 2 , 2 ) .
There we ran Jacobi on this Laplacian and observed that the slowest-decaying error mode was the Fiedler vector. We can perform a similar computation with Lanczos.
Deflating the known null vector. L L L is symmetric positive semidefinite and always has the eigenpair ( 0 , 1 ) (0,1) ( 0 , 1 ) , since every row sums to zero; we assume the graph connected, so that 0 0 0 is simple and λ 2 > 0 \lambda_2>0 λ 2 > 0 . The Fiedler value λ 2 \lambda_2 λ 2 is therefore not extremal, but it sits just above a known eigenvalue. By the proposition above, Lanczos would converge to 0 0 0 and to λ max \lambda_{\max} λ m a x long before it resolved λ 2 \lambda_2 λ 2 . The remedy is immediate: start from a vector orthogonal to 1 1 1 and keep the iteration in that subspace,
q 1 ⊥ 1 , w ← w − ( 1 T w ) 1 n at every step. q_1\perp1,\qquad w\leftarrow w-\frac{(1^{\mathsf T}w)1}{n}\quad\text{at every step.} q 1 ⊥ 1 , w ← w − n ( 1 T w ) 1 at every step.
On the deflated subspace λ 2 \lambda_2 λ 2 is the smallest eigenvalue, and the proposition applies again.
Example 4.2.4 (Lanczos on the two-triangle Laplacian). The exact spectrum is
{ 0 , 0.4384 , 3 , 3 , 3 , 4.5616 } . \{0,\ 0.4384,\ 3,\ 3,\ 3,\ 4.5616\}. { 0 , 0.4384 , 3 , 3 , 3 , 4.5616 } . Starting from a normalised random q 1 ⊥ 1 q_1\perp1 q 1 ⊥ 1 and deflating as above, the Ritz values are
Table 3.
After three steps the Ritz values are the exact nonzero eigenvalues, and this is not an accident. On 1 ⊥ 1^{\perp} 1 ⊥ the eigenvalues of L L L are { 0.4384 , 3 , 3 , 3 , 4.5616 } \{0.4384, 3, 3, 3, 4.5616\} { 0.4384 , 3 , 3 , 3 , 4.5616 } , that is, only three distinct values, so the minimal polynomial of L L L relative to any starting vector (Definition 4.1.8) has degree at most 3 and Lanczos must terminate at m = 3 m=3 m = 3 whatever q 1 q_1 q 1 is. The Ritz vector belonging to θ 1 = 0.4384 \theta_1=0.4384 θ 1 = 0.4384 is
v = Q 3 y 1 = ( 0.4647 , 0.4647 , 0.2610 , − 0.2610 , − 0.4647 , − 0.4647 ) T , v = Q_3 y_1 = (0.4647,\ 0.4647,\ 0.2610,\ -0.2610,\ -0.4647,\ -0.4647)^{\mathsf{T}}, v = Q 3 y 1 = ( 0.4647 , 0.4647 , 0.2610 , − 0.2610 , − 0.4647 , − 0.4647 ) T ,
the Fiedler vector to machine precision, and thresholding it is the simplest form of spectral clustering. Its sign pattern splits the vertices exactly as the two triangles, { 1 , 2 , 3 } \{1,2,3\} { 1 , 2 , 3 } against { 4 , 5 , 6 } \{4,5,6\} { 4 , 5 , 6 } , and the small entries ± 0.2610 \pm0.2610 ± 0.2610 identify nodes 3 and 4 as the bottleneck across the cut.
Takeaways
Lanczos is a Krylov projection: m m m matrix-vector products compress A A A into an m × m m \times m m × m tridiagonal T m T_m T m , whose eigenvalues (Ritz values) approximate those of A A A .
Convergence happens at the ends of the spectrum first. Extremal eigenvalues come cheaply; a few Lanczos steps therefore bracket λ min , λ max \lambda_{\min},\lambda_{\max} λ m i n , λ m a x hence κ ( A ) \kappa(A) κ ( A ) , which the CG bound of § 6.2 needs.
On graph Laplacians, deflating the constant vector makes the Fiedler value extremal, and Lanczos delivers the community structure in an optimal number of steps.
Krylov Methods and Machine Learning (optional)
Fitting a linear model to data by least squares is the simplest learning task and one of the most widely used. The same problem appears well beyond machine learning: reconstructing a sharp image from a blurred one, recovering a physical parameter from indirect measurements, and combining a forecast with observations in the data assimilation systems. In each case the matrix is ill-conditioned and the data are noisy, so the point of view differs from that of the earlier sections: there the system had a unique solution and we wanted it accurately, whereas here solving to machine precision reproduces the noise rather than the underlying model.2
Least squares and the SVD
We are given N N N observations, stored as the rows of a design matrix X ∈ R N × d X \in\mathbb{R}^{N \times d} X ∈ R N × d , together with the corresponding targets y ∈ R N y \in\mathbb{R}^{N} y ∈ R N , and we look for coefficients x ∈ R d x \in\mathbb{R}^{d} x ∈ R d for which X x Xx X x is as close to y y y as possible. This is the linear least squares problem
min x ∈ R d ∥ X x − y ∥ 2 2 ⟺ X T X x = X T y , (6) \min_{x \in\mathbb{R}^{d}} \lVert Xx-y\rVert_2^2 \quad\Longleftrightarrow\quad X^{\mathsf{T}}Xx=X^{\mathsf{T}}y, \tag*{(6)} x ∈ R d min ∥ X x − y ∥ 2 2 ⟺ X T X x = X T y , (6)
the equivalence being obtained by setting the gradient of the objective to zero. The matrix X T X X^{\mathsf{T}}X X T X is symmetric positive semidefinite, and positive definite whenever X X X has full column rank, so we are back in the setting of this chapter.
Proposition 4.3.1 (Least-squares solution via the SVD). Let X ∈ R N × d X \in\mathbb{R}^{N \times d} X ∈ R N × d have full column rank, with thin SVD X = U Σ V T X=U\Sigma V^{\mathsf{T}} X = U Σ V T and Σ = diag ( σ 1 , … , σ d ) \Sigma=\operatorname{diag}(\sigma_1,\ldots,\sigma_d) Σ = diag ( σ 1 , … , σ d ) , σ 1 ≥ ⋯ ≥ σ d > 0 \sigma_1 \geq\cdots\geq\sigma_d>0 σ 1 ≥ ⋯ ≥ σ d > 0 . Then the solution of (6) is unique and given by
x l s = ∑ i = 1 d u i T y σ i v i . (7) x^{\mathrm{ls}}=\sum_{i=1}^{d}\frac{u_i^{\mathsf{T}}y}{\sigma_i}v_i. \tag*{(7)} x ls = i = 1 ∑ d σ i u i T y v i . (7)
Proof. Since U U U has orthonormal columns, X T X = V Σ U T U Σ V T = V Σ 2 V T X^{\mathsf{T}}X = V\Sigma U^{\mathsf{T}}U\Sigma V^{\mathsf{T}} = V\Sigma^2V^{\mathsf{T}} X T X = V Σ U T U Σ V T = V Σ 2 V T , which is invertible because every σ i > 0 \sigma_i > 0 σ i > 0 ; the normal equations in (6) therefore have the unique solution x l s = ( X T X ) − 1 X T y = V Σ − 2 V T V Σ U T y = V Σ − 1 U T y x^{\mathrm{ls}} = (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}y = V\Sigma^{-2}V^{\mathsf{T}}V\Sigma U^{\mathsf{T}}y = V\Sigma^{-1}U^{\mathsf{T}}y x ls = ( X T X ) − 1 X T y = V Σ − 2 V T V Σ U T y = V Σ − 1 U T y , which written columnwise is (4.2). (If rank X = r < d X = r < d X = r < d the solution is no longer unique, but replacing d d d by r r r in (4.2) gives the one of smallest norm.)
Formula (4.2) also shows where the difficulty lies. Suppose for example that the target y y y carry noise, y y y being replaced by y + e y + e y + e . By (4.2) the i i i -th coefficient changes by u i T e / σ i u_i^{\mathsf{T}}e/\sigma_i u i T e / σ i . If σ i = 10 − 3 \sigma_i = 10^{-3} σ i = 1 0 − 3 , a noise component of size 10 − 3 10^{-3} 1 0 − 3 shifts that coefficient by 1. The coefficients the data determine the least well are thus the ones the noise affects the most, which is why x l s x^{\mathrm{ls}} x ls is of little use when X X X is ill-conditioned.
Spectral filters and the truncated SVD
The remedy is modify the form of (4.2) while damping the terms responsible for amplification of errors. We therefore replace x l s x^{\mathrm{ls}} x ls by
x ^ = ∑ i = 1 d f i u i T y σ i v i , f i ∈ [ 0 , 1 ] , (8) \widehat{x} = \sum_{i=1}^{d} f_i \frac{u_i^{\mathsf{T}}y}{\sigma_i}v_i,\qquad f_i \in[0,1], \tag*{(8)} x = i = 1 ∑ d f i σ i u i T y v i , f i ∈ [ 0 , 1 ] , (8)
where the coefficients f i f_i f i are called filter factors : f i = 1 f_i = 1 f i = 1 retains the i i i -th direction exactly as least squares had it, f i = 0 f_i = 0 f i = 0 discards it, and intermediate values damp it. Choosing the f i f_i f i , keeping the directions the data determine well and suppressing the others, is what regularisation means [19 ] . The simplest choice is a sharp cutoff: for a fixed k ≤ d k \le d k ≤ d , set f i = 1 f_i = 1 f i = 1 for i ≤ k i \le k i ≤ k and f i = 0 f_i = 0 f i = 0 otherwise, keeping the k k k best-determined directions. This is the truncated SVD , the most transparent of the filters but also the most expensive: it requires the singular value decomposition of X X X , out of reach for large d d d , together with a cutoff k k k chosen by hand. The next two sections present filters that require neither.
Ridge regression: a smooth filter
Rather than truncating, one may penalise large coefficients. Ridge regression solves
min x 1 2 ∥ X x − y ∥ 2 2 + λ 2 ∥ x ∥ 2 2 ⟺ ( X T X + λ I ) x = X T y , \min_x \frac{1}{2}\lVert Xx-y\rVert^2_2 + \frac{\lambda}{2}\lVert x\rVert^2_2 \qquad\Longleftrightarrow\qquad(X^{\mathsf{T}}X+\lambda I)x = X^{\mathsf{T}}y, x min 2 1 ∥ X x − y ∥ 2 2 + 2 λ ∥ x ∥ 2 2 ⟺ ( X T X + λ I ) x = X T y ,
where λ > 0 \lambda> 0 λ > 0 is a regularisation parameter. The matrix A λ = X T X + λ I A_\lambda= X^{\mathsf{T}}X+\lambda I A λ = X T X + λ I is SPD, so CG applies directly and never requires A λ A_\lambda A λ itself: A λ v = X T ( X v ) + λ v A_\lambda v = X^{\mathsf{T}}(Xv)+\lambda v A λ v = X T ( X v ) + λ v costs two passes over the data, and no d × d d \times d d × d matrix is ever formed. This is what makes Krylov methods natural here. It remains to determine the effect of λ \lambda λ .
Proposition 4.3.2 (Ridge regression is a spectral filter). Under the assumptions of Proposition 4.3.1 and for λ > 0 \lambda> 0 λ > 0 , the ridge solution x λ = ( X T X + λ I ) − 1 X T y x_\lambda= (X^{\mathsf{T}}X+\lambda I)^{-1}X^{\mathsf{T}}y x λ = ( X T X + λ I ) − 1 X T y has the form (8) with f i = σ i 2 / ( σ i 2 + λ ) ∈ ( 0 , 1 ) f_i = \sigma_i^2/(\sigma_i^2+\lambda) \in(0,1) f i = σ i 2 / ( σ i 2 + λ ) ∈ ( 0 , 1 ) .
Proof. As above, X T X + λ I = V ( Σ 2 + λ I ) V T X^{\mathsf{T}}X+\lambda I = V(\Sigma^2+\lambda I)V^{\mathsf{T}} X T X + λ I = V ( Σ 2 + λ I ) V T and X T y = V Σ U T y X^{\mathsf{T}}y = V\Sigma U^{\mathsf{T}}y X T y = V Σ U T y , so
x λ = V ( Σ 2 + λ I ) − 1 Σ U T y = ∑ i σ i σ i 2 + λ ( u i T y ) v i = σ i 2 σ i 2 + λ ⋅ 1 σ i ( u i T y ) v i x_\lambda= V(\Sigma^2+\lambda I)^{-1}\Sigma U^{\mathsf{T}}y = \sum_i \frac{\sigma_i}{\sigma_i^2+\lambda}(u_i^{\mathsf{T}}y)v_i = \frac{\sigma_i^2}{\sigma_i^2+\lambda}\cdot\frac{1}{\sigma_i}(u_i^{\mathsf{T}}y)v_i x λ = V ( Σ 2 + λ I ) − 1 Σ U T y = i ∑ σ i 2 + λ σ i ( u i T y ) v i = σ i 2 + λ σ i 2 ⋅ σ i 1 ( u i T y ) v i
which represents the solution in the form (8) .
Hence f i ≈ 1 f_i \approx1 f i ≈ 1 when σ i 2 ≫ λ \sigma_i^2 \gg\lambda σ i 2 ≫ λ and f i ≈ 0 f_i \approx0 f i ≈ 0 when σ i 2 ≪ λ \sigma_i^2 \ll\lambda σ i 2 ≪ λ : ridge regression is a smoothed truncated SVD, in which λ \lambda λ rather than k k k determines where the transition occurs (Figure 4.5 , left). No singular value decomposition is needed, but a value of λ \lambda λ still has to be supplied.
Figure 4.5. Left: the truncated SVD cuts small-σ i \sigma_i σ i directions off abruptly, ridge damps them smoothly, and the CG filter lies between the two. Right: semi-convergence — the unresolved part (dotted) falls while the admitted noise (dashed) grows, so the total (solid) is smallest at k ⋆ k^\star k ⋆ .
Early stopping: the filter generated by CG
We now consider a third possibility, in which no filter is designed and no parameter selected: solve the unregularised normal equations in (6) by CG from x ( 0 ) = 0 x^{(0)} = 0 x ( 0 ) = 0 , and stop after k k k steps. The iteration then produces a filter of its own.
Proposition 4.3.3 (CG is a spectral filter). Apply CG to X T X x = X T y X^\mathsf{T}X x = X^\mathsf{T}y X T X x = X T y from x ( 0 ) = 0 x^{(0)} = 0 x ( 0 ) = 0 , and let q k q_k q k be the CG residual polynomial of Proposition 4.1.9, so deg q k ≤ k \deg q_k \le k deg q k ≤ k and q k ( 0 ) = 1 q_k(0) = 1 q k ( 0 ) = 1 . Then x ( k ) x^{(k)} x ( k ) has the form (8) with f i = 1 − q k ( σ i 2 ) f_i = 1 - q_k(\sigma_i^2) f i = 1 − q k ( σ i 2 ) , and among all filters built this way from a polynomial, that of CG minimises the A A A -norm of the error at every step.
Proof. The matrix A = X T X A = X^\mathsf{T}X A = X T X is SPD with eigenpairs ( σ i 2 , v i ) (\sigma_i^2, v_i) ( σ i 2 , v i ) and A x = X T y Ax = X^\mathsf{T}y A x = X T y has solution x l s x^{\mathrm{ls}} x ls . By Proposition 4.1.9, e ( k ) = q k ( A ) e ( 0 ) e^{(k)} = q_k(A)e^{(0)} e ( k ) = q k ( A ) e ( 0 ) ; since x ( 0 ) = 0 x^{(0)} = 0 x ( 0 ) = 0 we have e ( 0 ) = − x l s e^{(0)} = -x^{\mathrm{ls}} e ( 0 ) = − x ls , so x ( k ) = x l s + e ( k ) = ( I − q k ( A ) ) x l s x^{(k)} = x^{\mathrm{ls}} + e^{(k)} = (I - q_k(A))x^{\mathrm{ls}} x ( k ) = x ls + e ( k ) = ( I − q k ( A )) x ls . Expanding x l s x^{\mathrm{ls}} x ls in the basis { v i } \{v_i\} { v i } and using q k ( A ) v i = q k ( σ i 2 ) v i q_k(A)v_i = q_k(\sigma_i^2)v_i q k ( A ) v i = q k ( σ i 2 ) v i gives the filter; the optimality is that of Proposition 4.1.9, read through this correspondence.
Stopping early is therefore a form of regularisation. The iteration count k k k plays the role that k k k plays for the truncated SVD and λ \lambda λ for ridge regression, but at no additional cost, since the iteration has to be carried out in any case.
To be noted that the first two filters are fixed before the data are seen, whereas q k q_k q k is built from K k ( A , X T y ) \mathcal{K}_k(A, X^\mathsf{T}y) K k ( A , X T y ) and hence depends on y y y . CG selects a filter adapted to the problem at hand, which is why it outperforms a fixed filter at the same k k k .
Remark 4.3.4 (When to stop). Filtering explains why early stopping helps, but not when to stop. Two effects compete, as shown on the right of Figure 4.5 :
the part of the solution not yet resolved decreases with k k k ;
the noise admitted through the filter grows with k k k , since the filter factors approach 1 on the small σ i \sigma_i σ i for which 1 / σ i 1/\sigma_i 1/ σ i is large.
Their sum therefore attains a minimum at some k ⋆ k^\star k ⋆ and increases again beyond it, a phenomenon known as semi-convergence . A stopping rule is thus indispensable.
If the noise level δ \delta δ is known, one stops once ∥ y − X x ( k ) ∥ 2 ≤ τ δ \lVert y - Xx^{(k)}\rVert_2 \le\tau\delta ∥ y − X x ( k ) ∥ 2 ≤ τ δ with τ > 1 \tau> 1 τ > 1 , since fitting the data more closely than the noise amounts to fitting the noise; this makes CG a genuine regularisation method [18 ] . In machine learning δ \delta δ is seldom known, and one stops when the error on a held-out validation set begins to rise.
Exercises
Problem 1 (A link between fixed point and Krylov iterative methods, exam 2023/2024). Let A , M ∈ R n × n A, M \in\mathbb{R}^{n \times n} A , M ∈ R n × n be non-singular, I : = I n I := I_n I := I n , and b ∈ R n b \in\mathbb{R}^n b ∈ R n . Let x 0 ∈ R n x^0 \in\mathbb{R}^n x 0 ∈ R n be arbitrary and define r k = b − A x k r^k = b - Ax^k r k = b − A x k for k = 0 , 1 , 2 , … k = 0, 1, 2, \ldots k = 0 , 1 , 2 , … . Abbreviate r : = b − A x 0 r := b - Ax^0 r := b − A x 0 . Assume one splits A = M − ( M − A ) A = M - (M - A) A = M − ( M − A ) .
(a) Show that A x = b Ax = b A x = b if and only if
x = x + M − 1 ( b − A x ) . (9) x = x + M^{-1}(b - Ax). \tag*{(9)} x = x + M − 1 ( b − A x ) . (9)
(b) Relation (9) leads to the fixed-point method
x k + 1 = x k + M − 1 r k . x^{k+1} = x^k + M^{-1}r^k. x k + 1 = x k + M − 1 r k .
Show that
r k + 1 = ( I − A M − 1 ) r k . r^{k+1} = (I - AM^{-1})r^k. r k + 1 = ( I − A M − 1 ) r k .
(c) Show that
M − 1 r k + 1 = ( I − M − 1 A ) M − 1 r k . M^{-1}r^{k+1} = (I - M^{-1}A)M^{-1}r^k. M − 1 r k + 1 = ( I − M − 1 A ) M − 1 r k .
(d) Define the difference u k = x k + 1 − x k u^k = x^{k+1} - x^k u k = x k + 1 − x k . Show that
u k = M − 1 r k . u^k = M^{-1}r^k. u k = M − 1 r k .
(e) Define U k = [ u 0 , u 1 , … , u k − 1 ] ∈ R n × k U^k = [u^0, u^1, \ldots, u^{k-1}] \in\mathbb{R}^{n \times k} U k = [ u 0 , u 1 , … , u k − 1 ] ∈ R n × k . Show that
U k = [ M − 1 r , ( I − M − 1 A ) M − 1 r , … , ( I − M − 1 A ) k − 1 M − 1 r ] . U^k = [M^{-1}r,\ (I - M^{-1}A)M^{-1}r,\ \ldots,\ (I - M^{-1}A)^{k-1}M^{-1}r]. U k = [ M − 1 r , ( I − M − 1 A ) M − 1 r , … , ( I − M − 1 A ) k − 1 M − 1 r ] .
Problem 2 (Krylov spaces, homework 2023/2024). Let n = 5 n = 5 n = 5 and
A = [ 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 ] . A =
\begin{bmatrix}
2 & -1 & 0 & 0 & 0 \\
-1 & 2 & -1 & 0 & 0 \\
0 & -1 & 2 & -1 & 0 \\
0 & 0 & -1 & 2 & -1 \\
0 & 0 & 0 & -1 & 2
\end{bmatrix}. A = 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 − 1 0 0 0 − 1 2 .
One finds that its Schur decomposition is A = S D S − 1 A = SDS^{-1} A = S D S − 1 , with S S S orthogonal (so S − 1 = S T S^{-1} = S^{\mathsf{T}} S − 1 = S T ) and
D = diag ( 2 − 3 , 1 , 2 , 3 , 2 + 3 ) , S = [ 1 12 − 1 2 1 3 1 2 − 1 12 1 2 − 1 2 0 − 1 2 1 2 1 3 0 − 1 3 0 1 3 1 2 1 2 0 1 2 1 2 1 12 1 2 1 3 − 1 2 − 1 12 ] . D = \operatorname{diag}(2-\sqrt{3}, 1, 2, 3, 2+\sqrt{3}), \qquad
S =
\begin{bmatrix}
\frac{1}{\sqrt{12}} & -\frac{1}{2} & \frac{1}{\sqrt{3}} & \frac{1}{2} & -\frac{1}{\sqrt{12}} \\
\frac{1}{2} & -\frac{1}{2} & 0 & -\frac{1}{2} & \frac{1}{2} \\
\frac{1}{\sqrt{3}} & 0 & -\frac{1}{\sqrt{3}} & 0 & \frac{1}{\sqrt{3}} \\
\frac{1}{2} & \frac{1}{2} & 0 & \frac{1}{2} & \frac{1}{2} \\
\frac{1}{\sqrt{12}} & \frac{1}{2} & \frac{1}{\sqrt{3}} & -\frac{1}{2} & -\frac{1}{\sqrt{12}}
\end{bmatrix}. D = diag ( 2 − 3 , 1 , 2 , 3 , 2 + 3 ) , S = 12 1 2 1 3 1 2 1 12 1 − 2 1 − 2 1 0 2 1 2 1 3 1 0 − 3 1 0 3 1 2 1 − 2 1 0 2 1 − 2 1 − 12 1 2 1 3 1 2 1 − 12 1 .
Write S = [ s 1 , … , s 5 ] S = [s_1, \ldots, s_5] S = [ s 1 , … , s 5 ] as a matrix of five columns s 1 , … , s 5 s_1, \ldots, s_5 s 1 , … , s 5 .
(a) Determine the minimal polynomial of A A A and the polynomial q q q for which A − 1 = q ( A ) A^{-1} = q(A) A − 1 = q ( A ) .
(b) Determine dim K k ( A , s 2 ) \dim\mathcal{K}_k(A, s_2) dim K k ( A , s 2 ) and dim K k ( A , e 1 ) \dim\mathcal{K}_k(A, e_1) dim K k ( A , e 1 ) for k = 1 , 2 , 3 , … k = 1, 2, 3, \ldots k = 1 , 2 , 3 , … .
(c) Let b = s 2 + s 4 b = s_2 + s_4 b = s 2 + s 4 and x 0 = 0 x_0 = 0 x 0 = 0 . Does there exist (in exact arithmetic) a k ∈ N k \in\mathbb{N} k ∈ N for which the CG iterate x k x_k x k equals the solution x ^ \hat{x} x ^ of A x ^ = b A\hat{x} = b A x ^ = b ? If yes, what is that value of k k k ?
Exercises
Problem 3 (Conjugate Gradient and Lanczos by hand). Let
A = [ 2 − 1 0 − 1 2 − 1 0 − 1 2 ] , b = e 1 = [ 1 0 0 ] , x ( 0 ) = 0. A = \begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{bmatrix}, \qquad b = e_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \qquad x^{(0)} = 0. A = 2 − 1 0 − 1 2 − 1 0 − 1 2 , b = e 1 = 1 0 0 , x ( 0 ) = 0.
(a) Perform two steps of CG (Definition 4.1.6 ): give α 0 \alpha_0 α 0 , x ( 1 ) x^{(1)} x ( 1 ) , r ( 1 ) r^{(1)} r ( 1 ) , β 0 \beta_0 β 0 , p ( 1 ) p^{(1)} p ( 1 ) , α 1 \alpha_1 α 1 , x ( 2 ) x^{(2)} x ( 2 ) and r ( 2 ) r^{(2)} r ( 2 ) . Check that r ( 0 ) r^{(0)} r ( 0 ) , r ( 1 ) r^{(1)} r ( 1 ) , r ( 2 ) r^{(2)} r ( 2 ) are mutually orthogonal and that ( p ( 0 ) ) T A p ( 1 ) = 0 (p^{(0)})^{\mathsf{T}} A p^{(1)} = 0 ( p ( 0 ) ) T A p ( 1 ) = 0 .
(b) Perform the third step and show that x ( 3 ) = A − 1 b x^{(3)} = A^{-1}b x ( 3 ) = A − 1 b . Use Proposition 4.1.9 to explain why three steps suffice, and why for this b b b CG cannot stop earlier. (The eigenvalues of A A A are 2 − 2 2 - \sqrt{2} 2 − 2 , 2 2 2 , 2 + 2 2 + \sqrt{2} 2 + 2 .)
(c) Compute ∥ e ( k ) ∥ A \lVert e^{(k)} \rVert_A ∥ e ( k ) ∥ A for k = 0 , 1 , 2 k = 0, 1, 2 k = 0 , 1 , 2 and compare the ratios ∥ e ( k ) ∥ A / ∥ e ( 0 ) ∥ A \lVert e^{(k)} \rVert_A / \lVert e^{(0)} \rVert_A ∥ e ( k ) ∥ A / ∥ e ( 0 ) ∥ A with the bound
2 ( κ − 1 κ + 1 ) k . 2\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^k. 2 ( κ + 1 κ − 1 ) k .
(d) Run two steps of the Lanczos algorithm (Algorithm 2) from q 1 = e 1 q_1 = e_1 q 1 = e 1 and verify the formulas of Proposition 4.2.2 (ii) for α 1 \alpha_1 α 1 , α 2 \alpha_2 α 2 and β 1 \beta_1 β 1 .
Problem 4 (Predicting iteration counts for the 1D Poisson matrix). Let A = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n A = \operatorname{tridiag}(-1, 2, -1) \in\mathbb{R}^{n \times n} A = tridiag ( − 1 , 2 , − 1 ) ∈ R n × n , with eigenvalues λ k = 2 − 2 cos ( k π n + 1 ) \lambda_k = 2 - 2\cos\left(\frac{k\pi}{n+1}\right) λ k = 2 − 2 cos ( n + 1 k π ) , k = 1 , … , n k = 1, \ldots, n k = 1 , … , n (Chapter 2).
(a) Show that κ ( A ) = cot 2 ( π 2 ( n + 1 ) ) \kappa(A) = \cot^2\left(\frac{\pi}{2(n+1)}\right) κ ( A ) = cot 2 ( 2 ( n + 1 ) π ) and that κ ( A ) ≈ 4 ( n + 1 ) 2 π 2 \kappa(A) \approx\frac{4(n+1)^2}{\pi^2} κ ( A ) ≈ π 2 4 ( n + 1 ) 2 for large n n n .
(b) Using the convergence bounds of steepest descent and of CG, estimate the number of iterations each method needs to reduce ∥ e ( k ) ∥ A \lVert e^{(k)} \rVert_A ∥ e ( k ) ∥ A by a factor 10 6 10^6 1 0 6 , for n = 100 n = 100 n = 100 and n = 1000 n = 1000 n = 1000 . You may use ln c + 1 c − 1 ≈ 2 c \ln\frac{c+1}{c-1} \approx\frac{2}{c} ln c − 1 c + 1 ≈ c 2 for large c c c .
(c) For n = 100 n = 100 n = 100 , compare the CG estimate of (b) with the finite-termination property of Proposition 4.1.9. Which of the two is the more useful prediction, and why is there no contradiction?
(d) Run the companion notebook with n = 100 n = 100 n = 100 , monitor ∥ e ( k ) ∥ A \lVert e^{(k)} \rVert_A ∥ e ( k ) ∥ A , and compare the measured iteration counts with (b) and (c).
Companion notebook: week5/script_week5_1.ipynb
Problem 5 (Kantorovich inequality). Let A A A be SPD with condition number κ ( A ) \kappa(A) κ ( A ) . Show that for any v ≠ 0 v \ne0 v = 0 ,
1 ≤ ( v T A v ) ( v T A − 1 v ) ( v T v ) 2 ≤ 1 4 ( κ ( A ) + 1 κ ( A ) ) 2 . 1 \le\frac{(v^{\mathsf{T}} A v)(v^{\mathsf{T}} A^{-1}v)}{(v^{\mathsf{T}}v)^2} \le\frac{1}{4}\left(\sqrt{\kappa(A)} + \frac{1}{\sqrt{\kappa(A)}}\right)^2. 1 ≤ ( v T v ) 2 ( v T A v ) ( v T A − 1 v ) ≤ 4 1 ( κ ( A ) + κ ( A ) 1 ) 2 .
Problem 6 (Orthogonality, A A A -conjugacy, best approximation). Let A A A be an SPD matrix. Prove the following fundamental CG properties:
(a) (Residual orthogonality) r ( i ) T r ( j ) = 0 r^{(i)\mathsf{T}}r^{(j)} = 0 r ( i ) T r ( j ) = 0 for i ≠ j i \ne j i = j .
(b) (A A A -orthogonality of search directions) p ( i ) T A p ( j ) = 0 p^{(i)\mathsf{T}} A p^{(j)} = 0 p ( i ) T A p ( j ) = 0 for i ≠ j i \ne j i = j .
(c) (Best A A A -norm approximation) x ( k ) x^{(k)} x ( k ) minimizes ∥ x − x ∗ ∥ A \lVert x - x^* \rVert_A ∥ x − x ∗ ∥ A over x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)} + K_k(A, r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) .
Problem 7 (Chebyshev bound and CG convergence rate). Let 0 < m = λ min ( A ) ≤ λ ≤ M = λ max ( A ) 0 < m = \lambda_{\min}(A) \le\lambda\le M = \lambda_{\max}(A) 0 < m = λ m i n ( A ) ≤ λ ≤ M = λ m a x ( A ) and κ ( A ) = M / m \kappa(A) = M/m κ ( A ) = M / m .
(a) Show that
∥ e ( k ) ∥ A ∥ e ( 0 ) ∥ A ≤ min p ∈ Π k p ( 0 ) = 1 max λ ∈ [ m , M ] ∣ p ( λ ) ∣ , \frac{\lVert e^{(k)}\rVert_A}{\lVert e^{(0)}\rVert_A} \le\min_{\substack{p\in\Pi_k\\p(0)=1}}\max_{\lambda\in[m,M]} \lvert p(\lambda)\rvert, ∥ e ( 0 ) ∥ A ∥ e ( k ) ∥ A ≤ p ∈ Π k p ( 0 ) = 1 min λ ∈ [ m , M ] max ∣ p ( λ )∣ , where Π k \Pi_k Π k denotes polynomials of degree ≤ k \le k ≤ k .
(b) We will assume as known the Chebyshev optimality property
max λ ∈ [ m , M ] ∣ p k ( λ ) ∣ = min p ∈ Π k p ( 0 ) = 1 max λ ∈ [ m , M ] ∣ p ( λ ) ∣ = 1 T k ( M + m M − m ) . \max_{\lambda\in[m,M]}\lvert p_k(\lambda)\rvert= \min_{\substack{p\in\Pi_k\\p(0)=1}}\max_{\lambda\in[m,M]}\lvert p(\lambda)\rvert= \frac{1}{T_k\left(\frac{M+m}{M-m}\right)}. λ ∈ [ m , M ] max ∣ p k ( λ )∣ = p ∈ Π k p ( 0 ) = 1 min λ ∈ [ m , M ] max ∣ p ( λ )∣ = T k ( M − m M + m ) 1 . where T k T_k T k is the Chebyshev polynomial having the complex representation
T k ( z ) = ( z + z 2 − 1 ) k + ( z − z 2 − 1 ) k 2 . T_k(z)=\frac{\left(z+\sqrt{z^2-1}\right)^k+\left(z-\sqrt{z^2-1}\right)^k}{2}. T k ( z ) = 2 ( z + z 2 − 1 ) k + ( z − z 2 − 1 ) k . Prove the bound
max λ ∈ [ m , M ] ∣ p k ( λ ) ∣ ≤ 2 ( κ ( A ) − 1 κ ( A ) + 1 ) k , \max_{\lambda\in[m,M]}\lvert p_k(\lambda)\rvert\le2\left(\frac{\sqrt{\kappa(A)}-1}{\sqrt{\kappa(A)}+1}\right)^k, λ ∈ [ m , M ] max ∣ p k ( λ )∣ ≤ 2 ( κ ( A ) + 1 κ ( A ) − 1 ) k , and conclude the classical CG estimate
∥ e ( k ) ∥ A ≤ 2 ( κ ( A ) − 1 κ ( A ) + 1 ) k ∥ e ( 0 ) ∥ A . \lVert e^{(k)}\rVert_A \le2\left(\frac{\sqrt{\kappa(A)}-1}{\sqrt{\kappa(A)}+1}\right)^k\lVert e^{(0)}\rVert_A. ∥ e ( k ) ∥ A ≤ 2 ( κ ( A ) + 1 κ ( A ) − 1 ) k ∥ e ( 0 ) ∥ A .
Companion notebook: week5/script_week5_1.ipynb
Problem 8 (Orthogonality of Lanczos vectors). Let A = A T A=A^{\mathsf{T}} A = A T and generate { q j } \{q_j\} { q j } by the Lanczos steps
w ← A q j − β j − 1 q j − 1 , α j ← q j T w , w ← w − α j q j , β j ← ∥ w ∥ 2 , q j + 1 ← w / β j ( β j ≠ 0 ) , \begin{aligned}
w &\leftarrow Aq_j-\beta_{j-1}q_{j-1}, \qquad\alpha_j \leftarrow q_j^{\mathsf{T}}w,\\
w &\leftarrow w-\alpha_jq_j, \qquad\beta_j \leftarrow\lVert w\rVert_2, \qquad q_{j+1}\leftarrow w/\beta_j \quad(\beta_j\ne0),
\end{aligned} w w ← A q j − β j − 1 q j − 1 , α j ← q j T w , ← w − α j q j , β j ← ∥ w ∥ 2 , q j + 1 ← w / β j ( β j = 0 ) , with ∥ q 1 ∥ 2 = 1 \lVert q_1\rVert_2=1 ∥ q 1 ∥ 2 = 1 , q 0 = 0 q_0=0 q 0 = 0 , β 0 = 0 \beta_0=0 β 0 = 0 . Show that, in exact arithmetic, the vectors q 1 , … , q m q_1,\ldots,q_m q 1 , … , q m are orthonormal.
Problem 9 (Networks: Lanczos Ritz values and interlacing). Let A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n be symmetric, and let T m = Q m T A Q m T_m=Q_m^{\mathsf{T}}AQ_m T m = Q m T A Q m be the Lanczos tridiagonal matrix. Show that the eigenvalues of T m T_m T m interlace those of A A A , i.e.
λ i ( A ) ≤ θ i ≤ λ i + n − m ( A ) , \lambda_i(A)\le\theta_i\le\lambda_{i+n-m}(A), λ i ( A ) ≤ θ i ≤ λ i + n − m ( A ) , where θ i \theta_i θ i are the eigenvalues of T m T_m T m . Hint: use the Courant–Fischer characterisation λ i ( A ) = min dim U = i max x ∈ U ∖ { 0 } R A ( x ) = max dim V = n − i + 1 min x ∈ V ∖ { 0 } R A ( x ) \lambda_i(A)=\min_{\dim U=i}\max_{x\in U\setminus\{0\}}R_A(x)=\max_{\dim V=n-i+1}\min_{x\in V\setminus\{0\}}R_A(x) λ i ( A ) = min d i m U = i max x ∈ U ∖ { 0 } R A ( x ) = max d i m V = n − i + 1 min x ∈ V ∖ { 0 } R A ( x ) , with R A ( x ) = x T A x / x T x R_A(x)=x^{\mathsf{T}}Ax/x^{\mathsf{T}}x R A ( x ) = x T A x / x T x , and apply it to T m T_m T m with x = Q m y x=Q_my x = Q m y .
Companion notebook: week5/script_week5_2.ipynb
Krylov Methods for Non-Symmetric Systems
Overview
In Chapter 4 we exploited symmetry to the full but unfortunately none of these results will hold when A A A is non-symmetric. Convection–diffusion discretisations with upwinding, linearised Navier–Stokes Jacobians, transport and reaction networks, and the Google matrix are all non-symmetric and often non-normal , meaning they have no orthogonal eigenbasis at all. This chapter develops the tools for that general setting. Replacing Lanczos by the Arnoldi process yields an upper Hessenberg projection H k H_k H k and a full (long) recurrence; minimising ∥ b − A x ∥ 2 \lVert b-Ax\rVert_2 ∥ b − A x ∥ 2 over x ( 0 ) + K k ( A , r ( 0 ) ) x^{(0)}+\mathcal{K}_k(A,r^{(0)}) x ( 0 ) + K k ( A , r ( 0 ) ) then gives GMRES [39 ] , at a storage and orthogonalisation cost that grows with k k k which is why restarting is needed in practice. The convergence theory changes character too: for non-normal A A A the eigenvalues alone no longer predict behaviour, and one turns to the field of values instead. This chapter applies the methodology to a wider class of problem: (1) Arnoldi and GMRES for A x = b Ax=b A x = b ; (2) directed networks, where the same Arnoldi decomposition is used for eigenvalue computations; (3) rectangular systems from machine learning, where least squares poses the same minimisation but where the Krylov space is built from A A A and A T A^{\mathsf{T}} A T .
By the end of this chapter, you should be able to:
Derive the Arnoldi decomposition A V k = V k + 1 H k AV_k=V_{k+1}H_k A V k = V k + 1 H k , explain why the projection is upper Hessenberg, and recover Lanczos as the symmetric special case.
Formulate GMRES as the least-squares problem min y ∥ β e 1 − H k y ∥ 2 \min_y\lVert\beta e_1-H_ky\rVert_2 min y ∥ β e 1 − H k y ∥ 2 , and state its residual-minimisation, monotonicity, and finite-termination properties.
Express the GMRES residual as q k ( A ) r ( 0 ) q_k(A)r^{(0)} q k ( A ) r ( 0 ) with q k ( 0 ) = 1 q_k(0)=1 q k ( 0 ) = 1 , and use eigenvalue and field-of-values bounds to explain why eigenvalues alone mispredict convergence when A A A is non-normal.
Formulate PageRank as an eigenproblem for the Google matrix, apply G G G without forming it, explain why ∣ λ 2 ( G ) ∣ ≤ α \lvert\lambda_2(G)\rvert\le\alpha ∣ λ 2 ( G )∣ ≤ α bounds the power-iteration rate.
Explain why rectangular A A A forces a Krylov space built from A A A and A T A^{\mathsf{T}} A T , and reduce LSQR to min y ∥ γ e 1 − B k y ∥ 2 \min_y\lVert\gamma e_1-B_ky\rVert_2 min y ∥ γ e 1 − B k y ∥ 2 , mirroring the GMRES methodology.
GMRES and Arnoldi for non-normal matrices
We now wish to solve A x = b Ax=b A x = b when A A A is no longer symmetric. Two routes are possible in this case. CG requires A A A to be SPD, so it does not apply. Forming the normal equations A T A x = A T b A^{\mathsf{T}}Ax=A^{\mathsf{T}}b A T A x = A T b restores symmetry, but squares the condition number and destroys the sparsity of A A A ; § 8.4 will explain why this is a bad idea. What remains is the natural alternative: work directly with A A A , build a Krylov basis of K k ( A , r ( 0 ) ) \mathcal{K}_k(A,r^{(0)}) K k ( A , r ( 0 ) ) , and choose from that subspace the iterate whose residual is smallest in the 2-norm.
Definition 5.1.1 (Krylov subspace and Arnoldi decomposition). Given A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n (not necessarily symmetric) and r ( 0 ) = b − A x ( 0 ) r^{(0)}=b-Ax^{(0)} r ( 0 ) = b − A x ( 0 ) ,
K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } . \mathcal{K}_k(A,r^{(0)})=\operatorname{span}\{r^{(0)},Ar^{(0)},\ldots,A^{k-1}r^{(0)}\}. K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } . The Arnoldi process [2] (with v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 v_1=r^{(0)}/\lVert r^{(0)}\rVert_2 v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 ) builds an orthonormal basis V k + 1 = [ v 1 , … , v k + 1 ] V_{k+1}=[v_1,\ldots,v_{k+1}] V k + 1 = [ v 1 , … , v k + 1 ] and an upper Hessenberg matrix H ‾ k ∈ R ( k + 1 ) × k \underline{H}_k\in\mathbb{R}^{(k+1)\times k} H k ∈ R ( k + 1 ) × k such that
A V k = V k + 1 H ‾ k , H ‾ k = [ H k h k + 1 , k e k T ] . AV_k=V_{k+1}\underline{H}_k,\qquad\underline{H}_k=\begin{bmatrix}H_k\\h_{k+1,k}e_k^{\mathsf{T}}\end{bmatrix}. A V k = V k + 1 H k , H k = [ H k h k + 1 , k e k T ] . where H k ∈ R k × k H_k\in\mathbb{R}^{k\times k} H k ∈ R k × k is upper Hessenberg with entries h i j = ( A v j , v i ) h_{ij}=(Av_j,v_i) h ij = ( A v j , v i ) for 1 ≤ i ≤ j + 1 ≤ k 1\leq i\leq j+1\leq k 1 ≤ i ≤ j + 1 ≤ k .
Figure 5.1. The Arnoldi decomposition A V k = V k + 1 H ‾ k AV_k=V_{k+1}\underline{H}_k A V k = V k + 1 H k , drawn for k = 4 k=4 k = 4 . Applying A A A to the orthonormal basis V k V_k V k of K k \mathcal{K}_k K k never leaves K k + 1 \mathcal{K}_{k+1} K k + 1 , so the result factors through exactly one extra basis vector. Column j j j of H ‾ k \underline{H}_k H k is nonzero only in its first j + 1 j+1 j + 1 entries, which is what upper Hessenberg means; in particular the first column has just two nonzeros. The matrix is ( k + 1 ) × k (k+1)\times k ( k + 1 ) × k , the last row containing the single entry h k + 1 , k h_{k+1,k} h k + 1 , k that carries v k + 1 v_{k+1} v k + 1 . When A A A is symmetric, H ‾ k \underline{H}_k H k is additionally symmetric and therefore tridiagonal, and Arnoldi reduces to the Lanczos algorithm of Chapter 4.
A matrix H = [ h i j ] H=[h_{ij}] H = [ h ij ] is upper Hessenberg if h i j = 0 h_{ij}=0 h ij = 0 whenever i > j + 1 i>j+1 i > j + 1 , i.e. all entries strictly below the first subdiagonal are zero. Example ( k = 6 ) (k=6) ( k = 6 ) :
H = [ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 0 ⋆ ⋆ ⋆ ⋆ 0 0 ⋆ ⋆ ⋆ 0 0 0 ⋆ ⋆ 0 0 0 0 ⋆ ] . H=\begin{bmatrix}
\star&\star&\star&\star&\star\\
\star&\star&\star&\star&\star\\
0&\star&\star&\star&\star\\
0&0&\star&\star&\star\\
0&0&0&\star&\star\\
0&0&0&0&\star
\end{bmatrix}. H = ⋆ ⋆ 0 0 0 0 ⋆ ⋆ ⋆ 0 0 0 ⋆ ⋆ ⋆ ⋆ 0 0 ⋆ ⋆ ⋆ ⋆ ⋆ 0 ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ .
Arnoldi produces such an H k H_k H k because A v j Av_j A v j only has components along v 1 , … , v j + 1 v_1,\ldots,v_{j+1} v 1 , … , v j + 1 .
Large non-symmetric systems were becoming routine in CFD and transport. Memory budgets were counted in megabytes , so one could not afford dense factorizations. Saad and Schultz’s 1986 idea [39 ] was elegantly pragmatic: let Arnoldi produce an orthonormal snapshot of the action of A A A , and within that small window solve the best possible least-squares.
GMRES: minimising the residual over the Krylov space
Arnoldi supplies a basis; it does not yet say which vector in the subspace to pick. GMRES makes the choice that CG made in Chapter 4, but in the 2-norm of the residual rather than the A A A -norm of the error, the latter no longer being available when A A A is not SPD.
Definition 5.1.2 (GMRES — the principle (method definition)). Given A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n , b b b , and an initial guess x ( 0 ) x^{(0)} x ( 0 ) with residual r ( 0 ) = b − A x ( 0 ) r^{(0)} = b - Ax^{(0)} r ( 0 ) = b − A x ( 0 ) , the Generalized Minimal Residual method (GMRES) defines the k k k -th iterate as
x ( k ) = arg min x ∈ x ( 0 ) + K k ( A , r ( 0 ) ) ∥ b − A x ∥ 2 , K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } . x^{(k)} = \underset{x \in x^{(0)} + \mathcal{K}_k(A,r^{(0)})}{\arg\min}\ \lVert b - Ax \rVert_2,\qquad\mathcal{K}_k(A,r^{(0)}) = \operatorname{span}\{r^{(0)},Ar^{(0)},\ldots,A^{k-1}r^{(0)}\}. x ( k ) = x ∈ x ( 0 ) + K k ( A , r ( 0 ) ) arg min ∥ b − A x ∥ 2 , K k ( A , r ( 0 ) ) = span { r ( 0 ) , A r ( 0 ) , … , A k − 1 r ( 0 ) } .
Proposition 5.1.3 (Equivalence of the GMRES definition and the Arnoldi least-squares). Let r ( 0 ) = b − A x ( 0 ) r^{(0)} = b - Ax^{(0)} r ( 0 ) = b − A x ( 0 ) and V k + 1 = [ v 1 , … , v k + 1 ] V_{k+1} = [v_1,\ldots,v_{k+1}] V k + 1 = [ v 1 , … , v k + 1 ] be an orthonormal Arnoldi basis of K k ( A , r ( 0 ) ) \mathcal{K}_k(A,r^{(0)}) K k ( A , r ( 0 ) ) with v 1 = r ( 0 ) ∥ r ( 0 ) ∥ 2 v_1 = \frac{r^{(0)}}{\lVert r^{(0)} \rVert_2} v 1 = ∥ r ( 0 ) ∥ 2 r ( 0 ) and set β = ∥ r ( 0 ) ∥ 2 \beta= \lVert r^{(0)} \rVert_2 β = ∥ r ( 0 ) ∥ 2 . Then the GMRES iterate x ( k ) x^{(k)} x ( k ) is obtained by choosing x ( k ) = x ( 0 ) + V k y ( k ) x^{(k)} = x^{(0)} + V_k y^{(k)} x ( k ) = x ( 0 ) + V k y ( k ) , where y ( k ) y^{(k)} y ( k ) solves the small least-squares
y ( k ) = arg min y ∈ R k ∥ β e 1 − H k y ∥ 2 , y^{(k)} = \underset{y \in\mathbb{R}^k}{\arg\min}\ \lVert\beta e_1 - H_k y \rVert_2, y ( k ) = y ∈ R k arg min ∥ β e 1 − H k y ∥ 2 , and the residual norm satisfies ∥ r ( k ) ∥ 2 = ∥ β e 1 − H k y ( k ) ∥ 2 \lVert r^{(k)} \rVert_2 = \lVert\beta e_1 - H_k y^{(k)} \rVert_2 ∥ r ( k ) ∥ 2 = ∥ β e 1 − H k y ( k ) ∥ 2 .
Proof. Every x ∈ x ( 0 ) + K k ( A , r ( 0 ) ) x \in x^{(0)} + \mathcal{K}_k(A,r^{(0)}) x ∈ x ( 0 ) + K k ( A , r ( 0 ) ) can be written uniquely as x = x ( 0 ) + V k y x = x^{(0)} + V_k y x = x ( 0 ) + V k y for some y ∈ R k y \in\mathbb{R}^k y ∈ R k , because V k V_k V k has orthonormal columns spanning K k \mathcal{K}_k K k . For such x x x ,
r ( y ) : = b − A x = b − A ( x ( 0 ) + V k y ) = r ( 0 ) − A V k y . r(y) := b - Ax = b - A(x^{(0)} + V_k y) = r^{(0)} - AV_k y. r ( y ) := b − A x = b − A ( x ( 0 ) + V k y ) = r ( 0 ) − A V k y .
By Arnoldi, A V k = V k + 1 H k AV_k = V_{k+1}H_k A V k = V k + 1 H k ; also r ( 0 ) = β v 1 = V k + 1 ( β e 1 ) r^{(0)} = \beta v_1 = V_{k+1}(\beta e_1) r ( 0 ) = β v 1 = V k + 1 ( β e 1 ) . Hence
r ( y ) = V k + 1 ( β e 1 − H k y ) . r(y) = V_{k+1}(\beta e_1 - H_k y). r ( y ) = V k + 1 ( β e 1 − H k y ) .
Because V k + 1 V_{k+1} V k + 1 has orthonormal columns, it is an isometry on R k + 1 \mathbb{R}^{k+1} R k + 1 : ∥ V k + 1 z ∥ 2 = ∥ z ∥ 2 \lVert V_{k+1}z \rVert_2 = \lVert z \rVert_2 ∥ V k + 1 z ∥ 2 = ∥ z ∥ 2 for all z ∈ R k + 1 z \in\mathbb{R}^{k+1} z ∈ R k + 1 . Therefore
∥ r ( y ) ∥ 2 = ∥ β e 1 − H k y ∥ 2 . \lVert r(y) \rVert_2 = \lVert\beta e_1 - H_k y \rVert_2. ∥ r ( y ) ∥ 2 = ∥ β e 1 − H k y ∥ 2 .
Minimizing ∥ r ( y ) ∥ 2 \lVert r(y) \rVert_2 ∥ r ( y ) ∥ 2 over x ∈ x ( 0 ) + K k x \in x^{(0)} + \mathcal{K}_k x ∈ x ( 0 ) + K k is thus exactly the ( k + 1 ) × k (k+1) \times k ( k + 1 ) × k least-squares
y ( k ) = arg min y ∥ β e 1 − H k y ∥ 2 , y^{(k)} = \underset{y}{\arg\min}\ \lVert\beta e_1 - H_k y \rVert_2, y ( k ) = y arg min ∥ β e 1 − H k y ∥ 2 ,
and the GMRES iterate is x ( k ) = x ( 0 ) + V k y ( k ) x^{(k)} = x^{(0)} + V_k y^{(k)} x ( k ) = x ( 0 ) + V k y ( k ) with residual norm ∥ r ( k ) ∥ 2 = ∥ β e 1 − H k y ( k ) ∥ 2 \lVert r^{(k)} \rVert_2 = \lVert\beta e_1 - H_k y^{(k)} \rVert_2 ∥ r ( k ) ∥ 2 = ∥ β e 1 − H k y ( k ) ∥ 2 . If h k + 1 , k = 0 h_{k+1,k} = 0 h k + 1 , k = 0 , then H k H_k H k has a zero last row, so the system is square (and upper Hessenberg) and the least-squares residual can be made zero; consequently r ( k ) = 0 r^{(k)} = 0 r ( k ) = 0 and GMRES terminates in k k k steps. ∎
Remark 5.1.4 (Why orthonormality is the key feature). The proof used V k + 1 V_{k+1} V k + 1 only once, but decisively: because its columns are orthonormal it preserves norms, so the n n n -dimensional residual and the ( k + 1 ) (k+1) ( k + 1 ) -dimensional vector β e 1 − H k y \beta e_1 - H_k y β e 1 − H k y have exactly the same length. Had the basis merely been linearly independent, the reduced problem would have acquired a weighting by V k + 1 T V k + 1 V_{k+1}^{\mathsf{T}}V_{k+1} V k + 1 T V k + 1 , and minimising it would no longer minimise the true residual. This is the same reason Lanczos orthonormalises in Chapter 4.
Proposition 5.1.3 turns each GMRES step into a small least-squares problem, but solving it from scratch at every step would be wasteful: H k H_k H k differs from H k − 1 H_{k-1} H k − 1 by a single new column. Maintaining a QR factorisation of H k H_k H k incrementally costs one rotation per step, and returns the residual norm as a by-product.
Definition 5.1.5 (Givens rotations (notation used by GMRES)). For scalars a a a , b b b , a Givens rotation G ( c , s ) G(c,s) G ( c , s ) is the 2 × 2 2 \times2 2 × 2 orthogonal matrix
G ( c , s ) = [ c s − s c ] , c = a a 2 + b 2 , s = b a 2 + b 2 , G(c,s)=\begin{bmatrix} c & s \\ -s & c \end{bmatrix}, \qquad c=\frac{a}{\sqrt{a^2+b^2}}, \qquad s=\frac{b}{\sqrt{a^2+b^2}}, G ( c , s ) = [ c − s s c ] , c = a 2 + b 2 a , s = a 2 + b 2 b , which maps
[ a b ] ↦ [ a 2 + b 2 0 ] . \begin{bmatrix} a \\ b \end{bmatrix}\mapsto\begin{bmatrix} \sqrt{a^2+b^2} \\ 0 \end{bmatrix}. [ a b ] ↦ [ a 2 + b 2 0 ] . In GMRES we apply G k G_k G k to rows k k k and k + 1 k+1 k + 1 of the tall matrix H k H_k H k to zero the subdiagonal entry h k + 1 , k h_{k+1,k} h k + 1 , k , and we apply the same rotation to the right-hand side g = β e 1 g=\beta e_1 g = β e 1 to keep g = Q T ( β e 1 ) g=Q^\mathsf{T}(\beta e_1) g = Q T ( β e 1 ) up to step k k k .
Definition 5.1.6 (Full GMRES (Givens form) — construction). Given x ( 0 ) x^{(0)} x ( 0 ) , set r ( 0 ) = b − A x ( 0 ) r^{(0)}=b-Ax^{(0)} r ( 0 ) = b − A x ( 0 ) , β = ∥ r ( 0 ) ∥ 2 \beta=\lVert r^{(0)}\rVert_2 β = ∥ r ( 0 ) ∥ 2 , v 1 = r ( 0 ) / β v_1=r^{(0)}/\beta v 1 = r ( 0 ) / β . For k = 1 , 2 , … k=1,2,\ldots k = 1 , 2 , … until convergence:
Arnoldi expansion. Compute w = A v k w=Av_k w = A v k . For i = 1 , … , k i=1,\ldots,k i = 1 , … , k :
h i , k = v i T w , w ← w − h i , k v i . h_{i,k}=v_i^\mathsf{T}w,\qquad w\leftarrow w-h_{i,k}v_i. h i , k = v i T w , w ← w − h i , k v i . Set h k + 1 , k = ∥ w ∥ 2 h_{k+1,k}=\lVert w\rVert_2 h k + 1 , k = ∥ w ∥ 2 . If h k + 1 , k = 0 h_{k+1,k}=0 h k + 1 , k = 0 (lucky breakdown), stop; else v k + 1 = w / h k + 1 , k v_{k+1}=w/h_{k+1,k} v k + 1 = w / h k + 1 , k .
Update small LS (Givens). Apply the stored rotations G 1 , … , G k − 1 G_1,\ldots,G_{k-1} G 1 , … , G k − 1 to the new column of H k H_k H k , then form G k G_k G k to annihilate h k + 1 , k h_{k+1,k} h k + 1 , k and apply G k G_k G k to g = β e 1 g=\beta e_1 g = β e 1 ; this maintains the QR of H k H_k H k and yields ∥ r ( k ) ∥ 2 = ∣ g k + 1 ∣ \lVert r^{(k)}\rVert_2=\lvert g_{k+1}\rvert ∥ r ( k ) ∥ 2 = ∣ g k + 1 ∣ .
Residual and test. The current residual norm is
∥ r ( k ) ∥ 2 = ∣ g k + 1 ∣ , \lVert r^{(k)}\rVert_2=\lvert g_{k+1}\rvert, ∥ r ( k ) ∥ 2 = ∣ g k + 1 ∣ , available without forming x ( k ) x^{(k)} x ( k ) . If ∥ r ( k ) ∥ 2 / ∥ b ∥ 2 ≤ t o l \lVert r^{(k)}\rVert_2/\lVert b\rVert_2\leq\mathrm{tol} ∥ r ( k ) ∥ 2 / ∥ b ∥ 2 ≤ tol , recover y ( k ) y^{(k)} y ( k ) by back-substitution in the k × k k\times k k × k upper triangular R R R and set x ( k ) = x ( 0 ) + V k y ( k ) x^{(k)}=x^{(0)}+V_ky^{(k)} x ( k ) = x ( 0 ) + V k y ( k ) .
Proposition 5.1.7 (GMRES basics in one shot). Let A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n , x ( 0 ) x^{(0)} x ( 0 ) be an initial guess, r ( 0 ) = b − A x ( 0 ) r^{(0)}=b-Ax^{(0)} r ( 0 ) = b − A x ( 0 ) , and let V k + 1 V_{k+1} V k + 1 and H k H_k H k come from k k k steps of Arnoldi with v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 v_1=r^{(0)}/\lVert r^{(0)}\rVert_2 v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 and A V k = V k + 1 H k AV_k=V_{k+1}H_k A V k = V k + 1 H k ; set β = ∥ r ( 0 ) ∥ 2 \beta=\lVert r^{(0)}\rVert_2 β = ∥ r ( 0 ) ∥ 2 . The GMRES iterate x ( k ) = x ( 0 ) + V k y ( k ) x^{(k)}=x^{(0)}+V_ky^{(k)} x ( k ) = x ( 0 ) + V k y ( k ) , where y ( k ) = arg min y ∥ β e 1 − H k y ∥ 2 y^{(k)}=\operatorname*{arg\,min}_y\lVert\beta e_1-H_ky\rVert_2 y ( k ) = arg min y ∥ β e 1 − H k y ∥ 2 . This satisfies:
(i) Orthogonality. r ( k ) ⊥ A K k ( A , r ( 0 ) ) r^{(k)}\perp AK_k(A,r^{(0)}) r ( k ) ⊥ A K k ( A , r ( 0 ) ) .
(ii) Monotonicity (full GMRES). ∥ r ( k + 1 ) ∥ 2 ≤ ∥ r ( k ) ∥ 2 \lVert r^{(k+1)}\rVert_2\leq\lVert r^{(k)}\rVert_2 ∥ r ( k + 1 ) ∥ 2 ≤ ∥ r ( k ) ∥ 2 .
(iii) Residual polynomial and finite termination. There exists q k ∈ Π k q_k\in\Pi_k q k ∈ Π k with q k ( 0 ) = 1 q_k(0)=1 q k ( 0 ) = 1 such that r ( k ) = q k ( A ) r ( 0 ) r^{(k)}=q_k(A)r^{(0)} r ( k ) = q k ( A ) r ( 0 ) , and q k q_k q k minimizes ∥ q ( A ) r ( 0 ) ∥ 2 \lVert q(A)r^{(0)}\rVert_2 ∥ q ( A ) r ( 0 ) ∥ 2 over all such polynomials. If m m m is the minimal polynomial of A A A relative to r ( 0 ) r^{(0)} r ( 0 ) (Definition 4.1.8) with deg m = d \deg m=d deg m = d , then GMRES terminates in at most d d d steps (i.e., r ( d ) = 0 r^{(d)}=0 r ( d ) = 0 ).
Proof. See Problem 4.
Figure 5.2. Why eigenvalues are not enough. For a normal matrix the eigenvalues alone control ∥ q ( A ) ∥ 2 = max j ∣ q ( λ j ) ∣ \lVert q(A)\rVert_2 = \max_j |q(\lambda_j)| ∥ q ( A ) ∥ 2 = max j ∣ q ( λ j ) ∣ , so a tight cluster away from the origin guarantees rapid GMRES convergence. For a non-normal matrix the eigenvector basis is ill-conditioned and the relevant object is the field of values F ( A ) = { x ∗ A x : ∥ x ∥ 2 = 1 } F(A) = \{x^*Ax : \lVert x\rVert_2 = 1\} F ( A ) = { x ∗ A x : ∥ x ∥ 2 = 1 } , a convex set containing σ ( A ) \sigma(A) σ ( A ) . If F ( A ) F(A) F ( A ) stays in a disk that excludes the origin, the Crouzeix–Palencia inequality [8] gives a genuine convergence rate; if the origin lies inside F ( A ) F(A) F ( A ) , no such bound is available, and GMRES can stagnate for n − 1 n - 1 n − 1 steps even with all eigenvalues equal to 1 (see the example at the end of this section).
Convergence: why eigenvalues are not enough
For CG, Chapter 4 gave a single convergence bound in terms of κ ( A ) \kappa(A) κ ( A ) , and that was the end of the story. For GMRES there is no such clean result. The natural first attempt is to bound the residual using the eigenvalues of A A A , and it works whenever A A A is diagonalisable with a well-conditioned eigenvector matrix, but it becomes vacuous exactly when A A A is strongly non-normal, which is the case we introduced the method for.
Proposition 5.1.8 (Diagonalizable-A A A bound). If A = V Λ V − 1 A = V\Lambda V^{-1} A = V Λ V − 1 , then
∥ r ( k ) ∥ 2 ∥ r ( 0 ) ∥ 2 ≤ κ 2 ( V ) min q ∈ Π k q ( 0 ) = 1 max i ∣ q ( λ i ) ∣ . \frac{\lVert r^{(k)}\rVert_2}{\lVert r^{(0)}\rVert_2} \le\kappa_2(V)\min_{\substack{q\in\Pi_k\\q(0)=1}}\max_i |q(\lambda_i)|. ∥ r ( 0 ) ∥ 2 ∥ r ( k ) ∥ 2 ≤ κ 2 ( V ) q ∈ Π k q ( 0 ) = 1 min i max ∣ q ( λ i ) ∣.
Proof. r ( k ) = q k ( A ) r ( 0 ) = V q k ( Λ ) V − 1 r ( 0 ) r^{(k)} = q_k(A)r^{(0)} = Vq_k(\Lambda)V^{-1}r^{(0)} r ( k ) = q k ( A ) r ( 0 ) = V q k ( Λ ) V − 1 r ( 0 ) , so ∥ r ( k ) ∥ ≤ ∥ V ∥ ∥ V − 1 ∥ max i ∣ q k ( λ i ) ∣ ∥ r ( 0 ) ∥ \lVert r^{(k)}\rVert\le\lVert V\rVert\lVert V^{-1}\rVert\max_i |q_k(\lambda_i)|\lVert r^{(0)}\rVert ∥ r ( k ) ∥ ≤ ∥ V ∥ ∥ V − 1 ∥ max i ∣ q k ( λ i ) ∣ ∥ r ( 0 ) ∥ .
Definition 5.1.9 (Field of values (numerical range)).
F ( A ) = { x ∗ A x x ∗ x : x ∈ C n ∖ { 0 } } . F(A) = \left\{ \frac{x^*Ax}{x^*x} : x\in\mathbb{C}^n\setminus\{0\} \right\}. F ( A ) = { x ∗ x x ∗ A x : x ∈ C n ∖ { 0 } } .
In words, F ( A ) F(A) F ( A ) is the set of all Rayleigh quotients of A A A . It is compact and convex, and it contains every eigenvalue of A A A , as one sees by taking x x x to be an eigenvector. The point of replacing the spectrum by F ( A ) F(A) F ( A ) is that the latter is insensitive to the conditioning of the eigenvectors: its shape and position control ∥ q ( A ) ∥ 2 \lVert q(A)\rVert_2 ∥ q ( A ) ∥ 2 directly, and hence the GMRES residual through r ( k ) = q k ( A ) r ( 0 ) r^{(k)} = q_k(A)r^{(0)} r ( k ) = q k ( A ) r ( 0 ) .
Proposition 5.1.10 (Disk-in-F ( A ) F(A) F ( A ) bound). If F ( A ) ⊂ { ∣ z − c ∣ ≤ s } F(A) \subset\{ |z-c| \le s\} F ( A ) ⊂ { ∣ z − c ∣ ≤ s } and 0 ∉ { ∣ z − c ∣ ≤ s } 0 \notin\{ |z-c| \le s\} 0 ∈ / { ∣ z − c ∣ ≤ s } , then
∥ r ( k ) ∥ 2 ≤ ( 1 + 2 ) ( s ∣ c ∣ ) k ∥ r ( 0 ) ∥ 2 . \lVert r^{(k)}\rVert_2 \le(1+\sqrt{2})\left(\frac{s}{|c|}\right)^k\lVert r^{(0)}\rVert_2. ∥ r ( k ) ∥ 2 ≤ ( 1 + 2 ) ( ∣ c ∣ s ) k ∥ r ( 0 ) ∥ 2 .
Proof. Crouzeix-Palencia: ∥ q ( A ) ∥ ≤ ( 1 + 2 ) max z ∈ F ( A ) ∣ q ( z ) ∣ \lVert q(A)\rVert\le(1+\sqrt{2})\max_{z\in F(A)}\lvert q(z)\rvert ∥ q ( A )∥ ≤ ( 1 + 2 ) max z ∈ F ( A ) ∣ q ( z )∣ . Test the degree-k k k polynomial q k ( z ) = ( 1 − z / c ) k q_k(z)=(1-z/c)^k q k ( z ) = ( 1 − z / c ) k ; then max F ( A ) ∣ q k ∣ ≤ ( s / ∣ c ∣ ) k \max_{F(A)}\lvert q_k\rvert\le(s/\lvert c\rvert)^k max F ( A ) ∣ q k ∣ ≤ ( s / ∣ c ∣ ) k .
When eigenvalues mislead: a companion matrix. The following example shows how far the spectrum alone can be from the truth. For a monic polynomial p ( z ) = z n + c n − 1 z n − 1 + ⋯ + c 0 p(z)=z^n+c_{n-1}z^{n-1}+\cdots+c_0 p ( z ) = z n + c n − 1 z n − 1 + ⋯ + c 0 , the companion matrix
C ( p ) = [ 0 1 0 ⋯ 0 0 0 1 ⋯ 0 ⋮ ⋮ ⋱ ⋱ ⋮ 0 0 ⋯ 0 1 − c 0 − c 1 ⋯ − c n − 2 − c n − 1 ] C(p)=\begin{bmatrix}
0 & 1 & 0 & \cdots& 0\\
0 & 0 & 1 & \cdots& 0\\
\vdots& \vdots& \ddots& \ddots& \vdots\\
0 & 0 & \cdots& 0 & 1\\
-c_0 & -c_1 & \cdots& -c_{n-2} & -c_{n-1}
\end{bmatrix} C ( p ) = 0 0 ⋮ 0 − c 0 1 0 ⋮ 0 − c 1 0 1 ⋱ ⋯ ⋯ ⋯ ⋯ ⋱ 0 − c n − 2 0 0 ⋮ 1 − c n − 1
has eigenvalues equal to the roots of p p p . These matrices are typically highly nonnormal . Take for example p ( z ) = ( z − 1 ) 3 = z 3 − 3 z 2 + 3 z − 1 p(z)=(z-1)^3=z^3-3z^2+3z-1 p ( z ) = ( z − 1 ) 3 = z 3 − 3 z 2 + 3 z − 1 . Then
A = C ( p ) = [ 0 1 0 0 0 1 1 − 3 3 ] , Λ ( A ) = { 1 , 1 , 1 } . A=C(p)=\begin{bmatrix}
0 & 1 & 0\\
0 & 0 & 1\\
1 & -3 & 3
\end{bmatrix},\qquad\Lambda(A)=\{1,1,1\}. A = C ( p ) = 0 0 1 1 0 − 3 0 1 3 , Λ ( A ) = { 1 , 1 , 1 } .
Run GMRES on A x = f Ax=f A x = f with x ( 0 ) = 0 x^{(0)}=0 x ( 0 ) = 0 and f = e 1 f=e_1 f = e 1 . One observes
∥ r ( 0 ) ∥ 2 = ∥ r ( 1 ) ∥ 2 = ∥ r ( 2 ) ∥ 2 = 1 , ∥ r ( 3 ) ∥ 2 = 0 , \lVert r^{(0)}\rVert_2=\lVert r^{(1)}\rVert_2=\lVert r^{(2)}\rVert_2=1,\qquad\lVert r^{(3)}\rVert_2=0, ∥ r ( 0 ) ∥ 2 = ∥ r ( 1 ) ∥ 2 = ∥ r ( 2 ) ∥ 2 = 1 , ∥ r ( 3 ) ∥ 2 = 0 ,
i.e., no decrease for two steps, then exact termination at k = 3 k=3 k = 3 . Moreover, e 1 T A e 1 = 0 e_1^{\mathsf{T}}Ae_1=0 e 1 T A e 1 = 0 , so 0 ∈ F ( A ) 0\in F(A) 0 ∈ F ( A ) and the disk bound above does not apply here. This illustrates that tightly clustered eigenvalues (even all at 1) do not guarantee fast GMRES if the matrix is nonnormal; the field of values and nonnormality matter. In fact any nonincreasing residual curve whatsoever can be realised by some matrix with any prescribed spectrum [15 ] . A systematic account of non-normal behaviour is given in [41 ] .
Networks: PageRank as an Eigenproblem
The original PageRank algorithm, developed by Larry Page and Sergey Brin in the late 1990s, was essentially an eigenvector computation on the web graph. At the time, the web already had tens of millions of pages, making full eigen-decompositions infeasible. Instead, iterative methods such as power iteration were the only scalable option. This numerical perspective i.e. treating web search as a giant linear algebra problem, was one of the key insights that allowed Google to leap ahead of earlier search engines. It remains a striking example of how classical numerical methods became transformative in an entirely new application domain.
Chapter 4 used the Krylov subspace twice: to solve A x = b Ax=b A x = b and, through Lanczos, to compute eigenvalues of a symmetric matrix. Section 5.1 has just repeated the first of these for non-symmetric A A A ; this section repeats the second. The Arnoldi decomposition A V m = V m + 1 H m AV_m=V_{m+1}H_m A V m = V m + 1 H m is again all we need for the eigenvalues of the small matrix H m H_m H m rather than for a linear solve. The application we envision here is ranking the nodes of a directed graph, where the matrix is non-symmetric by construction and far too large to factorise.
Definition 5.2.1 (Eigenvector centrality (directed graphs)). Let A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n be the (possibly non-symmetric) adjacency matrix. Right eigenvector centrality assigns scores x ≥ 0 x\ge0 x ≥ 0 to nodes by solving
A x = λ 1 x , ∥ x ∥ 1 = 1 , Ax=\lambda_1x,\qquad\lVert x\rVert_1=1, A x = λ 1 x , ∥ x ∥ 1 = 1 , while left eigenvector centrality assigns scores y ≥ 0 y\ge0 y ≥ 0 from A T y = λ 1 y , ∥ y ∥ 1 = 1 A^{\mathsf{T}}y=\lambda_1y,\qquad\lVert y\rVert_1=1 A T y = λ 1 y , ∥ y ∥ 1 = 1 .
The Google matrix Let P ∈ R n × n P \in\mathbb{R}^{n \times n} P ∈ R n × n be the column-stochastic matrix where
P i j = { 1 d j , if page j links to page i , 0 , otherwise , P_{ij} =
\begin{cases}
\frac{1}{d_j}, & \text{if page } j \text{ links to page } i,\\
0, & \text{otherwise},
\end{cases} P ij = { d j 1 , 0 , if page j links to page i , otherwise ,
with d j d_j d j the out-degree of page j j j . The Google matrix is defined as
G = α P + ( 1 − α ) 1 n 11 T , 0 < α < 1. G = \alpha P + (1-\alpha)\frac{1}{n}11^{\mathsf{T}}, \qquad0 < \alpha< 1. G = α P + ( 1 − α ) n 1 1 1 T , 0 < α < 1.
By construction, G G G is stochastic1 and irreducible2 ; the Perron-Frobenius theorem guarantees a unique dominant eigenvector x x x with positive entries such that ∥ x ∥ 1 = 1 \lVert x\rVert_1 = 1 ∥ x ∥ 1 = 1 , called the PageRank vector [27 , 34 ] .
Efficient application of the Google matrix We should notice that G G G is dense : every entry of 11 T 11^{\mathsf{T}} 1 1 T is nonzero, so storing G G G for a web-scale graph is out of the question, but it doesn’t have to be stored. For any x x x with 1 T x = 1 1^{\mathsf{T}}x = 1 1 T x = 1 the rank-one perturbation term acts as a constant vector,
G x = α P x + 1 − α n 1 ( 1 T x ) = α P x + 1 − α n 1 , Gx = \alpha Px + \frac{1-\alpha}{n}1(1^{\mathsf{T}}x) = \alpha Px + \frac{1-\alpha}{n}1, G x = α P x + n 1 − α 1 ( 1 T x ) = α P x + n 1 − α 1 ,
so one application of G G G costs one sparse product with P P P plus a vector addition. The power iteration of Chapter 3 (Algorithm 1) applied to the dominant eigenvector is therefore simply
x ( k + 1 ) = α P x ( k ) + 1 − α n 1 , (10) x^{(k+1)} = \alpha P x^{(k)} + \frac{1-\alpha}{n}1, \tag*{(10)} x ( k + 1 ) = α P x ( k ) + n 1 − α 1 , (10)
one sparse matrix–vector product per step, two vectors of storage, and no factorisation anywhere. This is the entire algorithm that ranked the early web.
Example 5.2.2 (Google Page Rank). Consider a graph of three webpages with links and find which is the most central node
Figure 6 (PDF p. 73).
The adjacency matrix is
A = [ 0 0 1 1 0 1 0 1 0 ] , P = [ 0 0 1 / 2 1 0 1 / 2 0 1 0 ] . A = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \end{bmatrix},
\qquad
P = \begin{bmatrix} 0 & 0 & 1/2 \\ 1 & 0 & 1/2 \\ 0 & 1 & 0 \end{bmatrix}. A = 0 1 0 0 0 1 1 1 0 , P = 0 1 0 0 0 1 1/2 1/2 0 . With damping factor α = 0.85 \alpha= 0.85 α = 0.85 ,
G = α P + 1 − α n 11 T = 0.85 P + 0.05 11 T = [ 0.05 0.05 0.475 0.90 0.05 0.475 0.05 0.90 0.05 ] , (column-stochastic). G = \alpha P + \frac{1-\alpha}{n}11^{\mathsf{T}} = 0.85P + 0.05\,11^{\mathsf{T}} =
\begin{bmatrix}
0.05 & 0.05 & 0.475\\
0.90 & 0.05 & 0.475\\
0.05 & 0.90 & 0.05
\end{bmatrix},
\qquad\text{(column-stochastic).} G = α P + n 1 − α 1 1 T = 0.85 P + 0.05 1 1 T = 0.05 0.90 0.05 0.05 0.05 0.90 0.475 0.475 0.05 , (column-stochastic). Start from the uniform vector x ( 0 ) = 1 3 ( 1 , 1 , 1 ) T x^{(0)} = \frac{1}{3}(1,1,1)^{\mathsf{T}} x ( 0 ) = 3 1 ( 1 , 1 , 1 ) T . Iterating with the power method gives
x ∗ ≈ [ 0.215 0.397 0.388 ] . x^* \approx\begin{bmatrix} 0.215 \\ 0.397 \\ 0.388 \end{bmatrix}. x ∗ ≈ 0.215 0.397 0.388 . hence node 2 is the most central.
The role of the damping factor
First, what α \alpha α means. The Google matrix models a random surfer who, at each step, follows one of the links on the current page with probability α \alpha α , and with probability 1 − α 1-\alpha 1 − α abandons the page and jumps to a page chosen randomly. So α \alpha α is a modelling choice. It is also the only thing that determines the cost of computing the ranking.
Proposition 5.2.3 (Role of the damping factor [21 ] ). Let P P P be column-stochastic, 0 < α < 1 0 < \alpha< 1 0 < α < 1 , and G = α P + 1 − α n 11 T G = \alpha P + \frac{1-\alpha}{n}11^{\mathsf{T}} G = α P + n 1 − α 1 1 T . Then every eigenvalue of G G G other than λ 1 = 1 \lambda_1 = 1 λ 1 = 1 satisfies ∣ λ ∣ ≤ α |\lambda| \le\alpha ∣ λ ∣ ≤ α . Consequently the power iteration (10) converges linearly with asymptotic factor at most α \alpha α , independently of the size of the graph.
Proof. Because G G G is column-stochastic, 1 T G = 1 T 1^{\mathsf{T}}G = 1^{\mathsf{T}} 1 T G = 1 T : the vector 1 1 1 is a left eigenvector for the eigenvalue 1. Let G x = λ x Gx = \lambda x G x = λ x with λ ≠ 1 \lambda\ne1 λ = 1 . Left and right eigenvectors belonging to different eigenvalues are orthogonal, so 1 T x = 0 1^{\mathsf{T}}x = 0 1 T x = 0 . Then we have,
G x = α P x + 1 − α n 1 ( 1 T x ) = α P x , Gx = \alpha Px + \frac{1-\alpha}{n}1(1^{\mathsf{T}}x) = \alpha Px, G x = α P x + n 1 − α 1 ( 1 T x ) = α P x ,
so x x x is also an eigenvector of P P P , with eigenvalue λ / α \lambda/\alpha λ / α . Since P P P is stochastic all its eigenvalues satisfy ∣ μ ∣ ≤ 1 |\mu| \le1 ∣ μ ∣ ≤ 1 , hence ∣ λ ∣ ≤ α |\lambda| \le\alpha ∣ λ ∣ ≤ α .
Two consequences are resulting from this but they pull in opposite directions.
The method is usable at scale. The bound involves α \alpha α and nothing else, in particular not n n n . Doubling the size of the web does not cost a single extra iteration, which is why (10) scaled from millions of pages to billions unchanged.
The model and the cost are linked. Taking α \alpha α close to 1 makes the ranking faithful to the link structure and simultaneously makes the iteration slow: to reach a tolerance ε \varepsilon ε needs about log ε / log α \log\varepsilon/\log\alpha log ε / log α steps, which behaves like log ( 1 / ε ) / ( 1 − α ) \log(1/\varepsilon)/(1-\alpha) log ( 1/ ε ) / ( 1 − α ) as α → 1 \alpha\to1 α → 1 and therefore blows up.
The three-node example makes this concrete. Its link matrix P P P has characteristic polynomial λ 3 − 1 2 λ − 1 2 = ( λ − 1 ) ( λ 2 + λ + 1 2 ) \lambda^3 - \frac{1}{2}\lambda- \frac{1}{2} = (\lambda- 1)(\lambda^2 + \lambda+ \frac{1}{2}) λ 3 − 2 1 λ − 2 1 = ( λ − 1 ) ( λ 2 + λ + 2 1 ) , so besides λ = 1 \lambda= 1 λ = 1 its eigenvalues are ( − 1 ± i ) / 2 (-1 \pm i)/2 ( − 1 ± i ) /2 , of modulus 1 / 2 1/\sqrt{2} 1/ 2 . By the proof above the corresponding eigenvalues of G G G are exactly α \alpha α times these, giving ∣ λ 2 ( G ) ∣ = α / 2 ≈ 0.707 α |\lambda_2(G)| = \alpha/\sqrt{2} \approx0.707\alpha ∣ λ 2 ( G ) ∣ = α / 2 ≈ 0.707 α , which is what Table 5.1 reports. The bound ∣ λ 2 ∣ ≤ α |\lambda_2| \le\alpha ∣ λ 2 ∣ ≤ α is thus attained here up to the factor ∣ λ 2 ( P ) ∣ |\lambda_2(P)| ∣ λ 2 ( P ) ∣ ; on a real web graph ∣ λ 2 ( P ) ∣ |\lambda_2(P)| ∣ λ 2 ( P ) ∣ is very close to 1 and the bound is essentially sharp.
Table 5.1. The damping factor on the three-node example of this section, where ∣ λ 2 ( G ) ∣ = α / 2 |\lambda_2(G)| = \alpha/\sqrt{2} ∣ λ 2 ( G ) ∣ = α / 2 exactly. The last column is the number of iterations observed to gain eight digits, in agreement with the asymptotic estimate 8 log 10 / log ( 1 / ∣ λ 2 ( G ) ∣ ) 8\log10/\log(1/|\lambda_2(G)|) 8 log 10/ log ( 1/∣ λ 2 ( G ) ∣ ) . The value α = 0.85 \alpha= 0.85 α = 0.85 used by Google is a compromise between fidelity to the link structure and cost, not a mathematical constant.
Beyond the power iteration: the Arnoldi process
Power iteration is attractive because (10) is one line of code, but it delivers a single eigenvector and its rate is fixed at α \alpha α : it cannot be accelerated, and it has no way of certifying its answer.
The remedy is the Arnoldi process, which we run on A A A (here A = G A=G A = G , or a shifted problem), and from the decomposition A V m = V m + 1 H m AV_m=V_{m+1}H_m A V m = V m + 1 H m take an eigenpair ( θ , y ) (\theta,y) ( θ , y ) of the square part H m H_m H m . The Ritz pair is then ( θ , u ) (\theta,u) ( θ , u ) with u = V m y u=V_my u = V m y , and the associated residual is available for free,
∥ A u − θ u ∥ 2 = ∣ h m + 1 , m ∣ ∣ e m T y ∣ , (11) \lVert Au-\theta u\rVert_2=|h_{m+1,m}||e_m^{\mathsf{T}}y|, \tag*{(11)} ∥ A u − θ u ∥ 2 = ∣ h m + 1 , m ∣∣ e m T y ∣ , (11)
because
A u − θ u = ( A V m − V m H m ) y = h m + 1 , m v m + 1 ( e m T y ) Au-\theta u=(AV_m-V_mH_m)y=h_{m+1,m}v_{m+1}(e_m^{\mathsf{T}}y) A u − θ u = ( A V m − V m H m ) y = h m + 1 , m v m + 1 ( e m T y )
and ∥ v m + 1 ∥ 2 = 1 \lVert v_{m+1}\rVert_2=1 ∥ v m + 1 ∥ 2 = 1 .
Takeaways
PageRank is the dominant eigenvector of the Google matrix G G G ; Perron–Frobenius guarantees it is unique and positive.
G G G is dense but never formed: one application costs one sparse product with P P P , so the power iteration (10) is one line of code.
The damping factor is the single difficult parameter; ∣ λ 2 ( G ) ∣ ≤ α |\lambda_2(G)|\le\alpha ∣ λ 2 ( G ) ∣ ≤ α bounds the convergence rate independently of the size of the graph.
Power iteration gives one eigenpair at a fixed rate and no error guarantees; in turn, Arnoldi i.e. the same decomposition built in § 8.2 gives several Ritz pairs at once, plus the residual ∣ h m + 1 , m ∣ ∣ e m T y ∣ |h_{m+1,m}||e_m^{\mathsf{T}}y| ∣ h m + 1 , m ∣∣ e m T y ∣ for free.
Krylov Methods for Large Least Squares (optional)
Section 5.1 solved A x = b Ax=b A x = b for square non-symmetric A A A by minimising ∥ b − A x ∥ 2 \lVert b-Ax\rVert_2 ∥ b − A x ∥ 2 over a Krylov subspace. Least squares asks for the minimiser of the same quantity when A A A is rectangular, and this section is about what changes. The answer is a single structural point: with A A A rectangular the sequence r ( 0 ) , A r ( 0 ) , A 2 r ( 0 ) , … r^{(0)},Ar^{(0)},A^2r^{(0)},\ldots r ( 0 ) , A r ( 0 ) , A 2 r ( 0 ) , … does not even make sense, because A r ( 0 ) Ar^{(0)} A r ( 0 ) lives in the wrong space. The Krylov space must be built by alternating A A A and A T A^{\mathsf{T}} A T , and everything else follows from this single change.3
A motivating example: sparse regression on text Consider a linear model fitted to a corpus of documents: row i i i of the design matrix X X X holds the word counts of document i i i over a vocabulary of n n n terms, and b i b_i b i is a response, as for example a rating in recommender systems, a click-through rate, or a price. Fitting by least squares means solving
min x ∥ X x − b ∥ 2 , X ∈ R m × n , \min_x \ \lVert Xx-b\rVert_2,\qquad X\in\mathbb{R}^{m\times n}, x min ∥ X x − b ∥ 2 , X ∈ R m × n ,
with m m m in the millions and n n n in the tens of thousands. Three features of this problem decide the choice of algorithm, and all of them are frequently encountered (recommender systems, genomics, and inverse problems).
In what follows will briefly explain why the obvious route fails.
Figure 5.3. Least squares as orthogonal projection. The minimiser of ∥ A x − b ∥ 2 \lVert Ax-b\rVert_2 ∥ A x − b ∥ 2 is the point of the column space R ( A ) \mathcal{R}(A) R ( A ) closest to b b b , so the residual is orthogonal to R ( A ) \mathcal{R}(A) R ( A ) , which leads exactly to the normal equations A T A x = A T b A^{\mathsf{T}}Ax=A^{\mathsf{T}}b A T A x = A T b . LSQR realises the same projection without forming A T A A^{\mathsf{T}}A A T A .
As explained in Chapter 4, the first ideas would be to solve min x ∥ X x − b ∥ 2 \min_x \lVert Xx-b\rVert_2 min x ∥ X x − b ∥ 2 by differentiating: the minimiser satisfies the normal equations X T X x = X T b X^{\mathsf{T}}Xx=X^{\mathsf{T}}b X T X x = X T b , a square SPD system of size n n n , to which the CG of Chapter 4 applies immediately. Indeed Section 4.3 did precisely this: it read least squares as an SPD problem and ran CG on X T X X^{\mathsf{T}}X X T X in order to exhibit the filter factors 1 − q k ( σ i 2 ) 1-q_k(\sigma_i^2) 1 − q k ( σ i 2 ) . That was the right way to understand regularisation, because the filtering is visible only in the eigenbasis of X T X X^{\mathsf{T}}X X T X . It is not the right way to compute . At any serious scale one never forms X T X X^{\mathsf{T}}X X T X , and never runs CG on it, for two independent reasons.
Sparsity is destroyed. Two terms contribute a nonzero to ( X T X ) j k (X^{\mathsf{T}}X)_{jk} ( X T X ) j k as soon as they co-occur in a single document, so the Gram matrix is far denser than X X X . On a representative sparse design matrix, for example 20 000 × 5000 20\,000 \times5000 20 000 × 5000 at 0.2 % 0.2\% 0.2% density, forming X T X X^{\mathsf{T}}X X T X produces a matrix that is 7.7 % 7.7\% 7.7% dense, nearly ten times as many nonzeros as X X X itself, and this ratio grows with the corpus.
Conditioning is squared. Since the singular values of X T X X^{\mathsf{T}}X X T X are σ i 2 \sigma_i^2 σ i 2 ,
κ 2 ( X T X ) = κ 2 ( X ) 2 . \kappa_2(X^{\mathsf{T}}X)=\kappa_2(X)^2. κ 2 ( X T X ) = κ 2 ( X ) 2 .
Running CG on X T X X^{\mathsf{T}}X X T X (the method known as CGNR) therefore converges at a rate governed by κ 2 ( X ) \kappa_2(X) κ 2 ( X ) rather than κ 2 ( X ) \sqrt{\kappa_2(X)} κ 2 ( X ) , and in floating point the damage can be worse than slow: information present in X X X can be destroyed by the time X T X X^{\mathsf{T}}X X T X has been formed.
A classical warning. Let ε > 0 \varepsilon> 0 ε > 0 and
X = [ 1 1 ε 0 0 ε ] , X T X = [ 1 + ε 2 1 1 1 + ε 2 ] . X = \begin{bmatrix} 1 & 1 \\ \varepsilon& 0 \\ 0 & \varepsilon\end{bmatrix}, \qquad X^{\mathsf{T}}X = \begin{bmatrix} 1+\varepsilon^2 & 1 \\ 1 & 1+\varepsilon^2 \end{bmatrix}. X = 1 ε 0 1 0 ε , X T X = [ 1 + ε 2 1 1 1 + ε 2 ] . X X X has rank 2 for every ε ≠ 0 \varepsilon\ne0 ε = 0 . But if ε < u \varepsilon< \sqrt{u} ε < u , where u ≈ 2.2 ⋅ 10 − 16 u \approx2.2 \cdot10^{-16} u ≈ 2.2 ⋅ 1 0 − 16 is the unit roundoff, then 1 + ε 2 1+\varepsilon^2 1 + ε 2 rounds to 1 and the Gram matrix is [ 1 1 1 1 ] \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} [ 1 1 1 1 ] , so exactly singular. In double precision this already happens at ε = 10 − 9 \varepsilon= 10^{-9} ε = 1 0 − 9 : the rank information is annihilated by forming X T X X^{\mathsf{T}}X X T X .
What we want, then, is a method with the convergence of CG on X T X X^{\mathsf{T}}X X T X but the conditioning of X X X , that means one that uses X X X and X T X^{\mathsf{T}} X T only as operators and never multiplies them together. That is exactly what Golub–Kahan bidiagonalisation provides [12]. From here on we revert to the neutral notation A ∈ R m × n A \in\mathbb{R}^{m \times n} A ∈ R m × n for the rectangular matrix, to keep the parallel with § 8.2 visible.
Golub–Kahan bidiagonalisation and LSQR
Definition 5.3.1 (Golub–Kahan bidiagonalization (core recurrence)). Choose u 1 = b ∥ b ∥ 2 u_1 = \frac{b}{\lVert b\rVert_2} u 1 = ∥ b ∥ 2 b and set β 0 = 0 \beta_0 = 0 β 0 = 0 , v 0 = 0 v_0 = 0 v 0 = 0 . For k = 1 , 2 , … k = 1,2,\ldots k = 1 , 2 , … :
(i) r = A T u k − β k − 1 v k − 1 r = A^{\mathsf{T}}u_k - \beta_{k-1}v_{k-1} r = A T u k − β k − 1 v k − 1 , α k = ∥ r ∥ 2 \alpha_k = \lVert r\rVert_2 α k = ∥ r ∥ 2 , v k = r / α k v_k = r/\alpha_k v k = r / α k ,
(ii) p = A v k − α k u k p = Av_k - \alpha_k u_k p = A v k − α k u k , β k = ∥ p ∥ 2 \beta_k = \lVert p\rVert_2 β k = ∥ p ∥ 2 , u k + 1 = p / β k u_{k+1} = p/\beta_k u k + 1 = p / β k .
Since A v k = α k u k + β k u k + 1 Av_k = \alpha_k u_k + \beta_k u_{k+1} A v k = α k u k + β k u k + 1 , we obtain A V k = U k + 1 B k AV_k = U_{k+1}B_k A V k = U k + 1 B k with B k ∈ R ( k + 1 ) × k B_k \in\mathbb{R}^{(k+1)\times k} B k ∈ R ( k + 1 ) × k lower bidiagonal, carrying α 1 , … , α k \alpha_1,\ldots,\alpha_k α 1 , … , α k on the diagonal and β 1 , … , β k \beta_1,\ldots,\beta_k β 1 , … , β k on the subdiagonal.
Figure 5.4. Golub–Kahan bidiagonalisation. Two orthonormal sequences are grown in alternation, one in the row space and one in the column space, each step costing one product with A A A and one with A T A^{\mathsf{T}} A T . The coefficients assemble into a lower bidiagonal B k B_k B k with A V k = U k + 1 B k AV_k = U_{k+1}B_k A V k = U k + 1 B k . This is Lanczos applied implicitly to A T A A^{\mathsf{T}}A A T A but the squaring never happens numerically, which is precisely why LSQR is stable where CG on the normal equations is not.
The reason this construction is worth the trouble is that it reduces the least-squares problem to a tiny one of exactly the shape met in § 8.2 .
Proposition 5.3.2 (LSQR as a small least-squares problem). Write γ = ∥ b ∥ 2 \gamma= \lVert b\rVert_2 γ = ∥ b ∥ 2 , so that u 1 = b / γ u_1 = b/\gamma u 1 = b / γ , and let k k k steps of Golub–Kahan produce V k V_k V k , U k + 1 U_{k+1} U k + 1 and the bidiagonal B k ∈ R ( k + 1 ) × k B_k \in\mathbb{R}^{(k+1)\times k} B k ∈ R ( k + 1 ) × k with A V k = U k + 1 B k AV_k = U_{k+1}B_k A V k = U k + 1 B k . Then for any x = V k y x = V_k y x = V k y ,
∥ b − A x ∥ 2 = ∥ γ e 1 − B k y ∥ 2 . \lVert b - Ax\rVert_2 = \lVert\gamma e_1 - B_k y\rVert_2. ∥ b − A x ∥ 2 = ∥ γ e 1 − B k y ∥ 2 . and consequently the minimiser of ∥ b − A x ∥ 2 \lVert b - Ax\rVert_2 ∥ b − A x ∥ 2 over x ∈ R ( V k ) x \in\mathcal{R}(V_k) x ∈ R ( V k ) is x ( k ) = V k y ( k ) x^{(k)} = V_k y^{(k)} x ( k ) = V k y ( k ) with
y ( k ) = arg min y ∈ R k ∥ γ e 1 − B k y ∥ 2 . y^{(k)} = \underset{y\in\mathbb{R}^k}{\operatorname{arg\,min}}\lVert\gamma e_1 - B_k y\rVert_2. y ( k ) = y ∈ R k arg min ∥ γ e 1 − B k y ∥ 2 .
Proof. Since b = γ u 1 = U k + 1 ( γ e 1 ) b = \gamma u_1 = U_{k+1}(\gamma e_1) b = γ u 1 = U k + 1 ( γ e 1 ) and A V k y = U k + 1 B k y AV_k y = U_{k+1}B_k y A V k y = U k + 1 B k y ,
b − A V k y = U k + 1 ( γ e 1 − B k y ) . b - AV_k y = U_{k+1}(\gamma e_1 - B_k y). b − A V k y = U k + 1 ( γ e 1 − B k y ) .
The columns of U k + 1 U_{k+1} U k + 1 are orthonormal, so U k + 1 U_{k+1} U k + 1 is an isometry on R k + 1 \mathbb{R}^{k+1} R k + 1 and the norm is unchanged. Minimising over y y y gives the second claim. □ \square □
Compare this with Proposition 5.1.3 : GMRES minimises ∥ β e 1 − H k y ∥ 2 \lVert\beta e_1 - H_k y\rVert_2 ∥ β e 1 − H k y ∥ 2 against an upper Hessenberg H k H_k H k , LSQR minimises ∥ γ e 1 − B k y ∥ 2 \lVert\gamma e_1 - B_k y\rVert_2 ∥ γ e 1 − B k y ∥ 2 against a bidiagonal B k B_k B k . The algorithms differ only in how the small matrix is generated and, consequently, in how cheaply it is factorised: a bidiagonal B k B_k B k is reduced to triangular form by a single Givens rotation per step. This is what allows the LSQR iterate to be updated by short recurrences in O ( m + n ) O(m+n) O ( m + n ) work and storage, with no need to keep V k V_k V k . Mathematically it produces the same iterates as CG on A T A A^\mathsf{T}A A T A , but the squaring never happens numerically, which is why it is stable where CGNR is not.
Early stopping and the truncated SVD
Figure 5.5. LSQR as an iterative truncated SVD. Because the Krylov space is built from A T b A^\mathsf{T}b A T b and grows by powers of A T A A^\mathsf{T}A A T A , components associated with large singular values are resolved first. Stopping early therefore acts as a spectral filter, suppressing the small-σ i \sigma_i σ i directions that magnify noise in an ill-posed problem. The iteration count plays the role of a regularisation parameter, the same mechanism as truncating the SVD, but obtained without ever computing one.
Running ℓ = k + p \ell= k+p ℓ = k + p steps of Golub–Kahan on a sparse term–document matrix M M M and computing the (tiny) SVD B ℓ = U ^ Σ ^ V ^ T B_\ell= \hat{U}\hat{\Sigma}\hat{V}^{\mathsf{T}} B ℓ = U ^ Σ ^ V ^ T gives approximate singular triplets of M M M through U ≈ U ℓ + 1 U ^ : , 1 : k U \approx U_{\ell+1}\hat{U}_{:,1:k} U ≈ U ℓ + 1 U ^ : , 1 : k and V ≈ V ℓ V ^ : , 1 : k V \approx V_\ell\hat{V}_{:,1:k} V ≈ V ℓ V ^ : , 1 : k , with a small oversampling p p p to improve accuracy of the last few. This is latent semantic analysis : the leading singular directions of M M M define a low-dimensional semantic space in which related documents cluster, and it is obtained from a few dozen matrix–vector products rather than a dense O ( m n min ( m , n ) ) O(mn\min(m,n)) O ( mn min ( m , n )) factorisation.
Figure 5.6. After truncated SVD, documents cluster in a low-dimensional semantic space.
Takeaways
Rectangular A A A forces the Krylov space to be built from A A A and A T A^{\mathsf{T}} A T ; Golub–Kahan bidiagonalisation is the resulting recurrence.
LSQR reduces the least-squares problem to min y ∥ γ e 1 − B k y ∥ 2 \min_y \lVert\gamma e_1 - B_k y \rVert_2 min y ∥ γ e 1 − B k y ∥ 2 , structurally identical to the GMRES problem of § 8.2 , but with a bidiagonal in place of a Hessenberg matrix.
Never form A T A A^{\mathsf{T}} A A T A : the geometry of the normal equations is right, the arithmetic is not.
The same bidiagonalisation, stopped early, acts as a spectral filter (regularisation); kept and factorised, it yields a truncated SVD.
Exercises
Problem 1 (Krylov spaces, resit exam 2024/2025). Consider the matrix
A = [ 2 − 1 0 − 1 2 − 1 0 − 1 2 ] , b = [ 1 0 1 ] . A = \begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{bmatrix}, \qquad b = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}. A = 2 − 1 0 − 1 2 − 1 0 − 1 2 , b = 1 0 1 .
(a) Define the Krylov space K k ( A , b ) K_k(A,b) K k ( A , b ) for k = 1 , 2 , … k=1,2,\ldots k = 1 , 2 , … .
(b) Compute the Krylov spaces K 1 ( A , b ) K_1(A,b) K 1 ( A , b ) , K 2 ( A , b ) K_2(A,b) K 2 ( A , b ) , and K 3 ( A , b ) K_3(A,b) K 3 ( A , b ) . Determine their dimensions.
(c) Explain the general principle of the GMRES method for solving A x = b Ax=b A x = b . How many iterations are needed for convergence?
Problem 2 (Companion matrices and GMRES convergence, exam October 2024). Let
A = [ 0 1 0 0 1 ⋱ ⋱ 0 1 c 0 c 1 ⋯ c n − 2 c n − 1 ] . A = \begin{bmatrix}
0 & 1 & & 0 \\
& 0 & 1 & \\
& & \ddots& \ddots\\
& & 0 & 1 \\
c_0 & c_1 & \cdots& c_{n-2} & c_{n-1}
\end{bmatrix}. A = 0 c 0 1 0 c 1 1 ⋱ 0 ⋯ 0 ⋱ 1 c n − 2 c n − 1 .
and the polynomial p ( z ) = z n − ∑ j = 0 n − 1 c j z j p(z)=z^n-\sum_{j=0}^{n-1}c_jz^j p ( z ) = z n − ∑ j = 0 n − 1 c j z j . Show that:
(a) p ( z ) p(z) p ( z ) is its characteristic and minimal polynomial.
(b) Let
A = [ 0 1 0 0 0 1 1 − 3 3 ] , p ( z ) = ( z − 1 ) 3 = z 3 − 3 z 2 + 3 z − 1 , b = e 1 = [ 1 0 0 ] . A = \begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & -3 & 3 \end{bmatrix}, \qquad p(z)=(z-1)^3=z^3-3z^2+3z-1, \qquad b=e_1=\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}. A = 0 0 1 1 0 − 3 0 1 3 , p ( z ) = ( z − 1 ) 3 = z 3 − 3 z 2 + 3 z − 1 , b = e 1 = 1 0 0 .
Compute the Krylov spaces K k ( A , b ) K_k(A,b) K k ( A , b ) , k = 1 , 2 , 3 k=1,2,3 k = 1 , 2 , 3 .
(c) Compute the GMRES iterates u k u_k u k for A u = b Au=b A u = b with u 0 = 0 u_0=0 u 0 = 0 and conclude on how many iterations are needed for convergence.
(d) Can this result be generalised to an n × n n \times n n × n companion matrix? Explain your answer.
Companion notebook: week6/script_week6_3.ipynb
Problem 3 (Arnoldi and GMRES by hand). Let
A = [ 1 2 0 0 1 2 0 0 1 ] , b = e 3 = [ 0 0 1 ] , x ( 0 ) = 0. A = \begin{bmatrix} 1 & 2 & 0 \\ 0 & 1 & 2 \\ 0 & 0 & 1 \end{bmatrix}, \qquad b=e_3=\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}, \qquad x^{(0)}=0. A = 1 0 0 2 1 0 0 2 1 , b = e 3 = 0 0 1 , x ( 0 ) = 0.
(a) Run two steps of the Arnoldi process from v 1 = b v_1=b v 1 = b . Give v 2 v_2 v 2 , v 3 v_3 v 3 and the 3 × 2 3 \times2 3 × 2 matrix H 2 H_2 H 2 , and check that A V 2 = V 3 H 2 AV_2=V_3H_2 A V 2 = V 3 H 2 .
(b) Using Proposition 5.1.3 , compute the GMRES iterates x ( 1 ) x^{(1)} x ( 1 ) , x ( 2 ) x^{(2)} x ( 2 ) and the residual norms ∥ r ( 1 ) ∥ 2 , ∥ r ( 2 ) ∥ 2 \lVert r^{(1)}\rVert_2,\lVert r^{(2)}\rVert_2 ∥ r ( 1 ) ∥ 2 , ∥ r ( 2 ) ∥ 2 . For k = 1 k=1 k = 1 , check that the Givens rotation which annihilates h 21 h_{21} h 21 returns the same residual norm as ∣ g 2 ∣ \lvert g_2\rvert ∣ g 2 ∣ .
(c) Show that x ( 3 ) x^{(3)} x ( 3 ) is the exact solution and compute it. (d) All eigenvalues of A A A equal 1. For which matrix with this spectrum would GMRES converge in one step? Show, with x = 1 2 ( 1 , − 2 , 1 ) T x = \frac{1}{2}(1,-\sqrt{2},1)^{\mathsf{T}} x = 2 1 ( 1 , − 2 , 1 ) T , that 0 ∈ F ( A ) 0 \in F(A) 0 ∈ F ( A ) , and explain why the disk-in-F ( A ) F(A) F ( A ) bound gives no information here.
Companion notebook: week6/script_week6_3.ipynb
Problem 4 (GMRES basics in one shot). Let A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n , x ( 0 ) x^{(0)} x ( 0 ) be an initial guess, r ( 0 ) = b − A x ( 0 ) r^{(0)} = b - Ax^{(0)} r ( 0 ) = b − A x ( 0 ) , and let V k + 1 V_{k+1} V k + 1 and H ‾ k \underline{H}_k H k come from k k k steps of Arnoldi with v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 v_1 = r^{(0)}/\lVert r^{(0)}\rVert_2 v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 and A V k = V k + 1 H ‾ k AV_k = V_{k+1}\underline{H}_k A V k = V k + 1 H k ; set β = ∥ r ( 0 ) ∥ 2 \beta= \lVert r^{(0)}\rVert_2 β = ∥ r ( 0 ) ∥ 2 . The GMRES iterate x ( k ) = x ( 0 ) + V k y ( k ) x^{(k)} = x^{(0)} + V_k y^{(k)} x ( k ) = x ( 0 ) + V k y ( k ) , where y ( k ) = arg min y ∥ β e 1 − H ‾ k y ∥ 2 y^{(k)} = \mathop{\operatorname{arg\,min}}_y \lVert\beta e_1 - \underline{H}_k y\rVert_2 y ( k ) = arg min y ∥ β e 1 − H k y ∥ 2 . Prove that it satisfies:
(i) Orthogonality. r ( k ) ⊥ A K k ( A , r ( 0 ) ) r^{(k)} \perp AK_k(A,r^{(0)}) r ( k ) ⊥ A K k ( A , r ( 0 ) ) .
(ii) Monotonicity (full GMRES). ∥ r ( k + 1 ) ∥ 2 ≤ ∥ r ( k ) ∥ 2 \lVert r^{(k+1)}\rVert_2 \le\lVert r^{(k)}\rVert_2 ∥ r ( k + 1 ) ∥ 2 ≤ ∥ r ( k ) ∥ 2 .
(iii) Residual polynomial and finite termination. There exists q k ∈ Π k q_k \in\Pi_k q k ∈ Π k with q k ( 0 ) = 1 q_k(0) = 1 q k ( 0 ) = 1 such that r ( k ) = q k ( A ) r ( 0 ) r^{(k)} = q_k(A)r^{(0)} r ( k ) = q k ( A ) r ( 0 ) , and q k q_k q k minimizes ∥ q ( A ) r ( 0 ) ∥ 2 \lVert q(A)r^{(0)}\rVert_2 ∥ q ( A ) r ( 0 ) ∥ 2 over all such polynomials. If m m m is the minimal polynomial of A A A relative to r ( 0 ) r^{(0)} r ( 0 ) (Definition 4.1.8) with deg m = d \deg m = d deg m = d , then GMRES terminates in at most d d d steps (i.e., r ( d ) = 0 r^{(d)} = 0 r ( d ) = 0 ).
Problem 5 (Field of values and GMRES convergence). Let A ∈ C n × n A \in\mathbb{C}^{n \times n} A ∈ C n × n and suppose its field of values (numerical range) F ( A ) F(A) F ( A ) is contained in a disk { z : ∣ z − c ∣ ≤ s } \{z : \lvert z-c\rvert\le s\} { z : ∣ z − c ∣ ≤ s } with 0 ∉ { z : ∣ z − c ∣ ≤ s } 0 \notin\{z : \lvert z-c\rvert\le s\} 0 ∈ / { z : ∣ z − c ∣ ≤ s } . Let r ( k ) r^{(k)} r ( k ) denote the k k k -th GMRES residual for A x = b Ax=b A x = b with some initial guess x ( 0 ) x^{(0)} x ( 0 ) and r ( 0 ) = b − A x ( 0 ) r^{(0)}=b-Ax^{(0)} r ( 0 ) = b − A x ( 0 ) .
(a) Using Problem 4(iii), show that ∥ r ( k ) ∥ 2 ≤ ∥ p ( A ) ∥ 2 ∥ r ( 0 ) ∥ 2 \lVert r^{(k)}\rVert_2 \le\lVert p(A)\rVert_2 \lVert r^{(0)}\rVert_2 ∥ r ( k ) ∥ 2 ≤ ∥ p ( A ) ∥ 2 ∥ r ( 0 ) ∥ 2 for every p ∈ Π k p \in\Pi_k p ∈ Π k with p ( 0 ) = 1 p(0)=1 p ( 0 ) = 1 .
(b) Prove the “disk-in-F ( A ) F(A) F ( A ) ” bound
∥ r ( k ) ∥ 2 ≤ C ( s ∣ c ∣ ) k ∥ r ( 0 ) ∥ 2 , \lVert r^{(k)}\rVert_2 \le C\left(\frac{s}{\lvert c\rvert}\right)^k \lVert r^{(0)}\rVert_2, ∥ r ( k ) ∥ 2 ≤ C ( ∣ c ∣ s ) k ∥ r ( 0 ) ∥ 2 ,
for a universal constant C C C . (Hint: Use Crouzeix–Palencia: ∥ p ( A ) ∥ ≤ ( 1 + 2 ) max z ∈ F ( A ) ∣ p ( z ) ∣ \lVert p(A)\rVert\le(1+\sqrt{2})\max_{z\in F(A)}\lvert p(z)\rvert ∥ p ( A )∥ ≤ ( 1 + 2 ) max z ∈ F ( A ) ∣ p ( z )∣ and test p ( z ) = ( 1 − z / c ) k p(z)=(1-z/c)^k p ( z ) = ( 1 − z / c ) k .)
(c) Explain briefly why this bound can be more descriptive than eigenvalue-only bounds for highly non-normal matrices. Give a concrete example of a matrix with tightly clustered eigenvalues but slow GMRES (Problems 2 and 3 contain two) and relate this to the geometry of F ( A ) F(A) F ( A ) .
Problem 6 (Arnoldi, Ritz values, and a posteriori eigen-residuals). Let A ∈ C n × n A \in\mathbb{C}^{n \times n} A ∈ C n × n and suppose k k k steps of Arnoldi with v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 v_1 = r^{(0)}/\lVert r^{(0)}\rVert_2 v 1 = r ( 0 ) / ∥ r ( 0 ) ∥ 2 produce
A V k = V k + 1 H ‾ k , V k + 1 = [ v 1 ⋯ v k + 1 ] , H ‾ k = [ H k h k + 1 , k e k T ] , AV_k = V_{k+1}\underline{H}_k,\qquad V_{k+1} = [v_1\ \cdots\ v_{k+1}],\qquad\underline{H}_k =
\begin{bmatrix}
H_k\\
h_{k+1,k}e_k^{\mathsf{T}}
\end{bmatrix}, A V k = V k + 1 H k , V k + 1 = [ v 1 ⋯ v k + 1 ] , H k = [ H k h k + 1 , k e k T ] ,
where V k + 1 V_{k+1} V k + 1 has orthonormal columns, H ‾ k ∈ C ( k + 1 ) × k \underline{H}_k \in\mathbb{C}^{(k+1)\times k} H k ∈ C ( k + 1 ) × k is upper Hessenberg, and H k ∈ C k × k H_k \in\mathbb{C}^{k\times k} H k ∈ C k × k is its leading square part.
(a) Show that if ( θ , y ) (\theta,y) ( θ , y ) is an eigenpair of H k H_k H k with ∥ y ∥ 2 = 1 \lVert y\rVert_2=1 ∥ y ∥ 2 = 1 , then ( θ , v ) (\theta,v) ( θ , v ) with v : = V k y v:=V_ky v := V k y is a Ritz pair for A A A (i.e. θ \theta θ is a Ritz value and v v v a Ritz vector).
(b) Prove the a posteriori residual formula
∥ A v − θ v ∥ 2 = ∣ h k + 1 , k ∣ ∣ e k T y ∣ . \lVert Av-\theta v\rVert_2 = \lvert h_{k+1,k}\rvert\lvert e_k^{\mathsf{T}}y\rvert. ∥ A v − θ v ∥ 2 = ∣ h k + 1 , k ∣ ∣ e k T y ∣ .
(c) Interpret this bound: under what condition is θ \theta θ a good eigenvalue approximation to A A A ? How does this relate to (i) small subdiagonal element ∣ h k + 1 , k ∣ |h_{k+1,k}| ∣ h k + 1 , k ∣ (a “lucky” or near-breakdown), and (ii) the last component ∣ e k T y ∣ |e_k^{\mathsf{T}}y| ∣ e k T y ∣ of the eigenvector of H k H_k H k ?
Companion notebook: week6/script_week6_3.ipynb
Problem 7 (PageRank on a four-page web). Four pages link as follows: page 1 → \to → 2, 3; page 2 → \to → 3; page 3 → \to → 1; page 4 → \to → 1, 3.
(a) Write down the link matrix P P P (with P i j = 1 / d j P_{ij}=1/d_j P ij = 1/ d j when page j j j links to page i i i ) and check that it is column-stochastic. Page 4 receives no links: what does that say about row 4 of P P P , and what will make its PageRank nonzero nevertheless?
(b) Take α = 0.85 \alpha=0.85 α = 0.85 . Without writing out the dense matrix G G G , give the formula that computes G x Gx G x from P x Px P x , and say how many nonzeros are touched per application.
(c) Starting from x ( 0 ) = 1 4 ( 1 , 1 , 1 , 1 ) T x^{(0)}=\frac{1}{4}(1,1,1,1)^{\mathsf{T}} x ( 0 ) = 4 1 ( 1 , 1 , 1 , 1 ) T , perform two steps of the iteration (10) and rank the four pages.
(d) Using Proposition 5.2.3 , bound the number of iterations needed to reduce the error by 10 − 6 10^{-6} 1 0 − 6 , first for α = 0.85 \alpha=0.85 α = 0.85 and then for α = 0.99 \alpha=0.99 α = 0.99 . Comment on what the ranking gains and what the computation loses when α \alpha α is increased.
(e) Suppose a fifth page is added that links to nothing. Show that P P P is then no longer column-stochastic, and describe a way of repairing it that keeps the cost of applying G G G unchanged.
Companion notebook: week6/script_week6_2.ipynb
Preconditioning and Accelerating Solvers
Overview
Chapters 4 and 5 showed that Krylov methods are optimal over the subspace they build, but that their speed is dictated by the spectrum of A A A : CG needs O ( κ ( A ) ) O(\sqrt{\kappa(A)}) O ( κ ( A ) ) iterations, and for the PDE matrices of Chapter 2 this means O ( h − 1 ) O(h^{-1}) O ( h − 1 ) iterations, growing without bound as the mesh is refined. Optimality over a Krylov subspace cannot fix a badly conditioned problem. This is what preconditioning does. Instead of solving A x = b Ax=b A x = b we solve an equivalent system involving M − 1 A M^{-1}A M − 1 A , where M M M approximates A A A but is cheap to invert, and we choose M M M so that the transformed spectrum is clustered and the condition number is bounded independently of the mesh size; broad surveys of the subject are given in [44 ] [4]. This chapter covers: (1) the algebraic framework of left, right, and symmetric preconditioning, preconditioned CG, and simple choices such as Jacobi and incomplete factorisations; (2) Schwarz domain decomposition, where the preconditioner is built from independent subdomain solves and the overlap width controls the convergence rate; (3) two-level and multigrid ideas, where a coarse-grid correction removes the smooth error components that relaxation cannot touch, yielding preconditioners whose quality is independent of h h h .
Learning objectives
By the end of this chapter, you should be able to:
Explain why Krylov convergence rates make preconditioning necessary for mesh-refined PDE problems, and state the left, right, and symmetric preconditioning formulations.
Derive the preconditioned conjugate gradient algorithm and explain why it requires only solves with M M M , never M − 1 / 2 M^{-1/2} M − 1/2 , and why M M M must be SPD.
Construct and assess simple preconditioners (Jacobi, SSOR, incomplete Cholesky/LU) and estimate their effect on κ ( M − 1 A ) \kappa(M^{-1}A) κ ( M − 1 A ) for the discrete Laplacian.
Formulate the alternating and additive Schwarz methods, analyse the 1D convergence factor in terms of the overlap δ \delta δ , and explain the role of a coarse space in restoring scalability.
Compute the smoothing factor of weighted Jacobi, explain why relaxation damps oscillatory but not smooth error, and describe how coarse-grid correction and the resulting two-grid/multigrid cycle give an h h h -independent preconditioner.
Motivation and Basic Preconditioners
Preconditioning accelerates Krylov iterations by modifying the spectrum of the coefficient matrix. We aim to accelerate the solution of A x = b Ax=b A x = b , A ∈ R n × n A\in\mathbb{R}^{n\times n} A ∈ R n × n , using a matrix M M M that approximates A A A but is cheaper to invert.
Definition 6.1.1 (Left, right, and symmetric preconditioning). We have three different options:
M L − 1 A x = M L − 1 b . M_L^{-1}Ax=M_L^{-1}b. M L − 1 A x = M L − 1 b . This changes the residual norm seen by the Krylov method.
A M R − 1 y = b , x = M R − 1 y . AM_R^{-1}y=b,\qquad x=M_R^{-1}y. A M R − 1 y = b , x = M R − 1 y . This preserves the original residual b − A x b-Ax b − A x but modifies the search space.
M − 1 / 2 A M − 1 / 2 x ~ = M − 1 / 2 b , x ~ = M 1 / 2 x . M^{-1/2}AM^{-1/2}\widetilde{x}=M^{-1/2}b,\qquad\widetilde{x}=M^{1/2}x. M − 1/2 A M − 1/2 x = M − 1/2 b , x = M 1/2 x . This preserves symmetry and positive definiteness.
Key idea
For SPD problems, the convergence of CG depends on the condition number of the operator. A good preconditioner replaces A A A by an operator whose eigenvalues are more tightly clustered, ideally near 1:
M − 1 A ≈ I , κ ( M − 1 A ) ≪ κ ( A ) . M^{-1}A\approx I,\qquad\kappa(M^{-1}A)\ll\kappa(A). M − 1 A ≈ I , κ ( M − 1 A ) ≪ κ ( A ) .
The gain can already be quantified exactly on simple model matrices.
Figure 6.1. Effect of preconditioning on the spectrum: eigenvalues of A A A (left) are spread, whereas those of M − 1 A M^{-1}A M − 1 A (right) are tightly clustered around 1.
Preconditioning SPD problems
For SPD problems, preconditioning is most naturally understood through the symmetrically preconditioned operator
A ~ = M − 1 / 2 A M − 1 / 2 . \widetilde{A}=M^{-1/2}AM^{-1/2}. A = M − 1/2 A M − 1/2 .
CG is then applied to
A ~ x ~ = b ~ , x ~ = M 1 / 2 x , b ~ = M − 1 / 2 b . \widetilde{A}\widetilde{x}=\widetilde{b},\qquad\widetilde{x}=M^{1/2}x,\qquad\widetilde{b}=M^{-1/2}b. A x = b , x = M 1/2 x , b = M − 1/2 b .
Equivalently, in the original variables, the search space is the preconditioned Krylov space
x ( 0 ) + K k ( M − 1 A , M − 1 r ( 0 ) ) . x^{(0)}+\mathcal{K}_k(M^{-1}A,M^{-1}r^{(0)}). x ( 0 ) + K k ( M − 1 A , M − 1 r ( 0 ) ) .
Proposition 6.1.2 (Preconditioned CG). Let A A A and M M M be SPD. Applying CG to
M − 1 / 2 A M − 1 / 2 x ~ = M − 1 / 2 b M^{-1/2}AM^{-1/2}\widetilde{x}=M^{-1/2}b M − 1/2 A M − 1/2 x = M − 1/2 b is equivalent to the following iteration in the original variables:
r ( 0 ) = b − A x ( 0 ) , z ( 0 ) = M − 1 r ( 0 ) , p ( 0 ) = z ( 0 ) , r^{(0)}=b-Ax^{(0)},\qquad z^{(0)}=M^{-1}r^{(0)},\qquad p^{(0)}=z^{(0)}, r ( 0 ) = b − A x ( 0 ) , z ( 0 ) = M − 1 r ( 0 ) , p ( 0 ) = z ( 0 ) , q ( k ) = A p ( k ) , α k = ⟨ r ( k ) , z ( k ) ⟩ ⟨ p ( k ) , q ( k ) ⟩ , q^{(k)}=Ap^{(k)},\qquad\alpha_k=\frac{\langle r^{(k)},z^{(k)}\rangle}{\langle p^{(k)},q^{(k)}\rangle}, q ( k ) = A p ( k ) , α k = ⟨ p ( k ) , q ( k ) ⟩ ⟨ r ( k ) , z ( k ) ⟩ , x ( k + 1 ) = x ( k ) + α k p ( k ) , r ( k + 1 ) = r ( k ) − α k q ( k ) , x^{(k+1)}=x^{(k)}+\alpha_kp^{(k)},\qquad r^{(k+1)}=r^{(k)}-\alpha_kq^{(k)}, x ( k + 1 ) = x ( k ) + α k p ( k ) , r ( k + 1 ) = r ( k ) − α k q ( k ) , z ( k + 1 ) = M − 1 r ( k + 1 ) , β k + 1 = ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ ⟨ r ( k ) , z ( k ) ⟩ , z^{(k+1)}=M^{-1}r^{(k+1)},\qquad\beta_{k+1}=\frac{\langle r^{(k+1)},z^{(k+1)}\rangle}{\langle r^{(k)},z^{(k)}\rangle}, z ( k + 1 ) = M − 1 r ( k + 1 ) , β k + 1 = ⟨ r ( k ) , z ( k ) ⟩ ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ , p ( k + 1 ) = z ( k + 1 ) + β k + 1 p ( k ) . p^{(k+1)}=z^{(k+1)}+\beta_{k+1}p^{(k)}. p ( k + 1 ) = z ( k + 1 ) + β k + 1 p ( k ) . Moreover,
spec ( M − 1 / 2 A M − 1 / 2 ) = spec ( M − 1 A ) , \operatorname{spec}(M^{-1/2}AM^{-1/2})=\operatorname{spec}(M^{-1}A), spec ( M − 1/2 A M − 1/2 ) = spec ( M − 1 A ) , so the usual CG convergence estimate holds with κ ( M − 1 A ) \kappa(M^{-1}A) κ ( M − 1 A ) replacing κ ( A ) \kappa(A) κ ( A ) .
Proof sketch. Set
x ~ = M 1 / 2 x , r ~ = M − 1 / 2 r , p ~ = M 1 / 2 p . \widetilde{x}=M^{1/2}x,\qquad\widetilde{r}=M^{-1/2}r,\qquad\widetilde{p}=M^{1/2}p. x = M 1/2 x , r = M − 1/2 r , p = M 1/2 p .
Then standard CG on A ~ = M − 1 / 2 A M − 1 / 2 \widetilde{A}=M^{-1/2}AM^{-1/2} A = M − 1/2 A M − 1/2 gives
α k = ⟨ r ~ ( k ) , r ~ ( k ) ⟩ ⟨ p ~ ( k ) , A ~ p ~ ( k ) ⟩ = ⟨ r ( k ) , M − 1 r ( k ) ⟩ ⟨ p ( k ) , A p ( k ) ⟩ , \alpha_k=\frac{\langle\widetilde{r}^{(k)},\widetilde{r}^{(k)}\rangle}{\langle\widetilde{p}^{(k)},\widetilde{A}\widetilde{p}^{(k)}\rangle}=\frac{\langle r^{(k)},M^{-1}r^{(k)}\rangle}{\langle p^{(k)},Ap^{(k)}\rangle}, α k = ⟨ p ( k ) , A p ( k ) ⟩ ⟨ r ( k ) , r ( k ) ⟩ = ⟨ p ( k ) , A p ( k ) ⟩ ⟨ r ( k ) , M − 1 r ( k ) ⟩ ,
which yields the stated recursion with z ( k ) = M − 1 r ( k ) z^{(k)}=M^{-1}r^{(k)} z ( k ) = M − 1 r ( k ) . Similarly,
β k + 1 = ⟨ r ~ ( k + 1 ) , r ~ ( k + 1 ) ⟩ ⟨ r ~ ( k ) , r ~ ( k ) ⟩ = ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ ⟨ r ( k ) , z ( k ) ⟩ . \beta_{k+1}=\frac{\langle\widetilde{r}^{(k+1)},\widetilde{r}^{(k+1)}\rangle}{\langle\widetilde{r}^{(k)},\widetilde{r}^{(k)}\rangle}=\frac{\langle r^{(k+1)},z^{(k+1)}\rangle}{\langle r^{(k)},z^{(k)}\rangle}. β k + 1 = ⟨ r ( k ) , r ( k ) ⟩ ⟨ r ( k + 1 ) , r ( k + 1 ) ⟩ = ⟨ r ( k ) , z ( k ) ⟩ ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ .
Finally,
M − 1 A = M − 1 / 2 ( M − 1 / 2 A M 1 / 2 ) M − 1 / 2 M^{-1}A=M^{-1/2}(M^{-1/2}AM^{1/2})M^{-1/2} M − 1 A = M − 1/2 ( M − 1/2 A M 1/2 ) M − 1/2
is similar to M − 1 / 2 A M − 1 / 2 M^{-1/2}AM^{-1/2} M − 1/2 A M − 1/2 , so they have the same eigenvalues.
Remark 6.1.3. Preconditioning does not alter the exact solution x ∗ x^* x ∗ but changes the trajectory of residual minimization. The effectiveness depends on how well M M M approximates A A A in the subspace explored by the Krylov method.
A general framework for preconditioners
Jacobi preconditioning. Let A = D + L + U A=D+L+U A = D + L + U , where D D D is the diagonal of A A A . The Jacobi preconditioner is
It is extremely cheap to apply, fully parallel, and often removes poor scaling between coordinates. Its main limitation is that it only uses diagonal information.
Example: Jacobi for the 1D discrete Laplacian. Consider now the tridiagonal matrix arising from the standard finite-difference discretisation of
− u ′ ′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0. -u''(x) = f(x), \qquad x \in(0,1), \qquad u(0) = u(1) = 0. − u ′′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0.
With m m m interior grid points and mesh size h = 1 / ( m + 1 ) h = 1/(m+1) h = 1/ ( m + 1 ) , the linear system is
A h u = f , A h = 1 h 2 tridiag ( − 1 , 2 , − 1 ) ∈ R m × m . A_h u = f, \qquad A_h = \frac{1}{h^2}\operatorname{tridiag}(-1,2,-1) \in\mathbb{R}^{m \times m}. A h u = f , A h = h 2 1 tridiag ( − 1 , 2 , − 1 ) ∈ R m × m .
The diagonal of A h A_h A h is constant: D = diag ( A h ) = 2 h 2 I D = \operatorname{diag}(A_h) = \frac{2}{h^2}I D = diag ( A h ) = h 2 2 I . Therefore the Jacobi-preconditioned operator is simply
D − 1 A h = h 2 2 A h = 1 2 tridiag ( − 1 , 2 , − 1 ) . D^{-1}A_h = \frac{h^2}{2}A_h = \frac{1}{2}\operatorname{tridiag}(-1,2,-1). D − 1 A h = 2 h 2 A h = 2 1 tridiag ( − 1 , 2 , − 1 ) .
Jacobi therefore rescales every eigenvalue by the same constant factor, and cannot improve the condition number κ ( A h ) ∼ h − 2 \kappa(A_h) \sim h^{-2} κ ( A h ) ∼ h − 2 , which grows quadratically as the mesh is refined.
Takeaways Jacobi preconditioning is effective when the main difficulty is simple coordinate scaling, but it cannot improve problems such as the 1D discrete Laplacian where the diagonal is already uniform. In PDEs, the hard part is often not local scaling but global error propagation, which motivates Schwarz methods, coarse spaces, and multigrid. A more powerful algebraic alternative is the incomplete factorisation A ≈ L ~ U ~ A \approx\widetilde{L}\widetilde{U} A ≈ L U , in which fill-in arising outside a prescribed sparsity pattern is simply discarded [29 ] .
One framework for the whole chapter
Before turning to better preconditioners it is worth noticing that many of them (and all those discussed in this lecture) are instances of a single construction. Let R i ∈ R n i × n R_i \in\mathbb{R}^{n_i \times n} R i ∈ R n i × n be a restriction onto a subspace of the unknowns, let A i = R i A R i T A_i = R_i A R_i^{\mathsf{T}} A i = R i A R i T be the corresponding local matrix, and set
M − 1 = ∑ i R i T A i − 1 R i . (12) M^{-1} = \sum_i R_i^{\mathsf{T}} A_i^{-1} R_i. \tag*{(12)} M − 1 = i ∑ R i T A i − 1 R i . (12)
Each term solves the problem exactly on one subspace and ignores the rest; the preconditioner is their sum. Applying M − 1 M^{-1} M − 1 costs one small solve per subspace, and the solves are independent of one another which is what makes the construction parallel.
Everything in this chapter is a choice of the subspaces in (12) .
Table 1 (PDF p. 86).
The table above is the plan of the chapter, and each row fixes the disadvantages of the one above it. For example, a pure local preconditioner handles local error well and global error not at all (for example Jacobi in its simple and block variants or Schwarz preconditioners). To fix this, one should add a global subspace to catch what the local solves miss.
Schwarz Domain Decomposition
The algorithms of this section, their convergence theory, and their parallel implementation are treated at length in [40 ] [9].
Schwarz Domain Decomposition
In the language of (12) , this section makes the first non-trivial choice of subspaces: instead of correcting each unknown independently as Jacobi does, we take the R i R_i R i to restrict to overlapping subdomains of the mesh, so that each local solve of A i − 1 A_i^{-1} A i − 1 inverts the operator exactly on a whole piece of the domain. The resulting preconditioner is additive Schwarz , and the question we must answer about it is the one posed in § 9.3 : how does its convergence behave?
It is easiest to answer that question by first studying the corresponding stationary iteration , where the local solves are applied one after another. That iteration is of independent historical interest and its convergence factor exposes the two parameters that will govern the preconditioner as well: the overlap width, and the number of subdomains.
Model problem: 1D Laplace equation. We consider
− u ′ ′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0. -u''(x) = f(x), \qquad x \in(0,1), \qquad u(0) = u(1) = 0. − u ′′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0.
We split the interval into two overlapping subdomains Ω 1 = ( 0 , β ) \Omega_1 = (0,\beta) Ω 1 = ( 0 , β ) , Ω 2 = ( α , 1 ) \Omega_2 = (\alpha,1) Ω 2 = ( α , 1 ) , 0 < α < β < 1 0 < \alpha< \beta< 1 0 < α < β < 1 . The overlap is the interval ( α , β ) (\alpha,\beta) ( α , β ) . The alternating Schwarz iteration is:
Figure 6.2. 1D domain split into two overlapping subdomains.
{ − ( u 1 ( n ) ) ′ ′ = f in ( 0 , β ) , u 1 ( n ) ( 0 ) = 0 , u 1 ( n ) ( β ) = u 2 ( n − 1 ) ( β ) , { − ( u 2 ( n ) ) ′ ′ = f in ( α , 1 ) , u 2 ( n ) ( 1 ) = 0 , u 2 ( n ) ( α ) = u 1 ( n ) ( α ) . \begin{cases}
-(u_1^{(n)})'' = f & \text{in }(0,\beta),\\
u_1^{(n)}(0) = 0,\\
u_1^{(n)}(\beta) = u_2^{(n-1)}(\beta),
\end{cases}
\qquad
\begin{cases}
-(u_2^{(n)})'' = f & \text{in }(\alpha,1),\\
u_2^{(n)}(1) = 0,\\
u_2^{(n)}(\alpha) = u_1^{(n)}(\alpha).
\end{cases} ⎩ ⎨ ⎧ − ( u 1 ( n ) ) ′′ = f u 1 ( n ) ( 0 ) = 0 , u 1 ( n ) ( β ) = u 2 ( n − 1 ) ( β ) , in ( 0 , β ) , ⎩ ⎨ ⎧ − ( u 2 ( n ) ) ′′ = f u 2 ( n ) ( 1 ) = 0 , u 2 ( n ) ( α ) = u 1 ( n ) ( α ) . in ( α , 1 ) ,
Each subproblem is solved exactly, using boundary data from the latest iterate on the neighbouring subdomain.
Proposition 6.2.1 (Contraction of alternating Schwarz in 1D). Let u u u be the exact solution of the Poisson BVP and let u 1 ( n ) , u 2 ( n ) u_1^{(n)},u_2^{(n)} u 1 ( n ) , u 2 ( n ) be the alternating Schwarz iterates on Ω 1 = ( 0 , β ) \Omega_1 = (0,\beta) Ω 1 = ( 0 , β ) and Ω 2 = ( α , 1 ) \Omega_2 = (\alpha,1) Ω 2 = ( α , 1 ) . Define the errors e 1 ( n ) = u ∣ Ω 1 − u 1 ( n ) e_1^{(n)} = u|_{\Omega_1} - u_1^{(n)} e 1 ( n ) = u ∣ Ω 1 − u 1 ( n ) , e 2 ( n ) = u ∣ Ω 2 − u 2 ( n ) e_2^{(n)} = u|_{\Omega_2} - u_2^{(n)} e 2 ( n ) = u ∣ Ω 2 − u 2 ( n ) . Then
− ( e 1 ( n ) ) ′ ′ = 0 in ( 0 , β ) , − ( e 2 ( n ) ) ′ ′ = 0 in ( α , 1 ) , -(e_1^{(n)})'' = 0 \quad\text{in }(0,\beta), \qquad-(e_2^{(n)})'' = 0 \quad\text{in }(\alpha,1), − ( e 1 ( n ) ) ′′ = 0 in ( 0 , β ) , − ( e 2 ( n ) ) ′′ = 0 in ( α , 1 ) , and the interface errors satisfy the exact relation
e 2 ( n ) ( β ) = ρ e 2 ( n − 1 ) ( β ) , ρ = α ( 1 − β ) β ( 1 − α ) ∈ ( 0 , 1 ) . e_2^{(n)}(\beta) = \rho e_2^{(n-1)}(\beta), \qquad\rho= \frac{\alpha(1-\beta)}{\beta(1-\alpha)} \in(0,1). e 2 ( n ) ( β ) = ρ e 2 ( n − 1 ) ( β ) , ρ = β ( 1 − α ) α ( 1 − β ) ∈ ( 0 , 1 ) . Hence e 2 ( n ) ( β ) = ρ n e 2 ( 0 ) ( β ) e_2^{(n)}(\beta) = \rho^n e_2^{(0)}(\beta) e 2 ( n ) ( β ) = ρ n e 2 ( 0 ) ( β ) : the alternating Schwarz method converges geometrically, and the convergence improves as the overlap β − α \beta-\alpha β − α increases.
Proof. The iterates and the exact solution satisfy the same equation with different boundary data, so each error is harmonic on its subdomain: − ( e i ( n ) ) ′ ′ = 0 -(e_i^{(n)})'' = 0 − ( e i ( n ) ) ′′ = 0 , with e 1 ( n ) ( 0 ) = 0 e_1^{(n)}(0) = 0 e 1 ( n ) ( 0 ) = 0 , e 1 ( n ) ( β ) = e 2 ( n − 1 ) ( β ) e_1^{(n)}(\beta) = e_2^{(n-1)}(\beta) e 1 ( n ) ( β ) = e 2 ( n − 1 ) ( β ) and e 2 ( n ) ( 1 ) = 0 e_2^{(n)}(1) = 0 e 2 ( n ) ( 1 ) = 0 , e 2 ( n ) ( α ) = e 1 ( n ) ( α ) e_2^{(n)}(\alpha) = e_1^{(n)}(\alpha) e 2 ( n ) ( α ) = e 1 ( n ) ( α ) . In one dimension, a harmonic function is affine, so each error is determined by its two boundary values:
e 1 ( n ) ( x ) = x β e 2 ( n − 1 ) ( β ) , e 2 ( n ) ( x ) = 1 − x 1 − α e 1 ( n ) ( α ) . e_1^{(n)}(x) = \frac{x}{\beta}e_2^{(n-1)}(\beta), \qquad e_2^{(n)}(x) = \frac{1-x}{1-\alpha}e_1^{(n)}(\alpha). e 1 ( n ) ( x ) = β x e 2 ( n − 1 ) ( β ) , e 2 ( n ) ( x ) = 1 − α 1 − x e 1 ( n ) ( α ) .
Evaluating the first at x = α x = \alpha x = α and substituting into the second at x = β x = \beta x = β gives e 2 ( n ) ( β ) = ρ e 2 ( n − 1 ) ( β ) e_2^{(n)}(\beta) = \rho e_2^{(n-1)}(\beta) e 2 ( n ) ( β ) = ρ e 2 ( n − 1 ) ( β ) with ρ = α ( 1 − β ) β ( 1 − α ) \rho= \frac{\alpha(1-\beta)}{\beta(1-\alpha)} ρ = β ( 1 − α ) α ( 1 − β ) . No absolute values and no inequality are needed: in one dimension the error is affine, hence determined exactly by its two interface values. Finally 0 < α < β < 1 0 < \alpha< \beta< 1 0 < α < β < 1 gives ρ > 0 \rho> 0 ρ > 0 , and α ( 1 − β ) < β ( 1 − α ) \alpha(1-\beta) < \beta(1-\alpha) α ( 1 − β ) < β ( 1 − α ) reduces to α < β \alpha< \beta α < β , so ρ < 1 \rho< 1 ρ < 1 .
Remark 6.2.2. The formula makes the role of the overlap explicit: as β − α → 0 \beta- \alpha\to0 β − α → 0 we get ρ → 1 \rho\to1 ρ → 1 and convergence stalls, while a wider overlap decreases ρ \rho ρ . Overlap is the mechanism by which information is shared by the neighbouring subdomains.
Figure 6.3. Alternating Schwarz converges geometrically in 1D. The contraction factor improves when the overlap increases.
Discrete formulation. Discretising with m = 9 m = 9 m = 9 interior points and taking β = 7 h \beta= 7h β = 7 h , α = 4 h \alpha= 4h α = 4 h gives local matrices A 1 ∈ R 6 × 6 A_1 \in\mathbb{R}^{6 \times6} A 1 ∈ R 6 × 6 and A 2 ∈ R 5 × 5 A_2 \in\mathbb{R}^{5 \times5} A 2 ∈ R 5 × 5 , each the 1D Laplacian on its own subdomain. The iteration becomes
A 1 u 1 ( n ) = f 1 + B ~ 12 u 2 ( n − 1 ) , A 2 u 2 ( n ) = f 2 + B ~ 21 u 1 ( n ) A_1 u_1^{(n)} = f_1 + \widetilde{B}_{12}u_2^{(n-1)}, \qquad A_2u_2^{(n)} = f_2 + \widetilde{B}_{21}u_1^{(n)} A 1 u 1 ( n ) = f 1 + B 12 u 2 ( n − 1 ) , A 2 u 2 ( n ) = f 2 + B 21 u 1 ( n )
where B ~ 12 , B ~ 21 \widetilde{B}_{12}, \widetilde{B}_{21} B 12 , B 21 inject the interface values from one subdomain into the other (Dirichlet transmission). Note that only A 1 A_1 A 1 and A 2 A_2 A 2 are factorised: the global matrix is never formed. This makes the method efficient at a larger scale.
The discrete contraction factor inherits the continuous one: the iteration converges geometrically at a rate set by the overlap and, crucially, independently of h h h such that refining the mesh does not slow it down, because the overlap is a fixed fraction of the domain.
Table 5.
From subdomain solves to a preconditioner. The discrete Schwarz viewpoint becomes clearer if we start from the global residual r ( n ) = f − A u ( n ) r^{(n)} = f - Au^{(n)} r ( n ) = f − A u ( n ) . Instead of trying to correct all unknowns at once, we restrict this residual to each subdomain and solve a local correction problem there. Let R i R_i R i be the restriction from the global vector to the unknowns belonging to subdomain Ω i \Omega_i Ω i , and let
A i = R i A R i T A_i = R_i A R_i^T A i = R i A R i T
be the local stiffness matrix on that subdomain. The local correction δ u i \delta u_i δ u i is defined by
A i δ u i = R i r ( n ) , so that δ u i = A i − 1 R i r ( n ) . A_i\delta u_i = R_i r^{(n)}, \qquad\text{so that}\qquad\delta u_i = A_i^{-1}R_i r^{(n)}. A i δ u i = R i r ( n ) , so that δ u i = A i − 1 R i r ( n ) .
This means that on each subdomain we solve exactly the local error equation driven by the restricted residual. The correction is then prolongated back to the global space by R i T R_i^T R i T . If the subdomains overlap, several local corrections contribute to the same global node, so we blend them with diagonal weights D i D_i D i satisfying the partition-of-unity condition
∑ i R i T D i R i = I . \sum_i R_i^T D_i R_i = I. i ∑ R i T D i R i = I .
This leads to the global correction
u ( n + 1 ) = u ( n ) + ∑ i R i T D i δ u i = u ( n ) + ∑ i R i T D i A i − 1 R i r ( n ) . u^{(n+1)} = u^{(n)} + \sum_i R_i^T D_i\delta u_i = u^{(n)} + \sum_i R_i^T D_i A_i^{-1}R_i r^{(n)}. u ( n + 1 ) = u ( n ) + i ∑ R i T D i δ u i = u ( n ) + i ∑ R i T D i A i − 1 R i r ( n ) .
Additive Schwarz preconditioner. The operator
M A S − 1 = ∑ i R i T D i A i − 1 R i M_{\mathrm{AS}}^{-1} = \sum_i R_i^T D_i A_i^{-1}R_i M AS − 1 = i ∑ R i T D i A i − 1 R i
is called the additive Schwarz preconditioner . With this notation, one step of preconditioned Richardson iteration reads
u ( n + 1 ) = u ( n ) + M A S − 1 ( f − A u ( n ) ) . u^{(n+1)} = u^{(n)} + M_{\mathrm{AS}}^{-1}(f-Au^{(n)}). u ( n + 1 ) = u ( n ) + M AS − 1 ( f − A u ( n ) ) .
So the preconditioner acts as an approximate inverse of A A A obtained by:
restricting the residual to each subdomain,
solving local problems there,
prolongating the local corrections,
blending the overlap contributions.
Remark 6.2.3. This additive preconditioner is closely related to the alternating Schwarz iteration, but it is not exactly the same algorithm. The alternating method is multiplicative : the second subdomain uses the updated information coming from the first. By contrast, additive Schwarz computes all local corrections from the same global residual and sums them together. The additive form is especially convenient as a preconditioner inside Krylov methods.
Key idea. Schwarz preconditioning improves on Jacobi by replacing pointwise updates with subdomain solves . Instead of using only diagonal information, it inverts the operator locally on each subdomain. This captures nearest-neighbour couplings much better and yields a significantly stronger preconditioner for discretised PDEs.
Example of a decomposition. To make the link with the preconditioned formulation explicit, consider again m = 9 m=9 m = 9 interior nodes and the decomposition
Ω 1 = ( 0 , 7 h ) , Ω 2 = ( 4 h , 1 ) , \Omega_1 = (0,7h), \qquad\Omega_2 = (4h,1), Ω 1 = ( 0 , 7 h ) , Ω 2 = ( 4 h , 1 ) ,
which gives an overlap of three grid points ( x 5 , x 6 , x 7 ) (x_5,x_6,x_7) ( x 5 , x 6 , x 7 ) . The global unknown vector is u = ( u 1 , u 2 , … , u 9 ) T u=(u_1,u_2,\ldots,u_9)^{\mathsf{T}} u = ( u 1 , u 2 , … , u 9 ) T , and the local vectors on each subdomain are
u 1 = ( u 1 , … , u 6 ) T , u 2 = ( u 5 , … , u 9 ) T . u_1=(u_1,\ldots,u_6)^{\mathsf{T}}, \qquad u_2=(u_5,\ldots,u_9)^{\mathsf{T}}. u 1 = ( u 1 , … , u 6 ) T , u 2 = ( u 5 , … , u 9 ) T .
Figure 6.4. Partition of unity in the overlap: D 1 D_1 D 1 (top) and D 2 D_2 D 2 (bottom). On the overlap, D 1 = D 2 = 1 2 D_1 = D_2 = \frac{1}{2} D 1 = D 2 = 2 1 and outside the overlap they sum to one: R 1 T D 1 R 1 + R 2 T D 2 R 2 = I R_1^T D_1 R_1 + R_2^T D_2 R_2 = I R 1 T D 1 R 1 + R 2 T D 2 R 2 = I .
The restriction matrices R 1 ∈ R 6 × 9 R_1 \in\mathbb{R}^{6 \times9} R 1 ∈ R 6 × 9 and R 2 ∈ R 5 × 9 R_2 \in\mathbb{R}^{5 \times9} R 2 ∈ R 5 × 9 are the corresponding rows of the identity: R 1 = [ I 6 0 ] R_1 = [I_6\ 0] R 1 = [ I 6 0 ] picks out u 1 , … , u 6 u_1,\ldots,u_6 u 1 , … , u 6 , and R 2 = [ 0 I 5 ] R_2 = [0\ I_5] R 2 = [ 0 I 5 ] picks out u 5 , … , u 9 u_5,\ldots,u_9 u 5 , … , u 9 . They are never stored as matrices since applying R i R_i R i is an indexing operation.
Each restriction operator extracts local unknowns, and its transpose R i T R_i^T R i T prolongates the local corrections into the global vector. In the overlap region (nodes x 5 , x 6 , x 7 x_5,x_6,x_7 x 5 , x 6 , x 7 ), contributions from both subdomains are combined using a discrete partition of unity :
D 1 = diag ( 1 , 1 , 1 , 1 , 1 2 , 1 2 , 1 2 , 0 , 0 ) , D 2 = diag ( 0 , 0 , 0 , 0 , 1 2 , 1 2 , 1 2 , 1 , 1 ) . D_1 = \operatorname{diag}\left(1,1,1,1,\frac{1}{2},\frac{1}{2},\frac{1}{2},0,0\right), \qquad D_2 = \operatorname{diag}\left(0,0,0,0,\frac{1}{2},\frac{1}{2},\frac{1}{2},1,1\right). D 1 = diag ( 1 , 1 , 1 , 1 , 2 1 , 2 1 , 2 1 , 0 , 0 ) , D 2 = diag ( 0 , 0 , 0 , 0 , 2 1 , 2 1 , 2 1 , 1 , 1 ) .
These satisfy
R 1 T D 1 R 1 + R 2 T D 2 R 2 = I , R_1^T D_1 R_1 + R_2^T D_2 R_2 = I, R 1 T D 1 R 1 + R 2 T D 2 R 2 = I ,
ensuring that the overlapping corrections are weighted consistently and the global operator remains a proper approximation of A − 1 A^{-1} A − 1 . The additive Schwarz preconditioner then reads M − 1 = R 1 T D 1 A 1 − 1 R 1 + R 2 T D 2 A 2 − 1 R 2 M^{-1} = R_1^T D_1 A_1^{-1} R_1 + R_2^T D_2 A_2^{-1} R_2 M − 1 = R 1 T D 1 A 1 − 1 R 1 + R 2 T D 2 A 2 − 1 R 2 , which applies local solves and blends them smoothly through the partition weights.
Two-Level Preconditioning and Multigrid Ideas
In order to understand the mechanism of error reduction or smoothing and then the need of a coarse space correction let us consider the discretisation of the 1D Poisson problem. The eigenvectors of A h A_h A h are the discrete sine modes
v j ( ℓ ) = sin ( j π ℓ m + 1 ) , ℓ = 1 , … , m , j = 1 , … , m , v_j(\ell) = \sin\left(\frac{j\pi\ell}{m+1}\right), \qquad\ell= 1,\ldots,m,\quad j = 1,\ldots,m, v j ( ℓ ) = sin ( m + 1 j π ℓ ) , ℓ = 1 , … , m , j = 1 , … , m ,
with eigenvalues
λ j ( A h ) = 4 h 2 sin 2 ( j π 2 ( m + 1 ) ) . \lambda_j(A_h) = \frac{4}{h^2}\sin^2\left(\frac{j\pi}{2(m+1)}\right). λ j ( A h ) = h 2 4 sin 2 ( 2 ( m + 1 ) j π ) .
Small values of j j j correspond to smooth, slowly varying modes, whereas large values of j j j correspond to oscillatory modes.
Figure 6.5. Low-frequency and high-frequency error modes on a 1D grid.
Two-Level Preconditioning and Multigrid Ideas
Consider the Jacobi iteration
u ( k + 1 ) = u ( k ) + D − 1 ( f − A h u ( k ) ) , D = 2 h 2 I . u^{(k+1)} = u^{(k)} + D^{-1}(f - A_h u^{(k)}), \qquad D = \frac{2}{h^2}I. u ( k + 1 ) = u ( k ) + D − 1 ( f − A h u ( k ) ) , D = h 2 2 I .
If e ( k ) = u − u ( k ) e^{(k)} = u - u^{(k)} e ( k ) = u − u ( k ) denotes the error, then
e ( k + 1 ) = T J e ( k ) , T J = I − D − 1 A h . e^{(k+1)} = T_J e^{(k)}, \qquad T_J = I - D^{-1}A_h. e ( k + 1 ) = T J e ( k ) , T J = I − D − 1 A h .
Proposition 6.3.1 (Mode-by-mode damping of Jacobi). For the 1D Laplacian, each sine mode v j v_j v j is an eigenvector of the Jacobi error propagator T J T_J T J , with eigenvalue
μ j = 1 − λ j ( A h ) 2 / h 2 = 1 − 2 sin 2 ( j π 2 ( m + 1 ) ) = cos ( j π m + 1 ) . \mu_j = 1 - \frac{\lambda_j(A_h)}{2/h^2} = 1 - 2\sin^2\left(\frac{j\pi}{2(m+1)}\right) = \cos\left(\frac{j\pi}{m+1}\right). μ j = 1 − 2/ h 2 λ j ( A h ) = 1 − 2 sin 2 ( 2 ( m + 1 ) j π ) = cos ( m + 1 j π ) . Hence, if
e ( 0 ) = ∑ j = 1 m c j v j , e^{(0)} = \sum_{j=1}^{m} c_j v_j, e ( 0 ) = j = 1 ∑ m c j v j , then after k k k Jacobi steps
e ( k ) = ∑ j = 1 m c j μ j k v j . e^{(k)} = \sum_{j=1}^{m} c_j \mu_j^k v_j. e ( k ) = j = 1 ∑ m c j μ j k v j .
Proof. Since the v j v_j v j are eigenvectors of A h A_h A h ,
A h v j = λ j ( A h ) v j . A_h v_j = \lambda_j(A_h)v_j. A h v j = λ j ( A h ) v j .
Therefore
T J v j = ( I − D − 1 A h ) v j = ( 1 − λ j ( A h ) 2 / h 2 ) v j . T_Jv_j = \left(I-D^{-1}A_h\right)v_j = \left(1-\frac{\lambda_j(A_h)}{2/h^2}\right)v_j. T J v j = ( I − D − 1 A h ) v j = ( 1 − 2/ h 2 λ j ( A h ) ) v j .
Using
λ j ( A h ) = 4 h 2 sin 2 ( j π 2 ( m + 1 ) ) , \lambda_j(A_h) = \frac{4}{h^2}\sin^2\left(\frac{j\pi}{2(m+1)}\right), λ j ( A h ) = h 2 4 sin 2 ( 2 ( m + 1 ) j π ) ,
we obtain
μ j = 1 − 2 sin 2 ( j π 2 ( m + 1 ) ) . \mu_j = 1 - 2\sin^2\left(\frac{j\pi}{2(m+1)}\right). μ j = 1 − 2 sin 2 ( 2 ( m + 1 ) j π ) .
The trigonometric identity 1 − 2 sin 2 ( θ ) = cos ( 2 θ ) 1 - 2\sin^2(\theta) = \cos(2\theta) 1 − 2 sin 2 ( θ ) = cos ( 2 θ ) gives
μ j = cos ( j π m + 1 ) . \mu_j = \cos\left(\frac{j\pi}{m+1}\right). μ j = cos ( m + 1 j π ) .
The expansion of e ( k ) e^{(k)} e ( k ) follows by diagonalising the iteration on the basis { v j } j = 1 m \{v_j\}_{j=1}^{m} { v j } j = 1 m .
Remark 6.3.2. This formula shows explicitly why Jacobi behaves differently on different scales:
j small ⟹ ∣ μ j ∣ ≈ 1 (slow damping) , j\ \text{small} \quad\Longrightarrow\quad|\mu_j| \approx1 \quad\text{(slow damping)}, j small ⟹ ∣ μ j ∣ ≈ 1 (slow damping) ,
whereas
j large ⟹ ∣ μ j ∣ is much smaller (fast damping) . j\ \text{large} \quad\Longrightarrow\quad|\mu_j|\ \text{is much smaller} \quad\text{(fast damping)}. j large ⟹ ∣ μ j ∣ is much smaller (fast damping) .
So Jacobi is not a good solver by itself, but it is a good smoother.
Weighted Jacobi and the smoothing factor
A common variant is weighted Jacobi :
u ( k + 1 ) = u ( k ) + ω D − 1 ( f − A h u ( k ) ) . u^{(k+1)} = u^{(k)} + \omega D^{-1}(f - A_h u^{(k)}). u ( k + 1 ) = u ( k ) + ω D − 1 ( f − A h u ( k ) ) .
Its error propagator is
T J , ω = I − ω D − 1 A h . T_{J,\omega} = I - \omega D^{-1} A_h. T J , ω = I − ω D − 1 A h .
For the sine mode v j v_j v j ,
T J , ω v j = μ j ( ω ) v j , μ j ( ω ) = 1 − 2 ω sin 2 ( j π 2 ( m + 1 ) ) . T_{J,\omega}v_j = \mu_j^{(\omega)}v_j,\qquad\mu_j^{(\omega)} = 1 - 2\omega\sin^2\left(\frac{j\pi}{2(m+1)}\right). T J , ω v j = μ j ( ω ) v j , μ j ( ω ) = 1 − 2 ω sin 2 ( 2 ( m + 1 ) j π ) .
To study smoothing, we focus on the upper half of the spectrum, m + 1 2 ≤ j ≤ m \frac{m+1}{2} \le j \le m 2 m + 1 ≤ j ≤ m , which corresponds to oscillatory modes. For these indices,
sin 2 ( j π 2 ( m + 1 ) ) ∈ [ 1 2 , 1 ] , \sin^2\left(\frac{j\pi}{2(m+1)}\right) \in\left[\frac{1}{2},1\right], sin 2 ( 2 ( m + 1 ) j π ) ∈ [ 2 1 , 1 ] ,
hence
μ j ( ω ) ∈ [ 1 − 2 ω , 1 − ω ] . \mu_j^{(\omega)} \in[1-2\omega,1-\omega]. μ j ( ω ) ∈ [ 1 − 2 ω , 1 − ω ] .
So the worst-case damping factor on the high-frequency range is
ρ h i g h ( ω ) = max ( ∣ 1 − 2 ω ∣ , ∣ 1 − ω ∣ ) . \rho_{\mathrm{high}}(\omega) = \max\left(|1-2\omega|,|1-\omega|\right). ρ high ( ω ) = max ( ∣1 − 2 ω ∣ , ∣1 − ω ∣ ) .
Proposition 6.3.3 (Optimal weighted Jacobi parameter for smoothing). For the 1D Laplacian, the value of ω \omega ω that minimizes the worst-case high-frequency damping factor is ω = 2 3 \omega=\frac{2}{3} ω = 3 2 . For this choice, ρ h i g h = 1 3 \rho_{\mathrm{high}}=\frac{1}{3} ρ high = 3 1 . Thus every high-frequency mode is reduced by at least a factor 1 / 3 1/3 1/3 per step.
Proof. We minimize
ρ h i g h ( ω ) = max ( ∣ 1 − 2 ω ∣ , ∣ 1 − ω ∣ ) . \rho_{\mathrm{high}}(\omega) = \max(|1-2\omega|,|1-\omega|). ρ high ( ω ) = max ( ∣1 − 2 ω ∣ , ∣1 − ω ∣ ) .
The optimum is obtained by balancing the two endpoint magnitudes:
∣ 1 − 2 ω ∣ = ∣ 1 − ω ∣ . |1-2\omega| = |1-\omega|. ∣1 − 2 ω ∣ = ∣1 − ω ∣.
For ω ∈ ( 1 / 2 , 1 ) \omega\in(1/2,1) ω ∈ ( 1/2 , 1 ) , this becomes 2 ω − 1 = 1 − ω 2\omega-1=1-\omega 2 ω − 1 = 1 − ω , hence ω = 2 3 \omega=\frac{2}{3} ω = 3 2 . Then ρ h i g h = 1 3 \rho_{\mathrm{high}}=\frac{1}{3} ρ high = 3 1 .
Takeaways
Weighted Jacobi with ω = 2 / 3 \omega=2/3 ω = 2/3 is not a scalable solver, but it is an effective smoother: it damps oscillatory components uniformly while leaving smooth components for the coarse grid to remove.
After several relaxation steps, the remaining error typically has the form
e = e s m o o t h + e o s c , ∥ e o s c ∥ ≪ ∥ e s m o o t h ∥ . e=e_{\mathrm{smooth}}+e_{\mathrm{osc}},\qquad\lVert e_{\mathrm{osc}}\rVert\ll\lVert e_{\mathrm{smooth}}\rVert. e = e smooth + e osc , ∥ e osc ∥ ≪ ∥ e smooth ∥ .
That is, the oscillatory part has already been reduced, while the smooth part remains.
The key multigrid observation is that a smooth error on the fine grid may look much more oscillatory on a coarser grid. This makes it easier to represent and eliminate there.
Key idea
A good solver for an elliptic problem must act on two scales at once:
local smoothing + global coarse correction . \text{local smoothing}\qquad+ \qquad\text{global coarse correction}. local smoothing + global coarse correction .
Coarse-grid correction
Section 6.3.1 left us with an iterate whose error is smooth : the smoother has removed the oscillatory components and stalls on the rest. What remains satisfies the error equation
A e = r , r = f − A u , e = u ∗ − u , Ae = r,\qquad r = f - Au,\qquad e = u^* - u, A e = r , r = f − A u , e = u ∗ − u ,
and solving that on the fine grid is of course just the original problem again. The way out rests on a single observation: a smooth vector is cheap to represent . It is determined, to good accuracy, by its values at half of the grid points, the rest being recovered by interpolation so the error equation may be solved in a space of half the dimension.
Formally, let P : R n c → R n P : \mathbb{R}^{n_c} \to\mathbb{R}^n P : R n c → R n be the prolongation (linear interpolation from the coarse grid), let R : R n → R n c R : \mathbb{R}^n \to\mathbb{R}^{n_c} R : R n → R n c be the restriction , and define the coarse operator by
A H = R A P . A_H = RAP. A H = R A P .
Taking R ∝ P T R \propto P^{\mathsf{T}} R ∝ P T makes A H A_H A H the Galerkin projection of A A A onto the coarse space; for the 1D Laplacian with full weighting R = 1 2 P T R = \frac{1}{2}P^{\mathsf{T}} R = 2 1 P T it coincides exactly with the direct discretisation on the coarse grid (Problem 4). The correction then reads: restrict the residual, solve A H e H = R r A_H e_H = Rr A H e H = R r , and prolongate, u ← u + P e H u \leftarrow u + Pe_H u ← u + P e H .
Coarse-grid correction cannot be an iterative method on its own. Its error propagation operator is
T = I − P A H − 1 R A , T = I - PA_H^{-1}RA, T = I − P A H − 1 R A ,
and a short computation using A H = R A P A_H = RAP A H = R A P shows that T 2 = T T^2 = T T 2 = T (Problem 5). A projection has eigenvalues 0 and 1 only, so ρ ( T ) = 1 \rho(T) = 1 ρ ( T ) = 1 : whatever lies in ker T \ker T ker T is annihilated in one step, and whatever does not is left completely untouched , forever. Coarse-grid correction removes the part of the error that the coarse space can see, and by construction can do nothing about the rest.
This is the precise sense in which the two ingredients are complementary, and it is why neither is optional. The smoother cannot touch smooth error; the coarse correction cannot touch anything outside its range. Each ingredient is useless on its own; the next subsection shows that together they are optimal.
From coarse correction to the two-grid method
The two ingredients are now in place, and Section 6.3.2 showed that each fails precisely where the other succeeds. Combining them in the obvious order gives the two-grid method : smooth to remove what the coarse space cannot see, correct on the coarse grid to remove what the smoother cannot touch, then smooth again to clean up the interpolation error introduced by P P P . Starting from an approximation u u u :
Pre-smoothing: u ← S ν 1 ( u ) u \leftarrow S^{\nu_1}(u) u ← S ν 1 ( u ) , with S S S weighted Jacobi or Gauss–Seidel.
Residual restriction: r H = R ( f − A u ) r_H = R(f - Au) r H = R ( f − A u ) .
Coarse solve: A H e H = r H A_H e_H = r_H A H e H = r H .
Coarse correction: u ← u + P e H u \leftarrow u + P e_H u ← u + P e H .
Post-smoothing: u ← S ν 2 ( u ) u \leftarrow S^{\nu_2}(u) u ← S ν 2 ( u ) .
From two grids to multigrid. Step 3 still requires an exact solve with n / 2 n/2 n /2 unknowns (n / 4 n/4 n /4 in 2D), which may itself be far too large to solve directly. But the observation that motivated the coarse grid applies again one level down: once the coarse problem has itself been smoothed, its error is smooth relative to H H H and is representable on a grid of spacing 2 H 2H 2 H . So replace the exact coarse solve by one two-grid step on Ω 2 h \Omega_{2h} Ω 2 h , its coarse solve by a two-grid step on Ω 4 h \Omega_{4h} Ω 4 h , and so on until the grid is small enough to solve directly.
That recursion is multigrid , and the grids visited on the way down and back up form the V-cycle of Figure 6.7 . Two properties follow. The work is O ( n ) O(n) O ( n ) , since the grids shrink geometrically; and the error reduction per cycle is independent of h h h , being governed by the smoothing factor of § 9.7.1 e.g. 1 3 \frac{1}{3} 3 1 per step for weighted Jacobi in 1D rather than by κ ( A ) \kappa(A) κ ( A ) .
Figure 6.7. The two-grid method (left) and the V-cycle obtained by applying it recursively (right). Each descending edge is a restriction of the residual, each ascending edge a prolongation of the correction, and relaxation is applied before and after every transfer. Recursion is what makes the method optimal: the coarse solve is itself replaced by a two-grid step until the problem is small enough to solve directly, so one cycle costs O ( n ) O(n) O ( n ) work and reduces the error by a factor independent of h h h . Used as a preconditioner for CG, this turns the O ( h − 1 ) O(h^{-1}) O ( h − 1 ) iteration count of Chapter 4 into a constant.
Takeaways
Smoother and coarse correction are exactly complementary [17 ] [7]: the first removes oscillatory error and cannot touch smooth error, the second removes what lies in the coarse space and cannot touch anything else. Recursion turns the pair into an O ( n ) O(n) O ( n ) solver whose contraction factor does not depend on h h h .
Multigrid as a preconditioner In practice, multigrid is often used not as a standalone solver, but as a preconditioner inside a Krylov method. One multigrid cycle is then viewed as an approximate inverse:
M M G − 1 ≈ A − 1 . M_{\mathrm{MG}}^{-1} \approx A^{-1}. M MG − 1 ≈ A − 1 .
The linear system is solved in preconditioned form, for example by CG or GMRES:
M M G − 1 A x = M M G − 1 b . M_{\mathrm{MG}}^{-1} A x = M_{\mathrm{MG}}^{-1} b. M MG − 1 A x = M MG − 1 b .
This fits naturally into the general framework of preconditioning: multigrid constructs a strong approximate inverse by combining local relaxation with a global coarse-space correction.
Exercises
Problem 1 (Convergence of a parallel Schwarz method in 1D, past exam). Consider the one-dimensional Laplace problem
− u ′ ′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0 , -u''(x)=f(x), \qquad x\in(0,1), \qquad u(0)=u(1)=0, − u ′′ ( x ) = f ( x ) , x ∈ ( 0 , 1 ) , u ( 0 ) = u ( 1 ) = 0 ,
and, as in Proposition 6.2.1 , the overlapping decomposition Ω = Ω 1 ∪ Ω 2 \Omega=\Omega_1\cup\Omega_2 Ω = Ω 1 ∪ Ω 2 with
Ω 1 = ( 0 , β ) , Ω 2 = ( α , 1 ) , 0 < α < β < 1 , δ = β − α > 0. \Omega_1=(0,\beta), \qquad\Omega_2=(\alpha,1), \qquad0<\alpha<\beta<1, \qquad\delta=\beta-\alpha>0. Ω 1 = ( 0 , β ) , Ω 2 = ( α , 1 ) , 0 < α < β < 1 , δ = β − α > 0.
Proposition 6.2.1 treats the alternating (multiplicative) iteration, in which Ω 2 \Omega_2 Ω 2 uses the value just computed on Ω 1 \Omega_1 Ω 1 . Here we study the parallel (additive) iteration, in which both subdomains use the previous iterate:
{ − ( u 1 ( n + 1 ) ) ′ ′ = f in ( 0 , β ) , u 1 ( n + 1 ) ( 0 ) = 0 , u 1 ( n + 1 ) ( β ) = u 2 ( n ) ( β ) , { − ( u 2 ( n + 1 ) ) ′ ′ = f in ( α , 1 ) , u 2 ( n + 1 ) ( α ) = u 1 ( n ) ( α ) , u 2 ( n + 1 ) ( 1 ) = 0. \begin{cases}
-\left(u_1^{(n+1)}\right)''=f & \text{in }(0,\beta),\\
u_1^{(n+1)}(0)=0,\quad u_1^{(n+1)}(\beta)=u_2^{(n)}(\beta),
\end{cases}
\qquad
\begin{cases}
-\left(u_2^{(n+1)}\right)''=f & \text{in }(\alpha,1),\\
u_2^{(n+1)}(\alpha)=u_1^{(n)}(\alpha),\quad u_2^{(n+1)}(1)=0.
\end{cases} ⎩ ⎨ ⎧ − ( u 1 ( n + 1 ) ) ′′ = f u 1 ( n + 1 ) ( 0 ) = 0 , u 1 ( n + 1 ) ( β ) = u 2 ( n ) ( β ) , in ( 0 , β ) , ⎩ ⎨ ⎧ − ( u 2 ( n + 1 ) ) ′′ = f u 2 ( n + 1 ) ( α ) = u 1 ( n ) ( α ) , u 2 ( n + 1 ) ( 1 ) = 0. in ( α , 1 ) ,
where u i ( n ) u_i^{(n)} u i ( n ) is the n n n th iterate on Ω i \Omega_i Ω i .
(a) Let e i ( n ) : = u ∣ Ω i − u i ( n ) e_i^{(n)}:=u|_{\Omega_i}-u_i^{(n)} e i ( n ) := u ∣ Ω i − u i ( n ) denote the errors on each subdomain (where u u u is the exact solution). Write the boundary value problems (BVPs) satisfied by e 1 ( n + 1 ) e_1^{(n+1)} e 1 ( n + 1 ) and e 2 ( n + 1 ) e_2^{(n+1)} e 2 ( n + 1 ) .
(b) Solve these BVPs explicitly (they are affine functions on each subdomain).
(c) Let
η n : = e 2 ( n ) ( β ) , ζ n : = e 1 ( n ) ( α ) . \eta_n:=e_2^{(n)}(\beta), \qquad\zeta_n:=e_1^{(n)}(\alpha). η n := e 2 ( n ) ( β ) , ζ n := e 1 ( n ) ( α ) .
Show that the interface errors satisfy the two coupled recursions
ζ n + 1 = α β η n , η n + 1 = 1 − β 1 − α ζ n , \zeta_{n+1}=\frac{\alpha}{\beta}\eta_n,\qquad
\eta_{n+1}=\frac{1-\beta}{1-\alpha}\zeta_n, ζ n + 1 = β α η n , η n + 1 = 1 − α 1 − β ζ n ,
and hence that every second iterate is contracted by the same factor as in Proposition 6.2.1 :
e 2 ( n ) ( β ) e 2 ( n − 2 ) ( β ) = e 1 ( n ) ( α ) e 1 ( n − 2 ) ( α ) = ρ , ρ : = α β ⋅ 1 − β 1 − α . \frac{e_2^{(n)}(\beta)}{e_2^{(n-2)}(\beta)}
=
\frac{e_1^{(n)}(\alpha)}{e_1^{(n-2)}(\alpha)}
=\rho,\qquad
\rho:=\frac{\alpha}{\beta}\cdot\frac{1-\beta}{1-\alpha}. e 2 ( n − 2 ) ( β ) e 2 ( n ) ( β ) = e 1 ( n − 2 ) ( α ) e 1 ( n ) ( α ) = ρ , ρ := β α ⋅ 1 − α 1 − β .
(d) Prove that 0 < ρ < 1 0<\rho<1 0 < ρ < 1 whenever α < β \alpha<\beta α < β (i.e. when the overlap δ \delta δ is positive), and conclude that the parallel Schwarz method converges. Comment on the price paid for parallelism: how many iterations of the parallel method correspond to one iteration of the alternating method?
Companion notebook: week7/script_week7_2.ipynb
Problem 2 (Spectral equivalence ⇒ \Rightarrow ⇒ PCG convergence). Let A , B A,B A , B be SPD and suppose there exist c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 such that
c 1 x T B x ≤ x T A x ≤ c 2 x T B x ∀ x ∈ R n . c_1x^{\mathsf T}Bx\leq x^{\mathsf T}Ax\leq c_2x^{\mathsf T}Bx
\qquad\forall x\in\mathbb{R}^n. c 1 x T B x ≤ x T A x ≤ c 2 x T B x ∀ x ∈ R n .
(a) Show that spec ( B − 1 / 2 A B − 1 / 2 ) ⊂ [ c 1 , c 2 ] \operatorname{spec}(B^{-1/2}AB^{-1/2})\subset[c_1,c_2] spec ( B − 1/2 A B − 1/2 ) ⊂ [ c 1 , c 2 ] , hence κ ( B − 1 A ) ≤ c 2 / c 1 \kappa(B^{-1}A)\leq c_2/c_1 κ ( B − 1 A ) ≤ c 2 / c 1 .
(b) Let A ~ = B − 1 / 2 A B − 1 / 2 \widetilde{A}=B^{-1/2}AB^{-1/2} A = B − 1/2 A B − 1/2 and e ~ k = B 1 / 2 e k \widetilde{e}_k=B^{1/2}e_k e k = B 1/2 e k , where e k = u k − u ∗ e_k=u_k-u_\ast e k = u k − u ∗ is the error of PCG in the original variables. Show that ∥ e ~ k ∥ A ~ = ∥ e k ∥ A \lVert\widetilde{e}_k\rVert_{\widetilde{A}}=\lVert e_k\rVert_A ∥ e k ∥ A = ∥ e k ∥ A , and conclude that PCG applied to A u = b Au=b A u = b with preconditioner B B B satisfies
∥ e k ∥ A ∥ e 0 ∥ A ≤ 2 ( κ ( B − 1 A ) − 1 κ ( B − 1 A ) + 1 ) k ≤ 2 ( c 2 / c 1 − 1 c 2 / c 1 + 1 ) k . \frac{\lVert e_k\rVert_A}{\lVert e_0\rVert_A}
\leq
2\left(
\frac{\sqrt{\kappa(B^{-1}A)}-1}
{\sqrt{\kappa(B^{-1}A)}+1}
\right)^k
\leq
2\left(
\frac{\sqrt{c_2/c_1}-1}
{\sqrt{c_2/c_1}+1}
\right)^k. ∥ e 0 ∥ A ∥ e k ∥ A ≤ 2 ( κ ( B − 1 A ) + 1 κ ( B − 1 A ) − 1 ) k ≤ 2 ( c 2 / c 1 + 1 c 2 / c 1 − 1 ) k .
In words: spectral equivalence with constants independent of h h h gives an iteration count independent of h h h .
Exercises
Problem 3 (Preconditioned conjugate gradient). Let A , M A, M A , M be SPD. Apply CG to A ~ x ~ = b ~ \widetilde{A}\widetilde{x}=\widetilde{b} A x = b with A ~ = M − 1 / 2 A M − 1 / 2 \widetilde{A}=M^{-1/2}AM^{-1/2} A = M − 1/2 A M − 1/2 and b ~ = M − 1 / 2 b \widetilde{b}=M^{-1/2}b b = M − 1/2 b . Show that the mapped iterates x ( k ) = M − 1 / 2 x ~ ( k ) x^{(k)}=M^{-1/2}\widetilde{x}^{(k)} x ( k ) = M − 1/2 x ( k ) satisfy the iteration in the proposition; in particular p ( 0 ) = M − 1 r ( 0 ) p^{(0)}=M^{-1}r^{(0)} p ( 0 ) = M − 1 r ( 0 ) , and
α k = ⟨ r ( k ) , z ( k ) ⟩ ⟨ p ( k ) , A p ( k ) ⟩ , β k + 1 = ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ ⟨ r ( k ) , z ( k ) ⟩ , z ( k ) = M − 1 r ( k ) . \alpha_k=\frac{\langle r^{(k)},z^{(k)}\rangle}{\langle p^{(k)},Ap^{(k)}\rangle},\qquad\beta_{k+1}=\frac{\langle r^{(k+1)},z^{(k+1)}\rangle}{\langle r^{(k)},z^{(k)}\rangle},\qquad z^{(k)}=M^{-1}r^{(k)}. α k = ⟨ p ( k ) , A p ( k ) ⟩ ⟨ r ( k ) , z ( k ) ⟩ , β k + 1 = ⟨ r ( k ) , z ( k ) ⟩ ⟨ r ( k + 1 ) , z ( k + 1 ) ⟩ , z ( k ) = M − 1 r ( k ) . Companion notebook: week7/script_week7_1.ipynb
Problem 4 (Transfer operators and the Galerkin coarse matrix). Let
A h = 1 h 2 tridiag ( − 1 , 2 , − 1 ) ∈ R m × m A_h=\frac{1}{h^2}\operatorname{tridiag}(-1,2,-1)\in\mathbb{R}^{m\times m} A h = h 2 1 tridiag ( − 1 , 2 , − 1 ) ∈ R m × m with m = 2 m H + 1 m=2m_H+1 m = 2 m H + 1 , h = 1 / ( m + 1 ) h=1/(m+1) h = 1/ ( m + 1 ) , and let P ∈ R m × m H P\in\mathbb{R}^{m\times m_H} P ∈ R m × m H be linear interpolation from the coarse grid H = 2 h H=2h H = 2 h : ( P e H ) 2 j = 1 2 ( e H ) j (Pe_H)_{2j}=\frac{1}{2}(e_H)_j ( P e H ) 2 j = 2 1 ( e H ) j , ( P e H ) 2 j + 1 = ( e H ) j (Pe_H)_{2j+1}=(e_H)_j ( P e H ) 2 j + 1 = ( e H ) j , ( P e H ) 2 j + 2 = 1 2 ( e H ) j (Pe_H)_{2j+2}=\frac{1}{2}(e_H)_j ( P e H ) 2 j + 2 = 2 1 ( e H ) j .
(a) Write P P P explicitly for m = 7 m=7 m = 7 , m H = 3 m_H=3 m H = 3 , and show that the full weighting restriction R = 1 2 P T R=\frac{1}{2}P^{\mathsf{T}} R = 2 1 P T applies the stencil 1 4 ( 1 , 2 , 1 ) \frac{1}{4}(1,2,1) 4 1 ( 1 , 2 , 1 ) .
(b) Compute A H = R A h P A_H=RA_hP A H = R A h P for m = 7 m=7 m = 7 and verify that it coincides exactly with the direct discretisation 1 H 2 tridiag ( − 1 , 2 , − 1 ) \frac{1}{H^2}\operatorname{tridiag}(-1,2,-1) H 2 1 tridiag ( − 1 , 2 , − 1 ) on the coarse grid.
(c) What do you get from P T A h P P^{\mathsf{T}}A_hP P T A h P instead, and why must the scalings of R R R and P P P be consistent?
Problem 5 (Coarse-grid correction alone cannot converge). With A h A_h A h , P P P , R R R and A H = R A h P A_H=RA_hP A H = R A h P as in Problem 4, the coarse-grid correction u ← u + P A H − 1 R ( f − A h u ) u\leftarrow u+PA_H^{-1}R(f-A_hu) u ← u + P A H − 1 R ( f − A h u ) has error propagation operator T = I − P A H − 1 R A h T=I-PA_H^{-1}RA_h T = I − P A H − 1 R A h .
(a) Show that T 2 = T T^2=T T 2 = T (use A H = R A h P A_H=RA_hP A H = R A h P to simplify P A H − 1 R A h P PA_H^{-1}RA_hP P A H − 1 R A h P ), and deduce that every eigenvalue of T T T is 0 or 1, that ρ ( T ) = 1 \rho(T)=1 ρ ( T ) = 1 when m H < m m_H<m m H < m , and that rank T = m − m H \operatorname{rank}T=m-m_H rank T = m − m H .
(b) Conclude that coarse-grid correction alone is not convergent, identify which errors lie in ker T \ker T ker T (compare Figure 6.6 ), and explain in one sentence why a smoother is therefore mandatory rather than merely helpful.
Figure 6.6. Why a coarse space can carry the correction. The thin grey curve is a random initial error on a grid with m = 31 m = 31 m = 31 ; the thick blue curve is the same error after five weighted-Jacobi sweeps (ω = 2 3 \omega= \frac{2}{3} ω = 3 2 ); the dashed orange curve is the piecewise-linear interpolant through its values at the 15 coarse nodes only.
Problem 6 (The two-grid contraction factor). Let S = I − ω D − 1 A h S=I-\omega D^{-1}A_h S = I − ω D − 1 A h be the weighted Jacobi smoother, D = diag ( A h ) D=\operatorname{diag}(A_h) D = diag ( A h ) , and T T T as in Problem 5; the two-grid operator with ν \nu ν pre-smoothing steps is M T G = T S ν M_{\mathrm{TG}}=TS^\nu M TG = T S ν .
(a) From the eigenvalues μ j ( ω ) = 1 − 2 ω sin 2 ( j π 2 ( m + 1 ) ) \mu_j(\omega)=1-2\omega\sin^2\left(\frac{j\pi}{2(m+1)}\right) μ j ( ω ) = 1 − 2 ω sin 2 ( 2 ( m + 1 ) j π ) of S S S , explain why ω = 1 \omega=1 ω = 1 is a poor smoother even though it minimises ρ ( S ) \rho(S) ρ ( S ) .
(b) Take ω = 2 3 \omega=\frac{2}{3} ω = 3 2 and m = 7 m=7 m = 7 . Compute ρ ( M T G ) \rho(M_{\mathrm{TG}}) ρ ( M TG ) numerically for ν = 1 \nu=1 ν = 1 and ν = 2 \nu=2 ν = 2 , and compare with the smoothing factor 1 3 \frac{1}{3} 3 1 of Proposition 6.3.3 . State the relationship you observe between ρ ( M T G ) \rho(M_{\mathrm{TG}}) ρ ( M TG ) and ν \nu ν .
(c) Repeat for m = 15 m=15 m = 15 and m = 31 m=31 m = 31 : is the contraction factor independent of h h h ? Contrast with the O ( h − 2 ) O(h^{-2}) O ( h − 2 ) iteration count of unpreconditioned Jacobi in Chapter 3.
(d) A V-cycle visits grids of size m m m , m / 2 m/2 m /2 , m / 4 m/4 m /4 , … \ldots … ; show its total cost is O ( m ) O(m) O ( m ) , and explain why this together with (c) makes multigrid an optimal solver.
Companion notebook: week7/script_week7_3.ipynb
About these notes
Acknowledgements. We thank Jan Willem van Beek, instructor for the course, for his contribution to the teaching and to these notes. We are also grateful to the students of previous editions, whose questions have shaped much of the exposition, and to colleagues who commented on earlier drafts.
Code. The numerical illustrations in these notes are reproducible. The accompanying Python code is available at
https://github.com/vicdolean/scicomp_examples
which is a fork of the repository maintained by Jan Willem van Beek, https://github.com/jwfvanbeek/scicomp_examples . The two are kept synchronised, and any divergence between them is stated in the repositories themselves. Readers are encouraged to run and modify the code: changing a parameter, a matrix or a stopping criterion and watching what happens to the convergence history is by far the fastest way to develop a feeling for when these methods work and when they do not.
Bibliography