betaFeedback

WorksMathematics

Statistical robustness and time consistency in risk averse optimization

Abstract

The aim of this manuscript is to investigate risk measures which are not necessarily convex. We show that law invariant coherent risk measures are not robust with respect to contamination of the probability measure, in the sense that an arbitrarily small contamination can change their values by an arbitrary amount. This contrasts with the Value-at-Risk, which is robust against such contamination. In addition, we show that for a given law invariant risk measure and a given policy, one can construct, under mild conditions, conditional risk measures with respect to which this policy is time consistent, in the frameworks of both Markov Decision Processes and Stochastic Optimal Control.

Introduction

In this manuscript, we study risk measures which do not necessarily satisfy the convexity axiom, and their use in sequential decision making under uncertainty. Modern theory of risk measures was started in [2], where the axioms of the so-called coherent risk measures were formulated. An important example of a coherent risk measure is the Average Value-at-Risk, which can be written as the infimum of a family of expectations [7]. More generally, spectral risk measures with monotonically nondecreasing spectral functions are coherent [1], and on a nonatomic probability space, law invariant coherent risk measures can be represented as the supremum of a set of coherent spectral risk measures (e.g., [12], Section 6.3.4). Coherent risk measures also possess the dual representation, which relates the risk averse and distributionally robust approaches to stochastic optimization (e.g., [12], Section 6.3). On the other hand, the Value-at-Risk, which is widely used in applications, does not satisfy the convexity axiom and does not have such dual representation.

The convexity of risk measures, however, comes at a price from the statistical point of view. It is well known that the Value-at-Risk is much less sensitive to outliers than the expected value (cf., [4, 8]). That is, when the distribution is contaminated by a small mass placed far away, the Value-at-Risk is not changed much, while the expected value can be moved by an arbitrary amount. It appears that the same property of statistical nonrobustness is shared by a large class of coherent risk measures. Indeed, the Average Value-at-Risk and spectral coherent risk measures are concave with respect to the probability measure, and consequently an arbitrarily small contamination of the distribution can increase their values by an arbitrary amount. Actually some authors argue that this is an advantage of the Average Value-at-Risk compared with the Value-at-Risk, since it accounts for possible large losses (cf., [7]). In any case, law invariant coherent risk measures are not robust from the statistical point of view, which motivates us to consider risk measures that do not satisfy the convexity axiom.

For sequential decision making, an approach to risk averse multistage stochastic programming based on nested compositions of conditional risk measures was suggested in [10], and was extended to Markov Decision Processes (MDPs) in [9]. A construction of nested risk functionals which does not rely on the convexity of risk measures was recently discussed in [13], for both Stochastic Optimal Control (SOC) and MDPs.

Closely related to the nested formulation is the time consistency of policies, that is, whether a policy optimal at the initial stage remains optimal at every later stage conditional on the history of the process. It is well known that the nested formulation is time consistent, while this is not the case for static formulations based on the Average Value-at-Risk. It should be noted that optimality at every later stage is measured by conditional risk measures, and hence time consistency depends on the choice of these conditional risk measures (e.g., [6]). For a given static risk measure, a natural choice is the one for which the static risk measure is equal to the composition of the conditional risk measures, since in that case the static problem can be written in the nested form. However, such a choice with law invariant coherent conditional risk measures exists only for the expectation and the max-risk measure (cf., [5], [11]), and in particular does not exist for the Average Value-at-Risk. In addition, even when such a choice exists, an optimal policy is not necessarily time consistent if the conditional risk measures are not strictly monotone [14, 15]. In this manuscript, for a given policy such that no policy is uniformly better, we construct conditional risk measures with respect to which this policy is time consistent, where the static risk measure is not necessarily convex.

The main contribution of this manuscript can be summarized as follows.

  • We show that risk measures concave in the probability measure, as well as their suprema, are not robust with respect to contamination of the probability measure. In particular, this

applies to spectral coherent risk measures, such as the Average Value-at-Risk, and to law invariant coherent risk measures on nonatomic probability spaces. This contrasts with the Value-at-Risk, which is robust against such contamination.

  • For an arbitrary law invariant risk measure satisfying the axioms of monotonicity, translation equivariance and positive homogeneity, and a policy such that no policy is uniformly better, we construct conditional risk measures with respect to which this policy is time consistent. The construction is based on the tail events on which the given policy is not worse than any other policy, and is carried out for both MDPs and SOC.

The remainder of this manuscript is organized as follows. In § 2, we discuss risk functionals which are not necessarily convex, and their concavity and statistical robustness with respect to the probability measure. In § 3.1 and § 3.2, we construct conditional risk measures with respect to which a given policy is time consistent, in the MDP and SOC settings, respectively. We conclude the manuscript in § 4.

We use the following notation and terminology. For a∈Ra \in\mathbb{R}, we denote [a]+:=max⁡{0,a}[a]_{+}:=\max\{0,a\}. A set Ω\Omega equipped with its sigma algebra F\mathcal{F} is called a measurable space. By P\mathcal{P} we denote the set of all probability measures on (Ω,F)(\Omega,\mathcal{F}). For P∈PP \in\mathcal{P}, (Ω,F,P)(\Omega,\mathcal{F},P) is called the probability space. It is said that P∈PP \in\mathcal{P} is nonatomic, if for any A∈FA \in\mathcal{F} such that P(A)>0P(A)>0, there exists A′∈FA' \in\mathcal{F} such that A′⊂AA' \subset A and P(A)>P(A′)>0P(A)>P(A')>0. By δa\delta_a we denote the Dirac measure of mass one at the point aa.

For P∈PP \in\mathcal{P} and measurable variable Z:Ω→RZ:\Omega\to\mathbb{R} we denote by EP[Z]=∫ΩZdP\mathbb{E}^{P}[Z]=\int_{\Omega} Z\mathrm{d}P its expected value, and by

FZP(z):=P(Z≤z),z∈R,F_Z^P(z):=P(Z\leq z),\quad z\in\mathbb{R},

its cumulative distribution function (cdf). By

(FZP)−1(α):=inf⁡{z:FZP(z)≥α},α∈[0,1],(F_Z^P)^{-1}(\alpha):=\inf\{z:F_Z^P(z)\geq\alpha\},\quad\alpha\in[0,1],

we denote the left side quantile. In particular if Z∼PZ\sim P we write FP(z)F^P(z) and (FP)−1(α)(F^P)^{-1}(\alpha) for the respective cdf and quantile. By F~\widetilde{\mathcal{F}} we denote the set of right side continuous monotonically nondecreasing functions ϕ:R→R\phi:\mathbb{R}\to\mathbb{R} such that lim⁡z→−∞ϕ(z)=0\lim_{z\to-\infty}\phi(z)=0 and lim⁡z→+∞ϕ(z)=1\lim_{z\to+\infty}\phi(z)=1, i.e. F~\widetilde{\mathcal{F}} is the space of cdfs. By Lp(Ω,F,P)L_p(\Omega,\mathcal{F},P) we denote the space of random variables ZZ such that

∫Ω∣Z∣pdP<∞,p∈[1,∞).\int_{\Omega}|Z|^p\mathrm{d}P<\infty,\quad p\in[1,\infty).

Risk functionals

Let (Ω,F)(\Omega,\mathcal{F}) be a measurable space. Assume that with every probability measure P∈PP\in\mathcal{P} is associated a linear space Z\mathcal{Z} of measurable functions (random variables) Z:Ω→RZ:\Omega\to\mathbb{R} and a functional R:Z→R\mathcal{R}:\mathcal{Z}\to\mathbb{R}. Consider the following axioms (conditions) that R\mathcal{R} may satisfy.

(A1) (monotonicity) if Z,Z′∈ZZ,Z'\in\mathcal{Z} and Z≥Z′Z\geq Z', PP-a.s., then R(Z)≥R(Z′)\mathcal{R}(Z)\geq\mathcal{R}(Z').

(A2) (translation equivariance) for Z∈ZZ\in\mathcal{Z} and τ∈R\tau\in\mathbb{R}, it follows that R(Z+τ)=R(Z)+τ\mathcal{R}(Z+\tau)=\mathcal{R}(Z)+\tau.

(A3) (positive homogeneity) if Z∈ZZ\in\mathcal{Z} and τ≥0\tau\geq0, then R(τZ)=τR(Z)\mathcal{R}(\tau Z)=\tau\mathcal{R}(Z).

(A4) (convexity) if Z,Z′∈ZZ,Z'\in\mathcal{Z} and τ∈[0,1]\tau\in[0,1], then R(τZ+(1−τ)Z′)≤τR(Z)+(1−τ)R(Z′)\mathcal{R}(\tau Z+(1-\tau)Z')\leq\tau\mathcal{R}(Z)+(1-\tau)\mathcal{R}(Z').

In order to emphasize that the functional R\mathcal{R} depends on P∈PP\in\mathcal{P}, we write it as RP\mathcal{R}^{P}. The corresponding linear space Z\mathcal{Z} may also depend on PP, but we suppress this in the notation. In any case, we assume that if Z∈ZZ\in\mathcal{Z} and τ∈R\tau\in\mathbb{R}, then Z+τ∈ZZ+\tau\in\mathcal{Z}. A typical example of the space Z\mathcal{Z} is Lp(Ω,F,P)L_p(\Omega,\mathcal{F},P), p∈[1,∞]p\in[1,\infty]. Note that it follows from axiom (A1) that if Z=Z′Z=Z', PP-a.s., then RP(Z)=RP(Z′)\mathcal{R}^{P}(Z)=\mathcal{R}^{P}(Z'). It follows from axiom (A3) that RP(0)=0\mathcal{R}^{P}(0)=0.

Definition 2.1. It is said that functional RP\mathcal{R}^{P} is a risk measure if it satisfies axioms (A1) and (A2).

Functionals satisfying axioms (A1)-(A4) are called coherent risk measures [2]. An important example of a non-convex risk measure is the Value-at-Risk, RP=V@R⁡αP\mathcal{R}^{P}=\operatorname{V@R}_{\alpha}^{P}, where

V@R⁡αP(Z):=inf⁡{z:FZP(z)≥1−α}=(FZP)−1(1−α),α∈(0,1).(1)\operatorname{V@R}_{\alpha}^{P}(Z) := \inf\{z : F_{Z}^{P}(z) \ge1-\alpha\} = (F_{Z}^{P})^{-1}(1-\alpha), \quad\alpha\in(0,1). \tag*{(1)}

That is, V@R⁡αP(Z)\operatorname{V@R}_{\alpha}^{P}(Z) is the left side (1−α)(1-\alpha)-quantile of the distribution of ZZ. In that example the corresponding space Z\mathcal{Z} consists of all measurable Z:Ω→RZ:\Omega\to\mathbb{R}. The V@R⁡αP(⋅)\operatorname{V@R}_{\alpha}^{P}(\cdot) functional satisfies axioms (A1)-(A3), but not the convexity axiom (A4). An important example of a coherent risk measure is the Average Value-at-Risk (also called Conditional Value-at-Risk, expected shortfall, expected tail loss), for α∈(0,1]\alpha\in(0,1],

AV@R⁡αP(Z)=inf⁡τ∈R{τ+α−1EP[Z−τ]+}=α−1∫0αV@R⁡τP(Z)dτ.(2)\operatorname{AV@R}_{\alpha}^{P}(Z)=\inf_{\tau\in\mathbb{R}}\{\tau+\alpha^{-1}\mathbb{E}^{P}[Z-\tau]_{+}\}=\alpha^{-1}\int_{0}^{\alpha}\operatorname{V@R}_{\tau}^{P}(Z)d\tau. \tag*{(2)}

The first of the above representations of the Average Value-at-Risk is due to [7]. In that example, the respective space is Z=L1(Ω,F,P)\mathcal{Z}=L_{1}(\Omega,\mathcal{F},P).

It is said that Z,Z′∈ZZ,Z'\in\mathcal{Z} are distributionally equivalent (with respect to P∈PP\in\mathcal{P}), if their cdfs do coincide, i.e., P(Z≤z)=P(Z′≤z)P(Z\le z)=P(Z'\le z) for all z∈Rz\in\mathbb{R}.

Definition 2.2. It is said that RP\mathcal{R}^{P} is law invariant if for any distributionally equivalent Z,Z′∈ZZ,Z'\in\mathcal{Z} it follows that RP(Z)=RP(Z′)\mathcal{R}^{P}(Z)=\mathcal{R}^{P}(Z').

That is, a law invariant risk measure RP(Z)\mathcal{R}^{P}(Z) is a function of the cdf FZPF_{Z}^{P}, i.e., it can be represented as

RP(Z)=ρ(FZP),Z∈Z,(3)\mathcal{R}^{P}(Z)=\rho(F_{Z}^{P}), \quad Z\in\mathcal{Z}, \tag*{(3)}

where ρ\rho is a mapping from the space of cdfs F\mathfrak{F} into R\mathbb{R}. For example, for RP=AV@R⁡αP\mathcal{R}^{P}=\operatorname{AV@R}_{\alpha}^{P} the corresponding

ρ(ϕ)=inf⁡τ∈R{τ+α−1∫−∞+∞[z−τ]+dϕ(z)},\rho(\phi)=\inf_{\tau\in\mathbb{R}}\left\{\tau+\alpha^{-1}\int_{-\infty}^{+\infty}[z-\tau]_{+}d\phi(z)\right\},

defined for such ϕ∈F\phi\in\mathfrak{F} that ∫−∞+∞[z−τ]+dϕ(z)<∞\int_{-\infty}^{+\infty}[z-\tau]_{+}d\phi(z)<\infty for all τ∈R\tau\in\mathbb{R}. For RP=V@R⁡αP\mathcal{R}^{P}=\operatorname{V@R}_{\alpha}^{P} the corresponding mapping is

ρ(ϕ)=inf⁡{z:ϕ(z)≥1−α},ϕ∈F.\rho(\phi)=\inf\{z:\phi(z)\ge1-\alpha\}, \quad\phi\in\mathfrak{F}.

Risk measure R\mathcal{R} is called spectral if it can be represented in the form

RP(Z)=∫01σ(τ)(FZP)−1(τ)dτ,(4)\mathcal{R}^{P}(Z)=\int_{0}^{1}\sigma(\tau)(F_{Z}^{P})^{-1}(\tau)d\tau, \tag*{(4)}

where σ(⋅)\sigma(\cdot) is a density function on [0,1][0,1]. Spectral risk measures were introduced in Acerbi [1]. Spectral risk measures are law invariant. It was shown in [1] that the spectral risk measure is coherent in the sense of [2], if the corresponding density σ(⋅)\sigma(\cdot) is monotonically nondecreasing on [0,1][0,1]. When the space Z=Lp(Ω,F,P)\mathcal{Z}=L_{p}(\Omega,\mathcal{F},P), it is assumed that the spectral function σ\sigma belongs to the dual space Z∗=Lq(Ω,F,P)\mathcal{Z}^{*}=L_{q}(\Omega,\mathcal{F},P).

Remark 2.1. Assuming that the probability space (Ω,F,P)(\Omega,\mathcal{F},P) is nonatomic, it is possible to show that any law invariant coherent risk measure can be represented as the supremum of a set of coherent spectral risk measures (e.g., [12 Section 6.3.4]). This motivates us to consider risk measures of the form

RP(Z):=sup⁡μ∈M∫01(FZP)−1(τ) dμ(τ),(5)\mathcal{R}^{P}(Z) := \sup_{\mu\in\mathcal{M}} \int_{0}^{1} (F_{Z}^{P})^{-1}(\tau) \, d\mu(\tau), \tag*{(5)}

where M\mathcal{M} is a set of probability measures on the interval [0,1][0,1].

Definition 2.3. We refer to risk measures of the form (5) as the generalized risk measures. We say that a generalized risk measure is generalized spectral if the set M={μˉ}\mathcal{M} = \{\bar{\mu}\} is a singleton.

The generalized risk measure also depends on a choice of the set M\mathcal{M}. In order to emphasize this, we sometimes write it as RMP(⋅)\mathcal{R}_{\mathcal{M}}^{P}(\cdot). The generalized risk measures are law invariant and satisfy axioms (A1)-(A3), but may not satisfy the convexity axiom (A4). By the above we have that a generalized risk measure is coherent if the probability measures μ∈M\mu\in\mathcal{M} have monotonically nondecreasing densities. An important example of a non-convex generalized spectral risk measure is V@R⁡αP\operatorname{V@R}_{\alpha}^{P}, which can be written in the form (5) with M\mathcal{M} being the singleton M={δ1−α}\mathcal{M} = \{\delta_{1-\alpha}\}.

Concavity of risk measures

Definition 2.4. A law invariant functional RP:Z→R\mathcal{R}^{P}: \mathcal{Z} \to\mathbb{R} is said to be concave in PP, if for P,P′∈PP, P' \in\mathcal{P} and τ∈[0,1]\tau\in[0,1] it follows for all Z∈ZZ \in\mathcal{Z} that

RτP+(1−τ)P′(Z)≥τRP(Z)+(1−τ)RP′(Z).(6)\mathcal{R}^{\tau P+(1-\tau)P'}(Z) \ge\tau\mathcal{R}^{P}(Z) + (1-\tau)\mathcal{R}^{P'}(Z). \tag*{(6)}

Such concavity property of risk measures is useful, for example, in construction of upper bounds in risk averse SDDP type algorithms [3], and in establishing existence of non-randomized policies in risk averse MDPs [13].

There is a large class of risk functionals which are concave with respect to the probability distribution. For example consider risk functionals of the form RP(Z)=inf⁡θ∈ΘEP[Ψ(Z,θ)]\mathcal{R}^{P}(Z) = \inf_{\theta\in\Theta} \mathbb{E}^{P}[\Psi(Z,\theta)], where Θ\Theta is a subset of a finite dimensional vector space and Ψ:R×Θ→R\Psi: \mathbb{R} \times\Theta\to\mathbb{R} is a real valued function. The AV@R⁡α\operatorname{AV@R}_{\alpha} risk measure is of that form. In that case RP(Z)\mathcal{R}^{P}(Z) is given by the infimum of functionals linear in PP, and hence is concave in PP. Spectral coherent risk measures are representable as the corresponding integral of the Average Value-at-Risk measures and hence are concave in PP.

We refer to the interval [0,1][0,1] equipped with its Borel sigma algebra as the standard sample space. We can view generalized risk measures as defined on the standard sample space. Consider the Bernoulli probability measure Bτ=τδ0+(1−τ)δ1B_{\tau} = \tau\delta_{0} + (1-\tau)\delta_{1}, τ∈[0,1]\tau\in[0,1]. With a law invariant risk measure RP\mathcal{R}^{P} and Z∈ZZ \in\mathcal{Z} we can associate the function

pZ(τ):=RBτ(Z),τ∈[0,1].(7)p_{Z}(\tau) := \mathcal{R}^{B_{\tau}}(Z), \quad\tau\in[0,1]. \tag*{(7)}

Since for γ,τ1,τ2∈[0,1]\gamma, \tau_{1}, \tau_{2} \in[0,1],

Bγτ1+(1−γ)τ2=γ(τ1δ0+(1−τ1)δ1)+(1−γ)(τ2δ0+(1−τ2)δ1)=γBτ1+(1−γ)Bτ2,B_{\gamma\tau_{1}+(1-\gamma)\tau_{2}} = \gamma(\tau_{1}\delta_{0} + (1-\tau_{1})\delta_{1}) + (1-\gamma)(\tau_{2}\delta_{0} + (1-\tau_{2})\delta_{1}) = \gamma B_{\tau_{1}} + (1-\gamma)B_{\tau_{2}},

it follows that if RP\mathcal{R}^{P} is concave in PP, then the function pZ(⋅)p_{Z}(\cdot) is concave on [0,1][0,1] for all Z∈ZZ \in\mathcal{Z}. For Z∼BτZ \sim B_{\tau} we write p(τ)p(\tau) for the corresponding function.

Let us consider some examples. Suppose that Z∼BτZ \sim B_{\tau}. Then we have that FZBτ(z)=0F^{B_{\tau}}_{Z}(z) = 0 for z<0z < 0, FZBτ(z)=τF^{B_{\tau}}_{Z}(z) = \tau for z∈[0,1)z \in[0,1) and FZBτ(z)=1F^{B_{\tau}}_{Z}(z) = 1 for z≥1z \ge1. Thus (FZBτ)−1(β)=0(F^{B_{\tau}}_{Z})^{-1}(\beta) = 0 for β∈(0,τ)\beta\in(0,\tau), and (FZBτ)−1(β)=1(F^{B_{\tau}}_{Z})^{-1}(\beta) = 1 for β∈[τ,1]\beta\in[\tau,1]. Consider RP(Z):=V@R⁡αP(Z)=(FZP)−1(1−α)\mathcal{R}^{P}(Z) := \operatorname{V@R}_{\alpha}^{P}(Z) = (F_{Z}^{P})^{-1}(1-\alpha) for some α∈(0,1)\alpha\in(0,1). Then for Z∼BτZ \sim B_{\tau},

p(τ)={0,for τ∈[0,α),1,for τ∈[α,1].(8)p(\tau) = \begin{cases} 0, & \text{for } \tau\in[0,\alpha),\\ 1, & \text{for } \tau\in[\alpha,1]. \end{cases} \tag*{(8)}

Here p(⋅)\mathfrak{p}(\cdot) cannot be represented as the supremum of functions concave on the interval [0,1][0,1], and hence V⁡@RαP(Z)\operatorname{V}@\mathcal{R}^{P}_{\alpha}(Z) cannot be represented as the supremum of law invariant risk measures concave in PP.

Consider now RP(Z):=AV⁡@RαP(Z)\mathcal{R}^{P}(Z) := \operatorname{AV}@\mathcal{R}^{P}_{\alpha}(Z). In that case by (2) we have that for Z∼BτZ \sim B_{\tau},

p(τ)={α−1τ,for τ∈[0,α),1,for τ∈[α,1].(9)\mathfrak{p}(\tau) = \begin{cases} \alpha^{-1}\tau, & \text{for } \tau\in[0,\alpha),\\ 1, & \text{for } \tau\in[\alpha,1]. \end{cases} \tag*{(9)}

Of course this function is concave on [0,1][0,1].

Statistical robustness of risk functionals

It is well known that the Value-at-Risk (quantile) functionals V⁡@RαP\operatorname{V}@\mathcal{R}^{P}_{\alpha} are much less sensitive to outliers than the expected value EP\mathbb{E}^{P}. That is, for a probability measure PP supported on the real line and ε∈(0,1)\varepsilon\in(0,1) consider the contamination (1−ε)P+εδκ(1-\varepsilon)P+\varepsilon\delta_{\kappa} of the original distribution PP by the scaled Dirac measure. Then

E(1−ε)P+εδκ[⋅]=(1−ε)EP[⋅]+εκ.\mathbb{E}^{(1-\varepsilon)P+\varepsilon\delta_{\kappa}}[\cdot] = (1-\varepsilon)\mathbb{E}^{P}[\cdot] + \varepsilon\kappa.

By choosing κ∈R\kappa\in\mathbb{R} large enough, the contaminated expectation can be moved from the expectation EP\mathbb{E}^{P} by an arbitrary amount for any positive value of ε\varepsilon. On the other hand the V⁡@RαP\operatorname{V}@\mathcal{R}^{P}_{\alpha} measure will not be changed much by contamination for ε<α\varepsilon< \alpha. The respective theory of the so-called Robust Statistics was well discussed already more than 40 years ago (cf., [4], [8]).

Suppose now that risk measure RP\mathcal{R}^{P} satisfies axioms (A1)-(A3) and is concave in PP. Then for ε∈(0,1)\varepsilon\in(0,1),

R(1−ε)P+εδκ(⋅)≥(1−ε)RP(⋅)+εRδκ(⋅)=(1−ε)RP(⋅)+εκ,(10)\mathcal{R}^{(1-\varepsilon)P+\varepsilon\delta_{\kappa}}(\cdot) \geq(1-\varepsilon)\mathcal{R}^{P}(\cdot) + \varepsilon\mathcal{R}^{\delta_{\kappa}}(\cdot) = (1-\varepsilon)\mathcal{R}^{P}(\cdot) + \varepsilon\kappa, \tag*{(10)}

where the last equality holds since δκ\delta_{\kappa} is the distribution of the constant κ\kappa, and hence Rδκ(⋅)=κ\mathcal{R}^{\delta_{\kappa}}(\cdot)=\kappa by axioms (A2)-(A3). As in the expected value example, the risk R(1−ε)P+εδκ(⋅)\mathcal{R}^{(1-\varepsilon)P+\varepsilon\delta_{\kappa}}(\cdot) can be moved from RP(⋅)\mathcal{R}^{P}(\cdot) by an arbitrary amount for any positive value of ε\varepsilon. As was already mentioned, spectral coherent risk measures are concave in PP, and hence are not robust from the statistical point of view. Moreover, consider functionals RP(⋅):=sup⁡i∈IRiP(⋅)\mathcal{R}^{P}(\cdot):=\sup_{i\in I}\mathcal{R}^{P}_{i}(\cdot), where {RiP}i∈I\{\mathcal{R}^{P}_{i}\}_{i\in I} is a family of risk measures such that for every i∈Ii\in I the risk measure RiP(⋅)\mathcal{R}^{P}_{i}(\cdot) is law invariant, satisfies axioms (A1)-(A3) and is concave in PP. Then by (10)

R(1−ε)P+εδκ(⋅)=sup⁡i∈IRi(1−ε)P+εδκ(⋅)≥sup⁡i∈I{(1−ε)RiP(⋅)+εκ}=(1−ε)RP(⋅)+εκ.\mathcal{R}^{(1-\varepsilon)P+\varepsilon\delta_{\kappa}}(\cdot) = \sup_{i\in I}\mathcal{R}^{(1-\varepsilon)P+\varepsilon\delta_{\kappa}}_{i}(\cdot) \geq\sup_{i\in I}\{(1-\varepsilon)\mathcal{R}^{P}_{i}(\cdot)+\varepsilon\kappa\} = (1-\varepsilon)\mathcal{R}^{P}(\cdot)+\varepsilon\kappa.

and hence RP\mathcal{R}^{P} is not robust from the statistical point of view. In particular, it follows from Remark 2.1 that law invariant coherent risk measures on a nonatomic probability space are not robust from the statistical point of view.

Time consistency

In this section, we discuss time consistency of policies with respect to nested formulations of risk averse problems. Specifically, in § 3.1, for a given policy of an MDP such that no policy is uniformly better, we construct conditional risk measures with respect to which this policy is time consistent, and in § 3.2, we carry out a similar construction for SOC.

Markov Decision Processes

Consider a finite horizon MDP with state spaces StS_t, action spaces AtA_t, cost functions ct(st,at,st+1)c_t(s_t,a_t,s_{t+1}), and transition probabilities Pt(st+1∣at,st)P_t(s_{t+1}|a_t,s_t), t=1,…,Tt=1,\ldots,T. A (non-randomized) policy π\pi is defined by mappings πt:(s1,…,st)↦At\pi_t:(s_1,\ldots,s_t)\mapsto A_t. A policy is Markovian if πt(st)\pi_t(s_t), t=1,…,Tt=1,\ldots,T, are functions of sts_t rather than the whole history (s1,…,st)(s_1,\ldots,s_t). To a policy π\pi corresponds the probability distribution defined on histories ht=(s1,…,st)h_t=(s_1,\ldots,s_t) with aτ=πτ(s1,…,sτ)a_\tau=\pi_\tau(s_1,\ldots,s_\tau), τ=1,…,t\tau=1,\ldots,t (cf., [16]).

We say that a policy πˉ\bar{\pi} is time consistent if at every stage tt, conditional on the history hth_t, the policy {πˉt,…,πˉT}\{\bar{\pi}_t,\ldots,\bar{\pi}_T\} is optimal. In order to make this precise we need to define what we mean by optimality at every stage tt. Suppose that for every tt the state space StS_t is a measurable space. Consider the set Π\Pi of Markovian policies. Let RP\mathcal{R}^{P} be a law invariant risk measure with PP being a probability measure defined on (s1,…,sT)∈S1×⋯×ST(s_1,\ldots,s_T)\in S_1\times\cdots\times S_T. Let Ptst,atP_t^{s_t,a_t}, t=1,…,Tt=1,\ldots,T, be reference transition kernels. Then to a policy π∈Π\pi\in\Pi corresponds the nested functional Rπ\mathcal{R}^{\pi} given by the composition of conditional risk measures RtPst,πt(st)\mathcal{R}_t^{P^{s_t,\pi_t(s_t)}} (cf., [13 Section 4]). Let us make the following observation.

Remark 3.1. Let P\mathcal{P} be the set of probability measures on a measurable space (Ω,F)(\Omega,\mathcal{F}) and RP\mathcal{R}^{P} be a law invariant risk measure associated with P∈PP\in\mathcal{P}. For a given P∈PP\in\mathcal{P} and a set C∈FC\in\mathcal{F} such that P(C)>0P(C)>0, consider the induced measure P\mathbb{P} supported on CC, i.e., P(A)=P(A)/P(C)\mathbb{P}(A)=P(A)/P(C) for any measurable set A⊂CA\subset C. Consider the corresponding risk measure RP\mathcal{R}^{\mathbb{P}}. Then for any measurable Z,Z′:Ω→RZ,Z':\Omega\to\mathbb{R} such that Z(ω)=Z′(ω)Z(\omega)=Z'(\omega) for all ω∈C\omega\in C, we have that ZZ and Z′Z' are distributionally equivalent with respect to P\mathbb{P} and hence RP(Z)=RP(Z′)\mathcal{R}^{\mathbb{P}}(Z)=\mathcal{R}^{\mathbb{P}}(Z').

This motivates the following construction. Let πˉ∈Π\bar{\pi}\in\Pi be a given policy (for example, πˉ\bar{\pi} can be an optimal policy with respect to Rπ\mathcal{R}^{\pi} for the total cost). Assume that there is no policy uniformly better than πˉ\bar{\pi}, that is:

Assumption 3.1. For every policy {πt}\{\pi_t\} and all st∈Sts_t\in S_t there is a set Ct=Ct(πt,st)⊂St+1C_t=C_t(\pi_t,s_t)\subset S_{t+1} of positive measure with respect to the reference probability measure Ptst,πt(st)P_t^{s_t,\pi_t(s_t)} on St+1S_{t+1}, such that Zπt(st+1)≥Zˉ(st+1)Z^{\pi_t}(s_{t+1})\geq\bar{Z}(s_{t+1}) for a.e. st+1∈Cts_{t+1}\in C_t, where

Zπt(st+1):=ct(st,πt(st),st+1)andZˉ(st+1):=ct(st,πˉt(st),st+1).Z^{\pi_t}(s_{t+1}) := c_t(s_t,\pi_t(s_t),s_{t+1}) \quad\text{and}\quad\bar{Z}(s_{t+1}) := c_t(s_t,\bar{\pi}_t(s_t),s_{t+1}).

Consider the probability measure PtP_t on CtC_t induced by Ptst,πt(st)P_t^{s_t,\pi_t(s_t)}. That is, for a measurable set A⊂CtA\subset C_t,

Pt(A)=Ptst,πt(st)(A)Ptst,πt(st)(Ct).P_t(A)=\frac{P_t^{s_t,\pi_t(s_t)}(A)}{P_t^{s_t,\pi_t(s_t)}(C_t)}.

Note that the measure Pt=Ptst,πt(st)P_t=P_t^{s_t,\pi_t(s_t)} is supported on the set Ct=Ct(πt,st)C_t=C_t(\pi_t,s_t) and depends on πt\pi_t and sts_t. Consider the corresponding conditional risk measures RtPtst,πt(st)\mathcal{R}_t^{P_t^{s_t,\pi_t(s_t)}}. In the following, we show that under Assumption 3.1, policy πˉ\bar{\pi} is time consistent with respect to the nested formulation determined by risk measures constructed from RtPtst,πt(st)\mathcal{R}_t^{P_t^{s_t,\pi_t(s_t)}}.

Specifically, we construct risk measures R~tst\widetilde{\mathcal{R}}_t^{s_t} such that πˉ\bar{\pi} is time consistent with respect to the corresponding nested formulation. Throughout the rest of our discussion, we assume that R\mathcal{R} also satisfies axiom (A3), and hence RP(0)=0\mathcal{R}^{P}(0)=0 for any probability measure PP. With the measures Ptst,πt(st)P_t^{s_t,\pi_t(s_t)} and the function Zˉ\bar{Z} defined above, we define the risk measure R~tst\widetilde{\mathcal{R}}_t^{s_t} of Z:St+1→RZ:S_{t+1}\to\mathbb{R} as

R~tst(Z):=sup⁡π∈ΠRtPtst,πt(st)(Z−Zˉ).(11)\widetilde{\mathcal{R}}_t^{s_t}(Z):=\sup_{\pi\in\Pi}\mathcal{R}_t^{P_t^{s_t,\pi_t(s_t)}}(Z-\bar{Z}). \tag*{(11)}

For any π∈Π\pi\in\Pi, we can write the value function of π\pi under the nested formulation determined by R~tst\widetilde{\mathcal{R}}_{t}^{s_t} recursively as VT+1π(⋅)≡0V_{T+1}^{\pi}(\cdot) \equiv0 and

Vtπ(st)=R~tst(ct(st,πt(st),⋅)+Vt+1π(⋅)),st∈St, t=T,…,1.(12)V_{t}^{\pi}(s_t)=\widetilde{\mathcal{R}}_{t}^{s_t}\left(c_t(s_t,\pi_t(s_t),\cdot)+V_{t+1}^{\pi}(\cdot)\right),\quad s_t\in S_t,\ t=T,\ldots,1. \tag*{(12)}

Proposition 3.1. Suppose Assumption 3.1 holds. Then for any π∈Π\pi\in\Pi, t=1,…,Tt=1,\ldots,T and st∈Sts_t\in S_t, we have

Vtπˉ(st)≤Vtπ(st).V_t^{\bar{\pi}}(s_t)\leq V_t^\pi(s_t).

That is, πˉ\bar{\pi} is time consistent with respect to the nested formulation determined by R~tst\widetilde{\mathcal{R}}_{t}^{s_t}.

Proof. Since VT+1πˉ≡0V_{T+1}^{\bar{\pi}}\equiv0 and R~tst(Zˉ)=0\widetilde{\mathcal{R}}_{t}^{s_t}(\bar{Z})=0, we obtain from (12) with π=πˉ\pi=\bar{\pi} that Vtπˉ≡0V_t^{\bar{\pi}}\equiv0 for t=T,…,1t=T,\ldots,1. Clearly the desired claim holds at t=T+1t=T+1. Suppose the claim holds at t+1t+1. Then we have

Vtπˉ(st)=RtPst,πt(st)(0)≤(a)RtPst,πt(st)(Zπt−Zˉ)≤(b)R~tst(Zπt)=(c)R~tst(ct(st,πt(st),⋅)+Vt+1πˉ(⋅))≤(d)R~tst(ct(st,πt(st),⋅)+Vt+1π(⋅))=Vtπ(st),\begin{aligned} V_t^{\bar{\pi}}(s_t) &=\mathcal{R}_{t}^{P^{s_t,\pi_t(s_t)}}(0) \overset{(a)}{\leq}\mathcal{R}_{t}^{P^{s_t,\pi_t(s_t)}}(Z^{\pi_t}-\bar{Z})\\ &\overset{(b)}{\leq}\widetilde{\mathcal{R}}_{t}^{s_t}(Z^{\pi_t})\\ &\overset{(c)}{=}\widetilde{\mathcal{R}}_{t}^{s_t}\left(c_t(s_t,\pi_t(s_t),\cdot)+V_{t+1}^{\bar{\pi}}(\cdot)\right)\\ &\overset{(d)}{\leq}\widetilde{\mathcal{R}}_{t}^{s_t}\left(c_t(s_t,\pi_t(s_t),\cdot)+V_{t+1}^{\pi}(\cdot)\right)=V_t^\pi(s_t), \end{aligned}

where (a) follows from Assumption 3.1 and (A1), (b) follows from (11), (c) follows from Vt+1πˉ≡0V_{t+1}^{\bar{\pi}}\equiv0, and (d) follows from the induction hypothesis. Hence the claim holds at tt. The proof is then completed.

It could be noted that the construction of R~tst\widetilde{\mathcal{R}}_{t}^{s_t} does not rely on the policies being Markovian. Consider history dependent policies πt(ht)\pi_t(h_t) with ht=(s1,…,st)h_t=(s_1,\ldots,s_t), and suppose that for every policy π\pi and every history hth_t there is a set Ct=Ct(πt,ht)⊂St+1C_t=C_t(\pi_t,h_t)\subset S_{t+1} of positive measure with respect to Ptst,πt(ht)P_t^{s_t,\pi_t(h_t)} such that ct(st,πˉt(ht),st+1)≤ct(st,πt(ht),st+1)c_t(s_t,\bar{\pi}_t(h_t),s_{t+1})\leq c_t(s_t,\pi_t(h_t),s_{t+1}) for a.e. st+1∈Cts_{t+1}\in C_t. Then the conclusion of Proposition 3.1 holds for history dependent policies, by following the same construction with sts_t replaced by hth_t. The same applies to the SOC setting discussed in § 3.2 below.

Stochastic Optimal Control

Let us now consider the stochastic optimal control (SOC) problem

min⁡π∈ΠET[∑t=1Tct(xt,ut,ξt)].\min_{\pi\in\Pi}\mathbb{E}^{\mathsf{T}}\left[\sum_{t=1}^{T}c_t(x_t,u_t,\xi_t)\right].

where the state evolves according to xt+1=Φt(xt,ut,ξt)x_{t+1}=\Phi_t(x_t,u_t,\xi_t), t=1,…,Tt=1,\ldots,T, from a deterministic initial state x1x_1, and the minimization is over policies that determine the controls ut∈Utu_t\in U_t. Here Xt⊂RntX_t\subset\mathbb{R}^{n_t}, Ut⊂RmtU_t\subset\mathbb{R}^{m_t} and Ξt⊂Rdt\Xi_t\subset\mathbb{R}^{d_t} are nonempty closed sets, ξt∈Ξt\xi_t\in\Xi_t is the random data at stage tt, Ft\mathcal{F}_t is the Borel sigma algebra of Ξt\Xi_t, and both the cost function ct:Xt×Ut×Ξt→Rc_t:X_t\times U_t\times\Xi_t\to\mathbb{R} and the mapping Φt:Xt×Ut×Ξt→Xt+1\Phi_t:X_t\times U_t\times\Xi_t\to X_{t+1} are continuous.

Throughout the rest of our discussion, we assume that the distribution of (ξ1,…,ξT)(\xi_1,\ldots,\xi_T) does not depend on states and controls, and is given by the product P=P1×⋯×PTP=P_1\times\cdots\times P_T of the marginal distributions PtP_t of ξt\xi_t on (Ξt,Ft)(\Xi_t,\mathcal{F}_t). For each PtP_t, let us consider a law invariant risk measure $\mathcal{R}^{P_t} satisfying axioms (A1)-(A3). The value functions of the risk averse counterpart of the SOC problem then depend only on the state, and satisfy the dynamic programming equations VT+1(⋅)≡0V_{T+1}(\cdot) \equiv0 and

Vt(xt)=inf⁡ut∈UtRPt[ct(xt,ut,ξt)+Vt+1(Φt(xt,ut,ξt))],t=T,…,1,V_t(x_t)=\inf_{u_t\in\mathcal{U}_t}\mathcal{R}^{P_t}\left[c_t(x_t,u_t,\xi_t)+V_{t+1}\left(\Phi_t(x_t,u_t,\xi_t)\right)\right],\quad t=T,\ldots,1,

where ξt∼Pt\xi_t\sim P_t. In view of this, we restrict our attention to the set Π\Pi of Markovian policies ut=πt(xt)u_t=\pi_t(x_t), t=1,…,Tt=1,\ldots,T.

Similar to § 3.1, for a given policy πˉ∈Π\bar{\pi}\in\Pi, we proceed to construct risk measures R~txt\widetilde{\mathcal{R}}_t^{x_t} such that πˉ\bar{\pi} is time consistent with respect to the corresponding nested formulation.

Assumption 3.2. For t=T,…,1t=T,\ldots,1, any policy π∈Π\pi\in\Pi and any xt∈Xtx_t\in\mathcal{X}_t, there is a set Ct=Ct(πt,xt)⊂ΞtC_t=C_t(\pi_t,x_t)\subset\Xi_t such that Pt(Ct)>0P_t(C_t)>0 and Zˉ(ξt)≤Zπt(ξt)\bar{Z}(\xi_t)\le Z^{\pi_t}(\xi_t) for a.e. ξt∈Ct\xi_t\in C_t, where

Zπt(ξt):=ct(xt,πt(xt),ξt)Z^{\pi_t}(\xi_t):=c_t(x_t,\pi_t(x_t),\xi_t) and Zˉ(ξt):=ct(xt,πˉt(xt),ξt)\bar{Z}(\xi_t):=c_t(x_t,\bar{\pi}_t(x_t),\xi_t).

For π∈Π\pi\in\Pi and xt∈Xtx_t\in\mathcal{X}_t, consider the probability measure PCtP_{C_t} on Ct=Ct(πt,xt)C_t=C_t(\pi_t,x_t) induced by PtP_t, that is, PCt(A)=Pt(A)/Pt(Ct)P_{C_t}(A)=P_t(A)/P_t(C_t) for measurable A⊂CtA\subset C_t. We then define the risk measure R~txt\widetilde{\mathcal{R}}_t^{x_t} of Z:Ξt→RZ:\Xi_t\to\mathbb{R} as

R~txt(Z):=sup⁡π∈ΠRPCt(πt,xt)(Z−Zˉ).(13)\widetilde{\mathcal{R}}_t^{x_t}(Z):=\sup_{\pi\in\Pi}\mathcal{R}^{P_{C_t(\pi_t,x_t)}}(Z-\bar{Z}). \tag*{(13)}

For any π∈Π\pi\in\Pi, we can write the value function of π\pi under the nested formulation determined by R~txt\widetilde{\mathcal{R}}_t^{x_t} recursively as VT+1π(⋅)≡0V_{T+1}^{\pi}(\cdot)\equiv0 and

Vtπ(xt)=R~txt(ct(xt,πt(xt),⋅)+Vt+1π(Φt(xt,πt(xt),⋅))),xt∈Xt, t=T,…,1.(14)V_t^\pi(x_t)=\widetilde{\mathcal{R}}_t^{x_t}\left(c_t(x_t,\pi_t(x_t),\cdot)+V_{t+1}^\pi\left(\Phi_t(x_t,\pi_t(x_t),\cdot)\right)\right),\quad x_t\in\mathcal{X}_t,\ t=T,\ldots,1. \tag*{(14)}

Proposition 3.2. Suppose Assumption 3.2 holds. Then for any π∈Π\pi\in\Pi, t=1,…,Tt=1,\ldots,T and xt∈Xtx_t\in\mathcal{X}_t, we have

Vtπˉ(xt)≤Vtπ(xt).V_t^{\bar{\pi}}(x_t)\le V_t^\pi(x_t).

That is, πˉ\bar{\pi} is time consistent with respect to the nested formulation determined by R~txt\widetilde{\mathcal{R}}_t^{x_t}.

Proof. Similar to the proof of Proposition 3.1, we obtain from (14) with π=πˉ\pi=\bar{\pi} and (13) that Vtπˉ≡0V_t^{\bar{\pi}}\equiv0 for t=T,…,1t=T,\ldots,1. Clearly the desired claim holds at t=T+1t=T+1. Suppose the claim holds at t+1t+1. With Ct=Ct(πt,xt)C_t=C_t(\pi_t,x_t), we have

Vtπˉ(xt)=RPCt(0)≤(a)RPCt(Zπt−Zˉ)≤(b)R~txt(Zπt)=(c)R~txt(ct(xt,πt(xt),⋅)+Vt+1πˉ(Φt(xt,πt(xt),⋅)))≤(d)R~txt(ct(xt,πt(xt),⋅)+Vt+1π(Φt(xt,πt(xt),⋅)))=Vtπ(xt),\begin{aligned} V_t^{\bar{\pi}}(x_t)&=\mathcal{R}^{P_{C_t}}(0)\mathrel{\overset{(a)}{\le}}\mathcal{R}^{P_{C_t}}(Z^{\pi_t}-\bar{Z})\\ &\mathrel{\overset{(b)}{\le}}\widetilde{\mathcal{R}}_t^{x_t}(Z^{\pi_t})\\ &\mathrel{\overset{(c)}{=}}\widetilde{\mathcal{R}}_t^{x_t}\left(c_t(x_t,\pi_t(x_t),\cdot)+V_{t+1}^{\bar{\pi}}\left(\Phi_t(x_t,\pi_t(x_t),\cdot)\right)\right)\\ &\mathrel{\overset{(d)}{\le}}\widetilde{\mathcal{R}}_t^{x_t}\left(c_t(x_t,\pi_t(x_t),\cdot)+V_{t+1}^{\pi}\left(\Phi_t(x_t,\pi_t(x_t),\cdot)\right)\right)=V_t^\pi(x_t), \end{aligned}

where (a) follows from Assumption 3.2 and (A1), (b) follows from (13), (c) follows from Vt+1πˉ≡0V_{t+1}^{\bar{\pi}}\equiv0, and (d) follows from the induction hypothesis with xt+1=Φt(xt,πt(xt),ξt)x_{t+1}=\Phi_t(x_t,\pi_t(x_t),\xi_t) for any ξt∈Ξt\xi_t\in\Xi_t. Hence the claim holds at tt. The proof is then completed.

Concluding remarks

In this manuscript, we discuss risk measures which are not necessarily convex. We show that concavity in the probability measure, which holds for a large class of coherent risk measures, makes the risk sensitive to an arbitrarily small contamination of the distribution, and hence such risk measures are not robust from the statistical point of view. Our findings suggest that when statistical robustness is of concern, it is natural to consider risk measures without the convexity axiom, such as the Value-at-Risk.

For any law invariant risk measure satisfying axioms (A1)-(A3) and any policy such that no policy is uniformly better, we construct conditional risk measures with respect to which this policy is time consistent, in both the MDP and SOC frameworks. It should be noted that the constructed risk measures depend on the given policy and are not law invariant. The important question of how to solve the resulting non-convex problems numerically is beyond the scope of this manuscript, and can be a topic of future research.

Acknowledgments We used Claude (Anthropic) to produce initial drafts of selected paragraphs, based on the main results provided by the authors, and to polish the writing and correct typos. All content was subsequently revised and verified by the authors.

References

  1. [1]C. Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 26:1505–1518, 2002.DOI
  2. [2]P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk. Mathematical Finance, 9:203–228, 1999.
  3. [3]V. Guigues, A. Shapiro, and Y. Cheng. Risk-averse stochastic optimal control: An efficiently computable statistical upper bound. Operations Research Letters, 51:393 – 400, 2023.arxiv.org/abs/2112.09757
  4. [4]P. Huber. Robust statistics. John Wiley & Sons, 1981.
  5. [5]M. Kupper and W. Schachermayer. Representation results for law invariant time consistent functions. Mathematics and Financial Economics, pages 189–210, 2009.DOI
  6. [6]A. Pichler, R. P. Liu, and A. Shapiro. Risk-averse stochastic programming: time consistency and optimal stopping. Operations Research, 70:2439–2455, 2022.arxiv.org/abs/1808.10807
  7. [7]R.T Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distributions. J. of Banking and Finance, 26(7):1443–1471, 2002.DOI
  8. [8]P.J. Rousseeuw and A.M. Leroy. Robust Regression and Outlier Detection. John Wiley & Sons, 1987.
  9. [9]A. Ruszczyński. Risk-averse dynamic programming for Markov decision processes. Mathematical Programming, 125:235–261, 2010.DOI
  10. [10]A. Ruszczyński and A. Shapiro. Conditional risk mappings. Mathematics of Operations Research, 31:544–561, 2006.
  11. [11]A. Shapiro. Time consistency of dynamic risk measures. Operations Research Letters, 40:436–439, 2012.DOI
  12. [12]A. Shapiro, D. Dentcheva, and A. Ruszczyński. Lectures on Stochastic Programming: Modeling and Theory. SIAM, Philadelphia, third edition, 2021.
  13. [13]A. Shapiro and Y. Li. Risk-averse formulations of Stochastic Optimal Control and Markov Decision Processes. arXiv preprint arXiv:2505.16651, 2025.arxiv.org/abs/2505.16651
  14. [14]A. Shapiro and K. Ugurlu. Decomposability and time consistency of risk averse multistage programs. Operations Research Letters, 44:663–665, 2016.arxiv.org/abs/1806.01497
  15. [15]A. Shapiro and L. Xin. Time inconsistency of optimal policies of distributionally robust inventory models. Operations Research, 68:1576–1584, 2020.DOI
  16. [16]Ionescu Tulcea. Mesures dans les espaces produits. Atti Accad. Naz. Lincei Rend, 7:208 – 211, 1949.

Paper details

vlnc:W_qgsmcw85
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents