Introduction
Artificial neural networks routinely outperform humans on narrow, well-specified tasks – from image classification to language modeling – yet the internal mechanisms behind their predictions remain largely opaque. This “black-box” character, present since Rosenblatt [37 ] ’s perceptron, now raises acute concerns of trust, reliability, and accountability wherever these models are deployed in high-stakes decisions – medical diagnosis, autonomous systems, financial decision-making – and has motivated a decade of work in eXplainable Artificial Intelligence (XAI).
Two broad families of attribution methods have emerged to answer the question of why a network produces a given output, each capturing a complementary facet of its behavior. Relevance propagation , epitomized by Layer-wise Relevance Propagation (LRP) (Bach et al., [6 ] ), decomposes the output score as a sum of contributions from individual input features by propagating a conservation signal backward through the network; it tells how the output is shared among the inputs, not how it would respond to a perturbation of them. Gradient-based attribution , such as Gradient×Input [44 , 7 ] , quantifies the sensitivity of the output to infinitesimal perturbations; it tells how the output would respond, but ignores the global flow of information through the network. Until now, LRP has been justified by local conservation and by deep Taylor decomposition [30 ] , which derives its rules from local expansions, but not as the solution of a stated optimization problem. In earlier, more applied work [33 ] , we proposed Relative LRP (R-LRP), a rule free of hyperparameters that avoids the divisions by small values of the classical rules by satisfying the conservation law only up to a multiplicative factor, and compared it with the standard LRP variants on image classification with convolutional, VGG and ResNet networks; that experience is what calls for the principled foundation developed here.
This paper replaces the conservation heuristic by a variational principle, which also relates the two families: relevance propagation is exactly the entropic projection of the backward transition law of the forward activation flow onto the set of divergence-free relevance flows on the active computational graph, the unique such flow with zero relative entropy to that law; for a non-negative reference, this relative entropy is the one of two Markov measures on the backward paths of the network. This principle recovers the classical L R P z + \mathrm{LRP}_{z+} LRP z + and L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β rules as special cases, L R P 0 \mathrm{LRP}_{0} LRP 0 through a signed extension of the same relative entropy, and gathers the three into one rule whose weights may vary from node to node, for instance with the activity of the node, in the spirit of Hebb’s principle [22 ] ; L R P 0 \mathrm{LRP}_{0} LRP 0 is Gradient×Input on networks of ReLU and max-pooling nodes. This entropic optimal-transport formulation of relevance is our main contribution: the rules become the choice of a reference, conservation becomes a constraint rather than a heuristic, and the tools of entropic transport – Sinkhorn iterations, Schrödinger bridges – become available for attribution. Carrying this program out rigorously demands a foundation broader than the smooth or piecewise-smooth networks classical LRP was built for: architectures such as hard-routed Mixture-of-Experts [42 , 17 ] can be outright discontinuous , so neither the classical nor the Clarke generalized gradient is everywhere defined, and a gradient-based attribution applied to them naively rests on undefined quantities exactly where the routing decision changes.
The paper is organized as follows. § 2 lays the foundation: a new, measure-theoretic formalization of feed-forward networks as directed acyclic graphs of Filippov operators, with a Clarke–Filippov generalized-gradient calculus yielding well-defined sets of admissible values wherever the network is non-differentiable or discontinuous; recast as a discrete-time optimal control problem in the spirit of LeCun [27 ] , the same formalism gives backpropagation itself a variational characterization, as the adjoint equation of a Lagrangian relaxation. § 3 builds the central variational principle on this foundation, recovering L R P z + \mathrm{LRP}_{z+} LRP z + and L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β as the unique solutions of relative-entropy minimization problems between backward transition laws, L R P 0 \mathrm{LRP}_{0} LRP 0 included, its reference being the signed forward contributions themselves. The three are instances of L R P N α , β \mathrm{LRPN}_{\alpha,\beta} LRPN α , β , a rule that splits the relevance of each node v v v between its excitatory and its inhibitory contributions with weights α v \alpha_v α v and β v = 1 − α v \beta_v = 1 - \alpha_v β v = 1 − α v that may change from node to node. With x ˉ v \bar{x}_v x ˉ v the value of v v v divided by the largest value of its layer, L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRPN}_{\alpha(\bar{x}),\beta(1-\bar{x})} LRPN α ( x ˉ ) , β ( 1 − x ˉ ) takes α v = x ˉ v \alpha_v = \bar{x}_v α v = x ˉ v and β v = 1 − x ˉ v \beta_v = 1 - \bar{x}_v β v = 1 − x ˉ v : in the spirit of Hebb’s principle, a more active node then sends more of its relevance to its excitatory contributions. On networks of ReLU and max-pooling nodes, each of these rules is an input times a modified backward pass – Gradient×Input for L R P 0 \mathrm{LRP}_{0} LRP 0 , which jumps across a kink where L R P z + \mathrm{LRP}_{z+} LRP z + stays continuous, whereas every rule may jump where the winner of a max-pooling node changes. Taking instead the relative entropy edge by edge, as in entropically regularized flows, § 3 introduces Sinkhorn Relevance Propagation (SRP), the conservative flow closest to the whole forward activation flow, computed by a Sinkhorn-type iteration, and closes with applications to two image classifiers, VGG-16 and ConvNeXt-Tiny (Appendix § 9 ). On VGG-16, the rules conserve the output to round-off and are told apart by the sign of their maps, the share of the output they leave to the bias nodes, their faithfulness, their localization and their dependence on the class, SRP following the edges and textures of the lower layers, and a perturbation of standard deviation 10 − 5 10^{-5} 1 0 − 5 moves the L R P 0 \mathrm{LRP}_{0} LRP 0 map by 1.5%, against 0.06 to 0.11% for the other rules, consistently with Corollary 3.5; on ConvNeXt-Tiny, whose hidden values take both signs, L R P N α , β \mathrm{LRPN}_{\alpha,\beta} LRPN α , β conserves the output and L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRPN}_{\alpha(\bar{x}),\beta(1-\bar{x})} LRPN α ( x ˉ ) , β ( 1 − x ˉ ) has the highest AOPC of the rules tested, whereas L R P z + \mathrm{LRP}_{z+} LRP z + loses part of the output and L R P 0 \mathrm{LRP}_{0} LRP 0 is no longer Gradient×Input. § 4 aggregates the relevance flow of a non-negative reference over a region of the input: it tells which hidden units carry the evidence of that region, with exact accounting, the flow conditioned on the region being a Schrödinger bridge whose potentials follow from one forward pass; on 200 annotated photographs of cats and dogs and VGG-16, with LRPz + _{z^+} z + and LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N_{\alpha(\bar{x}),\beta(1-\bar{x})}} N α ( x ˉ ) , β ( 1 − x ˉ ) , the head receives about 2.5 times as much relevance per pixel as the body, through largely different channels. § 5 concludes and discusses perspectives.
Feed-forward Filippov neural networks
This section lays the foundation of the paper. After the setting and the architectures that motivate it (§ 2.1 ), it introduces the Filippov regularization of a switching operator (§ 2.2 ), describes a network as a directed acyclic graph of such operators (§ 2.3 ), defines its forward propagation, its admissible branch selections, its active graph and the Clarke–Filippov calculus of its output (§ 2.4 ), and casts backpropagation as the adjoint equation of a discrete-time optimal control problem (§ 2.5 ).
Setting
The typical setting for supervised learning is to find an approximation of an unknown function f f f , given a training data set of its point values: { x ( i ) , f ( x ( i ) ) } i = 1 , … , m \{x^{(i)}, f(x^{(i)})\}_{i=1,\ldots,m} { x ( i ) , f ( x ( i ) ) } i = 1 , … , m where x ( i ) ∈ R d x^{(i)} \in\mathbb{R}^{d} x ( i ) ∈ R d and f ( x ( i ) ) ∈ R p f(x^{(i)}) \in\mathbb{R}^{p} f ( x ( i ) ) ∈ R p . For example, in classification problems for images, d d d is the number of pixels × \times × the number of channels in the images, typically somewhere in the range of 10 3 10^{3} 1 0 3 to 10 6 10^{6} 1 0 6 , and for videos it is even higher. The learning problem is then to numerically produce from this data a function f ^ \hat{f} f ^ that is in some sense a good predictor of f f f on new unseen draws x ∈ R d x \in\mathbb{R}^{d} x ∈ R d . In the classical formulation the input space is R d \mathbb{R}^{d} R d , where each coordinate corresponds to a fixed feature (e.g. a pixel in an image).
Large language models such as Claude [2 ] , Mistral [26 , 25 ] , GPT [11 , 34 ] , LLaMA [46 b] or DeepSeek [17 , 15 ] , are trained on collections of tokenized documents rather than on fixed-dimensional feature vectors. A training sample is a sequence of tokens x = ( t 1 , … , t n ) ∈ T n x = (t_{1}, \ldots, t_{n}) \in\mathcal{T}^{n} x = ( t 1 , … , t n ) ∈ T n , where T \mathcal{T} T is a finite vocabulary and n n n may vary from one example to another. The learning task consists in approximating the conditional distribution p ( t n + 1 ∣ t 1 , … , t n ) p(t_{n+1} \mid t_{1}, \ldots, t_{n}) p ( t n + 1 ∣ t 1 , … , t n ) , that is, predicting the next token given the previous context. After embedding, each token is represented by a vector in a latent space R d \mathbb{R}^{d} R d and the input to the network becomes a sequence X = ( x 1 , … , x n ) ∈ ( R d ) n X = (x_{1}, \ldots, x_{n}) \in(\mathbb{R}^{d})^{n} X = ( x 1 , … , x n ) ∈ ( R d ) n . Modern foundation models are typically based on transformer architectures combining attention layers, feed-forward networks and normalization operators. Several recent models additionally employ Mixture-of-Experts (MoE) [42 ] layers in order to increase the number of parameters without increasing the computational cost proportionally. In a MoE block, a routing network computes a score vector. A subset of experts is then selected according to the largest routing scores. In the hard-routing setting, the selection is typically performed through a top-k k k operator or an argmax rule.
Unlike attention and feed-forward blocks, which remain smooth or at worst piecewise-smooth (as with ReLU), the hard-routing level selects a different expert map across routing boundaries, where the network map itself, not merely one of its derivatives, may jump, unless the experts agree there. Neither the classical gradient nor the Clarke generalized gradient is then defined at such points, so that a gradient-based explanation applied naively to a hard-routed architecture rests on undefined quantities exactly where the routing decision changes, unless a selection convention is fixed. We therefore introduce, in place of ordinary differentiability, a Filippov set-valued regularization that remains well defined across these discontinuities, coincides with the classical gradient at regular points and contains the Clarke generalized gradient where the network is continuous.
Filippov regularization
Let us introduce piecewise-C 1 C^{1} C 1 switching operator.
Definition 2.1. Let Ω ⊆ R n \Omega\subseteq\mathbb{R}^{n} Ω ⊆ R n be an open set. A map Φ : Ω ∖ Σ → R m \Phi: \Omega\setminus\Sigma\to\mathbb{R}^{m} Φ : Ω ∖ Σ → R m (m ∈ N ∗ m \in\mathbb{N}^{*} m ∈ N ∗ ) is called a piecewise-C 1 C^{1} C 1 switching operator on Ω \Omega Ω , with switching region Σ \Sigma Σ , if there exist:
K K K , such that, writing
Σ : = Ω ∖ ⋃ k ∈ K Ω ( k ) , \Sigma:= \Omega\setminus\bigcup_{k\in K}\Omega^{(k)}, Σ := Ω ∖ k ∈ K ⋃ Ω ( k ) ,
the complement satisfies μ ( Σ ) = 0 \mu(\Sigma)=0 μ ( Σ ) = 0 where μ \mu μ denotes the Lebesgue measure – an explicit requirement on the family, not automatic from local finiteness and disjointness alone;
for every k ∈ K k\in K k ∈ K , an open set U ( k ) ⊆ R n U^{(k)}\subseteq\mathbb{R}^{n} U ( k ) ⊆ R n with Ω ( k ) ∩ Ω ⊆ U ( k ) \Omega^{(k)}\cap\Omega\subseteq U^{(k)} Ω ( k ) ∩ Ω ⊆ U ( k ) and a C 1 C^{1} C 1 map Φ ( k ) : U ( k ) → R m \Phi^{(k)}:U^{(k)}\to\mathbb{R}^{m} Φ ( k ) : U ( k ) → R m , called the branch of Φ \Phi Φ on Ω ( k ) \Omega^{(k)} Ω ( k ) , such that Φ = Φ ( k ) \Phi=\Phi^{(k)} Φ = Φ ( k ) on Ω ( k ) \Omega^{(k)} Ω ( k ) ; we write Φ ( k ) \Phi^{(k)} Φ ( k ) again for the restriction of this map to Ω ( k ) ∩ Ω \Omega^{(k)}\cap\Omega Ω ( k ) ∩ Ω . In particular Φ ( k ) \Phi^{(k)} Φ ( k ) is C 1 C^{1} C 1 , with continuous Jacobian D Φ ( k ) ∈ R m × n D\Phi^{(k)}\in\mathbb{R}^{m\times n} D Φ ( k ) ∈ R m × n , on a full open neighborhood of every point of Ω ( k ) ∩ Ω \Omega^{(k)}\cap\Omega Ω ( k ) ∩ Ω , including points of the switching region Σ \Sigma Σ itself, rather than merely admitting a continuous extension from inside Ω ( k ) \Omega^{(k)} Ω ( k ) .
For x ∈ Ω x\in\Omega x ∈ Ω , the active index set of Φ \Phi Φ at x x x is J Φ ( x ) : = { k ∈ K : x ∈ Ω ( k ) ‾ } J_{\Phi}(x):=\{k\in K:x\in\overline{\Omega^{(k)}}\} J Φ ( x ) := { k ∈ K : x ∈ Ω ( k ) } . We write Φ ∈ S 1 ( Ω , R m ) \Phi\in S^{1}(\Omega,\mathbb{R}^{m}) Φ ∈ S 1 ( Ω , R m ) for the class of all such operators, and simply Φ ∈ S 1 ( Ω ) \Phi\in S^{1}(\Omega) Φ ∈ S 1 ( Ω ) when m = 1 m=1 m = 1 .
The set Σ \Sigma Σ is automatically closed in Ω \Omega Ω . Moreover, we will show that J Φ ( x ) J_{\Phi}(x) J Φ ( x ) is finite and nonempty for every x ∈ Ω x\in\Omega x ∈ Ω . Since Φ ( k ) \Phi^{(k)} Φ ( k ) is C 1 C^{1} C 1 on the open neighborhood U ( k ) U^{(k)} U ( k ) , its restriction to Ω ( k ) ‾ ∩ Ω \overline{\Omega^{(k)}}\cap\Omega Ω ( k ) ∩ Ω automatically agrees with the limit taken from inside Ω ( k ) \Omega^{(k)} Ω ( k ) ,
Φ ( k ) ( z ) = lim ζ → z ζ ∈ Ω ( k ) Φ ( ζ ) , for every z ∈ Ω ( k ) ‾ ∩ Ω , \Phi^{(k)}(z)=\lim_{\substack{\zeta\to z\\\zeta\in\Omega^{(k)}}}\Phi(\zeta),\qquad\text{for every }z\in\overline{\Omega^{(k)}}\cap\Omega, Φ ( k ) ( z ) = ζ → z ζ ∈ Ω ( k ) lim Φ ( ζ ) , for every z ∈ Ω ( k ) ∩ Ω ,
so the branch is unambiguously determined by Φ \Phi Φ on Ω ( k ) \Omega^{(k)} Ω ( k ) alone; the open-neighborhood extension is what additionally guarantees that Φ ( k ) \Phi^{(k)} Φ ( k ) stays differentiable, not just continuous, arbitrarily close to a switching point.
The partition ( Ω ( k ) ) k ∈ K (\Omega^{(k)})_{k\in K} ( Ω ( k ) ) k ∈ K is motivated by the fact that many operators used in modern neural architectures are naturally described by different formulas on different regions of their input space; any discontinuity or nonsmooth behavior is then confined to the switching region Σ \Sigma Σ , contained in the union of the region boundaries. A first example is the max-pooling operator Φ ( z 1 , … , z n ) = max 1 ≤ i ≤ n z i \Phi(z_{1},\ldots,z_{n})=\max_{1\le i\le n}z_{i} Φ ( z 1 , … , z n ) = max 1 ≤ i ≤ n z i on Ω = R n \Omega=\mathbb{R}^{n} Ω = R n . For each k ∈ K = { 1 , … , n } k\in K=\{1,\ldots,n\} k ∈ K = { 1 , … , n } , let Ω ( k ) = { z ∈ R n : z k > z j for all j ≠ k } \Omega^{(k)}=\{z\in\mathbb{R}^{n}:z_{k}>z_{j}\text{ for all }j\ne k\} Ω ( k ) = { z ∈ R n : z k > z j for all j = k } , the open set where the k k k -th coordinate is the unique maximum. On Ω ( k ) \Omega^{(k)} Ω ( k ) , the operator reduces to the linear map Φ ( k ) ( z ) = z k \Phi^{(k)}(z)=z_{k} Φ ( k ) ( z ) = z k , and the switching region Σ = { z : z i = z j = max ℓ z ℓ for some i ≠ j } \Sigma=\{z:z_{i}=z_{j}=\max_{\ell}z_{\ell}\text{ for some }i\ne j\} Σ = { z : z i = z j = max ℓ z ℓ for some i = j } is the set of ties between maximal coordinates – a closed subset of finitely many hyperplanes, hence a null set. Max-pooling is thus piecewise linear and continuous, and it is not differentiable exactly on Σ \Sigma Σ , where several branches Φ ( k ) ( z ) = z k \Phi^{(k)}(z)=z_{k} Φ ( k ) ( z ) = z k coincide in value but have different gradients e k e_{k} e k . A second example is a hard-routing mixture-of-experts block ([42 , 19 , 15 , 25 ] ). Let g 1 , … , g M g_{1},\ldots,g_{M} g 1 , … , g M be C 1 C^{1} C 1 routing score functions and E 1 , … , E M E_{1},\ldots,E_{M} E 1 , … , E M be C 1 C^{1} C 1 expert maps. The routing rule selects the expert r ( z ) ∈ argmax 1 ≤ j ≤ M g j ( z ) r(z)\in\operatorname*{argmax}_{1\le j\le M}g_{j}(z) r ( z ) ∈ argmax 1 ≤ j ≤ M g j ( z ) , and the local operator is Φ ( z ) = E r ( z ) ( z ) \Phi(z)=E_{r(z)}(z) Φ ( z ) = E r ( z ) ( z ) . For each expert j j j , the routing region Ω ( j ) = { z : g j ( z ) > g ℓ ( z ) for all ℓ ≠ j } \Omega^{(j)}=\{z:g_{j}(z)>g_{\ell}(z)\text{ for all }\ell\ne j\} Ω ( j ) = { z : g j ( z ) > g ℓ ( z ) for all ℓ = j } is open, and on it the operator is the branch Φ ( j ) = E j \Phi^{(j)}=E_{j} Φ ( j ) = E j . The switching region is the set of routing ties g j ( z ) = g ℓ ( z ) = max i g i ( z ) g_{j}(z)=g_{\ell}(z)=\max_{i}g_{i}(z) g j ( z ) = g ℓ ( z ) = max i g i ( z ) , j ≠ ℓ j\ne\ell j = ℓ ; it is a null set as soon as no difference g j − g ℓ g_{j}-g_{\ell} g j − g ℓ vanishes on a set of positive measure, which holds for instance for affine scores g j ( z ) = a j T z + c j g_{j}(z)=a_{j}^{\mathsf T}z+c_{j} g j ( z ) = a j T z + c j with pairwise distinct a j a_{j} a j . Unlike max-pooling, the operator is in general discontinuous across such a tie, since the two experts need not agree there.
In both examples the local operator is C 1 C^{1} C 1 inside each region Ω ( k ) \Omega^{(k)} Ω ( k ) and loses regularity only on Σ \Sigma Σ : differentiability for max-pooling, possibly continuity for hard routing; the usual activation functions are further special cases, listed after Definition 2.3. The switching region is precisely where generalized notions of differentiation become relevant. For max-pooling or ReLU operators, the network map remains continuous and locally Lipschitz, so Clarke [13 ] ’s generalized gradients are well defined. For hard-routing architectures, the network map may become discontinuous across routing boundaries. In that case the Clarke framework is generally no longer applicable. For this, we introduce a Filippov set-valued extension based on ([20 , 9 , 4 , 5 ] ). In what follows, for x ∈ R n x\in\mathbb{R}^{n} x ∈ R n and δ > 0 \delta>0 δ > 0 , B ( x , δ ) : = { y ∈ R n : ∥ y − x ∥ < δ } B(x,\delta):=\{y\in\mathbb{R}^{n}:\lVert y-x\rVert<\delta\} B ( x , δ ) := { y ∈ R n : ∥ y − x ∥ < δ } denotes the open ball of radius δ \delta δ centred at x x x , co A \operatorname{co} A co A the closed convex hull of a set A A A , co A \operatorname{co} A co A its convex hull, and μ \mu μ the Lebesgue measure. Since Φ \Phi Φ and D Φ D\Phi D Φ are only defined on Ω ∖ Σ \Omega\setminus\Sigma Ω ∖ Σ , an image Φ ( A ) \Phi(A) Φ ( A ) , or a set { ( Φ ( y ) , D Φ ( y ) ) : y ∈ A } \{(\Phi(y),D\Phi(y)):y\in A\} {( Φ ( y ) , D Φ ( y )) : y ∈ A } , is taken over y ∈ A ∩ ( Ω ∖ Σ ) y\in A\cap(\Omega\setminus\Sigma) y ∈ A ∩ ( Ω ∖ Σ ) . As μ ( Σ ) = 0 \mu(\Sigma)=0 μ ( Σ ) = 0 , this removal of Σ \Sigma Σ changes nothing in the intersections over null sets N \mathcal{N} N below, which discard Σ \Sigma Σ like any other negligible set. The motivation for the double intersection is discussed, with examples, in Appendix § 6 .§ 1 .
Theorem 2.2. Let Φ ∈ S 1 ( Ω , R m ) \Phi\in S^{1}(\Omega,\mathbb{R}^{m}) Φ ∈ S 1 ( Ω , R m ) be a switching operator (see Definition 2.1 ) and x ∈ Ω x \in\Omega x ∈ Ω . Define the Filippov regularization of Φ \Phi Φ at x x x and the Filippov extension of ( Φ , D Φ ) (\Phi,D\Phi) ( Φ , D Φ ) by
F [ Φ ] ( x ) : = ⋂ μ ( N ) = 0 δ > 0 co ‾ Φ ( B ( x , δ ) ∖ N ) , F ~ [ Φ ] ( x ) : = ⋂ μ ( N ) = 0 δ > 0 co ‾ { ( Φ ( y ) , D Φ ( y ) ) : y ∈ B ( x , δ ) ∖ N } . \mathcal{F}[\Phi](x):=\bigcap_{\substack{\mu(\mathcal{N})=0\\\delta>0}}\overline{\operatorname{co}}\Phi\left(B(x,\delta)\setminus\mathcal{N}\right), \qquad\widetilde{\mathcal{F}}[\Phi](x):=\bigcap_{\substack{\mu(\mathcal{N})=0\\\delta>0}}\overline{\operatorname{co}}\left\{(\Phi(y),D\Phi(y)):y\in B(x,\delta)\setminus\mathcal{N}\right\}. F [ Φ ] ( x ) := μ ( N ) = 0 δ > 0 ⋂ co Φ ( B ( x , δ ) ∖ N ) , F [ Φ ] ( x ) := μ ( N ) = 0 δ > 0 ⋂ co { ( Φ ( y ) , D Φ ( y )) : y ∈ B ( x , δ ) ∖ N } . (i) For every x ∈ Ω x \in\Omega x ∈ Ω ,
F [ Φ ] ( x ) = co { Φ ( k ) ( x ) : k ∈ J Φ ( x ) } ⊂ R m , F ~ [ Φ ] ( x ) = co { ( Φ ( k ) ( x ) , D Φ ( k ) ( x ) ) : k ∈ J Φ ( x ) } ⊂ R m × R m × n . \mathcal{F}[\Phi](x)=\operatorname{co}\left\{\Phi^{(k)}(x):k\in J_{\Phi}(x)\right\}\subset\mathbb{R}^{m}, \qquad\widetilde{\mathcal{F}}[\Phi](x)=\operatorname{co}\left\{(\Phi^{(k)}(x),D\Phi^{(k)}(x)):k\in J_{\Phi}(x)\right\}\subset\mathbb{R}^{m}\times\mathbb{R}^{m\times n}. F [ Φ ] ( x ) = co { Φ ( k ) ( x ) : k ∈ J Φ ( x ) } ⊂ R m , F [ Φ ] ( x ) = co { ( Φ ( k ) ( x ) , D Φ ( k ) ( x )) : k ∈ J Φ ( x ) } ⊂ R m × R m × n . (ii) If x ∉ Σ x\notin\Sigma x ∈ / Σ , there is a unique k 0 ∈ K k_{0}\in K k 0 ∈ K with x ∈ Ω ( k 0 ) x\in\Omega^{(k_{0})} x ∈ Ω ( k 0 ) ; then J Φ ( x ) = { k 0 } J_{\Phi}(x)=\{k_{0}\} J Φ ( x ) = { k 0 } , and
F [ Φ ] ( x ) = { Φ ( k 0 ) ( x ) } = { Φ ( x ) } , F ~ [ Φ ] ( x ) = { ( Φ ( x ) , D Φ ( x ) ) } . \mathcal{F}[\Phi](x)=\{\Phi^{(k_{0})}(x)\}=\{\Phi(x)\}, \qquad\widetilde{\mathcal{F}}[\Phi](x)=\{(\Phi(x),D\Phi(x))\}. F [ Φ ] ( x ) = { Φ ( k 0 ) ( x )} = { Φ ( x )} , F [ Φ ] ( x ) = {( Φ ( x ) , D Φ ( x ))} .
Proof of the theorem is provided in Appendix § 6 .§ 1 . In the Filippov regularization formula, the first intersection is taken over the sets N ⊂ R n \mathcal{N}\subset\mathbb{R}^{n} N ⊂ R n with Lebesgue measure μ ( N ) \mu(\mathcal{N}) μ ( N ) equal to zero. By (ii), the Filippov regularization coincides with the classical evaluation of Φ \Phi Φ at every regular point.
We illustrate Theorem 2.2 on the hard-routing Mixture-of-Experts block introduced above, with M = 2 M=2 M = 2 experts on a scalar input z ∈ R z\in\mathbb{R} z ∈ R : routing scores g 1 ( z ) : = z g_{1}(z):=z g 1 ( z ) := z , g 2 ( z ) : = − z g_{2}(z):=-z g 2 ( z ) := − z , and affine experts E 1 ( z ) : = z + 1 E_{1}(z):=z+1 E 1 ( z ) := z + 1 , E 2 ( z ) : = 2 z − 1 E_{2}(z):=2z-1 E 2 ( z ) := 2 z − 1 , so that Ω ( 1 ) = { z > 0 } \Omega^{(1)}=\{z>0\} Ω ( 1 ) = { z > 0 } , Ω ( 2 ) = { z < 0 } \Omega^{(2)}=\{z<0\} Ω ( 2 ) = { z < 0 } , and Σ = { 0 } \Sigma=\{0\} Σ = { 0 } . At the switching point z = 0 z=0 z = 0 , both branches are active, J Φ ( 0 ) = { 1 , 2 } J_{\Phi}(0)=\{1,2\} J Φ ( 0 ) = { 1 , 2 } , with Φ ( 1 ) ( 0 ) = E 1 ( 0 ) = 1 \Phi^{(1)}(0)=E_{1}(0)=1 Φ ( 1 ) ( 0 ) = E 1 ( 0 ) = 1 and Φ ( 2 ) ( 0 ) = E 2 ( 0 ) = − 1 \Phi^{(2)}(0)=E_{2}(0)=-1 Φ ( 2 ) ( 0 ) = E 2 ( 0 ) = − 1 actually different values – unlike a ReLU kink, where every branch agrees at the switching point itself, this is exactly the network-map discontinuity that motivates the Filippov construction. Theorem 2.2 gives
F [ Φ ] ( 0 ) = co { Φ ( 1 ) ( 0 ) , Φ ( 2 ) ( 0 ) } = co { 1 , − 1 } = [ − 1 , 1 ] , \mathcal{F}[\Phi](0)=\operatorname{co}\{\Phi^{(1)}(0),\Phi^{(2)}(0)\}=\operatorname{co}\{1,-1\}=[-1,1], F [ Φ ] ( 0 ) = co { Φ ( 1 ) ( 0 ) , Φ ( 2 ) ( 0 )} = co { 1 , − 1 } = [ − 1 , 1 ] ,
an interval of admissible output values at the routing boundary, rather than the single value either expert alone would report (Figure 1 ). Including the branch derivatives D Φ ( 1 ) ( 0 ) = E 1 ′ ( 0 ) = 1 D\Phi^{(1)}(0)=E_{1}'(0)=1 D Φ ( 1 ) ( 0 ) = E 1 ′ ( 0 ) = 1 and D Φ ( 2 ) ( 0 ) = E 2 ′ ( 0 ) = 2 D\Phi^{(2)}(0)=E_{2}'(0)=2 D Φ ( 2 ) ( 0 ) = E 2 ′ ( 0 ) = 2 , the Filippov extension is likewise a non-degenerate segment,
Figure 1. The hard-routing MoE example: Φ \Phi Φ jumps from − 1 -1 − 1 to 1 1 1 across the routing boundary z = 0 z=0 z = 0 , where g 1 ( z ) = g 2 ( z ) g_{1}(z)=g_{2}(z) g 1 ( z ) = g 2 ( z ) . The Filippov regularization F [ Φ ] ( 0 ) \mathcal{F}[\Phi](0) F [ Φ ] ( 0 ) fills this jump with the whole segment [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] (dotted, in violet) rather than either endpoint alone, and the Filippov extension F ~ [ Φ ] ( 0 ) \widetilde{\mathcal{F}}[\Phi](0) F [ Φ ] ( 0 ) is the segment joining the pairs ( Φ ( k ) ( 0 ) , D Φ ( k ) ( 0 ) ) (\Phi^{(k)}(0),D\Phi^{(k)}(0)) ( Φ ( k ) ( 0 ) , D Φ ( k ) ( 0 )) of the two branches, ( 1 , 1 ) (1,1) ( 1 , 1 ) and ( − 1 , 2 ) (-1,2) ( − 1 , 2 ) .
F ~ [ Φ ] ( 0 ) = co { ( 1 , 1 ) , ( − 1 , 2 ) } , \widetilde{\mathcal{F}}[\Phi](0)=\operatorname{co}\{(1,1),(-1,2)\}, F [ Φ ] ( 0 ) = co {( 1 , 1 ) , ( − 1 , 2 )} ,
with both coordinates varying across branches – to be contrasted with the ReLU-kink example of Figure 3 , where the value coordinate collapses to a point and only the Jacobian coordinate remains set-valued.
Figure 3. The running example at the switching point x = ( − 2 , 1 ) ∈ D x = (-2,1) \in\mathcal{D} x = ( − 2 , 1 ) ∈ D .
To the best of our knowledge, Filippov regularization has not been applied to the discontinuous hard-routing maps of feed-forward Mixture-of-Experts architectures. Earlier work addresses related issues through other constructions. Jacobs et al. [24 ] interpret the softmax outputs of the gating network as the probabilities of a stochastic one-out-of-n n n selector, and train the system through a smooth objective of the mixture. Shazeer et al. (2017) note that sparse gating “creates some theoretically scary discontinuities”, without reporting practical difficulties attributable to them; their noisy gating also gives a differentiable estimate of the load of each expert, used for load balancing. Fedus et al. [19 ] keep the softmax probability of the selected expert as a multiplicative gate: away from the routing boundaries, backpropagation then computes the classical derivative of the active branch, but at a boundary where the branch values differ, these branch derivatives are neither a classical nor a Clarke gradient of the composite map. The Filippov construction below addresses these boundaries.
Directed Acyclic graph for networks
Let us consider a directed acyclic graph (DAG). They are used in many feed-forward neural network architectures (see Figure 2 ). We then propose a new definition of feed-forward neural networks based on DAGs, following DeVore et al. [18 ] ; Daubechies et al. [16 ] and adapted to our general Filippov framework.
Figure 2. A causal Transformer block on three tokens, drawn as a directed acyclic graph with input nodes i t i_t i t (the tokens) and output nodes o t o_t o t . Each node stands for a group of neurons (one d d d -dimensional vector per token); expanding it coordinate by coordinate gives a DAG in the sense of Definition 2.3 . The attention node a t a_t a t receives the query q t q_t q t and the keys and values k s , v s k_s,v_s k s , v s of every earlier token s ≤ t s \le t s ≤ t (solid orange edges), so the in-degree grows with the context length; dashed edges are residual connections skipping one or several stages, which prevent the graph from being layered (Definition 2.4 ). The double-circled node f t f_t f t is the feed-forward sub-layer, which becomes a switching operator in the sense of Definition 2.1 when it is a hard-routed Mixture-of-Experts. CNN, ResNet, DenseNet, Inception and U-Net architectures give DAGs of the same kind.
Definition 2.3. Let G = ( V , E ) \mathcal{G}=(\mathcal{V},\mathcal{E}) G = ( V , E ) be a finite directed acyclic graph. The set V \mathcal{V} V is the set of nodes and E \mathcal{E} E is the set of directed edges. We denote by I : = { v ∈ V : In ( v ) = ∅ } \mathcal{I}:=\{v\in\mathcal{V}:\operatorname{In}(v)=\varnothing\} I := { v ∈ V : In ( v ) = ∅ } the set of input nodes, and by O : = { v ∈ V : Out ( v ) = ∅ } \mathcal{O}:=\{v\in\mathcal{V}:\operatorname{Out}(v)=\varnothing\} O := { v ∈ V : Out ( v ) = ∅ } the set of output nodes where In ( v ) : = { u ∈ V : ( u , v ) ∈ E } \operatorname{In}(v):=\{u\in\mathcal{V}:(u,v)\in\mathcal{E}\} In ( v ) := { u ∈ V : ( u , v ) ∈ E } , and Out ( v ) : = { u ∈ V : ( v , u ) ∈ E } \operatorname{Out}(v):=\{u\in\mathcal{V}:(v,u)\in\mathcal{E}\} Out ( v ) := { u ∈ V : ( v , u ) ∈ E } . The following properties define a feed-forward Filippov neural network N F ( G ) \mathcal{N}_{\mathsf{F}}(\mathcal{G}) N F ( G ) associated to G \mathcal{G} G :
To every edge e = ( u , v ) ∈ E e=(u,v)\in\mathcal{E} e = ( u , v ) ∈ E is associated a scalar weight w u , v ∈ R w_{u,v}\in\mathbb{R} w u , v ∈ R .
For any node i ∈ I i\in\mathcal{I} i ∈ I , we associate a scalar value. For some input nodes, we prescribe the value to be fixed and equal to + 1 +1 + 1 . This subset B ⊂ I \mathcal{B}\subset\mathcal{I} B ⊂ I is called the bias set.
For every non-input node v ∈ V ∖ I v\in\mathcal{V}\setminus\mathcal{I} v ∈ V ∖ I , we assume that:
A local operator Φ v ∈ S 1 ( R ∣ In ( v ) ∣ ) \Phi_v\in S^1(\mathbb{R}^{|\operatorname{In}(v)|}) Φ v ∈ S 1 ( R ∣ In ( v ) ∣ ) is associated (see Definition 2.1 ), with switching region Σ v \Sigma_v Σ v , branches ( Φ v ( k ) ) k = 1 m v (\Phi_v^{(k)})_{k=1}^{m_v} ( Φ v ( k ) ) k = 1 m v , regions Ω v ( 1 ) , … , Ω v ( m v ) \Omega_v^{(1)},\ldots,\Omega_v^{(m_v)} Ω v ( 1 ) , … , Ω v ( m v ) , and active index set J v : = J Φ v J_v:=J_{\Phi_v} J v := J Φ v .
For any input vector x = ( x i ) i ∈ I ∖ B ∈ R d x=(x_i)_{i\in\mathcal{I}\setminus\mathcal{B}}\in\mathbb{R}^{d} x = ( x i ) i ∈ I ∖ B ∈ R d with d : = ∣ I ∖ B ∣ d:=|\mathcal{I}\setminus\mathcal{B}| d := ∣ I ∖ B ∣ ,
We denote the value of any input node by X i ( x ) = { x i } X_i(x)=\{x_i\} X i ( x ) = { x i } if i ∈ I ∖ B i\in\mathcal{I}\setminus\mathcal{B} i ∈ I ∖ B , and by X i ( x ) = { + 1 } X_i(x)=\{+1\} X i ( x ) = { + 1 } for any i ∈ B i\in\mathcal{B} i ∈ B .
We define Z v ( x ) : = ∏ u ∈ In ( v ) z u , v ⊂ R ∣ In ( v ) ∣ Z_v(x):=\prod_{u\in\operatorname{In}(v)}z_{u,v}\subset\mathbb{R}^{|\operatorname{In}(v)|} Z v ( x ) := ∏ u ∈ In ( v ) z u , v ⊂ R ∣ In ( v ) ∣ , with z u , v : = w u , v × X u ( x ) z_{u,v}:=w_{u,v}\times X_u(x) z u , v := w u , v × X u ( x ) for any u ∈ In ( v ) u\in\operatorname{In}(v) u ∈ In ( v ) .
We then define recursively the value X v ( x ) X_v(x) X v ( x ) of each node v ∈ V ∖ I v\in\mathcal{V}\setminus\mathcal{I} v ∈ V ∖ I by the propagated Filippov set of Φ v \Phi_v Φ v at Z v ( x ) Z_v(x) Z v ( x ) , extending the pointwise Filippov regularization F [ Φ v ] ( ζ ) F[\Phi_v](\zeta) F [ Φ v ] ( ζ ) of Theorem 2.2 to the whole box Z v ( x ) Z_v(x) Z v ( x ) by
X v ( x ) = F [ Φ v ] ( Z v ( x ) ) : = ⋃ k ∈ J v ( Z v ( x ) ) { Φ v ( k ) ( ζ ) : ζ ∈ Z v ( x ) ∩ Ω v ( k ) } ∪ ⋃ ζ ∈ Z v ( x ) ∩ Σ v F [ Φ v ] ( ζ ) . X_v(x)=F[\Phi_v](Z_v(x)):=\bigcup_{k\in J_v(Z_v(x))}\{\Phi_v^{(k)}(\zeta):\zeta\in Z_v(x)\cap\Omega_v^{(k)}\}\cup\bigcup_{\zeta\in Z_v(x)\cap\Sigma_v}F[\Phi_v](\zeta). X v ( x ) = F [ Φ v ] ( Z v ( x )) := k ∈ J v ( Z v ( x )) ⋃ { Φ v ( k ) ( ζ ) : ζ ∈ Z v ( x ) ∩ Ω v ( k ) } ∪ ζ ∈ Z v ( x ) ∩ Σ v ⋃ F [ Φ v ] ( ζ ) . where J v ( Z v ( x ) ) : = { k : Ω v ( k ) ∩ Z v ( x ) ≠ ∅ } J_v(Z_v(x)):=\{k:\Omega_v^{(k)}\cap Z_v(x)\ne\varnothing\} J v ( Z v ( x )) := { k : Ω v ( k ) ∩ Z v ( x ) = ∅ } is the set of regions met by the box Z v ( x ) Z_v(x) Z v ( x ) , extending the pointwise active index set J v ( ζ ) J_v(\zeta) J v ( ζ ) of Definition 2.1 . Each term of the first union is the image, under the single C 1 C^1 C 1 branch Φ v ( k ) \Phi_v^{(k)} Φ v ( k ) , of the part of the box lying in Ω v ( k ) \Omega_v^{(k)} Ω v ( k ) , since Φ v ( k ) = Φ v \Phi_v^{(k)}=\Phi_v Φ v ( k ) = Φ v on Ω v ( k ) \Omega_v^{(k)} Ω v ( k ) , this union is the set { Φ v ( ζ ) : ζ ∈ Z v ( x ) ∖ Σ v } \{\Phi_v(\zeta):\zeta\in Z_v(x)\setminus\Sigma_v\} { Φ v ( ζ ) : ζ ∈ Z v ( x ) ∖ Σ v } of the classical values at the regular points of the box, while the second union collects the local convex hulls at its switching points.
The graph G \mathcal{G} G fixes the topology of computation: since G \mathcal{G} G is directed and acyclic, a topological order on V \mathcal{V} V always exists, so the recursive definition of Z v ( x ) Z_v(x) Z v ( x ) and X v ( x ) X_v(x) X v ( x ) is well founded and terminates after finitely many steps – there is no risk of circular dependency, unlike in recurrent architectures. The weights w u , v w_{u,v} w u , v act exactly as in a classical network: they rescale the (set-valued) output of a node before it reaches the next one. Since scalar multiplication by w u , v w_{u,v} w u , v maps convex compact sets to convex compact sets, this operation is compatible with the Filippov formalism at every step. The bias set B \mathcal{B} B plays the same role as in a classical affine layer: fixing X i ( x ) = { + 1 } X_i(x) = \{+1\} X i ( x ) = { + 1 } for i ∈ B i \in\mathcal{B} i ∈ B allows constant (bias) terms to be absorbed into the weighted-sum structure of Z v ( x ) Z_v(x) Z v ( x ) without special-casing the definition.
As we can see, each node is associated to one neuron. The DAGs admit topological sorting and no cycles [10 ] . This allows us to deduce relationships in a logical order. Consequently, for any given feed-forward neural network, we can compute the list of all paths from one input node of I \mathcal{I} I to any output node of O \mathcal{O} O and conversely. Moreover, for any e = ( u , v ) ∈ E e=(u,v)\in\mathcal{E} e = ( u , v ) ∈ E the parameters w e w_e w e are then trainable. This definition encompasses virtually all network architectures used in practice – fully connected, convolutional, residual, U-Net, U2 ^2 2 -Net, and Transformer networks – with input, bias, hidden and output units corresponding to nodes of G \mathcal{G} G , directed edges representing dependencies between nodes, and the (possibly multi-valued) local operators Φ v \Phi_v Φ v representing the computation performed at each node.
The layer types of a standard architecture are instances of Definition 2.3 obtained by a choice of fan-in and of local operator: a convolution reads the weights of all its edges in one kernel θ \theta θ , w u , v = θ c ( u ) , c ( v ) , p ( u ) − p ( v ) w_{u,v}=\theta_{c(u),c(v),p(u)-p(v)} w u , v = θ c ( u ) , c ( v ) , p ( u ) − p ( v ) for u u u in the receptive field of v v v (c c c being the channel and p p p the spatial position of a unit); a max-pooling unit is one node carrying the max-of-∣ In ( v ) ∣ |\operatorname{In}(v)| ∣ In ( v ) ∣ switching operator of § 2.2 , whose ties are the only place where branches compete; and a softmax over n n n scores is n n n smooth nodes v 1 , … , v n v_1,\ldots,v_n v 1 , … , v n , whose switching regions are empty, Σ v j = ∅ \Sigma_{v_j}=\varnothing Σ v j = ∅ : it never switches, unlike the hard-routing argmax that replaces it in a Mixture-of-Experts, which switches where two scores tie. The local operator is where all the nonsmooth, possibly discontinuous behavior is concentrated – smooth activations, ReLU, sign-type relays and quantization are all finitely many C 1 C^1 C 1 branches – and the propagated value F [ Φ v ] ( Z v ( x ) ) \mathcal{F}[\Phi_v](Z_v(x)) F [ Φ v ] ( Z v ( x )) is the union of the pointwise Filippov values over ζ ∈ Z v ( x ) \zeta\in Z_v(x) ζ ∈ Z v ( x ) , not a global convexification of the incoming box, which is what keeps it nonempty and compact.
We now single out the special case where the nodes organize into a sequence of layers, each feeding only the next.
Definition 2.4. A feed-forward Filippov neural network N F ( G ) \mathcal{N}_{\mathcal{F}}(\mathcal{G}) N F ( G ) (see Definition 2.3 ) is layered if there exists a rank function ℓ : V → { 0 , 1 , … , K } \ell:\mathcal{V}\to\{0,1,\ldots,K\} ℓ : V → { 0 , 1 , … , K } with ℓ ( i ) = 0 \ell(i)=0 ℓ ( i ) = 0 for every non-bias input i ∈ I ∖ B i\in\mathcal{I}\setminus\mathcal{B} i ∈ I ∖ B , ℓ ( o ) = K \ell(o)=K ℓ ( o ) = K for every output o ∈ O o\in\mathcal{O} o ∈ O , and ℓ ( v ) = ℓ ( u ) + 1 \ell(v)=\ell(u)+1 ℓ ( v ) = ℓ ( u ) + 1 for every edge ( u , v ) ∈ E (u,v)\in\mathcal{E} ( u , v ) ∈ E . The level sets of ℓ \ell ℓ are the successive layers of the network.
The classical sequential architectures underlying Bach et al. [6 ] ’s Layer-wise Relevance Propagation – and, by inheritance, the acronyms LRP and SRP used throughout this paper – are exactly the layered case of Definition 2.4. The active graph G ( x ) G(x) G ( x ) of a general feed-forward Filippov network need not be layered: skip connections, hard-routed Mixture-of-Experts, and attention-induced edges all produce active graphs whose nodes cannot be assigned a single consistent rank. Every result of this paper is stated for the general, not necessarily layered, DAG G \mathcal{G} G ; the layered case of Definition 2.4 is recovered as the special instance considered in the classical literature.
Forward Propagation and the Active Graph
The recursion of Definition 2.3 propagates, at every switching node, the whole local convex hull of the active branches, so that node values are in general sets. An ordinary forward pass, or an automatic-differentiation library, computes none of these sets: it commits at each switching node to one active branch and propagates a single real number. The following definition formalizes this choice as a branch selection , and characterizes, through the active index sets J v J_v J v , the selections that are consistent with the values they themselves produce.
Definition 2.5. Let N F ( G ) \mathcal{N}_{F}(\mathcal{G}) N F ( G ) be a feed-forward Filippov neural network (see Definition 2.3) on a finite directed acyclic graph G = ( V , E ) \mathcal{G}=(\mathcal{V},\mathcal{E}) G = ( V , E ) , and let x ∈ R d x\in\mathbb{R}^{d} x ∈ R d , d : = ∣ I ∖ B ∣ d:=|\mathcal{I}\setminus\mathcal{B}| d := ∣ I ∖ B ∣ . A branch selection is a family
κ = ( κ v ) v ∈ V ∖ I ∈ ∏ v ∈ V ∖ I { 1 , … , m v } , \kappa=(\kappa_v)_{v\in\mathcal{V}\setminus\mathcal{I}}\in\prod_{v\in\mathcal{V}\setminus\mathcal{I}}\{1,\ldots,m_v\}, κ = ( κ v ) v ∈ V ∖ I ∈ v ∈ V ∖ I ∏ { 1 , … , m v } , assigning to every non-input node v v v one of the m v m_v m v branches of Φ v ∈ S 1 ( Ω v ) \Phi_v\in\mathcal{S}^{1}(\Omega_v) Φ v ∈ S 1 ( Ω v ) . It determines the recursion of Definition 2.3 in which the Filippov value F [ Φ v ] \mathcal{F}[\Phi_v] F [ Φ v ] is replaced, at every node v v v , by the single branch Φ v ( κ v ) \Phi_v^{(\kappa_v)} Φ v ( κ v ) : forward along a topological order of G \mathcal{G} G ,
X i ( κ ) ( x ) : = X i ( x ) ( i ∈ I ) , Z v ( κ ) ( x ) : = ∏ a ∈ In ( v ) w a , v X a ( κ ) ( x ) , X v ( κ ) ( x ) : = Φ v ( κ v ) ( Z v ( κ ) ( x ) ) ( v ∈ V ∖ I ) . (1) X_i^{(\kappa)}(x):=X_i(x)\quad(i\in\mathcal{I}),\qquad Z_v^{(\kappa)}(x):=\prod_{a\in\operatorname{In}(v)}w_{a,v}X_a^{(\kappa)}(x),\qquad X_v^{(\kappa)}(x):=\Phi_v^{(\kappa_v)}\left(Z_v^{(\kappa)}(x)\right)\quad(v\in\mathcal{V}\setminus\mathcal{I}). \tag*{(1)} X i ( κ ) ( x ) := X i ( x ) ( i ∈ I ) , Z v ( κ ) ( x ) := a ∈ In ( v ) ∏ w a , v X a ( κ ) ( x ) , X v ( κ ) ( x ) := Φ v ( κ v ) ( Z v ( κ ) ( x ) ) ( v ∈ V ∖ I ) . (1) Since X i ( x ) = { x i } X_i(x)=\{x_i\} X i ( x ) = { x i } or { + 1 } \{+1\} { + 1 } , every Z v ( κ ) ( x ) Z_v^{(\kappa)}(x) Z v ( κ ) ( x ) and X v ( κ ) ( x ) X_v^{(\kappa)}(x) X v ( κ ) ( x ) is a singleton, whose unique element we denote by z v ( κ ) ( x ) ∈ R ∣ In ( v ) ∣ z_v^{(\kappa)}(x)\in\mathbb{R}^{|\operatorname{In}(v)|} z v ( κ ) ( x ) ∈ R ∣ In ( v ) ∣ and x v ( κ ) ( x ) ∈ R x_v^{(\kappa)}(x)\in\mathbb{R} x v ( κ ) ( x ) ∈ R respectively. The selection κ \kappa κ is admissible at x x x if it picks, at every node, a branch that is active at the pre-activation it has itself produced,
κ v ∈ J v ( z v ( κ ) ( x ) ) for every v ∈ V ∖ I , (2) \kappa_v\in J_v\left(z_v^{(\kappa)}(x)\right)\qquad\text{for every }v\in\mathcal{V}\setminus\mathcal{I}, \tag*{(2)} κ v ∈ J v ( z v ( κ ) ( x ) ) for every v ∈ V ∖ I , (2) and we write
M ( x ) : = { κ : κ ∈ ∏ v ∈ V ∖ I J v ( z v ( κ ) ( x ) ) } \mathcal{M}(x):=\left\{\kappa:\ \kappa\in\prod_{v\in\mathcal{V}\setminus\mathcal{I}}J_v\left(z_v^{(\kappa)}(x)\right)\right\} M ( x ) := ⎩ ⎨ ⎧ κ : κ ∈ v ∈ V ∖ I ∏ J v ( z v ( κ ) ( x ) ) ⎭ ⎬ ⎫ for the set of branch selections admissible at x x x . For κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) and o ∈ O o\in\mathcal{O} o ∈ O , the scalar x o ( κ ) ( x ) x_o^{(\kappa)}(x) x o ( κ ) ( x ) is the output value along κ \kappa κ .
A branch selection is attached to the network, not to the input: like the weights w u , v w_{u,v} w u , v , κ \kappa κ is given before any x x x , and it is the set M ( x ) \mathcal{M}(x) M ( x ) of admissible ones that varies with x x x . Condition (2) is equivalent to z v ( κ ) ( x ) ∈ Ω v ( κ v ) z_v^{(\kappa)}(x)\in\Omega_v^{(\kappa_v)} z v ( κ ) ( x ) ∈ Ω v ( κ v ) , the condition under which the evaluation in (1) is meaningful at all: an admissible selection is one with all node values well defined. Along a topological order it is a recipe, not a test – once the predecessors of v v v carry values, J v ( z v ( κ ) ( x ) ) J_v(z_v^{(\kappa)}(x)) J v ( z v ( κ ) ( x )) is nonempty, so an admissible κ v \kappa_v κ v is available and M ( x ) ≠ ∅ \mathcal{M}(x)\ne\varnothing M ( x ) = ∅ (Theorem 2.7 (ii)) – and it carries information only at a switching node.
Two superscripts must be distinguished: ( κ ) (\kappa) ( κ ) is the global selection, ( κ v ) (\kappa_v) ( κ v ) the local branch index it prescribes at v v v . Since that branch changes the values downstream, the product of Definition 2.5 has κ \kappa κ -dependent factors: M ( x ) \mathcal{M}(x) M ( x ) is a set of fixed points, not a product of fixed sets, and it is fibered – along a topological order v 1 , v 2 , … v_1,v_2,\ldots v 1 , v 2 , … , each κ v j \kappa_{v_j} κ v j ranges over J v j J_{v_j} J v j at the pre-activation produced upstream, read at the point z v j ( κ ) ( x ) z_{v_j}^{(\kappa)}(x) z v j ( κ ) ( x ) and not on the box Z v j ( x ) Z_{v_j}(x) Z v j ( x ) , whose index set is empty at a pure switching point. Hence M ( x ) ⊆ ∏ ν ⋃ ζ ∈ Z v ( x ) J v ( ζ ) \mathcal{M}(x)\subseteq\prod_{\nu}\bigcup_{\zeta\in\mathcal{Z}_{v}(x)}J_v(\zeta) M ( x ) ⊆ ∏ ν ⋃ ζ ∈ Z v ( x ) J v ( ζ ) , in general strictly (Appendix § 6 .§ 2 ), with equality when the branches agree at the switching points, as for ReLU, maxout or max-pooling.
Alongside a branch selection, it is useful to isolate the nodes and edges of G \mathcal{G} G that actually carry a nonzero contribution for a given input, since a network typically activates only a subgraph of the full DAG.
Definition 2.6. Let N F ( G ) \mathcal{N}_{\mathcal{F}}(\mathcal{G}) N F ( G ) be a feed-forward Filippov neural network, then for any input x ∈ R d x \in\mathbb{R}^{d} x ∈ R d (d : = ∣ I \ B ∣ d := |\mathcal{I} \backslash\mathcal{B}| d := ∣ I \ B ∣ ), we then define G ( x ) = ( V ( x ) , E ( x ) ) \mathcal{G}(x) = (V(x), E(x)) G ( x ) = ( V ( x ) , E ( x )) the active graph of G \mathcal{G} G where V ( x ) = { v ∈ H ∪ I : X v ( x ) ≠ { 0 } } ∪ O V(x) = \{v \in\mathcal{H} \cup\mathcal{I}: X_v(x) \ne\{0\}\} \cup\mathcal{O} V ( x ) = { v ∈ H ∪ I : X v ( x ) = { 0 }} ∪ O is the set of active nodes and E ( x ) = { ( u , v ) ∈ E : u , v ∈ V ( x ) , z u , v ≠ { 0 } } E(x) = \{(u,v) \in E : u,v \in V(x), z_{u,v} \ne\{0\}\} E ( x ) = {( u , v ) ∈ E : u , v ∈ V ( x ) , z u , v = { 0 }} is the set of active edges where z u , v = w u , v X u ( x ) z_{u,v} = w_{u,v}X_u(x) z u , v = w u , v X u ( x ) is the weighted contribution of neuron u u u to neuron v v v for any edge ( u , v ) ∈ E ( x ) (u,v) \in E(x) ( u , v ) ∈ E ( x ) . We write I ( x ) : = I ∩ V ( x ) \mathcal{I}(x) := \mathcal{I} \cap V(x) I ( x ) := I ∩ V ( x ) for the set of active input nodes.
The active graph G ( x ) \mathcal{G}(x) G ( x ) is the graph of the nonzero forward contributions: a node of value zero leaves it with its edges, even if, at a linear node, the derivative of the output with respect to its value does not vanish. No relevance being attached to such a node, zero-valued nodes create no non-generic attribution ambiguity.
Switching nodes are exceptional for a generic input, but nothing said so far prevents a node from being switching for every input at once; this is what the non-degeneracy hypothesis of Theorem 2.7 (iii) below excludes. We now show that forward propagation is well-posed.
Theorem 2.7. Let N F ( G ) \mathcal{N}_{\mathcal{F}}(\mathcal{G}) N F ( G ) be a feed-forward Filippov neural network (see Definition 2.3 ) and x ∈ R d x \in\mathbb{R}^{d} x ∈ R d .
(i) Every node value is the nonempty compact set X v ( x ) = ⋃ ζ ∈ Z v ( x ) [ min k ∈ J v ( ζ ) Φ v ( k ) ( ζ ) , max k ∈ J v ( ζ ) Φ v ( k ) ( ζ ) ] X_v(x) = \bigcup_{\zeta\in Z_v(x)} [\min_{k\in J_v(\zeta)} \Phi_v^{(k)}(\zeta), \max_{k\in J_v(\zeta)} \Phi_v^{(k)}(\zeta)] X v ( x ) = ⋃ ζ ∈ Z v ( x ) [ min k ∈ J v ( ζ ) Φ v ( k ) ( ζ ) , max k ∈ J v ( ζ ) Φ v ( k ) ( ζ )] (v ∈ V \ I v \in\mathcal{V} \backslash\mathcal{I} v ∈ V \ I ), and the network output is the box S N ( x ) = ∏ o ∈ O X o ( x ) ⊂ R ∣ O ∣ \mathcal{S}_{\mathcal{N}}(x) = \prod_{o\in\mathcal{O}} X_o(x) \subset\mathbb{R}^{|\mathcal{O}|} S N ( x ) = ∏ o ∈ O X o ( x ) ⊂ R ∣ O ∣ .
(ii) M ( x ) \mathcal{M}(x) M ( x ) is nonempty and finite, with x v ( κ ) ( x ) ∈ X v ( x ) x_v^{(\kappa)}(x) \in X_v(x) x v ( κ ) ( x ) ∈ X v ( x ) at every node v ∈ V v \in\mathcal{V} v ∈ V and every κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) . For κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , v ∈ V v \in\mathcal{V} v ∈ V and o ∈ O o \in\mathcal{O} o ∈ O , the output x o ( κ ) x_o^{(\kappa)} x o ( κ ) of (1) is continuously differentiable with respect to the value x v ( κ ) x_v^{(\kappa)} x v ( κ ) of v v v , and
g v , o ( κ ) ( x ) : = ∂ x o ( κ ) ∂ x v ( κ ) ( x ) = ∑ p ∈ P ( v , o ) ∏ ( a , b ) ∈ p w a , b [ D Φ b ( κ b ) ( z b ( κ ) ( x ) ) ] a , (3) g_{v,o}^{(\kappa)}(x) := \frac{\partial x_o^{(\kappa)}}{\partial x_v^{(\kappa)}}(x) = \sum_{\mathbf{p}\in\mathcal{P}(v,o)} \prod_{(a,b)\in\mathbf{p}} w_{a,b} \left[D\Phi_b^{(\kappa_b)}\left(z_b^{(\kappa)}(x)\right)\right]_a, \tag*{(3)} g v , o ( κ ) ( x ) := ∂ x v ( κ ) ∂ x o ( κ ) ( x ) = p ∈ P ( v , o ) ∑ ( a , b ) ∈ p ∏ w a , b [ D Φ b ( κ b ) ( z b ( κ ) ( x ) ) ] a , (3) where P ( v , o ) \mathcal{P}(v,o) P ( v , o ) is the set of directed paths from v v v to o o o . Hence the branch-admissible set of value-gradient pairs
S ~ v , o ( x ) : = co { ( x o ( κ ) ( x ) , g v , o ( κ ) ( x ) ) : κ ∈ M ( x ) } (4) \widetilde{\mathcal{S}}_{v,o}(x) := \operatorname{co}\left\{\left(x_o^{(\kappa)}(x), g_{v,o}^{(\kappa)}(x)\right) : \kappa\in\mathcal{M}(x)\right\} \tag*{(4)} S v , o ( x ) := co { ( x o ( κ ) ( x ) , g v , o ( κ ) ( x ) ) : κ ∈ M ( x ) } (4) is a nonempty convex compact subset of R 2 \mathbb{R}^{2} R 2 .
(iii) Let D : = { x ∈ R d : Z v ( x ) ∩ Σ v ≠ ∅ for some v ∈ V \ I } D := \{x \in\mathbb{R}^{d} : Z_v(x) \cap\Sigma_v \ne\varnothing\text{ for some } v \in\mathcal{V} \backslash\mathcal{I}\} D := { x ∈ R d : Z v ( x ) ∩ Σ v = ∅ for some v ∈ V \ I } be the exceptional set. If x ∉ D x \notin D x ∈ / D , then every X v ( x ) X_v(x) X v ( x ) is a singleton, M ( x ) = { κ } \mathcal{M}(x) = \{\kappa\} M ( x ) = { κ } , and S ~ v , o ( x ) = { ( x o ( x ) , ∂ x o / ∂ x v ( x ) ) } \widetilde{\mathcal{S}}_{v,o}(x) = \{(x_o(x), \partial x_o/\partial x_v(x))\} S v , o ( x ) = {( x o ( x ) , ∂ x o / ∂ x v ( x ))} with ∂ x o / ∂ x v ( x ) = g v , o ( κ ) ( x ) \partial x_o/\partial x_v(x) = g_{v,o}^{(\kappa)}(x) ∂ x o / ∂ x v ( x ) = g v , o ( κ ) ( x ) . If moreover the network is non-degenerate, in the sense that
μ ( { x ∈ R d : z v ( κ ) ( x ) ∈ Σ v for some κ ∈ M ( x ) } ) = 0 for every v ∈ V \ I , \mu\left(\left\{x \in\mathbb{R}^{d} : z_v^{(\kappa)}(x) \in\Sigma_v \text{ for some } \kappa\in\mathcal{M}(x)\right\}\right) = 0 \qquad\text{for every } v \in\mathcal{V} \backslash\mathcal{I}, μ ( { x ∈ R d : z v ( κ ) ( x ) ∈ Σ v for some κ ∈ M ( x ) } ) = 0 for every v ∈ V \ I , then μ ( D ) = 0 \mu(D) = 0 μ ( D ) = 0 .
(iv) If the branches agree on the switching sets (Φ v ( k ) ( ζ ) = Φ v ( k ′ ) ( ζ ) \Phi_v^{(k)}(\zeta) = \Phi_v^{(k')}(\zeta) Φ v ( k ) ( ζ ) = Φ v ( k ′ ) ( ζ ) for every ζ ∈ Σ v \zeta\in\Sigma_v ζ ∈ Σ v and k , k ′ ∈ J v ( ζ ) k,k' \in J_v(\zeta) k , k ′ ∈ J v ( ζ ) , as for ReLU, maxout or max-pooling), then x ↦ x o ( x ) x \mapsto x_o(x) x ↦ x o ( x ) is single-valued and locally Lipschitz and x o ( κ ) ( x ) = x o ( x ) x_o^{(\kappa)}(x) = x_o(x) x o ( κ ) ( x ) = x o ( x ) for every κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) . If x ∉ D x \notin D x ∈ / D , then ∂ ∘ x o ( x ) = { ( g i , o ( κ ) ( x ) ) i ∈ I \ B } \partial^\circ x_o(x) = \{(g_{i,o}^{(\kappa)}(x))_{i\in\mathcal{I}\backslash\mathcal{B}}\} ∂ ∘ x o ( x ) = {( g i , o ( κ ) ( x ) ) i ∈ I \ B } for the unique κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , and
∂ ∘ x o ( x ) ⊆ co { ( g i , o ( κ ) ( x ) ) i ∈ I \ B : κ ∈ M ( x ) } for every x ∈ R d . \partial^\circ x_o(x) \subseteq\operatorname{co}\left\{(g_{i,o}^{(\kappa)}(x))_{i\in\mathcal{I}\backslash\mathcal{B}} : \kappa\in\mathcal{M}(x)\right\} \qquad\text{for every } x \in\mathbb{R}^{d}. ∂ ∘ x o ( x ) ⊆ co { ( g i , o ( κ ) ( x ) ) i ∈ I \ B : κ ∈ M ( x ) } for every x ∈ R d .
Proof and a worked example are in Appendices A.3 and A.2.
Non-degeneracy constrains preimages, not images: each Σ v \Sigma_v Σ v is null, but a pre-activation may pull it back to a large set. It is not implied by nonzero weights: at the second node of x ↦ ReLU ( ReLU ( x ) ) x \mapsto\operatorname{ReLU}(\operatorname{ReLU}(x)) x ↦ ReLU ( ReLU ( x )) , the pre-activation vanishes on ( − ∞ , 0 ) (-\infty,0) ( − ∞ , 0 ) , and a max-pooling window that follows ReLU nodes is a tie on an open set as soon as all its values vanish, which puts a whole neighborhood of a photograph in D D D for a network such as VGG-16. Sections 3–4 do not need it: the branches of these operators agree on the switching sets, so that the node values do not branch, by (iv), at every x x x .
The derivative in (3) is the computational-graph one: the value of v v v becomes a free variable, its descendants are recomputed with the branches prescribed by κ \kappa κ , the other nodes frozen – well defined since each Φ u ( κ u ) \Phi_u^{(\kappa_u)} Φ u ( κ u ) is C 1 C^1 C 1 near z u ( κ ) ( x ) z_u^{(\kappa)}(x) z u ( κ ) ( x ) even in Σ u \Sigma_u Σ u . At an input or bias node it is the ordinary partial derivative with respect to the prescribed value. At a regular point we write x v x_v x v for the unique element of X v ( x ) X_v(x) X v ( x ) , the convention used below; § 3 –§ 4 are stated at the inputs where the node values do not branch. An automatic-differentiation pass resolving each switching node consistently with its propagated values returns a pair of F v , o ( x ) \mathcal{F}_{v,o}(x) F v , o ( x ) , and why M ( x ) \mathcal{M}(x) M ( x ) is global rather than per path: z u ( k ) ( x ) z_u^{(k)}(x) z u ( k ) ( x ) depends on all ancestors of u u u .
At x ∈ D x \in D x ∈ D it is conservative both ways: its value coordinate may miss part of X o ( x ) X_o(x) X o ( x ) (Appendix § 6 .§ 2 ), and the inclusion of (iv) may be strict – for o = x h 1 − x h 2 o = x_{h_1} - x_{h_2} o = x h 1 − x h 2 with two ReLU nodes fed by one input, ∂ ∘ x o ( 0 ) = { 0 } \partial^\circ x_o(0) = \{0\} ∂ ∘ x o ( 0 ) = { 0 } while independent choices at the common kink give [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . We keep M ( x ) \mathcal{M}(x) M ( x ) : what a deterministic implementation can choose. The boxes, for their part, are outer enclosures: their products forget the correlations between the descendants of a node. For h = sign ( x ) h = \operatorname{sign}(x) h = sign ( x ) and o = h − h o = h - h o = h − h through two relays, X o ( 0 ) = [ − 2 , 2 ] X_o(0) = [-2,2] X o ( 0 ) = [ − 2 , 2 ] , while the composite map vanishes identically.
The example of Figure 3 illustrates Definition 2.3 at x = ( − 2 , 1 ) x = (-2,1) x = ( − 2 , 1 ) : ReLU hidden nodes h 1 h_1 h 1 , h 2 h_2 h 2 , linear outputs o 1 o_1 o 1 , o 2 o_2 o 2 , a skip edge i 1 → o 1 i_1 \to o_1 i 1 → o 1 and bias nodes b 1 b_1 b 1 , b 2 b_2 b 2 of value + 1 +1 + 1 , with weights w i 2 , h 1 = w h 2 , o 1 = w h 2 , o 2 = 2 w_{i_2,h_1} = w_{h_2,o_1} = w_{h_2,o_2} = 2 w i 2 , h 1 = w h 2 , o 1 = w h 2 , o 2 = 2 , w i 1 , h 2 = w h 1 , o 2 = w b 2 , o 2 = − 1 w_{i_1,h_2} = w_{h_1,o_2} = w_{b_2,o_2} = -1 w i 1 , h 2 = w h 1 , o 2 = w b 2 , o 2 = − 1 , zero on the two dashed edges and 1 on the others. Each arrow carries its contribution z u , v = w u , v X u ( x ) z_{u,v} = w_{u,v}X_u(x) z u , v = w u , v X u ( x ) , a set like every value here, so that the box Z v ( x ) = ∏ u ∈ In ( v ) z u , v Z_v(x) = \prod_{u\in\operatorname{In}(v)} z_{u,v} Z v ( x ) = ∏ u ∈ In ( v ) z u , v is read off the arrows entering v v v , and each node its value X v ( x ) X_v(x) X v ( x ) together with the derivative D Φ v ( k ) ( z v ( x ) ) D\Phi_v^{(k)}(z_v(x)) D Φ v ( k ) ( z v ( x )) of its operator over the active branches k ∈ J v ( z v ( x ) ) k \in J_v(z_v(x)) k ∈ J v ( z v ( x )) , written as a single number since every operator here sums its inputs, so that the entry [ D Φ v ( k ) ] a [D\Phi_v^{(k)}]_a [ D Φ v ( k ) ] a entering (3) is the same for every predecessor a a a – replaced, at the input and bias nodes, which carry no operator, by the path-gradient g v , o 1 ( k ) ( x ) g_{v,o_1}^{(k)}(x) g v , o 1 ( k ) ( x ) of (3) over κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) . Both are branch-wise values, whose convex hulls give the Filippov extension and the gradient coordinate of F v , o 1 ( x ) \mathcal{F}_{v,o_1}(x) F v , o 1 ( x ) , here [ − 1 , 0 ] [-1,0] [ − 1 , 0 ] at i 1 i_1 i 1 and [ 2 , 4 ] [2,4] [ 2 , 4 ] at i 2 i_2 i 2 . Each arrow takes the colour of the node it leaves, and red marks the switching node; the colours therefore label nodes, not paths. Paths and branches should not be confused on the picture either: i 1 i_1 i 1 reaches o 1 o_1 o 1 along three paths – the skip edge and one through each hidden node – which (3) sums, so the two values displayed at i 1 i_1 i 1 are one per admissible selection and not one per path, the two elements of M ( x ) \mathcal{M}(x) M ( x ) differing only at the kink node h 1 h_1 h 1 .
The arrows into h 1 h_1 h 1 carry z i 1 , h 1 = { − 2 } z_{i_1,h_1} = \{-2\} z i 1 , h 1 = { − 2 } , z i 2 , h 1 = { 2 } z_{i_2,h_1} = \{2\} z i 2 , h 1 = { 2 } and z b 1 , h 1 = { 0 } z_{b_1,h_1} = \{0\} z b 1 , h 1 = { 0 } , so the box Z h 1 ( x ) = { ( − 2 , 2 , 0 ) } Z_{h_1}(x) = \{(-2,2,0)\} Z h 1 ( x ) = {( − 2 , 2 , 0 )} is the single point ( − 2 , 2 , 0 ) (-2,2,0) ( − 2 , 2 , 0 ) of Σ h 1 \Sigma_{h_1} Σ h 1 – a ReLU kink, whence x ∈ D x \in D x ∈ D – while Z h 2 ( x ) = { ( 2 , 1 , 1 ) } Z_{h_2}(x) = \{(2,1,1)\} Z h 2 ( x ) = {( 2 , 1 , 1 )} lies in the open region where the ReLU is the identity. The operator carried by h 1 h_1 h 1 is Φ h 1 ( ζ ) = max ( ζ i 1 + ζ i 2 + ζ b 1 , 0 ) \Phi_{h_1}(\zeta) = \max(\zeta_{i_1} + \zeta_{i_2} + \zeta_{b_1},0) Φ h 1 ( ζ ) = max ( ζ i 1 + ζ i 2 + ζ b 1 , 0 ) on R ∣ In ( h 1 ) ∣ = R 3 \mathbb{R}^{|\operatorname{In}(h_1)|} = \mathbb{R}^3 R ∣ In ( h 1 ) ∣ = R 3 , whose switching set is the hyperplane where that sum vanishes; two of its branches are then admissible, the constant Φ h 1 ( 1 ) = 0 \Phi_{h_1}^{(1)} = 0 Φ h 1 ( 1 ) = 0 and the sum Φ h 1 ( 2 ) ( ζ ) = ζ i 1 + ζ i 2 + ζ b 1 \Phi_{h_1}^{(2)}(\zeta) = \zeta_{i_1} + \zeta_{i_2} + \zeta_{b_1} Φ h 1 ( 2 ) ( ζ ) = ζ i 1 + ζ i 2 + ζ b 1 , and they agree at the kink: X h 1 ( x ) = { 0 } X_{h_1}(x) = \{0\} X h 1 ( x ) = { 0 } under either, so every downstream value stays a singleton, X o 1 ( x ) = { 6 } X_{o_1}(x) = \{6\} X o 1 ( x ) = { 6 } and X o 2 ( x ) = { 7 } X_{o_2}(x) = \{7\} X o 2 ( x ) = { 7 } . Theorem 2.7 (iv) guarantee that a ReLU network’s output value never vanishes across branches, only its derivative can. At a regular input such as x = ( 1 , 1 ) x = (1,1) x = ( 1 , 1 ) , where Z h 1 ( x ) = { ( 1 , 2 , 0 ) } Z_{h_1}(x) = \{(1,2,0)\} Z h 1 ( x ) = {( 1 , 2 , 0 )} and Z h 2 ( x ) = { ( − 1 , 1 , 1 ) } Z_{h_2}(x) = \{(-1,1,1)\} Z h 2 ( x ) = {( − 1 , 1 , 1 )} avoid the switching sets, M ( x ) \mathcal{M}(x) M ( x ) is a singleton and every set in the figure collapses to one number.
For ReLU, the kink and X h 1 ( x ) = { 0 } X_{h_1}(x) = \{0\} X h 1 ( x ) = { 0 } are the same event, so h 1 ∉ V ( x ) h_1 \notin V(x) h 1 ∈ / V ( x ) (Definition 2.6 ): h 1 h_1 h 1 is inactive at this x x x , carrying no forward contribution to either output. This does not shrink the sum defining g i 1 , o 1 g_{i_1,o_1} g i 1 , o 1 : formula (3) sums over the paths of the full graph G G G , not over the smaller active graph G ( x ) G(x) G ( x ) , precisely so that a branch which would carry signal through a currently-inactive node is still represented. Dropping the path through h 1 h_1 h 1 and keeping only the skip edge and the path through h 2 h_2 h 2 – as restricting to G ( x ) G(x) G ( x ) would – gives the single value − 1 -1 − 1 below, not the set that follows; the branch Φ h 1 ( 2 ) \Phi_{h_1}^{(2)} Φ h 1 ( 2 ) is what recovers the second extreme point. The two branches disagree on the Jacobian D Φ h 1 ( κ h 1 ) D\Phi_{h_1}^{(\kappa_{h_1})} D Φ h 1 ( κ h 1 ) entering (3) , 0 0 0 against 1 1 1 , so for v = i 1 v=i_1 v = i 1 , o = o 1 o=o_1 o = o 1 , summing the skip edge and the two hidden paths gives
g i 1 , o 1 ( κ h 1 = 1 ) ( x ) = w i 1 , o 1 + w i 1 , h 1 ⋅ 0 ⋅ w h 1 , o 1 + w i 1 , h 2 ⋅ 1 ⋅ w h 2 , o 1 = − 1 , g i 1 , o 1 ( κ h 1 = 2 ) ( x ) = 1 + 1 ⋅ 1 ⋅ 1 − 2 = 0 , g_{i_1,o_1}^{(\kappa_{h_1}=1)}(x) = w_{i_1,o_1} + w_{i_1,h_1}\cdot0\cdot w_{h_1,o_1} + w_{i_1,h_2}\cdot1\cdot w_{h_2,o_1} = -1,\qquad g_{i_1,o_1}^{(\kappa_{h_1}=2)}(x) = 1+1\cdot1\cdot1-2=0, g i 1 , o 1 ( κ h 1 = 1 ) ( x ) = w i 1 , o 1 + w i 1 , h 1 ⋅ 0 ⋅ w h 1 , o 1 + w i 1 , h 2 ⋅ 1 ⋅ w h 2 , o 1 = − 1 , g i 1 , o 1 ( κ h 1 = 2 ) ( x ) = 1 + 1 ⋅ 1 ⋅ 1 − 2 = 0 ,
so that
F ~ i 1 , o 1 ( x ) = co { ( 6 , − 1 ) , ( 6 , 0 ) } = { 6 } × [ − 1 , 0 ] : \widetilde{\mathcal{F}}_{i_1,o_1}(x)=\operatorname{co}\{(6,-1),(6,0)\}=\{6\}\times[-1,0]: F i 1 , o 1 ( x ) = co {( 6 , − 1 ) , ( 6 , 0 )} = { 6 } × [ − 1 , 0 ] :
the value coordinate degenerates to the single point x o 1 = 6 x_{o_1}=6 x o 1 = 6 – ReLU’s continuity leaves it no room to branch – while the gradient coordinate is a nondegenerate segment, exactly the Clarke–Filippov ambiguity of an automatic-differentiation pass free to resolve the h 1 h_1 h 1 tie either way. The same computation for o 2 o_2 o 2 gives F ~ i 1 , o 2 ( x ) = { 7 } × [ − 3 , − 2 ] \widetilde{\mathcal{F}}_{i_1,o_2}(x)=\{7\}\times[-3,-2] F i 1 , o 2 ( x ) = { 7 } × [ − 3 , − 2 ] .
Backpropagation: Optimal Control
Formula (3) has a second derivation, in the spirit of LeCun [27 ] , which never enumerates paths. Read the evaluation along a branch selection as a constrained problem: the quantity of interest is the output ξ o \xi_o ξ o , while the state ξ = ( ξ v ) v ∈ V \xi=(\xi_v)_{v\in V} ξ = ( ξ v ) v ∈ V is tied by the state equations ξ v = Φ v ( κ v ) ( ζ v ( ξ ) ) \xi_v=\Phi_v^{(\kappa_v)}(\zeta_v(\xi)) ξ v = Φ v ( κ v ) ( ζ v ( ξ )) at the non-input nodes and by the data equations ξ i = x ^ i \xi_i=\hat{x}_i ξ i = x ^ i at the inputs – a discrete-time optimal control problem whose control is the weights, which enter nowhere but in the state equations. Adjoining every constraint with its own multiplier gives the Lagrangian (5) , and the construction pays off twice. On the trajectory all constraint terms vanish, so L ( κ ) = x o ( κ ) ( x ) \mathcal{L}^{(\kappa)}=x_o^{(\kappa)}(x) L ( κ ) = x o ( κ ) ( x ) whatever λ \lambda λ : the multipliers are free, and can be spent on cancelling the first-order effect of the state variations instead of tracking it. Imposing that cancellation is exactly the adjoint equation (6) .
Theorem 2.8. Let x ∈ R d x \in\mathbb{R}^{d} x ∈ R d , o ∈ O o \in\mathcal{O} o ∈ O and κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) (see Definition 2.5 ), and write x ^ i \hat{x}_{i} x ^ i for the value prescribed at an input node, x ^ i : = x i \hat{x}_{i}:=x_{i} x ^ i := x i if i ∈ I ∖ B i \in\mathcal{I} \setminus\mathcal{B} i ∈ I ∖ B and x ^ i : = 1 \hat{x}_{i}:=1 x ^ i := 1 if i ∈ B i \in\mathcal{B} i ∈ B . For state variables ξ = ( ξ v ) v ∈ V \xi=(\xi_{v})_{v\in\mathcal{V}} ξ = ( ξ v ) v ∈ V and multipliers λ = ( λ v ) v ∈ V \lambda=(\lambda_{v})_{v\in\mathcal{V}} λ = ( λ v ) v ∈ V , let
L ( κ ) ( ξ , λ ) : = ξ o + ∑ v ∈ V ∖ I λ v ( Φ v ( κ v ) ( ζ v ( ξ ) ) − ξ v ) + ∑ i ∈ I λ i ( x ^ i − ξ i ) , ζ v ( ξ ) : = ( w a , v ξ a ) a ∈ In ( v ) , (5) \mathcal{L}^{(\kappa)}(\xi,\lambda):=\xi_{o}+\sum_{v\in\mathcal{V}\setminus\mathcal{I}}\lambda_{v}\left(\Phi_{v}^{(\kappa_{v})}\left(\zeta_{v}(\xi)\right)-\xi_{v}\right)+\sum_{i\in\mathcal{I}}\lambda_{i}\left(\hat{x}_{i}-\xi_{i}\right),\qquad\zeta_{v}(\xi):=\left(w_{a,v}\xi_{a}\right)_{a\in\operatorname{In}(v)}, \tag*{(5)} L ( κ ) ( ξ , λ ) := ξ o + v ∈ V ∖ I ∑ λ v ( Φ v ( κ v ) ( ζ v ( ξ ) ) − ξ v ) + i ∈ I ∑ λ i ( x ^ i − ξ i ) , ζ v ( ξ ) := ( w a , v ξ a ) a ∈ In ( v ) , (5) for ξ \xi ξ near ξ ( κ ) : = ( x v ( κ ) ( x ) ) v ∈ V \xi^{(\kappa)}:=\left(x_{v}^{(\kappa)}(x)\right)_{v\in\mathcal{V}} ξ ( κ ) := ( x v ( κ ) ( x ) ) v ∈ V . Then ∂ L ( κ ) / ∂ ξ v ( ξ ( κ ) , λ ) = 0 \partial\mathcal{L}^{(\kappa)}/\partial\xi_{v}\left(\xi^{(\kappa)},\lambda\right)=0 ∂ L ( κ ) / ∂ ξ v ( ξ ( κ ) , λ ) = 0 for every v ∈ V v\in\mathcal{V} v ∈ V if and only if
λ v = 1 { v = o } + ∑ u ∈ Out ( v ) λ u w v , u [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] v , v ∈ V . (6) \lambda_{v}=\mathbf{1}_{\{v=o\}}+\sum_{u\in\operatorname{Out}(v)}\lambda_{u}w_{v,u}\left[D\Phi_{u}^{(\kappa_{u})}\left(z_{u}^{(\kappa)}(x)\right)\right]_{v},\qquad v\in\mathcal{V}. \tag*{(6)} λ v = 1 { v = o } + u ∈ Out ( v ) ∑ λ u w v , u [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] v , v ∈ V . (6) This backward recursion has a unique solution λ ( κ ) \lambda^{(\kappa)} λ ( κ ) , with λ v ( κ ) = g v , o ( κ ) ( x ) \lambda_{v}^{(\kappa)}=g_{v,o}^{(\kappa)}(x) λ v ( κ ) = g v , o ( κ ) ( x ) for every v ∈ V v\in\mathcal{V} v ∈ V . Consequently F v , o ( x ) = co { ( x o ( κ ) ( x ) , λ v ( κ ) ) : κ ∈ M ( x ) } F_{v,o}(x)=\operatorname{co}\left\{\left(x_{o}^{(\kappa)}(x),\lambda_{v}^{(\kappa)}\right):\kappa\in\mathcal{M}(x)\right\} F v , o ( x ) = co { ( x o ( κ ) ( x ) , λ v ( κ ) ) : κ ∈ M ( x ) } , and λ v ( κ ) = ∂ x o / ∂ x v ( x ) \lambda_{v}^{(\kappa)}=\partial x_{o}/\partial x_{v}(x) λ v ( κ ) = ∂ x o / ∂ x v ( x ) if x ∉ D x\notin\mathcal{D} x ∈ / D .
Proof of the theorem is provided in Appendix § 6 .§ 4 . The state variables ξ v \xi_{v} ξ v are free real numbers, distinct from the input x x x , which is fixed: stationarity in λ \lambda λ returns the constraints, that is the forward recursion (1) , whose unique solution is ξ ( κ ) \xi^{(\kappa)} ξ ( κ ) . The Lagrangian is well defined and C 1 C^{1} C 1 near ξ ( κ ) \xi^{(\kappa)} ξ ( κ ) even when x ∈ D x\in\mathcal{D} x ∈ D , because each branch Φ v ( κ v ) \Phi_{v}^{(\kappa_{v})} Φ v ( κ v ) is C 1 C^{1} C 1 on an open neighborhood of z v ( κ ) ( x ) z_{v}^{(\kappa)}(x) z v ( κ ) ( x ) (see Definition 2.1 ). The last sum in (5) enforces the prescribed input and bias values; its multipliers λ i \lambda_{i} λ i are, as usual in optimal control, the sensitivities of the output to those data.
Backpropagation is thus exactly the adjoint equation of the Lagrangian (5) : at every node v v v , the costate λ v \lambda_{v} λ v collects, from each successor u u u , the local Jacobian entry [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] v \left[D\Phi_{u}^{(\kappa_{u})}\left(z_{u}^{(\kappa)}(x)\right)\right]_{v} [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] v scaled by the weight w v , u w_{v,u} w v , u and by the already computed costate λ u \lambda_{u} λ u — one multiplier per node, obtained in a single backward pass, instead of the sum over the possibly exponentially many paths of (3) . Because the weights enter L ( κ ) \mathcal{L}^{(\kappa)} L ( κ ) only through the state equations, the same costates also give, at no further cost, the gradient with respect to every weight — the control variable, and the quantity of actual interest for training.
Corollary 2.9. Let x ∉ D x\notin\mathcal{D} x ∈ / D , o ∈ O o\in\mathcal{O} o ∈ O , κ \kappa κ the unique element of M ( x ) \mathcal{M}(x) M ( x ) and λ = λ ( κ ) \lambda=\lambda^{(\kappa)} λ = λ ( κ ) the solution of (6) . Then, for every edge ( u , v ) ∈ E (u,v)\in\mathcal{E} ( u , v ) ∈ E ,
∂ x o ∂ w u , v ( x ) = λ v x u [ D Φ v ( κ v ) ( z v ( x ) ) ] u = x u δ u , v , \frac{\partial x_{o}}{\partial w_{u,v}}(x)=\lambda_{v}x_{u}\left[D\Phi_{v}^{(\kappa_{v})}\left(z_{v}(x)\right)\right]_{u}=x_{u}\delta_{u,v}, ∂ w u , v ∂ x o ( x ) = λ v x u [ D Φ v ( κ v ) ( z v ( x ) ) ] u = x u δ u , v , where the edge costates δ u , v : = λ v [ D Φ v ( κ v ) ( z v ( x ) ) ] u \delta_{u,v}:=\lambda_{v}\left[D\Phi_{v}^{(\kappa_{v})}\left(z_{v}(x)\right)\right]_{u} δ u , v := λ v [ D Φ v ( κ v ) ( z v ( x ) ) ] u are the unique solution of the backward recursion
δ u , v = [ D Φ v ( κ v ) ( z v ( x ) ) ] u ( 1 { v = o } + ∑ r ∈ Out ( v ) w v , r δ v , r ) , ( u , v ) ∈ E . (7) \delta_{u,v}=\left[D\Phi_{v}^{(\kappa_{v})}\left(z_{v}(x)\right)\right]_{u}\left(\mathbf{1}_{\{v=o\}}+\sum_{r\in\operatorname{Out}(v)}w_{v,r}\delta_{v,r}\right),\qquad(u,v)\in\mathcal{E}. \tag*{(7)} δ u , v = [ D Φ v ( κ v ) ( z v ( x ) ) ] u 1 { v = o } + r ∈ Out ( v ) ∑ w v , r δ v , r , ( u , v ) ∈ E . (7)
Proof of the corollary is provided in Appendix § 6 .§ 5 . The recursion (7) computes every weight gradient in the same backward pass as (6) , which it rewrites edge by edge: the bracket is λ v \lambda_{v} λ v , and λ u = 1 { u = o } + ∑ v ∈ Out ( u ) w u , v δ u , v \lambda_{u}=\mathbf{1}_{\{u=o\}}+\sum_{v\in\operatorname{Out}(u)}w_{u,v}\delta_{u,v} λ u = 1 { u = o } + ∑ v ∈ Out ( u ) w u , v δ u , v . For a neuron Φ v ( ζ ) = σ v ( ∑ a ζ a ) \Phi_{v}(\zeta)=\sigma_{v}\left(\sum_{a}\zeta_{a}\right) Φ v ( ζ ) = σ v ( ∑ a ζ a ) , the Jacobian entry is σ v ′ ( z ~ v ) \sigma_{v}'(\tilde{z}_{v}) σ v ′ ( z ~ v ) for every u u u , so that δ u , v = λ v σ v ′ ( z ~ v ) \delta_{u,v}=\lambda_{v}\sigma_{v}'(\tilde{z}_{v}) δ u , v = λ v σ v ′ ( z ~ v ) is the classical “delta” of backpropagation, one per neuron, and ∂ x o / ∂ w u , v = x u δ v \partial x_{o}/\partial w_{u,v}=x_{u}\delta_{v} ∂ x o / ∂ w u , v = x u δ v .
Relevance Propagation as an Entropic Projection
Relevance flows on the active graph
As outlined in § 1 , classical relevance propagation arises from a single variational principle: the entropic projection (Villani, [48 ] ; Peyré and Cuturi, [36 ] ) of the backward transition law induced by the forward activation flow onto the set of divergence-free relevance flows consistent with the output, a projection of value zero, the flow it selects having exactly that transition law. Specifically, given the active computational graph G ( x ) G(x) G ( x ) induced by an input x x x , we define the admissible set A o ( x ) \mathcal{A}_{o}(x) A o ( x ) of flow relevances as those satisfying a conservation constraint at every hidden node — a discrete analogue of Kirchhoff’s current law — and we show in this section that the classical LRPz + _{z+} z + and LRPα , β _{\alpha,\beta} α , β rules of Bach et al. [6 ] (see also Montavon et al. [30 , 31 , 32 ] for extensions and overviews) are recovered as the unique solutions of relative-entropy minimization problems over this set, the relative entropy being taken between backward transition laws; LRP0 _0 0 is recovered in the same way, its reference being the signed forward contributions z z z themselves. For a non-negative reference and a non-negative flow, this is the relative entropy of two Markov measures on the backward paths of the network (Appendix § 7 .§ 3 ); for a signed reference, it is an extension under a sign constraint, without this reading.
This variational formulation offers three key advantages. First, it provides a rigorous foundation for relevance propagation, replacing the heuristic redistribution rules of classical LRP with a well-posed optimization problem whose solution is unique. Second, it reveals the geometric structure of attribution: the propagation kernels K u , v K_{u,v} K u , v are the transition laws of these minimizers, and the conservation of relevance is a constraint of the problem at every node, from which global conservation follows on the DAG, rather than an assumption. Third, it opens the door to new attribution methods that keep the same conservation constraint: changing the reference measure in the KL divergence changes the rule, and taking the relative entropy edge by edge yields Sinkhorn Relevance Propagation (§ 3.3 ).
Section 3.1 defines the admissible relevance flows and their conservation (Proposition 3.2 ). Section 3.2 proves the projection theorem (Theorem 3.3 ), derives from it LRP0 _0 0 , LRPz + _{z+} z + , LRPα , β _{\alpha,\beta} α , β and LRPNα , β _{\alpha,\beta} α , β (Corollary 3.4), and describes these rules on networks of ReLU and max nodes (Corollary 3.5): Gradient×Input for LRP0 _0 0 , continuity and jumps at a kink and at a change of winner. Section 3.3 introduces SRP, and Section 3.4 compares the rules on VGG-16 and ConvNeXt-Tiny.
We now formally define, for an output node o ∈ O o \in\mathcal{O} o ∈ O , the node relevances μ u o \mu_u^o μ u o together with the associated flow relevances π u , v o \pi_{u,v}^o π u , v o along each active edge of the DAG.
Definition 3.1. Let N F ( G ) \mathcal{N}_{\mathcal{F}}(\mathcal{G}) N F ( G ) be a feed-forward Filippov neural network , x ∈ R d x \in\mathbb{R}^d x ∈ R d and κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) . Write x v : = x v ( κ ) ( x ) x_v := x_v^{(\kappa)}(x) x v := x v ( κ ) ( x ) for the node values along κ \kappa κ and G ( κ ) ( x ) = ( V ( κ ) ( x ) , E ( κ ) ( x ) ) \mathcal{G}^{(\kappa)}(x) = (V^{(\kappa)}(x), E^{(\kappa)}(x)) G ( κ ) ( x ) = ( V ( κ ) ( x ) , E ( κ ) ( x )) for the active graph they determine, as in Definition 2.6 . For an active output o ∈ O o \in\mathcal{O} o ∈ O , define the admissible set
A o ( κ ) ( x ) : = { π o , κ ∈ R E ( κ ) ( x ) : ( div G ( κ ) ( x ) π o , κ ) ( v ) = 0 ∀ v ∈ H ( κ ) ( x ) , \mathcal{A}_o^{(\kappa)}(x) := \{ \pi^{o,\kappa} \in\mathbb{R}^{E^{(\kappa)}(x)} : (\operatorname{div}_{\mathcal{G}^{(\kappa)}(x)} \pi^{o,\kappa})(v) = 0 \ \forall v \in H^{(\kappa)}(x), A o ( κ ) ( x ) := { π o , κ ∈ R E ( κ ) ( x ) : ( div G ( κ ) ( x ) π o , κ ) ( v ) = 0 ∀ v ∈ H ( κ ) ( x ) , ∑ u : u → o π u , o o , κ = x o ( κ ) , ∑ u : u → o ′ π u , o ′ o , κ = 0 ∀ o ′ ∈ O ∖ { o } } \sum_{u:u\to o} \pi_{u,o}^{o,\kappa} = x_o^{(\kappa)}, \qquad\sum_{u:u\to o'} \pi_{u,o'}^{o,\kappa} = 0 \ \forall o' \in\mathcal{O} \setminus\{o\} \} u : u → o ∑ π u , o o , κ = x o ( κ ) , u : u → o ′ ∑ π u , o ′ o , κ = 0 ∀ o ′ ∈ O ∖ { o }} where H ( κ ) ( x ) = V ( κ ) ( x ) ∩ H H^{(\kappa)}(x) = V^{(\kappa)}(x) \cap\mathcal{H} H ( κ ) ( x ) = V ( κ ) ( x ) ∩ H are the hidden active nodes and
( div G ( κ ) ( x ) π o , κ ) ( v ) = ∑ u ∈ V ( κ ) ( x ) : u → v π u , v o , κ − ∑ w ∈ V ( κ ) ( x ) : v → w π v , w o , κ . \left(\operatorname{div}_{\mathcal{G}^{(\kappa)}(x)} \pi^{o,\kappa}\right)(v) =
\sum_{u\in V^{(\kappa)}(x):u\to v} \pi_{u,v}^{o,\kappa}
-
\sum_{w\in V^{(\kappa)}(x):v\to w} \pi_{v,w}^{o,\kappa}. ( div G ( κ ) ( x ) π o , κ ) ( v ) = u ∈ V ( κ ) ( x ) : u → v ∑ π u , v o , κ − w ∈ V ( κ ) ( x ) : v → w ∑ π v , w o , κ . is the divergence operator at node v v v . Both the active graph and the boundary mass x o ( κ ) x_o^{(\kappa)} x o ( κ ) are read off the values along κ \kappa κ , so A o ( κ ) ( x ) \mathcal{A}_o^{(\kappa)}(x) A o ( κ ) ( x ) does depend on the selection; when the node values do not branch at every x ∉ D x \notin D x ∈ / D (Theorem 2.7 (iii)), and at every x x x when the branches agree on the switching sets – it is the same for every selection, and we then drop κ \kappa κ , writing π u , v o \pi_{u,v}^o π u , v o , μ u o \mu_u^o μ u o and x o x_o x o , as we do throughout § 3 –§ 4 . The π u , v o , κ \pi_{u,v}^{o,\kappa} π u , v o , κ are called the flow relevances for the active output o ∈ O o \in\mathcal{O} o ∈ O . They are carried by the active edges alone, no variable being attached to the others. The relevance of a node u ∈ V ∖ O u \in\mathcal{V} \setminus\mathcal{O} u ∈ V ∖ O is their sum over the active edges leaving it,
μ u o , κ = ∑ v ∈ V ( κ ) ( x ) : u → v π u , v o , κ \mu_u^{o,\kappa} = \sum_{v\in V^{(\kappa)}(x):u\to v} \pi_{u,v}^{o,\kappa} μ u o , κ = v ∈ V ( κ ) ( x ) : u → v ∑ π u , v o , κ an empty sum, hence μ u o , κ = 0 \mu_u^{o,\kappa}=0 μ u o , κ = 0 , at a node from which no active edge leaves – in particular at every such node outside V ( κ ) ( x ) V^{(\kappa)}(x) V ( κ ) ( x ) . The relevance of an output is the sum of the flows entering it: μ o o , κ = x o ( κ ) \mu_o^{o,\kappa}=x_o^{(\kappa)} μ o o , κ = x o ( κ ) , and μ o ′ o , κ = 0 \mu_{o'}^{o,\kappa}=0 μ o ′ o , κ = 0 for $o' \ne o.
Figure 4 illustrates these notions on the running example of Figure 3 , at the same input x = ( − 2 , 1 ) x=(-2,1) x = ( − 2 , 1 ) and for the output o 1 o_1 o 1 . The kink node is the one casualty: X h 1 ( x ) = { 0 } X_{h_1}(x)=\{0\} X h 1 ( x ) = { 0 } , so h 1 ∉ V ( x ) h_1 \notin V(x) h 1 ∈ / V ( x ) and the five edges it carries leave E ( x ) E(x) E ( x ) , together with the zero-weight edges b 1 → h 1 b_1\to h_1 b 1 → h 1 and b 2 → o 1 b_2\to o_1 b 2 → o 1 . On what remains, a relevance flow π u , v o 1 , κ \pi^{o_1,\kappa}_{u,v} π u , v o 1 , κ is attached to every active edge, the relevance of a non-output node is the sum of the flows leaving it, μ u o 1 , κ = ∑ v ∈ V ( κ ) ( x ) : u → v π u , v o 1 , κ \mu^{o_1,\kappa}_u=\sum_{v\in V^{(\kappa)}(x):u\to v}\pi^{o_1,\kappa}_{u,v} μ u o 1 , κ = ∑ v ∈ V ( κ ) ( x ) : u → v π u , v o 1 , κ , the flows entering o 1 o_1 o 1 add up to its value, π h 2 , o 1 o 1 , κ + π i 1 , o 1 o 1 , κ = x o 1 ( κ ) = 6 \pi^{o_1,\kappa}_{h_2,o_1}+\pi^{o_1,\kappa}_{i_1,o_1}=x^{(\kappa)}_{o_1}=6 π h 2 , o 1 o 1 , κ + π i 1 , o 1 o 1 , κ = x o 1 ( κ ) = 6 , and no flow enters the other output: μ b 2 o 1 , κ = 0 \mu^{o_1,\kappa}_{b_2}=0 μ b 2 o 1 , κ = 0 , its only active edge feeding o 2 o_2 o 2 . Here M ( x ) \mathcal{M}(x) M ( x ) has two elements: the outputs being linear and h 2 h_2 h 2 active, κ o 1 = κ o 2 = 1 \kappa_{o_1}=\kappa_{o_2}=1 κ o 1 = κ o 2 = 1 and κ h 2 = 2 \kappa_{h_2}=2 κ h 2 = 2 are forced, while κ h 1 ∈ { 1 , 2 } \kappa_{h_1}\in\{1,2\} κ h 1 ∈ { 1 , 2 } is free at the kink. The two branches of h 1 h_1 h 1 agreeing in value, both selections give the same node values, hence the same active graph and the same flows: the figure holds for either.
Figure 4. Relevance flows and node relevances for the output o 1 o_1 o 1 , on the network of Figure 3 at x = ( − 2 , 1 ) x=(-2,1) x = ( − 2 , 1 ) , for either selection κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) .
Any admissible flow in A o ( x ) A_o(x) A o ( x ) already conserves the output relevance globally:
Proposition 3.2. Let N F ( G ) \mathcal{N}_{F}(\mathcal{G}) N F ( G ) be a feed-forward Filippov neural network, x ∈ R d x\in\mathbb{R}^{d} x ∈ R d , κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) and o ∈ O o\in\mathcal{O} o ∈ O . Every admissible relevance flow π o , κ ∈ A o ( κ ) ( x ) \pi^{o,\kappa}\in A_o^{(\kappa)}(x) π o , κ ∈ A o ( κ ) ( x ) satisfies
∑ u ∈ I μ u o , κ = x o ( κ ) ( x ) . \sum_{u\in\mathcal{I}}\mu^{o,\kappa}_u=x_o^{(\kappa)}(x). u ∈ I ∑ μ u o , κ = x o ( κ ) ( x ) .
Proof of the proposition is provided in Appendix § 7 .§ 1 . An input outside I ( x ) I(x) I ( x ) has x u = 0 x_u=0 x u = 0 , hence no active outgoing edge and, by Definition 3.1 , μ u o , κ = 0 \mu^{o,\kappa}_u=0 μ u o , κ = 0 ; so ∑ u ∈ I μ u o , κ = ∑ u ∈ I ( x ) μ u o , κ \sum_{u\in\mathcal{I}}\mu^{o,\kappa}_u=\sum_{u\in I(x)}\mu^{o,\kappa}_u ∑ u ∈ I μ u o , κ = ∑ u ∈ I ( x ) μ u o , κ : the conservation law is equivalently stated over the active input nodes I ( x ) I(x) I ( x ) (Definition 2.6 ) – the form used from here on, with κ \kappa κ dropped as agreed whenever the node values do not branch, so that the right-hand side is simply x o x_o x o .
Consequently, the relevance associated with output o o o is neither created nor destroyed inside the network; it is only redistributed among the active paths of G ( x ) G(x) G ( x ) .
The entropic projection and the classical rules
We fix an input x x x , a selection κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) and an active output o ∈ O o\in\mathcal{O} o ∈ O ; the node values, the active graph and the admissible sets are those of Definition 3.1 ; we keep κ \kappa κ in the statements of Theorem 3.3 and Corollary 3.4 and drop it elsewhere, as agreed there. For a ∈ R a\in\mathbb{R} a ∈ R we write a = a + − a − a=a^+-a^- a = a + − a − with a ± : = max ( ± a , 0 ) a^\pm:=\max(\pm a,0) a ± := max ( ± a , 0 ) , so that the edge contributions z u , v = w u , v x u z_{u,v}=w_{u,v}x_u z u , v = w u , v x u split into two non-negative measures z ± = ( z u , v ± ) ( u , v ) ∈ E ( x ) z^\pm=(z^\pm_{u,v})_{(u,v)\in E(x)} z ± = ( z u , v ± ) ( u , v ) ∈ E ( x ) on the active graph, and we write z ~ v ± : = ∑ u : u → v z u , v ± \widetilde{z}_v^\pm:=\sum_{u:u\to v}z^\pm_{u,v} z v ± := ∑ u : u → v z u , v ± .
A relevance propagation rule is a backward transition law : at every node v v v , it splits the relevance μ v \mu_v μ v of v v v among its predecessors, following a reference ρ ∈ R E ( x ) \rho\in\mathbb{R}^{E(x)} ρ ∈ R E ( x ) read off the forward pass, of either sign. Theorem 3.3 characterizes this rule as the unique admissible flow with zero relative entropy to the reference, the entropy being taken between backward transition laws rather than edge by edge. For a flow π ∈ R E ( x ) \pi\in\mathbb{R}^{E(x)} π ∈ R E ( x ) we write m v ( π ) : = ∑ u : u → v π u , v m_v(\pi):=\sum_{u:u\to v}\pi_{u,v} m v ( π ) := ∑ u : u → v π u , v and n v ( π ) : = ∑ u : u → v ∣ π u , v ∣ n_v(\pi):=\sum_{u:u\to v}|\pi_{u,v}| n v ( π ) := ∑ u : u → v ∣ π u , v ∣ for its inflow and absolute inflow at v v v .
Theorem 3.3. Let x ∈ R d x \in\mathbb{R}^{d} x ∈ R d , κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , o ∈ O o \in\mathcal{O} o ∈ O an active output, and ρ ∈ R E ( κ ) ( x ) \rho\in\mathbb{R}^{E^{(\kappa)}(x)} ρ ∈ R E ( κ ) ( x ) with ρ ~ v : = ∑ u : u → v ρ u , v ≠ 0 \widetilde{\rho}_{v} := \sum_{u:u\to v} \rho_{u,v} \ne0 ρ v := ∑ u : u → v ρ u , v = 0 for every v ∈ H ( κ ) ( x ) ∪ { o } v \in H^{(\kappa)}(x) \cup\{o\} v ∈ H ( κ ) ( x ) ∪ { o } . Define the kernel of ρ \rho ρ and its absolute law,
K u , v : = ρ u , v ρ ~ v , K ‾ u , v : = ∣ ρ u , v ∣ ∑ r : r → v ∣ ρ r , v ∣ , K_{u,v} := \frac{\rho_{u,v}}{\widetilde{\rho}_{v}}, \qquad\overline{K}_{u,v} := \frac{|\rho_{u,v}|}{\sum_{r:r\to v}|\rho_{r,v}|}, K u , v := ρ v ρ u , v , K u , v := ∑ r : r → v ∣ ρ r , v ∣ ∣ ρ u , v ∣ , both set to 0 0 0 if ρ ~ v = 0 \widetilde{\rho}_{v}=0 ρ v = 0 , and the conditional relative entropy of π ∈ R E ( κ ) ( x ) \pi\in\mathbb{R}^{E^{(\kappa)}(x)} π ∈ R E ( κ ) ( x ) ,
H ( π ∥ K ) : = ∑ v : n v ( π ) ≠ 0 n v ( π ) KL ( ∣ π ⋅ , v ∣ n v ( π ) ∥ K ‾ ⋅ , v ) (8) \mathcal{H}(\pi\mathbin{\|}K) := \sum_{v:n_{v}(\pi)\ne0} n_{v}(\pi)\,\operatorname{KL}\left(\frac{|\pi_{\cdot,v}|}{n_{v}(\pi)}\mathbin{\|}\overline{K}_{\cdot,v}\right) \tag*{(8)} H ( π ∥ K ) := v : n v ( π ) = 0 ∑ n v ( π ) KL ( n v ( π ) ∣ π ⋅ , v ∣ ∥ K ⋅ , v ) (8) if π \pi π is sign-coherent with K K K – at every node v v v there is σ v ∈ { − 1 , 1 } \sigma_{v} \in\{-1,1\} σ v ∈ { − 1 , 1 } with σ v π u , v K u , v ≥ 0 \sigma_{v}\pi_{u,v}K_{u,v} \ge0 σ v π u , v K u , v ≥ 0 for every edge ( u , v ) ∈ E ( κ ) ( x ) (u,v) \in E^{(\kappa)}(x) ( u , v ) ∈ E ( κ ) ( x ) entering v v v – and H ( π ∥ K ) : = + ∞ \mathcal{H}(\pi\mathbin{\|}K):=+\infty H ( π ∥ K ) := + ∞ otherwise. Then the problem
min π o , κ ∈ A o ( κ ) ( x ) H ( π o , κ ∥ K ) \min_{\pi^{o,\kappa}\in A^{(\kappa)}_{o}(x)} \mathcal{H}(\pi^{o,\kappa}\mathbin{\|}K) π o , κ ∈ A o ( κ ) ( x ) min H ( π o , κ ∥ K ) has a unique minimizer π o , κ , ∗ \pi^{o,\kappa,*} π o , κ , ∗ , its minimal value is 0 0 0 , and
π u , v o , κ , ∗ = K u , v μ v o , κ , ( u , v ) ∈ E ( κ ) ( x ) , \pi^{o,\kappa,*}_{u,v}=K_{u,v}\mu^{o,\kappa}_{v}, \qquad(u,v)\in E^{(\kappa)}(x), π u , v o , κ , ∗ = K u , v μ v o , κ , ( u , v ) ∈ E ( κ ) ( x ) , where the node relevances satisfy the backward recursion
μ o o , κ = x o ( κ ) , μ o ′ o , κ = 0 ( o ′ ∈ O ∖ { o } ) , μ u o , κ = ∑ v : u → v K u , v μ v o , κ ( u ∈ V ( κ ) ( x ) ∖ O ) . \mu^{o,\kappa}_{o}=x^{(\kappa)}_{o}, \qquad\mu^{o,\kappa}_{o'}=0\ (o'\in\mathcal{O}\setminus\{o\}), \qquad\mu^{o,\kappa}_{u}=\sum_{v:u\to v}K_{u,v}\mu^{o,\kappa}_{v}\quad(u\in V^{(\kappa)}(x)\setminus\mathcal{O}). μ o o , κ = x o ( κ ) , μ o ′ o , κ = 0 ( o ′ ∈ O ∖ { o }) , μ u o , κ = v : u → v ∑ K u , v μ v o , κ ( u ∈ V ( κ ) ( x ) ∖ O ) . In particular, ∑ u ∈ I ( x ) μ u o , κ = x o ( κ ) \sum_{u\in I(x)}\mu^{o,\kappa}_{u}=x^{(\kappa)}_{o} ∑ u ∈ I ( x ) μ u o , κ = x o ( κ ) . Letting κ \kappa κ range over M ( x ) \mathcal{M}(x) M ( x ) , the relevance of a node u u u is the set co { μ u o , κ : κ ∈ M ( x ) } \operatorname{co}\{\mu^{o,\kappa}_{u}:\kappa\in\mathcal{M}(x)\} co { μ u o , κ : κ ∈ M ( x )} , a singleton whenever neither the node values nor the reference depend on κ \kappa κ .
Proof of the theorem is provided in Appendix § 7 .§ 2 . Both ingredients of (8) are needed for a signed reference: with K = ( 2 , − 1 ) K=(2,-1) K = ( 2 , − 1 ) and inflow 1 1 1 , the naive form ∑ ∣ π u , v ∣ log ( ∣ π u , v ∣ / ( ∣ K u , v ∣ ∣ m v ( π ) ∣ ) ) \sum|\pi_{u,v}|\log\left(|\pi_{u,v}|/(|K_{u,v}||m_{v}(\pi)|)\right) ∑ ∣ π u , v ∣ log ( ∣ π u , v ∣/ ( ∣ K u , v ∣∣ m v ( π ) ∣ ) ) is − log 3 < 0 -\log3<0 − log 3 < 0 at π = ( 2 / 3 , 1 / 3 ) \pi=(2/3,1/3) π = ( 2/3 , 1/3 ) . Normalizing ∣ K ∣ |K| ∣ K ∣ into K ‾ \overline{K} K restores Gibbs’ inequality, and sign coherence excludes ( 2 / 3 , 1 / 3 ) (2/3,1/3) ( 2/3 , 1/3 ) , leaving ( 2 , − 1 ) = K (2,-1)=K ( 2 , − 1 ) = K as the only flow of inflow 1 1 1 with zero entropy.
The theorem turns the choice of a rule into the choice of a reference. It characterizes the flow once the reference is fixed, and only through the kernel K K K : rescaling the column ρ ⋅ , v \rho_{\cdot,v} ρ ⋅ , v of a node by a nonzero constant changes neither K K K nor the minimizer. It does not select a canonical reference, nor guarantee the faithfulness of the result, which § 3.4 measures. The minimizer rescales ρ \rho ρ at the head of each edge only, π u , v = ρ u , v μ v / ρ ~ v \pi_{u,v}=\rho_{u,v}\mu_{v}/\widetilde{\rho}_{v} π u , v = ρ u , v μ v / ρ v : the entropy fixes the proportions ∣ π u , v ∣ / n v ( π ) |\pi_{u,v}|/n_{v}(\pi) ∣ π u , v ∣/ n v ( π ) to those of ∣ ρ u , v ∣ |\rho_{u,v}| ∣ ρ u , v ∣ , and the coherence constraint fixes the signs, relevance of the sign of μ v \mu_{v} μ v flowing back along the edges where ρ \rho ρ has the sign of ρ ~ v \widetilde{\rho}_{v} ρ v , and of the opposite sign along the others. A non-negative reference therefore yields relevances of the sign of x o x_{o} x o , a signed one relevances of both signs. The natural references are built from the forward contributions z z z and their sign channels z ± z^{\pm} z ± , and the three classical rules are three ways of weighting these channels; we set z ~ v : = ∑ u : u → v z u , v = z ~ v + − z ~ v − \widetilde{z}_{v}:=\sum_{u:u\to v}z_{u,v}=\widetilde{z}^{+}_{v}-\widetilde{z}^{-}_{v} z v := ∑ u : u → v z u , v = z v + − z v − .
Corollary 3.4. Let x ∈ R d x \in\mathbb{R}^{d} x ∈ R d , κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) and o ∈ O o \in\mathcal{O} o ∈ O an active output, with contributions z u , v : = w u , v x u ( κ ) z_{u,v}:=w_{u,v}x^{(\kappa)}_{u} z u , v := w u , v x u ( κ ) on E ( κ ) ( x ) E^{(\kappa)}(x) E ( κ ) ( x ) . The minimizer π o , κ , ∗ \pi^{o,\kappa,*} π o , κ , ∗ of Theorem 3.3 is, for each of the following references ρ \rho ρ (those of (iii) and (iv) being given on the edges entering H ( κ ) ( x ) ∪ { o } H^{(\kappa)}(x)\cup\{o\} H ( κ ) ( x ) ∪ { o } , and 0 0 0 on the edges entering the other outputs):
(i) LRP0 _0 0 . If ρ = z \rho=z ρ = z and z ~ v ≠ 0 \widetilde{z}_{v}\ne0 z v = 0 for every v ∈ H ( κ ) ( x ) ∪ { o } v\in H^{(\kappa)}(x)\cup\{o\} v ∈ H ( κ ) ( x ) ∪ { o } , the minimizer is
π u , v o , κ , ∗ = z u , v z ~ v μ v o , κ . \pi^{o,\kappa,*}_{u,v}=\frac{z_{u,v}}{\widetilde{z}_{v}}\mu^{o,\kappa}_{v}. π u , v o , κ , ∗ = z v z u , v μ v o , κ . If moreover z ~ o ′ ≠ 0 \widetilde{z}_{o'}\ne0 z o ′ = 0 for every o ′ ∈ O ∖ { o } o'\in\mathcal{O}\setminus\{o\} o ′ ∈ O ∖ { o } , then, for π o , κ ∈ A o ( κ ) ( x ) \pi^{o,\kappa}\in A^{(\kappa)}_{o}(x) π o , κ ∈ A o ( κ ) ( x ) ,
H ( π o , κ ∥ K ) = ∑ v : n v ( π o , κ ) ≠ 0 n v ( π o , κ ) KL ( ∣ π ⋅ , v o , κ ∣ n v ( π o , κ ) ∥ ∣ z ⋅ , v ∣ ∑ r : r → v ∣ z r , v ∣ ) \mathcal{H}(\pi^{o,\kappa}\mathbin{\|}K)=\sum_{v:n_{v}(\pi^{o,\kappa})\ne0}n_{v}(\pi^{o,\kappa})\operatorname{KL}\left(\frac{|\pi^{o,\kappa}_{\cdot,v}|}{n_{v}(\pi^{o,\kappa})}\mathbin{\Big\|}\frac{|z_{\cdot,v}|}{\sum_{r:r\to v}|z_{r,v}|}\right) H ( π o , κ ∥ K ) = v : n v ( π o , κ ) = 0 ∑ n v ( π o , κ ) KL ( n v ( π o , κ ) ∣ π ⋅ , v o , κ ∣ ∑ r : r → v ∣ z r , v ∣ ∣ z ⋅ , v ∣ )
when, at every v v v , the products π u , v o , κ z u , v \pi^{o,\kappa}_{u,v}z_{u,v} π u , v o , κ z u , v over the edges ( u , v ) ∈ E ( κ ) ( x ) (u,v) \in E^{(\kappa)}(x) ( u , v ) ∈ E ( κ ) ( x ) entering v v v have a common sign, and H ( π o , κ ∥ K ) = + ∞ \mathcal{H}(\pi^{o,\kappa}\|K) = +\infty H ( π o , κ ∥ K ) = + ∞ otherwise.
(ii) LRPz + _{z+} z + . If ρ = z + \rho= z^+ ρ = z + and z ~ v + > 0 \widetilde{z}^+_v > 0 z v + > 0 for every v ∈ H ( κ ) ( x ) ∪ { o } v \in H^{(\kappa)}(x) \cup\{o\} v ∈ H ( κ ) ( x ) ∪ { o } , the minimizer is
π u , v o , κ , ∗ = z u , v + z ~ v + μ v o , κ . \pi^{o,\kappa,*}_{u,v} = \frac{z^+_{u,v}}{\widetilde{z}^+_v}\mu^{o,\kappa}_v. π u , v o , κ , ∗ = z v + z u , v + μ v o , κ .
(iii) LRPα , β _{\alpha,\beta} α , β . If α , β ∈ R \alpha, \beta\in\mathbb{R} α , β ∈ R with α + β = 1 \alpha+ \beta= 1 α + β = 1 , ρ u , v = α z u , v + / z ~ v + + β z u , v − / z ~ v − \rho_{u,v} = \alpha z^+_{u,v}/\widetilde{z}^+_v + \beta z^-_{u,v}/\widetilde{z}^-_v ρ u , v = α z u , v + / z v + + β z u , v − / z v − , and z ~ v + > 0 \widetilde{z}^+_v > 0 z v + > 0 , z ~ v − > 0 \widetilde{z}^-_v > 0 z v − > 0 for every v ∈ H ( κ ) ( x ) ∪ { o } v \in H^{(\kappa)}(x) \cup\{o\} v ∈ H ( κ ) ( x ) ∪ { o } , the minimizer is
π u , v o , κ , ∗ = ( α z u , v + z ~ v + + β z u , v − z ~ v − ) μ v o , κ . \pi^{o,\kappa,*}_{u,v} = \left(\alpha\frac{z^+_{u,v}}{\widetilde{z}^+_v} + \beta\frac{z^-_{u,v}}{\widetilde{z}^-_v}\right)\mu^{o,\kappa}_v. π u , v o , κ , ∗ = ( α z v + z u , v + + β z v − z u , v − ) μ v o , κ .
(iv) LRPN α , β _{\mathrm{N}\alpha,\beta} N α , β (LRPα , β _{\alpha,\beta} α , β with node-wise weights). If, for every v ∈ H ( κ ) ( x ) ∪ { o } v \in H^{(\kappa)}(x) \cup\{o\} v ∈ H ( κ ) ( x ) ∪ { o } , α v , β v ∈ R \alpha_v,\beta_v \in\mathbb{R} α v , β v ∈ R (which may depend on x x x ) satisfy α v + β v = 1 \alpha_v + \beta_v = 1 α v + β v = 1 and z ~ v + + z ~ v − > 0 \widetilde{z}^+_v + \widetilde{z}^-_v > 0 z v + + z v − > 0 , and
ρ u , v = { α v z u , v + / z ~ v + + β v z u , v − / z ~ v − if z ~ v + > 0 and z ~ v − > 0 , z u , v + / z ~ v + if z ~ v − = 0 , z u , v − / z ~ v − if z ~ v + = 0 , \rho_{u,v} =
\begin{cases}
\alpha_v z^+_{u,v}/\widetilde{z}^+_v + \beta_v z^-_{u,v}/\widetilde{z}^-_v & \text{if } \widetilde{z}^+_v > 0 \text{ and } \widetilde{z}^-_v > 0,\\
z^+_{u,v}/\widetilde{z}^+_v & \text{if } \widetilde{z}^-_v = 0,\\
z^-_{u,v}/\widetilde{z}^-_v & \text{if } \widetilde{z}^+_v = 0,
\end{cases} ρ u , v = ⎩ ⎨ ⎧ α v z u , v + / z v + + β v z u , v − / z v − z u , v + / z v + z u , v − / z v − if z v + > 0 and z v − > 0 , if z v − = 0 , if z v + = 0 ,
then ρ ~ v = 1 \widetilde{\rho}_v = 1 ρ v = 1 , and the minimizer is
π u , v o , κ , ∗ = ρ u , v μ v o , κ . \pi^{o,\kappa,*}_{u,v} = \rho_{u,v}\mu^{o,\kappa}_v. π u , v o , κ , ∗ = ρ u , v μ v o , κ .
Proof of the corollary is provided in Appendix § 7 .§ 4 .
The three references are of the form ρ u , v = c v + z u , v + − c v − z u , v − \rho_{u,v} = c^+_v z^+_{u,v} - c^-_v z^-_{u,v} ρ u , v = c v + z u , v + − c v − z u , v − , with ( c v + , c v − ) = ( 1 , 1 ) (c^+_v,c^-_v) = (1,1) ( c v + , c v − ) = ( 1 , 1 ) , ( 1 , 0 ) (1,0) ( 1 , 0 ) and ( α / z ~ v + , − β / z ~ v − ) (\alpha/\widetilde{z}^+_v,-\beta/\widetilde{z}^-_v) ( α / z v + , − β / z v − ) ; for β ≤ 0 ≤ α \beta\le0 \le\alpha β ≤ 0 ≤ α they are the rules of Bach et al. [6 ] ; Montavon et al. [30 , 32 ] , extended to the general active DAG G ( x ) G(x) G ( x ) . Case (iv) contains the other three wherever they apply: (iii) is ( α v , β v ) = ( α , β ) (\alpha_v,\beta_v) = (\alpha,\beta) ( α v , β v ) = ( α , β ) at every node, (ii) is ( α v , β v ) = ( 1 , 0 ) (\alpha_v,\beta_v) = (1,0) ( α v , β v ) = ( 1 , 0 ) , and (i) is ( α v , β v ) = ( z ~ v + / z ~ v , − z ~ v − / z ~ v ) (\alpha_v,\beta_v) = (\widetilde{z}^+_v/\widetilde{z}_v,-\widetilde{z}^-_v/\widetilde{z}_v) ( α v , β v ) = ( z v + / z v , − z v − / z v ) . Only (ii) yields relevances of the sign of x o x_o x o for every x x x ; (iv) does so whenever α v , β v ∈ [ 0 , 1 ] \alpha_v,\beta_v \in[0,1] α v , β v ∈ [ 0 , 1 ] , the reference being then non-negative.
Weights that depend on x x x give case (iv) a Hebbian-inspired reading. If α v \alpha_v α v grows with the activity x v x_v x v of v v v , then, at a given relevance μ v ≥ 0 \mu_v \ge0 μ v ≥ 0 and given channel shares z u , v + / z ~ v + z^+_{u,v}/\widetilde{z}^+_v z u , v + / z v + , a more active node sends a larger share α v \alpha_v α v of its relevance to its excitatory contributions z u , v + = ( w u , v x u ) + z^+_{u,v} = (w_{u,v}x_u)^+ z u , v + = ( w u , v x u ) + , and a weakly activated node a larger share β v \beta_v β v to its inhibitory ones, much as Hebb’s principle Hebb, [22 ] favours the connections whose two ends are active together. The analogy concerns this split only: the relevance of an edge is not monotone in the product of the activities of its two ends, the shares and μ v \mu_v μ v depending on x x x too, and no weight is updated. § 3.4 takes α v = x ˉ v \alpha_v = \bar{x}_v α v = x ˉ v , the value of v v v divided by the largest value of its layer.
Case (iii) requires every node to mix excitatory and inhibitory contributions, an assumption the LRPα , β _{\alpha,\beta} α , β literature leaves implicit; without it, the classical rule does not conserve relevance. Where z ~ v − = 0 \widetilde{z}^-_v = 0 z v − = 0 , the inhibitory term of (iii) is undefined; implementations drop it, keeping the reference ρ u , v = α z u , v + / z ~ v + \rho_{u,v} = \alpha z^+_{u,v}/\widetilde{z}^+_v ρ u , v = α z u , v + / z v + and the classical flow π = ρ μ \pi= \rho\mu π = ρ μ without normalizing it: its kernel sums to α \alpha α at v v v , so that v v v receives α μ v \alpha\mu_v α μ v and passes on μ v \mu_v μ v : the share β μ v \beta\mu_v β μ v is created or destroyed, and the flow leaves A o ( x ) \mathcal{A}_o(x) A o ( x ) . Symmetrically, where z ~ v + = 0 \widetilde{z}^+_v = 0 z v + = 0 , the kernel sums to β \beta β and the share α μ v \alpha\mu_v α μ v is created or destroyed. Being proportional to K = ρ / ρ ~ K = \rho/\widetilde{\rho} K = ρ / ρ at v v v , this flow still has H ( π ∥ K ) = 0 \mathcal{H}(\pi\|K) = 0 H ( π ∥ K ) = 0 ; the minimization over A o ( x ) \mathcal{A}_o(x) A o ( x ) discards it and selects LRPN α , β _{\mathrm{N}\alpha,\beta} N α , β with ( α v , β v ) = ( α , β ) (\alpha_v,\beta_v) = (\alpha,\beta) ( α v , β v ) = ( α , β ) , whose reference at such a node is the non-empty channel normalized, z u , v + / z ~ v + z^+_{u,v}/\widetilde{z}^+_v z u , v + / z v + (resp. z u , v − / z ~ v − z^-_{u,v}/\widetilde{z}^-_v z u , v − / z v − ): it sums to 1 and does not depend on ( α , β ) (\alpha,\beta) ( α , β ) . Relevance is then conserved, and LRPN α , β _{\mathrm{N}\alpha,\beta} N α , β falls back to LRPz + _{z+} z + (resp. to its inhibitory counterpart) at such nodes. In Figure 5 , at x = ( − 2 , 1 ) x = (-2,1) x = ( − 2 , 1 ) , h 2 h_2 h 2 receives only the contributions 2, 1, 1: the classical rule LRP2 , − 1 _{2,-1} 2 , − 1 sends 24 to its parents and the input total becomes 18, whereas LRPN 2 , − 1 _{\mathrm{N}2,-1} N 2 , − 1 splits μ h 2 = 12 \mu_{h_2} = 12 μ h 2 = 12 as 6, 3, 3, and the inputs receive 0, 3, 3, adding up to x o 1 = 6 x_{o_1} = 6 x o 1 = 6 .
Figure 5. LRP N 2 , − 1 \operatorname{LRP}_{N_{2,-1}} LRP N 2 , − 1 relevances for the output o 1 o_1 o 1 , on the network of Figure 3 at x = ( − 2 , 1 ) x=(-2,1) x = ( − 2 , 1 ) , for either selection κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) ; at h 2 h_2 h 2 , which has no inhibitory contribution, the rule falls back to LRP z + \operatorname{LRP}_{z+} LRP z + .
At a ReLU kink the node values do not branch, so that the relevance is a single vector; at a tie of a max node they do not branch either, but the winning edge, hence the reference, does. The behaviour of LRP 0 \operatorname{LRP}_0 LRP 0 , LRP z + \operatorname{LRP}_{z+} LRP z + and LRP N α , β \operatorname{LRP}_{N_{\alpha,\beta}} LRP N α , β near a kink and near a tie is given by Corollary 3.5(iii) and (iv).
In the following, we fix a single active output o ∈ O o\in O o ∈ O and omit it from the superscripts for readability; the dependence on o o o remains explicit through P ( u , o ) \mathcal{P}(u,o) P ( u , o ) wherever it appears.
Corollary 3.5. Let N F ( G ) N_F(G) N F ( G ) be a feed-forward Filippov neural network whose activations are all linear, ReLU or max, a max node v v v taking the value x v = max u : u → v w u , v x u x_v=\max_{u:u\to v}w_{u,v}x_u x v = max u : u → v w u , v x u , x ∈ R d x\in\mathbb{R}^d x ∈ R d an input at which the node values do not branch (see Theorem 2.7 (iv)), o ∈ O o\in O o ∈ O an active output, and κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , which picks at every max node v v v a predecessor κ v \kappa_v κ v with z κ v , v = x v z_{\kappa_v,v}=x_v z κ v , v = x v . With z ~ v : = ∑ u : u → v w u , v x u \widetilde{z}_v:=\sum_{u:u\to v}w_{u,v}x_u z v := ∑ u : u → v w u , v x u over all the edges of G G G entering v v v , let the gate η u , v ( x ) \eta_{u,v}(x) η u , v ( x ) of an edge be 1 1 1 if v v v is linear, ReLU with z ~ v > 0 \widetilde{z}_v>0 z v > 0 , or max with u = κ v u=\kappa_v u = κ v , and 0 0 0 otherwise, W u ( x ) W_u(x) W u ( x ) the sum over the paths of P ( u , o ) \mathcal{P}(u,o) P ( u , o ) of the products of w a , b η a , b ( x ) w_{a,b}\eta_{a,b}(x) w a , b η a , b ( x ) along them, and W v u ( x ) W_v^u(x) W v u ( x ) the same sum over P ( u , v ) \mathcal{P}(u,v) P ( u , v ) without the gate of the last edge. Let ρ \rho ρ be the reference of LRP 0 \operatorname{LRP}_0 LRP 0 , LRP z + \operatorname{LRP}_{z+} LRP z + or LRP N α , β \operatorname{LRP}_{N_{\alpha,\beta}} LRP N α , β , with weights continuous in x x x (see Corollary 3.4(i), (ii), (iv)), on the edges entering the linear and ReLU nodes, and ρ u , v : = z u , v \rho_{u,v}:=z_{u,v} ρ u , v := z u , v if u = κ v u=\kappa_v u = κ v , 0 0 0 otherwise, on the edges entering a max node v v v , at inputs where the hypotheses of that rule hold. A point x ˉ \bar{x} x ˉ is a simple switching point of a node v v v if z ~ r ( x ˉ ) ≠ 0 \widetilde{z}_r(\bar{x})\ne0 z r ( x ˉ ) = 0 at every linear or ReLU hidden node r ≠ v r\ne v r = v , and every max node r ≠ v r\ne v r = v has at x ˉ \bar{x} x ˉ a single maximizer, or the maximum 0 0 0 attained only at ReLU nodes r ′ r' r ′ with z ~ r ′ ( x ˉ ) ≠ 0 \widetilde{z}_{r'}(\bar{x})\ne0 z r ′ ( x ˉ ) = 0 .
(i) On V ( x ) ∖ ( O ∖ { o } ) V(x)\setminus(O\setminus\{o\}) V ( x ) ∖ ( O ∖ { o }) , μ u = x u W u u \mu_u=x_uW_u^u μ u = x u W u u , where W o o = 1 W_o^o=1 W o o = 1 and W u v = ∑ v ∈ H ( x ) ∪ { o } : u → v ω u , v s v W v u W_u^v=\sum_{v\in H(x)\cup\{o\}:u\to v}\omega_{u,v}s_vW_v^u W u v = ∑ v ∈ H ( x ) ∪ { o } : u → v ω u , v s v W v u , with ω u , v : = ρ u , v / x u \omega_{u,v}:=\rho_{u,v}/x_u ω u , v := ρ u , v / x u and s v : = x v / ρ ~ v s_v:=x_v/\widetilde{\rho}_v s v := x v / ρ v . For LRP 0 \operatorname{LRP}_0 LRP 0 , if z ~ v ≠ 0 \widetilde{z}_v\ne0 z v = 0 at every linear hidden node v v v , W ρ = W W^\rho=W W ρ = W , and W u ( x ) = ∂ x o / ∂ x u ( x ) W_u(x)=\partial x_o/\partial x_u(x) W u ( x ) = ∂ x o / ∂ x u ( x ) if x ∉ D x\notin D x ∈ / D (Gradient× \times × Input).
(ii) On an open set of inputs where the signs of the node values and of the contributions are constant, and so is κ \kappa κ , μ \mu μ is continuous, and, for LRP 0 \operatorname{LRP}_0 LRP 0 with z ~ v ≠ 0 \widetilde{z}_v\ne0 z v = 0 at every linear hidden node v v v , μ u = x u W u \mu_u=x_uW_u μ u = x u W u with W W W constant. If the hidden nodes are ReLU, or max of non-negative value, and x o > 0 x_o>0 x o > 0 , LRP z + \operatorname{LRP}_{z+} LRP z + satisfies 0 ≤ μ u ≤ ∣ x u ∣ W ‾ u 0\le\mu_u\le|x_u|\overline{W}_u 0 ≤ μ u ≤ ∣ x u ∣ W u , W ‾ u \overline{W}_u W u being the same sum as W u W_u W u with ∣ w a , b ∣ |w_{a,b}| ∣ w a , b ∣ in place of w a , b η a , b w_{a,b}\eta_{a,b} w a , b η a , b .
(iii) Let x ˉ \bar{x} x ˉ be a simple switching point of a ReLU node v v v with z ~ v ( x ˉ ) = 0 \widetilde{z}_v(\bar{x})=0 z v ( x ˉ ) = 0 , z ~ v \widetilde{z}_v z v taking both signs near x ˉ \bar{x} x ˉ , and v v v the maximizer of no max node, and let μ + \mu^+ μ + be the limits of μ ( x ) \mu(x) μ ( x ) as x → x ˉ x\to\bar{x} x → x ˉ with z ~ v ( x ) < 0 \widetilde{z}_v(x)<0 z v ( x ) < 0 , resp. z ~ v ( x ) > 0 \widetilde{z}_v(x)>0 z v ( x ) > 0 . Then μ − = μ ( x ˉ ) \mu^-=\mu(\bar{x}) μ − = μ ( x ˉ ) , and Δ : = μ + − μ ( x ˉ ) \Delta:=\mu^+-\mu(\bar{x}) Δ := μ + − μ ( x ˉ ) satisfies ∑ u ∈ I Δ u = 0 \sum_{u\in I}\Delta_u=0 ∑ u ∈ I Δ u = 0 , with: (a) for LRP 0 \operatorname{LRP}_0 LRP 0 , Δ u = x ˉ u W u v ( x ˉ ) W v ( x ˉ ) \Delta_u=\bar{x}_uW_u^v(\bar{x})W_v(\bar{x}) Δ u = x ˉ u W u v ( x ˉ ) W v ( x ˉ ) ; (b) for LRP N α , β \operatorname{LRP}_{N_{\alpha,\beta}} LRP N α , β , if x ˉ i ≠ 0 \bar{x}_i\ne0 x ˉ i = 0 at every input i i i , Δ = 0 \Delta=0 Δ = 0 unless some successor w w w of v v v other than a max node has at x ˉ \bar{x} x ˉ no contribution of the sign of w v , w w_{v,w} w v , w , and Δ = 0 \Delta=0 Δ = 0 for LRP z + \operatorname{LRP}_{z+} LRP z + .
(iv) Let x ˉ \bar{x} x ˉ be a simple switching point of a max node v v v whose maximum x v ( x ˉ ) ≠ 0 x_v(\bar{x})\ne0 x v ( x ˉ ) = 0 is attained at exactly two predecessors u , u ′ u,u' u , u ′ , with z u , v − z u ′ , v z_{u,v}-z_{u',v} z u , v − z u ′ , v taking both signs near x ˉ \bar{x} x ˉ ; let κ , κ ′ ∈ M ( x ˉ ) \kappa,\kappa'\in\mathcal{M}(\bar{x}) κ , κ ′ ∈ M ( x ˉ ) pick u u u , resp. u ′ u' u ′ at v v v and the
same predecessors elsewhere, and R i ( a ) R_i^{(a)} R i ( a ) be the sum over P ( i , a ) \mathcal{P}(i,a) P ( i , a ) of the products of the kernels of the rule along κ \kappa κ at x ˉ \bar{x} x ˉ . Then μ ( x ) \mu(x) μ ( x ) tends to μ κ ( x ˉ ) \mu^{\kappa}(\bar{x}) μ κ ( x ˉ ) as x → x ˉ x \to\bar{x} x → x ˉ with z u , v ( x ) > z u ′ , v ( x ) z_{u,v}(x) > z_{u',v}(x) z u , v ( x ) > z u ′ , v ( x ) , to μ κ ′ ( x ˉ ) \mu^{\kappa'}(\bar{x}) μ κ ′ ( x ˉ ) with z u ′ , v ( x ) > z u , v ( x ) z_{u',v}(x) > z_{u,v}(x) z u ′ , v ( x ) > z u , v ( x ) , and Δ : = μ κ ′ ( x ˉ ) − μ κ ( x ˉ ) \Delta:= \mu^{\kappa'}(\bar{x}) - \mu^{\kappa}(\bar{x}) Δ := μ κ ′ ( x ˉ ) − μ κ ( x ˉ ) satisfies ∑ i ∈ I Δ i = 0 \sum_{i \in I} \Delta_i = 0 ∑ i ∈ I Δ i = 0 , with: (a) for L R P 0 \mathrm{LRP}_0 LRP 0 , Δ i = x ˉ i W v ( x ˉ ) ( w u ′ , v W ^ u i ( x ˉ ) − w u , v W ^ u i ( x ˉ ) ) \Delta_i = \bar{x}_i W_v(\bar{x})(w_{u',v}\widehat{W}_u^{\,i}(\bar{x}) - w_{u,v}\widehat{W}_u^{\,i}(\bar{x})) Δ i = x ˉ i W v ( x ˉ ) ( w u ′ , v W u i ( x ˉ ) − w u , v W u i ( x ˉ )) , W ^ i a \widehat{W}_i^{\,a} W i a being the sum over P ( i , a ) \mathcal{P}(i,a) P ( i , a ) of the products of w η w_\eta w η along κ \kappa κ ; (b) for L R P z + \mathrm{LRP}_{z+} LRP z + and L R P N α , β \mathrm{LRP}_{N\alpha,\beta} LRP N α , β , if x ˉ i ≠ 0 \bar{x}_i \ne0 x ˉ i = 0 at every input i i i , Δ i = μ v ( x ˉ ) ( R i ( u ′ ) − R i ( u ) ) \Delta_i = \mu_v(\bar{x})(R_i^{(u')} - R_i^{(u)}) Δ i = μ v ( x ˉ ) ( R i ( u ′ ) − R i ( u ) ) .
Proof of the corollary is provided in Appendix § 7 .§ 5 .
For L R P 0 \mathrm{LRP}_0 LRP 0 , the recursion of (i) is the adjoint recursion (6) , the gates being its branch Jacobians; the hypothesis on linear nodes is sharp, a linear node with z ~ v = 0 \tilde{z}_v = 0 z ~ v = 0 leaving G ( x ) G(x) G ( x ) but keeping the gate 1. For the other rules, the gate becomes the secant ratio s v s_v s v – for L R P z + \mathrm{LRP}_{z+} LRP z + , the share z ~ v / z ~ v + ∈ ( 0 , 1 ] \tilde{z}_v/\tilde{z}_v^+ \in(0,1] z ~ v / z ~ v + ∈ ( 0 , 1 ] of the excitatory input of v v v that survives the inhibition –, so that W ρ W^\rho W ρ moves with x x x inside a region of constant pattern. The relevance of an input stays at most proportional to its value for L R P z + \mathrm{LRP}_{z+} LRP z + , whereas L R P N α , β \mathrm{LRP}_{N\alpha,\beta} LRP N α , β only bounds the absolute mass of the inputs by ∣ x o ∣ c L |x_o|c^L ∣ x o ∣ c L , with c = max v ( ∣ α v ∣ + ∣ β v ∣ ) c = \max_v(|\alpha_v| + |\beta_v|) c = max v ( ∣ α v ∣ + ∣ β v ∣ ) and L L L the length of the longest path: the amplification met in § 3.4 .
Part (iii) locates the jumps. For L R P 0 \mathrm{LRP}_0 LRP 0 , from the active side, v v v has a value x v → 0 x_v \to0 x v → 0 but splits μ v = x v W v \mu_v = x_v W_v μ v = x v W v in the proportions z u , v / x v z_{u,v}/x_v z u , v / x v , a product 0 ⋅ ∞ 0 \cdot\infty 0 ⋅ ∞ of limit z u , v W v z_{u,v}W_v z u , v W v : the size of the jump is set by the contributions that cancel at v v v , not by the perturbation. For L R P z + \mathrm{LRP}_{z+} LRP z + and L R P N α , β \mathrm{LRP}_{N\alpha,\beta} LRP N α , β , the kernel at v v v stays bounded, and what v v v receives from a successor w w w vanishes with x v x_v x v , unless v v v brings to w w w the first contribution of its sign: that channel then holds v v v alone and sends it a fixed share of μ w \mu_w μ w , α w \alpha_w α w or β w \beta_w β w , which is β w = 0 \beta_w = 0 β w = 0 for L R P z + \mathrm{LRP}_{z+} LRP z + . On the running example at x ˉ = ( − 2 , 1 ) \bar{x} = (-2,1) x ˉ = ( − 2 , 1 ) , L R P 0 \mathrm{LRP}_0 LRP 0 jumps by Δ = ( − 2 , 2 , 0 ) \Delta= (-2,2,0) Δ = ( − 2 , 2 , 0 ) (Figure 6 ), whereas L R P z + \mathrm{LRP}_{z+} LRP z + and L R P N 2 , − 1 \mathrm{LRP}_{N2,-1} LRP N 2 , − 1 are continuous (Figure 7 (a)), both successors of h 1 h_1 h 1 already having contributions of both signs; with w h 1 , o 1 = w i 1 , o 1 = − 1 w_{h_1,o_1} = w_{i_1,o_1} = -1 w h 1 , o 1 = w i 1 , o 1 = − 1 , o 1 o_1 o 1 has no negative contribution at x ˉ \bar{x} x ˉ , and L R P N 2 , − 1 \mathrm{LRP}_{N2,-1} LRP N 2 , − 1 jumps by Δ = ( 16 , − 18 , 2 ) \Delta= (16,-18,2) Δ = ( 16 , − 18 , 2 ) (Figure 7 (b)).
Figure 6. Corollary 3.5(iii)(a) at the switching point x ˉ = ( − 2 , 1 ) \bar{x} = (-2,1) x ˉ = ( − 2 , 1 ) of Figure 3 , for o 1 o_1 o 1 . Left: the weights w u , v w_{u,v} w u , v of the full graph; the kink node h 1 h_1 h 1 and its edges (dashed) lie outside G ( x ˉ ) G(\bar{x}) G ( x ˉ ) . At each input, the L R P 0 \mathrm{LRP}_0 LRP 0 relevance μ u ( x ˉ ) \mu_u(\bar{x}) μ u ( x ˉ ) (red) and the jump Δ u \Delta_u Δ u (orange) met when h 1 h_1 h 1 turns on. Right: the input relevances along x = ( − 2 , 1 + δ ) x = (-2,1+\delta) x = ( − 2 , 1 + δ ) ; at δ = 0 \delta= 0 δ = 0 , they equal the limit from δ < 0 \delta< 0 δ < 0 (red dot), and jump by Δ \Delta Δ for δ > 0 \delta> 0 δ > 0 (open dots).
Figure 7. Corollary 3.5(iii)(b) and (iv): input relevances μ i 1 \mu_{i_1} μ i 1 (blue), μ i 2 \mu_{i_2} μ i 2 (green) and μ b 1 \mu_{b_1} μ b 1 (black) along a line through a switching point, for L R P z + \mathrm{LRP}_{z+} LRP z + (solid) and L R P N 2 , − 1 \mathrm{LRPN}_{2,-1} LRPN 2 , − 1 (dashed); filled dots: the relevance at δ = 0 \delta= 0 δ = 0 , open dots: the limit from δ > 0 \delta> 0 δ > 0 where it differs. (a) The network of Figure 6 , along x = ( − 2 , 1 + δ ) x = (-2, 1 + \delta) x = ( − 2 , 1 + δ ) : where L R P 0 \mathrm{LRP}_0 LRP 0 jumps, both rules are continuous, o 1 o_1 o 1 already having contributions of both signs (μ i 1 \mu_{i_1} μ i 1 for L R P z + \mathrm{LRP}_{z+} LRP z + and μ b 1 \mu_{b_1} μ b 1 for L R P N 2 , − 1 \mathrm{LRPN}_{2,-1} LRPN 2 , − 1 coincide). (b) The same network with w h 1 , o 1 = w i 1 , o 1 = − 1 w_{h_1,o_1} = w_{i_1,o_1} = -1 w h 1 , o 1 = w i 1 , o 1 = − 1 (thick): at x ˉ = ( − 2 , 1 ) \bar{x} = (-2, 1) x ˉ = ( − 2 , 1 ) , o 1 o_1 o 1 has no negative contribution, and h 1 h_1 h 1 brings the first one when it turns on; L R P N 2 , − 1 \mathrm{LRPN}_{2,-1} LRPN 2 , − 1 jumps from ( 6 , 2 , 2 ) (6, 2, 2) ( 6 , 2 , 2 ) to ( 22 , − 16 , 4 ) (22, -16, 4) ( 22 , − 16 , 4 ) , L R P z + \mathrm{LRP}_{z+} LRP z + stays continuous, and the two rules coincide for δ < 0 \delta< 0 δ < 0 . (c) x o = x v = max ( x i 1 , x i 2 ) x_o = x_v = \max(x_{i_1}, x_{i_2}) x o = x v = max ( x i 1 , x i 2 ) along x = ( 1 , 1 + δ ) x = (1, 1 + \delta) x = ( 1 , 1 + δ ) : every rule jumps by Δ = ( − 1 , 1 ) \Delta= (-1, 1) Δ = ( − 1 , 1 ) , and at δ = 0 \delta= 0 δ = 0 the relevance is the segment between the two selections (shaded).
Part (iv) is the counterpart of (iii) at a max node, with the opposite outcome in general. Whatever the rule, the kernel of a max node is 1 on its winning edge and 0 on the others, so that at a change of winner the whole relevance μ v \mu_v μ v of the node leaves the paths through u u u for the paths through u ′ u' u ′ ; Δ = 0 \Delta= 0 Δ = 0 if and only if the node carries no relevance, μ v ( x ˉ ) = 0 \mu_v(\bar{x}) = 0 μ v ( x ˉ ) = 0 , or both predecessors pass a unit of relevance on to the inputs in the same way, R ( u ) = R ( u ′ ) R^{(u)} = R^{(u')} R ( u ) = R ( u ′ ) . At x ˉ \bar{x} x ˉ itself, the relevance is the segment between μ κ ( x ˉ ) \mu^{\kappa}(\bar{x}) μ κ ( x ˉ ) and μ κ ′ ( x ˉ ) \mu^{\kappa'}(\bar{x}) μ κ ′ ( x ˉ ) (Theorem 3.3 ). For x o = x v x_o = x_v x o = x v and x v = max ( x i 1 , x i 2 ) x_v = \max(x_{i_1},x_{i_2}) x v = max ( x i 1 , x i 2 ) , at x ˉ i 1 = x ˉ i 2 = a ≠ 0 \bar{x}_{i_1} = \bar{x}_{i_2} = a \ne0 x ˉ i 1 = x ˉ i 2 = a = 0 , every rule gives μ κ = ( a , 0 ) \mu^{\kappa} = (a,0) μ κ = ( a , 0 ) , μ κ ′ = ( 0 , a ) \mu^{\kappa'} = (0,a) μ κ ′ = ( 0 , a ) and Δ = ( − a , a ) \Delta= (-a,a) Δ = ( − a , a ) (Figure 7 (c)). The second alternative at the other max nodes, in (iii) and (iv), is a max-pooling window whose ReLU inputs are all inactive, as in VGG-16: its maximum 0 is attained several times, but the window stays inactive near x ˉ \bar{x} x ˉ , and its winner does not matter.
Sinkhorn Relevance Propagation
The minimizer of Theorem 3.3 rescales the reference at the head of each edge, π u , v = ρ u , v μ v / ρ ~ v \pi_{u,v} = \rho_{u,v}\mu_v/\widetilde{\rho}_v π u , v = ρ u , v μ v / ρ v ; it compares transition laws, node by node. Comparing instead the flow itself with ρ \rho ρ , edge by edge, gives another problem, whose solution rescales ρ \rho ρ at both ends of each edge, through one potential per node, computed by Sinkhorn iterations [14 ] . A reference of either sign is split into its sign parts, ρ = ρ + − ρ − \rho= \rho^+ - \rho^- ρ = ρ + − ρ − , and we look for two non-negative flows whose difference is conservative. The positive part carries mass along its edge, the negative part against it: let D ρ ( x ) D_\rho(x) D ρ ( x ) have an arc u → v u \to v u → v for every edge ( u , v ) ∈ E ( x ) (u,v) \in E(x) ( u , v ) ∈ E ( x ) with ρ u , v > 0 \rho_{u,v} > 0 ρ u , v > 0 and an arc v → u v \to u v → u for every edge with ρ u , v < 0 \rho_{u,v} < 0 ρ u , v < 0 ; let D ‾ ρ ( x ) \overline{D}_\rho(x) D ρ ( x ) merge the inputs into a single node S S S and add the arc o → S o \to S o → S ; let E ρ o ( x ) E^o_\rho(x) E ρ o ( x ) be the edges whose arc lies in the strongly connected component of S S S in D ‾ ρ ( x ) \overline{D}_\rho(x) D ρ ( x ) , and V ρ o ( x ) V^o_\rho(x) V ρ o ( x ) the nodes of that component other than S S S , together with the inputs. For ρ ≥ 0 \rho\ge0 ρ ≥ 0 , E ρ o ( x ) E^o_\rho(x) E ρ o ( x ) consists of the edges lying on a path of positive reference from I ( x ) I(x) I ( x ) to o o o .
Theorem 3.6. Let x x x be an input at which the node values do not branch, o ∈ O o \in O o ∈ O an active output, y > 0 y > 0 y > 0 , A o ( x ; y ) A_o(x;y) A o ( x ; y ) the admissible set of Definition 3.1 with x o x_o x o replaced by y y y , and ρ ∈ R E ( x ) \rho\in\mathbb{R}^{E(x)} ρ ∈ R E ( x ) vanishing off E ρ o ( x ) E^o_\rho(x) E ρ o ( x ) and on the edges entering the outputs other than o o o , with o ∈ V ρ o ( x ) o \in V^o_\rho(x) o ∈ V ρ o ( x ) . Then:
(i) The problem
min π + ≥ 0 π + − π − ∈ A o ( x ; y ) KL ( π + ∥ ρ + ) + KL ( π − ∥ ρ − ) , KL ( π ∥ ρ ) : = ∑ ( u , v ) ∈ E ( x ) ( π u , v log π u , v ρ u , v − π u , v + ρ u , v ) (9) \min_{\substack{\pi^+ \ge0\\ \pi^+ - \pi^- \in A_o(x;y)}} \operatorname{KL}(\pi^+ \mathbin\Vert\rho^+) + \operatorname{KL}(\pi^- \mathbin\Vert\rho^-), \qquad\operatorname{KL}(\pi\mathbin\Vert\rho) := \sum_{(u,v)\in E(x)} \left(\pi_{u,v}\log\frac{\pi_{u,v}}{\rho_{u,v}}-\pi_{u,v}+\rho_{u,v}\right) \tag*{(9)} π + ≥ 0 π + − π − ∈ A o ( x ; y ) min KL ( π + ∥ ρ + ) + KL ( π − ∥ ρ − ) , KL ( π ∥ ρ ) := ( u , v ) ∈ E ( x ) ∑ ( π u , v log ρ u , v π u , v − π u , v + ρ u , v ) (9) has a unique minimizer ( π S + , π S − ) (\pi^{S+},\pi^{S-}) ( π S + , π S − ) , and π S ± > 0 \pi^{S\pm} > 0 π S ± > 0 exactly where ρ ± > 0 \rho^\pm> 0 ρ ± > 0 .
(ii) There is a unique family of potentials ( φ v ) v ∈ V ρ o ( x ) (\varphi_v)_{v\in V_\rho^o(x)} ( φ v ) v ∈ V ρ o ( x ) , with φ v > 0 \varphi_v > 0 φ v > 0 and φ i = 1 \varphi_i = 1 φ i = 1 at the inputs, such that
π u , v S + = ρ u , v + φ v φ u , π u , v S − = ρ u , v − φ u φ v , ( u , v ) ∈ E ρ o ( x ) . (10) \pi^{S+}_{u,v}=\rho^+_{u,v}\frac{\varphi_v}{\varphi_u},\qquad\pi^{S-}_{u,v}=\rho^-_{u,v}\frac{\varphi_u}{\varphi_v},\qquad(u,v)\in E_\rho^o(x). \tag*{(10)} π u , v S + = ρ u , v + φ u φ v , π u , v S − = ρ u , v − φ v φ u , ( u , v ) ∈ E ρ o ( x ) . (10) In particular ∏ e ∈ p π e S + / ρ e = φ v \prod_{e\in p}\pi_e^{S+}/\rho_e=\varphi_v ∏ e ∈ p π e S + / ρ e = φ v for every path p p p of positive reference from an input to v v v .
(iii) Conversely, a pair of the form (10) is the minimizer if and only if the potentials solve
φ v 2 ( ∑ u ρ u , v + φ u + ∑ w ρ v , w − φ w ) = ∑ w ρ v , w + φ w + ∑ u ρ u , v − φ u ( v ∈ V ρ o ( x ) ∩ H ( x ) ) , \varphi_v^2\left(\sum_u\frac{\rho^+_{u,v}}{\varphi_u}+\sum_w\frac{\rho^-_{v,w}}{\varphi_w}\right)=\sum_w\rho^+_{v,w}\varphi_w+\sum_u\rho^-_{u,v}\varphi_u\qquad(v\in V_\rho^o(x)\cap H(x)), φ v 2 ( u ∑ φ u ρ u , v + + w ∑ φ w ρ v , w − ) = w ∑ ρ v , w + φ w + u ∑ ρ u , v − φ u ( v ∈ V ρ o ( x ) ∩ H ( x )) , φ o ∑ u ρ u , o + φ u − 1 φ o ∑ u ρ u , o − φ u = y . (11) \varphi_o\sum_u\frac{\rho^+_{u,o}}{\varphi_u}-\frac{1}{\varphi_o}\sum_u\rho^-_{u,o}\varphi_u=y. \tag*{(11)} φ o u ∑ φ u ρ u , o + − φ o 1 u ∑ ρ u , o − φ u = y . (11) the sums running over E ρ o ( x ) E_\rho^o(x) E ρ o ( x ) .
(iv) The Sinkhorn relevances μ u S : = ∑ w : u → w ( π u , w S + − π u , w S − ) \mu_u^S:=\sum_{w:u\to w}(\pi^{S+}_{u,w}-\pi^{S-}_{u,w}) μ u S := ∑ w : u → w ( π u , w S + − π u , w S − ) , u ∉ O u\notin O u ∈ / O , satisfy ∑ u ∈ I ( x ) μ u S = y \sum_{u\in I(x)}\mu_u^S=y ∑ u ∈ I ( x ) μ u S = y .
(v) Order V ρ o ( x ) ∖ I = { v 1 , … , v m } V_\rho^o(x)\setminus I=\{v_1,\ldots,v_m\} V ρ o ( x ) ∖ I = { v 1 , … , v m } , take φ ( 0 ) > 0 \varphi^{(0)}>0 φ ( 0 ) > 0 with φ i ( 0 ) = 1 \varphi_i^{(0)}=1 φ i ( 0 ) = 1 at the inputs, and obtain φ ( t + 1 ) \varphi^{(t+1)} φ ( t + 1 ) from φ ( t ) \varphi^{(t)} φ ( t ) by giving, for j = 1 , … , m j=1,\ldots,m j = 1 , … , m in turn, to φ v j \varphi_{v_j} φ v j the positive value for which (11) holds at v j v_j v j , the other potentials being fixed at their current values. Then φ ( t ) \varphi^{(t)} φ ( t ) converges to the potentials of (ii).
Proof of the theorem is provided in Appendix B.6. By default, SRP takes ρ = z + \rho=z^+ ρ = z + and y = x o y=x_o y = x o for x o > 0 x_o>0 x o > 0 , so that $\pi^{S-}=0.
Problem (9) is an entropically regularized minimum-cost flow (Peyré and Cuturi, [36 ] ). Writing ρ e ± = e − c e / ε \rho_e^\pm=e^{-c_e/\varepsilon} ρ e ± = e − c e / ε , the relative entropy is, up to a constant, ε − 1 ∑ e c e π e + ∑ e ( π e log π e − π e ) \varepsilon^{-1}\sum_e c_e\pi_e+\sum_e(\pi_e\log\pi_e-\pi_e) ε − 1 ∑ e c e π e + ∑ e ( π e log π e − π e ) : a linear transport cost c c c plus an entropy, minimized over the flows that are conservative at every hidden node and bring y y y to o o o . Entropic optimal transport (Cuturi, [14 ] ) prescribes instead the two marginals of a bipartite graph, and its solution π i , j = a i K i , j b j \pi_{i,j}=a_iK_{i,j}b_j π i , j = a i K i , j b j rescales the reference by one factor per node, the factors being computed by alternate rescalings, the Sinkhorn iterations. Theorem 3.6 has the same structure on a DAG, with free inputs: every hidden node and o o o carry a constraint, hence a potential φ v \varphi_v φ v , and, for ρ ≥ 0 \rho\geq0 ρ ≥ 0 , π u , v S = ρ u , v φ v / φ u \pi^S_{u,v}=\rho_{u,v}\varphi_v/\varphi_u π u , v S = ρ u , v φ v / φ u rescales each edge at its two ends; the sweeps of (v) are the Sinkhorn iterations. The Schrödinger bridge of § 4.1 has the same two-ended form, on paths.
The potentials are the exponentials of the Lagrange multipliers of the conservation constraints, and each update of (v) maximizes the dual function of (9) in one coordinate. For ρ ≥ 0 \rho\geq0 ρ ≥ 0 , it reads φ v = ( ∑ w ρ v , w φ w / ∑ u ρ u , v / φ u ) 1 / 2 \varphi_v=(\sum_w\rho_{v,w}\varphi_w/\sum_u\rho_{u,v}/\varphi_u)^{1/2} φ v = ( ∑ w ρ v , w φ w / ∑ u ρ u , v / φ u ) 1/2 at a hidden node and φ o = y / ∑ u ρ u , o / φ u \varphi_o=y/\sum_u\rho_{u,o}/\varphi_u φ o = y / ∑ u ρ u , o / φ u . One sweep costs O ( ∣ E ρ o ( x ) ∣ ) O(\lvert E_\rho^o(x)\rvert) O (∣ E ρ o ( x )∣) , like a pass of LRP, and the observed convergence is geometric, as for the Sinkhorn algorithm (Cuturi, [14 ] ; Peyré and Cuturi, [36 ] ).
Proposition 3.7. Under the assumptions of Theorem 3.6 , with ρ ≥ 0 \rho\geq0 ρ ≥ 0 , write π S : = π S + \pi^S:=\pi^{S+} π S := π S + : (i) if the restriction of ρ \rho ρ to E ρ o ( x ) E_\rho^o(x) E ρ o ( x ) lies in A o ( x ; y ) A_o(x;y) A o ( x ; y ) , then π S \pi^S π S is this restriction and all potentials equal 1 1 1 ; (ii) if every edge of E ρ o ( x ) E_\rho^o(x) E ρ o ( x ) enters o o o , then
π u , o S = y ρ u , o ∑ ( r , o ) ∈ E ρ o ( x ) ρ r , o , \pi^S_{u,o}=\frac{y\rho_{u,o}}{\sum_{(r,o)\in E_\rho^o(x)}\rho_{r,o}}, π u , o S = ∑ ( r , o ) ∈ E ρ o ( x ) ρ r , o y ρ u , o ,
which is the flow of Theorem 3.3 for the same reference, with x o x_o x o replaced by y y y ; (iii) in general π S \pi^S π S differs from that flow: for one input i i i , two hidden nodes h 1 , h 2 h_1,h_2 h 1 , h 2 and the output o o o , with E ρ o ( x ) = { ( i , h 1 ) , ( i , h 2 ) , ( h 1 , o ) , ( h 2 , o ) } E_\rho^o(x)=\{(i,h_1),(i,h_2),(h_1,o),(h_2,o)\} E ρ o ( x ) = {( i , h 1 ) , ( i , h 2 ) , ( h 1 , o ) , ( h 2 , o )} ,
π h 1 , o S π h 2 , o S = ( ρ i , h 1 ρ h 1 , o ρ i , h 2 ρ h 2 , o ) 1 / 2 , \frac{\pi^S_{h_1,o}}{\pi^S_{h_2,o}}=\left(\frac{\rho_{i,h_1}\rho_{h_1,o}}{\rho_{i,h_2}\rho_{h_2,o}}\right)^{1/2}, π h 2 , o S π h 1 , o S = ( ρ i , h 2 ρ h 2 , o ρ i , h 1 ρ h 1 , o ) 1/2 ,
whereas
π h 1 , o π h 2 , o = ρ h 1 , o ρ h 2 , o \frac{\pi_{h_1,o}}{\pi_{h_2,o}}=\frac{\rho_{h_1,o}}{\rho_{h_2,o}} π h 2 , o π h 1 , o = ρ h 2 , o ρ h 1 , o
for Theorem 3.3 .
Proof of the proposition is provided in Appendix B.8.
LRP allocates greedily, top-down, splitting the relevance of each node in proportion to the contributions of its parents; SRP rescales every path from an input to o o o by the same factor φ o \varphi_o φ o (Theorem 3.6 (ii)). With ρ = ( 1 , 1 , 4 , 1 ) \rho=(1,1,4,1) ρ = ( 1 , 1 , 4 , 1 ) and y = 5 y=5 y = 5 in Proposition 3.7(iii) (Figure 8 ), LRP sends 4 through h 1 h_1 h 1 , following the last layer only, and SRP 10 / 3 10/3 10/3 , balancing both layers. SRP needs no non-degeneracy condition: for ρ ≥ 0 \rho\geq0 ρ ≥ 0 , a node with ρ v = 0 \rho_v=0 ρ v = 0 has no arc entering it and lies outside V ρ o ( x ) V_\rho^o(x) V ρ o ( x ) . For x o < 0 x_o<0 x o < 0 and a reference whose reflection − ρ -\rho − ρ satisfies the assumptions of Theorem 3.6 , such as ρ λ \rho^\lambda ρ λ below, which changes sign with x o x_o x o , we take the minimizer for ( − ρ , ∣ x o ∣ ) (-\rho,\lvert x_o\rvert) ( − ρ , ∣ x o ∣) , on the support E − ρ o ( x ) E^{o}_{-\rho}(x) E − ρ o ( x ) , with its sign changed, and the zero flow for x o = 0 x_o=0 x o = 0 : exchanging the two sign parts maps (9) for ( ρ , y ) (\rho,y) ( ρ , y ) onto (9) for ( − ρ , − y ) (-\rho,-y) ( − ρ , − y ) , with flows of opposite sign. The reflection of z + ≥ 0 z^+\ge0 z + ≥ 0 has no path from the inputs to o o o ; for x o < 0 x_o<0 x o < 0 , SRP with ρ = z + \rho=z^+ ρ = z + is then defined as the flow of ( z + , ∣ x o ∣ ) (z^+,|x_o|) ( z + , ∣ x o ∣ ) with its sign changed, a convention rather than a minimizer for y = x o y=x_o y = x o .
Figure 8. The example of Proposition 3.7(iii) (w i , h 1 = w i , h 2 = 1 w_{i,h_1}=w_{i,h_2}=1 w i , h 1 = w i , h 2 = 1 , w h 1 , o = 4 w_{h_1,o}=4 w h 1 , o = 4 , w h 2 , o = 1 w_{h_2,o}=1 w h 2 , o = 1 , x i = 1 x_i=1 x i = 1 ), edge widths proportional to the values shown: the forward reference (not conservative at h 1 h_1 h 1 ), LRP and SRP, both conserving x o = 5 x_o=5 x o = 5 .
With a signed reference, a negative input relevance μ i S < 0 \mu_i^{\mathrm{S}}<0 μ i S < 0 is mass received back along negative edges – counter-evidence; restricting ρ \rho ρ to E ρ o ( x ) E^{o}_{\rho}(x) E ρ o ( x ) and to the edges not entering the other outputs is then a choice of the model, which keeps the problem on the output explained. The reference must also carry the scale of the output: the relative entropy keeps the flows close to the reference in mass, and z z z , whose absolute values add up to the whole forward activity of the network, is far larger than x o x_o x o . SRP also depends on the scale of the hidden units: multiplying a hidden unit by c > 0 c>0 c > 0 and its outgoing weights by 1 / c 1/c 1/ c leaves the network function unchanged, but multiplies by c c c the reference on the edges entering the unit. In Figure 8 , with h 1 h_1 h 1 scaled by 9, the ratio of Proposition 3.7(iii) becomes 6 and SRP sends 30 / 7 30/7 30/7 through h 1 h_1 h 1 instead of 10 / 3 10/3 10/3 , whereas LRPz + _{z^+} z + , which normalizes at the head of each edge, still sends 4: the Sinkhorn optimum is relative to the computational representation of the network.
Proposition 3.8. Under the assumptions of Theorem 3.6 , if the positive part ρ + \rho^+ ρ + lies in A o ( x ; y ) A_o(x;y) A o ( x ; y ) , the Sinkhorn flows have total mass ∑ ( u , v ) ( π u , v + + π u , v − ) ≤ e R \sum_{(u,v)}(\pi^+_{u,v}+\pi^-_{u,v})\le eR ∑ ( u , v ) ( π u , v + + π u , v − ) ≤ e R , where R : = ∑ ( u , v ) ∣ ρ u , v ∣ R:=\sum_{(u,v)}|\rho_{u,v}| R := ∑ ( u , v ) ∣ ρ u , v ∣ .
Proof of the proposition is provided in Appendix § 7 .7.
The LRPz + _{z^+} z + flow π z + \pi^{z^+} π z + of Corollary 3.4(ii), of node relevances μ + \mu^+ μ + , is conservative at the scale of x o x_o x o ; adding to it the inhibitory contributions at the same scale gives the reference
ρ u , v λ : = μ v + z v + ( z u , v + − λ z u , v − ) , λ ≥ 0 , (12) \rho^\lambda_{u,v}:=\frac{\mu^+_v}{z^+_v}(z^+_{u,v}-\lambda z^-_{u,v}),\qquad\lambda\ge0, \tag*{(12)} ρ u , v λ := z v + μ v + ( z u , v + − λ z u , v − ) , λ ≥ 0 , (12)
on the edges entering H ( x ) ∪ { o } H(x)\cup\{o\} H ( x ) ∪ { o } , and ρ u , v λ : = 0 \rho^\lambda_{u,v}:=0 ρ u , v λ := 0 on the edges entering the other outputs. Its positive part is π z + \pi^{z^+} π z + for x o > 0 x_o>0 x o > 0 ; for x o < 0 x_o<0 x o < 0 , it is minus the reference of ∣ x o ∣ |x_o| ∣ x o ∣ . We write SRPλ _\lambda λ for SRP with this reference, on its support E ρ o ( x ) E^o_\rho(x) E ρ o ( x ) and with y = x o y=x_o y = x o for x o > 0 x_o>0 x o > 0 , and with the convention above for x o ≤ 0 x_o\le0 x o ≤ 0 . SRP0 _0 0 is LRPz + _{z^+} z + (Proposition 3.7(i)); for λ > 0 \lambda>0 λ > 0 , the inhibitory contributions enter as counter-evidence, with a total mass at most e e e times that of the reference. The weight λ \lambda λ plays the part of β \beta β in LRPα , β _{\alpha,\beta} α , β , but the balance is global instead of node by node.
For ρ ≥ 0 \rho\ge0 ρ ≥ 0 , SRP is a rule of Theorem 3.3 : by (10) , the net inflow of v v v is μ v S = φ v ∑ u ρ u , v / φ u \mu_v^{\mathrm{S}}=\varphi_v\sum_u\rho_{u,v}/\varphi_u μ v S = φ v ∑ u ρ u , v / φ u , so that π u , v S = μ v S ( ρ u , v / φ u ) / ∑ r ( ρ r , v / φ r ) \pi^{\mathrm{S}}_{u,v}=\mu_v^{\mathrm{S}}(\rho_{u,v}/\varphi_u)/\sum_r(\rho_{r,v}/\varphi_r) π u , v S = μ v S ( ρ u , v / φ u ) / ∑ r ( ρ r , v / φ r ) is the flow of Theorem 3.3 on E ρ o ( x ) E^o_\rho(x) E ρ o ( x ) , with boundary mass y y y , for the reference ρ u , v / φ u \rho_{u,v}/\varphi_u ρ u , v / φ u . Every statement on the non-negative references of Theorem 3.3 then applies to SRP, the relevance by region of § 4.1 included.
Table 1 : The rules on VGG-16. On 200 Imagenette photographs: largest relative conservation error over the pixels and the bias nodes, mean share of x o x_o x o absorbed by the bias nodes, mean share of negative relevance in the pixel maps, median time per image (NVIDIA T1000), and faithfulness (ABPC, higher is better). On 200 Oxford-IIIT Pet photographs: relevance mass accuracy. On 50 Imagenette photographs: Spearman correlation ρ 2 \rho_2 ρ 2 of the absolute maps of the predicted and of the second-ranked class; 1 means the same ranking of the pixels for both classes.
Table 1.
Application: explaining an image classifier
We apply the rules to VGG-16 (Simonyan and Zisserman, [43 ] ) with its ImageNet weights (Russakovsky et al., [38 ] ) (Appendix D), a layered Filippov network (Definition 2.4 ) with one input node per pixel and colour channel and one bias node of value 1 per convolutional, dense and output node: a conservative rule distributes x o x_o x o , the logit of the predicted class, over the pixels and the bias nodes (Proposition 3.2 ). At a max-pooling node, every rule takes the reference x u x_u x u on the winning edge and 0 elsewhere. The rules are LRP0 _0 0 , here Gradient× \times × Input (Corollary 3.5(i)); LRPz + _{z+} z + ; LRPN 2 , − 1 _{N2,-1} N 2 , − 1 ; LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) , the Hebbian weights α v = x ˉ v : = x v / max w x w \alpha_v = \bar{x}_v := x_v/\max_w x_w α v = x ˉ v := x v / max w x w , the maximum running over the layer of v v v (x ˉ v : = 0 \bar{x}_v := 0 x ˉ v := 0 if it vanishes), and β v = 1 − x ˉ v \beta_v = 1 - \bar{x}_v β v = 1 − x ˉ v ; SRP with ρ = z + \rho= z^+ ρ = z + ; and SRP1 _1 1 ((12) , λ = 1 \lambda= 1 λ = 1 ), whose over-relaxed sweeps stop at a conservation defect of 1% of x o x_o x o , the recursion of LRP for ρ u , v / φ u \rho_{u,v}/\varphi_u ρ u , v / φ u then conserving x o x_o x o exactly.
On 200 Imagenette photographs (Howard, [23 ] ), we measure the quantities of Table 1 and faithfulness by region perturbation (Samek et al., [39 ] ): the 16 × 16 16 \times16 16 × 16 patches M M M are blurred in decreasing (MoRF) or increasing (LeRF) order of their region relevance μ ( M ) = ∑ i ∈ M μ i \mu(M) = \sum_{i \in M} \mu_i μ ( M ) = ∑ i ∈ M μ i (§ 4.1 ), and ABPC is the area under LeRF minus that under MoRF. Localization is the relevance mass accuracy (Arras et al., [3 ] ), the share of ∑ i ∣ μ i ∣ \sum_i \lvert\mu_i\rvert ∑ i ∣ μ i ∣ inside the animal, on 200 Oxford-IIIT Pet photographs (Parkhi et al., [35 ] ); class sensitivity, the Spearman correlation ρ 2 \rho_2 ρ 2 between the absolute maps of the first two classes, on 50 more photographs (Gu et al., [21 ] ; Sixt et al. [45 ] }).
The rules conserve x o x_o x o to single-precision round-off (Table 1). The LRP0 _0 0 map is half negative, alternating from pixel to pixel (Figure 9 ): its values of both signs compensate, leaving 92% of x o x_o x o to the bias nodes, and largely cancel within a patch; its ABPC is barely above that of a random map. LRPz + _{z+} z + and LRPN 2 , − 1 _{N2,-1} N 2 , − 1 have the highest ABPC, with no clear paired difference in this sample (+0.000 ± \pm ± 0.003), LRPN 2 , − 1 _{N2,-1} N 2 , − 1 localizing best; LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) , non-negative and more diffuse, is nearly as faithful and leaves less of x o x_o x o to the bias nodes. SRP follows the edges and textures of the lower layers (rank correlation 0.82 with LRPz + _{z+} z + ), is less faithful and localizes no better than the area of the animal (58%); SRP1 _1 1 comes close to the ABPC of LRPz + _{z+} z + , with a small but systematic deficit (paired difference − 0.004 ± 0.001 -0.004 \pm0.001 − 0.004 ± 0.001 ), at 93 s per image. ABPC and keeping pixels (Figure 10 ) reward the maps that cover the object, and every rule but LRP0 _0 0 ranks the pixels almost as for the second class (ρ 2 \rho_2 ρ 2 ): the class enters only through the output layer, through its weights w o + w^+_o w o + , and its bias for a non-negative reference and through its weights of both signs for LRPN 2 , − 1 _{N2,-1} N 2 , − 1 and SRP1 _1 1 , whereas LRP0 _0 0 depends on it through its costate.
Figure 9. Pixel maps for the class predicted by VGG-16 (below each photograph): L R P 0 \mathrm{LRP}_0 LRP 0 , L R P z + \mathrm{LRP}_{z^+} LRP z + , L R P N 2 , − 1 \mathrm{LRP}_{N_{2,-1}} LRP N 2 , − 1 , L R P N α , β \mathrm{LRP}_{N_{\alpha,\beta}} LRP N α , β with α v = x ˉ v \alpha_v=\bar{x}_v α v = x ˉ v (defined in the text), and S R P 1 \mathrm{SRP}_1 SRP 1 . Red: positive relevance, blue: negative; each map is scaled to the 99.5th percentile of its ∣ μ i ∣ |\mu_i| ∣ μ i ∣ .
Figure 10. Share of the 151 Imagenette photographs (of the 200) that VGG-16 classifies correctly, still classified in their true class when only the k % k\% k % pixels of largest ∣ μ i ∣ |\mu_i| ∣ μ i ∣ are kept, for the five rules and a random map (dotted: LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) , defined in the text), the other pixels being set to black (a) or replaced by a blurred copy of the image (b); at k = 100 k = 100 k = 100 , every curve is at 100% by construction.
On VGG-16, a Gaussian noise of standard deviation 10 − 5 10^{-5} 1 0 − 5 on 40 photographs changes x o x_o x o by 5 ⋅ 10 − 6 5 \cdot10^{-6} 5 ⋅ 1 0 − 6 (median) but switches 176 of the 13.6 million ReLU gates and the winners of 2785 max-pooling windows, many events at once rather than the isolated ones of Corollary 3.5. The LRP0 _0 0 map moves by 1.5% in ℓ 1 \ell^1 ℓ 1 norm (median), those of LRPz + _{z+} z + , LRPN 2 , − 1 _{N2,-1} N 2 , − 1 and LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) by 0.06%, 0.073% and 0.11%: consistent with the jumps of LRP0 _0 0 at a kink (part (iii)(a)), the continuity of the other rules there outside the case excluded in (iii)(b), and the jumps that a change of winner may cause for every rule (part (iv)).
In ConvNeXt-Tiny (Liu et al., [29 ] ) (Appendix § 9 ), GELU activations, layer normalizations and residual sums give the hidden values both signs. We read it at x x x as a network of linear and GELU nodes, each layer normalization being linear with its standard deviation held constant, and a GELU node splitting its relevance as a ReLU node does. On the same 200 photographs (Figure 11 and Figure 12 ), L R P z + \mathrm{LRP}_{z^+} LRP z + loses 45% of x o x_o x o (median) at the residual sums whose two contributions are negative, where z ~ v + = 0 \tilde{z}_v^+=0 z ~ v + = 0 (Corollary 3.4(ii)); L R P N 1 , 0 \mathrm{LRP}_{N_{1,0}} LRP N 1 , 0 , the same rule with the fallback of (iv), conserves it, as the other rules do. L R P 0 \mathrm{LRP}_0 LRP 0 is no longer Gradient× \times × Input (Spearman correlation 0.12): extending the algebra of Corollary 3.5(i) to GELU nodes, its factors s v s_v s v become GELU ( z ~ v ) / z ~ v \operatorname{GELU}(\tilde{z}_v)/\tilde{z}_v GELU ( z ~ v ) / z ~ v , and the standard deviations of the layer normalizations, held constant, do not move with x x x as they do in the gradient; L R P 0 \mathrm{LRP}_0 LRP 0 is the more faithful of the two (ABPC 0.216 against 0.063). L R P N 2 , − 1 \mathrm{LRP}_{N_{2,-1}} LRP N 2 , − 1 amplifies the absolute mass to 21 x o 21x_o 21 x o (median). L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRP}_{N_{\alpha(\bar{x}),\beta(1-\bar{x})}} LRP N α ( x ˉ ) , β ( 1 − x ˉ ) has the highest ABPC of the rules tested (ABPC 0.249, against 0.216 for L R P N 1 , 0 \mathrm{LRP}_{N_{1,0}} LRP N 1 , 0 ), and its 10% most relevant pixels, on a blurred background, keep the class of 47% of the photographs, against 41% for L R P N 1 , 0 \mathrm{LRP}_{N_{1,0}} LRP N 1 , 0 ; with x ^ v : = ( x v − min w x w ) / ( max w x w − min w x w ) \hat{x}_v := (x_v-\min_w x_w)/(\max_w x_w-\min_w x_w) x ^ v := ( x v − min w x w ) / ( max w x w − min w x w ) (x ^ v : = 0 \hat{x}_v:=0 x ^ v := 0 on a constant layer) in place of x ˉ v \bar{x}_v x ˉ v , which can now be negative, this share rises to 53%.
Figure 11. Pixel maps for the class predicted by ConvNeXt-Tiny (below each photograph), as in Figure 9 : LRP0 _0 0 , LRPN 1 , 0 _{N1,0} N 1 , 0 , LRPN 2 , − 1 _{N2,-1} N 2 , − 1 , LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{N\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) and LRPN α ( x ^ ) , β ( 1 − x ^ ) _{N\alpha(\hat{x}),\beta(1-\hat{x})} N α ( x ^ ) , β ( 1 − x ^ ) (defined in the text). Red: positive relevance, blue: negative; each map is scaled to the 99.5th percentile of its ∣ μ i ∣ |\mu_i| ∣ μ i ∣ .
Figure 12. As Figure 10 , for ConvNeXt-Tiny and the rules of Figure 11 , with a random map: share of the 173 Imagenette photographs (of the 200) that it classifies correctly, still classified in their true class when only the k % k\% k % pixels of largest ∣ μ i ∣ |\mu_i| ∣ μ i ∣ are kept, the other pixels being set to black (a) or replaced by a blurred copy of the image (b).
Relevance by Region
The flows of § 3 live on the edges of the active graph; this section aggregates them over sets of nodes. For a non-negative reference, a region of the input – a set of pixels – receives the relevance of the paths that end in it, and the hidden units that carry it are read off one forward pass (§ 4.1 ); we apply this to VGG-16 on 200 annotated photographs of cats and dogs, to read which hidden units carry their head, their body and the background (§ 4.2 ).
Region relevance and Schrödinger bridges
For a non-negative reference ρ \rho ρ and a non-negative flow π \pi π , (8) is, by the chain rule, the path-space entropy of Schrödinger’s problem [8 , 36 ] between the Markov measures on backward paths with transitions π u , v / m v ( π ) \pi_{u,v}/m_v(\pi) π u , v / m v ( π ) and K u , v K_{u,v} K u , v (Appendix § 7 .§ 3 ); a signed reference loses this reading.
In image classification, a relevance map says which pixels support the class, not through which hidden units. Concept relevance propagation [1 ] answers one direction of this question: it keeps, at a hidden layer, the relevance of a chosen channel only, and propagates it to the image, which shows where the evidence used by that channel lies. The converse question – which channels carry the evidence of a given region; are the head and the collar of a dog recognized by the same channels? – is answered by the Markov structure of the path measure, which treats both directions alike, and conditioning it on a region is an entropic transport problem with both marginals prescribed. Figure 13 shows the two readings, and LRP itself, on the running example.
Figure 13. The joint mass μ v R i ( v ) \mu_v R_i^{(v)} μ v R i ( v ) of Corollary 4.1(i) on the running example at x = ( 1 , 1 ) x=(1,1) x = ( 1 , 1 ) , for o 1 o_1 o 1 and ρ = z + \rho=z^+ ρ = z + , the paths being grouped by the hidden node they cross. LRP reads its column margins, concept relevance a row, and the region relevances a column.
Corollary 4.1. Let x ∈ R d x \in\mathbb{R}^{d} x ∈ R d , κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , o ∈ O o \in\mathcal{O} o ∈ O an active output with x o ( κ ) > 0 x_o^{(\kappa)}>0 x o ( κ ) > 0 , and ρ ∈ R ≥ 0 E ( κ ) ( x ) \rho\in\mathbb{R}_{\geq0}^{E^{(\kappa)}(x)} ρ ∈ R ≥ 0 E ( κ ) ( x ) satisfying the assumptions of Theorem 3.3 , of kernel K K K extended by 0 off E ( κ ) ( x ) E^{(\kappa)}(x) E ( κ ) ( x ) ; write μ \mu μ and π ∗ \pi^* π ∗ for μ o , κ \mu^{o,\kappa} μ o , κ and π o , κ , ∗ \pi^{o,\kappa,*} π o , κ , ∗ . For
nodes u u u , v v v , let
R u ( v ) : = ∑ p ∈ P ( u , v ) ∏ ( a , b ) ∈ p K a , b R_u^{(v)} := \sum_{p\in\mathcal{P}(u,v)} \prod_{(a,b)\in p} K_{a,b} R u ( v ) := p ∈ P ( u , v ) ∑ ( a , b ) ∈ p ∏ K a , b
the relevance of u u u when the recursion of Theorem 3.3 is started from v v v with unit mass, and, for M ⊂ I ( x ) M \subset\mathcal{I}(x) M ⊂ I ( x ) , R ( v ) ( M ) : = ∑ i ∈ M R i ( v ) R^{(v)}(M) := \sum_{i\in M} R_i^{(v)} R ( v ) ( M ) := ∑ i ∈ M R i ( v ) and μ ( M ) : = ∑ i ∈ M μ i \mu(M) := \sum_{i\in M} \mu_i μ ( M ) := ∑ i ∈ M μ i .
(i) For every v ∈ V ( κ ) ( x ) ∖ ( O ∖ { o } ) v \in V^{(\kappa)}(x) \setminus(\mathcal{O} \setminus\{o\}) v ∈ V ( κ ) ( x ) ∖ ( O ∖ { o }) , ( R i ( v ) ) i ∈ I ( x ) (R_i^{(v)})_{i\in\mathcal{I}(x)} ( R i ( v ) ) i ∈ I ( x ) is a probability vector, and the paths of the backward Markov chain of kernel K K K started at o o o with mass x o ( κ ) x_o^{(\kappa)} x o ( κ ) (Appendix § 7 .§ 3 ) that pass through v v v and end at i i i carry the mass μ v R i ( v ) \mu_v R_i^{(v)} μ v R i ( v ) .
(ii) The region relevances μ v M : = μ v R ( v ) ( M ) \mu_v^M := \mu_v R^{(v)}(M) μ v M := μ v R ( v ) ( M ) satisfy ∑ M ∈ M μ v M = μ v \sum_{M\in\mathcal{M}} \mu_v^M = \mu_v ∑ M ∈ M μ v M = μ v for every partition M \mathcal{M} M of I ( x ) \mathcal{I}(x) I ( x ) , and ∑ v ∈ C μ v M = μ ( M ) \sum_{v\in C} \mu_v^M = \mu(M) ∑ v ∈ C μ v M = μ ( M ) for every cut C C C , i.e. every set of nodes met exactly once by each directed path of G ( κ ) ( x ) G^{(\kappa)}(x) G ( κ ) ( x ) from an input to o o o .
(iii) If μ ( M ) > 0 \mu(M) > 0 μ ( M ) > 0 , the problem
min { H ( π ∥ K ) : π ∈ A o ( κ ) ( x ) , π ≥ 0 , ∑ v : i → v π i , v = x o ( κ ) μ ( M ) μ i 1 M ( i ) ( i ∈ I ( x ) ) } \min\left\{\mathcal{H}(\pi\mathbin{\|} K) : \pi\in\mathcal{A}^{(\kappa)}_o(x),\ \pi\ge0,\ \sum_{v:i\to v} \pi_{i,v} = \frac{x_o^{(\kappa)}}{\mu(M)}\mu_i\mathbf{1}_M(i)\ \ (i\in\mathcal{I}(x))\right\} min { H ( π ∥ K ) : π ∈ A o ( κ ) ( x ) , π ≥ 0 , v : i → v ∑ π i , v = μ ( M ) x o ( κ ) μ i 1 M ( i ) ( i ∈ I ( x )) }
has the unique minimizer π u , v M : = ( x o ( κ ) / μ ( M ) ) π u , v ∗ R ( u ) ( M ) \pi_{u,v}^M := (x_o^{(\kappa)}/\mu(M))\pi_{u,v}^*R^{(u)}(M) π u , v M := ( x o ( κ ) / μ ( M )) π u , v ∗ R ( u ) ( M ) , of node relevances ( x o ( κ ) / μ ( M ) ) μ u M (x_o^{(\kappa)}/\mu(M))\mu_u^M ( x o ( κ ) / μ ( M )) μ u M , and its minimal value is x o ( κ ) log ( x o ( κ ) / μ ( M ) ) x_o^{(\kappa)}\log(x_o^{(\kappa)}/\mu(M)) x o ( κ ) log ( x o ( κ ) / μ ( M )) .
Proof of the corollary is provided in Appendix § 8 .§ 1 .
The region relevance μ v M \mu_v^M μ v M is the relevance of v v v for o o o times the share of the evidence of v v v itself that lies in M M M . Over a partition of the image into regions – superpixels, or the parts of the object – the μ v M \mu_v^M μ v M form a table of hidden units against regions whose margins are the relevances of Theorem 3.3 : for the dog, the row of a channel tells on which parts of the image it relies, and the column of the head which channels carry its evidence. Both directions read the joint mass μ v R i ( v ) \mu_vR_i^{(v)} μ v R i ( v ) of (i): concept relevance by hidden unit, the region relevances by region. Part (iii) is the two-marginal problem whose one-marginal form is Theorem 3.3 : π M \pi^M π M is the Schrödinger bridge, the most likely routing of relevance given that it ends in M M M , and its potential R ( ⋅ ) ( M ) R^{(\cdot)}(M) R ( ⋅ ) ( M ) follows from one forward pass. With a signed reference, (ii) still holds by linearity, without its probabilistic meaning.
The bridge rescales each edge at both ends, π u , v M = ( x o / μ ( M ) ) K u , v μ v R ( v ) ( M ) \pi_{u,v}^M = (x_o/\mu(M))K_{u,v}\mu_vR^{(v)}(M) π u , v M = ( x o / μ ( M )) K u , v μ v R ( v ) ( M ) : by μ v \mu_v μ v at its head, as LRP does, and by the potential R ( v ) ( M ) R^{(v)}(M) R ( v ) ( M ) at its tail, the factor ( x o / μ ( M ) ) 1 M (x_o/\mu(M))\mathbf{1}_M ( x o / μ ( M )) 1 M of the endpoint propagated back along the paths. Sinkhorn Relevance Propagation has the same two-ended form, π u , v S = ρ u , v φ v / φ u \pi_{u,v}^{\mathrm{S}} = \rho_{u,v}\varphi_v/\varphi_u π u , v S = ρ u , v φ v / φ u (Theorem 3.6 ), but the two problems compare different objects: the bridge compares path measures, with a constraint at the end of the paths only; SRP compares flows on the edges, with a constraint at every node, and its flow is not in general the flow of the chain of transitions K K K (Proposition 3.7(iii)).
In Figure 14 (x = ( 1 , 1 ) x=(1,1) x = ( 1 , 1 ) , ρ = z + \rho=z^+ ρ = z + , regions M 1 = { i 1 } M_1=\{i_1\} M 1 = { i 1 } , M 2 = { i 2 } M_2=\{i_2\} M 2 = { i 2 } , M 3 = { b 1 , b 2 } M_3=\{b_1,b_2\} M 3 = { b 1 , b 2 } ), μ h 1 = 3 \mu_{h_1}=3 μ h 1 = 3 splits as 1 + 2 + 0 1+2+0 1 + 2 + 0 and μ h 2 = 2 \mu_{h_2}=2 μ h 2 = 2 as 0 + 1 + 1 0+1+1 0 + 1 + 1 : h 1 h_1 h 1 carries the evidence of i 1 i_1 i 1 , h 2 h_2 h 2 that of the bias, and both share that of i 2 i_2 i 2 . Conditioning on M = M 2 M=M_2 M = M 2 sends 4 4 4 through h 1 h_1 h 1 and 2 2 2 through h 2 h_2 h 2 , the whole mass x o 1 = 6 x_{o1}=6 x o 1 = 6 ending at i 2 i_2 i 2 .
Figure 14. Corollary 4.1 on the running example at x = ( 1 , 1 ) x=(1,1) x = ( 1 , 1 ) , for o 1 o_1 o 1 and ρ = z + \rho=z^+ ρ = z + (b 2 b_2 b 2 , o 2 o_2 o 2 omitted). Left: LRPz + _{z^+} z + flows and relevances, and the region relevances μ v M \mu_v^M μ v M of each node, for M 1 = { i 1 } M_1=\{i_1\} M 1 = { i 1 } (blue), M 2 = { i 2 } M_2=\{i_2\} M 2 = { i 2 } (green) and M 3 = { b 1 , b 2 } M_3=\{b_1,b_2\} M 3 = { b 1 , b 2 } (gray). Right: the bridge π M \pi^M π M for M = M 2 M=M_2 M = M 2 , with the forward potentials R ( v ) ( M ) R^{(v)}(M) R ( v ) ( M ) (teal).
Figure 15 makes the analogy explicit. In Schrödinger’s problem, particles follow a reference Brownian motion and are observed with an unexpected final distribution; the most likely evolution compatible with both marginals minimizes the relative entropy to the reference on path space, and is the reference reweighted by a function of the endpoint. Here the particles are the quanta of relevance, the reference is the chain of kernel K K K , the initial marginal is x o δ o x_o\delta_o x o δ o and the final one ν \nu ν , and the reweighting is the Doob transform by h = R ( ⋅ ) ( M ) h=R^{(\cdot)}(M) h = R ( ⋅ ) ( M ) .
Figure 15. Corollary 4.1(iii) as a Schrödinger problem, on the running example at x = ( 1 , 1 ) x=(1,1) x = ( 1 , 1 ) , for o 1 o_1 o 1 and ρ = z + \rho=z^+ ρ = z + . Both path measures start from the Dirac mass x o δ o x_o\delta_o x o δ o ; edges carry transition probabilities, with widths proportional to the flows, and bars give the endpoint masses divided by x o x_o x o . (a) The reference ends with the LRP relevances. (b) The bridge is forced to end in M M M , with the forward potentials h = R ( ⋅ ) ( M ) h=R^{(\cdot)}(M) h = R ( ⋅ ) ( M ) (teal); it sends 4 through h 1 h_1 h 1 and 2 through h 2 h_2 h 2 .
Application: regions of an image
We apply the relevance by region of § 4.1 to the image classifier of § 3.4 , to read which hidden units carry each part of an image.
The regions come from the Oxford-IIIT Pet dataset [35 ] , whose photographs carry two annotations: a trimap, which separates the animal, its boundary and the background, and a bounding box of the head of the animal. They fix the regions at the resolution of the pixels: the head is the set of pixels of the animal (foreground and boundary of the trimap) inside the head box, enlarged by 10% of its size on each side to take in the ears, which the annotated boxes leave out; the body is the set of the other pixels of the animal, the background the remaining pixels, and the bias nodes form one more region. Photographs and annotations are resized and cropped to 224 × 224 224 \times224 224 × 224 pixels in the same way. The trainval photographs are shuffled once and examined in that order; a photograph is kept if it has a head box, if VGG-16 predicts a cat or dog class, with x o > 0 x_o > 0 x o > 0 , if the crop keeps at least 95% of the head box, and if the head and the body cover at least 2% and 15% of the image, which leaves out the close-ups, where the animal outside the head box is little more than the ears; 200 photographs are kept (63 cats and 137 dogs) out of 581 examined, 2 having no head box, 24 not being predicted as a cat or a dog, 257 having the head cut by the crop and 98 too small a head or body.
Corollary 4.1 applies to L R P z + \mathrm{LRP}_{z+} LRP z + and to the L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRP}_{N_{\alpha(\bar{x}),\beta(1-\bar{x})}} LRP N α ( x ˉ ) , β ( 1 − x ˉ ) of § 3.4 : the hidden activations of VGG-16 being non-negative and x o > 0 x_o > 0 x o > 0 , its weights x ˉ v \bar{x}_v x ˉ v and 1 − x ˉ v 1-\bar{x}_v 1 − x ˉ v lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and its reference is non-negative, unlike that of L R P N 2 , − 1 \mathrm{LRP}_{N_{2,-1}} LRP N 2 , − 1 . For every region M M M and every node v v v of a cut, the region relevance μ v M = μ v R ( v ) ( M ) \mu_v^M=\mu_vR^{(v)}(M) μ v M = μ v R ( v ) ( M ) is the backward relevance times one forward potential, so that the three regions of pixels cost three forward passes, batched. Every path from a pixel to o o o meets a whole convolutional layer exactly once, which is all that the cut identity of Corollary 4.1 (ii) needs for a region of pixels (the paths from the bias nodes of the upper layers do not end in it); we use conv4_3 and conv5_3, summing μ v M \mu_v^M μ v M over the positions of each of their 512 channels c c c into μ c M \mu_c^M μ c M .
On each photograph and for each rule, we record the share of x o x_o x o carried by each region at conv5_3; the share of the pixel relevance that falls in the head, and its density relative to the body, i.e. the share of the pixel relevance per unit area of the head divided by that of the body; the number of channels of conv5_3 and conv4_3 that carry half of the evidence of the head; the purity of the ten channels that carry most of it, the mean share of their evidence lying in the head; and the number of channels common to the ten leading channels of the head and of the body. Table 2 gives their medians and quartiles over the 200 photographs.
Table 2. Relevance by region on the 200 Oxford-IIIT Pet photographs (Figure 16 ): median and, in parentheses, quartiles over the photographs. Area of the regions; share of x o x_o x o carried by each region and by the bias nodes, at conv5_3; for the head, its share of the pixel relevance, its relevance per unit area divided by that of the body, the number of channels of conv5_3 and conv4_3 that carry half of its evidence, the mean share of their evidence lying in the head over its ten leading channels (purity), and the number of channels common to the ten leading channels of the head and of the body.
The cut identity holds to within 8 ⋅ 10 − 7 x o 8 \cdot10^{-7}x_o 8 ⋅ 1 0 − 7 x o for both rules, at both layers, on every photograph. Under L R P z + \mathrm{LRP}_{z+} LRP z + , the head, which covers 21% of the image (median), carries 42% of x o x_o x o and 51% of the relevance of the pixels; per unit area, it receives 2.5 times as much relevance as the body (median), and more than the body
Table 2: Relevance by region on the 200 Oxford-IIIT Pet photographs (Figure 16 ): median and, in parentheses, quartiles over the photographs. Area of the regions; share of x o x_o x o carried by each region and by the bias nodes, at conv5_3; for the head, its share of the pixel relevance, its relevance per unit area divided by that of the body, the number of channels of conv5_3 and conv4_3 that carry half of its evidence, the mean share of their evidence lying in the head over its ten leading channels (purity), and the number of channels common to the ten leading channels of the head and of the body.
Figure 16. Relevance by region on VGG-16 (Corollary 4.1 ), Oxford-IIIT Pet. Left, top: the regions given by the annotations of the dataset on the first two cats and the first two dogs of the sample: head (blue, the pixels of the animal inside the enlarged head box), body (orange, the other pixels of the animal) and background. Left, bottom: share of x o x_o x o carried by each region and by the bias nodes, in percent, for L R P z + \mathrm{LRP}_{z+} LRP z + and L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRP}_{N_{\alpha(\bar{x}),\beta(1-\bar{x})}} LRP N α ( x ˉ ) , β ( 1 − x ˉ ) , computed at conv5_3. Right: share of the pixel relevance falling in the head against the area of the head, on the 200 photographs; dashed: equal shares.
on 198 of the 200 photographs (Figure 16 , right). Half of its evidence goes through 24 of the 512 channels of conv5_3 (65 at conv4_3), and the ten leading ones draw 76% of their evidence from the head, more than the share of the head in the pixel relevance – the purity of a channel whose evidence would be spread like the whole map – on 200 of the 200 photographs. The head and the body are carried by largely different channels, 3 of their ten leading channels being common (median). The concept maps of the leading channels of the head (Figure 17 ) lie on the face, those of the body on the rest of the animal, and a channel can carry evidence from both: a row of the table says where the evidence of a channel lies, as concept relevance propagation does ([1 ] ), and a column which channels carry the evidence of a region, with exact accounting in both directions. LRPN α ( x ˉ ) , β ( 1 − x ˉ ) _{\mathrm{N}\alpha(\bar{x}),\beta(1-\bar{x})} N α ( x ˉ ) , β ( 1 − x ˉ ) gives a close decomposition, more diffuse as its pixel maps are (§ 3.4 ): the bias nodes keep 9% of x o x_o x o instead of 17%, the body and the background gain a few points, the head carries 42% of x o x_o x o and 46% of the pixel relevance, with a density 2.1 times that of the body, and its evidence needs more channels (31 at conv5_3, 87 at conv4_3), with a similar purity (73%) and 2 channels common to the head and the body. The relevance by region therefore separates the rules as their pixel maps do, and says in addition through which hidden units each part of the image reaches the output. These shares account for the flow of the rule chosen, on this sample and these masks; they are not a causal test of the units.
Figure 17. Which conv5_3 channels carry the evidence of the head, for L R P z + \mathrm{LRP}_{z+} LRP z + (left) and L R P N α ( x ˉ ) , β ( 1 − x ˉ ) \mathrm{LRPN}_{\alpha(\bar{x}),\beta(1-\bar{x})} LRPN α ( x ˉ ) , β ( 1 − x ˉ ) (right), on the first cat and the first dog of Figure 16 . Bars: the eight channels c c c of largest region relevance μ C M \mu_C^M μ C M for M = h e a d M = \mathrm{head} M = head , each bar being the row ( μ C ′ M ) M ′ (\mu_{C'}^M)_{M'} ( μ C ′ M ) M ′ of the table – the relevance of the channel split by the region its evidence comes from, the bias nodes upstream included –, normalized by x o x_o x o . Below: the evidence of the two leading channels of the head and of the body, i.e. the pixel relevance obtained by propagating the relevance of that channel alone (concept relevance); the channels of the body are ranked by their own column, and need not appear among the bars.
Conclusion
We have shown that relevance propagation for feed-forward neural networks, classical or otherwise, is naturally cast as an entropic projection: the redistribution of a forward activation flow onto a set of admissible relevance flows on the network’s active computational graph, measured by a relative entropy between backward transition laws, which, for a non-negative reference, is a relative entropy on backward paths, as in Schrödinger’s problem of entropic optimal transport. This variational principle rests on a measure-theoretic Filippov formalization of feed-forward networks (§ 2 ), which covers smooth and piecewise-smooth architectures, such as ReLU and max-pooling networks, alongside outright discontinuous ones, such as hard-routed Mixture-of-Experts, and which, recast as a discrete-time optimal control problem, also characterizes backpropagation as the adjoint equation of a Lagrangian relaxation. On this formalization, the classical L R P z + \mathrm{LRP}_{z+} LRP z + and L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β rules are the unique solutions of relative-entropy minimization problems between backward transition laws, and so is L R P 0 \mathrm{LRP}_{0} LRP 0 , its reference being the signed forward contributions themselves (§ 3 ); taking the relative entropy edge by edge, as in entropically regularized flows, yields a new rule, Sinkhorn Relevance Propagation (§ 3.3 ).
The same principle lets the weights of L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β vary from node to node, a family, L R P N α , β \mathrm{LRPN}_{\alpha,\beta} LRPN α , β , that contains L R P 0 \mathrm{LRP}_{0} LRP 0 , L R P z + \mathrm{LRP}_{z+} LRP z + and L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β and keeps the sign of the output whenever its weights lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] (Corollary 3.4); with weights that grow with the activity of the node, a more active node sends more of its relevance to its excitatory contributions, a Hebbian-inspired reading. On networks of ReLU and max-pooling nodes, each rule of this family is an input times a modified backward pass (Corollary 3.5): L R P 0 \mathrm{LRP}_{0} LRP 0 is Gradient× \times × Input, linear on each region of constant gates, with an explicit jump across an isolated kink, while L R P z + \mathrm{LRP}_{z+} LRP z + is continuous there and, when the maxima are non-negative, bounded by the input times a path sum of the weights; where the winner of a max-pooling node changes, every rule may jump, the relevance of the node moving from one winner to the other.
Aggregated over a region of the input, the flow of a non-negative reference tells which hidden units carry the evidence of that region, with exact accounting; the flow conditioned on the region is a Schrödinger bridge, whose potentials follow from one forward pass (§ 4.1 ).
The applications of § 3.4 and § 4.2 check these statements on VGG-16: the conservation identities hold to round-off, and a perturbation of standard deviation 10 − 5 10^{-5} 1 0 − 5 moves the L R P 0 \mathrm{LRP}_{0} LRP 0 map by 1.5%, against 0.06 to 0.11% for the other rules, consistently with Corollary 3.5. They also show that the choice of the reference is not a detail: the rules differ in the sign of their maps, in the share of the output they leave to the bias nodes, in their faithfulness, in their localization and in their dependence on the class. SRP follows the edges and textures of the lower layers and is less faithful than L R P z + \mathrm{LRP}_{z+} LRP z + ; with the scaled reference, S R P 1 \mathrm{SRP}_{1} SRP 1 comes close to the faithfulness of L R P z + \mathrm{LRP}_{z+} LRP z + , at a higher cost. Node-wise weights normalized by the activations of their layer bring L R P N α , β \mathrm{LRPN}_{\alpha,\beta} LRPN α , β close to L R P z + \mathrm{LRP}_{z+} LRP z + on VGG-16. On ConvNeXt-Tiny, whose hidden values take both signs, L R P N α , β \mathrm{LRPN}_{\alpha,\beta} LRPN α , β conserves the output and these weights give the highest ABPC of the rules tested, whereas L R P z + \mathrm{LRP}_{z+} LRP z + loses part of the output where the hypothesis of Corollary 3.4(ii) fails and L R P 0 \mathrm{LRP}_{0} LRP 0 is no longer Gradient× \times × Input. On 200 annotated photographs of cats and dogs, the relevance by region gives the head about 2.5 times as much relevance per pixel as the body (median), carried by largely different channels.
Two directions follow from this work. First, the formalism of § 2 reaches well beyond the image classifiers evaluated here: hard-routed Mixture-of-Experts, for which its non-generic machinery was designed, and normalization layers entering the graph as nodes, rather than frozen at the input as in § 3.4 ; changing the reference of the projection while keeping its constraint gives further conservative rules for them, which the indicators of § 3.4 can compare. Second, the same entropic-projection language that recovers L R P α , β \mathrm{LRP}_{\alpha,\beta} LRP α , β suggests relaxing the conservation constraint into a soft, tunable marginal penalty, in the sense of unbalanced optimal transport ([12 , 41 , 28 ] ), when an exactly conserved flow is not what the modelling goal calls for. The characterization is relative to the reference: it does not by itself select one, nor guarantee faithfulness, which the applications compare. Taken together, these results suggest that optimal transport is not merely a convenient reformulation of existing relevance-propagation heuristics, but the organizing principle from which they, and their natural extensions, follow.
Proofs for Section 2: Feed-Forward Filippov Neural Networks
Proof of Theorem 2.2
Let Φ = ± 1 \Phi=\pm1 Φ = ± 1 on { ± z > 0 } ⊂ R \{\pm z>0\}\subset\mathbb{R} { ± z > 0 } ⊂ R , Σ = { 0 } \Sigma=\{0\} Σ = { 0 } . The closed convex hull fills the jump: F [ Φ ] ( 0 ) = [ − 1 , 1 ] \mathcal{F}[\Phi](0)=[-1,1] F [ Φ ] ( 0 ) = [ − 1 , 1 ] . The intersection over δ \delta δ localizes: F [ Φ ] ( 1 2 ) = { 1 } \mathcal{F}[\Phi](\frac{1}{2})=\{1\} F [ Φ ] ( 2 1 ) = { 1 } , although Φ ( B ( 1 2 , δ ) ∖ Σ ) = { − 1 , 1 } \Phi(B(\frac{1}{2},\delta)\setminus\Sigma)=\{-1,1\} Φ ( B ( 2 1 , δ ) ∖ Σ ) = { − 1 , 1 } for δ > 1 2 \delta>\frac{1}{2} δ > 2 1 . The intersection over null sets ignores negligible values: if Φ ~ = Φ \widetilde{\Phi}=\Phi Φ = Φ except Φ ~ ( 1 / ℓ ) = 5 \widetilde{\Phi}(1/\ell)=5 Φ ( 1/ ℓ ) = 5 , ℓ ∈ N ∗ \ell\in\mathbb{N}^{*} ℓ ∈ N ∗ , then co Φ ~ ( B ( 0 , δ ) ∖ Σ ) = [ − 1 , 5 ] \operatorname{co}\widetilde{\Phi}(B(0,\delta)\setminus\Sigma)=[-1,5] co Φ ( B ( 0 , δ ) ∖ Σ ) = [ − 1 , 5 ] for every δ \delta δ , and F [ Φ ~ ] ( 0 ) = [ − 1 , 1 ] \mathcal{F}[\widetilde{\Phi}](0)=[-1,1] F [ Φ ] ( 0 ) = [ − 1 , 1 ] ; in general F \mathcal{F} F only depends on the class of the map modulo null sets.
Let x ∈ Ω x\in\Omega x ∈ Ω and J : = J Φ ( x ) J:=J_{\Phi}(x) J := J Φ ( x ) . The family ( Ω ( k ) ) k (\Omega^{(k)})_k ( Ω ( k ) ) k being locally finite, J J J is finite and there is δ 0 > 0 \delta_0>0 δ 0 > 0 with B ( x , δ 0 ) ⊂ Ω B(x,\delta_0)\subset\Omega B ( x , δ 0 ) ⊂ Ω and J Φ ( y ) ⊆ J J_{\Phi}(y)\subseteq J J Φ ( y ) ⊆ J on B ( x , δ 0 ) B(x,\delta_0) B ( x , δ 0 ) . If J = ∅ J=\varnothing J = ∅ , a neighborhood of x x x lies in Σ \Sigma Σ , against μ ( Σ ) = 0 \mu(\Sigma)=0 μ ( Σ ) = 0 .
Let Ψ : Ω ∖ Σ → R p \Psi:\Omega\setminus\Sigma\to\mathbb{R}^{p} Ψ : Ω ∖ Σ → R p agree on each Ω ( k ) \Omega^{(k)} Ω ( k ) with a map Ψ ( k ) \Psi^{(k)} Ψ ( k ) continuous on Ω ( k ) ∩ Ω \Omega^{(k)}\cap\Omega Ω ( k ) ∩ Ω ; then
⋂ μ ( N ) = 0 ⋂ δ > 0 co Ψ ( B ( x , δ ) ∖ N ) = co { Ψ ( k ) ( x ) : k ∈ J } = : C . \bigcap_{\mu(\mathcal{N})=0}\ \bigcap_{\delta>0}\operatorname{co}\Psi(B(x,\delta)\setminus\mathcal{N})=\operatorname{co}\{\Psi^{(k)}(x):k\in J\}=:C. μ ( N ) = 0 ⋂ δ > 0 ⋂ co Ψ ( B ( x , δ ) ∖ N ) = co { Ψ ( k ) ( x ) : k ∈ J } =: C .
Let k ∈ J k\in J k ∈ J , N \mathcal{N} N null and δ , ε > 0 \delta,\varepsilon>0 δ , ε > 0 . For δ ′ ≤ δ \delta'\leq\delta δ ′ ≤ δ small, the open set B ( x , δ ′ ) ∩ Ω ( k ) B(x,\delta')\cap\Omega^{(k)} B ( x , δ ′ ) ∩ Ω ( k ) is nonempty, hence not contained in N \mathcal{N} N , and a point y y y of it outside N \mathcal{N} N has ∥ Ψ ( y ) − Ψ ( k ) ( x ) ∥ < ε \|\Psi(y)-\Psi^{(k)}(x)\|<\varepsilon ∥Ψ ( y ) − Ψ ( k ) ( x ) ∥ < ε . So Ψ ( k ) ( x ) \Psi^{(k)}(x) Ψ ( k ) ( x ) lies in the left-hand side, which is convex, and C C C too. Conversely, for ε > 0 \varepsilon>0 ε > 0 and δ \delta δ small, Ψ ( B ( x , δ ) ∖ Σ ) \Psi(B(x,\delta)\setminus\Sigma) Ψ ( B ( x , δ ) ∖ Σ ) lies in the closed ε \varepsilon ε -neighborhood of the compact convex set C C C , closed and convex; hence so does the left-hand side, for every ε \varepsilon ε .
Theorem Theorem 2.2 (i) is the case Ψ = Φ \Psi=\Phi Ψ = Φ and the case Ψ = ( Φ , D Φ ) \Psi=(\Phi,D\Phi) Ψ = ( Φ , D Φ ) , Ψ ( k ) = ( Φ ( k ) , D Φ ( k ) ) \Psi^{(k)}=(\Phi^{(k)},D\Phi^{(k)}) Ψ ( k ) = ( Φ ( k ) , D Φ ( k ) ) , continuous since Φ ( k ) \Phi^{(k)} Φ ( k ) is C 1 C^{1} C 1 on U ( k ) U^{(k)} U ( k ) . If x ∉ Σ x\notin\Sigma x ∈ / Σ , then x x x lies in a single open region Ω ( k 0 ) \Omega^{(k_0)} Ω ( k 0 ) , so J = { k 0 } J=\{k_0\} J = { k 0 } , and ( Φ , D Φ ) = ( Φ ( k 0 ) , D Φ ( k 0 ) ) (\Phi,D\Phi)=(\Phi^{(k_0)},D\Phi^{(k_0)}) ( Φ , D Φ ) = ( Φ ( k 0 ) , D Φ ( k 0 ) ) there. ■
Example for Theorem 2.7
Take two inputs i 1 , i 2 i_1,i_2 i 1 , i 2 , two hidden nodes v 1 , v 2 v_1,v_2 v 1 , v 2 and one output o o o , with w i 1 , v 1 = w i 2 , v 1 = w i 1 , v 2 = 1 w_{i_1,v_1}=w_{i_2,v_1}=w_{i_1,v_2}=1 w i 1 , v 1 = w i 2 , v 1 = w i 1 , v 2 = 1 , w i 2 , v 2 = − 1 w_{i_2,v_2}=-1 w i 2 , v 2 = − 1 , w v 1 , o = 3 w_{v_1,o}=3 w v 1 , o = 3 , w v 2 , o = 2 w_{v_2,o}=2 w v 2 , o = 2 , and
Φ b ( 1 ) ≡ − 1 on { a + c < 0 } , Φ b ( 2 ) ≡ + 1 on { a + c > 0 } ( b = v 1 , v 2 ) , \Phi_b^{(1)}\equiv-1\ \text{on }\{a+c<0\},\qquad
\Phi_b^{(2)}\equiv+1\ \text{on }\{a+c>0\}\qquad(b=v_1,v_2), Φ b ( 1 ) ≡ − 1 on { a + c < 0 } , Φ b ( 2 ) ≡ + 1 on { a + c > 0 } ( b = v 1 , v 2 ) ,
Φ o ( 1 ) = ( z 1 + z 2 ) 2 on { z 1 + z 2 < 1 } , Φ o ( 2 ) = z 1 + z 2 + 3 on { z 1 + z 2 > 1 } , \Phi_o^{(1)}=(z_1+z_2)^2\ \text{on }\{z_1+z_2<1\},\qquad
\Phi_o^{(2)}=z_1+z_2+3\ \text{on }\{z_1+z_2>1\}, Φ o ( 1 ) = ( z 1 + z 2 ) 2 on { z 1 + z 2 < 1 } , Φ o ( 2 ) = z 1 + z 2 + 3 on { z 1 + z 2 > 1 } ,
so that z v 1 = ( x i 1 , x i 2 ) z_{v_1}=(x_{i_1},x_{i_2}) z v 1 = ( x i 1 , x i 2 ) , z v 2 = ( x i 1 , − x i 2 ) z_{v_2}=(x_{i_1},-x_{i_2}) z v 2 = ( x i 1 , − x i 2 ) and z o = ( 3 x v 1 , 2 x v 2 ) z_o=(3x_{v_1},2x_{v_2}) z o = ( 3 x v 1 , 2 x v 2 ) .
At x = ( 2 , 1 ) x=(2,1) x = ( 2 , 1 ) , x ∉ D x\notin D x ∈ / D , x v 1 = x v 2 = 1 x_{v_1}=x_{v_2}=1 x v 1 = x v 2 = 1 , z o = ( 3 , 2 ) ∈ Ω o ( 2 ) z_o=(3,2)\in\Omega_o^{(2)} z o = ( 3 , 2 ) ∈ Ω o ( 2 ) , x o = 8 x_o=8 x o = 8 , M ( x ) = { ( 2 , 2 , 2 ) } \mathcal{M}(x)=\{(2,2,2)\} M ( x ) = {( 2 , 2 , 2 )} , ∂ x o / ∂ x v 1 = 3 \partial x_o/\partial x_{v_1}=3 ∂ x o / ∂ x v 1 = 3 , ∂ x o / ∂ x v 2 = 2 \partial x_o/\partial x_{v_2}=2 ∂ x o / ∂ x v 2 = 2 , ∂ x o / ∂ x i 1 = ∂ x o / ∂ x i 2 = 0 \partial x_o/\partial x_{i_1}=\partial x_o/\partial x_{i_2}=0 ∂ x o / ∂ x i 1 = ∂ x o / ∂ x i 2 = 0 and F ~ v 1 , o ( x ) = { ( 8 , 3 ) } \widetilde{\mathcal{F}}_{v_1,o}(x)=\{(8,3)\} F v 1 , o ( x ) = {( 8 , 3 )} .
At x = ( 1 , − 1 ) x=(1,-1) x = ( 1 , − 1 ) , z v 1 = ( 1 , − 1 ) ∈ Σ v 1 z_{v_1}=(1,-1)\in\Sigma_{v_1} z v 1 = ( 1 , − 1 ) ∈ Σ v 1 , so X v 1 ( x ) = [ − 1 , 1 ] X_{v_1}(x)=[-1,1] X v 1 ( x ) = [ − 1 , 1 ] , X v 2 ( x ) = { 1 } X_{v_2}(x)=\{1\} X v 2 ( x ) = { 1 } and Z o ( x ) = [ − 3 , 3 ] × { 2 } Z_o(x)=[-3,3]\times\{2\} Z o ( x ) = [ − 3 , 3 ] × { 2 } , cut by Σ o \Sigma_o Σ o at z 1 = − 1 z_1=-1 z 1 = − 1 :
X o ( x ) = { ( z 1 + 2 ) 2 : z 1 ∈ [ − 3 , − 1 ) } ∪ = [ 0 , 1 ] co { 1 , 4 } ∪ z 1 = − 1 { z 1 + 5 : z 1 ∈ ( − 1 , 3 ] } = = ( 4 , 8 ] [ 0 , 8 ] . X_o(x)=\{(z_1+2)^2:z_1\in[-3,-1)\}\underset{=[0,1]}{\cup}\operatorname{co}\{1,4\}\underset{z_1=-1}{\cup}\{z_1+5:z_1\in(-1,3]\}\underset{=(4,8]}{=}[0,8]. X o ( x ) = {( z 1 + 2 ) 2 : z 1 ∈ [ − 3 , − 1 )} = [ 0 , 1 ] ∪ co { 1 , 4 } z 1 = − 1 ∪ { z 1 + 5 : z 1 ∈ ( − 1 , 3 ]} = ( 4 , 8 ] = [ 0 , 8 ] .
By (2) , κ v 2 = 2 \kappa_{v_2}=2 κ v 2 = 2 , κ v 1 ∈ { 1 , 2 } \kappa_{v_1}\in\{1,2\} κ v 1 ∈ { 1 , 2 } is free and κ o \kappa_o κ o is forced by z o ( κ ) ( x ) z_o^{(\kappa)}(x) z o ( κ ) ( x ) . Hence M ( x ) = { κ ′ , κ ′ ′ } \mathcal{M}(x)=\{\kappa',\kappa''\} M ( x ) = { κ ′ , κ ′′ } and, by (3) ,
Table 1 (PDF p. 35).
with g i 1 , o ( κ ′ ) = g i 1 , o ( κ ′ ′ ) = 0 g_{i_1,o}^{(\kappa')}=g_{i_1,o}^{(\kappa'')}=0 g i 1 , o ( κ ′ ) = g i 1 , o ( κ ′′ ) = 0 , so that
F ~ v 1 , o ( x ) = co { ( 1 , − 6 ) , ( 8 , 3 ) } , F ~ v 2 , o ( x ) = co { ( 1 , − 4 ) , ( 8 , 2 ) } , F ~ i 1 , o ( x ) = [ 1 , 8 ] × { 0 } . \widetilde{\mathcal{F}}_{v_1,o}(x)=\operatorname{co}\{(1,-6),(8,3)\},\qquad
\widetilde{\mathcal{F}}_{v_2,o}(x)=\operatorname{co}\{(1,-4),(8,2)\},\qquad
\widetilde{\mathcal{F}}_{i_1,o}(x)=[1,8]\times\{0\}. F v 1 , o ( x ) = co {( 1 , − 6 ) , ( 8 , 3 )} , F v 2 , o ( x ) = co {( 1 , − 4 ) , ( 8 , 2 )} , F i 1 , o ( x ) = [ 1 , 8 ] × { 0 } .
Thus { x o ( κ ) ( x ) } = { 1 , 8 } ⊂ [ 0 , 8 ] = X o ( x ) \{x_o^{(\kappa)}(x)\}=\{1,8\}\subset[0,8]=X_o(x) { x o ( κ ) ( x )} = { 1 , 8 } ⊂ [ 0 , 8 ] = X o ( x ) : the values in [ 0 , 1 ) ∪ ( 1 , 4 ] [0,1)\cup(1,4] [ 0 , 1 ) ∪ ( 1 , 4 ] come from the interior of the hull at v 1 v_1 v 1 and are reached by no selection.
At x = ( 2 , 1 ) x=(2,1) x = ( 2 , 1 ) , (6) gives λ v 1 = 3 \lambda_{v_1}=3 λ v 1 = 3 , λ v 2 = 2 \lambda_{v_2}=2 λ v 2 = 2 , λ i 1 = λ i 2 = 0 \lambda_{i_1}=\lambda_{i_2}=0 λ i 1 = λ i 2 = 0 , and Corollary Corollary 2.9 gives ∂ x o / ∂ w v 1 , o = 1 \partial x_o/\partial w_{v_1,o}=1 ∂ x o / ∂ w v 1 , o = 1 and ∂ x o / ∂ w i 1 , v 1 = 3 × 2 × 0 = 0 \partial x_o/\partial w_{i_1,v_1}=3\times2\times0=0 ∂ x o / ∂ w i 1 , v 1 = 3 × 2 × 0 = 0 . At x = ( 1 , − 1 ) x=(1,-1) x = ( 1 , − 1 ) , the recursion gives ( λ v 1 , λ v 2 ) = ( − 6 , − 4 ) (\lambda_{v_1},\lambda_{v_2})=(-6,-4) ( λ v 1 , λ v 2 ) = ( − 6 , − 4 ) for κ ′ \kappa' κ ′ and ( 3 , 2 ) (3,2) ( 3 , 2 ) for κ ′ ′ \kappa'' κ ′′ , with λ i 1 = λ i 2 = 0 \lambda_{i_1}=\lambda_{i_2}=0 λ i 1 = λ i 2 = 0 .
Proof of Theorem 2.7
By Theorem 2.2(i), F [ Φ v ] ( ζ ) \mathcal{F}[\Phi_v](\zeta) F [ Φ v ] ( ζ ) is the interval between the extremes of the Φ v ( k ) ( ζ ) \Phi_v^{(k)}(\zeta) Φ v ( k ) ( ζ ) , k ∈ J v ( ζ ) k \in J_v(\zeta) k ∈ J v ( ζ ) , nonempty and finite, and it is { Φ v ( ζ ) } \{\Phi_v(\zeta)\} { Φ v ( ζ )} at a regular ζ \zeta ζ ; this is the formula of the statement. Let the X a ( x ) X_a(x) X a ( x ) , a ∈ In ( v ) a \in\operatorname{In}(v) a ∈ In ( v ) , be nonempty and compact; so is Z v ( x ) Z_v(x) Z v ( x ) . Near ζ 0 \zeta_0 ζ 0 , J v ( ζ ) ⊆ J v ( ζ 0 ) J_v(\zeta) \subseteq J_v(\zeta_0) J v ( ζ ) ⊆ J v ( ζ 0 ) and the branches of J v ( ζ 0 ) J_v(\zeta_0) J v ( ζ 0 ) are continuous, so F [ Φ v ] ( ζ ) \mathcal{F}[\Phi_v](\zeta) F [ Φ v ] ( ζ ) lies in the ϵ \epsilon ϵ -neighborhood of F [ Φ v ] ( ζ 0 ) \mathcal{F}[\Phi_v](\zeta_0) F [ Φ v ] ( ζ 0 ) . A finite cover of Z v ( x ) Z_v(x) Z v ( x ) thus bounds X v ( x ) X_v(x) X v ( x ) ; and if y n ∈ F [ Φ v ] ( ζ n ) y_n \in\mathcal{F}[\Phi_v](\zeta_n) y n ∈ F [ Φ v ] ( ζ n ) , y n → y y_n \to y y n → y , ζ n → ζ \zeta_n \to\zeta ζ n → ζ , then y ∈ F [ Φ v ] ( ζ ) y \in\mathcal{F}[\Phi_v](\zeta) y ∈ F [ Φ v ] ( ζ ) . Hence X v ( x ) X_v(x) X v ( x ) is compact, along a topological order.
M ( x ) \mathcal{M}(x) M ( x ) is finite. Along a topological order, the predecessors of v v v fix z v ( κ ) ( x ) z_v^{(\kappa)}(x) z v ( κ ) ( x ) and J v ( z v ( κ ) ( x ) ) ≠ ∅ J_v(z_v^{(\kappa)}(x)) \ne\varnothing J v ( z v ( κ ) ( x )) = ∅ provides κ v \kappa_v κ v : M ( x ) ≠ ∅ \mathcal{M}(x) \ne\varnothing M ( x ) = ∅ . For κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , induction gives z v ( κ ) ( x ) ∈ Z v ( x ) z_v^{(\kappa)}(x) \in Z_v(x) z v ( κ ) ( x ) ∈ Z v ( x ) and x v ( κ ) ( x ) = Φ v ( κ v ) ( z v ( κ ) ( x ) ) ∈ F [ Φ v ] ( z v ( κ ) ( x ) ) ⊆ X v ( x ) x_v^{(\kappa)}(x) = \Phi_v^{(\kappa_v)}(z_v^{(\kappa)}(x)) \in\mathcal{F}[\Phi_v](z_v^{(\kappa)}(x)) \subseteq X_v(x) x v ( κ ) ( x ) = Φ v ( κ v ) ( z v ( κ ) ( x )) ∈ F [ Φ v ] ( z v ( κ ) ( x )) ⊆ X v ( x ) .
Replace the value of v v v by t t t and recompute its descendants with the branches κ u \kappa_u κ u . At t 0 = x v ( κ ) ( x ) t_0=x_v^{(\kappa)}(x) t 0 = x v ( κ ) ( x ) , z u ( t 0 ) = z u ( κ ) ( x ) ∈ U u ( κ u ) z_u(t_0)=z_u^{(\kappa)}(x)\in U_u^{(\kappa_u)} z u ( t 0 ) = z u ( κ ) ( x ) ∈ U u ( κ u ) , so each x u ( t ) x_u(t) x u ( t ) is C 1 C^1 C 1 near t 0 t_0 t 0 , x ˙ v = 1 \dot{x}_v=1 x ˙ v = 1 , x ˙ i = 0 \dot{x}_i=0 x ˙ i = 0 at the inputs i ≠ v i\ne v i = v , and x ˙ u = ∑ a ∈ In ( u ) w a , u [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] a x ˙ a \dot{x}_u=\sum_{a\in\operatorname{In}(u)}w_{a,u}[D\Phi_u^{(\kappa_u)}(z_u^{(\kappa)}(x))]_a\dot{x}_a x ˙ u = ∑ a ∈ In ( u ) w a , u [ D Φ u ( κ u ) ( z u ( κ ) ( x )) ] a x ˙ a . A path of P ( v , u ) \mathcal{P}(v,u) P ( v , u ) is a path of P ( v , a ) \mathcal{P}(v,a) P ( v , a ) followed by ( a , u ) (a,u) ( a , u ) , which gives (3) by induction. F v , o ( x ) \mathcal{F}_{v,o}(x) F v , o ( x ) is the hull of a finite nonempty set.
Let D v : = { x : Z v ( x ) ∩ Σ v ≠ ∅ } D_v:=\{x:Z_v(x)\cap\Sigma_v\ne\varnothing\} D v := { x : Z v ( x ) ∩ Σ v = ∅ } and A v A_v A v the union of the D a D_a D a over the non-input strict ancestors a a a of v v v . Let x ∉ A v x\notin A_v x ∈ / A v and κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) . Along a topological order, each ancestor a a a of v v v has Z a ( x ) = { z a ( κ ) ( x ) } Z_a(x)=\{z_a^{(\kappa)}(x)\} Z a ( x ) = { z a ( κ ) ( x )} , outside Σ a \Sigma_a Σ a ; by Theorem 2.2(ii), κ a \kappa_a κ a is its unique active branch and X a ( x ) = { x a ( κ ) ( x ) } X_a(x)=\{x_a^{(\kappa)}(x)\} X a ( x ) = { x a ( κ ) ( x )} . Hence Z v ( x ) = { z v ( κ ) ( x ) } Z_v(x)=\{z_v^{(\kappa)}(x)\} Z v ( x ) = { z v ( κ ) ( x )} and
D v ⊆ A v ∪ { x : z v ( κ ) ( x ) ∈ Σ v for some κ ∈ M ( x ) } . D_v \subseteq A_v \cup\{x:z_v^{(\kappa)}(x)\in\Sigma_v\text{ for some }\kappa\in\mathcal{M}(x)\}. D v ⊆ A v ∪ { x : z v ( κ ) ( x ) ∈ Σ v for some κ ∈ M ( x )} .
Non-degeneracy gives μ ( D v ) = 0 \mu(D_v)=0 μ ( D v ) = 0 by induction along a topological order, and μ ( D ) = 0 \mu(D)=0 μ ( D ) = 0 .
If x ∉ D x\notin D x ∈ / D , this holds at every node: every X u ( x ) X_u(x) X u ( x ) is a singleton, every z u ( x ) z_u(x) z u ( x ) lies in an open region Ω u ( κ u ) \Omega_u^{(\kappa_u)} Ω u ( κ u ) , and M ( x ) = { κ } \mathcal{M}(x)=\{\kappa\} M ( x ) = { κ } , κ u = k u \kappa_u=k_u κ u = k u . These regions being open, κ \kappa κ remains the network when the value of v v v moves near t 0 t_0 t 0 , so ∂ x o / ∂ x v ( x ) = g v , o ( κ ) ( x ) \partial x_o/\partial x_v(x)=g_{v,o}^{(\kappa)}(x) ∂ x o / ∂ x v ( x ) = g v , o ( κ ) ( x ) .
Assume now that the branches agree on the switching sets. Then F [ Φ v ] ( ζ ) \mathcal{F}[\Phi_v](\zeta) F [ Φ v ] ( ζ ) is a singleton for every ζ \zeta ζ , the extension of Φ v \Phi_v Φ v to Σ v \Sigma_v Σ v is continuous, and, along a topological order, X v ( x ) = { x v ( κ ) ( x ) } X_v(x)=\{x_v^{(\kappa)}(x)\} X v ( x ) = { x v ( κ ) ( x )} for every κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , continuous in x x x . Let x 0 ∈ R d x_0\in\mathbb{R}^d x 0 ∈ R d . On a ball B B B around x 0 x_0 x 0 , every selection admissible at a point belongs to M ( x 0 ) \mathcal{M}(x_0) M ( x 0 ) : otherwise a single κ ∉ M ( x 0 ) \kappa\notin\mathcal{M}(x_0) κ ∈ / M ( x 0 ) is admissible at points x n → x 0 x_n\to x_0 x n → x 0 ; then z u ( κ ) ( x n ) = z u ( x n ) ∈ Ω u ( κ u ) z_u^{(\kappa)}(x_n)=z_u(x_n)\in\Omega_u^{(\kappa_u)} z u ( κ ) ( x n ) = z u ( x n ) ∈ Ω u ( κ u ) , where z u z_u z u is the pre-activation of the network, continuous, so z u ( x 0 ) ∈ Ω u ( κ u ) z_u(x_0)\in\Omega_u^{(\kappa_u)} z u ( x 0 ) ∈ Ω u ( κ u ) , and induction gives z u ( κ ) ( x 0 ) = z u ( x 0 ) z_u^{(\kappa)}(x_0)=z_u(x_0) z u ( κ ) ( x 0 ) = z u ( x 0 ) , the branches agreeing there: κ ∈ M ( x 0 ) \kappa\in\mathcal{M}(x_0) κ ∈ M ( x 0 ) . So x o x_o x o is on B B B a continuous selection of the C 1 C^1 C 1 maps x o ( κ ) x_o^{(\kappa)} x o ( κ ) , κ ∈ M ( x 0 ) \kappa\in\mathcal{M}(x_0) κ ∈ M ( x 0 ) , hence locally Lipschitz (Scholtes, [40 ] ).
If x ∉ D x\notin D x ∈ / D , the regions being open, x o = x o ( κ ) x_o=x_o^{(\kappa)} x o = x o ( κ ) near x x x , C 1 C^1 C 1 , of gradient ( g i , o ( κ ) ( x ) ) i (g_{i,o}^{(\kappa)}(x))_{i} ( g i , o ( κ ) ( x ) ) i . Let x x x be arbitrary, g κ : = ( g i , o ( κ ) ( x ) ) i ∈ I ∖ B g^\kappa:=(g_{i,o}^{(\kappa)}(x))_{i\in I\setminus\mathcal{B}} g κ := ( g i , o ( κ ) ( x ) ) i ∈ I ∖ B , v ∈ R d v\in\mathbb{R}^d v ∈ R d , m : = max κ ∈ M ( x ) ⟨ g κ , v ⟩ m:=\max_{\kappa\in\mathcal{M}(x)}\langle g^\kappa,v\rangle m := max κ ∈ M ( x ) ⟨ g κ , v ⟩ and ϵ > 0 \epsilon>0 ϵ > 0 . On a ball B B B around x x x , x o x_o x o is a selection of the x o ( κ ) x_o^{(\kappa)} x o ( κ ) , κ ∈ M ( x ) \kappa\in\mathcal{M}(x) κ ∈ M ( x ) , and x o ( κ ) ( y ′ ) − x o ( κ ) ( y ) ≤ ⟨ g κ , y ′ − y ⟩ + ϵ ∥ y ′ − y ∥ x_o^{(\kappa)}(y')-x_o^{(\kappa)}(y)\le\langle g^\kappa,y'-y\rangle+\epsilon\lVert y'-y\rVert x o ( κ ) ( y ′ ) − x o ( κ ) ( y ) ≤ ⟨ g κ , y ′ − y ⟩ + ϵ ∥ y ′ − y ∥ . Let y + s v ∈ B y+sv\in B y + s v ∈ B , s ∈ [ 0 , t ] s\in[0,t] s ∈ [ 0 , t ] . Near y + s v y+sv y + s v , by continuity, x o x_o x o only equals those maps with the value x o ( y + s v ) x_o(y+sv) x o ( y + s v ) at y + s v y+sv y + s v , so s ↦ x o ( y + s v ) s\mapsto x_o(y+sv) s ↦ x o ( y + s v ) , continuous, has right increments at most m + ϵ ∥ v ∥ m+\epsilon\lVert v\rVert m + ϵ ∥ v ∥ per unit length and x o ( y + t v ) − x o ( y ) ≤ ( m + ϵ ∥ v ∥ ) t x_o(y+tv)-x_o(y)\le(m+\epsilon\lVert v\rVert)t x o ( y + t v ) − x o ( y ) ≤ ( m + ϵ ∥ v ∥) t . Therefore
x o ∘ ( x ; v ) : = lim sup y → x , t ↓ 0 x o ( y + t v ) − x o ( y ) t ≤ m , x_o^\circ(x;v):=\limsup_{\substack{y\to x,\ t\downarrow0}}\frac{x_o(y+tv)-x_o(y)}{t}\le m, x o ∘ ( x ; v ) := y → x , t ↓ 0 lim sup t x o ( y + t v ) − x o ( y ) ≤ m ,
and every ξ ∈ ∂ ∘ x o ( x ) = { ξ : ⟨ ξ , v ⟩ ≤ x o ∘ ( x ; v ) ∀ v } \xi\in\partial^\circ x_o(x)=\{\xi:\langle\xi,v\rangle\le x_o^\circ(x;v)\ \forall v\} ξ ∈ ∂ ∘ x o ( x ) = { ξ : ⟨ ξ , v ⟩ ≤ x o ∘ ( x ; v ) ∀ v } (Clarke, [13 ] ) lies in co { g κ : κ ∈ M ( x ) } \operatorname{co}\{g^\kappa:\kappa\in\mathcal{M}(x)\} co { g κ : κ ∈ M ( x )} by separation. ■
Proof of Theorem 2.8
ζ v ( ξ ( κ ) ) = z v ( κ ) ( x ) ∈ U v ( κ v ) \zeta_v(\xi^{(\kappa)})=z_v^{(\kappa)}(x)\in U_v^{(\kappa_v)} ζ v ( ξ ( κ ) ) = z v ( κ ) ( x ) ∈ U v ( κ v ) and ζ v \zeta_v ζ v is linear, so L ( κ ) \mathcal{L}^{(\kappa)} L ( κ ) is C 1 C^1 C 1 near ξ ( κ ) \xi^{(\kappa)} ξ ( κ ) . The variable ξ v \xi_v ξ v enters the cost if v = o v=o v = o , the constraint of v v v with coefficient − λ v -\lambda_v − λ v , and the constraint of each u ∈ Out ( v ) u\in\operatorname{Out}(v) u ∈ Out ( v ) through w v , u ξ v w_{v,u}\xi_v w v , u ξ v :
∂ L ( κ ) ∂ ξ v ( ξ ( κ ) , λ ) = 1 { v = o } − λ v + ∑ u ∈ Out ( v ) λ u w v , u [ D Φ u ( κ u ) ( z u ( κ ) ( x ) ) ] v . \frac{\partial\mathcal{L}^{(\kappa)}}{\partial\xi_v}(\xi^{(\kappa)},\lambda)=\mathbf{1}_{\{v=o\}}-\lambda_v+\sum_{u\in\operatorname{Out}(v)}\lambda_u w_{v,u}\left[D\Phi_u^{(\kappa_u)}(z_u^{(\kappa)}(x))\right]_v. ∂ ξ v ∂ L ( κ ) ( ξ ( κ ) , λ ) = 1 { v = o } − λ v + u ∈ Out ( v ) ∑ λ u w v , u [ D Φ u ( κ u ) ( z u ( κ ) ( x )) ] v .
which vanishes for every v v v if and only if (6) holds. The right-hand side of (6) only involves the successors of v v v : the solution is unique, along a reverse topological order. A path of P ( v , o ) \mathcal{P}(v,o) P ( v , o ) , v ≠ o v\ne o v = o , is an edge ( v , u ) (v,u) ( v , u ) followed by a path of P ( u , o ) \mathcal{P}(u,o) P ( u , o ) , and P ( o , o ) \mathcal{P}(o,o) P ( o , o ) is the empty path: g v , o ( κ ) ( x ) g^{(\kappa)}_{v,o}(x) g v , o ( κ ) ( x ) solves the recursion. The rest follows from (4) and Theorem 2.7 (iii).
Proof of Corollary 2.9
The z n z_n z n lying in open regions, κ \kappa κ remains the network for w u , v w_{u,v} w u , v near its value, the node values ξ ( w u , v ) \xi(w_{u,v}) ξ ( w u , v ) are C 1 C^1 C 1 and satisfy the state equations, so that x o = L ( κ ) ( ξ ( w u , v ) , λ ( κ ) , w u , v ) x_o=\mathcal{L}^{(\kappa)}(\xi(w_{u,v}),\lambda^{(\kappa)},w_{u,v}) x o = L ( κ ) ( ξ ( w u , v ) , λ ( κ ) , w u , v ) . Differentiating, ∂ L ( κ ) / ∂ ξ n = 0 \partial\mathcal{L}^{(\kappa)}/\partial\xi_n=0 ∂ L ( κ ) / ∂ ξ n = 0 leaves
∂ x o ∂ w u , v = ∂ L ( κ ) ∂ w u , v = λ v [ D Φ v ( κ ) ( z v ) ] u x u . \frac{\partial x_o}{\partial w_{u,v}}=\frac{\partial\mathcal{L}^{(\kappa)}}{\partial w_{u,v}}=\lambda_v\left[D\Phi_v^{(\kappa)}(z_v)\right]_u x_u. ∂ w u , v ∂ x o = ∂ w u , v ∂ L ( κ ) = λ v [ D Φ v ( κ ) ( z v ) ] u x u .
(6) reads λ v = 1 { v = o } + ∑ r ∈ Out ( v ) w v , r δ v , r \lambda_v=\mathbf{1}_{\{v=o\}}+\sum_{r\in\operatorname{Out}(v)}w_{v,r}\delta_{v,r} λ v = 1 { v = o } + ∑ r ∈ Out ( v ) w v , r δ v , r ; multiplied by [ D Φ v ( κ ) ( z v ) ] u \left[D\Phi_v^{(\kappa)}(z_v)\right]_u [ D Φ v ( κ ) ( z v ) ] u , it is (7) . Conversely, for a solution d d d of (7) , 1 { v = o } + ∑ r w v , r d v , r \mathbf{1}_{\{v=o\}}+\sum_r w_{v,r}d_{v,r} 1 { v = o } + ∑ r w v , r d v , r solves (6) , so it is λ v \lambda_v λ v , and d = δ d=\delta d = δ .
Proofs for Section 3: Relevance Propagation as an Entropic Projection
Proof of Proposition 3.2
Let Σ \Sigma Σ be the sum of the divergences over H ( x ) H(x) H ( x ) and of the inflows at the outputs. Every edge of E ( x ) E(x) E ( x ) counts once with + + + at its head and with − - − at its tail if the tail is hidden, so Σ = ∑ u ∈ I ∑ v : u → v π u , v ∘ = ∑ u ∈ I μ u ∘ \Sigma=\sum_{u\in\mathcal{I}}\sum_{v:u\to v}\pi^\circ_{u,v}=\sum_{u\in\mathcal{I}}\mu^\circ_u Σ = ∑ u ∈ I ∑ v : u → v π u , v ∘ = ∑ u ∈ I μ u ∘ . The constraints of A o ( x ) \mathcal{A}_o(x) A o ( x ) give Σ = x o \Sigma=x_o Σ = x o . The same holds with x o x_o x o replaced by any b ∈ R b\in\mathbb{R} b ∈ R .
Proof of Theorem 3.3
Fix κ \kappa κ and drop it; π ∗ : = π o , κ , ∗ \pi^*:=\pi^{o,\kappa,*} π ∗ := π o , κ , ∗ . If ρ ~ v ≠ 0 \widetilde{\rho}_v\ne0 ρ v = 0 , ∑ u K u , v = 1 \sum_u K_{u,v}=1 ∑ u K u , v = 1 and K ‾ ⋅ , v = ∣ K ⋅ , v ∣ / ∥ K ⋅ , v ∥ 1 \overline{K}_{\cdot,v}=|K_{\cdot,v}|/\lVert K_{\cdot,v}\rVert_1 K ⋅ , v = ∣ K ⋅ , v ∣/ ∥ K ⋅ , v ∥ 1 is a probability vector. For probability vectors q , k q,k q , k , log t ≤ t − 1 \log t\le t-1 log t ≤ t − 1 , with equality only at t = 1 t=1 t = 1 , gives q u log ( q u / k u ) ≥ q u − k u q_u\log(q_u/k_u)\ge q_u-k_u q u log ( q u / k u ) ≥ q u − k u , so KL ( q ∥ k ) ≥ 0 \operatorname{KL}(q\mathbin{\|}k)\ge0 KL ( q ∥ k ) ≥ 0 with equality if and only if q = k q=k q = k .
Let n v ( π ) ≠ 0 n_v(\pi)\ne0 n v ( π ) = 0 , π \pi π coherent at v v v , q u : = ∣ π u , v ∣ / n v ( π ) q_u:=|\pi_{u,v}|/n_v(\pi) q u := ∣ π u , v ∣/ n v ( π ) . The term of (8) at v v v is + ∞ +\infty + ∞ if ρ ~ v = 0 \widetilde{\rho}_v=0 ρ v = 0 ; otherwise it is ≥ 0 \ge0 ≥ 0 , and it vanishes if and only if ∣ π u , v ∣ = c v ∣ K u , v ∣ |\pi_{u,v}|=c_v|K_{u,v}| ∣ π u , v ∣ = c v ∣ K u , v ∣ for all u u u , c v : = n v ( π ) / ∥ K ⋅ , v ∥ 1 > 0 c_v:=n_v(\pi)/\lVert K_{\cdot,v}\rVert_1>0 c v := n v ( π ) / ∥ K ⋅ , v ∥ 1 > 0 . Coherence then gives π u , v = σ v c v K u , v \pi_{u,v}=\sigma_vc_vK_{u,v} π u , v = σ v c v K u , v , and summing over u u u , π u , v = K u , v m v ( π ) \pi_{u,v}=K_{u,v}m_v(\pi) π u , v = K u , v m v ( π ) with m v ( π ) ≠ 0 m_v(\pi)\ne0 m v ( π ) = 0 . Conversely, if ρ ~ v ≠ 0 \widetilde{\rho}_v\ne0 ρ v = 0 and π u , v = K u , v m \pi_{u,v}=K_{u,v}m π u , v = K u , v m , m ≠ 0 m\ne0 m = 0 , then π \pi π is coherent at v v v with σ v = sgn ( m ) \sigma_v=\operatorname{sgn}(m) σ v = sgn ( m ) , n v ( π ) = ∣ m ∣ ∥ K ⋅ , v ∥ 1 n_v(\pi)=|m|\lVert K_{\cdot,v}\rVert_1 n v ( π ) = ∣ m ∣ ∥ K ⋅ , v ∥ 1 and q = K ‾ ⋅ , v q=\overline{K}_{\cdot,v} q = K ⋅ , v . A node with n v ( π ) = 0 n_v(\pi)=0 n v ( π ) = 0 has π u , v = 0 \pi_{u,v}=0 π u , v = 0 and contributes nothing. Hence H ( π ∥ K ) ≥ 0 \mathcal{H}(\pi\mathbin{\|}K)\ge0 H ( π ∥ K ) ≥ 0 , and
H ( π ∥ K ) = 0 ⟺ π u , v = K u , v m v ( π ) for every ( u , v ) ∈ E ( x ) . (13) \mathcal{H}(\pi\mathbin{\|}K)=0\Longleftrightarrow\pi_{u,v}=K_{u,v}m_v(\pi)\quad\text{for every }(u,v)\in E(x). \tag*{(13)} H ( π ∥ K ) = 0 ⟺ π u , v = K u , v m v ( π ) for every ( u , v ) ∈ E ( x ) . (13)
Let π ∈ A o ( x ) \pi\in\mathcal{A}_o(x) π ∈ A o ( x ) satisfy (13) . The constraints give π u , o = K u , o x o \pi_{u,o}=K_{u,o}x_o π u , o = K u , o x o and π u , o ′ = 0 \pi_{u,o'}=0 π u , o ′ = 0 ; along a reverse topological order, at a hidden v v v , the divergence gives m v ( π ) = ∑ w : v → w π v , w = : μ v m_v(\pi)=\sum_{w:v\to w}\pi_{v,w}=:\mu_v m v ( π ) = ∑ w : v → w π v , w =: μ v , already known, and π u , v = K u , v μ v \pi_{u,v}=K_{u,v}\mu_v π u , v = K u , v μ v . So π = π ∗ \pi=\pi^* π = π ∗ , π u , v ∗ : = K u , v μ v \pi^*_{u,v}:=K_{u,v}\mu_v π u , v ∗ := K u , v μ v , μ \mu μ being given by the backward recursion. Conversely, ρ ~ v ≠ 0 \widetilde{\rho}_v\ne0 ρ v = 0 on H ( x ) ∪ { o } H(x)\cup\{o\} H ( x ) ∪ { o } gives m v ( π ∗ ) = μ v m_v(\pi^*)=\mu_v m v ( π ∗ ) = μ v there, the recursion gives the outflow μ v \mu_v μ v at a hidden node v v v , μ o ′ = 0 \mu_{o'}=0 μ o ′ = 0 gives m o ′ ( π ∗ ) = 0 m_{o'}(\pi^*)=0 m o ′ ( π ∗ ) = 0 : π ∗ ∈ A o ( x ) \pi^*\in\mathcal{A}_o(x) π ∗ ∈ A o ( x ) , and it satisfies (13) . Thus π ∗ \pi^* π ∗ is the unique minimizer, of value 0 0 0 . Proposition 3.2 gives ∑ u ∈ I ( x ) μ u = x o \sum_{u\in\mathcal{I}(x)}\mu_u=x_o ∑ u ∈ I ( x ) μ u = x o . If neither the node values nor the reference depend on κ \kappa κ , neither do the active graph and the μ u κ \mu_u^{\kappa} μ u κ .
Path-space reading of the conditional entropy (8)
Under the assumptions of Theorem 3.3 , let x o > 0 x_o>0 x o > 0 , ρ ≥ 0 \rho\ge0 ρ ≥ 0 , so that K ‾ = K \overline{K}=K K = K , π ∈ A o ( x ) \pi\in\mathcal{A}_o(x) π ∈ A o ( x ) non-negative, so that n v ( π ) = m v ( π ) n_v(\pi)=m_v(\pi) n v ( π ) = m v ( π ) , with H ( π ∥ K ) < ∞ \mathcal{H}(\pi\mathbin{\|}K)<\infty H ( π ∥ K ) < ∞ , and Π \Pi Π the set of backward paths p = ( o = v 0 , … , v r ) p=(o=v_0,\ldots,v_r) p = ( o = v 0 , … , v r ) , v j + 1 → v j v_{j+1}\to v_j v j + 1 → v j in E ( x ) E(x) E ( x ) , v r ∈ I ( x ) v_r\in\mathcal{I}(x) v r ∈ I ( x ) . Let
P π ( p ) : = x o ∏ j = 0 r − 1 q v j + 1 , v j , Q K ( p ) : = x o ∏ j = 0 r − 1 K v j + 1 , v j , q u , v : = π u , v m v ( π ) . P_\pi(p):=x_o\prod_{j=0}^{r-1}q_{v_{j+1},v_j},\qquad Q_K(p):=x_o\prod_{j=0}^{r-1}K_{v_{j+1},v_j},\qquad q_{u,v}:=\frac{\pi_{u,v}}{m_v(\pi)}. P π ( p ) := x o j = 0 ∏ r − 1 q v j + 1 , v j , Q K ( p ) := x o j = 0 ∏ r − 1 K v j + 1 , v j , q u , v := m v ( π ) π u , v .
q . , v q_{.,v} q . , v being any transition law where m v ( π ) = 0 m_v(\pi)=0 m v ( π ) = 0 , be the backward chains from o o o of mass x o x_o x o . Then KL ( P π ∥ Q K ) = H ( π ∥ K ) \operatorname{KL}(P_\pi\mathbin{\|}Q_K)=\mathcal{H}(\pi\mathbin{\|}K) KL ( P π ∥ Q K ) = H ( π ∥ K ) .
Along a reverse topological order, the mass of the paths through a node v ∉ I ( x ) v\notin\mathcal{I}(x) v ∈ / I ( x ) is m v ( π ) m_v(\pi) m v ( π ) : it is x o x_o x o at o o o , and ∑ w : v → w m w ( π ) q v , w = ∑ w π v , w = m v ( π ) \sum_{w:v\to w}m_w(\pi)q_{v,w}=\sum_w\pi_{v,w}=m_v(\pi) ∑ w : v → w m w ( π ) q v , w = ∑ w π v , w = m v ( π ) at a hidden node, as π v , w = 0 \pi_{v,w}=0 π v , w = 0 where m w ( π ) = 0 m_w(\pi)=0 m w ( π ) = 0 ; through an input i i i , it is its outflow ∑ w π i , w \sum_w \pi_{i,w} ∑ w π i , w . So the paths using ( u , v ) (u,v) ( u , v ) have mass π u , v \pi_{u,v} π u , v , and those ending at an input carry its outflow. Expanding the products and exchanging the sums,
KL ( P π ∥ Q K ) = ∑ ( u , v ) ∈ E ( x ) π u , v log q u , v K u , v = ∑ ( u , v ) ∈ E ( x ) π u , v log π u , v K u , v m v ( π ) = H ( π ∥ K ) . \operatorname{KL}(P_{\pi}\|Q_K)=\sum_{(u,v)\in E(x)}\pi_{u,v}\log\frac{q_{u,v}}{K_{u,v}}=\sum_{(u,v)\in E(x)}\pi_{u,v}\log\frac{\pi_{u,v}}{K_{u,v}m_v(\pi)}=\mathcal{H}(\pi\|K). KL ( P π ∥ Q K ) = ( u , v ) ∈ E ( x ) ∑ π u , v log K u , v q u , v = ( u , v ) ∈ E ( x ) ∑ π u , v log K u , v m v ( π ) π u , v = H ( π ∥ K ) .
Proof of Corollary 3.4
Each case checks ρ ~ v ≠ 0 \widetilde{\rho}_v\ne0 ρ v = 0 on H ( x ) ∪ { o } H(x)\cup\{o\} H ( x ) ∪ { o } , computes K = ρ / ρ ~ K=\rho/\widetilde{\rho} K = ρ / ρ and applies Theorem 3.3 .
For ρ = z \rho=z ρ = z , ρ ~ v = z ~ v ≠ 0 \widetilde{\rho}_v=\widetilde{z}_v\ne0 ρ v = z v = 0 , K u , v = z u , v / z ~ v K_{u,v}=z_{u,v}/\widetilde{z}_v K u , v = z u , v / z v . If moreover z ~ o ′ ≠ 0 \widetilde{z}_{o'}\ne0 z o ′ = 0 , then K ‾ u , v = ∣ z u , v ∣ / ∑ r : r → v ∣ z r , v ∣ \overline{K}_{u,v}=\lvert z_{u,v}\rvert/\sum_{r:r\to v}\lvert z_{r,v}\rvert K u , v = ∣ z u , v ∣ / ∑ r : r → v ∣ z r , v ∣ at every non-input node (the formula can fail at an output with z ~ o ′ = 0 \widetilde{z}_{o'}=0 z o ′ = 0 , where K ‾ = 0 \overline{K}=0 K = 0 ), and, z ~ v \widetilde{z}_v z v having a fixed sign, coherence with K K K says that the products π u , v z u , v \pi_{u,v}z_{u,v} π u , v z u , v entering v v v have a common sign.
For ρ = z + \rho=z^+ ρ = z + , ρ ~ v = z ~ v + > 0 \widetilde{\rho}_v=\widetilde{z}_v^+>0 ρ v = z v + > 0 , K u , v = z u , v + / z ~ v + K_{u,v}=z_{u,v}^+/\widetilde{z}_v^+ K u , v = z u , v + / z v + . For LRPα , β _{\alpha,\beta} α , β , ρ ~ v = α + β = 1 \widetilde{\rho}_v=\alpha+\beta=1 ρ v = α + β = 1 and K = ρ K=\rho K = ρ , both channels being non-empty.
For LRPα , β N ^{N}_{\alpha,\beta} α , β N , the two channels are not both empty, and ∑ u z u , v ± / z ~ v ± = 1 \sum_u z_{u,v}^{\pm}/\widetilde{z}_v^{\pm}=1 ∑ u z u , v ± / z v ± = 1 when z ~ v ± > 0 \widetilde{z}_v^{\pm}>0 z v ± > 0 , so ρ ~ v = α v + β v = 1 \widetilde{\rho}_v=\alpha_v+\beta_v=1 ρ v = α v + β v = 1 when both are non-empty and ρ ~ v = 1 \widetilde{\rho}_v=1 ρ v = 1 otherwise: K = ρ K=\rho K = ρ . Theorem 3.3 only sees the numbers ρ u , v \rho_{u,v} ρ u , v , whatever the dependence of the weights on v v v and x x x .
Proof of Corollary 3.5
On E ( x ) E(x) E ( x ) , K u , v x v = x u ω u , v s v K_{u,v}x_v=x_u\omega_{u,v}s_v K u , v x v = x u ω u , v s v . The recursion of μ \mu μ , each path of P ( u , o ) \mathcal{P}(u,o) P ( u , o ) decomposing uniquely as a first edge followed by a path, gives μ u = x u W u ρ \mu_u=x_uW_u^{\rho} μ u = x u W u ρ by induction along a reverse topological order; off V ( x ) V(x) V ( x ) , x u = 0 = μ u x_u=0=\mu_u x u = 0 = μ u .
A kernel that vanishes with z u , v z_{u,v} z u , v has the same relevance on G ( x ) G(x) G ( x ) and on G G G , a node of value 0 0 0 sending nothing, and ρ ~ v \widetilde{\rho}_v ρ v is the same on both at v ∈ V ( x ) v\in V(x) v ∈ V ( x ) . So LRP0 _0 0 is the relevance on G G G of the kernel ρ u , v / ρ ~ v \rho_{u,v}/\widetilde{\rho}_v ρ u , v / ρ v . At a linear or ReLU hidden node or at o o o : if z ~ v ≠ 0 \widetilde{z}_v\ne0 z v = 0 , x v = η u , v z ~ v x_v=\eta_{u,v}\widetilde{z}_v x v = η u , v z v gives K u , v x v = x u ω u , v η u , v K_{u,v}x_v=x_u\omega_{u,v}\eta_{u,v} K u , v x v = x u ω u , v η u , v ; if z ~ v = 0 \widetilde{z}_v=0 z v = 0 , then K u , v = 0 K_{u,v}=0 K u , v = 0 and η u , v = 0 \eta_{u,v}=0 η u , v = 0 , a linear hidden node with z ~ v = 0 \widetilde{z}_v=0 z v = 0 being excluded and z ~ o ≠ 0 \widetilde{z}_o\ne0 z o = 0 . At a max node, the reference has the single term z κ v , v = x v z_{\kappa_v,v}=x_v z κ v , v = x v , so that K u , v x v = z u , v η u , v = x u ω u , v η u , v K_{u,v}x_v=z_{u,v}\eta_{u,v}=x_u\omega_{u,v}\eta_{u,v} K u , v x v = z u , v η u , v = x u ω u , v η u , v , both sides vanishing if x v = 0 x_v=0 x v = 0 . As above, μ u = x u W u \mu_u=x_uW_u μ u = x u W u , with W u = ∑ v : u → v w u , v η u , v W v W_u=\sum_{v:u\to v}w_{u,v}\eta_{u,v}W_v W u = ∑ v : u → v w u , v η u , v W v , W o = 1 W_o=1 W o = 1 , W o ′ = 0 W_{o'}=0 W o ′ = 0 ; hence W ρ = W W^\rho=W W ρ = W on V ( x ) ∖ ( O ∖ { o } ) V(x)\setminus(O\setminus\{o\}) V ( x ) ∖ ( O ∖ { o }) .
If x ∉ D x\notin D x ∈ / D , Theorem 2.7 (iii) gives a unique κ \kappa κ with every z v ( x ) z_v(x) z v ( x ) in an open region: no gate changes near x x x , x o x_o x o is affine there, and the branch Jacobians are the gates, that of a max node v v v being the coordinate vector of κ v \kappa_v κ v , so ∂ x o / ∂ x u ( x ) = W u ( x ) \partial x_o/\partial x_u(x)=W_u(x) ∂ x o / ∂ x u ( x ) = W u ( x ) by (3) . The recursion of W W W is (6) .
On U U U , the signs of the values and κ \kappa κ fix G ( x ) G(x) G ( x ) , the channels, the cases of Corollary 3.4 3.4 3.4 (iv) and the kernels of the max nodes. The shares z u , v ± / z ~ v ± z_{u,v}^{\pm}/\widetilde{z}_v^{\pm} z u , v ± / z v ± are continuous, or zero on an empty channel, so the kernels and μ \mu μ are continuous. For LRP0 _0 0 , with z ~ v ≠ 0 \widetilde{z}_v\ne0 z v = 0 at the linear hidden nodes, the gates are constant, so is W W W , and μ u = x u W u \mu_u=x_uW_u μ u = x u W u . For LRPz + _{z^+} z + with ReLU hidden nodes, or max hidden nodes of non-negative value, and z ~ o > 0 \widetilde{z}_o>0 z o > 0 , a reverse topological order gives 0 ≤ μ u ≤ ∣ x u ∣ ∣ W ‾ u ∣ 0\le\mu_u\le|x_u|\lvert\overline{W}_u\rvert 0 ≤ μ u ≤ ∣ x u ∣ ∣ W u ∣ : at a live ReLU head v v v , 0 < x v = z ~ v ≤ z v + 0<x_v=\widetilde{z}_v\le z_v^+ 0 < x v = z v ≤ z v + , so K u , v μ v ≤ z u , v + ∣ W ‾ v ∣ ≤ ∣ x u ∣ ∣ w u , v ∣ ∣ W ‾ v ∣ K_{u,v}\mu_v\le z_{u,v}^+|\overline{W}_v|\le|x_u||w_{u,v}||\overline{W}_v| K u , v μ v ≤ z u , v + ∣ W v ∣ ≤ ∣ x u ∣∣ w u , v ∣∣ W v ∣ ; at a live max head, x v = z κ v , v > 0 x_v=z_{\kappa_v,v}>0 x v = z κ v , v > 0 , and K u , v μ v K_{u,v}\mu_v K u , v μ v is 0 0 0 , or μ v ≤ x v ∣ W ‾ v ∣ = ∣ x u ∣ ∣ w u , v ∣ ∣ W ‾ v ∣ \mu_v\le x_v|\overline{W}_v|=|x_u||w_{u,v}||\overline{W}_v| μ v ≤ x v ∣ W v ∣ = ∣ x u ∣∣ w u , v ∣∣ W v ∣ for u = κ v u=\kappa_v u = κ v ; a dead head has μ v = 0 \mu_v=0 μ v = 0 .
Near x ˉ \bar{x} x ˉ , in (iii) and (iv), κ \kappa κ stays admissible at the max nodes other than the tie: a single maximizer stays the only one, and a maximum 0 0 0 attained at ReLU nodes r r r with z ~ r ( x ˉ ) < 0 \widetilde{z}_r(\bar{x})<0 z r ( x ˉ ) < 0 stays 0 0 0 , the other contributions staying negative. Such a node acts as a linear node with the single edge ( κ r , r ) (\kappa_r,r) ( κ r , r ) , whose value moves with that of κ r \kappa_r κ r , which is not v v v in (iii). Write η ˉ = η ( x ˉ ) \bar{\eta}=\eta(\bar{x}) η ˉ = η ( x ˉ ) and η r \eta_r η r for the gate shared by the edges entering a linear or ReLU node r r r , so η ˉ v = 0 \bar{\eta}_v=0 η ˉ v = 0 in (iii), and let U U U be a neighbourhood of x ˉ \bar{x} x ˉ on which every node r ≠ v r\ne v r = v , hidden or o o o , keeps its gates. A path does not cross v v v twice, so W v ( x ) = W v ( x ˉ ) W_v(x)=W_v(\bar{x}) W v ( x ) = W v ( x ˉ ) on U U U , and the first-edge recursion gives, by induction,
W u ( x ) = W u ( x ˉ ) + η v ( x ) W u v ( x ˉ ) W v ( x ˉ ) ( u ≠ v ) . W_u(x)=W_u(\bar{x})+\eta_v(x)W_u^v(\bar{x})W_v(\bar{x})\qquad(u\ne v). W u ( x ) = W u ( x ˉ ) + η v ( x ) W u v ( x ˉ ) W v ( x ˉ ) ( u = v ) .
With μ u ( x ) = x u W u ( x ) \mu_u(x)=x_uW_u(x) μ u ( x ) = x u W u ( x ) , this gives the two limits of LRP0 _0 0 ; at v v v , μ v ( x ) = x v W v ( x ˉ ) → 0 \mu_v(x)=x_vW_v(\bar{x})\to0 μ v ( x ) = x v W v ( x ˉ ) → 0 and x ˉ v = 0 \bar{x}_v=0 x ˉ v = 0 . Along a topological order, for r r r before v v v , x r = η r z ~ r x_r=\eta_r\widetilde{z}_r x r = η r z r at a linear or ReLU node and x r = z κ r , r x_r=z_{\kappa_r,r} x r = z κ r , r at a max node give ∑ i ∈ I x ~ i W ‾ i r = x ˉ r \sum_{i\in I}\widetilde{x}_i\overline{W}_i^r=\bar{x}_r ∑ i ∈ I x i W i r = x ˉ r , W ‾ i r \overline{W}_i^r W i r being the sum over P ( i , r ) \mathcal{P}(i,r) P ( i , r ) of the products of w η w\eta w η , gate of r r r included. Hence ∑ i ∈ I x ~ i W ‾ i ( x ˉ ) = ∑ r : r → v w r , v x ˉ r = z ~ v ( x ˉ ) = 0 \sum_{i\in I}\widetilde{x}_i\overline{W}_i(\bar{x})=\sum_{r:r\to v}w_{r,v}\bar{x}_r=\widetilde{z}_v(\bar{x})=0 ∑ i ∈ I x i W i ( x ˉ ) = ∑ r : r → v w r , v x ˉ r = z v ( x ˉ ) = 0 and ∑ u ∈ I Δ u = 0 \sum_{u\in I}\Delta_u=0 ∑ u ∈ I Δ u = 0 .
For LRPα , β N ^N_{\alpha,\beta} α , β N , the inputs and the z ~ r ( x ˉ ) \widetilde{z}_r(\bar{x}) z r ( x ˉ ) , r ≠ v r\ne v r = v linear or ReLU, are nonzero, so on a neighbourhood U U U of x ˉ \bar{x} x ˉ every node other than v v v keeps the sign of its value, zero at an inactive ReLU node, that of its winner at a max node, and so does every contribution it sends; G ( x ) = G ( x ˉ ) G(x)=G(\bar{x}) G ( x ) = G ( x ˉ ) where z ~ v ( x ) < 0 \widetilde{z}_v(x)<0 z v ( x ) < 0 , and G ( x ) G(x) G ( x ) is G ( x ˉ ) G(\bar{x}) G ( x ˉ ) augmented by v v v and its edges where z ~ v ( x ) > 0 \tilde{z}_v(x)>0 z ~ v ( x ) > 0 . By the recursion of μ \mu μ , it suffices that every kernel converge. Fix a node and a channel. Where z ~ v ( x ) < 0 \tilde{z}_v(x)<0 z ~ v ( x ) < 0 , x v = 0 x_v=0 x v = 0 : the channel is empty or not as at x ˉ \bar{x} x ˉ , and its shares converge to those of x ˉ \bar{x} x ˉ , so μ − = μ ( x ˉ ) \mu^-=\mu(\bar{x}) μ − = μ ( x ˉ ) . Where z ~ v ( x ) > 0 \tilde{z}_v(x)>0 z ~ v ( x ) > 0 , at a successor w w w of v v v other than a max node, z v , w = w v , w x v → 0 z_{v,w}=w_{v,w}x_v\to0 z v , w = w v , w x v → 0 with the sign of w v , w w_{v,w} w v , w ; a max successor keeps its kernel, v v v not being its winner. If the channel of that sign is non-empty at x ˉ \bar{x} x ˉ , its shares converge to those of x ˉ \bar{x} x ˉ , v v v having share → 0 \to0 → 0 . If it is empty, v v v fills it: share 1 for v v v , 0 for the others. In all cases the kernels converge, to those of LRPN α , β \operatorname{LRPN}_{\alpha,\beta} LRPN α , β on the augmented graph, at the values of x ˉ \bar{x} x ˉ , each successor w w w counting v v v in the channel of the sign of w v , w w_{v,w} w v , w ; they are the kernels of x ˉ \bar{x} x ˉ if no channel is filled, and then μ + = μ ( x ˉ ) \mu^+=\mu(\bar{x}) μ + = μ ( x ˉ ) . For LRP z + \operatorname{LRP}_{z^+} LRP z + , β w = 0 \beta_w=0 β w = 0 , so filling the negative channel of a node changes none of its kernels when its positive channel is non-empty, which holds at the live nodes, and a positive channel can only be filled at a dead node, of relevance 0: μ + = μ ( x ˉ ) \mu^+=\mu(\bar{x}) μ + = μ ( x ˉ ) . Finally, the relevances of the inputs add up to x o ( x ) x_o(x) x o ( x ) at the inputs x ∈ U x\in U x ∈ U with z ~ v ( x ) > 0 \tilde{z}_v(x)>0 z ~ v ( x ) > 0 ; as x → x ˉ x\to\bar{x} x → x ˉ , ∑ u ∈ I μ u + = x o ( x ˉ ) = ∑ u ∈ I μ u ( x ˉ ) \sum_{u\in I}\mu_u^+=x_o(\bar{x})=\sum_{u\in I}\mu_u(\bar{x}) ∑ u ∈ I μ u + = x o ( x ˉ ) = ∑ u ∈ I μ u ( x ˉ ) .
In (iv), the other predecessors of v v v stay below u u u and u ′ u' u ′ near x ˉ \bar{x} x ˉ , so κ \kappa κ is admissible where z u , v > z u ′ , v z_{u,v}>z_{u',v} z u , v > z u ′ , v and κ ′ \kappa' κ ′ where z u ′ , v > z u , v z_{u',v}>z_{u,v} z u ′ , v > z u , v . No gate, sign or channel changes there, and x v ≠ 0 x_v\ne0 x v = 0 , so μ = μ κ \mu=\mu^\kappa μ = μ κ on the first side, continuous at x ˉ \bar{x} x ˉ as in (ii), and μ = μ κ ′ \mu=\mu^{\kappa'} μ = μ κ ′ on the second. The kernels of κ \kappa κ and κ ′ \kappa' κ ′ differ only on the edges entering v v v , where they are 1 on ( u , v ) (u,v) ( u , v ) , resp. ( u ′ , v ) (u',v) ( u ′ , v ) , and 0 on the others. Splitting each path of P ( i , o ) \mathcal{P}(i,o) P ( i , o ) through v v v at the edge entering v v v , the paths to u u u and u ′ u' u ′ and the relevance μ v ( x ˉ ) \mu_v(\bar{x}) μ v ( x ˉ ) being the same for κ \kappa κ and κ ′ \kappa' κ ′ ,
μ i κ ′ ( x ˉ ) − μ i κ ( x ˉ ) = μ v ( x ˉ ) ∑ a : a → v R i ( a ) ( K a , v κ ′ − K a , v κ ) = μ v ( x ˉ ) ( R i ( u ′ ) − R i ( u ) ) . \mu_i^{\kappa'}(\bar{x})-\mu_i^\kappa(\bar{x})=\mu_v(\bar{x})\sum_{a:a\to v}R_i^{(a)}\left(K_{a,v}^{\kappa'}-K_{a,v}^{\kappa}\right)=\mu_v(\bar{x})\left(R_i^{(u')}-R_i^{(u)}\right). μ i κ ′ ( x ˉ ) − μ i κ ( x ˉ ) = μ v ( x ˉ ) a : a → v ∑ R i ( a ) ( K a , v κ ′ − K a , v κ ) = μ v ( x ˉ ) ( R i ( u ′ ) − R i ( u ) ) .
Both conserve x o ( x ˉ ) x_o(\bar{x}) x o ( x ˉ ) , so ∑ i ∈ I Δ i = 0 \sum_{i\in I}\Delta_i=0 ∑ i ∈ I Δ i = 0 . For LRP 0 \operatorname{LRP}_0 LRP 0 , μ v ( x ˉ ) = x ˉ v W v ( x ˉ ) \mu_v(\bar{x})=\bar{x}_vW_v(\bar{x}) μ v ( x ˉ ) = x ˉ v W v ( x ˉ ) , the paths to a a a give R i ( a ) = x ˉ i W ^ i a ( x ˉ ) / x ˉ a R_i^{(a)}=\bar{x}_i\widehat{W}_i^{a}(\bar{x})/\bar{x}_a R i ( a ) = x ˉ i W i a ( x ˉ ) / x ˉ a as in (i), and x ˉ v = w u , v x ˉ u = w u ′ , v x ˉ u ′ \bar{x}_v=w_{u,v}\bar{x}_u=w_{u',v}\bar{x}_{u'} x ˉ v = w u , v x ˉ u = w u ′ , v x ˉ u ′ .
Proof of Theorem 3.6
Write E o : = E ρ o ( x ) E^o:=E_\rho^o(x) E o := E ρ o ( x ) , V o : = V ρ o ( x ) V^o:=V_\rho^o(x) V o := V ρ o ( x ) , and attach to e = ( u , v ) ∈ E o e=(u,v)\in E^o e = ( u , v ) ∈ E o its arc, of tail tail ( e ) \operatorname{tail}(e) tail ( e ) and head head ( e ) \operatorname{head}(e) head ( e ) : u → v u\to v u → v if ρ e > 0 \rho_e>0 ρ e > 0 , v → u v\to u v → u if ρ e < 0 \rho_e<0 ρ e < 0 . For ν ∈ R V o \nu\in\mathbb{R}^{V^o} ν ∈ R V o , ν i = 0 \nu_i=0 ν i = 0 at the inputs, let φ : = e ν \varphi:=e^\nu φ := e ν and ( π + , π − ) (\pi^+,\pi^-) ( π + , π − ) be the pair (10) : the mass ∣ ρ e ∣ e ν head ( e ) − ν tail ( e ) |\rho_e|e^{\nu_{\operatorname{head}(e)}-\nu_{\operatorname{tail}(e)}} ∣ ρ e ∣ e ν head ( e ) − ν tail ( e ) along each arc. Let
g ( ν ) : = ∑ ( u , v ) ∈ E o ( ρ u , v + ( 1 − e ν v − ν u ) + ρ u , v − ( 1 − e ν u − ν v ) ) + y ν o , g(\nu):=\sum_{(u,v)\in E^o}\left(\rho_{u,v}^+(1-e^{\nu_v-\nu_u})+\rho_{u,v}^-(1-e^{\nu_u-\nu_v})\right)+y\nu_o, g ( ν ) := ( u , v ) ∈ E o ∑ ( ρ u , v + ( 1 − e ν v − ν u ) + ρ u , v − ( 1 − e ν u − ν v ) ) + y ν o ,
strictly concave, the dual function of (9) . Then ∂ g / ∂ ν v \partial g/\partial\nu_v ∂ g / ∂ ν v is the net outflow minus the net inflow of the pair at a hidden v v v , ∂ g / ∂ ν o = y \partial g/\partial\nu_o=y ∂ g / ∂ ν o = y minus its net inflow at o o o ; multiplying by φ v \varphi_v φ v , the pair is feasible if and only if ∇ g ( ν ) = 0 \nabla g(\nu)=0 ∇ g ( ν ) = 0 if and only if φ \varphi φ solves (11) .
Let c ∈ R c\in\mathbb{R} c ∈ R , g ( ν ) ≥ c g(\nu)\ge c g ( ν ) ≥ c , M : = max ( 0 , max e ( ν head ( e ) − ν tail ( e ) ) ) M:=\max(0,\max_e(\nu_{\operatorname{head}(e)}-\nu_{\operatorname{tail}(e)})) M := max ( 0 , max e ( ν head ( e ) − ν tail ( e ) )) , r : = min e ∣ ρ e ∣ r:=\min_e|\rho_e| r := min e ∣ ρ e ∣ , R : = ∑ e ∣ ρ e ∣ R:=\sum_e|\rho_e| R := ∑ e ∣ ρ e ∣ and L : = ∣ V o ∣ L:=|V^o| L := ∣ V o ∣ . Every node of V o V^o V o is reached from an input by at most L L L arcs, so ν v ≤ L M \nu_v\le LM ν v ≤ L M , and c ≤ g ( ν ) ≤ R + r ( 1 − e M ) + y L M c\le g(\nu)\le R+r(1-e^M)+yLM c ≤ g ( ν ) ≤ R + r ( 1 − e M ) + y L M bounds M M M . From g ( ν ) ≤ R + y ν o g(\nu)\le R+y\nu_o g ( ν ) ≤ R + y ν o , ν o \nu_o ν o is bounded below. From every node of V o V^o V o , at most L L L arcs lead to an input or to o o o , so ν v ≥ min ( 0 , ν o ) − L M \nu_v\ge\min(0,\nu_o)-LM ν v ≥ min ( 0 , ν o ) − L M . Thus { g ≥ c } \{g\ge c\} { g ≥ c } is compact and g g g has a maximizer ν ∗ \nu^* ν ∗ ; let ( π S + , π S − ) (\pi^{S+},\pi^{S-}) ( π S + , π S − ) be its pair, feasible, positive exactly where ρ ± ≠ 0 \rho^\pm\ne0 ρ ± = 0 .
Let ( π + , π − ) (\pi^+,\pi^-) ( π + , π − ) be feasible, of finite objective J J J . The difference d d d of the two net flows has zero divergence and zero inflow at o o o , so, summing by parts, ∑ ( u , v ) d u , v ( ν v − ν u ) = 0 \sum_{(u,v)}d_{u,v}(\nu_v-\nu_u)=0 ∑ ( u , v ) d u , v ( ν v − ν u ) = 0 for every ν \nu ν vanishing at the inputs. With ν ∗ \nu^* ν ∗ , as log ( π u , v S ± / π u , v ± ) = ± ( ν v ∗ − ν u ∗ ) \log(\pi_{u,v}^{S\pm}/\pi_{u,v}^{\pm})=\pm(\nu_v^*-\nu_u^*) log ( π u , v S ± / π u , v ± ) = ± ( ν v ∗ − ν u ∗ ) ,
J ( π + , π − ) = J ( π S + , π S − ) + KL ( π + ∥ π S + ) + KL ( π − ∥ π S − ) . J(\pi^+,\pi^-)=J(\pi^{S+},\pi^{S-})+\operatorname{KL}(\pi^+\Vert\pi^{S+})+\operatorname{KL}(\pi^-\Vert\pi^{S-}). J ( π + , π − ) = J ( π S + , π S − ) + KL ( π + ∥ π S + ) + KL ( π − ∥ π S − ) .
so ( π S + , π S − ) (\pi^{S+},\pi^{S-}) ( π S + , π S − ) is the unique minimizer. The same identity holds for every feasible pair of the form (10) : such a pair is the minimizer, and, conversely, the minimizer is feasible, whence (11) . On E o E^o E o , (10) fixes the ratios of potentials along edges, every node of V o V^o V o other than an input is reached from an input along E o E^o E o , and φ i = 1 \varphi_i=1 φ i = 1 at the inputs: φ \varphi φ is unique. Along a path of positive reference from an input to v v v , the products telescope to φ v \varphi_v φ v . Proposition 3.2 gives ∑ u ∈ I ( x ) μ u − = y \sum_{u\in I(x)}\mu_u^-=y ∑ u ∈ I ( x ) μ u − = y .
At a hidden v v v , ∂ g / ∂ ν v \partial g/\partial\nu_v ∂ g / ∂ ν v is strictly decreasing in ν v \nu_v ν v and vanishes exactly where (11) holds at v v v , at φ v = ( ( ∑ w ρ v , w + φ w + ∑ u ρ u , v − φ u ) / ( ∑ u ρ u , v + / φ u + ∑ w ρ v , w − / φ w ) ) 1 / 2 \varphi_v=\left((\sum_w\rho_{v,w}^+\varphi_w+\sum_u\rho_{u,v}^-\varphi_u)/(\sum_u\rho_{u,v}^+/\varphi_u+\sum_w\rho_{v,w}^-/\varphi_w)\right)^{1/2} φ v = ( ( ∑ w ρ v , w + φ w + ∑ u ρ u , v − φ u ) / ( ∑ u ρ u , v + / φ u + ∑ w ρ v , w − / φ w ) ) 1/2 ; at o o o , with A o + : = ∑ u ρ u , o + / φ u A_o^+:=\sum_u\rho_{u,o}^+/\varphi_u A o + := ∑ u ρ u , o + / φ u and A o − : = ∑ u ρ u , o − φ u A_o^-:=\sum_u\rho_{u,o}^-\varphi_u A o − := ∑ u ρ u , o − φ u , ∂ g ∂ ν o = y − φ o A o + + A o − / φ o \frac{\partial g}{\partial\nu_o}=y-\varphi_o A_o^+ + A_o^-/\varphi_o ∂ ν o ∂ g = y − φ o A o + + A o − / φ o vanishes at the positive root of A o + φ o 2 − y φ o − A o − A_o^+\varphi_o^2-y\varphi_o-A_o^- A o + φ o 2 − y φ o − A o − . So each update maximizes g g g in one coordinate, and the sweep T T T is continuous, with g ∘ T ≥ g g\circ T\ge g g ∘ T ≥ g , and g ( T ν ) = g ( ν ) g(T\nu)=g(\nu) g ( T ν ) = g ( ν ) only if ∇ g ( ν ) = 0 \nabla g(\nu)=0 ∇ g ( ν ) = 0 , that is, ν = ν ∗ \nu=\nu^* ν = ν ∗ . The iterates stay in the compact set { g ≥ g ( ν ( 0 ) ) } \{g\ge g(\nu^{(0)})\} { g ≥ g ( ν ( 0 ) )} and g ( ν ( t ) ) g(\nu^{(t)}) g ( ν ( t ) ) converges; at a limit point ν ˉ \bar{\nu} ν ˉ , g ( T ν ˉ ) = g ( ν ˉ ) g(T\bar{\nu})=g(\bar{\nu}) g ( T ν ˉ ) = g ( ν ˉ ) , so ν ˉ = ν ∗ \bar{\nu}=\nu^* ν ˉ = ν ∗ , and ν ( t ) → ν ∗ \nu^{(t)}\to\nu^* ν ( t ) → ν ∗ .
Proof of Proposition 3.8
( ρ + , 0 ) (\rho^+,0) ( ρ + , 0 ) is feasible, of objective KL ( 0 ∥ ρ − ) = R − ≤ R \operatorname{KL}(0\mathbin\Vert\rho^-)=R^-\le R KL ( 0 ∥ ρ − ) = R − ≤ R , so J ≤ R J\le R J ≤ R at the minimizer. For p , r ≥ 0 p,r\ge0 p , r ≥ 0 , p log ( p / r ) − p + r ≥ p − ( e − 1 ) r p\log(p/r)-p+r\ge p-(e-1)r p log ( p / r ) − p + r ≥ p − ( e − 1 ) r , the tangent at p = e r p=er p = er of the convex function p ↦ p log ( p / r ) p\mapsto p\log(p/r) p ↦ p log ( p / r ) . Summing over the two sign parts, J ≥ M − ( e − 1 ) R J\ge M-(e-1)R J ≥ M − ( e − 1 ) R , with M M M the total mass. Hence M ≤ e R M\le eR M ≤ e R .
Proof of Proposition 3.7
The restriction of ρ \rho ρ is feasible, of the form (10) with φ ≡ 1 \varphi\equiv1 φ ≡ 1 : by Theorem 3.6 (iii) it is π S \pi^S π S . If every edge enters o o o , its source is an input, π u , o S = ρ u , o φ o \pi^S_{u,o}=\rho_{u,o}\varphi_o π u , o S = ρ u , o φ o , and the boundary condition gives φ o = y / ∑ ( r , o ) ∈ E ρ o ( x ) ρ r , o = y / ρ ~ o \varphi_o=y/\sum_{(r,o)\in E_\rho^o(x)}\rho_{r,o}=y/\tilde{\rho}_o φ o = y / ∑ ( r , o ) ∈ E ρ o ( x ) ρ r , o = y / ρ ~ o , ρ \rho ρ vanishing off E ρ o ( x ) E_\rho^o(x) E ρ o ( x ) : this is K u , o y K_{u,o}y K u , o y . On the graph of the last statement, by Theorem 3.6 (ii), the products of π S / ρ \pi^S/\rho π S / ρ along the two paths agree, and conservation gives π i , h k S = π h k , o S \pi^S_{i,h_k}=\pi^S_{h_k,o} π i , h k S = π h k , o S , whence the stated ratio. For Theorem 3.3 , the flows into o o o are K h k , o y K_{h_k,o}y K h k , o y .
Proofs for Section 4: Relevance by Region
Proof of Corollary 4.1
Fix κ \kappa κ and drop it. A path of P ( u , v ) \mathcal{P}(u,v) P ( u , v ) , u ≠ v u\ne v u = v , is a path of P ( u , w ) \mathcal{P}(u,w) P ( u , w ) followed by ( w , v ) (w,v) ( w , v ) , or an edge ( u , w ) (u,w) ( u , w ) followed by a path of P ( w , v ) \mathcal{P}(w,v) P ( w , v ) :
R u ( v ) = ∑ w : w → v K w , v R u ( w ) = ∑ w : u → w K u , w R w ( v ) ( u ≠ v ) , R v ( v ) = 1. (14) R_u^{(v)}=\sum_{w:w\to v}K_{w,v}R_u^{(w)}=\sum_{w:u\to w}K_{u,w}R_w^{(v)}\quad(u\ne v),\qquad R_v^{(v)}=1. \tag*{(14)} R u ( v ) = w : w → v ∑ K w , v R u ( w ) = w : u → w ∑ K u , w R w ( v ) ( u = v ) , R v ( v ) = 1. (14)
The second form is the recursion of Theorem 3.3 from v v v with unit mass, so μ u = x o R u ( o ) \mu_u=x_oR_u^{(o)} μ u = x o R u ( o ) . R ( v ) ≥ 0 R^{(v)}\ge0 R ( v ) ≥ 0 . R ( v ) R^{(v)} R ( v ) is the Dirac mass at v v v if v v v is an input; for v ∈ H ( x ) ∪ { o } v\in H(x)\cup\{o\} v ∈ H ( x ) ∪ { o } , ∑ u K u , v = 1 \sum_uK_{u,v}=1 ∑ u K u , v = 1 , and summing (14) over i i i gives ∑ i R i ( v ) = 1 \sum_iR_i^{(v)}=1 ∑ i R i ( v ) = 1 along a topological order. By Appendix § 7 .§ 3 , with q = K q=K q = K , the path p p p has mass Q K ( p ) = x o ∏ j K v j + 1 , v j Q_K(p)=x_o\prod_jK_{v_{j+1},v_j} Q K ( p ) = x o ∏ j K v j + 1 , v j ; a path through v v v ending at i i i splits at v v v , so these paths have mass x o R o ( o ) R i ( v ) = μ v R i ( v ) x_oR_o^{(o)}R_i^{(v)}=\mu_vR_i^{(v)} x o R o ( o ) R i ( v ) = μ v R i ( v ) . Summing over the regions of a partition gives ∑ M μ v M = μ v \sum_M\mu_v^M=\mu_v ∑ M μ v M = μ v . A path of positive mass ends at an input and meets a cut C C C exactly once, so ∑ v ∈ C μ v R i ( v ) \sum_{v\in C}\mu_vR_i^{(v)} ∑ v ∈ C μ v R i ( v ) is the mass of the paths ending at i i i , which is μ i \mu_i μ i by Appendix § 7 .§ 3 .
Let c : = x o / μ ( M ) c:=x_o/\mu(M) c := x o / μ ( M ) , h ( u ) : = R ( u ) ( M ) h(u):=R^{(u)}(M) h ( u ) := R ( u ) ( M ) , so that h ( i ) = 1 M ( i ) h(i)=1_M(i) h ( i ) = 1 M ( i ) at the inputs, h ( v ) = ∑ u K u , v h ( u ) h(v)=\sum_uK_{u,v}h(u) h ( v ) = ∑ u K u , v h ( u ) elsewhere by (14) , and h ( o ) = 1 / c h(o)=1/c h ( o ) = 1/ c . Then π M ≥ 0 \pi^M\ge0 π M ≥ 0 . Its inflow at v ∈ H ( x ) ∪ { o } v\in H(x)\cup\{o\} v ∈ H ( x ) ∪ { o } is c μ v h ( v ) c\mu_vh(v) c μ v h ( v ) , its outflow at a hidden v v v too; it has inflow c x o h ( o ) = x o cx_oh(o)=x_o c x o h ( o ) = x o at o o o , 0 0 0 at the other outputs, and outflow c μ i h ( i ) c\mu_ih(i) c μ i h ( i ) at an input, and the node relevances are c μ u h ( u ) = c μ u M c\mu_uh(u)=c\mu_u^M c μ u h ( u ) = c μ u M .
Let π \pi π be feasible with H ( π ∥ K ) < ∞ \mathcal{H}(\pi\mathbin\Vert K)<\infty H ( π ∥ K ) < ∞ . By Appendix § 7 .§ 3 , H ( π ∥ K ) = KL ( P π ∥ Q K ) \mathcal{H}(\pi\mathbin\Vert K)=\operatorname{KL}(P_\pi\mathbin\Vert Q_K) H ( π ∥ K ) = KL ( P π ∥ Q K ) , and the endpoint has law the outflow ν \nu ν of the inputs under P π P_\pi P π , μ \mu μ under Q K Q_K Q K , with ν i = c μ i 1 M ( i ) \nu_i=c\mu_i1_M(i) ν i = c μ i 1 M ( i ) . By the chain rule for relative entropy,
KL ( P π ∥ Q K ) = KL ( ν ∥ μ ) + ∑ i ∈ M ν i KL ( P π ( ⋅ ∣ i ) ∥ Q K ( ⋅ ∣ i ) ) ≥ KL ( ν ∥ μ ) = x o log c , \operatorname{KL}(P_\pi\mathbin\Vert Q_K)=\operatorname{KL}(\nu\mathbin\Vert\mu)+\sum_{i\in M}\nu_i\operatorname{KL}\bigl(P_\pi(\cdot\mid i)\mathbin\Vert Q_K(\cdot\mid i)\bigr)\ge\operatorname{KL}(\nu\mathbin\Vert\mu)=x_o\log c, KL ( P π ∥ Q K ) = KL ( ν ∥ μ ) + i ∈ M ∑ ν i KL ( P π ( ⋅ ∣ i ) ∥ Q K ( ⋅ ∣ i ) ) ≥ KL ( ν ∥ μ ) = x o log c ,
with equality if and only if P π = P M P_\pi=P^M P π = P M , P M ( p ) : = c Q K ( p ) 1 M ( v r ) P^M(p):=cQ_K(p)1_M(v_r) P M ( p ) := c Q K ( p ) 1 M ( v r ) . The ratios h ( v j + 1 ) / h ( v j ) h(v_{j+1})/h(v_j) h ( v j + 1 ) / h ( v j ) telescope to c 1 M ( v r ) c1_M(v_r) c 1 M ( v r ) along a path of positive P M P^M P M -mass, so P M P^M P M is the chain of transitions K u , v h ( u ) / h ( v ) = π u , v M / m v ( π M ) K_{u,v}h(u)/h(v)=\pi^M_{u,v}/m_v(\pi^M) K u , v h ( u ) / h ( v ) = π u , v M / m v ( π M ) , any law at the nodes where h ( v ) = 0 h(v)=0 h ( v ) = 0 , which carry no P M P^M P M -mass: P M = P π M P^M=P_{\pi^M} P M = P π M . Hence H ( π M ∥ K ) = x o log c \mathcal{H}(\pi^M\mathbin\Vert K)=x_o\log c H ( π M ∥ K ) = x o log c , and a feasible π \pi π attaining the bound has P π = P π M P_\pi=P_{\pi^M} P π = P π M , so π = π M \pi=\pi^M π = π M , a flow being the edge marginal of its path measure.
Architectures of VGG-16 and ConvNeXt-Tiny
Both networks are used with their ImageNet weights from torchvision, on 224 × 224 224 \times224 224 × 224 colour images, and have 1000 linear outputs, the logits. Table 3 and Table 4 list their layers; the sizes are those of the output of each stage.
Table 3. VGG-16 ([43 ] ): 138.4 million parameters, 71.6% top-1 accuracy on ImageNet. Every convolution is 3 × 3 3 \times3 3 × 3 , of stride 1 and padding 1, followed by a ReLU; every max-pooling has 2 × 2 2 \times2 2 × 2 windows of stride 2.
Table 4. ConvNeXt-Tiny ([29 ] ): 28.6 million parameters, 82.5% top-1 accuracy on ImageNet. A block of width C C C maps a a a to a + γ ⊙ W 2 GELU ( W 1 LN ( D a ) ) a + \gamma\odot W_2 \operatorname{GELU}(W_1 \operatorname{LN}(D a)) a + γ ⊙ W 2 GELU ( W 1 LN ( D a )) , with D D D a depthwise 7 × 7 7 \times7 7 × 7 convolution (one kernel per channel, padding 3), LN a layer normalization over the C C C channels at each position, W 1 W_1 W 1 , W 2 W_2 W 2 two 1 × 1 1 \times1 1 × 1 convolutions C → 4 C → C C \to4C \to C C → 4 C → C , and γ ∈ R C \gamma\in\mathbb{R}^{C} γ ∈ R C the layer scale.
As a Filippov network, VGG-16 is layered (Definition 2.4 ): one input node per pixel and colour channel, one ReLU node per unit of a convolution or dense layer (13.6 million in all), with one bias node of value 1 each, one max node (Corollary 3.5) of four predecessors of weight 1 per window of a max-pooling layer, and 1000 linear outputs, each with its bias node. The adaptive average pooling of torchvision is the identity at this image size, and dropout is inactive at inference. The regions of § 4.2 are cut in conv5_3, of size 14 × 14 × 512 14 \times14 \times512 14 × 14 × 512 .
ConvNeXt-Tiny is not layered: each block adds its input to the output of its branch. Read at x x x as in § 3.4 , it has linear and GELU nodes only, with no ReLU and no max node: a convolution or dense unit is a linear node with its bias node, a depthwise unit having its predecessors in its own channel only; the node c c c of a layer normalization at a position whose C C C channels have mean m m m and stabilized standard deviation σ ε : = ( C − 1 ∑ c ′ ( x c ′ − m ) 2 + ε ) 1 / 2 \sigma_\varepsilon:= \left(C^{-1}\sum_{c'}(x_{c'} - m)^2 + \varepsilon\right)^{1/2} σ ε := ( C − 1 ∑ c ′ ( x c ′ − m ) 2 + ε ) 1/2 , ε = 10 − 6 \varepsilon= 10^{-6} ε = 1 0 − 6 being the constant of the module, held constant, receives the contributions g c ( δ c , c ′ − 1 / C ) x c ′ / σ ε g_c(\delta_{c,c'} - 1/C)x_{c'}/\sigma_\varepsilon g c ( δ c , c ′ − 1/ C ) x c ′ / σ ε and the contribution b c b_c b c of a bias node, g g g and b b b being the affine parameters of the normalization; each unit of W 1 W_1 W 1 is a GELU node (8.6 million in all); the layer scale γ \gamma γ is folded into W 2 W_2 W 2 and its bias; a residual sum is a node with two contributions of weight 1; and the global average pooling is a linear node of weights 1 / 49 1/49 1/49 .