A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networks

Introduction

Deep neural networks (DNNs) have reached impressive performance in computer vision problems such as object detection, making them a natural choice for problems like automated driving [1]. However, DNNs are known to be highly vulnerable to faults. For example, even small changes to the input such as adding a customized noise pattern that remains invisible to the human eye, can stimulate silent prediction errors [8]. Similarly, modifying a single out of millions of network parameters, in the form of a bit flip, is sufficient to cause severe accuracy drops [14].

Because DNNs are being deployed in safety-critical applications such as autonomous vehicles (AVs), we need efficient mechanisms to detect errors that cause such silent data corruptions (SDC). Beyond the functional part, trust in the safety of the application requires that the error detectors are interpretable by the user, so that he/she can develop an intuitive understanding of the regular and irregular behavior of the network [2]. In an AV, for example, a user who does not trust an automated perception component due to its opaque decision-making, will not trust a black-box fault monitor either. Therefore, it is important to build interpretable error detectors for DNNs.

The goal of error detection is to supervise a small, yet representative subset of activations - during a given network inference - for comparison with a previously extracted fault-free baseline. This leads to three key challenges: (1) How can one compress the relevant information into efficient abstractions? (2) How can one efficiently perform the anomaly detection process, for complex patterns? (3) Can the anomaly detection decision be understandable to a human, so that insights are gained about the inner workings of the network?

Unfortunately, no existing approach satisfactorily addresses all three of the above challenges (§ 1.1). This paper presents an solution using a monitoring architecture that taps into the intermediate activations only at selected strategic points and interprets those signatures in a transparent way, see Figure 1. Our approach is designed to detect SDC-causing errors due to input corruptions or hardware faults in the underlying platform memory. Our main observation that underpins the method is that an SDC occurs when a fault either increases the values of a few activations by a large margin (referred to here as an activation peak shift), or the values of many activations each by a small margin (activation bulk shift). As Figure 2 shows, the former is observed typically for platform faults, while the latter is observed for input faults. We then use discrete quantile markers to distill the knowledge about the variation of the activation distribution in a given layer. Conceptually, within a faulty layer, we can expect a large change of only the top quantiles for activation peak shifts, and small changes of the lower and medium quantiles for bulk shifts (Figure 2). This idea allows us to produce discriminative features for anomaly detection from a small number of monitored elements, with a single detector.

In summary, we make the following contributions in this paper:

• We demonstrate that even for complex object detection networks, we can identify anomalous behavior from quantile shifts in only a few layers.

• We identify minimal sets of relevant features and discuss their universality across models.

• We efficiently differentiate input and hardware fault classes with a single detector.

• We show that the anomaly detection process can be achieved with algorithmically transparent components, such as decision trees.

The article is structured as follows: Sec. 2 discusses related work, while Sec. 3 describes our experimental setup. We present our method in Sec. 4, and the results of our evaluation in Sec. 5.

Related Work

There are three main categories of related work.

Image-level Techniques: Input faults can be detected from the image itself (i.e., before network inference), in comparison with known fault-free data, resulting for example in specialized blur detectors [15]. However, these techniques do not necessarily relate to SDC in the network, as image-level corruptions may be tolerated by the model.

Activation Patterns: Methods to extract activation patterns range from activation vectors [5] to feature traces [24, 25]. However, these techniques do not scale well to deeper models as they result in a massive number of monitored features and large overheads. Zhao et al. [26] attempt to reduce the monitoring effort by leveraging only activations from selected layers and compressing them with customized convolution and pooling operations. This leads to a rather complex, non-interpretable detector component, and the selection of monitored layers remains empirical.

Anomaly Detection techniques establish clusters of regular and anomalous data to efficiently categorize new input. In single-label problems, such as image classification, fault-free clusters are typically formed by samples that belong to the same individual label [12], suggesting that those samples also share common attributes in the space of intermediate activations. This technique does not generalize to multi-label problems though, such as object detection, as many objects (in the form of bounding boxes and labels) are represented in the same image. More abstracted clustering rules such as the maximum activation range per layer have been proposed [4, 18]. However, these detectors omit more subtle errors within the activation spectrum, for example resulting from input faults. In other work [24, 25, 26], a secondary neural network is trained to perform the detection process. This comes at the cost that the detector then does not feature algorithmic transparency [2] and hence the anomaly decision is not understandable to a human. The same limitations are found in the context of detector subnetworks that are trained to identify adversarial perturbations [19].

Summary: We see that none of the prior techniques satisfactorily address the challenges outlined earlier. We present a new technique to overcome this problem in this paper.

Experimental Setup and Preliminary Study

Models and Datasets: We use the three classic object detection networks Yolo(v3), Single Shot Detector (SSD), and RetinaNet from the open-mmlab [3] framework, as well as the two standard image classification networks ResNet50 and AlexNet from torchvision [21]. Object detection networks are pretrained on Coco [20] and were retrained on Kitti [6], with the following AP50 baseline performances: Yolo+Coco: 55.5%, Yolo+Kitti: 72.8%, SSD+Coco: 52.5%, SSD+Kitti: 66.5%, RetinaNet+Coco: 59.0%, RetinaNet+Kitti: 81.8%. Image classification models were pretrained on ImageNet [17], providing accuracies of 78.0% (ResNet) and 60.0% (AlexNet) for the test setup. The data was split in a ratio of 2:1 for detector training and testing. All models are run in Pytorch with the IEEE-standard FP32 data precision [16].

Fault Modes: Input faults are modeled using torchvision [21] transform functions and are applied in three different magnitudes to the raw RGB images. We select three perturbation patterns that are popular in computer vision benchmarks such as ImageNet-C [11] for our analysis: i) Gaussian noise due to low lighting conditions or noise in the electronic signal of the sensor device. Low (0.1), medium (1), and high (10) noise is tested. ii) Gaussian blur, reflecting for example a camera lens being out of focus. We choose a kernel size of (5, 9) and a symmetric, variable standard deviation (0.3, 1, 3). iii) Contrast reductions simulate poor lighting conditions or foggy weather. We adjust the contrast by a factor between zero (no contrast, gray image) and one (original image). The selected models have different vulnerabilities to input faults, for example, the two image classification models ResNet and AlexNet are highly sensitive to contrast adjustments, but are rather robust to noise and blur faults. For the remaining models, the trend is reversed.

Hardware faults are modeled as single bit flips in the underlying memory and injected using PytorchAlfi [9]. Such flips can occur randomly either in the buffers holding temporary activation values (neuron fault), or in dedicated memory which holds the parameters of the network (weight faults). We group both neuron and weight faults into a single class memory fault. This approach is in line with previous work [4, 7, 13, 18, 23, 24, 25]. We target all convolutional layers.

Fault Metrics: First, detectable uncorrectable errors (DUE) can occur when invalid symbols such as $\mathrm{NaN}$ or $\mathrm{Inf}$ are found among the activations at inference time. During fault injection, we observe DUE events only for memory faults, with rates <1% across all models. DUEs can be generated also at the detector stage, in the process of adding up feature map sums that contain platform errors. The rates for such events vary between 0.2% and 5.1% with our method. While DUE errors may affect the system’s availability, they are considered less critical as they are readily detectable and there is no need for further activation monitoring [7].

In this article, we are concerned therefore only with silent data corruption (SDC), events that lead to a silent alteration of the predicted outcome. For image classification networks, this is represented by a change in the top-1 class prediction. For object detection systems, we use an asymmetric version of the IVMOD metric [23] as SDC criterion, i.e., an image-wise increment in the FP or FN object numbers is counted as SDC. Each experiment was done with a subset of 100 images of the test data set, running 100 random FIs on each image individually. For hardware faults, SDC rates are typically low (~1–3%) since drastic changes will result only from bit flips in the high exponential bits of the FP32 data type [7, 18]. Therefore, an additional 500 epochs with accelerated FI only into the three highest exponential bits are performed for both flavors of memory faults. Overall, the faulty and fault-free data is found to be balanced at a ratio of about 2:1.

Model

Notational Remarks: We use the range index convention, i.e., a vector is given as $\mathbf{x}=(x_i)=(x^i)$, a matrix reads $\mathbf{A}=(A_{ij})$, and similarly for higher-dimensional tensors.

Monitoring Approach: Let us denote a four-dimensional activation tensor that represents an intermediate state of a convolutional neural network as $\mathbf{T}=(T_{n,c,h,w})\in\mathbb{R}^{N\times C\times H\times W}$, where $N$ is the sample number, $C$ the number of channels, $H$ the height, and $W$ the width. We list $n$ as running global sample index, where samples may be further grouped in batches. An output tensor of a specific layer $l\in[1,\ldots,L]$ shall be given as $\mathbf{T}^l$, with $L$ being the total number of monitored layers of the model. Subsets of a tensor with fixed $n,c$ are called feature maps. Our monitoring approach first performs the summation of individual feature maps and subsequently calculates quantile values over the remaining kernels, see Figure 1,

Here $Q_p$ is the quantile function for the percentile $p$ which acts on the $n$-th row of $(F_{n,c})^l$. In other words, $Q_p$ reduces the kernel dimensions $c$ to a set of discrete values where we use the 10-percentiles, i.e., $p\in[0,10,20,30,\ldots,90,100]$. The result is a quantile value set, $q_p$, for a given image index $n$ and layer $l$. Note that both the summation and the quantile operations (and hence the detector) are invariant under input perturbations such as image rotations.

Supervised Layers: We intercept the output activations of all convolutional layers, as those layers provide the vast majority of operations in the selected computer vision DNNs. Yet, the same technique can be applied to any neural network layer.

Reference Bound Extraction: Applied to a separate data set $D_{\mathrm{bnds}}$, the above technique is used pre-runtime to extract reference bounds which represent the minimum and maximum feature sums during fault-free operation:

For $D_{\mathrm{bnds}}$, we randomly select 20% of the training data [4].

Anomaly Feature Extraction: For a given input during runtime, Eqs. (1) to (2) are used to obtain the quantile markers of the current activation distribution. Those are further processed to a so-called anomaly feature vector which quantifies the similarity of the observed patterns with respect to the baseline references of Eq. (3),

Here, $f_{\mathrm{norm}}$ normalizes the monitored quantiles to a range of $(-1,1)$ by applying element-wise ($\epsilon= 10^{-8}$ is a regularization offset)

Intuitively, the result of Eq. (5) will be positive if $a$ is outside the defined minimum ($a_{\min}$) and maximum ($a_{\max}$) bounds (approaching $+1$ for very large positive or negative values). The function is negative if $a$ is within the bounds (lowest when barely above the minimum), and will become zero when $a$ is of the order of the thresholds. In Eq. (4), a shift brings features to a range of $(0,1)$ to facilitate the interpretation of feature importance. Finally, all extracted features are unrolled into a single anomaly feature vector $\mathbf{q} = ((q^l)_p) = [q^1_0, q^2_0, \ldots q^L_0, q^1_{10}, \ldots q^L_{100}]$, that will be the input to the anomaly detector component.

Anomaly Detector: We use a decision tree [10] approach to train an interpretable classifier, leveraging the sklearn package [22]. The class weights are inversely proportionally to the number of samples in the respective class to compensate for imbalances in the training data. As a measure of the split quality of a decision node we use the Gini index [10]. To avoid overfitting of the decision tree, we perform cost-complexity pruning [22] with a factor varying between $1 \times10^{-5}$ and $2 \times10^{-5}$, that is optimized for the respective model.

To investigate fault class identification, we study three different detector modes with varying levels of fault class abstractions and quantify each mode $x \in\{cls, cat, sdc\}$ by precision, $P_x = \mathrm{TP}_x/(\mathrm{TP}_x+\mathrm{FP}_x)$ and recall $R_x = \mathrm{TP}_x/(\mathrm{TP}_x+\mathrm{FN}_x)$. Here we abbreviated true positives (TP), false positives (FP), and false negatives (FN). In the class mode ($cls$), we consider only those detections as true positives where the predicted and actual fault modes (see § 2) coincide exactly. Cases where SDC is detected correctly but the fault class does not match will be counted as either FP or FN in this setting. In the category mode ($cat$), those SDC detections are considered true positives where the predicted and actual fault class fall into the same category of either memory fault or input fault $= \{noise, blur, contrast\}$. That means, fault class confusions within a category will not reduce the performance in this mode. The final precision and recall values for the class and category mode are given as the average over all classes or categories, respectively. Finally, in the mode $sdc$, we consider all cases as true positives where SDC was correctly identified regardless of the specific fault class. This reflects a situation where one is only interested in the presence of SDC overall, rather than the specific fault class.

Results

Detector Performance

Error Detection: Tab. 1 shows the precision, recall, and decision tree complexity for the studied detectors and models. When all extracted features are leveraged by the decision tree classifier (referred to as full model), the average class-wise detection precision varies between 93.9% (ResNet) and 97.5% (RetinaNet+Kitti), while the recall is between 97.1% (RetinaNet+Coco) and 99.1% (Yolo+Kitti). If only the fault category needs to be detected correctly, we find $P_{\mathrm{cat}} > 95\%$ and $R_{\mathrm{cat}} > 94\%$. Correct decisions about the presence of SDC only are done with $P_{\mathrm{sdc}} > 96\%$ and $R_{\mathrm{sdc}} \ge98\%$. Across models, we observe (not shown in Table 1) that the most common confusion are false positive noise detections, leading to a reduced precision in the individual noise class (worst case 75.8% for ResNet). The recall is most affected by memory faults (lowest individual class recall 90.6% for RetinaNet+Coco).

Model	P(%)			R(%)			DT
	$P_{\mathrm{cls}}$	$P_{\mathrm{cat}}$	$P_{\mathrm{sdc}}$	$R_{\mathrm{cls}}$	$R_{\mathrm{cat}}$	$R_{\mathrm{sdc}}$	$N_{\mathrm{ft}}/N_{\mathrm{l}}$
Yolo+Coco
full	95.8	96.4	96.1	98.2	98.6	98.4	825/75
red (avg)	93.3	94.6	93.4	97.4	96.3	96.7	2/2
Yolo+Kitti
full	97.3	97.5	97.4	99.1	99.3	99.2	825/75
red (avg)	92.6	92.1	92.0	97.3	96.4	96.8	3/2
SSD+Coco
full	96.6	97.2	96.6	98.2	98.5	98.3	429/39
red (avg)	95.2	96.3	94.9	96.5	94.5	95.9	3/3
SSD+Kitti
full	96.0	97.1	96.2	98.4	98.7	98.6	429/39
red (avg)	92.8	94.6	92.1	98.0	97.7	98.2	2/2
RetinaNet+Coco
full	96.6	95.7	96.9	97.1	94.9	98.0	781/71
red (avg)	96.6	96.6	96.5	97.0	94.6	98.2	2/2
RetinaNet+Kitti
full	97.5	97.3	97.5	98.6	98.2	98.7	781/71
red (avg)	96.2	96.6	95.9	98.6	97.8	98.9	2/2
ResNet+Imagenet
full	93.9	98.3	97.6	98.1	99.6	99.4	583/53
red (avg)	92.1	97.6	96.7	98.3	99.6	99.5	3/3
AlexNet+Imagenet
full	96.1	98.3	97.3	98.4	99.2	99.0	55/5
red (avg)	93.2	96.8	95.0	98.0	99.0	98.8	4/3

The detection rates of the full model in Table 1 outperform the ones reported in the comparable approach of Schorn et al. [25] (using feature map tracing) and the blur detection in Huang et al. [15] in terms of precision and recall. When using alternative metrics (not shown in Table 1) for comparison with other detector designs, we find that our method achieves class-wise misclassification rates ranging between 0.7% and 2.0%, depending on the model, which is on par with the results for example in Cheng et al. [5]. Similarly, the calculated class-wise true negative rates vary between 99.6% and 99.8%, reaching or exceeding the classifier performance in Zhao et al. [26]. Note that all mentioned references are limited to image classification networks.

Feature Reduction: The number of monitored features can be drastically reduced without significantly affecting the detection performance. This means that many quantiles represent similar information and further distillation can be applied. For feature reduction, we follow two steps: First, all quantile features of the full model are ranked according to their Gini importance [22] in the decision tree. Then, we retrain the classifier with a successive number of features, starting from the most important one only, to the two most important ones, etc. A reduced model is accepted as efficient if it recovers at least 95% of both the precision and recall performance of the original model with all features.

Figure 3 shows the results of the feature reduction. Inspecting performance trends from larger to smaller feature numbers, we observe that the detection rate stagnates over most of the elimination process, before dropping abruptly when the number of used features reduces beyond a limit. On average, the number of monitored features and layers that are required to maintain close-to-original performance (as defined above) are as few as 2 to 4 and 2 to 3, respectively. For a model like Yolo, this means that only 2 out of the 75 convolution layers have to be supervised. The average characteristics of the resulting detector models is shown in Table 1 as reduced (red) model.

Minimal Monitoring Features

Minimal Feature Search: The feature reduction process in § 4.1 demonstrates that only few strategic monitoring markers are needed to construct an efficient detector model. In this section, we elaborate further to what extent the model can be compressed, and which features are the most relevant. We apply the following strategy, starting from a full classifier model using all quantile features: 1) Apply the feature reduction technique described in § 4.1 to identify minimal monitoring features that maintain at least 95% of the original precision and recall. This combination of features is added to a pool of minimal model candidates. 2) A new instance of the full model is initiated and all feature candidates from the pool are eliminated. Return to the first step to find alternative candidates until a certain search depth (we choose 24) is exhausted.

Universal Trends: The identified minimal feature combinations are shown in Figure 4. We find that just 2 features from 2 different layers are sufficient to constitute an efficient error detector for all studied models except for AlexNet (4 features from 3 layers).

Almost universally, one of the monitored layers needs to be among the very last layers of the deep neural network. Since memory faults are injected randomly across the network, errors in the last layers would go unnoticed otherwise. Only for SSD models, it turns out that most of the SDC faults occur in earlier layers, so that a supervision of the last layers is less crucial to achieve a similar statistical detection performance. We observe that it is favorable to supervise a higher percentile (e.g., $q_{100}$) in the later layers, especially in more shallow networks (AlexNet and SSD). This is because in shallow networks, peak shifts have a shorter propagation path and hence it is more important to intercept faults directly. This can only be achieved by the highest percentiles. In models with ReLU activation functions (all except Yolo here), the minimum quantile does not serve as a meaningful peak shift monitor as negative activations are clipped automatically.

A second monitoring marker should to be set in the first half of the network layer stack. This helps to identify input faults (which are interceptable from the very first layer) and discriminate them from memory faults. Either a low or high percentile can be chosen for supervision.

Explainability: Given the above generalizable trends and the fully transparent nature of the classifier, we can make statements about the inner workings of the DNN that correlate a given input with an anomalous or regular outcome. Those statements can be interpreted intuitively by a human as a proxy of a decision, and hence qualify as an explanation [2].

Overhead

We measure the average inference time per image when running the supervised model on random input, using the Torch profiler [21]. The profiled overall self compute time in Figure 5 is shared between CPU and GPU. Compared to the feature map tracing method of Schorn et al. [24, 25], the quantile operation introduces additional compute, but at the same time saves the time of storing large tensors, due to the compression of many feature sums into only a few quantiles. Between these two opposing trends, full quantile monitoring turns out to be faster than feature map tracing for all the studied models except for the shallow AlexNet, as shown in Figure 5. If only selected layers are monitored to create a reduced classifier, the overhead can be decreased significantly. We find that the impact of minimal quantile monitoring on the overall inference time is between $+0.3\%$ and $+1.6\%$ for all studied object detection DNNs. For the image classification networks, on the other hand, quantile monitoring imposes a more significant overhead of $+10.7\%$ (ResNet) and $+53.8\%$ (AlexNet). This is because those networks have a much smaller number of parameters such that the relative impact of quantile extraction with respect to the total number of operations is higher. Across all models, minimal quantile monitoring is $> 10\%$ faster than feature map tracing. In absolute numbers, the respective saving in inference time can be up to $\sim10$ms, which is a significant improvement for applications operating at real-time, for example object detection in a self-driving vehicle.

Comparison with Other Detector Approaches

Alternative to a decision tree, we can deploy a linear machine learning model for error detection (similar to [24]). We study the feasibility of doing so in this section. For this setup, we select Yolo+Kitti to train a classifier for 1000 epochs using the Adam optimizer and cross entropy loss. A batch size of 100 and learning rates optimized between $1 \times10^{-4}$ and $5 \times10^{-3}$ were chosen. In the simplest form, with a multi-layer-perceptron, the algorithmic transparency is preserved and we find $P_{\mathrm{cls}} = 86.0\%$ and $R_{\mathrm{cls}} = 95.7\%$. If more hidden linear layers are added, higher detection rates can be achieved at the cost of explainability. For example, including one extra hidden layer with 64 neurons [24], we find a performance of $P_{\mathrm{cls}} = 88.9\%$ and $R_{\mathrm{cls}} = 96.3\%$, with three such extra layers we obtain $P_{\mathrm{cls}} = 91.7\%$ and $R_{\mathrm{cls}} = 95.1\%$. Compared to decision trees, however, this strategy suffers from more complex hyperparameter tuning and large training times. Therefore, decision trees are a better fit for our use case.

Summary and Future Work

In this paper, we show that critical silent data corruptions in computer vision DNNs (originating either from hardware memory faults or input corruptions) can be efficiently detected by monitoring the quantile shifts of the activation distributions in specific layers. In most studied cases, it is sufficient to supervise two layers with one quantile marker each to achieve high error detection rates up to $\sim96\%$ precision and $\sim98\%$ recall. We also show that the strategic monitoring location can be associated with the concept of intercepting bulk and peak activation shifts, which gives a novel, unifying perspective on the dependability of DNNs. Due to the large degree of information compression in this approach, the compute overhead of the approach is in most cases only between 0.3% and 1.6% compared to the original inference time, and outperforms the comparable state of the art. In addition, we show that the method contributes to the model’s explainability as the error detection decision is interpretable and transparent. For future work, we can further guide the search for optimized minimal feature combinations, for example, by taking into account specifics of the model architecture.

Acknowledgement: We thank Neslihan Kose Cihangir and Yang Peng for helpful discussions. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 956123. This work was partially funded by the Federal Ministry for Economic Affairs and Climate Action of Germany, as part of the research project Safe-Wahr (Grant Number: 19A21026C), and the Natural Sciences and Engineering Research Council of Canada (NSERC).