WorksComputer Science
Understanding Contextual Bias in Agricultural CNNs Through Hidden-Neuron Concept Induction
Abstract
Convolutional Neural Networks (CNNs) can classify images using background-dominated patches that contain little or none of the target object, showing that they exploit contextual information. Previous studies detected this behavior through classification accuracy but say little about what CNN hidden layers represent. We investigate whether Concept Induction can reveal interpretable visual properties associated with hidden-neuron activations on contextual image patches. Using five agricultural datasets with high, moderate, and low contextual bias, we analyze four CNN architectures at three network depths and five patch locations. Quantitative image descriptors of each patch are encoded in a knowledge graph and used as background knowledge for Concept Induction, which derives logical expressions characterizing neuron activation. These expressions are tested on held-out images using the Mann-Whitney U test and Cliff's delta. Of 48,013 tested neuron-feature pairs, 13,164 (27.4%) were validated. Validation was most frequent at intermediate depths (mean 40.2% at mid layers versus 12.0% at deep layers), and no patch location dominated across datasets. Among the tested descriptors, edge, Gabor, and Tamura texture features had the highest validation rates. Stronger contextual bias did not correspond to higher neuron-feature validation rates: moderately biased Sugarcane had the highest rate (40.5%) and highly biased Cassava the lowest (19.2%). These results suggest that behavioral context dependence and the semantic interpretability of hidden neurons capture different aspects of CNN behavior, and that Concept Induction can complement patch-based bias detection.
Introduction
Convolutional Neural Networks (CNNs) have achieved remarkable performance across a wide range of image-classification tasks, establishing them as a fundamental component of modern computer vision systems [28, 21]. However, strong predictive performance does not necessarily imply that a model has learned the visual characteristics intended by the classification task. Deep neural networks can exploit unintended correlations and other predictive shortcuts present in the training data, achieving high performance without relying primarily on the characteristics of the target object [17, 26, 56]. Because CNNs learn statistical patterns directly from training data, such shortcuts can arise from contextual information, including background patterns, object co-occurrences, acquisition artifacts, and other incidental visual cues that correlate with class labels rather than from the object’s genuine defining features [6, 68]. Evidence of such behavior has been reported across diverse application settings, demonstrating that reliance on contextual or background information is not restricted to a particular dataset or domain [6, 35]. Such reliance can become problematic when these associations are specific to the training distribution, as changes in background composition, acquisition conditions, or other contextual characteristics may affect model robustness and generalization [17, 3, 35].
Standard evaluation measures, such as accuracy, precision, recall, and F1-score, quantify how successfully a model predicts class labels, but provide limited information about the visual evidence underlying those predictions. Consequently, a model that relies on unintended contextual cues may still achieve strong performance when the same correlations are present in the training and evaluation data[60, 50, 70]. This limitation has motivated diagnostic approaches that systematically modify the visual information available to a trained model, including object masking, background manipulation, image occlusion, cropping, and region- or patch-based evaluation[3, 6, 46, 37]. By restricting or altering portions of an image and observing the resulting model behavior, such approaches can reveal whether predictions remain supported when expected object information is reduced or contextual information is modified. In particular, spatial patch-based evaluation provides a direct means of examining whether restricted image regions retain sufficient information for classification. Predictive performance substantially above the expected chance level on such patches can therefore provide evidence that a model has learned class-discriminative information from spatially localized image content, including contextual regions that may not contain the complete target object[57, 37, 9].
Building on this form of diagnostic evaluation, our prior studies systematically investigated contextual dependence by evaluating CNNs on spatially restricted image regions across different classification settings. Experiments involving biomedical images, a manually curated object-classification dataset, and agricultural image datasets showed that trained CNNs can retain non-trivial classification capability even when predictions are made from only a portion of the original image [46, 37, 9]. In the agricultural setting, five predefined spatial regions such as top-left, top-right, center, bottom-left, and bottom-right were evaluated across multiple datasets and CNN architectures, revealing substantial differences in the extent to which models could classify images from restricted regions [9]. These findings provided a behavioral characterization of context dependence and demonstrated that its severity varies across datasets and model architectures.
Patch-based evaluation, however, describes whether predictive information is present within restricted image regions without explaining what visual information the network internally represents from those regions. Classification performance alone does not reveal which hidden neurons respond to properties of the patches, how such representations vary across network depth, or which visual characteristics are associated with their activation. Consequently, identifying context-dependent classification behavior represents only one part of understanding how contextual information is encoded within a trained CNN. Examining the internal representations associated with these image regions is necessary to move from behavioral observations of context dependence toward a semantic characterization of the learned representations.
Existing interpretability methods provide several ways to examine such internal representations. Attribution and localization methods identify image regions contributing to predictions [55, 61, 72], while concept- and neuron-level approaches such as TCAV [27], ACE [19], Network Dissection [5], and CLIP-Dissect [44] associate learned representations with human-interpretable concepts. However, these approaches generally characterize internal representations through attribution, concept association, or semantic labeling rather than logically structured, compositional descriptions of neuron activation grounded in quantitative visual properties or support reasoning over such properties using explicit background knowledge. This motivates a complementary approach in which hidden-neuron activation patterns are connected to structured symbolic representations, and formal concept learning characterizes the visual conditions associated with neuron responses [8].
In this work, we investigate the hidden-neuron representations associated with the contextual image patches used in our preceding agricultural context-bias analysis. We develop a neuro-symbolic analysis framework that integrates hidden-layer activation analysis with quantitative image descriptors and Description Logic Concept Induction. For selected hidden layers of four CNN architectures, neuron activations are extracted from predefined spatial image patches, while complementary visual characteristics of the same patches are represented using UDAT image descriptors. These descriptors are discretized and encoded within OWL knowledge graphs, providing structured background knowledge for Concept Induction. DL-Learner is then used to derive Description Logic concepts characterizing the visual properties associated with selected neuron activation patterns. The induced associations are subsequently evaluated on held-out images using statistical significance and effect-size criteria, providing a systematic mechanism for identifying semantically validated neuron–feature associations.
Using this framework, we examine representations across five agricultural image datasets selected to span the high-, moderate-, and low-context-bias conditions identified in the preceding patch-based study. The analysis considers multiple CNN architectures, three representative network depths, five spatial patch locations, and complementary families of quantitative image descriptors. This experimental design allows us to investigate not only whether semantic associations can be identified for hidden neurons, but also where these associations emerge within CNN feature hierarchies, how they vary spatially, which visual properties most frequently characterize them, and whether their prevalence corresponds to the degree of context dependence observed behaviorally.
Specifically, we investigate the following research questions:
RQ1: At which depths of CNN architectures do semantically interpretable neuron representations associated with contextual image patches emerge? At which layers of CNN architectures do semantically interpretable, context-sensitive representations emerge?
RQ1.1: Which spatial patch locations consistently produce semantically validated neuron representations across agricultural datasets?
RQ2: Which image-feature families most consistently characterize the hidden-neuron representations associated with contextual image patches?
RQ3: How does the emergence of validated semantic concepts relate to the degree of context bias observed across agricultural datasets?
The main contributions of this work are as follows:
We extend the analysis of context dependence in agricultural image classification from observable model behavior to internal CNN representations through a neuro-symbolic framework that associates hidden-neuron activation patterns with semantically interpretable visual properties using Description Logic Concept Induction.
We introduce a systematic analysis and validation pipeline for identifying neuron–feature associations, combining targeted neuron selection, symbolic concept generation, and independent statistical validation on held-out images.
We provide a multi-dimensional empirical characterization of semantically validated representations across five agricultural datasets, four CNN architectures, three representative network depths, five spatial patch locations, and multiple visual feature families, revealing how these associations vary across depth, spatial region, and visual characteristics.
We relate representation-level semantic validation to previously observed behavioral context dependence and show that stronger patch-based context bias does not necessarily correspond to a greater prevalence of semantically validated neuron–feature associations, highlighting that behavioral context dependence and semantic concept emergence capture distinct aspects of CNN behavior.
The remainder of this paper is organized as follows. § 2 reviews related work on contextual bias, CNN interpretability, and concept-based and neuro-symbolic explanation methods. § 3 introduces the background concepts and technical foundations necessary to understand the proposed framework. § 4 describes the datasets, hidden-layer analysis, symbolic representation construction, Concept Induction, and statistical validation framework. § 5 presents and discusses the experimental findings with respect to the research questions. Finally, § 6 summarizes the main findings, discusses limitations and implications, and outlines directions for future work.
Related Work
Context bias and shortcut learning in image classification
Image classifiers can learn regularities of a dataset instead of properties of the target object, a problem recognized even before deep learning became dominant. Early investigation showed that popular recognition benchmarks have their own distinctive signature [66]. A classifier could tell which dataset an image came from, and models trained on one dataset generalized poorly to others; such failures were later described as emph shortcut learning [17]. A shortcut is a decision rule that works on the benchmark but fails when conditions change because it relies on spurious cues that only happen to correlate with the label.
Background reliance is the best documented shortcut in natural images. Camera-trap classifiers lose much of their accuracy at locations not seen during training [6]. Images with the object removed still yield non-trivial accuracy on ImageNet [68]. One study found a model that recognized aeroplanes from artificial padding patterns and horses from a source watermark [29]. Similar reliance on unintended cues has been observed in medical imaging, where CNNs could identify the hospital that produced a chest radiograph with almost perfect accuracy [70]. Since pneumonia rates differed between hospitals, this site information leaked into the pneumonia predictions. Likewise, COVID-19 detectors were also found to rely on confounders rather than lung pathology, leading to poor generalization across hospitals [13]. In agricultural imaging, a model trained on only eight background pixels of PlantVillage images reached 49% accuracy, compared with a chance level of 2.6% [43]. Similarly, plant phenotyping models were found to make confounder-driven decisions, and correcting them required expert feedback during training [54].
A related line of work measures dataset bias by classifying image regions that contain little or no task-relevant information. Small, seemingly blank sub-images extracted from object recognition benchmarks were classified with accuracy well above chance [38]. Similar findings were reported in biomedical imaging, where CNNs achieved high classification accuracy using image regions without biomedical content [14].
Building on this direction, subsequent studies investigated contextual bias across different image-classification domains. Image transformations were used to distinguish background bias from useful contextual information across several image-classification benchmarks [16]. An investigation tested thirteen cancer pathology datasets with four CNN architectures [46]. Background regions without clinical content were classified with up to 93% accuracy. More recently, contextual bias was examined across eight agricultural image datasets, where background-dominated patches achieved above-chance classification accuracy in six datasets [9]. Although class imbalance explained part of this effect, it did not fully account for the results.
While these studies show that context bias appears in several domains through classification performance, , they do not investigate how the visual characteristics of contextual regions are reflected in CNN hidden-neuron activations. The present work addresses this gap by applying Concept Induction to analyze the relationships between visual properties of image patches and hidden-neuron activations, using agricultural datasets exhibiting high, moderate, and low levels of contextual bias identified in [9].
Explainability of CNN hidden representations
Early work on CNN interpretability focused on visualizing the patterns that activate individual units. Deconvolutional networks were used to project intermediate activations back into pixel space [71], while gradient-based approaches generated class-maximizing images and saliency maps [61]. Subsequent work introduced regularization techniques to make such visualizations easier to recognize [69]. These studies provided insights into the hierarchical representations learned by CNNs, where early layers respond to low-level features such as edges and colors, while deeper layers respond to more complex patterns such as object parts and objects. However, interpreting these visualizations remains largely qualitative. Attribution methods such as Grad-CAM highlight the image regions contributing to a particular prediction [55]. They show where the model looks, but not what it has learned to represent.
Network Dissection introduced a quantitative approach to interpreting individual neurons by comparing thresholded activation maps with pixel-level annotations of colors, textures, materials, parts, objects, and scenes [5]. This approach was subsequently extended to logical combinations of concepts, revealing both abstract neurons and neurons that respond to several unrelated concepts [40].
Other methods investigate concepts represented in the activation space rather than focusing exclusively on individual neurons. TCAV learns a concept direction from example images and measures its effect on a class prediction [27], while ACE identifies concepts automatically by clustering image segments in activation space [19]. More recent methods use vision-language models to generate natural-language descriptions of hidden neurons [44, 22]. Neuron-level analysis has also been extended to shortcut learning, where individual neurons can be scored by how strongly they depend on spurious features [30].
Despite these advances, existing approaches have limitations in the context of our study. Concept-based explanations can depend on the probe dataset and the available concept vocabulary, potentially producing different interpretations under different settings [49]. Traditional Dissection-style methods need pixel-level concept masks, which are generally unavailable in agricultural image datasets. Similarly, labels produced by vision-language models come from an opaque model themselves, so they inherit its lack of transparency. Finally, all of these methods provide labels, vectors, or scores; they do not generally induce logical statements grounded in explicit background knowledge. Logical statements of this kind would express the relationships between visual features and neuron activations to be represented as formally defined concepts that can be investigated within a symbolic reasoning framework.
Neuro-symbolic explanation and Concept Induction
Neuro-symbolic AI combines neural learning with symbolic knowledge representation and reasoning [24, 31]. In explainable AI, ontologies can provide a structured vocabulary for interpreting neural network behavior. For example, classifiers have been trained to map a neural network’s activations to ontology concepts, allowing model outputs to be explained using symbolic terms [12].
Concept Induction provides an alternative approach in which explanations are learned directly as logical class expressions. Given a knowledge base expressed in Description Logic (DL), along with sets of positive and negative individuals, the objective is to identify a class expression that covers the positive examples while excluding the negative examples [34, 51]. The resulting DL concepts have formal semantics and can express complex relationships through combinations of classes and properties using logical constructors such as conjunction, disjunction, and quantifiers. These expressions are interpreted with respect to the underlying background knowledge.
DL-Learner is one of the most established systems for concept induction that supports OWL and RDF background knowledge and provides access to reasoning-based learning algorithms, including refinement-based learners such as CELOE [32, 7]. However, refinement-based approaches can be computationally expensive due to the large number of reasoning operations required during concept search. To address this limitation, Efficient Concept Induction from Individuals (ECII) was introduced as a computationally efficient alternative, although with a more restricted search space [51].
Concept Induction has previously been explored for explaining neural network predictions and internal representations. A Semantic Web-based approach demonstrated how induced concepts could be used to explain the classifications of trained neural networks [52], while a Wikipedia-derived class hierarchy was used to generate meaningful explanations of differences between data sets in ways that people found meaningful [67]. This direction was subsequently extended to hidden-neuron analysis, where images that strongly activate a neuron were treated as positive examples and those producing weak activations as negative examples [11, 10]. The induced concepts are then validated against new, purpose-retrieved images to confirm that they describe the neuron’s behavior. This pipeline produced meaningful labels for dense-layer neurons of a scene classifier. The framework was subsequently automated [1] and later studies utilized this with different training targets and on the SUN2012 dataset [2, 53].
Previous applications of Concept Induction to hidden-neuron analysis have two limitations. Its background knowledge came from object annotations of whole images, and it analyzed only the last dense layer. The resulting concepts therefore say which objects a neuron activation responds to, but not which visual properties drive it, or how these change with network depth and image position. We address both of these limitations by building background knowledge from quantitative descriptors of contextual patches, and applying Concept Induction at three network depths.
Image descriptors and semantic background knowledge
Before the widespread adoption of deep learning, images were usually described with handcrafted descriptors that have exact mathematical definitions. Haralick features capture texture statistics derived from gray-level co-occurrence matrices (GLCMs), including measures such as contrast, correlation, and entropy [20], while Tamura features approximate perceived texture properties, such as coarseness, contrast, and directionality [64]. Gabor filters measure texture patterns across several scales and orientations [36], whereas Local binary patterns describe fine texture and are invariant to gray-scale changes and rotation [45]. These descriptors have explicit mathematical definitions and correspond to visual properties that can be named. A single value, such as high entropy, can therefore be read directly, unlike a learned CNN feature.
WND-CHARM extended the use of handcrafted features by introducing a comprehensive set of image descriptors applicable across different image domains [48, 59]. It extracts features from the raw image and its Fourier, wavelet, and Chebyshev transforms, including combinations of these transformations. The descriptor families include polynomial decompositions (Zernike and Chebyshev), texture features (Haralick, Tamura, and Gabor), edge and object statistics, and multiscale histograms. Together, these descriptors characterize different aspects of image appearance, including texture, frequency content, structure, and intensity distribution. The set was designed to work across image domains without tuning. UDAT builds on these descriptors as a general quantitative analysis tool [58]. Previous research has demonstrated that handcrafted and learned features can provide complementary information [41], while ImageNet-trained CNNs have been shown to exhibit a strong reliance on texture information [18]. These findings motivate the use of quantitative image descriptors to investigate the visual characteristics associated with CNN hidden-neuron activations.
While handcrafted descriptors have traditionally been used for image classification and quantitative analysis, we employ them to construct symbolic background knowledge for explaining CNN hidden-neuron activations. In this work, UDAT converts low-level visual characteristics of each patch into structured descriptors, which are subsequently discretized and represented in OWL. This representation enables Concept Induction to identify logical descriptions of the visual properties associated with neuron activation patterns.
Preliminaries
Knowledge graphs and Description Logic representation
Knowledge graphs provide a structured representation of entities and their relationships, typically expressed as nodes connected through labeled relations. When supported by an ontology, a knowledge graph additionally provides a formal vocabulary for describing the types of entities, their properties, and the relationships that may hold among them. This representation enables data from individual instances to be combined with formally defined domain knowledge and supports logical reasoning over the resulting structure [25, 23].
An ontology-based knowledge graph can be viewed as consisting of a terminological component (TBox) and an assertional component (ABox). The TBox defines the classes and properties used to describe a domain and the relationships among these classes, while the ABox contains assertions about individual instances. For example, an image may be represented as an individual of the class Image and connected through an object property to another individual representing one of its visual characteristics.
Description Logics (DLs) provide a formal knowledge-representation framework underlying ontology languages such as OWL. In addition to atomic classes and properties, DLs allow complex class expressions to be constructed using logical operators such as conjunction (), disjunction (), negation (), and existential () and universal () restrictions [25]. For example, an expression of the form
describes images that are related through hasFeature to an instance belonging to FeatureRange. Such expressions provide a symbolic representation that can be automatically reasoned over and form the basis for the Concept Induction approach used in this work.
Concept Induction
Concept Induction [34] is a symbolic learning approach that derives Description Logic class expressions from positive and negative examples represented within a knowledge base. Given a set of positive examples , a set of negative examples , and a knowledge base , Concept Induction searches for a class expression that characterizes the positive examples while distinguishing them from the negative examples. Ideally, an induced expression satisfies for all while for all . When perfect separation is not possible, candidate expressions can instead be evaluated according to how accurately they cover the positive and negative examples.
In this work, Concept Induction is performed using DL-Learner [7], a framework for learning Description Logic concepts from examples and structured background knowledge. Specifically, we employ its CELOE (Class Expression Learning for Ontology Engineering) [33] algorithm, which searches the space of possible class expressions using refinement operators and evaluates candidate expressions according to their ability to distinguish positive from negative examples. The resulting expressions can combine classes and relationships defined in the underlying ontology, producing symbolic descriptions of characteristics shared by the positive examples.
Within our analysis framework, this capability provides the connection between numerical CNN activation behavior and symbolic descriptions of image characteristics. Images exhibiting contrasting neuron activation behavior can be represented as positive and negative examples, while their visual characteristics are encoded in an OWL knowledge graph. Concept Induction can then identify class expressions describing visual properties associated with the neuron’s activation behavior. The construction of these examples and the application and validation of the induced concepts are described in § 4.
UDAT image descriptors
UDAT [58] is a quantitative image analysis tool based on WND-CHARM [48, 59] that extracts a broad range of numerical descriptors characterizing image content. These descriptors are computed from the original image as well as from image transformations, including Fourier, wavelet, Chebyshev, edge, color, and hue transforms, and combinations of these transformations (e.g., the Fourier transform of the wavelet transform). For each contextual image patch, UDAT generates 4,041 numerical descriptors covering different aspects of image appearance, including texture, intensity distribution, edges, and structural patterns.
In this study, we use only descriptors computed on the raw image (338 descriptors per patch, organized into 13 feature families). Descriptors of transformed images describe the image in a different domain (e.g., frequency coefficients of an edge map) and therefore cannot be related directly to visual properties of the patch. Whereas raw-image descriptors measure properties such as texture, edges, or intensity distribution that remain interpretable in the resulting explanations. Table 1 summarizes the feature families, the number of numerical descriptors associated with each family, and the visual properties they quantify. Together, they cover intensity and color distributions, texture, edges and objects, and frequency- and shape-related content. Each numerical descriptor is treated as an individual visual attribute for subsequent discretization and knowledge graph construction. The selection of feature families for individual CNN layers and their use in Concept Induction are described in § 4.
| Feature family | # Values | Visual property quantified |
| Intensity statistics | 8 | Global distribution of pixel intensities (e.g., mean, median, standard deviation, entropy) |
| Color histogram | 19 | Distribution of colors in the image |
| Combined first four moments | 48 | Mean, standard deviation, skewness and kurtosis of intensity over image regions |
| Multiscale histograms | 24 | Intensity histograms computed at several bin resolutions |
| Edge statistics | 28 | Magnitude and direction of intensity gradients (means, medians, variances, histograms, homogeneity) |
| Object statistics | 34 | Properties of segmented objects: count, area, distances, Euler number, centroid |
| Haralick texture | 28 | Statistics of the gray-level co-occurrence matrix, e.g., contrast, correlation, entropy [20] |
| Tamura texture | 6 | Perceptual texture properties: coarseness, contrast, directionality [64] |
| Gabor filters | 7 | Responses to oriented band-pass filters at several frequencies |
| Fractal features | 20 | Self-similarity and roughness of image structure |
| Radon transform | 12 | Intensity projections along multiple angles, capturing oriented structure |
| Chebyshev–Fourier | 32 | Coefficients of orthogonal polynomial expansions describing spatial-frequency content |
| Zernike moments | 72 | Orthogonal moments describing shape and spatial distribution of intensity |
| Total | 338 |
Table 1. UDAT feature families considered in this study, the number of numerical descriptors associated with each family, and the visual properties they quantify.
Prior patch-based context-bias study
This study builds directly on a previous empirical investigation of context bias in CNN-based agricultural image classification [9]. The earlier work evaluated whether CNN classifiers could retain predictive performance when semantically meaningful content was largely removed from the input images. The study covered eight widely used plant and agricultural image datasets: PlantVillage [39], PlantDoc [62], DeepWeeds [47], Cassava Leaf Disease [15], WeedCrop [63], OpenSprayer [4], Sugarcane Leaf Disease [65], and Oxford-102 Flowers [42]. These datasets span plant disease recognition, weed identification, crop classification, spraying-status recognition, and fine-grained flower classification, and include both controlled and field-acquired imagery. Four CNN architectures were evaluated: ResNet50V2, InceptionV3, DenseNet121, and VGG16. For each dataset, the models were trained on the original images and subsequently evaluated on predefined spatial image patches. Five spatial regions were considered for every image: top-left, top-right, center, bottom-left, and bottom-right. Each patch had an original size of () pixels and was resized to the input dimensions required by the corresponding CNN architecture. The original class label was preserved for each extracted patch, enabling direct evaluation of whether the trained classifier could predict the original class using only a spatially restricted region of the image. Most patches contained little or no agriculturally relevant visual content, although the center patch could occasionally include part of the primary subject.
Context dependence was assessed by comparing classification accuracy on the patch datasets with the corresponding random-chance accuracy of each classification task. For dataset containing classes, the uniform random-chance accuracy is:
Patch performance substantially above this baseline was interpreted as evidence that the CNN could exploit contextual or dataset-specific signals even when the primary agricultural object or disease evidence was absent. Such signals may include background texture, lighting patterns, framing characteristics, acquisition artifacts, or other recurring visual regularities associated with the class labels.
| Dataset | Tasks | # Images | Acquisition Setting | # Classes | Random Chance | Patch Acc. Range | Margin | Context-Bias Category |
| PlantVillage | Disease classification | ∼54,000 | Controlled | 15 | 6.7% | 9.1–19.1% | 6.2 pp | Moderate |
| PlantDoc | Disease classification | 2,569 | Field | 27 | 3.7% | 2.0–7.1% | 0.03 pp | Low |
| DeepWeeds | Species classification | 17,509 | Field | 9 | 11.1% | 51.3–55.0% | 40.6 pp | High |
| Cassava | Disease classification | 21,397 | Field | 5 | 20% | 16.0–55.8% | 13.6 pp | High |
| WeedCrop | Crop/weed classification | 2,822 | Field/UAV | 2 | 50% | 85.6–92.4% | 41.1 pp | High |
| OpenSprayer | Weed classification | ∼10,000 | UAV / aerial | 2 | 50% | 58.4–80.6% | 20.2 pp | High |
| Sugarcane | Disease detection | ∼3,000+ | Field | 11 | 9.1% | 11.4–22.8% | 7.8 pp | Moderate |
| Oxford 102 | Species classification | ∼8,200 | Field | 102 | 0.98% | 0.6–3.9% | 0.7 pp | Low |
Table 2. Summary of dataset characteristics and patch-based context-bias results reported in the previous study. “Chance Acc.” denotes random-chance accuracy, and “Patch Acc.” denotes the reported patch accuracy range.
The previous analysis revealed considerable variation in context dependence across datasets. Deep-Weeds, WeedCrop, OpenSprayer, and Cassava exhibited pronounced above-chance performance on patch-only inputs, whereas PlantVillage and Sugarcane showed more moderate context dependence. In contrast, PlantDoc and Oxford-102 Flowers produced patch accuracies close to their corresponding chance baselines, indicating comparatively lower dependence on restricted contextual regions. The study also found that susceptibility to context-dependent behavior varied across CNN architectures, demonstrating that architectural design alone does not guarantee robustness against spurious contextual correlations. These findings established that patch-level predictive behavior can reveal the presence and relative degree of contextual dependence in CNN-based agricultural classifiers. However, the analysis remained primarily behavioral: it demonstrated that models could exploit restricted image regions, but did not identify which visual characteristics within those regions were represented by the CNNs or how such representations emerged across hidden layers. The previous study therefore identified feature-level investigation as an important direction for future work, including the use of image descriptors such as those provided by UDAT [58] to characterize the visual information contained in contextual patches.
The present study extends this earlier work from detecting context dependence to explaining its internal representation. Using the same patch-based experimental setting, we investigate hidden-neuron activations across multiple CNN depths and characterize them using UDAT-derived visual descriptors, knowledge graph representations, Description Logic Concept Induction, and held-out statistical validation. This enables analysis of not only whether contextual bias is present, but also which semantically interpretable visual feature families are associated with the corresponding hidden-neuron representations.
Methodology
Neuro-Symbolic Analysis Framework
This study adapts a neuro-symbolic hidden-neuron explanation approach based on Concept Induction [11] to investigate the internal representations associated with contextual image patches. The analysis builds on the agricultural context-bias setting described in § 3.4 and shifts the focus from observing patch-level classification behavior to characterizing the visual properties represented within CNN hidden layers. Figure 1 presents an overview of the analysis pipeline. For each dataset and spatial patch location, patch images are provided to the corresponding trained CNN, and hidden-layer activations are collected at three network depths: mid, mid-deep, and deep. The resulting activation patterns are used to identify neurons suitable for subsequent explanation analysis.

Figure 1. Overview of the neuro-symbolic analysis pipeline for interpreting CNN hidden-neuron activations from contextual image patches.
In parallel, the visual characteristics of the patch images are extracted using UDAT image descriptors. The extracted feature values are transformed into symbolic representations and encoded in an OWL knowledge graph, which provides the background knowledge used for Concept Induction. For each selected neuron-feature-family combination, images associated with contrasting activation behavior are used as positive and negative examples. Concept Induction then searches for Description Logic expressions using the OWL knowledge graph that distinguish the positive examples from the negative examples, thereby generating candidate semantic explanations of neuron activation. The resulting neuron-feature associations are evaluated using held-out data and statistical validation. Associations that satisfy the predefined validation criteria are retained as validated neuron-feature pairs. These pairs form the fundamental observations used in the subsequent analysis. Finally, the validated neuron-feature pairs are aggregated along several experimental dimensions, including CNN depth, spatial patch location, dataset, and image-feature family.
These aggregated representations are used to examine where context-sensitive semantic representations emerge within CNNs. In particular, the analysis examines the network depths at which such representations arise, the spatial regions and visual feature families with which they are most consistently associated, and how their occurrence relates to the context-bias levels observed in the preceding patch-based study.
The prior patch-based study categorized the eight datasets according to context-bias severity observed from patch-level classification performance. Since the subsequent hidden-neuron and Concept Induction analysis is considerably more computationally intensive, we selected a representative subset of five datasets rather than repeating the analysis across all eight datasets. The selection is designed to span the observed bias spectrum while providing greater representation of datasets exhibiting strong contextual dependence, for which identifying the internal representations associated with image context is particularly relevant. Accordingly, Cassava, WeedCrop, and OpenSprayer from the high-bias group, Sugarcane from the moderate-bias group, and Oxford-102 from the low-bias group have been chosen. This subset therefore enables comparison of hidden-layer semantic representations across datasets exhibiting substantially different levels of previously observed context dependence.
Hidden-Layer Selection and Activation Extraction
To examine how context-sensitive representations evolve across network depth, three hidden layers were selected from each CNN architecture to represent mid, mid-deep, and deep stages of feature processing. The layers are selected from progressively deeper major processing blocks within each architecture, providing representative points along the feature hierarchy while avoiding adjacent layers that capture highly similar stages of computation. Because ResNet50V2, DenseNet121, InceptionV3, and VGG16 differ substantially in their connectivity patterns and architectural organization, we do not assume exact layer-wise equivalence across the four networks. Instead, the selected layers provide architecture-specific reference points corresponding broadly to intermediate, later-intermediate, and deep feature-processing stages. The selected layers are summarized in Table 3.
| Architecture | Mid | Mid-deep | Deep |
| ResNet50V2 | conv3_block4_out | conv4_block6_out | conv5_block3_out |
| DenseNet121 | conv3_block12_concat | conv4_block24_concat | relu |
| InceptionV3 | mixed5 | mixed7 | mixed10 |
| VGG16 | block3_conv3 | block4_conv3 | block5_conv3 |
Table 3. Hidden layers selected to represent mid, mid-deep, and deep stages of each CNN architecture
Earlier and intermediate convolutional layers primarily encode basic visual features, such as edges, colors, and textures, into local patterns, while progressively deeper layers combine information from larger image regions to represent more complex and abstract features [71]. Analyzing features at these different layer depths helps us understand how context-sensitive concepts gradually develop and change as information moves through the CNN.
For each dataset and spatial patch location, the corresponding patch images are forwarded through the trained CNN, and activation values are extracted from the selected hidden layers. Since the analyzed layers are convolutional or feature-map-producing layers, each layer contains a distinct set of feature channels. Consequently, neurons analyzed at the mid, mid-deep, and deep stages are not treated as the same neurons tracked across depth; rather, each selected layer contributes its own set of candidate neurons for subsequent analysis. For each selected layer, an intermediate model was constructed using the trained CNN’s input and the selected hidden layer as its output. Since the analyzed layers produce multi-channel feature maps, global average pooling (GAP) was applied to reduce each channel’s activation map to a single value by averaging over its spatial positions. Each feature channel is treated as a neuron in the subsequent analysis. Repeating this procedure over all patch images produced an image-by-neuron activation matrix for each combination of dataset, patch location, architecture, and selected hidden layer.
| Architecture | Depth | Layer | Initial neurons/ channels | After variance filter | After correlation ranking | Final selected |
| ResNet50V2 | Mid | conv3_block4_out | 512 | 384 | 100 | 50 |
| ResNet50V2 | Mid-deep | conv4_block6_out | 1,024 | 768 | 100 | 50 |
| ResNet50V2 | Deep | conv5_block3_out | 2,048 | 1,536 | 100 | 50 |
| DenseNet121 | Mid | conv3_block12_concat | 512 | 384 | 100 | 50 |
| DenseNet121 | Mid-deep | conv4_block24_concat | 1,024 | 768 | 100 | 50 |
| DenseNet121 | Deep | relu | 1,024 | 768 | 100 | 50 |
| InceptionV3 | Mid | mixed5 | 768 | 576 | 100 | 50 |
| InceptionV3 | Mid-deep | mixed7 | 768 | 576 | 100 | 50 |
| InceptionV3 | Deep | mixed10 | 2,048 | 1,536 | 100 | 50 |
| VGG16 | Mid | block3_conv3 | 256 | 192 | 100 | 50 |
| VGG16 | Mid-deep | block4_conv3 | 512 | 384 | 100 | 50 |
| VGG16 | Deep | block5_conv3 | 512 | 384 | 100 | 50 |
Table 4. Reduction in candidate neurons across the three-stage neuron-selection procedure for the analyzed CNN layers.
The extracted activation values are then used as the basis for neuron filtering and the construction of positive and negative example sets for Concept Induction. To avoid information leakage during concept evaluation, the constructed patch data are divided into analysis and held-out subsets, with the former used for neuron selection and concept generation and the latter for validation.
Neuron Selection
To restrict Concept Induction to neurons exhibiting informative and decision-relevant activation behavior, a three-stage neuron-selection procedure is applied independently to each analyzed hidden layer. Importantly, neuron selection is performed using only the 70% analysis split; the 30% holdout split was excluded from all filtering and concept-generation steps and reserved exclusively for subsequent validation. For each neuron, activation values are collected across the analysis images. The first filtering stage removed neurons exhibiting comparatively little variation in their responses. Neuron-wise activation variance was calculated, and neurons falling within the bottom 25th percentile of the variance distribution for the corresponding layer were discarded. Because this criterion is quantile-based rather than based on a fixed variance threshold, the cutoff adapts to the activation characteristics of each layer. Among the remaining neurons, a second filtering stage assessed association with the CNN’s prediction confidence. For each neuron , the Pearson correlation coefficient between its activation values and the corresponding model-confidence scores was calculated,
where denotes the activation of neuron for image , and denotes the CNN prediction confidence for that image. Neurons are ranked according to the absolute correlation magnitude, , and the 100 highest-ranked neurons are retained. Using the absolute value allows both positively and negatively associated activation patterns to be considered potentially relevant to model confidence. Finally, the retained 100 neurons are re-ranked according to their activation variance, and the 50 highest-variance neurons are selected for Concept Induction. Thus, each analyzed layer contributes at most 50 candidate neurons for each evaluated feature family.
UDAT Feature Extraction and Discretization
Each contextual patch is described with UDAT, using the 338 raw-image descriptors organized into 13 feature families (Table 1).
Not every family is tested against every hidden layer. Feature families are assigned to individual network layers based on an LLM-guided assessment. The assessment was conducted using Claude Sonnet (Anthropic, 2026). For each architecture, the model was prompted with the layer names, spatial resolutions, and channel depths alongside descriptions of the 13 UDAT feature families, and asked to classify each layer-feature-family combination as test, optional/borderline, or skip based on the type of visual representations expected at that network depth. Architecture-specific properties were also considered, including residual skip connections in ResNet50V2, dense feature concatenation in DenseNet121, and multi-scale parallel processing in InceptionV3. The full prompt template is provided in Appendix § 8. of their relevance to the type of visual representations expected at different network depths. of their relevance to the type of visual representations expected at different network depths. The assessment categorized each layer-feature-family combination as test, optional/borderline, or skip, and only the test combinations are run through Concept Induction. This gives 84 layer-feature-family combinations per dataset-patch configuration, the same for all 25 configurations. Of the 13 families, 11 are evaluated. Intensity Statistics and Color Histogram are marked test only for early layers, which are not analyzed in this study. Zernike moments are run on the mid-deep and deep layers, but CELOE returns only the generic concept Thing for every neuron (7,759 of 7,759 runs that return a concept). These concepts fall under the exclusion rule described in the validation subsection and are not validated, so Zernike moments produce no testable concepts and do not appear in the results.
The raw UDAT outputs are continuous numerical values and therefore cannot be used directly as symbolic concepts in the subsequent Description Logic representation. Each descriptor is therefore discretized into a finite set of ordered value intervals, and each interval is represented as a symbolic bin. A patch is then described by the interval its measurement falls into, not by the exact value. Interval boundaries are set from the minimum and maximum of each descriptor over all patch images of the corresponding dataset-patch configuration, i.e., analysis and held-out images together.
For descriptor and patch image , the discretization is
where is the numerical value of the descriptor for , and is the -th interval of , the one containing that value. For example, a low edge-statistics value, falls into a lower-order bin, and a comparatively high value for the same descriptor falls into a higher-order bin. The discretized representation provides two advantages for the neuro-symbolic analysis. First, it transforms continuous visual measurements into categorical concepts that can be represented explicitly in OWL. Second, it allows Concept Induction to search for human-interpretable combinations of visual characteristics rather than operating directly over raw numerical values.
Knowledge Graph Construction
The discretized UDAT descriptors are encoded as an OWL-based knowledge graph to provide the symbolic background knowledge required for Concept Induction. Each contextual patch image is represented as an individual of the class ‘Image’. For each UDAT descriptor, an object property connects the image individual to a bin individual representing its discretized feature value. The bin individual is subsequently typed using an OWL class corresponding to the numerical interval into which the original descriptor value fell.
For example, in Figure 2, for the Gabor feature family, properties such as hasGaborFilters_Bin0, hasGaborFilters_Bin1, ..., hasGaborFilters_Bin6 connect an image to the discretized values of the corresponding Gabor descriptors. The interval-specific classes are organized under broader descriptor classes using rdfs:subClassOf, thereby preserving both the feature identity and its discretized numerical range. Formally, for image and descriptor with object property and interval classes , the graph contains the assertions
Figure 2. Ontology pattern for one patch image and one UDAT sub-feature (Gabor filters, sub-feature Bin2). The patch individual is linked by hasGaborFilters Bin2 to a value individual (value 0.1225), which is typed with the interval class for [0.105, 0.150]; the five equal-width interval classes are subclasses of the sub-feature class GaborFilters_Bin2. The same pattern is repeated for each of the seven Gabor sub-features.
where is the value individual of for , and is the index of the interval containing the numerical value , as defined in the discretization step. Individual OWL representations generated for the patch images are subsequently combined into a single ontology for the corresponding dataset, patch location, and UDAT feature family. This combined knowledge graph is then supplied as the background knowledge in Concept Induction. The ontology therefore encodes each patch as a structured set of symbolic visual attributes. The discretized feature intervals are organized hierarchically, allowing Concept Induction to reason over both specific value ranges and broader descriptor-level abstractions.
Concept Induction
For each selected neuron–feature-family combination, Concept Induction is run with the CELOE algorithm in DL-Learner, using the OWL knowledge graph built from the corresponding UDAT descriptors as background knowledge. Only the 70% analysis split is used; the 30% holdout split remains unseen until the validation stage. For each neuron , positive and negative example sets are derived from its activations over the analysis images. Let denote the activation of neuron for image , and let be the maximum of over the analysis images. Images with activation of at least 80% of form the positive set, and images with nonzero activation of at most 20% of form the negative set:
Images with zero activation, and images with activation between 20% and 80% of the maximum are discarded. The two example sets and the OWL background knowledge are passed to DL-Learner as a posNegStandard learning problem with a closed-world reasoner. CELOE searches the space of Description Logic class expressions for those that cover the positive examples while excluding the negative ones. Each run is limited to 60 s. For each neuron-feature-family combination, only the top-ranked class expression, ranked by predictive accuracy, is retained and carried forward to validation. A run for which CELOE returns no solution yields no candidate concept.
Concept and Neuron-Feature Pair Validation
The concepts generated from the 70% analysis split are evaluated using the independent 30% holdout split. Concepts containing only generic classes, such as Thing or Image, and concepts that cannot be parsed into UDAT feature-bin constraints are excluded from statistical validation.
For each processable neuron-feature pair, holdout images are divided into concept-positive and concept-negative groups based on whether they satisfy the induced concept. An image satisfies a concept when its UDAT feature values fall within the bin intervals specified by the corresponding Description Logic expression. For negated constraints, the feature values must fall outside the specified intervals. Images satisfying the concept are assigned to the concept-positive group, while the remaining images form the concept-negative group.
Statistical validation is performed only when both groups contain at least three images. A one-sided Mann-Whitney U test is applied to examine whether neuron activations are significantly higher for concept-positive images than for concept-negative images. The alternative hypothesis is defined as
where and denote the activation distributions of neuron for concept-positive and concept-negative holdout images, respectively. Statistical significance is assessed at .
In addition to statistical significance, Cliff’s delta () is used to measure the effect size of the activation difference between the two groups. Effect sizes are categorized as negligible (), small (), medium (), and large (). A neuron–feature pair is considered validated only if it satisfies both criteria: and . This ensures that retained associations exhibit both statistically significant activation differences and at least medium effect-size magnitude.
The validation rate (VR) is calculated as the proportion of statistically tested neuron-feature pairs that satisfy the final validation criteria. For each CNN depth and dataset-patch configuration , it is defined as
where denotes the number of validated neuron-feature pairs and denotes the number of pairs for which the Mann-Whitney U test is completed. Each configuration represents one dataset-patch combination, giving 25 configurations in total. The mean validation rate at depth is calculated as
This aggregation gives equal weight to each dataset-patch configuration rather than pooling all neuron-feature pairs into a single rate.
For spatial analysis, validation results are aggregated across the analyzed architectures and layer depths for each dataset and patch position . The corresponding patch-level validation rate is
The same validated-to-tested ratio is used for feature-family analysis, with neuron-feature pairs grouped according to their corresponding UDAT descriptor families. These measures provide the basis for the subsequent comparisons across CNN depths, spatial patch locations, and visual feature families.
Results and Discussion
In this section, we first summarize the overall outcome of the neuron-feature validation pipeline. Across the 5 analyzed datasets and 5 spatial patch positions, the combined analysis comprised 25 dataset-patch configurations. For each configuration, 3 depths 4 architectures = 12 hidden layers are examined, with 50 selected neurons evaluated for each applicable layer-feature-family combination. Because the availability of UDAT feature families varied across layers, 84 layer-feature-family combinations are represented per configuration, yielding 84 50 = 4,200 neuron-feature evaluation cases. Across all 25 configurations, this resulted in a total of 25 4,200 = 105,000 evaluation cases. Figure 3 summarizes the progressive reduction of neuron-feature evaluation cases through concept induction, constraint parsing, held-out statistical testing, and final validation. A total of 48,013 neuron-feature pairs satisfied the requirements for statistical testing on the held-out data. Among them, 13,164 pairs met the final validation criteria, corresponding to an overall validation rate of approximately 27.4%. To examine whether this reduction is consistent across experimental settings, Table 5 presents the validation progression separately for all 25 dataset-patch configurations. A progressive decrease can be observed in every configuration, indicating that the overall filtering and validation behavior is broadly distributed rather than dominated by a single dataset or spatial region.

Figure 3. Progressive reduction of neuron-feature evaluation cases. Percentages indicate the proportion retained relative to the initial 105,000 evaluation cases.
| Dataset | Patch | All Cases | Concept Found | Constraints Parsed | MW Test Completed | Significant | Validated |
| Cassava | BL | 4200 | 3219 | 2356 | 2260 | 1200 | 362 |
| Cassava | BR | 4200 | 3184 | 2318 | 2223 | 1213 | 387 |
| Cassava | C | 4200 | 3155 | 2241 | 2170 | 1131 | 433 |
| Cassava | TL | 4200 | 3227 | 2351 | 2296 | 1236 | 487 |
| Cassava | TR | 4200 | 3242 | 2350 | 2302 | 1266 | 495 |
| OpenSprayer | BL | 4200 | 3005 | 2367 | 2003 | 791 | 468 |
| OpenSprayer | BR | 4200 | 3011 | 2282 | 1965 | 762 | 405 |
| OpenSprayer | C | 4200 | 3083 | 2422 | 2036 | 796 | 453 |
| OpenSprayer | TL | 4200 | 3016 | 2286 | 1937 | 701 | 385 |
| OpenSprayer | TR | 4200 | 3001 | 2367 | 2048 | 838 | 484 |
| Oxford | BL | 4200 | 3081 | 2534 | 2248 | 1071 | 622 |
| Oxford | BR | 4200 | 3039 | 2463 | 2216 | 1043 | 667 |
| Oxford | C | 4200 | 3243 | 2665 | 2259 | 824 | 423 |
| Oxford | TL | 4200 | 3059 | 2498 | 2211 | 1262 | 745 |
| Oxford | TR | 4200 | 3053 | 2449 | 2181 | 1074 | 650 |
| Sugarcane | BL | 4200 | 2985 | 2248 | 2016 | 1132 | 895 |
| Sugarcane | BR | 4200 | 2946 | 2194 | 1961 | 1074 | 783 |
| Sugarcane | C | 4200 | 3132 | 2438 | 2187 | 1045 | 650 |
| Sugarcane | TL | 4200 | 3016 | 2245 | 1916 | 1133 | 852 |
| Sugarcane | TR | 4200 | 2865 | 2178 | 1926 | 1098 | 877 |
| WeedCrop | BL | 4200 | 3315 | 2973 | 1169 | 244 | 244 |
| WeedCrop | BR | 4200 | 3054 | 2707 | 993 | 184 | 184 |
| WeedCrop | C | 4200 | 3020 | 2651 | 1303 | 562 | 562 |
| WeedCrop | TL | 4200 | 2960 | 2628 | 944 | 223 | 223 |
| WeedCrop | TR | 4200 | 3116 | 2730 | 1243 | 429 | 428 |
Table 5. Dataset-patch-wise progression of neuron-feature evaluation cases. Patch locations are denoted as BL (Bottom-Left), BR (Bottom-Right), C (Center), TL (Top-Left), and TR (Top-Right).
The following sections analyze these validated neuron–feature pairs across CNN depth, spatial patch location, feature family, and dataset-level context-bias characteristics to address the research questions.
RQ1 - Emergence Across CNN Depth
As the analysis is performed on the predefined patch regions used to assess contextual dependence, and the induced concepts are derived from visual properties of those same patches, the validated neuron-feature pairs are interpreted as context-sensitive internal representations. Table 6 summarizes the validation rates across different layer depths, including the mean, standard deviation, minimum, and maximum values.
| Layer depth | Configurations | Mean validation rate (%) | SD | Min (%) | Max (%) |
| Mid | 25 | 40.21 | 10.38 | 24.49 | 61.83 |
| Mid-deep | 25 | 26.91 | 12.89 | 14.43 | 54.02 |
| Deep | 25 | 11.99 | 5.65 | 4.41 | 29.07 |
Table 6. Summary of neuron-feature validation rates across the three analyzed CNN depth levels.
Across the analyzed configurations, the Mid layers exhibit the highest mean validation rate at 40.21%, followed by the Mid-deep layers at 26.91%, while the Deep layers exhibit a substantially lower mean rate of 11.99%. The Mid layers have produced validation rates ranging from approximately 24.49% to 61.83%, with a standard deviation of 10.38. Mid-deep layers range from 14.43% to 54.02%, with a standard deviation of 12.89, whereas Deep layers range from 4.41% to 29.07%, with a standard deviation of 5.65.
The layer-depth heatmap in Figure 4 further shows how this pattern varies across individual datasets and patch locations. In most configurations, the Mid layer has produced the highest validation rate, while the Deep layer has produced the lowest. This tendency is especially pronounced for Sugarcane, where Mid-layer rates reach 61.8% for the bottom-left patch and remain above 46% for all patch locations. In contrast, corresponding Deep-layer rates remain substantially lower. WeedCrop provides another notable case: the center patch reaches 52.4% at the Mid depth and decreases to 42.3% at Mid-deep and 29.1% at Deep.

Figure 4. Validation rates of context-sensitive neuron-feature pairs across different layers for each dataset-patch configuration.
At the same time, the heatmap indicates that the magnitude of the decrease is dataset-dependent. Cassava and OpenSprayer display comparatively low Deep-layer rates across all spatial regions, whereas Sugarcane and WeedCrop retain somewhat stronger validation at deeper levels in selected patches. Nevertheless, the overall ordering remains strongly weighted toward the earlier analyzed depth. These results indicate that semantically interpretable associations with contextual patch properties are most strongly represented at the Mid depth, become less prevalent at Mid-deep layers, and are least frequently validated at the Deep depth. This pattern suggests that contextual visual characteristics are encoded most prominently during intermediate stages of hierarchical feature formation, while deeper representations become increasingly selective and task-oriented. Thus, RQ1 is answered by the consistent concentration of validated context-sensitive neuron representations in Mid-level CNN layers, followed by a progressive reduction through Mid-deep and Deep layers.
Furthermore, Figure 5 reveals substantial variation across datasets rather than a single universally dominant spatial region. For Cassava, validation rates remain relatively similar across the top-left, top-right, and center patches, ranging from 20.0% to 21.5%, while the bottom-left and bottom-right regions show lower rates of 16.0% and 17.4%, respectively. OpenSprayer exhibits a comparatively uniform distribution, with rates ranging from 19.9% to 23.6% across the five patch locations.

Figure 5. Dataset-wise validation rates of context-sensitive neuron-feature representations across the five predefined spatial patch locations.
Oxford exhibits stronger spatial variation. The top-left patch reaches 33.6%, followed by the bottom-right at 30.1% and top-right at 29.8%, whereas the center patch decreases to 18.7%. Sugarcane has produced the highest overall patch-wise validation levels, with the top-right, top-left, and bottom-left regions reaching approximately 45.5%, 44.5%, and 44.4%, respectively. In contrast, WeedCrop demonstrates a different spatial pattern, where the center patch reaches 43.1%, substantially exceeding the top-left, bottom-left, and bottom-right regions. To examine whether any patch location is consistently dominant across datasets, the validation rates across all 15 dataset-depth observations associated with each patch position are also summarized in Table 7. The mean patch-wise validation rates are approximately 29.53% for top-right, 27.39% for top-left, 25.61% for center, 25.25% for bottom-left, and 24.06% for bottom-right. The relatively close overall means, combined with the dataset-specific patterns visible in Figure 5, indicate that no single spatial patch location consistently dominates across all datasets. Although the top-right and top-left regions exhibit modestly higher overall mean validation rates, the strongest spatial region varies by dataset. Sugarcane shows high validation across several peripheral patches, WeedCrop is dominated by the center region, and Oxford favors the top-left region. This variation suggests that context-sensitive neuron representations are not tied to a fixed image coordinate across datasets. Instead, the spatial distribution of semantically validated representations appears to depend on the visual characteristics and contextual structure associated with each dataset.
| Patch position | Observations | Mean validation rate (%) | SD | Min (%) | Max (%) |
| Top-left | 15 | 27.39 | 16.25 | 8.87 | 57.32 |
| Top-right | 15 | 29.53 | 16.08 | 6.77 | 56.20 |
| Center | 15 | 25.61 | 14.85 | 6.96 | 52.43 |
| Bottom-left | 15 | 25.25 | 15.97 | 4.41 | 61.83 |
| Bottom-right | 15 | 24.06 | 14.77 | 6.23 | 50.37 |
Table 7. Patch-wise summary of neuron-feature validation rates across datasets and analyzed layer depths.
Having established that validated context-sensitive representations are concentrated primarily at intermediate depths but vary spatially across datasets, the next analysis examines which visual feature families most consistently characterize these representations.
RQ2 - Which Feature Families Dominate?
RQ2 examines which visual feature families most consistently dominates in the hidden-neuron representations associated with contextual image patches. For this analysis, the statistically tested neuron-feature pairs are grouped by feature family across the analyzed datasets, patch locations, architectures, and layer depths. Within each feature family, the validation rate is calculated.
The overall analysis reveals a clear dominance of several edge- and texture-related descriptors. As shown in Figure 6, Edge features achieve the highest validation rate at 64.5%, with 1,402 validated pairs among 2,174 statistically tested pairs. Gabor features follow closely at 60.88% with 1,346 of 2,211 tested pairs validated, while Tamura features reach 48.52%, corresponding to 2,881 validated pairs among 5,938 tested cases. These three feature families therefore constitute the strongest overall associations with the validated hidden-neuron representations.

Figure 6. Overall validation outcomes for the analyzed UDAT feature families.
A second group of feature families exhibits more moderate validation rates. Fractal features reach 36.47%, followed by Chebyshev at 30.42% and Radon at 25.13%. In contrast, CombFour, Object, and Multiscale features show substantially lower validation rates of 12.00%, 8.51%, and 4.52%, respectively. Haralick features show a validation rate of 36.23%; however, this result is based on only 138 statistically tested pairs from one dataset and one patch location and is therefore can be interpreted separately from the feature families represented across all five datasets and five patch positions. The prominence of Edge, Gabor, and Tamura features indicates that the validated context-sensitive neuron representations are most frequently associated with visual characteristics related to edge organization, directional or frequency-sensitive texture, and perceptual texture structure. Importantly, these results are interpreted at the level of the neuron-feature pair rather than the neuron alone. An individual neuron may therefore be associated with more than one validated feature family. To determine whether the aggregate ranking is driven by only a subset of experimental conditions, the same validation-rate calculation is repeated after grouping the tested neuron-feature pairs separately by dataset and by patch position. The resulting comparisons are shown in Figure 7 and Figure 8.

Figure 7. Feature-family validation rates grouped by dataset across the analyzed patch locations, architectures, and layer depths.

Figure 8. Feature-family validation rates grouped by spatial patch position across the analyzed datasets, architectures, and layer depths.
Across datasets, Edge and Gabor features remain among the strongest families in nearly every case. Edge validation rates range from 60.13% for Cassava to 68.43% for OpenSprayer, while Gabor ranges from 44.10% for Oxford to 87.60% for Sugarcane. Tamura also exhibits relatively high validation in Cassava (44.12%), OpenSprayer (55.16%), Oxford (46.88%), and Sugarcane (59.02%), although its rate decreases to 24.68% for WeedCrop.
The remaining feature families show greater dataset dependence. Fractal features are relatively weak for Cassava (17.09%) and OpenSprayer (19.24%) but become substantially stronger for Oxford (39.29%), Sugarcane (59.51%), and WeedCrop (52.89%). A similar pattern appears for Chebyshev and Radon, both of which reach their strongest validation in Sugarcane at 58.80% and 61.94%, respectively. In contrast, CombFour, Object, and Multiscale remain comparatively low across most datasets. These results indicate that Edge and Gabor provide the most stable high-validation pattern across the five datasets, whereas several secondary feature families exhibit considerably greater dataset-specific variation. The patch-wise analysis in Figure 8 reveals a similarly consistent pattern. Edge features achieve the highest validation rate in four of the five spatial regions, reaching 67.58% in the top-left, 68.44% in the top-right, 74.55% in the center, and 63.84% in the bottom-left patches. In the bottom-right region, Gabor becomes the strongest feature family at 57.63%, compared with 46.57% for Edge. Gabor also remains consistently strong across the other patch positions, ranging from 55.51% to 67.23%, while Tamura maintains rates between approximately 45.29% and 52.63%. Fractal, Chebyshev, and Radon show moderate but more variable behavior across regions, whereas CombFour, Object, and Multiscale remain comparatively weak.
Taken together, the dataset-wise and patch-wise analyses show that the high overall validation rates of Edge and Gabor are not attributable to a single dataset or spatial location. Instead, both families remain consistently prominent across heterogeneous experimental settings, with Tamura pro-
viding an additional but somewhat more variable texture-related signal. Furthermore, to assess how feature-family associations vary with CNN depth, validation rates are computed for each layer-feature combination as validated pairs divided by statistically tested pairs and reported in Figure 9. Combinations with no statistically tested pairs are reported as NA.

Figure 9. Validation rates of neuron-feature pairs across feature families and CNN layer depths.
In the heatmap we can see that several of the strongest feature families are concentrated at the Mid depth. Edge and Gabor are represented only at the Mid depth in the tested data and achieve validation rates of 64.5% and 60.9%, respectively. Tamura is represented across all three depths and exhibits a progressive decline from 55.2% at Mid to 46.6% at Mid-deep and 33.6% at Deep. A similar depth-related decrease is observed for Chebyshev, from 54.4% at Mid to 33.8% at Mid-deep and 18.0% at Deep. Fractal and Haralick are represented at Mid and Mid-deep depths and also show higher validation at Mid, decreasing from 41.5% to 31.7% for Fractal and from 39.7% to 31.7% for Haralick. Radon shows a smaller reduction from 27.0% to 23.2%, while CombFour decreases from 17.5% at Mid to 11.8% at Mid-deep and 7.4% at Deep. Multiscale remains consistently low across all depths, ranging from 3.3% to 6.0%, whereas Object features are represented only at the Deep depth and achieve an 8.5% validation rate.
Overall, the rate-based comparison reinforces the depth pattern identified in RQ1. For feature families represented across multiple depths, validation is generally strongest at the Mid level and decreases toward deeper layers. This suggests that the strongest semantic associations with contextual visual properties are concentrated primarily in intermediate CNN representations, although the degree of decline varies by feature family. Having identified both where context-sensitive representations emerge and which visual feature families characterize them, the next analysis examines how these semantic validation patterns relate to the degree of context bias observed in the prior patch-based study.
RQ3 - Relationship Between Context Bias and Semantic Concept Emergence
RQ3 examines how the emergence of validated semantic concepts relates to the degree of context bias observed across the analyzed datasets. Table 8 represents the neuron-feature validation statistics on Dataset-level bias and above chance patch accuracy. The above-chance patch accuracy represents the model’s behavioral dependence on contextual image regions, whereas the dataset-level validation rate represents the proportion of statistically tested neuron-feature associations that can be semantically characterized and satisfy the final validation criterion. Sugarcane exhibits the highest semantic validation rate at 40.55%, despite having only moderate context bias. In contrast, Cassava and OpenSprayer exhibit high context bias but considerably lower validation rates of 19.23% and 21.97%, respectively. WeedCrop also exhibits high context bias, yet its validation rate reaches 29.03%, remaining below that of Sugarcane. Oxford provides another contrasting case, achieving a validation rate of 27.95% despite showing only minimal above-chance patch performance.
| Dataset | Bias category | Mean patch accuracy | Random chance | Above-chance patch accuracy | Tested | Validated | Validation rate |
| Cassava | High | 33.62% | 20.00% | 13.62% | 11,251 | 2,164 | 19.23% |
| OpenSprayer | High | 70.18% | 50.00% | 20.18% | 9,989 | 2,195 | 21.97% |
| Oxford | Low | 1.70% | 0.98% | 0.72% | 11,115 | 3,107 | 27.95% |
| Sugarcane | Moderate | 16.92% | 9.1% | 7.83% | 10,006 | 4,057 | 40.55% |
| WeedCrop | High | 91.06% | 50.00% | 41.06% | 5,652 | 1,641 | 29.03% |
Table 8. Dataset-level bias and neuron-feature validation statistics.
The dataset rankings differ between context-bias severity and semantic concept emergence. Stronger dependence on contextual regions does not necessarily lead to more semantically validated neuron-feature associations. Similarly, datasets with moderate or low context bias can still show relatively high semantic concept emergence. Overall, RQ3 shows that stronger context bias does not necessarily correspond to greater semantic concept emergence in the analyzed hidden representations.
These results indicate that behavioral context dependence and semantic concept emergence capture distinct aspects of CNN behavior. High patch-based classification performance reflects the extent to which a model can exploit information contained within restricted image regions, whereas the validation rate reflects the extent to which hidden-neuron activations can be associated with the visual feature concepts represented in the constructed knowledge graph. Consequently, a dataset may exhibit strong contextual dependence without producing a proportionally larger number of semantically validated neuron–feature associations.
Conclusion
Previous studies have demonstrated that CNNs can achieve above-chance classification accuracy using image patches containing little or no task-relevant content, revealing potential dependence on contextual information. Building on these findings, this study investigates how such information is represented within CNN hidden layers. We employ a neuro-symbolic approach that combines hidden-neuron activation analysis, UDAT image descriptors, OWL knowledge graphs, and Description Logic Concept Induction to identify interpretable associations between neuron activations and visual characteristics of contextual image patches. Across five datasets, five patch locations, four CNN architectures, and three network depths, the analysis evaluates 105,000 neuron-feature-family combinations, of which 77,027 produce candidate concepts, 48,013 undergo statistical testing, and 13,164 satisfy the validation criteria.
The findings reveal several patterns in the internal representations of CNNs. Mid layers exhibit the highest mean validation rate (40.2%), followed by mid-deep (26.9%) and deep layers (12.0%), indicating that the evaluated image descriptors characterize mid-layer activations more consistently. Spatial analysis reveals no universally dominant patch location, as the regions producing the highest validation rates vary across datasets. Among the evaluated feature families, Edge, Gabor, and Tamura achieve the strongest and most consistent validation results across datasets and patch locations, suggesting that edge structure and texture-related characteristics are particularly prominent in the representations associated with contextual image patches. However, the degree of concept validation does not directly correspond to the severity of context bias observed in the previous patch-based study. Sugarcane, which exhibits moderate context bias, achieves the highest overall validation rate (40.5%), whereas highly biased Cassava achieves the lowest (19.2%).
Together, these findings extend patch-based context-bias analysis from observing classification behavior to examining the visual characteristics associated with hidden-neuron activations. They also suggest that strong contextual dependence does not necessarily imply a greater proportion of statistically validated neuron–feature associations.
Several limitations affect how far these findings reach. The feature families were assigned to layers by an LLM-guided assessment rather than crossed with all layers. Edge and Gabor were evaluated only at mid layers, so their high validation rates cannot be separated from the depth effect, and only Tamura, among the strongest families, was tested at all three depths. Zernike moments yielded no testable concepts, and Haralick features were tested on a single dataset and patch location. While the proposed analysis provides interpretable evidence of internal representations, these associations alone do not establish which features causally influence classification decisions. Future work can investigate this relationship through controlled feature interventions and examine whether modifying or removing identified contextual characteristics changes model predictions, thereby supporting a more direct assessment of their contribution to context bias.
Code and Data Availability
The code and data associated with this study will be made publicly available upon publication at https://github.com/Samatha1994/XAI_Understanding_Bias_Project/.
LLM-Guided Feature-Layer Assignment
The feature-layer assessment described in § 4.4 was conducted using Claude Sonnet (Anthropic, 2026). For each architecture, the model was provided with the architecture name, the selected layer names, and the 13 UDAT feature family names, and queried as follows:
For the following layers of [ARCHITECTURE]: [LAYER NAMES] and the following feature families: Edge Statistics, Combined First Four Moments, Chebyshev-Fourier Transform, Multiscale Histogram, Tamura Texture, Haralick Texture, Gabor Filters, Zernike Moments, Object Statistics, Radon Transform, Color Histogram, Fractal Features, and Intensity Statistics — which features should be tested for each layer, and which should be skipped?
This prompt was issued separately for each of the four architectures, with layer names substituted accordingly. The model returned a classification of each layer–feature-family combination as test, optional/borderline, or skip, along with theoretical justification for each decision based on known properties of convolutional feature hierarchies.
References
- [1]Samantha Ereshi Akkamahadevi, Abhilekha Dalal, and Pascal Hitzler. Automating cnn neuron interpretation using concept induction. In ISWC (Posters/Demos/Industry), 2024.
- [2]Samantha Ereshi Akkamahadevi, Abhilekha Dalal, and Pascal Hitzler. Evaluation of concept induction in explainable AI using multiple datasets. In Proceedings of the AAAI Symposium Series, volume 5, 2025.DOI
- [3]Ananthu Aniraj, Cassio F. Dantas, Dino Ienco, and Diego Marcos. Masking strategies for background bias removal in computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 4454–4463, 2023.arxiv.org/abs/2308.12127
- [4]Gavin Armstrong. Open sprayer images, 2023. Accessed: June 2025.
- [5]David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6541–6549, 2017.arxiv.org/abs/1704.05796
- [6]Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pages 456–473, 2018.
- [7]Lorenz Bü hmann, Jens Lehmann, and Patrick Westphal. DL-Learner—a framework for inductive learning on the Semantic Web. Journal of Web Semantics, 39:15–24, 2016.
- [8]Abhilekha Dalal. Understanding Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning. Kansas State University, 2024.
- [9]Abhilekha Dalal, Michael Okonoda, Eder Martinez, and Lior Shamir. Evaluating contextual bias in cnn image classification: Evidence from agricultural benchmark datasets. arXiv preprint arXiv:2609.14654, 2026.arxiv.org/abs/2609.14654
- [10]Abhilekha Dalal, Rushruk h Rayan, Adrita Barua, Samatha Ereshi Akkamahadevi, Avishek Das, Cara Widmer, Eugene Y Vasserman, Md Kamruzzaman Sarker, and Pascal Hitzler. Toward a neurosymbolic understanding of hidden neuron activations. Neurosymbolic Artificial Intelligence, 2:29498732261419302, 2026.DOI
- [11]Abhilekha Dalal, Rushrukh Rayan, Adrita Barua, Eugene Y Vasserman, Md Kamruzzaman Sarker, and Pascal Hitzler. On the value of labeled data and symbolic methods for hidden neuron activation analysis. In International Conference on Neural-Symbolic Learning and Reasoning, pages 109–131. Springer, 2024.arxiv.org/abs/2404.13567
- [12]Manuel de Sousa Ribeiro and João Leite. Aligning artificial neural networks and ontologies towards explainable AI. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
- [13]Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3:610–619, 2021.DOI
- [14]Sanchari Dhar and Lior Shamir. Evaluation of the benchmark datasets for testing the efficacy of deep convolutional neural networks. Visual Informatics, 2021.DOI
- [15]ErnestMwebaze, Jesse Mostipak, Joyce, Julia Elliott, and Sohier Dane. Cassava leaf disease classification. https://kaggle.com/competitions/cassava-leaf-disease-classification, 2020. Kaggle.
- [16]Sai Teja Erukude, Akhil Joshi, and Lior Shamir. Identifying bias in deep neural networks using image transforms. Computers, 13(12):341, 2024.DOI
- [17]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.DOI
- [18]Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.arxiv.org/abs/1811.12231
- [19]Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.arxiv.org/abs/1902.03129
- [20]Robert M. Haralick, K. Shanmugam, and Its’hak Dinstein. Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics, SMC-3(6):610–621, 1973.DOI
- [21]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.arxiv.org/abs/1512.03385
- [22]Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2201.11114
- [23]Pascal Hitzler. A review of the semantic web field. Commun. ACM, 64(2):76–83, 2021.DOI
- [24]Pascal Hitzler, Federico Bianchi, Monireh Ebrahimi, and Md Kamruzzaman Sarker. Neural-symbolic integration and the Semantic Web. Semantic Web, 11(1):3–11, 2020.DOI
- [25]Pascal Hitzler, Markus Krötzsch, and Sebastian Rudolph. Foundations of Semantic Web Technologies. Chapman and Hall/CRC Press, 2010.
- [26]Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.arxiv.org/abs/1905.02175
- [27]Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 2668–2677, 2018.arxiv.org/abs/1711.11279
- [28]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097–1105, 2012.
- [29]Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10:1096, 2019.
- [30]Phuong Quynh Le, Jörg Schlötterer, and Christin Seifert. An xai-based analysis of shortcut learning in neural networks. In World Conference on Explainable Artificial Intelligence, pages 424–445. Springer, 2025.arxiv.org/abs/2504.15664
- [31]Jae Hee Lee, Sergio Lanza, and Stefan Wermter. From neural activations to concepts: A survey on explaining concepts in neural networks. Neurosymbolic Artificial Intelligence, 2024.arxiv.org/abs/2310.11884
- [32]Jens Lehmann. DL-Learner: Learning concepts in description logics. Journal of Machine Learning Research, 10:2639–2642, 2009.DOI
- [33]Jens Lehmann, Sören Auer, Lorenz Bühmann, and Sebastian Tramp. Class expression learning for ontology engineering. Journal of Web Semantics, 9(1):71–81, 2011.DOI
- [34]Jens Lehmann and Pascal Hitzler. Concept learning in description logics using refinement operators. Mach. Learn., 78(1-2):203–250, 2010.DOI
- [35]Cathy Ong Ly, Balagopal Unnikrishnan, Tony Tadic, Tirth Patel, Joe Duhamel, Sonja Kandel, Yasbanoo Moayedi, Michael Brudno, Andrew Hope, Heather Ross, and Chris McIntosh. Short-cut learning in medical ai hinders generalization: method for estimating ai model generalization without external data. npj Digital Medicine, 7(1):124, 2024.
- [36]B. S. Manjunath and W. Y. Ma. Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):837–842, 1996.DOI
- [37]Eder Martinez, Michael Okonoda, Abhilekha Dalal, and Lior Shamir. Detection and profiling of dataset bias in cnn analysis of images. Computing in Science & Engineering, 2026.DOI
- [38]Ian Model and Lior Shamir. Comparison of data set bias in object recognition benchmarks. IEEE Access, 3, 2015.DOI
- [39]Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7, Sep 2016.arxiv.org/abs/1604.03169
- [40]Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.arxiv.org/abs/2006.14032
- [41]Loris Nanni, Stefano Ghidoni, and Sheryl Brahnam. Handcrafted vs. non-handcrafted features for computer vision classification. Pattern recognition, 71:158–172, 2017.DOI
- [42]Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008.DOI
- [43]Mehmet Alican Noyan. Uncovering bias in the PlantVillage dataset. arXiv preprint arXiv:2206.04374, 2022.arxiv.org/abs/2206.04374
- [44]Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2204.10965
- [45]Timo Ojala, Matti Pietikäinen, and Topi Mäenpää. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971–987, 2002.
- [46]Michael Okonoda, Eder Martinez, Abhilekha Dalal, and Lior Shamir. Unmasking biases and reliability concerns in convolutional neural networks analysis of cancer pathology images. Electronics, 15(6), 2026.DOI
- [47]Alex Olsen, Dmitry A. Konovalov, Bronson Philippa, Peter Ridd, Jake C. Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, Brendan Calvert, Mostafa Rahimi Azghadi, and Ronald D. White. Deepweeds: A multiclass weed species image dataset for deep learning. Scientific Reports, 9(2058), 2 2019.
- [48]Nikita Orlov, Lior Shamir, Tomasz Macura, Josiah Johnston, D. Mark Eckley, and Ilya G. Goldberg. WND-CHARM: Multi-purpose image classification using compound image transforms. Pattern Recognition Letters, 29(11):1684–1693, 2008.pubmed.ncbi.nlm.nih.gov/18958301
- [49]Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
- [50]Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ” why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.arxiv.org/abs/1602.04938
- [51]Md Kamruzzaman Sarker and Pascal Hitzler. Efficient concept induction for description logics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3036–3043, 2019.arxiv.org/abs/1812.03243
- [52]Md Kamruzzaman Sarker, Ning Xie, Derek Doran, Michael Raymer, and Pascal Hitzler. Explaining trained neural networks with Semantic Web technologies: First steps. arXiv preprint arXiv:1710.04324, 2017.arxiv.org/abs/1710.04324
- [53]Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, and Pascal Hitzler. A case study on concept induction for neuron-level interpretability in cnn. arXiv preprint arXiv:2603.00197, 2026.arxiv.org/abs/2603.00197
- [54]Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2:476–486, 2020.arxiv.org/abs/2001.05371
- [55]Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
- [56]Atefeh Shahroudnejad. A survey on understanding, visualizations, and explanation of deep neural networks. arXiv preprint arXiv:2102.01792, 2021.arxiv.org/abs/2102.01792
- [57]Lior Shamir. Evaluation of face datasets as tools for assessing the performance of face recognition methods. International Journal of Computer Vision, 79(3):225–230, 2008.pubmed.ncbi.nlm.nih.gov/18776952
- [58]Lior Shamir. Udat: A multi-purpose data analysis tool. Astrophysics Source Code Library, pages ascl–1704, 2017.
- [59]Lior Shamir, Nikita Orlov, D. Mark Eckley, Tomasz Macura, Josiah Johnston, and Ilya G. Goldberg. Wndchrm – an open source utility for biological image analysis. Source Code for Biology and Medicine, 3:13, 2008.DOI
- [60]Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk–quantifying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8218–8226, 2019.arxiv.org/abs/1812.06707
- [61]Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2014.arxiv.org/abs/1312.6034
- [62]Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. Plantdoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 249–253. Association for Computing Machinery, 2020.DOI
- [63]Kaspars Sudars, Janis Jasko, Ivars Namatevs, Liva Ozola, and Niks Badaukis. Dataset of annotated food crops and weed images for robotic computer vision control. Data in brief, 31:105833, 2020.DOI
- [64]Hideyuki Tamura, Shunji Mori, and Takashi Yamawaki. Textural features corresponding to visual perception. IEEE Transactions on Systems, Man, and Cybernetics, 8(6):460–473, 1978.
- [65]Sandip Thite, Yogesh Suryawanshi, Kailas Patil, and Prawit Chumchu. Sugarcane leaf dataset: A dataset for disease detection and classification for machine learning applications. Data in Brief, 53:110268, 2024.
- [66]Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1521–1528, 2011.DOI
- [67]Cara Leigh Widmer, Md Kamruzzaman Sarker, Srikanth Nadella, Joshua Fiechter, Ion Juvina, Brandon Minnery, Pascal Hitzler, Joshua Schwartz, and Michael Raymer. Towards human-compatible XAI: Explaining data differentials with concept induction over background knowledge. Journal of Web Semantics, 79:100807, 2023.arxiv.org/abs/2209.13710
- [68]Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.arxiv.org/abs/2006.09994
- [69]Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015.arxiv.org/abs/1506.06579
- [70]John R Zech, Marcus A Badgley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e1002683, 2018.DOI
- [71]Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 818–833, 2014.arxiv.org/abs/1311.2901
- [72]Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2016.arxiv.org/abs/1512.04150
