betaFeedback

WorksComputer Science

Understanding Contextual Bias in Agricultural CNNs Through Hidden-Neuron Concept Induction

Abstract

Convolutional Neural Networks (CNNs) can classify images using background-dominated patches that contain little or none of the target object, showing that they exploit contextual information. Previous studies detected this behavior through classification accuracy but say little about what CNN hidden layers represent. We investigate whether Concept Induction can reveal interpretable visual properties associated with hidden-neuron activations on contextual image patches. Using five agricultural datasets with high, moderate, and low contextual bias, we analyze four CNN architectures at three network depths and five patch locations. Quantitative image descriptors of each patch are encoded in a knowledge graph and used as background knowledge for Concept Induction, which derives logical expressions characterizing neuron activation. These expressions are tested on held-out images using the Mann-Whitney U test and Cliff's delta. Of 48,013 tested neuron-feature pairs, 13,164 (27.4%) were validated. Validation was most frequent at intermediate depths (mean 40.2% at mid layers versus 12.0% at deep layers), and no patch location dominated across datasets. Among the tested descriptors, edge, Gabor, and Tamura texture features had the highest validation rates. Stronger contextual bias did not correspond to higher neuron-feature validation rates: moderately biased Sugarcane had the highest rate (40.5%) and highly biased Cassava the lowest (19.2%). These results suggest that behavioral context dependence and the semantic interpretability of hidden neurons capture different aspects of CNN behavior, and that Concept Induction can complement patch-based bias detection.

Introduction

Convolutional Neural Networks (CNNs) have achieved remarkable performance across a wide range of image-classification tasks, establishing them as a fundamental component of modern computer vision systems [28, 21]. However, strong predictive performance does not necessarily imply that a model has learned the visual characteristics intended by the classification task. Deep neural networks can exploit unintended correlations and other predictive shortcuts present in the training data, achieving high performance without relying primarily on the characteristics of the target object [17, 26, 56]. Because CNNs learn statistical patterns directly from training data, such shortcuts can arise from contextual information, including background patterns, object co-occurrences, acquisition artifacts, and other incidental visual cues that correlate with class labels rather than from the object’s genuine defining features [6, 68]. Evidence of such behavior has been reported across diverse application settings, demonstrating that reliance on contextual or background information is not restricted to a particular dataset or domain [6, 35]. Such reliance can become problematic when these associations are specific to the training distribution, as changes in background composition, acquisition conditions, or other contextual characteristics may affect model robustness and generalization [17, 3, 35].

Standard evaluation measures, such as accuracy, precision, recall, and F1-score, quantify how successfully a model predicts class labels, but provide limited information about the visual evidence underlying those predictions. Consequently, a model that relies on unintended contextual cues may still achieve strong performance when the same correlations are present in the training and evaluation data[60, 50, 70]. This limitation has motivated diagnostic approaches that systematically modify the visual information available to a trained model, including object masking, background manipulation, image occlusion, cropping, and region- or patch-based evaluation[3, 6, 46, 37]. By restricting or altering portions of an image and observing the resulting model behavior, such approaches can reveal whether predictions remain supported when expected object information is reduced or contextual information is modified. In particular, spatial patch-based evaluation provides a direct means of examining whether restricted image regions retain sufficient information for classification. Predictive performance substantially above the expected chance level on such patches can therefore provide evidence that a model has learned class-discriminative information from spatially localized image content, including contextual regions that may not contain the complete target object[57, 37, 9].

Building on this form of diagnostic evaluation, our prior studies systematically investigated contextual dependence by evaluating CNNs on spatially restricted image regions across different classification settings. Experiments involving biomedical images, a manually curated object-classification dataset, and agricultural image datasets showed that trained CNNs can retain non-trivial classification capability even when predictions are made from only a portion of the original image [46, 37, 9]. In the agricultural setting, five predefined spatial regions such as top-left, top-right, center, bottom-left, and bottom-right were evaluated across multiple datasets and CNN architectures, revealing substantial differences in the extent to which models could classify images from restricted regions [9]. These findings provided a behavioral characterization of context dependence and demonstrated that its severity varies across datasets and model architectures.

Patch-based evaluation, however, describes whether predictive information is present within restricted image regions without explaining what visual information the network internally represents from those regions. Classification performance alone does not reveal which hidden neurons respond to properties of the patches, how such representations vary across network depth, or which visual characteristics are associated with their activation. Consequently, identifying context-dependent classification behavior represents only one part of understanding how contextual information is encoded within a trained CNN. Examining the internal representations associated with these image regions is necessary to move from behavioral observations of context dependence toward a semantic characterization of the learned representations.

Existing interpretability methods provide several ways to examine such internal representations. Attribution and localization methods identify image regions contributing to predictions [55, 61, 72], while concept- and neuron-level approaches such as TCAV [27], ACE [19], Network Dissection [5], and CLIP-Dissect [44] associate learned representations with human-interpretable concepts. However, these approaches generally characterize internal representations through attribution, concept association, or semantic labeling rather than logically structured, compositional descriptions of neuron activation grounded in quantitative visual properties or support reasoning over such properties using explicit background knowledge. This motivates a complementary approach in which hidden-neuron activation patterns are connected to structured symbolic representations, and formal concept learning characterizes the visual conditions associated with neuron responses [8].

In this work, we investigate the hidden-neuron representations associated with the contextual image patches used in our preceding agricultural context-bias analysis. We develop a neuro-symbolic analysis framework that integrates hidden-layer activation analysis with quantitative image descriptors and Description Logic Concept Induction. For selected hidden layers of four CNN architectures, neuron activations are extracted from predefined spatial image patches, while complementary visual characteristics of the same patches are represented using UDAT image descriptors. These descriptors are discretized and encoded within OWL knowledge graphs, providing structured background knowledge for Concept Induction. DL-Learner is then used to derive Description Logic concepts characterizing the visual properties associated with selected neuron activation patterns. The induced associations are subsequently evaluated on held-out images using statistical significance and effect-size criteria, providing a systematic mechanism for identifying semantically validated neuron–feature associations.

Using this framework, we examine representations across five agricultural image datasets selected to span the high-, moderate-, and low-context-bias conditions identified in the preceding patch-based study. The analysis considers multiple CNN architectures, three representative network depths, five spatial patch locations, and complementary families of quantitative image descriptors. This experimental design allows us to investigate not only whether semantic associations can be identified for hidden neurons, but also where these associations emerge within CNN feature hierarchies, how they vary spatially, which visual properties most frequently characterize them, and whether their prevalence corresponds to the degree of context dependence observed behaviorally.

Specifically, we investigate the following research questions:

  • RQ1: At which depths of CNN architectures do semantically interpretable neuron representations associated with contextual image patches emerge? At which layers of CNN architectures do semantically interpretable, context-sensitive representations emerge?

  • RQ1.1: Which spatial patch locations consistently produce semantically validated neuron representations across agricultural datasets?

  • RQ2: Which image-feature families most consistently characterize the hidden-neuron representations associated with contextual image patches?

  • RQ3: How does the emergence of validated semantic concepts relate to the degree of context bias observed across agricultural datasets?

The main contributions of this work are as follows:

  • We extend the analysis of context dependence in agricultural image classification from observable model behavior to internal CNN representations through a neuro-symbolic framework that associates hidden-neuron activation patterns with semantically interpretable visual properties using Description Logic Concept Induction.

  • We introduce a systematic analysis and validation pipeline for identifying neuron–feature associations, combining targeted neuron selection, symbolic concept generation, and independent statistical validation on held-out images.

  • We provide a multi-dimensional empirical characterization of semantically validated representations across five agricultural datasets, four CNN architectures, three representative network depths, five spatial patch locations, and multiple visual feature families, revealing how these associations vary across depth, spatial region, and visual characteristics.

  • We relate representation-level semantic validation to previously observed behavioral context dependence and show that stronger patch-based context bias does not necessarily correspond to a greater prevalence of semantically validated neuron–feature associations, highlighting that behavioral context dependence and semantic concept emergence capture distinct aspects of CNN behavior.

The remainder of this paper is organized as follows. § 2 reviews related work on contextual bias, CNN interpretability, and concept-based and neuro-symbolic explanation methods. § 3 introduces the background concepts and technical foundations necessary to understand the proposed framework. § 4 describes the datasets, hidden-layer analysis, symbolic representation construction, Concept Induction, and statistical validation framework. § 5 presents and discusses the experimental findings with respect to the research questions. Finally, § 6 summarizes the main findings, discusses limitations and implications, and outlines directions for future work.

Related Work

Context bias and shortcut learning in image classification

Image classifiers can learn regularities of a dataset instead of properties of the target object, a problem recognized even before deep learning became dominant. Early investigation showed that popular recognition benchmarks have their own distinctive signature [66]. A classifier could tell which dataset an image came from, and models trained on one dataset generalized poorly to others; such failures were later described as emph shortcut learning [17]. A shortcut is a decision rule that works on the benchmark but fails when conditions change because it relies on spurious cues that only happen to correlate with the label.

Background reliance is the best documented shortcut in natural images. Camera-trap classifiers lose much of their accuracy at locations not seen during training [6]. Images with the object removed still yield non-trivial accuracy on ImageNet [68]. One study found a model that recognized aeroplanes from artificial padding patterns and horses from a source watermark [29]. Similar reliance on unintended cues has been observed in medical imaging, where CNNs could identify the hospital that produced a chest radiograph with almost perfect accuracy [70]. Since pneumonia rates differed between hospitals, this site information leaked into the pneumonia predictions. Likewise, COVID-19 detectors were also found to rely on confounders rather than lung pathology, leading to poor generalization across hospitals [13]. In agricultural imaging, a model trained on only eight background pixels of PlantVillage images reached 49% accuracy, compared with a chance level of 2.6% [43]. Similarly, plant phenotyping models were found to make confounder-driven decisions, and correcting them required expert feedback during training [54].

A related line of work measures dataset bias by classifying image regions that contain little or no task-relevant information. Small, seemingly blank sub-images extracted from object recognition benchmarks were classified with accuracy well above chance [38]. Similar findings were reported in biomedical imaging, where CNNs achieved high classification accuracy using image regions without biomedical content [14].

Building on this direction, subsequent studies investigated contextual bias across different image-classification domains. Image transformations were used to distinguish background bias from useful contextual information across several image-classification benchmarks [16]. An investigation tested thirteen cancer pathology datasets with four CNN architectures [46]. Background regions without clinical content were classified with up to 93% accuracy. More recently, contextual bias was examined across eight agricultural image datasets, where background-dominated patches achieved above-chance classification accuracy in six datasets [9]. Although class imbalance explained part of this effect, it did not fully account for the results.

While these studies show that context bias appears in several domains through classification performance, , they do not investigate how the visual characteristics of contextual regions are reflected in CNN hidden-neuron activations. The present work addresses this gap by applying Concept Induction to analyze the relationships between visual properties of image patches and hidden-neuron activations, using agricultural datasets exhibiting high, moderate, and low levels of contextual bias identified in [9].

Explainability of CNN hidden representations

Early work on CNN interpretability focused on visualizing the patterns that activate individual units. Deconvolutional networks were used to project intermediate activations back into pixel space [71], while gradient-based approaches generated class-maximizing images and saliency maps [61]. Subsequent work introduced regularization techniques to make such visualizations easier to recognize [69]. These studies provided insights into the hierarchical representations learned by CNNs, where early layers respond to low-level features such as edges and colors, while deeper layers respond to more complex patterns such as object parts and objects. However, interpreting these visualizations remains largely qualitative. Attribution methods such as Grad-CAM highlight the image regions contributing to a particular prediction [55]. They show where the model looks, but not what it has learned to represent.

Network Dissection introduced a quantitative approach to interpreting individual neurons by comparing thresholded activation maps with pixel-level annotations of colors, textures, materials, parts, objects, and scenes [5]. This approach was subsequently extended to logical combinations of concepts, revealing both abstract neurons and neurons that respond to several unrelated concepts [40].

Other methods investigate concepts represented in the activation space rather than focusing exclusively on individual neurons. TCAV learns a concept direction from example images and measures its effect on a class prediction [27], while ACE identifies concepts automatically by clustering image segments in activation space [19]. More recent methods use vision-language models to generate natural-language descriptions of hidden neurons [44, 22]. Neuron-level analysis has also been extended to shortcut learning, where individual neurons can be scored by how strongly they depend on spurious features [30].

Despite these advances, existing approaches have limitations in the context of our study. Concept-based explanations can depend on the probe dataset and the available concept vocabulary, potentially producing different interpretations under different settings [49]. Traditional Dissection-style methods need pixel-level concept masks, which are generally unavailable in agricultural image datasets. Similarly, labels produced by vision-language models come from an opaque model themselves, so they inherit its lack of transparency. Finally, all of these methods provide labels, vectors, or scores; they do not generally induce logical statements grounded in explicit background knowledge. Logical statements of this kind would express the relationships between visual features and neuron activations to be represented as formally defined concepts that can be investigated within a symbolic reasoning framework.

Neuro-symbolic explanation and Concept Induction

Neuro-symbolic AI combines neural learning with symbolic knowledge representation and reasoning [24, 31]. In explainable AI, ontologies can provide a structured vocabulary for interpreting neural network behavior. For example, classifiers have been trained to map a neural network’s activations to ontology concepts, allowing model outputs to be explained using symbolic terms [12].

Concept Induction provides an alternative approach in which explanations are learned directly as logical class expressions. Given a knowledge base expressed in Description Logic (DL), along with sets of positive and negative individuals, the objective is to identify a class expression that covers the positive examples while excluding the negative examples [34, 51]. The resulting DL concepts have formal semantics and can express complex relationships through combinations of classes and properties using logical constructors such as conjunction, disjunction, and quantifiers. These expressions are interpreted with respect to the underlying background knowledge.

DL-Learner is one of the most established systems for concept induction that supports OWL and RDF background knowledge and provides access to reasoning-based learning algorithms, including refinement-based learners such as CELOE [32, 7]. However, refinement-based approaches can be computationally expensive due to the large number of reasoning operations required during concept search. To address this limitation, Efficient Concept Induction from Individuals (ECII) was introduced as a computationally efficient alternative, although with a more restricted search space [51].

Concept Induction has previously been explored for explaining neural network predictions and internal representations. A Semantic Web-based approach demonstrated how induced concepts could be used to explain the classifications of trained neural networks [52], while a Wikipedia-derived class hierarchy was used to generate meaningful explanations of differences between data sets in ways that people found meaningful [67]. This direction was subsequently extended to hidden-neuron analysis, where images that strongly activate a neuron were treated as positive examples and those producing weak activations as negative examples [11, 10]. The induced concepts are then validated against new, purpose-retrieved images to confirm that they describe the neuron’s behavior. This pipeline produced meaningful labels for dense-layer neurons of a scene classifier. The framework was subsequently automated [1] and later studies utilized this with different training targets and on the SUN2012 dataset [2, 53].

Previous applications of Concept Induction to hidden-neuron analysis have two limitations. Its background knowledge came from object annotations of whole images, and it analyzed only the last dense layer. The resulting concepts therefore say which objects a neuron activation responds to, but not which visual properties drive it, or how these change with network depth and image position. We address both of these limitations by building background knowledge from quantitative descriptors of contextual patches, and applying Concept Induction at three network depths.

Image descriptors and semantic background knowledge

Before the widespread adoption of deep learning, images were usually described with handcrafted descriptors that have exact mathematical definitions. Haralick features capture texture statistics derived from gray-level co-occurrence matrices (GLCMs), including measures such as contrast, correlation, and entropy [20], while Tamura features approximate perceived texture properties, such as coarseness, contrast, and directionality [64]. Gabor filters measure texture patterns across several scales and orientations [36], whereas Local binary patterns describe fine texture and are invariant to gray-scale changes and rotation [45]. These descriptors have explicit mathematical definitions and correspond to visual properties that can be named. A single value, such as high entropy, can therefore be read directly, unlike a learned CNN feature.

WND-CHARM extended the use of handcrafted features by introducing a comprehensive set of image descriptors applicable across different image domains [48, 59]. It extracts features from the raw image and its Fourier, wavelet, and Chebyshev transforms, including combinations of these transformations. The descriptor families include polynomial decompositions (Zernike and Chebyshev), texture features (Haralick, Tamura, and Gabor), edge and object statistics, and multiscale histograms. Together, these descriptors characterize different aspects of image appearance, including texture, frequency content, structure, and intensity distribution. The set was designed to work across image domains without tuning. UDAT builds on these descriptors as a general quantitative analysis tool [58]. Previous research has demonstrated that handcrafted and learned features can provide complementary information [41], while ImageNet-trained CNNs have been shown to exhibit a strong reliance on texture information [18]. These findings motivate the use of quantitative image descriptors to investigate the visual characteristics associated with CNN hidden-neuron activations.

While handcrafted descriptors have traditionally been used for image classification and quantitative analysis, we employ them to construct symbolic background knowledge for explaining CNN hidden-neuron activations. In this work, UDAT converts low-level visual characteristics of each patch into structured descriptors, which are subsequently discretized and represented in OWL. This representation enables Concept Induction to identify logical descriptions of the visual properties associated with neuron activation patterns.

Preliminaries

Knowledge graphs and Description Logic representation

Knowledge graphs provide a structured representation of entities and their relationships, typically expressed as nodes connected through labeled relations. When supported by an ontology, a knowledge graph additionally provides a formal vocabulary for describing the types of entities, their properties, and the relationships that may hold among them. This representation enables data from individual instances to be combined with formally defined domain knowledge and supports logical reasoning over the resulting structure [25, 23].

An ontology-based knowledge graph can be viewed as consisting of a terminological component (TBox) and an assertional component (ABox). The TBox defines the classes and properties used to describe a domain and the relationships among these classes, while the ABox contains assertions about individual instances. For example, an image may be represented as an individual of the class Image and connected through an object property to another individual representing one of its visual characteristics.

Description Logics (DLs) provide a formal knowledge-representation framework underlying ontology languages such as OWL. In addition to atomic classes and properties, DLs allow complex class expressions to be constructed using logical operators such as conjunction (⊓\sqcap), disjunction (⊔\sqcup), negation (¬\neg), and existential (∃\exists) and universal (∀\forall) restrictions [25]. For example, an expression of the form

Image⊓∃ hasFeature.FeatureRange\mathrm{Image} \sqcap\exists\,\mathrm{hasFeature}.\mathrm{FeatureRange}

describes images that are related through hasFeature to an instance belonging to FeatureRange. Such expressions provide a symbolic representation that can be automatically reasoned over and form the basis for the Concept Induction approach used in this work.

Concept Induction

Concept Induction [34] is a symbolic learning approach that derives Description Logic class expressions from positive and negative examples represented within a knowledge base. Given a set of positive examples PP, a set of negative examples NN, and a knowledge base KK, Concept Induction searches for a class expression EE that characterizes the positive examples while distinguishing them from the negative examples. Ideally, an induced expression satisfies K⊨E(p)K \models E(p) for all p∈Pp \in P while K⊭E(q)K \not\models E(q) for all q∈Nq \in N. When perfect separation is not possible, candidate expressions can instead be evaluated according to how accurately they cover the positive and negative examples.

In this work, Concept Induction is performed using DL-Learner [7], a framework for learning Description Logic concepts from examples and structured background knowledge. Specifically, we employ its CELOE (Class Expression Learning for Ontology Engineering) [33] algorithm, which searches the space of possible class expressions using refinement operators and evaluates candidate expressions according to their ability to distinguish positive from negative examples. The resulting expressions can combine classes and relationships defined in the underlying ontology, producing symbolic descriptions of characteristics shared by the positive examples.

Within our analysis framework, this capability provides the connection between numerical CNN activation behavior and symbolic descriptions of image characteristics. Images exhibiting contrasting neuron activation behavior can be represented as positive and negative examples, while their visual characteristics are encoded in an OWL knowledge graph. Concept Induction can then identify class expressions describing visual properties associated with the neuron’s activation behavior. The construction of these examples and the application and validation of the induced concepts are described in § 4.

UDAT image descriptors

UDAT [58] is a quantitative image analysis tool based on WND-CHARM [48, 59] that extracts a broad range of numerical descriptors characterizing image content. These descriptors are computed from the original image as well as from image transformations, including Fourier, wavelet, Chebyshev, edge, color, and hue transforms, and combinations of these transformations (e.g., the Fourier transform of the wavelet transform). For each contextual image patch, UDAT generates 4,041 numerical descriptors covering different aspects of image appearance, including texture, intensity distribution, edges, and structural patterns.

In this study, we use only descriptors computed on the raw image (338 descriptors per patch, organized into 13 feature families). Descriptors of transformed images describe the image in a different domain (e.g., frequency coefficients of an edge map) and therefore cannot be related directly to visual properties of the patch. Whereas raw-image descriptors measure properties such as texture, edges, or intensity distribution that remain interpretable in the resulting explanations. Table 1 summarizes the feature families, the number of numerical descriptors associated with each family, and the visual properties they quantify. Together, they cover intensity and color distributions, texture, edges and objects, and frequency- and shape-related content. Each numerical descriptor is treated as an individual visual attribute for subsequent discretization and knowledge graph construction. The selection of feature families for individual CNN layers and their use in Concept Induction are described in § 4.

Feature family# ValuesVisual property quantified
Intensity statistics8Global distribution of pixel intensities (e.g., mean, median, standard deviation, entropy)
Color histogram19Distribution of colors in the image
Combined first four moments48Mean, standard deviation, skewness and kurtosis of intensity over image regions
Multiscale histograms24Intensity histograms computed at several bin resolutions
Edge statistics28Magnitude and direction of intensity gradients (means, medians, variances, histograms, homogeneity)
Object statistics34Properties of segmented objects: count, area, distances, Euler number, centroid
Haralick texture28Statistics of the gray-level co-occurrence matrix, e.g., contrast, correlation, entropy [20]
Tamura texture6Perceptual texture properties: coarseness, contrast, directionality [64]
Gabor filters7Responses to oriented band-pass filters at several frequencies
Fractal features20Self-similarity and roughness of image structure
Radon transform12Intensity projections along multiple angles, capturing oriented structure
Chebyshev–Fourier32Coefficients of orthogonal polynomial expansions describing spatial-frequency content
Zernike moments72Orthogonal moments describing shape and spatial distribution of intensity
Total338

Table 1. UDAT feature families considered in this study, the number of numerical descriptors associated with each family, and the visual properties they quantify.

Prior patch-based context-bias study

This study builds directly on a previous empirical investigation of context bias in CNN-based agricultural image classification [9]. The earlier work evaluated whether CNN classifiers could retain predictive performance when semantically meaningful content was largely removed from the input images. The study covered eight widely used plant and agricultural image datasets: PlantVillage [39], PlantDoc [62], DeepWeeds [47], Cassava Leaf Disease [15], WeedCrop [63], OpenSprayer [4], Sugarcane Leaf Disease [65], and Oxford-102 Flowers [42]. These datasets span plant disease recognition, weed identification, crop classification, spraying-status recognition, and fine-grained flower classification, and include both controlled and field-acquired imagery. Four CNN architectures were evaluated: ResNet50V2, InceptionV3, DenseNet121, and VGG16. For each dataset, the models were trained on the original images and subsequently evaluated on predefined spatial image patches. Five spatial regions were considered for every image: top-left, top-right, center, bottom-left, and bottom-right. Each patch had an original size of (20×2020 \times20) pixels and was resized to the input dimensions required by the corresponding CNN architecture. The original class label was preserved for each extracted patch, enabling direct evaluation of whether the trained classifier could predict the original class using only a spatially restricted region of the image. Most patches contained little or no agriculturally relevant visual content, although the center patch could occasionally include part of the primary subject.

Context dependence was assessed by comparing classification accuracy on the patch datasets with the corresponding random-chance accuracy of each classification task. For dataset dd containing KdK_d classes, the uniform random-chance accuracy is:

Cd=1Kd.(1)C_d = \frac{1}{K_d}. \tag*{(1)}

Patch performance substantially above this baseline was interpreted as evidence that the CNN could exploit contextual or dataset-specific signals even when the primary agricultural object or disease evidence was absent. Such signals may include background texture, lighting patterns, framing characteristics, acquisition artifacts, or other recurring visual regularities associated with the class labels.

DatasetTasks# ImagesAcquisition Setting# ClassesRandom ChancePatch Acc. RangeMarginContext-Bias Category
PlantVillageDisease classification∼54,000Controlled156.7%9.1–19.1%6.2 ppModerate
PlantDocDisease classification2,569Field273.7%2.0–7.1%0.03 ppLow
DeepWeedsSpecies classification17,509Field911.1%51.3–55.0%40.6 ppHigh
CassavaDisease classification21,397Field520%16.0–55.8%13.6 ppHigh
WeedCropCrop/weed classification2,822Field/UAV250%85.6–92.4%41.1 ppHigh
OpenSprayerWeed classification∼10,000UAV / aerial250%58.4–80.6%20.2 ppHigh
SugarcaneDisease detection∼3,000+Field119.1%11.4–22.8%7.8 ppModerate
Oxford 102Species classification∼8,200Field1020.98%0.6–3.9%0.7 ppLow

Table 2. Summary of dataset characteristics and patch-based context-bias results reported in the previous study. “Chance Acc.” denotes random-chance accuracy, and “Patch Acc.” denotes the reported patch accuracy range.

The previous analysis revealed considerable variation in context dependence across datasets. Deep-Weeds, WeedCrop, OpenSprayer, and Cassava exhibited pronounced above-chance performance on patch-only inputs, whereas PlantVillage and Sugarcane showed more moderate context dependence. In contrast, PlantDoc and Oxford-102 Flowers produced patch accuracies close to their corresponding chance baselines, indicating comparatively lower dependence on restricted contextual regions. The study also found that susceptibility to context-dependent behavior varied across CNN architectures, demonstrating that architectural design alone does not guarantee robustness against spurious contextual correlations. These findings established that patch-level predictive behavior can reveal the presence and relative degree of contextual dependence in CNN-based agricultural classifiers. However, the analysis remained primarily behavioral: it demonstrated that models could exploit restricted image regions, but did not identify which visual characteristics within those regions were represented by the CNNs or how such representations emerged across hidden layers. The previous study therefore identified feature-level investigation as an important direction for future work, including the use of image descriptors such as those provided by UDAT [58] to characterize the visual information contained in contextual patches.

The present study extends this earlier work from detecting context dependence to explaining its internal representation. Using the same patch-based experimental setting, we investigate hidden-neuron activations across multiple CNN depths and characterize them using UDAT-derived visual descriptors, knowledge graph representations, Description Logic Concept Induction, and held-out statistical validation. This enables analysis of not only whether contextual bias is present, but also which semantically interpretable visual feature families are associated with the corresponding hidden-neuron representations.

Methodology

Neuro-Symbolic Analysis Framework

This study adapts a neuro-symbolic hidden-neuron explanation approach based on Concept Induction [11] to investigate the internal representations associated with contextual image patches. The analysis builds on the agricultural context-bias setting described in § 3.4 and shifts the focus from observing patch-level classification behavior to characterizing the visual properties represented within CNN hidden layers. Figure 1 presents an overview of the analysis pipeline. For each dataset and spatial patch location, patch images are provided to the corresponding trained CNN, and hidden-layer activations are collected at three network depths: mid, mid-deep, and deep. The resulting activation patterns are used to identify neurons suitable for subsequent explanation analysis.

Overview of the neuro-symbolic analysis pipeline

Figure 1. Overview of the neuro-symbolic analysis pipeline for interpreting CNN hidden-neuron activations from contextual image patches.

In parallel, the visual characteristics of the patch images are extracted using UDAT image descriptors. The extracted feature values are transformed into symbolic representations and encoded in an OWL knowledge graph, which provides the background knowledge used for Concept Induction. For each selected neuron-feature-family combination, images associated with contrasting activation behavior are used as positive and negative examples. Concept Induction then searches for Description Logic expressions using the OWL knowledge graph that distinguish the positive examples from the negative examples, thereby generating candidate semantic explanations of neuron activation. The resulting neuron-feature associations are evaluated using held-out data and statistical validation. Associations that satisfy the predefined validation criteria are retained as validated neuron-feature pairs. These pairs form the fundamental observations used in the subsequent analysis. Finally, the validated neuron-feature pairs are aggregated along several experimental dimensions, including CNN depth, spatial patch location, dataset, and image-feature family.

These aggregated representations are used to examine where context-sensitive semantic representations emerge within CNNs. In particular, the analysis examines the network depths at which such representations arise, the spatial regions and visual feature families with which they are most consistently associated, and how their occurrence relates to the context-bias levels observed in the preceding patch-based study.

The prior patch-based study categorized the eight datasets according to context-bias severity observed from patch-level classification performance. Since the subsequent hidden-neuron and Concept Induction analysis is considerably more computationally intensive, we selected a representative subset of five datasets rather than repeating the analysis across all eight datasets. The selection is designed to span the observed bias spectrum while providing greater representation of datasets exhibiting strong contextual dependence, for which identifying the internal representations associated with image context is particularly relevant. Accordingly, Cassava, WeedCrop, and OpenSprayer from the high-bias group, Sugarcane from the moderate-bias group, and Oxford-102 from the low-bias group have been chosen. This subset therefore enables comparison of hidden-layer semantic representations across datasets exhibiting substantially different levels of previously observed context dependence.

Hidden-Layer Selection and Activation Extraction

To examine how context-sensitive representations evolve across network depth, three hidden layers were selected from each CNN architecture to represent mid, mid-deep, and deep stages of feature processing. The layers are selected from progressively deeper major processing blocks within each architecture, providing representative points along the feature hierarchy while avoiding adjacent layers that capture highly similar stages of computation. Because ResNet50V2, DenseNet121, InceptionV3, and VGG16 differ substantially in their connectivity patterns and architectural organization, we do not assume exact layer-wise equivalence across the four networks. Instead, the selected layers provide architecture-specific reference points corresponding broadly to intermediate, later-intermediate, and deep feature-processing stages. The selected layers are summarized in Table 3.

ArchitectureMidMid-deepDeep
ResNet50V2conv3_block4_outconv4_block6_outconv5_block3_out
DenseNet121conv3_block12_concatconv4_block24_concatrelu
InceptionV3mixed5mixed7mixed10
VGG16block3_conv3block4_conv3block5_conv3

Table 3. Hidden layers selected to represent mid, mid-deep, and deep stages of each CNN architecture

Earlier and intermediate convolutional layers primarily encode basic visual features, such as edges, colors, and textures, into local patterns, while progressively deeper layers combine information from larger image regions to represent more complex and abstract features [71]. Analyzing features at these different layer depths helps us understand how context-sensitive concepts gradually develop and change as information moves through the CNN.

For each dataset and spatial patch location, the corresponding patch images are forwarded through the trained CNN, and activation values are extracted from the selected hidden layers. Since the analyzed layers are convolutional or feature-map-producing layers, each layer contains a distinct set of feature channels. Consequently, neurons analyzed at the mid, mid-deep, and deep stages are not treated as the same neurons tracked across depth; rather, each selected layer contributes its own set of candidate neurons for subsequent analysis. For each selected layer, an intermediate model was constructed using the trained CNN’s input and the selected hidden layer as its output. Since the analyzed layers produce multi-channel feature maps, global average pooling (GAP) was applied to reduce each channel’s activation map to a single value by averaging over its spatial positions. Each feature channel is treated as a neuron in the subsequent analysis. Repeating this procedure over all patch images produced an image-by-neuron activation matrix for each combination of dataset, patch location, architecture, and selected hidden layer.

ArchitectureDepthLayerInitial neurons/ channelsAfter variance filterAfter correlation rankingFinal selected
ResNet50V2Midconv3_block4_out51238410050
ResNet50V2Mid-deepconv4_block6_out1,02476810050
ResNet50V2Deepconv5_block3_out2,0481,53610050
DenseNet121Midconv3_block12_concat51238410050
DenseNet121Mid-deepconv4_block24_concat1,02476810050
DenseNet121Deeprelu1,02476810050
InceptionV3Midmixed576857610050
InceptionV3Mid-deepmixed776857610050
InceptionV3Deepmixed102,0481,53610050
VGG16Midblock3_conv325619210050
VGG16Mid-deepblock4_conv351238410050
VGG16Deepblock5_conv351238410050

Table 4. Reduction in candidate neurons across the three-stage neuron-selection procedure for the analyzed CNN layers.

The extracted activation values are then used as the basis for neuron filtering and the construction of positive and negative example sets for Concept Induction. To avoid information leakage during concept evaluation, the constructed patch data are divided into analysis and held-out subsets, with the former used for neuron selection and concept generation and the latter for validation.

Neuron Selection

To restrict Concept Induction to neurons exhibiting informative and decision-relevant activation behavior, a three-stage neuron-selection procedure is applied independently to each analyzed hidden layer. Importantly, neuron selection is performed using only the 70% analysis split; the 30% holdout split was excluded from all filtering and concept-generation steps and reserved exclusively for subsequent validation. For each neuron, activation values are collected across the analysis images. The first filtering stage removed neurons exhibiting comparatively little variation in their responses. Neuron-wise activation variance was calculated, and neurons falling within the bottom 25th percentile of the variance distribution for the corresponding layer were discarded. Because this criterion is quantile-based rather than based on a fixed variance threshold, the cutoff adapts to the activation characteristics of each layer. Among the remaining neurons, a second filtering stage assessed association with the CNN’s prediction confidence. For each neuron nn, the Pearson correlation coefficient between its activation values and the corresponding model-confidence scores was calculated,

rn=∑i(ain−aˉn)(ci−cˉ)∑i(ain−aˉn)2∑i(ci−cˉ)2,r_n = \frac{\sum_i (a_{in} - \bar{a}_n)(c_i - \bar{c})}{\sqrt{\sum_i (a_{in} - \bar{a}_n)^2 \sum_i (c_i - \bar{c})^2}},

where aina_{in} denotes the activation of neuron nn for image ii, and cic_i denotes the CNN prediction confidence for that image. Neurons are ranked according to the absolute correlation magnitude, ∣rn∣\lvert r_n \rvert, and the 100 highest-ranked neurons are retained. Using the absolute value allows both positively and negatively associated activation patterns to be considered potentially relevant to model confidence. Finally, the retained 100 neurons are re-ranked according to their activation variance, and the 50 highest-variance neurons are selected for Concept Induction. Thus, each analyzed layer contributes at most 50 candidate neurons for each evaluated feature family.

UDAT Feature Extraction and Discretization

Each contextual patch is described with UDAT, using the 338 raw-image descriptors organized into 13 feature families (Table 1).

Not every family is tested against every hidden layer. Feature families are assigned to individual network layers based on an LLM-guided assessment. The assessment was conducted using Claude Sonnet (Anthropic, 2026). For each architecture, the model was prompted with the layer names, spatial resolutions, and channel depths alongside descriptions of the 13 UDAT feature families, and asked to classify each layer-feature-family combination as test, optional/borderline, or skip based on the type of visual representations expected at that network depth. Architecture-specific properties were also considered, including residual skip connections in ResNet50V2, dense feature concatenation in DenseNet121, and multi-scale parallel processing in InceptionV3. The full prompt template is provided in Appendix § 8. of their relevance to the type of visual representations expected at different network depths. of their relevance to the type of visual representations expected at different network depths. The assessment categorized each layer-feature-family combination as test, optional/borderline, or skip, and only the test combinations are run through Concept Induction. This gives 84 layer-feature-family combinations per dataset-patch configuration, the same for all 25 configurations. Of the 13 families, 11 are evaluated. Intensity Statistics and Color Histogram are marked test only for early layers, which are not analyzed in this study. Zernike moments are run on the mid-deep and deep layers, but CELOE returns only the generic concept Thing for every neuron (7,759 of 7,759 runs that return a concept). These concepts fall under the exclusion rule described in the validation subsection and are not validated, so Zernike moments produce no testable concepts and do not appear in the results.

The raw UDAT outputs are continuous numerical values and therefore cannot be used directly as symbolic concepts in the subsequent Description Logic representation. Each descriptor is therefore discretized into a finite set of ordered value intervals, and each interval is represented as a symbolic bin. A patch is then described by the interval its measurement falls into, not by the exact value. Interval boundaries are set from the minimum and maximum of each descriptor over all patch images of the corresponding dataset-patch configuration, i.e., analysis and held-out images together.

For descriptor fjf_j and patch image xx, the discretization is

fj(x)↦Bj,k,(2)f_j(x) \mapsto B_{j,k}, \tag*{(2)}

where fj(x)f_j(x) is the numerical value of the descriptor for xx, and Bj,kB_{j,k} is the kk-th interval of fjf_j, the one containing that value. For example, a low edge-statistics value, falls into a lower-order bin, and a comparatively high value for the same descriptor falls into a higher-order bin. The discretized representation provides two advantages for the neuro-symbolic analysis. First, it transforms continuous visual measurements into categorical concepts that can be represented explicitly in OWL. Second, it allows Concept Induction to search for human-interpretable combinations of visual characteristics rather than operating directly over raw numerical values.

Knowledge Graph Construction

The discretized UDAT descriptors are encoded as an OWL-based knowledge graph to provide the symbolic background knowledge required for Concept Induction. Each contextual patch image is represented as an individual of the class ‘Image’. For each UDAT descriptor, an object property connects the image individual to a bin individual representing its discretized feature value. The bin individual is subsequently typed using an OWL class corresponding to the numerical interval into which the original descriptor value fell.

For example, in Figure 2, for the Gabor feature family, properties such as hasGaborFilters_Bin0, hasGaborFilters_Bin1, ..., hasGaborFilters_Bin6 connect an image to the discretized values of the corresponding Gabor descriptors. The interval-specific classes are organized under broader descriptor classes using rdfs:subClassOf, thereby preserving both the feature identity and its discretized numerical range. Formally, for image xx and descriptor fjf_j with object property pjp_j and interval classes Bj,1,…,Bj,KB_{j,1}, \ldots, B_{j,K}, the graph contains the assertions

Figure 2. Ontology pattern for one patch image and one UDAT sub-feature (Gabor filters, sub-feature Bin2). The patch individual is linked by hasGaborFilters Bin2 to a value individual (value 0.1225), which is typed with the interval class for [0.105, 0.150]; the five equal-width interval classes are subclasses of the sub-feature class GaborFilters_Bin2. The same pattern is repeated for each of the seven Gabor sub-features.

pj(x,b),Bj,k(b),(3)p_j(x,b), \qquad B_{j,k}(b), \tag*{(3)}

where bb is the value individual of xx for fjf_j, and kk is the index of the interval containing the numerical value fj(x)f_j(x), as defined in the discretization step. Individual OWL representations generated for the patch images are subsequently combined into a single ontology for the corresponding dataset, patch location, and UDAT feature family. This combined knowledge graph is then supplied as the background knowledge in Concept Induction. The ontology therefore encodes each patch as a structured set of symbolic visual attributes. The discretized feature intervals are organized hierarchically, allowing Concept Induction to reason over both specific value ranges and broader descriptor-level abstractions.

Concept Induction

For each selected neuron–feature-family combination, Concept Induction is run with the CELOE algorithm in DL-Learner, using the OWL knowledge graph built from the corresponding UDAT descriptors as background knowledge. Only the 70% analysis split is used; the 30% holdout split remains unseen until the validation stage. For each neuron nn, positive and negative example sets are derived from its activations over the analysis images. Let aina_{in} denote the activation of neuron nn for image xix_i, and let anmax⁡a_n^{\max} be the maximum of aina_{in} over the analysis images. Images with activation of at least 80% of anmax⁡a_n^{\max} form the positive set, and images with nonzero activation of at most 20% of anmax⁡a_n^{\max} form the negative set:

Pn={xi∣ain≥0.8anmax⁡},(4)P_n = \{x_i \mid a_{in} \ge0.8a_n^{\max}\}, \tag*{(4)}
Nn={xi∣0<ain≤0.2anmax⁡}.(5)N_n = \{x_i \mid0 < a_{in} \le0.2a_n^{\max}\}. \tag*{(5)}

Images with zero activation, and images with activation between 20% and 80% of the maximum are discarded. The two example sets and the OWL background knowledge are passed to DL-Learner as a posNegStandard learning problem with a closed-world reasoner. CELOE searches the space of Description Logic class expressions for those that cover the positive examples while excluding the negative ones. Each run is limited to 60 s. For each neuron-feature-family combination, only the top-ranked class expression, ranked by predictive accuracy, is retained and carried forward to validation. A run for which CELOE returns no solution yields no candidate concept.

Concept and Neuron-Feature Pair Validation

The concepts generated from the 70% analysis split are evaluated using the independent 30% holdout split. Concepts containing only generic classes, such as Thing or Image, and concepts that cannot be parsed into UDAT feature-bin constraints are excluded from statistical validation.

For each processable neuron-feature pair, holdout images are divided into concept-positive and concept-negative groups based on whether they satisfy the induced concept. An image satisfies a concept when its UDAT feature values fall within the bin intervals specified by the corresponding Description Logic expression. For negated constraints, the feature values must fall outside the specified intervals. Images satisfying the concept are assigned to the concept-positive group, while the remaining images form the concept-negative group.

Statistical validation is performed only when both groups contain at least three images. A one-sided Mann-Whitney U test is applied to examine whether neuron activations are significantly higher for concept-positive images than for concept-negative images. The alternative hypothesis is defined as

H1:An+>An−,(6)H_1 : A_n^{+} > A_n^{-}, \tag*{(6)}

where An+A_n^{+} and An−A_n^{-} denote the activation distributions of neuron nn for concept-positive and concept-negative holdout images, respectively. Statistical significance is assessed at α=0.05\alpha= 0.05.

In addition to statistical significance, Cliff’s delta (δ\delta) is used to measure the effect size of the activation difference between the two groups. Effect sizes are categorized as negligible (∣δ∣<0.147|\delta| < 0.147), small (0.147≤∣δ∣<0.3300.147 \leq|\delta| < 0.330), medium (0.330≤∣δ∣<0.4740.330 \leq|\delta| < 0.474), and large (∣δ∣≥0.474|\delta| \geq0.474). A neuron–feature pair is considered validated only if it satisfies both criteria: p<0.05p < 0.05 and ∣δ∣≥0.330|\delta| \geq0.330. This ensures that retained associations exhibit both statistically significant activation differences and at least medium effect-size magnitude.

The validation rate (VR) is calculated as the proportion of statistically tested neuron-feature pairs that satisfy the final validation criteria. For each CNN depth dd and dataset-patch configuration cc, it is defined as

VRd,c=Nd,cvalidatedNd,ctested×100,(7)VR_{d,c} = \frac{N_{d,c}^{\mathrm{validated}}}{N_{d,c}^{\mathrm{tested}}} \times100, \tag*{(7)}

where Nd,cvalidatedN_{d,c}^{\mathrm{validated}} denotes the number of validated neuron-feature pairs and Nd,ctestedN_{d,c}^{\mathrm{tested}} denotes the number of pairs for which the Mann-Whitney U test is completed. Each configuration represents one dataset-patch combination, giving 25 configurations in total. The mean validation rate at depth dd is calculated as

VR‾d=125∑c=125VRd,c.(8)\overline{VR}_{d} = \frac{1}{25}\sum_{c=1}^{25} VR_{d,c}. \tag*{(8)}

This aggregation gives equal weight to each dataset-patch configuration rather than pooling all neuron-feature pairs into a single rate.

For spatial analysis, validation results are aggregated across the analyzed architectures and layer depths for each dataset ss and patch position pp. The corresponding patch-level validation rate is

VRs,p=Ns,pvalidatedNs,ptested×100.(9)VR_{s,p} = \frac{N_{s,p}^{\mathrm{validated}}}{N_{s,p}^{\mathrm{tested}}} \times100. \tag*{(9)}

The same validated-to-tested ratio is used for feature-family analysis, with neuron-feature pairs grouped according to their corresponding UDAT descriptor families. These measures provide the basis for the subsequent comparisons across CNN depths, spatial patch locations, and visual feature families.

Results and Discussion

In this section, we first summarize the overall outcome of the neuron-feature validation pipeline. Across the 5 analyzed datasets and 5 spatial patch positions, the combined analysis comprised 25 dataset-patch configurations. For each configuration, 3 depths ×\times 4 architectures = 12 hidden layers are examined, with 50 selected neurons evaluated for each applicable layer-feature-family combination. Because the availability of UDAT feature families varied across layers, 84 layer-feature-family combinations are represented per configuration, yielding 84 ×\times 50 = 4,200 neuron-feature evaluation cases. Across all 25 configurations, this resulted in a total of 25 ×\times 4,200 = 105,000 evaluation cases. Figure 3 summarizes the progressive reduction of neuron-feature evaluation cases through concept induction, constraint parsing, held-out statistical testing, and final validation. A total of 48,013 neuron-feature pairs satisfied the requirements for statistical testing on the held-out data. Among them, 13,164 pairs met the final validation criteria, corresponding to an overall validation rate of approximately 27.4%. To examine whether this reduction is consistent across experimental settings, Table 5 presents the validation progression separately for all 25 dataset-patch configurations. A progressive decrease can be observed in every configuration, indicating that the overall filtering and validation behavior is broadly distributed rather than dominated by a single dataset or spatial region.

Progressive reduction of neuron-feature evaluation cases

Figure 3. Progressive reduction of neuron-feature evaluation cases. Percentages indicate the proportion retained relative to the initial 105,000 evaluation cases.

DatasetPatchAll CasesConcept FoundConstraints ParsedMW Test CompletedSignificantValidated
CassavaBL42003219235622601200362
CassavaBR42003184231822231213387
CassavaC42003155224121701131433
CassavaTL42003227235122961236487
CassavaTR42003242235023021266495
OpenSprayerBL4200300523672003791468
OpenSprayerBR4200301122821965762405
OpenSprayerC4200308324222036796453
OpenSprayerTL4200301622861937701385
OpenSprayerTR4200300123672048838484
OxfordBL42003081253422481071622
OxfordBR42003039246322161043667
OxfordC4200324326652259824423
OxfordTL42003059249822111262745
OxfordTR42003053244921811074650
SugarcaneBL42002985224820161132895
SugarcaneBR42002946219419611074783
SugarcaneC42003132243821871045650
SugarcaneTL42003016224519161133852
SugarcaneTR42002865217819261098877
WeedCropBL4200331529731169244244
WeedCropBR420030542707993184184
WeedCropC4200302026511303562562
WeedCropTL420029602628944223223
WeedCropTR4200311627301243429428

Table 5. Dataset-patch-wise progression of neuron-feature evaluation cases. Patch locations are denoted as BL (Bottom-Left), BR (Bottom-Right), C (Center), TL (Top-Left), and TR (Top-Right).

The following sections analyze these validated neuron–feature pairs across CNN depth, spatial patch location, feature family, and dataset-level context-bias characteristics to address the research questions.

RQ1 - Emergence Across CNN Depth

As the analysis is performed on the predefined patch regions used to assess contextual dependence, and the induced concepts are derived from visual properties of those same patches, the validated neuron-feature pairs are interpreted as context-sensitive internal representations. Table 6 summarizes the validation rates across different layer depths, including the mean, standard deviation, minimum, and maximum values.

Layer depthConfigurationsMean validation rate (%)SDMin (%)Max (%)
Mid2540.2110.3824.4961.83
Mid-deep2526.9112.8914.4354.02
Deep2511.995.654.4129.07

Table 6. Summary of neuron-feature validation rates across the three analyzed CNN depth levels.

Across the analyzed configurations, the Mid layers exhibit the highest mean validation rate at 40.21%, followed by the Mid-deep layers at 26.91%, while the Deep layers exhibit a substantially lower mean rate of 11.99%. The Mid layers have produced validation rates ranging from approximately 24.49% to 61.83%, with a standard deviation of 10.38. Mid-deep layers range from 14.43% to 54.02%, with a standard deviation of 12.89, whereas Deep layers range from 4.41% to 29.07%, with a standard deviation of 5.65.

The layer-depth heatmap in Figure 4 further shows how this pattern varies across individual datasets and patch locations. In most configurations, the Mid layer has produced the highest validation rate, while the Deep layer has produced the lowest. This tendency is especially pronounced for Sugarcane, where Mid-layer rates reach 61.8% for the bottom-left patch and remain above 46% for all patch locations. In contrast, corresponding Deep-layer rates remain substantially lower. WeedCrop provides another notable case: the center patch reaches 52.4% at the Mid depth and decreases to 42.3% at Mid-deep and 29.1% at Deep.

Heatmap of validated context-sensitive representations by layer depth across Cassava, OpenSprayer, Oxford, and WeedCrop dataset-patch configurations

Figure 4. Validation rates of context-sensitive neuron-feature pairs across different layers for each dataset-patch configuration.

At the same time, the heatmap indicates that the magnitude of the decrease is dataset-dependent. Cassava and OpenSprayer display comparatively low Deep-layer rates across all spatial regions, whereas Sugarcane and WeedCrop retain somewhat stronger validation at deeper levels in selected patches. Nevertheless, the overall ordering remains strongly weighted toward the earlier analyzed depth. These results indicate that semantically interpretable associations with contextual patch properties are most strongly represented at the Mid depth, become less prevalent at Mid-deep layers, and are least frequently validated at the Deep depth. This pattern suggests that contextual visual characteristics are encoded most prominently during intermediate stages of hierarchical feature formation, while deeper representations become increasingly selective and task-oriented. Thus, RQ1 is answered by the consistent concentration of validated context-sensitive neuron representations in Mid-level CNN layers, followed by a progressive reduction through Mid-deep and Deep layers.

Furthermore, Figure 5 reveals substantial variation across datasets rather than a single universally dominant spatial region. For Cassava, validation rates remain relatively similar across the top-left, top-right, and center patches, ranging from 20.0% to 21.5%, while the bottom-left and bottom-right regions show lower rates of 16.0% and 17.4%, respectively. OpenSprayer exhibits a comparatively uniform distribution, with rates ranging from 19.9% to 23.6% across the five patch locations.

Validated context-sensitive representations by patch region

Figure 5. Dataset-wise validation rates of context-sensitive neuron-feature representations across the five predefined spatial patch locations.

Oxford exhibits stronger spatial variation. The top-left patch reaches 33.6%, followed by the bottom-right at 30.1% and top-right at 29.8%, whereas the center patch decreases to 18.7%. Sugarcane has produced the highest overall patch-wise validation levels, with the top-right, top-left, and bottom-left regions reaching approximately 45.5%, 44.5%, and 44.4%, respectively. In contrast, WeedCrop demonstrates a different spatial pattern, where the center patch reaches 43.1%, substantially exceeding the top-left, bottom-left, and bottom-right regions. To examine whether any patch location is consistently dominant across datasets, the validation rates across all 15 dataset-depth observations associated with each patch position are also summarized in Table 7. The mean patch-wise validation rates are approximately 29.53% for top-right, 27.39% for top-left, 25.61% for center, 25.25% for bottom-left, and 24.06% for bottom-right. The relatively close overall means, combined with the dataset-specific patterns visible in Figure 5, indicate that no single spatial patch location consistently dominates across all datasets. Although the top-right and top-left regions exhibit modestly higher overall mean validation rates, the strongest spatial region varies by dataset. Sugarcane shows high validation across several peripheral patches, WeedCrop is dominated by the center region, and Oxford favors the top-left region. This variation suggests that context-sensitive neuron representations are not tied to a fixed image coordinate across datasets. Instead, the spatial distribution of semantically validated representations appears to depend on the visual characteristics and contextual structure associated with each dataset.

Patch positionObservationsMean validation rate (%)SDMin (%)Max (%)
Top-left1527.3916.258.8757.32
Top-right1529.5316.086.7756.20
Center1525.6114.856.9652.43
Bottom-left1525.2515.974.4161.83
Bottom-right1524.0614.776.2350.37

Table 7. Patch-wise summary of neuron-feature validation rates across datasets and analyzed layer depths.

Having established that validated context-sensitive representations are concentrated primarily at intermediate depths but vary spatially across datasets, the next analysis examines which visual feature families most consistently characterize these representations.

RQ2 - Which Feature Families Dominate?

RQ2 examines which visual feature families most consistently dominates in the hidden-neuron representations associated with contextual image patches. For this analysis, the statistically tested neuron-feature pairs are grouped by feature family across the analyzed datasets, patch locations, architectures, and layer depths. Within each feature family, the validation rate is calculated.

The overall analysis reveals a clear dominance of several edge- and texture-related descriptors. As shown in Figure 6, Edge features achieve the highest validation rate at 64.5%, with 1,402 validated pairs among 2,174 statistically tested pairs. Gabor features follow closely at 60.88% with 1,346 of 2,211 tested pairs validated, while Tamura features reach 48.52%, corresponding to 2,881 validated pairs among 5,938 tested cases. These three feature families therefore constitute the strongest overall associations with the validated hidden-neuron representations.

Overall validation outcomes for the analyzed UDAT feature families

Figure 6. Overall validation outcomes for the analyzed UDAT feature families.

A second group of feature families exhibits more moderate validation rates. Fractal features reach 36.47%, followed by Chebyshev at 30.42% and Radon at 25.13%. In contrast, CombFour, Object, and Multiscale features show substantially lower validation rates of 12.00%, 8.51%, and 4.52%, respectively. Haralick features show a validation rate of 36.23%; however, this result is based on only 138 statistically tested pairs from one dataset and one patch location and is therefore can be interpreted separately from the feature families represented across all five datasets and five patch positions. The prominence of Edge, Gabor, and Tamura features indicates that the validated context-sensitive neuron representations are most frequently associated with visual characteristics related to edge organization, directional or frequency-sensitive texture, and perceptual texture structure. Importantly, these results are interpreted at the level of the neuron-feature pair rather than the neuron alone. An individual neuron may therefore be associated with more than one validated feature family. To determine whether the aggregate ranking is driven by only a subset of experimental conditions, the same validation-rate calculation is repeated after grouping the tested neuron-feature pairs separately by dataset and by patch position. The resulting comparisons are shown in Figure 7 and Figure 8.

Feature-family validation rates grouped by dataset across the analyzed patch locations, architectures, and layer depths.

Figure 7. Feature-family validation rates grouped by dataset across the analyzed patch locations, architectures, and layer depths.

Feature-family validation rates grouped by spatial patch position across the analyzed datasets, architectures, and layer depths.

Figure 8. Feature-family validation rates grouped by spatial patch position across the analyzed datasets, architectures, and layer depths.

Across datasets, Edge and Gabor features remain among the strongest families in nearly every case. Edge validation rates range from 60.13% for Cassava to 68.43% for OpenSprayer, while Gabor ranges from 44.10% for Oxford to 87.60% for Sugarcane. Tamura also exhibits relatively high validation in Cassava (44.12%), OpenSprayer (55.16%), Oxford (46.88%), and Sugarcane (59.02%), although its rate decreases to 24.68% for WeedCrop.

The remaining feature families show greater dataset dependence. Fractal features are relatively weak for Cassava (17.09%) and OpenSprayer (19.24%) but become substantially stronger for Oxford (39.29%), Sugarcane (59.51%), and WeedCrop (52.89%). A similar pattern appears for Chebyshev and Radon, both of which reach their strongest validation in Sugarcane at 58.80% and 61.94%, respectively. In contrast, CombFour, Object, and Multiscale remain comparatively low across most datasets. These results indicate that Edge and Gabor provide the most stable high-validation pattern across the five datasets, whereas several secondary feature families exhibit considerably greater dataset-specific variation. The patch-wise analysis in Figure 8 reveals a similarly consistent pattern. Edge features achieve the highest validation rate in four of the five spatial regions, reaching 67.58% in the top-left, 68.44% in the top-right, 74.55% in the center, and 63.84% in the bottom-left patches. In the bottom-right region, Gabor becomes the strongest feature family at 57.63%, compared with 46.57% for Edge. Gabor also remains consistently strong across the other patch positions, ranging from 55.51% to 67.23%, while Tamura maintains rates between approximately 45.29% and 52.63%. Fractal, Chebyshev, and Radon show moderate but more variable behavior across regions, whereas CombFour, Object, and Multiscale remain comparatively weak.

Taken together, the dataset-wise and patch-wise analyses show that the high overall validation rates of Edge and Gabor are not attributable to a single dataset or spatial location. Instead, both families remain consistently prominent across heterogeneous experimental settings, with Tamura pro-

viding an additional but somewhat more variable texture-related signal. Furthermore, to assess how feature-family associations vary with CNN depth, validation rates are computed for each layer-feature combination as validated pairs divided by statistically tested pairs and reported in Figure 9. Combinations with no statistically tested pairs are reported as NA.

Heatmap of validation rates of neuron-feature pairs across feature families and CNN layer depths

Figure 9. Validation rates of neuron-feature pairs across feature families and CNN layer depths.

In the heatmap we can see that several of the strongest feature families are concentrated at the Mid depth. Edge and Gabor are represented only at the Mid depth in the tested data and achieve validation rates of 64.5% and 60.9%, respectively. Tamura is represented across all three depths and exhibits a progressive decline from 55.2% at Mid to 46.6% at Mid-deep and 33.6% at Deep. A similar depth-related decrease is observed for Chebyshev, from 54.4% at Mid to 33.8% at Mid-deep and 18.0% at Deep. Fractal and Haralick are represented at Mid and Mid-deep depths and also show higher validation at Mid, decreasing from 41.5% to 31.7% for Fractal and from 39.7% to 31.7% for Haralick. Radon shows a smaller reduction from 27.0% to 23.2%, while CombFour decreases from 17.5% at Mid to 11.8% at Mid-deep and 7.4% at Deep. Multiscale remains consistently low across all depths, ranging from 3.3% to 6.0%, whereas Object features are represented only at the Deep depth and achieve an 8.5% validation rate.

Overall, the rate-based comparison reinforces the depth pattern identified in RQ1. For feature families represented across multiple depths, validation is generally strongest at the Mid level and decreases toward deeper layers. This suggests that the strongest semantic associations with contextual visual properties are concentrated primarily in intermediate CNN representations, although the degree of decline varies by feature family. Having identified both where context-sensitive representations emerge and which visual feature families characterize them, the next analysis examines how these semantic validation patterns relate to the degree of context bias observed in the prior patch-based study.

RQ3 - Relationship Between Context Bias and Semantic Concept Emergence

RQ3 examines how the emergence of validated semantic concepts relates to the degree of context bias observed across the analyzed datasets. Table 8 represents the neuron-feature validation statistics on Dataset-level bias and above chance patch accuracy. The above-chance patch accuracy represents the model’s behavioral dependence on contextual image regions, whereas the dataset-level validation rate represents the proportion of statistically tested neuron-feature associations that can be semantically characterized and satisfy the final validation criterion. Sugarcane exhibits the highest semantic validation rate at 40.55%, despite having only moderate context bias. In contrast, Cassava and OpenSprayer exhibit high context bias but considerably lower validation rates of 19.23% and 21.97%, respectively. WeedCrop also exhibits high context bias, yet its validation rate reaches 29.03%, remaining below that of Sugarcane. Oxford provides another contrasting case, achieving a validation rate of 27.95% despite showing only minimal above-chance patch performance.

DatasetBias categoryMean patch accuracyRandom chanceAbove-chance patch accuracyTestedValidatedValidation rate
CassavaHigh33.62%20.00%13.62%11,2512,16419.23%
OpenSprayerHigh70.18%50.00%20.18%9,9892,19521.97%
OxfordLow1.70%0.98%0.72%11,1153,10727.95%
SugarcaneModerate16.92%9.1%7.83%10,0064,05740.55%
WeedCropHigh91.06%50.00%41.06%5,6521,64129.03%

Table 8. Dataset-level bias and neuron-feature validation statistics.

The dataset rankings differ between context-bias severity and semantic concept emergence. Stronger dependence on contextual regions does not necessarily lead to more semantically validated neuron-feature associations. Similarly, datasets with moderate or low context bias can still show relatively high semantic concept emergence. Overall, RQ3 shows that stronger context bias does not necessarily correspond to greater semantic concept emergence in the analyzed hidden representations.

These results indicate that behavioral context dependence and semantic concept emergence capture distinct aspects of CNN behavior. High patch-based classification performance reflects the extent to which a model can exploit information contained within restricted image regions, whereas the validation rate reflects the extent to which hidden-neuron activations can be associated with the visual feature concepts represented in the constructed knowledge graph. Consequently, a dataset may exhibit strong contextual dependence without producing a proportionally larger number of semantically validated neuron–feature associations.

Conclusion

Previous studies have demonstrated that CNNs can achieve above-chance classification accuracy using image patches containing little or no task-relevant content, revealing potential dependence on contextual information. Building on these findings, this study investigates how such information is represented within CNN hidden layers. We employ a neuro-symbolic approach that combines hidden-neuron activation analysis, UDAT image descriptors, OWL knowledge graphs, and Description Logic Concept Induction to identify interpretable associations between neuron activations and visual characteristics of contextual image patches. Across five datasets, five patch locations, four CNN architectures, and three network depths, the analysis evaluates 105,000 neuron-feature-family combinations, of which 77,027 produce candidate concepts, 48,013 undergo statistical testing, and 13,164 satisfy the validation criteria.

The findings reveal several patterns in the internal representations of CNNs. Mid layers exhibit the highest mean validation rate (40.2%), followed by mid-deep (26.9%) and deep layers (12.0%), indicating that the evaluated image descriptors characterize mid-layer activations more consistently. Spatial analysis reveals no universally dominant patch location, as the regions producing the highest validation rates vary across datasets. Among the evaluated feature families, Edge, Gabor, and Tamura achieve the strongest and most consistent validation results across datasets and patch locations, suggesting that edge structure and texture-related characteristics are particularly prominent in the representations associated with contextual image patches. However, the degree of concept validation does not directly correspond to the severity of context bias observed in the previous patch-based study. Sugarcane, which exhibits moderate context bias, achieves the highest overall validation rate (40.5%), whereas highly biased Cassava achieves the lowest (19.2%).

Together, these findings extend patch-based context-bias analysis from observing classification behavior to examining the visual characteristics associated with hidden-neuron activations. They also suggest that strong contextual dependence does not necessarily imply a greater proportion of statistically validated neuron–feature associations.

Several limitations affect how far these findings reach. The feature families were assigned to layers by an LLM-guided assessment rather than crossed with all layers. Edge and Gabor were evaluated only at mid layers, so their high validation rates cannot be separated from the depth effect, and only Tamura, among the strongest families, was tested at all three depths. Zernike moments yielded no testable concepts, and Haralick features were tested on a single dataset and patch location. While the proposed analysis provides interpretable evidence of internal representations, these associations alone do not establish which features causally influence classification decisions. Future work can investigate this relationship through controlled feature interventions and examine whether modifying or removing identified contextual characteristics changes model predictions, thereby supporting a more direct assessment of their contribution to context bias.

Code and Data Availability

The code and data associated with this study will be made publicly available upon publication at https://github.com/Samatha1994/XAI_Understanding_Bias_Project/.

LLM-Guided Feature-Layer Assignment

The feature-layer assessment described in § 4.4 was conducted using Claude Sonnet (Anthropic, 2026). For each architecture, the model was provided with the architecture name, the selected layer names, and the 13 UDAT feature family names, and queried as follows:

For the following layers of [ARCHITECTURE]: [LAYER NAMES] and the following feature families: Edge Statistics, Combined First Four Moments, Chebyshev-Fourier Transform, Multiscale Histogram, Tamura Texture, Haralick Texture, Gabor Filters, Zernike Moments, Object Statistics, Radon Transform, Color Histogram, Fractal Features, and Intensity Statistics — which features should be tested for each layer, and which should be skipped?

This prompt was issued separately for each of the four architectures, with layer names substituted accordingly. The model returned a classification of each layer–feature-family combination as test, optional/borderline, or skip, along with theoretical justification for each decision based on known properties of convolutional feature hierarchies.

References

  1. [1]Samantha Ereshi Akkamahadevi, Abhilekha Dalal, and Pascal Hitzler. Automating cnn neuron interpretation using concept induction. In ISWC (Posters/Demos/Industry), 2024.
  2. [2]Samantha Ereshi Akkamahadevi, Abhilekha Dalal, and Pascal Hitzler. Evaluation of concept induction in explainable AI using multiple datasets. In Proceedings of the AAAI Symposium Series, volume 5, 2025.DOI
  3. [3]Ananthu Aniraj, Cassio F. Dantas, Dino Ienco, and Diego Marcos. Masking strategies for background bias removal in computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 4454–4463, 2023.arxiv.org/abs/2308.12127
  4. [4]Gavin Armstrong. Open sprayer images, 2023. Accessed: June 2025.
  5. [5]David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6541–6549, 2017.arxiv.org/abs/1704.05796
  6. [6]Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pages 456–473, 2018.
  7. [7]Lorenz Bü hmann, Jens Lehmann, and Patrick Westphal. DL-Learner—a framework for inductive learning on the Semantic Web. Journal of Web Semantics, 39:15–24, 2016.
  8. [8]Abhilekha Dalal. Understanding Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning. Kansas State University, 2024.
  9. [9]Abhilekha Dalal, Michael Okonoda, Eder Martinez, and Lior Shamir. Evaluating contextual bias in cnn image classification: Evidence from agricultural benchmark datasets. arXiv preprint arXiv:2609.14654, 2026.arxiv.org/abs/2609.14654
  10. [10]Abhilekha Dalal, Rushruk h Rayan, Adrita Barua, Samatha Ereshi Akkamahadevi, Avishek Das, Cara Widmer, Eugene Y Vasserman, Md Kamruzzaman Sarker, and Pascal Hitzler. Toward a neurosymbolic understanding of hidden neuron activations. Neurosymbolic Artificial Intelligence, 2:29498732261419302, 2026.DOI
  11. [11]Abhilekha Dalal, Rushrukh Rayan, Adrita Barua, Eugene Y Vasserman, Md Kamruzzaman Sarker, and Pascal Hitzler. On the value of labeled data and symbolic methods for hidden neuron activation analysis. In International Conference on Neural-Symbolic Learning and Reasoning, pages 109–131. Springer, 2024.arxiv.org/abs/2404.13567
  12. [12]Manuel de Sousa Ribeiro and João Leite. Aligning artificial neural networks and ontologies towards explainable AI. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
  13. [13]Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3:610–619, 2021.DOI
  14. [14]Sanchari Dhar and Lior Shamir. Evaluation of the benchmark datasets for testing the efficacy of deep convolutional neural networks. Visual Informatics, 2021.DOI
  15. [15]ErnestMwebaze, Jesse Mostipak, Joyce, Julia Elliott, and Sohier Dane. Cassava leaf disease classification. https://kaggle.com/competitions/cassava-leaf-disease-classification, 2020. Kaggle.
  16. [16]Sai Teja Erukude, Akhil Joshi, and Lior Shamir. Identifying bias in deep neural networks using image transforms. Computers, 13(12):341, 2024.DOI
  17. [17]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.DOI
  18. [18]Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.arxiv.org/abs/1811.12231
  19. [19]Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.arxiv.org/abs/1902.03129
  20. [20]Robert M. Haralick, K. Shanmugam, and Its’hak Dinstein. Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics, SMC-3(6):610–621, 1973.DOI
  21. [21]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.arxiv.org/abs/1512.03385
  22. [22]Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2201.11114
  23. [23]Pascal Hitzler. A review of the semantic web field. Commun. ACM, 64(2):76–83, 2021.DOI
  24. [24]Pascal Hitzler, Federico Bianchi, Monireh Ebrahimi, and Md Kamruzzaman Sarker. Neural-symbolic integration and the Semantic Web. Semantic Web, 11(1):3–11, 2020.DOI
  25. [25]Pascal Hitzler, Markus Krötzsch, and Sebastian Rudolph. Foundations of Semantic Web Technologies. Chapman and Hall/CRC Press, 2010.
  26. [26]Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.arxiv.org/abs/1905.02175
  27. [27]Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 2668–2677, 2018.arxiv.org/abs/1711.11279
  28. [28]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097–1105, 2012.
  29. [29]Sebastian Lapuschkin, Stephan Wä​ldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10:1096, 2019.
  30. [30]Phuong Quynh Le, Jörg Schlötterer, and Christin Seifert. An xai-based analysis of shortcut learning in neural networks. In World Conference on Explainable Artificial Intelligence, pages 424–445. Springer, 2025.arxiv.org/abs/2504.15664
  31. [31]Jae Hee Lee, Sergio Lanza, and Stefan Wermter. From neural activations to concepts: A survey on explaining concepts in neural networks. Neurosymbolic Artificial Intelligence, 2024.arxiv.org/abs/2310.11884
  32. [32]Jens Lehmann. DL-Learner: Learning concepts in description logics. Journal of Machine Learning Research, 10:2639–2642, 2009.DOI
  33. [33]Jens Lehmann, Sören Auer, Lorenz Bühmann, and Sebastian Tramp. Class expression learning for ontology engineering. Journal of Web Semantics, 9(1):71–81, 2011.DOI
  34. [34]Jens Lehmann and Pascal Hitzler. Concept learning in description logics using refinement operators. Mach. Learn., 78(1-2):203–250, 2010.DOI
  35. [35]Cathy Ong Ly, Balagopal Unnikrishnan, Tony Tadic, Tirth Patel, Joe Duhamel, Sonja Kandel, Yasbanoo Moayedi, Michael Brudno, Andrew Hope, Heather Ross, and Chris McIntosh. Short-cut learning in medical ai hinders generalization: method for estimating ai model generalization without external data. npj Digital Medicine, 7(1):124, 2024.
  36. [36]B. S. Manjunath and W. Y. Ma. Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):837–842, 1996.DOI
  37. [37]Eder Martinez, Michael Okonoda, Abhilekha Dalal, and Lior Shamir. Detection and profiling of dataset bias in cnn analysis of images. Computing in Science & Engineering, 2026.DOI
  38. [38]Ian Model and Lior Shamir. Comparison of data set bias in object recognition benchmarks. IEEE Access, 3, 2015.DOI
  39. [39]Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7, Sep 2016.arxiv.org/abs/1604.03169
  40. [40]Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.arxiv.org/abs/2006.14032
  41. [41]Loris Nanni, Stefano Ghidoni, and Sheryl Brahnam. Handcrafted vs. non-handcrafted features for computer vision classification. Pattern recognition, 71:158–172, 2017.DOI
  42. [42]Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008.DOI
  43. [43]Mehmet Alican Noyan. Uncovering bias in the PlantVillage dataset. arXiv preprint arXiv:2206.04374, 2022.arxiv.org/abs/2206.04374
  44. [44]Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2204.10965
  45. [45]Timo Ojala, Matti Pietikäinen, and Topi Mäenpää. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971–987, 2002.
  46. [46]Michael Okonoda, Eder Martinez, Abhilekha Dalal, and Lior Shamir. Unmasking biases and reliability concerns in convolutional neural networks analysis of cancer pathology images. Electronics, 15(6), 2026.DOI
  47. [47]Alex Olsen, Dmitry A. Konovalov, Bronson Philippa, Peter Ridd, Jake C. Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, Brendan Calvert, Mostafa Rahimi Azghadi, and Ronald D. White. Deepweeds: A multiclass weed species image dataset for deep learning. Scientific Reports, 9(2058), 2 2019.
  48. [48]Nikita Orlov, Lior Shamir, Tomasz Macura, Josiah Johnston, D. Mark Eckley, and Ilya G. Goldberg. WND-CHARM: Multi-purpose image classification using compound image transforms. Pattern Recognition Letters, 29(11):1684–1693, 2008.pubmed.ncbi.nlm.nih.gov/18958301
  49. [49]Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  50. [50]Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ” why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.arxiv.org/abs/1602.04938
  51. [51]Md Kamruzzaman Sarker and Pascal Hitzler. Efficient concept induction for description logics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3036–3043, 2019.arxiv.org/abs/1812.03243
  52. [52]Md Kamruzzaman Sarker, Ning Xie, Derek Doran, Michael Raymer, and Pascal Hitzler. Explaining trained neural networks with Semantic Web technologies: First steps. arXiv preprint arXiv:1710.04324, 2017.arxiv.org/abs/1710.04324
  53. [53]Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, and Pascal Hitzler. A case study on concept induction for neuron-level interpretability in cnn. arXiv preprint arXiv:2603.00197, 2026.arxiv.org/abs/2603.00197
  54. [54]Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2:476–486, 2020.arxiv.org/abs/2001.05371
  55. [55]Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
  56. [56]Atefeh Shahroudnejad. A survey on understanding, visualizations, and explanation of deep neural networks. arXiv preprint arXiv:2102.01792, 2021.arxiv.org/abs/2102.01792
  57. [57]Lior Shamir. Evaluation of face datasets as tools for assessing the performance of face recognition methods. International Journal of Computer Vision, 79(3):225–230, 2008.pubmed.ncbi.nlm.nih.gov/18776952
  58. [58]Lior Shamir. Udat: A multi-purpose data analysis tool. Astrophysics Source Code Library, pages ascl–1704, 2017.
  59. [59]Lior Shamir, Nikita Orlov, D. Mark Eckley, Tomasz Macura, Josiah Johnston, and Ilya G. Goldberg. Wndchrm – an open source utility for biological image analysis. Source Code for Biology and Medicine, 3:13, 2008.DOI
  60. [60]Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk–quantifying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8218–8226, 2019.arxiv.org/abs/1812.06707
  61. [61]Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2014.arxiv.org/abs/1312.6034
  62. [62]Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. Plantdoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 249–253. Association for Computing Machinery, 2020.DOI
  63. [63]Kaspars Sudars, Janis Jasko, Ivars Namatevs, Liva Ozola, and Niks Badaukis. Dataset of annotated food crops and weed images for robotic computer vision control. Data in brief, 31:105833, 2020.DOI
  64. [64]Hideyuki Tamura, Shunji Mori, and Takashi Yamawaki. Textural features corresponding to visual perception. IEEE Transactions on Systems, Man, and Cybernetics, 8(6):460–473, 1978.
  65. [65]Sandip Thite, Yogesh Suryawanshi, Kailas Patil, and Prawit Chumchu. Sugarcane leaf dataset: A dataset for disease detection and classification for machine learning applications. Data in Brief, 53:110268, 2024.
  66. [66]Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1521–1528, 2011.DOI
  67. [67]Cara Leigh Widmer, Md Kamruzzaman Sarker, Srikanth Nadella, Joshua Fiechter, Ion Juvina, Brandon Minnery, Pascal Hitzler, Joshua Schwartz, and Michael Raymer. Towards human-compatible XAI: Explaining data differentials with concept induction over background knowledge. Journal of Web Semantics, 79:100807, 2023.arxiv.org/abs/2209.13710
  68. [68]Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.arxiv.org/abs/2006.09994
  69. [69]Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015.arxiv.org/abs/1506.06579
  70. [70]John R Zech, Marcus A Badgley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e1002683, 2018.DOI
  71. [71]Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 818–833, 2014.arxiv.org/abs/1311.2901
  72. [72]Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2016.arxiv.org/abs/1512.04150

Paper details

vlnc:W_jghk9hj8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents