betaFeedback

WorksComputer Science

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Abstract

Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and cache-based adapters, typically supervise global image embeddings and therefore underuse token-level structure. Conversely, structure-aware methods exploit local evidence but often introduce additional inference-time computation. We propose SD-LoRA, a structural distillation framework that uses token-level graph reasoning only during training and transfers its effects into a lightweight LoRA-adapted CLIP model. During training, a heterogeneous graph teacher models patch-patch, global-local, and visual-text relations over high-resolution ViT tokens. Rather than retaining this graph at test time, SD-LoRA transfers its structural bias into the shared CLIP-LoRA backbone through implicit distillation and Prototype Predictive Alignment, which aligns graph-conditioned representations with support-derived class prototypes. At inference, the graph teacher is discarded; prediction requires only a single CLIP-LoRA forward pass and a cache-based classifier. Across 11 few-shot benchmarks, SD-LoRA improves over a strong CLIP-LoRA baseline by 2.4/1.8/3.0/2.4/2.0 percentage points in the 1/2/4/8/16-shot settings, respectively, without additional graph computation at test time. Results on base-to-novel generalization, cross-dataset transfer, out-of-distribution robustness, and calibration further show that training-time structural supervision improves both accuracy and transfer reliability. Code is available at https://github.com/MR-Sherif/SD-LoRA.git.

Introduction

Vision–language models (VLMs) such as CLIP Radford et al. [ [28] and ALIGN Jia et al. [ [13] are now standard foundations for open-vocabulary and few-shot recognition. Their strength comes from large-scale image–text pretraining, which produces transferable global representations and strong zero-shot performance. However, this same global representation that makes CLIP efficient can become a limitation when only a few labeled examples are available. In many few-shot tasks, especially fine-grained recognition, class distinctions depend not just on coarse semantic alignment but on subtle local cues such as texture, part shape, object configuration, or small discriminative regions.

Preprint.

Recent CLIP adaptation methods advance this setting along several axes. Prompt-learning adapts CLIP’s text interface Zhou et al. [ [45], Khattak et al. [ [15], distillation-based prompts improve transfer from stronger teachers Li et al. [ [22], and base-to-novel approaches explicitly manage the tradeoff between fitting base classes and preserving novel-class generalization Li et al. [2025]. In parallel, adapter and cache-based methods such as CLIP-Adapter and Tip-Adapter Gao et al. [ [8], Zhang et al. [ [40], along with recent kernel views of cache adaptation Bendou et al. [ [2], provide lightweight alternatives to full backbone fine-tuning. LoRA-based adaptation likewise achieves parameter efficiency by updating only low-rank components Hu et al. [ [12], Zanella and Ben Ayed [ [37]. Most methods still use a single global image embedding for supervision or retrieval, which improves efficiency but does not explicitly model interactions among local visual tokens or between local tokens and class semantics.

This creates a pooled-supervision bottleneck: the adapted model maps an image to the right coarse region of CLIP’s embedding space, but neglects local token interactions needed to separate fine-grained classes. Although CLIP’s ViT backbone encodes dense token-level information, standard adaptation collapses these tokens into a pooled feature before supervision, weakening or discarding cues such as part shape, texture, and spatial layout. Region-level alignment, dense prediction, multi-crop reasoning, or graph structure Rao et al. [ [29], Zhong et al. [ [43], Li et al. [ [20], Ahmad et al. [ [1] can recover such cues, but usually require extra detectors, dense computation, multiple crops, or graph modules at inference. Consequently, existing methods fall into two regimes: lightweight approaches that adapt global representations but underuse structure, and structure-aware approaches that better exploit local evidence but increase deployment cost.

This paper asks a simple question: can token-level structural reasoning improve few-shot CLIP adaptation without being used at test time? We answer this via proposing SD-LoRA, a structural distillation framework for few-shot vision–language adaptation. SD-LoRA uses graph reasoning only as training-time supervision, not for inference. During training, it builds a heterogeneous graph over high-resolution visual tokens and class-text embeddings, modeling patch–patch, global–local, and visual–text relations. The teacher is optimized to reveal a discriminative local structure that is never deployed. Its structural signal is distilled into a shared CLIP-LoRA backbone, so the final model is just a lightweight LoRA-adapted CLIP encoder with a cache-based classifier.

The core idea is implicit structural distillation. Instead of conventional logit-based distillation Hinton et al. [ [11], SD-LoRA transfers structure via shared-backbone optimization: the graph teacher and the deployed student use the same LoRA-adapted visual backbone, so gradients from the graph objective directly shape the low-rank updates used in inference. To stabilize this in the low-shot regime, we introduce Prototype Predictive Alignment (PPA), aligning graph-conditioned representations with support-based class prototypes, pushing the LoRA-updated features to form compact, structure-aware class regions instead of overfitting to individual support examples. Unlike recent transductive/test-time adaptation methods that rely on unlabeled test batches or update the model during deployment Zanella et al. [ [39] (2024), Yoon et al. [ [35], we target the inductive few-shot setting: after training in the support set, it performs single-image inference, with no test-time graph construction, unlabeled-batch assumptions or online adaptation.

Contributions: (1) We propose SD-LoRA, a training-time structural distillation framework that injects token-level graph reasoning into a CLIP-LoRA backbone while keeping inference as lightweight LoRA+cache adaptation. (2) We introduce a heterogeneous graph teacher over ViT tokens and class-text embeddings, enabling patch–patch, global–local, and visual–text relational supervision without test-time graph computation. (3) We introduce Prototype Predictive Alignment as a complementary regularizer of graph-conditioned supervision, boosting few-shot accuracy, base-to-novel generalization, cross-dataset transfer, and OOD robustness in the same lightweight inference pathway.

Related Work

Global adaptation of CLIP representations. Most few-shot CLIP adaptation methods tweak text prompts, image features, or the classifier while keeping the pretrained model fixed. Prompt-learning approaches like CoOp and CoCoOp Zhou et al. [ [45] replace hand-crafted templates with learnable context tokens, and MaPLe Khattak et al. [ [15] extends prompting to both visual and textual branches. Later work enhances generalization via regularization, prompt distillation, or base-to-novel balancing, including PromptSRC Khattak et al. [ [16], CoPrompt Roy and Etemad [ [30], PromptKD Li et al. [ [22], and DPC Li et al. [2025]. Adapter- and cache-based methods instead modify the visual representation or decision rule: CLIP-Adapter Gao et al. [ [8] adds residual adapters, Tip-Adapter Zhang et al. [ [40] builds a support-set key–value cache, and ProKeR Bendou et al. [ [2] interprets cache adaptation as kernel regularization. These methods provide strong lightweight adaptation but operate mostly on image-level embeddings, prompt logits, or support-set similarities. SD-LoRA is complementary: it remains lightweight but changes the training signal by adding token-level structural supervision.

LoRA and parameter-efficient VLM adaptation. Parameter-efficient fine-tuning updates only a small subset of parameters, keeping most of the pretrained model fixed. LoRA Hu et al. [ [12] does this via low-rank updates to selected linear layers, and CLIP-LoRA Zanella and Ben Ayed [ [37] shows it is a strong few-shot CLIP baseline. Robustness-focused extensions like AdvCLIP-LoRA Ghiasvand et al. [ [9] study LoRA under distribution shift. SD-LoRA uses LoRA differently: LoRA is not the main novelty but the target of structural supervision. The graph teacher and student share the same LoRA-adapted backbone, so teacher-side relational losses directly shape the low-rank updates that will later be used without the teacher, unlike standard LoRA, where low-rank parameters are trained mainly from global classification losses.

Region-, token-, and graph-level reasoning. Another research line augments CLIP with local-ized or structured visual evidence. DenseCLIP Rao et al. [ [29] targets dense prediction, while RegionCLIP Zhong et al. [ [43] and GLIP-style models Li et al. [ [20] align text with object regions. Patch- and crop-based methods further use localized views to refine predictions Ahmad et al. [ [1]. Graph-based approaches such as HeGraphAdapter Zhao et al. [ [42] and GraphAdapter Li et al. [ [21] encode relations among visual, textual, or class-level nodes. TOGA Mohammad et al. [2026] uses a training-only heterogeneous image–patch–text graph teacher to supervise a cache-based adapter while keeping CLIP frozen. SD-LoRA instead transfers structural supervision into shared LoRA parameters using high-resolution ViT tokens, while likewise discarding the graph teacher at inference.

Distillation, transduction, and test-time adaptation. Knowledge distillation transfers information from a teacher model Hinton et al. [ [11]. In VLM adaptation, PromptKD Li et al. [ [22] distills prompt knowledge from a larger CLIP teacher. SD-LoRA differs by distilling structure rather than only predictions: the teacher is a token-level graph reasoner, and the transfer occurs implicitly through shared LoRA parameters and Prototype Predictive Alignment. Our setting is also distinct from transductive or test-time adaptation methods such as TransCLIP Zanella et al. [ [38], C-TPT Yoon et al. [ [35], and StatA Zanella et al. [ [39], which exploit unlabeled test batches or update the model during deployment. SD-LoRA considers the standard inductive few-shot protocol: after support-set adaptation, each test image is processed independently with a fixed LoRA+cache model. Following recent concerns about standard CLIP few-shot evaluation Kravets et al. [ [18], we therefore report not only few-shot accuracy but also base-to-novel generalization, cross-dataset transfer, out-of-distribution (OOD) robustness, and calibration.

Figure 2. SD-LoRA framework. It trains a deployable CLIP-LoRA+cache student sθ,ηs_{\theta,\eta} using an auxiliary structural teacher gθ,ψg_{\theta,\psi} present only during adaptation. Teacher and student share the LoRA backbone, but the teacher reasons over high-resolution ViT tokens and class-text embeddings with a relation-aware graph transformer. Structural supervision from the graph branch plus Prototype Predictive Alignment updates the shared LoRA parameters θ\theta. At inference, the teacher parameters ψ\psi and graph computation are discarded, and only the lightweight student sθ,ηs_{\theta,\eta} makes predictions.

Structural Distillation into Low-Rank Adaptation

We formulate SD-LoRA as learning a deployable low-rank predictor within a richer training-time hypothesis class. The inference model sθ,ηs_{\theta,\eta} combines a LoRA-adapted CLIP image encoder with parameters θ\theta and a cache head with parameters η\eta. During training, we introduce a teacher gθ,ψg_{\theta,\psi} that shares the LoRA backbone but incorporates graph-reasoning parameters ψ\psi over visual tokens and class-text embeddings. Student and teacher share θ\theta: student losses optimize deployment-time discrimination, while teacher losses enforce structural constraints on the representation that persist at inference. After training, we discard ψ\psi and use only sθ,ηs_{\theta,\eta} for prediction.

Deployable Student and Training-Time Teacher

We consider CC-way KK-shot few-shot classification problem with support set S={(xi,yi)}i=1CKS = \{(x_i, y_i)\}_{i=1}^{CK}, where each of the CC classes has KK labeled support examples from the training split following standard few-shot CLIP adaptation protocols Zhou et al. [ [45], Zhang et al. [ [40], Zanella and Ben Ayed [2024]. For each class cc, we construct a prompt τc\tau_c and compute the corresponding CLIP text embedding tc=ft(τc)t_c = f_t(\tau_c), giving the class-text set T={tc}c=1CT = \{t_c\}_{c=1}^{C}.

The deployed student sθ,ηs_{\theta,\eta} uses a LoRA-adapted CLIP image encoder. Starting from pretrained CLIP Radford et al. [2021], the original image encoder weights are frozen and only low-rank updates are trained. For a pretrained weight matrix W0W_0, LoRA defines W=W0+ΔWW = W_0 + \Delta W, ΔW=BA\Delta W = BA, with AA and BB low-rank trainable matrices. Let fvθ:X→Rdf_v^{\theta}: \mathcal{X} \to\mathbb{R}^d be the pooled image embedding from the LoRA-adapted CLIP visual encoder, and define the normalized feature zθ(x)=fvθ(x)/∥fvθ(x)∥2z_{\theta}(x) = f_v^{\theta}(x) / \lVert f_v^{\theta}(x) \rVert_2. The zero-shot CLIP logits are LZS(x)=sim⁡(zθ(x),T)L_{\mathrm{ZS}}(x) = \operatorname{sim}(z_{\theta}(x), T), the cosine similarity between the image feature and all class-text embeddings. To use the few-shot support set at deployment, the student adds a cache-based head: support features give cache keys aicache=zθ(xi)a_i^{\mathrm{cache}} = z_{\theta}(x_i), which form the rows of Acache∈RCK×dA_{\mathrm{cache}} \in\mathbb{R}^{CK \times d} for CKCK support examples (KK shots per class), and the corresponding one-hot labels give cache values V∈RCK×CV \in\mathbb{R}^{CK \times C}. Given a query feature zθ(x)z_{\theta}(x), the cache head computes support affinities aη(x)=exp⁡[−β(1−Aη(zθ(x),Acache))]a_{\eta}(x) = \exp[-\beta(1 - \mathcal{A}_{\eta}(z_{\theta}(x), A_{\mathrm{cache}}))], where Aη\mathcal{A}_{\eta} is a learnable cache matcher, β\beta controls affinity sharpness, and aη(x)∈RCKa_{\eta}(x) \in\mathbb{R}^{CK} gives one score per support example. Multiplying these affinities by the one-hot cache values VV yields cache logits over the CC classes:

LCache(x)=aη(x)TV,sθ,η(x)=LZS(x)+ωLCache(x),(1)L_{\mathrm{Cache}}(x) = a_{\eta}(x)^{\mathsf{T}} V, \qquad s_{\theta,\eta}(x) = L_{\mathrm{ZS}}(x) + \omega L_{\mathrm{Cache}}(x), \tag*{(1)}

where LCache(x)∈RCL_{\mathrm{Cache}}(x) \in\mathbb{R}^{C} is the cache-branch logit vector, sθ,η(x)∈RCs_{\theta,\eta}(x) \in\mathbb{R}^{C} is the final deployed student logits, and ω\omega controls the cache contribution. Eq. (1) defines the inference-time student model.

During training, we introduce a structural teacher gθ,ψg_{\theta,\psi} that, unlike the student operating on the pooled image representation zθ(x)z_{\theta}(x), processes the full token sequence from the same LoRA-adapted visual backbone, Uθ(x)={ucls,u1,…,uM}U_{\theta}(x) = \{u_{\mathrm{cls}}, u_1, \ldots, u_M\}, consisting of the [CLS] token uclsu_{\mathrm{cls}} and MM patch tokens. The teacher fuses these visual tokens with the class-text embeddings TT via graph reasoning to obtain a structure-aware representation hgh_g and graph logits LGraphL_{\mathrm{Graph}}. Thus,

gθ,ψ(x,T)=(hg,LGraph).(2)g_{\theta,\psi}(x,T) = (h_g, L_{\mathrm{Graph}}). \tag*{(2)}

The important point is that both sθ,ηs_{\theta,\eta} and gθ,ψg_{\theta,\psi} depend on the same LoRA parameters θ\theta. Consequently, structural supervision applied to the teacher updates the representation used by the deployed student.

Relation-Aware Structural Supervision

The role of gθ,ψg_{\theta,\psi} is to define a richer training-time hypothesis class that captures local token interactions and class semantics. To obtain fine-grained evidence, we extract ViT tokens at higher spatial resolution and interpolate the pretrained positional embeddings to the new token grid. Let eclse_{\mathrm{cls}} be the pretrained positional embedding of the [CLS] token, and let EpospatchE_{\mathrm{pos}}^{\mathrm{patch}} be the pretrained patch positional embeddings. The adapted positional encoding is

Epos′=Concat⁡(ecls,Resize⁡bicubic(Epospatch,(h′,w′))),(3)E_{\mathrm{pos}}^{\prime} = \operatorname{Concat}\left(e_{\mathrm{cls}}, \operatorname{Resize}_{\mathrm{bicubic}}\left(E_{\mathrm{pos}}^{\mathrm{patch}}, (h^{\prime}, w^{\prime})\right)\right), \tag*{(3)}

where (h′,w′)(h^{\prime}, w^{\prime}) is the new token grid size. This yields denser visual tokens in a single backbone forward pass, without multi-crop inference or external region proposals. Given the visual tokens Uθ(x)U_{\theta}(x) and text embeddings TT, the teacher builds a heterogeneous graph

G=(N,E),N=Nv∪Nt,Nv={ucls,u1,…,uM},(4)\mathcal{G} = (\mathcal{N}, \mathcal{E}), \qquad\mathcal{N} = \mathcal{N}_{v} \cup\mathcal{N}_{t}, \qquad\mathcal{N}_{v} = \{u_{\mathrm{cls}}, u_1, \ldots, u_M\}, \tag*{(4)}

Nv\mathcal{N}_{v} is the set of visual nodes and Nt=T\mathcal{N}_{t} = T the class-text nodes. The edge set includes three relation types: patch–patch spatial edges between neighboring visual tokens, global–local edges between the CLS and patch tokens, and visual–text semantic edges between visual tokens and class-text nodes. These relations let the teacher jointly capture local visual consistency, image-level context, and class-conditioned token alignment in a single structural representation.

Patch–patch edges use 8-connected adjacency on the 2D ViT token grid: each patch token connects to horizontal, vertical, and diagonal neighbors within one grid step, and boundary tokens connect only to in-grid neighbors. Global–local edges are bidirectional between the [CLS] node and every patch node. Visual–text edges are bidirectional between visual nodes and class-text nodes. The graph is updated by relation-aware message passing. Let ϕ(i)∈{vis,text}\phi(i) \in\{\mathrm{vis}, \mathrm{text}\} be the type of node ii, and let r=rel⁡(j,i)r = \operatorname{rel}(j,i) be the relation type of edge j→ij \to i. Let fi(l)∈Rdf_{i}^{(l)} \in\mathbb{R}^{d} be the hidden state of node i∈Ni \in\mathcal{N} at layer ll, with fi(0)f_{i}^{(0)} initialized from the corresponding visual-token or class-text embedding. The neighborhood of node ii is defined by its incoming edges, Γ(i)={j∣(j,i)∈E}\Gamma(i) = \{j \mid(j,i) \in\mathcal{E}\}. In our heterogeneous graph, Γ(i)\Gamma(i) may contain spatially adjacent patch nodes (patch–patch relations), the [CLS] node or patch nodes (global–local relations), and connected class-text nodes (visual–text semantic relations). Attention normalization is thus performed over neighbors of the target node, with the edge type r=rel⁡(j,i)r = \operatorname{rel}(j,i) determining the relation-specific bias and message transformation.

For attention head hh, we construct a query vector qi(h)\mathbf{q}_{i}^{(h)}, a relation-conditioned key vector kj→i(h)\mathbf{k}_{j \to i}^{(h)}, and a relation-conditioned value vector vj→i(h)\mathbf{v}_{j \to i}^{(h)} in a dhd_h-dimensional space, where dh=d/Hd_h = d/\mathcal{H} for H\mathcal{H} heads. The unnormalized attention score ej→i(h)e_{j \to i}^{(h)}, normalized attention weight αj→i(h)\alpha_{j \to i}^{(h)}, and aggregated message mi(h)m_i^{(h)} are then computed as

ej→i(h)=(qi(h))Tkj→i(h)dh+br(h),αj→i(h)=ρj∈Γ(i)(ej→i(h)),mi(h)=∑j∈Γ(i)μr(h)αj→i(h)vj→i(h).(5)e_{j \to i}^{(h)} = \frac{\left(\mathbf{q}_{i}^{(h)}\right)^{\mathsf{T}}\mathbf{k}_{j \to i}^{(h)}}{\sqrt{d_h}} + b_{r}^{(h)}, \qquad\alpha_{j \to i}^{(h)} = \rho_{j \in\Gamma(i)}\left(e_{j \to i}^{(h)}\right), \qquad m_i^{(h)} = \sum_{j \in\Gamma(i)} \mu_{r}^{(h)}\alpha_{j \to i}^{(h)}\mathbf{v}_{j \to i}^{(h)}. \tag*{(5)}

Here, ej→i(h)e_{j \to i}^{(h)} is the relation-aware compatibility score, ρ\rho is the softmax over neighbors Γ(i)\Gamma(i), br(h)b_{r}^{(h)} is the relation-dependent attention bias, and μr(h)\mu_{r}^{(h)} is the relation-specific message scaling factor. The concatenated multi-head message is projected back to Rd\mathbb{R}^{d} by a type-specific learnable output matrix WO(ϕ(i))∈RHdh×dW_{O}^{(\phi(i))} \in\mathbb{R}^{H d_h \times d}. We use separate output projections for visual and text nodes, and optimize these matrices as part of the graph-teacher parameters ψ\psi, giving

fi(l+1)=LN⁡(fi(l)+FFN⁡(ϕ(i))(Concat⁡h=1H(mi(h)WO(ϕ(i))))),(6)f_i^{(l+1)} = \operatorname{LN}\left(f_i^{(l)} + \operatorname{FFN}^{(\phi(i))}\left(\operatorname{Concat}_{h=1}^{\mathcal{H}}\left(m_i^{(h)}W_{O}^{(\phi(i))}\right)\right)\right), \tag*{(6)}

Here, LN is layer normalization and FFN⁡(ϕ(i))\operatorname{FFN}^{(\phi(i))} is a type-specific feed-forward network. Thus, ϕ\phi controls modality-specific transformations and rr defines relation-specific interaction patterns, allowing the teacher to assign distinct roles to patch–patch, global–local, and visual–text evidence. After LL message-passing layers, the teacher obtains updated visual and text nodes. Instead of averaging all patch tokens, which mixes informative and background regions, we use a learned token-selection readout. Here, “saliency” is the scalar importance score predicted by the graph teacher for each refined patch node, not an external saliency annotation. Each refined patch node fif_i is assigned a score σi=ws⊤fi\sigma_i = w_s^\top f_i, ws∈Rdw_s \in\mathbb{R}^{d}, where wsw_s is a learnable scoring vector included in the teacher parameters ψ\psi. We keep the top ⌈κM⌉\lceil\kappa M\rceil patch nodes ranked by σi\sigma_i, denoted Ntop\mathcal{N}_{\mathrm{top}}, where 0<κ≤10 < \kappa\le1 is a token-retention ratio tuned on validation. These selected nodes, together with the refined [CLS] token, form the graph representation used to compute teacher logits by comparing it with the refined text nodes:

hg=fcls+1∣Ntop∣∑fj∈Ntopfj,LGraph(x)=sim⁡(hg,Ft),(7)h_g = f_{\mathrm{cls}} + \frac{1}{\lvert\mathcal{N}_{\mathrm{top}}\rvert}\sum_{f_j \in\mathcal{N}_{\mathrm{top}}} f_j,\qquad L_{\mathrm{Graph}}(x) = \operatorname{sim}(h_g,F_t), \tag*{(7)}

where FtF_t denotes the graph-refined class-text embeddings. The teacher, therefore, provides class-discriminative supervision based on token-level visual evidence and visual–text relations.

Implicit Distillation Objective

The teacher is not distilled into the student by matching logits. Instead, it performs implicit structural distillation via the shared LoRA backbone: because both the deployed student sθ,ηs_{\theta,\eta} and structural teacher gθ,ψg_{\theta,\psi} depend on θ\theta, teacher-side losses directly update the low-rank inference parameters.

∂Lstruct∂θ=∂Lstruct∂hg∂hg∂Uθ(x)∂Uθ(x)∂θ.(8)\frac{\partial\mathcal{L}_{\mathrm{struct}}}{\partial\theta} = \frac{\partial\mathcal{L}_{\mathrm{struct}}}{\partial h_g} \frac{\partial h_g}{\partial U_{\theta}(x)} \frac{\partial U_{\theta}(x)}{\partial\theta}. \tag*{(8)}

Thus, the graph teacher shapes the deployed representation without becoming part of the deployed model. To stabilize this transfer in the few-shot regime, we introduce Prototype Predictive Alignment (PPA). For each class cc, let Sc={i∣yi=c}\mathcal{S}_c = \{i \mid y_i = c\} denote the indices of support examples belonging to class cc. We compute the class prototype from the cache keys aicache=zθ(xi)a_i^{\mathrm{cache}} = z_{\theta}(x_i) as

πˉc=1∣Sc∣∑i∈Scaicache,πc=πˉc∥πˉc∥2.(9)\bar{\pi}_c = \frac{1}{\lvert\mathcal{S}_c\rvert}\sum_{i\in\mathcal{S}_c} a_i^{\mathrm{cache}},\qquad \pi_c = \frac{\bar{\pi}_c}{\lVert\bar{\pi}_c\rVert_2}. \tag*{(9)}

Here, πˉc\bar{\pi}_c is the unnormalized support mean and πc\pi_c is the ℓ2\ell_2-normalized class prototype used as the PPA target. Given a training image with label yy, the graph representation hgh_g is mapped to prototype space by a lightweight head P\mathcal{P}: π^=P(hg)\hat{\pi} = \mathcal{P}(h_g). The PPA loss aligns this prediction with the ground-truth class prototype: LPPA=1−sim⁡(π^,πy)\mathcal{L}_{\mathrm{PPA}} = 1 - \operatorname{sim}(\hat{\pi},\pi_y). Instead of fitting only instance-level labels, PPA encourages the structural branch to recover class-level visual concepts from token interactions. Since hgh_g depends on Uθ(x)U_{\theta}(x), this alignment also regularizes the LoRA feature space used by the deployed student. The complete training objective is

Ltrain=LStudent+λ1LTeacher+λ2LPPA,(10)\mathcal{L}_{\mathrm{train}} = \mathcal{L}_{\mathrm{Student}} + \lambda_1\mathcal{L}_{\mathrm{Teacher}} + \lambda_2\mathcal{L}_{\mathrm{PPA}}, \tag*{(10)}

where λ1\lambda_1 and λ2\lambda_2 balance the structural losses. The student loss is cross-entropy on the deployed logits and the teacher loss applies focal loss Lin et al. [ [23] to the graph logits:

LStudent=CE⁡(sθ,η(x),y),LTeacher=−(1−py)γlog⁡py,p=softmax⁡(LGraph),(11)\mathcal{L}_{\mathrm{Student}} = \operatorname{CE}(s_{\theta,\eta}(x),y),\qquad \mathcal{L}_{\mathrm{Teacher}} = -(1-p_y)^\gamma\log p_y,\qquad p = \operatorname{softmax}(L_{\mathrm{Graph}}), \tag*{(11)}

where pyp_y is the probability of the ground-truth class and γ≥0\gamma\ge0 is the focusing parameter. The factor (1−py)γ(1-p_y)^\gamma down-weights confident teacher predictions and keeps larger gradients for low-confidence or misclassified examples.

This objective makes parameter coupling explicit. The student term updates (θ,η)(\theta,\eta) for deployment-time discrimination. The teacher and PPA terms update (θ,ψ)(\theta,\psi), but only θ\theta is shared with the deployed model. After training, ψ\psi and P\mathcal{P} are discarded, and inference uses only sθ,ηs_{\theta,\eta} from Eq. (1).

Evaluation of Structural Distillation

We evaluate on 11 standard CLIP few-shot benchmarks, spanning general object recognition, fine-grained categories, textures, scenes, remote sensing, and action recognition. We follow the standard inductive few-shot protocol Zhou et al. [ [45], Zhang et al. [ [40], Zanella and Ben Ayed [2024]: for each dataset, we randomly sample K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\} labeled examples per class from the training split and evaluate on the official validation or test split. We report top-1 accuracy. Unless stated otherwise, SD-LoRA uses CLIP ViT-B/16 Radford et al. [ [28]; additional ViT-B/32 results are in Appendix § 8.§ 1. CLIP weights are frozen; only the LoRA parameters, cache head, and training-only teacher are optimized with AdamW (learning rate 1×10−51 \times10^{-5}, cosine decay). Further implementation details are in Appendix § 7, additional backbone and transfer results in Appendix § 8, and hyperparameter sensitivity and computational complexity in Appendix § 10.

Does Structural Distillation Improve Few-Shot Adaptation?

Table 1 reports average few-shot performance over 11 benchmarks, and Figure 3 shows per-dataset trends. SD-LoRA consistently outperforms the strongest lightweight CLIP adaptation baselines across all shot regimes. Relative to CLIP-LoRA, which uses the same low-rank backbone without structural supervision, SD-LoRA improves average accuracy by 2.4 points at 1-shot, 1.8 at 2-shot, 3.0 at 4-shot, 2.4 at 8-shot, and 2.0 at 16-shot. These gains reflect the full SD-LoRA system. To analyze their source, Table 4 separates the LoRA+cache student from the training-only graph branch: the student-only baseline isolates the effect of cache-based inference, while the additional improvement of SD-LoRA over LoRA+Cache captures the impact of structural supervision on the shared LoRA representation. Gains are largest on fine-grained, domain-specific datasets such as Aircraft, EuroSAT, DTD, and UCF101, where global CLIP embeddings can miss local or class-specific cues. This supports SD-LoRA’s hypothesis that token-level structural reasoning is most useful during adaptation, when it shapes the deployable representation, but need not be retained at inference. Full per-dataset comparisons with adapter- and prompt-based methods are in Appendix § 8 (Table 6 and Table 7).

ShotsTIP-FTaskResCLIP-AdaptCCAPLOT++MaPLePromptSRCCLIP-LoRACLIP-ASTSD-LoRA
170.9±0.570.8±0.765.7±0.666.3±0.570.7±0.666.4±0.572.1±0.772.5±0.572.3±0.274.9±0.1
273.7±0.472.9±0.466.8±0.668.9±0.474.0±0.670.5±0.375.2±0.275.4±0.675.8±0.377.2±0.3
475.6±0.170.3±0.274.7±0.472.2±0.376.9±0.175.8±0.478.3±0.477.4±0.178.9±0.280.4±0.1
878.3±0.378.4±0.371.4±0.275.0±0.379.6±0.276.4±0.380.6±0.380.3±0.281.7±0.182.7±0.1
1680.7±0.280.8±0.275.5±0.177.6±0.282.1±0.181.8±0.182.7±0.283.0±0.184.6±0.185.0±0.2

Table 1. Macro-average few-shot accuracy across 11 benchmarks. We report average top-1 accuracy (%) over all datasets for top-10 representative adapter-, prompt-, and LoRA-based CLIP adaptation methods. Full per-dataset results are provided in Appendix § 8. The best result is shown in bold, and the strongest previous baseline in each shot setting is underlined.

Figure 3. Few-shot accuracy trends across 11 benchmarks. Each panel shows top-1 accuracy for K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\} shots on one dataset; the first panel reports the macro-average over all 11 benchmarks. SD-LoRA is compared to recent CLIP adaptation methods. Positive values mean structural distillation improves the deployed LoRA+cache model, with especially strong gains in low-shot and fine-grained scenarios.

VariantCacheGraphAircraft 1-ShotAircraft 4-shotAircraft 16-shotCaltech101 1-shotCaltech101 4-shotCaltech101 16-shotEuroSAT 1-shotEuroSAT 4-shotEuroSAT 16-shot
LoRA + Cache✓✗32.243.559.292.793.095.174.688.391.8
LoRA + Graph✗✓33.144.760.793.294.896.575.390.293.1
SD-LoRA✓✓34.846.261.494.496.297.276.891.394.0

Table 4. Main component ablation. LoRA is enabled in all variants. Cache is a deployed inference component, while the graph is a training-only teacher and is removed after adaptation. SD-LoRA combines both components and achieves the best performance across all datasets and shot settings.

Table 6. Full few-shot comparison on 11 benchmarks. We report top-1 accuracy (%) for 1-, 2-, and 4-shot classification against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold. SD-LoRA is reported in the final row of each shot group. Dataset abbreviations: INet: ImageNet; SUN: SUN397; Air: FGVC-Aircraft; Euro: EuroSAT; Cars: StanfordCars; Food: Food101; Pets: OxfordPets; Flow: Flowers102; Cal: Caltech101; DTD: Describable Textures; UCF: UCF101.

ShotsTypeMethodVenueINetSUNAirEuroCarsFoodPetsFlowCalDTDUCFAvg
0CLIPICML’2266.771.462.624.747.565.386.189.192.943.666.765.1
AdapterTIP-Adapter-FECCV’2269.467.228.867.867.185.890.683.894.051.673.470.9
TaskResCVPR’2369.668.131.365.468.884.690.281.793.653.871.770.8
GraphAdapterNeurIPS’2361.561.920.963.359.775.484.480.088.951.864.964.8
CLIP-AdapterIJCV’2467.971.325.265.449.365.786.189.092.044.266.965.7
CLAPCVPR’2458.561.120.659.256.373.083.679.988.447.562.562.8
CCAICCV’2561.563.822.567.060.077.886.981.089.951.066.366.3
1PromptCoOp (4)IJCV’2268.067.326.250.967.182.690.372.793.250.170.767.2
CoOp (16)IJCV’2265.767.020.856.467.584.390.278.392.550.171.267.6
CoCoOpCVPR’2269.468.728.155.467.684.991.973.494.152.670.468.8
PLOT++ICLR’2366.466.728.665.468.886.191.880.494.354.574.370.7
KgCoOpCVPR’2368.968.426.861.966.786.492.174.794.252.772.869.6
MaPLeCVPR’2369.769.328.129.167.685.491.474.993.650.071.166.4
ProGradICCV’2367.067.028.857.068.284.991.480.993.552.873.369.5
PromptSRCICCV’2368.568.527.472.868.785.390.585.993.856.774.872.1
CLIP-LoRACVPR’2470.470.430.272.370.184.392.383.293.754.376.372.5
CLIP-ASTCVPR’2570.170.331.068.070.786.092.383.094.156.074.072.3
SD-LoRA (Ours)—70.571.634.876.870.986.292.588.294.459.278.574.9
2AdapterTIP-Adapter-FECCV’2270.068.632.873.270.886.091.690.193.957.876.273.7
TaskResCVPR’2370.270.532.770.272.185.690.784.494.355.675.272.9
GraphAdapterNeurIPS’2362.364.623.867.363.276.386.385.690.255.769.567.7
CLIP-AdapterIJCV’2468.267.227.051.266.686.289.771.793.445.468.466.8
CLAPCVPR’2458.563.323.265.661.474.984.984.289.853.067.866.1
CCAICCV’2562.166.325.070.064.077.987.988.091.055.069.368.9
PromptCoOp (4)IJCV’2268.768.028.166.270.582.689.980.993.053.773.570.5
CoOp (16)IJCV’2267.067.025.965.170.484.489.988.093.154.174.170.8
CoCoOpCVPR’2270.169.429.361.868.485.991.977.894.452.373.470.4
PLOT++ICLR’2368.268.031.176.873.186.392.289.894.656.776.774.0
KgCoOpCVPR’2369.669.628.069.268.286.692.379.894.555.374.671.6
MaPLeCVPR’2370.070.729.559.468.586.591.879.894.950.674.070.5
ProGradICCV’2369.169.031.166.372.484.891.587.593.656.075.672.4
PromptSRCICCV’2369.871.331.579.272.986.092.391.194.660.078.675.2
CLIP-LoRACVPR’2470.871.333.282.773.283.291.389.894.659.980.075.4
CLIP-ASTCVPR’2571.171.535.082.073.586.192.990.095.958.078.075.8
SD-LoRA (Ours)—70.771.636.484.973.586.892.491.795.864.680.877.2
4AdapterTIP-Adapter-FECCV’2270.770.835.776.874.186.591.992.194.859.878.175.6
TaskResCVPR’2363.667.325.773.867.476.186.390.291.060.770.970.3
GraphAdapterNeurIPS’2363.166.727.075.265.576.886.689.991.059.671.570.3
CLIP-AdapterIJCV’2471.072.733.474.276.086.091.985.095.060.176.274.7
CLAPCVPR’2460.765.925.673.165.575.986.586.790.658.869.869.1
CCAICCV’2563.369.029.080.168.078.288.191.192.063.072.172.2
PromptCoOp (4)IJCV’2269.770.629.765.873.483.592.386.694.558.578.173.0
CoOp (16)IJCV’2270.669.730.969.774.484.592.592.294.559.577.674.2
CoCoOpCVPR’2270.870.430.661.769.586.392.781.594.855.775.371.8
PLOT++ICLR’2370.471.735.283.276.286.492.592.995.162.479.776.9
KgCoOpCVPR’2370.271.532.271.869.586.992.687.095.058.777.673.9
MaPLeCVPR’2367.770.634.884.575.381.791.992.694.461.078.475.8
ProGradICCV’2371.371.734.169.675.085.492.191.194.459.777.973.9
PromptSRCICCV’2370.873.837.486.377.086.693.193.895.365.481.678.3
CLIP-LoRACVPR’2471.472.837.984.977.482.791.093.795.263.881.177.4
CLIP-ASTCVPR’2571.173.441.088.078.086.893.594.095.765.081.078.9
SD-LoRA (Ours)—72.273.546.291.378.287.192.794.396.270.182.680.4
ShotsTypeMethodVenueINetSUNAirEuroCarsFoodPetsFlowCalDTDUCFAvg
8AdapterTIP-Adapter-FECCV’2271.773.539.581.378.386.991.894.395.266.782.078.3
8AdapterTaskResCVPR’2372.374.640.377.579.686.492.096.095.366.781.678.4
8AdapterGraphAdapterNeurIPS’2364.268.931.480.270.577.787.694.192.464.575.773.4
8AdapterCLIP-AdapterIJCV’2462.767.526.277.967.978.087.691.791.461.073.371.4
8AdapterCLAPCVPR’2462.968.628.976.770.377.487.792.191.463.273.372.1
8AdapterCCAICCV’2564.971.035.083.174.379.289.094.492.565.077.575.0
8PromptCoOp (4)IJCV’2270.872.437.074.776.883.392.195.094.763.779.876.4
8PromptCoOp (16)IJCV’2270.671.938.577.179.082.791.394.994.564.880.076.8
8PromptCoCoOpCVPR’2270.871.532.469.170.487.093.386.394.960.175.973.8
8PromptPLOT++ICLR’2371.373.941.488.381.286.593.095.495.566.482.879.6
8PromptKgCoOpCVPR’2370.272.634.873.972.887.093.091.595.165.680.076.0
8PromptMaPLeCVPR’2371.373.233.882.871.387.293.190.595.163.079.576.4
8PromptProGradICCV’2371.373.037.777.878.786.192.295.094.863.980.577.4
8PromptPromptSRCICCV’2371.775.843.088.880.986.993.296.195.769.884.580.6
8PromptCLIP-LoRACVPR’2472.374.745.789.782.183.191.796.395.667.584.180.3
8PromptCLIP-ASTCVPR’2572.975.450.090.083.087.094.596.596.170.184.081.7
8PromptSD-LoRA (Ours)—73.275.653.292.784.487.493.196.896.372.284.882.7
16AdapterTIP-Adapter-FECCV’2273.476.044.685.982.386.892.696.295.770.883.980.7
16AdapterTaskResCVPR’2373.076.144.982.783.586.992.497.595.871.584.080.8
16AdapterGraphAdapterNeurIPS’2365.771.236.985.376.278.688.696.293.367.678.876.2
16AdapterCLIP-AdapterIJCV’2469.874.234.271.474.087.192.392.994.959.480.275.5
16AdapterCLAPCVPR’2465.070.833.680.175.178.588.594.291.966.476.374.6
16AdapterCCAICCV’2566.072.242.085.379.079.890.995.293.570.080.077.6
16PromptCoOp (4)IJCV’2271.574.640.183.579.185.192.496.495.569.281.979.0
16PromptCoOp (16)IJCV’2271.974.943.285.082.984.292.096.895.869.783.180.0
16PromptCoCoOpCVPR’2271.172.633.373.672.387.493.489.195.163.777.275.3
16PromptPLOT++ICLR’2372.676.046.792.084.587.193.597.596.071.485.382.1
16PromptKgCoOpCVPR’2370.473.336.576.274.887.293.293.495.268.781.777.3
16PromptMaPLeCVPR’2372.375.548.492.383.585.392.897.096.071.385.081.8
16PromptProGradICCV’2372.175.143.083.682.985.892.896.695.968.882.779.9
16PromptPromptSRCICCV’2372.877.250.692.183.687.493.297.496.072.686.782.7
16PromptCLIP-LoRACVPR’2473.676.154.792.186.384.292.498.096.472.086.783.0
16PromptCLIP-ASTCVPR’2573.977.460.093.087.087.594.498.097.175.087.084.6
16PromptSD-LoRA (Ours)—74.277.561.494.087.787.894.198.397.274.088.385.0

Table 7. Full 8- and 16-shot comparison on 11 benchmarks. We report top-1 accuracy (%) against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold. SD-LoRA is reported in the final row of each shot group. Dataset abbreviations follow Table 6.

Scalability with more support data. As shown in Figure 3, increasing the number of shots improves all strong adaptation methods and narrows their performance gap, making large margins at higher shots less likely, particularly for recent LoRA- and adapter-based baselines. Nonetheless, SD-LoRA is best in all shot settings, outperforming the strongest prior baseline by +2.4, +1.4, +1.5, +1.0, and +0.4 points for 1-, 2-, 4-, 8-, and 16-shots, respectively (Table 1). The smaller margin at 16 shots reflects that top baselines are already tightly clustered, yet SD-LoRA still achieves the highest average accuracy and clear gains on difficult datasets such as Aircraft, EuroSAT, DTD, and UCF101. This suggests that structural supervision is especially useful when local evidence is decisive, while remaining competitive as more labeled examples reduce ambiguity.

Does the Learned Representation Generalize?

Standard few-shot accuracy only measures adaptation to a fixed support set, not transfer to unseen classes or robustness to distribution shift. We therefore also evaluate SD-LoRA on base-to-novel generalization and ImageNet out-of-distribution robustness.

Base-to-novel generalization. Table 2 evaluates whether adaptation to base classes transfers to novel classes under the MaPLe protocol. SD-LoRA achieves the best average performance on all three metrics: 87.23% on base classes, 86.59% on novel classes, and 86.91% harmonic mean. Compared to the strongest baseline, it yields a small gain on base classes (+0.27) but clear gains on novel classes (+5.85) and harmonic mean (+3.18), indicating that SD-LoRA avoids overfitting to base classes and instead learns representations that transfer better to unseen categories. Improvements are strongest on fine-grained and domain-specific datasets: SD-LoRA improves novel-class accuracy over the strongest baseline by +9.33 on Cars, +14.79 on EuroSAT, +15.54 on Flowers, +10.16 on UCF101, and +9.31 on DTD. These gains indicate that token-level structural supervision preserves class-discriminative local evidence beyond the support classes. Although SD-LoRA is not always best on base-class accuracy, it consistently improves the base–novel trade-off, as shown by the harmonic mean.

MethodAvg.(11)CarsEuroSATFlowersUCFDTD
BaseNovelHMBaseNovelHMBaseNovelHMBaseNovelHMBaseNovelHMBaseNovelHM
CLIP69.3474.2271.7063.3774.8968.6556.4864.0560.0372.0877.8074.8370.5377.5073.8553.2459.9056.37
CoOp82.6963.2271.6678.1260.4068.1392.1954.7468.6997.6059.6774.0684.6956.0567.4679.4441.1854.24
CoCoOp80.4771.6975.8370.4973.5972.0187.4960.0471.2194.8771.7581.7182.3373.4577.6477.0156.0064.85
ProGrad82.4870.7576.1677.6868.6372.8890.1160.8972.6795.5471.8782.0384.3374.9479.3577.3552.3562.45
KgCoOp80.7373.6077.0071.7675.0473.3685.6464.3473.4895.0074.7383.6582.8976.6779.6577.5554.9964.35
MaPLe82.2875.1478.5572.9474.0073.4794.0773.2382.3595.9272.4682.5683.0078.6680.7780.3659.1868.16
PromptSRC84.2676.1079.9778.2774.9776.5892.9073.9082.3298.0776.5085.9587.1078.8082.7483.3762.9771.75
CLIP-AST85.9476.9981.0684.2174.0578.8095.9081.7288.2497.9177.7386.6687.3879.1283.0584.0365.3473.52
PromptKD86.9680.7383.7382.8083.3783.1397.5482.0889.1499.4282.6290.2489.7182.2786.1085.8471.3777.94
Ours87.2386.5986.9186.0692.7089.2597.2996.8797.0898.2098.1698.1890.0792.4391.2484.6880.6882.63
Δ+0.27+5.85+3.18+1.85+9.33+6.12-0.25+14.79+7.94-1.22+15.54+7.94+0.36+10.16+5.14-1.16+9.31+4.69

Table 2. Base-to-novel generalization under the MaPLe protocol. We report accuracy (%) on base classes, novel classes, and their harmonic mean (HM). Avg.(11) is the unweighted mean over 11 benchmarks. The Δ row shows the improvement of SD-LoRA over the strongest baseline for each metric. Best results are in bold.

Out-of-distribution robustness. Table 3 reports robustness by adapting on ImageNet and testing on four OOD variants: ImageNet-R, ImageNet-A, ImageNet-V2, and ImageNet-Sketch. SD-LoRA achieves the highest average accuracy across all five datasets, surpassing the strongest structured baseline, GraphAdapter, by 0.41 points. The margin is smaller than in the base-to-novel setting because all methods use a strong, shared ImageNet adaptation protocol, and the top baselines are closely matched. Notably, we achieve this without any graph- or token-level reasoning at inference.

DatasetCLIPCoOpCoCoOpMaPLeProGradKgCoOpGraphAdapterTransCLIPCLIP-ASTOurs
ImageNet66.7371.5171.0270.7272.2471.2073.6870.3073.8774.20
ImageNet-R73.9675.2176.1876.9874.5876.7077.2075.00—77.32
ImageNet-A47.7749.7150.6350.9049.3950.6949.2349.50—49.63
ImageNet-V260.8364.2064.0764.0764.7364.1065.5762.3066.3765.78
ImageNet-Sketch46.1547.9948.7549.1547.6148.9748.5749.0748.3749.38
Avg.59.0961.7262.1362.2361.7162.3362.8561.23—63.26

Table 3. Out-of-distribution (OOD) generalization. Models are adapted on ImageNet and evaluated on ImageNet and four OOD variants. Top-1 accuracy (%) is reported, and “Avg.” denotes the unweighted mean over all five datasets. Best results are shown in bold.

SD-LoRA achieves the best accuracy on ImageNet, ImageNet-R, and ImageNet-Sketch, and remains competitive on ImageNet-A and ImageNet-V2, reflecting the impact of the training-time graph teacher. Structural distillation strengthens local texture, shape, and part-level cues, which particularly benefits rendition and sketch-style shifts. In contrast, ImageNet-A contains adversarially filtered natural images with strong object, pose, and context biases, where improving local evidence alone may not fully address the shift. ImageNet-V2 is close to the original ImageNet validation set, and several source-adapted baselines already perform similarly, leaving limited room for gains. Overall, SD-LoRA improves the average OOD trade-off without requiring test-time graph reasoning, with its advantage largest when token-level visual evidence is directly informative. It also preserves the lightweight CLIP-LoRA+cache deployment: after adaptation, the graph teacher and prototype head are removed. These results support our central claim that structural reasoning can enhance representations during training without increasing test-time complexity.

Additional transfer and calibration results. Additional generalization results appear in Appendix § 8.§ 2 and Appendix § 8.§ 3. Adapted on ImageNet and evaluated on ten target datasets without target-domain training, SD-LoRA raises average cross-dataset accuracy over PromptKD from 71.33% to 74.98% (+3.65), with gains on 9 of 10 datasets. Under the same setting, calibration also improves: average ECE drops from 5.57 to 2.65, NLL from 1.05 to 0.62, and Brier score from 0.38 to 0.20, indicating more accurate, better-calibrated representations under domain shift.

What Components Drive the Improvement?

Table 4 isolates the effects of the cache branch and the training-only graph teacher. LoRA+Cache is the deployed student without structural supervision, and LoRA+Graph removes the cache but retains graph-guided training of the shared LoRA backbone. Both components help, but together they work best. Across Aircraft, Caltech101, and EuroSAT, SD-LoRA outperforms LoRA+Cache by +2.2, +3.0, and +2.2 points at 1-, 4-, and 16-shot learning, and exceeds LoRA+Graph by +1.5, +1.3, and +0.8 points. Thus, the cache and graph are complementary: the cache offers efficient support-set memory at inference, while the graph enhances LoRA training. Because the graph is discarded after adaptation, these gains add no test-time graph cost.

Effect of LoRA placement. Figure 4 shows that SD-LoRA consistently outperforms both fixed CLIP-LoRA and adaptive CLIP-AST across datasets and shot settings. This indicates that gains are not only from better LoRA placement: adaptive placement decides where to insert LoRA, while structural distillation improves how its parameters are trained.

Figure 4. Effect of LoRA placement. We compare fixed LoRA placement and adaptive layer selection under 1/4/16-shot settings. SD-LoRA outperforms both fixed CLIP-LoRA and adaptive CLIP-AST, indicating that adaptive placement complements SD-LoRA: layer selection optimizes where LoRA is inserted, while structural distillation improves how its parameters are trained.

More ablations are provided for graph edge families (Appx. C.2, Table 13), PPA (Appx. C.3, Table 14), token filtering (Appx. C.4, Table 15), input resolution (Appx. C.5, Table 16), graph design (Appx. C.7, Table 18), and loss terms (Appx. C.8, Table 19). Appx. D details computational complexity and hyperparameter selection, and Appx. E discusses scope, limitations, and responsible use.

VariantSpatialGlobal–localCLS–textPatch–textAircraft 1-shotAircraft 4-shotAircraft 16-shotCaltech101 1-shotCaltech101 4-shotCaltech101 16-shot
Full graph✓✓✓✓34.8046.2061.4094.4096.2097.20
w/o spatial edges✗✓✓✓33.2145.4760.4492.7595.4096.28
w/o global–local edges✓✗✓✓32.3643.9858.1291.9293.9293.84
w/o semantic text–vision edges✓✓✗✗31.5743.8757.3291.0293.8093.28
CLS-semantic only✓✓✓✗33.5445.8060.6893.2295.3396.44
Patch-semantic only✓✓✗✓32.7844.7258.3292.4594.8294.31

Table 13. Edge-type ablation of the graph teacher. Accuracy (%) is reported on Aircraft and Caltech101 under 1-, 4-, and 16-shot settings. The edge columns indicate whether each relation family is enabled: spatial patch–patch, global–local CLS–patch, semantic CLS–text, and semantic patch–text interactions. The full graph performs best across all settings. Removing semantic visual–text edges gives the largest degradation, followed by global–local edges, while spatial patch–patch edges provide a smaller but consistent gain.

VariantEnabled componentsAircraftCaltech101EuroSAT
PPAGraph1-shot4-Shot16-Shot1-shot4-Shot16-Shot1-shot4-Shot16-Shot
HR only✗✗32.243.559.292.793.095.174.688.391.8
HR + PPA✓✗31.743.459.592.593.095.373.986.492.1
HR + Graph✗✓33.645.160.294.296.197.176.189.993.2
Full model✓✓34.846.261.494.496.297.276.891.394.0

Table 14. Effect of PPA and the graph teacher. LoRA and cache-based inference are enabled in all variants. We compare high-resolution adaptation alone (HR only), adding Prototype Predictive Alignment (HR + PPA), adding the training-only graph teacher (HR + Graph), and the full model with both graph and PPA. Accuracy (%) is reported under 1-, 4-, and 16-shot settings on Aircraft, Caltech101, and EuroSAT. The graph teacher provides the dominant gain, while PPA is most effective when combined with graph-conditioned representations.

Pooling Strategy1-shot2-shot4-shot8-shot16-shot
EuroSAT∑All\sum_{\mathrm{All}} (Global)75.980.189.592.293.6
\sum_{\mathrm{Top\mathchar`-M=25}}77.081.290.092.693.9
\sum_{\mathrm{Top\mathchar`-M=50}}76.884.991.392.794.0
\sum_{\mathrm{Top\mathchar`-M=75}}76.781.088.892.493.2
Caltech∑All\sum_{\mathrm{All}} (Global)94.194.894.996.096.8
\sum_{\mathrm{Top\mathchar`-M=25}}94.295.095.096.296.9
\sum_{\mathrm{Top\mathchar`-M=50}}94.495.896.296.397.2
\sum_{\mathrm{Top\mathchar`-M=75}}94.094.895.396.097.0
Food∑All\sum_{\mathrm{All}} (Global)86.186.787.087.287.5
\sum_{\mathrm{Top\mathchar`-M=25}}86.086.886.987.387.6
\sum_{\mathrm{Top\mathchar`-M=50}}86.286.887.187.487.8
\sum_{\mathrm{Top\mathchar`-M=75}}86.186.786.887.287.4

Table 15. Effect of token filtering in the graph-teacher readout. We compare global pooling over all graph-refined tokens with saliency-based Top-MM filtering. Top-MM retains only the highest-scoring patch tokens before aggregation, reducing the influence of background or weakly informative tokens. A moderate selection size, Top-MM=50, provides the best overall trade-off across datasets and shot settings.

DatasetResolution (H′×W′H^{\prime} \times W^{\prime})1 Shot2 Shot4 Shot8 Shot16 Shot
Caltech224 ×\times 224 (Standard)94.195.695.496.896.9
Caltech336 ×\times 33694.495.896.296.397.2
Caltech512 ×\times 51291.592.693.794.895.9
Euro224 ×\times 224 (Standard)76.586.691.592.493.9
Euro336 ×\times 33676.884.991.392.794.0
Euro512 ×\times 51275.182.790.291.592.4
Food224 ×\times 224 (Standard)86.086.586.787.187.1
Food336 ×\times 33686.286.887.187.487.8
Food512 ×\times 51284.385.185.385.685.9

Table 16. Effect of input resolution. We compare the standard CLIP input resolution, 224 ×\times 224, with higher resolutions used for training-time graph reasoning. The 336 ×\times 336 setting generally provides the best trade-off between finer token granularity and pretrained feature stability, while 512 ×\times 512 introduces a stronger resolution shift and degrades performance.

ComponentsAircraftCaltech101EuroSAT
TransRGTFiltering1-shot4-shot16-shot1-shot4-shot16-shot1-shot4-shot16-shot
✗✓✓34.145.158.893.294.895.474.289.891.1
✓✗✓33.545.357.692.893.795.573.188.689.8
✓✓✗33.444.257.492.494.295.773.789.191.6
✗✓✗32.744.956.591.593.394.572.888.290.8
✓✗✗33.244.055.391.592.593.372.187.590.0
✗✗✓32.741.754.289.791.892.372.386.889.3
✓✓✓34.846.261.494.496.297.276.891.394.0

Table 18. Graph teacher architecture ablation. We evaluate the contributions of the visual refinement Transformer (Trans), the relation-aware graph Transformer (RGT), and discriminative node filtering. Accuracy (%) is reported on Aircraft, Caltech101, and EuroSAT under 1-, 4-, and 16-shot settings. The full architecture performs best across all datasets and shot settings, showing that token refinement, relation-aware message passing, and foreground-focused filtering provide complementary structural supervision.

Loss ComponentsEuroSATCaltech101Food101
LStudent\mathcal{L}_{\mathrm{Student}}LTeacher\mathcal{L}_{\mathrm{Teacher}}LPPA\mathcal{L}_{\mathrm{PPA}}1-shot4-shot8-shot1-shot4-shot8-shot1-shot4-shot8-shot
✓✗✗74.688.391.892.793.095.185.286.386.9
✓✓✗76.289.793.594.294.996.885.786.787.2
✓✗✓76.491.093.894.395.597.086.086.987.6
✓✓✓76.891.394.094.496.297.286.287.187.8

Table 19. Loss component ablation. We evaluate the contributions of the student loss LStudent\mathcal{L}_{\mathrm{Student}}, the graph-teacher loss LTeacher\mathcal{L}_{\mathrm{Teacher}}, and Prototype Predictive Alignment LPPA\mathcal{L}_{\mathrm{PPA}}. Accuracy (%) is reported on EuroSAT, Caltech101, and Food101 under 1-, 4-, and 8-shot settings. The full objective performs best across all settings, showing that graph-based structural supervision and prototype-level alignment provide complementary training signals.

Matched controls and retained structural supervision. On Aircraft, Caltech101, EuroSAT, and Food101, we match support splits, optimization, and cache-based deployment (three runs per configuration). Table 5 varies training resolution; all models deploy at 224×224224 \times224. Structural training adds +1.8/ + 1.7/ + 1.7/ + 1.7 pp over the matched 336-trained cache baseline. After teacher removal, the deployed student’s intra-class distance falls from 0.560 to 0.511 and inter-class distance rises from 0.713 to 0.890; prototype margin increases from 0.075 to 0.156. These results support a persistent structural-training effect beyond cache or resolution alone. Full controls and representation diagnostics appear in Appx. C.6 and C.1.

Training configuration1-shot4-shot8-shot16-shot
CLIP-LoRA+cache @22470.877.279.882.9
CLIP-LoRA+cache @33671.378.580.783.4
SD-LoRA @33673.180.282.485.1

Table 5. Resolution- and cache-matched controls. Top-1 accuracy (%) is averaged over four datasets, distinct from the 11-benchmark averages in Table 1. @ denotes training resolution; inference is always 224×224224 \times224.

Conclusion

We proposed SD-LoRA, a training-time structural distillation framework for few-shot CLIP adaptation that decouples training from deployment. A relation-aware graph teacher is used only during optimization to reason over visual tokens and class-text embeddings, transferring structural supervision into a shared LoRA backbone. At test time, the teacher is removed and inference is based solely on a lightweight CLIP-LoRA+cache student. Unlike many efficient VLM adapters that supervise mainly global image embeddings, SD-LoRA injects token-level relational bias during training while keeping the inference cost of the standard LoRA+cache. Experiments on few-shot accuracy, base-to-novel generalization, OOD robustness, and calibration show consistent gains over strong baselines, and ablations confirm the complementarity of graph-guided training and cache-based inference. SD-LoRA demonstrates that structural reasoning can enhance parameter-efficient VLM adaptation without adding inference-time overhead.

Acknowledgments and Disclosure of Funding

This work was supported by UK Research and Innovation (UKRI) through the Engineering and Physical Sciences Research Council (EPSRC) grant ATRACT (EP/X028631/1), the Economic and Social Research Council (ESRC) grant SCAnDi (ES/Y010655/1), and Cancer Research UK (CRUK) grants EDDISA–May21\100001 and EDDPMA–May22\100028.

Additional results, ablation studies, analyzes, visualization, and computational complexity that could not be accommodated in the main paper due to space constraints are provided in this Appendix.

Additional Implementation Details

This appendix provides implementation details for reproducing SD-LoRA. The main paper focuses on the experimental findings and analysis; here we describe support-set construction, preprocessing, LoRA adaptation, cache construction, the training-time graph teacher, and optimization.

Support-set construction. For each dataset, we follow the standard inductive few-shot protocol used in prior CLIP adaptation work Zhou et al. [ [45], Zhang et al. [ [40], Zanella and Ben Ayed [2024]. Given a CC-class dataset, we randomly sample K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\} labeled examples per class from the training split to form the support set S={(xi,yi)}i=1CK\mathcal{S} = \{(x_i, y_i)\}_{i=1}^{CK}. The resulting support set has CKCK labeled images. No class-manifold search or unlabeled test-time information is used. For each shot setting, we follow the standard inductive few-shot protocol and repeat the experiment over three independently sampled support splits, and report the mean top-1 accuracy.

Preprocessing and prompts. We use the standard CLIP image preprocessing pipeline. During adaptation, images are resized and augmented following common few-shot CLIP practice Zhang et al. [ [40], Zanella and Ben Ayed [2024]; at evaluation time, deterministic resizing and center cropping are used. Text embeddings are produced from class-name prompts using the CLIP text encoder. Unless otherwise stated, we use the same prompt templates as prior CLIP few-shot adaptation methods to ensure a fair comparison.

LoRA adaptation. The backbone is CLIP ViT-B/16 Radford et al. [ [28]. The pretrained CLIP parameters are frozen, and adaptation is performed using low-rank updates. For a pretrained projection matrix W0W_0, LoRA represents the adapted weight as

W=W0+ΔW,ΔW=BA,(12)W = W_0 + \Delta W,\qquad\Delta W = BA, \tag*{(12)}

where AA and BB are trainable low-rank matrices. Only the LoRA parameters, the cache head, and the training-only teacher parameters are optimized during adaptation. The vision and text encoder remains frozen throughout. This keeps the deployed model lightweight while allowing the visual representation to be shaped by structural supervision during training.

Cache construction. For each support image xix_i, we compute the normalized LoRA-adapted image feature

zθ(xi)=fvθ(xi)∥fvθ(xi)∥2.(13)z_\theta(x_i) = \frac{f_v^\theta(x_i)}{\lVert f_v^\theta(x_i)\rVert_2}. \tag*{(13)}

The support features form the cache keys, while their one-hot class labels form the cache values. For a query image xx, the cache branch computes similarities between the query feature zθ(x)z_\theta(x) and the support keys, followed by an affinity transformation controlled by the cache sharpness parameter. The resulting cache logits are combined with the zero-shot CLIP logits to obtain the deployed student prediction. After adaptation, the cache keys are recomputed once using the final LoRA-adapted encoder and then kept fixed for inference.

Training-time structural teacher. The graph teacher is used only during adaptation. It takes high-resolution ViT visual tokens from the same LoRA-adapted visual backbone as the deployed student, along with frozen class-text embeddings. Positional embeddings are interpolated for higher spatial resolutions. The teacher builds a heterogeneous graph with visual nodes ([CLS] and patch tokens) and class-text nodes (class-level text embeddings).

The graph has three relation types: patch–patch, global–local, and visual–text. Patch–patch edges capture local spatial interactions among neighboring tokens. Global–local edges link the [CLS] token to patch tokens, allowing image-level context to influence local evidence. Visual–text edges connect visual tokens to class-text embeddings, enabling class-conditioned structural reasoning. The graph is refined via relation-aware message passing, with node-type-specific transformations controlled by ϕ(i)\phi(i) and relation-specific interactions by r=rel⁡(j,i)r = \operatorname{rel}(j, i).

Structural readout. After graph refinement, we obtain updated visual and text node representations. To prevent background tokens from overwhelming local evidence, we apply a saliency-based readout over patch nodes: each refined patch node receives a learnable score, we keep the top-κ\kappa fraction, average them, and combine this with the refined [CLS] representation to form the graph-conditioned representation hgh_g. This representation is used only for training losses and is not computed at inference.

Training objective. The overall objective combines three terms:

Ltrain=LStudent+λ1LTeacher+λ2LPPA.\mathcal{L}_{\mathrm{train}} = \mathcal{L}_{\mathrm{Student}} + \lambda_1\mathcal{L}_{\mathrm{Teacher}} + \lambda_2\mathcal{L}_{\mathrm{PPA}}.

The student loss uses cross-entropy on the deployed CLIP-LoRA+cache logits. The teacher loss applies focal loss to the graph logits, down-weighting confident examples and emphasizing low-confidence or misclassified ones. Prototype Predictive Alignment (PPA) encourages the graph-conditioned representation to predict the support-derived prototype of the ground-truth class. Because the student and teacher share the LoRA-adapted backbone, gradients from the teacher and PPA losses update the same LoRA parameters used at inference.

Evaluation. All aggregate values are computed directly from the per-dataset results using unweighted arithmetic means. For base-to-novel evaluation, harmonic mean is computed as HM=2BN/(B+N)\mathrm{HM} = 2BN/(B + N) from the reported base and novel accuracies.

Optimization. We optimize with AdamW, an initial learning rate of 1×10−51 \times10^{-5}, and cosine learning-rate decay, using the same settings across datasets unless noted. To prevent test-set leakage, all hyperparameters are chosen only from training/validation data or via support-split cross-validation; test labels are never used. Hyperparameter sensitivity is reported in Appendix § 10.

Training-time structural transfer and deployment. Algorithm 1 summarizes the asymmetric train–deploy design. The structural branch and student jointly update the same LoRA parameters; there is no independently pretrained frozen teacher or teacher–student logit-matching loss.

Algorithm 1 SD-LoRA: Training-Time Structural Transfer and Lightweight Deployment

Require: Labeled support set SS, pretrained CLIP backbone, frozen class-text embeddings TT, selected training and cache settings Ensure: Adapted 224×224224 \times224 CLIP-LoRA+cache predictor 1: Initialize shared LoRA parameters θ\theta, cache parameters η\eta, and auxiliary teacher/PPA parameters 2: for each support-set optimization step do 3: Compute student logits using the LoRA backbone and support-derived cache Eq. (1) 4: Extract visual tokens and construct patch–patch, global–local, and visual–text relations 5: Compute graph-conditioned representations and teacher logits 6: Align graph-conditioned evidence with support-derived class prototypes Eq. (9) 7: Lstruct←λ1LTeacher+λ2LPPA\mathcal{L}_{\mathrm{struct}} \leftarrow\lambda_1\mathcal{L}_{\mathrm{Teacher}} + \lambda_2\mathcal{L}_{\mathrm{PPA}} 8: Optimize LStudent+Lstruct\mathcal{L}_{\mathrm{Student}} + \mathcal{L}_{\mathrm{struct}}, updating shared θ\theta Eqs. (8), (10) 9: end for 10: Discard the graph teacher, saliency/readout, PPA head, and high-resolution path 11: Recompute the support cache once using the full 224×224224 \times224 LoRA encoder 12: return the fixed LoRA+cache predictor sθ,η(x)=LZS(x)+ωLCache(x)s_{\theta,\eta}(x) = L_{\mathrm{ZS}}(x) + \omega L_{\mathrm{Cache}}(x)

The structurally supervised updates to θ\theta remain in the deployed encoder even though auxiliary parameters are discarded. Each test image is processed independently by a single LoRA image-encoder pass and the cache classifier, without graph construction or high-resolution processing. Code repository: https://github.com/MR-Sherif/SD-LoRA.

Inference. At test time, all teacher-specific components are removed, including the relation-aware graph transformer, high-resolution graph-token processing, structural readout, and prototype prediction head. Inference uses only the LoRA-adapted CLIP image encoder, frozen CLIP text embeddings, and the cache-based classifier:

sθ,η(x)=LZS(x)+ωLCache(x).s_{\theta,\eta}(x) = L_{\mathrm{ZS}}(x) + \omega L_{\mathrm{Cache}}(x).

Thus, SD-LoRA introduces structural reasoning only during training while keeping inference lightweight, with no graph computation at deployment.

Comparison to the State-of-the-art (SotA): Complete Table

Benchmarks. We evaluate our approach on 11 standard few-shot image classification benchmarks widely used in CLIP research: ImageNet Deng et al. [2009], Caltech101 Fei-Fei et al. [2004], OxfordPets Parkhi et al. [2012], StanfordCars Krause et al. [2013], Flowers102 Nilsback and Zisserman [2008], Food101 Bossard et al. [2014], FGVC-Aircraft Maji et al. [2013], SUN397 Xiao et al. [2010], DTD Cimpoi et al. [2014], EuroSAT Helber et al. [2019], and UCF101 Soomro et al. [2012], covering general object recognition, fine-grained categorization, textures, scene understanding, remote sensing, and action recognition.

Baselines. We compare our method with two main streams of state-of-the-art vision-language adaptation techniques. The first includes adapter-based methods: Tip-Adapter Zhang et al. [2022], TaskRes Yu et al. [2023], GraphAdapter Li et al. [2023], CLIP-Adapter Gao et al. [2024], CLAP Silva-Rodriguez et al. [2024], and CCA Jiang et al. [2025]. The second includes prompt-based and fine-tuning methods: CoOp Zhou et al. [2022b], CoCoOp Zhou et al. [2022a], PLOT++ Chen et al. [2022], KgCoOp Yao et al. [2023], MaPLe Khattak et al. [2023a], ProGrad Zhu et al. [2023], PromptSRC Khattak et al. [2023b], CLIP-LoRA Zanella and Ben Ayed [2024], and CLIP-AST Zhang et al. [2025].

Discussion of full few-shot results. Table 6 and Table 7 present per-dataset results corresponding to the average comparison in Table 1. SD-LoRA consistently improves performance across shot settings and dataset types, with especially strong gains on fine-grained and domain-specific benchmarks such as Aircraft, EuroSAT, DTD, Flowers102, and UCF101. These datasets rely on localized shape, texture, or class-specific cues, highlighting the benefit of the training-only graph teacher in shaping the deployed LoRA representation. Relative to CLIP-LoRA, SD-LoRA preserves the same lightweight inference design but achieves higher accuracy via structural supervision during adaptation. The results also show that SD-LoRA is complementary to adaptive LoRA placement methods such as CLIP-AST: layer selection controls where LoRA is inserted, whereas SD-LoRA controls how its parameters are trained. We include the full tables here for completeness; the main paper reports only compact averages and per-dataset trends to save space.

Additional base-to-novel results. Table 8 complements the main base-to-novel evaluation by reporting the remaining per-dataset results omitted from the main paper due to space constraints. The results further support the trend observed in the main table: SD-LoRA improves the base–novel trade-off rather than merely increasing base-class accuracy. Across Aircraft, Caltech101, Food101, OxfordPets, and SUN397, SD-LoRA achieves the best base, novel, and harmonic-score scores among all compared methods. The gains are especially clear on Aircraft, where SD-LoRA improves over the strongest baseline by +1.01 on base classes, +2.01 on novel classes, and +1.59 HM, indicating stronger transfer in a fine-grained recognition setting. On ImageNet, SD-LoRA is slightly below PromptKD on base-class accuracy, but still improves novel accuracy and HM, increasing the harmonic mean from 77.62 to 77.94. Together with the main base-to-novel results, these additional datasets show that training-time structural distillation improves novel-class generalization while preserving competitive base-class performance.

MethodAircraft BaseAircraft NovelAircraft HMCaltech BaseCaltech NovelCaltech HMFood BaseFood NovelFood HMImagenet BaseImagenet NovelImagenet HMPets BasePets NovelPets HMSUN BaseSUN NovelSUN HM
CLIP27.1936.2931.0996.8494.0095.4090.1091.2290.6672.4368.1470.2291.1797.2694.1269.3675.3572.23
CoCoOp33.4123.7127.7497.9693.8195.8490.7091.2990.9975.9870.4373.1095.2097.6996.4379.7476.8678.27
MaPLe37.4435.6136.5097.7494.3696.0290.7192.0591.3876.6670.5473.4795.4397.7696.5880.8278.7079.75
PromptSRC42.7337.8740.1598.1094.0396.0290.6791.5391.1077.6070.7374.0195.3397.3096.3082.6778.4780.52
CLIP-AST48.9838.2142.9398.7194.0096.3090.5791.1190.8478.4470.2274.1096.2397.3796.8083.0578.1280.51
PromptKD49.1241.8145.1798.9196.6597.7792.4393.6893.0580.8374.6677.6296.3098.0197.1583.6981.5482.60
Ours50.1343.8246.7699.2397.0398.1292.6794.0993.3780.4475.6077.9496.8998.4097.6483.9182.6783.28

Table 8. Additional base-to-novel generalization results under the MaPLe protocol. Due to space constraints in the main paper, we report the remaining per-dataset results here. We show accuracy (%) on base classes, novel classes, and their harmonic mean (HM).

Results with an Alternative ViT-B/32 Backbone

To examine whether training-time structural distillation depends on a specific CLIP backbone, we repeat the few-shot evaluation with CLIP ViT-B/32. Compared with ViT-B/16, ViT-B/32 offers a weaker visual backbone with coarser patches, making it a good test of SD-LoRA when the representation has less fine-grained spatial detail. Table 9 reports results on 10 benchmarks (excluding ImageNet) for K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\} shots.

ShotsMethodSUNAirEuroCarsFoodPetsFlowCalDTDUCFAvg
1CoOp (4)64.818.749.360.574.487.768.091.547.166.562.85
CoOp (16)63.315.752.959.675.987.671.591.747.266.363.17
CoCoOp65.615.546.760.479.588.268.692.245.767.362.97
TIP-Adapter-F65.422.259.761.180.487.581.192.450.966.566.72
CLIP-Adapter63.219.846.960.780.687.666.791.644.162.962.41
PLOT++64.522.067.361.479.289.473.793.050.669.767.08
KgCoOp65.720.955.461.280.788.668.192.250.266.864.98
TaskRes62.020.960.159.674.684.375.288.850.164.363.99
MaPLe66.913.732.360.179.787.668.792.044.566.461.19
ProGrad64.921.252.860.678.187.974.991.651.066.564.95
CLIP-LoRA67.622.967.163.578.488.777.493.049.872.368.07
TOGA66.224.360.862.981.088.584.592.652.570.268.35
SD-LoRA (Ours)69.427.569.865.681.587.884.592.757.071.970.77
2CoOp (4)65.620.856.562.174.286.677.191.549.471.565.53
CoOp (16)63.818.662.462.373.385.882.091.249.670.465.94
CoCoOp66.815.451.761.180.888.770.092.750.370.064.75
TIP-Adapter-F66.824.667.463.780.888.285.592.854.971.869.65
CLIP-Adapter64.820.950.161.581.087.967.191.444.965.963.55
PLOT++65.523.173.064.777.087.484.392.952.872.669.33
KgCoOp67.521.758.562.481.188.873.792.853.169.266.88
TaskRes64.422.464.862.875.985.278.089.953.667.866.48
MaPLe67.620.546.961.980.789.371.693.049.771.065.22
ProGrad67.522.356.764.078.387.082.292.452.171.467.39
CLIP-LoRA68.624.180.864.876.386.684.793.757.475.471.24
TOGA68.426.370.165.881.089.388.295.056.473.571.40
SD-LoRA (Ours)69.929.278.561.581.789.487.893.255.474.672.12
4CoOp (4)68.222.162.165.276.590.184.193.155.373.569.02
CoOp (16)67.124.068.766.275.688.887.993.055.375.070.16
CoCoOp67.817.358.562.081.189.874.693.252.371.666.82
TIP-Adapter-F68.328.871.567.680.988.688.994.658.075.172.23
CLIP-Adapter65.621.349.962.281.388.468.392.047.267.364.35
PLOT++69.226.281.668.577.889.190.293.957.275.672.93
KgCoOp69.222.664.963.281.289.576.893.855.171.668.79
TaskRes66.723.170.766.776.786.779.090.657.068.668.54
MaPLe69.423.464.762.281.490.578.194.055.070.968.96
ProGrad69.624.863.766.479.289.487.593.255.973.470.31
CLIP-LoRA70.327.785.668.375.686.390.194.360.376.573.50
TOGA70.229.179.869.481.389.892.194.861.477.074.49
SD-LoRA (Ours)71.635.987.065.882.089.791.494.364.978.776.13
8CoOp (4)70.028.072.869.077.289.590.293.860.877.172.84
CoOp (16)69.229.176.471.376.387.492.793.861.776.573.44
CoCoOp68.920.358.163.481.690.177.393.857.472.468.33
TIP-Adapter-F71.232.175.072.681.389.890.494.563.678.074.85
CLIP-Adapter69.323.555.265.481.589.378.093.950.873.067.99
PLOT++71.031.787.173.578.288.493.894.462.979.176.01
KgCoOp69.524.766.265.081.790.383.194.561.174.771.08
TaskRes71.931.974.973.880.689.193.594.864.578.475.34
MaPLe70.325.479.063.781.990.981.194.459.875.072.15
ProGrad71.129.073.571.880.089.192.194.262.375.773.88
CLIP-LoRA72.136.188.874.476.787.792.494.863.780.176.68
TOGA72.535.482.673.581.389.795.295.165.980.277.10
SD-LoRA (Ours)72.942.890.971.482.791.094.694.769.280.879.10
16CoOp (4)72.131.180.671.779.889.493.195.064.378.275.53
CoOp (16)72.232.983.376.078.688.795.494.965.378.676.59
CoCoOp69.822.670.464.681.991.082.594.359.775.371.21
TIP-Adapter-F74.134.883.477.081.790.494.395.168.080.577.93
CLIP-Adapter71.826.764.768.981.990.188.794.858.176.572.22
PLOT++73.436.391.177.479.789.196.394.967.081.578.67
KgCoOp71.023.770.167.381.790.886.194.465.177.572.77
TaskRes73.637.077.778.081.489.495.595.768.380.677.72
MaPLe72.028.083.366.982.191.789.095.163.477.374.88
ProGrad73.233.381.076.180.189.395.195.065.879.676.85
CLIP-LoRA74.044.991.879.778.288.896.295.268.282.879.98
SD-LoRA (Ours)75.351.092.378.983.091.496.095.872.184.882.06

Table 9. Table 9: Few-shot results with CLIP ViT-B/32. Top-1 accuracy (%) for K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\} shots on 10 datasets (excluding ImageNet); Avg is their unweighted mean. Mohammad et al. [2026] results for 1/2/4/8 shots are reproduced from supplementary Table S2. Best results are in bold.

SD-LoRA attains the best average accuracy at all shot levels, outperforming CLIP-LoRA by +2.70, +0.88, +2.63, +2.42, and +2.08 percentage points for 1-, 2-, 4-, 8-, and 16-shot learning, respectively. Gains are largest on fine-grained or domain-specific datasets (Aircraft, EuroSAT, DTD, Flowers102, UCF101), where discrimination relies on localized shape, texture, or class-specific cues. These results mirror the ViT-B/16 findings and indicate that SD-LoRA is not tied to a particular backbone. The training-only graph teacher instead provides structural supervision that improves the LoRA-adapted representation across CLIP visual encoders.

Under ViT-B/32, the inference model is unchanged: after adaptation, the graph teacher and prototype head are removed, and predictions use only the LoRA-adapted CLIP encoder with the cache-based classifier. Thus, the improvements in Table 9 confirm that SD-LoRA enhances deployable adaptation via training-time structural supervision, without extra test-time computation.

Additional Cross-Dataset Generalization Results

Table 10 shows cross-dataset generalization when models are adapted on ImageNet and evaluated on ten target datasets without target-domain training, testing whether the adapted representation remains transferable beyond the source distribution.

MethodSourceTarget datasetsAvg.
ImageNetCaltechDTDEuroSATAircraftFoodFlowersPetsCarsSUNUCF
MaPLe72.3093.5346.4948.0624.7486.2072.2390.4965.5767.0168.6966.30
PromptSRC72.8093.6046.8745.5023.9086.1570.2590.2565.7067.1068.7565.81
PromptKD—93.6155.0863.7426.2488.8475.3391.5973.9368.5776.3971.33
Ours74.2095.1761.5569.0531.7686.9085.2592.4876.2372.9878.4374.98
Δ vs. PromptKD—+1.56+6.47+5.31+5.52-1.94+9.92+0.89+2.30+4.41+2.04+3.65

Table 10. Cross-dataset generalization from ImageNet. Models are adapted on ImageNet and evaluated directly on target datasets without target-domain training. The ImageNet column reports source-domain accuracy, while Avg. denotes the unweighted mean over the ten target datasets. Δ\Delta reports the absolute difference between SD-LoRA and PromptKD. Best results are shown in bold.

SD-LoRA attains the best average target accuracy, improving over PromptKD from 71.33% to 74.98% (+3.65), with gains on 9 of 10 datasets. The largest improvements are on Flowers (+9.92), DTD (+6.47), EuroSAT (+5.31), Aircraft (+5.52), and SUN397 (+4.41), indicating that training-time structural supervision particularly aids transfer when recognition relies on local shape, texture, or domain-specific cues. The only drop occurs on Food101 (−1.94), where PromptKD remains better. Overall, SD-LoRA enhances the deployable LoRA representation instead of over-specializing it to the source dataset.

Calibration under Cross-Dataset Generalization

Accuracy alone does not capture reliability under domain shift, so we evaluate calibration in a cross-dataset setting using expected calibration error (ECE), negative log-likelihood (NLL), and Brier score, where lower is better.

As shown in Table 11, SD-LoRA consistently improves calibration across all target datasets and metrics. Relative to PromptKD, average ECE drops from 5.57 to 2.65, NLL from 1.05 to 0.62, and Brier score from 0.38 to 0.20, with particularly large gains on challenging targets such as EuroSAT, Flowers, DTD, Aircraft, and UCF101. This indicates that the structural teacher not only boosts accuracy but also yields a better-calibrated decision function after adaptation. Because the graph teacher is discarded after training, these gains require no extra calibration module or test-time adaptation.

MetricMethodCaltechDTDEuroSATAircraftFoodFlowersPetsCarsSUNUCFAvg.
ECE↓PromptKD5.014.9617.413.042.005.495.477.941.492.935.57
ECE↓Ours1.993.546.832.321.563.400.833.641.281.132.65
NLL↓PromptKD0.241.860.972.460.391.480.250.791.120.911.05
NLL↓Ours0.161.180.281.700.280.320.230.760.780.510.62
Brier↓PromptKD0.110.620.460.810.160.330.110.370.440.340.38
Brier↓Ours0.070.340.130.440.090.090.100.350.310.120.20

Table 11. Calibration under cross-dataset generalization. After ImageNet-source adaptation, models are evaluated on ten target datasets. We report expected calibration error (ECE), negative log-likelihood (NLL), and Brier score. Lower values indicate better calibration. Best results are shown in bold.

Additional Ablation Studies

The main paper presents a compact ablation study separating the effects of the cache branch and the training-only graph teacher. Here, we add diagnostic ablations to clarify SD-LoRA’s design choices, examining graph edge families, Prototype Predictive Alignment (PPA), token filtering, input resolution, graph teacher architecture, and loss components. Unless noted, all ablations use the same few-shot protocol as the main experiments, and the graph teacher is discarded after training.

Representation Geometry of the Deployed Student

We analyze the 224 × 224 student after removing all teacher-specific components: the graph, high-resolution pathway, saliency readout, and PPA head. The comparator is a matched CLIP-LoRA+cache student trained without structural supervision. These diagnostics concern encoder representations, rather than graph-classifier outputs, and use evaluation labels only for post-hoc analysis. They do not introduce labels or teacher components into deployment.

We measure within-class feature dispersion DintraD_{\mathrm{intra}} (lower is better), separation between class prototypes DinterD_{\mathrm{inter}} (higher is better), the correct-class nearest-prototype margin, silhouette score, and Davies–Bouldin (DB) index. The silhouette score contrasts within-class and nearest-other-class distances, while DB summarizes within-class scatter relative to between-class separation; lower DB indicates more distinct clusters. Table 12 reports unweighted averages over Aircraft, Caltech101, EuroSAT, and Food101 at 1/4/8/16 shots. These 16 dataset–shot conditions are descriptive evaluation cells, not 16 independent random repetitions.

Deployed modelDintra↓D_{\mathrm{intra}} \downarrowDinter↑D_{\mathrm{inter}} \uparrowMargin↑\uparrowSilhouette↑\uparrowDB↓\downarrow
CLIP-LoRA+cache0.5600.7130.0750.1043.241
SD-LoRA student0.5110.8900.1560.1353.051

Table 12. Deployed-student representation geometry after teacher removal. Values are unweighted means over Aircraft, Caltech101, EuroSAT, and Food101 at 1/4/8/16 shots (16 conditions). Features are produced by the deployed 224 × 224 student, without the graph, high-resolution path, readout, or PPA head.

After teacher removal, within-class distance decreases from 0.560 to 0.511, inter-class distance increases from 0.713 to 0.890, and prototype margin increases from 0.075 to 0.156. The corresponding relative changes are approximately −8.8%, +24.8%, and more than a twofold margin increase. Silhouette rises from 0.104 to 0.135 and DB decreases from 3.241 to 3.051. SD-LoRA has lower DintraD_{\mathrm{intra}} and higher prototype margin and silhouette in all 16 conditions; inter-class distance and DB improve in 15/16.

The diagnostics show that the structural-training effect persists in the deployed representation, rather than being confined to the discarded graph head. They are consistent with the patch-attention visualization in Figure 1. Improved class geometry alone does not uniquely identify a relational mechanism: the matched controls in Appx. C.6 and relation-family ablations in Appx. C.2 provide complementary attribution evidence. The structural objective is auxiliary supervision through shared LoRA parameters, rather than conventional frozen-teacher logit matching.

Patch-level attention maps for input images, frozen CLIP, and distilled CLIP

Figure 1. Structural distillation sharpens token-level evidence. We visualize patch-level attention maps for frozen CLIP and SD-LoRA using cosine similarity between the final [CLS] token and patch tokens. Frozen CLIP attends strongly to background foliage and misclassifies the image as clematis, while SD-LoRA shifts focus toward discriminative flower regions and correctly predicts balloon flower. The graph teacher is used only during training; the visualization is produced from the deployed LoRA backbone without test-time graph computation.

Effect of Graph Edge Families

Table 13 evaluates the impact of each relation family in the training-only graph teacher. Using the full graph yields the best performance on Aircraft and Caltech101, showing that SD-LoRA benefits from jointly modeling spatial, global–local, and visual–text relations. Removing semantic visual–text edges causes the largest average drop, indicating that class-conditioned links between visual tokens and text embeddings are key to structural supervision. Removing global–local edges also substantially hurts performance, confirming that interactions between the [CLS] token and patch tokens are important for spreading image-level context to fine-grained evidence. In contrast, removing spatial patch–patch edges yields a smaller but consistent drop, implying that local patch spatial structure helps but is less critical than semantic grounding.

The last two rows isolate the two semantic relation types. Keeping only CLS–text edges outperforms keeping only patch–text edges, suggesting that global class-level alignment is the strongest semantic cue. Still, the full graph is best, indicating that patch-level semantics provide complementary fine-grained evidence on top of CLS–text grounding.

Averaged over Aircraft and Caltech101 at 1/4/16 shots, removing visual–text relations reduces accuracy by 3.22 pp, removing global–local relations by 2.68 pp, and removing spatial relations by 1.11 pp. The hierarchy supports the design principle of class-semantic grounding, global context, and complementary local continuity; it is empirical evidence for this construction rather than a theoretical uniqueness claim.

Effect of PPA and the Graph Teacher

Table 14 disentangles the effects of Prototype Predictive Alignment (PPA) from the training-only graph teacher. All variants use LoRA and cache-based inference, so the comparison isolates the impact of high-resolution adaptation, prototype alignment, graph-guided training, and their combinations.

The graph teacher is the main driver of gains: relative to the HR-only baseline, adding the graph consistently improves performance across datasets and shot settings, showing that token-level relational reasoning provides a strong structural signal for the shared LoRA backbone. In contrast, PPA alone yields only small and sometimes inconsistent gains, indicating that prototype alignment is weaker without graph-conditioned features. Yet, combining PPA with the graph teacher yields the best results in all settings, suggesting that PPA works best as a complementary objective: the graph teacher first produces relation-aware visual evidence, and PPA then aligns this refined representation with support-derived class prototypes.

Both the graph teacher and PPA head are discarded after adaptation, so all gains are baked into the deployed LoRA+cache student with no extra inference-time cost.

Quantifying the conditional role of PPA. Across the nine dataset–shot cells in Table 14 (Aircraft, Caltech101, and EuroSAT at 1/4/16 shots), PPA without graph-conditioned representations changes the HR-only baseline by −0.29 pp on average. The graph without PPA contributes +1.68 pp, and adding PPA on top of the graph contributes a further +0.76 pp; the full model improves over HR-only by approximately +2.43 pp. Thus, PPA is a complementary regularizer of graph-conditioned evidence, not a consistently beneficial standalone branch. The small difference between sums of rounded component gains and the rounded total reflects rounding.

Effect of Token Filtering

Table 15 evaluates the readout strategy of the training-only graph teacher. The results show that which tokens are aggregated matters, not just how information propagates through the graph. Global pooling over all tokens consistently underperforms saliency-based Top-MM filtering, supporting our hypothesis that indiscriminate averaging mixes discriminative object regions with background or weakly informative tokens, weakening the structural signal transferred to the shared LoRA backbone.

A moderate number of high-saliency tokens works best. Top-MM=50 yields the strongest and most stable performance across datasets and shot settings. On EuroSAT, Top-MM=50 improves 4-shot accuracy from 89.5% (global pooling) to 91.3%, and also performs best at 8- and 16-shot. Similar but smaller gains appear on Caltech101 and Food101, which are less affected by background clutter or strong texture ambiguity.

The ablation reveals a selection trade-off. Too few tokens (Top-MM=25) remove useful context, while too many (Top-MM=75) reintroduce low-saliency, noisy tokens. On EuroSAT, Top-MM=75 drops to 88.8% at 4-shot versus 91.3% for Top-MM=50. Overall, the graph teacher benefits from foreground-focused aggregation: it preserves discriminative local evidence while avoiding the dilution of global averaging. Because the readout is used only during training, this structural guidance is distilled into the deployed CLIP-LoRA backbone without adding inference-time overhead.

Effect of Input Resolution

Table 16 analyzes the effect of input resolution on structure-aware adaptation. Because the graph teacher operates on ViT tokens, higher resolution yields a denser token grid and finer local cues for relational reasoning. Relative to the CLIP pretraining resolution of 224 × 224, 336 × 336 gives modest but non-uniform changes: some dataset–shot conditions improve while others decrease. The matched controls in Appx. C.6 isolate this resolution effect from cache and structural supervision. At 16-shot, accuracy rises from 96.9% to 97.2% on Caltech101, from 93.9% to 94.0% on EuroSAT, and from 87.1% to 87.8% on Food101.

Further increasing the resolution to 512 ×\times 512 consistently hurts performance, indicating that excessive upscaling mismatches the CLIP pretraining regime and destabilizes token-level interactions during graph propagation. The drop is most pronounced on Caltech101, where 16-shot accuracy falls from 97.2% at 336 ×\times 336 to 95.9% at 512 ×\times 512, and on Food101, where it falls from 87.8% to 85.9%. Overall, 336 ×\times 336 offers the best trade-off: it increases spatial granularity for the training-only graph teacher while preserving the stability of pretrained CLIP features.

Matched Resolution, Cache, and Structural-Training Controls

To separate the contributions of higher-resolution adaptation, cache-based prediction, and structural supervision, we evaluate six configurations on Aircraft, Caltech101, EuroSAT, and Food101 at 1/4/8/16 shots. Each cell is the unweighted mean across these four datasets, using identical support splits and three runs per configuration. Backbone, data partitions, support images, preprocessing, training schedule, and shared optimization settings are matched. The labels @224 and @336 denote adaptation resolution, not deployment resolution: all configurations use the original 224 ×\times 224 inference encoder after adaptation. The SD-LoRA inference path retains the same cache as the corresponding LoRA+cache control.

Training configuration1-shot4-shot8-shot16-shot
CLIP-LoRA @22470.175.278.581.9
CLIP-LoRA @33670.776.879.782.8
CLIP-LoRA+cache @22470.877.279.882.9
CLIP-LoRA+cache @33671.378.580.783.4
SD-LoRA @22472.679.381.984.8
SD-LoRA @33673.180.282.485.1

Table 17. Full matched comparison of training resolution, cache, and structural supervision. Top-1 accuracy (%) is averaged over four datasets. The graph teacher and PPA head are absent from every deployed SD-LoRA model.

Increasing LoRA+cache training resolution from 224 to 336 changes accuracy by +0.5 / + 1.3 / + 0.9 / + 0.5 pp. At matched 336 resolution and cache, the training-only structural branch adds +1.8 / + 1.7 / + 1.7 / + 1.7 pp. At 336, adding the cache to vanilla CLIP-LoRA contributes +0.6 / + 1.7 / +1.0 / + 0.6 pp. Finally, SD-LoRA trained at 224 exceeds the cache-equipped 336 control by +1.3 / + 0.8 / + 1.2 / + 1.4 pp. These controlled differences support an effect of structural training beyond either the cache or a denser token grid. They concern the four-dataset study above and should not be conflated with the 11-benchmark complete-system gains.

All 336-resolution variants use the same bicubic interpolation of the pretrained patch positional grid from 14×1414 \times14 to 21×2121 \times21, retaining the original [CLS] positional embedding. Higher-resolution adaptation provides more patch nodes for the teacher, rather than an additional inference path. Table 16 further shows that resolution is not uniformly beneficial: 336 sometimes reduces accuracy, and 512 degrades performance in the evaluated settings. Thus, the matched controls isolate interpolation as well as resolution, without claiming that upscaling alone explains the structural-training gains.

Graph Teacher Architecture Ablation

Table 18 ablates the main components of the training-only graph teacher: the visual refinement Transformer (Trans), the relation-aware graph Transformer (RGT), and discriminative node filtering. The full teacher consistently performs best on Aircraft, Caltech101, and EuroSAT, showing that all three provide complementary structural supervision.

Removing Trans hurts performance, especially at higher shots on Aircraft and EuroSAT, indicating that early token refinement stabilizes local visual features before relation-aware message passing. Removing RGT causes a larger drop, confirming that explicit relation-aware propagation is central: without typed visual–visual and visual–text interactions, the model struggles to encode structured evidence. Removing discriminative node filtering also reduces accuracy, indicating that suppressing weak or background-dominated tokens improves the graph-level training signal.

Combined ablations reinforce this view: variants with only one or two components are consistently weaker than the full architecture, with the largest drops when RGT or filtering is missing. Thus, SD-LoRA gains from the interaction of token refinement, relation-aware reasoning, and foreground-focused readout, rather than any single addition. Because the graph teacher is used only during adaptation, these designs strengthen supervision of the shared LoRA backbone without increasing inference-time complexity.

Loss Component Ablation

Table 19 evaluates the impact of the three training losses: deployed student loss LStudent\mathcal{L}_{\mathrm{Student}}, graph-teacher loss LTeacher\mathcal{L}_{\mathrm{Teacher}}, and Prototype Predictive Alignment LPPA\mathcal{L}_{\mathrm{PPA}}. The student-only objective is a strong baseline because it directly optimizes the LoRA+cache predictor used at inference, but its supervision operates mainly at the image-logit level and does not explicitly enforce token-level structure.

Adding graph-teacher loss improves performance in several cases (e.g., Caltech101 and Food101), showing that structure-aware supervision benefits the shared LoRA backbone. The conditional contribution of PPA is quantified separately in Appx. C.3; it is most effective as a complement to graph-conditioned supervision.

Prototype Predictive Alignment offers a more stable complementary signal. Relative to the student-only objective, adding LPPA\mathcal{L}_{\mathrm{PPA}} improves EuroSAT from 93.4% to 93.8% at 8-shot, Caltech101 from 93.9% to 95.5% at 4-shot, and Food101 from 86.9% to 87.6% at 8-shot. This indicates that prototype-level supervision better anchors the adapted representation to the support-induced class structure, beyond individual instance predictions.

The full objective, combining LStudent\mathcal{L}_{\mathrm{Student}}, LTeacher\mathcal{L}_{\mathrm{Teacher}}, and LPPA\mathcal{L}_{\mathrm{PPA}}, yields the best performance in all settings. This validates implicit structural distillation in SD-LoRA: the graph teacher provides relation-aware signals, and PPA stabilizes them by enforcing prototype-consistent geometry in the shared CLIP-LoRA feature space. Because all teacher components are discarded after training, the gains fully transfer to the deployed LoRA+cache student without logit matching or test-time graph computation.

Cross-modal Interaction Frequency

Why cross-modal interaction frequency matters? The SD-LoRA graph teacher refines local visual evidence using both visual structure and class semantics. Interaction placement specifies at which message-passing layers visual–text edges are active: all layers activates them at every layer; alternate at alternating layers; first only only in the first layer; last only only in the final layer; and none removes visual–text interaction, leaving purely visual message passing. Figure 5 reports Aircraft performance under 1/4/16-shot settings, where subtle local evidence is crucial. Marker height gives the best accuracy for each strategy; the label above each marker shows the epochs needed to reach 95% of that peak, separating final performance from optimization speed.

Figure 5. Effect of cross-modal interaction frequency. We ablate where semantic text–visual interaction is applied across graph layers on Aircraft under 1/4/16-shot settings. Each marker reports the peak accuracy, while the text label indicates the number of epochs required to reach 95% of the peak. Applying interaction in all layers consistently achieves the best peak accuracy across shot regimes and also converges efficiently in the 16-shot setting, reaching 95% of peak in 13 epochs compared with 18 epochs for last only and 22 epochs for none. This suggests that frequent cross-modal interaction improves both final adaptation quality and optimization efficiency.

Applying text–visual interaction in all layers is consistently best across all shot counts, indicating that gains come from repeated semantic–visual exchange during message passing, not just adding text nodes. First-only interaction injects semantics before visual features are well refined, while last-only waits too long to ground them. The none variant performs worst and converges slowest in the 16-shot case, showing that visual-only structure is inadequate for this fine-grained task. Thus, the ablation supports SD-LoRA’s central choice: frequent class-conditioned interaction during training yields a stronger LoRA, even though the graph is discarded at inference.

Computational Complexity Analysis and Hyperparameter Optimization

Table 20 separates the cost of high-resolution structural training from that of the graph branch. SD-LoRA uses a 336×336 CLIP forward pass only during adaptation, letting the training-only teacher see a denser ViT token grid and reason over finer local evidence. This addresses few-shot fine-grained tasks, where discriminative cues often come from small parts or textures that may vanish when adaptation relies only on a standard 224×224 global representation. Instead of multiple manual crops or repeated region encodings, SD-LoRA obtains richer spatial evidence with a single high-resolution ViT-B/16 pass, then applies graph reasoning over the resulting patch tokens.

Component / SettingParams (M)Trainable (M)GFLOPsRole
Standard adaptation reference
CLIP-LoRA, 224×224224 \times224152.081.2317.50Standard train/infer path
SD-LoRA training path
Frozen CLIP backbone, 336×336336 \times336150.850.0046.73HR token extraction
+ LoRA1.231.23≈0.00\approx0.00Shared deployed update
Cache head0.050.05≈0.00\approx0.00Deployed classifier
Graph reasoning teacher12.1612.160.61Training only
– Token projection2.102.100.47Training only
– Relation-aware graph layers6.896.890.14Training only
– Saliency/readout/projection heads3.173.17<0.01<0.01Training only
Total SD-LoRA training system164.2913.4447.34Adaptation only
Graph-only overhead vs. 336×336336 \times336 CLIP––+1.31%Teacher branch cost
Full training cost vs. 224×224224 \times224 CLIP-LoRA––2.71×2.71 \timesHR training cost
Deployed SD-LoRA inference path152.131.28 retained17.50 + cacheNo graph / no HR teacher

Table 20. Computational analysis. GFLOPs are reported per image using the same profiler. SD-LoRA separates training-time structural supervision from deployment. The 336×336336 \times336 pass is used only during adaptation to expose a denser ViT token grid for fine-grained graph supervision. The graph branch itself adds 0.61 GFLOPs on top of this high-resolution pass, corresponding to 1.31% overhead relative to the 336×336336 \times336 backbone pass. At inference, the graph reasoning module, saliency scorer, PPA head, and high-resolution teacher path are removed, reducing deployment to the 224×224224 \times224 CLIP-LoRA+cache student.

The main extra training-time cost is high-resolution token extraction, which increases the backbone from 17.50 GFLOPs at 224×224 to 46.73 GFLOPs at 336×336. On top of this, the graph reasoning teacher adds only 0.61 GFLOPs (1.31% of the 336×336 CLIP pass), so the relation-aware graph module remains lightweight; the dominant cost is the one-time extraction of denser tokens during adaptation. This matters because the graph branch is never deployed; it serves only as a training-time source of token-level structural supervision for the shared LoRA parameters.

The full training system has 164.29M parameters, with 13.44M trainable. Most trainable parameters lie in the training-only graph teacher (12.16M), while the deployed LoRA and cache use just 1.28M trainable parameters. After adaptation, the high-resolution teacher path, graph module, saliency scorer, and PPA head are discarded. Inference uses only the standard 224×224224 \times224 CLIP-LoRA encoder plus the cache head, with no graph construction or high-resolution computation. SD-LoRA thus trades a higher one-time adaptation cost for fine-grained structural supervision in the deployable LoRA backbone, while keeping inference lightweight for practical use.

Figure 6. A shows the one-time wall-clock overhead associated with the additional high-resolution training and re-encoding passes. B reports the computational overhead relative to the CLIP-LoRA training system, showing that the added graph branch increases computation by only 0.61 GFLOPs (+1.31%) while keeping parameter growth modest (+8.0%). C summarizes practical deployment-time metrics on ViT-B/16, including training time, memory footprint, and inference latency. D shows that the method remains well behaved across backbone choices: ViT-B/32 retains competitive average few-shot accuracy, while the higher cost of ViT-L/14 is primarily attributable to the larger backbone rather than to additional method-specific steps.

Table 21 summarizes the SD-LoRA hyperparameter search space, showing that adaptation uses a compact set of interpretable knobs. Most parameters follow standard optimization and PEFT practice (learning rate, weight decay, and a LoRA rank candidate set r∈{2,4,8,16}r \in\{2, 4, 8, 16\} with αLoRA∈{8,16,32}\alpha_{\mathrm{LoRA}} \in\{8, 16, 32\}). The remaining freedom lies in balancing three training signals: graph supervision (λ1\lambda_1), prototype alignment (λ2\lambda_2), and hard-example emphasis via focal loss (γ\gamma) plus a single inference-time cache weight (ω\omega). The ranges are chosen for stability: λ2∈[0.1,1.0]\lambda_2 \in[0.1, 1.0] regularizes rather than dominates learning, while λ1\lambda_1 spans a wider range to accommodate dataset granularity and structural cue strength. Overall, the search is tractable (few hyperparameters, narrow PEFT range) and cleanly decouples training-time structural guidance from deployment, since only ω\omega affects inference.

Selected Configuration and Model-Selection Protocol

The search space in Table 21 is distinct from the selected configuration in Table 22. Bayesian optimization uses mean validation top-1 accuracy over a predefined set of six representative datasets and all shot regimes to choose one global configuration. That configuration is then used unchanged across all 11 datasets and shot settings; there is no per-dataset or per-shot hyperparameter selection. The wide learning-rate search interval does not imply that the reported models use large learning rates.

HyperparameterSymbolSearch RangeScale
Optimizer & Backbone
Learning Rateη\eta[10−5,5×10−2][10^{-5}, 5 \times10^{-2}]Log-Uniform
Weight Decaywdw_d[10−6,10−3][10^{-6}, 10^{-3}]Log-Uniform
LoRA Rankrr[2,4,8,16][2, 4, 8, 16]Integer
LoRA AlphaαLoRA\alpha_{\mathrm{LoRA}}{8,16,32}\{8, 16, 32\}Discrete
Training Objectives
Graph Loss Weightλ1\lambda_1[0.1,5.0][0.1, 5.0]Uniform
PPA Alignment Weightλ2\lambda_2[0.1,1.0][0.1, 1.0]Uniform
Focal Loss Gammaγ\gamma[0.5,3.0][0.5, 3.0]Uniform
Inference
Inference Cache Weightω\omega[1.0,10.0][1.0, 10.0]Uniform

Table 21. Hyperparameter Optimization Space. The search selects one global configuration on validation data; selected values are reported in Table 22 and are fixed across datasets and shots.

HyperparameterSelected valueCandidates / search range
Shared LoRA learning rate1×10−51 \times10^{-5}[10−5,5×10−2][10^{-5}, 5 \times10^{-2}], log-uniform
Teacher/head learning rate1×10−51 \times10^{-5}[10−5,5×10−2][10^{-5}, 5 \times10^{-2}], log-uniform
Weight decay1×10−41 \times10^{-4}[10−6,10−3][10^{-6}, 10^{-3}], log-uniform
LoRA rank rr4{2,4,8,16}\{2, 4, 8, 16\}
LoRA alpha αLoRA\alpha_{\mathrm{LoRA}}8{8,16,32}\{8, 16, 32\}
Graph-loss weight λ1\lambda_11.0[0.1,5.0][0.1, 5.0], uniform
PPA weight λ2\lambda_20.5[0.1,1.0][0.1, 1.0], uniform
Focal-loss parameter γ\gamma2.0[0.5,3.0][0.5, 3.0], uniform
Cache weight ω\omega5.0[1.0,10.0][1.0, 10.0], uniform

Table 22. Selected SD-LoRA hyperparameters. These values are fixed globally after validation selection. Search ranges describe the selection procedure, not uncertainty in the final evaluation settings.

Support images are sampled only from training partitions. Model selection uses validation data, and test labels are not used for optimization or selection. Controlled comparisons, including CLIP-LoRA, CLIP-LoRA+cache, and SD-LoRA, share the same support samples, partitions, preprocessing, training schedules, validation criterion, tuning budget, and fixed shared hyperparameters; only the component being tested changes. Each shot setting uses three independently sampled support sets, with each controlled variant adapted separately on the same sets.

Published external baselines are taken from their original papers, rather than rerun or retuned. They provide literature-level comparisons under the reported benchmark protocols; we do not claim identical optimization settings or an equal search budget for those external results. The matched controls, rather than the full literature table alone, support attribution to structural training.

We report main few-shot means and SDs over the three support-split runs. The matched-control and deployed-feature tables report the aggregate values available for those studies; they do not imply new significance tests or per-condition confidence intervals. Protocol-specific differences in support-label access, validation access, and inference-time adaptation must be considered when interpreting comparisons.

Reproducibility and statistical reporting. For the main few-shot experiments, each result is the mean accuracy over three independently sampled support splits. Uncertainty values are the standard deviation across these runs, reflecting variation from support-set choice and stochastic optimization under a fixed evaluation protocol. Aggregate results are unweighted arithmetic means over datasets. For base-to-novel, cross-dataset, and ImageNet-OOD evaluations, we follow the standard fixed protocols from prior CLIP adaptation work and report point estimates unless noted otherwise. Hyperparameters are chosen using only training/validation data or support-split cross-validation; test labels are never used for model selection.

Scope, Limitations, and Responsible Use

SD-LoRA targets inductive few-shot adaptation of CLIP-like vision–language models with a small labeled support set and lightweight inference. It is best suited to settings where deployment must be efficient but training-time cost can be moderately higher. Its key benefit is a relation-aware graph teacher used only during adaptation and discarded afterward, improving the deployed LoRA+cache student without adding graph computation at test time; this also limits its applicability.

Training-time overhead. Despite no graph cost at inference, SD-LoRA is more expensive to adapt than standard CLIP-LoRA or cache-only methods because the teacher processes high-resolution visual tokens and class-text nodes. This is acceptable for offline few-shot adaptation on a small support set, but less suitable for frequent on-device updates or rapid continual adaptation.

Dependence on support-set quality. SD-LoRA relies on a high-quality few-shot support set. If labels are noisy, imbalanced, or visually unrepresentative, both the cache and the prototype-alignment objective can encode misleading class structure. The graph teacher refines local evidence but cannot compensate for unrepresentative support examples.

Backbone and modality scope. We study CLIP-style image–text models on standard few-shot benchmarks. The structural distillation idea is general, but our implementation relies on ViT token representations and class-text embeddings. Applying it to dense prediction, video, open-vocabulary detection, or multimodal temporal reasoning may require new graph constructions and task-specific readouts.

Resolution and graph design sensitivity. The Appendix shows that moderately higher-resolution tokenization helps, but excessively high resolution hurts due to mismatch with CLIP pretraining. Graph edge choices and token filtering also influence performance. Thus, structural supervision is beneficial but constrained: effective graph design must balance local detail, semantic grounding, and pretrained feature stability.

Responsible use. SD-LoRA improves adaptation efficiency and transferability, aiding applications with limited labeled data. Like other CLIP-based methods, it inherits biases and failure modes from the pretrained vision–language model and the support data. In sensitive domains (e.g., medical, forensic, public safety), SD-LoRA must not serve as an autonomous decision-maker; it should be evaluated with domain-specific protocols, carefully calibrated, and used only with human oversight and safeguards.

Existing assets, licenses, and data access. SD-LoRA relies on public benchmark datasets and pretrained CLIP-style vision–language models. We cite the original sources for all datasets, pretrained models, and baselines. We do not redistribute datasets or model weights; users must obtain them from official sources and follow the respective licenses or terms of use. Any released implementation will provide instructions for acquiring or preparing public datasets but will not repackage restricted or third-party assets.

References

  1. [1]Tasweer Ahmad, Arindam Sikdar, Sandip Pradhan, and Ardhendu Behera. Advancing cache-based few-shot classification via patch-driven relational gated graph attention. arXiv preprint arXiv:2512.12498, 2025.arxiv.org/abs/2512.12498
  2. [2]Yassir Bendou, Amine Ouasi, Vincent Gripon, and Adnane Boukhayma. Proker: A kernel perspective on few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25092–25102, 2025.arxiv.org/abs/2501.11175
  3. [8]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581–595, 2024.arxiv.org/abs/2110.04544
  4. [9]Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, and Ramtin Pedarsani. Few-shot adversarial low-rank fine-tuning of vision-language models. arXiv preprint arXiv:2505.15130, 2025.arxiv.org/abs/2505.15130
  5. [11]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  6. [12]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.arxiv.org/abs/2106.09685
  7. [13]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021.arxiv.org/abs/2102.05918
  8. [15]Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113–19122, 2023a.arxiv.org/abs/2210.03117
  9. [16]Muhammad Uzair Khattak, Syed Talal Wasim, Muzzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15190–15200, 2023b.arxiv.org/abs/2307.06948
  10. [18]Alexey Kravets, Da Chen, and Vinay P Namboodiri. Rethinking few shot clip benchmarks: A critical analysis in the inductive setting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1902–1911, 2025.arxiv.org/abs/2507.20834
  11. [20]Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuан Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965–10975, 2022.arxiv.org/abs/2112.03857
  12. [21]Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36: 13448–13466, 2023.arxiv.org/abs/2309.13625
  13. [22]Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26617–26626, 2024.arxiv.org/abs/2403.02781
  14. [23]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  15. [28]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
  16. [29]Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18082–18091, 2022.arxiv.org/abs/2112.01518
  17. [30]Shuyendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2306.01195
  18. [35]Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson, Yingzhen Li, and Chang D. Yoo. C-TPT: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In The Twelfth International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2403.14119
  19. [37]Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1593–1603, 2024.arxiv.org/abs/2405.18541
  20. [38]Maxime Zanella, Benoît Gérin, and Ismail Ayed. Boosting vision-language models with transduction. Advances in Neural Information Processing Systems, 37:62223–62256, 2024.arxiv.org/abs/2406.01837
  21. [39]Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25103–25112, 2025.arxiv.org/abs/2501.03729
  22. [40]Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaptation of clip for few-shot classification. In European conference on computer vision, pages 493–510. Springer, 2022.arxiv.org/abs/2207.09519
  23. [42]Yumiao Zhao, Bo Jiang, Xiao Wang, Qin Xu, and Jin Tang. Hegraphadapter: Tuning multi-modal vision-language models with heterogeneous graph adapter. arXiv preprint arXiv:2410.07854, 2024.arxiv.org/abs/2410.07854
  24. [43]Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16793–16803, 2022.arxiv.org/abs/2112.09106
  25. [45]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022b.DOI

Paper details

vlnc:W_zba3hx6x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents