betaFeedback

WorksComputer Science

Drift-adaptive Representation Preservation for Domain-Incremental Learning

Abstract

Pretrained vision models provide strong general-purpose representations, but in Domain-Incremental Learning (DIL), where the input distribution changes sequentially, adapting to new domains often degrades performance on previously learned ones. We observe that forgetting is not uniform across domains and classes and is not explained by training order alone. This motivates us to track each past domain-class concept individually. We use representation drift, defined as the deviation of a past concept's current representation from a fixed anchor, as a measurable surrogate for forgetting risk. The measured drift then serves two complementary roles. A consolidation loss assigns stronger preservation to high-drift pairs, while prototype construction down-weights highly drifted domains to reduce their influence on the class reference. Preservation is thus allocated by measured drift rather than assumed importance. To obtain this signal during training without storing past raw data, we repurpose a Joint-Embedding Predictive Architecture (JEPA). For each domain-class pair, we store only a compact context summary in the backbone feature space and rerun it through the learned context-to-target relation to measure its drift from the fixed anchor. Across DomainNet, Office-Home, CORe50, and CDDB-Hard, our method sustains a strong accuracy-forgetting balance on every benchmark, using substantially less training memory than accuracy-competitive methods and the smallest storage among compared methods.

Introduction

Pretrained vision models provide strong, broadly transferable representations and have substantially expanded the practical promise of DIL [21] [26]. In DIL, a model learns a sequence of domains that share a label space but differ in input distribution, and it must maintain stable performance on all domains seen so1 far without revisiting past data [22] [23]. The central difficulty is the stability–plasticity dilemma: adapting the representation to a new domain disturbs the structure formed on past domains and induces forgetting, whereas suppressing representational change too strongly limits adaptation [12] [24].

We study forgetting in DIL through the lens of representation drift. As the model learns a domain, the resulting per-class representation serves as a fixed reference point, or anchor, against which later changes can be measured. When the model subsequently adapts to new domains, a previously learned concept may move away from this reference. We use this deviation as a measurable signal of how strongly the concept has changed and as a surrogate for its forgetting risk. Crucially, these changes are not uniform across previously learned domains and classes. As shown in Figure 1, the amount of forgetting varies substantially across domains within the same training sequence and cannot be characterized by training order alone. This heterogeneous behavior motivates us to track representation drift separately for each domain–class pair rather than relying on a single global measure. This nonuniformity creates a specific problem for prototype-based classification: a pair with large drift has likely departed from the stable class semantics, and combining it with equal weight when forming the class prototype lets the distorted representation contaminate the class reference itself—an effect we call prototype contamination.

Per-domain forgetting when DINOv2 is sequentially fine-tuned on DomainNet, with example images of the same class across domains

Figure 1. Per-domain forgetting when DINOv2 is sequentially fine-tuned on DomainNet. Example images of the same class (e.g., crab) illustrate the visual variation across domains.

Motivated by this, we use drift in two ways at once. First, we use drift as a proxy for forgetting risk: a pair with larger drift receives stronger preservation because its current reconstruction is less aligned with the fixed class anchor. Second, drift is a reliability signal for prototype construction: a pair with small drift is trusted, whereas a pair with large drift is down-weighted, shielding the class reference from contamination. A single measured drift signal thus plays two complementary roles, so that preservation is allocated according to measured drift rather than assumed importance.

Using drift this way has a prerequisite: it must be measurable at the domain–class level, during training, and without access to past raw data. We observe that JEPA [1] provides a natural mechanism for this purpose. Because it learns, in latent space, to predict one observed cue from another, a past concept can be reexamined by storing only a compact context summary for each domain–class pair and rerunning it through this learned relation. Concretely, a frozen backbone’s CLS-to-patch attention splits each image into a target carrying class-relevant evidence and a context capturing domain-sensitive cues; the model learns to predict the target from the context, and stored past-domain context summaries are re-input to the current encoder and predictor to measure how far the prediction departs from the fixed anchor—this distance is the drift signal.

This design also brings a concrete efficiency benefit: no gradient flows through the frozen backbone, so training is confined to a lightweight encoder and predictor and the training memory footprint remains small, while past information is stored only as compact per-domain–class summaries, so storage grows only mildly with the number of domains. Our contributions are summarized as follows:

  • We propose a preservation framework in which a single measured drift signal drives two complementary mechanisms—a drift-adaptive consolidation loss that concentrates preservation on high-drift concepts, and a robust prototype construction that down-weights high-drift domains to prevent prototype contamination—allocating preservation by measured drift rather than assumed importance.

  • To make this signal available at the domain–class level during training, without storing any past image, we repurpose JEPA’s learned context-to-target relation as a drift meter, rerunning a stored domain–class context summary through the current encoder and predictor.

  • Across four DIL benchmarks, our method sustains a strong accuracy–forgetting balance on every benchmark while using substantially less training memory than accuracy-competitive methods and the smallest storage among the compared methods.

Related Work

Domain-Incremental Learning

Continual learning studies how a model can absorb new data distributions without forgetting previously learned ones. In domain-incremental learning, all domains share the same label space while the input distribution shifts sequentially, so the central difficulty is to keep past domains from degrading as the representation adapts to new ones. Rehearsal-based methods address this by storing or synthesizing past samples for replay [18] [2]. They are effective, but their storage cost grows with the data, they raise privacy concerns, and they are inapplicable when past images cannot be retained [27] [4]. Regularization-based methods instead constrain parameters or outputs deemed important for earlier tasks from changing substantially, as in EWC and LwF [11] [8]. Such preservation, however, is driven by an importance that is assumed or estimated globally, rather than by measuring which domain–class concepts are actually drifting under the current model. Recent domain-incremental and pretrained-model-based continual learning methods reduce reliance on raw data replay by keeping the pretrained backbone frozen and adapting to new domains through a small number of additional parameters or memory. Prompt-based methods such as CODA-Prompt [19], OS-Prompt [7], and OS-Prompt++ [7] adapt by accumulating or selecting prompt parameters per domain. In contrast, DIDD [15], GraphKeeper [6], SOYO [23], DUCT [26], and MoE [25] preserve or route domain knowledge through different mechanisms—feature- and graph-based memories, selective routing over domain-specific parameters or paths, backbone adaptation, and expert modules. Our method instead keeps the pretrained backbone frozen and stores only a compact summary per domain–class pair, using measured drift to decide what to preserve and which contributions to trust when forming each prototype.

Representation Drift and Predictive Representations

The instability of learned representations under sequential training is a well-recognized difficulty in continual learning. As a model adapts to new data, the features of previously learned concepts may move, causing stored statistics such as class prototypes to fall out of alignment with the current feature space—an effect often described as representation, feature, or semantic drift [10] [5] [4]. Prior work has mainly treated this drift as an error to be compensated for or suppressed, for example by estimating the displacement of stored prototypes or by stabilizing old representations through feature alignment and distillation [14]. In contrast, we use drift before suppressing it: we measure it at the domain–class level and use its magnitude as a nonuniform signal that determines both how strongly each concept should be preserved and how much each domain should contribute to the class prototype. Our drift measurement is enabled by a latent predictive relation inspired by JEPA, which learns representations by predicting the embedding of one part or view of an input from another in latent space, typically using an online encoder and an exponential moving average target encoder. In their original use, this predictive objective is designed for self-supervised representation learning. We repurpose the same principle for continual learning: the context-to-target relation trained on the current domain is reapplied to stored past-domain context summaries, and the predicted targets are compared with fixed class anchors to measure drift. Measurement and learning therefore share one function rather than relying on a separate probe, and the resulting drift signal drives both drift-adaptive consolidation and robust prototype construction. This use of a predictive relation as a per-concept meter of drift, with a single signal serving both roles, sets our approach apart from prior drift compensation and pretrained-model-based continual learning methods.

Proposed Method

Problem Formulation

In DIL, a model is trained on TT domains in sequence. Each domain t∈{1,…,T}t \in\{1,\ldots,T\} is associated with a distribution Pt(X,Y)\mathcal{P}_t(X,Y), from which a training set Dt=\mathcal{D}_t = {(xit,yit)}i=1Nt\{(x_i^t, y_i^t)\}_{i=1}^{N_t} is sampled, where xitx_i^t is an input image, yit∈Yy_i^t \in\mathcal{Y} is its label, and NtN_t is the number of samples in domain tt. All domains share a common label space Y={1,…,C}\mathcal{Y} = \{1,\ldots,C\} with CC classes, while the input distribution Pt(X)\mathcal{P}_t(X) differs across domains.

Overview of our framework

Figure 2. Overview of our framework. Evidence Decomposition splits each input image into a target and a context. Context-to-Target Prediction predicts the target from the context and performs classification. Drift-Adaptive Consolidation measures the drift from the fixed anchors using stored past-domain summaries while learning a new domain, and uses this signal to modulate the preservation strength and to weight each domain’s contribution when forming the class prototypes.

Training proceeds sequentially, one domain at a time: at step tt the model can access only the current domain DtD_t and cannot use the original images of the previous domains D1,…,Dt−1D_1,\ldots,D_{t-1}. Let Φt\Phi_t denote the model trained up to domain tt. The objective is to maximize the average performance over all domains seen so far,

A(Φt)=1t∑s=1tE(x,y)∼Ps[1(Φt(x)=y)],(1)A(\Phi_t) = \frac{1}{t}\sum_{s=1}^{t}\mathbb{E}_{(x,y)\sim\mathcal{P}_s}\left[\mathbf{1}(\Phi_t(x)=y)\right], \tag*{(1)}

where 1(⋅)\mathbf{1}(\cdot) is the indicator function and s≤ts \leq t indexes the domains seen so far. Keeping the objective in Eq. (1) without access to past data is the central challenge of DIL.

In this work, Φt\Phi_t is a prototype-based classifier that maps an input to a representation through a frozen pretrained backbone and a trainable encoder, and classifies it by the cosine similarity to each class prototype. Under this formulation, forgetting is reflected in the degradation of performance on previously learned domains. We track the underlying representational change through the deviation of each past concept from its fixed class anchor and use this representation drift as a measurable surrogate for forgetting risk. At each domain transition, before training on domain tt, we fix the representation of each class observed so far as a reference against which subsequent change is measured, and call it the anchor ac(t−1)a_c^{(t-1)} (constructed in § 3.3). As the encoder changes while later domains are learned, a past concept may be mapped to a location that departs from its anchor, and this distance serves as a measurable surrogate for the concept’s forgetting risk.

To make this distance concrete, we define the drift of class cc from domain ss, measured at step tt, as a nonnegative scalar δs,c(t)\delta_{s,c}^{(t)}. This construction has two important aspects. First, δs,c(t)\delta_{s,c}^{(t)} is obtained without any past image: a past concept is reconstructed through the current model from a stored domain–class context summary and compared against its anchor. Second, since the anchor is fixed per class, the per-domain difference in drift arises solely from the reconstructed past representation; drift is therefore not a single global quantity but one indexed by each domain–class pair (s,c)(s,c). We track these domain–class-specific representational changes through the per-pair surrogate δs,c(t)\delta_{s,c}^{(t)} during training, and use this signal to allocate preservation and construct robust prototypes.

Evidence Decomposition and Prediction

We now specify the representation used for classification, introduced abstractly in § 3.1, as two complementary cues derived from attention, and learn a predictive relation between them. This predictive relation later serves, in § 3.3, as the function that measures the drift of past concepts.

CLS-to-Patch Attention Passing an input xx through the frozen backbone f(⋅)f(\cdot) yields a CLS token and NN patch tokens. Let hn∈RDh_n \in\mathbb{R}^{D} be the nn-th patch token, where DD is the backbone token dimensionality, and let αn\alpha_n be the attention from the CLS token to that patch, normalized over the patches so that αn≥0\alpha_n \ge0 and ∑nαn=1\sum_n \alpha_n = 1; intuitively, αn\alpha_n reflects the regions that the pretrained backbone attends to more strongly and that therefore tend to carry the class-relevant evidence [3]. Using the attention weights αn\alpha_n and their complementary weights α~n\tilde{\alpha}_n, we aggregate the patch tokens into two representations,

uˉ=∑n=1Nαnhn,vˉ=∑n=1Nα~nhn,α~n=1−αn∑m=1N(1−αm),(2)\bar{u} = \sum_{n=1}^{N} \alpha_n h_n,\qquad \bar{v} = \sum_{n=1}^{N} \tilde{\alpha}_n h_n,\qquad \tilde{\alpha}_n = \frac{1-\alpha_n}{\sum_{m=1}^{N}(1-\alpha_m)}, \tag*{(2)}

which we then normalize to unit vectors,

uraw=uˉ∥uˉ∥2,vraw=vˉ∥vˉ∥2,uraw,vraw∈RD,(3)u^{\mathrm{raw}} = \frac{\bar{u}}{\lVert\bar{u}\rVert_2},\qquad v^{\mathrm{raw}} = \frac{\bar{v}}{\lVert\bar{v}\rVert_2},\qquad u^{\mathrm{raw}},v^{\mathrm{raw}} \in\mathbb{R}^{D}, \tag*{(3)}

where urawu^{\mathrm{raw}} is the target, formed from regions emphasized by the pretrained backbone and used as the classification cue, whereas vrawv^{\mathrm{raw}} is the context, formed with the complementary weights α~n\tilde{\alpha}_n and tending to capture cues sensitive to domain change, such as background, style, and texture.

Latent Encoding and Classification The two raw representations are passed through a shared trainable encoder gθ(⋅)g_{\theta}(\cdot) and mapped to the embedding space Rd\mathbb{R}^{d}:

u=gθ(uraw),v=gθ(vraw),u,v∈Rd,(4)u = g_{\theta}(u^{\mathrm{raw}}),\qquad v = g_{\theta}(v^{\mathrm{raw}}),\qquad u,v \in\mathbb{R}^{d}, \tag*{(4)}

where dd is the encoder output dimensionality, uu is the target representation used for classification, and vv is the context representation used as the input to prediction; the same embedding space is later used for drift measurement. Throughout, all encoder and predictor outputs are ℓ2\ell_2-normalized to unit vectors before any cosine comparison. As described in § 3.1, classification is performed via the cosine similarity between the target representation uu and the class prototype pc∈Rdp_c \in\mathbb{R}^d,

y^=arg⁡max⁡c∈Y cos⁡(u,pc).(5)\hat{y} = \underset{c \in\mathcal{Y}}{\arg\max}\ \cos(u, p_c). \tag*{(5)}

The prototype pcp_c is the embedding-space projection of a stored raw prototype μc∈RD\mu_c \in\mathbb{R}^D, whose construction and update we defer to § 3.3.

Context-to-Target Prediction The core of our framework is the relation that predicts the target from the context in the latent space. A predictor qϕ(⋅)q_\phi(\cdot), with trainable parameters ϕ\phi, estimates the target from the context representation:

u^=qϕ(v).(6)\hat{u} = q_\phi(v). \tag*{(6)}

The prediction is aligned to a teacher target u~\tilde{u}. The teacher is an exponential moving average (EMA) copy gξ(⋅)g_\xi(\cdot) of the encoder, whose parameters are updated at each training step as ξ←mξ+(1−m)θ\xi\leftarrow m\xi+ (1-m)\theta with EMA coefficient m∈[0,1)m \in[0,1), and no gradient flows along this path; using an EMA copy rather than the online encoder gives a slowly moving, stable target while gθg_\theta is still being learned. Using the label, we mix the normalized teacher output with the corresponding class prototype to stabilize the semantic target:

u~=normalize⁡(λgˉξ(uraw)+(1−λ)py),gˉξ(z)=normalize⁡(gξ(z)),(7)\tilde{u} = \operatorname{normalize}\left(\lambda\bar{g}_\xi(u^{\mathrm{raw}}) + (1-\lambda)p_y\right), \qquad\bar{g}_\xi(z) = \operatorname{normalize}(g_\xi(z)), \tag*{(7)}
Lpred=1−cos⁡(u^,sg⁡(u~)),(8)\mathcal{L}_{\mathrm{pred}} = 1 - \cos\left(\hat{u}, \operatorname{sg}(\tilde{u})\right), \tag*{(8)}

where yy is the input label, pyp_y is the prototype of class yy, λ∈[0,1]\lambda\in[0,1] is the mixing ratio between the normalized teacher output gˉξ(uraw)\bar{g}_\xi(u^{\mathrm{raw}}) and the class prototype, normalize⁡(⋅)\operatorname{normalize}(\cdot) denotes ℓ2\ell_2 normalization, and sg⁡(⋅)\operatorname{sg}(\cdot) is the stop-gradient operator. This loss aligns gθg_\theta and qϕq_\phi so that the target indicated by the context is consistently reconstructed, capturing the semantic structure that should persist across domains. This same context-to-target predictive relation is later reused in § 3.3 to measure the drift δs,c(t)\delta_{s,c}^{(t)} by re-inputting stored past context summaries.

Drift-Adaptive Consolidation

This section concretely constructs the anchor ac(t−1)a_c^{(t-1)} and the drift δs,c(t)\delta_{s,c}^{(t)} introduced conceptually in § 3.1, and reuses the predictive relation of § 3.2 to measure the drift of past concepts during training. The measured drift is then used in two ways: the consolidation loss concentrates preservation on high-drift pairs, while prototype construction lowers the contribution of high-drift domains. Notationally, we distinguish the raw token space produced by the backbone, RD\mathbb{R}^D, from the embedding space to which the encoder maps, Rd\mathbb{R}^{d}. Prototypes are stored as class means in the raw space and are mapped to the embedding space by the encoder at the time of use.

Anchor Construction At the end of training on each domain ss, we store for every observed class cc its raw target class mean ms,c∈RDm_{s,c} \in\mathbb{R}^{D}, the mean of the raw target representations urawu^{\mathrm{raw}} over the samples of class cc in domain ss. The prototype of class cc is maintained as a raw class mean μc∈RD\mu_c \in\mathbb{R}^{D} that combines these per-domain means over the domains in which the class was observed. The embedding prototype used for classification is its mapping through the current encoder, pc=gθ(μc)p_c = g_\theta(\mu_c). Immediately before training on domain tt (the domain boundary), we take a fixed snapshot of the teacher encoder gξ(t−1)g_{\xi^{(t-1)}} and the raw prototype μc(t−1)\mu_c^{(t-1)} at that point, and define the anchor of class cc as the normalized embedding representation that this teacher produces for the raw prototype:

ac(t−1)=normalize⁡(gξ(t−1)(μc(t−1)))∈Rd,μc(t−1)∈RD,(9)a_c^{(t-1)} = \operatorname{normalize}\left(g_{\xi^{(t-1)}}\left(\mu_c^{(t-1)}\right)\right) \in\mathbb{R}^{d}, \qquad\mu_c^{(t-1)} \in\mathbb{R}^{D}, \tag*{(9)}

where ξ(t−1)\xi^{(t-1)} denotes the teacher parameters fixed immediately before training on domain tt. The anchor is fixed once at the domain boundary and does not change during training on domain tt; this is the concrete realization of the “reference that does not move” described in § 3.1. Because the anchor is defined at the class level, the per-domain difference in drift arises not from the anchor but from the past representation given as input.

Drift Measurement via Replay Measuring drift requires re-evaluating a past concept under the current model without its original images. To this end, at the end of training on each domain we store, for every observed class, a raw context summary νs,c∈RD\nu_{s,c} \in\mathbb{R}^{D}, the mean of the raw context representations vrawv^{\mathrm{raw}} over the samples of class cc in domain ss. This is stored alongside the raw target class mean ms,cm_{s,c}; the two summaries are the only state we retain for each domain–class pair. At the current step tt, we reconstruct the target of a past concept by passing its stored context summary through the current encoder and predictor,

rs,c(t)=qϕ(gθ(νs,c))∈Rd,(10)r_{s,c}^{(t)} = q_\phi\left(g_\theta(\nu_{s,c})\right) \in\mathbb{R}^{d}, \tag*{(10)}

which applies the same context-to-target relation (6) used during current-domain training, now to the past summary rather than to a fresh image. The drift of class cc from domain ss is then the cosine distance between this reconstruction and the fixed anchor,

δs,c(t)=1−cos⁡(rs,c(t),ac(t−1)) for s<t,δt,c(t):=0,(11)\delta_{s,c}^{(t)} = 1 - \cos\left(r_{s,c}^{(t)}, a_c^{(t-1)}\right) \text{ for } s < t, \qquad\delta_{t,c}^{(t)} := 0, \tag*{(11)}

which is nonnegative and serves as a surrogate for whether the current model still maps the past context to a representation aligned with the class anchor; the current domain carries no accumulated drift yet, so its drift is defined as zero. Because the same predictive relation both renders representations on the current domain in (8) and serves as the meter of drift for past summaries, measurement and learning act through one function.

Adaptive Consolidation. The consolidation loss constrains gθg_\theta and qϕq_\phi so that the past context-to-target relation is preserved under the current model, that is, so that the measured drift is reduced. Let S(t)={(s,c):s<t, νs,c stored}\mathcal{S}^{(t)} = \{(s,c):s<t,\ \nu_{s,c}\ \text{stored}\} be the set of past domain–class pairs whose summaries are currently stored. Rather than treating all pairs equally, we place greater preservation pressure on pairs that have drifted more,

ωs,c=(sg⁡(δs,c(t))+ϵ)γ1∣S(t)∣∑(s′,c′)∈S(t)(sg⁡(δs′,c′(t))+ϵ)γ.(12)\omega_{s,c} = \frac{\left(\operatorname{sg}\left(\delta_{s,c}^{(t)}\right)+\epsilon\right)^\gamma} {\frac{1}{|\mathcal{S}^{(t)}|}\sum_{(s',c')\in\mathcal{S}^{(t)}}\left(\operatorname{sg}\left(\delta_{s',c'}^{(t)}\right)+\epsilon\right)^\gamma}. \tag*{(12)}
Lcons=1∣S(t)∣∑(s,c)∈S(t)ωs,cδs,c(t).(13)\mathcal{L}_{\mathrm{cons}} = \frac{1}{|\mathcal{S}^{(t)}|} \sum_{(s,c)\in\mathcal{S}^{(t)}}\omega_{s,c}\delta_{s,c}^{(t)}. \tag*{(13)}

where γ≥0\gamma\ge0 controls how sharply the weighting responds to drift, sg⁡(⋅)\operatorname{sg}(\cdot) is the stop-gradient operator, and ϵ>0\epsilon> 0 is a small constant for numerical stability, especially when some past pairs have near-zero drift.

The weight ωs,c\omega_{s,c} has two properties. First, the drift is detached through sg⁡\operatorname{sg}, so gradients flow only through the δs,c(t)\delta_{s,c}^{(t)} factor in (13): the weight rescales each pair by its drift magnitude but is not itself a learning signal. Second, ωs,c\omega_{s,c} is normalized to unit mean over S(t)\mathcal{S}^{(t)}, so the overall loss scale is preserved as γ\gamma varies, and dividing the sum by ∣S(t)∣|\mathcal{S}^{(t)}| keeps it stable as past pairs accumulate. At γ=0\gamma= 0 the weights reduce to ωs,c=1\omega_{s,c}=1 and all pairs are preserved uniformly; as γ\gamma grows, preservation concentrates on the highest-drift pairs.

Robust Prototype Construction. The same drift signal also governs how the prototype is built. Recall from the Anchor Construction above that the raw prototype μc(t)\mu_c^{(t)} combines the per-domain target means ms,c∈RDm_{s,c}\in\mathbb{R}^{D}; we now specify that combination. When a domain has drifted far, its target mean, once projected through the current encoder, is likely misaligned with the stable semantics of that class, so combining it with equal weight would let it contaminate the prototype. We therefore down-weight each domain by its measured drift,

ψs,c=clip⁡((sg⁡(δs,c(t))+ϵδˉc(t)+ϵ)−ρ,1κ,κ),δˉc(t)=1∣Hc(t)∣∑s∈Hc(t)sg⁡(δs,c(t)),(14)\psi_{s,c}=\operatorname{clip}\left(\left(\frac{\operatorname{sg}\left(\delta_{s,c}^{(t)}\right)+\epsilon}{\bar{\delta}_c^{(t)}+\epsilon}\right)^{-\rho},\frac{1}{\kappa},\kappa\right), \qquad \bar{\delta}_c^{(t)}=\frac{1}{|\mathcal{H}_c^{(t)}|}\sum_{s\in\mathcal{H}_c^{(t)}}\operatorname{sg}\left(\delta_{s,c}^{(t)}\right), \tag*{(14)}
μc(t)=∑s∈Hc(t)πs,cms,c,πs,c=ψs,c∑s′∈Hc(t)ψs′,c,(15)\mu_c^{(t)}=\sum_{s\in\mathcal{H}_c^{(t)}}\pi_{s,c}m_{s,c}, \qquad \pi_{s,c}=\frac{\psi_{s,c}}{\sum_{s'\in\mathcal{H}_c^{(t)}}\psi_{s',c}}, \tag*{(15)}

where Hc(t)={s≤t:ms,c stored}\mathcal{H}_c^{(t)}=\{s\le t:m_{s,c}\ \text{stored}\} is the set of domains in which class cc has been observed, δˉc(t)\bar{\delta}_c^{(t)} is the mean drift over those domains, ρ≥0\rho\ge0 controls how strongly reliability responds to drift, and κ≥1\kappa\ge1 bounds each unnormalized weight ψs,c\psi_{s,c} to [1/κ,κ][1/\kappa,\kappa]. As in (12), the drift is detached through sg⁡\operatorname{sg}, so ψs,c\psi_{s,c} reweights without acting as a learning signal, and the weights are normalized to sum to one per class. Unlike the consolidation weight, Hc(t)\mathcal{H}_c^{(t)} includes the current domain, whose drift is zero by construction (δt,c(t)=0\delta_{t,c}^{(t)} = 0). This turns δˉc(t)\bar{\delta}_c^{(t)} into a per-class reference against the latest observation: ignoring the small stabilizer ϵ\epsilon, a domain that has drifted more than the class average has a ratio δs,c(t)/δˉc(t)>1\delta_{s,c}^{(t)}/\bar{\delta}_c^{(t)} > 1 and, under the negative exponent −ρ-\rho, contributes less, while stable domains receive larger prototype weights. At ρ=0\rho= 0 all weights equal one and μc(t)\mu_c^{(t)} reduces to the uniform mean of its per-domain means, mirroring the γ=0\gamma= 0 case of consolidation.

A single drift signal δs,c(t)\delta_{s,c}^{(t)} thus plays two complementary roles. Consolidation (13) places stronger preservation pressure on high-drift pairs, so that concepts at higher risk of forgetting receive stronger protection; prototype construction (15) trusts high-drift domains less, so that the class reference stays stable. The former decides where to concentrate preservation, the latter which representation to trust: consolidation reduces drift over training, while robust construction keeps the prototype free of the contamination that residual drift would otherwise introduce.

Training Objective

At each training step, the model minimizes a weighted sum of three losses,

L=wclsLcls+wpredLpred+wconsLcons,(16)\mathcal{L} = w_{\mathrm{cls}}\mathcal{L}_{\mathrm{cls}} + w_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}} + w_{\mathrm{cons}}\mathcal{L}_{\mathrm{cons}}, \tag*{(16)}

where wcls,wpred,wcons≥0w_{\mathrm{cls}}, w_{\mathrm{pred}}, w_{\mathrm{cons}} \ge0 weight the respective terms. The classification loss is the cross-entropy over prototype logits, obtained by scaling the cosine similarity between the target embedding uu and each class prototype pcp_c by a learnable logit scale τ\tau,

Lcls=−log⁡exp⁡(τcos⁡(u,py))∑c∈Yexp⁡(τcos⁡(u,pc)),(17)\mathcal{L}_{\mathrm{cls}} = -\log\frac{\exp(\tau\cos(u,p_y))}{\sum_{c\in\mathcal{Y}}\exp(\tau\cos(u,p_c))}, \tag*{(17)}

where yy is the ground-truth label, pcp_c is the class prototype defined in § 3.3, and τ>0\tau> 0 is a learnable logit scale. The prediction loss Lpred\mathcal{L}_{\mathrm{pred}} and the consolidation loss Lcons\mathcal{L}_{\mathrm{cons}} are as defined in (8) and (13), respectively.

The three terms play distinct roles. The classification loss preserves discriminability on the current domain, the prediction loss learns the context-to-target relation, and the consolidation loss keeps that relation from breaking down on past domain–class pairs. On the first domain, no past summary has been stored, so S(1)=∅S^{(1)} = \varnothing and Lcons=0\mathcal{L}_{\mathrm{cons}} = 0; consolidation becomes active from the second domain onward.

Training follows the domain sequence, with the pretrained backbone kept frozen throughout. At each domain boundary (t≥2t \ge2), we fix the anchor ac(t−1)a_c^{(t-1)} from the snapshot teacher as described in § 3.3. During training on domain tt, we update the online encoder gθg_\theta, the predictor qϕq_\phi, and the logit scale τ\tau by minimizing (16), while the teacher parameters ξ\xi are updated by EMA after each step. When training on domain tt ends, we store the raw target mean mt,cm_{t,c} and raw context summary νt,c\nu_{t,c} for each observed class, and recompute the raw prototypes μc(t)\mu_c^{(t)} via (15), with the drift δs,c(t)\delta_{s,c}^{(t)} evaluated at the end of training on domain tt, for use in subsequent domains.

Experiments

Experiment Results

We evaluate our method on four standard DIL benchmarks whose domains are presented sequentially: DomainNet [17], CORe50 [13], Office-Home [20], and CDDB-Hard [9]. We report the two standard DIL metrics: the final average accuracy AvgAcc, averaged over all seen domains after the last stage, and the forgetting AvgF, the average drop between the peak accuracy each domain reached and its accuracy at the final stage. Their formal definitions, together with dataset details, data splits, and the hyperparameter settings, are provided in Appendices § 8–§ 10. All results are averaged over three runs with different random seeds. Table 1 reports AvgAcc and AvgF across the four benchmarks. Our method achieves the highest accuracy on CORe50 and Office-Home and remains competitive on DomainNet and CDDB-Hard, while keeping forgetting low on every benchmark. In contrast, the accuracy-leading baselines show pronounced weaknesses on other benchmarks: DUCT achieves the best accuracy on DomainNet but reaches 39.0% forgetting on CDDB-Hard, and MoE, the strongest method on CDDB-Hard, reaches 17.9% forgetting on DomainNet. These figures indicate substantial degradation on previously learned domains as new ones are learned. Our method instead maintains a strong accuracy–forgetting balance across all four benchmarks. Its lowest per-benchmark accuracy, 61.8% on DomainNet, trails the accuracy leaders DUCT and MoE but remains higher than every other compared method, and it comes with only 4.0% forgetting—an order of magnitude below DUCT’s worst-case 39.0%. High accuracy and low forgetting are thus achieved together, and consistently, rather than by improving one benchmark at the expense of severe forgetting on another.

ModelDomainNetCORe50OfficeHomeCDDB-Hard
AvgAcc ↑\uparrowAvgF ↓\downarrowAvgAcc ↑\uparrowAvgF ↓\downarrowAvgAcc ↑\uparrowAvgF ↓\downarrowAvgAcc ↑\uparrowAvgF ↓\downarrow
DIDD [15]9.42±3.239.42 \pm3.234.75±2.874.75 \pm2.8755.9±2.0755.9 \pm2.0722.6±1.8522.6 \pm1.8529.5±2.6029.5 \pm2.603.9±1.603.9 \pm1.6054.5±1.5454.5 \pm1.541.7±0.871.7 \pm0.87
CODA-Prompt [19]53.6±2.3353.6 \pm2.3321.5±3.2721.5 \pm3.2799.1±0.1599.1 \pm0.150.8±0.160.8 \pm0.1687.2±0.1387.2 \pm0.134.7±0.264.7 \pm0.2667.8±1.0167.8 \pm1.0112.9±0.9212.9 \pm0.92
OS-Prompt [7]52.1±1.5952.1 \pm1.5923.3±1.1323.3 \pm1.1399.2±0.0899.2 \pm0.080.8±0.080.8 \pm0.0886.6±0.1786.6 \pm0.175.5±0.145.5 \pm0.1466.9±0.7366.9 \pm0.7320.2±0.5120.2 \pm0.51
OS-Prompt++ [7]51.4±1.1851.4 \pm1.1824.3±1.8724.3 \pm1.8799.2±0.0399.2 \pm0.030.8±0.020.8 \pm0.0286.3±0.4786.3 \pm0.475.2±0.465.2 \pm0.4667.5±1.7667.5 \pm1.7617.9±2.1417.9 \pm2.14
MoE [25]63.6±0.8263.6 \pm0.8217.9±2.2317.9 \pm2.2399.3±0.0299.3 \pm0.020.5±0.080.5 \pm0.0887.5±0.0587.5 \pm0.051.8±0.271.8 \pm0.2781.4±0.4881.4 \pm0.488.3±1.538.3 \pm1.53
GraphKeeper [6]51.8±0.5151.8 \pm0.518.6±0.478.6 \pm0.4798.2±0.0798.2 \pm0.070.3±0.040.3 \pm0.0422.6±0.2222.6 \pm0.220.6±0.130.6 \pm0.1361.6±0.8661.6 \pm0.861.9±0.921.9 \pm0.92
SOYO [23]51.2±0.9351.2 \pm0.932.0±0.552.0 \pm0.5599.6±0.0499.6 \pm0.040.1±0.030.1 \pm0.0372.3±0.0872.3 \pm0.080.3±0.200.3 \pm0.2078.3±1.5478.3 \pm1.542.4±0.732.4 \pm0.73
DUCT [26]66.4±1.0266.4 \pm1.024.1±1.134.1 \pm1.1394.4±0.0594.4 \pm0.050.4±0.080.4 \pm0.0887.2±0.1087.2 \pm0.104.3±0.624.3 \pm0.6271.2±3.5871.2 \pm3.5839.0±4.8839.0 \pm4.88
Ours61.8±0.9761.8 \pm0.974.0±1.074.0 \pm1.0799.7±0.0499.7 \pm0.040.3±0.030.3 \pm0.0387.8±0.0487.8 \pm0.040.2±0.500.2 \pm0.5079.4±0.3379.4 \pm0.336.7±0.876.7 \pm0.87

Table 1. Experiment results. For each benchmark we report the AvgAcc (↑\uparrow) and AvgF (↓\downarrow), in percent. Bold marks the best and underline the second best in each column.

Efficiency Analysis

We analyze efficiency along two axes: the GPU memory used during training and the storage that accumulates as domains grow. Table 2 reports training-time GPU memory on DomainNet, measured under the same protocol, hardware, batch size, and resolution. Our method uses only 3.0 GB, 4.7×4.7 \times and 10.7×10.7 \times less than its strongest accuracy competitors DUCT (14.07 GB) and MoE (32.00 GB). This follows from our design: the backbone stays frozen with no gradients through it, and only a lightweight encoder and predictor are trained. GraphKeeper uses less memory and DIDD remains moderate, but both are markedly weaker in accuracy on several benchmarks.

ModelDIDDCODA-PromptOS-PromptOS-Prompt++MoEGraphKeeperSOYODUCTOurs
VRAM (GB)6.3819.9919.7920.2132.002.0015.2314.073.00

Table 2. Peak GPU memory usage on DomainNet (GB).

Figure 3 shows, on a log scale, the storage each method accumulates as domains grow. Because we store neither raw images nor a generative model, but only a compact context summary and target mean per domain–class pair, our storage grows only mildly, reaching 13.78 MB after the sixth domain—the smallest among all compared methods, and roughly 355×355 \times smaller than the heaviest, DUCT. Prompt- and feature-based methods accumulate far more (up to 4,217 MB), and the gap widens with the number of domains, making our advantage more pronounced in long-term continual learning. Overall, our method remains competitive with the strongest accuracy baselines while using 4.7×4.7 \times–10.7×10.7 \times less training memory and the smallest storage among all compared methods, occupying a favorable position in the accuracy–efficiency trade-off.

Cumulative storage (MB, log scale) on DomainNet as the number of learned domains grows

Figure 3. Cumulative storage (MB, log scale) on DomainNet as the number of learned domains grows. Numbers at the right denote the final storage after all six domains.

Ablation Study

We analyze the contribution of each component on DomainNet and CDDB-Hard. Table 3 reports the final AvgAcc and AvgF when the prediction loss, the consolidation loss, the drift-adaptive weighting, and the robust prototype weighting are each removed from the full model. The three drift-driven components affect forgetting far more than accuracy. On DomainNet, removing the consolidation loss is the largest effect, surging forgetting from 4.12% to 14.50% (roughly 3.5×3.5 \times), confirming it as the core mechanism for preserving prior knowledge. Treating all past domain–class pairs uniformly (γ=0\gamma= 0) and disabling the robust prototype weighting (ρ=0\rho= 0) increase forgetting by about 2.1×2.1 \times and 1.7×1.7 \times, respectively, indicating that both drift-aware weighting mechanisms contribute to reducing forgetting. Removing the prediction loss instead costs accuracy at least as much as stability, consistent with Lpred\mathcal{L}_{\mathrm{pred}} learning the context-to-target relation on which both the representation and the drift measurement rest. The same trends hold on CDDB-Hard, with smaller absolute changes given its fewer domains.

VariantDomainNet AvgAcc ↑DomainNet AvgF ↓CDDB-Hard AvgAcc ↑CDDB-Hard AvgF ↓
Full (Ours)61.804.1279.406.70
w/o Lcons\mathcal{L}_{\mathrm{cons}}58.1814.5076.827.74
w/o Lpred\mathcal{L}_{\mathrm{pred}}59.835.5376.288.22
w/o adaptive cons. (γ=0\gamma=0)58.258.5678.466.88
w/o robust proto. (ρ=0\rho=0)58.807.0777.717.28

Table 3. Ablation study. Each variant removes one component of the full model. AvgAcc and AvgF are the final average accuracy and forgetting (%).

While Table 3 quantifies these effects, Figure 4 and Figure 5 show them directly in the representation space. We measure the drift of a class as the cosine distance between the target reconstructed from its stored context and its anchor; for visualization, we plot every class by its distance to its own anchor and place all anchors at a single center, so that the radius denotes the magnitude of drift. Figure 4 plots the average drift over past domains at each training stage. Without the consolidation loss, past representations move steadily away from their anchors as domains accumulate, reaching a final average drift of 0.73, whereas the consolidation loss suppresses it to around 0.42. The suppression is largest at

Mean drift over all past domains at each training stage on DomainNet, with and without the consolidation loss

Figure 4. Mean drift over all past domains at each training stage on DomainNet, with and without the consolidation loss Lcons\mathcal{L}_{\mathrm{cons}}.

Polar visualization of drift with and without $\mathcal{L}_{\mathrm{cons}}$

Figure 5. Polar visualization of drift with and without Lcons\mathcal{L}_{\mathrm{cons}}, where all class anchors are placed at the center (⋆\star) and the radius denotes the drift magnitude.

the Quickdraw stage, where the measured drift also peaks, showing that the consolidation loss has its strongest effect when representation change is substantial. Figure 5 shows the same phenomenon through a polar visualization centered at each class anchor. In (a), each point is the drift of one class; without the consolidation loss the points of later domains spread widely into the outer region, whereas the consolidation loss shifts most points toward the center. In (b), each spoke is one past domain, and the trajectory along it depicts the change in drift from immediately after learning that domain to the final state. Without the consolidation loss, all trajectories extend outward as subsequent domains are learned; with it, the trajectories are shorter overall, and those of high-drift domains point back inward. Thus, the consolidation loss suppresses further drift while pulling already deviated representations back toward their anchors.

Conclusion

This work addresses nonuniform forgetting across domains and classes in domain-incremental learning by using representation drift as a measurable surrogate for forgetting risk. With a frozen backbone and a JEPA-based predictive relation, we measure how far each past domain–class concept deviates from its anchor during training without storing raw data. The measured drift guides two complementary mechanisms: adaptive consolidation emphasizes high-drift pairs, while robust prototype construction reduces their influence on class references. Across four benchmarks, our method maintains a strong accuracy–forgetting balance while remaining resource efficient. It achieves the best accuracy on two benchmarks and competitive performance on the others, with consistently low forgetting. By training only a lightweight encoder and predictor and storing compact domain–class summaries, the method requires low training memory and small storage overhead.

Acknowledgments

This work was supported partly by the BK21 FOUR Project and partly by the Technology development Program RS-2025-25454734 funded by the Ministry of SMEs and Startups (MSS, Korea).

Supplementary Material

Evaluation Metrics

We give the formal definitions of the two metrics reported in the main paper. Let as,ja_{s,j} denote the accuracy of the model trained up to stage ss, evaluated on domain jj. At the final stage TT,

AT=1T∑j=1TaT,j,FT=1T−1∑j=1T−1(max⁡s∈{j,…,T−1}as,j−aT,j).(18)A_T = \frac{1}{T}\sum_{j=1}^{T} a_{T,j}, \qquad F_T = \frac{1}{T-1}\sum_{j=1}^{T-1}\left(\max_{s\in\{j,\ldots,T-1\}} a_{s,j} - a_{T,j}\right). \tag*{(18)}

ATA_T (AvgAcc) is the empirical estimate, at the final stage, of the seen-domain average objective defined in Eq. (1), and FTF_T (AvgF) measures forgetting as the average drop between the peak accuracy each domain reached and its final accuracy.

Datasets and Splits

Benchmark#Domains#ClassesDomain order
DomainNet6345clipart, infograph, painting, quickdraw, real, sketch
CORe501150s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11
Office-Home465Art, Clipart, Product, Real World
CDDB-Hard52gaugan, biggan, wild, whichfaceisreal, san

Table 4. Benchmark statistics. Domains are presented sequentially in the listed order.

DomainNet [17] contains six visually distinct domains over 345 shared classes. All images are re-split per domain and class into train/validation/test with an approximate 8:1:1 ratio, stratified by class.

CORe50 [13] consists of 50 object classes recorded in eleven sessions with varying backgrounds and conditions; each session is treated as one domain. Images are re-split per domain and class into train/validation/test at approximately 8:1:1.

Office-Home [20] covers 65 classes across four domains. Since the dataset ships no official split, images are re-split per domain and class into train/validation/test at approximately 8:1:1.

CDDB-Hard [9] is the hard track of the continual deepfake detection benchmark: binary real/fake classification over five generator domains, following the standard sequence used by prior DIL work. We use the official per-domain train/val folders, with val serving as the test set, as is standard practice on this benchmark. All images are resized to 224×224224 \times224 and normalized with ImageNet statistics. The validation split, where present, is used only for early stopping (Sec. C) and never for training.

Implementation Details

Architecture. All experiments use a frozen DINOv2 ViT-B/14 [16] backbone (D=768D = 768); no gradient flows through it. The online encoder gθg_{\theta} is a two-layer residual MLP and the predictor qϕq_{\phi} is a two-layer MLP, both operating at the backbone feature dimension (d=D=768d = D = 768). The teacher gξg_{\xi} is an EMA copy of gθg_{\theta} with momentum m=0.996m = 0.996. Target/context pooling uses the soft attention weighting of Eq. (2).

Optimization. We train with AdamW (learning rate 10−310^{-3}, weight decay 10−410^{-4}), batch size 128, a cosine learning-rate schedule without warmup, gradient clipping at max-norm 1.0, and mixed precision. Early stopping monitors validation accuracy with patience 5. The classification and prediction loss weights are wcls=wpred=1w_{\mathrm{cls}} = w_{\mathrm{pred}} = 1 throughout; the per-benchmark consolidation weight wconsw_{\mathrm{cons}}, drift-weighting sharpness γ\gamma, and number of epochs are listed in Table 5.

DomainNetCORe50Office-HomeCDDB-Hard
Epochs per domain10103020
wconsw_{\mathrm{cons}}300330030
γ\gamma2112

Table 5. Per-benchmark training settings. All other hyperparameters are shared across benchmarks.

Method hyperparameters. The teacher-mixing ratio λ\lambda is 0.5; the logit scale τ\tau is learnable with initial value 10.0; the reliability exponent is ρ=2\rho= 2 with clip bound κ=10\kappa= 10; the numerical stabilizer is ϵ=10−6\epsilon= 10^{-6}.

wconsw_{\mathrm{cons}} differs across benchmarks because the scale of the consolidation term itself does: benchmarks with strong distributional heterogeneity (e.g., DomainNet) exhibit large drift, so a larger weight is needed for the preservation pressure to remain comparable to the current-domain losses, whereas benchmarks whose domains are visually close (e.g., CORe50), even with many stages, accumulate little drift and require only a small weight. Values were selected by a grid over wcons∈{10,30,100,300}w_{\mathrm{cons}} \in\{10, 30, 100, 300\} and γ∈{0,1,2}\gamma\in\{0, 1, 2\} on seed 0 (Sec. D). All experiments run on a single NVIDIA A100 GPU, and all results are averaged over three runs with different random seeds.

γ\wcons\gamma\backslash w_{\mathrm{cons}}1030100300
γ=0\gamma=060.59 / 11.5260.63 / 11.2361.47 / 9.8758.25 / 8.56
γ=1\gamma=159.80 / 12.7860.51 / 11.6361.06 / 10.5260.51 / 6.40
γ=2\gamma=259.90 / 12.6660.53 / 11.6161.19 / 9.8561.80 / 4.12

Table 6. Sensitivity to the consolidation weight wconsw_{\mathrm{cons}} on DomainNet. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration (wcons=300w_{\mathrm{cons}}=300, γ=2\gamma=2).

γ\wcons\gamma\backslash w_{\mathrm{cons}}1030100300
γ=0\gamma=087.12 / 1.8286.83 / 2.1487.75 / 0.2187.68 / 0.50
γ=1\gamma=184.93 / 4.0987.16 / 0.9987.51 / 0.8087.80 / 0.22
γ=2\gamma=286.60 / 2.3286.80 / 2.1887.43 / 0.6586.15 / 0.53

Table 7. Sensitivity to the consolidation weight wconsw_{\mathrm{cons}} on Office-Home. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration (wcons=300w_{\mathrm{cons}}=300, γ=1\gamma=1).

Sensitivity to the Consolidation Weight

Tables 6 and 7 report AvgAcc and AvgF on DomainNet and Office-Home as the consolidation weight wconsw_{\mathrm{cons}} and the drift-weighting sharpness γ\gamma are varied. Two trends are consistent. First, under drift-adaptive weighting (γ≥1\gamma\ge1), increasing wconsw_{\mathrm{cons}} monotonically reduces forgetting (e.g., 12.66% →\to 4.12% on DomainNet at γ=2\gamma= 2) while accuracy remains stable, so the benefit does not hinge on a narrowly tuned weight. Second, the adaptive weighting is what makes a large preservation budget safe: with uniform weighting (γ=0\gamma= 0), the same wcons=300w_{\mathrm{cons}} = 300 costs 3.6 accuracy points on DomainNet (58.25 vs. 61.80), whereas concentrating the same budget on high-drift pairs (γ=2\gamma= 2) attains the lowest forgetting without this accuracy loss. Office-Home shows the same pattern at a milder scale, with forgetting below 4.1% over the entire grid.

Training Algorithm

Algorithm 1 summarizes the full training procedure.

Visualization of CLS-to-Patch Attention

Figure 6 visualizes the CLS-to-patch attention αn\alpha_n of the frozen backbone, which the evidence decomposition of § 3.2 uses to split each image into a target and a context. For the same class (airplane), we show inputs and attention maps.

CLS-to-patch attention maps for the airplane class across six DomainNet domains

Figure 6. CLS-to-patch attention of the frozen backbone for the airplane class across the six DomainNet domains. Top rows: input images; bottom rows: attention maps.

Algorithm 1 Drift-adaptive training over a domain sequence

Require: frozen backbone ff, encoder gθg_{\theta}, predictor qϕq_{\phi}, logit scale τ\tau, EMA teacher gξg_{\xi}, per-domain class means {ms,c}\{m_{s,c}\} and context summaries {νs,c}\{\nu_{s,c}\} (initially empty)

  1. for domain t=1,…,Tt = 1, \ldots, T do

  2. &nbsp;&nbsp;if t=1t = 1 then

  3. &nbsp;&nbsp;&nbsp;&nbsp;initialize μc\mu_c with the raw target class-means of D1\mathcal{D}_1 from the frozen backbone

  4. &nbsp;&nbsp;else

  5. &nbsp;&nbsp;&nbsp;&nbsp;freeze snapshot gξ(t−1)g_{\xi^{(t-1)}}, μc(t−1)\mu_c^{(t-1)}

  6. &nbsp;&nbsp;&nbsp;&nbsp;ac(t−1)←normalize⁡(gξ(t−1)(μc(t−1)))a_c^{(t-1)} \leftarrow\operatorname{normalize}\left(g_{\xi^{(t-1)}}\left(\mu_c^{(t-1)}\right)\right) ▷ anchor construction

  7. &nbsp;&nbsp;end if

  8. &nbsp;&nbsp;for each mini-batch (x,y)∼Dt(x,y) \sim\mathcal{D}_t do

  9. &nbsp;&nbsp;&nbsp;&nbsp;pool uraw,vrawu^{\mathrm{raw}}, v^{\mathrm{raw}} from CLS-to-patch attention ▷ evidence decomposition

  10. &nbsp;&nbsp;&nbsp;&nbsp;u←gθ(uraw)u \leftarrow g_{\theta}(u^{\mathrm{raw}}); u^←qϕ(gθ(vraw))\hat{u} \leftarrow q_{\phi}\left(g_{\theta}(v^{\mathrm{raw}})\right)

  11. &nbsp;&nbsp;&nbsp;&nbsp;compute Lcls\mathcal{L}_{\mathrm{cls}} and Lpred\mathcal{L}_{\mathrm{pred}}

  12. &nbsp;&nbsp;&nbsp;&nbsp;if t≥2t \ge2 then

  13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rs,c←qϕ(gθ(νs,c))r_{s,c} \leftarrow q_{\phi}\left(g_{\theta}(\nu_{s,c})\right) for (s,c)∈S(t)(s,c) \in\mathcal{S}^{(t)} ▷ replay stored contexts

  14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;δs,c←1−cos⁡(rs,c,ac(t−1))\delta_{s,c} \leftarrow1 - \cos\left(r_{s,c}, a_c^{(t-1)}\right) ▷ drift measurement

  15. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;compute Lcons\mathcal{L}_{\mathrm{cons}} with drift-adaptive weights ωs,c\omega_{s,c}

  16. &nbsp;&nbsp;&nbsp;&nbsp;else

  17. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Lcons←0\mathcal{L}_{\mathrm{cons}} \leftarrow0

  18. &nbsp;&nbsp;&nbsp;&nbsp;end if

  19. &nbsp;&nbsp;&nbsp;&nbsp;update θ\theta, ϕ\phi, τ\tau by minimizing wclsLcls+wpredLpred+wconsLconsw_{\mathrm{cls}}\mathcal{L}_{\mathrm{cls}} + w_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}} + w_{\mathrm{cons}}\mathcal{L}_{\mathrm{cons}}; ξ←mξ+(1−m)θ\xi\leftarrow m\xi+ (1-m)\theta

  20. &nbsp;&nbsp;end for

  21. &nbsp;&nbsp;store mt,cm_{t,c} and νt,c\nu_{t,c} for each observed class cc

  22. &nbsp;&nbsp;recompute μc(t)\mu_c^{(t)} with drift-weighted combination, with δs,c\delta_{s,c} evaluated at the end of training on domain tt ▷ robust prototype construction

  23. end for

from all six DomainNet domains. Across domains with sharply different visual statistics—from photographs (real) to abstract line drawings (quickdraw) and cluttered layouts (infograph)—the attention concentrates on the object region rather than on the background or style.

References

  1. [1]Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15619–15629 (2023)arxiv.org/abs/2301.08243
  2. [2]Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems 33, 15920–15930 (2020)
  3. [3]Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: International conference on learning representations. vol. 2024, pp. 2632–2652 (2024)arxiv.org/abs/2309.16588
  4. [4]Gomez-Villa, A., Goswami, D., Wang, K., Bagdanov, A.D., Twardowski, B., Van De Weijer, J.: Exemplar-free continual representation learning via learnable drift compensation. In: European Conference on Computer Vision. pp. 473–490. Springer (2024)
  5. [5]Goswami, D., Soufii-Cormerais, A., Liu, Y., Kamath, S., Twardowski, B., Van De Weijer, J., et al.: Resurrecting old classes with new data for exemplar-free continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 28525–28534 (2024)
  6. [6]Guo, Z., Sun, Q., Zhang, Z., Yuan, H., Zhuang, H., Fu, X., Li, J.: Graphkeeper: Graph domain-incremental learning via knowledge disentanglement and preservation. Advances in Neural Information Processing Systems 38, 1145–1171 (2026)
  7. [7]Kim, Y., Li, Y., Panda, P.: One-stage prompt-based continual learning. In: European conference on computer vision. pp. 163–179. Springer (2024)arxiv.org/abs/2402.16189
  8. [8]Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
  9. [9]Li, C., Huang, Z., Paudel, D.P., Wang, Y., Shahbazi, M., Hong, X., Van Gool, L.: A continual deepfake detection benchmark: Dataset, methods, and essentials. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1339–1349 (2023)
  10. [10]Li, Q., Peng, Y., Zhou, J.: Fcs: Feature calibration and separation for non-exemplar class incremental learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 28495–28504 (2024)
  11. [12]Liang, Y.S., Li, W.J.: Inflora: Interference-free low-rank adaptation for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23638–23647 (2024)
  12. [13]Lomonaco, V., Maltoni, D.: Core50: a new dataset and benchmark for continuous object recognition. In: Conference on robot learning. pp. 17–26. PMLR (2017)
  13. [14]Magistri, S., Trinci, T., van de Weijer, J., Bagdanov, A., et al.: Elastic feature consolidation for cold start exemplar-free incremental learning. In: International Conference on Learning Representations. vol. 2024, pp. 31909–31931 (2024)
  14. [15]Oh, M., Sim, J.Y.: Asymmetric synthetic data update for domain incremental dataset distillation. In: The Fourteenth International Conference on Learning Representations (2026)
  15. [16]Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Domain2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)arxiv.org/abs/2304.07193
  16. [17]Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1406–1415 (2019)arxiv.org/abs/1812.01754
  17. [19]Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., Kira, Z.: Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11909–11919 (2023)arxiv.org/abs/2211.13218
  18. [20]Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5018–5027 (2017)
  19. [21]Wang, Q., He, Y., Dong, S., Gao, X., Wang, S., Gong, Y.: Non-exemplar domain incremental learning via cross-domain concept integration. In: European Conference on Computer Vision. pp. 144–162. Springer (2024)DOI
  20. [22]Wang, Q., He, Y., Dong, S., Song, X., Han, J., Luo, H., Gong, Y.: Dualcp: Rehearsal-free domain-incremental learning via dual-level concept prototype. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 21198–21206 (2025)arxiv.org/abs/2503.18042
  21. [23]Wang, Q., Song, X., He, Y., Han, J., Ding, C., Gao, X., Gong, Y.: Boosting domain incremental learning: Selecting the optimal parameters is all you need. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4839–4849 (2025)arxiv.org/abs/2505.23744
  22. [24]Xie, J., Yan, S., He, X.: General incremental learning with domain-aware categorical representations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14351–14360 (2022)arxiv.org/abs/2204.04078
  23. [25]Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., He, Y.: Boosting continual learning of vision-language models via mixture-of-experts adapters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23219–23230 (2024)arxiv.org/abs/2403.11549
  24. [26]Zhou, D.W., Cai, Z.W., Ye, H.J., Zhang, L., Zhan, D.C.: Dual consolidation for pre-trained model-based domain-incremental learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20547–20557 (2025)arxiv.org/abs/2410.00911

Paper details

vlnc:W_5f3y7j2x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents