WorksComputer Science
Drift-adaptive Representation Preservation for Domain-Incremental Learning
Abstract
Pretrained vision models provide strong general-purpose representations, but in Domain-Incremental Learning (DIL), where the input distribution changes sequentially, adapting to new domains often degrades performance on previously learned ones. We observe that forgetting is not uniform across domains and classes and is not explained by training order alone. This motivates us to track each past domain-class concept individually. We use representation drift, defined as the deviation of a past concept's current representation from a fixed anchor, as a measurable surrogate for forgetting risk. The measured drift then serves two complementary roles. A consolidation loss assigns stronger preservation to high-drift pairs, while prototype construction down-weights highly drifted domains to reduce their influence on the class reference. Preservation is thus allocated by measured drift rather than assumed importance. To obtain this signal during training without storing past raw data, we repurpose a Joint-Embedding Predictive Architecture (JEPA). For each domain-class pair, we store only a compact context summary in the backbone feature space and rerun it through the learned context-to-target relation to measure its drift from the fixed anchor. Across DomainNet, Office-Home, CORe50, and CDDB-Hard, our method sustains a strong accuracy-forgetting balance on every benchmark, using substantially less training memory than accuracy-competitive methods and the smallest storage among compared methods.
Introduction
Pretrained vision models provide strong, broadly transferable representations and have substantially expanded the practical promise of DIL [21] [26]. In DIL, a model learns a sequence of domains that share a label space but differ in input distribution, and it must maintain stable performance on all domains seen so1 far without revisiting past data [22] [23]. The central difficulty is the stability–plasticity dilemma: adapting the representation to a new domain disturbs the structure formed on past domains and induces forgetting, whereas suppressing representational change too strongly limits adaptation [12] [24].
We study forgetting in DIL through the lens of representation drift. As the model learns a domain, the resulting per-class representation serves as a fixed reference point, or anchor, against which later changes can be measured. When the model subsequently adapts to new domains, a previously learned concept may move away from this reference. We use this deviation as a measurable signal of how strongly the concept has changed and as a surrogate for its forgetting risk. Crucially, these changes are not uniform across previously learned domains and classes. As shown in Figure 1, the amount of forgetting varies substantially across domains within the same training sequence and cannot be characterized by training order alone. This heterogeneous behavior motivates us to track representation drift separately for each domain–class pair rather than relying on a single global measure. This nonuniformity creates a specific problem for prototype-based classification: a pair with large drift has likely departed from the stable class semantics, and combining it with equal weight when forming the class prototype lets the distorted representation contaminate the class reference itself—an effect we call prototype contamination.

Figure 1. Per-domain forgetting when DINOv2 is sequentially fine-tuned on DomainNet. Example images of the same class (e.g., crab) illustrate the visual variation across domains.
Motivated by this, we use drift in two ways at once. First, we use drift as a proxy for forgetting risk: a pair with larger drift receives stronger preservation because its current reconstruction is less aligned with the fixed class anchor. Second, drift is a reliability signal for prototype construction: a pair with small drift is trusted, whereas a pair with large drift is down-weighted, shielding the class reference from contamination. A single measured drift signal thus plays two complementary roles, so that preservation is allocated according to measured drift rather than assumed importance.
Using drift this way has a prerequisite: it must be measurable at the domain–class level, during training, and without access to past raw data. We observe that JEPA [1] provides a natural mechanism for this purpose. Because it learns, in latent space, to predict one observed cue from another, a past concept can be reexamined by storing only a compact context summary for each domain–class pair and rerunning it through this learned relation. Concretely, a frozen backbone’s CLS-to-patch attention splits each image into a target carrying class-relevant evidence and a context capturing domain-sensitive cues; the model learns to predict the target from the context, and stored past-domain context summaries are re-input to the current encoder and predictor to measure how far the prediction departs from the fixed anchor—this distance is the drift signal.
This design also brings a concrete efficiency benefit: no gradient flows through the frozen backbone, so training is confined to a lightweight encoder and predictor and the training memory footprint remains small, while past information is stored only as compact per-domain–class summaries, so storage grows only mildly with the number of domains. Our contributions are summarized as follows:
We propose a preservation framework in which a single measured drift signal drives two complementary mechanisms—a drift-adaptive consolidation loss that concentrates preservation on high-drift concepts, and a robust prototype construction that down-weights high-drift domains to prevent prototype contamination—allocating preservation by measured drift rather than assumed importance.
To make this signal available at the domain–class level during training, without storing any past image, we repurpose JEPA’s learned context-to-target relation as a drift meter, rerunning a stored domain–class context summary through the current encoder and predictor.
Across four DIL benchmarks, our method sustains a strong accuracy–forgetting balance on every benchmark while using substantially less training memory than accuracy-competitive methods and the smallest storage among the compared methods.
Related Work
Domain-Incremental Learning
Continual learning studies how a model can absorb new data distributions without forgetting previously learned ones. In domain-incremental learning, all domains share the same label space while the input distribution shifts sequentially, so the central difficulty is to keep past domains from degrading as the representation adapts to new ones. Rehearsal-based methods address this by storing or synthesizing past samples for replay [18] [2]. They are effective, but their storage cost grows with the data, they raise privacy concerns, and they are inapplicable when past images cannot be retained [27] [4]. Regularization-based methods instead constrain parameters or outputs deemed important for earlier tasks from changing substantially, as in EWC and LwF [11] [8]. Such preservation, however, is driven by an importance that is assumed or estimated globally, rather than by measuring which domain–class concepts are actually drifting under the current model. Recent domain-incremental and pretrained-model-based continual learning methods reduce reliance on raw data replay by keeping the pretrained backbone frozen and adapting to new domains through a small number of additional parameters or memory. Prompt-based methods such as CODA-Prompt [19], OS-Prompt [7], and OS-Prompt++ [7] adapt by accumulating or selecting prompt parameters per domain. In contrast, DIDD [15], GraphKeeper [6], SOYO [23], DUCT [26], and MoE [25] preserve or route domain knowledge through different mechanisms—feature- and graph-based memories, selective routing over domain-specific parameters or paths, backbone adaptation, and expert modules. Our method instead keeps the pretrained backbone frozen and stores only a compact summary per domain–class pair, using measured drift to decide what to preserve and which contributions to trust when forming each prototype.
Representation Drift and Predictive Representations
The instability of learned representations under sequential training is a well-recognized difficulty in continual learning. As a model adapts to new data, the features of previously learned concepts may move, causing stored statistics such as class prototypes to fall out of alignment with the current feature space—an effect often described as representation, feature, or semantic drift [10] [5] [4]. Prior work has mainly treated this drift as an error to be compensated for or suppressed, for example by estimating the displacement of stored prototypes or by stabilizing old representations through feature alignment and distillation [14]. In contrast, we use drift before suppressing it: we measure it at the domain–class level and use its magnitude as a nonuniform signal that determines both how strongly each concept should be preserved and how much each domain should contribute to the class prototype. Our drift measurement is enabled by a latent predictive relation inspired by JEPA, which learns representations by predicting the embedding of one part or view of an input from another in latent space, typically using an online encoder and an exponential moving average target encoder. In their original use, this predictive objective is designed for self-supervised representation learning. We repurpose the same principle for continual learning: the context-to-target relation trained on the current domain is reapplied to stored past-domain context summaries, and the predicted targets are compared with fixed class anchors to measure drift. Measurement and learning therefore share one function rather than relying on a separate probe, and the resulting drift signal drives both drift-adaptive consolidation and robust prototype construction. This use of a predictive relation as a per-concept meter of drift, with a single signal serving both roles, sets our approach apart from prior drift compensation and pretrained-model-based continual learning methods.
Proposed Method
Problem Formulation
In DIL, a model is trained on domains in sequence. Each domain is associated with a distribution , from which a training set is sampled, where is an input image, is its label, and is the number of samples in domain . All domains share a common label space with classes, while the input distribution differs across domains.

Figure 2. Overview of our framework. Evidence Decomposition splits each input image into a target and a context. Context-to-Target Prediction predicts the target from the context and performs classification. Drift-Adaptive Consolidation measures the drift from the fixed anchors using stored past-domain summaries while learning a new domain, and uses this signal to modulate the preservation strength and to weight each domain’s contribution when forming the class prototypes.
Training proceeds sequentially, one domain at a time: at step the model can access only the current domain and cannot use the original images of the previous domains . Let denote the model trained up to domain . The objective is to maximize the average performance over all domains seen so far,
where is the indicator function and indexes the domains seen so far. Keeping the objective in Eq. (1) without access to past data is the central challenge of DIL.
In this work, is a prototype-based classifier that maps an input to a representation through a frozen pretrained backbone and a trainable encoder, and classifies it by the cosine similarity to each class prototype. Under this formulation, forgetting is reflected in the degradation of performance on previously learned domains. We track the underlying representational change through the deviation of each past concept from its fixed class anchor and use this representation drift as a measurable surrogate for forgetting risk. At each domain transition, before training on domain , we fix the representation of each class observed so far as a reference against which subsequent change is measured, and call it the anchor (constructed in § 3.3). As the encoder changes while later domains are learned, a past concept may be mapped to a location that departs from its anchor, and this distance serves as a measurable surrogate for the concept’s forgetting risk.
To make this distance concrete, we define the drift of class from domain , measured at step , as a nonnegative scalar . This construction has two important aspects. First, is obtained without any past image: a past concept is reconstructed through the current model from a stored domain–class context summary and compared against its anchor. Second, since the anchor is fixed per class, the per-domain difference in drift arises solely from the reconstructed past representation; drift is therefore not a single global quantity but one indexed by each domain–class pair . We track these domain–class-specific representational changes through the per-pair surrogate during training, and use this signal to allocate preservation and construct robust prototypes.
Evidence Decomposition and Prediction
We now specify the representation used for classification, introduced abstractly in § 3.1, as two complementary cues derived from attention, and learn a predictive relation between them. This predictive relation later serves, in § 3.3, as the function that measures the drift of past concepts.
CLS-to-Patch Attention Passing an input through the frozen backbone yields a CLS token and patch tokens. Let be the -th patch token, where is the backbone token dimensionality, and let be the attention from the CLS token to that patch, normalized over the patches so that and ; intuitively, reflects the regions that the pretrained backbone attends to more strongly and that therefore tend to carry the class-relevant evidence [3]. Using the attention weights and their complementary weights , we aggregate the patch tokens into two representations,
which we then normalize to unit vectors,
where is the target, formed from regions emphasized by the pretrained backbone and used as the classification cue, whereas is the context, formed with the complementary weights and tending to capture cues sensitive to domain change, such as background, style, and texture.
Latent Encoding and Classification The two raw representations are passed through a shared trainable encoder and mapped to the embedding space :
where is the encoder output dimensionality, is the target representation used for classification, and is the context representation used as the input to prediction; the same embedding space is later used for drift measurement. Throughout, all encoder and predictor outputs are -normalized to unit vectors before any cosine comparison. As described in § 3.1, classification is performed via the cosine similarity between the target representation and the class prototype ,
The prototype is the embedding-space projection of a stored raw prototype , whose construction and update we defer to § 3.3.
Context-to-Target Prediction The core of our framework is the relation that predicts the target from the context in the latent space. A predictor , with trainable parameters , estimates the target from the context representation:
The prediction is aligned to a teacher target . The teacher is an exponential moving average (EMA) copy of the encoder, whose parameters are updated at each training step as with EMA coefficient , and no gradient flows along this path; using an EMA copy rather than the online encoder gives a slowly moving, stable target while is still being learned. Using the label, we mix the normalized teacher output with the corresponding class prototype to stabilize the semantic target:
where is the input label, is the prototype of class , is the mixing ratio between the normalized teacher output and the class prototype, denotes normalization, and is the stop-gradient operator. This loss aligns and so that the target indicated by the context is consistently reconstructed, capturing the semantic structure that should persist across domains. This same context-to-target predictive relation is later reused in § 3.3 to measure the drift by re-inputting stored past context summaries.
Drift-Adaptive Consolidation
This section concretely constructs the anchor and the drift introduced conceptually in § 3.1, and reuses the predictive relation of § 3.2 to measure the drift of past concepts during training. The measured drift is then used in two ways: the consolidation loss concentrates preservation on high-drift pairs, while prototype construction lowers the contribution of high-drift domains. Notationally, we distinguish the raw token space produced by the backbone, , from the embedding space to which the encoder maps, . Prototypes are stored as class means in the raw space and are mapped to the embedding space by the encoder at the time of use.
Anchor Construction At the end of training on each domain , we store for every observed class its raw target class mean , the mean of the raw target representations over the samples of class in domain . The prototype of class is maintained as a raw class mean that combines these per-domain means over the domains in which the class was observed. The embedding prototype used for classification is its mapping through the current encoder, . Immediately before training on domain (the domain boundary), we take a fixed snapshot of the teacher encoder and the raw prototype at that point, and define the anchor of class as the normalized embedding representation that this teacher produces for the raw prototype:
where denotes the teacher parameters fixed immediately before training on domain . The anchor is fixed once at the domain boundary and does not change during training on domain ; this is the concrete realization of the “reference that does not move” described in § 3.1. Because the anchor is defined at the class level, the per-domain difference in drift arises not from the anchor but from the past representation given as input.
Drift Measurement via Replay Measuring drift requires re-evaluating a past concept under the current model without its original images. To this end, at the end of training on each domain we store, for every observed class, a raw context summary , the mean of the raw context representations over the samples of class in domain . This is stored alongside the raw target class mean ; the two summaries are the only state we retain for each domain–class pair. At the current step , we reconstruct the target of a past concept by passing its stored context summary through the current encoder and predictor,
which applies the same context-to-target relation (6) used during current-domain training, now to the past summary rather than to a fresh image. The drift of class from domain is then the cosine distance between this reconstruction and the fixed anchor,
which is nonnegative and serves as a surrogate for whether the current model still maps the past context to a representation aligned with the class anchor; the current domain carries no accumulated drift yet, so its drift is defined as zero. Because the same predictive relation both renders representations on the current domain in (8) and serves as the meter of drift for past summaries, measurement and learning act through one function.
Adaptive Consolidation. The consolidation loss constrains and so that the past context-to-target relation is preserved under the current model, that is, so that the measured drift is reduced. Let be the set of past domain–class pairs whose summaries are currently stored. Rather than treating all pairs equally, we place greater preservation pressure on pairs that have drifted more,
where controls how sharply the weighting responds to drift, is the stop-gradient operator, and is a small constant for numerical stability, especially when some past pairs have near-zero drift.
The weight has two properties. First, the drift is detached through , so gradients flow only through the factor in (13): the weight rescales each pair by its drift magnitude but is not itself a learning signal. Second, is normalized to unit mean over , so the overall loss scale is preserved as varies, and dividing the sum by keeps it stable as past pairs accumulate. At the weights reduce to and all pairs are preserved uniformly; as grows, preservation concentrates on the highest-drift pairs.
Robust Prototype Construction. The same drift signal also governs how the prototype is built. Recall from the Anchor Construction above that the raw prototype combines the per-domain target means ; we now specify that combination. When a domain has drifted far, its target mean, once projected through the current encoder, is likely misaligned with the stable semantics of that class, so combining it with equal weight would let it contaminate the prototype. We therefore down-weight each domain by its measured drift,
where is the set of domains in which class has been observed, is the mean drift over those domains, controls how strongly reliability responds to drift, and bounds each unnormalized weight to . As in (12), the drift is detached through , so reweights without acting as a learning signal, and the weights are normalized to sum to one per class. Unlike the consolidation weight, includes the current domain, whose drift is zero by construction (). This turns into a per-class reference against the latest observation: ignoring the small stabilizer , a domain that has drifted more than the class average has a ratio and, under the negative exponent , contributes less, while stable domains receive larger prototype weights. At all weights equal one and reduces to the uniform mean of its per-domain means, mirroring the case of consolidation.
A single drift signal thus plays two complementary roles. Consolidation (13) places stronger preservation pressure on high-drift pairs, so that concepts at higher risk of forgetting receive stronger protection; prototype construction (15) trusts high-drift domains less, so that the class reference stays stable. The former decides where to concentrate preservation, the latter which representation to trust: consolidation reduces drift over training, while robust construction keeps the prototype free of the contamination that residual drift would otherwise introduce.
Training Objective
At each training step, the model minimizes a weighted sum of three losses,
where weight the respective terms. The classification loss is the cross-entropy over prototype logits, obtained by scaling the cosine similarity between the target embedding and each class prototype by a learnable logit scale ,
where is the ground-truth label, is the class prototype defined in § 3.3, and is a learnable logit scale. The prediction loss and the consolidation loss are as defined in (8) and (13), respectively.
The three terms play distinct roles. The classification loss preserves discriminability on the current domain, the prediction loss learns the context-to-target relation, and the consolidation loss keeps that relation from breaking down on past domain–class pairs. On the first domain, no past summary has been stored, so and ; consolidation becomes active from the second domain onward.
Training follows the domain sequence, with the pretrained backbone kept frozen throughout. At each domain boundary (), we fix the anchor from the snapshot teacher as described in § 3.3. During training on domain , we update the online encoder , the predictor , and the logit scale by minimizing (16), while the teacher parameters are updated by EMA after each step. When training on domain ends, we store the raw target mean and raw context summary for each observed class, and recompute the raw prototypes via (15), with the drift evaluated at the end of training on domain , for use in subsequent domains.
Experiments
Experiment Results
We evaluate our method on four standard DIL benchmarks whose domains are presented sequentially: DomainNet [17], CORe50 [13], Office-Home [20], and CDDB-Hard [9]. We report the two standard DIL metrics: the final average accuracy AvgAcc, averaged over all seen domains after the last stage, and the forgetting AvgF, the average drop between the peak accuracy each domain reached and its accuracy at the final stage. Their formal definitions, together with dataset details, data splits, and the hyperparameter settings, are provided in Appendices § 8–§ 10. All results are averaged over three runs with different random seeds. Table 1 reports AvgAcc and AvgF across the four benchmarks. Our method achieves the highest accuracy on CORe50 and Office-Home and remains competitive on DomainNet and CDDB-Hard, while keeping forgetting low on every benchmark. In contrast, the accuracy-leading baselines show pronounced weaknesses on other benchmarks: DUCT achieves the best accuracy on DomainNet but reaches 39.0% forgetting on CDDB-Hard, and MoE, the strongest method on CDDB-Hard, reaches 17.9% forgetting on DomainNet. These figures indicate substantial degradation on previously learned domains as new ones are learned. Our method instead maintains a strong accuracy–forgetting balance across all four benchmarks. Its lowest per-benchmark accuracy, 61.8% on DomainNet, trails the accuracy leaders DUCT and MoE but remains higher than every other compared method, and it comes with only 4.0% forgetting—an order of magnitude below DUCT’s worst-case 39.0%. High accuracy and low forgetting are thus achieved together, and consistently, rather than by improving one benchmark at the expense of severe forgetting on another.
Table 1. Experiment results. For each benchmark we report the AvgAcc () and AvgF (), in percent. Bold marks the best and underline the second best in each column.
Efficiency Analysis
We analyze efficiency along two axes: the GPU memory used during training and the storage that accumulates as domains grow. Table 2 reports training-time GPU memory on DomainNet, measured under the same protocol, hardware, batch size, and resolution. Our method uses only 3.0 GB, and less than its strongest accuracy competitors DUCT (14.07 GB) and MoE (32.00 GB). This follows from our design: the backbone stays frozen with no gradients through it, and only a lightweight encoder and predictor are trained. GraphKeeper uses less memory and DIDD remains moderate, but both are markedly weaker in accuracy on several benchmarks.
| Model | DIDD | CODA-Prompt | OS-Prompt | OS-Prompt++ | MoE | GraphKeeper | SOYO | DUCT | Ours |
| VRAM (GB) | 6.38 | 19.99 | 19.79 | 20.21 | 32.00 | 2.00 | 15.23 | 14.07 | 3.00 |
Table 2. Peak GPU memory usage on DomainNet (GB).
Figure 3 shows, on a log scale, the storage each method accumulates as domains grow. Because we store neither raw images nor a generative model, but only a compact context summary and target mean per domain–class pair, our storage grows only mildly, reaching 13.78 MB after the sixth domain—the smallest among all compared methods, and roughly smaller than the heaviest, DUCT. Prompt- and feature-based methods accumulate far more (up to 4,217 MB), and the gap widens with the number of domains, making our advantage more pronounced in long-term continual learning. Overall, our method remains competitive with the strongest accuracy baselines while using – less training memory and the smallest storage among all compared methods, occupying a favorable position in the accuracy–efficiency trade-off.

Figure 3. Cumulative storage (MB, log scale) on DomainNet as the number of learned domains grows. Numbers at the right denote the final storage after all six domains.
Ablation Study
We analyze the contribution of each component on DomainNet and CDDB-Hard. Table 3 reports the final AvgAcc and AvgF when the prediction loss, the consolidation loss, the drift-adaptive weighting, and the robust prototype weighting are each removed from the full model. The three drift-driven components affect forgetting far more than accuracy. On DomainNet, removing the consolidation loss is the largest effect, surging forgetting from 4.12% to 14.50% (roughly ), confirming it as the core mechanism for preserving prior knowledge. Treating all past domain–class pairs uniformly () and disabling the robust prototype weighting () increase forgetting by about and , respectively, indicating that both drift-aware weighting mechanisms contribute to reducing forgetting. Removing the prediction loss instead costs accuracy at least as much as stability, consistent with learning the context-to-target relation on which both the representation and the drift measurement rest. The same trends hold on CDDB-Hard, with smaller absolute changes given its fewer domains.
| Variant | DomainNet AvgAcc ↑ | DomainNet AvgF ↓ | CDDB-Hard AvgAcc ↑ | CDDB-Hard AvgF ↓ |
| Full (Ours) | 61.80 | 4.12 | 79.40 | 6.70 |
| w/o | 58.18 | 14.50 | 76.82 | 7.74 |
| w/o | 59.83 | 5.53 | 76.28 | 8.22 |
| w/o adaptive cons. () | 58.25 | 8.56 | 78.46 | 6.88 |
| w/o robust proto. () | 58.80 | 7.07 | 77.71 | 7.28 |
Table 3. Ablation study. Each variant removes one component of the full model. AvgAcc and AvgF are the final average accuracy and forgetting (%).
While Table 3 quantifies these effects, Figure 4 and Figure 5 show them directly in the representation space. We measure the drift of a class as the cosine distance between the target reconstructed from its stored context and its anchor; for visualization, we plot every class by its distance to its own anchor and place all anchors at a single center, so that the radius denotes the magnitude of drift. Figure 4 plots the average drift over past domains at each training stage. Without the consolidation loss, past representations move steadily away from their anchors as domains accumulate, reaching a final average drift of 0.73, whereas the consolidation loss suppresses it to around 0.42. The suppression is largest at

Figure 4. Mean drift over all past domains at each training stage on DomainNet, with and without the consolidation loss .

Figure 5. Polar visualization of drift with and without , where all class anchors are placed at the center () and the radius denotes the drift magnitude.
the Quickdraw stage, where the measured drift also peaks, showing that the consolidation loss has its strongest effect when representation change is substantial. Figure 5 shows the same phenomenon through a polar visualization centered at each class anchor. In (a), each point is the drift of one class; without the consolidation loss the points of later domains spread widely into the outer region, whereas the consolidation loss shifts most points toward the center. In (b), each spoke is one past domain, and the trajectory along it depicts the change in drift from immediately after learning that domain to the final state. Without the consolidation loss, all trajectories extend outward as subsequent domains are learned; with it, the trajectories are shorter overall, and those of high-drift domains point back inward. Thus, the consolidation loss suppresses further drift while pulling already deviated representations back toward their anchors.
Conclusion
This work addresses nonuniform forgetting across domains and classes in domain-incremental learning by using representation drift as a measurable surrogate for forgetting risk. With a frozen backbone and a JEPA-based predictive relation, we measure how far each past domain–class concept deviates from its anchor during training without storing raw data. The measured drift guides two complementary mechanisms: adaptive consolidation emphasizes high-drift pairs, while robust prototype construction reduces their influence on class references. Across four benchmarks, our method maintains a strong accuracy–forgetting balance while remaining resource efficient. It achieves the best accuracy on two benchmarks and competitive performance on the others, with consistently low forgetting. By training only a lightweight encoder and predictor and storing compact domain–class summaries, the method requires low training memory and small storage overhead.
Acknowledgments
This work was supported partly by the BK21 FOUR Project and partly by the Technology development Program RS-2025-25454734 funded by the Ministry of SMEs and Startups (MSS, Korea).
Supplementary Material
Evaluation Metrics
We give the formal definitions of the two metrics reported in the main paper. Let denote the accuracy of the model trained up to stage , evaluated on domain . At the final stage ,
(AvgAcc) is the empirical estimate, at the final stage, of the seen-domain average objective defined in Eq. (1), and (AvgF) measures forgetting as the average drop between the peak accuracy each domain reached and its final accuracy.
Datasets and Splits
| Benchmark | #Domains | #Classes | Domain order |
| DomainNet | 6 | 345 | clipart, infograph, painting, quickdraw, real, sketch |
| CORe50 | 11 | 50 | s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11 |
| Office-Home | 4 | 65 | Art, Clipart, Product, Real World |
| CDDB-Hard | 5 | 2 | gaugan, biggan, wild, whichfaceisreal, san |
Table 4. Benchmark statistics. Domains are presented sequentially in the listed order.
DomainNet [17] contains six visually distinct domains over 345 shared classes. All images are re-split per domain and class into train/validation/test with an approximate 8:1:1 ratio, stratified by class.
CORe50 [13] consists of 50 object classes recorded in eleven sessions with varying backgrounds and conditions; each session is treated as one domain. Images are re-split per domain and class into train/validation/test at approximately 8:1:1.
Office-Home [20] covers 65 classes across four domains. Since the dataset ships no official split, images are re-split per domain and class into train/validation/test at approximately 8:1:1.
CDDB-Hard [9] is the hard track of the continual deepfake detection benchmark: binary real/fake classification over five generator domains, following the standard sequence used by prior DIL work. We use the official per-domain train/val folders, with val serving as the test set, as is standard practice on this benchmark. All images are resized to and normalized with ImageNet statistics. The validation split, where present, is used only for early stopping (Sec. C) and never for training.
Implementation Details
Architecture. All experiments use a frozen DINOv2 ViT-B/14 [16] backbone (); no gradient flows through it. The online encoder is a two-layer residual MLP and the predictor is a two-layer MLP, both operating at the backbone feature dimension (). The teacher is an EMA copy of with momentum . Target/context pooling uses the soft attention weighting of Eq. (2).
Optimization. We train with AdamW (learning rate , weight decay ), batch size 128, a cosine learning-rate schedule without warmup, gradient clipping at max-norm 1.0, and mixed precision. Early stopping monitors validation accuracy with patience 5. The classification and prediction loss weights are throughout; the per-benchmark consolidation weight , drift-weighting sharpness , and number of epochs are listed in Table 5.
| DomainNet | CORe50 | Office-Home | CDDB-Hard | |
| Epochs per domain | 10 | 10 | 30 | 20 |
| 300 | 3 | 300 | 30 | |
| 2 | 1 | 1 | 2 |
Table 5. Per-benchmark training settings. All other hyperparameters are shared across benchmarks.
Method hyperparameters. The teacher-mixing ratio is 0.5; the logit scale is learnable with initial value 10.0; the reliability exponent is with clip bound ; the numerical stabilizer is .
differs across benchmarks because the scale of the consolidation term itself does: benchmarks with strong distributional heterogeneity (e.g., DomainNet) exhibit large drift, so a larger weight is needed for the preservation pressure to remain comparable to the current-domain losses, whereas benchmarks whose domains are visually close (e.g., CORe50), even with many stages, accumulate little drift and require only a small weight. Values were selected by a grid over and on seed 0 (Sec. D). All experiments run on a single NVIDIA A100 GPU, and all results are averaged over three runs with different random seeds.
| 10 | 30 | 100 | 300 | |
| 60.59 / 11.52 | 60.63 / 11.23 | 61.47 / 9.87 | 58.25 / 8.56 | |
| 59.80 / 12.78 | 60.51 / 11.63 | 61.06 / 10.52 | 60.51 / 6.40 | |
| 59.90 / 12.66 | 60.53 / 11.61 | 61.19 / 9.85 | 61.80 / 4.12 |
Table 6. Sensitivity to the consolidation weight on DomainNet. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration (, ).
| 10 | 30 | 100 | 300 | |
| 87.12 / 1.82 | 86.83 / 2.14 | 87.75 / 0.21 | 87.68 / 0.50 | |
| 84.93 / 4.09 | 87.16 / 0.99 | 87.51 / 0.80 | 87.80 / 0.22 | |
| 86.60 / 2.32 | 86.80 / 2.18 | 87.43 / 0.65 | 86.15 / 0.53 |
Table 7. Sensitivity to the consolidation weight on Office-Home. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration (, ).
Sensitivity to the Consolidation Weight
Tables 6 and 7 report AvgAcc and AvgF on DomainNet and Office-Home as the consolidation weight and the drift-weighting sharpness are varied. Two trends are consistent. First, under drift-adaptive weighting (), increasing monotonically reduces forgetting (e.g., 12.66% 4.12% on DomainNet at ) while accuracy remains stable, so the benefit does not hinge on a narrowly tuned weight. Second, the adaptive weighting is what makes a large preservation budget safe: with uniform weighting (), the same costs 3.6 accuracy points on DomainNet (58.25 vs. 61.80), whereas concentrating the same budget on high-drift pairs () attains the lowest forgetting without this accuracy loss. Office-Home shows the same pattern at a milder scale, with forgetting below 4.1% over the entire grid.
Training Algorithm
Algorithm 1 summarizes the full training procedure.
Visualization of CLS-to-Patch Attention
Figure 6 visualizes the CLS-to-patch attention of the frozen backbone, which the evidence decomposition of § 3.2 uses to split each image into a target and a context. For the same class (airplane), we show inputs and attention maps.

Figure 6. CLS-to-patch attention of the frozen backbone for the airplane class across the six DomainNet domains. Top rows: input images; bottom rows: attention maps.
Algorithm 1 Drift-adaptive training over a domain sequence
Require: frozen backbone , encoder , predictor , logit scale , EMA teacher , per-domain class means and context summaries (initially empty)
for domain do
if then
initialize with the raw target class-means of from the frozen backbone
else
freeze snapshot ,
▷ anchor construction
end if
for each mini-batch do
pool from CLS-to-patch attention ▷ evidence decomposition
;
compute and
if then
for ▷ replay stored contexts
▷ drift measurement
compute with drift-adaptive weights
else
end if
update , , by minimizing ;
end for
store and for each observed class
recompute with drift-weighted combination, with evaluated at the end of training on domain ▷ robust prototype construction
end for
from all six DomainNet domains. Across domains with sharply different visual statistics—from photographs (real) to abstract line drawings (quickdraw) and cluttered layouts (infograph)—the attention concentrates on the object region rather than on the background or style.
References
- [1]Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15619–15629 (2023)arxiv.org/abs/2301.08243
- [2]Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems 33, 15920–15930 (2020)
- [3]Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: International conference on learning representations. vol. 2024, pp. 2632–2652 (2024)arxiv.org/abs/2309.16588
- [4]Gomez-Villa, A., Goswami, D., Wang, K., Bagdanov, A.D., Twardowski, B., Van De Weijer, J.: Exemplar-free continual representation learning via learnable drift compensation. In: European Conference on Computer Vision. pp. 473–490. Springer (2024)
- [5]Goswami, D., Soufii-Cormerais, A., Liu, Y., Kamath, S., Twardowski, B., Van De Weijer, J., et al.: Resurrecting old classes with new data for exemplar-free continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 28525–28534 (2024)
- [6]Guo, Z., Sun, Q., Zhang, Z., Yuan, H., Zhuang, H., Fu, X., Li, J.: Graphkeeper: Graph domain-incremental learning via knowledge disentanglement and preservation. Advances in Neural Information Processing Systems 38, 1145–1171 (2026)
- [7]Kim, Y., Li, Y., Panda, P.: One-stage prompt-based continual learning. In: European conference on computer vision. pp. 163–179. Springer (2024)arxiv.org/abs/2402.16189
- [8]Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
- [9]Li, C., Huang, Z., Paudel, D.P., Wang, Y., Shahbazi, M., Hong, X., Van Gool, L.: A continual deepfake detection benchmark: Dataset, methods, and essentials. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1339–1349 (2023)
- [10]Li, Q., Peng, Y., Zhou, J.: Fcs: Feature calibration and separation for non-exemplar class incremental learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 28495–28504 (2024)
- [12]Liang, Y.S., Li, W.J.: Inflora: Interference-free low-rank adaptation for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23638–23647 (2024)
- [13]Lomonaco, V., Maltoni, D.: Core50: a new dataset and benchmark for continuous object recognition. In: Conference on robot learning. pp. 17–26. PMLR (2017)
- [14]Magistri, S., Trinci, T., van de Weijer, J., Bagdanov, A., et al.: Elastic feature consolidation for cold start exemplar-free incremental learning. In: International Conference on Learning Representations. vol. 2024, pp. 31909–31931 (2024)
- [15]Oh, M., Sim, J.Y.: Asymmetric synthetic data update for domain incremental dataset distillation. In: The Fourteenth International Conference on Learning Representations (2026)
- [16]Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Domain2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)arxiv.org/abs/2304.07193
- [17]Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1406–1415 (2019)arxiv.org/abs/1812.01754
- [19]Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., Kira, Z.: Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11909–11919 (2023)arxiv.org/abs/2211.13218
- [20]Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5018–5027 (2017)
- [21]Wang, Q., He, Y., Dong, S., Gao, X., Wang, S., Gong, Y.: Non-exemplar domain incremental learning via cross-domain concept integration. In: European Conference on Computer Vision. pp. 144–162. Springer (2024)DOI
- [22]Wang, Q., He, Y., Dong, S., Song, X., Han, J., Luo, H., Gong, Y.: Dualcp: Rehearsal-free domain-incremental learning via dual-level concept prototype. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 21198–21206 (2025)arxiv.org/abs/2503.18042
- [23]Wang, Q., Song, X., He, Y., Han, J., Ding, C., Gao, X., Gong, Y.: Boosting domain incremental learning: Selecting the optimal parameters is all you need. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4839–4849 (2025)arxiv.org/abs/2505.23744
- [24]Xie, J., Yan, S., He, X.: General incremental learning with domain-aware categorical representations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14351–14360 (2022)arxiv.org/abs/2204.04078
- [25]Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., He, Y.: Boosting continual learning of vision-language models via mixture-of-experts adapters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23219–23230 (2024)arxiv.org/abs/2403.11549
- [26]Zhou, D.W., Cai, Z.W., Ye, H.J., Zhang, L., Zhan, D.C.: Dual consolidation for pre-trained model-based domain-incremental learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20547–20557 (2025)arxiv.org/abs/2410.00911