WorksComputer Science
Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks
Abstract
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves accuracy unchanged. On CIFAR-10, the same protocol modestly improves the test accuracy of a channel-narrowed ResNet-10, a standard residual architecture, and of a second capacity-limited convolutional network. These results show that how class emphasis is distributed in time can shape both the solution a network reaches and the internal structure of the network.
Introduction
Neural networks are typically optimized under a fixed training objective, yet the trajectory followed to optimize that objective is itself part of the learning process. Choices such as the ordering of examples, curriculum, learning-rate schedule, optimizer, and batch construction can change which solutions are reached even when the final objective is unchanged [1, 2, 3, 4, 5]. This raises a broader question: how does a training protocol change not only which function a network learns, but also where in parameter space different parts of that function are implemented?
The question is particularly interesting for tasks that contain several related sub-problems. In a multiclass classifier, for example, a single network must implement class-specific decision rules while sharing a finite parameter budget. More generally, multitask learning asks a shared model to solve several objectives whose gradients may not point in the same direction. When task gradients conflict, optimizing their sum can produce inefficient or destructive interference, motivating methods that explicitly rebalance, project, or otherwise manipulate task gradients [6, 7, 8]. Continual-learning methods approach a related problem from the opposite temporal regime: when tasks are presented sequentially, parameter isolation, masks, pruning, or architectural expansion can protect task-specific computations from interference [9, 10, 11, 12, 13, 14], and whether tasks are presented in blocks or interleaved changes what is learned [15]. These approaches
demonstrate that which parameters are allowed to change is consequential, but they typically impose parameter isolation explicitly or address tasks one after another.
Here we study a different regime: the architecture and the objective are unchanged, but optimization pressure is deliberately cycled between related sub-problems. The protocol is closely related to dynamical loss functions [16, 17], an approach motivated by oscillating strategies in biological systems, materials science and physics [18, 19, 20, 21, 22, 23], and distinct from learned, teacher-driven loss schedules [24]. A dynamical loss periodically changes the class weights of the cross-entropy loss, and this can rescue under-parameterized networks that fail under a static loss and can improve generalization in over-parameterized ones [16, 17]. This process can be analyzed through Hessian and neural-tangent-kernel dynamics [16]. Our questions are complementary: can a similar temporal forcing, implemented through mini-batch composition, rescue under-parameterized networks that fail under Uniform Batching (UB), in which every mini-batch has equal class proportions? And does it reorganize the parameters so that switching between class-specific solutions becomes more economical? Acting on batch composition rather than on the loss leaves the per-example loss and the optimizer untouched, so the protocol can be dropped into any training pipeline without modification.
This question connects to a broader literature on modularity. Modularity is commonly understood as an organization in which subsets of a system support distinguishable functions while interacting relatively weakly with other subsets. In neural networks, modularity can be defined structurally through connectivity [25] or functionally through subsets of weights required for particular computations [26, 27]. Standard networks can exhibit some specialization without being explicitly modular, but the extent to which useful functional modules emerge from ordinary training remains an active question. In particular, Csordás et al. [26] found that common architectures often fail to reuse the same submodules across related functions, while work on superposition has emphasized that finite representational capacity can instead lead multiple features to share neurons or directions [28, 29]. We therefore do not assume that a network becomes modular in the graph-theoretic sense. Instead, we ask whether the parameter changes associated with switching between sub-problems become localized: a weaker notion than modularity, but a directly measurable one.
Our central hypothesis is that periodically emphasizing each class concentrates the class-to-class change of the network onto a small, stable set of weights. If so, moving from one class-specialized state to another should involve fewer weights, largely the same ones every time, and a shorter distance in parameter space. We test this prediction with three complementary measurements: the size of the switching set, the smallest fraction of weights that carries most of the class-to-class change; its persistence, the overlap between the switching sets of consecutive cycles; and the switching distance between class-specialized states.
We first map the operating regime of Dynamical Batching (DB) on a three-arm spiral, where a capacity-constrained MLP that plateaus under UB reaches near-perfect training accuracy under a suitable amplitude–period combination. We then follow the same training runs over time: the class-to-class change concentrates on a small set of weights that persists from cycle to cycle, while the distance between class-specialized states decreases, although the drift of ordinary training also decreases. Finally, we test whether the accuracy effect extends beyond the toy setting with a channel-narrowed ResNet-10 on CIFAR-10 and three further convolutional architectures. Our contributions are:
A batch-level protocol that rescues a capacity-limited network. Without changing the loss, optimizer or architecture, DB lets a one-hidden-layer MLP that plateaus under UB fit the spiral; a grid over amplitude and period shows a broad region of gain bordered by a failure region.
Weight-level measures of class switching. We define a drift-corrected switching set, its persistence, and a switching distance, and track them against a UB control that uses identical bookkeeping.
Persistent, localized switching as a signature of the protocol. Under DB, the switching set is smaller and approximately twice as persistent as under UB across all five synthetic datasets, and the same pattern holds for a wider network that fits the data under both protocols.
Evidence beyond the toy setting. On CIFAR-10, DB modestly improves the test accuracy of a channel-narrowed ResNet-10, acting as a regularizer; a plain CNN with a narrow fully connected bottleneck shows a similar gain, while two wider networks show none beyond seed-to-seed noise.
Dynamical Batching
Protocol
Figure 1 shows the DB protocol schematically. Each mini-batch is built with a time-dependent number of samples from each class. Panel (a) shows the fraction of samples drawn from each class over time, at low and high amplitude. Panel (b) is a cartoon of parameter space: the three colored cones represent regions where the class of that color is correctly classified, and the lines display training trajectories.
Figure 1. Learning by Dynamical Batching (DB). (a) Class proportion , the fraction of each mini-batch drawn from class , over two training cycles (). At low amplitude (top, ) every batch still contains all three classes: the focal-class proportion rises from to , while the other two fall to . At high amplitude (bottom, ) the emphasized class dominates the batch, its proportion saturating near , while the other two nearly vanish. A cycle is consecutive periods, one sweep through the classes. (b) Schematic of the parameter space (illustrative, not data). Each class-specialized objective is satisfied in an extended, cone-shaped region (one cone per class, colored as in (a)), and the cones meet only near a common apex (Optimum, star). The shape only encodes that each class alone is satisfied in a large region of parameter space, while all classes together are satisfied only in a much smaller region. In the under-parametrized regime, Uniform Batching (UB, dotted gray) moves from initialization (circle) to a point that satisfies no class fully (Suboptimal, cross). DB (solid green) alternates between the cones, each period pulling the trajectory toward one class’s region before the next pulls it toward another.
Our working hypothesis is that under UB a capacity-limited network reaches a compromise point that satisfies no class fully, whereas DB forces the network to visit a different region of parameter space each time a different class is emphasized and, as training progresses, the trajectory approaches a region where all classes are classified correctly. Moving from one class-emphasized region to another should then require increasingly small changes in parameter space, which we test directly in Sec. IV.
To integrate the oscillations into standard optimizers without altering the loss function (standard cross-entropy optimized with SGD or Adam), we implement the protocol as a dataset sampling schedule. For a dataset of classes and period , one focal class is emphasized during its window. Its relative sampling weight follows a triangular ramp of amplitude :
where and . We call consecutive periods a cycle. The class proportions are for the focal class and for each of the other classes (Figure 1a). recovers UB.
Each mini-batch of nominal size is built by deterministic stratification: class contributes
examples, drawn with replacement within the class. The class composition is therefore fixed at every step; only the choice of samples within each class is random.2 In the final 5% of training steps all batches are uniform, so that every run ends under the same objective.
Experimental setup
To understand the learning dynamics in detail, we use a three-arm spiral, a multi-class variant of the classic two-spirals benchmark [30], with 100 training points per class and another 100 test points per class, generated by the same procedure with a different random seed; each training seed draws a new dataset and a new initialization (Appendix A). We train two one-hidden-layer MLPs with ReLU activations and Glorot initialization [31]: a small network with hidden units (303 parameters; under-parametrized for this task) and a large one with (3,003 parameters; over-parametrized). All main-text MLP experiments use 125,000 training steps, batch size and Adam [32]} (learning rate 0.002, , ), with zero weight decay and no gradient clipping.
fall short of by fewer than samples. This happens even under UB: and give . At large amplitudes, flooring also removes the non-focal classes entirely, since whenever .
Dynamical Batching succeeds where Uniform Batching plateaus
We ask whether DB can improve learning, and under which conditions. Figure 2 maps the accuracy gain over the (UB) baseline across amplitude and period (5 seeds per cell). A broad region of positive test-accuracy gain emerges for moderate amplitudes and periods, where the oscillating emphasis lets the network leave the poor solution reached under UB. At large amplitudes combined with short periods a failure region appears, consistent with the emphasis shifting faster than the network can consolidate. For the small network (), the best cell of the grid, at , improves test accuracy by +37.9 points over the grid’s UB baseline. Re-running this configuration and its UB control with 20 seeds each gives training and test accuracy under UB, against and under DB (Table I). The UB shortfall is not a matter of slower convergence within our training budget: UB training accuracy gains less than 10 points over the last two thirds of training and remains far below DB throughout (Appendix A1, Figure 5a), and doubling the budget to 250,000 steps raises it only to (Table II). For the large network (), UB already fits the training set, and DB improves test accuracy only slightly, by at most +1.7 points in the grid, at ; with 20 seeds, test accuracy is under UB and under DB.
Figure 2. Dynamical Batching (DB) on the three-arm spiral. (a) Accuracy gain in percentage points over the Uniform Batching (UB, ) run with the same period , across oscillation amplitude and period , for the small () and large () networks; training and test gain shown separately. Color scales differ between rows. Each cell averages 5 seeds; the yellow and green boxes mark the UB baseline () and the best DB configuration, , used in (b) and in Sec. IV. For the small network, a broad region of positive gain reaches +37.87 points in test accuracy, next to a failure region at large and small . The large network already reaches 99.00% train and 95.93% test accuracy under UB in this grid, and DB improves test accuracy slightly (up to +1.74 points) in a narrow band at low amplitude, just above the failure region. (b) Decision boundaries for a single UB run () and a single DB run at : the network is evaluated on a grid and shaded by predicted class, with training (circles) and test (crosses) samples colored by true label. UB produces a fragmented boundary that does not follow the arms (test accuracy 56.60%); DB recovers the spiral structure (test accuracy 94.47%).
| Method | Train acc. (%) | Test acc. (%) | Persistence | |||
| UB | ||||||
| DB |
Table II. Spiral, small network, , trained for 250,000 steps (twice the budget used elsewhere), mean s.d. over 20 seeds. Columns as in Table I.
Figure 5. Training curves on the spiral. Full-dataset training (dark) and test (light) accuracy over training for UB (amber) and DB (green); lines are the mean and bands s.d. across 20 seeds. The final 5% of training is uniform for both protocols. (a) Small network (), . After an initial rise, UB levels off, gaining less than 10 points of training accuracy over the last two thirds of training, and ends at training and test accuracy. DB overtakes it after steps and ends at and . The fast oscillation of the DB curves follows the periodic class emphasis. (b) Large network (), . Both protocols fit the training set ( UB, DB), UB faster than DB; test accuracy ends at (UB) and (DB).
Why does UB plateau when the same network can fit the data under DB? One candidate is a gradient-conflict plateau, related to the conflicting-gradient problem of multi-task learning [7]. When per-class gradients disagree, the balanced gradient can vanish at a point where no individual class is satisfied ( for every ), so the mean step of UB is zero there, whereas a class-focused batch follows a single, undiluted . If and the per-class gradients have equal norm, their mean pairwise cosine must equal , i.e. an angle of for (Appendix § 0.2.2 2). We measured this cosine for gradients computed on fixed, single-class probe sets, with all gradients evaluated at the same parameters (Appendix § 0.2.2 2, Figure 8a,Figure 8b). Under UB it sits at or close to throughout training, both in the small network that plateaus near 71% training accuracy and in the large network that fits the data. Under DB, evaluated at the class-emphasized states, the class gradients remain strongly opposed but do not cancel (cosine from early in training to after – cycles). Two conclusions follow. First, because the cancelling arrangement is equally present when UB succeeds, it cannot by itself explain the plateau of the small network: it indicates only that UB training stays close to a stationary point of the balanced loss, which is poor in the small network and good in the large one. Second, DB does not remove the conflict between classes; it changes which combination of the conflicting gradients the optimizer follows. The DB batch weights the classes unequally, so its expected gradient is with . Even at a point where the balanced sum vanishes, this step equals , which is generically nonzero: the network keeps moving where UB would stall. We regard this as a hypothesis for how DB leaves the poor solution, not as its explanation.
Figure 8. Additional measures on the spiral. Solid green: DB; dashed amber: UB; bands s.d. across 20 seeds. (a) Mean pairwise cosine between the per-class probe gradients , each computed on the training examples of a single class, with all three gradients evaluated at the same parameters (Appendix B 2), small network (). Under UB it sits at (dotted) throughout training, while training plateaus near 71% (Figure 5a). Under DB it starts at and settles at after cycles: the class gradients remain strongly opposed but do not cancel. (b) The same for the large network (). UB again sits close to (between and ), although it fits the training set (): the arrangement is generic under UB, not a signature of failure. DB stays near for the first cycles and settles at after cycles. (c) Participation ratio of [41], the effective number of weights carrying the displacement, small network. Under DB it falls from 90 to 28, agreeing in direction with (Figure 3a); under UB it drops within the first cycles and then rises to 40, with large seed-to-seed variability, so the two protocols separate only after cycle 75–80. (d) Switching-set size under DB (solid green) and UB (dashed amber) for energy thresholds of 50% (thin, lower), 80% (thick, main text) and 90% (thin, upper). After the first few cycles, each DB curve lies below the UB curve of the same threshold, ending at 0.07, 0.19 and 0.30 against 0.10, 0.32 and 0.47: DB needs fewer weights than UB to carry the same fraction of the class-to-class variation, whichever fraction is chosen.
Class switching localizes onto a small, persistent set of weights
We quantify how the protocol organizes the weights with three measures. Let be the average of the parameter vector over the training steps in the central half, , of class ’s period in cycle , where class dominates the batch.
To ask which weights change when the emphasis moves from one class to the next, we measure for each weight how much it varies across the class-focused periods of a cycle: , the standard deviation of across classes , after removing the part of that variation that is simply steadily drift across the consecutive class windows (Appendix § 0.2.2, Eq. (6)). The switching set is the smallest set of weights that together carry 80% of the total variation ; all other weights form the scaffold. We report its size as a fraction of all weights, . The ordering between protocols is unchanged for thresholds of 50% and 90% (Appendix § 0.2.2, Figure 8d).
Whether the switch is carried by the same weights every cycle is measured by the Jaccard overlap of consecutive switching sets,
the number of weights that are switching weights in both cycles, divided by the number that are switching weights in at least one of them.
The simplest measure of how far the network moves when it switches classes is the switching distance, the mean distance over the class pairs,
UB uses the same schedule and the same bookkeeping, but with , so class sampling stays uniform. is then not a class-emphasized state but the average parameter vector over the corresponding window of ordinary training, and its oscillation, switching set and distance measure the fluctuations and drift of ordinary training over the same windows. UB is therefore the natural control for how concentrated and how stable the motion is, but not for how large it is: DB moves between genuinely different class solutions and UB does not, so the distance between class states is expected to be larger under DB.
Results
Throughout this section the small network is trained at ,3 and the UB control is the identical run with .
By the end of training, switching between classes under DB is carried by fewer weights than the fluctuations of UB training, and largely by the same weights every cycle (Figure 3a,Figure 3b, Table I). Under DB, 80% of the class-to-class variation is carried by only 14% of the weights after cycles and by at the end of training; under UB, the same fraction of the ordinary training fluctuations requires about a third of the weights throughout ( at the end; Figure 3a). The participation ratio of , a threshold-free measure of the same concentration, agrees in direction but separates the protocols only late in training (Appendix § 0.2.2, Figure 8c). A small switching set could still be redrawn in every cycle, but it is not: consecutive switching sets overlap with a Jaccard index that peaks near 0.8 around cycle 30 and ends at under DB, against under UB (Figure 3b). Both measures are thus most favorable around cycle 30 and relax partially afterwards, while staying well separated from UB. At the same time, the distance between class-emphasized states falls about 4.5-fold, from 2.9 to (Figure 3c). This decrease is not by itself specific to DB: the drift of ordinary training over the same windows also falls, from to , mostly within the first few cycles, and the DB distance remains above it, as it must. Relative to that drift, the DB distance still shrinks, from roughly five times the UB value around cycle 10 to 2.7 times at the end. What is specific to DB is that the class states remain distinct while the change between them is carried by a small, persistent set of weights. The three quantities are complementary: a network could switch through a small set of weights that is redrawn every cycle, or through a stable set that moves ever farther. Figure 3e sketches the picture they give together.
Figure 3. Class switching concentrates on a small, persistent set of weights, and the distance between class states shrinks over training. Spiral, small network, , . One cycle is steps, giving 130 cycles, of which the first 122 are dynamical and the rest fall in the uniform final 5% and are excluded from the analysis. Solid green: Dynamical Batching (DB); dashed amber: Uniform Batching (UB, ). (a) Switching-set size : the fraction of all weights needed to carry 80% of the class-to-class variation. (b) Persistence of the switching set: the fraction of weights switching in either cycle that switch in both. (c) Switching distance (Eq. (4)), the mean of over the three class pairs. (d) Full-dataset training accuracy (black, left axis) and training loss (coral red, right axis; note the different loss scales) over one focal-class period ( steps; the horizontal axis counts steps from the start of that period), early and late in one DB run. The dashed vertical line marks the middle of the period (), where the focal class is most emphasized. The step-like changes of the loss coincide with the steps at which the number of samples of each non-focal class per batch changes (Eq. (2)). (e) Weight-level schematic (no data): early in training the class-to-class change is spread over many weights and redrawn every cycle; late in training it is carried by a small set of weights (highlighted) that is largely the same in every cycle, while the remaining scaffold barely oscillates. In (a)–(c), each point is the mean over 20 seeds and bands are s.d. across seeds.
This organization is not a by-product of the accuracy gap of the small network. In the large network (, ), both protocols fit the training set (mean training accuracy ), and the first two observations hold more strongly: under DB the switching set shrinks to of the weights, against under UB, and its persistence reaches , against . The distance between class states again falls, from 3.6 to , but here it closely tracks the drift of ordinary training throughout and ends close to it (; Appendix B4, Figure 9, Table IV).
| Method | Train acc. (%) | Test acc. (%) | Persistence | ||
| UB | |||||
| DB |
Table IV. Spiral, large network (), , mean s.d. over 20 seeds. Columns as in Table I.
Figure 9. Studying the structure of the network (analogous to Figure 3a–Figure 3c) for the large network (), where both protocols fit the training set. , giving 277 cycles, of which 264 are dynamical. (a) Switching-set size : DB drops to 0.06 within a few cycles and ends at 0.03; UB stays at 0.15–0.25 and ends at 0.15. The concentration is stronger than in the small network (Figure 3a). (b) Persistence of the switching set (Jaccard overlap of consecutive cycles): DB starts below UB (0.1) while the switching set is being formed, rises to 0.7 by cycle and stays at 0.55–0.75; UB stays at 0.30–0.34. (c) Switching distance: DB falls from 3.6 to 0.18, about twentyfold, closely tracking the drift of ordinary training over the same windows (UB, from 2.8 to 0.13).
The first two observations also largely generalize across data geometries (Appendix A, Table III). On five synthetic datasets trained with the same schedule, persistence is 1.8–2.2 times higher under DB than under UB in every case. The switching set is 26–46% smaller under DB on Spiral, Checkerboard, Dartboard and Rings, and marginally smaller, within the seed-to-seed spread, on Blobs, which UB solves almost perfectly. Neither measure tracks the accuracy gain, however. The same smaller, persistent set appears where DB rescues the network, on Spiral and on Rings (test accuracy against ), and where it leaves accuracy unchanged within noise, on Checkerboard, Dartboard and Blobs.
| Dataset | Method | Train acc. (%) | Test acc. (%) | |||
| Spiral | UB | |||||
| DB | ||||||
| Blobs | UB | |||||
| DB | ||||||
| Checkerboard | UB | |||||
| DB | ||||||
| Dartboard | UB | |||||
| DB | ||||||
| Rings | UB | |||||
| DB |
Table III. Accuracy and weight-level measures across five toy datasets, , mean s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime. : switching-set size; : persistence of the switching set; : switching distance (Sec. IV).
Figure 3d illustrates the functional side of this picture in one representative DB run. Late in training, accuracy is at its ceiling for most of the period, drops in its middle part while the loss rises, and recovers within 5–10 steps once the emphasis on the focal class has decreased, a small fraction of the steps of the period. Early in training, accuracy stays between and for almost the whole period. The abrupt onset and end of the dip ( and ) coincide with the flooring of Eq. (2): at these steps the number of examples of each non-focal class drops from 3 to 2 per batch, and later returns to 3. The non-focal classes never vanish from the batch (at the peak, , the batch holds 45 focal-class examples and 2 of each other class), but the step from 3 to 2 is a one-third cut in each non-focal class, which is why its effect on the loss is so abrupt. The dip therefore marks the interval in which the other classes are most under-represented, and the recovery is the functional counterpart of the switch that Figure 3a–Figure 3c quantify in weight space. Fast recovery late in training is what one would expect from a switch carried by fewer, persistent weights, but it also coincides with the network having learned the task; our measurements show that the two views change together, not that one causes the other.
| Method | Train acc. (%) | Test acc. (%) | Persistence | ||
| UB | |||||
| DB |
Table 1. Spiral, small network, , mean s.d. over 20 seeds. Accuracies are measured at the end of training (after the uniform final phase); switching measures at the last cycle of each protocol’s regime (the last dynamical cycle for DB, the last cycle for UB). : switching-set size; : persistence of the switching set; : switching distance (Eq. (4)).
A channel-narrowed CNN on CIFAR-10
The measurements above concern a single hidden layer trained on synthetic data. To check whether the accuracy effect of Sec. III carries over to a standard architecture and dataset, we apply the same batch-composition schedule [Eq. (1)] to a ResNet-10 [33] trained on CIFAR-10 [34] (). The network has four stages of one basic residual block each, with all channel widths scaled by a factor of 0.15 to reduce capacity. It is trained for 100,000 steps with batch size 128 using SGD with momentum 0.9 and weight decay ; the learning rate starts at 0.0679 and is divided by 10 after one and two thirds of training. We scan 14 amplitudes from to and 15 periods from to steps (values listed in Appendix § 1.5), with 5 seeds per configuration. Accuracies are measured at the end of training on 5,000-example subsets of the training and test sets. Under UB, averaged over the 15 cells of the column, the network reaches training and test accuracy: unlike the small spiral network, it fits its training set. All values in this section are mean s.d. over seeds. In the heatmaps, the gain of each cell is measured relative to the UB cell () with the same period; the numbers quoted in the text instead use the mean of the whole column, which pools all UB runs. Both baselines give the same gain for the best configuration to within 0.01 points (Appendix § 1.5). We do not repeat the weight-level analysis of Sec. IV here and leave it to future work.
A band of positive test-accuracy gain runs along the edge of the failure region, at large amplitudes and periods of one to a few hundred steps (Figure 4): 29 of the 195 DB configurations exceed the UB test accuracy, 13 of them by more than half a point. The maximum test accuracy is at and , a gain of +1.43 points, about 2.3 times the combined seed-to-seed s.d. of the two configurations (0.62), and neighboring cells along the same band reach comparable values [81.01% at , 80.98% at (100, 150) and 80.86% at (150, 200)]. Because this is the maximum over a grid selected on test accuracy, it is biased upward; the band of positive gain, rather than the single best cell, is the evidence for the effect. The gain comes with a decrease in training accuracy, to for that configuration, and every configuration that improves test accuracy by more than half a point has training accuracy between 90% and 99.5%, whereas configurations that still fit the training set gain at most 0.35 points: here the oscillations act as a regularizer, trading training fit for generalization, rather than rescuing an optimization failure as the small network on the spiral. At large amplitudes combined with short periods, both training and test accuracy collapse, as in the failure region of the spiral, and the boundary of this region moves to longer periods as the amplitude grows. The size of the gain is comparable to that of the over-parameterized spiral network (+1.74 points), not to the rescue of the under-parameterized one (+37.87 points).
Figure 4. Accuracy change on a channel-narrowed ResNet-10. Gain in training (left) and test (right) accuracy, in percentage points, relative to the Uniform Batching (UB) run () with the same period (UB averaged over all periods: train 100.00%, test 79.84%), across amplitude and period , for a ResNet-10 with channel widths scaled by 0.15, trained on CIFAR-10. Each cell averages 5 seeds. The color scale is clipped at −30 points: dark blue cells are runs that collapse, mostly to chance level. Axes are categorical; only a subset of the tested values is labeled.
The architecture sweep of Appendix § 1.5 (Table V) indicates which models benefit. Alongside the narrow ResNet-10 we tested a plain CNN whose capacity is limited by a 50-unit fully connected bottleneck, a VGG-style Myrtle5 network, and the same ResNet-10 at half width. The plain CNN shows the largest and broadest gain (+2.00 0.96 points at , , with 42 configurations improving test accuracy by more than one point), again
with lower training accuracy: it does not quite fit its training set under UB (98.30%). Myrtle5 and the half-width ResNet-10 gain at most about half a point, within noise. The benefit is therefore not restricted to networks that fail to fit their data, since the narrow ResNet-10 fits it; across the four architectures it appears in the two narrowest ones, which are also those with the lowest UB test accuracy. With two architectures on each side, we read this as consistent with, not as a demonstration of, a benefit concentrated in capacity-limited models.
Discussion
Our results show that Dynamical Batching (DB), relative to Uniform Batching (UB), changes both whether a capacity-limited network reaches a good solution and how the change between class-specialized states is organized in its weights. DB introduces no modular architecture, routing mechanism, task mask, or sparsity penalty; it only changes which class receives the strongest optimization pressure. Under DB, the change between class-emphasized states is carried by a set of weights that is about twice as persistent from cycle to cycle as under UB on every dataset we tested, and smaller on all of them.
This picture connects three lines of work. First, dynamic-objective methods show that changing the optimization target over time can improve trainability and generalization [16]; we find that the same can be achieved through batch composition alone. Second, multitask-learning work identifies interference between task gradients as a central optimization obstacle [6, 7, 8]; our probe-gradient measurements show that per-class gradients nearly cancel under UB, equally when training plateaus and when it succeeds, so conflict alone does not explain the plateau, and that DB changes how the conflicting gradients are combined (Appendix B2). Third, modularity and continual-learning work shows that allocating different functions to distinguishable parameter subsets can reduce interference [10, 11, 26], and optimization is known to concentrate in small subspaces or subnetworks [35, 36]. Our experiments suggest a bridge: temporal class cycling lets a network carry the class-to-class change in a stable subset of its parameters, without that subset being imposed.
The connection to nonequilibrium training is suggestive. Periodically varying related goals can produce modular organization without explicitly selecting for it [37], can facilitate adaptation [38], and has an optimal time scale [39]; Falk et al. [22] showed that periodic target switching can teach a physical material to change functionality through a small subset of design variables. We observe a related effect in gradient-trained networks: repeated exposure to different but related class objectives is associated with a small, persistent parameter footprint for switching. In our setting, however, this footprint is not a set of class-specific modules.
Several limitations are important. First, the weightlevel evidence comes from one-hidden-layer MLPs on synthetic three-class tasks; on CIFAR-10 we show only an accuracy effect. Second, neither the size nor the persistence of the switching set tracks the accuracy gain. Third, our control is ordinary UB training, whose window-to-window motion is largely stochastic; a periodic emphasis on class-agnostic data subsets would separate effects of class structure from those of periodic forcing per se. Fourth, our notion of localization is weaker than the functional modularity of Csordás et al. [26] and Hod et al. [27], and our measurements are correlational: a causal test would freeze the switching set late in training and ask whether switching and recovery fail, compared with freezing the same number of random weights. Fifth, the protocol requires a meaningful partition of the data, and amplitude and period interact strongly; configurations were selected on test accuracy, and we have not compared DB with other regularizers or class-reweighting schemes on CIFAR-10.
Overall, the main result is not that cyclic training universally improves neural networks. Rather, it suggests that how optimization pressure is distributed in time can shape the internal organization of the solution: a fixed architecture trained under a changing sequence of related sub-problems can develop a small, persistent parameter response to those sub-problems.
Data availability
The protocol is fully specified in Sec. II [Eqs. (1)–(2)], the weight-level measures in Sec. IV and Appendix § 0.2.2 [Eq. (B2)], and the probe-gradient test in Sec. III and Appendix § 0.2.2 2. Dataset generation, grid-search configurations and baselines for the synthetic datasets and CNN architectures are reported in Appendixes A and C. The synthetic datasets are generated by the released code, and CIFAR-
10 is publicly available [34]. Code and configuration files to reproduce all results are available at https://github.com/samuellozanoiglesias/dynamical_batching and archived on Zenodo [40].
Use of AI tools
Generative AI tools were used to help refine hypotheses about the mechanism behind the observed accuracy gains, to provide feedback on experimental design (e.g., the choice of control conditions and ablations), to search for and summarize related literature while writing the Introduction and Discussion, to suggest structural edits to the manuscript, and to polish the writing. They were not used to generate datasets, to formulate or prove mathematical claims, to clean or reformat data, or to analyze data. The authors reviewed all AI-assisted content and take full responsibility for the content of this article.
Acknowledgments
The project that gave rise to these results received the support of a fellowship from Fundación Ramón Areces. We acknowledge support from Ramón y Cajal program (RYC2021-032055-I) funded by MCIN/AEI/10.13039/501100011033 and by European Union NextGenerationEU/PRTR, a Research Grant from HFSP (Ref.-No: RGEC33/2024) with the award DOI (https://doi.org/10.5204/HFSP.RGEC332024.pc.gr.194170) and grant PID2023-147067NB-I00 funded by MCIN/AEI/10.13039/501100011033 and by ERDF/EU. Samuel Lozano is a Graduate Fellow of the Madrid City Council (Spain) at the Residencia de Estudiantes, 2025-2026.
Additional Toy Experiments
Throughout the Appendices, DB denotes Dynamical Batching and UB Uniform Batching (). This appendix expands the toy-network experiments of the main text. We first show the training curves behind the spiral results, then test robustness to batch size on the spiral, and finally evaluate the weight-level measures on four additional synthetic datasets. The purpose is not to claim universal optimal hyper-parameters, but to determine whether the qualitative pattern survives changes in data geometry.
Training curves on the spiral: UB plateaus, DB does not
Figure 5 shows training and test accuracy over the whole run for the 20-seed experiments of Sec. IV and Appendix § 0.2.2 § 1.4.4. For the small network (panel a), UB rises during the first steps and then improves only slowly: over the last two thirds of training its training accuracy gains less than 10 points, ending at ( test), with a wide spread across seeds. The same network under DB passes the UB curve after steps and ends at training and test accuracy. For the large network (panel b), both protocols fit the training set, and UB does so faster than DB; DB ends with slightly higher test accuracy ( against ).
UB accuracy in the small network is still increasing slowly at the end of training, so we tested whether a longer budget closes the gap. Doubling the training length to 250,000 steps, with everything else unchanged (20 seeds per protocol, uniform final 5%), barely moves UB: training accuracy rises to and test accuracy to , that is, by 4.8 and 3.2 points for 125,000 additional steps. DB reaches training and test accuracy, so the gap remains 24.7 points in training and 32.5 points in test accuracy (Table II). The difference is therefore not a matter of DB reaching the UB solution sooner within a fixed budget. The weight-level picture is also unchanged at the longer budget: under DB the switching set is about half the size of that under UB and about 1.6 times as persistent. All other small-network results use 125,000 steps, for consistency across datasets and protocols.
Batch-size robustness on the spiral
We swept period and amplitude for two network widths ( and hidden units) on the three-arm spiral, with the batch size increased from to and the optimizer and architectures unchanged. The corresponding grid maps (Figure 6, the analogue of Figure 2a without decision boundaries) show the same qualitative structure: a broad region of positive gain and a failure region at large amplitudes and short periods.
Figure 6. Grid search on the spiral at a larger batch size (). Accuracy gain in percentage points over the UB run () with the same period , across oscillation amplitude and period , for the small () and large () networks; training and test gain are shown separately. As for in the main text, a broad region of positive gain unlocks the capacity-constrained network, bordered by a failure region at extreme amplitudes and short periods where the emphasis shifts faster than the network can consolidate. Each cell represents the average of 5 independent seeds.
For the small network (), the UB baseline reaches 70.20% training and 55.60% test accuracy, and the configuration at reaches 99.33% training and 94.93% test accuracy. For the large network (), the UB baseline reaches 99.33% training and 96.13% test accuracy, while the best tested oscillating configuration reaches 97.33% test accuracy at . The numerical optima differ from those of the main-text grid because the batch size and sweep ranges differ; we therefore treat the grid search as evidence for a broad operating regime rather than for a universal optimal .
Batch composition is deterministic (Eq. (2)), so the protocol contains no multinomial sampling noise at any batch size. What the batch size changes is the effect of the flooring: the realized composition, and the amplitude above which non-focal classes vanish from the batch (), both depend jointly on , and . That the phase structure survives a fourfold change in suggests that it is not tied to a particular flooring pattern. The failure regions do move: at it recedes to somewhat shorter periods, most visibly for the large network, so the boundary depends on the batch size as well as on and .
Weight-level measures across five toy datasets
To test whether the pattern of Sec. IV is specific to the spiral geometry, we repeated the measurements on five synthetic benchmarks. All use the small MLP of the main text (one hidden layer, 50 units, ReLU activations, Glorot initialization) trained for 125,000 steps with , with 20 seeds per configuration.
Spiral: 3 classes, 100 points per class. Class follows an arm with angular offset : its points have radii evenly spaced from 0.01 to 1 and angles evenly spaced over an interval of length (two full turns), and Gaussian noise of standard deviation 0.2 is added to each angle.
Blobs: 3 classes, 100 points per class. Fifteen Gaussian modes are equally spaced on a circle of radius 0.6 and assigned to the three classes in rotation, so that each class is a mixture of 5 modes apart and neighboring modes belong to different classes; noise standard deviation 0.05.
Checkerboard: 3 classes, 100 points per class, drawn from the cells of a grid whose labels alternate among the three classes; noise standard deviation 0.1.
Dartboard: 3 classes, 100 points per class. The disc between radii 0.1 and 1.0 is divided into 3 radial rings and 12 angular sectors, and the cell in ring and sector is assigned class . Points
are sampled uniformly in area within the central part of each cell, leaving an empty margin of 15% of the cell’s radial and angular width on each side; Gaussian noise of standard deviation 0.03 is then added to both coordinates.
Rings: 3 classes, 100 points per class. Each class consists of 2 concentric rings, interleaved with those of the other classes, starting at radius 0.1 with a radial separation of 0.15 between consecutive rings; noise standard deviation 0.05.
Figure 7 shows the five datasets. All five are normalized to the region , so the networks receive inputs on the same scale for every geometry. For every dataset, the test set (100 points per class) is generated by the same procedure as the training set with a different random seed. Each training seed also sets the dataset seed, so each seed draws a new dataset (training and test sets) and a new initialization, and the spread across seeds reported throughout includes dataset-to-dataset variability. For the spiral, whose radii are evenly spaced, training and test points therefore lie at the same radii and differ only in their angular noise.
Figure 7. The five synthetic datasets. Training samples (circles) and test samples (crosses), colored by class, for (a) Spiral, (b) Blobs, (c) Checkerboard, (d) Dartboard and (e) Rings, each with 3 classes and 100 training and 100 test points per class, all within the region . Each panel shows one realization (seed 0) generated from the exact configuration used for training; every training seed draws a new realization.
For all datasets we use , the values selected for the main-text spiral experiment. This is a transfer test rather than a per-dataset optimization, so Table III should not be read as a comparison of the maximum attainable accuracy on each dataset.
Three observations follow from Table III.
The switching set is more persistent under DB on every dataset. On all five datasets the switching set is more persistent under DB than under UB, by a factor of 1.8–2.2.
The switching set is smaller under DB on every dataset. DB needs fewer weights than UB to carry 80% of the class-to-class variation on Rings (0.13 vs. 0.24), Spiral (0.19 vs. 0.32), Checkerboard (0.16 vs. 0.26) and Dartboard (0.23 vs. 0.31), a reduction of 26–46%, and marginally fewer on Blobs (0.16 vs. 0.20, within the seed-to-seed spread). The concentration reported for the spiral in Sec. IV therefore transfers to all five geometries, although its magnitude varies.
Neither measure tracks the accuracy gain. DB improves test accuracy substantially on Spiral (+35.9 points) and Rings (+13.3), and leaves it unchanged within seed-to-seed variability on Checkerboard (+1.2), Dartboard (+1.5) and Blobs (−1.1). Persistence is higher under DB on every dataset, and the switching set is smaller, so both
properties appear with a gain (Spiral, Rings) and without one (Checkerboard, Dartboard, and Blobs). They are therefore a signature of the protocol, not of its accuracy benefit. The switching distance is larger under DB on every dataset, as expected, since only DB moves between class-specialized solutions (Sec. IV).
Taken together, the five datasets show that under DB the switch is carried by largely the same weights from cycle to cycle on all five geometries, and by fewer weights than under UB (within noise on Blobs), whereas the accuracy gain is specific to some of them. Nor is this organization a by-product of an accuracy gap: the set is smaller and more persistent under DB on Checkerboard and Dartboard, where both protocols reach the same accuracy, just as on Spiral and Rings; it is also about twice as persistent on Blobs, which UB already solves; and it holds in the large network of Appendix § 0.2.2 4, where both protocols fit the data.
Weight-level switching analysis
Unless stated otherwise, results use the setting of Sec. IV: small network (), , 20 seeds per protocol, dynamical cycles only. The large network () uses , the best configuration of its grid in Figure 2a, also with 20 seeds per protocol (277 cycles, of which the first 264 are dynamical). Solid green is DB and dashed amber is UB; bands are s.d. across seeds.
Definition of the switching set
Let be the position of class in the cycle, the cycle mean of weight (since each is itself a time average, is the average of weight over the central steps of the cycle), and its drift between consecutive cycles. Because the class windows are consecutive in time, a weight that drifts by per cycle moves by from one class window to the next, and would differ between class states even without any class-specific change. For each weight we therefore remove a linear trend across the cycle, with its slope set by this drift, and quantify class-to-class switching by the variance
of the residuals:
The intercept centers the residuals, , so is the variance across classes of the de-trended states; equivalently, . In the first cycle of each regime there is no previous cycle, and no trend is removed. All quantities are computed from the stored class states of every cycle. The switching set is the smallest set of weights with , obtained by ranking weights by .
a. Why the slope is taken from the previous cycle. Two simpler choices fail for identifiable reasons. Without any trend removal, , pure drift produces : a class pattern that repeats identically every cycle. A network that only drifts would then show a switching set made of its fastest-drifting weights, and that set would appear highly persistent. Fitting the slope by least squares to the class states of the current cycle removes this drift, but it also removes two of the degrees of freedom, the mean and the slope, leaving only residual directions. For a single direction is left: with classes indexed by their position in the cycle, and . A weight that separates the first and the last class of the cycle, pattern , would then have and never enter the switching set, and the result would depend on which class the cycle happens to start with. Within one cycle, a class pattern that is linear in cycle position cannot be distinguished from drift. The drift is estimated from outside the cycle, from the previous cycle mean, and is zero under a closed limit cycle whatever the oscillation amplitude, which is why Eq. (6) separates the two.
Per-class gradient geometry
a. What is measured. For each class , is the gradient, with respect to all network parameters, of the mean cross-entropy over a probe set that contains only examples of class . The probe sets are drawn once, before training, and kept fixed throughout; with 100 training examples per class, each contains all training examples of its class. The gradient is a diagnostic computed on stored parameters, not the gradient of any training mini-batch. This distinction matters for UB. A UB mini-batch mixes the three classes, and its gradient is approximately the balanced sum ; comparing such mixed gradients with each other would say nothing about conflict between classes. The probe gradients instead isolate the contribution of each class, for UB and DB alike.
In each cycle, and at each of the class states used for the switching measures (Sec. IV), we evaluate all probe gradients at that same point and take their mean pairwise cosine; the reported value is the mean over the states of the cycle. Evaluating the gradients at the same parameters is what the argument below requires. Evaluating each class gradient at its own window center instead compares gradients at different parameters. Under DB, whose class states are far apart early in training, this produces an apparent cosine near 0 in the first cycles that disappears when the gradients are evaluated at the same point.
b. Why . At a stationary point of the balanced objective, . Taking the inner product with gives , so for equal norms the mean pairwise cosine is exactly . A cosine of therefore indicates that the per-class gradients nearly cancel, so that the balanced step is a small residual of large, opposing class gradients. One might suspect that this cancellation is confined to the output layer: given the current hidden features, the output layer solves a convex problem that balanced sampling brings quickly to its optimum, where its per-class gradients cancel whatever the quality of the features. Our measurements do not support this: restricted to the hidden-layer parameters or to the readout parameters alone, the cosine takes the same values as over all parameters, in both networks, close to under UB and under DB.
The converse also deserves care. A mean cosine of implies exact cancellation only for gradients of equal norm; the cosine alone is therefore evidence of near-cancellation, not a proof of it. Moreover, at a class-emphasized DB state the optimizer follows the weighted sum , so near-stationarity of that weighted objective does not require the unweighted sum to vanish; a cosine above under DB is therefore expected and should not be read as reduced conflict.
Results. Under UB (Figure 8a,Figure 8b), the per-class gradients cancel throughout training in both networks, whether training ultimately succeeds or not. The cancelling arrangement is therefore the generic state of balanced training at this scale. It indicates that UB training stays close to a stationary point of the balanced loss over the time scale of a cycle, a poor one in the small network and a good one in the large network, and is not by itself the signature of a failed run. What distinguishes the small network is that it stays at a poor solution while in this arrangement (Sec. III, Figure 5a). Under DB, the per-class gradients at the class-emphasized states are almost as strongly opposed ( early in training, after –50 cycles in both networks) but do not cancel. DB therefore does not remove the conflict between classes. What it changes is the combination of the conflicting gradients that the optimizer follows: at each point, the batch weights the focal class more heavily, so the step is not the near-cancelling balanced sum. Whether this is what lets DB leave the poor solution remains a hypothesis.
Robustness of the concentration measure
The participation ratio, a threshold-free measure of how many weights carry the displacement between two class states, agrees in direction with the switching-set size, but separates the two protocols only late in training (Figure 8c). The switching-set size depends on the energy threshold, but its ordering between the protocols does not: at 50%, 80% and 90% it ends at 0.07, 0.19 and 0.30 under DB against 0.10, 0.32 and 0.47 under UB (Figure 8d).
Persistence does not require an accuracy gap: the large network
In the small network, DB fits the data and UB does not, so differences in weight dynamics could in principle be a by-product of the accuracy gap. The large network removes this confound: both protocols fit the training set (Figure 5b, Table IV); training accuracy 99.9 0.5% under UB and 100.0 0.0% under DB, test accuracy 96.6 1.5% and 98.2 1.2%, respectively (20 seeds).
The first two main-text observations hold, and are stronger, when there is no accuracy gap to explain them (Figure 9a,Figure 9b). The third needs the same qualification as in the main text: the distance between class states falls, but so does the drift of ordinary UB training over the same windows (from 2.8 to 0.13), and the two end close to each other (Figure 9c).
Additional CNN architectures on CIFAR-10
We repeated the grid search of § 0.5 on three further architectures to determine which models benefit from the protocol: a plain CNN with a fully connected bottleneck, a VGG-style Myrtle5 network, and the ResNet-10 of the main text at half width.
All architectures are trained for 100,000 steps with batch size 128 using SGD with momentum 0.9 and weight decay , without gradient clipping or Nesterov momentum. The initial learning rate is 0.05 for the plain CNN and 0.0679 for all other architectures, and in every case it is divided by 10 at steps 33,333 and 66,666, one and two thirds of training. Every grid covers the same 14 amplitudes, , and 15 periods, steps, that is, 210 configurations of which 195 are DB (). Training and test accuracy are evaluated at the end of training on 5,000-example subsets of the training and test sets. The UB baseline is the mean over the 15 cells of the column, whose runs differ only by seed-to-seed and run-to-run variability, and its uncertainty is the s.d. over all runs in that column. In the heatmaps (Figure 4 and Figure 10), the gain of each cell is instead computed relative to the cell with the same period . The two baselines differ only by the variability between cells (UB test accuracy 79.61–80.16% across periods for the narrow
Figure 10. Grid searches for the additional CNN architectures. Gain in training (left) and test (right) accuracy, in percentage points, relative to the UB run () with the same period , across amplitude and period , on CIFAR-10. Each cell averages 5 seeds. The color scale is clipped at points; dark blue cells are runs that collapse. Axes are categorical, as in Figure 4. (a) Plain CNN with a 50-unit fully connected bottleneck (UB: train 98.30%, test 75.97%): a broad region of positive test gain at intermediate to large amplitudes and periods of about a hundred steps or more, with reduced training accuracy. (b) Myrtle5 (UB: 100.00%, 85.10%). (c) ResNet-10 at width 0.5 (UB: 100.00%, 85.98%). In (b) and (c) the test accuracy outside the failure region is flat to within seed-to-seed noise, and the training set is fitted wherever training does not collapse.
ResNet-10), and the best-cell gains they give differ by at most 0.32 points: +1.42 against +1.43 for the narrow ResNet-10, +2.15 against +2.00 for the plain CNN, +0.21 against +0.53 for Myrtle5 and +0.08 against +0.22 for the half-width ResNet-10 (same- against pooled baseline). No conclusion depends on this choice. Uncertainties of single configurations are s.d. across seeds; for a difference we quote the combined s.d. . All “best” values are maxima and are therefore biased upward by the selection, equally for every architecture. All heatmap’s points include the average over 5 independent seeds.
A test-accuracy gain beyond the seed-to-seed variability appears only in the two narrowest models, the ResNet-10 at width 0.15 (main text) and the plain CNN with a 50-unit bottleneck, and in both it comes with lower training accuracy. The two wider models, Myrtle5 and the ResNet-10 at width 0.5, show no gain beyond noise. Whether a model fits its training set under UB does not separate the two groups: the narrow ResNet-10 reaches 100% training accuracy under UB and still benefits. With two architectures on each side, the protocol appears useful primarily in capacity-limited networks, but we have not isolated whether width, the resulting test accuracy, or another property of these models is the relevant variable, nor the mechanism in these CNNs. In every architecture, large amplitudes combined with short periods make training collapse, often to chance level, and the failure boundary moves to longer periods as the amplitude grows (Figure 4 and Figure 10).
Plain CNN with a fully connected bottleneck
Three convolutional stages with 16, 32 and 64 channels, kernels, ReLU activations, no batch normalization, max pooling after every stage and no residual connections, followed by a 50-unit fully connected bottleneck before the 10-way classifier. Its capacity is limited by the bottleneck, like that of the small spiral network by its hidden layer.
Results: The UB baseline reaches training and test accuracy, so the network does not quite fit its training set. A broad region of positive test gain appears [Figure 10(a)]: 85 of the 195 DB configurations improve on the UB test accuracy, 42 of them by more than one point. The best DB test accuracy is (at , ), a gain of points, and several other cells come within 0.1 points of it [, and ]. Training accuracy in the best configuration falls to , and the highest DB training accuracy, (at , ), is within noise of the baseline: as for the narrow ResNet-10, the protocol acts as a regularizer.
Myrtle5
A VGG-style Myrtle5 network [42]: four convolutional stages with one block each, channel widths doubling from 64 up to 512, kernels, batch normalization, and pooling at the last stage.
Results: The UB baseline reaches training and test accuracy. Under DB, training accuracy stays at wherever training does not collapse, and the best test accuracy is (at , ). The difference, points, is smaller than its combined s.d., so we treat it as no effect. The distinction from the capacity-limited models rests on this ratio of gain to seed noise, not on how the configurations were selected, which is the same for all architectures: points is about 2.3 times its combined s.d. (0.62) and about twice (0.96), whereas for the two wider networks the gain is below its combined s.d.
ResNet-10 at half width
The ResNet-10 of the main text, four stages of one basic block each, with width multiplier 0.5 instead of 0.15.
Results: The UB baseline reaches training and test accuracy. The best DB test accuracy is (at , ), a difference of points. No DB configuration improves test accuracy by more than half a point. Widening the same architecture from 0.15 to 0.5 thus raises the UB test accuracy by about six points and removes the benefit of the protocol.
| Architecture | Fits train (UB)? | Base train | Base test | Best test | Train at best | Test gain | |
| ResNet-10 (width 0.15) (main text) | Yes | Yes | |||||
| Plain CNN + FC bottleneck (50) | No | Yes | |||||
| Myrtle5 (base width 64) | Yes | No | |||||
| ResNet-10 (width 0.5) | Yes | No |
Table 5. Architectures tested on CIFAR-10 and their response to DB. Test-accuracy gains beyond seed-to-seed noise appear only in the two capacity-limited architectures (top two rows). “Base” values are UB () means over the 15 cells of the column; “Best test” is the maximum over the DB configurations () of the grid, and “Train at best” the training accuracy of that same configuration. Every configuration is run with 5 seeds. Values are mean s.d. over seeds (accuracies in %); the test gain is given with the combined s.d. .
References
References
- [1]Y. Bengio, J. Louradour, R. Collobert, and J. Weston, Curriculum learning, in Proceedings of the 26th Annual International Conference on Machine Learning (2009) pp. 41–48.DOI
- [2]L. N. Smith, Cyclical learning rates for training neural networks, in IEEE Winter Conference on Applications of Computer Vision (WACV) (2017) pp. 464–472.arxiv.org/abs/1506.01186
- [3]B. Neyshabur, Implicit Regularization in Deep Learning, Ph.D. thesis, Toyota Technological Institute at Chicago (2017), arXiv:1709.01953.arxiv.org/abs/1709.01953
- [4]J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalker, Gradient descent on neural networks typically occurs at the edge of stability, in International Conference on Learning Representations (2021).arxiv.org/abs/2103.00065
- [5]J. M. Cohen, A. Damian, A. Talwalkar, J. Z. Kolter, and J. D. Lee, Understanding optimization in deep learning with central flows, in International Conference on Learning Representations (2025) arXiv:2410.2406.arxiv.org/abs/2410.24206
- [6]O. Sener and V. Koltun, Multi-task learning as multi-objective optimization, in Advances in Neural Information Processing Systems, Vol. 31 (2018).arxiv.org/abs/1810.04650
- [7]T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, Gradient surgery for multi-task learning, in Advances in Neural Information Processing Systems, Vol. 33 (2020).arxiv.org/abs/2001.06782
- [8]Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks, in Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80 (PMLR, 2018) pp. 794–803.
- [9]J. Kirkpatrick, R. Pascanu, N. Rabinowitz, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences 114, 3521 (2017).
- [10]J. Serrà, D. Surís, M. Miron, and A. Karatzoglou, Overcoming catastrophic forgetting with hard attention to the task, in Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80 (PMLR, 2018) pp. 4548–4557.arxiv.org/abs/1801.01423
- [11]A. Mallya, D. Davis, and D. Lazebnik, Piggyback: Adapting a single network to multiple tasks by learning to mask weights, in Proceedings of the European Conference on Computer Vision (ECCV) (2018) pp. 67–82.arxiv.org/abs/1801.06519
- [12]A. Mallya and S. Lazebnik, PackNet: Adding multiple tasks to a single network by iterative pruning, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) pp. 7765–7773.arxiv.org/abs/1711.05769
- [13]A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016).arxiv.org/abs/1606.04671
- [14]M. Ruiz-Garcia, Model architecture can transform catastrophic forgetting into positive transfer, Scientific Reports 12, 10736 (2022).
- [15]T. Flesch, J. Balaguer, R. Dekker, H. Nili, and C. Summerfield, Comparing continual task learning in minds and machines, Proceedings of the National Academy of Sciences 115, E10313 (2018).DOI
- [16]M. Ruiz-Garcia, G. Zhang, S. S. Schoenholz, and A. J. Liu, Tilting the playing field: Dynamical loss functions for machine learning, in Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139 (PMLR, 2021) pp. 9157–9167.
- [17]E. Lavín-Pallero and M. Ruiz-García, Dynamical loss functions shape landscape topography and improve learning in artificial neural networks, Journal of Statistical Mechanics: Theory and Experiment 2025, 123402 (2025).arxiv.org/abs/2410.10690
- [18]A. Murugan and H. M. Jaeger, Bioinspired nonequilibrium search for novel materials, MRS Bulletin 44, 96 (2019).DOI
- [19]A. Murugan, K. Husain, M. J. Rust, C. Hepler, J. Bass, J. M. J. Pietsch, P. S. Swain, S. G. Jena, J. E. Toettcher, A. K. Chakraborty, et al., Roadmap on biology in time varying environments, Physical Biology 18, 041502 (2021).DOI
- [20]V. Mustonen and M. Lä ssig, From fitness landscapes to seascapes: non-equilibrium dynamics of selection and adaptation, Trends in Genetics 25, 111 (2009).
- [21]J. Cairns, F. Borse, T. Mononen, T. Hiltunen, and V. Mustonen, Strong selective environments determine evolutionary outcome in time-dependent fitness seascapes, Evolution Letters 6, 266 (2022).DOI
- [22]M. J. Falk, J. Wu, A. Matthews, V. Sachdeva, N. Pashine, M. L. Gardel, S. R. Nagel, and A. Murugan, Learning to learn by using nonequilibrium training protocols for adaptable materials, Proceedings of the National Academy of Sciences 120, e2219558120 (2023).DOI
- [23]M. Ruiz-Garcia, A. J. Liu, and E. Katifori, Tuning and jamming reduced to their minima, Physical Review E 100, 052608 (2019).
- [24]L. Wu, F. Tian, Y. Xia, Y. Fan, T. Qin, J.-H. Lai, and T.-Y. Liu, Learning to teach with dynamic loss functions, in Advances in Neural Information Processing Systems, Vol. 31 (2018) pp. 6466–6477.arxiv.org/abs/1810.12081
- [25]D. Filan, S. Casper, S. Hod, C. Wild, A. Critch, and S. Russell, Clusterability in neural networks, arXiv preprint arXiv:2103.03386 (2021).arxiv.org/abs/2103.03386
- [26]R. Csordás, S. van Steenkiste, and J. Schmidhuber, Are neural nets modular? Inspecting functional modularity through differentiable weight masks, in International Conference on Learning Representations (2021).
- [27]S. Hod, D. Filan, S. Casper, A. Critch, and S. Russell, Quantifying local specialization in deep neural networks, arXiv preprint arXiv:2110.08058 (2021).arxiv.org/abs/2110.08058
- [28]N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah, Toy models of superposition, Transformer Circuits Thread (2022), arXiv:2209.10652.
- [29]A. Scherlis, K. Sachan, A. S. Jermyn, J. Benton, and B. Shlegeris, Polysemanticity and capacity in neural networks, arXiv preprint arXiv:2210.01892 (2022).arxiv.org/abs/2210.01892
- [30]K. J. Lang and M. J. Witbrock, Learning to tell two spirals apart, in Proceedings of the 1988 Connectionist Models Summer School (Morgan Kaufmann, 1988) pp. 52–59.
- [31]X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 9 (PMLR, 2010) pp. 249–256.
- [32]D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, in International Conference on Learning Representations (2015).DOI
- [33]K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) pp. 770–778.arxiv.org/abs/1512.03385
- [34]A. Krizhevsky, Learning Multiple Layers of Features from Tiny Images, Tech. Rep. (University of Toronto, 2009).
- [35]G. Gur-Ari, D. A. Roberts, and E. Dyer, Gradient descent happens in a tiny subspace, arXiv preprint arXiv:1812.04754 (2018).
- [36]J. Frankle and M. Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks, in International Conference on Learning Representations (2019).
- [37]N. Kashtan and U. Alon, Spontaneous evolution of modularity and network motifs, Proceedings of the National Academy of Sciences 102, 13773 (2005).pubmed.ncbi.nlm.nih.gov/16174729
- [38]M. Parter, N. Kashtan, and U. Alon, Facilitated variation: How evolution learns from past environments to generalize to new environments, PLoS Computational Biology 4, e1000206 (2008).DOI
- [39]V. Sachdeva, K. Husain, J. Sheng, S. Wang, and A. Murugan, Tuning environmental timescales to evolve and maintain generalists, Proceedings of the National Academy of Sciences 117, 12693 (2020).DOI
- [40]N. Ratier-Werbin, S. Lozano, and M. Ruiz-García, Numerical Batching: code and configuration files, Zenodo (2026), version v1.0.1, doi:10.5281/zenodo.23471602; all versions: doi:10.5281/zenodo.23349474.
- [41]A. Ingrosso and S. Goldt, Data-driven emergence of convolutional structure in neural networks, Proceedings of the National Academy of Sciences 119, e2201854119 (2022).
- [42]V. Shankar, A. Fang, W. Guo, S. Fridovich-Keil, J. Ragan-Kelley, L. Schmidt, and B. Recht, Neural kernels without tangents, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020) pp. 8614–8623.arxiv.org/abs/2003.02237









