betaFeedback

WorksComputer Science

Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks

Abstract

Federated Learning enables collaborative model training across decentralized clients without sharing raw data. While the field has seen rapid growth, the datasets underpinning FL evaluation have not kept pace: real federated datasets remain scarce, and most studies rely on small-scale centralized classification datasets partitioned via Dirichlet label skew, a one-dimensional proxy for the multi-dimensional heterogeneity present in real federations. Moreover, label-based partitioning is by construction inapplicable to non-classification tasks such as segmentation and detection, for which no principled simulation method exists. We cast the conversion of centralized datasets into federated benchmarks as the task of Federated Dataset Simulation (FDS), and decompose it into a representation function ΦΦ and an assignment mechanism ππ. We provide two label-free instantiations – geometric partitioning via pretrained model embeddings and semantic partitioning via vision-language model-derived criteria – applicable to classification and dense prediction tasks alike. We evaluate across four datasets spanning classification, segmentation, and detection, and show that both methods induce heterogeneity that matches or exceeds that of label-skew partitioning, with measurable impact on downstream FL performance. We release fds, a modular open-source package for end-to-end federated dataset simulation, heterogeneity measurement, and FL training, at https://github.com/siomvas/fds.

Introduction

Federated learning (FL) [36] is a collaborative training paradigm that has matured into a central solution for privacy-preserving and decentralized ML, with production deployments in several industries [27]. Despite an exponential growth in research output, a well-documented gap persists in the scale, complexity, and diversity of the datasets and tasks underpinning FL evaluation [37, 23]: the vast majority of empirical studies are conducted on image classification at the scale and complexity of CIFAR [26]. While naturally federated datasets exist [7, 38, 56], expanding this dataset pool is laborious and costly, and the heterogeneity such datasets exhibit is often idiosyncratic and not transferable to other tasks or domains. It has therefore become standard practice to partition existing centralized datasets into virtual clients for FL evaluation [46]. For a virtual partition to provide a challenging FL evaluation, the consensus is it must reflect a central tenet of FL theory, namely that federated optimization performance degrades as the statistical heterogeneity (non-IIDness) of client data increases [36, 50].

This creates a need for methods simulating non-IIDness across the virtual clients of a split centralized dataset. The prevailing approach to do so is Dirichlet-based label partitioning [46], which assigns skewed class proportions across clients by sampling from a Dirichlet distribution [22]. This reduces the multi-dimensional heterogeneity (covariate shift, concept shift, domain shift) observed in real federations to a single axis, when in practice clients may differ not only in which classes they observe but in how those classes appear [24]. Moreover, Dirichlet partitioning operates on sample-level class labels, so is by construction inapplicable to any task where samples are not uniquely associated with a single category (e.g., segmentation, machine translation, and sound event detection). For these tasks, which constitute a growing share of practical FL applications, no principled heterogeneity simulation method exists, and practitioners are limited to random partitioning or ad-hoc splits [23]. As a result, current FL evaluations and benchmarks produce conclusions about FL research progress (in performance, convergence rates, fairness, and robustness) that may not generalize to real deployments.

We cast this problem as the task of Federated Dataset Simulation (FDS): converting a centralized dataset into a federated benchmark in a controllable manner, making heterogeneity simulation itself an object of systematic design and measurement. To enable simulation that is agnostic to the downstream task and label format, we decompose FDS into two sub-tasks: a representation function Φ\Phi, which maps data into a structured latent space that captures axes of variation related to heterogeneity, and an assignment mechanism π\pi, which allocates samples to virtual clients within that space. This Φ/π\Phi/\pi decomposition provides a unifying formalism under which existing approaches, like label Dirichlet partitioning, can be expressed as special cases. Because Φ\Phi operates on raw inputs rather than annotations, the framework is naturally label-free and modality-agnostic. We provide and evaluate two concrete instantiations of this framework in the vision domain, each targeting a different latent space. The first, geometric partitioning, defines Φgeo\Phi_{\mathrm{geo}} as the embedding function of a self-supervised pretrained foundation model and uses kk-means clustering as the assignment π\pi. Because such encoders are trained to organize semantically related inputs into coherent regions of their representation space, partitioning along geometric boundaries in this space induces covariate shift that reflects meaningful visual and semantic structure. The second, semantic partitioning, defines Φsem\Phi_{\mathrm{sem}} as a chain of pretrained vision-language and large language models which extracts per-sample sets of human-interpretable semantic criterion labels. A Dirichlet-weighted assignment π\pi then correlates client identity with specific criterion values to produce federations where each client can be characterized by the visual properties they tend to exhibit.

We validate the framework across four datasets encompassing classification, semantic segmentation, and object detection, and benchmark FL performance under a varying number of clients. We demonstrate that both geometric and semantic partitioning produce distinct heterogeneity profiles that surpass existing approaches, not only in terms of feature and label (where applicable) heterogeneity, but also on FL convergence. Our contributions are as follows:

  1. Task formulation. We formalize federated dataset simulation through the Φ/π\Phi/\pi decomposition, establishing a unified framework in which heterogeneity type, severity, and measurement are jointly specified and become the object of study and design.

  1. Partitioning methods. We introduce two distinct label-free partitioning pipelines that induce feature-level heterogeneity and apply to classification and dense prediction tasks alike, resulting in similar or greater heterogeneity than Dirichlet partitioning despite using no labels.

  1. Empirical evaluation and benchmark. We produce a multi-faceted evaluation spanning label and feature space heterogeneity along with FL performance and validate the framework across four challenging vision datasets, showing that the proposed partitioning methods result in a more challenging setting that can serve as a better benchmark for FL algorithm development.

  1. Software package. With the intention of lowering the barrier for researchers to move beyond label-skew-only evaluation and classification, we release fds, a modular open-source library providing an end-to-end pipeline for federated dataset simulation: feature extraction across multiple pretrained encoders, semantic clustering based on our Φsem\Phi_{\mathrm{sem}} VLM-LLM pipeline, controllable partitioning to clients, heterogeneity measurement in the feature or label space, and integrated downstream FL training for classification, segmentation, or detection.

The FDS Framework

General Problem Formulation

Definition 1 (Federated Dataset Simulation). Let D={(xi,yi)}i=1N\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N} denote a centralized dataset comprising NN samples, where xi∈Xx_i \in\mathcal{X} and yi∈Yy_i \in\mathcal{Y} represent the input and annotation space, respectively. The Federated Dataset Simulation (FDS) task seeks a partition {Dk}k=1K\{\mathcal{D}_k\}_{k=1}^{K} such that: ⋃kDk=D\bigcup_k \mathcal{D}_k = \mathcal{D} (all samples are assigned), Dk∩Dℓ≠k=∅\mathcal{D}_k \cap\mathcal{D}_{\ell\ne k} = \emptyset (samples are uniquely assigned), and the induced local distributions Pk(x,y)P_k(x,y) exhibit meaningful statistical heterogeneity.

Definition 2 (Partitioning Mechanism). Given D\mathcal{D}, a partitioning mechanism consists of:

  1. A representation function Φ:X×Y→Z\Phi: \mathcal{X} \times\mathcal{Y} \to\mathcal{Z}, which projects inputs into a structured latent space Z\mathcal{Z}.

  2. An assignment mechanism π\pi which produces a partitioning {Dk}k=1K\{\mathcal{D}_k\}_{k=1}^{K} from the set of latent representations {zi=Φ(xi)}i=1N\{z_i = \Phi(x_i)\}_{i=1}^{N}.

The assignment mechanism operates via a set of client parameters ω={ωk}k=1K\omega= \{\omega_k\}_{k=1}^{K}, which may be computed from the data, sampled from a prior, or both, and then allocates samples to clients based on the affinity between each sample’s representation and the client parameters. Label-skew Dirichlet partitioning [22] fits this framework as the special case where Φ(x)=y\Phi(x) = y, ωk=qk∼Dir⁡(αp)\omega_k = q_k \sim\operatorname{Dir}(\alpha p) and samples of each class are allocated to clients in proportion to {qk,y}k=1K\{q_{k,y}\}_{k=1}^{K}. The concentration parameter α\alpha controls heterogeneity from IID (α→∞\alpha\to\infty) to near class-disjoint (α→0\alpha\to0).

Unlike the Dirichlet special case, the two instantiations we propose next operate on Φ(x)\Phi(x) alone, making them applicable to settings where sample-level labels are unavailable, noisy, or dense (e.g., segmentation and detection tasks). This also extends the space of heterogeneity that can be expressed: because the label space Y\mathcal{Y} is a lossy projection of X\mathcal{X}, feature-based representations can capture distributional structure that label-based partitioning cannot.

Definition 3 (Heterogeneity Functional). Given a realized partition, we quantify heterogeneity via the mean pairwise divergence of client distributions in the representation space:

H(π,Φ)=1(K2)∑k<ℓDΦ(Pk,Pℓ)\mathcal{H}(\pi, \Phi) = \frac{1}{\binom{K}{2}} \sum_{k<\ell} D_{\Phi}(P_k, P_\ell)

where PkP_k denotes the empirical distribution of Φ(x)\Phi(x) for samples assigned to client kk, and DΦD_{\Phi} is a divergence measure appropriate to Z\mathcal{Z}.

Label Distribution Heterogeneity When sample-level class labels are available, we examine how client label distributions diverge. We measure this via the Jensen–Shannon divergence, a symmetric, bounded measure well-suited to discrete distributions:

JSD⁡(Pky∥Pℓy)=12DKL⁡(Pky∥M)+12DKL⁡(Pℓy∥M),M=(Pky+Pℓy)/2\operatorname{JSD}(P_k^y \| P_\ell^y) = \frac{1}{2}D_{\operatorname{KL}}(P_k^y \| M) + \frac{1}{2}D_{\operatorname{KL}}(P_\ell^y \| M), \quad M = (P_k^y + P_\ell^y) / 2

Feature-distribution heterogeneity To measure heterogeneity in a continuous representation space, we use the Maximum Mean Discrepancy (MMD) [17] with an RBF kernel and the Frechet Inception Distance (FID) [21], computed in the activation space of a frozen InceptionV3 network [47] independent of the embedding encoder used for partitioning, avoiding the circularity of evaluating partitions in the same embedding space used to create them:

MMD⁡2=∥μPk−μPℓ∥F2, where μP is the kernel mean embedding of P\operatorname{MMD}^{2} = \lVert\mu_{P_k} - \mu_{P_\ell} \rVert_{\mathcal{F}}^{2}, \text{ where } \mu_{P} \text{ is the kernel mean embedding of } P
FID⁡=∥μk−μℓ∥2+Tr⁡(Σk+Σℓ−2(ΣkΣℓ)1/2)\operatorname{FID} = \lVert\mu_k - \mu_\ell\rVert^{2} + \operatorname{Tr}(\Sigma_k + \Sigma_\ell- 2(\Sigma_k\Sigma_\ell)^{1/2})

Geometric Partitioning

This instantiation operates in the continuous feature space induced by pretrained foundation models. Let fθ:X→Rdf_{\theta} : \mathcal{X} \to\mathbb{R}^{d} be a model pretrained with a self-supervised or vision-language objective. We define Φgeo(x)\Phi_{\mathrm{geo}}(x) as fθ(x)∈Rdf_{\theta}(x) \in\mathbb{R}^{d}. The assignment mechanism partitions the embedding space via kk-means clustering with KK clusters equal to the number of target clients. The client parameters are the cluster centroids ω={μk}k=1K\boldsymbol{\omega} = \{\mu_k\}_{k=1}^{K}, computed from the full set of representations, and each sample is assigned to its nearest centroid ki=arg⁡min⁡k∈[K]∥Φgeo(xi)−μk∥2k_i = \arg\min_{k \in[K]} \lVert\Phi_{\mathrm{geo}}(x_i) - \mu_k\rVert_2.

Modern pretrained representations, whether trained via self-distillation [39] or vision-language alignment [43], organize semantically related inputs into coherent regions of the embedding space, so partitioning along geometric boundaries in this space induces covariate shift that reflects the visual and semantic structure captured by the encoder. The resulting clients specialize in different visual domains, object types, or scene compositions, depending on which axes of variation are salient in the encoder’s representation space, without requiring access to annotations. The number of clients KK acts as an implicit heterogeneity knob, where fewer clients produce broader clusters spanning more diverse manifold regions, while more clients yield finer partitions with greater inter-client divergence. We evaluate multiple encoder families and token aggregation strategies in Section 3 and report the selection procedure used to identify a recommended default configuration.

VLM-based Semantic Partitioning

The second instantiation operates over a discrete criterion space Zsem=L1×L2×⋯×LC\mathcal{Z}_{\mathrm{sem}} = \mathcal{L}_1 \times\mathcal{L}_2 \times\cdots\times\mathcal{L}_C, where each Lc\mathcal{L}_c is a finite set of categorical values corresponding to a human-interpretable semantic criterion (e.g., cuisine origin or texture for a food photography dataset). The representation function Φsem:X→Zsem\Phi_{\mathrm{sem}} : \mathcal{X} \to\mathcal{Z}_{\mathrm{sem}} maps each image to a tuple of criterion labels Φsem(xi)=(ℓ1(xi),…,ℓC(xi))\Phi_{\mathrm{sem}}(x_i) = (\ell_1(x_i), \ldots, \ell_C(x_i)), and the assignment mechanism π\pi allocates samples to clients based on these discrete representations using a product-of-Dirichlets mechanism.

Criterion label extraction Φsem\Phi_{\mathrm{sem}} is a multi-stage pipeline chaining a vision-language model (VLM) and a large language model (LLM) to perform text-conditioned image clustering. For each dataset, semantically meaningful axes of variation are identified through domain inspection, and candidate criteria are validated by querying the VLM on a sample subset to verify non-degenerate label distributions. After CC is established, per-sample labeling proceeds as an iterative procedure, inspired by Kwon et al. [28]: the VLM is prompted to produce a short description of each image focusing exclusively on criterion cc. An LLM converts each description into a single-word raw label summarizing the image with respect to cc to form a dictionary. The LLM then clusters this dictionary of raw labels into ∣Lc∣|\mathcal{L}_c| canonical categories to consolidate semantically close concepts. Each sample description is then classified into one of these ∣Lc∣|\mathcal{L}_c| categories, yielding the criterion label ℓc(xi)∈Lc\ell_c(x_i) \in\mathcal{L}_c, using the same LLM. This is repeated for each criterion to construct Φsem(xi)\Phi_{\mathrm{sem}}(x_i).

Assignment mechanism. Given the extracted criterion vectors, samples are assigned to clients via a product-of-Dirichlets scheme. For each client mm and each criterion cc, a preference vector pm,c∼Dir⁡(α⋅1∣Lc∣)\mathbf{p}_{m,c} \sim\operatorname{Dir}(\alpha\cdot\mathbf{1}_{|\mathcal{L}_c|}) is drawn. The assignment probability of sample ii to client mm is πm∝∏c=1Cpm,c[ℓc(xi)]\pi_m \propto\prod_{c=1}^{C} p_{m,c}[\ell_c(x_i)], where pm,c[ℓ]p_{m,c}[\ell] denotes the preference weight that client mm places on value ℓ\ell of criterion cc. This multiplicative composition ensures that a sample receives high affinity for a client only if it matches that client’s preferences across all criteria simultaneously. This mechanism generalizes the standard Dirichlet label-skew assignment to the multi-label setting. The concentration α\alpha serves as the heterogeneity knob, like in the typical label-skew case. The full set of criteria and prompt details used per dataset are reported in Appendix C and included in the code release. As this is a computationally expensive pipeline, our code release also includes the precomputed criterion labels for all 4 evaluated datasets.

Connection to Federated Learning Performance

The previous sections define methods for inducing distributional divergence across virtual clients, but not all such divergence is equally consequential for federated optimization. A partition that separates clients along axes to which the model can easily learn invariant features may produce large H\mathcal{H} while having negligible impact on model convergence. The optimization difficulty induced by a partition depends not only on the magnitude of H\mathcal{H} but on the mutual information I(Z;Y)I(\mathcal{Z};Y) between the representation and the task: when I(Z;Y)≈0I(\mathcal{Z};Y) \approx0 heterogeneity in Z\mathcal{Z} does not translate into optimization difficulty as opposed to when I(Z;Y)I(\mathcal{Z};Y) is high. This clarifies why ad-hoc heterogeneity methods such as additive Gaussian noise [30] or geometric rotations [14] do not typically produce meaningful FL difficulty especially if the training pipeline includes augmentations for such perturbations. To effectively benchmark FL algorithms across different tasks, client configurations, and optimization strategies, we therefore need partitioning methods that produce heterogeneity with high I(Z;Y)I(Z; Y).

Table 1: Heterogeneity metrics on classification datasets across client counts K∈{5,10,20}K \in\{5, 10, 20\}. JSD measures label-distribution divergence; MMD, FID, and centroid distance measure feature-distribution divergence (computed in InceptionV3 activation space). Higher values indicate greater heterogeneity. Geometric partitioning uses no labels yet induces the highest heterogeneity across all metrics, including JSD.

DatasetMethodJSD ↑\uparrow 51020MMD (×10−2\times10^{-2}) ↑\uparrow 51020FID ↑\uparrow 51020Centroid ↑\uparrow 51020
TinyRandom.003.005.0100.000.000.003.36.613.20.90.90.9
TinyLabel Dir. α=0.1\alpha=0.1.559.572.5690.400.550.7529.239.049.44.96.07.0
TinySemantic α=1.0\alpha=1.0.128.168.2070.700.931.1922.030.943.04.85.56.2
TinySemantic α=0.1\alpha=0.1.152.240.3150.581.332.1523.242.162.24.86.78.7
TinyGeometric.630.655.6685.319.7018.011114620113.821.631.1
FoodRandom.002.003.0070.000.000.002.75.410.50.80.91.0
FoodLabel Dir. α=0.1\alpha=0.1.564.587.5660.691.061.0616.524.629.98.39.911.1
FoodSemantic α=1.0\alpha=1.0.138.168.2150.670.751.1012.516.325.78.08.29.7
FoodSemantic α=0.1\alpha=0.1.213.284.3590.611.502.4714.325.339.88.311.214.3
FoodGeometric.620.660.6646.5816.413.874.012712227.039.040.5

Table 1.

Table 2: Heterogeneity metrics on dense prediction datasets. Dirichlet label-skew partitioning is inapplicable to segmentation and detection, so Random is the only existing baseline. Geometric and semantic partitioning are the first methods to produce non-trivial heterogeneity for these tasks.

DatasetMethodMMD (×10−2\times10^{-2}) ↑\uparrow 51020FID ↑\uparrow 51020Centroid ↑\uparrow 51020
ADE20KRandom0.000.000.0111.121.841.30.91.31.8
ADE20KSemantic α=1.0\alpha=1.02.132.272.1647.160.880.56.46.87.0
ADE20KSemantic α=0.1\alpha=0.12.804.565.1059.492.6124.68.110.111.0
ADE20KGeometric13.116.519.016719124720.725.631.9
Pascal VOCRandom0.010.020.0252.387.4129.62.02.83.9
Pascal VOCSemantic α=1.0\alpha=1.00.681.211.2968.6112.2152.15.16.77.6
Pascal VOCSemantic α=0.1\alpha=0.11.091.522.1679.8120.5170.66.37.59.4
Pascal VOCGeometric18.116.730.121922826722.023.629.7

Table 2.

Both proposed instantiations are designed to meet this requirement. Φgeo\Phi_{\mathrm{geo}} is extracted from encoders whose pretraining objectives aim to preserve semantic content, producing representations with high I(Z;Y)I(Z; Y) across a wide range of vision tasks [4]. Partitioning in this space induces distributional shifts that the model is likely to be sensitive to, which we verify empirically in Section 3. For Φsem\Phi_{\mathrm{sem}}, task-coupling depends on the choice of criteria, with criteria that describe properties correlated with the target variable (e.g., main object type, scene composition) producing higher I(Zsem;Y)I(Z_{\mathrm{sem}}; Y) than criteria more orthogonal to the task, as we show via criterion-level ablation in Section 3. This reflects an asymmetry between the two instantiations: geometric partitioning derives task-relevance implicitly from the pretraining objective, while semantic partitioning requires the explicit design of task-relevant criteria but offers interpretability, as the axes of heterogeneity are human-readable.

Evaluation

Evaluation Design

Encoder and model selection. Our geometric partitioning method depends on the choice of representation function Φgeo\Phi_{\mathrm{geo}}. We therefore evaluate several pretrained vision encoders, including DINOv2 [39], CLIP [43], and MAE [20] variants across several token extraction and aggregation strategies, and select a single default configuration based on average rank of MMD across datasets. Unless otherwise stated, we use DINOv2-B features obtained by concatenating the CLS tokens from the final four layers, followed by UMAP [35] dimensionality reduction before clustering. This choice provided a strong and computationally tractable default across classification, segmentation, and detection tasks. For semantic partitioning, we use the open-source Intern-VL3.5 [51] and Llama 3.3 70B [16] models as the VLM and LLM components, respectively, in order to keep the pipeline reproducible and accessible under a modest compute budget; all extracted criterion labels are cached and reused across experiments.

Datasets and tasks. We evaluate on four vision datasets spanning three prediction settings. Tiny-ImageNet [29] provides a more visually complex classification benchmark than CIFAR-scale datasets while remaining comparable to prior FL work. Food101 [6] introduces a fine-grained, domain-specific classification task with substantial intra-class variation and label noise. To test whether FDS extends beyond sample-level class labels, we further include ADE20K [58] for semantic segmentation and PASCAL VOC [12] for object detection. These dense prediction tasks are important for our setting because standard label Dirichlet partitioning is not naturally defined when each sample contains multiple labels, spatially structured labels, or object sets rather than a single class.

FL training protocol. For each dataset, we compare random partitioning, label Dirichlet partitioning where applicable, geometric partitioning, and semantic partitioning. We vary the number of clients K∈{5,10,20}K \in\{5, 10, 20\} to study how each partitioning mechanism scales with federation size. All FL experiments use FedAvg [36] with 1 local epoch for 100 communication rounds. We use a ResNet-18 [19], DeepLabv3 with ResNet-50 [8] for semantic segmentation, and a Faster R-CNN model with a ResNet-50-FPN backbone [32] for detection. All models are initialized from pretrained weights. We tune dataset-specific optimization hyperparameters on a held-out subset using Optuna [2] and then fix them across partitioning methods.

Metrics. We evaluate each partition along three axes. We measure feature-space heterogeneity using the MMD, FID, and feature centroid Euclidean distance averaged over pairwise distances between client distributions in the encoder representation space. Where sample-level labels are available, we report label-space heterogeneity using Jensen–Shannon divergence between client label distributions. Downstream FL performance is evaluated using task-appropriate metrics: top-1 accuracy for Tiny ImageNet and Food101, mIoU for ADE20K, and mAP@50 for PASCAL VOC. This separates the question of whether a partition induces measurable distributional heterogeneity from whether that heterogeneity produces optimization difficulty for FL.

Main Results

Induced Heterogeneity Tables 1 and 2 show that the proposed FDS instantiations induce feature-space heterogeneity across tasks. On classification datasets, label Dirichlet partitioning strongly increases label-distribution divergence but has comparatively modest effect in feature space. In contrast, geometric partitioning, despite using no labels, produces the largest feature-space divergence and also induces high label divergence as a byproduct of partitioning semantically structured representations. Semantic partitioning occupies an intermediate regime: it produces non-random feature heterogeneity with an interpretable severity parameter, and the α=0.1\alpha= 0.1 setting consistently increases heterogeneity relative to α=1.0\alpha= 1.0. On dense prediction datasets both geometric and semantic partitioning produce non-trivial feature heterogeneity relative to random splits, supporting the use of the Φ/π\Phi/\pi framework beyond classification.

Impact on downstream FL performance. The heterogeneity induced by FDS translates into measurable downstream optimization difficulty. Table 3 reports global validation performance under FedAvg for K∈{5,10,20}K \in\{5, 10, 20\} clients. Across all datasets and client counts, non-random partitions degrade FedAvg performance relative to random splitting, indicating that the induced client differences are not merely artifacts of the heterogeneity metrics. Geometric partitioning is the most challenging setting in most configurations, producing the largest degradation on all Tiny ImageNet, ADE20K, and PASCAL VOC settings, and on Food101 for K=5K = 5 and K=10K = 10. Semantic partitioning is generally less severe but remains consistently harder than random, while preserving an interpretable α\alpha-controlled assignment mechanism. On classification datasets, label-Dirichlet skew remains a strong baseline at α=0.1\alpha= 0.1; however, on dense prediction tasks it is not directly applicable, whereas both proposed FDS instantiations continue to induce non-trivial performance degradation.

DatasetMetricPartitionK=5K = 5K=10K = 10K=20K = 20
Tiny ImageNetAcc.Random74.53 (0.00)74.63 (0.00)74.29 (0.00)
Label Dir. α=0.1\alpha= 0.171.74 (2.79)71.56 (3.07)71.74 (2.55)
Label Dir. α=1.0\alpha= 1.074.30 (0.23)74.19 (0.44)73.57 (0.72)
Semantic α=0.1\alpha= 0.173.69 (0.84)70.43 (4.20)72.99 (1.30)
Semantic α=1.0\alpha= 1.073.56 (0.97)71.32 (3.31)72.31 (1.98)
Geometric67.41 (7.12)64.84 (9.79)69.24 (5.05)
Food101Acc.Random77.16 (0.00)74.43 (0.00)70.39 (0.00)
Label Dir. α=0.1\alpha= 0.171.31 (5.85)66.45 (7.98)64.20 (6.19)
Label Dir. α=1.0\alpha= 1.075.75 (1.41)72.89 (1.54)68.80 (1.59)
Semantic α=0.1\alpha= 0.172.93 (4.23)64.49 (9.94)63.97 (6.42)
Semantic α=1.0\alpha= 1.074.93 (2.23)71.66 (2.77)65.68 (4.71)
Geometric70.02 (7.14)63.15 (11.28)66.34 (4.05)
ADE20KmIoURandom40.23 (0.00)39.02 (0.00)36.91 (0.00)
Semantic α=0.1\alpha= 0.138.13 (2.10)36.74 (2.28)33.46 (3.45)
Semantic α=1.0\alpha= 1.038.98 (1.25)37.82 (1.20)34.01 (2.90)
Geometric35.37 (4.86)31.14 (7.88)29.34 (7.57)
PASCAL VOCmAP@50Random79.40 (0.00)83.10 (0.00)82.14 (0.00)
Semantic α=0.1\alpha= 0.178.15 (1.25)77.56 (5.54)77.99 (4.15)
Semantic α=1.0\alpha= 1.078.12 (1.28)78.12 (4.98)79.22 (2.92)
Geometric74.34 (5.06)73.19 (9.91)78.10 (4.04)

Table 3. Downstream FL performance under FedAvg across datasets and client counts. Parentheses report degradation relative to random partitioning. Lower non-random performance within each dataset/client-count block in bold. Label Dirichlet baselines are omitted for dense prediction tasks because they do not provide a single sample-level class label.

Ranking of Aggregation Methods

We compare seven aggregation methods, FedAvg [36], FedDyn [11], FedNova [49], FedProx [31], FedSAM [42], SCAFFOLD [25], and FedLC [57], under label Dirichlet skew and semantic partitioning at α=0.1\alpha= 0.1 and under geometric partitioning, on Tiny ImageNet with 10 clients and a ResNet-18 trained from random initialization or ImageNet weights (Table 4, settings in Appendix § 8). Geometric partitioning is the hardest protocol for every method, and it also orders the methods differently.

MethodRandom initialization: Label Dir.Random initialization: GeometricRandom initialization: SemanticPretrained weights: Label Dir.Pretrained weights: GeometricPretrained weights: Semantic
FedAvg32.97±\pm0.09 (3)24.89±\pm0.22 (4)35.14±\pm0.35 (2)54.73±\pm0.23 (3)42.11±\pm0.16 (4)55.97±\pm0.34 (5)
FedDyn30.92±\pm0.13 (6)25.11±\pm0.04 (2)33.91±\pm0.08 (5)53.06±\pm0.10 (7)36.41±\pm0.33 (6)56.73±\pm0.02 (3)
FedNova32.91±\pm0.36 (4)20.03±\pm7.69 (7)33.76±\pm0.06 (6)54.22±\pm0.05 (4)26.32±\pm1.86 (7)55.72±\pm0.39 (7)
FedProx33.07±\pm0.09 (2)24.92±\pm0.24 (3)34.98±\pm0.09 (4)54.17±\pm0.12 (5)42.95±\pm0.08 (2)55.98±\pm0.08 (4)
FedSAM30.16±\pm0.18 (7)25.97±\pm0.10 (1)32.66±\pm0.26 (7)55.45±\pm0.03 (2)45.66±\pm0.08 (1)57.27±\pm0.01 (2)
SCAFFOLD33.16±\pm0.10 (1)24.50±\pm3.57 (6)35.94±\pm0.15 (1)55.63±\pm0.45 (1)40.89±\pm2.56 (5)58.37±\pm0.18 (1)
FedLC32.27±\pm0.16 (5)24.80±\pm0.30 (5)35.09±\pm0.17 (3)53.35±\pm0.13 (6)42.42±\pm0.14 (3)55.84±\pm0.37 (6)
Spread3.005.943.282.5719.342.65

Table 4. Accuracy (%) of seven aggregation methods on Tiny ImageNet with K=10K = 10, from random and from ImageNet initializations. Settings differ from Table 3 (Appendix § 8). Mean ±\pm standard deviation over three seeds, rank in parentheses. Geometric partitioning is the most challenging protocol for every method.

Semantic partitioning preserves much of the ranking that label skew produces, with Spearman correlations of 0.750 from random initialization and 0.464 from pretrained weights, whereas the ranking under geometric partitioning correlates with it at −0.571 and 0.179. FedSAM, last of the seven under label skew from random initialization, is first under geometric partitioning from either starting point, while FedNova falls to last. From pretrained weights the seven lie within 2.57 points under label skew, each within a point of its nearest neighbor, whereas under geometric partitioning they spread over 19.34 points, or 9.25 without FedNova, and SCAFFOLD, which edges FedSAM by 0.18 points under label skew, trails it by 4.77. From random initialization the spread widens from 3.00 to 5.94 points only because of FedNova, without which it narrows to 1.47.

Comparison to Naturally Federated Datasets

We next ask how the magnitude of heterogeneity induced by representation-space partitioning compares against that naturally present in some naturally multi-centric/federated datasets. To do this we compare the feature space and label space heterogeneity induced by Φgeo\Phi_{\mathrm{geo}} in naturally federated datasets with the heterogeneity originally present, as measured by JSD, MMD, and kk-NN consistency, which is the average fraction of nearest neighbors in feature space belonging to the same client, measuring local clustering structure in feature space [11]. We choose Fed-ISIC2019, a skin lesion classification dataset with 6 clients [38], FEMNIST, a handwriting dataset of digits and letters separated by writer with 3,550 users [7], and Office-Caltech10 [15], a domain generalization dataset of 4 distinct domains (Amazon, Caltech, DSLR, and Webcam) with one client per domain, and compare their original client partitions against geometric partitioning under Φgeo\Phi_{\mathrm{geo}} with the same number of clients. Table 5 shows that geometric partitioning often matches or exceeds the natural client split in feature space. On FEMNIST and Office-Caltech10, geometric partitioning produces substantially larger MMD and kk-NN client consistency than the natural partition. On Fed-ISIC2019, the natural split exhibits higher label-space divergence, while geometric partitioning produces larger feature-space separation. This distinction is important: real federations can be heterogeneous for different reasons, and the proposed framework makes the choice of representation explicit rather than reducing all heterogeneity to label imbalance.

DatasetPartitionLabel JSDFeature MMDkk-NN consistency
Fed-ISIC2019Natural0.1150.0270.260
Geometric0.0280.1920.530
Office-Caltech10Natural0.0100.0160.446
Geometric0.6840.3340.982
FEMNISTNatural0.1300.0430.127
Geometric0.6430.4650.654

Table 5. Natural versus simulated heterogeneity. For each naturally federated dataset, we compare the original client split to a geometric partition with the same number of clients. Higher values indicate greater heterogeneity.

Ablating the Assignment Mechanism in Semantic Partitioning

To validate that the VLM-extracted criterion labels capture meaningful axes of variation and to examine which criteria contribute most to heterogeneity, we conduct a pure-criterion ablation. For each criterion cc and dataset, we create ∣Lc∣|L_c| clients by deterministically assigning all samples sharing the same criterion value to the same client, bypassing the stochastic Dirichlet assignment entirely. This represents the maximum heterogeneity achievable along a single semantic axis and isolates the contribution of each criterion independently of the assignment mechanism. Table 6 (Appendix § 10) reports feature-space heterogeneity under these pure-criterion splits across all four datasets. A consistent hierarchy emerges: criteria describing what appears in an image (main object, scene, cuisine) produce the largest distributional divergence, while criteria describing how it appears (lighting, color, size) produce substantially less. This ordering is consistent across metrics and aligns with the intuition that content-level properties carry more mutual information with the task label than stylistic ones, as discussed in § 2.4. To test whether the feature-space hierarchy translates into downstream optimization difficulty, we train FedAvg on Tiny ImageNet under each pure-criterion split. The ranking is preserved exactly: main_object, the strongest criterion in feature space, yields the lowest accuracy (67.58%), a degradation comparable to geometric partitioning at K=5K=5 (67.41%, Table 3), while time, the weakest, degrades accuracy by only 2.47 points to 72.06%. MMD and accuracy degradation across the five criteria are perfectly correlated, consistent with criteria with higher I(Zsem;Y)I(Z_{\mathrm{sem};Y}) producing more consequential heterogeneity (§ 2.4). Validating the pipeline. On 10,531 Clevr-4 images [48], whose shape, texture, color, and count taxonomies are equally valid, the label of each criterion agrees with its own taxonomy far more than with the others, with NMI from 0.297 for count to 0.860 for shape, against 0.012 on average off the diagonal. Our Food101 cuisine labels reach an AMI of 0.503 with the human labels of Food-4c [33] (Appendix § 12).

DatasetCriterionClientsFID ↑MMD ↑kkNN sep. ↑Centroid ↑
Tiny ImageNetmain object7115.30.03680.84013.7
location573.20.02370.78110.9
size460.20.01600.6218.6
color741.30.00920.4827.0
time429.80.00730.5836.2
ADE20Kscene4122.10.03960.85314.2
main object5118.40.02750.76111.6
camera5111.20.02590.56711.0
color7165.00.01970.63210.4
size447.70.00930.5386.9
light317.80.00280.4653.8
PASCAL VOCmain object6228.50.03580.85114.1
location397.80.01820.7299.8
size490.00.01310.5158.0
color689.10.00690.3296.3
light457.90.00350.3394.5
Food101cuisine534.70.06160.76416.5
preparation542.40.05840.74016.1
composition434.50.02740.58211.3
texture422.50.02310.62910.0
formality316.90.02120.7339.8

Table 6. Pure-criterion ablation for semantic partitioning. For each criterion, one client is created per criterion value and all samples with that value are assigned to the corresponding client. Higher values indicate stronger feature-space separation.

What Geometric Partitioning Separates

Geometric partitioning requires no criteria, leaving open what its clients differ in. Against the pure-criterion partitions, they follow content far more than appearance, with a mean AMI at K=10K = 10 of 0.377 for main object, scene, and cuisine against 0.060 for lighting, color, camera, and size (Appendix § 13).

The content that a partition follows is set by the representation. Removing the top pp principal components before clustering shifts the partition among the properties that the encoder captures (Figure 1). For DINOv2 it raises the NMI with texture from 0.035 at p=0p = 0 to 0.428 at p=10p = 10 while that with shape falls from 0.928 to 0.186, but leaves color below 0.012, and for MAE it raises the NMI with color from 0.073 to 0.496 at p=5p = 5, far beyond the variation across kk-means seeds. Encoder pretraining thus fixes what a representation can express while the projection fixes what it does express, and pp controls the axis of heterogeneity as α\alpha controls its severity.

NMI between $k$-means partitions for DINOv2 and MAE embeddings after removing principal components

Figure 1. NMI between kk-means partitions (K=10K = 10) of Clevr-4 and each of its taxonomies after removing the top pp principal components of DINOv2 (left) or MAE (right) embeddings. The kk-means seed standard deviation at p=0p = 0 is at most 0.021 (DINOv2, three seeds). No partition collapses, since every client holds at least 1.2% of the images, and the coefficient of variation of client sizes ranges from 0.22 to 0.37 for MAE and from 0.27 to 0.61 for DINOv2.

Discussion

Feature-space and label-space heterogeneity are not interchangeable. Label Dirichlet partitioning at α=0.1\alpha=0.1 produces high label divergence on classification datasets but comparatively modest feature-space divergence: on Tiny ImageNet with K=10K=10, it achieves JSD 0.572 but only MMD 0.55×10−20.55 \times10^{-2}, whereas geometric partitioning reaches JSD 0.655 alongside MMD 9.70×10−29.70 \times10^{-2}. This indicates that label-skew partitions create clients that differ in which classes they hold but not in how they look, limiting the diversity of optimization challenges they present. Evaluation conducted solely under label skew therefore exercises a single axis of heterogeneity, and § 3.3 shows that heterogeneity along a different axis can change which aggregation method appears best.

Not all heterogeneity is equally consequential for federated optimization. The criterion ablation (§ 3.5) shows that the relationship between distributional divergence and FL difficulty is mediated by the mutual information between the heterogeneity axis and the task, the main_object criterion on Tiny ImageNet costing 6.95 points of accuracy against the random baseline while time costs 2.47. Effective heterogeneity simulation therefore requires partitioning along task-relevant axes, not merely maximizing distributional distance.

Dense prediction tasks are underserved by current simulation practice. On ADE20K and PASCAL VOC, where label Dirichlet partitioning is not directly applicable, random splitting has been the only available baseline. Random splits leave these tasks untested under heterogeneity, and geometric partitioning degrades mIoU on ADE20K by up to 7.88 points and mAP@50 on PASCAL VOC by up to 9.91 points relative to random. Given the growing importance of segmentation and detection in deployed FL systems, this gap represents a blind spot in current evaluation methodology that the Φ/π\Phi/\pi framework directly addresses.

Geometric and semantic partitioning occupy complementary roles. Geometric partitioning produces the most challenging FL setting in most configurations and requires no design decisions beyond encoder selection, making it a strong default for stress-testing FL algorithms. Semantic partitioning trades peak heterogeneity for interpretability: the α\alpha parameter provides continuous control, and the criterion-level ablation allows practitioners to understand and select the axes along which their federation is heterogeneous. The choice between the two depends on whether the goal is maximal difficulty or controlled, explainable variation. Semantic partitioning thus extends the prevailing protocol to tasks it cannot reach, while geometric partitioning creates a regime in which aggregation methods rank differently than under label skew (Appendix § 14 gives practical guidance).

Federated dataset simulation is contemporaneous with FL itself, as the original FedAvg experiments constructed virtual non-IID clients by assigning each client a number of label shards [36]. Subsequent work by Hsu et al. [22] systematized and popularized the Dirichlet label-skew allocation first introduced in Yurochkin et al. [55], constructing a tunable protocol where the Dirichlet concentration α\alpha acts as a continuous control on client heterogeneity. While this remains the dominant synthetic partitioning protocol in empirical FL [46, 23], it is limited by construction to representing client heterogeneity through variations in label marginals and therefore assumes single, sample-level class labels. Attempts to simulate feature heterogeneity instead have largely remained at the level of input perturbations, including client-specific Gaussian noise [30], image rotations [14], corruption-based perturbations [13], affine transformations [44], and Dirichlet allocation over binned feature values [18]. These methods are generally not guaranteed to affect task-relevant representations, making their induced heterogeneity difficult to interpret, measure, or transfer across tasks and label formats.

More recently, Borazjani et al. [5] explore the use of learned embedding spaces for partitioning data into non-IID clients. Their procedure fine-tunes a task-specific model, extracts latent representations, clusters them with kk-means, and treats the resulting clusters as pseudo-labels for a standard Dirichlet split. They demonstrate that representation-aware heterogeneity can expose failure modes missed by class-based partitions across multiple aggregation methods. However, the number of pseudo-labels kk is hard to interpret, and this method relies on task-specific training, making the resulting partitions difficult to standardize across studies. In contrast, we formulate federated dataset simulation as a decomposition into a representation function Φ\Phi and an assignment mechanism π\pi, enabling label-free simulation in both continuous and semantic representation spaces and supporting evaluation through feature-space, label-space, and downstream FL metrics.

Conclusion

We introduce Federated Dataset Simulation (FDS), a framework that decomposes heterogeneity simulation into a representation function Φ\Phi and an assignment mechanism π\pi, unifying existing label-based approaches with novel feature-level and semantic-level partitioning strategies. Through two concrete instantiations and an accompanying evaluation suite, we provide the community with a comprehensive benchmark for controlled heterogeneity research. The Φ/π\Phi/\pi decomposition is inherently extensible with new representation functions and assignment mechanisms.

Broader impact. FDS lets privacy-preserving FL algorithms be stress-tested under controllable heterogeneity along several dimensions, for dense prediction as well as classification, without access to naturally federated data. The open-source release is intended to lower the barrier for FL researchers to move beyond label-skew-only evaluation.

Limitations. The current evaluation is restricted to computer vision tasks but extending the framework to text, tabular, and multi-modal data is a natural direction for future work. Greater feature-space heterogeneity and greater FedAvg degradation do not by themselves make a benchmark more realistic or more useful, and the framework offers controllable heterogeneity along several dimensions that need not resemble any particular deployment. Our evidence on what geometric partitions separate covers content and scene but not acquisition factors such as device, sensor, and compression, which would require a representation function sensitive to them. The semantic partitioning pipeline depends on the quality and consistency of VLM and LLM outputs, which may introduce noise, and it is weakest on enumeration criteria, with an NMI of 0.297 for count against 0.52 to 0.86 for the other Clever-4 criteria, a gap we attribute to the weakness of current VLMs at counting.

Acknowledgments and Disclosure of Funding

This work was supported by free cloud compute credits that RunPod provided to Giacomo Tarroni and by the Hyperion HPC cluster of City St George’s, University of London. Vasilis Siomos is a full-time employee of Ultromics and Jonathan Passerat-Palmbach is employed by Flashbots, and neither company had any involvement in this work. The other authors declare no competing interests.

The fds package

fds, available at https://github.com/siomvas/fds, is a Python package for controlled federated learning data simulation, partition evaluation, and FL training. It was designed to make the client-partitioning process explicit, reproducible, and measurable, while also providing the training infrastructure required to study how different forms of data heterogeneity affect FL convergence and final model quality.

Core Capabilities

  • Federated data simulation: fds generates client-level datasets for FL experiments under IID, label-skewed, embedding-based, and VLM-informed heterogeneous settings.

  • Reusable partition artifacts: Client splits are serialized as durable artifacts, allowing the same partition to be inspected, scored, reused for training, and evaluated post hoc.

  • Multi-task support: The package supports image classification, semantic segmentation, and object detection workflows through task-specific data modules, models, training loops, and evaluation metrics.

  • End-to-end FL execution: Beyond data splitting, fds can launch full federated training jobs, construct client dataloaders, instantiate models, aggregate client updates, checkpoint global models, and evaluate final models.

  • Experiment analysis: The package includes tools for quantifying partition heterogeneity, summarizing experiment grids, producing reports, and comparing convergence behavior across partitioning methods.

Datasets and Tasks

Datasets are normalized into sample dictionaries. Classification samples contain image and label fields, segmentation samples contain image and mask references, and detection samples contain image annotations such as boxes, labels, and image identifiers. Federated partitions are represented as mappings from client identifiers to sample indices. This decouples data partitioning from model training and makes each partition independently reproducible. The package supports shared global validation, client-local validation, and no-validation training. This allows experiments to distinguish between during-training validation and post-hoc evaluation of the final global model. Splitters are seed-controlled, enabling repeated generation of the same client partitions when the same configuration is used. Saved split files can be reused across model architectures, FL optimizers, or evaluation scripts without regenerating the underlying client assignments.

The package includes an easily extensible dataset factory, which currently supports classification datasets such as Tiny-ImageNet and Food101, segmentation datasets such as ADE20K, and objection detection datasets such as Pascal VOC and BDD100K. Dataset modules apply appropriate preprocessing for each task, including classification image normalization and augmentation, segmentation image-mask transforms, and detection-specific annotation handling. The same data-loading machinery can support centralized baselines and federated client-specific training. fds also includes a model factory that instantiates classification, segmentation, and detection models from supported providers timm and torchvision, transformers and segmentation-models-pytorch backbones.

Partitioning Methods

Random splitting. The random splitter provides IID or near-IID baselines by distributing samples approximately evenly across clients.

Label-Dirichlet splitting. The Dirichlet splitter creates label-skewed non-IID clients. The concentration parameter α\alpha controls the severity of the label imbalance, with smaller values producing more heterogeneous client label distributions.

Embedding-based splitting. Embedding-based partitioning constructs clients according to similarity in feature space rather than according to labels alone.

VLM-based multivariate splitting. The multivariate splitter uses semantic attributes derived from vision-language annotations, allowing client heterogeneity to vary along interpretable axes such as object type, color, size, scene, location, lighting, or viewpoint.

Feature Extractors. The embedding pipeline supports representations from models such as DINOv2, CLIP, SigLIP, Inception-style networks, MAE-style encoders, raw-pixel projections, and lightweight autoencoder representations. Four registered representation functions use no pretrained weights, namely a raw-pixel projection, a sparse random projection, an untrained autoencoder, and a wavelet scattering transform. For transformer-style backbones, fds supports different token-selection and pooling strategies, such as class-token features, patch-token aggregation, and pooled multi-layer representations. Feature normalization and dimensionality reduction, including UMAP, PCA, and sliced-Wasserstein projections, can be applied before clustering, allowing the user to study how representation geometry affects client construction. Embedding partitions can be generated using methods such as K-means, HDBSCAN, KNN-based grouping, and spectral clustering.

Embedding cache. The package also includes a cache manager. Extracted features are cached, so expensive embedding computation can be reused across partitioning methods, client counts, and hyper-parameter searches.

Partition Evaluation Metrics

Label-distribution metrics: fds computes class-balance and label-divergence statistics to quantify label skew across clients.

Feature-distribution metrics: The package also evaluates heterogeneity in embedding space using distance-based and distributional measures like FID, MMD, feature-space energy distance, sliced Wasserstein distance, PCA-based multivariate KS diagnostics, kNN separability, and centroid distance.

Client balance diagnostics: The package records sample-count imbalance and related diagnostics so that pathological or undersized client splits can be identified.

Federated Training Infrastructure

fds uses NVIDIA FLARE [45] to launch and coordinate federated training jobs. Experiments are configured using structured Hydra configs that specify the dataset, partition file, model, optimizer, scheduler, FL method, client count, number of local epochs, validation policy, and output directory. Each client reconstructs its local dataloader from the saved split file and trains on its assigned samples for the configured number of local epochs. The server saves global model states during training, enabling later post-hoc evaluation or offline continuation. Runs can emit logs, manifests, TensorBoard metrics, validation summaries, and model checkpoints for downstream analysis. The comparison of aggregation methods in § 3.3 instead runs the implementations of the fluke library [41] through an adapter in fds (Appendix § 8).

Released Artifacts

The release includes the precomputed criterion labels of the four datasets, the prompts and label sets of Appendix § 9, and the artifacts and configurations of the Clevr-4 and Food-4c experiments of § 3.5.

Hyperparameter details

All federated learning experiments used FedAvg [36] with full client participation in every communication round. We train for one local epoch per round using mixed-precision training and resumed optimizer state across rounds. Experiments were trained for 100 communication rounds. Validation was performed on the global validation set during training.

For Tiny-ImageNet, we used an ImageNet-pretrained timm [52] ResNet-18 classifier with 200 output classes, cross-entropy loss, batch size 32, SGD with learning rate 0.01, momentum 0.9, weight decay 5e-6, and a cosine learning-rate schedule with minimum learning rate 1e-4. For Food101, we used 101 classes, cross-entropy loss, batch size 64, SGD with learning rate 0.01, momentum 0.9, weight decay 0, and no scheduler. For Pascal VOC, we used a COCO-pretrained torchvision Faster R-CNN ResNet50-FPN-v2 detector with 21 classes including background, batch size 2, AdamW with learning rate 1.10e-5, weight decay 0, and no scheduler. For ADE20K, we used a COCO/VOC-pretrained DeepLabV3-ResNet50 segmentation model with 150 classes and Dice loss, batch size 2, AdamW [34] with learning rate 4.5e-5, weight decay 1e-4, and a cosine schedule.

Model hyperparameters were selected with Optuna [2] using a single-client proxy training setup and validation objectives matched to each task: accuracy for classification, mean IoU for segmentation, and [email protected] for detection. We use TPE sampling with median pruning and search over batch size, optimizer, scheduler, log-scaled learning rate, weight decay, SGD momentum, and the pretrained flag for classification backbones. Separately, embedding-based partition selection was tuned with Optuna performing a full grid search over the embedding extractor, token/pooling strategy, normalization, dimensionality reduction using MMD-based heterogeneity scoring.

Comparison of aggregation methods. The experiment of § 3.3 uses Tiny ImageNet at its native resolution of 64×6464 \times64 with 10 clients, a ResNet-18 with group normalization trained from random initialization and an ImageNet-pretrained ResNet-18 with batch normalization, and 100 rounds of one local epoch with all clients participating, with a batch size of 64, SGD with a learning rate of 0.01, momentum 0.9, no weight decay, and no scheduler, and a server learning rate of 1.0. The methods are the implementations of fluke [41] (version 0.8.1), run through fds, which adapts the fluke implementations of FedDyn, FedNova, and SCAFFOLD. Each configuration is run with three seeds on the same partitions. The batch size, input resolution, weight decay, and schedule differ from those of the main Tiny ImageNet experiments, which accounts for the lower accuracies of Table 4.

Semantic Clustering Pipeline Details

The semantic partitioning pipeline follows IC|TC [28], with the key differences being modified prompts and additional guardrails.

Step 1: Criterion-Focused Image Description

Step 1 takes each image and asks a VLM to describe the image while focusing on a single criterion. The selected criterion is substituted into the dataset-specific step1_prompt.txt. The output for each image is a record containing the image path, original dataset label, generated description, and original sample index. The exact Step 1 prompt template is:

Describe the image focusing exclusively on [__CRITERION__].
Avoid proper nouns. Answer in <=80 tokens.

For example, Tiny-ImageNet main_object uses the prompt label Main Object, while Food101 cuisine uses cuisine or culinary origin.

Step 2a: Raw Criterion Label Extraction

Step 2a takes the Step 1 description and asks an LLM to produce a one-word answer for the selected criterion. For example, for Tiny-ImageNet, the exact Step 2a system prompt template is:

You will be given a description of an image. Your job is
to determine the [__CRITERION__]
within the image based on the provided description.
Please respond in a single word.
For example, given the following description:

"""
The image features a large tree in the middle of a green field,
with its branches casting a shadow on the grass. The tree
appears a willow tree, and its branches are covered in green leaves.
The sun is shining, creating a beautiful morning scene.
"""

An exemplar answer is "[__EXEMPLAR__]".

The exact Step 2a user prompt generated for each caption is:

Task: Read the image description and respond
with one word about the "{args.prompt_label}".
Image description: "{caption}"
Your response:

Step 2a writes the LLM output into the label field of each record in step2a_result.jsonl. These labels are raw, unconstrained strings and may include synonyms, spelling variants, or overly specific words.

Ontology Induction from Raw Labels

Step 2b takes all the raw labels, removes punctuation, counts label frequencies, filters low-frequency labels, and asks the LLM to cluster the remaining labels into a requested number of ontology categories. The exact Step 2b system prompt template, shared across datasets, is:

You will be provided a list of [__LEN__] [__CRITERION__]
and the number of occurences in a given dataset.
Your job is to cluster [__LEN__] words into
[__NUM_CLASSES_CLUSTER__] categories.
Provide your answer as a list of [__NUM_CLASSES_CLUSTER__] words,
each word representing a [__CRITERION__].

You must provide your answer in the following format
"Answer {index}: {object}", where {index} is the
index of the category and {object} is the object name
representing the category. For example, if you think the first category
is "object", then you should provide your answer as "Answer 1: object".

When categorizing classes, consider the following criteria:

1. Merge clusters with similar meanings.
2. Each cluster represents a general [__CRITERION__]
and should not be too specific.

Also, please provide a reason you chose the word for each category.
You can provide your reason in the following format
"Reason {index}: {reason}", where {index} is the index of the
category and {reason} is the reason you chose the word for the category.

The exact Step 2b user prompt generated by step2b.format_prompt is:

Task: Cluster these following list of labels into
{num_of_class} categories.
List of labels: {labels}
Your response:

Final Constrained Classification

The final stage reloads the Step 1 image descriptions and the Step 2b ontology labels, then asks the LLM to assign each image description to exactly one ontology label. This is the annotation file consumed by the VLM partition splitter. The exact Step 3 system prompt template is:

You are a classification system.
Your task: Based on the image description, determine the most
appropriate category that best classifies the [__CRITERION__] in the image.
You must choose a category from this list: [__CLASSES__]
IMPORTANT RULES:
- Output only the category name
  • No explanations or extra text

  • Choose the most specific matching category

  • If ambiguous, select the closest match based on dominant feature, no explanations needed

  • If multiple categories could apply, choose the most dominant one

  • Format: CategoryName (exactly as provided in options)

The exact Step 3 user prompt generated by step3.format_prompt is:

Task: Based on the image description, determine the category for
the {args.prompt_label} in the image.
You must choose one option from this list: {class_list}

Image description: "{caption}".

Base your decision on the primary/dominant characteristic.
If a situation arises where nothing is allocated,
please assign it to the object that has the closest resemblance.

Output only the category name.
Your response:

Selected Criteria

Tiny-ImageNet: main_object, size, time, color, location.

Food101: cuisine, preparation, composition, texture, formality.

ADE20K: main_object, scene, color, light, size, camera.

PASCAL: main_object, location, color, light, size.

Criterion Prompt Labels, Exemplars, and Requested Ontology Sizes

DatasetCriterionPrompt labelExemplar / requested KK
tinymain_objectMain ObjectTree / 7
tinysizeSize and Scale of the main objectLarge / 4
tinytimeTime of dayMorning / 4
tinycolorDominant ColorGreen / 7
tinylocationLocationField / 5
food101cuisinecuisine or culinary originItalian / 5
food101preparationcooking or preparation methodfried / 5
food101compositionarrangement and structure of food componentsmixed / 4
food101texturedominant surface texturecrispy / 4
food101formalityplating style and presentation formalityformal / 3
ade20kmain_objectMain ObjectTree / 8
ade20ksceneLocationField / 4
ade20kcolorDominant ColorGreen / 7
ade20klightLighting IntensityBright / 4
ade20ksizeSize of the main objectLarge / 5
ade20kcameraCamera PerspectiveWide-shot / 5
PascalVOCmain_objectMain ObjectTree / 5
PascalVOClocationLocationField / 4
PascalVOCcolorDominant ColorGreen / 6
PascalVOClightLighting IntensityBright / 4
PascalVOCsizeSize of the main objectLarge / 5

Table 1 (PDF p. 20).

Final Label Ontology per Criterion

The following label lists are the final ontology labels extracted from the saved training-split step2b_result.txt files. These labels are the candidate classes supplied to the final constrained classification stage. Tiny-ImageNet

color Brown/Yellow, Red, Green, Blue, Black, White, Other.

location Indoors, Urban Outdoors, Nature Outdoors, Aquatic, Unknown.

main_object Human, Animal, Vehicle, Building, Food, Landscape, Inanimate Object.

size Small, Large, Medium, Expansive.

time Day, Night, Unknown, Sunset/Dusk.

Food101

composition Mixed, Layered/Stacked, Scattered/Spread, Uniform.

cuisine American, Italian, Asian, Mexican, French.

formality Casual, Elegant, Modern.

preparation Baked, Fried, Steamed, Raw, Mixed.

texture Smooth, Rough, Crispy, Soft.

ADE20K

camera Lowangle, Eyelevel, Highangle, Aerial, Wideangle.

color Brown/Beige, White, Green, Gray, Blue, Red/Orange, Black.

light Intense, Moderate, Dim.

main_object Building, Furniture, Vehicle, Person, Nature.

scene Urban Outdoors, Residential Indoors, Natural Outdoors, Office Indoors.

size Large, Small, Medium, Huge.

Pascal VOC

color Red/Orange, Blue/Purple, Green, Black, Brown/Yellow, White.

light Dim, Moderate, Bright, Intense.

location Indoors, Urban Outdoors, Nature.

main_object Animal, Person, Vehicle, Nature, Unanimated Object, Building.

size Large, Small, Medium, Huge.

Pure-Criterion Ablation

Table 6 gives the feature-space heterogeneity of the pure-criterion splits of § 3.5.

Sensitivity of Geometric Partitioning to the Representation

Geometric partitioning involves several choices beyond the encoder, and we measured how much the representation function, normalization, and dimensionality reduction each matter. Over 733 configurations at K=10K = 10, spanning the four datasets, eight representation functions, and reduction by UMAP to 32 or 128 dimensions or no reduction, the representation function accounts for between 58.8% and 76.3% of the variance in MMD depending on the dataset. The reduction accounts for between 0.02% and 7.8% and normalization for less than 0.2%, and FID and centroid distance order the factors in the same way. UMAP raises MMD by a median factor of 1.39 on Food101 and 1.08 on Tiny ImageNet, has no effect on the two dense prediction datasets, and ranks configurations as the unreduced embeddings do, with Spearman correlations between 0.885 and 0.909.

A further sweep of 160 configurations crosses the four datasets with four encoders and ten reduction settings, including PCA to 32 and 128 dimensions and variants of the UMAP parameters. PCA never changes which encoder produces the most heterogeneous partition, although at 128 components it shifts the magnitude of MMD by +17% on ADE20K and −22% on Tiny ImageNet. DINOv2-B, selected for the main experiments by its average MMD rank, remains the encoder with the highest MMD under the UMAP settings deployed there. Other UMAP settings change the leading encoder in six arms on Tiny ImageNet and Food101, while DINOv2-B leads throughout on ADE20K and PASCAL VOC.

Semantic Pipeline: Validation, Models, and Compute

Clevr-4. Clevr-4 [48] consists of synthetic scenes that admit four equally valid taxonomies, by shape, texture, color, and count. We run the pipeline with InternVL3.5-38B and Llama 3.3-70B on 10,531 Clevr-4 images and compare the partition produced for each criterion with each of the four taxonomies (Table 7). No stage of the pipeline reads an annotation, but the texture prompt was revised once during the run, after the descriptions produced so far had been clustered and scored against the texture taxonomy. Measured by AMI, each criterion agrees with its own taxonomy far more than with any other, by factors of 277 for shape, 34 for color, 21 for count, and 8.5 for texture over the strongest other taxonomy, and the AMI off the diagonal averages 0.010. Two controls bound these values. The four annotated taxonomies are nearly independent of one another, with a mean AMI of 0.0007, and the four criterion partitions share little with one another, with a mean AMI of 0.012. Count, the one enumeration criterion, is recovered least completely, which we attribute to the weakness of current vision-language models at counting. With Qwen3-VL-8B [3] and Qwen3-8B [53] in place of the two models, every criterion still peaks on its own taxonomy, while the NMI falls from 0.860 to 0.674 for shape and from 0.793 to 0.711 for color, so the capability of the models governs how completely a criterion is recovered and not which axis is recovered. This run kept the original texture prompt, so its texture NMI of 0.272 does not isolate the effect of the models.

Table 7: Clevr-4 validation of the semantic pipeline (InternVL3.5-38B and Llama 3.3-70B). (a) NMI between the partition produced for each criterion and each annotated taxonomy. (b) AMI between the four criterion partitions. (c) AMI between the four annotated taxonomies.

(a) Criterion (rows) against taxonomy (columns), NMI
TextureTextureShapeColorCount
Texture0.5240.0630.0120.003
Shape0.0040.8600.0020.005
Color0.0250.0080.7930.002
Count0.0020.0160.0030.297
(b) Criterion against criterion, AMI
TextureTextureShapeColorCount
Texture–0.0410.0190.001
Shape0.041–0.0050.008
Color0.0190.005–0.001
Count0.0010.0080.001–
(c) Taxonomy against taxonomy, AMI
TextureTextureShapeColorCount
Texture–0.0000-0.00010.0001
Shape0.0000–0.00000.0043
Color-0.00010.0000–-0.0001
Count0.00010.0043-0.0001–

Table 7.

Food-4c. Food-4c [33] supplies human annotations of Food101 images along four criteria, cuisine among them. On 68,250 Food101 images, with the human cuisine labels coarsened to five categories, the cuisine labels of our pipeline reach an adjusted mutual information (AMI) of 0.503 with the human labels, an adjusted Rand index of 0.465, and direct agreement on 73.0% of images. Pipeline criteria that do not correspond to cuisine score far lower against the same human labels, 0.092 for preparation and 0.018 for texture, as do the pipeline’s cuisine labels against an unrelated human criterion, at 0.095. The five criteria of the pipeline are also nearly independent of one another, with a mean pairwise AMI of 0.032 and a maximum of 0.071, whereas the human taxonomies of Food-4c overlap far more, with a mean pairwise AMI of 0.360 and a maximum of 0.625.

Models. We tried the vision-language models MiniCPM-V 2.6 [54], InternVL2 [10], InternVL2.5-38B [9], InternVL3.5 at 8B and 38B parameters [51], and Qwen3-VL at 8B and 30B parameters [3], and the language models Llama 3.1-8B-Instruct and Llama 3.3-70B [16]. Larger models and newer generations performed better, and above a threshold of capability the choice of model family mattered little.

Compute. The labeling pipeline ran on two A100 GPUs with 80 GB of memory each. Labeling took about three days per criterion on the classification datasets of roughly 100,000 images and less than a day per criterion on ADE20K and PASCAL VOC, with criteria labeled concurrently. Labeling the 10,531 Clevr-4 images for all four criteria took about 16 hours and 4.5 million tokens.

Client Composition of Geometric Partitions

We compare each geometric partition of the heterogeneity tables with the pure-criterion partition of each criterion, for the four datasets and K∈{5,10,20}K \in\{5, 10, 20\}, which gives 63 combinations of dataset, criterion, and client count Table 8. The criterion labels cover every sample of each partition (100,000 on Tiny ImageNet, 75,750 on Food101, 25,574 on ADE20K, and 5,717 on PASCAL VOC). Over all datasets, the content criteria (main object, scene, and cuisine) reach a mean AMI of 0.379, 0.377, and 0.363 at K=5K = 5, 10, and 20, against 0.056, 0.060, and 0.061 for lighting, color, camera, and size. On ADE20K at K=10K = 10 the ordering runs from scene (0.417) and main object (0.334) through camera (0.095), size (0.078), and color (0.033) to lighting (0.013), and total variation distances agree. Alignment also shifts with the number of clients, since on ADE20K alignment with scene falls from 0.492 at K=5K = 5 to 0.382 at K=20K = 20, whereas on Food101 alignment with cuisine rises from 0.117 to 0.255.

DatasetCriterionK=5K = 5K=10K = 10K=20K = 20
Tiny ImageNetmain objectc0.4830.5150.486
sizes0.0730.0840.085
time0.0800.0810.075
colors0.0530.0590.072
location0.2410.2370.233
Food101cuisinec0.1170.1480.255
preparation0.1460.2190.257
composition0.0640.0670.090
texture0.0250.0500.061
formality0.0380.0500.055
ADE20Kmain objectc0.3720.3340.309
scenec0.4920.4170.382
colors0.0360.0330.031
lights0.0070.0130.014
sizes0.0830.0780.078
cameras0.1070.0950.087
PASCAL VOCmain objectc0.4320.4720.381
location0.1110.2260.190
colors0.0410.0520.060
lights0.0130.0280.032
sizes0.0860.0960.094

Table 8. AMI between each geometric partition and the pure-criterion partition of each criterion, for K∈{5,10,20}K \in\{5, 10, 20\}. cContent criteria. sCriteria of lighting, color, camera, and size.

Individual clients carry recognizable signatures Table 9. On Tiny ImageNet with 10 clients, one client holds vehicles at 10.6 times their global rate together with urban outdoor scenes at 4.3 times, another food at 9.8 times with indoor scenes at 2.1 times, and a third humans at 4.7 times. On ADE20K with five clients, one client consists of 96.1% residential indoor scenes against 31.7% in the dataset as a whole.

DatasetClientSignature
Tiny ImageNet0location=Aquatic (3.66×\times), main object=Animal (2.65×\times), size=Small (1.79×\times)
1main object=Building (6.44×\times), size=Expansive (5.70×\times), main object=Landscape (5.33×\times)
2main object=Inanimate Object (2.73×\times), time=Unknown (1.98×\times), time=Night (1.87×\times)
3main object=Animal (3.53×\times), location=Nature Outdoors (2.60×\times), color=Brown/Yellow (1.97×\times)
4main object=Vehicle (10.55×\times), location=Urban Outdoors (4.34×\times), color=White (2.01×\times)
5main object=Human (4.68×\times), size=Medium (1.86×\times), time=Unknown (1.74×\times)
6main object=Food (9.84×\times), location=Indoors (2.06×\times), color=Red (1.97×\times)
7main object=Animal (3.62×\times), location=Aquatic (3.24×\times), location=Nature Outdoors (2.33×\times)
8color=Black (4.75×\times), main object=Animal (3.51×\times), location=Nature Outdoors (2.44×\times)
9main object=Animal (3.51×\times), size=Medium (1.49×\times), location=Indoors (1.44×\times)
Food1010preparation=Raw (1.86×\times), formality=Modern (1.75×\times), formality=Elegant (1.56×\times)
1cuisine=Mexican (2.50×\times), preparation=Mixed (1.86×\times), preparation=Baked (1.38×\times)
2texture=Crispy (3.31×\times), preparation=Fried (3.01×\times), composition=Scattered/Spread (1.68×\times)
3preparation=Baked (2.29×\times), composition=Uniform (1.84×\times), cuisine=French (1.66×\times)
4composition=Uniform (6.82×\times), cuisine=American (3.36×\times), preparation=Raw (2.67×\times)
5preparation=Mixed (5.77×\times), cuisine=Italian (5.52×\times), composition=Mixed (3.84×\times)
6preparation=Steamed (4.92×\times), cuisine=French (3.37×\times), preparation=Raw (2.28×\times)
7preparation=Steamed (9.82×\times), cuisine=Asian (4.55×\times), composition=Mixed (2.47×\times)
8cuisine=Mexican (2.12×\times), texture=Crispy (1.76×\times), texture=Soft (1.67×\times)
9preparation=Steamed (10.50×\times), cuisine=Asian (4.64×\times), composition=Scattered/Spread (2.41×\times)
ADE20K0main object=Person (2.27×\times), scene=Natural Outdoors (2.02×\times), camera=Highangle (1.71×\times)
1scene=Residential Indoors (3.14×\times), main object=Furniture (3.02×\times), color=Red/Orange (2.65×\times)
2camera=Wideangle (3.44×\times), color=Black (2.79×\times), main object=Furniture (2.73×\times)
3scene=Residential Indoors (3.03×\times), main object=Furniture (2.75×\times), size=Medium (2.62×\times)
4scene=Urban Outdoors (2.77×\times), main object=Vehicle (2.26×\times), main object=Building (2.26×\times)
5main object=Nature (3.82×\times), scene=Natural Outdoors (2.98×\times), size=Small (1.85×\times)
6scene=Office Indoors (4.15×\times), color=Black (2.74×\times), main object=Person (2.49×\times)
7scene=Residential Indoors (2.78×\times), camera=Wideangle (2.67×\times), main object=Furniture (2.42×\times)
8scene=Urban Outdoors (2.58×\times), main object=Building (2.26×\times), main object=Vehicle (1.60×\times)
9camera=Wideangle (3.13×\times), main object=Furniture (2.85×\times), scene=Residential Indoors (2.58×\times)
PASCAL VOC0main object=Animal (2.75×\times), size=Small (2.27×\times), color=Brown/Yellow (1.41×\times)
1main object=Vehicle (2.73×\times), location=Urban Outdoors (2.15×\times), color=Red/Orange (1.71×\times)
2main object=Vehicle (3.44×\times), size=Huge (2.65×\times), color=White (2.23×\times)
3color=Green (2.88×\times), location=Nature (2.69×\times), main object=Animal (2.64×\times)
4main object=Unanimated Object (3.29×\times), main object=Person (2.36×\times), location=Indoors (1.99×\times)
5size=Huge (5.37×\times), main object=Vehicle (3.29×\times), location=Urban Outdoors (2.26×\times)
6main object=Animal (2.88×\times), location=Nature (2.59×\times), color=Green (1.50×\times)
7main object=Animal (2.89×\times), location=Indoors (2.12×\times), size=Medium (1.98×\times)
8main object=Animal (2.54×\times), location=Nature (2.44×\times), size=Large (1.98×\times)
9main object=Building (6.07×\times), size=Huge (2.65×\times), main object=Vehicle (2.26×\times)

Table 9. Client signatures of the geometric partitions at K=10K = 10: for each client, the three criterion values with the largest lift (client rate over global rate) among values covering at least 5% of the client.

Practical Guidance

Choosing the representation. Content-level heterogeneity, which § 3.6 shows geometric partitions to follow, needs no intervention beyond the default encoder. Other axes call for attention to the biases of the encoder, since objectives that reconstruct masked pixels and objectives that align embeddings capture different properties of an image [40, 48], and to the projection, removing the top pp principal components with pp as the control. Where neither suffices, the representation functions that use no pretrained weights (Appendix § 7), or an encoder trained on the domain in question, can take the place of a general-purpose encoder.

Choosing criteria. To choose criteria, we start by listing the ways in which the images of a dataset actually differ. On Food101, candidate criteria for viewpoint, color, and lighting returned nearly degenerate labels, and none of the three appears among the final criteria (Appendix § 9). Each remaining candidate should be validated on a subset for a non-degenerate label distribution, the survivors ranked with the pure-criterion screen of § 3.5, and the final set checked for redundancy, which on Food101 is low, the five criteria sharing a mean pairwise AMI of 0.032 (Appendix § 12).

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: We substantiate the presented claims through our framework in Section 2 and an extensive empirical analysis in Section 3. The code and artifacts are released at https://github.com/siomvas/fds.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our work as part of the Conclusion section.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: This paper introduces only some definitions relating to the task of federated dataset simulation, which serve to form a common language when making this task the object of study and design. Their relevance and utility is shown empirically in the corresponding section.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a best-effort explanation of all the hyper-parameters used. Along with the released code, this should be enough to replicate our results.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is released with instructions at https://github.com/siomvas/fds.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: We do this in Section 3 and Appendix § 8.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The reason we do not run multiple seeds is that the computational burden is too high, and cannot be justified. After preliminary runs established the performance variance across seed was much lower compared to the gap between partitioning methods, we were satisfied that the insights from a single seed were correct. This can be attributed to our use of pre-trained models for initialization, which is known to greatly stabilize performance in FL optimization. The comparison of aggregation methods in § 3.3 reports the mean and standard deviation over three seeds.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix F reports the hardware and time of the semantic labeling pipeline. For the remaining experiments we provide part of this information, since collecting all of it would require estimating it post hoc.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and verified the submission conforms with it.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We briefly discuss the potential positive impact of better FL algorithm in the conclusion.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: This is not applicable to our paper.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original owners of assets we have used, and have checked that the corresponding licenses allow the assets to be used for research purposes.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] Justification: The repository at https://github.com/siomvas/fds documents the released code and artifacts and states their license, Apache 2.0.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: This is not applicable to our paper.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: This is not applicable to our paper.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Use of an LLM is part of our semantic partitioning method. It is documented as a part of the pipeline, we reference the used model, which is open-source, and provide the exact prompts in the appendix.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [2]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019.arxiv.org/abs/1907.10902
  2. [3]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  3. [4]Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210, 2023.arxiv.org/abs/2304.12210
  4. [5]Kasra Borazjani, Payam Abdisarabshali, Naji Khosravan, and Seyyedali Hosseinipour. Redefining non-iid data in federated learning for computer vision tasks: Migrating from labels to embeddings for task-specific data distributions. IEEE Transactions on Artificial Intelligence, 2026.
  5. [6]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In European conference on computer vision, pages 446–461. Springer, 2014.DOI
  6. [7]Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.arxiv.org/abs/1812.01097
  7. [8]Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.arxiv.org/abs/1706.05587
  8. [9]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.arxiv.org/abs/2412.05271
  9. [10]Zhe Chen, Jiannan Wu, Weihan Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024.
  10. [11]Chris Ding and Xiaofeng He. K-nearest-neighbor consistency in data clustering: incorporating local information into global optimization. In Proceedings of the 2004 ACM symposium on Applied computing, pages 584–589, 2004.DOI
  11. [12]Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88 (2):303–338, 2010.DOI
  12. [13]Xiuwen Fang, Mang Ye, and Xiyuan Yang. Robust heterogeneous federated learning under data corruption. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4997–5007, 2023. doi: 10.1109/ICCV51070.2023.000463.DOI
  13. [14]Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in neural information processing systems, 33: 19586–19597, 2020.arxiv.org/abs/2006.04088
  14. [15]Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066–2073, 2012. doi: 10.1109/CVPR.2012.6247911.DOI
  15. [16]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  16. [17]Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723–773, 2012.DOI
  17. [18]Daniel Mauricio Jimenez Gutierrez, Aris Anagnostopoulos, Ioannis Chatziagiannakis, and Andrea Vitaletti. Fedartml: A tool to facilitate the generation of non-iid datasets in a controlled way to support federated learning research. IEEE Access, 12:81004–81016, 2024. doi: 10.1109/ACCESS.2024.3410026.DOI
  18. [19]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.arxiv.org/abs/1603.05027
  19. [20]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, 2022. doi: 10.1109/CVPR52688.2022.01553.DOI
  20. [21]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
  21. [22]Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification, 2019. URL https://arxiv.org/abs/1909.06335.
  22. [23]Hyejun Jeong, Shiqing Ma, and Amir Houmansadr. A survey on federated unlearning: Challenges and opportunities. IEEE Transactions on Big Data, pages 1–20, 2026. doi: 10.1109/TBDA.2026.3668358.
  23. [24]Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.arxiv.org/abs/1912.04977
  24. [25]Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020.arxiv.org/abs/1910.06378
  25. [26]Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
  26. [27]Krishan Kumar, Komal Sharma, Vinay Kukreja, and Abhishek Bhattacherjee. Exploring federated learning trends: a bibliometric analysis and word2vec nlp approach. Progress in Artificial Intelligence, pages 1–17, 2025.DOI
  27. [28]Sehyun Kwon, Jaeseung Park, Minkyu Kim, Jaewoong Cho, Ernest K Ryu, and Kangwook Lee. Image clustering conditioned on text criteria. arXiv preprint arXiv:2310.18297, 2023.arxiv.org/abs/2310.18297
  28. [29]Yann Le, Xuan Yang, et al. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  29. [30]Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pages 965–978. IEEE, 2022.arxiv.org/abs/2102.02079
  30. [31]Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  31. [32]Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021.arxiv.org/abs/2111.11429
  32. [33]Mingxuan Liu, Zhun Zhong, Jun Li, Gianni Franchi, Subhankar Roy, and Elisa Ricci. Organizing unstructured image collections using natural language. In Findings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8907–8918, 2026.arxiv.org/abs/2410.05217
  33. [34]Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5(5):5, 2017.
  34. [35]Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.arxiv.org/abs/1802.03426
  35. [36]Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.arxiv.org/abs/1602.05629
  36. [37]Adam Narożniak. Federated datasets in research. Flower Blog, December 2024. URL https://flower.ai/blog/2024-12-02-federated-datasets-in-research/. Accessed: 2026-05-04.
  37. [38]Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. Advances in Neural Information Processing Systems, 35:5315–5334, 2022.arxiv.org/abs/2210.04620
  38. [39]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=a68SUt6zFt. Featured Certification.
  39. [40]Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? In International Conference on Learning Representations, 2023.arxiv.org/abs/2305.00729
  40. [41]Mirko Polato. fluke: Federated learning utility framework for experimentation and research. arXiv preprint arXiv:2412.15728, 2024.arxiv.org/abs/2412.15728
  41. [42]Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, pages 18250–18280. PMLR, 2022.arxiv.org/abs/2206.02618
  42. [43]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
  43. [44]Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. Advances in neural information processing systems, 33:21554–21565, 2020.arxiv.org/abs/2006.08907
  44. [45]Holger R Roth, Yan Cheng, Yuhong Wen, Isaac Yang, Ziyue Xu, Yuan-Ting Hsieh, Kristopher Kersten, Ahmed Harouni, Can Zhao, Kevin Lu, et al. Nvidia flare: Federated learning from simulation to real-world, 2022. URL https://arxiv.org/abs/2210.13291.
  45. [46]David Solans, Mikko Heikkila, Andrea Vitaletti, Nicolas Kourtellis, Aris Anagnostopoulos, Ioannis Chatziogiannakis, et al. Non-iid data in federated learning: A survey with taxonomy, metrics, methods, frameworks and future directions. arXiv preprint arXiv:2411.12377, 2024.arxiv.org/abs/2411.12377
  46. [47]Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re-thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  47. [48]Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. In Advances in Neural Information Processing Systems, volume 36, pages 19962–19989, 2023.arxiv.org/abs/2311.17055
  48. [49]Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.arxiv.org/abs/2007.07481
  49. [50]Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Agüera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Adwait Gadkari, Zachary Garrett, Antoniouѕ M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization, 2021. URL https://arxiv.org/abs/2107.06917.
  50. [51]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
  51. [52]Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
  52. [53]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengchen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  53. [54]Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. MiniCPM-V: A GPT-4V level MLLM on your phone. arXiv preprint arXiv:2408.01800, 2024.arxiv.org/abs/2408.01800
  54. [55]Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pages 7252–7261. PMLR, 2019.arxiv.org/abs/1905.12022
  55. [56]Maximilian Zenk, Ujjwal Baid, Sarthak Pati, Akis Linardos, Brandon Edwards, Micah Sheller, Patrick Foley, Alejandro Aristizabal, David Zimmerer, Alexey Gruzdev, et al. Towards fair decentralized benchmarking of healthcare ai algorithms with the federated tumor segmentation (fets) challenge. Nature communications, 16(1):6274, 2025.
  56. [57]Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pages 26311–26329. PMLR, 2022.arxiv.org/abs/2209.00189
  57. [58]Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.DOI

Paper details

vlnc:W_spqejr7k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents