betaFeedback

WorksComputer Science

Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

Abstract

Evaluating candidate changes (hypotheses) to recommendation models requires full-scale training runs consuming thousands of GPU-hours, severely limiting the hypothesis space engineers can feasibly explore. We introduce Predictable Scaling (PS), a systematic framework that exploits empirically verified regularity conditions to predict full-scale model performance from small-scale proxies. PS constructs scaled-down variants via composable scaling functions along distinct resource axes, and projects metrics to the target scale via curve fitting with calibrated uncertainty estimates. We compare three prediction strategies of increasing expressiveness: even the simplest (raw proxy) achieves prediction error within the empirical noise floor of repeated-run variance; the most expressive (Gaussian Process) provides an indicative 15% relative MAE improvement and calibrated uncertainty. Evaluation on two ranking models of increasing complexity—a task-specific model and a foundation model—demonstrates that PS correctly ranks actionable hypotheses across diverse hypothesis types at 79-88% compute reductions.

Introduction

In industrial recommendation systems, model exploration—evaluating candidate changes against established baselines—dominates computational requirements. Each candidate requires a full training run, and the candidate set grows multiplicatively with the design space (architectures ×\times features ×\times hyperparameters). While scaling laws have been extensively studied for LLMs [9, 8], their exploitation for predicting hypothesis-level performance differences in recommendation systems remains largely unexplored. Our key empirical observation is that relative performance gaps between hypotheses behave as smooth, low-dimensional functions of scale, even when absolute curves are noisy. We present Predictable Scaling (PS), a framework that leverages this regularity to predict hypothesis performance at a fraction of full training compute. Our contributions:

  • Formalization and scaling functions. We formalize reduced-scale evaluation as prediction over relative gaps with explicit regularity conditions, and introduce six composable scaling functions that reduce compute along distinct resource axes (§ 3).

  • Predictor comparison with uncertainty. Three strategies: simplest achieves error within AA noise floor; GP provides calibrated uncertainty for reliability screening (§ 4).

  • Large-scale evaluations. PS validated on two ranking models achieving 79–88% compute reduction while correctly ranking actionable hypotheses (§ 4).

Related Work

Scaling laws in deep learning. Kaplan et al. (Kaplan et al., [9] and Hoffmann et al. (Hoffmann et al., [8] established power-law scaling for language models. Gadre et al. (Gadre et al., [5] show these laws hold under over-training, relevant to PS’s under-trained proxy regime. However, these results derive from homogeneous Transformers; recommendation systems break this assumption with heterogeneous architectures coupling sparse embeddings, dense layers, and sequential components.

Efficient candidate evaluation. Multi-fidelity methods (Hyperband [11], BOHB [4]) adaptively terminate unpromising configurations within a fixed architecture. μ\muP [17] achieves zero-shot hyperparameter transfer. PS is complementary: it predicts impact of structural changes (architecture, features, data) that each require distinct training runs and cannot be framed as early-stopping decisions.

Small-to-large prediction. Tay et al. [15] show best architectures can change across scales; Choshen et al. [2] and Lee et al. [10] propose methodology and Bayesian extrapolation for LLMs. All predict absolute LLM losses; PS predicts relative gaps between recommendation model hypotheses.

Recommendation system scaling. Ardalani et al. [1] show DLRM quality follows power-law-plus-constant with parameter scaling saturating before data scaling. Wukong (Zhang et al., 2024) and HSTU (Zhai et al., 2024) confirm power-law scaling. These target absolute performance for capacity planning; PS targets the complementary problem of ranking hypotheses relative to a baseline.

Predictable Scaling: Problem Formulation and Framework

Setting

Let P0\mathcal{P}_0 denote a baseline model, and {P1,…,PN}\{\mathcal{P}_1,\ldots,\mathcal{P}_N\} denote NN hypothesis configurations. Each yields scalar metric ℓi\ell_i (normalized entropy) after convergent training. The relative performance gap is:

Δi=ℓi−ℓ0ℓ0,i∈[N],(1)\Delta_i = \frac{\ell_i-\ell_0}{\ell_0}, \quad i \in[N], \tag*{(1)}

where Δi<0\Delta_i < 0 indicates improvement. A scaling mechanism sjs_j produces a more efficient variant with scaled gap Δi(j)=(ℓi(j)−ℓ0(j))/ℓ0(j)\Delta_i^{(j)} = (\ell_i^{(j)}-\ell_0^{(j)})/\ell_0^{(j)}. The scaled gaps form observation vector di=(Δi(1),…,Δi(M))∈RM\mathbf{d}_i = (\Delta_i^{(1)},\ldots,\Delta_i^{(M)}) \in\mathbb{R}^M.

Scaling Functions

Each scaling mechanism sj=(ϕk,αl)s_j = (\phi_k,\alpha_l) binds a function family to a reduction ratio α∈(0,1]\alpha\in(0,1]. We implement six families (Table 1); full details in section C.

FnDescriptionScaledDimrk(α)r_k(\alpha)
DDUniform subsampling, hash-seededData volumeFLOPsα\alpha
HSRReduce table rows (min 5K)Embedding rowsMemoryα\alpha
ESShrink vector width, alignedEmbedding widthFLOPs+Memα\alpha
FSPrune by importance scoresInput featuresFLOPsα\alpha
WASScale all dims uniformlyAll hidden dimsFLOPs+Memα2\alpha^2
DSTruncate to ⌊αL⌋\lfloor\alpha L \rfloor layersLayersFLOPsα\alpha

Table 1. Scaling functions in PS. Each parameterized by α∈(0,1]\alpha\in(0,1].

Multiple functions compose multiplicatively: r(c)=∏t=1mrkt(αlt)r(c) = \prod_{t=1}^{m} r_{k_t}(\alpha_{l_t}). We define scaling profiles—curated compositions producing proxies at a target computational reduction (e.g., 10×\times).

Gap Extrapolation

A predictor fθ:RM→Rf_\theta: \mathbb{R}^M \to\mathbb{R} maps proxy observations to predicted full-scale gaps Δ^i=fθ(di)\widehat{\Delta}_i = f_\theta(\mathbf{d}_i). Beyond the raw proxy baseline (Δ^i=Δi(1)\widehat{\Delta}_i = \Delta_i^{(1)}), we consider:

ModelPrimary Task(s)FLOPs/fwd (M)TrainersSamples (B)HardwareProxy Budget Constraint
TS-1 (Task-Specific)Feed CTR + CVR3,40311280NVIDIA H100ΔTrainers≤−50%\Delta\mathrm{Trainers} \le-50\%
FM-1 (Foundation)Multi-task CTR + CVR117,7341,024122NVIDIA H100ΔSamples≤−50%\Delta\mathrm{Samples} \le-50\%

caption: tbl:2 Summary of the two evaluation models. FLOPs denote the Floating-point Operations per forward pass and QPS denotes the queries per second for each model type.

:::

Linear regression. Δ^i=wTdi+b\hat{\Delta}_i = \mathbf{w}^{\mathsf{T}}\mathbf{d}_i + b, fit via OLS. Effective when the gap transfer function is approximately affine.

Empirical Gaussian Process (EGP). A GP whose prior is learned from historical training curves [13]; the mean function is the pointwise average and the covariance is the empirical sample covariance of historical NE trajectories. The EGP extrapolates NE over training steps at a fixed scale point, predicting where the proxy’s learning curve will converge. The posterior provides calibrated uncertainty; predictions exceeding a coefficient-of-variation threshold are rejected, signaling that full-scale training is needed.

Optimization Objective and Success Criteria

PS replaces full-scale training with efficient proxy runs and extrapolates to the target scale. It jointly optimizes scaling mechanisms SS and predictor fθf_\theta:

Δ^i=fθ(di),min⁡S,g∈G,θEs.t. ∣S∣=M, compute⁡(S)≤B.(2)\hat{\Delta}_i = f_\theta(\mathbf{d}_i), \qquad\min_{S,g\in\mathcal{G},\theta} \mathcal{E} \quad\text{s.t. } |S| = M,\ \operatorname{compute}(S) \le B. \tag*{(2)}

The primary metric is MAE: E=1N∑i=1N∣Δ^i−Δi∣\mathcal{E} = \frac{1}{N}\sum_{i=1}^{N}|\hat{\Delta}_i-\Delta_i|. The secondary metric is Spearman ρ\rho over hypotheses with ∣Δi∣>τAA|\Delta_i| > \tau_{\mathrm{AA}} (the A/A noise floor; formal definition in section B.1).

Regularity conditions. Four assumptions under which reduced-scale evaluation succeeds: (A1) Convergence stability—metrics are stable and reproducible; (A2) Gap transferability—a consistent function f∗f^* maps scaled to full-scale gaps; (A3) Monotonic signal preservation—scaling preserves gap sign with high probability; (A4) Low-dimensional predictive structure—f∗f^* admits a low-complexity form. Empirical verification in section D.

Compute Budget-Optimal Proxy Selection

The selection procedure enumerates feasible configurations satisfying (N+1)⋅r(c)⋅Cfull≤B(N+1)\cdot r(c)\cdot C_{\mathrm{full}} \le B, trains proxies on a benchmark corpus with known full-scale gaps, and selects (S∗,g∗)(S^*,g^*) minimizing E\mathcal{E} via cross-validation. The enumeration is tractable (KK small, coarse ratio grid, depth m≤3m \le3) and pruned via compute-tier filtering. Full procedure in algorithm 1 (section F).

Experiments and Results

We evaluate PS on large-scale industrial datasets. A reproducible public evaluation on Criteo (Criteo Labs, 2014) with four public models is provided in section H.

Models. Two ranking models of increasing complexity (table 2): TS-1, a task-specific feed ranking model; FM-1, a foundation model (≈35×\approx35\times TS-1 FLOPs). Both are multi-task models jointly predicting CTRCTR and CVRCVR across recommendation surfaces. Both train on O(1010)O(10^{10})–O(1011)O(10^{11}) user interaction logs with ∼1500\sim1500 dense and ∼700\sim700 sparse, high-cardinality categorical features.

Setup. We use Normalized Entropy (NE) [7] as the evaluation metric (NE<1<1 indicates improvement over base rate). All evaluations use M=1M=1 proxy. TS-1 evaluates 14 held-out hypotheses across 3 predictors; FM-1 evaluates 7 held-out hypotheses comparing raw proxy and GP. MAE values are normalized by per-model τAA\tau_{\mathrm{AA}}.

PredictorMAE (×τAA\times\tau_{\mathrm{AA}})SEΔ\DeltaMAEρ\rhoΔρ\Delta\rho
Raw proxy0.865±0.776\pm0.776—0.755—
Linear0.846±0.738\pm0.738−4%0.7550%
GP (EGP)0.738±0.657\pm0.657−15%0.800+6%

caption: TS-1 prediction accuracy (CTR head, 14 held-out hypotheses).

:::

TS-1: Predictor ablation. Table 3 compares three strategies on TS-1 (CTR head). The raw proxy achieves 0.865τAA0.865\tau_{\mathrm{AA}} MAE (ρ=0.755\rho= 0.755); the EGP achieves the best at 0.738τAA0.738\tau_{\mathrm{AA}} (ρ=0.8\rho= 0.8)—a 15% relative MAE improvement. Among the 6 actionable hypotheses (∣Δ∣>τAA|\Delta| > \tau_{\mathrm{AA}}), PS achieves ρ=1.0\rho= 1.0 under all predictors. Figure 1 visualizes this progression.

TS-1 predictive fidelity for raw proxy, linear, and GP (EGP)

Figure 1. TS-1 predictive fidelity (CTR). Each point is a hypothesis; diagonal = perfect prediction; shaded band = ±τAA\pm\tau_{\mathrm{AA}}. Filled: held-out; open: training.

FM-1: GP on a foundation model. Table 4 compares raw proxy and GP on FM-1 across 7 held-out hypotheses. The raw proxy exceeds the noise floor on CTR heads (2.15τAA2.15\tau_{\mathrm{AA}}, 1.75τAA1.75\tau_{\mathrm{AA}}). The GP reduces these to 0.700.70–0.85τAA0.85\tau_{\mathrm{AA}}—a 57–67% MAE improvement—confirming that curve fitting provides substantial value on foundation models where the raw proxy alone is insufficient.

HeadPredictorMAE (×τAA\times\tau_{\mathrm{AA}})SEΔ\DeltaMAEρ\rhoΔρ\Delta\rho
CTR-1Raw proxy2.150±1.692\pm1.692—0.607—
CTR-1GP (EGP)0.701±0.552\pm0.552−67%0.714+18%
CTR-2Raw proxy1.750±1.466\pm1.466—0.643—
CTR-2GP (EGP)0.753±0.631\pm0.631−57%0.714+11%
CVR-1Raw proxy0.950±0.556\pm0.556—0.714—
CVR-1GP (EGP)0.854±0.500\pm0.500−10%0.7140%

Table 4. FM-1 prediction accuracy (7 held-out hypotheses). Δ\Delta columns: change vs. raw proxy.

Compute reduction. TS-1 and FM-1 reduce trainers by 86%/88%, and samples by 80%/50%. Overall, PS achieves 79–88% compute reduction across both models (Table 6, section A.4).

ModelΔ\DeltaQPSΔ\DeltaSamplesΔ\DeltaTrainersCompute Red.
TS-1−76%−80%−86%88%
FM-1−70%−50%−88%79%

Table 6. Resource efficiency of selected proxy configurations.

Discussion and Conclusion

PS performs best with fast NE convergence, low AA variance, and effect sizes exceeding τAA\tau_{\mathrm{AA}}; when these hold, even the raw proxy suffices for triage. The GP’s benefit increases with complexity (15% on TS-1 vs. 57–67% on FM-1), suggesting curve fitting becomes essential as proxy fidelity degrades at scale. Design trade-offs include scaling aggressiveness (WAS reduces NE gap by 38% relative to ES at α=0.5\alpha=0.5; section E) and composition risk—sign reversals are detectable on a benchmark corpus and should be rejected above 15–20% rates. Unlike classical scaling laws [9, 8] predicting absolute performance for homogeneous architectures, PS predicts relative gaps across coupled axes in heterogeneous models, extending to LLMs where DD maps to token subsampling and WAS/DS to width/depth scaling [15], reducing per-hypothesis compute and carbon footprint by ∼5–8×\sim5\text{--}8\times.

Across both evaluation models, PS achieves 79–88% compute reduction with noise-floor prediction accuracy and perfect rank preservation (ρ=1.0\rho= 1.0) among actionable hypotheses. Future directions include multi-scale configurations (M>1M > 1), extending to larger foundation models, adapting PS to online metrics beyond NE, and architecture-aware scaling ratio selection (motivated by DCN-v2 findings in section H.3). For newer architectures where regularity conditions are unverified, hybrid strategies combining PS screening with selective full-scale validation offer the best quality balance.

Acknowledgments

This paper was partially written with the assistance of large language models (LLMs) for text drafting, editing, and structural refinement. All technical content, experimental design, results, and scientific claims were produced and verified by the authors.

Additional Experimental Results

Hypothesis Categories

In large-scale recommendation systems, model development proceeds through hypotheses—atomic configuration changes applied to a baseline that target a specific modeling dimension. Each hypothesis modifies exactly one aspect of the model and falls into one of three categories:

  1. Architecture hypotheses alter the network topology or training procedure: layer fusion, precision changes, distillation loss modifications, attention mechanism changes, activation function swaps, normalization layer modifications, or residual connection restructuring.

  1. Feature hypotheses add, remove, or transform input features: deprecation of redundant sparse features, embedding table restructuring, feature crossing modifications, addition of new user or item signals, or changes to feature preprocessing pipelines.

  1. Data hypotheses modify the training data pipeline: sampling strategies, deduplication filters, data augmentation schemes, label smoothing, temporal windowing changes, or modifications to negative sampling ratios.

Evaluating hypotheses at full scale is the primary computational bottleneck in the model development cycle: each candidate change requires a complete training run to assess its impact on offline metrics. PS aims to replace these expensive full-scale runs with efficient proxy evaluations while preserving ranking fidelity among hypotheses.

Scaling Configurations

Table 5 summarizes the scaling function compositions selected for each model via the compute-optimal proxy selection procedure (§ 3.5).

ModelDDHSRESOtherConstraint
TS-1α=0.317\alpha=0.317α=0.5\alpha=0.5α=0.25\alpha=0.25—ΔTrainers≤−50%\Delta\mathrm{Trainers} \le-50\%
FM-1α=0.5\alpha=0.5α=0.2\alpha=0.2α=0.2\alpha=0.2FS 0.2, DS 0.5ΔSamples≤−50%\Delta\mathrm{Samples} \le-50\%

Table 5. Scaling configurations per model.

For TS-1, the composition of DD, HSR, and ES reflects the infrastructure constraint of halving the trainer count while preserving data distribution fidelity. For FM-1, aggressive scaling across five axes (DD, HSR, FS, ES, DS) achieves 79% compute reduction under a 50% data-reduction ceiling.

Extended Per-Head and Predictor Analysis

On TS-1, all three prediction strategies fall within τAA\tau_{\mathrm{AA}}, and the absolute improvement from raw proxy to EGP is modest (0.13 τAA\tau_{\mathrm{AA}} in mean, 0.12 τAA\tau_{\mathrm{AA}} in SE). Nonetheless, curve fitting adds meaningful value: the EGP improves Spearman ρ\rho from 0.755 to 0.8, tightening the ranking among near-neutral hypotheses, while simultaneously reducing prediction variance across the held-out pool. More importantly, the EGP’s primary advantage lies in its calibrated uncertainty estimates (§ 3.3), which enable automatic rejection of unreliable predictions—a capability absent from both the raw proxy and the linear predictor.

On FM-1, Table 4 reveals a striking contrast between prediction heads. The raw proxy is severely degraded on CTR heads (CTR-1: 2.15 τAA\tau_{\mathrm{AA}}, CTR-2: 1.75 τAA\tau_{\mathrm{AA}}) but already near the noise floor on CVR-1 (0.95 τAA\tau_{\mathrm{AA}}). The GP dramatically improves the CTR heads—reducing MAE by 67% and 57% respectively—while providing only a modest 10% improvement on CVR-1. This asymmetry suggests that the FM-1 scaling configuration (DD + HSR + FS + ES + DS; section A.2) introduces greater distortion for CTR-related representations than for CVR, possibly because CTR predictions rely more heavily on fine-grained embedding interactions that are disrupted by aggressive scaling across five axes. The GP’s empirical prior, learned from 9 training hypotheses, successfully corrects this distortion. The identical Spearman ρ=0.714\rho= 0.714 across all three GP-predicted heads reflects the same 7 held-out hypotheses being ranked in the same order regardless of prediction objective, indicating stable ranking structure. On the raw proxy, rank preservation is lower on CTR heads (ρ=0.607\rho= 0.607–0.6430.643) than on CVR-1 (ρ=0.714\rho= 0.714), mirroring the MAE pattern.

Confirming these patterns and extending the predictor comparison with a larger foundation model hypothesis pool remain important directions for future work.

Compute reduction

Compute Reduction⁡(θfull,θproxy)=1−(1+ΔTrainers)⋅(1+ΔSamples)1+ΔQPS(3)\operatorname{Compute\ Reduction}(\theta_{\mathrm{full}},\theta_{\mathrm{proxy}}) = 1 - \frac{(1+\Delta_{\mathrm{Trainers}})\cdot(1+\Delta_{\mathrm{Samples}})}{1+\Delta_{\mathrm{QPS}}} \tag*{(3)}

where θfull\theta_{\mathrm{full}} and θproxy\theta_{\mathrm{proxy}} denote the full-scale and proxy configurations, respectively, and ΔQPS\Delta_{\mathrm{QPS}}, ΔTrainers\Delta_{\mathrm{Trainers}}, and ΔSamples\Delta_{\mathrm{Samples}} are the relative percentage changes in throughput or Queries per Sample (QPS), trainer count, and sample volume between the two configurations.

This formula models total training compute as proportional to trainers ×\times samples/throughput: training computation scales with parallelism (trainer count) and data volume (samples), and inversely with per-device throughput (QPS). A decrease in QPS increases per-sample processing time, partially offsetting reductions in other axes. Since PS proxy runs are executed on the same hardware type as their corresponding full-scale baselines (table 2), the relative change in trainer count between proxy and full-scale configurations directly reflects the reduction in compute infrastructure, without confounding differences in per-device throughput.

Formal Definitions

Rank Preservation

We define the non-neutral hypothesis set as

I={i∈[N]∣∣Δi∣>τAA},(4)\mathcal{I} = \{ i \in[N] \mid|\Delta_i| > \tau_{\mathrm{AA}} \}, \tag*{(4)}

and compute rank correlation only over this set:

ρrank=Spearman⁡(rank⁡(Δ^I),rank⁡(ΔI)),(5)\rho_{\mathrm{rank}} = \operatorname{Spearman}(\operatorname{rank}(\widehat{\Delta}_{\mathcal{I}}), \operatorname{rank}(\Delta_{\mathcal{I}})), \tag*{(5)}

where Δ^I\widehat{\Delta}_{\mathcal{I}} and ΔI\Delta_{\mathcal{I}} denote the restriction of the predicted and ground-truth gap vectors to the indices in I\mathcal{I}. Hypotheses with ∣Δi∣≤τAA|\Delta_i| \le\tau_{\mathrm{AA}} are excluded because their gaps are indistinguishable from noise—any ranking among them is arbitrary, and including them would dilute the correlation signal with uninformative ties. A perfect system achieves ρrank=1\rho_{\mathrm{rank}} = 1. We note that low proxy error generally entails high rank correlation, but the converse does not hold: a predictor may preserve rankings while systematically distorting gap magnitudes.

Normalized Entropy (NE)

NE quantifies the predictive quality of a model relative to the empirical base-rate entropy:

NE=H(y^,y)H(p)=−1n∑k=1n[yklog⁡y^k+(1−yk)log⁡(1−y^k)]−[plog⁡p+(1−p)log⁡(1−p)](6)\mathrm{NE} = \frac{\mathcal{H}(\hat{y},y)}{\mathcal{H}(p)} = \frac{-\frac{1}{n}\sum_{k=1}^{n}[y_k\log\hat{y}_k + (1-y_k)\log(1-\hat{y}_k)]}{-[p\log p + (1-p)\log(1-p)]} \tag*{(6)}

where yk∈{0,1}y_k \in\{0,1\} is the binary label, y^k∈(0,1)\hat{y}_k \in(0,1) is the predicted probability, p=1n∑kykp = \frac{1}{n}\sum_k y_k is the marginal positive rate, and nn is the number of evaluation examples. A model that perfectly matches the base rate achieves NE = 1; lower values indicate superior predictive performance. NE is preferred over raw log-loss (binary cross-entropy) in industrial settings because it normalizes for variation in base rates across tasks, time periods, and data splits, thereby enabling meaningful comparison across heterogeneous model types and prediction heads. For multi-task models, we report NE independently for each prediction head to capture per-objective scaling behavior.

Scaling Function Details

This appendix provides implementation details for the scaling functions summarized in Table 1.

Quantized Scaling Operator

Many scaling functions reduce integer-valued parameters (embedding dimensions, hash sizes, layer widths). Naively multiplying by α\alpha and rounding can produce values that are misaligned with hardware (e.g., not multiples of 8 for tensor cores) or degenerate (e.g., embedding dimension of 0). We use a quantized scaling operator throughout:

QScale⁡(v,α,q,τ)={vif v<τ,q⋅⌈max⁡(⌊α⋅v⌋,1)q⌉otherwise.(7)\operatorname{QScale}(v,\alpha,q,\tau) = \begin{cases} v & \text{if } v < \tau, \\ q \cdot\left\lceil\frac{\max(\lfloor\alpha\cdot v\rfloor,1)}{q} \right\rceil& \text{otherwise}. \end{cases} \tag*{(7)}

where vv is the original value, qq is the alignment granularity (e.g., 8), and τ\tau is a minimum threshold below which values are left unchanged. This ensures scaled values are always positive, hardware-aligned integers.

Scaling Function Descriptions

Data Downsampling (DD) is the simplest scaling function: train on a uniformly sampled α\alpha-fraction of the data using a fixed hash seed for determinism. If the configuration already has a downsampling rate r0r_0, the effective rate becomes r0⋅αr_0 \cdot\alpha. Compute reduction is approximately linear in α\alpha, and because the data distribution is preserved, this is the most reliable scaling function for order preservation.

Hash-Size Reduction (HSR) targets the sparse embedding tables that dominate memory in recommendation models. Each categorical feature is mapped to an embedding table with hfh_f rows; hash-size reduction scales these to hf′=QScale⁡(hf,α,1,5000)h'_f = \operatorname{QScale}(h_f,\alpha,1,5000)—features with hash sizes below 5,000 are left unchanged, as further reduction degrades feature resolution. After scaling, all affected tables use explicit hashing to ensure consistent collision behavior. This primarily reduces memory footprint and enables fitting the model on fewer GPUs.

Embedding Scaling (ES) is complementary to HSR: instead of reducing the number of rows, it reduces the width of each embedding vector. Each embedding dimension is scaled as di′=QScale⁡(di,α,qi,10)d'_i = \operatorname{QScale}(d_i,\alpha,q_i,10). The alignment factor qiq_i must simultaneously satisfy memory alignment restrictions (di′ mod 4=0d'_i \bmod4 = 0), attention head compatibility (dmodel mod nheads=0d_{\mathrm{model}} \bmod n_{\mathrm{heads}} = 0), and quantization-specific requirements. We compute qiq_i as the least common multiple of these individual constraints. Embeddings using FP16 quantization are excluded from scaling entirely.

Compression invariant. In our architectures, embedding dimension and a compression factor satisfy demb×kcompress=dmodeld_{\mathrm{emb}} \times k_{\mathrm{compress}} = d_{\mathrm{model}}. After scaling, we must find a new (demb′,kcompress′)(d'_{\mathrm{emb}},k'_{\mathrm{compress}}) pair satisfying this invariant with demb′ mod 8=0d'_{\mathrm{emb}} \bmod8 = 0. We enumerate all valid pairs and select the one minimizing a weighted deviation that prioritizes preserving embedding dimension (which has greater impact on quality) over compression factor:

min⁡d′⋅k′=dmodel′2⋅∣d′−dembdemb∣+∣k′−kcompresskcompress∣.(8)\min_{d' \cdot k'=d'_{\mathrm{model}}} 2 \cdot\left|\frac{d'-d_{\mathrm{emb}}}{d_{\mathrm{emb}}}\right|+ \left|\frac{k'-k_{\mathrm{compress}}}{k_{\mathrm{compress}}}\right|. \tag*{(8)}

Feature Scaling (FS) prunes low-importance features using pre-computed feature importance scores, retaining the top ⌊α⋅F⌋\lfloor\alpha\cdot F\rfloor features. Two design choices handle the complexity of feature sets: Unified embedding groups. Features sharing a single embedding table are treated atomically—the group is kept or removed as a whole, based on its average importance, to avoid corrupting shared table structures.

Dependency resolution. Features derived from other features via hash transformations create dependency chains. When a feature is removed, all its dependents must also be removed. We resolve this via breadth-first traversal from the removed set. Sparse and dense features are scaled independently.

Whole Architecture Scaling (WAS) reduces all hidden dimensions simultaneously—embedding widths, dense layers, specialized sub-networks, task-specific heads, cross-network ranks, attention dimensions, and transformer token counts—by the same ratio α\alpha. Each architecture parameter pjp_j is scaled to max⁡(⌈α⋅pj⌉align,pjmin⁡)\max(\lceil\alpha\cdot p_j \rceil_{\mathrm{align}}, p_j^{\min}), rounding up to the nearest multiple of the alignment (typically 8) and enforcing a minimum value (typically 1). Because every dimension shrinks proportionally, compute reduction is approximately quadratic in α\alpha (i.e., rk(α)=α2r_k(\alpha) = \alpha^2).

The challenge is that architecture parameters are not independent. They are connected by structural constraints:

  • demb×kcompress=dmodeld_{\mathrm{emb}} \times k_{\mathrm{compress}} = d_{\mathrm{model}} (embedding-to-model dimension);

  • dmodelmod⁡nheads=0d_{\mathrm{model}} \mathbin{\operatorname{mod}} n_{\mathrm{heads}} = 0 (attention head divisibility);

  • Expert output dimensions must sum to a target (MoE layers).

These constraints form a dependency graph. A dependency resolver scales independent parameters first via topological ordering, then adjusts dependent ones to satisfy constraints.

Depth Scaling (DS) reduces the number of transformer layers by truncating the layout to the first ⌊α⋅L⌋\lfloor\alpha\cdot L \rfloor layers (minimum 1). The deepest layers are removed first, and all dependent configurations—prefetch schedules, expert-parallelism assignments, per-layer compressed embeddings, residual connections to removed layers, and per-layer auxiliary outputs—are pruned. In practice, we implement this as architecture-specific variants, one per model family, each handling its particular dependency structure. WAS and DS are orthogonal and can be composed.

Assumption Verification

Assumptions (A1)–(A3) from § 3 can be partially verified empirically for any given model family and scaling mechanism.

(A1) Convergence stability. We verify convergence by monitoring the NE trajectory of each training run: all runs (both proxy and full-scale) exhibit monotonically decreasing NE that plateaus within the final 10% of training steps. The standard deviation of terminal NE across repeated baseline runs defines τAA\tau_{\mathrm{AA}}, which is small relative to the hypothesis gaps of interest.

(A2) Gap transferability. The scatter plots in Figure 1 provide visual evidence of gap transferability: proxy gaps cluster tightly around the identity line, with residuals uncorrelated with gap magnitude. The low MAE values (<1.03 τAA< 1.03\,\tau_{\mathrm{AA}} across all models and heads) further support this assumption.

(A3) Monotonic signal preservation. For each scaling function, we compute the sign-reversal rate: the fraction of hypotheses for which sign⁡(Δi(j))≠sign⁡(Δi)\operatorname{sign}(\Delta_i^{(j)}) \ne\operatorname{sign}(\Delta_i). On TS-1, the selected proxy configuration (DD + HSR + ES) achieves a sign-reversal rate below 10%, concentrated among near-neutral hypotheses with ∣Δi∣<τAA|\Delta_i| < \tau_{\mathrm{AA}}. When a scaling mechanism exhibits high sign-reversal rates for actionable hypotheses, the framework provides a principled signal to exclude that mechanism from S\mathcal{S}.

Comparison of Scaling Functions

We compare two architecture scaling strategies: ES, which applies the scale ratio only to embedding-related dimensions while leaving dense layers, specialized sub-networks, task heads, and cross-network ranks at full capacity; and WAS, which applies the ratio uniformly to all dimension parameters.

ComponentESWAS
Embedding tables✓✓
User-history embeddings✓✓
Interaction dmodeld_{\mathrm{model}}✓✓
Dense layers—✓
Specialized sub-networks—✓
Task-specific heads—✓
Cross-network rank—✓
Attention tokens—✓

Table 7. Components scaled by ES and WAS.

NE fidelity and training efficiency. Table 8 compares ES and WAS at scaling ratio 0.5 on a large foundation model (≈100× TS-1 FLOPs, 1,024 trainers on NVIDIA B200). WAS reduces the NE gap from −0.0090 to −0.0056 (38% improvement) while delivering 25% higher QPS and 42.9% lower FLOPs per sample.

MetricESWASΔ\Delta
NE Gap−0.0090−0.005638%↓
QPS (k)194.9244.1+25.2%
FLOPs/sample (M)109,357.962,440.4−42.9%
FLOPs reduction vs. full68.4%82.0%—

Table 8. ES vs. WAS at α=0.5\alpha=0.5 on Foundation Model. NE gap: signed difference (negative = proxy underperforms).

Analysis. The fidelity gap reflects dimensional asymmetry in tightly coupled architectures. Under ES, embedding-adjacent dimensions shrink while downstream components remain at full width. For a foundation model whose architecture features deep multiplicative interactions between embeddings, specialized sub-networks, and attention mechanisms, this width mismatch disrupts learned feature interactions, inflating the NE gap. WAS eliminates this asymmetry by scaling all components uniformly. We adopt WAS for tightly coupled models (FM-1), and ES for simpler, embedding-dominated models (TS-1).

Compute Budget-Optimal Proxy Selection: Full Procedure

Design space. Let Φ={ϕ1,…,ϕK}\Phi= \{\phi_1,\ldots,\phi_K\} denote the set of KK candidate scaling functions (Table 1), and let A={α1,…,αL}A = \{\alpha_1,\ldots,\alpha_L\} denote a discrete grid of candidate scaling ratios (e.g., {0.1,0.2,…,0.9}\{0.1, 0.2,\ldots,0.9\}). A scaling configuration is a tuple c=((ϕk1,αl1),…,(ϕkm,αlm))c = ((\phi_{k_1},\alpha_{l_1}),\ldots,(\phi_{k_m},\alpha_{l_m})) specifying which functions to apply and at what ratios. The effective compute reduction is r(c)=∏t=1mrkt(αlt)r(c) = \prod_{t=1}^{m} r_{k_t}(\alpha_{l_t}), where each rk(α)r_k(\alpha) is given in Table 1. Let G={g1,…,gP}G = \{g_1,\ldots,g_P\} denote the set of candidate prediction models (linear regression and EGP as described in § 3.3). The full design space is C×G\mathcal{C} \times G.

Benchmark corpus. Selection is performed using a benchmark corpus B={(P0,P1,…,PN)}B = \{(P_0,P_1,\ldots,P_N)\} of hypotheses for which ground-truth full-scale gaps Δ=(Δ1,…,ΔN)\Delta= (\Delta_1,\ldots,\Delta_N) are known. This corpus is accumulated over time from experiments that have been trained to convergence. The complete procedure is given in algorithm 1.

Practical considerations. The full enumeration in algorithm 1 is tractable because: (i) the number of scaling function families KK is small, (ii) the ratio grid AA is coarse (typically 5–9 values), and (iii) we limit composition depth to m≤Mmax⁡=3m \le M_{\max}=3. The dominant compute is Stage 2—training the scaled models on the benchmark corpus—which is a one-time offline compute investment. Once the observation matrix DD is computed, Stage 3 (model fitting and evaluation) is computationally negligible. We further reduce the search space via compute-tier filtering: we retain only configurations whose compute reduction falls within a target range

Algorithm 1 Joint Selection of Scaling Configuration and Prediction Model

Require: Scaling function families Φ={ϕ1,…,ϕK}\Phi= \{\phi_1, \ldots, \phi_K\}; ratio grid A={α1,…,αL}A = \{\alpha_1, \ldots, \alpha_L\}; prediction model candidates G={g1,…,gP}G = \{g_1, \ldots, g_P\}; benchmark corpus B\mathcal{B} with ground-truth gaps Δ\Delta; budget BB

Ensure: Optimal scaling configuration set S∗S^* and prediction model g∗g^*

  1. Stage 1: Enumerate feasible scaling configurations

  2. C←∅\mathcal{C} \leftarrow\emptyset

  3. for each composition (ϕk1,αl1),…,(ϕkm,αlm)(\phi_{k_1}, \alpha_{l_1}), \ldots, (\phi_{k_m}, \alpha_{l_m}) with m≤Mmax⁡m \le M_{\max} do

  4. c←((ϕk1,αl1),…,(ϕkm,αlm))c \leftarrow((\phi_{k_1}, \alpha_{l_1}), \ldots, (\phi_{k_m}, \alpha_{l_m}))

  5. if (N+1)⋅r(c)⋅Cfull≤B(N + 1) \cdot r(c) \cdot C_{\mathrm{full}} \le B then

  6. C←C∪{c}\mathcal{C} \leftarrow\mathcal{C} \cup\{c\}

  7. end if

  8. end for

  9. Stage 2: Run scaled experiments on benchmark corpus

  10. for each c∈Cc \in\mathcal{C} do

  11. Apply scaling configuration cc to all hypotheses in B\mathcal{B}

  12. Train P0(c),P1(c),…,PN(c)\mathcal{P}_0^{(c)}, \mathcal{P}_1^{(c)}, \ldots, \mathcal{P}_N^{(c)} to convergence

  13. Compute scaled gaps: Δi(c)=(ℓi(c)−ℓ0(c))/ℓ0(c)\Delta_i^{(c)} = (\ell_i^{(c)} - \ell_0^{(c)})/\ell_0^{(c)} for all i∈[N]i \in[N]

  14. end for

  15. Stage 3: Train and evaluate prediction models

  16. best_score←∞\mathrm{best\_score} \leftarrow\infty

  17. for each subset S⊆CS \subseteq\mathcal{C} with ∣S∣=M|S| = M do

  18. Construct observation matrix D∈RN×M\mathbf{D} \in\mathbb{R}^{N \times M} from SS

  19. for each prediction model gp∈Gg_p \in G do

  20. Fit fθ←gp(D,Δ)f_\theta\leftarrow g_p(\mathbf{D}, \Delta) via cross-validation on B\mathcal{B}

  21. Compute predictions: Δ^i=fθ(di)\hat{\Delta}_i = f_\theta(d_i) for all i∈[N]i \in[N]

  22. Evaluate: E=1N∑i=1N∣Δ^i−Δi∣\mathcal{E} = \frac{1}{N}\sum_{i=1}^{N}|\hat{\Delta}_i - \Delta_i|

  23. if E<best_score\mathcal{E} < \mathrm{best\_score} then

  24. S∗←SS^* \leftarrow S; g∗←gpg^* \leftarrow g_p; best_score←E\mathrm{best\_score} \leftarrow\mathcal{E}

  25. end if

  26. end for

  27. end for

  28. return (S∗,g∗)(S^*, g^*)

(e.g., 5×5\times–15×15\times), discarding configurations that are either too computationally expensive to provide meaningful savings or too aggressively scaled to retain useful signal.

Curve Fitting Details

Reference Curve Selection

Both linear regression and the EGP require selecting appropriate reference curves from the historical library. We apply two filters sequentially:

Gap trend filter. For each reference rr, we compute the relative NE gap with the target at each time step: gt=(ℓtarget(t)−ℓr(t))/ℓtarget(t)g_t = (\ell_{\mathrm{target}}(t) - \ell_r(t))/\ell_{\mathrm{target}}(t). The gap trend is the range of this relative gap (in percentage points): GapTrend⁡(r)=100⋅∣max⁡tgt−min⁡tgt∣\operatorname{GapTrend}(r) = 100 \cdot\lvert\max_t g_t - \min_t g_t \rvert. References exceeding a threshold δgap\delta_{\mathrm{gap}} are excluded—a large range indicates the reference and target have divergent dynamics.

Correlation filter. Among remaining references, we compute Pearson correlation with the target over the observed interval (skipping the first 20 time steps to avoid noisy early-training data). We keep only the top-K′K' by correlation, requiring ρ≥0.9999\rho\ge0.9999. If no reference meets this threshold, the prediction is rejected.

Why K′=1K' = 1? We find that using a single reference is more robust than larger bases. With K′>1K' > 1, OLS can overfit to noise in the reference-target alignment, especially when the observed window is short. The extremely high correlation threshold (0.9999) compensates—that this level, a single reference already provides an excellent basis.

Empirical Gaussian Process Details

Preprocessing. Training progress is min-max normalized to [0,1][0,1]. NE values are standardized using the global mean and standard deviation across all reference curves. Predictions are un-standardized before returning.

Empirical prior. The mean function is the pointwise average of RR historical curves (R=13R = 13 for TS-1, R=9R = 9 for FM-1, corresponding to training hypotheses), linearly interpolated to query points. The covariance is the sample covariance:

k(x,x′)=1R−1∑i=1R[yi(x)−μ(x)][yi(x′)−μ(x′)].(9)k(x,x') = \frac{1}{R-1}\sum_{i=1}^{R}[y_i(x)-\mu(x)][y_i(x')-\mu(x')]. \tag*{(9)}

This non-parametric kernel directly captures the correlation patterns in training curves without requiring manual kernel engineering.

Rejection criterion. The mean coefficient of variation CV=mean⁡(σ^/∣μ^∣)\mathrm{CV} = \operatorname{mean}(\hat{\sigma}/|\hat{\mu}|) over the prediction horizon serves as a confidence check. When CV exceeds a task-dependent threshold, the prediction is deemed unreliable.

Predictable Scaling on Public Data

To complement the evaluation and support reproducibility, we evaluate PS on four widely-used public recommendation model architectures—DLRM (Naumov et al., [14]), DeepFM (Guo et al., [6]), xDeepFM (Lian et al., [12]), and DCN-v2 (Wang et al., [16])—trained on the Criteo Terabyte dataset (Criteo Labs, [3]), a standard large-scale click-through rate prediction benchmark comprising approximately 4 billion labeled user interaction records with 13 integer and 26 categorical features. These four architectures span the dominant paradigms in industrial recommendation: DLRM couples sparse embedding lookups with dense feature interactions via dot products, DeepFM jointly models low-order feature interactions through a factorization machine and high-order interactions through a deep network, xDeepFM extends this with a Compressed Interaction Network (CIN) that learns explicit bounded-degree feature interactions at the vector level, and DCN-v2 introduces explicit cross-network layers that learn bounded-degree feature interactions alongside a deep network. For each model, we construct scaled-down proxies using the scaling functions described in § 3.2 and evaluate all three prediction strategies—raw proxy, linear predictor (LP), and Empirical Gaussian Process (EGP)—across six scaling configurations (Table 10–table 13). This public evaluation serves two purposes: it validates that PS generalizes beyond full scale model families to architecturally diverse public models, and it provides a fully reproducible experimental setup against which future work can be compared.

ConfigPredictorMAE (×τAA\times\tau_{AA})SE (×τAA\times\tau_{AA})Δ\DeltaMAESpearman ρ\rhoΔρ\Delta\rho
hash_onlyRaw proxy0.7310.731±0.617\pm0.617—0.4170.417—
hash_onlyLinear0.7310.731±0.617\pm0.6170%0\%0.4170.4170.0000.000
hash_onlyGP (EGP)0.7280.728±0.479\pm0.4790%0\%0.4170.4170.0000.000
emb_onlyRaw proxy1.1741.174±0.494\pm0.494—−0.050-0.050—
emb_onlyLinear0.9910.991±0.875\pm0.875−16%-16\%−0.050-0.0500.0000.000
emb_onlyGP (EGP)0.9470.947±0.771\pm0.771−19%-19\%−0.050-0.0500.0000.000
data_dsRaw proxy0.7810.781±1.056\pm1.056—0.5330.533—
data_dsLinear0.5230.523±0.397\pm0.397−33%-33\%0.5330.5330.0000.000
data_dsGP (EGP)0.4910.491±0.407\pm0.407−37%-37\%0.7830.783+0.250+0.250
hash_embRaw proxy1.3491.349±1.474\pm1.474—−0.050-0.050—
hash_embLinear1.2191.219±1.217\pm1.217−10%-10\%−0.050-0.0500.0000.000
hash_embGP (EGP)1.2191.219±1.217\pm1.217−10%-10\%−0.050-0.0500.0000.000
hash_dataRaw proxy4.5674.567±0.901\pm0.901—0.4830.483—
hash_dataLinear0.9210.921±1.051\pm1.051−80%-80\%0.4830.4830.0000.000
hash_dataGP (EGP)0.8930.893±1.055\pm1.055−80%-80\%0.6500.650+0.167+0.167
all_scaledRaw proxy0.9700.970±0.545\pm0.545—0.7500.750—
all_scaledLinear0.8520.852±0.805\pm0.805−12%-12\%0.7500.7500.0000.000
all_scaledGP (EGP)0.8520.852±0.805\pm0.805−12%-12\%0.7500.7500.0000.000

Table 10. Public DLRM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of τAA\tau_{AA}. Δ\DeltaMAE: relative % change vs. raw proxy; Δρ\Delta\rho: absolute difference vs. raw proxy.

Hypothesis Design

To evaluate PS on public models, we design a pool of 27 hyperparameter hypotheses applied to a shared baseline configuration: embedding dimension demb=16d_{\mathrm{emb}} = 16, hidden layers [256][256], dropout rate 0.3, and learning rate 10−310^{-3}. The hypotheses span five categories that systematically probe distinct modeling dimensions:

  1. Embedding dimension sweep (7 hypotheses). Vary the sparse embedding dimension demb∈{2,4,8,16,32,64,128}d_{\mathrm{emb}} \in\{2, 4, 8, 16, 32, 64, 128\}, controlling the representational capacity allocated to each categorical feature while holding all other hyperparameters at baseline. Category: Feature Hypothesis.

  1. DNN width sweep (7 hypotheses). Vary the hidden layer widths from a single narrow layer ([32])([32]) through two-layer ([64,32],[128,64])([64, 32], [128, 64]), three-layer ([256,128,64],[512,256,128])([256, 128, 64], [512, 256, 128]), and four-layer ([1024,512,256,128],[1024,1024,512,256])([1024, 512, 256, 128], [1024, 1024, 512, 256]) configurations, jointly scaling width and depth of the dense interaction network. Category: Architecture Hypothesis.

  1. DNN depth sweep (5 hypotheses). Fix the per-layer width at 256 and vary only depth: [256]×k[256]^{\times k} for k∈{1,2,4,6,8}k \in\{1, 2, 4, 6, 8\}, isolating the effect of network depth from width. Category: Architecture Hypothesis.

  1. Joint embedding–architecture sweep (4 hypotheses). Co-vary embedding dimension and DNN configuration simultaneously, ranging from a minimal model (demb=2d_{\mathrm{emb}}=2, [32]) through intermediate configurations (demb=4d_{\mathrm{emb}}=4, [64, 32]; demb=64d_{\mathrm{emb}}=64, [512, 256, 128]) to a maximal model (demb=128d_{\mathrm{emb}}=128, [1024, 512, 256, 128]), testing how sparse and dense capacity interact when scaled together. Category: Architecture Hypothesis.

  1. Regularization and training (4 hypotheses). Vary dropout rate ∈{0.0,0.5,0.8}\in\{0.0, 0.5, 0.8\} (baseline: 0.3) and learning rate (10−210^{-2}; baseline: 10−310^{-3}), probing sensitivity to training-time hyperparameters that do not alter model architecture. Category: Architecture Hypothesis: Training Hyperparameters.

Effect size distribution. The hypothesis pool exhibits a diverse range of effect sizes: six hypotheses are neutral, nine exhibit weak effects, and twelve show strong effects. This distribution ensures that the PS predictor is evaluated across the full spectrum of hypothesis magnitudes—from subtle, near-threshold changes that stress the predictor’s sensitivity, to large-effect changes where correct ranking is most actionable. We categorize hypotheses into three actionability tiers based on NE gap magnitude relative to the empirical noise floor: neutral (∣Δ∣<0.55%|\Delta| < 0.55\%), weak (0.55–1.5%), and strong (> 1.5%).

Train–test split. The 27 hypotheses are partitioned into 17 training and 10 held-out evaluation hypotheses via stratified sampling, ensuring each effect category (neutral, weak, strong) is represented in both sets. This prevents the predictor from being evaluated exclusively on easy (strong-signal) or hard (near-neutral) hypotheses.

Scaling Configurations

Unlike the full scale models (TS-1, FM-1), where proxy compute budgets are constrained by infrastructure-specific limits (trainer counts, QPS, sample volumes), the public benchmark operates under a simpler objective: reducing the overall training computation. Since no infrastructure constraints apply, we select multiple scaling configurations at a uniform factor of 0.5 along different resource axes, enabling a systematic comparison of how each scaling dimension affects proxy fidelity. The six configurations (Table 9) span single-axis and multi-axis scaling:

ConfigHSRESDD
hash_only0.51.01.0
emb_only1.00.51.0
data_downsample1.01.00.5
hash_emb0.50.51.0
hash_data0.51.00.5
all_scaled0.50.50.5

Table 9. Scaling configurations for public benchmark evaluation. Each configuration scales the indicated dimensions by a factor of 0.5, targeting overall training compute reduction.

By varying which dimensions are scaled, we isolate the contribution of each resource axis to proxy fidelity, and by including multi-axis compositions (hash_emb, hash_data, all_scaled), we test whether combining scaling dimensions introduces interaction effects that degrade prediction quality. The uniform 0.5 ratio across all configurations simplifies interpretation and ensures that observed differences in proxy fidelity are attributable to the choice of scaling dimensions rather than differing reduction magnitudes.

Cross-Model Results and Analysis

Table 10–table 13 present prediction accuracy across all four public models, six scaling configurations, and three predictors. Figure 2–figure 5 show the corresponding predicted-vs.-actual scatter plots. We organize the analysis around the paper’s three core claims, identifying which results strengthen each claim and where limitations emerge.

Hash-only scaling

(a) Hash-only scaling

Embedding-only scaling

(b) Embedding-only scaling

Data downsampling

(c) Data downsampling

Hash + embedding scaling

(d) Hash + embedding scaling

Hash + data scaling

(e) Hash + data scaling

All dimensions scaled

(f) All dimensions scaled

Figure 2. Predicted vs. actual accuracy gap for six scaling configurations on the public DLRM benchmark. Each point represents one hypothesis; the dashed line indicates perfect prediction. Tighter clustering around the diagonal indicates higher proxy fidelity.

GP Predictor Value

Consistent MAE recovery across architectures. The GP predictor consistently reduces MAE relative to the raw proxy, with the largest improvements occurring precisely when the raw proxy is most degraded—the regime where PS adds the most practical value. Across all four models, when the raw proxy exceeds 1τAA1 \tau_{\mathrm{AA}}, the GP reduces MAE by 37–80%:

  • DLRM hash_data: 4.567→0.893τAA4.567 \to0.893 \tau_{\mathrm{AA}} (−80%; Figure 2e)

  • DCN-v2 hash_data: 1.919→0.559τAA1.919 \to0.559 \tau_{\mathrm{AA}} (−71%; Figure 5e)

  • DCN-v2 data_ds: 1.821→0.819τAA1.821 \to0.819 \tau_{\mathrm{AA}} (−55%; Figure 5c)

  • xDeepFM all_scaled: 1.170→0.541τAA1.170 \to0.541 \tau_{\mathrm{AA}} (−54%; Figure 4f)

  • xDeepFM hash_data: 0.704→0.332τAA0.704 \to0.332 \tau_{\mathrm{AA}} (−53%; Figure 4e)

  • DeepFM hash_data: 0.835→0.448τAA0.835 \to0.448 \tau_{\mathrm{AA}} (−46%; Figure 3e)

  • DLRM data_ds: 0.781→0.491τAA0.781 \to0.491 \tau_{\mathrm{AA}} (−37%; Figure 2c)

Predicted versus actual accuracy gap for six scaling configurations on the public DeepFM benchmark, shown in six rows of three scatter plots

Figure 3. (a) Hash-only scaling; (b) Embedding-only scaling; (c) Data downsampling; (d) Hash + embedding scaling; (e) Hash + data scaling; (f) All dimensions scaled. Predicted vs. actual accuracy gap for six scaling configurations on the public DeepFM benchmark. Each point represents one hypothesis; the dashed line indicates perfect prediction. Tighter clustering around the diagonal indicates higher proxy fidelity.

Predicted vs. actual accuracy gap for six scaling configurations

Figure 4. Predicted vs. actual accuracy gap for six scaling configurations on the public xDeepFM benchmark. Each point represents one hypothesis; the dashed line indicates perfect prediction. Tighter clustering around the diagonal indicates higher proxy fidelity.

Predicted versus actual accuracy gap for six scaling configurations on the public DCN-v2 benchmark

Figure 5. (a) Hash-only scaling; (b) Embedding-only scaling; (c) Data downsampling; (d) Hash + embedding scaling; (e) Hash + data scaling; (f) All dimensions scaled. Predicted vs. actual accuracy gap for six scaling configurations on the public DCN-v2 benchmark. Each point represents one hypothesis; the dashed line indicates perfect prediction. Tighter clustering around the diagonal indicates higher proxy fidelity.

This pattern—the GP providing the largest improvements where the raw proxy is worst—directly validates the finding from § 4 that curve fitting adds the most value when the raw signal is noisy, and least when the scaling function alone already satisfies (A2)–(A3). For configurations where the raw proxy is already accurate (e.g., DeepFM hash_only: 0.381τAA0.381 \tau_{\mathrm{AA}}; DLRM hash_only: 0.731τAA0.731 \tau_{\mathrm{AA}}), the GP provides negligible or modest additional benefit, as expected.

The GP also improves rank preservation (ρ\rho) in several cases where it matters most: DLRM data_ds (+0.250), DeepFM all_scaled (+0.242), DLRM hash_data (+0.167), and xDeepFM hash_data (+0.063).

Linear predictor limitations. The linear predictor adds value on DLRM (improving MAE on 5/6 configurations) but is identical to the raw proxy on 5/6 configurations for both xDeepFM and DCN-v2. This suggests that the gap transfer function f∗f^{*} (§ 3) is approximately affine for DLRM’s simple dot-product interactions but nonlinear for models with explicit feature-crossing layers (CIN in xDeepFM, cross-network in DCN-v2). Rather than weakening the case for PS, this validates the GP as the default predictor choice: the GP subsumes the linear predictor and additionally captures nonlinear gap relationships through its empirical prior (§ 3.3).

Scaling Function Effectiveness and Architecture Dependence

Best configurations vary by architecture. No single scaling function dominates across all four models, directly validating the budget-optimal proxy selection procedure (algorithm 1, § 3.5):

ModelBest config (GP)GP MAE (×τAA\times\tau_{\mathrm{AA}})GP ρ\rho
DLRMdata_ds0.4910.783
DeepFMhash_only0.2850.939
xDeepFMhash_data0.3320.745
DCN-v2hash_emb0.4300.515

Table 2.

This architecture dependence is a central finding: blindly applying a fixed scaling configuration across models would select a suboptimal proxy for at least three of four architectures. The PS framework addresses this through per-model selection, and the public benchmark provides empirical evidence that this selection is necessary.

Strong proxy fidelity is achievable. Under the best GP configuration, all four models achieve MAE well below the noise floor: 0.2850.285–0.491τAA0.491 \tau_{\mathrm{AA}} for DLRM, DeepFM, and xDeepFM, and 0.430τAA0.430 \tau_{\mathrm{AA}} for DCN-v2. The top-performing configurations also achieve meaningful rank preservation (ρ=0.515\rho= 0.515–0.9390.939). These results demonstrate that PS generalizes beyond our full scale models to architecturally diverse public models at 50% compute reduction per scaling axis.

Data downsampling: robust on DLRM, variable elsewhere. In the evaluation (§ 4), data downsampling (DD) was the most reliable scaling function for order preservation. On DLRM, this finding replicates strongly (GP: ρ=0.783\rho= 0.783, MAE = 0.491 τAA0.491\,\tau_{\mathrm{AA}}). However, DD achieves only moderate rank preservation on DeepFM (ρ=0.430\rho= 0.430) and DCN-v2 (ρ=0.430\rho= 0.430), and near-random rankings on xDeepFM (ρ=0.182\rho= 0.182). We attribute this discrepancy to dataset scale: full scale models train on O(1010)O(10^{10})–O(1011)O(10^{11}) examples, where 50% downsampling still retains billions of samples with minimal distributional shift. On the smaller Criteo dataset, 50% downsampling may remove sufficient signal to degrade proxy fidelity for architectures with complex feature interactions (CIN layers in xDeepFM, cross-network layers in DCN-v2). This highlights that DD’s robustness is scale-dependent, and practitioners should verify DD effectiveness on their specific data volume.

Regularity Condition Violations and Failure Modes

Embedding scaling violates (A3) across architectures. Embedding-only scaling (ES) produces poor rank preservation on all four models: DLRM (ρ=−0.050\rho= -0.050), DeepFM (ρ=0.115\rho= 0.115), xDeepFM (ρ=0.518\rho= 0.518), and DCN-v2 (ρ=0.224\rho= 0.224 raw, 0.3330.333 GP). Even the GP cannot repair the rank ordering when embedding scaling is the sole scaling axis—the (A3) violation is structural rather than correctable by curve fitting. This contrasts with the results where ES performs well on TS-1, and likely reflects that full scale models’ deeper interaction networks are more robust to embedding dimension changes than these public architectures’ simpler structures. The failure is detectable during the proxy selection phase (algorithm 1), where configs with poor benchmark-corpus ρ\rho are automatically excluded.

Multi-axis compositions can degrade fidelity. Combining scaling dimensions sometimes introduces interaction effects that violate (A3) more severely than either axis alone:

  • DeepFM hash_emb (ρ=−0.491\rho= -0.491) combines hash-only (ρ=0.939\rho= 0.939) and emb-only (ρ=0.115\rho= 0.115)—the composition is dramatically worse than either component. Simultaneously reducing embedding table rows and embedding vector widths creates a compounding distortion in the factorization machine layer.

  • DeepFM all_scaled (ρ=−0.442\rho= -0.442 raw) is similarly degraded, though the GP partially recovers (ρ=−0.200\rho= -0.200).

These interaction effects validate the PS design principle that scaling functions should not be composed blindly: the compute budget-optimal selection procedure (§ 3.5) evaluates compositions on a benchmark corpus and rejects those that degrade proxy fidelity.

DCN-v2 presents the hardest case. DCN-v2 is the most challenging architecture for PS in this benchmark. Hash-only scaling produces essentially random rankings (ρ=0.006\rho= 0.006) despite achieving reasonable MAE (0.653 τAA0.653\,\tau_{\mathrm{AA}} under GP), indicating that MAE and rank preservation can diverge—a systematic bias can produce low MAE while scrambling relative ordering. The best GP ρ\rho across all configs is 0.515(hashemb)0.515 (hash_emb), substantially below the other models. On all_scaled, the GP MAE (1.063 τAA1.063\,\tau_{\mathrm{AA}}) exceeds the noise floor, and the GP slightly degrades ρ\rho on both hash_data (0.321→0.2730.321 \to0.273) and all_scaled (0.491→0.4790.491 \to0.479).

We attribute DCN-v2’s difficulty to its cross-network layers, which create tightly coupled multiplicative interactions across all feature dimensions. The uniform α=0.5\alpha= 0.5 scaling used in this benchmark may be too aggressive for DCN-v2’s cross-network rank—analogous to the finding in section E that tightly coupled full scale models (FM-1) require WAS rather than ES. This motivates architecture-aware scaling ratio selection as future work: rather than applying a uniform factor, the scaling ratio could be adapted based on architectural coupling strength.

Cross-Architecture Synthesis

The public benchmark validates the paper’s three core claims while revealing important boundary conditions:

Validated claims.

  1. GP consistently improves over raw proxy. The GP reduces MAE on all 24 model×\timesconfig pairs, with improvements ranging from 3% (xDeepFM hash_only) to 80% (DLRM hash_data).

  1. Per-model scaling function selection is essential. The best configuration differs across all four architectures (data_ds, hash_only, hash_data, hash_emb), and applying any single configuration universally would select a suboptimal proxy for at least three models.

  1. Regularity violations are detectable. Configurations that violate (A3) (negative ρ\rho) are identifiable on the benchmark corpus and would be excluded by algorithm 1, preventing their use on new hypotheses.

Boundary conditions and limitations.

  1. Architecture sensitivity. Models with tightly coupled feature interactions (DCN-v2’s cross-network, xDeepFM’s CIN) are harder to scale than models with simpler interaction structures (DLRM). The GP cannot fully compensate when the scaling function itself destroys the signal.

  1. Embedding scaling is unreliable. ES produces poor rank preservation across all four public architectures, in contrast to its effectiveness on full scale models. This suggests ES is best suited for models with deep, redundant embedding representations rather than the relatively compact embeddings in these public benchmarks.

  1. Data downsampling is scale-dependent. DD’s robustness on data (O(1010)O(10^{10}) examples) does not fully transfer to the smaller Criteo dataset, where 50% downsampling may remove too much signal for architectures with complex feature interactions.

  1. Linear predictor has narrow applicability. The linear predictor is effective only when the gap transfer function is approximately affine (DLRM), and is superseded by the GP on all architectures—supporting the GP as the default predictor choice.

Taken together, these results demonstrate that PS generalizes across four architecturally diverse public models, with the GP predictor and per-model scaling function selection jointly achieving GP MAE of 0.285–0.541 τAA\tau_{\mathrm{AA}} on the best configuration for three of four models. The fourth (DCN-v2) achieves 0.430 τAA\tau_{\mathrm{AA}} MAE but with limited rank preservation (ρ=0.515\rho= 0.515), identifying architecture-aware scaling ratios as a key direction for future work.

ConfigPredictorMAE (×τAA)(\times\tau_{\mathrm{AA}})SE (×τAA)(\times\tau_{\mathrm{AA}})Δ\DeltaMAESpearman ρ\rhoΔρ\Delta\rho
hash_onlyRaw proxy0.381±0.303\pm0.303—0.939—
hash_onlyLinear0.381±0.303\pm0.3030%0.9390.000
hash_onlyGP (EGP)0.285±0.304\pm0.304−25%0.9390.000
emb_onlyRaw proxy0.848±0.487\pm0.487—0.115—
emb_onlyLinear0.848±0.487\pm0.4870%0.1150.000
emb_onlyGP (EGP)0.795±0.460\pm0.460−6%0.1150.000
data_dsRaw proxy0.883±0.569\pm0.569—0.430—
data_dsLinear0.754±0.403\pm0.403−15%0.4300.000
data_dsGP (EGP)0.641±0.327\pm0.327−27%0.4300.000
hash_embRaw proxy0.944±0.628\pm0.628—−0.491—
hash_embLinear0.944±0.628\pm0.6280%−0.4910.000
hash_embGP (EGP)0.944±0.628\pm0.6280%−0.4910.000
hash_dataRaw proxy0.835±0.486\pm0.486—0.624—
hash_dataLinear0.497±0.394\pm0.394−40%0.6240.000
hash_dataGP (EGP)0.448±0.309\pm0.309−46%0.6240.000
all_scaledRaw proxy0.995±0.724\pm0.724—−0.442—
all_scaledLinear1.008±0.505\pm0.505+1%−0.4420.000
all_scaledGP (EGP)0.780±0.497\pm0.497−22%−0.200+0.242

Table 11. Table 11 Public DeepFM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of τAA\tau_{\mathrm{AA}}. Δ\DeltaMAE: relative % change vs. raw proxy; Δρ\Delta\rho: absolute difference vs. raw proxy.

ConfigPredictorMAE (×τAA\times\tau_{\mathrm{AA}})SE (×τAA\times\tau_{\mathrm{AA}})Δ\DeltaMAESpearman ρ\rhoΔρ\Delta\rho
hash_onlyRaw proxy0.814±1.575\pm1.575—0.836—
Linear0.814±1.575\pm1.5750%0.8360.000
GP (EGP)0.791±1.609\pm1.609−3%0.8360.000
emb_onlyRaw proxy0.699±0.517\pm0.517—0.518—
Linear0.699±0.517\pm0.5170%0.5180.000
GP (EGP)0.564±0.435\pm0.435−19%0.5180.000
data_dsRaw proxy1.038±0.985\pm0.985—0.100—
Linear1.038±0.985\pm0.9850%0.1000.000
GP (EGP)0.993±1.568\pm1.568−4%0.182+0.082
hash_embRaw proxy0.735±0.737\pm0.737—0.718—
Linear0.735±0.737\pm0.7370%0.7180.000
GP (EGP)0.642±0.806\pm0.806−13%0.700−0.018
hash_dataRaw proxy0.704±0.748\pm0.748—0.682—
Linear0.704±0.748\pm0.7480%0.6820.000
GP (EGP)0.332±0.196\pm0.196−53%0.745+0.063
all_scaledRaw proxy1.170±0.934\pm0.934—0.736—
Linear0.812±0.591\pm0.591−31%0.7360.000
GP (EGP)0.541±0.346\pm0.346−54%0.745+0.009

Table 12. Table 12 Public xDeepFM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of τAA\tau_{\mathrm{AA}}. Δ\DeltaMAE: relative % change vs. raw proxy; Δρ\Delta\rho: absolute difference vs. raw proxy.

ConfigPredictorMAE (×τAA\times\tau_{\mathrm{AA}})SE (×τAA\times\tau_{\mathrm{AA}})Δ\DeltaMAESpearman ρ\rhoΔρ\Delta\rho
hash_onlyRaw proxy0.686±0.706\pm0.706—0.006—
hash_onlyLinear0.686±0.706\pm0.7060%0.0060.000
hash_onlyGP (EGP)0.653±0.618\pm0.618−5%0.0060.000
emb_onlyRaw proxy0.581±0.323\pm0.323—0.224—
emb_onlyLinear0.581±0.323\pm0.3230%0.2240.000
emb_onlyGP (EGP)0.545±0.302\pm0.302−6%0.333+0.109
data_dsRaw proxy1.821±1.755\pm1.755—0.370—
data_dsLinear1.821±1.755\pm1.7550%0.3700.000
data_dsGP (EGP)0.819±1.091\pm1.091−55%0.430+0.060
hash_embRaw proxy0.520±0.266\pm0.266—0.467—
hash_embLinear0.520±0.266\pm0.2660%0.4670.000
hash_embGP (EGP)0.430±0.324\pm0.324−17%0.515+0.048
hash_dataRaw proxy1.919±1.240\pm1.240—0.321—
hash_dataLinear0.943±0.865\pm0.865−51%0.3210.000
hash_dataGP (EGP)0.559±0.667\pm0.667−71%0.273−0.048
all_scaledRaw proxy1.190±0.784\pm0.784—0.491—
all_scaledLinear1.190±0.784\pm0.7840%0.4910.000
all_scaledGP (EGP)1.063±0.913\pm0.913−11%0.479−0.012

Table 13. Table 13 Public DCN-v2 prediction accuracy across six scaling configurations. MAE and SE reported as multiples of τAA\tau_{\mathrm{AA}}. Δ\DeltaMAE: relative % change vs. raw proxy; Δρ\Delta\rho: absolute difference vs. raw proxy.

References

  1. [1]Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Understanding scaling laws for recommendation models. arXiv preprint arXiv:2208.08489, 2022.
  2. [2]Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker’s guide to scaling law estimation. arXiv preprint arXiv:2410.11840, 2025.arxiv.org/abs/2410.11840
  3. [3]Criteo Labs. Criteo terabyte click logs dataset. https://huggingface.co/datasets/criteo/CriteoClickLogs/, 2014.
  4. [4]Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pages 1437–1446, 2018.arxiv.org/abs/1807.01774
  5. [5]Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024.
  6. [6]Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: A factorization-machine based neural network for CTR prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1725–1731, 2017.arxiv.org/abs/1703.04247
  7. [7]Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, and Joaquin Quinonero-Candela. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pages 1–9, 2014.
  8. [8]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. Advances in Neural Information Processing Systems, 35:30016–30030, 2022.arxiv.org/abs/2203.15556
  9. [9]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  10. [10]Dongwoo Lee, Dong Bok Lee, Steven Adriaansen, Juho Lee, Sung Ju Hwang, Frank Hutter, Seon Joo Kim, and Hae Beom Lee. Bayesian neural scaling law extrapolation with prior-data fitted networks. arXiv preprint arXiv:2505.23032, 2025.
  11. [11]Lisha Li, Kevin Jamieson, Giulia DeSalvo, Aditya Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1–52, 2018.
  12. [12]Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1754–1763, 2018.
  13. [13]Jihao Andreas Lin, Sebastian Ament, Louis C. Tiao, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Empirical gaussian processes. arXiv preprint arXiv:2602.12082, 2026.arxiv.org/abs/2602.12082
  14. [14]Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaram, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, et al. Deep learning recommendation model for personalization and recommendation systems. arXiv preprint arXiv:1906.00091, 2019.arxiv.org/abs/1906.00091
  15. [15]Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022.
  16. [16]Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021, pages 1785–1797, 2021.
  17. [17]Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.

Paper details

vlnc:W_jysfx2ru
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

large language models (LLMs)

View statement
Looking for related papers…
Contents