betaFeedback

WorksComputer Science

PREFACE: Separating Shared Corrections from Prefix Conditioning in PEFT

Abstract

Does input-conditioned parameter-efficient fine-tuning (PEFT) improve because it uses the current input, or because it adds a useful learned correction? We investigate this distinction with PREFACE, a bounded channel controller for PEFT-adapted projections that requires no test-time gradients. The controller reads observed context; in multiple-choice likelihood scoring, its state is independent of the candidate continuation. Our central study freezes IA3, tuned LoRA, and DoRA adapters and matches controller architecture, initialization, batches, and updates across current-prefix and learned-global controls. Across ARC-Challenge and SciQ, current-prefix control improves all six mean accuracies over frozen adapters, but none of its paired contrasts with learned-global control excludes zero. Replacing its corrections with their training-prefix mean changes accuracy by at most 0.34 percentage points; input-varying corrections contribute only 1.8-2.8% of correction RMS. On ARC test, frozen-adapter current/global controls score 25.68%/25.77%, distinct from the earlier jointly trained 23.1% to 26.4% gain over IA3. We derive and evaluate static compilation of fixed corrections, separating deployment simplification from statistical equivalence. Broader GPT-2, Pythia, and Qwen experiments show family-, task-, and scoring-dependent gains, with modest recent-model effects. The study provides an intervention-based evaluation of conditional PEFT: improvements over a static adapter do not by themselves demonstrate a benefit from input matching.

Introduction

Pretrained language models are usually deployed with fixed weights. Parameter-efficient fine-tuning (PEFT) methods such as LoRA [8] and Infused Adapter by Inhibiting and Amplifying Inner Activations (IA3; [12]) freeze the backbone and train compact adapters, but the learned adapters are typically static after training. Recent inference-time adaptation methods introduce more dynamic behavior. Transformer-Squared modulates singular components through task dispatch and expert vectors [20], and In-Place Test-Time Training updates MLP fast weights with a next-token-prediction objective during inference [4]. These methods suggest that MLP blocks are promising adaptation sites, but they also raise a practical question: how much context-dependent adaptation is possible without test-time gradient updates or large per-task states?

Our starting point is an attribution problem: comparing an input-conditioned adapter only with its static predecessor changes both parameterization and input dependence. Input-dependent strength might avoid under-correcting some inputs and over-correcting others, but a shared correction learned by the larger parameterization could also explain the gain. We separate these explanations with trained global controls and interventions on a fixed trained controller. PREFACE learns a small bounded change in channel strength from the observed prefix, either jointly with the static scale or around a frozen adapted projection. The residual parameterization exactly recovers the static adapter when the controller output is zero and prevents the controller from generating an unconstrained replacement adapter.

For example, a static IA3 adapter uses the same scale vector a0a_0 for every instruction, question, or preceding text context. PREFACE compresses the observed context into c(q)c(q) and uses a(q)=a0+ρtanh⁡(Bc(q))a(q) = a_0 + \rho\tanh(Bc(q)). The resulting scale modulates subsequent inference; in our MCQ experiments it is reused while scoring all candidate answers, so candidates cannot influence it. For LoRA or DoRA, the evaluated wrapper rescales the input to the adapted projection rather than synthesizing a new low-rank matrix. This yields two distinct questions: does adding the correction help, and does assigning it to the matching input help beyond a shared correction? The second question is central to our study, not an assumption implicit in the method’s name.

Our contributions are:

  • We define a common, candidate-independent correction interface and a matched IA3/LoRA/DoRA study separating added correction, input matching, and boundedness. Frozen/global/current comparisons and same-checkpoint interventions are the primary evidence.

  • We quantify the difference between a variable controller representation and a functionally variable correction. Fixed-mean replacement and prediction transitions show why internal-state variation alone is insufficient evidence for useful input adaptation.

  • We derive static compilation of fixed corrections and evaluate its numerical fidelity and runtime. Broader backbone, scoring, and objective diagnostics delimit the findings rather than imply universal transfer.

Figure 1 gives a plain-language view of the shared prefix controller used in our experiments.

Figure 1. The frozen-adapter PREFACE wrapper. Observed context is encoded into a bounded correction applied to the input of an adapted projection, with the backbone and base adapter fixed. The construction supports subsequent token- or sequence-level inference. Our MCQ protocol additionally reuses one context state across candidates. Earlier jointly trained multiplicative variants are distinguished in § 3.

Related Work

Parameter-efficient adaptation. PEFT methods differ in where they attach task-specific state. Adapter modules insert small trainable bottlenecks between frozen Transformer components ([7]), prefix tuning and

prompt tuning learn continuous conditioning vectors while keeping the backbone fixed ([10, 9]), and BitFit updates only bias terms ([1]). LoRA learns low-rank additive updates to frozen weights ([8]). Recent LoRA variants improve this additive-update family through SVD-based initialization or weight decomposition, including PiSSA ([15]) and DoRA ([13]). IA3 instead learns multiplicative vectors that rescale inner activations, giving a very small parameter footprint ([12]). We use IA3 as a static multiplicative reference point, not as the definition of prefix-conditioned adaptation. PREFACE-IA3-shared instantiates the controller with residual channel scaling around the identity, while PREFACE-LoRA and PREFACE-DoRA test the same bounded-gating mechanism on tuned additive LoRA and DoRA adapters.

Input-conditioned and representation-side control. Several PEFT methods make adaptation depend on the input rather than using only a fixed adapter. Hypernetworks generate one model’s weights from another network’s conditioning signal ([6]). Within PEFT, Hyper-Adapter uses task or example summaries to generate adapter parameters or mixtures ([26]), while AdapterFusion composes trained adapters without changing their task-specific parameters ([16]). Representation finetuning methods such as ReFT/LoReFT intervene directly on hidden states rather than low-rank weight updates ([24]). Input-adaptive prompt methods such as ADePT ([22]) make prompt-side offsets depend on the input, while PARA generates prompt-aware representation adjustments inside Transformer layers for multi-tenant PEFT [14]. PEDRO similarly uses lightweight prompt-dependent vector generators to modify hidden representations through dot-product interactions [25]. PREFACE uses two design constraints to study this family of methods. First, the controller state is computed only from context observed before downstream target tokens, enabling causal prefix-conditioned control for subsequent token- or sequence-level inference. For MCQ likelihood scoring, that state is reused unchanged for every candidate continuation, preventing the adaptive state from seeing answer-specific tokens. Second, the controller changes only channel-wise gains inside MLP PEFT modules: the same prefix state can adjust IA3-style scales, tuned LoRA, tuned DoRA, spectral diagnostics, or compact representation-side interventions without changing the base adapter’s internal parameterization. This makes our intervention narrower than general representation editing while keeping the controller independent of a particular downstream scoring format and allowing a controlled test of transfer across PEFT families. We do not claim a new class of hypernetwork or superiority over these methods: our main comparison asks whether the input dependence of our controller is necessary once the added correction and training protocol are controlled.

Spectral and test-time adaptation. SVFT and SVFit adapt pretrained singular directions or values [11, 19]; Transformer-Squared uses spectral expert dispatch [20]. Test-time training updates model states using inference-time self-supervision [21, 4]. PREFACE uses a trained feed-forward controller with no inference-time gradients. MLP key-value interpretations [5] motivate our intervention site, but do not establish its performance mechanism. Our compact LoReFT- and PARA/PEDRO-style references share the local evaluation pipeline; they are not official-code reproductions. Appendix § 25 summarizes the design differences and spectral implementation.

Method

PREFACE learns bounded channel corrections from an observed context and applies them to a PEFT-adapted computation. The construction supports subsequent token- or sequence-level inference. We distinguish the controller’s input dependence, which follows from its definition, from whether matching a correction to a particular input improves performance, which requires an intervention experiment.

Bounded Controller

Let qℓq_\ell summarize the observed context at layer ℓ\ell. A shared small network produces coefficients cθ(qℓ)c_\theta(q_\ell), and a layer-specific basis BℓB_\ell maps them to a channel correction:

cθ(qℓ)=W2GELU⁡(W1LN⁡(qℓ)+b1)+b2,(1)c_\theta(q_\ell) = W_2\operatorname{GELU}(W_1\operatorname{LN}(q_\ell) + b_1) + b_2, \tag*{(1)}
δℓ(qℓ)=ρtanh⁡(Bℓcθ(qℓ)).(2)\delta_\ell(q_\ell) = \rho\tanh(B_\ell c_\theta(q_\ell)). \tag*{(2)}

Here qℓq_\ell is a mean of context-token activations, cθ(qℓ)∈Rmc_\theta(q_\ell) \in\mathbb{R}^{m}, Bℓ∈Rdℓ×mB_\ell\in\mathbb{R}^{d_\ell\times m}, and ρ>0\rho> 0 is a fixed bound. Thus each coordinate satisfies ∣δℓ,j∣≤ρ|\delta_{\ell,j}| \le\rho. The final controller layer is initialized to zero, recovering the static computation at initialization. The encoder and bases are learned on training examples; inference uses a forward pass without parameter updates. The same architecture also defines a learned-global control by replacing qℓq_\ell with a constant, and an unbounded control by removing tanh⁡\tanh.

Layer-wise order, without a cycle. Let xℓ,tx_{\ell,t} be the already-computed input to the adapted projection at token tt, before its current-layer correction, and let PP index observed-context tokens. We first compute qℓ=∣P∣−1∑t∈Pxℓ,tq_\ell= |P|^{-1}\sum_{t\in P} x_{\ell,t}, then δℓ(qℓ)\delta_\ell(q_\ell), and finally the corrected projection in Eq. (3). Earlier layers’ controls can affect xℓx_\ell, but the control at layer ℓ\ell is computed only after xℓx_\ell is available. There is no fixed-point iteration or dependence on that layer’s future output; sharing controller weights across layers adds no backward computational edge.

Application Across Adapter Families

Write Fϕ,ℓF_{\phi,\ell} for a projection containing a trained PEFT adapter. Our frozen-adapter input wrapper computes

yℓ=Fϕ,ℓ((1+δℓ(qℓ))⊙xℓ).(3)y_\ell= F_{\phi,\ell}\left((1+\delta_\ell(q_\ell))\odot x_\ell\right). \tag*{(3)}

All parameters ϕ\phi remain fixed. For an affine adapted projection Fϕ,ℓ(x)=Wϕ,ℓx+bℓF_{\phi,\ell}(x)=W_{\phi,\ell}x+b_\ell, its local change at a fixed activation is exactly

Δyℓ=Wϕ,ℓ(δℓ⊙xℓ).(4)\Delta y_\ell= W_{\phi,\ell}(\delta_\ell\odot x_\ell). \tag*{(4)}

Consequently, equal correction norms need not imply equal output changes: the activation and adapted projection also matter. We measure these changes rather than assuming that a particular adapter family must benefit. The coordinate bound implies ∥Δyℓ∥2≤ρ∥Wϕ,ℓ∥2∥xℓ∥2\lVert\Delta y_\ell\rVert_2 \le\rho\lVert W_{\phi,\ell}\rVert_2\lVert x_\ell\rVert_2, not a guarantee of improved predictions. If one fixed correction δˉℓ\bar{\delta}_\ell suffices, it can be folded into Wϕ,ℓdiag⁡(1+δˉℓ)W_{\phi,\ell}\operatorname{diag}(1+\bar{\delta}_\ell) without a dynamic controller; our fixed-mean intervention tests the accuracy consequence of this simplification.

For LoRA and DoRA, the evaluated input wrapper rescales the input to the complete adapted projection. It does not rescale only the additive low-rank branch. Its frozen weight is W+ΔWW+\Delta W for LoRA and the magnitude-normalized adapted weight for DoRA. A learned input-independent scale is additionally present in the earlier LoRA/DoRA runs. The matched study removes this extra scale and uses Eq. (3) identically across families.

The earlier multiplicative sweeps instead use

yℓ=Wℓ((a0,ℓ+δℓ(qℓ))⊙xℓ)+bℓ.(5)y_\ell= W_\ell\left((a_{0,\ell}+\delta_\ell(q_\ell))\odot x_\ell\right)+b_\ell. \tag*{(5)}

Their static IA3 scale a0,ℓa_{0,\ell} is jointly optimized in the main DistilGPT-2 runs and frozen in the two-stage gated variants. These runs establish empirical performance but do not alone isolate adapter family from training protocol. The new matched study freezes every base adapter and uses the same controller architecture, training schedule, and intervention site.

Representation-side instances rescale low-rank intervention values, while the spectral diagnostic rescales singular channels. These are alternative intervention sites, not evidence that every site performs equally well. Architecture-specific target layers, sharing variants, parameter counts, and the spectral equations appear in Appendix § 25.

Shared and input-varying components. Let δˉℓ\bar{\delta}_\ell be the mean post-nonlinearity correction over training prefixes and let ϵℓ(p)=δℓ(p)−δˉℓ\epsilon_\ell(p)=\delta_\ell(p)-\bar{\delta}_\ell. At a fixed incoming activation xx, replacing current control by its mean changes the affine projection by exactly Wϕ,ℓ(ϵℓ(p)⊙x)W_{\phi,\ell}(\epsilon_\ell(p)\odot x). This separates the shared correction from its input-varying residual; it is not a bound on final-network accuracy because changes propagate to later layers. For a fixed correction, compilation is exact in real arithmetic:

W~ϕ,ℓ=Wϕ,ℓdiag⁡(1+δˉℓ).(6)\widetilde{W}_{\phi,\ell}=W_{\phi,\ell}\operatorname{diag}(1+\bar{\delta}_\ell). \tag*{(6)}

Experimental Setup

Models and data. We evaluate DistilGPT-2, GPT-2 small/medium, Pythia-410M, Qwen2.5-0.5B/1.5B-Instruct, and Qwen3-0.5B/0.8B ([17]) within an 8GB GPU budget. Tasks are SciQ ([23]), ARC-Challenge ([3]), WinoGrande ([18]), and PIQA ([2]). Most results use full labeled validation splits. One fixed DistilGPT-2 IA3/PREFIX pair is additionally evaluated on the untouched 1,172-example ARC-Challenge test split. Validation-selected results, including the matched study below, are development diagnostics rather than independent confirmatory estimates. Uniform guessing over each question’s actual choice count gives 25.07% on ARC validation and 25.02% on ARC test (some questions have three or five choices), 25% on SciQ, and 50% on PIQA/WinoGrande.

Scoring and objective. Context-prefix (CP) scoring appends each answer text to Question: {question}\nAnswer:; WinoGrande instead appends each option and sentence suffix to the prefix before the blank. We report summed and length-normalized continuation log-probability. First-token prediction with prefilling (FTP+prefill) scores the first discriminative answer token after any shared answer prefix, with choices listed in the prompt. Multiple-choice prompting (MCP) lists all options and scores either the answer label or answer text. Appendix Table 10 gives exact definitions. For input-conditioned scoring, only context activations enter the controller; its state is identical across candidates, with no answer-token pooling. Training minimizes cross-entropy over candidate mean log-probabilities. A separate SciQ teacher-forced answer-token negative log-likelihood (NLL) experiment tests a standard token objective beyond candidate ranking.

Reference experiments. Static references include IA3, tuned LoRA/DoRA, prompt/prefix tuning, and compact local LoReFT- and PARA/PEDRO-style implementations. Headline GPT-style runs use 500 updates, batch size four, and maximum training length 96. IA3 and its jointly trained controller use learning rate 3×10−33 \times10^{-3}. LoRA ranks and learning rates are selected on validation; DoRA uses rank eight and 10−310^{-3}. Qwen3.5 uses one fixed configuration across three tasks: 1,024 training examples, 200 updates, and three seeds. Most studies use seeds 13/21/42; GPT-2 small ARC uses five. Appendix § 25 preserves all sweep settings, prompts, target layers, and variant details.

Matched frozen-adapter study. To isolate the effect of input-dependent corrections from protocol differences, we freeze existing IA3, tuned LoRA, and DoRA checkpoints on DistilGPT-2 and apply Eq. (3) identically. For each family, task (ARC/SciQ), and seed (13/21/42), we train three arms: current-prefix bounded control; learned-global bounded control with constant qℓq_\ell; and current-prefix unbounded control. All arms have the same 129,108 nominal trainable parameters, initial weights, sampled training batches, and 200-update budget, without a separately trainable static scale. Equal nominal counts do not imply equal input-dependent expressivity or identical gradients when the input is constant. The shared controller has hidden width 16, four coefficients, and ρ=0.2\rho= 0.2. We use 1,024 training examples, batch size four, AdamW at 5×10−45 \times10^{-4}, weight decay 0.01, clipping 1.0, and dropout disabled. The protocol is fixed before examining these runs; existing base-adapter selection still limits independence. No newly observed outcome selects a configuration.

Interventions and uncertainty. On each bounded current-prefix checkpoint, we evaluate the current correction, zero correction (exact frozen-adapter recovery), training-mean correction, random correction, and three shuffled donor permutations. The mean and random-scale estimates use 256 training prefixes, never validation prefixes. Random corrections are norm-matched around this mean and clipped to the coordinate bound; they do not preserve learned covariance. Shuffles reuse complete correction vectors from other validation examples, with no self-donors or labels. All choices use the same assigned vectors. For the matched study, paired crossed bootstrap resamples seeds and common example IDs (200,000 replicates); permutations are averaged within seeds, not treated as independent trials. We distinguish pointwise 95% intervals from Bonferroni sensitivity intervals over 12 comparisons (six task–family cells against both frozen and global controls), and over all 36 matched-study contrasts.

frozen and global controls), and over all 36 matched-study contrasts. The additional ARC test contrast has a separate 37-comparison sensitivity analysis (Appendix § 22). These adjustments do not cover every comparison in the paper or remove validation-selection bias. Earlier results retain their reported, pointwise hierarchical intervals. All estimates remain exploratory with few seeds; pooled McNemar is only an appendix diagnostic.

Compilation audit. We compile the saved training-mean corrections for all 18 bounded-current checkpoints, without training or selecting a new setting, and check all answer scores on the full validation splits against the uncompiled fixed-mean intervention. Runtime uses the first 24 preserved examples per task, seed 13, one question per batch with all candidates, float32, four warmup questions and five interleaved timing blocks on an RTX 5060 Laptop GPU. Dynamic timing includes a prefix-only pass and candidate scoring without KV-cache reuse; compiled timing requires neither a controller nor a prefix-only pass. CUDA-synchronized timing excludes loading and tokenization. This is a matched implementation benchmark, not an optimal implementation comparison; full protocol and memory definitions appear in Appendix § 24.

Results

Matched Mechanism Controls

Table 1 separates three questions: whether adding the controller helps a frozen adapter; whether the learned correction must be matched to the current input; and whether independently trained global control is sufficient. Unlike the earlier sweeps, all families use the same frozen-projection wrapper and controller budget.

TaskAdapterFrozenGlobalCurrentFixed meanRandomShuffled
ARC-CIA324.4±0.324.4\pm0.327.6±1.927.6\pm1.928.2±0.828.2\pm0.828.5±0.728.5\pm0.728.4±0.728.4\pm0.728.3±0.728.3\pm0.7
ARC-CLoRA32.6±2.732.6\pm2.733.9±1.633.9\pm1.634.1±1.234.1\pm1.234.2±1.434.2\pm1.434.1±1.234.1\pm1.234.3±1.234.3\pm1.2
ARC-CDoRA34.3±4.034.3\pm4.035.1±1.235.1\pm1.235.9±1.535.9\pm1.536.0±1.536.0\pm1.536.0±1.436.0\pm1.435.9±1.535.9\pm1.5
SciQIA351.5±0.951.5\pm0.953.0±0.653.0\pm0.652.9±0.652.9\pm0.652.9±0.652.9\pm0.653.0±0.753.0\pm0.753.0±0.753.0\pm0.7
SciQLoRA51.4±1.651.4\pm1.652.5±1.052.5\pm1.052.7±1.052.7\pm1.052.8±0.952.8\pm0.952.8±0.952.8\pm0.952.8±0.952.8\pm0.9
SciQDoRA50.6±0.850.6\pm0.853.6±0.653.6\pm0.653.5±0.653.5\pm0.653.3±0.653.3\pm0.653.3±0.653.3\pm0.653.3±0.653.3\pm0.6

Table 1. Matched frozen-adapter study: DistilGPT-2, full validation splits (299 ARC-C; 1,000 SciQ), three seeds. Uniform guessing gives 25.07%/25%, respectively. Current/global each add 129,108 nominal parameters with matched initialization, batches and 200 updates. Global receives zero input; mean/random/shuffled intervene on the current checkpoint without retraining. Shuffled averages three permutations per seed. Bold marks the highest displayed mean, including rounding ties, not significance.

Transfer across families. Current-prefix control improves the mean over the frozen adapter in all six task–family cells. ARC gains are +3.79, +1.56, and +1.56 points for IA3, LoRA, and DoRA; the corresponding SciQ gains are +1.40, +1.33, and +2.87. Only SciQ–DoRA excludes zero pointwise: +2.87 [1.00, 4.77], improving 50.6% to 53.5%; its B12 interval [0.00, 5.83] and B36 interval [-0.40, 6.30] do not exclude zero. This is exploratory evidence beyond IA3/ARC, not a familywise-significant or universal advantage. These development results do not replace the independent ARC test result below.

Does the current prefix matter? Replacing each input’s correction by the training-mean correction changes accuracy by at most 0.34 points in any cell, with all current-minus-mean intervals including or touching zero. Current-prefix control also does not reliably outperform shuffled or learned-global control in any cell. Table 2 directly reports this conditioning residual per family and task, and extends current versus global to ARC test: 25.68% versus 25.77%, or -0.09 [-1.25, 0.91] points. These are not equivalence tests, but the observed current–mean differences are small, and prediction disagreement is only 0.11–0.56% across the six cells. The results support a largely input-independent correction in this regime, not a demonstrated semantic-matching advantage. They also motivate a concrete deployment test: the fixed-mean correction can be folded into the frozen projection weights (Eq. (6)). § 5.2 tests this compilation, without treating a small dynamic–mean difference as statistical equivalence.

TaskAdapterCurrent − frozen [95%]Current − global [95%]Global contrast [B12]
ARC-CIA3+3.79 [0.00, 7.47]+0.56 [-1.45, 3.01][-2.34, 4.35]
ARC-CLoRA+1.56 [-1.78, 4.91]+0.22 [-1.67, 2.34][-2.68, 3.68]
ARC-CDoRA+1.56 [-2.34, 5.91]+0.78 [-1.67, 2.79][-3.46, 3.90]
SciQIA3+1.40 [-0.80, 3.60]-0.03 [-0.87, 0.80][-1.30, 1.23]
SciQLoRA+1.33 [-0.70, 3.30]+0.23 [-0.60, 1.10][-1.07, 1.67]
SciQDoRA+2.87 [1.00, 4.77]-0.10 [-0.90, 0.67][-1.37, 1.10]

Table 2. Additional frozen-IA3 test diagnostic: 1,172 ARC examples, three seeds Frozen: 23.07±0.40; global: 25.77±0.45; current: 25.68±0.98. Current − global: -0.09 [-1.25, 0.91]; B37: [-2.30, 1.54]. Paired conditioning residuals in percentage points. [95%] denotes pointwise crossed-bootstrap intervals (200,000 draws); B12 targets 95% familywise coverage over six current–frozen and six current–global contrasts. The added test comparison uses existing frozen-adapter checkpoints without retraining or test-driven selection; B37 includes the 36 earlier matched-study contrasts. It is distinct from the jointly trained 26.4% headline test result. No current–global interval excludes zero; this is not an equivalence claim.

Which corrections are useful? Figure 2 examines the output correction rather than merely probing the controller’s internal state. The input-varying residual around the training mean is only 1.8–2.8% of total correction RMS, averaged over layers and seeds within each task–family cell. A non-collapsed prefix representation therefore need not produce materially different correction vectors. Output perturbations vary across layers and families, as Eq. (4) predicts; their magnitude alone does not explain accuracy. On ARC, IA3 corrects 9.92% of examples while harming 6.13%, compared with LoRA’s 6.47%/4.91% and DoRA’s 7.47%/5.91%. On SciQ, DoRA instead has the strongest net trade-off, 9.53% corrected versus 6.67% harmed. The reversal across tasks argues against a universal IA3-only explanation. Adapted projections, activation statistics, and decision margins can change the effect of the same correction, as Eq. (4) makes explicit; these diagnostics localize the prediction trade-offs but do not identify which factor causes them.

Figure 2. Direct correction diagnostics in the matched frozen-adapter study. (a) Relative local output change at each MLP projection, holding its incoming activation fixed; solid lines denote ARC and dashed lines SciQ. This is not an attribution of the final prediction to one layer. (b) Input-varying correction RMS around the training mean, relative to total correction RMS; dots are seeds. (c) Percentages of predictions corrected and harmed relative to the frozen base adapter. A large correction is not necessarily useful: accuracy gain is the difference between corrected and harmed examples.

Is the bound necessary? The equally initialized unbounded arm has no reliably different accuracy in any cell (Appendix Table 23). The coordinate bound is a design constraint and yields the local norm inequality in § 3; these results do not establish it as the cause of the accuracy gains. Random-control clipping is zero in all runs, and its near-current accuracy cannot be attributed to clipping away the perturbation. All accuracy, NLL, margin, and intervention outcomes are retained in the accompanying evidence package.

Static Compilation

Table 3 tests a deployment consequence of the fixed-mean intervention, not a newly trained adapter. Compilation reproduces the uncompiled fixed-mean prediction on 11,690 of 11,691 seed–example pairs across all 18 checkpoints. The largest absolute answer-score discrepancy is 3.82×10−53.82 \times10^{-5}; one near-tied SciQ–IA3 prediction changes under floating-point reassociation without changing correctness. All 18 full-split accuracies are unchanged. In this prefix-reuse implementation, compiled latency is 2.16–2.25 times lower than dynamic latency and close to the merged frozen adapter. These ratios include the dynamic prefix-only pass, but no KV-cache optimization; they should not be read as a hardware-independent speedup. Compilation removes the need to keep or evaluate the controller, but materializing a separately adapted dense model does not reduce backbone storage or guarantee a multi-tenant memory benefit.

TaskAdapterAccuracy (%, three seeds)Latency (ms/question)
CurrentFixed meanCompiledFrozenCurrentCompiled
ARC-CIA328.2128.5428.548.5718.908.43
ARC-CLoRA34.1134.2334.238.4718.298.48
ARC-CDoRA35.9036.0136.018.4718.308.44
SciQIA352.9352.9352.938.1418.168.19
SciQLoRA52.7352.8052.808.1718.238.19
SciQDoRA53.5053.3353.338.1518.258.13

Table 3. Static compilation using the training-prefix mean. Accuracy uses full validation splits; fixed-mean values come from the same checkpoints as Table 1. Latency is the median of five blocks on 24 questions per task, seed 13, RTX 5060 Laptop GPU, float32, batch size one question with all choices. Bold identifies the lower latency among current/compiled deployment modes, not accuracy significance. This is not a comparison of differently trained methods.

Broader Empirical Evidence

Appendix Table 5 preserves earlier same-family pairs: IA3 versus PREFACE-IA3-small, tuned LoRA versus PREFACE-LoRA, and tuned DoRA versus PREFACE-DoRA. These are within-family comparisons; the matched frozen-adapter study in § 5.1 additionally controls training protocol. The strongest pattern is on ARC-Challenge, where multiplicative prefix control improves over static IA3 on DistilGPT-2, GPT-2 small, and Pythia-410M. The additive-family result is mixed: PREFACE-LoRA improves DistilGPT-2 separation-style ARC-Challenge from 32.6% to 34.3%, while PREFACE-DoRA is neutral under length-normalized scoring. Appendix Table 6 compares CP text scoring with first-token prefilling; mixed MCP stress tests appear in Appendix Table 11. Appendix Table 26 compares learned virtual-token methods, while near-chance WinoGrande results remain a stability diagnostic.

Prompt tuning is smaller than PREFACE-IA3-small but weaker on both datasets; prefix tuning wins on ARC-Challenge with nearly nine times as many parameters and a large SciQ regression. Thus PREFACE is a compact controller over existing PEFT adapters, not a uniformly dominant replacement for prompt-side PEFT.

Fixed DistilGPT-2 checkpoints retain the ARC-Challenge gain on the untouched test split in all seeds. With one Qwen3.5 configuration across three datasets, all mean deltas are positive but only SciQ excludes zero; the macro average rises from 67.45 to 67.82.

PREFACE-IA3-small also lowers SciQ answer-token NLL relative to IA3 for all three seeds, suggesting that bounded multiplicative control can help outside answer-choice ranking, though we keep this as a scoped evaluation rather than a broad generation benchmark.

Additional diagnostics. Appendix diagnostics report full SciQ accuracy, Qwen SciQ, MCP-style scoring stress tests, prefix-source diagnostics, controller ablations, boundedness, paired evidence, and latency; together they support the ARC-Challenge multiplicative result while preserving the mixed-task interpretation.

Conclusion

We studied PREFACE, a prefix-conditioned controller that makes small bounded changes around static PEFT adapters for token- or sequence-level inference without test-time gradients; causal prefix reuse additionally prevents candidate-answer leakage in our MCQ protocol. The controller can modify IA3-adapted projections, LoRA/DoRA-adapted projections, and representation-side low-rank channels. Earlier cross-backbone evidence is strongest for IA3 on ARC-Challenge; the matched study also has a positive SciQ–DoRA point estimate, but its multiplicity-adjusted interval touches zero. Fixed-mean and learned-global controls largely retain the gains, including in the ARC test diagnostic, and bounded versus unbounded differences remain uncertain. The contribution is a controlled study separating added correction from input matching, with a tested static-compilation route when the input-varying component is dispensable in the evaluated setting. The results do not establish statistical equivalence, a universal failure of conditional PEFT, or boundedness as the cause of the gains.

Limitations

Limitations. The evidence is concentrated on small GPT-2/GPT-NeoX models plus Qwen2.5 and Qwen3.5-0.8B, mostly MCQ likelihood evaluation and one SciQ answer-token likelihood evaluation. The untouched ARC-Challenge test result reduces validation-selection risk for the main DistilGPT-2 claim, but most other rows remain validation diagnostics. Results are strongest on ARC-Challenge and mixed on SciQ, Wino-Grande, PIQA, GPT-2 medium, additive PEFT, DoRA, and MCP-style option prompting. The prompt-side baseline comparison is also limited: prefix tuning outperforms PREFACE-IA3small on DistilGPT-2 ARC-Challenge at much higher parameter cost, while prompt tuning and SciQ prefix tuning are weaker under our loop. Full external-code LoReFT/PARA/PEDRO sweeps, exhaustive prompt/prefix hyperparameter searches, larger modern backbones, broader generation/classification tasks, and controller compression remain open. Three-seed matched diagnostics and validation-selected base settings limit inference even when evaluation uses complete splits. Test results already examined here cannot serve as an untouched selection-independent test for future modifications. Static compilation validates this implementation on the reported data and precision, not all future inputs, hardware, decoding modes, or conditional-PEFT methods.

Ethical Considerations

We introduce no new dataset of people or deployed application. Input-conditioned adapters may amplify context-specific biases, so we report regressions and avoid factuality or safety conclusions without direct evaluation.

Acknowledgements

AI assistance was used for research-idea exploration, code development and debugging, analysis scripts, manuscript drafting and revision, and preparation of submission materials. Responsibility for the research, verification, and final submission remains with the human authors.

Code and numerical artifacts. The accompanying arXiv ancillary archive, PREFACE_code.zip, contains implementation code, evaluation outputs, and instructions for recomputing the reported analyses. Pretrained model weights and training datasets are obtained separately under their respective licenses.

References

Earlier Same-Family and Scoring Comparisons

These previously reported comparisons preserve the original training protocols and separate their adapter gains from the matched input-conditioning tests in the main paper.

SciQ Accuracy Details

Table 7 reports the full SciQ validation accuracy table under both summed and length-normalized continuation scoring. It complements the main-text SciQ likelihood result by showing that PREFACE-IA3 is competitive but only modestly better under the standard MCQ scoring setup. The table also makes the metric dependence explicit: LoRA-tuned is strongest under summed continuation likelihood, whereas PREFACE-IA3-small is strongest under length-normalized scoring. This is why the paper treats SciQ as supportive rather than decisive evidence for the controller design.

MethodTrainable paramsSeedsSumMean
DistilGPT-20340.3±0.140.5±0.1
LoRA-r8 (initial)184,320327.1±1.234.0±2.1
LoRA-tuned368,640343.5±0.751.4±1.6
DoRA188,928340.4±2.150.6±0.8
IA318,432342.0±0.551.5±0.9
PREFACE-IA3-shared270,632341.9±0.852.4±0.9
PREFACE-IA3-small147,540342.5±0.952.6±0.5
PREFACE-Spectral4,802,358341.0±0.341.8±0.2

Table 7. SciQ accuracy on the full validation set, averaged over seeds 13, 21, and 42. “Sum” uses summed continuation log-probability; “Mean” uses length-normalized continuation log-probability. Bold marks the best score and underline marks the second-best score within each metric column. Input-conditioned rows use context-prefix-only controller states.

Stress-Test Diagnostics

Table 8 reports complementary Qwen2.5 SciQ evaluations. Unlike the ARC-Challenge rows in Table 4, these outcomes are mixed and the paired intervals overlap zero, so they serve as scope-calibration evidence rather than the core backbone-transfer result. They help show that the modern-backbone effect is task-sensitive rather than uniformly positive across datasets.

EvaluationnnIA3PREFACEPaired Δ\Delta [95%]
DistilGPT-2 / ARC test1,17223.1±0.426.4±1.4+3.36 [1.39, 5.26]
Qwen3.5 / ARC val.29947.4±0.747.9±0.4+0.56 [-0.56, 1.67]
Qwen3.5 / SciQ val.1,00084.7±0.885.1±0.9+0.40 [0.03, 0.80]
Qwen3.5 / PIQA val.1,83870.2±0.470.4±0.5+0.15 [-0.20, 0.47]

Table 4. Held-out evaluation and recent-model transfer, with length-normalized scoring and three seeds. The ARC row evaluates the jointly trained -small checkpoint against IA3 on the test split withheld from its selection (uniform guessing: 25.02%); it is not the matched frozen-adapter test in Table 2. Qwen3.5-0.8B uses a fixed configuration and full validation splits. Intervals are pointwise hierarchical estimates.

Base adapterStaticPREFACEPaired delta
IA381.7±0.581.4±0.1-0.3 [-0.8, 0.2]
LoRA75.0±2.875.0±2.80.0 [-0.1, 0.0]

Table 8. Modern-backbone SciQ stress-test diagnostics on Qwen2.5-0.5B-Instruct. These rows are mixed or tied, so they are reported as complementary scope-calibration evidence alongside the ARC-Challenge backbone-transfer table.

Table 9 reports both scoring rules for the Qwen3.5 study summarized in the main paper. The same fixed hyperparameters are used for all three datasets.

DatasetScorennIA3PREFACEΔ\DeltaHierarchical 95% interval
ARC-Challengesum29943.03±1.3543.03\pm1.3543.14±1.0043.14\pm1.00+0.11+0.11[−0.67,0.89][-0.67, 0.89]
ARC-Challengemean29947.38±0.7047.38\pm0.7047.94±0.3947.94\pm0.39+0.56+0.56[−0.56,1.67][-0.56, 1.67]
SciQsum1,00082.30±0.9882.30\pm0.9882.63±0.8182.63\pm0.81+0.33+0.33[0.07,0.67][0.07, 0.67]
SciQmean1,00084.73±0.7684.73\pm0.7685.13±0.8585.13\pm0.85+0.40+0.40[0.03,0.80][0.03, 0.80]
PIQAsum1,83869.44±0.0869.44\pm0.0869.73±0.1769.73\pm0.17+0.29+0.29[0.00,0.58][0.00, 0.58]
PIQAmean1,83870.24±0.4470.24\pm0.4470.38±0.5470.38\pm0.54+0.15+0.15[−0.20,0.47][-0.20, 0.47]

Table 9. Complete Qwen3.5-0.8B results over seeds 13, 21, and 42. Separate adapters are trained for each dataset using one fixed configuration. Positive effects are modest; SciQ excludes zero under both scoring rules, PIQA sum touches zero, and the remaining intervals overlap zero.

Scoring Protocol Definitions

Table 10 makes the controller-visible and scored tokens explicit for every reported protocol.

ProtocolController-visible prefixScored tokensShared-prefix handling
CP textQuestion/context onlyCandidate answer textNone; one prefix representation is reused across choices.
FTP+prefillQuestion plus visible optionsFirst discriminative answer tokenTokens shared by all answer strings are prefixed but not scored.
MCP labelQuestion plus labeled optionsNext option-label tokenNo answer-text continuation is exposed to the controller.
MCP textQuestion plus labeled optionsCandidate answer textEach candidate text is masked from the controller.
Options-trainedQuestion plus labeled optionsCandidate answer textTraining and evaluation use the same options-visible template.

Table 10. Scoring protocols. CP denotes context-prefix answer-text scoring, FTP denotes first-token prefilling, and MCP denotes multiple-choice prompting. Candidate tokens receiving likelihood scores never enter the controller representation.

Table 11 gives the option-prompting stress tests referenced in § 5 and shows why CP text and first-token prefilling remain the main scoring protocols. The label-prompt and full-text MCP formats change the ranking behavior enough that gains no longer line up cleanly across adapter families. Reporting these rows separately keeps the main scoring table centered on the primary evaluation protocols while testing whether the same prefix-conditioned interface transfers across scoring formats.

Base adapterProtocolStatic adapterPREFACEΔ\Delta
IA3MCP label21.3±0.221.3\pm0.221.4±0.021.4\pm0.0+0.1+0.1 [0.0,0.4][0.0, 0.4]
LoRAMCP label24.3±0.524.3\pm0.523.2±2.623.2\pm2.6−1.1-1.1 [−4.9,2.7][-4.9, 2.7]
DoRAMCP label26.4±2.926.4\pm2.925.9±1.325.9\pm1.3−0.6-0.6 [−3.6,2.2][-3.6, 2.2]
IA3MCP text29.2±0.829.2\pm0.828.9±4.028.9\pm4.0−0.3-0.3 [−5.0,4.7][-5.0, 4.7]
LoRAMCP text31.8±4.131.8\pm4.133.0±3.933.0\pm3.9+1.2+1.2 [−1.3,3.8][-1.3, 3.8]
DoRAMCP text33.2±4.533.2\pm4.532.9±2.032.9\pm2.0−0.3-0.3 [−4.8,3.8][-4.8, 3.8]

Table 11. MCP-style ARC-Challenge scoring stress tests on DistilGPT-2. MCP label lists all choices and scores the next label token; MCP text lists all choices and scores each answer text. The mixed pattern is why the main scoring-protocol table focuses on CP text and first-token prefilling.

Backbone Transfer Details

Table 12 collects the same-family cross-backbone comparisons in one place. It shows that the IA3-family improvement pattern extends beyond DistilGPT-2 to GPT-2 small, Pythia-410M, and Qwen2.5-1.5B. At the same time, additive adapters show smaller or neutral effects, which is why the main empirical conclusion stays centered on multiplicative control rather than adapter-independent gains. The table therefore acts as the compact backbone-transfer summary behind the scattered headline rows in the main text.

Base adapterBackboneSame-family comparisonParamsSeedsBasePREFACEΔ\Delta
IA3DistilGPT-2IA3 → PREFACE-IA3-small147,540324.4±0.324.4 \pm0.328.7±2.828.7 \pm2.8+4.2+4.2
LoRADistilGPT-2LoRA-tuned → PREFACE-LoRA270,632332.6±2.732.6 \pm2.734.3±0.734.3 \pm0.7+1.8+1.8
DoRADistilGPT-2DoRA → PREFACE-DoRA270,632334.3±4.034.3 \pm4.034.4±3.234.4 \pm3.2+0.1+0.1
IA3GPT-2 smallIA3 → PREFACE-IA3-small239,700524.5±1.524.5 \pm1.529.0±2.329.0 \pm2.3+4.5+4.5
IA3GPT-2 mediumIA3 → PREFACE-IA3-small565,332332.2±1.032.2 \pm1.033.6±5.733.6 \pm5.7+1.3+1.3
IA3Pythia-410MIA3 → PREFACE-IA3-gated467,052331.5±1.531.5 \pm1.534.0±0.834.0 \pm0.8+2.5+2.5
IA3Qwen2.5-0.5BIA3 → PREFACE-IA3554,604340.6±1.040.6 \pm1.041.8±1.241.8 \pm1.2+1.2+1.2
IA3Qwen2.5-1.5BIA3 → PREFACE-IA3-gated1,164,912351.5±1.051.5 \pm1.053.1±0.253.1 \pm0.2+1.6+1.6
LoRAQwen2.5-0.5BLoRA-tuned → PREFACE-LoRA87,704327.8±1.227.8 \pm1.228.1±1.528.1 \pm1.5+0.3+0.3

Table 12. ARC-Challenge validation accuracy by base adapter and backbone using separation-style continuation scoring. “Params” counts trainable parameters in the reported adaptation stage, including the static scale in jointly trained -small rows. Two-stage -gated rows freeze the static scale. Rows compare against same-family references rather than IA3 as a universal baseline.

Representation-Side Baseline

Table 13 summarizes the direct representation-side baselines discussed in the main text. The table clarifies that PARA/PEDRO-style adapters are competitive, but the gains are smaller and less uniform than the multiplicative IA3-family result. It also shows that the best raw accuracy in this appendix section comes from a representation-side adapter, not from the IA3 family, which keeps the paper from overclaiming universal dominance. What remains distinctive for the main proposal is not winning every adapter family, but providing one bounded prefix controller across PEFT families and downstream inference formats; its strongest measured gains appear in multiplicative settings.

MethodParamsARC Acc.Paired evidence
IA318,43224.4±0.324.4 \pm0.3–
LoReFT-minimal18,552<u>28.8±0.628.8 \pm0.6</u>–
PREFACE-LoReFT14,00429.0±1.329.0 \pm1.3+0.2+0.2 [-0.9, 1.7] vs LoReFT
PARA/PEDRO-small60,056<u>30.5±2.330.5 \pm2.3</u>–
PREFACE-PARA/PEDRO-small13,90831.5±1.731.5 \pm1.7+1.0+1.0 [0.0, 2.5] vs PARA/PEDRO
PREFACE-IA3-small147,54028.7±2.828.7 \pm2.8−1.9-1.9 [-7.9, 3.3] vs PARA/PEDRO

Table 13. Direct representation-side diagnostics on DistilGPT-2 ARC-Challenge over seeds 13, 21, and 42. Bold marks the best accuracy and underline marks the second-best accuracy. LoReFT-minimal is a compact local LoReFT-style reproduction with rank-4 low-rank subspace interventions on context-token hidden states. PREFACE-LoReFT freezes this base adapter and trains a prefix-conditioned controller over its rank channels. PARA/PEDRO-small uses prefix-pooled representation control and token–prompt dot-product gates to add low-rank hidden-state residuals. PREFACE-PARA/PEDRO-small freezes that base adapter and trains a prefix-conditioned controller over its value channels. Paired intervals are hierarchical bootstrap intervals in percentage points.

PEFT familySettingSame-family comparisonParamsSeedsStaticPREFACEΔ
IA3DistilGPT-2 / ARC-CIA3 → PREFACE-IA3-small147,540324.4±0.328.7±2.8+4.2
IA3GPT-2 small / ARC-CIA3 → PREFACE-IA3-small239,700524.5±1.529.0±2.3+4.5
IA3Pythia-410M / ARC-CIA3 → PREFACE-IA3-gated467,052331.5±1.534.0±0.8+2.5
IA3Qwen2.5-1.5B / ARC-CIA3 → PREFACE-IA3-gated1,164,912351.5±1.053.1±0.2+1.6
LoRADistilGPT-2 / ARC-CLoRA-tuned → PREFACE-LoRA270,632332.6±2.734.3±0.7+1.8
DoRADistilGPT-2 / ARC-CDoRA → PREFACE-DoRA270,632334.3±4.034.4±3.2+0.1
IA3DistilGPT-2 / SciQIA3 → PREFACE-IA3-small147,540351.5±0.952.6±0.5+1.1
IA3DistilGPT-2 / SciQ answer NLLIA3 → PREFACE-IA3-small147,54034.488±0.0054.464±0.007-0.024

Table 5. Headline same-family comparisons, not parameter-matched conditioning tests. ARC uniform-guessing accuracy is 25.07%. DistilGPT-2/GPT-2 IA3 rows use the jointly trained -small variant; Pythia/Qwen2.5-1.5B use the two-stage -gated variant. Params count trainable parameters in the reported adaptation stage, including the static scale when jointly trained. SciQ answer NLL is lower-is-better, with pointwise paired interval -0.024 [-0.040, -0.007].

PEFT familyProtocolStatic adapterPREFACEΔ
IA3CP text24.4±0.328.7±2.8+4.2 [0.1, 8.1]
LoRACP text32.6±2.734.3±0.7+1.8 [-1.4, 5.1]
DoRACP text34.3±4.034.4±3.2+0.1 [-3.1, 3.5]
IA3FTP+prefill19.0±0.823.0±2.3+4.0 [1.3, 6.8]
LoRAFTP+prefill25.9±1.726.2±0.4+0.3 [-1.9, 2.6]
DoRAFTP+prefill25.2±1.825.9±1.8+0.7 [-1.7, 3.0]

Table 6. ARC-Challenge protocol diagnostic on DistilGPT-2 using the same CP-trained checkpoints (uniform-guessing accuracy: 25.07%). CP scores answer text; FTP+prefill lists choices and scores the first discriminative token after any shared prefix. Deltas use pointwise hierarchical intervals, not multiplicity-adjusted evidence. MCP stress tests appear in Appendix Table 11.

Controller Architecture Ablation

Table 14 details the controller ablation behind the TokenMLP choice. The main takeaway is that lightweight token-local processing is sufficient, while stronger token mixing does not reliably improve accuracy. TokenMLP gives the best compact result in this sweep, but MeanMLP and TokenCNN remain close, which suggests that performance is not overly sensitive to small controller-architecture changes. By contrast, heavier attention-style token mixing increases parameters substantially without delivering a clear accuracy advantage.

ControllerParamsAcc.Δ vs IA3Summary path
Static IA318k24.4±0.3–static scale
MeanMLP148k28.7±2.8+4.2mean → MLP
TokenMLP148k29.2±2.5+4.8shared MLP → mean
Attention pooling203k27.9±0.8+3.5learned weighted mean
Self-attention149k28.5±2.8+4.1self-attn → mean
TokenCNN240k28.7±2.9+4.2Conv1D → mean
TokenSelfAtt37.9M28.1±1.2+3.7full token self-attn
GatedTokenMLP197k27.7±2.3+3.2token MLP + gate

Table 14. DistilGPT-2 ARC-Challenge controller ablation over seeds 13, 21, and 42 using separation-style scoring and current-prefix control. Bold marks the best score and underline marks tied second-best scores. h1,…,hTh_1, \ldots, h_T denote the prefix-token hidden states seen by the controller. MeanMLP first averages these states and then applies the controller MLP; TokenMLP applies a shared MLP to each token state and then averages. TokenMLP is the best compact variant here, but stronger token mixing with attention or CNN layers does not reliably improve accuracy.

Paired Evidence Details

Table 15 reports the full paired-evidence breakdown supporting the headline rows. It distinguishes intervals that exclude zero from intervals that overlap zero and makes the adapter-family-specific pattern explicit. The cleanest evidence remains the IA3-family ARC-Challenge rows on DistilGPT-2, GPT-2 small, Pythia-410M, and Qwen2.5-1.5B, whereas SciQ, LoRA, and DoRA are weaker or null. This table is therefore the main statistical evidence for the paper’s conclusion that the controller effect is real but not universal.

FamilySettingReference → prefix variantRef.PrefixΔ [95%]Seed-level evidence
Mult.SciQIA3 → PREFACE-IA3-shared51.552.5+1.0 [-0.5, 2.5]2/3 seeds; boot pp = 0.191; McNemar diag. pp = 0.172
Mult.ARCIA3 → PREFACE-IA3-small24.428.7+4.2 [0.1, 8.1]3/3 seeds; boot pp = 0.046; McNemar diag. pp = 0.0049
Mult.WinoGrandeIA3 → PREFACE-IA3-small48.849.8+1.0 [-0.6, 2.5]3/3 seeds; boot pp = 0.205; McNemar diag. pp = 0.196
Mult.ARC, GPT-2IA3 → PREFACE-IA3-small24.529.0+4.5 [1.0, 8.3]5/5 seeds; boot pp = 0.0088; McNemar diag. pp = 0.00085
Mult.ARC, GPT-2-MIA3 → PREFACE-IA3-small32.233.6+1.3 [-4.3, 7.6]1/3 seeds; boot pp = 0.686; McNemar diag. pp = 0.502
Mult.ARC, PythiaIA3 → PREFACE-IA3-gated31.534.0+2.5 [0.7, 4.2]3/3 seeds; boot pp = 0.013; McNemar diag. pp = 0.0032
Mult.ARC, QwenIA3 → PREFACE-IA340.641.8+1.2 [-0.2, 2.7]3/3 seeds; boot pp = 0.101
Mult.ARC, Qwen-1.5BIA3 → PREFACE-IA3-gated51.553.1+1.6 [0.2, 3.0]3/3 seeds; boot pp = 0.033
Mult.SciQ, QwenIA3 → PREFACE-IA381.781.4-0.3 [-0.8, 0.2]0/3 seeds; boot pp = 0.298
LoRAARC, sep.LoRA-tuned → PREFACE-LoRA32.634.3+1.8 [-1.6, 5.1]2/3 seeds; boot pp = 0.294; McNemar diag. pp = 0.205
LoRAARC, QwenLoRA-tuned → PREFACE-LoRA27.828.1+0.3 [-1.2, 1.9]1/3 seeds; boot pp = 0.708
LoRASciQ, QwenLoRA-tuned → PREFACE-LoRA75.075.00.0 [-0.1, 0.0]0/3 seeds; boot pp = 0.981
DoRAARC, sep.DoRA → PREFACE-DoRA34.334.4+0.1 [-3.1, 3.5]1/3 seeds; boot pp = 0.982; McNemar diag. pp = 1.000

Table 15. Seed-level paired evidence by adapter family on length-normalized accuracy. Deltas are candidate–reference accuracy in percentage points. The interval is a hierarchical bootstrap that resamples seeds and then examples within each seed; this is the primary uncertainty estimate. Pooled McNemar values, when shown, are ancillary diagnostics because seed–example pairs are not independent.

Checkpoint and Runtime Details

Tokenizers are loaded from the same checkpoint revisions as the corresponding backbones.

Controller Overhead

Table 17 reports the latency cost of the controller relative to static adapters. The overhead is measurable but remains small in absolute memory usage on the laptop-GPU setup. In throughput terms, PREFACE variants are slower than static IA3, LoRA, or DoRA because they add a prefix encoder and channel-wise modulation. Even so, the peak allocated memory stays below 0.4GB in this benchmark, which helps frame the method as lightweight in deployment rather than just in trainable parameters.

Prefix-Source Diagnostics

Table 18 gives a parameter-matched multi-task mechanism diagnostic on Qwen3.5-0.8B. The learned-global control exactly matches IA3, while current, shuffled, and random representations differ by at most 0.19 pooled points and all current-minus-control intervals include zero. No prefix-source contrast is significant before correction, so Holm correction does not change any conclusion. The learned representations nevertheless retain coarse task information: a held-out nearest-centroid probe predicts task identity at 82.1%, 80.2%, and 83.7% across seeds (33.3% chance), with effective ranks 8.24, 8.30, and 15.38. Thus the controller is non-constant and task-structured, but these controls do not establish fine-grained semantic matching to the current question.

Table 19 gives the separation-style prefix-source diagnostic for all three adapter families. It shows that IA3 and LoRA benefit more than DoRA, and that shuffled prefixes remain surprisingly competitive. That pattern constrains the interpretation: the controller is not best understood as a pure donor-question semantic mechanism. Instead, the evidence points to a bounded prefix-derived control channel whose gains partly survive even when the source prefix is perturbed.

FamilyMethodPrefix sourceParamsAcc.Seed-level evidence vs. same-family reference
Mult.IA3static18k24.4±0.3–
Mult.Static basis (92k)learned global92k27.5±3.0+3.1 [-0.7, 6.6]; 2/3 seeds; boot pp = 0.120
Mult.PREFACE-IA3-smallcurrent prefix148k28.7±2.8+4.2 [0.2, 8.0]; 3/3 seeds; boot pp = 0.049
Mult.PREFACE-IA3-smallrandom prefix148k27.6±1.7+3.2 [0.0, 6.4]; 3/3 seeds; boot pp = 0.054
Mult.PREFACE-IA3-smallshuffled prefix148k28.8±2.7+4.3 [0.4, 8.1]; 3/3 seeds; boot pp = 0.042
LoRALoRA-tunedstatic184k32.6±2.7–
LoRAPREFACE-LoRAcurrent prefix271k34.3±0.7+1.8 [-1.6, 5.2]; 2/3 seeds; boot pp = 0.319
LoRAPREFACE-LoRArandom prefix271k32.3±2.0-0.2 [-2.0, 1.7]; 1/3 seeds; boot pp = 0.833
LoRAPREFACE-LoRAshuffled prefix271k34.7±1.0+2.1 [-1.3, 5.6]; 2/3 seeds; boot pp = 0.253
DoRADoRAstatic189k34.3±4.0–
DoRAPREFACE-DoRAcurrent prefix271k34.4±3.2+0.1 [-3.0, 3.6]; 2/3 seeds; boot pp = 0.957
DoRAPREFACE-DoRArandom prefix271k33.7±1.6-0.7 [-3.2, 2.2]; 1/3 seeds; boot pp = 0.653
DoRAPREFACE-DoRAshuffled prefix271k34.2±3.0-0.1 [-3.2, 3.3]; 2/3 seeds; boot pp = 0.989

Table 19. DistilGPT-2 ARC-Challenge separation-style prefix-source diagnostic over seeds 13, 21, and 42. Bold marks the best score and underline marks the second-best score within each adapter family. Rows compare within adapter family: multiplicative rows use IA3 as the reference, PREFACE-LoRA rows use tuned LoRA, and PREFACE-DoRA rows use tuned DoRA. The 92k static basis is not parameter-matched to the 148k controller; matched controls appear in Table 1–Table 2. PREFACE-LoRA preserves the current-prefix improvement pattern, random prefix removes most of it, and shuffled prefix remains close to current, while PREFACE-DoRA is neutral.

Table 20 adds the options-format multiplicative diagnostic to show that the same static-versus-prefix pattern is not limited to one prompt format. As in the separation-style table, the shuffled-prefix row remains close to the current-prefix row, while the learned global control already captures part of the gain. This pattern shows that gains can survive a changed donor prefix; it does not isolate boundedness from other parameterization or optimization effects.

FamilyMethodPrefix sourceParamsAcc.ΔDiagnostic note
Mult.IA3static18k27.2±0.7–Static low-parameter multiplicative reference.
Mult.Parameter-matched staticlearned global147k29.7±2.5+2.45 [0.33, 4.57]Adds scale capacity without input conditioning.
Mult.PREFACE-IA3-smallcurrent prefix148k30.5±1.6+3.34 [0.45, 6.24]Prefix-controlled scale; +0.89 [-1.78, 3.57] vs. the parameter-matched static control.
Mult.PREFACE-IA3-smallrandom prefix148k28.4±0.3current +2.12 [-0.45, 4.68]Replaces the context representation with fixed random vectors.
Mult.PREFACE-IA3-smallshuffled prefix148k30.9±1.6current -0.33 [-0.78, 0.00]Uses donor-prefix summaries; not worse than current-prefix states.

Table 20. DistilGPT-2 ARC-Challenge options-format multiplicative diagnostic. Bold marks the best score and underline marks the second-best score. All rows use the options-text train/eval format and report seed mean ± standard deviation over seeds 13, 21, and 42. Bracketed values are seed-level bootstrap 95% intervals in percentage points.

Boundedness Mechanism Diagnostic

Table 21 tests the regularization induced by the bounded correction δ(q)=ρtanh⁡(Bc(q))\delta(q) = \rho\tanh(Bc(q)) while keeping the DistilGPT-2 backbone, frozen IA3 adapter, controller width, basis size, training data, and update budget fixed. A very small bound (ρ=0.05\rho= 0.05) changes fewer predictions but also yields fewer net corrections. The main setting (ρ=0.20\rho= 0.20) gives the best correction trade-off in this diagnostic. A looser bound (ρ=1.00\rho= 1.00) produces larger corrections and more prediction flips without improving the net number of corrected decisions. This pattern is consistent with moderate bounded control acting as a regularizer, although the overlapping accuracy intervals do not establish a uniquely optimal value of ρ\rho.

ρ\rhoMean ∣δ∣|\delta|P95 ∣δ∣|\delta|Sum Δ\DeltaMean Δ\DeltaSum net/flipsMean net/flipsSeeds
0.050.02970.0491+1.00 [-0.33, 2.45]-0.11 [-1.45, 1.23]+9 / 69-1 / 803
0.200.06620.1776+2.01 [0.00, 4.12]+2.01 [-2.01, 5.80]+18 / 147+18 / 2343
1.000.11990.3852+1.78 [-0.56, 4.12]+1.56 [-1.11, 4.46]+16 / 185+14 / 3123

Table 21. Boundedness diagnostic on DistilGPT-2 ARC-Challenge. Accuracy deltas are percentage points relative to the same frozen IA3 adapter, with paired hierarchical 95% bootstrap intervals. “Net/flips” is wrong-to-right minus right-to-wrong decisions, followed by the total number of changed predictions pooled across three seeds. Correction magnitudes are measured from actual controller outputs over the validation examples.

Model identifierRevisionPrecision
distilgpt22290a62682d06624634c1f46a6ad5be0f47f38aafloat32
gpt2607a30d783dfe6a63ca3e96633721c8d4fecd7efloat32
gpt2-medium6dcaa7a952f72f9298047f d5137cd6e4f05f41dafloat32
EleutherAI/pythia-410m-dedupedc4fc8d586d62df49f1f9b69d66d3ca419992d3efloat32
Qwen/Qwen2.5-0.5B-Instruct7ae557604adf67be50417f59c2cf167def9a775float16
Qwen/Qwen2.5-1.5B-Instruct989aa7980e4cf806f80c7fef2b1adb7b71aa306float16
Qwen/Qwen3.5-0.8B2f0c6364715b967f1860aea9cf38778875588b17float16

Table 16. Exact checkpoint revisions recovered from the local Hugging Face cache metadata used by the reported runs.

Matched-Study Diagnostics

Tables 1 and 2 place the complete primary accuracy controls in the main paper. The learned-global arm retains the complete controller parameterization but receives a constant input. Its nominal parameter count matches the dynamic arm; input variation and effective functional degrees of freedom do not. This is why we also use a fixed-mean intervention on the same trained dynamic checkpoint. That intervention requires no retraining and removes input variation without changing learned weights. The random control matches marginal scale but not learned correlations. We therefore interpret it together with three donor shuffles, which preserve the empirical correction distribution. All reported outcomes are retained. Three seeds and the use of development data limit

MethodParamsms/qq/schoices/sPeak MB
Frozen051.619.477.1377.8
IA318k52.519.075.7377.9
LoRA-tuned184k60.616.565.7378.5
DoRA189k58.017.268.6378.5
PREFACE-IA3-small148k75.013.353.1378.5
PREFACE-LoRA271k85.611.746.5379.7
PREFACE-DoRA271k88.911.244.8379.7

caption: Runtime on 100 ARC-Challenge validation questions using DistilGPT-2 on the laptop GPU. Choices are scored sequentially and prefix representations are not cached. PREFACE adds a small prefix encoder and bounded channel-wise modulation, increasing latency relative to static adapters while remaining below 0.4GB peak allocated CUDA memory in this DistilGPT-2 benchmark.

:::

Control sourcePooled acc.Current minus control95% interval
IA3 (no controller)67.74+0.11[-0.37, 0.59]
Learned global67.74+0.11[-0.37, 0.59]
Current prefix67.86––
Within-task shuffle67.82+0.04[-0.30, 0.37]
Cross-task shuffle67.74+0.11[-0.22, 0.45]
Random vector67.93-0.07[-0.48, 0.30]

caption: Matched Qwen3.5 mechanism diagnostic pooled over 299 fixed examples from each of ARC-Challenge, SciQ, and PIQA and three seeds. One controller is shared across tasks. Intervals hierarchically resample seeds and examples.

:::

confirmatory interpretation. The unbounded arm changes only the output nonlinearity: ρtanh⁡(v)\rho\tanh(v) becomes ρv\rho v. It is an optimization-matched diagnostic, not a separately tuned competitor. A difference cannot establish that boundedness is universally necessary.

Multiplicity and Frozen-Adapter Test Details

Table 2 separates pointwise and multiplicity-adjusted evidence. The 12-comparison sensitivity family includes six current–global and six current–frozen contrasts, not all paper comparisons. Table 22 supplies the current–frozen intervals omitted from that main-paper table. The 36-comparison sensitivity additionally includes current versus mean, shuffled, random, and unbounded corrections in all six cells. Each interval uses 200,000 paired seed/example draws, checked with a second bootstrap random seed. Monte Carlo precision does not resolve uncertainty from having only three training seeds.

TaskAdapterCurrent − frozen B12B36
ARC-CIA3[-2.01, 3.96][-2.79, 10.03]
ARC-CLoRA[-3.46, 6.69][-4.24, 7.36]
ARC-CDoRA[-4.35, 8.36][-5.02, 9.36]
SciQIA3[-1.87, 4.80][-2.30, 5.30]
SciQLoRA[-1.90, 4.40][-2.37, 4.90]
SciQDoRA[0.00, 5.83][-0.40, 6.30]

Table 22. Bonferroni sensitivity intervals in percentage points. The complete 36-contrast record is retained with raw predictions; these adjustments do not address development-set selection or every comparison in the paper.

ARC test comparison. After freezing the three current/global controller pairs used in the matched ARC validation diagnostic, we score all 1,172 official test examples with the same mean-answer-log-probability protocol, without retraining or test-driven selection. Both arms add 129,108 nominal parameters during training; base IA3 remains frozen. Recomputed zero-correction predictions agree with all 3,516 original baseline correctness flags. Table 2 reports the current–global result and B37 sensitivity (the earlier 36 diagnostic comparisons plus this test contrast). This does not change the jointly trained headline checkpoint or its 26.4% accuracy.

Exploratory Shared-Task Controller

We additionally test whether task heterogeneity makes actual-prefix information useful. One DistilGPT-2 IA3 adapter and one controller are shared across ARC-Challenge, SciQ, PIQA, and WinoGrande within each seed (13/21/42), with no task-specific adapter or head. We sample 1,024 training examples per task, train joint IA3 for 500 updates at learning rate 0.003, then freeze it and train current/global controllers for 200 updates at 0.0005. Every batch contains exactly one question per task. The matched 129,108-parameter architecture, initialization, objective, length-96 training limit, and controller batches are fixed across arms; no validation tuning is performed.

Within-task and cross-task interventions reuse complete donor correction bundles from the current checkpoint, with three fixed bijections and no self-donors; cross-task donors always come from another task. Bijections preserve the global correction marginal, although unequal task sizes preclude uniform donor-task frequencies. The mean intervention averages actual post-tanh corrections over 64 training prefixes per task. All evaluations use full validation splits and task-balanced macro averaging.

Macro current-minus-global is +0.24 points [-0.51, 0.94], current-minus-within-shuffle is -0.06 [-0.18, 0.09], and current-minus-cross-shuffle is +0.02 [-0.30, 0.35]. Current-minus-training-mean is -0.10 [-0.36, 0.11]. Intervals use 200,000 crossed seed/example draws, sharing seed resamples across tasks and resampling examples within each task. Neither task-level nor macro current–global/current–cross contrasts have strictly positive pointwise lower bounds. Bonferroni sensitivity across these ten primary contrasts also includes zero.

The macro input-varying correction RMS is

7.60% of total residual-correction RMS, excluding identity and centering on the balanced training mean. Training-centroid task prediction from correction vectors reaches 63.8–72.9% macro accuracy (chance 25%), but this descriptive task information does not establish prediction utility. Relative to cross-task donors, current prefixes correct 0.42% and harm 0.40% of examples under macro averaging. Thus the experiment does not demonstrate either a current-prefix advantage or useful coarse task routing; similar accuracy is not an equivalence result. The joint base adapter is weaker than the separately trained headline models, and this conclusion is limited to the fixed short-budget diagnostic. The prespecified positive-evidence gate for Qwen replication was not met, so no outcome-driven Qwen search was

TaskAdapterCurrent boundedUnbounded
ARC-CIA328.2±0.827.9±0.8
ARC-CLoRA34.1±1.234.1±1.5
ARC-CDoRA35.9±1.534.8±1.5
SciQIA352.9±0.652.7±0.4
SciQLoRA52.7±1.052.6±1.3
SciQDoRA53.5±0.653.6±1.0

caption: Matched bounded/unbounded validation controls. Unbounded replaces ρtanh⁡(v)\rho\tanh(v) by ρv\rho v without retuning. Bold denotes the higher displayed mean, including rounding ties. All paired pointwise intervals for this contrast include zero.

:::

run.

Static-Compilation Protocol and Runtime

The compilation audit uses every bounded-current checkpoint of the matched study: two tasks, three families, and three seeds. Corrections are the mean of saved post-tanh vectors over 256 training prefixes; no evaluation examples or labels enter their estimation. No training or hyperparameter search is performed. Full candidate scores and predictions are compared against the preserved fixed-mean intervention, with source hashes and per-example compiled outputs supplied in the software package. GPT-2’s implementation stores projection weights transposed relative to Eq. (6), so compilation multiplies the input-channel rows of the stored matrix. Bias is unchanged. Equivalence is algebraic in real arithmetic; float32 matrix reassociation can change argmax for near-tied answers. One such disagreement occurs in SciQ–IA3 seed 42. This is distinct from the statistically uncertain current–global contrast.

Timing protocol. First 24 examples in preserved validation order; seed 13; float32, TF32 disabled, SDPA attention; four warmup questions per mode and five blocks in deterministically shuffled order. Each question packs all candidates into one batch. Dynamic mode computes a prefix-only pass, retains layer corrections on the GPU, then scores candidates. Fixed-unfolded uses the same training-mean vector without a prefix pass; compiled folds that vector into the weights; frozen merges the trained base adapter without correction. There is no KV-cache reuse in any arm. CUDA synchronization brackets each block. Tokenization, model loading, compilation, and file I/O are excluded. Each block’s order, latency, throughput, and memory are retained.

Memory interpretation. Model variants are kept resident while timing to avoid loading differences. We therefore report incremental peak allocated CUDA memory above the resident baseline, not an isolated process’s total deployment memory. This includes transient tensors and does not estimate optimizer memory or multi-tenant storage. The same benchmark must be rerun for other decoding implementations or hardware.

Implementation and Reference Settings

Layer-Shared Input Controller

The non-shared controller form learns a separate controller network for each adapted layer. This improves flexibility but makes the controller much larger than a static vector adapter. Most large sweeps use multiplicative PEFT, so the notation below writes the layerwise control as a channel scale aℓa_\ell; additive and representation-side PEFT use the same sharing idea with their own channel values. PREFACE-IA3-shared shares the prefix encoder across all adapted MLP layers while keeping layer-specific static scale vectors and low-rank scaling bases:

aℓ(qℓ)=a0,ℓ+ρtanh⁡(Bℓcθ(qℓ)).(7)a_\ell(q_\ell) = a_{0,\ell} + \rho\tanh(B_\ell c_\theta(q_\ell)). \tag*{(7)}

Here cθc_\theta is shared, while a0,ℓa_{0,\ell} and BℓB_\ell are layer-specific. This preserves per-layer adaptation capacity but amortizes the expensive prefix encoder. In DistilGPT-2, the non-shared controller has 794k trainable parameters, whereas PREFACE-IA3-shared uses 270k parameters with the same basis size and hidden width. We also test PREFACE-IA3-small, a compact version with basis size 4 and hidden width 16, using 148k trainable parameters. For architecture-transfer diagnostics, we also test PREFACE-IA3-gated, a conservative two-stage variant that first trains a static scale baseline, freezes the resulting scale, and then trains only a context-conditioned correction with a learned scalar gate per layer. For this variant,

aℓ(qℓ)=a^0,ℓ+gℓρtanh⁡(Bℓcθ(qℓ)),(8)a_\ell(q_\ell) = \hat{a}_{0,\ell} + g_\ell\rho\tanh(B_\ell c_\theta(q_\ell)), \tag*{(8)}

where a^0,ℓ\hat{a}_{0,\ell} is the frozen static baseline and gℓ=σ(γℓ)g_\ell= \sigma(\gamma_\ell) is a learned scalar gate. The gates are initialized small and regularized so the dynamic component must improve a strong static baseline rather than overwrite it.

TaskFrozen IA3GlobalCurrentWithin shuffleCross shuffleTrain mean
ARC-C21.6322.4123.6323.6023.5223.97
SciQ49.2749.7749.6049.7649.7449.77
PIQA59.0759.1858.9258.9358.9458.89
WinoGrande49.3849.3849.5449.6449.4149.46
Macro44.8445.1845.4245.4845.4045.52

Table 24. Exploratory heterogeneous-task accuracy, mean over three seeds, one shared adapter/controller per seed. These checkpoints differ from all single-task headline rows. Bold marks highest displayed means, not statistical significance.

TaskAdapterModeLatency median [min,max] msQuestions/sIncremental peak MiB
ARC-CIA3frozen8.57 [8.43, 10.73]116.734.8
ARC-CIA3dynamic18.90 [18.35, 25.23]52.935.0
ARC-CIA3fixed-unfolded9.30 [9.22, 9.48]107.635.1
ARC-CIA3compiled8.43 [8.41, 10.93]118.634.8
ARC-CLoRAfrozen8.47 [8.46, 13.64]118.034.8
ARC-CLoRAdynamic18.29 [18.07, 18.61]54.735.0
ARC-CLoRAfixed-unfolded9.14 [9.10, 9.29]109.435.1
ARC-CLoRAcompiled8.48 [8.38, 10.69]117.934.8
ARC-CDoRAfrozen8.47 [8.34, 8.64]118.134.8
ARC-CDoRAdynamic18.30 [18.18, 19.16]54.635.0
ARC-CDoRAfixed-unfolded9.17 [9.14, 9.39]109.035.1
ARC-CDoRAcompiled8.44 [8.35, 8.60]118.534.8
SciQIA3frozen8.14 [8.11, 8.26]122.910.2
SciQIA3dynamic18.16 [17.82, 18.47]55.110.3
SciQIA3fixed-unfolded8.91 [8.81, 8.97]112.210.4
SciQIA3compiled8.19 [8.12, 8.99]122.110.2
SciQLoRAfrozen8.17 [8.04, 8.21]122.410.2
SciQLoRAdynamic18.23 [18.14, 18.47]54.810.3
SciQLoRAfixed-unfolded8.93 [8.74, 9.01]111.910.4
SciQLoRAcompiled8.19 [8.08, 8.28]122.110.2
SciQDoRAfrozen8.15 [8.07, 8.31]122.710.2
SciQDoRAdynamic18.25 [18.06, 18.46]54.810.3
SciQDoRAfixed-unfolded8.94 [8.88, 9.07]111.810.4
SciQDoRAcompiled8.13 [8.06, 8.19]123.110.2

Table 25. Compilation runtime audit. Block ranges describe measurement variability, not confidence intervals over tasks or training seeds. All rows use the same fixed protocol; earlier runtime tables use a different benchmark and should not be compared directly.

Spectral Diagnostic

Let W∈Rdout×dinW \in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}} be a pretrained MLP projection and let

W≈Urdiag⁡(σr)VrT(9)W \approx U_r\operatorname{diag}(\sigma_r)V_r^{\mathsf{T}} \tag*{(9)}

be a rank-rr truncated SVD. We define W⊥=W−Urdiag⁡(σr)VrTW_{\perp} = W - U_r\operatorname{diag}(\sigma_r)V_r^{\mathsf{T}}. PREFACE-Spectral scales only the dominant singular channels:

f(x;q)=W⊥x+Urdiag⁡(σr⊙s(q))VrTx,(10)f(x;q) = W_{\perp}x + U_r\operatorname{diag}\left(\sigma_r \odot s(q)\right)V_r^{\mathsf{T}}x, \tag*{(10)}

where qq is a context-prefix summary and s(q)∈Rrs(q) \in\mathbb{R}^{r} is an input-conditioned scaling vector. Equivalently,

f(x;q)=Wx+Urdiag⁡(σr⊙(s(q)−1))VrTx.(11)f(x;q) = Wx + U_r\operatorname{diag}\left(\sigma_r \odot(s(q)-1)\right)V_r^{\mathsf{T}}x. \tag*{(11)}

Thus the pretrained projection is recovered exactly when s(q)=1s(q)=1. We parameterize the spectral scaling as a bounded residual update around the identity:

s(q)=1+ρtanh⁡(Bcθ(q)),(12)s(q) = 1 + \rho\tanh(Bc_{\theta}(q)), \tag*{(12)}

where ρ\rho is a small scale, cθ(q)∈Rmc_{\theta}(q) \in\mathbb{R}^{m} is predicted by a lightweight controller network, and B∈Rr×mB \in\mathbb{R}^{r\times m} is a learned basis with m≪rm \ll r. The spectral row is a diagnostic for whether singular channels are a good basis for prefix-conditioned control. The earlier LoRA/DoRA implementations scale the input to the complete adapted projection and include a trainable static input scale; they do not isolate the low-rank update branch.

Reference Experiment Settings

The following settings describe the earlier sweeps, not the matched frozen-adapter study.

Models. We use GPT-2-style causal language models so experiments can run under a fixed 8GB GPU budget. Most experiments use DistilGPT-2. We also run targeted GPT-2 small, GPT-2 medium, and Pythia-410M experiments on ARC-Challenge and SciQ to test whether the main pattern survives larger backbones. For modern evaluations, we use Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen3.5-0.8B (Qwen Team, [17]) under the same local MCQ scoring protocol.

Tasks. The primary reported experiment uses SciQ (Welbl et al., [23]). We also include WinoGrande (Sakaguchi et al., [18]), ARC-Challenge (Clark et al., [3]), and PIQA (Bisk et al., [2]). Choices are scored by continuation likelihood on held-out validation splits, reporting both summed and length-normalized mean log-probability. Most rows use validation splits as reproducible labeled evaluation sets. After selecting the DistilGPT-2 configuration, we additionally evaluate the fixed IA3 and PREFACE-IA3-small checkpoints on the untouched ARC-Challenge test split (1,172 examples), which was not used for architecture or hyperparameter selection. To test beyond MCQ ranking, we also evaluate SciQ answer generation by teacher-forced answer-token NLL and perplexity. For SciQ and ARC-Challenge, the separation-style template is Question: {question}\nAnswer: followed by each candidate answer text. For WinoGrande, the context is the sentence prefix before the blank, and each candidate continuation is {option}{suffix}. In the options-based ARC diagnostic, all choices are shown as Options:\nA. ... before Answer:, and candidate answer texts are scored after that marker. We additionally report a post-hoc scoring-protocol diagnostic on ARC-Challenge: the main table includes the standard CP answer-text score and FTP+prefill, which scores only the first discriminative answer token after prefilling any token prefix shared by all answer choices. Appendix Table 11 reports MCP-label and MCP-text stress tests, where all choices are shown before scoring the label token or answer text.

Baselines. We compare pretrained models with no adaptation, LoRA and DoRA on MLP projections, IA3 on MLP projections, PEFT prompt tuning, PEFT prefix tuning, static spectral scaling, compact LoReFT- and PARA/PEDRO-style references, PREFACE-LoReFT, PREFACE-PARA/PEDRO-small, PREFACE-Spectral, PREFACE-IA3, PREFACE-LoRA, and PREFACE-DoRA as prefix-conditioned controller diagnostics. The SciQ study includes full-validation results for LoRA, DoRA, and the strongest multiplicative variants. The LoReFT-style diagnostic uses rank-4 low-rank subspace interventions on GPT-2 block outputs, masks interventions to context tokens under MCQ scoring, and trains 18.6k parameters, closely matching IA3’s parameter count. PREFACE-LoReFT freezes this LoReFT adapter and trains a 14.0k-parameter prefix controller over its rank-channel intervention. The PARA/PEDRO-style reference uses a prefix-pooled representation controller and token–prompt dot-product gates to add low-rank hidden-state residuals, training 60.1k parameters. PREFACE-PARA/PEDRO-small freezes that adapter and trains a 13.9k-parameter controller over its value channels. Prompt tuning and prefix tuning are trained with the same MCQ contrastive objective and candidate-scoring pipeline as the adapter rows, providing a prompt-side PEFT comparison in the same code path. Together with parameter-matched static, learned-global, and random/shuffled prefix-source controls, these rows are our matched in-code comparison to nearby input-conditioned and hypernetwork-style PEFT ideas. Transformer-Squared and In-Place TTT are discussed above at the design level; reproducing them would require separate implementations and broader benchmark protocols, so our empirical comparisons are restricted to baselines run in the same code path.

Implementation. Adapters target the MLP output projection: c_proj for GPT-2-style models and dense_4h_to_h for GPT-NeoX/Pythia. For input-conditioned adapters, the controller summary is masked to the context prefix during both training and evaluation, so candidate answer tokens never contribute to the adapter state used for continuation scoring. The backbone is frozen. In jointly trained PREFACE-IA3, including the main DistilGPT-2 small variant, the static scale uid=a0,ℓu_{id} = a_{0,\ell} is optimized with the controller; in two-stage gated runs, the trained IA3 scale is frozen before controller training. LoRA, DoRA, and representation-side base adapters are likewise frozen, with uid=1u_{id} = 1. Adapters are trained for 500 steps on up to 2,048 training examples from each dataset. IA3 and PREFACE-IA3-family adapters use AdamW with learning rate 3×10−33 \times10^{-3}, batch size 4, max sequence length 96, and gradient clipping at 1.0. The initial LoRA row uses rank 8, alpha 16, and learning rate 3×10−33 \times10^{-3}. To avoid under-tuning LoRA, we run a single-seed validation sweep over ranks {4,8,16}\{4, 8, 16\} and learning rates {10−4,3×10−4,10−3,3×10−3}\{10^{-4}, 3 \times10^{-4}, 10^{-3}, 3 \times10^{-3}\} with α=2r\alpha= 2r, then rerun the best settings over three seeds. Because selection and reporting use the same validation split, these LoRA rows are development results rather than confirmatory estimates. For additive adapters, PREFACE-LoRA freezes tuned LoRA and trains only the prefix controller; PREFACE-DoRA freezes tuned DoRA and trains the same controller. For the options-based ARC diagnostic, we also include a parameter-matched static control with 147k parameters: it uses IA3 plus learned layer-specific scaling bases, but replaces the prefix encoder with learned global coefficients. DoRA uses rank 8, alpha 16, and learning rate 10−310^{-3} after a small stability sweep. Prompt tuning and prefix tuning use 20 virtual tokens. Prompt tuning uses learning rate 3×10−23 \times10^{-2}. Prefix tuning uses projection, encoder hidden size 128, and learning rate 5×10−35 \times10^{-3}. The answer-generation evaluation uses the same DistilGPT-2 IA3 and PREFACE-IA3-small adapters, 200 training steps, and full SciQ validation. The Qwen3.5 study fixes one configuration across ARC-Challenge, SciQ, and PIQA: 1,024 training examples, 200 updates, batch size 1, maximum length 96, and float16 precision. All Qwen3.5 rows use seeds 13, 21, and 42 and the complete local validation splits (299 ARC-Challenge, 1,000 SciQ, and 1,838 PIQA examples). Experiments use PyTorch 2.11.0 with CUDA 12.8, Transformers 5.5.4, and PEFT 0.18.1; GPT-style runs use float32, and no run uses gradient accumulation. Headline PREFACE checkpoints use mean pooling followed by the controller MLP; the token-mixing sweep in Appendix Table 14 is post-hoc and did not select these checkpoints. We report full validation results averaged over seeds 13, 21, and 42 unless otherwise noted. The GPT-2 small ARC-Challenge study uses five seeds: 7, 13, 21, 42, and 99. ARC-Challenge uses all 1,119 training examples, with no rows discarded during preprocessing. The study is designed as a controlled efficiency evaluation rather than a broad benchmark comparison.

MethodParamsARC-CSciQNote
IA318,43224.4±0.3<u>51.5±0.9</u>static scale
Prompt tuning15,36027.4±2.349.0±0.320 virtual tokens
Prefix tuning1,302,65631.7±2.532.3±0.6projected prefix
PREFACE-IA3-small147,540<u>28.7±2.8</u>52.6±0.5IA3 + prefix controller

Table 26. Prompt-side PEFT baselines on DistilGPT-2 under the same training and scoring pipeline. Bold marks the best and underline the second-best score. Prefix tuning is strongest on ARC-Challenge at 1.3M parameters; PREFACE-IA3-small is strongest on SciQ with fewer parameters.

References

  1. [1]Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 1–9.
  2. [2]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence.arxiv.org/abs/1911.11641
  3. [3]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.arxiv.org/abs/1803.05457
  4. [4]Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, and Tianle Cai. 2026. In-place test-time training. In International Conference on Learning Representations.arxiv.org/abs/2604.06169
  5. [5]Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.arxiv.org/abs/2012.14913
  6. [6]David Ha, Andrew M. Dai, and Quoc V. Le. 2017. Hypernetworks are in. In International Conference on Learning Representations.
  7. [7]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790–2799.arxiv.org/abs/1902.00751
  8. [8]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  9. [9]Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.arxiv.org/abs/2104.08691
  10. [10]Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4582–4597.
  11. [11]Vijay Lingam, Atula Tejaswi Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Funsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. 2024. Svft: Parameter-efficient fine-tuning with singular vectors. In Advances in Neural Information Processing Systems.arxiv.org/abs/2405.19597
  12. [12]Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems.arxiv.org/abs/2205.05638
  13. [13]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32100–32121. PMLR.
  14. [14]Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, and Aaron Xuxiang Tian. 2024b. PARA: Parameter-efficient fine-tuning with prompt-aware representation adjustment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 728–737, Miami, Florida, US. Association for Computational Linguistics.arxiv.org/abs/2502.01033
  15. [15]Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. PiSSA: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems.arxiv.org/abs/2404.02948
  16. [16]Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503, Online. Association for Computational Linguistics.arxiv.org/abs/2005.00247
  17. [17]Qwen Team. 2026. Qwen3.5-0.8B. Hugging Face model card.
  18. [18]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.arxiv.org/abs/1907.10641
  19. [19]Chengwei Sun, Jiwei Wei, Yujia Wu, Yiming Shi, Shiyuan He, Zeyu Ma, Ning Xie, and Yang Yang. 2024. Svfit: Parameter-efficient fine-tuning of large pre-trained models using singular values. arXiv preprint arXiv:2409.05926.arxiv.org/abs/2409.05926
  20. [20]Qi Sun, Edoardo Cetin, and Yujin Tang. 2025. Transformer-squared: Self-adaptive LLMs. In International Conference on Learning Representations.arxiv.org/abs/2501.06252
  21. [21]Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexaei A. Efros, and Moritz Hardt. 2020. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning.arxiv.org/abs/1909.13231
  22. [22]Pengwei Tang, Xiaolin Hu, and Yong Liu. 2025. ADePT: Adaptive decomposed prompt tuning for parameter-efficient fine-tuning. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2501.03291
  23. [23]Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209.
  24. [24]Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2024. ReFT: Representation fine-tuning for language models. In Advances in Neural Information Processing Systems.
  25. [25]Tianfang Xie, Tianjing Li, Wei Zhu, Wei Han, and Yi Zhao. 2026. PEDRO: Parameter-efficient fine-tuning with prompt dependent representation modification. In Neural Information Processing (ICONIP 2024), volume 15295 of Lecture Notes in Computer Science, pages 253–268. Springer.DOI
  26. [26]Hao Zhao, Jie Fu, and Zhaofeng He. 2023. Prototype-based hyperadapter for sample-efficient multi-task tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.arxiv.org/abs/2310.11670

Paper details

vlnc:W_xcf2u2hu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI assistance

View statement
Looking for related papers…
Contents