betaFeedback

WorksComputer Science

Reasoning Supervision and Capability Retention in LoRA Domain Adaptation

Abstract

Fine-tuning foundation models on financial data has a wide range of applications, including identifying event types from news, monitoring risk and compliance language, and answering questions over filings and earnings calls. In this setting, the main challenge is to improve task performance without degrading the generalization capabilities of the model. In this paper, we present an extensive set of ablations for fine-tuning Qwen3-8B with low-rank adaptation (LoRA). We compare three training regimes: (1) response-only LoRA, where the supervised response is the label alone; (2) single-adapter reasoning, where reasoning traces from a teacher model are included alongside the labels in the training loss; and (3) dual-adapter composite training, where separate LoRA adapters are trained on traces and labels. Response-only LoRA improves in-domain performance and leaves general capabilities near those of the frozen model. The in-domain change is largest for calibration and argmax accuracy. Under single-adapter reasoning, the in-domain metrics remain behind those of the response-only regime, while MMLU-Pro can rise slightly and the other retention benchmarks stay within ordinary variation. Dual-adapter composite training remains on the same side of this comparison and yields no further gain. Finally, we compress the teacher traces by summarizing them before training, and show that shorter summaries move calibration toward the response-only regime without reaching it.

Introduction

Low-rank adaptation (LoRA) [1] is a widely used fine-tuning method for resource-constrained settings. For a weight matrix W∈Rm×nW \in\mathbb{R}^{m \times n} of a linear layer, LoRA keeps WW frozen and parameterizes its update as the scaled low-rank product ΔW=(α/r)AB\Delta W = (\alpha/r) AB, where A∈Rm×rA \in\mathbb{R}^{m \times r} and B∈Rr×nB \in\mathbb{R}^{r \times n} are the trainable factors (or adapters), r≪min⁡(m,n)r \ll\min(m,n) is the rank, and α\alpha is a scaling coefficient. This reduces the number of trainable parameters per layer from mnmn to r(m+n)r(m+n), which is what makes fine-tuning feasible on limited hardware.

LoRA makes domain specialization computationally cheap. Yet specialization is only useful if the adapted model keeps its broad capabilities. After fine-tuning, it should still generalize to unseen problems, follow instructions, and express calibrated uncertainty. Retaining these capabilities is especially challenging for classification-like tasks with long inputs. In this setting, the adapter must learn a specialized decision boundary while preserving the base model’s general reasoning.

One potential solution to this problem is to distill knowledge from a larger, more capable model (the teacher) into the student during fine-tuning. Teacher-generated reasoning traces show the intermediate steps that lead to the final answer and thus provide far more supervision per example than the answer alone. However, these traces also contain repetition, hedging, self-correction, and prose specific to the teacher, and training on full traces asks the student to reproduce all of it. Reasoning traces therefore offer a rich domain signal, but whether the student keeps its general capabilities depends on how this signal changes the model during fine-tuning.

Training on traces from a larger model is slow, because each example contains many more tokens than the answer alone. This makes trace-based fine-tuning hard to run in low-resource settings, which are the main use case for LoRA. One way to address this is to summarize the teacher traces before they are used to fine-tune the student, so that the student is trained on only a fraction of the original tokens. However, the choice of compression method and compression ratio is critical: too much compression can remove the information that makes the traces useful and leave fine-tuning with little to learn from.

In this study, we examine how supervising with the response (or label) alone, compared with supervising with a teacher reasoning trace, affects domain performance and capability retention. We study two ways of incorporating traces into LoRA fine-tuning. The first is a blended loss, where a single adapter is trained on a weighted combination of the trace loss and the label loss, with a weight that controls how much each part contributes. The second is dedicated adapters, where the rank budget is split between one adapter for the trace and one for the label, and the two are merged into a single update at inference. The trace is either the full teacher trace or a summary of it, and we vary the degree of compression. All comparisons use matched training configurations, so that a consistent trend can be separated from a single favorable setting.

Contributions. Our main contributions are as follows:

  1. We present a systematic study of reasoning supervision in LoRA fine-tuning for a financial classification task. We use a dataset of 200K examples as our main training set.

  2. To this end, we first build a strong response-only baseline by tuning the LoRA hyperparameters of a Qwen3-8B [2] model on our dataset.

  3. We study how to incorporate the reasoning traces of a larger teacher model into LoRA fine-tuning through a blended label/trace objective, and we ablate the weight that balances the two terms.

  4. We propose a composite adapter that uses separate LoRA modules for the labels and the traces and ablate the allocation of ranks between the two adapters under a fixed rank budget.

  5. We summarize the teacher traces before training and ablate the effect of several compression levels.

  6. We evaluate all configurations on both the in-domain task and a set of retention benchmarks, so that gains on the target task can be weighed against losses in general capabilities.

The rest of the paper is organized as follows. § 2 reviews related work. § 3 describes the dataset, the supervision methods, and the staged protocol. § 4 reports the results. § 5 presents the conclusions. Appendix § 6 contains the hyperparameter search (Appendix § 6.§ 1), the detailed results for the response-only, single-adapter reasoning, and dual-adapter composite stages (Appendices § 6.§ 2, § 6.§ 3, and § 6.§ 4), the trace-compression prompt and the resulting summary lengths (Appendix § 6.§ 5), and the detailed results for the summarized-traces stage (Appendix § 6.6).

Related Work

Low-rank adaptation (LoRA) updates a frozen model through trainable low-rank matrices [1], while QLoRA extends this setup to quantized base models [3]. A small trainable footprint does not, however, guarantee retention. Compared with full fine-tuning, LoRA often learns less and forgets less, with rank controlling the tradeoff [4]; forgetting also grows with the number of updated parameters and optimization steps [5]. With sufficient capacity, broad layer coverage, and appropriate tuning, LoRA can nevertheless match full fine-tuning [6]. This dependence on the training regime motivates evaluating adaptation and retention jointly rather than treating LoRA as a capability-preserving baseline.

Prior financial language models span mixed financial and general pretraining in BloombergGPT [7], efficient specialization in FinGPT [8], and instruction and evaluation resources such as FinMA and FLARE [9]. The closest work to our setting studies label-only and blended chain-of-thought support for LoRA and logit distillation on pruned students [10]. At a fixed student size, blended LoRA is stable across three seeds and surpasses blended chain-of-thought distillation on MMLU and MMLU-Pro at the largest data budgets. Its classification gains plateau, while gold-label NLL and Brier score first improve and then worsen. We instead keep a dense Qwen3-8B backbone fixed and compare response-only LoRA, in which the response is the label alone; single-adapter reasoning; and dual-adapter composite LoRA under matched budgets.

Our reasoning supervision can also be understood as sequence distillation. Classical knowledge distillation matches softened teacher and student distributions [11]; sequence distillation replaces that distribution with the teacher’s highest-scoring decoded response [12]. Our sampled traces are likewise hard sequence targets. Prior rationale distillation has trained rationale generation and label prediction as separate tasks [13], whereas we train on a single response containing a trace followed by its label. We treat the trace as a verbalized rationale rather than as a faithful account of the teacher’s internal process.

Which rationales help depends on both the student and the objective. The useful level of granularity varies with student capacity, and stronger teachers do not always yield better students [14]. Existing methods have separated rationale and answer learning [15] or combined LoRA experts trained on different reasoning tasks [16]. Our narrower intervention trains two always-active additive factors on a single response containing a trace and a label. The trace factor receives both losses, the label factor receives only the label loss, and the total rank is matched to the single-adapter control. Because the factors merge into one update, the intervention changes the training procedure rather than the inference architecture.

Trace length introduces a related efficiency tradeoff. Prior work shows that full traces give the highest accuracy, while compressed traces reduce training and inference cost and usually outperform truncation at the same length [17]. We therefore compare full traces with semantic summaries at several compression levels, rather than assuming that shorter traces are inherently better or worse.

Methods of LoRA Supervision

In this section, we first describe the models and dataset used in our study, along with practical considerations such as hyperparameter tuning. We then describe the supervision methods used in LoRA fine-tuning. Finally, we present the staged protocol that organizes our study.

Dataset Generation and Model Architecture

We study LoRA fine-tuning in the financial domain. To this end, we fine-tune Qwen3-8B [2] on a training set of 200K examples whose inputs are headlines from the FinHeadlineMix dataset [18].1 As supervision, we use the labels and reasoning traces generated by Qwen3-32B for the same inputs. In the rest of the paper, we refer to the 8B and 32B models as the “student” and the “teacher”, respectively. The choice of teacher and dataset size follows [10].

Response-only LoRA

In the first stage, we apply LoRA fine-tuning to the student using only the labels from our dataset. This allows us to assess how training on the labels affects domain performance and retention on MMLU-Pro, GPQA, and IFEval. In this stage, we also perform hyperparameter tuning to find the best settings, which we then use in the subsequent stages. This establishes how much domain utility and calibration the available labels can provide while preserving the model’s held-out capabilities. Across all experiments, the LoRA rank is r=32r = 32.

Single-adapter Reasoning

In the second stage, we apply a reasoning method which keeps a single adapter and changes only the supervision signal. For traced examples, the teacher response is serialized as a reasoning region followed by the final label. The adapter is trained with a per-example, region-balanced loss

Li=λLˉlabel,i+(1−λ)Lˉtrace,i,(1)\mathcal{L}_i = \lambda\bar{\mathcal{L}}_{\mathrm{label},i} + (1-\lambda)\bar{\mathcal{L}}_{\mathrm{trace},i}, \tag*{(1)}

where each bar denotes the mean cross-entropy over the tokens in that region and λ\lambda controls the weight given to the label. Because the loss is averaged within each region, the weight of the trace does not depend on its length: a long chain of thought cannot dominate the label simply by having more tokens. A high λ\lambda focuses training on the label, while a low λ\lambda encourages imitating the teacher’s reasoning. In our experiments, we ablate a range of λ\lambda values to study the role of traces during fine-tuning. We note that λ=1\lambda= 1 is equivalent to response-only training in terms of the loss in Eq. (1). However, the two settings are not identical: although the losses are equivalent, both the trace and the label tokens are passed to the model when λ=1\lambda= 1 (see § 4 for details).

Dual-adapter Composite LoRA

The third stage applies a second reasoning method, which uses the same serialized regions and the same blended loss (§ 3.3) but splits the rank budget across two adapters within a single model: a trace adapter and a label adapter. Both adapters are active on every token and are merged into one static update at inference. The trace adapter is trained by the full blended loss, and the label adapter is trained by the label term only. This comparison keeps the total rank, training budget, and evaluation protocol fixed relative to single-adapter reasoning. For a given rank budget rr, we evaluate three cases:

  • Equal Split, (rtrace,rlabel)=(r/2,r/2)(r_{\mathrm{trace}}, r_{\mathrm{label}}) = (r/2, r/2): the rank is divided evenly between the trace and label adapters;

  • Trace-Heavy Split, (rtrace,rlabel)=(3r/4,r/4)(r_{\mathrm{trace}}, r_{\mathrm{label}}) = (3r/4, r/4): more rank is allocated to the trace adapter;

  • Label-Heavy Split, (rtrace,rlabel)=(r/4,3r/4)(r_{\mathrm{trace}}, r_{\mathrm{label}}) = (r/4, 3r/4): more rank is allocated to the label adapter.

In all cases, the ratio α/r\alpha/r is held fixed for each adapter. For a given value of λ\lambda, selected from the top-performing runs of the previous stage, we ablate all three configurations to evaluate the rank allocation in the dual-adapter setup.

Summarized Traces

In the fourth stage, we repeat our reasoning experiments with summarized traces, for both the single and the composite adapter. Following [17], we summarize each trace with our 32B teacher model to a target token budget. To this end, each trace is wrapped in a prompt that asks the model to rewrite it as a summarized version of a given length. We ablate several length ratios to measure the effect of summarization on the in-domain and retention performance of the fine-tuned model. In each case, we allow the model to generate up to 10% (with a constant offset) more tokens than the target, so that the summary is not cut off before it is complete. The implementation details and the results of this ablation are presented in § 4.6.

Sequential Experimental Design

Comparing fine-tuning schemes is only meaningful if differences in accuracy reflect the schemes themselves rather than the hyperparameters chosen for any one of them. However, a full hyperparameter search is impractical because the search space grows exponentially. We therefore organize the study in stages, each building on the runs selected in the previous one. We first evaluate the frozen model without fine-tuning and use it as the primary baseline. The fine-tuning experiments then proceed in four stages:

Stage 1. We run response-only LoRA (§ 3.2) over the selected hyperparameter grid and keep the runs that satisfy the selection criterion. Details of the hyperparameter selection are given in Appendix § 6.§ 1.

Stage 2. Starting from these runs, we train single-adapter reasoning LoRA (§ 3.3) over a range of λ\lambda values in Eq. (1) and keep the top runs, satisfying the selection criterion.

Stage 3. For each run kept in Stage 2, we train dual-adapter composite LoRA (§ 3.4) with equal, trace-heavy, and label-heavy splits. We repeat these experiments at the λ\lambda values adjacent to the selected one, so that the final comparison does not depend on the exact λ\lambda chosen in Stage 2.

Stage 4. Finally, we repeat the experiments of Stage 2 and Stage 3 with summarized traces, where each trace in the training data is replaced by its summary (Section 3.5). We ablate different summarization levels in this stage.

Figure 1 summarizes the above stages in our study.

Figure 1. Overview of the staged protocol. The teacher (Qwen3-32B) labels 200K FinHeadlineMix headlines and generates a reasoning trace for each. We then fine-tune the student (Qwen3-8B) with LoRA in four stages: (1) response-only LoRA, trained on the label alone over a hyperparameter grid, (2) single-adapter reasoning, trained on the trace and the label with the blended loss of Eq. (1), (3) dual-adapter composite LoRA, which splits the rank budget between a trace adapter and a label adapter, and (4) summarized traces, which repeats Stages 2 and 3 with teacher-generated summaries at target lengths of 25%, 50%, and 75%. In every stage, each run uses three seeds and is evaluated on the in-domain task and the retention benchmarks. A run is carried to the next stage when it outperforms the frozen model on every task. If none does, we fall back to the run with the highest average retention score among those that beat the frozen model on the in-domain tasks.

Selection criterion. At every stage we apply the same rule: a run is kept only if it outperforms the frozen baseline on all tasks, both in-domain tasks and retention (general-capability) tasks. We use this rule to satisfy an important practical consideration: a model that gains in-domain accuracy at the cost of its general capabilities is of limited use. Using one rule throughout also keeps the comparison fair across schemes. When no run satisfies this rule, we fall back to the run with the highest average retention score among those that outperform the baseline on all in-domain tasks. Full task details are given in § 4.1.

Experimental Results

This section presents the experimental results of our study. For each experiment stage, we first describe the setup and methodology, then report the results and discuss the findings.

Training and Evaluation Setup

All experiments fine-tune a single base model, Qwen3-8B, with LoRA on a dataset of headline inputs from FinHeadlineMix [18], annotated by the teacher model (see § 3.1). We train every model with AdamW [19] for a single epoch, repeat each configuration with three random seeds, and report the mean and standard deviation across seeds. The training hyperparameters are ablated in § 4.3. We evaluate every fine-tuned model along two axes: (i) In-domain and (ii) Retention.

In-domain Tasks. We measure in-domain performance on a held-out, class-stratified set of 10,000 financial-headline event-classification examples. We report greedy accuracy (“Acc”) and macro-F1 (“F1”) over the 35 event classes. To assess calibration, we additionally report the negative log-likelihood (“NLL”) and Brier score [20] (“Brier”) of the model’s label distribution, together with argmax accuracy (“Acc. argmax”). Argmax accuracy scores each of the 35 class strings as a teacher-forced continuation of the prompt and counts a prediction as correct when the gold class receives the highest likelihood. Unlike greedy accuracy, which parses the label from the model’s greedy generation and therefore also penalizes formatting failures, argmax accuracy isolates how well the model’s class-token distribution ranks the correct label (it is computed from the same distribution as NLL and Brier).

Retention of General Capabilities Tasks. We measure retention on three public benchmarks: MMLU-Pro [21] for broad multi-task knowledge and reasoning, GPQA-Diamond [22] (“GPQA”) for graduate-level scientific reasoning, and IFEval [23] (“IFEval”) for instruction following, for which we report prompt-level strict accuracy. All evaluations use greedy decoding with the model’s thinking mode disabled and the same prompt template as in training, so that scores are directly comparable across all fine-tuning configurations and against the untuned base model.

Baseline Results

Table 1 reports the performance of the untuned base model (Qwen3-8B without fine-tuning) on all metrics. Each evaluation is run three times, and we report the mean and standard deviation (std). The largest run-to-run variation is on GPQA (std ≈ 4.3%) among the retention benchmarks and on macro-F1 (std ≈ 1.1%) among the in-domain metrics. GPQA contains only 198 questions, so each one is worth roughly 0.5 percentage points, and the questions are difficult enough that the model sits near the decision boundary on many of them, so the score moves noticeably. Macro-F1 is similarly sensitive because it weights every class equally regardless of its size: for small classes, a few flipped predictions can shift that class’s F1 substantially.

Task TypeMetricRun 1Run 2Run 3Mean ± Std
In-domainNLL (↓\downarrow)5.69935.70085.70335.7011 ± 0.0020
In-domainBrier (↓\downarrow)0.85950.85960.85970.8596 ± 0.0001
In-domainAcc. argmax (↑\uparrow)0.52640.52640.52600.5262 ± 0.0002
In-domainMacro-F1 (↑\uparrow)0.65060.63130.65210.6447 ± 0.0116
In-domainAcc (↑\uparrow)0.77390.77090.77310.7726 ± 0.0016
RetentionGPQA (↑\uparrow)0.47980.56570.53540.5269 ± 0.0435
RetentionIFEval (↑\uparrow)0.83730.85580.83550.8429 ± 0.0112
RetentionMMLU-Pro (↑\uparrow)0.62230.62220.62290.6225 ± 0.0004

Table 1. In-domain and retention results for the frozen Qwen3-8B model over three independent evaluation runs (mean ± std). (↑): higher is better, (↓): lower is better.

Response-only LoRA Results

Response-only LoRA improves in-domain performance and leaves general capabilities near the frozen model. The in-domain change is largest for calibration and for which class the model ranks first. Greedy accuracy and macro-F1 move less. We LoRA-fine-tune Qwen3-8B on this dataset with AdamW for a single epoch, and we ablate the LoRA scaling coefficient α\alpha, the peak and minimum learning rate, the AdamW β1\beta_1, and the number of warm-up steps (see Appendix § 6.§ 1 for hyperparameter details). We then select the top runs that outperform the frozen base model on all tasks, both in-domain and retention. This criterion yields 5 runs for the 200K dataset, and these are the runs carried forward to Stage 2.

Figure 2 shows the effect of each hyperparameter on the top runs of response-only LoRA. Learning rate and α\alpha have the largest impact: top runs favor low learning rates (4 of the 5 top runs use 5×10−55 \times10^{-5}) and small α\alpha (4 of the 5 use α=8\alpha= 8, and the fifth uses α=16\alpha= 16). Consistent with this, the effective LoRA learning rate appears to be the decisive factor as three of the five top runs have effective step size 1.25×10−51.25 \times10^{-5}, and the other two sit at 2.5×10−52.5 \times10^{-5}. Among the remaining parameters, β1=0.8\beta_1 = 0.8 and a minimum learning rate of 1×10−51 \times10^{-5} give better results, while the number of warm-up steps has little effect. We present the complete top-run results in Appendix § 6.§ 2.

Distribution of the top runs over the hyperparameter grid

Figure 2. Distribution of the top runs over the hyperparameter grid for the 200K dataset (5 runs) for response-only LoRA (Stage 1). The last plot uses the effective step size αη/r\alpha\eta/r with r=32r = 32, where η\eta is the learning rate.

Single-adapter Reasoning Results

Single-adapter reasoning on full traces leaves the in-domain metrics behind the response-only run that each configuration starts from. Calibration, argmax accuracy, macro-F1, and NLL remain behind those runs. MMLU-Pro can rise slightly, while GPQA and IFEval stay within ordinary variation. We apply the region-balanced loss (see § 3.3) to the top runs from Stage 1. We train each run with six different λ\lambda values {0.1,0.3,0.5,0.7,0.9,1}\{0.1, 0.3, 0.5, 0.7, 0.9, 1\} as in Eq. (1) and select the top runs that satisfy our criterion (defined in § 3.5).

Only two runs in this stage (bold squares, ■, in Figure 3) outperform the frozen model on all tasks. They remain behind the response-only runs they start from on macro-F1, argmax accuracy, NLL, and Brier. Although the loss in Eq. (1) is equivalent to the response-only objective at λ=1.0\lambda= 1.0, the λ=1.0\lambda= 1.0 runs perform far worse than response-only training. The reason is that the training inputs differ: in the reasoning setting, the model is trained on sequences that include the reasoning text, which changes the context in which the label is learned. We present the complete results in Appendix § 6.§ 3.

Figure 3. Single-adapter reasoning (Stage 2) across λ\lambda values, initialized from the top response-only runs of Table 4. Each curve is one hyperparameter configuration (α\alpha, LR, min. LR, β1\beta_1, warm-up); the leftmost point (“Response-only”) is the corresponding Stage 1 run. Points are means over three seeds and error bars show one standard deviation. Gray horizontal lines mark the frozen baseline model. Bold squares (■) mark the two selected top runs, which improve over the baseline on every retention and in-domain metric.

Dual-adapter Composite LoRA Results

Dual-adapter training adds nothing beyond the single adapter. Against the response-only runs, calibration and argmax accuracy stay behind, as they do for the single adapter. Against the single adapter, retention behaves the same way, and the split of rank between the trace adapter and the label adapter does not separate the runs. For each top run from Stage 2, we train dual-adapter models with the selected λ\lambda as well as two adjacent λ\lambda values. The number of trainable parameters is kept fixed and the total rank is r=32r = 32, as in the previous stages. Following § 3.4, we evaluate the splits (rtrace,rlabel)=(16,16)(r_{\mathrm{trace}}, r_{\mathrm{label}}) = (16, 16), (24,8)(24, 8), and (8,24)(8, 24) for each run, with α/r\alpha/r held fixed per adapter.

Figure 4 shows this comparison on the retention benchmarks and on the in-domain metrics. MMLU-Pro remains the retention benchmark that moves, as it does for the single adapter. IFEval stays near the frozen model and near the single-adapter and response-only runs. GPQA differences stay within the run-to-run variation of the frozen model. NLL is lowest for response-only training, whose objective is the label itself and so matches the in-domain task directly. Brier, argmax accuracy, macro-F1, and greedy accuracy follow the same pattern. The full results are in Appendix § 6.§ 4.

Figure 4. Dual-adapter composite stage (Stage 3). Each row corresponds to one top run from the single-adapter reasoning stage, trained with different optimizer settings, and is evaluated at the same λ\lambda as that run and at adjacent λ\lambda values. All runs share α=8\alpha= 8, a learning rate of 5×10−55 \times10^{-5}, and a minimum learning rate of 1×10−51 \times10^{-5}.

Summarized Traces Results

In the fourth stage, we study the effect of training on summarized reasoning traces. To construct length-controlled summaries, we compress each teacher-generated trace with the teacher model itself (Qwen3-32B, greedy decoding) under a token-level length budget. Given a trace of TT tokens and a target length ratio K∈(0,1]K \in(0, 1], we instruct the teacher to rewrite the trace as a condensed version of approximately ⌊K⋅T⌉\lfloor K \cdot T \rceil tokens, where ⌊⋅⌉\lfloor\cdot\rceil denotes rounding to the nearest integer. The summary must preserve the essential reasoning steps, the decisive evidence, and the final category choice, while keeping the first-person deliberative style of the original trace (the exact prompt and implementation details are given in Appendix § 6.§ 5). To prevent runaway generations without forcing mid-sentence truncation, we cap the output at 1.1⋅⌊K⋅T⌉+321.1 \cdot\lfloor K \cdot T \rceil+ 32 tokens, i.e., 10% above the target length plus a fixed margin of 32 tokens. Table 2 shows the results of applying summarization on our dataset with different ratios.

Table 2: Target vs. realized trace-compression ratios of the compressed training sets, measured in tokens on 200K dataset. K^\hat{K} is the per-trace ratio of compressed to original trace length (mean ±\pm std); the aggregate ratio is the total compressed token count divided by the total original token count. The first row gives the uncompressed source traces.

Target KKK^\hat{K} (mean ±\pm std)Aggregate token ratioTrace tokens (mean ±\pm std)
Original—1.000474±315474 \pm315
0.250.252±0.0600.252 \pm0.0600.233110±57110 \pm57
0.500.426±0.0700.426 \pm0.0700.414196±129196 \pm129
0.750.582±0.0870.582 \pm0.0870.576273±193273 \pm193

Table 2.

Figure 5 shows training on compressed traces for the single adapter and the composite adapter. The horizontal axis is the target length. The 25% target is reached; the 50% and 75% targets come out shorter than requested (Table 2). The prompt is in Appendix § 6.§ 5. Shorter summaries move NLL and Brier toward the response-only run, which stays ahead on both. Macro-F1 stays about the same across lengths. Greedy accuracy is lower at the 50% and 75% targets and close to the full trace at the 25% target. IFEval sits about two points lower on the summaries than on the full trace, in both carried runs, and the seed intervals overlap. MMLU-Pro and GPQA stay within the seed variation. The composite adapter follows the single adapter, and no summary reaches the response-only run on NLL or Brier. The rank split is visible mainly in NLL, where more rank on the label adapter tends to leave NLL worse. The detailed results are in Appendix § 6.6.

Figure 5. Effect of summarizing the reasoning traces (Stage 4) in the top runs of Stages 2 and 3. The xx-axis is the target length of the summarized trace relative to the original (100% = no summary). The two runs are β1=0.8\beta_1=0.8 with no warm-up (λ=0.7\lambda=0.7) and β1=0.9\beta_1=0.9 with 400 warm-up steps (λ=0.5\lambda=0.5), and use α=8\alpha=8, LR 0.5×10−40.5 \times10^{-4} and min. LR 1×10−51 \times10^{-5}. Points are means over three seeds, error bars show one standard deviation, and the gray line is the frozen baseline.

Conclusion

We presented a systematic study of reasoning supervision in LoRA fine-tuning for a financial classification task, comparing response-only training with two ways of using teacher reasoning traces: blending the trace into the loss of a single adapter and splitting the rank budget between a trace adapter and a label adapter. We also studied the effect of summarizing the traces before training. All regimes were compared under a staged protocol with matched hyperparameters and evaluated on both the in-domain task and retention benchmarks.

Response-only LoRA is the regime that improves in-domain performance while leaving general capabilities near the frozen model. The gain is largest for calibration and for which class the model ranks first. Adding a full teacher trace does not extend this gain. The in-domain metrics stay behind the response-only runs, MMLU-Pro can rise slightly, and the other retention benchmarks stay within ordinary variation. Splitting the rank across a trace adapter and a label adapter does not change this comparison. The allocation only affects NLL, which worsens as more rank is given to the label adapter. Summarizing the traces moves calibration back toward the response-only regime without reaching it, lowers IFEval by a few points, and reduces accuracy and macro-F1 at intermediate summary lengths. The composite adapter follows the single adapter at every summary length.

Several extensions remain open. The same comparison can be repeated at other data scales and on other classification-like financial tasks, such as risk and compliance language or questions over filings. Attaching a trace to only part of the training examples would test whether the MMLU-Pro lift can be kept at a smaller cost to calibration. Decision models in the style of Jev [24] return one choice from a fixed list together with a probability. They address the same problems and have not been compared with these LoRA regimes. Finally, all of our experiments use AdamW, and the comparison could be repeated with other optimizers.

Appendix

Hyperparameter Search for Response-only Stage

In this stage, we perform an extensive hyperparameter search. Table 3 summarizes the fixed and searched hyperparameters used in this stage. The full grid amounts to 432 training runs for response-only hyperparameter tuning.

HyperparameterValues
FixedTraining examples in Dataset{200K}\{200\mathrm{K}\}
FixedSeeds{617,1234,2468}\{617, 1234, 2468\}
FixedLoRA rank (rr)32
FixedAdam β2\beta_{2}0.999
FixedGlobal batch size256
FixedMicro batch size (MBS)2
SearchedLoRA α\alpha{8,16,32}\{8, 16, 32\}
SearchedLearning rate{5×10−5,1×10−4,2×10−4,5×10−4}\{5 \times10^{-5}, 1 \times10^{-4}, 2 \times10^{-4}, 5 \times10^{-4}\}
SearchedMin. learning rate{2.5×10−6,1×10−5}\{2.5 \times10^{-6}, 1 \times10^{-5}\}
SearchedAdam β1\beta_{1}{0.8,0.9}\{0.8, 0.9\}
SearchedWarm-up steps (ww){0,200,400}\{0, 200, 400\}

Table 3. Hyperparameters for response-only LoRA. For Learning rate, we use linear warm-up followed by cosine decay to a floor (Min. learning rate). Each experiment has 781 training steps.

Detailed Results for Response-only Stage

Table 4 lists the top runs defined in § 4.3.

α\alphaLR (×10−4\times10^{-4})Min. LR (×10−5\times10^{-5})β1\beta_{1}Warm-upGPQA (↑\uparrow)IFEval (↑\uparrow)MMLU-Pro (↑\uparrow)Acc. (↑\uparrow)Macro-F1 (↑\uparrow)Acc. argmax (↑\uparrow)NLL (↓\downarrow)Brier (↓\downarrow)
Baseline (Frozen Model)0.527±0.0440.527 \pm0.0440.843±0.0110.843 \pm0.0110.622±0.0000.622 \pm0.0000.773±0.0020.773 \pm0.0020.645±0.0120.645 \pm0.0120.526±0.0000.526 \pm0.0005.701±0.0025.701 \pm0.0020.860±0.0000.860 \pm0.000
80.50.250.800.534±0.0210.534 \pm0.0210.843±0.0090.843 \pm0.0090.625±0.0020.625 \pm0.0020.806±0.0040.806 \pm0.0040.690±0.0060.690 \pm0.0060.825±0.0020.825 \pm0.0020.582±0.0100.582 \pm0.0100.267±0.0020.267 \pm0.002
80.510.800.539±0.0240.539 \pm0.0240.845±0.0050.845 \pm0.0050.625±0.0020.625 \pm0.0020.806±0.0030.806 \pm0.0030.698±0.0100.698 \pm0.0100.833±0.0020.833 \pm0.0020.545±0.0110.545 \pm0.0110.254±0.0020.254 \pm0.002
8110.82000.537±0.0320.537 \pm0.0320.845±0.0040.845 \pm0.0040.628±0.0040.628 \pm0.0040.807±0.0030.807 \pm0.0030.703±0.0070.703 \pm0.0070.841±0.0010.841 \pm0.0010.562±0.0100.562 \pm0.0100.245±0.0040.245 \pm0.004
160.50.250.94000.545±0.0130.545 \pm0.0130.847±0.0060.847 \pm0.0060.627±0.0010.627 \pm0.0010.807±0.0040.807 \pm0.0040.691±0.0060.691 \pm0.0060.844±0.0030.844 \pm0.0030.476±0.0110.476 \pm0.0110.234±0.0040.234 \pm0.004
80.510.94000.530±0.0350.530 \pm0.0350.848±0.0060.848 \pm0.0060.626±0.0000.626 \pm0.0000.800±0.0010.800 \pm0.0010.693±0.0100.693 \pm0.0100.799±0.0030.799 \pm0.0030.847±0.0130.847 \pm0.0130.324±0.0050.324 \pm0.005

Table 4. Top runs of the response-only stage (Stage 1). Each configuration is trained with three random seeds and we report the mean ±\pm standard deviation. Hyperparameters follow Table 3.

Detailed Results for Single-adapter Reasoning Stage

Table 5 presents the detailed runs defined in § 4.4.

Detailed Results for Dual-adapter Composite Stage

Table 6 presents the detailed runs defined in § 4.5.

λ\lambda(rtrace,rlabel)(r_{\mathrm{trace}}, r_{\mathrm{label}})GPQA (↑)IFEval (↑)MMLU-Pro (↑)Acc. (↑)Macro-F1 (↑)Acc. argmax (↑)NLL (↓)Brier (↓)
Baseline (Frozen Model)0.527±0.0440.527 \pm0.0440.843±0.0110.843 \pm0.0110.622±0.0000.622 \pm0.0000.773±0.0020.773 \pm0.0020.645±0.0120.645 \pm0.0120.526±0.0000.526 \pm0.0005.701±0.0025.701 \pm0.0020.860±0.0000.860 \pm0.000
β1=0.8\beta_1=0.8, Warm-up=0
Response-only0.539±0.0240.539 \pm0.0240.845±0.0050.845 \pm0.0050.625±0.0020.625 \pm0.0020.806±0.0030.806 \pm0.0030.698±0.0100.698 \pm0.0100.833±0.0020.833 \pm0.0020.545±0.0110.545 \pm0.0110.254±0.0020.254 \pm0.002
Reasoning loss, single adapter (λ=0.7\lambda=0.7)0.529±0.0190.529 \pm0.0190.850±0.0160.850 \pm0.0160.628±0.0020.628 \pm0.0020.801±0.0020.801 \pm0.0020.676±0.0080.676 \pm0.0080.722±0.0020.722 \pm0.0020.934±0.0160.934 \pm0.0160.413±0.0040.413 \pm0.004
0.50.5(8,24)(8, 24)0.500±0.0320.500 \pm0.0320.845±0.0060.845 \pm0.0060.627±0.0010.627 \pm0.0010.797±0.0040.797 \pm0.0040.672±0.0030.672 \pm0.0030.696±0.0030.696 \pm0.0031.062±0.0101.062 \pm0.0100.451±0.0040.451 \pm0.004
0.50.5(16,16)(16, 16)0.522±0.0300.522 \pm0.0300.828±0.0140.828 \pm0.0140.629±0.0000.629 \pm0.0000.798±0.0020.798 \pm0.0020.675±0.0150.675 \pm0.0150.717±0.0020.717 \pm0.0020.936±0.0170.936 \pm0.0170.419±0.0040.419 \pm0.004
0.50.5(24,8)(24, 8)0.535±0.0320.535 \pm0.0320.833±0.0120.833 \pm0.0120.626±0.0020.626 \pm0.0020.795±0.0010.795 \pm0.0010.675±0.0080.675 \pm0.0080.721±0.0060.721 \pm0.0060.869±0.0300.869 \pm0.0300.403±0.0100.403 \pm0.010
0.70.7(8,24)(8, 24)0.535±0.0330.535 \pm0.0330.837±0.0060.837 \pm0.0060.634±0.0030.634 \pm0.0030.797±0.0020.797 \pm0.0020.666±0.0060.666 \pm0.0060.676±0.0060.676 \pm0.0061.174±0.0311.174 \pm0.0310.480±0.0080.480 \pm0.008
0.70.7(16,16)(16, 16)0.488±0.0180.488 \pm0.0180.839±0.0080.839 \pm0.0080.631±0.0010.631 \pm0.0010.796±0.0020.796 \pm0.0020.656±0.0050.656 \pm0.0050.698±0.0050.698 \pm0.0051.045±0.0201.045 \pm0.0200.447±0.0060.447 \pm0.006
0.70.7(24,8)(24, 8)0.505±0.0220.505 \pm0.0220.837±0.0010.837 \pm0.0010.633±0.0000.633 \pm0.0000.797±0.0020.797 \pm0.0020.675±0.0110.675 \pm0.0110.712±0.0040.712 \pm0.0040.965±0.0200.965 \pm0.0200.426±0.0060.426 \pm0.006
0.90.9(8,24)(8, 24)0.525±0.0220.525 \pm0.0220.832±0.0100.832 \pm0.0100.637±0.0020.637 \pm0.0020.771±0.0040.771 \pm0.0040.641±0.0130.641 \pm0.0130.634±0.0060.634 \pm0.0061.578±0.0051.578 \pm0.0050.561±0.0050.561 \pm0.005
0.90.9(16,16)(16, 16)0.527±0.0130.527 \pm0.0130.837±0.0080.837 \pm0.0080.637±0.0010.637 \pm0.0010.777±0.0030.777 \pm0.0030.640±0.0060.640 \pm0.0060.643±0.0070.643 \pm0.0071.437±0.0281.437 \pm0.0280.539±0.0050.539 \pm0.005
0.90.9(24,8)(24, 8)0.530±0.0050.530 \pm0.0050.843±0.0170.843 \pm0.0170.637±0.0020.637 \pm0.0020.776±0.0030.776 \pm0.0030.634±0.0080.634 \pm0.0080.649±0.0050.649 \pm0.0051.338±0.0321.338 \pm0.0320.523±0.0060.523 \pm0.006
β1=0.9\beta_1=0.9, Warm-up=400
Response-only0.530±0.0350.530 \pm0.0350.848±0.0060.848 \pm0.0060.626±0.0000.626 \pm0.0000.800±0.0010.800 \pm0.0010.693±0.0100.693 \pm0.0100.799±0.0030.799 \pm0.0030.847±0.0130.847 \pm0.0130.324±0.0050.324 \pm0.005
Reasoning loss, single adapter (λ=0.5\lambda=0.5)0.556±0.0100.556 \pm0.0100.847±0.0080.847 \pm0.0080.630±0.0020.630 \pm0.0020.801±0.0020.801 \pm0.0020.666±0.0090.666 \pm0.0090.705±0.0030.705 \pm0.0031.050±0.0141.050 \pm0.0140.445±0.0050.445 \pm0.005
0.30.3(8,24)(8, 24)0.512±0.0180.512 \pm0.0180.839±0.0100.839 \pm0.0100.630±0.0010.630 \pm0.0010.799±0.0020.799 \pm0.0020.667±0.0220.667 \pm0.0220.675±0.0020.675 \pm0.0021.227±0.0161.227 \pm0.0160.491±0.0030.491 \pm0.003
0.30.3(16,16)(16, 16)0.508±0.0030.508 \pm0.0030.839±0.0060.839 \pm0.0060.631±0.0020.631 \pm0.0020.802±0.0010.802 \pm0.0010.661±0.0040.661 \pm0.0040.688±0.0010.688 \pm0.0011.124±0.0071.124 \pm0.0070.466±0.0010.466 \pm0.001
0.30.3(24,8)(24, 8)0.525±0.0100.525 \pm0.0100.844±0.0070.844 \pm0.0070.628±0.0010.628 \pm0.0010.801±0.0050.801 \pm0.0050.667±0.0080.667 \pm0.0080.701±0.0020.701 \pm0.0021.025±0.0031.025 \pm0.0030.443±0.0020.443 \pm0.002
0.50.5(8,24)(8, 24)0.524±0.0030.524 \pm0.0030.840±0.0040.840 \pm0.0040.634±0.0010.634 \pm0.0010.796±0.0010.796 \pm0.0010.664±0.0100.664 \pm0.0100.665±0.0010.665 \pm0.0011.308±0.0201.308 \pm0.0200.507±0.0020.507 \pm0.002
0.50.5(16,16)(16, 16)0.527±0.0250.527 \pm0.0250.845±0.0110.845 \pm0.0110.633±0.0010.633 \pm0.0010.795±0.0030.795 \pm0.0030.652±0.0190.652 \pm0.0190.684±0.0020.684 \pm0.0021.167±0.0251.167 \pm0.0250.475±0.0050.475 \pm0.005
0.50.5(24,8)(24, 8)0.535±0.0180.535 \pm0.0180.843±0.0050.843 \pm0.0050.632±0.0020.632 \pm0.0020.798±0.0010.798 \pm0.0010.660±0.0050.660 \pm0.0050.694±0.0040.694 \pm0.0041.103±0.0131.103 \pm0.0130.460±0.0050.460 \pm0.005
0.70.7(8,24)(8, 24)0.520±0.0450.520 \pm0.0450.842±0.0030.842 \pm0.0030.635±0.0020.635 \pm0.0020.781±0.0020.781 \pm0.0020.647±0.0060.647 \pm0.0060.643±0.0030.643 \pm0.0031.403±0.0171.403 \pm0.0170.536±0.0030.536 \pm0.003
0.70.7(16,16)(16, 16)0.527±0.0300.527 \pm0.0300.826±0.0060.826 \pm0.0060.634±0.0030.634 \pm0.0030.789±0.0010.789 \pm0.0010.651±0.0020.651 \pm0.0020.659±0.0020.659 \pm0.0021.320±0.0081.320 \pm0.0080.515±0.0030.515 \pm0.003
0.70.7(24,8)(24, 8)0.510±0.0090.510 \pm0.0090.833±0.0020.833 \pm0.0020.638±0.0010.638 \pm0.0010.790±0.0040.790 \pm0.0040.648±0.0100.648 \pm0.0100.666±0.0030.666 \pm0.0031.270±0.0211.270 \pm0.0210.502±0.0050.502 \pm0.005

Table 6. Dual-adapter composite stage (Stage 3) for the two selected configurations, for adjacent λ\lambda values and different rank splits (rtrace,rlabel)(r_{\mathrm{trace}}, r_{\mathrm{label}}). Each run is trained with three random seeds and we report the mean ±\pm standard deviation. For reference, each block starts with the response-only run of that configuration (Table 4) and its selected single-adapter run with the reasoning loss (Table 5). Both configurations use α=8\alpha=8, LR=0.5×10−40.5 \times10^{-4} and Min. LR=1×10−51 \times10^{-5}.

λ\lambdaGPQA (↑)IFEval (↑)MMLU-Pro (↑)Acc. (↑)Macro-F1 (↑)Acc. argmax (↑)NLL (↓)Brier (↓)
Baseline (Frozen Model)0.527 ± 0.0440.843 ± 0.0110.622 ± 0.0000.773 ± 0.0020.645 ± 0.0120.526 ± 0.0005.701 ± 0.0020.860 ± 0.000
α=8\alpha=8, LR=0.5×10−40.5\times10^{-4}, Min. LR=0.25×10−50.25\times10^{-5}, β1=0.8\beta_1=0.8, Warm-up=0
Response-only0.534 ± 0.0210.843 ± 0.0090.625 ± 0.0020.806 ± 0.0040.690 ± 0.0060.825 ± 0.0020.582 ± 0.0100.267 ± 0.002
0.10.532 ± 0.0290.829 ± 0.0160.628 ± 0.0010.811 ± 0.0020.694 ± 0.0030.728 ± 0.0020.834 ± 0.0050.395 ± 0.003
0.30.527 ± 0.0080.835 ± 0.0050.629 ± 0.0020.799 ± 0.0020.681 ± 0.0080.728 ± 0.0040.860 ± 0.0280.398 ± 0.008
0.50.517 ± 0.0250.830 ± 0.0130.627 ± 0.0040.800 ± 0.0010.679 ± 0.0120.724 ± 0.0030.889 ± 0.0010.405 ± 0.002
0.70.502 ± 0.0440.839 ± 0.0160.633 ± 0.0030.799 ± 0.0040.672 ± 0.0060.705 ± 0.0031.033 ± 0.0210.441 ± 0.005
0.90.519 ± 0.0520.841 ± 0.0080.639 ± 0.0040.780 ± 0.0020.647 ± 0.0180.631 ± 0.0031.474 ± 0.0180.556 ± 0.005
1.00.505 ± 0.0150.831 ± 0.0130.626 ± 0.0030.763 ± 0.0030.656 ± 0.0070.565 ± 0.0064.653 ± 0.1030.754 ± 0.014
α=8\alpha=8, LR=0.5×10−40.5\times10^{-4}, Min. LR=1×10−51\times10^{-5}, β1=0.8\beta_1=0.8, Warm-up=0
Response-only0.539 ± 0.0240.845 ± 0.0050.625 ± 0.0020.806 ± 0.0030.698 ± 0.0100.833 ± 0.0020.545 ± 0.0110.254 ± 0.002
0.10.535 ± 0.0050.827 ± 0.0070.625 ± 0.0010.812 ± 0.0040.680 ± 0.0070.730 ± 0.0050.803 ± 0.0240.388 ± 0.007
0.30.495 ± 0.0170.835 ± 0.0110.628 ± 0.0040.805 ± 0.0010.686 ± 0.0020.737 ± 0.0030.798 ± 0.0140.381 ± 0.003
0.50.530 ± 0.0150.841 ± 0.0020.629 ± 0.0030.799 ± 0.0020.680 ± 0.0040.736 ± 0.0010.818 ± 0.0060.384 ± 0.002
■ 0.70.529 ± 0.0190.850 ± 0.0160.628 ± 0.0020.801 ± 0.0020.676 ± 0.0080.722 ± 0.0020.934 ± 0.0160.413 ± 0.004
0.90.532 ± 0.0260.834 ± 0.0040.637 ± 0.0030.783 ± 0.0040.635 ± 0.0040.648 ± 0.0021.335 ± 0.0160.523 ± 0.002
1.00.513 ± 0.0300.835 ± 0.0050.623 ± 0.0010.764 ± 0.0010.651 ± 0.0020.567 ± 0.0044.551 ± 0.0790.746 ± 0.010
α=8\alpha=8, LR=1×10−41\times10^{-4}, Min. LR=1×10−51\times10^{-5}, β1=0.8\beta_1=0.8, Warm-up=200
Response-only0.537 ± 0.0320.845 ± 0.0040.628 ± 0.0040.807 ± 0.0030.703 ± 0.0070.841 ± 0.0010.562 ± 0.0100.245 ± 0.004
0.10.524 ± 0.0450.826 ± 0.0080.625 ± 0.0040.820 ± 0.0040.691 ± 0.0070.723 ± 0.0030.857 ± 0.0050.405 ± 0.004
0.30.527 ± 0.0030.832 ± 0.0150.629 ± 0.0010.813 ± 0.0030.688 ± 0.0050.740 ± 0.0020.771 ± 0.0070.375 ± 0.004
0.50.545 ± 0.0050.829 ± 0.0140.627 ± 0.0010.808 ± 0.0020.674 ± 0.0010.745 ± 0.0020.777 ± 0.0100.372 ± 0.002
0.70.530 ± 0.0050.831 ± 0.0120.628 ± 0.0030.801 ± 0.0020.672 ± 0.0100.738 ± 0.0010.845 ± 0.0150.386 ± 0.002
0.90.500 ± 0.0090.828 ± 0.0080.636 ± 0.0010.789 ± 0.0010.653 ± 0.0060.682 ± 0.0011.180 ± 0.0210.475 ± 0.005
1.00.524 ± 0.0110.840 ± 0.0150.627 ± 0.0010.763 ± 0.0030.646 ± 0.0080.571 ± 0.0094.394 ± 0.1370.734 ± 0.017
α=16\alpha=16, LR=0.5×10−40.5\times10^{-4}, Min. LR=0.25×10−50.25\times10^{-5}, β1=0.9\beta_1=0.9, Warm-up=400
Response-only0.545 ± 0.0130.847 ± 0.0060.627 ± 0.0010.807 ± 0.0040.691 ± 0.0060.844 ± 0.0030.476 ± 0.0110.234 ± 0.004
0.10.505 ± 0.0270.834 ± 0.0210.626 ± 0.0020.810 ± 0.0010.687 ± 0.0170.726 ± 0.0060.837 ± 0.0240.400 ± 0.008
0.30.542 ± 0.0200.822 ± 0.0030.628 ± 0.0010.808 ± 0.0020.692 ± 0.0190.739 ± 0.0040.762 ± 0.0090.374 ± 0.004
0.50.540 ± 0.0250.832 ± 0.0110.627 ± 0.0010.804 ± 0.0030.686 ± 0.0020.739 ± 0.0010.776 ± 0.0040.376 ± 0.001
0.70.559 ± 0.0330.835 ± 0.0050.627 ± 0.0030.800 ± 0.0010.675 ± 0.0110.732 ± 0.0020.873 ± 0.0110.398 ± 0.004
0.90.508 ± 0.0110.839 ± 0.0090.639 ± 0.0030.787 ± 0.0010.651 ± 0.0260.687 ± 0.0041.158 ± 0.0430.472 ± 0.009
1.00.510 ± 0.0130.839 ± 0.0130.623 ± 0.0020.761 ± 0.0020.640 ± 0.0120.574 ± 0.0064.389 ± 0.1390.732 ± 0.012
α=8\alpha=8, LR=0.5×10−40.5\times10^{-4}, Min. LR=1×10−51\times10^{-5}, β1=0.9\beta_1=0.9, Warm-up=400
Response-only0.530 ± 0.0350.848 ± 0.0060.626 ± 0.0000.800 ± 0.0010.693 ± 0.0100.799 ± 0.0030.847 ± 0.0130.324 ± 0.005
0.10.492 ± 0.0110.840 ± 0.0110.625 ± 0.0010.806 ± 0.0010.679 ± 0.0050.725 ± 0.0000.874 ± 0.0110.403 ± 0.003
0.30.519 ± 0.0150.841 ± 0.0140.627 ± 0.0020.806 ± 0.0030.670 ± 0.0060.713 ± 0.0070.959 ± 0.0360.425 ± 0.010
■ 0.50.556 ± 0.0100.847 ± 0.0080.630 ± 0.0020.801 ± 0.0020.666 ± 0.0090.705 ± 0.0031.050 ± 0.0140.445 ± 0.005
0.70.542 ± 0.0280.837 ± 0.0100.634 ± 0.0030.792 ± 0.0030.653 ± 0.0020.676 ± 0.0021.213 ± 0.0090.489 ± 0.009
0.90.503 ± 0.0080.840 ± 0.0030.637 ± 0.0020.768 ± 0.0040.644 ± 0.0200.617 ± 0.0011.706 ± 0.0630.591 ± 0.009
1.00.505 ± 0.0050.836 ± 0.0120.624 ± 0.0020.766 ± 0.0020.652 ± 0.0030.561 ± 0.0054.841 ± 0.1030.762 ± 0.012

Table 5. Single-adapter reasoning stage (Stage 2; 200K dataset, rank r=32r=32) for each of the top response-only configurations of Table 4 and each λ\lambda. Each configuration is trained with three random seeds and we report the mean ± standard deviation; the Response-only row of each block repeats the corresponding run of Table 4. Bold rows (■) are the two top runs that outperform the baseline on all tasks.

Details of Trace-Compression

Each compression request uses the prompt below, sent through the teacher’s chat template with thinking mode disabled and greedy decoding. For a trace of TT tokens and a target length ratio KK, {target_tokens} = round(K⋅TK \cdot T), and {target_words} and {ratio_pct} give the same budget as an approximate word count and as a percentage, respectively. {trace} is the original reasoning trace. Generation is hard-capped at 1.1⋅1.1 \cdot round(K⋅TK \cdot T) + 32 tokens, and outputs that reach the cap are trimmed back to the last complete sentence. Table 2 reports the resulting trace lengths for each length ratio on the 200K dataset.

Below is a reasoning trace that explains how a financial news headline was classified
into an event category. Rewrite it as a condensed version that preserves the essential
reasoning steps and the conclusion.

Requirements:
- The condensed version must be approximately {target_tokens} tokens long (roughly
{target_words} words; about {ratio_pct}% of the original length). Stay within 5% of that
target length. Your output will be cut off shortly past it, so finish your reasoning and
state the category choice before the limit.
- Keep the same first-person deliberative style as the original.
- Preserve the decisive evidence and the final category choice.
- Output ONLY the condensed reasoning text, with no preamble, headers, or quotation
marks.

Original reasoning trace:
---
{trace}
---

Detailed Results for Summarized Traces Stage

Tables 7, 8, and 9 present the detailed runs defined in § 4.6.

Length ratioRetentionIn-domain task
GPQA (↑)IFEval (↑)MMLU-Pro (↑)Acc. (↑)Macro-F1 (↑)Acc. argmax (↑)NLL (↓)Brier (↓)
Baseline (Frozen Model)0.527 ± 0.0440.843 ± 0.0110.622 ± 0.0000.773 ± 0.0020.645 ± 0.0120.526 ± 0.0005.701 ± 0.0020.860 ± 0.000
β1=0.8\beta_1=0.8, Warm-up=0, λ=0.7\lambda=0.7
Response-only0.539 ± 0.0240.845 ± 0.0050.625 ± 0.0020.806 ± 0.0030.698 ± 0.0100.833 ± 0.0020.545 ± 0.0110.254 ± 0.002
25%0.554 ± 0.0180.829 ± 0.0070.630 ± 0.0010.799 ± 0.0040.681 ± 0.0080.769 ± 0.0020.837 ± 0.0240.353 ± 0.005
50%0.515 ± 0.0400.828 ± 0.0100.630 ± 0.0010.766 ± 0.0080.657 ± 0.0130.751 ± 0.0030.870 ± 0.0160.375 ± 0.002
75%0.520 ± 0.0130.826 ± 0.0060.628 ± 0.0020.714 ± 0.0030.652 ± 0.0130.736 ± 0.0060.880 ± 0.0330.393 ± 0.006
100% (No summary)0.529 ± 0.0190.850 ± 0.0160.628 ± 0.0020.801 ± 0.0020.676 ± 0.0080.722 ± 0.0020.934 ± 0.0160.413 ± 0.004
β1=0.9\beta_1=0.9, Warm-up=400, λ=0.5\lambda=0.5
Response-only0.530 ± 0.0350.848 ± 0.0060.626 ± 0.0000.800 ± 0.0010.693 ± 0.0100.799 ± 0.0030.847 ± 0.0130.324 ± 0.005
25%0.519 ± 0.0160.829 ± 0.0160.631 ± 0.0000.789 ± 0.0050.673 ± 0.0040.763 ± 0.0030.908 ± 0.0150.365 ± 0.002
50%0.515 ± 0.0280.824 ± 0.0120.627 ± 0.0010.761 ± 0.0080.672 ± 0.0120.731 ± 0.0010.934 ± 0.0700.400 ± 0.009
75%0.529 ± 0.0030.826 ± 0.0140.630 ± 0.0010.703 ± 0.0070.655 ± 0.0170.717 ± 0.0040.987 ± 0.0290.427 ± 0.007
100% (No summary)0.556 ± 0.0100.847 ± 0.0080.630 ± 0.0020.801 ± 0.0020.666 ± 0.0090.705 ± 0.0031.050 ± 0.0140.445 ± 0.005

Table 7. Single-adapter reasoning with summarized traces (Stage 4), for the two top runs. The length ratio is the target length of the summarized trace relative to the original one (100% = no summary). Response-only is the run of the same configuration trained without traces (Table 4). Each run is trained with three random seeds and we report the mean ± standard deviation. Both runs use α=8\alpha=8, LR=0.5×10−40.5 \times10^{-4} and Min. LR=1×10−51 \times10^{-5}.

Table 8. Dual-adapter composite LoRA with summarized traces (Stage 4) for the top run with β1=0.8\beta_1=0.8, warm-up=0 (α=8\alpha=8, LR=0.5×10−40.5 \times10^{-4}, Min. LR=1×10−51 \times10^{-5}), for every length ratio, λ\lambda and rank split (rtrace,rlabelr_{\mathrm{trace}}, r_{\mathrm{label}}). Length ratio 100% corresponds to the original traces. Each block starts with the single-adapter run with the reasoning loss (λ=0.7\lambda=0.7) trained on traces of the same length; Response-only is the run of the same configuration trained without traces (Table 4). Each run is trained with three random seeds and we report the mean ±\pm standard deviation.

λ\lambda(rtrace,rlabel)(r_{\mathrm{trace}}, r_{\mathrm{label}})Retention: GPQA (↑\uparrow)Retention: IFEval (↑\uparrow)Retention: MMLU-Pro (↑\uparrow)In-domain task: Acc. (↑\uparrow)In-domain task: Macro-F1 (↑\uparrow)In-domain task: Acc. argmax (↑\uparrow)In-domain task: NLL (↓\downarrow)In-domain task: Brier (↓\downarrow)
Baseline (Frozen Model)0.527 ±\pm 0.0440.843 ±\pm 0.0110.622 ±\pm 0.0000.773 ±\pm 0.0020.645 ±\pm 0.0120.526 ±\pm 0.0005.701 ±\pm 0.0020.860 ±\pm 0.000
Response-only0.539 ±\pm 0.0240.845 ±\pm 0.0050.625 ±\pm 0.0020.806 ±\pm 0.0030.698 ±\pm 0.0100.833 ±\pm 0.0020.545 ±\pm 0.0110.254 ±\pm 0.002
Length ratio 25%
Single adapter (λ=0.7\lambda=0.7)0.554 ±\pm 0.0180.829 ±\pm 0.0070.630 ±\pm 0.0010.799 ±\pm 0.0040.681 ±\pm 0.0080.769 ±\pm 0.0020.837 ±\pm 0.0240.353 ±\pm 0.005
0.5(8, 24)0.507 ±\pm 0.0310.832 ±\pm 0.0030.631 ±\pm 0.0020.794 ±\pm 0.0040.666 ±\pm 0.0200.750 ±\pm 0.0080.932 ±\pm 0.0280.379 ±\pm 0.009
0.5(16, 16)0.520 ±\pm 0.0380.827 ±\pm 0.0050.631 ±\pm 0.0020.793 ±\pm 0.0040.683 ±\pm 0.0080.773 ±\pm 0.0030.760 ±\pm 0.0060.337 ±\pm 0.000
0.5(24, 8)0.547 ±\pm 0.0130.822 ±\pm 0.0130.631 ±\pm 0.0010.800 ±\pm 0.0040.686 ±\pm 0.0050.781 ±\pm 0.0040.696 ±\pm 0.0270.321 ±\pm 0.005
0.7(8, 24)0.535 ±\pm 0.0130.821 ±\pm 0.0060.629 ±\pm 0.0030.783 ±\pm 0.0040.656 ±\pm 0.0080.727 ±\pm 0.0070.969 ±\pm 0.0220.411 ±\pm 0.008
0.7(16, 16)0.522 ±\pm 0.0030.827 ±\pm 0.0110.633 ±\pm 0.0010.793 ±\pm 0.0020.683 ±\pm 0.0050.754 ±\pm 0.0060.946 ±\pm 0.1110.379 ±\pm 0.014
0.7(24, 8)0.517 ±\pm 0.0150.828 ±\pm 0.0050.632 ±\pm 0.0020.794 ±\pm 0.0120.667 ±\pm 0.0130.759 ±\pm 0.0040.828 ±\pm 0.0240.359 ±\pm 0.005
0.9(8, 24)0.544 ±\pm 0.0370.835 ±\pm 0.0080.630 ±\pm 0.0020.779 ±\pm 0.0010.649 ±\pm 0.0130.663 ±\pm 0.0031.528 ±\pm 0.1330.544 ±\pm 0.015
0.9(16, 16)0.520 ±\pm 0.0100.834 ±\pm 0.0070.630 ±\pm 0.0020.783 ±\pm 0.0040.652 ±\pm 0.0140.670 ±\pm 0.0071.372 ±\pm 0.1090.518 ±\pm 0.019
0.9(24, 8)0.530 ±\pm 0.0350.833 ±\pm 0.0040.630 ±\pm 0.0020.775 ±\pm 0.0060.661 ±\pm 0.0070.693 ±\pm 0.0031.225 ±\pm 0.1090.478 ±\pm 0.016
Length ratio 50%
Single adapter (λ=0.7\lambda=0.7)0.515 ±\pm 0.0400.828 ±\pm 0.0100.630 ±\pm 0.0010.766 ±\pm 0.0080.657 ±\pm 0.0130.751 ±\pm 0.0030.870 ±\pm 0.0160.375 ±\pm 0.002
0.5(8, 24)0.532 ±\pm 0.0110.835 ±\pm 0.0060.633 ±\pm 0.0010.783 ±\pm 0.0060.659 ±\pm 0.0040.729 ±\pm 0.0020.943 ±\pm 0.0220.407 ±\pm 0.002
0.5(16, 16)0.522 ±\pm 0.0150.835 ±\pm 0.0130.629 ±\pm 0.0010.763 ±\pm 0.0080.656 ±\pm 0.0070.750 ±\pm 0.0030.808 ±\pm 0.0070.369 ±\pm 0.001
0.5(24, 8)0.492 ±\pm 0.0290.834 ±\pm 0.0080.629 ±\pm 0.0020.754 ±\pm 0.0040.662 ±\pm 0.0020.762 ±\pm 0.0050.727 ±\pm 0.0180.345 ±\pm 0.008
0.7(8, 24)0.537 ±\pm 0.0530.818 ±\pm 0.0060.629 ±\pm 0.0020.773 ±\pm 0.0030.645 ±\pm 0.0110.717 ±\pm 0.0030.973 ±\pm 0.0470.423 ±\pm 0.007
0.7(16, 16)0.493 ±\pm 0.0340.821 ±\pm 0.0080.630 ±\pm 0.0040.782 ±\pm 0.0060.654 ±\pm 0.0200.722 ±\pm 0.0030.969 ±\pm 0.0340.415 ±\pm 0.006
0.7(24, 8)0.530 ±\pm 0.0390.829 ±\pm 0.0070.632 ±\pm 0.0020.767 ±\pm 0.0070.654 ±\pm 0.0080.744 ±\pm 0.0010.896 ±\pm 0.0170.388 ±\pm 0.003
0.9(8, 24)0.519 ±\pm 0.0110.824 ±\pm 0.0120.631 ±\pm 0.0030.765 ±\pm 0.0050.646 ±\pm 0.0100.666 ±\pm 0.0011.315 ±\pm 0.0290.522 ±\pm 0.005
0.9(16, 16)0.517 ±\pm 0.0240.830 ±\pm 0.0110.634 ±\pm 0.0020.749 ±\pm 0.0030.645 ±\pm 0.0060.676 ±\pm 0.0061.239 ±\pm 0.0640.500 ±\pm 0.012
0.9(24, 8)0.532 ±\pm 0.0150.820 ±\pm 0.0020.631 ±\pm 0.0020.749 ±\pm 0.0020.638 ±\pm 0.0150.681 ±\pm 0.0041.136 ±\pm 0.0490.481 ±\pm 0.010
Length ratio 75%
Single adapter (λ=0.7\lambda=0.7)0.520 ±\pm 0.0130.826 ±\pm 0.0060.628 ±\pm 0.0020.714 ±\pm 0.0030.652 ±\pm 0.0130.736 ±\pm 0.0060.880 ±\pm 0.0330.393 ±\pm 0.006
0.5(8, 24)0.524 ±\pm 0.0190.821 ±\pm 0.0160.624 ±\pm 0.0020.727 ±\pm 0.0050.653 ±\pm 0.0100.714 ±\pm 0.0010.964 ±\pm 0.0240.425 ±\pm 0.003
0.5(16, 16)0.534 ±\pm 0.0150.824 ±\pm 0.0050.625 ±\pm 0.0020.722 ±\pm 0.0130.627 ±\pm 0.0190.740 ±\pm 0.0040.842 ±\pm 0.0180.383 ±\pm 0.006
0.5(24, 8)0.529 ±\pm 0.0130.826 ±\pm 0.0140.623 ±\pm 0.0030.711 ±\pm 0.0060.635 ±\pm 0.0040.747 ±\pm 0.0010.799 ±\pm 0.0120.373 ±\pm 0.002
0.7(8, 24)0.519 ±\pm 0.0440.824 ±\pm 0.0050.626 ±\pm 0.0030.745 ±\pm 0.0010.649 ±\pm 0.0120.697 ±\pm 0.0031.098 ±\pm 0.0260.462 ±\pm 0.004
0.7(16, 16)0.530 ±\pm 0.0050.819 ±\pm 0.0070.626 ±\pm 0.0020.732 ±\pm 0.0060.641 ±\pm 0.0120.711 ±\pm 0.0081.011 ±\pm 0.0310.433 ±\pm 0.008
0.7(24, 8)0.508 ±\pm 0.0300.819 ±\pm 0.0100.628 ±\pm 0.0030.725 ±\pm 0.0110.638 ±\pm 0.0230.726 ±\pm 0.0040.896 ±\pm 0.0130.404 ±\pm 0.001
0.9(8, 24)0.533 ±\pm 0.0090.835 ±\pm 0.0080.631 ±\pm 0.0020.738 ±\pm 0.0060.660 ±\pm 0.0190.655 ±\pm 0.0031.344 ±\pm 0.0750.535 ±\pm 0.010
0.9(16, 16)0.495 ±\pm 0.0270.845 ±\pm 0.0050.632 ±\pm 0.0030.730 ±\pm 0.0020.648 ±\pm 0.0060.656 ±\pm 0.0051.422 ±\pm 0.1000.542 ±\pm 0.017
0.9(24, 8)0.525 ±\pm 0.0570.829 ±\pm 0.0050.632 ±\pm 0.0020.727 ±\pm 0.0010.637 ±\pm 0.0190.661 ±\pm 0.0031.346 ±\pm 0.0350.528 ±\pm 0.006
Length ratio 100% (no summary)
Single adapter (λ=0.7\lambda=0.7)0.529 ±\pm 0.0190.850 ±\pm 0.0160.628 ±\pm 0.0020.801 ±\pm 0.0020.676 ±\pm 0.0080.722 ±\pm 0.0020.934 ±\pm 0.0160.413 ±\pm 0.004
0.5(8, 24)0.500 ±\pm 0.0320.845 ±\pm 0.0060.627 ±\pm 0.0010.797 ±\pm 0.0040.672 ±\pm 0.0030.696 ±\pm 0.0031.062 ±\pm 0.0100.451 ±\pm 0.004
0.5(16, 16)0.522 ±\pm 0.0300.828 ±\pm 0.0140.629 ±\pm 0.0000.798 ±\pm 0.0020.675 ±\pm 0.0150.717 ±\pm 0.0020.936 ±\pm 0.0170.419 ±\pm 0.004
0.5(24, 8)0.535 ±\pm 0.0320.833 ±\pm 0.0120.626 ±\pm 0.0020.795 ±\pm 0.0010.675 ±\pm 0.0080.721 ±\pm 0.0060.869 ±\pm 0.0300.403 ±\pm 0.010
0.7(8, 24)0.535 ±\pm 0.0330.837 ±\pm 0.0060.634 ±\pm 0.0030.797 ±\pm 0.0020.666 ±\pm 0.0060.676 ±\pm 0.0061.174 ±\pm 0.0310.480 ±\pm 0.008
0.7(16, 16)0.488 ±\pm 0.0180.839 ±\pm 0.0080.631 ±\pm 0.0010.796 ±\pm 0.0020.656 ±\pm 0.0050.698 ±\pm 0.0051.045 ±\pm 0.0200.447 ±\pm 0.006
0.7(24, 8)0.505 ±\pm 0.0220.837 ±\pm 0.0010.633 ±\pm 0.0000.797 ±\pm 0.0020.675 ±\pm 0.0110.712 ±\pm 0.0040.965 ±\pm 0.0200.426 ±\pm 0.006
0.9(8, 24)0.525 ±\pm 0.0220.832 ±\pm 0.0100.637 ±\pm 0.0020.771 ±\pm 0.0040.641 ±\pm 0.0130.634 ±\pm 0.0061.578 ±\pm 0.0050.561 ±\pm 0.005
0.9(16, 16)0.527 ±\pm 0.0130.837 ±\pm 0.0080.637 ±\pm 0.0010.777 ±\pm 0.0030.640 ±\pm 0.0060.643 ±\pm 0.0071.437 ±\pm 0.0280.539 ±\pm 0.005
0.9(24, 8)0.530 ±\pm 0.0050.843 ±\pm 0.0170.637 ±\pm 0.0020.776 ±\pm 0.0030.634 ±\pm 0.0080.649 ±\pm 0.0051.338 ±\pm 0.0320.523 ±\pm 0.006

:::

Table 9. Table 9: Dual-adapter composite LoRA with summarized traces (Stage 4) for the top run with β1=0.9\beta_1=0.9, warm-up=400 (α=8\alpha=8, LR=0.5×10−40.5 \times10^{-4}, Min. LR=1×10−51 \times10^{-5}), for every length ratio, λ\lambda and rank split (rtrace,rlabelr_{\mathrm{trace}}, r_{\mathrm{label}}). Length ratio 100% corresponds to the original traces. Each block starts with the single-adapter run with the reasoning loss (λ=0.5\lambda=0.5) trained on traces of the same length; Response-only is the run of the same configuration trained without traces (Table 4). Each run is trained with three random seeds and we report the mean ±\pm standard deviation.

RetentionIn-domain task
λ\lambda(rtrace,rlabel)(r_{\mathrm{trace}}, r_{\mathrm{label}})GPQA (↑\uparrow)IFEval (↑\uparrow)MMLU-Pro (↑\uparrow)Acc. (↑\uparrow)Macro-F1 (↑\uparrow)Acc. argmax (↑\uparrow)NLL (↓\downarrow)Brier (↓\downarrow)
Baseline (Frozen Model)0.527 ±\pm 0.0440.843 ±\pm 0.0110.622 ±\pm 0.0000.773 ±\pm 0.0020.645 ±\pm 0.0120.526 ±\pm 0.0005.701 ±\pm 0.0020.860 ±\pm 0.000
Response-only0.530 ±\pm 0.0350.848 ±\pm 0.0060.626 ±\pm 0.0000.800 ±\pm 0.0010.693 ±\pm 0.0100.799 ±\pm 0.0030.847 ±\pm 0.0130.324 ±\pm 0.005
Length ratio 25%
Single adapter (λ=0.5\lambda=0.5)0.519 ±\pm 0.0160.829 ±\pm 0.0160.631 ±\pm 0.0000.789 ±\pm 0.0050.673 ±\pm 0.0040.763 ±\pm 0.0030.908 ±\pm 0.0150.365 ±\pm 0.002
0.3(8, 24)0.502 ±\pm 0.0180.834 ±\pm 0.0080.631 ±\pm 0.0020.792 ±\pm 0.0040.662 ±\pm 0.0020.707 ±\pm 0.0011.338 ±\pm 0.0770.451 ±\pm 0.007
0.3(16, 16)0.517 ±\pm 0.0320.821 ±\pm 0.0040.630 ±\pm 0.0010.794 ±\pm 0.0030.679 ±\pm 0.0080.745 ±\pm 0.0070.975 ±\pm 0.0880.383 ±\pm 0.014
0.3(24, 8)0.527 ±\pm 0.0420.827 ±\pm 0.0040.630 ±\pm 0.0030.794 ±\pm 0.0030.664 ±\pm 0.0090.751 ±\pm 0.0080.946 ±\pm 0.0500.374 ±\pm 0.012
0.5(8, 24)0.500 ±\pm 0.0180.826 ±\pm 0.0080.634 ±\pm 0.0030.789 ±\pm 0.0040.652 ±\pm 0.0080.717 ±\pm 0.0031.112 ±\pm 0.0520.430 ±\pm 0.006
0.5(16, 16)0.532 ±\pm 0.0370.826 ±\pm 0.0080.630 ±\pm 0.0010.789 ±\pm 0.0040.657 ±\pm 0.0090.738 ±\pm 0.0041.051 ±\pm 0.0280.399 ±\pm 0.004
0.5(24, 8)0.529 ±\pm 0.0190.825 ±\pm 0.0090.630 ±\pm 0.0020.787 ±\pm 0.0090.674 ±\pm 0.0170.759 ±\pm 0.0010.884 ±\pm 0.0170.365 ±\pm 0.003
0.7(8, 24)0.525 ±\pm 0.0150.834 ±\pm 0.0090.630 ±\pm 0.0030.777 ±\pm 0.0040.650 ±\pm 0.0010.692 ±\pm 0.0051.218 ±\pm 0.0910.472 ±\pm 0.013
0.7(16, 16)0.519 ±\pm 0.0130.829 ±\pm 0.0060.629 ±\pm 0.0010.790 ±\pm 0.0010.663 ±\pm 0.0090.716 ±\pm 0.0061.033 ±\pm 0.0390.427 ±\pm 0.008
0.7(24, 8)0.539 ±\pm 0.0280.826 ±\pm 0.0060.631 ±\pm 0.0010.784 ±\pm 0.0020.659 ±\pm 0.0100.722 ±\pm 0.0041.050 ±\pm 0.0430.420 ±\pm 0.008
Length ratio 50%
Single adapter (λ=0.5\lambda=0.5)0.515 ±\pm 0.0280.824 ±\pm 0.0120.627 ±\pm 0.0010.761 ±\pm 0.0080.672 ±\pm 0.0120.731 ±\pm 0.0010.934 ±\pm 0.0700.400 ±\pm 0.009
0.3(8, 24)0.529 ±\pm 0.0150.834 ±\pm 0.0140.628 ±\pm 0.0050.774 ±\pm 0.0060.658 ±\pm 0.0110.705 ±\pm 0.0041.077 ±\pm 0.0340.438 ±\pm 0.003
0.3(16, 16)0.545 ±\pm 0.0300.828 ±\pm 0.0100.628 ±\pm 0.0010.766 ±\pm 0.0090.683 ±\pm 0.0140.725 ±\pm 0.0021.030 ±\pm 0.0160.418 ±\pm 0.003
0.3(24, 8)0.520 ±\pm 0.0050.828 ±\pm 0.0110.629 ±\pm 0.0010.766 ±\pm 0.0060.665 ±\pm 0.0080.731 ±\pm 0.0051.000 ±\pm 0.0580.407 ±\pm 0.009
0.5(8, 24)0.522 ±\pm 0.0080.821 ±\pm 0.0160.629 ±\pm 0.0010.778 ±\pm 0.0030.668 ±\pm 0.0140.694 ±\pm 0.0001.159 ±\pm 0.0380.459 ±\pm 0.005
0.5(16, 16)0.500 ±\pm 0.0170.834 ±\pm 0.0090.631 ±\pm 0.0010.774 ±\pm 0.0030.662 ±\pm 0.0110.715 ±\pm 0.0060.985 ±\pm 0.0380.424 ±\pm 0.008
0.5(24, 8)0.483 ±\pm 0.0080.832 ±\pm 0.0040.626 ±\pm 0.0040.759 ±\pm 0.0050.657 ±\pm 0.0130.722 ±\pm 0.0030.983 ±\pm 0.0170.417 ±\pm 0.002
0.7(8, 24)0.530 ±\pm 0.0350.826 ±\pm 0.0070.629 ±\pm 0.0040.758 ±\pm 0.0030.646 ±\pm 0.0090.673 ±\pm 0.0061.250 ±\pm 0.0270.500 ±\pm 0.004
0.7(16, 16)0.513 ±\pm 0.0290.826 ±\pm 0.0030.630 ±\pm 0.0020.759 ±\pm 0.0030.648 ±\pm 0.0110.683 ±\pm 0.0021.206 ±\pm 0.0640.477 ±\pm 0.008
0.7(24, 8)0.515 ±\pm 0.0180.823 ±\pm 0.0090.632 ±\pm 0.0010.762 ±\pm 0.0050.648 ±\pm 0.0110.695 ±\pm 0.0011.074 ±\pm 0.0290.455 ±\pm 0.006
Length ratio 75%
Single adapter (λ=0.5\lambda=0.5)0.529 ±\pm 0.0030.826 ±\pm 0.0140.630 ±\pm 0.0010.703 ±\pm 0.0070.655 ±\pm 0.0170.717 ±\pm 0.0040.987 ±\pm 0.0290.427 ±\pm 0.007
0.3(8, 24)0.532 ±\pm 0.0320.829 ±\pm 0.0080.624 ±\pm 0.0020.734 ±\pm 0.0060.656 ±\pm 0.0160.697 ±\pm 0.0051.042 ±\pm 0.0350.448 ±\pm 0.007
0.3(16, 16)0.513 ±\pm 0.0120.837 ±\pm 0.0050.628 ±\pm 0.0050.708 ±\pm 0.0170.646 ±\pm 0.0110.719 ±\pm 0.0030.925 ±\pm 0.0260.416 ±\pm 0.002
0.3(24, 8)0.524 ±\pm 0.0200.829 ±\pm 0.0030.626 ±\pm 0.0020.713 ±\pm 0.0150.626 ±\pm 0.0040.733 ±\pm 0.0010.914 ±\pm 0.0640.402 ±\pm 0.008
0.5(8, 24)0.517 ±\pm 0.0180.824 ±\pm 0.0140.626 ±\pm 0.0020.729 ±\pm 0.0060.642 ±\pm 0.0130.678 ±\pm 0.0031.194 ±\pm 0.0600.484 ±\pm 0.007
0.5(16, 16)0.505 ±\pm 0.0130.819 ±\pm 0.0070.626 ±\pm 0.0030.706 ±\pm 0.0050.650 ±\pm 0.0060.701 ±\pm 0.0011.008 ±\pm 0.0200.442 ±\pm 0.004
0.5(24, 8)0.532 ±\pm 0.0180.820 ±\pm 0.0070.629 ±\pm 0.0000.704 ±\pm 0.0020.648 ±\pm 0.0080.708 ±\pm 0.0031.009 ±\pm 0.0180.437 ±\pm 0.002
0.7(8, 24)0.475 ±\pm 0.0440.826 ±\pm 0.0130.629 ±\pm 0.0010.721 ±\pm 0.0080.635 ±\pm 0.0070.659 ±\pm 0.0021.417 ±\pm 0.0630.534 ±\pm 0.009
0.7(16, 16)0.513 ±\pm 0.0280.829 ±\pm 0.0090.632 ±\pm 0.0010.712 ±\pm 0.0080.652 ±\pm 0.0090.672 ±\pm 0.0021.169 ±\pm 0.0260.490 ±\pm 0.007
0.7(24, 8)0.529 ±\pm 0.0110.827 ±\pm 0.0030.631 ±\pm 0.0020.706 ±\pm 0.0080.643 ±\pm 0.0080.676 ±\pm 0.0051.198 ±\pm 0.0200.489 ±\pm 0.004
Length ratio 100% (no summary)
Single adapter (λ=0.5\lambda=0.5)0.556 ±\pm 0.0100.847 ±\pm 0.0080.630 ±\pm 0.0020.801 ±\pm 0.0020.666 ±\pm 0.0090.705 ±\pm 0.0031.050 ±\pm 0.0140.445 ±\pm 0.005
0.3(8, 24)0.512 ±\pm 0.0180.839 ±\pm 0.0100.630 ±\pm 0.0010.799 ±\pm 0.0020.667 ±\pm 0.0220.675 ±\pm 0.0021.227 ±\pm 0.0160.491 ±\pm 0.003
0.3(16, 16)0.508 ±\pm 0.0030.839 ±\pm 0.0060.631 ±\pm 0.0020.802 ±\pm 0.0010.661 ±\pm 0.0040.688 ±\pm 0.0011.124 ±\pm 0.0070.466 ±\pm 0.001
0.3(24, 8)0.525 ±\pm 0.0100.844 ±\pm 0.0070.628 ±\pm 0.0010.801 ±\pm 0.0050.667 ±\pm 0.0080.701 ±\pm 0.0021.025 ±\pm 0.0030.443 ±\pm 0.002
0.5(8, 24)0.524 ±\pm 0.0030.840 ±\pm 0.0040.634 ±\pm 0.0010.796 ±\pm 0.0010.664 ±\pm 0.0100.665 ±\pm 0.0011.308 ±\pm 0.0200.507 ±\pm 0.002
0.5(16, 16)0.527 ±\pm 0.0250.845 ±\pm 0.0110.633 ±\pm 0.0010.795 ±\pm 0.0030.652 ±\pm 0.0190.684 ±\pm 0.0021.167 ±\pm 0.0250.475 ±\pm 0.005
0.5(24, 8)0.535 ±\pm 0.0180.843 ±\pm 0.0050.632 ±\pm 0.0020.798 ±\pm 0.0050.660 ±\pm 0.0050.694 ±\pm 0.0041.103 ±\pm 0.0130.460 ±\pm 0.005
0.7(8, 24)0.520 ±\pm 0.0450.842 ±\pm 0.0030.635 ±\pm 0.0020.781 ±\pm 0.0020.647 ±\pm 0.0060.643 ±\pm 0.0031.403 ±\pm 0.0170.536 ±\pm 0.003
0.7(16, 16)0.527 ±\pm 0.0300.826 ±\pm 0.0060.634 ±\pm 0.0030.789 ±\pm 0.0010.651 ±\pm 0.0020.659 ±\pm 0.0021.320 ±\pm 0.0080.515 ±\pm 0.003
0.7(24, 8)0.510 ±\pm 0.0090.833 ±\pm 0.0020.638 ±\pm 0.0010.790 ±\pm 0.0040.648 ±\pm 0.0100.666 ±\pm 0.0031.270 ±\pm 0.0210.502 ±\pm 0.005

References

  1. [1]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  2. [2]Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  3. [3]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.14314.
  4. [4]Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2405.09673.
  5. [5]Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models. arXiv preprint arXiv:2401.05605, 2024. URL https://arxiv.org/abs/2401.05605.
  6. [6]John Schulman and Thinking Machines Lab. LoRA without regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250929. URL https://thinkingmachines.ai/blog/lora/.DOI
  7. [7]Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabrovolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023. URL https://arxiv.org/abs/2303.17564.
  8. [8]Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. FinGPT: Open-source financial large language models. In FinLLM Workshop at IJCAI, 2023. URL https://arxiv.org/abs/2306.06031.
  9. [9]Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. PIXIU: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443, 2023. URL https://arxiv.org/abs/2306.05443.
  10. [10]Lavinia Ghita, Dhruv Desai, and Ioana Boier. Scaling laws for task-specific LLM distillation. arXiv preprint arXiv:2606.24747, 2026. URL https://arxiv.org/abs/2606.24747v2.
  11. [11]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531.
  12. [12]Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.07947.
  13. [13]Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, 2023. URL https://arxiv.org/abs/2305.02301.
  14. [14]Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, and Xiaoyu Shen. Unveiling the key factors for distilling chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15094–15119, 2025. doi: 10.18653/v1/2025.findings-acl.782. URL https://aclanthology.org/2025.findings-acl.782/.DOI
  15. [15]Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. Improve student’s reasoning generalizability through cascading decomposed CoTs distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15623–15643, 2024. doi: 10.18653/v1/2024.emnlp-main.875. URL https://aclanthology.org/2024.emnlp-main.875/.
  16. [16]Xiang Li, Shizhu He, Jiayu Wu, Zhao Yang, Yao Xu, Jun Yang, Haifeng Liu, Kang Liu, and Jun Zhao. MoDE-CoTD: Chain-of-thought distillation for complex reasoning tasks with mixture of decoupled LoRA-experts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 11475–11485, 2024. URL https://aclanthology.org/2024.lrec-main.1003/.DOI
  17. [17]Maxime Griot, Paul Steven Scotti, and Tanishq Mathew Abraham. Compress-Distill: Reasoning trace compression for efficient knowledge distillation. arXiv preprint arXiv:2606.05988, 2026. URL https://arxiv.org/abs/2606.05988.
  18. [18]Dhruv Desai and Lavinia Ghita. FinHeadlineMix: Large-scale synthetic financial news headlines. https://huggingface.co/datasets/nvidia/FinHeadlineMix, 2026. Hugging Face dataset.
  19. [19]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  20. [20]Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
  21. [21]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.01574
  22. [22]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2311.12022
  23. [23]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911
  24. [24]Diogo Almeida. Introducing system one models and Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026. TypeSafe, September 15, 2026.

Paper details

vlnc:W_jew2s958
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents