Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs

Introduction

Which cognitive abilities are required to reason about time? Prior work on large language models (LLMs) commonly decomposes temporal reasoning into two modes: temporal arithmetic, involving calculations over times and dates, and temporal semantics, involving the interpretation of before–after relations and temporal causality [40, 6, 8, 21]. Both require an understanding of time, whereas temporal arithmetic additionally requires mathematical capabilities. This distinction has become a common organising principle for benchmark design, model training, and performance reporting [40, 6, 33, 8]. However, it remains unclear whether it reflects meaningful differences in the cognitive demands tested by current benchmarks.

This question matters because benchmarks may not measure what they claim to measure due to poor construct validity [4]. As benchmarks become more complex, the true bottleneck may diverge from the target ability named in the task description [29, 37]. For example, success on a temporal arithmetic question may depend more on numerical calculation than on temporal understanding, whereas success on a temporal semantics question may depend more on language comprehension or memorised world knowledge. The arithmetic/semantics distinction may therefore classify tasks by their surface form and not by the abilities that actually drive performance.

We test this assumption in two ways. First, we ask whether the arithmetic/semantics distinction predicts transfer across benchmarks. We train LLMs on temporal reasoning datasets and evaluate whether performance improvements generalise to unseen datasets, both within and across the two modes. If each mode reflects a stable capability with shared reasoning patterns, transfer should be strong between datasets in the same mode [31, 17, 20]. Instead, we find that improvements remain highly localised to the training data and barely generalise to unseen data in the same purported mode.

Second, we test whether the arithmetic/semantics split reflects differences in cognitive demands. Using the Demand-Level-Annotation (DeLeAn) rubrics (Zhou et al., 2026), we score each benchmark item on 18 general cognitive dimensions. We compare arithmetic and semantic datasets across all DeLeAn dimensions, but find no clear separation between the two modes. We then focus on the Quantitative Reasoning (QLq) and Logical Reasoning (QLl) cognitive demands, which are most naturally aligned with temporal arithmetic ([33, 8]) and temporal semantics ([44, 35]), respectively. We train models on data with relatively high demands for each of the two dimensions and test whether this produces larger gains in the corresponding mode. It does not. Instead, we find that tasks depend more strongly on the cognitive demands for knowledge.

Taken together, these results suggest that temporal reasoning benchmarks are better characterised as knowledge-heavy tasks than as tests of distinct arithmetic or semantic capabilities, with performance depending more on learned regularities and background knowledge than on explicit reasoning over temporal quantities or relations.

Background

Temporal reasoning benchmarks commonly distinguish tasks requiring semantic temporal understanding from those that also require mathematical operations. For example, Su et al. [33] note that 19 of the 38 tasks in TRAM (Wang and Zhao, [40]) involve mathematical reasoning. TimeBench (Chu et al., [6]), which covers 10 temporal reasoning benchmarks, distinguishes symbolic tasks such as date arithmetic from those requiring only semantic temporal understanding, while Test of Time (ToT) (Fatemi et al., [8]) explicitly has an Arithmetic and a Semantic subset. Across these efforts, the arithmetic/semantic split is treated as a meaningful axis of variation, yet none tests whether it reflects separable underlying abilities.

However, growing evidence questions the construct validity of LLM benchmarks—the extent to which they measure the abilities they claim to measure. Reviewing 445 benchmarks from major NLP and ML venues, Bean et al. [4] find that many studies do not establish a clear link between the ability named by a benchmark and the abilities required to perform well on it. Alaa et al. [2] identify a similar gap in medicine, where high scores on exam-style benchmarks do not reliably predict performance on real clinical data. More broadly, Rameshkumar et al. [29] and Varela et al. [37] show that the main performance bottleneck can diverge from the capability named in the task description as tasks become more complex. These findings suggest that benchmark categories require empirical validation. We apply this principle to temporal reasoning by testing whether the arithmetic/semantic labels of existing benchmarks predict transfer behaviour and cognitive abilities.

If temporal arithmetic and temporal semantics reflect genuinely distinct and coherent abilities, then improvements in one should transfer, at minimum, to other tasks within the same mode. This logic follows a standard rationale in the reasoning literature: training in a narrow domain should generalise to tasks that draw on the same underlying capability. Shao et al. [31] and Guo et al. [13] show that reinforcement learning on mathematical benchmarks reliably transfers to other mathematical tasks and even to non-mathematical reasoning benchmarks, confirming that fine-tuning can yield cross-benchmark transfer when the underlying capability is shared. Whether the same holds for temporal reasoning remains untested. Xiong et al. [44] show that fine-tuning on temporal data can improve temporal reasoning, but do not test cross-mode transfer, and recent work suggests that transfer is not guaranteed across all domains ([17, 19]). We use fine-tuning transfer as a direct probe of the arithmetic/semantic distinction: if gains transfer within mode, the distinction has empirical grounding; if they do not, it lacks support as a description of separable abilities.

Experimental setup

Datasets

We use nine temporal reasoning datasets from eight benchmarks, counting ToT Arithmetic and Semantic separately (Table 1). For the demand-profile analyses and item-level experiments, TCP is further divided into Long and Short subsets, yielding ten subsets. These datasets span a broad range of temporal phenomena, task formats, and dataset construction methods. This variation allows us to test whether the arithmetic/semantic distinction generalises beyond a particular benchmark or task format. We report training, validation, and test splits for each dataset partition. The size of each split is reported in the Appendix (Tables 4 and 5).

Dataset	Format	Temporal phenomena
TCP ([7])	Free-form	Constraint-based planning
ToT Arithmetic ([8])	Free-form	Addition, duration, scheduling, time zones
ToT Semantic ([8])	Free-form	Event relationships, before/after, timelines
TRAM ([40])	Multiple-choice	Order, arithmetic, frequency, duration, NLI
TempReason ([34])	Free-form	Temporal relation extraction
TimeQA ([5])	Free-form	Time-sensitive question answering
MenatQA ([41])	Free-form	Counterfactual, shuffled context, varying time scope
Date Arithmetic ([34])	Free-form	Date calculation
TimeDial ([27])	Multiple-choice	Temporal common sense in dialogue

Cognitive capability labelling

We annotate each benchmark item $X_j$ using the DeLeAn rubric of Zhou et al. (2026). GPT-4o assigns an annotation vector $\mathbf{x}^{(j)} = (\vec{x}^{(j)}_1,\ldots,x^{(j)}_{18}) \in\{0,\ldots,5^+\}^{18}$, where $x^{(j)}_i$ is the item’s demand score on cognitive dimension $i$, with higher values indicating greater demand. The dimensions include knowledge, metacognition, attention, comprehension, Quantitative Reasoning (QLq), and Logical Reasoning (QLl) (full definitions in Appendix § 8).

The DeLeAn rubrics build on established cognitive taxonomies and demonstrate good agreement among human raters and between the human consensus ratings and GPT-4o annotations. Five of the nine datasets in our study (Table 1) were included in the original evaluation of the framework: Date Arithmetic, MenatQA, TempReason, TimeDial, and TimeQA. Nevertheless, we confirm label reliability by relabelling data with Gemini 3.1 Flash Lite¹ and comparing its labels with GPT-4o’s, using within-group interrater reliability (James et al., [22]) as in the original study, but also explore ordinal Krippendorff’s alpha (Hayes and Krippendorff, [15]). More details are in the Appendix § 12.

For the purposes of this study, QLq and QLl are the most relevant dimensions, as they map most directly onto the conventional arithmetic/semantic distinction. QLq captures the numerical demands associated with temporal arithmetic (Su et al., [33]; Fatemi et al., [8]), while QLl captures the relational and causal reasoning demands associated with temporal semantics (Wang and Zhao, [40]; Xiong et al., [44]). By comparing demand levels on these two dimensions at the item level, we can test whether the arithmetic/semantic distinction corresponds to a genuine difference in cognitive requirements, independently of the labels assigned by benchmark designers.

Operationalising the arithmetic/semantic distinction

We use fine-tuning transfer (see 3.3) between temporal arithmetic and semantics as a probe of their separability. If both modes reflect distinct abilities, then training on one should improve performance on tasks from the same mode, and not on tasks from the other. We use two complementary operationalisations to partition the data into these modes.

The first is a dataset-level operationalisation based on existing mode labels. We restrict this controlled transfer analysis to ToT and TRAM, as these are the only benchmarks that provide explicit or reconstructable arithmetic/semantic partitions, minimising confounds from differences in dataset construction, such as annotation assumptions, or task format. ToT provides an explicit split: its Arithmetic subset covers time/date arithmetic, duration, scheduling, and time zone conversion tasks, while its Semantic subset targets event–event/time, before/after, timeline, and first/last relations. TRAM does not provide such a binary partition. Instead, we follow Su et al.’s [33] classification of TRAM subtasks as math-time tasks (equivalent to arithmetic mode) and pure-time tasks (equivalent to semantic mode) (see Appendix § 9.§ 2 for details).

The second is an item-level operationalisation based on DeLeAn annotations (§ 3.1.1). We partition items by their QLq and QLl demands to construct training sets intended to probe both temporal reasoning modes. Specifically, we filter all datasets to items where demand for QLq exceeds that for QLl, and vice versa, yielding six training conditions. These include two zero-demand controls (Tab. 2). It is worth noting that QLq and QLl are rarely the most demanding dimensions in our data. Out of 8,530 training items, QLq is the domi-

caption: Item-level operationalisation based on relative QLq and QLl demand.

:::

nant dimension in only 15 items (1,288 including ties), and QLl in only 1 item (771 including ties). We will discuss the limited role these dimensions seem to play in current benchmarks later.

Non-temporal QL-heavy training

QL demand is low in the temporal data (mean$(\max(\mathrm{QLq},\mathrm{QLl})) = 1.57$; Figure 6). To test whether a stronger QL signal improves temporal arithmetic and semantics, we select non-temporal examples from ChemLLMBench (Guo et al. [14]), MMLU-Pro (Wang et al. [40]), AGIEval (Zhong et al. (2024)), TruthQuest (Mondorf and Plank [26]), OmniMath (Gao et al. [9]), and LiveBench (White et al. [42]). Filtering for $\mathrm{QL}=\max(\mathrm{QLq},\mathrm{QLl})\ge3$ and $\mathrm{QL}>\max_{d\in D\setminus\{\mathrm{QL}\}} d$ yields 428 items with a mean QL demand of 4.13.

Evaluation

We use accuracy for evaluation and additionally normalise temporal expressions before scoring, following Abbood et al. [1]. The expected answer type in the dataset guides the normalisation of the model’s response. E.g., “11 Aug. 2023” and “11/08/2023” are then treated as equivalent. This prevents surface-level differences in temporal formatting from being counted as reasoning failures (details in Appendix § 10).

Aggregate accuracy, however, does not isolate performance on any single cognitive demand dimension, since each item carries a multidimensional DeLeAn label. To disentangle the contribution of individual dimensions, we construct subject characteristic curves using dominant slices. For a dominant dimension $i$ and level $l$, we retain only items satisfying $x_i^{(j)}=l$ and $x_{i'}^{(j)}\le l$ for all $i'\ne i$, i.e., items for which dimension $i$ is the dominant demand. Accuracy is averaged within each level and a two-parameter logistic curve is fitted to the resulting five points, using the weighting scheme of Zhou et al. (2026). We report the area under the curve (AUC) of the fitted curve as the model’s ability on that dimension.

Four of the 18 dimensions are excluded from this analysis due to insufficient coverage in our data. Zhou et al. suggests excluding dimensions that have a range narrower than 1–3 or have fewer than 30 items. Spatio-physical Reasoning (SNs) and Verbal Expression (CEe) span only demand levels 1 and 2, too narrow a range for meaningful curve fitting. Conceptualisation, Learning and Abstraction (CL) and Knowledge of Natural Sciences (KNn) yield only 9 and 25 items, respectively, across all dominant slices, making estimates unreliable.

Fine-tuning

Our main method for investigating the separability between temporal arithmetic and temporal semantics is to fine-tune models on data for each mode and test the transfer of performance to the corresponding mode.

GRPO. Temporal reasoning tasks can require multi-step reasoning. We initially attempted supervised fine-tuning on reasoning traces from a larger teacher model (Hsieh et al. [16]), without success. We attribute this to a mismatch between the teacher’s and student’s response style (Li et al. [24]). Instead, we use Group Relative Policy Optimization (GRPO) (Guo et al. [13]), which sidesteps this issue by optimising directly on self-generated reasoning traces. To ensure comparability across experiments despite substantial variation in dataset sizes, we fix the compute budget to 400 optimisation steps with a batch size of 16. We use 8 generations per rollout for dataset-level operationalisation and 16 for the item-level operationalisation, due to training instabilities with 8 rollouts (details in Appendix § 11).

Reward. We define two reward settings to probe the effect of training from complementary directions. In the accuracy setting, the reward is 1 if the model’s response is correct and 0 otherwise, determined by the semantic equivalence metric described in § 3.2. In the one-minus-accuracy setting, we invert this signal to explore a form of targeted unlearning similar to Zaraduokas et al. (2026). The model receives a reward of 1 only if its answer is incorrect but correctly formatted, 0.5 if the answer is both correct and correctly formatted (for richer feedback), and 0 otherwise. The format requirement mitigates reward hacking (Skalse et al.,

2022. by preventing models from maximising reward through degenerate outputs. If temporal arithmetic and semantics are separable, unlearning one mode should degrade performance primarily on the same mode, leaving the other largely intact. For experiments using the item-level operationalisation, we additionally scale the reward by $1+QLq+QLl$, so that items with higher quantitative or logical demand contribute a stronger optimisation signal. The goal is to offset the low-resolution binary reward signal, which initially provided limited learning signal across many samples.

Models. We run our experiments with six models from three families and two size ranges, ensuring that our findings are not an artefact of a single architecture or scale. Specifically, we use Qwen3-4B, Qwen3-8B (Yang et al. [45]), Llama3.2-3B-Instruct, Llama-3.1-8B-Instruct ([11]), gemma-3-4B-it, and gemma-3-12B-it ([10]). Qwen models are run in non-thinking mode for comparability with the other models. During evaluation, we allow up to 4,000 output tokens and set the temperature to 0 for deterministic decoding.

Results

Cognitive demand profiles do not reflect the temporal reasoning dichotomy

If the arithmetic/semantic distinction reflects a genuine structural property of temporal reasoning, it should be recoverable through the data’s cognitive demands. Specifically, items labelled as temporal arithmetic should exhibit elevated $QLq$ demands relative to $QLl$, while items labelled as temporal semantics should show the reverse. We test this prediction in two ways: first by examining the joint distribution of $QLq$ and $QLl$ across all benchmarks, and then by examining whether the full 18-dimensional demand profiles cluster by mode.

$QLq$ and $QLl$ do not diverge. Figure 1 shows that the joint distribution of $QLq$ and $QLl$ demand levels across all benchmark items is mostly similar. Items primarily concentrate near the diagonal, with the modal demand level at 1 for both dimensions. $QLl$ tends to exceed $QLq$ slightly more often and reaches higher levels, but the two dimensions are broadly correlated rather than orthogonal. This pattern persists even in ToT, which is explicitly designed to separate arithmetic from semantic tasks. ToT Semantic has 679 items where $QLq \ge3$,

while the Arithmetic subset has only 41 items, making the Semantic subset quantitatively more $QLq$ demanding. This is partly explained by two of the eight Semantic question types (RelationDuration and NumEventsInTimeInterval) which require arithmetic operations (see Appendix § 9.§ 3 for a breakdown of $QLq$ and $QLl$ scores per question type). The full distribution of demand levels for both ToT subsets is depicted in the Appendix (Figure 7).

DeLeAn separates benchmarks by overall demand, not by mode. Because $QLq$ and $QLl$ alone do not distinguish the two purported modes, we ask whether the distinction emerges when all 18 cognitive dimensions are considered. We therefore apply principal component analysis (PCA) to the DeLeAn-labelled items (Figure 2). The first two principal components’ explained variance is PC1: 40.5% and PC2: 15.5%. If the arithmetic/semantics distinction were a major organising principle, we would expect items from both modes to occupy different regions of this projection. Instead, the dominant structure appears to reflect overall cognitive demand²: Temporal Constraint-based Planning (TCP) and ToT Semantic, the two most demanding datasets in this study, each form a distinct cluster. The remaining benchmarks show no mode-consistent grouping. In short, the leading principal components capture differences in overall benchmark demand but reveal no clustering that supports the arithmetic/semantics distinction.

Performance gains are dataset-specific and do not transfer across modes

The cognitive demand analysis above examines what benchmarks require; we now test what models actually learn. If temporal arithmetic and temporal semantics are separable abilities, then fine-tuning on one mode should improve performance on other datasets from the same mode, and not on datasets from another mode.

To test transfer, we treat the arithmetic and semantic portions of each benchmark as separate subsets, yielding four dataset-mode combinations: ToT-Arithmetic, ToT-Semantic, TRAM-Arithmetic, and TRAM-Semantic. We fine-tune six models separately on each combination and evaluate resulting variants on all four test sets. This design allows us to test transfers across datasets within and across temporal reasoning modes. For each training–test combination, Figure 3 reports the change in accuracy relative to the corresponding untrained model, averaged across the six models.

Within- and cross-mode transfer fails. In the accuracy reward setting, in-domain gains are substantial and consistent (diagonal in Figure 3, left). These gains, however, do not transfer across benchmarks, even when the source and target task share a mode. If the arithmetic/semantic distinction captured coherent, transferable abilities, within-mode transfer should be positive and reliable³. Instead, off-diagonal transfer is essentially absent, with one partial exception: training on TRAM arithmetic improves TRAM semantic by 11.1 pp (though the within-dataset gain is far larger, 23.1 pp), and training on TRAM semantic improves TRAM arithmetic by 3.4 pp (with a 15.2 pp gain on TRAM semantic itself). Training on ToT arithmetic, by contrast, degrades TRAM performance ($-3.2$ and $-2.9$ pp). Inspecting model responses shows within-TRAM transfer is largely a formatting effect. The two TRAM modes share the prompt template and a single multiple-choice-letter answer space. Thus, improving answer formatting in one mode recovers otherwise-unparseable responses in the other. The effect does not replicate for ToT, whose two modes differ in both content and answer format (structured JSON vs. event-identifier sets).

Unlearning confirms the pattern. The one-minus-accuracy experiments (Figure 3, right) provide a complementary perspective. Performance degradation mirrors the localised, dataset-specific nature of accuracy gains. This effect is most pronounced for ToT arithmetic ($-26$ pp) and ToT Semantic ($-22.8$ pp). However, degradation spreads more broadly off-diagonal than improvement does.

Notably, gains and losses are not symmetric across reward settings. For TRAM, accuracy training transfers arithmetic improvements to semantics more strongly than the reverse, whereas unlearning reverses this asymmetry. This suggests that the two TRAM subsets share surface-level regularities that are learned and unlearned asymmetrically, rather than drawing on a common underlying ability.

Overall, these results suggest that abilities learned during fine-tuning are highly localised to individual datasets rather than transferable improvements in temporal arithmetic or semantic reasoning. Per-model results (Appendix § 13, Table 22) confirm that this pattern is consistent across all six models and not driven by individual outliers.

Reward	Train	Test	Baseline			$\Delta$ (pp)		Imp./Worse
Acc.	TRAM-Ari.	TRAM-Ari.	25.74	70.09	87.74	23.12	14.65	6/0
		TRAM-Sem.	10.00	61.40	70.60	11.13	1.90	5/0
		ToT-Ari.	15.37	50.22	72.08	0.83	1.73	4/2
		ToT-Sem.	27.86	42.50	55.86	0.86	1.07	4/2
	TRAM-Sem.	TRAM-Ari.	25.74	70.09	87.74	3.35	1.52	3/3
		TRAM-Sem.	10.00	61.40	70.60	15.17	5.00	6/0
		ToT-Ari.	15.37	50.22	72.08	-0.72	-0.43	3/3
		ToT-Sem.	27.86	42.50	55.86	1.12	0.43	3/3
	ToT-Ari.	TRAM-Ari.	25.74	70.09	87.74	-3.22	-2.31	2/4
		TRAM-Sem.	10.00	61.40	70.60	-2.87	0.40	4/2
		ToT-Ari.	15.37	50.22	72.08	7.61	7.79	6/0
		ToT-Sem.	27.86	42.50	55.86	1.40	0.86	3/3
	ToT-Sem.	TRAM-Ari.	25.74	70.09	87.74	-1.01	-0.85	2/4
		TRAM-Sem.	10.00	61.40	70.60	-0.00	0.60	4/2
		ToT-Ari.	15.37	50.22	72.08	1.01	1.19	5/1
		ToT-Sem.	27.86	42.50	55.86	6.81	7.00	6/0
1-Acc.	TRAM-Ari.	TRAM-Ari.	25.74	70.09	87.74	-17.17	-16.67	2/4
		TRAM-Sem.	10.00	61.40	70.60	0.80	0.40	3/3
		ToT-Ari.	15.37	50.22	72.08	-5.77	-4.76	1/5
		ToT-Sem.	27.86	42.50	55.86	-1.62	-0.79	3/3
	TRAM-Sem.	TRAM-Ari.	25.74	70.09	87.74	-1.22	-0.78	2/4
		TRAM-Sem.	10.00	61.40	70.60	-12.93	-11.50	1/5
		ToT-Ari.	15.37	50.22	72.08	-1.12	-0.22	2/4
		ToT-Sem.	27.86	42.50	55.86	-6.36	-2.29	1/5
	ToT-Ari.	TRAM-Ari.	25.74	70.09	87.74	-10.07	-6.06	1/5
		TRAM-Sem.	10.00	61.40	70.60	-6.03	-1.60	0/6
		ToT-Ari.	15.37	50.22	72.08	-25.97	-18.51	0/6
		ToT-Sem.	27.86	42.50	55.86	-12.31	-10.21	1/5
	ToT-Sem.	TRAM-Ari.	25.74	70.09	87.74	-0.95	-1.33	2/4
		TRAM-Sem.	10.00	61.40	70.60	-0.60	-1.30	1/5
		ToT-Ari.	15.37	50.22	72.08	-2.63	-1.73	1/5
		ToT-Sem.	27.86	42.50	55.86	-22.76	-25.00	0/6

An exploratory experiment with Qwen3-32B produces smaller and mixed transfer effects, including a 3.7-point gain from TRAM Arithmetic to ToT Arithmetic, but does not establish a consistent mode-specific pattern (Appendix § 15).

Quantitative and logical reasoning are not the main drivers for performance

The preceding transfer experiments use benchmark-defined labels to distinguish the two temporal-reasoning modes. We complement them with an item-level operationalisation by training on data filtered by relative QLq and QLl demands (§ 3.1.2) and measuring changes in DeLeAn-defined capabilities. This asks a finer-grained question: when trained on data with high quantitative or logical demands, do models improve most on those dimensions, as the arithmetic/semantic distinction predicts, or on other capabilities?

Accuracy for evaluation is insufficient here, because we are interested in changes across the full capability profile rather than end-task performance alone. We therefore compute DeLeAn-based subject characteristic curves as described in § 3.2 and use the AUC across 14 retained dimensions as a summary measure of model ability on each dimension. Figure 4 reports the average change in AUC relative to baseline, averaged across all six models for both experimental settings. Per-model results are in the Appendix § 13.

QLq and QLl’s subordinate role in improving temporal reasoning. The arithmetic and semantic modes contrast temporal reasoning that requires explicit calculation with temporal reasoning that does not. If true, we would expect QLq-dominant training to primarily improve QLq (and hence temporal arithmetic), and the equivalent for QLl-dominant training. This is not what we observe. Training on data where QLl = QLq + 1 yields the largest improvements for both QLq and QLl, rather than selectively boosting QLl. The QLl = QLq + 2 condition produces weaker gains, possibly due to its small dataset size (54 items). QLq-dominant training, where QLq = QLl + {1,2}, also does not preferentially improve QLq. Because QLq demands are generally lower than QLl demands across the benchmarks (Figure 6 in the Appendix), QLq-dominant items may not provide a sufficiently challenging learning signal for that dimension.

A stronger QL signal does not improve temporal accuracy. To test whether low QL demand in the temporal training data (mean demand of 1.57) explains the limited performance gains, we fine-tune models on QL-dominant, non-temporal data (mean demand of 4.13). Although QLl and QLq both increase by 0.15 AUC—the largest capability gains among all dimensions—accuracy on the temporal mix data remains unchanged (mean $\Delta= -0.06$ pp; range $-1.23$ to $+0.55$; Appendix § 14, Table 23,Table 24). Thus, stronger QL capability is insufficient to improve temporal reasoning and QL is not the principal bottleneck in model performance.

Dimension	$\Delta$ AUC
AS	+0.01
AT	-0.02
CEc	+0.14
KNa	-0.04
KNc	+0.02
KNf	+0.08
KNs	+0.07
MCr	+0.03
MCt	+0.02
MCu	-0.02
MS	+0.07
QLl	+0.15
QLq	+0.15
VO	-0.08

	Baseline	QL-dominant
	Acc.	Acc.	$\Delta$
Llama-3.2-3B	24.85	25.40	+0.55
Llama-3.1-8B	33.33	32.87	-0.46
gemma-3-4b	43.42	43.71	+0.29
gemma-3-12b	62.53	62.78	+0.25
Qwen3-4b	53.59	52.36	-1.23
Qwen3-8b	55.78	56.03	+0.25

Knowledge benefits the most. The most revealing finding (Figure 4, top) emerges when we examine which dimensions benefit most across all six training conditions. Gains in Knowledge of Social Sciences (KNs), Customary Everyday Knowledge (KNc), and Verbal Comprehension (CEc) consistently exceed gains in QLl and QLq (often by a wide margin). This pattern holds regardless of whether the training data was selected for its quantitative or logical demands. In other words, even when we deliberately select items that emphasise arithmetic or logical reasoning, models primarily improve on KNc, KNs and CEc. The one-minus-accuracy results (Figure 4, bottom) corroborate this from the reverse direction: the largest capability degradations occur for the $KNs$ dimension, confirming that these are the abilities most exercised by current temporal benchmarks. The per-benchmark demand distributions (Appendix § 9.§ 4, Figure 7) provide further support. Most benchmarks concentrate their $QLq$ and $QLl$ demands at levels 0–2, with few items at the higher levels, which would be necessary to expect reasoning bottlenecks.

Relabelling with Gemini largely reproduces the $KN$-over-$QL$ pattern seen in Figure 4, although $KNs$ rather than $KNc$ benefits most and $CEc$ gains are smaller (details in Appendix § 12).

These results suggest that logical and quantitative reasoning are not the primary drivers of performance on current temporal reasoning benchmarks. Training on examples selected for their logical or quantitative demands does affect those dimensions, but the dominant effect falls on knowledge dimensions. This finding reinforces the conclusion from § 4.1 and § 4.2: existing temporal reasoning benchmarks do not cleanly instantiate the arithmetic/semantics distinction. Instead, they largely measure models’ capacity to retrieve and apply learned temporal regularities.

Conclusions

This study scrutinises whether temporal arithmetic and temporal semantics correspond to distinct reasoning abilities. Capability-aware analyses of the dataset and fine-tuning experiments show that they do not. Cognitive demand profiles fail to separate arithmetic from semantic benchmark items, despite supposed differences in required quantitative and logical reasoning. Furthermore, finetuning gains do not transfer between benchmarks of the same temporal reasoning mode.

Capability changes after finetuning on temporal data show that DeLeAn knowledge dimensions improve more than $QLq$ or $QLl$. Specialised $QL$-heavy training could raise $QL$ capability, but still did not improve performance in either temporal reasoning modes. Our results suggest that current benchmarks test the retrieval and application of time-related regularities, not arithmetic or semantic abilities. The arithmetic/semantics taxonomy may describe task form, but it should not be treated as a salient decomposition of temporal reasoning. Future benchmarks should control for knowledge demands and validate proposed categories through capability-based analyses.

Limitations

Our experiments use models of up to 12 billion parameters. In one instance, we trained a larger model, which showed small transfer (improvement of $\approx3$ percentage points) in the arithmetic mode. Future work could explore transfer dynamics in even larger models; however, the consistency of our findings across three families and two scales suggests that this pattern is robust. Furthermore, our transfer experiments are limited to ToT and TRAM, which are the only benchmarks with explicit or reconstructable arithmetic and semantic partitions. Extending this analysis to future benchmarks that provide more detailed task taxonomies would strengthen the generalisability of our findings.

Acknowledgments

JHO’s research is supported by OpenAI’s grant to the “AI Progress through the Lens of Predictable AI Ecosystems” programme, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. JHO’s research has also been supported by Google’s grant to the “Evaluations and Benchmarks” project, which is based at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. AA is supported by the Economic and Social Research Council [ES/Y001788/1].

Demand level annotations (DeLeAn)

Demand-Level Annotations (DeLeAn) are a set of general-scale, human-interpretable rubrics that map a benchmark item $X_j$ to a multi-dimensional demand profile intended to capture which cognitive and knowledge demands a fully competent solver would need to meet in order to answer correctly (Zhou et al., 2026). Concretely, each item is assigned an annotation vector $\mathbf{x}^{(j)} \in\{0,\ldots,5^{+}\}^{18}$, where each component $x_i^{(j)}$ is an open-scale demand level on dimension $i$ (0 = no meaningful demand; larger values indicate increasing demand; $5^{+}$ indicates demands beyond the highest anchor point of the rubric). In this paper, we use these demand vectors as structured, interpretable item-level features for analysing which cognitive capabilities current temporal reasoning benchmarks actually require.

The DeLeAn rubrics build upon seven broad capabilities proposed in [36], which are refined into 11 core cognitive dimensions. In addition, the framework incorporates five knowledge-related dimensions and two extraneous ones (atypicality and volume) designed to capture task-specific complexity that goes beyond basic cognitive or knowledge demands. While the authors also define an additional extraneous dimension, unguessability, computed algorithmically based on prompt-specific characteristics, it is not considered in this work. Table 3 summarises the dimensions included in the full rubric set.

Dimension (Broad)	Dimension (Specific)	Demand description
AS Attention and Scan	AS Attention and Scan	Focus on or locate specific elements within a given stream of information or environment in the whole process of solving a task.
CE Comprehension and Expression	CEc Verbal Comprehension	Understand text, stories or the semantic content of other representations of ideas in different formats or modalities.
CE Comprehension and Expression	CEe Verbal Expression	Generate and articulate ideas, stories, or semantic content in different formats or modalities.
CL Conceptualisation, Learning and Abstraction	CL Conceptualisation, Learning and Abstraction	Build new concepts, engage in inductive and analogical reasoning, map relationships between domains, and generate abstractions from concrete examples.
MC Metacognition and Critical Thinking	MCr Identifying Relevant Information	Recognise what information helps solve the task or does not, and how this recognition process unfolds as they work toward the solution.
MC Metacognition and Critical Thinking	MCt Critical Thinking Processes	Monitor or regulate multiple thought processes to answer the question effectively, ranging from simple recall to high-level critical thinking.
MC Metacognition and Critical Thinking	MCu Calibrating Knowns and Unknowns	Recognise the boundaries of one’s knowledge and confidently identify what one knows they know, knows they don’t know, or is uncertain about.
MS Mind Modelling and Social Cognition	MS Mind Modelling and Social Cognition	Model the minds of other agents or reasoning about how the beliefs, desires, intentions, and emotions of multiple other agents might interact to determine future behaviours.
QL Quantitative and Logical Reasoning	QLl Logical Reasoning	Match and apply rules, procedures, algorithms or systematic steps to premises to solve problems, derive conclusions and make decisions.
QL Quantitative and Logical Reasoning	QLq Quantitative Reasoning	Work with and reason about quantities, numbers, and numerical relationships.
SN Spatial Reasoning and Navigation	SNs Spatio-physical Reasoning	Understand spatial relationships between objects and predicting physical interactions.
KN Knowledge	KNa Knowledge of Applied Sciences	Knowledge or conceptual understanding in applied sciences (e.g., medicine, law, education, business, agriculture, engineering except IT).
KN Knowledge	KNc Customary Everyday Knowledge	Knowledge in information that most people in a given society typically acquire through daily life experiences, social interactions, and media.
KN Knowledge	KNf Knowledge of Formal Sciences	Knowledge or conceptual understanding in formal sciences (e.g., mathematics, logic, computer science, statistics).
KN Knowledge	KNn Knowledge of Natural Sciences	Knowledge or conceptual understanding in natural sciences (e.g., physics, chemistry, biology, astronomy, earth sciences, ecology).
KN Knowledge	KNs Knowledge of Social Sciences	Knowledge or conceptual understanding in social sciences and humanities (e.g., history, psychology, sociology, literature, art, philosophy).
AT Atypicality	AT Atypicality	How uncommon the task is or how unlikely it is that the instance has appeared in various sources (internet, textbooks, tests).
VO Volume	VO Volume	Proportional to the logarithm of the time a fully competent human needs to read and complete the task in ideal conditions, excluding interruptions.

All data used in this study is derived from Zhou et al., 2026. ToT, and TCP were labelled for this study and are licenced under CC BY 4.0 and MIT respectively.

Data

Benchmark sizes

Our experiments use two distinct data configurations. In the first configuration (see § 4.2), we use existing benchmark labels to separate ToT and TRAM into semantic and arithmetic modes for the controlled transfer analysis, yielding four dataset-mode combinations. Their cardinalities are reported in Table 4. In the second configuration (see § 4.1 and § 4.3), we annotate items from all nine datasets with DeLeAn cognitive demand profiles, retaining TCP Long and Short as separate subsets. The resulting ten subsets are reported in Table 5. Because TRAM is substantially larger than the other benchmarks, we subsample it to ensure that no single benchmark dominates the training conditions. The cost of annotation with GPT-4o was also a factor in this decision.

Dataset	Train	Validation	Test
ToT Arithmetic	1295	93	462
ToT Semantic	1959	141	700
TRAM Arithmetic	25499	2834	2700
TRAM Semantic	20121	2236	500

Dataset	Train	Validation	Test
TRAM	2736	322	739
ToT Semantic	2042	208	544
ToT Arithmetic	1297	141	401
TimeQA	530	58	139
MenatQA	490	49	141
TempReason	455	55	147
Date Arithmetic	347	41	105
TimeDial	256	30	54
TCP Long	207	21	57
TCP Short	170	23	43

Partitioning TRAM into arithmetic and semantic tasks

Unlike ToT, which provides an explicit arithmetic/semantic split, TRAM organises its tasks into categories such as foundational temporal understanding, temporal interpretation and computation, and advanced temporal concepts, rather than assigning a binary mode label. To derive an arithmetic/semantic partition for TRAM, we follow Su et al. [33]’s distinction between math-time tasks and pure-time tasks. TRAM’s subtasks Frequency, Duration, Arithmetic, and Ambiguity are classified as math-time tasks, corresponding to the arithmetic mode. We therefore assign these four subtasks to the arithmetic split.

This leaves six remaining TRAM subtasks that can be classified as pure-time tasks, corresponding to the semantic mode. To keep the two splits balanced, however, we include only four subtasks in the semantic partition. TRAM’s Ordering and Storytelling tasks align closely with the ToT Semantic subtasks FirstLast and Timeline, respectively, so we assign them to the semantic split.

This leaves Typical Time, Causality, Relation, and NLI. Because we include only two additional subtasks, we use QLq and QLl scores as a tiebreaker. We select Relation and NLI, as these subtasks show the largest differences in QLq and QLl. The scores are reported in Tab. 6.

To further assess the quality of this partition, we project all TRAM items’ demand profiles into a two-dimensional PCA space and colour them by assigned mode (Figure 5). The resulting separation is weak, consistent with the findings in § 4.1 that arithmetic and semantic tasks are not strongly distinguishable by demand vectors.

Cognitive demand profiles of ToT Semantic question types

The ToT Semantic subset contains eight question types that vary in their reliance on quantitative reasoning. Tab. 7 reports the average QLq and QLl scores for each type. Two question types, Relation-Duration and Timeline, have a positive QLq−QLl difference, indicating that they place slightly higher quantitative than logical demands despite being labelled as semantic. Conversely, FirstLast, *Be-

caption: Average QLq and QLl scores and their difference for TRAM tasks.

:::

foreAfter, and EventAtTimeOfOtherEvent show the largest negative differences, consistent with their reliance on relational reasoning. This within-subset heterogeneity supports our finding (§ 4.1) that the semantic label subsumes tasks with meaningfully different cognitive profiles.

Per-benchmark cognitive demand profiles

Figure 6 shows the average demand level per cognitive dimension for each of the ten dataset subsets, sorted by overall difficulty (computed as the mean across all 18 dimensions). The most demanding subsets (TCP Short and TCP Long) show elevated scores across nearly all dimensions, while the least demanding (Date Arithmetic) has near-zero demand on most dimensions except $\mathrm{QLq}$ and $\mathrm{KNc}$. Notably, $\mathrm{QLq}$ and $\mathrm{QLl}$ are not among the highest-demand dimensions for any subset. Instead, $\mathrm{KNc}$, and $\mathrm{CEc}$, among others, dominate. This pattern is consistent with our main finding that temporal benchmarks are knowledge-heavy rather than reasoning-heavy.

Figure 7 provides a complementary view, showing the full distribution of demand levels (0 through 5+) for each cognitive dimension within each of the ten dataset subsets. These distributions confirm that most subsets concentrate their $\mathrm{QLq}$ and $\mathrm{QLl}$ demands at levels 0–2, with few items reaching the higher levels that would be expected if genuine arithmetic or logical reasoning were the primary bottleneck.

Metrics

To reduce spurious mismatches caused by formatting variability, we evaluate model outputs with benchmark-specific post-processing before computing final scores. Across our evaluation modules, normalisation generally takes one of three forms: extracting the answer from a larger generated response, canonicalising the representation of the answer, and then comparing either normalised strings, normalised structured objects, or normalised sets. The exact procedure differs by benchmark.

TRAM accuracy

For TRAM, we normalise predictions by extracting the final option letter from responses matching the template “The final answer is (X)”, with $X \in\{A,B,C,D\}$. We then compute exact-match accuracy against the gold option label. This removes variability in preceding reasoning text but still requires a standardised final-answer template.

TCP accuracy

For TCP, the evaluator expects the model’s final answer to appear in boxed LaTeX notation, using the

caption: tbl:7: Average QLq and QLl scores and their difference for ToT Semantic question types

:::

pattern \boxed{...}. It extracts the boxed span with the regular expression `\boxed{(<sup>4</sup>*)}` and strips surrounding white space from the extracted content. Comparison is then performed by exact string match.

In the TCP Short subset only, the evaluator removes the literal string “GMT” from both the prediction and the reference before comparison; no analogous normalisation is applied for TCP Long. The system prompt includes the exact expected answer format.

Test of Time

For ToT, we extract the first valid JSON object from the model output, after escaping unescaped control characters inside JSON strings. We discard the explanation field and compare only the answer-bearing content. In the Arithmetic subset, we additionally normalise schema mismatches such as age versus answer, lowercase items in ordered_list and unordered_list answer types, cast prediction values to the reference types when possible, and sort unordered_list values to make comparison order-invariant. In the Semantic subset, we normalise both prediction and reference by converting them to strings before exact comparison.

TimeBench derived benchmarks

For TimeBench tasks, we first extract the final answer from the text following the marker “Thus, the

ends_mid=1

correct answer is:”. We retain only the first subsequent line, strip trailing sentence punctuation, and canonicalise any response containing unanswerable to the label unanswerable. For TempReason, TimeQA, and MenatQA, the extracted answer is scored with standard SQuAD exact match and F1. For Date Arithmetic, predictions and references are parsed into date objects and normalised by setting the day to 1, so comparison is effectively at the month-year level. For TimeDial, prediction and reference are mapped to sets of selected options in A,B,C,D and scored using set exact match and set-based F1.

Non-temporal multiple-choice accuracy

For single-answer multiple-choice examples from the Civil Service Examination, LSAT, MMLU-Pro, and the relevant Math tasks, we inspect the final 300 characters of the completion. We preferentially extract the final option letter in $\{A,\ldots,J\}$ appearing in an answer-bearing expression such as “the answer is X”, “answer: X”, “option X”, or “choice X”, or as a standalone line. If no such expression is found, we fall back to the final isolated letter in this range. The extracted letter is uppercased and compared by exact match with the gold option label.

TruthQuest accuracy

For TruthQuest, the reference annotation is converted to the set of option labels corresponding to knights. From the final 300 characters of the completion, we collect option letters in answer-bearing expressions; if none are present, we collect all isolated letters in $\{A,\ldots,J\}$. Both prediction and reference are treated as sets, so ordering and duplicate labels are ignored. A reward of 1 is assigned only when the predicted and reference sets are identical.

Math and OmniMath accuracy

Math examples use several answer representations. Multiple-choice items follow the option-letter procedure above. For integer-valued Math and OmniMath items, we extract the final integer appearing in the last 200 characters of the completion, cast it to an integer, and compare it exactly with the reference. For sequence-valued Math items, we extract the last valid JSON array from the complete response and compare it directly with the reference list; this comparison is order-sensitive.

ChemLLMBench and Reasoning accuracy

For fraction-valued OmniMath items, we parse a LaTeX fraction written using \frac, \dfrac, or \tfrac. If no valid LaTeX fraction is found, we use the final plain-text fraction of the form $p/q$. The numerator and denominator are converted to an exact rational number and reduced to the canonical string $p/q$ before exact comparison with the similarly normalised reference.

For the SMILES-valued ChemLLMBench examples, we scan the completion from right to left for a whitespace-delimited SMILES-like token that can be parsed by RDKit. The predicted molecule is converted to canonical isomeric SMILES and compared by exact string match with the canonicalised reference.

For free-text Reasoning examples, we inspect the final 300 characters of the completion, lowercase the text, remove punctuation, and collapse consecutive whitespace. We apply the same normalisation to the reference and award a match when the normalised reference occurs as a substring of the normalised completion.

Training settings

We used Transformers 4.57.3 [43] to load the models, TRL 0.27.2 [38] for GRPO, and PEFT 0.18.0 [25] for LoRA fine-tuning [18]. LoRA used rank $r = 16$ and scaling parameter $\alpha= 32$ and was applied to the q_proj, k_proj, v_proj, and o_proj modules. The maximum GRPO completion length was 3,000 tokens.

All GRPO runs used a learning rate of $5 \times10^{-5}$ with linear decay, training seed 42, no KL penalty ($\beta= 0$), and no warmup. Dataset splits used seed 6; the temporal mixture’s initial stratified train/test split used seed 42.

Training used PyTorch 2.9.0 [3], Accelerate 1.12.0 [12], and DeepSpeed 0.18.3 [30]. Regular single-node training used DeepSpeed ZeRO-2, whereas Qwen3-32B training used ZeRO-3 [28]. Evaluation was performed using vLLM 0.11.2 [23].

DeLeAn-filtered and non-temporal training runs allocated one GPU for training and one GPU for vLLM generation. Single-dataset training runs allocated two GPUs for training and one GPU for generation. Qwen3-32B training used three compute nodes equipped with four NVIDIA GH200 GPUs each: eight GPUs across two nodes were used for training, while four GPUs on the third node were used for generation. Evaluation used one GPU per run. Full training runs generally took several hours, with longer runs approaching or exceeding 12 hours. Individual evaluations generally took approximately 1–8 minutes, although some larger evaluations took longer.

Label reliability

We test label agreement using the within-group interrater reliability, which is the main metric used for the DeLeAn paper, between the labels from GPT-4o and Gemini 3.1 Flash Lite. Even though this is the metric reported in the original paper (Zhou et al., 2026), we additionally report exact score matches, within one matches, and Krippendorff’s alpha.

For each item, within-group interrater agreement is

The reference variance is $s_{E}^{2}=(A^{2}-1)/12=35/12$, assuming a discrete uniform distribution over $A=6$ equally spaced category codes $\{0,1,2,3,4,5\}$. The open-ended top category $5+$ is encoded as 5. We use all six rubric categories for this reference distribution, even when fewer levels are observed.

We use the population variance across the two annotators, clip each item’s $r_{WG}$ to $[0,1]$, and report the mean over items with both labels present in each dimension.

The capability analyses retain dimensions with at least three observed demand levels and at least 30 items in the dominant slices (§ 3.2). The focus of our work lies in inspecting QLl and QLq. However, we are also interested in confirming whether the improvements observed for CEc, KNc, and KNs in Figure 4 are not due to label biases but can be reproduced by Gemini 3.1 Flash Lite.

Table 8 shows that overall agreement according to $r_{WG}$ is high with values close to the maximum of 1. KNc is an outlier, showing a lower agreement of only 0.66. CEc, while scoring high for $r_{WG}$, has a lower value for Krippendorff’s alpha. The issue is that both models assign items a demand level of 1–2. Absolute closeness is therefore high, explaining the high value for $r_{WG}$, but on the other hand, it increases the by-chance agreement, which Krippendorff’s alpha penalises.

Dimension	Exact (%)	Within one (%)	Ordinal $\alpha$	$r_{\mathrm{WG}}$
KNc	27	45	-0.23	0.66
CEe	62	98	-0.01	0.96
SNs	99	100	0.06	1.00
MCu	42	85	0.34	0.90
CL	70	93	0.42	0.95
CEc	60	99	0.44	0.96
KNa	83	92	0.44	0.94
KNf	62	78	0.47	0.87
KNn	96	98	0.53	0.99
MCt	65	99	0.56	0.97
KNs	79	89	0.57	0.93
AT	37	81	0.59	0.88
VO	60	97	0.60	0.96
QLl	68	95	0.68	0.96
MS	92	95	0.69	0.97
MCr	56	94	0.71	0.94
QLq	63	97	0.74	0.96
AS	46	94	0.76	0.94

Table 9 compares the mean demand levels between Gemini and GPT. We can see that the low agreement in KNc can be explained by GPT-4o assigning much higher demands (1.59 higher in average).

Dimension	Gemini	GPT-4o	GPT-4o − Gemini
KNc	0.36	1.96	1.59
KNf	0.61	1.27	0.66
MCu	0.73	1.36	0.63
AT	1.67	2.19	0.53
AS	1.16	1.67	0.52
KNs	0.26	0.57	0.30
CL	0.23	0.52	0.29
KNa	0.19	0.47	0.28
VO	1.46	1.70	0.24
MCt	1.58	1.75	0.17
QLl	1.21	1.37	0.15
MS	0.16	0.25	0.09
QLq	1.17	1.25	0.08
KNn	0.04	0.10	0.06
MCr	1.31	1.33	0.03
CEe	1.11	1.11	0.01
SNs	0.01	0.01	0.01
CEc	1.77	1.76	-0.00

Gemini treats KNc as the everyday knowledge a solver needs to bring to the task. When the prompt supplies the information required to obtain the answer, Gemini treats it as requiring little or no prior everyday knowledge.

GPT-4o appears to score the topical complexity of the prompt (e.g., German political parties, medieval history, museum associations), conflating the necessity of specialised content with the requirement of everyday knowledge—which the rubric explicitly excludes (specialised academic knowledge is not KNc).

The main findings still reproduce with Gemini 3.1 Flash Lite, even with its lower KNc scores. Instead of KNc, KNs and in one instance Knowledge of Formal Sciences (KNf) show the largest increase in capability after finetuning across all training settings. Most importantly, QLl and QLq barely improve (Figure 8). The same pattern also holds in the one-minus-accuracy setting, except for Atypicality (AT), which shows the strongest decrease in the QLl = QLq + 2 setting. CEc does not survive this ablation. However, although, Table 9 shows strong agreement in mean demand for CEc, annotation agreement shows lower values (Table 8). The reason is that GPT-4o assigns demand 3 levels more often than Gemini 3.1 Flash Lite. As a result, the potential gains in capability are capped more often at level 2.

Per-model results

The transfer results reported in § 4.2 are averaged across six models. Table 22 disaggregates these results, reporting the baseline accuracy range (min, median, max across models), the mean and median performance delta, and the number of models that improved versus worsened for each train–test combination under both reward settings.

Several patterns are worth noting. First, in-domain gains under the accuracy reward are consistent across all six models: every model improves on its own training set (6/0 in all diagonal cells). Second, the absence of cross-benchmark transfer

is not driven by individual outliers. For most off-diagonal cells, the number of models that improve versus worsen is close to 3/3, indicating that the lack of transfer is a stable property of the task structure rather than an artefact of averaging. Third, under the one-minus-accuracy reward, unlearning is similarly concentrated: for example, training on ToT Arithmetic degrades all six models on ToT Arithmetic (0/6) and five of six on TRAM Arithmetic (1/5), confirming that the unlearning signal is reliably absorbed. Finally, the wide baseline range (e.g., 25.74–87.74 for TRAM Arithmetic) reflects genuine variation in model strength, yet the direction of transfer effects is consistent across this range, further supporting the robustness of our conclusions.

Non-temporal QL-heavy training

To test whether a stronger quantitative and logical reasoning signal transfers to temporal reasoning, we fine-tune all six models on the non-temporal QL-dominant data described in § 3.1.3. Table 23 reports the mean capability AUC change per dimension, averaged across models, and Table 24 the per-model accuracy on the temporal mixture before and after this training. The training achieves its intended effect on the targeted capabilities: QLl and QLq show the largest gains of all 14 dimensions (+0.15 AUC each). Accuracy on the temporal mixture, however, is essentially unchanged for every model (mean $\Delta= -0.06$ pp; per-model range $-1.23$ to $+0.55$ pp), indicating that increased QL capability alone does not translate into better performance on the evaluated temporal benchmarks (see § 4.3).

This appendix also collects the per-model results for the item-level experiments of § 4.3, which Figure 4 reports only as averages. Table 10–Table 21 report, for each of the six models, the baseline capability AUC and its change under each of the six training conditions, separately for the accuracy and one-minus-accuracy reward settings.

	Baseline	$QL_l = 0$	$QL_q = 0$	$QL_l > QL_q$		$QL_q > QL_l$
				+1	+2	+1	+2
AS	2.10	0.05	0.25	0.21	0.28	0.39	0.11
AT	2.69	0.10	0.16	0.17	0.20	0.05	0.01
CEc	2.93	-0.30	0.41	-0.08	0.16	-0.34	-0.09
KNa	3.08	0.05	0.05	-0.07	-0.04	0.08	-0.12
KNc	2.84	0.04	0.36	0.04	0.17	0.22	0.05
KNf	2.59	0.01	0.05	0.22	0.35	0.11	0.34
KNs	2.83	0.20	0.09	0.07	-0.02	0.04	0.03
MCr	1.93	0.01	0.33	0.16	0.17	0.51	0.04
MCt	2.61	-0.21	0.37	-0.03	0.04	-0.05	-0.07
MCu	2.30	0.25	0.37	-0.29	0.29	0.24	-0.02
MS	3.20	-0.13	-0.11	-0.07	-0.05	0.02	0.02
QLl	2.07	0.38	0.62	0.72	0.80	0.61	0.60
QLq	2.91	-0.26	0.07	-0.25	-0.21	-0.31	-0.12
VO	1.83	0.10	0.21	0.21	0.13	0.64	0.34

	Baseline	QLl = 0	QLq = 0	QLl > QLq		QLq > QLl
				+1	+2	+1	+2
AS	3.03	-1.16	-0.26	-0.20	-1.05	-0.27	-0.34
AT	3.54	-1.83	-0.55	-0.26	-1.96	-0.31	-0.39
CEc	2.83	-1.70	0.05	0.14	-0.46	-0.48	-0.32
KNa	3.05	-2.41	-0.17	-0.26	-0.25	-0.17	-0.19
KNc	3.58	-2.78	-0.42	-0.11	-0.91	-0.04	-0.01
KNf	3.47	-1.36	-0.04	-0.07	-1.19	-0.53	-0.65
KNs	2.97	-2.80	-0.28	-0.17	-0.71	0.11	0.06
MCr	2.77	-1.59	-0.30	-0.10	-0.88	-0.13	-0.21
MCt	2.94	-1.91	-0.13	-0.27	-0.61	-0.56	-0.59
MCu	2.68	-1.50	-0.31	-0.18	-0.85	0.08	-0.29
MS	2.80	-2.34	-0.18	-0.26	-0.12	0.10	-0.07
QLl	3.17	-1.55	0.00	0.04	-1.13	-0.44	-0.25
QLq	2.30	-1.15	0.09	0.07	-0.10	-0.44	-0.38
VO	2.83	-1.39	0.07	0.07	-0.50	-0.78	-0.80

Larger models

We also explored scaling the experiments to larger models, but the computational cost of GRPO prevented us from running the full experimental suite. For example, the length of some ToT prompts, which can span several thousand tokens, would

	Baseline	$QL_l = 0$	$QL_q = 0$	$QL_l > QL_q$		$QL_q > QL_l$
				+1	+2	+1	+2
AS	2.10	-0.16	-0.21	0.30	0.01	-0.47	-0.01
AT	2.69	-0.47	-0.44	0.03	0.00	-0.45	-0.02
CEc	2.93	-1.28	-0.27	-0.31	0.09	-0.64	-0.22
KNa	3.08	-1.05	-0.14	0.17	0.01	-0.24	-0.22
KNc	2.84	-1.08	-0.13	0.14	0.07	-0.19	0.08
KNf	2.59	-0.46	0.11	-0.04	0.54	-0.82	-0.06
KNs	2.83	-1.51	-0.25	0.26	0.27	0.00	0.10
MCr	1.93	-0.45	0.16	0.47	0.16	0.11	0.19
MCt	2.61	-1.02	-0.07	0.09	0.34	-0.73	-0.10
MCu	2.30	-0.83	-0.03	0.25	0.36	-0.20	-0.24
MS	3.20	-0.66	-0.15	-0.21	-0.11	-0.35	-0.10
QLl	2.07	-0.26	0.34	0.46	0.83	-0.40	0.30
QLq	2.91	-1.13	-0.22	-0.45	0.19	-1.42	-0.47
VO	1.83	-0.38	0.22	0.49	0.05	-0.36	0.15

	Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=$ 0	QLq $=$ 0	QLl $>$ QLq		QLq $>$ QLl
				$+1$	$+2$	$+1$	$+2$
AS	3.19	0.17	-0.18	0.05	0.03	-0.08	0.17
AT	3.74	0.23	0.29	0.35	0.12	0.03	0.07
CEc	3.51	0.17	0.52	0.71	0.46	-0.29	0.04
KNa	3.25	0.10	0.10	-0.14	-0.00	0.15	-0.02
KNc	3.57	0.13	0.09	0.15	0.11	0.09	0.04
KNf	3.45	0.03	0.52	0.57	0.16	0.07	-0.05
KNs	2.99	0.51	0.14	-0.03	0.16	0.22	0.55
MCr	2.77	0.09	-0.13	-0.20	-0.02	0.04	0.20
MCt	3.23	0.03	0.65	0.80	0.18	0.04	-0.06
MCu	2.88	0.22	0.19	0.30	0.05	-0.03	0.15
MS	3.04	0.24	0.38	0.24	0.10	0.10	0.23
QLl	3.50	0.00	0.30	0.50	0.16	-0.10	-0.13
QLq	3.64	0.05	0.46	1.09	0.83	-0.35	0.15
VO	3.15	0.08	-0.26	-0.03	0.03	0.03	-0.06

	Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=$ 0	QLq $=$ 0	QLl $>$ QLq		QLq $>$ QLl
				$+1$	$+2$	$+1$	$+2$
AS	3.19	0.09	-0.02	-0.24	-0.21	-0.39	-0.16
AT	3.74	-0.06	-0.13	-0.26	-0.09	-0.45	0.21
CEc	3.51	-1.24	-1.33	-1.27	-0.17	-1.62	-0.91
KNa	3.25	-1.46	-1.78	-1.42	-0.05	-1.71	-0.71
KNc	3.57	-1.39	-1.40	-1.29	-0.02	-1.59	-0.63
KNf	3.45	-0.39	-0.50	-0.40	-0.35	-1.02	-0.24
KNs	2.99	-1.55	-1.73	-1.55	-0.07	-1.79	-1.06
MCr	2.77	-0.16	-0.72	-0.58	-0.08	-0.64	-0.28
MCt	3.23	-1.02	-1.20	-1.24	-0.03	-1.37	-0.75
MCu	2.88	-0.71	-0.95	-0.77	-0.03	-1.05	-0.27
MS	3.04	-0.96	-1.74	-0.94	0.09	-1.00	0.38
QLl	3.50	-0.59	-0.64	-0.63	-0.25	-0.48	-0.67
QLq	3.64	-0.98	-1.08	-1.16	0.14	-1.52	-1.10
VO	3.15	-0.29	-0.54	-0.82	0.09	-1.20	-0.38

	Baseline	QLl = 0	QLq = 0	QLl > QLq		QLq > QLl
				+1	+2	+1	+2
AS	1.79	0.09	0.35	0.19	0.32	0.28	0.12
AT	2.22	0.40	0.47	0.27	0.46	0.53	0.40
CEc	1.24	0.82	0.61	0.72	0.71	1.01	0.62
KNa	1.23	1.47	1.42	1.11	1.21	1.29	1.31
KNc	1.55	1.25	1.11	0.90	0.84	1.25	1.09
KNf	1.29	0.25	0.53	0.25	0.32	0.62	0.31
KNs	1.44	1.30	1.22	1.14	1.04	1.10	1.23
MCr	1.41	0.27	0.54	0.35	0.52	0.66	0.55
MCt	1.35	0.27	0.44	0.33	0.31	0.77	0.29
MCu	1.37	0.48	0.75	0.63	0.47	0.52	0.71
MS	1.02	1.47	1.98	1.83	1.17	1.72	1.71
QLl	1.50	-0.12	0.26	-0.11	0.04	0.21	-0.16
QLq	0.85	0.28	0.23	0.05	0.22	0.69	0.25
VO	1.08	0.15	0.39	0.24	0.20	0.33	0.17

	Baseline	QLl = 0	QLq = 0	QLl > QLq		QLq > QLl
				+1	+2	+1	+2
AS	1.79	-1.01	-0.86	-1.30	-1.78	-1.19	-0.99
AT	2.22	-1.37	-1.36	-1.63	-1.98	-1.54	-1.31
CEc	1.24	-1.06	-0.87	-0.84	-1.12	-0.87	-0.53
KNa	1.23	-1.00	-0.84	-0.43	-1.10	-0.88	-0.76
KNc	1.55	-1.31	-1.22	-0.95	-1.35	-0.94	-0.57
KNf	1.29	-0.68	-0.72	-1.02	-1.25	-1.05	-0.81
KNs	1.44	-1.33	-1.16	-0.64	–	-0.63	-0.55
MCr	1.41	-0.91	-0.97	-1.15	–	-1.12	-0.65
MCt	1.35	-1.13	-1.02	-1.10	-1.28	-1.12	-0.85
MCu	1.37	-0.85	-0.87	-1.15	–	-1.01	-0.91
MS	1.02	–	-0.89	-0.77	–	-0.65	-0.28
QLl	1.50	-0.89	-0.89	-1.27	-1.45	-1.21	-1.04
QLq	0.85	-0.63	-0.49	-0.54	-0.77	-0.61	-0.44
VO	1.08	-0.83	-0.73	-0.76	-1.04	-0.69	-0.38

	Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl$=$ 0	QLq$=$ 0	QLl$>$QLq		QLq$>$QLl
				+1	+2	+1	+2
AS	2.42	0.08	0.18	0.35	-0.01	0.37	0.21
AT	2.95	0.15	0.36	0.75	0.37	0.67	0.51
CEc	1.82	0.30	1.09	1.13	-0.16	0.68	1.09
KNa	1.62	0.78	0.96	0.80	-0.20	1.34	0.89
KNc	2.23	0.80	0.90	1.11	0.18	1.29	0.99
KNf	2.07	-0.07	0.30	0.76	-0.22	0.32	0.21
KNs	2.15	0.70	0.85	1.21	0.29	0.84	1.18
MCr	1.84	0.25	0.34	0.51	0.39	0.83	0.46
MCt	1.60	0.26	0.84	0.76	-0.00	0.62	0.69
MCu	1.89	0.33	0.86	1.15	0.32	0.69	0.75
MS	2.31	0.79	0.65	0.54	0.21	0.64	0.54
QLl	1.74	0.27	0.91	1.23	0.21	0.75	0.71
QLq	1.06	0.02	0.70	0.89	0.08	0.84	0.92
VO	1.23	0.05	0.35	0.26	0.15	0.77	0.46

	Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl$=$ 0	QLq$=$ 0	QLl$>$QLq		QLq$>$QLl
				+1	+2	+1	+2
AS	2.42	-0.87	-0.77	-2.27	-2.12	-1.67	-1.70
AT	2.95	-1.38	-1.41	-2.76	-2.35	-1.88	-1.84
CEc	1.82	-1.19	-1.25	-1.56	-1.48	-1.26	-1.13
KNa	1.62	-1.34	-1.42	-1.57	-1.49	-1.32	-0.97
KNc	2.23	-1.74	-1.74	-2.02	-1.74	-1.09	-1.00
KNf	2.07	-0.58	-0.68	-1.89	-1.71	-1.63	-1.57
KNs	2.15	-2.11	-2.07	-1.94	-1.83	-1.16	-1.16
MCr	1.84	-0.62	-0.71	-1.73	-1.67	-1.47	-1.46
MCt	1.60	-1.04	-1.10	-1.44	-1.28	-1.16	-1.00
MCu	1.89	-0.88	-0.98	-1.83	-1.73	-1.60	-1.55
MS	2.31	-2.18	–	–	-2.18	-1.63	-1.90
QLl	1.74	-0.42	-0.56	-1.62	-1.46	-1.46	-1.29
QLq	1.06	-0.79	-0.73	-0.91	-0.79	-0.62	-0.47
VO	1.23	-0.68	-0.69	-1.12	-1.00	-0.81	-0.65

	Baseline	QLl= 0	QLq= 0	QLl>QLq		QLq>QLl
				+1	+2	+1	+2
AS	2.80	-0.05	0.08	0.20	0.20	0.07	-0.02
AT	3.38	0.02	0.16	0.22	0.25	0.16	-0.14
CEc	2.64	0.26	0.13	0.53	0.20	0.21	0.24
KNa	3.17	-0.27	-0.22	-0.24	-0.24	-0.06	-0.02
KNc	3.28	0.08	-0.04	0.33	0.34	0.21	0.04
KNf	3.06	-0.11	-0.05	0.33	0.14	0.17	0.03
KNs	3.12	-0.16	-0.06	-0.07	-0.12	0.27	-0.20
MCr	2.54	-0.05	0.03	0.10	0.07	0.13	-0.09
MCt	2.72	-0.10	-0.02	0.16	0.16	0.00	0.04
MCu	2.70	-0.04	0.17	0.02	-0.06	0.22	-0.16
MS	3.09	0.06	-0.19	-0.20	-0.14	0.06	0.00
QLl	2.92	-0.20	-0.17	0.07	0.04	-0.13	-0.16
QLq	2.11	-0.01	-0.05	0.08	0.15	0.29	-0.02
VO	3.02	-0.42	-0.26	-0.17	-0.22	-0.13	-0.19

	Baseline	QLl= 0	QLq= 0	QLl>QLq		QLq>QLl
				+1	+2	+1	+2
AS	2.80	-1.29	-1.29	-0.16	-0.42	-0.22	-0.24
AT	3.38	-1.67	-1.71	-0.29	-1.17	-0.39	-0.30
CEc	2.64	-1.66	-0.68	0.00	-0.04	0.17	-0.01
KNa	3.17	-2.61	-1.52	-0.40	-0.49	-0.14	-0.19
KNc	3.28	-2.40	-1.62	-0.23	-0.55	0.23	-0.16
KNf	3.06	-1.48	-1.12	-0.01	-0.44	0.13	-0.21
KNs	3.12	-2.94	-2.02	-0.22	-0.72	-0.13	-0.06
MCr	2.54	-1.37	-1.32	-0.34	-0.27	-0.03	-0.12
MCt	2.72	-1.82	-0.73	-0.19	-0.23	-0.01	-0.20
MCu	2.70	-1.47	-1.40	-0.44	-0.39	-0.34	-0.25
MS	3.09	–	-2.09	-0.44	-0.34	-0.23	-0.05
QLl	2.92	-1.73	-1.29	-0.16	-0.64	-0.12	-0.34
QLq	2.11	-0.95	-0.20	-0.04	0.05	-0.05	-0.10
VO	3.02	-1.50	-1.17	-0.29	-0.49	-0.34	-0.13

	Baseline	QLl = 0	QLq = 0	QLl > QLq		QLq > QLl
				+1	+2	+1	+2
AS	3.03	-0.08	-0.01	0.04	0.03	0.03	-0.03
AT	3.54	0.06	0.11	0.25	0.13	0.16	0.03
CEc	2.83	0.21	0.03	0.34	0.19	0.23	0.43
KNa	3.05	-0.31	-0.19	-0.32	0.05	-0.04	-0.20
KNc	3.58	-0.03	-0.02	0.13	0.22	0.12	-0.02
KNf	3.47	0.31	0.24	0.26	0.37	0.14	0.01
KNs	2.97	-0.14	0.08	0.16	0.17	0.17	0.09
MCr	2.77	-0.09	-0.15	0.06	-0.03	-0.06	0.09
MCt	2.94	0.12	-0.18	0.01	0.06	0.05	0.04
MCu	2.68	0.15	0.41	0.21	0.19	0.12	0.17
MS	2.80	-0.25	-0.09	0.02	0.51	0.13	0.07
QLl	3.17	-0.16	0.10	0.11	0.13	-0.12	-0.02
QLq	2.30	-0.12	-0.08	0.13	0.03	0.26	0.03
VO	2.83	-0.32	-0.33	-0.01	-0.00	0.10	0.01

have been much higher than for TRAM. Nevertheless, we trained Qwen3-32B separately on the TRAM Arithmetic and Semantic subsets using the standard accuracy-reward configuration and evaluated both variants on all TRAM and ToT subsets. Figure 9 reports the resulting changes in accuracy. Training on TRAM Arithmetic improves ToT Arithmetic by 3.7 percentage points, nearly matching the 3.9-point in-domain gain, while ToT Semantic improves by 1.9 points. Training on TRAM Semantic improves ToT Semantic by 1.9 points, but also increases TRAM Arithmetic by 2.7 points. Overall, however, the changes range from −0.2 to +3.9 points, substantially smaller than the in-domain gains of +6.8 to +23.1 points observed across the models in Figure 3. These results therefore provide tentative evidence of arithmetic transfer, but not a consistent mode-specific transfer pattern. Because they come from a single larger model, it remains unclear whether the observed pattern reflects model scale or ordinary performance variation.