betaFeedback

WorksComputer Science

Atomic MathReasoner: Instance-Level Composition of Atomic Reasoning Skills for Mathematical LLMs

Abstract

Most existing approaches enhance mathematical reasoning in large language models (LLMs) as a monolithic capability, despite the fact that different mathematical problems require different reasoning skills. Recent work has explored a more fine-grained perspective by decomposing mathematical reasoning into atomic skills and developing specialized modules for individual capabilities. However, how to effectively utilize these atomic abilities remains underexplored. To address this challenge, we propose Atomic MathReasoner (AMR), an instance-level framework for composing atomic reasoning skills. AMR first trains a library of capability-specific LoRA experts and performs gradient-free optimization over held-out validation subsets to discover a diverse set of low-loss composition prototypes. It evaluates the preference of each prototype using the token-level negative log-likelihood of reference solutions, filters ambiguous assignments through a confidence margin, and distills the resulting pseudo-labels into a lightweight router. During inference, the router selects an appropriate composition prototype for each problem, enabling problem-specific utilization of atomic skills without costly online optimization. Experiments on the AtomicMath benchmark across general and mathematics-specialized LLMs demonstrate consistent improvements. On Qwen2.5-Math-Instruct-7B, AMR increases average field accuracy from 64.4% to 68.1% at Level 1 and from 59.3% to 63.7% at Level 2, while improving average logical reasoning performance from 39.4% to 46.8%. These results demonstrate that effectively composing specialized atomic skills provides a promising approach to strengthening mathematical reasoning in LLMs.

Introduction

Mathematical reasoning is a fundamental capability of artificial intelligence [3]. Large language models (LLMs) have achieved substantial progress in mathematical problem solving. Recent advances have been driven by several paradigms, including instruction tuning [18, 21, 35, 36], tool-integrated reasoning [4, 8, 9, 5], and reinforcement learning over long chains of thought [17, 26, 20, 10, 32]. Despite their effectiveness, most of these paradigms treat mathematical reasoning as a monolithic capability, optimizing either the entire model or the complete solution trajectory.

This limitation motivates a more fine-grained view of mathematical reasoning. Human problem solving often decomposes complex tasks into simpler sub-skills and coordinates them according to problem-specific demands [2, 37]. Inspired by this perspective, Kuang et al. [16] decomposed mathematical reasoning in LLMs into fine-grained atomic skills, including domain-specific and logical reasoning capabilities.

However, decomposing mathematical reasoning into atomic skills does not by itself determine how these skills should be combined for a specific problem. A central challenge therefore remains at inference time: how can a model select and compose the appropriate atomic skills for an unseen input? Although Kuang et al. [16] showed that activating relevant atomic skills can improve performance, their work primarily focused on capability decomposition and did not provide an efficient mechanism for skill composition.

To bridge this gap, we introduce Atomic MathReasoner (AMR), a two-stage framework for instance-adaptive composition of atomic mathematical skills. In the first stage, following the atomic-skill taxonomy of Kuang et al. [16], we train a dedicated LoRA expert for each skill. We then conduct multiple gradient-free searches on a held-out validation set to optimize the linear composition coefficients under the causal language modeling loss, yielding a diverse set of compositional prototypes. In the second stage, we formulate problem-specific prototype selection as a classification task, avoiding costly per-instance search during inference. For each routing example, we assign the prototype with the lowest token-level negative log-likelihood on the reference solution as a pseudo-label and apply margin-based filtering to retain high-confidence examples. We subsequently train a lightweight router based on TF-IDF features and multinomial logistic regression. At inference time, the router maps each unseen problem to a compositional prototype, whose coefficients are used to combine the atomic LoRA experts for answer generation.

Our extensive experiments on the AtomicMath benchmark (Kuang et al., [16]) demonstrate the robust effectiveness of AMR across various base models. Specifically, AMR improves the average Level 1 and Level 2 field accuracies of InternLM2-Math-Plus-7B to 42.1% (+4.3 percentage points) and 38.7% (+2.2 percentage points), respectively, while increasing the average logic score of Qwen2.5-Math-Instruct-7B by 7.4 percentage points to 46.8%. Notably, our framework enables smaller models to outperform larger math-specialized counterparts in specific categories. For instance, InternLM2-Math-Plus-7B with AMR achieves 48.4% in Analysis (Level 1) and 38.8% in Topology (Level 2), surpassing Abel-13B (40.4%) and Xwin-Math-13B (38.3%), respectively. AMR also remains competitive with commercial models on several challenging tasks. In particular, Qwen2.5-Math-Instruct-7B with AMR achieves 73.5% in Analysis (Level 2) and 53.8% in Geometry (Level 2), outperforming GPT-4o (53.2% and 36.3%) and Claude 3.7 Sonnet (68.1% and 39.0%) on these two tasks, respectively. Additionally, a comprehensive ablation study further validates the effectiveness of the proposed framework.

  • We propose AMR, a two-stage framework that composes independently trained atomic skill experts at the instance level. By constructing problem-conditioned skill compositions, AMR enables different mathematical problems to leverage complementary reasoning capabilities.

  • We propose a gradient-free composition search method and lightweight offline router. By distilling prototype labels into the routing classifier, AMR enables efficient instance-level capability matching during inference and reduces latency.

  • Our framework, when integrated into diverse baseline models, demonstrates performance gains across both low and high difficulty levels on the AtomicMath benchmark. It enables 7B models to exhibit strong competitiveness against 13B specialized counterparts and even GPT-4o in specific domains.

Related Work

Mathematical Reasoning in LLM

Mathematical reasoning for Large language models (LLMs) has achieved substantial progress through several complementary paradigms including chain-of-thought and self-consistency decoding ([29, 27]), domain-specific pretraining and instruction tuning on mathematical corpora ([18, 21]), tool-integrated reasoning that delegates computation to external programs or symbolic solvers ([8, 4] (Guo et al. 2024)), and reinforcement-learning-based reasoning models ([17, 26, 20, 10, 32]). Recent work on Atomic Thinking argues that mathematical intelligence can be decomposed into fine-grained atomic capabilities (Kuang et al., [16]). However, decoupling abilities alone does not determine how these heterogeneous skills should be composed for each unseen problem. Our work addresses this gap by dynamically composing atomic mathematical LoRA experts at the instance level.

Skill Decomposition and Expert Composition

Decomposing complex reasoning tasks into simpler subproblems or specialized skills has been shown to improve performance on challenging tasks ([31, 24, 23, 15, 38]). This perspective is also consistent with cognitive accounts in which complex problem solving involves coordinating multiple specialized capabilities ([2, 37]). While these studies primarily investigate task decomposition or the construction of intermediate subgoals, AMR focuses on the complementary problem of composing a fixed library of independently trained atomic mathematical experts for each input. From a methodological perspective, AMR is related to parameter-efficient adapter composition ([22]), mixture-of-experts routing ([7]), and model merging ([25]), which investigate how specialized modules or parameter updates can be combined or selected. AMR applies this general composition perspective to atomic mathematical skills and studies whether problem-conditioned expert mixtures can improve mathematical reasoning.

Figure 1. Conventional LLMs treat mathematical reasoning as a monolithic capability, often failing across diverse mathematical problems. In contrast, AMR dynamically composes atomic reasoning experts for each input through instance-level routing, enabling problem-adaptive capability synergy and more accurate mathematical reasoning.

Methodology

In this section, we introduce the three main components of AMR. We first construct a library of atomic LoRA experts based on fine-grained mathematical capabilities (LoRA for Each Atomic Ability). Next, we build a compositional prototype set through gradient-free search on the validation set (Composition Prototype Set Construction). Finally, we generate prototype-based pseudo-labels and train a lightweight router to select a suitable expert composition for each input during inference (Instance-Level Routing via Prototype-Based Pseudo-Labels).

LoRA for Each Atomic Ability

Following the taxonomy of mathematical reasoning abilities defined by Kuang et al. [16], we denote the set of NN atomic abilities as A={a1,a2,…,aN}\mathcal{A} = \{a_1, a_2, \ldots, a_N\}. To ensure a clear decoupling of these distinct cognitive skills, we train a dedicated LoRA module ψi\psi_i (Hu et al. [14]) for each corresponding atomic ability ai∈Aa_i \in\mathcal{A}. Specifically, each module is fine-tuned independently using the specialized datasets provided by Kuang et al. [16]. This process results in a comprehensive library of LoRAs, where each ψi\psi_i encapsulates the specific atomic ability of its corresponding task.

Composition Prototype Set Construction

Each atomic LoRA module has its specific mathematical knowledge. However, solving real mathematical problems requires the dynamic synergy of these capabilities. For instance, an analytic geometry problem demands a deep integration of both algebraic manipulation and spatial geometric intuition, whereas a pure algebraic equation requires concentrated algebraic reasoning. A static combination such as applying fixed global weights to all problems fails to adapt to the diverse demands of different questions, often introducing irrelevant noise.

To overcome the limitation that a single composition strategy struggles to adapt to diverse mathematical problems, we discard the reliance on a unique global compositional solution. Instead, we collect a diverse set of compositional prototypes through multiple gradient-free searches. Specifically, given a designated search subset of the validation set, SsearchS_{\mathrm{search}}, we search for weight vectors w=[w1,…,wN]T\mathbf{w} = [w_1, \ldots, w_N]^{\mathsf{T}} within a continuous bounded search space Ω⊂RN\Omega\subset\mathbb{R}^N. For a pre-trained weight W(0)W^{(0)} of a target linear layer in the base model, we denote the low-rank update corresponding to the ii-th atomic LoRA module as ΔW(i)=B(i)A(i)\Delta W^{(i)} = B^{(i)}A^{(i)}, and adopt a linear incremental composition:

Overview of the proposed Atomic MathReasoner framework

Figure 2. Overview of the proposed Atomic MathReasoner (AMR) framework. Top (Stage 1): Prototype Construction. We first decouple mathematical reasoning into atomic capabilities and train corresponding LoRA modules. A gradient-free optimization algorithm is then applied to the validation set to search for localized capability compositions, constructing a Compositional Prototype Set. Bottom (Stage 2): Route Selection Mechanism. We generate pseudo-labels via token-level NLL and apply a margin-based filtering strategy to obtain a noise-removed training set. Finally, a lightweight router utilizing TF-IDF feature extraction is trained to dynamically predict a suitable combination for unseen problems.

W(w)=W(0)+∑i=1NwiΔW(i)=W(0)+∑i=1NwiB(i)A(i).(1)W(\mathbf{w}) = W^{(0)} + \sum_{i=1}^{N} w_i \Delta W^{(i)} = W^{(0)} + \sum_{i=1}^{N} w_i B^{(i)} A^{(i)}. \tag*{(1)}

Subsequently, we employ a gradient-free optimization algorithm (e.g., NGOpt) to search for the composition coefficients that minimize the validation loss:

w∗=arg min⁡w∈Ω{Lval(w)+λ1N∑i=1N∣wi∣}.(2)\mathbf{w}^{*} = \underset{\mathbf{w}\in\Omega}{\operatorname{arg\,min}} \left\{ \mathcal{L}_{\mathrm{val}}(\mathbf{w}) + \lambda\frac{1}{N} \sum_{i=1}^{N} \lvert w_i \rvert\right\}. \tag*{(2)}

where Lval(w)\mathcal{L}_{\mathrm{val}}(\mathbf{w}) denotes the causal language modeling loss evaluated on the validation samples, and λ\lambda is the L1 sparsity regularization coefficient. By repeatedly executing this process across different support subsets of the validation set and under varying search budget settings, we obtain multiple validation-loss local optima in the composition space. After deduplicating the repeated solutions, these vectors constitute the compositional prototype set:

Π={π1,…,πM}.(3)\Pi= \{\pi_1, \ldots, \pi_M\}. \tag*{(3)}

where each prototype πm\pi_m corresponds to an atomic capability synergy pattern.

Instance-Level Routing via Prototype-Based Pseudo-Labels

While the prototype set constructed in the previous stage encapsulates diverse synergistic strategies, dynamically matching an unseen problem to a suitable composition remains a challenge. To establish a fast and direct mapping from the problem text to a composition, we frame instance-level prototype selection as a classification task.

Given the compositional prototype set Π={π1,…,πM}\Pi= \{\pi_1, \ldots, \pi_M\}, where each prototype corresponds to a fixed vector of composition weights, we generate prototype-based pseudo-labels by evaluating the token-level negative log-likelihood (NLL) for each input-output pair (xn,yn)(x_n, y_n) in a routing subset of the held-out validation set SrouteS_{\mathrm{route}}, which is disjoint

Algorithm 1 Inference Pipeline of AMR

Require: Problem xx, base weights W(0)W^{(0)}, atomic LoRAs {ΔW(i)}i=1N\{\Delta W^{(i)}\}_{i=1}^{N}, prototypes Π\Pi, trained router

  1. ϕ(x)←TFIDF⁡(x)\phi(x) \leftarrow\operatorname{TFIDF}(x)

  2. m∗←Router⁡(ϕ(x))m^* \leftarrow\operatorname{Router}(\phi(x))

  3. w∗←πm∗w^* \leftarrow\pi_{m^*}

  4. W(w∗)←W(0)+∑iwi∗ΔW(i)W(w^*) \leftarrow W^{(0)} + \sum_i w_i^* \Delta W^{(i)}

  5. y←LLM.Generate⁡(x∣W(w∗))y \leftarrow\operatorname{LLM.Generate}(x \mid W(w^*))

  6. return yy

from the prototype-search subset SsearchS_{\mathrm{search}}. The prototype achieving the lowest NLL is assigned as the prototype-based pseudo-label znz_n:

zn=arg⁡min⁡m∈{1,…,M} LNLL(xn,yn;πm).(4)z_n = \underset{m \in\{1,\ldots,M\}}{\arg\min}\ \mathcal{L}_{\mathrm{NLL}}(x_n,y_n;\pi_m). \tag*{(4)}

We use this NLL-selected label as a tractable proxy for prototype preference, rather than assuming that it is the unique semantic optimum for the problem. Directly learning from these pseudo-labels introduces noise, as multiple prototypes may yield comparable NLLs for a given problem. To address this, we introduce a margin-based filtering mechanism that governs a critical trade-off between label purity and sample coverage.

Let ℓn(1)\ell_n^{(1)} and ℓn(2)\ell_n^{(2)} denote the lowest and second-lowest NLLs across all prototypes. We define the margin Δn=ℓn(2)−ℓn(1)\Delta_n = \ell_n^{(2)}-\ell_n^{(1)}. By enforcing a margin threshold δ>0\delta> 0, we filter out ambiguous samples and construct a refined training set:

Drouter={(xn,zn)∣Δn≥δ},Δn=ℓn(2)−ℓn(1).(5)\mathcal{D}_{\mathrm{router}} = \{(x_n,z_n) \mid\Delta_n \ge\delta\}, \qquad\Delta_n = \ell_n^{(2)}-\ell_n^{(1)}. \tag*{(5)}

A properly tuned δ\delta ensures the router learns from high-confidence mappings without excessively truncating the problem-type distribution. To avoid introducing excessive latency during testing, we do not employ neural network routers such as Transformer encoders. Instead, we extract non-parametric TF-IDF features ϕ(xn)\phi(x_n) from the text and train a lightweight multinomial logistic regression classifier.

The classifier is optimized using standard cross-entropy loss with L2L_2 regularization over Drouter\mathcal{D}_{\mathrm{router}}. During inference, for any unseen problem xx, the router predicts a prototype index in negligible time. The corresponding weight vector is then retrieved to linearly combine the atomic LoRA modules. Ultimately, this paradigm replaces costly online dynamic searches with a highly efficient offline-trained classifier, allowing each problem to use a composition of reasoning abilities.

Experiment

Experimental Details

Baseline. In this paper, we employ Llama-3-8B-Instruct [1] as a general LLM, and InternLM2-Math-Plus-7B [34] and Qwen2.5-Math-Instruct-7B [33] as math-specialized LLMs, to serve as the baselines for applying our method. Furthermore, we select diverse open-source and state-of-the-art closed-source LLMs for comparison; detailed in Appendix. For reasoning enhancement frameworks, we compare AMR with methods which aim to improve mathematical reasoning, including DPO [17], Step-DPO [17], Math-Shepherd [26], RSD [20], MathFimer [32]. Datasets. We evaluate performance on AtomicMath benchmark [16], which includes both field and logical dimensions. For field capabilities, we collect and categorize data from MATH [13], GSM8K [6], Gaokao-Bench [39], OlympiadBench [11], AIME, MMLU [28], and DeepMath [12] into four fields (algebra, geometry, analysis, topology) across two difficulty levels. For logical capabilities, we derive conceptual understanding tasks from NaturalProofs (Wellend et al., 2021), formal language forward reasoning tasks from Lean Workbook, and counterexample-driven backward reasoning statements from CounterMath [19]. Notably, we select GSM8K [6] and MATH [13] for comparison with other methods for mathematical reasoning enhancement. We follow the official AtomicMath train/test split and we hold out 10% training data as validation. The training and validation splits are disjoint from the test split: atomic LoRA experts are trained only on the training split, prototype search and router fitting are performed only on the validation split, and the test split is reserved strictly for evaluation. Evaluation and Metrics. Following Kuang et al. [16], we adopt default Chain-of-Thought (CoT) prompting for all evaluations. For field capabilities, conceptual understanding, and forward reasoning tasks, we use Accuracy computed via exact match with reference answers. For backward reasoning, we evaluate truthfulness judgment using the F-1 score. Additionally, to assess counterexample construction, we use Example extraction alongside Strict and Loose alignment metrics to measure consistency with reference examples.

Implementation Details. We train a separate atomic LoRA adapter for each capability using LLaMA-Factory (Zheng et al. [40]). To ensure a fair comparison, all atomic adapters share the same training recipe, including 4-bit quantization, LoRA rank 16, batch size of 16, learning rate 10−410^{-4}, and 2 training epochs. For the gradient-free composition search, the continuous bounded search space Ω\Omega is set to [−1.5,1.5]N[-1.5, 1.5]^N (where N=7N = 7 represents the number of decoupled atomic experts), and the L1 sparsity regularization coefficient λ\lambda is set to 0.05. Unless otherwise specified, the router uses all M=9M = 9 prototypes and a margin threshold of δ=0.02\delta= 0.02. We sample 512 examples from validation split as the candidate pool for pseudo-label assignment; after margin filtering, the retained high-confidence examples are used to train the router. All composition search, prototype extraction, and router fitting are conducted exclusively on the validation split, while the test split is used only for final evaluation. Unless otherwise specified, all reported results are obtained from three runs under three random seeds. More details are in Appendix.

Main Results

Field Ability. As shown in Table 1, AMR improves the field abilities of Llama-3-8B-Instruct. Beyond improving its Level 1 and 2 average accuracy to 38.6% and 27.9%, it achieves 54.3% in Algebra and 45.2% in Analysis on Level 1 tasks, outperforming math-specific baselines like Deepseek-math-RL-7B and NuminaMath-7B-TIR. AMR’s enhancement is not limited to general models; it consistently boosts the performance of math-specialized models as well. Integrating AMR with the InternLM2-Math-Plus-7B enhances its average accuracy to 46.2% (+8.4%) on Level 1 tasks and 42.7% (+6.2%) on Level 2 tasks. Notably, it outperforms 13B specialized models in some domains: it achieves 48.4% in Analysis (Level 1) and 38.8% in Topology (Level 2), successfully surpassing Abel-13B (40.4%) and Xwin-Math-13B (38.3%), respectively. For Qwen2.5 empowered by AMR, the average accuracy increases to 68.1% (+3.7%) on Level 1 and 63.7% (+4.4%) on Level 2. This performance indicates that it outperforms across all open-source models; for instance, in Algebra Level 2, its score of 71.4% far surpasses InternLM2-Math-Plus-20B (49.6%) and Xwin-Math-13B (51.6%). Impressively, even when compared with commercial models, it still remains remarkably competitive and even surpasses them in specific domains. For instance, its Analysis (Level 2) score of 73.5% surprisingly exceeds all evaluated commercial models. Furthermore, its Geometry (Level 2) score of 53.8% significantly outperforms both Claude 3.7 Sonnet (39.0%) and GPT-4o (36.3%).

MethodAlgebra LowAlgebra HighGeometry LowGeometry HighAnalysis LowAnalysis HighTopology LowTopology HighAvg. LowAvg. High
Llama-3-8B-Instruct48.228.730.524.339.127.522.020.135.025.2
InternLM2-Math-Plus-7B49.235.933.031.441.941.527.237.037.836.5
Deepseek-math-RL-7B52.033.835.539.344.637.622.633.738.736.1
NuminaMath-7B-TIR52.428.239.932.146.833.626.425.141.429.8
Qwen2.5-Math-Instruct-7B80.565.257.351.967.766.552.153.464.459.3
Abel-13B63.447.342.741.840.435.630.224.644.237.3
Xwin-Math-13B78.351.649.546.755.754.846.738.357.647.9
InternLM2-Math-Plus-20B67.849.649.346.154.347.845.438.954.245.6
GPT-4o69.350.548.536.355.153.253.049.456.547.4
Claude 3.7-Sonnet83.572.058.039.080.368.156.058.669.559.4
Gemini 2.5-Pro80.579.670.273.167.563.868.166.271.670.7
Llama-3 + AMR (Ours)54.333.231.725.445.231.523.121.438.627.9
InternLM2 + AMR (Ours)55.140.335.433.148.442.629.538.842.138.7
Qwen2.5 + AMR (Ours)84.171.459.953.874.373.553.955.968.163.7

Table 1. Performance of AMR on the field atomic capabilities of AtomicMath (three runs). Values are accuracies under low and high difficulty.

Logic Ability. As shown in Table 2, AMR effectively enhances the logical reasoning capabilities of Llama-3-8B-Instruct. It raises the overall average to 31.6%, driven by steady improvements in Concept Attribute (43.5%) and Forward Reasoning (21.8%). The enhancements also consistently extend to math-specialized models. Equipping InternLM2-Math-Plus-7B with AMR elevates its average logic score to 42.9% (+6.1%). Notably, it achieves a remarkable surge in Forward Reasoning, jumping from 23.7% to 36.5%. This enhancement enables it to comprehensively surpass the logic performance of larger open-source models, including InternLM2-Math-Plus-20B (36.9%) and Xwin-Math-13B (34.8%). With AMR, the average performance of Qwen2.5 improves by 7.4% to 46.8%. It shows strong growth in complex reasoning tasks, achieving 46.8% (+12.4%) in Forward Reasoning and 46.5% (+8.2%) in Backward Reasoning. Impressively, even when compared to commercial models, it stays very competitive and wins in specific areas. Most notably, it achieves an overall average score of 46.8% and a Forward Reasoning accuracy of 46.8%, successfully outperforming GPT-4o which scores 45.8% and 37.7% in the respective metrics.

MethodConcept Attr.Concept Def.Forward Acc.Backward F1Avg.
Llama-3-8B-Instruct42.135.220.524.330.5
InternLM2-Math-Plus-7B46.243.223.733.936.8
Deepseek-math-RL-7B46.539.423.127.634.2
NuminaMath-7B-TIR27.322.830.230.427.7
Qwen2.5-Math-Instruct-7B50.334.434.438.339.4
Abel-13B31.248.037.222.434.7
Xwin-Math-13B45.929.833.130.234.8
InternLM2-Math-Plus-20B38.352.937.818.436.9
GPT-4o38.148.337.759.045.8
Claude 3.7-Sonnet60.745.856.864.857.0
Gemini 2.5-Pro60.448.656.777.060.7
Llama-3 + AMR (Ours)43.537.821.823.131.6
InternLM2 + AMR (Ours)47.649.136.538.442.9
Qwen2.5 + AMR (Ours)52.741.246.846.546.8

Table 2. Logic atomic capabilities of AtomicMath (three runs); accuracy for concept understanding and forward reasoning, F1 for backward reasoning.

Results on Other Benchmarks. To further verify the effectiveness of AMR, we further evaluate AMR on GSM8K and MATH datasets comparing with other frameworks for enhancing mathematical reasoning in LLMs. As summarized in Table 3, AMR exhibits highly competitive performance, achieving absolute gains of 3.7% on GSM8K and 3.2% on MATH over the base model. These improvements clearly surpass those obtained by recent advanced methods such as MathFimer ([32]) (+2.4% and +2.3%, respectively) and RSD ([20]) (+1.9% and +1.8%, respectively).

MethodGSM8KMATH
Baseline79.634.7
DPO80.435.2
Step-DPO81.336.0
Math-Shepherd81.035.8
RSD81.536.5
MathFimer82.037.0
AMR (Ours)83.337.9

Table 3. Comparison on GSM8K and MATH. Baseline is Llama-3-8B-Instruct.

Ablation Study

we conduct an ablation study on Qwen2.5-Math-7B-Instruct to evaluate their importance, as summarized in Table 4. We first evaluate Joint LoRA, which is trained on the mixture of all atomic datasets,improves the base model by 1.2% overall, while the Best Single LoRA achieves a larger gain of 2.8%, suggesting that atomic specialization is more effective than directly mixing heterogeneous mathematical abilities. We further compare different composition strategies. Static Composition learns a globally shared combination of atomic experts and already improves the Overall score from 50.6% to 52.7%, demonstrating that composing multiple atomic capabilities can produce effective reasoning synergy. Dynamic Composition further boosts performance by 1.8% over Static Composition through support-dependent combination weights, indicating that different support sets benefit from different expert mixtures rather than a universal composition. Building on this observation, AMR introduces instance-level prototype routing to select a more suitable composition for each problem, achieving the best overall performance of 56.4% in AtomicMath benchmark.

MethodOverallField Avg.Logic Avg.
Base50.661.939.4
Joint LoRA51.862.740.9
Best Single LoRA53.463.843.0
Static Composition52.763.342.2
Dynamic Composition54.564.544.5
AMR (Ours)56.465.946.8

Table 4. Ablation study on Qwen2.5-Math-7B-Instruct.

LOO Analysis of Atomic Experts

To better understand the contribution of each atomic expert within AMR, we conduct a leave-one-out (LOO) ablation, where one expert is removed from the full expert pool at a time while keeping the rest of the framework unchanged. As shown in Table 5, the most critical component is Conceptual Understanding. Removing this expert causes the largest overall drop, from 56.4 to 52.7, and leads to a particularly severe degradation on Logic Avg., which falls from 46.8 to 42.4. This result indicates that conceptual reasoning plays a foundational role in coordinating the higher-level logical behaviors required by mathematical problem solving. Among the remaining experts, Analysis and Algebra also make substantial contributions. Excluding Analysis reduces overall performance to 55.4 and lowers Field Avg. from 65.9 to 64.3, suggesting that it is especially important for mathematical tasks. Removing Algebra similarly decreases performance to 55.7, confirming that symbolic manipulation remains a core ingredient in the composed reasoning process. By contrast, removing Forward Reasoning, Backward Reasoning, Geometry, or Topology leads to only minor changes, implying that these experts are helpful but more complementary in the presence of the full expert pool. Overall, the LOO analysis supports our hypothesis that atomic experts are not equally important: AMR derives most of its gains from a small subset of highly influential capabilities, especially concept and analysis.

Model / Drop ModuleOverallField Avg.Logic Avg.
AMR (Ours)56.465.946.8
w/o Algebra55.764.846.6
w/o Analysis55.464.346.5
w/o Backward Rea.56.165.846.4
w/o Concept52.763.042.4
w/o Forward Rea.56.265.846.6
w/o Geometry56.165.446.8
w/o Topology56.365.846.8

Table 5. Leave-One-Out (LOO) ablation of atomic expert modules.

Efficiency and Parameter Analysis

Efficiency. To evaluate inference efficiency, we test the latency on a 256-example subset of Qwen2.5-Math-7B-Instruct with a single NVIDIA GeForce RTX 3090 GPU. Compared with the baseline Dynamic Composition (budget=10) that costs 184.0 seconds for inference, our framework only consumes 66.7 seconds, achieving a remarkable 2.76× speedup. The significant efficiency improvement stems from abandoning online gradient-free search during inference. Instead of performing iterative optimization for each test instance, AMR pre-constructs a prototype pool in the offline training phase. At test time, it merely conducts one lightweight forward pass and Logistic Regression classifier to rapidly route the input question to the prototype configuration.

Hyperparameter Analysis. We conduct sensitivity experiments on three hyperparameters. For prototype pool size, we expand the pool from top-4 to all 9 prototypes yields improvement across all metrics, with overall accuracy rising from 53.8% to 56.4% and the Field Avg. from 63.0% to 65.9%, showing that higher prototype size could enhance the overall performance. On the pseudo-label margin threshold δ\delta, δ=0\delta=0 retains ambiguous labels that blur decision boundaries, while δ=0.05\delta=0.05 excessively filters valid instances, δ=0.02\delta=0.02 balances label quality and the number of retained samples, achieving best result. Regarding the size of training set, AMR attains 56.4% overall accuracy with 512 samples, reducing to 256 samples leads to underfitting, whereas scaling up to the full set degrades accuracy to 55.8%, indicating that excessive training data introduces noise with marginal overfitting. More details in the Appendix.

HyperparameterOverallField Avg.Logic Avg.
Prototype Pool Size
Top-4 prototypes53.863.044.6
Top-6 prototypes54.363.545.1
All (M=9M = 9) prototypes56.465.946.8
Margin Threshold (δ\delta)
δ=0\delta= 055.965.446.4
δ=0.02\delta= 0.0256.465.946.8
δ=0.05\delta= 0.0556.165.746.5
Training Set Size
256 samples55.465.145.7
512 samples56.465.946.8
2048 samples56.065.546.5
All (N=7439N = 7439) samples55.865.346.3

Table 6. Hyperparameter sensitivity of AMR on Qwen2.5-Math-7B-Instruct.

Conclusion

In this paper, we presented Atomic MathReasoner (AMR), a two-stage framework for enhancing mathematical reasoning in LLMs through instance-level composition of atomic mathematical capabilities. AMR first constructs a diverse set of compositional prototypes by performing gradient-free search over independently trained atomic LoRA experts, and then distills these prototypes into a lightweight router that efficiently selects a suitable expert composition for each input during inference. Extensive experiments on the AtomicMath benchmark, together with evaluations on GSM8K and MATH, demonstrate that AMR consistently improves the mathematical reasoning performance of both general-purpose and math-specialized LLMs. Comprehensive ablation studies further verify the effectiveness of atomic specialization, compositional prototype construction, and instance-level routing, highlighting the importance of adaptive expert composition for mathematical reasoning.

References

  1. [1]AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  2. [2]John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger, and Ray Pelletier. Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2):167–207, 1995. doi: 10.1207/s15327809jls0402_2. URL https://doi.org/10.1207/s15327809jls0402_2.
  3. [3]Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Lemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4WnqR9l5j.arxiv.org/abs/2310.10631
  4. [4]Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.arxiv.org/abs/2211.12588
  5. [5]Yifei Chen, Guanting Dong, and Zhicheng Dou. Toward effective tool-integrated reasoning via self-evolved preference learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=mNeitRAdWV.arxiv.org/abs/2509.23285
  6. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reirichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  7. [7]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  8. [8]Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International conference on machine learning, pages 10764–10799. PMLR, 2023.arxiv.org/abs/2211.10435
  9. [9]Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In International Conference on Learning Representations, volume 2024, pages 48362–48395, 2024.arxiv.org/abs/2309.17452
  10. [10]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  11. [11]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
  12. [12]Zhiwei He, Tian Liang, Jiahao Xu, Quizhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=kHB5TeE5Wm.arxiv.org/abs/2504.11456
  13. [13]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
  14. [14]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.arxiv.org/abs/2106.09685
  15. [15]Yuxuan Jiang and Francis Ferraro. Scribe: Structured mid-level supervision for tool-using language models. arXiv preprint arXiv:2601.03555, 2026.arxiv.org/abs/2601.03555
  16. [16]Jiayi Kuang, Haojing Huang, Yinghui Li, Xinnian Liang, Zhikun Xu, Yangning Li, Xiaoyu Tan, Chao Qu, Meishan Zhang, Ying Shen, and Philip S. Yu. Atomic thinking of LLMs: Decoupling and exploring mathematical reasoning abilities. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=iBFfb6bG0z.
  17. [17]Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024.arxiv.org/abs/2406.18629
  18. [18]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022.arxiv.org/abs/2206.14858
  19. [19]Yinghui Li, Jiayi Kuang, Haojing Huang, Zhikun Xu, Xinnian Liang, Yi Yu, Wenlian Lu, Yangning Li, Xiaoyu Tan, Chao Qu, Ying Shen, Hai-Tao Zheng, and Philip S. Yu. One example shown, many concepts known! counterexample-driven conceptual reasoning in mathematical LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=A31Ep22iQ7.arxiv.org/abs/2502.10454
  20. [20]Baoh​ao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient LLM reasoning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=AVeskAAETB.
  21. [21]Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.arxiv.org/abs/2308.09583
  22. [22]Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume, pages 487–503, 2021.arxiv.org/abs/2005.00247
  23. [23]Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560, 2022.arxiv.org/abs/2207.07560
  24. [24]Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018, 2025.arxiv.org/abs/2502.12018
  25. [25]Jinping Wang, Zhiqiang Gao, Zhang Dinggen, and Zhiwu Xie. Escaping optimization stagnation: Taking steps beyond task arithmetic via difference vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 26310–26318, 2026.
  26. [26]Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, 2024a.
  27. [27]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.arxiv.org/abs/2203.11171
  28. [28]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhramil Chandra, Shiguang Guo, Weiming Ren, Aaron Aulraj, Xuan He, Xiyan Jiang, et al. Alm-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024b.arxiv.org/abs/2406.01574
  29. [29]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
  30. [31]Kun Xiang, Zhili Liu, Zihao Jiang, Yunshuang Nie, Haoxiang Fan, Weiran Huang, et al. Can atomic step decomposition enhance the self-structured reasoning of multimodal large models? arXiv preprint arXiv:2503.06252, 2025.arxiv.org/abs/2503.06252
  31. [32]Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Xin Xu, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Mathfimer: Enhancing mathematical reasoning by expanding reasoning steps through fill-in-the-middle task. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=14iwzPPfn.arxiv.org/abs/2502.11684
  32. [33]An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024.arxiv.org/abs/2409.12122
  33. [34]Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayuan Wang, Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning, 2024.arxiv.org/abs/2402.06332
  34. [35]Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=N8N0hgNDRt.arxiv.org/abs/2309.12284
  35. [36]Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=yJClGs770T.arxiv.org/abs/2309.05653
  36. [37]BeiLei Zhang, Gino M. Salituro, Deborah Szalkowski, Zhihua Li, Yan Zhang, Inmaculada Royo, Dolores Vilella, María Teresa Diéz, Fernando Peláez, Caroline Ruby, Richard L. Kendall, Xianzhi Mao, Patrick Griffin, Jimmy Calaycay, Juleen R. Zierath, James V. Heck, and Roy G. Smith. Sources of mathematical thinking: behavioral and brain-imaging evidence. Science, 284 5416:970–4, 1999. URL https://api.semanticscholar.org/CorpusID:10523949.
  37. [38]Likui Zhang, Tao Tang, Zhihao Zhan, Xiuwei Chen, Zisheng Chen, Jianhua Han, Jiangtong Zhu, Pei Xu, Hang Xu, Hefeng Wu, et al. Atomicvla: Unlocking the potential of atomic skill learning in robots. arXiv preprint arXiv:2603.07648, 2026.arxiv.org/abs/2603.07648
  38. [39]Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. 2023.arxiv.org/abs/2305.12474
  39. [40]Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Lla-mafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372.

Paper details

vlnc:W_8uyth2th
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents