betaFeedback

WorksComputer Science

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

Abstract

Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM. Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.

github.com/berkearda/skilleval | berkearda.github.io/skilleval

Introduction

Benchmarks for large language models (LLMs) have rapidly proliferated in recent years. While this creates new opportunities, it also raises a key challenge: how to effectively leverage a large collection of benchmarks for comprehensive LLM evaluation? Current evaluation practices remain quite raw. The common approach is to evaluate a model on all items in a benchmark and then report the average score. This naive statistical way collapses performance across items into a single number, which obscures fine-grained capability differences [7]. For example, two LLMs with similar overall accuracy on a math benchmark (0.63 vs. 0.65) may exhibit very different profiles across sub-skills such as algebra and geometry. This leads to a natural question: *Given a set of test items, how can we go beyond a scalar score to a more informative, human-understandable capability estimate?

To answer this, we develop a new evaluation framework that aims to assess the latent abilities underlying observed item scores, and represents them as fine-grained skill profiles of LLMs. This problem has long been studied in psychometrics and the education community using various latent variable models. A representative family of approaches is Item Response Theory [19], which jointly models test-taker traits (e.g., ability) and item traits (e.g., difficulty and discrimination). IRT models are also increasingly used in machine learning, but typically for data-centric tasks such as dataset analysis or distillation [18, 20, 33, 24, 8], and less often for LLM evaluation as it still produces a scalar overall ability score that suffers from capability collapse. Some recent work explores multidimensional-IRT [21] and its neural variants that learn ability vectors [3, 6, 27, 12]. However, these multidimensional models are known to suffer from poor interpretability and non-identifiability, making them less suitable for reliable evaluation.

In this study, we propose SKILLEVAL, a new evaluation framework that identifies latent skills and estimates LLM abilities at the skill level. The framework consists of a skill discovery module, which extracts shared skills from problems, and a neural cognitive diagnosis model (CDM; Wang et al., 2020), which estimates per-skill abilities based on model performance on skill-associated items. We evaluate SKILLEVAL on a large-scale dataset of 36M model–item response records, covering 3,811 models and 9,523 items across five benchmarks from various domains, including MATH [15], BBH [29], GPQA [23], MuSR [28], and IFEval [32]. We conduct extensive experiments and demonstrate the superiority of SKILLEVAL from various aspects:

  • Compared to MIRT-style methods, SKILLEVAL extracts high-quality, human-understandable skills and produces stable ability estimates across runs, ensuring reproducible and robust evaluation.

  • SKILLEVAL accurately predicts model performance at both the item (with AUC=81.73\mathrm{AUC}=81.73) and benchmark (with Pearson r=0.989r=0.989) levels, outperforming strong baselines. In a downstream routing task where the goal is to assign queries to a cheap yet capable model, SKILLEVAL matches the strongest LLM at 25% of the cost (τ=0.80\tau=0.80, 99.9% of strongest accuracy), and exceeds it by 3.7 points at 37% of the cost (τ=0.95\tau=0.95). It also profiles previously unseen LLMs efficiently without retraining the shared network: 20 adaptively chosen answers yield a mean benchmark-score error comparable to 100 randomly chosen answers.

  • In addition, SKILLEVAL serves as a useful tool for analyzing dataset structure and model behavior, such as skill coverage across benchmarks, co-mastery patterns among skills, and skill-level differences between base and instruction-tuned models, offering useful insights into data curation and model training.

Profiling LLMs' Skills with Cognitive Diagnostic Models

Problem Statement

Given a set of models to be evaluated {M1,…,MM}\{\mathcal{M}_1,\ldots,\mathcal{M}_M\} and a collection of test items {x1,…,xN}\{x_1,\ldots,x_N\} from one or multiple benchmarks, we represent their performance using a matrix S∈RM×N\mathbf{S}\in\mathbb{R}^{M\times N}, where each entry SijS_{ij} denotes the score of the ii-th model on the jj-th item. We formulate LLM profiling as a function F\mathcal{F} that takes S\mathbf{S} as input and outputs some estimates for each model’s capabilities, i.e., F(S)→{θ1,…,θM}\mathcal{F}(\mathbf{S})\rightarrow\{\theta_1,\ldots,\theta_M\}, where θi\theta_i is a quantitative measurement of the capabilities of model ii.

The common practice in evaluation is to model F\mathcal{F} as a simple average over individual items, i.e., θi=1n∑j=1nSij\theta_i=\frac{1}{n}\sum_{j=1}^{n}S_{ij}. While this provides a concise summary of model performance, it suffers from two key limitations. First, it compresses the model’s performance into a single scalar metric, failing to capture variations across fine-grained skills. Second, the naive average can be affected by bias induced by the difficulty distribution of the test items, and may not faithfully reflect the model’s true ability. Indeed, this type of ability modeling assumes that all items in a given benchmark contribute equally to the estimation of a single ability dimension, which is typically an implausible assumption.

To address these limitations, we propose a new paradigm for profiling LLMs. We assume that the items in the benchmark are associated with a set of underlying latent skills C={c1,…,cK}\mathcal{C} = \{c_1, \ldots, c_K\}. Inspired by psychometric approaches for assessing latent traits, we use Cognitive Diagnosis Models (CDMs) to construct the profiling function FCD(S)→Θ∈RM×K\mathcal{F}_{\mathrm{CD}}(S) \to\Theta\in\mathbb{R}^{M \times K}, where the profile θi\theta_i for each model is a kk-dimensional vector representing its proficiency over the set of skills. An overview of our method is shown in Figure 1.

Overview of the profiling method

Figure 1. We first extract skills from the questions and construct a question-to-skill mapping (i.e., the QQ-matrix). Then, using this QQ-matrix together with the performance matrix of models on items, we train a neural CDM to estimate each model’s skill-level abilities θ\theta.

Q-Matrix Construction

We use a binary QQ-matrix Q∈{0,1}N×KQ \in\{0, 1\}^{N \times K} to denote the mapping between NN items and KK skills, where each entry Qij=1Q_{ij} = 1 indicates the ii-th item is associated with the jj-th skill and 0 otherwise. Manually defining and annotating skills is challenging, costly, and difficult to scale. Recent work shows that LLMs can automatically identify meaningful skills from problem content [10, 14]. We adapt this approach to our setting, consisting of two main steps: skill labeling and skill clustering.

Skill labeling. For each item jj, we prompt GPT-4o-mini with the question stem and ask for (i) a single primary skill and (ii) the full list of skills required for the solution (exact prompt in Appendix DD). We use the full list rather than only the primary skill: primary-skill assignments produce QQ-matrix rows of mean row-sum 1.0, which under-constrains NCDM. Full-list extraction yields a mean of 2.9 skills per item and 13,276 unique skill strings across 9,523 items.

Skill clustering. Because skills are extracted independently from different problems, they may contain redundancy and noise. To obtain a more compact and coherent skill set, we cluster similar skills and merge them accordingly. Didolkar et al. [10] employ an LLM to perform this grouping process; however, such approaches may not scale well to a large number of skills. Therefore, we adopt an embedding-based clustering method, which is more efficient and scalable. Each unique skill string is first encoded by the pretrained SBERT all-mpnet-base-v2 [22] into R768\mathbb{R}^{768}. Then, we use the hierarchical agglomerative clustering (HAC) algorithm with cosine distance and average linkage, specifying the number of final clusters as KK.

Q-matrix assembly. We set Qjk=1Q_{jk} = 1 iff at least one of item jj’s extracted skills falls into cluster kk. This multi-skill assignment reduces the fraction of clusters with fewer than ten items from 43% under primary-skill assignment to 36% at K=100K = 100,2 yielding a mean column-sum of ∼139\sim139 items per skill (mean row-sum 1.5 clusters per item, reflecting the compression of the 2.9 extracted skill strings into shared clusters), the regime in which NCDM trains stably.

Neural CDM

We adopt the Neural Cognitive Diagnosis Model (Wang et al., 2020). The probability that LLM ss answers item jj correctly is estimated as:

p(Sij=1)=sigmoid⁡(f(αj(θi−dj)⊙qj)),(1)p(S_{ij}=1)=\operatorname{sigmoid}\left(f\left(\alpha_j\left(\theta_i-d_j\right)\odot q_j\right)\right), \tag*{(1)}

where θi∈[0,1]K\theta_i\in[0,1]^K represents the learned skill mastery of model Mi\mathcal{M}_i, αj∈[0,1]\alpha_j\in[0,1] is the discrimination parameter of item jj, dj∈[0,1]Kd_j\in[0,1]^K is its per-skill difficulty vector, and qj∈{0,1}Kq_j\in\{0,1\}^K is the corresponding Q-matrix row. The scalar αj\alpha_j scales all skill dimensions equally. The function ff is a three-layer MLP with PosLinear layers. Each PosLinear layer uses non-negative weights, W+=max⁡(W,0)W^+=\max(W,0). Together with the non-decreasing sigmoid activations, this ensures that increasing any mastery component cannot decrease the predicted probability, holding all other inputs fixed.

Learning Item Parameters from Text

The original NCDM treats item discrimination αj\alpha_j and difficulty djd_j as model parameters learned during training, preventing predictions on unseen items. To overcome this limitation, we follow previous context-aware approaches ([12, 6]) to use a lightweight neural network to learn item parameters based on their text features. Specifically, for a given item xjx_j, we first use a pretrained encoder to obtain its embedding eje_j. Then, we compute αj\alpha_j and djd_j via:

αj=σ(wαTej+bα),dj=σ(Wdej+bd).(2)\alpha_j=\sigma\left(\mathbf{w}_{\alpha}^{\mathsf{T}}e_j+b_{\alpha}\right),\qquad d_j=\sigma\left(W_d e_j+b_d\right). \tag*{(2)}

To maintain computational efficiency, we freeze the pretrained encoder and only optimize the projection parameters wα\mathbf{w}_{\alpha}, WdW_d and bαb_{\alpha}, bdb_d.

Evaluation of Skill Profiles

Experiment Setup. We train our text-conditioned NCDM on a subset of the RouterEval dataset (Huang et al., [17]), including 3,811 LLMs evaluated on 9,523 items from five benchmarks: MATH, BBH, GPQA, MuSR, and IFEval. For the number of skills, we set K=100K=100 as the main experimental configuration, as it achieves a better balance between interpretability and performance. We provide comparisons with other values of KK in Appendix § 12.

Quality of Discovered Skills

Since our evaluation is centered around skills, it is crucial to assess the quality of these LLM-discovered skills. We consider the following three criteria: (1) Semantic clarity, which measures whether a skill description is clear, unambiguous, and self-contained; (2) Level of abstraction, which captures the granularity of a skill, where a good skill should be neither overly broad nor overly specific to particular questions; (3) Diversity, which evaluates whether different skills are non-redundant and capture distinct aspects. For the first two criteria, we adopt LLM-as-a-judge. For the last one, we use intra-list diversity defined as averaged pair-wise dissimilarities, which is widely used in recommendation systems.

The experimental results are shown in Table 1. As a reference, we compare our skills with the Common Core State Standards3, a widely used expert-curated framework for K–12 math education. Our skills achieve a similar level of semantic clarity to the Common Core. In terms of granularity, they are slightly more fine-grained. They also show lower variance (std. 0.29 vs. 0.55), indicating more consistent granularity across skills, which is desirable. In addition, our skills are more diverse (avg. 0.83 vs. 0.76), likely because they cover more domains and reduce redundancy through clustering. Overall, our automatically extracted skills are competitive with manually designed ones along key dimensions.

LLM-extractedExpert-defined
Clarity4.07 ▬▬▰4.14 ▬▬▰
Granularity2.26 ▰▰▬2.72 ▬▰▬
Diversity0.83 ▬▰0.76 ▬▬

Table 1. Comparison between the LLM-extracted skill set (K = 100) and an expert-defined mathematics skill set (the Common Core standards; K = 406).

Human validation of skill–question assignments. Clear skill descriptions do not by themselves establish that the skills are relevant to their assigned questions. We therefore evaluate the skill–question assignments through human annotation. Three annotators, each holding an MSc in engineering, evaluated 780 skill–question pairs covering all 100 skills. We sampled up to 10 questions uniformly at random from each skill’s assigned items, retaining all items for smaller clusters. Annotators were shown revised descriptive skill names and question text, and judged whether answering each question required the named skill. Majority voting accepted 652 of 780 assignments (83.6%), with an acceptance rate of 81.7% when averaged equally across skills. Inter-annotator agreement was 0.715 in mean pairwise Cohen’s kappa and 0.714 in Krippendorff’s nominal alpha; both measures equal 1 for perfect agreement and 0 for agreement expected by chance. These judgments provide evidence for the semantic relevance of the sampled assignments.

Stability

For an evaluation method, the stability of its results is essential. However, multidimensional latent ability models commonly suffer from non-identifiability: the θ\theta vectors learned in repeated training runs are inconsistent, as they are affected by rotation, permutation, and random initialization. This prevents such models from serving as a reliable and reproducible means of evaluation. SKILLEVAL instead ties each dimension of θ\theta to a named skill through a fixed Q-matrix, which anchors the coordinate system across runs.

We test whether ability profiles remain consistent across retraining, and compare SKILLEVAL with IRTNet (Chen et al., [3]), a high-dimensional neural IRT model that represents each LLM with a dd-dimensional latent ability vector. We train SKILLEVAL with five random seeds and IRTNet with different dimensionalities. We compute Pearson correlations across learned Θ\Theta across different runs in Table 2. Averaged over dimensions and run pairs, the correlation is 0.96 for SKILLEVAL and ≈0.00\approx0.00 for each IRTNet configuration. To rule out differences that arise merely from different coordinate systems, we also compare runs after Procrustes alignment, which finds the rotation that best maps one run’s θ\theta vectors onto another’s. Despite this, IRTNet’s mean correlations rise only to 0.10–0.43. Thus, SKILLEVAL supports more reliable and consistent evaluation of LLMs.

ModelFlatPer-LLMPer-dim RawPer-dim OrthogonalFull
IRTNet (d=232d=232)0.02 (±0.06\pm0.06)0.02 (±0.06\pm0.06)0.00 (±0.01\pm0.01)0.43 (±0.22\pm0.22)0.43 (±0.21\pm0.21)
IRTNet (d=512d=512)0.00 (±0.04\pm0.04)0.00 (±0.04\pm0.04)0.00 (±0.00\pm0.00)0.17 (±0.02\pm0.02)0.17 (±0.01\pm0.01)
IRTNet (d=1024d=1024)0.01 (±0.00\pm0.00)0.01 (±0.00\pm0.00)0.00 (±0.00\pm0.00)0.10 (±0.02\pm0.02)0.10 (±0.02\pm0.02)
SKILLEVAL (K=100K = 100)0.99 (±0.00\pm0.00)0.99 (±0.01\pm0.01)0.96 (±0.00\pm0.00)0.96 (±0.00\pm0.00)0.96 (±0.00\pm0.00)

Table 2. Mean Pearson correlations between ability matrices Θ\Theta across run pairs, with standard deviations across pairs in parentheses. Flat correlates all entries of the two ability matrices after arranging them into vectors. Per-LLM correlates each LLM’s ability profile between runs and averages over LLMs. These summaries can be high when some dimensions consistently take larger values than others. Per-dim instead compares values across LLMs separately for each dimension, then averages the correlations over dimensions. It therefore tests whether each dimension assigns similar relative values to LLMs across runs. Raw uses the dimensions as learned. Orthogonal applies the rotation or reflection that best aligns one run’s ability matrix with the other ([25]), and Full additionally centres and rescales the matrices before alignment ([13]).

Predictive Validity

Finally, we evaluate how well Θ\Theta predicts model behavior in independent tasks. On held-out items, the mean per-LLM correlation between Θ\Theta and per-skill accuracy is 0.73 across five seeds,4 validating Θ\Theta as a measurement target. We further test the full predictive power of Θ\Theta via item- and benchmark-level prediction with the trained NCDM.

Item-level Prediction. We consider both in-distribution (ID) and out-of-distribution (OOD) settings. In the ID setting, we perform an (LLM ×\times Item) interaction-wise split, where all items are observed during training. In the OOD setting, we use an item-wise split, holding out a subset of items that are never seen during training.

We compare SKILLEVAL with 2PL IRT and IRTNet in Table 3. The 2PL model scores highest in the ID setting, as it learns a separate difficulty and discrimination for every item, but for the same reason it cannot score items unseen during training. In the ID setting, SKILLEVAL performs on par with IRTNet, even though IRTNet has 34 to 39 times as many trainable parameters: 22.0M to 25.2M across the three sizes, against 0.64M for SKILLEVAL. In the OOD setting, IRTNet is ahead by 3.2 AUC and 1.8 accuracy points. This gap is expected: IRTNet lets every item use all of its latent dimensions, whereas SKILLEVAL restricts each question to the dimensions of its assigned skills. The same restriction is what gives each dimension of Θ\Theta a clear meaning and keeps it stable across runs, which IRTNet’s dimensions lack (§ 3.2). SKILLEVAL thus remains competitive in prediction while providing a far more stable and interpretable ability profile.

ModelAUCAcc
IDOODIDOOD
IRT (2PL)86.16–78.76–
IRTNet (dd=232)81.3875.0474.2169.05
IRTNet (dd=512)81.3675.1474.2169.07
IRTNet (dd=1024)81.7475.3574.6869.38
SKILLEVAL (KK = 100)81.7372.1575.5767.62

Table 3. AUC and accuracy (Acc) under two settings: ID (interaction-wise split) and OOD (item-wise split). For Acc, a response is predicted correct when its predicted probability is at least 0.5. IRT 2PL has no item features and therefore cannot generalise to held-out items; its OOD column is left blank. OOD values are means over seeds, five for SKILLEVAL and three for IRTNet. ID values are single runs.

Benchmark-level prediction. Benchmark accuracy is the quantity most often used to compare LLMs, so we ask whether item-level predictions recover each LLM’s score on each benchmark. In the OOD setting, we average an LLM’s predicted probabilities over a benchmark’s held-out items and compare the result with its observed accuracy on those items (Figure 2). Although the two methods reach similar correlations on the pooled score (rr=0.989 vs. 0.993), SKILLEVAL is more accurate within every benchmark, most clearly on MATH (0.977 vs. 0.852) and IFEval (0.966 vs. 0.718). SKILLEVAL’s predictions lie in a narrow band along the diagonal, whereas IRTNet’s spread widely: LLMs with nearly identical observed accuracy can receive very different predicted scores. IRTNet also tends to overestimate accuracy, with most points above the diagonal on MATH, GPQA and MuSR, while SKILLEVAL stays better calibrated. This is because that IRTNet’s per-benchmark errors for the same LLM point in different directions and largely cancel when averaged into the pooled score. In addition, the pooled score is also dominated by BBH, which contributes 61% of the held-out items and on which both methods already agree closely. A good pooled correlation can therefore hide inaccurate per-benchmark estimates, and the per-benchmark comparison is the one that matters when LLMs are compared on a given benchmark.

Figure 2. Predicted versus observed benchmark accuracy on held-out items in the OOD setting. Each point represents one LLM. The horizontal axis shows its observed accuracy on the benchmark’s held-out items; the vertical axis shows the mean predicted probability of a correct answer on those items. The diagonal indicates exact agreement, and rr is the Pearson correlation across 3,811 LLMs. IRTNet uses dd=232.

Item-side versus model-side accuracy. Taken together, Table 3 and Figure 2 reveal a trade-off between predicting items and comparing models. IRTNet lets every item draw on all dd latent dimensions, so it can fit item-specific idiosyncrasies and attains higher pooled AUC on unseen items (+3.2 points). SKILLEVAL restricts each item to the dimensions of its assigned skills and summarises its difficulty along those skills with a single discrimination scalar, which limits how finely it can model individual items. The same restriction forces all items that share a skill to constrain the same ability dimension, so each dimension of Θ\Theta is estimated from many related items. As a result, SKILLEVAL estimates where each LLM stands within a domain more accurately: it ranks LLMs better on every benchmark (Figure 2) and yields ability profiles that are stable across retraining (Table 2). The cost lies on the item side: difficulty and discrimination learned from text are less precise, and errors in the Q-matrix propagate directly into predictions. For an evaluation framework, whose purpose is to compare models rather than to predict individual responses, we consider this trade-off favourable.

Applications

Routing LLMs via Ability Profiles

LLM routing selects a model for each incoming query, with the aim of balancing answer accuracy and inference cost ([16, 17]). We first test whether skill profiles improve accuracy by selecting an LLM for each query, rather than always using the strongest model overall. For a test item jj SKILLEVAL estimates the probability Pi(j)=p(Sij=1)P_i(j)=p(S_{ij}=1) that LLM ii answers it correctly (Equation (1)). The router selects the LLM with the highest predicted probability: i^(j)=arg max⁡i∈{1,…,M}Pi(j)\hat{i}(j)=\operatorname*{arg\,max}_{i\in\{1,\ldots,M\}} P_i(j). We evaluate on the Ntest=1,905N_{\mathrm{test}}=1{,}905 held out items of the OOD split with M=3,811M=3{,}811 candidate LLMs. For routers that rank multiple candidates, Acc@k is the fraction of test items for which at least one of the top-kk LLMs answer correctly. Thus, Acc@1 measures the accuracy of the selected LLM, while larger values of kk measure whether a shortlist contains a correct answer.

Baselines. We compare against five baselines. Random routing selects an LLM uniformly at random for each item. The strongest single LLM always uses the model with the highest accuracy on the training items, calme-3.2-instruct-78b.5 The IRT 2PL router ranks LLMs by the single ability value that a fitted 2PL model assigns to each of them ([19]), giving the same ranking for every query. Related work, IRT-Router ([27]), instead uses multidimensional IRT for routing. The KNN router ([3]) retrieves the 10 training items with the most similar text embeddings to item jj. For each candidate LLM, it estimates the probability of a correct answer as the fraction of these retrieved items that the LLM answered correctly. IRTNet routes like SKILLEVAL, selecting the LLM with the highest probability predicted by its own model.

Among the methods in Table 4, SKILLEVAL achieves the highest Acc@1: 65.67%, compared with 61.57% for the strongest single LLM, 61.10% for the IRT 2PL router, 62.05% for KNN, and 63.08% for IRTNet over three runs. Across three retrainings, SKILLEVAL’s Acc@1 is (65.46±0.34)%(65.46 \pm0.34)\% (mean ±\pm standard deviation). IRTNet predicts individual responses more accurately (Table 3) but picks a correct LLM less often, since routing depends only on the top-ranked LLM for each question. For larger shortlists, KNN is competitive: its Acc@3 is similar to that of SKILLEVAL, and it achieves higher Acc@5 and Acc@10. The per-benchmark breakdown (Appendix § 14, Table 10) shows how these results vary across domains. The largest gains over the IRT 2PL router occur on MuSR (+12.5 percentage points) and IFEval (+14.0 percentage points).

MethodAcc@1Acc@3Acc@5Acc@10
Random40.47–––
Strongest single LLM61.57
IRT 2PL61.1077.7481.9485.93
KNN62.0578.0183.1088.98
IRTNet, 3 seeds63.0877.3181.2186.44
(±0.98)(±0.55)(±1.60)(±0.76)
SKILLEVAL (K=100)65.6777.9582.9487.51
SKILLEVAL, 3 seeds65.4677.5582.4387.49
(±0.34)(±1.26)(±0.37)(±0.47)

Table 4. Routing performance on 1,905 held-out items with 3,811 candidate LLMs. All scores are percentages. Rows marked “3 seeds” report the mean over three trainings with different seeds, with the standard deviation in parentheses; IRTNet uses d=1024d=1024, its best size in Table 3. The strongest single LLM and KNN involve no training or randomness, Random is a single random draw, and the other rows are single training runs. Best results are bold.

Table 10: Routing Acc@1 by source benchmark on the held-out test items. Strongest is the train-selected top-1 LLM. SkillEval beats the single-LLM baseline and IRT 2PL on all five benchmarks.

BenchmarknnRandomStrongestIRT 2PLSkillEval
MATH2640.1520.3750.5300.542
BBH11590.4710.7160.6900.735
GPQA2460.2800.3700.4020.407
MuSR1360.3010.5290.4190.544
IFEval1000.4500.8100.6800.820
Overall19050.4050.6160.6110.657

Table 10.

Cost–accuracy trade-off. The preceding experiment selects models for accuracy alone. We now consider inference cost, estimated as the computation per query in billions of floating-point operations (GFLOPs). For each item, the router selects the cheapest LLM whose predicted probability of a correct answer exceeds a threshold τ\tau. If none qualifies, it selects the LLM with the highest predicted probability. Figure 3 compares SKILLEVAL, IRTNet (d=232d=232), and KNN using this same rule, varying τ\tau from 0.10 to 0.95 and expressing cost relative to the strongest single LLM. At τ=0.80\tau=0.80, SKILLEVAL reaches 99.9% of that LLM’s accuracy at 25% of its cost; at τ=0.95\tau=0.95, it exceeds that LLM’s accuracy by 3.7 percentage points at 37% of its cost. KNN is more accurate at low cost but levels off at 98% of the strongest LLM’s accuracy. Its probability estimates take only the eleven values 0/10,…,10/100/10,\ldots,10/10, giving many tied candidates at the highest score. At matched cost budgets in the plotted range, SKILLEVAL achieves higher accuracy than IRTNet.

Figure 3. Accuracy vs. inference cost for three routers using the same threshold rule. Both axes are expressed relative to the strongest single LLM selected on training items. Each point corresponds to one threshold τ\tau.

Skill-level “weak-beats-strong” inversions. A fixed ranking of LLMs cannot adapt to questions on which a model with lower average accuracy performs better. To assess how often such opportunities occur, we examine the observed test responses. On 723 of the 1,905 test items (38.0%), an LLM with at most 13B parameters answers correctly while the strongest single LLM selected on the training items fails. We call these cases “weak-beats-strong” inversions. They occur in every benchmark (BBH: 329 items, MATH: 156, GPQA: 155, MuSR: 64, IFEval: 19) and span 41 skill clusters (Figure 6 in Appendix § 15). These cases illustrate why selecting an LLM for each query can be useful even when one model leads on average.

Figure 6. Distribution of “weak-beats-strong” items by source benchmark. 38% of held-out items (n=723/1,905n=723/1{,}905) admit at least one ≤13\le13B LLM that out-performs the global strongest model, spanning 41 skill clusters: BBH 329, MATH 156, GPQA 155, MuSR 64, IFEval 19.

Profiling Unseen LLMs

When a new LLM becomes available, we want to estimate its skill profile from a small set of its answers without retraining the entire diagnostic network. We train the text-conditioned NCDM using only training-item responses from 3,048 LLMs. We then freeze the network, including the item difficulty and discrimination parameters, and estimate only the KK-dimensional profile of each new LLM from its answers to NN items drawn from the training-item pool. This is ability estimation with fixed item parameters, also called scoring in psychometrics [26]. We evaluate this procedure on 200 held-out LLMs, using separate test items whose responses never enter profile estimation.

Starting point and related skills. We fit a multivariate Gaussian to the raw (pre-sigmoid) profiles of the 3,048 training LLMs and use it as a prior. Starting from the prior mean, we update the new LLM’s profile with L-BFGS, using Equation (1) with a Gaussian prior regularizer. Because the Gaussian captures correlations between skills, an answer also shifts skills that are correlated with the item’s assigned skills. As more answers are observed, the data term increasingly outweighs the fixed prior (Appendix § 13).

Choosing the questions. We compare uniform random selection with adaptive selection. Computerized adaptive testing (CAT) chooses each next question based on the answers given so far; CD-CAT applies this idea to cognitive diagnosis models [4]. Starting from the prior mean, we use the current profile to predict the probability pjp_j of a correct response to each unused item jj and select:

j∗=arg⁡max⁡jpj(1−pj)∥qj∥1.(3)j^* = \arg\max_j p_j(1-p_j)\lVert\mathbf{q}_j\rVert_1. \tag*{(3)}

The uncertainty term pj(1−pj)p_j(1-p_j) is largest at pj=0.5p_j=0.5. The skill count ∥qj∥1\lVert q_j\rVert_1 favours items assigned to several skills. After observing the answer, we update the profile using all answers so far and repeat until NN items have been answered. Both selection methods use the same final profile-estimation procedure.

Measures. We measure what the answers reveal about each LLM in two ways. For each benchmark, we average the predicted probabilities over its test items and compare this predicted accuracy with the observed accuracy on those items. The benchmark-score error is their absolute difference, averaged over the five benchmarks and the LLMs. Profile agreement compares the relative skill estimates from NN answers with a reference profile estimated from all 7,618 pool answers without the prior. We compute Pearson correlation across LLMs for each skill, then average over skills. A simple-count baseline uses the same randomly selected items to estimate each benchmark score as the fraction answered correctly. If no items from a benchmark have been answered, it uses the prediction from the zero-answer profile.

Results. With no answers, the starting profile predicts benchmark scores with a mean error of 0.095 (Figure 4). With 100 random answers, SKILLEVAL lowers the error to 0.061, and with 500 answers to 0.049, approaching the 0.042 obtained from all 7,618 answers. Adaptive selection reaches similar errors with fewer answers: its mean error after 10 answers is 0.068, comparable to 50 random answers, and after 20 answers it is 0.061, comparable to 100 random answers. It also yields higher profile agreement: 0.53 after 100 answers, versus 0.45 for random selection. Adaptive selection yields lower benchmark-score error and higher profile agreement than random selection at every tested budget from 5 to 500 answers (Appendix § 13). At the nonzero tested budgets up to 100 answers, the simple count has higher error than the zero-answer prediction; with 500 answers, its error is 0.052.

Figure 4. Profiling 200 held-out LLMs from NN answers. (a) Mean absolute benchmark-score error across LLMs and the five benchmarks; the dotted line uses all 7,618 pool answers with the same estimator. (b) Pearson correlation with profiles estimated from all answers without the prior, computed across LLMs for each skill and averaged over skills.

Diagnostic Analyses

The Q-matrix and the learned mastery profiles support complementary analyses of benchmark coverage and model behavior. We examine skill coverage across benchmarks and differences between base and instruction-tuned models.

Skill-level differences between base and instruction-tuned models. We compare 81 matched pairs, each comprising a base model and its instruction-tuned version. Averaged across pairs, estimated mastery is more than 0.01 higher in instruction-tuned models on 38 skills and more than 0.01 lower on 36 skills. Separately, paired Wilcoxon signed-rank tests identify 14 skills with significant differences after Bonferroni correction, which divides the significance level by the 100 tested skills (Figure 5). Eleven of these differences are positive, mostly on instruction-following skills, and three are negative, with magnitudes of at most 0.06. When skills are grouped by the benchmark contributing most of their items, IFEval has the largest mean difference (Δ=+0.126\Delta=+0.126, p<10−10p<10^{-10}); Appendix § 20 reports the differences by model family. Statistical significance does not establish a large effect, and these observational comparisons do not isolate the causal effect of instruction tuning.

Figure 5. Estimated mastery differences between base and instruction-tuned models. (a) Benchmark means, assigning each skill to its primary benchmark by item count; stars indicate unadjusted pp-values (∗p<0.05\ast p<0.05, ∗∗p<0.01\ast\ast p<0.01, ∗∗∗p<0.001\ast\ast\ast p<0.001). (b) The 14/100 skills significant after Bonferroni correction (α=5×10−4\alpha=5\times10^{-4}), sorted by mean difference and coloured by primary benchmark; names are abbreviated. Error bars: ±1\pm1 SEM across 81 matched model pairs. Omitted skills are not necessarily unchanged.

Skill coverage across benchmarks. To identify distinct and overlapping skill coverage across benchmarks, we count a skill as covered when at least one item in a benchmark is assigned to it in the Q-matrix. Under this criterion, 30 of the 100 skills occur in only one benchmark, including 13 unique to IFEval, 9 to GPQA and 6 to MATH (Figure 7 in Appendix § 19). MATH and IFEval together cover 87 skills; adding GPQA increases this to 98. BBH adds one further skill to this combined coverage despite contributing 5,761 items. Shared coverage suggests potential redundancy, but benchmarks can still differ in difficulty, item diversity and how well they distinguish between models. The concentration of unique skills in IFEval is consistent with the transfer result in Appendix § 16: withholding IFEval from training removes all training items for these skills, and routing performance on IFEval deteriorates.

Figure 7. Skill coverage per benchmark. Dark segments show skills unique to that benchmark; light segments show shared skills. IFEval contributes 13 unique skills, while BBH has only 1. MATH + IFEval together cover 87/100 skills.

Conclusion

SkillEval applies a Neural Cognitive Diagnostic Model to 3,811 LLMs across 9,523 items and five benchmarks, producing interpretable mastery profiles over K=100K=100 named skills. Unlike IRTNet’s latent dimensions, these profiles are stable across retraining: each skill’s estimates correlate at 0.96 on average between runs. Skill-conditioned routing reaches Acc@1=65.67% (vs. 61.57% for the strongest single LLM, 62.05% for KNN, 63.08% for IRTNet over three runs and 61.10% for the IRT 2PL router) and matches the strongest LLM at 4×4\times lower inference cost. SkillEval also profiles previously unseen LLMs without retraining the shared network: 20 adaptively chosen answers give a mean benchmark-score error comparable to 100 randomly chosen answers. Skill profiles recover benchmark-level scores at Pearson r=0.989r=0.989. The framework surfaces diagnostic findings invisible to aggregate scores: differences between base and instruction-tuned models that are statistically significant after Bonferroni correction on 14/100 skills, and overlapping skill coverage across benchmarks. An exploratory analysis in Appendix § 18 examines asymmetric co-mastery: LLMs mastering one skill often also master another, while the reverse is less common. Testing how these patterns generalise across LLM populations is a direction for future work. Limitations are in Appendix § 21.

Related Work

LLM Evaluation and Routing. Standard LLM evaluation aggregates item-level correctness into a single benchmark score [11, 15, 5], creating leaderboards whose rankings are influenced as much by evaluation protocol (prompt format, shot count, decoding parameters) as by true model capability [1]. Platforms such as the Open LLM Leaderboard and RouterBench Hu et al. [16] release per-item correctness matrices for thousands of LLMs, enabling but not themselves performing fine-grained analysis. Routing systems match queries to the cheapest capable model, typically using latent query-model embeddings without explicit semantic skills. Closest in spirit to our work, CAIMIRA Gor et al. [12] applies IRT to factual QA, and Didolkar et al. [10] use LLM-generated skill labels to probe mathematical reasoning. We build on the latter by scaling skill extraction to five benchmarks and fitting a full CDM over the extracted skills.

Cognitive Diagnostic Models. Cognitive Diagnostic Models (CDMs) originate in educational psychometrics and estimate latent skill mastery from observed item responses, extending Item Response Theory (IRT) by modelling proficiency as a vector over named skills rather than a scalar or a continuous embedding. Classical CDMs such as DINA de la Torre [9] assume conjunctive skill requirements (an item is correct only if all required skills are mastered) and binary mastery. The Neural CDM (NCDM) Wang et al. (2020) relaxes these assumptions with neural interaction functions while preserving mastery-to-score monotonicity via non-negative weight constraints, and is generalised in NeuralCD Wang et al. [31]; reference implementations are collected in the EduCDM library bigdata ustc (2021). CDMs have been widely deployed for human assessment over hundreds of skills and tens of thousands of examinees, making them a natural fit for LLM evaluation where response matrices are now orders of magnitude larger.

Psychometric Methods for LLMs. Recent work applies psychometric models to LLMs at scale. IRT-Router Song et al. [27] uses multidimensional IRT with a query-cluster relevance vector for routing. IRTNet Chen et al. [3] jointly learns dd-dimensional LLM ability embeddings, query discrimination, and query difficulty via a Mixture-of-Experts architecture, evaluated on 112 LLMs and 10 benchmarks. Metabench Kipnis et al. [18] compresses six public benchmarks to under 3% of their combined items using 2-PL IRT and Fisher-information filtering on ∼5,000\sim5{,}000 LLMs, showing that a single latent factor explains 79% of benchmark variance. TinyBenchmarks Maia Polo et al. [20] similarly uses IRT-based item selection to reduce evaluation cost. All of these approaches treat LLM ability as a low-dimensional latent embedding without a named skill structure. SKILLEVAL commits to an a priori Q-matrix over K=100K=100 semantically labeled skills derived from LLM metacognition and SBERT clustering, enabling skill-level analyses of co-mastery patterns, skill interaction within items, and base versus instruction-tuned differences that are not expressible over a continuous latent embedding. We scale to 3,811 LLMs (34×34 \times IRTNet, comparable to metabench’s ∼5,000\sim5{,}000) across five benchmarks.

Notation

Table Table 5 summarises the notation used throughout the paper.

SymbolMeaning
MMNumber of LLMs (M=3,811M=3{,}811)
NNNumber of items (N=9,523N=9{,}523); also the number of answers used to profile a new LLM
KKNumber of skill clusters (K=100K=100 default)
i∈{1,…,M}i \in\{1,\ldots,M\}LLM index
j∈{1,…,N}j \in\{1,\ldots,N\}Item index
k∈{1,…,K}k \in\{1,\ldots,K\}Skill index
S∈{0,1}M×NS \in\{0,1\}^{M \times N}Binary response matrix; Sij=1S_{ij}=1 iff LLM ii answers item jj correctly
Q∈{0,1}N×KQ \in\{0,1\}^{N \times K}Q-matrix; qjk=1q_{jk}=1 iff item jj tests skill kk
θi∈[0,1]K\boldsymbol{\theta}_i \in[0,1]^KSkill mastery profile of LLM ii
dj∈[0,1]Kd_j \in[0,1]^KItem difficulty per skill
αj∈[0,1]\alpha_j \in[0,1]Item discrimination
ej∈R768e_j \in\mathbb{R}^{768}Frozen SBERT text embedding of item jj
σ(⋅)\sigma(\cdot)Sigmoid function

Table 5. Notation summary.

Skill Cluster Examples

Table Table 6 shows four representative skill clusters with example items. Each cluster groups items that share a common cognitive requirement despite coming from different benchmarks or subtasks. For instance, the Quadratic Formula cluster contains 451 items dominated by MATH algebra problems but also includes 20 GPQA items that happen to require algebraic manipulation.

Table 6: Representative skill clusters with example items. Each cluster aggregates items from multiple benchmarks that share a common skill requirement.

Skill clusterItemsExample items
Quadratic Formula (MATH-primary, 429/451)451“Find the domain of f(x)=x2−16−3f(x)=\sqrt{\sqrt{x^2-16}-3}” “If −6≤a≤−2-6\leq a\leq-2 and 3≤b≤53\leq b\leq5, find the greatest value of a+ba\frac{a+b}{a}”
Truthfulness of Nested Statements (BBH-primary, 1836/1880)1,880“Then he tells Billy: ‘Be sure that you do not come in at all tomorrow morning’ ” (causal judgement) “Teressa lies. Willian says Teressa lies. Jamey says Willian lies. Ryan says Jamey lies. Kandi says Ryan lies. Does Kandi tell the truth?” (web of lies)
Structural Isomers in Organic Compounds (GPQA-primary, 442/488)488“Identify product D [from NMR spectrum]: (A) 2-bromonaphthalene (B) 2-bromo-1,2,3,4-tetrahydronaphthalene. . . ”
Word Count Constraints (IFEval-primary, 414/418)418“Write a social media post about the hajj pilgrimage. Use all lowercase and include the word ‘story’ at least twice”

Table 6.

Divergent model performance. Table 7 illustrates how skill profiles explain item-level performance differences. On a BBH book-trading problem that requires both sequential event tracking and object trade tracking (conjunctive), models mastering both skills solve it 74% of the time while models mastering only one skill drop to 22–33%. For example, Llama-3.1-Tulu-3-70B (masters both) answers correctly, while Llama-3.2-Korean-5B (masters only event tracking) fails. On a compensatory IFEval item requiring word count constraints or Urdu dialogue adaptation, mastering either skill suffices: accuracy is 97% with word count skill alone and 42% with Urdu skill alone, both far above the 12% baseline.

Table 7: Accuracy by mastery group on two held-out items. The BBH item is conjunctive (needing both skills). The IFEval item is compensatory (either skill suffices).

BothSkill 1Skill 2Neither
BBH (conjunctive)0.740.330.220.10
IFEval (compensatory)0.930.970.420.12

Table 7.

Q-Matrix Construction Details

Skill extraction prompt. For each item we prompt GPT-4o-mini at temperature 0 with the benchmark, the subtask and the item text, and ask for two to four skills, each with a label, the cognitive process and the content domain, together with the single most important skill, which we call the primary skill. Labels on the list of generic labels in the prompt are removed after extraction. The system prompt and the user template are reproduced below; long lines are wrapped for display. We extract the full skill list rather than only the primary skill, yielding a mean of 2.9 skills per item and 13,276 unique skill strings across 9,523 items. The primary skill alone is too coarse: it covers only the headline category and produces Q-matrix rows with mean row-sum 1.0, which under-constrains the diagnostic model.

System prompt.

You are a psychometrician designing a Q-matrix for cognitive diagnostic
    assessment of AI systems. Your task: identify the specific cognitive
    skills each test item measures.

RULES:
1. Each skill must be a compound phrase (5–10 words):
[specific_cognitive_process] + [specific_content_domain]
2. Extract 2-4 skills per item. At least one must be domain-specific.
3. FORBIDDEN generic labels (will be rejected):
- "reading comprehension", "logical reasoning", "critical thinking"
- "mathematical reasoning", "problem solving", "analytical thinking"
- "scientific knowledge", "attention to detail", "numerical reasoning"
- Any label that would apply to >15% of items in the benchmark
4. Specificity test: would fewer than 100 items in a 10,000-item test share this exact skill? If not, decompose further.
5. Use lowercase snake_case labels.

GOOD examples by benchmark type:

MATH: "factoring_higher_degree_polynomials_over_integers",
"applying_pigeonhole_principle_to_combinatorial_bounds",
"computing_modular_arithmetic_in_residue_classes"

BBH: "tracing_boolean_operator_precedence_in_nested_expressions",
"identifying_causal_direction_from_correlational_evidence",
"tracking_object_positions_through_spatial_transformations"

GPQA: "applying_gauss_law_to_cylindrical_charge_distributions",
"predicting_reaction_products_via_retrosynthetic_analysis",
"interpreting_phylogenetic_trees_from_molecular_sequence_data"

MuSR: "integrating_alibis_and_motives_to_identify_suspects",
"tracking_object_locations_across_sequential_room_transfers",
"resolving_team_allocation_under_mutual_exclusion_constraints"

IFEval: "enforcing_exact_word_count_in_structured_output",
"maintaining_consistent_formatting_across_nested_lists",
"embedding_required_keywords_while_preserving_coherence"

BAD examples (too generic, rejected):
- "problem solving" -> decompose to
  "applying_substitution_to_solve_nonlinear_systems"
- "reading comprehension" -> decompose to
  "extracting_temporal_ordering_from_narrative_clues"
- "scientific knowledge" -> decompose to
  "applying_conservation_of_angular_momentum_to_rotating_bodies"

User template. The benchmark name, subtask and item text replace the placeholders in braces.

Benchmark: {benchmark} | Subtask: {subtask}

Item:
{question_text}

Respond in JSON:
{
  "skills": [
    {
      "label": "5-10 word compound skill in snake_case",
      "process": "the specific cognitive operation",
      "domain": "the specific content area"
    }
  ],
  "primary_skill": "label of the single most important skill"
}

Embedding and clustering. The 13,276 unique skill strings are embedded with the all-mpnet-base-v2 sentence transformer into R768\mathbb{R}^{768} and L2L_2-normalised. We compare three clustering families: HDBSCAN after UMAP reduction to 5D, KK-Means, and HAC with cosine distance and average linkage. We select HAC because it yields the highest downstream test AUC at every KK we tried, despite a lower silhouette score (0.053 at K=50K=50 vs. 0.144 for KK-Means and 0.160 for HDBSCAN at K=281K=281). Intrinsic clustering quality is therefore not a reliable proxy for diagnostic utility.

Coverage fix. Our first pipeline iteration assigned each item to a single cluster via the primary skill only, which left 43% of clusters with fewer than 10 items and 10 empty columns in QQ at K=100K=100. Switching to the multi-skill assignment described above removed all empty columns, reduced the fraction of under-populated clusters to 36% and raised the mean column-sum from 95 to 139 items per skill.

Reference-answer ablation. Our default skill extraction does not use reference solutions. To test whether showing the extractor a reference solution changes the result, we ran two new extractions for the 2,470 MATH and GPQA items that have a reference solution, one from the question and one from the question plus the solution. Both use the same prompt, the same extraction model and the same question text, and they are separate from the extraction that produced the QQ-matrix. The solution changes which skills are named: the token-level Jaccard overlap between the two label sets is 0.37, against 0.73 between two independent question-only runs. A blind judge (GPT-4.1-mini) compared both label sets on 600 items, each in both presentation orders. We retain the 357 items with a consistent, non-tied preference in both orders; the remaining judgments do not enter the percentages below. Among the retained items the judge preferred the solution-aware labels in 49.6% and the question-only labels in 50.4%, so it showed no clear aggregate preference for either extraction condition. Within the retained subset the preference differs by benchmark: question-only labels are preferred on MATH (59%) and solution-aware labels on GPQA (61%). We keep question-only extraction, which also applies to item banks without reference solutions.

Training Details

The text-conditioned NCDM is trained end-to-end with binary cross-entropy loss on the 80% training split of the response matrix, which covers 7,618 training items and all 3,811 LLMs. The skill mastery embedding θi\theta_i, the item difficulty and discrimination heads Wk∈RK×768W_k \in\mathbb{R}^{K \times768} and we∈R768w_e \in\mathbb{R}^{768}, and the PosLinear layers all use Xavier normal initialisation. The PosLinear prediction network has hidden widths K→512→256→1K \to512 \to256 \to1; non-negativity is enforced by W+=max⁡(W,0)W^+ = \max(W, 0) at each forward pass. We optimise with Adam at a learning rate 0.002 for 15 epochs, with dropout 0.5 after each hidden layer. The main model and the KK-sweep models use batch size 1024. The five-seed runs and the interaction-wise model use batch size 64. Model selection uses validation AUC on a held-out triplet split of 10% of the training pairs, keeping the weights of the best-AUC epoch. Training the main model takes about three hours on a single NVIDIA RTX 2080 Ti. The default seed is 42. The multi-seed results in § 3.2 use seeds 42–46.

Full K-Sweep Results

Table 8 reports a sweep over K∈{50,100,150,200,300,400,500}K \in\{50, 100, 150, 200, 300, 400, 500\}. Test AUC is essentially flat (0.716–0.721), confirming that the diagnostic fit is robust to clustering granularity rather than dependent on a particular elbow. Routing Acc@1 improves modestly from 0.640 at K=50K=50 to 0.676 at K=500K=500, and is 0.657 at K=100K=100. We adopt K=100K=100 as the default because it preserves interpretable, named clusters while using about four times fewer parameters than K=500K=500.

KKTest AUCAcc@1Acc@3Acc@5Acc@10
500.7200.6400.7830.8290.877
1000.7170.6570.7800.8290.875
1500.7190.6540.7780.8310.887
2000.7160.6610.7970.8460.885
3000.7190.6720.7940.8430.882
4000.7210.6690.8020.8410.887
5000.7200.6760.8040.8470.896

Table 8. Effect of the number of skill clusters KK on diagnostic AUC and routing Acc@kk. Diagnostic fit is essentially constant across KK, and routing accuracy rises modestly with KK. Each row is a single training run with batch size 1024; the KK=100 row is the main model, so its test AUC differs from the five-seed mean in Table 3.

Profiling Unseen LLMs: Details

Network. The frozen network is trained with the recipe of the main model in Appendix § 11 (seed 42, batch size 1024, 15 epochs), using only the training-item responses of the 3,048 training LLMs.

Estimation. Each profile is represented by unconstrained raw parameters r\mathbf{r}, with θ=sigmoid⁡(r)\boldsymbol{\theta} = \operatorname{sigmoid}(\mathbf{r}). The prior on r\mathbf{r} is a multivariate normal distribution. Its mean μ\boldsymbol{\mu} is the mean raw profile of the 3,048 training LLMs; its covariance Σ\boldsymbol{\Sigma} is either empirical covariance shrunk by 10% towards its diagonal.

Let AA be the set of answered items, yjy_j indicate whether item jj was answered correctly, and pjp_j be the predicted probability of correctness from Equation (1). We estimate rr by minimising

−∑j∈A[yjlog⁡pj+(1−yj)log⁡(1−pj)]+12(r−μ)TΣ−1(r−μ).(4)-\sum_{j \in A} \left[y_j \log p_j + (1-y_j)\log(1-p_j)\right] + \frac{1}{2}(r-\mu)^{\mathsf{T}}\Sigma^{-1}(r-\mu). \tag*{(4)}

The first term measures fit to the observed answers; the second penalises departures from profiles supported by the prior. We optimise with L-BFGS, starting from μ\mu. With no answers, the raw estimate is μ\mu and the skill profile is its sigmoid transform. In a check on the first 20 LLMs using all 7,618 pool answers, fits with and without the prior have a mean absolute difference of 0.005 in their test-item probabilities. During adaptive selection, each update starts from the previous raw estimate. At each evaluated budget, we refit from μ\mu, using the same procedure for random and adaptive selection.

Evaluation. We use the first 200 of the 763 held-out LLMs. Random-selection metrics are averaged over three seeds; adaptive selection is deterministic for a given LLM and network. Both selection methods are evaluated at the same answer budgets up to 500. Table 9 lists the values shown in Figure 4. At each tested budget from 5 to 500 answers, paired 95% bootstrap intervals over LLMs favour adaptive selection on both measures: the adaptive-minus-random difference is below zero for benchmark-score error and above zero for profile agreement.

Table 9: Profiling 200 held-out LLMs from NN answers. Error: mean absolute benchmark-score error over LLMs and benchmarks. Agreement: mean per-skill Pearson correlation with profiles estimated from all pool answers without the prior. AUC: mean per-LLM test-item AUC. Both selection methods are evaluated up to 500 answers; random-selection results are averaged over three seeds. The all-answers error and AUC use the estimator with the prior.

NNErrorAgreementAUC
RandomAdaptiveCountRandomAdaptiveRandomAdaptive
00.0950.0950.0950.0000.0000.6820.682
50.0870.0790.2020.2070.3010.6820.684
100.0820.0680.2180.2520.3510.6820.686
200.0750.0610.2170.3290.4170.6820.687
500.0680.0560.1540.3960.4700.6840.688
1000.0610.0530.1070.4510.5320.6860.689
2000.0550.0480.0750.5020.5950.6880.690
5000.0490.0460.0520.5700.6550.6900.691
All 7,6180.0420.693

Table 9.

Reference profiles and robustness. Profile agreement uses a reference estimated from all pool answers without the prior, avoiding a shared prior term in the reference and the estimates being evaluated. This reference is itself an estimated profile. We assign zero agreement to a skill whose estimates are constant across LLMs; this includes the zero-answer setting, where Pearson correlation is undefined. Using a reference estimated with the prior, or restricting the measure to the 50 skills with at least 20 pool items, changes the agreement values but preserves the ordering of the compared methods.

Per-LLM AUC. Test-item AUC changes little in this setting, from 0.68 with no answers to 0.69 with all answers. The zero-answer prediction uses the same profile for every LLM, so reaching this AUC does not require estimating differences between their skills. We therefore use benchmark-score error and profile agreement to assess what additional answers reveal about each LLM.

Per-Benchmark Routing

Table 10 expands the per-benchmark routing breakdown summarised in § 4.1. SkillEval beats the strongest-model baseline and IRT 2PL on every benchmark, with the largest absolute gains over IRT on MuSR (+12.5 pts) and IFEval (+14.0 pts).

Weak-Beats-Strong Inversions

Figure 6 expands the per-benchmark distribution of “weak-beats-strong” inversions summarised in § 4.1. Across the 1,905 held-out test items, 723 (38%) admit at least one LLM with ≤13B\le13\mathrm{B} parameters that out-performs the global strongest model, spanning 41 distinct skill clusters. The phenomenon is broad rather than driven by a single benchmark: BBH contributes 329, MATH 156, GPQA 155, MuSR 64, IFEval 19. This is the headroom that skill-conditional routing exploits and scalar-ability routers cannot reach.

Cross-Benchmark Transfer

To probe whether the learned skill space generalises across benchmark families, we hold out two entire benchmarks (MuSR and IFEval) and train SkillEval on the remaining three (MATH, BBH, GPQA), using the same K=100K=100 HAC clusters, M=3,811M=3{,}811 LLMs, and 15-epoch Adam schedule as the main run. The training set covers 8,231 items; the held-out evaluation set covers 1,292 items.

Aggregate diagnostic fit on the held-out items collapses to AUC 0.469 (test accuracy 0.460), indicating that on combined unseen-benchmark items the model is no better than chance. The picture sharpens when split by benchmark (Table 11). On MuSR, where the multi-step reasoning skills overlap with BBH, transfer is intact: SkillEval reaches Acc@1 0.470, matching the strongest-model baseline (0.467) but below the full-training reference (0.544). On IFEval, however, SkillEval drops to 0.410, below random (0.437) and 32 points below the strongest model (0.731).

Held-outnnRandomStrongestSkillEval (transfer)Reference
MuSR7560.4150.4670.4700.544
IFEval5360.4370.7310.4100.820
Combined12920.4000.5770.445—

Table 11. Cross-benchmark transfer with MuSR and IFEval held out entirely. “Reference” is the full-training Acc@1 from § 4.1. Transfer succeeds on MuSR (overlap with BBH multi-step reasoning) but fails on IFEval, where 14 of the 62 assigned skills receive no training items.

Skill coverage in the transfer experiment. With MuSR and IFEval held out, 15 of the 100 skills have no training item. These skills were defined during Q-matrix construction on the full item bank, but their mastery parameters receive no response-based training in this experiment. IFEval items are assigned to 62 skills, 14 of which are among these 15, and 23% of IFEval items involve at least one of them; for MuSR the share is below 1%. This coverage gap is consistent with the larger failure on IFEval, but it does not explain it fully, since most IFEval items involve only skills that were trained. We therefore report the IFEval result as a limit of cross-benchmark transfer: deployments should include the target benchmark family in the training mixture rather than rely on zero-shot transfer.

Controls for the Validity of Θ\Theta

Section 3.3 reports the correlation between Θ\Theta and held-out per-skill accuracy. Such a correlation could arise if Θ\Theta only reflected how strong an LLM is overall. For each skill kk we therefore compute the partial correlation, across LLMs, between the mastery θk\theta_k and the held-out accuracy on skill kk after regressing out the LLM’s overall training accuracy. Over the 41 skills with at least ten held-out items, the mean partial correlation is 0.35, averaged over five seeds. For a descriptive null comparison, we fix the seed-42 mastery estimates and permute held-out item-to-skill assignments within each benchmark, preserving each skill’s held-out size and benchmark composition; the mean partial correlation over 20 such permutations is 0.22. Controlling instead for the LLM’s five per-benchmark training accuracies gives 0.135, against a placebo mean of −0.02. Controlling for the finer profile of per-subtask training accuracies (38 subtasks) reduces the mean partial correlation to 0.021, indicating substantial overlap with subtask structure. These comparisons indicate that Θ\Theta carries information beyond overall and benchmark-level accuracy; they do not separate it from subtask-level accuracy.

Empirical Co-Mastery Patterns

Mastery and link rule. This exploratory analysis asks whether LLMs that master one skill also tend to master another, even when the reverse is less common. We find such asymmetric co-mastery patterns across most skills, with similar patterns visible in the observed responses. In Equation (1), estimated mastery θ\theta is interpreted relative to item difficulty, so its value alone does not specify expected accuracy on a skill’s items. For this analysis, we classify an LLM as mastering a skill when its predicted probability of a correct answer, averaged over the items assigned to that skill, exceeds 0.5, that is, when the model expects it to answer more than half of those items correctly.

For two skills AA and BB, let P(A∣B)P(A \mid B) denote the share of LLMs mastering AA among those mastering BB. For example, 52 of the 55 LLMs mastering combinatorics of dice rolls also master divisibility rules for factorials, so P(divisibility∣dice)=52/55≈0.95P(\mathrm{divisibility} \mid\mathrm{dice}) = 52/55 \approx0.95. In the reverse direction, only 52 of the 167 LLMs mastering divisibility rules also master dice-roll combinatorics, so P(dice∣divisibility)=52/167≈0.31P(\mathrm{dice} \mid\mathrm{divisibility}) = 52/167 \approx0.31. Almost every LLM that masters the dice-roll skill thus also masters the divisibility skill, whereas most LLMs that master the divisibility skill do not master the dice-roll skill. We represent this pattern by a link from divisibility rules to dice rolls.

We draw a link from AA to BB when two conditions hold. First, we require P(A∣B)−P(B∣A)>0.3P(A \mid B) - P(B \mid A) > 0.3, capturing an asymmetry like the one above. Second, we require a strong association. The odds of an event are its probability divided by the probability that it does not occur, so a probability of 0.75 corresponds to odds of 3. The odds ratio compares the odds of mastering AA among LLMs that master BB with those among LLMs that do not. A ratio of 1 indicates no association; a ratio above 1 means mastery of AA is more likely among LLMs mastering BB. We require an estimated odds ratio above 10. Without this condition, unrelated skills could be linked simply because one is mastered more frequently than the other.

Odds ratio. For two skills AA and BB, let nABn_{AB} be the number of LLMs that master both, nABˉn_{A\bar{B}} the number that master AA but not BB, nAˉBn_{\bar{A}B} the number that master BB but not AA, and nAˉBˉn_{\bar{A}\bar{B}} the number that master neither. The Haldane correction adds 0.5 to each count, which keeps the odds ratio finite when a count is zero:

OR⁡(A,B)=(nAB+0.5)(nAˉBˉ+0.5)(nABˉ+0.5)(nAˉB+0.5).\operatorname{OR}(A,B) = \frac{(n_{AB}+0.5)(n_{\bar{A}\bar{B}}+0.5)}{(n_{A\bar{B}}+0.5)(n_{\bar{A}B}+0.5)}.

Co-mastery graph. Applying this rule to every pair of skills gives a co-mastery graph with skills as nodes. The graph orders two skills when following links in their direction leads from one to the other. To simplify the graph, we remove a direct link when the original graph also connects the same pair through one intermediate skill. This leaves 304 links connecting 86 of the 100 skills; some retained links may still be implied by longer paths. Of these 86 skills, 18 have no incoming link and 13 have no outgoing link. Every link points from a more commonly mastered skill to a less commonly mastered one, because P(A∣B)P(A \mid B) can exceed P(B∣A)P(B \mid A) only when more LLMs master AA than BB. Its direction therefore follows from the construction and does not show that AA must be learned before BB. Repeating the construction and link removal with a mastery cut-off of 0.4 instead of 0.5 gives 391 links connecting 89 skills, and with 0.6 it gives 209 links connecting 68 skills. Only 29 direct links appear at all three cut-offs, so individual links are sensitive to the cut-off.

Checks. Two checks support these patterns. First, we construct a second graph using the same LLMs, items and skill assignments, but classify an LLM as mastering a skill if it answers more than half of that skill’s items correctly. This graph orders 1,325 pairs. The co-mastery graph orders 983 in the same direction and one in the opposite direction, leaving the rest unordered. Wherever both graphs order a pair, they agree in all but one case. Second, we independently shuffle each skill’s mastery labels across LLMs, preserving the share mastering each skill while disrupting the observed associations. Before link removal, the rule finds at most 51 links in any of 200 shuffles, compared with 1,031 in the original profiles. This supports associations beyond individual skill frequencies. Across the 304 retained links, the median of P(A∣B)P(A \mid B) is 0.87 and the median of P(B∣A)P(B \mid A) is 0.26. A benchmark average does not expose these relationships between named skills. For dataset design, the graph can suggest skills to assess within groups of LLMs sharing a strength, as dice-roll combinatorics distinguishes among the 167 LLMs that master divisibility rules.

Cross-family overlap. We also apply the link rule separately to each model family with at least 100 LLMs: Llama, Qwen, Mistral, Gemma and Phi. We compare the resulting link sets before removing any links. For two families, the Jaccard index is the number of links shared by both divided by the number found in at least one. Across the ten family pairs, it ranges from 0.09 to 0.31. The detected links are therefore only partly shared across the sampled families; whether these patterns persist in other LLM populations remains to be tested.

Benchmark Redundancy Details

Figure 7 shows skill coverage per benchmark: dark segments mark skills unique to that benchmark, light segments mark skills shared with at least one other benchmark. IFEval contributes the most unique skills (13 of 62), while BBH contributes only 1 unique skill of 42. MATH and IFEval together cover 87 of 100 curated skills.

Figure 8 shows the pairwise coverage matrix: cell (i,j)(i, j) reports the fraction of benchmark ii’s skills also covered by benchmark jj. MuSR is the most redundant row: 82% of its skills are covered by BBH, and 82% by IFEval. IFEval is the most independent row, with the lowest average overlap with the other benchmarks.

Figure 8. Pairwise skill coverage matrix. Cell (i,j)(i, j): fraction of row benchmark’s skills also covered by column benchmark. Red = low overlap (distinct), blue = high overlap (redundant). MuSR’s skills are 82% covered by BBH and IFEval.

Differences Between Base and Instruction-Tuned Models: Details

Figure 9 reports estimated mastery differences by model family and benchmark, with each skill assigned to the benchmark that contributes most of its items. Mean differences over IFEval skills are positive for every evaluated family. The MuSR column is uniformly negative, and the MATH/GPQA columns are near-zero or slightly negative for most families. Llama shows the broadest gains; Qwen is essentially flat.

Figure 9. Mean estimated mastery difference (instruction-tuned−-base) per model family and benchmark. Blue = improved, red = degraded. The IFEval column is positive for every evaluated family; the MuSR column is negative for every evaluated family.

Limitations

Scope of the new-LLM experiments. The new-LLM results use 200 of the 763 held-out LLMs and one trained network. We evaluate one adaptive selection rule at answer budgets up to 500. The prior is estimated from profiles produced by that network, while observed scores for the smaller benchmarks are based on relatively few test items. The results describe the evaluated LLMs, network and item pool; they do not establish an advantage for adaptive selection in every setting.

Co-mastery vs. causal prerequisites. The co-mastery graph in Appendix § 18 is constructed from statistical co-mastery (OR >10> 10), not causal dependency. The observed structure likely reflects a mix of skill-difficulty ordering and shared training-data signal rather than experimentally verifiable cognitive prerequisites. Establishing causal prerequisite relations would require intervention experiments (skill-specific fine-tuning) beyond our scope; we therefore phrase all structural claims in terms of co-mastery.

Q-matrix inter-annotator agreement. The Q-matrix is constructed semi-automatically via LLM-proposed skill phrases clustered by SBERT and HAC. Cluster choice is validated by downstream NCDM AUC rather than by human adjudication. A human validation of a sample of skill–question assignments is reported in § 3.1. It tests whether an assigned skill is needed, not whether needed skills are missing. Items near cluster boundaries should be interpreted with additional caution. We sweep the number of skill clusters KK and report sensitivity to KK, but we do not separately vary the clustering algorithm (HAC), the skill-extraction prompt, or the sentence-embedding source (SBERT); we leave a joint pipeline-robustness study to future work.

Evaluation scope. Our item-wise OOD setting holds out all responses to the test items while retaining the same benchmarks and LLM population. We treat the Q-matrix as part of the test design: the skill taxonomy and Q-matrix are constructed before this split, using question text and benchmark/subtask metadata from the complete item bank, including test items. Skill extraction and clustering use item content rather than the evaluated LLMs’ responses. However, the clustering method and the number of clusters were chosen with reference to downstream performance (Appendices § 10 and § 12). Item difficulty and discrimination for test items are predicted from their text by projections trained on training items only. The evaluation is therefore transductive with respect to the skill representation, while test-item responses are excluded from parameter fitting: it measures prediction of held-out responses within a fixed, item-bank-derived taxonomy. Generalization remains limited. IRTNet outperforms SKILLEVAL on held-out-item prediction (Table 3). When MuSR and IFEval are jointly excluded from response-based training, observed routing accuracy on MuSR is similar to that of the single-model baseline selected by training accuracy (0.470 versus 0.467), whereas IFEval routing falls below the random-routing baseline (0.410 versus 0.437; Appendix § 16). These results limit the scope of our routing and profiling claims; reliable transfer to new benchmark domains or skill families remains to be established.

Benchmark coverage. SkillEval covers five English-language benchmarks (MATH, BBH, GPQA, MuSR, IFEval). Notable gaps include multilingual reasoning, tool use, long-context retrieval, safety/red-team benchmarks, and open-ended generation. The Q-matrix pipeline is designed to extend to new benchmarks without retraining existing skill clusters: adding a new benchmark requires running the LLM skill extraction pipeline on its items, assigning each extracted skill to its nearest existing cluster (or expanding KK), and retraining NCDM.

References

  1. [1]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julien Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviva Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models, 2024. URL https://arxiv.org/abs/2405.14782.
  2. [3]Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Ou, and Shuyue Hu. Learning compact representations of llm abilities via item response theory, 2025. URL https://arxiv.org/abs/2510.00844.
  3. [4]Ying Cheng. When cognitive diagnosis meets computerized adaptive testing: Cd-cat. Psychometrika, 74(4):619–632, Dec 2009. ISSN 1860-0980. doi: 10.1007/s11336-009-9123-2. URL https://doi.org/10.1007/s11336-009-9123-2.
  4. [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
  5. [6]Peng Cui and Mrinmaya Sachan. Investigating the zone of proximal development of language models for in-context learning. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 6485–6498, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.362. URL https://aclanthology.org/2025.findings-naacl.362/.
  6. [7]Peng Cui, Heejin Do, and Mrinmaya Sachan. Do llms exhibit coherent knowledge structures in mathematical reasoning? a perspective from knowledge space theory, 2026a. URL https://arxiv.org/abs/2609.05245.
  7. [8]Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, and Mrinmaya Sachan. What makes something hard(er)? explaining question difficulty in natural language, 2026b. URL https://arxiv.org/abs/2610.01627.
  8. [9]Jimmy de la Torre. Dina model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1):115–130, 2009. doi: 10.3102/1076998607309474. URL https://doi.org/10.3102/1076998607309474.
  9. [10]Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving, 2024. URL https://arxiv.org/abs/2405.12205.
  10. [11]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haoan Li, Kyle McDonnell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602.
  11. [12]Maharshi Gor, Hal Daumé III, Tianyi Zhou, and Jordan Boyd-Graber. Do great minds think alike? investigating human-AI complementarity in question answering with CAIMIRA. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21533–21564, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1201. URL https://aclanthology.org/2024.emnlp-main.1201/.
  12. [13]J. C. Gower. Generalized procrustes analysis. Psychometrika, 40(1):33–51, 1975. doi: 10.1007/BF02291478. URL https://doi.org/10.1007/BF02291478.
  13. [14]Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. STAT: Skill-targeted adaptive training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=m3jG3GaNIj.arxiv.org/abs/2510.10023
  14. [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/be83ab3ecd0db773eb2dc1b0a17836a1-Paper-round2.pdf.
  15. [16]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shri yash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system, 2024. URL https://arxiv.org/abs/2403.12031.
  16. [17]Zhongzhan Huang, Guoming Ling, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin. RouterEval: A comprehensive benchmark for routing LLMs to explore model-level scaling up in LLMs. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 3860–3887, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.208. URL https://aclanthology.org/2025.findings-emnlp.208/.DOI
  17. [18]Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench - a sparse benchmark of reasoning and knowledge in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4I33izzFpK.
  18. [19]Frederic M. Lord and Melvin R. Novick. Statistical Theories of Mental Test Scores. Addison-Wesley, 1968.
  19. [20]Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 34303–34326. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/maia-polo24a.html.arxiv.org/abs/2402.14992
  20. [21]Mark D. Reckase. 18 multidimensional item response theory. In C.R. Rao and S. Sinharay (eds.), Psychometrics, volume 26 of Handbook of Statistics, pp. 607–642. Elsevier, 2006. doi: https://doi.org/10.1016/S0169-7161(06)26018-8. URL https://www.sciencedirect.com/science/article/pii/S0169716106260188.
  21. [22]Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410/.DOI
  22. [23]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.arxiv.org/abs/2311.12022
  23. [24]Pedro Rodriguez, Joe Barrow, Alexander Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4486–4503, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.346. URL https://aclanthology.org/2021.acl-long.346/.DOI
  24. [25]Peter H. Schö nemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, 1966. doi: 10.1007/BF02289451. URL https://doi.org/10.1007/BF02289451.
  25. [26]Daniel O. Segall. Multidimensional adaptive testing. Psychometrika, 61(2):331–354, Jun 1996. ISSN 1860-0980. doi: 10.1007/BF02294343. URL https://doi.org/10.1007/BF02294343.
  26. [27]Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. IRT-router: Effective and interpretable multi-LLM routing via item response theory. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15629–15644, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.761. URL https://aclanthology.org/2025.acl-long.761/.DOI
  27. [28]Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jenyYQzue1.arxiv.org/abs/2310.16049
  28. [29]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoko Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https://aclanthology.org/2023.findings-acl.824/.DOI
  29. [31]Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuralcd: A general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 35(8):8312–8327, August 2023. doi: 10.1109/TKDE.2022.3201037.DOI
  30. [32]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911.
  31. [33]Vilém Zouhar, Peng Cui, and Minmay Sachan. How to select datapoints for efficient human evaluation of NLG models? Transactions of the Association for Computational Linguistics, 13:1789–1811, 2025. doi: 10.1162/tacl.a.60. URL https://aclanthology.org/2025.tacl-1.80/.DOI

Paper details

vlnc:W_2m4cy7ud
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents