Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models

Introduction

Consider an instructor tasked with grading essays. She typically first inspects a sample of the essays to inform her grading policy: some may show excellent writing while others have an impressive vocabulary, etc. These comparable attributes between essays are called features. From this small sample, she can draw up a rubric, namely the detailed features she will use to grade with limited bias. As she grades, she is likely to realize that the original sample that first calibrated her grading policy was too small to contain every important dimension: as the grading goes on, she realizes that grammar is also distinguishing and important, so she needs to go back to the first essays to revise their grading.

This is clearly inefficient, and the instructor would hope to use AI tools to assist in this process of discriminative feature extraction, namely the extraction of the relevant features of the data on which she can base her decisions. If she had essays from previous years together with the grade they obtained, she could train a machine learning model to predict the essays’ grades from their embeddings, which would serve as features for grading. This would automate the grading, but the features used would be opaque and uninterpretable, which is incompatible with transparent decision-making: a crucial aspect in such applications. Current techniques cannot automate the interpretable rubric extraction step. A crucial challenge for decision-makers is therefore: given an unstructured set of items, discover interpretable and discriminative features for this dataset, along with per-item per-feature scores, that they can reliably base their decisions on. Additionally, when decision-making is delegated to an LLM, the same features should make the LLM’s decisions auditable.

A natural approach is to give an LLM the whole set of items and ask it directly for the features that distinguish them (a grading rubric, in the case of essays). However, processing an entire dataset requires an extremely long context, and LLM performance typically degrades as input length increases [1, 2]. Consistent with this, we find that direct LLM extraction misses features that are present in the data and conflates others (§ 4). The difficulty is therefore one of context management: how to use LLMs on the set of items so that they can reliably identify the features that matter. We introduce an end-to-end approach that, given an unstructured dataset, discovers human-interpretable, dataset-relevant features using LLMs.

Contributions. Our idea is that the relevance of a feature can be revealed by a set of pairwise comparisons of items, and that LLM judges can articulate these comparisons in human language. We call such an articulated comparison a rationale: which features the judge cites, and which item it favors on each. The key is that, with enough comparisons, a statistical model can reverse-engineer the judge’s rationales and recover which features distinguish the items and how each item scores along each feature.

We treat the rationales as data. A statistical choice model (§ 3) describes each rationale as a two-stage outcome: whether the LLM cites a feature, and, when it does, which item wins on it. The model absorbs the LLM’s potential biases, such as a preference for the first or second position in the prompt. Fitting it yields per-item, per-feature scores with confidence intervals, together with diagnostics that filter out features with unreliably estimated scores. Estimation is non-trivial: the likelihood is non-convex and the number of parameters grows with the number of items. We nonetheless prove, under mild assumptions, that the likelihood admits a local maximizer that is consistent for every item score at once in the regime where the number of items grows (Theorems 1 and 2).

We validate our approach extensively in three settings. First, we benchmark it on a synthetic hiring dataset of cover letters with known ground-truth features, where our method correctly recovers all planted features, and most discovered features track a single planted one, whereas methods that prompt an LLM with the whole dataset/a sample of it miss some features and blur several together (§ 4). We then apply it to a real dataset of online debates (the ChangeMyView subreddit) to discover and quantify which argumentation techniques are the most effective. We find that our method reaches an accuracy comparable to the state of the art while providing interpretable features (§ 5) along with insights on argumentation techniques. Finally, we show in Appendix § 9 how an LLM’s rationales can also be used to audit its decisions and represent, in interpretable terms, what actually mattered in its decision when comparing two items, yielding interpretability insights for LLM-based scoring.

Our approach provides a concrete procedure that can be supplied as instructions to an LLM when tasked with feature discovery. It offers a principled way to organize multiple LLM agents for this purpose: a lead agent delegates pairwise comparisons along a graph to parallel sub-agents and aggregates their rationales through a statistical model. This aligns with recent efforts to scale AI performance through the orchestration of multiple agents [3], where how the work is split and recombined is critical to performance [4].

Related work

The LLM-as-a-Judge paradigm The LLM-as-a-Judge paradigm is an approach to several problems that uses LLMs to evaluate or rank items where one could have solicited a human judge. Examples include model benchmarking: in their foundational work, Zheng et al. [5] asked both LLMs and humans which of two AI-generated answers to a prompt they preferred, and found substantial agreement between the two, suggesting that LLMs could replace humans for comparing items. Nowadays, LLMs are used as judges in a variety of domains, spanning information retrieval [6], recommendation systems [7], alignment training [8], automated essay scoring [9], etc. LLM biases. Nonetheless, several works [5, 10, 11] document biases in LLMs, such as a positional bias (when asked to compare two items, the LLM prefers the item that appears first/second in the prompt for no particular reason), a verbosity bias (a tendency to prefer more verbose items, such as longer essays if the items being assessed are essays), etc. These phenomena can severely undermine LLM-based decision-making if not accounted for.

Feature evaluation with LLMs. Closer to our work, general LLM-based approaches typically work from a predetermined pool of features: the features are scored, but never discovered from the data. Evaluation features are either fixed in the prompt [5, 12], drawn from a predefined skill taxonomy [13], or supplied by the user as a scoring rubric [14]. These approaches work by asking the judge to output a few tokens per feature: either a pointwise score [12, 14] or the verdict of a pairwise comparison [5] (Liusie et al. 2024) (“A” if item A is better, “B” otherwise). The judge’s rationales are often discarded or used as a qualitative confirmation, while we argue that they could be treated as data useful for feature discovery. Some approaches [16] leverage them to create scoring guidelines to explain existing scores and preference labels, but to our knowledge no approach uses them for interpretable feature extraction.

Method

Our method proceeds in four steps, illustrated in Figure 1. First, LLMs compare pairs of items along a comparison graph, describing the features on which they differ and indicating which item shows more of each feature. These local comparisons require only a small context but are myopic, missing dataset-level information. Second, we group these descriptions into a shared set of features. Third, we fit a statistical choice model that aggregates these local observations into dataset-level, per-item feature scores, using both the cited differences and the features left unmentioned. Finally, we derive per-feature identifiability and reliability diagnostics that flag features whose scores are not pinned down by the observations.

Rationale collection. Let $1,\ldots,n$ denote the items. We compare them along a comparison graph, a set of $N$ pairs $p=(i,j)$ where item $i$ is shown first. For each pair, an LLM judge is asked how the two items differ: it answers in free text, naming the features on which they differ and which item is stronger on each. These raw features, phrased differently across pairs, are then passed to a second LLM call that groups them into $K$ shared features. The outcome is a sparse verdict tensor $T \in\{-1,0,+1\}^{N \times K}$: $T_{p,k}=0$ if feature $k$ was not invoked for pair $p$, and $T_{p,k}=+1$ (resp. $-1$) if it was invoked and the first (resp. second) item of the pair was cited as stronger.

Statistical choice model. Given the verdicts $T$, our objective is to reverse-engineer the items’ scores along each feature and the judge’s biases. Our statistical model takes inspiration from Bradley–Terry models [17] to estimate latent scores from pairwise comparisons (see Appendix § 10.§ 1 for related work).

We assume that each item has a latent score along each feature, collected in a score matrix $S \in\mathbb{R}^{n \times K}$. If pair $p$ compares items $i$ (shown first) and $j$, we write $\Delta_{p,k}=S_{i,k}-S_{j,k}$ for the true difference of their scores along feature $k$. Our model posits that these score differences drive the LLM’s behavior in two ways: a) the more the items’ scores differ on $k$, the more likely the LLM is to cite feature $k$; b) when feature $k$ is cited, the item having the larger score along it is more likely to be chosen.

Importantly, we do not assume the LLM does any of this reliably: its citations and choices are noisy and may also carry the judge’s biases. We hence model the citation of feature $k$ on pair $p$ as a coin flip of probability $\sigma(f(\Delta_{p,k})+\gamma_k)$, where $\sigma$ is the sigmoid function and $f$ is an even function increasing in $|\Delta|$ (we use $f=\log(2\cosh)$, a smooth surrogate of the absolute value), so that the non-citation probability is

Here $\gamma_k$ is the baseline citation propensity of feature $k$, which accounts for a feature-specific eagerness or reluctance of the LLM to cite it. Under this model, silence is also informative: a non-citation suggests that the judge did not find the pair’s difference on that feature salient enough to cite it.

Our model also assumes that, conditioned on citing feature $k$, the LLM declares the first item the winner with probability $\sigma(\Delta_{p,k}+\beta)$, where $\beta$ captures the positional bias widely reported in the literature [5, 10, 11], which we also observe in our experiments. Multiplying the two gives:

Hence, $\beta$ and $\gamma_k$ capture the LLM’s biases (a preference for a position in the prompt, and feature-specific citation habits) so that they do not contaminate the estimated scores. The position coefficient $\beta$ is shared across features; as we show in Appendix § 10.§ 2, performing comparisons with presentation order randomized makes the scores essentially insensitive to this choice, and a per-feature refit confirms it empirically.

Parameter estimation. We estimate $\theta:= (\operatorname{vec}(S),(\gamma_k)_{k=1}^{K},\beta) \in\mathbb{R}^{nK+K+1}$ from the verdict tensor $T$ by penalized maximum likelihood,

where $\mathcal{L}$ is the negative log-likelihood of $T$ and $\lambda\ge0$ is a ridge parameter. Under conditional independence of the verdicts across pairs and features given $\theta$, $\mathcal{L}$ factorizes as

The likelihood is invariant to shifting all scores of a feature by a common scalar; the ridge penalty resolves this by selecting, within each invariance class, the centered representative $\mathbf{1}^{\top}s_k=0$, where $s_k$ is the $k$-th column of $S$.

Theoretical guarantees. Our estimator raises a nonstandard consistency question: the number of parameters grows with the number of items $n$, and the negative log-likelihood (4) is mildly non-convex. We prove, under reasonable assumptions, that the likelihood admits a local maximizer that is consistent for every item score at once as $n$ grows with a bounded number of comparisons per pair, in the case $K = 1$ (single feature).

In Appendix B, we prove a more general version of Theorem 1 (Theorem 2) in which both channels carry arbitrary comparison-level covariates and $\gamma$ and $\beta$ are coefficient vectors.

In practice, we optimize the regularized likelihood (3) numerically on sparse comparison graphs, and the fitted model is well calibrated out of fold (Figure 13 and Figure 14). Theorem 1 assumes a dense design and does not cover this regime; what it guarantees is that, near the truth, the non-convex likelihood has a unique stationary point, which is a consistent local maximizer, so a local optimizer targets a well-defined estimator rather than an arbitrary critical point.

Extracting score reliability. Because $\hat{\theta}$ is an M-estimator, it satisfies a central limit theorem as the number of comparisons grows, which yields confidence intervals on every parameter (see Appendix § 10.§ 3 for more details). The same machinery gives two per-feature diagnostics: a condition number $\kappa_k$ of the score block of the Fisher information, which flags features whose scores are poorly identified by the data ([18]), and a reliability coefficient $\rho_k$, the fraction of the observed score spread attributable to true between-item differences rather than estimation noise. Features with $\kappa_k > \kappa_{\max}$ or $\rho_k < \rho_{\min}$, for thresholds fixed a priori, are filtered out before a final refit.

Scoring new items. Running the full pipeline on the entire dataset allows the statistical model to aggregate comparisons globally, correct the judge’s biases, and provide reliability diagnostics. With $d$ comparisons per item, this requires $dn/2$ pairwise LLM calls. A cheaper alternative is to discover features on a representative subset and then ask an LLM to score additional items independently along those features, requiring one call per item. This pointwise scoring requires specifying precisely how each feature should be scored, which may require human contextual knowledge of the dataset. It is also subject to well-documented biases of pointwise LLM scoring ((Liusie et al., 2024)). Alternatively, one can compare each new item pairwise against a fixed number of already-scored items along the fixed features and fit a Bradley–Terry model for each feature. This retains dataset-level aggregation and position-bias correction without repeating feature discovery. The choice trades cost against robustness to pointwise biases; both are evaluated in § 5.

Empirical Results on Interpretable Feature Extraction

We first evaluate our approach in a synthetic hiring application where we control the ground-truth features, allowing us to test which features each method recovers and how accurately it recovers them. We construct a dataset of $n = 200$ cover letters responding to a real Google security software engineering job advertisement, with known features (which we call dials) embedded in each letter. See Appendix § 11 for details about the experiment.

Dataset. Each letter realizes 18 engineered ground-truth dials: 8 primary ones (years of experience, education level, school prestige, number of languages, production launches, skill coverage, employer prestige and prose fluency), 6 binary role-skill presences (e.g. Python, security assessments), and 4 three-level niche dials (e.g. open-source involvement, security certifications) exactly two of which are present in each letter.

Dials are drawn near-orthogonally and with capped correlation to letter length, so that each dial is a separately recoverable signal and length cannot proxy for quality. Of the 18 generated features, $D = 16$ are relevant recovery targets, while two serve as controls: skill coverage is a redundant composite of the individual skill features, and prose fluency is irrelevant to the job.

Metrics. A discovered feature is said to track a dial if the cover letters’ scores along the feature correlate sufficiently with the dial. A discovered feature is said to be about the dial if its name and the dial’s name are sufficiently aligned in TEXT-EMBEDDING-3-LARGE embedding space. We will compare methods along the following metrics (defined mathematically in Appendix C.2.1).

• Cost. Total method cost in USD.

• Statistical recovery (Tracked & Stat.R.). The fraction of tracked dials (Tracked), and how strongly on average each dial is tracked (Stat.R.).

• Semantic recovery (Sem.R.). The fraction of dials that have a discovered feature that is both about it and tracks it. By definition, it is at most Tracked: the gap between the two counts dials tracked under a misaligned name.

• Attributable features (Attrib.). The fraction of the discovered features that track a single dial.

• Leakage (LK). The average correlation between a dial and the attributable features other than the one that best tracks it. Since dials are near-orthogonal by construction, a feature that measures one dial should be uncorrelated with the others; leakage quantifies how much of the other dials nonetheless bleeds into its scores (lower is better).

Benchmarks. We compare our approach to direct LLM feature extraction and its variants. Specifically, we provide a powerful LLM (GPT-5.4 (HIGH REASONING)) with the whole dataset or a subset and ask it to identify features and formulate dataset-specific guidelines for scoring them. We then use a cheaper model (GPT-5.4 MINI) to score each cover letter on each feature. More precisely, we consider the following variants:

• Direct LLM: An LLM is queried on the full dataset to discover features, after which a cheaper LLM scores each item along each feature. We also consider Direct LLM–Multi-score, where multiple agents score each item with slightly different prompts, and their scores are averaged. This variant approximately matches our approach’s cost, allowing us to evaluate whether the benefits arise from using multiple agents or from our specific way of orchestrating them.

• Local LLMs: Multiple agents each examine a randomly selected subset of the dataset and discover features. We pool their features and use an LLM to merge equivalent descriptions into a shared set (see Remark 7 in Appendix C for the choice of merging procedure). A single agent then scores each item along these features. Analogously, we also consider Local LLMs–Multi-score. These variants help assess whether the performance benefits arise from distributing feature discovery across multiple agents or from our pairwise comparisons and statistical aggregation.

Ablations. To examine the contributions of statistical scoring and reliability filtering, we retain the features discovered through our sub-agents’ pairwise comparisons and consider two ablations, each with single-agent and multi-agent scoring.

• Pairwise comparisons + LLM scoring: LLMs directly score all our discovered features, replacing statistical scoring and its reliability filtering.

• Reliable features + LLM scoring: An LLM scores only the features retained by our statistical model’s reliability filter. Comparing these two ablations isolates the contribution of filtering under LLM scoring.

Results. Table 1 reports the metrics of each benchmark and ablation. Our approach outperforms direct LLM feature extraction, even though that benchmark uses one of OPENAI’s most capable models available at the time of the experiments. It tracks all 16 ground-truth features, compared with 13 for Direct LLM, and achieves complete semantic recovery: the discovered features both track the underlying signals and describe them meaningfully. The three missing dials are a discovery failure rather than a scoring failure: at matched cost, Direct LLM–Multi-score still tracks the same

13. More of our features are attributable to a single ground-truth feature (16 of 21 versus 13 of 21), and leakage is lower (0.090 versus 0.109). Direct LLM is however cheaper ($0.26 versus $3.45).

Our approach also outperforms the multi-agent alternatives, whether multiple agents are used for feature discovery (Local LLMs), scoring (Multi-score), or both, with statistically significant gains in tracking, attribution, and leakage over Local LLMs (Figure 15). Multi-agent scoring improves statistical recovery but increases leakage, while multi-agent feature discovery reduces recovery compared with a single agent with the full dataset. The benefits therefore do not follow simply from using multiple agents: how we divide their work into pairwise comparisons and aggregate their outputs through a statistical model matters.

The ablations also achieve high tracking and semantic recovery, consistent with their use of the same features as the full pipeline, but without statistical scoring and its reliability filter, attribution is substantially lower (13–14 attributable features versus 16) and leakage higher (0.103–0.104 for Multi-score methods versus 0.090), despite a slightly higher Stat.R. (0.860–0.868 versus 0.850) and twice the cost. The resulting scores therefore separate the ground-truth signals less clearly. Filtering alone raises the attributable fraction under multi-agent scoring from 14 of 32 to 14 of 21 while preserving complete tracking and semantic recovery, showing that it removes features that track no dial. Statistical scoring on the same 21 features then raises it to 16.

Application to a Real Dataset

We next evaluate our approach on a real-life dataset whose underlying features are unknown: online debates from Reddit’s ChangeMyView forum. The central question is whether automatically discovered features provide interpretable insights into persuasion while retaining sufficient signal to predict which arguments change a participant’s view.