WorksComputer Science
Synthesizing symbolic models of human and machine reasoning at scale
Abstract
Symbolic models offer a powerful account of cognition, but their scope has historically been limited by the need for hand-engineering and computational tractability. We show how advances in artificial intelligence (AI) can scale symbolic modeling and help overcome these engineering limitations. We introduce Behavior-Conditioned Symbolic Synthesis (BCSS)—an approach for automated search of interpretable, deterministic symbolic programs that verifiably reproduce behavior—and apply it to model both human and machine behavior across hundreds of challenging reasoning problems. Analyzing these programs reveals that both people and machines tend to oversimplify, adopt rules that are partial explanations for their observations, and misinfer dependencies between problem features. These shared signatures are modulated by available resources, with reasoning effort predicting the extent to which machines simplify and explain their observations. Together, these results point to underlying mechanisms reminiscent of satisficing in both people and machines, and illustrate how BCSS offers a general approach for interpreting and comparing rule-based reasoning of humans and black-box AI systems.
Symbolic models of cognition provide a long-standing and influential framework for understanding the human mind [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. Beyond theoretical arguments that the mind is inherently symbolic and rule-like [2, 3, 7, 11], these models offer an alternative to contemporary deep neural networks and foundation models [12, 13, 14] that is typically more interpretable and explicit in its representation. Historically, a key obstacle to constructing symbolic models has been the need for extensive hand-engineering of these representations [15, 16], which can include specifying underlying symbolic primitives and their compositions [5, 10, 17, 18, 19, 20, 21]. These demands are tractable in restricted experimental tasks where the relevant hypotheses can be expressed compactly and plausible strategies are approximately enumerable [5], but have posed substantial challenges in more open-ended domains.
Here, we leverage recent advances in artificial intelligence (AI) to automatically synthesize symbolic models at scale in an open-ended domain that is challenging for both people and machines. In particular, we exploit the emergence of strong programming abilities in LLMs [22, 23, 24] to address the issue of combinatorial search within vast discrete program spaces. We developed Behavior-Conditioned Symbolic Synthesis (BCSS), an approach that uses LLMs to automatically search for programs that verifiably reproduce human and machine behavior on problem-solving tasks.
To demonstrate the potential of this approach, we use BCSS to analyze the Abstraction and Reasoning Corpus (ARC), an open-ended set of abstract rule induction tasks spanning a wide range of concepts [25]. Rather than focus on correct responses, we focus on modeling patterns of errors in a recently published large dataset of humans solving these problems [26], alongside a comparison of errors made by state-of-the-art open weight large reasoning models (LRMs). Our hypothesis was that incorrect responses can serve as a rich window into underlying cognitive processes. However, the diversity of errors in a dataset of this size (over 1,700 participants and four model configurations, yielding 6,261 unique human and 4,166 configuration-unique machine errors) means that hand-engineering solutions is impossibly costly.
Our application of BCSS to ARC problems reveals three main insights into human and machine reasoning. First, approximately half of both human and machine incorrect responses we considered are captured by BCSS, suggesting that human and LRM errors are often clustered in regions of program space close to correct reference solutions of ARC problems. Second, analysis of these reverse-engineered programs reveals shared error signatures: both people and LRMs simplify relative to the target rule, find rules that only explain some of the observed evidence, and infer incorrect dependencies between problem features. Third, these signatures are modulated by available resources, with reasoning effort predicting the extent of simplification, partial explanation and reliance on misinferred feature dependencies. More broadly, BCSS programs support investigating patterns of generalization, can expose representational biases and provide bottom-up insights into cognitive mechanisms supporting reasoning. Unlike previous methods, each synthesized symbolic artifact provides a detailed explanation of individual-level behavior, enhancing interpretability of human or machine behavior. We use this fact to expose similarities and differences in abstract reasoning between people and machines using a shared symbolic representational space. We conclude by discussing potential applications of this approach, for example to study mathematical reasoning errors, and the broader implications it has for models of cognition.
Behavior-Conditioned Symbolic Synthesis
Each ARC task presents a small number of input-output pairs of colored grids (typically two to five) that demonstrate a hidden transformation rule. The solver must infer the rule and apply it to a new test input by constructing an output grid cell-by-cell (Figure 1A). Each task is a challenging problem of program induction given the limited number of examples and the effectively unbounded space of possible transformations. Reasoners can go wrong in many ways: adopting a rule that fits only some of the examples, attending to the wrong feature of the objects involved, or identifying the correct rule but carrying out only part of it.

Figure 1. Reverse-engineering individual rules in a complex cognitive domain using programs. (A): Schematic task corpus and associated behavioral data. Each of the four representative ARC tasks presents an agent with a small set of input–output training examples from which a hidden transformation rule must be inferred. From a test input, the agent must then construct the corresponding output grid (hidden with a grey square and “?” for illustrative purposes). We use H-ARC, a dataset of responses from over 1,700 participants to the 800 publicly available ARC-AGI-1 tasks. On the right of each task, we show 4 of the most frequent incorrect responses, and the correct response at the bottom. Under each grid, we show the associated response frequency (red for incorrect, green for correct). (B): Behavior-conditioned symbolic synthesis (BCSS). For each unique incorrect response, a large language model is prompted with a full specification of the domain-specific language, task grids, a reference program implementing the target rule for task , and the observed incorrect response (here, from a human participant). It proposes independent sets of at most line edits to that program. The schematic edit proposals show LLM-sampled proposals which are then applied to . Each proposed set of line edits is applied programmatically to the reference program to obtain a new program . A custom parser rewrites the programs such that only causally relevant code is retained (i.e., unreferenced lines of code are removed). Each new program is then executed with the test input as argument. Programs whose output reproduces the observed response are kept (green checkmark); all others are discarded (red cross). Because a response is often reachable by more than one unique program, BCSS returns a collection of programs per response rather than a single model. The individual response shown here is also shown surrounded by a red square to the right of task in panel A. The constant THREE represents the color green.
The four subsets of participant-level response examples in Figure 1A illustrate that incorrect responses often reflect systematic patterns of inference rather than random or defaulting behavior (incorrect solutions cluster near the correct one shown at the bottom). Similarly to prior work in diagnosing errors in children’s mathematical reasoning [27, 28], we hypothesized that such responses may correspond to constrained deviations from a normative standard. Additionally, we assume that the rule-like inferences of people and machines in this domain are best understood as programs [29].
We operationalize this hypothesis using simple but expressive programs manipulating grids and implemented in a symbolic domain-specific language (DSL). The DSL we use was developed for expressing programmatic solutions to ARC and includes correct, human-written reference programs for the 400 training tasks of ARC-AGI-1 [30]. Although BCSS generates a symbolic representation of the underlying algorithm a person or AI system may have applied to produce their response, it is not by itself a process-level description of how a program was inferred. Future work using our approach while leveraging more detailed behavioral traces from the reasoning process (e.g., think aloud protocols) could help expand our understanding of how people infer these rules. Here, we instead focus on characterizing the space of final programs applied to produce responses in people and machines.
Specifically, for a given individual response, our method searches for programs representing output behavior by sampling constrained sets of deviations from a reference program. As illustrated in Figure 1B, given a reference program implementing the target rule for a task , an LLM proposes independent sets of at most line edits to that program. Importantly, proposal sets are sampled conditioned on single human or machine behavioral responses. Each resulting candidate program is then executed and retained only if it reproduces the response the proposal LLM was conditioned on. Thus, the LLM guides search over a discrete program space by supplying a proposal distribution for plausible deviations implemented as line edits, whereas program execution provides a hard verification mechanism.
Results
We leverage the H-ARC dataset, a large corpus of human responses from over 1,700 participants to the full set of 800 publicly available ARC-AGI-1 tasks [26]. H-ARC contains up to three attempts per task from each participant, with correct/incorrect feedback after each attempt. We filtered H-ARC to narrow down the dataset to responses more likely to arise from conceptual rather than procedural mistakes (§ 4.1 for more details). The final filtered H-ARC dataset contains 6,261 unique incorrect outputs across 723 tasks and 1,387 participants. For the 400 evaluation tasks which did not have a human-written solver, we successfully synthesized reference programs for over 90% of the tasks (see § 4.2.1 for more details), and conditioned error program synthesis using those programs. Unless specified, we report human results aggregated over modeled attempts from the training and evaluation splits of ARC-AGI-1. Wherever substantial divergence was observed, we report results separately. To compare human and machine errors, we additionally evaluate a range of large reasoning models (LRMs) on the 400 training tasks of ARC-AGI-1 (see § 2.5 for more details). Because each response is typically explained by several candidate programs, and tasks differ widely in how many errors they elicit, observations are unevenly distributed across programs, errors and tasks. We therefore report estimates from hierarchical Bayesian regressions [31]. This approach allows us to pool information within each error and across errors within each task or group, so that heavily sampled errors and tasks do not dominate the results (§ 4.8). Accordingly, we report posterior means or medians with 94% highest-density intervals (HDIs).
Synthesized programs capture human errors
After verification and filtering (see § 4.3 for more details), we obtained 8,246 total programs matching individual responses to tasks in both the training and evaluation splits of ARC-AGI-1. These programs account for 52.5% of non-trivial incorrect attempts on the training split, and 46.2% on the evaluation split (49.1%, 3,636/7,402 pooled; Supplementary Table S2 for the full per-split breakdown). Given the edit constraints on the synthesizer, our results demonstrate that a large proportion of incorrect human responses can be modeled by programs a few line edits away (, , § 4.3) from reference programs implementing the target rule for each task. Further, among successfully modeled incorrect responses, we recover a median of 2 distinct programs per unique error grid (IQR: [1, 4]). This many-to-one correspondence is illustrated in Figure 1B with two distinct, but successful edit sets. The first replaces the leastcolor(I) primitive operation with a hardcoded color, whereas the second additionally substitutes a selection operator. Both yield programs whose output reproduces the same participant response, and both are retained.
We compared BCSS against an exhaustive search over all single-line rewrites of the training reference programs (§ 4.3). Despite enumerating over 16M edits, this baseline captures only 25.8% (899/3,485) of erroneous attempts—roughly half of what BCSS achieves on the same data. With only 15 synthesized samples per response, BCSS still recovers 89% of the exhaustively searched programs that match human responses. Because the number of candidates grows combinatorially with edit depth, exhaustive search is intractable at the edit distances where many human errors lie (see § 4.5 for validation of BCSS using manually verified synthetic errors).
Under the view of ARC as program induction, our results suggest that people often infer incorrect programs that are structurally similar to the program implementing the target rule for a given task, providing support for our initial hypothesis that incorrect responses in this open-ended domain are not merely idiosyncratic failures.
Inferred programs are systematically simpler
Human learners intrinsically value simple representations over more complex ones [32, 33, 34], and this fact is sometimes an essential component of models posited to capture human behavior [5]. Here, we examine whether this bias for simple codes is a property of the behavioral outputs which we modeled using BCSS, beyond the shortening bias intrinsic to the BCSS edit procedure. As a test of this idea, we applied the same procedure used for producing error programs using identical reference programs, but instead conditioned the synthesizer on correct output grids (see § 4.8 for more details on statistical models). This measures the inherent tendency of BCSS to produce shorter programs independent of the nature of the errors. Because the programs are adapted using line-level edits, we use relative line-count change as a proxy for difference in program complexity with respect to the reference programs. Statistical analyses confirm that DSL programs that capture human responses are credibly simpler than reference programs they are adapted from, on both splits (Figure 2C). To obtain these results, we fit a regression predicting average relative line-count change for error programs and control programs. On the training split, where each reference program is hand-written by a human, error programs are 19.1% shorter on average (94% HDI [−0.206, −0.176]), compared with only 5.2% for baseline (correct-output) programs (94% HDI [−0.076, −0.028]). This corresponds to an additional shortening of 13.9 percentage points attributable to conditioning on human errors (94% HDI [−0.164, −0.116]). The same pattern holds on the evaluation split, with the caveat that its reference programs are also model-synthesized rather than hand-written. Error programs from the evaluation split are 24.3% shorter on average (94% HDI [−0.259, −0.225]) versus 2.7% for baseline programs (94% HDI [−0.055, −0.002]). This corresponds to an additional 21.5 percentage points attributable to conditioning on human errors (94% HDI [−0.240, −0.189]). Panel D of Figure 2 illustrate a drastic but intuitive example of simplification: the target rule for this task requires searching over the eight rotations and reflections of a framed sub-grid for the one consistent with the surrounding cells, but the programs reproducing the two bottom responses instead use fixed transforms (mirror and rot180, respectively), rendering fourteen lines of the reference program irrelevant to the final output.

Figure 2. Synthesized programs suggest people often infer rules simpler than the target rule for a given task. (A): A representative ARC task (94133066). Three training examples demonstrate the target rule—return the contents of the framed sub-grid under whichever of its eight rotations and reflections is consistent with the isolated colored cells scattered around it—followed by the test input for which participants construct an output grid (hidden with a grey square and “?” for illustrative purposes). (B): The reference program implementing the target rule for this task. Because this task belongs to the evaluation split, is itself model-synthesized rather than human-written. (C): Distribution of relative program length change, , over all synthesized programs, shown separately for the training and evaluation splits of ARC-AGI-1. Treatment programs (blue) are those conditioned on incorrect human responses; control programs (grey) are produced by an identical procedure conditioned instead on the correct output grid. Because edits replace lines in-place and causally irrelevant code is subsequently removed, synthesized programs can either shorten, or remain the same, but never lengthen. Therefore, the control condition controls for the shortening bias attributable to the procedure itself, against which the leftward displacement of the treatment distribution should be read. The dashed line marks the median shortening of modeled responses for the task displayed in this figure. (D): The correct response (top row) and the three of the most frequent incorrect responses to that task, with response frequencies below each grid (green, correct; red, incorrect). For each, we show the diff with after applying BCSS, the length of the resulting program relative to (blue, retained lines; gray, removed lines; red rule, reference length), which training examples the program reproduces (green, reproduced), and its error category. A single edit can account for a large simplification: for example, replacing the final extract over the eight candidate transforms by one fixed transform renders fourteen lines of the reference program inconsequential, and they are removed. Thus, the simpler resulting program hardcodes a transform instead of exhaustively searching over possible transformations.
To examine this effect further, we fit an additional Bayesian regression predicting relative line-count change in error programs as a function of reference program line count, empirical task success rate and error categories for training split tasks only (see § 2.4 for further discussion of error categories; see § 4.8.4 for details on the regression). The statistical model finds that reference program complexity modulates the magnitude of relative shortening. For each additional line in a reference program, we find 0.47 percentage points more shortening on average in corresponding error programs (94% ). For example, error programs derived from a 7-line reference program are predicted to be 5.8% shorter on average, whereas those from a 28-line reference are 15.7% shorter. This corresponds to a 9.8 percentage-point difference for a four-fold increase in reference program length (Supplementary Figure S3A).
Additionally, we found suggestive evidence that conditional on a reference program’s line count, errors from tasks with higher empirical success rate are modeled by slightly less relative shortening (94% HDI ). The estimated coefficient suggests that easier tasks may tend to elicit error rules closer to the reference programs in complexity, but the credible interval includes zero so these results remain inconclusive.
Although the procedure is biased toward shortening, our analysis suggests that, beyond this bias, the incorrect rules that people infer are often represented by substantially simpler programs relative to the program implementing the target rule for a given task. These results hint at a bounded cognition account, where people possibly adopt increasingly simpler rules when the target rule is costlier to infer or verify.
Inferred programs partially satisfy available evidence
Because we represent the rules that people inferred as programs, we can verify how these programs behave on the training examples that participants observed. For each program that reproduces a human response and its associated task, we executed it on each of the training input grids to determine whether it produced any correct output grids. To aggregate over multiple programs fitting each error, we fit a Bayesian categorical regression predicting the extent to which errors are modeled by programs that reproduce none, some or all demonstrations seen by participants (see § 4.8.2 for more details on the statistical model). Overall, we find that 22.6% (94% HDI [21.9%, 23.4%]) of modeled incorrect attempts correspond to programs consistent with at least one training example, with no major differences between splits (see Supplementary Table S3 for per-split results). This subset of programs is split between two rule types: partial-match and alternative rule (Figure 3C, top). Partial-match programs correspond to rules where a strict subset of training outputs are correctly generated by the error program and account for 20.9% of reproduced incorrect attempts (94% HDI [20.2%, 21.7%]). Conversely, alternative rules correspond to programs that correctly reproduce all observed training examples and the behavioral response (and thus are arguably equally valid), but differ in their prediction for the test output. These alternative rules captured 1.7% (94% HDI [1.4%, 2.0%]) of modeled attempts. The first incorrect response in Figure 3D illustrates a partial-match rule: the program that captures it selects the rightmost object rather than the least common one, a rule that happens to satisfy the first two training examples but fails on the third.

Figure 3. Synthesized programs suggest people infer rules that only explain some of the data they observe. (A): A representative ARC task (b230c067). Three training examples specify the target rule—recolor every shape blue except the least common one, which is recolored red—followed by the test input for which participants construct an output grid (hidden with a grey square and “?” for illustrative purposes). (B): A human-written reference program implementing the target rule for this task. (C): Top, prevalence of each rule type among all modeled incorrect attempts. A program is a partial-match if it reproduces at least one, but not all, of the training outputs when executed on training inputs; an alternative rule if it reproduces all of them yet still fails the test example; and neither otherwise. Bottom, conditional on an attempt being explained by a partial-match program, the positions of the training examples it reproduces: the first- or last- example(s) shown on screen, a contiguous block of at least two examples that doesn’t include the first or last examples, or any other configuration. In both plots, bars give the posterior median attempt-weighted prevalence and error bars the 94% HDI, shown separately for the training (dark) and evaluation (light) splits of ARC-AGI-1; orange dots give the corresponding proportion for the task shown in panel A. (D): The correct response (top row) and three of the most frequent incorrect responses to that task, with response frequencies below each grid (green, correct; red, incorrect). For each, we show the diff with after applying BCSS, the length of the resulting program relative to (blue, retained lines; gray, removed lines; red rule, reference length), which training examples the program reproduces (green, reproduced), and its error category. The most frequent incorrect response is captured by a program that selects the rightmost object rather than the least common one, a rule that satisfies the first two training examples but not the third.
Table S3 Rule types and order effects by split (attempt-weighted). Posterior median and 94% HDI. A program is a partial match if it reproduces at least one but not all training outputs, an alternative rule if it reproduces all of them yet still fails the test example, and neither otherwise. Order effects are conditional on a partial match and describe which of the training examples the program reproduces, ordered as models or participants saw them.
| Training | Evaluation | Combined | |
| (attempts / errors) | 1,827 / 1,274 | 1,809 / 1,507 | 3,636 / 2,781 |
| Rule type (all modeled attempts) | |||
| Neither | 76.2% [74.9, 77.4] | 78.7% [77.8, 79.6] | 77.4% [76.6, 78.1] |
| Partial match | 22.1% [20.8, 23.3] | 19.7% [18.8, 20.6] | 20.9% [20.2, 21.7] |
| Alternative rule | 1.8% [1.3, 2.3] | 1.6% [1.2, 2.1] | 1.7% [1.4, 2.0] |
| Consistent with example | 23.8% [22.6, 25.1] | 21.3% [20.4, 22.2] | 22.6% [21.9, 23.4] |
| Order effects (conditional on a partial match) | |||
| (attempts / errors) | 507 / 302 | 472 / 311 | 979 / 613 |
| First- (prefix run) | 41.0% [38.2, 44.0] | 30.1% [27.6, 32.6] | 35.8% [34.0, 37.7] |
| Last- (suffix run) | 17.0% [14.8, 19.4] | 15.7% [13.8, 17.8] | 16.4% [14.9, 17.8] |
| Contiguous (interior run > 1) | 4.4% [2.9, 5.9] | 2.6% [1.6, 3.7] | 3.5% [2.6, 4.5] |
| Other (single interior or scattered) | 37.5% [34.4, 40.6] | 51.5% [48.7, 54.4] | 44.3% [42.2, 46.3] |
| Structured subset | 62.5% [59.4, 65.6] | 48.5% [45.6, 51.3] | 55.7% [53.7, 57.8] |
Table S3. Rule types and order effects by split (attempt-weighted).
We further examined synthesized programs for order effects: do partial-match programs suggest order-dependent inference or verification of rules? We labeled each partial-match program by which examples it reproduces: the first or the last examples shown on screen, a contiguous block of at least two examples, and anything else as “Other” (Figure 3C, bottom). We then fit a Bayesian categorical regression predicting the extent to which errors tend to be modeled by programs belonging to one of these four categories (see § 4.8.2 for more details on the statistical model). On the training split, 62.5% (94% HDI [59.4%, 65.6%]) of partial-match errors form a structured subset, dominated by the first- pattern at 41.0% (94% HDI [38.2%, 44.0%]), followed by 17.0% (94% HDI [14.8%, 19.4%]) matching the last- and 4.4% (94% HDI [2.9%, 5.9%]) a contiguous block; the remaining 37.5% (94% HDI [34.4%, 40.6%]) match a single interior example or a non-contiguous set. On the evaluation split, the structured share is lower at 48.5% (94% HDI [45.6%, 51.3%]), with 30.1% (94% HDI [27.6%, 32.6%]) first-, 15.7% (94% HDI [13.8%, 17.8%]) last-, and 2.6% (94% HDI [1.6%, 3.7%]) contiguous block.
Despite BCSS not having explicit constraints to encourage the types of programs we describe here, these results suggest that people often find rules that are incomplete explanations of the data they observe. We hypothesize that such rules may be the consequence of ordered, sequential rule induction or verification, paired with early stopping. Taken together, these results are consistent with satisfying [35, 36]: people adopt a rule based on whether it satisfies a “good enough” threshold, rather than verifying it against all evidence or expending more cognitive effort to find better, potentially more complex rules that explain all the observed data.
Inferred programs follow common error patterns
Prior work has sought to characterize reasoning error kinds, for example on intelligence tests [37, 38]. Contrary to success rate, which is a sparse signal indicating differences in degree, error analyses afford characterizing differences in kind, which is informative about underlying cognitive strategies. Unlike the data we model here, researchers often collect forced-choice responses (e.g., Raven’s Progressive Matrices) rather than richer, free-form responses. This implies that the scope of possible error types is predefined by the structure of the incorrect alternative responses of each task. In comparison, we can more richly characterize the space of reasoning errors that people make, without strong constraints on their responses.
To classify types of errors, we devised a novel error taxonomy comprised of seven cognitive categories and a catch-all “unclassifiable” category (see Table 1). For each valid synthesized program of tasks in the training split, we sampled labels using an LLM classifier. We then aggregated program-level classifications at the attempt level using a hierarchical Bayesian model (see § 4.4 for more details). Overall, the model finds a dominant category of errors: attribute misselection at 45.2% (94% HDI [43.9, 46.6]) of modeled incorrect attempts (Figure 4C). Inspired by [39]’s work on attribute substitutions in intuitive judgment, we define attribute misselections as errors that arise when a reasoner fails to identify or compute the correct dependency between an attribute and a target parameter. The first incorrect response shown in Figure 3D demonstrates an example of such a rule: determining which object to color red (i.e., the target parameter) as a function of which object is odd with respect to the others (i.e., the attribute). Instead, participants often rely on possibly simpler or more salient dependencies. Observing that the top-right object is always red is a correct inference based on the first two examples, but it does not hold in the third example. This error category aims to capture simplifications of this kind, and our results suggest that people often rely on such heuristics.
| Category | Definition | Example |
| Attribute misselection | The correct operation is applied, but with a misselected attribute, parameter, or input value. | inferred color hardcoded color |
| Scope restriction | The correct computation is applied, but only to a subset of the entities or regions where it should apply. | all objects subset |
| Operator confusion | The correct class of operation is identified, but the wrong member within that class is selected. | rotate CW rotate CCW |
| Transformation misselection | The correct targets are identified, but a transformation from an entirely different computational family is applied. | mirror recolor |
| Sequencing | All required operations are identified, but they are composed in the wrong order or bound to the wrong targets. | |
| Partial program | A subset of the required steps are executed correctly; one or more computational blocks are entirely missing. | extract + tile extract only |
| Boundary precision | The correct quantity is computed from the correct source, but with a small additive offset (off-by-one). | |
| Unclassifiable | The response reflects an idiosyncratic rule with no recognizable structural relationship to the reference program. | ad-hoc rule |
Table 1. Taxonomy of error categories used to classify synthesized programs. Each category identifies a way in which programs that reproduce incorrect responses deviate from the reference program implementing the target rule for the same task. Categories are defined over structural differences between the two programs, but the classifier is also conditioned on the response grids. Categories are defined as mutually exclusive and the “unclassifiable” catch-all makes the set exhaustive. The examples in the third column are schematic illustrations of the contrast between reference and response programs that defines each category. Every synthesized program was classified by an LLM given a reference program, task grids, the synthesized program, and its execution trace on the training and test inputs, with five independent classifications sampled per program (§3.4).

Figure 4. Synthesized programs reveal a restricted set of recurring deviations from target rules. (A): A representative ARC task (7fe24cdd). Three training examples specify the target rule—tile the input into a arrangement in which each quadrant holds the input rotated by a further clockwise, starting from the input at the top left—followed by the test input for which participants construct an output grid (hidden with a gray square and “?” for illustrative purposes). (B): A human-written reference program implementing the target rule for this task. (C): Prevalence of each error category among modeled incorrect attempts. An LLM classifier assigned every program one of seven cognitive categories, given the reference program, the task grids, the synthesized program and its execution trace (Table 1 for definitions). Bars give the posterior median of the attempt-weighted prevalence derived from a hierarchical Bayesian categorical model, and error bars the 94% HDI; the categories are mutually exclusive. Classification was performed on programs fitting responses to training split tasks only. (D): The correct response (top row) and three of the most frequent modeled incorrect responses to the task in panel A, with the response frequency below (green, correct; red, incorrect). For each, we show the diff with after applying BCSS, the length of the resulting program relative to (blue, retained lines; gray, removed lines; red rule, reference length), which training examples the program reproduces when executed on their inputs (green, reproduced), and its error category. The programs that reproduce these three incorrect responses diverge from in different ways: omitting the rotations and tiling the input four times, carrying out reflections instead of rotations, or correctly carrying out the rule on a restricted portion of the input.
Additionally, we find that people often infer rules that correspond to partial programs, representing 11.9% (94% HDI [10.7, 13.2]) of modeled incorrect attempts. Such errors arise when participants correctly applied a rule that captures a subset of the correct steps for solving a task. Finally, we find that 10.8% (94% HDI [9.8, 11.7]) of incorrect attempts are transformation misselections, where the participant applied an incorrect transformation to the right parts of the input. The remaining four cognitive categories represent less than 5% of modeled incorrect attempts, and 22.3% (94% HDI [21.5, 23.3]) are determined to be unclassifiable under our taxonomy.
Every modeled response we display is labeled with its modal error category (panel D, last column of Figure 2–Figure 4), illustrating the different ways in which participants can deviate from a single target rule. For task 7fe24cdd in Figure 4, the rule is to tile the input into a arrangement of successive rotations. In that figure’s panel D, the first response omits the rotations and simply tiles the input four times (partial program, four lines shorter), the next preserves the construction but substitutes reflections for rotations (transformation misselection, same length), and the third applies the construction to a subset of the input (scope restriction, one line shorter). As suggested by how these examples differ in shortening, beyond the effect of target rule complexity (§ 2.2), we observe general quantitative differences in relative shortening across error categories, with localized errors such as boundary precision leaving program length largely unchanged, and partial program errors substantially shorter than the target rule (Supplementary § 9).
Taken together, these results provide evidence that errors on ARC tasks are not idiosyncratic deviations from the target rule. Most modeled incorrect attempts fall into a small set of categories, with attribute misselection as the most commonly occurring error type. Individual categories also tell us how much of the target rule’s structure is kept. Localized errors retain most of it, while partial programs and transformation misselections are coarser approximations. Error analyses of BCSS programs can therefore not only capture whether people’s failures are rule-like, but also how their inferences diverge from the target rule.
Comparing human and machine errors
Since the advent of reasoning-tuned LLMs [40, 41], performance on ARC has drastically increased, reaching near-ceiling performance among the most capable models.1 Given observed scaling trends of performance with respect to reasoning effort on these problems, we investigate whether the patterns of results we established in people differ for these models, and how modulating reasoning effort might vary results. We study two reasoning-tuned LLMs (hereafter referred to as large reasoning models or LRMs), Qwen3.5-27B [42] and GPT-OSS-120B [43], both of which generate long chains of thought (CoTs) before producing an answer. Because GPT-OSS-120B allows us to vary how much the model dynamically allocates tokens for CoT before providing an answer (i.e., varying test-time compute), we evaluate it at high, medium, and low reasoning effort. We sample ten grid responses from each model and configuration on each of the 400 training tasks in ARC-AGI-1 (§ 4.6).
Figure 5A shows the average accuracy across independent attempts and its standard deviation across runs. Performance varies substantially across models and reasoning efforts. Qwen3.5-27B, with approximately 27 billion active parameters, performs best at 80.63% accuracy, followed by GPT-OSS-120B, with approximately 5.1 billion active parameters, at 72.28% under high reasoning effort. Both models outperform the estimated average human accuracy of 59.9% on a single attempt, as reported by [26]. Reducing reasoning effort substantially lowers GPT-OSS-120B’s performance to 50.96% at medium effort and 25.02% at low effort. After filtering (§ 4.1), we observe 4,166 configuration-unique errors across all configurations, spanning 357 of the 400 tasks.

Figure 5. Machine errors are systematic and performance-dependent. (A): Average accuracy across responses of large reasoning models at different reasoning efforts and humans on the training set of ARC-AGI-1. Note that GPT-OSS-120B uses a mixture-of-experts architecture and thus only has 5.1 billion active parameters during inference. (B): Relative shortening of programs synthesized from model errors compared with programs for correct-output controls; all model configurations exhibit a simplicity bias, which is stronger for lower-performing configurations. (C): The proportion of error programs that partially match or fully reproduce the support examples increases with model performance. (D) Schematic of the classification analysis distinguishing human and machine errors using features derived from synthesized error programs. (E) Within-task ROC-AUC of three fitted classifiers provides evidence of varying degrees of discriminability of human and machine errors on held-out tasks, with less performant models better discriminated from people. Classifiers are trained using linearized program-level feature vectors. ROC-AUC is computed among the errors of each of the 56 shared tasks and averaged across tasks, with error bars showing 95% task-level bootstrap CIs. Panels B and C report results on the subset of 56 tasks where both reasoning language models and people generated incorrect responses (see Appendix § 11 for similar trends across all errors). All machine responses are independently sampled, and we restrict comparisons to human first-attempt responses.
BCSS captures a large fraction of LRM errors, indicating that many admit systematic programmatic descriptions. After verification and filtering (§ 4.1), we obtain 3,804 programs that capture 62.8% (1,565/2,491) of GPT-OSS-120B incorrect attempts at low reasoning effort, 2,760 programs that capture 59.3% (1,031/1,738) at medium reasoning effort, and 944 programs that capture 54.4% (355/652) of its incorrect attempts at high reasoning effort. For Qwen3.5-27B, 823 programs capture 51.5% (296/575) of its incorrect attempts. For comparisons across models and humans, we restrict subsequent analyses to the 56 tasks on which humans and every model configuration make at least one error, so that comparisons are made on errors in a common set of tasks. We find similar trends hold when considering the full pool of errors across models (see Supplementary Tables S6–S8). Finally, we only include first-attempt human errors to match the machine evaluation procedure where responses are independently sampled.
We investigate whether LRM errors exhibit a similar simplicity bias as observed in people by looking at whether programs synthesized from LRM errors are shorter than those synthesized for the correct-output control group (see § 2.2 for human results; see § 4.8.1 for more details on the regression model). Figure 5B shows the average relative shortening of error programs compared with baseline programs. Two notable patterns emerge. First, all model configurations exhibit a simplicity bias similar to humans, but to a varying degree. For the strongest model we tested, Qwen3.5-27B, its error programs are 11.9% shorter on average than baseline programs (94% HDI [−0.164, −0.073]). For GPT-OSS-120B, error programs are 18.6% shorter at high reasoning effort (94% HDI ), 23.3% shorter at medium reasoning effort (94% HDI ), and 29.3% shorter at low effort (94% HDI ). Second, the magnitude of relative shortening is correlated with accuracy: programs that represent errors of lower-performing systems are shorter relative to baseline than those of higher-performing systems (Pearson’s , 94% HDI , ; § 4.8.1). Each additional 10 points of accuracy corresponds to 3.0 percentage points less shortening (94% HDI ). These results provide
model-based support for our hypothesis that simplification is driven by resource constraints: less capable systems, or those devoting less compute to reasoning, tend to infer simpler rules.
As with people, we also analyze the prevalence of rule types and order effects (§ 2.3) within programs synthesized from LRM errors. Figure 5C shows that the proportion of partial-match rules increases with model capacity, and alternative rules are substantially more frequent for the two highest-performing configurations. For GPT-OSS-120B at low reasoning effort, only 20.2% (94% HDI [17.2%, 23.2%]) of programs produce partial matches and 3.0% (94% HDI [1.9%, 4.2%]) represent alternative rules. For Qwen3.5-27B, these proportions rise to 40.0% (94% HDI [36.6%, 43.5%]) and 6.7% (94% HDI [4.9%, 8.7%]), respectively (Supplementary Table S7 for all values across the full set of errors). Only GPT-OSS-120B at high reasoning effort shows a human-like order effect, with more first- than last- partial matches (36.1%, 94% HDI [30.8%, 41.4%] vs. 21.8% [16.9%, 26.9%]; Supplementary Figure S4B). The other configurations show no consistent preference for explaining the first examples, with GPT-OSS-120B at medium reasoning effort instead leaning towards last- matches (Supplementary Figure S4B and Supplementary Table S7). Similarly to people, models find simpler, often incomplete rules that nonetheless capture meaningful aspects of the underlying task structure. These findings suggest that the latest generation of more capable LRMs make more systematic errors than was previously found [44, 45].
Furthermore, we find that the distributions of error types show a similar correlation with performance. As in humans, attribute misselection (see § 2.4 for the taxonomy) is the most frequent error type across models, and its prevalence increases with performance. It accounts for 63.66% (94% HDI [60.63%, 67.16%]) of Qwen3.5-27B errors and 61.58% (94% HDI [58.70%, 64.62%]) of GPT-OSS-120B errors at high reasoning effort, compared with 49.11% (94% HDI [46.79%, 51.56%]) and 42.63% (94% HDI [39.60%, 45.38%]) at medium and low effort, respectively. Lower-performing models instead produce a larger proportion of unclassifiable errors (Supplementary Figure S4A and Supplementary Table S8), suggesting less systematicity.
Finally, we investigate whether meaningful structural differences can differentiate human and LRM errors based on their synthesized programs. We trained classifiers to predict whether an error was made by a human or an LRM based on simple, linearized program-level features representing primitive count changes (see Figure 5D and § 4.7 for a description of these features). We fit logistic L2 regression, explainable boosting machine (EBM) and gradient boosting classifiers. To avoid the confound of between-task program differences, we report ROC-AUC among each shared task’s human and LRM errors and average across tasks. All three classifiers discriminate human from pooled LRM errors above chance and perform comparably (within-task ROC-AUC = 0.65–0.69; Figure 5E). Classifiers trained separately on humans versus each LRM configuration reliably discriminate LRM configurations which are less accurate than people (GPT-OSS-120B medium and low: 0.68–0.77 across classifiers, all 95% CI lower bounds ≥ 0.59). However, discrimination is weaker and not reliably above chance for configurations which are more accurate than people (Qwen3.5-27B: 0.58–0.61; GPT-OSS-120B high: 0.57–0.63; Supplementary Table S10). Consistent with coarse feature representations, inspection of fitted coefficients and EBM shape functions indicates that this signal is distributed across many weak cues rather than a few diagnostic operations. We find that edits that remove higher-order primitives (e.g., fork, chain, mapply) or insert arithmetic (e.g., double, increment) are relatively important features for predicting LRM errors. Arithmetic insertions appear in about 5% of human errors versus 10–16% for each LRM configuration. Conversely, skipping a single drawing or recoloring step (e.g., fill, paint) is more characteristic of human errors (about 12% vs. 2–9%). More generally, we find that people and LRMs tend to fail on different tasks: per-task success rates agree only weakly between people and each LRM configuration (Spearman , 95% CIs within [0.20, 0.44], across the 400 ARC-AGI-1 training tasks), substantially less than the LRM configurations which tend to agree with one another (). Although human and machine errors share broad patterns in simplicity bias, rule-type distributions, and error categories, measurable differences remain in the specific primitive operations that are used to reproduce their respective errors and problem-level difficulty profiles. This suggests that potentially different inductive biases support rule-search in people versus reasoning language models. Together, these results illustrate how BCSS can support investigating similarities and differences in human and machine reasoning through a shared symbolic representational space.
Discussion
Symbolic models are psychologically motivated, interpretable, and directly comparable, but are difficult to scale beyond tasks with approximately enumerable hypothesis spaces. We address this scaling challenge by proposing BCSS. BCSS uses an LLM as a proposal distribution over program edits to recover, for individual responses, the rules people and machines may have applied in an open-ended domain. Three regularities emerge from the human data. Programs explaining errors are systematically shorter than the target rule, beyond the shortening attributable to the procedure itself. Many modeled errors appear to be explained by mental rules that fit some but not all of the training examples. And the most common error type is attribute misselection: computing a target parameter from the wrong feature. Together, these patterns are consistent with satisficing [35]: settling on a rule that fits enough of the evidence rather than searching for one that fits all of it.
Large reasoning models show the same broad signatures as people, graded by resources. In GPT-OSS-120B, lowering reasoning effort produces errors explained by simpler programs and fewer partial fits to the evidence, paralleling human results. In combination with other research demonstrating a positive correlation between reasoning token allocation and thinking time from human responses in the H-ARC dataset [46], these results suggest that the patterns observed here reflect limits on search in both humans and LRMs. Yet, people and models fail on largely different tasks, and their error programs remain partially discriminable, pointing to differing inductive biases. Together, our results hint at a resource-rational characterization of reasoning in open-ended domains [47], though establishing this formally would require explicit modeling of the costs people and machines trade off [34].
Modeling incorrect responses was central to the results we report here. Correct answers in isolation reveal little about how a solution was reached, whereas understanding systematic errors can enhance our explanations of underlying mechanisms [48, 49, 50]. The order asymmetry among partial fits illustrates this. Exact Bayesian inference is order-invariant, whereas sequential sampling algorithms predict a dependence on presentation order [51], and because all examples remain visible throughout solving, degraded encoding is an unlikely cause of the observed patterns. An alternative is that task designers tend to place simpler examples first, so that a rule ignoring a later exception fits the early examples by construction. However, whether people form hypotheses from a limited subset of the examples, or form them more broadly but only verify them sequentially against part of the evidence remains an open question for future work.
Like much prior work in cognitive science [52, 53, 54], BCSS treats inferences as deviations from a normative model. Here the normative standard takes the form of a reference program expressed in a DSL, with a locality prior keeping error programs within a few line edits of it. This strategy for understanding incorrect inferences has several precedents, for example in accounts of arithmetic errors as “bugs” in a correct procedure [27, 28], and approaches that attribute rules to individuals from a pre-specified candidate set of strategies, distractors, or grammar-generated hypotheses [5, 38, 55, 56]. Conditioning an LLM proposal distribution on the observed response removes the need to enumerate hand-design that set. However, one caveat is that each reference program is only one of many equivalent encodings of the target rule, so edit distance and relative length are defined with respect to that encoding. The training references were written independently of this work [30], and the main effects replicate on the evaluation split, whose references were obtained by an entirely different procedure (§ 4.2.1).
BCSS has further limitations. First, it captures about half of non-trivial errors, and because edits replace lines rather than add them, errors that elaborate on the target rule are likely underrepresented in the analyzed set. It is also possible that these responses require more edits, or simply reflect more idiosyncratic behavior not governed by coherent rules. Second, the DSL was designed for solving ARC rather than for cognitive modeling. Although many of the operations in it are cognitively plausible (e.g., geometric transformations, object groupings, and other spatial predicates), some primitives (e.g., type coercion) have less clear cognitive counterparts. Consequently, the correspondence between cognitive complexity and DSL program complexity is unlikely to be one-to-one. Library learning, potentially guided by language [29, 57, 58] and an objective that encodes agreement with human thinking time, could instead automatically infer them. Relatedly, several distinct programs often reproduce the same response, and we do not yet weight them, for instance by complexity or cognitive cost [34]. Third, programs are proposed and classified by LLMs from the same family, whose editing distributions may shape the categories we observe. Finally, BCSS identifies which rule an agent may have executed. Although we recover aspects of process-level signatures such as order-effects, our pipeline does not explicitly model how an agent arrived at that rule. Addressing process-level considerations will require integrating dynamic behavioral metrics such as response time, eye-tracking as well as carrying out controlled experimental conditions to further illuminate how human reasoners search and revise candidate rules. For LRMs, reasoning traces potentially offer a complementary and analogous window.
BCSS builds on multiple strands of research from AI and cognitive science. Recent advances have led to impressive developments in science automation [59, 60, 61, 62, 63], with particular developments of this line of research to engineer systems for automated cognitive model discovery [64, 65, 66, 67, 68, 69]. Here, we develop similar tools to scale longstanding methods of psychological inquiry which have often faced a scalability bottleneck [70, 71, 72]. Inspired by seminal work such as Newell & Simon’s [73] early attempts to develop theories of human problem-solving by analyzing detailed, individual-level behavioral protocols, we instantiate a similar but automated approach using AI tools. By using neural models as proposal distributions over discrete program spaces, BCSS makes symbolic, individual-level accounts of behavior feasible in open-ended domains. Further, our approach demonstrates that cognitive footprints can be decoded from the structure of synthesized programs that explain behavior in different systems, offering a common ground for comparing how people and black-box AI systems solve complex problems. We see exciting prospects for the application of BCSS to other domains, shedding light on both human and machine cognition.
Methods
Data and filtering
We use the H-ARC dataset to understand how people reason on abstract rule-induction problems [25, 26]. The dataset consists of output grid responses across 800 distinct ARC-AGI-1 few-shot visual rule inference problems from over 1700 online participants. For the scope of the human results reported here, we focus our modeling efforts on a total of 723 ARC-AGI-1 tasks (362 tasks from the training split; 361 from the evaluation split), yielding a total of 6261 unique incorrect output grids across 1387 participants and 7402 attempts.
We pre-processed the H-ARC dataset by identifying all near-misses and “trivial” errors, corresponding to 15.6% (1,222/7,809) of the full set of incorrect unique grids, and filtered them out under a noise tolerance criterion. Qualitative analyses revealed that for larger grids, “errors” could often be plausibly attributed to procedural mistakes (i.e., mis-clicks; Supplementary Figure S1). Given our modeling approach and goals, capturing this kind of noise within programs would not be informative about rule inference since we aim to understand conceptual mistakes. As such, we apply a noise tolerance criterion such that for any “incorrect” response grid of ten or more cells, we filter it out as a “near-miss” if it matches the correct test output grid with up to two mismatched cells; of the 353 unique near-miss grids we found, 51.3% differed from the correct output by a single cell and 47.9% by exactly two (Supplementary Figure S2, Supplementary Table S1). Additionally, participants sometimes copied the test input grid (7.9% of unique responses) or training inputs and outputs (3.5% of unique responses). For test-input copies we applied the same tolerance criterion—for instance, if a participant produced a output identical to the test input grid modulo one or two incorrect cells, we filtered it out as an identity error, whereas copies of a training example were removed only on an exact match. For grids smaller than ten cells, we apply a strict filter by only removing grids that are exact copies of the test input or training examples. Finally, we remove all “empty” grids (1.9% of unique responses) because this grid state corresponds to the initial state of the test output grid in our task interface (i.e., a blank response).
Domain-specific language and reference programs
We model rules as programs in a domain-specific language (DSL) for ARC. To do so, we leverage an expressive open-source DSL which was released alongside 400 human-written reference programs implementing the correct rule for each task [30]. The DSL consists of 160 primitive functions spanning object detection and manipulation (e.g., objects, ofcolor, recolor, etc), grid transformations (e.g., rot90, fill, etc), spatial queries (e.g., height, adjacent, manhattan, etc), selection and ranking (e.g., argmax, mostcommon, etc), and higher-order compositional and programmatic operations (e.g., compose, fork, branch, etc).
The DSL is typed over a small set of object-centric types built from Python immutable types: grids as nested tuples of integers, objects as nested frozen sets of colored cells, etc. Additionally, the DSL contains a small set of primitive constants: a ten color palette, boolean values, and directional constants.
Programs in this DSL are written in A-normal form (ANF; [74]), a canonical form in which every non-trivial sub-expression is named via direct assignment, and function arguments must be atoms (i.e., variables or constants). The programs also respect static single assignment, where every variable is named once and only once. Although there are many possible expressions of correct programs for a given task rule, we focus on using the 400 human-written ones available from the open-source DSL for the training split. For the remaining 400 tasks from the evaluation split, we use the shortest synthesized solver for tasks where we successfully synthesized at least one solver.
Synthesizing additional reference programs for full coverage
As described above, the DSL we utilize in this work was released with 400 human-written solvers for half of the publicly available tasks from ARC-AGI-1, excluding the 400 tasks in the evaluation split. To obtain full coverage of H-ARC incorrect attempts, we prompted an LLM to generate synthetic solvers for the remaining 400 tasks. We successfully generated solvers for 361 tasks of the 392 tasks from the evaluation split of ARC-AGI-1 that have incorrect responses. To do so, we simply prompted Gemini Pro 3.1 (Supplementary Figure S6) using retrieval-augmented generation [75] with five in-context examples of other tasks and corresponding human-written solvers from the training split. For each queried evaluation task, we sampled five independent responses, and selected the shortest program from the set of valid and correct DSL programs for each task. Incorrect responses on the remaining 31 evaluation tasks (326 of the 3,892 non-trivial unique errors in that split) could not be modeled and are excluded, which leaves the 6,261 unique errors (2,695 training, 3,566 evaluation) analyzed here.
Symbolic model synthesis
Behavior-Conditioned Symbolic Synthesis. BCSS involves automating inference over behavioral outputs to produce programs representing the rules that an agent may have used to generate its responses. To so do, we leverage an LLM by prompting it to synthesize a program representation of each incorrect attempt. We sample from the LLM structured outputs of line edits restricted to a JSON schema, and its responses on task grids, a complete DSL reference, the reference program for the in-context task, and an incorrect output grid produced by a human participant or an LLM (Supplementary Figure S5). The LLM is instructed to generate a restricted set of edits to the in-context reference program such that the resulting program would produce the observed response when executed on the test input grid. The edits consist of in-place edits to referenced lines of code from the reference program. The LLM is instructed to generate syntactically valid, ANF lines of code written exclusively using the DSL. For each unique incorrect response, we collect 5 independent samples with edit budgets of 2, 4 and 8 lines, yielding a total of 15 samples per error. For human responses, we obtained a total of 93,855 program samples (40,365 for the training set split responses; 53,490 for the evaluation set split responses) across 723 tasks. Four training-split errors (five attempts) were missing from the synthesis run and thus were not sampled; they are retained in the error counts reported here. Samples were generated using Gemini 3 Pro for the training split and Gemini 3.5 Flash for the evaluation split, with temperature set to 1.0, reasoning budget set to high, and default parameters otherwise.
For each sampled structured output of edits, we programmatically replace each tagged line with its corresponding edited line, and verify that the resulting programs respect ANF syntactic constraints and exclusively use DSL operators. Any edits that violate the instructions are rejected as invalid and discarded. Often, editing one or multiple lines has the side-effect of disconnecting variables in the reference program from the computation graph processing the input into the output grid. Thus, each program is rewritten using a custom parser which removes these pieces of code after applying the edits. We then execute each program using the test input grid for the corresponding task and verify its output against the target error grid. We only accept a program as an explanation for a given error if it produces an output that is equivalent to the closest match from the pool of generated programs under the same noise tolerance criterion as for filtering errors (§ 4.1 for more details). Concretely, we apply a noise tolerance criterion such that for any synthesized program, if its output grid has ten or more cells, we consider it a valid model of the observed response if it matches the target response grid with up to two mismatched cells. For example, for a given target output grid, if the best program in the pool of 15 samples produces a correctly sized output that is at least 10 cells large, but differs by a single cell from the target, then any other program from that pool must produce outputs that are equally accurate to be accepted. After filtering, we obtained a total of 8,246 unique valid programs explaining human errors (3,690 for the training set split responses; 4,556 for the evaluation set split responses; see § 4.6 for machine errors). Across modeled errors, we computed the per-error average minimum edits: the minimum number of edits averaged across errors. This serves as an upper bound on the shortest path in line-edit space from the reference program for explaining a given response.
Exhaustive depth-1 program enumeration. To characterize the space of programs adjacent to human-written ARC solutions, we exhaustively enumerated all single-edit mutations of the 400 training split reference programs. Because the programs are written in ANF and respect static single assignment, the set of syntactically well-formed edits is finite and enumerable. We adapted code from previous work [76] where a stochastic search algorithm over DSL program mutations in the ARC setting was proposed as a form of data augmentation to train deep learning models for program induction.
Each program is parsed into an abstract syntax tree and a concrete type is inferred for every variable by propagating the types of the task’s first training input through the DSL’s primitive signatures. For every assignment statement in the program, we then enumerated four kinds of type-correct edits:
Argument replacement: substituting one argument of the call for any other in-scope variable or primitive constant of the same type
Function replacement: substituting the called primitive for any primitive with a compatible output type
Right-hand-side replacement: substituting the entire expression for any type-correct application of a DSL primitive to in-scope variables and constants
Argument replacement with variable introduction: inserting a new assignment immediately before a given statement and passing its result as the argument, with downstream variables renumbered accordingly
This procedure yielded a total of 16,661,032 candidate programs. We then executed each candidate on all input grids of its parent task. To do so, we wrote a custom line-by-line interpreter with a 0.25s per-statement timeout and a 1000-cell bound on grid and intermediate-state size, discarding executions that raised, diverged, or exceeded the size bound. Only candidates producing a valid output grid on both the training and test inputs were retained. This left 9,422,295 executable programs (56.6% of those enumerated). Enumeration and execution were parallelized across 52 CPU cores and completed in approximately 16 hours on the NYU high-performance computing cluster. Programs were determined to match a human error under the same noise tolerance criterion as LLM-generated programs.
Error classification
To characterize the kinds of inferential shortcuts that people and machines utilize to solve ARC tasks, we developed a taxonomy of error types based on comparisons between programs that explain human and machine responses, and the reference programs from which they were adapted (see Table 1). Although some errors only require single-line edits, or even simpler node-level edits such as swapping an argument or a primitive function for another, the majority of synthesized programs that reproduce human responses required multi-line edits. Such program differences are difficult to programmatically interpret since their consequences can span different parts of the computation graph, and as a result they cannot be easily attributed to a particular high-level cognitive category without semantic interpretation or the contextual information from the training examples. Thus, we developed a precise taxonomy of seven error categories which we believed would capture meaningful cognitive constructs, and prompted an LLM (Supplementary Figure S7) to categorize each program from the final filtered collection of programs that match human or machine responses.
For each program, we prompt an LLM to return a structured JSON schema output containing one of eight possible labels: seven cognitive categories and an “unclassifiable” catch-all category. In addition to definitions for each category, we include the following elements in the model’s prompt: the reference program for the given task, the task training input and output grids, the synthesized program, and the synthesized program execution trace on the training and test input grids. This final portion explicitates how the program behaves on the training examples, as well as what test output grid it produces (i.e., a human or machine response). Concretely, the classifier must assign a label to structural differences between the reference program and the error program while taking into account the task data. For instance, a sub-tree that computes height relative to color count might instead be collapsed to a constant integer value in the error program, which coincides with some training examples. We sampled 5 independent classifications per program, yielding 18,450 classifications for the 3,690 programs that explain human errors on the training split, and 41,655 classifications for the 8,331 programs that explain machine errors (4,115 for Qwen3.5-27B; 4,720, 13,800, and 19,020 for GPT-OSS-120B at high, medium, and low reasoning effort, respectively). All samples were generated using Gemini 3.1 Pro, with temperature set to 1.0, reasoning budget set to high, and default parameters otherwise.
Pipeline validation
To assess the validity and sources of bias of BCSS, we manually constructed a validation set of synthetic errors by mechanically producing targeted, simple edits on the full set of human-written and synthetic reference programs. For each of the eight categories, we defined a function that implements a limited set of mutation operators, applies them to a given input program, and returns new, modified programs that fall under the syntactic constraints of the corresponding error category. In total, this yielded 2,375 synthetic errors across the 8 categories. We then manually verified a small random subset of errors for each category (444 in total) to obtain a target of ∼20 errors per category, balanced between the human-written solvers of the training split and the LLM-synthesized solvers of the evaluation split. Manual verification is required to ensure semantic interpretation of errors conditioned on task grids. In total, we obtained 148 synthetic errors which we use for validation, with attribute misselection containing 19 programs, and sequencing only 9. We refer to the errors generated and verified using the procedure just described as validation errors/programs. Validation errors refer to the grids obtained by executing the validation programs.
We passed the validation errors through BCSS end-to-end to evaluate how accurately the predicted categories matched the gold-labeled ones. Overall, this resulted in an end-to-end category agreement of 64.9%, 75.0%, and 76.4% at edit budgets of , respectively. We evaluated these budgets because in this synthetic setting, program edit distances are known by construction—124 of the 148 errors are single-line edits, 141 are reachable within two edits, and all 148 within three—though in practice the true edit distance is unknown. To understand what governs this performance and why accuracy plateaus around 76%, we decompose the pipeline into three potential sources of loss: classification error (labeling), search failure (recovery), and semantic shift under alternative programs (aliasing).
We first isolated labeling by running the classifier directly on the validation programs, which establishes an accuracy ceiling of 85.1%. Classifications across five runs per program were highly consistent at ∼95% agreement, and we used modal labels as the unit of analysis. Second, we evaluated whether search failure accounts for the drop from this ceiling. We found that given an adequate edit budget, it did not account for drop in accuracy. Increasing from 1 to 3 raised recovery from 84.5% to 99.3% (95.3% at = 2), successfully finding a candidate program for all but one validation error grid. Instead, the remaining gap is driven almost entirely by what we call “program aliasing”: functionally equivalent programs that are written differently. For instance, consider a task in which a set of isolated cells must be expanded into a square of a new color. An error in which the central cell keeps its original color can be produced by restricting the recoloring to background cells surrounding the center cell (an injected scope restriction), but also by omitting the step that fills the square’s interior so that only the ring around the cell is recolored, a program that the classifier instead labels as a partial program. Across all budgets, BCSS recovered the exact injected program in roughly two-thirds of cases (62%–65%). When search found an alternative, aliased program instead, the classifier still identified the injected category in 57.3% of cases (remaining essentially flat at 52.6%–59.4% across budgets). Thus, program aliasing rather than search failure or classifier noise, accounts for the primary drop between the 85.1% ceiling and end-to-end performance.
Evaluating machine errors
As described in § 2.5, we evaluate two reasoning-tuned LLMs on the training split of ARC-AGI-1 to study their error behavior: Qwen3.5-27B [42] in thinking mode and GPT-OSS-120B [43] at high, medium, and low reasoning effort. We evaluate each model configuration across ten random seeds using a standard evaluation prompt (see Supplementary Figure S8), on all 416 test inputs of the 400 tasks (4,160 responses per configuration). H-ARC participants attempted only the first test input of the 14 tasks that have two so we only consider responses to that attempt in our analyses. For Qwen3.5-27B, we follow the recommended sampling settings for general tasks in thinking mode2 with a temperature of 1.0, a top- value of 0.95, a top- value of 20, a minimum- value of 0.0, a presence penalty of 1.5, and a repetition penalty of 1.0. For GPT-OSS-120B, we follow GPT-OSS’s official vLLM example3 and use a temperature of 1.0, a top- value of 1.0, disabled top- filtering, a minimum- value of 0.0, a presence penalty of 0.0, and a repetition penalty of 1.0 across all three reasoning-effort settings. We set a maximum output budget of 65,536 generated tokens, including reasoning and the final answer, for both models. For GPT-OSS-120B, we control reasoning behavior through the reasoning-effort setting without imposing a separate reasoning-token budget. For Qwen3.5-27B, we additionally impose a reasoning-token budget of 49,152 tokens to limit time and resources spent on inference. If Qwen3.5-27B reaches this budget before ending its reasoning, the end-of-thinking marker </think> is appended to its current output, terminating the reasoning block and allowing final-answer generation within the remaining output budget. We observe that only 4.35% (181/4160) of responses reach the model’s maximum reasoning budget.
A response is counted as correct only if generation finishes within the maximum token budget, and the extracted grid exactly matches the target in both dimensions and cell values. We parse predicted grids by selecting the last valid group of bracketed grid rows, accepting spaces or commas between entries, and falling back to parsing Python/JSON-style nested lists. A valid grid must be nonempty, rectangular, and contain integer values from 0 to 9. Completed responses from which no valid grid can be extracted are classified as parsing errors. Both unfinished responses and completed responses with no valid grid remain included in the overall accuracy calculations but are excluded from subsequent analyses (see Supplementary Table S6). We find that Qwen3.5-27B generates 806 incorrect responses, of which 689 contain valid but incorrect grid predictions. For GPT-OSS-120B, 742/1153 incorrect responses at high reasoning effort contain valid but incorrect grid predictions, compared with 2002/2040 at medium reasoning effort and 3059/3119 at low reasoning effort. After removing incorrectly formatted grid outputs, we applied the additional filtering steps described in § 4.1 (see Supplementary Table S5 for the full per-configuration breakdown).
We pass each incorrect grid prediction to BCSS as described in § 4.3. As with human responses, we collect 5 independent program samples at edit budgets of 2, 4, and 8 lines, yielding a total of 15 samples per machine error. Unlike for humans, identical grids from different seeds were sampled separately, but programs were subsequently deduplicated and pooled per unique error. In total, we obtain 8,625 program samples for errors from Qwen3.5-27B and 9,780 samples for GPT-OSS-120B at high reasoning effort, 26,070 at medium reasoning effort, and 37,365 at low reasoning effort. At the time when we ran machine evaluations, Gemini 3 Pro was unfortunately not available anymore on the Gemini API. Thus, samples are generated using Gemini 3.5 Flash, with the temperature set to 1.0, the reasoning budget set to high, and all other parameters left at their default values. All programs are subsequently verified and filtered as described in § 4.3. After filtering, we obtain 823 unique valid programs for Qwen3.5-27B’s errors, 944 programs describing GPT-OSS-120B’s errors at high reasoning effort, 2,760 at medium reasoning effort, and 3,804 at low reasoning effort. For comparisons across models and humans, we restrict our analyses in § 2.5 to the 56 tasks on which humans and every model configuration make at least one error, thereby avoiding task difficulty as a confounding factor. On these 56 tasks, Qwen3.5-27B produces 112 errors captured by 486 programs, whereas GPT-OSS-120B produces 128 errors described by 608 programs at high reasoning effort, 221 errors described by 1,008 programs at medium reasoning effort, and 206 errors described by 916 programs at low reasoning effort. On the same tasks, humans produce 144 first-attempt errors described by 450 programs. All statistical models in § 4.8 are fit separately for each model configuration and for humans, using only the first-attempt errors and programs from these 56 tasks unless stated otherwise (§ 4.8.1).
Predicting human versus machine errors
We train classifiers on program features to predict whether an error was made by a human or a large reasoning model (LRM). As described in § 4.3, DSL programs are written in A-normal form as sequences of primitive function calls. We describe each error program by how it differs from its reference program (Figure 5D).
Primitive delta vectors record the signed difference in call counts between the error-producing and reference programs, with one entry per DSL primitive. For example, if one rotate180 call is replaced by hmirror and all other calls remain unchanged, the vector contains −1 for rotate180, +1 for hmirror, and zeros elsewhere.
We fit L2-regularized logistic regression, explainable boosting machines (EBMs), and histogram gradient boosting classifiers on the extracted features. Each candidate program explaining an observed error provides one training example. Source-balanced weights allocate half the total training weight to humans and half to LRMs, with the LRM share divided equally among the four LRM configurations. Within each source, the weight is divided equally among tasks, then among errors within each task, and finally among candidate programs within each error. Logistic regression features are scaled by training-fold standard deviations without centering; boosting models use raw values.
Classifiers are trained on errors from all tasks, which maximizes the number of examples available for training. Because classifiers are fit at the candidate-program level, we obtain error-level predictions by averaging probabilities across all candidate programs explaining a given error. We use nested cross-validation with five outer and five inner folds, keeping all errors and candidate programs from a task together at both levels to avoid information leakage. The outer and inner folds are defined over the 56 shared tasks on which humans and every LRM configuration make at least one modeled error (§ 4.6) and the remaining 304 tasks are included as training data in every outer and inner fold. Thus, reported metrics are computed only on the out-of-fold predictions for the 56 shared tasks. In the outer loop, one fold of shared tasks is held out for testing and the remaining shared tasks are partitioned into the five inner folds for hyperparameter selection. We select hyperparameters to maximize mean within-task ROC-AUC across the five inner validation folds. The winning hyperparameter configuration is then used to refit the classifier on the full outer training split and generate out-of-fold predictions on the held-out outer test fold. We search the inverse L2 regularization strength for logistic regression, pairwise interactions and boosting rounds for EBM, and maximum leaf nodes and iterations for histogram gradient boosting. Pre-processing and interaction selection use only data from the corresponding training fold.
We score predictions using within-task ROC-AUC, treating LRM errors as the positive class and averaging equally across tasks. We pool the outer test-fold predictions across all five outer splits, with 95% percentile confidence intervals estimated from 10,000 bootstrap resamples of whole tasks holding the out-of-fold predictions fixed. To examine whether discriminability differs across LRM configurations, we also fit each classifier to distinguish human errors from those of a single configuration, using the same features, hyperparameter grids, task folds, shared evaluation tasks, and bootstrap resamples, with the LRM half of the training weight assigned entirely to that configuration.
Statistical analyses
All analyses were performed using a Bayesian framework. Continuous outcomes were modeled as Gaussian linear mixed-effects regressions, using Bambi v0.17.2 for the main simplicity model and PyMC v5.28.5 directly for error category simplicity model; categorical outcomes were modeled as non-centered hierarchical categorical regressions directly in PyMC v5.28.5 with a softmax inverse link. All posteriors were approximated using the No-U-Turn Sampler (NUTS; [77]) via the JAX/NumPyro backend [78], with a target acceptance rate set to 0.9. We ran inference using 4 chains, with warm-up samples specified for each model below. For all models, we use non-centered parameterizations of the random effects, which is standard practice for inference using NUTS. Convergence was assessed by the rank-normalized split- statistic and bulk/tail effective sample size, using thresholds of and ESS per parameter; we additionally verified that no divergent transitions occurred. For every parameter of interest we report the posterior mean (or median for probabilities) and the 94% highest-density interval (HDI).
Simplicity bias
To determine whether error-producing programs were simpler than baseline programs synthesized for correct outputs under an identical synthesis procedure, we fit a hierarchical Bayesian linear model. Let denote the relative line-count change of the -th synthesized program for unique error in task , and let indicate the treatment condition (where for control and for treatment). We compute the relative line-count change as , where is the line count of the synthesized program, and is the line count of the reference program from which it was adapted. We modeled the outcome as:
We include task-level random intercepts to account for baseline variations in program length across ARC tasks, and nested per-error intercepts to handle the fact that each unique error contributes multiple candidate programs that are not independent observations. Thus, the model answers whether synthesized programs that fit incorrect human or machine responses, tend to be more or less complex than reference programs subjected to an identical procedure modulo the conditioned response data. We dummy-coded the condition factor with the control baseline as the reference level, so that the slope is directly interpretable as the treatment-minus-control contrast. Analyses were restricted to tasks present in both conditions to ensure the random-effects structure was identified. We report the treatment and control mean relative line-count change and the treatment–control contrast corresponding to additional shortening attributable to conditioning synthesis on incorrect responses.
Control programs were synthesized with the same model as the treatment programs they are compared against: Gemini 3 Pro for human errors on the training split, and Gemini 3.5 Flash for human errors on the evaluation split and for machine errors (§ 4.3, § 4.6). For the comparison with machines (Figure 5B), fits are restricted to the 55 of the 56 matched tasks (§ 4.6) that have a control program. The four machine fits share a single control set of 150 programs, and the human fit uses 132 control programs.
To relate the simplicity bias to performance, we propagated posterior uncertainty in the treatment-minus-control contrasts into the correlation. For each of 20,000 posterior draws, we took the relative shortening from each of the five fits, computed its Pearson correlation with the five accuracy rates, and fit an ordinary least-squares slope of shortening on accuracy. We report the posterior median and 94% HDI of both quantities. Accuracy is treated as fixed, so these intervals reflect uncertainty in the shortening estimates only.
Finally, we placed weakly informative priors on all coefficients and variance components:
The residual standard deviation was assigned Bambi’s default data-scaled prior.
Rule types and order effects
Given error-producing programs, we modeled how often such programs also produce the correct output grids for at least one training example but not all (i.e., partial-match programs), or all the training examples but not the test example (i.e., alternative rules). To do so, we fit a hierarchical categorical Bayesian model, and each program was run on all the training examples of its associated task and labeled according to the outputs it produced. Let be the rule-type classification of program for unique error , where categories: “alternative rule” (set as reference, ), “neither” () and “partial-match” (). We modeled the likelihood and linear predictors as:
where is the error-specific probability vector on the -simplex, shared by all programs belonging to error . As in the previous model, we include per-error intercepts since there are often multiple programs per error, each with a possibly different behavior on the training examples. The “alternative rule” category is fixed at zero as the reference, since the softmax link is translation invariant. Finally, we placed weakly informative priors on the coefficient and variance components:
With this model, we seek to understand how likely it is that an incorrect rule is modeled by a program that also satisfies some or all of the constraints imposed by the training examples human or machine reasoners observed. Accordingly, we define the attempt-weighted rule-type prevalence over the empirical error pool as a derived quantity of the model parameters:
where , is defined as above, is the number of attempts that produced error (the number of participant attempts for human data; for model configurations, the number of sampled responses), and is the total number of unique errors. Evaluating at each posterior draw , , yields a Monte Carlo sample from the posterior distribution of the prevalence, which we summarize by its median and 94% HDI. Because each error contributes its realized random effect weighted by , corresponds to the expected rule-type prevalence one would observe when sampling incorrect attempts uniformly at random from the human or machine incorrect response distributions.
To characterize which output structure is reproduced by the partial-match rules, we fit a second instance of the same hierarchical categorical model, restricted to the partial-match programs, on ordering categories of the reproduced training pairs. For a partial-match program that reproduces of the training outputs, we assign an ordering category based on the positions of the matched pairs: “first-” when they form a block anchored at the first training example (similar to a primacy pattern), “last-” when the block is anchored at the last example (similar to a recency pattern), “contiguous” when the run is interior (length , not including the first or last example), and “other” for a single interior match or a scattered, non-contiguous set. Let be the ordering category of partial-match program for unique error , where and “contiguous” is the reference (). The likelihood, softmax link, per-error random effects , and priors are exactly as in the rule-type model above; only the observation set (partial-match programs), the category set, and hence the number of categories change. We summarize the same attempt-weighted prevalence by its posterior median and 94% HDI, with the weights renormalized over the errors that admit at least one partial-match program. The resulting prevalences are therefore conditional on an error being explained by a partial-match program, and describe, among such attempts, how the reproduced training examples are distributed by position. For both models, we report only the attempt-weighted prevalences , expressed as percentages.
Error categories
Similarly to the rule-type model described above, we modeled the prevalence of the different error types among the programs that reproduce incorrect human or machine attempts on ARC tasks. For each program, we first obtain five classification samples using our LLM classifier (§ 4.4). Given this pool of samples, each unique response may have between 5 and label samples, where is the largest number of unique error programs retained for a single error. Let be the error category assigned by sample to program of unique error , where categories: “attribute misselection” (set as reference, ), “boundary precision” (), “sequencing” (), “operator confusion” (), “partial program” (), “scope restriction” (), “transformation misselection” (), and “unclassifiable” (). We modeled,
The difference with the rule-type model is the additional program-level random effect , nested within the per-error effect , and the number of categories. We include both effects because each unique error may be explained by several distinct programs, and each program is in turn labeled multiple times by the classifier. The per-error intercepts absorb systematic differences between errors, and the program-level intercepts absorb variation between the programs generated for the same error together with the labeling noise across the replicated samples. As before, “attribute misselection” is fixed at zero as the reference, since the softmax link is translation invariant. We placed the same weakly informative priors on the coefficient and both variance components:
As with the rule-type model, we define the attempt-weighted error-category prevalence over the empirical error pool as a derived quantity of the model parameters:
where , is the error-level linear predictor evaluated with the program-level effects held at zero (), is the number of attempts that produced error (the number of participant attempts for human data; for model configurations, the number of sampled responses), and is the total number of unique errors. Evaluating at each posterior draw , , yields a Monte Carlo sample from the posterior distribution of the prevalence, which we summarize by its median and 94% HDI. Holding evaluates each error at its typical program; the prevalences therefore describe the error category one would expect for a typical program explaining an incorrect attempt sampled uniformly at random from the human or machine incorrect response distributions. Similarly to rule types, we report the attempt-weighted category prevalences .
Relative length change and error categories
Having established that error programs are simpler than the reference programs they are adapted from (§ 4.8.1), we next asked how this relative shortening varies with reference program length, task difficulty, and error type. We fit a Bayesian linear regression predicting the relative line-count change of each error program, with reference program line count and empirical task success rate as continuous covariates and the error category as an additional term. Since we only ran error classification on programs fitting incorrect responses to tasks from the training split, this model is only run on that subset of programs. Let denote the relative line-count change (defined as in § 4.8.1) of the -th error program for unique error in task , let be the H-ARC empirical success rate of task , and let be the line count of the task’s reference program. Both covariates were mean-centered, with and denoting their sample means. We modeled the outcome as:
As in the previous models, we include per-error random intercepts to account for the several candidate programs that may explain a single unique error. Centering the covariates makes the expected relative length change at the mean success rate and mean reference length. The slopes and capture how relative shortening covaries with task difficulty and reference program complexity, respectively.
The vector is a per-category effect on relative length change for the categories of § 4.8.3, and is a fixed -vector of category weights for program . Instead of collapsing the classifier’s five samples per program to a single majority-vote label, we carry that uncertainty into the covariate. Let be the program’s category count vector across its classification samples. We set to the Dirichlet–multinomial posterior mean under a symmetric Dirichlet() prior,
so that the category effect enters the mean linearly as the probability-weighted average . We used a minimal smoothing constant () so that a program with unanimous classifications loads its weight almost entirely on its own category, keeping on the true per-category magnitude scale, with non-unanimous programs distributing weight across their sampled categories. Treating as fixed data (rather than as a latent allocation informed by the length outcome) keeps the category term a plain linear-Gaussian covariate with unimodal geometry. Unlike the corresponding latent-mixture formulation which failed to converge, this model samples cleanly under NUTS.
The seven cognitive categories share a sum-to-zero prior, so that each of their effects is interpretable as a deviation from their common grand mean . The “unclassifiable” category is held out of this constraint and given its own free effect. We set the model this way since this category collects short, degenerate programs whose shortening is several times larger than that of any cognitive category, and pooling it into the sum-to-zero prior would stretch that scale and distort the grand-mean anchor. We placed weakly informative priors on the coefficients and variance components:
Because of the custom sum-to-zero structure on the category effects, this model was written and sampled directly in PyMC rather than through Bambi. We report the change in relative shortening per additional reference-program line () and per unit of task success rate () in percentage points. We also report each category’s predicted relative length change () and its deviation from the grand mean (), and the predicted relative length change at a given reference length ().
Supplementary information
This article includes Supplementary Information: Supplementary Notes S1–S6, Supplementary Figures S1–S8 and Supplementary Tables S1–S10.
AI use statement
We acknowledge the use of generative AI tools for minor language revisions, such as grammar checks and reformulations. In addition, we used generative AI tools such as Claude Code and Codex to assist with code implementation, review and testing. SL and PM reviewed all code generated by Claude Code and Codex and tested it for correctness. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI.
Funding
SL was supported by a combination of NSF Award 2121101 to TMG, the NYU Training Program in Computational Neuroscience (NIH R90 DA060339), and the MacCracken Fellowship at NYU. PM was supported by the LMU–NYU Cooperation Program during his visit to NYU and Princeton University making this collaboration possible. AKJ is supported by a Natural and Artificial Minds (NAM) Fellowship from the Scully Peretsman foundation at Princeton University. We thank Google for generously supporting this research with a financial gift to BML by providing Google Cloud Credits to utilize Gemini models for research.
Data availability
This study uses the publicly available H-ARC dataset [26]. All relevant data will be made publicly available with a permanent DOI upon publication.
Code availability
Code for BCSS and all analyses is available at https://github.com/Le-Gris/bcss.
Author contribution
SL, AKJ, BML and TMG conceptualized the study. SL developed the methodology, implemented the software and performed the main analyses. JJ contributed to preliminary analyses. PM implemented the machine evaluations, and PM and SL analysed the machine results. SL created the figures. BML and TMG supervised the project, and AKJ provided guidance throughout. SL wrote the manuscript, with PM drafting the machine evaluation and analysis sections. AKJ, BML and TMG provided substantial feedback and edits, and all authors reviewed the manuscript.
Supplementary Information
This supplement reports (i) data for understanding the noise-tolerance criterion (§S1), (ii) per-split versions of every quantity reported as pooled results in the main text (§S2–§S4), (iii) per-model values for the human–machine comparison (§S5), and (iv) the prompts used at each stage of BCSS (§S6).
Filtering policy
Our filtering policy (§ 4.1) treats an incorrect response of ten or more cells as a near-miss, and filters it out as approximately correct, when it differs from the correct test output in at most two cells. We apply the same criterion for synthesized programs that approximately reproduce an error (§ 4.3). We provide some rationale for this two-cell constant in the pipeline.
Supplementary Figure S1 shows representative responses within the tolerance interval. In each case the participant’s grid and the best-matching program output differ in one or two isolated cells against an otherwise identical structure.
Supplementary Figure S2 quantifies the same point over the full dataset. Panel A gives the coverage of incorrect attempts as a function of the tolerance : coverage is the empirical CDF of , the minimum number of mismatched cells between any of the fifteen sampled programs for an error and the target response, restricted to size-matched outputs. At , coverage is 52.5% on the training split and 46.2% on the evaluation split. The curve keeps climbing beyond the cutoff, so the threshold is conservative. Panel B shows that the share of mismatched cells that are scattered singletons (no mismatched neighbor) falls from % at to % by and to % beyond . Residuals of three or more cells are therefore likely to represent objects or other structures, but more analyses are needed to confirm this. In any case, accepting them would include programs implementing a different transformation, as opposed to a slightly imprecise one. Panel C shows the same cutoff from the filtering side: the distribution of cell-level distances between incorrect responses and the correct test output, with the removed near-miss data ( cells) shaded.
Supplementary Table S1 reports what the trivial-error filter removes.

Figure S1. Responses within the two-cell noise tolerance. (A–C) Three representative ARC tasks. Left: the training examples and the test input shown to participants (the test output is hidden with a grey square and “?”). Right: for three response grids per task, the output of the best-matching synthesized program (left column) beside the participant responses it is accepted as reproducing (remaining columns); hatched cells indicate that fewer than two distinct responses share that program. The fraction to the left of each row gives the number of participants producing that response out of all participants who attempted the task (red, incorrect; green, correct). Within each row the grids differ in at most two isolated cells (surrounded by white) against the best-match program output.

Figure S2. The two-cell cutoff separates procedural noise from rule-level divergence. (A) Coverage of non-trivial incorrect responses as a function of the tolerance , attempt-weighted, shown separately for the training (blue) and evaluation (grey) splits. Circles mark the value used throughout (52.5% and 46.2% respectively). (B) Share of mismatched cells that are scattered singletons—no mismatched cell among their four neighbors—as a function of , the mismatch count of the best-scoring candidate program. The steep fall immediately past the cutoff shows that small residuals are isolated misclick-like cells, whereas higher values of indicate structured regions/objects. (C) Distribution of cell-level distances between incorrect responses and the correct test output, per split. Grey bars ( mismatched cells) are the near-misses removed by the filter.
| Training | Evaluation | Combined | |
| By incorrect responses (attempts) | |||
| Incorrect responses | 4,293 | 5,156 | 9,449 |
| Filtered (all trivial) | 808 (18.8%) | 891 (17.3%) | 1,699 (18.0%) |
| Near-miss ( correct) | 189 (4.4%) | 251 (4.9%) | 440 (4.7%) |
| Test-input copy (identity) | 368 (8.6%) | 377 (7.3%) | 745 (7.9%) |
| Training-example copy | 130 (3.0%) | 197 (3.8%) | 327 (3.5%) |
| Empty grid | 121 (2.8%) | 66 (1.3%) | 187 (2.0%) |
| By unique incorrect grids | |||
| Unique incorrect grids | 3,220 | 4,589 | 7,809 |
| Filtered (all trivial) | 525 (16.3%) | 697 (15.2%) | 1,222 (15.6%) |
| Near-miss ( correct) | 140 (4.3%) | 213 (4.6%) | 353 (4.5%) |
| Test-input copy (identity) | 209 (6.5%) | 282 (6.1%) | 491 (6.3%) |
| Training-example copy | 88 (2.7%) | 142 (3.1%) | 230 (2.9%) |
| Empty grid | 88 (2.7%) | 60 (1.3%) | 148 (1.9%) |
Table S1. Table S1 Trivial-error filtering by split (§ 4.1). Counts and percentages of incorrect responses removed before modeling, under the noise tolerance criterion: i) shapes must match, ii) grids of ten or more cells tolerate up to two mismatched cells, iii) smaller grids must match exactly and iv) copies of a training example are removed only on an exact match at any size. Percentages are over the full pre-filter pool of responses of each split.
Modeling coverage and program yield by split
Supplementary Table S2 gives the per-split decomposition of the coverage results reported pooled in § 2.1. training reference programs are hand-written whereas evaluation reference programs are model-synthesized (§ 4.2.1). Edits were sampled with Gemini 3 Pro on training and Gemini 3.5 Flash on evaluation.
Table S2 Modeling coverage and program yield by split. Unless specified, all reported numbers refer to errors modeled under the two-cell tolerance criterion. Attempt-weighted quantities weight each unique error by the number of participants who produced it. Exact-match reproduced is the percentage of errors with at least one program reproducing the grid with zero mismatched cells. Effective edits are counted after the parser removes causally irrelevant lines. Four training-split errors were missing from the synthesis run, so program samples cover 2,691 of the 2,695 training errors.
| Training | Evaluation | Combined | |
| Reference program | hand-written | synthesized | — |
| Proposal model | Gemini 3 Pro | Gemini 3.5 Flash | — |
| Tasks modeled | 362 | 361 | 723 |
| Unique non-trivial errors | 2,695 | 3,566 | 6,261 |
| Incorrect attempts | 3,485 | 3,917 | 7,402 |
| Program samples collected | 40,365 | 53,490 | 93,855 |
| Tasks with error modeled | 304 | 304 | 608 |
| Errors reproduced | 1,274 (47.3%) | 1,507 (42.3%) | 2,781 (44.4%) |
| Attempts reproduced | 1,827 (52.4%) | 1,809 (46.2%) | 3,636 (49.1%) |
| Participants modeled | 548 | 611 | 1,159 |
| Exact-match coverage (errors) | 39.9% | 35.5% | 37.4% |
| Exact-match coverage (attempts) | 45.9% | 39.9% | 42.7% |
| Total filtered programs | 3,690 | 4,556 | 8,246 |
| Programs per error () | |||
| Programs per error (median [IQR]) | 2 [1, 4] | 3 [1, 4] | 2 [1, 4] |
| Programs per error (max) | 10 | 12 | 12 |
| Effective edits / program (, ) | 3.30, 2.00 | 3.63, 2.16 | 3.49, 2.10 |
| Effective edits, per-error min. () |
Table S2.
Rule types and order effects by split
Supplementary Table S3 reports the per-split values behind the pooled rule-type and order-effect estimates of § 2.3. Estimates are attempt-weighted population marginals from the hierarchical categorical model (§ 4.8.2). The prevalence of programs consistent with at least one training example is close across splits (23.8% versus 21.3%), whereas the structured share of partial matches is substantially higher on training (62.5% versus 48.5%), with the largest proportion coming from the first- tier.
Program length, reference complexity and error categories
Supplementary Figure S3 and Supplementary Table S4 report the full output of the program length model of § 4.8.4, which regresses relative length change on reference-program line count and task success rate with a soft error-category term.

Figure S3. Relative length change as a function of reference complexity and error category. Partial residuals of relative length change against reference-program line count, with the posterior mean slope (dashed) and its 94% HDI (band). Annotations mark the predicted shortening at , 14 and 28 lines.
| Category | [94% HDI] | % shorter [94% HDI] | |
| Unclassifiable † | −0.369 [−0.395, −0.344] | 0.000 | +46.1 [+44.2, +48.0] |
| Partial program | −0.231 [−0.264, −0.198] | 0.000 | +32.3 [+29.1, +35.6] |
| Transformation misselection | −0.121 [−0.150, −0.091] | 0.000 | +21.3 [+18.4, +24.0] |
| Attribute misselection | +0.014 [−0.007, +0.034] | 0.893 | +7.8 [+6.2, +9.3] |
| Scope restriction | +0.045 [−0.006, +0.098] | 0.947 | +4.6 [−1.1, +10.4] |
| Operator confusion | +0.080 [+0.041, +0.118] | 1.000 | +1.2 [−2.8, +5.1] |
| Sequencing | +0.102 [+0.039, +0.164] | 0.999 | −1.1 [−7.9, +6.0] |
| Boundary precision | +0.111 [+0.065, +0.159] | 1.000 | −2.0 [−7.3, +3.0] |
Table S4. Error-category effects on relative program length (training split). is the deviation from the grand mean of the seven cognitive categories, which share a sum-to-zero prior; negative means more relative shortening than the average error type. The unclassifiable catch-all (†) is held outside that constraint. “% shorter” is the absolute per-category prediction at mean success rate and mean reference length, summed per posterior draw (grand-mean shortening ); a negative value means the error programs are longer than the reference. programs over 1,274 errors.
Beyond the growing relative shortening predicted as a function of target rule complexity (§ 2.2), our results suggest substantial variation in relative shortening across error categories (Supplementary Table S4). Around a grand-mean shortening of 9.2% (94% HDI [7.6, 10.8]), the seven cognitive categories span roughly 34 percentage points.
At one end, partial-program errors produce the greatest simplification with programs 32.3% shorter than the target rule (94% HDI [29.1, 35.6]; , 94% HDI [-0.264, -0.198]). Conversely, boundary precision errors yield no reliable simplification: although they deviate credibly from the grand mean (, 94% HDI [0.065, 0.159]), their predicted length is 2.0% longer than the reference program, with an interval that includes zero (94% HDI [-3.0, 7.3]).
Overall, we find that our analysis recovers an interpretable hierarchy between the categories. This result suggests that when people make simple localized errors such as boundary precision or operator confusion, the rules they infer, as modeled by our programs, approximate much of the structure of the target rule. Conversely, when people only partially execute the rule or select the wrong transformation, their approximations are coarser, which manifests as substantially shorter programs.
Machine errors
Machine errors
Supplementary Table S5 reports what the trivial-error filter removes from each model configuration’s incorrect responses before modeling, as the counterpart to Supplementary Table S1 for human responses. Supplementary Tables S6–S8 then report the extended LRM evaluation and associated error analyses on the full set of errors for comparison with the restricted-subset results reported in the main text. Figure S4 provides additional visualization of error analyses on the subset of tasks where both models and people produced modeled incorrect responses. Supplementary Table S9 reports within-task and pooled ROC-AUC for the human-versus-LRM source classifiers on this same subset and for each classifier and Supplementary Table S10 reports classifiers trained on humans versus each LRM configuration separately (§ 4.7).

Figure S4. Language models display human-like error category profiles, but order effects are weaker and less consistent than in humans. All panels report values corresponding to the subset of 56 tasks where evaluated LRMs and people both generated at least one incorrect response. Human responses are restricted to first-attempt comparisons. (A) Prevalence of each error category among modeled incorrect machine errors. As with human results, an LLM classifier assigned every program one of seven cognitive categories or unclassifiable, conditioned on the reference program, the task grids, the synthesized program and its execution trace (Table 1 for definitions). Bars give the posterior median of the attempt-weighted prevalence derived from a hierarchical Bayesian categorical model, and error bars the 94% HDI. The categories are mutually exclusive. (B) Conditional on an attempt being explained by a partial-match program, we show the positions of the training examples it reproduces: the first- or last- example(s) shown, a contiguous block of at least two examples that does not include the first or last examples, or any other configuration. In both plots, bars give the posterior median attempt-weighted prevalence and error bars the 94% HDI.
Table S5 Trivial-error filtering by model configuration (§ 4.1). Incorrect model responses with a valid grid removed before modeling, under the same noise tolerance criterion applied to human responses (Supplementary Table S1), counted by response (top) and by unique incorrect grid (bottom). Percentages are over each configuration’s pre-filter total in the same block.
| Qwen3.5-27B | GPT-OSS-120B | |||
| high | mid | low | ||
| By incorrect responses (attempts) | ||||
| Responses with a valid grid | 689 | 742 | 2,002 | 3,059 |
| Filtered (all trivial) | 114 (16.5%) | 90 (12.1%) | 264 (13.2%) | 568 (18.6%) |
| Near-miss ( correct) | 73 (10.6%) | 54 (7.3%) | 84 (4.2%) | 61 (2.0%) |
| Test-input copy (identity) | 22 (3.2%) | 12 (1.6%) | 110 (5.5%) | 354 (11.6%) |
| Training-example copy | 19 (2.8%) | 24 (3.2%) | 70 (3.5%) | 153 (5.0%) |
| By unique incorrect grids | ||||
| Unique incorrect grids | 536 | 562 | 1,522 | 2,059 |
| Filtered (all trivial) | 79 (14.7%) | 57 (10.1%) | 148 (9.7%) | 229 (11.1%) |
| Near-miss ( correct) | 57 (10.6%) | 42 (7.5%) | 72 (4.7%) | 47 (2.3%) |
| Test-input copy (identity) | 13 (2.4%) | 7 (1.2%) | 48 (3.2%) | 110 (5.3%) |
| Training-example copy | 9 (1.7%) | 8 (1.4%) | 28 (1.8%) | 72 (3.5%) |
Table S5.
| Qwen3.5-27B | GPT-OSS-120B | |||
| high | mid | low | ||
| Accuracy (%) | 80.63 | 72.28 | 50.96 | 25.02 |
| Tasks with error | 145 | 173 | 284 | 344 |
| Incorrect responses | 806 | 1,153 | 2,040 | 3,119 |
| with a valid grid | 689 | 742 | 2,002 | 3,059 |
| Attempts | 575 | 652 | 1,738 | 2,491 |
| Unique errors | 457 | 505 | 1,374 | 1,830 |
| Program samples | 8,625 | 9,780 | 26,070 | 37,365 |
| Attempts modelled | 296 (51.5%) | 355 (54.4%) | 1,031 (59.3%) | 1,565 (62.8%) |
| Errors modelled | 197 (43.1%) | 236 (46.7%) | 714 (52.0%) | 991 (54.2%) |
| Filtered programs | 823 | 944 | 2,760 | 3,804 |
Table S6. Table S6 Evaluation and modeling yield by model configuration (across all tasks with errors). Accuracy is the mean over ten seeds on the 400 training tasks of ARC-AGI-1. Attempts correspond to every incorrect response that survived filtering; unique errors deduplicate identical grid outputs. Attempts modeled and errors modeled are the counts with at least one verified program reproducing the error within the two-cell noise tolerance criterion—with coverage as a percentage of attempts and of unique errors, respectively.
| Qwen3.5-27B | GPT-OSS-120B | |||
| high | mid | low | ||
| Simplicity bias (relative length change vs. correct-output control) | ||||
| −0.147 | −0.153 | −0.211 | −0.245 | |
| 94% HDI | [−0.185,−0.112] | [−0.187,−0.119] | [−0.236,−0.185] | [−0.268,−0.221] |
| 1.00 | 1.00 | 1.00 | 1.00 | |
| Rule type (all modeled attempts) | ||||
| Neither | 59.1 [56.5,61.8] | 60.2 [57.8,62.8] | 78.6 [77.2,79.8] | 84.2 [83.2,85.4] |
| Partial match | 33.6 [31.0,36.2] | 33.0 [30.5,35.4] | 19.6 [18.3,20.8] | 14.7 [13.6,15.7] |
| Alternative rule | 7.3 [5.6,8.9] | 6.7 [5.3,8.2] | 1.8 [1.3,2.3] | 1.1 [0.7,1.5] |
| Order effects (conditional on a partial match) | ||||
| (errors) | 146 | 160 | 291 | 388 |
| First- (prefix run) | 28.7 [24.0,33.7] | 41.3 [37.2,45.6] | 40.1 [36.8,43.2] | 37.6 [34.1,41.2] |
| Last- (suffix run) | 27.9 [22.4,33.0] | 19.7 [15.3,23.6] | 25.4 [22.2,28.6] | 26.2 [22.9,29.6] |
| Contiguous (interior) | 3.8 [1.9,6.4] | 2.3 [0.9,4.1] | 1.3 [0.4,2.3] | 1.7 [0.7,3.0] |
| Other | 39.3 [34.0,45.0] | 36.5 [32.0,41.1] | 33.2 [29.6,36.7] | 34.4 [30.4,38.0] |
Table S7. Table S7 Simplicity bias, rule types and order effects by model configuration (across all tasks with errors). is the posterior mean relative shortening of error programs against correct-output controls, where negative values indicate shorter programs. Rule types and order effects are attempt-weighted population marginals from the hierarchical categorical model (§ 4.8.2); posterior median and 94% HDI, in percent. A program is a partial match if it reproduces at least one but not all training outputs, an alternative rule if it reproduces all of them yet still fails the test example, and neither otherwise. Order effects are conditional on a partial match and describe which training examples the program reproduces, ordered as models or participants saw them.
Table S8 Error categories by model configuration (across all tasks with errors). Attempt-weighted population marginals from the hierarchical categorical model (§ 4.8.3); posterior median and 94% HDI, in percent. Error categories are the seven cognitive types of § 2.4 and unclassifiable (†) is the catch-all category. is the number of errors with at least one validly classified program; one medium-effort and two low-effort errors (7 and 3 programs) had no valid classification sample and are excluded, hence 713/989 rather than 714/991 in Supplementary Table S6.
| Qwen3.5-27B | GPT-OSS-120B | |||
| high | mid | low | ||
| (errors) | 197 | 236 | 713 | 989 |
| Attribute misselection | 61.0 [58.7,63.5] | 58.4 [55.9,60.9] | 45.7 [44.3,47.3] | 38.0 [36.6,39.4] |
| Transformation misselection | 13.0 [11.4,14.5] | 12.6 [11.1,14.3] | 11.0 [10.0,12.1] | 10.1 [9.3,10.9] |
| Partial program | 7.2 [5.4,8.8] | 10.2 [8.2,12.2] | 10.8 [9.6,12.0] | 9.5 [8.4,10.6] |
| Scope restriction | 3.2 [2.3,4.3] | 3.9 [3.0,4.7] | 2.7 [2.4,3.1] | 3.0 [2.5,3.6] |
| Operator confusion | 0.9 [0.1,2.3] | 3.2 [2.0,4.7] | 2.7 [1.9,3.5] | 3.6 [2.7,4.5] |
| Boundary precision | 3.4 [2.4,4.4] | 1.3 [0.6,2.0] | 1.8 [1.3,2.3] | 1.5 [1.1,1.8] |
| Sequencing | 1.6 [0.8,2.3] | 0.5 [0.0,1.1] | 2.3 [1.8,2.9] | 0.9 [0.5,1.3] |
| Unclassifiable † | 9.6 [8.2,10.9] | 9.7 [8.3,11.2] | 22.8 [21.8,24.1] | 33.4 [32.3,34.5] |
Table S8.
Table S9 Human-versus-LLM source prediction (§ 4.7). Out-of-fold ROC-AUC on the 811 errors (144 human, 667 LLM) from the 56 matched tasks, for classifiers trained on primitive delta vectors from the errors on all tasks (651 human, 2,138 LLM). LLM errors are the positive class. Within-task ROC-AUC is computed among the errors of each task and averaged with equal task weight. Pooled ROC-AUC is computed over all errors at once. Point estimates with 95% percentile confidence intervals from 10,000 bootstrap resamples of whole tasks, shared across rows.
| Within-task ROC-AUC | Pooled ROC-AUC | |
| L2 logistic regression | 0.69 [0.63,0.76] | 0.64 [0.59,0.70] |
| EBM | 0.68 [0.61,0.74] | 0.68 [0.62,0.74] |
| Hist. gradient boosting | 0.65 [0.57,0.73] | 0.68 [0.62,0.74] |
Table S9.
| LLM errors | Within-task ROC-AUC | |||||
| Acc. (%) | Train | Eval. | Logistic L2 | EBM | Grad. boosting | |
| Qwen3.5-27B | 80.63 | 197 | 112 | 0.61 [0.52,0.69] | 0.59 [0.49,0.69] | 0.58 [0.48,0.67] |
| GPT-OSS (high) | 72.28 | 236 | 128 | 0.60 [0.51,0.69] | 0.57 [0.47,0.66] | 0.63 [0.53,0.72] |
| GPT-OSS (mid) | 50.96 | 714 | 221 | 0.68 [0.59,0.76] | 0.69 [0.61,0.76] | 0.68 [0.60,0.76] |
| GPT-OSS (low) | 25.02 | 991 | 206 | 0.74 [0.67,0.82] | 0.77 [0.69,0.84] | 0.77 [0.68,0.84] |
Table S10. Human-versus-LLM source prediction for each LLM configuration (§ 4.7). Each row reports classifiers trained on primitive delta vectors to distinguish human errors from those of a single configuration, with the same features, weighting scheme, task folds and evaluation tasks as Supplementary Table S9. Train includes the configuration’s modeled errors on all tasks, and Eval includes those on the 56 matched tasks, on which out-of-fold within-task ROC-AUC is computed. Every row shares the same 651 human training errors and 144 human evaluation errors. Acc. is mean single-attempt accuracy (humans: 59.90%). Accuracy, reasoning effort and the number of training errors vary together across configurations, so differences between rows should not be attributed to accuracy alone. Point estimates with 95% percentile confidence intervals from 10,000 bootstrap resamples of whole tasks.
Prompts
Supplementary Figures S5–S8 reproduce the three prompt templates used by the pipeline. Placeholders in double braces are substituted at run time with the corresponding task or program text.
Figure S5. Error-program synthesis prompt (§8.3). The model is given the full DSL specification, the task grids, the reference program implementing the target rule, and the respondent’s incorrect output grid, and returns a structured set of line edits under a fixed JSON schema.
Figure S6. Reference-program (solver) synthesis prompt (§ 4.2.1). Used to generate reference programs for evaluation-split tasks, which have no hand-written solver. The prompt is retrieval-augmented with five solved tasks and their human-written solvers from the training split.
Solver synthesis prompt
prompts/synthesize_solver_program.md
You are an expert program synthesis researcher. Your task is to write a program in a domain-specific language (DSL) that solves a visual pattern transformation task. The task consists of input–output grid pairs that demonstrate a consistent transformation rule. You must discover this rule and express it as a DSL program.
DSL Reference
The following DSL is available for writing programs:
{{dsl_code}} DSL signatures + docstrings
{{in_context_examples}} retrieved demo solvers; empty under zero-shot
Target Task
Write a solver program for the following task. Study the training examples to discover the transformation rule, then express it as a DSL program.
{{target_examples}} train pairs, test input; test output hidden by default
{{task_description}} optional human NL description of the rule
Instructions
Write a DSL program that transforms the input grid to the output grid for the target task above. The program should generalize the transformation rule demonstrated by the training examples.
Syntax rules:
Each line of code must follow this form:
x[k] = function_name(arg1, . . . , argL)for line and arguments.The final output variable must be named
O.Arguments can only be: variable names from previous lines in the program, constants from the DSL, or function names from the DSL passed as references (e.g.
mapply(outbox, x2)where outbox is passed by name, not called).Do not nest function calls — that is, do not use a function call as an argument to another function (e.g.
f(g(x))is not allowed).You cannot use any other operations than the ones in the DSL.
Do not use nested functions like a def or nested function calls like
func(func(x)). Instead, compose functions using existing DSL functions.Do not use lambda expressions.
Do not use Python primitive operations like
*,//,+,-and others. For those operations, use the corresponding DSL functions, when available.Do not directly create native Python data structures like lists, tuples, dictionaries, sets.
Do not use double variable assignments or use of
*operator for arguments.Do not perform direct assignment of numbers. Instead, use constants where appropriate.
The input grid is always available as the variable
I.
Response Format
Respond only with a JSON object of this form:
{
"program": "x1 = ...\nx2 = ...\nO = ..."
}
program: The complete solver program body, with lines separated by
\n. Each line assigns to a variable. The last meaningful assignment should produce the output grid, assigned toO.
Figure S7. Error-classification prompt (§ 4.4). The classifier sees the reference program, the task grids, the synthesized error program, and a side-by-side comparison of their behavior on the training examples, and returns one of the seven cognitive categories or the unclassifiable catch-all.
Error classification prompt (abbreviated)
prompts/classify_errors.md
You are an expert computational cognitive scientist studying human errors on visual program synthesis tasks (ARC — Abstraction and Reasoning Corpus). For each participant error, we have synthesized a program that reproduces the participant’s incorrect output. Your task is to characterize the cognitive error that this program represents using a structured taxonomy.
DSL Reference
{{dsl_code}} signatures + docstrings
Task Data
{{ref_program}} ground-truth program, solves the task correctly
{{ref_examples}} training examples, Input → Output
Error Program
{{error_program}} synthesized to reproduce the participant’s incorrect output
{{training_comparison}} side-by-side ground-truth vs. error-program outputs, per training example
{{participant_output}} error program’s output on the test input
{{participant_description}} optional participant NL
[one framing sentence per input section omitted, plus the two subheadings under Error Program]
Error Category Taxonomy
Classify the error into exactly one of the following 8 categories. Read all definitions before deciding.
Attribute Substitution — replaces a quantity or feature that must be computed with a different, often simpler or hardcoded one.
Scope Restriction — the correct computation, applied correctly, but only to a subset of the targets it should cover.
Operator Confusion — right class of operation, wrong member of it (rotating left instead of right, mirroring on the wrong axis).
Transformation Substitution — right targets and spatial structure, but an operation from an entirely different family.
Compositional Sequencing Error — all the right operations, wrongly ordered (sequential_ordering) or wrongly bound to targets (parallel_binding).
Partial Program — the pipeline is incomplete; the steps present are correct, some are missing entirely.
Boundary Precision Error — correct quantity, off by one (or a small constant), not explainable by overfitting to training examples.
Unclassifiable / Ad-hoc Rule — an idiosyncratic rule bearing little structural relationship to the ground truth.
Quick Decision Guide
Ask these questions in order. Stop at the first “yes”:
Is the participant’s program a proper subset of the ground-truth pipeline, with missing steps representing unperformed computation? →
partial_programIs the correct operation applied, but not everywhere it should be? →
scope_restrictionIs a different attribute/feature (e.g. colour, dimension) used in place of the computed/inferred one, perhaps resorting to a simpler strategy or hardcoded value? →
attribute_substitution
Sub-check: if the difference is only ±1, do the training examples discriminate between the two values? If no → attribute_substitution (hardcoded_constant). If yes → boundary_precision (skip to 7).
Is the right operation class identified, but the wrong member of that class selected (e.g. wrong rotation direction, wrong mirroring axis)? →
operator_confusionAre the correct targets identified, but is a transformation from an entirely different computational family applied? →
transformation_substitutionAre all the right operations identified and applied to all targets, but assembled in the wrong order or bound to the wrong targets? →
compositional_sequencingIs the computation off by exactly ±1 from the correct value, and do the training examples rule out overfitting? →
boundary_precisionDoes the participant’s program bear little structural resemblance to the ground truth? →
unclassifiable
Response Instructions
Follow these three steps in order:
Step 1—Rule descriptions.
correct_rule_descriptionanderror_rule_description: what the ground-truth program and the error program each do, as a rule a human could articulate. Cognitive/perceptual terms, never DSL ones; describe behavior, not syntax; translate colour codes to colour names.Step 1b—Operation domain. Name the mental operation at stake (“spatial placement”, “object counting”, “colour attribution”), focusing on what the participant failed at.
Step 2—Category selection. Walk through the Quick Decision Guide above; select the single best-fitting
error_category.Step 3—Category-specific fields. Fill in ONLY the fields for that category. All fields for other categories must be null.
[ vocabulary guidance paragraph omitted ]
[the source’s Field-to-Category Mapping table — 8 rows naming the 2–5 non-null fields required for each category — is omitted here for space; the fields are listed in src/category_definitions.py]
Response Format
Respond only with a JSON object containing all fields below; each field carries a one-line description of what to write there. Fields not belonging to the selected category must be null.
{
"correct_rule_description": "...", "error_rule_description": "...",
"operation_domain": "...", "error_category": <one of the 8 categories above>,
<28 category-specific fields, non-null only for the selected category>
}
[≈30 category-specific fields follow in the source, each with a one-line description and an explicit null option]
Figure S8. Evaluation prompt (§ 4.6). Used to collect machine responses on ARC tasks for comparison with the human data. The model is shown the training pairs and asked to complete the held-out pair.
References
- [1]Feldman, J. Minimization of Boolean complexity in human concept learning. Nature 407, 630–633 (2000). URL https://doi.org/10.1038/35036586.
- [2]Fodor, J. A. The Language of Thought (Harvard University Press, Cambridge, MA, 1975).
- [3]Newell, A. Physical symbol systems. Cogn. Sci. 4, 135–183 (1980). URL https://onlinelibrary.wiley.com/doi/10.1207/s15516709cog0402_2.DOI
- [4]Braine, M. D. & O’Brien, D. P. Mental Logic (Psychology Press, 1998).
- [5]Goodman, N. D., Tenenbaum, J. B., Feldman, J. & Griffiths, T. L. A rational analysis of rule-based concept learning. Cognitive science 32, 108–154 (2008).
- [6]Sloman, S. Causal Models: How People Think about the World and Its Alternatives (Oxford University Press, New York, 2005).
- [7]Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to Grow a Mind: Statistics, Structure, and Abstraction. Science 331, 1279–1285 (2011).DOI
- [8]Lake, B. M., Salakhutdinov, R. & Tenenbaum, J. B. Human-level concept learning through probabilistic program induction. Science 350, 1332–1338 (2015).DOI
- [9]Yang, Y. & Piantadosi, S. T. One model for the learning of language. Proceedings of the National Academy of Sciences 119, e2021865119 (2022). URL https://www.pnas.org/doi/abs/10.1073/pnas.2021865119.DOI
- [10]Rule, J. S. et al. Symbolic metaprogram search improves learning efficiency and explains rule learning in humans. Nat. Commun. 15, 6847 (2024). URL https://www.nature.com/articles/s41467-024-50966-x.DOI
- [11]Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science (MIT Press, Cambridge, MA, 2001).
- [12]Lake, B. M. & Baroni, M. Human-like systematic generalization through a meta-learning neural network. Nature 623, 115–121 (2023). URL https://www.nature.com/articles/s41586-023-06668-3.DOI
- [13]Webb, T., Holyoak, K. J. & Lu, H. Emergent analogical reasoning in large language models. Nature Human Behaviour 7, 1526–1541 (2023). URL https://www.nature.com/articles/s41562-023-01659-w.DOI
- [14]Binz, M. et al. A foundation model to predict and capture human cognition. Nature 644, 1002–1009 (2025). URL https://www.nature.com/articles/s41586-025-09215-4.DOI
- [15]McClelland, J. L. et al. Letting structure emerge: Connectionist and dynamical systems approaches to cognition. Trends in Cognitive Sciences 14, 348–356 (2010). URL https://linkinghub.elsevier.com/retrieve/pii/S1364661310001245.
- [16]Griffiths, T. L., Lake, B. M., McCoy, R. T., Pavlick, E. & Webb, T. W. Whither symbols in the era of advanced neural networks? Trends in Cognitive Sciences 0 (2026). URL https://www.cell.com/trends/cognitive-sciences/abstract/S1364-6613(26)00028-8.DOI
- [17]Tenenbaum, J. B. Solla, S., Leen, T. & Müller, K. (eds), Rules and Similarity in Concept Learning. Advances in Neural Information Processing Systems (eds Solla, S., Leen, T. & Müller, K.), Vol. 12, 59–65 (MIT Press, 2000). URL https://proceedings.neurips.cc/paper_files/paper/1999/hash/86d7c8a08b4aaa1bc7c599473f5ddd da-Abstract.html.
- [18]Piantadosi, S. T., Tenenbaum, J. B. & Goodman, N. D. The logical primitives of thought: Empirical foundations for compositional cognitive models. Psychological Review 123, 392–424 (2016). URL https://doi.org/10.1037/a0039980.
- [19]Lake, B. M. & Piantadosi, S. T. People Infer Recursive Visual Concepts from Just a Few Examples. Computational Brain & Behavior 3, 54–65 (2020). URL http://link.springer.com/10.1007/s42113-019-00053-y.arxiv.org/abs/1904.08034
- [20]Sablé-Meyer, M., Ellis, K., Tenenbaum, J. & Dehaene, S. A language of thought for the mental representation of geometric shapes. Cognitive Psychology 139, 101527 (2022). URL https://www.sciencedirect.com/science/article/pii/S0010028522000639.
- [21]Zhou, Y., Feinman, R. & Lake, B. M. Compositional diversity in visual concept learning. Cognition 244, 105711 (2024). URL https://www.sciencedirect.com/science/article/abs/pii/S0010027723003451.DOI
- [22]Chen, M. et al. Evaluating Large Language Models Trained on Code (2021). Preprint at https://arxiv.org/abs/2107.03374.
- [23]Austin, J. et al. Program Synthesis with Large Language Models (2021). Preprint at https://arxiv.org/abs/2108.07732.
- [24]DeepSeek-AI et al. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024). Preprint at https://arxiv.org/abs/2406.11931.
- [25]Chollet, F. On the measure of intelligence (2019). Preprint at https://arxiv.org/abs/1911.01547.
- [26]LeGris, S., Vong, W. K., Lake, B. M. & Gurreksis, T. M. A Comprehensive Behavioral Dataset for the Abstraction and Reasoning Corpus. Scientific Data 12, 1380 (2025). Preprint at https://arxiv.org/abs/2409.01374.
- [27]Brown, J. S. & Burton, R. R. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2, 155–192 (1978). URL https://www.sciencedirect.com/science/article/pii/S0364021378800044.DOI
- [28]VanLehn, K. Mind Bugs: The Origins of Procedural Misconceptions Mind Bugs: The Origins of Procedural Misconceptions (The MIT Press, Cambridge, MA, US, 1990).
- [29]Acquaviva, S. et al. Communicating Natural Programs to Humans and Machines. Advances in Neural Information Processing Systems, Vol. 35, 3731–3743 (Neural Information Processing Systems Foundation, Inc., 2022).arxiv.org/abs/2106.07824
- [30]Hodel, M. Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation (2024). Preprint at https://arxiv.org/abs/2404.07353.
- [31]Mc McElreath, R. Statistical Rethinking: A Bayesian Course with Examples in R and Stan 2nd edn (Chapman and Hall/CRC, Boca Raton, FL, 2020). URL https://www.taylorfrancis.com/books/mono/10.1201/9780429029608/statistical-rethinking-richard-mcelreath.
- [32]Chater, N. & Vitányi, P. Simplicity: A unifying principle in cognitive science? Trends in Cognitive Sciences 7, 19–22 (2003). URL https://linkinghub.elsevier.com/retrieve/pii/S1364661302000050.pubmed.ncbi.nlm.nih.gov/12517354
- [33]Feldman, J. The Simplicity Principle in Human Concept Learning. Current Directions in Psychological Science 12, 227–232 (2003). URL https://doi.org/10.1046/j.0963-7214.2003.01267.x.
- [34]Kendall, C. & Oprea, R. On the complexity of forming mental models. Quantitative Economics 15, 175–211 (2024). URL https://www.econometricsociety.org/doi/10.3982/QE2264.DOI
- [35]Simon, H. A. Rational choice and the structure of the environment. Psychological Review 63, 129–138 (1956).pubmed.ncbi.nlm.nih.gov/13310708
- [36]Simon, H. A. Models of Bounded Rationality (MIT Press, Cambridge, Mass, 1982).
- [37]Carpenter, P. A., Just, M. A. & Shell, P. What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review 97, 404–431 (1990). URL https://doi.org/10.1037/0033-295X.97.3.404.
- [38]Kunda, M., Soulières, I., Rozga, A. & Goel, A. K. Error patterns on the Raven’s Standard Progressive Matrices Test. Intelligence 59, 181–198 (2016). URL https: //www.sciencedirect.com/science/article/pii/S0160289616300149.DOI
- [39]Kahneman, D. & Frederick, S. in Representativeness Revisited: Attribute Substitution in Intuitive Judgment Heuristics and Biases: The Psychology of Intuitive Judgment (eds Griffin, D., Kahneman, D. & Gilovich, T.) 49–81 (Cambridge University Press, Cambridge, 2002).
- [40]Guo, D. et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633–638 (2025). URL https://www.nature.com/articles/s41586-025-09422-z.DOI
- [41]OpenAI et al. OpenAI o1 System Card (2024). Preprint at https://arxiv.org/abs/2412.16720.
- [42]Qwen Team. Qwen3.5: Towards native multimodal agents (2026). URL https://qwen.ai/blog?id=qwen3.5.
- [43]OpenAI. gpt-oss-120b & gpt-oss-20b model card (2025). Preprint at https://arxiv.org/abs/2508.10925.
- [44]LeGris, S., Vong, W. K., Lake, B. M. & Gureckis, T. M. H-ARC: A robust estimate of human performance on the Abstraction and Reasoning Corpus benchmark (2024). URL http://arxiv.org/abs/2409.01374.
- [45]Mitchell, M., Palmarini, A. B. & Moskvichev, A. Comparing Humans, GPT-4, and GPT-4V on Abstraction and Reasoning Tasks. Proceedings of the LLM-CAP Workshop, AAAI-24 (2024). Preprint at https://arxiv.org/abs/2311.09247.
- [46]de Varda, A. G., D’Elia, F. P., Kean, H., Lampinen, A. & Fedorenko, E. The cost of thinking is similar between large reasoning models and humans. Proceedings of the National Academy of Sciences 122, e2520077122 (2025). URL https://www.pnas.org/doi/abs/10.1073/pnas.2520077122.
- [47]Lieder, F. & Griffiths, T. L. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences 43, e1 (2020). URL https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/resourcerational-analysis-understanding-human-cognition-as-the-optimal-use-of-limited-computational-resources/5868660D9A1D1E A7A1EECE217D392F4A.
- [48]Tversky, A. & Kahneman, D. Judgment under Uncertainty: Heuristics and Biases. Science 185, 1124–1131 (1974). URL https://doi.org/10.1126/science.185.4157.1124.
- [49]Marcus, G. F. et al. Overregularization in Language Acquisition. Monographs of the Society for Research in Child Development 57, i–178 (1992).pubmed.ncbi.nlm.nih.gov/1518508
- [50]Khemlani, S. & Johnson-Laird, P. N. Theories of the syllogism: A meta-analysis. Psychological Bulletin 138, 427–457 (2012).
- [51]Sanborn, A. N., Griffiths, T. L. & Navarro, D. J. Rational approximations to rational models: Alternative algorithms for category learning. Psychological Review 117, 1144–1167 (2010).DOI
- [52]Chomsky, N. Aspects of the Theory of Syntax 50 edn (The MIT Press, 1965). URL https://www.jstor.org/stable/j.ctt17kk81z. jstor.j.ctt17kk81z.
- [53]Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (Henry Holt and Co., Inc., USA, 1982).
- [54]Anderson, J. R. The Adaptive Character of Thought The Adaptive Character of Thought (Lawrence Erlbaum Associates, Inc, Hillsdale, NJ, US, 1990).
- [55]Siegler, R. S. Three aspects of cognitive development. Cognitive Psychology 8, 481–520 (1976). URL https://doi.org/10.1016/0010-0285(76)90016-5.
- [56]Nosofsky, R. M., Palmeri, T. J. & McKinley, S. C. Rule-plus-exception model of classification learning. Psychological Review 101, 53–79 (1994). URL https://doi.org/10.1037/0033-295X.101.1.53.
- [57]Ellis, K. et al. DreamCoder: Bootstrapping inductive program synthesis with wake-sleep library learning. Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, 835–850 (ACM, New York, NY, USA, 2021). URL https://dl.acm.org/doi/10.1145/3453483.3454080.DOI
- [58]Wong, L., Ellis, K. M., Tenenbaum, J. & Andreas, J. Meila, M. & Zhang, T. (eds), Leveraging language to learn program abstractions and search heuristics. Proceedings of the 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.), Vol. 139 of Proceedings of Machine Learning Research, 11193–11204 (PMLR, 2021). URL https://proceedings.mlr.press/v139/wong21a.html.
- [59]Novikov, A. et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery (2025). Preprint at https://arxiv.org/abs/2506.13131.
- [60]Fawzi, A. et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610, 47–53 (2022). URL https://www.nature.com/articles/s41586-022-05172-4.DOI
- [61]Jansen, P. et al. CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. Findings of the Association for Computational Linguistics: ACL 2025, 13370–13467 (Association for Computational Linguistics, Vienna, Austria, 2025).arxiv.org/abs/2503.22708
- [62]Ghareeb, A. E. et al. A multi-agent system for automating scientific discovery. Nature 655, 497–505 (2026). URL https://www.nature.com/articles/s41586-026-10652-y.
- [63]Romera-Paredes, B. et al. Mathematical discoveries from program search with large language models. Nature 625, 468–475 (2024). URL https://www.nature.com/articles/s41586-023-06924-6.
- [64]Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D. & Griffiths, T. L. Using large-scale experiments and machine learning to discover theories of human decision-making. Science 372, 1209–1214 (2021). URL https://doi.org/10.1126/science.abe2629.
- [65]Rmus, M., Jagadish, A. K., Mathony, M., Ludwig, T. & Schulz, E. Generating Computational Cognitive Models using Large Language Models. Advances in Neural Information Processing Systems, Vol. 38, 97582–97619 (Neural Information Processing Systems Foundation, Inc., 2025).
- [66]Castro, P. S. et al. Discovering Symbolic Cognitive Models from Human and Animal Behavior (2025). Preprint at https://doi.org/10.1101/2025.02.05.636732.
- [67]Jagadish, A. K. et al. Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist (2026). Preprint at https://arxiv.org/abs/2606.26448.
- [68]Prystawski, B. et al. Auto-psych: Automating the science of mind using agent-driven theory discovery and experimentation (2026). Preprint at https://arxiv.org/abs/2606.26460.
- [69]Wang, C., Kasenberg, D., Stachenfeld, K. & Castro, P. S. Discovering Differences in Strategic Behavior Between Humans and LLMs (2026). Preprint at https://arxiv.org/abs/2602.10324.
- [70]Waterman, D. & Newell, A. Protocol analysis as a task for artificial intelligence. Artificial Intelligence 2, 285–318 (1971). URL https://linkinghub.elsevier.com/retrieve/pii/0004370271900142.DOI
- [71]Ericsson, K. A. & Simon, H. A. Protocol Analysis: Verbal Reports as Data Revised edn (The MIT Press, Cambridge, MA, 1993). URL https://doi.org/10.7551/mitpress/5657.001.0001.
- [72]Wurgat, D. et al. Scaling up the think-aloud method. Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 47 (2025). Preprint at https://arxiv.org/abs/2505.23931.
- [73]Newell, A. & Simon, H. A. Human Problem Solving (Prentice-Hall, Englewood Cliffs, NJ, 1972).
- [74]Flanagan, C., Sabry, A., Duba, B. F. & Felleisen, M. The essence of compiling with continuations. Proceedings of the ACM SIGPLAN 1993 Conference on Programming Language Design and Implementation, 237–247 (Association for Computing Machinery, New York, NY, USA, 1993). URL https://doi.org/10.1145/155090.155113.
- [75]Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, Vol. 33, 9459–9474 (Curran Associates, Inc., 2020). URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html.arxiv.org/abs/2005.11401
- [76]Butt, N. et al. CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay. Forty-first International Conference on Machine Learning (2024). URL https://openreview.net/forum?id=SXVn5IFsrs.arxiv.org/abs/2402.04858
- [77]Hoffman, M. D., Gelman, A. et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res. 15, 1593–1623 (2014).DOI
- [78]Phan, D., Pradhan, N. & Jankowiak, M. Composable effects for flexible and accelerated probabilistic programming in numpyro (2019). Preprint at https://arxiv.org/abs/1912.11554.



