From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution

Introduction

Language models are increasingly deployed as agents that must act on natural-language instructions: editing, drawing, operating software, manipulating an environment, and a lot more. However, the instructions users write are rarely explicit sequences of actions. A user asking an agent to “add a ‘total’ row to each table in the report” expects it to recover the many concrete edits compressed into this phrase, rather than be instructed for each one.

This tendency for compression is unsurprising: human speakers typically provide the minimal amount of information that is needed for a successful interpretation [13]. Speakers thus convey multiple actions through abstraction, defined by Wing (2010) as “the representation of many instances through a single higher-level object, capturing essential structure while suppressing lower-level details.”

For a language model to carry out abstract instructions, it must identify the abstract concepts and reconstruct the sequence of actions they leave implicit [3] (MacMahon et al. 2006). Models are unevenly good at this: performance declines as instructions grow more abstract [1] (Shridhar et al. 2020; Ruis et al. 2020; Wu et al. 2021), particularly evident in HEXAGONS (Lachmy et al., 2022), where instructions are annotated with abstraction levels.

This suggests a simple hypothesis: if the gap between language and action arises because abstract instructions leave task-relevant steps implicit, then unfolding those abstractions explicitly should help bridge it. Code generation is a natural first step to explore [8] (Liang et al. 2022): programming languages provide built-in mechanisms for unfolding abstraction — variables, loops, functions, conditions — so mapping instructions to code gives the model computational structure that exposes the structures implicit in language.

However, general-purpose code may not suffice: the model may still need to recover task-specific abstractions from scratch, e.g., what counts as a “table” or a “header” in a document, and reflect them as concrete operations.

In business environments, such recurring domain concepts are often provided as reusable operations through a domain-specific language (DSL) (Mernik et al., 2005) that is then used in code to map abstraction to action. In this work we ask: (i) can language models use task-specific DSLs to effectively follow natural language instructions? and, moreover, (ii) may language models come up with such DSLs themselves?

Our primary empirical investigation focuses on HEXAGONS, a visually grounded instruction-following benchmark designed to elicit abstraction in natural language. Its instructions refer to higher-level structures: complex objects, patterns, arrangements; and successful execution depends on producing the right low-level actions (Figure 2). Furthermore, HEXAGONS’s abstraction-level annotation lets us control performance on abstraction, and isolate where the interpretation breaks down.

We investigate the first question via a three-way comparison of strategies to solve the HEXAGONS board: direct inference, unrestricted Python generation, and Python generation over a DSL. In particular, we assess performance given HEXAGEN, a human-authored Python DSL we designed for this purpose, which defines decomposable abstractions on top of the HEXAGON tiles: named shapes, spatial queries and geometric transformations. The model may write an ordinary Python program and invoke DSL functions as needed, retaining full interpretive responsibility while offloading low-level domain-specific reasoning to the DSL (Figure 1).

Our results show that a human-curated DSL indeed leads to substantial performance gains, closing much of the gap with the human skyline, an effect we show on both HEXAGON (§ 5) and a TEXT2CAD real-world scenario (§ 6).

This success begs our second question, namely, can LLMs induce such DSLs themselves?

We investigate this by augmenting the HEXAGONS task training data, instructions paired with the denotations on the board, with a minimal API covering only low-level drawing actions. Models are asked to identify recurring patterns and implement respective DSL functions, producing executable Python code and documentation for downstream use. Replacing HEXAGEN with such induced DSLs shows mixed results relative to a minimal-API baseline, with improvements varying across models, but consistently underperforming the human-curated DSL counterpart.

Taken together, our experiments show that interpreting and executing natural language instruction using a curated DSL consistently outperforms both direct inference and unrestricted code generation, with the largest gains where interpretation is hardest: at higher abstraction levels. At the same time, we demonstrate that current LLMs fall short of human experts on the task of reliably inducing such abstraction layers themselves from natural language domain-specific data — leaving the automatic discovery of task-relevant abstraction layers from natural language texts as an open challenge for further investigation.

From Abstraction to Action:

Challenge and Key Hypotheses

Interpreting and grounding abstract instructions poses a genuine challenge for LLMs, as it requires mapping natural-language abstractions to their underlying implicit and concrete actions to execute. Here we argue that tackling this challenge is inherently multifaceted.

First we observe that formal coding languages are a natural, unambiguous way to express abstractions and how they unfold. Providing the model access to explicit computation — variables, loops, functions — alongside the language utterances, may help recover the actions that natural language leaves implicit. We therefore hypothesize that mapping instructions to executable code (rather than, e.g., relying on thinking trajectories) is a better strategy than direct inference on the task at hand.

Following this intuition, we cast instruction-following as a program-synthesis problem over a grounding environment: given a natural-language instruction $x$ and an initial state $s_0$, the model must produce a program $p$ whose execution yields the target state $s^*$.

Yet, code generation in a general-purpose language does not fully close the abstraction gap for a specific task. Plain Python lacks task-specific notions, (in HEXAGONS (§ 2.2), for example, these may be hexagonal grids, spatial neighbors, or geometric objects,) forcing the model to learn and implement every domain concept from scratch. Generally, we argue that code generation alone may not suffice for many real-world tasks, whose relevant abstractions are often domain-specific and lack, or are divorced from, their interpretation in a general-purpose language. So, just as SQL encapsulates relational operations like joins and aggregations rather than requiring repetitive iterations, a domain-specific language (DSL) can encapsulate recurring domain concepts as reusable operations ((Mernik et al., 2005)).

We draw from this a key conclusion: while code generation provides the machinery of computation, what is still missing is a way to encapsulate recurring domain concepts as reusable operations that the model can compose and reuse rather than rederive. For this, a DSL can help.

The Hexagons benchmark

The HEXAGONS benchmark ((Lachmy et al., 2022)) is an instruction-following benchmark designed to elicit abstraction in natural language, derived from a collaborative drawing game in which one player describes how to recreate an image on a hexagonal grid while the other follows the instructions on an empty board. Since players are rewarded for both accuracy and speed, speakers naturally resort to compressed, abstract descriptions characteristic of natural communication — referring to objects, loops, spatial transformations, and conditions — each standing for many low-level actions, with missing details left to be filled by the hearer.

The core challenge in solving the HEXAGONS task is that carrying out such instructions requires more than understanding explicit content: the model must identify the abstract constructions in the instruction, ground them in exact tile coordinates, and execute the implied actions correctly. As illustrated in Figure 2, “a blue flower” compresses a spatial arrangement of seven tiles into a single reference, while “on each of the outermost corners” implicitly specifies four locations dependent on the geometry of the previously constructed object. These stages are tightly coupled — errors in interpretation propagate through grounding into execution — and each grows harder as abstraction increases, as confirmed empirically by Lachmy et al. (2022). HEXAGONS therefore provides a controlled testbed for studying how to improve the interpretation, grounding, and execution of abstract instructions in contemporary LLMs.

DSLs for Natural Language Abstraction

From General Code to Task-Specific Libraries

The notion of a domain-specific language (DSL) is known from coding environments, and it refers to a specialized library tailored to a particular application domain, for defining, breaking down and spelling out domain-specific abstract notions ((Mernik et al., 2005)). By raising the level of abstraction to match the problem at hand, DSLs reduce both the effort and the error rate of programming within their target domain. In our setting, we aim to use DSLs for interpreting naturally occurring language instructions in different domains, likewise reducing the effort and errors in interpreting and following natural language instructions.

On the way to obtain this, we face several concerns in the design of DSLs for NL. The first is representation: the DSL should provide first-class constructs for the abstractions that genuinely organize solutions in the target domain. The second is transparency: because our DSL is to be consumed by a language model, what each primitive does and how it should be called must be clearly evident from its name, signature, and documentation (Qu et al., 2025). The third is robustness: constrained semantics, safe defaults, and failure-resistant interfaces should ensure that when a program fails, it fails because the intended construction is wrong, not because the programming interface is fragile.

From Hexagons to HexaGen We apply these principles to HEXAGONS to deliver HEXAGEN, an expert-curated Python DSL that supplies the missing abstraction layer for following instructions on the HEXAGONS board. On the representational side, we identified its core primitives by manually analyzing training examples, guided by the abstraction categories in Lachmy et al. (2022). For transparency, the API exposes non-overlapping, self-explanatory methods with thorough documentation. For robustness, it validates inputs eagerly, provides safe defaults, and raises descriptive exceptions for common pitfalls, thus errors reflect flawed usage, not interface fragility. The full DSL specification, including all constructors, actions, queries, and their signatures, is provided in in Appendix § 15. A representative example is given in Figure 1.

Inducing DSLs from Data

Creating useful and adequate DSLs, such as HEXAGEN, is a human-intensive and costly engineering effort. The theoretical benefits of DSLs in general, and of HEXAGEN in particular, beg a natural follow-up question: can LLMs induce such abstraction layers themselves, given only examples of the task?

We formulate this problem as follows. Let $\mathcal{A}_0$ denote a foundation library exposing only the task-specific primitives. Let $\mathcal{D} = \{(x_i, s_i)\}_{i=1}^{N}$ be a training set of paired natural-language instructions and their denotation (in HEXAGONS (§ 2.2) for example, these are instruction paired with target board states represented as tile-level descriptions). The induction task is, given $\mathcal{A}_0$ and $\mathcal{D}$ produce an extended library $\mathcal{A} \supseteq\mathcal{A}_0$, consisting of new callable functions, together with API documentation $\operatorname{Doc}(\mathcal{A})$ suitable for downstream prompting.

We suggest that the induction pipeline be comprised of three stages: (i) abstraction discovery, which identifies recurring, non-trivial computational patterns in the instruction corpus, (ii) method implementation, which translates the discovered method specifications into validated executable code on top of the foundation library, and (iii) documentation authoring, which produces a compact API specification that serves as the sole downstream prompting interface.

Experimental Setup

Goal. We set out to empirically investigate two questions: whether a task-aligned DSLs improve LLM performance over plain code generation, and whether LLMs can induce useful DSLs themselves.

Data. We use the original HEXAGONS dataset (Lachmy et al., 2022), consisting of 620 procedures and 4,177 language instructions overall, where each procedure is a sequence of instructions paired with the corresponding board state. Later, we also use the Text2CAD (Khan et al., 2024) benchmark.

Task. The HEXAGONS task is defined as follows. The input is a procedure describing a complex image, composed of multiple instructions. The procedure is processed all at once.² We evaluated models under one of three execution scenarios: (i) Direct Prediction, which outputs a list of (row, col, color) tuples specifying the tiles to paint; (ii) Unrestricted Python Generation, which outputs Python code implementing get_tiles() to return the same tuples; and finally, (iii) DSL-based Code Generation, where the input includes the DSL API documentation and the output is code using the DSL, whose execution yields the target.

Prompts. All scenarios use a two-phase prompt: a system prompt defining the board, colors, output requirements, and scenario-specific information, if any (API specification, tuple format, or board dimensions), and a user message containing the instructions. The exact are stated in Appendix § 17.

Models. We evaluate both proprietary and open-weight models, both low and high reasoning (thinking) settings: GPT-5.1 and Gemini-3-Pro (high/low), Gemini-3.1-Pro (high), Qwen3-30B-A3B (a reasoning model) (Yang et al., 2025), and Gemma-3-27B-IT (Team, 2025). The open models are served locally via vLLM on 2 A100 GPUs.

Inference. All models are decoded with temperature 0 and a maximum output length of 15K tokens. Generated programs are executed in an isolated environment with a 60-sec. timeout and up to 3 retries conditioned on the resulting exception message.

Metrics. Following Lachmy et al. (2022), we report F1 and exact match on the final board state. In the all-at-once setting, board-level F1 equals action-level F1 in Lachmy et al. (2022). Table 1 reports mean ± standard deviation over three execution runs. Table 3 reports mean ± standard deviation across three independently induced libraries, each evaluated over three execution runs.

Proprietary Models

Open-Source Models

GPT-5.1 (Low)

GPT-5.1 (High)

Gemini 3 Pro (Low)

Gemini 3 Pro (High)

Qwen3 30B

Gemma3 27B

F1

EM

F1

EM

F1

EM

F1

EM

F1

EM

F1

EM

Tiles

$0.683 \pm0.006$

$0.236 \pm0.018$

$0.765 \pm0.016$

$0.322 \pm0.032$

$0.743 \pm0.009$

$0.301 \pm0.009$

$0.745 \pm0.02$

$0.382 \pm0.037$

$0.556 \pm0.016$

$0.177 \pm0.013$

$0.407 \pm0.005$

$0.123 \pm0.007$

Unrestricted Python

$0.708 \pm0.011$

$0.231 \pm0.009$

$0.757 \pm0.009$

$0.296 \pm0.009$

$0.807 \pm0.003$

$0.376 \pm0.040$

$0.793 \pm0.021$

$0.419 \pm0.000$

$0.587 \pm0.012$

$0.172 \pm0.007$

$0.495 \pm0.005$

$0.145 \pm0.000$

Human-Curated DSL

$\mathbf{0.774 \pm0.021}$

$\mathbf{0.371 \pm0.000}$

$\mathbf{0.831 \pm0.010}$

$\mathbf{0.441 \pm0.025}$

$\mathbf{0.825 \pm0.002}$

$\mathbf{0.457 \pm0.024}$

$\mathbf{0.853 \pm0.016}$

$\mathbf{0.511 \pm0.046}$

$\mathbf{0.663 \pm0.019}$

$\mathbf{0.237 \pm0.033}$

$\mathbf{0.512 \pm0.007}$

$0.140 \pm0.007$

	Proprietary Models		Open-Source Models
	GPT-5.1 (High)	Gemini 3.1 Pro (High)	Qwen3 30B Thinking	Gemma3 27B Instruct
	F1	EM	F1	EM	F1	EM	F1	EM
Minimal DSL	$0.740 \pm0.008$	$0.282 \pm0.034$	$0.839 \pm0.033$	$0.484 \pm0.028$	$0.542 \pm0.003$	$0.145 \pm0.016$	$0.371 \pm0.019$	$0.097 \pm0.000$
All-at-once	$0.757 \pm0.082$	$0.314 \pm0.103$	$0.776 \pm0.069$	$0.321 \pm0.099$	$0.565 \pm0.019$	$0.152 \pm0.018$	$0.416 \pm0.055$	$0.129 \pm0.023$
Batches	$0.745 \pm0.088$	$0.339 \pm0.160$	$0.828 \pm0.032$	$0.518 \pm0.023$	$0.480 \pm0.041$	$0.100 \pm0.034$	$0.389 \pm0.060$	$0.105 \pm0.034$
Semantic Batches	$0.704 \pm0.007$	$0.225 \pm0.022$	$0.788 \pm0.077$	$0.396 \pm0.129$	$0.531 \pm0.025$	$0.149 \pm0.019$	$0.441 \pm0.020$	$0.121 \pm0.011$
Human-Curated DSL	$0.831 \pm0.010$	$0.441 \pm0.025$	$0.861 \pm0.016$	$0.591 \pm0.019$	$0.663 \pm0.019$	$0.237 \pm0.033$	$0.512 \pm0.007$	$0.140 \pm0.007$

Does Explicit Abstraction Help?

Results Table 1 presents execution results across all models and strategies. Overall in terms of F1, Unrestricted Python Generation improves over Direct Prediction in five of six settings, while exact match remains broadly similar overall. This suggests that generating an executable program is often more effective than predicting tiles directly. Then, DSL-based Code Generation pushes the results further, essentially closing the gap with the human skyline of F1=82.06 reported by Lachmy et al. (2022). It achieves the highest F1 across all six models and the highest exact match in five of six, indicating that explicit, executable abstractions improve grounding beyond code execution alone. We further see that weaker models benefit more from the DSL-based strategy: relative to Direct Prediction, the three weakest models improve F1 by 13.3–25.8%, compared with 8.6–14.5% for the three strongest.

Analysis Lachmy et al. (2022) showed that model performance degrades as abstraction increases. We now ask the respective question, whether the DSL is especially helpful in the harder, more abstract, scenarios. Figure 3 suggests that it is indeed the case. At Level 0 the DSL method performs on par with the other strategies for every model, but at Levels 2–3 it leads by up to 0.21 F1, essentially showing that the more abstract the instruction, the more the DSL helps.

We next ask how models use the DSL. Figure 4 reports the breadth of DSL usage (i.e., the number of unique DSL methods invoked, out of the 41 available) and the volume of DSL usage (i.e., the total number of DSL method calls issued), each plotted against board F1. In terms of breadth, stronger models use more of the DSL; Gemini 3 Pro (High) and GPT-5.1 (High) invoke 23–24 unique methods, versus 19–20 for Gemma 3 and Qwen 3 (Spearman $\rho= +0.80$ with F1). Volume runs the other way, and tracks F1 almost perfectly (Spearman $\rho= -1.00$): weaker models issue substantially more DSL calls. This suggests that weaker models rely on fewer methods and call them more repetitively, while stronger models spread their calls across a wider slice of the DSL and invoke it more sparingly overall, hinting at more targeted abstraction selection. All in all, the DSL helps all models, but the gain scales with the model’s ability to exploit the structure it provides.

Finally, we test whether the models’ errors stem from lacking visual feedback. Showing the board after each instruction does not help: when adding the visual signal, for HEXAGEN-based generation the difference is negligible, for direct tile prediction performance degrades markedly (Appendix § 11).

Level	Qwen3-30B-A3B-Thinking DSL	Qwen3-30B-A3B-Thinking Direct	Qwen3-30B-A3B-Thinking Undec.	Gemini-3.1-Pro DSL	Gemini-3.1-Pro Direct	Gemini-3.1-Pro Undec.
Expert	43.8%	20.9%	35.3%	13.8%	4.8%	81.4%
Intermediate	25.6%	20.4%	54.0%	17.6%	12.1%	70.3%
Beginner	23.0%	22.1%	54.9%	17.1%	17.5%	65.3%
Abstract	31.1%	30.1%	38.9%	31.1%	24.0%	45.0%

Does This Hold Beyond Hexagons?

To test whether our results generalize beyond the HEXAGONS task, we turn to Text2CAD ((Khan et al., 2024)) with its CadQuery annotations ((Xie and Ju, 2025)), where each object is paired with design instructions at four levels of abstraction, from expert to abstract. Taking CadQuery as the foundation library, we hand-curate a small DSL of eight recurring abstractions following the design principles of § 3. We follow the evaluation setup of Khan et al. (2024) and simplify by using a single protocol across all four abstraction levels: a VLM judge compares renders of the DSL and direct programs pairwise, with position randomized, and returns a preference or declares the pair indistinguishable. Programs that fail to execute count as losses, and pairs where both fail are dropped. Full details are provided in Appendix § 13.

Table 2 shows that the DSL is preferred in seven of eight model–level comparisons, the exception being Gemini-3.1-Pro at the beginner level, where the two are effectively tied (17.1% vs. 17.5%). Some of this gain is a matter of reliability: the DSL lowers the execution-failure rate for both models, most sharply at the expert level, from 51.3% to 35.1% for Qwen3-30B-A3B-Thinking and from 21.7% to 13.4% for Gemini-3.1-Pro.

Can LLMs Induce DSLs Themselves?

Observing the substantial gains using HEXAGEN, our human-curated DSL, we ask whether LLMs can induce useful and effective DSLs themselves.

Experimental Setup We design an experiment where each evaluated model induces its own DSL. Each induction variant is run 3 times per model, producing 3 independently induced libraries. Each library is evaluated over 3 execution runs. For proprietary models, we use only the high reasoning setting.

Induction As mentioned in § 4, the induction pipeline has three stages: abstraction discovery, method implementation, and documentation authoring, which we detail here in turn.

In the discovery stage, we vary how the instruction corpus is presented to the model: (1) flat, which seeks and proposes abstractions for all instructions in a single prompt; (2) arbitrary batching, which partitions the corpus into fixed-size batches, analyzes each independently, and then deduplicates and merges the resulting proposals; and (3) semantic batching, which replaces arbitrary batches with sets of similar instructions, obtained by $k$-means clustering over BGE-M3 instruction embeddings ([7]), targeting batches of 100 instructions for both batched variants. Grouping related examples before prompting resembles the clustered-sampling variant of [12 Appendix C.2]. In the implementation stage, given the proposed method signatures, the model receives the foundation-library source and doc and generates complete implementations, including docstrings and executable examples. Methods that fail to compile or fail their doctest examples are removed. Validation and deduplication are standard in LLM tool-creation pipelines ((Yuan et al., 2024)).

In the documentation stage, the model produces the final documentation, which serves as the only interface provided for inference. It is written for LLMs rather than human readability, favoring clarity and unambiguous information. This follows LILO’s AutoDoc procedure ([12]).

Results. Table 3 shows that the human-curated HEXAGEN achieves the highest F1 and exact match in all four model settings, outperforming the best induced variant by 3–10 F1 points. Still, some induced DSLs perform strongly in individual runs: all-at-once induction with GPT-5.1 (High) reaches 0.816 F1 / 0.452 EM. The main limitation is thus consistency rather than capability, as the same induction settings that produce strong libraries also produce runs falling below the minimal baseline. For Gemini 3.1 Pro the pattern is stronger: no induced variant beats the minimal API they extend.³

Beyond execution quality, the strategies differ in how many methods they induce: batched discovery yields substantially more methods than flat analysis, though library size does not correlate with downstream performance (Appendix § 14).

Finally, we manually inspect two induced libraries to learn about the type of errors they produce. This shows how a single foundational error in the library’s design can make an induced library perform worse than no library at all. Both come from the same setup, GPT-5.1 (High) with the flat induction variant, yet Run A reaches 0.816 board-F1 while Run B reaches 0.700, below both non-DSL baselines for this model (0.765 Tiles, 0.757 Unrestricted Python). They cover about the same abstractions and differ in one fundamental choice: Run B falsely treats the board as a rectangular lattice. Seven of its ten methods rest on that assumption, and all seven run without failing while producing wrong tiles. Run A gets the geometry right, and its remaining distance to HEXAGEN is of a different kind: it has no compositional layer, no first-class Shape with set operations, spatial queries, or anchor-based copy (Table 4).

Capability	HEXAGEN	Run A	Run B
Shape constructors (rays, lines, rings, triangles, fill)	✓	✓	✗
Rotation / mirror / translation	✓	✓	✗
Hex distance metric	✓	✓	–
Hex adjacency (neighbors)	✓	✓	✗
Compositional Shape algebra	✓	–	–
Spatial / topological queries	✓	–	–
Tiling / anchor-based copy	✓	–	✗
Board & color queries	✓	–	–
Board-F1	0.831 ±0.010	0.816	0.700

Related Work

Grounded instruction following maps natural language to executable actions in situated environments, spanning navigation ([6, 2] (MacMahon et al. 2006)), robotic manipulation ([1] (Tellex et al. 2011)), and collaborative construction ([3] (MacMahon et al. 2006)). Across these settings, performance degrades as instructions become more abstract ([1] (Shridhar et al. 2020; Ruis et al. 2020; Wu et al. 2021)). In HEXAGONS ((Lachmy et al., 2022)), where instructions are annotated by abstraction level, this trend is measured directly. We take this gap as a starting point and ask whether an executable target language that makes abstractions explicit can help close it.

A growing body of work treats grounded reasoning as program synthesis: instead of predicting outputs directly, the model generates executable code. This paradigm has been effective in mathematical reasoning (e.g., [11, 9]), visual question answering ((Gupta and Kembhavi, 2022; Suris et al., 2023)), and embodied control ([1] (Liang et al. 2022; Singh et al. 2022)). Code generation is a natural fit for abstract instructions because programming languages already provide mechanisms such as variables, loops, and functions. However, when models generate in a general-purpose language or over low-level action sets, they must still recover task-specific abstractions on their own. Our comparison of direct inference, unrestricted Python, and DSL-based generation shows that making these abstractions explicit and executable is itself a major factor in execution quality.

Domain-specific abstractions for code generation have often taken the form of external DSLs that restrict the target language, both in classical program synthesis and semantic parsing ([14, 4]) and in recent LLM systems that translate natural language into formal domain-specific targets ((Wong et al., 2023)). HEXAGEN, by contrast, is an internal DSL: the model writes unrestricted Python, while the DSL supplies callable abstractions such as named shapes, spatial queries, and composable transformations. In this respect, it is closer to API-augmented generation ((Liang et al., 2022; Singh et al., 2022)), though those systems expose fixed low-level action primitives rather than abstractions aligned with the level at which people give instructions.

The question of whether models can discover reusable abstractions connects to prior work on library learning and tool induction, which differs from ours in the signal abstractions are found in: recurring code rather than recurring language. In the external DSL setting, DreamCoder ([10]) compresses solved programs, keeping an abstraction when it shortens the corpus by more than it costs to define; LAPS ((Wong et al., 2021)) adds word alignments to that objective, but only to score candidates compression already extracted; and LILO ([12]) adds an LLM searcher and documentation, reporting abstractions proposed by an LLM prompted with programs as an appendix negative result only. On the internal side, LATM ([5]) and CREATOR ((Qian et al., 2023)) build per-task tools rather than a library, while Voyager ((Wang et al., 2023)) admits skills by execution success during open-ended play. We instead induce a library from instructions paired with board states, with no solved programs to compress: what recurs across the instructions is the only signal available, and the abstractions we seek are those a speaker compresses into a phrase—not the fragments that recur across solutions.

Conclusions

The core challenge in grounded instruction following is mapping abstract, underspecified language to concrete, executable actions. We show that a DSL is a useful device for making task-relevant abstractions explicit. Concretely, we show on HEXAGONS that generating executable code with HEXAGEN, a human-curated DSL, substantially outperforms both direct prediction and unrestricted code generation—with higher gains at higher abstraction levels. However, current LLMs still do not reliably induce such DSLs with similar efficacy on their own right. Model-authored DSLs in our settings occasionally approach the human-curated baseline but remain far more unstable, with no discovery strategy consistently resolving this instability. Enabling models to automatically construct robust, transparent, task-specific abstraction layers therefore remains an intriguing open challenge. To promote research progress on this matter, we release our code, data, evaluation pipeline, and all DSLs, encouraging the community to advance work on executable abstraction and DSL induction.

Limitations

Benchmark coverage. Our evaluation covers HEXAGONS and TEXT2CAD. The two differ in domain and output space, but broader coverage across additional testbeds would strengthen the generality of our claims. The TEXT2CAD replication is also deliberately lightweight: its DSL comprises eight functions, against the 41 of HEXAGEN. It therefore establishes that the effect transfers, and does not measure how large that effect can be in a second domain.

Induction without execution feedback. All three induction strategies share one design: the model authors a DSL from a fixed pool of examples and never runs the result. Run B suggests this may matter — its rectangular-lattice assumption breaks seven of ten methods and would surface on the first execution — though a single case does not establish how much an iterative variant would recover.

Future work. Two directions follow from these results. The first is multi-turn interaction, which our setting does not consider. The second is DSL induction itself. Our results establish that a good abstraction layer helps and that current models do not reliably produce one, which leaves the induction problem open.

Visual Feedback

We test whether giving the model visual access to the board it is constructing improves execution.

Setup. We use the step-by-step protocol (Appendix Appendix § 12) rather than our full-procedure formulation, since the latter emits the entire procedure in one turn and leaves no point at which an intermediate board could be shown. After each instruction is executed, we render the resulting board and supply the image alongside the next instruction. We evaluate both execution strategies, Tiles and HEXAGEN, with and without the rendered board, using Gemini 3.1 Pro on the validation set.

Setup	EM	B-F1	A-F1
Tiles	0.590	0.864	0.819
Tiles + Vision	0.464	0.830	0.782
HEXAGEN	0.744	0.902	0.867
HEXAGEN + Vision	0.727	0.896	0.860

Results. Visual feedback does not help either strategy. For HEXAGEN the difference is negligible; for direct tile prediction it costs over twelve points of exact match. We report this as a negative preliminary result, and leave the interactive setting — where the model can act on visual feedback rather than only observe it — to future work.

Full-Procedure vs. Step-by-Step Prompting

To determine the querying setup for our generation framework, we compared our full-procedure formulation against the default HEXAGONS evaluation protocol. The original benchmark uses step-by-step prompting with access to the full conversational history, whereas our framework provides the model with the entire instruction sequence at once.

As shown in Table 6, full-procedure prompting performs comparably to the original benchmark protocol across both tested models, and is even slightly better. We therefore use it throughout. This choice simplifies the generation pipeline, reduces inference overhead, better matches real-life settings where the full task is available upfront, and avoids conflating execution quality with dialogue-history effects.

Setup	GPT-5.1 (Low)	Gemma 3	DeBERTa	T5
Step-by-step + Full History	0.520±0.003	0.404±0.002	0.404	0.386
Full-Procedure	0.541±0.007	0.407±0.005	–	–

Text2CAD Evaluation Details

Data. Text2CAD (Khan et al., 2024) with CadQuery annotations (Xie and Ju, 2025); the 1,000-object test split, each object with instructions at four abstraction levels (expert, intermediate, beginner, abstract) and a gold CadQuery program. We retain the 885 objects whose gold program executes and evaluate on these at every level. Gold executions are cached and deduplicated. Single seed.

Conditions. Direct: raw CadQuery. DSL: an 8-function hand-curated library over CadQuery, built following the design principles of § 3. Both conditions are generated by Qwen3-30B-A3B-Thinking and Gemini-3.1-Pro.

Evaluation. We apply one protocol at all four levels. Renders of the two conditions are shown paired to Qwen3.6-27B as a VLM judge, which selects the better match to the instruction or returns undecided; the gold object is never shown and presentation order is randomized. An example on which exactly one condition fails to execute is awarded to the other; where both fail, no comparison is possible and the example is dropped.

Method Counts Across Induction Strategies

Figure 5 tracks how many candidate methods survive each stage of induction — proposal, deduplication, and validation — under the three induction strategies. For the three stronger models, both batched strategies produce substantially more validated methods than flat analysis, suggesting that a single pass over the full instruction corpus is too coarse for the model to attend to recurring patterns. Gemma 3 is the least consistent, with method counts varying greatly across strategies. A larger library is not in itself better, however: many candidates are removed during deduplication or validation, and only a subset is ever called downstream. Method count is therefore best read as a diagnostic of the induction process rather than as a quality measure, with the execution results in Table 3 providing the relevant comparison.

HexaGen API

Table 7 lists the public classes, methods, properties, and operators available in the HEXAGEN DSL. This is the full API exposed to the language model; internal mathematical and coordinate helper functions remain private and are not accessible during generation.

Constants. The library also exposes the constants WIDTH=18, HEIGHT=10, the color set {white, black, yellow, green, red, blue, purple, orange}, and the six hex-grid directions {up, down, up_right, down_left, up_left, down_right}.

Minimal API

Table 8 lists the public classes, methods, and properties available in the minimal HEXAGEN API. This is the full API exposed to the language model; internal helper classes and mathematical or coordinate utilities remain private and are not accessible during generation.

Class	Method / Property	Description
Game	__init__(width, height)	Initializes the board with the given dimensions.
Game	start(width, height)	Creates a new game instance with the given dimensions.
Game	__enter__(), __exit__(...)	Supports context-managed execution inside a Game block.
Game	plot(...)	Visualizes the board state, optionally against a gold reference.
Tile	__init__(row, column, game)	Creates a tile at the specified coordinates.
Tile	draw(color)	Paints the tile with the specified color.
Tile	color, row, column	Properties returning the current color and 1-based coordinates.

Constants. The API also exposes the board dimensions WIDTH=18 and HEIGHT=10, the color set {white, black, yellow, green, red, blue, purple, orange}, and the six hex-grid directions {up, down, up_right, down_left, up_left, down_right}.

Standard Experiment Prompts

Shared game-context system prompt.

# ========== HEXAGONS GAME OVERVIEW ==========

# Board geometry
- 18 columns x 10 rows, both **1-indexed** (column 1 = left
  edge; row 1 = top).
- Address any tile by the **tuple (row, column)**.
- Every tile starts as **white** and keeps its color until
  repainted.

# Colors
- Allowed colors: ['white', 'black', 'yellow', 'green', 'red',
  'blue', 'purple', 'orange'].

# Game flow

Class	Method / Property	Description
Game	__init__(width, height)	Initializes a game with board dimensions (defaults to WIDTH=18, HEIGHT=10).
	start(width, height)	Factory method returning a new game instance.
	__enter__(), __exit__(...)	Context-manager interface for writing programs inside a Game block.
	record_step(name)	Saves a snapshot of the current painted tiles under a step name.
	get_record(names)	Returns the tiles painted in one or more named steps.
	plot(...)	Visualizes the board state, optionally against a gold reference.
Shape	__init__(...)	Constructs a shape from tiles or other shapes.
	draw(color)	Paints all active tiles in the shape with the specified color.
	copy_paste(...)	Copies the shape along a direction, shift vector, or source–destination mapping.
	reflect(...)	Reflects the shape across a line, coordinate axis, or directional axis.
	grid(...)	Repeats the shape multiple times along a direction.
	rotate(center, angle)	Rotates the shape around a tile by a multiple of 60 degrees.
	recolor(color_map)	Recolors tiles inside the shape according to a mapping dictionary.
	get(criterion)	Selects tiles satisfying a geometric or color criterion (e.g., outside, above, a direction, or a color).
	neighbors(criterion)	Returns neighboring tiles satisfying a criterion such as all, white, inside, or a direction.
	boundary(...)	Returns the shape’s boundary tiles, including outer or inner boundary modes.
	edge(direction)	Returns the outermost boundary segment in a specified direction.
	extreme(direction)	Returns the furthest tile in a specified hex direction.
	intersection(other)	Returns the tiles shared by this and another shape.
	overlaps(other)	Returns whether this shape shares any tiles with another.
	is_empty()	Returns whether the shape contains no tiles.
	center()	Returns the center-of-mass tile of the shape.
	__add__, __sub__, __mul__	Set-style union, difference, and intersection operators over shapes.
	get_entire_board()	Static method returning a shape containing all board tiles.
	get_board_perimeter()	Static method returning the perimeter of the board.
	get_color(color)	Static method returning all active tiles of a given color (or all colors).
	get_column(column)	Static method returning all tiles in a given column.
	polygon(...)	Static method connecting consecutive vertices with Lines.
Tile	__init__(row, column)	Creates a tile at the specified 1-based coordinates.
	neighbor(direction)	Returns the adjacent tile in a hex-grid direction.
	on_board()	Returns whether the tile lies on the board.
	color, row, column	Properties returning the current color and 1-based coordinates.
	offset	Property returning the (row, column) pair.
Line	__init__(...)	Creates a line from a start tile and either an end tile or a direction and length.
	parallel(shift, space)	Returns a parallel line translated by the specified spacing.
Circle	__init__(center, radius)	Creates a circular outline around a center tile with a given radius.
Triangle	__init__(...)	Creates a directional triangular border shape from a start tile, orientation, and side length.

• A human Instructor writes natural-language steps.

• You are the Executor who must ground each step on the board.

• Previously painted tiles stay as they are unless a step changes them.

Abstraction note

• Instructions may employ higher-level abstractions (objects, loops, conditions, symmetry); a single sentence can cover many low-level paint actions.

Code-generation prompt.

System prompt:

# ==== CODE-GENERATION ====

# Role
- You are an expert **Python** assistant for the Hexagons drawing game.

# Task
- Each turn you receive an *open* Python code block that ends with a `# TODO`.
- Complete the code so it satisfies the `# TODO`, then finish with:
    board_state = g.board_state

# Rules
1. **Write only Python -- no prose -- until the fence closes **
2. Do **not** add imports or create a Game; the script already begins with:
       from hexagen import Game, Tile, Shape, Line, Circle,
           Triangle
       with Game() as g:
3. Store every new shape or tile in a descriptive variable (e.g. `red_flower`, `flower_stack`) so later steps can reuse them.
4. Write all lines of code using the **same** **4-space indentation** that the `# TODO` line uses; indent the first line as well. This instruction is super important as the code won't run without the proper indentations.
5. Never overwrite previously defined variables and add **no ** extra comments, prints, or plots.
6. Columns and rows are **1-indexed**.
7. Use **only** the functions, classes, and parameters explicitly defined in the provided library API specification. Do not invent, guess, or call any methods, arguments, or attributes that are not listed there.

[... Injected HexaGen API Spec ...]

User message template:

##### CONTEXT (completed instructions so far)
{HISTORY_BLOCK}

##### CURRENT SCRIPT
{CODE_SO_FAR}
{NEXT_STEP}

Tile-prediction prompt.

System prompt:

# ===== TILE-SELECTION =====

# Role
- You are an assistant for a hexagonal drawing game.

# Task
- Given natural-language instructions, output which tiles should be colored.

# Return format
- A list of tuples in the form **(row, column, color)**.

# Rules
1. Rows and columns start at **1**.
2. Allowed colors: ['white','black','yellow','green','red','blue','purple','orange'].
3. Reply **only** with the list of tuples -- no extra prose.

User message template:

##### CONTEXT (completed instructions so far)
{HISTORY_BLOCK}

##### NEXT INSTRUCTION
{NEXT_STEP}

Provide the tiles as a list of (row,column,color) tuples:

Unrestricted-Python ablation prompt.

System prompt:

# ===== PYTHON-BASED TILE GENERATION =====

# Role
- You are an assistant for a hexagonal drawing game.

# Task
- Given natural-language instructions, write unrestricted Python code to compute which tiles should be colored.
- Output the tiles as a list of tuples in the form **(row, column, color)**.

# Constraints
- You may use any Python logic (loops, conditions, functions, list comprehensions, etc.).
- You are NOT given a drawing DSL or library. Use pure Python to calculate tile positions.
- The board has 10 rows and 18 columns, both **1-indexed**.

# Return format
- Write a Python function named `get_tiles()` that returns a list of (row, column, color) tuples.
- The function must return the result as a list: `return [( row, col, color), ...]`
- Reply **only** with Python code -- no extra prose before or after.
- Do **not** include markdown code blocks.

User message template:

##### CONTEXT (completed instructions so far)
{HISTORY_BLOCK}

##### INSTRUCTIONS
{NEXT_STEP}

Write Python code with a `get_tiles()` function that returns the list of (row, column, color) tuples:

HexaGen API specification.

# ==== HEXAGEN API SPEC ====
# One entity per line; two-space indentation shows containment.

# Extra 'Notes:' lines clarify hidden behaviour.

Import: from hexagen import Game, Tile, Shape, Line, Circle,
    Triangle

Constant: WIDTH = 18
Constant: HEIGHT = 10
Constant: COLORS = ['white','black','yellow','green','red',
    'blue','purple','orange']

# These are the hexagon board directions, most methods use
# these directions, some of them also extends the list to
# more directions.
Constant: DIRECTIONS = ['up','down','up_right','down_left',
    'up_left','down_right']

# ------
Class: Game
  Method: __init__(width:int = WIDTH, height:int = HEIGHT)
  Method: start(self) -> Game
  Method: __enter__() -> Game
  Method: __exit__(exc_type, exc, tb) -> bool
  Method: record_step(step_names:str | list[str]) -> Shape
    # Notes: closes previous step snapshot
  Method: get_record(steps_names:str | list[str]) -> Shape
    # tiles drawn in named step(s)
  Method: plot(gold_boards:list[int | list[list[int]] | list[list[int]]] | None
    = None,

multiple:bool = False, file_name:str | None =
None, show:bool = True) -> Figure
# ------
Class: Shape        # Base class for collections of tiles
    Method: __init__(tiles:list[Tile] | Tile | Shape, from_linds
        :bool = False,
        from_hexagons:bool = False, game:Game |
        None = None)
    Notes: duplicate tiles removed; single-tile => auto-up-
        casts to Tile.

    # -- Painting helpers (ALL mutate active board **and**
        return the new Shape) --
    Method: draw(color:str) -> Shape
                # auto-paints
    Notes: draw() overwrites the tile's previous color. To
        find where two shapes overlap, compute the intersection BEFORE drawing: overlap = shape_a
        * shape_b (or shape_a.intersection(shape_b)).
    Method: copy_paste(shift_direction:str | None = None,
                # auto-paints
        spacing:int = 0, reference_shape:Shape |
        None = None,
        source:Tile | None = None, destination:
        Tile | None = None,
        shift:_Vec | None = None) -> Shape
    Notes: Returns a NEW Shape at the destination. To chain
        multiple copies, call copy_paste on the RETURNED shape. For repeated copies
        in one direction, use grid() instead.
    Param options: shift_direction in {'left','right'} OR
        DIRECTIONS OR None
    Method: reflect(axis_line:Line | None = None, column:int |
        None = None,                 # auto-paints
        axis_direction:str | None = None,
        tile_on_axis:Tile | None = None) -> Shape
    Param options: axis_direction in {'horizontal','vertical'}
        OR DIRECTIONS
    Method: grid(shift_direction:str, spacing:int, num_copies:
        int | None = None) -> Shape  # auto-paints
        shift_direction in {'left','right'} OR DIRECTIONS OR None;
        num_copies=None => 'fill board'.
    Method: rotate(center_tile:Tile, angle:int) -> Shape
                # auto-paints
        angle must be multiple of 60.
    Method: recolor(color_map:dict[str,str]) -> Shape
                # auto-paints

    # -- Geometric queries
    -------------------------------------------------
    Method: get(criterion:str) -> Shape  # the method returns a
        new Shape object that has the criterion geometric
        relation to the original shape or the tiles with
        selected colors
    Param options: {'outside','inside','above','below','top',
        'bottom',
                'corners','endpoints'} OR DIRECTIONS OR
        COLORS
    Method: neighbors(criterion:str = 'all') -> Shape
        Param options: {'all','right','left','above','below',
        'outside','inside','white'} OR DIRECTIONS
    Method: boundary(criterion:str = 'all') -> Shape  #
        criterion in {'all','outer','inner'} # the method
        returns the boundary tiles belonging to the shape itself,
        without including any tiles outside the shape.
    Method: edge(direction:str) -> Shape  #
        direction in {'left','right','top','bottom'} OR
        DIRECTIONS
    Method: extreme(direction:str) -> Shape  #
        direction in DIRECTIONS
    Method: is_empty() -> bool
    Method: overlaps(other:Shape) -> bool
    Method: intersect(other:Shape) -> Shape  #
        same as shape_a * shape_b
    Method: center() -> Tile  #
        center of mass (rounded)

    # -- Set operators
    -------------------------------------------------
    Operator: shape_a + shape_b -> Shape  #
        union
    Operator: shape_a * shape_b -> Shape
        intersection
    Operator: shape_a - shape_b -> Shape  #
        difference

    # -- Utilities (static)
    -------------------------------------------------
    Static: get_entire_board(game:Game | None = None) -> Shape
    Static: get_board_perimeter(game:Game | None = None) ->
        Shape
    Static: get_color(color:str | {'all','any'}, game:Game |
        None = None) -> Shape
    Static: get_column(column:int, game:Game | None = None) ->
        Shape
    Static: polygon(vertices:Shape | list[Tile] | Tile, *
        extra_vertices:Tile) -> Shape
        # CRITICAL: polygon() connects consecutive vertices with
        # Line(), so every
        # consecutive vertex pair must be collinear on a hex axis.
        # If vertices are not hex-collinear, draw each edge as a separate Line(
        # start, direction=..., length=...).

# ------
Class: Tile (inherits Shape)
    Method: __init__(row:int, column:int, game:Game | None =
        None)
    Notes: negative indices wrap from opposite edge.
    Method: neighbor(direction:str) -> Tile        #
        direction in DIRECTIONS only -- 'left'/'right' are NOT
        valid here
    Method: on_board() -> bool
    Property: color:str
    Property: column:int
    Property: row:int
    Property: offset:tuple[int,int]                # (
        row,col) 1-based

# ------
Class: Line (inherits Shape)
        # Note creating a line from two tiles is only valid
        if the tiles lie on a straight hex direction
    Method: __init__(start_tile:Tile, end_tile:Tile | None =
        None,
                direction:str | None = None, length:int |
        None = None,
                end_tiles:Shape | None = None,
                include_start_tile:bool = True,
                include_end_tile:bool = True)
    Param options: direction in DIRECTIONS only ('left'/'right
        ' NOT valid here); if end_tile supplied, direction/
        length ignored.
    Notes: If direction is given but length is omitted (None),
        the line extends to the board edge.
        Only specify length when you need a shorter line.
    # CRITICAL: Line(start, end) raises an error if the two
        tiles are not collinear
        # on a hex axis (i.e. do NOT share the same q, r, or s
        cube coordinate).
        # Prefer Line(start, direction=..., length=...)
        # for reliable line creation.
    Method: parallel(shift_direction:str, spacing:int) -> Line
        # shift direction in {'left','right'} OR DIRECTIONS
        OR None
    Method: draw(color:str) -> Line
        # inherited auto-paints

# ------
Class: Circle (inherits Shape)          # The created Circle
        has a hollow center
    Method: __init__(center_tile:Tile, radius:int = 1)
    Method: draw(color:str) -> Shape    # inherited auto-
        paints

# ------
Class: Triangle (inherits Shape)        # The created
        triangle has only sides with the center being hollow
    Method: __init__(start_tile:Tile, point:start_tile_type
        :str, side_length:int = 2)
    Param options: point in {'left','right'}; start_tile_type
        in {'side', 'top','bottom'}.
    # CRITICAL: side_length must be >= 2. For a single-tile
        annotation, use Tile.draw() instead.
    Method: draw(color:str) -> Triangle    # inherited auto-
        paints

Library Generation Pipeline Prompts

Stage 1: API discovery.

You have a minimal base library:

{base_lib_docs}

Analyze these {len(batch)} instructions and design NEW methods
    to extend the base library:
{io_note}
{format_instructions(batch)}

OUTPUT FORMAT:

Class: ClassName
    Method: method_name(param: type) -> ReturnType
        Description: [1 sentence - what it does]
        Rationale: [What problem this solves + how often this
            pattern appears in the instructions]
        Necessity: [Why users need this abstraction]


DESIGN CRITERIA:
Each method must:
- Address a complex pattern found in the instructions above
- Be non-trivial - users would struggle to implement correctly
- Involve non-trivial algorithms or domain-specific logic
- Require knowledge of internal data structures or complex
    rules


NOT:
- Easily composable from existing methods
- Simple wrapper or utility
- Simple property access, collection builders, thin wrappers

Quality > Quantity. Propose only methods meeting all criteria.

Class: ClassName                                      # optional brief
    role comment
    # -- Category Name ------------------------        # group related
        methods
    Method: method_name(param1:type, param2:type) -> ReturnType
        # brief inline note
        Param options: param1 in {'val1','val2'} OR SOME_CONSTANT
        Notes: One sentence max. Only when inline comment isn't
            enough.
        # CRITICAL: warning about non-obvious constraint or common
            mistake
    ...


    RULES:
    1. Document ONLY public methods (no _ prefix). Skip __repr__,
        __str__, __hash__, __eq__.
    2. Be CONCISE: prefer inline `#` comments over multi-line
        Notes.
    3. GROUP methods by function (e.g. 'Drawing/painting', '
        Navigation'). Use `# --` headers.
    4. Mark methods that modify state with `# auto-paints` or `#
        mutates` inline.
    5. Add `# CRITICAL:` warnings for methods that are easy to
        misuse.
    6. Use compact set notation for param options.

Method implementation

Implement the NEW methods from this API proposal{
    domain_context}.

API PROPOSAL:
{api_proposal}

BASE LIBRARY (reference):
...

OUTPUT FORMAT - Use this exact structure:

# CLASS: ClassName
def method_name(self, param1: type1, param2: type2) ->
    ReturnType:
    '''Brief description.

    Args: ...
    Returns: ...
    Examples:
    >>> obj = ClassName(...)
    >>> obj.method_name(arg1, arg2)
    expected_result
    '''
    # Implementation
    pass


REQUIREMENTS:
- Complete implementations with docstrings
- Each method MUST include at least one doctest example in its
    docstring
- Handle edge cases (None, empty, out-of-bounds)
- Match base library coding style

Provide ONLY the new method implementations.

Documentation spec generation

Write the API spec section for class `{class_name}`{
    domain_context}.

CLASS SOURCE CODE:
...

EXAMPLE INSTRUCTIONS this library handles:
{instructions_text}

OUTPUT FORMAT -- produce ONLY a Class section like this: