betaFeedback

WorksComputer Science

Program Distillation: Transferring LLM Capability from System 2 to System 0

Abstract

Recent studies leverage large language models (LLMs) to construct executable programs through environment interaction, feedback analysis, and code revision. However, prior work has mainly treated LLMs as optimizers, overlooking their ability to transfer knowledge across system levels. We propose Program Distillation (PD), a framework that formalizes the transfer of abilities from System 2 to System 0. During learning, the LLM provides task knowledge and draft strategies; at deployment, a student program executes the resulting policy. We show that this approach can improve performance while reducing inference time. We introduce PDBench, a benchmark comprising 27 discrete- and continuous-control tasks with fixed initial policies, and PDMetaHarness, a model-harness-independent meta-system for monitoring and guiding the learning process. The strongest model achieves a normalized mean score of 0.728, surpassing PPO (0.656) and Jev-like decision models (0.239), while running exclusively on CPUs and reducing inference costs. Our analysis shows that performance is largely driven by the first breakthrough during learning, and structural modifications have a slightly lower success rate than parameter modifications but typically yield larger gains. We further introduce style distillation, which enables interpretable stylistic control over the learning process. Overall, we present an analytical framework for PD and demonstrate its practical viability.

CCS Concepts

• Computing methodologies → Artificial intelligence; Natural language processing; Machine learning.

Introduction

Recent studies have used large language models (LLMs) to construct executable policies [17]. Given an initial program, an LLM interprets task rules and iteratively develops a policy through environment interaction, feedback analysis, and code revision [9, 11, 14, 18, 33]. The resulting program encodes task experience in its structure, rules, and parameters, and can be executed in sequential decision-making tasks.

However, prior work has primarily treated LLMs as program optimizers that generate, search for, and improve programs, overlooking their potential to transfer knowledge across system levels.1

We propose program distillation (PD), which formalizes this process as knowledge distillation from System 2 to System 0. System 2 denotes deliberative, slow thinking, whereas System 0 refers to a pre-cognitive, embodied process that does not require explicit neural processing [28]. In conventional knowledge distillation, teacher guidance is used to train a student and improve its capabilities [10]. Prior work has demonstrated distillation within the same system, including the transfer of decision-making capabilities from System 1 to System 1 [25] and reasoning capabilities from System 2 to System 2 [8]. Other approaches transfer capabilities across systems. For example, System 2-to-System 1 distillation trains a model to respond directly based on the outputs of deliberative computation [5, 36]. By contrast, PD transfers capabilities from System 2 to System 0.

During learning, a System 2 LLM deliberates to develop a task-specific control strategy. At deployment, this strategy is embodied in a task-specific System 0 program, enabling repeated execution with low computational overhead and explicit, interpretable logic. Like knowledge distillation, PD aims to transfer capabilities within or across systems to support efficient deployment and inference on edge devices. More broadly, this perspective aligns with calls to return to System 1 [12]. Both approaches seek high capability, low latency, structured outputs, and reliable execution at low cost.

Figure 1 illustrates our architecture, in which the LLM serves as both teacher and learning optimizer. As a teacher, it contributes task knowledge and draws on its parametric knowledge to propose control strategies. As a learning optimizer, it organizes experiments, invokes tools, and interprets feedback. To support this architecture, we develop two complementary components: PDBench and PDMetaHarness.

Figure 1. Program distillation overview. A frozen System 2 LLM serves as both the distillation teacher and the learning optimizer. As a teacher, it supplies task-specific knowledge and uses its parametric knowledge to propose control strategies; as an optimizer, it organizes experiments, invokes tools, and interprets feedback. The environment evaluates candidate programs, and the final program is frozen and deployed as a System 0 policy.

PDBench is built on 27 Gym tasks, comprising 11 discrete-action and 16 continuous-action environments. For each task, we specify the available state fields, action bounds, environment configuration, scoring objective, and initial policy program. The model’s goal is to improve the initial policy and achieve the highest normalized score. PDMetaHarness is a meta-control system that operates independently of the model and harness. It monitors the learning workspace, automatically steers execution, and provides follow-up guidance. It also enables the LLM to serve as both a distillation teacher and a learning optimizer, thereby ensuring the effective execution of the distillation process. Together, these components allow different models and harnesses to follow a common workflow across environments and optimization objectives, enabling fair comparisons.

We evaluate ten LLMs and five additional harnesses on PDBench across 27 discrete- and continuous-control tasks. The best standard learning configuration achieves a normalized mean score of 0.728, exceeding the score of 0.656 achieved by the PPO baseline using conventional reinforcement learning and the score of 0.239 achieved by Jev-like decision models. It also achieves this performance in less time than all non-PD baselines when inference is performed exclusively on a CPU, demonstrating that PD is a more effective and cost-efficient alternative.

Trajectory analysis shows that performance is largely driven by the first breakthrough during learning. Structural modifications have a slightly lower success rate than parameter modifications, but yield greater gains. We also introduce a distillation strategy called style distillation, which injects interpretable, stylized guidance during distillation. Our results further suggest that some tasks may have a unique optimal policy. Overall, these results demonstrate the practical viability of PD.

Our contributions are as follows:

  1. Knowledge distillation interpretation. We formulate PD as knowledge distillation, with a System 2 LLM as the teacher and a System 0 program as the student.

  2. Infrastructure for PD training and evaluation. PDBench enables fair comparisons of models, base harnesses, and learning-control policies. PDMetaHarness coordinates the learning process and maintains effective system execution through adaptive guidance.

  3. Cross-system leading performance. We show that PD achieves the highest performance while requiring the shortest inference time compared with conventional RL and Jev-like decision models.

  4. PD optimization strategies and style distillation. We introduce optimization strategies that further advance the performance frontier, as well as a style-distillation approach for interpretable control of model behavior.

  5. Trajectory insights from large-scale learning. We show how performance gains in PD relate to the first breakthrough and structural modifications during learning.

Methodology

Program Distillation

We formulate PD as a form of capability distillation from System 2 to System 0 [28]. An LLM with fixed parameters serves as the teacher, and a program serves as the student. Through task-specific reasoning and experimentation in the environment, it transfers task-relevant execution capabilities into standalone decision rules that can be deployed on edge devices. Complex reasoning is concentrated in the learning phase, while the student acts continuously at deployment time with low online computational cost. PD thus involves two related transformations. First, it changes the representation of capability: knowledge in an LLM is converted into the structure and parameters of a specialized program. Second, it changes the mode of execution: decisions that require online model inference are replaced by direct execution of a local program.

Correspondence to conventional distillation. Given a teacher TθT_{\theta}, a student SϕS_{\phi}, and transfer data D\mathcal{D}, a common distillation objective accounts for both task supervision and teacher guidance [10]:

ϕ⋆∈arg min⁡ϕ[Ltask(Sϕ;D)+λLdist(Sϕ,Tθ;D)].(1)\phi^{\star} \in\underset{\phi}{\operatorname{arg\,min}}\left[\mathcal{L}_{\mathrm{task}}(S_{\phi};\mathcal{D})+\lambda\mathcal{L}_{\mathrm{dist}}(S_{\phi},T_{\theta};\mathcal{D})\right]. \tag*{(1)}

Here, Ltask\mathcal{L}_{\mathrm{task}} denotes task supervision, Ldist\mathcal{L}_{\mathrm{dist}} denotes teacher guidance, and λ≥0\lambda\ge0 controls the trade-off between them. The teacher parameters θ\theta are fixed, whereas the student parameters ϕ\phi are learnable. In PD, the student is a fixed policy program SϕS_{\phi}. Outcomes from the environment provide task supervision, while the teacher draws on its existing knowledge to reason about effective rules and code, then revises the student’s policy program. The overall principle is similar to that of conventional distillation.

System capability and speed. We distinguish two dimensions in our objective: decision capability, which describes how well a system performs a task, and execution speed, which measures the time from receiving an observation to producing a usable action.

Let the task define a metric for which higher values are better, and let ff denote either the teacher TT or the student SS. We define

C(f)=Eξ[Score⁡(τf,ξ(0))].(2)C(f)=\mathbb{E}_{\xi}\left[\operatorname{Score}\left(\tau_{f,\xi}^{(0)}\right)\right]. \tag*{(2)}

Here, τf,ξ(0)\tau_{f,\xi}^{(0)} is an idealized model outcome under seed ξ\xi, assuming the task is not time-sensitive. Thus, C(f)C(f) represents the system’s decision capability under this idealized setting.

In practical PD applications, however, many systems are time-sensitive, and the inference latency of most deployed models is constrained. We therefore account for a factor Δ\Delta that affects system performance. Actual execution performance is determined by rollout on the real system:

τf,ξ(Δ)=Rollout⁡(E,f;ξ),\tau_{f,\xi}^{(\Delta)} = \operatorname{Rollout}(E,f;\xi),
JΔ(f)=Eξ[Score⁡T(τf,ξ(Δ))]=C(f)−ΛΔ(f).(3)J_{\Delta}(f) = \mathbb{E}_{\xi}\left[\operatorname{Score}_{\mathcal{T}}\left(\tau_{f,\xi}^{(\Delta)}\right)\right] = C(f)-\Lambda_{\Delta}(f). \tag*{(3)}

Here, ΛΔ(f)\Lambda_{\Delta}(f) is a time-related penalty that represents the gap between actual and ideal performance. Accordingly, JΔ(f)J_{\Delta}(f) measures the model’s performance in the real environment.

Finally, we define the following metric for execution speed:

vΔ(f)=1ℓf,ξ(Δ).(4)v_{\Delta}(f)=\frac{1}{\ell_{f,\xi}^{(\Delta)}}. \tag*{(4)}

Here, ℓf,ξ(Δ)\ell_{f,\xi}^{(\Delta)} is the end-to-end inference time for rollout on the real system, and vΔ(f)v_{\Delta}(f) is its reciprocal.

Notably, in time-sensitive decision problems, a student can trade some decision capability for lower response latency. When the benefit of faster responses outweighs the capability gap, deployment performance can exceed that of the online teacher. For example, the difference in actual performance between the teacher and the student can be written as

JΔ(T)−JΔ(S)=[C(T)−C(S)]−[ΛΔ(T)−ΛΔ(S)].(5)J_{\Delta}(T)-J_{\Delta}(S)=[C(T)-C(S)]-[\Lambda_{\Delta}(T)-\Lambda_{\Delta}(S)]. \tag*{(5)}

The left-hand side represents the teacher’s deployment advantage over the student. On the right-hand side, the first term represents the capability gap, and the second represents the temporal gap. When the impact of timing exceeds the capability gap, this expression is negative, yielding JΔ(S)>JΔ(T)J_{\Delta}(S)>J_{\Delta}(T). In this case, the student outperforms the teacher. If we also observe vΔ(S)>vΔ(T)v_{\Delta}(S)>v_{\Delta}(T), the program student achieves better deployment performance through faster responses.

Learning Process

To perform program distillation in a general manner that is independent of the model and harness framework, we design PDMetaHarness, a meta-control system defined as K\mathcal{K}.

Design of PDMetaHarness. PDMetaHarness is defined over a model MθM_{\theta} and a harness HH. It guides the model MθM_{\theta} to act as both the teacher TθT_{\theta} and the optimizer OθO_{\theta}.

(Tθ,Oθ)=K(Mθ,H).(6)(T_{\theta},O_{\theta})=\mathcal{K}(M_{\theta},H). \tag*{(6)}

MθM_{\theta} is an LLM with fixed parameters. The base harness HH handles model calls, tool interactions, and session execution. The meta-control rules K\mathcal{K} comprise common user control instructions, Steering and Followup. In our setting, PDMetaHarness triggers these automatically based on feedback from the learning workspace context CC. In practical applications, it can handle issues such as stalled progress and premature termination without user involvement. Concretely, Steer(C)(C) monitors the workspace during each distillation step and can inject contextual prompts when learning stalls, whereas Follow(C)(C) reopens the run with the existing context after the model stops automatically.

Algorithm 1 Feedback-Driven Learning Process
Require: Initial policy Sϕ0S_{\phi_0}, candidate budget BB, environment EE, initial context C0C_0, steering rule Steer, follow-up rule Follow.
Ensure: Final policy Sϕ∗S_{\phi^*}, test score J∗J^*
1: C←C0C \leftarrow C_0, Sϕ←Sϕ0S_{\phi} \leftarrow S_{\phi_0}, J←J^dev(Sϕ)J \leftarrow\hat{J}_{\mathrm{dev}}(S_{\phi}), k←0k \leftarrow0
2: while k<Bk < B do
3: Sϕ′←Distill⁡(Sϕ,Tθ,E,C;Oθ)S'_{\phi} \leftarrow\operatorname{Distill}(S_{\phi},T_{\theta},E,C;O_{\theta}), with Steer(C)(C) during execution
4: J′←J^dev(Sϕ′)J' \leftarrow\hat{J}_{\mathrm{dev}}(S'_{\phi})
5: if Sϕ′S'_{\phi} is valid and J′>JJ' > J then
6: Freeze Sϕ′S_{\phi'} as the current best: Sϕ←Sϕ′S_{\phi} \leftarrow S_{\phi'}, J←J′J \leftarrow J'
7: end if
8: k←k+1k \leftarrow k+1
9: C←Follow⁡(C)C \leftarrow\operatorname{Follow}(C)
10: end while
11: Sϕ∗←SϕS_{\phi^*} \leftarrow S_{\phi}
12: J∗←J^test(Sϕ∗)J^* \leftarrow\hat{J}_{\mathrm{test}}(S_{\phi^*})
13: return Sϕ∗,J∗S_{\phi^*},J^*

Table 1.

Program distillation objective. Let MθM_{\theta} denote the frozen LLM, t∈Tt \in\mathcal{T} a task, and EE the environment. Starting from Sϕ0S_{\phi_0}, the teacher proposes modifications, and the optimizer uses development feedback to decide which modifications to integrate and how to refine them. The teacher’s weights remain fixed, while the program and workspace context are updated continuously. The meta-harness orchestrates the learning loop: the teacher generates candidate programs, and the benchmark evaluates their task performance.

During distillation, the program first generates execution trajectories, and the evaluator then computes the mean task score over a fixed set of seeds:

τSϕ,ξ=Rollout⁡(E,Sϕ;ξ),\tau_{S_{\phi},\xi}=\operatorname{Rollout}(E,S_{\phi};\xi),
J^q(Sϕ)=1∣Dq∣∑ξ∈DqScore⁡T(τSϕ,ξ).(7)\widehat{J}_{q}(S_{\phi})=\frac{1}{|D_q|}\sum_{\xi\in D_q}\operatorname{Score}_{\mathcal{T}}(\tau_{S_{\phi},\xi}). \tag*{(7)}

Here, τSϕ,ξ\tau_{S_{\phi},\xi} is the complete trajectory under seed ξ\xi, and q∈{dev,test}q \in\{\mathrm{dev},\mathrm{test}\} denotes development or test evaluation. Development and test evaluation use independent seed sets DdevD_{\mathrm{dev}} and DtestD_{\mathrm{test}}.

For the maximization tasks considered in this paper, the learning objective is

Sϕ∗∈arg⁡max⁡SϕJ^dev(Sϕ).(8)S_{\phi^*}\in\arg\max_{S_{\phi}}\widehat{J}_{\mathrm{dev}}(S_{\phi}). \tag*{(8)}

where Sϕ∗S_{\phi^*} is the optimal control policy. Algorithm 1 details the complete procedure for executing this learning strategy.

Benchmark Design

To ensure that model optimization and evaluation are conducted under fair conditions, we design PDBench, a standalone benchmark tailored to program distillation.

Design of PDBench. PDBench is built on 27 publicly available classic Gym tasks [30]. The tasks cover rule selection, path planning, feedback control of posture, target control, and multi-joint coordination. They include 11 discrete-action tasks and 16 continuous-action tasks. Each task is initialized with a valid, unoptimized initial policy. The model can access the specified state fields, action ranges, environment configuration, time limits, and scoring objectives, and iteratively optimize the policy based on this information.

CategoryCountRepresentative tasksPrimary decision requirement
Game scoring and rule-based decisions2Pong, BlackjackAdversarial control and rule selection under hidden information
Map navigation and safe paths5Taxi, FrozenLake, CliffWalkingPath planning and risk management on known maps
Postural stability and swinging6CartPole, Acrobot, HumanoidStandupBalancing, swinging up, and postural recovery
Goal reaching and manipulation6MountainCar, LunarLander, PusherReaching goals, landing, and object manipulation
Sustained locomotion and propulsion8BipedalWalker, Hopper, Ant, HumanoidJoint coordination and sustained locomotion
Total27—Discrete and continuous control

Table 1 (PDF p. 4). Task coverage of PDBench. Each of the 27 tasks is assigned to exactly one primary semantic category; counts and representative tasks summarize the five groups.

Score normalization. Raw returns differ in scale across tasks. To enable meaningful comparisons, we use normalized scores to measure model performance:

Score=1N∑t=1NJ~t,J~t=clip⁡(Jt−atbt−at,0,1).(9)\mathrm{Score} = \frac{1}{N}\sum_{t=1}^{N}\widetilde{J}_t,\qquad\widetilde{J}_t = \operatorname{clip}\left(\frac{J_t-a_t}{b_t-a_t},0,1\right). \tag*{(9)}

Here, [at,bt][a_t,b_t] is the valid score range, and JtJ_t is the raw test score of the frozen policy.

Experiment

Experimental Setup

Baseline Methods. We evaluate three categories of methods under the same task contract and normalized scoring. (1) Program distillation (PD): we run ten LLM configurations with MiniSWEAgent as the default harness, and in a harness-replacement study we hold GLM-5.3 fixed while varying the harness across OpenClaw, OpenCode, ClaudeCode, OpenHands, and Codex. This category covers standard PD, teacher replacement, and harness replacement. (2) Jev-like decision models: SemIf-4B [15] and Laya [21] produce an action at every environment step without within-task policy learning. (3) Reinforcement learning: PPO [26], SPO [34], RPO [6], and A3C [20] train neural policies through environment interaction and parameter updates. The same-model PD category spans ten configurations from multiple providers to reduce dependence on any single provider. Table 2 consolidates the suite-level results for all three categories and reports the timing details.

MethodCont. control (16)Discrete control (11)Overall (27)Build time (h)Inference time (s)
Jev-like
SemIf-4B0.2400.2370.239–264960
Laya0.1750.2170.192–33480
Classic RL
PPO0.6030.7330.6565.1147
SPO0.5580.6430.5935.1146
RPO0.6080.5290.5765.6164
A3C0.4900.6160.5429.9122
PD (MiniSWEAgent)
Claude-Opus-50.6330.8640.72823.096
MiMo-V2.6-Pro0.6080.8840.72067.397
GLM-5.3<u>0.618</u>0.8640.71838.0110
DeepSeek-V4.1-Flash0.597<u>0.814</u>0.68613.488
GLM-5.3-Flash0.5630.8070.66231.659
GPT-6-Sol0.5240.8460.65516.674
Claude-Sonnet-50.5410.7800.63817.165
Kimi-K2.60.4480.7780.58234.273
Qwen3.8-27B0.3880.6960.5139.655
Qwen3.6-35B-A3B0.3430.5880.44311.862
PD (GLM-5.3)
OpenClaw0.613<u>0.881</u><u>0.722</u>13.4560
OpenCode0.6110.8430.70613.4127
ClaudeCode0.5800.8210.67811.776
OpenHands0.5600.8450.6769.3105
Codex0.3740.6000.4661.878

Table 2. Performance comparison across the three baseline categories. The best and second-best values across the entire table are shown in bold and underlined, respectively. Build time is cumulative learning-session time. Inference time is the total testing duration for the entire batch; – denotes not applicable.

Basic Configurations. PD executes at most 30 candidates per iteration and at most 5 standard follow-ups, retaining the best-performing strategy according to development-set performance. The Classic RL baseline uses an MLP, with 2 hidden layers of 64 tanh units in each network. All reinforcement learning experiments use training seed 42 and run for at most 1,000,000 steps per task. The checkpoint with the best development-set performance is retained as the model. The Jev-like model is not trained and is run using the officially released weights. Training of the PD model calls the LLM API, and testing is conducted on a single Intel Core i7-14700KF CPU. Training and inference for Classic RL, as well as inference for Jev-like, are conducted on a single RTX 4070 Ti Super GPU.

RQ1: How Does PD Compare with Other Learning Paradigms?

The main results are shown in Table 2 and the Pareto frontier across paradigms is summarized in Figure 2. Jev-like methods perform poorly: SemIf-4B and Laya obtain overall scores of only 0.239 and 0.192, respectively. Classic reinforcement learning is substantially stronger, with PPO achieving the best overall score among the conventional RL baselines at 0.656. In contrast, the best program distillation (PD) configuration reaches 0.728, exceeding PPO by 0.072. This gap indicates that the main advantage of PD comes from more effective interaction and program-construction strategies rather than from policy optimization alone.

Figure 2. Normalized score versus inference time for the program distillation, reinforcement-learning, and online-decision routes. The best PD configuration attains a normalized score of 0.728 with shorter inference time than PPO, A3C, SemIf-4B, and Laya. The horizontal axis is logarithmic and runs from slower to faster.

RQ2: How Do LLM MθM_{\theta} and Harness HH Affect Student SϕS_{\phi} Performance?

Across the ten model configurations in Table 2, Claude-Opus-5 achieves the highest overall score of 0.728, followed by MiMo-V2.6-Pro and GLM-5.3 with scores of 0.720 and 0.718, respectively. All three substantially outperform the unoptimized initial policy, whose average score is only 0.204. The gains are also differentiated by action type: Claude-Opus-5 performs best on continuous control with a score of 0.633, whereas MiMo-V2.6-Pro leads on discrete control with 0.884. This specialization suggests that model capability is not uniformly transferred across action spaces, and that the strongest overall configuration is determined by the balance between continuous execution and discrete decision-making.

The harness comparison further shows that the execution substrate is a major determinant of performance. With GLM fixed, changing the base harness produces overall scores ranging from 0.466 to 0.722. OpenClaw achieves the best overall score of 0.722 and the strongest discrete-control score of 0.881, while MiniSWEAgent obtains the best continuous-control score of 0.618 and a comparable overall score of 0.718. The large spread across harnesses indicates that tool availability and session execution procedures can be as important as the underlying model when constructing executable programs.

RQ3: How Do Meta-Control Methods K\mathcal{K} Affect Student SϕS_{\phi} Performance?

Other Control Methods. We fix the model to GLM-5.3 in the ablation study and use MiniSWEAgent to compare four alternative meta-harness designs.

Meta: Cross-task experience transfer. Reusable experience is extracted from the development checkpoints of completed tasks and injected as a skill into the context of the target task.

MethodContin. control (16)Discrete control (11)Overall (27)Build time (h)Inference time (s)
Alternatives
Meta0.5010.8520.644236.395
History0.5290.8360.654207.2112
Population0.5690.8660.690176.5124
Force0.5960.8720.709121.91567
PDMetaHarness
Standard0.6180.8640.71838.0110
Reset0.6440.8890.744172.0133
Swarm0.6520.8820.746400.3127

Table 3. Performance and cost of the four alternative learning-control paths (Meta, History, Population, and Force) and the Standard, Reset, and Swarm PDMetaHarness strategies. Scores are test-evaluation results after development-set selection for each task; costs include the initial execution of each alternative and all additional reruns following failures.

History: Historical policy comparison. Each candidate is paired in turn with all previously accepted policies on the same complete set of development seeds. For each historical policy, the candidate must achieve a strictly higher mean development score and a higher task score on at least half of the development seeds.

Population: Population search from a shared parent. In each generation, two isolated candidates are generated from the same parent policy. After completing the official development evaluation, the candidate with the higher raw development score is selected. At most one candidate advances per generation.

Force: Forced continuation with context retention. The agent is forced to run for 30 rounds. If it declares completion early, the controller always injects a requirement to continue learning, while retaining the same session context throughout.

Advanced Strategies. In addition to these four alternative construction paths, we investigate two execution strategies on the current meta-harness that increase model performance by allocating more training effort.

Reset: Continuation with a fresh context. After each run, the policy at the end of that run is used as the initial policy for the next run. Only the current best policy and development evaluation records are carried over; the original conversational context is not retained.

Swarm: Parallel repeated learning. Five runs are launched in parallel using the same learning protocol. For each task, the result with the highest raw development score is selected, and we report the results obtained by selecting the best across all five runs.

Results. The table summarizes the learning control paths and the results of development-set selection. The four alternatives achieve overall scores between 0.644 and 0.709, none surpassing our original PDMetaHarness. With more extensive training, Reset and Swarm achieve overall scores of 0.744 and 0.746, respectively, despite incurring higher training costs. These results represent improvements of approximately 3.6% and 3.9% over the standard method. The results show that more extensive training can further improve performance. In practice, Reset may offer a favorable trade-off between performance gains and training cost.

ModelContin. control (16)Discrete control (11)Overall (27)Build time (h)Inference time (s)
GLM-5.3 (single)0.6180.8640.71838.0110
GLM-5.3 + DeepSeek-V4.1-Flash0.5940.8650.70549.9101
GLM-5.3 + Kimi-K2.60.5010.8480.64243.8116
GLM-5.3 + Qwen3.6-35B-A3B0.5500.7850.64625.3134

Table 4. Test-evaluation performance when the GLM-5.3 tool agent is fixed as the optimizer and only the distillation teacher is replaced. The continuous and discrete columns are grouped by action type, so that corresponding settings can be compared directly across teachers. Failed tasks still enter test evaluation through their frozen submissions rather than being discarded, and the overall scores are computed over all 27 tasks.

RQ4: Can the Capability of Teacher TθT_{\theta} Be Distinguished from That of Optimizer OθO_{\theta}?

To investigate the role of the distillation teacher, we design a distinctive isolation scheme that separates the effects of different models and then uses multiple models in combination to solve each problem. Under the standard setting, GLM-5.3 is fixed as the optimizer, while only the teacher used to distill candidate programs is replaced. This setup separates the model that proposes programs from the model that conducts experiments, allowing the capability of a model as a distillation teacher to be analyzed independently.

Table 4 shows a clear positive relationship between the capability of the distillation teacher and the final evaluation performance. Measured by the standalone PD total score in Table 2, GLM-5.3 and DeepSeek-V4.1-Flash achieve relatively strong overall performance, whereas Kimi-K2.6 and Qwen3.6-35B-A3B perform relatively poorly overall, indicating a substantial generational gap between the two groups of models.

Although this gap is reduced after replacing the optimizer with GLM-5.3, it remains clearly identifiable overall. The two models in the higher-performing group achieve scores of 0.718 and 0.705, while those in the lower-performing group achieve 0.642 and 0.646. These results are consistent with the expectation that the capabilities of the LLM teacher are distilled into the student program: stronger distillation teachers provide more useful task priors, control structures, and parameter candidates. After passing through the same filtering process, the resulting student programs also achieve better evaluation results on the test set. This shows that a pure optimizer view cannot fully explain teacher capability, whereas isolating the teacher role can.

RQ5: How Does the Learning Process Justify the Distillation View?

Capability Acquisition by Models Is Breakthrough-Driven. Table 2 reports the execution results for 27 tasks across 10 models. After the initial policy, these runs made a total of 3,643 valid attempts, and 244 runs achieved positive gains. We ask whether models acquire capabilities through breakthroughs or gradual improvement.

To answer this question, Figure 3(a) presents the main result: the first improvement in policy performance accounts for an average of 68.7% of the total gain over a run, while the first three improvements account for an average of 87.6%. The across-run distribution in Figure 3(b) reveals an even more striking pattern. Based on the median, the first improvement alone accounts for 90.9% of a run’s total gain. Moreover, 31.6% of the runs achieve 100% of their total gain in a single attempt, indicating that nearly all gains in many runs come from this one event. These results show that learning does not typically proceed through steady, incremental improvements along a single trajectory. Instead, it is usually characterized by a sharp breakthrough, with gains concentrated almost entirely in the model’s first policy improvement. Models almost always improve through sudden insight rather than gradual accumulation, so capability acquisition is more one-shot than repeated tuning.

Figure 3. Contribution of successive breakthroughs. (a) Mean share of total development gain by breakthrough order, with 95% bootstrap intervals and the cumulative share; (b) Distribution of the first-breakthrough share across 244 positive-gain runs.

Structural Modifications Are Low-Hit but High-Reward. We divide policy modifications into two categories, structural modifications and parameter modifications, and examine them using this binary classification. Specifically, we parse the policy program files before and after each modification and compare their code skeletons, ignoring numerical values. If the skeletons are identical, we classify the modification as a parameter modification; otherwise, we classify it as a structural modification. Structural modifications account for 52.0% of all modifications, while parameter modifications account for 48.0%. We ask how structural and parameter modifications to policies affect model capabilities.

As shown in Figure 4(a), structural and parameter modifications have similar overall success rates: 33.1% of structural modification attempts improve on the previous rule, compared with 34.6% of parameter modification attempts. Structural modifications therefore have a slightly lower success rate. Their advantage lies instead in the magnitude of their effects. Structural and parameter modifications produce gains of at least 25% of the total gain in 12.8% and 2.3% of attempts, respectively. Further, Figure 4(b) shows that the median fraction of total improvement contributed by a successful modification is 9.28% for structural modifications, compared with 1.44% for parameter modifications. At the 75th percentile, the corresponding values are 72.5% and 6.0%. These results indicate that structural modifications account for most of the development gains. Structural modifications thus exhibit a low-hit, high-reward pattern: they succeed slightly less often, but their gains have a substantially heavier tail. Parameter modifications, by contrast, tend to provide small, somewhat more frequent adjustments. As an illustrative example, Figure 5 presents four representative learning trajectories. Overall, performance comes more from structural adjustments than from parameter fine-tuning, and teacher knowledge and capability largely determine these structural outcomes.

Figure 4. Structural modifications as low-hit, high-reward events. (a) Attempt-level exceedance, the probability that an attempt’s payoff reaches a given share of run gain; structural and parameter modifications have similar success rates (33.1% versus 34.6%). (b) Distribution of event share conditional on setting a new record; the median contribution is 9.28% for structural modifications versus 1.44% for parameter modifications, and the 75th-percentile shares are 72.5% and 6.0%. Structural modifications succeed slightly less often but have a substantially heavier payoff tail.

Figure 5. Four representative learning trajectories: an early structural jump, a later structural jump, a parameter jump near the end of the budget, and a case with substantial gain but weak test evaluation. Gray marks candidate development scores, blue points mark accepted candidates that set records, the orange line marks the historical best, and the dashed line marks the initial policy. Gray shading indicates the pre-jump interval without a new record.

RQ6: How Does the Distillation View Enable Exploration Beyond Pure Performance Optimization?

Style Objectives. We propose a style-distillation training method that exploits the LLM teacher’s ability to interpret behavioral characteristics. During distillation, the model is instructed to follow a particular behavioral style. This approach can support the generation of personalized characters or levels in domains such as games.

To illustrate how task performance and behavioral style jointly shape policy construction, and how the resulting programs can be used to generate a population of players with diverse styles, we use Pong, a classic adversarial game in PDBench, as an example. We refer to longer and shorter scoring rallies as conservative and adventurous, respectively, and to lower and higher paddle reversal tendencies as stable and change-oriented. These labels correspond to measurable model behaviors and align closely with intuitive human perceptions of these styles.

We characterize Pong style using two behavioral proxies that can be recomputed from trajectories: rally length and the frequency of actual paddle reversals. Let NvisN_{\mathrm{vis}}, NrallyN_{\mathrm{rally}}, and NrevN_{\mathrm{rev}} denote the number of ball-visible decision steps, completed scoring rallies, and actual paddle reversals, respectively, over the full development match. We define

L(Sϕ)=NvisNrally,V(Sϕ)=1000NrevNvis.(10)L(S_{\phi}) = \frac{N_{\mathrm{vis}}}{N_{\mathrm{rally}}}, \qquad V(S_{\phi}) = 1000\frac{N_{\mathrm{rev}}}{N_{\mathrm{vis}}}. \tag*{(10)}

The conservative/adventurous metric L(Sϕ)L(S_{\phi}) measures the number of visible decision steps per scoring rally, while the stable/change-oriented metric V(Sϕ)V(S_{\phi}) measures the number of actual paddle reversals per thousand visible decision steps.

Training and Capability Ranking. We combine the two style dimensions to form four training-objective quadrants and launch three training runs for each quadrant, for a total of 12 runs. Before each run, the target style is specified to the model through task instructions and skill descriptions. During training, development feedback includes both the score difference and the style metrics.

For each new candidate, the style objective is incorporated into learning through a style-specific prompt. This effectively adds a distinct form of teacher guidance, distilling the teacher’s understanding of a particular policy style. It also introduces a policy-style objective alongside the objective of achieving the highest score. We expect that, if an optimal style exists, the model can learn both objectives simultaneously.

Retaining only the single highest-scoring program would discard behavioral diversity among intermediate candidates. We therefore collect all policies generated during development as evaluation candidates, forming a development style calibration set of 198 policies. We assign each policy to one of four quadrants according to the strength of its style characteristics. We then evaluate every pair of code-based policies through direct matches in Pong and use the outcomes to rank their capabilities. In total, we conduct 2,000 matches and rank the policies using the Elo rating system.

Learning Results. A key advantage of program distillation is that it allows us to investigate whether an optimal control style exists for a given class of tasks.

Figure 6(a) shows a clear pattern in model capability: nearly all policies with the highest Elo ratings fall in the conservative-stable quadrant. Moreover, the more strongly a policy exhibits this style, the higher its capability. Figure 6(b) shows the degree of alignment between model behavior and the target style during training. Because distillation aims to align the model with the teacher while improving its performance, a natural conflict between the style objective and high performance would suggest that achieving high performance while maintaining that style may be infeasible. We find that the conservative-stable style is maintained more consistently than the other styles across all training methods. This result suggests an inherent conflict between the two objectives that training may not be able to resolve simultaneously. Taken together, these results indicate that the conservative-stable style is not only beneficial for improving capability, but may also represent the optimal style. This demonstrates that style distillation enables stylistic learning and identifies the optimal style.

Measured style, internal Elo rank, and style strength plots

Figure 6. Measured style, internal Elo rank, and style strength, reported by assigned training target. (a) Measured style positions, where darker color indicates a higher internal Elo rank. (b) Style strength under each assigned target; across all training methods, the conservative-stable style is maintained more consistently than the other styles, indicating that this target is easier to preserve during training.

Related Work

Knowledge Distillation

Knowledge distillation transfers capabilities from a computationally expensive teacher to a more efficient student [10]. In sequential decision making, policy distillation compresses a teacher policy or consolidates multiple expert policies into a single student [25]. System 2 distillation similarly transfers explicit reasoning capabilities

to direct-answer models, thereby reducing inference-time computation [36]. Related studies distill neural policies into symbolic programs. PIRL searches for interpretable policies in a domain-specific language, guided by a neural policy [31], while VIPER extracts verifiable decision trees from a teacher policy and its value function [3]. Code as Policies further shows that LLMs can generate executable programs that invoke perception and control interfaces [17]. Our work extends these ideas to interactive policy construction from System 2 to System 0. The LLM supplies task knowledge and proposes or revises policy code, while the resulting program preserves the acquired behavior and executes independently at deployment.

Feedback-Driven Program Evolution

Feedback-driven methods improve behavior or executable artifacts without updating model parameters. Self-Refine revises outputs using model-generated feedback, whereas Reflexion records linguistic reflections from previous attempts to guide subsequent behavior [19, 27]. FunSearch and AlphaEvolve combine LLM-based generation with automated evaluation to search over programs [22, 24]. EoH evolves program descriptions and implementations, while ReEvo exploits performance differences among candidates to guide further search [18, 35]. Several studies apply this paradigm to policy code. LangProp uses execution errors and task feedback to optimize policies [11], and Generative Code Optimization revises Atari policies based on trajectories and linguistic feedback [14]. PromptPO generates and selects Python policies from trajectory feedback, achieving strong sample efficiency across multiple tasks [9]. These studies highlight the importance of incorporating both the strategies being optimized and the full environmental context into the feedback process, an idea closely related to the concept of a harness. However, no standardized benchmark currently enables fair comparisons across different approaches. Consequently, it remains difficult to disentangle the contributions of the underlying model from those of the harness architecture. This limitation motivates the need for a unified, model- and harness-agnostic meta-system that organizes the entire harness-learning process systematically. Addressing this challenge is a central objective of our work.

Efficient Decision Making and Control

Real-time control requires balancing action quality against inference latency. Real-time reinforcement learning explicitly models the concurrent evolution of agent computation and the environment, exposing limitations of simulators that wait for each action before proceeding [23]. Studies of LLM-based decision making likewise report a trade-off between response speed and decision quality [5, 13]. Hierarchical systems mitigate this trade-off by assigning high-level reasoning to LLMs and low-level execution to faster controllers or skills [2, 29]. Action chunking further reduces latency by overlapping action execution with generation [4]. An alternative approach is to compile model capabilities into independently executable programs. Cardiverse optimizes LLM-generated programs to reduce repeated model calls during gameplay [16]. Code to Control synthesizes reactive controllers and optimizes their parameters without requiring LLM calls during execution [1]. Related program-search methods have also been applied to active flow control and traffic signal control [7, 32]. In contrast to the prevailing view that action quality and inference latency are inherently coupled, we theoretically show that this trade-off is not fundamental. By distilling capabilities from System 2 to System 0, our approach produces policies that improve decision quality while reducing inference latency.

Conclusion

We propose Program Distillation (PD), a knowledge distillation framework that converts LLM-guided reasoning into standalone decision programs. PD combines a frozen LLM with iterative program refinement and supports reproducible evaluation through PDBench and PDMetaHarness. Across 27 control tasks, PD achieves a normalized mean score of 0.728, surpassing PPO and online-decision baselines, while producing CPU-executable programs with no model calls at deployment. Further analyses show that stronger teachers yield better programs, early breakthroughs account for most performance gains, and style distillation enables interpretable behavioral control. These results establish PD as an effective and deployable approach for transforming model-guided experimentation into reusable decision policies.

Task Protocol

The experimental environment is fixed to Gymnasium 1.3.0, MuJoCo 3.3.7, Box2D 2.3.10, ALE 0.12.0, and NumPy 2.4.6. Digests of the MuJoCo XML files are stored with the protocol. Table 5 lists the environment ID, action space, decision limit, and scoring anchors for each of the 27 tasks. LunarLander disables wind, standard BipedalWalker disables hardcore terrain, Taxi disables rain and passenger variation, and FrozenLake retains stochastic slipping. Pong uses frame skip 4 and sticky-action probability 0.250. The policy interface prohibits external-module imports, classes, nested helper functions, asynchronous calls, mutable global state, file or network access, pretrained weights, and seed-indexed lookup tables. Gymnasium policies may use only restricted built-ins and approved mathematical operations. Discrete actions must belong to the legal action set. Continuous actions must have the correct dimensionality, contain finite values, and lie within the specified range.

TaskEnvironment IDAction spaceMaximum decision stepsLower / upper anchor
atari_pongALE/Pong-v56 discrete actions100,000; ends at 11 points-11.000 / 11.000
cartpoleCartPole-v12 discrete actions5000.000 / 500.000
mountain_carMountainCar-v03 discrete actions200-200.000 / -80.000
mountain_car_continuousMountainCarContinuous-v01 dimension, range [-1, 1]999-100.000 / 100.000
pendulumPendulum-v11 dimension, range [-2, 2]200-1600.000 / 0.000
acrobotAcrobot-v13 discrete actions500-500.000 / -80.000
lunar_landerLunarLander-v34 discrete actions1,000-300.000 / 300.000
bipedal_walkerBipedalWalker-v34 dimensions, range [-1, 1]1,600-150.000 / 300.000
taxiTaxi-v46 discrete actions200-200.000 / 20.000
frozen_lakeFrozenLake8x8-v14 discrete actions2000.000 / 1.000
blackjackBlackjack-v12 discrete actions32-1.000 / 1.000
cliff_walkingCliffWalking-v14 discrete actions200-20000.000 / -13.000
cliff_walking_slipperyCliffWalkingSlippery-v14 discrete actions200-20000.000 / -13.000
frozen_lake4x4FrozenLake-v14 discrete actions1000.000 / 1.000
lunar_lander_continuousLunarLanderContinuous-v32 dimensions, range [-1, 1]1,000-300.000 / 300.000
bipedal_walker_hardcoreBipedalWalkerHardcore-v34 dimensions, range [-1, 1]2,000-150.000 / 300.000
inverted_pendulumInvertedPendulum-v51 dimension, range [-3, 3]1,0000.000 / 1000.000
inverted_double_pendulumInvertedDoublePendulum-v51 dimension, range [-1, 1]1,0000.000 / 10000.000
reacherReacher-v52 dimensions, range [-1, 1]50-100.000 / 0.000
pusherPusher-v57 dimensions, range [-2, 2]100-200.000 / 0.000
swimmerSwimmer-v52 dimensions, range [-1, 1]1,000-100.000 / 400.000
hopperHopper-v53 dimensions, range [-1, 1]1,000-100.000 / 4000.000
half_cheetahHalfCheetah-v56 dimensions, range [-1, 1]1,000-1000.000 / 12000.000
walker2dWalker2d-v56 dimensions, range [-1, 1]1,000-100.000 / 5000.000
antAnt-v58 dimensions, range [-1, 1]1,000-1000.000 / 6000.000
humanoidHumanoid-v517 dimensions, range [-0.400, 0.400]1,0000.000 / 6000.000
humanoid_standupHumanoidStandup-v517 dimensions, range [-0.400, 0.400]1,0000.000 / 150000.000

Table 5. Complete task protocol. Each task is listed with its environment ID, action space, maximum decision steps, and lower/upper scoring anchors. The gym_ prefix is omitted from task names.

Cumulative Gain over Observed Attempts

Figure 7 reports cumulative gain over observed attempts. Panel (a) shows the mean marginal increment and the cumulative share of total development gain by observed candidate attempt, while panel (b) shows the distribution of cumulative gain at attempts 1, 3, and 10. Gains are front-loaded relative to both the candidate budget and the observed trajectory: models obtain 72.7%–97.4% of their gains within the first 25.0% of the budget, but only 10.1%–64.9% within the first 25.0% of the observed trajectory. Early termination directly affects this difference, because a run with a 30-round budget but only one executed round places its sole experiment at 3.3% of the budget and 100.0% of the observed trajectory.

Figure 7. Cumulative gain over observed attempts. (a) Mean marginal increment and cumulative share of total development gain by observed candidate attempt. (b) Distribution of cumulative gain at attempts 1, 3, and 10.

The Echo Court Local Game

From the Policy Pool to a Desktop Game

Style distillation enables a further use of the resulting policy pool: we design a desktop game around it. The game contains 500 virtual players, each with a distinct policy and a complete Elo ranking. Playing the game locally reproduces the experience of online matchmaking. Echo Court is a local Pong application implemented in Python and pygame. It supports human-versus-policy and two-player games, spectating, rankings, matchmaking, and persistent match records. The application packages source files and JSON definitions for 500 composite players, 197 base policies, and a frozen composition runtime. We combine different composition mechanisms to generate this larger set of 500 players.

Local Rankings and Matchmaking

In a new season, all 500 policy players and new users start at an Elo rating of 100.000. Automatic matchmaking first samples uniformly from the complete candidate set within a rating difference of 50.000. If this set is empty, the range expands to 100.000; if it remains empty, matchmaking stops. When a match is confirmed, both ratings are checked again. Upon completion, ratings are updated using zero-sum Elo with update factor 32 and scale 400. Match records preserve the matchmaking interval, candidate count, both pre-match ratings, and provenance. Figure 8 shows the complete leaderboard for a snapshot.

Complete leaderboard for a snapshot of the 500-player Echo Court league

Figure 8. Complete leaderboard for a snapshot of the 500-player Echo Court league. Players are ordered by Elo, with wins, losses, and mean score difference per game retained.

A 50-Match Experience

We use a virtual player to demonstrate the full game flow. Rally100 is a virtual robot player that uses the unoptimized initial program from the reference commit to control the right paddle. It completes 50 matches through the actual interface flow of entry, matchmaking, confirmation, play, and settlement.

Across the 50 valid matches, Rally100 faced 42 distinct opponents, won 24 and lost 26, and accumulated a score of 430:427 for a mean score difference of +0.060 per match. Its Elo rating moved from 100.000 to 63.998.

The following two screenshots, Figures 9 and 10, were produced by off-screen rendering of the actual pygame window.

Direct LLM Decision Baseline

Execution Protocol

To demonstrate the capability of the model teacher, we used a more aggressive scheme, using an LLM, GLM-5.3, to output structured instructions and using it to directly control the single-step model. The LLM uses the same model-request configuration as the original learning runs. At every environment step, it reads the current state and outputs a JSON action. The executor advances the environment after validating the response; an invalid response triggers another request, and the environment-step count increases only after a legal action is executed. The model continues to make decisions directly throughout this process, with zero within-task training episodes. One episode was ultimately completed on test-evaluation seed 10000 for CartPole-v1.

A new player entering the local game

Figure 9. A new player entering the local game; screenshot produced by off-screen rendering of the actual pygame window.

An actual match in the 50-match fixed-policy experience

Figure 10. An actual match in the 50-match fixed-policy experience between Rally100, which uses the unoptimized initial program from the reference commit to control the right paddle, and a composite player; screenshot produced by off-screen rendering of the actual pygame window.

Score and Inference Cost

The episode remains balanced through the 500-step limit and reaches the maximum CartPole return. Output tokens are used primarily for model inference; under a protocol that waits for an action at every step, cumulative request latency becomes execution time. If 30 episodes each execute 500 steps with the same request overhead as this episode, linear extrapolation gives approximately 281.308 cumulative hours to execute this task. This is a budget estimate under the stated conditions.

MetricResult
CartPole return500.000
Input tokens212633
Output tokens3342490
Cumulative model-request latency (h)9.377

Table 6. Return and execution investment for the single successful CartPole-v1 episode of the direct JSON-decision baseline. Request latencies come from usage records; extrema and the median follow the experiment report.

This result reveals the powerful capability of general-purpose LLMs, far stronger than Jev-like models. However, due to time constraints, it is impossible to fully use such System 2 LLMs for inference. This is precisely the reason we turned the system into System 0.

Details of Model and Ablation Results

Model Heatmap

Figure 11 presents per-task test normalized scores for the ten models across 27 tasks. Rows are grouped into five semantic task categories, and the bar on the left distinguishes discrete and continuous actions. The matrix supports comparison of model-task differences and action-type groups; model means appear at the bottom.

Figure 11. Test normalized scores for ten models across 27 tasks. Rows are grouped into five semantic task categories, the bar on the left distinguishes discrete and continuous actions, and model means appear at the bottom. The matrix supports comparison of model-task differences and action-type groups. Red indicates higher scores and blue lower scores; each cell is labeled to three decimal places.

Ablation Heatmap

Figure 12 reports complete results for all ablation experiments, including the individual results for the Meta, History, Population, and Force categories, as well as Standard, Reset, and Swarm for PDMetaHarness. Interestingly, although these methods do not surpass our Standard method in overall performance, they are actually somewhat complementary to one another. If we select the best-performing methods based on development-set performance and combine these methods, the average can still reach 0.742, which is slightly inferior to the 0.746 of the PDMetaHarness swarm method, but still a large improvement over the original baseline of 0.718. Testing and using more strategies may be a key direction for future research.

Figure 12. The ablation experiments are executed on the normalized scores on each task in the test set, and the bottom shows the average of each method across 27 tasks.

Environment Seed Setting

Basic Rules. If a suitable evaluation score requires running nn times, then for all models, the seeds used for development are 0 to nn, and the seeds used for testing are 10000 to 10000+n10000+n.

Failed Seed Replacement Rules. There are some special cases. That is, the environment may crash because it takes too long or falls into an infinite loop, with Atari-pong being a typical example. This is not a problem with the policy itself, but with the environment setup. We replace seeds where this occurs according to specific mapping rules. Specifically, the run result of the kk-th failed seed is replaced by the result of 100000 + kk. For example: in the History supplementary evaluation, the failed environment seeds 10016, 10020, and 10027 for Atari-pong were replaced by 100000, 100001, and 100002. Backup seeds are assigned independently according to each policy’s own failure evidence and a fixed order.

Token Costs

We aggregate the input and output token count fields separately for each task. All 10 models have 27 trajectories with computable durations. The token counts reflect the usage recorded by the adapter. We did not calculate monetary cost because it depends on model pricing and cache hit rates. The results show that Claude-Opus-5 used the most candidate rounds, 645, while MiMo-V2.6-Pro had the highest consumption of both input and output tokens.

System 0/1/2/3

The System 0/1/2/3 framework adopted in this paper is based on the quad-process cognitive framework [28]. It extends the conventional fast/slow distinction of dual-process theory to four temporal scales: super-fast, fast, slow, and super-slow. Each scale corresponds to a distinct class of cognitive processes, as summarized in Table 8.

ModelCandidate RoundsInput tokens (M)Output tokens (M)
Claude-Opus-564586.4211.897
MiMo-V2.6-Pro404449.3477.091
GLM-5.3453385.1334.052
DeepSeek-V4.1-Flash45763.4321.533
GLM-5.3-Flash324235.4513.789
GPT-6-Sol51261.2300.433
Claude-Sonnet-5425211.9162.584
Kimi-K2.6332113.9533.992
Qwen3.8-27B7859.1542.123
Qwen3.6-35B-A3B92179.4091.331

Table 7. Training token and round investment of ten models in selected sessions. Token units are in millions.

System 0 encompasses pre-cognitive embodied processes that operate on a super-fast timescale. These processes arise directly from physical interactions between an agent’s body and its environment, with morphological computation and passive dynamics serving as their primary mechanisms.

DimensionSystem 0System 1System 2System 3
Temporal scaleSuper-fastFastSlowSuper-slow
Underlying modelPhysical dynamicsWorld modelLanguage modelEmergent symbolic system
RepresentationPre-representationalImplicitExplicit and internalExternal and shared
Cognitive processReflexiveIntuitiveDeliberateCollective
Role of languageNon-linguisticSub-linguisticInner languageExternal language

Table 8. Key characteristics of the System 0/1/2/3 framework.

System 1 encompasses fast, intuitive, and automatic processes associated with world models and implicit representations. It primarily relies on tacit knowledge that is difficult to verbalize, enabling subconscious pattern recognition and immediate responses. Such processes operate mainly at the individual level.

System 2 encompasses slow, deliberate, and effortful reasoning based on language models and explicit internal representations. It supports conscious, step-by-step reasoning and enables the organization of thought through inner language. Such processes operate mainly at the personal level.

System 3 encompasses collective intelligence and symbol emergence on a super-slow timescale. Its central mechanism is collective prediction, through which multiple agents collaboratively construct and refine a shared world model via predictive processing. This process enables external symbolic systems, such as language, cultural norms, and social institutions, to emerge and evolve over time.

Within this framework, we classify chain-of-thought reasoning in LLMs as System 2. [sic] Jev-like models and classical reinforcement learning methods are classified as System 1 because they rely on implicit representations and relatively simple model-based intuitive inference. Finally, executable programs obtained through program distillation are classified as System 0 because their execution requires no online model inference.

Definition and Distinction of Program Distillation

Program Distillation versus Knowledge Distillation

It should be noted that, although we use the term distillation, program distillation (PD), as defined in this paper, is not identical to knowledge distillation in the conventional sense. PD refers to the transfer of capabilities from System 2 to System 0, whereas conventional knowledge distillation typically trains a student to match a teacher’s logits or soft labels on observed inputs. The key distinction is that the student in PD is a program, and its learning does not rely on soft labels. Instead, the student inherits a programmatic representation in which the teacher’s knowledge for solving the control problem has been encoded. Thus, PD differs from conventional knowledge distillation in both the representation of knowledge and the mechanism by which that knowledge is transferred. Nevertheless, the two share the same underlying objective: compressing the decision tendencies of a strong teacher into the behavior of a weaker student.

This perspective on program distillation, together with the introduction of teacher alignment, is essential for understanding our approach. Unlike conventional distillation, the direction of model updates in PD is not determined by gradients. Instead, updates are restricted to the hypothesis space and behavioral priors specified by the teacher. Furthermore, the student does not search for a solution from scratch. Its capabilities do not emerge solely through independent discovery based on task feedback. Rather, they are transferred directly from the teacher through a programmatic representation of the teacher’s knowledge. During deployment, the student also inherits the teacher’s action style in the form of an executable program, as demonstrated by our style distillation strategy.

Formal Differences from Related Work

Recent studies often formulate code as a policy and optimize it directly. While related to our setting, this line of research diverges from our focus in a fundamental way. Rather than attributing performance gains solely to the optimization procedure, our study traces the observed performance gap to the underlying capability that generates the policy. Our core claim is that distillation and optimization serve as coexisting sources of capability, meaning that neither can be overlooked when accounting for simulation performance.

A formal comparison clarifies this distinction. Existing approaches can be formulated as

Oθ=(Mθ,Hspecial),Sϕ′=Learn⁡(Sϕ,E,C;Oθ).(11)\mathcal{O}_{\theta} = (M_{\theta}, H_{\mathrm{special}), \qquad S'_{\phi} = \operatorname{Learn}(S_{\phi}, E, C; \mathcal{O}_{\theta}).} \tag*{(11)}

where MθM_{\theta} denotes the base model, HspecialH_{\mathrm{special}} is a task-specific harness, Oθ\mathcal{O}_{\theta} is the resulting optimization operator, SϕS_{\phi} represents the student policy, EE and CC denote the environment and context, and Learn is the optimization algorithm. Under this paradigm, policy capability stems primarily from optimization guided by a bespoke harness.

In contrast, our formulation introduces an explicit teacher:

(Tθ,Oθ)=K(Mθ,H),Sϕ′=Distill⁡(Sϕ,Tθ,E,C;Oθ).(12)(T_{\theta}, \mathcal{O}_{\theta}) = \mathcal{K}(M_{\theta}, H), \qquad S'_{\phi} = \operatorname{Distill}(S_{\phi}, T_{\theta}, E, C; \mathcal{O}_{\theta}). \tag*{(12)}

where HH represents general knowledge regarding the target behavior, and the operator K\mathcal{K} constructs both the teacher TθT_{\theta} and the optimization operator Oθ\mathcal{O}_{\theta} from the base model and this knowledge. The improved student policy Sϕ′S'_{\phi} is subsequently obtained through joint distillation and optimization. This formulation departs from prior work in two key respects. First, it assigns an explicit role to the teacher, requiring the base model to be structured into a source of transferable behavioral knowledge. Second, it provides a generalizable procedure that eliminates the reliance on a manually engineered, task-specific harness such as HspecialH_{\mathrm{special}}, which is typically required by code-as-policy methods such as Evolution of Heuristics [18]. Consequently, distillation and optimization act as complementary sources of capability, and our analysis emphasizes the former just as much as the latter.

References

  1. [1]Zergham Ahmed, Joshua B Tenenbaum, Chris Bates, and Samuel J Gershman. 2026. Code to Control: Synthesis-Based Parameterized Reactive Controllers. arXiv preprint arXiv:2609.38733 (2026).arxiv.org/abs/2609.38733
  2. [2]Michael Ahn, Anthony Bohn, Noah Brown, Yevgeny Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022).
  3. [3]Osbert Bastani, Yevgeny Pu, and Armando Solar-Lezama. 2018. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems 31 (2018).arxiv.org/abs/1805.08328
  4. [4]Kevin Black, Manuel Galliker, and Sergey Levine. 2026. Real-time execution of action chunking flow policies. Advances in Neural Information Processing Systems 38 (2026), 33383–33407.arxiv.org/abs/2506.07339
  5. [5]Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, and Mingyuan Zhou. 2026. System-1Arena: Which Decisions Can Be Left to System 1? arXiv preprint arXiv:2610.21244 (2026).arxiv.org/abs/2610.21244
  6. [6]Xiazhong Gan, Renye Yan, Zhe Wu, and Junliang Xing. 2024. Reflective policy optimization. arXiv preprint arXiv:2406.03678 (2024).
  7. [7]Paul Garnier, Jonathan Viquerat, and Elie Hachem. 2026. Heuristic Learning for Active Flow Control Using Coding Agents. arXiv preprint arXiv:2607.11565 (2026).
  8. [8]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025).DOI
  9. [9]Stephane Hatgis-Kessell and Emma Brunskill. 2026. When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks? arXiv preprint arXiv:2605.30719 (2026).
  10. [10]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).arxiv.org/abs/1503.02531
  11. [11]Shu Ishida, Gianluca Corrado, George Fedoseev, Hudson Yeo, Lloyd Russell, Jamie Shotton, Joáo F Henriques, and Anthony Hu. 2024. Langprop: A code optimization framework using large language models applied to driving. arXiv preprint arXiv:2401.10314 (2024).
  12. [12]Jev AI. 2026. Jev: The System One Model for Fast, Calibrated AI Decisions. https://jevai.net/articles/what-is-system-one-jev/. Guide.
  13. [13]Hao Kang, Qingru Zhang, Han Cai, Weiyuan Xu, Tushar Krishna, Yihun Du, and Tschay Weissman. 2026. Win fast or lose slow: Balancing speed and accuracy in latency-sensitive decisions of llms. Advances in Neural Information Processing Systems 38 (2026), 150862–150884.arxiv.org/abs/2505.19481
  14. [14]Zhiyi Kuang, Ryan Rong, YuCheng Yuan, and Allen Nie. 2025. Learning Game-Playing Agents with Generative Code Optimization. arXiv preprint arXiv:2508.19506 (2025).arxiv.org/abs/2508.19506
  15. [15]Theo Lee. 2026. Semlf-OpenJev. https://github.com/TheoLeeCJ/Semlf-OpenJev. GitHub repository.
  16. [16]Danrui Li, Sen Zhang, Samuel S Sohn, Kaidong Hu, Muhammad Usman, and Mubbassir Kapadia. 2025. Cardiverse: Harnessing llms for novel card game prototyping. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 29723–29750.arxiv.org/abs/2502.07128
  17. [17]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA). IEEE, 9493–9500.arxiv.org/abs/2209.07753
  18. [18]Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. 2024. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. arXiv preprint arXiv:2401.02051 (2024).arxiv.org/abs/2401.02051
  19. [19]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrami Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems 36 (2023), 46534–46594.arxiv.org/abs/2303.17651
  20. [20]Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. PMLR, 1928–1937.arxiv.org/abs/1602.01783
  21. [21]M. Nandha Kishor. 2026. Laya. https://github.com/NandhaKishorM/laya. GitHub repository.
  22. [22]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Émilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. Alphaveolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131 (2025).arxiv.org/abs/2506.13131
  23. [23]Simon Ramstedt and Chris Pal. 2019. Real-time reinforcement learning. Advances in neural information processing systems 32 (2019).arxiv.org/abs/1911.04448
  24. [24]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawam Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature 625, 7995 (2024), 468–475.
  25. [25]Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. 2015. Policy distillation. arXiv preprint arXiv:1511.06295 (2015).arxiv.org/abs/1511.06295
  26. [26]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).arxiv.org/abs/1707.06347
  27. [27]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36 (2023), 8634–8652.arxiv.org/abs/2303.11366
  28. [28]Tadahiro Taniguchi, Yasushi Hirai, Masahiro Suzuki, Shingo Murata, Takato Horii, and Kazutoshi Tanaka. 2025. System 0/1/2/3: Quad-process theory for multitimesscale embodied collective cognitive systems. Artificial Life 31, 4 (2025), 465–496.
  29. [29]PUBG Ally Team, Irene Chen, Youngin Cho, Seungjun Chung, Jimin Hong, Hyeonbin Hwang, Hyeongjun Im, Insub Im, Jaeseung Jeon, Seohyeon Jung, et al. 2026. PUBG Ally: A Conversational Embodied Agent as an AI Teammate. arXiv e-prints (2026), arXiv–2609.arxiv.org/abs/2609.29837
  30. [30]Mark Towers, Ariel Kwiatkowski, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Kallinteris Andreas, Markus Krimmel, Arjun Kg, Rodrigo Perez-Vicente, et al. 2026. Gymnasium: A standard interface for reinforcement learning environments. Advances in Neural Information Processing Systems 38 (2026).arxiv.org/abs/2407.17032
  31. [31]Abhinav Verma, Vijayavaradhan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. 2018. Programmatically interpretable reinforcement learning. In International conference on machine learning. PMLR, 5045–5054.arxiv.org/abs/1804.02477
  32. [32]Leizhen Wang, Peibo Duan, Zhenlin Qin, Yancheng Ling, Jian Xu, Yue Wang, Hao Wang, and Zhenjiang Ma. 2026. EvoSignal: LLM-Guided Evolutionary Design of Modular Traffic Signal Control Programs. arXiv preprint arXiv:2610.09563 (2026).arxiv.org/abs/2610.09563
  33. [33]Jiayi Wen. 2026. Learning Beyond Gradients. https://trinkle228397.github.io/learning-beyond-gradients/. Blog post.
  34. [34]Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Rening Xu. 2024. Simple policy optimization. arXiv preprint arXiv:2401.16025 (2024).arxiv.org/abs/2401.16025
  35. [35]Haoran Ye, Jiarui Wang, Zhiqiang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. 2024. Reavo: Large language models as hyper-heuristics with reflective evolution. Advances in neural information processing systems 37 (2024), 43571–43608.
  36. [36]Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. 2024. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023 (2024).arxiv.org/abs/2407.06023

Paper details

vlnc:W_64kwv6m4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents