betaFeedback

WorksComputer Science

Interpretable Meta-Policies for Neural Decision Making

Abstract

Deep Reinforcement Learning remains difficult to interpret and align due to the black-box nature of neural networks. While (neuro-)symbolic approaches improve transparency by incorporating symbolic reasoning, applying them directly to low-level actions results in overly complex, unreadable policies. We introduce NEXUS, a hierarchical RL framework that resolves this tension by combining neural skills with symbolic meta-policies. By shifting transparent reasoning to high-level skills rather than atomic actions, NEXUS drastically reduces symbolic complexity without sacrificing environment performance. Object-centric observations enable high-level reasoning and the design of reward functions and meta-policies by interaction with LLMs. We demonstrate that NEXUS agents learn disentangled skills, are interpretable regarding high-level goals, are better aligned to goals described in the game manual, less prone to reward hacking and more robust to game simplifications than deep RL. Finally, we isolate and evaluate the impact of different meta-policy architectures (neural, symbolic, neuro-symbolic) on environmental return and aligned task performance.

Introduction

Deep reinforcement learning (DRL) policies are highly capable ([51, 55, 19]) but operate as black boxes, exhibiting behaviors that are difficult to interpret and

potentially misaligned [54]. This opacity makes identifying and correcting undesirable behaviors a persistent challenge for practitioners [70, 71, 14].

Neuro-symbolic approaches address this issue by combining neural networks for perceptual grounding with symbolic reasoning modules for decision-making [11, 24, 1, 14, 44]. These architectures aim to enhance transparency by representing the policies through symbolic structures that are more readily interpretable. Despite their promise, applying symbolic reasoning directly to low-level action spaces often results in policies of prohibitive complexity (cf. Figure 1 center, Appendix A). The combinatorial explosion of symbolic rules at fine-grained action levels undermines interpretability and scalability.

Figure 1. NEXUS provides high-level interpretability through symbolic reasoning and abstraction. Neural network-based policies are non-modular black box models. While symbolic approaches are fully transparent by leveraging logical rules or decision trees, they quickly become excessively complex if applied to low-level actions. NEXUS abstracts low-level actions into skills and provides transparent high-level decisions.

Hierarchical Reinforcement Learning offers an alternative by abstracting sequences of actions into higher-level skills or options [60, 17, 7]. While the framework provides a natural structure, the resulting options, when learned autonomously, are often entangled or have overlapping goals, and the meta-policy remains opaque, both hindering interpretability.

To address these limitations, we draw inspiration from the dual-process theory of cognition (Kahneman, 2011), where fast, intuitive processes and slower, deliberative reasoning continuously interact. We emulate this interplay in our hierarchical framework NEXUS, Neural EXecution Under Symbols: a low-level neural policy provides efficient, automatic execution (analogous to System 1), while a meta-policy layer of simple, interpretable rule sets (analogous to System 2) constantly guides skill selection, preserving clarity about the reasoning behind the active skill in complex environments (cf. Figure 1, right). We ensure disentangled skills, i.e. each skill corresponds to a distinct situation with clear semantics, by defining their subgoals explicitly through reward functions on objects extracted from the image.

To reduce the manual effort required, we leverage the reasoning capabilities of Large Language Models (LLMs) to assist in identifying those skills, defining the high-level meta-policy functions, and generating the reward signals that guide the training of the low-level neural controllers. Both the low-level skills and the meta-policy QQ-function are learned jointly in an off-policy manner. Overall, our neuro-symbolic design enables inspectable high-level policies without compromising the efficiency of neural agents.

Our primary contributions are as follows:

  • (i) We extend Parallelised Q-Networks to the hierarchical setting for efficient and scalable meta-policy and skill learning (§ 3.1).

  • (ii) We introduce NEXUS and compare three meta-policy variants with varying degrees of interpretability and flexibility (§ 3.1 - § 3.3).

  • (iii) We demonstrate that object-centric rewards and high-level meta-policy functions guide training towards disentangled skills and an interpretable high-level skill selection (§ 4.2).

  • (iv) We show that NEXUS agents are competitive to DRL methods while being less susceptible to reward hacking of environment rewards and more robust to game simplifications. (§ 4.4).

Background

NEXUS builds upon the deep Reinforcement Learning method Parallelised Q-Networks, which is a Deep Q-Networks variant that scales favorably and Hierarchical Reinforcement Learning that decomposes complex tasks into hierarchies of simpler sub-tasks, thereby abstracting actions into skills.

Deep and Parallelised Q-Networks. As sequential decision making framework we adopt Reinforcement Learning (RL), in which an agent learns to interact with an environment in order to maximize cumulative reward. The environment is typically modeled as a Markov Decision Process (MDP), defined by the tuple M=⟨S,A,P,R,γ⟩\mathcal{M} = \langle S, A, P, R, \gamma\rangle, where SS is the set of states, AA is the set of actions, P(s′∣s,a)P(s' \mid s, a) is the transition probability from state ss to state s′s' under action aa, R(s,a)R(s, a) is the reward function and γ∈[0,1)\gamma\in[0, 1) is the discount factor. The goal of the agent is to learn a policy π:S→A\pi: S \to A that maximizes the expected discounted return: Eπ[∑t=0∞γtR(st,at)]\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t)]. In QQ-learning, a value-based RL algorithm, the agent seeks to learn the optimal action-value function Q∗(s,a)Q^*(s, a), which satisfies the Bellman optimality equation: Q∗(s,a)=Es′[R(s,a)+γmax⁡a′Q∗(s′,a′)]Q^*(s, a) = \mathbb{E}_{s'}[R(s, a) + \gamma\max_{a'} Q^*(s', a')]. This function is updated iteratively via the Q-learning update rule: Q(st,at)←Q(st,at)+α[rt+γmax⁡a′Q(st+1,a′)−Q(st,at)]Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha[r_t + \gamma\max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)], where α\alpha is the learning rate. With growing state space SS, it becomes infeasible to store and update a tabular QQ-function. Deep Q-Networks (DQN) [51] address this by using a deep neural network Qθ(s,a)Q_{\theta}(s, a), parameterized by θ\theta, to approximate

Q(s,a)Q(s, a). DQN introduces several key modifications to stabilize learning, including experience replay, where transitions (st,at,rt,st+1)(s_t, a_t, r_t, s_{t+1}) are stored in a replay buffer and sampled randomly to break correlations between consecutive updates and target networks, where a separate network Qθ−(s,a)Q_{\theta^-}(s, a) is used to compute the target value and is updated periodically: yt=rt+γmax⁡a′Qθ−(st+1,a′)y_t = r_t + \gamma\max_{a'} Q_{\theta^-}(s_{t+1}, a').

More recently, Gallici et al. [19] introduced Parallelised Q-Networks (PQN), a simplified variant of DQN that eliminates the use of experience replay and target networks. Instead, PQN leverages a large number of parallel (ideally vectorized) environments and applies normalization techniques to mitigate training instabilities. This high degree of parallelization enables substantially faster training while maintaining performance.

Hierarchical Reinforcement Learning. Hierarchical reinforcement learning (HRL) decomposes complex tasks into simpler sub-tasks, often modeled as temporally extended actions or options [60]. At inference time, a meta policy selects the option, enabling more efficient exploration and transfer in environments with long horizons or sparse rewards. Sub-policies can be learned autonomously using intrinsic objectives [5, 62] or be guided by manually designed reward functions [60, 17]. However, autonomous discovery often suffers from option entanglement or collapse, where multiple options overlap or only a single option is actively used [23, 26]. NEXUS instead enforces its skills to be disentangled by explicitly defining reward functions for each skill, which can be LLM-generated.

Neural Execution Under Symbols

To bridge the gap between well performing deep neural RL and human-interpretable logic, we introduce NEXUS, a hierarchical RL framework that integrates symbolic reasoning with neural skill execution. The overall pipeline is described in Figure 2. At a high level, NEXUS decomposes decision-making into two layers: (1) a high-level meta-policy and (2) a set of low-level neural skill policies optimized for specific sub-goals. At each step, the decision process proceeds as follows. First, object-centric representations are extracted from pixel-based inputs. Second, the meta-policy selects the most suitable skill for the current situation with the goal of maximizing environmental reward. Finally, the chosen skill executes its low-level policy to maximize the corresponding sub-goal. Object-centric representations are crucial for enabling symbolic rule definition and skill-specific reward design, both of which are difficult to obtain without such structured representations. Moreover, they allow LLMs to generate those rules and reward functions.

NEXUS with a neuro-symbolic meta-policy

Figure 2. NEXUS with a neuro-symbolic meta-policy. First, the symbolic state is extracted from the observation. Then, the LLM-generated rules filter potential relevant skills, which the trained meta-policy Q-function selects between. Finally, the selected neural skill outputs the low-level action. Using the same experience, the meta-policy’s Q-function is trained to optimize the environmental reward, while each skill is trained using its own LLM-generated reward, yielding disentangled skills with interpretable objectives.

In the following, we describe, three viable types of meta-policy function: purely neural, purely symbolic, and neuro-symbolic. The first method NEXUS (neural), cf. § 3.1, corresponds to a hierarchical PQN method, which learns both a neural meta-policy and neural skills (based on skill-specific rewards) and which serves as the foundation for subsequent variants. While it offers disentangled skills and high flexibility due to the learned meta-policy, it is missing the ability to reason about why a given sub-goal (and the corresponding skill) is selected in the current step. This shortcoming can be facilitated by replacing the learned meta-policy with a fixed, interpretable function which can either be manually defined or LLM-generated. This variant, NEXUS (symbolic), cf. § 3.2, is fully symbolic at the meta-policy level and thus produces the easiest to interpret meta-policy of the investigated options. Finally, we explore a hybrid approach NEXUS (nesy), cf. § 3.3, which filters candidate skills via predefined symbolic rules, but then selects the final skill using Q-learning on the abstract skill level in ambiguous situations.

Hierarchical PQN

We accommodate the hierarchical learning setup by modeling the environment as two layers of MDPs. On the action level, we define a collection of MDPs M={⟨S,A,P,Rn,γ⟩}n=1N\mathcal{M} = \{\langle\mathcal{S}, \mathcal{A}, P, R_n, \gamma\rangle\}_{n=1}^{N} for NN skills, each associated with a distinct option policy πn∈Π\pi_n \in\Pi that decides over the actual environment actions. The meta-level MDP Mmeta=⟨S,Π,P,Renv,γ⟩\mathcal{M}_{\mathrm{meta}} = \langle\mathcal{S}, \Pi, P, R_{\mathrm{env}}, \gamma\rangle governs option selection through a meta-policy πmeta\pi_{\mathrm{meta}} that selects over the options Π\Pi. Note that the option-level MDPs differ only in their reward functions RnR_n, while the meta-level MDP retains the environment’s original reward RenvR_{\mathrm{env}}.

To enable learning across sub-policies even when they are not actively selected, we adopt off-policy Q-learning. This allows all the skills to learn from trajectories generated by the active skill and meta-policy while optimizing for their respective reward structures. Exploration within each skill is conducted via ϵ\epsilon-greedy action selection. For scalability through vectorization, we build our hierarchical training approach upon PQL with λ\lambda-returns ([19]). However, instead of learning a single shared Q-network QϕQ_\phi as in PQL, we learn separate Q-functions QϕnQ_{\phi_n} for each low-level skill, along with a meta-level Q-network QϕmetaQ_{\phi_{\mathrm{meta}}}.

For updating, we roll out trajectories (s0,…,sT)(s_0, \ldots, s_T) following the global policy by selecting the next active skill πn∈Π\pi_n \in\Pi using the meta-policy Q-function QϕmetaQ_{\phi_{\mathrm{meta}}} and find the most promising action according to the active skill’s Q-function QϕnQ_{\phi_n}:

πn=arg⁡max⁡πn∈Π Qϕmeta(st,πn),a′=arg⁡max⁡a′∈A Qϕn(st,a′).(1)\pi_n = \underset{\pi_n \in\Pi}{\arg\max}\ Q_{\phi_{\mathrm{meta}}}(s_t,\pi_n), \qquad a' = \underset{a' \in\mathcal{A}}{\arg\max}\ Q_{\phi_n}(s_t,a'). \tag*{(1)}

By executing the action in the environment, we obtain the rewards and the next environmental state for the next iteration. We employ ϵ\epsilon-greedy for exploration during both the skill and the action selection.

Next to the environment reward renv,tr_{\mathrm{env},t}, we require skill-specific reward functions rn,tr_{n,t} that are based on the object-centric state sts_t and may be automatically generated by LLMs (cf. Appendix § 17), which we use to compute the λ\lambda-returns recursively back in time (for details, we refer the reader to [19, 10]):

Rn,tλ=rn,t+γ[λRn,t+1λ+(1−λ)max⁡a′∈AQϕn(st+1,a′)],(2)R^\lambda_{n,t} = r_{n,t} + \gamma\left[\lambda R^\lambda_{n,t+1} + (1-\lambda)\max_{a' \in\mathcal{A}} Q_{\phi_n}(s_{t+1},a')\right], \tag*{(2)}

and similarly, for the learned meta-policy using environment rewards:

Renv,tλ=renv,t+γ[λRenv,t+1λ+(1−λ)max⁡πnQϕmeta(st+1,πn)],(3)R^\lambda_{\mathrm{env},t} = r_{\mathrm{env},t} + \gamma\left[\lambda R^\lambda_{\mathrm{env},t+1} + (1-\lambda)\max_{\pi_n} Q_{\phi_{\mathrm{meta}}}(s_{t+1},\pi_n)\right], \tag*{(3)}

or, if sts_t is terminal Rn,tλ=rn,tR^\lambda_{n,t} = r_{n,t} and Renv,tλ=renv,tR^\lambda_{\mathrm{env},t} = r_{\mathrm{env},t}. All learned Q-functions are updated towards their λ\lambda-returns. We provide the full algorithm in Appendix § 9.

Interpretable Meta-Policy Function

While the hierarchical structure allows to identify the active skill, it does not expose the decision process that leads to its selection. To make this process transparent, we introduce interpretable meta-policy functions implemented as rule-based programs. We choose this representation for two key reasons. First, rule-based programs offer simplicity and accessibility since conditional statements can typically be understood and modified even by users with limited programming background. Second, LLMs are highly effective at generating and editing code-like structures due to extensive pretraining on programming data. This enables generation of human-interpretable meta-policies and greatly reduces manual effort. Further details on how we employ LLMs to generate the meta-policy function are available in Appendix § 17.

Formally, instead of relying on a learned Q-function QϕmetaQ_{\phi_{\mathrm{meta}}}, we define a meta-policy πmeta:S→Π\pi_{\mathrm{meta}} : \mathcal{S} \to\Pi as a set of human-readable rules that directly maps the current object-centric state st∈Ss_t \in\mathcal{S} to the selected low-level policy πn∈Π\pi_n \in\Pi:

πn=πmeta(st).(4)\pi_n = \pi_{\mathrm{meta}}(s_t). \tag*{(4)}

We hereby constrain πmeta\pi_{\mathrm{meta}} to a set of rules {(Ak,π(k))}\{(A_k,\pi_{(k)})\}, where each condition set Ak⊆SA_k \subseteq\mathcal{S} defines the state subspace where policy π(k)\pi_{(k)} is triggered. To enable transparent inspection of policy decisions, we require these activation conditions to be pairwise disjoint (i.e., Ai∩Aj=∅A_i \cap A_j = \varnothing for all i≠ji \ne j). This mutual exclusivity ensures that the selected skill can always be unambiguously traced back to the single unique rule that evaluated to true given the current state. Example rules are available in the evaluation section (cf. Listing 1).

Neuro-symbolic Meta-Policy Function

The symbolic meta-policy is a deterministic function that selects mutually exclusive skills based on the rule with the highest priority. This approach may become ineffective in situations where multiple skills are simultaneously applicable. Consider an agent controlling a submarine that can shoot a nearby enemy or surface to replenish oxygen. The most viable rule depends on the specific context: if the oxygen level is critically low, resurfacing may be prioritized; if an enemy is dangerously close, attacking may take precedence. Crafting hard-coded rules to handle all these situations, could lead to growing, less interpretable policies.

As an alternative meta-policy variant aimed at automatically resolving such situations, we also explore a neuro-symbolic (NeSy) approach. Similar to the symbolic meta-policy, this variant maintains a set of high-level, interpretable conditions (e.g., “go to surface if oxygen is low”), represented as a binary condition vector ct∈{0,1}Nc_t \in\{0,1\}^N indicating rule activation at time tt. These rule-sets are again formulated as simple code functions derived from the symbolic meta-policies; however, in this variant, we allow multiple conditions to be active concurrently. The skill selection is then restricted by these conditions: the meta-policy selects the skill with the highest Q-value among those whose condition is active:

πn=arg⁡max⁡πn∈Πt Qϕmeta(st,πn),(5)\pi_n = \underset{\pi_n \in\Pi_t}{\arg\max}\ Q_{\phi_{\mathrm{meta}}}(s_t,\pi_n), \tag*{(5)}

where Πt={πn∈Π∣ct,n=1}\Pi_t = \{\pi_n \in\Pi\mid c_{t,n}=1\}.

where Πt⊆Π\Pi_t \subseteq\Pi denotes the set of skills whose conditions are active at time tt. In practice, this corresponds to a masked argmax in which the Q-values of inactive skills are set to −∞-\infty. This design successfully maintains interpretability at the symbolic level and simultaneously enables the agent to resolve ambiguous scenarios adaptively based on learned preferences. An overview over the entire NEXUS algorithm with the different meta-policy variations is provided in Algorithm 1.

Algorithm 1 NEXUS Variants
Require: Skill policies {πn}n=1N\{\pi_n\}_{n=1}^{N} with Q-functions {Qϕn}n=1N\{Q_{\phi_n}\}_{n=1}^{N}, meta-policy QϕmetaQ_{\phi_{\mathrm{meta}}} (for learned/soft)
1: for each episode do
2: for each environment step (in parallel) do
3: Extract object-centric state sts_t
4: Select skill πn,t\pi_{n,t} as follows:
A) Neural: πn,t=arg⁡max⁡πnQϕmeta(st,πn)\pi_{n,t} = \arg\max_{\pi_n} Q_{\phi_{\mathrm{meta}}}(s_t,\pi_n)
B) Symbolic: πn,t=πmeta(st)\pi_{n,t} = \pi_{\mathrm{meta}}(s_t)
C) NeSy: πn,t=arg⁡max⁡πn:ct,n=1Qϕmeta(st,πn)\pi_{n,t} = \arg\max_{\pi_n:c_{t,n}=1} Q_{\phi_{\mathrm{meta}}}(s_t,\pi_n)
5: Select action ata_t using ϵ\epsilon-greedy from πn,t\pi_{n,t}
6: Execute ata_t and observe transition ({rn,t}n=0N,renv,t,st+1)(\{r_{n,t}\}_{n=0}^{N},r_{\mathrm{env},t},s_{t+1})
7: end for
8: for each gradient step do
9: Compute Rn,tλR^\lambda_{n,t} and Renv,tλR^\lambda_{\mathrm{env},t} (if A or C)
10: Update each QϕnQ_{\phi_n} using skill-specific Rn,tλR^\lambda_{n,t}
11: Update QϕmetaQ_{\phi_{\mathrm{meta}}} using Renv,tλR^\lambda_{\mathrm{env},t} (if A or C)
12: end for
13: end for

Table 1.

Experimental Evaluation

This work aims to develop an object-centric pipeline for agents capable of solving complex environments while maintaining interpretability of high-level goals. We assess this objective by addressing the following research questions:

(Q1) Does NEXUS learn meaningful disentangled skills?

(Q2) Is the high-level skill selection of NEXUS interpretable?

(Q3) Can NEXUS compete with other deep methods?

(Q4) Does NEXUS improve robustness to game simplifications?

Experimental Setup

Our experiments assume access to pre-extracted object representations and attributes. We therefore provide the agent with symbolic environment states directly, allowing for a minimum agnostic to the specific object extraction pipeline. We conduct experiments on JAXAtari ([16]), a JAX-based reimplementation of the Atari Learning Environment (ALE) ([8]), specifically using the games Kangaroo and Seaquest, with object-centric representations similar to OCAtari ([13]). Additionally, we evaluate on Crafter ([22]), using the JAX reimplementation by Matthews et al. ([49]), an environment that integrates elements from the games Minecraft and NetHack (cf. Appendix § 10.1). Finally, for generalization we also test the trained agents on slightly modified versions of the games. These modifications are designed to highlight the sensitivity of standard reinforcement learning agents to game simplifications. For example, the simplification of removing enemies (i.e., sharks and submarines) in Seaquest already leads to a substantial drop in many deep agents performances [15]. We compare the three developed meta-policy variations of NEXUS to the default, non-interpretable PQN [19], and, in case of the Atari games, with the actor-critic PPO [55] as baselines. Additionally, we compare to HPQN, a hierarchical PQN variation that does not employ skill-specific rewards and has a purely neural meta-policy. All results (including baselines) are directly trained on object-centric inputs, which may affect performance compared to image based training. We adhere to the standard frame budgets of both games: 200M frames for JAXAtari and 1B frames for Crafter (cf. Appendix § 10). All metrics shown are aggregated over a total of six seeds.

Interpretability Results

Disentangled skill learning (Q1). We first evaluate whether NEXUS enables efficient learning of meaningful skills. Since each skill requires its own reward function, we leverage the reasoning capabilities of LLMs to generate them based on the game manual, skill definitions, and the object-centric state (cf. Appendix § 17). Figure 3 demonstrates that NEXUS is able to learn most target skills successfully from a single source of off-policy data. This confirms that a skill can learn from another skills action and is not required to be activated to do so. Unlike the baselines, which tend to mostly focus on a single skill that maximizes environment reward (e.g., Shoot Enemies for Seaquest, Handle Threats for Kangaroo), NEXUS approaches

Figure 3. NEXUS learns disentangled skills from off-policy data. In Seaquest (top), the baseline methods mainly focus on shooting enemies, while the NEXUS approaches acquire the target skills more evenly. Similarly, in Kangaroo (bottom), the NEXUS approaches learn ’Move Up’ and ’Collect Fruits’ reliably, while the baselines focus mostly on ’Handle Threats’. Aggregated over 6 seeds, ±1 standard deviation.

def meta_policy(st: State):
    if oxygen_low(st) or \
        all_collected(st.divers):
        return surface
    elif enemy_closeby(st):
        return fight_enemies
    if diver_available(st):
        return rescue_divers
    return fight_enemies

Listing 1: Symbolic meta-policy for Seaquest.

def meta_policy_rules(st: State):
    fight_enemies, rescue_divers, surface = \
        True, False, False
    if diver_available(st):
        rescue_divers = True
    if oxygen_low(st) or \
        all_collected(st.divers):
        surface = True
    return [fight_enemies, rescue_divers, surface]

Listing 2: Neurosymbolic meta-policy rules Seaquest.

promote balanced skill acquisition. We experience a similar picture when evaluating on Crafter (cf. Figure 4). Using the game manual and state description, we query an LLM for essential skills. Recurrent versions of NEXUS are then trained explicitly for these skills and compared to a recurrent PQN baseline. NEXUS approaches are able to learn all five skills simultaneously, while the PQN baseline focuses only on some skills. An additional baseline comparison to PQN with shaped rewards is available in Appendix § 15.1.

Figure 4. NEXUS successfully learns LLM-proposed Crafter skills. Compared to the baseline, NEXUS approaches often converge faster on the proposed skills. Aggregated over 6 seeds, ±1 standard deviation.

Interpretability of NEXUS meta-policies (Q2). Next, we assess the interpretability of NEXUS by inspecting a fixed meta-policy for Seaquest, shown in Listing 1. By abstracting raw observations into object-centric representations and low-level actions into high-level skills, the high-level decision-making process becomes transparent, while the execution of each skill remains neural. Each option has a clear, mutually exclusive activation condition, enabling unambiguous skill selection. For example, the surfacing skill takes precedence whenever oxygen is low or all divers are collected. The resulting meta-policy is fully transparent on the high-level decisions and easily modifiable by humans. Additionally, the simplicity in rule design allows for both manual design by technical users and automated generation by LLMs (cf. Appendix § 17). For neuro-symbolic meta-policies, multiple conditions may be satisfied simultaneously (e.g., an enemy is nearby and oxygen is low; cf. Listing 2), which occurs frequently in practice (cf. Section D.2). In these cases, the rules transparently restrict the set of admissible skills, but the final choice among them is made by the learned meta-policy Q-function and is therefore only partially interpretable. The neuro-symbolic variant thus trades interpretability for flexibility. Figure 5 visualizes the neuro-symbolic meta-policy controlling a Seaquest agent while maintaining transparency regarding the current goals and decision logic. We analyze the neuro-symbolic meta-policy and skill activation frequencies in Appendix § 15.2.

Figure 5. NEXUS (nesy) restricts skill selection through interpretable rules but remains flexible in ambiguous scenarios. Left: enemy_closeby and diver_available both hold; the meta-policy prefers fighting, likely as the diver is near the enemy. Center: with the diver more accessible, rescuing is preferred. Right: with all divers collected and an enemy nearby, surfacing has the higher estimated return.

Policy complexity

To quantify the interpretability of the high-level decision logic without human studies, we measure its simulatability [41] with two metrics: Average Decision Path Length (ADPL) [63], the expected number of condition checks per decision, and Rule Size / Length [35], the number of rules and their total length. Both cover only the symbolic layer, not the neural skills.

Compared to the flat interpretable baseline NUDGE [11] on Seaquest, NEXUS reduces policy complexity by almost an order of magnitude (cf. Table 1, Appendix § 10.3): NUDGE requires 16 rules with 46 literals and an ADPL of 23.56, whereas delegating low-level control to neural skills leaves NEXUS with 2–4 rules, 3–4 predicates and an ADPL of 2.5–3, well within human working-memory bounds (7 ± 2 concepts; [50, 34]).

PolicyRule Size and Length Lakkaraju et al. [35]ADPL Wu et al. [63]
NUDGE (Listing 4)16 rules, 46 predicates23.56 evaluations
NEXUS (Listing 1)4 rules, 4 predicates3 evaluations
NEXUS (Listing 2)2 rules, 3 predicates2.50 evaluations

Table 1 (PDF p. 8). Quantitative interpretability metrics of the symbolic decision logic, comparing NUDGE (action level) and NEXUS (skill level) on Seaquest. NEXUS metrics cover the meta-policy only; low-level skills are neural.

Performance Results

Comparison to other approaches (Q3). We evaluate NEXUS against baselines using two metrics: Game returns and aligned environment goals, which track progress toward goals defined in game manuals (e.g., divers rescued in Seaquest, level completion in Kangaroo). The latter aims to capture the overall alignment with the intended game objectives, which differ from simply maximizing the environmental reward, a phenomenon in deep RL known as reward hacking. As shown in Figure 6,

Figure 6. NEXUS is competitive. In Seaquest, NEXUS outperforms the baselines on both game reward and rescuing divers. In Kangaroo, NEXUS approaches progress farther. Aggregated over 6 seeds, ±1 standard deviation.

NEXUS approaches are competitive to the baselines. In most cases, they achieve comparable or even higher game reward and notably outperform the baselines on the games’ actual goal described in the manual, demonstrating the baselines reward hacking tendency. The results suggest that NEXUS mitigates reward misalignment by incorporating domain priors (game manual) into the decision-making process, while still achieving good overall game performance. We provide additional evaluations on more environments and compare NEXUS to other neuro-symbolic RL methods in Appendix § 15.

Robustness to game simplifications (Q4). Most RL algorithms struggle to adapt to even minor variations in the environment ([15]). Surprisingly, their performances drop even in settings that simplify the game for humans, such as removing deadly threats like enemies and their projectiles in Seaquest and Kangaroo. We assess whether NEXUS agents can generalize to such simplified variants by training the agents on the standard versions of the games and evaluating on the unseen simplifications. In Seaquest and Kangaroo we remove all threats, while in Crafter we remove the need to drink water for survival. Figure 7 presents the results. As expected, the baselines suffer performance degradation under the simplifications or already had low scores. In contrast, the NEXUS variants display varying degrees of robustness. In Atari, the symbolic and neuro-symbolic variants demonstrate improved robustness, with small performance drops in Kangaroo and even gains in Seaquest, while the fully neural variant shows limited robustness. In Crafter, all NEXUS variants maintain or improve performance. We attribute this robustness to symbolic steering of the meta-policy, which deactivates unnecessary skills (handling threats, finding water).

Figure 7. NEXUS remains practical on simplified games. While baseline performance on aligned metrics remains low, neuro-symbolic and fully symbolic NEXUS improve on modified Seaquest (left). Modified kangaroo (middle) reduces performance of symbolic NEXUS, while neuro-symbolic NEXUS remains effective. Crafter performance (right) remains consistent or improves for all NEXUS versions. Aggregated over 6 seeds, ±1 standard deviation.

Discussion

Overall, the empirical results highlight the strengths of our approach across multiple dimensions. Compared to the DRL baselines, NEXUS achieves competitive game rewards and exhibits a strong advantage when evaluated on metrics measuring game progress. Furthermore, we observe improved robustness against environmental simplifications. Taken together, these findings strongly suggest that (neuro-)symbolic meta-policies effectively steer reinforcement learning agents toward better-aligned and more robust policies, all while offering the crucial benefit of enhanced interpretability.

On the choice of the meta-policy. Based on our findings, we can offer concrete recommendations regarding the optimal meta-policy design. The purely neural meta-policy presents the simplest training paradigm as it circumvents the need for a separate symbolic layer and demonstrates strong performance in terms of raw game reward; however, it suffers from limited interpretability and degraded performance when evaluated on simplified games. In contrast, the purely symbolic meta-policy enhances both interpretability and robustness, though it introduces the overhead of rule definition. Because this process can be largely automated using LLMs, this minor investment is typically well justified. Lastly, the neuro-symbolic variant achieves game reward and goal metrics performance comparable to the symbolic meta-policy and is also similarly robust to game simplifications. However, it is more flexible in ambiguous situations. Based on these results, we recommend the purely symbolic meta-policy for practitioners prioritizing high interpretability and alignment without sacrificing performance. Meanwhile, the neuro-symbolic variation is well-suited for settings with frequent ambiguity.

Scope of interpretability. NEXUS provides interpretability at three levels of decreasing transparency: (i) the meta-policy, whose decisions are fully traceable to a single rule in the symbolic variant and restricted to rule-admissible skills in the neuro-symbolic variant, where the final choice in ambiguous states is made by the learned QQ-function; (ii) the skill objectives, defined by explicit, human-readable reward functions; and (iii) the low-level action selection within each skill, which remains neural. Our claims therefore concern why a skill is selected and what it is trained to achieve, not how it is executed. Accordingly, our complexity metrics cover only the symbolic layer, and a user study evaluating human understanding of NEXUS agents is left for future work.

Prior knowledge requirements and manual adaptation. Two components of NEXUS require prior domain knowledge: the specification of skill reward functions and the definition of meta-policy rules for the symbolic and neurosymbolic variants. While these elements could be engineered manually, we leverage LLMs to minimize the need for human expert intervention. Utilizing the prompts detailed in Appendix § 17, the framework mainly requires the raw game manual. After obtaining the generated meta-policy and skill rewards, the user is expected to verify correctness of the output, ensure the code respects framework dependencies, and refine thresholds, rule priorities and reward terms where necessary (cf. Table 5).

Env.ComponentChange from LLM output to final implementationType
SeaquestSkill set4 skills →\to 3; avoid obstacles dropped (death penalty moved into fight)S
Seaquestrcollectr_{\mathrm{collect}}diver proximity count →\to direct divers_collected counterF
Seaquestrfightr_{\mathrm{fight}}enemy count decrease (+0.5) →\to score increase (+1), life lost (−1)S
Seaquestrsurfacer_{\mathrm{surface}}surfacing with 6 divers / low oxygen →\to oxygen refill (+0.1)S
SeaquestMeta-policysame conditions; thresholds set (enemy <40px, oxygen <10), priority reorderedF+S
KangarooSkill set4 skills →\to 3; goal reaching merged into navigateS
Kangaroornavigater_{\mathrm{navigate}}+0.1 per upward step →\to clipped Δy\Delta y, penalty below floor 1, 0 when crashingS
Kangaroorenemiesr_{\mathrm{enemies}}collision (−1) →\to monkey killed (+1), life lost (−1)S
Kangaroorfruitsr_{\mathrm{fruits}}fruit (+0.5) and bell (+0.2) →\to fruit (+1)S
KangarooMeta-policysame structure without goal rule; distance thresholds set (35/50px)F+S
CrafterSkill set5 skills kept (renamed)–
CrafterRewardswritten manuallyM
CrafterMeta-policyhelper predicates implemented; thresholds changed; combat before survival; crafting before collectionF+S
AllNeSy rulesderived manually from the symbolic meta-policy by making conditions non-exclusiveM

Table 5. Human edits to LLM-generated components (GPT-4o, App. F). F: framework adaptation (JAX, state fields); S: semantic change; M: written manually.

Limitations. NEXUS relies on object-centric scene decoders that accurately provide the agent with objects and their positions and sizes from raw images, which we assume to exist in this work. For Atari games, multiple approaches are viable ([37, 42, 40, 12]) with some achieving near-perfect sprite extraction ([59]). For real-world data, recent advancements have significantly improved robustness, with models like SAM2 ([53]) reaching up to 90% J&F\mathit{J}\&\mathit{F} accuracy on zero-shot segmentation. However, even the best object detector will produce misdetections, so we additionally validate the performance of NEXUS under noisy detections (cf. Appendix § 16). The LLM based generation of reward functions and meta-policies require a pre-defined description of the task, i.e. the game’s manual (as done in Wu et al. [64]). Moreover, while the generated functions generally capture valid semantics and are logically consistent, the implementations still require framework adaptations and semantic refinements (cf. Table 5). Prior works on LLM-based reward design (e.g., Xie et al. [65], Ma et al. [46], Kaufmann et al. [31]) have documented recurring issues such as reward misspecification, proxy objective selection, and over- or under-constrained preconditions, highlighting the need for careful verification. NEXUS currently addresses these risks through manual inspection of the generated code. Extending our framework by incorporating automatic mechanisms, e.g., refinement from trajectory feedback and planning, as explored by SCALAR (Zabounidis et al., [69], is a crucial direction for enhancing robustness and scalability. Lastly, the presented approach is based on QQ-learning and therefore limited to discrete action spaces, however, extension to continuous action spaces seems viable, e.g., by adopting an off-policy actor critic instead of ϵ\epsilon-greedy action selection (Lillicrap et al., [39]; Haarnoja et al. [21]).

Related Work

Interpretable and Hierarchical Reinforcement Learning. Interpretability in RL can be introduced at various stages of the pipeline (Glanois et al., [20]), often by deriving symbolic state representations from raw observations via object recognition or segmentation (Li et al., [37]; Locatello et al. [42]; Kirillov et al. [32]; Lin et al. [40]; Delfosse et al. [12]). Such object-centric states have enabled interpretable policies through decision trees (Silva et al., [58]; Likmeta et al. [38]; Delfosse et al. [14]), logic rules (Maes et al., [48]; Akrour et al. [2]; Delfosse et al. [11]), parametric functions (Luo et al., [44]) or programmatic policies and trees (Verma et al., [61]; Anderson et al. [3]; Kohler et al. [33]). Neural and logical policies can also be efficiently combined (Shindo et al., [57]). Complementary efforts introduce hierarchical decompositions, where high-level interpretable policies select among low-level sub-policies, leveraging annotated task sketches (Andreas et al., [4]) or (differentiable) symbolic planning (Leonetti et al., [36]; Yang et al. [66]; Jin et al. [29]; Lyu et al. [45]; Ye et al. [68]). Hierarchical symbolic planning based on object-centric representations has been shown to be beneficial for task transferability (James et al., [28]) and robotics applications (Sharma et al., [56]). Our work extends these directions by enforcing semantic separation of skills, introducing a symbolic meta-policy to improve understandability and integrating LLMs throughout the pipeline.

Reward Shaping and Reward Machines. Prior work has explored structured reward specifications to improve learning efficiency. Potential-based reward shaping (Ng et al., 1999) provides a theoretically grounded method for augmenting rewards while preserving optimality, but relies on manually specified potential functions. Reward machines offer a more expressive automata-based formalism for encoding non-Markovian reward logic (Icarte et al., [27]), with recent work even leveraging LLMs for automated design (Castanyer et al. [9]). However, these approaches typically learn a single value function over a flat action space and have not been evaluated on visually complex domains such as Atari or Crafter. In contrast, our approach learns a separate QQ-function per skill within a hierarchical framework, using object-centric reward specifications to enforce semantic disentanglement.

LLM-guided skill learning. Concurrent to our work, SCALAR (Zabounidis et al., [69] uses LLMs to propose skills as STRIPS-like operators with preconditions, effects and reward functions, sequences them with a symbolic planner, and trains the corresponding options with PPO in a skill-by-skill curriculum, refining the specifications from rollouts via trajectory analysis. While SCALAR targets sample-efficient learning of long prerequisite chains in crafting domains and executes skills in a fixed planned order, NEXUS targets interpretable high-level decision making: its (neuro-)symbolic meta-policy re-selects skills at every step from the object-centric state, all skills are learned concurrently from shared off-policy experience without state resets, and skill rewards are kept separate from the environment reward to enforce disentanglement. The two directions are complementary; SCALAR’s trajectory-based refinement could automate the verification of NEXUS’s LLM-generated rewards and meta-policies.

Relation to the Options framework. HRL has been studied extensively, with the Options framework (Sutton et al., [60]) as the most prominent formulation. NEXUS can be interpreted as an instantiation of the Options framework with intra-option learning, where multiple options are learned simultaneously from shared off-policy experience, using sub-policies guided by option-specific reward functions and coordinated by a neuro-symbolic meta-policy. Key differences arise in temporal abstraction and in the treatment of initiation and termination conditions. Rather than executing an option until termination, the meta-policy selects the active option at every time step, jointly determining activation and termination. The logical rule set as the meta-policy serves as a generalized initiation set, enabled by the object-centric encoding of observations.

Conclusion

In this work, we present NEXUS, a hierarchical RL method that combines high interpretability on meta-policy level with neural, low-level action execution. The evaluation suggest several advantages of our approach. It learns disentangled sub-policies corresponding to identifiable skills, provides high-level interpretable and modular structures for inspection and intervention, reduces reward-hacking through integration of domain priors, and remains robust to environment simplifications where standard deep RL agents fail. We also found that LLMs can successfully be integrated into the NEXUS pipeline to generate skill decompositions, reward functions, and symbolic meta-policies, enabling interpretability and dynamic adaptation to simplified environments. Future work should increase the autonomy of policy adaptation by incorporating mechanisms to detect when new skills or meta-policy updates are required, e.g. via causal world models ([67, 18]), which could enable scaling to complex, open-ended environments. Updates to the skills or meta-policy could be retrieved by re-querying an LLM. In conclusion, this work advances RL interpretability and modularity through symbolic and object-centric reasoning while supporting human-in-the-loop control at the skill level, offering a promising path toward transparent, adaptable, and aligned agents.

Broader Impact. This work advances RL interpretability and modularity by integrating symbolic structure and object-centric reasoning. Beyond enhanced transparency, it enables human-in-the-loop control: practitioners may inspect, modify, or constrain agent behavior at the skill level. We believe that this precise, skill-level alignment and the flexible adaptation are among the most promising avenues opened by this approach. While LLM-generated rewards and meta-policies are viable, they remain susceptible to misspecification and subsequent undesirable actions, making human oversight essential for safety-critical applications.

Motivation - Shortcomings of existing approaches

Figure 8. Existing approaches are not interpretable. Neural RL agents are black-boxes and learn shortcuts. Hierarchical RL remains a black box and typically produces entangled skills. Symbolic agents are transparent, but grow massively with policy complexity.

Interpretable Policy Sizes

We visualize the actual logic policies retrieved from existing interpretable RL methods in Listing 3 and Listing 4. While these policies have maximum transparency on low-level action selection, they are difficult to interpret for humans due to the massive size. NEXUS on the other hand presents a simple solution by incorporating hierarchical abstraction and thus allowing for compact and interpretable high-level policies, cf. Listing 1 and Listing 2.

def play(state):
    if state.Ball_0.prev_x <= 1.24:
        if state.Ball_0.y − state.Enemy_0.prev_y <= −0.41:
            if state.Ball_0.x − state.Ball_0.prev_x <= 0.09:
                if state.Player_0.y − state.Enemy_0.y <= −0.91:
                    if state.Ball_0.x <= 0.73:
                        if state.Player_0.y − state.Ball_0.x <= −1.20:
                            return "LEFT"
                        else:
                            return "RIGHT"
                    else:
                        return "NOOP"
                else:
                    if state.Ball_0.x <= −0.05:
                        return "NOOP"
                    else:
                        if state.Player_0.y − state.Player_0.prev_y <= −0.19:
                            return "NOOP"
                        else:
                            if state.Ball_0.prev_x − state.Ball_0.prev_y <= −0.38:
                                return "NOOP"
                            else:
                                if state.Ball_0.x − state.Ball_0.prev_x <= −0.09:
                                    return "RIGHT"
                                else:
                                    return "RIGHT"
            else:
                return "RIGHT"
        else:
            if state.Ball_0.x − state.Enemy_0.y <= −1.46:
                if state.Player_0.y − state.Ball_0.prev_x <= −0.92:
                    return "LEFT"
                else:
return "RIGHT"
else :
    if state.Player_0.y − state.Ball_0.y <= −0.63:
        if state.Ball_0.prev_x <= −0.05:
            return "LEFT"
        else :
            if state.Player_0.y − state.Ball_0.y <= −0.80:
                return "LEFT"
            else :
                if state.Ball_0.prev_x <= 0.80:
                    return "LEFT"
                else :
                    return "NOOP"
    else :
        if state.Ball_0.x <= −0.07:
            if state.Ball_0.prev_y <= −0.15:
                if state.Ball_0.prev_y <= −1.96:
                    return "LEFT"
                else :
                    return "LEFT"
            else :
                return "NOOP"
        else :
            if state.Ball_0.y <= 1.24:
                if state.Player_0.y − state.Ball_0.y <= −0.48:
                    if state.Ball_0.prev_x <= 0.68:
                        return "NOOP"
                    else :
                        return "NOOP"
                else :
                    if state.Ball_0.x <= 0.54:
                        if state.Player_0.y <= −0.82:
                            return "NOOP"
                        else :
                            if state.Enemy_0.y − state.Enemy_0.prev_y <= −0.09:
                                return "RIGHT"
                            else :
                                return "RIGHT"
                    else :
                        if state.Ball_0.x − state.Ball_0.y <= −0.21:
                            return "NOOP"
                        else :
                            return "RIGHT"
            else :
                if state.Ball_0.y <= 1.54:
                    return "NOOP"
                else :
                    return "LEFT"
    else :
        if state.Player_0.y − state.Ball_0.y <= −0.54:
            if state.Player_0.y − state.Ball_0.y <= −0.88:
                return "LEFT"
            else :
                if state.Ball_0.y <= −1.40:
return "RIGHT"
else:
    return "NOOP"
else:
    if state.Enemy_0.y − state.Enemy_0.prev_y <= 0.93:
        return "RIGHT"
    else:
        return "LEFT"

Listing 3: Pong policy of SCoBots [14]

up_air(X):-oxygen_low(B).
up_divers_collected(X):-all_divers_collected(D).
fire_left(X):-same_depth_enemy(P,E),visible_enemy(E),facing_left(P),right_of_enemy(P,E).
fire_right(X):-same_depth_enemy(P,E),visible_enemy(E),facing_right(P),left_of_enemy(P,E).
left_aim(X):-right_of_enemy(P,E),facing_right(P),same_depth_enemy(P,E),visible_enemy(E).
right_aim(X):-left_of_enemy(P,E),facing_left(P),same_depth_enemy(P,E),visible_enemy(E).
down_aim(X):-higher_than_enemy(P,E),visible_enemy(E).
up_aim(X):-deeper_than_enemy(P,E),visible_enemy(E).
up_evade(X):-close_by_enemy(P,E),same_depth_enemy(P,E),visible_enemy(E).
down_evade(X):-close_by_enemy(P,E),same_depth_enemy(P,E),visible_enemy(E).
up_evade(X):-close_by_missile(P,M),same_depth_missile(P,M),visible_missile(M).
down_evade(X):-close_by_missile(P,M),same_depth_missile(P,M),visible_missile(M).
left_to_diver(X):-right_of_diver(P,D),close_by_diver(P,D),visible_diver(D).
right_to_diver(X):-left_of_diver(P,D),close_by_diver(P,D),visible_diver(D).
up_to_diver(X):-deeper_than_diver(P,D),close_by_diver(P,D),visible_diver(D).
down_to_diver(X):-higher_than_diver(P,D),close_by_diver(P,D),visible_diver(D).

Listing 4: Seaquest policy of NUDGE [11]

logits_noop2 = −0.074*y_agent_1*y_opponent_2 + 0.059*y_agent_1*y_opponent_3 − 0.097*y_agent_4 − 0.16*y_opponent_1*y_opponent_2 − 0.18*y_opponent_2**2 − 0.27*y_opponent_2
    + 0.063*y_opponent_4
logits_up1 = 0.23*y_agent_1**2 + 0.59*y_agent_1*y_agent_2 + 0.4*y_agent_2**2 + 0.11*y_agent_2 − 1.5*y_agent_4**2 − 3.6*y_agent_4 + 0.068*y_opponent_3 + 1.1
logits_down1 = 0.09*x_ball_3 + 0.12*x_ball_4 − 0.21*y_agent_1**2 + 0.12*y_agent_1*y_opponent_1 + 0.27*y_agent_1*y_opponent_2 − 0.43*y_agent_1*y_opponent_3 − 0.28*y_agent_1*y_opponent_4 + 0.13*y_agent_2 + 0.14*y_agent_4**2 + 0.43*y_agent_4 + 0.087*y_ball_3 + 0.15*y_ball_4 + 0.14*y_opponent_1**2 + 0.6*y_opponent_1*y_opponent_2 + 0.61*y_opponent_1 + 0.65*y_opponent_2**2 + 1.1*y_opponent_2 − 0.2*y_opponent_3**2 − 0.26*y_opponent_3*y_opponent_4 − 2.8*y_opponent_3 − 0.085*y_opponent_4**2 − 0.14*y_opponent_4 − 2.3
logits_up2 = 0.063*x_ball_4 − 0.078*y_agent_1 + 0.18*y_agent_2**2 + 0.52*y_agent_2*y_agent_3 + 0.35*y_agent_2*y_opponent_1 + 0.29*y_agent_2*y_opponent_2 + 0.26*y_agent_2 + 0.38*y_agent_3**2 + 0.51*y_agent_3*y_opponent_1 + 0.4*y_agent_3*y_opponent_2 + 1.6*y_agent_3**2 + 8.2*y_agent_4 − 0.085*y_ball_3 + 0.17*y_ball_4 + 0.28*y_opponent_1**2 + 0.28*y_opponent_1*y_opponent_2 + 0.25*y_opponent_1 + 0.11*y_opponent_2**2 + 0.15*y_opponent_2 − 0.074*y_opponent_3 + 0.26
logits_down2 = −0.052*x_ball_1 − 0.068*x_ball_3 − 0.093*x_ball_4 + 0.18*y_agent_1 − 0.17*y_agent_2**2 − 0.49*y_agent_2*y_agent_3 − 0.33*y_agent_2*y_opponent_1 − 0.27*y_agent_2*y_opponent_2 − 0.39*y_agent_2 − 0.35*y_agent_3**2 − 0.48*y_agent_3*y_opponent_1 − 0.4*y_agent_3*y_opponent_2 − 0.38*y_agent_3 + 0.15*y_agent_4**2 + 0.54*y_agent_4 − 0.06*y_ball_1 − 0.064*y_ball_3 − 0.11*y_ball_4 − 0.17*y_opponent_1**2 − 0.28*y_opponent_1*y_opponent_2 − 0.58*y_opponent_1 − 0.13*y_opponent_2**2 − 0.38*y_opponent_2 + 2.2*y_opponent_3 − 0.052*y_opponent_4 − 3.6
action_noop = [exp(logits_noop1) + exp(logits_noop2)] / sum(exp(logits))
action_up = [exp(logits_up1) + exp(logits_up2)] / sum(exp(logits))
action_down = [exp(logits_down1) + exp(logits_down2)] / sum(exp(logits))

caption: Listing 5: Pong policy of INSIGHT (Luo et al. [44])

Hierarchical PQN Algorithm

The complete algorithm for hierarchical PQN with a neural meta-policy is provided in Algorithm 2.

Algorithm 2 Hierarchical PQN

Require: Update period UU, number of parallel environments EE, number of skills NN, exploration probability ϵ\epsilon

Ensure: Learned Q-network parameters {ϕn}n=1N,ϕmeta\{\phi_n\}_{n=1}^{N}, \phi_{\mathrm{meta}}

  1. Initialize Q-network parameters {ϕn}n=1N,ϕmeta\{\phi_n\}_{n=1}^{N}, \phi_{\mathrm{meta}}

  2. Sample initial states s0e∼P0s_0^e \sim P_0 for e∈{0,…,E−1}e \in\{0,\ldots,E-1\}

  3. t←0t \leftarrow0

  4. for each episode do

  5. &nbsp;&nbsp;&nbsp;&nbsp;for all e∈{0,…,E−1}e \in\{0,\ldots,E-1\} in parallel do

  6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample skill πte∼πmeta\pi_t^e \sim\pi_{\mathrm{meta}}

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;With probability ϵ\epsilon: ate∼Unif⁡a_t^e \sim\operatorname{Unif}, else ate∼πtea_t^e \sim\pi_t^e

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample rewards rte∼PR(ste,ate)r_t^e \sim P_R(s_t^e,a_t^e), skill rewards rte,n∼PR,n(ste,ate)r_t^{e,n} \sim P_{R,n}(s_t^e,a_t^e) for all nn

  9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample next state st+1e∼PS(ste,ate)s_{t+1}^e \sim P_S(s_t^e,a_t^e)

  10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t←t+1t \leftarrow t+1

  11. &nbsp;&nbsp;&nbsp;&nbsp;end for

  12. &nbsp;&nbsp;&nbsp;&nbsp;if t mod U=0t \bmod U = 0 then

  13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute meta λ\lambda-returns Rλ,t−1eR_{\lambda,t-1}^e to Rλ,t−UeR_{\lambda,t-U}^e for all ee

  14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute skill λ\lambda-returns Rλ,t−1e,nR_{\lambda,t-1}^{e,n} to Rλ,t−Ue,nR_{\lambda,t-U}^{e,n} for all e,ne,n

  15. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for number of epochs do

  16. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for number of minibatches do

  17. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample minibatch BB of size b≤EUb \le EU from {(t−U,0),…,(t−1,E−1)}\{(t-U,0),\ldots,(t-1,E-1)\}

  18. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update meta:

ϕmeta←ϕmeta+αt2b∇ϕmeta∑(j,τ)∈B(Rλ,τj−Qϕmeta(sτj))2\phi_{\mathrm{meta}} \leftarrow\phi_{\mathrm{meta}}+\frac{\alpha_t}{2b}\nabla_{\phi_{\mathrm{meta}}}\sum_{(j,\tau)\in B}\left(R_{\lambda,\tau}^j-Q_{\phi_{\mathrm{meta}}}(s_\tau^j)\right)^2
  1. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update skills:

ϕn←ϕn+αt2b∇ϕn∑(j,τ)∈B(Rλ,τj,n−Qϕn(sτj,n))2,∀n∈{1,…,N}\phi_n \leftarrow\phi_n+\frac{\alpha_t}{2b}\nabla_{\phi_n}\sum_{(j,\tau)\in B}\left(R_{\lambda,\tau}^{j,n}-Q_{\phi_n}(s_\tau^{j,n})\right)^2,\quad\forall n\in\{1,\ldots,N\}
  1. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end for

  2. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end for

  3. &nbsp;&nbsp;&nbsp;&nbsp;end if

  4. end for

Implementation Details

Setup. We base our implementations on CleanRL (Huang et al. [25]) and PureJaxRL (Lu et al. [43]), adapting them to object-centric inputs by replacing convolutional encoders with lightweight MLPs for feature extraction. Hyperparameters are listed in Table 2 and Table 3 and remain largely consistent with the original implementations, except for an increased number of parallel environments enabled by the efficiency of JAX-based code.

ParameterPQN(-based)PPO
Total Timesteps5×1075 \times10^{7}5×1075 \times10^{7}
Num Environments1024128
Num Steps per Update3232
Learning Rate2.5×10−42.5 \times10^{-4}2.5×10−42.5 \times10^{-4}
Max Grad Norm100.5
Discount Factor (γ\gamma)0.990.99
GAE Lambda (λ\lambda)0.65 (0.5 with learned meta)0.95
GAE Meta Lambda (λ\lambda)0.9–
Num Epochs12
Num Minibatches2564
Hidden Size64–
Num Layers3–
NormalizationLayer Norm–
Clip ϵ\epsilon–0.2
Entropy Coef–0.01
Value Function Coef–0.5
Anneal LRFalseTrue
ϵ\epsilon-Start/End/Decay1.0 / 0.001 / 0.1–
Meta ϵ\epsilon-Start/End/Decay1.0 / 0.001 / 0.3–

Table 2. Atari Hyperparameters

ParameterPQN(-based)
Total Timesteps1×1091 \times10^{9}
Num Environments512
Num Steps per Update128
Learning Rate3.0×10−43.0 \times10^{-4}
Max Grad Norm0.5
Discount Factor (γ\gamma)0.99
GAE Lambda (λ\lambda)0.5
GAE Meta Lambda (λ\lambda)0.5
Num Epochs4
Num Minibatches4
Hidden Size512
Num Layers1
NormalizationLayer Norm
Anneal LRTrue
ϵ\epsilon-Start/End/Decay1.0 / 0.005 / 0.1
Meta ϵ\epsilon-Start/End/Decay1.0 / 0.005 / 0.1

Table 3. Crafter Hyperparameters

The exploration parameter ϵ\epsilon was selected via a brief hyperparameter sweep in the range [1,0.001][1, 0.001], using final test return as the selection criterion.

Each experiment is run with six random seeds (0-5) to ensure robust reporting. Reported plots include the corresponding standard deviation.

All experiments were conducted on a single NVIDIA Tesla V100-SXM3-32GB-H GPU on an NVIDIA DGX Server (Version 5.1.0) with CUDA 12.4.

Our code repository1 includes all necessary code and config files to reproduce the results from the experiments section, including plots.

Environments

Screenshots of the environments Crafter, Seaquest and Kangaroo are available in Figure 9.

Figure 9. The evaluation environments: Crafter, Seaquest and Kangaroo.

These games are selected because their gameplay naturally decomposes into low-level skills that must be combined to solve the overall task. In Kangaroo, the objective is to ascend through platforms while avoiding enemies and collecting berries. In Seaquest, the player navigates a submarine to rescue divers and return them to the surface, while avoiding hazards such as sharks and enemy submarines. These two games are known to exhibit reward hacking behavior, meaning that the agents find simple shortcuts of increasing their return rather than the intended way that humans would typically choose. Examples include exclusively shooting enemies in Seaquest and only catching falling apples or boxing monkeys in Kangaroo.

Unlike Atari games, Crafter presents a more complex, open-ended challenge that requires agents to acquire and coordinate multiple skills, including navigation, combat, resource collection, and crafting.

Evaluation Metrics

We empirically evaluate agent performance using three metrics: (1) Skill Returns to test whether the skills were learned successfully, (2) Human-Normalized Score (HNS) for absolute performance relative to human and random baselines and (3) Aligned Environment Goal Scores that measure performance based on the main goals described in the game’s manual.

Human Normalized Score. Human-Normalized Score standardizes agent performance across Atari environments by accounting for differences in reward scales [51, 47]. Given the average agent score AA, human score HH, and random score RR, HNS is defined as:

HNS=A−R∣H−R∣\mathrm{HNS} = \frac{A - R}{\lvert H - R \rvert}

A value of 1.0 indicates human-level performance, values greater than 1.0 indicate superhuman performance, and values below 0 denote sub-random behavior. We adopt the human and random baselines from Badia et al. [6], derived from professional human play.

Aligned Environment Goal Scores. Environment reward signals may not always align with the intended task objectives and can be susceptible to reward hacking. In such cases, agents may learn high-reward behaviors that are non-intuitive and deviate from human-like solutions. To better capture progress toward the actual environment goals, we define two aligned goal-based metrics grounded in the objectives stated in the game manuals.

For Seaquest, the goal is to rescue as many divers as possible; for Kangaroo, it is to help the mother kangaroo reach and rescue her baby, located on the topmost platform. Accordingly, we track the number of divers retrieved and the number of platforms reached, respectively.

Interpretability Proxies - Detailed Computation

To rigorously compare policy complexity between NUDGE (Delfosse et al. [11]) and NEXUS, we adopt the formal rule-based complexity metrics defined by Lakkaraju et al. [35] alongside Average Decision Path Length (Wu et al. [63]):

  • Rule Size (∣R∣|\mathcal{R}|): The total number of distinct rules or conditional logic branches comprising the policy.

  • Total Rule Length (Ltotal\mathcal{L}_{\mathrm{total}}): The sum of all body literals (conditions) evaluated across the entire policy (Ltotal=∑r∈R∣r∣\mathcal{L}_{\mathrm{total}} = \sum_{r \in\mathcal{R}} |r|).

  • Average Decision Path Length (ADPL): The expected number of condition checks executed to make a single decision under a uniform distribution over execution paths Wu et al. [63].

Metric Definitions

∣R∣=Total count of rules/branches in policy R|\mathcal{R}| = \text{Total count of rules/branches in policy } \mathcal{R}
Ltotal=∑r∈R∣r∣where ∣r∣ is the number of body literals in rule r\mathcal{L}_{\text{total}} = \sum_{r \in\mathcal{R}} |r| \quad\text{where } |r| \text{ is the number of body literals in rule } r
ADPL⁡=1∣K∣∑k=1∣K∣Ckwhere Ck is the cumulative evaluations to trigger path k\operatorname{ADPL} = \frac{1}{|\mathcal{K}|}\sum_{k=1}^{|\mathcal{K}|} C_k \quad\text{where } C_k \text{ is the cumulative evaluations to trigger path } k

NUDGE Policy Computations

NUDGE for Seaquest (cf. Listing 4) consists of ∣R∣=16|\mathcal{R}| = 16 sequential Prolog rules evaluated top-to-bottom.

Rule Size and Total Rule Length

  • Rule Size (∣R∣NUDGE|\mathcal{R}|_{\mathrm{NUDGE}}): 16 rules

  • Total Rule Length (LNUDGE\mathcal{L}_{\mathrm{NUDGE}}):

LNUDGE=∑i=116∣ri∣=1+1+4+4+4+4+2+2+3+3+3+3+3+3+3+3=46 literals\begin{aligned} \mathcal{L}_{\mathrm{NUDGE}} = \sum_{i=1}^{16} |r_i| \\ &= 1 + 1 + 4 + 4 + 4 + 4 + 2 + 2 + 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 \\ &= 46\ \text{literals} \end{aligned}

Average Decision Path Length (ADPL⁡NUDGE\operatorname{ADPL}_{\mathrm{NUDGE}})

Evaluating sequentially, reaching rule kk requires evaluating and failing all conditions in rules 1…k−11 \ldots k - 1 plus evaluating rule kk. The cumulative cost Ck=∑i=1k∣ri∣C_k = \sum_{i=1}^{k} |r_i| for each rule k∈{1,…,16}k \in\{1, \ldots, 16\} is:

C1=1,C2=2,C3=6,C4=10,C5=14,C6=18,C7=20,C8=22,C_1 = 1,\quad C_2 = 2,\quad C_3 = 6,\quad C_4 = 10,\quad C_5 = 14,\quad C_6 = 18,\quad C_7 = 20,\quad C_8 = 22,
C9=25,C10=28,C11=31,C12=34,C13=37,C14=40,C15=43,C16=46.C_9 = 25,\quad C_{10} = 28,\quad C_{11} = 31,\quad C_{12} = 34,\quad C_{13} = 37,\quad C_{14} = 40,\quad C_{15} = 43,\quad C_{16} = 46.

Summing cumulative checks across all 16 outcomes:

∑k=116Ck=377⇒ADPL⁡NUDGE=37716=23.56 evaluations\sum_{k=1}^{16} C_k = 377 \quad\Rightarrow\quad\operatorname{ADPL}_{\mathrm{NUDGE}} = \frac{377}{16} = \mathbf{23.56}\ \text{evaluations}

NEXUS (symbolic) Computations

Listing 1 defines an explicit control-flow hierarchy over 4 execution branches.

A. Rule Size and Total Rule Length

  • Rule Size (∣R∣NEXUS−1|\mathcal{R}|_{\mathrm{NEXUS-1}}): 4 execution branches (3 conditional statements + 1 default fallback)

  • Total Rule Length (LNEXUS−1\mathcal{L}_{\mathrm{NEXUS-1}}):

LNEXUS−1=∣r1∣+∣r2∣+∣r3∣+∣r4∣=2+1+1+0=4 conditions\mathcal{L}_{\mathrm{NEXUS-1}} = |r_1| + |r_2| + |r_3| + |r_4| = 2 + 1 + 1 + 0 = 4\ \text{conditions}

B. Average Path Length (ADPLNEXUS−1\mathrm{ADPL}_{\mathrm{NEXUS-1}})

Accounting for short-circuit evaluation across the 4 possible decision paths (K={1,2,3,4}\mathcal{K} = \{1, 2, 3, 4\}), where a satisfied disjunction requires one check and a failed disjunction two:

  • Path 1 (surface): C1=1C_1 = 1 check

  • Path 2 (fight_enemies): C2=2+1=3C_2 = 2 + 1 = 3 checks

  • Path 3 (rescue_divers): C3=2+1+1=4C_3 = 2 + 1 + 1 = 4 checks

  • Path 4 (fight_enemies default): C4=2+1+1=4C_4 = 2 + 1 + 1 = 4 checks

ADPLNEXUS−1=1+3+4+44=3.00 evaluations\mathrm{ADPL}_{\mathrm{NEXUS-1}} = \frac{1 + 3 + 4 + 4}{4} = \mathbf{3.00}\ \text{evaluations}

NEXUS (nesy) Computations

Listing 2 evaluates two independent conditional blocks sequentially; fight_enemies is always admissible.

A. Rule Size and Total Rule Length

  • Rule Size (∣R∣NEXUS−2|\mathcal{R}|_{\mathrm{NEXUS-2}}): 2 rule blocks

  • Total Rule Length (LNEXUS−2\mathcal{L}_{\mathrm{NEXUS-2}}):

LNEXUS−2=∣r1∣+∣r2∣=1+2=3 conditions\mathcal{L}_{\mathrm{NEXUS-2}} = |r_1| + |r_2| = 1 + 2 = \mathbf{3}\ \text{conditions}

B. Average Decision Path Length (ADPLNEXUS−2\mathrm{ADPL}_{\mathrm{NEXUS-2}})

Both blocks execute on every forward pass:

  • Block 1 (if diver_available): Always 1.0 check.

  • Block 2 (if oxygen_low or all_collected): Evaluates oxygen_low (1 check) + 50% chance of evaluating all_collected (+0.5 check) = 1.5 checks.

ADPLNEXUS−2=1.0+1.5=2.50 evaluations\mathrm{ADPL}_{\mathrm{NEXUS-2}} = 1.0 + 1.5 = \mathbf{2.50}\ \text{evaluations}

Figure 10. Reward-shaping is not sufficient. Replacing NEXUS’s neuro-symbolic meta-policy with LLM-designed or statically weighted reward shaping leads to severe performance degradation across both environments. While baselines successfully acquire isolated skills (e.g., surface in Seaquest), they fail to learn all skills equally well or to coordinate them—demonstrating that NEXUS’s skill disentanglement is essential for achieving high overall performance.

Additional Results

Comparative evaluation against reward shaping baselines

To isolate the benefits of NEXUS, we evaluate its performance against PQN trained on shaped rewards. To ensure a rigorous comparison, we designed three distinct reward-shaping strategies:

  1. PQN + shaped (symbolic): Utilizes the same LLM-designed symbolic meta-policy as NEXUS (symbolic) to dynamically select the active skill reward at each time step, functioning analogously to a reward machine [27, 9].

  1. PQN + shaped (uniform): Applies a static, equal weighting across all three skill rewards throughout training.

  1. PQN + shaped (LLM-weighted): Leverages LLM-generated scalar weights to combine the individual skill rewards into a single fixed objective.

Across both environments, all three reward-shaping baselines fall significantly short of NEXUS in terms of overall performance. Although these baselines successfully acquire specific localized behaviors, such as rescue_divers or surface, in Seaquest, they focus on single skills and fail to effectively coordinate them, resulting in low overall environment returns. These results demonstrate that simply augmenting RL methods with shaped or LLM-designed rewards is insufficient, whereas NEXUS’s neuro-symbolic architecture enables equal skill learning and coordination.

Analysis of neuro-symbolic meta-policy impact and skill activation dynamics

To evaluate the contribution of the neuro-symbolic meta-policy, we analyze both the performance impact of its decision-making mechanism and the activation frequencies of individual skills across Seaquest and Kangaroo (Figure 11). We first test the necessity of the Meta-Q-value maximization by replacing it in ambiguous situations where multiple skill conditions evaluate to true with a uniformly random choice over the valid skills. To prevent frequent switching, random skill selections are held for a minimum duration of 20 steps. We observe a notable drop in overall environment return and aligned evaluation metrics in both domains, with Seaquest exhibiting a stronger performance degradation than Kangaroo. This drop confirms that the neuro-symbolic meta-policy plays a critical role in selecting context-appropriate actions rather than relying on arbitrary execution. Analyzing skill activation frequencies reveals that gameplay in both environments is heavily anchored by dedicated default skills: fight_enemies in Seaquest and navigate in Kangaroo. The meta-policy selectively departs from these defaults only when specific conditions permit alternative skills to maximize expected returns. In Seaquest, the agent heavily balances default combat with targeted objectives.

Figure 11. Skill activation dynamics and performance impact of the neuro-symbolic meta-policy in Seaquest and Kangaroo. Replacing the meta-policy with random skill choices significantly degrades performance, demonstrating its critical role in maximizing returns. Default skills (fight_enemies in Seaquest, navigate in Kangaroo) dominate usage, while specialized skills trigger selectively based on context: collect_divers (20% total / 30% eligible) and upward (7% total / 70% eligible) in Seaquest, versus collect_fruits (13% total / 46% eligible) and handle_enemies (9% total / 40% eligible) in Kangaroo.

The collect_divers skill activates in roughly 20% of all execution steps, corresponding to 30% of the steps where it was eligible. Conversely, the upward skill accounts for only 7% of total gameplay steps, but triggers in 70% of all valid opportunities, indicating highly efficient, situational utilization. In Kangaroo, the navigate skill overwhelmingly dominates total runtime. Secondary skills execute far more conservatively: collect_fruits accounts for 13% of all steps (46% of eligible opportunities), while handle_enemies is invoked in just 9% of total steps (40% of eligible opportunities).

Additional environments

To assess the generality of NEXUS, we extend our evaluation to three additional Atari environments: Pong, Breakout, and Freeway. Skill learning curves for these environments are shown in Figure 12, while comparisons to baseline agents and ablations—evaluated via human-normalized scores—are presented in Figure 13. Note that these games are generally less complex than Kangaroo and Seaquest, and the learned skills are not strictly necessary to achieve the environment goals. In particular, the skills "Move Up" and "Avoid Crash" in Freeway largely correspond to atomic actions such as forward or noop. As such, a simple fixed meta-policy operating directly on primitive actions could suffice for solving this task.

Figure 12. NEXUS successfully learns skills in Pong (top), Breakout (middle), and Freeway (bottom).

Figure 13. NEXUS achieves performance comparable to baseline methods across the three evaluated environments.

Neuro-symbolic RL baselines.

For improved comparison, we provide baseline scores of existing interpretable and neuro-symbolic methods on both the default games (cf. Figure 14) and the simplifications (cf. Figure 15). We compare to NUDGE (Delfosse et al., [11]) and BlendRL (Shindo et al., [57]). All scores, together with the reported scores from SCOBots (Delfosse et al., [14]) (game simplification scores from Delfosse et al. [15]) are reported in Table 4. While prior neuro-symbolic methods achieve competitive scores, they suffer from significant limitations regarding transparency and flexibility. Primarily, interpretability in these baselines is restricted to low-level actions. Consequently, the overall behavioral policy must be defined at the action granularity, resulting in excessive complexity that is difficult to parse (cf. Section A). Furthermore, these approaches lack explicit representations of the high-level objectives currently being pursued (e.g., “collect all divers”).

Figure 14. Comparison to other neuro-symbolic RL methods.

Figure 15. Comparison to other neuro-symbolic RL methods on simplified games.

AlgorithmSeaquest DefaultDivers CollectedSeaquest SimplificationDivers CollectedKangaroo DefaultLevel Completion (%)Kangaroo SimplificationLevel Completion (%)
PPO915.3 ± 101.1(6.7 ± 0.1)0.0 ± 0.0(4.7 ± 0.6)550.0 ± 353.6(33.3 ± 0.6)150.0 ± 212.1(33.3 ± 0.6)
PQN1361.3 ± 294.6(4.8 ± 2.8)0.0 ± 0.0(4.0 ± 1.5)949.7 ± 279.9(47.3 ± 0.4)338.9 ± 209.1(40.5 ± 13.6)
NEXUS (neural)2100.7 ± 614.5(9.2 ± 4.2)0.0 ± 0.0(6.7 ± 6.8)1889.0 ± 1770.6(70.6 ± 28.8)551.0 ± 513.2(58.9 ± 30.9)
NEXUS (symbolic)2290.1 ± 172.9(12.4 ± 0.7)9416.7 ± 4363.2(41.0 ± 13.1)1030.3 ± 553.3(83.8 ± 2.6)448.8 ± 114.7(52.0 ± 9.4)
NEXUS (nesy)2185.6 ± 1000.1(10.8 ± 2.9)2483.3 ± 1607.7(15.3 ± 9.0)909.4 ± 519.1(85.6 ± 12.8)674.8 ± 115.8(72.1 ± 16.9)
NUDE46.7 ± 23.1(4.1 ± 0.7)113.3 ± 99.6(7.3 ± 4.5)1522.2 ± 661.9(110.0 ± 17.3)1966.7 ± 290.6(101.7 ± 2.9)
BlendRL2138.9 ± 1636.0(15.8 ± 8.6)24755.6 ± 29438.4(55.6 ± 43.0)1955.6 ± 887.8(101.7 ± 2.9)1944.4 ± 300.6(118.3 ± 27.5)
SCoBots1055.3 ± 272.6(- ± -)0.0 ± 0.0(- ± -)2776.6 ± 1332.4(- ± -)0.0 ± 0.0(- ± -)

Table 4. Performance Comparison on Seaquest and Kangaroo

The logical rules within NUDGE [11] and BlendRL [57] are static and strictly followed, resulting in rigid policies. While BlendRL attempts to mitigate this via a switching mechanism that activates a neural policy, this compromise diminishes transparency; the neural component maximizes extrinsic reward rather than adhering to an interpretable sub-goal.

Moreover, all baseline approaches necessitate substantial manual engineering. NUDGE and BlendRL require the definition of specific valuation functions based on detected objects and attributes, while SCoBots [14] relies on the manual definition and selection of relational functions. In contrast, NEXUS operates at the abstracted level of skills, facilitating the interpretation and manipulation of the agent’s active goals. As a result, the meta-policy definitions in NEXUS are significantly more concise than the action-level policies of SCoBots, NUDGE, and BlendRL. Additionally, NEXUS eliminates the need for manual valuation functions by autonomously learning skills via LLM-generated reward functions.

Finally, the performance of NUDGE and BlendRL in simplified environments is largely attributable to extensive hard-coded logic and specific valuation functions (e.g. ‘fire if same_depth(player, enemy) and visible(enemy) and facing\_right(player) and left\_of(player, enemy)’). Deriving these rigid rules is significantly more labor-intensive than generating meta-policy and reward functions in NEXUS, rendering the baselines less scalable to complex environments. This suggests an inherent trade-off between high-level interpretability, manual labor, and performance. While previous algorithms prioritize maximum action-level transparency and performance by incorporating extensive manual inductive biases, this work targets high-level interpretability with a significantly reduced reliance on such biases.

Noisy Detections

While object detection methods become increasingly reliable, misdetections still happen. To simulate unreliable object extractors, we incorporate a misdetection chance and gaussian noise to each object attribute during both training and testing. We conduct experiments with 10% misdetection rate and 3px gaussian noise. To emulate the effect of a kalman filter estimating the objects movements during each step, instead of zeroing out the detected objects, we keep the previous time step detections. On top of that, we add gaussian noise with a standard deviation of 3 at each attribute.

We observe significant differences between Seaquest and Kangaroo. For Seaquest, when introducing noise only during evaluation (Figure 16), all methods lose performance significantly. However, both the symbolic and neuro-symbolic approaches keep rescuing a relatively high number of divers. In Kangaroo, on the other hand, the experiments surprisingly indicate that all approaches increase in performance.

Figure 16. Noisy evaluation under 10% misdetection and noise.

The performance drops show different characteristics when the approaches also experience noise during training (Figure 17).

Figure 17. Noisy training and evaluation under 10% misdetection and noise.

In Seaquest, all NEXUS approaches drop only a small amount and remain reliable. We again observe only small performance drops (and even improvements) in Kangaroo. Our working hypothesis for the observed robustness in Kangaroo is that the noise induced a higher exploration and reduced overfitting to spurious correlations during training.

LLM Interaction

We outline the procedure for leveraging an LLM to generate task-relevant skills, associated reward functions, and a fixed meta-policy rule set. The LLM is conditioned on the game’s original manual and structured type information describing the object-centric observations available to the agent.

Initially, the LLM is queried for a set of skills and corresponding reward functions. These outputs can be manually refined before querying the LLM for a meta-policy function that selects which skill to execute. Prior prompts and responses are retained to maintain conversational context, consistent with standard chat behavior. We provide the entire prompt for the game Kangaroo in Figure 18. The prompt for Seaquest was generated equivalently.

Subsequent prompts for generating a skill-list and corresponding reward functions and a meta-policy function for the game Kangaroo

Figure 18. Subsequent prompts (chat-function) to generate a skill-list and corresponding reward functions (top) and meta-policy function (bottom) for the game Kangaroo.

Unedited responses from GPT-4o (via chatgpt.com on 21-07-2025) for the games Kangaroo and Seaquest are included in Figure 19–Figure 22.

Essential Skills for Solving Kangaroo (Atari 2600)

Figure 19. Generated reward functions for Kangaroo by GPT4-O

Seaquest Skill Selection Function

Figure 22. Generated fixed meta-policy for Seaquest by GPT4-O

For Crafter, we used the LLM to generate the list of important tasks, as well as the symbolic meta-policy function. Prompt and answers are available in Figure 23 and Figure 24. Rewards for the specific skills were crafted manually.

Essential Skills for Solving Crafter

Essential Skills for Solving _Crafter_

Basic Skills. We identify the following five skills as sufficient building blocks for solving _Crafter_:

  1. Gather Resources: Mine wood, stone, coal, iron, or other materials when available.

  2. Craft Tools/Weapons: Use gathered resources to craft pickaxes and swords for efficiency and combat readiness.

  3. Eat/Drink/Sleep: Replenish food, water, or energy to maintain survival.

  4. Fight/Defend: Attack hostile mobs when they are nearby and health is sufficient.

  5. Explore/Move: Traverse unexplored areas to discover resources, water, and safe locations.

Policy Function. The following pseudo-code outlines a rule-based strategy for selecting the active skill based on the current game state.

def select_skill(state: EnvState) -> int:
    """
    Returns the skill index:
    0 = Gather Resources
    1 = Craft Tools/Weapons
    2 = Eat/Drink/Sleep
    3 = Fight/Defend
    4 = Explore/Move
    """

    # --- Survival priority ---
    if state.player_food <= 2 or state.player_drink <= 2 or state.player_energy <= 2:
        return 2 # Eat/Drink/Sleep

    # --- Combat ---
    if mob_nearby(state) and state.player_health > 3:
        return 3 # Fight/Defend

    # --- Resource management ---
    if resources_nearby(state) and inventory_needs_tools(state.inventory):
        return 0 # Gather Resources

    if can_craft_better_tools(state.inventory):
        return 1 # Craft Tools/Weapons

    # --- Default action ---
    return 4

Figure 24. Generated list of essential skills and a rule-based policy for solving _Crafter_ by GPT4-O. The policy prioritizes survival, then combat, followed by resource gathering and crafting, and defaults to exploration otherwise.

The LLM outputs served as a starting point: while the proposed skill decompositions and the structure of the meta-policies were largely retained, the final implementations required both framework adaptations (e.g., JAX compatibility, mapping to the available state variables) and semantic refinements by the authors, such as setting distance and resource thresholds, reordering rule priorities, and simplifying or rebalancing reward terms. The neuro-symbolic rule sets were derived manually from the symbolic meta-policies by making their conditions non-exclusive. Table 5 lists all changes per environment and component; the final code is available in our repository2.

You are a RL expert and develop a hierarchical agent to play the atari game Seaquest. Below, I have provided a detailed description of the game. Your task is to come up with a short list of essential skills (3-4) that are needed to solve the game.

Game Description:

<Atari game manualᵃ>

Think about what the main goal of this game is, then come up with a list of the required skills to solve the game and provide it in the output. Finally, create a pseudo reward function for each of these skills that a RL agent can use to learn the skills.

The reward functions are called at each step in the environment and retrieve the current and previous symbolic observation of the game. Here is the type definition for the observation:

class KangarooObservation(NamedTuple):
    player_x: chex.Array
    player_y: chex.Array
    player_o: chex.Array
    platform_positions: chex.Array
    ladder_positions: chex.Array
    fruit_positions: chex.Array
    bell_position: chex.Array
    child_position: chex.Array
    falling_coco_position: chex.Array
    monkey_positions: chex.Array
    coco_positions: chex.Array

Each skill and it’s reward function should be as simple as possible that achieves the wanted behavior. Provide: List of skills and description, pseudo-code reward functions for each skill.

You are now creating a function that decides at each time step in the game, which skill to choose. The skills areᵃ:

  • Obstacle Avoidance

  • Vertical Navigation

  • Fruit Collection

  • Goal Reaching (Child Rescue)

The function again has access to the current and previous observation.

Think about at what point in the game you want to use which skill, then provide the function.

Kangaroo Skill Selection Function

Kangaroo Skill Selection Function

To select appropriate skills, the agent follows a fixed priority:

  1. Obstacle Avoidance: If any apple or monkey is in close proximity.

  2. Goal Reaching: If the player is near the child on the top platform.

  3. Fruit Collection: If fruit is visible and reachable.

  4. Vertical Navigation: Default behavior to explore or ascend.

Skill Selector Function:

def select_skill (obs, prev_obs):
    player_pos = np.array([obs.player_x, obs.player_y])

    def in_danger():
        danger_entities = obs.apple_positions + obs.monkey_positions
        return any(np.linalg.norm(player_pos − np.array([x, y])) <= 1.0
                   for (x, y) in danger_entities)

    def near_child():
        child_pos = np.array([obs.child_position[0], obs.child_position[1]])
        return np.linalg.norm(player_pos − child_pos) <= 1.5

    def fruit_visible():
        return any(abs(y − obs.player_y) <= 1 and
                   abs(x − obs.player_x) <= 3
                   for (x, y) in obs.fruit_positions)

    if in_danger():
        return obstacle_avoidance
    elif near_child():
        return goal_reaching
    elif fruit_visible():
        return fruit_collection
    else:
        return vertical_navigation

This rule-based selection policy balances safety, progression, and reward acquisition, enabling effective hierarchical control.

Figure 20. Generated fixed meta-policy function for Kangaroo by GPT4-O

Essential Skills for Solving Kangaroo (Atari 2600)

A hierarchical RL agent for Kangaroo can be built upon the following four primitive skills:

  1. Obstacle Avoidance

Description: Avoid contact with apples or monkeys via dodging or punching.

Reward Function:

def reward_obstacle_avoidance(obs, prev_obs):
    collided = any(np.array_equal(obs.player_x, x) and
                   np.array_equal(obs.player_y, y)
                   for (x, y) in obs.apple_positions + obs.monkey_positions)
    return −1.0 if collided else 0.0
  1. Vertical Navigation

Description: Climb ladders to ascend levels and progress.

Reward Function:

def reward_vertical_navigation(obs, prev_obs):
    dy = prev_obs.player_y − obs.player_y
    return 0.1 if dy > 0 else 0.0
  1. Fruit Collection

Description: Collect fruits and optionally ring a bell to regenerate them.

Reward Function:

def reward_fruit_collection(obs, prev_obs):
    reward = 0.0
    if len(obs.fruit_positions) < len(prev_obs.fruit_positions):
        reward += 0.5
    if np.array_equal(obs.player_x, obs.bell_position[0]) and \
       np.array_equal(obs.player_y, obs.bell_position[1]):
        reward += 0.2
    return reward
  1. Goal Reaching (Child Rescue)

Description: Reach the child’s location to complete the level.

Reward Function:

def reward_reach_child(obs, prev_obs):
    return 1.0 if np.array_equal(obs.player_x, obs.child_position[0]) and \
                  np.array_equal(obs.player_y, obs.child_position[1]) else 0.0

These modular rewards enable a high-level controller to invoke appropriate behaviors depending on context.

Essential Skills for Solving Seaquest (Atari 2600)

To solve Seaquest effectively, a hierarchical RL agent must master the following four core skills:

1. Avoid Obstacles (Survival Navigation)

Description: Avoid collisions with sharks, enemy submarines, missiles, and the patrol sub, both underwater and when surfacing.

Reward Function (pseudo-code):

def avoid_obstacles_reward(prev_obs, curr_obs):
    if player_collided(curr_obs):
        return −1.0
    return 0.0

def player_collided(obs):
    return check_collision_with_sharks(obs) or \
           check_collision_with_subs(obs) or \
           check_collision_with_missiles(obs) or \
           check_collision_with_surface_sub(obs)

2. Collect Divers

Description: Navigate to and collect divers to accumulate score and progress.

Reward Function (pseudo-code):

def collect_divers_reward(prev_obs, curr_obs):
    prev_count = count_nearby_divers(prev_obs)
    curr_count = count_nearby_divers(curr_obs)
    return float(prev_count > curr_count) * 1.0

def count_nearby_divers(obs):
    return sum(is_near(obs.player_x, obs.player_y, dx, dy)
               for dx, dy in obs.diver_positions)

3. Surface Strategically

Description: Surface safely when carrying six divers or when oxygen is low, to drop off divers and replenish oxygen.

Reward Function (pseudo-code):

def surface_reward(prev_obs, curr_obs):
    surfaced_now = prev_obs.player_y > 0 and curr_obs.player_y == 0
    has_6_divers = num_divers_collected(prev_obs) == 6
    low_oxygen = prev_obs.oxygen < OXYGEN_THRESHOLD
    if surfaced_now and (has_6_divers or low_oxygen):
        return 2.0 if has_6_divers else 0.5
    return 0.0

def num_divers_collected(obs):
    return 6 − sum(is_valid(dx, dy) for dx, dy in obs.diver_positions)

4. Eliminate Threats (Combat)

Description: Fire missiles to destroy sharks and enemy submarines, clearing paths and increasing score.

Reward Function (pseudo-code):

def eliminate_threats_reward(prev_obs, curr_obs):
    num_enemies_prev = count_enemies(prev_obs)
    num_enemies_curr = count_enemies(curr_obs)
    return float(num_enemies_curr < num_enemies_prev) * 0.5

def count_enemies(obs):
    return sum(is_valid(dx, dy) for dx, dy in obs.shark_positions) + \

You are a reinforcement learning specialist. The following is the basic description of the game Crafter, which you are trying to solve.

—

Crafter is a game about exploring dungeons, mining, crafting and fighting enemies. The player can move in the four cardinal directions using WASD and can interact using SPACE. Interacting can cause the player to attempt to mine (a block), attack (a creature), drink (water or from a fountain), or eat (fruit).

The player has the ’intrinsics’ health, hunger, thirst and energy. Hunger, thirst and energy will naturally decrease and must be replenished by eating, drinking and sleeping respectively. Health will recover when hunger, thirst and energy are non-zero and will decrease if any of these are 0. If the players health falls beneath 0 they will die and the game will restart.

—

First, come up with 5 basic skills that can be used to solve the game. Only provide necessary skills without duplications. After that, please come up with a simple function (pseudo code) that gets the game-state as input and returns an integer deciding on the active skill to select in the current state. The skills correspond to the ones that you chose before. Try to solve the game as far as possible. The game-state looks like this:

@struct.dataclass
class Inventory:
    wood: int = 0
    stone: int = 0
    coal: int = 0
    iron: int = 0
    diamond: int = 0
    sapling: int = 0
    wood_pickaxe: int = 0
    stone_pickaxe: int = 0
    iron_pickaxe: int = 0
    wood_sword: int = 0
    stone_sword: int = 0
    iron_sword: int = 0

@struct.dataclass
class Mobs:
    position: jnp.ndarray
    health: int
    mask: bool
    attack_cooldown: int

@struct.dataclass
class EnvState:
    map: jnp.ndarray
    mob_map: jnp.ndarray
    player_position: jnp.ndarray
    player_direction: int
    # Intrinsics
    player_health: int
    player_food: int
    player_drink: int
    player_energy: int
    is_sleeping: bool
    # Second order intrinsics
    player_recover: float
    player_hunger: float
    player_thirst: float
    player_fatigue: float
    inventory: Inventory
    zombies: Mobs
    cows: Mobs
    skeletons: Mobs
    arrows: Mobs
    arrow_directions: jnp.ndarray
    growing_plants_positions: jnp.ndarray
    growing_plants_age: jnp.ndarray

References

  1. [1]Kamal Acharya, Waleed Raza, Carlos M. J. M. Dourado Júnior, Alvaro Velasquez, and Houbing Herbert Song. Neurosymbolic Reinforcement Learning and Planning: A Survey. IEEE Transactions on Artificial Intelligence, 5(5):1939–1953, 2024.DOI
  2. [2]Riad Akrour, Davide Tateo, and Jan Peters. Towards reinforcement learning of human readable policies. In The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases: The 1st Workshop on Deep Continuous-Discrete Machine Learning, 2019.
  3. [3]Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic Reinforcement Learning with Formally Verified Exploration. In Neural Information Processing Systems, volume abs/2009.12612, 2020.arxiv.org/abs/2009.12612
  4. [4]Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforcement Learning with Policy Sketches. In International Conference on Machine Learning (ICML), pp. 166–175, 2017.arxiv.org/abs/1611.01796
  5. [5]Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. In AAAI Conference on Artificial Intelligence (AAAI), pp. 1726–1734, 2017.arxiv.org/abs/1609.05140
  6. [6]Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari Human Benchmark. In International Conference on Machine Learning (ICML), pp. 507–517, 2020.arxiv.org/abs/2003.13350
  7. [7]Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13:341–379, 2003.
  8. [8]Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research (JAIR), 47:253–279, 2013.
  9. [9]Roger Creus Castanyer, Faisal Mohamed, Pablo Samuel Castro, Cyrus Neary, and Glen Berseth. Arm-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning. abs/2510.14176, 2025.arxiv.org/abs/2510.14176
  10. [10]Brett Daley and Christopher Amato. Reconciling λ-Returns with Experience Replay. In Conference on Neural Information Processing Systems (NeurIPS), 2019.DOI
  11. [11]Quentin Delfosse, Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction. In Conference on Neural Information Processing Systems (NeurIPS), 2023a.arxiv.org/abs/2306.01439
  12. [12]Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbächer, Dwarak Vittal, and Kristian Kersting. Boosting Object Representation Learning via Motion and Object Continuity. Springer Nature Switzerland, 2023b.DOI
  13. [13]Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Object-Centric Atari 2600 Reinforcement Learning Environments. Reinforcement Learning Conference (RLC), 1:400–449, 2024a.arxiv.org/abs/2306.08649
  14. [14]Quentin Delfosse, Sebastian Sztwiertnia, Mark Rothermel, Wolfgang Stammer, and Kristian Kersting. Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents. In Conference on Neural Information Processing Systems (NeurIPS), volume abs/2401.05821, 2024b.arxiv.org/abs/2401.05821
  15. [15]Quentin Delfosse, Jannis Blüml, Fabian Tatai, Théo Vincent, Bjarne Gregori, Elisabeth Dillies, Jan Peters, Constantin Rothkopf, and Kristian Kersting. Deep reinforcement learning agents are not even close to human intelligence. arXiv preprint, 2025.arxiv.org/abs/2505.21731
  16. [16]Quentin Delfosse, Raban Emunds, Paul Seitz, Sebastian Wette, Jannis Blüml, and Kristian Kersting. Jaxatari: A high-performance framework for reasoning agents. https://github.com/k4ntz/JAXAtari/, 2026.
  17. [17]Thomas G. Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition. Journal of Artificial Intelligence Research (JAIR), 13:227–303, 2000.DOI
  18. [18]Elisabeth Dillies, Quentin Delfosse, Jannis Blüml, Raban Emunds, Florian Peter Busch, and Kristian Kersting. Better decisions through the right causal world model. 2025.arxiv.org/abs/2504.07257
  19. [19]Matteo Gallici, Mattie Fellows, Benjamin Ellis, B. Pou, Ivan Masmitja, J. Foerster, and Mario Martin. Simplifying Deep Temporal Difference Learning. arXiv.org, abs/2407.04811, 2024.arxiv.org/abs/2407.04811
  20. [20]Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. Machine Learning, 113(8):5847–5890, 2024.arxiv.org/abs/2112.13112
  21. [21]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML). Pmlr, 2018.arxiv.org/abs/1801.01290
  22. [22]Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780, 2021.arxiv.org/abs/2109.06780
  23. [23]J. Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When Waiting is not an Option : Learning Options with a Deliberation Cost. In AAAI Conference on Artificial Intelligence, volume abs/1709.04571, 2017.arxiv.org/abs/1709.04571
  24. [24]Rishi Hazra and Luc De Raedt. Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach. In European Conference on Machine Learning and Knowledge Discovery in Databases (PKDD), pp. 213–229, 2023.arxiv.org/abs/2304.08349
  25. [25]Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL http://jmlr.org/papers/v23/21-1342.html.arxiv.org/abs/2111.08819
  26. [26]Minsung Hyun, Junyoung Choi, and Nojun Kwak. Disentangling Options with Hellinger Distance Regularizer. arXiv.org, abs/1904.06887, 2019.arxiv.org/abs/1904.06887
  27. [27]Rodrigo Toro Icarte, T​​oryn Q. Klassen, R. Valenzano, and Sheila A. McIlraith. Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning. In International Conference on Machine Learning, pp. 2112–2121, 2018.
  28. [28]Steven James, Benjamin Rosman, and George Konidaris. Autonomous Learning of Object-Centric Abstractions for High-Level Planning. In International Conference on Learning Representations (ICLR), 2022.
  29. [29]Mu Jin, Zihao Ma, Kebing Jin, Hankz Hankui Zhuo, Chen Chen, and Chao Yu. Creativity of AI: Automatic Symbolic Option Discovery for Facilitating Deep Reinforcement Learning. In AAAI Conference on Artificial Intelligence (AAAI), pp. 7042–7050, 2022.
  30. [31]Timo Kaufmann, Jannis Blüml, Antonia Wüst, Quentin Delfosse, Kristian Kersting, and Eyke Hüllermeier. Ocalm: Object-centric assessment with language models. arXiv preprint arXiv:2406.16748, 2024.arxiv.org/abs/2406.16748
  31. [32]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment Anything. In IEEE International Conference on Computer Vision (ICCV), pp. 3992–4003, 2023.arxiv.org/abs/2304.02643
  32. [33]Hector Kohler, Quentin Delfosse, R. Akrour, Kristian Kersting, and Philippe Preux. Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning. arXiv.org, abs/2405.14956, 2024.arxiv.org/abs/2405.14956
  33. [34]Isaac Lage, Emily Chen, Jeffrey He, M. Narayanan, Been Kim, Samuel Gershman, and Finale Doshi-Velez. An Evaluation of the Human-Interpretability of Explanation. arXiv.org, abs/1902.00006, 2019.arxiv.org/abs/1902.00006
  34. [35]Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. Interpretable Decision Sets - A Joint Framework for Description and Prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1675–1684. ACM, 2016.pubmed.ncbi.nlm.nih.gov/27853627
  35. [36]Matteo Leonetti, Luca Iocchi, and Peter Stone. A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artificial Intelligence, 241:103–130, 2016.DOI
  36. [37]Yuez​​hang Li, Katia P. Sycara, and Rahul Iyer. Object-sensitive Deep Reinforcement Learning. In Global Conference on Artificial Intelligence (GCAI), pp. 20–35, 2017.arxiv.org/abs/1809.06064
  37. [38]Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano. Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving. Robotics and Autonomous Systems, 131:103568, 2020.DOI
  38. [39]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.arxiv.org/abs/1509.02971
  39. [40]Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/2001.02407
  40. [41]Zachary C. Lipton. The mythos of model interpretability. Communications of the ACM, 61(10):36–43, 2018.arxiv.org/abs/1606.03490
  41. [42]Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  42. [43]Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455–16468, 2022.arxiv.org/abs/2210.05639
  43. [44]Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2403.12451
  44. [45]Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. Sdrl: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning. In AAAI Conference on Artificial Intelligence (AAAI), pp. 2970–2977, 2019.arxiv.org/abs/1811.00090
  45. [46]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=IEduRUO55F.arxiv.org/abs/2310.12931
  46. [47]Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents (extended abstract). In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018.DOI
  47. [48]Francis Maes, Raphael Fonteneau, Louis Wehenkel, and Damien Ernst. Policy Search in a Space of Simple Closed-form Formulas: Towards Interpretability of Reinforcement Learning. In International Conference on Discovery Science (DS), pp. 37–51, 2012.
  48. [49]Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning. In International Conference on Machine Learning, volume abs/2402.16801, 2024.arxiv.org/abs/2402.16801
  49. [50]George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
  50. [51]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumar, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.DOI
  51. [53]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. Sam 2: Segment Anything in Images and Videos. In The Thirteenth International Conference on Learning Representations, volume abs/2408.00714, 2025.arxiv.org/abs/2408.00714
  52. [54]Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.DOI
  53. [55]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv.org, abs/1707.06347, 2017.arxiv.org/abs/1707.06347
  54. [56]Mohit Sharma, Jacky Liang, Jialiang Zhao, Alex LaGrassa, and Oliver Kroemer. Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation. In Conference on Robot Learning (CoRL), pp. 822–844, 2020.arxiv.org/abs/2011.04627
  55. [57]Hikaru Shindo, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Blendr: A framework for merging symbolic and neural policy learning. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.11689
  56. [58]Andrew Silva, Taylor W. Killian, I. D. Rodriguez, Sung-Hyun Son, and M. Gombolay. Optimization Methods for Interpretable Differential Decision Trees in Reinforcement Learning. arXiv: Learning, 2019.arxiv.org/abs/1903.09338
  57. [59]Dmitriy Smirnov, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, and Justin M. Solomon. Marionette: Self-Supervised Sprite Learning. In Conference on Neural Information Processing Systems (NeurIPS), pp. 5494–5505, 2021.arxiv.org/abs/2104.14553
  58. [60]Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 112(1-2):181–211, 1999.DOI
  59. [61]Abhinav Verma, Hoang Minh Le, Yisong Yue, and Swarat Chaudhuri. Imitation-Projected Programmatic Reinforcement Learning. In Conference on Neural Information Processing Systems (NeurIPS), pp. 15726–15737, 2019.arxiv.org/abs/1907.05431
  60. [62]Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal Networks for Hierarchical Reinforcement Learning. In International Conference on Machine Learning (ICML), pp. 3540–3549, 2017.arxiv.org/abs/1703.01161
  61. [63]Mike Wu, M. Hughes, S. Parbhoo, M. Zazzi, Volker Roth, and Finale Doshi-Velez. Beyond Sparsity: Tree Regularization of Deep Models for Interpretability. In AAAI Conference on Artificial Intelligence, pp. 1670–1678, 2017.
  62. [64]Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M. Mitchell. Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals. In Conference on Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2302.04449
  63. [65]Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.arxiv.org/abs/2309.11489
  64. [66]Fangkai Yang, Daoming Lyu, Bo Liu, and Steven Gustafson. Peorl: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 4860–4866, 2018.arxiv.org/abs/1804.07779
  65. [67]Yupei Yang, Biwei Huang, Fan Feng, Xinyue Wang, Shikui Tu, and Lei Xu. Towards generalizable reinforcement learning via causality-guided self-adaptive representations. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2407.20651
  66. [68]Zihan Ye, O. Arenz, and Kristian Kersting. Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning. arXiv, 2025.arxiv.org/abs/2505.11661
  67. [69]Renos Zabounidis, Yue Wu, Simon Stepputis, Woojun Kim, Yuanzhi Li, Tom Mitchell, and Katia Sycara. Scalar: Learning and composing skills through llm guided symbolic planning and deep rl grounding. arXiv preprint arXiv:2603.09036, 2026.arxiv.org/abs/2603.09036
  68. [70]Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding DQNs. In International Conference on Machine Learning (ICML), pp. 1899–1908, 2016.arxiv.org/abs/1602.02658
  69. [71]Chiyuan Zhang, O. Vinyals, R. Munos, and Samy Bengio. A Study on Overfitting in Deep Reinforcement Learning. arXiv.org, abs/1804.06893, 2018.arxiv.org/abs/1804.06893

Paper details

vlnc:W_nd8qnphw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents