WorksComputer Science
Distilling Safety Filters for Filter-free Multi-agent RL
Abstract
Reinforcement learning (RL) can teach robot teams to navigate shared spaces, but trial and error causes collisions that real robots cannot afford. Control barrier function (CBF) safety filters prevent these collisions by correcting unsafe actions, yet in a team the filter becomes a centralized optimization whose number of constraints grows quadratically with the team size. We propose an approach that builds the filter's safety into the RL policy and examine how such a system behaves during filter-free deployment. However, the filter that keeps training safe also hides what the policy must learn: it masks the cost of unsafe actions, which actor-critic methods then overvalue, and it keeps near-collision states out of the training data. Our approach therefore learns only from executed safe actions with implicit Q-learning and distills the filter into the policy at synthetic near-collision states. We prove that filter distillation loss bounds the policy's constraint violation and hence how far agents can cross their safety margin, so errors never accumulate. When distillation states cover deployed states, the loss also bounds the collision probability at every step. Without a filter and acting only on local observations, our policies have no collisions in 1,800 test episodes with multiple agents. A policy trained with three agents transfers zero-shot to teams of twenty at constant per-agent computation and stays collision-free with up to eight agents. We test our policies on Crazyflie quadrotors with up to five flying at the same time.
Introduction
Robots increasingly work in teams that share space, where each robot must reach its goal while avoiding collisions with other robots and obstacles. Reinforcement learning (RL) offers a promising approach to learning such behaviours, but its trial-and-error nature causes collisions during training that can damage hardware, so real-world multi-agent RL requires safe exploration from the start. Control barrier function (CBF) safety filters ([3, 4]) address this challenge: at each step, the filter minimally modifies the actions proposed by the policy to satisfy the collision-avoidance constraints, so only safe actions are executed during training ([17]). In multi-agent systems, however, pairwise constraints couple the team into a centralized quadratic program (QP) whose number of constraints grows quadratically with the team size. Solving it requires every agent’s state and proposed action; for 20 agents, the joint QP takes 16.9 ms, a third of the control period, against 0.6 ms per agent for our policy (§ 5.2). This motivates our objective: use a centralized safety filter to guarantee collision-free exploration during training, but learn a decentralized policy that no longer requires the filter at deployment.
Simply removing the filter after training, however, leads to a fundamental distribution mismatch: the policy is trained under filtered action-to-transition mapping but deployed under an unfiltered one. CBF-RL ([53]) adds barrier-based reward shaping so that single-agent policies, trained with the filter, run without it at deployment. Because the filter maps many raw actions to the same safe action ([35], actor-critic methods such as soft actor-critic (SAC) can overvalue unsafe raw actions: SAC trained with the filter in the loop collides in 98% of episodes when deployed without it (§ 5.3).
This observation motivates RL methods that learn only from actions in the data: we extract the policy from the executed safe actions with implicit Q-learning (IQL) (Kostrikov et al., [27]) (§ 3.2). IQL alone is insufficient, however: because the filter prevents collisions, the replay buffer lacks the near-collision states that the unfiltered policy may reach. To cover them, off-support safety-filter distillation (OSFD; § 3.3) trains the policy on the actions of the training filter, queried at synthetic near-collision states, without environment interaction. Following centralized training with decentralized execution, only training uses the joint state, and the deployed policy is an attention-based actor on local observations whose parameters do not depend on the team size (§ 3.4).
We prove theoretically that this training controls safety once the filter is removed (§ 4). The distillation loss bounds a closed-form CBF residual that measures how much the unfiltered policy violates the filter’s constraints, a bounded residual lowers the barrier by a bounded amount uniformly in time, so violations never accumulate, and, when the distillation states cover the deployed states, the loss bounds the collision probability at every step. The training loss thus certifies the safety of the filter-free policy, and every bound held on the 75 trained policies we audited.
Our contributions are summarized as follows: (1) Filter-free multi-agent RL: we propose a novel framework that trains collision-free with a centralized CBF filter and deploys a decentralized policy without it, by combining IQL-based policy extraction with OSFD. (2) Safety guarantees: bounds that tie the distillation loss to the safety of the unfiltered policy under single- and double-integrator dynamics (§ 4). (3) Scalable filter-free collision avoidance: our local-sensing policies have no collision in 1,800 episodes on the training scenes, where CBF-RL collides in up to 44%, a policy trained with three agents transfers zero-shot to twenty at a constant 0.6 ms per agent (§ 5), and our policies fly up to five Crazyflie quadrotors without a filter (§ 5.4).
Related Work
Safe RL with safety filters. Constrained RL (Achiam et al., [1]) bounds expected cumulative costs but not state-wise safety during training. Safety filters enforce constraints at every step through shielding (Alshiekh et al., [2]), safety layers (Dalal et al., [15]), Hamilton-Jacobi reachability (Fisac et al., [17]; Bansal et al. [6]), also learned by RL (Fisac et al., [18]), CBFs (Cheng et al., [12]; Pizarro Bejarano et al. [42]) or control barrier-value functions that unify the two (Choi et al., [14]). Learned filters handle model uncertainty (Choi et al., [13]) and black-box dynamics (Lavanakul et al., [30]), but generally remain in loop at execution. CBF-RL (Yang et al., [53]) trains with a CBF filter and shapes reward so the policy runs without it. We instead address coupled multi-agent constraints and decentralized policies under partial observability. We close the train-deploy mismatch by in-support policy extraction and supervision beyond data rather than reward shaping.
Multi-agent collision avoidance and safety constraints. Learning-based collision avoidance has been explored for ground and aerial swarms (Chen et al., [11]; Long et al. [33]; Batra et al. [7]; Huang et al. [25]). Decentralized barrier certificates (Wang et al., [49]), shields (ElSayed-Aly et al., [16]; Melcer et al. [36]) and CBF filters within multi-agent RL (Cai et al., [8]) avoid a central solve by splitting the constraints among agents, but remain in the loop at deployment, and neural approaches learn decentralized CBFs together with their controllers (Qin et al., [44]; Zhang et al. [56, 57]). In centralized training with decentralized execution (CTDE) (Lowe et al., [34]; Yu et al. [55]), critics use joint information during training, while each deployed policy acts on its own observation. We follow CTDE and additionally use a centralized filter for safe exploration and as a teacher for a filter-free decentralized policy.
Data-driven RL, imitation learning and partial observability. Offline RL counters value overestimation at actions absent from the data [32] by constraining the policy to the behavior policy [20, 28, 50, 38, 19], by penalizing out-of-distribution values [29] or by learning values in-sample [27, 52, 21], also with multiple agents [54, 40]. In-sample learning suits filtered transitions, where unsafe raw actions share safe outcomes, but leaves states missing from the buffer unsupervised. DAgger [47] queries an expert on the learner’s own roll-outs; OSFD instead queries the training filter at synthetic near-collision states, without environment interaction. As in learning with privileged information [9, 31], we use the global state only during training and our local-observation attention actor builds on Zhang et al. [57].
Our Approach
We now present our approach for learning decentralized collision-avoidance policies using a centralized safety filter during training (Figure 1(a)). The actor is trained with two losses: an RL loss on the executed safe actions (§ 3.2) and a distillation loss at synthetic near-collision states (§ 3.3).
Figure 1. (a) The CBF-QP filter keeps training collision-free, the policy learns with IQL on the filtered buffer and with OSFD on filter labels at sampled states and is deployed without the filter, with local sensing within . (b) Three Crazyflies and four stacked-bucket obstacles (§ 5.4), lines are illustrative paths to the goals.
Problem Formulation
We consider disc-shaped agents of radius navigating toward individual goals in an environment with circular obstacles. Each agent follows either single-integrator (SI) or double-integrator (DI) dynamics, discretized with time step :
Here, , and are position, velocity and control, with and, for DI agents, . Unsubscripted norms are Euclidean. Agent observes , either the full scene or its own state and the agents and obstacles within a sensing radius , and receives an individual reward for goal progress, goal completion and collisions (App. § 15). We seek a policy shared by all agents, a Gaussian squashed by , that is trained without collisions and deployed on local observations without a filter or communication.
Safety filter. During training, a centralized CBF filter corrects unsafe actions. With joint state, we index by agent-obstacle, -wall and -agent constraints and give each constraint a barrier :
where obstacle has center and radius , is arena half-width and are margins. With margin for , tightened safe set is ; collision requires . A CBF makes forward invariant: if the CBF condition holds along the trajectory, never becomes negative, so a system that starts in stays in it ([3]). This condition for SI dynamics and the high-order CBF (HOCBF) condition with ([51]) for DI dynamics each give an affine constraint on the joint control, the row of in the filter below, with and under SI dynamics (App. § 9).
Given the joint action proposed by the agents, the filter solves ([49])
Only is executed. Under SI dynamics the constraints are hard (), feasible on , and keep the system in (Prop. 5). Under DI dynamics, states with , such as an agent coasting toward an obstacle, are reachable and may admit no feasible acceleration ([51]), so a penalized slack () keeps the QP feasible (App. § 9). The transitions , with and as current and next-step observations and rewards , are stored in a replay buffer .
Learning from Safe Experience
Under the filter, many raw actions map to the same executed action and to identical transitions. This action aliasing ([35]) lets actor-critic methods overvalue unsafe proposals whose consequences the filter corrected, so we extract the policy from the executed safe actions.
We adopt IQL ([27]) with centralized critics and per-agent values ([34]): with -th critic of agent , its value and parameters of target critic, we minimize
where is the expectile regression loss. The policy is then extracted through advantage-weighted regression ([41]),
with temperature . Minimizing it regresses the policy onto the executed safe actions, weighted toward higher advantage (weights are clipped, normalized and up-weighted where the filter intervened; App. § 10). Since every regression target satisfies the filter’s constraints, the population optimum satisfies every obstacle, wall and agent-agent constraint the agent senses.
Why IQL? Unlike SAC, IQL never maximizes the critic over arbitrary raw actions. Most offline RL methods ([20, 29, 19]) still bootstrap in the next actions sampled from the policy, each of which would need a joint filter to follow the executed dynamics (App. C). IQL instead bootstraps from and evaluates only buffered actions.
Off-support Safety-Filter Distillation
To supervise the policy beyond the replay buffer, OSFD queries the training filter at synthetic safety-critical states instead of executing additional actions. At each update, we draw joint states from a distribution that mixes buffer states, perturbed buffer states and synthetic configurations that place agents close to obstacles, walls or one another, moving toward them. Under DI dynamics, these include states with , which the filtered trajectories rarely visit. Each state is labeled by the training filter, , where is a goal-directed reference controller with a tangential component against deadlock (Reis et al., [46]). The actor imitates these labels with
where excludes agents held at their goals (“active agents”). OSFD needs no environment interaction and, unlike DAgger (Ross et al., [47], never executes the learner’s action, so training remains collision-free. A penalty on the part of the policy’s action that violates the filter’s constraints further sharpens the policy at the constraint boundary (App. § 10).
Partial Observability and Decentralized Execution
We follow CTDE (Lowe et al., [34]) with a strict split: the filter, critics and OSFD labels use the joint state only during training, and the deployed system is the actor alone, acting on local observations. Agent senses the set of agents whose center, and obstacles whose surface, lies within of . It observes an ego vector (goal direction and distance, wall clearances, previous action and, under DI dynamics, its velocity) and one token per sensed entity . Each token holds the normal and value of the filter’s barrier (2) between and , the radius, the relative velocity and the type (one-hot: agent or obstacle), so the actor perceives safety in the filter’s coordinates. The ego features query the encoded tokens by cross-attention (Vaswani et al., [48]),
where is a learned empty token and is an MLP. Attention aggregates any number of tokens into one fixed-size vector, so the actor’s parameters depend on neither the number of agents nor that of obstacles, and a policy trained with three agents runs unchanged on larger teams. Each agent’s computation grows with and needs no communication.
The actor minimizes , where the weights and balance reward against imitation of the filter (§ 5.3), with the constraint-violation penalty and an entropy term (App. § 10). Gradients never pass through the QP. At deployment, each agent executes .
Deployment-Time Safety Analysis
The deployed system is the closed loop (1) under the joint action without a filter. For the filter row of a constraint in (3) and the OSFD label , we define the policy’s CBF residual, its label error and the label’s own violation,
where . The residual is closed-form and needs no QP. OSFD minimizes the label error, when every agent active. The label violation vanishes wherever the hard rows are feasible, in particular on under SI dynamics, and is at most the slack under DI dynamics (App. § 9). Our guarantee chains three bounds,
and in the following, Corollary 3 turns the last bound into a collision probability.
Proposition 1 (Distillation bounds the residual). For every labeled state and every constraint , . Consequently, with every agent active,
where equals for obstacle and wall constraints and for pair constraints, under both dynamics.
Proof. by Cauchy–Schwarz, and the right side is non-negative. Minkowski’s inequality and give (10).
Proposition 1 uses only the label’s feasibility, so it holds for any reference controller, both dynamics and any observation model. Under SI dynamics, , so is the rate at which the policy decreases beyond the rate that the CBF condition allows.
Theorem 2 (The residual bounds the barrier). Let be the CBF gain, the time step with , and the margin of constraint . Along the unfiltered SI loop, for every constraint and every ,
Consequently, for : (i) if for all , then for all , for any horizon; (ii) if for all , there is no collision on during the first steps.
Proof sketch. Convexity of the barriers (Lemma 4) and (8) give one-step bound . Unrolling it gives (11), and summing the geometric series gives (i) and (ii) (full proof in App. § 12).
Residual errors therefore do not accumulate: (11) contracts at rate , so the barrier falls below only after a sustained run of large residuals. By convexity of the barriers, the bound is exact in discrete time and needs no Lipschitz constant of the policy or the dynamics. To bound the collision probability, the deployed states must also be represented in .
Assumption 1 (Coverage). There is a constant such that, for every , the distribution of the deployed state satisfies for every set of states .
is a concentrability coefficient, standard in offline RL ([37, 10]). It bounds how much more often the deployed policy visits any region than OSFD samples it, and, as in offline RL, coverage of the deployed policy alone suffices ([45]). Unlike a fixed dataset, is ours to design, and since the proof uses only through , can be measured (App. § 14).
Corollary 3 (Collision probability). Under Assumption 1 and SI dynamics, with , for every constraint and every step ,
Consequently, the probability of a collision on within steps is at most times the right side.
Proof sketch. By (11), a collision at step requires the discounted residual sum to exceed . By Minkowski’s inequality and Assumption 1, its second moment is at most . Markov’s inequality gives the first bound, Proposition 1 the second, and a union bound over the steps the last claim (App. § 12).
Corollary 3 turns the training loss into a deployment guarantee: every term on its right side is set during training, and the bound holds at every step for any horizon. It also shows why filtered data fall short: the unfiltered loop departs from the filtered one only where the residual is positive (Lemma 6), reaching states absent from the buffer that only OSFD covers. On 75 trained policies, every link of (9) held, and a smaller loss went with a smaller residual and fewer collisions (App. § 14).
Local observation and double integrators. These results hold for any observation model. A decentralized policy cannot match the pair labels, which depend on the neighbor’s intent (Prop. 7), but if every agent keeps the violation of its own half of each pair row below , no two agents collide (Prop. 8). Under DI dynamics, Proposition 1 applies unchanged, and two nested contractions bound the barrier uniformly in time up to an discretization term (Theorem 10, App. § 13). This term can use most of the margin at our control step, so we measure DI safety empirically.
Experiments
We evaluate filter-free safety against CBF-RL (§ 5.1), zero-shot scaling and its cost against the joint QP (§ 5.2), the role of each component (§ 5.3) and transfer to hardware (§ 5.4).
Setup. Agents of radius 0.15 m move in a 10 10 m arena with 2 or 4 circular obstacles of radius 0.4–1.2 m, redrawn every episode ( s, m/s, DI m/s). In the training circle layout, agents start on a circle with antipodal goals, so every episode is an -way crossing through the obstacles at the center, a stress test hardening with ; in the random layout, starts and goals are uniform. Policies use full observation (MLP actor) or local observation (attention actor, m for SI, 3 m for DI). We evaluate the final policy without filter on 3 seeds of 100 episodes each, reporting collisions (% of episodes with any collision) and goals reached (% of agents reaching their goal). Every training run of our method recorded zero collisions (App. § 15).
Baselines. CBF-RL [53] trains with the same filter and adds its barrier-condition reward and intervention penalty, on the multi-agent SAC of our ablations, so it differs from “SAC only” only in the reward (weight screened in App. § 16).
Filter-Free Safety
With local observation, our method had no collision in any of the 1,800 episodes across both dynamics and all three settings (Table 1). CBF-RL matches it with one agent, but with several it collides in 23–67% of DI episodes and, with four obstacles, reaches almost no goals, even with its training filter in place (App. § 16): its barrier reward penalizes approaching any constraint, and in a crowded crossing every route approaches one. Our full-observation MLP collides noticeably only at 3 agents and 4 obstacles (7–8%, all with obstacles), cutting inside the margin where two obstacles leave a narrow gap; the local actor waits instead, at a cost of 4–6% of goals, a ceiling set by the scenes, where the filter itself reaches only 95%. Local sensing is thus the safer observation model.
| Agents / obstacles | Obs. | Single integrator: Ours | Single integrator: CBF-RL | Double integrator: Ours | Double integrator: CBF-RL |
| 1 / 4 | full | 0.3 / 99.3 | 0.3 / 99.3 | 0.0 / 100 | 4.0 / 97.3 |
| local | 0.0 / 100 | 0.0 / 99.3 | 0.0 / 100 | 2.7 / 99.0 | |
| 3 / 2 | full | 0.0 / 100 | 3.3 / 94.3 | 0.3 / 100 | 23.7 / 100 |
| local | 0.0 / 100 | 5.7 / 37.7 | 0.0 / 100 | 22.7 / 99.3 | |
| 3 / 4 | full | 7.3 / 99.7 | 20.7 / 0.2 | 8.3 / 98.9 | 67.3 / 12.4 |
| local | 0.0 / 95.4 | 4.7 / 0.0 | 0.0 / 93.4 | 44.0 / 11.4 |
Table 1. Unfiltered deployment on the training scenes: collisions (% of episodes) / goals reached (% of agents), 3 seeds 100 episodes in simulation.
Zero-Shot Scaling under Partial Observability
We deploy the three-agent DI policies, without retraining, on teams up to twenty (Figure 2 and Figure 3). Safety degrades gracefully: up to eight agents, no episode collides in either layout, and in random layouts collisions stay below 1% up to sixteen agents and reach 3.3% at twenty, nearly seven times the training team, reaching over 99% of goals. In the circle stress test, collisions appear only beyond twelve, mostly with obstacles, whose small margin leaves the smallest residual budget (§ 4).
Figure 2. Unfiltered roll-outs of one DI policy trained with 3 agents and 4 obstacles (local sensing) in simulation on random selection of layouts tested. Open circles: starts; crosses: goals; filled discs: final positions. The three right panels are zero-shot. All agents reach their goals without collision.
Figure 3. Zero-shot scaling of the 3-agent DI policies (local sensing) to up to 20 agents with 2 obstacles, without the filter, in simulation. (a, b) Mean of 3 seeds × 100 episodes; band: per-seed range. (c) Median per-step compute on one CPU thread: agent’s policy versus joint CBF-QP.
Computational cost. Each agent’s local observation policy takes 0.6 ms at every team size, without communication (Figure 3c), whereas the joint QP grows from 0.3 ms at three agents to 16.9 ms at twenty, a third of the 50 ms control period, before any state is gathered. Removing the filter thus removes the one component whose cost grows with the team.
Table 2: Component ablation (DI, local sensing m, trained with 3 agents and 4 obstacles), evaluated without the filter on the training cell and zero-shot with 16 agents. Collisions (% of episodes) / goals reached (% of agents), mean over 3 seeds × 100 episodes in simulation.
| Variant | RL term | OSFD | 3 agents, 4 obs. | 16 agents, 2 obs. (zero-shot) |
| Ours | IQL | ✓ | 0.0 / 93.4 | 7.0 / 99.6 |
| OSFD only | – | ✓ | 0.0 / 93.2 | 2.0 / 92.6 |
| IQL only | IQL | – | 8.0 / 99.9 | 100.0 / 99.9 |
| SAC + OSFD | SAC | ✓ | 0.0 / 95.4 | 13.0 / 99.6 |
| SAC only | SAC | – | 98.3 / 100.0 | 100.0 / 99.7 |
Table 2. Component ablation (DI, local sensing m, trained with 3 agents and 4 obstacles), evaluated without the filter on the training cell and zero-shot with 16 agents. Collisions (% of episodes) / goals reached (% of agents), mean over 3 seeds × 100 episodes in simulation.
Ablations
We remove one component at a time on the hardest setting (DI, local observations, 3 agents and 4 obstacles) and also evaluate each policy zero-shot with 16 agents, where the default policy is not saturated (Table 2).
OSFD makes the policy safe beyond the data. Without OSFD, SAC collides in 98.3% of episodes with the filter removed, the failure predicted by action aliasing, and IQL in 8.0%; zero-shot with 16 agents, both collide in every episode, since the buffer contains no crowded near-collision states, the coverage gap of Assumption 1. With OSFD, no variant collides in training and zero-shot collisions fall to 2–13%, where in-support extraction with IQL has about half of SAC’s (7.0% vs. 13.0%).
The RL term drives the policy toward optimality. OSFD alone imitates the filtered reference controller and receives no reward signal, so it cannot improve on that controller. The RL term supplies this signal: strengthened relative to OSFD, through a larger AWR weight or a smaller OSFD weight , it raises goal reaching from 93.2% up to 97.9%, while every setting remains collision-free on the training setting (Table 5, App. § 16). Zero-shot collisions grow with it, so sets how far beyond the training distribution safety extends.
| Hyperparameter | Value | 3 agents, 4 obs. | 16 agents, 2 obs. (zero-shot) |
| OSFD weight | 10 | 0.0 / 97.9 | 36.0 / 99.7 |
| 25 | 0.0 / 94.7 | 18.3 / 99.7 | |
| 50* | 0.0 / 93.4 | 7.0 / 99.6 | |
| 100 | 0.0 / 93.3 | 1.3 / 99.6 | |
| AWR weight | 3 | 0.0 / 93.8 | 1.7 / 99.5 |
| 10* | 0.0 / 93.4 | 7.0 / 99.6 | |
| 30 | 0.0 / 96.4 | 36.0 / 99.6 | |
| Expectile | 0.6 | 0.0 / 93.2 | 4.3 / 99.6 |
| 0.7* | 0.0 / 93.4 | 7.0 / 99.6 | |
| 0.8 | 0.0 / 94.2 | 12.0 / 99.6 | |
| 0.9 | 0.0 / 94.2 | 13.3 / 99.4 | |
| Training radius (m) | 1.5 | 0.0 / 94.7 | 13.0 / 99.8 |
| 2.0 | 0.0 / 93.9 | 12.0 / 99.6 | |
| 2.5 | 0.0 / 94.2 | 15.7 / 99.6 | |
| 3.0* | 0.0 / 93.4 | 7.0 / 99.6 | |
| (full) | 0.0 / 95.0 | 13.0 / 99.6 |
Table 5. One-hyperparameter sweeps around the default (*) on the cell of Table 2, unfiltered. Collisions / goals (%), mean over 3 seeds.
Hardware Experiments
We fly the 3-agent DI policies on three and five Crazyflie quadrotors ([22, 43]) in a plane at fixed height, with no filter at any point: first the full-observation policies of Table 1, with 2 and 4 obstacles, then the local-sensing policy, all without retraining. Motion capture provides positions, a ground computer builds each drone’s observation and runs the policy, and each drone tracks the training dynamics in a scaled copy of the training arena with stacked-bucket obstacles (App. § 17). With both observation models, every drone reached its goal without contact in 17 of 18 flights: 8 with full observation, 7 with local sensing and 2 with five local-sensing drones (Figure 1(b), Table 6 in App. § 17). No two drones came closer than 0.18 m center to center in these flights (propellers touch at 0.13 m). With local sensing, drones start beyond each other’s sensing radius and resolve crossing from local information alone.
| Policy and layout | Flights | Collisions | Goals (%) | Separation (m) | Clearance (m) | Time (s) |
| Local sensing, 3 drones, 4 obstacles | ||||||
| Scattered | 6 | 0 | 100 | 0.25 | 0.03 | 10.7–13.4 |
| Row, spread | 1 | 0 | 100 | 0.19 | 0.23 | 9.4 |
| Row, tight | 1 | 1 | 66.7 | contact | – | – |
| Local sensing, 5 drones, 2 obstacles (zero-shot) | ||||||
| Circle layout | 2 | 0 | 100 | 0.20 | 0.16 | 10.4–11.4 |
| Full observation, 3 drones | ||||||
| 4 obstacles | 2 | 0 | 100 | 0.26 | 0.16 | 12.9–13.6 |
| 2 obstacles | 3 | 0 | 100 | 0.30 | 0.10 | 10.1–18.2 |
| 2 virtual obstacles | 3 | 0 | 100 | 0.18 | 0.04 | 9.5–13.6 |
Table 6. Hardware flights without a filter (DI, trained with 3 agents). The five-drone flights are zero-shot in the number of drones. Separation: smallest center-to-center distance between two drones. Clearance: smallest horizontal distance from a propeller tip to an obstacle rim (virtual obstacles: to the declared disc). Time: until the last drone reaches its goal.
Conclusion
We present a multi-agent RL framework that trains collision-free with a centralized CBF filter and deploys a decentralized policy with no filter, no communication and only local sensing. This removes the centralized solve whose cost grows with the number of agents. In-support extraction with IQL makes the policy agree with the filter on the data, and off-support safety-filter distillation extends this agreement to near-collision states that the filtered data never contain. The distillation loss provably bounds a closed-form CBF residual of the unfiltered policy, which bounds the barrier uniformly in time and, under coverage, the collision probability at every step. Without a filter, our local-sensing policies had no collisions on their training settings, transferred zero-shot from 3 to 20 agents at constant per-agent cost and flew up to five Crazyflie quadrotors. Since each agent can evaluate its share of every residual locally, future work will monitor the residuals online to decide where to distill next, and will replace motion capture with on-board sensing.
Generative AI Use
Generative AI tools were used to search the literature, to implement code and experiments, to assist with some mathematical derivations and proofs and to write and edit the paper. The research ideas and methodology are the authors’ own. The authors reviewed and verified all AI-assisted content and take responsibility for the final content of the paper.
Acknowledgments
This work was supported by the NSF Safe Learning Enabled Systems project, the DARPA TIAMAT project, the U.S. Army Research Laboratory and the U.S. Army Research Office under Grant W911NF2010219, the Office of Naval Research and the National Science Foundation. It used Jetstream2 at Indiana University through allocation CIS251274 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603 and #2138296. We thank Alex Zalles and the HSL group members for reviewing project ideas and manuscript revisions. We also thank Ebonye Smith, Ian Chuang and JoonHo Lee for their help with the hardware experiments.
Safety Filter: Coefficients, Feasibility and Training-Time Safety
Coefficients. Let denote the admissible CBF action set. Each constraint gives one row of (3), with under both dynamics. For SI dynamics, . For DI dynamics, and give
where for a distance barrier at distance (center to center distance) with unit normal (relative velocity for pairs), and for walls. The DI filter also carries hard rows and , which keep the velocity bound inactive, and it pins the action of agents held at their goals to zero.
Lemma 4 (Convexity). Each barrier in (2) is convex in the joint position , so for every displacement , with equality for walls.
Proof. Obstacles and pair barriers are a Euclidean norm of an affine function of minus a constant; wall barriers are affine.
Proposition 5 (SI: feasibility and collision-free training). For SI dynamics, for every , so the hard filter is feasible on . If and , the filtered loop satisfies for every and : training never collides and the replay buffer contains no collision.
Proof. At each row reads , which holds on . For the executed action , Lemma 4 with gives ; induct on .
DI feasibility. For DI dynamics, and states with are reachable from , for example by coasting toward an obstacle. There the HOCBF row may be unattainable within , so the filter uses the relaxing slack : every is then feasible, and at an active row the executed action violates the row by at most (from the KKT condition of the slack, with the row’s multiplier), which is small at . This slack bounds the label violation of Sec. 4. On a radial approach along , the row is attainable without slack whenever ( at our gains). We claim no discrete-time training guarantee for DI dynamics; every DI replay buffer recorded zero collisions.
Learning Objectives in Full
The value and critic losses are (4), with twin critics , the bootstrap term multiplied by for a termination flag , and one output head per agent, since the game is general-sum.
Advantage weights. In (5), and
The factor up-weights the transitions in which the filter intervened. Every target remains an executed safe action, so the weights change which safe actions the policy prefers, not whether its targets are safe.
Constraint-violation penalty. At the OSFD query states,
where is the violation, by agent ’s own action , of its share of row : a row at least as strict as the filter’s for obstacles and walls, and for pairs the reciprocal half , an antisymmetric shift of the halves in (14) that the joint label satisfies whenever it satisfies the pair row. Under DI dynamics the HOCBF rows are used, with each target capped at so that it stays attainable. The penalty only sharpens the policy at the constraint boundary. The analysis uses the exact rows and bounds the residual through .
Full actor objective. The actor minimizes , where is a sampled action and is tuned toward a target entropy as in SAC (Haarnoja et al., [24]). The entropy term only keeps exploration alive. The regression terms act on the mean tanh , which is the deployed action.
Why IQL over CQL?
CQL (Kumar et al., [29]) penalizes at actions sampled from the policy, so it evaluates the critic at exactly the raw actions whose values action aliasing leaves unidentified. Its Bellman target must also follow the filtered dynamics: the executed policy is , so every sampled next action must pass through the joint filter. That is one QP per transition, 8,192 per environment step at our batch size and update ratio, or about 3 s at 0.4 ms per solve, a cost that dominates training and grows with the team. IQL samples no new action: it bootstraps from and reads only from the buffer, so value learning never calls the filter. The same cost arises for every method whose Bellman target uses a next action sampled from the policy, such as the policy-constraint methods (Fujimoto et al., [20]; Kumar et al., [28]; Fujimoto and Gu, [19]); in-sample methods (Kostrikov et al., [27]; Xu et al., [52]; Garg et al., [21]) avoid it.
Proofs and local observation certificate for Section 4
We analyze the deployed loop without collision handling, i.e., loop up to the first collision. Agents held at their goals execute zero action, and so do their labels, so they contribute to neither side of Proposition 1. For pairs, because the pair barrier has gradient in the joint position.
Proof of Theorem 2. Each barrier in (2) is convex in the positions, so for every displacement (Lemma 4). The SI step is , and (8) gives . Hence
and unrolling gives (11). For (i), the geometric sum is at most . For (ii), it is less than , so , whereas a collision requires . □
Proof of Corollary 3. Fix , let and . By Theorem 2 and , . Assumption 1 applied to gives for every step , so by Minkowski’s inequality
Markov’s inequality gives , the first bound of (12); Proposition 1 gives the second, and a union bound over the episode-level claim. The proof uses only the moment condition , which we measure in App. § 14.
Lemma 6 (The train-deploy shift arises only through the residual). For the hard SI filter, if for every along the filtered trajectory from , then the unfiltered trajectory from coincides with it.
Proof. If , then attains the minimum 0 of (3), so ; induct on .
Local observation. The following results hold for SI dynamics.
Proposition 7 (The pair label depends on the neighbour’s intent). With only the pair row active and the box inactive, the filter’s output for agent is , where are the actions passed to the filter. If does not determine , every decentralized policy has when the row is active.
Proof. Projecting onto the active half-space, with and , moves by with . Given , the only random term in is , and the best squared-loss predictor is its conditional mean, whose error is the conditional variance.
Safety does not require matching the label. A pair row splits into half per agent,
and agent ’s violation of its half, depends only on its own action and its token .
Proposition 8 (Decentralized pair certificate). For every pair, . Hence, if every agent keeps for every agent at every step, no agent-agent collision occurs.
Proof. With the two halves in (14), , and Theorem 2(ii) applies to the pair barrier.
The same holds for any antisymmetric shift of the halves, such as the reciprocal split used in training (App. § 10), and the HOCBF rows of DI dynamics split in the same way.
Unsensed constraints are inactive. An unsensed agent has ; so agent ’s half-row holds for every admissible action whenever ; at our SI settings the two sides are 3.0 and 2.83 m/s. An unsensed obstacle’s row holds since m/s, and walls are always observed. No constraint that the policy could violate lies outside the sensing radius.
Obstacles and walls, agent by agent. For any obstacle or wall row of agent , acts only on agent ’s action, has unit norm, and is determined by when the obstacle is sensed. The proofs above then go through with and with Assumption 1 stated for the distribution of agent ’s observation rather than the joint state.
Double Integrator Certificate
The DI labels satisfy each HOCBF row up to the slack, so and Proposition 1 applies unchanged. For the barrier, write ; for a distance constraint let be the relative velocity and the contact distance ( or ). We assume, up to step , that the velocity bound is inactive and that with . Before the first collision, every segment between samples then stays at distance at least from the obstacle center (or the other agent), since each sample is at distance at least and each segment has length at most . Along such segments, and is Lipschitz with constant (Lemma 9); for walls both are zero. These are hypotheses on the deployed trajectory, not consequence of training.
Lemma 9 (Curvature of a distance barrier). Let on and, at a point with , let . Then , so and . If the segment from to stays at distance at least from , then for every , with .
Proof. Since and , the Hessian is , with eigenvalue along and along the tangent . For the Lipschitz bound, fix a unit and differentiate along a unit direction , using and :
Writing , this is , whose largest magnitude over unit is . Over this peaks at , where it equals . Integrating along the segment from to , on which , and scaling by gives the bound.
Theorem 10 (Sampled-data DI certificate). Let and . Start at rest in , assume the conditions above up to step , and let for the HOCBF row of constraint . Then for all ,
with
so . In particular, if and , no collision on occurs before step .
Proof. Position step. Lemma 4 with gives , with the discrete HOCBF function . Velocity step. Let and . The residual’s definition at gives , so
Since with , we have with . Substituting this and the position step,
using (Lemma 9 with and ), (Lemma 9) and before the first collision. Cascade. At rest, . Unrolling the velocity step, ; unrolling the position step from gives , which is (14).
For a pair, the same argument applies in relative positions and velocities: in the joint position, with and the Hessian of Lemma 9 at . The bound is conservative, but not by much at our control step: in simulation, an adversarial tangential approach at s, and m produces clearance deficits of 0.08–0.10 m, nearly the whole margin, while radial approaches produce none. This is why we measure DI safety empirically.
Audit of the Bounds
Protocol. For each trained policy we rebuild at its training cell as in training, label it with the training filter, and run 100 unfiltered episodes of the deterministic policy, recording the label error , the residual and the barrier of every row at every step. We audit 75 policies: 21 SI policies (ours with full and local observation, CBF-RL and SAC) and the 54 DI policies of Table 2 and Table 5 and of the 2-obstacle setting, at the training cell and, for the 51 ablation policies, zero-shot with 16 agents.
Each link of the chain holds. The three links of (9) were checked directly. Loss to label error: on , equals for SI policies and is 87–97% of it for DI policies, whose goal-held agents are excluded from the loss. Label error to residual: Proposition 1 held on all rows evaluated. Residual to barrier: along the unfiltered SI roll-outs, the one-step inequality behind (11) held on all row-steps before the first collision, and each of the 724 SI collisions, 503 of them by SAC and CBF-RL, was preceded by a discounted residual sum above , as Theorem 2 requires.
Lowering one quantity lowers the next. Across the 21 policies, Spearman rank correlations along the chain are 0.97 from to , 0.98 from to on , and 0.88 from to the collision rate; the loss alone predicts collisions at 0.86 (Table 3, top). Under DI dynamics, and on are ranked identically, and on the deployed states predicts at 0.90. On itself, the DI residual of every policy trained with OSFD is 1.15–1.25 m/s² whatever its loss: the synthetic states include approaches that no admissible acceleration can stop, where the label’s own violation dominates . The two variants trained without OSFD have losses of 0.55 and 1.8, against at most 0.063 with OSFD, residuals on of 4.4 and 9.1 m/s², and are the only DI policies that collide on the training scenes.
| On | Deployed | ||||||
| Policy | Agents/obs. | Coll. (%) | |||||
| SI Ours, local | 5/4 | 0.012 | 0.49 | 0.05 | 0.40 | 0.05 | 0.7 |
| SI Ours, local, zero-shot | 16/2 | 0.012 | 0.49 | 0.05 | 1.70 | 0.17 | 16.3 |
| SI Ours, full obs. | 5/4 | 0.060 | 1.09 | 0.10 | 1.00 | 0.05 | 4.3 |
| SI Ours, full obs. | 8/4 | 0.083 | 1.63 | 0.13 | 1.46 | 0.13 | 10.7 |
| SI Ours, full obs. | 3/8 | 0.327 | 1.98 | 0.38 | 1.95 | 0.26 | 98.0 |
| SI SAC only | 3/4 | 0.938 | 3.36 | 0.78 | 3.47 | 0.13 | 100.0 |
| SI CBF-RL () | 3/4 | 1.459 | 4.18 | 0.81 | 3.39 | 0.04 | 22.0 |
| DI Ours, local | 3/4 | 0.032 | 1.78 | 1.16 | 1.12 | 0.06 | 0.0 |
| DI Ours, local, zero-shot | 16/2 | 0.032 | 1.78 | 1.16 | 5.09 | 0.50 | 7.7 |
| DI OSFD only, zero-shot | 16/2 | 0.026 | 1.63 | 1.18 | 5.29 | 0.29 | 0.7 |
| DI IQL only | 3/4 | 0.550 | 7.38 | 4.43 | 5.63 | 1.98 | 8.3 |
| DI SAC only | 3/4 | 1.802 | 13.45 | 9.07 | 15.48 | 10.19 | 97.7 |
Table 3. Audit of the chain (9) without the filter. On at the training cell: the OSFD loss and the root-mean-square label error and residual ; on the deployed states: the same error and residual, and collisions over 100 episodes per seed. is the larger of the obstacle/wall and pair groups; units m/s (SI) and m/s (DI). Means over 3 seeds; †one seed.
| Group | Single integrator | Double integrator |
| Environment | , , , 1000 steps | same, |
| Filter | , , | , , , |
| IQL | , , weight cap 100, , , target entropy | |
| OSFD | , , 256 states, mixture (0.4, 0.3, 0.3) | same, adverse speed |
| Sensing | ||
| Optimization | Adam, lr , batch 4096, , Polyak 0.005, 2 updates/step, 16 envs | |
| Exploration | 5k goal-directed warm-up steps (noise 0.5); goal-directed proposals w.p. 0.3 per env | |
| Training | 200k steps (1 agent), 300k steps (3 agents) |
Table 4. Hyperparameters.
Coverage. The proof Corollary 3 needs only , which we measure as the smallest such over 50-step windows. On the training scenes it is at most 0.83 for the obstacle and wall rows of every SI policy of our method and 0.04 for every DI policy trained with OSFD, since concentrates on near-collision states; pair rows need up to 8 (SI) and 0.18 (DI).
Implementation Details
Scenes and reward. The arena is (), , and obstacle radii lie in , redrawn every episode. Circle layout: starts evenly spaced on a circle of radius 4 m with antipodal goals. Random layout: starts and goals uniform in , pairwise at least 1 m apart, each start at least 4 m from its goal. An agent within 0.5 m of its goal is held at rest. The reward is , with (Ng et al., [39]); under DI dynamics the directional term uses the velocity, and under the filter the collision terms never fire.
Observations and networks. Full observation: own position, goal offset, offsets to all agents and obstacles, obstacle radii, wall clearances, previous action and, under DI dynamics, all velocities divided by . Local observation: the tokens of § 3.4, 8-dimensional, whose motion slot holds the relative velocity divided by (DI) or the network’s last action (SI); masking by is applied before any learned layer. Actor: squashed Gaussian, a MLP with LayerNorm (full observation) or a per-token MLP of width 128 and 4-head attention followed by a trunk (local observation). Twin critics and value network: MLPs with LayerNorm (Ba et al., [5]) over all agents’ observations, one head per agent. All networks use orthogonal initialization and Adam (Kingma and Ba, [26]).
Training loop and OSFD. Sixteen environments run in parallel. For the first 5k steps, proposals are goal-directed with Gaussian noise; afterwards each agent samples , replaced by the goal-directed proposal with probability 0.3 per environment to keep goal-reaching transitions in the buffer. Each environment step is followed by two updates, each sampling 4096 transitions and building 256 OSFD query states. mixes synthetic boundary states (40%), buffer states (30%) and buffer states with positions perturbed by and DI velocities by (30%). A synthetic state keeps a buffer row’s obstacles and goals and places agents within 0.6 m of an obstacle, a wall or one another, heading toward it (under DI dynamics at speed ). The reference is with , clearance and clockwise tangent ; under DI dynamics, . All QPs are solved with the Goldfarb-Idnani method (Goldfarb and Idnani, [23]). The reported policy is the final actor.
Training curves. Figure 4 shows the training of the full-observation policies of Table 1 with 3 agents and 4 obstacles. Every 10k steps, the current policy is evaluated without the filter on 50 episodes of the training scenes. The filtered training rollouts never collide, yet the unfiltered policy initially collides in almost every episode. As the OSFD loss falls, filter-free collisions drop below 20% within 50k (SI) and 80k (DI) steps and stay near 5% over the last 100k steps, while goal reaching rises above 99% (final values in Table 1). The actor loss follows the OSFD term, which makes up most of it at .
Figure 4. Training with full observation, 3 agents and 4 obstacles: SI (blue) and DI (red), mean over 3 seeds; band: per-seed range. (a, b) Filter-free evaluation of the current policy on the training scenes, 50 episodes every 10k steps. (c) The OSFD term and (d) the full actor objective (App. § 10), logged every 2k steps on one batch and smoothed by a running median over 10k steps; logarithmic axes.
Baselines and Hyperparameters
All baselines use the same filter, environment, networks and evaluation. CBF-RL adds to each agent’s reward , the barrier term and bounded intervention penalty of Yang et al. [53], computed with the rows of the joint filter (a pair row is charged to both agents), with and the row scale, and trains with multi-agent SAC. A screen over (3 agents, 4 obstacles, one seed) gave the fewest collisions at on both dynamics. With its training filter still on, CBF-RL reaches only 0–13% of goals with 3 agents and 4 obstacles, so its low goal rate there is not caused by removing the filter. SAC only runs SAC on the raw action, with the filter treated as part of the environment; SAC + OSFD adds the OSFD terms; IQL only sets ; OSFD only sets , so the critics train but do not reach the actor. Table 5 varies one hyperparameter at a time around the default.
Hardware Setup
Platform and scaling. Three or five Crazyflie 2.x quadrotors (Giernacki et al., [22]) fly at 0.8 m through Crazyswarm2 (Preiss et al., [43]), tracked by a Vicon system at 100 Hz, with a Mellinger controller tracking full-state setpoints. With length scale , time stretch and arena center , we map , , and to , so a trajectory of the training plant is a lab trajectory slowed down by : walls at m, a 10 Hz loop and a sensing radius of 1.125 m.
Control loop. Every 0.1 s a laptop maps the Vicon positions to the training frame, builds each drone’s observation exactly as in training (masked to the sensing radius), evaluates in 0.46 ms for all three drones (0.60 ms for five) with no QP, integrates it with the training plant from the measured position, and sends the result to the drone as a setpoint, which was tracked to within 2.4 cm (95th percentile) in every flight. The velocities the policy sees are the commanded plant velocities. A drone within 0.6 training units of its goal (0.5 in training, enlarged so that a real drone has room to brake) is held in place. The drones exchange nothing; motion capture stands in for on-board sensing.
Obstacles and flights. Each obstacle is a stack of three buckets of rim radius 0.16 m, declared to the policy with radius 0.30 m (0.8 training units) to absorb the drone’s size, its tracking error and the gap between the lab and the training plant; over the 18 flights, the agent’s disc entered a declared disc in four, and no propeller came within 3 cm of a rim. The five-drone flights use the training policy (3 agents, 4 obstacles) unchanged, with five starts and antipodal goals evenly spaced on the training circle and two obstacles; neighbouring drones start 4.6 training units apart, beyond the sensing radius of 3, so the drones sense each other only once the crossing begins. The first full-observation flights used virtual obstacles. Each flight starts with the drones hovering at rest at the starts of the circle layout and ends when all are held at their goals. In the tight row layout, two drones met in the central opening and the policy’s setpoints converged until they touched, so the contact came from the policy rather than from tracking error. Videos of the flights are available at [srikar babug.github.io/ff-safemail](https://srikar babug.github.io/ff-safemail).
References
- [1]Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), 2017.arxiv.org/abs/1705.10528
- [2]Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI Conference on Artificial Intelligence, 2018.arxiv.org/abs/1708.08611
- [3]Aaron D. Ames, Xiangru Xu, Jessy W. Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861–3876, 2017. doi: 10.1109/TAC.2016.2638961.DOI
- [4]Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In European Control Conference (ECC), 2019.arxiv.org/abs/1903.11199
- [5]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.arxiv.org/abs/1607.06450
- [6]Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J. Tomlin. Hamilton-Jacobi reachability: A brief overview and recent advances. In IEEE Conference on Decision and Control (CDC), 2017.arxiv.org/abs/1709.07523
- [7]Sumeet Batra, Zhehui Huang, Aleksei Petrenko, Tushar Kumar, Artem Molchanov, and Gaurav S. Sukhatme. Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning. In Conference on Robot Learning (CoRL), 2021.arxiv.org/abs/2109.07735
- [8]Zhiyuan Cai, Huanhui Cao, Wenjie Lu, Lin Zhang, and Hao Xiong. Safe multi-agent reinforcement learning through decentralized multiple control barrier functions. arXiv preprint arXiv:2103.12553, 2021.arxiv.org/abs/2103.12553
- [9]Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning (CoRL), 2020.arxiv.org/abs/1912.12294
- [10]Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1905.00360
- [11]Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P. How. Decentralized non-communicating multi-agent collision avoidance with deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2017.DOI
- [12]Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI Conference on Artificial Intelligence, 2019.arxiv.org/abs/1903.08792
- [13]Jason Choi, Fernando Castañeda, Claire J. Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control Lyapunov functions and control barrier functions. In Robotics: Science and Systems (RSS), 2020. doi: 10.15607/RSS.2020.XVI.088.DOI
- [14]Jason J. Choi, Donggun Lee, Koushil Sreenath, Claire J. Tomlin, and Sylvia L. Herbert. Robust control barrier-value functions for safety-critical control. In IEEE Conference on Decision and Control (CDC), pages 6814–6821, 2021.arxiv.org/abs/2104.02808
- [15]Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018.arxiv.org/abs/1801.08757
- [16]Ingy Elsayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
- [17]Jaime F. Fisac, Anayo K. Akametalu, Melanie N. Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J. Tomlin. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737–2752, 2019a.arxiv.org/abs/1705.01292
- [18]Jaime F. Fisac, Neil F. Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J. Tomlin. Bridging Hamilton-Jacobi safety analysis and reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2019b.DOI
- [19]Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.06860
- [20]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1812.02900
- [21]Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-learning: MaxEnt RL without entropy. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2301.02328
- [22]Wojciech Giernacki, Mateusz Skwierczyński, Wojciech Witwicki, Paweł Wroński, and Piotr Kozierksi. Crazyflie 2.0 quadrotor as a platform for research and education in robotics and control engineering. In International Conference on Methods and Models in Automation and Robotics (MMAR), 2017.DOI
- [23]Donald Goldfarb and Ashok Idnani. A numerically stable dual method for solving strictly convex quadratic programs. Mathematical Programming, 27(1):1–33, 1983. doi: 10.1007/BF02591962.DOI
- [24]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 1861–1870, 2018.arxiv.org/abs/1801.01290
- [25]Zhehui Huang, Zhaojing Yang, Rahul Krupani, Başkın Şenbaşlar, Sumeet Batra, and Gaurav S. Sukhatme. Collision avoidance and navigation for a quadrotor swarm using end-to-end deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), pages 300–306, 2024.arxiv.org/abs/2309.13285
- [26]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980
- [27]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2110.06169
- [28]Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1906.00949
- [29]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.arxiv.org/abs/2006.04779
- [30]Will Lavanakul, Jason J. Choi, Koushil Sreenath, and Claire J. Tomlin. Safety filters for black-box dynamical systems by learning discriminating hyperplanes. In Learning for Dynamics and Control Conference (L4DC), volume 242 of Proceedings of Machine Learning Research, pages 1278–1291, 2024.arxiv.org/abs/2402.05279
- [31]Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47), 2020.
- [32]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.arxiv.org/abs/2005.01643
- [33]Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2018.arxiv.org/abs/1709.10082
- [34]Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.02275
- [35]Hannah Markgraf, Shambhuraj Sawant, Hanna Krasowski, Lukas Schäfer, Sebastien Gros, and Matthias Althoff. Safe reinforcement learning using action projection: Safeguard the policy or the environment? arXiv preprint arXiv:2509.12833, 2025.arxiv.org/abs/2509.12833
- [36]Daniel Melcer, Christopher Amato, and Stavros Tripakis. Shield decentralization for safe multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.DOI
- [37]Rémi Munos. Error bounds for approximate policy iteration. In International Conference on Machine Learning (ICML), 2003.
- [38]Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.arxiv.org/abs/2006.09359
- [39]Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), pages 278–287, 1999.
- [40]Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 17221–17237, 2022.arxiv.org/abs/2111.11188
- [41]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
- [42]Federico Pizarro Bejarano, Lukas Brunke, and Angela P. Schoellig. Safety filtering while training: Improving the performance and sample efficiency of reinforcement learning agents. IEEE Robotics and Automation Letters, 10(1):788–795, 2025.DOI
- [43]James A. Preiss, Wolfgang Hönig, Gaurav S. Sukhatme, and Nora Ayanian. Crazyswarm: A large nano-quadcopter swarm. In IEEE International Conference on Robotics and Automation (ICRA), 2017.DOI
- [44]Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi-agent control with decentralized neural barrier certificates. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2101.05436
- [45]Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2103.12021
- [46]Matheus F. Reis, A. Pedro Aguiar, and Paulo Tabuada. Control barrier function-based quadratic programs introduce undesirable asymptotically stable equilibria. IEEE Control Systems Letters, 5(2):731–736, 2021. doi: 10.1109/LCSYS.2020.3004797.DOI
- [47]Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 15 of PMLR, pages 627–635, 2011.arxiv.org/abs/1011.0686
- [48]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.03762
- [49]Li Wang, Aaron D. Ames, and Magnus Egerstedt. Safety barrier certificates for collisions-free multirobot systems. IEEE Transactions on Robotics, 33(3):661–674, 2017. doi: 10.1109/TRO.2017.2659727.DOI
- [50]Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.arxiv.org/abs/1911.11361
- [51]Wei Xiao and Calin Belta. Control barrier functions for systems with high relative degree. In IEEE Conference on Decision and Control (CDC), pages 474–479, 2019. doi: 10.1109/CDC40024.2019.9029455.DOI
- [52]Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2303.15810
- [53]Lizhi Yang, Blake Werner, Massimiliano de Sa, and Aaron D. Ames. CBF-RL: Safety filtering reinforcement learning in training with control barrier functions. arXiv preprint arXiv:2510.14959, 2025.arxiv.org/abs/2510.14959
- [54]Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
- [55]Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2022.arxiv.org/abs/2103.01955
- [56]Songyuan Zhang, Kunal Garg, and Chuchu Fan. Neural graph control barrier functions guided distributed collision-avoidance multi-agent control. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2311.13014
- [57]Songyuan Zhang, Oswin So, Kunal Garg, and Chuchu Fan. GCBF+: A neural graph control barrier function framework for distributed safe multiagent control. IEEE Transactions on Robotics, 41:1533–1552, 2025.DOI



