WorksComputer Science
When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning
Abstract
Cooperative multi-agent reinforcement learning (MARL) supports coordinated decision-making in distributed Web services, multi-robot control, and collaborative games through learned coordination conventions. When deployment requirements change, a previously useful convention may become obsolete or unsafe. For example, revised cache-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery. Teams must therefore remove the specified convention while preserving useful cooperation. This is challenging because the same relational pattern can be reproduced by different agents across different histories. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement unlearning and trajectory-deletion methods do not directly target role-invariant conventions. This paper formalizes this problem as semantic coordination-strategy unlearning and proposes CARVE.CARVE uses causal coordination attribution to identify influential interactions and retain-aware subspace updates to limit interference with unrelated behaviors. Counterfactual re-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered. Accepted updates are merged into decentralized actors for direct deployment. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention-completion and worst-case elicitation rates by 91.4% and 83.3%, respectively.The median utility loss is 5.9%, and measured inference latency matches that of the original policies.
Introduction
Cooperative multi-agent reinforcement learning (MARL) enables agents to pursue shared objectives in distributed Web services, multi-robot systems, and collaborative games. Its effectiveness often depends on learned coordination conventions: distributed edge caches coordinate content placement and retrieve objects from neighboring caches to reduce content-delivery latency 1, allied units focus fire in StarCraft micromanagement [11, 32], partners divide cooking and delivery roles in Overcooked [4, 31], and robots negotiate passage through narrow corridors [2, 29]. These conventions simplify joint decision-making and can contribute substantially to policy return. Once formed, however, they also become behavioral patterns that deployed systems may repeatedly reproduce. Removing an obsolete or unsafe coordination pattern while preserving service efficiency and effectiveness is therefore an important problem for reliable multi-agent deployment.
In real-world settings, changing deployment requirements may render some coordination conventions no longer permissible. For example, in a cooperative Web caching service, controllers may learn a division of roles in which one cache leaves an object uncached locally while another retains it and serves subsequent peer fetches. This convention can improve storage utilization and reduce origin requests. If updated service rules prohibit sharing a designated content class across an administrative boundary, the team must abandon conventions that depend on the revoked sharing relation while continuing to serve requests through local caches, permitted peers, or the origin. Such a request targets a specific mode of cooperation, not the entire task and not one particular agent. Removing the target convention therefore requires targeted intervention: the system should abandon the specified convention while retaining the coordination skills needed to make progress in other ways. We call this problem semantic coordination-strategy unlearning.
A natural response is to prohibit the target convention at execution time to satisfy the semantic unlearning objective. In the caching example, a runtime rule can reject disallowed peer fetches, but learned controllers may continue to omit local copies, increasing origin traffic or delivery latency when those fetches are rejected. Our passage experiment in the Vectorized Multi-Agent Simulator (VMAS), a two-dimensional physics simulator for multi-agent robotic learning [2], illustrates this tradeoff between convention suppression and task performance in a controlled coordination setting. The policy is trained with Multi-Agent Proximal Policy Optimization (MAPPO), which uses centralized value estimation during training while allowing agents to choose actions in a decentralized manner at execution time [45]. In this task, the policy learns a bottleneck convention: one agent yields or blocks while another agent passes. Starting from a restored checkpoint, we apply an automaton-based runtime shield that intercepts actions completing the forbidden convention [18, 19]. Although the shield changes the executed action, the original actor network continues to propose the same action pattern. The agents are repeatedly intervened on near the passage and fail to make efficient progress. As shown in Table 1, the shield reduces WER from 0.755 to 0.341, where WER = is the worst-case elicitation, computed over held-out or adversarial contexts . Here, denotes a trajectory, is a relational temporal formula specifying the forbidden convention, and means that the trajectory realizes an accepting instance of that convention, such as one agent blocking or yielding followed by another agent passing within a bounded time window. Lower WER means the forbidden convention is harder to recover under role permutations, starts, or context changes. However, the task return drops from 47.5 to 29.6, and normalized task utility falls to 0.623. Return is the episodic task reward measuring task performance, while utility normalizes how much of the original performance is retained. Thus, an apparent improvement in a convention-suppression metric can hide a severe loss of performance. The real challenge is not only to block the old convention, but to change the learned coordination behavior while preserving a viable way to complete the task.
| Policy | WER | Return | Utility |
| Original checkpoint | 0.755 | 47.5 | 1.000 |
| Runtime shield | 0.341 | 29.6 | 0.623 |
| CARVE (Ours) | 0.340 | 158.2 | 1.000 |
Table 1. Comparison on VMAS Passage from the same restored MAPPO checkpoint. Similar WER values can correspond to different task performance.
This difficulty is not limited to runtime constraints, because a coordination convention is not owned by one agent or one trajectory. Existing reinforcement-unlearning and trajectory-deletion methods do not directly address semantic strategy removal [14, 43]. For example, in the caching scenario, the revoked convention concerns the relation between a cache that omits a local copy and a peer that subsequently supplies the object across the restricted boundary. After deleting trajectories involving one cache pair, other nodes may still reproduce the same placement and fetching pattern. Similarly, forgetting focus fire must cover the same tactic across different allied units and enemy targets. We call this property role invariance: the target convention is defined by roles and interactions rather than fixed agent identities. Moreover, the influence of the convention may be distributed across shared policies, value networks, communication modules, and agents’ responses to teammates. These properties show that fixed agents, trajectories, or state-action pairs are too narrow to serve as the basic unit of such unlearning requests.
These observations motivate a behavioral notion of semantic coordination-strategy unlearning: an edited policy should suppress the specified convention across nominal and eliciting contexts while preserving useful cooperation. We think target conventions should be specified after deployment, because shared team rewards entangle their contributions with useful cooperation, making it difficult to withhold positive credit selectively during training. We therefore study request-driven post-training policy editing, with the accepted update merged into decentralized actors for direct deployment. It provides a concrete comparison target beyond the frequency of forbidden events and includes the preservation of original coordination competence as part of the unlearning requirement. The final deployed policy should satisfy these behavioral requirements without relying on a detachable inference-time prohibition layer.
To address this MARL problem, we propose CARVE. The framework first uses counterfactual action interventions to attribute target-convention risk to agent-time interactions. It then selects parameter subspaces according to the behavior that should be retained and restricts updates to these subspaces, reducing interference with other coordination skills. Counterfactual re-coordination helps agents learn alternative ways to complete the task, avoiding the progress failures caused by merely blocking the original action pattern. Finally, adversarial auditing tests whether the target convention can still be recovered. CARVE connects the semantic scope of an unlearning request with the cooperative behaviors that must be preserved, enabling targeted model updates for semantic coordination-strategy unlearning. Our evaluation spans five representative MARL environments and a set of representative baselines to assess the effectiveness and efficiency of policy unlearning. Compared with runtime shielding, CARVE achieves relative improvements of 167.8% in average task utility. Unlearning is 101.92–584.83 times faster than original backbone training. Meanwhile, the two methods achieve similar average reductions in WER, indicating that reporting target-convention recoverability alone is insufficient; evaluation must also consider task utility, return retention. Our main contributions are summarized as follows:
We first identify limitations of MARL runtime shield and provide the formalization of role-invariant semantic coordination-strategy unlearning in finite-horizon cooperative Markov games.
We propose CARVE, combining causal interaction attribution, counterfactual re-coordination, and adversarial auditing for efficient subspace updates that remove target strategies while preserving performance.
We evaluate CARVE across five MARL benchmarks through basic and stronger elicitation tests, efficiency measurements, and ablations. Compared with the strongest parameter-editing baseline, CARVE improves unlearning effectiveness by 16.9%.
Related Work
MARL benchmarks and algorithms. Reinforcement learning [1, 9] is gradually evolving towards multi-agent systems. A large body of cooperative MARL research has developed benchmarks and algorithms for learning coordination. SMAC established StarCraft micromanagement as a cooperative MARL benchmark [32]; SMACv2 introduced procedural generation and stronger closed-loop evaluation pressure [11]. Overcooked-AI studies human-AI coordination [4], and the Overcooked Generalisation Challenge (OGC) expands evaluation to novel partners and layouts [31]. VMAS provides fast vectorized multi-robot simulation [2]. RWARE and LBF are sparse-reward coordination benchmarks popularized through systematic cooperative MARL evaluation [29]. Melting Pot complements these with test-time social generalization [21]. Standard backbones include MAPPO [45], QMIX [30], VDN [34], MADDPG [26], and COMA [12]. Our work builds on these established environments and backbones, but changes the evaluation question from whether agents can acquire coordination to whether a trained system can selectively forget it while retaining task competence.
Reinforcement and machine unlearning. Machine unlearning [3, 28] is commonly motivated by retraining equivalence: the post-unlearning model should resemble a model trained without the removed data or influence [24, 25, 37, 39, 40] due to the requirements of GDPR [46], CCPA [20] or users’ privacy needs [36, 38]. Reinforcement unlearning extends this idea to RL environments [43], while TrajDeleter targets offline RL trajectory removal [14]. These settings do not cover a distributed, role-invariant coordination convention that can be re-enacted by different agents.
Policy repair, safe MARL, and temporal logic. Another line of work studies how to constrain or repair policies after undesirable behavior has been identified. Temporal logic and reward machines provide automata-based structure for non-Markovian tasks [6, 18, 19]. Safe MARL methods such as MACPO and MAPPO-Lagrangian optimize constraints during policy improvement [16]. PolicyCleanse mitigates RL backdoors [17], and MUTE removes low-value communication [47]; both are relevant conditional baselines for triggered or communication-mediated target strategies. These approaches provide closely related risk–return objectives and are essential baselines for behavioral strategy unlearning. CARVE focuses on request-driven editing of a trained team, using interaction attribution, retain-aware sparse updates, and teacher-guided replacement coordination to suppress role-invariant conventions while preserving cooperative task performance.
Preliminaries
We study a finite-horizon cooperative Markov game as follows:
where is the set of agents, is the global state space, and are the observation and action spaces of agent , is the transition kernel, is the shared team reward, is the discount factor, and is the horizon. At time , agent observes , acts with , and conditions on its local history . We use and for the joint history and joint action. A decentralized policy factorizes as
where denotes the actor parameters. Parameter sharing is allowed: multiple agents may use the same parameter block while receiving different histories. The return of a policy is
where denotes a complete trajectory.
Semantic Requests
The object to be forgotten is not a data point, one fixed agent, or one exact trajectory. It is a role-invariant coordination convention, which can be specified as a relational temporal formula , where denotes “forget”. Atomic predicates are event labels computed from the state and joint action, such as Block, Pass, or Attack. The variables range over agents, and ranges over task entities, such as a specific enemy in a game. For example, the semantic request “forget the convention where one agent blocks a door so another passes shortly after” can be written as
Here means “eventually” and means “within steps.” means that the convention should be removed no matter which concrete agents instantiate the blocker and passer roles. The same notation also covers combat conventions such as focus-fire:
which asks the policy to forget the role-invariant tactic where two allies successively attack the same enemy . For monitoring, we compile into a finite automaton
is the set of monitor states, is the initial state, updates the monitor after seeing the current event labels, and is the set of accepting states indicating that the forbidden convention has occurred. The product state
combines the environment state with the current monitor state . Thus the monitor turns a history-dependent semantic request into a Markovian signal that can be used by critics, attribution, and evaluation.
Unlearning Goal
Let denote an evaluation context, such as a layout, initial-state distribution, teammate population, or adversarial partner. The semantic risk of a policy is
The first term is the probability that the policy executes the forbidden convention in context . The second is the worst-case risk over the evaluation contexts. Given an original policy , an unlearning algorithm returns a new policy . We want to satisfy two behavioral requirements:
The first condition asks for low residual semantic risk under nominal and adversarial evaluation contexts. The second preserves retained task performance; is return measured on target-unrelated or retained contexts, and is the allowed loss. We do not require a separately trained reference policy. Instead, the evaluation directly measures whether the forbidden convention occurs or can be elicited again, using semantic monitors under nominal rollouts, role permutations, held-out contexts, and adversarial partners. This distinction is important because a runtime shield may block the final forbidden action while leaving the underlying coordination preference unchanged. The request is intentionally behavioral in approximate unlearning [22, 35, 41], which does not claim that every parameter-level trace of the original strategy disappears. Instead, they ask whether the delivered policy acts like one that no longer relies on the forbidden convention, while keeping the rest of its cooperative competence.
System Design: CARVE
The strategy unlearning MARL framework CARVE takes as input the original policy , its parameters , rollout logs or simulator access, the semantic request , and optional retain specifications of behaviors or task performance to retain. It outputs a merged policy whose actor parameters are changed. The automaton is used during unlearning and auditing, but not as an inference-time shield. Figure 1 summarizes the four stages.
Figure 1. Overview of CARVE. The framework takes an original policy, rollout data or simulator access, and a semantic request , and returns an edited policy . CARVE first attributes forbidden-convention risk to agent-time interactions, then selects retain-aware sparse edit directions, constructs replacement coordination through a counterfactual teacher, and finally audits the edited policy before merging the accepted parameter update.
Causal Coordination Attribution
The first stage asks a simple question: which agents, at which times, make the forbidden convention more likely? We estimate two centralized critics on the product state :
estimates future task return, and estimates the probability that the monitor eventually accepts. Because records formula progress, can assign credit to early decisions that only become visibly forbidden later.
To measure an agent’s contribution, we compare the observed joint action with counterfactual actions sampled from a task-plausible baseline . For a coalition ,
where keeps the observed actions of agents in , and replaces the other agents’ current actions. Specifically, samples alternative actions from the original policy’s high-probability actions at the current local histories, excluding the observed actions and renormalizing their probabilities. A Shapley-style attribution is then
where is the standard Shapley distribution [33, 44] over coalitions not containing . Thus, means that restoring agent ’s original action increases semantic risk on average. To keep pairwise coordination visible [15], we also estimate
This second score is positive when two agents jointly create more risk than their separate effects explain. In practice, we can approximate these expectations with sampled coalitions rather than enumerating all subsets.
Retain-Aware Strategy Subspace
The second stage turns attribution into parameter directions. Let
be the policy score of agent ’s decision. We convert the attribution scores into one nonnegative weight
where and controls how much pairwise coordination affects the weight. The forbidden-strategy direction for trajectory is
This vector points toward parameters that support decisions causally involved in .
Aggregating over high-risk trajectories gives
the forbidden-strategy sensitivity matrix.
We also estimate a retained Fisher matrix [42] from trajectories and decision points that do not trigger the target convention and should remain stable. Its empirical estimate is computed as
where is the set of retained decision points, , and all policy score vectors are evaluated at the original parameters . Intuitively, a is large if direction changes the forbidden strategy, while is large if disturbs retained behavior.
CARVE chooses edit directions by solving
Here is an edit direction, is its strategy-to-retain ratio, and stabilizes the retain metric. Let contain the top- candidate directions. To obtain sparse edits, we select a fixed coordinate mask containing the top coordinates under the retain-aware semantic sensitivity score, and form , which is a sparsification heuristic. Subsequent optimization is restricted to
Only is optimized; the masked basis remains fixed.
Counterfactual Re-Coordination
Removing a convention can leave a coordination gap: agents may stop using the forbidden strategy but fail to coordinate on an alternative. This stage therefore builds a counterfactual teacher at product states with high . Here, counterfactual refers to alternative joint actions evaluated at the same decision context to construct replacement coordination. For each such state, the teacher favors joint actions that keep task value high and semantic risk low:
normalizes the distribution over joint actions. The coefficient rewards useful alternatives, and penalizes actions likely to complete the forgotten convention. This is a compact way to express a KL-near replacement teacher: stay close to , but tilt probability mass away from the target strategy and toward valuable alternatives. Because the teacher is centralized, we distill its agent-wise marginals into the decentralized actors:
denotes teacher-generated or teacher-relabeled trajectories, and is the teacher’s marginal distribution for agent . The same attribution weight focuses distillation on decisions that matter for the target convention, rather than forcing the entire policy to imitate the teacher everywhere.
Robust Auditing and Policy Merge
The final stage chooses and repeatedly audits the candidate policy . The training objective contains only two terms:
where
keeps target-unrelated behavior close to the original policy. The semantic monitor is used for auditing:
is the empirical worst-case semantic risk over adversarial contexts. These contexts include held-out partners, starts, layouts, and role permutations selected to elicit . If the audit risk is too high, the eliciting trajectories are added to the training set and the update is repeated. We accept the final policy only if it passes the semantic-risk and return checks from the formulation section. The delivered model is obtained by merging into the actor parameters; no monitor or shield is required at deployment.
Experiments
We conduct experiments to answer four questions about CARVE: i) RQ1: whether it removes the requested coordination strategy (§5.2.1, §5.2.3, §5.4)? ii) RQ2: whether it preserves MARL task performance (§5.2.2)? iii) RQ3: whether it is efficient enough to be a post-training edit rather than retraining in disguise (§5.3)? iv) RQ4: which components and hyperparameters control the erasure–retention frontier (§5.5, §C)?
Experimental Setup
5.1.1 Testbed. Experiments are implemented in Python 3.11.8 with PyTorch 2.6.0+cu124 and TorchRL 0.7.2. They run on an Ubuntu 18.04.6 LTS server with two 26-core Intel Xeon Platinum 8269CY CPUs, 256 GB RAM, and four NVIDIA A100-40GB GPUs. Unless otherwise stated, backbone training uses five random seeds.
5.1.2 Evaluation Benchmarks and Unlearning Requests. Our evaluation cross five MARL environments pairs established cooperative MARL testbeds with role-invariant unlearning requests. For SMACv2 [11], the target is a relational combat tactic; for Overcooked-AI/OGC [4, 31], it is a bottleneck or handoff convention; for VMAS [2], it is a continuous-control yielding or leader-follower rule; for RWARE [29], it is a warehouse aisle convention; and for LBF [29], it is a sparse-reward partner-selection convention, which is used as an additional sparse-reward stress test. Details of the scenarios, role-invariant unlearning requests, retain specifications, backbone algorithms, and hyperparameters, are provided in Appendix § 11.
5.1.3 Baselines. The baselines are chosen to answer distinct alternative explanations: whether runtime shielding merely hides the behavior, whether constrained MARL is enough, whether generic parameter unlearning is enough, and whether trajectory removal is enough. We compare against the methods as shown in Table 2.
| Method | Role in evaluation |
| Orig. (trained backbone) | Pre-unlearning policy and target-strategy prevalence. |
| Shield (runtime automaton) | Blocks forbidden completions without model updates. |
| CARVE (Ours) | Sparse retain-aware semantic strategy unlearning. |
| Full-FT | Full-parameter semantic fine-tuning without sparse subspace using negative strategy gradient. |
| RP (semantic reward penalty) | Fixed semantic-cost penalty during post-training fine-tuning. |
| Lag. (Lagrangian MARL) [16] | Adaptive Lagrangian penalty for semantic-cost suppression. |
| MACPO [16] | Conservative constrained policy optimization baseline. |
| Fisher [13] | Fisher-ratio parameter dampening without causal attribution. |
| TrajDel [14] | Tests trajectory removal for role-invariant strategies. |
Table 2. Baseline suite used in the main result tables.
5.1.4 Evaluation Metrics. We assess convention removal using the nominal semantic satisfaction rate and the worst-case elicitation rate . Here, denotes the red-team elicitation set, including held-out teammates or partners when applicable, held-out starts, layouts, procedural context seeds, and role permutations. Both metrics measure the occurrence of the forbidden convention, with lower values indicating more effective suppression. We assess task retention using episodic return (Return) and normalized task-utility retention
where is the original policy’s mean return recorded in the same run, . The evaluation scripts use for SMACv2 and VMAS, which accommodates negative and near-zero original returns, and for Overcooked-AI, RWARE, and LBF, whose reported original returns are positive. Util. is normalized relative to the restored original policy, with 1 indicating full task-performance retention. We measure efficiency using unlearning speedup , inference latency (Infer.), and the fraction of actor parameters changed relative to the original checkpoint (ParamFrac).
Overall Performance
5.2.1 Unlearning Performance. Table 3 shows that CARVE reduces mean SSR and WER by 91.4% and 83.3%, respectively, relative to the original policies. No target-strategy completions are observed under either protocol in Overcooked-AI, RWARE, and LBF. In the two benchmarks with residual risk, CARVE matches the lowest reported WER in SMACv2 and achieves the lowest WER in VMAS. Relative to MACPO, the strongest parameter-editing baseline by mean WER, the largest within-benchmark SSR and WER reductions reach 29.1% and 39.8%, respectively. Compared with Shield, CARVE matches WER in four benchmarks and reduces it by 53.8% in VMAS, while requiring no inference-time monitor. For Full-FT, RP, and Lag., the largest within-benchmark WER reductions range from 33.3% to 56.0%. These gains are consistent with targeting causally attributed coordination decisions, constructing useful alternatives, and incorporating eliciting trajectories during auditing, rather than relying solely on broad updates or semantic penalties. Fisher leaves greater residual semantic risk, while TrajDel still permits target-strategy completions in every benchmark. This shows that parameter dampening may miss the interactions underlying a convention, while deleting selected trajectories does not prevent the strategy’s reproduction through other histories. SMACv2 remains challenging, with a residual WER of 0.500. This matches the lowest observed WER among the evaluated methods. At this risk level, however, CARVE retains a utility of 0.941, compared with 0.749 for Shield and 0.544 for MACPO. Thus, although substantial residual risk remains, CARVE preserves the most task utility among methods achieving the same worst-case suppression. These results highlight the importance of evaluating convention removal jointly with retained coordination competence.
| Method | Metric | SMACv2 Zero | OC-AI Asym. | VMAS Flocking | RWARE Tiny | LBF 8x8-2p |
| Orig. | SSR↓ | 0.469 | 1.000 | 0.357 | 0.875 | 0.792 |
| WER↓ | 0.875 | 1.000 | 0.793 | 0.875 | 0.792 | |
| Shield | SSR↓ | 0.125 | 0.000 | 0.180 | 0.000 | 0.000 |
| WER↓ | 0.500 | 0.000 | 0.480 | 0.000 | 0.000 | |
| CARVE | SSR↓ | 0.144 | 0.000 | 0.157 | 0.000 | 0.000 |
| WER↓ | 0.500 | 0.000 | 0.222 | 0.000 | 0.000 | |
| Full-FT | SSR↓ | 0.312 | 0.000 | 0.292 | 0.000 | 0.000 |
| WER↓ | 0.875 | 0.000 | 0.361 | 0.000 | 0.000 | |
| RP | SSR↓ | 0.250 | 0.000 | 0.187 | 0.000 | 0.000 |
| WER↓ | 0.750 | 0.000 | 0.274 | 0.000 | 0.000 | |
| Lag. | SSR↓ | 0.344 | 0.000 | 0.219 | 0.000 | 0.000 |
| WER↓ | 0.625 | 0.000 | 0.505 | 0.000 | 0.000 | |
| MACPO | SSR↓ | 0.203 | 0.000 | 0.164 | 0.000 | 0.000 |
| WER↓ | 0.500 | 0.000 | 0.369 | 0.000 | 0.000 | |
| Fisher | SSR↓ | 0.422 | 0.000 | 0.399 | 0.000 | 0.042 |
| WER↓ | 0.750 | 0.000 | 0.743 | 0.000 | 0.042 | |
| TrajDel | SSR↓ | 0.403 | 1.000 | 0.292 | 0.750 | 0.792 |
| WER↓ | 0.625 | 1.000 | 0.586 | 0.750 | 0.792 |
Table 3. Unlearning performance across five MARL benchmarks. OC-AI denotes Overcooked-AI.
MARL Retain Performance
Table 4 shows CARVE reduces a median utility of only 5.9% relative to the original policies, with four of the five benchmarks losing at most 6.2%. The retention comparison is particularly informative in Overcooked-AI, RWARE, and LBF: Shield, Full-FT, RP, Lag., and MACPO achieve zero observed SSR/WER alongside zero task return. Their suppression is therefore accompanied by task failure, whereas CARVE maintains positive returns in all three settings. In SMACv2, CARVE matches Shield and MACPO in WER while improving return by up to 80.6%. This pattern is consistent with the intended combination of retain-aware subspace selection, KL retention, and teacher-guided replacement coordination. TrajDel preserves slightly higher mean utility, but the target convention remains elicitable on every benchmark. Relative to TrajDel, CARVE gives up 2.9% in mean utility while reducing mean WER by 80.8%. Fisher preserves utility well in SMACv2 and VMAS but does not consistently retain performance across the remaining tasks. Overcooked-AI exhibits the largest utility loss with a 45.8% reduction. Productive workflows can depend strongly on the convention being removed, making effective replacement coordination difficult. Nevertheless, among methods attaining zero observed SSR and WER, CARVE achieves the highest retained utility; all corresponding baselines yield zero task return. These comparisons show that CARVE offers the strongest observed retention at the lowest reported semantic risk in this setting. The remaining utility gap may reflect both dependence on the removed convention and limitations of replacement coordination.
| Method | Metric | SMACv2 Zerg | OC-AI Asym. | VMAS Flocking | RWARE Tiny | LBF 8x8-2p |
| Orig. | Return | 8.407 | 216.0 | 0.483 | 1.000 | 0.667 |
| Orig. | Util. | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Shield | Return | 4.025 | 0.0 | 0.481 | 0.000 | 0.000 |
| Shield | Util. | 0.749 | 0.000 | 0.902 | 0.000 | 0.000 |
| CARVE | Return | 7.268 | 109.0 | 0.557 | 1.250 | 0.625 |
| CARVE | Util. | 0.941 | 0.542 | 1.000 | 1.000 | 0.938 |
| Full-FT | Return | 5.111 | 0.0 | 0.401 | 0.000 | 0.000 |
| Full-FT | Util. | 0.802 | 0.000 | 0.918 | 0.000 | 0.000 |
| RP | Return | 4.163 | 0.0 | 0.426 | 0.000 | 0.000 |
| RP | Util. | 0.673 | 0.000 | 0.943 | 0.000 | 0.000 |
| Lag. | Return | 4.702 | 0.0 | 0.405 | 0.000 | 0.000 |
| Lag. | Util. | 0.747 | 0.000 | 0.922 | 0.000 | 0.000 |
| MACPO | Return | 4.524 | 0.0 | 0.460 | 0.000 | 0.000 |
| MACPO | Util. | 0.544 | 0.000 | 0.976 | 0.000 | 0.000 |
| Fisher | Return | 7.211 | 0.0 | 0.526 | 0.000 | 0.021 |
| Fisher | Util. | 0.903 | 0.000 | 1.000 | 0.000 | 0.031 |
| TrajDel | Return | 4.783 | 207.0 | 0.400 | 0.875 | 0.667 |
| TrajDel | Util. | 0.803 | 0.958 | 0.916 | 0.875 | 1.000 |
Table 4. MARL retain performance across five benchmarks. OC-AI denotes Overcooked-AI.
Robustness with Stronger Elicitation under Different Unlearning Requests
We further test whether the unlearning performance persist under stronger elicitation across the GiveWay, Passage, and Flocking requests in VMAS dataset, using their respective MADDPG [26], MAPPO [45], and IPPO [10] checkpoints. We use two red-team attacks. The role-permutation attack swaps agent identities before policy evaluation and maps the resulting actions back before stepping. The role-invariant semantic request remains unchanged, so this attack tests whether reassignment of the same roles to different agent indices can recover the target convention. The held-out-context attack evaluates the same edited checkpoint on unseen VMAS context seeds {4001, 4002, 4003}, which vary initial states and procedural task contexts. No additional parameter updates are performed during these tests; CARVE executes its merged actors, and the semantic monitor scores the resulting trajectories.
Table 5 reports standard-context WER (Norm. WER), WER under role permutation (Role WER), and the maximum WER across held-out-context seeds (Ctx. WER). Under role permutation, CARVE achieves the lowest reported Role WER for all three unlearning requests. Relative to the original policies, the reduction across requests has a median of 53.0% and reaches 63.6%. Across parameter-editing baselines, the median Role WER reductions range from 9.3% to 58.9%, with the largest reduction against MACPO reaching 46.9%. Compared with Shield, the median reduction is 12.0%, and the largest reaches 42.1%. These comparisons show that CARVE’s advantage persists when the same roles are reassigned to different agent indices. Under held-out contexts, CARVE reduces Ctx. WER relative to the original policies by a median of 54.6%, with the largest reduction reaching 64.9%. Its median Ctx. WER reductions against parameter-editing baselines range from 5.2% to 59.0%, including 22.2% against MACPO. Relative to Shield, the median reduction is 10.2%, and the largest reaches 48.2%. CARVE achieves the lowest Ctx. WER in GiveWay and Passage, while RP has slightly lower Ctx. WER in Flocking. When risk is summarized by the maximum over the nominal, role-permutation, and held-out-context protocols for each method and request, CARVE nevertheless achieves the lowest value for all three requests. This supports its comparative robustness across the tested role assignments and unseen contexts, while residual strategy completions remain possible.
| Task | Method | SSR↓ | Return↑ | Util.↑ | Norm. WER | Role WER | Ctx. WER |
| GiveWay (MADDPG) | Orig. | 0.502 | 3.708 | 1.000 | 0.502 | 0.509 | 0.517 |
| GiveWay (MADDPG) | Shield | 0.326 | 3.125 | 0.982 | 0.326 | 0.399 | 0.401 |
| GiveWay (MADDPG) | CARVE | 0.313 | 3.679 | 0.998 | 0.313 | 0.351 | 0.360 |
| GiveWay (MADDPG) | Full-FT | 0.399 | 3.172 | 0.990 | 0.399 | 0.407 | 0.432 |
| GiveWay (MADDPG) | RP | 0.404 | 3.276 | 0.991 | 0.404 | 0.416 | 0.423 |
| GiveWay (MADDPG) | Lag. | 0.425 | 3.259 | 0.987 | 0.425 | 0.398 | 0.431 |
| GiveWay (MADDPG) | MACPO | 0.405 | 3.170 | 0.990 | 0.405 | 0.374 | 0.395 |
| GiveWay (MADDPG) | Fisher | 0.481 | 3.152 | 0.985 | 0.481 | 0.487 | 0.491 |
| GiveWay (MADDPG) | TrajDel | 0.501 | 3.713 | 0.892 | 0.501 | 0.501 | 0.506 |
| Passage (MAPPO) | Orig. | 0.338 | 47.5 | 1.000 | 0.755 | 0.747 | 0.756 |
| Passage (MAPPO) | Shield | 0.203 | 29.6 | 0.623 | 0.341 | 0.360 | 0.374 |
| Passage (MAPPO) | CARVE | 0.208 | 58.2 | 1.000 | 0.340 | 0.351 | 0.343 |
| Passage (MAPPO) | Full-FT | 0.248 | 103.1 | 1.000 | 0.365 | 0.398 | 0.367 |
| Passage (MAPPO) | RP | 0.217 | 128.1 | 1.000 | 0.373 | 0.406 | 0.362 |
| Passage (MAPPO) | Lag. | 0.229 | 133.9 | 1.000 | 0.512 | 0.521 | 0.623 |
| Passage (MAPPO) | MACPO | 0.213 | 143.5 | 1.000 | 0.406 | 0.387 | 0.441 |
| Passage (MAPPO) | Fisher | 0.441 | 103.7 | 0.000 | 0.821 | 0.853 | 0.836 |
| Passage (MAPPO) | TrajDel | 0.337 | 48.5 | 1.000 | 0.754 | 0.745 | 0.754 |
| Flocking (IPPO) | Orig. | 0.357 | 0.483 | 1.000 | 0.793 | 0.797 | 0.756 |
| Flocking (IPPO) | Shield | 0.180 | 0.481 | 0.902 | 0.480 | 0.501 | 0.541 |
| Flocking (IPPO) | CARVE | 0.157 | 0.557 | 1.000 | 0.222 | 0.290 | 0.280 |
| Flocking (IPPO) | Full-FT | 0.292 | 0.401 | 0.918 | 0.361 | 0.443 | 0.387 |
| Flocking (IPPO) | RP | 0.187 | 0.426 | 0.943 | 0.274 | 0.358 | 0.271 |
| Flocking (IPPO) | Lag. | 0.219 | 0.405 | 0.922 | 0.505 | 0.653 | 0.533 |
| Flocking (IPPO) | MACPO | 0.164 | 0.460 | 0.976 | 0.369 | 0.546 | 0.435 |
| Flocking (IPPO) | Fisher | 0.399 | 0.526 | 1.000 | 0.743 | 0.858 | 0.813 |
| Flocking (IPPO) | TrajDel | 0.292 | 0.400 | 0.916 | 0.586 | 0.737 | 0.719 |
Table 5. Stronger unlearning audit under two different attacks on three VMAS forgetting tasks. Bold numbers indicate the largest reported value among the three audit settings.
Unlearning Efficiency
Table 6 shows that CARVE achieves the highest reported unlearning time speedup across all five benchmarks, ranging from 101.92× to 584.83× relative to original backbone training, including loading data, gradient-mask construction, and update time. It is 1.25–2.29 times faster than Full-FT. Its inference latency matches the original policy, while Shield increases latency by 19.2%–31.2%. This overhead arises from additional runtime automaton monitoring and action filtering. Figure 2 reports ParamFrac, the fraction of actor parameters changed relative to the original checkpoint. CARVE modifies 13.1% on average, ranging from 10.5% to 20.0%, compared with 90.6%–100.0% for RP, Lag., MACPO, Full-FT, and TrajDel. Fisher also has a small edit footprint of 12.0% in every benchmark, but records lower unlearning speedups than CARVE. This illustrates that edit locality alone does not determine runtime, which also depends on parameter selection and optimization costs. CARVE optimizes only the coefficients in the retain-aware update . The accepted update is merged into the actors, enabling deployment without an inference-time monitor or shield.
Figure 2. Fraction of actor parameters modified by each unlearning method across five MARL benchmarks
| Benchmark | Metric | Orig. | Shield | CARVE | Full-FT | RP | Lag. | MACPO | Fisher | TrajDel |
| SMACv2 (Zerg) | Speedup() | – | – | 584.83 | 255.06 | 290.30 | 266.87 | 273.08 | 575.38 | 482.07 |
| Infer.(ms) | 120.326 | 143.460 | 120.326 | 120.326 | 120.326 | 120.326 | 120.326 | 120.326 | 120.326 | |
| Overcooked-AI (Asym.) | Speedup() | – | – | 101.92 | 44.77 | 48.68 | 45.59 | 47.20 | 94.32 | 82.37 |
| Infer.(ms) | 80.750 | 104.776 | 80.750 | 80.750 | 80.750 | 80.750 | 80.750 | 80.750 | 80.750 | |
| VMAS (Flocking) | Speedup() | – | – | 454.53 | 363.68 | 391.25 | 326.47 | 357.24 | 416.54 | 423.68 |
| Infer.(ms) | 86.718 | 107.856 | 86.718 | 86.718 | 86.718 | 86.718 | 86.718 | 86.718 | 86.718 | |
| RWARE (Tiny) | Speedup() | – | – | 247.47 | 178.54 | 199.10 | 182.61 | 188.69 | 228.85 | 209.02 |
| Infer.(ms) | 100.735 | 122.740 | 100.735 | 100.735 | 100.735 | 100.735 | 100.735 | 100.735 | 100.735 | |
| LBF (8x8-2p) | Speedup() | – | – | 312.94 | 209.17 | 228.46 | 216.34 | 220.46 | 264.57 | 291.89 |
| Infer.(ms) | 70.605 | 92.617 | 70.605 | 70.605 | 70.605 | 70.605 | 70.605 | 70.605 | 70.605 |
Table 6. Unlearning efficiency across five MARL benchmarks. Speedup is ; Infer. is inference latency.
Visualization
We use VMAS Flocking to inspect whether semantic unlearning changes the policy itself. The target convention is leader-follower dominance: agents move cohesively while one agent exerts disproportionate control. Figure 3 compares the original restored actor and CARVE on the same rollout diagnostics. In the monitor row, , , and denote low-risk, recoverable, and accepted forbidden-convention states. The original actor shows the target signature across all views: clustered trajectories, a semantic-score peak above
Figure 3. Visualization for VMAS Flocking. Rows from top to bottom show the rotated physical trajectory, the semantic score for leader-follower dominance, per-agent action norms, and occupancy of the monitor states , , .
the accepting threshold, uneven action norms, and visible monitor occupancy. CARVE weakens this signature. Its trajectories are less organized around a single leading path, the semantic score remains below the accepting region in the shown rollout, action magnitudes are more balanced, and monitor mass shifts toward . This supports the main claim: CARVE removes the leader-follower convention while preserving coordinated flocking, rather than merely blocking actions at runtime.
Ablation Study
We ablate five components on VMAS GiveWay, Passage, and Flocking using the same checkpoints, semantic requests, retain sets, and audit contexts. The full model uses the complete pipeline in Eqs. (12)–(25). The ablated variants replace the attribution scores in Eqs. (12)–(13) with random sparse weights, replace the retain-aware eigenspace in Eq. (19) with semantic-gradient coordinates, drop from Eq. (23), drop the counterfactual teacher in Eqs. (21)–(22), or optimize the full actor instead of the subspace update in Eq. (20). Table 7 reports task averages. Causal attribution increases ΔSSR and ΔWER by 147.1% and 142.0%, respectively, compared with random weights, supporting more effective targeting of the convention. Retain-aware selection improves ΔWER by 27.3% and utility by 10.5%, while the retention loss improves utility by 54.9% with nearly unchanged semantic reductions. These components therefore provide complementary protection for retained cooperation. Counterfactual replacement improves WER by 43.1% and mean return by 61.6%, supporting productive alternative coordination. Subspace restriction improves utility by 35.5% while reducing the changed-parameter fraction by 87.8%, indicating that limiting edit scope helps preserve task competence. Together, these ablations show that the components complement one another in balancing effective convention removal with efficient retained cooperation.
| Variant | SSR | WER | Util. | Return | ParamFrac |
| Full CARVE | 0.173 | 0.392 | 0.999 | 54.15 | 0.120 |
| w/o attribution | 0.070 | 0.162 | 0.963 | 51.7 | 0.118 |
| w/o retain-aware mask | 0.159 | 0.308 | 0.904 | 33.3 | 0.120 |
| w/o retain loss | 0.174 | 0.390 | 0.645 | 32.8 | 0.120 |
| w/o replacement reg. | 0.129 | 0.274 | 0.987 | 33.5 | 0.120 |
| w/o sparse subspace | 0.130 | 0.340 | 0.737 | -103.6 | 0.986 |
Table 7. Ablation study on VMAS GiveWay, Passage, and Flocking. SSR and WER are reductions relative to the restored original checkpoint.
Conclusion
This paper studied semantic coordination-strategy unlearning, where a trained cooperative policy must abandon a role-invariant convention while retaining useful cooperation. CARVE combines causal coordination attribution, retain-aware subspace updates, counterfactual re-coordination, and adversarial auditing, then merges the accepted update into decentralized actors. Experiments across five MARL benchmarks show that CARVE balances target-convention removal with the preservation of cooperative task performance and no additional inference latency.
References
Algorithm Details
CARVE Pseudocode
As shown in Algorithm 1, CARVE is a post-training unlearning framework. It starts from a restored cooperative policy, uses the automaton monitor to locate decisions that causally support the forbidden convention, restricts the update to retain-aware strategy-sensitive directions, distills replacement coordination, and finally audits the edited actor under stronger elicitation contexts. The automaton and red-team contexts are used during editing and verification; the delivered policy is the edited actor itself and does not require an inference-time shield.
Complexity Analysis
Let be the number of collected rollouts, the horizon, the number of agents, the number of actor parameters, the number of sampled coalitions per target-positive decision, the retained subspace rank, the editable actor-parameter fraction, the number of projected optimization updates, the number of audit contexts, and the number of audit episodes per context. Monitor annotation is linear in trajectory length, , excluding simulator cost. Attribution requires semantic-critic evaluations in the sampled-coalition implementation, rather than enumerating all coalitions. Building the retain-aware subspace costs to accumulate score-gradient statistics plus for iterative generalized-eigen updates with diagonal or low-rank sensitivities are used; a dense eigensolver would be and is not the intended implementation. Selecting the edit mask costs to score coordinates plus a top- selection. Projected optimization costs for minibatch actor forward/backward passes, but only actor coordinates receive nonzero updates. Materializing the masked subspace costs . The final audit costs simulator steps. In practice, the dominant costs are simulator rollouts and actor-gradient passes; memory is for the actor and masked subspace, plus the rollout buffer.
Evaluation and Hyperparameters Details
Table A1 summarizes the task scenarios, role-invariant semantic unlearning requests, retained behaviors, and backbone algorithms across five cooperative MARL environments. Unless otherwise stated, SMACv2 and VMAS backbone training uses five random seeds, 3.072M environment frames per run, 32 evaluation episodes every 122,880 frames, and checkpointing every 307,200 frames. SMACv2 uses four environment workers and 2,000 collected frames per batch, while VMAS uses 32 workers and 6,144 collected frames per batch. For post-checkpoint CARVE runs, the default setting is 800 update steps, actor edit fraction , retain-action weight , learning rate , , , and . For deterministic actors such as MADDPG, evaluation actions are generated by adding independent Gaussian noise in the atanh latent space of normalized original actions, followed by a tanh transformation and rescaling to the native action range.
| Benchmark | Scenario | Semantic request | Retain behavior | Backbone |
| SMACv2 [11] | Zerg 5v5 | Role-invariant focus-fire: two or more allied units attack the same enemy at the same decision step, independent of ally identities. | Episode return, battle win rate, and killed-enemy count. | MAPPO [45] |
| Overcooked-AI/OGC [4, 31] | Asymmetric Advantages | Fixed cook–deliver split: one player contributes cooking/potting events while the partner contributes dish, soup, or delivery events. | Sparse/shaped soup return and total cooking plus delivery event count. | IPPO [10] |
| RWARE [29] | Tiny 2-agent warehouse | Fixed carrier–yielder aisle convention: the target carrier handles shelf deliveries while the paired robot yields during pickup or carrying. | Shelf-delivery return and non-target delivery count. | QMIX [30] |
| LBF [29] | 8x8-2p | Fixed-pair collection: the target pair jointly collects sparse-reward food. | Food-collection return and non-target pair collections. | VDN [34] |
| VMAS [2] | GiveWay | Yield-then-cross: at the narrow crossing, one agent stays low-motion near the center while another crosses with high motion. | Task return, completion rate, and collision avoidance. | MADDPG [26] |
| VMAS [2] | Passage | Single-file gate priority: near the passage, one agent queues or yields while another takes priority through the bottleneck. | Passage return, completion rate, and collision avoidance. | MAPPO [45] |
| VMAS [2] | Flocking | Leader-follower dominance: a cohesive flock moves while one agent has a disproportionately large action norm and drives the group. | Flocking return, completion rate, and collision avoidance. | IPPO [10] |
Table A1. Concrete scenarios used in the main experiments. Each row reports one implemented semantic request, the corresponding retained behavior, and the single backbone used for the reported checkpoint.
Parameter Sensitivity
We sweep the hyperparameters that control the erasure–retention frontier on the three VMAS post-checkpoint tasks: GiveWay, Passage, and Flocking. The validation grid is update steps in , edit fraction in , retain weight in , and learning rate in . Table A2 reports the average semantic reduction, retained utility, task reward, and parameter locality together. Increasing the edit fraction from 0.06 to 0.12 improves WER by 16.7%, whereas expanding it to 0.48 yields only a further 1.5% gain with four times as many changed parameters. Increasing the retain weight from 0.5 to 4.0 reduces WER by 44.6%, with little additional utility retention. Among the tested learning rates, achieves the strongest semantic reductions. More updates do not consistently improve forgetting: 800 steps increase mean reward by 50.1% over 200 steps with comparable semantic reductions, while 1200 steps weaken forgetting. These results support the default configuration as a practical balance between convention removal, task performance, and edit locality.
| Sweep | Value | |||||
| Edit fraction | 0.03 | 0.122 | 0.253 | 0.966 | 49.1 | 0.030 |
| Edit fraction | 0.06 | 0.164 | 0.336 | 0.999 | 52.8 | 0.060 |
| Edit fraction | 0.12 | 0.173 | 0.392 | 0.999 | 54.2 | 0.120 |
| Edit fraction | 0.24 | 0.176 | 0.398 | 0.999 | 54.3 | 0.240 |
| Edit fraction | 0.48 | 0.181 | 0.398 | 0.999 | 54.8 | 0.480 |
| Retain weight | 0.1 | 0.187 | 0.395 | 0.977 | 52.1 | 0.120 |
| Retain weight | 0.5 | 0.173 | 0.392 | 0.999 | 54.2 | 0.120 |
| Retain weight | 1.0 | 0.170 | 0.387 | 0.999 | 53.0 | 0.120 |
| Retain weight | 2.0 | 0.164 | 0.278 | 0.999 | 52.8 | 0.120 |
| Retain weight | 4.0 | 0.118 | 0.217 | 1.000 | 50.9 | 0.120 |
| Learning rate | 1e-4 | 0.134 | 0.338 | 0.965 | 52.8 | 0.120 |
| Learning rate | 2e-4 | 0.173 | 0.392 | 0.999 | 54.2 | 0.120 |
| Learning rate | 5e-4 | 0.144 | 0.377 | 0.998 | 53.3 | 0.120 |
| Update steps | 200 | 0.177 | 0.399 | 1.000 | 36.1 | 0.120 |
| Update steps | 400 | 0.137 | 0.318 | 0.998 | 53.5 | 0.120 |
| Update steps | 800 | 0.173 | 0.392 | 0.999 | 54.2 | 0.120 |
| Update steps | 1200 | 0.130 | 0.335 | 0.997 | 56.7 | 0.120 |
Table A2. Parameter sensitivity on VMAS.
Teacher–Actor Gap Experiment
Marginal distillation may discard joint-action correlations encoded by the centralized teacher. Table A3 therefore compares the joint teacher with the independently sampled edited actor on paired nominal episodes, using one trained checkpoint per VMAS scenario. Both are nominal-context score diagnostics. The observed gap is scenario-dependent. GiveWay shows nearly unchanged semantic scores and return. In Passage, the actor’s SSR and WER point estimates decrease by 0.023 and 0.022, respectively, alongside a higher return point estimate. In Flocking, they increase by 0.019 and 0.039, respectively, alongside a lower return point estimate. Thus, the actor is not uniformly worse than the teacher. The teacher’s local, critic-based KL-regularized objective does not guarantee dominance on episode-level metrics. All paired bootstrap 95% confidence intervals for contain zero, leaving return differences statistically unresolved. This does not establish equivalence, particularly given the wide Passage interval. The comparison quantifies the combined Algorithm 1 CARVE semantic coordination-strategy unlearning.
| Scenario | Paired 95% CI of | ||||||
| GiveWay | 0.312 | 0.313 | 0.315 | 0.313 | 3.678 | 3.679 | |
| Passage | 0.231 | 0.208 | 0.362 | 0.340 | 121.1 | 158.2 | |
| Flocking | 0.138 | 0.157 | 0.183 | 0.222 | 0.604 | 0.557 |
Table A3. Joint-teacher versus actor performance on three VMAS scenarios. WER is measured in the nominal context. Confidence intervals bootstrap paired evaluation episodes.
Input: Restored policy with actor parameters ; simulator ; forbidden formula ; retain data ; monitor automaton ; edit rank ; actor edit fraction ; update budget ; audit contexts ; minimum semantic reduction ; utility threshold ; required consecutive passes ; audit interval .
Output: Accepted edited policy , or NotAccepted.
Evaluate baseline scores and on .
Roll out in training and retain contexts; update product states with .
Mark target-positive and retain decision points from the monitored trajectories.
Fit for target-convention risk and for task value.
for each target-positive decision point do
Sample coalitions and estimate by Eqs. (12) and (13).
Convert attribution scores into weights by Eq. (15).
end for
Compute by Eq. (16); estimate and .
Solve Eq. (19); retain the top- candidate directions .
Select the top coordinates by retain-aware semantic sensitivity; form and as a heuristic sparsification.
Fix and restrict subsequent optimization to .
Build the centralized replacement teacher on high-risk product states using Eq. (21).
Define the decentralized distillation loss by Eq. (22).
Initialize .
for do
Update with Eq. (23), using from Eq. (24).
if or then
Audit on ; obtain , , and .
if and and then
Accept ; merge into the actors.
Report SSR/WER, retention metrics, and ParamFrac.
return
end if
end if
end for
return NotAccepted.
effects of correlation loss, local observability, and constrained actor fitting.
Theory
We analyze attribution, surrogate subspace selection, the replacement teacher, conditional return retention, and decentralized information limits.
Attribution consistency.
Proposition 1 (Shapley efficiency and estimation error). For a fixed intervention baseline, let and be the exact Shapley values of and its estimate . If , then
The pair term in Eq. (13) vanishes for additive . These statements concern signed attributions before clipping.
Proof. Shapley values average permutation marginals, whose sum telescopes from to . Each estimated marginal has error at most . Additive coalition functions have zero second differences.
Retain-aware subspace. Let and , with . The semantic sensitivity surrogate is
Theorem 1 (Retain-normalized sensitivity). For , the top generalized eigenvectors of , normalized by , solve
The optimum is the sum of the top generalized eigenvalues.
Proof. Set and . The problem becomes ; the Ky Fan maximum principle gives the result.
Because Eq. (27) is unchanged by , this theorem does not guarantee semantic-risk reduction. For a fixed coordinate mask, it applies to an eigenproblem solved on the selected principal submatrices, with no larger than the selected dimension; it does not imply optimality of the mask or of a subsequently masked unrestricted basis.
Replacement teacher. At a fixed context, write and .
Proposition 2 (KL-regularized teacher). For bounded , the teacher in Eq. (21) uniquely minimizes
over distributions .
Proof. With ,
which is minimized uniquely at . □
This optimum concerns the centralized teacher; marginal distillation need not preserve its joint-action correlations.
KL-based retention. For , define
Here is the population retain-history marginal, with the time weights of Eq. (24); is its empirical counterpart.
Proposition 3 (Conditional return retention). Fix an -step context with common initial, transition, observation, and reward kernels, , and . Let be the original-policy history law. Suppose, for all ,
where the first inequality is between measures, , and is finite and valid for the fitted edit. With ,
Proof. Common environment kernels and factorized actors give
Bounded returns and Pinsker’s inequality yield , proving the claim. □
The KL direction matches Eq. (24). Coverage and generalization are additional conditions, not certified by a small replay loss or action-matching MSE; the experiments do not verify them.
Decentralized information limit.
Theorem 2 (Indistinguishable decision situations). If two same-time situations give every agent identical local histories but require disjoint joint-action sets and , no history-conditioned decentralized policy satisfies both requirements with probability one. At least one situation has violation probability at least .
Proof. Identical histories induce the same joint-action law . Disjointness implies , so at least one required set has probability at most . □
This theorem identifies an information limit that parameter editing alone cannot resolve. Its error bound is conditional on the two indistinguishable decision situations and does not directly bound overall SSR/WER or expected return.
Limitations
The formulation assumes cooperative finite-horizon MARL, CTDE training, target strategies expressible through observable relational predicates, and limited simulator access for suffix regeneration and red-team auditing. If the original reward continues to make the forgotten strategy uniquely optimal, subsequent unconstrained training can relearn it; preventing that is continual semantic revocation rather than one-shot unlearning. The proposed guarantees are behavioral and semantic, not direct legal-data-deletion guarantees. Future work will investigate replacement policies that better preserve joint-action correlations during decentralized distillation and broader auditing over partners and deployment contexts.
References
- [1]Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE signal processing magazine 34, 6 (2017), 26–38.
- [2]Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. 2022. Vmas: A vectorized multi-agent simulator for collective robot learning. In International symposium on distributed autonomous robotic systems. Springer, Cham, 42–56.arxiv.org/abs/2207.03530
- [3]Lucas Bourtou le, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP). IEEE, Virtual Site, 141–159.
- [4]Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019), 12 pages.arxiv.org/abs/1910.05789
- [6]Linghan Chen, Yudong Gao, Jiaoy Wang, Kaiyan Ji, and Honglong Chen. 2026. Do system prompts leave behavioral fingerprints? a large-scale empirical study of clone detection via output similarity. arXiv:2608.24461 [cs.CR].
- [9]Qiuli Dai, Honglong Chen, Zhichen Ni, Yubin Yang, Linghan Chen, Xinglong Fan, and Xiangyu Li. 2025. Reinforcement learning-based resource reservation for mobile edge computing with probable failure. IEEE Transactions on Vehicular Technology 74, 5 (2025), 7985–7996.
- [10]Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviichuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. 2020. Is independent learning all you need in the starcraft multi-agent challenge? arXiv:2011.09533 [cs.AI].arxiv.org/abs/2011.09533
- [11]Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. 2023. An improved benchmark for cooperative multi-agent reinforcement learning. Advances in neural information processing systems 36 (2023), 37567–37593.
- [12]Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. AAAI Press, New Orleans, Louisiana, USA, 9 pages.arxiv.org/abs/1705.08926
- [13]Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. AAAI Press, Vancouver, Canada, 12043–12051.arxiv.org/abs/2308.07707
- [14]Chen Gong, Kecen Li, Jin Yao, and Tianhao Wang. 2024. Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS). Internet Society, San Diego, CA, USA.arxiv.org/abs/2404.12530
- [15]Michel Grabisch and Marc Roubens. 1999. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory 28, 4 (1999), 547–565.
- [16]Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. 2021. Multi-agent constrained policy optimisation. arXiv:2110.02793 [cs.AI].arxiv.org/abs/2110.02793
- [17]Junfeng Guo, Ang Li, Lixu Wang, and Cong Liu. 2023. Polycycles: Backdoor detection and mitigation for competitive reinforcement learning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Paris, France, 4676–4685.DOI
- [18]Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael Wooldridge. 2021. Multi-agent reinforcement learning with temporal logic specifications. In Proceedings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 583–592.arxiv.org/abs/2102.00582
- [19]Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. 2018. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning. PMLR, Stockholm, Sweden, 2107–2116.
- [20]Erin Illman and Paul Temple. 2019. California consumer privacy act. The Business Lawyer 75, 1 (2019), 1637–1646.
- [21]Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jay Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. 2021. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning. PMLR, Virtual Site, 6187–6199.
- [22]Na Li, Chunyi Zhou, Yansong Gao, Hui Chen, Zhi Zhang, Boyu Kuang, and Anmin Fu. 2025. Machine unlearning: Taxonomy, metrics, applications, challenges, and prospects. IEEE Transactions on Neural Networks and Learning Systems 36, 8 (2025), 13709–13729.DOI
- [24]Huanghuang Liang, Weihan Wu, Rui Ge, Jiawei Jiang, and Dazhao Cheng. 2026. Decoupled Attribute Subspace Unlearning for Private Federated Recommender Systems. IEEE Transactions on Dependable and Secure Computing early access (2026), 1–13.DOI
- [25]Huanghuang Liang, Weihan Wu, Zheng Gong, Zhili He, Yili Gong, Jiawei Jiang, and Dazhao Cheng. 2026. Federated Unlearning via Representation Misdirection with Adaptive Anchor Generation. IEEE Transactions on Dependable and Secure Computing early access (2026), 1–14.DOI
- [26]Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017), 6382–6393.arxiv.org/abs/1706.02275
- [28]Alex Oesterling, Jiaqi Ma, Flavio Calmon, and Himabindu Lakkaraju. 2024. Fair machine unlearning: Data removal while mitigating disparities. In International Conference on Artificial Intelligence and Statistics. PMLR, Valencia, Spain, 3736–3744.arxiv.org/abs/2307.14754
- [29]Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. 2021. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems 2021 Datasets and Benchmarks Track. Curran Associates, Inc., Virtual Site.arxiv.org/abs/2006.07869
- [30]Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1–51.
- [31]Richard Ruhdörfer, Kalesha Bullard, Adam Gleave, Edward Grinfenstette, Tim Rõcktäsch el, and Mikayel Samvelyan. 2024. The Overlooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design. arXiv:2406.17949 [cs.LR]arxiv.org/abs/2406.17949
- [32]Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2186–2188.arxiv.org/abs/1902.04043
- [33]Lloyd S. Shapley. 1952. A Value for n-Person Games. Paper P-295. RAND Corporation, Santa Monica, CA.
- [34]Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018. Value-decomposition networks for cooperative multi-agent learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Stockholm, Sweden, 2085–2087.arxiv.org/abs/1706.05296
- [35]Weiqi Wang, Zhiyi Tian, Chenhan Zhang, and Shui Yu. 2024. Machine unlearning: A comprehensive survey. arXiv:2405.07406 [cs.CR]arxiv.org/abs/2405.07406
- [36]Weihan Wu, Yuli Gong, Jiawei Jiang, Chuang Hu, Xiaobo Zhou, and Dazhao Cheng. 2025. Defending against attribute inference attacks in post-training of recommendation systems via unlearning. In 2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, Hong Kong SAR, 2656–2669.
- [37]Weihan Wu, Zhili He, Huanghuang Liang, Yili Gong, Jiawei Jiang, Chuang Hu, and Dazhao Cheng. 2026. REMISFVU: Vertical Federated Unlearning via Representation Misdirection for Intermediate Output Feature. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. AAAI Press, Singapore, 26983–26991.
- [38]Weihan Wu, Jiawei Jiang, and Chuang Hu. 2025. Aegis: Post-training attribute unlearning in federated recommender systems against attribute inference attacks. In Proceedings of the ACM Web Conference 2025. ACM New York, NY, USA, Sydney, Australia, 3783–3793.DOI
- [39]Weihan Wu, Huanghuang Liang, Tianyu Tu, Jiawei Jiang, Chuang Hu, and Dazhao Cheng. 2025. Mimir: Data-free federated unlearning through client-specific prompt generation for personalized models. IEEE Transactions on Mobile Computing 24, 10 (2025), 10537–10556.DOI
- [40]Wenhan Wu, Huanghuang Liang, Jingling Yuan, Jiawei Jiang, Kanyie Ye Wang, Chuang Hu, Xiaobo Zhou, and Dazhao Cheng. 2025. Zero-shot Federated Un-learning via Transforming from Data-Dependent to Personalized Model-Centric.. In IJCAI. IJCAI Press, Montreal, Canada, 6588–6596.
- [41]Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. 2024. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence 8, 3 (2024), 2150–2168.DOI
- [42]Minghan Yang, Dong Xu, Qiuwei Cui, Zaiwen Wen, and Pengxiang Xu. 2022. An efficient Fisher matrix approximation method for large-scale neural network optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2022), 5391–5403.DOI
- [43]Dazhong Ye, Tianqiang Zhu, Congcong Zhu, Derui Wang, Kun Gao, Zewei Shi, Sheng Shen, Wanlei Zhou, and Minhui Xue. 2025. Reinforcement unlearning. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS). Internet Society, San Diego, CA, USA.arxiv.org/abs/2312.15910
- [44]Lei You, Yijun Bian, and Lele Cao. 2026. Joint distribution-informed shapley values for sparse counterfactual explanations. In International Conference on Learning Representations, Vol. 2026. OpenReview.net, Seoul, South Korea, 100183–100205.
- [45]Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35 (2022), 24611–24624.arxiv.org/abs/2103.01955
- [46]Razieh Nokkbeh Zaeem and K Suzanne Barber. 2020. The effect of the GDPR on privacy policies: Recent progress and future promise. ACM Transactions on Management Information Systems (TMIS) 12, 1 (2020), 1–20.
- [47]Rui Zuo, Qinwei Huang, Mingyang Li, Zhenhang Zhang, Simon Khan, and Qinru Qiu. 2023. MUTE: Return-Preserving Communication Unlearning for Efficient Multi-Agent Coordination. arXiv:2607.03473 [cs.MA]


