betaFeedback

WorksComputer Science

When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning (MARL) supports coordinated decision-making in distributed Web services, multi-robot control, and collaborative games through learned coordination conventions. When deployment requirements change, a previously useful convention may become obsolete or unsafe. For example, revised cache-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery. Teams must therefore remove the specified convention while preserving useful cooperation. This is challenging because the same relational pattern can be reproduced by different agents across different histories. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement unlearning and trajectory-deletion methods do not directly target role-invariant conventions. This paper formalizes this problem as semantic coordination-strategy unlearning and proposes CARVE.CARVE uses causal coordination attribution to identify influential interactions and retain-aware subspace updates to limit interference with unrelated behaviors. Counterfactual re-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered. Accepted updates are merged into decentralized actors for direct deployment. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention-completion and worst-case elicitation rates by 91.4% and 83.3%, respectively.The median utility loss is 5.9%, and measured inference latency matches that of the original policies.

Introduction

Cooperative multi-agent reinforcement learning (MARL) enables agents to pursue shared objectives in distributed Web services, multi-robot systems, and collaborative games. Its effectiveness often depends on learned coordination conventions: distributed edge caches coordinate content placement and retrieve objects from neighboring caches to reduce content-delivery latency 1, allied units focus fire in StarCraft micromanagement [11, 32], partners divide cooking and delivery roles in Overcooked [4, 31], and robots negotiate passage through narrow corridors [2, 29]. These conventions simplify joint decision-making and can contribute substantially to policy return. Once formed, however, they also become behavioral patterns that deployed systems may repeatedly reproduce. Removing an obsolete or unsafe coordination pattern while preserving service efficiency and effectiveness is therefore an important problem for reliable multi-agent deployment.

In real-world settings, changing deployment requirements may render some coordination conventions no longer permissible. For example, in a cooperative Web caching service, controllers may learn a division of roles in which one cache leaves an object uncached locally while another retains it and serves subsequent peer fetches. This convention can improve storage utilization and reduce origin requests. If updated service rules prohibit sharing a designated content class across an administrative boundary, the team must abandon conventions that depend on the revoked sharing relation while continuing to serve requests through local caches, permitted peers, or the origin. Such a request targets a specific mode of cooperation, not the entire task and not one particular agent. Removing the target convention therefore requires targeted intervention: the system should abandon the specified convention while retaining the coordination skills needed to make progress in other ways. We call this problem semantic coordination-strategy unlearning.

A natural response is to prohibit the target convention at execution time to satisfy the semantic unlearning objective. In the caching example, a runtime rule can reject disallowed peer fetches, but learned controllers may continue to omit local copies, increasing origin traffic or delivery latency when those fetches are rejected. Our passage experiment in the Vectorized Multi-Agent Simulator (VMAS), a two-dimensional physics simulator for multi-agent robotic learning [2], illustrates this tradeoff between convention suppression and task performance in a controlled coordination setting. The policy is trained with Multi-Agent Proximal Policy Optimization (MAPPO), which uses centralized value estimation during training while allowing agents to choose actions in a decentralized manner at execution time [45]. In this task, the policy learns a bottleneck convention: one agent yields or blocks while another agent passes. Starting from a restored checkpoint, we apply an automaton-based runtime shield that intercepts actions completing the forbidden convention [18, 19]. Although the shield changes the executed action, the original actor network continues to propose the same action pattern. The agents are repeatedly intervened on near the passage and fail to make efficient progress. As shown in Table 1, the shield reduces WER from 0.755 to 0.341, where WER = max⁡ν∈VPr⁡π,ν(τ⊨φf)\max_{\nu\in V} \Pr_{\pi,\nu}(\tau\models\varphi_f) is the worst-case elicitation, computed over held-out or adversarial contexts ν\nu. Here, τ\tau denotes a trajectory, φf\varphi_f is a relational temporal formula specifying the forbidden convention, and τ⊨φf\tau\models\varphi_f means that the trajectory realizes an accepting instance of that convention, such as one agent blocking or yielding followed by another agent passing within a bounded time window. Lower WER means the forbidden convention is harder to recover under role permutations, starts, or context changes. However, the task return drops from 47.5 to 29.6, and normalized task utility falls to 0.623. Return is the episodic task reward measuring task performance, while utility normalizes how much of the original performance is retained. Thus, an apparent improvement in a convention-suppression metric can hide a severe loss of performance. The real challenge is not only to block the old convention, but to change the learned coordination behavior while preserving a viable way to complete the task.

PolicyWER↓\downarrowReturn↑\uparrowUtility↑\uparrow
Original checkpoint0.75547.51.000
Runtime shield0.34129.60.623
CARVE (Ours)0.340158.21.000

Table 1. Comparison on VMAS Passage from the same restored MAPPO checkpoint. Similar WER values can correspond to different task performance.

This difficulty is not limited to runtime constraints, because a coordination convention is not owned by one agent or one trajectory. Existing reinforcement-unlearning and trajectory-deletion methods do not directly address semantic strategy removal [14, 43]. For example, in the caching scenario, the revoked convention concerns the relation between a cache that omits a local copy and a peer that subsequently supplies the object across the restricted boundary. After deleting trajectories involving one cache pair, other nodes may still reproduce the same placement and fetching pattern. Similarly, forgetting focus fire must cover the same tactic across different allied units and enemy targets. We call this property role invariance: the target convention is defined by roles and interactions rather than fixed agent identities. Moreover, the influence of the convention may be distributed across shared policies, value networks, communication modules, and agents’ responses to teammates. These properties show that fixed agents, trajectories, or state-action pairs are too narrow to serve as the basic unit of such unlearning requests.

These observations motivate a behavioral notion of semantic coordination-strategy unlearning: an edited policy should suppress the specified convention across nominal and eliciting contexts while preserving useful cooperation. We think target conventions should be specified after deployment, because shared team rewards entangle their contributions with useful cooperation, making it difficult to withhold positive credit selectively during training. We therefore study request-driven post-training policy editing, with the accepted update merged into decentralized actors for direct deployment. It provides a concrete comparison target beyond the frequency of forbidden events and includes the preservation of original coordination competence as part of the unlearning requirement. The final deployed policy should satisfy these behavioral requirements without relying on a detachable inference-time prohibition layer.

To address this MARL problem, we propose CARVE. The framework first uses counterfactual action interventions to attribute target-convention risk to agent-time interactions. It then selects parameter subspaces according to the behavior that should be retained and restricts updates to these subspaces, reducing interference with other coordination skills. Counterfactual re-coordination helps agents learn alternative ways to complete the task, avoiding the progress failures caused by merely blocking the original action pattern. Finally, adversarial auditing tests whether the target convention can still be recovered. CARVE connects the semantic scope of an unlearning request with the cooperative behaviors that must be preserved, enabling targeted model updates for semantic coordination-strategy unlearning. Our evaluation spans five representative MARL environments and a set of representative baselines to assess the effectiveness and efficiency of policy unlearning. Compared with runtime shielding, CARVE achieves relative improvements of 167.8% in average task utility. Unlearning is 101.92–584.83 times faster than original backbone training. Meanwhile, the two methods achieve similar average reductions in WER, indicating that reporting target-convention recoverability alone is insufficient; evaluation must also consider task utility, return retention. Our main contributions are summarized as follows:

  • We first identify limitations of MARL runtime shield and provide the formalization of role-invariant semantic coordination-strategy unlearning in finite-horizon cooperative Markov games.

  • We propose CARVE, combining causal interaction attribution, counterfactual re-coordination, and adversarial auditing for efficient subspace updates that remove target strategies while preserving performance.

  • We evaluate CARVE across five MARL benchmarks through basic and stronger elicitation tests, efficiency measurements, and ablations. Compared with the strongest parameter-editing baseline, CARVE improves unlearning effectiveness by 16.9%.

Related Work

MARL benchmarks and algorithms. Reinforcement learning [1, 9] is gradually evolving towards multi-agent systems. A large body of cooperative MARL research has developed benchmarks and algorithms for learning coordination. SMAC established StarCraft micromanagement as a cooperative MARL benchmark [32]; SMACv2 introduced procedural generation and stronger closed-loop evaluation pressure [11]. Overcooked-AI studies human-AI coordination [4], and the Overcooked Generalisation Challenge (OGC) expands evaluation to novel partners and layouts [31]. VMAS provides fast vectorized multi-robot simulation [2]. RWARE and LBF are sparse-reward coordination benchmarks popularized through systematic cooperative MARL evaluation [29]. Melting Pot complements these with test-time social generalization [21]. Standard backbones include MAPPO [45], QMIX [30], VDN [34], MADDPG [26], and COMA [12]. Our work builds on these established environments and backbones, but changes the evaluation question from whether agents can acquire coordination to whether a trained system can selectively forget it while retaining task competence.

Reinforcement and machine unlearning. Machine unlearning [3, 28] is commonly motivated by retraining equivalence: the post-unlearning model should resemble a model trained without the removed data or influence [24, 25, 37, 39, 40] due to the requirements of GDPR [46], CCPA [20] or users’ privacy needs [36, 38]. Reinforcement unlearning extends this idea to RL environments [43], while TrajDeleter targets offline RL trajectory removal [14]. These settings do not cover a distributed, role-invariant coordination convention that can be re-enacted by different agents.

Policy repair, safe MARL, and temporal logic. Another line of work studies how to constrain or repair policies after undesirable behavior has been identified. Temporal logic and reward machines provide automata-based structure for non-Markovian tasks [6, 18, 19]. Safe MARL methods such as MACPO and MAPPO-Lagrangian optimize constraints during policy improvement [16]. PolicyCleanse mitigates RL backdoors [17], and MUTE removes low-value communication [47]; both are relevant conditional baselines for triggered or communication-mediated target strategies. These approaches provide closely related risk–return objectives and are essential baselines for behavioral strategy unlearning. CARVE focuses on request-driven editing of a trained team, using interaction attribution, retain-aware sparse updates, and teacher-guided replacement coordination to suppress role-invariant conventions while preserving cooperative task performance.

Preliminaries

We study a finite-horizon cooperative Markov game as follows:

G=(N,S,{Oi},{Ai},P,r,γ,H),(1)\mathcal{G} = (\mathcal{N},\mathcal{S},\{\mathcal{O}_i\},\{\mathcal{A}_i\},P,r,\gamma,H), \tag*{(1)}

where N={1,…,N}\mathcal{N} = \{1,\ldots,N\} is the set of agents, S\mathcal{S} is the global state space, Oi\mathcal{O}_i and Ai\mathcal{A}_i are the observation and action spaces of agent ii, PP is the transition kernel, rr is the shared team reward, γ\gamma is the discount factor, and HH is the horizon. At time tt, agent ii observes otio_t^i, acts with atia_t^i, and conditions on its local history htih_t^i. We use ht=(ht1,…,htN)h_t = (h_t^1,\ldots,h_t^N) and at=(at1,…,atN)a_t = (a_t^1,\ldots,a_t^N) for the joint history and joint action. A decentralized policy factorizes as

πθ(at∣ht)=∏i=1Nπθi(ati∣hti),(2)\pi_\theta(a_t \mid h_t) = \prod_{i=1}^{N} \pi_{\theta_i}(a_t^i \mid h_t^i), \tag*{(2)}

where θ\theta denotes the actor parameters. Parameter sharing is allowed: multiple agents may use the same parameter block while receiving different histories. The return of a policy is

J(π)=Eτ∼π[∑t=0H−1γtr(st,at)],(3)J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right], \tag*{(3)}

where τ\tau denotes a complete trajectory.

Semantic Requests

The object to be forgotten is not a data point, one fixed agent, or one exact trajectory. It is a role-invariant coordination convention, which can be specified as a relational temporal formula φf\varphi_f, where ff denotes “forget”. Atomic predicates are event labels computed from the state and joint action, such as Block(i,door⁡)(i,\operatorname{door}), Pass(j,door⁡)(j,\operatorname{door}), or Attack(i,e)(i,e). The variables i,ji,j range over agents, and ee ranges over task entities, such as a specific enemy in a game. For example, the semantic request “forget the convention where one agent blocks a door so another passes shortly after” can be written as

φf=◊[∃i≠j:Block⁡(i,door⁡)∧◊≤kPass⁡(j,door⁡)].(4)\varphi_f = \Diamond[\exists i \ne j : \operatorname{Block}(i,\operatorname{door}) \land\Diamond_{\leq k}\operatorname{Pass}(j,\operatorname{door})]. \tag*{(4)}

Here ◊\Diamond means “eventually” and ◊≤k\Diamond_{\leq k} means “within kk steps.” ∃i≠j\exists i \ne j means that the convention should be removed no matter which concrete agents instantiate the blocker and passer roles. The same notation also covers combat conventions such as focus-fire:

φf=◊[∃i≠j,e:Attack⁡(i,e)∧◊≤kAttack⁡(j,e)],(5)\varphi_f = \Diamond[\exists i \ne j,e : \operatorname{Attack}(i,e) \land\Diamond_{\leq k}\operatorname{Attack}(j,e)], \tag*{(5)}

which asks the policy to forget the role-invariant tactic where two allies successively attack the same enemy ee. For monitoring, we compile φf\varphi_f into a finite automaton

Aφ=(Q,q0,δ,F).(6)\mathcal{A}_{\varphi} = (Q,q_0,\delta,F). \tag*{(6)}

QQ is the set of monitor states, q0q_0 is the initial state, δ\delta updates the monitor after seeing the current event labels, and FF is the set of accepting states indicating that the forbidden convention has occurred. The product state

sˉt=(st,qt)(7)\bar{s}_t = (s_t,q_t) \tag*{(7)}

combines the environment state sts_t with the current monitor state qtq_t. Thus the monitor turns a history-dependent semantic request into a Markovian signal that can be used by critics, attribution, and evaluation.

Unlearning Goal

Let c∈Cc \in\mathcal{C} denote an evaluation context, such as a layout, initial-state distribution, teammate population, or adversarial partner. The semantic risk of a policy is

Rφ(π;c)=Pr⁡τ∼(Gc,π)[τ⊨φf],Rφrob(π)=sup⁡c∈CRφ(π;c).(8)\mathcal{R}_{\varphi}(\pi;c)=\Pr_{\tau\sim(\mathcal{G}_{c},\pi)}[\tau\models\varphi_{f}],\qquad\mathcal{R}_{\varphi}^{\mathrm{rob}}(\pi)=\sup_{c\in\mathcal{C}}\mathcal{R}_{\varphi}(\pi;c). \tag*{(8)}

The first term is the probability that the policy executes the forbidden convention in context cc. The second is the worst-case risk over the evaluation contexts. Given an original policy π0\pi_{0}, an unlearning algorithm returns a new policy π−=U(π0,φf)\pi^{-}=\mathcal{U}(\pi_{0},\varphi_{f}). We want π−\pi^{-} to satisfy two behavioral requirements:

Rφrob(π−)≤ϵf,Jr(π−)≥Jr(π0)−δJ.(9)\mathcal{R}_{\varphi}^{\mathrm{rob}}(\pi^{-})\leq\epsilon_{f},\qquad J_{r}(\pi^{-})\geq J_{r}(\pi_{0})-\delta_{J}. \tag*{(9)}

The first condition asks for low residual semantic risk under nominal and adversarial evaluation contexts. The second preserves retained task performance; JrJ_{r} is return measured on target-unrelated or retained contexts, and δJ\delta_{J} is the allowed loss. We do not require a separately trained reference policy. Instead, the evaluation directly measures whether the forbidden convention occurs or can be elicited again, using semantic monitors under nominal rollouts, role permutations, held-out contexts, and adversarial partners. This distinction is important because a runtime shield may block the final forbidden action while leaving the underlying coordination preference unchanged. The request is intentionally behavioral in approximate unlearning [22, 35, 41], which does not claim that every parameter-level trace of the original strategy disappears. Instead, they ask whether the delivered policy acts like one that no longer relies on the forbidden convention, while keeping the rest of its cooperative competence.

System Design: CARVE

The strategy unlearning MARL framework CARVE takes as input the original policy π0\pi_{0}, its parameters θ0\theta_{0}, rollout logs or simulator access, the semantic request φf\varphi_{f}, and optional retain specifications of behaviors or task performance to retain. It outputs a merged policy π−\pi^{-} whose actor parameters are changed. The automaton is used during unlearning and auditing, but not as an inference-time shield. Figure 1 summarizes the four stages.

Figure 1. Overview of CARVE. The framework takes an original policy, rollout data or simulator access, and a semantic request φf\varphi_f, and returns an edited policy π−\pi^-. CARVE first attributes forbidden-convention risk to agent-time interactions, then selects retain-aware sparse edit directions, constructs replacement coordination through a counterfactual teacher, and finally audits the edited policy before merging the accepted parameter update.

Causal Coordination Attribution

The first stage asks a simple question: which agents, at which times, make the forbidden convention more likely? We estimate two centralized critics on the product state sˉt=(st,qt)\bar{s}_{t}=(s_{t},q_{t}):

QJ(sˉt,at)=E[∑k=tH−1γk−trk∣sˉt,at],Qφ(sˉt,at)=Pr⁡(qH∈F∣sˉt,at).(10)Q^{J}(\bar{s}_{t},a_{t})=\mathbb{E}\left[\sum_{k=t}^{H-1}\gamma^{k-t}r_{k}\mid\bar{s}_{t},a_{t}\right],\qquad Q^{\varphi}(\bar{s}_{t},a_{t})=\Pr(q_{H}\in F\mid\bar{s}_{t},a_{t}). \tag*{(10)}

QJQ^{J} estimates future task return, and QφQ^{\varphi} estimates the probability that the monitor eventually accepts. Because qtq_{t} records formula progress, QφQ^{\varphi} can assign credit to early decisions that only become visibly forbidden later.

To measure an agent’s contribution, we compare the observed joint action with counterfactual actions sampled from a task-plausible baseline bb. For a coalition S⊆NS\subseteq\mathcal{N},

vt(S)=EaS,t′∼b[Qφ(st,aS,t,aS,t′)].(11)v_{t}(S)=\mathbb{E}_{a_{S,t}^{\prime}\sim b}\left[Q^{\varphi}(s_{t},a_{S,t},a_{S,t}^{\prime})\right]. \tag*{(11)}

where aS,ta_{S,t} keeps the observed actions of agents in SS, and aS,t′a_{S,t}^{\prime} replaces the other agents’ current actions. Specifically, bb samples alternative actions from the original policy’s high-probability actions at the current local histories, excluding the observed actions and renormalizing their probabilities. A Shapley-style attribution is then

χi,t=ES∼ωi[vt(S∪{i})−vt(S)],(12)\chi_{i,t}=\mathbb{E}_{S\sim\omega_{i}}\left[v_{t}(S\cup\{i\})-v_{t}(S)\right], \tag*{(12)}

where ωi\omega_{i} is the standard Shapley distribution [33, 44] over coalitions not containing ii. Thus, χi,t>0\chi_{i,t}>0 means that restoring agent ii’s original action increases semantic risk on average. To keep pairwise coordination visible [15], we also estimate

χi,j,t(2)=ES∼ωij[vt(S∪{i,j})−vt(S∪{i})−vt(S∪{j})+vt(S)].(13)\chi_{i,j,t}^{(2)}=\mathbb{E}_{S\sim\omega_{ij}}\left[v_{t}(S\cup\{i,j\})-v_{t}(S\cup\{i\})-v_{t}(S\cup\{j\})+v_{t}(S)\right]. \tag*{(13)}

This second score is positive when two agents jointly create more risk than their separate effects explain. In practice, we can approximate these expectations with sampled coalitions rather than enumerating all subsets.

Retain-Aware Strategy Subspace

The second stage turns attribution into parameter directions. Let

ψi,t=∇θlog⁡πθi(ait∣hit)(14)\psi_{i,t}=\nabla_{\theta}\log\pi_{\theta_{i}}(a_{i}^{t}\mid h_{i}^{t}) \tag*{(14)}

be the policy score of agent ii’s decision. We convert the attribution scores into one nonnegative weight

wi,t=[χi,t]++η∑j≠i[χi,j,t(2)]+,(15)w_{i,t}=[\chi_{i,t}]_{+}+\eta\sum_{j\ne i}[\chi_{i,j,t}^{(2)}]_{+}, \tag*{(15)}

where [x]+=max⁡(x,0)[x]_{+}=\max(x,0) and η\eta controls how much pairwise coordination affects the weight. The forbidden-strategy direction for trajectory τ\tau is

gf(τ)=∑i,twi,tψi,t.(16)g_{f}(\tau)=\sum_{i,t}w_{i,t}\psi_{i,t}. \tag*{(16)}

This vector points toward parameters that support decisions causally involved in φf\varphi_{f}.

Aggregating over high-risk trajectories gives

Gf=E[gf(τ)gf(τ)T],(17)G_{f}=\mathbb{E}[g_{f}(\tau)g_{f}(\tau)^{\mathsf{T}}], \tag*{(17)}

the forbidden-strategy sensitivity matrix.

We also estimate a retained Fisher matrix [42] FrF_{r} from trajectories and decision points that do not trigger the target convention and should remain stable. Its empirical estimate is computed as

F^r=1M∑(i,t)∈Irψi,tψi,tT,(18)\widehat{F}_{r}=\frac{1}{M}\sum_{(i,t)\in\mathcal{I}_{r}}\psi_{i,t}\psi_{i,t}^{\mathsf{T}}, \tag*{(18)}

where Ir\mathcal{I}_{r} is the set of retained decision points, M=∣Ir∣M=|\mathcal{I}_{r}|, and all policy score vectors are evaluated at the original parameters θ0\theta_{0}. Intuitively, a uTGfuu^{\mathsf{T}}G_{f}u is large if direction uu changes the forbidden strategy, while uTFruu^{\mathsf{T}}F_{r}u is large if uu disturbs retained behavior.

CARVE chooses edit directions by solving

Gfuk=λk(Fr+λ0I)uk.(19)G_{f}u_{k}=\lambda_{k}(F_{r}+\lambda_{0}I)u_{k}. \tag*{(19)}

Here uku_{k} is an edit direction, λk\lambda_{k} is its strategy-to-retain ratio, and λ0>0\lambda_{0}>0 stabilizes the retain metric. Let B=[u1,…,uK]B=[u_{1},\ldots,u_{K}] contain the top-KK candidate directions. To obtain sparse edits, we select a fixed coordinate mask mf∈{0,1}Pm_{f}\in\{0,1\}^{P} containing the top ⌈fP⌉\lceil fP\rceil coordinates under the retain-aware semantic sensitivity score, and form Bf=diag⁡(mf)BB_f = \operatorname{diag}(m_f)B, which is a sparsification heuristic. Subsequent optimization is restricted to

θα=θ0+Bfα,α∈RK.(20)\theta_\alpha= \theta_0 + B_f\alpha,\qquad\alpha\in\mathbb{R}^{K}. \tag*{(20)}

Only α\alpha is optimized; the masked basis remains fixed.

Counterfactual Re-Coordination

Removing a convention can leave a coordination gap: agents may stop using the forbidden strategy but fail to coordinate on an alternative. This stage therefore builds a counterfactual teacher at product states with high QφQ^\varphi. Here, counterfactual refers to alternative joint actions evaluated at the same decision context to construct replacement coordination. For each such state, the teacher favors joint actions that keep task value high and semantic risk low:

μ~t(a∣sˉt)=1Ztπ0(a∣ht)exp⁡(βQJ(sˉt,a)−λQφ(sˉt,a)).(21)\tilde{\mu}_t(\mathbf{a}\mid\bar{s}_t)=\frac{1}{Z_t}\pi_0(\mathbf{a}\mid h_t)\exp\left(\beta Q^J(\bar{s}_t,\mathbf{a})-\lambda Q^\varphi(\bar{s}_t,\mathbf{a})\right). \tag*{(21)}

ZtZ_t normalizes the distribution over joint actions. The coefficient β≥0\beta\ge0 rewards useful alternatives, and λ≥0\lambda\ge0 penalizes actions likely to complete the forgotten convention. This is a compact way to express a KL-near replacement teacher: stay close to π0\pi_0, but tilt probability mass away from the target strategy and toward valuable alternatives. Because the teacher is centralized, we distill its agent-wise marginals μ~i,t\tilde{\mu}_{i,t} into the decentralized actors:

Lcf(α)=Eτ~∑i,twi,tDKL(μ~i,t(⋅∣sˉt)∥πθα,i(⋅∣hti)).(22)\mathcal{L}_{\mathrm{cf}}(\alpha)=\mathbb{E}_{\tilde{\tau}}\sum_{i,t}w_{i,t}D_{\mathrm{KL}}\left(\tilde{\mu}_{i,t}(\cdot\mid\bar{s}_t)\mathbin{\|}\pi_{\theta_\alpha,i}(\cdot\mid h_t^i)\right). \tag*{(22)}

τ~\tilde{\tau} denotes teacher-generated or teacher-relabeled trajectories, and μ~i,t\tilde{\mu}_{i,t} is the teacher’s marginal distribution for agent ii. The same attribution weight wi,tw_{i,t} focuses distillation on decisions that matter for the target convention, rather than forcing the entire policy to imitate the teacher everywhere.

Robust Auditing and Policy Merge

The final stage chooses α\alpha and repeatedly audits the candidate policy πθα\pi_{\theta_\alpha}. The training objective contains only two terms:

min⁡αLcf(α)+λrLretain(α),(23)\min_{\alpha}\quad\mathcal{L}_{\mathrm{cf}}(\alpha)+\lambda_r\mathcal{L}_{\mathrm{retain}}(\alpha), \tag*{(23)}

where

Lretain(α)=Eτ∼Dr∑i,tDKL(π0,i(⋅∣hti)∥πθα,i(⋅∣hti))(24)\mathcal{L}_{\mathrm{retain}}(\alpha)=\mathbb{E}_{\tau\sim\mathcal{D}_r}\sum_{i,t}D_{\mathrm{KL}}\left(\pi_{0,i}(\cdot\mid h_t^i)\mathbin{\|}\pi_{\theta_\alpha,i}(\cdot\mid h_t^i)\right) \tag*{(24)}

keeps target-unrelated behavior close to the original policy. The semantic monitor is used for auditing:

R^adv(π)=max⁡c∈CadvR^φ(π;c)(25)\widehat{\mathcal{R}}_{\mathrm{adv}}(\pi)=\max_{c\in\mathcal{C}_{\mathrm{adv}}}\widehat{\mathcal{R}}_{\varphi}(\pi;c) \tag*{(25)}

is the empirical worst-case semantic risk over adversarial contexts. These contexts include held-out partners, starts, layouts, and role permutations selected to elicit φf\varphi_f. If the audit risk is too high, the eliciting trajectories are added to the training set and the update is repeated. We accept the final policy only if it passes the semantic-risk and return checks from the formulation section. The delivered model is obtained by merging Δθ=Bfα\Delta\theta=B_f\alpha into the actor parameters; no monitor or shield is required at deployment.

Experiments

We conduct experiments to answer four questions about CARVE: i) RQ1: whether it removes the requested coordination strategy (§5.2.1, §5.2.3, §5.4)? ii) RQ2: whether it preserves MARL task performance (§5.2.2)? iii) RQ3: whether it is efficient enough to be a post-training edit rather than retraining in disguise (§5.3)? iv) RQ4: which components and hyperparameters control the erasure–retention frontier (§5.5, §C)?

Experimental Setup

5.1.1 Testbed. Experiments are implemented in Python 3.11.8 with PyTorch 2.6.0+cu124 and TorchRL 0.7.2. They run on an Ubuntu 18.04.6 LTS server with two 26-core Intel Xeon Platinum 8269CY CPUs, 256 GB RAM, and four NVIDIA A100-40GB GPUs. Unless otherwise stated, backbone training uses five random seeds.

5.1.2 Evaluation Benchmarks and Unlearning Requests. Our evaluation cross five MARL environments pairs established cooperative MARL testbeds with role-invariant unlearning requests. For SMACv2 [11], the target is a relational combat tactic; for Overcooked-AI/OGC [4, 31], it is a bottleneck or handoff convention; for VMAS [2], it is a continuous-control yielding or leader-follower rule; for RWARE [29], it is a warehouse aisle convention; and for LBF [29], it is a sparse-reward partner-selection convention, which is used as an additional sparse-reward stress test. Details of the scenarios, role-invariant unlearning requests, retain specifications, backbone algorithms, and hyperparameters, are provided in Appendix § 11.

5.1.3 Baselines. The baselines are chosen to answer distinct alternative explanations: whether runtime shielding merely hides the behavior, whether constrained MARL is enough, whether generic parameter unlearning is enough, and whether trajectory removal is enough. We compare against the methods as shown in Table 2.

MethodRole in evaluation
Orig. (trained backbone)Pre-unlearning policy and target-strategy prevalence.
Shield (runtime automaton)Blocks forbidden completions without model updates.
CARVE (Ours)Sparse retain-aware semantic strategy unlearning.
Full-FTFull-parameter semantic fine-tuning without sparse subspace using negative strategy gradient.
RP (semantic reward penalty)Fixed semantic-cost penalty during post-training fine-tuning.
Lag. (Lagrangian MARL) [16]Adaptive Lagrangian penalty for semantic-cost suppression.
MACPO [16]Conservative constrained policy optimization baseline.
Fisher [13]Fisher-ratio parameter dampening without causal attribution.
TrajDel [14]Tests trajectory removal for role-invariant strategies.

Table 2. Baseline suite used in the main result tables.

5.1.4 Evaluation Metrics. We assess convention removal using the nominal semantic satisfaction rate SSR⁡(π)=Pr⁡τ∼π(τ⊨φf)\operatorname{SSR}(\pi) = \Pr_{\tau\sim\pi}(\tau\models\varphi_f) and the worst-case elicitation rate WER⁡(π)=max⁡ν∈VPr⁡π,ν(τ⊨φf)\operatorname{WER}(\pi) = \max_{\nu\in\mathcal{V}} \Pr_{\pi,\nu}(\tau\models\varphi_f). Here, V\mathcal{V} denotes the red-team elicitation set, including held-out teammates or partners when applicable, held-out starts, layouts, procedural context seeds, and role permutations. Both metrics measure the occurrence of the forbidden convention, with lower values indicating more effective suppression. We assess task retention using episodic return (Return) and normalized task-utility retention

Util.⁡(π)=clip⁡[0,1](1−∣J0−J(π)∣Db),\operatorname{Util.}(\pi)=\operatorname{clip}_{[0,1]}\left(1-\frac{|J_0-J(\pi)|}{D_b}\right),

where J0J_0 is the original policy’s mean return recorded in the same run, [x]+=max⁡(x,0)[x]_+ = \max(x,0). The evaluation scripts use Db=max⁡(∣J0∣,1)D_b=\max(|J_0|,1) for SMACv2 and VMAS, which accommodates negative and near-zero original returns, and Db=J0D_b=J_0 for Overcooked-AI, RWARE, and LBF, whose reported original returns are positive. Util. is normalized relative to the restored original policy, with 1 indicating full task-performance retention. We measure efficiency using unlearning speedup Tbackbone/TunlearnT_{\mathrm{backbone}}/T_{\mathrm{unlearn}}, inference latency (Infer.), and the fraction of actor parameters changed relative to the original checkpoint (ParamFrac).

Overall Performance

5.2.1 Unlearning Performance. Table 3 shows that CARVE reduces mean SSR and WER by 91.4% and 83.3%, respectively, relative to the original policies. No target-strategy completions are observed under either protocol in Overcooked-AI, RWARE, and LBF. In the two benchmarks with residual risk, CARVE matches the lowest reported WER in SMACv2 and achieves the lowest WER in VMAS. Relative to MACPO, the strongest parameter-editing baseline by mean WER, the largest within-benchmark SSR and WER reductions reach 29.1% and 39.8%, respectively. Compared with Shield, CARVE matches WER in four benchmarks and reduces it by 53.8% in VMAS, while requiring no inference-time monitor. For Full-FT, RP, and Lag., the largest within-benchmark WER reductions range from 33.3% to 56.0%. These gains are consistent with targeting causally attributed coordination decisions, constructing useful alternatives, and incorporating eliciting trajectories during auditing, rather than relying solely on broad updates or semantic penalties. Fisher leaves greater residual semantic risk, while TrajDel still permits target-strategy completions in every benchmark. This shows that parameter dampening may miss the interactions underlying a convention, while deleting selected trajectories does not prevent the strategy’s reproduction through other histories. SMACv2 remains challenging, with a residual WER of 0.500. This matches the lowest observed WER among the evaluated methods. At this risk level, however, CARVE retains a utility of 0.941, compared with 0.749 for Shield and 0.544 for MACPO. Thus, although substantial residual risk remains, CARVE preserves the most task utility among methods achieving the same worst-case suppression. These results highlight the importance of evaluating convention removal jointly with retained coordination competence.

MethodMetricSMACv2 ZeroOC-AI Asym.VMAS FlockingRWARE TinyLBF 8x8-2p
Orig.SSR↓0.4691.0000.3570.8750.792
WER↓0.8751.0000.7930.8750.792
ShieldSSR↓0.1250.0000.1800.0000.000
WER↓0.5000.0000.4800.0000.000
CARVESSR↓0.1440.0000.1570.0000.000
WER↓0.5000.0000.2220.0000.000
Full-FTSSR↓0.3120.0000.2920.0000.000
WER↓0.8750.0000.3610.0000.000
RPSSR↓0.2500.0000.1870.0000.000
WER↓0.7500.0000.2740.0000.000
Lag.SSR↓0.3440.0000.2190.0000.000
WER↓0.6250.0000.5050.0000.000
MACPOSSR↓0.2030.0000.1640.0000.000
WER↓0.5000.0000.3690.0000.000
FisherSSR↓0.4220.0000.3990.0000.042
WER↓0.7500.0000.7430.0000.042
TrajDelSSR↓0.4031.0000.2920.7500.792
WER↓0.6251.0000.5860.7500.792

Table 3. Unlearning performance across five MARL benchmarks. OC-AI denotes Overcooked-AI.

MARL Retain Performance

Table 4 shows CARVE reduces a median utility of only 5.9% relative to the original policies, with four of the five benchmarks losing at most 6.2%. The retention comparison is particularly informative in Overcooked-AI, RWARE, and LBF: Shield, Full-FT, RP, Lag., and MACPO achieve zero observed SSR/WER alongside zero task return. Their suppression is therefore accompanied by task failure, whereas CARVE maintains positive returns in all three settings. In SMACv2, CARVE matches Shield and MACPO in WER while improving return by up to 80.6%. This pattern is consistent with the intended combination of retain-aware subspace selection, KL retention, and teacher-guided replacement coordination. TrajDel preserves slightly higher mean utility, but the target convention remains elicitable on every benchmark. Relative to TrajDel, CARVE gives up 2.9% in mean utility while reducing mean WER by 80.8%. Fisher preserves utility well in SMACv2 and VMAS but does not consistently retain performance across the remaining tasks. Overcooked-AI exhibits the largest utility loss with a 45.8% reduction. Productive workflows can depend strongly on the convention being removed, making effective replacement coordination difficult. Nevertheless, among methods attaining zero observed SSR and WER, CARVE achieves the highest retained utility; all corresponding baselines yield zero task return. These comparisons show that CARVE offers the strongest observed retention at the lowest reported semantic risk in this setting. The remaining utility gap may reflect both dependence on the removed convention and limitations of replacement coordination.

MethodMetricSMACv2 ZergOC-AI Asym.VMAS FlockingRWARE TinyLBF 8x8-2p
Orig.Return8.407216.00.4831.0000.667
Orig.Util.1.0001.0001.0001.0001.000
ShieldReturn4.0250.00.4810.0000.000
ShieldUtil.0.7490.0000.9020.0000.000
CARVEReturn7.268109.00.5571.2500.625
CARVEUtil.0.9410.5421.0001.0000.938
Full-FTReturn5.1110.00.4010.0000.000
Full-FTUtil.0.8020.0000.9180.0000.000
RPReturn4.1630.00.4260.0000.000
RPUtil.0.6730.0000.9430.0000.000
Lag.Return4.7020.00.4050.0000.000
Lag.Util.0.7470.0000.9220.0000.000
MACPOReturn4.5240.00.4600.0000.000
MACPOUtil.0.5440.0000.9760.0000.000
FisherReturn7.2110.00.5260.0000.021
FisherUtil.0.9030.0001.0000.0000.031
TrajDelReturn4.783207.00.4000.8750.667
TrajDelUtil.0.8030.9580.9160.8751.000

Table 4. MARL retain performance across five benchmarks. OC-AI denotes Overcooked-AI.

Robustness with Stronger Elicitation under Different Unlearning Requests

We further test whether the unlearning performance persist under stronger elicitation across the GiveWay, Passage, and Flocking requests in VMAS dataset, using their respective MADDPG [26], MAPPO [45], and IPPO [10] checkpoints. We use two red-team attacks. The role-permutation attack swaps agent identities before policy evaluation and maps the resulting actions back before stepping. The role-invariant semantic request remains unchanged, so this attack tests whether reassignment of the same roles to different agent indices can recover the target convention. The held-out-context attack evaluates the same edited checkpoint on unseen VMAS context seeds {4001, 4002, 4003}, which vary initial states and procedural task contexts. No additional parameter updates are performed during these tests; CARVE executes its merged actors, and the semantic monitor scores the resulting trajectories.

Table 5 reports standard-context WER (Norm. WER), WER under role permutation (Role WER), and the maximum WER across held-out-context seeds (Ctx. WER). Under role permutation, CARVE achieves the lowest reported Role WER for all three unlearning requests. Relative to the original policies, the reduction across requests has a median of 53.0% and reaches 63.6%. Across parameter-editing baselines, the median Role WER reductions range from 9.3% to 58.9%, with the largest reduction against MACPO reaching 46.9%. Compared with Shield, the median reduction is 12.0%, and the largest reaches 42.1%. These comparisons show that CARVE’s advantage persists when the same roles are reassigned to different agent indices. Under held-out contexts, CARVE reduces Ctx. WER relative to the original policies by a median of 54.6%, with the largest reduction reaching 64.9%. Its median Ctx. WER reductions against parameter-editing baselines range from 5.2% to 59.0%, including 22.2% against MACPO. Relative to Shield, the median reduction is 10.2%, and the largest reaches 48.2%. CARVE achieves the lowest Ctx. WER in GiveWay and Passage, while RP has slightly lower Ctx. WER in Flocking. When risk is summarized by the maximum over the nominal, role-permutation, and held-out-context protocols for each method and request, CARVE nevertheless achieves the lowest value for all three requests. This supports its comparative robustness across the tested role assignments and unseen contexts, while residual strategy completions remain possible.

TaskMethodSSR↓Return↑Util.↑Norm. WERRole WERCtx. WER
GiveWay (MADDPG)Orig.0.5023.7081.0000.5020.5090.517
GiveWay (MADDPG)Shield0.3263.1250.9820.3260.3990.401
GiveWay (MADDPG)CARVE0.3133.6790.9980.3130.3510.360
GiveWay (MADDPG)Full-FT0.3993.1720.9900.3990.4070.432
GiveWay (MADDPG)RP0.4043.2760.9910.4040.4160.423
GiveWay (MADDPG)Lag.0.4253.2590.9870.4250.3980.431
GiveWay (MADDPG)MACPO0.4053.1700.9900.4050.3740.395
GiveWay (MADDPG)Fisher0.4813.1520.9850.4810.4870.491
GiveWay (MADDPG)TrajDel0.5013.7130.8920.5010.5010.506
Passage (MAPPO)Orig.0.33847.51.0000.7550.7470.756
Passage (MAPPO)Shield0.20329.60.6230.3410.3600.374
Passage (MAPPO)CARVE0.20858.21.0000.3400.3510.343
Passage (MAPPO)Full-FT0.248103.11.0000.3650.3980.367
Passage (MAPPO)RP0.217128.11.0000.3730.4060.362
Passage (MAPPO)Lag.0.229133.91.0000.5120.5210.623
Passage (MAPPO)MACPO0.213143.51.0000.4060.3870.441
Passage (MAPPO)Fisher0.441103.70.0000.8210.8530.836
Passage (MAPPO)TrajDel0.33748.51.0000.7540.7450.754
Flocking (IPPO)Orig.0.3570.4831.0000.7930.7970.756
Flocking (IPPO)Shield0.1800.4810.9020.4800.5010.541
Flocking (IPPO)CARVE0.1570.5571.0000.2220.2900.280
Flocking (IPPO)Full-FT0.2920.4010.9180.3610.4430.387
Flocking (IPPO)RP0.1870.4260.9430.2740.3580.271
Flocking (IPPO)Lag.0.2190.4050.9220.5050.6530.533
Flocking (IPPO)MACPO0.1640.4600.9760.3690.5460.435
Flocking (IPPO)Fisher0.3990.5261.0000.7430.8580.813
Flocking (IPPO)TrajDel0.2920.4000.9160.5860.7370.719

Table 5. Stronger unlearning audit under two different attacks on three VMAS forgetting tasks. Bold numbers indicate the largest reported value among the three audit settings.

Unlearning Efficiency

Table 6 shows that CARVE achieves the highest reported unlearning time speedup across all five benchmarks, ranging from 101.92× to 584.83× relative to original backbone training, including loading data, gradient-mask construction, and update time. It is 1.25–2.29 times faster than Full-FT. Its inference latency matches the original policy, while Shield increases latency by 19.2%–31.2%. This overhead arises from additional runtime automaton monitoring and action filtering. Figure 2 reports ParamFrac, the fraction of actor parameters changed relative to the original checkpoint. CARVE modifies 13.1% on average, ranging from 10.5% to 20.0%, compared with 90.6%–100.0% for RP, Lag., MACPO, Full-FT, and TrajDel. Fisher also has a small edit footprint of 12.0% in every benchmark, but records lower unlearning speedups than CARVE. This illustrates that edit locality alone does not determine runtime, which also depends on parameter selection and optimization costs. CARVE optimizes only the coefficients α\alpha in the retain-aware update Δθ=Bfα\Delta\theta= B_f\alpha. The accepted update is merged into the actors, enabling deployment without an inference-time monitor or shield.

Figure 2. Fraction of actor parameters modified by each unlearning method across five MARL benchmarks

BenchmarkMetricOrig.ShieldCARVEFull-FTRPLag.MACPOFisherTrajDel
SMACv2 (Zerg)Speedup(×\times)↑\uparrow––584.83255.06290.30266.87273.08575.38482.07
Infer.(ms)↓\downarrow120.326143.460120.326120.326120.326120.326120.326120.326120.326
Overcooked-AI (Asym.)Speedup(×\times)↑\uparrow––101.9244.7748.6845.5947.2094.3282.37
Infer.(ms)↓\downarrow80.750104.77680.75080.75080.75080.75080.75080.75080.750
VMAS (Flocking)Speedup(×\times)↑\uparrow––454.53363.68391.25326.47357.24416.54423.68
Infer.(ms)↓\downarrow86.718107.85686.71886.71886.71886.71886.71886.71886.718
RWARE (Tiny)Speedup(×\times)↑\uparrow––247.47178.54199.10182.61188.69228.85209.02
Infer.(ms)↓\downarrow100.735122.740100.735100.735100.735100.735100.735100.735100.735
LBF (8x8-2p)Speedup(×\times)↑\uparrow––312.94209.17228.46216.34220.46264.57291.89
Infer.(ms)↓\downarrow70.60592.61770.60570.60570.60570.60570.60570.60570.605

Table 6. Unlearning efficiency across five MARL benchmarks. Speedup is Tbackbone/TunlearnT_{\mathrm{backbone}}/T_{\mathrm{unlearn}}; Infer. is inference latency.

Visualization

We use VMAS Flocking to inspect whether semantic unlearning changes the policy itself. The target convention is leader-follower dominance: agents move cohesively while one agent exerts disproportionate control. Figure 3 compares the original restored actor and CARVE on the same rollout diagnostics. In the monitor row, q0q_0, q1q_1, and q2q_2 denote low-risk, recoverable, and accepted forbidden-convention states. The original actor shows the target signature across all views: clustered trajectories, a semantic-score peak above

Figure 3. Visualization for VMAS Flocking. Rows from top to bottom show the rotated physical trajectory, the semantic score for leader-follower dominance, per-agent action norms, and occupancy of the monitor states q0q_0, q1q_1, q2q_2.

the accepting threshold, uneven action norms, and visible q1/q2q_1/q_2 monitor occupancy. CARVE weakens this signature. Its trajectories are less organized around a single leading path, the semantic score remains below the accepting region in the shown rollout, action magnitudes are more balanced, and monitor mass shifts toward q0q_0. This supports the main claim: CARVE removes the leader-follower convention while preserving coordinated flocking, rather than merely blocking actions at runtime.

Ablation Study

We ablate five components on VMAS GiveWay, Passage, and Flocking using the same checkpoints, semantic requests, retain sets, and audit contexts. The full model uses the complete pipeline in Eqs. (12)–(25). The ablated variants replace the attribution scores in Eqs. (12)–(13) with random sparse weights, replace the retain-aware eigenspace in Eq. (19) with semantic-gradient coordinates, drop Lretain\mathcal{L}_{\mathrm{retain}} from Eq. (23), drop the counterfactual teacher in Eqs. (21)–(22), or optimize the full actor θ\theta instead of the subspace update in Eq. (20). Table 7 reports task averages. Causal attribution increases ΔSSR and ΔWER by 147.1% and 142.0%, respectively, compared with random weights, supporting more effective targeting of the convention. Retain-aware selection improves ΔWER by 27.3% and utility by 10.5%, while the retention loss improves utility by 54.9% with nearly unchanged semantic reductions. These components therefore provide complementary protection for retained cooperation. Counterfactual replacement improves Δ\DeltaWER by 43.1% and mean return by 61.6%, supporting productive alternative coordination. Subspace restriction improves utility by 35.5% while reducing the changed-parameter fraction by 87.8%, indicating that limiting edit scope helps preserve task competence. Together, these ablations show that the components complement one another in balancing effective convention removal with efficient retained cooperation.

VariantΔ\DeltaSSR↑\uparrowΔ\DeltaWER↑\uparrowUtil.↑\uparrowReturn↑\uparrowParamFrac↓\downarrow
Full CARVE0.1730.3920.99954.150.120
w/o attribution0.0700.1620.96351.70.118
w/o retain-aware mask0.1590.3080.90433.30.120
w/o retain loss0.1740.3900.64532.80.120
w/o replacement reg.0.1290.2740.98733.50.120
w/o sparse subspace0.1300.3400.737-103.60.986

Table 7. Ablation study on VMAS GiveWay, Passage, and Flocking. Δ\DeltaSSR and Δ\DeltaWER are reductions relative to the restored original checkpoint.

Conclusion

This paper studied semantic coordination-strategy unlearning, where a trained cooperative policy must abandon a role-invariant convention while retaining useful cooperation. CARVE combines causal coordination attribution, retain-aware subspace updates, counterfactual re-coordination, and adversarial auditing, then merges the accepted update into decentralized actors. Experiments across five MARL benchmarks show that CARVE balances target-convention removal with the preservation of cooperative task performance and no additional inference latency.

References

Algorithm Details

CARVE Pseudocode

As shown in Algorithm 1, CARVE is a post-training unlearning framework. It starts from a restored cooperative policy, uses the automaton monitor to locate decisions that causally support the forbidden convention, restricts the update to retain-aware strategy-sensitive directions, distills replacement coordination, and finally audits the edited actor under stronger elicitation contexts. The automaton and red-team contexts are used during editing and verification; the delivered policy is the edited actor itself and does not require an inference-time shield.

Complexity Analysis

Let MM be the number of collected rollouts, HH the horizon, NN the number of agents, PP the number of actor parameters, CC the number of sampled coalitions per target-positive decision, KK the retained subspace rank, ff the editable actor-parameter fraction, UU the number of projected optimization updates, RR the number of audit contexts, and EE the number of audit episodes per context. Monitor annotation is linear in trajectory length, O(MH)O(MH), excluding simulator cost. Attribution requires O(MHNC)O(MHNC) semantic-critic evaluations in the sampled-coalition implementation, rather than enumerating all 2N2^N coalitions. Building the retain-aware subspace costs O(MHNP)O(MHNP) to accumulate score-gradient statistics plus O(IeigKP)O(I_{\mathrm{eig}}KP) for IeigI_{\mathrm{eig}} iterative generalized-eigen updates with diagonal or low-rank sensitivities are used; a dense P×PP \times P eigensolver would be O(P3)O(P^3) and is not the intended implementation. Selecting the edit mask costs O(P)O(P) to score coordinates plus a top-⌈fP⌉\lceil fP\rceil selection. Projected optimization costs O(UBmP)O(UB_mP) for minibatch actor forward/backward passes, but only fPfP actor coordinates receive nonzero updates. Materializing the masked subspace costs O(KfP)O(KfP). The final audit costs O(REH)O(REH) simulator steps. In practice, the dominant costs are simulator rollouts and actor-gradient passes; memory is O(P+KfP)O(P+KfP) for the actor and masked subspace, plus the rollout buffer.

Evaluation and Hyperparameters Details

Table A1 summarizes the task scenarios, role-invariant semantic unlearning requests, retained behaviors, and backbone algorithms across five cooperative MARL environments. Unless otherwise stated, SMACv2 and VMAS backbone training uses five random seeds, 3.072M environment frames per run, 32 evaluation episodes every 122,880 frames, and checkpointing every 307,200 frames. SMACv2 uses four environment workers and 2,000 collected frames per batch, while VMAS uses 32 workers and 6,144 collected frames per batch. For post-checkpoint CARVE runs, the default setting is 800 update steps, actor edit fraction f=0.12f = 0.12, retain-action weight λr=0.5\lambda_r = 0.5, learning rate 2×10−42 \times10^{-4}, β=1\beta= 1, K=8K = 8, λ0=10−3\lambda_0 = 10^{-3} and λ=8\lambda= 8. For deterministic actors such as MADDPG, evaluation actions are generated by adding independent Gaussian noise ϵ∼N(0,0.152I)\epsilon\sim\mathcal{N}(0, 0.15^2 I) in the atanh latent space of normalized original actions, followed by a tanh transformation and rescaling to the native action range.

BenchmarkScenarioSemantic request φf\varphi_fRetain behaviorBackbone
SMACv2 [11]Zerg 5v5Role-invariant focus-fire: two or more allied units attack the same enemy at the same decision step, independent of ally identities.Episode return, battle win rate, and killed-enemy count.MAPPO [45]
Overcooked-AI/OGC [4, 31]Asymmetric AdvantagesFixed cook–deliver split: one player contributes cooking/potting events while the partner contributes dish, soup, or delivery events.Sparse/shaped soup return and total cooking plus delivery event count.IPPO [10]
RWARE [29]Tiny 2-agent warehouseFixed carrier–yielder aisle convention: the target carrier handles shelf deliveries while the paired robot yields during pickup or carrying.Shelf-delivery return and non-target delivery count.QMIX [30]
LBF [29]8x8-2pFixed-pair collection: the target pair jointly collects sparse-reward food.Food-collection return and non-target pair collections.VDN [34]
VMAS [2]GiveWayYield-then-cross: at the narrow crossing, one agent stays low-motion near the center while another crosses with high motion.Task return, completion rate, and collision avoidance.MADDPG [26]
VMAS [2]PassageSingle-file gate priority: near the passage, one agent queues or yields while another takes priority through the bottleneck.Passage return, completion rate, and collision avoidance.MAPPO [45]
VMAS [2]FlockingLeader-follower dominance: a cohesive flock moves while one agent has a disproportionately large action norm and drives the group.Flocking return, completion rate, and collision avoidance.IPPO [10]

Table A1. Concrete scenarios used in the main experiments. Each row reports one implemented semantic request, the corresponding retained behavior, and the single backbone used for the reported checkpoint.

Parameter Sensitivity

We sweep the hyperparameters that control the erasure–retention frontier on the three VMAS post-checkpoint tasks: GiveWay, Passage, and Flocking. The validation grid is update steps in {200,400,800,1200}\{200, 400, 800, 1200\}, edit fraction in {0.03,0.06,0.12,0.24,0.48}\{0.03, 0.06, 0.12, 0.24, 0.48\}, retain weight in {0.1,0.5,1.0,2.0,4.0}\{0.1, 0.5, 1.0, 2.0, 4.0\}, and learning rate in {10−4,2×10−4,5×10−4}\{10^{-4}, 2 \times10^{-4}, 5 \times10^{-4}\}. Table A2 reports the average semantic reduction, retained utility, task reward, and parameter locality together. Increasing the edit fraction from 0.06 to 0.12 improves Δ\DeltaWER by 16.7%, whereas expanding it to 0.48 yields only a further 1.5% gain with four times as many changed parameters. Increasing the retain weight from 0.5 to 4.0 reduces Δ\DeltaWER by 44.6%, with little additional utility retention. Among the tested learning rates, 2×10−42 \times10^{-4} achieves the strongest semantic reductions. More updates do not consistently improve forgetting: 800 steps increase mean reward by 50.1% over 200 steps with comparable semantic reductions, while 1200 steps weaken forgetting. These results support the default configuration as a practical balance between convention removal, task performance, and edit locality.

SweepValueΔSSR↑\Delta\mathrm{SSR}\uparrowΔWER↑\Delta\mathrm{WER}\uparrowUtil.↑\mathrm{Util.}\uparrowReward↑\mathrm{Reward}\uparrowParamFrac↓\mathrm{ParamFrac}\downarrow
Edit fraction0.030.1220.2530.96649.10.030
Edit fraction0.060.1640.3360.99952.80.060
Edit fraction0.120.1730.3920.99954.20.120
Edit fraction0.240.1760.3980.99954.30.240
Edit fraction0.480.1810.3980.99954.80.480
Retain weight0.10.1870.3950.97752.10.120
Retain weight0.50.1730.3920.99954.20.120
Retain weight1.00.1700.3870.99953.00.120
Retain weight2.00.1640.2780.99952.80.120
Retain weight4.00.1180.2171.00050.90.120
Learning rate1e-40.1340.3380.96552.80.120
Learning rate2e-40.1730.3920.99954.20.120
Learning rate5e-40.1440.3770.99853.30.120
Update steps2000.1770.3991.00036.10.120
Update steps4000.1370.3180.99853.50.120
Update steps8000.1730.3920.99954.20.120
Update steps12000.1300.3350.99756.70.120

Table A2. Parameter sensitivity on VMAS.

Teacher–Actor Gap Experiment

Marginal distillation may discard joint-action correlations encoded by the centralized teacher. Table A3 therefore compares the joint teacher TT with the independently sampled edited actor ASA_S on paired nominal episodes, using one trained checkpoint per VMAS scenario. Both are nominal-context score diagnostics. The observed gap is scenario-dependent. GiveWay shows nearly unchanged semantic scores and return. In Passage, the actor’s SSR and WER point estimates decrease by 0.023 and 0.022, respectively, alongside a higher return point estimate. In Flocking, they increase by 0.019 and 0.039, respectively, alongside a lower return point estimate. Thus, the actor is not uniformly worse than the teacher. The teacher’s local, critic-based KL-regularized objective does not guarantee dominance on episode-level metrics. All paired bootstrap 95% confidence intervals for J(AS)−J(T)J(A_S) - J(T) contain zero, leaving return differences statistically unresolved. This does not establish equivalence, particularly given the wide Passage interval. The comparison quantifies the combined Algorithm 1 CARVE semantic coordination-strategy unlearning.

ScenarioSSR↓\mathrm{SSR}\downarrow TTSSR↓\mathrm{SSR}\downarrow AsA_sWER↓\mathrm{WER}\downarrow TTWER↓\mathrm{WER}\downarrow AsA_sReturn↑\mathrm{Return}\uparrow TTReturn↑\mathrm{Return}\uparrow AsA_sPaired 95% CI of J(As)−J(T)J(A_s)-J(T)
GiveWay0.3120.3130.3150.3133.6783.679[−0.0038,0.0056][-0.0038, 0.0056]
Passage0.2310.2080.3620.340121.1158.2[−8.098,230.861][-8.098, 230.861]
Flocking0.1380.1570.1830.2220.6040.557[−0.286,0.035][-0.286, 0.035]

Table A3. Joint-teacher versus actor performance on three VMAS scenarios. WER is measured in the nominal context. Confidence intervals bootstrap paired evaluation episodes.

Input: Restored policy π0\pi_0 with actor parameters θ0\theta_0; simulator GG; forbidden formula φf\varphi_f; retain data DrD_r; monitor automaton AφfA_{\varphi_f}; edit rank KK; actor edit fraction ff; update budget UU; audit contexts CauditC_{\mathrm{audit}}; minimum semantic reduction δ>0\delta> 0; utility threshold ρ\rho; required consecutive passes MM; audit interval AA.

Output: Accepted edited policy π−\pi^-, or NotAccepted.

  1. Evaluate baseline scores S0=SSR⁡(π0)S_0 = \operatorname{SSR}(\pi_0) and W0=WER⁡(π0)W_0 = \operatorname{WER}(\pi_0) on CauditC_{\mathrm{audit}}.

  2. Roll out π0\pi_0 in training and retain contexts; update product states sˉt=(st,qt)\bar{s}_t = (s_t, q_t) with AφfA_{\varphi_f}.

  3. Mark target-positive and retain decision points from the monitored trajectories.

  4. Fit Qφf(sˉt,at)Q^{\varphi_f}(\bar{s}_t, a_t) for target-convention risk and QJ(sˉt,at)Q^J(\bar{s}_t, a_t) for task value.

  5. for each target-positive decision point do

  6.     Sample coalitions and estimate χi,t,χij,t(2)\chi_{i,t}, \chi_{ij,t}^{(2)} by Eqs. (12) and (13).

  7.     Convert attribution scores into weights wi,tw_{i,t} by Eq. (15).

  8. end for

  9. Compute gf(τ)g_f(\tau) by Eq. (16); estimate GfG_f and FrF_r.

  10. Solve Eq. (19); retain the top-KK candidate directions B=[u1,…,uK]B = [u_1, \ldots, u_K].

  11. Select the top ⌈fP⌉\lceil fP \rceil coordinates by retain-aware semantic sensitivity; form mfm_f and Bf=diag⁡(mf)BB_f = \operatorname{diag}(m_f)B as a heuristic sparsification.

  12. Fix BfB_f and restrict subsequent optimization to θα=θ0+Bfα\theta_\alpha= \theta_0 + B_f\alpha.

  13. Build the centralized replacement teacher μ~t\tilde{\mu}_t on high-risk product states using Eq. (21).

  14. Define the decentralized distillation loss Lcf\mathcal{L}_{\mathrm{cf}} by Eq. (22).

  15. Initialize α←0\alpha\leftarrow0.

  16. for u=1,…,Uu = 1, \ldots, U do

  17.     Update α\alpha with Eq. (23), using Lretain\mathcal{L}_{\mathrm{retain}} from Eq. (24).

  18.     if u mod A=0u \bmod A = 0 or u=Uu = U then

  19.         Audit πθα\pi_{\theta_\alpha} on CauditC_{\mathrm{audit}}; obtain Su=SSR⁡(πθα)S_u = \operatorname{SSR}(\pi_{\theta_\alpha}), Wu=WER⁡(πθα)W_u = \operatorname{WER}(\pi_{\theta_\alpha}), and Vu=Util⁡(πθα)V_u = \operatorname{Util}(\pi_{\theta_\alpha}).

  20.         if S0−Su≥δS_0 - S_u \ge\delta and W0−Wu≥δW_0 - W_u \ge\delta and Vu≥ρV_u \ge\rho then

  21.             Accept α⋆←α\alpha^\star\leftarrow\alpha; merge Δθ=Bfα⋆\Delta\theta= B_f\alpha^\star into the actors.

  22.             Report SSR/WER, retention metrics, and ParamFrac.

  23.             return π−=πθα⋆\pi^- = \pi_{\theta_{\alpha^\star}}

  24.         end if

  25.     end if

  26. end for

  27. return NotAccepted.

effects of correlation loss, local observability, and constrained actor fitting.

Theory

We analyze attribution, surrogate subspace selection, the replacement teacher, conditional return retention, and decentralized information limits.

Attribution consistency.

Proposition 1 (Shapley efficiency and estimation error). For a fixed intervention baseline, let χi,t\chi_{i,t} and χ^i,t\widehat{\chi}_{i,t} be the exact Shapley values of vtv_t and its estimate v^t\widehat{v}_t. If max⁡S∣v^t(S)−vt(S)∣≤εv\max_S \lvert\widehat{v}_t(S) - v_t(S) \rvert\le\varepsilon_v, then

∑iχi,t=vt(N)−vt(∅),∣χ^i,t−χi,t∣≤2εv.(26)\sum_i \chi_{i,t} = v_t(\mathcal{N}) - v_t(\varnothing), \qquad\lvert\widehat{\chi}_{i,t} - \chi_{i,t} \rvert\le2\varepsilon_v. \tag*{(26)}

The pair term in Eq. (13) vanishes for additive vtv_t. These statements concern signed attributions before clipping.

Proof. Shapley values average permutation marginals, whose sum telescopes from vt(∅)v_t(\varnothing) to vt(N)v_t(\mathcal{N}). Each estimated marginal has error at most 2εv2\varepsilon_v. Additive coalition functions have zero second differences. □\square

Retain-aware subspace. Let Gf,Fr⪰0G_f,F_r \succeq0 and M=Fr+λ0IM = F_r + \lambda_0 I, with λ0>0\lambda_0 > 0. The semantic sensitivity surrogate is

ΔθTGfΔθ=E[(gf(τ)TΔθ)2].(27)\Delta\theta^\mathsf{T}G_f\Delta\theta= \mathbb{E}[(g_f(\tau)^\mathsf{T}\Delta\theta)^2]. \tag*{(27)}

Theorem 1 (Retain-normalized sensitivity). For 1≤K≤d1 \le K \le d, the top KK generalized eigenvectors of Gfu=λMuG_f u = \lambda M u, normalized by BTMB=IKB^\mathsf{T}MB = I_K, solve

max⁡B∈Rd×Ktr⁡(BTGfB)s.t.BTMB=IK.(28)\max_{B \in\mathbb{R}^{d \times K}} \operatorname{tr}(B^\mathsf{T}G_fB) \quad\text{s.t.}\quad B^\mathsf{T}MB = I_K. \tag*{(28)}

The optimum is the sum of the top KK generalized eigenvalues.

Proof. Set Y=M1/2BY = M^{1/2}B and A=M−1/2GfM−1/2A = M^{-1/2}G_fM^{-1/2}. The problem becomes max⁡YTY=IKtr⁡(YTAY)\max_{Y^\mathsf{T}Y=I_K}\operatorname{tr}(Y^\mathsf{T}AY); the Ky Fan maximum principle gives the result. □\square

Because Eq. (27) is unchanged by Δθ↦−Δθ\Delta\theta\mapsto-\Delta\theta, this theorem does not guarantee semantic-risk reduction. For a fixed coordinate mask, it applies to an eigenproblem solved on the selected principal submatrices, with KK no larger than the selected dimension; it does not imply optimality of the mask or of a subsequently masked unrestricted basis.

Replacement teacher. At a fixed context, write p0(a)=π0(a∣ht)p_0(a)=\pi_0(a\mid h_t) and wt(a)=βQJ(sˉt,a)−λQφ(sˉt,a)w_t(a)=\beta Q^J(\bar{s}_t,a)-\lambda Q^\varphi(\bar{s}_t,a).

Proposition 2 (KL-regularized teacher). For bounded wtw_t, the teacher in Eq. (21) uniquely minimizes

DKL(μ∥p0)−Eμ[wt](29)D_{\mathrm{KL}}(\mu\Vert p_0)-\mathbb{E}_{\mu}[w_t] \tag*{(29)}

over distributions μ≪p0\mu\ll p_0.

Proof. With Zt=Ep0[ewt]∈(0,∞)Z_t=\mathbb{E}_{p_0}[e^{w_t}]\in(0,\infty),

DKL(μ∥p0)−Eμ[wt]=DKL(μ∥μ~t)−log⁡Zt,(30)D_{\mathrm{KL}}(\mu\Vert p_0)-\mathbb{E}_{\mu}[w_t]=D_{\mathrm{KL}}(\mu\Vert\widetilde{\mu}_t)-\log Z_t, \tag*{(30)}

which is minimized uniquely at μ=μ~t\mu=\widetilde{\mu}_t. □

This optimum concerns the centralized teacher; marginal distillation need not preserve its joint-action correlations.

KL-based retention. For π−=πθα\pi^-=\pi_{\theta_\alpha}, define

κα(ht)=∑iDKL(π0,i(⋅∣hti)∥πi−(⋅∣hti)),Lretain=∑tEqr,t[κα].(31)\kappa_\alpha(h_t)=\sum_i D_{\mathrm{KL}}\left(\pi_{0,i}(\cdot\mid h_t^i)\Vert\pi_i^-(\cdot\mid h_t^i)\right),\qquad\mathcal{L}_{\mathrm{retain}}=\sum_t\mathbb{E}_{q_{r,t}}[\kappa_\alpha]. \tag*{(31)}

Here qr,tq_{r,t} is the population retain-history marginal, with the time weights of Eq. (24); Lretain\mathcal{L}_{\mathrm{retain}} is its empirical counterpart.

Proposition 3 (Conditional return retention). Fix an HH-step context cc with common initial, transition, observation, and reward kernels, ∣rt∣≤Rmax⁡|r_t|\le R_{\max}, and 0≤γ≤10\le\gamma\le1. Let dπ0,tcd_{\pi_{0,t}}^c be the original-policy history law. Suppose, for all tt,

dπ0,tc≤Ccqr,t,L‾retain(α)≤Lretain(α)+εgen,(32)d_{\pi_{0,t}}^c\le C_cq_{r,t},\qquad\overline{\mathcal{L}}_{\mathrm{retain}}(\alpha)\le\mathcal{L}_{\mathrm{retain}}(\alpha)+\varepsilon_{\mathrm{gen}}, \tag*{(32)}

where the first inequality is between measures, Cc<∞C_c<\infty, and εgen≥0\varepsilon_{\mathrm{gen}}\ge0 is finite and valid for the fitted edit. With RH=Rmax⁡∑t=0H−1γtR_H=R_{\max}\sum_{t=0}^{H-1}\gamma^t,

∣Jc(π−)−Jc(π0)∣≤2RHCc(Lretain(α)+εgen)2.(33)\left|J_c(\pi^-)-J_c(\pi_0)\right|\le2R_H\sqrt{\frac{C_c\left(\mathcal{L}_{\mathrm{retain}}(\alpha)+\varepsilon_{\mathrm{gen}}\right)}{2}}. \tag*{(33)}

Proof. Common environment kernels and factorized actors give

DKL(Pπ0c∥Pπ−c)=∑tEdπ0,tc[κα]≤CcL‾retain≤Cc(Lretain+εgen).(34)D_{\mathrm{KL}}(P_{\pi_0}^c\Vert P_{\pi^-}^c)=\sum_t\mathbb{E}_{d_{\pi_{0,t}}^c}[\kappa_\alpha]\le C_c\overline{\mathcal{L}}_{\mathrm{retain}}\le C_c\left(\mathcal{L}_{\mathrm{retain}}+\varepsilon_{\mathrm{gen}}\right). \tag*{(34)}

Bounded returns and Pinsker’s inequality yield ∣Jc(π−)−Jc(π0)∣≤2RHDKL(Pπ0c∥Pπ−c)/2\left|J_c(\pi^-)-J_c(\pi_0)\right|\le2R_H\sqrt{D_{\mathrm{KL}}(P_{\pi_0}^c\Vert P_{\pi^-}^c)/2}, proving the claim. □

The KL direction matches Eq. (24). Coverage and generalization are additional conditions, not certified by a small replay loss or action-matching MSE; the experiments do not verify them.

Decentralized information limit.

Theorem 2 (Indistinguishable decision situations). If two same-time situations x,x′x,x' give every agent identical local histories but require disjoint joint-action sets Agood(x)\mathcal{A}_{\mathrm{good}}(x) and Agood(x′)\mathcal{A}_{\mathrm{good}}(x'), no history-conditioned decentralized policy satisfies both requirements with probability one. At least one situation has violation probability at least 1/21/2.

Proof. Identical histories induce the same joint-action law μ\mu. Disjointness implies μ(Agood(x))+μ(Agood(x′))≤1\mu(\mathcal{A}_{\mathrm{good}}(x))+\mu(\mathcal{A}_{\mathrm{good}}(x'))\le1, so at least one required set has probability at most 1/21/2. □

This theorem identifies an information limit that parameter editing alone cannot resolve. Its error bound is conditional on the two indistinguishable decision situations and does not directly bound overall SSR/WER or expected return.

Limitations

The formulation assumes cooperative finite-horizon MARL, CTDE training, target strategies expressible through observable relational predicates, and limited simulator access for suffix regeneration and red-team auditing. If the original reward continues to make the forgotten strategy uniquely optimal, subsequent unconstrained training can relearn it; preventing that is continual semantic revocation rather than one-shot unlearning. The proposed guarantees are behavioral and semantic, not direct legal-data-deletion guarantees. Future work will investigate replacement policies that better preserve joint-action correlations during decentralized distillation and broader auditing over partners and deployment contexts.

References

  1. [1]Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE signal processing magazine 34, 6 (2017), 26–38.
  2. [2]Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. 2022. Vmas: A vectorized multi-agent simulator for collective robot learning. In International symposium on distributed autonomous robotic systems. Springer, Cham, 42–56.arxiv.org/abs/2207.03530
  3. [3]Lucas Bourtou le, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP). IEEE, Virtual Site, 141–159.
  4. [4]Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019), 12 pages.arxiv.org/abs/1910.05789
  5. [6]Linghan Chen, Yudong Gao, Jiaoy Wang, Kaiyan Ji, and Honglong Chen. 2026. Do system prompts leave behavioral fingerprints? a large-scale empirical study of clone detection via output similarity. arXiv:2608.24461 [cs.CR].
  6. [9]Qiuli Dai, Honglong Chen, Zhichen Ni, Yubin Yang, Linghan Chen, Xinglong Fan, and Xiangyu Li. 2025. Reinforcement learning-based resource reservation for mobile edge computing with probable failure. IEEE Transactions on Vehicular Technology 74, 5 (2025), 7985–7996.
  7. [10]Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviichuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. 2020. Is independent learning all you need in the starcraft multi-agent challenge? arXiv:2011.09533 [cs.AI].arxiv.org/abs/2011.09533
  8. [11]Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. 2023. An improved benchmark for cooperative multi-agent reinforcement learning. Advances in neural information processing systems 36 (2023), 37567–37593.
  9. [12]Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. AAAI Press, New Orleans, Louisiana, USA, 9 pages.arxiv.org/abs/1705.08926
  10. [13]Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. AAAI Press, Vancouver, Canada, 12043–12051.arxiv.org/abs/2308.07707
  11. [14]Chen Gong, Kecen Li, Jin Yao, and Tianhao Wang. 2024. Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS). Internet Society, San Diego, CA, USA.arxiv.org/abs/2404.12530
  12. [15]Michel Grabisch and Marc Roubens. 1999. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory 28, 4 (1999), 547–565.
  13. [16]Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. 2021. Multi-agent constrained policy optimisation. arXiv:2110.02793 [cs.AI].arxiv.org/abs/2110.02793
  14. [17]Junfeng Guo, Ang Li, Lixu Wang, and Cong Liu. 2023. Polycycles: Backdoor detection and mitigation for competitive reinforcement learning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Paris, France, 4676–4685.DOI
  15. [18]Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael Wooldridge. 2021. Multi-agent reinforcement learning with temporal logic specifications. In Proceedings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 583–592.arxiv.org/abs/2102.00582
  16. [19]Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. 2018. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning. PMLR, Stockholm, Sweden, 2107–2116.
  17. [20]Erin Illman and Paul Temple. 2019. California consumer privacy act. The Business Lawyer 75, 1 (2019), 1637–1646.
  18. [21]Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jay Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. 2021. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning. PMLR, Virtual Site, 6187–6199.
  19. [22]Na Li, Chunyi Zhou, Yansong Gao, Hui Chen, Zhi Zhang, Boyu Kuang, and Anmin Fu. 2025. Machine unlearning: Taxonomy, metrics, applications, challenges, and prospects. IEEE Transactions on Neural Networks and Learning Systems 36, 8 (2025), 13709–13729.DOI
  20. [24]Huanghuang Liang, Weihan Wu, Rui Ge, Jiawei Jiang, and Dazhao Cheng. 2026. Decoupled Attribute Subspace Unlearning for Private Federated Recommender Systems. IEEE Transactions on Dependable and Secure Computing early access (2026), 1–13.DOI
  21. [25]Huanghuang Liang, Weihan Wu, Zheng Gong, Zhili He, Yili Gong, Jiawei Jiang, and Dazhao Cheng. 2026. Federated Unlearning via Representation Misdirection with Adaptive Anchor Generation. IEEE Transactions on Dependable and Secure Computing early access (2026), 1–14.DOI
  22. [26]Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017), 6382–6393.arxiv.org/abs/1706.02275
  23. [28]Alex Oesterling, Jiaqi Ma, Flavio Calmon, and Himabindu Lakkaraju. 2024. Fair machine unlearning: Data removal while mitigating disparities. In International Conference on Artificial Intelligence and Statistics. PMLR, Valencia, Spain, 3736–3744.arxiv.org/abs/2307.14754
  24. [29]Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. 2021. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems 2021 Datasets and Benchmarks Track. Curran Associates, Inc., Virtual Site.arxiv.org/abs/2006.07869
  25. [30]Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1–51.
  26. [31]Richard Ruhdörfer, Kalesha Bullard, Adam Gleave, Edward Grinfenstette, Tim Rõcktäsch el, and Mikayel Samvelyan. 2024. The Overlooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design. arXiv:2406.17949 [cs.LR]arxiv.org/abs/2406.17949
  27. [32]Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2186–2188.arxiv.org/abs/1902.04043
  28. [33]Lloyd S. Shapley. 1952. A Value for n-Person Games. Paper P-295. RAND Corporation, Santa Monica, CA.
  29. [34]Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018. Value-decomposition networks for cooperative multi-agent learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Stockholm, Sweden, 2085–2087.arxiv.org/abs/1706.05296
  30. [35]Weiqi Wang, Zhiyi Tian, Chenhan Zhang, and Shui Yu. 2024. Machine unlearning: A comprehensive survey. arXiv:2405.07406 [cs.CR]arxiv.org/abs/2405.07406
  31. [36]Weihan Wu, Yuli Gong, Jiawei Jiang, Chuang Hu, Xiaobo Zhou, and Dazhao Cheng. 2025. Defending against attribute inference attacks in post-training of recommendation systems via unlearning. In 2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, Hong Kong SAR, 2656–2669.
  32. [37]Weihan Wu, Zhili He, Huanghuang Liang, Yili Gong, Jiawei Jiang, Chuang Hu, and Dazhao Cheng. 2026. REMISFVU: Vertical Federated Unlearning via Representation Misdirection for Intermediate Output Feature. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. AAAI Press, Singapore, 26983–26991.
  33. [38]Weihan Wu, Jiawei Jiang, and Chuang Hu. 2025. Aegis: Post-training attribute unlearning in federated recommender systems against attribute inference attacks. In Proceedings of the ACM Web Conference 2025. ACM New York, NY, USA, Sydney, Australia, 3783–3793.DOI
  34. [39]Weihan Wu, Huanghuang Liang, Tianyu Tu, Jiawei Jiang, Chuang Hu, and Dazhao Cheng. 2025. Mimir: Data-free federated unlearning through client-specific prompt generation for personalized models. IEEE Transactions on Mobile Computing 24, 10 (2025), 10537–10556.DOI
  35. [40]Wenhan Wu, Huanghuang Liang, Jingling Yuan, Jiawei Jiang, Kanyie Ye Wang, Chuang Hu, Xiaobo Zhou, and Dazhao Cheng. 2025. Zero-shot Federated Un-learning via Transforming from Data-Dependent to Personalized Model-Centric.. In IJCAI. IJCAI Press, Montreal, Canada, 6588–6596.
  36. [41]Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. 2024. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence 8, 3 (2024), 2150–2168.DOI
  37. [42]Minghan Yang, Dong Xu, Qiuwei Cui, Zaiwen Wen, and Pengxiang Xu. 2022. An efficient Fisher matrix approximation method for large-scale neural network optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2022), 5391–5403.DOI
  38. [43]Dazhong Ye, Tianqiang Zhu, Congcong Zhu, Derui Wang, Kun Gao, Zewei Shi, Sheng Shen, Wanlei Zhou, and Minhui Xue. 2025. Reinforcement unlearning. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS). Internet Society, San Diego, CA, USA.arxiv.org/abs/2312.15910
  39. [44]Lei You, Yijun Bian, and Lele Cao. 2026. Joint distribution-informed shapley values for sparse counterfactual explanations. In International Conference on Learning Representations, Vol. 2026. OpenReview.net, Seoul, South Korea, 100183–100205.
  40. [45]Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35 (2022), 24611–24624.arxiv.org/abs/2103.01955
  41. [46]Razieh Nokkbeh Zaeem and K Suzanne Barber. 2020. The effect of the GDPR on privacy policies: Recent progress and future promise. ACM Transactions on Management Information Systems (TMIS) 12, 1 (2020), 1–20.
  42. [47]Rui Zuo, Qinwei Huang, Mingyang Li, Zhenhang Zhang, Simon Khan, and Qinru Qiu. 2023. MUTE: Return-Preserving Communication Unlearning for Efficient Multi-Agent Coordination. arXiv:2607.03473 [cs.MA]

Paper details

vlnc:W_f2agfb5q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents