betaFeedback

WorksComputer Science

Who Taught the Robot That? Removing One User's Corrections from a Robot Policy That Learns Online

Abstract

A robot that keeps learning after deployment is corrected by several people in turn, and one of them may later ask to be removed. We show that whether the robot can forget that person depends on how people correct it. The goal of removal is the robot that would exist if the person had never been there, which we compute exactly in two simulated tasks with scripted users. When users correct the robot by taking over control, an ordered log of training updates is enough: retraining on the remaining sessions in their original order gives a robot as close to the goal as training noise allows. Deleting the person's data and retraining in one pass does not, even when the other users' corrections do not depend on the removed person, because it loses the order of training. When users only nudge the robot's action, the other users' corrections carry the removed person's influence, and no method we tested recovers the goal robot. Continuing to train without the person removes most of their habits, but not what their data in the training buffer keep alive. The log can identify one clearly harmful user, but not rank users by influence better than counting their corrections.

Introduction

Picture a robot in a shared home or a small clinic. Several people correct it over weeks: one steers it away from the stairs, another changes where it sets things down, a third slows it down. After each person’s corrections, the robot’s control policy, the function that chooses its action in each situation, is retrained. Later one of these people has to be removed. They withdraw consent, they move out, or their corrections were wrong and the robot now fails at its task. The operator needs to undo what that person taught the robot.

Machine unlearning studies how to remove one contributor’s data from a trained model. In federated learning, methods such as FedEraser [1] keep a log of every contributor’s past model updates and use it to rebuild the model without one of them [2, 3]. These methods assume that each contributor’s data are fixed and do not depend on the other contributors. Robot corrections break this assumption. A person corrects the robot while it acts, in situations the robot reached because of everyone who corrected it before. If user A teaches the robot to use the lower of two gaps in a wall, user B later corrects a robot that already heads for the lower gap, so B’s corrections reflect what A taught. Deleting A’s data may therefore leave some of A’s influence in the robot.

We study this question in simulation, where we can compute the robot that never met the removed user by rerunning the whole deployment without them. Our main finding is that whether a robot can forget one person’s corrections depends on how people correct it. If people take over control, their influence can be removed using an ordered log of training updates. If people only nudge the robot’s action, it cannot be removed with any method we tested. In detail:

  1. Deleting a user’s data and retraining on the rest in one pass gives the wrong robot, mainly because it loses the order of training. It fails by the same amount when we make the users’ corrections independent of each other by design. Retraining on the remaining sessions in their original order gives the right robot, up to random training noise; the same sessions in shuffled order do not.

  1. When users only nudge the robot, no method we tested gives the right robot. A nudge contains part of the robot’s own action, which the removed user had shaped. For nudges that move the action 20 to 60% of the way toward the user’s choice, retraining in order stays farther from the right robot than training noise explains, and for nudges up to 45% it keeps too much of the removed user’s habits. Rebuilding the corrections from the log fixes part of this, but not all, because the removed user also changed which situations the others corrected.

  1. Continuing to train without the user removes most of their habits, except what their old data keep alive. A habit that all remaining users oppose shrinks to 15% of its initial size and stays there, because the removed user’s corrections are still reused in training. Deleting those corrections, without any other change, shrinks it to 2%.

  1. The log does not reliably say who taught the robot a behaviour. It identifies one clearly harmful user among three in 85 to 95% of runs. Among six users with random preferences, its ranking of who influenced a behaviour most agrees only weakly with the true ranking, and no better than simply counting each user’s corrections.

A 2D delivery task and a simulated robot arm give the same answers for the two correction types we ran on both (findings 1 and 2).

Related work

Machine unlearning. FedEraser [1] stores each contributor’s past updates in federated learning and uses them to remove a contributor; surveys cover the wider field [2, 3, 4]. SISA [5] makes retraining exact by saving model checkpoints and retraining only from the last checkpoint before the removed data. Our replay method is exact retraining from a checkpoint in this sense. We add a goal for removal when data are collected interactively, a test of when replay reaches it, and the finding that training order matters even for independent data.

Unlearning in robot policies. Li et al. [6] evaluate removal of demonstrations from imitation-learning policies against a policy retrained without them, and measure both the policy’s behaviour and statistical traces of the removed data. VLA-Forget [7] and TrajDeleter [8] remove data from a fixed dataset, and MoRE [9] suppresses unwanted behaviours by editing the network’s weights. All of these work with a fixed dataset; we study a robot that learns from several people over time, and compare against a rerun of the deployment rather than a retrained policy.

Learning from interventions and shared control. Our takeover users follow HG-Dagger [10], in which a person takes control when the robot errs and the robot learns from what the person did. Our partial users follow policy blending in shared control [11], in which the executed action is a mix of the robot’s and the person’s. Data attribution [12, 13] and multi-user preference learning [14, 15] do not consider removing a person after deployment.

Setting

How the robot learns from users

The robot starts with a base policy π0\pi_0, trained by imitating 200 demonstrations. It is then deployed, and users take turns in sessions. In a session, one user watches the robot for 10 episodes (attempts at the task). When the robot does something the user dislikes, the user intervenes, and the actions the user supplies are recorded as corrections: pairs of a state and the action the robot should have taken there. After the 10 episodes, the policy is fine-tuned for 200 gradient steps. Each training batch is drawn 50% from this session’s corrections, 25% from the corrections of all earlier sessions (we call these stored earlier corrections the replay buffer) and 25% from the original demonstrations.

Three users, A, B and C, each get four sessions. A round gives each user one session. The schedule’s block length kk sets how often users alternate: with k=1k=1 the order is A B C A B C . . .; with k=4k=4 it is AAAA BBBB CCCC.

What removing a user should produce

To remove user uu, the goal is the policy the robot would have if uu had never taken part. We call it the counterfactual policy π−u\pi_{-u} and compute it by rerunning the whole deployment without uu’s sessions, using the same random numbers for every other session. This is not the same as training on everyone’s data except uu’s: without uu, the other users would have seen a different robot, intervened at different moments and given different corrections. A real operator cannot compute π−u\pi_{-u}, because they cannot ask the other users to repeat weeks of corrections. Scripted users let us compute it, so we can measure how close each removal method gets.

Two ways of correcting

Each user has a preferred expert action aexpa^{\mathrm{exp}} in every state. A takeover user, at the first moment the robot does something they dislike, takes control and drives the robot with aexpa^{\mathrm{exp}} until the end of the episode. A partial user, from that moment on, only nudges the robot: the executed and recorded action is

a=(1−α)arobot+αaexp,0<α≤1,a = (1-\alpha)a^{\mathrm{robot}}+\alpha a^{\mathrm{exp}}, \qquad0 < \alpha\leq1,

where arobota^{\mathrm{robot}} is what the robot would have done and α\alpha, the correction strength, is how far the user moves the action toward their own choice. At α=1\alpha=1 the partial user behaves exactly like a takeover user. At α=0.3\alpha=0.3, the recorded action is 70% the robot’s own action. Since the robot’s action reflects what earlier users taught it, so does the recorded correction.

Correction log

After each session we store one log record: who corrected and when, the corrections themselves, the robot’s own action at each correction, the random seed and number of steps used for the update, and a copy of the policy before the update (a checkpoint). Repeating the logged updates in order, starting from π0\pi_0, reproduces the deployed policy exactly.

Removal methods

Retrain. Start from π0\pi_0 and train once on all logged corrections except uu’s, mixed together in random batches, for the same total number of steps. This is what “delete the user’s data and retrain” usually means.

Replay. Start from the checkpoint saved just before uu’s first session, then repeat every later session except uu’s, in the original order, with the same seeds and the same batch rule. The result is the same as replaying from π0\pi_0; the checkpoint only saves computation.

Shuffled replay. Replay with the remaining sessions in random order. This tests whether the order matters.

Relabelled replay. For partial users only. Replay, but rebuild each correction using the policy being replayed instead of the deployed policy: the new recorded action is the replayed policy’s action plus the user’s logged nudge, πreplay(s)+(a−arobot)\pi_{\mathrm{replay}}(s)+(a-a^{\mathrm{robot}}). An oracle version instead recomputes the correction from the user’s true expert action. An operator cannot use the oracle, because they do not know each user’s expert action; it shows the best that relabelling could do.

Other methods. Gradient-ascent unlearning starts from the deployed policy and trains it to fit uu’s corrections worse while still fitting everyone else’s. Per-user adapters keep π0\pi_0 fixed and give each user a small add-on network (a LoRA adapter) trained only on their sessions; removal deletes uu’s adapter. Continuing without the user keeps the deployed policy and goes on training with the remaining users.

How we measure success

Action distance. To compare two policies we give them the same states and measure how differently they act: d(π1,π2)d(\pi_1,\pi_2) is the average Euclidean distance between their actions, over 500 states visited by π−u\pi_{-u} and 500 states where uu corrected the robot. The units are those of the action (in Delivery2D, a velocity per step, at most 0.05).

Noise floor. Training is random: two runs of the same deployment with different random seeds end with slightly different policies. We measure this by rerunning π−u\pi_{-u} with three new training seeds and averaging their distance to π−u\pi_{-u}. We call this the noise floor. A removal method whose distance to π−u\pi_{-u} equals the noise floor is as close as one can expect; we often report a distance as a multiple of the floor, so that 1 means “as close as training noise allows”. Replay repeats the sessions before uu’s first session exactly as π−u\pi_{-u} did, which removes some of the randomness. For replay we therefore use a prefix floor, computed by rerunning π−u\pi_{-u} with new seeds only from uu’s first session onward. The floor differs between experimental conditions, so when we compare conditions we use raw distances.

Share of influence left. To show how much of uu’s influence a method removes, we divide its distance to π−u\pi_{-u} by the distance of the deployed policy, which still contains all of uu’s influence:

r=d(πmethod,π−u)d(πdeployed,π−u),rmin⁡=noise floord(πdeployed,π−u).r = \frac{d(\pi_{\mathrm{method}},\pi_{-u})}{d(\pi_{\mathrm{deployed}},\pi_{-u})}, \qquad r_{\min} = \frac{\text{noise floor}}{d(\pi_{\mathrm{deployed}},\pi_{-u})}.

r=1r=1 means the method removed nothing. rr cannot fall much below rmin⁡r_{\min}, because even a perfect method stays one noise floor away from π−u\pi_{-u}. We report r−rmin⁡r-r_{\min}: 0 means the method did as well as possible.

Preference match. The share of 100 test episodes in which the robot does what user uu prefers: it passes the obstacle the way uu likes and places the object within 0.08 of uu’s preferred position. We report a method’s preference match minus that of π−u\pi_{-u}. A positive value means the method keeps more of uu’s habits than the counterfactual robot would have; a negative value means it removes more.

Statistics. Every comparison is made seed by seed: both methods are run on the same 20 seeds, and we report the mean difference, a 95% bootstrap confidence interval and the pp-value of a Wilcoxon signed-rank test. When we claim that two quantities are equal, we use a paired equivalence test (two one-sided tests) with the margin stated.

Tasks

Delivery2D (Figure 1, top). The state is the robot’s position, the object’s position and the goal’s position. The action is a velocity, at most 0.05 per step, and an episode lasts 150 steps. A wall in the middle has a low gap and a high gap. User A prefers the low gap and a low drop position; B the high gap and a high drop; C the high gap, a middle drop and a slow speed. User H prefers a drop position just outside the goal band, so a robot that follows H fails the task. A user intervenes when the robot heads for the gap they do not want, puts the object down too far from their preferred position, moves faster than they allow, or drives away from the object before picking it up.

Figure 1. The two tasks. Top: Delivery2D, seen from above. (a) The robot picks up an object on the left, passes through the low or the high gap in a wall, and puts the object down in the goal band on the right. Users differ in which gap they prefer, where in the band they want the object, and how fast the robot may move. (b) The base policy heads for the wrong gap; user A takes over (circle) and finishes the episode their way. Bottom: Arm3, a planar arm with three joints. Thick lines show each user’s preferred path of the hand; thin lines show the arm at several moments. The arm picks up an object from the table, carries it over the central block and places it on the shelf. Users differ in whether the elbow points up or down while carrying, where on the shelf the object goes, and how fast the joints may move. User H places the object off the shelf.

Arm3 (Figure 1, bottom). The arm moves in a vertical plane, and the action is a velocity for each of its three joints. The preferences are the elbow direction while carrying (the counterpart of the gap in Delivery2D), the position on the shelf, and a joint speed limit. The scripted users compute their actions with standard inverse-kinematics control and succeed in 98 to 100% of episodes.

In both tasks the policy is a neural network with two hidden layers of 256 units. The base policy π0\pi_0 succeeds in 90% of episodes on Delivery2D (between 52% and 100% depending on the seed) and 92% on Arm3. We report all seeds; restricting the results to seeds where π0\pi_0 succeeds at least 80% of the time changes none of them noticeably (Appendix § 13). Unless stated, results use 20 seeds; the Arm3 decomposition in § 5.1 and the experiments in § 5.4 use 10.

Results

With takeover users, one-pass retraining fails because it loses the order of training

Retrain and replay use the same logged corrections and differ only in how they train on them: retrain mixes everything in one pass, replay repeats the sessions in order. This lets us split retrain’s distance from the counterfactual policy into two parts:

d(retrain,π−u)⏟total gap,d(retrain,replay)⏟due to the training procedure,d(replay,π−u)⏟due to the data.\underbrace{d(\mathrm{retrain},\pi_{-u})}_{\text{total gap}}, \qquad \underbrace{d(\mathrm{retrain},\mathrm{replay})}_{\text{due to the training procedure}}, \qquad \underbrace{d(\mathrm{replay},\pi_{-u})}_{\text{due to the data}}.

The last part is what remains when the procedure is right; it measures how much the other users’ corrections still carry the removed user’s influence.

To test directly whether that influence is the problem, we also run a frozen-robot control. In every session, the user corrects the base policy π0\pi_0 instead of the current, already-corrected robot; training is unchanged. No user then sees what earlier users taught, so no user’s corrections depend on anyone else’s. In this control the logged data are exactly the data the counterfactual run would collect, and replay equals π−u\pi_{-u} exactly. If retrain still fails here, the dependence between users is not the cause.

The dependence between users is not what makes retrain fail. On Delivery2D, retrain ends 1.30 times the noise floor away from π−u\pi_{-u} in the normal deployment and also 1.30 times in the frozen-robot control (paired difference −0.00-0.00, 95% CI [−0.09-0.09, +0.09+0.09], p=0.87p=0.87; Figure 2, Table 1). On Arm3 the control lowers retrain’s error by 26%, so most of the error remains.

Figure 2. Figure 2: Where retrain’s error comes from, Delivery2D, takeover users. Each group of bars is one schedule (kk is the block length). Blue: total distance from retrain to the counterfactual policy π−u\pi_{-u}. Orange: distance from retrain to replay (due to the training procedure). Green: distance from replay to π−u\pi_{-u} (due to the data). All are divided by the noise floor; the horizontal line at 1 is as close as training noise allows. (a) Normal deployment. (b) Frozen-robot control, where no user’s corrections depend on another’s, so replay equals π−u\pi_{-u} exactly. Retrain’s total error is the same in both panels.

Replay reaches the counterfactual policy. Replay ends 0.91 times the prefix floor from π−u\pi_{-u} on Delivery2D and 0.90 on Arm3: as close as training noise allows (equal to the floor within ±0.25\pm0.25, p=0.001p=0.001 on Delivery2D). It leaves the removed user’s preference match where π−u\pi_{-u} has it (+0.001+0.001, equal within ±0.05\pm0.05), and it removes as much of the user’s influence as noise allows (r−rmin⁡<0r-r_{\min}<0).

What retrain loses is the order. Replaying the same sessions in shuffled order moves the result 0.44 noise floors farther from π−u\pi_{-u} (p<10−5p<10^{-5}), and 1.28 floors farther in the frozen-robot control. Two attempts to fix retrain without replaying in order, matching the deployment’s mix of data in each batch and weighting each session by how often it was used, did not help (Appendix § 10). What retrain loses is not how often each session is seen, but that the late sessions are trained on last.

Retrain’s error is not specific to the removed user. Whichever user is removed, retrain moves the robot toward the preferences of the user who comes first in each round and away from the others. When we reverse the order within each round from A B C to C B A, the favoured user changes from A (+0.25 preference match) to C (+0.19). This order effect also explains why retrain seems to remove too much of the removed user in Table 1: with B removed, B’s preference match under retrain is 0.20 below π−u\pi_{-u}’s in one order and 0.23 above it in the other.

taskcorrectionnormalcontroldistancereplayretrainreplay / retrain
Delivery2Dtakeover3.83.80.91+0.00−0.10−0.09 / +0.15
partial, α=0.3\alpha=0.34.72.31.50+0.06+0.00+0.23 / +0.25
Arm3takeover7.05.20.90+0.00−0.07−0.04 / +0.03
partial, α=0.3\alpha=0.38.53.51.69+0.03+0.01+0.26 / +0.25

Table 1. Table 1: Main results. Each row is one task and one way of correcting, averaged over block lengths and over which user is removed. Retrain gap: distance from retrain to the counterfactual policy π−u\pi_{-u}, in action units (×10−3\times10^{-3}), in the normal deployment and in the frozen-robot control, where no user’s corrections depend on another’s. Replay distance: distance from replay to π−u\pi_{-u} as a multiple of the prefix noise floor (1 is as close as training noise allows). Preference: the removed user’s preference match minus that of π−u\pi_{-u} (0 is right; positive keeps too much of the user). r−rmin⁡r-r_{\min}: the share of the user’s influence left beyond what training noise forces (0 is the best possible). Bold marks results within training noise of π−u\pi_{-u}.

The dependence exists but is small. When uu is present, the other users intervene for somewhat different reasons and in different places, but not more often, and the difference disappears when users do not alternate (k=4k=4). Its one visible consequence is that retrain’s error grows with alternation in the normal deployment but not in the control (Appendix § 9).

With partial corrections, no method we tested reaches the counterfactual policy

We now vary the correction strength α\alpha over 0.1, 0.2, 0.3, 0.45, 0.6, 0.8 and 1 on Delivery2D (Figure 3).

Figure 3. Effect of correction strength α\alpha, Delivery2D, schedule k=1k=1. “Take-over” is the takeover user, identical to α=1\alpha=1. (a) Distances in action units. Green: replay to π−u\pi_{-u} (due to the data). Orange: retrain to replay (due to the training procedure). Blue: retrain to π−u\pi_{-u} in the frozen-robot control. Grey: prefix noise floor. (b) The removed user’s preference match minus that of π−u\pi_{-u} for each method; 0 is right, and positive values keep too much of the user.

For intermediate correction strengths, the dependence between users matters. For α\alpha from 0.1 to 0.6, retrain’s error in the normal deployment is larger than in the frozen-robot control by 1.4 to 3.1×10−33.1 \times10^{-3} action units (p<10−5p<10^{-5} at each α\alpha). For α≥0.8\alpha\ge0.8 the difference is not significant. The effect is not symmetric: it is present at α=0.2\alpha=0.2 and absent at α=0.8\alpha=0.8. What matters most is how much of the robot’s own action stays in each correction (1−α1-\alpha), more than how strongly the removed user changed the robot.

Replay keeps too much of the removed user. For α\alpha from 0.2 to 0.6, replay ends farther from π−u\pi_{-u} than the prefix floor (p≤0.03p \le0.03), and for α≤0.45\alpha\le0.45 it keeps more of the removed user’s preferences than π−u\pi_{-u} does (preference match +0.024 to +0.058). The reason is that the other users’ corrections contain the robot’s own action, which carried the removed user’s habit, and replay trains on those corrections as they were recorded. Retrain is no better: for every α≤0.6\alpha\le0.6 its rr is about 1 or more, so it removes no more of the user’s influence than doing nothing. Replay removes as much as noise allows at α=0.1\alpha=0.1 and at α≥0.8\alpha\ge0.8; in between it is 0.16 to 0.36 above that limit. Arm3 at α=0.3\alpha=0.3 gives the same picture (Table 1).

Rebuilding the corrections helps only in part. The oracle relabelling, which knows each user’s true expert action, removes 43 to 64% of replay’s preference error for α≤0.3\alpha\le0.3, but changes replay’s action distance to π−u\pi_{-u} by only −13 to +8%. The relabelling an operator can compute is worse than plain replay at every α\alpha: its rebuilt correction differs from the true one by α(πreplay(s)−πdeployed(s))\alpha(\pi_{\mathrm{replay}}(s) - \pi_{\mathrm{deployed}}(s)), which grows with α\alpha. Since even perfect labels leave the action distance unchanged, the rest of the error comes from where the corrections were made. The removed user changed which situations the other users saw and corrected, and the log cannot supply the corrections the others would have given in the situations the counterfactual robot reaches.

Other removal methods

Gradient-ascent unlearning and deleting a user’s adapter both damage the robot: on Arm3 they lower task success from 0.76 to 0.54 and to 0.02. A membership-inference test finds traces of users only on Arm3 with partial corrections, and replay and retrain remove them (Appendix § 8).

Continuing without the user, and the replay buffer

In a deployment with users A, B, and the harmful user H, the robot ends up failing its task often: on Arm3 it succeeds in 33% of episodes, against 60% for the robot that never met H. An operator might simply stop listening to H and keep training with the others. H’s harmful habit, putting the object down outside the goal, then fades quickly, because every remaining user corrects it each time it happens: on Delivery2D it returns to the level of π−u\pi_{-u} within 8 rounds, and on Arm3 two rounds bring it to 7% of episodes, against 15% for π−u\pi_{-u}.

H’s other habit, the choice of gap, does not fade fully on Delivery2D. When all remaining users prefer the other gap, H’s habit falls from +0.66 to +0.18 in two rounds and then stays near +0.10, 15% of its initial size (Table 2). It persists because H’s old corrections remain in the replay buffer and are reused in every later batch. Deleting them from the buffer, and changing nothing else, lowers the habit to +0.01 after 8 rounds (paired difference −0.09, pp=0.02). When the remaining users disagree among themselves about the gap, the habit fades fully with or without the deletion. Appendix D reports how the result changes when users notice only some violations, and with partial users.

R=0R=0R=2R=2R=4R=4R=8R=8
continue, H’s corrections kept in the replay buffer+0.66+0.18+0.12+0.10
continue, H’s corrections deleted from the replay buffer+0.66+0.02−0.00+0.01

Table 2. How H’s preferred gap fades when training continues with the remaining users, Delivery2D, takeover users, all remaining users preferring the other gap, 10 seeds. Each entry is the robot’s share of trips through H’s preferred gap minus the same share for the counterfactual policy π−u\pi_{-u} (0 means none of H’s habit is left), after RR further rounds without H.

Can the log say which user taught a behaviour?

An operator may also want to answer the question in our title: which user taught the robot a behaviour it now shows? We test this on Delivery2D with six users whose preferences are drawn at random, three sessions each, interleaved, for three groups of users and 20 seeds each. We look at three behaviours bb: the average drop position, the share of trips through the low gap, and the average speed. The true influence of user vv on a behaviour is how much the behaviour changes when vv is removed by rerunning the deployment:

Iv=b(πdeployed)−b(π−v).I_v = b(\pi_{\mathrm{deployed}}) - b(\pi_{-v}).

We compare four ways an operator could estimate IvI_v from the log, described in Appendix E. The main one removes vv by replay and measures the change in the behaviour; the simplest counts each user’s logged corrections that push the behaviour, which needs no training. We score each by the Spearman rank correlation between its ranking of the six users and their true ranking: 1 means the same order, 0 means no relation.

Removing each user by replay gives a mean rank correlation of only 0.25 with takeover users and 0.32 with partial users at α\alpha=0.3, no better than counting corrections (0.24 and 0.25; pp=0.92 and 0.15), although counting needs no training and replay about 18,000 gradient steps. Replay is better only at naming the single most influential user (45 to 50% of cases, against 17% for counting and for chance). With one clearly harmful user among A, B and H, replay names H as the most influential user in 85 to 95% of seeds. Details are in Appendix E.

Limitations

Simulation and scripted users. Our users are programs with fixed preferences, one correction strength and perfect consistency. Real people correct inconsistently, change their minds and mix both ways of correcting. The counterfactual policy exists only in simulation, so an operator with real users cannot check replay against it.

Small effects and small policies. The deployed policy is only 1.0 to 1.75 noise floors from π−u\pi_{-u}, so our conclusions rest on paired tests over 20 seeds and on preference measures, not on large differences; the same noise limits the attribution results. Both policies are small networks with low-dimensional inputs, and on Arm3 learning from corrections lowers task success a lot (from 0.92 to 0.60 in the harmful-user setting). Whether larger policies keep traces of a user that behaviour does not show is untested (Appendix A).

Conclusion

Whether a robot can forget one person’s corrections depends on how people correct it. Corrections that replace the robot’s action can be undone with an ordered log of training updates; one-pass retraining cannot undo them, because it loses the order of training. Partial corrections, as in shared control, could not be undone by any method we tested, because the other people’s corrections carry the removed person’s influence. Continuing to train without the person is not removal either, because their old data in the replay buffer keep part of their habits alive.

In practice, the log should store the order and random seeds of the updates, not only who gave which corrections. Checkpoints are optional, since replaying from π0\pi_0 gives the same result. The log is itself a privacy risk: it and every checkpoint saved after a user’s first session contain that user, so a removal request means deleting both once the replay is done.

Other removal methods

Gradient-ascent unlearning starts from the deployed policy and, for 200 steps, trains it to fit uu’s corrections worse while still fitting everyone else’s, with a bound that keeps the loss finite. On Arm3 with takeover users it ends 10 noise floors from π−u\pi_{-u} and lowers task success from 0.76 to 0.54. Nothing in its objective says how much of the user to remove, so it removes too much and damages the rest of the policy.

Per-user adapters keep π0\pi_0 fixed and give each user a small add-on network (a rank-4 LoRA adapter on both hidden layers) trained only during their sessions. Deleting one user’s adapter lowers task success on Arm3 to 0.02, because each adapter was trained while the others were active and learned to work together with them. An earlier Delivery2D run with the same protocol gave the same ordering: 23 noise floors for gradient ascent, and success 0.14 after deleting an adapter.

Membership inference. A loss-threshold membership-inference test [16] asks whether a policy fits a user’s logged corrections more closely than fresh corrections from the same user, a sign that it was trained on them. Its score, the area under the ROC curve (AUC), is 0.5 when there is no trace of the user. On Delivery2D it is 0.49 to 0.51 for every policy, including the deployed one. On Arm3 with partial users at α=0.3\alpha=0.3 it finds a trace: 0.59 for the removed user in the deployed policy, and 0.55 for H both in the deployed policy and after two more rounds without H. Retrain and replay bring it back to 0.50 to 0.52. With takeover users on Arm3 it is 0.49 to 0.51.

How the other users’ corrections change when the removed user is present

We compare, session by session, the other users’ corrections in the normal deployment (with uu) and in the counterfactual run (without uu), which share the same random numbers. As a noise reference we compare the counterfactual run with a rerun that uses new training seeds. Delivery2D, takeover users, 20 seeds.

  • Why they intervene. Each intervention has a reason: wrong gap, wrong drop position, too fast, or leaving the object behind. The share of each reason differs between the two runs by 0.055 more than noise, measured as total variation distance, which is half the sum of the absolute differences in the shares (p<10−4p<10^{-4}).

  • Where they intervene. The sets of states where interventions start differ by 0.010 more than noise in squared maximum mean discrepancy (MMD), a standard measure of how different two sets of points are, which is 0 when they come from the same distribution (p<10−4p<10^{-4}).

  • How much they correct. The number of corrections per session does not change beyond noise (p=0.20p=0.20).

  • Effect of the schedule. When users do not alternate (k=4k=4), the difference in where they intervene disappears (p=0.96p=0.96).

  • Effect on retrain. Retrain’s error grows with alternation, by 0.29 noise floors from k=4k=4 to k=1k=1 (p<10−3p<10^{-3}), in the normal deployment but not in the frozen-robot control.

Two attempts to fix retrain without replaying in order

Matching the batch mix. During deployment, each batch is 50% the current session, 25% earlier sessions and 25% demonstrations. A version of retrain whose batches have the same average mix, still in one pass, changed retrain’s distance to π−u\pi_{-u} by +0.08 noise floors, so it did not help.

Weighting each session by how often it was used. We gave each session’s data the weight it effectively had during deployment, that is, its expected share of all gradient updates. Under our batch rule this weight decreases with the session’s position, because every later session reuses the earlier ones: in one example the first session had a share of 0.19 and the last 0.09. Weighted retrain was equal to or worse than plain retrain in all 8 settings we tried (takeover and α=0.3\alpha=0.3; k=1k=1 and 4; orders A B C and C B A), and it strengthened the bias toward the user who corrects first in each round. Replay sees early sessions more often in total but late sessions last, and it is the second property that matters.

Fading of a habit: attention and partial users

We vary the share of remaining users who prefer the gap H did not (0, 0.25, 0.5, 0.75 or 1), the number of remaining users (2, 4 or 6) and the probability that a user notices a violation (1.0, 0.7 or 0.4); 10 seeds.

Attention. When users notice only 40% of violations, slightly more of H’s gap habit remains after 4 rounds (+0.07 compared with full attention; pp=0.08 to 0.11 with 10 seeds). A noticing probability of 0.7 makes no difference.

Partial users. With partial users at α\alpha=0.3, H’s gap habit starts smaller (+0.17 instead of +0.66) and fades fully within 2 rounds at full attention (+0.005 after 8 rounds). Lower attention leaves a small residue (+0.056 at a noticing probability of 0.4 after 4 rounds, pp=0.002). Deleting H’s corrections from the buffer helps when three quarters of the remaining users oppose H’s gap (−0.049, pp=0.037).

Disagreement among the remaining users. When at most half of the remaining users oppose H’s gap, the habit fades to zero with takeover users whether or not H’s corrections are deleted from the buffer.

Attribution methods and detailed results

Methods. We compare four ways to estimate a user’s influence IvI_v from the log.

  1. Remove the user by replay: replay without vv and measure the change in the behaviour.

  2. Remove one session at a time: replay without each session in turn and add up the changes for each user.

  3. A quick estimate without retraining: for each logged update, multiply the stored change in the network’s weights by the direction in which the weights would have to move to change the behaviour, and add these up for each user. This is a first-order estimate in the style of TracIn.

  4. Counting: count each user’s logged corrections that push the behaviour, with a plus or minus sign for the direction (the logged action minus the robot’s action at that state, along the behaviour’s axis). This uses the log’s records only and needs no training.

We also report a random ranking.

Results. Mean Spearman rank correlation with the true influence, over 60 groups of users and seeds:

users1 remove user2 remove session3 quick estimate4 countingrandom
takeover0.25 [0.19, 0.31]0.190.110.24−0.01
partial, α\alpha=0.30.32 [0.25, 0.38]0.310.180.250.05

Table 1 (PDF p. 11).

Removing one session at a time is no better than removing the user and costs 1.7 times as much. The quick estimate is the weakest, and in the harmful-user case its sign flips between takeover and partial users (−0.30 and +0.72).

Per behaviour. With method 1, the correlation is moderate for the share of low-gap trips (0.56) and close to zero for average drop position (0.12 with takeover users, 0.30 with partial users) and average speed (0.08 to 0.11). Unlike removal, attribution does not get worse with partial users (+0.07, pp=0.16).

Noise in the true influence. The true influence of each user comes from one rerun of the deployment without that user, and training noise is large in this task (§ 6). Part of the low correlation may therefore come from noise in this reference; we did not measure how much.

Seeds where the base policy is weak

On Delivery2D, π0\pi_0 succeeds in fewer than 80% of episodes on 3 of 20 seeds (52%, 55% and 56%); on Arm3, on 4 of 20 (60% to 78%). On Arm3 we repeated every analysis with only the 16 seeds where π0\pi_0 reaches 80%; no result changed by more than a few hundredths, and no conclusion changed. On Delivery2D we report all 20 seeds.

References

  1. [1]G. Liu, X. Ma, Y. Yang, C. Wang, and J. Liu. FedEraser: Enabling efficient client-level data removal from federated learning models. In IEEE/ACM International Symposium on Quality of Service (IWQoS), 2021.DOI
  2. [2]N. Romandini, A. Mora, C. Mazzocca, R. Montanari, and P. Bellavista. Federated unlearning: A survey on methods, design guidelines, and evaluation metrics. IEEE Transactions on Neural Networks and Learning Systems, 2024. arXiv:2401.05146.DOI
  3. [3]Z. Liu, Y. Jiang, J. Shen, M. Peng, K.-Y. Lam, X. Yuan, and X. Liu. A survey on federated unlearning: Challenges, methods, and future directions. ACM Computing Surveys, 2024. arXiv:2310.20448.arxiv.org/abs/2310.20448
  4. [4]Y. Zhao, J. Yang, Y. Tao, L. Wang, X. Li, and D. Niyato. Exploring federated unlearning: Review, comparison, and insights. arXiv preprint arXiv:2310.19218, 2024.arxiv.org/abs/2310.19218
  5. [5]L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot. Machine unlearning. In IEEE Symposium on Security and Privacy, 2021.
  6. [6]J. Li, Y. Zhang, Y. Fei, K. Dong, X. Zhu, H. Liu, and J. Tao. Rethinking demonstration unlearning in imitation learning for robotics. arXiv preprint arXiv:2608.20784, 2026.arxiv.org/abs/2608.20784
  7. [7]R. Ranjan and A. Polyzou. VLA-Forget: Vision-language-action unlearning for embodied foundation models. arXiv preprint arXiv:2604.03956, 2026. Also in the KnowFM 2026 workshop proceedings.arxiv.org/abs/2604.03956
  8. [8]C. Gong, K. Li, J. Yao, and T. Wang. TrajDeleter: Enabling trajectory forgetting in offline reinforcement learning agents. In Network and Distributed System Security Symposium (NDSS), 2025.arxiv.org/abs/2404.12530
  9. [9]H. Wang, J. Lei, D. Li, B. Liu, M. Zheng, M. Li, R. Zhang, and Z. Fan. Behavior uncloning: Distilling mode redirection into policy weights without inference-time steering. arXiv preprint arXiv:2606.29201, 2026.arxiv.org/abs/2606.29201
  10. [10]M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. In IEEE International Conference on Robotics and Automation (ICRA), 2019.arxiv.org/abs/1810.02890
  11. [11]A. D. Dragan and S. S. Srinivasa. A policy-blending formalism for shared control. The International Journal of Robotics Research, 32(7):790–805, 2013.DOI
  12. [12]C. Agia, R. Sinha, J. Yang, R. Antonova, M. Pavone, H. Nishimura, M. Itkina, and J. Bohg. CUPID: Curating data your robot loves with influence functions. In Conference on Robot Learning (CoRL), PMLR 305, 2025.arxiv.org/abs/2506.19121
  13. [13]H. Lee, T. Min, J. Kim, S. Kang, F. Liu, L. Pinto, and K. Lee. Quality over quantity: Demonstration curation via influence functions for data-centric robot learning. arXiv preprint arXiv:2603.09056, 2026.arxiv.org/abs/2603.09056
  14. [14]T. Kim, G. Lee, M. Chang, S. Lim, and J. Choi. Deployable human preference alignment in robotics: Learning representative rewards from diverse human preferences. arXiv preprint arXiv:2607.12466, 2026.arxiv.org/abs/2607.12466
  15. [15]A. Civit, A. Andriella, C. Sierra, and G. Alenyà. Multi-user personalisation in human-robot interaction: Resolving preference conflicts using gradual argumentation. arXiv preprint arXiv:2511.03576, 2025.arxiv.org/abs/2511.03576
  16. [16]S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium (CSF), 2018.arxiv.org/abs/1709.01604

Paper details

vlnc:W_cbaxf8nt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents