betaFeedback

WorksComputer Science

Revisiting Critic Estimation in Soft Actor-Critic Algorithms

Abstract

Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and λλ-return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler λλ-return critic. What does improve performance, most clearly on continuous control, is training a λλ-return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft λλ-return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms.

Introduction

Soft Actor-Critic [18] is one of the most widely adopted algorithms in deep reinforcement learning. Its entropy regularization sustains exploration [18], improves robustness [12], and yields a unique optimal policy, unlike the unregularized objective [23]. Modern implementations [35] (Bu et al. 2024) follow Haarnoja et al. [19] in learning one or two soft Q-functions from one-step temporal-difference errors. This design is a convention rather than a requirement: SAC is defined by its actor-critic structure and entropy regularization, not by a particular critic. This leaves considerable freedom in how the critic is learned – freedom that remains largely unexplored.

We study three dimensions of this freedom, which together span the grid of critic variants in Figure 1: (i) learning soft advantages instead of soft Q-values; (ii) the rollout depth of the target, through soft λ\lambda-returns; and (iii) whether multi-step targets are computed from on-policy rollouts or replayed from a

Figure 1. Design choices and resulting variants. The critic is trained on replayed or on-policy data, with one-step or multi-step λ\lambda-return targets, and predicts soft Q-values QαπQ^\pi_\alpha or soft advantages AαπA^\pi_\alpha. SAC occupies the one-step, replay-based Q-value cell; the other five cells are the variants we study.

buffer. Since all variants share the policy update, the implementation and the evaluation protocol, differences in outcome can be attributed to the critic.

In A3C (Mnih et al. [29]) and Proximal Policy Optimization (PPO; Schulman et al. [37]), advantages reduce the variance of score-function policy gradients. This role does not carry over to SAC, whose policy gradient is unaffected by state-dependent shifts of the critic (§ 2). Advantages can nevertheless be numerically favorable: SAC’s policy depends only on the differences between actions, which can be small relative to the value itself and hard to resolve with an approximate Q-function (Tallec et al. [42]), whereas an advantage critic learns them directly ([6, 47]). We test whether this makes SAC more robust to its hyperparameters, learning soft advantages with an explicit advantage loss (SAC-A) or jointly with a soft state value in a dueling architecture (SAC-DA); all other components are inherited from SAC.

The rollout depth of the target trades off bias and variance in the critic’s learning signal. Generalized Advantage Estimation (GAE; Schulman et al. [36]) interpolates between one-step bootstrapped targets and Monte Carlo returns, and PQN (Gallici et al. [15]) recently carried the same idea over to Q-learning, replacing the replay buffer with many parallel environments. Following the idea of PQN, we drop the replay buffer, and train a soft Q-function on soft λ\lambda-returns from parallel on-policy rollouts (SAC-λ\lambda). Combining both axes, we also implement a variant in which we estimate soft advantages using GAE (SAC-GAE).

Multi-step targets are harder to combine with replay, since a replayed trajectory was generated by an older policy and biases the target. Many methods correct for this with importance weights or truncated traces (Espeholt et al. [11]), including concurrent work on SAC (Łyskawa et al. [55]), yet uncorrected multi-step targets from a buffer have been demonstrated to work well in value-based methods ([20, 13, 9]) and in actor-critic continuous control (Kozuno et al. [24]). For the entropy-regularized target, the picture is less settled: Kozuno et al. [24] found its per-step entropy terms destabilizing and kept only the final one. We train both multi-step critics on replayed segments with the full soft λ\lambda-return and no correction (SAC-λ\lambda-R, SAC-GAE-R), and compare them with their on-policy counterparts to see whether the bias outweighs the benefit of reusing data.

All of these choices solely concern critic learning and fit into existing SAC implementations with little overhead. We present them for reinforcement learning with a general convex regularizer, of which SAC’s entropy bonus is the most prominent instance, and then measure what each buys in SAC across an entire random hyperparameter search on fourteen tasks from three benchmark suites. Soft advantages improve on soft Q-values neither in return nor in robustness, and the GAE critic adds nothing over the λ\lambda-return critic. In continuous control, multi-step gains come with replay rather than depth: within our sampled ranges, neither a larger λ\lambda nor a longer horizon helps, whereas training on replayed segments does, despite the uncorrected bias. This extends earlier evidence for uncorrected multi-step targets under replay to the full soft λ\lambda-return. In practice, the critic parameterization can be chosen on grounds other than return, and in SAC, too, multi-step targets can be paired with replay without an off-policy correction.

Preliminaries

We assume a standard Markov Decision Process (MDP) defined by a tuple M=(S,A,p0,p,γ)\mathcal{M} = (\mathcal{S}, \mathcal{A}, p_0, p, \gamma). Here, S\mathcal{S} is the state space, A\mathcal{A} is the action space, p0∈Δ(S)p_0 \in\Delta(\mathcal{S}) is the initial state distribution, p(⋅∣s,a)∈Δ(S×R)p(\cdot\mid s,a) \in\Delta(\mathcal{S} \times\mathbb{R}) is the probabilistic transition and reward function, and γ∈[0,1)\gamma\in[0,1) is the discount factor. We usually abbreviate the reward by r(s,a)=Es′,r∼p(s′,r∣s,a)[r]r(s,a) = \mathbb{E}_{s',r \sim p(s',r \mid s,a)}[r].

The behavior of an agent interacting with this MDP is defined by a policy, that is, a mapping π:S→Δ(A)\pi: \mathcal{S} \to\Delta(\mathcal{A}) assigning a distribution over actions to each state. Each policy induces a distribution pπ(τ)p_\pi(\tau) on the set of trajectories τ=(s0,a0,s1,…)\tau= (s_0, a_0, s_1, \ldots) (Agarwal et al. [2]). The value function is defined as Vπ(s)=Eτ∼pπ(τ)∣s0=s[∑t=0∞γtr(st,at)]V^\pi(s) = \mathbb{E}_{\tau\sim p_\pi(\tau) \mid s_0=s}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t)\right], resulting in the RL objective π∗(⋅∣s)=arg⁡max⁡π∈ΠVπ(s)\pi^*(\cdot\mid s) = \arg\max_{\pi\in\Pi} V^\pi(s).

Regularized MDP. A natural extension of the RL objective adds a regularization term at the reward level, also referred to as MDP regularization (Ziebart [54]; ); Lee et al. [27]). More precisely, given a convex function h:Δ(A)→Rh : \Delta(\mathcal{A}) \to\mathbb{R}, the corresponding regularized value function for some regularization weight (temperature) α≥0\alpha\ge0 is defined as Vαπ(s):=Eτ∼pπ(τ)∣s0=s[∑t=0∞γt{r(st,at)−αh(π(⋅∣st))}]V^\pi_\alpha(s) := \mathbb{E}_{\tau\sim p_\pi(\tau)|s_0=s}\left[\sum_{t=0}^{\infty}\gamma^t\{r(s_t,a_t)-\alpha h(\pi(\cdot|s_t))\}\right] [30, 25]. The resulting regularized MDP objective can be expressed as π∗(⋅∣s)=arg⁡max⁡π∈ΠVαπ(s)\pi^*(\cdot|s)=\arg\max_{\pi\in\Pi}V^\pi_\alpha(s).

Following the convention of prior literature [18, 28, 51], we define the regularized Q-function as Qαπ(s,a):=r(s,a)+γEs′∼p(⋅∣s,a)[Vαπ(s′)]Q^\pi_\alpha(s,a):=r(s,a)+\gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V^\pi_\alpha(s')]. This entails the relationship Vαπ(s)=Ea∼π(⋅∣s)[Qαπ(s,a)−αh(π(⋅∣s))]V^\pi_\alpha(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}[Q^\pi_\alpha(s,a)-\alpha h(\pi(\cdot|s))] between the two soft functions (e.g. Zhan et al. [51]), and gives rise to the soft Bellman Operator [18]:

TαπQ(s,a)=r(s,a)+γEs′∼p(⋅∣s,a),a′∼π(⋅∣s′)[Q(s′,a′)−αh(π(⋅∣s′))].(1)\mathcal{T}^\pi_\alpha Q(s,a)=r(s,a)+\gamma\mathbb{E}_{s'\sim p(\cdot|s,a),a'\sim\pi(\cdot|s')}[Q(s',a')-\alpha h(\pi(\cdot|s'))]. \tag*{(1)}

Throughout, we fix hh to the negative Shannon entropy, h(p)=Ex∼p[log⁡(p(x))]=−H(p)h(p)=\mathbb{E}_{x\sim p}[\log(p(x))]=-\mathcal{H}(p) for p∈Δ(A)p\in\Delta(\mathcal{A}), which biases the search towards more exploratory policies and forms the basis of the SAC algorithm [54, 17, 18].

Soft action-values and soft advantages

Advantages measure how much better an action is than the policy’s average in a given state, separating the action-dependent part of the value from its state-dependent level. For soft Q-functions, the difference Qαπ−VαπQ^\pi_\alpha-V^\pi_\alpha does not generally vanish in expectation, so we follow prior work [28, 8] and define the soft advantages as

Aαπ(s,a):=Qαπ(s,a)−Ea∼π(⋅∣s)[Qαπ(s,a)].(2)A^\pi_\alpha(s,a):=Q^\pi_\alpha(s,a)-\mathbb{E}_{a\sim\pi(\cdot|s)}[Q^\pi_\alpha(s,a)]. \tag*{(2)}

For α=0\alpha=0 this recovers the standard advantage, since then Vπ(s)=Ea∼π(⋅∣s)[Qπ(s,a)]V^\pi(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}[Q^\pi(s,a)]. As QαπQ^\pi_\alpha and AαπA^\pi_\alpha differ only by a state-dependent term, soft advantages can replace the soft Q-function in the SAC policy update:

πk+1(⋅∣s)=arg⁡min⁡π∈ΠEa∼π(⋅∣s)[−Qαπk(s,a)]+αh(π(⋅∣s))=arg⁡min⁡π∈ΠEa∼π(⋅∣s)[−Aαπk(s,a)]+αh(π(⋅∣s)).(3)\begin{aligned} \pi_{k+1}(\cdot|s)&=\arg\min_{\pi\in\Pi}\mathbb{E}_{a\sim\pi(\cdot|s)}[-Q^{\pi_k}_\alpha(s,a)]+\alpha h(\pi(\cdot|s))\\ &=\arg\min_{\pi\in\Pi}\mathbb{E}_{a\sim\pi(\cdot|s)}[-A^{\pi_k}_\alpha(s,a)]+\alpha h(\pi(\cdot|s)). \tag*{(3)} \end{aligned}

The two forms yield identical gradients, exactly for discrete actions and for every sample of SAC’s reparameterized estimator, so a Q-critic and an advantage critic can differ only in how well they are learned. We will use the following properties of soft advantages.

Lemma 2.1. The soft advantages AαπA^\pi_\alpha satisfy the following properties for any (s,a)∈S×A(s,a)\in\mathcal{S}\times\mathcal{A}, α≥0\alpha\ge0:

(i) The soft advantages can be obtained from the soft value function as Aαπ(s,a)=r(s,a)−αh(π(⋅∣s))+γEs′∼p(⋅∣s,a)[Vαπ(s′)]−Vαπ(s)A^\pi_\alpha(s,a)=r(s,a)-\alpha h(\pi(\cdot|s))+\gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V^\pi_\alpha(s')]-V^\pi_\alpha(s),

(ii) The soft advantages vanish in expectation, i.e. Ea∼π(⋅∣s)[Aαπ(s,a)]=0\mathbb{E}_{a\sim\pi(\cdot|s)}[A^\pi_\alpha(s,a)]=0.

Property (i) expresses the soft advantages in terms of the soft value function and is the key ingredient for several algorithms in the next section. Property (ii) confirms that they retain only the action-dependent part of the Q-function. For completeness, we provide proofs in Appendix A.

Estimating the targets

The soft Bellman operator of Eq. (1) and Lemma 2.1 are all that is needed to train critics other than SAC’s one-step Q-function. We now formalize the variants of Figure 1, with training pipelines shown in Figure 2: after the SAC critic, we introduce one-step advantage critics, then critics trained on soft λ\lambda-returns and soft GAE, and finally their replay-based counterparts. All variants share the policy objective of Eq. 3; pseudo-code and the remaining implementation differences are given in Appendix D.

Training pipelines of the SAC variants

Figure 2. Training pipelines of the SAC variants. (a) SAC-A and SAC-DA keep SAC’s replay-based one-step pipeline but learn soft advantages. (b) SAC trains its Q-network on replayed transitions, SAC-λ\lambda on parallel on-policy rollouts, and SAC-λ\lambda-R on replayed trajectory segments. (c) SAC-GAE and SAC-GAE-R learn a soft value network, on rollouts or replayed segments, and compute soft GAEs for the policy update.

Soft Actor-Critic

Following Haarnoja et al. [19], in the now common form without a separate value network, SAC trains QθQ_\theta on the soft one-step temporal difference error of Eq. (1)

LcriticSAC(θ)=E(s,a,s′)∼D[{Qθ(s,a)−r(s,a)−γEa′∼π(⋅∣s′)[Qθˉ(s′,a′)+αγh(π(⋅∣s′))]}2].(4)L_{\mathrm{critic}}^{\mathrm{SAC}}(\theta) = \mathbb{E}_{(s,a,s')\sim\mathcal{D}}\left[\left\{Q_\theta(s,a)-r(s,a)-\gamma\mathbb{E}_{a'\sim\pi(\cdot|s')}\left[Q_{\bar{\theta}}(s',a')+\alpha\gamma h(\pi(\cdot|s'))\right]\right\}^{2}\right]. \tag*{(4)}

Here, θˉ\bar{\theta} refers to a target network, and D\mathcal{D} to a buffer of stored one-step transitions (Raffin et al., [35]; Bou et al. [7]). Throughout, the expectations over π\pi are estimated by sampling for continuous actions.

Soft Actor-Critic with Advantages

Based on Lemma 2.1 we can provide a Bellman-like equation for the soft values, and hence a loss for an approximation of the soft values VϕV_\phi:

LV(ϕ)=E(s,a,s′)∼D[{Vϕ(s)−r(s,a)+αh(π(⋅∣s))−γVϕˉ(s′)}2].(5)L_V(\phi) = \mathbb{E}_{(s,a,s')\sim\mathcal{D}}\left[\left\{V_\phi(s)-r(s,a)+\alpha h(\pi(\cdot|s))-\gamma V_{\bar{\phi}}(s')\right\}^{2}\right]. \tag*{(5)}

Using the value estimates, an advantage network AθA_\theta can be trained, based on Lemma 2.1:

LcriticA(θ)=E(s,a,s′)∼D[{Aθ(s,a)−r(s,a)+αh(π(⋅∣s))−γVϕ(s′)+Vϕ(s)}2].(6)L_{\mathrm{critic}}^{\mathrm{A}}(\theta) = \mathbb{E}_{(s,a,s')\sim\mathcal{D}}\left[\left\{A_\theta(s,a)-r(s,a)+\alpha h(\pi(\cdot|s))-\gamma V_\phi(s')+V_\phi(s)\right\}^{2}\right]. \tag*{(6)}

Since Eq. 5 uses the action stored in the buffer, under replay VϕV_\phi targets the soft value of the behavior policy rather than of π\pi, and SAC-A inherits this bias. The dueling formulation below avoids it, as its combined critic is trained on the standard soft Q-target, which is valid off-policy.

Dueling Q-Networks. Advantage and value networks can also be trained under a single objective, following the dueling architecture of Wang et al. [47], in which the two are recombined into a Q-function that is then trained on the one-step temporal difference error. Since Qαπ(s,a)=Aαπ(s,a)+Vαπ(s)+αh(π(⋅∣s))Q_\alpha^\pi(s,a)=A_\alpha^\pi(s,a)+V_\alpha^\pi(s)+\alpha h(\pi(\cdot|s)) by definition, we set

Qθ,ϕ(s,a)=Aθ(s,a)+Vϕ(s)+αh(π(⋅∣s))−κdE^[Aθ(s,⋅)],(7)Q_{\theta,\phi}(s,a)=A_\theta(s,a)+V_\phi(s)+\alpha h(\pi(\cdot|s))-\kappa_d\widehat{\mathbb{E}}[A_\theta(s,\cdot)], \tag*{(7)}

where the last term is the regularizer of Wang et al. [47]. For a finite action set we compute it exactly, E^[Aθ(s,⋅)]=1∣A∣∑a′∈AAθ(s,a′)\widehat{\mathbb{E}}[A_\theta(s,\cdot)]=\frac{1}{|\mathcal{A}|}\sum_{a'\in\mathcal{A}}A_\theta(s,a'); for continuous actions the mean is estimated from samples a′∼π(⋅∣s)a'\sim\pi(\cdot|s). The combined critic is trained with the SAC loss of Eq. (4),

LcriticDA(θ,ϕ)=E(s,a,s′)∼D[{Qθ,ϕ(s,a)−r(s,a)−γEa′∼π[Qθˉ,ϕˉ(s′,a′)+αγh(π(⋅∣s′))]}2].(8)L_{\mathrm{critic}}^{\mathrm{DA}}(\theta,\phi)=\mathbb{E}_{(s,a,s')\sim\mathcal{D}}\left[\left\{Q_{\theta,\phi}(s,a)-r(s,a)-\gamma\mathbb{E}_{a'\sim\pi}\left[Q_{\bar{\theta},\bar{\phi}}(s',a')+\alpha\gamma h(\pi(\cdot|s'))\right]\right\}^{2}\right]. \tag*{(8)}

In our experiments, κd=0\kappa_d=0 worked best, leaving the split into AθA_\theta and VϕV_\phi unidentifiable: SAC-DA thus tests the dueling parameterization of the soft Q-critic, while SAC-A learns soft advantages explicitly. By design, both variants use a single advantage and value network instead of the two SAC critics; otherwise, they train on one-step transitions from a replay buffer and drop into existing SAC implementations unchanged (Figure 2a). We refer to them as SAC-A and SAC-DA.

Soft Actor-Critic with λ\lambda-returns

The second dimension is the rollout depth of the target (Figure 1, middle row), for which we adopt the two changes that lead from DQN to PQN. First, we replace the one-step target with a truncated λ\lambda-return ([41, 48, 38]), defined through the soft Bellman operator as

Tλ,απQ:=(1−λ)∑n=1N−1λn−1(Tαπ)nQ+λN−1(Tαπ)NQ,(9)T_{\lambda,\alpha}^{\pi}Q := (1-\lambda)\sum_{n=1}^{N-1}\lambda^{n-1}(T_{\alpha}^{\pi})^{n}Q+\lambda^{N-1}(T_{\alpha}^{\pi})^{N}Q, \tag*{(9)}

where λ∈[0,1]\lambda\in[0,1], NN is the horizon, and (Tαπ)n(T_{\alpha}^{\pi})^{n} applies Eq. (1) nn times. Second, we replace the replay buffer by parallel environments (Figure 2b).

For our choice h=−Hh=-\mathcal{H}, the regularizer contributes the entropy bonus αH(π(⋅∣s))\alpha\mathcal{H}(\pi(\cdot|s)) at every visited state, and a trajectory sampled from π\pi yields an unbiased estimate of (Tαπ)nQ(st,at)(T_{\alpha}^{\pi})^{n}Q(s_t,a_t), the soft nn-step return

Rα,t(n):=∑k=0n−1γk{rt+k+γαH(π(⋅∣st+k+1))}+γnEa∼π(⋅∣st+n)[Q(st+n,a)].(10)R_{\alpha,t}^{(n)}:=\sum_{k=0}^{n-1}\gamma^{k}\{r_{t+k}+\gamma\alpha\mathcal{H}(\pi(\cdot|s_{t+k+1}))\}+\gamma^{n}\mathbb{E}_{a\sim\pi(\cdot|s_{t+n})}[Q(s_{t+n},a)]. \tag*{(10)}

The soft λ\lambda-return Rα,tλ:=(1−λ)∑n=1N−1λn−1Rα,t(n)+λN−1Rα,t(N)R_{\alpha,t}^{\lambda}:=(1-\lambda)\sum_{n=1}^{N-1}\lambda^{n-1}R_{\alpha,t}^{(n)}+\lambda^{N-1}R_{\alpha,t}^{(N)} then satisfies the recursion

Rα,tλ=rt+γ{(1−λ)Ea∼π(⋅∣st+1)[Q(st+1,a)]+αH(π(⋅∣st+1))+λRα,t+1λ}(11)R_{\alpha,t}^{\lambda}=r_t+\gamma\{(1-\lambda)\mathbb{E}_{a\sim\pi(\cdot|s_{t+1})}[Q(s_{t+1},a)]+\alpha\mathcal{H}(\pi(\cdot|s_{t+1}))+\lambda R_{\alpha,t+1}^{\lambda}\} \tag*{(11)}

with Rα,tλ=rtR_{\alpha,t}^{\lambda}=r_t if st+1s_{t+1} is terminal, which allows it to be computed efficiently backwards along a trajectory (Appendix § 8). This is the soft analogue of Peng’s Q(λ\lambda) ([33]), the target that PQN uses with a maximum and that Kozuno et al. [24] combine with SAC. The critic is trained on

Lcritic(θ)=Eτ[∑t=0∣τ∣{Qθ(st,at)−Rα,tλ}2],(12)L_{\mathrm{critic}}(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{|\tau|}\{Q_{\theta}(s_t,a_t)-R_{\alpha,t}^{\lambda}\}^{2}\right], \tag*{(12)}

with targets computed by target networks (Appendix § 11); we refer to this variant as SAC-λ\lambda.

Generalized Soft Advantage Estimation

Combining multi-step targets with soft advantages (Figure 1, middle right) allows the advantages themselves to be estimated by Generalized Advantage Estimation (GAE; Schulman et al. [36]). Formally, for λ∈(0,1]\lambda\in(0,1] given a trajectory τ=(s1,a1,…,sT)\tau=(s_1,a_1,\ldots,s_T) we estimate the advantages via

Aλ(st,at)=∑i=0T−t−1(λγ)iδ(st+i,at+i),(13)A^{\lambda}(s_t,a_t)=\sum_{i=0}^{T-t-1}(\lambda\gamma)^i\delta(s_{t+i},a_{t+i}), \tag*{(13)}

where δ(st,at)=r(st,at)+γV(st+1)−V(st)\delta(s_t,a_t)=r(s_t,a_t)+\gamma V(s_{t+1})-V(s_t). For λ=1\lambda=1 it recovers the (T−t)(T-t)-step advantage estimate of Mnih et al. [29], Aλ=1(st,at)=QT−t(st,at)−V(st)A^{\lambda=1}(s_t,a_t)=Q^{T-t}(s_t,a_t)-V(s_t). Following Choe & Kim [8] and property (i) of Lemma 2.1, this extends to soft advantages as

Aαλ(st,at):=∑i=0T−t−1(λγ)iδα(st+i,at+i),(14)A_{\alpha}^{\lambda}(s_t,a_t):=\sum_{i=0}^{T-t-1}(\lambda\gamma)^i\delta_{\alpha}(s_{t+i},a_{t+i}), \tag*{(14)}

where δα(st,at):=r(st,at)−αh(π(⋅∣st))+γVαπ(st+1)−Vαπ(st)\delta_{\alpha}(s_t,a_t):=r(s_t,a_t)-\alpha h(\pi(\cdot|s_t))+\gamma V_{\alpha}^{\pi}(s_{t+1})-V_{\alpha}^{\pi}(s_t); again, λ=1\lambda=1 recovers the analogous (T−t)(T-t)-step expression (Appendix § 8). We train a soft value network on the soft λ\lambda-returns, the multi-step counterpart of Eq. 5, use it to compute soft GAEs from sampled rollouts (Figure 2c) and refer to this variant as SAC-GAE. Since GAE provides advantages only at sampled actions, SAC-GAE updates its policy with the soft score-function gradient (Shi et al. [39]; Choe & Kim [8]) (Appendix § 11); here, unlike in SAC-A and SAC-DA, the advantage acts as a variance-reducing baseline.

Experience Replay

We consider SAC an off-policy algorithm, because its one-step target can be computed from transitions generated by any policy: the current policy only enters at the next state, through the expectation over a′∼π(⋅∣s′)a^\prime\sim\pi(\cdot\mid s^\prime). SAC-DA inherits this property; SAC-A only in part (§ 3.2). A multi-step target, in contrast, follows the actions taken along a trajectory, so without a correction such as importance weighting or trace cutting ([11, 55]), it is unbiased only for trajectories sampled from the policy being evaluated.

Replay is nonetheless attractive, since reusing data typically improves sample efficiency. Following other methods that combine uncorrected multi-step targets with a replay buffer ([20, 13, 24, 9]), we train both multi-step critics on trajectory segments drawn from a buffer and deliberately apply no correction. The result is an off-policy estimate of an on-policy target, which is biased by the age of the stored segments; for SAC-GAE-R, whose score-function update also uses the replayed actions, the same holds for the policy gradient. Whether that bias matters in practice is an empirical question we return to in the next section. These two variants, SAC-λ\lambda-R and SAC-GAE-R, complete the grid in Figure 1.

Experiments

We evaluate all variants on fourteen tasks from three suites: classic control (Acrobot, CartPole, MountainCar, Pendulum; [32]) and MinAtar (Freeway, Asterix, Breakout, SpaceInvaders; [50]), both from Gymnax ([26]), and six MuJoCo control tasks from Brax (Ant, HalfCheetah, Hopper, Humanoid, Reacher, Walker2d; [14]). Since Pendulum and Brax tasks have continuous action spaces and the remaining seven tasks discrete ones, we report the two groups separately. As external reference points we include PPO and, on discrete tasks, PQN, sampled under the same protocol. All implementations build on Jaxnasium ([34]); additional results, experimental details, and pseudo-code are given in Appendices B, C and D.

Evaluation protocol. We treat hyperparameters as part of the algorithm rather than tuning them away before reporting: a method that performs well only in a narrow region of its hyperparameter space demands more expertise to deploy on a new task than one that performs well broadly. For a fixed algorithm, let Xh,mX_{h,m} denote the normalized return after training with hyperparameters h∈Λh \in\Lambda on task m∈Mm \in\mathcal{M}. Even for fixed hh and mm it is random, through initialization, training seeds and hardware nondeterminism ([3]), and through the stochasticity of environment and policy at evaluation. We therefore treat Xh,m:Ωtrain×Ωtest→RX_{h,m} : \Omega_{\mathrm{train}} \times\Omega_{\mathrm{test}} \to\mathbb{R} as a random variable and report properties of its distribution over hh and mm. Returns are normalized per task so that zero corresponds to the lowest achievable return and one to the 95th percentile of per-run mean returns pooled over all algorithms, which ties the scale neither to a single method nor to a single noisy draw.

Configurations are drawn by Sobol sampling from a region centered on a configuration tuned once for SAC, with no prior tuning of any variant, which amounts to roughly 27,000 training runs; all sampled ranges are listed in Appendix C. Two properties of this protocol shape how the results should be read: Any statement about a distribution over hyperparameters is conditional on Λ\Lambda: widening it lowers every score, and re-centering it can reorder methods. Centering on the baseline’s optimum is the conservative choice with respect to our variants. Moreover, Λ\Lambda does not have the same dimension for every method: variants with a second critic carry additional learning rates and update counts, and are therefore sampled more sparsely at a fixed budget of HH configurations per task, which again works against them. Each metric is defined in the subsection that reports it.

Robustness to Hyperparameter Selection

We first ask how likely a randomly drawn configuration is to work at all. The tail probability P(h,m)∼U(Λ,M)(Xh,m>τ)\mathbb{P}_{(h,m)\sim\mathcal{U}(\Lambda,\mathcal{M})}(X_{h,m}>\tau) is the probability that an algorithm reaches performance level τ\tau on a randomly drawn task under uniformly drawn hyperparameters. We estimate it with an extended version of the score distribution of Agarwal et al. [3],

F^X(τ):=1M∑m=1M1H∑h=1H1S∑s=1S1S′∑s′=1S′1{xh,m,s,s′>τ},(15)\widehat{F}_X(\tau) := \frac{1}{M}\sum_{m=1}^{M}\frac{1}{H}\sum_{h=1}^{H}\frac{1}{S}\sum_{s=1}^{S}\frac{1}{S'}\sum_{s'=1}^{S'}\mathbf{1}_{\{x_{h,m,s,s'}>\tau\}}, \tag*{(15)}

where xh,m,s,s′x_{h,m,s,s'} is the normalized return of configuration hh on task mm under training seed ss and evaluation seed s′s', and MM, HH, SS, and S′S' denote the numbers of tasks, configurations per task, training seeds per configuration, and post-training evaluations per training seed.

In discrete-action tasks (Figure 3a), no SAC variant rises clearly above SAC, while the two on-policy multi-step variants, SAC-λ\lambda and SAC-GAE, fall clearly below it. On continuous-action tasks (Figure 3b), SAC-λ\lambda-R dominates every other method at every threshold and SAC-λ\lambda is second, both holding a larger share of workable configurations than SAC, the advantage variants and PPO. Two contrasts within the figure already localize the effect. Exchanging soft QQ-values for soft advantages moves nothing: SAC-A and SAC-DA stay close to SAC throughout. Adding a replay buffer moves a great deal: SAC-λ\lambda-R and SAC-GAE-R sit above SAC-λ\lambda and SAC-GAE across the whole range, without any importance weighting or trace cutting of the replayed targets.

Performance profiles and peak performance

Figure 3. Performance profiles and peak performance. (a) & (b) Performance profiles: the share of configurations reaching normalized score τ\tau (Eq. (15)). (c) & (d) Peak score against sensitivity Φ\Phi; dashed lines mark constant relative sensitivity Φ/Peak\Phi/\mathrm{Peak}, and the lower right corner is best. Bands and bars are 95% stratified bootstrap confidence intervals. PQN is only defined for discrete action spaces and is omitted for continuous action spaces.

Peak vs. Sensitivity

A high chance of drawing a well-performing configuration does not need to come with a high ceiling. Peak performance measures the best score attainable per task,

Peak⁡m(X)=max⁡hxh,m,Peak⁡(X)=1M∑m=1MPeak⁡m(X),(16)\operatorname{Peak}_m(X)=\max_h x_{h,m},\qquad\operatorname{Peak}(X)=\frac{1}{M}\sum_{m=1}^{M}\operatorname{Peak}_m(X), \tag*{(16)}

with xh,m=1S∑s=1S1S′∑s′=1S′xh,m,s,s′x_{h,m}=\frac{1}{S}\sum_{s=1}^{S}\frac{1}{S'}\sum_{s'=1}^{S'}x_{h,m,s,s'} the empirical mean performance of a configuration on a task. Since peak performance permits per-task tuning, it says little about how well a configuration generalizes across tasks. We therefore pair it with the sensitivity ([1]), the performance lost when a single configuration must serve all tasks,

Φ(X):=Peak⁡(X)−max⁡h1M∑m=1Mxh,m.(17)\Phi(X):=\operatorname{Peak}(X)-\max_h\frac{1}{M}\sum_{m=1}^{M}x_{h,m}. \tag*{(17)}

In discrete tasks (Figure 3c), peaks cluster near 1.1 for SAC, SAC-A, SAC-DA and SAC-λ\lambda-R, with relative sensitivities of roughly 20–30%. The on-policy multi-step variants lose about a third of that peak, and the GAE variants trade peak for a modest reduction in sensitivity without reaching PPO.

Four-panel plots showing normalized score for SAC variants across discrete and continuous tasks and target horizons

Figure 4. Hyperparameter transfer performance and effect of the horizon. (a) & (b) Best per-task configuration against the best configuration transferred from all other tasks. (c) & (d) Normalized score against target horizon NN. The on-policy variants (solid) are sampled at N≤32N \le32, the replay-based ones (dashed) at N≥32N \ge32. Bands and bars are 95% stratified bootstrap confidence intervals.

In continuous tasks (Figure 3d) the picture is clearer. SAC-λ\lambda-R reaches a clearly higher peak than SAC at roughly the same absolute sensitivity. Because Φ\Phi is an absolute quantity, it flatters methods whose ceiling is low to begin with, so we read it relative to the peak: here SAC-λ\lambda-R leads with 0.138, against 0.175 for SAC and 0.379 for PPO, and is thus at once the strongest variant and among the cheapest to tune. It ranks first in peak score on five of the seven continuous tasks (Figure 6) and leads at every random-search budget (Figure 7). SAC-GAE-R attains the second-highest peak, far above SAC-GAE, though at a higher sensitivity; replacing on-policy rollouts by replayed segments thus lifts both target types. The advantage variants do not improve on SAC here either: SAC-A, which by design has a single critic, reaches a lower peak, and SAC-DA a similar peak at a higher sensitivity.

Figure 6. Per-task ranks (1 = best). Left: by peak score max⁡hxh,m\max_h x_{h,m}; right: by IQM⁡h(xh,m)\operatorname{IQM}_h(x_{h,m}). PQN is not defined for continuous action spaces.

Figure 7. Random-search budget curves. Expected normalized score of the best shared configuration among nn sampled ones. Bands are 95% bootstrap confidence intervals.

Hyperparameter Transfer Performance

Sensitivity charges one configuration with serving every task at once. A milder question is what a configuration tuned on other tasks is worth on an unseen one. Let h−m=arg max⁡h1M−1∑m′≠mxh,m′h_{-m} = \operatorname*{arg\,max}_{h} \frac{1}{M-1}\sum_{m'\ne m} x_{h,m'} be the best shared configuration over all tasks except mm; its score on mm defines the transfer performance

Tm(X)=xh−m,m,T(X)=1M∑m=1MTm(X).(18)T_m(X) = x_{h_{-m},m}, \qquad T(X) = \frac{1}{M}\sum_{m=1}^{M} T_m(X). \tag*{(18)}

fig:4a plots transfer against peak score. On discrete tasks, SAC-A and SAC-DA match SAC on both bars, and the replay-based variants retain a larger share of their peak than the one-step variants, even where that peak is lower, as for SAC-GAE-R; given the intervals, however, no variant separates from SAC. On continuous tasks (Fig. 4b), SAC-λ\lambda-R leads on both bars and retains about 75% of its peak after transfer, followed by SAC-λ\lambda, whereas PPO loses more than half of its peak.

A closer look at the multi-step variants

In continuous tasks, the λ\lambda-return critics, and SAC-λ\lambda-R in particular, are the clearest improvement over SAC. Three ingredients set them apart: the λ\lambda-weighting of the target, its horizon, and, for SAC-λ\lambda-R and SAC-GAE-R, the replay buffer. All three vary within our search (Appendix § 10), so each can be examined without further runs, within the sampled ranges.

More weight on long returns does not help. Since λ\lambda is sampled independently of every other hyperparameter, its effect can be read off directly. Performance is flat or mildly decreasing in λ\lambda on both splits: of the sixteen slope estimates in Appendix B.5, two exclude zero, and both are negative.

Nor does a longer horizon. The on-policy variants use rollouts of 8 to 32 steps and the replay-based ones replay segments of 32 to 256, so an aggregate comparison between the two families conflates replay with a horizon up to eight times longer. Within each family, the score does not show a consistent increase with the horizon NN on either split (Fig. 4c,d).

Replay does, despite the uncorrected bias. At N=32N = 32, the one horizon both families share, the replay-based variants lead for both the λ\lambda-return and the GAE critic and on both splits (Fig. 4c,d), so the gain is not due to the horizon. Across all fourteen tasks, a randomly drawn run of a replay-based variant outperforms one of its on-policy counterparts for both critics (see also Fig. 8). This gain persists despite the uncorrected bias of the replayed targets. Since the replay-based variants also use larger batches and their own update schedule, this comparison bounds the combined effect of replay rather than isolating the bias.

These results neither make the multi-step target dispensable nor isolate its contribution. Both the λ\lambda and the horizon ranges begin well above the one-step limit, so they show only that additional depth beyond them buys nothing. Relative to the one-step SAC, SAC-λ\lambda-R is clearly ahead on continuous tasks (Figure 3b,Figure 3d), but the two differ in more than their target, since the replay-based λ\lambda-return critic is trained on sampled segments, with larger batches and its own update schedule. We therefore attribute the gain of SAC-λ\lambda-R on continuous control to the combination it implements, λ\lambda-return targets computed on replayed segments, rather than to any single ingredient. What our data does rule out is that this gain depends on a large λ\lambda, a long horizon, or an off-policy correction.

Related Work

SAC as an instance of Policy Mirror Descent. Policy Mirror Descent (PMD; Geist et al. [16]; Vieillard et al. [46]; Xiao [49]; Vaswani et al. [45]; Lan [25]; Zhan et al. [51]) has emerged as a unifying framework for common RL algorithms. The PMD policy update (e.g. Lan [25], Algorithm 1)

πk+1(⋅∣s)=arg⁡min⁡π∈Π{Ea∼π(⋅∣s)[−Qπkα(s,a)]+αh(π(⋅∣s))+ηkBω(π,πk;s)}(19)\pi_{k+1}(\cdot\mid s) = \arg\min_{\pi\in\Pi} \left\{ \mathbb{E}_{a \sim\pi(\cdot\mid s)}[-Q_{\pi_k}^{\alpha}(s,a)] + \alpha h(\pi(\cdot\mid s)) + \eta_k B_{\omega}(\pi,\pi_k;s) \right\} \tag*{(19)}

where ηk>0\eta_k > 0 is a sequence of step sizes, hh some convex function and BωB_{\omega} some Bregman Divergence, which acts as a trust-region term that prevents large deviations from the previous policy. SAC is thus an instance of PMD, with hh the negative Shannon entropy and no trust-region term, and the design choices studied here should transfer to other PMD algorithms, including those built on more general convex regularizers (Lee et al., [27]; Alfano et al., [4]). They are also complementary to efforts that add trust-region terms to SAC (Tomar et al., [44]; Neumann et al., [31]).

Multi-step returns & advantages in maximum-entropy RL. Soft advantage estimators are not new: Huang et al. [22] introduced a variant of GAE for entropy-regularized objectives, and Choe & Kim [8] use a closely related soft GAE in PPO-style algorithms, whereas we integrate soft advantages into SAC. Multi-step targets have likewise been studied in entropy-regularized settings (Shil et al., [40]), including for transformer-based policies (Tian et al., [43]); Huang et al. [21] and Lyskawa et al. (2025) train SAC on replayed λ\lambda- and nn-step returns, corrected by importance sampling and truncated importance sampling, respectively. Kozuno et al. [24] show that Peng’s Q(λ)Q(\lambda), which applies no correction, often outperforms conservative operators in continuous control, and characterize when its bias is benign; for SAC, however, they drop the entropy bonus from all but the last step of the target for stability. Our results add to these findings: the full soft λ\lambda-return, entropy terms included and with a learned temperature, trains stably from replay across a distribution of hyperparameters.

More broadly, recent work revisits core design assumptions in SAC, particularly regarding critic learning and target construction (Shil et al., [40]; Zhou et al., [53]; Asad et al., [5]); we complement it with a systematic study of soft advantages, multi-step targets and replay within a unified SAC framework.

Conclusion

We asked which choices in SAC’s critic influence its outcomes by keeping the policy objective fixed and varying three of these choices: the learned quantity, the rollout depth of the target, and experience replay. In total, we evaluated across six critic variants over an entire random hyperparameter search on fourteen tasks. Learning soft advantages instead of soft Q-values neither makes SAC more robust to its hyperparameters nor improves its performance: the advantage and dueling critics stay close to SAC in the share of well-performing configurations and in transfer, and on continuous tasks SAC-A even reaches a lower peak. Since SAC’s policy gradient is unaffected by state-dependent shifts of the critic, this is a finding about critic learning: in our setting, fitting the state-dependent level of the value is not what limits SAC’s critic. The soft GAE critic, in turn, adds nothing over the simpler λ\lambda-return critic.

The clearest gain comes from training a λ\lambda-return critic on replayed segments. In continuous tasks, SAC-λ\lambda-R holds the largest share of well-performing configurations, reaches the highest peak score at a lower relative sensitivity than SAC, and keeps most of that peak under transfer across tasks. Its lead over the on-policy SAC-λ\lambda persists at a matched horizon, while neither a larger λ\lambda nor a longer horizon helps within the sampled ranges; on discrete tasks, no variant improves on SAC. In line with earlier evidence for uncorrected multi-step targets, the missing off-policy correction does not prevent these gains, and our results extend this observation to the full soft λ\lambda-return. In practice, SAC’s critic parametrization can thus be chosen on grounds other than return, and multi-step targets can be paired with replay without correction.

Several limitations bound these conclusions. Our continuous-control evidence rests on a limited number of continuous action tasks, and given SAC’s prevalence in that domain, further benchmarks are warranted. The replay comparison bounds the combined effect of target staleness, larger batches and a different update schedule rather than isolating the bias, and since SAC-λ\lambda-R also differs from SAC in its training regime, we attribute its gain to the combination rather than to the multi-step target alone. Any statement about a distribution over hyperparameters is conditional on the search space, which we centered on a configuration tuned for SAC. Finally, we studied the variants in isolation from other recent SAC extensions to attribute effects cleanly; combining them remains open.

Acknowledgments

This work was financially supported by Shell Information Technology International Limited and the Netherlands Enterprise Agency under the grant PPS23-3-03529461.

Proofs

Soft Advantages

As highlighted in the related work, we can see the SAC policy update as an instance of Policy Mirror Descent [51]. In fact, for all instances of PMD, the (soft) action-value function QαπQ_{\alpha}^{\pi} can be replaced by the corresponding (soft) advantage function.

πk+1(⋅∣s)=arg⁡min⁡π∈Π {Ea∼π(⋅∣s)[−Qαπk(s,a)]+αh(π(⋅∣s))+ηkBω(π,πk;s)}=arg⁡min⁡π∈Π {Ea∼π(⋅∣s)[−Aαπk(s,a)]−Ea∼πk(⋅∣s)[Qαπk(s,a)]+αh(π(⋅∣s))+ηkBω(π,πk;s)}.(20)\begin{aligned} \pi_{k+1}(\cdot|s) &= \underset{\pi\in\Pi}{\arg\min}\ \left\{\mathbb{E}_{a\sim\pi(\cdot|s)}[-Q_{\alpha}^{\pi_k}(s,a)] + \alpha h(\pi(\cdot|s)) + \eta_k B_{\omega}(\pi,\pi_k;s)\right\} \\ &= \underset{\pi\in\Pi}{\arg\min}\ \left\{\mathbb{E}_{a\sim\pi(\cdot|s)}[-A_{\alpha}^{\pi_k}(s,a)] - \mathbb{E}_{a\sim\pi_k(\cdot|s)}[Q_{\alpha}^{\pi_k}(s,a)] + \alpha h(\pi(\cdot|s)) + \eta_k B_{\omega}(\pi,\pi_k;s)\right\}. \tag*{(20)} \end{aligned}

The term Ea∼πk(⋅∣s)[Qαπk(s,a)]\mathbb{E}_{a\sim\pi_k(\cdot|s)}[Q_{\alpha}^{\pi_k}(s,a)] in the final expression can be dropped because it does not depend on π\pi. This holds for any convex hh and is not limited to the negative Shannon entropy.

Proof of Lemma 2.1. We prove the two properties of Lemma 2.1 together with a third, which relates soft and unregularized advantages.

Lemma A.1 (Lemma 2.1, extended). The soft advantages AαπA_{\alpha}^{\pi} satisfy the following properties for any (s,a)∈S×A(s,a) \in\mathcal{S} \times\mathcal{A}, α≥0\alpha\ge0:

(i) Aαπ(s,a)=r(s,a)−αh(π(⋅∣s))+γEs′∼p(⋅∣s,a)[Vαπ(s′)]−Vαπ(s)A_{\alpha}^{\pi}(s,a) = r(s,a) - \alpha h(\pi(\cdot|s)) + \gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V_{\alpha}^{\pi}(s')] - V_{\alpha}^{\pi}(s),

(ii) Ea∼π(⋅∣s)[Aαπ(s,a)]=0\mathbb{E}_{a\sim\pi(\cdot|s)}[A_{\alpha}^{\pi}(s,a)] = 0,

(iii) Aαπ(s,a)=Aπ(s,a)+αH(π∣s)−αH(π∣s,a)A_{\alpha}^{\pi}(s,a) = A^{\pi}(s,a) + \alpha H(\pi|s) - \alpha H(\pi|s,a), where H(π∣s,a)H(\pi|s,a) and H(π∣s)H(\pi|s) denote the cumulative discounted negative entropy starting with state-action pair s,as,a and in state ss, respectively:

H(π∣s,a):=Eτ∼pπ∣s0=s,a0=a[∑t≥0γth(π(⋅∣st))]H(\pi|s,a) := \mathbb{E}_{\tau\sim p_{\pi}|s_0=s,a_0=a}\left[\sum_{t\ge0}\gamma^t h(\pi(\cdot|s_t))\right]
H(π∣s):=Ea∼π(⋅∣s)[H(π∣s,a)]H(\pi|s) := \mathbb{E}_{a\sim\pi(\cdot|s)}[H(\pi|s,a)]

Proof. Recall that the soft Q-function is defined as

Qαπ(s,a)=r(s,a)+γEs′∼p(⋅∣s,a)[Vαπ(s′)].(21)Q_{\alpha}^{\pi}(s,a) = r(s,a) + \gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V_{\alpha}^{\pi}(s')]. \tag*{(21)}

Additionally, we can re-express the relation between QαπQ_{\alpha}^{\pi} and VαπV_{\alpha}^{\pi} as

Ea∼π(⋅∣s)[Qαπ(s,a)]=Vαπ(s)+αh(π(⋅∣s)).(22)\mathbb{E}_{a\sim\pi(\cdot|s)}[Q_{\alpha}^{\pi}(s,a)] = V_{\alpha}^{\pi}(s) + \alpha h(\pi(\cdot|s)). \tag*{(22)}

(i) Now, we can insert the relations of Eqs. (21) and (22) into the definition of the soft advantages

Aαπ(s,a)=Qαπ(s,a)−Ea∼π(⋅∣s)[Qαπ(s,a)]=r(s,a)+γEs′∼p(⋅∣s,a)[Vαπ(s′)]−Vαπ(s)−αh(π(⋅∣s)).(23)\begin{aligned} A_{\alpha}^{\pi}(s,a) &= Q_{\alpha}^{\pi}(s,a) - \mathbb{E}_{a\sim\pi(\cdot|s)}[Q_{\alpha}^{\pi}(s,a)] \\ &= r(s,a) + \gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V_{\alpha}^{\pi}(s')] - V_{\alpha}^{\pi}(s) - \alpha h(\pi(\cdot|s)). \tag*{(23)} \end{aligned}

(ii) follows from definition as

Ea∼π(⋅∣s)[Aαπ(s,a)]=Ea∼π(⋅∣s)[Qαπ(s,a)]−Ea∼π(⋅∣s)[Qαπ(s,a)]=0.(24)\mathbb{E}_{a\sim\pi(\cdot|s)}[A_{\alpha}^{\pi}(s,a)] = \mathbb{E}_{a\sim\pi(\cdot|s)}[Q_{\alpha}^{\pi}(s,a)] - \mathbb{E}_{a\sim\pi(\cdot|s)}[Q_{\alpha}^{\pi}(s,a)] = 0. \tag*{(24)}

(iii) By definition again, we can express

Vαπ(s)=Vπ(s)−αH(π∣s).(25)V_{\alpha}^{\pi}(s) = V^{\pi}(s) - \alpha H(\pi|s). \tag*{(25)}

Inserting this into the expression obtained in (i) yields (after some reorganization of terms)

Aαπ(s,a)=r(s,a)+γEs′∼p(⋅∣s,a)[Vαπ(s′)]−Vαπ(s)−αh(π(⋅∣s))(26)A_{\alpha}^{\pi}(s,a) = r(s,a) + \gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V_{\alpha}^{\pi}(s')] - V_{\alpha}^{\pi}(s) - \alpha h(\pi(\cdot|s)) \tag*{(26)}
=r(s,a)+γEs′∼p(⋅∣s,a)[Vπ(s′)]−Vπ(s)(27)= r(s,a) + \gamma\mathbb{E}_{s'\sim p(\cdot|s,a)}[V^{\pi}(s')] - V^{\pi}(s) \tag*{(27)}
+αH(π∣s)(28)+ \alpha H(\pi|s) \tag*{(28)}
−αγEs′∼p(⋅∣s,a),a′∼π(⋅∣s′)[H(π∣s′,a′)]−αh(π(⋅∣s))(29)- \alpha\gamma\mathbb{E}_{s'\sim p(\cdot|s,a),a'\sim\pi(\cdot|s')}[H(\pi|s',a')] - \alpha h(\pi(\cdot|s)) \tag*{(29)}
=Aπ(s,a)+αH(π∣s)−αH(π∣s,a).(30)= A^{\pi}(s,a) + \alpha H(\pi|s) - \alpha H(\pi|s,a). \tag*{(30)}

Notably, we can also immediately see that if we use the soft value function as baseline subtraction, i.e. A~απ(s,a):=Qαπ(s,a)−Vαπ(s)\widetilde{A}^{\pi}_{\alpha}(s,a) := Q^{\pi}_{\alpha}(s,a) - V^{\pi}_{\alpha}(s), we obtain

Ea∼π(⋅∣s)[A~απ(s,a)]=Ea∼π(⋅∣s)[Qαπ(s,a)]−Vαπ(s)=αh(π(⋅∣s)),\mathbb{E}_{a\sim\pi(\cdot\mid s)}[\widetilde{A}^{\pi}_{\alpha}(s,a)] = \mathbb{E}_{a\sim\pi(\cdot\mid s)}[Q^{\pi}_{\alpha}(s,a)] - V^{\pi}_{\alpha}(s) = \alpha h(\pi(\cdot\mid s)),

(by Eq. (22)) which does not vanish for non-deterministic policies in the regularized case (α>0\alpha> 0).

Soft n-step returns

Though soft nn-step returns have been used before with SAC ([40, 55, 43]), we give the details behind Eqs. (10) and 11 for completeness. Throughout, we write EπQ(s):=Ea∼π(⋅∣s)[Q(s,a)]\mathbb{E}_{\pi}Q(s) := \mathbb{E}_{a\sim\pi(\cdot\mid s)}[Q(s,a)] and H(s):=H(π(⋅∣s))\mathcal{H}(s) := \mathcal{H}(\pi(\cdot\mid s)).

Claim. The nn-step soft Bellman Operator satisfies the following expression:

(Tαπ)nQ(st,at)=Eτ∣st,at[∑k=0n−1γkrt+k−α∑k=1nγklog⁡π(at+k∣st+k)+γnQ(st+n,at+n)].(T^{\pi}_{\alpha})^{n}Q(s_t,a_t) = \mathbb{E}_{\tau\mid s_t,a_t}\left[\sum_{k=0}^{n-1}\gamma^k r_{t+k} - \alpha\sum_{k=1}^{n}\gamma^k\log\pi(a_{t+k}\mid s_{t+k}) + \gamma^n Q(s_{t+n},a_{t+n})\right].

Proof. Let π\pi be a fixed policy and define the soft Bellman operator

(TαπQ)(s,a)=r(s,a)+γ(Pπ(Q−αlog⁡π))(s,a),(T^{\pi}_{\alpha}Q)(s,a) = r(s,a) + \gamma(P^{\pi}(Q-\alpha\log\pi))(s,a),

where (Pπf)(s,a):=Es′∼p(⋅∣s,a), a′∼π(⋅∣s′)[f(s′,a′)](P^{\pi}f)(s,a) := \mathbb{E}_{s'\sim p(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}[f(s',a')] and log⁡π(s,a):=log⁡π(a∣s)\log\pi(s,a) := \log\pi(a\mid s).

We proceed by induction on nn.

Base case (n=1n=1). By definition of the soft Bellman operator,

(TαπQ)(st,at)=rt+γEτ∣st,at[Q(st+1,at+1)−αlog⁡π(at+1∣st+1)],(T^{\pi}_{\alpha}Q)(s_t,a_t) = r_t + \gamma\mathbb{E}_{\tau\mid s_t,a_t}[Q(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}\mid s_{t+1})],

which coincides with the claimed expression for n=1n=1.

Inductive step. Assume the statement holds for some n≥1n\geq1, i.e.

(Tαπ)nQ(st,at)=Eτ∣st,at[∑k=0n−1γkrt+k−α∑k=1nγklog⁡π(at+k∣st+k)+γnQ(st+n,at+n)].(T^{\pi}_{\alpha})^{n}Q(s_t,a_t) = \mathbb{E}_{\tau\mid s_t,a_t}\left[\sum_{k=0}^{n-1}\gamma^k r_{t+k} - \alpha\sum_{k=1}^{n}\gamma^k\log\pi(a_{t+k}\mid s_{t+k}) + \gamma^n Q(s_{t+n},a_{t+n})\right].

Consider n+1n+1. Using the operator definition,

(Tαπ)n+1Q(st,at)=rt+γEτ∣st,at[(Tαπ)nQ(st+1,at+1)−αlog⁡π(at+1∣st+1)].(T^{\pi}_{\alpha})^{n+1}Q(s_t,a_t) = r_t + \gamma\mathbb{E}_{\tau\mid s_t,a_t}[(T^{\pi}_{\alpha})^nQ(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}\mid s_{t+1})].

Applying the induction hypothesis at time t+1t+1 and using the law of total expectation yields

(Tαπ)n+1Q(st,at)=Eτ∣st,at[rt+∑k=0n−1γk+1rt+1+k−α(γlog⁡π(at+1∣st+1)+∑k=1nγk+1log⁡π(at+1+k∣st+1+k))+γn+1Q(st+n+1,at+n+1)].\begin{aligned} (T^{\pi}_{\alpha})^{n+1}Q(s_t,a_t) ={}&\mathbb{E}_{\tau\mid s_t,a_t}\Bigg[r_t+\sum_{k=0}^{n-1}\gamma^{k+1}r_{t+1+k}\\ &\quad-\alpha\left(\gamma\log\pi(a_{t+1}\mid s_{t+1})+\sum_{k=1}^{n}\gamma^{k+1}\log\pi(a_{t+1+k}\mid s_{t+1+k})\right)\\ &\quad+\gamma^{n+1}Q(s_{t+n+1},a_{t+n+1})\Bigg]. \end{aligned}

Re-indexing the sums gives

(Tαπ)n+1Q(st,at)=Eτ∣st,at[∑k=0nγkrt+k−α∑k=1n+1γklog⁡π(at+k∣st+k)+γn+1Q(st+n+1,at+n+1)],(T^{\pi}_{\alpha})^{n+1}Q(s_t,a_t) = \mathbb{E}_{\tau\mid s_t,a_t}\left[\sum_{k=0}^{n}\gamma^k r_{t+k} - \alpha\sum_{k=1}^{n+1}\gamma^k\log\pi(a_{t+k}\mid s_{t+k}) + \gamma^{n+1}Q(s_{t+n+1},a_{t+n+1})\right],

which is exactly the desired form. This completes the induction.

Along a trajectory sampled from π\pi, at+k∼π(⋅∣st+k)a_{t+k} \sim\pi(\cdot\mid s_{t+k}), so by the tower property each term −αlog⁡π(at+k∣st+k)-\alpha\log\pi(a_{t+k}\mid s_{t+k}) can be replaced by its conditional expectation αH(st+k)\alpha\mathcal{H}(s_{t+k}), and the bootstrap Q(st+n,at+n)Q(s_{t+n},a_{t+n}) by EπQ(st+n)\mathbb{E}_{\pi}Q(s_{t+n}), without changing the expectation:

(Tαπ)nQ(st,at)=Eτ∣st,at[∑k=0n−1γkrt+k+α∑k=1nγkH(st+k)+γnEπQ(st+n)].(T_{\alpha}^{\pi})^{n}Q(s_t,a_t)=\mathbb{E}_{\tau\mid s_t,a_t}\left[\sum_{k=0}^{n-1}\gamma^k r_{t+k}+\alpha\sum_{k=1}^{n}\gamma^k\mathcal{H}(s_{t+k})+\gamma^n\mathbb{E}_{\pi}Q(s_{t+n})\right].

This motivates the soft nn-step return of Eq. (10) as an unbiased estimate of the nn-step soft Bellman operator,

Rα,t(n)=∑k=0n−1γk{rt+k+γαH(st+k+1)}+γnEπQ(st+n).(31)R_{\alpha,t}^{(n)}=\sum_{k=0}^{n-1}\gamma^k\{r_{t+k}+\gamma\alpha\mathcal{H}(s_{t+k+1})\}+\gamma^n\mathbb{E}_{\pi}Q(s_{t+n}). \tag*{(31)}

Unlike in Tian et al. [43], the entropy enters as a discounted bonus for the next state. This matches the soft Bellman operator, since Rα,t(1)=rt+γ(EπQ(st+1)+αH(st+1))R_{\alpha,t}^{(1)}=r_t+\gamma\left(\mathbb{E}_{\pi}Q(s_{t+1})+\alpha\mathcal{H}(s_{t+1})\right) is the one-step target of Eq. (1). Based on Eq. (9), we further define the λ\lambda-return for a trajectory of length NN as

Rα,tλ:=(1−λ)∑n=1N−1λn−1Rα,t(n)+λN−1Rα,t(N).R_{\alpha,t}^{\lambda}:=(1-\lambda)\sum_{n=1}^{N-1}\lambda^{n-1}R_{\alpha,t}^{(n)}+\lambda^{N-1}R_{\alpha,t}^{(N)}.

While the recursive relation in Eq. 11 was used before, e.g. by Huang et al. [21], we provide a formal proof for completeness.

Claim. The soft λ\lambda-returns satisfy the recursion

Rα,tλ={rtif st+1 is terminal,rt+γ{(1−λ)EπQ(st+1)+αH(st+1)+λRα,t+1λ}otherwise.(32)R_{\alpha,t}^{\lambda}= \begin{cases} r_t & \text{if }s_{t+1}\text{ is terminal},\\ r_t+\gamma\left\{(1-\lambda)\mathbb{E}_{\pi}Q(s_{t+1})+\alpha\mathcal{H}(s_{t+1})+\lambda R_{\alpha,t+1}^{\lambda}\right\} & \text{otherwise}. \end{cases} \tag*{(32)}

Proof. Write rtα,π:=rt+γαH(st+1)r_t^{\alpha,\pi}:=r_t+\gamma\alpha\mathcal{H}(s_{t+1}). Then Rα,t(1)=rtα,π+γEπQ(st+1)R_{\alpha,t}^{(1)}=r_t^{\alpha,\pi}+\gamma\mathbb{E}_{\pi}Q(s_{t+1}) and, separating the first term, Rα,t(n)=rtα,π+γRα,t+1(n−1)R_{\alpha,t}^{(n)}=r_t^{\alpha,\pi}+\gamma R_{\alpha,t+1}^{(n-1)} for n≥2n\ge2. Inserting both into the definition,

Rα,tλ=(1−λ)(rtα,π+γEπQ(st+1))+(1−λ)∑n=2N−1λn−1(rtα,π+γRα,t+1(n−1))+λN−1(rtα,π+γRα,t+1(N−1))=rtα,π+(1−λ)γEπQ(st+1)+γλ{(1−λ)∑n=1N−2λn−1Rα,t+1(n)+λN−2Rα,t+1(N−1)},\begin{aligned} R_{\alpha,t}^{\lambda} ={}&(1-\lambda)(r_t^{\alpha,\pi}+\gamma\mathbb{E}_{\pi}Q(s_{t+1}))+(1-\lambda)\sum_{n=2}^{N-1}\lambda^{n-1}(r_t^{\alpha,\pi}+\gamma R_{\alpha,t+1}^{(n-1)})\\ &+\lambda^{N-1}(r_t^{\alpha,\pi}+\gamma R_{\alpha,t+1}^{(N-1)}) \\ &=r_t^{\alpha,\pi}+(1-\lambda)\gamma\mathbb{E}_{\pi}Q(s_{t+1})+\gamma\lambda\left\{(1-\lambda)\sum_{n=1}^{N-2}\lambda^{n-1}R_{\alpha,t+1}^{(n)}+\lambda^{N-2}R_{\alpha,t+1}^{(N-1)}\right\}, \end{aligned}

where we grouped rtα,πr_t^{\alpha,\pi} and re-indexed the sum. The expression in braces is Rα,t+1λR_{\alpha,t+1}^{\lambda}, hence

Rα,tλ=rtα,π+(1−λ)γEπQ(st+1)+γλRα,t+1λ=rt+γ{(1−λ)EπQ(st+1)+αH(st+1)+λRα,t+1λ}.R_{\alpha,t}^{\lambda}=r_t^{\alpha,\pi}+(1-\lambda)\gamma\mathbb{E}_{\pi}Q(s_{t+1})+\gamma\lambda R_{\alpha,t+1}^{\lambda}=r_t+\gamma\left\{(1-\lambda)\mathbb{E}_{\pi}Q(s_{t+1})+\alpha\mathcal{H}(s_{t+1})+\lambda R_{\alpha,t+1}^{\lambda}\right\}.

By convention, EπQ(st+1)=0\mathbb{E}_{\pi}Q(s_{t+1})=0, Rα,t+1λ=0R_{\alpha,t+1}^{\lambda}=0 and rtα,π=rtr_t^{\alpha,\pi}=r_t if st+1s_{t+1} is terminal, which yields the stated recursion. □

Algorithm 4 computes the same recursion in the equivalent form Rα,tλ=rt+γ{(1−λ)Vˉ(st+1)+λ(Rα,t+1λ+αH(st+1))}R_{\alpha,t}^{\lambda}=r_t+\gamma\{(1-\lambda)\bar{V}(s_{t+1})+\lambda(R_{\alpha,t+1}^{\lambda}+\alpha\mathcal{H}(s_{t+1}))\}, with the soft value Vˉ(s)=EπQ(s)+αH(s)\bar{V}(s)=\mathbb{E}_{\pi}Q(s)+\alpha\mathcal{H}(s).

GAE for soft advantages

As in the unregularized case (α=0\alpha= 0), for λ=1\lambda= 1 the soft GAE recovers a (T−t)(T-t)-step lookahead minus the value of the current state:

Aαλ=1(st,at)=∑i=0T−t−1γi{r(st+i,at+i)−αh(π(⋅∣st+i))+γVα(st+i+1)−Vα(st+i)}=∑i=0T−t−1γi{r(st+i,at+i)−αh(π(⋅∣st+i))}+γT−tVα(sT)−Vα(st)(telescoping)=Rα,t(T−t)−Vα(st)−αh(π(⋅∣st)),\begin{aligned} A_{\alpha}^{\lambda=1}(s_t, a_t) &= \sum_{i=0}^{T-t-1} \gamma^i \{r(s_{t+i}, a_{t+i}) - \alpha h(\pi(\cdot\mid s_{t+i})) + \gamma V_{\alpha}(s_{t+i+1}) - V_{\alpha}(s_{t+i})\} \\ &= \sum_{i=0}^{T-t-1} \gamma^i \{r(s_{t+i}, a_{t+i}) - \alpha h(\pi(\cdot\mid s_{t+i}))\} + \gamma^{T-t}V_{\alpha}(s_T) - V_{\alpha}(s_t) \quad\text{(telescoping)} \\ &= \mathcal{R}_{\alpha,t}^{(T-t)} - V_{\alpha}(s_t) - \alpha h(\pi(\cdot\mid s_t)), \end{aligned}

where the last step holds for h=−Hh = -\mathcal{H}, with Q=QαπQ = Q_{\alpha}^{\pi} in Eq. (31) and Vα(sT)=EπQ(sT)+αH(sT)V_{\alpha}(s_T) = \mathbb{E}_{\pi}Q(s_T) + \alpha\mathcal{H}(s_T).

Additional Results

This appendix complements the main results with a variance decomposition, per-task rankings, probabilities of improvement at matched horizon, random-search budget curves, the λ\lambda ablation and per-task performance profiles.

Variance Decomposition

An interesting question is whether the design choices change where performance variance originates. Performance varies with the task, the configuration, the training seed and the evaluation episode, and we decompose the total sum of squares accordingly. With SmS_m training seeds on task mm, nm=HSmS′n_m = HS_mS', xh,m,s=1S′∑s′xh,m,s,s′x_{h,m,s} = \frac{1}{S'}\sum_{s'} x_{h,m,s,s'}, xh,mx_{h,m} as in Section 4, μm=1H∑hxh,m\mu_m = \frac{1}{H}\sum_h x_{h,m} and μ=1M∑mμm\mu= \frac{1}{M}\sum_m \mu_m,

SStask=∑mnm(μm−μ)2,SSconfig=∑mSmS′∑h(xh,m−μm)2,SSseed=∑mS′∑h,s(xh,m,s−xh,m)2,SSepisode=∑h,m,s,s′(xh,m,s,s′−xh,m,s)2.(33)\begin{aligned} SS_{\mathrm{task}} &= \sum_m n_m(\mu_m-\mu)^2, \\ SS_{\mathrm{config}} &= \sum_m S_mS'\sum_h (x_{h,m}-\mu_m)^2, \\ SS_{\mathrm{seed}} &= \sum_m S'\sum_{h,s}(x_{h,m,s}-x_{h,m})^2, \\ SS_{\mathrm{episode}} &= \sum_{h,m,s,s'}(x_{h,m,s,s'}-x_{h,m,s})^2. \tag*{(33)} \end{aligned}

and each share is SSℓ/∑ℓSSℓSS_\ell/ \sum_\ell SS_\ell.

The resulting profiles are shown in Figure 5. On continuous tasks the one-step methods draw about 75% of their variance from differences between tasks, whereas the multi-step variants draw only 24–45%, with correspondingly more coming from configuration, seed and episode noise. A smaller between-task share means performance depends less on which task happens to be drawn, which is consistent with the stronger transfer reported above. On discrete tasks the pattern reverses, and what distinguishes the multi-step variants there is instead a smaller training-noise share at comparable evaluation noise.

Variance decomposition across discrete and continuous tasks

Figure 5. Variance decomposition. Share of total variance attributable to task, configuration, training seed and evaluation episode, over all configurations (a, c) and over each method’s top-quartile configurations (b, d), on discrete (a, b) and continuous (c, d) tasks.

The comparison is biased by performance level, since a method that fails uniformly has little seed-to-seed variance left to decompose. Restricting the decomposition to each method’s top-quartile configurations (Figure 5b,Figure 5d) shrinks the configuration share and raises the seed and episode shares, as expected, but leaves the between-task trend mostly intact.

Per Task Ranking

Due to the amount of environment-algorithm combinations, most of our analysis has been focused on aggregate metrics. To complement this view Figure 6 shows the per-task ranks of all methods, by peak score max⁡hxh,m\max_h x_{h,m} (left) and by the interquartile mean over configurations, IQM⁡h(xh,m)\operatorname{IQM}_h(x_{h,m}) (right), a proxy for performance under limited tuning. By peak score, SAC-λ\lambda-R ranks first on nine of the fourteen tasks, including five of the seven continuous ones, and never below fourth. By IQM the picture is similar but less pronounced: SAC-λ\lambda-R ranks first or second on every continuous task, but drops to seventh on MountainCar.

Random-Search Budget Curve

Ranking by the IQM in Section B.2 was used as a proxy for how rankings change with limited tuning. To further investigate that effect, we use budget curves [10]: they give the expected score of the best shared configuration among nn randomly drawn ones,

B(n)=ES⊆{1,…,H}, ∣S∣=n[max⁡h∈S1M∑m=1Mxh,m].(34)B(n) = \mathbb{E}_{S \subseteq\{1,\ldots,H\},\,|S|=n}\left[\max_{h \in S} \frac{1}{M}\sum_{m=1}^{M}x_{h,m}\right]. \tag*{(34)}

On continuous tasks (Figure 7a), SAC-λ\lambda-R leads at every budget and SAC-λ\lambda is second, so their advantage does not depend on how much tuning is spent. On discrete tasks (Figure 7b), SAC, the advantage variants and the replay-based variants overlap almost fully and gain similarly from tuning, the on-policy multi-step variants stay below, and PQN gains most from additional configurations.

Probability of improvement for X=SAC-GAE-R and Y=SAC-GAE across tasks

Figure 8a. X=SAC-GAE-R, Y=SAC-GAE

Figure 8b. X=SAC-λ\lambda-R, Y=SAC-λ\lambda

fig:8: Probability of improvement at N=32N = 32 of the replay-based over the on-policy variant, per task and aggregated. (a): SAC-GAE-R vs. SAC-GAE; (b): SAC-λ\lambda-R vs. SAC-λ\lambda; the dashed line marks 0.5. Bars are 95% stratified bootstrap confidence intervals.

Replay at Matched Horizon: Probability of Improvement

To complement Figure 4c,d, we compare each replay-based variant with its on-policy counterpart at the shared horizon N=32N = 32, using the probability of improvement of Agarwal et al. [3], extended to configurations and evaluation seeds. For methods XX and YY,

P(X>Y)=1M∑m=1MPm(X>Y),P(X > Y) = \frac{1}{M} \sum_{m=1}^{M} P_m(X > Y),
Pm(X>Y)=1(HSS′)2∑h,s,s′∑h~,s~,s~′(1{xh,m,s,s′>yh~,m,s~,s~′}+121{xh,m,s,s′=yh~,m,s~,s~′}).(35)P_m(X > Y) = \frac{1}{(HSS')^2} \sum_{h,s,s'} \sum_{\tilde{h},\tilde{s},\tilde{s}'} \left( \mathbb{1}_{\{x_{h,m,s,s'} > y_{\tilde{h},m,\tilde{s},\tilde{s}' }\}} + \frac{1}{2}\mathbb{1}_{\{x_{h,m,s,s'} = y_{\tilde{h},m,\tilde{s},\tilde{s}' }\}} \right). \tag*{(35)}

where the sums run over the configurations sampled with N=32N = 32. For both replay-base variants the CI for the probability of improvement does not contain zero on either suite (Figure 8a,b), demonstrating a statistically significant improvement.

λ\lambda Ablations

λ\lambda is sampled for every multi-step variant, independently of all other hyperparameters, so the score against λ\lambda within one variant is a genuine ablation of the multi-step target: same architecture, data and update schedule, but a longer effective horizon (Figure 9). We quantify it by two weighted least-squares slopes of the task-averaged score of a configuration, 1M∑mxh,m\frac{1}{M}\sum_m x_{h,m}, in λ\lambda (Table 1). The marginal slope regresses on λ\lambda alone and thus averages over all other hyperparameters; the partial slope is the λ\lambda coefficient in a regression on all hyperparameters sampled for the method, with learning rates, sizes and counts on a log scale. Since λ\lambda is sampled independently of the other hyperparameters, both estimate the same effect, but the partial slope is more precise, as the controls absorb variance that λ\lambda does not explain. Of the sixteen estimates, only two exclude zero, and both are negative: within the sampled ranges, placing more weight on long-range returns does not improve performance.

Figure 9. Effect of the sampled λ\lambda. normalized score against λ\lambda, binned at bin centers, for every method that samples it. The λ\lambda-return critics are sampled on [0.50, 0.95] and the GAE critics on [0.85, 0.99], so the two families occupy different parts of the axis and should not be compared by slope across families. Bands are 95% bootstrap confidence intervals; slopes are given in Table 1.

MethodMarginal slopeMarginal lowerMarginal upperPartial slopePartial lowerPartial upper
Discrete
SAC-λ\lambda+0.3−1.3+2.0+0.3−1.1+1.8
SAC-λ\lambda-R−2.0−4.1+0.1−2.2−3.6−0.6
SAC-GAE−2.6−8.2+3.1−3.2−6.9+0.4
SAC-GAE-R−0.8−7.3+5.2−0.9−5.5+3.5
Continuous
SAC-λ\lambda−2.7−5.3−0.1−1.4−3.2+0.5
SAC-λ\lambda-R−1.1−3.3+1.1−1.4−3.1+0.2
SAC-GAE−1.4−6.0+3.1+0.5−3.5+4.1
SAC-GAE-R−3.5−10.0+3.3−1.1−6.5+4.8

Table 1. Effect of λ\lambda on performance. Slopes of the task-averaged normalized score of a configuration (in percentage points) per +0.1 in λ\lambda, from weighted least squares. The marginal slope regresses on λ\lambda alone; the partial slope is the λ\lambda coefficient in a regression on all hyperparameters sampled for the method, with learning rates, sizes and counts on a log scale. For the GAE critics, λ\lambda is the GAE coefficient, sampled on [0.85, 0.99]; the λ\lambda-return critics sample $\lambda \in [0.50, 0.95]. Lower and upper bounds are 95% bootstrap confidence intervals; bold marks intervals that exclude zero.

Per Environment Performance Profiles

Figure 10. Per-task performance profiles: share of configurations and seeds reaching normalized score τ\tau on each task (Eq. (15) with M=1M = 1).

Experiment Details

Random Search Protocol

For each of the MM environments, we perform a random hyperparameter search consisting of HH sampled configurations. For each configuration, we train an agent SS times with different random seeds. After training each agent, we additionally perform S′=50S^{\prime}=50 evaluation rollouts per seed, yielding robust estimates of post-training performance. Hyperparameter configurations are drawn by Sobol sampling over the ranges specified in Table 3. All remaining hyperparameters are held fixed to the default SAC settings. The numbers for each sweep are specified in Table 2.

MethodHmin⁡H_{\min}Hmax⁡H_{\max}SSTraining runs
BraxCCMinAtarBraxCCMinAtarBraxCCMinAtarBraxCCMinAtar
PPO64646464646435311521280768
PQN–6464–6464–53–960768
SAC64586464656435311521255768
SAC-A64646464646435311521280768
SAC-DA64646464646435311521280768
SAC-GAE64646464646435311521280768
SAC-GAE-R64646464646435311521280768
SAC-λ\lambda64646464646435311521280768
SAC-λ\lambda-R64646464646435311521280768

Table 2. Configurations per task (HH; minimum and maximum across the tasks of a suite), training seeds per configuration (SS) and resulting training runs per suite.

Hyperparameter
Shared by all methods
Discount factor γ\gamma[0.95, 0.995][0.95,\,0.995]
Actor learning rate[5×10−5, 3×10−3][5 \times10^{-5},\,3 \times10^{-3}], log
Critic learning rate[5×10−5, 3×10−3][5 \times10^{-5},\,3 \times10^{-3}], log
Temperature learning rate[5×10−5, 3×10−3][5 \times10^{-5},\,3 \times10^{-3}], log
One-step replay critics (SAC, SAC-A, SAC-DA)
Batch size{64,128,512,1024,2048}\{64, 128, 512, 1024, 2048\}
Replay buffer size{105,2.5×105,106}\{10^{5}, 2.5 \times10^{5}, 10^{6}\}
Critic updates per iteration{4,8,16,32,64}\{4, 8, 16, 32, 64\}
On-policy multi-step critics (SAC-λ\lambda, SAC-GAE)
Rollout length NN{8,16,32}\{8, 16, 32\}
Critic epochs per iteration{4,8,16,32}\{4, 8, 16, 32\}
Critic minibatches{2,4,8,16}\{2, 4, 8, 16\}
Replay-based multi-step critics (SAC-λ\lambda-R, SAC-GAE-R)
Replayed segment length NN{32,64,128,256}\{32, 64, 128, 256\}
Batch size{1024,2048,4096}\{1024, 2048, 4096\}
Replay buffer size{105,2.5×105,1.5×106}\{10^{5}, 2.5 \times10^{5}, 1.5 \times10^{6}\}
Critic updates per iteration{4,8,16,32}\{4, 8, 16, 32\}
Method-specific
λ\lambda (λ\lambda-return critics)[0.50,0.95][0.50, 0.95]
λ\lambda (GAE critics)[0.85,0.99][0.85, 0.99]
Advantage-critic learning rate[5×10−5, 3×10−3][5 \times10^{-5},\,3 \times10^{-3}], log
Advantage-critic updates (SAC-A only){4,8,16,32,64}\{4, 8, 16, 32, 64\}

Table 3. Sampled hyperparameter ranges. Continuous ranges are given as intervals, discrete ones as sets; “log” denotes log-uniform sampling.

Hyperparameters that exist only in a subset of the variants are sampled on the same footing as the shared ones: λ∈[0.50,0.95]\lambda\in[0.50, 0.95] for the λ\lambda-return critics, the GAE coefficient in [0.85,0.99][0.85, 0.99] for the GAE critics, and the horizon NN, i.e. the rollout length for the on-policy variants and the replayed segment length for the replay-based ones.

Notably, the replay variants of SAC-λ\lambda and SAC-GAE allow for a higher rollout length in our implementation. For SAC-λ\lambda and SAC-GAE NN had to be short, so that there were a reasonable amount of actor updates.

Environment Return ranges

Table 4 lists the reference returns used to normalize scores (§ 4).

TaskFloor95th percentileMaximum
valueloweruppervaluelowerupper
Discrete
CartPole0500500500500500500
Acrobot−500−78.0−78.5−77.5−65.6−69.4−65.6
MountainCar−200−99.3−100.2−98.9−95.2−95.9−95.2
MinAsterix025.322.928.157.445.357.4
MinBreakout093.492.195.0112.0107.7112.0
MinFreeway051.449.652.860.559.160.5
MinSpaceInvaders0166165167176172176
Continuous
Pendulum−500−143.3−143.9−142.7−137.9−139.0−137.9
Reacher−400−8.03−8.79−7.59−4.94−5.82−4.94
Ant0594057286130791173687911
HalfCheetah0161001480016900214001990021400
Hopper0271020753248428638254286
Humanoid0576053386488941287739412
Walker2d0446540104793879875908798

Table 4. Reference returns used for normalization. The floor is the minimum achievable return and maps to 0; the 95th percentile of per-run mean returns, pooled over all algorithms, maps to 1. The pooled maximum is given for reference. Lower and upper bounds are 95% bootstrap confidence intervals.

Algorithm Details

Following the main text, θ\theta denotes critic parameters, ϕ\phi value-network parameters and ψ\psi policy parameters; target networks are indicated by θˉ\bar{\theta} and ϕˉ\bar{\phi}. NN denotes the horizon (rollout or segment length), as in the main text, and CC the period of the hard target-network updates.

Implementation differences at a glance. All variants share the policy objective of Eq. 3, the learned temperature with its target-entropy schedule (Remark D.3) and the optimizer. Beyond the critic target, they differ as follows: (i) SAC, SAC-λ\lambda and SAC-λ\lambda-R use two critics (Remark D.1), whereas SAC-A and SAC-DA use, by design, a single advantage and value network; (ii) the GAE variants use an unsquashed Gaussian policy, the score-function gradient of Eq. (39) and batch-normalized advantages, which rescales the advantages relative to α\alpha and thus the effective temperature of their policy update; (iii) the on-policy variants take a single full-batch policy step per iteration, all others KπK_{\pi} minibatch steps; (iv) the GAE variants use no target network.

SAC

Remark D.1 (double Q function). To reduce the bias induced by overestimation of the QQ-values, we implement an ensemble of QQ-networks, here of size two. The operator agg⁡j=1,2\operatorname{agg}_{j=1,2} aggregates the two critics and depends on the action space: for continuous actions we take the minimum, min⁡j=1,2Qθˉj\min_{j=1,2} Q_{\bar{\theta}_j}, whereas for discrete actions we take the mean, 12∑j=1,2Qθˉj\frac{1}{2}\sum_{j=1,2}Q_{\bar{\theta}_j} (Zhou et al. [52]). The same aggregation is used in the critic target (with target critics) and in the policy loss (with online critics).

Remark D.2 (Discrete vs. continuous action spaces). The algorithm handles either discrete or continuous action spaces, but not mixed ones. For discrete actions, πψ(⋅∣s)\pi_{\psi}(\cdot\mid s) is a categorical distribution, and the critics output one value per action, Qθj(s,⋅)∈R∣A∣Q_{\theta_j}(s,\cdot) \in\mathbb{R}^{|\mathcal{A}|}, from which Qθj(si,ai)Q_{\theta_j}(s_i,a_i) is selected. All expectations over a∼πψa \sim\pi_{\psi} (in the critic target, the policy loss and the temperature loss) are then computed exactly as ∑a∈Aπψ(a∣s)[⋅]\sum_{a\in\mathcal{A}}\pi_{\psi}(a \mid s)[\cdot]. For continuous actions, πψ\pi_{\psi} is a tanh-squashed Gaussian, the critics take (s,a)(s,a) as input, and every expectation is estimated with a single reparameterized sample a∼πψ(⋅∣s)a \sim\pi_{\psi}(\cdot\mid s).

Remark D.3 (α\alpha-schedule). The temperature is learned, parameterized as log⁡α\log\alpha and initialized at α0=0.2\alpha_0 = 0.2, following (Haarnoja et al. [19]). The target entropy is linearly annealed via a scale σt\sigma_t that goes from

Algorithm 1 Vanilla Soft Actor-Critic (SAC)

Initialize: critics Qθ1Q_{\theta_1}, Qθ2Q_{\theta_2}, target critics Q‾θ‾j←Qθj\overline{Q}_{\overline{\theta}_j} \leftarrow Q_{\theta_j}, policy network πψ\pi_\psi

Initialize learned temperature α=exp⁡(log⁡α)\alpha= \exp(\log\alpha) and target-entropy schedule H‾t\overline{\mathcal{H}}_t

Initialize replay buffer D←∅D \leftarrow\emptyset and fill it with WW warm-up transitions collected by πψ\pi_\psi

for t=1,…,budgett = 1,\ldots,\text{budget} do

    environment step: sample action a∼πψ(⋅∣s)a \sim\pi_\psi(\cdot\mid s), next state s′∼p(⋅∣s,a)s' \sim p(\cdot\mid s,a)

    Store transition (s,a,r(s,a),d,s′)(s,a,r(s,a),d,s') in DD, with dd the termination flag

    if tmodU=0t \mathbin{\mathrm{mod}} U = 0 then

        (Policy Evaluation Step)

        for k=1,…,KQk = 1,\ldots,K_Q do

            Sample minibatch B={(si,ai,ri,di,si′)}i=1bB = \{(s_i,a_i,r_i,d_i,s'_i)\}_{i=1}^{b} from DD

            Compute soft target

Ti=ri+(1−di)γEa′∼πψ(⋅∣si′)[agg⁡j=1,2 Q‾θ‾j(si′,a′)−αlog⁡(πψ(a′∣si′))]T_i = r_i + (1-d_i)\gamma\mathbb{E}_{a'\sim\pi_\psi(\cdot\mid s'_i)}\left[\underset{j=1,2}{\operatorname{agg}}\,\overline{Q}_{\overline{\theta}_j}(s'_i,a')-\alpha\log(\pi_\psi(a'\mid s'_i))\right]

cf. (1)

            For each critic calculate critic loss

LcriticSAC(θj)=1b∑i=1b{Qθj(si,ai)−Ti}2L_{\mathrm{critic}}^{\mathrm{SAC}}(\theta_j)=\frac{1}{b}\sum_{i=1}^{b}\left\{Q_{\theta_j}(s_i,a_i)-T_i\right\}^{2}

cf. (4)

            Update critics via gradient descent on ∇θjLcriticSAC\nabla_{\theta_j}L_{\mathrm{critic}}^{\mathrm{SAC}}

            (Target network update) θ‾j←τθj+(1−τ)θ‾j\overline{\theta}_j \leftarrow\tau\theta_j + (1-\tau)\overline{\theta}_j

        end for

        (Policy Improvement Step)

        for k=1,…,Kπk = 1,\ldots,K_\pi do

            Sample minibatch of states {si}i=1b\{s_i\}_{i=1}^{b} from DD and compute

Lπ(ψ)=1b∑i=1bEa∼πψ(⋅∣si)[−agg⁡j=1,2 Qθj(si,a)+αlog⁡(πψ(a∣si))]L_\pi(\psi)=\frac{1}{b}\sum_{i=1}^{b}\mathbb{E}_{a\sim\pi_\psi(\cdot\mid s_i)}\left[-\underset{j=1,2}{\operatorname{agg}}\,Q_{\theta_j}(s_i,a)+\alpha\log(\pi_\psi(a\mid s_i))\right]

cf. (38)

            Update policy via gradient descent on ∇ψLπ\nabla_\psi L_\pi

        end for

        (Temperature Step)

        for k=1,…,Kαk = 1,\ldots,K_\alpha do

            Sample minibatch of states {si}i=1b\{s_i\}_{i=1}^{b} from DD and compute

L(α)=−1b∑i=1bαEa∼πψ(⋅∣si)[log⁡(πψ(a∣si))+H‾t]L(\alpha)=-\frac{1}{b}\sum_{i=1}^{b}\alpha\mathbb{E}_{a\sim\pi_\psi(\cdot\mid s_i)}\left[\log(\pi_\psi(a\mid s_i))+\overline{\mathcal{H}}_t\right]

            Update temperature via gradient descent on ∇log⁡αL\nabla_{\log\alpha}L

        end for

    end if

end for

1 to 0.1 over the course of training: for discrete actions H‾t=0.89 σtlog⁡∣A∣\overline{\mathcal{H}}_t = 0.89\,\sigma_t\log|\mathcal{A}|, and for continuous actions H‾t=−(2−σt)dim⁡(A)\overline{\mathcal{H}}_t = -(2-\sigma_t)\dim(\mathcal{A}). In both cases the target entropy decreases, driving the policy towards less stochastic behavior.

Remark D.4 (Target network updates). We use hard target updates, θˉ←θ\bar{\theta} \leftarrow\theta every CC critic gradient steps, i.e. τ(c)=1{C mod N=0}\tau(c) = \mathbb{1}_{\{C \bmod N=0\}} (not environment steps), which performed better than Polyak averaging in our implementation. The same holds for all variants with target networks.

SAC-A

Algorithm 2 SAC-A

Initialize: advantage critic AθA_\theta, soft value network VϕV_\phi, target value network V‾ϕ←Vϕ\overline{V}_\phi\leftarrow V_\phi, policy network πψ\pi_\psi

Initialize learned temperature α\alpha and target-entropy schedule H‾t\overline{\mathcal{H}}_t (cf. Remark D.3)

Initialize replay buffer D←∅D \leftarrow\varnothing and fill it with WW warm-up transitions collected by πψ\pi_\psi

for t=1,…,budgett = 1, \ldots, \text{budget} do

environment step: sample action a∼πψ(⋅∣s)a \sim\pi_\psi(\cdot\mid s), next state s′∼p(⋅∣s,a)s' \sim p(\cdot\mid s,a)

Store transition (s,a,r(s,a),d,s′)(s,a,r(s,a),d,s') in DD, with dd the termination flag

if t mod U=0t \bmod U = 0 then

(Policy Evaluation Step)

Freeze a reference copy Vϕold←VϕV_{\phi_{\mathrm{old}}} \leftarrow V_\phi for value clipping

for k=1,…,KVk = 1,\ldots,K_V do

Sample minibatch B={(si,ai,ri,di,si′)}i=1bB = \{(s_i,a_i,r_i,d_i,s'_i)\}_{i=1}^{b} from DD

Compute soft target TiV=ri+αH(πψ(⋅∣si))+(1−di)γV‾ϕ(si′)T_i^V = r_i + \alpha\mathcal{H}(\pi_\psi(\cdot\mid s_i)) + (1-d_i)\gamma\overline{V}_\phi(s'_i), and the clipped value loss

LV(ϕ)=1b∑i=1bmax⁡{(Vϕ(si)−TiV)2,(Vϕclip(si)−TiV)2}(36)L_V(\phi)=\frac{1}{b}\sum_{i=1}^{b}\max\left\{\left(V_\phi(s_i)-T_i^V\right)^2,\left(V_\phi^{\mathrm{clip}}(s_i)-T_i^V\right)^2\right\} \tag*{(36)}

with Vϕclip(si)=Vϕold(si)+clip⁡(Vϕ(si)−Vϕold(si),−ϵ,ϵ)V_\phi^{\mathrm{clip}}(s_i)=V_{\phi_{\mathrm{old}}}(s_i)+\operatorname{clip}(V_\phi(s_i)-V_{\phi_{\mathrm{old}}}(s_i),-\epsilon,\epsilon)

update the value network via gradient descent on ∇ϕLV\nabla_\phi L_V

(Target network update) ϕ‾←τϕ+(1−τ)ϕ‾\overline{\phi}\leftarrow\tau\phi+(1-\tau)\overline{\phi}

end for

for k=1,…,KAk = 1,\ldots,K_A do

Sample minibatch B={(si,ai,ri,di,si′)}i=1bB = \{(s_i,a_i,r_i,d_i,s'_i)\}_{i=1}^{b} from DD

Compute advantage targets TiA=ri+αH(πψ(⋅∣si))+(1−di)γVϕ(si′)−Vϕ(si)T_i^A=r_i+\alpha\mathcal{H}(\pi_\psi(\cdot\mid s_i))+(1-d_i)\gamma V_\phi(s'_i)-V_\phi(s_i), and the advantage loss

LcriticA(θ)=1b∑i=1b{Aθ(si,ai)−TiA}2(37)L_{\mathrm{critic}}^A(\theta)=\frac{1}{b}\sum_{i=1}^{b}\left\{A_\theta(s_i,a_i)-T_i^A\right\}^2 \tag*{(37)}

Update critic via gradient descent on ∇θLcriticA\nabla_\theta L_{\mathrm{critic}}^A

end for

(Policy Improvement Step)

for k=1,…,Kπk = 1,\ldots,K_\pi do

Sample minibatch of states {si}i=1b\{s_i\}_{i=1}^{b} from DD and compute

Lπ(ψ)=1b∑i=1b(Ea∼πψ(⋅∣si)[−Aθ(si,a)]−αH(πψ(⋅∣si)))(38)L_\pi(\psi)=\frac{1}{b}\sum_{i=1}^{b}\left(\mathbb{E}_{a\sim\pi_\psi(\cdot\mid s_i)}[-A_\theta(s_i,a)]-\alpha\mathcal{H}(\pi_\psi(\cdot\mid s_i))\right) \tag*{(38)}

Update policy via gradient descent on ∇ψLπ\nabla_\psi L_\pi

end for

(Temperature Step) as in Algorithm 1

end if

end for

Remark D.5 (Shared components). Data collection, the temperature step, the gradient-based updates and the hard target updates (now counted in value-network steps) are identical to SAC. By design, and in contrast to SAC, there is no critic ensemble: a single advantage network AθA_\theta and a single value network VϕV_\phi are used.

Remark D.6 (Consistency weight κ\kappa). The consistency term enforces Ea∼πψ[Aθ(s,a)]=0\mathbb{E}_{a\sim\pi_\psi}[A_\theta(s,a)]=0. In our experiments we set κ=0\kappa=0, i.e. the advantage critic is trained on the regression loss alone.

SAC-DA

Algorithm 3 SAC-DA

Initialize: critic AθA_\theta, soft value network VϕV_\phi, target networks Aˉθ←Aθ\bar{A}_\theta\leftarrow A_\theta, Vˉϕ←Vϕ\bar{V}_\phi\leftarrow V_\phi, policy network πψ\pi_\psi

Initialize learned temperature α\alpha and target-entropy schedule Hˉt\bar{H}_t (cf. Remark D.3)

Initialize replay buffer D←∅D\leftarrow\varnothing and fill it with WW warm-up transitions collected by πψ\pi_\psi

for t=1,…,budgett=1,\ldots,\text{budget} do

environment step: sample action a∼πψ(⋅∣s)a\sim\pi_\psi(\cdot\mid s), next state s′∼p(⋅∣s,a)s'\sim p(\cdot\mid s,a)

Store transition (s,a,r(s,a),d,s′)(s,a,r(s,a),d,s') in DD, with dd the termination flag

if t mod U=0t\bmod U=0 then

(Policy Evaluation Step)

for k=1,…,KQk=1,\ldots,K_Q do

Sample minibatch B={(si,ai,ri,di,si′)}i=1bB=\{(s_i,a_i,r_i,d_i,s'_i)\}_{i=1}^{b} from DD

Combine advantages and values to Q-values

Qθ,ϕ(si,ai)=Aθ(si,ai)+Vϕ(si)−αH(πψ(⋅∣si))−κdE^[Aθ(si,⋅)]Q_{\theta,\phi}(s_i,a_i)=A_\theta(s_i,a_i)+V_\phi(s_i)-\alpha\mathcal{H}(\pi_\psi(\cdot\mid s_i))-\kappa_d\widehat{\mathbb{E}}[A_\theta(s_i,\cdot)]

cf. Eq. (7)

Compute soft targets using the joint soft Q of the target networks

Ti=ri+(1−di)γ(Ea′∼πψ(⋅∣si′)[Qθˉ,ϕˉ(si′,a′)]+αH(πψ(⋅∣si′)))T_i=r_i+(1-d_i)\gamma\left(\mathbb{E}_{a'\sim\pi_\psi(\cdot\mid s'_i)}\left[Q_{\bar{\theta},\bar{\phi}}(s'_i,a')\right]+\alpha\mathcal{H}(\pi_\psi(\cdot\mid s'_i))\right)

cf. Eq. (1)

Calculate the critic loss

LcriticDA(θ,ϕ)=1b∑i=1b{Qθ,ϕ(si,ai)−Ti}2\mathcal{L}^{\mathrm{DA}}_{\mathrm{critic}}(\theta,\phi)=\frac{1}{b}\sum_{i=1}^{b}\left\{Q_{\theta,\phi}(s_i,a_i)-T_i\right\}^{2}

cf. Eq. (8)

Update advantages and values jointly via gradient descent on ∇θ,ϕLcriticDA\nabla_{\theta,\phi}\mathcal{L}^{\mathrm{DA}}_{\mathrm{critic}}

(Target network updates) ϕˉ←τϕ+(1−τ)ϕˉ\bar{\phi}\leftarrow\tau\phi+(1-\tau)\bar{\phi} and θˉ←τθ+(1−τ)θˉ\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}

end for

(Policy Improvement Step) as in Algorithm 2

(Temperature Step) as in Algorithm 1

end if

end for

Remark D.7 (Shared components). Data collection, the temperature step, the gradient-based updates and the hard target updates are identical to SAC. By design, a single advantage and value network are used, as in SAC-A. In contrast to SAC-A AθA_\theta and VϕV_\phi are trained jointly on a single loss. Both target networks Aˉθ\bar{A}_\theta and Vˉϕ\bar{V}_\phi receive a hard update every NN joint critic steps.

Remark D.8 (Regularizer weight κd\kappa_d). κd=0\kappa_d=0 worked best in preliminary experiments and is used throughout (§ 3.2). The split into AθA_\theta and VϕV_\phi is then not identifiable, and SAC-DA tests the dueling parametrization of the soft Q-critic.

SAC-lambda

Image not extracted

Figure 4 (PDF p. 31).

Remark D.9 (Shared components). The critic ensemble and its aggregation agg⁡\operatorname{agg}, the discrete/continuous handling, the temperature step, the optimizer and the hard target updates (counted in critic gradient steps) are identical to SAC.

Remark D.10 (On-policy training). SAC-λ\lambda uses no replay buffer. Every iteration collects E⋅TE \cdot T fresh transitions, where the segments continue from the previous iteration rather than starting new episodes. The λ\lambda-returns are computed once per iteration with the current target critics and temperature, and they stay fixed during the subsequent updates. The critics are trained for KQepK_Q^{\mathrm{ep}} epochs over KQmbK_Q^{\mathrm{mb}} shuffled minibatches each, while the policy and the temperature each take a single full-batch gradient step per iteration.

Remark D.11 (Soft Peng’s Q(λ\lambda)). With Vˉ(s)=Ea∼πψ[Qθˉ(s,a)]+αH(πψ(⋅∣s))\bar{V}(s) = \mathbb{E}_{a \sim\pi_\psi}[Q_{\bar{\theta}}(s,a)] + \alpha\mathcal{H}(\pi_\psi(\cdot|s)), the recursion equals (32): with weight 1−λ1-\lambda the return bootstraps from the soft value of the next state under the current policy rather

than from the action actually taken, and with weight λ\lambda it continues with the next return plus the entropy bonus at the next state. This is the soft analogue of Peng’s Q(λ)Q(\lambda) [33]; unlike Kozuno et al. [24], we keep the entropy bonus at every step. At segment ends and episode boundaries (ei,n=1e_{i,n}=1) the trace is cut, and bootstrapping is suppressed only on termination (di,n=1d_{i,n}=1), so on truncation the return bootstraps from the true final state.

SAC-GAE

In order to get a suitable gradient estimator for our empirical policy loss

Lπ(ψ)=1b∑i=1bEa∼πψ(⋅∣si)[−A^(si,a)+αlog⁡(πψ(a∣si))],L_\pi(\psi) = \frac{1}{b}\sum_{i=1}^{b}\mathbb{E}_{a\sim\pi_\psi(\cdot\mid s_i)}[-\hat{A}(s_i,a) + \alpha\log(\pi_\psi(a\mid s_i))],

some care is needed. We only have access to the advantage estimator A^(si,ai)\hat{A}(s_i,a_i) at the sampled action aia_i, not under all actions, so the loss above cannot be implemented directly. The soft policy gradient theorem (Shi et al. [39]; Choe & Kim [8]) resolves this, and the same proof yields

∇ψEa∼πψ(⋅∣s)[A^(s,a)−αlog⁡(πψ(a∣s))]=Ea∼πψ(⋅∣s)[∇ψlog⁡(πψ(a∣s)){A^(s,a)−α(log⁡(πsg⁡(ψ)(a∣s))+1)}],\nabla_\psi\mathbb{E}_{a\sim\pi_\psi(\cdot\mid s)}[\hat{A}(s,a)-\alpha\log(\pi_\psi(a\mid s))] = \mathbb{E}_{a\sim\pi_\psi(\cdot\mid s)}[\nabla_\psi\log(\pi_\psi(a\mid s))\{\hat{A}(s,a)-\alpha(\log(\pi_{\operatorname{sg}(\psi)}(a\mid s))+1)\}],

where sg⁡(⋅)\operatorname{sg}(\cdot) refers to a stop gradient. We can use this expression to obtain a gradient estimator for the above loss as

∇ψLπ(ψ)=1b∑i=1b∇ψlog⁡(πψ(ai∣si)){−A^(si,ai)+α(log⁡(πsg⁡(ψ)(ai∣si))+1)}(39)\nabla_\psi L_\pi(\psi) = \frac{1}{b}\sum_{i=1}^{b}\nabla_\psi\log(\pi_\psi(a_i\mid s_i))\{-\hat{A}(s_i,a_i)+\alpha(\log(\pi_{\operatorname{sg}(\psi)}(a_i\mid s_i))+1)\} \tag*{(39)}

Remark D.12 (No target network). In contrast to all other variants, SAC-GAE uses no target network. The advantages and returns are computed once per iteration with the value network VϕoldV_{\phi_{\mathrm{old}}} from before the update and stay fixed during the subsequent updates. The same VϕoldV_{\phi_{\mathrm{old}}} serves as reference for the value clipping with ϵ=10\epsilon=10.

Replay-buffer variants

SAC-λ\lambda-R and SAC-GAE-R combine the off-policy training loop of SAC (Algorithm 1) with the return estimators of SAC-λ\lambda (Algorithm 4) and SAC-GAE (Algorithm 5), respectively. Data collection, the replay buffer, the update schedule (KQK_Q critic steps, then KπK_\pi policy steps and KαK_\alpha temperature steps every UU environment steps), the temperature step and the hard target updates are exactly as in SAC. The networks and losses are those of the on-policy counterparts: SAC-λ\lambda-R uses the twin critics QθjQ_{\theta_j} and the SAC policy loss, and SAC-GAE-R uses a single value network VϕV_\phi without target network, the untanh’d Gaussian policy and the policy-gradient loss of Eq. (39). The only structural change is how a minibatch is formed: instead of bb independent transitions, each gradient step samples contiguous traces from DD and computes the returns on them with the current networks.

The critic (resp. value) loss is then the one of the on-policy counterpart on BB, i.e. 1b∑(i,m)∈B(Qθj(si,m,ai,m)−Ri,m)2\frac{1}{b}\sum_{(i,m)\in B}(Q_{\theta_j}(s_{i,m},a_{i,m})-R_{i,m})^2 for each critic in SAC-λ\lambda-R, and the value loss of Algorithm 5 in SAC-GAE-R. The policy loss of SAC-GAE-R uses the advantages of a separately sampled trace minibatch, computed after the critic updates and normalized over that minibatch. SAC-λ\lambda-R and the temperature step only need the states si,ms_{i,m} and use no returns.

In contrast to the on-policy variants, where returns are computed once per rollout and kept fixed, here they are recomputed for every sampled minibatch with the current target critics (SAC-λ\lambda-R) or the current value network (SAC-GAE-R), and with the current policy and temperature. Hence the traces are stale, but their returns are not.

Algorithm 5 SAC-GAE

Initialize: soft value network VϕV_{\phi}, policy network πψ\pi_{\psi}

Initialize learned temperature α\alpha and target-entropy schedule H‾t\overline{H}_{t} (cf. Remark D.3)

for t=1,…,budgett = 1, \ldots, \text{budget} do

environment step: continue EE parallel environments for TT steps under πψ\pi_{\psi}

Store trajectory segments in trajectory batch {τi}i=1E\{\tau_i\}_{i=1}^{E}, τi=(si,n,ai,n,ri,n,di,n,ei,n,si,n′)n=1T\tau_i = (s_{i,n}, a_{i,n}, r_{i,n}, d_{i,n}, e_{i,n}, s'_{i,n})_{n=1}^{T}, with dd the termination and ee the termination-or-truncation flag

Freeze a reference copy Vϕold←VϕV_{\phi_{\mathrm{old}}} \leftarrow V_{\phi}

(Calculate soft GAEs) cf. Eq. 14

for i=1,…,Ei = 1, \ldots, E do

set Ai,T+1=0A_{i,T+1} = 0

for n=T,…,1n = T, \ldots, 1 do

δi,n=ri,n+αH(πψ(⋅∣si,n))+γ(1−di,n)Vϕold(si,n′)−Vϕold(si,n)\delta_{i,n} = r_{i,n} + \alpha\mathcal{H}(\pi_{\psi}(\cdot\mid s_{i,n})) + \gamma(1-d_{i,n}) V_{\phi_{\mathrm{old}}}(s'_{i,n}) - V_{\phi_{\mathrm{old}}}(s_{i,n})
Ai,n=δi,n+γλ(1−ei,n)Ai,n+1A_{i,n} = \delta_{i,n} + \gamma\lambda(1-e_{i,n}) A_{i,n+1}
Ri,n=Ai,n+Vϕold(si,n)R_{i,n} = A_{i,n} + V_{\phi_{\mathrm{old}}}(s_{i,n})

end for

end for

(Policy Evaluation Step)

for k=1,…,KVepk = 1, \ldots, K_{V}^{\mathrm{ep}} epochs, and each of KVmbK_{V}^{\mathrm{mb}} shuffled minibatches B⊂{(i,n)}B \subset\{(i,n)\} do

Calculate clipped value loss

LV(ϕ)=12∣B∣∑(i,n)∈Bmax⁡{(Vϕ(si,n)−Ri,n)2,(Vϕclip(si,n)−Ri,n)2}L_V(\phi) = \frac{1}{2|B|} \sum_{(i,n)\in B} \max\left\{ \left(V_{\phi}(s_{i,n}) - R_{i,n}\right)^2, \left(V_{\phi}^{\mathrm{clip}}(s_{i,n}) - R_{i,n}\right)^2 \right\}

cf. Eq. 5

with Vϕclip(s)=Vϕold(s)+clip⁡(Vϕ(s)−Vϕold(s),−ϵ,ϵ)V_{\phi}^{\mathrm{clip}}(s) = V_{\phi_{\mathrm{old}}}(s) + \operatorname{clip}(V_{\phi}(s) - V_{\phi_{\mathrm{old}}}(s), -\epsilon, \epsilon)

Update the value network via gradient descent on ∇ϕLV\nabla_{\phi}L_V

end for

(Policy Improvement Step)

Normalize advantages A~i,n=(Ai,n−μA)/(σA+10−8)\widetilde{A}_{i,n} = (A_{i,n} - \mu_A)/(\sigma_A + 10^{-8}) over the batch and compute

Lπ(ψ)=1E∑i=1E1T∑n=1Tlog⁡(πψ(ai,n∣si,n)){−A~i,n+α(log⁡(πsg⁡(ψ)(ai,n∣si,n))+1)}L_{\pi}(\psi) = \frac{1}{E}\sum_{i=1}^{E}\frac{1}{T}\sum_{n=1}^{T}\log(\pi_{\psi}(a_{i,n}\mid s_{i,n}))\left\{-\widetilde{A}_{i,n}+\alpha\left(\log(\pi_{\operatorname{sg}(\psi)}(a_{i,n}\mid s_{i,n}))+1\right)\right\}

cf. Eq. 35

Update policy via gradient descent on ∇ψLπ\nabla_{\psi}L_{\pi}

(Temperature Step) as in Algorithm 1, on the states of the current rollout

end for

References

  1. [1]Jacob Adkins, Michael Bowling, and Adam White. A method for evaluating hyperparameter sensitivity in reinforcement learning. Advances in Neural Information Processing Systems, 37:124820–124842, 2024.
  2. [2]Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms, 2021a.
  3. [3]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pp. 29304–29320. Curran Associates, Inc., 2021b.arxiv.org/abs/2108.13264
  4. [4]Carlo Alfano, Sebastian Rene Towers, Silvia Sapora, Chris Lu, and Patrick Rebeschini. Learning mirror maps in policy mirror descent. In Seventeenth European Workshop on Reinforcement Learning, 2024.arxiv.org/abs/2402.05187
  5. [5]Reza Asad, Reza Babanezhad, and Sharan Vaswani. Revisiting actor-critic methods in discrete action off-policy reinforcement learning, 2025. URL https://arxiv.org/abs/2509.09838.
  6. [6]Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 4, pp. 2448–2453. IEEE, 1994.
  7. [7]Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-making library for pytorch. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 1778–1811, 2024.arxiv.org/abs/2306.00577
  8. [8]Jean Seong Bjorn Choe and Jong-kook Kim. Maximum entropy softmax policy gradient via entropy advantage estimation. In James Kwok (ed.), Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pp. 4958–4966. International Joint Conferences on Artificial Intelligence Organization, 8 2025. doi: 10.24963/ijcai.2025/552. Main Track.
  9. [9]Tyler Clark, Mark Towers, Christine Evers, and Jonathon Hare. Beyond the rainbow: High performance deep reinforcement learning on a desktop pc. arXiv preprint arXiv:2411.03820, 2024.arxiv.org/abs/2411.03820
  10. [10]Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2185–2194, 2019.arxiv.org/abs/1909.03004
  11. [11]Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407–1416. PMLR, 2018.arxiv.org/abs/1802.01561
  12. [12]Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022.arxiv.org/abs/2103.06257
  13. [13]William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pp. 3061–3071. PMLR, 2020.arxiv.org/abs/2007.06700
  14. [14]C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/brax.arxiv.org/abs/2106.13281
  15. [15]Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024.arxiv.org/abs/2407.04811
  16. [16]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A Theory of Regularized Markov Decision Processes. In Proceedings of the 36th International Conference on Machine Learning, pp. 2160–2169. PMLR, may 2019.arxiv.org/abs/1901.11275
  17. [17]Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies. In Proceedings of the 34th International Conference on Machine Learning, pp. 1352–1361. PMLR, jul 2017.arxiv.org/abs/1702.08165
  18. [18]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861–1870. PMLR, jul 2018a.arxiv.org/abs/1801.01290
  19. [19]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.arxiv.org/abs/1812.05905
  20. [20]Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, 2018.arxiv.org/abs/1710.02298
  21. [21]Shiyu Huang, Bin Wang, Hang Su, Dong Li, Jianye Hao, Jun Zhu, and Ting Chen. Off-policy training for truncated td (λ) boosted soft actor-critic. In Pacific Rim International Conference on Artificial Intelligence, pp. 46–59. Springer, 2021.DOI
  22. [22]Yubo Huang, Xuechun Wang, Luobao Zou, Zhiwei Zhuang, and Weidong Zhang. Soft policy optimization using dual-track advantage estimator. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 1064–1069. IEEE, 2020.arxiv.org/abs/2009.06858
  23. [23]Yash Jhaveri, Harley Wiltzer, Patrick Shafto, Marc G Bellemare, and David Meger. Convergence theorems for entropy-regularized and distributional reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
  24. [24]Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting peng’s q (λ) for modern reinforcement learning. In International Conference on Machine Learning, pp. 5794–5804. PMLR, 2021.arxiv.org/abs/2103.00107
  25. [25]Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming, 198(1):1059–1106, mar 2023. ISSN 1436-4646. doi: 10.1007/s10107-022-01816-5.DOI
  26. [26]Robert Tjarko Lange. gymnax: A JAX-based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax.
  27. [27]Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv:1902.00137, 2019.arxiv.org/abs/1902.00137
  28. [28]Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp. 6820–6829. PMLR, 2020.arxiv.org/abs/2005.06392
  29. [29]Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. PMLR, 2016.arxiv.org/abs/1602.01783
  30. [30]Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017.arxiv.org/abs/1705.07798
  31. [31]Samuel Neumann, Jiamin He, Adam White, and Martha White. Investigating the utility of mirror descent in off-policy actor-critic. In Reinforcement Learning Conference, 2025.
  32. [32]Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygf-kSYwH.arxiv.org/abs/1908.03568
  33. [33]Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceedings 1994, pp. 226–232. Elsevier, 1994.
  34. [34]Koen Ponse, Aske Plaat, and Thomas M. Moerland. Jaxnasium: Embracing pytrees for any-agent, cross-suite reinforcement learning, 2026. URL https://arxiv.org/abs/2610.36075.
  35. [35]Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1–8, 2021.
  36. [36]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.arxiv.org/abs/1506.02438
  37. [37]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  38. [38]Harm Seijen and Rich Sutton. True online td (lambda). In International Conference on Machine Learning, pp. 692–700. PMLR, 2014.
  39. [39]Wenjie Shi, Shiji Song, and Cheng Wu. Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning, September 2019.arxiv.org/abs/1909.03198
  40. [40]Manas Shil, G. N. Pillai, and M. K. Gupta. Improved Soft Actor-Critic: Reducing Bias and Estimation Error for Fast Learning. In 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–6, February 2023. doi: 10.1109/SCEECS57921.2023.10063058.DOI
  41. [41]Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–44, 1988.
  42. [42]Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pp. 6096–6104. PMLR, 2019.arxiv.org/abs/1901.09732
  43. [43]Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-based soft actor-critic with n-step returns. arXiv preprint arXiv:2503.03660, 2025.arxiv.org/abs/2503.03660
  44. [44]Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror Descent Policy Optimization, jun 2021.arxiv.org/abs/2005.09814
  45. [45]Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 8619–8649. PMLR, 2022.arxiv.org/abs/2108.05828
  46. [46]Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. Advances in Neural Information Processing Systems, 33:12163–12174, 2020.
  47. [47]Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995–2003. PMLR, 2016.arxiv.org/abs/1511.06581
  48. [48]Christopher John Cornish Hellaby Watkins. Learning from delayed rewards, 1989.DOI
  49. [49]Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23 (282):1–36, 2022.arxiv.org/abs/2201.07443
  50. [50]Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019.arxiv.org/abs/1903.03176
  51. [51]Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization, 33(2):1061–1091, 2023.arxiv.org/abs/2105.11066
  52. [52]Haibin Zhou, Tong Wei, Zichuan Lin, Junliang Xing, Xuanchun Shi, Li Shen, Chao Yu, Deheng Ye, et al. Revisiting discrete soft actor-critic. arXiv preprint arXiv:2209.10081, 2022.arxiv.org/abs/2209.10081
  53. [53]Haibin Zhou, Tong Wei, Zichuan Lin, junyou Ji, Junliang Xing, Yuanchun Shi, Li Shen, Chao Yu, and Deheng Ye. Revisiting Discrete Soft Actor-Critic, November 2024.
  54. [54]Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.DOI
  55. [55]Jakub Łyskawa, Jakub Lewandowski, and Paweł Wawrzyński. Sacn: Soft actor-critic with n-step returns, 2025. URL https://arxiv.org/abs/2512.13165.

Paper details

vlnc:W_bpbpuy6w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents