WorksComputer Science
Revisiting Critic Estimation in Soft Actor-Critic Algorithms
Abstract
Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and -return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler -return critic. What does improve performance, most clearly on continuous control, is training a -return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft -return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms.
Introduction
Soft Actor-Critic [18] is one of the most widely adopted algorithms in deep reinforcement learning. Its entropy regularization sustains exploration [18], improves robustness [12], and yields a unique optimal policy, unlike the unregularized objective [23]. Modern implementations [35] (Bu et al. 2024) follow Haarnoja et al. [19] in learning one or two soft Q-functions from one-step temporal-difference errors. This design is a convention rather than a requirement: SAC is defined by its actor-critic structure and entropy regularization, not by a particular critic. This leaves considerable freedom in how the critic is learned – freedom that remains largely unexplored.
We study three dimensions of this freedom, which together span the grid of critic variants in Figure 1: (i) learning soft advantages instead of soft Q-values; (ii) the rollout depth of the target, through soft -returns; and (iii) whether multi-step targets are computed from on-policy rollouts or replayed from a
Figure 1. Design choices and resulting variants. The critic is trained on replayed or on-policy data, with one-step or multi-step -return targets, and predicts soft Q-values or soft advantages . SAC occupies the one-step, replay-based Q-value cell; the other five cells are the variants we study.
buffer. Since all variants share the policy update, the implementation and the evaluation protocol, differences in outcome can be attributed to the critic.
In A3C (Mnih et al. [29]) and Proximal Policy Optimization (PPO; Schulman et al. [37]), advantages reduce the variance of score-function policy gradients. This role does not carry over to SAC, whose policy gradient is unaffected by state-dependent shifts of the critic (§ 2). Advantages can nevertheless be numerically favorable: SAC’s policy depends only on the differences between actions, which can be small relative to the value itself and hard to resolve with an approximate Q-function (Tallec et al. [42]), whereas an advantage critic learns them directly ([6, 47]). We test whether this makes SAC more robust to its hyperparameters, learning soft advantages with an explicit advantage loss (SAC-A) or jointly with a soft state value in a dueling architecture (SAC-DA); all other components are inherited from SAC.
The rollout depth of the target trades off bias and variance in the critic’s learning signal. Generalized Advantage Estimation (GAE; Schulman et al. [36]) interpolates between one-step bootstrapped targets and Monte Carlo returns, and PQN (Gallici et al. [15]) recently carried the same idea over to Q-learning, replacing the replay buffer with many parallel environments. Following the idea of PQN, we drop the replay buffer, and train a soft Q-function on soft -returns from parallel on-policy rollouts (SAC-). Combining both axes, we also implement a variant in which we estimate soft advantages using GAE (SAC-GAE).
Multi-step targets are harder to combine with replay, since a replayed trajectory was generated by an older policy and biases the target. Many methods correct for this with importance weights or truncated traces (Espeholt et al. [11]), including concurrent work on SAC (Łyskawa et al. [55]), yet uncorrected multi-step targets from a buffer have been demonstrated to work well in value-based methods ([20, 13, 9]) and in actor-critic continuous control (Kozuno et al. [24]). For the entropy-regularized target, the picture is less settled: Kozuno et al. [24] found its per-step entropy terms destabilizing and kept only the final one. We train both multi-step critics on replayed segments with the full soft -return and no correction (SAC--R, SAC-GAE-R), and compare them with their on-policy counterparts to see whether the bias outweighs the benefit of reusing data.
All of these choices solely concern critic learning and fit into existing SAC implementations with little overhead. We present them for reinforcement learning with a general convex regularizer, of which SAC’s entropy bonus is the most prominent instance, and then measure what each buys in SAC across an entire random hyperparameter search on fourteen tasks from three benchmark suites. Soft advantages improve on soft Q-values neither in return nor in robustness, and the GAE critic adds nothing over the -return critic. In continuous control, multi-step gains come with replay rather than depth: within our sampled ranges, neither a larger nor a longer horizon helps, whereas training on replayed segments does, despite the uncorrected bias. This extends earlier evidence for uncorrected multi-step targets under replay to the full soft -return. In practice, the critic parameterization can be chosen on grounds other than return, and in SAC, too, multi-step targets can be paired with replay without an off-policy correction.
Preliminaries
We assume a standard Markov Decision Process (MDP) defined by a tuple . Here, is the state space, is the action space, is the initial state distribution, is the probabilistic transition and reward function, and is the discount factor. We usually abbreviate the reward by .
The behavior of an agent interacting with this MDP is defined by a policy, that is, a mapping assigning a distribution over actions to each state. Each policy induces a distribution on the set of trajectories (Agarwal et al. [2]). The value function is defined as , resulting in the RL objective .
Regularized MDP. A natural extension of the RL objective adds a regularization term at the reward level, also referred to as MDP regularization (Ziebart [54]; ); Lee et al. [27]). More precisely, given a convex function , the corresponding regularized value function for some regularization weight (temperature) is defined as [30, 25]. The resulting regularized MDP objective can be expressed as .
Following the convention of prior literature [18, 28, 51], we define the regularized Q-function as . This entails the relationship between the two soft functions (e.g. Zhan et al. [51]), and gives rise to the soft Bellman Operator [18]:
Throughout, we fix to the negative Shannon entropy, for , which biases the search towards more exploratory policies and forms the basis of the SAC algorithm [54, 17, 18].
Soft action-values and soft advantages
Advantages measure how much better an action is than the policy’s average in a given state, separating the action-dependent part of the value from its state-dependent level. For soft Q-functions, the difference does not generally vanish in expectation, so we follow prior work [28, 8] and define the soft advantages as
For this recovers the standard advantage, since then . As and differ only by a state-dependent term, soft advantages can replace the soft Q-function in the SAC policy update:
The two forms yield identical gradients, exactly for discrete actions and for every sample of SAC’s reparameterized estimator, so a Q-critic and an advantage critic can differ only in how well they are learned. We will use the following properties of soft advantages.
Lemma 2.1. The soft advantages satisfy the following properties for any , :
(i) The soft advantages can be obtained from the soft value function as ,
(ii) The soft advantages vanish in expectation, i.e. .
Property (i) expresses the soft advantages in terms of the soft value function and is the key ingredient for several algorithms in the next section. Property (ii) confirms that they retain only the action-dependent part of the Q-function. For completeness, we provide proofs in Appendix A.
Estimating the targets
The soft Bellman operator of Eq. (1) and Lemma 2.1 are all that is needed to train critics other than SAC’s one-step Q-function. We now formalize the variants of Figure 1, with training pipelines shown in Figure 2: after the SAC critic, we introduce one-step advantage critics, then critics trained on soft -returns and soft GAE, and finally their replay-based counterparts. All variants share the policy objective of Eq. 3; pseudo-code and the remaining implementation differences are given in Appendix D.

Figure 2. Training pipelines of the SAC variants. (a) SAC-A and SAC-DA keep SAC’s replay-based one-step pipeline but learn soft advantages. (b) SAC trains its Q-network on replayed transitions, SAC- on parallel on-policy rollouts, and SAC--R on replayed trajectory segments. (c) SAC-GAE and SAC-GAE-R learn a soft value network, on rollouts or replayed segments, and compute soft GAEs for the policy update.
Soft Actor-Critic
Following Haarnoja et al. [19], in the now common form without a separate value network, SAC trains on the soft one-step temporal difference error of Eq. (1)
Here, refers to a target network, and to a buffer of stored one-step transitions (Raffin et al., [35]; Bou et al. [7]). Throughout, the expectations over are estimated by sampling for continuous actions.
Soft Actor-Critic with Advantages
Based on Lemma 2.1 we can provide a Bellman-like equation for the soft values, and hence a loss for an approximation of the soft values :
Using the value estimates, an advantage network can be trained, based on Lemma 2.1:
Since Eq. 5 uses the action stored in the buffer, under replay targets the soft value of the behavior policy rather than of , and SAC-A inherits this bias. The dueling formulation below avoids it, as its combined critic is trained on the standard soft Q-target, which is valid off-policy.
Dueling Q-Networks. Advantage and value networks can also be trained under a single objective, following the dueling architecture of Wang et al. [47], in which the two are recombined into a Q-function that is then trained on the one-step temporal difference error. Since by definition, we set
where the last term is the regularizer of Wang et al. [47]. For a finite action set we compute it exactly, ; for continuous actions the mean is estimated from samples . The combined critic is trained with the SAC loss of Eq. (4),
In our experiments, worked best, leaving the split into and unidentifiable: SAC-DA thus tests the dueling parameterization of the soft Q-critic, while SAC-A learns soft advantages explicitly. By design, both variants use a single advantage and value network instead of the two SAC critics; otherwise, they train on one-step transitions from a replay buffer and drop into existing SAC implementations unchanged (Figure 2a). We refer to them as SAC-A and SAC-DA.
Soft Actor-Critic with -returns
The second dimension is the rollout depth of the target (Figure 1, middle row), for which we adopt the two changes that lead from DQN to PQN. First, we replace the one-step target with a truncated -return ([41, 48, 38]), defined through the soft Bellman operator as
where , is the horizon, and applies Eq. (1) times. Second, we replace the replay buffer by parallel environments (Figure 2b).
For our choice , the regularizer contributes the entropy bonus at every visited state, and a trajectory sampled from yields an unbiased estimate of , the soft -step return
The soft -return then satisfies the recursion
with if is terminal, which allows it to be computed efficiently backwards along a trajectory (Appendix § 8). This is the soft analogue of Peng’s Q() ([33]), the target that PQN uses with a maximum and that Kozuno et al. [24] combine with SAC. The critic is trained on
with targets computed by target networks (Appendix § 11); we refer to this variant as SAC-.
Generalized Soft Advantage Estimation
Combining multi-step targets with soft advantages (Figure 1, middle right) allows the advantages themselves to be estimated by Generalized Advantage Estimation (GAE; Schulman et al. [36]). Formally, for given a trajectory we estimate the advantages via
where . For it recovers the -step advantage estimate of Mnih et al. [29], . Following Choe & Kim [8] and property (i) of Lemma 2.1, this extends to soft advantages as
where ; again, recovers the analogous -step expression (Appendix § 8). We train a soft value network on the soft -returns, the multi-step counterpart of Eq. 5, use it to compute soft GAEs from sampled rollouts (Figure 2c) and refer to this variant as SAC-GAE. Since GAE provides advantages only at sampled actions, SAC-GAE updates its policy with the soft score-function gradient (Shi et al. [39]; Choe & Kim [8]) (Appendix § 11); here, unlike in SAC-A and SAC-DA, the advantage acts as a variance-reducing baseline.
Experience Replay
We consider SAC an off-policy algorithm, because its one-step target can be computed from transitions generated by any policy: the current policy only enters at the next state, through the expectation over . SAC-DA inherits this property; SAC-A only in part (§ 3.2). A multi-step target, in contrast, follows the actions taken along a trajectory, so without a correction such as importance weighting or trace cutting ([11, 55]), it is unbiased only for trajectories sampled from the policy being evaluated.
Replay is nonetheless attractive, since reusing data typically improves sample efficiency. Following other methods that combine uncorrected multi-step targets with a replay buffer ([20, 13, 24, 9]), we train both multi-step critics on trajectory segments drawn from a buffer and deliberately apply no correction. The result is an off-policy estimate of an on-policy target, which is biased by the age of the stored segments; for SAC-GAE-R, whose score-function update also uses the replayed actions, the same holds for the policy gradient. Whether that bias matters in practice is an empirical question we return to in the next section. These two variants, SAC--R and SAC-GAE-R, complete the grid in Figure 1.
Experiments
We evaluate all variants on fourteen tasks from three suites: classic control (Acrobot, CartPole, MountainCar, Pendulum; [32]) and MinAtar (Freeway, Asterix, Breakout, SpaceInvaders; [50]), both from Gymnax ([26]), and six MuJoCo control tasks from Brax (Ant, HalfCheetah, Hopper, Humanoid, Reacher, Walker2d; [14]). Since Pendulum and Brax tasks have continuous action spaces and the remaining seven tasks discrete ones, we report the two groups separately. As external reference points we include PPO and, on discrete tasks, PQN, sampled under the same protocol. All implementations build on Jaxnasium ([34]); additional results, experimental details, and pseudo-code are given in Appendices B, C and D.
Evaluation protocol. We treat hyperparameters as part of the algorithm rather than tuning them away before reporting: a method that performs well only in a narrow region of its hyperparameter space demands more expertise to deploy on a new task than one that performs well broadly. For a fixed algorithm, let denote the normalized return after training with hyperparameters on task . Even for fixed and it is random, through initialization, training seeds and hardware nondeterminism ([3]), and through the stochasticity of environment and policy at evaluation. We therefore treat as a random variable and report properties of its distribution over and . Returns are normalized per task so that zero corresponds to the lowest achievable return and one to the 95th percentile of per-run mean returns pooled over all algorithms, which ties the scale neither to a single method nor to a single noisy draw.
Configurations are drawn by Sobol sampling from a region centered on a configuration tuned once for SAC, with no prior tuning of any variant, which amounts to roughly 27,000 training runs; all sampled ranges are listed in Appendix C. Two properties of this protocol shape how the results should be read: Any statement about a distribution over hyperparameters is conditional on : widening it lowers every score, and re-centering it can reorder methods. Centering on the baseline’s optimum is the conservative choice with respect to our variants. Moreover, does not have the same dimension for every method: variants with a second critic carry additional learning rates and update counts, and are therefore sampled more sparsely at a fixed budget of configurations per task, which again works against them. Each metric is defined in the subsection that reports it.
Robustness to Hyperparameter Selection
We first ask how likely a randomly drawn configuration is to work at all. The tail probability is the probability that an algorithm reaches performance level on a randomly drawn task under uniformly drawn hyperparameters. We estimate it with an extended version of the score distribution of Agarwal et al. [3],
where is the normalized return of configuration on task under training seed and evaluation seed , and , , , and denote the numbers of tasks, configurations per task, training seeds per configuration, and post-training evaluations per training seed.
In discrete-action tasks (Figure 3a), no SAC variant rises clearly above SAC, while the two on-policy multi-step variants, SAC- and SAC-GAE, fall clearly below it. On continuous-action tasks (Figure 3b), SAC--R dominates every other method at every threshold and SAC- is second, both holding a larger share of workable configurations than SAC, the advantage variants and PPO. Two contrasts within the figure already localize the effect. Exchanging soft -values for soft advantages moves nothing: SAC-A and SAC-DA stay close to SAC throughout. Adding a replay buffer moves a great deal: SAC--R and SAC-GAE-R sit above SAC- and SAC-GAE across the whole range, without any importance weighting or trace cutting of the replayed targets.

Figure 3. Performance profiles and peak performance. (a) & (b) Performance profiles: the share of configurations reaching normalized score (Eq. (15)). (c) & (d) Peak score against sensitivity ; dashed lines mark constant relative sensitivity , and the lower right corner is best. Bands and bars are 95% stratified bootstrap confidence intervals. PQN is only defined for discrete action spaces and is omitted for continuous action spaces.
Peak vs. Sensitivity
A high chance of drawing a well-performing configuration does not need to come with a high ceiling. Peak performance measures the best score attainable per task,
with the empirical mean performance of a configuration on a task. Since peak performance permits per-task tuning, it says little about how well a configuration generalizes across tasks. We therefore pair it with the sensitivity ([1]), the performance lost when a single configuration must serve all tasks,
In discrete tasks (Figure 3c), peaks cluster near 1.1 for SAC, SAC-A, SAC-DA and SAC--R, with relative sensitivities of roughly 20–30%. The on-policy multi-step variants lose about a third of that peak, and the GAE variants trade peak for a modest reduction in sensitivity without reaching PPO.

Figure 4. Hyperparameter transfer performance and effect of the horizon. (a) & (b) Best per-task configuration against the best configuration transferred from all other tasks. (c) & (d) Normalized score against target horizon . The on-policy variants (solid) are sampled at , the replay-based ones (dashed) at . Bands and bars are 95% stratified bootstrap confidence intervals.
In continuous tasks (Figure 3d) the picture is clearer. SAC--R reaches a clearly higher peak than SAC at roughly the same absolute sensitivity. Because is an absolute quantity, it flatters methods whose ceiling is low to begin with, so we read it relative to the peak: here SAC--R leads with 0.138, against 0.175 for SAC and 0.379 for PPO, and is thus at once the strongest variant and among the cheapest to tune. It ranks first in peak score on five of the seven continuous tasks (Figure 6) and leads at every random-search budget (Figure 7). SAC-GAE-R attains the second-highest peak, far above SAC-GAE, though at a higher sensitivity; replacing on-policy rollouts by replayed segments thus lifts both target types. The advantage variants do not improve on SAC here either: SAC-A, which by design has a single critic, reaches a lower peak, and SAC-DA a similar peak at a higher sensitivity.
Figure 6. Per-task ranks (1 = best). Left: by peak score ; right: by . PQN is not defined for continuous action spaces.
Figure 7. Random-search budget curves. Expected normalized score of the best shared configuration among sampled ones. Bands are 95% bootstrap confidence intervals.
Hyperparameter Transfer Performance
Sensitivity charges one configuration with serving every task at once. A milder question is what a configuration tuned on other tasks is worth on an unseen one. Let be the best shared configuration over all tasks except ; its score on defines the transfer performance
fig:4a plots transfer against peak score. On discrete tasks, SAC-A and SAC-DA match SAC on both bars, and the replay-based variants retain a larger share of their peak than the one-step variants, even where that peak is lower, as for SAC-GAE-R; given the intervals, however, no variant separates from SAC. On continuous tasks (Fig. 4b), SAC--R leads on both bars and retains about 75% of its peak after transfer, followed by SAC-, whereas PPO loses more than half of its peak.
A closer look at the multi-step variants
In continuous tasks, the -return critics, and SAC--R in particular, are the clearest improvement over SAC. Three ingredients set them apart: the -weighting of the target, its horizon, and, for SAC--R and SAC-GAE-R, the replay buffer. All three vary within our search (Appendix § 10), so each can be examined without further runs, within the sampled ranges.
More weight on long returns does not help. Since is sampled independently of every other hyperparameter, its effect can be read off directly. Performance is flat or mildly decreasing in on both splits: of the sixteen slope estimates in Appendix B.5, two exclude zero, and both are negative.
Nor does a longer horizon. The on-policy variants use rollouts of 8 to 32 steps and the replay-based ones replay segments of 32 to 256, so an aggregate comparison between the two families conflates replay with a horizon up to eight times longer. Within each family, the score does not show a consistent increase with the horizon on either split (Fig. 4c,d).
Replay does, despite the uncorrected bias. At , the one horizon both families share, the replay-based variants lead for both the -return and the GAE critic and on both splits (Fig. 4c,d), so the gain is not due to the horizon. Across all fourteen tasks, a randomly drawn run of a replay-based variant outperforms one of its on-policy counterparts for both critics (see also Fig. 8). This gain persists despite the uncorrected bias of the replayed targets. Since the replay-based variants also use larger batches and their own update schedule, this comparison bounds the combined effect of replay rather than isolating the bias.
These results neither make the multi-step target dispensable nor isolate its contribution. Both the and the horizon ranges begin well above the one-step limit, so they show only that additional depth beyond them buys nothing. Relative to the one-step SAC, SAC--R is clearly ahead on continuous tasks (Figure 3b,Figure 3d), but the two differ in more than their target, since the replay-based -return critic is trained on sampled segments, with larger batches and its own update schedule. We therefore attribute the gain of SAC--R on continuous control to the combination it implements, -return targets computed on replayed segments, rather than to any single ingredient. What our data does rule out is that this gain depends on a large , a long horizon, or an off-policy correction.
Related Work
SAC as an instance of Policy Mirror Descent. Policy Mirror Descent (PMD; Geist et al. [16]; Vieillard et al. [46]; Xiao [49]; Vaswani et al. [45]; Lan [25]; Zhan et al. [51]) has emerged as a unifying framework for common RL algorithms. The PMD policy update (e.g. Lan [25], Algorithm 1)
where is a sequence of step sizes, some convex function and some Bregman Divergence, which acts as a trust-region term that prevents large deviations from the previous policy. SAC is thus an instance of PMD, with the negative Shannon entropy and no trust-region term, and the design choices studied here should transfer to other PMD algorithms, including those built on more general convex regularizers (Lee et al., [27]; Alfano et al., [4]). They are also complementary to efforts that add trust-region terms to SAC (Tomar et al., [44]; Neumann et al., [31]).
Multi-step returns & advantages in maximum-entropy RL. Soft advantage estimators are not new: Huang et al. [22] introduced a variant of GAE for entropy-regularized objectives, and Choe & Kim [8] use a closely related soft GAE in PPO-style algorithms, whereas we integrate soft advantages into SAC. Multi-step targets have likewise been studied in entropy-regularized settings (Shil et al., [40]), including for transformer-based policies (Tian et al., [43]); Huang et al. [21] and Lyskawa et al. (2025) train SAC on replayed - and -step returns, corrected by importance sampling and truncated importance sampling, respectively. Kozuno et al. [24] show that Peng’s , which applies no correction, often outperforms conservative operators in continuous control, and characterize when its bias is benign; for SAC, however, they drop the entropy bonus from all but the last step of the target for stability. Our results add to these findings: the full soft -return, entropy terms included and with a learned temperature, trains stably from replay across a distribution of hyperparameters.
More broadly, recent work revisits core design assumptions in SAC, particularly regarding critic learning and target construction (Shil et al., [40]; Zhou et al., [53]; Asad et al., [5]); we complement it with a systematic study of soft advantages, multi-step targets and replay within a unified SAC framework.
Conclusion
We asked which choices in SAC’s critic influence its outcomes by keeping the policy objective fixed and varying three of these choices: the learned quantity, the rollout depth of the target, and experience replay. In total, we evaluated across six critic variants over an entire random hyperparameter search on fourteen tasks. Learning soft advantages instead of soft Q-values neither makes SAC more robust to its hyperparameters nor improves its performance: the advantage and dueling critics stay close to SAC in the share of well-performing configurations and in transfer, and on continuous tasks SAC-A even reaches a lower peak. Since SAC’s policy gradient is unaffected by state-dependent shifts of the critic, this is a finding about critic learning: in our setting, fitting the state-dependent level of the value is not what limits SAC’s critic. The soft GAE critic, in turn, adds nothing over the simpler -return critic.
The clearest gain comes from training a -return critic on replayed segments. In continuous tasks, SAC--R holds the largest share of well-performing configurations, reaches the highest peak score at a lower relative sensitivity than SAC, and keeps most of that peak under transfer across tasks. Its lead over the on-policy SAC- persists at a matched horizon, while neither a larger nor a longer horizon helps within the sampled ranges; on discrete tasks, no variant improves on SAC. In line with earlier evidence for uncorrected multi-step targets, the missing off-policy correction does not prevent these gains, and our results extend this observation to the full soft -return. In practice, SAC’s critic parametrization can thus be chosen on grounds other than return, and multi-step targets can be paired with replay without correction.
Several limitations bound these conclusions. Our continuous-control evidence rests on a limited number of continuous action tasks, and given SAC’s prevalence in that domain, further benchmarks are warranted. The replay comparison bounds the combined effect of target staleness, larger batches and a different update schedule rather than isolating the bias, and since SAC--R also differs from SAC in its training regime, we attribute its gain to the combination rather than to the multi-step target alone. Any statement about a distribution over hyperparameters is conditional on the search space, which we centered on a configuration tuned for SAC. Finally, we studied the variants in isolation from other recent SAC extensions to attribute effects cleanly; combining them remains open.
Acknowledgments
This work was financially supported by Shell Information Technology International Limited and the Netherlands Enterprise Agency under the grant PPS23-3-03529461.
Proofs
Soft Advantages
As highlighted in the related work, we can see the SAC policy update as an instance of Policy Mirror Descent [51]. In fact, for all instances of PMD, the (soft) action-value function can be replaced by the corresponding (soft) advantage function.
The term in the final expression can be dropped because it does not depend on . This holds for any convex and is not limited to the negative Shannon entropy.
Proof of Lemma 2.1. We prove the two properties of Lemma 2.1 together with a third, which relates soft and unregularized advantages.
Lemma A.1 (Lemma 2.1, extended). The soft advantages satisfy the following properties for any , :
(i) ,
(ii) ,
(iii) , where and denote the cumulative discounted negative entropy starting with state-action pair and in state , respectively:
Proof. Recall that the soft Q-function is defined as
Additionally, we can re-express the relation between and as
(i) Now, we can insert the relations of Eqs. (21) and (22) into the definition of the soft advantages
(ii) follows from definition as
(iii) By definition again, we can express
Inserting this into the expression obtained in (i) yields (after some reorganization of terms)
Notably, we can also immediately see that if we use the soft value function as baseline subtraction, i.e. , we obtain
(by Eq. (22)) which does not vanish for non-deterministic policies in the regularized case ().
Soft n-step returns
Though soft -step returns have been used before with SAC ([40, 55, 43]), we give the details behind Eqs. (10) and 11 for completeness. Throughout, we write and .
Claim. The -step soft Bellman Operator satisfies the following expression:
Proof. Let be a fixed policy and define the soft Bellman operator
where and .
We proceed by induction on .
Base case (). By definition of the soft Bellman operator,
which coincides with the claimed expression for .
Inductive step. Assume the statement holds for some , i.e.
Consider . Using the operator definition,
Applying the induction hypothesis at time and using the law of total expectation yields
Re-indexing the sums gives
which is exactly the desired form. This completes the induction.
Along a trajectory sampled from , , so by the tower property each term can be replaced by its conditional expectation , and the bootstrap by , without changing the expectation:
This motivates the soft -step return of Eq. (10) as an unbiased estimate of the -step soft Bellman operator,
Unlike in Tian et al. [43], the entropy enters as a discounted bonus for the next state. This matches the soft Bellman operator, since is the one-step target of Eq. (1). Based on Eq. (9), we further define the -return for a trajectory of length as
While the recursive relation in Eq. 11 was used before, e.g. by Huang et al. [21], we provide a formal proof for completeness.
Claim. The soft -returns satisfy the recursion
Proof. Write . Then and, separating the first term, for . Inserting both into the definition,
where we grouped and re-indexed the sum. The expression in braces is , hence
By convention, , and if is terminal, which yields the stated recursion. □
Algorithm 4 computes the same recursion in the equivalent form , with the soft value .
GAE for soft advantages
As in the unregularized case (), for the soft GAE recovers a -step lookahead minus the value of the current state:
where the last step holds for , with in Eq. (31) and .
Additional Results
This appendix complements the main results with a variance decomposition, per-task rankings, probabilities of improvement at matched horizon, random-search budget curves, the ablation and per-task performance profiles.
Variance Decomposition
An interesting question is whether the design choices change where performance variance originates. Performance varies with the task, the configuration, the training seed and the evaluation episode, and we decompose the total sum of squares accordingly. With training seeds on task , , , as in Section 4, and ,
and each share is .
The resulting profiles are shown in Figure 5. On continuous tasks the one-step methods draw about 75% of their variance from differences between tasks, whereas the multi-step variants draw only 24–45%, with correspondingly more coming from configuration, seed and episode noise. A smaller between-task share means performance depends less on which task happens to be drawn, which is consistent with the stronger transfer reported above. On discrete tasks the pattern reverses, and what distinguishes the multi-step variants there is instead a smaller training-noise share at comparable evaluation noise.

Figure 5. Variance decomposition. Share of total variance attributable to task, configuration, training seed and evaluation episode, over all configurations (a, c) and over each method’s top-quartile configurations (b, d), on discrete (a, b) and continuous (c, d) tasks.
The comparison is biased by performance level, since a method that fails uniformly has little seed-to-seed variance left to decompose. Restricting the decomposition to each method’s top-quartile configurations (Figure 5b,Figure 5d) shrinks the configuration share and raises the seed and episode shares, as expected, but leaves the between-task trend mostly intact.
Per Task Ranking
Due to the amount of environment-algorithm combinations, most of our analysis has been focused on aggregate metrics. To complement this view Figure 6 shows the per-task ranks of all methods, by peak score (left) and by the interquartile mean over configurations, (right), a proxy for performance under limited tuning. By peak score, SAC--R ranks first on nine of the fourteen tasks, including five of the seven continuous ones, and never below fourth. By IQM the picture is similar but less pronounced: SAC--R ranks first or second on every continuous task, but drops to seventh on MountainCar.
Random-Search Budget Curve
Ranking by the IQM in Section B.2 was used as a proxy for how rankings change with limited tuning. To further investigate that effect, we use budget curves [10]: they give the expected score of the best shared configuration among randomly drawn ones,
On continuous tasks (Figure 7a), SAC--R leads at every budget and SAC- is second, so their advantage does not depend on how much tuning is spent. On discrete tasks (Figure 7b), SAC, the advantage variants and the replay-based variants overlap almost fully and gain similarly from tuning, the on-policy multi-step variants stay below, and PQN gains most from additional configurations.

Figure 8a. X=SAC-GAE-R, Y=SAC-GAE
Figure 8b. X=SAC--R, Y=SAC-
fig:8: Probability of improvement at of the replay-based over the on-policy variant, per task and aggregated. (a): SAC-GAE-R vs. SAC-GAE; (b): SAC--R vs. SAC-; the dashed line marks 0.5. Bars are 95% stratified bootstrap confidence intervals.
Replay at Matched Horizon: Probability of Improvement
To complement Figure 4c,d, we compare each replay-based variant with its on-policy counterpart at the shared horizon , using the probability of improvement of Agarwal et al. [3], extended to configurations and evaluation seeds. For methods and ,
where the sums run over the configurations sampled with . For both replay-base variants the CI for the probability of improvement does not contain zero on either suite (Figure 8a,b), demonstrating a statistically significant improvement.
Ablations
is sampled for every multi-step variant, independently of all other hyperparameters, so the score against within one variant is a genuine ablation of the multi-step target: same architecture, data and update schedule, but a longer effective horizon (Figure 9). We quantify it by two weighted least-squares slopes of the task-averaged score of a configuration, , in (Table 1). The marginal slope regresses on alone and thus averages over all other hyperparameters; the partial slope is the coefficient in a regression on all hyperparameters sampled for the method, with learning rates, sizes and counts on a log scale. Since is sampled independently of the other hyperparameters, both estimate the same effect, but the partial slope is more precise, as the controls absorb variance that does not explain. Of the sixteen estimates, only two exclude zero, and both are negative: within the sampled ranges, placing more weight on long-range returns does not improve performance.
Figure 9. Effect of the sampled . normalized score against , binned at bin centers, for every method that samples it. The -return critics are sampled on [0.50, 0.95] and the GAE critics on [0.85, 0.99], so the two families occupy different parts of the axis and should not be compared by slope across families. Bands are 95% bootstrap confidence intervals; slopes are given in Table 1.
| Method | Marginal slope | Marginal lower | Marginal upper | Partial slope | Partial lower | Partial upper |
| Discrete | ||||||
| SAC- | +0.3 | −1.3 | +2.0 | +0.3 | −1.1 | +1.8 |
| SAC--R | −2.0 | −4.1 | +0.1 | −2.2 | −3.6 | −0.6 |
| SAC-GAE | −2.6 | −8.2 | +3.1 | −3.2 | −6.9 | +0.4 |
| SAC-GAE-R | −0.8 | −7.3 | +5.2 | −0.9 | −5.5 | +3.5 |
| Continuous | ||||||
| SAC- | −2.7 | −5.3 | −0.1 | −1.4 | −3.2 | +0.5 |
| SAC--R | −1.1 | −3.3 | +1.1 | −1.4 | −3.1 | +0.2 |
| SAC-GAE | −1.4 | −6.0 | +3.1 | +0.5 | −3.5 | +4.1 |
| SAC-GAE-R | −3.5 | −10.0 | +3.3 | −1.1 | −6.5 | +4.8 |
Table 1. Effect of on performance. Slopes of the task-averaged normalized score of a configuration (in percentage points) per +0.1 in , from weighted least squares. The marginal slope regresses on alone; the partial slope is the coefficient in a regression on all hyperparameters sampled for the method, with learning rates, sizes and counts on a log scale. For the GAE critics, is the GAE coefficient, sampled on [0.85, 0.99]; the -return critics sample $\lambda \in [0.50, 0.95]. Lower and upper bounds are 95% bootstrap confidence intervals; bold marks intervals that exclude zero.
Per Environment Performance Profiles
Figure 10. Per-task performance profiles: share of configurations and seeds reaching normalized score on each task (Eq. (15) with ).
Experiment Details
Random Search Protocol
For each of the environments, we perform a random hyperparameter search consisting of sampled configurations. For each configuration, we train an agent times with different random seeds. After training each agent, we additionally perform evaluation rollouts per seed, yielding robust estimates of post-training performance. Hyperparameter configurations are drawn by Sobol sampling over the ranges specified in Table 3. All remaining hyperparameters are held fixed to the default SAC settings. The numbers for each sweep are specified in Table 2.
| Method | Training runs | |||||||||||
| Brax | CC | MinAtar | Brax | CC | MinAtar | Brax | CC | MinAtar | Brax | CC | MinAtar | |
| PPO | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| PQN | – | 64 | 64 | – | 64 | 64 | – | 5 | 3 | – | 960 | 768 |
| SAC | 64 | 58 | 64 | 64 | 65 | 64 | 3 | 5 | 3 | 1152 | 1255 | 768 |
| SAC-A | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| SAC-DA | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| SAC-GAE | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| SAC-GAE-R | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| SAC- | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
| SAC--R | 64 | 64 | 64 | 64 | 64 | 64 | 3 | 5 | 3 | 1152 | 1280 | 768 |
Table 2. Configurations per task (; minimum and maximum across the tasks of a suite), training seeds per configuration () and resulting training runs per suite.
| Hyperparameter | |
| Shared by all methods | |
| Discount factor | |
| Actor learning rate | , log |
| Critic learning rate | , log |
| Temperature learning rate | , log |
| One-step replay critics (SAC, SAC-A, SAC-DA) | |
| Batch size | |
| Replay buffer size | |
| Critic updates per iteration | |
| On-policy multi-step critics (SAC-, SAC-GAE) | |
| Rollout length | |
| Critic epochs per iteration | |
| Critic minibatches | |
| Replay-based multi-step critics (SAC--R, SAC-GAE-R) | |
| Replayed segment length | |
| Batch size | |
| Replay buffer size | |
| Critic updates per iteration | |
| Method-specific | |
| (-return critics) | |
| (GAE critics) | |
| Advantage-critic learning rate | , log |
| Advantage-critic updates (SAC-A only) |
Table 3. Sampled hyperparameter ranges. Continuous ranges are given as intervals, discrete ones as sets; “log” denotes log-uniform sampling.
Hyperparameters that exist only in a subset of the variants are sampled on the same footing as the shared ones: for the -return critics, the GAE coefficient in for the GAE critics, and the horizon , i.e. the rollout length for the on-policy variants and the replayed segment length for the replay-based ones.
Notably, the replay variants of SAC- and SAC-GAE allow for a higher rollout length in our implementation. For SAC- and SAC-GAE had to be short, so that there were a reasonable amount of actor updates.
Environment Return ranges
Table 4 lists the reference returns used to normalize scores (§ 4).
| Task | Floor | 95th percentile | Maximum | ||||
| value | lower | upper | value | lower | upper | ||
| Discrete | |||||||
| CartPole | 0 | 500 | 500 | 500 | 500 | 500 | 500 |
| Acrobot | −500 | −78.0 | −78.5 | −77.5 | −65.6 | −69.4 | −65.6 |
| MountainCar | −200 | −99.3 | −100.2 | −98.9 | −95.2 | −95.9 | −95.2 |
| MinAsterix | 0 | 25.3 | 22.9 | 28.1 | 57.4 | 45.3 | 57.4 |
| MinBreakout | 0 | 93.4 | 92.1 | 95.0 | 112.0 | 107.7 | 112.0 |
| MinFreeway | 0 | 51.4 | 49.6 | 52.8 | 60.5 | 59.1 | 60.5 |
| MinSpaceInvaders | 0 | 166 | 165 | 167 | 176 | 172 | 176 |
| Continuous | |||||||
| Pendulum | −500 | −143.3 | −143.9 | −142.7 | −137.9 | −139.0 | −137.9 |
| Reacher | −400 | −8.03 | −8.79 | −7.59 | −4.94 | −5.82 | −4.94 |
| Ant | 0 | 5940 | 5728 | 6130 | 7911 | 7368 | 7911 |
| HalfCheetah | 0 | 16100 | 14800 | 16900 | 21400 | 19900 | 21400 |
| Hopper | 0 | 2710 | 2075 | 3248 | 4286 | 3825 | 4286 |
| Humanoid | 0 | 5760 | 5338 | 6488 | 9412 | 8773 | 9412 |
| Walker2d | 0 | 4465 | 4010 | 4793 | 8798 | 7590 | 8798 |
Table 4. Reference returns used for normalization. The floor is the minimum achievable return and maps to 0; the 95th percentile of per-run mean returns, pooled over all algorithms, maps to 1. The pooled maximum is given for reference. Lower and upper bounds are 95% bootstrap confidence intervals.
Algorithm Details
Following the main text, denotes critic parameters, value-network parameters and policy parameters; target networks are indicated by and . denotes the horizon (rollout or segment length), as in the main text, and the period of the hard target-network updates.
Implementation differences at a glance. All variants share the policy objective of Eq. 3, the learned temperature with its target-entropy schedule (Remark D.3) and the optimizer. Beyond the critic target, they differ as follows: (i) SAC, SAC- and SAC--R use two critics (Remark D.1), whereas SAC-A and SAC-DA use, by design, a single advantage and value network; (ii) the GAE variants use an unsquashed Gaussian policy, the score-function gradient of Eq. (39) and batch-normalized advantages, which rescales the advantages relative to and thus the effective temperature of their policy update; (iii) the on-policy variants take a single full-batch policy step per iteration, all others minibatch steps; (iv) the GAE variants use no target network.
SAC
Remark D.1 (double Q function). To reduce the bias induced by overestimation of the -values, we implement an ensemble of -networks, here of size two. The operator aggregates the two critics and depends on the action space: for continuous actions we take the minimum, , whereas for discrete actions we take the mean, (Zhou et al. [52]). The same aggregation is used in the critic target (with target critics) and in the policy loss (with online critics).
Remark D.2 (Discrete vs. continuous action spaces). The algorithm handles either discrete or continuous action spaces, but not mixed ones. For discrete actions, is a categorical distribution, and the critics output one value per action, , from which is selected. All expectations over (in the critic target, the policy loss and the temperature loss) are then computed exactly as . For continuous actions, is a tanh-squashed Gaussian, the critics take as input, and every expectation is estimated with a single reparameterized sample .
Remark D.3 (-schedule). The temperature is learned, parameterized as and initialized at , following (Haarnoja et al. [19]). The target entropy is linearly annealed via a scale that goes from
Algorithm 1 Vanilla Soft Actor-Critic (SAC)
Initialize: critics , , target critics , policy network
Initialize learned temperature and target-entropy schedule
Initialize replay buffer and fill it with warm-up transitions collected by
for do
environment step: sample action , next state
Store transition in , with the termination flag
if then
(Policy Evaluation Step)
for do
Sample minibatch from
Compute soft target
cf. (1)
For each critic calculate critic loss
cf. (4)
Update critics via gradient descent on
(Target network update)
end for
(Policy Improvement Step)
for do
Sample minibatch of states from and compute
cf. (38)
Update policy via gradient descent on
end for
(Temperature Step)
for do
Sample minibatch of states from and compute
Update temperature via gradient descent on
end for
end if
end for
1 to 0.1 over the course of training: for discrete actions , and for continuous actions . In both cases the target entropy decreases, driving the policy towards less stochastic behavior.
Remark D.4 (Target network updates). We use hard target updates, every critic gradient steps, i.e. (not environment steps), which performed better than Polyak averaging in our implementation. The same holds for all variants with target networks.
SAC-A
Algorithm 2 SAC-A
Initialize: advantage critic , soft value network , target value network , policy network
Initialize learned temperature and target-entropy schedule (cf. Remark D.3)
Initialize replay buffer and fill it with warm-up transitions collected by
for do
environment step: sample action , next state
Store transition in , with the termination flag
if then
(Policy Evaluation Step)
Freeze a reference copy for value clipping
for do
Sample minibatch from
Compute soft target , and the clipped value loss
with
update the value network via gradient descent on
(Target network update)
end for
for do
Sample minibatch from
Compute advantage targets , and the advantage loss
Update critic via gradient descent on
end for
(Policy Improvement Step)
for do
Sample minibatch of states from and compute
Update policy via gradient descent on
end for
(Temperature Step) as in Algorithm 1
end if
end for
Remark D.5 (Shared components). Data collection, the temperature step, the gradient-based updates and the hard target updates (now counted in value-network steps) are identical to SAC. By design, and in contrast to SAC, there is no critic ensemble: a single advantage network and a single value network are used.
Remark D.6 (Consistency weight ). The consistency term enforces . In our experiments we set , i.e. the advantage critic is trained on the regression loss alone.
SAC-DA
Algorithm 3 SAC-DA
Initialize: critic , soft value network , target networks , , policy network
Initialize learned temperature and target-entropy schedule (cf. Remark D.3)
Initialize replay buffer and fill it with warm-up transitions collected by
for do
environment step: sample action , next state
Store transition in , with the termination flag
if then
(Policy Evaluation Step)
for do
Sample minibatch from
Combine advantages and values to Q-values
cf. Eq. (7)
Compute soft targets using the joint soft Q of the target networks
cf. Eq. (1)
Calculate the critic loss
cf. Eq. (8)
Update advantages and values jointly via gradient descent on
(Target network updates) and
end for
(Policy Improvement Step) as in Algorithm 2
(Temperature Step) as in Algorithm 1
end if
end for
Remark D.7 (Shared components). Data collection, the temperature step, the gradient-based updates and the hard target updates are identical to SAC. By design, a single advantage and value network are used, as in SAC-A. In contrast to SAC-A and are trained jointly on a single loss. Both target networks and receive a hard update every joint critic steps.
Remark D.8 (Regularizer weight ). worked best in preliminary experiments and is used throughout (§ 3.2). The split into and is then not identifiable, and SAC-DA tests the dueling parametrization of the soft Q-critic.
SAC-lambda
Figure 4 (PDF p. 31).
Remark D.9 (Shared components). The critic ensemble and its aggregation , the discrete/continuous handling, the temperature step, the optimizer and the hard target updates (counted in critic gradient steps) are identical to SAC.
Remark D.10 (On-policy training). SAC- uses no replay buffer. Every iteration collects fresh transitions, where the segments continue from the previous iteration rather than starting new episodes. The -returns are computed once per iteration with the current target critics and temperature, and they stay fixed during the subsequent updates. The critics are trained for epochs over shuffled minibatches each, while the policy and the temperature each take a single full-batch gradient step per iteration.
Remark D.11 (Soft Peng’s Q()). With , the recursion equals (32): with weight the return bootstraps from the soft value of the next state under the current policy rather
than from the action actually taken, and with weight it continues with the next return plus the entropy bonus at the next state. This is the soft analogue of Peng’s [33]; unlike Kozuno et al. [24], we keep the entropy bonus at every step. At segment ends and episode boundaries () the trace is cut, and bootstrapping is suppressed only on termination (), so on truncation the return bootstraps from the true final state.
SAC-GAE
In order to get a suitable gradient estimator for our empirical policy loss
some care is needed. We only have access to the advantage estimator at the sampled action , not under all actions, so the loss above cannot be implemented directly. The soft policy gradient theorem (Shi et al. [39]; Choe & Kim [8]) resolves this, and the same proof yields
where refers to a stop gradient. We can use this expression to obtain a gradient estimator for the above loss as
Remark D.12 (No target network). In contrast to all other variants, SAC-GAE uses no target network. The advantages and returns are computed once per iteration with the value network from before the update and stay fixed during the subsequent updates. The same serves as reference for the value clipping with .
Replay-buffer variants
SAC--R and SAC-GAE-R combine the off-policy training loop of SAC (Algorithm 1) with the return estimators of SAC- (Algorithm 4) and SAC-GAE (Algorithm 5), respectively. Data collection, the replay buffer, the update schedule ( critic steps, then policy steps and temperature steps every environment steps), the temperature step and the hard target updates are exactly as in SAC. The networks and losses are those of the on-policy counterparts: SAC--R uses the twin critics and the SAC policy loss, and SAC-GAE-R uses a single value network without target network, the untanh’d Gaussian policy and the policy-gradient loss of Eq. (39). The only structural change is how a minibatch is formed: instead of independent transitions, each gradient step samples contiguous traces from and computes the returns on them with the current networks.
The critic (resp. value) loss is then the one of the on-policy counterpart on , i.e. for each critic in SAC--R, and the value loss of Algorithm 5 in SAC-GAE-R. The policy loss of SAC-GAE-R uses the advantages of a separately sampled trace minibatch, computed after the critic updates and normalized over that minibatch. SAC--R and the temperature step only need the states and use no returns.
In contrast to the on-policy variants, where returns are computed once per rollout and kept fixed, here they are recomputed for every sampled minibatch with the current target critics (SAC--R) or the current value network (SAC-GAE-R), and with the current policy and temperature. Hence the traces are stale, but their returns are not.
Algorithm 5 SAC-GAE
Initialize: soft value network , policy network
Initialize learned temperature and target-entropy schedule (cf. Remark D.3)
for do
environment step: continue parallel environments for steps under
Store trajectory segments in trajectory batch , , with the termination and the termination-or-truncation flag
Freeze a reference copy
(Calculate soft GAEs) cf. Eq. 14
for do
set
for do
end for
end for
(Policy Evaluation Step)
for epochs, and each of shuffled minibatches do
Calculate clipped value loss
cf. Eq. 5
with
Update the value network via gradient descent on
end for
(Policy Improvement Step)
Normalize advantages over the batch and compute
cf. Eq. 35
Update policy via gradient descent on
(Temperature Step) as in Algorithm 1, on the states of the current rollout
end for
References
- [1]Jacob Adkins, Michael Bowling, and Adam White. A method for evaluating hyperparameter sensitivity in reinforcement learning. Advances in Neural Information Processing Systems, 37:124820–124842, 2024.
- [2]Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms, 2021a.
- [3]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pp. 29304–29320. Curran Associates, Inc., 2021b.arxiv.org/abs/2108.13264
- [4]Carlo Alfano, Sebastian Rene Towers, Silvia Sapora, Chris Lu, and Patrick Rebeschini. Learning mirror maps in policy mirror descent. In Seventeenth European Workshop on Reinforcement Learning, 2024.arxiv.org/abs/2402.05187
- [5]Reza Asad, Reza Babanezhad, and Sharan Vaswani. Revisiting actor-critic methods in discrete action off-policy reinforcement learning, 2025. URL https://arxiv.org/abs/2509.09838.
- [6]Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 4, pp. 2448–2453. IEEE, 1994.
- [7]Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-making library for pytorch. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 1778–1811, 2024.arxiv.org/abs/2306.00577
- [8]Jean Seong Bjorn Choe and Jong-kook Kim. Maximum entropy softmax policy gradient via entropy advantage estimation. In James Kwok (ed.), Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pp. 4958–4966. International Joint Conferences on Artificial Intelligence Organization, 8 2025. doi: 10.24963/ijcai.2025/552. Main Track.
- [9]Tyler Clark, Mark Towers, Christine Evers, and Jonathon Hare. Beyond the rainbow: High performance deep reinforcement learning on a desktop pc. arXiv preprint arXiv:2411.03820, 2024.arxiv.org/abs/2411.03820
- [10]Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2185–2194, 2019.arxiv.org/abs/1909.03004
- [11]Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407–1416. PMLR, 2018.arxiv.org/abs/1802.01561
- [12]Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022.arxiv.org/abs/2103.06257
- [13]William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pp. 3061–3071. PMLR, 2020.arxiv.org/abs/2007.06700
- [14]C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/brax.arxiv.org/abs/2106.13281
- [15]Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024.arxiv.org/abs/2407.04811
- [16]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A Theory of Regularized Markov Decision Processes. In Proceedings of the 36th International Conference on Machine Learning, pp. 2160–2169. PMLR, may 2019.arxiv.org/abs/1901.11275
- [17]Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies. In Proceedings of the 34th International Conference on Machine Learning, pp. 1352–1361. PMLR, jul 2017.arxiv.org/abs/1702.08165
- [18]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861–1870. PMLR, jul 2018a.arxiv.org/abs/1801.01290
- [19]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.arxiv.org/abs/1812.05905
- [20]Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, 2018.arxiv.org/abs/1710.02298
- [21]Shiyu Huang, Bin Wang, Hang Su, Dong Li, Jianye Hao, Jun Zhu, and Ting Chen. Off-policy training for truncated td (λ) boosted soft actor-critic. In Pacific Rim International Conference on Artificial Intelligence, pp. 46–59. Springer, 2021.DOI
- [22]Yubo Huang, Xuechun Wang, Luobao Zou, Zhiwei Zhuang, and Weidong Zhang. Soft policy optimization using dual-track advantage estimator. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 1064–1069. IEEE, 2020.arxiv.org/abs/2009.06858
- [23]Yash Jhaveri, Harley Wiltzer, Patrick Shafto, Marc G Bellemare, and David Meger. Convergence theorems for entropy-regularized and distributional reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
- [24]Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting peng’s q (λ) for modern reinforcement learning. In International Conference on Machine Learning, pp. 5794–5804. PMLR, 2021.arxiv.org/abs/2103.00107
- [25]Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming, 198(1):1059–1106, mar 2023. ISSN 1436-4646. doi: 10.1007/s10107-022-01816-5.DOI
- [26]Robert Tjarko Lange. gymnax: A JAX-based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax.
- [27]Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv:1902.00137, 2019.arxiv.org/abs/1902.00137
- [28]Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp. 6820–6829. PMLR, 2020.arxiv.org/abs/2005.06392
- [29]Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. PMLR, 2016.arxiv.org/abs/1602.01783
- [30]Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017.arxiv.org/abs/1705.07798
- [31]Samuel Neumann, Jiamin He, Adam White, and Martha White. Investigating the utility of mirror descent in off-policy actor-critic. In Reinforcement Learning Conference, 2025.
- [32]Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygf-kSYwH.arxiv.org/abs/1908.03568
- [33]Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceedings 1994, pp. 226–232. Elsevier, 1994.
- [34]Koen Ponse, Aske Plaat, and Thomas M. Moerland. Jaxnasium: Embracing pytrees for any-agent, cross-suite reinforcement learning, 2026. URL https://arxiv.org/abs/2610.36075.
- [35]Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1–8, 2021.
- [36]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.arxiv.org/abs/1506.02438
- [37]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [38]Harm Seijen and Rich Sutton. True online td (lambda). In International Conference on Machine Learning, pp. 692–700. PMLR, 2014.
- [39]Wenjie Shi, Shiji Song, and Cheng Wu. Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning, September 2019.arxiv.org/abs/1909.03198
- [40]Manas Shil, G. N. Pillai, and M. K. Gupta. Improved Soft Actor-Critic: Reducing Bias and Estimation Error for Fast Learning. In 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–6, February 2023. doi: 10.1109/SCEECS57921.2023.10063058.DOI
- [41]Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–44, 1988.
- [42]Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pp. 6096–6104. PMLR, 2019.arxiv.org/abs/1901.09732
- [43]Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-based soft actor-critic with n-step returns. arXiv preprint arXiv:2503.03660, 2025.arxiv.org/abs/2503.03660
- [44]Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror Descent Policy Optimization, jun 2021.arxiv.org/abs/2005.09814
- [45]Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 8619–8649. PMLR, 2022.arxiv.org/abs/2108.05828
- [46]Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. Advances in Neural Information Processing Systems, 33:12163–12174, 2020.
- [47]Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995–2003. PMLR, 2016.arxiv.org/abs/1511.06581
- [48]Christopher John Cornish Hellaby Watkins. Learning from delayed rewards, 1989.DOI
- [49]Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23 (282):1–36, 2022.arxiv.org/abs/2201.07443
- [50]Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019.arxiv.org/abs/1903.03176
- [51]Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization, 33(2):1061–1091, 2023.arxiv.org/abs/2105.11066
- [52]Haibin Zhou, Tong Wei, Zichuan Lin, Junliang Xing, Xuanchun Shi, Li Shen, Chao Yu, Deheng Ye, et al. Revisiting discrete soft actor-critic. arXiv preprint arXiv:2209.10081, 2022.arxiv.org/abs/2209.10081
- [53]Haibin Zhou, Tong Wei, Zichuan Lin, junyou Ji, Junliang Xing, Yuanchun Shi, Li Shen, Chao Yu, and Deheng Ye. Revisiting Discrete Soft Actor-Critic, November 2024.
- [54]Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.DOI
- [55]Jakub Łyskawa, Jakub Lewandowski, and Paweł Wawrzyński. Sacn: Soft actor-critic with n-step returns, 2025. URL https://arxiv.org/abs/2512.13165.





