WorksPhysics and Astronomy
Advancing Quantum Reinforcement Learning with Differentiable Quantum Architecture Search and Adaptive Non-local Observables
Abstract
Quantum Reinforcement Learning (QRL) using variational quantum circuits (VQCs) is constrained by fixed measurement protocols and hand-designed circuit ansätze. Adaptive Non-local Observables (ANO) and Differentiable Quantum Architecture Search (DiffQAS) address these issues independently, but to our knowledge have not been combined or compared in a single QRL framework. We present an ablation of ANO and DiffQAS within the Quantum Time-series Transformer (QTSTransformer), evaluated on a 2 x 2 factorial design across six environments–two classic-control tasks and four Atari 2600 games. Three findings emerge. First, the quantum agent outperforms a matched classical transformer baseline on five of six environments, with absolute scores ranging from 360.0 vs 304.0 (MarioBros) to 17.2 vs 2.5 (Breakout), while using approximately 12.5x fewer transformer parameters on the four Atari environments (more modestly so on the two classic-control tasks). Second, ANO and DiffQAS provide complementary benefits: ANO yields the largest gain on continuous-state CartPole (347.7 vs 223.2), while DiffQAS yields the largest absolute Atari gains (Breakout 17.2 vs 3.0; DonkeyKong 70.0 vs 37.0). Third, combining both innovations exhibits an asymmetric interaction: ANO-dominant tasks tolerate DiffQAS, whereas DiffQAS-dominant tasks degrade when ANO is added. These results provide design guidance for QRL agents: select between ANO and DiffQAS according to whether the bottleneck is the measurement basis or the gate topology, rather than applying both indiscriminately.
Introduction
Quantum Reinforcement Learning (QRL) has emerged as a promising frontier in quantum machine learning, using quantum parallelism and entanglement for potential gains in convergence and representation power [1]. Variational Quantum Circuits (VQCs) have been deployed as function approximators across classic control, arcade games, robotics, and financial prediction tasks [2, 3, 4, 5, 6]. However, current QRL research remains constrained by two bottlenecks: fixed measurement protocols and manually-crafted circuit architectures.
The first limitation is the measurement layer: standard VQCs use fixed local observables (e.g., single-qubit Pauli-Z), restricting output to a confined eigenvalue spectrum that fails to capture complex state-action correlations [7]. Adaptive Non-local Observables (ANO) address this by treating the Hermitian observable as a learnable parameter, letting the model expand its functional space and capture non-local correlations without added circuit depth [7], adapting its measurement basis to the reward landscape [8].
The second hurdle is VQC architectural design: most QRL agents rely on hand-crafted ansätze requiring domain expertise and lacking flexibility across environments [9]. Differentiable Quantum Architecture Search (DiffQAS) relaxes the discrete search space into a continuous, gradient-optimizable domain [10], in principle allowing task-specific architecture selection and reducing the manual trial-and-error burden [9, 11].
Despite the individual successes of ANO and DiffQAS, no study to date has combined or systematically compared these two paradigms, leaving open whether adaptive measurements, automated architectures, or their combination is optimal for high-dimensional sequential decision problems. We address this gap by proposing a QRL framework based on the Quantum Time-series Transformer (QTSTransformer) [12], which integrates both DiffQAS and ANO. By evaluating our model on a spectrum of environments—ranging from simple OpenAI Gymnasium tasks to complex, pixel-based Atari 2600 games—this work provides a systematic analysis of how learnable measurements and automated architecture search collectively influence the performance and scalability of QRL agents.
Background
Quantum Reinforcement Learning (QRL)
Reinforcement learning models sequential decision-making as a Markov Decision Process ; the agent learns a policy maximizing the expected return (not the immediate reward ) by approximating a value function [13, 14].
Quantum Deep Quality-Networks (DQN) replaces the classical DQN with a VQC-based function approximator [15]: a state is encoded via and processed by a trainable ansatz . Rather than the textbook single-observable-per-action formulation, most practical QRL implementations—including ours—measure a fixed set of observables independent of and feed the expectation values into a classical fully-connected (FC) head layer :
where are fixed Pauli operators or learnable ANO Hermitians; this accommodates varying across environments (2 for CartPole, up to 18 for DonkeyKong/MarioBros) at a fixed circuit width of 8 qubits ( baseline, ANO; mapping detailed below).
To stabilize training we use experience replay and the Double DQN target [16], which decouples action selection (online network ) from value evaluation (target network ) and reduces the upward bias of the standard operator.
Quantum Time-series Transformer (QTSTransformer)
The Quantum Time-series Transformer (QTSTransformer) is a hybrid sequence-processing architecture whose idealized quantum subroutine for the time-mixing operation has cost per query in sequence length , compared to for explicit classical self-attention [12]. This asymptotic statement applies only to the abstract subroutine; in this paper the entire pipeline runs on a classical state-vector simulator, and end-to-end cost is dominated by convolutional neural network (CNN) preprocessing, the FC head, replay, and simulator overhead. We therefore use the QTSTransformer as an architectural and inductive-bias choice, not as a wall-clock speedup. The architecture has four stages:
A sequence of latent feature vectors is mapped to rotation angles that parameterize unique unitaries , encoding temporal information directly into a quantum gate sequence. Time-mixing is motivated by the linear-combination-of-unitaries (LCU) primitive [17]: a control register in superposition directs the application of sequence unitaries on a target register, yielding
where is the (unnormalized) attention weight for time step (Born-rule weight from the control register).
Non-linearity is motivated by the quantum singular value transformation (QSVT) framework [18], expressed as a degree- polynomial acting on the singular values of an appropriately block-encoded . We evaluate by accumulating with learnable (degree in all runs). This state is then renormalized and passed through one further single-layer ansatz—the quantum feed-forward block—whose four gate slots are included in the architecture-parameter count of Table II. The resulting state is measured to produce expectation values , with either fixed single-qubit Paulis () or learnable -local Hermitians on 8 sliding windows (), processed by a compact FC head—Linear(, 128) ReLU Linear(128, ) on Atari, Linear(, 64) ReLU Linear(64, ) on classic control—to produce the -dimensional Q-value vector, realizing Eq. (1) uniformly across all environments.
Adaptive Non-local Observables (ANO)
As noted above, fixed local measurements confine VQC output to , limiting expressivity where -values span broader ranges. ANO addresses this by treating the Hermitian measurement operator as a learnable parameter [7, 8, 19].
Inspired by the Heisenberg picture, ANO makes the readout observable trainable rather than fixed—as opposed to standard fixed-Pauli VQCs, restricted to observables in the spectral equivalence class of the chosen Pauli basis. While the fully general case spans the full -dimensional Hermitian space [7], our implementation parameterizes Hermitians on -qubit subsystems only, so the agent navigates the space of -local Hermitians on the chosen sliding windows (we use ). For a -qubit system, let , an adaptive -local observable is Hermitian by design, built from a lower-triangular matrix with three sets of learnable parameters indexed by —real strictly-lower entries , imaginary strictly-lower entries (each parameters), and real diagonal entries ( parameters)—giving where represents trainable real parameters in total: diagonal entries and Hermitian-conjugate off-diagonal pairs , guaranteeing and real-valued expectation values.
By the Rayleigh quotient, the output of an ANO-VQC is bounded by the spectrum of : . Strictly, this argument only shows that the achievable output range becomes learnable; it does not by itself imply improved nonlocal-correlation modeling. Capturing 2-local correlations is plausible since each Hermitian on a 2-qubit subsystem is generally not separable into single-qubit Paulis [7], but isolating range from correlation effects would require a fixed-spectrum 2-local control that we do not include here. In a QRL setting, the agent jointly optimizes the variational parameters and the observable parameters by minimizing the same Double DQN loss, now taken jointly over with target-network copies ; since is differentiable with respect to all learnable entries of , the gradients are computed via automatic differentiation through the quantum circuit simulator [19].
Differentiable Quantum Architecture Search (DiffQAS)
As noted above, DiffQAS relaxes the discrete choice of quantum gates into a continuous, end-to-end optimizable formulation [9, 10, 11, 20]. The general framework [10, 20] assigns each candidate gate a learnable structural weight, relaxed via softmax into a weighted ensemble over candidates, and discretizes to the highest-weight candidate upon convergence.
Euler Angle Parameterization
An alternative to the weighted ensemble is to parameterize each gate slot via continuous Euler angles, allowing smooth interpolation between canonical gates within a single circuit [11]. Each single-qubit gate is replaced by , where is the variational angle (a single per-gate scalar, distinct from the global parameter set used for the rest of the network) and are learnable architecture parameters. With free, the Euler identity implies
, by contrast, cannot be realized for arbitrary by any fixed , since reduces to a -axis rotation only when is an integer multiple of , in which case it equals and is independent of . The rotations realizable by the continuous parameterization are therefore (and for controlled slots). Our implementation places the three discretization references at , labeled , , and . Two implementation-level caveats—a sign absorbed by the learnable at the point, and the point actually realizing during Phase 1 (the search phase; Eq. (2)) before Phase 2 (the production phase) substitutes the genuine —did not affect the production-phase circuits, since all six environments converged to the initialization (a finding we interpret with caution below). Architecture parameters start at (), are jointly optimized with during the search phase, and are discretized to the nearest canonical reference by Euclidean distance at the end.
Experiment
We evaluate the QTSTransformer agent on classic-control tasks [21] (for precise ANO/DiffQAS ablations) and high-dimensional Atari 2600 games [22]. Episode counts are not directly comparable because episode length differs sharply: despite a gap in episode count, CartPole ( episodes steps) and FrozenLake (3,000 episodes – steps) see comparable environment interactions. Atari quantum runs use –M steps each versus –M for the classical baselines (trained to 10,000 episodes). Training ends where the plateau/decline stopping rule or the 48-hour wall-clock limit triggered, not at an arbitrary cutoff.
Classic Control Environments
Classic control tasks provide a controlled regime for fundamental learning efficiency, with low-dimensional state vectors encoded directly into quantum circuit parameters.
CartPole-v1
In CartPole-v1 [21], the agent balances a pole on a cart via horizontal force; the episode ends if the pole angle exceeds , the cart moves more than 2.4 units from center, or 500 timesteps elapse. The 4-dimensional observation (cart position, velocity, pole angle, angular velocity) pairs with a discrete two-action space (push left/right) and a reward per timestep upright (max return 500). A sliding window of 4 states forms a tensor, projected via a linear layer to a 64-dimensional feature space and scaled via sigmoid to to parameterize the rotation gates.
FrozenLake-v1
FrozenLake-v1 [21] requires navigating a grid from a starting tile to a goal tile under stochastic “slippery” dynamics (only a chance of moving in the intended direction). The discrete state is one-hot encoded (16-dimensional); the action space has 4 discrete movements. Rewards are sparse (+1 only on reaching the goal, 0 otherwise). As with CartPole, a sliding window of 4 timesteps forms a tensor, mapped via a linear layer to 64 rotation parameters scaled to via a sigmoid.
Atari2600 Environments
We test high-dimensional feature processing and long-range temporal modeling on four Atari 2600 games [22].
Common Preprocessing and Quantum Interface
Atari observations are preprocessed to reduce dimensionality while preserving motion cues: actions are repeated across 4 frames (frame skipping), raw RGB frames (210 × 160 × 3) are converted to grayscale and resized to 84 × 84, and pixel values are normalized to and stacked across the 4 most recent timesteps to form a input tensor. A DQN-style CNN [14] extracts 128-dimensional features per timestep; standard sinusoidal positional encoding [23] represents temporal order within the transformer. The resulting vectors are projected to the rotation parameter space and scaled to via a centered sigmoid: .
Benchmark Games
SpaceInvaders (ALE/SpaceInvaders-v5): a 6-action shoot-’em-up (NOOP, FIRE, LEFT, RIGHT, LEFTFIRE, RIGHTFIRE) with score-based rewards. Breakout (ALE/Breakout-v5): a 4-action ball-and-paddle game (NOOP, FIRE, LEFT, RIGHT); rewards scale with brick row height, and the game accelerates as bricks clear. DonkeyKong (ALE/DonkeyKong-v5): an 18-action platforming task with rewards from item collection, obstacle avoidance, and stage-completion bonuses. MarioBros (ALE/MarioBros-v5): an 18-action platforming task based on the 1983 arcade game, with rewards for defeating enemies and coin collection.
QRL Agent Design and RL Techniques
The agent is a hybrid classical-quantum DQN with the QTSTransformer-based readout over sequences of four states (or four CNN-encoded frames for Atari). We use Double DQN [16] and uniform experience replay. All models are trained until the 100-episode moving-average return plateaus or begins to decline.
ANO and DiffQAS Ablation Methodology
We conducted a factorial ablation study to isolate the contributions of our two primary quantum innovations:
ANO
The ANO condition replaces the baseline’s fixed Pauli readout with eight learnable -local Hermitian observables. Each is a Hermitian with 16 learnable parameters (6 real off-diagonal, 6 imaginary off-diagonal, 4 diagonal). The observables are placed in circular sliding windows across 8 qubits, allowing the agent to adapt its measurement basis to the reward landscape.
DiffQAS
DiffQAS automates the selection of gate types in the variational circuit using the Euler angle parameterization described earlier. Each ansatz layer contains four gate slots—two single-qubit slots searching over and two controlled slots searching over (though are reachable only as discretization targets, not during the continuous search itself)—parameterized via and , respectively, where is the variational rotation angle. Training proceeds in two phases: a Search Phase in which variational parameters and architecture parameters are jointly optimized, followed by a Production Phase in which each is discretized to the nearest canonical gate by Euclidean distance and frozen for the remainder of training.
Full and Baseline Models
The Full model optimizes VQC, ANO Hermitian, and DiffQAS architecture parameters jointly. The Baseline model uses a fixed RY-CRX circuit with fixed PauliX/Y/Z measurements on all qubits (24 observables for 8 qubits). A Classical model replaces the quantum core with a standard nn.TransformerEncoder for benchmarking; Figure 1 overviews the pipeline and how the four ablation conditions differ in ansatz and measurement.
Figure 1. Overview of the 2 × 2 ablation design. Top: The QTSTransformer pipeline, with the two variable components highlighted—Ansatz (gate topology, green) and Measurement (observables, orange). Middle-left: DiffQAS’s gate search. Middle-right: ANO’s learnable observables. Bottom: The four ablation conditions from the factorial combination of DiffQAS and ANO.
This Classical model is a matched-protocol baseline, not a parameter-matched or separately tuned classical RL agent: it shares the same CNN feature extractor, replay buffer, -greedy schedule, and Double DQN loop as the quantum conditions, so only the temporal module (VQC vs. nn.TransformerEncoder) changes. The low classical CartPole score (132.6) is thus the ceiling of this matched-protocol transformer head under our budget, not a statement about classical RL on CartPole—a standard, tuned classical DQN or PPO solves CartPole-v1 to near-maximal return; we simply did not run one. Our claim is that the quantum core outperforms a matched classical core under the same training protocol and budget, with far fewer parameters, not that it beats the best achievable classical agent.
All conditions use 8 qubits, 2 ansatz layers, and QSVT degree 2, optimized with Adam (batch 64/32 for classic control/Atari); VQC/ANO/DiffQAS learning rates are 0.001/0.01/0.001 (classic control) and 0.00025/0.01/0.001 (Atari), with a DiffQAS search phase of 50/100 episodes, discount factor , schedule 1.0→0.01 (decay 0.995) / 1.0→0.1 (decay 0.99), replay memory 10,000/100,000, and target-network sync every 50/1,000 steps.
Computational Infrastructure
We used PyTorch (v2.5.0) and PennyLane [24] (v0.42.3) to implement our RL agents. All experiments ran on the Perlmutter supercomputer (NERSC, Lawrence Berkeley National Laboratory) on NVIDIA A100-80GB GPU nodes.
Results
We evaluate the QSTransformer agent across six reinforcement learning environments spanning two complexity regimes: classic control (CartPole-v1, FrozenLake-v1) and Atari games (SpaceInvaders, Breakout, DonkeyKong, MarioBros). For each environment, we compare four conditions in a 2 × 2 factorial ablation design—Baseline, ANO-only, DiffQAS-only, Full (ANO + DiffQAS)—plus a matched classical transformer baseline. All quantum conditions use 8 qubits, 2 ansatz layers, and QSVT degree 2. Performance is measured by the best 100-episode rolling-average return (BestAvgR), with Atari scores computed after the first 500 episodes to exclude exploration artifacts.
Quantum vs. Classical Performance
Table 1 presents the complete results across all six environments and five experimental conditions (bold/underline legend in the caption).
| Environment | Baseline | ANO-only | DiffQAS-only | Full | Classical | Best Q vs. C Advantage | Best-Q Retention % | |
| Classic Control | ||||||||
| CartPole-v1 | 2 | 223.2 | 347.7 | 24.8 | 359.9 | 132.6 | +171% | 52.1 |
| FrozenLake-v1 | 4 | 56% | 49% | 56% | 51% | 76% | −26% | — |
| Atari | ||||||||
| SpaceInvaders | 6 | 339.2 | 347.0 | 311.8 | 347.9 | 290.8 | +19.6% | 80.9 |
| Breakout | 4 | 3.0 | 2.7 | 17.2 | 11.1 | 2.5 | +588% | 92.9 |
| DonkeyKong | 18 | 37.0 | 43.0 | 70.0 | 25.0 | 38.0 | +84% | 80.0 |
| MarioBros | 18 | 360.0 | 296.0 | 344.0 | 336.0 | 304.0 | +18% | 40.0 |
Table 1. Performance comparison across all environments and ablation conditions. Values report the best 100-episode rolling-average return (BestAvgR); for FrozenLake, values indicate the success rate. Retention %: end-of-training value as a percentage of the bolded (best-quantum) condition’s own peak (FrozenLake excluded — not its story). Bold: best quantum condition per environment. Underline: overall best per environment. : action space size. Each value is point estimate from a single training run.
The quantum agent outperforms the matched classical transformer on five of six environments (Table I). The percentage gains over classical are +588% for Breakout and +84% for DonkeyKong; both are computed against small classical baselines and should be read alongside the absolute scores in the table. Because BestAvgR reports an achieved peak rather than a retained value, we also report each best-quantum condition’s end-of-training retention (Table I, last column): Breakout, DonkeyKong, and SpaceInvaders peaks are durable (80–93% retained), but CartPole and especially MarioBros show substantial forgetting (52.1% and 40.0% retained, respectively). DonkeyKong and MarioBros—the two 18-action games—also show the highest late-training variance (coefficient of variation 1.34–2.53 and 2.55–3.82 across the four quantum conditions, respectively, versus 0.22–1.18 for the other four environments), consistent with a genuine post-exploration decline in both games rather than a stable plateau. The quantum model also uses substantially fewer parameters throughout.
The sole exception is FrozenLake, where the classical transformer achieves 76% success vs. best quantum 56%. FrozenLake is the only environment with discrete observations (16-dimensional one-hot) and inherent stochasticity (slip probability 2/3). The classical transformer’s larger parameter budget (∼2K vs. ∼1.2K) is well-suited to such tabular-like stochastic mappings, whereas the quantum circuit’s rotation-angle encoding aligns more naturally with continuous states.
The episode at which best-quantum performance peaks varies widely by game (DonkeyKong ∼1,200 and MarioBros ∼760 to SpaceInvaders ∼4,900 and Breakout ∼7,100; full spread ∼565–8,076 across the sixteen Atari quantum configurations), yet the classical transformer, trained for up to 10,000 episodes, surpasses the quantum peak on no Atari game. We suspect this owes something to the VQC’s Hilbert-space structure acting as a useful inductive bias [25], [26], but the quantum model also has far fewer parameters, and a smaller model is often just easier to optimize; separating these factors would need a parameter-matched classical control, which we did not build here.
Effect of DiffQAS and ANO
DiffQAS: Circuit Architecture Search
DiffQAS shows highly task-dependent effects (Table I): on Breakout, DiffQAS-only shows a monotonic climb that separates from other conditions early in training; on DonkeyKong, it achieves the largest absolute Atari gain in the study. In both cases the search discretizes to RY/CRY rather than the baseline’s RY/CRX; the evidence below indicates this reflects the fixed CRY gate rather than an adaptively discovered structure.
Across all six environments the search discretized to RY/CRY. On CartPole this topology is detrimental (24.8 vs. baseline 223.2, near random); on SpaceInvaders, FrozenLake, and MarioBros it gives no measurable improvement. DiffQAS therefore does not universally improve performance: it converges to a fixed alternative topology that benefits tasks for which the default RY-CRX is mismatched, and is harmful when the default is already well-suited.
This convergence to RY/CRY—the search’s own initialization—held tightly: each gate slot’s architecture parameters moved, on average, only 1.6% of the distance to the nearest alternative gate reference (over all 144 slots of the DiffQAS-only and Full conditions across the six environments; exactly 0% for Breakout). Because every peak is reached in the discretized production phase, the DiffQAS-only advantage over the RY-CRX baseline reduces to a fixed RY/CRY vs. RY-CRX comparison under identical readout—it reflects the CRY entangling gate rather than an adaptively discovered structure, so the outcome is governed by search-space design and initialization, not task-specific discovery. Two caveats bound this: we log only the converged parameters, not the search trajectory (a small endpoint distance is consistent with either never moving or moving and returning—certain only for Breakout at 0.0000), and we ran no from-scratch fixed-RY/CRY control (so the search machinery need not be inert, only that it discovered nothing task-specific here).
b) ANO: Learnable Observables.
ANO shows a complementary pattern (Table I): it gives the largest gain on CartPole, and on SpaceInvaders a modest peak gain accompanies a +11% improvement in last-100-episode average, indicating better late-training stability. A similar end-of-training pattern appears on MarioBros (+61% in the last-100-episode window despite a lower peak).
ANO does not help on Breakout (2.7 vs. 3.0), where DiffQAS-only’s large gain (17.2 vs. 3.0 baseline) instead points to the gate topology as the bottleneck. On FrozenLake, neither ANO-only (49%) nor DiffQAS-only (56%) beats the baseline (56%); as discussed above, its discrete, stochastic observations more likely suit neither quantum intervention rather than isolating a specific measurement or topology limitation.
Together, ANO and DiffQAS act on complementary parts of the circuit: ANO optimizes what is measured, benefiting continuous-state tasks with suboptimal default Pauli measurements; DiffQAS optimizes how entanglement is generated, benefiting tasks whose reward landscape rewards a different connectivity.
c) Combined Model (Full = ANO + DiffQAS).
Combining ANO and DiffQAS exhibits an asymmetric interaction that depends on which innovation dominates in isolation.
When ANO is the dominant innovation, Full matches or slightly exceeds ANO-only (Table I): on CartPole, Full is the highest quantum score even though DiffQAS-only alone is near-random; on SpaceInvaders, Full is essentially tied with ANO-only. The parametric Rot/CRot gates active during Phase 1 plausibly supply additional gradient pathways that aid ANO’s measurement optimization.
When DiffQAS is the dominant innovation, Full degrades. On Breakout, Full (11.1) falls below DiffQAS-only (17.2); on DonkeyKong, Full (25.0) falls below both DiffQAS-only (70.0) and the baseline (37.0). Adding ANO’s 128 Hermitian parameters and separate optimizer appears to interfere with the converged CRY topology in these reward landscapes.
When neither innovation helps in isolation (MarioBros, FrozenLake), Full is at or below the baseline (336.0 vs. 360.0; 51% vs. 56%).
The asymmetry has a practical reading: ANO adjusts a downstream readout largely decoupled from the circuit, so it is robust to DiffQAS, whereas DiffQAS modifies the circuit itself and is more vulnerable to interference from ANO’s concurrently optimized degrees of freedom.
Parameter Efficiency
| Component | Quantum | Classical |
| Atari | ||
| CNN feature extractor | 1,946K | 1,946K |
| Transformer / VQC | ~12K | ~150K |
| Classic Control | ||
| Feature projection | <1K | <1K |
| Transformer / VQC | ~1.2K | ~2K |
| Environments won (of 6) | 5 | 1 |
Table 2. Parameter counts for the quantum and classical transformer architectures; the quantum model uses 12.5× fewer transformer-level parameters than the classical model on Atari. Transformer/VQC rows include +128 ANO and +24 architecture parameters (12 gate slots × two Euler angles: four slots in each of the two timestep ansatz layers and four in the quantum feed-forward block).
Table II compares parameter counts across architectures. On Atari, the quantum model shares its CNN feature extractor with the classical nn.TransformerEncoder (, 4 heads, 1 layer, ) (Table II), isolating its 12.5× parameter reduction to the temporal processing module; despite this reduction, it outperforms the classical baseline on all four Atari games.
One candidate explanation is the quantum circuit’s inductive bias: an 8-qubit circuit operates in a -dimensional Hilbert space, so the state space it acts on grows exponentially with qubit count even though the parameter count grows linearly; we offer this as a hypothesis rather than a demonstrated mechanism, since our experiments do not isolate it. The QSVT polynomial mixing supplies a temporal inductive bias different from self-attention. Model size and architectural differences remain confounded with this explanation in our design, so we read the quantum-versus-classical result as a parameter-efficiency comparison rather than as evidence of a quantum inductive-bias advantage, and we claim no asymptotic or wall-clock speedup.
Conclusion
We presented an ablation of ANO and DiffQAS within the QTSTransformer across six RL environments. Three findings stood out: the QTSTransformer outperformed a matched classical transformer on 5 of 6 environments while using substantially fewer parameters; ANO and DiffQAS provided complementary gains, with ANO strongest on continuous-state tasks and DiffQAS strongest on Atari; and combining them showed an asymmetric interaction, with ANO-dominant tasks tolerating DiffQAS while DiffQAS-dominant tasks degraded when ANO was added.
These findings offer practical guidance: rather than optimizing all circuit components simultaneously, practitioners should diagnose whether the bottleneck is the measurement basis or the gate topology and apply ANO or DiffQAS accordingly—ANO as the safer default on continuous-state tasks since it tolerates DiffQAS, DiffQAS alone when the default topology is mismatched to the reward landscape.
All results are obtained in noiseless state-vector simulation: we do not model finite-shot readout, so the reported expectation values are exact rather than sampled, and shot noise could change the comparisons above. Several directions extend this work. First, evaluating the framework on noisy intermediate-scale quantum (NISQ) hardware would clarify how gate noise, decoherence, and finite-shot readout interact with the ANO and DiffQAS optimizations. Second, matched-output controls (e.g., eight fixed 2-local Hermitians) would help isolate ANO’s adaptivity contribution from its locality and output-count changes. Third, expanding the DiffQAS search space to include genuine and richer entangling primitives, with longer search phases and varied initializations, would better test whether it performs genuine task-specific discovery rather than remaining near initialization. Fourth, complementing BestAvgR with area-under-the-curve summaries and multiseed confidence intervals would further clarify late-training behavior and the statistical robustness of the differences we report, motivating anti-forgetting mechanisms—such as adaptive target-network schedules or quantum-aware regularization—for the substantial forgetting observed on CartPole.
Acknowledgment
The authors extend their gratitude to the members of the Connectome Lab for their invaluable support and critical contributions to the development of this study. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1C1C1006503, RS-2023-00266787, RS-2023-00265406, RS-2024-00421268, RS-2024-00342301, RS-2024-00435727, RS-2025-25457239, RS-2021-NR061370, NRF-2021M3E5D2A01022515, and NRF-2021S1A3A2A02090597), by the Researchers Program through Seoul National University (No. 200-20250071, 200-20250049, 200-20250116, 200-20250115, 200-20250113, 0670-20250039, 200-20260009). Additional support was provided by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS-2021-II211343 Artificial Intelligence Graduate School Program, Seoul National University] and by the Global Research Support Program in the Digital Field (RS-2024-00421268). This work was also supported by the Artificial Intelligence Industrial Convergence Cluster Development Project funded by the Ministry of Science and ICT and Gwangju Metropolitan City, by the Korea Brain Research Institute (KBRI) basic research program (25-BR-05-01), by the Korea Health Industry Development Institute (KHIDI) and the Ministry of Health and Welfare, Republic of Korea (HR22C1605), and by the Korea Basic Science Institute (National Research Facilities and Equipment Center) grant funded by the Ministry of Education (RS-2024-00435727). We acknowledge the National Supercomputing Center for providing supercomputing resources and technical support (KSC-2023-CRE-0568, KSC-2024-CRE-0198, KSC-2025-CRE-0340). An award for computer time was provided by the U.S. Department of Energy’s (DOE) ASCR Leadership Computing Challenge (ALCC). This research used resources of the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility, under ALCC award m4750-2024, and supporting resources at the Argonne and Oak Ridge Leadership Computing Facilities, U.S. DOE Office of Science user facilities at Argonne National Laboratory and Oak Ridge National Laboratory.
References
- [1]J. Kaldari, S. Tariq, S. Al-Kuwari, S. Y.-C. Chen, S. Chatzinotas, and H. Shin, “Quantum reinforcement learning: Recent advances and future directions,” 2025.arxiv.org/abs/2510.14595
- [2]S. Y.-C. Chen, C.-H. H. Yang, J. Qi, P.-Y. Chen, X. Ma, and H.-S. Goan, “Variational quantum circuits for deep reinforcement learning,” IEEE Access, vol. 8, pp. 141007–141024, 2020.arxiv.org/abs/1907.00397
- [3]A. Skolik, S. Jerbi, and V. Dunjko, “Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning,” Quantum, vol. 6, p. 720, May 2022.DOI
- [4]O. Lockwood and M. Si, “Playing Atari with hybrid quantum-classical reinforcement learning,” in NeurIPS 2020 Workshop on Pre-registration in Machine Learning (L. Bertinetto, J. F. Henriques, S. Albanie, M. Paganini, and G. Varol, eds.), vol. 148 of Proceedings of Machine Learning Research, pp. 285–301, PMLR, 11 Dec 2021.arxiv.org/abs/2107.04114
- [5]H. Hohenfeld, D. Heimann, F. Wiebe, and F. Kirchner, “Quantum deep reinforcement learning for robot navigation tasks,” IEEE Access, vol. 12, pp. 87217–87236, 2024.DOI
- [6]S. Dutta, N. Innan, A. Marchisio, S. B. Yahia, and M. Shafique, “QADN: Quantum attention deep q-network for financial market prediction,” in 2024 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 02, pp. 341–346, 2024.DOI
- [7]H.-Y. Lin, H.-H. Tseng, S. Y.-C. Chen, and S. Yoo, “Adaptive non-local observable on quantum neural networks,” in 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 01, pp. 1884–1893, 2025.arxiv.org/abs/2504.13414
- [8]H.-Y. Lin, S. Y.-C. Chen, H.-H. Tseng, and S. Yoo, “Quantum reinforcement learning by adaptive non-local observables,” in 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 02, pp. 241–246, 2025.arxiv.org/abs/2507.19629
- [9]S. Y.-C. Chen, “Differentiable Quantum Architecture Search in Asynchronous Quantum Reinforcement Learning,” in 2024 IEEE International Conference on Quantum Computing and Engineering (QCE), (Los Alamitos, CA, USA), pp. 1516–1524, IEEE Computer Society, Sept. 2024.arxiv.org/abs/2407.18202
- [10]S.-X. Zhang, C.-Y. Hsieh, S. Zhang, and H. Yao, “Differentiable quantum architecture search,” Quantum Science and Technology, vol. 7, p. 045023, aug 2022.DOI
- [11]Y. Sun, Y. Ma, and V. Tresp, “Differentiable quantum architecture search for quantum reinforcement learning,” in 2023 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 02, pp. 15–19, 2023.arxiv.org/abs/2309.10392
- [12]J. J. Park, J. Seo, S. Bae, S. Y.-C. Chen, H.-H. Tseng, J. Cha, and S. Yoo, “Resting-state fMRI analysis using quantum time-series transformer,” in 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 01, pp. 2352–2363, 2025.arxiv.org/abs/2509.00711
- [13]R. Sutton and A. Barto, Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning series, MIT Press, 2018.
- [14]V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, p. 529–533, 2015.
- [15]S. Y.-C. Chen, “An introduction to quantum reinforcement learning (QRL),” in 2024 15th International Conference on Information and Communication Technology Convergence (ICTC), pp. 1139–1144, 2024.arxiv.org/abs/2409.05846
- [16]H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, p. 2094–2100, AAAI Press, 2020.
- [17]A. M. Childs and N. Wiebe, “Hamiltonian simulation using linear combinations of unitary operations,” Quantum Information & Computation, vol. 12, no. 11–12, p. 901–924, 2012.
- [18]A. Gilyén, Y. Su, G. H. Low, and N. Wiebe, “Quantum singular value transformation and beyond: exponential improvements for quantum matrix arithmetics,” in Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, (New York, NY, USA), p. 193–204, Association for Computing Machinery, 2019.DOI
- [19]S. Y.-C. Chen, H.-H. Tseng, H.-Y. Lin, and S. Yoo, “Learning to measure quantum neural networks,” in 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 1–5, 2025.arxiv.org/abs/2501.05663
- [20]S. Y.-C. Chen and P. Tiwari, “Quantum long short-term memory with differentiable architecture search,” in 2025 IEEE International Conference on Quantum Artificial Intelligence (QAI), pp. 13–18, 2025.arxiv.org/abs/2508.14955
- [21]M. Towers, A. Kwiatkowski, J. U. Balis, G. D. Cola, T. Deleu, M. Goulão, K. Andreas, M. Krimmel, A. KG, R. D. L. Perez-Vicente, J. K. Terry, A. Pierré, S. V. Schulhoff, J. J. Tai, H. Tan, and O. G. Younis, “Gymnasium: A standard interface for reinforcement learning environments,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.arxiv.org/abs/2407.17032
- [22]M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research, vol. 47, pp. 253–279, jun 2013.
- [23]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017.
- [24]V. Bergholm, J. Isaac, M. Schuld, C. Gogolin, S. Ahmed, V. Ajith, et al., “PennyLane: Automatic differentiation of hybrid quantum-classical computations,” 2022.
- [25]M. Schuld, R. Sweke, and J. J. Meyer, “Effect of data encoding on the expressive power of variational quantum-machine-learning models,” Phys. Rev. A, vol. 103, p. 032430, Mar 2021.DOI
- [26]A. Abbas, D. Sutter, C. Zoufal, A. Lucchi, A. Figalli, and S. Woerner, “The power of quantum neural networks,” Nature Computational Science, vol. 1, no. 6, p. 403–409, 2021.DOI
