betaFeedback

WorksComputer Science

Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback

Abstract

Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.

Introduction

The open radio access network (O-RAN) paradigm promotes the disaggregation and virtualization of radio access functions, enabling the deployment of artificial intelligence (AI)-driven control through the near-real-time RAN Intelligent Controller (RIC) [1, 2]. Reinforcement learning (RL) has emerged as a promising tool to optimize such control processes, allowing agents to adaptively manage radio resources, slicing, and admission policies in highly dynamic environments [3, 4, 5].

However, while the use of RL for O-RAN control has been widely studied in simulation, practical deployments face a critical yet often overlooked challenge, i.e., stochastic delay in feedback signals. When an RL agent issues a control action (e.g., modifying physical resource block allocations or scheduling priorities), its effect on observable performance metrics such as throughput or latency does not manifest immediately due to queuing, MAC scheduling, and telemetry reporting cycles [6, 7]. From the perspective of learning, these effects appear as delayed rewards, where the agent receives feedback corresponding to a past action after an uncertain

X. Wu and J. Farooq are with the Department of Electrical and Computer Engineering, University of Michigan-Dearborn, MI 48128 USA, Emails: {xingqiwu, mjfarooq}@umich.edu.

J. Chen is with the Department of Computer and Information Science, Fordham University, New York, NY USA, Email: [email protected].

T. Li is with the Department of Systems Engineering, City University of Hong Kong, Hong Kong, Email: [email protected].

Y. Wang is with the Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ USA, Email: [email protected].

interval. Such delays distort the temporal association between actions and outcomes, corrupt the replay buffer, and degrade policy convergence.

Empirical evidence of this phenomenon is shown in Figure 1, based on measurements from the Open AI Cellular (OAIC) O-RAN testbed [6]. When the PRB allocation is reduced to zero, the downlink data rate does not decrease instantaneously but only after some delay, highlighting the stochastic feedback latency inherent in real O-RAN control loops. Prior works on RL-driven O-RAN control have studied problems such as network slice admission [3], admission among heterogeneous slices [8], throughput-maximizing admission in edge computing [9], and multi-agent O-RAN resource allocation [4]. However, most of these approaches rely on offline training in simulators where feedback is synchronous. This assumption does not hold in real deployments, where delays are inevitable. On the other hand, the RL theory community has investigated delayed rewards in online learning and bandit settings [10, 11], off-policy learning with delayed feedback [12], and reward decomposition methods [13, 14]. These studies demonstrate fundamental impacts of delayed feedback, but generally assume the delay is fixed or provided to the learner. Such assumptions are impractical in O-RAN, where delays vary unpredictably across implementations and must be inferred from observed system behavior.

Empirical observation of delayed feedback in the OAIC O-RAN testbed

Figure 1. Empirical observation of delayed feedback in the OAIC O-RAN testbed. The PRB allocation is reduced to zero at time tt, but the downlink data rate only drops after a delay interval dtd_t.

To address this limitation, this letter introduces a delay-aware deep reinforcement learning (DRL) framework for O-RAN, designed to learn effectively under stochastic and unpredictable reward delays. The proposed framework integrates an LSTM-based delay predictor with an off-policy RL agent to realign rewards with their originating state–action pairs before policy updates. This enables the agent to recover the correct temporal credit assignment despite delayed feedback. The key contributions of this work are as follows:

  • We formulate O-RAN control as a Markov decision process (MDP) with stochastic delayed rewards, capturing the effect of non-Markovian feedback caused by network control latency.

  • We propose a delay-aware deep RL framework that uses an LSTM-based predictor to estimate reward delay and perform temporal realignment during off-policy training.

  • We validate the framework using delay traces derived from the OAIC testbed, demonstrating superior stability and performance over conventional RL baselines.

System Model and Problem Formulation

We consider an O-RAN downlink system where a near-RT RIC employs a RL agent to optimize radio resource management across heterogeneous user slices. The network consists of a centralized unit (O-CU), a distributed unit (O-DU), and a radio unit (O-RU) interconnected via the F1 and E2 interfaces. The near-RT RIC communicates control actions to the O-DU through the E2 interface. The O-DU executes physical-layer and MAC-layer scheduling decisions, which in turn affect the transmission performance experienced by the user equipments (UEs).

Let M={1,2,…,M}\mathcal{M} = \{1, 2, \ldots, M\} denote the set of active UEs, each belonging to either the enhanced Mobile Broadband (eMBB) or ultra-reliable low-latency communication (URLLC) slice, represented by Me\mathcal{M}_{e} and Mu\mathcal{M}_{u}, respectively. Each slice type enforces a distinct maximum tolerable latency τemax⁡\tau_{e}^{\max} or τumax⁡\tau_{u}^{\max} according to its quality-of-service (QoS) class.

At each decision epoch t∈T={0,…,Tn−1}t \in\mathcal{T} = \{0,\ldots,T_{n}-1\}, the near-RT RIC observes a network state vector st={τˉe(t),τˉu(t),bˉe(t),bˉu(t)}s_{t} = \{\bar{\tau}_{e}(t), \bar{\tau}_{u}(t), \bar{b}_{e}(t), \bar{b}_{u}(t)\}, where τˉx(t)\bar{\tau}_{x}(t) and bˉx(t)\bar{b}_{x}(t) represent the average latency and throughput for slice x∈{e,u}x \in\{e,u\}. Based on sts_{t}, the agent issues a control action at={Bm(t),Cm(t)}m∈Ma_{t} = \{B_{m}(t), C_{m}(t)\}_{m\in\mathcal{M}}, where Bm(t)B_{m}(t) and Cm(t)C_{m}(t) represent the bandwidth and computational resource allocated to UE mm, subject to ∑m∈MBm(t)≤B\sum_{m\in\mathcal{M}} B_{m}(t) \leq B and ∑m∈MCm(t)≤C\sum_{m\in\mathcal{M}} C_{m}(t) \leq C.

Latency and Queueing Model

Each UE generates data packets that must be transmitted and processed before their latency deadline. The end-to-end latency τm(t)\tau_{m}(t) is composed of transmission and computation delays:

τm(t)=τmtx(t)+τmcomp(t).(1)\tau_{m}(t) = \tau_{m}^{\mathrm{tx}}(t) + \tau_{m}^{\mathrm{comp}}(t). \tag*{(1)}

Both components are determined by the service rates induced by allocated bandwidth and compute capacity. Let δm(t)\delta_{m}(t) and Nm(t)N_{m}(t) denote the average packet size and arrival rate (packets per slot) for UE mm. The instantaneous achievable data rate, given by the Shannon model, is

bm(t)=Bm(t)log⁡2(1+P∣hm(t)∣2βm(t)Bm(t)N0+Im(t)),(2)b_{m}(t) = B_{m}(t)\log_{2}\left(1 + \frac{P|h_{m}(t)|^{2}\beta_{m}(t)}{B_{m}(t)N_{0} + I_{m}(t)}\right), \tag*{(2)}

where PP is the transmit power, ∣hm(t)∣2|h_{m}(t)|^{2} the small-scale fading gain, βm(t)\beta_{m}(t) the large-scale path loss, N0N_{0} the noise spectral density, and Im(t)I_{m}(t) the interference term. The transmission and computation queues evolve as

Qmtx(t+1)=max⁡[Qmtx(t)−bm(t)Ts,0]+δm(t)Nm(t),(3)Q_{m}^{\mathrm{tx}}(t+1) = \max\left[Q_{m}^{\mathrm{tx}}(t) - b_{m}(t)T_{s}, 0\right] + \delta_{m}(t)N_{m}(t), \tag*{(3)}
Qmcomp(t+1)=max⁡[Qmcomp(t)−Cm(t)Ts,0]+Qmtx(t+1),(4)Q_{m}^{\mathrm{comp}}(t+1) = \max[Q_{m}^{\mathrm{comp}}(t) - C_{m}(t)T_{s}, 0] + Q_{m}^{\mathrm{tx}}(t+1), \tag*{(4)}

where TsT_{s} is the scheduling interval. The corresponding average latencies are approximated via Little’s law as

τmtx(t)≈Qmtx(t)bm(t),τmcomp(t)≈Qmcomp(t)Cm(t).(5)\tau_{m}^{\mathrm{tx}}(t) \approx\frac{Q_{m}^{\mathrm{tx}}(t)}{b_{m}(t)}, \qquad\tau_{m}^{\mathrm{comp}}(t) \approx\frac{Q_{m}^{\mathrm{comp}}(t)}{C_{m}(t)}. \tag*{(5)}

These queue dynamics form the environment’s transition function P(s′∣s,a)\mathcal{P}(s'|s,a) that governs the system evolution perceived by the RL agent.

Modeling Feedback Delay in O-RAN

In real O-RAN systems, the effect of a control decision does not appear instantaneously in telemetry feedback. When the near-RT RIC applies an action ata_{t}, its measurable impact on system KPIs (e.g., throughput, latency) is observed after a stochastic delay dtd_{t}. Thus, the feedback at time tt corresponds to the system response under an earlier state-action pair at an effective time ℓ=max⁡(1,t−dt)\ell= \max(1,t-d_{t}), where dt∈D={0,1,…,D}d_{t} \in\mathcal{D} = \{0,1,\ldots,D\} represents the delay at time tt and has an unknown probability mass function Pr⁡(dt=k)=pk\Pr(d_{t}=k)=p_{k}, such that ∑k=0Dpk=1\sum_{k=0}^{D}p_{k}=1.

This asynchronous control–feedback loop introduces a stochastic temporal misalignment between the agent’s action and its corresponding reward signal, thereby violating the standard Markov property. If unaccounted for, such delay contaminates the replay buffer and destabilizes policy convergence. To enable delay-aware learning, the system must explicitly model this stochastic feedback process within the MDP formulation. The overall control and feedback process in the O-RAN architecture, showing the stochastic reward delay experienced across the E2 interface between the near-RT RIC and the O-DU/O-RU, is illustrated in Figure 2.

Control loop in O-RAN showing stochastic feedback delay between RIC-issued actions and observed rewards across the E2 interface

Figure 2. Control loop in O-RAN showing stochastic feedback delay between RIC-issued actions and observed rewards across the E2 interface.

Resource Update and Reliability Definition

For each resource type Rm(t)∈{Bm(t),Cm(t)}R_{m}(t) \in\{B_{m}(t), C_{m}(t)\}, the near-RT RIC applies incremental updates of the form

Rm(t)=Rm(t−1)+ρRϕtR^m(t)∑mR^m(t),(6)R_{m}(t) = R_{m}(t-1) + \rho R\phi^{t}\frac{\hat{R}_{m}(t)}{\sum_{m}\hat{R}_{m}(t)}, \tag*{(6)}

where ρ\rho is a step-size coefficient, ϕt\phi^{t} is the action increment chosen by the policy, and R^m(t)\hat{R}_{m}(t) denotes the normalized demand estimate computed as:

C^m(t)=δm(l)Nm(l)+(τmcomp(l−1)−Ts)++Cm(l−1)τmmax⁡,(7)\hat{C}_{m}(t) = \frac{\delta_{m}(l)N_{m}(l) + (\tau_{m}^{\mathrm{comp}}(l-1)-T_{s})^{+} + C_{m}(l-1)}{\tau_{m}^{\max}}, \tag*{(7)}
B^m(t)=δm(l)Nm(l)+(τmtx(l−1)−Ts)++bm(l−1)τmmax⁡,(8)\hat{B}_{m}(t) = \frac{\delta_{m}(l)N_{m}(l) + (\tau_{m}^{\mathrm{tx}}(l-1)-T_{s})^{+} + b_{m}(l-1)}{\tau_{m}^{\max}}, \tag*{(8)}

with (x)+=max⁡(0,x)(x)^{+}=\max(0,x). A minimum-adjustment mechanism rescales the allocations to satisfy the overall capacity constraints. The reliability for UE mm is defined as the probability that latency remains below its threshold:

P(τm(t)≤τmmax⁡)=∑kpkPr⁡[τm(t)≤τmmax⁡∣dt=k].(9)\mathbb{P}(\tau_{m}(t) \leq\tau_{m}^{\max}) = \sum_{k}p_{k}\Pr[\tau_{m}(t) \leq\tau_{m}^{\max}\mid d_{t}=k]. \tag*{(9)}

The system aims to maintain P(τm(t)≤τmmax⁡)≥ϱ\mathbb{P}(\tau_m(t) \le\tau_m^{\max}) \ge\varrho for all m∈Mm \in\mathcal{M}.

Reward Function and Optimization Objective

The instantaneous reward reflects both latency satisfaction and efficient resource utilization:

r(st,at)=1M∑m∈M[2Fab(τmmax⁡−τm(t))+νFcd(∑mrm(t))](10)r(s_t,a_t) = \frac{1}{M} \sum_{m \in\mathcal{M}} \left[2F_a^b(\tau_m^{\max} - \tau_m(t)) + \nu F_c^d\left(\sum_m r_m(t)\right)\right] \tag*{(10)}

where Fab(⋅):R→[0,1]F_a^b(\cdot): \mathbb{R} \to[0,1] and Fcd(⋅):R→[0,1]F_c^d(\cdot): \mathbb{R} \to[0,1] are smooth, strictly increasing functions parameterized by (a,b)(a,b) and (c,d)(c,d) respectively to approximate the indicator function, and ν∈[−1,0]\nu\in[-1,0] penalizes over-provisioning. rm(t)r_m(t) is the normalized resource utilization, evaluated as rm(t)=Bm(t)2B+Cm(t)2Cr_m(t) = \frac{B_m(t)}{2B} + \frac{C_m(t)}{2C}. The agent seeks to learn a stationary policy π(at∣st)\pi(a_t|s_t) that maximizes the expected discounted cumulative reward under stochastic feedback delays.

Delay-Aware Deep Reinforcement Learning Framework

Building on the stochastic delayed-reward formulation in § 2, this section presents the proposed Delay-Reward Alignment (DRA) framework for DRL in O-RAN control. The goal is to ensure robust and temporally consistent policy learning when the reward corresponding to each control action is received after a stochastic delay across the E2 interface.

MDP Formulation with Unknown Delayed Feedback

The environment is modeled as a Markov decision process with stochastic delayed feedback using the tuple (S,A,P,Rd,γ)(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}_d,\gamma), where S\mathcal{S} and A\mathcal{A} denote the state and action spaces, P(s′∣s,a)\mathcal{P}(s'|s,a) the transition kernel, Rd\mathcal{R}_d the delay-dependent reward function, and γ∈[0,1)\gamma\in[0,1) the discount factor. The stochastic delay horizon k=dtk=d_t is drawn from an unknown latent distribution D∗\mathcal{D}^*, and the expected cumulative delayed reward is

Rd(τt−k:t)=Ek∼D∗[∑i=0kγir(st+i−k,at+i−k)].(11)\mathcal{R}_d(\tau_{t-k:t}) = \mathbb{E}_{k \sim\mathcal{D}^*}\left[\sum_{i=0}^{k}\gamma^i r(s_{t+i-k},a_{t+i-k})\right]. \tag*{(11)}

This formulation generalizes standard off-policy learning by embedding unknown stochastic delay in the reward generation process.

State, Action, and Reward Specification

At each decision epoch tt, the near-RT RIC observes st=[C,B,1M∑mCm(t−1),1M∑mBm(t−1),1M∑mSINRmt−1,1M∑mδm(t−1),1M∑mτm(t−1)]s_t = \left[\mathbf{C},\mathbf{B},\frac{1}{M}\sum_m C_m(t-1),\frac{1}{M}\sum_m B_m(t-1),\frac{1}{M}\sum_m \mathrm{SINR}_m^{t-1},\frac{1}{M}\sum_m \delta_m(t-1),\frac{1}{M}\sum_m \tau_m(t-1)\right], where SINRmt=Pβm(t)∣hm(t)∣2Bm(t)N0+Im(t)\mathrm{SINR}_m^t = \frac{P\beta_m(t)|h_m(t)|^2}{B_m(t)N_0+I_m(t)}. The agent’s action ata_t updates the bandwidth and computational allocations {Bm(t),Cm(t)}\{B_m(t),C_m(t)\} for all m∈Mm \in\mathcal{M}, subject to total resource constraints ∑mBm(t)≤B\sum_m B_m(t) \le B and ∑mCm(t)≤C\sum_m C_m(t) \le C. The instantaneous reward follows the definition in (11), combining latency satisfaction and utilization efficiency, while the realized delayed reward at time tt is as follows: Rdt(τt−k:t∣k)=∑i=0kγir(st+i−k,at+i−k)\mathcal{R}_d^t(\tau_{t-k:t} \mid k) = \sum_{i=0}^{k}\gamma^i r(s_{t+i-k},a_{t+i-k}).

Delay Prediction and Reward Realignment

Figure 3 illustrates the integrated DRA training architecture that combines an LSTM-based delay predictor with an off-policy DRL orchestrator. To mitigate policy corruption caused by delayed rewards, the DRA framework integrates a DRL-based orchestrator and an LSTM-based delay predictor operating under strict modular separation. Three buffers are maintained: (1) H\mathcal{H} for raw trajectories, (2) G\mathcal{G} for observed delay history {kt}\{k_t\}, and (3) F\mathcal{F} for corrected replay transitions constructed using the predicted delay k^t\hat{k}_t.

Integrated training architecture for mitigating delayed reward corruption using LSTM-driven delay prediction and off-policy DRL

Figure 3. Integrated training architecture for mitigating delayed reward corruption using LSTM-driven delay prediction and off-policy DRL.

At each training iteration, the predictor uses past delay samples K={kt−1,…,kt−K}\mathcal{K} = \{k_{t-1},\ldots,k_{t-K}\} to estimate k^t\hat{k}_t, reassigning the delayed reward Rdt\mathcal{R}_d^t to the appropriate historical (st−k^t,at−k^t)(s_{t-\hat{k}_t},a_{t-\hat{k}_t}) pair. This temporally aligned transition is inserted into F\mathcal{F}, ensuring high-purity experiences for policy learning. The LSTM predictor is trained to minimize the mean-squared delay prediction error,

Ldelay=1∣M∣∑t∈M(dt−d^t)2(12)\mathcal{L}_{\mathrm{delay}} = \frac{1}{|\mathcal{M}|}\sum_{t \in\mathcal{M}}(d_t-\hat{d}_t)^2 \tag*{(12)}

while the DRL agent minimizes the mean-squared Bellman error:

L(θ)=E[(Qθ(st−k,at−k)−yt)2],(13)L(\theta) = \mathbb{E}\left[(Q_\theta(s_{t-k},a_{t-k})-y_t)^2\right], \tag*{(13)}

where yt=r^t+γQθ−(st−k+1,πθ−(st−k+1))y_t = \hat{r}_t + \gamma Q_{\theta^-}(s_{t-k+1},\pi_{\theta^-}(s_{t-k+1})) and r^t=rt+k^t\hat{r}_t = r_{t+\hat{k}_t}.

Algorithm 1 LSTM-Enhanced Off-Policy DRL for Delay-Aware Resource Orchestration

  1. Initialize: environment Pσ\mathcal{P}_{\sigma}, replay buffers H\mathcal{H}, F\mathcal{F}, G\mathcal{G}; policy πθ\pi_{\theta}, critic QϕQ_{\phi}, and LSTM predictor Fψ\mathcal{F}_{\psi}

  2. for each episode do

  3. for each time step tt do

  4. Observe sts_t; choose at∼πθ(st)a_t \sim\pi_{\theta}(s_t); apply ata_t to update {Bm,Cm}\{B_m,C_m\}

  5. Observe delayed reward Rtd(τt−kt:t∣kt)R^d_t(\tau_{t-k_t:t}\mid k_t), where kt∼Dk_t \sim\mathcal{D}

  6. Update delay predictor using past {ki}\{k_i\} and estimate k^t\hat{k}_t

  7. Construct corrected tuple (st−k^t,at−k^t,Rtd,st−k^t+1)(s_{t-\hat{k}_t},a_{t-\hat{k}_t},R^d_t,s_{t-\hat{k}_t+1})

  8. Store aligned tuple in F\mathcal{F} and true delay ktk_t in G\mathcal{G}

  9. Sample minibatch from F\mathcal{F}; update QϕQ_{\phi} and πθ\pi_{\theta} via gradient descent

  10. end for

  11. end for

  12. Return: optimized delay-aware policy π∗\pi^{*}

Online Training Procedure

The full online training process is summarized in Algorithm 1, which alternates between environment interaction, delay prediction, and policy update phases. This design decouples delay estimation from reinforcement updates, thereby preserving gradient stability under stochastic and non-Markovian delays. The resulting formulation is compatible with standard off-policy algorithms such as DQN, DDPG, and SAC, providing a plug-and-play extension for delay-aware training in near-RT RIC implementations.

Experimental Evaluation

This section evaluates the proposed DRA mechanism integrated into three popular off-policy RL algorithms, i.e., Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Soft Actor–Critic (SAC), for O-RAN control under stochastic reward delay. The experiments employ empirical delay traces from the OAIC platform, enabling realistic modeling of control–feedback latency between the near-RT RIC and O-DU.

Experimental Setup

Our experimental setup leveraged the OAIC testbed [6] and Python 3.7 with PyTorch 1.10.1 running on an NVIDIA GeForce MX350 GPU (8GB VRAM). Delay-response measurements were recorded from real-time OAIC experiments and discretized into time slots for training and evaluation. A delay model of the following form was empirically fitted on the empirical measurements:

dt=min⁡{D,Xt+Asin⁡(2πft)+Φ}d_t=\min\{D,X_t+A\sin(2\pi f t)+\Phi\}

where Xt∼Exp⁡(1/λ)X_t \sim\operatorname{Exp}(1/\lambda) with mean 1/λ1/\lambda seconds, while AA, ff, and Φ\Phi specify the amplitude, frequency, and constant shift of the periodic component, respectively. The total training cycle, including baseline and proposed models, spanned approximately 48 hours. Table I summarizes the key environment and learning parameters. For reward shaping, we employed sigmoid functions of the form

Lr ParamsValueEnv ParamsValue
KK15M,dm(t)M,d_m^{(t)} (km)4∼9, [0.4, 0.5]
αsoft−update\alpha_{\mathrm{soft-update}}1e-3CC (Mbps)1000
Ns1,Ns2,Ns3N_s^1,N_s^2,N_s^310, 10, 10fcf_c (GHz)3
μDRL,μLSTM\mu_{\mathrm{DRL}},\mu_{\mathrm{LSTM}}5e-5, 1e-3δm(t)\delta_m(t) (Mb)[0.003, 0.3]
SbatchDRLS_{\mathrm{batch}}^{\mathrm{DRL}}256ψξ\psi_{\xi} (dB)3.5
SbatchLSTMS_{\mathrm{batch}}^{\mathrm{LSTM}}32Tn,TsT_n,T_s (s)150, 0.2
HDDPGH_{\mathrm{DDPG}}{64, 64}τe\tau_e (s), τu\tau_u (s)0.1, 0.001
HDQNH_{\mathrm{DQN}}{25, 25}PP (dBm)32.3
HSACH_{\mathrm{SAC}}{64, 64}BB (MHz)1000
a,ba,b20, 5e-4Nm(t)N_m(t)[?, ?]
ν,ϱ,γ\nu,\varrho,\gamma-0.15, 0.99, 0.1η\eta (dB)0.1
HLSTMH_{\mathrm{LSTM}}{64, 64}N0N_0 (dBm/Hz)-163
SbufferS_{\mathrm{buffer}}5e5f,Φf,\Phi0.1/2π\pi, [3, 8]
c,dc,d10, 0.6λ,A\lambda,A[0.4, 1], [?, 5]

Table I. Simulation Parameters.

Fab(x)=11+exp⁡(−a(x−b))F_a^b(x)=\frac{1}{1+\exp(-a(x-b))}

where parameters aa and bb tune the sharpness and shift of the response. This flexibility enabled precise alignment of the reward signal with critical latency and utilization thresholds.

For each algorithm (DQN, SAC, and DDPG), we train two variants:

  • Baseline: the standard algorithm without delay compensation;

  • DRA-variant: the same algorithm enhanced with the proposed DRA framework described in Section III.

All agents are trained for 5×1045 \times10^4 steps with discount factor γ=0.95\gamma=0.95, learning rate 10−410^{-4}, and batch size 64. The LSTM-based delay predictor within DRA uses a single hidden layer with 64 units and a temporal window of 10 steps.

Performance is evaluated using the following metrics relevant to near-RT RIC operation: (i) Average episodic return, indicating policy convergence stability; (ii) Latency violation ratio, percentage of users exceeding τmmax⁡\tau_m^{\max}; (iii) Packet drop ratio, fraction of packets not delivered within deadlines; and (iv) Resource efficiency, throughput-to-bandwidth utilization ratio.

Results and Discussion

Figure 4 compares the convergence behavior of all six agents. In Figure 4(a), the DRA-enhanced algorithms (DRA-DQN, DRA-SAC, DRA-DDPG) exhibit smooth and monotonic improvement in expected return, whereas their non-DRA counterparts show slower or unstable convergence when stochastic delay is present. The advantage of the DRA mechanism is further evident in Figure 4(b), where all DRA variants maintain substantially lower packet-drop ratios across episodes, highlighting improved temporal credit assignment under delayed feedback.

Expected return and packet drop ratio training curves

Figure 4. Training rewards of resource orchestration agents under varying strategies.

The end-to-end latency distributions are characterized by the empirical complementary cumulative distribution functions (ECCDFs) shown in Figure 5. For eMBB users, as shown in Figure 5(a), DRA-augmented policies achieve tighter latency concentration, with 95th-percentile delays below 15 ms. For URLLC users, as shown in Figure 5(b), only the DRA variants maintain sub-1 ms latency at reliability above 99.9%, while standard baselines exhibit heavy tails due to uncorrected reward misalignment. These results confirm that DRA preserves QoS isolation across heterogeneous slices even under feedback uncertainty.

ECCDF curves for per-slice latency

Figure 5. ECCDF of per-slice latency for DQN, SAC, and DDPG with and without DRA compensation.

The system-level metrics in Figure 6 further reinforce this observation. Across increasing UE density, the DRA variants consistently achieve lower latency-violation and packet-drop ratios, as shown in Figure 6(a)–(Figure 6b), as well as higher resource efficiency and reliability shown in Figure 6(c)–(Figure 6d). Among all, DRA-SAC delivers the best trade-off between learning stability and resource efficiency, followed by DRA-DDPG and DRA-DQN. Quantitatively, the DRA framework reduces latency violations by up to 25% and improves spectral–computational efficiency by 12% compared with the non-compensated baselines.

Aggregate QoS metrics for increasing UE count comparing DQN, SAC, DDPG, and their DRA-enhanced counterparts

Figure 6. Aggregate QoS metrics for increasing UE count comparing DQN, SAC, DDPG, and their DRA-enhanced counterparts.

Conclusion

This letter addressed the challenge of stochastic reward delay in reinforcement learning (RL)-based control for O-RAN systems. We formulated the O-RAN resource allocation problem as an MDP with delayed rewards and proposed a delay-aware deep RL framework that integrates an LSTM-based delay predictor for temporal reward realignment. Using empirical delay traces derived from the OAIC testbed, the results show that the proposed method achieves faster convergence, higher reliability, and better resource efficiency than conventional RL baselines. Future work will explore multi-agent extensions for cooperative delay compensation and on-device deployment within near-RT RIC platforms.

References

  1. [1]B. Balasubramanian, E. S. Daniels, M. Hiltunen, R. Jana, K. Joshi, R. Sivaraj, T. X. Tran, and C. Wang, “RIC: A RAN intelligent controller platform for AI-enabled cellular networks,” IEEE Internet Computing, vol. 25, no. 2, pp. 7–17, 2021.DOI
  2. [2]L. Bonati, S. D’Oro, M. Polese, S. Basagni, and T. Melodia, “Intelligence and learning in O-RAN for data-driven NextG cellular networks,” IEEE Communications Magazine, vol. 59, no. 10, pp. 21–27, 2021.DOI
  3. [3]M. A. Haque and V. Kirova, “5G network slice admission control using optimization and reinforcement learning,” in IEEE Wireless Commun. Netw. Conf. (WCNC 2022), Austin, TX, USA, Apr. 2022.DOI
  4. [4]F. Rezazadeh, L. Zanzi, F. Devoti, S. Barrachina-Muñoz, E. Zeydan, X. Costa-Pérez, and J. Manges-Bafalluy, “A multi-agent deep reinforcement learning approach for RAN resource allocation in O-RAN,” in IEEE Conf. Comput. Commun. Wkshps, New York, USA, May. 2023.arxiv.org/abs/2307.02414
  5. [5]N. Frangoniotis, E. Stai, I. Dimolitsas, A. Zafeiropoulos, and S. Papavassiliou, “Dynamic, reconfigurable and green network slice admission control and resource allocation in the O-RAN using model predictive control,” in IFIP Netw. Conf., Thessaloniki, Greece, Jun. 2024.
  6. [6]P. S. Upadhyaya, N. Tripathi, J. Gaeddert, and J. H. Reed, “Open AI cellular (OAIC): An open source 5G O-RAN testbed for design and testing of AI-based RAN management algorithms,” IEEE Network, vol. 37, no. 5, pp. 7–15, 2023.DOI
  7. [7]E. Municio, G. Garcia-Aviles, A. Garcia-Saavedra, and X. Costa-Pérez, “O-ran: Analysis of latency-critical interfaces and overview of time sensitive networking solutions,” IEEE Communications Standards Magazine, vol. 7, no. 3, pp. 82–89, 2023.
  8. [8]M. Dai, G. Sun, H. Yu, and D. Niyato, “Maximize the long-term average revenue of network slice provider via admission control among heterogeneous slices,” IEEE/ACM Trans. Netw., vol. 32, no. 1, pp. 745–760, 2024.arxiv.org/abs/2304.10057
  9. [9]Y. Zhou, Q. Ye, H. Huang, and H. Du, “Deep reinforcement learning based admission control for throughput maximization in mobile edge computing,” in IEEE 94th Veh. Technol. Conf. (VTC-Fall 2021), Norman, OK, USA, Sept. 2021.DOI
  10. [10]P. Joulani, A. Gyorgy, and C. Szepesvari, “Online learning under delayed feedback,” in Proceedings of the 30th International Conference on Machine Learning (ICML 2013), S. Dasgupta and D. McAllester, Eds., vol. 28, Atlanta, Georgia, USA, 17–19 Jun 2013, pp. 1453–1461.arxiv.org/abs/1306.0686
  11. [11]K. Quanrud and D. Khashabi, “Online learning with adversarial delays,” in Advances in Neural Information Processing Systems (NeurIPS 2015), C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28, 2015.
  12. [12]B. Han, Z. Ren, Z. Wu, Y. Zhou, and J. Peng, “Off-policy reinforcement learning with delayed rewards,” in Proceedings of the 39th International Conference on Machine Learning (ICML 2022), Baltimore, MD, USA, Jul. 2022, pp. 8280–8303.arxiv.org/abs/2106.11854
  13. [13]W. U. Mondal and V. Aggarwal, “Reinforcement learning with delayed, composite, and partially anonymous reward,” Transactions on Machine Learning Research, 2023.arxiv.org/abs/2305.02527
  14. [14]J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter, “RUDDER: Return decomposition for delayed rewards,” in 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, Dec. 2019.arxiv.org/abs/1806.07857

Paper details

vlnc:W_5bjd3rh6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents