WorksComputer Science
Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback
Abstract
Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.
Introduction
The open radio access network (O-RAN) paradigm promotes the disaggregation and virtualization of radio access functions, enabling the deployment of artificial intelligence (AI)-driven control through the near-real-time RAN Intelligent Controller (RIC) [1, 2]. Reinforcement learning (RL) has emerged as a promising tool to optimize such control processes, allowing agents to adaptively manage radio resources, slicing, and admission policies in highly dynamic environments [3, 4, 5].
However, while the use of RL for O-RAN control has been widely studied in simulation, practical deployments face a critical yet often overlooked challenge, i.e., stochastic delay in feedback signals. When an RL agent issues a control action (e.g., modifying physical resource block allocations or scheduling priorities), its effect on observable performance metrics such as throughput or latency does not manifest immediately due to queuing, MAC scheduling, and telemetry reporting cycles [6, 7]. From the perspective of learning, these effects appear as delayed rewards, where the agent receives feedback corresponding to a past action after an uncertain
X. Wu and J. Farooq are with the Department of Electrical and Computer Engineering, University of Michigan-Dearborn, MI 48128 USA, Emails: {xingqiwu, mjfarooq}@umich.edu.
J. Chen is with the Department of Computer and Information Science, Fordham University, New York, NY USA, Email: [email protected].
T. Li is with the Department of Systems Engineering, City University of Hong Kong, Hong Kong, Email: [email protected].
Y. Wang is with the Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ USA, Email: [email protected].
interval. Such delays distort the temporal association between actions and outcomes, corrupt the replay buffer, and degrade policy convergence.
Empirical evidence of this phenomenon is shown in Figure 1, based on measurements from the Open AI Cellular (OAIC) O-RAN testbed [6]. When the PRB allocation is reduced to zero, the downlink data rate does not decrease instantaneously but only after some delay, highlighting the stochastic feedback latency inherent in real O-RAN control loops. Prior works on RL-driven O-RAN control have studied problems such as network slice admission [3], admission among heterogeneous slices [8], throughput-maximizing admission in edge computing [9], and multi-agent O-RAN resource allocation [4]. However, most of these approaches rely on offline training in simulators where feedback is synchronous. This assumption does not hold in real deployments, where delays are inevitable. On the other hand, the RL theory community has investigated delayed rewards in online learning and bandit settings [10, 11], off-policy learning with delayed feedback [12], and reward decomposition methods [13, 14]. These studies demonstrate fundamental impacts of delayed feedback, but generally assume the delay is fixed or provided to the learner. Such assumptions are impractical in O-RAN, where delays vary unpredictably across implementations and must be inferred from observed system behavior.

Figure 1. Empirical observation of delayed feedback in the OAIC O-RAN testbed. The PRB allocation is reduced to zero at time , but the downlink data rate only drops after a delay interval .
To address this limitation, this letter introduces a delay-aware deep reinforcement learning (DRL) framework for O-RAN, designed to learn effectively under stochastic and unpredictable reward delays. The proposed framework integrates an LSTM-based delay predictor with an off-policy RL agent to realign rewards with their originating state–action pairs before policy updates. This enables the agent to recover the correct temporal credit assignment despite delayed feedback. The key contributions of this work are as follows:
We formulate O-RAN control as a Markov decision process (MDP) with stochastic delayed rewards, capturing the effect of non-Markovian feedback caused by network control latency.
We propose a delay-aware deep RL framework that uses an LSTM-based predictor to estimate reward delay and perform temporal realignment during off-policy training.
We validate the framework using delay traces derived from the OAIC testbed, demonstrating superior stability and performance over conventional RL baselines.
System Model and Problem Formulation
We consider an O-RAN downlink system where a near-RT RIC employs a RL agent to optimize radio resource management across heterogeneous user slices. The network consists of a centralized unit (O-CU), a distributed unit (O-DU), and a radio unit (O-RU) interconnected via the F1 and E2 interfaces. The near-RT RIC communicates control actions to the O-DU through the E2 interface. The O-DU executes physical-layer and MAC-layer scheduling decisions, which in turn affect the transmission performance experienced by the user equipments (UEs).
Let denote the set of active UEs, each belonging to either the enhanced Mobile Broadband (eMBB) or ultra-reliable low-latency communication (URLLC) slice, represented by and , respectively. Each slice type enforces a distinct maximum tolerable latency or according to its quality-of-service (QoS) class.
At each decision epoch , the near-RT RIC observes a network state vector , where and represent the average latency and throughput for slice . Based on , the agent issues a control action , where and represent the bandwidth and computational resource allocated to UE , subject to and .
Latency and Queueing Model
Each UE generates data packets that must be transmitted and processed before their latency deadline. The end-to-end latency is composed of transmission and computation delays:
Both components are determined by the service rates induced by allocated bandwidth and compute capacity. Let and denote the average packet size and arrival rate (packets per slot) for UE . The instantaneous achievable data rate, given by the Shannon model, is
where is the transmit power, the small-scale fading gain, the large-scale path loss, the noise spectral density, and the interference term. The transmission and computation queues evolve as
where is the scheduling interval. The corresponding average latencies are approximated via Little’s law as
These queue dynamics form the environment’s transition function that governs the system evolution perceived by the RL agent.
Modeling Feedback Delay in O-RAN
In real O-RAN systems, the effect of a control decision does not appear instantaneously in telemetry feedback. When the near-RT RIC applies an action , its measurable impact on system KPIs (e.g., throughput, latency) is observed after a stochastic delay . Thus, the feedback at time corresponds to the system response under an earlier state-action pair at an effective time , where represents the delay at time and has an unknown probability mass function , such that .
This asynchronous control–feedback loop introduces a stochastic temporal misalignment between the agent’s action and its corresponding reward signal, thereby violating the standard Markov property. If unaccounted for, such delay contaminates the replay buffer and destabilizes policy convergence. To enable delay-aware learning, the system must explicitly model this stochastic feedback process within the MDP formulation. The overall control and feedback process in the O-RAN architecture, showing the stochastic reward delay experienced across the E2 interface between the near-RT RIC and the O-DU/O-RU, is illustrated in Figure 2.

Figure 2. Control loop in O-RAN showing stochastic feedback delay between RIC-issued actions and observed rewards across the E2 interface.
Resource Update and Reliability Definition
For each resource type , the near-RT RIC applies incremental updates of the form
where is a step-size coefficient, is the action increment chosen by the policy, and denotes the normalized demand estimate computed as:
with . A minimum-adjustment mechanism rescales the allocations to satisfy the overall capacity constraints. The reliability for UE is defined as the probability that latency remains below its threshold:
The system aims to maintain for all .
Reward Function and Optimization Objective
The instantaneous reward reflects both latency satisfaction and efficient resource utilization:
where and are smooth, strictly increasing functions parameterized by and respectively to approximate the indicator function, and penalizes over-provisioning. is the normalized resource utilization, evaluated as . The agent seeks to learn a stationary policy that maximizes the expected discounted cumulative reward under stochastic feedback delays.
Delay-Aware Deep Reinforcement Learning Framework
Building on the stochastic delayed-reward formulation in § 2, this section presents the proposed Delay-Reward Alignment (DRA) framework for DRL in O-RAN control. The goal is to ensure robust and temporally consistent policy learning when the reward corresponding to each control action is received after a stochastic delay across the E2 interface.
MDP Formulation with Unknown Delayed Feedback
The environment is modeled as a Markov decision process with stochastic delayed feedback using the tuple , where and denote the state and action spaces, the transition kernel, the delay-dependent reward function, and the discount factor. The stochastic delay horizon is drawn from an unknown latent distribution , and the expected cumulative delayed reward is
This formulation generalizes standard off-policy learning by embedding unknown stochastic delay in the reward generation process.
State, Action, and Reward Specification
At each decision epoch , the near-RT RIC observes , where . The agent’s action updates the bandwidth and computational allocations for all , subject to total resource constraints and . The instantaneous reward follows the definition in (11), combining latency satisfaction and utilization efficiency, while the realized delayed reward at time is as follows: .
Delay Prediction and Reward Realignment
Figure 3 illustrates the integrated DRA training architecture that combines an LSTM-based delay predictor with an off-policy DRL orchestrator. To mitigate policy corruption caused by delayed rewards, the DRA framework integrates a DRL-based orchestrator and an LSTM-based delay predictor operating under strict modular separation. Three buffers are maintained: (1) for raw trajectories, (2) for observed delay history , and (3) for corrected replay transitions constructed using the predicted delay .

Figure 3. Integrated training architecture for mitigating delayed reward corruption using LSTM-driven delay prediction and off-policy DRL.
At each training iteration, the predictor uses past delay samples to estimate , reassigning the delayed reward to the appropriate historical pair. This temporally aligned transition is inserted into , ensuring high-purity experiences for policy learning. The LSTM predictor is trained to minimize the mean-squared delay prediction error,
while the DRL agent minimizes the mean-squared Bellman error:
where and .
Algorithm 1 LSTM-Enhanced Off-Policy DRL for Delay-Aware Resource Orchestration
Initialize: environment , replay buffers , , ; policy , critic , and LSTM predictor
for each episode do
for each time step do
Observe ; choose ; apply to update
Observe delayed reward , where
Update delay predictor using past and estimate
Construct corrected tuple
Store aligned tuple in and true delay in
Sample minibatch from ; update and via gradient descent
end for
end for
Return: optimized delay-aware policy
Online Training Procedure
The full online training process is summarized in Algorithm 1, which alternates between environment interaction, delay prediction, and policy update phases. This design decouples delay estimation from reinforcement updates, thereby preserving gradient stability under stochastic and non-Markovian delays. The resulting formulation is compatible with standard off-policy algorithms such as DQN, DDPG, and SAC, providing a plug-and-play extension for delay-aware training in near-RT RIC implementations.
Experimental Evaluation
This section evaluates the proposed DRA mechanism integrated into three popular off-policy RL algorithms, i.e., Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Soft Actor–Critic (SAC), for O-RAN control under stochastic reward delay. The experiments employ empirical delay traces from the OAIC platform, enabling realistic modeling of control–feedback latency between the near-RT RIC and O-DU.
Experimental Setup
Our experimental setup leveraged the OAIC testbed [6] and Python 3.7 with PyTorch 1.10.1 running on an NVIDIA GeForce MX350 GPU (8GB VRAM). Delay-response measurements were recorded from real-time OAIC experiments and discretized into time slots for training and evaluation. A delay model of the following form was empirically fitted on the empirical measurements:
where with mean seconds, while , , and specify the amplitude, frequency, and constant shift of the periodic component, respectively. The total training cycle, including baseline and proposed models, spanned approximately 48 hours. Table I summarizes the key environment and learning parameters. For reward shaping, we employed sigmoid functions of the form
Table I. Simulation Parameters.
where parameters and tune the sharpness and shift of the response. This flexibility enabled precise alignment of the reward signal with critical latency and utilization thresholds.
For each algorithm (DQN, SAC, and DDPG), we train two variants:
Baseline: the standard algorithm without delay compensation;
DRA-variant: the same algorithm enhanced with the proposed DRA framework described in Section III.
All agents are trained for steps with discount factor , learning rate , and batch size 64. The LSTM-based delay predictor within DRA uses a single hidden layer with 64 units and a temporal window of 10 steps.
Performance is evaluated using the following metrics relevant to near-RT RIC operation: (i) Average episodic return, indicating policy convergence stability; (ii) Latency violation ratio, percentage of users exceeding ; (iii) Packet drop ratio, fraction of packets not delivered within deadlines; and (iv) Resource efficiency, throughput-to-bandwidth utilization ratio.
Results and Discussion
Figure 4 compares the convergence behavior of all six agents. In Figure 4(a), the DRA-enhanced algorithms (DRA-DQN, DRA-SAC, DRA-DDPG) exhibit smooth and monotonic improvement in expected return, whereas their non-DRA counterparts show slower or unstable convergence when stochastic delay is present. The advantage of the DRA mechanism is further evident in Figure 4(b), where all DRA variants maintain substantially lower packet-drop ratios across episodes, highlighting improved temporal credit assignment under delayed feedback.

Figure 4. Training rewards of resource orchestration agents under varying strategies.
The end-to-end latency distributions are characterized by the empirical complementary cumulative distribution functions (ECCDFs) shown in Figure 5. For eMBB users, as shown in Figure 5(a), DRA-augmented policies achieve tighter latency concentration, with 95th-percentile delays below 15 ms. For URLLC users, as shown in Figure 5(b), only the DRA variants maintain sub-1 ms latency at reliability above 99.9%, while standard baselines exhibit heavy tails due to uncorrected reward misalignment. These results confirm that DRA preserves QoS isolation across heterogeneous slices even under feedback uncertainty.

Figure 5. ECCDF of per-slice latency for DQN, SAC, and DDPG with and without DRA compensation.
The system-level metrics in Figure 6 further reinforce this observation. Across increasing UE density, the DRA variants consistently achieve lower latency-violation and packet-drop ratios, as shown in Figure 6(a)–(Figure 6b), as well as higher resource efficiency and reliability shown in Figure 6(c)–(Figure 6d). Among all, DRA-SAC delivers the best trade-off between learning stability and resource efficiency, followed by DRA-DDPG and DRA-DQN. Quantitatively, the DRA framework reduces latency violations by up to 25% and improves spectral–computational efficiency by 12% compared with the non-compensated baselines.

Figure 6. Aggregate QoS metrics for increasing UE count comparing DQN, SAC, DDPG, and their DRA-enhanced counterparts.
Conclusion
This letter addressed the challenge of stochastic reward delay in reinforcement learning (RL)-based control for O-RAN systems. We formulated the O-RAN resource allocation problem as an MDP with delayed rewards and proposed a delay-aware deep RL framework that integrates an LSTM-based delay predictor for temporal reward realignment. Using empirical delay traces derived from the OAIC testbed, the results show that the proposed method achieves faster convergence, higher reliability, and better resource efficiency than conventional RL baselines. Future work will explore multi-agent extensions for cooperative delay compensation and on-device deployment within near-RT RIC platforms.
References
- [1]B. Balasubramanian, E. S. Daniels, M. Hiltunen, R. Jana, K. Joshi, R. Sivaraj, T. X. Tran, and C. Wang, “RIC: A RAN intelligent controller platform for AI-enabled cellular networks,” IEEE Internet Computing, vol. 25, no. 2, pp. 7–17, 2021.DOI
- [2]L. Bonati, S. D’Oro, M. Polese, S. Basagni, and T. Melodia, “Intelligence and learning in O-RAN for data-driven NextG cellular networks,” IEEE Communications Magazine, vol. 59, no. 10, pp. 21–27, 2021.DOI
- [3]M. A. Haque and V. Kirova, “5G network slice admission control using optimization and reinforcement learning,” in IEEE Wireless Commun. Netw. Conf. (WCNC 2022), Austin, TX, USA, Apr. 2022.DOI
- [4]F. Rezazadeh, L. Zanzi, F. Devoti, S. Barrachina-Muñoz, E. Zeydan, X. Costa-Pérez, and J. Manges-Bafalluy, “A multi-agent deep reinforcement learning approach for RAN resource allocation in O-RAN,” in IEEE Conf. Comput. Commun. Wkshps, New York, USA, May. 2023.arxiv.org/abs/2307.02414
- [5]N. Frangoniotis, E. Stai, I. Dimolitsas, A. Zafeiropoulos, and S. Papavassiliou, “Dynamic, reconfigurable and green network slice admission control and resource allocation in the O-RAN using model predictive control,” in IFIP Netw. Conf., Thessaloniki, Greece, Jun. 2024.
- [6]P. S. Upadhyaya, N. Tripathi, J. Gaeddert, and J. H. Reed, “Open AI cellular (OAIC): An open source 5G O-RAN testbed for design and testing of AI-based RAN management algorithms,” IEEE Network, vol. 37, no. 5, pp. 7–15, 2023.DOI
- [7]E. Municio, G. Garcia-Aviles, A. Garcia-Saavedra, and X. Costa-Pérez, “O-ran: Analysis of latency-critical interfaces and overview of time sensitive networking solutions,” IEEE Communications Standards Magazine, vol. 7, no. 3, pp. 82–89, 2023.
- [8]M. Dai, G. Sun, H. Yu, and D. Niyato, “Maximize the long-term average revenue of network slice provider via admission control among heterogeneous slices,” IEEE/ACM Trans. Netw., vol. 32, no. 1, pp. 745–760, 2024.arxiv.org/abs/2304.10057
- [9]Y. Zhou, Q. Ye, H. Huang, and H. Du, “Deep reinforcement learning based admission control for throughput maximization in mobile edge computing,” in IEEE 94th Veh. Technol. Conf. (VTC-Fall 2021), Norman, OK, USA, Sept. 2021.DOI
- [10]P. Joulani, A. Gyorgy, and C. Szepesvari, “Online learning under delayed feedback,” in Proceedings of the 30th International Conference on Machine Learning (ICML 2013), S. Dasgupta and D. McAllester, Eds., vol. 28, Atlanta, Georgia, USA, 17–19 Jun 2013, pp. 1453–1461.arxiv.org/abs/1306.0686
- [11]K. Quanrud and D. Khashabi, “Online learning with adversarial delays,” in Advances in Neural Information Processing Systems (NeurIPS 2015), C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28, 2015.
- [12]B. Han, Z. Ren, Z. Wu, Y. Zhou, and J. Peng, “Off-policy reinforcement learning with delayed rewards,” in Proceedings of the 39th International Conference on Machine Learning (ICML 2022), Baltimore, MD, USA, Jul. 2022, pp. 8280–8303.arxiv.org/abs/2106.11854
- [13]W. U. Mondal and V. Aggarwal, “Reinforcement learning with delayed, composite, and partially anonymous reward,” Transactions on Machine Learning Research, 2023.arxiv.org/abs/2305.02527
- [14]J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter, “RUDDER: Return decomposition for delayed rewards,” in 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, Dec. 2019.arxiv.org/abs/1806.07857