betaFeedback

WorksEngineering

Intelligent Carbon Emission-aware Radio Unit Activation Control in Dense Heterogeneous Energy 6G Networks

Abstract

Base stations account for a dominant share of energy consumption and operational carbon emissions in radio access networks. In O-RAN deployments with heterogeneous energy sources, however, minimizing energy consumption alone does not necessarily minimize carbon emissions. In this work, we investigate carbon-aware DRL for RU sleep control. Under the adopted source-matched operational accounting model, renewable-matched RUs are assigned zero direct operational emissions, whereas non-renewable-supplied RUs incur operational emissions. The problem is formulated as a Markov decision process and evaluated using two carbon-aware reward formulations and a conventional energy–QoS reward baseline. We implement and compare TD3, PPO, and a heuristic method under realistic mobility and 3GPP channel models across multiple renewable-matched RU layouts and traffic loads. Simulation results show that TD3 learns more energy-efficient activation policies than PPO, achieving approximately 3%–6% lower energy consumption in representative scenarios. TD3 also achieves 8%–12% lower operational emissions than PPO in scenarios with a low proportion of renewable-matched RUs and up to a 20% reduction relative to the heuristic baseline under moderate renewable availability and high traffic load. Moreover, the joint energy,carbon and QoS reward achieves approximately 12%–28% lower operational emissions than the energy and QoS reward, while maintaining comparable energy consumption.

Introduction

The explosive growth of mobile data traffic has driven the large-scale densification of cellular infrastructures. While dense deployments significantly enhance network capacity and coverage, they simultaneously introduce substantial operational energy demands. Recent reports indicate that Base Stations (BSs) account for approximately 73%–77% of the total energy consumption in mobile networks [1], making them the dominant contributors to both operational expenditure and greenhouse gas emissions. A typical 5G macro BS consumes between 3.3 and 9 kW, with even higher values observed in massive MIMO and mmWave deployments [2]. As networks evolve toward ultra-dense 6G architectures, energy efficiency alone is no longer sufficient; reducing carbon emissions associated with network operation has become a central sustainability objective. Traditional optimization efforts have primarily focused on minimizing instantaneous power consumption. However, energy efficiency does not necessarily translate into carbon efficiency [3]. With the increasing adoption of renewable energy sources at cellular sites, modern radio access networks are progressively transitioning toward hybrid energy supply models, where some infrastructure elements are powered by renewable sources (e.g., solar or wind), while others remain dependent on carbon-intensive grid electricity. In such heterogeneous energy environments, minimizing total energy usage may not lead to minimal carbon emissions. Instead, sustainable network operation requires explicit consideration of the carbon intensity of different energy sources. This shifts the optimization paradigm from purely energy-aware control toward carbon-aware and sustainability-driven orchestration.

O-RAN provides a promising architectural foundation for enabling such sustainability-oriented control. By disaggregating the traditional monolithic BS into Central Unit (CU), Distributed Unit (DU), and RU components, O-RAN introduces programmability and openness through standardized interfaces [4]. Among these components, RUs are responsible for lower-PHY processing and RF transmission, and thus constitute a significant portion of radio access energy consumption [5]. Since RUs are typically dimensioned for peak traffic, placing a subset into sleep mode during lower demand periods provides operational flexibility. In hybrid renewable–grid deployments, such flexibility enables differentiated activation strategies, where renewable-powered RUs can be prioritized over grid-powered ones, facilitating structural carbon reduction beyond conventional energy-saving mechanisms.

The introduction of the RAN Intelligent Controller (RIC) further enhances O-RAN’s capability for sustainability-driven control. The Non Real Time RIC (Non-RT RIC) provides long-term policy guidance and optimization objectives via the A1 interface [6], while the Near Real Time RIC (Near-RT RIC) executes fine-grained radio control decisions through xApps interacting with O-DU and O-RU nodes via the E2 interface [7]. This hierarchical control paradigm naturally supports the integration of sustainability objectives into operational decision-making: long-term carbon-reduction policies can be translated into near-real-time resource management actions. As a result, O-RAN evolves from a flexible architectural framework into a programmable platform capable of supporting intelligent, carbon-aware radio access network operation.

BS sleeping [8] has been extensively studied as a means to reduce energy consumption in cellular networks, especially during off-peak hours. Wu et al. [9] proposed two traffic-aware sleep strategies, threshold-based and vacation-based, that jointly optimize BS activation and transmit power, revealing a fundamental tradeoff between energy savings and user delay. In heterogeneous networks, energy savings can be achieved by selectively deactivating underutilized BSs. Zhou et al. [10] formulated a joint user association and BS operation framework to optimize long-term network utility under energy constraints. Their results show that activating only a subset of Small Base Stations (SBSs) based on traffic demand can significantly reduce energy use without degrading service quality. To incorporate renewable energy sources, Gong et al. [11] considered a two-stage optimization scheme for BS sleeping and subcarrier allocation, aiming to minimize non-renewable power consumption. Their approach dynamically adjusts BS states based on both harvested energy availability and user traffic, striking a balance between energy cost and blocking probability. Recent architectural advances enable finer-grained sleep control. Sun et al. [12] investigated BS sleeping in uplink–downlink fully decoupled RANs, jointly optimizing user association, power allocation, and BS activity. Similarly, Wu et al. [13] modeled BS sleep behavior using a Semi-Markov Decision Process (SMDP), enabling dynamic sleep scheduling in macro-femto networks based on user arrival patterns and energy efficiency. In addition to optimization-based methods, traffic prediction has been used to guide sleep decisions. Lin et al. [14] proposed a data-driven BS sleeping strategy based on spatio-temporal traffic forecasting using Graph Neural Network (GNN) and LSTMs. Spatio–temporal correlations are further leveraged in [15], where convolutional-Long Short-term Memory (LSTM) forecasting is coupled with an actor–critic controller to improve robustness under fluctuating traffic.

Recent studies have increasingly adopted DRL-based paradigms to tackle the BS sleep/active control problem, motivated by their ability to handle high-dimensional state spaces, long-term dependencies, and non-linear decision boundaries. Several representative learning-based solutions have been proposed in this context. For instance, [16], the authors propose a Deep Q-Learning Network (DQN)-based framework augmented with an auxiliary action-selection network that filters infeasible actions, thereby improving exploration efficiency and stabilizing training. To explicitly account for switching overheads and service degradation, [17] introduces a modular DRL architecture that combines a Deep Deterministic Policy Gradient (DDPG) policy with a supervised cost estimator and a traffic prediction module. This hybrid design allows the controller to internalize the long-term impact of sleep transitions. In [18], a PPO-based solution is developed to simultaneously coordinate sleep scheduling, cell zooming, user association, and Reconfigurable Intelligent Surface (RIS) configuration, addressing the coupling between discrete and continuous control variables in dense multi-cell environments. Risk-awareness and robustness have also received growing attention. In [19], a digital twin is integrated with a DQN controller to evaluate delay risk prior to action execution, enabling online retraining or feature suppression under abnormal traffic conditions. From a distributed control perspective, [20] models each BS as an autonomous agent and applies a multi-agent PPO with a state-similarity mechanism, jointly optimizing BS sleeping and MIMO antenna activation while mitigating oscillatory behavior.

Beyond pure energy minimization, recent green RAN research increasingly emphasizes where the energy is drawn from and how operational decisions translate into carbon footprint. In renewable energy powered cellular networks, the key objective is often reframed from minimizing total energy to minimizing on-grid energy, since renewable consumption is typically treated as carbon-free and its availability is both stochastic and spatially imbalanced across sites. For example, the paper [21] studies BSs powered by hybrid supplies and jointly optimize traffic management and energy cooperation to reduce grid energy usage while meeting throughput constraints, highlighting that renewable-aware topology/traffic control and energy exchange between sites are complementary levers rather than substitutes. Building on this viewpoint, a line of work investigates joint sleep control and renewable energy sharing in dense deployments. [22] formulate SBS sleep decisions together with renewable energy sharing as a unified Markov Decision Process (MDP), aiming to reduce grid energy consumption and improve renewable utilization under time-varying traffic and harvested energy. To address the curse of dimensionality in Ultra Dense Network (UDN)-scale deployments, they adopt a multi-discrete PPO design that factorizes the combinatorial control space into tractable sub-actions, demonstrating notable gains over schemes that treat sleeping and energy sharing separately. Complementary to terrestrial-only designs, vertical-assisted architectures have also been explored as a sustainability enabler. Paper [23] leverage a High Altitude Platform Station (HAPS) layer to offload traffic and enable more aggressive SBS sleeping, reporting sizeable reductions in grid power consumption while maintaining Quality of Service (QoS). Another strand explicitly models carbon footprint rather than energy alone and shows that operational policies interact with both operational and embodied emissions. [24] analyze network sharing as a mechanism to enable recurrent BS deactivation during low-load periods and quantify carbon-footprint mitigation beyond energy savings, including the effect of switching frequency on equipment degradation and embodied carbon through life-time extension. From an architectural perspective, O-RAN sustainability studies provide foundational power-consumption models and highlight that open and cloud-native O-RAN implementations may shift energy burdens across RU, DU and CU and transport components, motivating careful design of energy-aware control loops. For example, Baldini et al. [25] develop a parametric power model for O-RAN and use real RU deployment datasets to expose how placement and processing/transport assumptions shape end-to-end power trends, thus informing where carbon-aware operation is most impactful.

Contribution

Although the above studies are closely related to sustainable wireless network operation, they differ fundamentally from the focus of this paper. In this work, we do not consider energy sharing, battery dynamics, or joint traffic management. Instead, In this work, we consider an O-RAN system where RUs are powered by heterogeneous energy sources, including renewable energy and the conventional power grid. By explicitly distinguishing renewable-powered RUs, which incur zero operational carbon emissions, from grid-powered RUs, we reformulate the RU control problem as a carbon-aware sequential decision making task. This formulation fundamentally alters the structure of the control problem: RU activation decisions are no longer symmetric, and the same action may lead to different carbon outcomes depending on the underlying energy source. As a result, conventional energy-centric control policies may become suboptimal when carbon reduction is a primary objective. Moreover, unlike [22], which develops a customized multi-discrete PPO framework for large-scale joint control, we conduct a systematic comparison between deterministic off-policy learning and stochastic on-policy learning under the same carbon-aware MDP formulation. This enables us to reveal how different DRL paradigms respond to reward shaping and heterogeneous energy supply in sustainable O-RAN operation. Through extensive simulations, we compare representative policy-gradient and actor–critic methods and analyze their learning behaviors under the proposed sustainability-aware formulation.

The main contributions of this paper are summarized as follows:

  • We formulate a carbon-aware RU sleep mode control problem by explicitly modeling heterogeneous energy sources, where renewable-powered and grid-powered RUs lead to different carbon implications under identical activation decisions.

  • We design a sustainability-oriented reward function within the MDP framework that jointly captures network energy consumption, operational carbon emissions, and QoS satisfaction, enabling the learning agent to balance efficiency and environmental impact.

  • We implement and evaluate representative DRL approaches, including deterministic off-policy (TD3) and stochastic on-policy (PPO) methods, under a unified simulation framework, ensuring fair comparison across learning paradigms.

  • We provide a comprehensive performance analysis across multiple deployment scenarios, demonstrating that carbon-aware optimization introduces a trade-off between energy efficiency and emissions, and showing that TD3 achieves more stable convergence and consistently better energy–carbon performance compared to PPO and heuristic model.

System Model

We consider an O-RAN-based cellular network comprising a fixed set of RUs, denoted by M={1,…,M}\mathcal{M} = \{1,\ldots,M\}, and a set of User Equipments (UEs), denoted by K={1,…,K}\mathcal{K} = \{1,\ldots,K\}.

The RU positions are predetermined and remain unchanged, as shown in Figure 1. Time is divided into TT slots indexed by t∈{1,…,T}t \in\{1,\ldots,T\}.

Proposed heterogeneous energy network

Figure 1. Proposed heterogeneous energy network

To model heterogeneous electricity-procurement arrangements, the set of RUs is partitioned into two disjoint subsets: (i) renewable-matched RUs, denoted by MR\mathcal{M}_{R}, whose electricity demand is contractually matched with renewable generation, and (ii) non-renewable-supplied RUs, denoted by MG\mathcal{M}_{G}, whose electricity demand is assigned the emission factor of the residual non-renewable grid supply. These subsets satisfy

MR∪MG=M,MR∩MG=∅.(1)\mathcal{M}_{R} \cup\mathcal{M}_{G} = \mathcal{M}, \qquad\mathcal{M}_{R} \cap\mathcal{M}_{G} = \varnothing. \tag*{(1)}

Unlike deployments with on-site renewable generation, we adopt a source-matched, market-based operational accounting model. Electricity consumed by both RU categories may be physically delivered through the same power grid; therefore, the partition does not represent physically separated electricity flows. Instead, it represents different contractual generation attributes assigned to the electricity consumed by the RUs.

The renewable-energy contracts are assumed to provide sufficient matching for the electricity demand of RUs in MR\mathcal{M}_{R} over the considered control horizon. This is an exogenous scenario assumption rather than a decision made by the DRL agent. Renewable-generation forecasting, electricity procurement, storage operation, curtailment, and energy sharing are outside the scope of this work.

Under the adopted operational accounting boundary, RUs in MR\mathcal{M}_{R} are assigned zero direct operational emissions at the point of renewable-electricity generation, whereas RUs in MG\mathcal{M}_{G} are assigned a positive source-specific operational emission factor. This boundary does not include embodied or life-cycle emissions associated with energy-generation infrastructure. The resulting abstraction enables us to isolate the effect of heterogeneous electricity-procurement arrangements on carbon-aware RU activation control.

Each RU m∈Mm \in\mathcal{M} is equipped with QmQ_m Physical Resource Blocks (PRBs). Denote by nm,ktn_{m,k}^{t} the number of PRBs allocated from RU mm to UE kk at time slot tt. The total PRB usage of RU mm is

Nmt=∑k∈Knm,kt,lmt=NmtQm.(2)N_m^t = \sum_{k\in\mathcal{K}} n_{m,k}^t, \qquad l_m^t = \frac{N_m^t}{Q_m}. \tag*{(2)}

where lmtl_m^t is the normalized PRB load. The UE association and PRB allocation are handled by the simulation environment. Initially, each UE is associated with its nearest RU. When the serving RU is switched off, the UE is reassigned to the nearest active RU. During UE mobility, a handover to another active candidate RU is performed when the candidate requires fewer PRBs to meet the target rate and has sufficient residual PRB capacity. For each UE, the requested number of PRBs is calculated as the minimum number required to achieve Rkmin⁡R_k^{\min} under the current channel condition. If the total number of requested PRBs does not exceed the capacity QmQ_m of RU mm, each UE receives its requested PRBs. Otherwise, the available PRBs are allocated proportionally among the UEs associated with that RU while satisfying the RU-level PRB capacity. The achieved UE data rates are subsequently recalculated based on the allocated PRBs.

The Signal-to-Noise Ratio (SNR) of UE kk served by RU mm at slot tt is modeled as

γm,kt=PTXgm,ktQmN0WPRB,(3)\gamma_{m,k}^{t} = \frac{P_{\mathrm{TX}}g_{m,k}^{t}}{Q_mN_0W_{\mathrm{PRB}}}, \tag*{(3)}

where PTXP_{\mathrm{TX}} is the transmission power, gm,ktg_{m,k}^{t} is the gain of the channel including path loss and fading, N0N_0 is the spectral density of the noise power and WPRBW_{\mathrm{PRB}} is the bandwidth of a PRB. The achievable data rate of UE kk is then

Rkt=∑m∈Mnm,ktWPRBlog⁡2(1+γm,kt).(4)R_k^{t} = \sum_{m\in\mathcal{M}} n_{m,k}^{t}W_{\mathrm{PRB}}\log_2\left(1+\gamma_{m,k}^{t}\right). \tag*{(4)}

To guarantee service quality, each UE must satisfy a minimum data rate requirement.

Power Consumption Model

The activation status of RU mm is described by a binary variable αmt∈{0,1}\alpha_m^t \in\{0,1\}, where αmt=1\alpha_m^t=1 if RU mm is active and αmt=0\alpha_m^t=0 if RU mm is in sleep mode. The power consumption of each RU is composed of three components [16]. The first is fixed power consumption, denoted by Pfix,mtP_{\mathrm{fix},m}^{t}, which represents the energy used by signal processing, cooling systems, and power supply units. This component depends solely on the operational mode of the RU. The second component is the load-dependent power consumption, denoted by Pdata,mtP_{\mathrm{data},m}^{t}, primarily attributed to the Power Amplifier (PA). In this work, the load is quantified based on the PRB utilization, as defined in (2). The third component is the transition power Ptrans,mtP_{\mathrm{trans},m}^{t}, which is incurred only when the RU switches from sleep mode to active mode. By summing these three components, the total power consumption of RU mm at time slot tt is given by:

Pmt=Pfix,mt+Pdata,mt+Ptrans,mt,(5)P_m^t=P_{\mathrm{fix},m}^t+P_{\mathrm{data},m}^t+P_{\mathrm{trans},m}^t, \tag*{(5)}

with

Pfix,mt=αmtPmactive+(1−αmt)Pmsleep,(6)P_{\mathrm{fix},m}^t=\alpha_m^tP_m^{\mathrm{active}}+(1-\alpha_m^t)P_m^{\mathrm{sleep}}, \tag*{(6)}
Pdata,mt=αmtPTXηlmt,(7)P_{\mathrm{data},m}^t=\alpha_m^t\frac{P_{\mathrm{TX}}}{\eta}l_m^t, \tag*{(7)}
Ptrans,mt=max⁡(αmt−αmt−1,0)⋅Vmtrans.(8)P_{\mathrm{trans},m}^t=\max\left(\alpha_m^t-\alpha_m^{t-1},0\right)\cdot V_m^{\mathrm{trans}}. \tag*{(8)}

Here PmactiveP_m^{\mathrm{active}} and PmsleepP_m^{\mathrm{sleep}} are the baseline power levels in active and sleep states, respectively; η\eta is the power amplifier efficiency; and VmtransV_m^{\mathrm{trans}} captures the overhead due to switching from sleep to active states.

Carbon Emission Model

We consider direct operational emissions associated with the electricity consumed by the RUs. Let cmtc_m^t [kgCO2_2/kWh] denote the source-specific operational carbon intensity assigned to RU mm at time slot tt [26]. Under the adopted market-based accounting boundary, the carbon intensity used in the simulations is

cmt={0,m∈MR,cG,m∈MG,(9)c_m^t = \begin{cases} 0, & m\in\mathcal{M}_{R},\\ c_G, & m\in\mathcal{M}_{G}, \end{cases} \tag*{(9)}

where cG>0c_G>0 is the operational carbon intensity assigned to the non-renewable electricity supply. The zero value assigned to renewable-matched RUs represents zero direct emissions at the point of renewable-electricity generation. It does not include embodied or life-cycle emissions associated with energy-generation infrastructure.

Since PmtP_m^t is measured in watts and the slot duration Δt\Delta t is measured in seconds, the electricity consumed by RU mm during time slot tt is Emt=Δt3.6×106PmtE_m^t=\frac{\Delta t}{3.6\times10^6}P_m^t [kWh]. Accordingly, the operational carbon emissions of RU mm are

CO2,mt=Emtcmt=Δt3.6×106cmtPmt [kgCO2].(10)\mathrm{CO}_{2,m}^t=E_m^tc_m^t=\frac{\Delta t}{3.6\times10^6}c_m^tP_m^t\ [\mathrm{kgCO}_2]. \tag*{(10)}

The network-wide operational carbon emissions during time slot tt are

CO2t=∑m∈MCO2,mt=Δt3.6×106∑m∈McmtPmt.(11)\mathrm{CO}_2^t=\sum_{m\in\mathcal{M}}\mathrm{CO}_{2,m}^t=\frac{\Delta t}{3.6\times10^6}\sum_{m\in\mathcal{M}}c_m^tP_m^t. \tag*{(11)}

The cumulative operational emissions over the horizon {1,…,T}\{1,\ldots,T\} are

CO2,tot=∑t=1TCO2t=Δt3.6×106∑t=1T∑m∈McmtPmt.(12)\mathrm{CO}_{2,\mathrm{tot}}=\sum_{t=1}^{T}\mathrm{CO}_2^t=\frac{\Delta t}{3.6\times10^6}\sum_{t=1}^{T}\sum_{m\in\mathcal{M}}c_m^tP_m^t. \tag*{(12)}

The notation cmtc_m^t retains both RU and time indices so that the general formulation can accommodate RU-specific or time-varying carbon-intensity inputs. In the present simulations, however, we use the two source-specific values in (9) to isolate the effect of heterogeneous electricity procurement on RU activation control.

Problem Formulation

Given the system model, the network operation optimizes RU activity {αmt}\{\alpha_m^t\} over slots t=1,…,Tt=1,\ldots,T. The objective is to minimize a weighted sum of renewable and non-renewable energy consumption:

min⁡{αmt}∑t=1T(λ∑m∈MRPmt+(1−λ)∑m∈MGPmt),(13)\begin{aligned} \min_{\{\alpha_m^t\}}\sum_{t=1}^{T}(\lambda\sum_{m\in\mathcal{M}_{R}}P_m^t \\ &+(1-\lambda)\sum_{m\in\mathcal{M}_{G}}P_m^t), \tag*{(13)} \end{aligned}

s.t.

Rkt≥Rkmin⁡,∀k,t,(14)R_k^t\ge R_k^{\min},\quad\forall k,t, \tag*{(14)}
∑k∈Knm,kt≤Qm,∀m,t,(15)\sum_{k\in\mathcal{K}}n_{m,k}^t\le Q_m,\quad\forall m,t, \tag*{(15)}
∑m∈Mδm,kt=1,∀k,t,(16)\sum_{m\in\mathcal{M}}\delta_{m,k}^t=1,\quad\forall k,t, \tag*{(16)}
nm,kt∈Z0,αmt,δm,kt∈{0,1},n_{m,k}^t\in\mathbb{Z}_0,\quad\alpha_m^t,\delta_{m,k}^t\in\{0,1\},
∀m,k,t.(17)\forall m,k,t. \tag*{(17)}

Here MR\mathcal{M}_{R} and MG\mathcal{M}_{G} denote the sets of renewable-energy and non-renewable-energy RUs, respectively. PmtP_m^t is the instantaneous power consumption of RU mm given its operational state and traffic load. The scalar λ∈[0,1]\lambda\in[0,1] controls the relative emphasis between renewable and non-renewable energy consumption. RktR_k^t is the downlink rate of UE kk at slot tt and must satisfy the QoS requirement Rkmin⁡R_{k}^{\min}. Constraint (15) limits the PRB allocation per RU, while (16) enforces unique RU association for each UE.

Note that explicit carbon-emission equations are omitted from the formulation. Since only non-renewable consumption contributes to carbon emissions, minimizing the weighted energy consumption in (13) naturally encourages deactivation of non-renewable RUs, thereby reducing the carbon footprint.

Markov Decision Process Formulation

In this section, we formulate the RU sleep/active control problem as a MDP, which enables sequential decision-making under dynamic traffic, user mobility, and heterogeneous energy sources. At each time slot, the agent observes the network state, selects a joint RU activation action, and receives a reward that reflects the sustainability and service performance of the network. The environment then evolves according to user mobility, traffic demand variations, and wireless channel dynamics. This formulation allows the agent to learn long-term policies that balance energy efficiency, carbon emission reduction, and QoS satisfaction over time.

Formally, the MDP is defined by the tuple (S,A,P,r)(\mathcal{S}, \mathcal{A}, \mathcal{P}, r), where S\mathcal{S} denotes the state space, A\mathcal{A} the action space, P\mathcal{P} the state transition probability, and rr the reward function. The detailed definitions of the state space, action space, and reward function are given as follows.

1) State Space: The state comprises essential information required for policy learning. At each time step tt, the state includes the real data rates of all UEs, expressed as

Rdt=[Rd,1t,Rd,2t,…,Rd,Kt]T,(18)\mathbf{R}_{d}^{t} = [R_{d,1}^{t}, R_{d,2}^{t}, \ldots, R_{d,K}^{t}]^{T}, \tag*{(18)}

which reflects the system’s ability to meet user data rate requirements. The activation states of the RUs from the previous time step are given by

αt−1=[α1t−1,α2t−1,…,αMt−1]T.(19)\boldsymbol{\alpha}^{t-1} = [\alpha_{1}^{t-1}, \alpha_{2}^{t-1}, \ldots, \alpha_{M}^{t-1}]^{T}. \tag*{(19)}

This information captures the temporal coupling between successive RU activation decisions and allows the agent to account for switching behavior.

The PRB utilization levels of the RUs are represented as

Lt=[l1t,l2t,…,lMt]T,(20)\mathbf{L}^{t} = [l_{1}^{t}, l_{2}^{t}, \ldots, l_{M}^{t}]^{T}, \tag*{(20)}

which quantify the instantaneous traffic load of each RU. In addition, the spatial positions of the UEs are included as

Ut=[(x1t,y1t),(x2t,y2t),…,(xKt,yKt)]T,(21)\mathbf{U}^{t} = [(x_{1}^{t},y_{1}^{t}), (x_{2}^{t},y_{2}^{t}), \ldots, (x_{K}^{t},y_{K}^{t})]^{T}, \tag*{(21)}

where (xkt,ykt)(x_{k}^{t}, y_{k}^{t}) denote the coordinates of UE kk at time tt. The UE positions directly affect path loss, user association, and achievable data rates, and thus influence RU load distributions. Including spatial information in the state enables the agent to anticipate future handovers, traffic shifts, and coverage-critical regions, rather than reacting solely to instantaneous load conditions.

Overall, the state at time tt is defined as

st=[Rdt,αt−1,Lt,Ut]T.(22)s_{t} = [\mathbf{R}_{d}^{t}, \boldsymbol{\alpha}^{t-1}, \mathbf{L}^{t}, \mathbf{U}^{t}]^{T}. \tag*{(22)}

All state variables are normalized to comparable ranges to facilitate stable and efficient learning.

2) Action Space: The action at∈Aa_t \in\mathcal{A} represents the operational states of all RUs and is defined as

A=[α1,α2,…,αM],(23)\mathcal{A} = [\alpha_{1}, \alpha_{2}, \ldots, \alpha_{M}], \tag*{(23)}

where each element αm∈{0,1}\alpha_m \in\{0,1\} denotes the operating mode of RU mm. Specifically, αm=1\alpha_m = 1 indicates that RU mm is active, while αm=0\alpha_m = 0 corresponds to sleep mode. The action is generated by the DRL policy and determines the network-wide RU activation pattern at each time slot.

3) Reward Function: The reward function is designed to guide the agent toward sustainability-aware RU control decisions by balancing energy efficiency, carbon emission reduction, and QoS satisfaction. To evaluate different sustainability objectives, we consider two carbon-aware reward formulations.

a) Energy–Carbon–QoS reward:

rt(1)=−w1⋅norm⁡(Ptott)−w2⋅norm⁡(Ctott)−w3⋅norm⁡(Kunsatt),(24)r_{t}^{(1)} = -w_{1} \cdot\operatorname{norm}(P_{\mathrm{tot}}^{t}) - w_{2} \cdot\operatorname{norm}(C_{\mathrm{tot}}^{t}) - w_{3} \cdot\operatorname{norm}(K_{\mathrm{unsat}}^{t}), \tag*{(24)}

where PtottP_{\mathrm{tot}}^{t} denotes the total network energy consumption at time tt, CtottC_{\mathrm{tot}}^{t} represents the corresponding carbon emissions, and KunsattK_{\mathrm{unsat}}^{t} is the number of UEs whose data rates fall below the minimum QoS requirement. Each term is normalized to ensure comparable scales and to prevent dominance by any single component. This reward formulation jointly optimizes energy efficiency, carbon reduction, and service reliability.

b) Carbon–QoS-oriented reward:

rt(2)=−w4⋅norm⁡(Ctott)−w5⋅norm⁡(Kunsatt),(25)r_{t}^{(2)} = -w_{4} \cdot\operatorname{norm}(C_{\mathrm{tot}}^{t}) - w_{5} \cdot\operatorname{norm}(K_{\mathrm{unsat}}^{t}), \tag*{(25)}

which focuses explicitly on minimizing operational carbon emissions while maintaining QoS constraints. By removing the explicit energy term, this reward highlights scenarios where carbon reduction, rather than energy minimization, is the dominant objective, particularly in networks with heterogeneous renewable-powered and grid-powered RUs. The parameter λ\lambda in Eq. (13) provides a high-level representation of the relative importance of renewable and non-renewable energy consumption. This trade-off is implemented in the MDP through the reward weights. AG1 penalizes both total energy consumption and carbon emissions, AG2 penalizes only carbon emissions and therefore places the strongest preference on renewable-matched operation. Thus, the two reward formulations correspond to different implementations of the source-weighting principle represented by λ\lambda. And in both reward formulations, QoS constraints Rd,kt≥Rmin⁡,d,kR_{d,k}^{t} \geq R_{\min,d,k} are enforced through the penalty term associated with KunsattK_{\mathrm{unsat}}^{t}. The temporal accumulation of this penalty discourages persistent QoS violations and promotes stable network operation.

Figure 2. Illustrates the workflow of TD3 training process. Red solid lines indicate critic training based on temporal-difference loss; green dashed lines represent actor updates guided by critic gradients and soft update target network.

Sustainable DRL-Based RU Sleep Control

To solve the formulated MDP, we adopt DRL methods to learn optimal RU activation policies under dynamic traffic variations and heterogeneous energy conditions. In particular, we consider two representative DRL paradigms: deterministic off-policy learning and stochastic on-policy learning.

Off-policy methods, such as TD3, learn the optimal policy using transitions sampled from a replay buffer, which may contain experiences generated by previous policies. This allows for improved sample efficiency and stable learning, as the agent can reuse past experiences and decorrelate data samples. Such properties are particularly advantageous in our setting, where the RU activation problem exhibits a high-dimensional combinatorial action space and strong coupling between decisions across RUs. The ability to leverage historical interactions enables more effective exploration of the action space and refinement of activation patterns over time.

In contrast, on-policy methods, such as PPO, update the policy using only data collected from the current policy, ensuring that the learning process closely follows the latest policy behavior. This results in more stable and theoretically grounded updates, as the policy gradient is computed with respect to the current data distribution. However, this also leads to lower sample efficiency and may limit exploration in complex environments. In the context of RU sleep control, where the reward landscape can be highly non-convex due to QoS constraints and heterogeneous energy sources, on-policy methods may converge faster in early stages but risk settling into suboptimal solutions.

Overall, the comparison between off-policy and on-policy learning provides valuable insights into the trade-offs between exploration, stability, and convergence performance in sustainability-aware RU control. This design enables a systematic evaluation of how different learning paradigms respond to the structural complexity of the problem.

TD3-Based Training for RU Sleep Control

The joint sleep/active control of multiple RUs inherently leads to a high-dimensional and strongly coupled action space. When the number of controllable RUs increases, discrete value-based approaches face an exponential growth of possible action combinations, which severely degrades exploration efficiency and learning stability. To overcome this limitation, we employ a continuous-control formulation based on the TD3 algorithm, which enables learning by relaxing the discrete RU activation decisions into a continuous action space during training.

  1. Continuous Relaxation of RU Activation: In the proposed framework, the actor network parameterized by θ\theta outputs a continuous action vector

μθ(s)=[μθ,1(s),μθ,2(s),…,μθ,M(s)]∈[0,1]M,(26)\boldsymbol{\mu}_{\theta}(s) = [\mu_{\theta,1}(s),\mu_{\theta,2}(s),\ldots,\mu_{\theta,M}(s)] \in[0,1]^M, \tag*{(26)}

where each element represents the activation tendency of an RU. This continuous representation allows the policy to express fine-grained preferences over RU activation levels, thereby avoiding direct enumeration of all binary activation patterns. Note that, the continuous actor output is interpreted as a proto-action rather than the physical operating state of an RU. Binary actions are executed in the environment and used to train the critics. During the actor update, the critic is evaluated at the continuous proto-action and provides a surrogate gradient over the relaxed action space. This approximation enables deterministic policy-gradient training without explicitly enumerating all joint binary actions. However, the resulting gradient is not the exact gradient of the thresholded binary policy, and approximation errors may arise between the interior of the relaxed action space and its binary vertices.

During interaction with the environment, the continuous actions are converted into executable binary decisions through a deterministic discretization mechanism combined with exploration noise. Specifically, Gaussian noise is added to each action dimension to encourage exploration,

a~i=μθ,i(s)+ηi,ηi∼N(0,σ2),(27)\tilde{a}_i = \mu_{\theta,i}(s) + \eta_i,\qquad\eta_i \sim\mathcal{N}(0,\sigma^2), \tag*{(27)}

followed by threshold-based discretization;

ai={1,if a~i>0.5,0,otherwise,(28)a_i = \begin{cases} 1, & \text{if }\tilde{a}_i > 0.5,\\ 0, & \text{otherwise}, \end{cases} \tag*{(28)}

where aia_i denotes the operational state of RU ii. This hybrid continuous–discrete execution strategy enables efficient exploration in a smooth action space while ensuring compatibility with binary RU sleep/active control.

  1. Critic Learning and Target Estimation: The learning process relies on an experience replay buffer that stores transitions (st,at,rt,st+1)(s_t,a_t,r_t,s_{t+1}), where the reward reflects energy consumption, carbon emissions, and QoS satisfaction according to the adopted sustainability objective.

To mitigate overestimation bias in value approximation, TD3 employs two independent critic networks, Qϕ1Q_{\phi_1} and Qϕ2Q_{\phi_2}, both of which estimate the state–action value function.

For each sampled transition, the target Q-value is computed using the minimum of the two target critics,

yt=rt+γmin⁡i∈{1,2}Qϕi′(st+1,f(μθ′(st+1)+ϵ)),(29)y_t = r_t + \gamma\min_{i\in\{1,2\}} Q_{\phi_i'}\left(s_{t+1},f\left(\mu_{\theta'}(s_{t+1})+\epsilon\right)\right), \tag*{(29)}

where θ′\theta' and ϕi′\phi_i' denote the parameters of the target actor and target critics, respectively. The noise term ϵ∼clip⁡(N(0,σtarget2))\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma_{\mathrm{target}}^2)) is used for target policy smoothing, which reduces sensitivity to small perturbations in the action space and improves learning robustness.

Each critic network is updated by minimizing the mean squared temporal-difference loss

L(ϕi)=E(s,a,r,s′)∼D[(Qϕi(s,a)−yt)2].(30)\mathcal{L}(\phi_i)=\mathbb{E}_{(s,a,r,s')\sim\mathcal{D}}\left[\left(Q_{\phi_i}(s,a)-y_t\right)^2\right]. \tag*{(30)}
  1. Actor Update and Delayed Policy Optimization: The actor network is updated using the deterministic policy gradient, which adjusts the policy parameters to maximize the expected Q-value estimated by the critics,

∇θJ(θ)=Es∼D[∇aQϕ1(s,a)∣a=μθ(s)⋅∇θμθ(s)].(31)\nabla_{\theta}J(\theta)=\mathbb{E}_{s\sim\mathcal{D}}\left[\left.\nabla_{a}Q_{\phi_1}(s,a)\right|_{a=\mu_{\theta}(s)}\cdot\nabla_{\theta}\mu_{\theta}(s)\right]. \tag*{(31)}

To enhance stability, actor updates are performed less frequently than critic updates, following the delayed policy update strategy of TD3. This separation allows the critics to provide more accurate value estimates before the policy is adjusted.

  1. Target Network Updates: Both the actor and critic target networks are updated using soft updates,

θ′←τθ+(1−τ)θ′,ϕi′←τϕi+(1−τ)ϕi′,i∈{1,2},(32)\theta' \leftarrow\tau\theta+(1-\tau)\theta',\qquad\phi'_i\leftarrow\tau\phi_i+(1-\tau)\phi'_i,\quad i\in\{1,2\}, \tag*{(32)}

where τ≪1\tau\ll1 controls the update rate. This mechanism smooths parameter evolution and further stabilizes training.

PPO-Based Training for RU Sleep Control

To provide a representative stochastic policy-gradient baseline, we implement PPO for the same RU sleep/active control task and environment as the TD3 agent. In our setting, the controller must decide, at every time slot, whether each RU should be active or sleeping, resulting in a multi-dimensional binary action vector at∈{0,1}Ma_t\in\{0,1\}^{M}. Unlike TD3, which learns a deterministic continuous policy and applies thresholding for execution, PPO directly models the joint RU activation decision as a stochastic policy over binary actions. This design enables on-policy learning with clipped policy updates while preserving the discrete nature of RU switching.

  1. Policy Parameterization for Multi-RU Binary Actions: We adopt an actor-critic architecture with a shared feature extractor and two heads: an actor head producing RU-wise logits and a critic head estimating the state value. Given the observed state sts_t, the network outputs

ℓt,Vψ(st)=fψ(st),(33)\ell_t,V_{\psi}(s_t)=f_{\psi}(s_t), \tag*{(33)}

where ℓt∈RM\ell_t\in\mathbb{R}^{M} denotes the logits for all RUs, and Vψ(st)V_{\psi}(s_t) is the state-value estimate. The RU activation policy is modeled as a factorized Bernoulli distribution

πψ(at∣st)=∏m=1MBernoulli⁡(at,m;pt,m),\pi_{\psi}(a_t\mid s_t)=\prod_{m=1}^{M}\operatorname{Bernoulli}(a_{t,m};p_{t,m}),
pt,m=σ(ℓt,m/τt),(34)p_{t,m}=\sigma(\ell_{t,m}/\tau_t), \tag*{(34)}

where σ(⋅)\sigma(\cdot) is the sigmoid function and τt\tau_t is a temperature parameter. Sampling from this distribution yields the RU activation vector ata_t, which is executed directly in the environment. This formulation maps naturally to RU sleep scheduling: each dimension corresponds to turning one RU active/sleep, and the stochastic policy allows PPO to explore different activation patterns under the same state.

Algorithm 1: TD3 Training for Sustainability-Aware RU Sleep Control

Input: Initial state s0s_0; actor μθ\mu_{\theta}; critics Qϕ1,Qϕ2Q_{\phi_1},Q_{\phi_2}; replay buffer D\mathcal{D}; discount γ\gamma; noise scales σ,σtgt\sigma,\sigma_{\mathrm{tgt}}; update delay dd; soft update rate τ\tau

Output: Trained actor policy μθ\mu_{\theta} for joint RU activation control

  1. for episode =1,2,…=1,2,\ldots do

    1. Reset environment and observe s←s0s\leftarrow s_0;

    2. for t=1,2,…,Tt=1,2,\ldots,T do

      1. // (1) Continuous activation scores for all RUs

      2. Draw exploration noise η∼N(0,σ2I)\eta\sim\mathcal{N}(0,\sigma^2\mathbf{I});

      3. Compute activation scores z←μθ(s)+ηz\leftarrow\mu_{\theta}(s)+\eta, where z∈RMz\in\mathbb{R}^{M};

      4. Clip zz to [0,1]M[0,1]^{M};

      5. // (2) Discretize to executable RU on/off decisions

      6. Construct RU action vector a∈{0,1}Ma\in\{0,1\}^{M} by am←⟦zm>δ⟧, ∀m∈{1,…,M}a_m\leftarrow\llbracket z_m>\delta\rrbracket,\ \forall m\in\{1,\ldots,M\};

      7. // Optionally: δ=0.5\delta=0.5; interpret {m:am=1}\{m:a_m=1\} as the selected active RU set

      8. // (3) Environment interaction

      9. Execute aa, obtain reward rr and next state s′s' (reward uses the chosen carbon-aware objective);

      10. Store transition (s,a,r,s′)(s,a,r,s') into D\mathcal{D};

      11. Set s←s′s\leftarrow s';

      12. // (4) Critic update

      13. Sample a mini-batch {(sj,aj,rj,sj′)}j=1N\{(s_j,a_j,r_j,s'_j)\}_{j=1}^{N} from D\mathcal{D};

      14. Draw target smoothing noise ϵ∼clip⁡(N(0,σtgt2I))\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma_{\mathrm{tgt}}^2\mathbf{I}));

      15. Compute target scores zj′←μθ′(sj′)+ϵz'_j\leftarrow\mu_{\theta'}(s'_j)+\epsilon, clip to [0,1]M[0,1]^{M};

      16. Discretize target action aj′a'_j by aj,m′←⟦zj,m′>δ⟧a'_{j,m}\leftarrow\llbracket z'_{j,m}>\delta\rrbracket;

      17. Compute TD target: yj←rj+γmin⁡i∈{1,2}Qϕi′(sj′,aj′)y_j\leftarrow r_j+\gamma\min_{i\in\{1,2\}}Q_{\phi'_i}(s'_j,a'_j);

      18. Update critics by minimizing, for i∈{1,2}i\in\{1,2\}: L(ϕi)=1N∑j=1N(Qϕi(sj,aj)−yj)2\mathcal{L}(\phi_i)=\frac{1}{N}\sum_{j=1}^{N}(Q_{\phi_i}(s_j,a_j)-y_j)^2;

      19. // (5) Delayed actor and target updates

      20. if t mod d=0t\bmod d=0 then // update every dd steps

        1. Update actor using deterministic policy gradient: ∇θJ≈1N∑j=1N∇aQϕ1(sj,a)∣a=μθ(sj)∇θμθ(sj)\nabla_{\theta}J\approx\frac{1}{N}\sum_{j=1}^{N}\left.\nabla_aQ_{\phi_1}(s_j,a)\right|_{a=\mu_{\theta}(s_j)}\nabla_{\theta}\mu_{\theta}(s_j);

        2. Soft update target networks: θ′←τθ+(1−τ)θ′\theta'\leftarrow\tau\theta+(1-\tau)\theta'; ϕi′←τϕi+(1−τ)ϕi′,i∈{1,2}\phi'_i\leftarrow\tau\phi_i+(1-\tau)\phi'_i,\quad i\in\{1,2\};

Exploration Design in RU Scheduling

To avoid premature saturation of Bernoulli probabilities (i.e., pt,m→0p_{t,m}\to0 or 11 leading to near-zero entropy and poor exploration), we incorporate several exploration mechanisms during training. First, we inject Gaussian noise into the actor logits before sampling,

ℓ~t=ℓt+ξt,ξt∼N(0,σℓ,t2I),(35)\widetilde{\ell}_t=\ell_t+\xi_t,\qquad\xi_t\sim\mathcal{N}(0,\sigma_{\ell,t}^{2}\mathbf{I}), \tag*{(35)}

and apply temperature scaling through τt\tau_t to control stochasticity. Both σℓ,t\sigma_{\ell,t} and τt\tau_t are annealed over training to transition from exploration to exploitation. Second, an entropy bonus is added to the objective to encourage diverse RU activation decisions at early stages. These mechanisms are particularly important in our scenario because RU control is high-dimensional and strongly coupled via traffic load, user association, and QoS constraints; insufficient exploration may cause the policy to collapse to a fixed RU pattern that fails to adapt to changing UE distributions.

3) Rollout Collection and Advantage Estimation with Episode Boundaries: PPO is trained on on-policy rollouts collected by interacting with the same environment dynamics used by TD3, including mobility, handovers, and the RU-specific energy/carbon models. At each time step, the transition (st,at,rt,st+1)(s_t, a_t, r_t, s_{t+1}) is recorded along with the log-probability log⁡πψ(at∣st)\log\pi_\psi(a_t|s_t) and value estimate Vψ(st)V_\psi(s_t). We collect rollouts for a fixed number of steps and then perform multiple epochs of PPO updates.

To compute low-variance policy gradients, we use generalized advantage estimation (GAE). Importantly, in our implementation, advantage accumulation is reset at episode boundaries (either environment termination or time-limit truncation), preventing leakage of advantages across independent episodes:

A^t=∑l=0∞(γλ)lδt+l,δt=rt+γVψ(st+1)−Vψ(st).(36)\hat{A}_t = \sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}, \qquad \delta_t = r_t + \gamma V_\psi(s_{t+1}) - V_\psi(s_t). \tag*{(36)}

This is crucial for RU scheduling since episodes are time-limited and the environment may reset due to simulation horizon; improper bootstrapping would distort the estimated returns and destabilize training.

4) Clipped PPO Objective for RU Sleep Decisions: Given a batch of rollout data, PPO updates the policy by maximizing the clipped surrogate objective:

LPPO(ψ)=E[min⁡(ρt(ψ)A^t,clip⁡(ρt(ψ),1−ϵ,1+ϵ)A^t)](37)\mathcal{L}_{\mathrm{PPO}}(\psi)=\mathbb{E}\left[\min\left(\rho_t(\psi)\hat{A}_t,\operatorname{clip}(\rho_t(\psi),1-\epsilon,1+\epsilon)\hat{A}_t\right)\right] \tag*{(37)}

where

ρt(ψ)=exp⁡(log⁡πψ(at∣st)−log⁡πψold(at∣st)),(38)\rho_t(\psi)=\exp\left(\log\pi_\psi(a_t|s_t)-\log\pi_{\psi_{\mathrm{old}}}(a_t|s_t)\right), \tag*{(38)}

and ϵ\epsilon is the clipping parameter. The value function is trained with a squared error loss, and an entropy regularizer is used to promote exploration:

L(ψ)=−LPPO(ψ)+cvE[(R^t−Vψ(st))2]−cHE[H(πψ(⋅∣st))],(39)\mathcal{L}(\psi)=-\mathcal{L}_{\mathrm{PPO}}(\psi)+c_v\mathbb{E}\left[(\hat{R}_t-V_\psi(s_t))^2\right]-c_H\mathbb{E}\left[\mathcal{H}(\pi_\psi(\cdot|s_t))\right], \tag*{(39)}

where R^t\hat{R}_t is the estimated return, H(⋅)\mathcal{H}(\cdot) denotes entropy, and cvc_v and cHc_H control the value and entropy terms, respectively. In the RU sleep control context, this objective updates the RU activation probabilities so that actions yielding lower energy/carbon cost under QoS constraints become more likely, while the clipped update prevents abrupt policy shifts that could cause unstable RU switching.

5) Stability Measures and Implementation Details: To improve numerical stability in high-dimensional binary control, we clamp logits to bounded ranges before computing log-probabilities and apply gradient norm clipping during optimization. Moreover, reward normalization is applied per rollout to reduce scale sensitivity and to stabilize the advantage estimates. The actor and critic are optimized jointly using

separate learning rates for the shared backbone/actor head and the critic head. These design choices help mitigate training collapse (e.g., entropy going to zero) that can occur when the policy quickly becomes overconfident in a particular RU activation pattern.

Overall, PPO observes the same state representation as TD3 and outputs a stochastic RU activation vector at\boldsymbol{a}_t that is executed directly in the environment. This provides a strong and widely adopted policy-gradient baseline for evaluating the suitability of stochastic versus deterministic control paradigms in sustainability-aware RU sleep scheduling.

Simulation Environment

Simulation Settings

In our simulations, each scenario is modeled as an O-RAN environment where MM RUs serve KK UEs within a square area of size L×L m2L \times L\ \mathrm{m}^2. The considered scenarios explicitly differentiate between renewable-powered RUs and grid-powered RUs, enabling the evaluation of sustainability-aware RU sleep control under heterogeneous energy sources. Renewable-powered RUs are assumed to operate with zero operational carbon emissions, while grid-powered RUs incur both energy consumption and carbon emissions according to the adopted energy and carbon models.

In practical O-RAN deployments, the Near-RT RIC is typically hosted at edge cloud or regional edge data centers, with latency budgets ranging from 10 ms to 1 s. In our system model, the inference of the DRL agents is executed within the Near-RT RIC as an xApp, ensuring that RU sleep/active decisions satisfy the real-time control requirements. Model training is performed offline using simulated interaction data, while the trained policies are deployed for online inference. This deployment setting aligns with current O-RAN architectural principles and ensures realistic decision-making latency.

As illustrated in Figure 3, multiple spatial layouts are considered to examine the impact of renewable penetration and spatial distribution. The scenarios include different numbers of renewable-powered RUs (e.g., 2, 4, 6, and 8), as well as different spatial arrangements such as centralized, evenly distributed, and random placements. These layouts are designed to evaluate the robustness of the learned policies under varying renewable availability and spatial heterogeneity.

Spatial deployment scenarios with different renewable RU penetration levels and layouts

Figure 3. Spatial deployment scenarios with different renewable RU penetration levels and layouts. Green circles indicate renewable-powered RUs, while gray circles represent grid-powered RUs. These scenarios are designed to evaluate the robustness of RU sleep scheduling policies under varying renewable availability and spatial distributions.

UEs move according to a periodic mobility model with constant speed v=vavg±vstdv = v_{\mathrm{avg}} \pm v_{\mathrm{std}}, where the mean speed is vavg=2v_{\mathrm{avg}} = 2 m/s and the standard deviation is vstd=0.5v_{\mathrm{std}} = 0.5 m/s. Individual UE speeds are randomly drawn from this range. UEs periodically travel from the boundary of the service area toward the center and back, forming a structured cyclic mobility pattern. This mobility model induces time-varying traffic loads and handovers across RUs, which is critical for evaluating adaptive RU sleep control policies.

The wireless channel model follows the 3GPP Urban Microcell (UMi) specification [27] with both Line of Sight (LOS) and Non-Line of Sight (NLOS) conditions. For LOS links, the path loss is given by

Pm,kLOS={32.4+21log⁡10(dm,k)+20log⁡10(f),dm,k≤dBP,32.4+40log⁡10(dm,k)+20log⁡10(f)−9.5,dm,k>dBP,(40)P^{\mathrm{LOS}}_{m,k} = \begin{cases} 32.4 + 21\log_{10}(d_{m,k}) + 20\log_{10}(f), & d_{m,k} \le d_{\mathrm{BP}}, \\ 32.4 + 40\log_{10}(d_{m,k}) + 20\log_{10}(f) - 9.5, & d_{m,k} > d_{\mathrm{BP}}, \end{cases} \tag*{(40)}

where dm,kd_{m,k} denotes the distance between RU mm and UE kk, ff is the carrier frequency in GHz, and dBP=4hRUhUEfcd_{\mathrm{BP}} = \frac{4h_{\mathrm{RU}}h_{\mathrm{UE}}f}{c} is the breakpoint distance. For NLOS links, the path loss is modeled as

Pm,kNLOS=35.3+22.4log⁡10(dm,k)+21.3log⁡10(f)−0.3(hUE−1.5).(41)P^{\mathrm{NLOS}}_{m,k} = 35.3 + 22.4\log_{10}(d_{m,k}) + 21.3\log_{10}(f) - 0.3(h_{\mathrm{UE}} - 1.5). \tag*{(41)}

The LOS probability is defined as

PLOS=min⁡(18dm,k,1)(1−e−dm,k/36)+e−dm,k/36,(42)P_{\mathrm{LOS}} = \min\left(\frac{18}{d_{m,k}},1\right)\left(1-e^{-d_{m,k}/36}\right) + e^{-d_{m,k}/36}, \tag*{(42)}

with PNLOS=1−PLOSP_{\mathrm{NLOS}} = 1 - P_{\mathrm{LOS}}, and PLOS=1P_{\mathrm{LOS}} = 1 when dm,k<18d_{m,k} < 18 m.

Each training episode consists of 200 time steps. To account for RU state transition latency and to match practical network operation timescales, each time step corresponds to 1.75 s. This value is motivated by the slow-timescale BS sleep-control model in [16], where the decision interval is selected according to the coherence time of large-scale fading and BS transition latency. In the present work, 1.75 s is used as the interval between two consecutive RU activation decisions and provides time for telemetry collection, policy inference, E2 signaling, and RU actuation. RU sleep/active decisions, energy consumption measurement, and carbon emission accounting are all performed at this temporal granularity. The instantaneous energy and carbon emission values are recorded at each step, while episode-level metrics are obtained by accumulation over the entire episode.

We evaluate two representative DRL approaches for centralized RU sleep control: the deterministic policy-based TD3 and

ParameterValue
DRL Training Parameters
TD3 actor learning rate (αT\alpha_{\mathrm{T}})1×10−41 \times10^{-4}
TD3 critic learning rate (αQ\alpha_{\mathrm{Q}})1×10−31 \times10^{-3}
PPO learning rate (αPPO\alpha_{\mathrm{PPO}})3×10−43 \times10^{-4}
PPO clipping parameter (ϵ\epsilon)0.2
GAE parameter (λ\lambda)0.95
Entropy coefficient (cHc_H)0.01
Value loss coefficient (cVc_V)0.5
Mini-batch size (BB)128
Replay buffer size (TD3 only)5×1045 \times10^4
Discount factor (γ\gamma)0.99
Training episodes (NepsN_{\mathrm{eps}})2000
Soft update coefficient (TD3, τ\tau)0.01
Random seed42
Network and Channel Parameters
Carrier frequency (ff)2 GHz
RU height (hRUh_{\mathrm{RU}})15 m
UE height (hUEh_{\mathrm{UE}})1.7 m
Network size (LL)1000 m
Number of RUs (MM)12
Number of UEs (KK)40, 80
Noise power density (σn2\sigma_n^2)−174-174 dBm/Hz
Mobility and Traffic Parameters
Average UE speed (vavgv_{\mathrm{avg}})2 m/s
Std. deviation of UE speed (vstdv_{\mathrm{std}})0.5 m/s
Minimum data rate (Rmin⁡R_{\min})6 Mbps
Energy and Carbon Model Parameters
Active mode RU power (PactiveP_{\mathrm{active}})20 W
Sleep mode RU power (PsleepP_{\mathrm{sleep}})5 W
Mode transition power (PtransP_{\mathrm{trans}})3 W
Maximum transmit power (PTXP_{\mathrm{TX}})1 W (30 dBm)
Power amplifier efficiency (η\eta)0.5
Reward weights (w1,w2,w3w_1,w_2,w_3)(1, 3, 10)
Reward weights (w4,w5w_4,w_5)(1, 2.5)

Table I. Simulation Parameters

the stochastic policy-gradient-based PPO. Both agents share the same state representation, action space, and environment dynamics to ensure a fair and consistent comparison. To investigate the impact of sustainability-aware optimization, three different reward formulations are considered, summarized in Table II. AG1 jointly accounts for normalized energy consumption, carbon emissions, and QoS satisfaction, enabling a balanced trade-off between efficiency and sustainability. AG2 focuses on carbon-aware optimization by considering carbon emissions and QoS satisfaction only, while AG3 represents a conventional energy-driven design that optimizes energy consumption and QoS without explicitly incorporating carbon emissions [8]. This setup allows us to systematically analyze the interaction and trade-off between energy efficiency and carbon reduction. Detailed network architectures, optimizer configurations and system parameters are summarized in Table I.

Reward functionDescription
AG1Energy + Carbon + QoS
AG2Carbon + QoS
AG3Energy + QoS

Table II. Comparison of Reward Function Designs

To provide a non-learning baseline, a green-aware hysteresis heuristic is implemented. The heuristic follows five rules. First, any RU serving at least one UE is kept active. Second, an RU is considered idle if it has no associated UEs and its PRB utilisation is below the idle threshold UoffU_{\mathrm{off}}. Third, an idle RU is switched off after it has remained idle for ToffT_{\mathrm{off}} consecutive time steps. Fourth, if the average user satisfaction falls below Smin⁡S_{\min}, or if any active RU has PRB utilisation above the congestion threshold UonU_{\mathrm{on}}, the heuristic activates nearby sleeping RUs to relieve congestion. When multiple candidate RUs are available, renewable-powered RUs are prioritised before grid-powered RUs. Finally, at least one RU is always kept active as a safety condition.

The idle threshold is set to Uoff=0.20U_{\mathrm{off}} = 0.20, the congestion threshold is set to Uon=0.90U_{\mathrm{on}} = 0.90, and the minimum average user satisfaction is set to Smin⁡=0.95S_{\min} = 0.95. The hysteresis duration is Toff=0T_{\mathrm{off}} = 0, meaning that an idle RU can be switched off immediately once the idle condition is satisfied. To avoid activating too many RUs in one control step, the maximum number of additionally activated RUs is limited to two. The heuristic is evaluated over 50 episodes, with 200 time steps per episode.

Numerical Results

Figure 4 compares the training convergence of TD3 and PPO under the 6-renewable-RU random deployment scenario for both 40-UE and 80-UE traffic loads. Two reward formulations, AG1 and AG2, are evaluated. For the moderate traffic case with 40 UEs, both algorithms converge within approximately the first 1000 episodes. PPO shows faster improvement during the initial training phase, which can be attributed to its stochastic exploration over binary RU activation patterns. However, TD3 improves more steadily and reaches higher final rewards under both reward functions. This suggests that, after sufficient exploration, the deterministic off-policy structure of TD3 is more effective in refining RU activation decisions.

Training convergence comparison plots for 40 UEs and 80 UEs under AG1 and AG2

Figure 4. Training convergence comparison between PPO and TD3 under the 6 renewable RU (random layout) scenario. The moving-average episode rewards are shown for 40 UE and 80 UE traffic loads, each evaluated with two independent reward functions (AG1 and AG2).

Under the heavier 80-UE load, the performance gap between TD3 and PPO becomes more pronounced. The increased traffic demand tightens QoS constraints and strengthens the coupling among RU activation, user association, and PRB utilization. In this more complex setting, PPO converges quickly but tends to stabilize in a suboptimal region, whereas TD3 continues to improve over a longer training horizon and achieves higher asymptotic rewards. This demonstrates the advantage of experience replay and twin-critic learning in handling high-dimensional RU sleep control under stringent service requirements.

The reward formulation also affects the convergence behavior. AG1, which jointly considers energy, carbon emissions, and QoS, generally leads to smoother learning because the energy term regularizes excessive RU activation. In contrast, AG2 focuses more directly on carbon reduction, creating a sharper and more asymmetric reward landscape between renewable- and grid-powered RUs. This makes learning more challenging, especially for PPO. TD3 remains comparatively stable across both reward functions, showing stronger robustness to reward shaping and traffic-load variations.

Figure 5 presents the average number of active renewable- and grid-powered RUs under different renewable deployment layouts. A clear operational transition is observed as the number of renewable RUs increases. In low-renewable scenarios, such as the 2-renewable-RU case, learning-based strategies activate nearly all renewable RUs but still rely on grid-powered RUs to satisfy coverage and QoS requirements. As renewable penetration increases, both TD3 and PPO progressively reduce grid-powered activation and shift toward renewable-dominant operation.

Average number of active renewable-powered and grid-powered RUs under different deployment scenarios

Figure 5. Average number of active renewable-powered and grid-powered RUs under different control strategies for various renewable penetration levels and spatial layouts. Each subfigure corresponds to a different deployment scenario. Results are shown for both 40 UEs and 80 UEs.

Among the evaluated methods, TD3 achieves the most compact activation structure. Across all renewable layouts and traffic loads, TD3 activates on average about one fewer RU than PPO and about 2.5 fewer RUs than the heuristic baseline. The difference is particularly visible under the 80-UE case, where PPO tends to keep more grid-powered RUs active to preserve QoS stability. By contrast, TD3 learns a more selective activation policy that better exploits the spatial distribution of renewable-powered RUs. In the 8-renewable-RU scenario under 40 UEs, TD3 nearly eliminates grid-powered RU activation, indicating strong carbon-aware prioritization.

The influence of reward design is also evident from the activation patterns. AG2 encourages stronger renewable utilization than AG1 because carbon emissions are directly penalized while total energy consumption is not explicitly included. As a result, AG2 may activate additional renewable-powered RUs to reduce grid dependence. AG1, on the other hand, balances energy and carbon objectives, leading to a more compact active set. Compared with the heuristic baseline, both DRL methods are more sensitive to renewable placement and traffic distribution, while TD3 provides the most effective carbon-aware reconfiguration.

Figure 6 presents the average network energy consumption for AG1 and AG2. For 40 UEs, the heuristic baseline has the highest energy consumption, remaining above 180 W in low-renewable scenarios. DRL-based methods reduce energy consumption across all renewable configurations, with TD3 consistently outperforming PPO. In the 2-renewable-RU case, TD3 saves approximately 5–10 W compared with PPO, corresponding to about 3%–6% improvement. Similar trends are

Average network energy consumption under different control strategies for 40 and 80 UEs

Figure 6. Average network energy consumption under different control strategies as a function of the number of renewable RUs for the scenario with 40 (left) and 80 (right) UEs. Results are shown for TD3, PPO with two reward functions (AG1 and AG2) and heuristic baselines.

observed for 80 UEs, where the absolute consumption is higher due to increased traffic demand, but TD3 still achieves typical savings of 5–8 W over PPO.

Reward design also affects energy consumption. AG1 usually consumes less energy than AG2 because it explicitly penalizes total power. AG2, by focusing more strongly on carbon reduction, may activate additional renewable-powered RUs to reduce emissions, leading to slightly higher total energy use. This difference is most visible in moderate-renewable scenarios, where AG1 provides an additional 2–4 W energy reduction.

Figure 7 compares AG1 with AG3, where AG3 includes energy and QoS terms only. AG3 achieves lower energy consumption than AG1 in most scenarios for both traffic loads, as expected, since it does not consider carbon emissions. However, this lower energy use comes at the cost of higher reliance on grid-powered RUs, as shown later in the carbon-emission results. Therefore, AG3 represents a pure energy-minimization policy, while AG1 provides a more sustainable balance between energy efficiency and carbon reduction.

Average network energy consumption under different control strategies for 40 and 80 UEs

Figure 7. Average network energy consumption under different control strategies as a function of the number of renewable RUs for the scenario with 40 (left) and 80 (right) UEs. Results are shown for TD3, PPO with two reward functions (AG1 and AG3) and heuristic baselines.

Figure 8 reports the average operational carbon emissions under AG1 and AG2. Carbon emissions decrease monotonically as the number of renewable RUs increases, confirming that renewable penetration shifts the network toward low-carbon operation. For 40 UEs, TD3 achieves the lowest emissions across all renewable configurations. In the 2-renewable-RU case, TD3-AG2 reduces emissions by approximately 8%–12% compared with PPO-AG2 and by more than 15% compared with the heuristic baseline. With higher renewable penetration, TD3 maintains a 6%–10% reduction over PPO and over 18% compared with the heuristic strategy.

Figure 8. Average operational carbon emissions under different control strategies as a function of the number of renewable RUs for the scenario with 40 (left) and 80 (right) UEs. The results illustrating how carbon-aware control policies increasingly rely on renewable-powered RUs and reduce grid-dependent emissions as renewable availability grows.

For 80 UEs, emissions increase due to heavier traffic, but the relative ranking remains unchanged. TD3-based policies again achieve the lowest emissions, particularly under 4–6 renewable RUs, where TD3-AG2 obtains about 10% lower emissions than PPO-AG2 and up to 20% lower emissions than the heuristic baseline. AG2 consistently produces lower emissions than AG1 because it directly prioritizes carbon reduction, while AG1 retains energy awareness and therefore offers a more balanced policy.

Figure 9 compares AG1 and AG3 in terms of carbon emissions. AG1 consistently outperforms AG3 across all renewable deployments and traffic loads, reducing emissions by approximately 12%–28%, with an average reduction of about 18%. This improvement comes from explicitly including carbon emissions in the reward, which encourages the agent to prioritize renewable-powered RUs and reduce grid-powered activation. By contrast, AG3 minimizes energy and QoS penalties only, and therefore may select energy-efficient but carbon-intensive RU configurations.

Figure 9. Average operational carbon emissions under different control strategies (AG1 and AG3) as a function of the number of renewable RUs for the scenario with 40 (left) and 80 (right) UEs.

Together, Figs. 7 and 9 demonstrate the trade-off between pure energy minimization and carbon-aware control. AG3 provides the lowest energy consumption, whereas AG1 achieves substantial carbon reduction with only a marginal energy increase. These results confirm that explicit carbon-aware reward design is necessary for sustainable RU sleep control in heterogeneous-energy O-RAN systems.

Table III reports the QoS performance of TD3 and PPO under different renewable-RU deployments and traffic loads. In most scenarios, both algorithms achieve a user satisfaction rate above 95%, indicating that the RU sleep-control policies generally maintain the required 6 Mbps user data rate. TD3 provides more stable QoS performance under the higher traffic load, with satisfaction rates ranging from 95.35% to 99.66%. In comparison, PPO falls below 95% in several cases, particularly under AG3 with six evenly distributed renewable RUs and under AG1 with eight renewable RUs. These results demonstrate that the energy and carbon reductions achieved by TD3 are generally obtained while maintaining a high level of user satisfaction.

ScenarioReward40 UEs40 UEs80 UEs80 UEs
TD3PPOTD3PPO
2 renewable RUsAG139.70±0.5139.70 \pm0.51 (99.25%)39.93±0.1339.93 \pm0.13 (99.83%)78.60±1.2178.60 \pm1.21 (98.25%)77.88±1.3377.88 \pm1.33 (97.35%)
AG239.59±0.5939.59 \pm0.59 (98.98%)39.76±0.2539.76 \pm0.25 (99.40%)79.56±0.4279.56 \pm0.42 (99.45%)79.11±0.8679.11 \pm0.86 (98.89%)
AG339.90±0.2139.90 \pm0.21 (99.75%)39.45±0.5639.45 \pm0.56 (98.63%)79.73±0.3479.73 \pm0.34 (99.66%)79.69±0.4179.69 \pm0.41 (99.61%)
4 renewable RUs (corner)AG139.18±0.8939.18 \pm0.89 (97.95%)39.88±0.2939.88 \pm0.29 (99.70%)78.42±1.3578.42 \pm1.35 (98.03%)79.87±0.1879.87 \pm0.18 (99.84%)
AG239.80±0.5639.80 \pm0.56 (99.50%)39.90±0.2639.90 \pm0.26 (99.75%)79.20±0.7479.20 \pm0.74 (99.00%)79.78±0.2679.78 \pm0.26 (99.72%)
AG339.89±0.2039.89 \pm0.20 (99.72%)39.77±0.2739.77 \pm0.27 (99.43%)79.25±0.7179.25 \pm0.71 (99.06%)78.93±0.8978.93 \pm0.89 (98.66%)
4 renewable RUs (middle)AG138.67±0.8638.67 \pm0.86 (96.67%)38.86±0.9238.86 \pm0.92 (97.15%)78.89±1.0678.89 \pm1.06 (98.61%)78.06±1.0478.06 \pm1.04 (97.58%)
AG239.70±0.5239.70 \pm0.52 (99.25%)38.96±0.7838.96 \pm0.78 (97.40%)78.70±1.1178.70 \pm1.11 (98.38%)76.01±1.3276.01 \pm1.32 (95.01%)
AG339.77±0.3339.77 \pm0.33 (99.43%)39.45±0.5639.45 \pm0.56 (98.63%)78.97±0.7878.97 \pm0.78 (98.71%)78.93±0.8978.93 \pm0.89 (98.66%)
6 renewable RUs (random)AG139.72±0.3639.72 \pm0.36 (99.30%)39.70±0.4639.70 \pm0.46 (99.25%)79.24±0.6379.24 \pm0.63 (99.05%)79.37±0.6879.37 \pm0.68 (99.21%)
AG239.91±0.2139.91 \pm0.21 (99.77%)39.91±0.1539.91 \pm0.15 (99.77%)78.93±0.8078.93 \pm0.80 (98.66%)79.09±0.7679.09 \pm0.76 (98.86%)
AG339.80±0.2639.80 \pm0.26 (99.50%)39.50±0.5139.50 \pm0.51 (98.75%)78.95±0.8578.95 \pm0.85 (98.69%)78.86±0.9778.86 \pm0.97 (98.58%)
6 renewable RUs (even)AG139.75±0.4239.75 \pm0.42 (99.38%)39.11±1.0539.11 \pm1.05 (97.78%)76.28±3.0776.28 \pm3.07 (95.35%)77.30±2.4877.30 \pm2.48 (96.63%)
AG239.49±0.4939.49 \pm0.49 (98.73%)39.14±0.5439.14 \pm0.54 (97.85%)79.19±0.7679.19 \pm0.76 (98.99%)79.60±0.4379.60 \pm0.43 (99.50%)
AG339.36±0.5039.36 \pm0.50 (98.40%)37.57±1.4237.57 \pm1.42 (93.92%)79.39±0.4879.39 \pm0.48 (99.24%)74.13±2.8174.13 \pm2.81 (92.66%)
8 renewable RUsAG139.44±0.5839.44 \pm0.58 (98.60%)39.39±0.7239.39 \pm0.72 (98.47%)78.11±1.2178.11 \pm1.21 (97.64%)71.69±2.5771.69 \pm2.57 (89.61%)
AG239.58±0.4039.58 \pm0.40 (98.95%)39.52±0.3739.52 \pm0.37 (98.80%)78.13±0.8978.13 \pm0.89 (97.66%)76.73±1.9876.73 \pm1.98 (95.91%)
AG339.71±0.4539.71 \pm0.45 (99.28%)38.73±0.8638.73 \pm0.86 (96.82%)79.53±0.4179.53 \pm0.41 (99.41%)79.60±0.3279.60 \pm0.32 (99.50%)

Table III. QoS SATISFACTION ACHIEVED BY TD3 AND PPO UNDER DIFFERENT RENEWABLE-RU DEPLOYMENT SCENARIOS AND TRAFFIC LOADS.

Conclusion

This paper investigated carbon-aware RU sleep control in O-RAN under heterogeneous energy supply, where renewable-powered RUs are carbon-neutral and grid-powered RUs incur operational emissions. By formulating the problem as a sequential decision-making task and embedding sustainability objectives into reward design, we systematically evaluated TD3, PPO, and heuristic control across multiple renewable penetration levels, spatial layouts, and traffic conditions. The results reveal a clear operational transition from grid-dominant to renewable-dominant activation as renewable availability increases. In structural terms, TD3 activates on average about one fewer RU than PPO and about 2.5 fewer RUs than the heuristic baseline, indicating a more compact and carbon-aware activation pattern. In energy terms, TD3 consistently outperforms PPO, reducing average network energy consumption by 5–10 W in the 40UE case and 5–8 W in the 80-UE case, while AG1 further achieves 2–4 W lower energy consumption than AG2 in most moderate-renewable scenarios. In carbon terms, TD3-AG2 achieves 8%–12% lower emissions than PPO-AG2 in low-renewable scenarios, maintains 6%–10% lower emissions under higher renewable penetration, and reaches up to 20% lower emissions than the heuristic baseline under 80 UEs. In addition, AG1 reduces carbon emissions by approximately 12%–28% relative to AG3, with an average reduction of around 18%, confirming that explicit carbon awareness substantially reshapes RU activation behavior. These findings demonstrate that carbon-aware reward design is essential for sustainable O-RAN operation, and that deterministic off-policy learning provides a more effective solution for large-scale RU control under heterogeneous energy conditions.

References

  1. [1]J. S. Thompson, S. Fletcher, V. Friderikos, Y. Gao, L. Hanzo, M. R. Nakhai, T. O’Farrell, and P. D. Wells, “Editorial a decade of green radio and the path to “net zero”: A united kingdom perspective,” IEEE Transactions on Green Communications and Networking, vol. 6, no. 2, pp. 657–664, 2022.DOI
  2. [2]X. Liang, Q. Wang, A. Al-Tahmeesschi, S. B. Chetty, D. Grace, and H. Ahmadi, “Energy consumption of machine learning enhanced open RAN: A comprehensive review,” IEEE Access, vol. 12, pp. 81 889–81 910, 2024.DOI
  3. [3]H. Ahmadi, M. Rahmani, S. B. Chetty, E. E. Tsoropoulou, H. Arslan, M. Debbah, and T. Quek, “Towards sustainability in 6g and beyond: Challenges and opportunities of open ran,” IEEE Communications Standards Magazine, 2025.arxiv.org/abs/2503.08353
  4. [4]O-RAN Allience WG1, “O-RAN Architecture Description,” June 2025, accessed: June 2025. [Online]. Available: https://https://specifications.o-ran.org/specifications
  5. [5]A. I. Abubakar, O. Onireti, Y. Sambo, L. Zhang, G. Ragesh, and M. A. Imran, “Energy efficiency of open radio access network: A survey,” in 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring). IEEE, 2023, pp. 1–7.DOI
  6. [6]O-RAN Allience WG2, “A1 Interface: general aspects and principles 3.0,” June 2025, accessed: June 2025. [Online]. Available: https://https://specifications.o-ran.org/specifications
  7. [7]O-RAN Allience WG3, “O-RAN E2 General Aspects and Principles (E2GAP) 8.0,” October 2025, accessed: October 2025. [Online]. Available: https://https://specifications.o-ran.org/specifications
  8. [8]X. Liang, A. Al-Tahmeesschi, S. Chetty, C. Avdar, B. Canberk, and H. Ahmadi, “Scalable machine learning-based approaches for energy saving in densely deployed open ran,” IEEE Transactions on Green Communications and Networking, vol. 10, pp. 2710–2722, 2026.
  9. [9]J. Wu, S. Zhou, and Z. Niu, “Traffic-aware base station sleeping control and power matching for energy-delay tradeoffs in green cellular networks,” IEEE Transactions on Wireless Communications, vol. 12, no. 8, pp. 4196–4209, 2013.DOI
  10. [10]T. Zhou, Y. Fu, D. Qin, X. Li, and C. Li, “Joint user association and bs operation for green communications in ultra-dense heterogeneous networks,” IEEE Transactions on Vehicular Technology, vol. 73, no. 2, pp. 2305–2319, 2023.DOI
  11. [11]J. Gong, J. S. Thompson, S. Zhou, and Z. Niu, “Base station sleeping and resource allocation in renewable energy powered cellular networks,” IEEE Transactions on Communications, vol. 62, no. 11, pp. 3801–3813, 2014.DOI
  12. [12]Y. Sun, H. Zhou, K. Yu, Y. Xu, B. Qian, and L. X. Cai, “Flexible base station sleeping and resource allocation for green uplink fully-decoupled ran,” IEEE Transactions on Wireless Communications, 2025.DOI
  13. [13]J. Wu, Y. Li, H. Zhuang, Z. Pan, G. Wang, and Y. Xian, “Smdp-based sleep policy for base stations in heterogeneous cellular networks,” Digital Communications and Networks, vol. 7, no. 1, pp. 120–130, 2021.DOI
  14. [14]J. Lin, Y. Chen, H. Zheng, M. Ding, P. Cheng, and L. Hanzo, “A data-driven base station sleeping strategy based on traffic prediction,” IEEE Transactions on Network Science and Engineering, vol. 11, no. 6, pp. 5627–5643, 2021.DOI
  15. [15]Q. Wu, X. Chen, Z. Zhou, L. Chen, and J. Zhang, “Deep reinforcement learning with spatio-temporal traffic forecasting for data-driven base station sleep control,” IEEE/ACM Transactions on Networking, vol. 29, no. 2, pp. 935–948, 2021.arxiv.org/abs/2101.08391
  16. [16]H. Ju, S. Kim, Y. Kim, and B. Shim, “Energy-efficient ultra-dense network with deep reinforcement learning,” IEEE Transactions on Wireless Communications, vol. 21, no. 8, pp. 6539–6552, 2022.
  17. [17]J. Ye and Y.-J. A. Zhang, “Drag: Deep reinforcement learning based base station activation in heterogeneous networks,” IEEE Transactions on Mobile Computing, vol. 19, no. 9, pp. 2076–2087, 2019.arxiv.org/abs/1809.02159
  18. [18]S. Sun, C. Huang, G. Chen, P. Xiao, and R. Tafazolli, “Deep learning-based traffic-aware base station sleep mode and cell zooming strategy in ris-aided multi-cell networks,” IEEE Transactions on Cognitive Communications and Networking, 2024.arxiv.org/abs/2412.18983
  19. [19]M. Masoudi, E. Soroush, J. Zander, and C. Cavdar, “Digital twin assisted risk-aware sleep mode management using deep q-networks,” IEEE Transactions on Vehicular Technology, vol. 72, no. 1, pp. 1224–1239, 2022.arxiv.org/abs/2208.14380
  20. [20]Y. Zhen, L. Tao, D. Wu, T. Tang, and R. Wang, “Energy-saving control strategy for ultra-dense network base stations based on multi-agent reinforcement learning,” Digital Communications and Networks, 2024.DOI
  21. [21]H.-S. Lee and J.-W. Lee, “Adaptive traffic management and energy cooperation in renewable-energy-powered cellular networks,” IEEE Systems Journal, vol. 14, no. 1, pp. 132–143, 2019.DOI
  22. [22]J. Gan, H. Kou, G. Yang, H. Zhang, Z. Cao, and W. Xu, “Joint sleep control and energy sharing strategy with deep reinforcement learning in green ultra-dense networks,” IEEE Transactions on Green Communications and Networking, 2025.
  23. [23]M. Salamotmaghdisi, A. Mehrabian, and H. Yanikomeroglus, “Energy sustainability in dense radio access networks via high altitude platform stations,” IEEE Networking Letters, vol. 6, no. 1, pp. 21–25, 2023.
  24. [24]D. Renga, M. Meo, and L. Nuaymi, “Network sharing: a pathway to sustainability and carbon footprint mitigation in radio access networks,” in 2025 36th International Teletraffic Congress (ITC-36). IEEE, 2025, pp. 1–9.DOI
  25. [25]G. Baldini, R. Bolla, R. Bruschi, A. Carrega, F. Davoli, C. Lombardo, and R. Rabbani, “Toward sustainable o-ran deployment: An in-depth analysis of power consumption,” IEEE Transactions on Green Communications and Networking, 2024.DOI
  26. [26]Y. Ding, H. Duan, M. Xie, R. Mao, J. Wang, and W. Zhang, “Carbon emissions and mitigation potentials of 5g base station in china,” Resources, Conservation and Recycling, vol. 182, p. 106339, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S092134492200177XDOI
  27. [27]3GPP, “Study on channel model for frequencies from 0.5 to 100 ghz,” ETSI, Technical Report TR 138 901, 2017. [Online]. Available: https://www.etsi.org/deliver/etsi_tr/138900_138999/138901.pdf

Paper details

vlnc:W_dd56yk87
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents