WorksComputer Science
Feedback-Aware Residual Multi-Agent Reinforcement Learning for Timely Communication in Cognitive Radio Networks
Abstract
Supporting time-sensitive services in cognitive radio networks requires distributed access strategies that translate shared spectrum opportunities into timely packet delivery. Achieving this objective is challenging because reception conditions are only partially observable to individual transmitters, while dynamic competition and access overhead constrain the service that can be completed within packet deadlines. We formulate a distributed access problem that captures the coupling among interference-dependent reception, queue evolution, and protocol durations. To address this problem, we propose feedback-aware residual multi-agent proximal policy optimization (FAR-MAPPO) under centralized training and decentralized execution. The method constructs channel-specific historical feedback representations that complement instantaneous sensing with evidence from completed access attempts. A residual policy network integrates this experience with current observations and packet urgency to guide local access decisions. Numerical results show improved timely delivery over the baseline methods, particularly under heavy traffic and bursty arrivals with mixed packet deadlines.
Introduction
Timely communication is essential to time-sensitive wireless services [1]. Information with a limited useful lifetime must reach its destination before that lifetime expires. When multiple communication links share limited radio resources, achieving this objective requires terminals to adapt their access behavior to changing competition and reception conditions. Cognitive radio networks support such adaptation through environmental sensing, interaction feedback, and behavioral learning [2], [3]. Under distributed operation, however, translating locally available information into effective timely delivery remains challenging.
This challenge arises from the interaction between incomplete information and limited service time. Transmitter-side sensing cannot fully describe the interference experienced by the intended receiver, and an apparent access opportunity may disappear as competing activity changes. Meanwhile, obtaining information, acquiring access, and completing a transmission all consume the remaining lifetime of a packet. A channel that appears favorable at the decision instant may therefore fail to support timely delivery. This distinction becomes particularly consequential when traffic arrives in bursts and multiple links compete for few channels. Access decisions must account for both the reliability of a communication opportunity and the time required to exploit it.
Wireless access under dynamic competition has been studied extensively. Distributed learning and online allocation improve channel utilization under uncertain network conditions [4, 5, 6, 7], while adaptive contention control addresses competition and coexistence [8, 9, 10]. Recent studies further support cooperation among heterogeneous users [11], [12] and adaptation across network configurations [13], [14]. Historical sensing and reception feedback also help terminals infer hidden activity and exploit temporal dependencies [15, 16, 17]. These advances establish a foundation for access decisions based on local communication experience. For time-sensitive services, existing work considers deadline-constrained access with switching overhead [18] and distributed learning for timely industrial communication [19]. Traffic-aware resource allocation and access control further reduce packet delay or support differentiated service requirements [20, 21, 22]. These studies highlight the importance of incorporating service needs into access design.
For autonomous transmitter–receiver links, a key unresolved issue is how to jointly account for receiver-dependent reception conditions and deadline consumption throughout access. Channel utilization and average delay alone do not determine whether an individual packet is delivered before expiry. Moreover, the access behavior of each link changes the interference experienced by other receivers, coupling their delivery opportunities. The scientific question is therefore how each link can use incomplete local evidence and communication experience to coordinate access when reception reliability and service completion time jointly determine timely delivery.
Multi-agent reinforcement learning (MARL) offers a promising approach to this problem. Each link can act as an agent that selects actions from local observations and learns how its behavior affects subsequent communication opportunities and service outcomes. Centralized training with decentralized execution supports cooperative learning from joint information while retaining local decision-making after deployment [23]. Multi-agent proximal policy optimization (MAPPO) provides a suitable actor–critic framework [24]. Directly applying a general policy network, however, still requires extracting channel-specific experience from delayed, sparse, and unevenly sampled feedback and relating that experience to current traffic conditions. A recurrent network can exploit observation sequences, and explicitly organizing feedback for the access task provides a more direct representation of the evidence needed for timely decisions.
Two related design challenges follow. First, communication experience must remain informative under changing competition. Long-term outcomes describe relatively persistent access behavior, whereas recent outcomes help expose changes in competing activity. The amount and freshness of the available evidence also affect its relevance to the next decision. Second, that experience must be used according to current service needs. A channel with favorable historical outcomes may be unsuitable for a packet with little remaining time because switching and access incur additional delays. Conversely, a channel with poor recent outcomes may become usable as competition changes. The policy must therefore combine historical feedback, current sensing, and packet urgency to make state-dependent access decisions.
To address these challenges, we propose feedback-aware residual multi-agent proximal policy optimization (FAR-MAPPO). The method constructs channel-specific feedback representations that describe experienced access conditions through outcomes at multiple time scales, evidence freshness, and operation duration. These representations are combined with the current local state and supplied to a residual policy network [25]. A direct feature path retains encoded information, while learnable residual transformations represent interactions between communication experience and the current service state. A centralized critic supports cooperative training, and each deployed actor makes decisions using only its own observations and feedback memory. Together, these components provide a task-specific policy representation for distributed timely access.
The main contributions are summarized as follows.
We formulate distributed timely access in a cognitive radio network with multiple transmitter–receiver links and external competition. The model jointly captures the mismatch between local sensing and receiver conditions, dynamic interference, and deadline consumption during access operations. The objective measures the fraction of offered packets whose delivery is confirmed before expiry.
We develop FAR-MAPPO by integrating a channel-specific feedback representation with a residual policy network in the MAPPO framework. Multi-time-scale outcomes, evidence freshness, and access duration organize local communication experience, while the residual actor combines this experience with current service conditions for decentralized access decisions.
We establish a packet-level evaluation that examines controlled-link population, external competition, channel availability, and offered traffic separately. Traffic-burstiness and deadline-configuration comparisons, component ablations, and delivery analysis across deadline classes assess the operating conditions and component effects relevant to the proposed design.
The remainder of this paper is organized as follows. Section II presents the system model and problem formulation. Section III develops FAR-MAPPO. Section IV presents the performance evaluation, and Section V concludes the paper.
Notation: Bold symbols denote vectors, and denotes the -dimensional real vector space. For a set , denotes its cardinality. The operators and denote expectation and probability, respectively. The indicator equals one when its argument is true and zero otherwise. The notation denotes vector concatenation, and denotes the sequence from time index 0 through . The operator restricts to the interval .
System Model and Problem Formulation
We consider distributed timely communication among cognitive transmitter–receiver links sharing orthogonal channels with external competing links. As illustrated in Figure 1, each controlled transmitter selects access actions from local service, sensing, and feedback information. The actions progress through protocol stages, alter transmission activity, and generate observations for subsequent decisions. The model below connects this local access loop to receiver conditions and packet deadlines.
Figure 1. System model: (a) shared wireless environment with partial local observability; (b) local information and access decisions; (c) protocol execution under a packet deadline.
Network and Wireless Environment
Consider controlled links and external competing links sharing orthogonal channels. Their index sets are , , and , respectively. Each link consists of a transmitter and its intended receiver. A controlled link uses one channel at a time, and its receiver follows the channel selected by its transmitter. Time is divided into physical slots of duration , indexed by . Unless otherwise stated, protocol durations and packet deadlines below are expressed in slots, and multiplication by gives physical time. Link locations remain fixed within a realization, while fading and transmission activity evolve over time.
a) Propagation and reception: Using a large-scale attenuation and block-fading model [26], the channel power gain from transmitter to endpoint is
where is their distance, is the reference distance, is the reference power gain, and is the path-loss exponent. The shadowing term is Gaussian in decibels and fixed within a realization. The fading power is exponentially distributed with unit mean and remains constant over a coherence block. Shadowing and fading are sampled for the directed propagation paths and channels.
Let contain controlled and external transmitters in their data phase on channel . With denoting the gain from transmitter to receiver , the signal-to-interference-plus-noise ratio (SINR) of active controlled link is
where is the transmit power of link and is the receiver noise power. A data transmission is decoded successfully if in every slot of its data phase, where is the decoding SINR threshold in linear scale.
b) Local sensing: Sensing measures aggregate power at the transmitter. Let be the gain from transmitter to transmitter . A measurement on channel is
where is zero-mean Gaussian measurement error in decibels. Energy is observed through sensing or clear-channel assessment (CCA) on the selected channel. Measurements of other channels retain their last values and associated ages.
c) External competition: Each external link is assigned a channel at initialization and follows an ON/OFF activity process with geometric state durations. During an ON period, it uses carrier sensing and random backoff to initiate data transmissions, following the carrier-sense/backoff principle analyzed in [27]. A busy channel freezes its countdown, whereas an idle channel permits the countdown to advance. Its backoff window and transmission duration are fixed for that link within a realization. An ongoing data transmission completes even if the activity process becomes OFF. External transmissions enter both the sensed energy and receiver interference above, while controlled transmissions can influence external carrier sensing. External links supply autonomous competing traffic.
Traffic and Deadline Model
Let indicate an exogenous packet arrival at controlled link in slot , with mean rate packets per unit physical time. Under non-bursty traffic, arrivals are independent Bernoulli variates with probability . Under bursty traffic, a Markov state modulates arrivals:
where is the stationary ON probability and and are mean state durations in slots. The ON/OFF transition probabilities are and , respectively. We initialize this process in stationarity and require . Consequently, both traffic models have the same mean arrival rate, and their difference is temporal concentration of demand.
Packet arrives at slot with relative deadline , independently drawn from a prescribed deadline distribution. Its absolute deadline and remaining slack are
Each link has a queue of capacity . A new access attempt selects the queued packet with the earliest absolute deadline, with arrival order breaking ties, and retains that packet throughout the operation. New arrivals do not preempt an ongoing attempt. A packet remains counted in the queue while in service until it is delivered or expires. An unsuccessful attempt leaves an unexpired packet available for another attempt, and arrivals to a full queue are discarded.
Let be the number of timely deliveries confirmed at the end of slot , and let count the other queued packets that expire at that boundary. Confirmations are processed before expiry, so a successful confirmation exactly at the deadline is credited. Writing , the queue-size evolution is
The corresponding excess arrivals are overflow losses. Timely delivery requires successful data reception and completion of the acknowledgment (ACK) phase by the packet deadline. The ACK phase provides reliable confirmation with a fixed duration and contributes no interference.
Distributed Access and Local Observations
Local actions and protocol progression
Figure 1(b) and Figure 1(c) connect local decisions to their protocol execution. Let denote the action selected by controlled link at slot . When idle, the link selects its action from the set
giving action indices. WAIT advances one slot. SENSE switches to the selected channel when necessary and then obtains an energy measurement. ATTEMPT selects a queued packet, switches when necessary, and performs CCA. A busy CCA ends the operation without a data transmission. If CCA is idle and the selected packet is still present and unexpired, the link enters its data phase and then the feedback phase. The CCA measurement also updates the local sensing record.
Let be the current channel of link . The stage durations , , , , and denote the numbers of slots required for switching, sensing, CCA, data transmission, and ACK confirmation, respectively. The total sensing duration and the duration of an uninterrupted attempt on channel that reaches confirmation are
Protocol stages proceed to completion once initiated, while packet lifetimes decrease on the same physical clock. Busy CCA or packet expiry before data initiation ends an attempt early. Entry into the data phase requires an unexpired packet, and evaluating whether its remaining lifetime can accommodate data and confirmation is part of the access decision. While idle, WAIT and all sensing actions are legal, and attempt actions additionally require a nonempty queue. During an ongoing operation, CONTINUE advances the current protocol stage and is the sole legal action, using the same index as WAIT. All links act from pre-transition observations, after which interference is evaluated and ongoing stages advance on the shared physical clock. This accommodates overlapping operations and different decision opportunities across links.
Information available to a terminal
The local observation comprises three groups:
The service group includes link identity, current channel, protocol phase, queue length, earliest-deadline packet slack and attempt count, and the arrival indicator. The sensing group records the latest measured energy, validity, age, and busy indication for each channel. The feedback group contains the most recent initiated action, completed-operation outcome, and elapsed operation time. A packet selected by an ongoing attempt may differ from the earliest-deadline packet after a new arrival and the policy does not interrupt that attempt.
The actual receiver SINR, current external activity and backoff states, and the local observations of other links are unavailable to the deployed actor. Operation outcomes include successful timely delivery, unsuccessful delivery, busy assessment, and expiry-related termination. These outcomes summarize experienced access conditions across contention, reception, and packet expiry. Each completed operation generates one feedback event associated with its selected channel and duration. Section III describes how the policy organizes this local experience.
Problem Formulation
Let be the set of link–packet pairs generated during a fixed measurement window, including packets discarded on arrival. Let denote the collection of decentralized policies, where is the local policy of link . Define as the successful confirmation time and set if the packet is never successfully confirmed. The timely-delivery objective is
where is the expected timely-delivery ratio, and averages over the system randomness and actions generated under policy . The denominator includes all offered packets, and the numerator counts their timely confirmations. Packets are followed through delivery or expiry, and traffic and competition continue throughout this observation period.
Let be the causal local history of link . Define the joint action vector and the collection of local histories . For an action , let denote the legal action mask. The quantity is the conditional probability of the local action. We seek decentralized policies satisfying
for every link and decision time, under the physical, traffic, and protocol transitions above. The policies choose when to wait, obtain information, or attempt service and which channel to use. Packet service order, transmit power, and external access rules are fixed components of the system model.
Three coupled properties make this problem difficult. First, independent actions affect one another through receiver interference and carrier sensing, so a locally favorable choice may reduce the service opportunities of other links. Second, the observations provide incomplete and delayed evidence of reception conditions. Third, an action consumes time and changes the queue and deadline conditions under which later actions are taken. An instantaneous energy score captures only part of these consequences, while a fixed allocation cannot respond to changes in competition and demand. Direct model-based optimization would require a sufficiently informative state and a tractable transition model for the interacting links.
MARL can learn these temporally extended consequences from interaction while maintaining local execution. Centralized training supplies a critic with the joint observations of the controlled links, and each deployed actor conditions only on its own information. Effective learning still requires a representation that relates experienced access outcomes to present service urgency.
Feedback-Aware Residual Multi-Agent Reinforcement Learning
This section develops FAR-MAPPO to solve the distributed timely-access problem in (12). The method combines a channel-specific representation of local feedback with a residual actor that interprets this experience under current service conditions. MAPPO provides the cooperative training framework [24]. As illustrated in Figure 2, each terminal maintains a local feedback memory and executes a recurrent residual policy, while a centralized critic supports training through a shared timely-delivery objective.
Figure 2. Architecture of FAR-MAPPO: (a) local feedback-aware residual access policy; (b) channel-specific feedback memory; (c) centralized training.
Framework Overview
The communication system defines a cooperative decentralized partially observable Markov decision process [23]. Its latent state comprises propagation conditions, external activity, queues, and protocol stages. Each agent acts on its local observation. The shared reward counts timely confirmations in physical slot :
Using the discount factor , the training objective accumulates these rewards over time:
The discounted return supplies the learning objective, while (11) measures finite-cohort timely delivery. Confirmation rewards assign value to earlier waiting, sensing, and access decisions through their subsequent delivery outcomes.
At each physical slot, terminal receives the local observation defined in (10). The local memory in Figure 2(a) incorporates newly completed attempts and produces a feedback vector . The augmented policy input is , where is the numerical vector encoding of the local observation , and brackets denote concatenation. The actor encodes this input, applies residual feature transformations, and updates its recurrent state before selecting a legal action. Each terminal maintains its own memory and recurrent state, while the actor parameters, collectively denoted by , are shared across controlled links. Link identity in the observation allows a shared policy to distinguish terminals.
The feedback representation addresses an information problem: local sensing provides an incomplete account of receiver conditions, whereas completed attempts reveal whether access actually produced timely service. The residual actor addresses a representation problem: the relevance of that experience depends on current channel measurements, queue conditions, and deadline slack. A centralized critic estimates the future team delivery return during training. During execution, each terminal uses only its local input, memory, and actor, and no joint-observation exchange is required.
Historical Feedback Representation
Figure 2(b) illustrates how completed-attempt feedback updates the corresponding channel record to form the feedback representation.
a) Event association and causal updates: A terminal records the selected channel and start time whenever it initiates ATTEMPT. When the operation completes, its outcome and elapsed duration are assigned to that channel, and the pending record is cleared. Attempt statistics are updated exactly once for each completed ATTEMPT event.
Let when an attempt on channel completes during the transition from to , and zero otherwise. For a completed attempt, denotes timely confirmed delivery and denotes busy CCA, and each indicator is zero otherwise. Both outcome indicators are set to zero when no attempt completes. An unsuccessful outcome includes transmission failure, busy assessment, or expiry-related termination. The statistics below therefore characterize the empirical success of access attempts in achieving timely delivery, jointly reflecting contention, reception conditions, and service duration.
b) Cumulative and recent outcomes: For each link–channel pair, initialize smoothed success and failure counts as and update
The first update adds a successful completion, while the second adds an unsuccessful completion. Both counts remain unchanged when . Their ratio defines the smoothed cumulative success statistic:
The actual number of observed attempts is , where subtracting four removes the initial smoothing counts.
To expose changes on different time scales, maintain discounted success and attempt counts and , where denotes fast and slow memory. Let be decay time constants expressed in the same physical-time units as , and set . The two counts evolve as
The first count tracks recent successes, and the second tracks all recent attempts. Both start at zero and decay every physical slot, including slots without completed attempts. The corresponding success statistic is
The smoothing constants give an initial value of . As a channel remains unsampled, its discounted evidence loses weight and the statistic gradually returns toward this initial value.
To record recent contention separately, a busy count follows
Here, , and a completed attempt adds one only when CCA reports a busy channel. Normalizing by the discounted attempt count yields the busy-feedback statistic
c) Evidence freshness and service duration: The memory also stores the last completion time , the consecutive unsuccessful-attempt count , and a smoothed operation duration . At a completed successful attempt, the streak is reset to zero, and at any other completed attempt, it increases by one up to a cap .
If an operation started at and completed at , its observed duration is . The smoothed duration is updated as
with . Duration and streak remain unchanged between completion events. The initial duration is the nominal CCA–data–confirmation time on the current channel, and the initial streak is zero.
Feedback age is after at least one completion. An unobserved channel is assigned the maximum normalized age. This quantity allows the policy to distinguish recently observed access conditions from older experience.
d) Compact feature construction: To place counts, ages, and durations on a common scale, normalize a nonnegative scalar using a positive cap :
Apply this rule to , , , , and , using the respective caps , , , , and . Since is already capped, its normalized value is .
For compact notation, suppress the common indices on the right-hand side below. The channel representation is
The first four entries summarize access outcomes, the next three describe sample support and freshness, and the last two record consecutive failures and experienced time cost.
Concatenating the channel vectors produces the feedback input of the actor:
The memory resets with a new environment realization and is constructed entirely from the causal interaction record of the terminal.
These features serve complementary purposes. Cumulative outcomes summarize longer-term access experience, and disagreement with recent outcomes makes a change in observed performance visible to the actor. Counts and age distinguish repeatedly supported observations from sparse or stale evidence. Busy feedback and failure streaks describe recent access difficulties. Duration records the time actually consumed by attempts, including attempts that terminate early. Together with the current protocol state and deadline slack, it provides evidence of the time cost of accessing a channel. This representation organizes the interaction history of the terminal into explicit inputs for timely access decisions.
Residual Policy Network
The feature-level identity path in Figure 2(a) connects the input and output of each residual block. The actor first maps the augmented input to a feature vector, , where is the encoded input feature vector, is an elementwise activation function, and and are the trainable encoding weight matrix and bias vector, respectively.
Let be the number of feature-residual blocks [25]. For block index and input feature vector , a two-layer branch computes
where and are the branch weight matrices, and are their bias vectors, and collects these parameters. Writing for the input of block , the branch output is added through an identity connection:
for . The input and output widths of each block agree.
The identity path carries the encoded current observations and historical feedback, while the nonlinear branch learns transformations of their combined representation. Under dynamic competition, the same access history can have different implications for current service needs. For example, a channel with favorable historical outcomes may incur excessive switching delay for an urgent packet. The residual blocks provide a feature representation in which access experience can be interpreted jointly with current sensing, queue conditions, and remaining packet lifetime.
A gated recurrent unit (GRU) [28] then updates the local hidden-state vector from its previous value and the final residual feature vector: . Explicit feedback memory and recurrence have different roles: the former maintains predefined per-channel outcome statistics, while the latter encodes sequential context across current observations, service states, and those statistics. Recurrent state is retained across an ongoing operation and reset when the environment is reset.
A linear action head maps the updated recurrent state to one logit per action index: , where and are the trainable action-head weight matrix and bias vector. The logit vector contains the unnormalized action scores, with denoting the component for action . Applying a masked softmax [29] with the protocol mask from Section II gives
At least the WAIT/CONTINUE index is legal. When an operation is underway, CONTINUE has probability one. At a free decision point, the distribution selects among waiting, sensing, and eligible attempts on the available channels.
The policy combines complementary evidence when selecting an action. Repeated failures accompanied by favorable sensed energy indicate different experienced conditions from recent successes with stale sensing. Channel-specific feedback makes these distinctions available to the actor, and residual feature transformations and recurrence learn how they relate to timely delivery. The resulting recurrent representation produces action logits, which the masked softmax converts into a distribution over feasible actions.
Centralized Training and Decentralized Execution
Figure 2(c) shows how joint training information produces value targets and shared actor updates.
a) Value estimation and temporal credit assignment: During training, the critic receives the concatenated joint-input vector and its previous hidden-state vector . Its value function estimates the expected discounted team return, where collects the critic parameters. Its input is constructed entirely from the augmented local observations of the controlled links. The reward is the timely-confirmation count in (13), replicated across agents for cooperative training. For a rollout collected with the old policy, write for the value of the critic in the unnormalized reward scale. The bootstrap indicator equals one when a continuation value is included and zero at a true terminal transition. The one-step temporal-difference residual is . This residual compares the observed reward plus the estimated continuation value with the current value estimate. Generalized advantage estimation (GAE) [30] accumulates these residuals backward along the rollout:
where is the team advantage estimate, controls the weighting of later residuals, and equals one when the trace continues to the next recorded transition in the same realization and zero when the trace ends. Adding the estimated advantage to the old value gives the return target for critic training:
At a rollout boundary, the final observation supplies the bootstrap value and the finite sampled trace ends. At an artificial environment time limit, the pre-reset final observation supplies the bootstrap value, while the trace and recurrent states reset before the next realization. A true terminal transition uses . These rules preserve continuation-value estimates while separating independently reset episodes [31].
The same physical-slot discount is applied while a link is sensing, switching, or transmitting. Consequently, those stages influence the return even though their only legal action may be CONTINUE. For a singleton action set, the masked distribution has zero entropy and unit selected-action probability, and the direct actor-loss term at that slot has no policy dependence. Its reward and value information still contribute to temporal credit assignment for earlier decisions.
b) Actor and critic objectives: Within the MAPPO training framework, the actor uses a clipped proximal policy optimization (PPO) objective with entropy regularization [32], and the critic learns normalized return targets [24].
For each stored local sequence, let be the ratio of the new and stored old probabilities of the selected action, evaluated using the corresponding recurrent inputs and legal-action masks. Clipping this ratio to gives , where is the policy clipping threshold. Let denote standardized using the mean and standard deviation of the rollout advantages. The actor minimizes
where denotes the empirical average over sampled agent–time pairs. The minimum uses the smaller of the original and clipped surrogate terms. The categorical entropy is computed over legal actions and encourages exploration with nonnegative weight .
The critic uses a normalized, clipped Huber loss. Let denote the return target normalized using the running mean and standard deviation of return targets. The value predictions and their training targets use the same normalization statistics. Let be the current normalized value prediction and its stored old counterpart. The clipped prediction limits the change from the old prediction using threshold :
For prediction error , the Huber penalty with threshold is
This penalty is quadratic for small errors and linear for large errors. The value objective uses the larger penalty obtained from the current and clipped predictions:
The value optimizer minimizes , where is the value-loss coefficient. Actor and critic use separate optimizers, and their PPO, value-normalization, and recurrent-training settings are held fixed across matched component comparisons.
c) Complete procedure and execution: Feedback vectors are computed online and stored with observations, actions, masks, old probabilities, rewards, and recurrent states. Optimization reuses the recorded vectors so that each action remains paired with the information available when it was selected. Feedback memory evolves according to deterministic online update rules, while actor and critic parameters are optimized through their respective objectives.
Training samples legal actions, while evaluation selects the maximum-probability legal action, consistently with the learned baselines. The complete procedure is summarized in Algorithm 1.
This procedure connects the two design components to the communication task. Local memory supplies explicit evidence of experienced service conditions, and the residual recurrent actor learns how to use that evidence under current traffic and protocol constraints. Centralized training supports cooperative policy learning through a shared timely-delivery objective. After deployment, each link executes its policy using its own observations and interaction history. Section IV evaluates the combined design and its component variants.
Performance Evaluation
This section evaluates access performance under different network configurations, traffic patterns, and packet deadlines. We first compare FAR-MAPPO with five baseline methods, then examine its components and delivery performance across deadline classes.
Experimental Setup
Network and Traffic Settings
The reference configuration consists of controlled links, external competing links, and orthogonal channels. The mean packet arrival rate is packets/s per controlled link. Transmitters and receivers are deployed in a m area, with an intended link distance between 10 and 25 m.
Channel gains follow (1). The path-loss exponent is three, and the reference gain is determined by free-space propagation at 2.4 GHz over a distance of 1 m. Shadowing has a standard Algorithm 1 Feedback-Aware Residual Multi-Agent Reinforcement Learning
Require: Parallel environments, interaction budget, and PPO settings.
Ensure: Shared actor parameters for decentralized execution.
1: Initialize actor , critic , local feedback memories, and recurrent states. 2: while the interaction budget is not exhausted do 3:  for each physical slot in the rollout do 4:   Form local inputs , legal-action masks, and joint critic input . 5:   Sample legal actions using (27). 6:   Record the selected channel and start time for each new attempt. 7:   Advance all links by one physical slot and obtain the shared reward and local feedback. 8:   Decay recent statistics and incorporate each newly completed attempt exactly once. 9:   Store inputs, actions, masks, old probabilities, rewards, and recurrent states. 10:   if an environment reaches its time limit then 11:    Bootstrap from its pre-reset final observation. 12:    Reset its feedback memory and recurrent states. 13:   end if 14:  end for 15:  Compute advantages using (28) and return targets using (29). 16:  Update the actor and critic with recurrent mini-batches using (30) and (33). 17: end while
Execution: Each link uses the shared actor, maintains its own feedback memory and recurrent state, selects the highest-probability legal action, and continues feedback updates.
deviation of 4 dB, and exponential power fading is refreshed every 10 ms. The transmit power is 0 dBm, the noise floor is −107 dBm, and the sensing-error standard deviation is 2 dB. Reception and channel assessment follow the models in Section II.
Unless otherwise specified, controlled links generate bursty traffic with mean ON and OFF durations of 20 and 60 ms, respectively. The stationary ON probability is 1/4. Consequently, the packet arrival probability during an ON slot is , maintaining an overall mean arrival rate of . Each arriving packet independently receives a deadline of 20, 40, or 80 ms with equal probability.
External links have mean ON and OFF durations of 20 ms each. Their backoff windows and data-transmission durations are selected from and slots, respectively, and remain fixed within each realization. Table I summarizes the key parameters.
| Parameter | Setting |
| Controlled links | 12; sweep: 8, 10, 12, 14, 16, 18, 20 |
| External links | 12; sweep: 4, 6, 8, 10, 12, 14, 16 |
| Orthogonal channels | 4; sweep: 2, 3, 4, 5, 6, 7, 8 |
| Per-link arrival rate | 40 packets/s; sweep: 20, 30, 40, 50, 60, 70, 80 |
| Deployment area; link distance | 100 × 100 m; 10–25 m |
| Transmit power; noise floor | 0 dBm; −107 dBm |
| SINR / CCA thresholds | 6 dB / −90 dBm |
| Physical slot | 1 ms |
| Switch / sensing / CCA / data / ACK | 1 / 1 / 1 / 4 / 1 ms |
| Queue capacity; service order | 20 packets; earliest deadline first |
| Packet deadlines | Equiprobable 20, 40, and 80 ms |
| Controlled ON/OFF means | 20/60 ms |
| External ON/OFF means | 20/20 ms |
Table I. Key system parameters.
Compared Methods
The main comparisons include the following five baselines.
Random: Each link selects an attempt channel uniformly at random and applies a random cooldown after unsuccessful or busy feedback.
Fixed: Link is assigned the preferred channel and applies a random cooldown after unsuccessful or busy feedback.
Greedy: An idle link with a nonempty queue selects the channel with the lowest local score and attempts transmission. The score combines the latest measured channel energy with a 3-dB switching penalty. An unmeasured channel uses a default energy of −100 dBm. Score clipping and the first-minimum tie-breaking rule are fixed across tests.
Independent proximal policy optimization (IPPO): IPPO [33] uses a recurrent local actor and a critic based on local observations. Actor parameters are shared across links, with link identity included in the input.
MAPPO: MAPPO [24] uses recurrent local actors and a centralized critic that receives the joint observations of the controlled links.
Training and Evaluation Protocol
The feedback-memory decay scales are ms and ms. The normalization caps are , , ms, , and ms.
For each network configuration, the learning methods use 524,288 network-level physical time steps accumulated across eight parallel environments. The rollout length is 256, the number of PPO epochs per update is 15, and the recurrent hidden dimension is 64. The discount factor and GAE parameter are 0.99 and 0.95, respectively. Five training seeds are used: 32001, 32043, 32089, 32137, and 32183. Policies are trained separately for each configuration and evaluated using their final-budget checkpoints. The interaction budget is matched across methods within each configuration. Repeated training and held-out evaluation account for variation across learned policies and network realizations, an important consideration in reinforcement-learning comparisons [34].
Evaluation uses twenty held-out environment seeds, 730001–730020. All methods are evaluated using the same environment-seed set at a given configuration. Learned policies select the highest-probability legal action, while randomized baselines retain their prescribed randomization.
Figure 3. Timely delivery versus the number of controlled links, with , , and packets/s/link.
Figure 4. Timely delivery versus the number of external competing links, with , , and packets/s/link.
Each evaluation realization includes a 2,000 ms warmup, a 10,000 ms measurement arrival window, and an 80 ms settling period. Arrivals and external competition continue during the settling period. Packet accounting verifies that every packet generated within the measurement window has been resolved. Packets discarded due to queue overflow or deadline expiry remain in the delivery-ratio denominator.
Let be the number of training seeds and the number of test environments. Using and as their enumeration indices, let denote the measured delivery ratio of the policy trained with seed in test environment . The reported mean for a learning method is
where and . Rule-based methods are averaged over the same twenty test environments. Numerical differences reported below are expressed in percentage points and are calculated before rounding.
Four parameter sweeps vary , , , and individually while keeping the other parameters at their reference values. Each sweep contains seven values, giving 25 distinct configurations after accounting for the shared reference configuration. Increasing raises both aggregate traffic and the number of interacting links. Increasing provides additional channels while keeping per-channel propagation settings and packet-transmission duration unchanged.
Performance under Different Network Configurations
Figs. 3–6 compare the methods across the four parameter sweeps. Over the 25 configurations, the mean timely-delivery ratio of FAR-MAPPO exceeds the strongest baseline mean by 2.59–9.38 percentage points.
Number of Controlled Links
Fig. 3 varies from 8 to 20 with , , and packets/s/link. The aggregate arrival rate therefore increases from 320 to 800 packets/s while the available channel resources remain fixed. Greedy is the strongest baseline at both endpoints, with its delivery ratio decreasing from 86.06% to 70.44%. The corresponding FAR-MAPPO values are 91.44% and 78.68%, yielding improvements of 5.37 and 8.24 percentage points, respectively.
As the number of controlled links increases, more packets compete for transmission opportunities, and additional interference relationships emerge. The growing performance difference indicates the value of adapting access decisions to a larger interacting population under a fixed channel budget.
Number of External Competing Links
Fig. 4 increases from 4 to 16 while fixing , , and packets/s/link. At , IPPO achieves the highest baseline delivery ratio of 89.93%, compared with 94.16% for FAR-MAPPO. At , Greedy is the strongest baseline at 77.16%, while FAR-MAPPO achieves 83.58%. The corresponding improvements are 4.24 and 6.42 percentage points.
External competition affects both channel availability during CCA and reception quality during transmission. Consequently, local sensing and completed-attempt feedback provide complementary information for access decisions. The baseline ranking also changes with the competition level, showing that the relative effectiveness of sensing rules and learned policies depends on network conditions.
Number of Orthogonal Channels
Fig. 5 varies from 2 to 8 with and packets/s/link. With two channels, Greedy achieves 56.39%, whereas FAR-MAPPO achieves 62.42%. With eight channels, their delivery ratios increase to 94.13% and 96.87%, respectively.
Figure 5. Timely delivery versus the number of orthogonal channels, with and packets/s/link.
Additional channels allow concurrent transmissions to separate in frequency and reduce repeated contention for the same resource. The improvement over the strongest baseline is 6.03 percentage points at , 7.06 points at , and 2.75 points at .
These results indicate that the performance benefit depends on the available service opportunities. Severe channel scarcity limits the delivery capability of all methods, while abundant channels allow the baselines to approach a high delivery ratio. The largest improvement among these three points occurs at an intermediate channel availability.
Packet Arrival Rate
4) Packet Arrival Rate: Figure 6 varies the mean per-link arrival rate from 20 to 80 packets/s with and . At 20 packets/s/link, IPPO achieves 93.34%, compared with 95.93% for FAR-MAPPO, giving a difference of 2.59 percentage points. At 80 packets/s/link, Greedy provides the strongest baseline result of 50.36%, while FAR-MAPPO achieves 59.73%, increasing the difference to 9.38 percentage points.
Figure 6. Timely delivery versus the mean per-link packet arrival rate, with and .
Higher traffic loads place more packets within a limited service-time budget. Queueing, unsuccessful attempts, and protocol overhead consequently consume a larger share of the time available before packet expiry. The larger gain at high load shows that the performance difference becomes more pronounced when queueing and access overhead consume a greater share of packet lifetimes.
Impact of Traffic Burstiness and Deadline Composition
Table 2 compares four traffic and deadline combinations at , , and packets/s/link. The arrival process is either non-bursty or bursty, and packet deadlines are either uniformly 80 ms or independently selected from ms with equal probability.
| Traffic and deadline | Random | Fixed | Greedy | IPPO | MAPPO | FAR-MAPPO |
| Non-bursty, 80 ms | 89.16 | 88.59 | 93.51 | 96.59 | 92.63 | 98.54 |
| Non-bursty, mixed | 77.87 | 80.85 | 88.62 | 88.93 | 87.17 | 93.59 |
| Bursty, 80 ms | 74.33 | 79.33 | 87.56 | 88.95 | 85.30 | 93.74 |
| Bursty, mixed | 62.57 | 69.90 | 79.78 | 80.13 | 78.61 | 87.19 |
Table 2. Timely delivery (%) under different traffic and deadline combinations, with , , and packets/s/link.
The mean arrival rate remains unchanged across these combinations. Burstiness therefore changes the temporal concentration of demand. Introducing mixed deadlines adds shorter packet lifetimes and reduces the mean deadline from 80 to 46.67 ms, jointly changing deadline composition and tightness.
IPPO is the strongest baseline in all four combinations. Its delivery ratio is 96.59% with non-bursty arrivals and an 80-ms deadline. Introducing mixed deadlines reduces this value to 88.93%, while introducing bursty arrivals with the 80-ms deadline yields 88.95%. Combining burstiness and mixed deadlines further reduces the delivery ratio to 80.13%.
The corresponding FAR-MAPPO results are 98.54%, 93.59%, 93.74%, and 87.19%. Its improvements over the strongest baseline are 1.95, 4.67, 4.78, and 7.06 percentage points, respectively.
The same average offered load can thus impose substantially different timely-service requirements. Concentrated arrivals increase short-term competition for service, while shorter deadlines reduce the time available to recover from waiting or unsuccessful attempts. Their combination produces a larger separation between the compared access strategies.
Ablation Study
Table 3 compares MAPPO, Feedback-Aware MAPPO (FA-MAPPO), Residual-Network MAPPO (ResNet-MAPPO), and FAR-MAPPO. FA-MAPPO augments the ordinary MAPPO architecture with historical feedback, ResNet-MAPPO introduces the residual policy network, and FAR-MAPPO combines both components.
| Method | |||
| MAPPO | 78.61 | 52.59 | 47.14 |
| FA-MAPPO | 81.07 | 60.49 | 57.54 |
| ResNet-MAPPO | 86.72 | 62.18 | 58.28 |
| FAR-MAPPO | 87.19 | 62.42 | 59.73 |
Table 3. Component-comparison means: timely delivery (%) with .
All three configurations use , with set to , , and , where is measured in packets/s/link. They represent the reference configuration, reduced channel availability, and increased traffic load, respectively.
Relative to MAPPO, FA-MAPPO improves the mean timely-delivery ratio by 2.47, 7.90, and 10.40 percentage points across the three configurations. ResNet-MAPPO improves the mean timely-delivery ratio by 8.11, 9.59, and 11.14 points. FAR-MAPPO achieves the highest mean timely-delivery ratio in each configuration, exceeding MAPPO by 8.58, 9.83, and 12.59 points, respectively.
The comparison with each individual-component row further characterizes the combined result. FAR-MAPPO exceeds FA-MAPPO by 6.11, 1.93, and 2.19 percentage points, and exceeds ResNet-MAPPO by 0.47, 0.24, and 1.46 points. Thus, the difference from the original MAPPO backbone is larger than the additional improvement over the stronger individual-component row.
The method design assigns complementary roles to the two components: historical feedback provides channel-specific evidence about completed access attempts, while residual feature processing learns how this evidence and the current local state should influence the decision.
Delivery Performance across Deadline Classes
Fig. 7 reports timely delivery separately for packets with 20 ms, 40 ms, and 80 ms deadlines under the same three configurations as Table III. For each deadline class, the denominator includes all packets generated in that class. This comparison examines how the aggregate delivery pattern extends to packets with different time budgets.
At and packets/s/link, FAR-MAPPO achieves 80.12%, 87.79%, and 93.70% for the 20-, 40-, and 80-ms classes, respectively. For the shortest-deadline class, Greedy is the strongest baseline at 72.69%, giving an improvement of 7.43 percentage points.
When the number of channels decreases to two, the 20-ms delivery ratio of FAR-MAPPO is 52.42%, compared with 48.17% for the strongest baseline. At packets/s/link, with four channels, the corresponding values are 55.90% and 47.38%. These differences are 4.25 and 8.52 percentage points, respectively.
Conclusion and Future Work
This paper investigated timely distributed communication in cognitive radio networks with partially observable reception conditions, dynamic competition, and deadline-consuming access procedures. We developed FAR-MAPPO, which combines historical feedback representation with a residual policy network under centralized training and decentralized execution. This design integrates channel-specific access experience with current service conditions to guide local decisions. The numerical results show improved mean timely delivery across the evaluated configurations, with larger gains under heavy traffic and bursty arrivals with mixed deadlines. The improvements also extend to short-deadline packets.
Future work will extend this framework toward joint channel selection, transmission scheduling, and power control, enabling coordinated use of spectrum, time, and energy resources. We will also investigate transferable feedback representations that facilitate policy reuse across network scales and service profiles. Building on the current delivery objective, another direction is to incorporate differentiated service priorities and energy budgets into multi-agent coordination. These extensions would support a broader range of wireless applications while retaining the use of local feedback by the framework for decentralized decision-making.
Figure 7. Timely delivery across deadline classes with : (a) and packets/s/link; (b) and packets/s/link; (c) and packets/s/link. Each delivery ratio uses all generated packets in the corresponding deadline class as its denominator.
References
- [1]M. Bennis, M. Debbah, and H. V. Poor, “Ultrareliable and low-latency wireless communication: Tail, risk, and scale,” Proc. IEEE, vol. 106, no. 10, pp. 1834–1853, 2018.arxiv.org/abs/1801.01270
- [2]S. Haykin, “Cognitive radio: brain-empowered wireless communications,” IEEE J. Sel. Areas Commun., vol. 23, no. 2, pp. 201–220, 2005.DOI
- [3]Y. Wang, J. Lei, F. Shang, and Y. Li, “A comprehensive survey of multi-agent deep reinforcement learning for wireless spectrum management,” Neurocomputing, vol. 653, 2025, art. no. 131236.DOI
- [4]O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for distributed dynamic spectrum access,” IEEE Trans. Wireless Commun., vol. 18, no. 1, pp. 310–323, 2019.arxiv.org/abs/1704.02613
- [5]X. Tan, L. Zhou, H. Wang, Y. Sun, H. Zhao, B.-C. Seet, J. Wei, and V. C. M. Leung, “Cooperative multi-agent reinforcement-learning-based distributed dynamic spectrum access in cognitive radio networks,” IEEE Internet Things J., vol. 9, no. 19, pp. 19477–19488, 2022.arxiv.org/abs/2106.09274
- [6]A. Tondwalkar and A. Kwasinski, “A deep Q-learning algorithm with guaranteed convergence for distributed and uncoordinated operation of cognitive radios,” IEEE Access, vol. 13, pp. 19678–19693, 2025.DOI
- [7]S. Ye and S. Wang, “Toward order optimal channel access in unknown environments: An online learning method,” IEEE Trans. Commun., vol. 73, no. 1, pp. 292–302, 2025.
- [8]T. Lee and O. Jo, “Learning backoff: Deep reinforcement learning-based wireless channel access,” IEEE Syst. J., vol. 18, no. 1, pp. 351–354, 2024.DOI
- [9]X. Du, X. Fang, R. He, L. Yan, L. Lu, and C. Luo, “Federated deep reinforcement learning-based intelligent channel access in dense Wi-Fi deployments,” arXiv:2409.01004, 2024. [Online]. Available: https://arxiv.org/abs/2409.01004
- [10]Z. Yan, X. Gu, Z. Liu, and L. Lu, “Collaborative channel access and transmission for NR sidelink and Wi-Fi coexistence over unlicensed spectrum,” IEEE Internet Things J., vol. 12, no. 18, pp. 37 223–37 238, 2025.DOI
- [11]Q. Wang, W. Xu, and H.-H. Chen, “A heterogeneous-agent deep reinforcement learning approach for dynamic spectrum access in cognitive wireless networks,” IEEE Trans. Cogn. Commun. Netw., vol. 12, pp. 2221–2235, 2026.
- [12]J. Yu, L. Liang, C. Guo, Z. Guo, S. Jin, and G. Y. Li, “Heterogeneous multi-agent reinforcement learning for distributed channel access in WLANs,” arXiv:2412.14218, 2024. [Online]. Available: https://arxiv.org/abs/2412.14218
- [13]M. Han, Z. Chen, and X. Sun, “Multiple access via curriculum multitask HAPPO based on dynamic heterogeneous wireless network,” IEEE Internet Things J., vol. 11, no. 21, pp. 35 073–35 085, 2024.DOI
- [14]Z. Chen, X. Sun, Y. Jin, and F. Wang, “Multi-task reinforcement learning-based multiple access for dynamic wireless networks,” IEEE Trans. Mobile Comput., vol. 24, no. 9, pp. 9153–9167, 2025.DOI
- [15]Y. Shao, Y. Cai, T. Wang, Z. Guo, P. Liu, J. Luo, and D. Gündüz, “Learning-based autonomous channel access in the presence of hidden terminals,” IEEE Trans. Mobile Comput., vol. 23, no. 5, pp. 3680–3695, 2024.arxiv.org/abs/2207.03605
- [16]G. Aissou, S. Habib, R. A. Chou, and T. Kim, “Feedback-assisted decentralized Q-learning for dynamic spectrum access,” in Proc. 59th Asilomar Conf. Signals, Syst., Comput., 2025, pp. 1832–1836.DOI
- [17]N. Mohammadi, L. Liu, Y. Song, and Y. Yi, “Energy-efficient dynamic and spatiotemporal spectrum access via spiking reservoir computing,” IEEE Trans. Wireless Commun., vol. 25, pp. 1500–1514, 2026.DOI
- [18]Z. Xue, A. Gong, Y.-H. Lo, S. Tian, and Y. Zhang, “Deadline-constrained opportunistic spectrum access with spectrum handoff,” arXiv:2305.12430, 2023. [Online]. Available: https://arxiv.org/abs/2305.12430
- [19]S. Abdelrahman, H. Farag, and G. Berardinelli, “Distributed learning for reliable and timely communication in 6G industrial subnetworks,” in Proc. IEEE Middle East Conf. Commun. Netw. (MECOM), 2025, pp. 59–64.arxiv.org/abs/2506.11749
- [20]Y. Zhang and D. Guo, “Multi-agent reinforcement learning for multicell spectrum and power allocation,” IEEE Trans. Commun., vol. 73, no. 8, pp. 5980–5992, 2025.arxiv.org/abs/2312.05746
- [21]Q. Chen, X. Sun, H. Li, C. Feng, X. Wang, Q. Xue, and T. Q. S. Quek, “Reinforcement learning-based distributed channel access for delay optimization,” IEEE Internet Things J., vol. 13, no. 7, pp. 13 925–13 939, 2026.DOI
- [22]Z. Zuo, D. Wang, X. Nie, X. Pan, M. Deng, and M. Ma, “PDCF-DRL: a contention window backoff scheme based on deep reinforcement learning for differentiating access categories,” J. Supercomput., vol. 81, no. 1, 2025, art. no. 213.
- [23]S. V. Albrecht, F. Christianos, and L. Schäfer, Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. Cambridge, MA, USA: MIT Press, 2024. [Online]. Available: https://mipr ess.mit.edu/9780262049375/multi-agent-reinforcement-learning/
- [24]C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative multi-agent games,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 24 611–24 624. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/9c1535a02f0ce07943344e14d910597-Abstract-Datasets_and_Benchmarks.htmlarxiv.org/abs/2103.01955
- [25]K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778.arxiv.org/abs/1512.03385
- [26]A. Goldsmith, Wireless Communications. Cambridge, U.K.: Cambridge University Press, 2005.
- [27]G. Bianchi, “Performance analysis of the IEEE 802.11 distributed coordination function,” IEEE J. Sel. Areas Commun., vol. 18, no. 3, pp. 535–547, 2000.DOI
- [28]K. Cho, B. van Merriënboer, C. Gulcehre, Y. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in Proc. Conf. Empirical Methods Natural Language Process. (EMNLP), 2014, pp. 1724–1734.arxiv.org/abs/1406.1078
- [29]S. Huang and S. Ontañón, “A closer look at invalid action masking in policy gradient algorithms,” in Proc. Int. FLAIRS Conf., vol. 35, 2022.DOI
- [30]J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2016. [Online]. Available: https://arxiv.org/abs/1506.02438
- [31]F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev, “Time limits in reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proceedings of Machine Learning Research, vol. 80, 2018, pp. 4045–4054. [Online]. Available: https://proceedings.mlr.press/v80/pardo18a.htmlarxiv.org/abs/1712.00378
- [32]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
- [33]C. Schroeder de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson, “Is independent learning all you need in the StarCraft multi-agent challenge?” arXiv:2011.09533, 2020. [Online]. Available: https://arxiv.org/abs/2011.09533
- [34]R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” in Advances in Neural Information Processing Systems, vol. 34, 2021. [Online]. Available: https://agarwl.github.io/rlable/






