WorksPhysics and Astronomy
Reinforcement Learning designs Quantum Repeater Protocols on Networks beyond the Chain Scenario
Abstract
Small-scale quantum repeater protocols have been shown to distribute entanglement over existing telecommunication infrastructure by means of short range quantum communications and long range classical messages. For larger networks, there are a number of theoretical proposals about how to operate the network to achieve a good performance under varying metrics. Moreover, since the dimension of the space of possible protocols grows exponentially with network size, plenty of those proposals are based on heuristics that reduce the search space. One of these is entanglement routing, which simplifies the problem on non-linear networks by allocating a series of disjoint paths and running chain protocols simultaneously on them. However, in this work we claim routing leads to suboptimal protocols. In fact, inspired by works on linear chains, we apply reinforcement learning to non-linear networks and optimize over the larger non-routing protocol space. Then, we find protocols that outperform a routed extension of the swap-ASAP protocol, even in the parameter regimes where the latter is widely recognized as the state of the art. These results indicate that routing must be discarded in favor of non-routing methods, while they pave the way towards the design of a sensible Quantum Internet architecture.
Introduction
First generation quantum repeaters were proposed to fight losses when distributing long-range entanglement by splitting that distance with repeaters, generating short-range entanglement between adjacent repeaters, and then perform local measurements and long-range classical communication [1], an operation known as entanglement swapping [2]. In recent years, multiple experiments have implemented this idea and consumed the final entanglement to perform characteristic tasks, such as teleportation [3] and key distribution along an untrusted repeater [4], to name a few. Meanwhile, prepare-and-measure protocols are in a more mature state, sharing production networks with commercial traffic [5]. Comparing these two technologies, it stands to reason that future experiments will consider larger networks for entanglement distribution as well, which has led into the proposal of a road map towards the Quantum Internet [6]. In this work we focus on one of its challenges, which is to find efficient ways to manage quantum networks beyond the chain scenario such that we maximize the entanglement rate established between two given points. That is, our objective is to find efficient repeater protocols in non-chain networks.
Previous work has typically considered long chains of repeaters and their asymptotic performance under many different metrics, for a review see [7]. This has resulted in the proposal of multiple repeater protocols, such as tree-like arrangements of entanglement swapping rounds [1, 8] or protocols that have repeaters swap short-range entanglement as soon as possible (swap-ASAP) [9].
Some other protocols are the result of applying machine learning methods [10, 11] and, most notably, are shown to outperform the fixed, human designed, protocols mentioned above. We claim that, for arbitrary networks, there is good reason to believe that there are efficient protocols to be designed with the assistance of machine learning methods too. First of all, machine learning is known to deal well with the curse of dimensionality. With larger networks, chain or not, the dimension of state and operation spaces grow exponentially. This problem is often reduced by limiting the state space to pair-wise entangled states, but the operation space remains exponential. Secondly, proper performance metrics such as rates of some entanglement measure have complicated optimization landscapes over the network parameters, which rules out optimization methods tailored to specific metrics such as linear programming or flow problems. Because of these two reasons, being able to efficiently compute gradients often yields good results.
Another benefit of this machine learning approach is that it searches the full protocol space, even protocols outside of the set defined by a common heuristic, entanglement routing [12]. This term stands for the design of protocols that allocate network paths and then distribute entanglement through each path simultaneously. This heuristic is often regarded as a very practical approximation, because it greatly reduces the search space to an addition of parallel routes each of them executing well known protocols tailored to chains. However, we claim that routing policies are not optimal when compared against arbitrary protocols because both entanglement and the fundamental operation of repeaters, swapping, need not follow network connectivity. Then, limiting the optimization space to routing protocols might exclude the optimal one from search space. Therefore, in this work we focus on exploring with machine learning methods the high-dimensional space of non-routing quantum repeater protocols in networks beyond the chain scenario. In particular, we model networks with intersecting paths between target nodes, faulty generation and swapping operations, decoherence on its memories and, finally, inhomogeneity in all these parameters, as described in Section II. Then, we define the routing heuristic and extend the swap-ASAP policy to non-linear networks in Section III. We briefly introduce some concepts from reinforcement learning and describe the process of training non-routing policies in Section IV as well as establish a benchmark based on the expected rate of logarithmic negativity. We present our results in § 1.4, which are the comparison of these learnt protocols against routed swap-ASAP in terms of said entanglement rate, which is higher for the non-routing learnt policies even in networks where routed swap-ASAP is expected to have high performance. Finally, we present our conclusions in § 1.5.
Network Model
We model the quantum network as an undirected graph where nodes represent individual quantum memories (qubits) rather than full repeater stations. Unlike traditional network abstractions that treat entire repeater stations as graph nodes [10, 11], our framework shifts the fundamental unit of the graph to individual quantum memories and introduces a new type of edge to connect qubits within a repeater station. Modeling the network at repeater station level introduces unnecessary complexity, as repeaters are inherently a composite entity whose internal state and operations depends on the state and interactions of its constituent qubits among them and with the rest of the network. By defining individual qubits as the atomic graph nodes and a new type of edge that connects qubits within a repeater station, the network naturally simplifies into a collection of qubits that interact and share entanglement among them.
Formally, our model is a graph where is the set of memories and is the set of edges connecting them. To accurately reflect both the physical infrastructure and the logical state of the network, we define three different sets of edges: fiber edges , representing direct connections between qubits in different stations; station edges , linking memories within the same repeater station; and finally Bell edges , indicating entanglement between any two qubits. The first two kinds of edges represent the infrastructure of the network, while the last type represents its dynamical state. Figure 1 offers an example of all these three edges.

Figure 1. Schematic representation of one of the smallest networks. (a) In its initial state, three stations (grey rectangles) are arranged in a chain with one repeater between two end points. Each station is connected by a fiber edge (dashed line) to its neighbors. For each incoming fiber edge, there are two quantum memories (blue circles), each on a different station. Memories on the same station may be connected by a station edge (solid lines), meaning that a Bell Basis Measurement is possible between them. (b) Heralded entanglement generation (EG) takes place between memories connected by fiber edges. Successful EG creates faulty entangled states represented by Bell edges with an initial Werner parameter. (c) Entanglement swapping at the repeater station consumes two existing Bell edges to establish a Bell edge between the end points.
The state of the network, that is, the Bell edges, changes over time due to three processes: entanglement generation between adjacent repeaters, entanglement swapping at repeaters containing two halves of entangled pairs, and decoherence. There are many different experiments showcasing each of these phenomena [3, 13, 14, 15], but, in order to draw conclusions that apply to all of them, we need to formulate a model that abstracts the details of each experiment to a common model.
We start by introducing entanglement generation between adjacent repeaters. This operation is possible between memories and connected by a fiber edge . Furthermore, those memories must not be entangled with any other, so there is no Bell edge containing them. We choose to impose this restriction to limit ourselves to model bipartite pair-wise entanglement. Entanglement generation is a heralding operation, in the sense that it is stochastic and its success or failure are known to the network. The probability of generation is . Conditional on its success, it generates a Bell edge between those two memories, that we interpret as a Werner state
where is a Bell pair and the Werner parameter between the memories and . From now on, we omit the Bell state and only consider the Werner parameter as a label representing the state in that edge. If, alternatively, the operation fails, no Bell edge is added to the graph. The first and second subplots of Figure 1 provide a visual representation of this entanglement generation process in two different memory pairs.
The second process that has an impact on the network’s dynamics is entanglement swapping. This operation is the cornerstone of quantum repeaters and allows for the generation of longer range entanglement by performing a local measurement in two memories inside the same repeater. Here, we consider that two memories on the same station that share a station edge are able to perform said measurement. However, another condition must be met before performing the measurement: those two memories in must be entangled with different stations via Bell edges and . Once those two conditions are met, this operation is possible and with a heralded probability of success . Upon success, it consumes the two Bell edges and and creates a new and longer Bell edge between the memories that were not nor in and . See the second and third subplots of Figure 1 for a representation of this process. After a short calculation, the Werner parameter of the final Bell edge is
the product of the input Werner parameters [16]. Note, however, that this new Werner parameter might refer to a Werner state defined with respect a different Bell pair than the one introduced in Equation (1). That Bell pair can be determined by the outcome of the measurement at the repeater, which we assume is sent to two privileged stations that are the targets to entangle, often denoted as Alice and Bob. We do not consider part of the model performing the local operations required to fix the Bell pair shared between the output memories.
The final process that has an impact on network dynamics is decoherence. As we did with entanglement generation, we abstract the details of the many different experiments performed so far and assume decoherence takes the form of the depolarizing channel, which for Werner states results in an exponential attenuation of the Werner parameter
where is the time step of the simulation and is the composite coherence time of the two memories storing this state. Finally, if the Werner parameter of a Bell edge drops below one third, there is no entanglement between those memories and the Bell edge is discarded from the simulation.
Now that we have introduced the three fundamental processes responsible for the network’s dynamics we turn our attention to their composition over discrete time steps. Both generation and swapping act on a pair of memories connected by either a fiber or a station edge. Two operations that act on different memories commute within the formalism of quantum mechanics and therefore we assume that they can be run on the same time step. However, there is a caveat with regards to the evolution of the network when two or more copies of a swapping operation are performed simultaneously. If two repeaters are ready to swap on station edges and , and one Bell edge is an input to both swaps, then the probability of success of the whole process is the multiplication of the two and . On success they delete their input Bell edges and a new edge is created between the two memories outside the repeaters. But, if either of the swaps fails, a simple calculation shows that all entanglement is lost, including both input pairs of the successful repeater. Some of us denominate this composite operation a two-repeater entanglement swapping or 2-ES for short [17]. The generalization to -ES is straightforward. Finally, note that for larger networks in the number of memories , the number of possible Bell edges grows as . The number of fiber and station edges grows as as long as there is an upper bound to both repeater and memory degree in the graph. However, since commuting actions can be run in parallel, creating new composite actions, the set of actions grows as . Moreover, those composite actions have a great number of possible final states, and they are iterated over time, leading to a large set of possible trajectories for the system. In order to apply heuristics or machine learning to find protocols that map network states to useful actions, we need to have a performance metric to act as a compass towards efficient protocols in this exponentially large space.
We introduce two distant target nodes in the network, Alice and Bob. Broadly speaking, the objective of a protocol is to produce entanglement between them, also known as end-to-end (E2E) entanglement. Previous works measure E2E entanglement with metrics that can be sorted into, at least, four groups. Some maximize the secret key rate [16]. We discard this approach on the grounds that it binds the protocol to a specific application. Others try to maximize the number of Bell edges created between Alice and Bob, or to minimize the time taken creating them [10]. The third group notes that there are two objectives: the number/frequency of those pairs and their quality, often characterized by fidelity [18]. Here, we take a joint approach and use the addition of logarithmic negativity over all E2E Bell edges produced over time [19]. This quantity is an entanglement measure for two-qubit states. Moreover, it is an additive entanglement monotone, that is, if Alice and Bob share two pairs of memories in states and then
This property is crucial to our model. Because of it, protocols that produce many low quality E2E Bell edges can be compared against low-rate and high-quality protocols by adding this quantity over all the entangled pairs generated by both protocols over a common period of time and dividing by that time. This is the entanglement rate that we use in this work.
Routing swap-ASAP policies
Given the network model, its operations and the entanglement rate defined in Section II, we now introduce heuristics that try to guide the design of efficient repeater policies while not using machine learning.
The first example of such a policy is swap-ASAP [9], which results from the intuition that, given the state of a chain of repeaters, doing as many commuting actions as possible should lead to good results. So, if the network starts with no entanglement, the first action attempts to generate all fiber links. The second action generates again in the links that failed and, if some repeaters contain two Bell pairs, it swaps that entanglement right away. It stands to reason that this policy produces quickly E2E entanglement in the chain.
However, it is not clear how to generalize that heuristic to networks beyond the chain scenario. If given a state one chooses to run as many commuting operations as possible, short range entanglement will be swapped indeed, creating entanglement of a longer range. But it might not be getting closer to the target nodes and will be lost in the non-trivial layout of the network. Conveniently enough, previous literature considering these kind of networks has created another heuristic, entanglement routing [12]. This heuristic reduces the task of designing network protocols by taking the pair of target nodes and slicing the network infrastructure into a set of disjoint paths connecting those nodes. Then, chain based methods can be used on each path or route in parallel.
Here, we propose to apply this idea to the swap-ASAP policy in the following way. Given a network and two target stations, there is a set of possible simple paths connecting them. We select the path that has highest multiplication of heralding probabilities along its fiber and swap edges. And then we select the second highest ranking path that is disjoint from the first, and so on until we run out of paths. Then, chain-based swap-ASAP can be executed simultaneously on all selected paths. We call the resulting policy routed swap-ASAP. Just as with swap-ASAP, it stands to reason that this policy will often entangle the target nodes.
However, these are just heuristics. It has been shown in chains that swap-ASAP is not optimal under realistic conditions [10]. At least one circumstance in which swap-ASAP is clearly suboptimal can be easily understood: If a repeater contains two halves of Werner states with vastly different quality, it might make sense not to swap them, but discard the worst one and generate it again. In fact, there might be other settings in which swap-ASAP leads to suboptimal results. We conclude that if swap-ASAP is not optimal, neither is routed swap-ASAP. However, we believe that there are more reasons behind that disadvantage than memory decoherence. Therefore, in the research to follow we consider networks with high coherence times so that swap-ASAP stays close to optimal in the routed paths. Then, if we manage to find better policies than routed swap-ASAP in this parameter regime, we can conclude that routing is a heuristic that, although helpful at first, leads to suboptimal policies. This will be the objective of the remainder of this work.
Reinforcement Learning Model
Deep Reinforcement Learning (DRL) can be considered the application of neural networks to control theory. It deals with the relation of two elements: an agent and an environment. In this work, the agent can be identified as a neural network with depth greater than one composed with other functions to become the quantum repeater policy, that is, a map from states to actions. Meanwhile the environment is the quantum network. It is unfortunate that the word environment is used in the study of open quantum systems too, but we point out that our network is an open system only in the sense that its memories decohere according to Equation (3) and the word environment will only be used in the DRL sense. And in DRL, the environment is characterized by states that are feed into the policy, and it evolves depending on the actions taken by said policy. DRL has the advantage of being able to approximate arbitrary policies and estimate gradient loss through efficient backpropagation, see [20] for an introductory reference.
From the many algorithms in DRL we use Reinforce [21]. We choose this algorithm because of both its simplicity and the fact that it is well suited for continuous state and discrete action spaces, as is the case of the model in Section II. In Reinforce, time is discretized in time-steps . Its agent takes the current state of the environment , which must be a vector, and applies sequentially the neural layers indexed by . On each layer, the input vector is multiplied by learnable weights and added biases . A non-linear activation function , which in this work is set to the ReLU function, is then applied to each entry of the resulting vector
for each layer and where is . In order to identify the output of the network with actions, we set this vector to have as many entries as actions allows the environment. Then, we turn the entries of into probabilities by applying a softmax map
then the agent creates the action by sampling an index from the policy vector in Equation (6) and sends it to the environment. It, in turns, updates its state due to by sampling from its own transition probability distribution given by the rules introduced in Section II. Notice that the composition of and creates a Markov decision process that might be tuned by the learnable weights and biases . These transitions might result in rewards and this process is iterated until either a halt state or a maximum number of iterations are reached, resulting in an episode or trajectory . The sum of rewards over an episode is the total return of said episode. Finally, the objective of a RL algorithm is to update the weights of the agent in such a way that it maximizes the expected total return:
Now that we have introduced the basic notation of DRL, we refer the reader to [20] for implementation details on how to numerically solve Equation (7) by stochastic gradient descent. Here, we build on the notation introduced so far to discuss the application of this algorithm to the design of non-routing quantum repeater protocols specifically.
As mentioned in Section II, there is an exponential number of actions to be run at a given time in the network. Since in Equation (6) needs to have a dimension equal to the number of actions, then the last layer of the neural network has an exponential number of neurons with the quantum network size. To solve this problem, we turn to action serialization [22]. We consider that the space of actions is only composed of elementary actions, that is, single generations or swaps which grows as . We sort the actions in some arbitrary but constant order and then, at each reinforcement loop iteration, the agent chooses a single elementary action and is sent to the environment. The network does not evolve because of it, but rather keeps it in an action cache . Broadly speaking, the agent now creates composite actions over multiple iterations. Once the agent is done building a composite action, it selects a new action which tells the environment to take , run it, evolve a single physical time-step and empty the cache.
For the state encoding into vectors, we first considered computing the adjacency matrix of the network’s Bell edges. Then, we put in the non-zero entries of that matrix the Werner parameter of the edge it represents. However, in order to keep reinforcement loop iterations markovian, we need to extend the state with an encoding of the agents action cache. Intuitively, it makes sense that the agent needs to take the cache as input in order to know what actions to commit next. Therefore, we create a vector with entries and use the previously introduced action sorting to put ones in its entries if the action associated with that index is present in the cache. Then, we append this vector to the flattened Bell adjacency matrix and set the first layer of the neural network to match this dimension. This results in a state encoding with growth , a fair price to pay for a polynomial action space.
Another DRL technique we use is action masking [23]. Given a state, the policy discussed so far may produce an output action that makes no physical sense. For instance, it may output an action that does not commute with the action cache, leading to an invalid composite action. Another example is the choice of actions we know will fail, such as swapping with no input entanglement. These situations can be avoided if the last layer of the neural network in Equation (5) is modified with a mask that sets the activations of impossible actions to the smallest representable floating point number. Then, applying the softmax layer in Equation (6) results in a policy that assigns zero probability to impossible actions. Note that this mask must be built from the information of the state encoding alone. Other actions removed by the mask are entanglement generation in links with at least an entangled memory and the action on an empty cache.
Finally, we introduce the reward function that leads to the total return which DRL maximizes in Equation (7). Back in Section II we introduced the multiple performance metrics the literature studies when analyzing quantum repeater protocols. We settled for the expected rate of E2E logarithmic negativity, so it makes sense to choose a reward function that turns total return into that rate. In order to do so, we set to the sum of the logarithmic negativities of all E2E Bell edges in . More informative reward functions could be used, but they tend to reduce the performance of the final protocols and the serializing and masking techniques introduced above have been enough to make the model trainable.
Results
In this section we introduce the results of training the Reinforce agent described in Section IV in multiple networks. We start with a representative network that can be found in Figure 2. It consists of eight stations of varying degree. Two of these stations are regarded as the target of entanglement distribution and labeled as Alice and Bob. Notably, these two stations have a very different degree: One is connected to four fibers and the other one to two. The intermediate stations have varying degree as well, between two and three. This leads to a network with multiple intersecting simple paths between Alice and Bob, which will be a crucial property when it comes to show the advantage of the Reinforce policy against routing policies. The network parameters are inhomogeneous with probabilities of heralding generation and swap varying from 0.7 to 1. The final Werner parameter after successful link generation is 1 and the memories time of coherence is infinite. Again, these are ideal parameters intended to make swap-ASAP optimal in each route.

Figure 2. A typical network used to train Reinforce. For the definition of each node, edge colors and labels see Figure 1. In this network, probability of generation and swap range between 0.7 and 1. Initial Werner parameter of generated states is 1 and the memories have no decoherence. These parameters are ideal so that we can use routed swap-ASAP as a baseline with which to compare our Reinforce algorithm. Notice that between the end nodes Alice and Bob there are multiple possible paths, and plenty of them intersect with each other, a crucial property to make apparent that routing leads to suboptimal policies.
Notice that the layout of this network bounds entanglement rate between zero and one ebit per time step. This is due to the fact that, per time step, Bob can hold at most two perfect Bell pairs with Alice. And those two Bell pairs require at least two time steps to be created: one time step for the generation of all intermediate Bell pairs and one time steps for the -body swaps. The rest of the parameters of the network only reduce this rate by introducing failures in the generation and swapping of some links for both heuristic and machine-learnt policies. Knowing the range of values for the entanglement rate, we present the performance of both Reinforce and routed swap-ASAP on Figure 3. The Reinforce agent is trained during 400 batches of 100 episodes of 20 time-steps each, with a learning rate of 0.0025 using the ADAM optimizer. In the upper plot of Figure 3 we present a histogram of the logarithmic negativity rate obtained over 200 simulations after training the agent. The obtained average entanglement rate and standard deviation are 0.825 ± 0.101 bits per time-step. Meanwhile, in the lower plot of Figure 3 we present the logarithmic negativity of routed swap-ASAP which has and average value and standard deviation of 0.680 ± 0.101 bits per time-step, well below the rate of Reinforce in units of their standard deviations. By inspecting the trajectories in the simulations, we find that Reinforce is able to adapt to both the stochastic and non-commuting nature of the elementary operations of the network while routing policies do not. That is, the network might end up by chance in a state that could lead to long range entanglement if the swaps were arranged in a path that does not commute with the best routes, the paths used in routed swap-ASAP.

Figure 3. Histograms showing the distribution (blue) and mean (red) log negativity rates defined in Section II obtained in 200 simulations using either the trained Reinforce policy (upper plot) or routed swap-ASAP (lower plot) in the network of Figure 2. Each simulation lasted 20 time steps. The log negativity rate is obtained by adding up the log negativity of each Alice-Bob memory pair over time and dividing by the simulation’s duration. Non-routing Reinforce obtains a rate of 0.825±0.101 ebits/time-step while routed swap-ASAP obtains a lower rate of 0.680 ± 0.101 ebits/time-step, leading to the claim that the optimal protocol does not route entanglement.
In order to strengthen this claim, we indicate that the results presented so far hold for other networks as well.
Though, not all networks. As a consistency check, we report that both Reinforce and routed swap-ASAP have the same performance in chains of repeaters containing memories with very high coherence times, as it is expected [10, 11]. For the non-linear networks where we expect Reinforce to improve on routed swap-ASAP, we could repeat the analysis presented in Figure 2 and Figure 3. But we believe this approach adds little to this work. Instead, we use our insight obtained by inspecting the actions taken by trained agents and we introduce a function that assigns a score to a quantum network that summarizes whether we expect Reinforce to improve over routed swap-ASAP. Later on, we will present the improvement in entanglement rate as a function of this score for multiple networks.
We name this quantity the accumulated non-commuting score, since it represents how many non-commuting resources a network has. It is computed as follows: Given a network and a pair of target nodes, there is a set of simple paths connecting them. These paths can be ranked based on a score which is equal to the product of all the probabilities of success in their edges. The higher the score for a path the higher the entanglement rate expected from that route. As stated in Section III, routed swap-ASAP takes as a route the path with best score. The second route is the next best disjoint path to the first route, and so on until we run out of disjoint paths. After this routing process ends, there are still paths that are discarded, because they intersect with the selected paths. It stands to reason that the presence of these paths is responsible for the advantage of non-routing policies. Therefore, we define the accumulated non-commuting score as the sum of the scores of all these discarded paths. In order to increase the familiarity of the reader with this score, we point out that its values range from zero to the number of discarded paths. If the score is zero it means there are no discarded paths or it is very unlikely to obtain E2E entanglement from them. If it is equal to the number of discarded paths, then all actions on those paths (and the selected routes) are ideal and a high E2E entanglement rate is expected.
As our final results, we introduce Figure 4. It represents the average increase and standard deviation of entanglement rate of Reinforce minus routed swap-ASAP’s as a function of this accumulated non-commuting score for multiple different networks. These networks are the chain of length 5, the kite network, the and the square lattices in addition to the previously presented Figure 2. Their exact layouts, parameters and agents’ weights are provided in the publicly available repository accompanying this work [24]. We choose these networks because they are small in the number of memories while having an increasing amount of intersecting paths, that is, an increasing accumulated non-commuting score. We find that these networks and the performance of the agents trained on them confirm our insight obtained by inspecting the actions of the agent acting on the network from Figure 2. When the score is high, it is likely that the operation of the network will entangle its memories in ways that could lead to E2E entanglement if the next operations were not arranged along the initially best scoring routes. Then, Reinforce is able to adapt and produce that E2E pair while routed swap-ASAP cannot, leading to an increase in Reinforce’s entanglement rate.

Figure 4. Logarithmic negativity rate of Reinforce minus the rate of routed swap-ASAP for different networks, as a function of the proposed accumulated non-commuting score of each network. We present this figure as a summary of the main point of this article: If entanglement routing is used in a network with many intersecting routes, the resulting protocol may be outperformed by non-routing strategies, such as Reinforce. We point out that the agent trained in the network of Figure 2 contributes the data point at score around 3. The network with zero score is a five station chain and, as expected, the trained agent has the same performance as the routing strategy, resulting in zero advantage. The rest of the networks require training an agent for each of them, and for information on their layouts and parameters we refer the reader to the accompanying code repository [24].
Conclusions
In this work we have introduced a network model that summarizes the three processes that are expected to be central to entanglement distribution in telecommunication networks: entanglement generation, swapping and decoherence. Moreover, all these processes are allowed to be faulty and inhomogeneous throughout the network. A brief summary of performance metrics about end-to-end (E2E) entanglement generation in these networks is presented, leading to our choice, the rate of logarithmic negativity, that we use in the remainder of this work. Then, we introduce the reader to the exponential growth of the set of possible protocols that try to produce E2E entanglement as a function of network size and we mention that previous literature typically turns to heuristics to reduce the size of the protocol search space. Entanglement routing is a very common one, which sends the entanglement along predefined routes in the network. Our central claim is that this heuristic does not solve the optimization problem of finding efficient quantum
repeater protocols in networks beyond the chain scenario. To back this claim, we develop a reinforcement learning model tailored to the exponentially large set of non-routing protocols. In particular, we introduce a model based on Deep Reinforcement Learning as well as an action serialization procedure, together with action and state vector encodings and masking, resulting in a polynomial model growth. Finally, we train the model on multiple networks of various types and show that Reinforce obtains a higher E2E entanglement rate than heuristic entanglement-routing protocols consistently. Additionally, we provide a function that maps a given network to a score that is shown to be correlated with the improvement of Reinforce over routing policies.
The results provide valuable information in the road map towards the Quantum Internet and the design of its architecture. Entanglement must not be regarded as a package of information to be routed along paths of the network, but a correlation among the network’s elements which may not follow network connectivity. We conclude that entanglement routing should be discarded or, if used, justified with strong evidence about its convenience due to other merits than E2E entanglement rate. As part of said road map toward the Quantum Internet, this model could be extended in the following ways: Firstly, the network model could be extended to include distillation operations, which could have a fundamental impact on its performance. Secondly, the model could be extended to include classical communication delays in a more nuanced way. At present time, heralded signals are all sent to the reinforcement learning agent, which replies to each node with the actions expected from them. These delays set the minimal time step the model can use. This time step could be reduced if the agent is replaced by agents distributed over the network that work on partial information about the network [25]. Finally, the distant nodes to be entangled are fixed in this framework. However, more pairs of client nodes could be added, even in a dynamical way, with little change to the algorithm.
Acknowledgements
This work has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101297140 (QIA-Phase2) and No 101114043 (QSNP). A.A.C. acknowledges funding from the CAM Programa TEC2024/COM-84 QUITEMAD-CM. The authors gratefully acknowledge the computer resources at Artemisa, funded by the European Union ERDF and Comunitat Valenciana as well as the technical support provided by the Instituto de Fisica Corpuscular, IFIC (CSIC-UV).
References
References
- [1]H.-J. Briegel, W. Dür, J. I. Cirac, and P. Zoller, Physical Review Letters 81, 5932 (1998), publisher: American Physical Society.
- [2]M. Żukowski, A. Zeilinger, M. A. Horne, and A. K. Ekert, Physical Review Letters 71, 4287 (1993), publisher: American Physical Society.
- [3]J. M. Thomas, F. I. Yeh, J. H. Chen, J. J. Mambretti, S. J. Kohlert, G. S. Kanter, and P. Kumar, Optica 11, 1700 (2024), publisher: Optica Publishing Group.
- [4]S. Langenfeld, P. Thomas, O. Morin, and G. Rempe, Physical Review Letters 126, 230506 (2021), publisher: American Physical Society.
- [5]V. Martin, J. P. Brito, L. Ortíz, R. B. Méndez, J. S. Buruaga, R. J. Vicente, A. Sebastián-Lombraña, D. Rincón, F. Pérez, C. Sánchez, M. Peev, H. H. Brunner, F. Fung, A. Poppe, F. Fröwis, A. J. Shields, R. I. Woodward, H. Griesser, S. Roehrich, F. de la Iglesia, C. Abellán, M. Hentschel, J. M. Rivas-Moscoso, A. Pastor-Perales, J. Folgueira, and D. López, npj Quantum Information 10, 80 (2024).
- [6]S. Wehner, D. Elkouss, and R. Hanson, Science 362, eaam9288 (2018), https://www.science.org/doi/pdf/10.1126/science.aam9288.
- [7]K. Azuma, S. E. Economou, D. Elkouss, P. Hilaire, L. Jiang, H.-K. Lo, and I. Tzitrin, Rev. Mod. Phys. 95, 045006 (2023).
- [8]N. Sangouard, C. Simon, H. de Riedmatten, and N. Gisin, Reviews of Modern Physics 83, 33 (2011), publisher: American Physical Society.
- [9]L. Kamin, E. Schuchkin, F. Schmidt, and P. Van Loock, Physical Review Research 5, 023086 (2023).
- [10]Á. G. Iñesta, G. V. Ramasco, S. Cacavasso, and S. Wehner, npj Quantum Information 9, 46 (2023).
- [11]S. Haldar, P. J. Barge, S. Khatri, and H. Lee, Phys. Rev. Appl. 21, 024041 (2024).
- [12]A. Abane, M. Cubeddu, V. S. Mai, and A. Battou, IEEE Transactions on Quantum Engineering 6, 1 (2025).
- [13]S. L. N. Hermans, M. Pompili, H. K. C. Beukers, S. Baier, J. Borregaard, and R. Hanson, Nature 605, 663 (2022), publisher: Nature Publishing Group.
- [14]V. Krutyansky, M. Canteri, M. Meraner, J. Bate, V. Krčmarsky, J. Schupp, N. Sangouard, and B. Lanyon, Physical Review Letters 130, 213601 (2023), publisher: American Physical Society.
- [15]Z.-B. Cui, Z.-Q. Wang, P.-C. Lai, Y. Wang, J.-X. Shi, P.-Y. Liu, Y.-D. Sun, Z.-C. Tian, Y.-B. Liang, B.-X. Qi, Y.-Y. Huang, Z.-C. Zhou, Y.-K. Wu, Y. Xu, L.-M. Duan, and Y.-F. Pu, Nature Communications 17, 697 (2025), publisher: Nature Publishing Group.
- [16]B. Li, T. Coopmans, and D. Elkouss, IEEE Transactions on Quantum Engineering 2, 1 (2021).
- [17]J. Faba, A. A. Casado, L. Robledo, and V. Martin, in 2026 International Conference on Quantum Communications, Networking, and Computing (QCNC) (2026) pp. 734–740.
- [18]Fast and reliable entanglement distribution with quantum repeaters: Principles for improving protocols using reinforcement learning | Phys. Rev. Applied.
- [19]G. Vidal and R. F. Werner, Physical Review A 65, 032314 (2002), publisher: American Physical Society.
- [20]Foundations of Deep Reinforcement Learning: Theory and Practice in Python, iSBN: 9780135172490.
- [21]R. J. Williams, Machine Learning 8, 229 (1992).
- [22]T. Pierrot, V. Macé, J.-B. Sevestre, L. Monier, A. Laterre, N. Perrin, K. Beguir, and O. Sigaud, Factored Action Spaces in Deep Reinforcement Learning (2020).
- [23]Y. Hou, X. Liang, J. Zhang, Q. Yang, A. Yang, and N. Wang, Applied Sciences 13, 8283 (2023), publisher: Multidisciplinary Digital Publishing Institute.
- [24]To be released upon publication, reviewers can find the code and models as supplementary material to the article’s submission.
- [25]S. Haldar, P. J. Barge, X. Cheng, K.-C. Chang, B. T. Kirby, S. K. Shatri, C. W. Wong, and H. Lee, Communications Physics 8, 132 (2025), publisher: Nature Publishing Group.