betaFeedback

WorksPhysics and Astronomy

Application of Stochastic Control Algorithms for the Improvement of the Electron Injection Efficiency of BESSY II

Abstract

Synchrotron light source storage rings aim to maintain a continuous beam current without observable beam motion during injection. One element that paves the way to this target is the non-linear kicker (NLK). The field distribution it generates poses challenges for optimising the topping-up operation. Within this study, a reinforcement learning agent was developed and trained to optimise the NLK operation parameters. We present the models employed, the optimisation process, and the achieved results.

Dem Institut für Mathematik der Technischen Universität Berlin vorgelegte

Masterarbeit

von

Matrikelnummer 399787.

Betreuer und Erstgutachter: Prof. Dr. Christoph Knochenhauer

Zweitgutachter: PD Dr. Pierre Schnizer

Eidesstattliche Erklärung

Hiermit erkläre ich, dass ich die vorliegende Arbeit selbstständig und eigenhändig sowie ohne unerlaubte fremde Hilfe und ausschließlich unter Verwendung der aufgeführten Quellen und Hilfsmittel angefertigt habe.

Berlin, den 31.01.2024

Alexander Schütt

Zusammenfassung

In dieser Arbeit wird das Prinzip des Reinforcement Learnings auf ein reales Problem angewandt. Ziel der Arbeit ist die Injektionseffizienz von BESSY II zu optimieren.

Zuerst werden die Grundlagen des Reinforcement Learnings zusammengetragen. Wir beweisen unter welchen Annahmen sich mit dem Optimalitätsprinzip von Bellman optimale Strategien finden lassen und stellen zwei auf dem genannten Prinzip basierenden Algorithmen vor. Zusätzlich wird gezeigt, wie die Parameter der Algorithmen automatisch auf das Problem optimiert werden können.

Danach widmen wir uns der Anwendung. Zunächst wird die Synchrotronstrahlungsquelle BESSY II vorgestellt. Wir werden auf die einzelnen Bauteile genauer eingehen, insbesondere auf den Non-Linear Kicker. Jener wird bei der zu optimierenden Injektion verwendet. Um ein Verständnis der Injektion von Elektronen mit dem Non-Linear Kicker aufzubauen, wird gezeigt unter welcher notwendigen Bedingung eine Injektion erfolgreich sein kann.

Daraufhin stellen wir drei verschiedene Modelle vor, wie man die Injektion mathematisch beschreiben kann. Die Modelle unterscheiden sich darin, wie viele Elektronen injiziert werden und wann entschieden werden kann in welcher Runde der Non-Linear Kicker aktiviert wird.

Um gute Modelle mit den vorgestellten Reinforcement Learning Algorithmen zu erhalten, zeigen wir, wie wir eine Simulation der BESSY II Injektion approximiert haben, um diese signifikant zu beschleunigen.

Zum Schluss zeigen wir die besten Injektionsstrategien in allen 3 Modellen und wie man diese für die reale Synchrotronstrahlungsquelle BESSY II benutzen kann.

Introduction

In this thesis, we show how stochastic control algorithms are used to increase the injection efficiency of BESSY II. We also give an outlook on how to fully automate the injection process at BESSY II. We start with an introduction to the electron injection at BESSY II and reinforcement learning.

Electron Injection at BESSY II

This thesis explores the application of stochastic control algorithms, specifically reinforcement learning algorithms, to a real world application. The goal is to improve the electron injection efficiency of BESSY II.

BESSY II is a synchrotron light source located in Adlershof, Berlin. First, it accelerates electrons in a synchrotron and then stores them in a storage ring. Then, the stored electrons are accelerated in various directions to generate synchrotron light. To maintain the energy of the synchrotron light at a constant level, new electrons need to be injected regularly. Typically, these injections happen roughly every three minutes. Currently, electrons are injected by a 4-kicker bump:

In a first step, the stored electrons, that create the synchrotron light, are kicked out of their initial path by 2 magnets (called K1,K2 in Figure 1.1). The redirected electrons

4-Kicker bump

Figure 1.1. 4-Kicker bump, by [1].

Introduction

are then merged with newly injected electrons before being kicked back (by magnets K3,K4K_3,K_4) to the stored beam position.

This process causes disturbances in the path of the stored electrons, which reduces the quality of the synchrotron light and additionally risks losing a significant fraction of the electrons in the storage ring during this injection procedure.

A new injection procedure, called the Non-Linear Kicker Injection, has been introduced to BESSY II, see [2]. This method uses a single magnet with a non-linear magnetic field to inject the electrons, as shown in the figure below.

This injection method has the advantage of only minimally affecting the stored electrons whose xx values are very close to 0, as the magnetic field strength is almost 0 in this region (see Figure 1.2). As a result, the stored electrons are only slightly disturbed and the quality of the synchrotron light is not deteriorated.

The magnetic field of the non-linear kicker

Figure 1.2. The magnetic field of the non-linear kicker. On the x axis the x-dimensional distance to the centre of the beam line is shown. On the y axis the corresponding amount of magnetic field strength is shown.

This non-linear kicker magnet has already been installed at BESSY II in 2011, see [2]. It is currently not used because the injection efficiency is lower compared to the 4-kicker bump. Recently some progress has been made to increase the efficiency by optimising the strength of the magnets leading to the non-linear kicker.

Reinforcement Learning

Reinforcement learning is a subfield of machine learning that deals with decision-making. Its foundation is based on learning by interaction, which is inspired by the way humans learn.

Through interactions, we humans can analyse how the environment around us responds to our actions. Given such a response, we get the information on whether the environment has changed in the desired way, or if we need to adapt our actions.

Consider, for example, a game of chess. The outcome of a game is influenced by the actions taken, whether it results in a win, loss or draw. Based on the outcome, we may adjust our play style to improve our chances of winning future games.

In reinforcement learning, this concept is applied. Reinforcement learning algorithms explore the environment to find a policy that maximise a numerical reward.

The mathematical foundation of reinforcement learning are Markov Decision Models. They describe the environment by providing the state and action spaces, the dynamics between them, such that the Markov property holds and additionally a reward system that rewards/punishes the agent for taking good/bad actions. Social science speaks about positive/negative reinforcement.

Once we have defined Markov Decision Models, we use the Markov property of the environment to show fundamental mathematical connections that allow us to compute optimal policies.

As these optimal policies can only be computed efficiently in discrete and finite cases, we describe different reinforcement learning algorithms. They use the mathematical connections, but in an approximate way, using neural networks. Allowing efficient calculations for continuous action and state spaces. Finally, we apply them to the injection problem.

Scope of the Thesis

This thesis is structured as follow:

Chapter 2 introduces the mathematical foundation of Reinforcement Learning. We define Markov Decision Processes and prove under which conditions the Bellman equation holds. After that, we present different algorithms for the search of good strategies.

In chapter 3, the physics models are introduced, which are the basis of the used simulation. These shall give a concise description of the physics involved.

Chapter 4 describes how the Markov decision models are formulated. In total we give 4 formulations: The first formulation describes the actual problem in BESSY II. After that we see three formulations that are able to be solved by reinforcement learning algorithms. We show the 1000-step single-electron injection, the 1000-step 1000-electron injection and finally the 1-step 1000-electron injection.

As the preceding chapters laid out the foundations, chapter 5 presents how the reinforcement learning algorithms are applied to the injection problem and gives its training results.

Reinforcement Learning

This chapter is based on chapter 2 of the book Markov decision processes with applications to finance [3] by Nicole Bäuerle and Ulrich Rieder. Some adaptations have been made, to fit the setting of our application better. For example, the set of admissible actions is identical in each state, as this is the case in our application and it simplifies the model.

We start this chapter by presenting the foundations of reinforcement learning. After the definition of a Markov Decision Model/Process, policies and the value function, we show the Bellman equation and prove that under some conditions we can find optimal policies using the Bellman equation. Next, we present temporal difference learning and two algorithms to find good policies. After that, we introduce partially observable Markov decision processes, that are needed in the non-linear kicker injection problem. At the end, we describe the process of the hyperparameter search and present different algorithms for the hyperparameter search.

Markov Decision Processes

Definition 2.1.1 (Markov Decision Model). A Markov Decision Model with planning horizon N∈NN \in\mathbb{N} is a tuple (E,A,p,p0,r,β,g)(E,A,p,p_0,r,\beta,g), where:

  • EE is a measurable space with σ\sigma-algebra E\mathcal{E}, called state space

  • AA is a measurable space with σ\sigma-algebra A\mathcal{A}, called action space

  • pp is a stochastic transition kernel from E×AE \times A to EE, i.e. for all (x,a)∈E×A(x,a) \in E \times A and B∈EB \in\mathcal{E}, the function B↦p(B∣x,a)B \mapsto p(B|x,a) is a probability measure on E\mathcal{E} and the mapping (x,a)↦p(B∣x,a)(x,a) \mapsto p(B|x,a) is measurable. The value p(B∣x,a)p(B|x,a) is the probability that if at any time nn in state x∈Ex \in E the action aa is taken, the next state at time n+1n+1 is in BB.

  • p0p_0 is called initial probability on EE

  • r:E×A→Rr : E \times A \to\mathbb{R} is a measurable function, called reward function

  • discount factor β∈(0,1]\beta\in(0,1]

  • g:E→Rg : E \to\mathbb{R} is a measurable function, called terminal reward function

Our goal now is to define a random process on the Markov Decision Model (E,A,p,p0,r,β,g)(E, A, p, p_0, r, \beta, g) with planning horizon N∈NN \in\mathbb{N}. The random process is going to take values in the state space, that can be influenced by the actions taken. For this, we consider the following measurable space:

Ω:=EN+1with σ-algebra F:=⨂n=0NE,\Omega:= E^{N+1} \quad\text{with } \sigma\text{-algebra } \mathcal{F} := \bigotimes_{n=0}^{N} \mathcal{E},

where (E,E)(E,\mathcal{E}) is the state space.

Definition 2.1.2 (Markov Decision Process). Let M=(E,A,p,p0,r,β,g)M = (E, A, p, p_0, r, \beta, g), N∈NN \in\mathbb{N} and (Ω,F)(\Omega, \mathcal{F}) be as above. For all n∈{0,…,N}n \in\{0,\ldots,N\} and ω=(ω0,…,ωN)∈Ω\omega= (\omega_0,\ldots,\omega_N) \in\Omega, we define

Xn(ω)=Xn((ω0,…,ωN)):=ωn.X_n(\omega) = X_n((\omega_0,\ldots,\omega_N)) := \omega_n.

The random process X={Xn}n=0,…,NX = \{X_n\}_{n=0,\ldots,N} is called Markov Decision Process.

Since F\mathcal{F} is the product σ\sigma-algebra on Ω\Omega, the functions XnX_n are measurable, as they are projections.

In the following, we assume that (E,A,p,p0,r,β,g)(E, A, p, p_0, r, \beta, g) denotes a Markov Decision Model with planning horizon N∈NN \in\mathbb{N} and X={Xn}n=0,…,NX = \{X_n\}_{n=0,\ldots,N} denotes a Markov Decision Process on (Ω,F)(\Omega,\mathcal{F}).

To allow the Markov Decision Process to depend on the actions taken, we first define the set of valid actions:

Definition 2.1.3 (decision rule/ policy). We call a measurable map f:E→Af:E \to A a decision rule. A sequence of decision rules π=(f0,…,fN−1)\pi= (f_0,\ldots,f_{N-1}) is called policy. The set of all policies is denoted by Ξ\Xi.

Note that if we assume that the state space EE and action space AA are not empty, the set of all policies Ξ\Xi is not empty, as all constant mappings f:E→Af:E \to A are measurable.

Remark 2.1.4. Now, we can describe a Markov Decision Process which depends on a policy.

Let π=(f0,…,fN−1)\pi= (f_0,\ldots,f_{N-1}) be a fixed policy and x∈Ex \in E an initial state. One can show that there exists a unique probability measure Pxπ\mathbb{P}^{\pi}_x on (Ω,F)(\Omega,\mathcal{F}), such that for all B∈EB \in\mathcal{E}:

  • Pxπ(X0∈B)=δx(B)\mathbb{P}^{\pi}_x(X_0 \in B) = \delta_x(B),

  • Pxπ(Xn+1∈B∣X1,…,Xn)=Pxπ(Xn+1∈B∣Xn)=p(B∣Xn,fn(Xn))\mathbb{P}^{\pi}_x(X_{n+1} \in B \mid X_1,\ldots,X_n) = \mathbb{P}^{\pi}_x(X_{n+1} \in B \mid X_n) = p(B \mid X_n, f_n(X_n)).

Reinforcement Learning

The second property is called Markov property, which tells us, that the next state only depends on the current state and action chosen, and it does not depend on the history of states and chosen actions.

Note that in our analysis, we will not go into detail about the initial probability p0p_0, as the next definitions all depend on a current state x∈Ex \in E at some time point nn.

Further, note that in our definition of Pxπ\mathbb{P}^{\pi}_{x}, the actions are applied indirectly by having the measure Pxπ\mathbb{P}^{\pi}_{x} depend on the policy π\pi.

From now on, we denote by Pxπ\mathbb{P}^{\pi}_{x} the probability measure from above, given policy π∈Ξ\pi\in\Xi and initial state x∈Ex \in E, and by Exπ\mathbb{E}^{\pi}_{x} the expectation w.r.t. Pxπ\mathbb{P}^{\pi}_{x}. Further, we define Pnxπ\mathbb{P}^{\pi}_{n x} as conditional probability Pnxπ(⋅):=Pπ(⋅∣Xn=x)\mathbb{P}^{\pi}_{n x}(\cdot) := \mathbb{P}^{\pi}(\cdot\mid X_n = x), which is well-defined by the Markov property. Denote by En,xπ\mathbb{E}^{\pi}_{n,x} the associated expectation.

We move on to compare the performances of different policies within a Markov Decision Model. In order to ensure that the following definitions are well defined, we impose following assumption:

Let r+:E×A→R≥0r^{+}: E \times A \to\mathbb{R}_{\ge0} and g+:E→R≥0g^{+}: E \to\mathbb{R}_{\ge0} denote the positive part of rr and gg, i.e. for all x∈Ex \in E and a∈Aa \in A

r+(x,a)=max⁡{r(x,a),0}andg+(x)=max⁡{g(x),0}.r^{+}(x,a) = \max\{r(x,a),0\}\quad\text{and}\quad g^{+}(x) = \max\{g(x),0\}.

Henceforth, we assume that the following inequality holds for all n∈{0,…,N}n \in\{0,\ldots,N\} and x∈Ex \in E:

sup⁡πEn,xπ[∑k=nN−1βkr+(Xk,fk(Xk))+βNg+(XN)]<∞,(1)\sup_{\pi}\mathbb{E}^{\pi}_{n,x}\left[\sum_{k=n}^{N-1}\beta^{k}r^{+}(X_k,f_k(X_k))+\beta^{N}g^{+}(X_N)\right] < \infty, \tag*{(1)}

where the supremum is taken over all policies π=(f0,…,fN−1)∈Ξ\pi=(f_0,\ldots,f_{N-1}) \in\Xi.

Definition 2.1.5 (policy reward). Let π=(f0,…,fN−1)∈Ξ\pi=(f_0,\ldots,f_{N-1}) \in\Xi be a policy. The expected total reward starting at time nn, given policy π\pi and state x∈Ex \in E is defined as

Vn,π(x):=En,xπ[∑k=nN−1βkr(Xk,fk(Xk))+βNg(XN)]∈[−∞,∞).V_{n,\pi}(x) := \mathbb{E}^{\pi}_{n,x}\left[\sum_{k=n}^{N-1}\beta^{k}r(X_k,f_k(X_k))+\beta^{N}g(X_N)\right] \in[-\infty,\infty).

The maximal expected reward starting at time nn is given as

Vn(x):=sup⁡πVn,π(x)∈[−∞,∞),V_n(x) := \sup_{\pi} V_{n,\pi}(x) \in[-\infty,\infty),

for x∈Ex \in E. We call the function Vn:E→RV_n:E \to\mathbb{R} value function.

Remark 2.1.6. By Assumption ANA_N the mappings Vn,πV_{n,\pi} and VnV_n are both well defined. If there exists a policy π∈Ξ\pi\in\Xi with V0,π(x)=V0(x)V_{0,\pi}(x) = V_0(x) for all x∈Ex \in E, we call the policy π\pi optimal and denote the optimal policy π\pi by π∗\pi^*.

Bellman Equation

In the last section we defined Markov Decision Processes on Markov Decision Models, together with policies. In this section we show how to iteratively calculate the value function Vn,πV_{n,\pi} of a policy π\pi.

References

  1. [16]James Bergstra and Yoshua Bengio. “Random Search for Hyper-Parameter Optimization”. In: Journal of Machine Learning Research 13.10 (2012), pp. 281–305. URL: http://jmlr.org/papers/v13/bergstra12a.html.
  2. [17]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005. ISBN: 9780262256834. DOI: 10.7551/mitpress/3206.001.0001.DOI
  3. [18]Julian Schwinger. “On the Classical Radiation of Accelerated Electrons”. In: (1949). DOI: 10.1103/PhysRev.75.1912. URL: https://link.aps.org/doi/10.1103/PhysRev.75.1912.DOI
  4. [19]H. A. Haus and J. R. Melcher. Electromagnetic fields and energy. Prentice-Hall International Editions, 1988.
  5. [20]J. Bengtsson, W. Rogers, and T. Nicholls. “A CAD Tool for Linear Optics Design: A Con-trols Engineer’s Geometric Approach to Hill’s Equation”. In: (2021). arXiv: 2109 . 15066 [physics.acc-ph].arxiv.org/abs/2109.15066
  6. [21]P. Schnizer et al. “Progress on Thor SCSI development”. English. In: Proc. IPAC’23 (Venice, Italy). IPAC’23 - 14th International Particle Accelerator Conference 14. JACoW Publishing, Geneva, Switzerland, May 2023, pp. 3413–3416. ISBN: 978-3-95450-231-8. DOI: 10.18429/JACoW-IPAC2023-WEPL127. URL: https://indico.jacow.org/event/41/contributions/2085.DOI
  7. [22]Lin Liu et al. Injection Dynamics for Sirius Using a Nonlinear Kicker. THPMR011 Proceedings of IPAC2016, Busan, Korea pp 3406-3408. 2016. URL: https://accelconf.web.cern.ch/ipac2016/papers/thpmr011.pdf (visited on 01/30/2024).DOI
  8. [23]Felix Armborst. “Transverse Resonance Island Buckets at BESSY II”. PhD thesis. Humboldt-Universität zu Berlin, Mathematisch-Naturwissenschaftliche Fakultät, 2022. DOI: http://dx.doi.org/10.18452/23851.
  9. [24]Charles R. Harris et al. “Array programming with NumPy”. In: Nature 585.7825 (Sept. 2020), pp. 357–362. DOI: 10.1038/s41586-020-2649-2. URL: https://doi.org/10.1038/s41586-020-2649-2.
  10. [25]Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. 2018. arXiv: 1603.07285.arxiv.org/abs/1603.07285

Paper details

vlnc:W_h244up6v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents