betaFeedback

WorksComputer Science

Synthesis of separation processes with reinforcement learning

Abstract

This paper shows the implementation of reinforcement learning (RL) in commercial flowsheet simulator software (Aspen Plus V12) for designing and optimising a distillation sequence. The aim of the SAC agent was to separate a hydrocarbon mixture in its individual components by utilising distillation. While doing so it tries to maximise the profit produced by the distillation sequence. All actions of the agent were set by the SAC agent in Python and communicated in Aspen Plus via an API. Here the distillation column was simulated by use of the build-in RADFRAC column. With this a connection was established for data transfer between Python and Aspen and the agent succeeded to show learning behaviour, while increasing profit. Although results were generated, the use of Aspen was slow (190 hours) and Aspen was found unsuitable for parallelisation. This makes that Aspen is incompatible for solving RL problems. Code and thesis are available at https://github.com/lollcat/Aspen-RL

Introduction

Reinforcement learning is a sub-field of machine learning, in which an agent aims to learn an optimal policy which maximises its expected reward. The policy is improved by setting actions inside the environment and updating the agent based on these experiences. Previously RL has shown to be capable of outperforming humans in Chess and Go [1], but also shown success in process control [2, 3]. Recent research has shown the applicability of RL in process synthesis [4, 5, 6, 7, 8]. Reinforcement learning shows potential in handling open-end problems. Where superstructures have their optimal solution embedded in the user-defined superstructure, allows the reinforcement learning for exploration outside this pre-defined superstructure.

Reinforcement learning

A key characteristic of an RL problem is that it is defined as a Markov Decision Process (MDP). This implies that the choice of the current action only depends on the current state, see Figure 1. An MDP consists of 2 entities, an agent and environment. The agent observes the state of the environment, StS_t, and sets an action accordingly, AtA_t. This action is selected from the possible actions defined by policy π\pi. Due to the action, the environment transitions to its new state, St+1S_{t+1}. With this new created state a reward is gained, Rt+1R_{t+1}. This sequence continues until the terminal state is reached, which can be user defined. The agent will set actions such that it maximises its total expected reward, max⁡πEτ∼π[∑t=0∞γtRt]\max_{\pi} \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t R_t\right].

Markov Decision Process schematic

Figure 1. Markov Decision Process schematic [9]

Soft Actor Critic

Actor critic agents have been created to benefit from both policy optimisation and Q-learning methods. The key characteristic of an actor-critic algorithm is that the agent is composed of 2 “sub-agents”, an actor and a critic, Figure 2. An actor is learning a policy, π\pi, which performs actions maximising the future expected reward.

Actor Critic architecture

Figure 2. Actor Critic architecture [9]

To maximise exploration by the agent a soft actor critic (SAC) algorithm was implemented. With SAC, the RL objective is modified such that is tries to maximise the expected return and entropy of the agent, max⁡πEτ∼π[∑t=0∞γt(R+αH(P))]\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}(R+\alpha H(P))\right]. The temperature parameter α\alpha determines the contribution of HH in the agent’s objective where HH is the entropy which is a measure of how random the actions of the agent are. Both the actor and critic are represented by a feed-forward neural network and are parameterised by ϕ\phi and θ\theta respectively.

The network of the actor takes in the state parameters and outputs a mean, μϕ\mu_{\phi}, and standard deviation, σϕ\sigma_{\phi}, over the action-space of each action. Parameters of the network are updated with the aim to minimise the KL-divergence, equation (1). Zπold(st)Z^{\pi_{\mathrm{old}}}(s_t) is a normalisation factor [10].

πnew=arg⁡min⁡π′∈Π  DKL(π′(⋅∣st)∥exp⁡(1αQπold(st,⋅))Zπold(st))(1)\pi_{\mathrm{new}}=\underset{\pi'\in\Pi}{\arg\min}\;D_{\mathrm{KL}}\left(\pi'(\cdot\mid s_t)\middle\Vert\frac{\exp\left(\frac{1}{\alpha}Q^{\pi_{\mathrm{old}}}(s_t,\cdot)\right)}{Z^{\pi_{\mathrm{old}}}(s_t)}\right) \tag*{(1)}

Minimising the KL-divergence is done by updating the network parameters via stochastic gradient decent, equation (2) [9].

Jπ(ϕ)=Est∼D,at∼πϕ[αlog⁡(πϕ(at∣st))−Qθ(st,at)−Zπold(st)](2)J_{\pi}(\phi)=\mathbb{E}_{s_t\sim D,a_t\sim\pi_{\phi}}\left[\alpha\log(\pi_{\phi}(a_t\mid s_t))-Q_{\theta}(s_t,a_t)-Z^{\pi_{\mathrm{old}}}(s_t)\right] \tag*{(2)}

Estimating the gradient with respect to ϕ\phi is not possible, since the expectation depends on ϕ\phi. Therefore the reparameterization trick is applied, such that at=fϕ(st,ϵt)=tanh⁡(μϕ+σϕ⊙ϵ)a_t=f_\phi(s_t,\epsilon_t)=\tanh(\mu_\phi+\sigma_\phi\odot\epsilon), where ϵ\epsilon is a noise vector sampled from a normal distribution, ϵ∼N(0,1)\epsilon\sim\mathcal{N}(0,1) equation 3.

Jπ(ϕ)=Est∼D,ϵ∼N[αlog⁡(πϕ(fϕ(ϵt,st)∣st))−Qθ(st,fϕ(ϵt,st))](3)J_\pi(\phi)=\mathbb{E}_{s_t\sim\mathcal{D},\epsilon\sim\mathcal{N}}\left[\alpha\log\left(\pi_\phi(f_\phi(\epsilon_t,s_t)\mid s_t)\right)-Q_\theta(s_t,f_\phi(\epsilon_t,s_t))\right] \tag*{(3)}

Zoldπ(st)Z^\pi_{\mathrm{old}}(s_t) is neglected, since it is independent of ϕ\phi and is regarded a constant when estimating the gradient with respect to ϕ\phi. The critics’ aim is to learn an optimal soft Q function, equation 4, by minimising the mean squared error, equation 5.

Qπ(st,at)=E(st,at)∼D[r+γ(Qθ(st+1,at+1)−αlog⁡π(at+1∣st+1))](4)Q^\pi(s_t,a_t)=\mathbb{E}_{(s_t,a_t)\sim\mathcal{D}}\left[r+\gamma\left(Q_\theta(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}\mid s_{t+1})\right)\right] \tag*{(4)}
Qπ(st,at)=E(st,at)∼D[12(Qθ(st,at)−(r+γEst+1∼p[Qθˉ(st+1,at+1)−αlog⁡π(at+1∣st+1)]))2](5)Q^\pi(s_t,a_t)=\mathbb{E}_{(s_t,a_t)\sim\mathcal{D}}\left[\frac{1}{2}\left(Q_\theta(s_t,a_t)-\left(r+\gamma\mathbb{E}_{s_{t+1}\sim p}\left[Q_{\bar{\theta}}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}\mid s_{t+1})\right]\right)\right)^2\right] \tag*{(5)}

Updates of the critic network is performed by applying stochastic gradient descent following equation 6.

∇^θJQ(θ)=∇θQθ(Qθ(st,at)−(r+γ(Qθˉ(st+1,at+1)−αlog⁡(πϕ(at+1∣st+1)))))(6)\widehat{\nabla}_\theta J_Q(\theta)=\nabla_\theta Q_\theta\left(Q_\theta(s_t,a_t)-\left(r+\gamma\left(Q_{\bar{\theta}}(s_{t+1},a_{t+1})-\alpha\log\left(\pi_\phi(a_{t+1}\mid s_{t+1})\right)\right)\right)\right) \tag*{(6)}

QθˉQ_{\bar{\theta}} in equation 6 is the Q value of a target Q network. This target network is introduced to improve stability of the critic and its parameters are updated by Polyak averaging, θˉ=τθ+(1−τ)θˉ\bar{\theta}=\tau\theta+(1-\tau)\bar{\theta}. τ\tau is the target smoothing coefficient, which makes for soft target parameter updates.

To determine the relative importance between the reward and entropy the temperature α\alpha is introduced. This parameter determines the stochasticity of the policy and is updated via a multi-constrained objective given in equation 7, where H‾\overline{H} is a set target entropy.

{π∗=arg⁡max⁡∑tE(st,at)∼π[r(st,at)]s.t. E(st,at)∼π[−log⁡π(at∣st)]≥H‾, ∀t(7)\left\{\begin{aligned} \pi^*&=\arg\max\sum_t\mathbb{E}_{(s_t,a_t)\sim\pi}[r(s_t,a_t)]\\ s.t.\ &\mathbb{E}_{(s_t,a_t)\sim\pi}[-\log\pi(a_t\mid s_t)]\geq\overline{H},\ \forall t \end{aligned}\right. \tag*{(7)}

The target entropy was fixed, whilst α\alpha was automatically tuned via the following objective function 8.

J(α)=Eat∼πt[−αlog⁡πt(at∣st)−αH‾](8)J(\alpha)=\mathbb{E}_{a_t\sim\pi_t}\left[-\alpha\log\pi_t(a_t\mid s_t)-\alpha\overline{H}\right] \tag*{(8)}

Case description

For this research, a hydrocarbon mixture was considered for the feed stream with the goal to separate it in individual components with a minimum purity of 95 mol%. The feed was processed at 12.400 kmol/h with the composition stated in appendix 2. The agents’ reward was defined as

r(st,at)=revenuetop+revenuebottom−TAC−penalty(9)r(s_t,a_t)=revenue_{top}+revenue_{bottom}-TAC-penalty \tag*{(9)}

where revenue was only generated when the purity specification was 95 mol%. The Total Annualized Cost (TAC) was based on the total capital expenditures (CAPEX) and operational expenditures (OPEX) of the distillation sequence. Appendix D shows an overview of the design and TAC calculations. A set of penalties were defined to discourage non-sensible column designs, see appendix B. The state of the environment was defined as the stream conditions, such as temperature, pressure, molar flows, and revenue. The agent’s actionspace was limited to setting the number of stages, feed stage, condenser pressure, reflux -and reboil ratio. Boundaries of these parameters were set and are shown in appendix C. To keep track of all streams, a stream table was created which stores all data of a stream if the molar flow is greater or equal to 3.6 kmol/h and the purity specification is not met. Molar flows smaller then 3.6 kmol/h was regarded an outlet and was neither added to the stream table nor was it sold. To decide which stream to separate next was based on the largest molar flow present in the stream table. In order to compare the design generated by the RL agent, two base cases were created by use of Aspen Plus with optimisation via design specs and a random agent was programmed to take random actions. The settings used to run the SAC agent are shown in appendix

Case results

Base case

For the base cases two separation sequences were considered, direct separation or a tree-like sequence. Both these setups were able of generating profit without generating any waste streams.

For the linear case, four columns were put in series to separate the feed stream in its respective component. Appendix F shows a schematic of the separation train including the column specifications. Each column was able to produce a top and/or bottom stream with an equal or greater molar purity than 95 mol%. This result was gained with a stages range of 21 to 77 and a diameter ranging from 4.9 to 10.4 meters. Optimised values for the reflux ratio and distillate to feed ratio ranged from 2.72 to 11.28, and 0.35 to 0.67, respectively. The TAC of the linear separation train was 107 M€ and a yearly revenue of 1,552 M€. This results in a yearly profit of 1,445 M€.

In the case of a tree-like separation sequence two columns were put in series and the last two columns were put in parallel. This configuration allows for parallel processing of the top and bottom stream of the second column.

Appendix G shows the final column configuration with the columns respective dimensions, TAC and revenue. The range of diameter and number of stages in this configuration is from 6.5 to 9.5 meter and 21 to 85, respectively. Optimisation of the columns was done with the build-in design spec option of Aspen. For optimisation the distillate to feed ratio and reflux ratio were chosen, since this directly impacts the desired product, the distillate. The optimised values reflux ratio and distillate to feed ratio ranged from 1.09 to 8.56 and 0.34 to 0.77, respectively. A tree-like separation setup had a TAC of 107 M€, with a yearly revenue of 1,505 M€. This yields a profit of 1,398 M€.

SAC agent

For the SAC agent 2 cases will be evaluated, a maximum profit and a best average case.

Max profit

Utilising the SAC agent with the settings provided in appendix E generated a maximum profit of 1,311 M€ and an average profit of 384 M€. The maximum profit was generated by utilising 8 columns, which are shown in appendix H. Dimensions of the columns vary between 21 and 92 number of stages and a diameter ranging from 5 to 9.9 meters. All open streams in this configuration are at spec besides the bottom stream of column 4. This stream is not considered an outlet, since the molar flow is greater then 3.6 kmol/h. Hence the agent would continue separating this stream if it was allowed to. Figure 3 shows the trend in return of the RL agent. Here can be seen that there is almost no increase in return after 3000 episodes.

Episodic return and moving average

Figure 3. Return per episode and moving average of RL agent yielding the highest profit

Observing the agents’ actions for the first column provides a sense of how the agent develops over time, because for the first column the input stream is always the same. Figure 4 shows a converging trend for the reflux ratio over episodes. This is indicative for the agent adjusting the parameters to converge to an optimum. Appendix § 15 shows the critic loss and entropy of the agent. These plots together with the Figure 4 indicate a fast learning agent which takes little time to explore and start exploiting actions early on in the process. Hence it would be desired to decrease the rate at which the agent learns in order to provide sufficient time to explore new configurations.

Reflux ratio of first column

Figure 4. Reflux ratio of first column of each episode

Best average

Increasing performance based on the average profit was found with increasing the number of agent updates per episode to 4 and keeping the other parameters similar to appendix E. This adjustment yielded an average profit of 457 M€ and a maximum profit of 856 M€ within 9405 episodes. Figure 5 shows the trend in return and moving average. At first the profit increases sharp, but decreases as it continues. In comparison to the max profit case shows the adjustment a steeper increase overall. This increase is observed, since the agent observes the replay buffer more often. From Figure 5 can be seen that the agent converges towards until the 6500th episode. After this episode it starts exploring again, hence the increased variance over returns. This trend was also observed in the set column parameters.

Return per episode and a moving average of the RL agent yielding the best average

Figure 5. Return per episode and a moving average of the RL agent yielding the best average

Figure 6 shows the number of stages in the first column of each episode. It shows the same converging trend until the 6500th episode after which it start setting new values and gaining new experiences.

Number of stages of first column

Figure 6. Number of stage of the first column of each episode

Overall

The stability and speed were tracked and are presented in Table 1. From this table can be seen that the RL agent almost always chooses to separate the stream, which means that it takes every opportunity to gain experience. Another good result is the low percentage of convergence error of the flowsheet in Aspen. This indicates that the agent almost always takes actions which are solvable by the flowsheet. However, the implementation of RL in commercial simulation software showed some pitfalls with regard to speed and stability. It took the agent 193 hours to complete 10,151 episodes. This extensive time in combination with the unstable connection which rises RPC errors makes for a tedious way of performing a sensitivity analysis of the agents’ hyperparameters.

AgentEpisodesDurationSeparate
YesNo
RL agent, standard10,151193:43:1399.8%0.2%
RL agent, 4 agent updates9405160:58:4398.9%1.1%
Aspen runs
AgentLost contactConvergence errorConverged
With warningsWithout warnings
RL agent, standard0.2%0.01%23.8%76.0%
RL agent, 4 agent updates0.2%0.03%35.4%64.4%

Table 1. Convergence statistics

Conclusion and future

This paper showed the applicability of RL in a chemical engineering context. However, the RL agent was not able to outperform the created base case. To potentially outperform the base case it is necessary to increase insight on the effect of the hyperparameters. Based on the result in this study, the agent stops exploring too early, therefore the learning of the agent should be slowed down. Parameters which influence this learning are the learning rate of α\alpha and the target smoothing parameter of the QQ-target network. Lowering these will lead to an increased learning time, but will allow for more exploration by the agent.

To speed up the convergence to optimal solutions, good estimates can be passed on to the agent by first simulating the column with a DSTWU in Aspen Plus and feed the DSTWU column design to the SAC agent as initialisation.

The connection between Python and Aspen Plus was experienced as unstable and Aspen Plus acts as a rather slow RL environment. Therefore it would be advised to create a distillation model in Python, such that no external communication is required.

Appendix

Feed stream

ComponentMol%
C2H6C_{2}H_{6}0.06
C3H8C_{3}H_{8}33.69
iC4H10C_{4}H_{10}35.65
nC4H10C_{4}H_{10}15.32
iC5H12C_{5}H_{12}10.21
nC5H12C_{5}H_{12}5.13
Molar flow[kmol/h] 12,400
Temperature[°C] 105
Pressure[bar] 17

Table 2. Feed stream specifications

Agent penalties

ParameterNon-sensible conditionPenalty
QcondenserQ_{condenser}≥0\ge05 M€
QreboilerQ_{reboiler}≤0\le05 M€
TreboilerT_{reboiler}≥Tsteam\ge T_{steam}5 M€
ΔTlmcondenser\Delta T_{lm condenser}≤Tcool\le T_{cool}5 M€
Flowsheet error-10 M€

Table 3. Agent penalties

Action boundaries

ParameterTypeBound
MinMax
Number of stagesDiscrete20100
Feed stageDiscrete0.2∗nstages0.2 \ast n_{stages}0.8∗nstages0.8 \ast n_{stages}
Condenser pressureReal0.530
Reflux ratioReal120
Boilup ratioReal

Table 4. Parameter boundaries

Equations

Design equations
Column diameter
D=1.1∗max⁡(Deff)D = 1.1 * \max(D_{eff})
Where
DD= diameter[m][m]
DeffD_{eff}= effective diameter[m][m]
nsn_s= 0.2<ns<0.80.2 < n_s < 0.8[−][-]
Deff=4Φmol,nsvapor3.1416∗f∗R∗Tns∗Mwnsvapor∗1000PD_{eff} = \sqrt{\frac{4\Phi^{vapor}_{mol,n_s}}{3.1416*f} * \frac{R*T_{n_s}*Mw^{vapor}_{n_s}*1000}{P}}
Φmol,nsvapor\Phi^{vapor}_{mol,n_s}= molar vapor flow rate[mols][\frac{mol}{s}]
ff= gas load factor (fixed, 1.6)[Pa0.5][Pa^{0.5}]
RR= molar gas factor (fixed, 1.6)[Jmol∗K][\frac{J}{mol*K}]
TnsT_{n_s}= temperature[K][K]
MwnsvaporMw^{vapor}_{n_s}= vapor molar weight[gmol][\frac{g}{mol}]
PP= column pressure[Pa][Pa]
Column height
L=ns∗HETP+H0L = n_s * HETP + H_0
Where
LL= column height[m][m]
HETP= height equivalent to a theoretical plate[m][m]
H0H_0= clearance (fixed, 4)[m][m]
Condenser area
Acnd=−QcndKcnd∗ΔTlm,cndA_{cnd} = -\frac{Q_{cnd}}{K_{cnd}*\Delta T_{lm,cnd}}
Where
AcndA_{cnd}= condenser area[m2][m^2]
QcndQ_{cnd}= condenser duty, negative value[W][W]
KcndK_{cnd}= heat transfer coefficient (fixed, 500)[\frac{W}{m^2*^\circC}]
ΔTlm,cnd\Delta T_{lm,cnd}= logarithmic mean temperature difference[^\circC]
\Delta T_{lm,cnd} = \sqrt[3]{\frac{(T_{cnd}-T^{cool}_{in})(T_{cnd}-T^{cool}_{out})((T_{cnd}-T^{cool}_{in})+(T_{cnd}-T^{cool}_{out}))}{2}}
Where
TcndT_{cnd}= condenser temperature[^\circC]
TincoolT^{cool}_{in}= supply temperature cooling water (fixed, 30)[^\circC]
ToutcoolT^{cool}_{out}= return temperature cooling water (fixed, 40)[^\circC]

Table 5. Design and economical equations

Reboiler area

Arbl=QrblKrbl∗ΔTlm,rblA_{rbl} = \frac{Q_{rbl}}{K_{rbl} \ast\Delta T_{lm,rbl}}

Where

Arbl=reboiler area[m2]Qrbl=reboiler duty, positive value[W]Krbl=heat transfer coefficient (fixed, 800)[Wm2∗∘C]ΔTlm,rbl=temperature difference[∘C]\begin{aligned} A_{rbl} &= \text{reboiler area} && [m^2] \\ Q_{rbl} &= \text{reboiler duty, positive value} && [W] \\ K_{rbl} &= \text{heat transfer coefficient (fixed, 800)} && \left[\frac{W}{m^2 \ast{}^\circ C}\right] \\ \Delta T_{lm,rbl} &= \text{temperature difference} && [{}^\circ C] \end{aligned}
ΔTlm,rbl=Tsteam−Trbl\Delta T_{lm,rbl} = T_{steam} - T_{rbl}

Where

Tsteam=steam temperature (fixed, 201)[∘C]Trbl=reboiler temperature[∘C]\begin{aligned} T_{steam} &= \text{steam temperature (fixed, 201)} && [{}^\circ C] \\ T_{rbl} &= \text{reboiler temperature} && [{}^\circ C] \end{aligned}

Investment cost

I=Fcap∗CinvI = F_{cap} \ast C_{inv}

Where

I=investment[M€]Fcap=capital charge factor[−]Cinv=investment cost[M€]\begin{aligned} I &= \text{investment} && [M€] \\ F_{cap} &= \text{capital charge factor} && [-] \\ C_{inv} &= \text{investment cost} && [M€] \end{aligned}
Cinv=FL∗CeqpC_{inv} = F_L \ast C_{eqp}
FL=Lang factor (fixed, 5)[−]F_L = \text{Lang factor (fixed, 5)} \quad[-]

Equipment cost

Ceqp=Ccol+Cint+Ccnd+Crbl1000000C_{eqp} = \frac{C_{col}+C_{int}+C_{cnd}+C_{rbl}}{1000000}
Ceqp=equipment cost[M€]Ccol=column cost[€]Cint=internals cost[€]Ccnd=condenser cost[€]Crbl=reboiler cost[€]\begin{aligned} C_{eqp} &= \text{equipment cost} && [M€] \\ C_{col} &= \text{column cost} && [€] \\ C_{int} &= \text{internals cost} && [€] \\ C_{cnd} &= \text{condenser cost} && [€] \\ C_{rbl} &= \text{reboiler cost} && [€] \end{aligned}

Column cost

Ccol=0.9∗MS2018∗937.64∗D1.066∗L0.802∗Fc,col280C_{col} = \frac{0.9 \ast M_{S2018} \ast937.64 \ast D^{1.066} \ast L^{0.802} \ast F_{c,col}}{280}
Ccol=column cost[€]MS2018=Marshall & Swift equipment index 2018 (fixed, 1638.2)[−]Fc,col=correction factor for column[−]\begin{aligned} C_{col} &= \text{column cost} && [€] \\ M_{S2018} &= \text{Marshall \& Swift equipment index 2018 (fixed, 1638.2)} && [-] \\ F_{c,col} &= \text{correction factor for column} && [-] \end{aligned}
Fc,col=Fm+FpF_{c,col}=F_m+F_p
Fm=correction factor for column shell material (fixed, 1)[−]Fp=correction factor for column pressure (fixed, 1)[−]\begin{aligned} F_m&=\text{correction factor for column shell material (fixed, 1)} &&[-]\\ F_p&=\text{correction factor for column pressure (fixed, 1)} &&[-] \end{aligned}

Internal costs

Cint=0.9∗Ms2018∗97.24∗D1.55∗L∗Fc,int280C_{int}=\frac{0.9\ast M_{s2018}\ast97.24\ast D^{1.55}\ast L\ast F_{c,int}}{280}
Cint=internal cost[€]Fc,int=correction factor for internals[−]\begin{aligned} C_{int}&=\text{internal cost} &&[\text{€}]\\ F_{c,int}&=\text{correction factor for internals} &&[-] \end{aligned}
Fint,c=Fint,s+Fint,t+Fint,mF_{int,c}=F_{int,s}+F_{int,t}+F_{int,m}
Fint,s=correction factor for tray spacing (fixed, 1.4)[−]Fint,t=correction factor for tray type (fixed, 0)[−]Fint,m=correction factor for internals material (fixed, 0)[−]\begin{aligned} F_{int,s}&=\text{correction factor for tray spacing (fixed, 1.4)} &&[-]\\ F_{int,t}&=\text{correction factor for tray type (fixed, 0)} &&[-]\\ F_{int,m}&=\text{correction factor for internals material (fixed, 0)} &&[-] \end{aligned}

Condenser cost

Ccnd=0.9∗Ms2018∗474.67∗Acnd0.65∗Fc,cnd280C_{cnd}=\frac{0.9\ast M_{s2018}\ast474.67\ast A_{cnd}^{0.65}\ast F_{c,cnd}}{280}
Ccnd=condenser cost[€]Fcnd,c=correction factor for condenser[−]\begin{aligned} C_{cnd}&=\text{condenser cost} &&[\text{€}]\\ F_{cnd,c}&=\text{correction factor for condenser} &&[-] \end{aligned}
Fcnd,c=(Fhtx,P+Fhtx,d)∗Fhtx,mF_{cnd,c}=(F_{htx,P}+F_{htx,d})\ast F_{htx,m}
Fhtx,P=correction factor for pressure (fixed, 0)[−]Fhtx,d=correction factor for design type: fixed - tube sheet (fixed, 0.8)[−]Fhtx,m=correction factor for material (fixed, 1)[−]\begin{aligned} F_{htx,P}&=\text{correction factor for pressure (fixed, 0)} &&[-]\\ F_{htx,d}&=\text{correction factor for design type: fixed - tube sheet (fixed, 0.8)} &&[-]\\ F_{htx,m}&=\text{correction factor for material (fixed, 1)} &&[-] \end{aligned}

Reboiler cost

Crbl=0.9∗Ms2018∗474.67∗Arbl0.65∗Fc,rbl280C_{rbl}=\frac{0.9\ast M_{s2018}\ast474.67\ast A_{rbl}^{0.65}\ast F_{c,rbl}}{280}
Frbl,c=condenser duty, negative value[W]F_{rbl,c}=\text{condenser duty, negative value} \qquad[\text{W}]
Frbl,c=(Fhtx,P+Fhtx,d)∗Fhtx,mF_{rbl,c}=(F_{htx,P}+F_{htx,d})\ast F_{htx,m}

Condenser operation cost

Ccnd,operation=Ecnd∗Ecost,cnd1,000,000C_{cnd,operation}=\frac{E_{cnd}\ast E_{cost,cnd}}{1{,}000{,}000}
Ccnd,operation=operational cost condenser[M€]Ecnd=condenser energy[kW hyear]Ecost,cnd=energy price[ekW h]\begin{aligned} C_{cnd,operation}&=\text{operational cost condenser} &&[\text{M€}]\\ E_{cnd}&=\text{condenser energy} &&\left[\frac{\text{kW h}}{\text{year}}\right]\\ E_{cost,cnd}&=\text{energy price} &&\left[\frac{\text{e}}{\text{kW h}}\right] \end{aligned}
Ecnd=uptime∗Qcnd1000E_{cnd}=\frac{uptime\ast Q_{cnd}}{1000}
Ecost,cnd=6∗10−6∗Tcool2−0.0006∗Tcool+0.0163E_{cost,cnd}=6\ast10^{-6}\ast T_{cool}^{2}-0.0006\ast T_{cool}+0.0163

Reboiler operation cost

Crbl,operation=Qrbl∗M∗csteam∗3600∗uptimeΔHvap∗1,000,000C_{rbl,operation} = \frac{Q_{rbl} * M * c_{steam} * 3600 * uptime}{\Delta H_{vap} * 1,000,000}
Crbl,operation=operational costs reboiler[M€]M=molar weight of water (fixed, 18)[gmol]csteam=steam price at 16 bar (fixed, 18)[€ton]ΔHvap=molar heat of vaporization of 16 bar steam (fixed, 34794)[Jmol]\begin{aligned} C_{rbl,operation} &= \text{operational costs reboiler} && [M€] \\ M &= \text{molar weight of water (fixed, 18)} && \left[\frac{g}{mol}\right] \\ c_{steam} &= \text{steam price at 16 bar (fixed, 18)} && \left[\frac{€}{ton}\right] \\ \Delta H_{vap} &= \text{molar heat of vaporization of 16 bar steam (fixed, 34794)} && \left[\frac{J}{mol}\right] \end{aligned}

Reinforcement learning agent parameters

AgentMax profitBest average
OptimiserAdam
Learning rate3e-4
Discount factor0.99
TemperatureAdaptive
Target smoothing coefficient0.005
Target entropy-5 (=-dim[A])
Replay buffer sizeMin. 320,
Max. 320.000
Reward scale10
Network updates per episode14
Network
Hidden layers2
Hidden layer size128
Type of networkFeedForward

Table 6. Reinforcement learning agent parameters

Base case results - Linear

Base case linear separation sequence. The sequence contains Column 1: # of stages: 21, Diameter: 9.2, RR: 1.1, BR: 4.1, TAC: 41.7 M€; Column 2: # of stages: 77, Diameter: 9.4, RR: 5.2, BR: 5.8, TAC: 38.2 M€; Column 3: # of stages: 25, Diameter: 4.9, RR: 2.7, BR: 3.3, TAC: 11.1 M€; and Column 4: # of stages: 72, Diameter: 8.5, RR: 5.0, BR: 1.1, TAC: 16.4 M€. Arrows are numbered 1 through 9. The transitions are labelled Revenue: 291 M€ and Propane: 95 mol%; Revenue: 518 M€ and Butane: 95 mol%; Revenue: 197 M€ and nButane: 95 mol%; Revenue: 369 M€ and iPentane: 95 mol%; and Revenue: 182 M€ and nPentane: 95 mol%.

Figure 7. Base case linear separation sequence

ColumnUnit1234
Condenser pressure[bar]17.27.27.22.0
Reflux ratio[-]11.285.242.725.04
Boilup ratio[-]4.075.763.3110.64
Distillate/Feed[-]0.350.550.490.67
Condenser duty[MW]-181-130-34-49
Reboiler duty[MW]1301143444

Table 7. Linear column specifications with optimised reflux ratio and distillate/feed ratio

123456789
Molar purity mol%
Stream number123456789
C2H6C_{2}H_{6}0.10.10.00.00.00.00.00.00.0
C3H8C_{3}H_{8}33.295.00.50.90.00.00.00.00.0
iC4H10C_{4}H_{10}35.84.652.495.01.12.30.00.00.0
nC4H10C_{4}H_{10}15.40.223.54.146.895.00.91.40.0
iC5H12C_{5}H_{12}10.30.015.80.034.72.465.495.05.0
nC5H12C_{5}H_{12}5.20.07.90.017.40.333.73.695.0
Molar flow kmol/h
Stream number123456789
C2H6C_{2}H_{6}660000000
C3H8C_{3}H_{8}39953955404000000
iC4H10C_{4}H_{10}4312192412040794040000
nC4H10C_{4}H_{10}18571018471741673165617170
iC5H12C_{5}H_{12}12400124001240421198116830
nC5H12C_{5}H_{12}62306230623561845574
Total120334164786942933576174318331229604
Temperature [°C]10551.610666.152.280.565.310251.8
48.657.8
Pressure [bar]17.217.217.27.27.27.27.27.22.0
2.02.0

Table 8. Stream table linear base case

Base case results - Tree

ColumnUnit1234
Condenser pressure[bar]17.27.26.72.0
Reflux ratio[-]8.61.14.76.7
Boilup ratio[-]2.74.813.313.4
Distillate/Feed[-]0.340.770.710.66
Condenser duty[MW]-139-64-126-61
Reboiler duty[MW]874912556

Table 9. Tree-like column specifications with optimised reflux ratio and distillate/feed ratio

Base case tree-like separation sequence

Figure 8. Base case tree-like separation sequence

Molar purity [mol%]
Stream number123456789
C2_2H6_60.10.10.00.00.00.00.00.00.0
C3_3H8_833.295.01.21.60.02.30.00.00.0
iC4_4H10_{10}35.84.652.067.30.195.00.70.20.0
nC4_4H10_{10}15.40.323.329.81.02.795.01.50.0
iC5_5H12_{12}10.30.015.61.264.50.04.195.05.0
nC5_5H12_{12}5.20.07.90.134.30.00.23.395.0
Molar flow [kmol/h]
Stream number123456789
C2_2H6_6660000000
C3_3H8_8399538979999099000
iC4_4H10_{10}431218941234121241071320
nC4_4H10_{10}18571218451826181181709180
iC5_5H12_{12}124001239731166073113631
nC5_5H12_{12}623062336200339581
Total120334104792961221807432417991195611
Temperature [°C]10551.610565.055.351.310251.848.562.948.457.8
Pressure [bar]17.217.217.27.27.26.77.22.06.76.72.02.0

Table 10. Stream table tree-like base case

Reinforcement learning agent results - max profit

Reinforcement learning separation sequence - max profit

Figure 9. Reinforcement learning separation sequence - max profit

Critic loss

Figure 10. Critic loss - max profit

Entropy

Figure 11. Entropy - max profit

Reinforcement learning agent results - best average

Critic loss - Best average

Figure 12. Critic loss - Best average

Entropy - best average

Figure 13. Entropy - best average

References

  1. [1]David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi and Go through self-play.
  2. [2]J. C. Hoskins and D. M. Himmelblau. Process control via artificial neural networks and reinforcement learning. Computers & Chemical Engineering, 16(4):241–251, 4 1992.DOI
  3. [3]Joohyun Shin, Thomas A. Badgwell, Kuang Hung Liu, and Jay H. Lee. Reinforcement Learning – Overview of recent progress and implications for process control. Computers & Chemical Engineering, 127:282–294, 8 2019.DOI
  4. [4]Laurence Midgley and Michael Thomson. Reinforcement learning for chemical engineering process synthesis. Bachelor Thesis, University of Cape Town, 11 2019.DOI
  5. [5]Laurence Illing Midgley. Deep Reinforcement Learning for Process Synthesis. https://arxiv.org/abs/2009.13265v1, 9 2020.
  6. [6]Laura Stopps, Roel Leenhouts, Qinghe Gao, and Artur M. Schweidtmann. Flowsheet synthesis through hierarchical reinforcement learning and graph neural networks. https://arxiv.org/abs/2207.12051, 7 2022.
  7. [7]Quirin Göttl, Yannic Tön ges, Dominik G. Grimm, and Jakob Burger. Automated Flowsheet Synthesis Using Hierarchical Reinforcement Learning: Proof of Concept. Chemie Ingenieur Technik, 93(12):2010–2018, 12 2021.DOI
  8. [8]Quirin Göttl, Dominik G. Grimm, and Jakob Burger. Automated synthesis of steady-state continuous processes using reinforcement learning. Frontiers of Chemical Science and Engineering, 16(2):288–302, 2 2022.arxiv.org/abs/2101.04422
  9. [9]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, second edition, 2018.
  10. [10]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications. https://arxiv.org/abs/1812.05905, 12 2018.

Paper details

vlnc:W_rf2d5zdb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents