betaFeedback

WorksComputer Science

Adaptive optimization of wave energy conversion in oscillatory wave surge converters via SPH simulation and deep reinforcement learning

Abstract

The nonlinear damping characteristics of the oscillating wave surge converter (OWSC) significantly impact the performance of the power take-off system. This study presents a framework by integrating deep reinforcement learning (DRL) with numerical simulations of OWSC to identify optimal adaptive damping policy under varying wave conditions, thereby enhancing wave energy harvesting efficiency. Firstly, the open-source multiphysics libraries SPHinXsys and Simbody are employed to establish the numerical environment for wave interaction with OWSCs. Subsequently, a comparative analysis of three DRL algorithms-proximal policy optimization (PPO), twin delayed deep deterministic policy gradient (TD3), and soft actor-critic (SAC)-is conducted using the two-dimensional (2D) numerical study of OWSC interacting with regular waves. The results reveal that artificial neural networks capture the nonlinear characteristics of wave-structure interactions and provide efficient PTO policies. Notably, the SAC algorithm demonstrates exceptional robustness and accuracy, achieving a 10.61% improvement in wave energy harvesting. Furthermore, policies trained in a 2D environment are successfully applied to the three-dimensional (3D) study, with an improvement of 22.54% in energy harvesting. Additionally, the study shows that energy harvesting is improved by 6.42% for complex irregular waves. However, for the complex dual OWSC system, optimizing the damping characteristics alone is insufficient to enhance energy harvesting.

Introduction

Considering the significant environmental issues caused by the extensive use of fossil fuels, including pollution, greenhouse gas emissions, and ecological destruction, there has been a marked increase in the study of clean and renewable energy sources. Wind and solar power have seen rapid technological advancements over the past few decades, achieving successful commercial applications. However, as energy demand grows, a broader range of renewable energy sources has come into focus. Among these, wave energy stands out due to its substantial potential (with a minimum estimated capacity of around 0.2 TW), high energy density, strong stability, and the advantage of not occupying land resources [1]. These features have attracted significant research and development investments, making wave energy a crucial component in transitioning to a sustainable energy future. Historically, most research has focused on extracting energy from the heave motion of deep-water systems, mainly due to the common belief that nearshore wave resources are significantly lower than those in deeper waters [2, 3]. However, since the beginning of the 21st century, the concept of exploitable wave energy resources has become more realistic [4]. In many nearshore locations, the exploitable resource is typically only 10-20% lower than that offshore [5]. As a result, the extraction of wave energy from the surge motion of waves in nearshore waters has gained increasing attention.

Typical wave energy converters (WECs) can be classified into three main categories based on their working principles: oscillating water column (OWC) devices, which use the oscillating water column to compress air and drive a turbine [6], over-topping devices, which utilize the potential energy of waves as they spill over a barrier [7], and wave-activated bodies, which exploit the heave, surge, roll, or pitch motions depending on their construction [8, 9]. oscillating wave surge converters (OWSCs) are typical wave-activated bodies used in nearshore waters, usually employing bottom-hinged flap mechanisms. Notable examples include the products of WaveRoller and Oyster [10, 11]. The primary distinction between these two lies in the positioning of their flaps: WaveRoller’s flap is wholly submerged in seawater, whereas Oyster’s flap has an upper edge that protrudes above the water surface [12]. Research conducted at Queen’s University Belfast suggests that while partial submersion enhances the impact pressure exerted by the rotating flap, the flap inherently decouples from the wave as the oscillation amplitude increases [13]. This decoupling effect ensures that the wave-induced loads remain manageable, thereby safeguarding the structural integrity of the Oyster, even under extreme sea conditions. The structure of the Oyster is shown in Figure 1. The flap is connected to the base via a hinge and oscillates back and forth in response to the incident waves. This oscillatory motion drives the power take-off (PTO) system, which utilizes a hydraulic pump to channel high-pressure water through a pipeline to a hydroelectric turbine, generating electricity [14].

Schematic of Oyster® (a type of OWSC) under waves

Figure 1. Schematic of Oyster® (a type of OWSC) under waves.

According to the experimental study of the wave interacting with OWSCs by Henry et al. [15], the flap of an OWSC actively impacts the trough of incoming waves, generating breaking waves. Chow et al. [16] discovered that a system of two tandem OWSCs could improve overall wave energy conversion efficiency and determined the optimal separation distance between them based on Bragg reflection. Brito et al. [17] demonstrated through scale model experiments that the capture width ratio (CWR) and response amplitude operator (RAO) of an OWSC exhibit weak correlation under both regular and irregular wave conditions.

Numerical study is attracting more and more attention as it can be applied to to explore the detailed mechanisms of wave-structure interactions (WSI), optimize the structure of OWSCs, and improve the energy harvesting efficiency of PTO systems. For example, Folley et al. [11] utilized a linearized frequency domain model based on the Boundary Element Method (BEM). As a following, Cheng et al. [12, 18] developed a two-dimensional (2D) and three-dimensional (3D) higher-order boundary element method (HOBEM) model to estimate the performance of an OWSC. Renzi and Dias [19] also proposed a semi-analytical model for 3D computation of OWSCs. Although these methods offer high computational efficiency, they lack the accuracy of predicting nonlinear phenomena such as slamming and overtopping. Two approaches for this problem are based on the Navier-Stokes equations with either mesh-based methods or particle-based methods. Mesh-based methods are primarily implemented using commercial software such as Ansys Fluent and the open-source platform OpenFOAM. Schmitt et al. were the first to use Reynolds-averaged Navier–Stokes (RANS) equations to explore the nonlinear relationship between wave height and the optimal damping of the OWSC [20]. The numerical simulations by Wei et al. [21, 22] also agreed with experimental results in predicting the wave height and the pressure distribution on the flap. They found that the effects of viscosity and Froude scaling on OWSCs are minimal, whereas re-reflection significantly enhances or suppresses the incident waves and impacts. Jiang et al. [23] discussed the impact of different damping strategies on power generation efficiency in PTO systems. They found that nonlinear damping strategies can enhance system stability but do not significantly increase power output. However, these methods come with significant computational costs due to the inclusion of additional Volume of Fluid (VOF) equations and the necessity of using dynamic mesh techniques to solve for the motion of the flap [24, 25].

Particle-based methods, i.e., SPH, are particularly well-suited for addressing challenges involving large deformations, complex free surface flows, and multiphase flows [26, 27, 28], making them a compromising alternative to study the hydrodynamic interactions between waves and WECs. Specifically, for OWSC calculations, Henry et al. [15] and Rafiee et al. [29] used a modified SPH method to perform 2D and 3D numerical simulations. They employed a Riemann solver to compute wave density and incorporated the Lagrangian form of the RANS k−ϵk-\epsilon model into the equations to capture the turbulence characteristics. The results indicated that 3D simulations can more accurately predict the pressure distribution on the flap. Brito et al. [30] presented a numerical model combining DualSPHysics for wave computation and Chrono for nonlinear mechanical constraint systems of OWSCs. Later, they updated the numerical model to consider most constraints, such as the PTO system, revolute joints, and frictional contacts [31]. Wei et al. [32] proposed a similar solution, replacing the SPH library with the GPUSPH code. Their simulation results showed that when the natural frequency of the OWSC structure is similar to the wave period, it can maximize power capture. Liu et al. [33] quantitatively analyzed the effects of parameters such as load, flap mass, thickness, hinge height, and damping of the PTO system on motion resonance and wave absorption of OWSC. Zhang et al. [34] used a Riemann-based weakly compressible method based on SPHinXsys to compute incident waves and WSI. Simobdy is employed for solid dynamics.

Their results demonstrated that this solver could accurately predict wave height and the pressure distribution on the flap while significantly reducing computation time, showing great potential for practical applications.

Currently, the performance optimization of OWSCs primarily relies on experiments, theoretical analysis, and numerical simulations under specific wave conditions to conduct parametric studies on the structure and position of flaps or damping of the PTO system [35, 36, 37, 38]. There is still a lack of research on effective control strategies to enhance the performance of OWSCs. In comparison, control strategy methods have already been applied to optimize the wave energy absorption performance of other WEC devices [39, 40, 41]. However, the interaction between waves, especially irregular waves, and OWSCs is complex, highly stochastic, and nonlinear. Models such as latching and model predictive control (MPC) exhibit poor robustness and are highly dependent on the accuracy of the predictive model. Inaccurate models can significantly affect performance [42]. In the past decade, with the rapid advancement of artificial intelligence (AI), deep reinforcement learning (DRL) has demonstrated significant capabilities in the field of active flow control [43, 44, 45, 46]. DRL combines artificial neural networks (ANN) and reinforcement learning (RL). Since ANN uses nonlinear activation functions and can effectively fit any function, DRL, compared to traditional RL, enhances the exploration and capture of high-dimensional state spaces, making it suitable for WSI problems [47]. Rabault et al. [48] were the first to use DRL to achieve drag reduction in the problem of flow around a cylinder through real-time control of the jets on both sides. Research on optimizing WECs using DRL employs potential wave theory as the environment. Studies have shown that DRL can enhance the energy harvesting efficiency of WECs [49, 50]. Additionally, compared to directly using computational fluid dynamics (CFD) as the environment, this approach significantly reduces the time required to train the ANN. However, potential wave theory is less effective at capturing the coupling effects between multiple physical fields. Only Liang et al. [51] have combined 2D CFD with DRL to optimize the wave energy conversion of horizontal floating cylinders under irregular waves.

In this paper, we will first establish a new platform that combines CFD with DRL. Based on our previous work, we will use SPHinXsys, a multi-physics library based on the SPH method, and Simbody as the numerical computation platform for the bottom-hinged OWSC [52]. Given that mainstream DRL platforms like Tianshou employ neural networks such as PyTorch, which are based on the Python environment, we will use Pybind11 to package the relevant OWSC code into a dynamic link library for invocation in the OpenAI Gymnasium environment [53]. This standardized environment will facilitate the direct application of various DRL algorithms available on the Tianshou platform, enabling us to explore the impact of these algorithms on performance improvement. Additionally, in the specific training process, we will not simply define the damping coefficient of the PTO system as linear, rather, it will be a parameter that varies in real-time with the incident waves and the motion state of the flap. This parameter will be controlled using a neural network optimized by the RL algorithm.

The remainder of this paper is organized as follows: § 2 introduces the Riemann-based SPH method for hydrodynamic modeling in SPHinXsys, the simulation of dynamic damping characteristics using Simbody, the de tailed coupling process, wave-making theory, and validation against theoretical and experimental results. § 3 discusses the details of our CFD-DRL framework, the DRL training environment, and the mainstream RL algorithms. § 4 analyzes the performance differences of various DRL algorithms in the OWSC problem, explores the applicability of 2D2\mathrm{D} and 3D3\mathrm{D} numerical computations to the policies, verifies that the adaptive damping coefficient policy can be applied to random irregular waves, and investigates the feasibility of wave energy optimization under the dual OWSC systems comparing it with the optimal layout and the best combination of constant damping coefficients.

Numerical modeling

Accurate numerical simulation of the wave interaction with OWSCs is crucial for building a reliable environment for RL. In this section, we utilize the Lagrange-based numerical computing library SPHinXsys to perform precise calculations. Specifically, we introduce an improved weakly compressible SPH (WCSPH) method [54], corrected with weighted kernel gradient (WKG) [55] to handle the accurate prediction of wave dynamic and its interaction with the OWSC. Furthermore, we present a comprehensive coupling approach between SPHinXsys and the Simbody library to capture the WSI.

Governing equation

In the Lagrangian framework, the mass and momentum conservation equations for incompressible and viscous fluid can be written as

{dρdt=−ρ∇⋅vdvdt=−1ρ∇p+ν∇2v+g.(1)\begin{cases} \frac{d\rho}{dt} = -\rho\nabla\cdot\mathbf{v} \\ \frac{d\mathbf{v}}{dt} = -\frac{1}{\rho}\nabla p + \nu\nabla^2\mathbf{v} + \mathbf{g}. \tag*{(1)} \end{cases}

Here, ρ\rho is the density of the fluid, v\mathbf{v} the velocity, pp the pressure, ν\nu the kinematic viscosity, and g\mathbf{g} is the gravity. An artificial isothermal equation of state (EoS) is used to close the system of Eq. (1)

p=c2(ρ−ρ0),(2)p = c^2(\rho-\rho^0), \tag*{(2)}

where cc is the artificial speed of sound, ρ0\rho^0 the initial density. Following the weakly-compressive assumption, c=10vmaxc = 10v_{max} is applied to ensure that the density varies around 1% [56]. Here, vmaxv_{max} is the maximum anticipated particle velocity in the flow. In this paper, we set vmax=2ghv_{max} = 2\sqrt{gh} with hh denoting the water depth.

Riemann-based WCSPH method

In this work, Eq. (1) is discretized with Riemann-SPH method [54] as

{dρidt=2ρi∑jVj(U∗−vi⋅eij)∂Wij∂rijdvidt=−mi∑j2P∗ρiρj∇iWij+mi∑j2μ vijρiρjrij∂Wij∂rij+gi.(3)\begin{cases} \frac{d\rho_i}{dt} = 2\rho_i\sum_j V_j(U^*-\mathbf{v}_i\cdot\mathbf{e}_{ij})\frac{\partial W_{ij}}{\partial r_{ij}} \\ \frac{d\mathbf{v}_i}{dt} = -m_i\sum_j\frac{2P^*}{\rho_i\rho_j}\nabla_iW_{ij}+m_i\sum_j\frac{2\mu\,\mathbf{v}_{ij}}{\rho_i\rho_jr_{ij}}\frac{\partial W_{ij}}{\partial r_{ij}}+\mathbf{g}_i. \tag*{(3)} \end{cases}

Here, mim_i and ρi\rho_i are the mass and density of particle ii, VjV_j the particle volume, vij=vi−vj\mathbf{v}_{ij} = \mathbf{v}_i-\mathbf{v}_j particle relative velocity, and μ\mu is the dynamic viscosity. Also, ∇Wij=eij(∂W(rij,h)/∂rij)\nabla W_{ij} = \mathbf{e}_{ij}(\partial W(r_{ij},h)/\partial r_{ij}) with eij=rij/rij\mathbf{e}_{ij} = \mathbf{r}_{ij}/r_{ij} and rij=ri−rj\mathbf{r}_{ij} = \mathbf{r}_i-\mathbf{r}_j, and WijW_{ij} represents the Kernel gradient [57]. Besides, U∗U^* and P∗P^* are the solutions of the one-dimensional Riemann problem constructed along the line pointing from particle ii to jj. The left and right initial states of the Riemann problem can be reconstructed as

(ρL,UL,PL,cL)=(ρi,−vi⋅eij,pi,ci)(ρR,UR,PR,cR)=(ρj,−vj⋅eij,pj,cj).(4)\begin{aligned} (\rho_L,U_L,P_L,c_L) &= (\rho_i,-\mathbf{v}_i\cdot\mathbf{e}_{ij},p_i,c_i) \\ (\rho_R,U_R,P_R,c_R) &= (\rho_j,-\mathbf{v}_j\cdot\mathbf{e}_{ij},p_j,c_j). \tag*{(4)} \end{aligned}

To solve this one-dimensional Riemann problem, Zhang et al. [54] adopted a linearised Riemann solver coupled with a dissipation limiter

U∗=ρLcLUL+ρRcRUR+PL−PRρLcL+ρRcRP∗=ρLcLPR+ρRcRPL+ρLcLρRcRβ(UL−UR)ρLcL+ρRcR,(5)\begin{aligned} U^* &= \frac{\rho_L c_L U_L+\rho_R c_R U_R+P_L-P_R}{\rho_L c_L+\rho_R c_R} \\ P^* &= \frac{\rho_L c_L P_R+\rho_R c_R P_L+\rho_L c_L\rho_R c_R\beta(U_L-U_R)}{\rho_L c_L+\rho_R c_R}, \tag*{(5)} \end{aligned}

with low dissipation limiter β=min⁡(3max⁡(UL−UR,0)/cˉ,1)\beta= \min(3\max(U_L-U_R,0)/\bar{c},1) where cˉ=(ρLcL+ρRcR)/(ρL+ρR)\bar{c} = (\rho_L c_L+\rho_R c_R)/(\rho_L+\rho_R).

Weighted kernel gradient correction

Current research indicates that using the Riemann-based WCSPH method to solve the wave dynamics leads to significant numerical dissipation, which affects the simulation of small waves [58]. The kernel gradient correction (KGC) method can effectively reduce the numerical dissipation and improve energy conservation [59].

Following Ref. [60], the correction matrix in Eq. (6) is introduced to modify the kernel gradient calculation to ensure first-order consistency.

Bi=(−∑jrij⊗∇iWijVj)−1=(Ai)−1(6)\mathbf{B}_i = \left(-\sum_j \mathbf{r}_{ij}\otimes\nabla_i W_{ij}V_j\right)^{-1} = (\mathbf{A}_i)^{-1} \tag*{(6)}

However, this correction encounters problems such as computational instability when the determinant of the matrix Ai\mathbf{A}_i approaches zero. In order to improve the robustness of numerical simulation, a generalized and consistent weighted kernel gradient correction (WKGC) proposed by Ren et al. [55] is adopted as

B~i=ω1Bi+(1−ω1)I,ω1=∣Ai∣ϵ+∣Ai∣.(7)\begin{aligned} \widetilde{\mathbf{B}}_i &= \omega_1\mathbf{B}_i + (1-\omega_1)\mathbf{I},\\ \omega_1 &= \frac{|\mathbf{A}_i|}{\epsilon+|\mathbf{A}_i|}. \tag*{(7)} \end{aligned}

Here, I\mathbf{I} is the identity matrix, and ϵ=0.3\epsilon=0.3.

For the Riemann-based WCSPH, the kernel correction matrix is introduced to modify the solution of pressure item P∗P^* in Eq. (5) as

P∗=ρLcLPRB~i+ρRcRPLB~j+ρLcLρRcRβ(UL−UR)ρLcL+ρRcR.(8)P^*=\frac{\rho_L c_L P_R\widetilde{\mathbf{B}}_i+\rho_R c_R P_L\widetilde{\mathbf{B}}_j+\rho_L c_L\rho_R c_R\beta(U_L-U_R)}{\rho_L c_L+\rho_R c_R}. \tag*{(8)}

Dual-criteria time stepping

In order to improve computational efficiency, dual-criteria time-stepping method is adopted here. Specifically, the update frequency of the particle configuration is controlled by the advection criterion, while the integration of pressure relaxation is determined by a smaller time step size based on the acoustic criterion. Following Zhang et al. [54], the time step size of the advection criterion Δtad\Delta t_{ad} and the acoustic criterion Δtac\Delta t_{ac} are

Δtad=CFLadmin⁡(h∣v∣max⁡,h2ν),Δtac=CFLac(hc+∣v∣max⁡).(9)\begin{aligned} \Delta t_{ad} &= CFL_{ad}\min\left(\frac{h}{|\mathbf{v}|_{\max}},\frac{h^2}{\nu}\right),\\ \Delta t_{ac} &= CFL_{ac}\left(\frac{h}{c+|\mathbf{v}|_{\max}}\right). \tag*{(9)} \end{aligned}

where CFLad=0.25CFL_{ad}=0.25 and CFLac=0.6CFL_{ac}=0.6. More details are refered to Ref. [54].

SPHinXsys and Simbody coupling

To compute the solid-body kinematics in wave interaction with OWSCs, we adopted the Simbody library, which provides an object-oriented C++ API. This feature allows it to be coupled with SPHinXsys and exposed to the Python environment. In the current framework, all media, including rigid bodies such as the flap and hinge, are presented with particles. The forces exerted from the fluid on the rigid bodies and damping coefficients are transmitted to Simbody via SPHinXsys. For predicting the combined translational and rotational motion by solving the Newton-Euler equations.

In detail, the entire calculation sequence begins with the calculation of the fluid properties. The fluid density ρi\rho_i will be initialized firstly at the beginning of the advection time step Δtad\Delta t_{ad} as

ρi=max⁡(ρ∗,ρ0∑Wij∑Wij0)(10)\rho_i = \max\left(\rho^*, \rho^0 \frac{\sum W_{ij}}{\sum W_{ij}^0}\right) \tag*{(10)}

where ρ∗\rho^* denotes the density before re-initialization and ρ0\rho^0 represents the initial reference value. The viscous force faν\mathbf{f}_{a\nu} is also computed at this stage. Subsequently, pressure relaxation is carried out over the next several acoustic time steps Δtac\Delta t_{ac} using the position-based Verlet scheme proposed by Zhang et al. [61]

{ρin+12=ρin+12Δtac(dρidt)n+12rin+12=rin+12Δtacvin.(11)\begin{cases} \rho_i^{n+\frac{1}{2}} = \rho_i^n + \frac{1}{2}\Delta t_{ac}\left(\frac{d\rho_i}{dt}\right)^{n+\frac{1}{2}} \\ \mathbf{r}_i^{n+\frac{1}{2}} = \mathbf{r}_i^n + \frac{1}{2}\Delta t_{ac}\mathbf{v}_i^n . \tag*{(11)} \end{cases}

After that the particle’s velocity vi\mathbf{v}_i, density ρi\rho_i, and position ri\mathbf{r}_i are updated to the mid-points as

vin+1=vin+Δtac(dvidt)n+1ρin+1=ρin+12+12Δtac(dρidt)n+12rin+1=rin+12+12Δtacvin+1.(12)\begin{aligned} \mathbf{v}_i^{n+1} &= \mathbf{v}_i^n + \Delta t_{ac}\left(\frac{d\mathbf{v}_i}{dt}\right)^{n+1}\\ \rho_i^{n+1} &= \rho_i^{n+\frac{1}{2}} + \frac{1}{2}\Delta t_{ac}\left(\frac{d\rho_i}{dt}\right)^{n+\frac{1}{2}}\\ \mathbf{r}_i^{n+1} &= \mathbf{r}_i^{n+\frac{1}{2}} + \frac{1}{2}\Delta t_{ac}\mathbf{v}_i^{n+1}. \tag*{(12)} \end{aligned}

Then, the forces acting on the flap of the OWSC will be computed, which are composed of two main components, the pressure force fap\mathbf{f}_{ap} and the viscous force fav\mathbf{f}_{av}, as

fap=−2∑iViVaρidpad+pidρadρi+ρad∇Waifav=2∑iνViVavi−vadrai∂Wai∂rai.(13)\begin{aligned} \mathbf{f}_{ap} &= -2\sum_i V_i V_a \frac{\rho_i^d p_a^d + p_i^d\rho_a^d}{\rho_i+\rho_a^d}\nabla W_{ai}\\ \mathbf{f}_{av} &= 2\sum_i \nu V_iV_a\frac{\mathbf{v}_i-\mathbf{v}_a^d}{r_{ai}}\frac{\partial W_{ai}}{\partial r_{ai}}. \tag*{(13)} \end{aligned}

Here, the subscript aa denotes the solid particle index. The imaginary pressure padp_a^d and velocity vad\mathbf{v}_a^d read as follows

pad=pi+ρimax⁡(0,(g−dvadt)⋅n)(rai⋅n)vad=2vi−va,(14)\begin{aligned} p_a^d &= p_i+\rho_i\max\left(0,\left(\mathbf{g}-\frac{d\mathbf{v}_a}{dt}\right)\cdot\mathbf{n}\right)(\mathbf{r}_{ai}\cdot\mathbf{n})\\ \mathbf{v}_a^d &= 2\mathbf{v}_i-\mathbf{v}_a, \tag*{(14)} \end{aligned}

where n\mathbf{n} represents the normal direction of the solid body to fluid. The total force and total torque τ\tau acting on the flap can be written as

F=∑a∈Nfa=∑a∈N(fap+fav)τ=∑a∈N(ra−rG)×fa,(15)\begin{aligned} \mathbf{F} &= \sum_{a\in N}\mathbf{f}_a=\sum_{a\in N}(\mathbf{f}_{ap}+\mathbf{f}_{av})\\ \tau&= \sum_{a\in N}(\mathbf{r}_a-\mathbf{r}_G)\times\mathbf{f}_a, \tag*{(15)} \end{aligned}

where NN denotes the total number of solid particles and rG\mathbf{r}_G is the position vector of the flap mass center.

At the end of each time step, the total force and total torque calculated from SPHinXsys are transmitted to Simbody for solving the Newton–Euler

F=mI0dvdtτ=J0dΩdt−kdΩ.(16)\begin{aligned} \mathbf{F} &= m\mathbf{I}_{0}\frac{d\mathbf{v}}{dt}\\ \boldsymbol{\tau} &= \mathbf{J}_{0}\frac{d\boldsymbol{\Omega}}{dt}-k_{d}\boldsymbol{\Omega}. \tag*{(16)} \end{aligned}

where mm is the mass of the flap, I0\mathbf{I}_{0} the identity matrix, J0\mathbf{J}_{0} the moment of the inertia about the center of mass, ω\boldsymbol{\omega} the angular velocity and kdk_{d} the damping coefficient. After updating the position, velocity, and normal direction, the new kinematic state will be imported back to SPHinXsys, and the loop will continue.

It is worth noting that previous research [34] introduced a linear damper for modeling the PTO system of the OWSC. The damping coefficient kdk_{d} was fixed during the simulation. In this paper, during the computation of Eq. (16), the kdk_{d} from the policy is variable within a specific range, and it can be rapidly implemented through the damping update definition in Simbody.

Wave generation

In the Riemann-based WCSPH method, the solid wall boundary can be discretized by dummy particles. For fluid particles around the wall region, the interaction is determined by solving a one-sided Riemann problem along the wall-normal direction [54, 62] where the left state is defined

(ρL,UL,PL)=(ρi,−nw⋅vi,pi),(17)(\rho_{L}, U_{L}, P_{L})=(\rho_{i},-\mathbf{n}_{w}\cdot\mathbf{v}_{i},p_{i}), \tag*{(17)}

where nw\mathbf{n}_{w} is the normal direction of the wall, and ii represents the fluid particles. The right state of the velocity URU_{R} and pressure PRP_{R} are assumed as

UR=−UL+2uwPR=PL+ρig⋅riw,(18)\begin{aligned} U_{R} &= -U_{L}+2u_{w}\\ P_{R} &= P_{L}+\rho_{i}\mathbf{g}\cdot\mathbf{r}_{iw}, \tag*{(18)} \end{aligned}

where uw\mathbf{u}_w is wall velocity, riw=rw−ri\mathbf{r}_{iw}=\mathbf{r}_w-\mathbf{r}_i, ρi\rho_i is computed from Eq. (2).

A piston-type wave maker consisting of a group of dummy particles whose displacement is determined by the linear and nonlinear wave maker theory [63] can generate regular and irregular waves. Based on the linear wave maker theory, the displacement function of the wave maker rar_a relies on

ra=Esin⁡(2πft+ψ)E=H(sinh⁡(2kh)+2kh)sinh⁡(2kh)tanh⁡(2kh).(19)\begin{aligned} r_a &= E\sin(2\pi f t+\psi) \\ E &= \frac{H(\sinh(2kh)+2kh)}{\sinh(2kh)\tanh(2kh)}. \tag*{(19)} \end{aligned}

Here, EE is the wave stroke, ff the wave frequency, ψ\psi the wave phase, HH the wave height, and hh the water depth. The wave number kk followed by the dispersion relation [64]

ω2=gktanh⁡(kh),(20)\omega^2=gk\tanh(kh), \tag*{(20)}

where ω=2πf\omega=2\pi f is the wave angular frequency.

For the second-order Stokes wave, the theory by Madsen [64] is adopted, where the wave maker motion is

ra=−S0cos⁡(2πft)−S0(3Hsin⁡(4πft)4n0h(4sinh⁡2(kh)−n0/2))n0=12(1+2khsinh⁡(2kh))S0=H2n0tanh⁡(kh).(21)\begin{aligned} r_a &= -S_0\cos(2\pi f t)-S_0\left(\frac{3H\sin(4\pi f t)}{4n_0h(4\sinh^2(kh)-n_0/2)}\right) \\ n_0 &= \frac{1}{2}\left(1+\frac{2kh}{\sinh(2kh)}\right) \\ S_0 &= \frac{H}{2}\frac{n_0}{\tanh(kh)}. \tag*{(21)} \end{aligned}

The Pierson-Moskowitz and JONSWAP models primarily represent the spectra of irregular waves. The JONSWAP spectrum exhibits a more concentrated wave energy than the Pierson-Moskowitz spectrum, making it more effective in describing the energy distribution of waves in the developmental stage [6]. Considering that OWSCs are predominantly placed in nearshore areas, where waves are typically in this developmental stage, it is appropriate to use the JONSWAP spectrum to generate irregular waves [65]

{S(f)=βJHp2Tp−4f−5exp⁡[−1.25(Tpf)−4]γJexp⁡[−(Tpf−1)22δJ2]βJ=0.0624(1.094−0.01915ln⁡γJ)0.230+0.0336γJ−0.185(1.9+γJ)−1,(22)\left\{ \begin{aligned} S(f) &= \beta_J H_p^2 T_p^{-4} f^{-5} \exp\left[-1.25(T_p f)^{-4}\right]\gamma_J^{\exp\left[-\frac{(T_p f-1)^2}{2\delta_J^2}\right]} \\ \beta_J &= \frac{0.0624(1.094-0.01915\ln\gamma_J)}{0.230+0.0336\gamma_J-0.185(1.9+\gamma_J)^{-1}}, \end{aligned} \right. \tag*{(22)}

where HpH_p is the main wave height, TpT_p the peak wave period, f=ω/2πf=\omega/2\pi the wave frequency, and γJ=3.3\gamma_J=3.3 the peak enhancement factor, δJ\delta_J dependent on the peak frequency fp=1/Tpf_p=1/T_p [66]. Our study employed a combination of NN random regular waves to simulate nonlinear waves. The wave number knk_n for each regular wave was consistent with Eq. (20). The displacement equation rNr_N can be written as

rN=∑n=1NE(fn)cos⁡(2πfnt+ψn).(23)r_N=\sum_{n=1}^{N}E(f_n)\cos(2\pi f_n t+\psi_n). \tag*{(23)}

Here, E(fn)=2Sω(fn)ΔfE(f_n)=\sqrt{2S_\omega(f_n)\Delta f}, f1=0f_1=0 Hz, fN=3fpf_N=3f_p, Δf=1/Ttotal\Delta f=1/T_{total} with TtotalT_{total} the total time of simulation, ψn\psi_n represents the random phases, and Sω(fn)S_\omega(f_n) is defined as

Sω(fn)=S(fn)(4sinh⁡2(kh)2kh+sinh⁡(2kh))−2.(24)S_\omega(f_n)=S(f_n)\left(\frac{4\sinh^2(kh)}{2kh+\sinh(2kh)}\right)^{-2}. \tag*{(24)}

In addition, to prevent numerical divergence caused by excessive movement of the wave maker during the initial computation, we introduced a relaxation time trex=1t_{rex}=1 s, and the final wave maker displacement function is as follows

rN={sin⁡(πt2)rN,t≤trexrN,t>trex.(25)r_N= \begin{cases} \sin\left(\frac{\pi t}{2}\right)r_N, & t\le t_{rex} \\ r_N, & t>t_{rex}. \end{cases} \tag*{(25)}

Also, to mitigate the impact of wave reflection off the wall on the motion of the OWSC, the wave-particle velocity v\mathbf{v} in the damping zone is given by

v=v0(1.0−αΔt(r−r0r1−r0)).(26)\mathbf{v}=\mathbf{v}_0\left(1.0-\alpha\Delta t\left(\frac{\mathbf{r}-\mathbf{r}_0}{\mathbf{r}_1-\mathbf{r}_0}\right)\right). \tag*{(26)}

v0\mathbf{v}_0 the fluid particle velocity at the entrance of the damping zone, the reduction coefficient α\alpha is set as 5.0, r0\mathbf{r}_0 and r1\mathbf{r}_1 are the initial and final position vectors of the damping zone.

Numerical validations

In this section, we primarily verify the accuracy of generating regular and irregular waves and validate the numerical simulations of the 2D and 3D OWSC by comparing them with experimental results.

Regular waves

In this part, we consider the regular wave equation in 2D for validation. The geometry of the numerical simulation is shown in Figure 2. The total length of the tank LL is 15 m, the water depth hh is 0.691 m, and the damping zone is 4.47 m long, the same size as the wavelength λ\lambda. The wave parameters are H=0.2H=0.2 m and wave period T=2T=2 s. We place a wave height sensor at a distance of 4 m from the wave maker. As shown in Figure 3, our numerical result of wave frequency and amplitude is very close to the analytical solution conducted by Madsen [64], which indicates that the wave is generated correctly.

The 2D water tank geometry for wave generation verification

Figure 2. The 2D water tank geometry for wave generation verification.

Comparison of wave heights under different resolutions

Figure 3. Comparison of wave heights under different resolutions at x=4.0x = 4.0 m with theoretical results.

Furthermore, we set three different particle resolutions from coarse to fine. The resolution of dp=0.015dp = 0.015 m not only improves the accuracy of the calculation results but also reduces the computation time. Therefore, this resolution will be used for all subsequent 2D simulations.

Irregular waves

For the irregular wave generation, the tank geometry and water depth are the same as regular waves. The typical parameters for irregular waves are Hp=0.2H_p = 0.2 m and Tp=2.0T_p = 2.0 s. Note that TtotalT_{total} is set for 40 s and 100 s for comparison. Figure 4 presents the wave height time series near the wave maker at x=0.2x = 0.2 m. The frequency spectrum is obtained by applying the Fast Fourier Transform (FFT) to the wave height, as shown in Figure 5. Notably, our numerical results demonstrate a high degree of agreement with the analytical results derived from the JONSWAP spectrum, and for RL training, Ttotal=40T_{total} = 40 s is set for one training episode.

Time series of the free-surface elevation at $x = 0.2$ m for the irregular wave

Figure 4. Time series of the free-surface elevation at x=0.2x = 0.2 m for the irregular wave.

Comparison between the analytical JONSWAP spectrum of {xref}`Eq. (22)` and the SPH spectrum corresponding to different time series

Figure 5. Comparison between the analytical JONSWAP spectrum of Eq. (22) and the SPH spectrum corresponding to different time series.

2D and 3D validaitons

In this part, the present method for modeling wave interaction with the OWSC is validated in both 2D and 3D. The water tank and OWSC geometries are based on the experiment conducted at the Marine Research Group’s hydraulics laboratory at Queen’s University Belfast [21], as shown in Figure 6. The length, width, and height of the wave tank are 18.4 m, 4.58 m, and 1.0 m. The flap shape of the OWSC is simplified as a box-type with the dimensions of 0.12 m ×\times 1.04 m ×\times 0.48 m. It is set at 7.92 m far from the wave maker in the xx direction and the middle in the zz direction. The water depth hh is 0.691 m, and the hinge height is 0.16 m. The mass and inertia of the flap are 33 kg and 1.84 kg⋅m21.84\ \mathrm{kg}\cdot\mathrm{m}^2. The damping coefficient kdk_d is set to 20. In addition, we set three wave height sensors in the xx direction, which are WP04 (3.99 m), WP05 (7.02 m), and WP12 (8.82 m).

Schematic of the wave tank and the OWSC model

Figure 6. Schematic of the wave tank and the OWSC model.

In the current simulation, the wave maker creates a simple harmonic wave with the wave height H=0.2H = 0.2 m and wave period T=2T = 2 s. The simulation parameter is 1 : 25 in scale, and both 2D and 3D results presented herein have been converted to full scale as in Ref. [34]. The 2D simulation is computed on a Mac OS system, with an Apple M1 Max core and 32GB RAM, while the 3D simulation is carried on a Windows system, with an AMD Ryzen 9 7950X and 48GB RAM. The total fluid particle numbers are 40027 and 1.542 million, for 2D and 3D simulations, respectively. From Figure 7, we can see that WP04 is far away from the flap, so the interaction of the wave and flap has little influence on the wave height, and our results show that the wave maker can generate an accurate sine wave with minor errors. WP05 and WP12 are set near the flap, and we can find that our simulations can capture the influence of interaction on wave height in both places. Furthermore, by comparing the wave heights at WP05 and WP12, it is evident that the wave height significantly decreases after passing through the OWSC, indicating an energy reduction. This reduction indirectly demonstrates that a portion of the energy has been converted into wave energy. Figure 8 shows that the flap rotation simulation results are consistent with the experiment. The 3D results are much better, where antisymmetric diffracted waves traveling in all directions, including tangentially to the flap, can induce near-resonant phenomena that enhance the exciting torque on the converter [67].

Comparison of free surface elevations at WP04

(a) WP04

Comparison of free surface elevations at WP05

(b) WP05

Comparison of free surface elevations at WP12

(c) WP12

Figure 7. Comparison of free surface elevations for wave height H=5.0H = 5.0 m and wave period T=10T = 10 s at different wave probes. (a) x=3.99x = 3.99 m for WP04, (b) x=7.02x = 7.02 m for WP05, and (c) x=8.82x = 8.82 m for WP12

Comparison of the flap rotation

Figure 8. Comparison of the flap rotation.

The results demonstrate that our simulations based on SPHinXsys are suitable for setting the physical environment for RL. Notably, a 2D simulation requires only 3.5 minutes of computing time for 12 seconds of physical time calculation, whereas a 3D simulation demands 3.5 hours. Given that each DRL training session necessitates at least 200 episodes, we employ 2D simulations for training and utilize 3D simulations for policy validation.

Direct deep reinforcement learning

Figure 9 provides an overview of the CFD-DRL platform. The DRL training process consists of two key components: the environment and the agent. The previous chapter details the CFD environment. Here, the focus shifts to the agent. The sampled state vector from SPHinXsys is normalized and passed to the agent. The reward is calculated based on the change in state between two consecutive time steps and the variation in reward parameters during the action application. All experiences are uniformly stored in the replay buffer for future use. We show a typical framework of the DRL algorithm: soft actor critics (SAC), which takes actor-critic architecture [68]. The policy network (actor) outputs actions fed into Simbody and critic networks in real-time. The critic networks evaluate the quality of these actions, and the smaller QiQ_i is chosen to update the policy network with gradient ascent. The critic networks will be updated with target networks using gradient descent.

The current CFD-DRL framework encompassing the CFD Environment and the DRL Agent

Figure 9. The current CFD-DRL framework encompasses an integrated interaction process between two primary components: the CFD Environment and the DRL Agent.

DRL environments

The numerical simulation environment, corresponding observation probes, and action transition functions are created using SPHinXsys. Pybind11 is utilized to package the entire computational module as dynamic link libraries (DLLs) or shared objects (SOs), which can then be imported and executed directly within the Python environment [69].

A standard DRL environment is built based on OpenAI’s Gym library [70], which includes two essential functions: reset and step. In the reset function, the SPHinXsys-based numerical simulation environment is initialized, and the initial observations are collected using probes. The step function receives action values from the DRL algorithm, passes them to SPHinXsys for numerical calculations under the current action, collects new observations, and calculates the reward.

Rabault et al. [48] shown that more observations will give the agent more information to update its network and improve the training result. Therefore, 38 observations are set as the state vector sns_n from the environment to capture the structure of the regular wave and its impact on the flap. The components of the observations are based on two parts, as shown in Figure 10. The first part is the wave properties, including velocity and wave height at five positions, starting from x=3.0x = 3.0 m to x=7.6x = 7.6 m, which is close to one wavelength. Another 5 points from y=0.3y = 0.3 m to y=0.7y = 0.7 m on the front panel of the OWSC device are also set to get the wave velocity. The second part is the characteristics of the flap, including flap rotation and angular velocity, and damping coefficient kdk_d of the PTO system. The x-axis and y-axis total force on the flap are not considered as they fluctuate broadly with too much noise.

Two main components of the observation vector

Figure 10. Two main components of the observation vector.

Besides, it is ideal that the observations from 2D and 3D simulations can be close. Thus, the observation positions are set on the middle plane in the zz-axis as shown in Figure 6, and only xx-axis and yy-axis properties are considered.

In the present work, the action ana_n in one action time step ta=0.1t_a = 0.1 s of variation in the damping coefficient ∣Δkd∣≤25\lvert\Delta k_d\rvert\le25. However, directly changing the damping coefficient has the potential risk to the computation divergence. Therefore, the current numerical damping coefficient kdnk_d^n and the subsequent numerical damping coefficient kdn+1k_d^{n+1} have a simple linearly increasing change of Δkd/M\Delta k_d/M during the time step of ta/Mt_a/M. Considering that the Δtac=0.00021\Delta t_{\mathrm{ac}} = 0.00021 s, M=10M = 10 is sufficient to ensure accuracy in the 2D simulation. Note that a large damping coefficient will make the flap’s rotation challenging. A small damping coefficient will result in larger deflection angles, which may cause the OWSC break in real applications. In present work, 0≤kd(N⋅m⋅s/rad)≤1000 \le k_d(N \cdot m \cdot s/rad) \le100 is applied. As observed in Figure 8, we can see that even if the damping coefficient is set to 0, the most extensive rotation of the flap is 50∘50^\circ, which is smaller than the critical value. Although the entire OWSC can still run under extreme boundary conditions, a penalty term is given in the reward as P∗=−1⋅[(kd+Δkd<0)∨(kd+Δkd>100)]P^* = -1 \cdot[(k_d + \Delta k_d < 0) \lor(k_d + \Delta k_d > 100)].

Based on Senol and Raessi’s work [36], the instance energy harvesting in one action time step can be defined as

Ptake−off=∑n=0M−1kdn(Ωn+1+Ωn2)2.(27)P_{take-off}=\sum_{n=0}^{M-1} k_d^n\left(\frac{\Omega_{n+1}+\Omega_n}{2}\right)^2. \tag*{(27)}

Here, Ωn\Omega_n is the flap angular velocity. It is evident that larger Ptake−offP_{take-off} means more instantaneous wave energy harvesting by the OWSC. The study of Zhang et al. [34] shows that the average wave energy harvesting factor (CF) of 3D OWSCs can reach a peak when kd∗k_d^* is around 40 N⋅m⋅s/rad40\ \mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad}. Combined with Table 1, the verification results show that in the case of 2D OWSC, the total wave energy conversion reaches a maximum value at kd∗=45 N⋅m⋅s/radk_d^*=45\ \mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad}. Then, the instance energy harvesting in one action time step with kd∗k_d^* is recorded as the baseline PbaselineP_{baseline} in the reward to help the agent distinguish good from harmful actions. The final reward can be calculated as

kdk_d (N⋅m⋅s/rad\mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad})10203040455060708090100
EtE_t (J)195.19281.38309.56321.26321.96316.52310.86300.92291.02279.37267.47

Table 1. The variations of the total wave energy conversion EtE_t in terms of damping coefficients.

rn=Ptake−off−Pbaseline+P∗.(28)r_n=P_{take-off}-P_{baseline}+P^*. \tag*{(28)}

DRL algorithms

RL algorithms can be broadly categorized into two types: on-policy and off-policy. On-policy algorithm updates its policy network after one episode and uses the newly updated policy to collect data in the next episode. The typical algorithm is proximal policy optimization (PPO) [71]. On the other hand, the policy for updating and collecting is different for the off-policy algorithm. The typical methods are twin delayed deep deterministic policy gradient (TD3) [72] and SAC.

More specifically, the action value function Qπθ(sn,an)Q_{\pi_\theta}(s_n, a_n) and the state value function Vπθ(sn)V_{\pi_\theta}(s_n) can be defined as:

{Qπθ(sn,an)=Eπθ[∑t=n∞(γtrt∣st,at)]Vπθ(sn)=Ean∼πθ[∑t=n∞(γtrt∣st)],(29)\left\{ \begin{aligned} Q_{\pi_\theta}(s_n,a_n) &= \mathbb{E}_{\pi_\theta}\left[\sum_{t=n}^{\infty}(\gamma_t r_t\mid s_t,a_t)\right] \\ V_{\pi_\theta}(s_n) &= \mathbb{E}_{a_n\sim\pi_\theta}\left[\sum_{t=n}^{\infty}(\gamma_t r_t\mid s_t)\right], \end{aligned} \right. \tag*{(29)}

where πθ\pi_\theta is the policy network with parameter θ\theta, γt∈[0,1]\gamma_t \in[0,1] the discount factor for the future reward, and ∑t=n∞γtrt\sum_{t=n}^{\infty}\gamma_t r_t usually defined as return GnG_n. The policy network plays an important role in the optimization process, as its input is the current state sns_n, and its output is the action ana_n or the probability density function of the action πθ(⋅∣sn)\pi_\theta(\cdot\mid s_n). The action value function Qπθ(sn,an)Q_{\pi_\theta}(s_n,a_n) represents the expected return obtained by taking action ana_n in the current state sns_n and then following the policy πθ\pi_\theta. The state value function Vπθ(sn)V_{\pi_\theta}(s_n) represents the expected return obtained by following the policy πθ\pi_\theta and the current state sns_n.

The PPO algorithm

The core of the PPO algorithm is to build an objective function JJ to characterize the return GnG_n under the parameter θ\theta of the current policy network. The best policy and return are obtained by updating θ\theta to maximize the objective function. Based on the Policy Gradient Theorem (PGT), the objective function J(θ)J(\theta) can be written as [71]

J(θ)=Esn∼D[Ean∼πθk[πθ(an∣sn)πθk(an∣sn)⋅Aπθk(sn,an)]],(30)J(\theta) = \mathbb{E}_{s_n\sim\mathcal{D}}\left[\mathbb{E}_{a_n\sim\pi_{\theta_k}}\left[\frac{\pi_{\theta}(a_n|s_n)}{\pi_{\theta_k}(a_n|s_n)}\cdot A^{\pi_{\theta_k}}(s_n,a_n)\right]\right], \tag*{(30)}

where D\mathcal{D} is the replay buffer, θk\theta_k the old parameter of the policy network. Also, Aπθk(sn,an)A^{\pi_{\theta_k}}(s_n,a_n) is the advantage function which can be defined as

Aπθk(sn,an)=rn+γVϕπθk(sn+1)−Vϕπθk(sn),(31)A^{\pi_{\theta_k}}(s_n,a_n) = r_n + \gamma V_{\phi}^{\pi_{\theta_k}}(s_{n+1}) - V_{\phi}^{\pi_{\theta_k}}(s_n), \tag*{(31)}

where Vϕπθk(sn)V_{\phi}^{\pi_{\theta_k}}(s_n) is the estimated value of state sns_n by the critic network with parameters ϕ\phi, the superscript πθk\pi_{\theta_k} indicates that the data is collected using the policy employed during the kk-th iteration, and γ\gamma is the discount factor. The advantage function will not affect the expectation but improve the policy’s performance [73]. A key feature of PPO is the clipped surrogate objective, which is designed to prevent huge policy updates. The objective function of J(θ)J(\theta) is then rewritten as follows

J(θ)=Esn∼D[Ean∼πθk[min⁡(rθ(sn,an),clip⁡(rθ(sn,an),1−σ,1+σ))⋅Aπθk(sn,an)]],(32)J(\theta) = \mathbb{E}_{s_n\sim\mathcal{D}}\left[\mathbb{E}_{a_n\sim\pi_{\theta_k}}\left[\min\left(r_{\theta}(s_n,a_n),\operatorname{clip}(r_{\theta}(s_n,a_n),1-\sigma,1+\sigma)\right)\cdot A^{\pi_{\theta_k}}(s_n,a_n)\right]\right], \tag*{(32)}

where rθ(sn,an)=πθ(an∣sn)/πθk(an∣sn)r_{\theta}(s_n,a_n)=\pi_{\theta}(a_n|s_n)/\pi_{\theta_k}(a_n|s_n), and σ=0.2\sigma=0.2. Note that the objective function J(θ)J(\theta) is only relevant with the policy network parameter θ\theta, and our target is to update the policy by maximizing J(θ)J(\theta) with stochastic gradient ascent with Adam scheme [74]:

θk+1←θk+α∇θJ(θ),(33)\theta_{k+1}\leftarrow\theta_k+\alpha\nabla_{\theta}J(\theta), \tag*{(33)}

where α\alpha is the learning rate.

The critic network VϕV_{\phi} in PPO is primarily used to represent the state value function and its loss function is defined as

L(ϕ)=Esn∼D[(Vϕ(sn)−Rn)2]Rn=rn+γVϕk(sn+1).(34)\begin{aligned} L(\phi) &= \mathbb{E}_{s_n\sim\mathcal{D}}\left[(V_{\phi}(s_n)-R_n)^2\right]\\ R_n &= r_n+\gamma V_{\phi_k}(s_{n+1}). \tag*{(34)} \end{aligned}

To update the critic network parameters ϕ\phi, the value function loss is minimized using gradient descent as

ϕk+1←ϕk−α0∇ϕL(ϕ).(35)\phi_{k+1}\leftarrow\phi_k-\alpha_0\nabla_{\phi}L(\phi). \tag*{(35)}

The TD3 algorithm

The TD3 algorithm is based on policy gradient methods, where the policy network πθ\pi_{\theta} outputs actions ana_n directly. The value networks QϕiQ_{\phi_i}, called critics, provide value estimates based on sns_n and ana_n suggested by the policy network. TD3 employs two value networks to mitigate overestimation issues and obtains more reliable value estimates using the minimum value predicted by the two networks [72]. Each network (the policy and the two value networks) is paired with a corresponding target network. Therefore, a total of six networks are utilized during the training process.

The objective function of a TD3 policy network is defined as

J(θ)=Esn∼D[Qϕ1(sn,πθ(sn))].(36)J(\theta)=\mathbb{E}_{s_n\sim\mathcal{D}}\left[Q_{\phi_1}(s_n,\pi_{\theta}(s_n))\right]. \tag*{(36)}

Also the loss for the critic networks is given by temporal difference (TD)

L(ϕi)=Esn∼D[(Qϕi(sn,an)−yn)2], i=1,2.(37)L(\phi_i)=\mathbb{E}_{s_n\sim\mathcal{D}}\left[(Q_{\phi_i}(s_n,a_n)-y_n)^2\right],\ i=1,2. \tag*{(37)}

Here, the target value yny_n is calculated based on the Bellman Equation (BE)

yn=rn+γmin⁡i=1,2Qϕi′(sn+1,πθ′(sn+1)+ϵ),(38)y_n=r_n+\gamma\min_{i=1,2}Q_{\phi_i^{\prime}}\left(s_{n+1},\pi_{\theta^{\prime}}(s_{n+1})+\epsilon\right), \tag*{(38)}

where Qϕi′Q_{\phi_i^{\prime}} means the target critic network, πθ′\pi_{\theta^{\prime}} is the target policy network and ϵ\epsilon is the truncated Gaussian noise.

Gradient ascent on the policy network and gradient descent on the critic networks are performed to update their corresponding parameters. Note that the update frequency of the policy network and three target networks is generally lower than the critic networks.

The SAC algorithm

SAC consists of a policy network that outputs an action probability density function, two value networks, and two target networks corresponding to the value networks.

In the PPO state space exploration primarily relies on sampling from the action probability distribution output by the policy network, whereas TD3 achieves exploration by artificially adding noise to the output of the action. Compared with the PPO and TD3 algorithms, the SAC algorithm incorporates the entropy of the policy into the state-value function, encouraging exploration by maximizing the return regularized by entropy

Vπθ(sn)=Ean∼πθ[∑t=n∞(γtrt∣st)+βH(πθ(sn))],(39)V_{\pi_\theta}(s_n) = \mathbb{E}_{a_n \sim\pi_\theta}\left[\sum_{t=n}^{\infty}(\gamma^t r_t \mid s_t) + \beta\mathcal{H}(\pi_\theta(s_n))\right], \tag*{(39)}

where β\beta is the regularization coefficient.

The objective function of the policy network is

J(θ)=Esn∼D[min⁡i=1,2Qϕi(sn,a~n)−βlog⁡πθ(a~n∣sn)],(40)J(\theta) = \mathbb{E}_{s_n \sim\mathcal{D}}\left[\min_{i=1,2} Q_{\phi_i}(s_n,\tilde{a}_n) - \beta\log\pi_\theta(\tilde{a}_n \mid s_n)\right], \tag*{(40)}

where a~n\tilde{a}_n is the sample from πθ(⋅∣sn)\pi_\theta(\cdot\mid s_n).

Also, the loss for the critic networks is also calculated by TD, while with a different definition of yny_n

{L(ϕi)=Esn∼D[(Qϕi(sn,an)−yn)2],i=1,2yn=rn+γ(min⁡i=1,2Qϕi′(sn+1,a~n+1)−βlog⁡πθ(a~n+1∣sn+1)),(41)\left\{ \begin{aligned} \mathcal{L}(\phi_i) &= \mathbb{E}_{s_n\sim\mathcal{D}}\left[\left(Q_{\phi_i}(s_n,a_n)-y_n\right)^2\right], \quad i=1,2\\ y_n &= r_n+\gamma\left(\min_{i=1,2}Q_{\phi_i'}(s_{n+1},\tilde{a}_{n+1})-\beta\log\pi_\theta(\tilde{a}_{n+1}\mid s_{n+1})\right), \end{aligned} \right. \tag*{(41)}

where a~n+1\tilde{a}_{n+1} is the sample from πθ(⋅∣sn+1)\pi_\theta(\cdot\mid s_{n+1}).

Result

In this section, various RL algorithms for optimizing the energy harvesting of the PTO system of an OWSC under regular wave conditions will be first analyzed. Then, the generalization capability of 2D training policy in 3D simulations will be verified. Subsequently, the training of agents under irregular wave conditions is investigated. Last but not least, optimizing the damping coefficient in a strongly nonlinear system comprising a dual OWSC system is explored.

Study of DRL algorithms

Initially, three typical DRL algorithms are trained:i.e., PPO, TD3, and SAC. The parameters of the policy and critic networks under the three algorithms are consistent, with two hidden layers and 512 neurons in each layer. Other settings of the neural network and algorithm hyperparameters are shown in Table 2.

AlgorithmPPOTD3SAC
Activation functiontanhtanhtanh
Learning rate (α\alpha)3e-43e-41e-3
Steps per epoch204820482048
Batch size256256256
Discount factor (γ\gamma)0.990.990.99
Soft update (τ\tau)-0.0050.005

Table 2. Basic hyperparameters of different DRL algorithms.

Figure 11 illustrates the wave height at x=7.6x=7.6 m in front of the flap. It is observed that around the 4-second mark, the wave reaches and interacts with the flap. Consequently, the training phase is initiated at the 4-second mark and continues until the 24-second mark, encompassing 20 seconds and 200 actions. The testing phase extends over 40 seconds, incorporating 400 actions.

Wave height in front of the flap

Figure 11. Wave height in front of the flap.

The overall reward curves with standard deviation shadows are depicted in Figure 12. The training consisted of 220 episodes, with TD3 and SAC requiring the pre-collection of data for the first 20 episodes, a process of gathering initial data before the actual training. Pre-collection resulted in oscillations around -80 in the reward curves during this phase. However, both algorithms quickly identified effective energy enhancement strategies within the first ten episodes after training commenced. Considering TD3 incorporates noise artificially to enrich exploration, it displayed instability and only began to stabilize after 90 episodes.

Total reward curves in the training process with three agents.

Figure 12. Total reward curves in the training process with three agents.

On the other hand, SAC explored and converged to optimal strategies by approximately 60 episodes with entropy regularization. PPO, not requiring initial data collection, showed a slower but steady improvement from the beginning, converging to effective strategies around 80 episodes. From the reward trends observed between episodes 160 and 220, SAC demonstrates its best performance with the slightest standard deviation, indicating superior stability over other two algorithms. In addition, since the incident wave is regular, the dynamic curve of the damping coefficient is also periodic, as shown in Figure 13. It can be observed that the overall fluctuation of the PPO algorithm is substantial, whereas the TD3 algorithm converges to a locally optimal solution due to insufficient exploration. Therefore, SAC will be applied for subsequent agent training.

Adaptive damping coefficients of the PTO system with different agents

Figure 13. Adaptive damping coefficients of the PTO system with different agents.

Figure 14 presents the velocity field in the xx-direction and the motion state of the flap under both fixed and dynamic damping coefficients, and Figure 15 focuses on quantitatively analyzing the flap’s rotation and angular velocity. It can be observed that the adaptive change of the damping coefficient does not alter the flow field structure or the angular amplitude of the flap, while shift the equilibrium position of the flap, from −1.81∘-1.81^\circ to 4.31∘4.31^\circ. Further analysis indicates that the periodic characteristics of the wave height at the flap are consistent with the damping coefficient of the PTO system. When the wave crest passes, the damping coefficient increases to its peak value. Given that the energy density of the wave crest is high, the angular velocity is reduced slightly, leading to an overall improvement in the PTO system, as shown in Figure 15 (c). In addition, the wave energy density of the trough itself is lower than that of the peak, and some energy has already been absorbed during the crest phase. Maintaining a high damping coefficient during this phase would rapidly decrease angular velocity. Although reducing the damping coefficient can increase the flap’s angular velocity, the PTO system’s power output still decreases compared to a constant damping coefficient. Overall, as shown in Figure 16 (b), during a complete wave period, the average energy harvesting by the dynamic damping system is 27.3 J, compared to 24.4 J captured under the optimal constant damping coefficient, resulting in a 10.61% improvement in wave energy harvesting.

Free surfaces and wave-structure interactions in one wave period

Figure 14. Free surfaces and wave-structure interactions in one wave period. The fluid particles are colored by velocity magnitude on the x-axis.

Influence of the damping coefficient on the rotation of the flap, angular velocity of the flap, and instantaneous power capture

Figure 15. Influence of the damping coefficient on the (a) rotation of the flap, (b) angular velocity of the flap, and (c) instantaneous power capture.

Comparison of peak and trough energy in a complete cycle under the optimal constant damping coefficient and SAC strategy, (a) 2D and (b) 3D

Figure 16. Comparison of peak and trough energy in a complete cycle under the optimal constant damping coefficient and SAC strategy, (a) 2D and (b) 3D.

The energy harvesting efficiency of the OWSC can be quantified by CWR [31, 36]

CWR=PoutP0.(42)\mathrm{CWR} = \frac{P_{out}}{P_{0}}. \tag*{(42)}

Here, PoutP_{out} is the capture of instantaneous energy within a wave period and P0P_{0} is the mean incident power of unidirectional regular waves based on the

linear theory

Pout=1Tperiod∫0TperiodkdΩ2 dtP0=ρgH2Bω16k(1+2khsinh⁡(2kh)),(43)\begin{aligned} P_{\mathrm{out}} &= \frac{1}{T_{\mathrm{period}}}\int_{0}^{T_{\mathrm{period}}} k_d\Omega^2\,\mathrm{d}t \\ P_0 &= \frac{\rho g H^2 B \omega}{16k}\left(1+\frac{2kh}{\sinh(2kh)}\right), \tag*{(43)} \end{aligned}

with BB denoting the width of the flap. Currently, CWR under optimal constant damping coefficient is 13.12%, and 14.68% for the adaptive damping coefficient in 2D simulations, as shown in Table 3.

Fixed kdk_d (%)DRL (%)
2D Regular Wave13.1214.68
3D Regular Wave34.1541.86
2D Irregular Wave86.7992.38

Table 3. Comparison of constant and adaptive damping coefficients for different wave types

Effects of 3D simulations

Previous studies have shown that fixed flaps in 2D simulations simplify the diffraction waves, which are theoretically equal in size to the incident waves and opposite in direction. This results in standing waves forming on the windward side of the flap [67]. In 3D simulations, diffraction waves propagate in all directions, and antisymmetric shear waves along the flap can trigger near-resonance, enhancing the torque acting on the converter. Therefore, 3D simulations can more accurately capture the motion characteristics of the OWSC device in actual operation. In this section, we conduct experiments in 3D environment with the policy obtained from 2D training environment. Notably, Zhang et al. [34] have proved that in 3D simulations, kd=40 N⋅m⋅s/radk_d = 40\ \mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad} is the optimal constant damping coefficient.

As shown in Figure 17, the dynamic changes in the damping coefficient do not alter the structure of the flow field in the 3D simulations. Also as shown in Figure 18, the output of the damping coefficient in both 2D and 3D simulations is essentially consistent. This consistency indicates that the 2D assumption can accurately represent the coupling effect between the waves and the OWSC, and it also demonstrates the robustness of the trained policy network, which can be applied in real-world scenarios.

3D simulations of OWSC: free surfaces and flap motion in a wave period

Figure 17. 3D simulations of OWSC: Free surfaces and the flap motion in a wave period. left side is the optimal constant damping coefficient and right side is the damping coefficient controlled with SAC.

Comparison of the damping coefficient with the same policy under 2D and 3D simulations

Figure 18. Comparison of the damping coefficient with the same policy under 2D and 3D simulations.

Similar to the 2D simulation, Figure 19 shows that under adaptive damping control, the equilibrium position of the flap shifts to the left by 11.46∘11.46^\circ, approaching nearly vertical position to the base. Considering the fluid incompressibility, this shift increases the force perpendicular to the flap, as illustrated in Figure 19a. A control period is assumed to begin when the flap

(a) Vertical force on the flap

(b) Rotation of the flap

(c) Angular velocity of the flap

(d) Instantaneous power capture

Figure 19. Effects of different policies on (a) vertical force on the flap, (b) rotation of the flap, (c) angular velocity of the flap, and (d) instantaneous power capture.

rotates to the far left, with the wave crest reaching the flap. During the first quarter of the period, the damping coefficient continues to rise. Due to the increased thrust on the flap, the angular velocity remains almost unchanged compared to the constant damping coefficient, enhancing energy harvesting.

As the wave trough passes, the force on the flap decreases, and the reduction in the damping coefficient helps maintain the flap’s angular velocity. Since the energy of the wave crest is inherently higher than that of the trough, the overall energy harvesting improves due to the difference in energy levels, as shown in Figure 16 (b). Over a complete wave period, the average energy harvesting by the dynamic damping system is 77.88 J, compared to 63.55 J under the optimal constant damping coefficient, resulting in a 22.54% improvement. Also, in 3D simulation, CWR under optimal constant damping coefficient is 34.15% and 41.86% with the adaptive damping coefficient.

Optimization under irregular waves

In this section, we investigate optimization problems under irregular wave conditions, the most common scenarios encountered in practical engineering applications. Considering the strong nonlinearity of the induced motion of the OWSC under irregular waves, the number of observation points increases. The initial observation position of the wave height is set at x=3.5x = 3.5 m, with one point placed every 0.264 m for total 17 probes. The locations of observation points for wave speed are also increased accordingly, and the value of the observation vector increased to 74 at last. In addition, to verify that the policy network can resist the random characteristics of irregular waves, two sets of random seeds are used to characterize the wave phases in Eq. (23) during the training and testing stages while HpH_p and TpT_p remained

unchanged. The relation between the total energy conversion and linear damping coefficient for irregular waves is shown in Table 4. It is clear that with kd=60 N⋅m⋅s/radk_d = 60\ \mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad}, the energy harvesting factor is the highest.

kdk_d (N⋅m⋅s/rad\mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad})102030405060708090100
EtrainE_{\mathrm{train}} (J)195.21290.79340.34365.88378.75383.36382.02379.29377.94368.95
EtestE_{\mathrm{test}} (J)206.12300.77350.17378.48383.89390.49381.86376.41368.35363.64

Table 4. The variations of the total energy conversion in terms of damping coefficients.

From Figure 20 (a), it can be observed that, compared to regular waves, less than one-third of the wave heights of irregular waves exceed 0.2 m throughout the entire period, which is also the region where the wave energy is primarily concentrated. The dynamic response of the damping coefficient is related to the wave height. When the peak period occurs, the damped vibration also increases accordingly. This relationship can also be observed in the test section, indicating that the agent can accurately capture the wave characteristics under the specific spectrum.

Figure 20. The wave height curve in front of the flap and the corresponding damping coefficient under the (a) training wave (c) testing wave, and the difference of instantaneous energy harvesting ΔE=Et−E60\Delta E = E_t - E_{60} under the (a) training wave (c) testing wave.

Further combined with Figure 20 (b), we can see that compared with the constant damping coefficient, the difference in energy harvesting is mainly concentrated in the peak period, which is essentially consistent with the improvement of energy harvesting by regular waves. The near-simple harmonic damping motion will improve Capture energy in the crest section and reduce energy harvesting in the trough section. For secondary period waves, since the instantaneous energy they carry is small, the response of the damping coefficient will not cause significant changes in flap motion and energy harvesting. This part of the energy cannot be effectively improved. Therefore,

over the entire 40-second period, the wave energy harvesting increased by 24.67 J, an increase of 6.42%, compared to the energy harvesting under a constant damping coefficient. In addition, for the average incident energy of the irregular wave, Eq. (43) is modified as

P0=∑i=0NρgHi2Bωi16ki(1+2kihsinh⁡(2kih)).(44)P_0 = \sum_{i=0}^{N} \frac{\rho g H_i^2 B \omega_i}{16 k_i} \left(1 + \frac{2 k_i h}{\sinh(2 k_i h)}\right). \tag*{(44)}

Then, CWR under optimal constant damping coefficient is 86.79% and 92.38% for the adaptive damping coefficient.

Study of dual OWSC system

The dual OWSC system is illustrated in Figure 21. Previous research has indicated that for a dual OWSC system, the maximum total energy conversion is achieved when the spacing between the two OWSCs is seven-eighths of the wavelength [16]. In this section, we initially set the damping coefficients of both flaps to 50 N⋅m⋅s/rad50\ \mathrm{N}\cdot\mathrm{m}\cdot\mathrm{s}/\mathrm{rad} to investigate the impact of different spacings on total energy conversion. As shown in Table 5, when the spacing is 3.5 m, approximately three-quarters of the wavelength, the total energy conversion reaches its maximum.

The geometry of the dual OWSC system

Figure 21. The geometry of the dual OWSC system. The overall structure has stayed the same. Only the length of the plane where the base is located has been increased.

Δx\Delta x (m)2.02.252.52.753.03.253.53.754.0
EtE_t (J)449.01366.32422.91526.69608.52661.68675.47650.44541.84

Table 5. The influence of the spacing on the total energy conversion of dual OWSCs.

Subsequently, we examine the effect of different damping coefficient combinations on total energy harvesting at this optimal spacing, as depicted in

Figure 22. It can be observed that the damping coefficient of flap-2 has a linear relationship with the total energy conversion, where increasing the damping coefficient results in a gradual increase in energy harvesting. On the other hand, for flap-1, total energy conversion increases initially with the damping coefficient and then decreases. Notably, the peak value occurs around kd1=20 N⋅m⋅s/radk_{d1} = 20\ \mathrm{N} \cdot\mathrm{m} \cdot\mathrm{s} / \mathrm{rad}, which remains unaffected by changes in kd2k_{d2}. Therefore, in this section, the condition with kd1=20 N⋅m⋅s/radk_{d1} = 20\ \mathrm{N} \cdot\mathrm{m} \cdot\mathrm{s} / \mathrm{rad} and kd2=80 N⋅m⋅s/radk_{d2} = 80\ \mathrm{N} \cdot\mathrm{m} \cdot\mathrm{s} / \mathrm{rad} will be used as the baseline for RL training. In addition, considering that the dual OWSC system has strong nonlinear characteristics and flap-2 has low energy harvesting, in the training of this section, the damping coefficient of OWSC-2 is kept unchanged and only the damping coefficient of OWSC-1 is controlled. The training starts from the 24th second, when the coupling effect of the wave structure tends to be stable.

Three-dimensional surface plot showing total energy conversion as a function of damping coefficients

Figure 22. The variations of the total energy conversion in terms of the damping coefficients in dual OWSCs.

Subsequently, the effect of varying damping coefficient combinations on the total energy harvesting at the identified optimal spacing is investigated, as depicted in Figure 22. The analysis reveals a linear relationship between the damping coefficient of the OWSC-2 and the total energy conversion, with higher damping coefficients leading to a gradual increase in energy harvesting. Conversely, for the OWSC-1, the total energy conversion initially increases with the damping coefficient, reaching a peak before declining. Notably, the peak value occurs around kd1=20 N⋅m⋅s/radk_{d1} = 20\ \mathrm{N} \cdot\mathrm{m} \cdot\mathrm{s} / \mathrm{rad}, and this peak remains unaffected by variations in kd2k_{d2}. Therefore, for the subsequent RL training, the condition with kd1=20k_{d1} = 20 N · m · s / rad and kd2=80k_{d2} = 80 N · m · s / rad is established as the baseline. Given the strong nonlinear characteristics of the dual OWSC system and the observed lower wave energy harvesting by the OWSC-2, the damping coefficient of the OWSC-2 is held constant during training. This approach allows us to focus on optimizing the damping coefficient of the OWSC-1. The RL training commences at the 24th second, a point in time when the wave-structure interactions have stabilized, meaning the system has reached a steady state in terms of energy harvesting and conversion.

The training results are illustrated in Figure 23. The damping coefficient variation is consistent with the trend of wave height changes before the flap, demonstrating apparent periodicity that aligns with previous research findings. Further analysis based on Figure 24 reveals that, after 39 seconds, the system’s state is stabilized. An increase in the damping coefficient during the wave peak phase significantly reduces the angular velocity, resulting in only a limited increase in energy harvesting during the peak. Conversely, reducing the damping during the trough phase does not enhance the angular velocity, which remains lower than under constant damping conditions. This reduction in energy harvesting during the trough phase leads to a notable decrease in overall energy acquisition over the entire wave period, thereby failing to improve energy harvesting efficiency. During the 60-second test period, the energy harvesting for the OWSC-1 using the adaptive damping coefficient was 1560.37 J, compared to 1626.52 J with constant damping, representing a reduction of 4.07%.

The wave height in front of the first flap and the corresponding damping coefficient

Figure 23. The wave height in front of the first flap (x=7.6 mx = 7.6\ \mathrm{m}) and the corresponding damping coefficient.

Angular velocity of the flap and instantaneous power capture

Figure 24. The influence of the damping coefficient on the (a) angular velocity of the first flap, and (b) instantaneous power capture.

Additionally, for the OWSC-2, where the damping coefficient remained unchanged, the energy harvesting efficiency dropped significantly from 362.94 J to 197.91 J, a decrease of 45.47%, as shown in Figure 25. This indicates

The influence of the damping coefficient on the instantaneous power capture of the OWSC-2

Figure 25. The influence of the damping coefficient on the instantaneous power capture of the OWSC-2.

that after the wave passes through the OWSC-1 with adaptive damping, the energy loss is more significant than with constant damping. Combined with Figure 26, it is evident that under constant damping, significant harmonics are generated between the two OWSCs, which is beneficial for enhancing energy harvesting.

The spectrum diagram at $x = 8.2\ \mathrm{m}$

Figure 26. The spectrum diagram at x=8.2 mx = 8.2\ \mathrm{m}.

Therefore, in the dual OWSC system, the nonlinear characteristics are pronounced, and single damping control is insufficient to improve overall wave energy acquisition efficiency. Moreover, the 2D simulations constrain the design and optimization of the OWSC layout, necessitating further analysis and discussion in subsequent work.

Conclusion

This paper establishes a framework coupling a CFD environment based on an open-source SPH-based library with DRL, aimed at optimizing the adaptive damping coefficient of the PTO system in OWSCs for wave energy conversion. Initially, the wave-making model and the numerical model of WSI were validated. Subsequently, the performance of various RL algorithms for the optimization process was investigated. The results indicated that SAC considers policy entropy, balanced exploration, and exploitation well and provides effective policies to enhance wave energy conversion.

For regular waves, the strategy primarily utilizes the difference in energy density between wave crests and troughs. Increasing energy harvesting during the crest phase and reducing it during the trough phase achieves a positive net energy harvesting over each wave period. The policy trained in 2D simulations can be effectively applied in 3D simulations. Although the 2D simulations simplify wave diffraction and result in a slight decrease in calculated CWR, they accurately capture the coupling characteristics between waves and OWSCs, allowing DRL to learn practical policies that are robust and transferable. This provides a theoretical foundation for validation in experiments.

The DRL algorithm could still learn effective energy conversion optimization policies for irregular waves, primarily targeting regular waves with high energy density in the main period. The optimization principle is similar to that for regular waves, with limited enhancement in energy harvesting from the dynamic damping response for waves in the secondary period due to their lower energy density.

Finally, this paper explores the optimization of wave energy absorption in a dual OWSC system. The interaction between incident waves and OWSCs in the dual system generates harmonics with strong nonlinearity. Optimization focused on the primary OWSC showed that using similar optimization policies can not enhance energy harvesting, as it significantly reduces the wave energy density between the OWSCs, leading to a substantial decrease in energy absorption by the secondary OWSC and an overall reduction in the system’s energy harvesting. Therefore, considering the dynamic response of a single OWSC’s damping coefficient is insufficient to optimize the energy conversion in complex dual OWSC systems.

Future work will introduce multi-agent reinforcement learning to directly learn corresponding energy optimization strategies for various OWSC layouts in 3D simulations.

CRediT authorship contribution statement

M. Ye: Validation, Software, Methodology, Investigation, Formal analysis, Writing - original draft, Writing - review & editing, Visualization. C. Zhang: Software, Writing – review & editing, Investigation. Y.R. Ren: Resources, Formal analysis, Conceptualization. Z.Y. Liu: Methodology, Data curation. O.J. Haidn: Validation, Supervision, Investigation. X.Y. Hu: Writing – review & editing, Supervision, Methodology, Investigation, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The corresponding code of this work is available on GitHub at https://github.com/Xiangyu-Hu/SPHinXsys. The corresponding data of this work will be made available on reasonable request.

Acknowledgments

M. Ye was supported by China Scholarship Council (No.202006120018) when he conducted this work.

References

References

  1. [1]E. Callaway, To catch a wave: ocean wave energy is trying to break into the renewable-energy market, but many challenges remain, Nature 450 (7167) (2007) 156–160, https://doi.org/10.1038/450156a.
  2. [2]K. Budal, J. Falnes, Wave power conversion by point absorbers: A norwegian project, International Journal of Ambient Energy 3 (2) (1982) 59–67, https://doi.org/10.1080/01430750.1982.9675829.
  3. [3]D. Evans, Maximum wave-power absorption under motion constraints, Applied Ocean Research 3 (4) (1981) 200–203, https://doi.org/10. 1016/0141-1187(81)90063-8.
  4. [4]M. Folley, T. Whittaker, Analysis of the nearshore wave energy resource, Renewable energy 34 (7) (2009) 1709–1715, https://doi.org/10.1016/j.renene.2009.01.003.
  5. [5]M. Folley, B. Elsaesser, T. Whittaker, Analysis of the wave energy resource at the european marine energy centre, in: Coasts, marine structures and breakwaters: Adapting to change: Proceedings of the 9th international conference organised by the Institution of Civil Engineers and held in Edinburgh on 16 to 18 September 2009, Thomas Telford Ltd, 2010, pp. 660–669, https://doi.org/10.1680/cmsb.41301.0058.
  6. [6]I. López, R. Carballo, G. Iglesias, Site-specific wave energy conversion performance of an oscillating water column device, Energy conversion and management 195 (2019) 457–465, https://doi.org/10.1016/j.enconman.2019.05.030.
  7. [7]J. P. Kofoed, P. Frigaard, E. Friis-Madsen, H. C. Sørensen, Prototype testing of the wave energy converter wave dragon, Renewable energy 31 (2) (2006) 181–189, https://doi.org/10.1016/j.renene.2005.09.005.
  8. [8]A. Ilyas, S. A. Kashif, M. A. Saqib, M. M. Asad, Wave electrical energy systems: Implementation, challenges and environmental issues, Re- newable and Sustainable Energy Reviews 40 (2014) 260–268, https://doi.org/10.1016/j.rser.2014.07.085.
  9. [9]A. Day, A. Babarit, A. Fontaine, Y.-P. He, M. Kraskowski, M. Murai, I. Penesis, F. Salvatore, H.-K. Shin, Hydrodynamic modelling of marine renewable energy devices: A state of the art review, Ocean Engineering 108 (2015) 46–69, https://doi.org/10.1016/j.oceaneng.2015.05.036.
  10. [10]T. Whittaker, M. Folley, Nearshore oscillating wave surge converters and the development of oyster, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 370 (1959) (2012) 345–364, https://doi.org/10.1098/rsta.2011.0152.
  11. [11]M. Folley, T. Whittaker, J. Van’t Hoff, The design of small seabed-mounted bottom-hinged wave energy converters, in: Proceedings of the 7th European wave and tidal energy conference, Vol. 455, Citeseer, 2007, p. 312.
  12. [12]Y. Cheng, C. Ji, G. Zhai, Fully nonlinear analysis incorporating viscous effects for hydrodynamics of an oscillating wave surge converter with nonlinear power take-off system, Energy 179 (2019) 1067–1081, https://doi.org/10.1016/j.energy.2019.04.189.
  13. [13]T. Whittaker, D. Collier, M. Folley, M. Osterried, A. Henry, M. Crowley, The development of oyster—a shallow water surging wave energy converter, in: Proceedings of the 7th European wave and tidal energy conference, 2007, pp. 11–14.
  14. [14]E. Renzi, K. Doherty, A. Henry, F. Dias, How does oyster work? the simple interpretation of oyster mathematics, European Journal of Mechanics-B/Fluids 47 (2014) 124–131, https://doi.org/10.1016/j.euromechflu.2014.03.007.
  15. [15]A. Henry, P. Schmitt, T. Whittaker, A. Rafee, F. Dias, The characteristics of wave impacts on an oscillating wave surge converter, in: ISOPE International Ocean and Polar Engineering Conference, ISOPE, 2013, pp. ISOPE–I.
  16. [16]Y.-C. Chow, Y.-C. Chang, C.-C. Lin, J.-H. Chen, S.-Y. Tzang, Experimental investigations on wave energy capture of two bottom-hinged-flap wecs operating in tandem, Ocean Engineering 164 (2018) 322–331, https://doi.org/10.1016/j.oceaneng.2018.06.010.
  17. [17]M. Brito, R. M. Ferreira, L. Teixeira, M. G. Neves, R. B. Canelas, Experimental investigation on the power capture of an oscillating wave surge converter in unidirectional waves, Renewable Energy 151 (2020) 975–992, https://doi.org/10.1016/j.renene.2019.11.094.
  18. [18]Y. Cheng, G. Li, C. Ji, T. Fan, G. Zhai, Fully nonlinear investigations on performance of an owsc (oscillating wave surge converter) in 3d (three-dimensional) open water, Energy 210 (2020) 118526, https://doi.org/10.1016/j.energy.2020.118526.
  19. [19]E. Renzi, F. Dias, Resonant behaviour of an oscillating wave energy converter in a channel, Journal of Fluid Mechanics 701 (2012) 482–510, https://doi.org/10.1017/jfm.2012.194.
  20. [20]P. Schmitt, H. Asmuth, B. Elsäßer, Optimising power take-off of an oscillating wave surge converter using high fidelity numerical simulations, International journal of marine energy 16 (2016) 196–208, https://doi.org/10.1016/j.ijome.2016.07.006.
  21. [21]Y. Wei, A. Rafiee, A. Henry, F. Dias, Wave interaction with an oscillating wave surge converter, part i: Viscous effects, Ocean Engineering 104 (2015) 185–203, https://doi.org/10.1016/j.oceaneng.2015.05.002.
  22. [22]Y. Wei, T. Abadie, A. Henry, F. Dias, Wave interaction with an oscillating wave surge converter. part ii: Slamming, Ocean Engineering 113 (2016) 319–334, https://doi.org/10.1016/j.oceaneng.2015.12.041.
  23. [23]X. Jiang, S. Day, D. Clelland, Hydrodynamic responses and power efficiency analyses of an oscillating wave surge converter under different simulated pto strategies, Ocean Engineering 170 (2018) 286–297, https://doi.org/10.1016/j.oceaneng.2018.10.050.
  24. [24]P. Schmitt, B. Elsaesser, On the use of openfoam to model oscillating wave surge converters, Ocean Engineering 108 (2015) 98–104, https://doi.org/10.1016/j.oceaneng.2015.07.055.
  25. [25]H. R. Mottahedi, M. Anbarsooz, M. Passandideh-Fard, Application of a fictitious domain method in numerical simulation of an oscillating wave surge converter, Renewable energy 121 (2018) 133–145, https://doi.org/10.1016/j.renene.2018.01.021.
  26. [26]M. Luo, C. Koh, Shared-memory parallelization of consistent particle method for violent wave impact problems, Applied Ocean Research 69 (2017) 87–99, https://doi.org/10.1016/j.apor.2017.09.013.
  27. [27]C. Zhang, X. Y. Hu, N. A. Adams, A generalized transport-velocity formulation for smoothed particle hydrodynamics, Journal of Compu-tational Physics 337 (2017) 216–232, https://doi.org/10.1016/j.jcp.2017.02.016.
  28. [28]A. Khayyer, H. Gotoh, Y. Shimizu, K. Gotoh, H. Falahaty, S. Shao, Development of a projection-based sph method for numerical wave flume with porous media of variable porosity, Coastal Engineering 140 (2018) 1–22, https://doi.org/10.1016/j.coastaleng.2018.05.003.
  29. [29]A. Rafiee, B. Elsaesser, F. Dias, Numerical simulation of wave in-teraction with an oscillating wave surge converter, in: International Conference on Offshore Mechanics and Arctic Engineering, Vol. 55393, American Society of Mechanical Engineers, 2013, p. V005T06A013, https://doi.org/10.1115/OMAE2013–10195.
  30. [30]M. Brito, R. Canelas, R. Ferreira, O. García-Feal, J. Domínguez, A. Crespo, M. Neves, Coupling between dualsphysics and chrono-engine: towards large scale hpc multiphysics simulations, in: 11th International SPHERIC Workshop, Munich, Germany, 2016.
  31. [31]M. Brito, R. Canelas, O. García-Feal, J. Domínguez, A. Crespo, R. M. Ferreira, M. G. Neves, L. Teixeira, A numerical tool for modelling os-cillating wave surge converter with nonlinear mechanical constraints, Renewable Energy 146 (2020) 2024–2043, https://doi.org/10.1016/j.renene.2019.08.034.
  32. [32]Z. Wei, B. L. Edge, R. A. Dalrymple, A. Hérault, Modeling of wave energy converters by gpusph and project chrono, Ocean Engineering 183 (2019) 332–349, https://doi.org/10.1016/j.oceaneng.2019.04.029.
  33. [33]Z. Liu, Y. Wang, X. Hua, Numerical studies and proposal of design equations on cylindrical oscillating wave surge converters under regular waves using sph, Energy conversion and management 203 (2020) 112242, https://doi.org/10.1016/j.enconman.2019.112242.
  34. [34]C. Zhang, Y. Wei, F. Dias, X. Hu, An efficient fully lagrangian solver for modeling wave interaction with oscillating wave surge converter, Ocean Engineering 236 (2021) 109540, https://doi.org/10.1016/j.oceaneng.2021.109540.
  35. [35]R. P. Gomes, M. F. Lopes, J. C. Henriques, L. M. Gato, A. F. d. O. Falcão, The dynamics and power extraction of bottom-hinged plate wave energy converters in regular and irregular waves, Ocean Engineering 96 (2015) 86–99, https://doi.org/10.1016/j.oceaneng.2014.12.024.
  36. [36]K. Senol, M. Raessi, Enhancing power extraction in bottom-hinged flap-type wave energy converters through advanced power take-off techniques, Ocean Engineering 182 (2019) 248–258, https://doi.org/10.1016/j.oceaneng.2019.04.067.
  37. [37]Y. Liu, N. Mizutani, Y.-H. Cho, T. Nakamura, Performance enhancement of a bottom-hinged oscillating wave surge converter via resonant adjustment, Renewable Energy 201 (2022) 624–635, https://doi.org/10.1016/j.renene.2022.10.130.
  38. [38]Y. Liu, N. Mizutani, Y.-H. Cho, T. Nakamura, Nonlinear hydrodynamic analysis and optimization of oscillating wave surge converters under irregular waves, Ocean Engineering 250 (2022) 110888, https://doi.org/10.1016/j.oceaneng.2022.110888.
  39. [39]J. V. Ringwood, G. Bacelli, F. Fusco, Energy-maximizing control of wave-energy converters: The development of control system technology to optimize their operation, IEEE control systems magazine 34 (5) (2014) 30–55, https://doi.org/10.1109/MCS.2014.2333253.
  40. [40]Y. Jia, K. Meng, L. Dong, T. Liu, C. Sun, Z. Y. Dong, Economic model predictive control of a point absorber wave energy converter, IEEE transactions on sustainable energy 12 (1) (2020) 578–586, https://doi.org/10.1109/TSTE.2020.3012755.
  41. [41]M. Shadman, G. O. G. Avalos, S. F. Estefen, On the power performance of a wave energy converter with a direct mechanical drive power take-off system controlled by latching, Renewable Energy 169 (2021) 157–177, https://doi.org/10.1016/j.renene.2021.01.004.
  42. [42]H. Liang, D. Qiao, X. Wang, G. Zhi, J. Yan, D. Ning, J. Ou, Energy capture optimization of heave oscillating buoy wave energy con- verter based on model predictive control, Ocean Engineering 268 (2023) 113402, https://doi.org/10.1016/j.oceaneng.2022.113402.
  43. [43]D. Fan, G. Jodin, T. Consi, L. Bonfiglio, Y. Ma, L. Keyes, G. E. Karniadakis, M. S. Triantafyllou, A robotic intelligent towing tank for learning complex fluid-structure dynamics, Science Robotics 4 (36) (2019) eaay5063, https://doi.org/10.1126/scirobotics.aay5063.
  44. [44]D. Fan, L. Yang, Z. Wang, M. S. Triantafyllou, G. E. Karniadakis, Reinforcement learning for bluff body active flow control in experiments and simulations, Proceedings of the National Academy of Sciences 117 (42) (2020) 26091–26098, https://doi.org/10.1073/pnas.2004939117.
  45. [45]J. Rabault, F. Ren, W. Zhang, H. Tang, H. Xu, Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization, Journal of Hydrodynamics 32 (2020) 234–246, https://doi.org/10.1007/s42241-020-0028-y.
  46. [46]P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, E. Hachem, A review on deep reinforcement learning for fluid mechanics, Computers & Fluids 225 (2021) 104973, https://doi.org/10.1016/j.compfluid.2021.104973.
  47. [47]K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are universal approximators, Neural networks 2 (5) (1989) 359–366, https://doi.org/10.1016/0893-6080(89)90020-8.
  48. [48]J. Rabault, M. Kuchta, A. Jensen, U. Réglade, N. Cerardi, Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control, Journal of fluid mechanics 865 (2019) 281–302, https://doi.org/10.1017/jfm.2019.62.
  49. [49]E. Anderlini, S. Husain, G. G. Parker, M. Abusara, G. Thomas, Towards real-time reinforcement learning control of a wave energy converter, Journal of Marine Science and Engineering 8 (11) (2020) 845, https://doi.org/10.3390/jmse8110845.
  50. [50]S. Zou, X. Zhou, I. Khan, W. W. Weaver, S. Rahman, Optimization of the electricity generation of a wave energy converter using deep reinforcement learning, Ocean Engineering 244 (2022) 110363, https://doi.org/10.1016/j.oceaneng.2021.110363.
  51. [51]H. Liang, H. Qin, H. Su, Z. Wen, L. Mu, Environmental-sensing and adaptive optimization of wave energy converter based on deep reinforcement learning and computational fluid dynamics, Energy 297 (2024) 131254, https://doi.org/10.1016/j.energy.2024.131254.
  52. [52]C. Zhang, M. Rezavand, Y. Zhu, Y. Yu, D. Wu, W. Zhang, J. Wang, X. Hu, Sphinxsys: An open-source multi-physics and multi-resolution library based on smoothed particle hydrodynamics, Computer Physics Communications 267 (2021) 108066, https://doi.org/10.1016/j.cpc.2021.108066.
  53. [53]J. Weng, H. Chen, D. Yan, K. You, A. Duburcq, M. Zhang, Y. Su, H. Su, J. Zhu, Tianshou: A highly modularized deep reinforcement learning library, Journal of Machine Learning Research 23 (267) (2022) 1–6, http://jmlr.org/papers/v23/21-1127.html.arxiv.org/abs/2107.14171
  54. [54]C. Zhang, X. Hu, N. A. Adams, A weakly compressible sph method based on a low-dissipation riemann solver, Journal of Computational Physics 335 (2017) 605–620, https://doi.org/10.1016/j.jcp.2017.01.027.
  55. [55]Y. Ren, P. Lin, C. Zhang, X. Hu, An efficient correction method in riemann sph for the simulation of general free surface flows, Computer Methods in Applied Mechanics and Engineering 417 (2023) 116460, https://doi.org/10.1016/j.cma.2023.116460.
  56. [56]J. P. Morris, P. J. Fox, Y. Zhu, Modeling low reynolds number incompressible flows using sph, Journal of computational physics 136 (1) (1997) 214–226, https://doi.org/10.1006/jcph.1997.5776.
  57. [57]H. Wendland, Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree, Advances in computational Mathematics 4 (1995) 389–396, https://doi.org/10.1007/BF02123482.
  58. [58]T. Kanehira, M. L. McAllister, S. Draycott, T. Nakashima, D. M. Ingram, T. S. van den Bremer, H. Mutsuda, The effects of smoothing length on the onset of wave breaking in smoothed particle hydrodynamics (sph) simulations of highly directionally spread waves, Computational Particle Mechanics 9 (5) (2022) 1031–1047, https://doi.org/10.1007/s40571-022-00463-z.
  59. [59]H. Wen, B. Ren, X. Yu, An improved sph model for turbulent hydrody- namics of a 2d oscillating water chamber, Ocean Engineering 150 (2018) 152–166, https://doi.org/10.1016/j.oceaneng.2017.12.047.
  60. [60]J. J. Monaghan, Smoothed particle hydrodynamics, In: Annual review of astronomy and astrophysics. Vol. 30 (A93-25826 09-90), p. 543-574. 30 (1992) 543–574, https://doi.org/10.1146/annurev.aa.30.090192.002551.
  61. [61]C. Zhang, M. Rezavand, X. Hu, Dual-criteria time stepping for weakly compressible smoothed particle hydrodynamics, Journal of Computational Physics 404 (2020) 109135, https://doi.org/10.1016/j.jcp.2019.109135.
  62. [62]S. Adami, X. Y. Hu, N. A. Adams, A generalized wall boundary condition for smoothed particle hydrodynamics, Journal of Computational Physics 231 (21) (2012) 7057–7075, https://doi.org/10.1016/j.jcp.2012.05.005.
  63. [63]J.-F. Lee, J.-R. Kuo, C.-P. Lee, Transient wavemaker theory, Journal of hydraulic research 27 (5) (1989) 651–663, https://doi.org/10.1080/00221688909499116.
  64. [64]O. S. Madsen, On the generation of long waves, Journal of Geophysical Research 76 (36) (1971) 8672–8683, https://doi.org/10.1029/JC076i036p08672.
  65. [65]E. Renzi, F. Dias, Application of a moving particle semi-implicit numerical wave flume (mps-nwf) to model design waves, Coastal Engineering 172 (2022) 104066, https://doi.org/10.1016/j.coastaleng.2021.104066.
  66. [66]Y. Goda, Random seas and design of maritime structures, Vol. 33, World Scientific Publishing Company, 2010, https://doi.org/10.1142/7425.
  67. [67]E. Renzi, A. Abdolali, G. Bellotti, F. Dias, Wave-power absorption from a finite array of oscillating wave surge converters, Renewable Energy 63 (2014) 55–68, https://doi.org/10.1016/j.renene.2013.08.046.
  68. [68]T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International conference on machine learning, PMLR, 2018, pp. 1861–1870, https://proceedings.mlr.press/v80/haarnoja18b.html.arxiv.org/abs/1801.01290
  69. [69]W. Jakob, J. Rhinelander, D. Moldovan, pybind11 – seamless operability between c++11 and python, https://github.com/pybind/pybind11 (2017).DOI
  70. [70]G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, W. Zaremba, Openai gym, arXiv preprint arXiv:1606.01540https://doi.org/10.48550/arXiv.1606.01540 (2016).DOI
  71. [71]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347https://doi.org/10.48550/arXiv.1707.06347 (2017).arxiv.org/abs/1707.06347
  72. [72]S. Fujimoto, H. Hoof, D. Meger, Addressing function approximation error in actor-critic methods, in: International conference on machine learning, PMLR, 2018, pp. 1587–1596, https://proceedings.mlr.press/v80/fujimoto18a.html.
  73. [73]J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel, High-dimensional continuous control using generalized advantage estimation, arXiv preprint arXiv:1506.02438https://doi.org/10.48550/arXiv.1506.02438 (2015).DOI
  74. [74]D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980https://doi.org/10.48550/arXiv.1412.6980 (2014).DOI

Paper details

vlnc:W_8fuhd78m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents