betaFeedback

WorksComputer Science

Effects of boundary conditions in fully convolutional networks for learning spatio-temporal dynamics

Abstract

Accurate modeling of boundary conditions is crucial in computational physics. The ever increasing use of neural networks as surrogates for physics-related problems calls for an improved understanding of boundary condition treatment, and its influence on the network accuracy. In this paper, several strategies to impose boundary conditions (namely padding, improved spatial context, and explicit encoding of physical boundaries) are investigated in the context of fully convolutional networks applied to recurrent tasks. These strategies are evaluated on two spatio-temporal evolving problems modeled by partial differential equations: the 2D propagation of acoustic waves (hyperbolic PDE) and the heat equation (parabolic PDE). Results reveal a high sensitivity of both accuracy and stability on the boundary implementation in such recurrent tasks. It is then demonstrated that the choice of the optimal padding strategy is directly linked to the data semantics. Furthermore, the inclusion of additional input spatial context or explicit physics-based rules allows a better handling of boundaries in particular for large number of recurrences, resulting in more robust and stable neural networks, while facilitating the design and versatility of such networks.

Introduction

Recent advances in deep learning have shown an increased use of neural networks to create surrogate models for physics-related problems. In particular, Convolutional Neural Networks (CNN) have been employed in a wide variety of applications, leveraging their efficient parameter sharing property and their ability to capture long-range spatial correlations. However, most of the existing works limit themselves to one particular problem setup, keeping the same boundary conditions (BCs) throughout the entire training data [11], thus being unable to be generalized to other types of boundary conditions. Ideally, a flexible neural network framework should be able to work with several types of boundary conditions, without the need of retraining the network for each new problem setup. It is thus crucial to understand how boundary conditions are treated by data-driven CNNs in order to improve their generalization capabilities. The general theme of border effect for CNNs has been broadly studied in the image processing community [13, 6]. Still today, such border effects can have a strong influence in state-of-the-art architectures employed in image segmentation [2]. The usual zero-padding strategy leads to border pixel artifacts and blind spots where the network accuracy drops. Solutions have been proposed to treat borders through separate filters [7] for the edges, corners and inner pixels or to consider the padded pixels as missing information, through the use of Partial Convolution strategies [12]. Other works demonstrate how CNNs implicitly learn spatial position [9, 8], using the padded pixels to serve as anchors, i.e., as a reference for filter activation in border regions. The use of circular convolution [19] eliminates such border effects, but can only be employed on “panoramic” datasets.

Yet, the previous works have been devised for image segmentation/classification tasks and it is still unclear whether such studies can be directly transposed for regression and recurrent tasks, as usually encountered when modeling physics. In such contexts, a small error in the boundary prediction may lead to large errors elsewhere in the computational domain. A typical example of such a phenomenon is the development of non-reflecting boundary conditions in computational acoustics [17], which avoids the unphysical reflection of waves back into the computational domain. If left untreated, undesired reflections can pollute the calculated solution. To the author’s knowledge, there is a lack of clear results regarding what is the optimal strategy for the treatment of such boundary conditions when employing CNNs for spatio-temporal evolving problems, for which BC errors can propagate and contaminate the whole computational domain. Indeed, only few works specifically focus on the boundary treatment problem. Some employ explicit rules in relatively simple cases, such as in periodic domains, as in the case of turbulence modeling [16]. For more complex boundary treatments, previous works are found in the context of Physical-Informed Neural Networks (PINN) [18] employed in combination with CNN [4], where hard constraints are imposed through the padding mechanism. However, only simple Dirichlet or Neumann conditions on static problems are considered, leaving dynamical conditions out of the study. In the case of spatio-temporal modeling, CNNs and Recurrent Neural Networks (RNN) architectures have been employed indistinctly. Mathieu et al. [15] designed a Multi-Scale CNN for video prediction, which was later employed in physics-related applications [11, 1]. Fotiadis et al. [3] compared recurrent and convolutional approaches in physics-based applications and found that CNNs can be successfully employed as spatio-temporal predictors, with greater accuracy than RNNs and lesser training costs. In all previous cases, the treatment of boundary conditions was not explicitly studied, leaving the effect of BC treatments on such spatio-temporal evolving systems as an open question.

In this paper, the effects of several boundary condition treatments are characterized when modeling a space-time evolving problem using Convolutional Neural Networks. Three strategies are employed for handling the boundary conditions: (i) an implicit treatment through padding only, (ii) adding some extra spatial context to the network input and finally, (iii) explicitly encoding the boundary condition rules into the network output. These strategies are tested on a series of datasets with varying boundary conditions, modeling the two dimensional wave and heat equations.

The main contributions are the following:

  • (i) For problems with simple Neumann boundary conditions, padding allows CNNs to efficiently model the physics; yet, mimicking the actual semantics of the dataset makes it only possible for problems with fixed BCs;

  • (ii) The addition of an extra spatial context makes the neural network less sensitive to the choice of padding;

  • (iii) Explicit coding of neural networks gives the best accuracy in the more challenging test cases where dynamical effects are needed to model BCs, allowing more robust predictions by first enforcing physics.

Method

This section describes the methodology to predict the spatio-temporal dynamics of physics-related quantities using a convolutional neural network. The trained network follows a typical auto-regressive strategy [5] to produce time series of high-dimensional state vectors. The focus is put on the treatment of boundary conditions in the context of convolutional networks, in order to reproduce the desired physics. Several algorithms for the treatment of such BCs are presented and later evaluated in § 4.

Learning an Auto-Regressive Model

Dynamical systems can be modeled through a discrete time-invariant model ff acting on a delayed state vector Xt={st,st−1,…,st−k−1}X^t = \{s^t, s^{t-1}, \ldots, s^{t-k-1}\} composed of kk discrete temporal states sis^i which may lie on a high dimensional space. Formally, the discretized time-dynamics read:

Yt+1=f(Xt),(1)Y^{t+1} = f(X^t), \tag*{(1)}

where Yt+1={st+1}Y^{t+1} = \{s^{t+1}\} corresponds to the next state in the time series.

In order to generate an approximate model for ff, a neural network f^θ\hat{f}_{\theta}, parametrized by its weights and biases θ\theta, is trained on a dataset composed of input-target tuples {Xt,Yt+1}i\{X^{t},Y^{t+1}\}_{i} through a supervised optimisation problem, based on an error metric L\mathcal{L}, such that

f^θ=arg⁡min⁡fθ∑iL[fθ(Xt),Yt+1]i.(2)\hat{f}_{\theta}=\underset{f_{\theta}}{\arg\min}\sum_{i}\mathcal{L}\left[f_{\theta}(X^{t}),Y^{t+1}\right]_{i}. \tag*{(2)}

Once an approximate solution is obtained, any time state STS^{T} can be reached by employing an auto-regressive iterative strategy on the learned model, namely:

YT=f^θ∘f^θ∘…∘f^θ⏟T times(X0)(3)Y^{T}=\underbrace{\hat{f}_{\theta}\circ\hat{f}_{\theta}\circ\ldots\circ\hat{f}_{\theta}}_{T\ \text{times}}(X^{0}) \tag*{(3)}

where ∘\circ is the function composition operator.

Neural Network Convolutional Architecture

The auto-regressive strategy can be employed to create surrogate models for physics-based quantities. In traditional fluid solvers, it is common to discretize both the time and space dimensions of physical quantities, such as pressure or velocity fields. This results in high dimensional state where the modeled equations are solved for each degree of freedom. To reduce the computational costs associated with training a neural network surrogate on such high-dimensional states, a convolutional neural network is employed due to its weight-sharing capabilities. In order to efficiently treat the intrinsic multi-scale features of fluid flows, a Multi-Scale fully convolutional neural network [15, 11] is employed, as shown in Figure 1. The input state is composed of four consecutive vectors Xt={st−3,st−2,st−1,st}X^{t}=\{s^{t-3},s^{t-2},s^{t-1},s^{t}\}, in order to provide the network with additional temporal context.

Figure 1. Multi-Scale fully convolutional neural network with 4 consecutive input states of size N×NN \times N and 3 resolution banks (N/4N/4, N/2N/2 and NN). Grey arrows represent 2D convolutions and width of boxes the number of features.

The usage of a pure convolutional approach for the temporal regression problem instead of a Recurent Architecture (RNN, LSTMs etc) is justified because of the direct prediction of full states, which only require a short temporal span for their accurate time-stepping (in traditional PDE solvers, the discretization of the time-derivatives). A comparison of the fully convolutional approach with LSTM approaches performed in [3] confirms this observation, as the convolutional architecture achieves better accuracies on such types of problems.

Boundary Condition Treatment

In traditional solvers, the proper modeling of boundary conditions is key to accurate numerical resolution of the partial differential equations. Therefore, understanding the boundary condition treatment is fundamental if convolution neural networks are to be employed as surrogates for physics-based models. Boundary conditions are intrinsically linked to the concept of padding in fully convolutional networks: additional information must be created at the borders before each convolution in order to keep the same image input resolution at the output. However, the value of the additional pixel information is not known a priori, and several padding strategies are available to encode this information. Zero padding, where the additional information is filled with zero values; replication padding, where the values at border pixels are replicated multiple times into the padding area; reflection padding, where an axial symmetry is performed along boundary edges; and circular padding (or periodic) which wraps values from the opposite boundary in the same spatial direction.

While padding is the most straightforward strategy to impose BCs in CNNs, it is unclear how the padding type affects the predictions, neither how an optimal choice is connected to the physical BC type (Dirichlet, Neumann, etc.). Moreover, padding is fixed for the entire database considered, which lacks of versatility when targeting physical systems with multiple possible BCs. Consequently, the objective of this work is to study if an optimal boundary treatment strategy can be found when employing CNNs for physics-based regression. Three types of strategies are considered:

Implicit: It consists in applying exclusively padding to the input and successive feature maps. This is the most common approach in convolutional networks and forces the network to implicitly learn the boundary physics. It is up to the network designer to chose an adequate padding strategy which usually results in a long trial-and-error process until finding an optimal solution. It constitutes the baseline method of this work (Figure 2a) and the four padding strategies presented above are investigated (zeros, replication, reflection and circular).

Figure 2. Three boundary condition strategies: (a) implicit treatment using only padding, (b) adding an additional spatial context to the network input or (c) explicitly encoding boundary condition after the neural network prediction.

Spatial context: The second strategy consists in concatenating an additional channel to the network input, which is consisting in a Boolean mask indicating the position of border pixels. This is formalized as follows:

I(x)={1,if x∈∂D0,otherwise(4)\mathbb{I}(x) = \begin{cases} 1, & \text{if } x \in\partial\mathcal{D} \\ 0, & \text{otherwise} \end{cases} \tag*{(4)}

where D\mathcal{D} represents the domain of interest, ∂D\partial\mathcal{D} its boundaries and xx are the spatial coordinate. The motivation for such a strategy is to provide the network with an increased spatial context. Figure 2b depicts such a strategy. Note that padding is still employed in order to maintain the size of feature maps. This strategy is inspired by other works such as Liu et al. [14], where giving explicit spatial information to CNNs is shown to crucially improve their generalization capability. Yet, the correlation between the spatial extended context, the padding type, and the actual physical BC is still unclear, thus being studied here.

Explicit encoding: Finally, the third strategy (Figure 2c) consists in explicitly encoding the boundary condition. In practice, the boundary pixel values are imposed after the network output [4], before stepping into the optimization step during the training phase. The way the value is imposed depends on the mathematical modeling of the boundary (Dirichlet, Neumann condition, etc.).

All three strategies are compared in this work, by combining them with the four types of padding previously mentioned.

Loss Function

The loss function for training the aforementioned Multi-Scale network is defined as:

L=1N∑k=1N{λL2L2+λGDLLGDL}(5)\mathcal{L} = \frac{1}{N}\sum_{k=1}^{N}\left\{\lambda_{L2}\mathcal{L}_{2}+\lambda_{GDL}\mathcal{L}_{GDL}\right\} \tag*{(5)}

where L2=∥Yt+1−Y^t+1∥22\mathcal{L}_2 = \lVert Y^{t+1} - \hat{Y}^{t+1} \rVert_2^2 and LGDL=∥∂xYt+1−∂xY^t+1∥22+∥∂yYt+1−∂yY^t+1∥22\mathcal{L}_{GDL} = \lVert\partial_x Y^{t+1} - \partial_x \hat{Y}^{t+1} \rVert_2^2 + \lVert\partial_y Y^{t+1} - \partial_y \hat{Y}^{t+1} \rVert_2^2, where a classical mean square metric is employed, both for the state vector and its spatial derivatives, denoted as Gradient Difference Loss (GDL) as in [15]. This loss drives the optimization towards achieving sharper predictions, and compensates for the smoothing of the predicted signal in the long term prediction of spatio-temporal series.

Note that the training focuses only on the next time-step prediction. Therefore, the auto-regressive prediction of a long time series of state vectors is a generalization problem.

Applications: Time-evolving PDEs

The studied modeled is applied to create data-driven surrogates of spatio-temporal evolving partial differential equations. In practice, two applications are investigated: a hyperbolic PDE (acoustic wave propagation) and parabolic PDE (heat equation). The emphasis is put on the influence of the boundary conditions on the ability of surrogate data-driven models to reproduce accurately the underlying dynamics. Thus, several types of boundary conditions are studied for each case, which are detailed next.

Acoustic Propagation of Gaussian Pulses:

The first application corresponds to the surrogate modelling of a 2D acoustic wave equation in a quiescent medium with speed of sound c0c_0, written in terms of the acoustic density ρ=ρ(x,y,t)\rho= \rho(x,y,t), with pp Gaussian density pulses as initial conditions:

∂2ρ∂t2+c0∇2ρ=0(6)\frac{\partial^2 \rho}{\partial t^2} + c_0 \nabla^2 \rho= 0 \tag*{(6)}
ρ(x,y,t=0)=∑i=1pεiexp⁡{−log⁡2(bi)2[(x−x0i)2+(y−y0i)2]}(7)\rho(x,y,t=0) = \sum_{i=1}^{p} \varepsilon^i \exp\left\{-\frac{\log2}{(b^i)^2}\left[(x-x_0^i)^2+(y-y_0^i)^2\right]\right\} \tag*{(7)}

where ∇2\nabla^2 is the Laplacian operator, (x0i,y0i)(x_0^i,y_0^i), εi\varepsilon^i and bib^i represent respectively the spatial positions of the center, the amplitude and the half-width of the ithi^{th} initial pulse.

Boundary Conditions: Three cases of boundary conditions are considered, representative of typical configurations found in acoustics. Each one of the BC constitutes a dataset to be employed for training a surrogate model:

  • Reflecting walls (Dataset 1 - D1): Hard-reflecting walls, representing interior acoustics, which is modeled with a Neumann boundary condition: ∇ρ⋅n=0\nabla\rho\cdot\mathbf{n} = 0.

  • Periodic walls (Dataset 2 - D2): Periodic conditions to model infinitely repeating domains.

  • Absorbing walls (Dataset 3 - D3): Radiation boundary conditions, modeling propagation of waves into the far-field (exterior acoustics). The challenge is to avoid spurious reflections that can pollute the computational domain.

Diffusion of Temperature Spots:

Second, the diffusion of temperature spots is studied, modeled by the following heat equation on the temperature T=T(x,y,t)T = T(x, y, t) with pp Gaussian density pulses as initial conditions:

∂T∂t+α∇2T=0(8)\frac{\partial T}{\partial t} + \alpha\nabla^2 T = 0 \tag*{(8)}

where α\alpha denotes the thermal diffusivity of the medium. The initial conditions are identical as those employed in Eq. (6b) (Gaussian temperature spots).

Boundary Conditions: Here, an additional dataset is generated, called Adiabatic walls (Dataset 4 - D4): Zero-flux adiabatic walls, modeled as Neumann boundary conditions ∇T⋅n=0\nabla T \cdot n = 0.

Datasets Generation and Parameters

The datasets of input-target fields is generated offline with the multi-physics open-source Palabos Lattice-Boltzmann Method (LBM) [10] numerical solver. We solve equations (6) and (7) for a duration T with a time-step of Δt\Delta t. A two-dimensional square domain is considered.

Acoustic Datasets (D1 to D3): Each set is composed of 600 LBM simulations, each with T=231T = 231 discrete time snapshots. Only the acoustic density fields ρ\rho are recorded in square domains of physical length L×LL \times L, discretized with N=200N = 200 cells per spatial direction and a spatial step of Δx=L/N=0.5\Delta x = L/N = 0.5. The LB time step is set to ΔtLBM=0.0029D/c0\Delta t_{LBM} = 0.0029D/c_0, with c0=1/3Δx/ΔtLBMc_0 = 1/\sqrt{3}\Delta x/\Delta t_{LBM} is the speed of sound. The four input density fields fed into the Neural network are equally spaced in time with ΔtNN=ΔtLBM\Delta t_{NN} = \Delta t_{LBM}. A random number of Gaussian pulses in the range p∈[1,5]p \in[1, 5] are used as initial conditions, with fixed amplitude and half-width, ε=0.001\varepsilon= 0.001 and b/Δx=12b/\Delta x = 12. The initial location of the pulses (x0,y0)(x_0, y_0) is also randomly sampled inside the domain following an uniform distribution. A 500/100 training/validation random split is employed for the simulations.

Temperature Datasets (D4): Each set is composed of 550 LBM simulations, each with T=160T = 160 discrete time snapshots. The temperature fields TT are recorded in square domains of physical length L×LL \times L, discretized with N=200N = 200 cells per spatial direction and a spatial step of Δx=L/N=0.005\Delta x = L/N = 0.005. The heat diffusivity is set to α=8Δx2/ΔtLBM\alpha= 8\Delta x^2/\Delta t_{LBM}, where the LB time step is set to ΔtLBM=1\Delta t_{LBM} = 1. The four input temperature fields fed into the Neural network are sampled so that ΔtNN=4ΔtLBM\Delta t_{NN} = 4\Delta t_{LBM}. The same initial conditions as in D1-D3 are employed. A 400/150 training/validation random split is employed for the simulations.

Results

To perform the evaluation of the presented boundary condition treatment, a Multi-Scale network with 0.4 million parameters is trained on the four proposed

datasets. As discussed in 2.4 the network is trained to minimize Eq. (5) for a single-step prediction. The Adam optimizer is employed, with a learning rate initially set to 10−410^{-4}, and decaying by 20% each time the loss reaches a plateau. The loss weights are set to λL2=0.02\lambda_{L2} = 0.02 and λGDL=0.98\lambda_{GDL} = 0.98. Data augmentation is employed through the random rotation of input-target tuples, and input fields are normalized by their mean and standard deviation. Batch size is kept at 32. Trainings are performed on a NVIDIA V100 GPU and convergence is achieved at about 1000 epochs for each run.

Wave Equation with Reflecting Boundary · The first case of study corresponds to Dataset 1. Density waves are fully reflected back into the domain after the interaction with boundaries. Therefore, the waves stay in the computational domain for infinitely long times, as no viscous dissipation is present. To test the presented approaches, 24 initial conditions with 1 to 5 Gaussian pulses randomly located in the initial domain are used as inputs for the auto-regressive model. A 25th initial condition is also tested, with the particular case of a Gaussian pulse initially centered in the domain, whose solution leads to strong symmetric solutions and is thus challenging for the neural network. For each initial condition (generated with the LBM solver), the auto-regressive strategy can recursively predict the density fields over a time horizon of TT iterations, by using the previous prediction as a new input. In order to improve the neural network robustness versus the error accumulation over time, an a posteriori correction is employed to improve the predictions after each time step [1], based on the conservation of acoustic energy over time in this particular application.

The error is evaluated in terms of relative root mean square error at each neural network iteration, namely E(ρ)=∥ρt−ρ^t∥2∥ρt∥2E(\rho) = \sqrt{\frac{\lVert\rho_t - \hat{\rho}_t \rVert_2}{\lVert\rho_t \rVert_2}}, where ρ^t\hat{\rho}_t is the high-dimensional density prediction at iteration tt. Here, the time horizon is set at T=600T = 600 iterations. For the explicit method, Neumann boundary conditions on the density fields are used to model such conditions. A first-order finite difference discretization is employed.

Results are qualitatively evaluated in Figure 3. For two different initial conditions (centered pulse and 3 randomly sampled pulses), the LBM reference (top row) is compared to several of the proposed approaches. For the implicit case (i.e., only padding) strategies, the best model regarding the employed padding is shown in the second row, corresponding to the replication padding. For both initial conditions, the auto-regressive strategy follows closely the ground truth data. In the third row, the implicit strategy employs circular padding and shows a good agreement in the case of the initially centered Gaussian pulse. However, for the other initial condition, the prediction diverges after some iterations. At iteration it=80it = 80, the pulse arriving on the left wall is re-injected at the right wall, mimicking the behavior of periodic boundaries instead of the reflecting ones, on which the network has been trained.

Snapshots of propagating density waves for CNN trained on Dataset 1

Figure 3. Snapshots of propagating density waves for CNN trained on Dataset 1, for two different initial conditions (it=0it = 0): centered Gaussian pulse (4 left columns) and randomly sampled Gaussian pulses (4 right columns). Different boundary condition treatments are compared to the LBM reference (top row). For the implicit strategy, the best (resp. worse) results regarding the employed padding are shown in the second row (replication) and third row (circular). For the spatial context strategy, results with circular padding are shown in the last row.

Figure 4 shows the time-evolution of the error averaged over 25 initial conditions for the different evaluated methods and Table 1 presents the error values for the last prediction at it=600it = 600. For the implicit strategy, results show that choosing the optimal padding crucially depends on the data physics. With circular padding the network is incapable of reproducing the desired physics except for some particular initial conditions, illustrated by the increased variance area signaling the presence of outliers. This is due to the artifacts discussed previously. For the rest of available padding (zeros, replication and reflection), the replication padding solution performs better than the other two even if error levels remain acceptable (below 2% relative error).

Figure 4. Relative error evolution for case D1 (reflecting walls). 3 boundary treatments are employed: implicit (full lines), spatial context (lines and star marker), and explicit BC treatment (dashed-dotted lines). Results are averaged over 25 different initial conditions and shaded area represent the standard deviation.

MethodZeros minZeros maxZeros avgCircular minCircular maxCircular avgReplicate minReplicate maxReplicate avgReflect minReflect maxReflect avg
Implicit0.1180.2710.1890.306∞\infty∞\infty0.0980.1450.1190.1330.3190.234
Context0.1140.2130.1760.1340.4530.6700.1610.2350.2040.1380.2970.226
Explicit0.7680.5801.3920.2910.7800.5450.0790.3410.1480.0640.1570.094

Table 1. Averaged relative error for Dataset 1 (hard reflecting walls) at iteration it=600it = 600 for 25 random initial conditions. Bolded results represent best padding for each strategy.

Such observations agree with other studies performed in image segmentation [2]: circular padding limits the CNNs ability to encode position information and can only be used with spatially periodic data. To further investigate this claim, an additional spatial context channel is employed, while maintaining circular padding. As observed in Figure 4 (middle plot, red curve), the error is significantly lower than the one obtained with the implicit strategy. This suggests that the additional input serves as an spatial anchor for the CNN to encode the hard-reflecting wall, which was not possible using only circular padding. The last row in Figure 3 demonstrates this improvement, even though the prediction error is higher than the one obtained with other padding methods. Furthermore, the addition of the spatial context channel reduces the overall variability of the chosen padding effects. While the error slightly increases for the replication case versus the implicit strategy, all three padding methods converge to similar error evolutions. This suggests that the additional context channel may force the network to explicitly learn similar convolutional kernels for boundary treatment, while this is not guaranteed by the implicit case.

For the explicit case, results in Figure 4 (right) show that the replicate and reflection padding cases achieve the lowest errors, while zero and circular paddings have larger errors. Note that the network is only trained for a one-step prediction and the explicit enforcing of the boundary is performed after each prediction. Thus, the explicit enforcing is only processed by the network in the auto-regressive context. Results show whether the employed padding strategy is compatible with the enforced boundary values: reflecting and replication padding can be though as first-order finite difference approximations of spatial derivatives, for 1-pixel padding. Zero and circular paddings are on the other hand not compatible with the enforced boundary values, performing worse in both cases.

This behavior highlights the possible benefits of employing explicitly boundary rules as long as the subsequent padding follows the same logic. It also calls to directly enforce such explicit rules in the input padding mechanism, which is left for future work.

Wave Equation with Periodic BCs : The second case of study corresponds to Dataset 2, where all four wall boundaries are set as periodic walls in the training data. The objective is that the neural network reproduces the wave propagation in an infinitely-repeating domain.

The relative error evolution over time is depicted in Figure 5a, for the implicit and spatial context methods. The explicit method is not employed here as it is equivalent to the implicit one: physical solvers employ additional “ghost cells” to wrap values from one boundary to another [16]. Also, reflect padding is not shown as it behaves very similarly to the replicate strategy. Results show that for both implicit and spatial context strategies, only circular padding is able to achieve acceptable error levels, with an average relative error of 15% for the implicit case and 11% at t=600t = 600 for the spatial context. The use of a padding strategy other than circular produces unphysical behavior at the boundaries, as the network is incapable of copying by itself the values arriving at one border to the opposite one.

Figure 5. Relative errors for (a) D2 (periodic) and (b) D3 (absorbing) cases. Full lines represent the implicit strategy and lines and star markers the spatial context strategy.

Wave Equation with Absorbing BCs : The third test case corresponds to the neural network trained on dataset 3, with non-reflecting (absorbing) boundary conditions. This case is significantly more challenging than the two previous cases: the initial Gaussian pulse is expected to propagate into the far field and completely leave the computational domain. Thus, the underlying data distribution is changing over time: the initial acoustic energy ∥ρ∥2\|\rho\|_2 tends towards 0 as t→∞t \to\infty. The challenge for the network is to correctly propagate the initial pulses outside the domain without spurious reflections at boundaries. As the signal energy tends towards zero, the error is now calculated relatively to the initial density, i.e., E(ρ)=∥ρt−ρ^t∥2∣ρt=0∣\mathcal{E}(\rho) = \sqrt{\frac{\|\rho_t-\hat{\rho}_t\|_2}{|\rho_{t=0}|}}. Similarly to previous experiments, 25 different initial conditions are employed for the auto-regressive tests.

For the explicit method, a local one-dimensional (LODI) non-reflecting equation is used to impose the values a the boundaries, which reads ∂tρ+c0∇ρ⋅n=0\partial_t\rho+c_0\nabla\rho\cdot n=0 [17]. First order finite differences are employed to discretize both the spatial and temporal derivatives.

The evolution of the error for 600 auto-regressive iterations is shown in Figure 5b. For both the implicit and spatial context cases, the results show a high variability between the different cases. While the implicit strategy with zero and reflect padding manage to produce low-error results, the other two padding strategies lead to diverging simulations. In contrast, when the spatial context is employed, the circular padding performs better, while the other three methods diverge. This unstable network behavior shows the complexity of the non-reflecting case in comparison to the previous ones. Instabilities can be directly related to the appearance of artifacts when the pulse impinges the walls, as can be seen in Figure 6: before the pulse arrival at the wall (it=120it = 120), all methods show stable and accurate predictions, while larger errors are shown after the first interaction with the BCs (it>160it > 160). Such artifacts lead in some cases to the unbounded growth of the density amplitude, leading to instabilities.

Density fields for Dataset 3 and replicate padding, comparing the three investigated methods.

Figure 6. Density fields for Dataset 3 and replicate padding, comparing the three investigated methods. The initial conditions is a centered pulse.

Interestingly, the explicit encoding of boundaries has an important stabilizing effect: all four padding methods show a very similar error evolution. Figure 6 show the main differences between implicit, spatial context and explicit approaches for the same padding (replicate): the first two cases initially damp the outgoing waves more efficiently. However, some unphysical reflections (shown by the asymmetry of density the fields) lead to artifacts remaining in the computational domain, eventually leading to instabilities in the implicit case. In the explicit case, even though reflections also exist, they follow a symmetric pattern, which corresponds very closely to the one found for reflecting walls, as seen in Figure 3. This suggests that the explicit encoding adds physical consistency to the network BC treatment. The error in that approach arises mainly from the lack of proper BC modeling, as the aforementioned LODI hypothesis cannot handle efficiently two-dimensional effects, typical of Gaussian pulses, for example at corners. Future research should reach out towards the treatment of such transverse effects [17]. This demonstrates that blending prior physics knowledge when handling boundary conditions can improve CNNs accuracy and robustness.

Heat Equation with Adiabatic Boundary: The last studied application corresponds to the diffusion of temperature pulses, modeled by the heat equation (8), when employing Neumann boundary conditions in the temperature (adiabatic walls). The same testing strategy is employed, by simulating 25 initial conditions for T=120T = 120 neural network recurrences. Two metrics are employed to analyze the network, the relative mean square error E(T)=∥Tt−T^t∥2/∥Tt∥2\mathcal{E}(T) = \sqrt{\lVert T_t - \hat{T}_t\rVert_2/\lVert T_t\rVert_2} and the evolution of the average spatial temperature TrmsT_{rms} that tends to a constant value in time, as no heat flux exists at the adiabatic walls.

Results in Figure 7 and Figure 8 confirm some of the previously made observations for Neumann BCs (reflecting acoustic case): the implicit padding strategy seems appropriate to correctly reproduce the dynamics, when those remain simple, such as in the diffusion of temperature fields. However, the error evolution shows that the use of padding that mimics the physical boundary condition (e.g. replicate) results in a lower long-term error, as well as a closer approximation to the TrmsT_{rms} constant level. Furthermore, the use of additional spatial context reduces the variability of the different padding choices, thus confirming the interest of employing such an additional input to make the predictions more robust to boundary condition effects. Here the explicit strategy does not results in an improved accuracy with respect to the baseline methods. The use of such an additional a-priori encoding of boundary conditions may only be justified in the presence of complex conditions, such as the one presented previously for the non-reflecting acoustics case.

Temperature fields for Dataset 4 and replicate padding, comparing the three investigated methods.

Figure 7. Temperature fields for Dataset 4 and replicate padding, comparing the three investigated methods. The initial condition is a centered pulse.

Figure 8. Results for case D4. (Top) Relative MSE on temperature fields, (bottom) temporal evolution of the spatially averaged temperature TrmsT_{rms}.

Conclusion

This paper presents an exhaustive comparison between several available methods to treat boundary conditions in fully convolutional neural networks for spatio-temporal regression, in the context of hyperbolic and parabolic PDEs. Such temporal regression tasks are highly sensitive to the well-posedness of boundary conditions, as small localized errors can propagate in time and space, producing instabilities in some cases. The characterization of such boundaries is crucial to improve the neural network accuracy.

The main outcomes are summarized next: employing padding alone yields accurate results only when the chosen padding is compatible with the underlying data. The addition of a spatial context channel seems to increase the robustness of the network in simple cases (Neumann boundaries), but fails for the more complex non-reflecting boundary case. Finally, the explicit encoding of boundaries, which enforces some physics constraints on border pixels, clearly demonstrates its superiority in such cases, allowing to design more robust neural networks. Such an approach should be further investigated in order to understand its coupling with the neural network behavior, and its extension to problems with several types of boundary conditions.

References

  1. [1]Alguacil, A., Bauerheim, M., Jacob, M.C., Moreau, S.: Predicting the Propagation of Acoustic Waves using Deep Convolutional Neural Networks. In: AIAA Aviation Forum. p. 2513. Reston, VA (2020)
  2. [2]Alsallakh, B., Kokhlikyan, N., Miglani, V., Yuan, J., Reblitz-Richardson, O.: Mind the Pad - CNNs can develop blind spots. In: 9th International Conference on Learning Representations (ICLR). Vienna, Austria (2021)arxiv.org/abs/2010.02178
  3. [3]Fotiadis, S., Pignatelli, E., Bharath, A.A., Lino Valencia, M., Cantwell, C.D., Storkey, A.: Comparing Recurrent and Convolutional Neural Networks for Predicting Wave 3. Alguacil et al. Propagation. ICLR 2020 Workshop on Deep Learning and Differential Equations (2020)
  4. [4]Gao, H., Sun, L., Wang, J.X.: PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parametric PDEs on Irregular Domain. Journal of Computational Physics 428, 110079 (mar 2021)
  5. [5]Geneva, N., Zabaras, N.: Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks. Journal of Computational Physics 403 (2019)DOI
  6. [6]Hamey, L.G.: A Functional Approach to Border Handling in Image Processing. In: International Conference on Digital Image Computing: Techniques and Applications, (DICTA). Adelaide, Australia (2015)DOI
  7. [7]Innamorati, C., Ritschel, T., Weyrich, T., Mitra, N.J.: Learning on the Edge: Investigating Boundary Filters in CNNs. International Journal of Computer Vision 128(4), 773–782 (apr 2020)DOI
  8. [8]Islam, A., Jia, S., Bruce, N.D.B.: How much Position Information do Convolutional Neural Networks encode? In: 8th International Conference on Learning Representations (ICLR). Addis Ababa, Ethiopia (2020)arxiv.org/abs/2001.08248
  9. [9]Kayhan, O.S., Van Gemert, J.C.: On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14274–14285. Virtual Event (2020)arxiv.org/abs/2003.07064
  10. [10]Latt, J., Malaspinas, O., Kontaxakis, D., Parmigiani, A., Lagrava, D., Brogi, F., Belgacem, M.B., Thorimbert, Y., Leclaire, S., Li, S., Marson, F., Lemus, J., Kotsalos, C., Conradin, R., Coreixas, C., Petkantchin, R., Raynaud, F., Beny, J., Chopard, B.: Palabos: Parallel Lattice Boltzmann Solver. Computers and Mathematics with Applications (apr 2020)
  11. [11]Lee, S., You, D.: Data-driven prediction of unsteady flow over a circular cylinder using deep learning. Journal of Fluid Mechanics 879(1), 217–254 (nov 2019)
  12. [12]Liu, G., Shih, K.J., Wang, T.C., Reda, F.A., Sapra, K., Yu, Z., Tao, A., Catanzaro, B.: Partial Convolution based Padding. arXiv preprint arXiv:1811.11718 (nov 2018)arxiv.org/abs/1811.11718
  13. [13]Liu, R., Jia, J.: Reducing boundary artifacts in image deconvolution. In: 15th IEEE International Conference on Image Processing. pp. 505–508. San Diego, CA (2008)DOI
  14. [14]Liu, R., Lehman, J., Molino, P., Such, F.P., Frank, E., Sergeev, A., Yosinski, J.: An intriguing failing of convolutional neural networks and the CoordConv solution. In: Advances in Neural Information Processing Systems 31. pp. 9605–9616. Montréal, Canada (2018)arxiv.org/abs/1807.03247
  15. [15]Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. 4th International Conference on Learning Representations, ICLR (nov 2016)arxiv.org/abs/1511.05440
  16. [16]Mohan, A.T., Lubbers, N., Livescu, D., Chertkov, M.: Embedding Hard Physical Constraints in Neural Network Coarse-Graining of 3D Turbulence. In: ICLR 2020 Workshop Tackling Climate Change with Machine Learning. arXiv (jan 2020)arxiv.org/abs/2002.00021
  17. [17]Poinsot, T.J., Lele, S.K.: Boundary conditions for direct simulations of compressible viscous flows. Journal of Computational Physics 101(1), 104–129 (1992)
  18. [18]Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686–707 (2019)DOI
  19. [19]Schubert, S., Neubert, P., Poschmann, J., Pretzel, P.: Circular convolutional neural networks for panoramic images and laser data. In: 2019 IEEE Intelligent Vehicles Symposium (IV). pp. 653–660. Paris, France (2019)DOI

Paper details

vlnc:W_683sgkbs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents