betaFeedback

WorksComputer Science

Climate-Invariant Machine Learning

Abstract

Projecting climate change is a generalization problem: we extrapolate the recent past using physical models across past, present, and future climates. Current climate models require representations of processes that occur at scales smaller than model grid size, which have been the main source of model projection uncertainty. Recent machine learning (ML) algorithms hold promise to improve such process representations, but tend to extrapolate poorly to climate regimes they were not trained on. To get the best of the physical and statistical worlds, we propose a new framework - termed "climate-invariant" ML - incorporating knowledge of climate processes into ML algorithms, and show that it can maintain high offline accuracy across a wide range of climate conditions and configurations in three distinct atmospheric models. Our results suggest that explicitly incorporating physical knowledge into data-driven models of Earth system processes can improve their consistency, data efficiency, and generalizability across climate regimes.

Teaser: Physically-informed transformations aid the machine learning of Earth system model processes that generalize across climates.

Introduction

Background

Following its success in computer vision and natural language processing, machine learning (ML) is rapidly percolating through climate science (e.g., reviews by [1, 2, 3, 4, 5]). We use the term ML here to broadly describe algorithms that learn a task from data without being explicitly programmed for that task. Applications of ML in atmospheric science include the emulation of radiative transfer algorithms (e.g., [6, 7, 8, 9]), momentum fluxes (e.g., [10, 11, 12, 13]) and microphysical schemes (e.g., [14, 15, 16]), the bias correction of climate predictions (e.g., [17, 18]), the detection and classification of clouds and storms (e.g., [19, 20, 21, 22]), and the development of subgrid-scale “closures” (i.e. representation based on coarse-scale processes only) from high-resolution simulation data (e.g., [23, 24, 25]), which is the main application discussed in this manuscript.

ML algorithms typically optimize an objective on a training dataset and make implicit assumptions when extrapolating. Here, extrapolation refers to predictions outside of the training data range, henceforth referred to as out-of-distribution predictions. As an example, multiple linear regressions (MLR) assume that the linear relationship that best describes the training set is valid outside of that training set. Alternatively, when confronted with out-of-distribution inputs, random forests (RF, [26]) find the closest inputs in their training sets and assign the corresponding outputs regardless of the out-of-distribution input values. Neural networks (NN), which are powerful nonlinear regression and classification tools, rely on nonlinear activation functions and fitted weights to extrapolate. Except in specific situations (e.g., samples in the close neighborhood of the training set or described by the same nonlinear mapping as the training set), there is no reason why NNs should generalize well far outside of their training sets. We show later that different NN training approaches on the same data can lead to drastically different out-of-distribution predictions, highlighting the uncertainty associated with such predictions.

In climate applications, this extrapolation issue means that ML algorithms typically fail when exposed to dynamic, thermodynamic, or radiative conditions that differ substantially from the range of conditions they were trained on. Examples include [27], who showed that an RF-based moist convection scheme generalizes poorly in the tropics of a climate 6.5K warmer than the training climate; and [28], who showed that NNs and support vector machines down-scaling surface air temperature made substantial extrapolation errors when exposed to temperatures 2-3K warmer than in the training set. [29] showed that an NN-based thermodynamic subgrid-scale closure generalizes well to climates 1-2K warmer than the training one, but makes large errors as soon as the test climate is 4K warmer than the training one. [30] confirmed that these generalization errors remain even when the NN subgrid closure is modified to enforce conservation laws to within machine precision. This has led several studies to recommend training ML models in multiple climates if possible [31, 32]. Interestingly, both Guillaumin et al. [33] who trained an NN parameterization for subgrid oceanic momentum transport and Molina et al. [34] who trained convolutional NNs [35] to classify thunderstorms in high-resolution model outputs found that their ML models generalized well to a warmer climate. While this may be because both models relied heavily on velocity inputs and their gradients, whose distributions changed only slightly when the climate warmed, Molina et al. noted that using two types of ML layers, namely batch normalization (BN, [36]) followed by dropout (DP, [37]), was key to this successful generalization.

DP and BN are two examples of a larger set of methods that help NNs generalize and avoid overfitting, broadly referred to as “regularizations” [38]. Most empirical regularization methods (e.g., L1 regularization) rely on the parsimony principle, i.e. that simpler models, accurately describing the training set with fewer fitted parameters, are preferable to more complex models and generalize better to unseen conditions. More systematic approaches to regularization have been developed to use ML models in out-of-distribution situations that still require the same inputs/outputs, referred to as domain adaptation (e.g., [39, 40, 41]), a particular case of transfer learning (e.g., [42]). While not all domain adaptation approaches (sample-based, statistics-based, ensemble-based, domain-invariant feature learning, domain mapping, etc.) need supervision [43], they usually require at least a few samples in the generalization domain.

Without dismissing existing domain adaptation methods, we here focus on physically-informed methods that do not require samples in the generalization domain for three reasons: (1) one of the climate science community’s long-term goals is to train ML models that rely on historical observations only as we cannot, by definition, observe the future climate; (2) as shown later, even if we have access to simulation data across climates, ML models that intrinsically generalize to climates they have not been trained on tend to be more data-efficient and robust to other changes (e.g., configuration changes); and (3) physically-informed methods can be readily combined with existing domain adaptation and regularization methods. Motivated by these challenges, we ask:

How can we enhance ML algorithms with physical knowledge to make accurate predictions in climate conditions that – in standard variables – lie far outside of the training set?

Problem Definition

Our scientific contribution is to transform a mapping constructed using the original data’s features, henceforth referred to as “raw-data” mapping, into a mapping that remains nearly constant across climates, here referred to as “climate-invariant”. Inspired by invariants in physics and self-similarity in fluid mechanics [44], we make the ML-emulated mapping climate-invariant by transforming the input and output vectors so that their distributions shift minimally across different climates (see Figure 1).

Diagram showing raw inputs and outputs transformed through physical transformations implemented via custom pipelines/layers into climate-invariant inputs and outputs

Figure 1. By transforming inputs xx and outputs yy to match their probability density functions across climates, the algorithms can learn a transformed mapping ϕ~\tilde{\phi} that holds across climates. To illustrate this, we show the marginal distributions of inputs and outputs in two different climates using blue and red lines, before (top) and after (bottom) the physical transformation.

We demonstrate this framework’s utility by adapting ML closures of subgrid atmospheric thermodynamics (i.e., coarse-scale thermodynamic tendencies resulting from subgrid convection, radiation, gravity waves, and turbulence) so that they generalize better across climates. The motivation for this application is two-fold. First, purely physically-based subgrid closures remain one of the largest sources of uncertainties in Earth system models [45, 46, 47]. While ML-based closures have emerged as a promising alternative to traditional semi-empirical models [48], they lack robustness [49, 50] and, as discussed earlier, usually fail to generalize across climates [27, 29, 30]. Second, atmospheric thermodynamic processes are directly affected by global temperature changes, e.g., in response to anthropogenically-forced climate change [51]. Therefore, predicting subgrid thermodynamics in a warm climate with an ML model trained in a cold climate leads to very apparent failure modes [30] that we can transparently tackle.

In mathematical terms, our goal is to build a climate-invariant mapping between the input vector xx representing the large-scale (≈100km\approx100\mathrm{km}) climate state and the output vector yy grouping large-scale thermodynamic tendencies due to explicitly-resolved convection and parameterized radiative transfer and turbulent mixing at the ∼1km\sim1\mathrm{km}-scale (see SM B1 for details). We keep the overall structure of the mapping x↦yx \mapsto y fixed throughout the manuscript and aim to predict yy as accurately as possible in training and generalization climates (out-of-distribution prediction). Note that this mapping makes some implicit assumptions based on successful past work [29] [52], including locality in horizontal space and time (outputs only depend on inputs in the same atmospheric column at the same time step) and determinism (only one possible output vector for a given input vector). We include cloud radiative effects in all heating terms (total heating T˙\dot{T}, long-wave heating lw, and shortwave heating sw), but for simplicity do not predict changes in cloud liquid water and ice, and exclude cloud water and greenhouse gases other than water vapor qvq_v from the input vector xx.

After introducing the climate simulations and training/validation/test split (§ 2), we define the climate invariant mapping and feature transformations (§ 3), and demonstrate and explain their ability to generalize (§ 4) before concluding. We refer the reader to the Supplementary Materials (SM) for data availability (SM A), additional derivations and descriptions of the mapping and physical transformations (SM B), the implementation of our ML framework (SM C), and additional results (SM D).

Data

To test the robustness of our framework across model formulations and configurations, we use three distinct storm-resolving climate models and experimental set-ups: aquaplanet simulations using the Super-Parameterized Community Atmosphere Model version 3.0 (SPCAM3), Earth-like simulations (i.e., with continents) using the Super-Parameterized Community Earth System Model version 2 (SPCESM2), and quasi-global aquaplanet hypohydrostatic simulations using the System for Atmospheric Modeling version

6.3 (SAM). SPCAM3 and SPCESM2 assume a strict scale separation between the resolved coarse scales and subgrid processes, making them ideal testbeds to machine learn local subgrid closures [29] [53]. In contrast, SAM does not assume scale separation as a global storm-resolving model. This improves realism but requires coarse-graining SAM’s output for ML parameterization purposes [54, 55]. For each climate model, we run three simulations with three different prescribed surface temperature distributions:

  • (+0K) A reference simulation with a temperature range analogous to the present climate

  • (-4K) A cold simulation with surface temperatures 4K cooler than the (+0K) simulation

  • (+4K) A warm simulation with surface temperatures 4K warmer than the (+0K) simulation,

with the exception of SAM for which only the (-4K) and (+0K) simulations are available. By prescribing surface temperature, we focus on ML’s ability to consistently predict the atmospheric response to climate change across configurations. Projecting climate change involves a broader range of processes and is beyond this work’s scope. We summarize the simulations and indicate their spatiotemporal resolutions in Table S1. Figure 2 gives a visualization of surface temperatures in each model and Figure S1 provides snapshots of mid-tropospheric subgrid heating (Figure 2b), which is one of our ML models’ outputs.

Surface temperatures in the three utilized atmospheric models

Figure 2. Surface temperatures in the three utilized atmospheric models. Prescribed surface temperature (in K) for (left) the aquaplanet SPCAM3 model and (right) the hypohydrostatic SAM model. (Center) Annual-mean, near-surface air temperatures in the Earth-like SPCESM2 model.

RowInputSPCAM3SPCESM2SAM
1q600hPaq_{600\mathrm{hPa}}20.3, 35.117.1, 29.522.1
2qdeficit,600hPaq_{\mathrm{deficit},600\mathrm{hPa}}24.9, 36.518.1, 31.030.0
3RH600hPa\mathrm{RH}_{600\mathrm{hPa}}3.6, 8.23.2, 5.34.3
4T850hPaT_{850\mathrm{hPa}}53.2, 64.325.7, 37.351.9
5Tfrom NS,850hPaT_{\mathrm{from\ NS},850\mathrm{hPa}}5.1, 6.23.3, 6.410.5
6Bplume,850hPaB_{\mathrm{plume},850\mathrm{hPa}}9.4, 14.73.6, 7.65.8
7T150hPaT_{150\mathrm{hPa}}30.2, 33.531.6, 34.465.6
8Tfrom NS,150hPaT_{\mathrm{from\ NS},150\mathrm{hPa}}38.0, 53.714.5, 28.751.0
9Bplume,150hPaB_{\mathrm{plume},150\mathrm{hPa}}35.1, 42.210.4, 20.921.1
10LHF8.6, 14.59.7, 12.5
11LHFq\mathrm{LHF}_{q}4.7, 9.510.0, 10.7
12LHFΔq\mathrm{LHF}_{\Delta q}6.3, 9.99.9, 14.0

Table S1. Hellinger distance (in %) away from the (-4K) simulation for the PDFs of key inputs (q600hPaq_{600\mathrm{hPa}}, T850hPaT_{850\mathrm{hPa}}, T150hPaT_{150\mathrm{hPa}}, LHF) and their transformations. (+0K) distance in gray and (+4K) distance in red.

Surface temperatures and subgrid heating rate in the three utilized atmospheric models

Figure S1. Surface temperatures and subgrid heating rate in the three utilized atmospheric models. (a) Prescribed surface temperature (in K) for (left) the aquaplanet SPCAM3 model and (right) the hypohydrostatic SAM model. (center) Annual-mean, near-surface air temperatures in the Earth-like SPCESM2 model. (b) Snapshots of near-surface subgrid heating rate (in K/day). For each model, we show the cold (-4K), reference (+0K), and warm (+4K) simulations.

Super-Parameterized Aquaplanet Simulations

We use data from two-year SPCAM3 [56] climate simulations in an aquaplanet configuration [57], with zonally-symmetric surface temperatures fixed to a realistic meridionally-asymmetric profile [58]. The insolation is fixed to boreal summer conditions with a full diurnal cycle. A two-dimensional storm-resolving model is embedded in each grid cell of SPCAM3, namely 8 SAM atmospheric columns using a spatiotemporal resolution of 4km×30levels×25s, and the default one-moment microphysical scheme [59]. SPCAM3 combines a spectral primitive equation solver with a semi-Lagrangian dynamical core for advection [57]. The (+0K) SPCAM3 simulation was first presented in [53] and subsequently used to train ML subgrid closures in [29] [50, 60]. Inspired by the generalization experiment of [27], the (+4K) simulation was introduced in [29], and we ran the (-4K) simulation for the work presented here to increase the surface temperature generalization gap from 4K to 8K.

Super-Parameterized Earth-like Simulations

We run three two-year SPCESM2 [61] climate simulations in an Earth-like configuration with realistic surface boundary conditions, including a land surface model, seasonality, aerosol conditions representative of the year 2000, and a zonally-asymmetric annual climatology of sea surface temperatures derived from the “HadOIB1” dataset [62]. We use CESM v2.1.3 to couple CAM v4.0 with the Community Land Model version 4.0, and similarly embed 32 SAM columns in each atmospheric grid cell to explicitly represent deep convection. Our (+0K) simulation is similar to [52]’s, which showed the potential of ML for subgrid closures in Earth-like conditions.

Quasi-Global Aquaplanet Hypohydrostatic Simulations

While super-parameterization is well-adapted to statistically learning subgrid closures thanks to its explicit scale separation, this scale separation comes at the cost of distorted mesoscale systems and momentum fluxes [63]. Furthermore, most ML subgrid closures are based on coarse-graining high-resolution simulations (e.g., [64, 65]). This motivates us to also test the climate-invariant framework in hypohydrostatic SAM simulations in which the dynamics are not affected by a prescribed scale separation. Computational expense is reduced through hypohydrostatic scaling, which multiplies the vertical acceleration in the equations of motion by a factor of 16 to increase the horizontal scale of convection without overly affecting the larger-scale flow [66, 67]. While these simulations use idealized settings, such as aquaplanet configurations, an anelastic dynamical core, a quasi-global equatorial beta plane domain, and perpetual equinox without a diurnal cycle, [68] showed that they produce tropical rainfall intensity and cluster-area distributions that are close to satellite observations. The prescribed surface temperature distribution in the con-

trol simulation of [68] is designed to be close to zonal-mean observations [69], and its maximum value is roughly 2K colder than that of the distribution used for the (+0K) SPCAM3 simulation. To better match the SPCAM3 maxima of distributions of upper-level temperatures and humidities, we choose to treat this SAM control simulation as the (-4K) SAM simulation, and the warm simulation of [68] as the (+0K) SAM simulation. We refer the reader interested in the details of the simulations and the coarse-graining (here by a factor of 8) to [54]. Differences in climate model formulation and ML parameterization design lead to key differences in the mappings learned for SAM as compared to SPCAM3/SPCESM2, which we summarize below:

  • the input vector does not contain specific humidity, surface pressure, sensible heat fluxes, or latent heat fluxes, but instead contains the total non-precipitating water concentration and uses distance to the equator as a proxy for solar insolation,

  • the output vector includes the subgrid total non-precipitating water tendency instead of the subgrid specific humidity tendency, and the subgrid liquid/ice static energy tendency instead of the subgrid temperature tendency,

  • the output vector does not contain subgrid longwave and shortwave heating,

  • SAM uses a height-based vertical coordinate rather than a pressure-based one, and

  • the generalization experiment is from (-4K) to (+0K) (unavailable (+4K) simulation).

Normalization and Training, Validation, and Test Split

Both generalization experiments expose ML models to out-of-distribution inputs they have not been trained on. Following best ML practices [70], we use the training set to optimize the ML model’s trainable parameters, save the trainable parameters that led to the best performance on the validation set to avoid overfitting the training set, and evaluate the final model on samples from a separate test set. We split each of the 8 simulations into training/validation/test sets by using non-contiguous 3-month periods (reported in Table S1) to avoid high temporal correlations between training/validation/test set samples [71]. Following [29], the normalization procedure involves subtracting the mean value of each input variable at each vertical level and dividing by the maximum range of that variable across the entire atmospheric column.

To understand which solutions are most promising for helping ML algorithms generalize to unseen conditions, we design two generalization experiments: (1) Training and validating ML models on cold simulations (-4K) and testing them on warm simulations (+4K for SPCAM3/SPCESM2 and +0K for SAM); and (2) training and validating ML models on aquaplanet simulations (SPCAM3) and testing them on Earth-like simulations with continents (SPCESM2).

Theory

We formally define a climate-invariant mapping as a mapping that is unchanged across climates. In practice, it is difficult to find mappings that are exactly invariant and we will use the terminology climate-invariant for any mapping that remains approximately constant across climates. To achieve climate invariance, we introduce physically-based feature transformations, defined as physically-informed functions that map the inputs/outputs to different inputs/outputs whose distributions vary little across climates. We deem the physical transformation to be climate-invariant if it is successful at limiting distributions variations of the inputs/outputs across climates. Note that climate-invariant transformations are distinct from nondimensionalization in dimensional analysis, as nondimensionalization does not necessarily alter distribution shape while climate-invariant transformations may yield variables that have physical units.

Throughout the following section, we compare two transformation options for each input, whose univariate PDFs are depicted for all three atmospheric models in Figure 3: No transformation (top) and our most successful transformation (bottom). All transformations are derived in SM B2. Our comparison relies on the Hellinger and Jensen-Shannon PDF distance metrics defined and calculated in § 6 and SM D1. To prevent information leaks from generalization test sets into the physically-informed ML framework, we take two precautions: (1) the physical transformations are fixed, meaning that their structure and parameters are non-trainable; and (2) transformations are ranked based on their generalization from (-4K) to (+0K) in SPCAM3. Our (+4K) results across models and configurations independently confirm this ranking.

Physical transformations can align distributions across climates

Figure 3. Physical transformations can align distributions across climates. We show the univariate distributions of selected raw inputs xx: (a) 600hPa specific humidity, (b) 850hPa temperature, and (c) latent heat flux in the cold (blue), reference (gray), and warm (red) simulations of each model (SPCAM3, SPCESM2, and SAM). For each variable, we also show the PDFs of the transformed inputs x~\tilde{x} as discussed in text. From top to bottom, the variables are qq (g/kg), RH, TT (K), BplumeB_{\mathrm{plume}} (m/s2^2), LHF (W/m2^2), and LHFΔq\mathrm{LHF}_{\Delta q} (kg m−2^{-2}s−1^{-1}). For a given variable and transformation, we use the same vertical logarithmic scale across models.

Specific Humidity: Without any transformation, the PDF of specific humidity qq (Figure 3a, top) extends through a considerably larger range as the climate warms. This is because, barring supersaturation, qq has a theoretical upper bound in a given climate, namely the saturation specific humidity, which increases quasi-exponentially with temperature through the Clausius-Clapeyron relation (e.g., [72, 73]). The relative humidity (RH) transformation q~RH\tilde{q}_{\mathrm{RH}} (Figure 3a, bottom) normalizes specific humidity by its saturation value. As a result, most of the RH PDF lies within [0, 1], except for a few atmospheric columns exhibiting super-saturation in SPCAM, and that PDF changes little as the climate warms [74]. In addition to capturing grid-scale saturation, q~RH\tilde{q}_{\mathrm{RH}} helps predict the subgrid effects of dry-air entrainment, known to regulate tropical convection [75, 76, 77] (see SM B2b for details of RH calculations).

Temperature: The PDF of temperature TT (Figure 3b, top) shifts quasi-linearly as the climate warms. To address this shift without compromising the approximate invariance of tropopause temperatures with warming [78, 79, 80], we derive a temperature transformation directly relevant for moist convection: the buoyancy of a non-entraining, moist static energy-conserving plume T~buoyancy\tilde{T}_{\mathrm{buoyancy}} (Figure 3b, bottom, see SM B2c for this buoyancy’s derivation). This transformation is inspired by recently introduced lower-tropospheric buoyancy measures [81, 82], but with an extension to the full troposphere [83]. While T~buoyancy\tilde{T}_{\mathrm{buoyancy}} does not explicitly include entrainment effects, the mapping of T~buoyancy(p)\tilde{T}_{\mathrm{buoyancy}}(p) and q~RH(p)\tilde{q}_{\mathrm{RH}}(p) to heating and moisture sink will implicitly include these. This transformation captures leading order effects needed to yield approximate climate invariance (Figure 3b). T~buoyancy\tilde{T}_{\mathrm{buoyancy}} increases physical interpretability by linking the vertical temperature structure and near-surface humidity changes to a metric that correlates well with deep convective activity [84]. T~buoyancy\tilde{T}_{\mathrm{buoyancy}} also captures the role of near-surface humidity relative to the temperature structure aloft in contributing to moist convective instability in the tropics.

Latent Heat Flux: The last input whose distribution changes visibly with warming is the latent heat flux LHF (Figure 3c, top; the remaining inputs – sensible heat flux and surface pressure – change less with warming and are discussed in SM B2d). Similar to specific humidity, the increase of latent heat fluxes with warming is directly linked to the Clausius-Clapeyron relationship (e.g., [85]). To address this shift, we leverage the bulk aerodynamic formula to represent surface fluxes and to provide a physics-motivated transformation of LHF using the near-surface saturation deficit (Figure 3c, bottom). This transforms LHF, a thermodynamic variable, into LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q}, approximately proportional to the magnitude of near-surface horizontal winds and density (e.g., [85]), whose distributions vary less with warming. Note that this scaling is less effective over land (e.g., in SPCESM2) where evapotranspiration changes do not follow a Clausius-Clapeyron scaling.

We now show that all three input transformations (q~RH(p)\tilde{q}_{\mathrm{RH}}(p), T~buoyancy(p)\tilde{T}_{\mathrm{buoyancy}}(p), and LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q}) lead to statistically significant improvements in the ML models’ ability to generalize.

Results

The results are organized as follows. After demonstrating the benefits of progressively transforming the ML models’ inputs (Figure 4), we show how climate invariant models learn subgrid closures across climates and configurations during training (Figure 5, Figure S7). We then discuss the global skill of different models after training (Figure 6, Figure S8&Figure 9). Finally, we investigate the structure of climate-invariant mappings to understand why they generalize better across climates (Figure 7), even when data from multiple climates are available (Figure 8).

Mean-squared error profiles in cold and warm tropics

Figure 4. All neural networks, trained in the cold climate, exhibit low error in the cold climate’s test set (a), but much larger error in the warm climate’s test set (b). This generalization error decreases as inputs are incrementally transformed: First no transformation (blue), then the vertical profile of specific humidity (orange), then the vertical profile of temperature (green), and finally latent heat fluxes (red). For reference, the purple line depicts an NN trained in the warm climate. We depict the tendencies’ mean-squared error versus pressure, horizontally-averaged over the Tropics of SPCAM3 aquaplanet simulations, for the four model outputs: total moistening (q˙\dot{q}), total heating (T˙\dot{T}), longwave heating (lw) and shortwave heating (sw). Given that the raw-data NN’s generalization error (blue line) greatly exceeds that of the transformed NNs, we zoom in on each panel to facilitate visualization.

Trained onModelTraining Set<br>Same temperature<br>Same conf.Validation Set<br>Same temperature<br>Same config.Training Set<br>Different temp.<br>Same config.Training Set<br>Same temperature<br>Different config.
Cold<br>Aquaplanet<br>(-4K)<br>SPCAM3MLR RD287 (288) , 1.0288 (289) , 1.0759 (785) , 1.0242 (265) , 1.0
MLR CI294 (294) , 1.0295 (295) , 1.0671 (671) , 0.9245 (265) , 1.0
NN RD172 (172) , 0.6172 (173) , 0.62167 (2242) , 2.91098 (1117) , 4.5
NN RD+DN222 (222) , 0.8205 (206) , 0.73225 (3509) , 4.3258 (318) , 1.1
NN CI168 (169) , 0.6169 (169) , 0.6422 (425) , 0.6297 (316) , 1.2
NN CI+DN214 (214) , 0.7198 (198) , 0.7483 (487) , 0.6224 (227) , 0.9
Warm<br>Aquaplanet<br>(+4K)<br>SPCAM3MLR RD642 (644) , 1.0643 (645) , 1.0328 (330) , 1.0507 (543) , 1.0
MLR CI650 (651) , 1.0650 (651) , 1.0315 (315) , 1.0529 (598) , 1.0
NN RD363 (364) , 0.6362 (364) , 0.6386 (472) , 1.21354 (1612) , 2.7
NN RD+DN473 (473) , 0.7434 (436) , 0.7248 (248) , 0.8728 (1275) , 1.4
NN CI354 (355) , 0.6355 (356) , 0.6199 (202) , 0.61241 (1293) , 2.4
NN CI+DN451 (451) , 0.7412 (414) , 0.6210 (212) , 0.6567 (659) , 1.1
Cold<br>Earth-like<br>(-4K)<br>SPCAM5MLR RD197 (197) , 1.0197 (197) , 1.0508 (562) , 1.0299 (299) , 1.0
MLR CI203 (203) , 1.0202 (202) , 1.0459 (459) , 0.9307 (307) , 1.0
NN RD114 (114) , 0.6116 (116) , 0.6709 (803) , 1.4247 (251) , 0.8
NN RD+DN165 (165) , 0.8152 (152) , 0.8512 (575) , 1.0247 (247) , 0.8
NN CI112 (112) , 0.6114 (114) , 0.6292 (294) , 0.6312 (323) , 1.0
NN CI+DN158 (158) , 0.8144 (144) , 0.7346 (347) , 0.7229 (233) , 0.8

Figure 5. Model error across temperatures and configurations. Mean-Squared Error (MSE, in units W2 m−4\mathrm{W}^{2}\,\mathrm{m}^{-4}) of six models trained in three simulations (first column) and evaluated over the training or validation set of the same and two other simulations (last four columns). The models (second column) are raw-data (RD) or climate-invariant (CI), multiple linear regressions (MLR) or neural nets (NN), and sometimes include dropout layers preceded by a batch normalization layer (DN). The models are trained for 20 epochs. We first provide the MSE corresponding to the epoch of minimal validation loss, then the MSE averaged over the 5 epochs with lowest validation losses (in parentheses), and finally the MSE divided by the baseline MSE, where we use the raw-data multiple linear regression as baseline. Note that “Different Temperature” refers to (+4K) for (-4K) training sets and vice versa. In each application case, we highlight the best model’s error using bold font.

Raw-data mapping and climate-invariant neural networks for SPCAM3, SPCESM2, and SAM

Figure 6. Climate-invariant NNs (b) address the raw-data NNs’ generalization problems in the “warm Tropics” (a). This is demonstrated by the 500hPa subgrid heating’s coefficient of determination R2R^2 calculated over the test set. We train NNs using the cold (-4K) training set of each model (SPCAM3, SPCESM2, and SAM). We note that these NNs do not use dropout nor batch normalization and refer readers to Figure S8 for latitude-pressure cross-sections.

SHAP Feature Importance Matrix for raw-data and climate-invariant mappings

Figure 7. Explainable artificial intelligence suggests that climate-invariant mappings are more spatially local. We depict M\mathcal{M} for the (a) raw-data and (b) climate-invariant NNs trained in the SPCAM3 (+4K) warm aquaplanet simulation. The x-axes indicate the inputs’ vertical levels, from the surface (left, 10310^{3}hPa) to the top of the atmosphere (right, 0hPa), while the y-axes indicate the outputs’ vertical levels, from the surface (bottom, 10310^{3}hPa) to the top of the atmosphere (top, 0hPa). We additionally indicate the 200hPa vertical level with dotted black lines.

Test mean-squared error of neural networks trained in both climates

Figure 8. Climate-invariant (CI) neural networks trained on datasets containing both cold (-4K) and warm (+4K) samples outperform raw-data (RD) models offline in ≈95%\approx95\% of cases, with less sensitivity to the data partition used for training. Dots on the left represent the median RD error from a 10-fold cross-validation without replacement, with horizontal ticks indicating the 1st and 9th deciles. Stars on the right correspond to the median CI error. Ticks denote the majority of cases, for which CI models outperform RD models, even when data from both climates are available; crosses indicate the rare exceptions. We use a logarithmic scale for both axes.

Five-step workflow showing transformation discovery, verification, training, and testing across reference and target climates

Figure 9. Proposed five-step workflow to find climate-invariant transformations. The transformations help Machine Learning (ML) models generalize from a reference (ref.) climate to a target one, using (top) a baseline Multiple Linear Regression (MLR) as an initial guide.

Learning curves of neural networks and multiple linear regressions tested in cold and warm aquaplanet and cold Earth-like simulations

Figure S7. Unlike raw-data models, climate-invariant models continuously learn about subgrid thermodynamics in the warm aquaplanet as they are trained in the cold aquaplanet. More generally, they can learn information about configurations that differ from the one they were trained in. Learning curves of neural nets (left) and multiple linear regressions (right) tested in the (-4K) cold aquaplanet simulation (a, top row), the (+4K) warm aquaplanet simulation (b, middle row), and the (-4K) cold Earth-like simulation (c, bottom row). The lines’ colors indicate the training dataset, while their symbols refer to whether the ML model is raw-data (circle), climate-invariant (star), or climate-invariant with dropout layers before each activation function and batch normalization (square). (b) We additionally zoom in on the climate-invariant neural network’s learning curve in the (+4K) simulation.

Latitude-Pressure cross-sections of subgrid heating’s coefficient of determination for raw-data and climate-invariant neural networks

Figure S8. Latitude-Pressure cross-section of subgrid heating’s coefficient of determination R2R^2. We train (a) raw-data and (b) climate-invariant NNs using the cold (-4K) training set of each model (SPCAM3, SPCESM2, and SAM) and test them in the cold (-4K) and warm (+4K) climates. See Figure 5 or Figure S9 for the colorbar.

Benefits of Incremental Input Transformations

In this section, we demonstrate that incrementally transforming the inputs of neural networks progressively improves their generalization abilities from the cold (-4K) aquaplanet (SPCAM3) simulation to the

warm (+4K) aquaplanet simulation. The largest surface temperature jump tested in this study is between the cold aquaplanet simulation and the tropics of the warm aquaplanet simulation (“warm Tropics” for short), defined as the regions with out-of-distribution surface temperatures, whose latitudes are between -15°S and 23°N (approximately the red regions in Figure 2’s top-left subplots). To expose the failure modes of the “brute force” model and the benefits of progressively transforming the inputs, we first trained several NNs on the cold aquaplanet until they reached high accuracy (Figure 4a) before testing their out-of-distribution generalizability in the warm Tropics (Figure 4b).

In the cold Tropics, the vertical profiles of the mean-squared error (MSE) are nearly indistinguishable for all types of NNs and roughly follow the vertical structure of subgrid variance, as discussed in [53, 60]. When evaluated in the warm Tropics, the MSE of the “brute force” NN (blue line) increases by a factor of ≈10\approx10 and peaks above 100 W2m−4100\ \mathrm{W}^{2}\mathrm{m}^{-4}, underlining how raw-data NNs fail to generalize across climates. As discussed in § 3, we progressively transformed the inputs starting with specific humidity, which is transformed to relative humidity (orange line). This first transformation decreases the MSE so much (by a factor of 5-10) that we need to zoom in on each panel to distinguish the generalization abilities of additional NNs. Adding the transformations of temperature to plume buoyancy (green line) improves the generalization MSE for all variables. Adding the LHF to LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q} transformation (red line) further decreases generalization MSE, except for shortwave heating where the MSE improves in the cold but not the warm climate. Impressively, transforming all three inputs decreases the MSE so much that the resulting climate-invariant NN’s MSE (red line) is within ≈25%\approx25\% of the MSE of a “raw-data” NN that was directly trained in the warm climate (purple line).

Hereafter, we use “climate-invariant” to refer to models for which all three inputs (qq, TT, LHF) but no outputs were transformed, solely based on physical principles. After demonstrating their success in the aquaplanet case, we are now ready to investigate how these climate-invariant models learn in the other climates and simulations introduced in § 2.

Learning across Climates and Configurations

In this section, we show that climate-invariant models learn mappings that are valid across climates and configurations, and that their efficacy improves when used in conjunction with ML regularization techniques like BN and DP layers.

Fig 5 shows the MSE of ML models trained in three different datasets, and evaluated over their training and validation sets, and over test sets of different temperature and configuration. As discussed previously, climate-invariant NNs (NN CI) generalize better to warmer climates than raw-data NNs (NN RD). We go one step further by examining learning curves, defined as the MSE of an ML model at the end of each epoch during training (One epoch corresponds to the ML model being fed the entire training set once). Impressively, the learning curve of the climate-invariant NN trained in the cold aquaplanet but tested in the warm aquaplanet (starred blue line in Figure S7) is mostly decreasing, supporting this manuscript’s key result: climate-invariant NNs continuously learn about subgrid thermodynamics in the warm aquaplanet as they are trained in the cold aquaplanet. In contrast, the “raw-data” NN trained in the cold aquaplanet but tested in the warm aquaplanet makes extremely large generalization errors, which worsen as the model is trained in the cold aquaplanet (see SM D2 for details).

Climate-invariant NNs also facilitate learning across configurations, i.e., from the aquaplanet to the Earth-like simulations and vice-versa (see NN CI rows in Figure 5). Climate-invariant transformations additionally improve the MLR baseline’s generalization ability, albeit less dramatically. This smaller improvement in MLR’s generalization abilities is linked to its relatively small number of trainable parameters, resulting in (1) “raw-data” MLRs generalizing better than “raw-data” NNs; and (2) MLRs having lower descriptiveness and fitting their training sets less well, limiting the maximal accuracy of climate-invariant MLRs on test sets.

There are a few cases in which transforming inputs does not fully solve the generalization problem, e.g., when trying to generalize from the aquaplanet to the Earth-like simulation. In that case, we leverage the fact that input transformations can easily be combined with standard techniques to improve generalization tion, such as DP layers before each activation function and a single BN layer before the first DP layer [36]. As DP layers randomly drop a fixed proportion of the trainable parameters during training, NNs with DP fit their training set less well (see NN CI+DN row of Figure 5). However, they improve generalization in difficult cases (e.g., between cold aquaplanet and Earth-like simulations) and do not overly deteriorate generalization in cases where the input transformations work particularly well (e.g., from cold to warm aquaplanet). Our results suggest that combining physics-guided generalization methods (e.g., physical transformation of the inputs/outputs) with regularization methods (e.g., DP) is advantageous and deserves further investigation. After analyzing the overall MSE during training, we now turn to the spatial characteristics of our ML models’ skill after training.

Global Performance after Training

In this section, we first compare the spatial characteristics of the brute force and climate-invariant NNs’ skill across climates of different temperatures to further establish the advantages of our climate-invariant transformation. These advantages are clearly visible in Figure 6, where the “raw-data” models struggle to generalize to the warm Tropics for all simulations despite fitting the cold training set well (Figure 6a). We can trace these generalization errors to warm temperature and moist atmospheric conditions the NNs were not exposed to during training, visible when comparing Figure 6a with Figure 2a. In contrast, the climate-invariant models fit the warm climate almost as well as the cold climate they were trained in (Figure 6b). Note that Figure 6 focuses on the horizontal map of a single output, i.e., the total subgrid heating at 500hPa, but that the horizontal map of other outputs, such as the near-surface subgrid heating (see Figure S9) all exhibit the same pattern of “raw-data” models failing in the warm Tropics and the climate-invariant models mostly correcting these generalization errors. Finally, the spatial distribution of the skill in the training set (e.g., middle column of Figure S8) is reassuringly consistent with the skill map of highly tuned NNs trained in similar conditions (e.g., [52]). This confirms that the “raw-data” models, representative of state-of-the-art ML subgrid closures, fail to generalize. This failure is confirmed in the latitude-pressure map of the subgrid heating at all vertical levels shown in Figure S8 and discussed in SM D3.

Maps of the coefficient of determination $R^2$ for SPCAM3, SPCAM5, and SAM under raw-data and climate-invariant conditions

Figure S9. Climate-invariant NNs mitigate generalization issues in the “Warm Tropics” for near-surface subgrid heating. Same as Figure 6 for near-surface subgrid heating.

To fully compare ML models across climate and configurations, we evaluate their overall MSEs in the training, validation, and both generalization test sets in Figure 5. In addition to the MSE of minimal validation loss, we show the MSE averaged over the 5 epochs of minimal validation loss in parentheses to confirm that our models have converged. Consistent with Figure S7’s learning curves, climate-invariant NNs with DP and BN layers often demonstrate the highest level of generalizability (two rightmost columns of each row’s NN CI+DN models). While they fit their training sets less well, “raw-data” MLRs generalize better than “raw-data” NNs because they have fewer trainable parameters (see MLR RD and NN RD models). In Figure 5, we also show that while DP and BN layers generally increase the generalization performance of “raw-data” NNs (NN RD+DN models), we can systematically improve these standard ML regularization methods by combining them with input transformations (NN CI+DN models).

Understanding Climate Invariance

To interpret our NN results, we use a game theory-based explainable ML approach, called SHAP [86, 87], to dissect climate-invariant mappings and provide insight into why they generalize better across climates and configurations. Note that MLRs are interpretable by construction, and we can draw preliminary conclusions by visualizing MLR weights without the need for explainable ML libraries (see SM D4). Similarly, we can directly visualize the NNs’ linear responses [80, 88, 89, 90] by calculating their Jacobians (gradients) via automatic differentiation [49]. However, the difference between RD and CI MLRs is small and the Jacobians [88] cannot always be reliably used to explain nonlinear NN predictions as they are first-order derivatives calculated over a sample [91].

Therefore, as climate-invariant NNs have shown superior generalizability from cold to warm climates (see NN CI errors in Figure 5), we employ SHAP’s Kernel Explainer to elucidate the NNs’ generalizability. We choose this attribution method for its versatility, as it can be used for any multi-input/output ML model. SHAP estimates the impact of a particular input value xix_i on each output yjy_j of our model. It is designed with a local accuracy property, ensuring that the sum of the effects of individual inputs equals the difference be-

tween yjy_j and its average value in the training set:

∑iSHAP⁡(xi,yj)≔yj′(1)\sum_i \operatorname{SHAP}(x_i,y_j) \coloneqq y_j' \tag*{(1)}

where we have introduced the deviation yj′y_j', defined as the difference between yjy_j and its training set average yj′≔yj−⟨yi⟩Ey_j' \coloneqq y_j-\langle y_i\rangle_{\mathcal{E}}. We use these “Shapley values” to build a nonlinear feature matrix M\mathcal{M} capturing the influence of an input xix_i on an output yjy_j:

Mij≔⟨sign⁡(xi′)×SHAP⁡(xi,yj)⟩E(2)\mathcal{M}_{ij} \coloneqq\left\langle\operatorname{sign}(x_i') \times\operatorname{SHAP}(x_i,y_j)\right\rangle_{\mathcal{E}} \tag*{(2)}

where we use the sign of the input deviation xi′x_i' to make Mij\mathcal{M}_{ij} positive if xi′x_i' and yj′y_j' have the same sign, e.g., if a positive input deviation leads to a positive output deviation. In the particular case of the MLR defined in SM Eq 25, the nonlinear feature matrix becomes the regression weight matrix multiplied by the absolute value of the input deviations: Mij=⟨Aij∣xi′∣⟩E\mathcal{M}_{ij}=\langle A_{ij}|x_i'|\rangle_{\mathcal{E}}, confirming that the feature matrix M\mathcal{M} is a nonlinear extension of the Jacobian (AA in the MLR case).

In Figure 7, each panel depicts the SHAP feature matrix M\mathcal{M} for a given model: raw-data (a) and climate-invariant (b). Each model’s inputs (e.g., specific humidity qq and temperature TT) are organized on the x-axis from the surface to the top of the atmosphere. Each model’s outputs (subgrid moistening q˙\dot{q} and subgrid heating T˙\dot{T}; see Figure S14/15 for subgrid longwave heating lw and subgrid shortwave heating sw) are organized on the y-axis, from the surface to the top of the atmosphere. Following a horizontal line shows how different inputs contribute to a given output, while following a vertical line shows how a given input influences different outputs.

SHAP feature importance matrices for raw-data and climate-invariant neural nets

Figure S14. SHAP feature importance matrix for the (a) raw-data and (b) climate-invariant neural nets trained in the cold (-4K) aquaplanet simulation (left), the warm (+4) warm aquaplanet simulation (middle), and their difference (right). To calculate these matrices, we sample inputs from the (+4K) warm aquaplanet simulation for all ML models to facilitate inter-model comparison. The x-axes indicate the vertical levels of the inputs, from the surface (left, 10310^{3}hPa) to the top of the atmosphere (right, 0hPa), while the y-axes indicate the vertical levels of the outputs, from the surface (bottom, 10310^{3}hPa) to the top of the atmosphere (top, 0hPa). We additionally indicate the 200hPa vertical level with dotted black lines.

Figure 7 contains a wealth of information about subgrid closures trained in aquaplanet simulations; we focus here on visualizing how the climate-invariant NNs (b) operate in ways that generalize better than their raw-data mapping counterpart (a). Consider the row for subgrid heating T˙\dot{T}. In the raw-data case (a), M\mathcal{M} has large coefficients in most of the troposphere (in the entire square below the dashed lines depicting the approximate tropopause level). This means that specific humidity and temperature deviations at all levels impact subgrid heating at a given level, i.e. there are large non-local relations in the vertical. Some non-local relations are physically plausible for convection, since buoyant plumes tend to rise from the surface,

and indeed near-surface TT and qq influence T˙\dot{T} through the entire troposphere. However, in this model, moisture at higher altitudes appears to influence q˙\dot{q} at lower altitudes, raising suspicions that some of the raw-data NN’s non-localities are not causal, but rather due to high auto-correlations within the input’s vertical profile, as [49] showed could happen. Temperature variations are observed to have strong vertical correlations [92] in part because of deep convective effects. Because temperature affects the saturation threshold for moisture, the RD NN will have to correctly capture the effects of both temperature and moisture wherever either has influence. In contrast, in the BplumeB_{\mathrm{plume}}-RH climate-invariant case (b), leading non-local effects between the boundary layer and the free troposphere have already been taken into account in the buoyancy formulation, and the temperature-dependent saturation threshold is built into RH. Thus, M\mathcal{M} for T˙\dot{T} tends to be concentrated near the red diagonal, meaning that positive deviations of plume buoyancy and relative humidity increase subgrid heating near the same vertical level. The use of domain knowledge has effectively reduced the effects that must be estimated by the NN for the climate-invariant models. This tends to yield differences between the models trained in the cold and the warm climates that are much smaller than for the raw-data models (see last column of Figure S15).

SHAP feature importance profiles for raw-data and climate-invariant mappings

Figure S15. Same as Figure S14, but for the four scalar inputs used in addition to the temperature and specific humidity inputs. The scalar inputs are surface pressure (psp_s, purple line), solar insolation (S0S_0, yellow line), surface sensible heat flux (SHF, green line), and surface latent heat flux (LHF, blue line). For the climate-invariant mapping (b), LHF is transformed to LHFΔq\mathrm{LHF}_{\Delta q} as described in the “Theory” section, which in conjunction with the temperature and humidity transformations, changes the SHAP feature importance matrix for all input variables.

Advantages of Climate Invariance with Multi-Climate Data

It is natural to wonder if the benefits of climate invariance carry over to training scenarios that entail data from multiple simulations spanning diverse temperatures. In contrast, until now, we have only trained ML models on single climate simulations. To find out, we examine the benefits of our climate invariant transformation approach under the ideal scenario where we have access to data from multiple climates. We conduct experiments where we train NNs on both the cold (-4K) and warm (+4K) aquaplanet simulations. We progressively increase the amount of training data to assess data efficiency. Throughout the experiment, we use eight batches and systematically increase the batch size by powers of 2, starting with a batch size of 4. Note that we obtain similar results when increasing the number of batches while keeping the batch size fixed (not shown). To obtain well-defined uncertainty es-

t imates, we employ a 10-fold cross-validation procedure via random sampling without replacement. Our findings, depicted in Figure 8, demonstrate that CI NNs (1) consistently outperform RD NNs, particularly in data-rich scenarios; and (2) exhibit lower sensitivity to the training data partition, resulting in more reliable offline performance with reduced variability in test errors. This confirms that our climate-invariant mapping enhances data efficiency, performance, and fit reproducibility across different climates, even when training data from multiple climates are available.

Discussion

In the context of climate change, we hypothesized that ML models emulating climate-invariant mappings (Figure 1), for which the inputs/outputs distributions change little across climates (Figure 3), generalize much better than ML models emulating raw-data mappings, for which the inputs/outputs distributions change substantially across climates. Tested on a suite of storm-resolving atmospheric simulations with different surface temperatures in three atmospheric models with distinct configurations (Figure 2), physically-transformed NNs generalize better as their inputs are progressively transformed (Figure 4). Climate-invariant NNs whose inputs have all been transformed learn mappings that are robust to temperature and configuration changes (Figure 5), and hence exhibit superior generalization skill almost everywhere on the globe (Figure 6), including when data from multiple climates are available (Figure 8). Finally, attribution maps reveal that in addition to providing control on the features’ distributions, climate-invariant NNs learn more spatially local mappings that facilitate generalization across climates and configurations (Figure 7).

From a computational perspective, incorporating physical knowledge, here of climate change, into an ML framework to improve its generalization skill is a successful example of using domain knowledge to extract more informative predictors, informally referred to as “feature engineering” (e.g., [93]). This also aids interpretability of the mapping. From a climate science perspective, requiring that a nonlinear statistical model of the atmosphere generalize across climate is a stringent test that helped us discover new mappings. This climate-invariant mapping is more robust to climate and configuration changes, and is more advantageous than directly using model and observational outputs (e.g., specific humidity, temperature), even when data are available in various climate regimes. In the particular case of subgrid thermodynamics, our generalization results suggest the possibility of NN-powered closures that could work in Earth-like settings, even in vastly different climate conditions. Finally, the attribution maps suggest the possibility of new analytic representations of convection from data, facilitated by the more local climate-invariant representation of subgrid thermodynamics. Our strategy paves the way for the successful use of machine learning models for climate change studies.

Materials and Methods

This section outlines how to find feature transformations yielding climate invariance. Figure 9 illustrates our proposed workflow for finding robust inputs/outputs transformations that transform the initial “raw-data” mapping into a climate-invariant mapping when combined. Note that this workflow assumes that we cannot or do not want to retrain ML algorithms in the target climate, which excludes automatically finding a transformation by training a model. This limitation could arise because the data in the target climate are insufficient or less reliable, or because we seek to uncover new physical relations that hold across an even wider range of climates.

The first step is to propose a physical transformation to implement. We can do this through knowledge of robust physical or statistical relations that link and/or preserve distributions (e.g., state equations, self-similarities, conservation laws, accurate empirical relations, etc) as modeled in SM B2. These relations help derive invariants (e.g., [94]) under a change in thermodynamic conditions. Before taking the time to implement this transformation in the ML workflow, we can verify that the PDFs of the transformed inputs/outputs (approximately) match in the training and target climates. Ideally, the joint PDFs of the transformed inputs/outputs would match. In practice, because it is easier to transform one variable at a time and the data are often insufficient in the target climate, we can fall back on the necessary (but not sufficient) condition that the univariate PDFs of the transformed inputs/outputs must match in the training and target climates. Mathematically, this match can be quantified using PDF distance metrics.

An additional challenge is that the original and transformed variables may have different units and range, meaning that any nonlinear distance metric will complicate the PDF comparison. To address this, we normalize the PDFs and their support variables XX so that the PDFs’ domains strictly lie within [0,1][0,1]. For a given variable, we use the same normalization factors across climates:

Xnorm=defX−min⁡clXmax⁡clX−min⁡clX,(3)X_{\mathrm{norm}} \overset{\mathrm{def}}{=} \frac{X-\min_{\mathrm{cl}} X}{\max_{\mathrm{cl}} X-\min_{\mathrm{cl}} X}, \tag*{(3)}
PDFnorm=defPDF∫01dXnorm×PDF(Xnorm),(4)\mathrm{PDF}_{\mathrm{norm}} \overset{\mathrm{def}}{=} \frac{\mathrm{PDF}}{\int_{0}^{1} \mathrm{d}X_{\mathrm{norm}}\times\mathrm{PDF}(X_{\mathrm{norm}})}, \tag*{(4)}

where PDFnorm\mathrm{PDF}_{\mathrm{norm}} is the transformed PDF and XnormX_{\mathrm{norm}} is its transformed support, and max⁡cl\max_{\mathrm{cl}} and min⁡cl\min_{\mathrm{cl}} respectively refer to the maximum and minimum operators over the variables’ domains and across climates, i.e. over the (−4K)(-4\mathrm{K}), (+0K)(+0\mathrm{K}), and (+4K)(+4\mathrm{K}) simulations.

Once the PDFs of each variable are normalized, we may pick any informative PDF distance metric to quantify how PDFs match across climates. Here, we pick the commonly-used Hellinger distance between two PDFs pp and qq, formally defined [95] as:

H(p,q)=def∫01dx2[p(x)−q(x)]2.(5)\mathcal{H}(p,q) \overset{\mathrm{def}}{=} \sqrt{\int_{0}^{1}\frac{\mathrm{d}x}{2}\left[\sqrt{p(x)}-\sqrt{q(x)}\right]^{2}}. \tag*{(5)}

This distance is symmetric (i.e., the arguments’ order does not affect the outcome) and easy to interpret: H(p,q)\mathcal{H}(p,q) is bounded by 0 (when p=qp=q) and 100% (when pp is zero whenever qq is positive and vice-versa). In SM D1, we show that our results using the Hellinger distance (see Tab S1) are consistent with those using the Jensen-Shannon distance [96] (see Tab S3), a PDF distance metric giving large weights to the PDFs’ tails that tend to be particularly problematic for generalization purposes.

Once the univariate PDFs of the physically transformed variables match across climates, the second step is to train two inexpensive or “baseline” models on the reference climate to quickly check whether the transformation improves an ML model’s generalization ability: (1) A “raw-data” model without the transformation; and (2) a “climate-invariant” model with the transformation. If the transformation does not improve the baseline model’s generalization abilities (i.e., (2) performs worse than (1) in the target

climate), then the transformation may not be appropriate. Note that we trained MLR baselines to create climate-invariant NNs, but the ML model used to define the baseline should be tailored to the desired final ML model.

If the transformation improves the inexpensive baseline model’s performances, the last step is to train the “raw-data” and “climate-invariant” versions of the desired ML model (usually nonlinear) on the reference climate. If the physical transformation improves the desired ML model’s generalization abilities (i.e., the climate-invariant model beats the raw-data model in the target climate using the same performance metric calculated over a validation set), then we may keep the transformation. This workflow may be repeated for the ML model’s additional input/output variables until the emulated mapping is as climate-invariant as possible.

Before applying this workflow to subgrid thermodynamics closures, we underline one of its key challenges: Because some transformations are much more impactful than others, it is often not possible to develop each physical transformation independently. In our case, the specific humidity inputs vary the most across climates, meaning that transforming specific humidity affects the model’s generalization abilities the most. As a result, initial experiments that independently tested the effect of transforming temperature suggested a negative impact of temperature transformation on generalization ability (not shown). This initial result was later invalidated by experiments that jointly transformed specific humidity and temperature. Following this, we adopt a progressive input transformation approach, where the most important inputs are transformed first: Specific humidity, then temperature, and finally surface energy fluxes.

Acknowledgments

We thank William Boos for providing the SAM hy-drostatic simulation output, Joyce Meyerson for helping us improve figures, and Andrea Jenney, Imme Ebert-Uphoff, and the CBRAIN discussion group for help and advice that greatly benefited the present manuscript.

Funding: TB and MP acknowledge funding from NSF grants OAC1835863 and AGS-1734164. PG,

MP, SY, and JL acknowledge NSF funding from the LEAP Science and Technology center grant (2019625-STC). MP acknowledges the DOE’s Exascale Computing Project (17-SC-20-SC), Biological and Environmental Research (BER) program (DE-SC0022331), Advanced Scientific Computing Research (ASCR) program (DE-SC0022255). MP and SY also acknowledge the DOE EAGLES project (74358). JY acknowledges support from the EAPS Houghton-Lorenz postdoctoral fellowship. PG thanks NSF grant AGS-1734164 and USMILE European Research Council grant. PG, POG, and JY received M²LInES research funding by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program. FA and JDN acknowledge NSF AGS-1936810; FA acknowledges NSF AGS-2225956. We also thank the Extreme Science and Engineering Discovery Environment supported by NSF grant number ACI-1548562 (charge numbers TG-ATM190002 and TG-ATM170029), the Texas Advanced Computing Center (TACC; ATM-20009), and HPC support from Cheyenne (doi:10.5065/D6RX99HX) provided by NCAR’s Computational and Information Systems Laboratory (sponsored by the NSF) for computational resources.

Contributions: Conceptualization: TB, MP, SR, and PG. Methodology: All authors. Investigation: TB, MP, JY, AG, LP, JL, and SY. Visualization: TB, MP, PG, DN, and JY. Supervision: TB, MP, and PG. Writing—original draft: TB. Writing—review & editing: All authors.

Competing interests: The authors declare that they have no competing interests.

Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. See SM A for details about code and data availability.

Supplementary materials

Supplementary Text (SM A to SM D)

Figs. S1 to S15

Tables S1 to S3

Supplementary Materials for Climate-Invariant Machine Learning

Tom Beucler; Pierre Gentine; Janni Yuval; Ankitesh Gupta; Liran Peng; Jerry Lin; Sungduk Yu; Stephan Rasp; Fiaz Ahmed; Paul A. O’Gorman; J. David Neelin; Nicholas J. Lutsko; Michael Pritchard

This PDF file includes:

Supplementary Text (SM A to SM D)

Figs. S1 to S15

Tables S1 to S3

The Supplementary Materials (SM) is organized as follows: First, we discuss code and data availability (SM A), including links to multiple repositories to reproduce the different ML-based closures and climate simulations discussed in the manuscript. Then, we present the characteristics of the emulated mapping (SM B1), derive the input transformations used in the manuscript (SM B2), the output transformations tested in this SM (SM B3), and discuss possible vertical coordinate transformations (SM B4). We provide a guide to find new climate-invariant transformations in SM B5. SM C details the practical implementation of our climate-invariant ML workflow. Finally, we present supplementary results in SM D, including the Hellinger and Jensen-Shannon distances between input distributions (SM D1), the learning curves of climate-invariant models across climates and geographies (SM D2), the generalization skill of climate-invariant NNs near the surface (SM D3), and three methods to visualize the “raw-data” and climate-invariant mappings and compare them in the cold (-4K) and warm (+4K) climates (SM D4).

Code and Data Availability

The code used to process data, train models, and produce this manuscript’s figure can be found in the following Github repository: https://github.com/tbeucler/CBRAIN-CAM, which is archived using Zenodo https://zenodo.org/record/8140413 [19]. This repository includes a minimal reproducible example on how to train a climate-invariant neural network and verify its improved generalization ability: https://colab.research.google.com/github/tbeucler/CBRAIN-CAM/blob/master/Climate_Invariant_Guide.ipynb and a notebook to generate all figures in this manuscript requiring model data https://github.com/tbeucler/CBRAIN-CAM/blob/master/notebooks/tbeucler_devlog/090_Climate_Invariant_Paper_Figures_v2.ipynb.

Both scripts rely on the manuscript’s accompanying data, archived in the following Zenodo repository: https://doi.org/10.5281/zenodo.8140536 [20].

The above Github repository is forked from (and builds upon) Stephan Rasp’s CBRAIN repository https://github.com/raspstephan/CBRAIN-CAM, also archived using Zenodo https://zenodo.org/record/1402384#.YaJsg9BKiuk [21]. This repository contains a quickstart guide https://github.com/raspstephan/CBRAIN-CAM/blob/master/quickstart.ipynb to preprocess raw climate model output, train a neural network and benchmark it.

As described in § 2, we use data from eight climate simulations using three climate models (SP-CAM3, SPCESM2, and SAM) to form our training, validation, and test sets. We report the exact characteristics of the splits in Table S2 and information to re-generate the full simulation output below.

ModelSpatiotemporal ResolutionTraining SetValidation SetTest Set
SPCAM3(2.8∘×2.8∘)T42×30lev×30min(2.8^\circ\times2.8^\circ)_{\mathrm{T42}} \times30\mathrm{lev} \times30\mathrm{min}Yr2, Mo1-4 →\to 47MYr2, Mo5-8 →\to 48MYr1, Mo6-9 →\to 48M
SPCESM22.5∘×1.9∘×30lev×15min2.5^\circ\times1.9^\circ\times30\mathrm{lev} \times15\mathrm{min}Yr1, Day0-9/Mo →\to 143MYr2, Day0-9/Mo →\to 143MYr2, Day20-28/Mo →\to 118M
SAM (-4K)96km×96km×48lev×180min96\mathrm{km} \times96\mathrm{km} \times48\mathrm{lev} \times180\mathrm{min}day 225-545 →\to 13.8Mday 545-562 →\to 0.7Mday 562-587 →\to 2.6M
(+0K)96km×96km×48lev×180min96\mathrm{km} \times96\mathrm{km} \times48\mathrm{lev} \times180\mathrm{min}--day 380-405 →\to 2.6M

Table S2. Characteristics of the training/validation/test sets used in this manuscript. The spatiotemporal resolution uses the format longitude ×\times latitude ×\times vertical levels ×\times time. For SPCAM3, which uses a T42 spectral truncation, we use months 1 to 4 of the second simulation year to build the training set, resulting in ≈47M\approx47M samples. For SPCESM2, we use the first 9 days of every month of the first simulation year to build the training set, resulting in ≈143M\approx143M samples.

SPCAM3 The codebase for running the “SPCAM3” simulation is the same employed by [2], which is archived at https://gitlab.com/mspritch/spcam3.0-neural-net/-/tree/sp-diagnostic for the (+0K) simulation. The sea surface temperature is uniformly cooled by 4K to produce the (-4K) simulation and uniformly warmed by 4K to produce the (+4K) simulation. Raw output of the (+0K) simulation can be found at https://zenodo.org/record/1402384#.YaUCsdDMI-w [21]. The full simulations output, which is several TB, is archived on the GreenPlanet cluster at UC Irvine and available upon request.

SPCESM2 The codebase for running the “SPCESM2” simulations is the same employed by [22], which is archived at https://github.com/mspritch/UltraCAM-spcam2_0_cesm1_1_1; this code was in turn forked from a development version of the CESM1.1.1 located on the NCAR central subversion repository under tag spcam_cam5_2_00_forCESM1_1_1Rel_V09, which dates to February 25, 2013. The full simulations output, which is several TB, is also archived on the GreenPlanet cluster at UC Irvine and available upon request. We additionally archived the input data and run scripts necesary to re-run all three simulations as part of the manuscript’s accompanying data using Zenodo https://zenodo.org/record/5775541#.YbeMHNDMKUl [20].

SAM The codebase for running the “SAM” simulations is the same employed by [8]. The initial sounding, meridional surface temperature profile, and source code to re-run the simulation can be found at https://zenodo.org/record/4118346#.YaT_WtBKg-w [23]. To produce the (+0K) simulation, the initial sounding and surface temperature profiles are both uniformly warmed by 4K. The output from the SAM simulations, which is several TB, is archived at MIT and is available upon request.

Feature Transformations

Mapping

In this subsection, we present the characteristics of the emulated mapping, highlighting the differences between the superparameterized models (SPCAM3, SPCESM2) and the storm-resolving model (SAM). In both cases, the input vector xx encodes the large-scale (≈100 km\approx100\,\mathrm{km}) climate state:

x={[q(p)  T(p)  ps  S0  SHF  LHF]T(SPCAM3, SPCESM2)[qnp(p)  T(p)  dEquator]T(SAM)(6)x = \begin{cases} \left[ q(p)\ \ T(p)\ \ p_s\ \ S_0\ \ \mathrm{SHF}\ \ \mathrm{LHF} \right]^T & \text{(SPCAM3, SPCESM2)} \\ \left[ q_{\mathrm{np}}(p)\ \ T(p)\ \ d_{\mathrm{Equator}} \right]^T & \text{(SAM)} \end{cases} \tag*{(6)}

where q(p)q(p) is the vertical profile of specific humidity in units of kg/kg\mathrm{kg}/\mathrm{kg} (written as a function of the background pressure coordinate pp in units of Pa), T(p)T(p) is the temperature’s vertical profile in units of K, psp_s is surface pressure in units Pa, S0S_0 is solar insolation in units of W/m2\mathrm{W}/\mathrm{m}^2, SHF is surface sensible heat flux in units W/m2\mathrm{W}/\mathrm{m}^2, LHF is surface latent heat flux in units of W/m2\mathrm{W}/\mathrm{m}^2, qnp(p)q_{\mathrm{np}}(p) is the vertical profile of non-precipitating water concentration in units of kg/kg\mathrm{kg}/\mathrm{kg}, and dEquatord_{\mathrm{Equator}} is the distance to the Equator, which is used as a proxy for solar insolation in the mapping learned for SAM. The output vector yy groups subgrid-scale thermodynamic tendencies:

y={[Lvq˙(p)  cpT˙(p)  cplw(p)  cpsw(p)]T(SPCAM3, SPCESM2)[Lvq˙np(p)  H˙L(p)]T(SAM)(7)y = \begin{cases} \left[ L_v\dot{q}(p)\ \ c_p\dot{T}(p)\ \ c_p\mathrm{lw}(p)\ \ c_p\mathrm{sw}(p) \right]^T & \text{(SPCAM3, SPCESM2)} \\ \left[ L_v\dot{q}_{\mathrm{np}}(p)\ \ \dot{H}_L(p) \right]^T & \text{(SAM)} \end{cases} \tag*{(7)}

where LvL_v in units of J kg−1\mathrm{J\,kg}^{-1} is the latent heat of vaporization of water in standard conditions, q˙(p)\dot{q}(p) is the subgrid moistening vertical profile, cpc_p in units of J kg−1 K−1\mathrm{J\,kg}^{-1}\,\mathrm{K}^{-1} is the specific heat of dry air at constant pressure in standard atmospheric conditions, T˙(p)\dot{T}(p) is the total subgrid heating vertical profile (including subgrid radiation), lw(p)\mathrm{lw}(p) is the subgrid longwave radiative heating vertical profile, sw(p)\mathrm{sw}(p) is the subgrid shortwave radiative heating vertical profile, q˙np(p)\dot{q}_{\mathrm{np}}(p) is the non-precipitating water condensation vertical profile, and H˙L(p)\dot{H}_{L}(p) is the subgrid time-tendency of the liquid/ice static energy vertical profile. Following [10], all components of the output vector y\mathbf{y} are mass-weighted and vertically integrated within each vertical layer to yield energy flux units (W/m2^{2}). Assuming vertical profiles have NpN_{p} vertical levels, x\mathbf{x} is of length (2Np+4)(2N_{p}+4) for SPCAM3 and SPCESM2, and of length (2Np+1)(2N_{p}+1) for SAM. y\mathbf{y} is of length 4Np4N_{p} for SPCAM3 and SPCESM2, and of length 2Np2N_{p} for SAM.

Physically-Based Input Transformations

In Figure S2, we compare three transformation options for each input, whose univariate PDFs are depicted in Figure S2: No transformation (top), our most successful transformation (bottom), and our second best transformation (middle). After defining our second best input transformations (B2a), we delve into the details of our relative humidity (B2b) and plume buoyancy (B2c) transformations before discussing the other inputs’ distribution shift (B2d).

Univariate PDFs of specific humidity, temperature, and latent heat flux transformations

Figure S2. Figure S2: Univariate PDFs of the (a) 600hPa specific humidity, (b) 850hPa temperature, and (c) latent heat flux in the cold (blue), reference (gray), and warm (red) simulations of each model (SPCAM3, SPCESM2, and SAM). For each variable, we also show the PDFs of the two transformations discussed in SM SB.2. From top to bottom, the variables are qq (g/kg), qdeficitq_{\mathrm{deficit}} (g/kg), RH, TT (K), Tfrom NST_{\mathrm{from\ NS}} (K), BplumeB_{\mathrm{plume}} (m/s2^2), LHF (W/m2^2), LHFq\mathrm{LHF}_{q} (kg m−2^{-2}s−1^{-1}), and LHFΔq\mathrm{LHF}_{\Delta q} (kg m−2^{-2}s−1^{-1}). For a given variable and transformation, we use the same vertical logarithmic scale across models. Note that unlike for qq, the best options for TT and LHF do not decrease distribution distance more than the second best options, which is discussed in text.

Second Best Input Transformations

Along the way to our optimal feature transformations, we explored candidate options that proved second best. For completeness these are reviewed first in the SM.

Saturation Deficit: We explored saturation deficit but found it did not lead to climate invariance. Similar to qq, saturation deficit (Figure S2, middle) still has a corresponding expansion of the PDF with warming as a result of the Clausius-Clapeyron relation. It is defined as the amount by which the water vapor concentration must be increased to achieve saturation without changing the environmental temperature and pressure:

q~deficit(T,p)=defqsat(T,p)−q,(8)\widetilde{q}_{\mathrm{deficit}}(T,p) \mathrel{\mathop{=}^{\mathrm{def}}} q_{\mathrm{sat}}(T,p)-q, \tag*{(8)}

where qsat(T,p)q_{\mathrm{sat}}(T,p) is the saturation specific humidity.

In contrast, the relative humidity transformation q~RH(p)=defRH(q,T,p)\widetilde{q}_{\mathrm{RH}}(p) \mathrel{\mathop{=}^{\mathrm{def}}} \mathrm{RH}(q,T,p) (Figure S2, bottom) results in a climate-invariant PDF, as evidenced by PDFs that mostly overlap across all three climates.

Temperature minus Near-Surface Temperature: Assuming that the temperature’s PDF shift with warming is almost uniform with height, we can derive an approximate invariant by subtracting the temperature at all levels T(p)T(p) from the near-surface temperature T(pNS)T(p_{\mathrm{NS}}):

T~from NS(p)=defT(pNS)−T(p),(9)\widetilde{T}_{\mathrm{from\ NS}}(p) \mathrel{\mathop{=}^{\mathrm{def}}} T(p_{\mathrm{NS}})-T(p), \tag*{(9)}

where pNSp_{\mathrm{NS}} is the lowest atmospheric pressure level see (Figure S2, middle). However, this linear transformation fails in the upper atmosphere, especially near the tropopause where temperatures are approximately invariant with warming [14, 15, 16] and therefore decoupled from surface temperature changes. This is why the buoyancy of a moist static energy-conserving plume:

T~buoyancy(p)=defBplume(qNS,T,p),(10)\widetilde{T}_{\mathrm{buoyancy}}(p) \mathrel{\mathop{=}^{\mathrm{def}}} B_{\mathrm{plume}}(q_{\mathrm{NS}},T,p), \tag*{(10)}

where qNS=q(pNS)q_{\mathrm{NS}}=q(p_{\mathrm{NS}}) is the near-surface specific humidity, yields approximate climate invariance (Figure S2, bottom), unlike the temperature minus near surface temperature transformation.

Scaling Latent Heat Fluxes by Near-Surface Specific Humidity: To address the increase of LHF with warming, we scale LHF by near-surface specific humidity qq (pNSp_{\mathrm{NS}}) (Figure S2, middle):

LHF~q≔defLHFLvmax⁡{ϵq,q(pNS)},(11)\widetilde{\mathrm{LHF}}_{q} \mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \frac{\mathrm{LHF}}{L_v \max\{\epsilon_q,q(p_{\mathrm{NS}})\}}, \tag*{(11)}

where ϵq\epsilon_q is a user-chosen parameter that we set to 10−410^{-4} to avoid division by zero. While better than directly using LHF, this transformation fails for very dry atmospheres when the latent heat flux is negative, e.g. in polar oceans where atmospheric water vapor may be condensing on the surface, or when the near-surface specific humidity is very small, e.g. in subtropical regions. This is why scaling LHF using the near-surface saturation deficit (Figure S2, bottom):

LHF~Δq≔defLHFLvmax⁡{ϵq,qsat[T(pNS),pNS]−q(pNS)}.(12)\widetilde{\mathrm{LHF}}_{\Delta q} \mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \frac{\mathrm{LHF}}{L_v \max\{\epsilon_q,q_{\mathrm{sat}}[T(p_{\mathrm{NS}}),p_{\mathrm{NS}}]-q(p_{\mathrm{NS}})\}}. \tag*{(12)}

yields better generalizability. While both the Jensen-Shannon and Hellinger distances would suggest that LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q} is a slightly less good transformation than LHF~q\widetilde{\mathrm{LHF}}_q, the LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q} transformation leads to improved generalization performance compared to LHF~q\widetilde{\mathrm{LHF}}_q (not shown). This confirms that only considering the PDF distances is not always sufficient to find the optimal transformations (discussed in SM B4).

Relative Humidity

Relative humidity (RH) provides our optimal transformation for the specific humidity inputs. RH is defined as the ratio of the partial pressure of water vapor e(p,q)e(p,q) to its saturation value esat(T)e_{\mathrm{sat}}(T), and can be expressed analytically:

RH≔defe(p,q)esat(T)≈q≪1RvRdpqesat(T),(13)\mathrm{RH} \mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \frac{e(p,q)}{e_{\mathrm{sat}}(T)} \mathrel{\mathop{\approx}^{q\ll1}} \frac{R_v}{R_d}\frac{pq}{e_{\mathrm{sat}}(T)}, \tag*{(13)}

where Rv≈461 J kg−1 K−1R_v \approx461\,\mathrm{J\,kg^{-1}\,K^{-1}} is the specific gas constant for water vapor, Rd≈287 J kg−1 K−1R_d \approx287\,\mathrm{J\,kg^{-1}\,K^{-1}} is the specific gas constant for dry air, pp (in units Pa) is the total atmospheric pressure, qq (in units kg/kg) is specific humidity, and esat(T)e_{\mathrm{sat}}(T) (in units Pa) is the saturation pressure of water vapor, whose analytic expression in our case is given below. Consistent with Eq (13), the saturation specific humidity qsatq_{\mathrm{sat}} corresponding to RH =1=1, is

qsat(T,p)=Rdesat(T)Rvp.(14)q_{\mathrm{sat}}(T,p)=\frac{R_d e_{\mathrm{sat}}(T)}{R_v p}. \tag*{(14)}

SAM’s single-moment microphysics scheme [24], which is also used in the SPCAM3 and SPCESM2 simulations, partitions water between the liquid and ice phases using a weight ω\omega that is a linear function of the absolute temperature:

ω≔defT−T00T0−T00.(15)\omega\mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \frac{T-T_{00}}{T_0-T_{00}}. \tag*{(15)}

Under the assumptions of this microphysics scheme, the saturation pressure of water vapor can then be found by integrating the Clausius-Clapeyron equation with respect to temperature, expressed analytically as:

esat(T)={eliq(T)T>T0eice(T)T<T00ωeliq(T)+(1−ω)eice(T)T∈[T00,T0](16)e_{\mathrm{sat}}(T)= \begin{cases} e_{\mathrm{liq}}(T) & T>T_0 \\ e_{\mathrm{ice}}(T) & T<T_{00} \\ \omega e_{\mathrm{liq}}(T)+(1-\omega)e_{\mathrm{ice}}(T) & T\in[T_{00},T_0] \end{cases} \tag*{(16)}

where T0=273.16 KT_0=273.16\,\mathrm{K} and T00=253.16 KT_{00}=253.16\,\mathrm{K}. In Eq (16), as temperature increases, the saturation pressure of water vapor goes from the saturation vapor pressure with respect to liquid eliqe_{\mathrm{liq}}, to the saturation vapor pressure with respect to ice eicee_{\mathrm{ice}}. These are given by the following polynomial approximations:

eliq(T)=100 Pa×∑i=08aliq,i[max⁡(−193.15 K,T−T0)]i,(17)e_{\mathrm{liq}}(T)=100\,\mathrm{Pa}\times\sum_{i=0}^{8}a_{\mathrm{liq},i}\left[\max(-193.15\,\mathrm{K},T-T_0)\right]^i, \tag*{(17)}

where aliq\mathbf{a}_{\mathrm{liq}} is a vector of length 9 containing nonzero polynomial coefficients. The polynomial approximation for eicee_{\mathrm{ice}}, with the same temperature switches as Eq. (16), is:

eice(T)={eliq(T)100 Pa×{cice,1+C(T)[cice,4+cice,5C(T)]}100 Pa×∑i=08aice,i(T−T0)i(18)e_{\mathrm{ice}}(T) = \begin{cases} e_{\mathrm{liq}}(T) \\ 100\,\mathrm{Pa} \times\{c_{\mathrm{ice},1} + \mathcal{C}(T)[c_{\mathrm{ice},4} + c_{\mathrm{ice},5}\mathcal{C}(T)]\} \\ 100\,\mathrm{Pa} \times\sum_{i=0}^{8} a_{\mathrm{ice},i}(T-T_0)^i \end{cases} \tag*{(18)}

where C(T)\mathcal{C}(T) is a ramp function of temperature given by:

C(T)=defmax⁡(cice,2,T−T0),(19)\mathcal{C}(T) \mathrel{\mathop{=}^{\mathrm{def}}} \max(c_{\mathrm{ice},2},T-T_0), \tag*{(19)}

and (aice,cice)(\mathbf{a}_{\mathrm{ice}},\mathbf{c}_{\mathrm{ice}}) are vectors of length 9 and 5 containing nonzero elements, respectively. Between temperatures of T00T_{00} and T0T_0, the saturation pressure of water vapor is a weighted mean of eliqe_{\mathrm{liq}} and eicee_{\mathrm{ice}}. The reader interested in the numerical details of this transformation is referred to our implementation of relative humidity at https://colab.research.google.com/github/tbeucler/CBRAIN-CAM/blob/master/Climate_Invariant_Guide.ipynb.

Plume Buoyancy

Plume Buoyancy (BplumeB_{\mathrm{plume}}) is our most successful transformation for the temperature inputs. Buoyancy is defined as the upward acceleration exerted upon parcels by virtue of the density difference between the parcel and the surrounding air of the atmospheric column (e.g., [25]). Because our ML model’s inputs represent the large-scale thermodynamic state, the ML model does not have information about the storm-scale buoyancy field, and we must rely on idealized approximations to estimate the buoyancy that a plume would have for given specific humidity and temperature profiles. To be consistent with the model’s conserved quantities [24], we derive a simple buoyancy metric based on a moist static energy (hh) conserving plume below following similar derivations in [17] and [26]. We refer the reader interested in the numerical details of this transformation to https://colab.research.google.com/github/tbeucler/CBRAIN-CAM/blob/master/Climate_Invariant_Guide.ipynb.

For purposes of this transformation, we omit virtual temperature effects and condensate loading (effects of the environmental water vapor on heating/moisture sink are being estimated separately). Thus the parcel buoyancy is simply proportional to the relative difference between its temperature TparT_{\mathrm{par}} and the environmental temperature TT:

Bplume≈gTpar−TT,(20)B_{\mathrm{plume}} \approx g\frac{T_{\mathrm{par}}-T}{T}, \tag*{(20)}

where gg is the gravity constant. Further assuming that the plume is non-entraining, obeys hydrostatic balance, and lifts parcels from the near-surface, the lifted parcel’s moist static energy is conserved and equal to its near-surface value (at pressure pNSp_{\mathrm{NS}}:

hpar≈h(pNS)=defLvq(pNS)+cpT(pNS),(21)\begin{aligned} h_{\mathrm{par}} &\approx h(p_{\mathrm{NS}}) \\ &\mathrel{\mathop{=}^{\mathrm{def}}} L_vq(p_{\mathrm{NS}})+c_pT(p_{\mathrm{NS}}), \tag*{(21)} \end{aligned}

where we have used the environmental moist static energy’s definition:

h=defLvq+cpT+gz,(22)h \mathrel{\mathop{=}^{\mathrm{def}}} L_vq+c_pT+gz, \tag*{(22)}

where zz is geopotential height, and we neglected z(pNS)z(p_{\mathrm{NS}}) as the near-surface is close to the surface by definition. To express the parcel’s buoyancy as a function of the environmental thermodynamic state, we finally assume that the parcel is saturated (not necessarily true close to the surface), and that the thermodynamic differences between the parcel and the environment are small, which allows us to linearize the Clausius-Clapeyron equation about the environmental temperature:

Tpar−T≃Claus.−Clap.(∂T∂q∗)T,p(qsat,par−qsat)=RvT2Lvqsat(qsat,par−qsat)=RvT2Lv2qsat[hpar−hsat−cp(Tpar−T)].(23)\begin{aligned} T_{\mathrm{par}}-T &\mathrel{\mathop{\simeq}^{\mathrm{Claus.-Clap.}}} \left(\frac{\partial T}{\partial q^*}\right)_{T,p}(q_{\mathrm{sat,par}}-q_{\mathrm{sat}}) \\ &= \frac{R_vT^2}{L_vq_{\mathrm{sat}}}(q_{\mathrm{sat,par}}-q_{\mathrm{sat}}) \\ &= \frac{R_vT^2}{L_v^2q_{\mathrm{sat}}}\left[h_{\mathrm{par}}-h_{\mathrm{sat}}-c_p(T_{\mathrm{par}}-T)\right]. \tag*{(23)} \end{aligned}

Using Eq (23) to write Tpar−TT_{\mathrm{par}}-T as a function of the environmental thermodynamic state and substituting the resulting expression into Eq (20) yields an estimation of plume buoyancy from (q,T,p)(q,T,p):

Bplume(q,T,p)=g[hpar−hsat(q,T,p)]κ(T,p)×cpT,(24)B_{\mathrm{plume}}(q,T,p)=\frac{g\left[h_{\mathrm{par}}-h_{\mathrm{sat}}(q,T,p)\right]}{\kappa(T,p)\times c_pT}, \tag*{(24)}

where the parcel’s moist static energy is expressed as a function of near-surface (q,T)(q,T) in Eq (21), the environmental saturated moist static energy in pressure coordinates is defined as:

hsat(q,T,p)=defLvqsat(T,p)+cpT+gz(q,T,p),(25)h_{\mathrm{sat}}(q,T,p)\mathrel{\mathop{=}^{\mathrm{def}}}L_vq_{\mathrm{sat}}(T,p)+c_pT+gz(q,T,p), \tag*{(25)}

and we have introduced the dimensionless factor:

κ(T,p)=1+Lv2qsat(T,p)RvcpT2.(26)\kappa(T,p)=1+\frac{L_v^2q_{\mathrm{sat}}(T,p)}{R_vc_pT^2}. \tag*{(26)}

Note that in pressure coordinates, we calculate the geopotential height by vertically integrating the hydrostatic equation after using the ideal gas law:

z(q,T,p)=∫ppNSdp′T(p′)gp′{Rd+[Rv−Rd]q(p′)}.(27)z(q,T,p)=\int_p^{p^{\mathrm{NS}}} \mathrm{d}p'\frac{T(p')}{gp'}\left\{R_d+[R_v-R_d]q(p')\right\}. \tag*{(27)}

Sensible Heat Fluxes and Surface Pressure

In this appendix, we discuss the univariate PDFs of the two inputs we did not transform in the main manuscript (see § 3) in Figure S3 for both super-parameterized models and all three surface temperatures (-4K, +0K, and +4K). The PDF of sensible heat fluxes changes very little with warming. There is a slight expansion of the left tail of the surface pressure PDF with warming as the most extreme low-pressure systems become more intense, but we hypothesize that these changes are small enough not to require a dedicated input transformation.

Univariate PDFs of sensible heat flux and surface pressure for Aquaplanet (SPCAM3) and Earth-like (SPCESM2) simulations

Figure S3. Univariate PDFs of the sensible heat flux and surface pressure in the cold (blue), reference (gray), and warm (red) simulations of SPCAM3 and SPCESM2. For a given variable, we use the same vertical logarithmic scale across models.

Output Transformation

Theory

In contrast to input transformation, transforming our ML models’ outputs, namely subgrid thermodynamics, only marginally improves the models’ ability to generalize. In the absence of physical theory on how the full vertical profile of subgrid thermodynamics changes with warming, we place ourselves in an idealized scenario: Assuming we know how the outputs’ marginal PDF changes with warming, can we help our ML models generalize via output transformation?

We note that assuming knowledge of how the marginal (univariate) PDFs (or equivalently the CDF) of convective heating and moistening change with warming is more realistic than assuming full knowledge of how their joint PDFs change with warming. This knowledge could come from e.g. convection theory (e.g., [13] or shorter simulations than those required to train a subgrid closure. Under this assumption, a natural transformation is the outputs’ cumulative distribution function (CDF):

y~=CDF⁡(y).(28)\tilde{y} = \operatorname{CDF}(y). \tag*{(28)}

In essence, we are assuming that the mapping is more likely to be invariant in quantile than in physical space, which is a common practice when debiasing the outputs of climate models referred to as quantile mapping (e.g., review by [27]). In practice, we test two distinct methods for transforming the outputs using their CDFs and report the results for SPCAM3 aquaplanet simulations in SM B2b.

Quantile mapping after training: The first method is to transform the ML model’s input during training, and then transform the ML model’s output after training. This is akin to standard, post-hoc, quantile mapping. In the particular case of trying to generalize from a (-4K) cold simulation to a (+4K) warm simulation, the entire transformation to yield outputs in physical units can be mathematically written as:

y↦CDF⁡+4K−1[CDF⁡−4K(y)],(29)y \mapsto\operatorname{CDF}_{+4\mathrm{K}}^{-1}\left[\operatorname{CDF}_{-4\mathrm{K}}(y)\right], \tag*{(29)}

where for simplicity but without loss of generality, we have considered a singular input yy whose CDF is CDF⁡−4K\operatorname{CDF}_{-4\mathrm{K}} in the (-4K) cold simulation and CDF⁡+4K\operatorname{CDF}_{+4\mathrm{K}} in the (+4K) warm simulation.

Quantile mapping before training: The second method is to transform the ML model’s output before training. In that case, we directly train the ML model to predict y~=CDF⁡(y)\tilde{y} = \operatorname{CDF}(y) as accurately as possible. We then map the output back to physical units using CDF⁡−1\operatorname{CDF}^{-1} after training.

Results

The two methods to transform outputs presented above are depicted in Figure S4 and the generalization results presented in Figure S5. Transforming outputs after training slightly improves generalization skill (from an overall R2R^2 of 0.58 to 0.62 for the generalization (+4K) set). In contrast, transforming outputs before training leads to equally bad results both on the training and generalization sets, which is a negative result underlining the challenges of designing the appropriate loss function in probability space. A possible solution would be to convert back the outputs to physical space before feeding them to the loss function during training, and further investigation is required to fully assess the potential and limitations of training these ML models in probability space.

Two types of bias-correction methods used to transform outputs in SM B2

Figure S4. Two types of bias-correction methods used to transform outputs in SM B2. (1, dark green, “post-processing” method) Quantile mapping is typically done after training the model to bias-correct the outputs, and (2, light green, “pre-processing” method) we additionally test directly making predictions in probability space by converting the outputs to their CDF values before training. Note that this usually changes the loss function.

Coefficient of determination for four model configurations

Figure S5. Transforming outputs via quantile mapping after training slightly improves the climate invariant model’s ability to generalize from a cold to a warm climate. Coefficient of determination R2R^2 for 500-hPa subgrid heating of raw-data (a), climate-invariant (b), climate-invariant with outputs transformed after training (c), climate-invariant with outputs transformed before training (d) NNs trained using the cold (-4K) training set of SPCAM3 and calculated over the warm (+4K) training set of SPCAM3.

Spatial Coordinate Transformation

Another transformation to consider when input/output variables are functions of spatiotemporal coordinates is coordinate transformation, resulting in a coordinate change. In our specific example, it is possible to transform the vertical coordinate, i.e. the hybrid pressure coordinate pp. Possible transformations include:

  1. the temperature (p~=T\tilde{p} = T), e.g. for radiative heating, which tends to vary less in temperature coordinates [28],

  1. the saturation specific humidity (p~=qsat\tilde{p} = q_{\mathrm{sat}}) which is consistent with a transformation of the primitive equations that captures an upward shift of the circulation as the climate warms [29],

  1. the geopotential height or the altitude (p~=z\tilde{p} = z), which could more consistently capture gravity wave propagation,

  1. or a coordinate with fixed values for characteristic vertical levels in the atmosphere, such as the top of the planetary boundary layer or the tropopause [14].

While we were able to transform the vertical coordinate using interpolation functions (not shown), the benefits were not visible in our particular case. This could be because input transformation already addresses some of the upward shift of convective activity warming, as shown by the influence of humidity inputs in Figure S14.

Implementation

For reproducibility purposes [30], we now detail the practical implementation of a climate-invariant ML workflow (see Figure 9), from its overall structure (SM C1) to its benchmarking (SM C4) via the characteristics of the multiple linear regressions (MLR, SM C2) and neural networks (NN, SM C3) presented in this manuscript.

Overall Workflow

We present three ways to implement physical transformations. The first way is to physically transform the inputs/outputs before training. While this option is easiest to implement and debug, it usually comes at the cost of disk space: Every time we try a new transformation, we need to duplicate our training/validation/test datasets for all the climates/geographies we are interested in, which can quickly be prohibitive when trying multiple transformation combinations.

Therefore, it can be advantageous to transform the input/output variables within the ML framework, so that the transformations occur during training. In essence, we are trading disk space for computational time. In that spirit, the second method is to transform the inputs/outputs via custom layers (e.g., Ch 12 of [12]) in the ML algorithm itself. Since this second method tends to substantially slow down training as it adds sequential operations on the GPU, we take advantage of the fact that the transformations occur before and after the emulated mapping, and propose a third method that can happel in parallel on the CPU: Transforming inputs/outputs by customizing the pipeline or “data generator”, which is the algorithm responsible for feeding numbers to the ML model after reading the training data files. For each batch, the custom data generator then transforms inputs before feeding them to the ML algorithm. In our case, note that we transform outputs independently via quantile mapping (see SM B2).

For the rest of this manuscript, we will train our ML models using custom data generators: For “raw-data” models, the transformations are set to None (no transformation), while for “climate-invariant” models, the qq transformation is set to q~RH\tilde{q}_{\mathrm{RH}}, the TT transformation is set to T~buoyancy\widetilde{T}_{\mathrm{buoyancy}}, and the LHF transformation is set to LHF~Δq\widetilde{\mathrm{LHF}}_{\Delta q}. For all models, we additionally subtract the mean from each input before dividing it by its range to feed the ML algorithm floating-point numbers between (-1) and 1. Note that for each transformation, numbers are “de-normalized” before the transformation and “re-normalized” after following the normalization procedure described in Sec 2.4. Therefore, all transformations are done in physical units while the ML algorithm is always fed single-precision floating-point numbers in [−1,1][-1, 1].

For simplicity and building upon previous ML-powered subgrid closures [2, 11, 8], we use the mean-squared error (MSE) of the prediction in physical units (here W2^{2}m−4^{-4}) as our loss function. Motivated by the framework presented in Figure 9, we first train MLRs (SM C2) before training NNs (SM C3) and benchmarking our ML models to quantify their accuracy and ability to generalize (SM C4).

Multiple Linear Regressions

To use the same data generator for both MLRs and NNs, we implement our MLRs in Tensorflow 2.0 [31] and train them using the Adam optimizer, which builds on stochastic gradient descent [32]. Training a climateinvariant MLR results in a weight matrix AA of size 4Np×(2Np+4)4N_p \times(2N_p + 4) and a bias vector bb of length 4Np4N_p such that:

y≈Ax~+b,(30)\mathbf{y} \approx A\tilde{\mathbf{x}} + \mathbf{b}, \tag*{(30)}

where stochastic optimization means that there is no unique optimal solution for AA and bb. We train MLRs for 20 epochs using the default Keras learning rate of 0.001 and save the weights and biases corresponding to the minimal loss over the validation set.

Neural Network Design

To isolate the effects of physically transforming the NN’s inputs, we fix the hyperparameters of all NNs trained in this study, and leave the joint investigation of hyperparameter tuning and physical transformations for future work. Informed by [6] and [7], we fix the architecture to a multilayer perceptron of 7 layers of 128 neurons separated by Leaky Rectified Linear Unit activation functions of slope 0.3, resulting in 122,872 trainable parameters for each NN. We implement the SPCAM NNs using Tensorflow 2.0 [31], train them for 20 epochs using the Adam optimizer with the default Keras learning rate of 0.001 and a default batch size of 1024, and save the parameters corresponding to the minimal validation loss.

Following the supplemental material (Sec 2) of [9], some of the hyperparameters used for the NNs trained on SAM data are different. The SAM NNs are implemented using PyTorch 1.4.0 [33], have 5 dense layers of 128 neurons each, and use cyclic learning rate [34]: Starting with an initial learning rate in [2×10−4,2×10−3][2 \times10^{-4}, 2 \times10^{-3}] for the first epoch out of 10, we then reduce the minimal and maximal learning rates by 10% for the next 6 epochs before further reducing them by a factor 10 for the last 3 epochs.

For SPCAM and following [3], we augment some of our NNs with BN and DP layers, more specifically one DP layer before each activation function and a single BN layer before the first DP layer. Following [5], we use the default DP rate of 30% and the default parameters of the Keras BN layer that normalize each feature using its mean and standard deviation in a given batch [4]. Note that we do not adjust the default parameters of DP and BN to optimize generalization skill as this would require misusing the generalization test set as a validation test.

Benchmarking

We benchmark our ML models using two different metrics: their MSEs and their coefficient of determination R2R^2, defined for a singular output yky_k as:

R2=def1−⟨yErr,k2⟩samp⟨(yTruth,k−⟨yTruth,k⟩samp)2⟩samp,(31)R^2 \mathrel{\mathop{=}^{\mathrm{def}}} 1 - \frac{\left\langle y_{\mathrm{Err},k}^{2} \right\rangle_{\mathrm{samp}}}{\left\langle\left(y_{\mathrm{Truth},k} - \left\langle y_{\mathrm{Truth},k} \right\rangle_{\mathrm{samp}}\right)^2 \right\rangle_{\mathrm{samp}}}, \tag*{(31)}

where ⟨⋅⟩samp\langle\cdot\rangle_{\mathrm{samp}} is the averaging operator over the samples of interest. For instance, if we want a horizontal map of R2R^2, we average samples at a given location over time, while we average over time and horizontal space if we want a single R2R^2 value for yky_k. Similarly, if we want one value of MSE per output yky_k, we only average the MSE over time and horizontal space rather than over all outputs, as when calculating the loss function.

While comparing MSE and R2R^2 in the reference and target generalization climates is enough to assess generalization skill after training, we are also interested in how a given ML model learns to generalize during training. To address that question, we augment our SPCAM ML models with a function (technically a “Keras callback” [35]) that calculates the MSE over two datasets that correspond to the two generalization experiments at the end of each epoch: (1) a dataset of different temperature (warm when training on cold, and vice-versa); and (2) a dataset of different geography (Earth-like when training on Aquaplanet, and vice-versa). At the end of training, we hence obtain three learning curves for each ML model: the validation loss, and the loss in the two generalization sets as a function of number of epochs. Note that these callbacks are computationally expensive as they require evaluating the ML model over ≈100M\approx100M samples at the end of each epoch, which means they should be avoided when purely seeking performance, e.g. during hyperparameter tuning.

Supplementary Results

Jensen-Shannon Distance between PDFs across Climates

As an alternative to the Hellinger PDF distance, we pick the Jensen-Shannon distance [18] because it is a symmetric distance (i.e., the arguments’ order does not affect the outcome) that uses the logarithms of the PDFs, hence giving large weights to the PDFs’ tails that tend to be particularly problematic for generalization purposes:

JS⁡(p,q)=defKL⁡(p,q)+KL⁡(q,p)2(32)\operatorname{JS}(p,q) \overset{\mathrm{def}}{=} \sqrt{\frac{\operatorname{KL}(p,q)+\operatorname{KL}(q,p)}{2}} \tag*{(32)}

where pp and qq are the normalized PDFs to compare and KL is the Kullback–Leibler divergence, defined for continuous PDFs as:

KL⁡(p,q)=def∫01dx×p(x)ln⁡[p(x)q(x)].(33)\operatorname{KL}(p,q) \overset{\mathrm{def}}{=} \int_{0}^{1} \mathrm{d}x \times p(x) \ln\left[\frac{p(x)}{q(x)}\right]. \tag*{(33)}

Learning across Climates and Geographies

This section complements § 4.2 and confirms that climate-invariant models learn mappings that are valid across climates and geographies during training. For this purpose, we track the models’ generalizability throughout the training process as explained below.

Figure S7 shows learning curves; the color of each line indicates the dataset the model was trained in, while the color of the row indicates the dataset the model was tested in. To gain intuition, we can start by looking at lines that have the same color as their axes: These are the “standard” learning curve showing that each model’s validation loss in the same climate/geography monotonically decreases as the model is trained, confirming that we are not overfitting the training set.

We are now ready to zoom in on a key result of this manuscript: The learning curve of the “climate-invariant” NN trained in the cold aquaplanet but tested in the warm aquaplanet (starred blue line in the red box (a)). Impressively, this learning curve is mostly decreasing, confirming that “climate-invariant” NNs are able to continuously learn about subgrid thermodynamics in the warm aquaplanet as they are trained in the cold aquaplanet. In contrast, the “raw-data” NN trained in the cold aquaplanet but tested in the warm aquaplanet (circled blue line in the red box (a)) makes extremely large generalization errors, which worsen as the model is trained in the cold aquaplanet.

“Climate-invariant” NNs also facilitate learning across geographies, i.e., from the aquaplanet to the Earth-like simulations (starred blue line in green box (b) is consistently below circled blue line) and vice-versa (starred green line in blue box (c) is consistently below circled green line). “Climate-invariant” transformations additionally improve the MLR baseline’s generalization ability (see right column, e.g., starred blue line in red box (a) and starred green line in blue box (c)), albeit less dramatically. This smaller improvement in MLR’s generalization abilities is linked to its relatively small number of free parameters, resulting in (1) “raw-data” MLRs generalizing better than “raw-data” NNs; and (2) MLRs having lower representation power and fitting their training sets less well, limiting the maximal accuracy of “climate-invariant” MLRs on the test set.

There are a few cases in which transforming inputs does not fully solve the generalization problem, e.g., when trying to generalize from the aquaplanet to the Earth-like simulation (starred blue line in green box (b)). NNs with DP fit their training set less well (squared lines that have the same color as their boxes are above corresponding circled/starred lines). However, they improve generalization in difficult cases (e.g., squared blue line in green box (b)) and do not overly deteriorate generalization in cases where the input transformations work particularly well (e.g., squared green line in blue box (a)). This confirms that combining physics-guided generalization methods (e.g., physical transformation of the inputs/outputs) with standard ML generalization methods (e.g., DP) is advantageous.

Geographic Skill

This section complements § 4.3 by presenting different cross-sections of NN skill after training. Our results confirm that while raw-data NN trained in the cold climate struggle to generalize to the warm climate’s Tropics, the climate-invariant mapping alleviates this limitation.

Figure S8, which shows cross-section of the coefficient of determination R2R^{2} (1 or yellow for perfect predictions, and -1 or blue for errors larger or equal to two standard deviations) exposes the raw-data NN’s poor generalization skill in the warm (+4K) Tropics. In contrast, Figure S8 underlines how climate-invariant NNs improve generalization throughout the atmosphere in the warm Tropics without deteriorating skill in the mid-latitudes and poles of the warm simulation. This consideration helped us choose our final input transformation, as the T~from NS\widetilde{T}_{\mathrm{from\ NS}} temperature transformation significantly deteriorated generalization in the mid-latitudes, while the T~buoyancy\widetilde{T}_{\mathrm{buoyancy}} transformation helps generalization in the Tropics without overly compromising skills at other latitudes. There is a slight skill compromise at high latitudes, as can be seen by comparing the second rows of Figure S8 and Figure S8, which is especially apparent in the SAM case and can be partially traced back to challenges in generalizing subgrid ice sedimentation (not shown here, see [8] for details).

To show that the improved generalization skill of climate-invariant NNs for subgrid heating is not unique to the mid-troposphere (see Figure 5), in Figure S9 we also show the generalization skill of climate-invariant NNs near the surface. Consistent with [7, 1], the highest skill for the training climate is over land for all NNs as most of the variability comes from the diurnal cycle, which is easy to predict for NNs. Similarly to Figure 5, the generalization error is apparent for the raw-data NN (a) and mostly solved by making the NN climate-invariant (b).

Visualizing Climate-Invariant Mappings

Before using SHAP in Section 4.4 to visualize the difference between raw-data and climate-invariant mappings, we test simple linear methods to analyze ML models. First, we directly plot the weights AA (see Eq (30)) of our multi-linear regressions in Figure S10/Figure S11. Second, we plot the mean Jacobian of our NN calculated via automatic differentiation in Figure S12/Figure S13. Unlike SHAP, the MLR weights and the Jacobian matrices both suggest that the climate-invariant mapping is non-local in the vertical. Figure S10/Figure S11 is consistent with the climate-invariant MLR generalizing only slightly better than the raw-data MLR (see top-right panel of Figure S7). Meanwhile, comparing Figure S12/Figure S13 to the full SHAP feature importance matrix (Figure S14/Figure S15) suggests that while the linear sensitivity of subgrid heating/moistening with respect to lower-tropospheric plume buoyancy is high (top panels of Figure S12), which is expected, subgrid heating/moistening can be well-predicted using mostly local plume buoyancy information (top panels of Figure S14).

Weights of the raw-data and climate-invariant multi-linear regressions trained in the cold and warm aquaplanet simulations and their difference

Figure S10. Weights of the (a) raw-data and (b) climate-invariant multi-linear regressions trained in the cold (-4K) aquaplanet simulation (left), the warm (+4) warm aquaplanet simulation (middle), and their difference (right). The x-axes indicate the vertical levels of the inputs, from the surface (left, 10310^3hPa) to the top of the atmosphere (right, 0hPa), while the y-axes indicate the vertical levels of the outputs, from the surface (bottom, 10310^3hPa) to the top of the atmosphere (top, 0hPa). We additionally indicate the 200hPa vertical level with dotted black lines.

Six-panel plots of MLR weights for trained cold aqua, trained warm aqua, and their difference, shown for raw data and climate-invariant mappings

Figure S11. Same as Figure S10, but for the four scalar inputs used in addition to the temperature and specific humidity inputs. The scalar inputs are surface pressure (psp_s, purple line), solar insolation (S0S_0, yellow line), surface sensible heat flux (SHF, green line), and surface latent heat flux (LHF, blue line). For the climate-invariant mapping (b), LHF is transformed to LHFΔq\mathrm{LHF}_{\Delta q} as described in § 3, which in conjunction with the temperature and humidity transformations, changes the multi-linear regression weights for all input variables.

Jacobian matrices of raw-data and climate-invariant neural networks

Figure S12. Jacobian matrices of the (a) raw-data and (b) climate-invariant neural networks trained in the cold (-4K) aquaplanet simulation (left), the warm (+4) warm aquaplanet simulation (middle), and their difference (right). The x-axes indicate the vertical levels of the inputs, from the surface (left, 10310^{3}hPa) to the top of the atmosphere (right, 0hPa), while the y-axes indicate the vertical levels of the outputs, from the surface (bottom, 10310^{3}hPa) to the top of the atmosphere (top, 0hPa). We additionally indicate the 200hPa vertical level with dotted black lines.

Jacobians for scalar inputs in raw-data and climate-invariant mappings

Figure S13. Same as Figure S12, but for the four scalar inputs used in addition to the temperature and specific humidity inputs. The scalar inputs are surface pressure (psp_s, purple line), solar insolation (S0S_0, yellow line), surface sensible heat flux (SHF, green line), and surface latent heat flux (LHF, blue line). For the climate-invariant mapping (b), LHF is transformed to LHFΔq\mathrm{LHF}_{\Delta q} as described in § 3, which in conjunction with the temperature and humidity transformations, changes the Jacobian matrices for all input variables.

RowInputSPCAM3SPCESM2SAM
1q600hPaq_{600\mathrm{hPa}}0.5, 0.80.4, 0.70.5
2qdeficit,600hPaq_{\mathrm{deficit},600\mathrm{hPa}}0.7, 1.00.4, 0.80.8
3RH600hPa\mathrm{RH}_{600\mathrm{hPa}}0.1, 0.20.1, 0.10.1
4T850hPaT_{850\mathrm{hPa}}1.4, 1.90.5, 0.81.3
5Tfrom NS,850hPaT_{\mathrm{from\ NS},850\mathrm{hPa}}0.1, 0.10.1, 0.10.2
6Bplume,850hPaB_{\mathrm{plume},850\mathrm{hPa}}0.2, 0.30.1, 0.20.1
7T150hPaT_{150\mathrm{hPa}}0.6, 0.70.7, 0.71.5
8Tfrom NS,150hPaT_{\mathrm{from\ NS},150\mathrm{hPa}}0.9, 1.40.3, 0.61.4
9Bplume,150hPaB_{\mathrm{plume},150\mathrm{hPa}}1.0, 1.20.2, 0.40.5
10LHF0.2, 0.30.2, 0.3
11LHFq\mathrm{LHF}_{q}0.1, 0.20.2, 0.2
12LHFΔq\mathrm{LHF}_{\Delta q}0.1, 0.20.2, 0.3

Table S3. Jensen-Shannon distance away from the (-4K) simulation for the PDFs of key inputs (q600hPaq_{600\mathrm{hPa}}, T850hPaT_{850\mathrm{hPa}}, T150hPaT_{150\mathrm{hPa}}, LHF) and their transformations. (+0K) distance in gray and (+4K) distance in red.

Implementation of the climate-invariant ML framework

Figure S6. Implementation of the climate-invariant ML framework. The physical transformations can be implemented by (Option 1) transforming the training set, (Option 2) adding custom layers to the ML model, or (Option 3) customizing the data generator so that it automatically transforms the model inputs/outputs.

References

  1. [1]Han, Yilun, Guang J. Zhang, Xiaomeng Huang, and Yong Wang. ”A moist physics parameterization based on deep learning.” Journal of Advances in Modeling Earth Systems 12, no. 9 (2020): e2020MS002076.DOI
  2. [2]Rasp, Stephan, Michael S. Pritchard, and Pierre Gentine. ”Deep learning to represent subgrid processes in climate models.” Proceedings of the National Academy of Sciences 115, no. 39 (2018): 9684-9689.DOI
  3. [3]Molina, Maria J., David John Gagne, and Andreas F. Prein. ”A benchmark to test generalization capabilities of deep learning methods to classify severe convective storms in a changing climate.” Earth and Space Science 8, no. 9 (2021): e2020EA001490.DOI
  4. [4]Ioffe, Sergey, and Christian Szegedy. ”Batch normalization: Accelerating deep network training by reducing internal covariate shift.” In International conference on machine learning, pp. 448-456. pmlr, 2015.
  5. [5]Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. ”Dropout: a simple way to prevent neural networks from overfitting.” The journal of machine learning research 15, no. 1 (2014): 1929-1958.
  6. [6]Ott, Jordan, Mike Pritchard, Natalie Best, Erik Linstead, Milan Curcic, and Pierre Baldi. ”A Fortran-Keras deep learning bridge for scientific computing.” Scientific Programming 2020 (2020): 1-13.arxiv.org/abs/2004.10652
  7. [7]Mooers, Griffin, Michael Pritchard, Tom Beucler, Jordan Ott, Galen Yacalis, Pierre Baldi, and Pierre Gentine. ”Assessing the potential of deep learning for emulating cloud superparameterization in climate models with real-geography boundary conditions.” Journal of Advances in Modeling Earth Systems 13, no. 5 (2021): e2020MS002385.
  8. [8]Yuval, Janni, and Paul A. O’Gorman. ”Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions.” Nature communications 11, no. 1 (2020): 3295.DOI
  9. [9]Yuval, Janni, Paul A. O’Gorman, and Chris N. Hill. ”Use of neural networks for stable, accurate and physically consistent parameterization of subgrid atmospheric processes with good performance at reduced precision.” Geophysical Research Letters 48, no. 6 (2021): e2020GL091363.
  10. [10]Beucler, Tom, Michael Pritchard, Stephan Rasp, Jordan Ott, Pierre Baldi, and Pierre Gentine. ”Enforcing analytic constraints in neural networks emulating physical systems.” Physical Review Letters 126, no. 9 (2021): 098302.DOI
  11. [11]Brenowitz, Noah D., and Christopher S. Bretherton. ”Spatially extended tests of a neural network parametrization trained by coarse-graining.” Journal of Advances in Modeling Earth Systems 11, no. 8 (2019): 2728-2744.DOI
  12. [12]Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. ” O’Reilly Media, Inc.”, 2022.
  13. [13]Siebesma, A. Pier, Sandrine Bony, Christian Jakob, and Bjorn Stevens, eds. Clouds and climate: Climate science’s greatest challenge. Cambridge University Press, 2020.
  14. [14]Seeley, Jacob T., Nadir Jeevanjee, and David M. Romps. ”FAT or FiTT: Are anvil clouds or the tropopause temperature invariant?.” Geophysical Research Letters 46, no. 3 (2019): 1842-1850.DOI
  15. [15]Hartmann, Dennis L., and Kristin Larson. ”An important constraint on tropical cloud-climate feedback.” Geophysical research letters 29, no. 20 (2002): 12-1.DOI
  16. [16]Kuang, Zhiming, and Dennis L. Hartmann. ”Testing the fixed anvil temperature hypothesis in a cloud-resolving model.” Journal of Climate 20, no. 10 (2007): 2051-2057.
  17. [17]Ahmed, Fiaz, Angel F. Adames, and J. David Neelin. ”Deep convective adjustment of temperature and moisture.” Journal of the Atmospheric Sciences 77, no. 6 (2020): 2163-2186.DOI
  18. [18]Endres, Dominik Maria, and Johannes E. Schindelin. ”A new metric for probability distributions.” IEEE Transactions on Information theory 49, no. 7 (2003): 1858-1860.DOI
  19. [19]Stephan Rasp, Tom Beucler, Gael Reinaudi, and Pierre Gentine. “Tbeucler/cbrain-cam: Climate-invariant Branch, Second Release”. Zenodo, July 12, 2023. https://doi.org/10.5281/zenodo.8140413.
  20. [20]Tom Beucler, Michael Pritchard, Liran Peng, and Janni Yuval. “Climate-invariant Machine Learning”. Zenodo, July 12, 2023. https://doi.org/10.5281/zenodo.8140536.
  21. [21]Stephan Rasp. “Raspstephan/cbrain-cam”. Zenodo, August 23, 2018. https://doi.org/10.5281/zenodo.1402384.
  22. [22]Parishani, Hossein, Michael S. Pritchard, Christopher S. Bretherton, Christopher R. Terai, Matthew C. Wyant, Marat Khairoutdinov, and Balwinder Singh. ”Insensitivity of the cloud response to surface warming under radical changes to boundary layer turbulence and cloud microphysics: Results from the ultraparame­terized CAM.” Journal of Advances in Modeling Earth Systems 10, no. 12 (2018): 3139-3158.DOI
  23. [23]Janni Yuval. “Yaniyuval/neural_nework_parameterization: Associated Code and Data for “use of Neural Networks for Stable, Accurate and Physically Consistent Parameterization of Subgrid Atmospheric Processes with Good Performance at Reduced Precision””. Zenodo, October 22, 2020. https://doi.org/10.5281/zenodo.4118346.
  24. [24]Khairoutdinov, Marat F., and David A. Randall. ”Cloud resolving modeling of the ARM summer 1997 IOP: Model formulation, results, uncertainties, and sensitivities.” Journal of the Atmospheric Sciences 60, no. 4 (2003): 607-625.doi.org/10.1175/1520-0469(2003)060<0607:crmota>2.0.co;2
  25. [25]Emanuel, Kerry A. Atmospheric convection. Oxford University Press, USA, 1994.
  26. [26]Adames, Angel F., Scott W. Powell, Fiaz Ahmed, Víctor C. Mayta, and J. David Neelin. ”Tropical precipitation evolution in a buoyancy-budget framework.” Journal of the Atmospheric Sciences 78, no. 2 (2021): 509-528.DOI
  27. [27]Maraun, Douglas. ”Bias correcting climate change simulations-a critical review.” Current Climate Change Reports 2 (2016): 211-220.DOI
  28. [28]Jeevanjeе, Nadir, and David M. Romps. ”Mean precipitation change from a deepening troposphere.” Proceedings of the National Academy of Sciences 115, no. 45 (2018): 11465-11470.DOI
  29. [29]Singh, Martin S., and Paul A. O’Gorman. ”Upward shift of the atmospheric general circulation under global warming: Theory and simulations.” Journal of Climate 25, no. 23 (2012): 8259-8276.DOI
  30. [30]Irving, Damien. ”A minimum standard for publishing computational results in the weather and climate sciences.” Bulletin of the American Meteorological Society 97, no. 7 (2016): 1149-1158.DOI
  31. [31]Abadi, Martín, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado et al. ”Tensorflow: Large-scale machine learning on heterogeneous distributed systems.” arXiv preprint arXiv:1603.04467 (2016).arxiv.org/abs/1603.04467
  32. [32]Kiefer, Jack, and Jacob Wolfowitz. ”Stochastic estimation of the maximum of a regression function.” The Annals of Mathematical Statistics (1952): 462-466.
  33. [33]Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen et al. ”Pytorch: An imperative style, high-performance deep learning library.” Advances in neural information processing systems 32 (2019).arxiv.org/abs/1912.01703
  34. [34]Smith, Leslie N. ”Cyclical learning rates for training neural networks.” In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464-472. IEEE, 2017.arxiv.org/abs/1506.01186
  35. [35]Chollet, François. ”Keras: The python deep learning library.” Astrophysics source code library (2018): ascl-1806.

Paper details

vlnc:W_dcfartkt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents