betaFeedback

WorksComputer Science

PointZip: Statistically Lossless Compression for Climate Data with Pointwise Error Bounds

Abstract

High-resolution climate simulations generate data volumes that increasingly strain storage, I/O, and data dissemination. Error-bounded lossy compressors can reduce these costs, but their tolerances are usually user-chosen and difficult to standardize across heterogeneous climate datasets. We address this limitation with statistically lossless compression, where the pointwise compression error is bounded by the pointwise data uncertainty. We propose PointZip, a two-stage compressor that combines a rate-distortion-efficient base layer with error-bound-aware residual coding to enforce pointwise guarantees. On outputs from the widely used climate modeling framework ICON, PointZip makes 1km simulations practical by reducing storage bandwidth requirements by 72x. On the ERA5 climate dataset, PointZip reaches 69.3x on the most representative variables, reducing the commonly used subset to 4 TB. In downstream evaluations, PointZip preserves data spectra, extreme events, and air parcel trajectories, while increasing neural weather forecast RMSE by less than 1%, with negligible impact on the Fractions Skill Score (FSS).

Introduction

Climate simulations and reanalysis systems are continuously increasing in spatiotemporal resolution, and their storage demands are growing accordingly [1, 2, 3]. This pressure is already evident in extreme-scale simulation workflows. Recently, the ICON model [4, 5, 6] has been awarded the Gordon Bell Prize for Climate for high-resolution simulations at 1km spatial resolution and 10s temporal resolution [7]. With a 3-hourly output frequency, it generates 16 TiB of data per checkpoint and requires 200 GiB/s of storage write bandwidth and 620 GiB/s of read bandwidth, even in a configuration focused on showcasing the compute capability. The Earth Virtualization Engines project [8] has reported up to 1 TB/s storage bandwidth requirements for high-resolution climate simulations.

These numbers show that storage and I/O are becoming primary bottlenecks in high-resolution climate simulations [7], [8]. Institutions around the world generate enormous volumes of climate data every day. In one of the highest-resolution ICON runs [7], the simulation sustained 145.7 simulated days per day, meaning that a 3-hour simulation can be completed in 74.1s. However, the I/O time needed to output a single time step at full spatial resolution is 82.2s, which is longer than the simulation time itself.

This imbalance is problematic because it limits our ability to analyze the high-spatiotemporal-resolution data generated

by these simulations. To better understand the climate system, we need to be able to store and analyze the data produced by these simulations at intervals much shorter than 3 hours. That requires not only simulation models with higher computational performance but also the ability to store and transmit much larger volumes of data. The challenge is not limited to a single model. In particular, the European Centre for Medium-Range Weather Forecasts (ECMWF) reported producing 230 TB of data per day in 2021, and next-generation systems are expected to approach the petabyte-per-day scale [1], [2]. However, even for current-generation simulations, the required 1 TB/s of storage bandwidth is available at only a few supercomputers. For example, Alps [9] provides 1.1 TB/s of write bandwidth. Current workloads therefore place severe pressure on storage systems, I/O bandwidth, and data dissemination.

As supercomputers continue to become more powerful, this challenge becomes even greater, because data can be generated faster than the available storage and I/O systems can handle. We therefore need to reduce the amount of data that must be stored and transmitted while preserving its quality for downstream applications.

Data compression is a promising solution to reduce storage and I/O costs. Purely lossless compressors [10, 11, 12] provide limited help at this scale. For climate data, typical compression ratios are often around 2× [13], which is valuable but insufficient when the primary bottleneck is dataset size. Lossy compression can achieve compression ratios above 100×100\times on climate data, but its practical adoption remains limited. Without error control, lossy compression can introduce large distortions that affect downstream analyses. Although prior evaluations [14], [15] suggest that some downstream tasks degrade only modestly at high compression ratios, users must be careful for unknown applications without explicit control of the compression error.

To address this issue, several error-bounded lossy compressors have been developed for scientific datasets [16, 17, 18, 19, 20, 21]. Recent work has also proposed error-bounded designs tailored to climate data, including EBCC [22]. These methods provide strict guarantees on the maximum compression error. However, in most existing workflows the tolerance remains a user-chosen global parameter, typically expressed as an absolute, relative, or range-relative bound. For a heterogeneous climate dataset such as ERA5 [23], a single global tolerance may be too strict in some regions and too permissive in others, and appropriate values differ across variables and downstream tasks. As a result, users must repeatedly tune and validate bounds for their own applications, which makes it difficult to apply them in practice.

To make climate and weather data compression practical at scale, an ideal compression algorithm should satisfy four properties. (1) The error guarantee should be directly linked to intrinsic properties of the data rather than to application-specific tuning. (2) The algorithm should achieve a high compression ratio to efficiently reduce pressure on the I/O system and make the compressed dataset more portable. (3) The algorithm should have high compression throughput to handle large data volumes with low resource usage. (4) The compressed dataset should not introduce significant degradation in downstream applications.

Definition of statistically lossless compression. In this work, we propose statistically lossless compression that meets the desired properties for climate data compression. In this new paradigm, instead of using artificially defined global error bounds, we define a compressed dataset to be statistically lossless with respect to the original dataset if the pointwise compression error is less than or equal to the pointwise uncertainty at every data point. In other words, statistically lossless compression ensures that the compression error at each point does not exceed the intrinsic data uncertainty at that point. Under this definition, compression does not move a value outside the pointwise uncertainty interval already represented by the dataset.

Statistically lossless compression requires two key components: a pointwise uncertainty estimate and a compressor that supports pointwise error bounds. ERA5 includes a 10-member ensemble of data assimilations with inputs perturbed within the raw-measurement uncertainty range [23, 24]. The standard deviation of these ensemble members, i.e., the ensemble spread, provides a pointwise uncertainty estimate for the final reanalysis fields [25, 26]. We use this ensemble spread as the pointwise error bound for compression. To realize this idea, we propose PointZip, a two-stage compressor for floating-point climate data that enforces pointwise error bounds. The first stage uses a rate-distortion-efficient base codec to exploit spatial structure, and the second stage encodes the residuals to satisfy the pointwise error bounds while maximizing compression ratio. An overview of our approach is shown in Figure 1.

Overview of PointZip

Figure 1. Overview of PointZip. ① - Base layer (JPEG2000 in our current implementation). ② - Residual coding step to ensure pointwise error bounds. ⓐ - Error-bound-aware precision truncation by masking and rounding. ⓑ - Lossless compression for the sparse representation of truncated residuals. ③ - Optional feedback optimization to improve the overall compression ratio. In our evaluation, we show that PointZip achieves a high storage bandwidth reduction and compression ratio, while preserving data quality for downstream applications.

In this paper, we demonstrate that our implementation of statistically lossless compression for climate data satisfies these four properties. (1) In statistically lossless compression, the error target is directly tied to intrinsic properties of the data, namely the uncertainty estimate provided by the climate dataset. The error bound is therefore not an arbitrary user choice but is grounded in the physical and statistical characteristics of the data. (2) Our approach achieves a high compression ratio, reaching 37.7×37.7\times on the full ERA5 dataset and 69.3×69.3\times on the most representative variables. The commonly used, most representative subset of ERA5 can be compressed to 4 TB and stored on a commodity hard drive. Measured on real ICON output, our approach reduces I/O bandwidth requirements by 72×72\times and can effectively increase the data output frequency from 3 hours to 7.5 minutes of simulated time at the same storage bandwidth. (3) Our feedback-free design achieves a high compression throughput. Measurements on real ICON output show that a single node provides the compression throughput required for the data volumes reported by current climate centers. (4) We evaluate the compressed dataset on a comprehensive benchmark suite of downstream tasks, where it achieves the highest compression ratio and a near-optimal rate-distortion trade-off among all evaluated methods, while having only a limited impact on downstream applications. We show that the compressed dataset preserves small-scale spectral features, extreme weather events, and trajectory density, while it does not degrade weather forecast quality by more than 1% in terms of RMSE with negligible impact on FSS.

Method

Overview

PointZip takes a floating-point array of shape (N,H,W)(N,H,W) and a pointwise error-bound array of the same shape. The bounds are required only during compression and are not included in the compressed payload. Its objective is to maximize compression ratio while satisfying every pointwise bound. Figure 1 shows the pipeline.

① Base layer. A spatial codec first compresses each (H,W)(H,W) slice independently. The current implementation uses Open-JPEG [27, 28] and JPEG2000 [29], although any codec with a good rate-distortion trade-off can serve as the base layer. A user-selected base-layer compression level controls the initial distortion.

② Residual coding. The base codec does not guarantee pointwise bounds, so PointZip encodes a correction for points that need one. It has two substeps: precision truncation (ⓐ) and sparse coding (ⓑ). Precision truncation retains only the residual bits required to satisfy each bound. The resulting residuals contain many zeros and are stored using a bitmap and a packed array of nonzero values. Both arrays are compressed losslessly with ZSTD [11], and the value array is bit-shuffled before compression [30], [31]. Section II-B gives the truncation rule and its guarantee.

③ Feedback optimization. The residual layer guarantees the bounds for any base-layer compression level, but this parameter affects the combined size of the base and residual streams. It can be selected in advance or optimized with golden-section search on a small sample. Section II-C provides more details on this component.

Decompression. The decoder reconstructs the base layer and adds the decoded residuals to produce the reconstructed data.

Floating-point mapping and final check. JPEG2000 operates on integers, so the compressor maps the data to uint16 by linearly scaling it to the uint16 range and scales the error bounds to the same domain. The inverse mapping recovers floating-point values after decompression. Rounding to uint16 introduces at most half a quantization step, approximately 8×10−68 \times10^{-6} of the data range. In typical scientific datasets, the pointwise bounds are 1% of the data range, so this mapping error is at least three orders of magnitude smaller. After inverse mapping, the compressor checks for any floating-point values that would violate their original bounds. It stores the original values at those positions using the same lossless sparse representation and restores them during decompression.

NaN handling. Some climate variables contain NaN values. For example, sea surface temperature is defined only over ocean grid points, and land points in both reanalysis and ensemble data are masked by NaN. PointZip preserves NaN values in the reconstruction by losslessly storing their locations in a bitmap. Before compression, it replaces NaNs with 32768 and assigns their bounds the uint16 maximum, so these locations do not affect residual coding. The decoder restores the NaNs from the bitmap.

Bitstream composition and overhead. In our evaluation, sparse residual coding typically accounts for less than 5% of the final bitstream. For inputs containing NaNs, the NaN bitmap also accounts for less than 5%. The final-check correction stream occupies 0 bytes because no violations occurred in our evaluation, and metadata adds a constant 20 bytes to store the offsets of the bitstream sections.

Error-Bound-Aware Precision Truncation

Precision truncation (ⓐ) of the residuals is the core component of PointZip. Its primary role is to enforce the pointwise error bound. At the same time, it is designed to maximize the compression ratio of the residual layer by zeroing out as many bits in the residuals as possible. In this section, we first show that the overall compression error is entirely determined by the precision truncation error, and then present the truncation rule and its guarantee.

1) Why only precision truncation matters: Let uu denote the original field in the integer domain and u^\hat{u} the base-layer reconstruction. The base-layer residual is defined as r:=u−u^r := u - \hat{u}.

Let r^\hat{r} be its reconstruction after residual coding. Then the final reconstruction is u^′:=u^+r^\hat{u}^{\prime} := \hat{u} + \hat{r}. By construction, the final compression error ErrErr equals the residual coding error:

Err=u−u^′=u−(u^+r^)=r−r^Err = u - \hat{u}^{\prime} = u - (\hat{u} + \hat{r}) = r - \hat{r}

Because the truncated residuals are stored losslessly in the sparse coding stage (ⓑ), this means that the final compression error is determined solely by the precision truncation step (ⓐ). Therefore, to guarantee the pointwise error bound, it is sufficient to guarantee the same bound for the precision truncation error.

2) Precision truncation rule and guarantee: To guarantee the pointwise error bound while maximizing the compression ratio, our precision truncation rule keeps only the most significant bits of the residual that are accurate enough to satisfy the bound. Figure 2 illustrates this idea in the 8-bit case.

Precision truncation process

Figure 2. Precision truncation process (ⓐ). The truncation error is never larger than the error bound. When the residual exceeds the error bound, apply rounding that maps the residual to a coarser lattice determined by the error bound. When the residual is within the error bound, apply masking to set the residual to zero.

For each point, let ebeb denote the error bound and rr denote the residual, and define S=2⌊log⁡2(2eb)⌋S = 2^{\lfloor\log_2(2eb) \rfloor}. We then have the following precision truncation rule for the residual:

r^={RNE⁡(r/S)S,∣r∣>eb,0,∣r∣≤eb,\hat{r} = \begin{cases} \operatorname{RNE}(r/S)S, & |r| > eb, \\ 0, & |r| \le eb, \end{cases}

where RNE denotes round-to-nearest with ties-to-even. This rule has two branches.

Round. If the residual exceeds the pointwise bound, we keep a quantized version of the residual. The step size SS is chosen as the largest power of two no greater than 2eb2eb, so rounding introduces no more than ebeb error while zeroing as many least significant bits as possible.

Mask. If the residual is already within the pointwise bound, we set it to zero. In this case, the base-layer reconstruction already satisfies the error bound at this point. Therefore, we can set the residual to zero without violating the error bound.

Consequently, precision truncation ensures that the final reconstruction error remains within the pointwise error bound while trying to maximize the number of zero bits in the truncated residual to achieve a high overall compression ratio.

Feedback Optimization

Although precision truncation guarantees that the final reconstruction satisfies the pointwise error bound for any base-layer compression level, the choice of this parameter can affect the overall compression ratio. The overall compression ratio first increases and then decreases as the base layer becomes more aggressive. Stronger base compression reduces the base stream but produces larger residuals. Weaker base compression has the opposite effect. The resulting unimodal objective allows golden-section search to select a near-optimal level. However, this search process requires multiple compression passes, which can be expensive for large datasets and lower the compression throughput.

To study the amount of data required for feedback optimization, we evaluate PointZip across four commonly used single-level variables from ERA5: 10m u component (zonal) of wind, 10m v component (meridional) of wind, 2m temperature, and mean sea level pressure. The data are evaluated at full spatial resolution with 6-hour temporal intervals in December 2024, and the pointwise error bound is set equal to the ensemble spread.

Figure 3 shows that optimizing on the first slice changes the resulting compression ratio by less than 1% across all four variables. This suggests that feedback optimization can be run on a small subset of the data to reduce tuning cost without materially affecting compression throughput.

Compression-ratio difference when feedback optimization uses the full input or only its first 2D slice

Figure 3. Compression-ratio difference when feedback optimization uses the full input or only its first 2D slice.

Evaluation

Overview

Compression ratio and throughput are central metrics for evaluating compression methods. They directly reflect storage bandwidth reduction and computational efficiency. However, assessing compression for climate data such as ERA5 requires more than reporting compression ratios and throughput for a given error target. Climate data is used for various downstream applications, including visualization and analysis of extreme events, tracing atmospheric transport, and evaluating forecasting models. These applications are sensitive to different forms of reconstruction error, and a good compression method should have limited impact on all of these scientific uses of the data.

To give a comprehensive evaluation of PointZip, we therefore combine algorithm- and application-level validation in our evaluation, following prior work [22, 32, 33]. The following evaluations are organized around six questions. (1) Can PointZip efficiently enforce statistically lossless compression under pointwise uncertainty bounds? (2) Can its compression performance translate into substantial storage-bandwidth reduction for real weather-simulation output? (3) Under conventional global error bounds, how does PointZip compare to existing error-bounded compressors? (4) Does the compressed dataset produced by PointZip introduce artifacts that affect downstream applications? (5) Does our approach remain effective when scaled from the most representative variables to all variables in ERA5? (6) How robust is PointZip to changes in the supplied uncertainty estimates?

Statistically Lossless Compression of the Most Representative ERA5 Variables

This section answers question (1) by evaluating PointZip under the main setting of this work, namely statistically lossless compression under pointwise error bounds defined by the ERA5 ensemble spread.

1) Evaluation setup: We evaluate December 2024 ERA5 data at hourly resolution on the global 0.25° grid. Each variable and pressure level is compressed as a separate chunk containing 744 time steps and 721 × 1440 spatial points. The lower-resolution ensemble spread is linearly interpolated to the reanalysis grid and used as the pointwise error bound.

To simulate realistic usage scenarios, we focus on the most representative subset of ERA5 variables. This subset contains four single-level variables, 2m temperature, 10m u component of wind, 10m v component of wind, and mean sea level pressure, and five pressure-level variables, temperature, u component of wind, v component of wind, specific humidity, and geopotential, on all 37 pressure levels. They are the main variables used in weather and climate applications, including both traditional physics-based analysis [34] and neural weather modeling [35, 36, 37]. The raw data size of this subset is 584 GB in our evaluation period, and our performance measurements are performed on an AMD EPYC 7742 CPU.

No baseline compressor natively accepts pointwise error bounds as constraints. EBCC [22] is the most relevant compression method as it is designed specifically for weather and climate data. It uses a feedback loop to adjust its compression parameters until all points satisfy the error target. We therefore extend EBCC to support pointwise error bounds by modifying its feedback loop so that tuning stops only when all points satisfy their pointwise bounds. This extended version of EBCC is the baseline method for comparison in this section.

  1. Results: Figure 4 summarizes the results. PointZip has 15.1 MB/s single-threaded compression throughput and 149.6 MB/s single-threaded decompression throughput. Its compression throughput is 12.6×12.6\times higher than that of EBCC, while decompression throughput is of the same order of magnitude. This aligns with the design of the method. PointZip uses predefined compression parameters and a single-pass encoding process, whereas EBCC must repeatedly recompress inside a feedback loop until pointwise constraints are met.

Statistically lossless compression results for the most representative ERA5 variables

Figure 4. Statistically lossless compression results for the most representative ERA5 variables. Left: single-threaded compression and decompression throughput. Error bars denote standard deviation. Middle: overall compression ratio for different compressors. Right: histogram of PointZip compression ratios across variables and pressure levels.

Besides high compression throughput, PointZip has an overall compression ratio of 69.3×69.3\times on the full subset. This is 3.4×3.4\times higher than that of EBCC under the same pointwise error bounds. When admissible error varies strongly across space, a method that does not explicitly isolate the hard points is effectively limited by the tightest tolerances. In contrast, PointZip handles those points by a residual coding step, which allows it to achieve a much higher compression ratio while still satisfying the pointwise error bounds.

Across individual variables, PointZip achieves compression ratios between 25×25\times and 60×60\times for the single-level variables, and between 25×25\times and 270×270\times for the pressure-level variables. This broad range reflects the strong variation in compressibility across variables and pressure levels, but PointZip consistently delivers substantial size reduction throughout the benchmark.

To put those results in an operational context, applications are mainly interested in the recent 40 years of the most representative ERA5 variables, which have a total size of 275 TB. Our method can reduce this to 4 TB in a statistically lossless manner, which can be stored on a single commodity hard drive or transferred over the network quickly. This makes it possible to distribute a single compressed version of the dataset that is broadly usable.

Storage Bandwidth Reduction for Weather Simulation

This section answers question (2) by evaluating PointZip on real output from the ICON weather model [4, 5, 38], and analyzing the implications of our method for storage bandwidth in weather-simulation workflows.

  1. Evaluation setup: To analyze the implications for storage bandwidth in weather simulation, we apply our method to raw output from the ICON model [4, 5, 38]. ICON is widely used in major weather centers and is capable of global modeling at 1 km resolution [7]. We use the output of a 16-step air bubble simulation example provided by ICON in the default setup, with results remapped to a 0.25∘0.25^\circ rectangular grid. Pointwise error bounds are set based on the model discretization error, which is estimated by measuring the difference in results when doubling the spatial and temporal resolution.

In addition to the baseline method in § 2-B, we also compare to other state-of-the-art compressors that only provide global error bounds, including SPERR [21], SZ [19], SZ3 [20], and HPEZ [39]. We also exclude the constant fields in the data because they can be compressed with a simple constant-value encoding and do not reflect the general performance of the method on floating-point data. Each node uses 288 threads to compress all variables and pressure levels. Different nodes compress different time steps in parallel. The raw data size is 42.7 GB per node in our evaluation period, matching the load per GH200 node of the 1.25km resolution climate simulation [7].

  1. Results: Under the pointwise error bounds derived from the ICON discretization error, PointZip achieves a compression ratio of 108×108\times, compared with 13.2×13.2\times for the pointwise-bound extension of EBCC. The left panel of Figure 5 shows multi-threaded compression throughput on a single node with 288 threads. PointZip reaches 4.3 GB/s, corresponding to approximately 372 TB/day. This exceeds the 230 TB/day data-production rate reported by ECMWF [1], indicating that a single node can sustain an operational-scale compression workload. EBCC reaches only 0.5 GB/s under the same pointwise constraints and is therefore 8.6×8.6\times slower. SPERR, SZ, SZ3, and HPEZ achieve higher raw compression throughput, ranging from 8.8 to 13.4 GB/s, but do not provide the spatially varying pointwise guarantees evaluated here. The right panel shows that PointZip achieves approximately 14×14\times weak-scaling speedup on 16 nodes. This demonstrates that its throughput scales effectively as the data volume and number of compression nodes increase.

Multi-threaded compression performance on real ICON output

Figure 5. Multi-threaded compression performance on real ICON output. Left: single-node compression throughput. Right: weak-scaling speedup from 1 to 16 nodes. The dashed line denotes ideal linear scaling.

We project our results to a large-scale weather-simulation workflow by adopting numerical values from the Gordon Bell Prize run [7]. As shown in Figure 6, this run uses 2579 processes to output 16 TiB of data at the end of the 3-hour simulation. To apply our method, we first remap the data to a rectangular grid, then apply PointZip using 288 threads per process. This grid mapping step increases the data size by 50%, so our method achieves 72×72\times overall compression on the original data, thus reducing the time to output the data to storage from 82.2s to 3.33s.

Projected impact of PointZip on the 1.25 km ICON output workflow

Figure 6. Projected impact of PointZip on the 1.25 km ICON output workflow.

This 24.7×24.7\times reduction in end-to-end output time can alter

natively allow for an increase in the output frequency from 3 hours to 7.5 minutes of simulated time without increasing the storage bandwidth or runtime requirements. This timescale is important for post hoc air parcel tracing and for precipitation and convection lifecycle analysis. Under the same output frequency, the bandwidth requirement is reduced from 198.19 GiB/s to 2.75 GiB/s, lowering the required storage bandwidth by 72×72 \times and enabling the use of commodity storage systems for global 1.25km resolution climate simulations.

Compression under Global Error Bounds of the Most Representative ERA5 Variables

This section answers question (3) by evaluating PointZip under global error bounds and comparing it to existing error-bounded compressors.

1) Evaluation setup: We use the same data and measurement setup as in § 2-B, except that all methods are evaluated under global error bounds. However, the choice of global error bound is usually application-dependent in practice, and it is not meaningful to set a single global error bound across variables with different units and dynamic ranges. We therefore use a range-relative error bound that is defined as a constant fraction of the data range (maximum value minus minimum value) of each data chunk. We evaluate fractions of 0.1%, 0.5%, 1%, 5%, 10% to span strict, moderate, and relaxed error regimes. This provides a consistent basis across variables and levels.

In addition to the baseline method in § 2-C, we also include SRN-SZ [40] as our baseline. SRN-SZ is a recent neural-network-based compressor built on top of SZ3. We train SRN-SZ on ERA5 data using the official training scripts.

2) Results: We evaluate the rate-distortion trade-off of different compressors by measuring the compression ratio and the structural similarity index (SSIM) of the reconstructed fields at different error bounds. Figure 7 summarizes the results. Following Baker et al. [32], we consider compressed fields with SSIM values above 0.99995 to be visually indistinguishable from the original fields. This region is highlighted in blue in the plot. PointZip provides the strongest rate-distortion trade-offs together with SPERR and EBCC. This means that even though PointZip is not designed to optimize for global error bounds, it still achieves high compression ratios while maintaining high visual fidelity under conventional settings.

Rate-distortion trade-off, compression ratio vs. SSIM

Figure 7. Rate-distortion trade-off, compression ratio vs. SSIM. The 0.99995 SSIM threshold proposed by Baker et al. [32] is marked by the blue region on the plot.

Downstream Applications at Matched Compression Ratios

This section answers question (4) by evaluating the compressed datasets in a wide range of downstream applications including both traditional physics-based applications and modern AI-based applications.

To compare the impact of compression from different compressors, we evaluate the compressed datasets under the same storage budget. We chunk data in the same way as in § 2-B and compress it under a statistically lossless setup with PointZip. For the same set of baseline methods as in § 2-D, we tune their constant global error bounds with a binary search until their overall compression ratio matches the PointZip target within ±1\pm1.

These compressed datasets at matched compression ratios are then evaluated in a comprehensive benchmark suite of downstream analyses. Those analyses include general analysis of the data spectrum, visual analysis of extreme events, traditional physics-based Lagrangian trajectory simulations, and weather forecasting by neural networks.

1) Spectrum analysis: To examine scale-dependent fidelity of different compressors at matched compression ratios, we compute the spherical-harmonics power spectra of the compressed fields and compare them to the original spectra. Figure 8 shows two representative examples. Even at a moderate compression ratio (42.0×42.0 \times), SZ and SRN-SZ deviate visibly from the reference spectrum in the high-frequency range for temperature at 1000 hPa. At a higher compression ratio (139.4×139.4 \times), SPERR, EBCC and PointZip remain close to the original spectrum, whereas other methods introduce larger artifacts in the high-frequency range.

Spherical-harmonics power spectra under matched compression ratios

Figure 8. Spherical-harmonics power spectra under matched compression ratios. Top: geopotential (z) at 500 hPa. Bottom: temperature (t) at 1000 hPa. Insets show zoomed views of the high-frequency range.

These spectral differences matter because high-frequency components encode sharp gradients and small-scale structures that can affect downstream analyses. The spectral results therefore suggest that PointZip preserves physically relevant fine-scale structures rather than smoothing them away.

2) Case study of Hurricane Milton: High-frequency artifacts are visible not only in spectra but also directly in spatial fields around extreme events. Here we present a case study of Hurricane Milton, which formed in the Gulf of Mexico in October 2024 and reached Category 5 intensity. Figure 9 shows synthetic satellite views of the storm and the derived divergence field at 1000 hPa. PointZip, EBCC and SPERR preserve the storm structure with better quality than other methods. On the divergence field, PointZip shows slightly higher reconstruction error than EBCC and SPERR, but still maintains a higher quality than SZ, SZ3, HPEZ and SRN-SZ. This result is consistent with the spectrum analysis and shows that PointZip preserves the structure of extreme-weather events rather than smoothing it away.

Synthesized satellite views of wind and wind-derived fields during Hurricane Milton

Figure 9. Synthesized satellite views of wind and wind-derived fields during Hurricane Milton at 1000 hPa under matched compression ratios. Top: uu component of wind. Bottom: divergence derived from the compressed wind fields. For each derived field, the upper row shows the reconstructed field and the lower row shows the error relative to the reference derived from the original wind fields.

3) Case study of Bezymianny volcano eruption: Traditional physics-based applications are also important downstream uses of ERA5. Here we evaluate the effect of compression on Lagrangian trajectory simulation. We present a case study of the Bezymianny volcano eruption on December 24, 2024. Volcanic eruptions inject material into the atmosphere, and the subsequent transport of this material is important for understanding the eruption’s impact on air quality and for aviation safety. This is a challenging application for compression because errors in the wind fields can accumulate over time and lead to large deviations in the simulated trajectories, so it is important to understand how compression affects the results.

We run a seven-day simulation with LAGRANTO [34] using compressed datasets under matched compression ratios. A total of 50,000 air parcels are initialized near the volcano summit. To compare, we compute the particle density field at the end of each day and evaluate the Fractions Skill Score (FSS) of the final density field relative to the reference density field obtained from the original wind fields. The FSS measures the spatial agreement between neighborhood event fractions in reconstructed and reference fields. It ranges from 0 to 1, with higher values indicating better agreement. EBCC is closest to the reference. PointZip, HPEZ and SPERR are slightly worse but still maintain a higher FSS than other methods (Table I).

Method1 day3 days5 days7 days
PointZip (Ours)0.9940.9070.8430.804
EBCC0.9970.9630.9450.875
SPERR0.8730.8900.8760.791
SZ0.2880.0000.0000.017
SZ30.6260.2960.3110.335
HPEZ0.9840.8950.8950.797
SRN-SZ0.9270.6190.2190.258

Table I. AIR PARCEL DENSITY FSS ↑ VS. TIME AFTER ERUPTION.

Figure 10 shows the simulated trajectories and the final particle-density maps at the end of the seven-day period with a resolution of 0.2∘0.2^\circ. For high-fidelity methods including PointZip, the details in the trajectories are preserved and lead to a more accurate density field.

Seven-day air-parcel trajectories under matched compression ratios for the Bezymianny eruption

Figure 10. Seven-day air-parcel trajectories under matched compression ratios for the Bezymianny eruption. Trajectories are initialized from 50,000 locations near the top of the volcano. Top: all trajectories, colored by initial pressure level. Bottom: particle-density maps at the end of the seven-day period with a resolution of 0.2°.

4) Weather Forecast Impact: Modern AI-based applications are increasingly important downstream uses of ERA5. One typical example is neural weather modeling, so it is important to understand how compression affects forecast skill. We evaluate this using Aurora [37], a neural foundation model for Earth-system prediction, and measure forecast skill when compressed ERA5 data are used as input. Predictions are based on compressed ERA5 data and are fed back autoregressively for four steps at 6-hour intervals, corresponding to 24 hours.

Table II shows the normalized latitude-weighted root mean squared error (RMSE) and the Fractions Skill Score (FSS) of Aurora rollouts. PointZip changes forecast skill only marginally. Across all lead times, the RMSE increase remains below 1% and the FSS decrease is negligible.

Statistical Lossless Compression of all ERA5 Variables

This section answers question (5) by evaluating statistically lossless compression of all ERA5 variables.

1) Evaluation setup: We apply statistically lossless compression to all ERA5 variables under the same chunking setup and time period as in § 2-B. ERA5 contains both single-level and pressure-level variables, with a total of 854 variables across different levels. 30 of these variables contain NaN values for padding, and 328 variables have zero values in their ensemble spread fields, which means lossless compression is required at those points. PointZip can handle these cases natively without the need for special preprocessing. However, we skip the following two groups of variables. The first group lacks ensemble spread files, so the pointwise error bounds required by our method are unavailable. These fields are typically derived variables that are less important than other variables. The second group consists of constant boundary-condition fields. These fields can be stored separately without needing to be compressed. Table III provides a summary of these cases. The remaining 796 variables have an uncompressed size of 2.35 TB and are compressed with PointZip under a statistically lossless setup.

  1. Results: For all ERA5 variables, our method achieves an overall compression ratio of 37.7×37.7\times. Across retained variables, single-level variables are compressed by 23.2×23.2\times, whereas pressure-level variables are compressed by 45.9×45.9\times, as summarized in Table III. Single-level variables are generally less compressible because many of them contain finer spatial detail and smaller ensemble spreads. Pressure-level variables are usually smoother and often have larger admissible pointwise bounds, which allow higher compression ratios while still satisfying the local uncertainty constraint. This evaluation shows that the method remains effective in dataset size reduction when scaled from the most representative variables to all ERA5 variables with different characteristics and scientific uses.

Robustness to Uncertainty Estimates

This section answers question (6) by examining how sensitive PointZip is to changes in the supplied uncertainty estimates. Although many climate datasets provide ensemble spread as an uncertainty estimate, this uncertainty estimation can be imperfect. PointZip is designed to maintain a similar compression ratio and reconstruction quality when the supplied error bounds are changed within a reasonable range. We use the same ERA5 setup as in § 2-B. Under changes in the uncertainty estimates, we compare PointZip with the same base layer compression level, PointZip that reoptimizes the base layer for each modified uncertainty field, and EBCC.

First, we consider the case where the uncertainty field is not accurate for the local variation. This is a common situation in practice. We simulate this by averaging the original ensemble spread by spatial blocks. This progressively removes local variation while preserving its average magnitude. This is shown in the left panel of Figure 11, where we average the uncertainty field over 4×44 \times4, 16×1616 \times16, and 64×6464 \times64 blocks, followed by global averaging. PointZip shows changes of less than 25% in compression ratio and less than 21% in reconstruction MSE, even when all spatial structure in the ensemble spread is removed. These changes are much smaller than those observed for EBCC, which shows a 61% change in compression ratio and a change of more than 700% in reconstruction MSE. This means that PointZip still gives a similar compressed dataset even when the spatial structure of the uncertainty field is lost.

Sensitivity to changes in the supplied uncertainty estimates

Figure 11. Sensitivity to changes in the supplied uncertainty estimates. Both axes report percentage changes relative to compression using the original pointwise ERA5 ensemble spread. Left: the uncertainty field is progressively smoothed using 4×44 \times4, 16×1616 \times16, and 64×6464 \times64 spatial block averaging, followed by global averaging. Right: the uncertainty field is globally averaged and then rescaled from −40%-40\% to +40%+40\%. PointZip reuse retains the base-layer compression level selected for the original uncertainty field, whereas PointZip reopt. optimizes this level for each modified field.

MethodStep 1 RMSE ↓Step 1 FSS ↑Step 2 RMSE ↓Step 2 FSS ↑Step 3 RMSE ↓Step 3 FSS ↑Step 4 RMSE ↓Step 4 FSS ↑
REF0.007450.99800.010440.99620.012640.99430.014580.9924
PointZip (Ours)0.00748 (+0.40%)0.9980 (-0.00%)0.01045 (+0.10%)0.9962 (-0.00%)0.01265 (+0.08%)0.9943 (-0.00%)0.01459 (+0.07%)0.9924 (-0.00%)
EBCC0.00747 (+0.27%)0.9980 (-0.00%)0.01045 (+0.10%)0.9962 (-0.00%)0.01265 (+0.08%)0.9943 (-0.00%)0.01459 (+0.07%)0.9924 (-0.00%)
SPERR0.00775 (+4.03%)0.9979 (-0.01%)0.01057 (+1.25%)0.9961 (-0.01%)0.01272 (+0.63%)0.9943 (-0.00%)0.01463 (+0.34%)0.9924 (-0.00%)
SZ0.01617 (+117%)0.9898 (-0.82%)0.01751 (+67.7%)0.9886 (-0.76%)0.01892 (+49.7%)0.9866 (-0.77%)0.02030 (+39.2%)0.9844 (-0.81%)
SZ30.01396 (+87.4%)0.9927 (-0.53%)0.01524 (+46.0%)0.9919 (-0.43%)0.01639 (+29.7%)0.9906 (-0.37%)0.01764 (+21.0%)0.9891 (-0.33%)
HPEZ0.00748 (+0.40%)0.9980 (-0.00%)0.01045 (+0.10%)0.9962 (-0.00%)0.01265 (+0.08%)0.9943 (-0.00%)0.01459 (+0.07%)0.9924 (-0.00%)
SRN-SZ0.00819 (+9.93%)0.9978 (-0.02%)0.01093 (+4.69%)0.9960 (-0.02%)0.01303 (+3.09%)0.9942 (-0.01%)0.01491 (+2.26%)0.9923 (-0.01%)

Table 2. AURORA FORECAST PERFORMANCE ACROSS 6-HOUR ROLLOUT STEPS UNDER MATCHED COMPRESSION RATIOS.

LevelSinglePressureTotal
Variables262592854
Variables with Special Values
NaN-Padding50050
Zero in Ensemble Spread143185328
Skipped Variables
Missing Ensemble Spread File36036
Constant Boundary Conditions22022
Compression Results
Variables204592796
Uncompressed Size [TB]0.521.832.35
Compression Ratio23.245.937.7

Table 3. COMPRESSION SUMMARY FOR ALL ERA5 VARIABLES.

Second, we consider the case where the dataset provides no uncertainty estimate at all, so users use a single global uncertainty estimate to compress the data. This is simulated by replacing the original uncertainty field with its global mean and scaling it by −40%, −20%, 0%, +20%, and +40%, as shown in the right panel of Figure 11. PointZip shows a 67% change in compression ratio and a change of less than 121% in reconstruction MSE. For EBCC, the corresponding changes are 103% and more than 1331%, respectively. This means that for a dataset with no provided uncertainty estimate, PointZip is more robust to the choice of global uncertainty estimate than EBCC.

Related Work

  • a) Lossless Compression for Climate Data:* Common storage backends for climate data include GRIB/GRIB2 [41], NetCDF [42], HDF5 [43], and Zarr [44]. These systems are typically configured to be lossless or near-lossless. Besides the DEFLATE family [45, 46, 47], scientific storage systems increasingly use codecs such as bzip2 [48], LZ4 [10], Zstandard [11], and XZ [12] to balance throughput and compression ratio. Some lossless codecs improve the compression ratio of floating-point arrays through transformations. Blosc applies shuffle-style transformations [30, 49]. Other floating-point-specific compressors such as FPZIP [50] and FPC [51] further exploit structure in the floating-point representation.

  • b) Lossy Compression for Climate Data without Error Guarantees:* Classical lossy compression methods optimize rate-distortion performance under average-error metrics such as MSE or PSNR without strict error guarantees. This includes image codecs such as JPEG and JPEG2000 [29, 52], video codecs such as H.264, H.265, and AV1 [53, 54, 55], and neural compression models [56, 57, 58, 59]. CRA5 [14] learns a compact latent representation for ERA5 and reconstructs fields with a neural decoder. Pellicer-Valero et al. [15] use FFmpeg [60] to compress climate data as video sequences and report limited degradation in downstream evaluation scores when compressed data are used to evaluate neural models trained on the original dataset.

  • c) Error-Bounded Lossy Compression for Climate Data:* Following earlier studies [61] on how lossy compression affects downstream scientific analysis, error-bounded lossy compression has become a major direction in scientific data compression. It combines substantial size reduction with explicit reconstruction guarantees [16, 17, 18, 19, 20, 21, 33, 62]. Representative methods include SZ [19], SZ3 [20], ZFP [17], MGARD [18], and SPERR [21], which use prediction-, transform-, or multiresolution-based designs to enforce error control while retaining good compression performance. These methods typically guarantee a global maximum absolute error bound, and some also support relative error bounds. Compression methods designed specifically for weather and climate data have also been explored [1, 22, 32]. EBCC [22] is a two-stage compressor that first applies image-based transformations and then compresses residuals in a feedback loop to guarantee error bounds. Learned compressors, such as SRN-SZ [40], AE-SZ [63], and GraphComp [64], can provide strong rate-distortion performance by exploiting data-specific representations, but they often require dataset-specific training and an extra non-trainable layer to enforce error bounds. These algorithms are typically computationally intensive and suffer from limited throughput in real-world applications.

  • d) Uncertainty Estimation of ERA5:* All physical measurements are uncertain. Modern metrology therefore represents uncertainty probabilistically. Under the common Gaussian approximation, a measurement is represented as μ±σ\mu\pm\sigma, where μ\mu is the mean and σ\sigma is the standard deviation. The true value is expected to lie within that interval with approximately 68% probability. These conventions are standard in scientific reporting and are reflected in guidance from NIST [65] and JCGM [66, 67].

In numerical weather prediction and reanalysis [68, 69] uncertainty is commonly characterized with ensemble-based methods [25]. In ECMWF’s workflow, uncertainty arises from imperfect initial conditions and uncertainty in the underlying observations, and it is propagated through the data-assimilation system that combines observations with the forecast model [26]. For ERA5, ECMWF provides this through a 10-member ensemble with perturbed inputs drawn within the uncertainty range of the measurements [23, 24].

Conclusion

In this work, we introduce PointZip, an uncertainty-aware compressor for climate data that uses the ensemble spread as a pointwise error bound. This replaces user-chosen global tolerances with a spatially adaptive and scientifically meaningful error target, yielding a statistically lossless compressed dataset with respect to data uncertainty. It enforces pointwise guarantees while remaining practical for standard scientific-data workflows. Our approach makes global climate simulations at 1km resolution more practical by reducing storage bandwidth requirements by 72×72\times and enabling an output frequency of 7.5 minutes of simulated time using the same storage bandwidth as an uncompressed output frequency of 3 hours of simulated time. PointZip reaches compression ratios of 69.3×69.3\times and 37.7×37.7\times on the most representative variables and all variables in ERA5, respectively. At matched storage budgets, PointZip preserves spectra, extreme events, and air parcel trajectories while introducing less than a 1% increase in RMSE in neural weather forecasts. These results show that PointZip is robust against noise in the uncertainty estimation and can deliver high-quality compressed data while enabling efficient storage and distribution of large climate datasets.

Acknowledgment

This work was supported by the WeatherGenerator project (grant agreement No. 101187947), the ERC PSAP project (grant agreement No. 101002047) and the SERI NCCR Clim+ project.

References

  1. [1]M. Klöwer, M. Razinger, J. J. Dominguez, P. D. Düben, and T. N. Palmer, “Compressing atmospheric data into its real information content,” Nature Computational Science, vol. 1, no. 11, pp. 713–724, Nov 2021. [Online]. Available: https://doi.org/10.1038/s43588-021-00156-2
  2. [2]P. Bauer, A. Thorpe, and G. Brunet, “The quiet revolution of numerical weather prediction,” Nature, vol. 525, no. 7567, pp. 47–55, Sep 2015. [Online]. Available: https://doi.org/10.1038/nature14956
  3. [3]J. T. Overpeck, G. A. Meehl, S. Bony, and D. R. Easterling, “Climate data challenges in the 21st century,” Science, vol. 331, no. 6018, pp. 700–702, 2011. [Online]. Available: https://doi.org/10.1126/science.1197869
  4. [4]G. Zängl, D. Reinert, P. Rípodas, and M. Baldauf, “The ICON (ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M: Description of the non-hydrostatic dynamical core,” Quarterly Journal of the Royal Meteorological Society, vol. 141, 01 2015. [Online]. Available: https://doi.org/10.1002/qj.2378
  5. [5]ICON contributors, “Icon source code,” https://gitlab.dkrz.de/icon/icon-model, 2026.
  6. [6]The ICON Partners, “Amip configuration,” https://docs.icon-model.org/buildrun/buildrun_recommmended/aes_configurations/amip.html, 2026.
  7. [7]D. Klocke, C. Frauen, J. F. Engels, D. Alexeev, R. Redler, R. Schnur, H. Haak, L. Kornblueh, N. Brüggemann, F. Chegini, M. Römmert, L. Hoffmann, S. Griessbach, M. Bode, J. Coles, M. Gila, W. Sawyer, A. Calotou, Y. Budanaz, P. Mazumder, M. Copik, B. Weber, A. Herten, H. Boeckelmann, T. Hoefler, C. Hohenegger, and B. Stevens, “Computing the full earth system at 1km resolution,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 125–136. [Online]. Available: https://doi.org/10.1145/37712285.3771789
  8. [8]T. Hoefler, B. Stevens, A. F. Prein, J. Baehr, T. Schulthess, T. F. Stocker, J. Taylor, D. Klocke, P. Mamminien, P. M. Forster, T. Kölling, N. Gruber, H. Anzt, C. Frauen, F. Ziemen, M. Klöwer, K. Kashinath, C. Schär, O. Fuhrer, and B. N. Lawrence, “Earth virtualization engines: A technical perspective,” Computing in Science & Engineering, vol. 25, no. 3, pp. 50–59, 2023. [Online]. Available: https://doi.org/10.1109/MCSE.2023.3311148
  9. [9]CSCS, “Alps storage documentation,” https://docs.cscs.ch/alps/storage/, 2026.
  10. [10]LZ4 contributors, “Lz4 - extremely fast compression,” https://github.com/lz4/lz4, 2026.
  11. [11]Y. Collet and M. S. Kucherawy, “RFC 8878: Zstandard compression and the application/zstd media type,” https://www.rfc-editor.org/rfc/rfc8878, 2021, accessed 2026-02-10.
  12. [12]XZ Developers, “XZ utils,” https://github.com/tukaani-project/xz, 2026, accessed 2026-02-10.
  13. [13]M. Kuhn, J. Kunkel, and T. Ludwig, “Data compression for climate data,” Supercomputing Frontiers and Innovations, vol. 3, no. 1, pp. 75–94, Jun 2016. [Online]. Available: https://sfr.cisr.ufmsu.ru/index.php/sf/article/view/101DOI
  14. [14]T. Han, Z. Chen, S. Guo, W. Xu, and L. Bai, “Cra5: Extreme compression of era5 for portable global climate and weather research via an efficient variational transformer,” 2024. [Online]. Available: https://arxiv.org/abs/2405.03376
  15. [15]O. J. Pellicer-Valero, C. Aybar, and G. C. Valls, “Video compression for spatiotemporal earth system data,” 2025. [Online]. Available: https://arxiv.org/abs/2506.19656
  16. [16]S. Di, J. Liu, K. Zhao, X. Liang, R. Underwood, Z. Zhang, M. Shah, Y. Huang, J. Huang, X. Yu, C. Ren, H. Guo, G. Wilkins, D. Tao, J. Tian, S. Jin, Z. Jian, D. Wang, M. R. Hahman, B. Zhang, S. Song, J. Calhoun, G. Li, K. Yoshii, K. Alharthi, and F. Cappello, “A survey on error-bounded lossy compression for scientific datasets,” ACM Comput. Surv., vol. 57, no. 11, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3733104
  17. [17]P. Lindstrom, “Fixed-rate compressed floating-point arrays,” IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 12, pp. 2674–2683, 2014. [Online]. Available: https://doi.org/10.1109/TVCG.2014.2346458
  18. [18]Q. Gong, J. Chen, B. Whitney, X. Liang, V. Reshniak, T. Banerjee, J. Lee, A. Rangarajan, L. Wan, N. Vidal, Q. Liu, A. Gainaru, N. Podhorszki, R. Archibald, S. Ranka, and S. Klasky, “Mgard: A multigrid framework for high-performance, error-controlled data compression and refactoring,” SoftwareX, vol. 24, p. 101590, Dec. 2023. [Online]. Available: http://dx.doi.org/10.1016/j.softx.2023.101590
  19. [19]S. Di and F. Cappello, “Fast error-bounded lossy hpc data compression with sz,” in 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2016, pp. 730–739. [Online]. Available: https://doi.org/10.1109/IPDPS.2016.11
  20. [20]X. Liang, K. Zhao, S. Di, S. Li, R. Underwood, A. M. Gok, J. Tian, J. Deng, J. C. Calhoun, D. Tao, X. Chen, and F. Cappello, “Sz3: A modular framework for composing prediction-based error-bounded lossy compressors,” 2021. [Online]. Available: https://arxiv.org/abs/2111.02925
  21. [21]S. Li, P. Lindstrom, and J. Clyne, “Lossy scientific data compression with sperr,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2023, pp. 1007–1017. [Online]. Available: https://doi.org/10.1109/IPDPS54959.2023.00104
  22. [22]L. Huang, L. Fusco, F. Scheidl, J. Zibell, M. A. Sprenger, S. Schemm, and T. Hoefler, “Error bounded compression for weather and climate applications,” 2025. [Online]. Available: https://arxiv.org/abs/2510.22265
  23. [23]H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers, A. Simmons, C. Soti, S. Abdalla, X. Abellan, G. Balsamo, P. Bechtold, G. Biavati, J.-R. Bidlot, M. Bonavita, and J.-N. Thépaut, “The era5 global reanalysis,” Quarterly Journal of the Royal Meteorological Society, vol. 146, no. 730, pp. 1999–2049, 2020. [Online]. Available: https://doi.org/10.1002/qj.3803
  24. [24]Copernicus Climate Change Service (C3S), “Era5 dataset documentation,” https://confluence.ecmwf.int/display/CKB/ERA5, accessed 2026-02-10.
  25. [25]M. Leutbecher and T. Palmer, “Ensemble forecasting,” Journal of Computational Physics, vol. 227, no. 7, pp. 3515–3539, 2008, predicting weather, climate and extreme events. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0021999107000812DOI
  26. [26]D. P. Dee, S. M. Uppala, A. J. Simmons, P. Berrisford, P. Poli, S. Kobayashi, U. Andrae, M. A. Balmaseda, G. Balsamo, P. Bauer, R. Bechtold, A. C. M. Beljaars, J. Bidlot, N. Bormann, C. Delsol, R. Dragani, M. Fuentes, A. J. Geer, L. Haimberger, S. B. Healy, H. Hersbach, E. V. Hólm, L. Isaksen, P. Kållberg, M. Köhler, M. Matricardi, A. P. McNally, B. M. Monge-Sanz, J.-J. Morcrette, B.-K. Park, C. Peubey, P. de Rosnay, C. Tavolato, J.-N. Thépaut, and F. Vitart, “The ERA-interim reanalysis: Configuration and performance of the data assimilation system,” Quarterly Journal of the Royal Meteorological Society, vol. 137, no. 656, pp. 553–597, 2011. [Online]. Available: https://doi.org/10.1002/qj.828
  27. [27]OpenJPEG Developers, “Openjpeg: An open-source jpeg 2000 codec written in c language,” https://github.com/uclouvain/openjpeg, 2026, accessed 2026-01-26.
  28. [28]Glymur Developers, “Glymur: a python interface for jpeg 2000,” https://github.com/quintusdias/glymur, 2026, accessed 2026-01-26.
  29. [29]A. Skodras, C. Christopoulos, and T. Ebrahimi, “The jpeg 2000 still image compression standard,” IEEE Signal Processing Magazine, vol. 18, no. 5, pp. 36–58, 2001. [Online]. Available: https://doi.org/10.1197/79.52804
  30. [30]Blosc Developers, “Blosc: A blocking, shuffling and lossless compression library,” 2024, accessed 2026-02-10. [Online]. Available: https://github.com/Blosc/c-blosc
  31. [31]Python-Blosc Developers, “Python-Blosc: A python wrapper for the extremely fast blosc compression library,” 2026, accessed 2026-02-10. [Online]. Available: https://github.com/pythons-blosc/python-blosc
  32. [32]A. H. Baker, H. Xu, J. M. Dennis, M. N. Levy, D. Nychka, S. A. Mickelson, J. Edwards, M. Versteinen, and A. Wegener, “A methodology for evaluating the impact of data compression on climate simulation data,” in Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’14. New York, NY, USA: Association for Computing Machinery, 2014, p. 203–214. [Online]. Available: https://doi.org/10.1145/2600212.2600217
  33. [33]D. Tao, S. Di, H. Guo, Z. Chen, and F. Cappello, “Z-checker: A framework for assessing lossy compression of scientific data,” The International Journal of High Performance Computing Applications, vol. 33, no. 2, p. 285–303, Nov. 2017. [Online]. Available: http://dx.doi.org/10.1177/1094342017737147
  34. [34]M. Sprenger and H. Wernli, “The lagranto lagrangian analysis tool - version 2.0,” Geoscientific Model Development, vol. 8, no. 8, pp. 2569–2586, 2015. [Online]. Available: https://gmd.copernicus.org/articles/8/2569/2015/DOI
  35. [35]K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, “Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast,” 2022. [Online]. Available: https://arxiv.org/abs/2211.02556
  36. [36]R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Ale, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Hu et al., “Learning skillful medium-range global weather forecasting,” Science, vol. 382, no. 6677, pp. 1416–1421, 2023.DOI
  37. [37]C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Allen, J. Brandstetter, P. Garvan, M. Riechert, J. A. Weyn, H. Dong, J. K. Gupta, K. Thambiratnam, A. T. Archibald, C.-C. Wu, E. Heider, M. Welling, R. E. Turner, and P. Perdikari, “A foundation model for the earth system,” Nature, May 2025. [Online]. Available: https://doi.org/10.1038/s41586-025-09005-y
  38. [38]N. Koldunov, T. Kölling, X. Pedruzo-Bagazgoitia, T. Rackow, R. Redler, D. Sidorenkov, K.-H. Wieners, and F. A. Ziemen, “nextgems: output of the model development cycle 3 simulations for icon and ifs,” 2023. [Online]. Available: https://doi.org/10.26050/WDCC/nextGEMS_cyc3
  39. [39]J. Liu, S. Di, K. Zhao, X. Liang, S. Jin, Z. Jian, J. Huang, S. Wu, Z. Chen, and F. Cappello, “High-performance effective scientific error-bounded lossy compression with auto-tuned multi-component interpolation,” Proc. ACM Manag. Data, vol. 2, no. 1, Mar. 2024. [Online]. Available: https://doi.org/10.1145/3639259
  40. [40]J. Liu, S. Di, S. Jin, K. Zhao, X. Liang, Z. Chen, and F. Cappello, “Scientific error-bounded lossy compression with super-resolution neural networks,” in 2023 IEEE International Conference on Big Data (Big-Data), 2023, pp. 229–236.DOI
  41. [41]World Meteorological Organization, Manual on Codes (WMO-No. 306), Volume I.2: International Codes — Part B: Binary Codes (FM 92-XIV GRIB, Edition 2), 2025th ed., World Meteorological Organization, Geneva, Switzerland, 2025, accessed 2026-02-10. [Online]. Available: https://community.wmo.int/about-manual-codes-volume-i2
  42. [42]R. Rew and G. Davis, “netCdf: an interface for scientific data access,” IEEE Computer Graphics and Applications, vol. 10, no. 4, pp. 76–82, 1990. [Online]. Available: https://doi.org/10.1109/38.56302
  43. [43]M. Folk, G. Heber, Q. Koziol, E. Pourmal, and D. Robinson, “An overview of the hdf5 technology suite and its applications,” in Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases, ser. AD ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 36–47. [Online]. Available: https://doi.org/10.1145/1966895.1966900
  44. [44]Zarr Developers, “Zarr: chunked, compressed, N-dimensional arrays,” https://github.com/zarr-developers/zarr-python, 2026, accessed 2026-01-26.
  45. [45]P. Deutsch and J.-L. Gailly, “RFC 1950: Zlib compressed data format specification version 3.3,” https://www.rfc-editor.org/rfc/rfc1950, 1996, accessed 2026-02-10.
  46. [46]P. Deutsch and J.-L. Gailly, “RFC 1951: DEFLATE compressed data format specification version 1.3,” https://www.rfc-editor.org/rfc/rfc1951, 1996, accessed 2026-02-10.
  47. [47]P. Deutsch and J.-L. Gailly, “RFC 1952: GZIP file format specification version 4.3,” https://www.rfc-editor.org/rfc/rfc1952, 1996, accessed 2026-02-10.
  48. [48]J. Seward, “bzip2 and libbzip2,” 1998, accessed 2026-02-10. [Online]. Available: https://sourceware.org/bzip2/manual/manual.html
  49. [49]K. Masui, M. Amiri, L. Connor, M. Deng, M. Fandino, C. Höfer, M. Halpern, D. Hanna, A. Hincks, G. Hinshaw, J. Parra, L. Newburgh, J. Shaw, and K. Vanderlinde, “A compression scheme for radio data in high performance computing,” Astronomy and Computing, vol. 12, p. 181–190, Sep. 2015. [Online]. Available: http://dx.doi.org/10.1016/j.ascom.2015.07.002
  50. [50]P. Lindstrom and M. Isenburg, “Fast and efficient compression of floating-point data,” IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 5, pp. 1245–1250, 2006. [Online]. Available: https://doi.org/10.1109/TVCG.2006.143
  51. [51]M. Burtscher and P. Ratanaworabhan, “Fpc: A high-speed compressor for double-precision floating-point data,” IEEE Transactions on Computers, vol. 58, no. 1, pp. 18–31, 2009. [Online]. Available: https://doi.org/10.1109/TC.2008.131
  52. [52]G. Wallace, “The jpeg still picture compression standard,” IEEE Transactions on Consumer Electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992. [Online]. Available: https://doi.org/10.1109/30.125072
  53. [53]ITU-T, “Advanced video coding for generic audiovisual services (H.264),” https://www.itu.int/rec/T-REC-H.264, 2026, accessed 2026-02-10.
  54. [54]——, “High efficiency video coding (H.265),” https://www.itu.int/rec/T-REC-H.265, 2026, accessed 2026-02-10.
  55. [55]Alliance for Open Media (AOMedia), “AV1 bitstream & decoding process specification,” https://aomediacodec.github.io/av1-spec/, 2026, accessed 2026-02-10.
  56. [56]J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” 2017. [Online]. Available: https://arxiv.org/abs/1611.01704
  57. [57]D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” 2018. [Online]. Available: https://arxiv.org/abs/1809.02736
  58. [58]Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” 2020. [Online]. Available: https://arxiv.org/abs/2001.01568
  59. [59]J. Liu, H. Sun, and J. Katto, “Learned image compression with mixed transformer-cnn architectures,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1–10.arxiv.org/abs/2303.14978
  60. [60]S. Tomar, “Converting video formats with ffmpeg,” Linux J., vol. 2006, no. 146, p. 10, Jun. 2006.DOI
  61. [61]J. Woodring, S. Mniszewski, C. Brislawn, D. DeMarle, and J. Ahrens, “Revisiting wavelet compression for large-scale climate data using JPEG 2000 and ensuring data precision,” in 2011 IEEE Symposium on Large Data Analysis and Visualization, 2011, pp. 31–38. [Online]. Available: https://doi.org/10.1109/LDAV.2011.6092314
  62. [62]S. Lakshminarayanan, H. Shah, S. Ethier, S.-H. Ku, C. S. Chang, S. Klasky, R. Latham, R. Ross, and N. F. Samatova, “Isabela for effective in situ compression of scientific data,” Concurrency and Computation: Practice and Experience, vol. 25, no. 4, pp. 524–540, 2013. [Online]. Available: https://doi.org/10.1002/cpe.2887
  63. [63]J. Liu, S. Di, K. Zhao, S. Jin, D. Tao, X. Liang, Z. Chen, and F. Cappello, “Exploring autoencoder-based error-bounded compression for scientific data,” 2023. [Online]. Available: https://arxiv.org/abs/2105.11730
  64. [64]G. Li, M. Alhumaidi, S. Skiadopoulos, I. Hoteit, and P. Kalnis, “Extreme error-bounded compression of scientific data via temporal graph autoencoders,” IEEE Transactions on Knowledge and Data Engineering, vol. 38, no. 6, pp. 3597–3610, 2026.
  65. [65]B. N. Taylor and C. E. Kuyatt, “Guidelines for evaluating and expressing the uncertainty of NIST measurement results,” National Institute of Standards and Technology, Tech. Rep. Technical Note 1297, 1994, accessed 2026-02-10. [Online]. Available: https://www.nist.gov/pml/nist-technical-note-1297
  66. [66]Joint Committee for Guides in Metrology (JCGM), “Evaluation of measurement data — guide to the expression of uncertainty in measurement (GUM),” https://www.bipm.org/en/committees/jc/jcgm/publications, 2008, accessed 2026-02-10.DOI
  67. [67]——, “International vocabulary of metrology — basic and general concepts and associated terms (VIM), 3rd edition,” https://www.bipm.org/en/committees/jc/jcgm/publications, 2012, accessed 2026-02-10.DOI
  68. [68]P. Courtier, J.-N. Thépaut, and A. Hollingsworth, “A strategy for operational implementation of 4D-Var, using an incremental approach,” Quarterly Journal of the Royal Meteorological Society, vol. 120, no. 519, pp. 1367–1387, 1994. [Online]. Available: https://doi.org/10.1002/qj.49712051912
  69. [69]E. Kalnay, Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 2002.DOI

Paper details

vlnc:W_yucsz738
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents