WorksPhysics and Astronomy
ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction
Abstract
Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable. We recast the reconstruction of the underlying matter field from observed lightcones as a video-to-video translation task by introducing ViViViT, a vision transformer that factorizes attention across transverse frames and radial evolution. On reconstructed matter fields from 21cm neutral hydrogen simulations over , ViViViT preserves spectral power and phase alignment within a few percent of the ground truth up to the Nyquist scale, and three-point correlations for all triangle configurations with sides up to , outperforming a convolutional 3D-U-Net baseline across all measured statistics. By encoding the evolving geometry of the cosmological observable itself, ViViViT marks a step toward field-level architectures that natively respect cosmic time, opening the path to time-aware reconstruction of cosmic tracers and targeted searches for redshift-dependent signatures in 21cm data.
Introduction
The redshifted 21 cm hyperfine transition of neutral hydrogen opens a direct, volumetric window onto the evolution of the Universe. The 21 cm signal is a tracer of the underlying distribution of dark matter, following its gravitational growth across cosmic time. Mapping its emission from resolved and unresolved galaxies and diffuse gas allows us to probe the local Universe, track how the first luminous sources heated and ionized the intergalactic medium, and potentially access the Dark Ages prior to the formation of the first stars (see, e.g., [1, 2, 3]). The differential brightness temperature (relative to the CMB) is, however, not a direct image of the matter density : the signal is modulated through the neutral hydrogen fraction, gas and spin temperatures, radiation backgrounds, and line-of-sight velocities. The matter distribution shapes the signal through both the gas density and the abundance and clustering of galaxies, whose radiation heats and ionizes the surrounding intergalactic medium [4]. Exotic energy injection from sources such as decaying dark matter and primordial black holes can also alter the thermal history in a redshift-dependent way [5, 6, 7]. All these effects combine into a complex, non-linear mapping between the underlying matter field and the observed 21 cm signal, with both local and time-varying contributions.
Reconstructing the matter distribution. Inverting this mapping would give direct access to the matter field shared by all cosmological tracers. The evolution and statistical properties of are governed by gravity, providing a more robust and direct connection to early-Universe imprints [8, 9].
Accessing this field would enable clean -point statistical measurements, cross-correlations with other tracers, and a powerful probe to test the nature of cosmic acceleration, dark-matter physics, and departures from general relativity (see, e.g., [10, 11, 12, 9, 13]). Moreover, while depending on the astrophysical forward model, such a reconstruction would isolate how specific heating and ionization effects shape the signal, separating the coupled contributions of astrophysical processes from cosmic structure formation [14].
Towards tomographic 21 cm observations. Current radio interferometers (HERA [15], LOFAR [16], MWA [17]) target statistical detection of 21 cm power spectra at –12 [18, 19]. The Square Kilometre Array Observatory (SKAO) will extend the sensitivity to with baseline lengths enabling arcminute angular resolution, potentially achieving tomographic imaging of the neutral hydrogen structure [20, 21].
Related work. Cosmological reconstruction has long inferred primordial initial conditions from late-time large-scale structure through Bayesian forward modeling (e.g., [22]), score-based posterior sampling (e.g., [23]) and convolutional methods (e.g., [24, 25]). Those approaches connect a late-time field to a primordial snapshot, while our aim is to map between two evolving fields on the same lightcone. Within 21 cm cosmology, existing reconstruction approaches largely inherit the inductive biases of convolutional networks, which have been used for parameter inference [26, 27], matter-density reconstruction from 2D 21 cm slices [14], and for foreground and missing-mode mitigation [28]. Generative models have also been used for foreground mitigation [29]. A ViT–U-Net model has been employed to emulate 21 cm volumes from matter and halo fields [30] and recently Pietschke et al. [31] introduced a self-supervised ViT encoder of 21 cm lightcones for cross-simulator transfer learning and parameter inference. Time dependence has been included as conditioning to CNNs to reconstruct the ionization fraction [32], and in RNN architectures for parameter inference [33]. In this context, video models provide a mature basis to probe and generate spatiotemporal data, based on space–time transformers [34, 35] and diffusion systems [36, 37, 38].
Our contribution. The geometry of the lightcone implies that the radial axis is simultaneously comoving distance and look-back time: each transverse plane encodes a different epoch (and spatial location), so their sequence can be interpreted as a video of cosmic evolution (Figure 1, left), and reconstruction must account for correlations along that sequence. We therefore recast 21 cm field reconstruction as video-to-video translation and develop ViViViT, a deterministic encoder–decoder transformer based on the ViViT architecture of [35]. By factorizing attention into spatial and temporal components, our approach breaks the quadratic cost of full spacetime attention, allowing us to scale the model to larger volumes. Moreover, by separating transverse and radial attention, we provide a natural framework to capture signatures with different morphology and time dependence. We test the model on a matter reconstruction task and compare it to a convolutional U-Net baseline. Our results show that the transformer performs accurate field-to-field translation across the entire lightcone, improving the reconstruction of the measured two- and three-point statistics relative to the tested convolutional baseline.

Figure 1. A Video-to-Video Translation. (a) Longitudinal temperature and density slices from simulated 21 cm lightcones. Redshift changes along the line of sight, with both brightness temperature and matter density evolving across cosmic time. Dashed lines bound the reconstructed interval. (b) ViViViT alternates attention within transverse frames and along the line of sight. (c) An encoder–decoder translates the complete input sequence into a density sequence on the same grid. The decoder is initialized with a direct projection of the encoder’s output.
Method
21 cm physics and forward model
The differential brightness temperature describes the redshifted 21 cm signal in emission or absorption relative to the CMB. Neglecting peculiar-velocity gradients, the optically thin signal from the diffuse intergalactic medium can be parameterized as
where is the baryon density contrast, is the neutral hydrogen fraction (with the ionized fraction), K is the CMB temperature and is the spin temperature, set by radiative coupling to the CMB, collisions and Wouthuysen–Field coupling to the Ly background. Peculiar velocities modify the line optical depth through line-of-sight velocity gradients and shift the correspondence between comoving position and observed frequency [1, 2]. In standard thermal history models, the gas cools adiabatically after thermally decoupling from the CMB. While collisions remain effective, they couple toward the gas kinetic temperature , so that the signal appears in absorption. As the Universe expands and gas density drops, collisional coupling becomes highly inefficient and radiative coupling to the CMB then dominates, pulling toward , suppressing the diffuse 21 cm signal. During cosmic dawn, Ly photons from the first stars introduce Wouthuysen–Field coupling, driving toward the Ly colour temperature, which is similar to . This produces absorption, if the gas remains colder than the CMB. X-ray heating raises ; with effective spin coupling, the signal then appears in emission (). During patchy reionization, ionized regions locally suppress the diffuse 21 cm signal. Fluctuations in can dominate the brightness temperature structure when spin temperature fluctuations become subdominant. Temporal structure of different stages depends on the source population, possible exotic sources, and their radiative properties.
We generate paired temperature and density lightcones using 21cmFAST [39, 40], a simulation framework which combines perturbation theory for the density field with semi-numerical prescriptions for the ionization and thermal state of the intergalactic medium, to produce large-volume realizations of the 21 cm signal. Our task maps to on a volume spanning , with voxels of comoving size 4 Mpc. We choose this interval to include morphologically rich structure across absorption, heating and patchy reionization, testing reconstruction as the temperature–density relation evolves on a fixed grid. Each channel uses one global mean and standard deviation; we do not normalize individual slices, so the model must learn the evolution of the field amplitude along the lightcone.
The cosmological parameters are fixed to the Planck 2018 values: , , , , and [41]. For the astrophysical parameters, we employ the parametrization of [42], with prior ranges following [29]: the star formation efficiency at is sampled from , the ionizing photon escape fraction from , and the X-ray luminosity per unit star formation rate from in units of . Mass dependence of these quantities employs power-law scaling with indices and . From each simulation run, we extract two volumetric fields: differential brightness temperature encoding the observable 21 cm signal and matter density contrast . With multiple output channels, the model could jointly reconstruct matter density and the neutral fraction , including its spatial topology. We train on 3000 realizations with a 90/10 train–validation split and reserve a separate additional set of 200 lightcones for all reported statistics.
21cmFAST evolves the field in coeval cubic boxes of side Mpc, periodic on all three Cartesian axes, at a discrete set of node redshifts. It then connects the lightcone by extracting and interpolating transverse slices from the boxes whose redshifts bracket each radial pixel [39, 40]. The temperature–density relation hence evolves, but periodicity imprints from the same comoving structure enter the simulated lightcone at different times. This construction differs from an observed lightcone, where separated radial intervals occupy distinct comoving regions. A model with global line-of-sight reach can use the repeated structure imprints, while one with a shorter receptive field cannot. In Appendix § 10, we test our model in a setting without any simulator artifacts to isolate the receptive-field advantage.
Video-to-video transformer
We build an encoder–decoder transformer that translates an input temperature lightcone into a density lightcone on the same grid. In this work, we focus on a one-channel volume. Additional input channels could combine the 21 cm lightcone with co-spatial galaxy or line-intensity maps, while multiple output channels could jointly reconstruct density and neutral fraction. Before encoding, a strided 3D convolution forms overlapping spatial patches, which mitigates high-frequency discontinuities at patch boundaries. Full attention over all tokens would be computationally impractical; therefore, we factorize attention into alternating spatial and temporal components (see central panel of Figure 1), reducing the cost from to , where [35]. We then project each patch to a -dimensional embedding space, including a learned factorized positional encoding to retain spatial and temporal information (see Appendix § 7 for details).
As visually summarized in the right panel of Figure 1, the encoder forms a global contextual representation of the input tokens. A feed-forward projection initializes the decoder from the encoder output. This initialization preserves the token-wise correspondence between the input and target lightcones while passing the encoder’s spatial and radial context to the decoder. The decoder uses self- and cross-attention, followed by a local convolutional upsampling head that restores the full grid. Temporal self-attention is lower-triangular at patch resolution. Appendix § 7 provides the full architecture and training details.
We use a pixel-wise loss that penalizes absolute deviations between predicted and target fields, combined with a gradient-difference loss that enforces similarity in local structure and edge sharpness. The complete objective and a controlled – ablation are reported in Appendix § 8.
Experiments
We compare ViViViT with a four-level 3D U-Net. In Appendix § 10, we first compare the temporal receptive fields of the two architectures on a synthetic benchmark, varying the U-Net depth: ViViViT retains global context across the entire temporal direction, while the U-Net loses long-range accuracy as the target correlation length grows. For our main cosmological task, we train both models on the same 3000 lightcones and evaluate them on a held-out set of 200 lightcones by measuring the two-point and three-point statistics of the reconstructed density fields, which probe two-point correlations and non-Gaussian mode coupling relevant for cosmological inference [43, 44, 45, 46]. The code and trained checkpoints are available.
Field morphology. Figure 2 presents a qualitative comparison of the reconstructed density fields with the ground truth , along a longitudinal slice through the lightcone. The input 21 cm brightness temperature (left panel) exhibits the characteristic evolution of the signal: absorption deepening toward –, followed below by heated and ionized regions that grow until ionized bubbles with dominate at . The reconstruction task requires inverting this non-monotonic mapping while disentangling astrophysical evolution from gravitational clustering. Both architectures successfully recover the large-scale cosmic web, correctly placing major overdensities and underdensities. The architectures differ in the morphological details they recover. The U-Net baseline tends to produce overly smooth fields, blurring the sharp contrast between filaments and voids. In contrast, the ViViViT model retains sharper features and better preserves the texture of the density field. Crucially, the reconstruction remains stable also at low redshift (), where the 21 cm signal is highly non-linear and dominated by ionized regions with near-zero brightness temperature, indicating that the models successfully adapt to the evolving relationship between and .
Figure 2. ViViViT retains finer density structure than the tested U-Net on a held-out longitudinal slice. Input 21 cm brightness temperature (in mK), target matter density, and both reconstructions share a color scale within each field type.
Two-point fidelity. We divide each reconstructed lightcone into three adjacent regions and measure power after tapering only the non-periodic radial direction (see Appendix § 9 for details on the estimator). The power spectrum measures the variance of Fourier modes as a function of scale, whereas the cross-correlation coefficient measures phase alignment between the reconstructed and target fields (see Appendix § 9 for definitions). In Figure 3, the U-Net loses small-scale power earliest in the lowest-redshift bin, where growing ionized regions locally erase the temperature signal. ViViViT keeps the median power ratio near unity over most measured modes. More importantly, it maintains cross-correlation coherence to in the central bin, while the tested U-Net crosses that threshold near . These diagnostics show that ViViViT improves both small-scale power recovery and cross-correlation with the target field relative to the tested U-Net. The longitudinal slices in Figure 2 illustrate the corresponding differences in reconstructed morphology.

Figure 3. ViViViT recovers the measured two- and three-point statistics more accurately than the tested U-Net. Top: Reconstructed-to-target power spectrum ratio and cross-correlation coefficient in three consecutive redshift bins; Bottom: reconstructed-to-target isotropic bispectrum ratio for all retained closed triangles, ordered by their largest mode and shape. Curves are medians over 200 held-out lightcones and bands span the 16th–84th and 2.5th–97.5th percentiles. ViViViT (blue) stays closer to unit power and phase coherence at large and preserves most three-point configurations, while the U-Net (orange) develops a scale- and redshift-dependent deficit.
Non-Gaussian fidelity. Non-linear gravitational collapse couples Fourier modes of , so we next measure the bispectrum to quantify whether the non-Gaussian structure is preserved (see Appendix § 9 for the estimator construction). ViViViT reconstructs the ground-truth bispectrum to within about 5% for most retained triangle configurations, while the U-Net reconstruction error grows toward smaller scales and lower redshift (bottom panel of Figure 3). Once local brightness temperature is suppressed inside an ionized region, recovering the surrounding density requires context from outside that region.
Line-of-sight-aligned three-point structure. The isotropic result in Figure 3 averages over triangle orientation. To target the configurations that encode long-range correlations along the line of sight, we bin each bispectrum mode into five bins. We then isolate the bispectrum configurations aligned along the line of sight by keeping only triangles whose long mode falls in the line-of-sight-aligned angular bin (). This selection fixes the long radial mode while leaving the transverse shape free, and Figure 4 shows the resulting amplitude ratio for the squeezed (, in practice and ) and folded ( and ) families. ViViViT better preserves three-point amplitudes for triangles whose longest-wavelength leg is aligned with the line of sight. The U-Net loses the radial long-mode power progressively with decreasing redshift: the squeezed ratio falls from near unity in the farthest redshift bin to at the smallest scales of the closest one, and the folded family follows the same trend more mildly. The U-Net architecture does not preserve accurate spectral information when processing the lightcone section corresponding to the epoch of reionization, failing to fully recover structure information from the geometry of ionized regions.
Figure 4. ViViViT retains line-of-sight-aligned three-point structure more accurately than the tested U-Net. Reconstructed-to-target bispectrum ratios for squeezed (top) and folded (bottom) triangle families across the three redshift bins. Curves are medians over the held-out set and bands span the 16th–84th percentiles. Sharp excursions occur where the target bispectrum approaches zero and the ratio becomes ill-conditioned; they should not be interpreted as enhanced reconstruction signal. The plotted ratios and percentile bands are clipped to the displayed vertical range.
Conclusions
A cosmological lightcone can be viewed as a video across cosmic time, with successive transverse slices encoding successive epochs of structure formation at distinct spatial regions. We have introduced ViViViT, a spacetime-aware transformer that processes the lightcone through factorized attention, learning a one-shot translation from one cosmological field to another. On held-out 21cmFAST lightcones, we showed that ViViViT outperforms the tested convolutional baseline in preserving the measured two- and three-point statistics of the target field. A separate synthetic benchmark demonstrates its ability to exploit long-range radial information when the reconstruction task requires it.
Limitations. As discussed in § 2.1, the 21cmFAST forward model introduces periodicity artifacts from the same comoving structure at different times, differently from a real observed lightcone and that may provide additional reconstruction information. The synthetic benchmark (see Appendix § 10 shows that ViViViT can exploit long-range radial information when it is physically present. This benchmark establishes long-context capability, but does not isolate its contribution to the cosmological task. We aim to extend our method to more refined lightcone simulations and further recast the field translation task as biased tracer emulation from matter density to observed lightcone. Our mapping is a deterministic reconstruction: the model produces the entire volume in a single forward pass, but it does not preserve posterior uncertainty. In future work, our aim is to adapt this infrastructure to conditional sampling, accessing the conditional distribution rather than its median. Moreover, applications to real datasets must include observational and astrophysical systematics and project the physical field to an instrumental measurement. In particular, the galactic and extragalactic foregrounds confine reliable spectral information outside a wedge-shaped region in Fourier space. Neural approaches to foreground mitigation [28, 29] could be combined with our framework, and the effectively global receptive field of factorized attention is well suited to inpainting the lost modes from the surviving, correlated ones outside the wedge. A complementary route against both the wedge and the intrinsic information loss inside ionized regions is to feed the network several tracers of the same underlying mass field: ViViViT extends naturally to multi-channel input, exploiting cross-correlations between tracers to potentially break the foreground degeneracy. We leave a detailed study of these systematics and their mitigation for future work.
Beyond the present reconstruction, a natural possible extension is to target redshift-dependent physical processes whose signature is distributed across the lightcone. Exotic energy injection from decaying or annihilating dark matter and from primordial black holes can modify the thermal and ionization history of the intergalactic medium [5, 6, 7, 47, 48], while modified growth histories alter the evolution of the matter field. These effects are typically small in amplitude and easily confused with astrophysical uncertainties in power-spectrum analyses, yet coherent across the lightcone.
To our knowledge, ViViViT is the first video-to-video transformer applied to cosmological matter-field lightcone reconstruction. More broadly, it provides a flexible framework that directly encodes the lightcone spacetime structure, enabling more robust extraction of physical information from these complex datasets.
Acknowledgments and Disclosure of Funding
The authors thank Nicola Bellomo, Jessie de Kruijf, Sarah Libanore, Phoenix, Francesco Spezzati and Benjamin Wandelt for insightful feedback and discussions. The authors also thank the anonymous reviewers for their constructive feedback, which was useful in refining the presentation of our work. This work is partly supported by ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing, funded by European Union - NextGenerationEU. ML acknowledges support by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) Bando 2022 - grant 20228RMX4A. AR acknowledges funding from the Italian Ministry of University and Research (MIUR) through the “Dipartimenti di eccellenza” project “Science of the Universe”.
LLM Usage
In preparing this paper, we used large language models as assistive tools. In particular, LLMs were used for editing the draft for clarity, distributed training setup, running numerical experiments, and polishing figures. The authors take full responsibility for the content of this paper.
Architecture and training details
Tables 1 and 2 report the two model specifications; Table 3 lists the optimization settings shared by both architectures. On four A100 GPUs, one complete training-and-validation epoch takes minutes for ViViViT and minutes for U-Net. We extracted minimum-validation-loss trained checkpoints after 260 and 409 epochs, respectively.
Tokenization and position. The tokenizer applies a 3D convolution with kernel and stride , where denotes the temporal patch size and the spatial patch size in voxel units. Spatial neighbors therefore overlap by two voxels, whereas the radial patches remain aligned and non-overlapping. We use this strided convolution to avoid high-frequency discontinuities at patch boundaries, which manifest as spurious power in the Fourier spectrum at wavenumbers where is the spatial patch size, converted to physical units. The tokenizer maps the volume to , where is the batch size, is the number of temporal patches, is the number of spatial patches, and is the embedding dimension. We also add factorized learned position vectors,
where , , and are independent learnable parameters initialized with small random values. This additive decomposition provides unique encodings while using positional parameters instead of a table over the full token grid. The input lightcone is thus represented as a sequence of tokens with embedded positional information.
Attention and decoding. Each block applies spatial attention independently within every radial patch plane, followed by temporal attention independently at every spatial token. Temporal self-attention uses lower-triangular masks, while temporal cross-attention to the encoder is unmasked. The decoder is initialized with a direct projection of the encoder’s output via a non-linear MLP, and decoder blocks combine masked self-attention with encoder cross-attention. A local convolutional head mixes the coarse token grid and restores resolution through two joint sub-pixel rearrangement stages. Token representations are first projected from embedding dimension to an internal feature dimension . The resulting features are reshaped to form a 3D grid matching patch resolution: . Before upsampling, this coarse feature grid undergoes mixing through circular convolutions, distributing information across tile boundaries before upsampling to avoid the magnification of potential artifacts. Resolution is recovered through two stages, each applying upsampling jointly along all three axes by employing pixel shuffle operations followed by residual refinement. The transformation is a deterministic tensor rearrangement, requiring no interpolation or learned kernels. This approach, first introduced in super-resolution tasks [49], effectively separates the upsampling (rearrangement) from the feature generation (convolution), preventing artifacts from propagating both spatially and in Fourier space. The final layers apply residual refinement and project to the output channel dimension through additional circular convolutions, producing the reconstructed field at full resolution .
| Patch kernel / stride | / |
| Token grid | |
| Embedding dimension | |
| Attention heads | |
| Encoder blocks | |
| Decoder blocks | |
| Feed-forward dimension | |
| Decoder initialization | MLP projection |
| Upsampling feature width | |
| Dropout |
Table 1. ViViViT
| Convolution kernel | |
| Initial feature width | |
| Encoder widths | |
| Bottleneck width | |
| Decoder widths | |
| Downsampling levels | |
| Upsampling levels | |
| Bottleneck grid |
Table 2. 3D U-Net
| Loss | pixel gradient difference |
| Optimizer | AdamW, , weight decay |
| Learning rate | with warmup and cosine decay |
| Gradient clipping | norm |
| Batch / accumulation / GPUs | |
| Maximum epochs / patience | |
| Precision | automatic mixed precision |
Table 3. Shared optimization settings for the cosmological task.
Training objective and loss ablation
For target and reconstruction over volume , the training objective is
The pixel term measures voxel-wise fidelity without quadratically upweighting locally large residuals, which can arise where the 21 cm signal weakly constrains the density; the directional term instead penalizes mismatches in local field variations. We isolate the norm choice on a reduced task with 500 simulations and the same 80-epoch budget for and versions of both terms. Figure 5 shows compatible large-scale performance with a modest advantage for at large scales.

Figure 5. Loss-norm ablation for . Lines show medians over 27 held-out volumes; bands span the 16th–84th percentiles. The two objectives are largely consistent across the measured scales, while the power spectrum ratio mildly favors the training objective.
Summary-statistic estimators
We split each reconstructed volume into three contiguous regions. The transverse axes inherit periodic simulation boundaries, but the radial field evolves and is not periodic within a redshift bin, requiring radial tapering for proper power spectrum and bispectrum estimation.
Before transforming to Fourier space, we then multiply both reconstruction and target ground-truth fields by the same one-dimensional cosine-tapered window along the radial axis, where represents the radial frame index within a given redshift bin, with frames per bin,
with taper fraction , i.e. unity over the central half of the region and half-cosine ramps over the outer quarters. Each region is first reduced to a fluctuation field by subtracting its mean, and the tapered field
is exactly zero-mean by construction. The window multiplies ground truth and reconstruction identically, modifying the measured spectra of both fields. The window attenuation is compensated by the moments , entering the estimators below. No mass-assignment deconvolution is applied because the fields are defined directly on the grid rather than painted from particle simulations.
Power spectrum. For two tapered fields on the grid we estimate
where the bin contains the Fourier modes with magnitudes in the spherical shell of width , and is their mean wavenumber. Applying Eq. (8) to the reconstruction and the target , we define the cross-correlation coefficient
in which the factors cancel.
Bispectrum. We bin Fourier modes into spherical shells of width centered at (), so that a shell collects the modes with . For each shell we construct the band-limited real-space field by inverse-transforming with all modes outside the shell set to zero, and use the standard FFT estimator [50, 45, 51]
where counts closed mode triplets . We retain ordered closed shell triples with . We further select triangles with each side satisfying , close to two-thirds of the grid Nyquist scale, which guarantees that the real-space triple product of Eq. (10) picks up no spurious, aliased triangle closures.
The estimator defined in Eq. (10) averages over all triangle orientations. To resolve radial long-mode correlations, we measure the alignment of a mode with the line of sight by and split the range into five equal-width bins with index (with collecting the most radially aligned modes, ). The band-limited field is then built from the shell modes within the alignment bin . Replacing in Eq. (10) then defines the bispectrum of one orientation cell , normalized by the mode triplets of that cell. Summing the cells with weights recovers the isotropic estimator. A directional selection retains a subset of orientation cells (e.g. on the smallest- mode in Figure 4). For each triangle configuration and simulation , we then compute the mode-count-weighted average over the selected orientation cells,
estimating the bispectrum over the union of the retained cells. Since the same window is applied to both fields, the ratio and cross-correlation statistics are only mildly sensitive to the window choice.
Non-periodic global-receptive-field benchmark
We first directly compare long-horizon reconstruction capabilities and the temporal receptive fields of the two architectures using a synthetic benchmark. We build paired realizations of scalar “videos” on a grid, driven by the same per-slice Gaussian random field with transverse spectrum , unit variance and zero mean. The fields are defined by the recursive relations
with the recursion acting independently at every transverse pixel and each frame being unit-variance by construction. Since no transverse context exists by construction, reconstructing from forces a model to integrate the input over the temporal axis. We fix so the input field loses memory of its past states within a few slices. We tune to control the long-range correlation of the target field across frames, parameterizing it by the frame separation defined in terms of the temporal autocorrelation . We sweep on videos and compare a ViViViT with U-Net models of resolution levels.
Since the input is an invertible filter of the same Gaussians that drive the target, a model that sees the entire input history can potentially reconstruct the target exactly. We quantify reconstruction quality by the normalized mean squared error,
evaluated per deep-axis slice over the 50 held-out realizations. In Figure 6 we show the NMSE averaged over the 64 central slices and over test simulations. The 3D U-Net variants, due to their finite receptive fields, show increasing reconstruction error as the target’s memory grows. Each added resolution level extends the usable correlation length, but the reconstruction error follows the same trend, and by even the deepest U-Net misses the target almost entirely (NMSE ). In contrast, ViViViT reconstructs the target at the level at every . The reconstructed slices in the right panel of Figure 6 visually show this result for the case: the deepest U-Net recovers the short-range component of the target, but loses the structure spanning the temporal axis, which the transformer reproduces. When a target is coupled to the observable through correlations longer than the convolutional receptive field, a bounded-reach model has an intrinsic reconstruction limit that a ViViViT can lift through a global temporal reach. In this work, we are using factorized attention to achieve both short and long range couplings, but various time-aware architectures can be explored to efficiently encode such multi-range correlations.

Figure 6. Global temporal receptive field. A bounded receptive field caps reconstruction quality once the target’s memory exceeds it, while the global temporal reach of ViViViT does not degrade. (a) Normalized mean-squared error, Eq. (13), pooled over 50 held-out realizations and averaged over the 64 central slices. ViViViT stays near as the memory reaches the full sequence; each bounded U-Net degrades as exceeds its usable reach. (b) The input decorrelates within a few slices, while the ground-truth target carries coherent structure spanning the whole axis. ViViViT reproduces that slice-spanning structure; the best-performing U-Net () recovers only its short-range component.
The cosmological task of § 3 is different in this respect. In a lightcone, distinct spatial regions at distinct epochs share a causal history that connects the evolution of the observed field. The brightness temperature is foremost an equal-time tracer of at a given redshift, so density reconstruction does not require global temporal reach by itself; intrinsically redshift-dependent signals, such as exotic energy injection or radiative couplings that accumulate along the lightcone, are the applications for which this capability may become necessary and where a bounded-reach U-Net may struggle to recover the full signal. This benchmark demonstrates that our transformer can exploit long-range temporal structure when the reconstruction task requires it.
References
- [1]Steven R. Furlanetto, S. Peng Oh, and Frank H. Briggs. Cosmology at low frequencies: The 21 cm transition and the high-redshift universe. Physics Reports, 433(4-6):181–301, 2006. doi: 10.1016/j.physrep.2006.08.002.DOI
- [2]Jonathan R. Pritchard and Abraham Loeb. 21 cm cosmology in the 21st century. Reports on Progress in Physics, 75(8):086901, 2012. doi: 10.1088/0034-4885/75/8/086901.DOI
- [3]Steven R. Furlanetto. The 21-cm Line as a Probe of Reionization. In Andrei Mesinger, editor, Understanding the Epoch of Cosmic Reionization: Challenges and Progress, volume 423 of Astrophysics and Space Science Library, pages 247–280. Springer International Publishing, 2016. doi: 10.1007/978-3-319-21957-8_9.DOI
- [4]Andrei Mesinger, Aaron Ewall-Wice, and Jacqueline Hewitt. Reionization and beyond: detecting the peaks of the cosmological 21 cm signal. Mon. Not. Roy. Astron. Soc., 439(4):3262–3274, 2014. doi: 10.1093/mnras/stu125.DOI
- [5]José Luis Bernal, Alvise Raccanelli, Licia Verde, and Joseph Silk. Signatures of primordial black holes as seeds of supermassive black holes. JCAP, 05:017, 2018. doi: 10.1088/1475-7516/2018/05/017. [Erratum: JCAP 01, E01 (2020)].DOI
- [6]Katie Short, José Luis Bernal, Alvise Raccanelli, Licia Verde, and Jens Chluba. Enlightening the dark ages with dark matter. JCAP, 07:020, 2020. doi: 10.1088/1475-7516/2020/07/020.DOI
- [7]Yitian Sun, Joshua W. Foster, Hongwan Liu, Julian B. Muñoz, and Tracy R. Slatyer. Inhomogeneous energy injection in the 21-cm power spectrum: Sensitivity to dark matter decay. Phys. Rev. D, 111(4):043015, 2025. doi: 10.1103/PhysRevD.111.043015.DOI
- [8]Julian B. Muñoz, Yacine Ali-Haïmoud, and Marc Kamionkowski. Primordial non-gaussianity from the bispectrum of 21-cm fluctuations in the dark ages. Phys. Rev. D, 92(8):083508, 2015. doi: 10.1103/PhysRevD.92.083508.DOI
- [9]Jessie de Kruijf, Eleonora Vanzan, Kimberly K. Boddy, Alvise Raccanelli, and Nicola Bartolo. Searching for blue-tiled power spectra in the dark ages. Phys. Rev. D, 111(6):063507, 2025. doi: 10.1103/PhysRevD.111.063507.DOI
- [10]Sabino Matarrese and Licia Verde. The effect of primordial non-Gaussianity on halo bias. Astrophys. J. Lett., 677:L77–L80, 2008. doi: 10.1086/587840.DOI
- [11]L. Guzzo et al. A test of the nature of cosmic acceleration using galaxy redshift distortions. Nature, 451:541–545, 2008. doi: 10.1038/nature06555.DOI
- [12]Mustapha Ishak. Testing General Relativity in Cosmology. Living Rev. Rel., 22(1):1, 2019. doi: 10.1007/s41114-018-0017-4.DOI
- [13]M. Abdul Karim et al. DESI DR2 results. II. Measurements of baryon acoustic oscillations and cosmological constraints. Phys. Rev. D, 112(8):083515, 2025. doi: 10.1103/tr6y-kpc6.
- [14]Pablo Villanueva-Domingo and Francisco Villaescusa-Navarro. Removing Astrophysics in 21 cm maps with Neural Networks. Astrophys. J., 907(1):44, 2021. doi: 10.3847/1538-4357/abd245.
- [15]David R. DeBoer et al. Hydrogen Epoch of Reionization Array (HERA). Publ. Astron. Soc. Pac., 129(974):045001, 2017. doi: 10.1088/1538-3873/129/974/045001.DOI
- [16]M. P. van Haarlem et al. Lofar: The low-frequency array. Astronomy & Astrophysics, 556:A2, 2013. doi: 10.1051/0004-6361/201220873.DOI
- [17]S. J. Tingay et al. The murchison widefield array: The square kilometre array precursor at low radio frequencies. Publications of the Astronomical Society of Australia, 30:e007, 2013. doi: 10.1017/pasa.2012.007.DOI
- [18]Aaron R. Parsons, Jonathan C. Pober, James E. Aguirre, Christopher L. Carilli, Daniel C. Jacobs, and David F. Moore. A per-baseline, delay-spectrum technique for accessing the 21 cm cosmic reionization signature. The Astrophysical Journal, 756(2):165, 2012. doi: 10.1088/0004-637X/756/2/165.DOI
- [19]Joshua S. Dillon, Adrian Liu, and Max Tegmark. A fast method for power spectrum and foreground analysis for 21 cm cosmology. Phys. Rev. D, 87(4):043005, 2013. doi: 10.1103/PhysRevD.87.043005.DOI
- [20]L. V. E. Koopmans et al. The Cosmic Dawn and Epoch of Reionization with the Square Kilometre Array. PoS, AASKA14:001, 2015. doi: 10.22323/1.215.0001.DOI
- [21]Suman Majumdar et al. Cosmology with Intensity Mapping via Statistics Beyond the Power Spectrum in the SKAO Era. 6 2026.arxiv.org/abs/2606.30200
- [22]Jens Jasche and Benjamin D. Wandelt. Bayesian physical reconstruction of initial conditions from large scale structure surveys. Mon. Not. Roy. Astron. Soc., 432:894–913, 2013. doi: 10.1093/mnras/stt449.DOI
- [23]Ronan Legin, Matthew Ho, Pablo Lemos, Laurence Perreault-Levasseur, Shirley Ho, Yashar Hezaveh, and Benjamin Wandelt. Posterior sampling of the initial conditions of the universe from non-linear large scale structures using score-based generative models. Mon. Not. Roy. Astron. Soc. Lett., 527(1):L173–L178, 2023. doi: 10.1093/mnrasl/slad152.DOI
- [24]Thomas Flöss and P. Daniel Meerburg. Improving constraints on primordial non-Gaussianity using neural network based reconstruction. JCAP, 02:031, 2024. doi: 10.1088/1475-7516/2024/02/031.DOI
- [25]Xinyi Chen, Fangzhou Zhu, Sasha Gaines, and Nikhil Padmanabhan. Effective cosmic density field reconstruction with convolutional neural network. Mon. Not. Roy. Astron. Soc., 523(4):6272–6281, 2023. doi: 10.1093/mnras/stad1868.DOI
- [26]Nicolas Gillet, Andrei Mesinger, Bradley Greig, Adrian Liu, and Graziano Ucci. Deep learning from 21-cm tomography of the Cosmic Dawn and Reionization. Mon. Not. Roy. Astron. Soc., 484(1):282–293, 2019. doi: 10.1093/mnras/stz010.DOI
- [27]Sultan Hassan, Sambatra Andrianomena, and Caitlin Doughty. Constraining the astrophysics and cosmology from 21 cm tomography using deep learning with the SKA. Mon. Not. Roy. Astron. Soc., 494(4):5761–5774, 2020. doi: 10.1093/mnras/staa1151.DOI
- [28]T. Lucas Makinen, Lachlan Lancaster, Francisco Villalescusa-Navarro, Peter Melchior, Shirley Ho, Laurence Perreault-Levasseur, and David N. Spergel. deep21: a deep learning method for 21 cm foreground removal. JCAP, 04:081, 2021. doi: 10.1088/1475-7516/2021/04/081.DOI
- [29]Nashwan Sabti, Ram Purandhar Reddy Sudha, Julian B. Muñoz, Siddharth Mishra-Sharma, and Taewook Youn. A generative modeling approach to reconstructing 21-cm tomographic data. Mach. Learn. Sci. Technol., 6(1):015039, 2025. doi: 10.1088/2632-2153/adb19c.DOI
- [30]Prasad Rajesh Posture, Yashrajsinh Mahida, Suman Majumdar, and Leon Noble. CosmoUiT: A Vision Transformer–UNet Hybrid for Fast and Accurate Emulation of 21-cm Maps from the Epoch of Reionization. JCAP, 06:011, 2026. doi: 10.1088/1475-7516/2026/06/011.DOI
- [31]Yannic Pietschke, Caroline Heneka, Ayodele Ore, and Romain Meriot. Cross-simulator transfer with foundation model summaries: Towards robust ska-era reionization inference, 2026. URL https://arxiv.org/abs/2608.26354.
- [32]Damien Korber, Michele Bianco, Emma Tolley, and Jean-Paul Kneib. PINION: Physics-informed neural network for accelerating radiative transfer simulations for cosmic reionization. Mon. Not. Roy. Astron. Soc., 521(1):902–915, 2023. doi: 10.1093/mnras/stad615.DOI
- [33]David Prelogovic, Andrei Mesinger, Steven Murray, Giuseppe Fiameni, and Nicolas Gillet. Machine learning astrophysics from 21 cm lightcones: impact of network architectures and signal contamination. Mon. Not. Roy. Astron. Soc., 509(3):3852–3867, 2021. doi: 10.1093/mnras/stab3215.DOI
- [34]Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 813–824. PMLR, 2021.arxiv.org/abs/2102.05095
- [35]Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision, pages 6816–6826, 2021. doi: 10.1109/ICCV48922.2021.00676.DOI
- [36]Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. In Advances in Neural Information Processing Systems, volume 35, pages 8633–8646, 2022. doi: 10.52202/068431-0628.DOI
- [37]Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanchen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A Space-Time Diffusion Model for Video Generation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11. ACM, 2024. doi: 10.1145/3680528.3687614.
- [38]Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models, 2025.arxiv.org/abs/2503.20314
- [39]Andrei Mesinger, Steven Furlanetto, and R enyeue Cen. 21cmfast: a fast, seminumerical simulation of the high-redshift 21-cm signal. Monthly Notices of the Royal Astronomical Society, 411 (2):955–972, 2011. doi: 10.1111/j.1365-2966.2010.17731.x.DOI
- [40]Julian B. Muñoz, Yuxiang Qin, Andrei Mesinger, Steven G. Murray, Bradley Greig, and Charlotte Mason. The impact of the first galaxies on cosmic dawn and reionization. Mon. Not. Roy. Astron. Soc., 511(3):3657–3681, 2022. doi: 10.1093/mnras/stac185.DOI
- [41]N. Aghanim et al. Planck 2018 results. VI. Cosmological parameters. Astron. Astrophys., 641: A6, 2020. doi: 10.1051/0004-6361/201833910. [Erratum: Astron.Astrophys. 652, C4 (2021)].DOI
- [42]Jaehong Park, Andrei Mesinger, Bradley Greig, and Nicolas Gillet. Inferring the astrophysics of reionization and cosmic dawn from galaxy luminosity functions and the 21-cm signal. Mon. Not. Roy. Astron. Soc., 484(1):933–949, 2019. doi: 10.1093/mnras/stz032.DOI
- [43]S. Matarrese, Licia Verde, and A. F. Heavens. Large scale bias in the universe: Bispectrum method. Mon. Not. Roy. Astron. Soc., 290:651–662, 1997. doi: 10.1093/mnras/290.4.651.DOI
- [44]Daniel J. Eisenstein and Wayne Hu. Baryonic features in the matter transfer function. Astrophys. J., 496:605, 1998. doi: 10.1086/305424.DOI
- [45]C. A. Watkinson, S. Majumdar, J. R. Pritchard, and R. Mondal. A fast estimator for the bispectrum and beyond – a practical method for measuring non-Gaussianity in 21-cm maps. Mon. Not. Roy. Astron. Soc., 472(2):2436–2446, 2017. doi: 10.1093/mnras/stx2130.DOI
- [46]William R. Coulton, Francisco Villaescusa-Navarro, Drew Jamieson, Marco Baldi, Gabriel Jung, Dionysios Karagiannis, Michele Liguori, Licia Verde, and Benjamin D. Wandelt. Quijote-png: Simulations of primordial non-Gaussianity and the information content of the matter field power spectrum and bispectrum. Astrophys. J., 943(1):64, 2023. doi: 10.3847/1538-4357/aca8a7.DOI
- [47]Yitian Sun, Joshua W. Foster, and Julian B. Muñoz. Constraining inhomogeneous energy injection from annihilating dark matter and primordial black holes with 21-cm cosmology, 2025.
- [48]Jessie de Kruijf, Nicola Bellomo, Kimberly K. Boddy, and Alyse Raccanelli. Signatures of primordial black holes on the dark ages. In preparation.
- [49]Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016. doi: 10.1109/CVPR.2016.207.DOI
- [50]Roman Scoccimarro. Gravitational clustering from χ² initial conditions. Astrophys. J., 542:1–8, 2000. doi: 10.1086/309519.DOI
- [51]Joseph Tomlinson, Donghui Jeong, and Juhan Kim. Efficient parallel algorithm for estimating higher-order polyspectra. Astron. J., 158(3):116, 2019. doi: 10.3847/1538-3881/ab3223.DOI

