betaFeedback

WorksComputer Science

FastGRT: Accelerating Training of 3D Gaussian Ray Tracing

Abstract

Ray Tracing enables rendering capabilities that are difficult to achieve with rasterization-based approaches such as 3D Gaussian Splatting (3DGS). However, the training cost of Gaussian-based ray tracing (GRT) is much higher than that of 3DGS. In this paper, we study how ray tracing using Gaussians executes on GPUs and show that a considerable fraction of the training cost arises from how the computation is organized. We develop FastGRT, which decouples ray traversal from blending and reuses work done in the forward pass for the backward pass. FastGRT further incorporates optimizations such as fast gradient accumulation and a hybrid BVH update strategy to tackle other major bottlenecks in the training pipeline. We implement FastGRT in Vulkan. To the best of our knowledge, it is the first Vulkan-based GRT implementation that supports both training and rendering. Compared to 3DGRT, FastGRT improves training time by 1.9×\times to 4.97×\times, while preserving the original rendering formulation. Our code is open source.

Ray tracing using 3D Gaussians

Figure 1. Ray tracing using 3D Gaussians. A ray is cast from each pixel and the hit Gaussians’ contributions are blended together to produce the pixel color.

Introduction

Novel view synthesis (NVS) reconstructs and learns a 3D scene from a collection of observed images from certain viewpoints and render photorealistic images from previously unseen viewpoints. Recent advances in 3D Gaussian Splatting [8] (3DGS) have made this NVS highly efficient both in terms of speed and quality. 3DGS is a rasterization-based algorithm. Its efficiency comes from projecting each 3D Gaussian into screen space using geometric transformations, which maps extremely well to modern GPUs. Applications like autonomous cars require wide-angle cameras (e.g., fisheye cameras), whose lenses are

prone to damage, which leads to distortion. Real cameras are often not ideal pinhole cameras; for such complex camera models, the projected 2D Gaussian no longer represents the underlying 3D Gaussian with minimal approximation, and this becomes a source of reconstruction error for complex models. Additionally, consider a camera that may observe an object only through its reflection in a mirror. Here, light from the object to the camera does not follow a straight path; instead, it is reflected by the mirror. This type of secondary light effect is difficult to represent with direct projections used by splatting.

Ray tracing solves the problem from the opposite direction (Figure 1): instead of projecting every scene element onto the image plane, it launches a ray for each pixel and queries which scene elements intersect that ray. Arbitrary camera models can therefore be expressed simply by changing the ray origin and direction, while additional rays can be spawned at interactions to model reflection, refraction, and shadowing. 3D Gaussian Ray Tracing (3DGRT) [16] brings this formulation to Gaussian scenes, replacing Gaussian projections with direct evaluation of Gaussians along rays.

This flexibility comes at the cost of rendering and training speed. Modern ray-tracing systems organize geometry using spatial acceleration structures like BVH (Bounding Volume Hierarchy) for efficiently finding ray–Gaussian hits. Dedicated hardware like NVIDIA RT cores, AMD ray-tracing accelerators, and Intel ray-tracing units helps accelerate BVH traversal. Despite the acceleration, computing intersection remains a major cost. Existing Gaussian ray tracing systems such as 3DGRT retrace the Gaussian hits during its backward pass, even though the forward pass has already traced each camera ray and discovered the Gaussians contributing to its pixel. However, reusing the forward-pass hits instead of re-tracing is not straightforward: the number of hits per ray is unknown in advance and varies widely across pixels and training iterations, and recording them inside the tracing kernel adds memory traffic that can slow the forward pass enough to cancel the savings. Moreover, many adjacent rays compute gradients of the same Gaussian, requiring large numbers of global atomic accumulation operations, resulting in large synchronization overhead. BVH maintenance can also be costly. A BVH represents the spatial layout of the Gaussians, but this layout continuously changes to better fit the 3D scene. Existing systems update BVH structure at each training iteration. So for large scenes containing millions of Gaussians, this maintenance becomes a considerable fraction of training time.

These costs stem from how training is organized around ray traversal rather than from traversal itself, so they can be reduced without changing what 3DGRT computes. Instead of retracing rays, our forward pass keeps the hits it already discovers, and splits tracing into short rounds so that recording them does not slow it down. Instead of accumulating every gradient directly to global memory, neighboring rays that update the same Gaussian first combine their contributions on-chip. And instead of rebuilding the BVH from scratch every iteration, we refit it in place whenever the set of Gaussians is unchanged. Each of these changes preserves the Gaussians blended for every pixel, so our system follows the rendering formulation of 3DGRT exactly while training substantially faster.

To study these mechanisms, we implement our method in Vulkan (The Khronos Vulkan Working Group, 2024). In contrast to prior OptiX-based implementations [20], Vulkan is cross-vendor and exposes lower-level control over hardware.

In summary, we make the following contributions:

  • We eliminate retracing in the backward pass. With adaptive multi-round traversal, the forward pass records every hit in a coalesced per-pixel archive without capping the number of rounds per ray, so the backward pass replays these hits instead of traversing the BVH again.

  • We reduce the two other major training costs. A fast gradient accumulator combines the gradients of rays that update the same Gaussian in on-chip memory before issuing global atomics, speeding up the backward pass by up to 10×10\times at full resolution. A hybrid BVH update strategy refits the BVH in most iterations and rebuilds it only when the Gaussian population changes or periodically thereafter.

  • We implement these techniques in FastGRT, the first open-source3 Vulkan implementation of 3DGRT that supports both training and rendering. Across Mip-NeRF 360, Tanks and Temples, Deep Blending, and Zip-NeRF, FastGRT trains 1.91.9–4.97×4.97\times faster than 3DGRT while matching its rendering quality and performance.

Background and Motivation

This section gives an overview of the standard 3D Gaussian ray tracing (3DGRT) algorithm [16] and its training pipeline, discusses its bottlenecks, and explains the optimization challenges.

3D Gaussian Scene Primitive

3D Gaussian rendering algorithms represent a scene with a large collection of anisotropic semi-transparent Gaussian primitives (aka ellipsoids) [8] distributed in the 3D world space. Each Gaussian ii is parameterized by a mean position μi\boldsymbol{\mu}_i, covariance Σi\boldsymbol{\Sigma}_i, opacity αi\alpha_i, and view-dependent color represented by spherical harmonics (SH). In practice, each Gaussian is truncated to a finite ellipsoidal extent. The covariance is parameterized as Σi=RiSiSiTRiT\boldsymbol{\Sigma}_i = \mathbf{R}_i\mathbf{S}_i\mathbf{S}_i^{T}\mathbf{R}_i^{T}, where Ri\mathbf{R}_i and Si\mathbf{S}_i determine the ellipsoid’s rotation and scale. SH represents view-dependent color using 48 coefficients per Gaussian.

Differentiable Gaussian Ray Tracing

In this paper, we use the term Gaussian Ray Tracing (GRT) to broadly refer to approaches that optimize a Gaussian scene representation through a differentiable ray-traced renderer (Condor et al., 2025; Moenne-Loccoz et al., 2024; Blanc et al., 2025; Byrski et al., 2025; Poirier-Ginter et al., 2026; Xu et al., 2026; Vit et al., 2026). During training, each iteration goes through three sequential phases: forward pass, backward pass, and BVH update. In the forward pass, a ray is cast from each pixel into the scene, and the contributions of the Gaussians it intersects are blended to produce the pixel color. Testing every ray against millions of Gaussians would be prohibitively expensive, so ray tracers organize the scene in a Bounding Volume Hierarchy (BVH): a tree of nested bounding boxes, where a ray descends only into the boxes it intersects and thereby skips most of the scene. Because rendering is differentiable, the loss against the ground-truth image is back-propagated in the backward pass to obtain gradients for each Gaussian’s position, covariance, opacity, and color, which the optimizer uses to update the Gaussians. These updates move and resize Gaussians, and periodically add or remove them, so the bounding boxes stored in the BVH no longer match the scene. The BVH must therefore be updated before the next iteration can trace rays correctly.

3DGRT Pipeline and Ray Tracing Algorithm

We build on 3DGRT [16], a representative and widely used GRT method with an open-source implementation [19]. It follows a pipeline similar to that of 3DGS where, instead of splatting, it uses ray tracing for rendering the scene. BVH traversal, accelerated by the GPU’s ray-tracing hardware, passes each hit (aka intersecting Gaussian) to the any-hit shader in no guaranteed order. The shader therefore maintains the k=16k = 16 nearest hits in sorted front-to-back order within the ray payload. A hit is accepted only if blending actually uses it: its contribution along the ray must exceed a minimum threshold, and the ray must not have terminated yet. After each batch, the accepted hits are blended in front-to-back order to update the pixel color. If further contribution is still required (the ray transmittance has not reached the minimum threshold), traversal restarts with the ray’s search interval starting just beyond the last hit. 3DGRT runs this entire loop of traversal, sorting, and blending inside a single ray-generation kernel, which exits only once its ray has gathered and blended all contributions. We refer to such a kernel as a megakernel, following the ray-tracing literature [12]. Unlike the operator-fusing megakernels of ML systems, the term here only denotes that the entire per-ray loop runs in one kernel. The resulting pixel colors form the rendered image, which is then compared with the ground-truth image to compute the loss.

3DGRT stores the farthest accepted hit distance for each pixel during the forward pass. This hit distance is used to bound the retracing during the backward pass to recover the same contributing Gaussians and compute gradients for their parameters. These gradients are summed together in global memory using atomic operations, and used to update Gaussian parameters. Since these updates can modify Gaussian’s location, size and shape, 3DGRT rebuilds the BVH at the end of every training iteration. Like 3DGS, 3DGRT also periodically densify, prune and reset the opacity of Gaussians at scheduled intervals during training.

Where is the Bottleneck in 3DGRT Training?

Figure 2 shows the training-time breakdown for Bonsai and Bicycle from Mip-NeRF 360 (Barron et al., 2022). In particular, Figure 2 shows that for full resolution, the backward pass dominates the training time across all iterations, making it the biggest target for optimization. Even for lower (e.g., 1/2) resolution, it stays the bottleneck for some indoor scenes like Bonsai; however, for outdoor scenes such as Bicycle, at 1/4 resolution, the BVH update takes over to become the bottleneck after a few iterations. In all the workloads, the backward pass contributes considerable time and thus is a major target for our optimization.

Breakdown of 3DGRT training time for the Bonsai and Bicycle scenes across training stages and image resolutions

Figure 2. Breakdown of 3DGRT training time for the Bonsai and Bicycle scenes across training stages and image resolutions.

This shift in bottleneck can be explained by an increase in the Gaussian population, which we will discuss further in Opportunity 3 of § 2.5. Effectively, with lower resolution, BVH time increases because of an increase in the Gaussian population (refer to Table 1), and at the same time, the backward-pass cost decreases because fewer rays are processed, and the larger spacing between neighboring rays reduces overlap in their Gaussian contributions and therefore atomic operation contention.

Resolution/SceneBonsaiRoomCounterKitchenBicycleGardenStump
Standard1.51.21.21.46.84.58.5
Full1.41.01.01.22.23.21.9

Table 1. Gaussian population (in millions) at the final trained checkpoint for scenes in the Mip-NeRF 360 dataset (Barron et al., 2022). Standard denotes the commonly used downsampled resolution (1/2 for indoor and 1/4 for outdoor scenes); Full denotes the original resolution.

Opportunities and Challenges

We identify three opportunities to reduce the cost of 3DGRT training. Each raises a challenge, which we address in Sections 3.2, 3.3, and 3.4, respectively.

Opportunity 1: Reuse forward-pass hits instead of retracing. 3DGRT retraces rays during backpropagation to recover the Gaussian hit sequence required for gradient computation. The choice to recompute rather than reuse is inherited from Gaussian-splatting-based approaches where recomputing projections is very fast. However, this is not true for ray tracing based approaches because of costlier BVH traversal and intersection processing. Reusing forward-pass intersections would eliminate this redundant traversal altogether.

Challenge: enabling reuse efficiently is non-trivial. Recording hits decreases work in the backward pass but increases work in the forward pass. It increases memory pressure in the megakernel and can slow the forward pass enough to offset the backward-pass savings.

Deciding on the amount of memory to allocate for a pixel is also non-trivial. Although the number of accepted Gaussian hits per pixel is theoretically bounded as shown in Appendix § 12, directly storing only accepted hits leads to irregular memory accesses. Rays within the same subgroup can accept different numbers of hits, so subsequent writes from neighboring threads diverge to distant addresses, reducing memory coalescing and degrading performance.

Storing every gathered hit produces a more regular memory access because the write location for each traversal step is known in advance. This allows the archive to be arranged so that neighboring rays write to nearby addresses, improving memory coalescing. The challenge, however, is capacity: the number of gathered hits per ray is not known beforehand and varies significantly across pixels and training iterations. For example, on the Bonsai scene, the maximum hit count across all pixels spikes to roughly 2,700 at iteration 3,001, while remaining closer to 1,000 for most other iterations.

Forward-state reuse therefore requires careful design of the tracing pipeline, storage layout, and hit-capacity policy rather than simply adding a hit buffer.

Opportunity 2: Reduce global atomics for gradient accumulation Using atomics for gradient accumulation results in significant cost for the backward pass. To isolate its effect, we compare a backward pass with the global atomics removed while retaining everything else, using the known traversal ranges for each pixel computed during the forward pass against the complete backward pass. The difference in these times is directly attributed to global atomics as shown in Figure 3, which presents results for the Bonsai and Bicycle scenes at full resolution.

Figure 3. Breakdown of 3DGRT backward-pass time into traversal, blending, and gradient computation versus gradient atomics, for Bonsai and Bicycle at full resolution.

Challenge: How to reduce gradient atomics? If rays contributing to the same Gaussian at a given step can be identified, their gradients can be combined in faster on-chip memory before issuing global atomic updates. However, for our Vulkan-based implementation, identifying such rays requires communication across invocations within a warp (a subgroup in Vulkan), and Vulkan does not guarantee which pixels share a subgroup.

Opportunity 3: Avoid rebuilding the BVH every iteration

3DGRT represents each Gaussian using an icosahedral proxy which bounds the Gaussian primitive tightly (hence, reducing false-positive hits) using 20 triangular faces. This allows traversal to exploit hardware-accelerated ray–triangle intersection. This improves traversal efficiency, but each Gaussian adds 20 triangle primitives to the acceleration structure. The issue is more pronounced for outdoor scenes at lower resolutions, where the Gaussian count can increase substantially, unlike indoor scenes where it remains relatively stable (Table 1). For example, Bicycle reaches about 6.8M Gaussians, corresponding to ∼136\sim136M triangle primitives, making a full BVH rebuild at every iteration a significant systems cost.

Challenge: can one refit an existing BVH as Gaussians change? Alternative BVH organizations based on instancing can reduce this geometry footprint at the cost of traversal time. For a direct comparison with 3DGRT, however, we retain its original icosahedral representation and instead avoid rebuilding the BVH at every iteration. On the other hand, the cheaper alternative of refitting cannot accommodate changes in the number of Gaussians, and repeated refits gradually degrade traversal efficiency.

System Design

Training and Ray Tracing Pipeline

Our system follows the general pipeline of GRT with the rendering formulation of 3DGRT. In particular, our optimizations do not alter the mathematical computation performed by 3DGRT. In other words, our system gathers and blends the same Gaussians as 3DGRT for the same scene and camera view for every pixel.

Our training pipeline differs from that of 3DGRT in how information is carried from the forward to the backward pass. Instead of retracing rays during backward computation, as discussed in Opportunity 1 of § 2.5, we preserve forward-pass hit information for reuse. To make this practical, we introduce **Adaptive Multi-Round Traversal(§ 3.2), which combines a round-based traversal pipeline with an adaptive mechanism that adjusts the number of rounds across training iterations.

During the forward pass, each pixel maintains a hit archive storing a fixed number of Gaussians. During the backward pass, each pixel processes its archived hits sequentially and computes the corresponding gradients. To efficiently accumulate these updates without race conditions, we introduce Fast Gradient Accumulator(§ 3.3).

Finally, the optimizer updates the Gaussian parameters using the computed gradients. As Gaussian positions and scales change during training, the acceleration structure must also be updated to reflect the new spatial distribution. To reduce this maintenance cost, we introduce Efficient BVH Update Strategy (§ 3.4).

Figure 4 gives an overview of the resulting pipeline. We discuss each of these three components in the following subsections.

Overview of our Vulkan Training and Ray Tracing pipeline

Figure 4. Overview of our Vulkan Training and Ray Tracing pipeline.

Adaptive Multi-Round Traversal

To reuse hits without slowing the forward pass (Opportunity 1), we restructure tracing so that every hit can be recorded cheaply. Motivated by prior work showing the drawbacks of megakernels for ray tracing ([12]) and by the best practice of keeping ray-generation shaders lightweight, we split the megakernel into two kernels: gather and blend. This splitting reduces memory pressure, but introduces a new execution challenge. Ray traversal cannot proceed independently of blending: after each batch of hits is gathered, their contributions must be blended to update the ray transmittance and determine whether further traversal is required. We refer to one traversal followed by one blending stage as a round. The forward pass therefore becomes a sequence of dependent gather–blend rounds.

As gather and blend are different kernels, the accumulation of a round’s blend must be saved in global memory for the next blend to further accumulate. In particular, the following state must be preserved between gather and blend: T (remaining ray transmittance), accumulated radiance (R, G, B), hitDist (where the next gather should start), cursor (represents whether the ray is live or terminated), accumulated depth and hit count.

Hit archive. Each pixel stores every gathered hit, not only the accepted ones, in a hit archive of 512 slots. This is so that neighboring rays can write to nearby addresses (§ 4). Because we preallocate 512 slots per ray, a few rays may gather more than 512 Gaussians. For these rays, the backward pass retraverses the BVH to recover the remaining hits and computes their gradients following 3DGRT.

Adaptive round budget. A fixed round budget RR causes two problems. First, for rays that require more than RR rounds (which we call long rays), a fixed RR would terminate the ray before gathering considerable contributions, diverging from the 3DGRT rendering formulation. Second, as training progresses, many rays (which we call short rays) terminate within only a few rounds. Even though they no longer traverse, they must still read their state in every remaining round only to determine that they should exit, wasting time until all RR rounds complete. With a fixed value such as R=16R = 16, this overhead can become substantial once most rays terminate after only a few rounds.

To account for long rays, we therefore execute the forward pass in two phases. The first phase performs a bounded number of lightweight traversal–blend rounds. Short rays finish within the first RR rounds, after which the remaining long rays are handled by a fallback megakernel, which continues the 3DGRT traversal until their termination criterion is satisfied. The fallback is launched over all pixels, but already terminated rays immediately exit, leaving only the small tail of long-running rays to perform additional work.

As the Gaussian representation becomes denser and better fits the scene, rays generally accumulate opacity more quickly and tend to terminate earlier. To account for short rays, we therefore adapt the round budget from the behavior observed in the previous iteration. Let ptp_t denote the fraction of rays that remain active after reaching the round budget RtR_t. We initialize R0=16R_0 = 16 and update

Rt+1={Rt+1,pt≥1%,max⁡(1,Rt−1),pt<0.25%,Rt,otherwise.(1)R_{t+1} = \begin{cases} R_t + 1, & p_t \ge1\%, \\ \max(1, R_t - 1), & p_t < 0.25\%, \\ R_t, & \text{otherwise.} \end{cases} \tag*{(1)}

Changing the budget by only one round at a time prevents transient behavior from causing large scheduling changes between consecutive iterations. If too many rays exhaust the current budget, we increase RR; if almost all rays terminate earlier, we decrease it. In our experiments, the initial budget of R=16R = 16 gradually decreases to approximately R=4R = 4 (Figure 5), while automatically increasing again when training events temporarily require longer traversal.

Adaptive round budget over training iterations for the Bonsai scene at full resolution

Figure 5. Adaptive round budget over training iterations for the Bonsai scene at full resolution.

This design makes it possible to record hits during the forward pass without additional overhead and without capping the number of rounds per ray, eliminating retraversal in the backward pass. Figure 6 shows speedup in backpass with little overhead in forward pass.

Forward-pass and Backward-pass time of FastGRT and 3DGRT over training iterations

Figure 6. Forward-pass and Backward-pass time of FastGRT and 3DGRT over training iterations.

Fast Gradient Accumulator

As discussed in Opportunity 2 (§ 2.5), gradient accumulation is one of the major bottlenecks. We reduce these atomics by aggregating contributions within each subgroup before writing to global memory.

Our backward compute kernel processes a workgroup corresponding to an 8×88 \times8 tile of adjacent pixels, comprising two 32-lane subgroups on our target NVIDIA GPUs. Neighboring pixels often contribute to the same Gaussians, creating opportunities for aggregation. Although Vulkan does not generally guarantee the mapping of local invocation indices to subgroup lanes, all invocations remain within the same pixel tile. Our method therefore exploits tile-level spatial locality without requiring a particular subgroup layout.

Algorithm 1 Subgroup-Aggregated Gradient Accumulation

Require: Gaussian ID gig_i and gradient did_i

  1. A←A \leftarrow active lanes ▷ Find lanes having accepted hit

  2. while A≠0A \ne0 do

  3. &nbsp;&nbsp;&nbsp;&nbsp;l←FIRSTACTIVELANE⁡(A)l \leftarrow\operatorname{FIRSTACTIVELANE}(A) &nbsp;&nbsp;&nbsp;&nbsp;▷ Get gId of lane l and mask lanes with same gId

  4. &nbsp;&nbsp;&nbsp;&nbsp;g←BROADCAST⁡(gl,l)g \leftarrow\operatorname{BROADCAST}(g_l, l)

  5. &nbsp;&nbsp;&nbsp;&nbsp;M←BALLOT⁡(A∧(gi=g))M \leftarrow\operatorname{BALLOT}(A \land(g_i = g)) &nbsp;&nbsp;&nbsp;&nbsp;▷ Distribute 59 gradient components to accumulate among all lanes

  6. &nbsp;&nbsp;&nbsp;&nbsp;for d←LANEID⁡()d \leftarrow\operatorname{LANEID}() to D−1D - 1 step 32 do

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;s←0s \leftarrow0

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for all j∈Mj \in M do

  9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;s←s+dj[d]s \leftarrow s + d_j[d]

  10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end for ▷ One atomic

  11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ATOMICADD⁡(∇g[d],s)\operatorname{ATOMICADD}(\nabla_g[d], s)

  12. &nbsp;&nbsp;&nbsp;&nbsp;end for

  13. &nbsp;&nbsp;&nbsp;&nbsp;A←A∖MA \leftarrow A \setminus M

  14. end while

The kernel processes the cached Gaussian lists one entry at a time. At each list position, lanes with valid entries compute their gradient contributions and stage them in separate shared-memory rows (Figure 7) indexed by (subgroupID, laneID). Each row contains 59 gradient components, requiring approximately 14.75 KiB of shared memory per 64-invocation workgroup.

Image not extracted

Figure 7. Per-lane Gaussian gradients are staged in shared memory, and the accumulation work is divided among lanes.

Within each subgroup, we identify valid lanes (those with accepted Gaussian hits) using a ballot and partition their contributions by Gaussian ID. For each distinct Gaussian, lanes cooperatively reduce the corresponding rows, distributing gradient components across lanes: lane ℓ\ell accumulates component ℓ\ell and, when present (i.e., if ℓ+32<59\ell+ 32 < 59), component ℓ+32\ell+ 32. Each accumulated component is then written to global memory with a single atomic addition. This

Per-lane Gaussian gradients are staged in shared memory, and the accumulation work is divided among lanes. BVH update time per step of FastGRT and 3DGRT over training iterations.Vector (PDF)

distributes the reduction across the subgroup rather than assigning the entire gradient vector to one lane. Algorithm 1 summarizes the procedure, and Figure 6 compares the resulting backward pass with that of 3DGRT.

Efficient BVH Update Strategy

BVH refitting offers a cheaper alternative to rebuilding: it updates only the node bounds and preserves the existing hierarchy, without reconstructing any nodes. However, repeated refitting can degrade traversal efficiency as the geometry moves away from the spatial partitioning established during construction. Refitting also cannot accommodate changes in primitive count, requiring a rebuild when densification or pruning modifies the Gaussian population.

Hybrid rebuild/refit strategies are common for dynamic scenes [13, 10], but to the best of our knowledge they have not been explored for 3DGRT training. We therefore adopt a hybrid strategy that combines refitting with selective rebuilding. During the first 15,000 iterations, when densification is active, we rebuild at densification and pruning events and refit at all other iterations. After densification ends, the Gaussian population remains fixed, but continued parameter updates can still degrade BVH quality. We therefore rebuild every 16 iterations and refit in between, balancing maintenance cost against traversal efficiency (Figure 8).

Image not extracted

Figure 8. BVH update time per step of FastGRT and 3DGRT over training iterations.

Implementation

Command Buffers. Launching kernels separately for each of the RR traversal rounds introduces repeated host-side submission overhead. Vulkan allows multiple GPU commands to be recorded into a command buffer and submitted together to a VkQueue, reducing this overhead. However, dependencies between these commands are not inserted automatically; required execution and memory ordering must be explicitly specified using synchronization primitives such as pipeline barriers.

Shader Launch Parameters. For the blending and backward compute shaders, we use an 8×88 \times8 workgroup layout. The ray-tracing gather stage differs because Vulkan ray-tracing shaders do not use workgroups, instead, shader invocations are organized into subgroups by the implementation.

Any-hit and KK value. In 3DGRT, each retained hit stores both its distance and Gaussian ID in the ray payload, although only the distance is needed during hit selection. We move Gaussian IDs to global memory and maintain an unsorted distance buffer, tracking the farthest hit for replacement and sorting only before blending. These changes reduce payload pressure and increase the optimal KK from 11 to 24 in our Vulkan implementation. A larger KK reduces traversal rounds but increases per-round traversal and sorting costs. Refer Appendix § 13 for further details.

Memory Layout. Throughout our design, we organize stored state to favor coalesced memory accesses. For hit replay, nearby pixels should access Gaussian hits from nearby memory locations. We therefore use a slot-major layout, where each column corresponds to a pixel/ray and each row corresponds to one of the 512 hit slots. As a result, the same hit slot for neighboring pixels is stored contiguously, enabling more coalesced accesses during both recording and replay.

Experiments and Evaluation

We evaluate the following implementations:

  • 3DGRT Baseline: OptiX implementation4 of the original 3DGRT [16], which follows the same densification strategy as 3DGS and uses a density learning rate of 0.09. During inference, it uses density clamping and a transmittance threshold of 0.03.

  • FastGRT (Ours): our Vulkan implementation with all

the optimizations described above, using exactly the same rendering and training configuration as 3DGRT.

We do not compare against GRTX [14], the only other Vulkan-based Gaussian ray tracer we are aware of, because its source code is not publicly available. Its proposed hardware extensions would also require architectural simulation rather than execution on existing GPUs.

Experimental Setup. Experiments were conducted on an NVIDIA RTX PRO 6000 Blackwell GPU with driver 610.43.02, Vulkan 1.4, and CUDA 12.8.1. We evaluate on Mip-NeRF 360 [1], Tanks and Temples [9], Deep Blending [7], and Zip-NeRF [2]. For Mip-NeRF 360, we use the standard 1/2 indoor and 1/4 outdoor resolutions. Following 3DGRT, we report mean PSNR, LPIPS, and SSIM over test views for reconstruction quality and FPS for rendering performance.

Performance

End-to-end training performance. Figure 9 reports end-to-end training time for 3DGS, 3DGRT and FastGRT (Ours), along with our speedup over 3DGRT. The speedup is larger at full resolution (Figure 10), mainly because the backward pass becomes much more expensive at high resolution. Our optimizations provide the largest benefit in the backward pass Figure 11, translating into greater end-to-end speedups at full resolution. 3DGS does not run on distorted cameras and thus its results are produced.

End-to-end training performance at Standard resolution

Figure 9. End-to-end training performance at Standard resolution.

End-to-end training time for MipNeRF360 scenes at full resolution

Figure 10. End-to-end training time for MipNeRF360 scenes at full resolution

Performance benefit of each of the three proposed optimizations at selected training checkpoints

Figure 11. Performance benefit of each of the three proposed optimizations at selected training checkpoints.

Rendering performance. At inference, we run the megakernel without hit storage for comparison with 3DGRT (Figure 12). Since inference has no backward pass and BVH update, the megakernel is the better choice there: when

Comparable Inference Time Rendering Speed

Figure 12. Comparable Inference Time Rendering Speed

memory pressure is low and the computation around traversal does not limit occupancy.

Complex Camera Models We evaluate our approach on the Zip-NeRF dataset [2], which consists of four scenes (Alameda, Berlin, London, and NYC) captured with fisheye cameras and distorted inputs. Following 3DGRT, we use 1/8 resolution for a fair comparison.

Because 3DGS relies on projection, it handles such camera models poorly, so we compare only against 3DGRT. Figure 9 shows our training speedups, and Table 2 shows that FastGRT achieves comparable rendering quality.

Method/MetricMip-NeRF 360 PSNR↑\uparrowMip-NeRF 360 LPIPS↓\downarrowMip-NeRF 360 SSIM↑\uparrowDeep Blending PSNR↑\uparrowDeep Blending LPIPS↓\downarrowDeep Blending SSIM↑\uparrowTanks and Temples PSNR↑\uparrowTanks and Temples LPIPS↓\downarrowTanks and Temples SSIM↑\uparrowZip-NeRF PSNR↑\uparrowZip-NeRF LPIPS↓\downarrowZip-NeRF SSIM↑\uparrow
3DGRT27.880.2410.85129.450.3320.89622.540.2160.82924.330.2570.848
FastGRT (Ours)28.080.2410.85229.450.3300.89622.540.2140.83124.340.2610.847

Table 2. Rendering quality of the final converged checkpoints at standard resolution (mean over all scenes in each dataset).

Which optimization contributes most?

Figure 11 shows the effect of each optimization on the individual phases. At full resolution, most of the benefit comes from the atomic reduction (from C to D), which reduces iteration time by 300 ms. At standard resolution, the gains come from both the atomic reduction (e.g., Bonsai at 1/2 resolution) and the BVH update (e.g., Bicycle at 1/4 resolution).

Peak GPU Memory Usage

Table 3 reports peak GPU memory usage over the entire training run. At standard resolution, the number of Gaussians grows substantially for outdoor scenes, which is reflected in the table: 3DGRT uses around 30 GiB for these workloads. FastGRT consistently uses more memory than 3DGRT because it stores forward-pass hits, whose footprint scales with image resolution.

Reconstruction quality at Inference

Table 2 shows the reconstruction quality by rendering the final model checkpoint on test views. Each entry is the mean over all scenes in the dataset, evaluated at standard resolution following previous work on GRT.

We observe a small difference in mean PSNR on Mip-NeRF 360 despite using the same rendering formulation as 3DGRT. In earlier experiments, we identified an upstream 3DGRT issue where floating-point precision cause the final forward-pass hit to be skipped during backward traversal, because of which we see this difference.

Related Work

Gaussian Splatting Systems. 3DGS [8] established 3D Gaussians as an effective differentiable representation for novel-view synthesis. Subsequent work improves different parts of this pipeline: FastGS [22] reduces training time, CLM-GS (Zhao et al., 2026) improves scalability under limited GPU memory, 3DGUT (Wu et al., 2025) improves Gaussian projection under non-linear camera models using the unscented transform, and EDGS [11] revisits densification and pruning for faster convergence.

Ray-Traced Gaussian Representations. Several works render Gaussian or ellipsoidal primitives by ray tracing rather than splatting. RayGauss [3] performs volumetric ray casting over irregularly distributed Gaussian basis functions using a BVH. EVER [15] replaces Gaussians with constant-density ellipsoids whose volume rendering integral can be evaluated exactly, eliminating popping artifacts. Condor et al. [5] use volumetric ray-traced Gaussian primitives to model scattering and emissive media. These works focus on the rendering formulation and its quality, whereas we keep the formulation of 3DGRT fixed and optimize how its training executes on the GPU.

Traversal and BVH Optimizations. GRTX [14] targets the traversal and intersection cost of Gaussian ray tracing. It transforms anisotropic Gaussians into a normalized unit-sphere representation to simplify intersection tests and reduce acceleration-structure complexity, and additionally proposes hardware support to improve BVH traversal. Its focus is therefore complementary to ours: GRTX primarily optimizes the traversal mechanism, while we study the systems organization of the complete training pipeline.

Optimizing 3DGRT Training. GRay [21] explores algorithmic and representation-level changes for faster Gaussian ray-tracing training, including transferring dense initialization strategies from splatting. Stochastic ray tracing (Xu et al., 2026) replaces sorted front-to-back blending with an unbiased Monte Carlo estimator that evaluates only a sampled subset of Gaussians per ray, avoiding sorting altogether. Both approaches gain speed by changing the rendering formulation, while, we preserve the exact formulation of 3DGRT and instead reorganize how its training executes. 3DGART (Vit et al., 2026) identifies gradient accumulation in the backward pass as a major bottleneck and proposes a primitive-major organization using tile–primitive mappings. Its formulation relies on perspective projection, however, limiting its applicability to more general camera models. Our work also targets backward-pass overhead, but preserves the ray-based formulation and does not assume perspective projection. Outside ray tracing, DISTWAR (Durvasula et al., [6]) reduces atomic contention in differentiable rasterization pipelines, including 3DGS, by performing warp-level reduction of gradient updates before issuing global atomics. Our fast gradient accumulator follows a similar principle, but must operate without Vulkan guarantees on how invocations map to subgroups, and aggregates over per-ray replayed hit lists rather than rasterized fragments.

Scene/MethodFastGRT (GiB)3DGRT (GiB)
Bonsai13.17.5
Counter10.95.3
Kitchen13.05.8
Room10.05.4
Bicycle49.032.6
Garden30.719.5
Stump40.433.0

Table 3. Peak GPU memory during training at standard resolution

Forward-State Reuse. RaySplats (Byrski et al., [4]) also stores forward-pass Gaussian intersections for reuse during backpropagation, avoiding repeated traversal. However, it assumes a fixed cap on the number of hits per ray and changes the ray termination condition to account for this approximation. Our work makes no such approximation. RaySplats also does not report the performance benefit of this reuse.

Discussion and Future Work

The forward pass is the new bottleneck. In the OptiX baseline the backward pass dominates a training step: on bicycle at full resolution it takes 484 ms of a 659 ms step, about 4.6×4.6\times the forward pass. Hit replay, atomic reduction and hybrid BVH updates remove most of that cost. With all three, the backward pass falls to 43 ms and the BVH update to 16 ms.

The forward pass, at 96 ms, is now more than half of the step and over twice the backward pass. Our forward pass is only on par with OptiX’s (1.02–1.06× at convergence), so it is where further gains in training time must come from. Possible directions include adapting the number of hits gathered per round to how deep each ray’s march goes, reordering rays [17] to improve traversal coherence, and reduced-precision shading.

Atomic Reduction Our optimisation benefits from nearby rays hitting same gaussians. Because Vulkan does not guarantee invocation to pixel mapping, it is possible that nearby rays that hit same gaussians are present in different subgroups, thus giving no benefit of accumulating gradients within a subgroup. Additionally, aggregation does not benefits reflective or refractive secondary rays where divergence increases and shared-Gaussian contention decreases, reducing the benefit of atomic aggregation while increasing traversal and memory-access irregularity.

Conclusion

FastGRT reduces the training time of 3DGRT without compromising rendering quality. Our key observation is that a considerable fraction of the training cost lies not in ray traversal itself, but in the computation organized around it. By reusing forward-pass hits through adaptive multi-round traversal, aggregating gradients within subgroups before issuing global atomics, and combining BVH refitting with selective rebuilding, FastGRT substantially accelerates end-to-end training while gathering and blending the same Gaussians as 3DGRT. These techniques rely on general properties of Gaussian ray-tracing training rather than on a particular workload, and our Vulkan implementation is not tied to a single vendor’s ray-tracing API.

AI USAGE DISCLOSURE

We used OpenAI’s ChatGPT and Anthropic’s Claude as assisting tools during this work. They were used for language editing, rephrasing, manuscript organization, LaTeX formatting, technical discussion, literature and documentation lookup, and code generation and debugging assistance. Generative AI tools were also used to help create or edit the appearance a small number of figures, while all underlying experimental data were produced by the authors. The technical contributions, system design, implementation, experiments, measurements, analysis, and conclusions were developed and validated by the authors.

Theoretical case bound on number of accepted hits per ray in 3DGRT

The 3DGRT setup provides a theoretical upper bound through its transmittance termination criterion. Training terminates a ray once its transmittance falls below τ=0.001\tau= 0.001.

The longest traversal occurs when each accepted hit lets as much light through as possible.

An accepted Gaussian must have minimum opacity of αmin⁡=1/255\alpha_{\min} = 1/255. In the worst case, where every accepted Gaussian blocks exactly this minimum amount, the transmittance after nn hits is (1−αmin⁡)n(1 - \alpha_{\min})^n, and the ray remains active only while

(1−αmin⁡)n≥τ.(2)(1 - \alpha_{\min})^n \ge\tau. \tag*{(2)}

Thus, the number of accepted Gaussians before termination satisfies

n≤log⁡(0.001)log⁡(254255)≈1758.02.(3)n \le\frac{\log(0.001)}{\log\left(\frac{254}{255}\right)} \approx1758.02. \tag*{(3)}

Therefore, approximately 1759 accepted Gaussian contributions are sufficient to terminate a ray in the worst case.

Any-hit implementation and KK value

Hit storage and payload usage. In 3DGRT, each retained hit is represented by two values in the ray payload: its distance and its Gaussian ID. For a buffer of K=16K = 16 hits, these occupy 32 payload entries. During traversal, hit distances are repeatedly read to compare incoming hits with those already retained. Gaussian IDs, however, are only needed after traversal to identify the Gaussians to be processed. Consequently, half of the payload entries hold values that must be preserved but are not needed for hit selection.

Our implementation separates the storage of these two values. We retain hit distances in the payload and store the corresponding Gaussian IDs in global memory. Each retained hit has a slot index that associates its distance with its ID. When a hit is inserted or replaced, we update the distance and the corresponding ID at that slot. Moving IDs out of the payload frees space for additional hit distances, allowing more hits to be gathered in a single traversal. This trades additional global-memory accesses for reduced payload storage requirements.

Maintaining an unsorted hit buffer. We also avoid maintaining a fully sorted hit buffer during traversal. While free slots remain, incoming hits are appended to the next available slot. Once the buffer contains KK hits, we only need to determine whether a new hit belongs among the KK closest hits encountered so far. This requires comparing its distance with the largest distance currently stored, rather than with every entry in sorted order.

We therefore maintain two additional values: the maximum stored distance, maxDistance, and the index of the slot containing it. If an incoming hit is farther away than the current farthest hit, it is discarded. If it is closer, it replaces the farthest hit in that slot, and the corresponding Gaussian ID is updated in global memory.

After a replacement, the previous maximum is no longer valid. Because the buffer is unsorted, we scan the retained distances to find the new maximum and its index. Hits that do not replace an entry require no such scan. This organization avoids shifting entries to maintain sorted order during traversal, at the cost of recomputing the maximum whenever the farthest hit is replaced. Once traversal finishes, the retained hits are sorted by distance before front-to-back blending.

Effect on the optimal buffer size. These changes allow us to retain more hits within the available payload space. In our initial Vulkan implementation, which followed the original 3DGRT buffer organization, the best-performing value was K=11K = 11. After changing the storage and buffer organization, the best-performing value increased to K=24K = 24. Although moving Gaussian IDs out of the payload frees approximately half of the original per-hit payload storage, the best-performing KK is determined by execution costs as well as storage capacity.

Tradeoff between traversal rounds and per-round cost. Gathering more hits per traversal can reduce the number of gather–blend rounds needed by a ray. For a ray that processes the same sequence of contributions, increasing KK allows each round to process a larger batch. However, fewer rounds do not necessarily imply lower execution time, because the cost of each round also changes.

First, KK affects traversal pruning. Once the buffer is full, the distance of the farthest retained hit can be used to tighten the traversal range. A smaller buffer fills sooner and can establish a tighter distance bound earlier, allowing more distant hits to be excluded. A larger buffer generally retains a farther-reaching set of hits, so traversal may explore more hits before it finishes.

Second, blending determines whether the ray has accumulated enough opacity to terminate. Larger batches may gather hits that are ultimately unnecessary because the ray reaches its termination threshold partway through the batch. Smaller batches allow this termination condition to be checked more frequently, but require more traversal rounds.

Figure 13. Per-phase time per training iteration, Mip-NeRF 360 at standard resolution.

Figure 14. Per-phase time per training iteration, Mip-NeRF 360 at full resolution.

Figure 15. Per-phase time per training iteration, Tanks&Temples and Deep Blending.

References

  1. [1]Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, P. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5460–5469, 2022. doi: 10.1109/CVPR52688.2022.00539.DOI
  2. [2]Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, P. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19640–19648, 2023. doi: 10.1109/ICCV51070.2023.01804.
  3. [3]Blanc, H., Deschaud, J.-E., and Paljic, A. RayGauss: Volumetric Gaussian-based ray casting for photorealistic novel view synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1808–1817, 2025. doi: 10.1109/WACV61041.2025.00183.DOI
  4. [4]Byrski, K., Mazur, M., Tabor, I., Dziarmaga, T., Kądziolka, M., Baran, D., and Spurek, P. Rayplasts: Ray tracing based gaussian splatting, 2025. URL https://arxiv.org/abs/2501.19196.
  5. [5]Condor, J., Speierer, S., Bode, L., Bozic, A., Green, S., Didyk, P., and Jarabo, A. Don’t splat your Gaussians: Volumetric ray-traced primitives for modeling and rendering scattering and emissive media. ACM Transactions on Graphics, 44(1):1–17, 2025. doi: 10.1145/3711853.DOI
  6. [6]Durvasula, S., Zhao, A., Chen, F., Liang, R., Sanjaya, P. K., and Vijaykumar, N. DISTWAR: Fast differentiable rendering on raster-based rendering pipelines, 2023. URL https://arxiv.org/abs/2401.05345.
  7. [7]Hedman, P., Philip, J., Price, T., Frahm, J.-M., Drettakis, G., and Brostow, G. Deep blending for free-viewpoint image-based rendering. 37(6):257:1–257:15, 2018.DOI
  8. [8]Kerbl, B., Kopanas, G., Leimkuehler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 2023.arxiv.org/abs/2308.04079
  9. [9]Knapitsch, A., Park, J., Zhou, Q.-Y., and Koltun, V. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
  10. [10]Kopta, D., Ize, T., Spjut, J., Brunvand, E., Davis, A., and Kessler, A. Fast, effective BVH updates for animated scenes. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), pp. 197–204, 2012. doi: 10.1145/2159616.2159649.DOI
  11. [11]Kotovenko, D., Grebenkova, O., and Ommer, B. Edges: Eliminating densification for efficient convergence of 3dgs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 41065–41076, June 2026.arxiv.org/abs/2504.13204
  12. [12]Laine, S., Karras, T., and Aila, T. Megakernels considered harmful: wavefront path tracing on gpus. In Proceedings of the 5th High-Performance Graphics Conference, HPG ’13, pp. 137–143, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450321358. doi: 10.1145/2492045.2492060. URL https://doi.org/10.1145/2492045.2492060.
  13. [13]Lauterbach, C., eui Yoon, S., Tuft, D., and Manocha, D. RT-DEFORM: Interactive ray tracing of dynamic scenes using BVHs. In Proceedings of the IEEE Symposium on Interactive Ray Tracing, pp. 39–46, 2006. doi: 10.1109/RT.2006.280213.DOI
  14. [14]Lee, J., Jeon, S., Lee, J., Park, J., and Sim, J. GRTX: Efficient ray tracing for 3D Gaussian-based rendering. In Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1–14, 2026. doi: 10.1109/HPCA68181.2026.11408531.DOI
  15. [15]Mai, A., Hedman, P., Kopanas, G., Verbin, D., Futchik, D., Xu, Q., Kuester, F., Barron, J. T., and Zhang, Y. EVER: Exact volumetric ellipsoid rendering for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4930–4939, 2025. doi: 10.1109/ICCV51701.2025.00469.
  16. [16]Moenne-Loccoz, N., Mirzaei, A., Perel, O., de Lutio, R., Esturo, J. M., State, G., Fidler, S., Sharp, N., and Gojcic, Z. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Transactions on Graphics, 2024.
  17. [17]NVIDIA. Shader execution reordering. NVIDIA Whitepaper, https://developer.nvidia.com/sites/default/files/akamai/gameworks/ser-whitepaper.pdf, 2022.
  18. [18]NVIDIA. vk_gaussian_splatting: Vulkan gaussian splatting sample. https://github.com/nvpro-samples/vk_gaussian_splatting, 2025.
  19. [19]NVIDIA Toronto AI Lab. 3DGRUT: Official implementation of 3DGRT and 3DGUT. https://github.com/nv-tlabs/3dgrut, 2025.
  20. [20]Parker, S. G., Bigler, J., Dietrich, A., Friedrich, H., Hoberock, J., Luebke, D., McAllister, D., McGuire, M., Morley, K., Robinson, A., and Stich, M. Optix: a general purpose ray tracing engine. ACM Trans. Graph., 29(4), July 2010. ISSN 0730-0301. doi: 10.1145/1778765.1778803. URL https://doi.org/10.1145/1778765.1778803.
  21. [21]Poirier-Ginter, Y., Lalonde, J.-F., and Drettakis, G. Gray: Ray tracing 3D Gaussians near the speed of splats. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 9(1):1–19, 2026. doi: 10.1145/3804496.
  22. [22]Ren, S., Wen, T., Fang, Y., and Lu, B. FastGS: Training 3D Gaussian splatting in 100 seconds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.arxiv.org/abs/2511.04283

Paper details

vlnc:W_pfb3mf9h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT, Claude

View statement
Looking for related papers…
Contents