3D Gaussian Splatting for Real-Time Radiance Field Rendering

NeRF represents a scene as a continuous function stored inside a neural network: given any 3D point and viewing direction, the network outputs a color and density. To render a single pixel, you march a ray into the scene, query the network at dozens of sample points along the ray, then composite those samples into a final color via volume rendering.

This works beautifully — NeRF produces photorealistic images — but it is fundamentally slow. Every pixel requires multiple neural network forward passes. At 1080p, that means over two million pixels × dozens of samples each = hundreds of millions of network evaluations per frame. Even with spatial hash grids (Instant-NGP) or baked representations (Plenoxels), real-time rendering of full unbounded scenes remained elusive.

The deeper issue is architectural: NeRF uses an implicit representation — the scene lives inside the network's weights and can only be queried, never directly drawn. 3D Gaussian Splatting asks: what if the scene representation were explicit — a collection of simple geometric primitives that the GPU already knows how to paint?

A 3D Gaussian is the simplest possible "soft blob" in three-dimensional space. It is defined by a center point (mean μ) and a covariance matrix Σ that controls its size, shape, and orientation. Think of it as a fuzzy ellipsoid: dense in the center, fading smoothly to zero at the edges.

The key insight is that a single 3D Gaussian can represent a wide range of local scene geometry — a flat surface patch (a very thin, flat ellipsoid), a thin edge (a long, narrow ellipsoid), or a soft volume (a round blob). By using anisotropic Gaussians (different scales along different axes), each primitive adapts to the local geometry it needs to represent.

Each Gaussian carries five learnable properties: its position μ ∈ ℝ³, a covariance matrix Σ ∈ ℝ³ˣ³ (parameterized as a rotation quaternion q and a scale vector s for stable optimization), an opacity α ∈ [0,1], and spherical harmonic (SH) coefficients for view-dependent color. The SH coefficients let each Gaussian change color depending on the viewing angle — crucial for capturing specular highlights and other view-dependent effects.

Rendering 3D Gaussians into a 2D image follows the same alpha compositing equation used in volume rendering, but computed via rasterization instead of ray marching. The process has three stages:

1. Projection: Each 3D Gaussian is projected onto the image plane using the camera's viewing transformation. The 3D covariance Σ transforms into a 2D covariance Σ′ via a first-order (Jacobian) approximation of the projective mapping. The result is a 2D ellipse on screen — each Gaussian's "splat."

2. Sorting: All Gaussians are sorted by depth. This is essential for correct alpha compositing — nearby Gaussians must be drawn before distant ones.

3. Alpha compositing: For each pixel, the sorted Gaussians that overlap it are blended front-to-back. Each Gaussian contributes its color weighted by its opacity and the 2D Gaussian evaluation at that pixel, multiplied by the transmittance (how much light has not yet been absorbed by closer Gaussians). Accumulation stops when transmittance drops below a threshold.

Computing the alpha compositing equation naively — testing every Gaussian against every pixel — would be far too slow. The paper's rendering engine uses a tile-based approach inspired by how modern GPUs render triangles:

Step 1 — Frustum culling: Discard any Gaussian outside the camera's viewing frustum, using a 99% confidence interval to bound each Gaussian's extent.

Step 2 — Tiling: Divide the screen into 16×16 pixel tiles. Assign each surviving Gaussian to every tile its 2D splat overlaps.

Step 3 — Sorting: For each tile, sort its Gaussians by depth using a fast GPU radix sort. The key trick: pack the tile ID and depth into a single integer so one global sort handles all tiles simultaneously.

Step 4 — Compositing: Each tile is processed by one GPU thread block. Threads within the block share a common list of sorted Gaussians via shared memory, compositing in parallel — one thread per pixel. Accumulation stops early when transmittance drops below 1/255.

This design maps perfectly to GPU hardware: tiles correspond to thread blocks, pixels to threads, and the shared Gaussian list exploits shared memory bandwidth. The result: 30–200+ fps at 1080p depending on scene complexity.

The optimization starts from a sparse point cloud produced by Structure-from-Motion (SfM). These initial points are far too few to represent a full scene — imagine trying to paint a detailed landscape with only a hundred dots. The adaptive density control scheme grows and refines the Gaussian set during training:

Cloning — when a Gaussian is too small and its positional gradient is large (the optimizer wants to move it a lot because the region is under-reconstructed), create a copy and move it in the direction of the gradient. This fills in missing geometry. Think of it as mitosis: one cell divides into two to cover more territory.

Splitting — when a Gaussian is too large and its gradient is also large (it's covering too much area and can't represent the fine details within), replace it with two smaller Gaussians. This refines coarse regions into sharper detail. Think of a large paintbrush being traded for two finer ones.

Pruning — periodically remove Gaussians that have become nearly transparent (α close to 0) or excessively large. Also reset all opacities to near-zero periodically, forcing Gaussians to re-earn their existence — those that don't contribute to any view fade away.

This grow-and-prune cycle runs every 100 optimization steps. A typical scene converges to 1–5 million Gaussians — orders of magnitude fewer than the number of MLP evaluations NeRF requires per frame.

The full pipeline optimizes all Gaussian parameters jointly using stochastic gradient descent, with the differentiable rasterizer providing gradients. The loss function combines an L1 photometric loss with a structural similarity term:

The L1 loss penalizes per-pixel color differences between the rendered image and the ground-truth photograph. D-SSIM captures structural similarity — it is sensitive to contrast and structure at a patch level, catching errors that pixel-level L1 would miss. The weighting λ = 0.2 was found empirically.

Training typically runs for 30,000 iterations, taking 20–50 minutes on a single GPU. This is competitive with or faster than NeRF methods, and the resulting representation renders in real time — a dramatic improvement over NeRF which required seconds per frame even after training.

The paper evaluated on 13 real scenes from three established benchmarks — Mip-NeRF 360, Tanks & Temples, and Deep Blending — plus the synthetic Blender dataset. The results were striking:

• Quality: On Mip-NeRF 360, Gaussian Splatting matched or exceeded the PSNR of Mip-NeRF 360 itself — the reigning state-of-the-art — while also achieving higher SSIM and lower LPIPS scores on most scenes.

• Speed: Rendering at 1080p ranged from 30 fps on complex outdoor scenes to over 200 fps on smaller indoor scenes — the first method to achieve real-time performance at this quality level on unbounded scenes.

• Training time: 20–50 minutes on a single NVIDIA A6000 GPU — competitive with the fastest NeRF variants and far faster than the original NeRF (hours to days).

• Representation size: 1–5 million Gaussians per scene, storing position, covariance, SH coefficients, and opacity. This is a few hundred MB — compact enough for practical deployment.

The quality-speed combination was unprecedented. Previous methods formed a clear trade-off curve; Gaussian Splatting broke through it.

Real surfaces rarely look the same from every angle. A metal table has bright specular highlights that shift as you move; a matte wall has subtle shading variations. To capture these view-dependent effects without a neural network, each Gaussian stores spherical harmonic (SH) coefficients instead of a single fixed color.

Spherical harmonics are a set of basis functions defined on the surface of a sphere. By storing coefficients for each SH band (up to degree 3, giving 16 coefficients per color channel), each Gaussian can represent how its color varies with viewing direction. Low-order terms capture diffuse color; higher-order terms model specular highlights and complex reflectance.

During rendering, the viewing direction from camera to each Gaussian is computed, and the SH coefficients are evaluated to produce the view-specific RGB color. This is much cheaper than querying a neural network and still captures rich appearance variation.

The deepest legacy of 3D Gaussian Splatting is the paradigm shift it embodied: moving from implicit neural representations back to explicit geometric primitives, but now equipped with differentiable optimization and modern GPU rasterization. It showed that classical computer graphics ideas — point splatting, alpha compositing, tile-based rendering — gain new power when combined with gradient-based learning. The field did not need a better neural network; it needed a better representation.