Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

Monocular depth estimation asks a deceptively simple question: given a single photograph, how far is each pixel from the camera? Unlike stereo vision, which uses two cameras to triangulate distance, monocular estimation relies on a single viewpoint — the same information a human uses to judge depth from a flat painting.

Before Depth Anything, models like MiDaS trained on collections of labeled datasets gathered from LiDAR sensors, stereo matching, and structure-from-motion. These sources are expensive, limited in diversity, and hard to scale. MiDaS v3.1 assembled about 1.5 million labeled images from up to 12 datasets — impressive, but still a tiny sample of the visual world. When encountering fog, low light, unusual indoor layouts, or ultra-remote landscapes, MiDaS often failed catastrophically.

The core insight of Depth Anything is that unlabeled monocular images are essentially free: they exist on the internet in billions, cover every conceivable scene, and require no special sensors. The challenge is how to use them effectively.

The pipeline has two stages. In the first stage, a teacher model is trained on 1.5 million labeled images from six public datasets (BlendedMVS, DIML, HRWSI, IRS, MegaDepth, and TartanAir). The encoder uses DINOv2 pre-trained weights for strong visual features. Training uses an affine-invariant loss that aligns each prediction's scale and shift with the ground truth — essential because different datasets measure depth in different units.

In the second stage, this teacher annotates 62 million unlabeled images collected from eight large-scale public datasets: SA-1B, Open Images, BDD100K, ImageNet-21K, LSUN, Objects365, Places365, and Google Landmarks. Each unlabeled image is simply passed through the teacher to produce a dense pseudo depth map — one forward pass per image, no stereo matching or expensive reconstruction needed.

The result is a combined training set of labeled and pseudo-labeled images, spanning indoor, outdoor, driving, aerial, and artistic scenes. This diversity is the key to generalization.

Different depth datasets use different scales and offsets. A LiDAR dataset might report depth in meters, while a synthetic dataset uses arbitrary units. Training on mixed datasets requires a loss function that ignores these differences and focuses on the relative ordering of depth values.

The solution is to normalize both the prediction and ground truth before computing the error. Each depth map is centered by subtracting its median and scaled by dividing by its mean absolute deviation. This affine alignment ensures that only the shape of the depth map matters, not its absolute values.

Here is the surprise the authors discovered. When they naively combined labeled images with pseudo-labeled images and trained a student, performance did not improve. The student was essentially memorizing what the teacher already knew — no new knowledge was gained.

The fix was counterintuitive: make the student's job harder. During training on unlabeled images, the student receives heavily corrupted versions: strong color jittering, Gaussian blur, and CutMix (random rectangular patches swapped between images). But the teacher's pseudo labels come from clean, undistorted images. The student must reconstruct the clean depth from a damaged input.

Think of it like a musician practicing with a metronome set slightly too fast. The difficulty forces development of skills that would not emerge from comfortable repetition. The student is compelled to seek invariant visual cues — edges, textures, spatial relationships — that survive the perturbations, building representations that generalize far better to unseen scenes.

CutMix was originally designed for image classification. Depth Anything adapts it for depth estimation. A random rectangular region from one unlabeled image is pasted onto another, creating a composite image $u_{ab}$. The student must predict depth for the whole composite, but the loss is computed separately in the two regions — each region compared to its own teacher pseudo label.

This is mathematically precise. The composite is formed as $u_{ab} = u_a \odot M + u_b \odot (1-M)$, where $M$ is a binary mask with a rectangle set to 1. The loss on each region uses its own affine normalization, because the two source images can have completely different depth distributions:

Depth and semantics are deeply linked. Knowing that a region is "sky" tells the model it is infinitely far; knowing it is a "table" constrains its possible depth. Prior works tried auxiliary semantic segmentation tasks, but the authors found these failed when the depth model was already strong — mapping rich visual features to a small set of class labels discards too much information.

The solution was elegant: instead of predicting semantic classes, align the student's feature space with DINOv2's rich, continuous feature space. DINOv2 has extraordinary semantic capabilities (image retrieval, segmentation) even with frozen weights. A simple cosine similarity loss transfers this semantic knowledge:

The final training objective is an average of three losses, each serving a distinct role. The labeled loss $\mathcal{L}_l$ teaches accurate depth from human annotations. The unlabeled loss $\mathcal{L}_u$ expands coverage to millions of diverse scenes through pseudo labels and hard perturbations. The feature alignment loss $\mathcal{L}_{feat}$ transfers semantic understanding from DINOv2.

Depth Anything uses a DINOv2 encoder with a DPT (Dense Prediction Transformer) decoder, following MiDaS. The encoder extracts multi-scale features using a Vision Transformer, and the DPT decoder fuses features from multiple layers to produce a dense depth prediction at the input resolution.

The model comes in three sizes: ViT-S (24.8M parameters), ViT-B (97.5M), and ViT-L (335.3M). Remarkably, the small ViT-S model — less than 1/10 the size of MiDaS — outperforms MiDaS on several benchmarks. The ViT-B model already clearly surpasses MiDaS ViT-L, demonstrating that the training strategy matters more than raw model size.

Key implementation details: images are resized with the shorter side to 518 pixels and cropped to 518 × 518 during training. The encoder learning rate is 5e-6, with the decoder at 10× higher. The labeled-to-unlabeled ratio in each batch is 1:2. The teacher always uses the ViT-L encoder for maximum annotation quality, even when training smaller student models.

Zero-shot relative depth. Depth Anything ViT-L surpasses MiDaS ViT-L on all six unseen benchmarks (KITTI, NYUv2, Sintel, DDAD, ETH3D, DIODE). On KITTI, the AbsRel error drops from 0.127 to 0.076 and $\delta_1$ accuracy jumps from 0.850 to 0.947 — and this is comparing against a MiDaS that actually trained on KITTI data, while Depth Anything never saw it.

Metric depth. Fine-tuning the Depth Anything encoder for metric depth estimation using ZoeDepth's framework sets new records on both NYUv2 ($\delta_1$: 0.964 → 0.984) and KITTI ($\delta_1$: 0.978 → 0.982). The gains also transfer to zero-shot metric depth on unseen indoor and outdoor datasets.

Semantic segmentation. The encoder also advances semantic segmentation: 86.2 mIoU on Cityscapes (vs. 84.6 for ConvNeXt-XL) and 59.4 mIoU on ADE20K (vs. 58.3 for ViT-Adapter with BEiT-L). This confirms the encoder's potential as a universal multi-task backbone for both middle-level and high-level visual perception.

The ablation studies tell a clear story. Starting from labeled-only training (baseline AbsRel 0.085 on KITTI), adding unlabeled images without perturbations gives virtually no gain (0.085). Adding strong perturbations drops the error to 0.081 — the unlabeled data finally contributes. Adding the feature alignment loss drops it further to 0.076, a 10.6% relative improvement. The pattern is consistent across all six benchmarks.

The tolerance margin $\alpha$ also matters. Without it ($\alpha = 1.0$, full alignment), the mean AbsRel across benchmarks is 0.188. With $\alpha = 0.85$, it drops to 0.175. This confirms that blindly aligning features hurts depth discrimination within objects.

Another key finding: applying feature alignment only to unlabeled data works better than applying it to labeled data. The authors explain that labeled data has high-quality ground truth, so the semantic loss can interfere with learning from precise annotations. For pseudo-labeled data, however, the semantic constraint combats noise in the pseudo labels while adding knowledge.