YOLOv3: An Incremental Improvement

By 2018, object detection had split into two camps:

• Two-stage detectors (Faster R-CNN, R-FCN): first propose regions, then classify each one. High accuracy but too slow for real-time video — hundreds of milliseconds per frame.

• Single-stage detectors (YOLOv2, SSD): process the whole image at once and predict boxes directly. Fast enough for video, but weaker on small objects and unable to handle overlapping categories.

YOLOv2 in particular had a glaring weakness: it used a relatively shallow backbone, detected at a single scale, and relied on softmax — which forced every object into exactly one class. A "woman" wearing a "dress" could only be labeled as one or the other.

The first upgrade is the feature extractor. YOLOv2 used Darknet-19 (19 convolutional layers). YOLOv3 replaces it with Darknet-53 — 53 convolutional layers organized into residual blocks, borrowing the skip-connection idea from ResNet.

Each residual block pairs a 1×1 convolution (to compress channels) with a 3×3 convolution (to extract spatial features), then adds the input back. This is the same "shortcut highway" pattern from ResNet: gradients flow through the skip connection without decaying, so the network can be much deeper without suffering from the degradation problem.

The result: Darknet-53 achieves the same classification accuracy as ResNet-152 on ImageNet, but runs nearly twice as fast because it uses strided convolutions instead of max-pooling and has no fully connected layers — every parameter is a convolutional filter.

YOLOv3 doesn't predict box coordinates from scratch. Instead, it starts from anchor boxes — pre-computed box shapes found by running k-means clustering on the training data's ground-truth boxes. Think of anchors as a set of "starting guesses" for common object shapes: tall and thin (like a person standing), wide and short (like a car), or roughly square (like a face).

The network then predicts offsets from these anchors. For each bounding box, the network outputs four numbers — $t_x, t_y, t_w, t_h$ — plus an objectness score $t_o$.

Why the sigmoid on center coordinates? Without it, the predicted center could land anywhere in the image — making training unstable. The sigmoid squeezes the output between 0 and 1, so the center stays within its own grid cell. It's like saying: "you can nudge the anchor's center anywhere inside this one cell, but you can't teleport it across the image."

The width and height use exponential scaling: the network predicts how much to stretch or shrink the anchor. Predicting $t_w = 0$ means "keep the anchor's width"; $t_w = 1$ means "multiply it by $e \approx 2.7$."

The objectness score $\sigma(t_o)$ tells us: "is there actually an object here, or is this box just background noise?" Each ground-truth object is assigned to exactly one anchor — the one with the highest IoU. If an anchor overlaps a ground-truth object by more than 0.5 but isn't the best match, it is simply ignored during training.

The biggest leap in YOLOv3 is predicting objects at three different scales, inspired by Feature Pyramid Networks (FPN). Here's the intuition:

Imagine you're looking at a city photo. To find skyscrapers, you need a wide-angle view — a coarse grid where each cell covers a large area. To find street signs, you need to zoom in — a fine grid where each cell covers a small area. YOLOv3 does both, plus one in between.

The network produces feature maps at three resolutions. For a 416×416 input:

• 13×13 (stride 32) — each cell sees a 32×32 pixel region. Large objects detected here using the 3 largest anchors.

• 26×26 (stride 16) — each cell sees a 16×16 region. Medium objects with mid-sized anchors.

• 52×52 (stride 8) — each cell sees an 8×8 region. Small objects with the 3 smallest anchors.

To build the finer scales, YOLOv3 upsamples the deeper (13×13) feature map and concatenates it with the earlier (26×26) feature map from the backbone. This is the FPN idea: the deep layers carry rich semantic meaning ("this is a dog") while the shallow layers carry fine spatial detail ("the edge is at pixel 47"). Merging them gives both.

YOLOv2 used softmax for class prediction, which forces all class probabilities to sum to 1. This means each detected object gets exactly one label. But real-world datasets often have overlapping categories: "woman" and "person", "car" and "vehicle".

YOLOv3 replaces softmax with independent logistic classifiers — one sigmoid per class. Each class gets its own binary yes/no decision. A bounding box can now be simultaneously "woman" (0.95) and "person" (0.98), because the scores don't compete.

The training loss for class prediction switches from cross-entropy (which assumes mutual exclusion) to binary cross-entropy — one loss term per class, independent of the others. This seemingly small change unlocked YOLO for datasets with hierarchical or overlapping labels.

The complete YOLOv3 architecture works as an information pipeline with three stages:

Stage 1 — Feature extraction (Darknet-53): The input image passes through 53 convolutional layers with residual blocks. Each Darknet block uses batch normalization and Leaky ReLU activation. Strided convolutions (stride=2) replace max-pooling for downsampling, reducing information loss.

Stage 2 — Feature pyramid (multi-scale neck): Features from three depths of the backbone are tapped off. The deepest features are upsampled and concatenated with earlier features, creating a pyramid where each level combines semantic richness with spatial precision.

Stage 3 — Detection heads: At each of the three scales, a small convolutional sub-network predicts a 3D tensor of shape $S \times S \times [3 \times (4 + 1 + C)]$ — for 3 anchors, each with 4 box coordinates, 1 objectness score, and C class scores. For COCO (80 classes) that's $S \times S \times 255$.

In total, for a 416×416 input, YOLOv3 predicts $(13^2 + 26^2 + 52^2) \times 3 = 10{,}647$ bounding boxes. Most are suppressed by low objectness and non-maximum suppression.

YOLOv3 hits a distinctive sweet spot on the speed-accuracy curve:

• At 320×320, it runs in 22 ms (45 FPS) at 28.2 mAP, matching SSD but 3× faster.

• At 416×416, it achieves 31.0 mAP in 29 ms.

• At 608×608, it reaches 33.0 mAP in 51 ms — comparable to RetinaNet's 57.5 mAP@50 but 3.8× faster.

On the older mAP@50 metric (which YOLOv3 excels at), it scores 57.9 mAP@50 at 608×608, nearly matching RetinaNet. However, on the stricter mAP@[.5:.95] metric, YOLOv3 falls behind — the paper honestly notes it struggles with precise box localization at higher IoU thresholds.

The honest reporting is part of what makes this paper special. Redmon openly lists what didn't work: focal loss dropped mAP by 2 points, linear x/y predictions destabilized training, and dual IoU thresholds "produced similar results." This transparency is rare and valuable.

YOLOv3 was not a single breakthrough moment — it was a masterclass in engineering refinement. By combining three proven ideas (deeper residual backbones, feature pyramids, and independent classifiers) into one cohesive system, Redmon showed that incremental improvements, done right, can transform a detector's capabilities.

More importantly, YOLOv3 set the architectural template that the entire YOLO family follows to this day: a deep convolutional backbone → multi-scale feature fusion → detection heads at each scale. YOLOv4, v5, v7, v8, and v11 all iterate on this same skeleton.