Focal Loss for Dense Object Detection

Object detection works by examining thousands of candidate boxes across the image. A two-stage detector like Faster R-CNN handles this in two phases: first a Region Proposal Network filters ~100k raw anchors down to ~2,000 promising ones, then a classifier examines only those 2,000. This filtering elegantly avoids the imbalance problem — most background is thrown away before it can affect training.

One-stage detectors skip the filtering. They classify all ~100k anchors at once. This is faster, but it means the training loss is computed over all 100k, and roughly 99.9% are easy background. The cross-entropy loss treats every example equally: an anchor the model already classifies correctly at 99% confidence still contributes a small but nonzero loss. Multiply that small loss by 99,000 easy negatives, and it drowns the signal from the ~100 anchors that actually contain objects.

The standard cross-entropy loss for binary classification is:

$CE(p, y) = -\log(p_t)$

where $p_t = p$ if the ground truth $y = 1$, and $p_t = 1 - p$ otherwise. In other words, $p_t$ is the model's estimated probability for the correct class.

The key weakness: even when $p_t$ is high (say 0.9 — a correctly classified, easy example), $-\log(0.9) = 0.105$ is still a nontrivial loss. One such example is harmless; 99,000 of them contribute a collective loss of ~10,000, which overwhelms the few hundred hard positives.

A common patch is α-balancing: weighting the loss by class frequency. Setting α = 0.75 for the rare foreground class and 0.25 for the common background class helps, but it only rebalances how much each class matters — it doesn't distinguish easy from hard within each class.

The insight is elegant: multiply the cross-entropy by a modulating factor that automatically fades to zero as the model becomes more confident. Easy examples suppress themselves; hard examples stay loud.

Let's make this concrete with numbers. Consider an easy negative with $p_t = 0.9$ and γ = 2:

• Cross-entropy: $-\log(0.9) = 0.105$
• Focal loss factor: $(1 - 0.9)^2 = 0.01$
• Focal loss: $0.01 \times 0.105 = 0.00105$ — 100× smaller

Now a hard example with $p_t = 0.1$:

• Cross-entropy: $-\log(0.1) = 2.30$
• Focal loss factor: $(1 - 0.1)^2 = 0.81$
• Focal loss: $0.81 \times 2.30 = 1.86$ — barely reduced

The easy example's contribution drops by two orders of magnitude. The hard example's loss stays almost intact. This is the whole mechanism: a single multiplicative term that converts cross-entropy from a "count every vote equally" function into a "listen to the struggling students" function.

The paper includes a revealing analysis (Figure 4): after training, plot the cumulative distribution of the loss across all foreground and background examples.

With standard cross-entropy (γ = 0), loss is spread nearly uniformly — the flood of easy backgrounds contributes roughly as much total loss as the hard cases. With focal loss (γ = 2), the picture changes dramatically: for background examples, nearly all the loss is concentrated in the hardest few percent. The easy 90%+ contribute almost nothing.

For foreground examples, the distribution barely changes — focal loss doesn't penalize hard foreground examples, it only silences the background noise.

To prove that focal loss — not architecture — was the missing ingredient, the authors deliberately kept the detector simple. RetinaNet has three parts:

1. Backbone: Feature Pyramid Network (FPN) — built on top of ResNet, FPN constructs a multi-scale feature pyramid. Lower levels (P3) have high resolution for small objects; upper levels (P7) have low resolution but large receptive fields for big objects. Lateral connections blend fine spatial detail from bottom-up features with semantic richness from top-down features.

2. Classification subnet — a small fully-convolutional network (4 conv layers + sigmoid) attached to each FPN level. For every spatial position and every anchor (A = 9), it predicts K class probabilities. Parameters are shared across all pyramid levels.

3. Box regression subnet — an identical structure but predicting 4 offset values (dx, dy, dw, dh) per anchor, specifying how to shift and resize each anchor box to fit the detected object.

At every spatial position on every FPN level, RetinaNet places A = 9 anchor boxes: 3 aspect ratios (1:2, 1:1, 2:1) × 3 scales ($2^0$, $2^{1/3}$, $2^{2/3}$). These anchors serve as reference templates. The box regression subnet doesn't predict absolute coordinates — it predicts small adjustments to these anchors.

Anchors with IoU ≥ 0.5 against a ground-truth box are assigned as positives; those with IoU < 0.4 are negatives; the rest are ignored during training. This yields ~100k anchors per image, of which typically only ~10–100 are positive.

A critical training trick: the final classification layer is initialized with bias $b = -\log((1-\pi)/\pi)$ where π = 0.01, so every anchor starts with a prior probability of only 1% for foreground. Without this, the massive initial loss from 100k anchors all predicting 50% foreground would destabilize training.

Before this paper, the standard explanation for the one-stage accuracy gap was: "Two-stage detectors have better architecture." The region proposal network provides spatial attention — it tells the classifier where to look. One-stage detectors lack this.

The paper's key insight is different: the gap is not about where you look — it's about how you train. Two-stage detectors achieve class balance implicitly: the proposal stage discards 99% of easy negatives, and the second stage uses a 1:3 positive-to-negative sampling ratio. Focal loss achieves the same balance explicitly, through the loss function itself.

The proof is in the numbers: RetinaNet with focal loss (γ = 2, α = 0.25) achieves 36.0 AP on COCO minival — far above the best OHEM variant at 32.8 AP, and above Faster R-CNN with FPN at 36.2 AP. A cleaner approach to training beat architectural sophistication.

Before focal loss, the main strategy for handling imbalance in one-stage detectors was Online Hard Example Mining (OHEM): sort examples by loss, apply non-maximum suppression, then keep only the highest-loss ones for each mini-batch.

OHEM has two weaknesses: (1) it completely discards easy examples, losing any remaining learning signal they might offer; (2) it introduces extra hyperparameters — NMS threshold and batch size — that require tuning. The paper tests many OHEM configurations (batch sizes 128, 256, 512; NMS thresholds 0.5, 0.7; with and without 1:3 ratio enforcement) and finds the best reaches 32.8 AP. Focal loss reaches 36.0 with no sampling, no NMS on training data, and only two hyperparameters (γ, α) that are robust across a wide range.

The difference is philosophical: OHEM is a sampling strategy that selects what to train on. Focal loss is a continuous reweighting that trains on everything but listens more carefully to what matters.