Fast R-CNN

The original R-CNN pipeline had three separate stages, each with its own training procedure:

1. Feature extraction — run a full CNN forward pass on each of ~2,000 cropped-and-warped region proposals, then cache the features to disk.
2. Classification — train a linear SVM for each object class on the cached features.
3. Bounding-box regression — train a separate regressor to refine each proposal's coordinates.

This multi-stage design was slow (84 hours to train on VOC 2007), wasteful (hundreds of gigabytes of cached features), and fundamentally limited: because each stage was trained independently, errors in one stage could not be corrected by another. The CNN backbone could not be fine-tuned through the SVM or regressor, so detection-specific learning stopped at the feature level.

The most expensive part of a CNN is the convolutional layers — the early layers that extract edges, textures, and patterns. R-CNN ran these layers 2,000 times per image (once per proposal). But here's the insight: all proposals come from the same image, so they should share the same convolutional features.

Fast R-CNN's solution: run the CNN on the whole image once to produce a single shared feature map. Then, for each region proposal, simply look up the corresponding region on that feature map. This is like building a single detailed blueprint of the entire building, then using it to answer questions about any room — instead of surveying each room from scratch.

SPPnet (He et al., 2014) had the same sharing idea, but its spatial pyramid pooling layer blocked gradient flow to the convolutional layers. Fast R-CNN replaced it with a simpler RoI pooling layer that allows full backpropagation through every layer.

Region proposals come in all shapes and sizes — a person is tall and thin, a car is wide and low — but the fully connected layers at the end of the network demand a fixed-size input. RoI pooling solves this mismatch.

Here's how it works. Given a region proposal (say 21 × 14 on the feature map) and a target output size (say 7 × 7), RoI pooling divides the region into a 7 × 7 grid of sub-windows. Each sub-window is approximately 3 × 2 cells. Within each sub-window, max pooling selects the largest value. The result: a fixed 7 × 7 × C feature tensor regardless of the original region's shape.

Think of it as a cookie cutter on the feature map: no matter the region's shape, RoI pooling stamps out the same fixed-size summary. This is a simplified single-level version of spatial pyramid pooling from SPPnet — but critically, it allows gradients to flow backward through the pooling into the convolutional layers, enabling end-to-end training.

The full pipeline flows as follows:

1. The entire input image passes through a pre-trained CNN backbone (e.g., VGG16) up to the last convolutional layer, producing a shared feature map.

2. An external algorithm (Selective Search) proposes ~2,000 candidate regions on the original image.

3. Each proposal is projected onto the shared feature map, and RoI pooling extracts a fixed-size (e.g., 7 × 7 × 512) feature vector.

4. The feature vector passes through two fully connected layers (fc6 and fc7, each 4096 units with ReLU and dropout).

5. The output forks into two sibling heads:

   • A (K+1)-way softmax producing class probabilities (K object classes + background).
   • A bounding-box regressor producing 4 refined coordinates per class.

6. Both heads are trained jointly with a multi-task loss that combines classification cross-entropy and bounding-box smooth L1 loss.

The genius is that steps 1–6 form a single differentiable computation graph. Gradients flow from both output heads, through the fully connected layers, through RoI pooling, and into the convolutional backbone — updating every weight simultaneously.

Instead of training classification and localization as separate stages (as R-CNN did), Fast R-CNN trains both tasks simultaneously using a single combined loss function. This is one of the paper's core innovations: the multi-task loss.

For each RoI, the network outputs two things: a probability distribution $p = (p_0, \ldots, p_K)$ over K+1 classes (including background), and bounding-box regression offsets $t^u = (t^u_x, t^u_y, t^u_w, t^u_h)$ for each class $u$. The true class is $u$ and the true bounding-box target is $v$.

The key design choice is the indicator $[u \geq 1]$: background proposals (class 0) have no ground-truth box, so their regression loss is zeroed out. Only foreground objects contribute to the localization loss. The hyperparameter $\lambda$ controls the balance between the two tasks — the paper uses $\lambda = 1$, giving equal weight to classification and localization.

For bounding-box regression, Fast R-CNN introduced the smooth L1 loss, which blends the best properties of L1 and L2 losses:

• When the error is small (|x| < 1), it behaves like L2 (squared error): smooth, differentiable, and converges precisely.
• When the error is large (|x| ≥ 1), it behaves like L1 (absolute error): grows linearly rather than quadratically, preventing outliers from producing explosive gradients.

This is critical for training stability. Bounding-box targets can include large offsets (when a proposal is far from the ground truth), and L2 loss would turn those into enormous gradients that destabilize training. Smooth L1 caps the gradient magnitude at 1 for large errors, making training robust to outliers without requiring aggressive gradient clipping.

A subtle but important training innovation is hierarchical sampling of mini-batches. Instead of randomly sampling RoIs from across many images (which would require computing a separate feature map for each image in the batch), Fast R-CNN samples 2 images per mini-batch and 64 RoIs from each image, for a total of 128 RoIs.

Of these 128 RoIs, 25% are positive (IoU ≥ 0.5 with a ground-truth box) and 75% are negative (IoU in [0.1, 0.5)). This 1:3 ratio ensures the network sees enough negative examples (background) without being overwhelmed by them.

Because all 64 RoIs from the same image share the same convolutional feature map, the expensive forward pass happens only twice per mini-batch (once per image) rather than 128 times. This makes training both faster and memory-efficient.

On PASCAL VOC 2007, Fast R-CNN with VGG16 achieves 70.0% mAP — surpassing R-CNN's 66.0%. On VOC 2012 it reaches 68.4% mAP vs. R-CNN's 62.4%.

But the real story is speed. Training time dropped from 84 hours (R-CNN) to 9.5 hours — a 9× speedup. Test-time inference dropped from 47 seconds per image to 0.32 seconds (excluding proposal generation) — a 146× speedup. Including Selective Search (~2 seconds), total inference is ~2.3 seconds, which is still a 213× speedup over R-CNN's full pipeline.

The paper also compares against SPPnet, showing that Fast R-CNN trains 3× faster and tests 10× faster while achieving higher accuracy, thanks to the ability to fine-tune all layers.

Fast R-CNN solved the training and feature-sharing problems, but one piece of the pipeline remained outside the neural network: Selective Search. This CPU-based algorithm takes about 2 seconds per image — longer than Fast R-CNN's entire neural network inference (0.32s). It accounts for 86% of total test time.

This observation pointed directly to the next breakthrough: what if we replace Selective Search with a small neural network that proposes regions on the GPU? That idea became the Region Proposal Network (RPN) in Faster R-CNN, published just one month later by Ren, He, Sun, and Girshick himself.

Fast R-CNN established three principles that shaped every subsequent object detector:

Shared convolutional features. Never repeat the same computation. Run the CNN once, reuse the feature map for all proposals. This idea lives on in every modern detector — from Faster R-CNN to YOLO to DETR.

Multi-task learning. Classification and localization benefit from being trained together. Joint losses became the standard for all detection architectures.

End-to-end trainability. Every component should be differentiable so gradients can optimize the entire system. When something blocks gradient flow (like SPPnet's pooling or Selective Search), it becomes the target for the next innovation.