Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

The evolution from R-CNN to Fast R-CNN solved the repeated-computation problem: instead of running a CNN on every proposed region independently, Fast R-CNN runs the CNN once on the whole image and pools features from the shared feature map for each region. Detection became fast — but the proposals still came from Selective Search, a handcrafted CPU algorithm that tested ~2000 region candidates per image in roughly 2 seconds.

The GPU detection network sat idle, waiting for the CPU to finish generating proposals. It was as if you had built a high-speed highway but were stuck at a manual toll booth at the entrance.

The key observation is beautifully simple: the convolutional feature map that Fast R-CNN already computes for detection contains rich spatial information about where objects are. Why not use that same feature map to also propose regions?

The Region Proposal Network (RPN) is a small network that slides over this shared feature map. At each spatial location, it does two things simultaneously:

• Predicts an objectness score — is there any object here, regardless of class?
• Refines the coordinates of k anchor boxes — predefined reference rectangles of different sizes and shapes centered at that location.

Because the RPN operates on features the backbone has already computed, generating proposals costs almost nothing extra — roughly 10 milliseconds per image.

How does the RPN handle objects of vastly different sizes — a car versus a person versus a traffic light — without using image pyramids or filter pyramids?

The answer is anchor boxes. At each sliding-window position on the feature map, the network places k reference rectangles (anchors) of different scales and aspect ratios. The original paper uses 3 scales (128², 256², 512²) × 3 aspect ratios (1:1, 1:2, 2:1) = 9 anchors per position.

Think of it as placing 9 transparent frames of different sizes and shapes at every point on the feature map — like a photographer trying different viewfinders. The network then learns to say "yes, there's an object in this frame" or "no, skip this one," and for the positive frames, it learns how to nudge the frame boundaries to better fit the actual object.

This is far more efficient than scanning the image at multiple resolutions. The anchors act as built-in multi-scale priors, and the network only needs to learn refinements from these starting positions rather than raw coordinates from scratch.

The RPN architecture is surprisingly compact. On top of the shared convolutional feature map (e.g. from the last convolutional layer of VGG-16 or ResNet), two things happen:

Step 1: Sliding window. A 3×3 convolutional layer slides over the feature map, producing a 256-dimensional (or 512-dimensional) feature vector at each spatial position. This is the "looking through every viewfinder" step.

Step 2: Twin heads. Two sibling 1×1 convolutional layers branch from this intermediate layer:

• The classification head outputs 2k scores (object vs. not-object for each of k anchors).
• The regression head outputs 4k coordinates (dx, dy, dw, dh refinements for each anchor).

For k = 9 anchors, each position produces 18 classification scores and 36 bounding-box refinements. On a feature map of size W×H, the total number of anchors is W × H × 9 — typically around 20,000 for a 1000×600 input image.

With ~20,000 anchors per image, most will be background. To train the RPN, each anchor is labeled as either positive (contains an object) or negative (background):

• An anchor is positive if it has the highest IoU with any ground-truth box, or if its IoU with any ground-truth box exceeds 0.7.
• An anchor is negative if its IoU with all ground-truth boxes is below 0.3.
• Anchors between 0.3 and 0.7 IoU are ignored during training — they neither help nor hurt.

From these labeled anchors, a mini-batch of 256 is sampled per image (up to 128 positive, the rest negative). This balanced sampling prevents the overwhelming majority of negatives from dominating the learning signal.

The RPN is trained with a multi-task loss that combines two objectives. First, understand the intuition: we want the network to learn both where objects are (by labeling anchors as object/background) and how to adjust anchor boundaries to fit objects tightly. These are a classification problem and a regression problem respectively, combined into one loss.

The term $p_i^*$ in the regression loss acts as a gate: regression loss is computed only for positive anchors. There is no point in refining a box if there is no object inside it. The two normalizations ($N_{cls}$ for mini-batch size and $N_{reg}$ for the number of anchor positions) ensure the two tasks contribute equally to learning.

Here is the complete detection flow, from raw image to final bounding boxes:

Stage 1 — Shared backbone. The input image passes through a deep CNN (VGG-16 or ResNet). The output is a feature map — a compressed, semantically rich representation of the image.

Stage 2 — RPN proposals. The RPN slides over this feature map, producing ~20,000 anchors. After filtering by objectness score and applying non-maximum suppression (keeping only the highest-scoring, non-overlapping proposals), roughly 300 top proposals remain.

Stage 3 — RoI Pooling. Each proposal is mapped back to the shared feature map and pooled into a fixed-size feature vector (e.g. 7×7) using RoI Pooling. This allows the downstream classifier to accept proposals of any size.

Stage 4 — Fast R-CNN head. The pooled features pass through fully connected layers that produce two outputs per proposal: a class probability distribution (softmax over C+1 classes including background) and 4 refined bounding-box coordinates per class.

Stage 5 — Final NMS. Non-maximum suppression removes duplicate detections, yielding the final set of bounding boxes with class labels and confidence scores.

Training a unified network where proposal and detection share features is not trivial. The paper proposes a four-step alternating training strategy:

• Step 1: Train the RPN alone, initialized from an ImageNet-pretrained backbone.
• Step 2: Train a separate Fast R-CNN detector using the proposals from Step 1's RPN. This detector is also initialized from ImageNet but does not yet share features with the RPN.
• Step 3: Re-initialize the RPN with the detector's shared convolutional layers (from Step 2), but freeze these shared layers and fine-tune only the RPN-specific layers. Now both networks share the same backbone.
• Step 4: Keeping the shared layers frozen, fine-tune the Fast R-CNN-specific layers.

After these four steps, both networks share the same convolutional features, forming a single unified network. In practice, later work showed that approximate joint training (backpropagating through both networks simultaneously) also works well.

A critical property of the anchor mechanism is translation invariance. The same set of anchors and the same prediction function are applied at every position on the feature map. If an object shifts in the image, the same anchor at a different position will detect it.

This is analogous to how a convolutional filter detects the same edge everywhere — the RPN extends this principle to object proposals. Translation invariance means the model size does not grow with image size, and the network generalizes to objects at any location.

In contrast, earlier methods like MultiBox used position-specific predictions — they needed separate parameters for each spatial location, couldn't generalize across positions, and required far more parameters.

On PASCAL VOC 2007, Faster R-CNN with VGG-16 achieved 73.2% mAP at 5 fps — a significant speed-up over the Selective Search baseline while matching or exceeding its accuracy. With the deeper ResNet-101 backbone, accuracy reached 78.8% mAP on VOC 2012.

The RPN itself generates proposals with higher recall at fewer proposals than Selective Search: 300 RPN proposals achieve the same recall as 2,000 Selective Search regions. This is because the RPN proposals are learned and tuned for the specific task, while Selective Search is a generic, hand-engineered algorithm.

On the MS COCO benchmark, Faster R-CNN set new state-of-the-art results, demonstrating that the framework scales to larger, more challenging datasets with more object categories.

Both the RPN and the final detection stage produce overlapping predictions — many neighboring anchors may fire for the same object. Non-maximum suppression (NMS) cleans this up:

• Sort all proposals by their objectness score (or class confidence).
• Take the highest-scoring proposal.
• Remove any remaining proposal whose IoU with the selected one exceeds a threshold (typically 0.7 for RPN, 0.3 for final detection).
• Repeat until no proposals remain.

Think of NMS as a "winner-takes-all" filter: when multiple overlapping frames claim the same object, only the most confident one survives. This is essential because without it, the same car might be detected dozens of times.

From R-CNN's 47 seconds per image to Faster R-CNN's 0.2 seconds — a 200× speedup in 18 months. The key was moving proposals inside the network and sharing computation. This principle — unify, share, learn end-to-end — is the thread connecting every advance in this timeline.