Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

By 2013, object detection on PASCAL VOC had been stuck for years. The dominant approach — the Deformable Parts Model (DPM) — built detectors from hand-crafted HOG features arranged in a star-shaped grammar of parts. DPM was a decade-old recipe: compute HOG descriptors, build part models, run sliding windows at multiple scales. It was slow, complex, and its accuracy had plateaued.

Meanwhile, AlexNet had just won ImageNet 2012 by a stunning margin, proving that deep CNNs learn far richer features than any hand-designed descriptor. The burning question was: can these powerful classification features also solve detection?

Detection is fundamentally harder than classification — you must predict what and where. A classifier only needs one label per image; a detector needs a bounding box and label for every object instance. This demands both recognition power and spatial precision.

Instead of sliding a window over every possible location and scale — which would mean millions of evaluations — R-CNN uses Selective Search to propose roughly 2,000 candidate regions per image. Think of it as a fast pre-filter: before deploying the expensive CNN, a lightweight algorithm identifies bounding boxes that are likely to contain something.

Selective Search works bottom-up: it starts from a fine-grained over-segmentation of the image (many tiny regions), then iteratively merges similar neighboring regions based on color, texture, size, and containment. At every merge step, it records the bounding box of the merged region as a proposal. This hierarchical grouping naturally produces proposals at multiple scales — from small objects like cups to large ones like cars — without any sliding window.

The result is a set of roughly 2,000 class-agnostic bounding boxes. "Class-agnostic" means the algorithm doesn't know what the object is — it just flags regions that look like they contain something interesting. The what question comes later, when the CNN examines each region.

Each of the ~2,000 proposed regions is warped (resized) to 227×227 pixels — the fixed input size that AlexNet expects — regardless of the region's original aspect ratio. This brute-force resizing distorts some objects, but R-CNN shows it works well enough in practice.

The warped region is then fed through a CNN (originally AlexNet, later VGGNet) that was pre-trained on ImageNet for the 1000-class classification task. But instead of taking the final classification output, R-CNN extracts the 4096-dimensional activation vector from the layer just before the final classifier (fc7). This vector is the region's feature representation — a compact summary of everything the CNN "sees" in that patch.

Here is the crucial insight about transfer learning: the CNN was never trained on PASCAL VOC or on detection at all, yet its features are incredibly useful because ImageNet taught it to recognize edges, textures, parts, and objects. The lower layers detect universal visual patterns (edges, corners, color blobs), while higher layers detect object-specific features (eyes, wheels, fur). These features transfer to new tasks with remarkable effectiveness.

To bridge the gap between ImageNet classification and VOC detection, R-CNN fine-tunes the pre-trained CNN on the detection data. It replaces AlexNet's 1000-way classifier with a (K+1)-way classifier (K object classes + background) and continues training with warped region proposals, using an IoU threshold of 0.5 to decide positive vs. negative examples.

The deepest lesson of R-CNN is not the pipeline itself but the transfer learning recipe it validated. ImageNet has 1.2 million labeled images for classification; PASCAL VOC has only a few thousand for detection. Training a deep CNN from scratch on VOC would overfit immediately.

R-CNN's solution: pre-train the CNN on ImageNet's abundant data, where it learns rich visual features — a universal "visual vocabulary". Then fine-tune on the small detection dataset, where the network adapts those general features to the specific task. This two-step recipe — pre-training then fine-tuning — turned out to be one of the most important ideas in deep learning, eventually extending to NLP (BERT, GPT) and beyond.

The ablation studies in the paper tell a striking story: without fine-tuning, R-CNN's CNN features already outperformed HOG-based methods. With fine-tuning, mAP jumped by another 8 points. The fine-tuned fc7 features were the most discriminative — proof that the highest layers adapt most to the new domain.

After extracting a 4096-d feature vector from each region proposal, R-CNN feeds it into one linear SVM per object class. For a 20-class dataset like PASCAL VOC, that means 20 separate binary classifiers, each asking: "Does this region contain class X?"

Why SVMs instead of the CNN's own softmax classifier? The paper explored this question carefully. The softmax classifier trained during fine-tuning actually performs slightly worse than SVMs. The reason: fine-tuning uses loosely labeled data (any region with IoU ≥ 0.5 is "positive"), while SVM training uses stricter labels (only ground-truth boxes are positive, and IoU < 0.3 is negative). This stricter positive/negative definition gives SVMs a sharper decision boundary.

Once all SVMs have scored every proposal for every class, Non-Maximum Suppression (NMS) eliminates redundant overlapping detections. For each class independently, NMS ranks proposals by SVM score and iteratively discards any proposal whose IoU with a higher-scoring proposal exceeds a threshold (typically 0.3). The result: one clean bounding box per detected object.

Selective Search proposes rough bounding boxes, but they rarely align perfectly with the actual object. R-CNN adds a bounding box regressor — a simple linear model trained to predict four offsets ($t_x, t_y, t_w, t_h$) that shift and resize the proposed box to better fit the ground-truth object.

Think of it as fine-tuning the crop: the initial proposal says "there's probably a dog around here", and the regressor says "shift left by 5 pixels, shrink the height by 10%". This refinement typically boosts mAP by 3–4 points.

R-CNN uses IoU everywhere: to decide if a proposal is positive or negative during training, to set the threshold for NMS, and to evaluate final detection quality. IoU measures how well a predicted bounding box aligns with the ground-truth box:

R-CNN's training is a three-stage sequential process — one of its main drawbacks compared to later methods:

Stage A — Fine-tune the CNN. Take AlexNet pre-trained on ImageNet. Replace the 1000-way classifier with a randomly initialized (K+1)-way classifier. Train on warped region proposals: proposals with IoU ≥ 0.5 with any ground-truth box are positive; the rest are negative. Use SGD with a learning rate of 0.001 (1/10 of the original ImageNet rate).

Stage B — Train SVMs. Freeze the fine-tuned CNN and extract features for all proposals. Train one linear SVM per class, using ground-truth boxes as positives and proposals with IoU < 0.3 as hard negatives. Note the stricter labels compared to Stage A.

Stage C — Train bounding box regressors. Using the same frozen features, train per-class linear regressors to predict box offsets. Only proposals with IoU ≥ 0.6 with a ground-truth box are used as training examples.

This sequential pipeline is complex and disk-space-hungry (features must be cached for all proposals across all images). Fast R-CNN would later unify all three stages into a single end-to-end trainable network.

R-CNN's impact was immediate and dramatic:

On PASCAL VOC 2012, R-CNN achieved 53.3% mAP — a relative improvement of more than 30% over the previous best (DPM at ~40%).

On ILSVRC 2013 detection, R-CNN achieved 31.4% mAP, compared to the winner OverFeat's 24.3%.

The ablation studies revealed a clear pattern: (1) CNN features alone (pool5) already outperform HOG. (2) Higher layers (fc6, fc7) give progressively better features. (3) Fine-tuning adds ~8 mAP points. (4) Bounding box regression adds another 3–4 points. (5) Replacing AlexNet with the deeper VGGNet-16 pushed mAP to 66.0% on VOC 2007.

R-CNN also tackled semantic segmentation — classifying every pixel in the image, not just drawing boxes. The approach: for each region proposal, compute CNN features on (a) the full rectangular region, (b) only the foreground pixels (with background masked out), and (c) the foreground of a slightly larger region. Concatenate these feature vectors and train an SVM per class.

This achieved competitive results on VOC 2011 segmentation, demonstrating that the same CNN features useful for detection also capture the information needed for pixel-level classification. Later work like Fully Convolutional Networks (FCN) would surpass this approach, but R-CNN showed the path.

R-CNN had clear limitations that motivated its successors:

Speed. Running the CNN on ~2,000 proposals independently took ~47 seconds per image — far too slow for real-time use. Most computation was redundant since overlapping proposals share pixels.

Multi-stage training. Three separate training stages (CNN fine-tuning, SVM training, bbox regressor training) made the pipeline complex and hard to optimize jointly.

Disk space. Caching features for all proposals across all images required hundreds of gigabytes.

Fixed warping. Forcing every region to 227×227 destroys aspect ratio information that could help recognition.

These problems inspired a rapid succession of improvements: