Mask R-CNN

Before Mask R-CNN, computer vision had three separate tools for understanding images, each with its own blind spot:

Object detection draws a bounding box around each object and labels its class. It tells you where and what, but not the exact shape — the box includes background pixels too.

Semantic segmentation labels every pixel by category — all "car" pixels become blue, all "tree" pixels become green. But if two cars overlap, their pixels merge into one blob. You lose the count and individual identities.

Instance segmentation is the holy grail: detect each object separately and trace its exact pixel boundary. You know there are three cars, and you know the precise silhouette of each one. Before 2017, no system did this cleanly in a single pass.

Mask R-CNN is an extension of Faster R-CNN, so understanding the base is essential. The pipeline has two stages:

Stage 1 — Region Proposal Network (RPN). A lightweight convolutional network slides over the backbone's feature map. At every spatial position, it evaluates a set of anchor boxes of different sizes and aspect ratios, predicting whether each anchor contains an object and refining its coordinates. The output is a few hundred candidate regions (proposals).

Stage 2 — Classification + Box Regression. For each proposal, a small feature patch is extracted from the feature map (via RoI pooling), flattened, and fed through fully connected layers to predict (a) the object's class and (b) refined bounding-box coordinates.

Mask R-CNN's insight: add a third output head in stage 2 that predicts a binary segmentation mask for each proposal — in parallel with classification and box regression, sharing the same extracted features.

Here is where Mask R-CNN makes its most impactful technical contribution. Faster R-CNN used RoI pooling to extract a fixed-size feature patch from each proposal. But RoI pooling performs two rounds of quantization — first rounding the proposal's coordinates to the nearest integer on the feature map, then rounding again when dividing the RoI into bins. Each rounding step shifts features by a fraction of a pixel.

For bounding-box classification, these tiny shifts barely matter — a box just needs to roughly localize the object. But for pixel-level masks, a one-pixel misalignment at 1/16 resolution means a 16-pixel shift in the original image. The mask head receives features that no longer correspond to the right pixels, and accuracy collapses.

Think of it like a projector slightly out of focus: the audience can still recognize the slide (classification works), but they cannot read the fine print (mask precision is destroyed).

RoIAlign fixes this with a conceptually simple change: never round anything. Instead of snapping coordinates to integer grid points, RoIAlign keeps all coordinates as floating-point numbers and uses bilinear interpolation to compute feature values at exact sub-pixel locations.

The algorithm: given an RoI of arbitrary size, divide it into a fixed grid (e.g. 7×7). In each bin, sample at 4 regularly-spaced points. Each sample point falls between grid pixels on the feature map, so its value is computed as a weighted average of the 4 nearest feature-map pixels (bilinear interpolation). Then average or max-pool the 4 samples within each bin.

No rounding. No quantization. The extracted features faithfully reflect the exact spatial region proposed by the RPN. This seemingly minor change improved mask AP by 10–50% relative, with the largest gains on strict localization metrics.

Once RoIAlign extracts a spatially-aligned feature map for each proposal, the mask branch processes it through a small fully convolutional network — a series of convolutional layers that preserve spatial layout, ending with a deconvolution that upsamples to the final mask resolution (typically 28×28 pixels for each RoI).

The crucial design choice: the mask branch predicts K binary masks — one per class — using a per-pixel sigmoid activation, not a softmax. This means mask prediction is completely decoupled from class prediction. The network doesn't need to decide what the object is while simultaneously deciding where it is pixel by pixel. The classification head handles the "what"; the mask head only asks "is this pixel part of the object, yes or no?"

Why does this matter? When you use softmax across classes at the pixel level, the mask for "cat" and the mask for "dog" compete — activating one suppresses the other. But once you know (from the box classifier) that the object is a cat, this competition is wasteful and harmful. Decoupling improved mask AP by 5.5 points in the paper's ablations.

Mask R-CNN trains end-to-end with a multi-task loss that simply sums three components. The elegance is that no component interferes with the others — they share features but compute independent gradients:

Mask R-CNN's architecture is modular — each component can be swapped independently. The paper evaluates two backbone configurations:

ResNet-C4: Uses the last convolutional stage (C4) of ResNet as the feature extractor. RoIAlign extracts 14×14 features from the C4 map, which are processed by the heavy C5 stage (ResNet's final block) shared between detection and mask heads. Computationally heavier but conceptually simpler.

ResNet-FPN: Combines ResNet with a Feature Pyramid Network that builds multi-scale features. Proposals at different scales are extracted from different FPN levels. The mask head is a separate lightweight FCN (4 conv layers + deconv). This is faster and more accurate, and became the standard configuration.

Real images contain objects at many scales — a person close to the camera fills most of the frame, while a distant car occupies just 20×15 pixels. A single feature map at one resolution cannot serve both well: fine resolution captures small objects but lacks semantic depth, while coarse resolution has rich semantics but loses small objects entirely.

The Feature Pyramid Network (FPN) solves this by building a top-down pathway with lateral connections. It takes the natural multi-scale feature hierarchy of a CNN (shallow layers have high resolution, deep layers have low resolution) and augments each level with strong semantic information from the top. The result is a pyramid where every level has both high resolution and rich semantics.

In Mask R-CNN with FPN, each proposal is assigned to a specific pyramid level based on its area. Small proposals get features from higher-resolution levels; large proposals from lower-resolution levels. This is why Mask R-CNN handles objects from tiny to huge.

On the COCO 2016 benchmark, Mask R-CNN with ResNet-101-FPN achieved:

• Instance segmentation: 35.7 mask AP — outperforming all prior single-model entries.
• Object detection: 39.8 box AP — surpassing the Faster R-CNN baseline by simply adding the mask branch (the mask branch actually improved detection).
• Keypoint detection: 65.5 keypoint AP — by replacing the mask output with keypoint heatmaps, demonstrating the framework's flexibility.

A remarkable finding: adding the mask branch helped detection. The shared features learned richer representations because the mask task forced the backbone to preserve spatial information that pure box regression didn't demand. This is the benefit of multi-task learning — auxiliary tasks regularize the shared backbone.

The paper's ablation studies reveal which design choices carry the most weight:

RoIAlign vs. RoI Pool. Switching from RoI Pool to RoIAlign improved mask AP from 23.6 to 30.9 on the ResNet-50-C4 backbone — a massive 7.3 point jump (31% relative). On stricter IoU thresholds (AP₇₅), the gap was even larger. This confirms that spatial alignment is critical for pixel-level tasks.

Decoupled binary masks vs. multinomial masks. Using per-pixel sigmoid (decoupled) instead of per-pixel softmax (multinomial) gained 5.5 AP points. The lesson: once the box branch has classified the object, the mask branch should focus solely on foreground/background separation without competing across classes.

Class-specific vs. class-agnostic masks. Surprisingly, class-agnostic masks (one binary mask regardless of class) performed nearly as well as class-specific masks (29.7 vs. 30.3 AP). This confirms the decoupling principle — the mask really only needs to separate foreground from background.

Backbone depth. Deeper backbones helped significantly: ResNet-101 gained 1.1 AP over ResNet-50, and ResNeXt-101 added another 0.7 AP. The mask branch benefits from richer feature representations.

One of Mask R-CNN's most elegant demonstrations is human pose estimation. By replacing the mask head with a head that predicts K keypoint heatmaps (one per joint: left shoulder, right knee, etc.), the same framework achieves state-of-the-art keypoint detection. No architectural changes to the backbone or RPN — just a different output head.

This proves that Mask R-CNN is not just an instance segmentation model — it is a general framework for instance-level recognition tasks. Any task that requires (1) detecting objects and (2) predicting something spatially precise within each detection can be plugged into this framework. The recipe is always the same: backbone → FPN → RPN → RoIAlign → task-specific head.

Mask R-CNN's lasting contribution is not just a model — it is the proof that detection and segmentation are not separate problems. By adding one branch and fixing one alignment bug, the gap between "where is the object?" and "what shape is the object?" collapsed. Every modern instance-level vision system — from autonomous driving to medical imaging — inherits this unification.