Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows

The original Vision Transformer (ViT) splits an image into 16×16 patches and treats each patch as a token. Self-attention then lets every token attend to every other token — exactly like words in a sentence. This works beautifully for image classification where the image is typically 224×224 pixels (producing only 196 tokens).

But vision is much more than classification. Object detection needs to find small objects in large images. Semantic segmentation needs to label every single pixel. These dense prediction tasks require high-resolution inputs — 800×1200 or more. At that resolution with 16×16 patches, you get 3,750 tokens. Global self-attention over 3,750 tokens means over 14 million pair-wise comparisons per layer per head. The cost grows quadratically: double the resolution, quadruple the computation.

There is a second problem. CNNs naturally produce hierarchical feature maps — early layers capture fine-grained details at high resolution, while deeper layers capture abstract semantics at lower resolution. This multi-scale pyramid is essential for frameworks like Feature Pyramid Networks (FPN) used in detection and segmentation. ViT, by contrast, produces a single-scale feature map — all tokens remain at the same resolution from start to finish. This makes it architecturally incompatible with the dense prediction pipelines that power modern computer vision.

Swin Transformer borrows the best idea from CNNs — the multi-scale feature pyramid — and implements it using Transformers. The architecture has four stages:

Stage 1: The image is split into non-overlapping 4×4 patches. Each patch is flattened into a 48-dimensional vector (4 × 4 × 3 channels) and linearly projected to dimension $C$. For a 224×224 image, this produces 56×56 = 3,136 tokens at the highest resolution.

Stage 2: A patch merging layer groups each 2×2 neighborhood of tokens, concatenates their features (producing a 4$C$-dimensional vector), and projects them down to 2$C$ dimensions. The spatial resolution halves to 28×28, while the channel depth doubles. This is analogous to a strided convolution or pooling.

Stages 3 and 4: Patch merging repeats, producing 14×14 tokens with 4$C$ channels, then 7×7 tokens with 8$C$ channels. Each stage applies multiple Swin Transformer blocks before merging.

The result is four feature maps at resolutions $\frac{H}{4} \times \frac{W}{4}$, $\frac{H}{8} \times \frac{W}{8}$, $\frac{H}{16} \times \frac{W}{16}$, and $\frac{H}{32} \times \frac{W}{32}$ — exactly what FPN-style frameworks expect. Swin can therefore serve as a drop-in replacement for ResNet and other CNN backbones in existing detection and segmentation pipelines.

The core efficiency trick in Swin is simple: instead of computing self-attention across the entire feature map, partition it into non-overlapping windows of $M \times M$ patches (typically $M = 7$) and compute attention independently within each window.

In global self-attention, the computational cost for a feature map of $h \times w$ patches with $C$ channels is:

With window-based attention (W-MSA), attention is computed within each $M \times M$ window independently. The cost becomes:

Within each window, the attention mechanism is the standard multi-head attention with one important addition — a learnable relative position bias $B$:

Window-based attention has a limitation: tokens in one window cannot see tokens in neighboring windows. The model is blind at window boundaries — like reading a book through a slot that cuts off every sentence at the same position.

Swin solves this with a beautifully simple idea: alternate between two window configurations. In layer $\ell$, use the regular window partition. In layer $\ell+1$, shift the partition by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ pixels. The shifted windows straddle the boundaries of the previous layer's windows, creating cross-window connections.

Think of it like laying tiles: in one layer you lay a 7×7 grid. In the next layer, you offset the grid by 3.5 tiles. Every tile in the shifted grid overlaps with parts of four tiles from the previous grid. Information flows across boundaries without any explicit cross-window attention — just by alternating the grid.

The two consecutive Swin Transformer blocks compute:

A naive implementation of shifted windows creates a problem: the shift produces windows of varying sizes at the borders. Some windows are smaller than $M \times M$, which breaks the uniform batch computation that GPUs rely on for speed.

The solution is elegant: instead of creating irregular border windows, cyclically shift the entire feature map toward the top-left by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ positions. This wraps the bottom and right borders around to the top and left, creating a toroidal layout where all windows are again $M \times M$. After attention, the feature map is shifted back to its original position.

But there is a catch: after the cyclic shift, some windows contain patches that are not spatially adjacent in the original image. For example, a window might contain patches from both the top-right and bottom-left corners. To prevent these non-adjacent patches from attending to each other, a masking mechanism is applied: the attention scores between non-adjacent sub-windows are set to $-\infty$ before softmax, ensuring they contribute zero weight.

The result is that the number of windows stays exactly the same as in regular partitioning, enabling efficient batched computation with no wasted operations.

Each Swin Transformer block follows the standard Transformer architecture with one key difference: the attention operates on local windows instead of the full sequence. The block has four components:

Layer Normalization (LN): Applied before both the attention and MLP sub-layers (pre-norm style). This stabilizes training and allows higher learning rates.

Window Attention (W-MSA or SW-MSA): Multi-head self-attention computed within each window of $M \times M$ patches, with relative position bias added to each head.

Residual Connection: The input to each sub-layer is added to its output, ensuring gradient flow through deep networks. This is the same skip connection used in ResNets and the original Transformer.

MLP: A two-layer feed-forward network with GELU activation, expanding the hidden dimension by a ratio of 4× before projecting back. This is where the model adds non-linear capacity — the attention is, at its core, a weighted average, and the MLP adds the representational power to transform those averaged features.

Between stages, patch merging reduces spatial resolution and increases channel depth — exactly like max pooling or strided convolution in a CNN. The process is straightforward:

Take a feature map of size $H \times W \times C$. Group every 2×2 neighborhood of tokens. Concatenate the four tokens along the channel dimension, producing $\frac{H}{2} \times \frac{W}{2} \times 4C$. Then apply a linear layer that projects from $4C$ to $2C$ dimensions.

The result: spatial resolution halves, channel depth doubles, and total tokens reduce by 4×. This is how Swin transitions between its four stages, producing the multi-scale feature pyramid that dense prediction frameworks need.

Think of it as zooming out while switching to a higher-capacity lens. You see a larger area, but you now have richer features to describe it. Each stage operates at a different resolution, and the feature maps from all four stages are available for downstream tasks — just like FPN taps into multiple layers of a ResNet.

Swin Transformer comes in four sizes, scaling the base channel dimension $C$ and the number of blocks per stage:

Swin-T (Tiny): $C = 96$, layers = [2, 2, 6, 2], ~29M parameters, 4.5 GFLOPs. Comparable in cost to ResNet-50.

Swin-S (Small): $C = 96$, layers = [2, 2, 18, 2], ~50M parameters, 8.7 GFLOPs. Comparable to ResNet-101.

Swin-B (Base): $C = 128$, layers = [2, 2, 18, 2], ~88M parameters, 15.4 GFLOPs. Comparable to ViT-B/DeiT-B.

Swin-L (Large): $C = 192$, layers = [2, 2, 18, 2], ~197M parameters, 34.5 GFLOPs. The largest variant, pre-trained on ImageNet-22K.

Notice the pattern: Stage 3 has the most blocks (6 or 18) because it operates at 14×14 resolution — small enough to be computationally reasonable but large enough to capture rich spatial patterns. This design mirrors how ResNets concentrate their depth in the third stage (res4 block).

Swin Transformer achieved state-of-the-art results across three major vision benchmarks:

Image Classification (ImageNet-1K): Swin-B achieved 83.5% top-1 accuracy without extra data. With ImageNet-22K pre-training, Swin-L reached 87.3% — surpassing all prior ViT and CNN models at comparable FLOPs.

Object Detection (COCO): Using Cascade Mask R-CNN with Swin-L backbone, the model achieved 58.7 box AP and 51.1 mask AP on COCO test-dev, surpassing the previous best by +2.7 box AP and +2.6 mask AP. This was the first time a pure Transformer backbone dominated COCO detection.

Semantic Segmentation (ADE20K): Swin-L achieved 53.5 mIoU, a +3.2 improvement over the previous state-of-the-art. The hierarchical multi-scale features proved essential for per-pixel prediction.

The key takeaway: Swin proved that a single Transformer architecture, with the right inductive biases (locality through windows, hierarchy through merging), can match and surpass the best CNNs on tasks they were specifically designed for.

Swin Transformer's deepest contribution is not the specific architecture but the design recipe it established: local attention with global communication through shifting, hierarchical feature maps through patch merging, and relative position bias instead of absolute embeddings. These principles now appear across the vision landscape — in video understanding (Video Swin), medical imaging, point cloud processing, and beyond. Even ConvNeXt, which deliberately avoided the Transformer attention mechanism, adopted Swin's overall blueprint and validated it as the right way to structure a vision backbone.