Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Before this paper, image captioning followed a simple two-stage pipeline:

1. A convolutional neural network (typically VGGNet or GoogLeNet) processes the image and outputs a single feature vector — a fixed-length numerical summary of the entire scene.

2. An LSTM decoder receives that vector once, at the first time step, then generates the caption word by word purely from its hidden state.

The problem is a bottleneck: thousands of pixels, dozens of objects, and all their spatial relationships must compress into one vector of 512 or 4096 numbers. Early words get the freshest signal; later words rely on the LSTM's fading memory of that initial snapshot. As scenes grow more complex, the descriptions grow more generic — the model says "a man on a field" instead of "a football player catching a ball near the goal post".

The key insight is borrowed from Bahdanau's attention for machine translation: instead of encoding the image into a single vector, keep a grid of feature vectors and let the decoder choose which grid cells to focus on at each time step. The architecture has three parts:

Encoder (CNN): Extract features not from the final fully connected layer, but from a lower convolutional layer — specifically the fourth convolutional layer of VGGNet. This produces $L = 14 \times 14 = 196$ annotation vectors, each of dimension $D = 512$. Each vector encodes information about a particular spatial region of the image. Think of it as a 14×14 grid overlay on the image, where each cell contains a rich description of what's in that cell.

Attention mechanism: At each time step, a small neural network scores how relevant each of the 196 locations is, given the decoder's current state. The scores become weights (via softmax), and the weighted sum of annotation vectors produces a context vector — a dynamic summary that highlights the part of the image relevant to the current word.

Decoder (LSTM): Generates one word at a time, conditioned on the previous word, the previous hidden state, and the context vector from the attention mechanism. This means the decoder's input changes at every time step — it sees a different "view" of the image depending on what word it's about to generate.

At each time step $t$, the model must decide which of the $L = 196$ image regions to attend to. It does this in three stages:

Stage 1 — Energy score: A small alignment network takes each annotation vector $a_i$ and the previous hidden state $h_{t-1}$, and produces a scalar energy $e_{ti}$ measuring how relevant region $i$ is to the current decoding step. This is additive attention — the same mechanism Bahdanau used for translation.

Stage 2 — Attention weights: The energies are normalized via softmax so they sum to 1: $\alpha_{ti}$ is the probability that location $i$ is the right place to focus on when generating word $t$.

Stage 3 — Context vector: The context vector $\hat{z}_t$ is a weighted combination of annotation vectors. This single vector — a dynamic, word-specific summary of the image — is fed to the LSTM along with the previous word embedding.

The paper proposes two variants of the attention mechanism, each with different tradeoffs:

Soft (deterministic) attention takes the weighted average of all annotation vectors. Every region contributes, but regions with higher $\alpha$ weights contribute more. This is fully differentiable — you can train it end-to-end with standard backpropagation. Think of it as looking at the whole painting through a filter that makes relevant areas brighter and irrelevant areas dimmer.

Hard (stochastic) attention samples one region according to the $\alpha$ distribution. At each time step, the model picks exactly one grid location $s_t$ and uses only $a_{s_t}$ as the context. This is like putting a spotlight on one part of the painting and ignoring everything else. Because sampling is not differentiable, hard attention must be trained with the REINFORCE algorithm — a policy gradient method from reinforcement learning that treats the attention decisions as a sequence of actions.

The soft variant is simpler to train and more stable; the hard variant can sometimes achieve slightly higher performance because it forces sharper focus, but at the cost of higher variance during training.

By construction, the attention weights $\alpha_{ti}$ at each time step sum to 1 over locations (each word's focus is a valid probability distribution). But nothing prevents the model from always attending to the same few regions and ignoring the rest of the image.

To fix this, the authors introduce doubly stochastic regularization: in addition to the row constraint ($\sum_i \alpha_{ti} = 1$), they add a penalty encouraging the column sums to also be approximately 1 ($\sum_t \alpha_{ti} \approx 1$). This means every region of the image should be attended to roughly equally over the full caption — no region should be completely ignored.

Intuitively, think of a conference where every speaker (word) must pick audience members (regions) to look at. The first constraint says each speaker's attention must be a valid distribution. The second says no audience member should be totally ignored over the whole conference — everyone gets looked at by someone.

Not every word in a caption needs visual information. Function words like "a", "the", "of" are determined by grammar, not by the image. The paper introduces a learned gating scalar $\beta_t = \sigma(f_\beta(h_{t-1}))$ that modulates the context vector: $\hat{z}_t = \beta_t \sum_i \alpha_{ti} a_i$. When $\beta$ is close to 1, the LSTM relies heavily on the image; when close to 0, it relies on language context alone.

This is a simple but powerful idea: the model learns when to look at the image and when to rely on language patterns. When generating "a", $\beta$ drops — there's nothing to look at. When generating "frisbee", $\beta$ rises — the model needs to see the object.

Soft attention training is straightforward: minimize the negative log-likelihood of the correct caption words, plus the doubly stochastic penalty. Since soft attention is a differentiable weighted average, gradients flow through the attention weights normally.

Hard attention training is more complex. Because sampling a single region is non-differentiable, the authors use the REINFORCE algorithm. The objective becomes a variational lower bound on the log-likelihood, and the gradient is estimated using Monte Carlo sampling. To reduce variance, they use a moving average baseline and multiply the REINFORCE gradient with the reward, then add an entropy term to encourage exploration.

Both models use the same CNN encoder (VGGNet pre-trained on ImageNet), and the decoder is an LSTM with 512 hidden units. The initial hidden and cell states of the LSTM are computed from the mean of the annotation vectors through learned linear projections.

The model was evaluated on three standard image captioning benchmarks: Flickr8k, Flickr30k, and MS COCO, using BLEU and METEOR metrics. Both soft and hard attention models outperformed the baseline Show and Tell model that used no attention, validating that attention provides a genuine improvement rather than just added complexity.

But the most striking result wasn't in the numbers — it was in the attention maps. By visualizing $\alpha$ weights at each time step, the authors showed that the model genuinely learns to fixate on the correct objects: when generating "bird", the attention highlights the bird; when generating "water", it shifts to the water. This provided the first compelling evidence that attention in neural networks corresponds to something humanly interpretable.

This paper occupies a pivotal position in the history of visual attention. It took an idea proven in NLP (Bahdanau attention for translation) and showed it works across modalities — from sequences of words to grids of image patches. It demonstrated that attention can serve as an interpretability tool, not just a performance booster. And it established the encoder-decoder-with-attention template that would dominate image captioning, VQA, and eventually multi-modal models.