Graph Learning2018intermediate9 min read

Graph Attention Networks

شبكات الانتباه البيانية

Veličković, P. · Cucurull, G. · Casanova, A. · Romero, A. · Liò, P. · Bengio, Y. — ICLR

The problem

Graph Convolutional Networks (GCNs) aggregate neighbor features using fixed weights derived from the graph structure — typically the node degree. A node with five neighbors gives each one exactly 1/5 of the total , regardless of how relevant each neighbor's information actually is. Spectral methods depend on the graph Laplacian eigenbasis, tying the learned filters to one specific graph and preventing transfer to unseen graphs (inductive settings). Non-spectral methods like GraphSAGE sample fixed-size neighborhoods, losing access to the full neighborhood.

The contribution

Networks (GATs) replace the fixed neighborhood weights with a learned mechanism. Each node computes an attention coefficient for every neighbor using a shared feedforward neural network, then normalizes with to get weights that sum to 1. The key insight: the importance of a neighbor is determined by the content of both nodes' features, not the graph topology alone. (K heads concatenated or averaged) stabilizes learning and captures different relationship patterns. The method is efficient (parallelizable across edges), structure-agnostic (works on unseen graphs), and achieved state-of-the-art on Cora (83.0%), Citeseer (72.5%), and PPI (97.3% micro-F1).

The impact

GAT established attention as a first-class primitive in graph neural networks. Its design — learned, content-based neighbor weighting without eigendecompositions — became the template for a family of successors including GATv2, Graphormer, and Graph Transformers. The paper bridged Transformers and GNNs, showing that the same attention principle that revolutionized NLP could be adapted for irregular, non-grid data structures.

Imagine a neighborhood meeting. In a GCN world, every person at the table gets exactly the same speaking time, regardless of expertise. The city planner, the electrician, and the person who just moved in yesterday — all weighted equally when deciding how to fix the street.

GAT changes the rules: before the vote, each resident looks at what each neighbor knows and assigns them a personal relevance score. The electrician's opinion on wiring gets high weight; the newcomer's opinion on local history gets low weight. The decision becomes a weighted average where expertise matters.

That is exactly what a graph attention layer does: each node looks at its neighbors' features, scores their relevance, and aggregates their information proportionally.

The problem: one size does not fit all neighbors

A Graph Convolutional Network updates each node by averaging its neighbors' features with weights derived from the degree matrix. If node $i$ has neighbors $j_1, j_2, j_3$ , each gets a weight based on $1/\sqrt{d_i \cdot d_j}$ — a formula that depends only on how many connections each node has, not on what those nodes represent.

This fixed weighting has three consequences:

No selectivity. A citation network node treating a seminal paper and a tangentially related one as equally important loses discriminative power.
Spectral dependence. Methods based on the graph Laplacian learn filters tied to one specific graph's eigenstructure. Train on graph A, and the cannot transfer to graph B — a serious limitation for inductive tasks like predicting protein functions in unseen biological networks.
Sampling artifacts. GraphSAGE samples a fixed number of neighbors to keep computation constant, but this means some neighbors are randomly dropped, introducing noise.

Open in Lab

Toggle between GCN (fixed weights from degree) and GAT (learned attention weights). Notice how GAT assigns different importance to each neighbor.

The demo wakes as you arrive…

The idea: let node features decide neighbor importance

The GAT attention mechanism works in four steps. Before seeing any math, here is the mental model: imagine each node holding a business card describing its skills. When node $i$ wants to update itself, it reads its own card and each neighbor's card, computes a compatibility score for each pair, normalizes these scores, and then collects a weighted mix of neighbors' features.

Step 1 — Linear . Every node's vector $\vec{h}_i \in \mathbb{R}^F$ is projected to a new space via a shared weight matrix $\mathbf{W} \in \mathbb{R}^{F' \times F}$ , producing $\mathbf{W}\vec{h}_i \in \mathbb{R}^{F'}$ . This is the same idea as the learned projections in attention — lifting raw features into a space where comparison is meaningful.

Step 2 — Attention coefficients. A shared attention function $a$ takes the concatenation of two projected features and outputs a scalar relevance score. In GAT's implementation, $a$ is a single-layer feedforward network with a LeakyReLU activation.

Step 3 — Masked softmax. The scores are normalized across each node's neighborhood using softmax, ensuring they sum to 1. Crucially, attention is only computed between connected nodes (masked attention) — node $i$ only scores neighbors $j \in \mathcal{N}_i$ , not the entire graph.

Step 4 — Weighted aggregation. The normalized attention weights multiply each neighbor's projected features, and the results are summed to produce the node's new .

Open in Lab

Click each step to see how node 1 computes its new features from its neighbors.

The demo wakes as you arrive…

The math: from features to attention-weighted output

Now that the intuition is clear, here are the formulas. Each one maps directly to a step you already understand.

e_{ij} = \text{LeakyReLU}\!\left(\vec{a}^{\,T} [\mathbf{W}\vec{h}_i \| \mathbf{W}\vec{h}_j]\right)

Raw attention coefficient — how relevant is neighbor j to node i? — Concatenate the projected features of node i and neighbor j (the ‖ symbol), pass through a learned weight vector

\vec{a}

and LeakyReLU. The result is a scalar score before normalization.

\alpha_{ij} = \text{softmax}_j(e_{ij}) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_i} \exp(e_{ik})}

Normalized attention weight — a probability over the neighborhood — Softmax across all neighbors of node i turns raw scores into weights that sum to 1. High

\alpha_{ij}

means node i considers neighbor j highly relevant.

\vec{h}'_i = \sigma\!\left(\sum_{j \in \mathcal{N}_i} \alpha_{ij} \, \mathbf{W}\vec{h}_j\right)

Node update — attention-weighted aggregation — The new representation of node i is a nonlinear activation σ applied to the weighted sum of projected neighbor features. Each neighbor contributes proportionally to its attention weight.

Open in Lab

Hover over a node to see its attention weights toward each neighbor. Thicker edges mean higher attention.

The demo wakes as you arrive…

Stabilizing learning: multi-head attention on graphs

A single attention head might latch onto one relationship pattern and miss others. The solution — borrowed directly from the Transformer — is multi-head attention. GAT runs $K$ independent attention mechanisms in parallel, each with its own weight matrix $\mathbf{W}^k$ and attention vector $\vec{a}^k$ . In the hidden layers, the $K$ outputs are concatenated:

\vec{h}'_i = \overset{K}{\underset{k=1}{\Big\|}} \, \sigma\!\left(\sum_{j \in \mathcal{N}_i} \alpha_{ij}^k \, \mathbf{W}^k \vec{h}_j\right)

Multi-head attention (hidden layers) — concatenation — K independent attention heads, each producing F' features, are concatenated into a KF'-dimensional output. Each head learns a different aspect of the node relationships.

At the final (prediction) layer, concatenation would multiply the output dimension by $K$ , so instead the heads are averaged before the final activation:

\vec{h}'_i = \sigma\!\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{j \in \mathcal{N}_i} \alpha_{ij}^k \, \mathbf{W}^k \vec{h}_j\right)

Multi-head attention (output layer) — averaging — At the output layer, K attention heads are averaged instead of concatenated, keeping the output dimension equal to the number of classes.

Open in Lab

Each color is a different attention head. Watch how different heads attend to different neighbors, then get concatenated or averaged.

The demo wakes as you arrive…

Putting it together: the GAT architecture

The full GAT architecture for a transductive task like stacks two graph attention layers. The first layer uses $K = 8$ attention heads, each computing $F' = 8$ features (total 64 features after concatenation), with an ELU nonlinearity. The second layer uses a single attention head computing $C$ features (one per class), followed by softmax for classification.

is critical given the small training sets (only 20 labeled nodes per class in Cora): L2 regularization ( $\lambda = 0.0005$ ) and ( $p = 0.6$ ) on both the input features and the attention coefficients. Dropping attention coefficients means each training step exposes a node to a stochastically sampled subset of its neighborhood — a form of on graphs.

For the inductive PPI task, GAT uses three layers with $K = 4$ heads (256 features each) and skip connections. The final layer averages $K = 6$ heads for multi-label classification with sigmoid activation.

Open in Lab

Click any layer to see its configuration and role in the architecture.

The demo wakes as you arrive…

The same idea in code

A single graph attention head, completepython

Simplified to show the idea — not the real implementation.

import numpy as np

def leaky_relu(x, alpha=0.2):
    return np.where(x > 0, x, alpha * x)

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

def gat_head(h, W, a, adj):
    """
    h:   (N, F)   — node feature matrix
    W:   (F', F)  — shared linear projection
    a:   (2F',)   — attention weight vector
    adj: (N, N)   — adjacency matrix (1 = connected)
    """
    N = h.shape[0]
    Wh = h @ W.T                       # (N, F') — project all nodes
    F_prime = Wh.shape[1]
    h_new = np.zeros_like(Wh)

    for i in range(N):
        neighbors = np.where(adj[i] > 0)[0]  # includes self-loop
        scores = []
        for j in neighbors:
            concat = np.concatenate([Wh[i], Wh[j]])   # (2F',)
            e_ij = leaky_relu(a @ concat)              # scalar
            scores.append(e_ij)
        alpha = softmax(np.array(scores))    # attention weights sum to 1

        # Weighted aggregation
        h_new[i] = sum(alpha[k] * Wh[neighbors[k]]
                       for k in range(len(neighbors)))
    return h_new   # (N, F')

Why GAT matters: three key properties

Results: attention outperforms fixed weighting

GAT was evaluated on four benchmarks — three transductive citation networks (Cora, Citeseer, Pubmed) and one inductive protein-protein interaction dataset (PPI).

On the transductive tasks, GAT improved over GCN by 1.5% on Cora and 1.6% on Citeseer. The PPI results were even more dramatic: GAT achieved 97.3% micro-F1, improving 20.5% over the best GraphSAGE variant, directly demonstrating the benefit of observing the full neighborhood and weighting it adaptively.

A controlled experiment comparing GAT to "Const-GAT" (same architecture, but with constant attention $a(x,y) = 1$ , equivalent to GCN-style uniform aggregation) showed a 3.9% improvement on PPI, isolating the contribution of the attention mechanism itself.

Open in Lab

Compare GAT, GCN, and GraphSAGE across all four benchmarks.

The demo wakes as you arrive…

Connection: GAT attention vs Transformer attention

The Transformer can be seen as a special case of GAT operating on a fully connected graph where every is connected to every other token. GAT's key differences:

Sparse attention. GAT only computes attention between connected nodes, not all pairs. This makes it $O(|E|)$ instead of $O(N^2)$ .
Additive vs dot-product. Transformers use scaled dot-product attention ( $Q K^T / \sqrt{d_k}$ ); GAT uses an additive mechanism (concatenate + linear layer). The additive form is more expressive per pair but lacks the matrix-multiplication efficiency of dot-product attention.
No needed. Graph structure provides the "positions" — the tells the model who can attend to whom.

What came after

2018
GAT — this paper
First to apply masked self-attention to graph neural networks, enabling learned, content-based neighbor weighting that is inductive and parallelizable.
2021
GATv2
Brody et al. showed that GAT's original attention is "static" — the ranking of neighbors is the same regardless of the query node. GATv2 fixes this with a modified attention order that is truly dynamic.
2021
Graphormer
Combined Transformer architecture with graph structural encodings (degree, shortest path, edge features). Won the OGB-LSC quantum chemistry challenge, showing that full-graph attention with structural priors can scale.
2023
Graph Transformers at scale
Models like GPS (General Powerful Scalable) and Exphormer combined local message passing (GAT-style) with global attention (Transformer-style), handling graphs with millions of nodes.

CitationVeličković, Cucurull, Casanova, Romero, Liò, Bengio. Graph Attention Networks. ICLR, 2018.

Terms in this paper

Graph Neural Network (GNN)الشبكات العصبية الرسومية (البيانية)
Graph Convolutionالالتفاف الرسومي
Attentionآلية الانتباه
Message Passingتمرير الرسائل
Node Classificationتصنيف العُقد
Multi-Head Attentionالانتباه المتعدد المسارات
Adjacency Matrixمصفوفة التجاور
Softmaxسوفت ماكس
Transductive Learningالتعلم التبادلي
Graph Attentionانتباه الرسوم البيانية