GloVe: Global Vectors for Word Representation

Before GloVe, word embedding methods split into two camps.

Count-based methods (LSA, HAL) build a giant matrix where row $i$, column $j$ holds how often word $i$ appears near word $j$ across the entire corpus. They then apply matrix factorization — typically SVD — to compress this matrix into dense vectors. These methods see the whole corpus at once, so they capture global statistics well. But the resulting vectors performed poorly on the word analogy task (king − man + woman ≈ queen) that had become the gold standard.

Prediction-based methods (Word2Vec's skip-gram and CBOW) never build an explicit matrix. Instead, they slide a small window across the text and train a shallow neural network to predict context words from a center word (or vice versa). These models excelled at analogies and captured fine-grained semantic regularities, but they only ever saw local windows — they had no way to leverage global co-occurrence patterns directly.

The question GloVe asked was: can we get the best of both worlds?

The starting point of GloVe is a simple but powerful observation. Consider two target words, "ice" and "steam", and various context words. The raw co-occurrence probabilities $P(\text{solid}|\text{ice})$ and $P(\text{solid}|\text{steam})$ are noisy — "water" co-occurs with both, "fashion" co-occurs with neither, and neither of these uninformative contexts tells you anything about the difference between ice and steam.

But the ratio $P(\text{solid}|\text{ice}) / P(\text{solid}|\text{steam})$ is large (≫ 1), because "solid" is specific to ice. Conversely, $P(\text{gas}|\text{ice}) / P(\text{gas}|\text{steam})$ is small (≪ 1), because "gas" is specific to steam. For non-discriminative words like "water" and "fashion", the ratio is close to 1.

This ratio is what encodes meaning — and GloVe's objective is designed so that vector differences capture exactly these ratios.

GloVe's derivation starts from a requirement: we want a function $F$ of word vectors that encodes the co-occurrence ratio. Specifically, for words $i$, $j$, and context word $k$:

$F(w_i - w_j, \tilde{w}_k) = \frac{P_{ik}}{P_{jk}}$

The difference $w_i - w_j$ captures the contrast between two words, and $F$ maps that contrast (combined with the context vector $\tilde{w}_k$) to the ratio of their co-occurrence probabilities.

Through a series of mathematical constraints — requiring $F$ to be a homomorphism between addition in vector space and multiplication in probability space — the authors show that the only solution is:

$w_i^T \tilde{w}_k = \log(X_{ik}) - b_i - \tilde{b}_k$

In words: the dot product of a word vector and a context vector should equal the log of their co-occurrence count, minus bias terms that absorb frequency effects.

Not all co-occurrence counts are equally informative. A word pair that co-occurs once might be noise; a pair that co-occurs a million times (like "the" + "of") dominates the loss if treated equally. GloVe addresses this with a weighting function $f(x)$ that has three properties:

• $f(0) = 0$: zero co-occurrence contributes nothing.
• $f(x)$ is non-decreasing: more frequent pairs matter more, up to a point.
• $f(x)$ saturates at a cap $x_{\max}$: beyond this threshold (typically 100), the weight stays at 1 — preventing hyper-frequent pairs from dominating.

The chosen form is a piecewise function: below $x_{\max}$ the weight grows as $(x / x_{\max})^\alpha$ with $\alpha = 0.75$, and above $x_{\max}$ the weight is clamped to 1. The sub-linear exponent $\alpha = 0.75$ means that doubling the count increases the weight by only about 68%, not 100%.

Before training can begin, GloVe needs the global co-occurrence matrix $X$. This is built by sliding a window of size $L$ (typically 10 words on each side) across the entire corpus. For each center word, every context word within the window contributes to $X_{ij}$, but with a twist: distant words within the window contribute less. Specifically, a context word at distance $d$ adds $1/d$ to the count, so the immediate neighbor adds 1, a word two positions away adds 0.5, and so on.

This distance-weighted counting is elegant: it makes the matrix capture not just whether two words co-occur, but how closely they tend to appear. The resulting matrix is symmetric ($X_{ij} = X_{ji}$) and sparse — most word pairs never co-occur, so only nonzero entries need storage and computation. Training iterates only over observed (nonzero) cells, which gives GloVe a significant speed advantage over methods that must process every cell or every token.

With the co-occurrence matrix built, training is straightforward. The parameters are: a word vector $w_i$ and bias $b_i$ for each word, plus a separate context vector $\tilde{w}_j$ and bias $\tilde{b}_j$. The model randomly samples nonzero entries from $X$ and updates the parameters by gradient descent on the weighted least-squares loss.

An interesting detail: GloVe maintains two sets of vectors — word and context. After training, both carry useful information. The final embedding is their sum: $w_i + \tilde{w}_i$. This sum consistently outperforms either set alone, because the word and context vectors encode slightly different perspectives of the same word — averaging smooths out noise and captures a richer representation.

Training complexity scales as $O(|C|^{0.8})$ where $|C|$ is the corpus size — better than the worst case and competitive with window-based methods.

The most celebrated property of word vectors is the analogy: $w_{king} - w_{man} + w_{woman} \approx w_{queen}$. GloVe explains why this happens.

Since $w_i^T \tilde{w}_k \approx \log P(k|i)$, the difference $w_{king}^T \tilde{w}_k - w_{man}^T \tilde{w}_k \approx \log\frac{P(k|\text{king})}{P(k|\text{man})}$. This ratio isolates the "royalty" component. Adding $w_{woman}$ re-introduces the "female" component. The resulting vector is closest to "queen" because queen = royalty + female.

In other words, vector arithmetic works because GloVe's objective is log-probability ratios, and log turns division into subtraction. The analogy property is not a happy accident — it is a direct consequence of the objective function.

Perhaps GloVe's deepest contribution is showing that count-based and prediction-based methods are not fundamentally different. The paper demonstrates that the skip-gram model with negative sampling (SGNS) is implicitly factorizing a word-context matrix whose entries are shifted pointwise mutual information (PMI):

$w_i^T \tilde{w}_j \approx \text{PMI}(i,j) - \log k$

where $k$ is the number of negative samples. GloVe makes this factorization explicit and replaces the noisy stochastic training of skip-gram with a clean weighted least-squares regression on the full co-occurrence matrix.

This unification was influential: it meant researchers could analyze both families with the same mathematical tools. Later work by Levy & Goldberg (2014) formalized this connection further, showing that many apparent differences between methods boil down to hyperparameter choices rather than fundamental architecture.

GloVe was evaluated on three standard benchmarks:

• Word analogy (Google dataset, 19,544 questions): GloVe achieved 75% accuracy with 300-dimensional vectors trained on 42B tokens — the best result at the time. It outperformed skip-gram, CBOW, and SVD-based methods.
• Word similarity (WordSim-353, MEN, etc.): GloVe matched or surpassed existing methods, confirming that its vectors capture semantic similarity.
• Named Entity Recognition (CoNLL-2003): Using GloVe vectors as input features for a CRF-based NER system improved F1 scores, demonstrating that the vectors carry useful information for downstream tasks beyond intrinsic benchmarks.

The authors also studied the effect of hyperparameters: larger corpora, larger vectors (up to 300 dimensions), and symmetric context windows all helped. The training data matters: Wikipedia + Gigaword outperformed either alone.

GloVe demonstrated two lasting principles. First, explicit matrix factorization and implicit neural prediction are mathematically equivalent — a unification that clarified the entire field of distributional semantics. Second, simple models with the right objective can compete with or surpass complex neural architectures, a reminder that inductive bias matters more than model complexity.

The pre-trained GloVe vectors served as the default initialization for downstream NLP models from 2014 to 2018, before contextualized embeddings (ELMo and BERT) replaced static word vectors. Even then, GloVe's intellectual DNA persists: the embedding layers in Transformers learn a similar factorization, and modern retrieval systems still use GloVe-inspired dot-product similarity.

GloVe's pre-trained vectors were the invisible backbone of NLP for half a decade. Every sentiment classifier, every question-answering system, every machine translation model between 2014 and 2018 likely started with GloVe or Word2Vec embeddings. The era of static word vectors may have passed, but the mathematical insight — that co-occurrence ratios encode meaning, and that counting and predicting are the same — remains foundational.