Core ML2003intermediate12 min read

Latent Dirichlet Allocation

تخصيص ديريكليه الكامن

Blei, D. M. · Ng, A. Y. · Jordan, M. I. — JMLR

The problem

By the early 2000s, text collections were exploding in size — digital libraries, the web, email archives — but the tools to organize them were shallow. Bag-of-words models like TF-IDF treated every word independently, missing the fact that "bank", "loan", and "interest" co-occur because they share a hidden theme. Probabilistic Latent Semantic Analysis (pLSA) could discover topics, but it had no proper for documents, suffered from on small corpora, and could not assign probabilities to new, unseen documents.

The contribution

LDA: a fully generative Bayesian for text. Each draws a mixture of topics from a Dirichlet , each word draws its topic from that mixture, and each word is generated from the topic's . Because everything flows from priors, LDA generalizes to new documents, resists overfitting, and provides a principled for any document. is intractable, so the authors introduce a variational EM algorithm that scales to large corpora. LDA outperformed pLSA and TF-IDF on document classification and information retrieval benchmarks.

The impact

LDA became the foundational and one of the most cited machine learning papers ever. It established the paradigm of Bayesian generative modeling for discrete data and inspired hundreds of extensions — dynamic topic models, correlated topics, supervised LDA, author-topic models. Beyond NLP, LDA was adopted in bioinformatics, computer vision, and recommendation systems. Its framework influenced an entire generation of approximate Bayesian methods, eventually leading to variational autoencoders and modern amortized inference.

Imagine a kitchen where every dish is made by combining a few base recipes — Thai curry, Italian ragu, French roux — in different proportions. A pad Thai is mostly Thai curry with a dash of ragu for depth. A fusion risotto might be equal parts ragu and roux.

Now imagine you only see the finished dishes — you never see the recipes. LDA is the food critic who tastes enough dishes to reverse-engineer the hidden recipes and figure out each dish's blend.

The "dishes" are documents, the "base recipes" are topics, and the "ingredients" are words. LDA discovers the recipes nobody wrote down.

The problem: flat word counts miss hidden structure

The simplest way to represent a document is as a — count how many times each word appears, throw away order. TF-IDF improved on raw counts by down-weighting common words, but it still treats every word as independent: it doesn't know that "neuron", "synapse", and "cortex" cluster together because they share an underlying theme.

Word2Vec would later learn word similarities, but at a single-word level. What was missing was a document-level model that could say: "this paper is 60% neuroscience and 40% statistics" — capturing the mixture of themes that makes each document unique.

Probabilistic Latent Semantic Analysis (pLSA) by Hofmann (1999) took a step in this direction by modeling documents as mixtures of topics. But pLSA had a fatal flaw: the topic proportions were free parameters for each document, so the number of parameters grew linearly with the size. This made pLSA prone to overfitting, and there was no principled way to assign a probability to a document the model had never seen.

Open in Lab

Left: Bag-of-words sees each word independently. Right: LDA discovers that words cluster into hidden topics.

The demo wakes as you arrive…

The engine: the Dirichlet distribution

Before we see LDA's generative story, we need to meet its engine: the . A Dirichlet distribution is a distribution over probability distributions. Think of it as a machine that produces pie charts: you give it a $\alpha$ , and it hands you back a random probability vector whose entries sum to 1.

The parameter $\alpha$ controls the shape of the pie charts the machine produces. When all entries of $\alpha$ are large (say, 10), most pie charts come out roughly equal — every slice gets a fair share. When all entries are small (say, 0.1), most pie charts are sparse — one or two slices dominate and the rest are near zero. When entries differ, the machine favors some slices over others.

In LDA, the Dirichlet serves two roles. It produces a topic-mixture vector $\theta$ for each document (how much of each topic this document uses), and a word-distribution vector $\phi$ for each topic (how likely each vocabulary word is under this topic).

Open in Lab

Drag α to see how the Dirichlet distribution changes. Low α → sparse (few topics dominate). High α → uniform (topics share equally).

The demo wakes as you arrive…

The generative story: how LDA imagines a document was written

LDA tells a fictional story of how each document in a corpus came to exist. Nobody believes documents are actually written this way — the story is a modeling assumption that captures the key statistical pattern: documents exhibit multiple topics, and words are drawn from those topics.

Here is the full generative process. For each document $d$ in the corpus:

Step 1. Draw a topic-mixture vector $\theta_d \sim \text{Dir}(\alpha)$ . This is a point on the probability simplex — it says how much of each topic this document will use.

Step 2. For each word position $n$ in the document:

2a. Draw a topic assignment $z_{d,n} \sim \text{Multinomial}(\theta_d)$ . This picks which topic generates this particular word.

2b. Draw the word $w_{d,n} \sim \text{Multinomial}(\phi_{z_{d,n}})$ . This picks a word from the vocabulary distribution of the chosen topic.

The topic-word distributions $\phi_k$ are themselves drawn from a Dirichlet prior: $\phi_k \sim \text{Dir}(\beta)$ for each topic $k$ .

Notice what is observed and what is hidden. We see the words $w$ . Everything else — the topic mixtures $\theta$ , the topic assignments $z$ , and the topic-word distributions $\phi$ — is latent. The entire challenge of LDA is to infer these hidden variables from the observed words.

Open in Lab

Click "Generate" to watch LDA write a document step by step: draw θ, then for each word pick a topic and sample a word from it.

The demo wakes as you arrive…

The graphical model: reading the plate diagram

Bayesian models are often drawn as graphical models — diagrams where nodes are random variables and edges show dependencies. LDA's plate diagram is compact but encodes the entire generative story:

The outer plate (rectangle) is replicated $D$ times — once per document.
Inside it, the inner plate is replicated $N_d$ times — once per word in document $d$ .
$\alpha$ sits outside all plates: it is a corpus-wide shared by every document.
$\theta_d$ sits inside the document plate: each document has its own topic mixture.
$z_{d,n}$ and $w_{d,n}$ sit inside the word plate: each word has its own topic assignment.
$\phi_k$ and $\beta$ sit in a separate plate replicated $K$ times — once per topic.
The only shaded (observed) node is $w_{d,n}$ . Everything else is latent.

This compact picture fully specifies the joint distribution. The challenge is computing the posterior of all the latent variables given the observed words.

Open in Lab

Hover over any node to see its role and distribution. Shaded = observed, white = latent.

The demo wakes as you arrive…

The math: from story to equations

The generative story translates directly into a joint probability. For a single document with $N$ words, the joint distribution over the topic mixture $\theta$ , topic assignments $\mathbf{z}$ , and words $\mathbf{w}$ is:

p(\theta, \mathbf{z}, \mathbf{w} \mid \alpha, \beta) = p(\theta \mid \alpha) \prod_{n=1}^{N} p(z_n \mid \theta) \, p(w_n \mid z_n, \beta)

Joint distribution for a single document — p(θ|α) is the Dirichlet prior on topic mixtures · p(zₙ|θ) picks a topic for each word · p(wₙ|zₙ,β) picks a word from the chosen topic's vocabulary

To get the probability of the observed words alone, we need to integrate out $\theta$ and sum over all possible topic assignments $\mathbf{z}$ . This gives the marginal of a document:

p(\mathbf{w} \mid \alpha, \beta) = \int p(\theta \mid \alpha) \left( \prod_{n=1}^{N} \sum_{z_n} p(z_n \mid \theta) \, p(w_n \mid z_n, \beta) \right) d\theta

Marginal likelihood — the intractable integral — This integral has no closed form because of the coupling between θ and β inside the product. This intractability is what motivates approximate inference.

Variational inference: the tractable shortcut

Since exact inference is impossible, Blei, Ng, and Jordan proposed a variational approximation. The idea: choose a simpler family of distributions $q$ that factorizes (breaks the couplings), and find the member of that family closest to the true posterior.

The variational family for LDA assumes that the topic mixture $\theta$ and each topic assignment $z_n$ are independent — a "mean-field" assumption. This breaks the coupling that made exact inference intractable:

q(\theta, \mathbf{z} \mid \gamma, \boldsymbol{\varphi}) = q(\theta \mid \gamma) \prod_{n=1}^{N} q(z_n \mid \varphi_n)

Mean-field variational family — q(θ|γ) is a Dirichlet with free parameter γ · each q(zₙ|φₙ) is a Multinomial with free parameter φₙ · the key: θ and z are assumed independent, making the math tractable.

"Closest" is measured by between $q$ and the true posterior $p$ . Minimizing this KL divergence is equivalent to maximizing a quantity called the Evidence Lower Bound ():

\mathcal{L}(\gamma, \varphi; \alpha, \beta) = \mathbb{E}_q[\log p(\theta, \mathbf{z}, \mathbf{w} \mid \alpha, \beta)] - \mathbb{E}_q[\log q(\theta, \mathbf{z})]

Evidence Lower Bound (ELBO) — First term: expected log-joint under q (how well q explains the data). Second term: entropy of q (how spread out q is). Maximizing ELBO tightens the bound on log p(w).

The ELBO is maximized by coordinate ascent: iteratively update $\gamma$ and each $\varphi_n$ while holding the other fixed. The update equations have elegant closed forms thanks to the Dirichlet-Multinomial conjugacy:

\varphi_{n,k} \propto \beta_{k, w_n} \exp\!\bigl(\psi(\gamma_k) - \psi(\textstyle\sum_j \gamma_j)\bigr)

Variational update for topic assignment φ — Each word's soft topic assignment balances two forces: how well the topic explains this word (β term) and how much the document uses this topic (ψ/digamma terms from γ).

\gamma_k = \alpha_k + \sum_{n=1}^{N} \varphi_{n,k}

Variational update for topic mixture γ — The document's approximate topic mixture is the prior α plus the soft counts of how many words were assigned to each topic.

Open in Lab

Watch the E-step and M-step alternate. The E-step infers topic assignments for each document; the M-step updates the global topic-word distributions.

The demo wakes as you arrive…

The alternative: collapsed Gibbs sampling

Two years after LDA was published, Griffiths and Steyvers (2004) proposed a simpler inference method: . Instead of approximating the full posterior, you integrate out $\theta$ and $\phi$ analytically (thanks to Dirichlet-Multinomial conjugacy) and sample only the topic assignments $z$ .

The collapsed sampler works word by word: for each word $w_{d,n}$ , remove its current topic assignment, then re-sample a topic proportional to two counts: (1) how often topic $k$ appears in document $d$ (minus this word), and (2) how often word $w$ appears under topic $k$ (minus this word). After enough iterations, the samples converge to the posterior.

Collapsed became the most popular LDA inference method in practice because it is simple to implement, gives good results, and the sampling equation is intuitive:

p(z_{d,n} = k \mid \mathbf{z}_{-(d,n)}, \mathbf{w}, \alpha, \beta) \propto (n_{d,k}^{-(d,n)} + \alpha_k) \cdot \frac{n_{k,w}^{-(d,n)} + \beta_w}{\sum_{v} n_{k,v}^{-(d,n)} + \beta_v}

Collapsed Gibbs sampling equation — First factor: how much document d already uses topic k (document-topic affinity). Second factor: how well topic k explains word w (topic-word affinity). The word gravitates toward the topic that fits both the document and the word.

LDA vs pLSA: why the Bayesian layer matters

pLSA and LDA look similar on the surface — both model documents as mixtures of topics. The critical difference is the Dirichlet prior. In pLSA, each document's topic mixture $\theta_d$ is a free parameter estimated directly. In LDA, $\theta_d$ is a random variable drawn from a prior. This has three consequences:

. LDA can assign a probability to any new document by integrating over $\theta$ . pLSA cannot — it only knows the documents it trained on.
Overfitting control. The Dirichlet prior acts as a regularizer. With small $\alpha$ , the model prefers sparse topic mixtures, which is realistic — most documents cover only a few topics, not all of them.
Parameter count. LDA's model complexity is fixed (determined by $K$ and $V$ , not $D$ ). pLSA grows linearly with the number of documents, eventually overfitting.

Open in Lab

Compare LDA and pLSA on a small corpus. Notice how pLSA overfits when the corpus is small, while LDA's Dirichlet prior keeps topics clean.

The demo wakes as you arrive…

The same idea in code

LDA collapsed Gibbs sampler, completepython

Simplified to show the idea — not the real implementation.

import numpy as np

def lda_gibbs(docs, V, K, alpha, beta, n_iter=1000):
    """Collapsed Gibbs sampler for LDA.
    docs: list of lists of word indices
    V: vocabulary size, K: number of topics
    alpha: Dirichlet prior on doc-topic, beta: prior on topic-word
    """
    # Initialize: random topic assignment per word
    z = [[np.random.randint(K) for _ in doc] for doc in docs]

    # Count matrices
    n_dk = np.zeros((len(docs), K))   # doc d, topic k count
    n_kv = np.zeros((K, V))           # topic k, word v count
    n_k  = np.zeros(K)                # total words in topic k

    for d, doc in enumerate(docs):
        for n, w in enumerate(doc):
            k = z[d][n]
            n_dk[d, k] += 1
            n_kv[k, w] += 1
            n_k[k]     += 1

    # Gibbs iterations
    for it in range(n_iter):
        for d, doc in enumerate(docs):
            for n, w in enumerate(doc):
                k_old = z[d][n]
                # Remove current assignment
                n_dk[d, k_old] -= 1
                n_kv[k_old, w] -= 1
                n_k[k_old]     -= 1

                # Compute conditional: doc-topic × topic-word
                prob = (n_dk[d] + alpha) * (n_kv[:, w] + beta) / (n_k + V * beta)
                prob /= prob.sum()

                # Re-sample topic
                k_new = np.random.choice(K, p=prob)
                z[d][n] = k_new
                n_dk[d, k_new] += 1
                n_kv[k_new, w] += 1
                n_k[k_new]     += 1

    # Recover topic-word distributions
    phi = (n_kv + beta) / (n_k[:, None] + V * beta)
    # Recover doc-topic distributions
    theta = (n_dk + alpha) / (n_dk.sum(axis=1, keepdims=True) + K * alpha)
    return phi, theta

Impact: what LDA started

2003
LDA published
Blei, Ng, and Jordan introduce Latent Dirichlet Allocation in JMLR. First fully generative Bayesian topic model with variational inference.
2004
Collapsed Gibbs sampling for LDA
Griffiths and Steyvers propose collapsed Gibbs sampling — simpler, widely adopted. Published in PNAS under "Finding Scientific Topics."
2006
Dynamic & Correlated Topic Models
Blei and Lafferty extend LDA to model topic evolution over time and correlations between topics, spawning a rich family of extensions.
2010
Online LDA
Hoffman, Blei, and Bach introduce stochastic variational inference for LDA, scaling to millions of documents by processing mini-batches.
2013
Word2Vec — a different path
Mikolov's Word2Vec learns word embeddings via prediction rather than generative modeling. The two paradigms — topic models and embeddings — would eventually converge in neural topic models.
2014
LDA in bioinformatics
Researchers apply LDA to gene expression data, treating genes as "words" and cell samples as "documents." LDA discovers cell-type signatures automatically.
2017
Neural Topic Models (ProdLDA)
Srivastava and Sutton replace variational inference with a VAE-style encoder, bringing LDA into the deep learning era.

LDA's influence extends far beyond topic modeling. Its variational inference framework became a template for approximate Bayesian methods. The idea that documents have mixed memberships inspired mixed-membership models in genetics, social network analysis, and recommendation systems. And its core insight — that observed patterns arise from latent categorical choices governed by continuous priors — is the same principle behind variational autoencoders and modern generative models.

CitationBlei, Ng, Jordan. Latent Dirichlet Allocation. JMLR, 2003.

Terms in this paper

Topic Modelنموذج المواضيع
Dirichlet Distributionتوزيع ديريكليه
Latent Variableالمتغير الكامن
Variational Inferenceالاستدلال الاحتمالي المتغيِّر
Generative Modelالنموذج التوليدي
Bag of Wordsحقيبة الكلمات
Documentالمستند الرقمي
Corpusالمدونة النصية
Posteriorالاحتمال البعدي الـمُحدث
Priorالاحتمال القبلي المبدئي
Likelihoodالأرجحية
ELBOالحد الأدنى الاختلافي
Maximum Likelihood Estimationتقدير الأرجحية القصوى
KL Divergenceتباعد KL
Gibbs Samplingعيّنة غيبس