Azimuth

Language Models2019advanced12 min read

XLNet: Generalized Autoregressive Pretraining for Language Understanding

XLNet: تدريب مسبق ذاتي الانحدار مُعمَّم لفهم اللغة

Yang, Z. · Dai, Z. · Yang, Y. · Carbonell, J. · Salakhutdinov, R. · Le, Q. V. — NeurIPS

The problem

By 2019, pre-trained language models had split into two camps with complementary strengths and weaknesses. models like GPT decompose a sentence's probability left-to-right, producing clean likelihood estimates — but they only see context in one direction. Autoencoding models like BERT see context in both directions by masking random tokens — but they introduce an artificial mismatch: [MASK] tokens appear during but never during . Worse, BERT assumes the masked tokens are independent of each other, ignoring dependencies among them. No existing model captured context without these compromises.

The contribution

XLNet: a generalized autoregressive method that achieves bidirectional context without masking. The key idea is — instead of predicting tokens left-to-right, XLNet maximizes the expected log-likelihood over all possible factorization orders of the sequence. This lets every see context from both directions while preserving the autoregressive product rule, avoiding the independence assumption and of BERT. XLNet introduces ( + ) to make this work and integrates -XL's for long-range dependency. It outperformed BERT on 20 NLP tasks, achieving state-of-the-art on 18 including SQuAD, GLUE, RACE, and document ranking.

The impact

XLNet demonstrated that the autoregressive framework could match and exceed BERT-style bidirectional models when combined with permutation-based training. It validated Transformer-XL's recurrence mechanism as essential for long-context language modeling. While later models like RoBERTa showed that careful BERT training could close part of the gap, XLNet's permutation language modeling influenced subsequent work including UniLM and MPNet. Its two-stream mechanism became a reference architecture for tasks requiring position-aware prediction without information leakage.

Imagine you're assembling a jigsaw puzzle, but instead of following the picture on the box from left to right, you pick pieces in a random order each time. Sometimes you start with the sky, sometimes with the grass, sometimes with the house in the middle. After assembling the puzzle in hundreds of different orders, you understand every piece's relationship to every other piece — not just its left neighbor.

That's what XLNet does with sentences. BERT solves a different puzzle: it blacks out pieces and guesses what's missing, but it never learns the order in which pieces connect. GPT always goes left-to-right, seeing only what came before. XLNet shuffles the assembly order — keeping the autoregressive "one piece at a time" discipline — but eventually learns every connection from every direction.

The problem: two camps, two compromises

By 2019, language model pretraining had split into two paradigms, each with a fundamental limitation.

Autoregressive (AR) models like GPT factor the joint probability of a sequence as a product of conditional probabilities, always left-to-right: P(x₁) · P(x₂|x₁) · P(x₃|x₁,x₂) · ... This is clean and mathematically principled — each token is predicted from a proper probability distribution. But the model only sees leftward context. For a task like question answering where the answer depends on words both before and after a position, this one-way view is a handicap.

Autoencoding (AE) models like BERT solve this by masking random tokens and predicting them from the full surrounding context — both left and right. But this creates two problems. First, the pretrain-finetune discrepancy: during pretraining BERT sees artificial [MASK] tokens, but during fine-tuning those tokens never appear — a distribution mismatch. Second, the independence assumption: BERT predicts all masked tokens simultaneously, assuming they are conditionally independent of each other given the unmasked tokens. If "New" and "York" are both masked, BERT predicts each one separately without modeling their joint dependency.

Open in Lab
Compare how GPT (autoregressive) and BERT (autoencoding) process the same sentence — and see the compromises each makes.
The demo wakes as you arrive…

The key idea: permutation language modeling

XLNet's insight is elegant: instead of choosing between left-to-right factorization (GPT) and masking (BERT), factorize the sequence probability in every possible order.

Consider a sentence with four tokens: [x₁, x₂, x₃, x₄]. A standard always factorizes as: P(x₁) · P(x₂|x₁) · P(x₃|x₁,x₂) · P(x₄|x₁,x₂,x₃).

But this is just one of 4! = 24 possible orderings. What if we also tried the order [x₃, x₁, x₄, x₂]? Then we'd compute: P(x₃) · P(x₁|x₃) · P(x₄|x₃,x₁) · P(x₂|x₃,x₁,x₄).

In this order, predicting x₂ uses x₃, x₁, and x₄ as context — meaning x₂ benefits from both left and right context, even though we used a strictly autoregressive factorization. By training over all permutations (in expectation), every token eventually sees every other token as context. The result: bidirectional understanding with a clean autoregressive objective — no [MASK] tokens, no independence assumption.

Crucially, XLNet does not actually permute the input sequence. The tokens stay in their original positions. Instead, XLNet uses attention masks to control which tokens can attend to which — effectively simulating the permuted . This preserves the natural while varying the prediction order.

Open in Lab
Click "Shuffle" to see different factorization orders. Notice how each order gives different tokens access to different contexts.
The demo wakes as you arrive…
max⁡θ    Ez∼ZT[∑t=1Tlog⁡ pθ ⁣(xzt∣xz<t)]\max_\theta \;\; \mathbb{E}_{\mathbf{z} \sim \mathcal{Z}_T} \left[ \sum_{t=1}^{T} \log \, p_\theta\!\left(x_{z_t} \mid \mathbf{x}_{\mathbf{z}_{<t}}\right) \right]
Permutation Language Modeling Objective — For a sequence of length T, sample a permutation z from all T! possible orderings. For each position z_t in the permutation, predict token x_{z_t} using only the tokens that precede it in this permuted order (x_{z_{<t}}). Maximize the expected log-likelihood over all permutations. This single objective captures bidirectional dependencies through the autoregressive product rule.

The mechanism: two-stream self-attention

Permutation language modeling creates a subtle technical challenge. When predicting a token at position z_t, the model needs to know where the token is (its position) but must not see what the token is (its content) — otherwise the prediction is trivial. Yet for tokens that are part of the context (positions before z_t in the permutation), the model needs both their positions and their content.

Think of it like a conference table. Each participant (token) holds a name tag (position) and a folder of information (content). When it's your turn to present, the chairperson collects your name tag but takes away your folder — they know where you sit but not what you know. Everyone else at the table keeps both their name tag and their folder visible. This asymmetry is exactly what XLNet needs.

The standard Transformer can't do this — it either sees a token completely or not at all. XLNet solves this with two parallel attention streams:

The content stream works exactly like standard . It has access to both the position and the content of all tokens including itself. This builds the contextual representation.

The query stream can see the positions of all tokens, but can only see the content of tokens that precede it in the current permutation order. It cannot see its own content. This stream is what actually makes the prediction.

Open in Lab
Toggle between content stream and query stream to see what each token can access. The query stream hides the target token's content.
The demo wakes as you arrive…
gzt(m)←Attention ⁣(Q=gzt(m−1),  KV=hz<t(m−1);  θ)g_{z_t}^{(m)} \leftarrow \mathrm{Attention}\!\left( Q = g_{z_t}^{(m-1)},\; KV = \mathbf{h}_{\mathbf{z}_{<t}}^{(m-1)};\; \theta \right)
Query Stream Update — At layer m, the query representation g for position z_t attends to the content representations h of all positions that precede z_t in the permutation, but NOT to h_{z_t} itself. This prevents the target token's content from leaking into the prediction.
hzt(m)←Attention ⁣(Q=hzt(m−1),  KV=hz≤t(m−1);  θ)h_{z_t}^{(m)} \leftarrow \mathrm{Attention}\!\left( Q = h_{z_t}^{(m-1)},\; KV = \mathbf{h}_{\mathbf{z}_{\leq t}}^{(m-1)};\; \theta \right)
Content Stream Update — The content stream update is identical to standard Transformer self-attention. Position z_t attends to all positions up to and including itself (z_{≤t}), seeing both positions and content. This stream builds the full contextual representation that other tokens' query streams can then use.

Long-range context: Transformer-XL integration

Standard Transformers have a fixed — typically 512 tokens for BERT. Anything beyond that window is invisible. XLNet borrows two ideas from Transformer-XL to break this limit:

Segment-level recurrence: After processing a segment, XLNet caches the hidden states from all layers. When processing the next segment, these cached states become extra context that the model can attend to. Think of it as a sliding memory window: the model processes text in fixed-length chunks, but each chunk can look back at the previous chunk's representations. This extends the effective context far beyond any single segment's length.

: Because the model now attends across segment boundaries, embeddings (like BERT's) would create confusion — position 0 in segment 2 is not the same as position 0 in segment 1. XLNet uses relative positional encodings that capture the distance between tokens rather than their absolute locations. This lets the model seamlessly extend attention across segments without positional collisions.

Open in Lab
Watch how cached hidden states from Segment 1 extend the context available to Segment 2, effectively doubling the context window.
The demo wakes as you arrive…

Partial prediction: efficiency without sacrificing quality

A naive implementation of permutation language modeling would predict every token in every permutation — extremely expensive. XLNet uses a practical optimization: partial prediction. Instead of predicting all T tokens, XLNet only predicts the last c tokens in each permuted sequence (where c is a hyperparameter, chosen so that 1/c ≈ 1/K of the tokens get predicted, similar to BERT's 15%).

Why only the last tokens? In a permuted order, the last tokens have the most context available — they've seen the most other tokens. Early tokens in the permutation have very little context, making their predictions noisy and uninformative. By focusing on the information-rich later positions, XLNet gets more learning signal per computation.

max⁡θ    Ez∼ZT[∑t=c+1∣z∣log⁡ pθ ⁣(xzt∣xz<t)]\max_\theta \;\; \mathbb{E}_{\mathbf{z} \sim \mathcal{Z}_T} \left[ \sum_{t=c+1}^{|\mathbf{z}|} \log \, p_\theta\!\left(x_{z_t} \mid \mathbf{x}_{\mathbf{z}_{<t}}\right) \right]
Partial Prediction Objective — Only compute the loss for the last (|z| - c) tokens in each permutation. The first c tokens serve as context but do not contribute to the training objective. This focuses learning on predictions with the richest context available.

The idea in code

XLNet permutation language modeling — core logicpython

Simplified to show the idea — not the real implementation.

import numpy as np

def sample_permutation(seq_len):
    """Sample a random factorization order."""
    return np.random.permutation(seq_len)

def build_attention_mask(perm, predict_start):
    """Build the attention mask that enforces the permuted order.

    For position perm[t]:
      - Content stream: can attend to perm[0], ..., perm[t] (including self)
      - Query stream: can attend to perm[0], ..., perm[t-1] (excluding self)
    Only predict positions from predict_start onward (partial prediction).
    """
    T = len(perm)
    content_mask = np.zeros((T, T), dtype=bool)
    query_mask   = np.zeros((T, T), dtype=bool)

    for t in range(T):
        pos = perm[t]
        # Positions visible: everything earlier in the permutation
        for s in range(t + 1):
            content_mask[pos, perm[s]] = True   # content sees self + earlier
        for s in range(t):
            query_mask[pos, perm[s]] = True      # query sees only earlier

    return content_mask, query_mask

def xlnet_loss(model, tokens, predict_ratio=0.15):
    """One training step for XLNet's permutation LM."""
    T = len(tokens)
    perm = sample_permutation(T)

    # Only predict the last ~15% of tokens in this permutation
    predict_start = int(T * (1 - predict_ratio))
    content_mask, query_mask = build_attention_mask(perm, predict_start)

    # Forward pass with two-stream attention
    content_h, query_g = model.two_stream_forward(
        tokens, content_mask, query_mask
    )

    # Compute loss only on predicted positions
    loss = 0
    count = 0
    for t in range(predict_start, T):
        pos = perm[t]
        logits = query_g[pos] @ model.vocab_proj.T
        loss += cross_entropy(logits, tokens[pos])
        count += 1
    return loss / count

# KEY INSIGHT: The tokens never move! Only the attention masks change.
# The permutation determines WHO can see WHOM — not where tokens sit.

Architecture: putting it all together

XLNet comes in two sizes, mirroring BERT for fair comparison:

  • XLNet-Base: 12 layers, 768 hidden dim, 12 attention heads — comparable to BERT-Base
  • XLNet-Large: 24 layers, 1024 hidden dim, 16 attention heads — comparable to BERT-Large

The architecture is a modified Transformer-XL with two-stream self-attention. Each layer runs both the content stream and query stream in parallel, with separate attention masks for each. The content stream uses standard self-attention (attending to self and prior context). The query stream uses a restricted mask that excludes the target's own content embedding.

For fine-tuning, the query stream is discarded — only the content stream is needed, which behaves identically to a standard Transformer. This means XLNet's fine-tuning procedure is as simple as BERT's: add a task-specific head on top and tune all parameters.

Open in Lab
Explore XLNet's architecture layer by layer. Click any component to see its role.
The demo wakes as you arrive…

Results: outperforming BERT on 20 tasks

Under comparable settings (same data, similar model size), XLNet outperformed BERT on every tested:

  • SQuAD 1.1: F1 jumped from BERT's 94.9 to XLNet's 95.5 (reading comprehension with guaranteed answers)
  • SQuAD 2.0: F1 from 89.1 to 90.6 (includes unanswerable questions)
  • RACE (reading comprehension): Accuracy from 81.8 to 85.0 — a particularly strong result since RACE requires reasoning over long passages
  • : XLNet-Large achieved state-of-the-art on multiple tasks
  • Text classification: Improvements on Yelp, IMDB, DBpedia, AG News, and Amazon reviews
  • Document ranking (ClueWeb09-B): Significant gains over BERT

Notably, XLNet achieved these results without . The confirmed that NSP did not help, aligning with later findings from RoBERTa.

Open in Lab
Compare XLNet vs BERT across major benchmarks. Hover over each axis for details.
The demo wakes as you arrive…

XLNet in the landscape of language models

  1. 2017

    Transformer

    The self-attention architecture that replaced recurrence. The encoder-decoder design set the stage for both BERT and GPT.

  2. 2018

    GPT-1

    Autoregressive pretraining with left-to-right language modeling. Proved that unsupervised pretraining + fine-tuning works, but limited to unidirectional context.

  3. 2018

    BERT

    Masked language modeling enables bidirectional context. Dominates 11 benchmarks but introduces [MASK] token mismatch and independence assumption.

  4. 2019

    Transformer-XL

    Segment-level recurrence and relative positional encoding for long-range dependencies. Provided the architectural backbone that XLNet builds upon.

  5. 2019

    XLNet

    Permutation language modeling + two-stream attention + Transformer-XL recurrence. Outperforms BERT on 20 tasks without [MASK] tokens.

  6. 2019

    RoBERTa

    Showed that careful training of BERT (more data, longer training, no NSP) closes part of the gap with XLNet, validating both approaches.

  7. 2020

    ELECTRA

    Replaced masking with replaced-token detection. Every token gets a training signal, achieving BERT-level results with far less compute.

XLNet's legacy is dual. On one hand, it showed that the autoregressive framework is more flexible than previously thought — with permutation training, it can capture bidirectional context just as well as masking-based methods. On the other hand, it demonstrated the importance of long-range context modeling through Transformer-XL's recurrence mechanism. While later models moved in different directions — RoBERTa back to simpler BERT training, ELECTRA to more efficient objectives — XLNet's innovations influenced architectures like UniLM and MPNet that continued to explore the space between autoregressive and autoencoding pretraining.

CitationYang, Dai, Yang, Carbonell, Salakhutdinov, Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. NeurIPS, 2019.

Terms in this paper