BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension

Before BART, the pre-training landscape was split into two camps. On one side, BERT and its variants used bidirectional encoders — every token sees every other token — ideal for classification, question answering, and any task where you need to understand a whole passage. But BERT predicts masked tokens independently (not one after the other), so it struggles to generate fluent text.

On the other side, GPT used an autoregressive decoder — each token only sees what came before it — perfect for text generation, but limited for understanding tasks because it cannot look ahead. It is like writing a story while covering the page below your pen: you can only build on what you have already written.

Models like XLNet, UniLM, and MASS tried to bridge this gap with clever masking schemes, but each made compromises. No single pre-training strategy could match state-of-the-art on both discriminative and generative benchmarks. The question was: can we design one model that inherits the encoder's comprehension and the decoder's generation ability simultaneously?

BART's core idea is elegant: pre-train a full encoder-decoder Transformer as a denoising autoencoder. The process has two steps. First, take a clean document and apply a noising function — any transformation that corrupts the text. Second, train the model to reconstruct the original document from the corrupted version. The encoder reads the corrupted text bidirectionally (like BERT), and the decoder produces the clean text autoregressively (like GPT).

The training objective is simply the cross-entropy loss between the decoder's output and the original document. This means BART learns to map any corrupted input back to the original text — and the flexibility is the key: unlike BERT which only masks individual tokens, BART can handle deletions, insertions, span replacements, and even sentence reordering.

Think of it as training a translator, but instead of translating between languages, BART translates between damaged text and clean text. The encoder must understand the damaged input deeply enough to pass its meaning forward, and the decoder must generate fluent output — exactly the two skills needed for downstream tasks.

A core contribution of BART is the systematic study of different text corruption strategies. The key insight is that how you corrupt the text determines what the model learns. Each strategy teaches the model a different skill:

Token Masking replaces random tokens with [MASK], identical to BERT. The model learns to predict individual words from context — good for local understanding, but the model always knows how many tokens are missing.

Token Deletion removes random tokens entirely. Now the model must figure out which positions are missing, not just what the words are. This is harder than masking because the input and output lengths differ.

Text Infilling samples random spans (with lengths drawn from a Poisson distribution, λ=3) and replaces each span with a single [MASK] token, regardless of span length. A span of five words becomes one mask. A span of zero words inserts a mask between existing words. The model must predict how many tokens are missing behind each mask — a skill no previous method taught.

Sentence Permutation shuffles the order of sentences in the document. This forces the model to learn about discourse structure and how sentences relate to each other — a skill crucial for summarization and long-form generation.

Document Rotation picks a random token and rotates the document so it starts with that token. The model must learn to identify the true beginning of the document.

BART's architecture is the standard sequence-to-sequence Transformer from the original Transformer paper — an encoder and a decoder, just like what was designed for machine translation. While BERT uses only the encoder half and GPT uses only the decoder half, BART uses the complete architecture.

The encoder is bidirectional: every token attends to every other token with no causal mask. The decoder is autoregressive: each token can only attend to previous tokens (via a causal mask) plus the encoder output (via cross-attention). This combination means BART can understand input from all directions and generate output sequentially.

Two model sizes were explored. BART-Base has 6 encoder layers and 6 decoder layers with 768 hidden dimensions. BART-Large has 12 layers each with 1024 hidden dimensions. Both use GeLU activations instead of ReLU and initialize parameters from a normal distribution N(0, 0.02). BART-Large contains roughly 10% more parameters than a similarly-sized BERT model because of the cross-attention layers in the decoder.

Because BART has both an encoder and a decoder, it naturally adapts to a wider range of tasks than encoder-only or decoder-only models:

Sequence classification (sentiment analysis, NLI): Feed the same input to both encoder and decoder. Use the final decoder token's hidden state — which has attended to the entire input — as the classification vector. This is analogous to BERT's [CLS] token, but placed at the end so it can attend to the full decoder output.

Token classification (NER, SQuAD): Feed the full document to encoder and decoder. Use each decoder token's final hidden state to classify that position.

Sequence generation (summarization, abstractive QA): The most natural fit. The encoder receives the input document and the decoder generates the output autoregressively. This directly mirrors the pre-training setup — corrupted text in, clean text out — which is why BART excels at generation tasks.

Machine translation: A creative approach. Replace BART's encoder embedding layer with a new randomly initialized encoder trained on the source language. The new encoder learns to map foreign text into the same representation space that BART's decoder was pre-trained to denoise. This lets BART act as a pre-trained target-language model for translation, even though it was only pre-trained on English.

BART-Large was pre-trained with text infilling (masking 30% of tokens) and sentence permutation, using the same 160GB corpus as RoBERTa and training for 500,000 steps with a batch size of 8000.

Discriminative tasks: BART matched RoBERTa on GLUE and SQuAD, despite having a decoder that is not needed for these tasks. This proved that adding a decoder does not hurt understanding performance — a concern many researchers had.

Summarization: BART achieved new state-of-the-art on both CNN/DailyMail (44.16 ROUGE-1) and XSum (45.14 ROUGE-1), a gain of roughly 6 ROUGE points on XSum over the previous best. XSum requires highly abstractive summaries, making it the perfect test for BART's generation abilities.

Dialogue: On ConvAI2, BART outperformed all previous methods in both F1 and perplexity.

Abstractive QA: On ELI5, BART set a new state-of-the-art with 30.6 ROUGE-1.

Machine translation: Using the stacked encoder approach, BART improved over a strong back-translation baseline by 1.1 BLEU on WMT Romanian→English, reaching 37.96 BLEU — using only English pre-training.

The controlled ablation study across six tasks revealed several critical lessons about pre-training design:

Token-level corruption is essential. Rotation and sentence shuffling alone performed poorly. The methods that worked all involved token masking or deletion. The model needs fine-grained word-level signal during pre-training.

Left-to-right generation matters. The masked language model (BERT-style) and permuted language model underperformed on generation tasks. These are the only approaches that lack autoregressive left-to-right decoding during pre-training. Generating coherent text requires practice at sequential generation.

Bidirectional encoding is crucial for understanding. A pure left-to-right language model collapsed on SQuAD (76.7 F1 vs 90.8 for BART), confirming that comprehension tasks need the encoder to see the full context.

The pre-training objective is not the only factor. Controlling for data and optimization, different objectives led to very different results. But even within the same architecture, hyperparameters like learning rate and layer normalization placement mattered significantly.

BART's deepest insight is that pre-training is restoration, not prediction. BERT predicts masked tokens. GPT predicts next tokens. But BART frames pre-training as reconstructing corrupted text — a more general objective that naturally spans understanding and generation. This framing opened the door to creative corruption strategies and showed that the diversity of corruption matters as much as the architecture. Every modern encoder-decoder model, from T5 to mBART to PEGASUS, builds on this foundation.