OPT: Open Pre-Trained Transformer Language Models

GPT-3 demonstrated in 2020 that a single 175-billion-parameter language model could perform dozens of tasks with zero or few examples — translation, question answering, arithmetic, code generation — simply by conditioning on a text prompt. The results were extraordinary, but the model was accessible only through a paid API. The weights were never released, the training data was described but not shared, and the training process was summarized in a handful of paragraphs.

This created a two-tier research ecosystem. Labs with hundreds of millions of dollars could train and study frontier models. Everyone else could only observe from outside, guessing at architectural details, unable to run ablation studies, safety audits, or fairness evaluations on the actual weights. Reproducibility — a cornerstone of science — was impossible.

OPT was Meta AI's response: replicate GPT-3's capability at the 175B scale, then release everything — model weights across eight sizes (125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 175B), the full training code, and a detailed logbook of the training process. The goal was not to advance the state of the art but to democratize access to it.

OPT deliberately mirrors the GPT-3 architecture to enable fair comparison. Each model is a decoder-only Transformer — a stack of layers where every token can attend only to previous tokens through a causal attention mask. This autoregressive design means the model generates text one token at a time, each prediction conditioned on all tokens that came before.

The architecture choices closely follow GPT-3 with a few notable differences. OPT uses ReLU activation instead of GeLU, and applies weight initialization from the Megatron-LM codebase. All models use a sequence length of 2048 tokens and are tokenized with the GPT-2 byte-level BPE tokenizer. Batch sizes range from 0.5M to 4M tokens depending on model size.

The training data is a combination of publicly available datasets: the text corpora used for RoBERTa (BookCorpus, CCNews, CC-Stories), a filtered subset of The Pile (including CommonCrawl, OpenWebText2, Wikipedia, and several others), and PushShift.io Reddit conversations. After deduplication — which reduced The Pile by roughly 66% — the final corpus contains approximately 180 billion tokens (about 800 GB of text). All data is tokenized with the GPT-2 byte-level BPE tokenizer for consistency with prior open language models.

Training OPT-175B required 992 NVIDIA A100 GPUs (80 GB each), organized using Fully Sharded Data Parallel (FSDP) combined with Megatron-LM tensor parallelism. This combination achieved up to 147 TFLOP/s per GPU — 17% more efficient than prior NVIDIA-published benchmarks. The AdamW optimizer was used with FP32 states and FP16 model weights, along with gradient clipping and dynamic loss scaling to maintain numerical stability.

But the true contribution is not the infrastructure recipe — it is the logbook. Over the course of two months, the team documented every significant event: at least 35 manual restarts, over 100 hosts cycled due to hardware failures, and multiple loss divergences that required lowering the learning rate and rolling back to earlier checkpoints.

The logbook reveals a pattern: loss divergences were detected by monitoring the dynamic loss scalar crashing to zero and the L2-norm of final-layer activations spiking. When this happened, the team rolled back to a checkpoint where the loss scalar was still healthy (≥1.0), lowered the learning rate, and resumed training. This mid-flight adjustment was not a one-time fix — it happened repeatedly throughout the training run.

This level of transparency was unprecedented. Before OPT, the published account of training a 175B model was essentially "we trained it and it worked." The logbook showed that training at this scale is fragile, iterative, and requires constant human intervention — a reality that the polished final papers had been hiding.

A key claim of the paper is that OPT-175B was developed with approximately 1/7th the carbon footprint of GPT-3. This estimate accounts for the compute used during the successful training run. The authors emphasize that most published carbon footprint numbers exclude the experimental and research phases — the failed runs, the hyperparameter searches, the debugging — which can dwarf the final training cost. OPT's logbook makes at least some of this hidden cost visible.

The improved efficiency comes from combining FSDP with Megatron-LM tensor parallelism, which achieved 17% better GPU utilization than previously published results. These infrastructure improvements were released alongside the model, meaning the entire community benefits from Meta's engineering investment — not just Meta.

The authors evaluated OPT across 16 standard NLP benchmarks in zero-shot and few-shot settings, following the GPT-3 evaluation protocol. The results show that OPT-175B achieves performance that is, on average, comparable to GPT-3 Davinci across model sizes. On some tasks OPT performs slightly better; on others, slightly worse.

However, the authors note an important caveat: OPT consistently underperforms GPT-3 on a subset of tasks. They hypothesize that differences in the exact one-shot and few-shot evaluation setup — prompt formatting, example selection, tokenization nuances — may account for some of this gap. This highlights a broader problem in LLM evaluation: small differences in how you format a prompt can meaningfully change benchmark scores.

OPT was also evaluated on dialogue tasks using five benchmark datasets: ConvAI2, Wizard of Wikipedia, Empathetic Dialogues, Blended Skill Talk, and Wizard of the Internet. In a fully unsupervised setting — no fine-tuning on dialogue data — OPT-175B performed competitively against models that were fully supervised on these tasks. This demonstrates the strong emergent conversational ability that comes from large-scale pretraining on diverse text including Reddit conversations.

The authors conducted responsible AI evaluations using three established benchmarks. On hate speech detection (ETHOS dataset), OPT-175B considerably outperformed GPT-3 Davinci in all zero-shot through few-shot settings — a notable strength.

However, on CrowS-Pairs — which measures stereotypical biases across nine categories including gender, race, religion, and socioeconomic status — OPT-175B performed worse than GPT-3 in most categories (lower is better, indicating less bias). The overall bias score was 69.5% for OPT versus 67.2% for GPT-3. The authors attribute this to the PushShift.io Reddit corpus, which has a higher incidence of stereotypical and discriminatory text compared to other corpora.

On StereoSet, which measures stereotype awareness across professions, gender, religion, and race, OPT-175B and GPT-3 performed similarly. The authors are transparent about these limitations, noting that opening the model allows the community to study and mitigate these biases — something impossible with a closed model.

Like GPT-3, OPT is trained with the standard causal language modeling objective. The intuition is simple: given a sequence of tokens, predict what comes next. The model reads tokens left-to-right, and at each position it produces a probability distribution over the entire vocabulary for the next token. Training maximizes the probability of the correct next token across the entire training corpus.

OPT's deepest legacy is not its benchmark numbers — it is the norm it established. Before OPT, the default for frontier language models was secrecy. After OPT, openness became a competitive advantage and a research expectation. Meta followed OPT with LLaMA, which pushed the efficiency frontier further. The BigScience consortium released BLOOM, a 176B multilingual model trained by over a thousand researchers. Falcon, Mistral, and dozens of others followed.

The logbook tradition also took root. Later open training runs published increasingly detailed accounts of training dynamics, loss spikes, and infrastructure challenges. Training a large model was no longer presented as a solved problem but as an active engineering challenge — and OPT was the paper that forced that honesty.