The Llama 3 Herd of Models

Most frontier labs chase architectural novelty — mixture of experts, state space models, new attention patterns. Meta took the opposite bet: use a standard dense Transformer and pour all effort into three levers:

1. Data quality — 15 trillion tokens, aggressively filtered and deduplicated. The filtering pipeline alone uses heuristic classifiers, fastText language identifiers, and model-based quality classifiers trained on Llama 2 itself.

2. Scale — 405 billion parameters trained on 3.8 × 10²⁵ FLOPs across 16,000 H100 GPUs. This is roughly compute-optimal for their training budget according to their own scaling laws.

3. Managing complexity — by keeping the architecture simple, the team could iterate faster on data recipes, training stability, and post-training alignment without debugging novel components.

The philosophy is: don't complicate what works. Scale the recipe, not the architecture.

Llama 3's training corpus is 15 trillion tokens — over 8× larger than Llama 2's 1.8 trillion. But the raw size is only half the story. The team built an elaborate multi-stage filtering pipeline:

Stage 1: Deduplication. URL-level, document-level, and line-level deduplication removes near-duplicate crawl data. MinHash is used for fuzzy document deduplication.

Stage 2: Heuristic filtering. Rules eliminate documents with too many dirty words, excessive special characters, or abnormally low token counts.

Stage 3: Model-based quality scoring. A fastText classifier trained on curated high-quality pages (e.g. Wikipedia references) assigns quality scores. Only documents above a threshold pass through.

Stage 4: Domain-specific pipelines. Separate pipelines for code (syntax validation, decontamination), math (LaTeX extraction, problem filtering), and multilingual data (per-language quality classifiers).

The data mix itself is carefully tuned: roughly 50% English web data, 25% code, and 25% a mixture of multilingual, math, and curated knowledge sources. The mix shifts during training — in the final annealing phase, they upsample the highest-quality sources.

Llama 3 is a decoder-only dense Transformer. The architecture deliberately avoids trendy additions like mixture of experts — simplicity is the point. But several carefully chosen modifications make it efficient at scale:

Grouped Query Attention (GQA): Instead of giving every query head its own key-value pair, 8 key-value heads are shared across all 128 query heads (in the 405B model). Think of it like a conference room: instead of 128 private secretaries taking notes, 8 shared note-takers serve everyone. This dramatically reduces the KV cache size during inference without hurting quality.

Rotary Position Embedding (RoPE): Positions are encoded by rotating the query and key vectors in 2D planes. The base frequency is set to 500,000, enabling stable context lengths up to 128K tokens. During pre-training, the context starts at 8K and is extended to 128K in a dedicated long-context training stage.

SwiGLU activation: The feed-forward sublayers use SwiGLU — a gated variant that multiplies two parallel projections, one through a Swish activation. This consistently outperforms ReLU and GELU in practice.

RMSNorm pre-normalization: Each sublayer is preceded by RMS normalization (not LayerNorm). This is simpler, computationally cheaper, and has become the standard for large-scale training.

Training a 405B model at this scale is an engineering feat. The compute infrastructure used 16,384 NVIDIA H100 GPUs, each delivering roughly 400 TFLOPS of BF16 compute. Three parallelism strategies were combined:

Tensor Parallelism (TP): Each layer's weight matrices are split across 8 GPUs within one node. Think of this as eight people each holding one column of a spreadsheet — they do their part and pass results around.

Pipeline Parallelism (PP): The 126 layers are divided into stages across 16 pipeline stages, so different GPUs process different layers of the model simultaneously — like stations on an assembly line.

Data Parallelism (DP/FSDP): The same batch is split across replicas. With Fully Sharded Data Parallelism, optimizer states and gradients are sharded too, dramatically reducing per-GPU memory.

The total training compute was approximately 3.8 × 10²⁵ FLOPs. To put this in perspective: if a single H100 ran continuously, it would take over 30 million years to complete the same training run. The team achieved 38–43% Model FLOPs Utilization (MFU), meaning they extracted nearly half of the theoretical GPU throughput — a strong result at this scale. During training they experienced 466 job interruptions, of which 47 were planned, demonstrating the need for robust checkpointing and automatic recovery.

Pre-training produces a next-token predictor. Post-training turns it into an assistant that follows instructions, reasons carefully, and refuses harmful requests. Llama 3's post-training is an iterative loop with three stages:

1. Rejection Sampling: Generate multiple candidate responses using the current model, score them with a reward model, and keep only the best. This produces high-quality synthetic training data that pushes quality higher than human-written examples alone.

2. Supervised Fine-Tuning (SFT): Fine-tune on a curated mix of human-written and synthetic demonstrations. The data covers dialogue, coding, math reasoning, tool use, multilingual interaction, and long-context tasks.

3. Direct Preference Optimization (DPO): Instead of training a separate reward model and running PPO (as in Llama 2), DPO directly optimizes the policy by contrasting chosen vs rejected response pairs. This is simpler, more stable, and the team found it performed as well as or better than PPO.

This loop runs multiple rounds — each round uses the improved model to generate better rejection samples for the next round, creating a self-improving cycle. The key insight: by the later rounds, the model generates training data that exceeds what human annotators can produce on complex tasks like coding and math.

Llama 3 uses a BPE tokenizer with a vocabulary of 128,256 tokens — 4× larger than Llama 2's 32K vocabulary. This larger vocabulary achieves a compression ratio of 3.94 characters per token (vs ~4.5 for Llama 2), meaning the same text requires fewer tokens and the model can process more text within its context window.

Critically, 28,000 tokens were specifically added for non-English languages. This dramatically improved compression for languages like Arabic, Hindi, Thai, and others that were poorly served by the smaller vocabulary. Better compression means the model can process longer passages in these languages and training efficiency improves.

Safety is woven into every stage. The post-training mix includes safety-focused data: adversarial prompts with safe refusals, borderline examples where the model should engage rather than over-refuse, and red-teaming data to handle edge cases.

Llama Guard 3 is a separately trained classifier that acts as an input/output filter. It evaluates both user prompts and model responses against a taxonomy of harmful categories (violence, sexual content, criminal advice, etc.) and can block or flag unsafe content. This modular design means developers can adjust safety thresholds without retraining the main model.

The team reports achieving a much better balance between helpfulness and harmlessness compared to Llama 2, which was criticized for excessive refusals. They specifically optimized to reduce false refusals while maintaining safety.

Llama 3 405B matches or closely approaches GPT-4 across major benchmarks:

• MMLU (general knowledge): 88.6% — competitive with GPT-4's ~86.4%
• HumanEval (code generation): 89.0% — matching frontier code models
• GSM8K (math reasoning): 96.8% — near-ceiling performance
• MGSM (multilingual math): 91.6% — strong across 8 languages
• MATH (competition math): 73.8% — substantial improvement over Llama 2

The smaller models are equally impressive relative to their class. The 8B model outperforms Mistral 7B and Gemma 7B across most tasks. The 70B model competes with much larger models including Mixtral 8x22B. These smaller models were intentionally over-trained — trained far beyond compute-optimal on more tokens — sacrificing training efficiency for better final quality.

The paper also presents experimental results for multimodal extensions of Llama 3, using a compositional approach — specialized encoders are connected to the language model via cross-attention adapter layers, without modifying the core Transformer.

Vision: A ViT-H/14 image encoder (630M parameters) produces patch embeddings that are injected into the text model through cross-attention layers every four Transformer blocks. This adds roughly 100B parameters to the 405B model.

Video: Frames are encoded by the same ViT, then a temporal aggregator (Perceiver Resampler) merges every 32 frames into summary representations.

Speech: A 1B-parameter Conformer encoder processes audio spectrograms, and its outputs are integrated through the same adapter architecture.

These multimodal versions performed competitively with state-of-the-art models but were not publicly released as they were still under development at the time of publication.

One of the most practically impactful contributions is Llama 3's approach to scaling laws. Before committing billions of GPU-hours, the team trained hundreds of small models (40M to 16B parameters) and used them to predict the performance of the 405B model.

Their key finding: the standard Chinchilla scaling law (which predicts compute-optimal model size) works well for determining the 405B flagship size, but smaller models benefit from being trained far beyond compute-optimal. The 8B model, for instance, was trained on tokens far exceeding what Chinchilla would recommend — because at inference time, compute cost depends on model size, not training tokens. A smaller model trained longer is cheaper to deploy than a compute-optimal larger model.

They also developed a two-stage scaling law for predicting benchmark performance: first predict the negative log-likelihood from compute, then predict the benchmark score from the log-likelihood. This pipeline accurately predicted the 405B model's benchmark scores from small-scale experiments.