PaLM: Scaling Language Modeling with Pathways

By 2022, the largest language models had hit a training infrastructure wall. A single TPU v3 Pod contained up to 2048 chips — enough for models like LaMDA and GLaM. But 540 billion dense parameters needed more. Megatron-Turing NLG (530B) used 2240 GPUs with pipeline parallelism, where the model is split into sequential stages across machines. This creates "pipeline bubbles" — idle time when earlier stages wait for later stages — limiting efficiency.

PaLM took a fundamentally different approach: instead of splitting the model into a pipeline, it used the Pathways system to coordinate training across 6144 TPU v4 chips in two full Pods, using only data parallelism at the Pod level and standard data+model parallelism within each Pod. No pipeline parallelism at all.

PaLM is a dense, decoder-only Transformer — the same family as GPT-3 — but with six engineering modifications that collectively boosted both quality and speed:

1. SwiGLU activations: Replace the standard ReLU/GELU in the feed-forward network with SwiGLU, which uses a gating mechanism: Swish(xW₁) ⊙ (xW₃). This requires three matrix multiplications instead of two, but the quality gains outweigh the compute cost.

2. Parallel layers: Instead of running attention and the feed-forward network sequentially (x + Attention(LayerNorm(x)) then + FFN(LayerNorm(...))), PaLM runs them in parallel: x + Attention(LayerNorm(x)) + FFN(LayerNorm(x)). This gives a ~15% training speedup at scale with no quality loss.

3. Multi-query attention: Standard multi-head attention projects separate key and value vectors for each head. Multi-query attention shares one key and one value projection across all heads, dramatically reducing memory and compute during autoregressive decoding.

4. RoPE embeddings: Rotary Position Embeddings encode position through rotation matrices applied to query and key vectors, enabling better generalization to longer sequences than absolute or learned position embeddings.

5. Shared input-output embeddings: The same embedding matrix is used for both the input token lookup and the output softmax projection, reducing parameter count.

6. No biases: All dense kernels and layer norms omit bias terms, improving training stability at large scale.

PaLM was trained at three scales to study scaling behavior:

• PaLM 8B: 32 layers, 16 heads, 4096 hidden dim — the baseline for ablation studies.
• PaLM 62B: 64 layers, 32 heads, 8192 hidden dim — competitive with Chinchilla (70B).
• PaLM 540B: 118 layers, 48 heads, 18432 hidden dim — the flagship model. Each head has dimension 256. The feed-forward intermediate size is 73728 (4× hidden dim).

All three were trained on 780 billion tokens from a mixture of filtered web pages, books, Wikipedia, news articles, source code (from GitHub), and social media conversations across 124 languages. The vocabulary uses a SentencePiece tokenizer with 256,000 tokens.

In a standard Transformer block, attention and the feed-forward network run sequentially: the output of attention feeds into the FFN, which feeds into the residual. This creates a dependency chain — the FFN must wait for attention to finish.

PaLM flips this into a parallel formulation. Both attention and FFN receive the same layer-normalized input, compute independently, and their outputs are summed into the residual stream together. Mathematically:

• Standard: y = x + FFN(LayerNorm(x + Attention(LayerNorm(x))))
• Parallel: y = x + Attention(LayerNorm(x)) + FFN(LayerNorm(x))

The parallel version allows the GPU/TPU to fuse the attention and FFN computations into fewer operations and better saturate the hardware. At PaLM's scale (540B), this yields a ~15% wall- clock training speedup. At smaller scales (8B), it causes a slight quality degradation — but at 62B and above, the quality matches the sequential version exactly.

The standard Transformer FFN applies a single activation function: FFN(x) = ReLU(xW₁)W₂. SwiGLU replaces this with a gated activation that learns which information to let through:

FFN(x) = (Swish(xW₁) ⊙ xW₃) · W₂

where Swish(u) = u · σ(u) is the Swish function. The key idea is that xW₃ acts as a gate: it modulates the output of Swish(xW₁) element-wise. This gating mechanism lets the network be more selective about which features to amplify or suppress. Think of it as a valve system — one pipe carries the signal, another pipe controls how much of each element flows through.

Despite requiring a third weight matrix (and thus ~33% more parameters in the FFN), SwiGLU consistently outperformed standard activations in quality across all model sizes, making it a net win when paired with the training speedup from parallel layers.

PaLM's training dataset was a carefully curated mixture designed to cover diverse use cases:

• Filtered web pages — the bulk of the data, filtered for quality.
• Books — long-form, structured text for coherent generation.
• Wikipedia — factual knowledge across languages.
• News articles — current events and journalistic style.
• Source code — from GitHub, enabling code generation and understanding.
• Social media conversations — dialogue-style text for conversational ability.

The multilingual aspect is notable: while English dominated (the majority of tokens), the dataset included text in 124 languages. This, combined with a large SentencePiece vocabulary of 256,000 tokens, gave PaLM surprisingly strong multilingual capabilities — especially for a model not specifically designed for multilingual use.

An important design choice: PaLM was trained for a single epoch (no data repetition). The 780B tokens were seen exactly once. This stands in contrast to Chinchilla's finding that smaller models trained on more data can match larger ones — PaLM showed that raw scale still matters when the data is large enough.

One of PaLM's most striking findings was about how performance scales. For most tasks, scaling from 8B → 62B → 540B followed a smooth, log-linear improvement curve — consistent with the scaling laws described by Chinchilla. More parameters, more performance, predictably.

But for certain tasks — particularly those requiring multi-step reasoning, logical deduction, or understanding jokes — the improvement was discontinuous. Performance barely budged from 8B to 62B, then jumped dramatically from 62B to 540B. These are what researchers call emergent abilities: capabilities that appear to "switch on" only at sufficient scale, as though the model needs to reach a critical mass of knowledge before a new skill clicks into place.

This discovery was crucial: it showed that log-linear scaling laws underestimate the benefits of large models on hard reasoning tasks. If you only looked at 8B → 62B performance, you'd predict modest gains at 540B. The actual gains blew past those predictions.

Perhaps PaLM's most impressive result was on multi-step reasoning. Previous large models struggled with tasks requiring logical chains: arithmetic word problems, commonsense reasoning, multi-hop question answering. PaLM 540B didn't just improve on these tasks — it outperformed fine-tuned specialist models using only few-shot prompting.

The key enabler was chain-of-thought prompting: instead of asking for a direct answer, the prompt includes examples that show step-by-step reasoning. For instance:

Q: Roger has 5 tennis balls. He buys 2 more cans of 3. How many does he have? A: Roger started with 5. He bought 2 cans × 3 = 6. Total: 5 + 6 = 11. The answer is 11.

With chain-of-thought prompting, PaLM 540B achieved 58.1% on GSM8K (grade-school math) with just 8-shot prompting — surpassing the fine-tuned state-of-the-art. It achieved 75.6% on StrategyQA (commonsense reasoning), also beating the fine-tuned SOTA.

This was a paradigm shift: it showed that at sufficient scale, a general-purpose model with the right prompting strategy can match or beat task-specific fine-tuned models — without any gradient updates.

PaLM 540B was evaluated across hundreds of benchmarks. Here are the highlights:

BIG-bench (150+ tasks): PaLM 540B with 5-shot prompting outperformed the average human rater on a majority of BIG-bench tasks. This was the first time a language model achieved this. On tasks like logical deduction, causal reasoning, and English proverbs, PaLM showed dramatic improvements over prior models.

Reasoning tasks: With chain-of-thought prompting, PaLM 540B achieved 58.1% on GSM8K and 75.6% on StrategyQA — both surpassing the fine-tuned state-of-the-art at the time.

NLP benchmarks (29 tasks): PaLM achieved state-of-the-art few-shot results on tasks including TriviaQA, Natural Questions, LAMBADA, and HellaSwag. On SuperGLUE (after fine-tuning), it scored 92.6%.

Code generation: PaLM showed strong code generation abilities, competitive with specialized code models, thanks to the GitHub source code in its training data.

Multilingual: Despite being primarily English-focused, PaLM set new records on multilingual question answering and generation tasks, outperforming even models trained on more non-English data.

PaLM's legacy extends in three directions. First, its infrastructure: the Pathways system proved that multi-Pod training without pipeline parallelism was viable, paving the road for even larger models. Second, its architecture: SwiGLU, parallel layers, MQA, and RoPE became the default recipe in subsequent models including LLaMA, Mistral, and Gemini. Third, its science: the discovery of emergent abilities and the power of chain-of-thought prompting reshaped how the field thinks about what scale can achieve.

Perhaps most importantly, PaLM showed that the scaling hypothesis was still alive. At a time when Chinchilla suggested that data efficiency matters more than raw size, PaLM demonstrated that both matter — and that beyond a certain scale, qualitatively new capabilities emerge that no amount of data alone can produce.