BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

When GPT-3 appeared in 2020, it stunned the world with its ability to write essays, translate languages, and solve problems from just a few examples. But OpenAI never released its weights. Google followed with PaLM and Chinchilla — equally impressive, equally closed. The pattern was clear: only organizations with billions of dollars and thousands of GPUs could build frontier models, and they had no obligation to share.

This created a deep asymmetry. Researchers at universities could read the papers, but not reproduce the results. Developers in the Global South had to rely on API access controlled by companies in Silicon Valley. Languages other than English were afterthoughts — GPT-3 was trained almost entirely on English text. The most powerful technology of the decade was being shaped by a narrow group, for a narrow audience.

BigScience set out to challenge this. Organized by Hugging Face with support from the French government, it gathered over 1,000 researchers from more than 70 countries into working groups spanning engineering, data governance, ethics, and evaluation. The goal: build a large language model in the open, document every decision, and release everything.

A language model is only as good as the text it trains on. Most LLMs use crawled web data with minimal curation — Common Crawl, the Pile — and the result is models that reflect the biases and language distribution of the English-language internet.

BLOOM was trained on the ROOTS corpus: 1.6 terabytes of text spanning 498 sources across 46 natural languages and 13 programming languages. But ROOTS was not just big — it was deliberately curated. A dedicated data governance working group sourced data from books, scientific papers, government documents, Wikipedia, and community-gathered datasets. Each source was documented with a data card describing its provenance, license, language, and potential biases.

Crucially, the team made conscious choices about language representation. Rather than letting data availability dictate proportions (which would make the model 90%+ English), they actively sought high-quality sources for underrepresented languages including several African and Southeast Asian languages. The corpus was deduplicated, personally identifiable information was redacted, and a quality filtering pipeline removed low-quality text.

The tokenizer — the component that splits text into tokens the model processes — is often treated as a detail. But for a multilingual model, it is a make-or-break design decision. A tokenizer trained mostly on English will split Chinese or Arabic words into many tiny pieces, wasting the model's context window and degrading performance.

BLOOM uses Byte-Pair Encoding at the byte level with a vocabulary of 250,680 tokens. Three design choices were critical. First, no Unicode normalization was applied — this preserves scripts like Arabic diacritics and CJK characters faithfully. Second, the vocabulary size was set large enough to keep fertility (the average number of tokens per word) close to monolingual tokenizers for most languages. Third, the vocabulary size was constrained to be divisible by 128 (for GPU efficiency) and by 4 (for tensor parallelism).

The result: a single tokenizer that handles 46 natural languages and 13 programming languages without destroying any of them. Text in Arabic, French, Vietnamese, and Python all tokenize efficiently through the same pipeline.

BLOOM follows the decoder-only Transformer architecture — the same family as GPT-3. The model has 70 layers, 112 attention heads, a hidden dimension of 14,336, and processes sequences of 2,048 tokens. The total parameter count is 176 billion, of which 3.6 billion are in the embedding layer alone. It uses GeLU activations and tied input-output embeddings.

But the BigScience team did not simply copy GPT-3. Through a systematic ablation study at the 1-billion-parameter scale, they tested architectural variations and adopted two key deviations from the standard Transformer.

ALiBi Positional Embeddings. Standard Transformers add positional information to token embeddings — either through learned vectors or sinusoidal functions. ALiBi (Attention with Linear Biases) takes a completely different approach: instead of modifying the embeddings, it adds a linear penalty directly to the attention scores. The further apart two tokens are, the larger the penalty — like a rubber band that makes distant tokens harder to attend to. Each attention head gets a different slope for this penalty, so some heads focus locally while others reach further.

The team found that ALiBi produced smoother training dynamics and better downstream performance than both learned embeddings and rotary embeddings (RoPE), even without its original motivation of sequence length extrapolation.

Embedding LayerNorm. The second deviation is an additional layer normalization immediately after the embedding layer — before the first Transformer block. In preliminary experiments at 104B parameters (using float16), this extra normalization dramatically improved training stability, preventing the loss spikes and divergence that plague large-scale training runs.

There is a trade-off: embedding LayerNorm slightly penalizes zero-shot generalization. The team accepted this cost because training stability at 176B scale was non-negotiable — a single divergence could waste weeks of supercomputer time. Later research suggested that switching from float16 to bfloat16 (which BLOOM's final training used) may reduce the need for this extra normalization.

Training a 176B-parameter model does not fit on a single GPU — or even a single machine. BLOOM was trained on the Jean Zay supercomputer in Paris using 384 NVIDIA A100 80GB GPUs organized into 48 nodes of 8 GPUs each. The training ran for 117 days, from March 11 to July 6, 2022, processing approximately 366 billion tokens.

To distribute the work across GPUs, the team combined three parallelism strategies, often called "3D parallelism," building on NVIDIA's Megatron-LM framework with DeepSpeed integration.

Data Parallelism replicates the entire model on multiple GPU groups. Each group processes a different mini-batch, then gradients are averaged via all-reduce. Think of it as photocopying the model and giving each copy a different homework assignment, then combining the answers.

Tensor Parallelism splits individual layers across GPUs within a single node. A 14,336-dimension matrix multiplication can be distributed across 4 GPUs, each handling a slice. This requires fast interconnects (NVLink) because GPUs must exchange intermediate results.

Pipeline Parallelism assigns different layers to different GPU groups. GPU group 1 handles layers 1–18, group 2 handles layers 19–35, and so on. The data flows through GPUs sequentially, like an assembly line. The challenge is keeping all GPUs busy — "pipeline bubbles" occur when some stages wait for others.

BLOOM used Data Parallelism with 8 copies, Tensor Parallelism of degree 4 within each node, and Pipeline Parallelism of degree 12 across nodes. These fused CUDA kernels from Megatron-LM — fused LayerNorm, fused GeLU with bias — squeezed maximum throughput from the hardware.

The pretrained BLOOM model performs competitively with GPT-3 on English benchmarks and shows strong multilingual capabilities. On SuperGLUE in zero-shot and few-shot settings, BLOOM's performance is similar to comparably-sized GPT models trained on the Pile.

But the real story is what happens after multitask prompted finetuning. The team created BLOOMZ by finetuning BLOOM on xP3 — a multilingual collection of prompted tasks spanning 46 languages. The result was dramatic: BLOOMZ showed substantially stronger zero-shot task generalization across languages, closing the gap with models that had been finetuned on much more English-centric data.

This demonstrated a crucial insight: a multilingual model, when properly finetuned with multilingual prompts, can generalize to new tasks in languages it saw during finetuning — and sometimes even in languages it did not see at all, through cross-lingual transfer.

BLOOM was not released under a standard open-source license. Instead, the team created the Responsible AI License (RAIL) — a novel approach that grants broad access and modification rights while restricting specific harmful uses. Users can download, modify, and deploy BLOOM freely, but the license prohibits using it to generate disinformation, conduct surveillance, or cause deliberate harm.

This was a deliberate middle path between two extremes: fully closed models (GPT-3, PaLM) that restrict access entirely, and fully permissive licenses (MIT, Apache) that impose no use restrictions. The RAIL approach acknowledges that a 176B-parameter model capable of generating fluent text in 46 languages is powerful enough to warrant some guardrails, even when the goal is maximum openness.

BLOOM's most important contribution was not its benchmark scores — it was the proof of concept. Before BLOOM, many assumed that building a frontier LLM required the resources of a big tech company. BLOOM showed that an international collaboration, organized openly and transparently, could match the scale of corporate efforts. It opened the door for every open model that followed.

The model also established norms that the field now takes for granted: publishing training details, documenting data sources, measuring carbon footprints, and thinking carefully about licensing. These practices were pioneering when BLOOM did them. They are now expected.