Gemma: Open Models Based on Gemini Research and Technology

By early 2024, the most powerful language models were closed: accessible only through APIs, with no visibility into weights, training data, or safety processes. Open alternatives like LLaMA-2 (7B, 13B, 70B) and Mistral 7B offered strong performance, but lacked the training infrastructure, data curation, and systematic safety evaluation that characterize frontier labs.

Google's response was Gemma — not just another open model, but a deliberate transfer of frontier methodology to the open ecosystem. Gemma models are built from the same research pipeline as Gemini: same architecture family, similar data processing, same safety evaluation framework. The goal is to give researchers and developers access to models that are simultaneously high-performing, small enough to run locally, and rigorously evaluated for safety.

The name "Gemma" comes from the Latin for "gem" — a small, precious stone. Two sizes were released: a 7 billion parameter model for GPU and TPU deployment, and a 2 billion parameter model designed for CPU and on-device applications. Both come as pre-trained checkpoints and instruction-tuned variants.

Gemma uses a decoder-only transformer — the same fundamental architecture behind GPT and Gemini. But the original 2017 transformer has been upgraded with four key improvements that emerged from the community over the following years. Think of these as engine upgrades on a proven car design: the chassis is the same, but each component has been refined for better efficiency and performance.

1. Multi-Query Attention (MQA) vs Multi-Head Attention (MHA). Standard multi-head attention maintains separate Key and Value projections for every attention head. Multi-Query Attention shares a single Key-Value pair across all heads, dramatically reducing memory during inference. Gemma 2B uses MQA (1 KV head), while Gemma 7B uses standard MHA (16 KV heads) — a choice made through ablation studies showing each variant performed best at its respective scale.

2. Rotary Position Embeddings (RoPE). Instead of adding absolute position numbers to tokens, RoPE encodes position through rotation in the embedding space. Each position rotates the query and key vectors by an angle proportional to its index. This gives the model a natural sense of distance: nearby tokens have similar rotations, distant tokens diverge. RoPE generalizes better to sequence lengths not seen during training.

3. GeGLU Activations. The standard ReLU activation ("pass positive values, block negatives") is replaced by GeGLU — a gated variant of GELU that multiplies the activation by a learned gate. Think of it as a smart valve: instead of simply blocking or passing, the network learns how much to let through, giving finer control over information flow.

4. RMSNorm. Instead of LayerNorm (which centers and scales), Gemma uses RMSNorm — normalization by root-mean-square alone, without the centering step. This is computationally cheaper and empirically works just as well for transformers, applied to the input of each attention and feedforward sub-layer.

Both Gemma models share the same architectural blueprint but differ in scale. The 7B model has 28 transformer layers with a model dimension of 3072 and 16 attention heads. The 2B model has 18 layers with a model dimension of 2048 and 8 attention heads. Both use a head size of 256 and a shared vocabulary of 256,128 tokens via a SentencePiece tokenizer inherited from Gemini.

The feedforward hidden dimension is intentionally large relative to the model dimension — 49,152 for 7B and 32,768 for 2B — because GeGLU activations split this dimension in half internally (one half for the gate, one half for the value), so the effective width is half the listed number. Both models share their input and output embeddings to reduce parameter count.

Gemma models are trained on Google's TPUv5e hardware. The 7B model trains across 16 pods (4,096 TPUv5e chips total), using 16-way model parallelism and 16-way data parallelism within each pod, with optimizer state sharding similar to ZeRO-3. The 2B model uses 2 pods (512 chips) with simple 256-way data replication. Cross-pod communication uses the Pathways system.

The training data is primarily English, sourced from web documents, mathematics, and code. The 2B model trains on 2 trillion tokens, while the 7B model trains on 6 trillion tokens. Notably, Gemma is not multimodal — unlike Gemini, it processes text only.

Data filtering is thorough: heuristics and model-based classifiers remove harmful or low-quality content, personal information is filtered out, evaluation sets are excluded from training data to prevent contamination, and the data mixture is staged — meaning the proportion of high-quality data increases toward the end of training, a technique shared with Gemini.

A pre-trained language model can complete text, but it does not know how to follow instructions or hold a conversation. Gemma's instruction-tuned (IT) variants go through two stages of alignment.

Stage 1: Supervised Fine-Tuning (SFT). The model is trained on a curated mix of English-only, text-only prompt-response pairs — both human-written and synthetic. The data mixture is selected using LM-based side-by-side evaluations: given held-out prompts, a high-capability judge model compares responses from the candidate and a baseline, expressing a preference. This selects for helpfulness, factuality, creativity, and safety simultaneously.

Stage 2: Reinforcement Learning from Human Feedback (RLHF). Human raters compare pairs of model responses. A reward model is trained under the Bradley-Terry framework to predict which response humans prefer. The policy is then optimized using a variant of REINFORCE with a KL-divergence regularization term toward the SFT model — this prevents the model from drifting too far from its supervised baseline while maximizing the reward signal.

Gemma also defines a specific conversation format using control tokens. Every user turn is wrapped with <start_of_turn>user and <end_of_turn>, and every model turn with <start_of_turn>model and <end_of_turn>. This formatting is required at inference time — generating without it is possible but produces worse results because the model is out of distribution.

Gemma was evaluated on 18 academic benchmarks spanning reasoning, knowledge, mathematics, coding, and safety. The headline result: Gemma 7B outperforms comparably sized open models on 11 of 18 tasks, and even surpasses the larger LLaMA-2 13B on several benchmarks.

On MMLU (broad knowledge), Gemma 7B scores 64.3% — higher than Mistral 7B (62.5%) and far above LLaMA-2 7B (45.3%). On GSM8K (grade school math), Gemma 7B achieves 46.4%, beating Mistral 7B (35.4%) by over 10 points. On HumanEval (code generation), Gemma 7B scores 32.3% — outperforming even the code-specialized CodeLLaMA 7B (which scores 41.4% on MBPP where Gemma scores 44.4%).

The 2B model holds its own at its smaller scale, scoring 42.3% on MMLU and 22.0% on HumanEval — competitive for a model designed to run on CPUs and mobile devices.

In human preference evaluations against Mistral 7B v0.2 Instruct, Gemma 7B IT achieved a 51.7% win rate on instruction following and a 58% win rate on safety. Even Gemma 2B IT achieved a 56.5% safety win rate against the much larger Mistral 7B — showing that safety is not purely a function of model size.

Safety is not an afterthought in Gemma — it is embedded at every stage. The paper presents evaluations on 10 safety benchmarks including RealToxicity (measuring toxic text generation), BOLD and CrowS-Pairs (social bias), BBQ (ambiguous bias questions), Winogender and Winobias (gender bias), TruthfulQA (factual accuracy), and Toxigen (adversarial toxicity prompts).

Three layers of safety defense are applied. First, data filtering: training data is scrubbed using heuristic and model-based classifiers to remove harmful, toxic, and personal content. Second, instruction tuning for safety: the SFT and RLHF stages explicitly optimize for safety alongside helpfulness, with data subsets that encourage hedging, refusals, and attribution. Third, memorization evaluation: the paper tests whether the model can regurgitate training data verbatim, checking both exact and approximate memorization, and specifically measures leakage of personal and sensitive information.

A critical safety concern for any language model is whether it memorizes and can reproduce training data — especially personal information. The Gemma team tested this by sampling 10,000 documents, using the first 50 tokens as a prompt, and checking if the model generates the next 50 tokens verbatim.

The key findings: Gemma memorizes data at rates comparable to PaLM models of similar size. No sensitive personal data was found to be memorized. Some potentially personal data was detected, but the automated detection tools used (Google Cloud Sensitive Data Protection) are known to produce many false positives. Approximate memorization (using a 10% edit distance threshold) showed roughly 50% more data is approximately memorized than exactly — consistent across all dataset categories.

Gemma's lasting contribution is not just performance numbers — it is the demonstration that responsible open releases are feasible at the frontier. By combining high-quality data curation, modern architectural improvements, systematic safety evaluation, and memorization testing, Google showed that "open" and "responsible" are not contradictory. The recipe book is available; the community decides what to cook with it.