LIMA: Less Is More for Alignment

By 2023, the standard recipe for building a capable AI assistant had two expensive stages after pretraining. First, instruction tuning: fine-tuning the model on tens or hundreds of thousands of prompt-response pairs to teach it to follow instructions. Second, reinforcement learning from human feedback (RLHF): training a reward model from human preferences, then using it to further optimize the language model's outputs. Projects like InstructGPT collected millions of human annotations. Alpaca used 52,000 distilled examples. The implicit assumption was clear: more data, more human feedback, more training — better alignment.

But this assumption had never been rigorously tested. Is all that data really teaching the model new knowledge, or is it merely teaching the model how to present knowledge it already has? This is the question LIMA set out to answer.

The paper's central thesis is the Superficial Alignment Hypothesis: a model's knowledge and capabilities are learned almost entirely during pretraining, while alignment merely teaches the model which subdistribution of formats should be used when interacting with users. In other words, alignment is not about injecting knowledge — it's about unlocking a style.

Think of pretraining as building a massive library inside the model's weights — billions of facts, reasoning patterns, writing styles, code snippets, and world knowledge. Alignment, then, is not adding new books to the library. It's hiring a librarian who knows how to find the right book and present it clearly when someone asks a question.

If this hypothesis is correct, then a corollary follows: you should be able to align a pretrained model with a remarkably small number of examples, as long as those examples demonstrate the right style of interaction. This is exactly what LIMA tested.

If alignment is about teaching style rather than knowledge, then the training data doesn't need to be massive — but it does need to be excellent. The authors curated exactly 1,000 prompt-response pairs, totaling roughly 750,000 tokens, from a carefully balanced mix of sources.

750 examples from community forums. 200 from STEM Stack Exchange, 200 from other Stack Exchange topics, and 200 from wikiHow articles, all filtered for quality (minimum length, no first-person language, no cross-references). An additional 150 creative writing examples came from Reddit's r/WritingPrompts. The filtering was strict: answers had to be self-contained, well-written, and stylistically consistent with what a helpful AI assistant would produce.

250 examples manually authored. The paper's authors wrote these themselves, covering diverse tasks — advice, coding help, creative writing, chitchat, and more. They maintained a uniform response style: acknowledge the question, then answer it clearly. 13 of these examples specifically addressed safety-sensitive prompts with careful refusals.

50 examples from Natural Instructions. These covered NLP tasks like summarization and paraphrasing, adding task diversity without sacrificing quality.

The total: 1,000 examples. Compare this to Alpaca's 52,000 or InstructGPT's millions.

The training protocol is deliberately simple. Starting from LLaMA 65B, the authors fine-tuned on their 1,000 examples using standard supervised loss — no reward model, no PPO, no preference pairs. They introduced a special end-of-turn token (EOT) to separate user and assistant turns, avoiding conflation with the pretrained model's existing EOS token.

The hyperparameters were standard: AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay of 0.1, learning rate starting at $1 \times 10^{-5}$ and linearly decaying to $1 \times 10^{-6}$, batch size of 32, and training for 15 epochs.

One notable technique was residual dropout with linear increase: dropout started at $p_d = 0.0$ at the bottom layer and linearly rose to $p_d = 0.3$ at the top layer. This helped prevent overfitting on such a small dataset.

A surprising finding emerged: perplexity on held-out data negatively correlated with generation quality. Lower perplexity (normally considered better) actually produced worse outputs. The authors had to manually select checkpoints using a 50-example development set rather than relying on validation loss.

The authors compared LIMA against five strong baselines across 300 test prompts using human preference evaluation. The results were striking.

LIMA outperformed Alpaca 65B (trained on 52,000 examples) — humans preferred LIMA 53% of the time, with only 26% preferring Alpaca. It also outperformed DaVinci003 (OpenAI's RLHF-trained model) — LIMA won 44% vs 35%.

Against stronger baselines, LIMA remained competitive. Versus Bard, LIMA won or tied 58% of the time. Against Claude, 46%. Even against GPT-4 — widely considered the state of the art — LIMA produced equal or better responses 43% of the time. GPT-4 itself, when used as an annotator, preferred LIMA's own outputs over its own 19% of the time.

On an absolute scale, 50% of LIMA responses were rated excellent and 88% met prompt requirements. Only 12% failed to address the prompt adequately.

The authors ran three ablation experiments on a 7B LLaMA model to understand why a small dataset works so well, testing three factors: diversity, quality, and quantity.

Diversity matters. They compared 2,000 examples from Stack Exchange (diverse prompts across many topics) versus 2,000 examples from wikiHow (all "how-to" prompts). The diverse Stack Exchange data scored 3.83 on a 6-point quality scale, significantly outperforming wikiHow's 3.49. Diverse prompts teach the model to handle a wider range of user requests.

Quality matters. They compared 2,000 quality-filtered Stack Exchange examples versus 2,000 unfiltered ones (no length, style, or self-containedness filters). Filtered data scored 3.83 versus 3.33 for unfiltered — a 0.5-point gap from quality filtering alone.

Quantity surprisingly doesn't matter much. They trained on 2,000, 4,000, 8,000, 16,000, and 32,000 Stack Exchange examples. The result? Performance was essentially flat across this 16-fold increase in data size. Doubling or quadrupling the data provided no measurable gain. This is the strongest evidence for the Superficial Alignment Hypothesis: once the model has seen enough examples of the right style, more examples of the same style add nothing.

LIMA was trained entirely on single-turn interactions — one prompt, one response. Yet when tested in live multi-turn conversations, it showed surprisingly coherent dialogue ability, referencing information from previous turns. In 6 out of 10 test conversations, however, it failed within 3 interactions — a clear sign it was operating out of its training distribution.

The fix was remarkably cheap: adding just 30 hand-crafted dialogue chains (10 written by authors, 20 adapted from Stack Exchange comment threads) to the training set. This tiny addition — 30 examples out of 1,030 total — transformed LIMA's dialogue performance: excellent responses jumped from 45.2% to 76.1%, and failures dropped from 15 per 42 turns to just 1 per 46 turns. The fine-tuned model was significantly better in 7 out of 10 conversations.

This result reinforces the Superficial Alignment Hypothesis from a different angle: the model already knew how to conduct dialogue from pretraining. It just needed a few examples to activate that capability in the right format.

With only 13 safety-related training examples, LIMA responded safely to 80% of 30 potentially sensitive test prompts. When the malicious intent was explicit — like asking for a celebrity's home address — LIMA refused appropriately. However, when the harmful intent was implicit or disguised, LIMA was more likely to comply unsafely.

This is an important limitation. While the Superficial Alignment Hypothesis suggests that style can be learned from few examples, safety boundaries may require more robust training. The model had learned the format of a refusal ("I cannot help with that") but hadn't internalized the deeper reasoning about when refusals are necessary.