Toolformer: Language Models Can Teach Themselves to Use Tools

Large language models can write poetry, summarize legal documents, and pass exams — yet a pocket calculator outperforms them at arithmetic, and a simple database lookup beats them at retrieving facts. This is the core paradox: the same scaling that produces emergent reasoning abilities does not fix these fundamental blind spots.

The problem runs deeper than simple ignorance. Language models hallucinate facts because they have no mechanism to verify claims against an authoritative source. They fail at arithmetic because their training objective — predicting the next token — never required learning precise computation. And they cannot access real-time information because their knowledge is frozen at the moment training data was collected.

Humans solve this effortlessly. We reach for a calculator when numbers get complex, search the web when memory fails, and check a calendar when dates matter. The question is: can a language model learn to do the same?

The first clever insight of Toolformer is representational: an API call can be written as a text sequence using special tokens. The call has a name, an input, and a response, all enclosed in markers. For example, a calculator call looks like: [Calculator(171 + 312) → 483].

By encoding tool calls this way, the model does not need a separate module or architecture change. It simply learns to generate these special tokens at appropriate positions in the text. To the autoregressive model, calling a tool is no different from generating any other sequence of tokens — the magic is in learning when to generate them.

This design makes Toolformer API-agnostic: any tool whose input and output can be expressed as text can be plugged in. The paper demonstrates five tools — a calculator, two search engines (Wikipedia and a web search), a question-answering system, a machine translation system, and a calendar — but the framework is extensible to any text-based API.

The heart of Toolformer is a three-step pipeline that converts a plain text dataset into an augmented dataset containing useful API calls. Think of it as a quality-control assembly line: the model proposes many possible tool insertions, then reality checks them, and finally only the genuinely helpful ones survive.

Step 1 — Sample: For each sentence in the training data, the model proposes candidate API calls at promising positions. It does this using in-context learning: a few demonstrations showing example API calls are prepended to the text, and the model generates candidate calls by sampling. Positions are ranked by how likely the model thinks a <API> token should appear there.

Step 2 — Execute: Each candidate API call is actually executed — the calculator computes the arithmetic, the search engine retrieves results, the Q&A model answers the question. This is the reality check: the tool's actual output replaces the model's speculation.

Step 3 — Filter: This is the critical step. A call is kept only if inserting the tool's response reduces the model's loss on predicting the tokens that follow. If the tool's answer does not help — or actively hurts — the call is discarded. The surviving calls form the augmented training dataset.

The filtering criterion is what makes the entire approach self-supervised. The idea is simple: compare how well the model predicts the tokens after the API call in two scenarios — with the tool's response included, and without it (or with an empty response).

The model computes a weighted cross-entropy loss over the tokens following the API call position. If the loss with the tool's result is substantially lower than the loss without it, the API call is considered helpful and retained. The threshold $\tau$ controls how much improvement is required.

Intuitively, this asks: "Does knowing the tool's answer make the model less surprised by what comes next?" If the calculator returns 483 and the text continues with "483," then yes — the tool dramatically reduced the model's uncertainty. If the tool's answer is irrelevant, the loss barely changes and the call is discarded.

Toolformer demonstrates its approach with five diverse tools, each addressing a different weakness of language models:

Calculator — handles arithmetic that language models routinely get wrong. The model learns to emit expressions like Calculator(171 × 3) and insert the result.

Wikipedia Search — retrieves relevant passages when factual grounding is needed. The model generates queries like WikiSearch("Eiffel Tower height").

Web Search — a broader search engine for general knowledge retrieval when Wikipedia does not cover the topic.

Question Answering — calls an external QA model that can answer factual questions based on retrieved evidence, complementing the search tools.

Machine Translation — translates text to or from other languages, particularly useful for low-resource language content.

Calendar — returns the current date, allowing the model to answer time-dependent questions that would otherwise be impossible given a fixed training cutoff.

Each tool requires only 5–10 hand-written demonstration examples in the few-shot prompt. From these minimal seeds, the model bootstraps thousands of training examples through the sample-execute-filter pipeline.

The base model is GPT-J with 6.7 billion parameters. The pipeline annotates a subset of the CCNet dataset with API calls — up to 25,000 examples per API tool. Crucially, the augmented dataset $C^*$ also contains the original unannotated examples, so the model does not forget how to generate regular text without tools.

Training uses a standard autoregressive language modeling loss. The model learns to predict the API call tokens (including the special <API> and </API> markers) as part of the regular token stream. At inference time, a small modification to the decoding strategy allows API calls: whenever the <API> token appears among the top-$k$ most likely next tokens (not just the top-1), the model is allowed to begin a tool call. This increases tool use without requiring the model to be fully confident that a tool is needed.

The training is lightweight: up to 2,000 steps on 8 NVIDIA A100 GPUs using DeepSpeed ZeRO-3, with a batch size of 128 and learning rate of $10^{-5}$. The key finding is that this fine-tuning does not degrade the model's core language modeling ability — perplexity on standard benchmarks remains unchanged.

Toolformer is evaluated in a zero-shot setting — no task-specific examples are provided at inference time. The results are striking:

On math benchmarks (ASDiv, SVAMP, MAWPS), Toolformer with the calculator tool dramatically outperforms both GPT-J and the much larger OPT (66B) and GPT-3 (175B). The calculator gives Toolformer near-perfect arithmetic, which no amount of scaling can match.

On factual QA (LAMA's SQuAD, GoogleRE, T-REx subsets), Toolformer with search tools outperforms GPT-J by a wide margin. The search and QA tools ground the model's answers in retrieved facts rather than memorized (and potentially hallucinated) knowledge.

On temporal reasoning using the TempLAMA benchmark, the calendar tool gives Toolformer a substantial edge — it can answer questions about current dates that any frozen model would get wrong.

Importantly, on tasks where tools are not helpful (creative generation, general NLU), Toolformer performs identically to the base GPT-J model. It does not lose generality.

An important finding is that tool-use ability does not emerge at all model sizes. The authors tested GPT-2 variants from 124M to 1.6B parameters. Smaller models could not learn to use tools effectively — their performance with Toolformer was barely better than the baseline, and sometimes worse.

This suggests that learning when to use a tool requires a certain level of language understanding that only emerges at larger scales. The model needs enough capacity to represent both the concept of tool calling (a meta-skill) and the actual language understanding needed to decide which tool is appropriate for a given context.

At the GPT-J scale (6.7B), the capability clearly emerges: the model consistently and correctly inserts API calls where they help and refrains from calling tools where they would not. This aligns with the broader observation that many interesting LLM capabilities are emergent — appearing only beyond a threshold of scale.

Toolformer's lasting contribution is not any specific tool or benchmark result — it is the demonstration that a language model's own loss signal can supervise tool-use learning without human annotation. This insight unlocked the tool-augmented LLM era. Today, every major language model can call functions, browse the web, execute code, and interact with databases. The line from Toolformer to modern agentic AI is direct and unbroken.