MusicLM: Generating Music From Text

MusicLM does not train a single giant model from scratch. Instead, it assembles three independently pre-trained and frozen components, each specializing in a different aspect of audio understanding. Think of them as three experts who already know their craft — MusicLM just learns how to coordinate them.

SoundStream is a neural audio codec. It compresses raw audio waveforms into a compact sequence of discrete tokens — called acoustic tokens — using residual vector quantization (RVQ). RVQ chains multiple codebooks together: the first codebook captures the coarsest structure, each subsequent codebook refines the residual error, like adding layers of detail to a sketch. For 24 kHz audio, SoundStream produces 600 tokens per second across 12 quantization levels, at a bitrate of just 6 kbps. These tokens are sufficient to reconstruct high-fidelity audio.

w2v-BERT is a self-supervised speech model trained with masked language modeling on audio. From an intermediate layer (layer 7), MusicLM extracts representations that capture semantic content — the kind of information that tells you whether the audio is jazz or classical, whether the tempo is fast or slow, whether there is singing or only instruments. These representations are quantized into 25 semantic tokens per second using k-means clustering with 1024 centroids.

MuLan is a joint music-text embedding model, analogous to CLIP in the image domain. It has two towers — one processes audio, one processes text — both projecting into a shared 128-dimensional space trained with contrastive learning. If a piece of music and a text description refer to the same concept, their embeddings land close together. MuLan embeddings are quantized into just 12 MuLan tokens per audio sequence using RVQ. This is the bridge that connects text to music.

MusicLM generates music through a cascade of three autoregressive Transformer stages. Each stage takes as input the conditioning tokens plus the output of the previous stage, and predicts the next level of tokens.

Stage 1 — Semantic modeling. This stage takes the MuLan tokens (which encode "what kind of music") and generates the semantic tokens from w2v-BERT. It models the distribution $p(S_t | S_{<t}, M_A)$. Think of this as sketching the composition: deciding the melodic contour, rhythmic feel, and temporal structure. This stage is trained on 30-second crops of audio.

Stage 2 — Coarse acoustic modeling. Given the MuLan tokens and the semantic tokens from Stage 1, this stage predicts the first 4 levels of the SoundStream RVQ. It models $p(A_t | A_{<t}, S, M_A)$. These coarse acoustic tokens add harmonic content, rough timbre, and approximate dynamics — like filling in the colors of a painting.

Stage 3 — Fine acoustic modeling. The remaining 8 RVQ levels are generated conditioned on everything above. This adds the final layer of acoustic detail — precise timbre, reverb, stereo texture. The resolution goes from a sketch to a studio-quality recording.

Each stage uses a decoder-only Transformer with 24 layers, 16 attention heads, an embedding dimension of 1024, and 430M parameters. They are trained independently on audio-only data.

The central insight of MusicLM is how it sidesteps the paired-data bottleneck. In the image domain, DALL·E 2 has billions of image-text pairs. In the music domain, such data barely exists. MusicLM's solution: train on audio only, and swap the conditioning signal at inference time.

During training, the conditioning signal is the MuLan audio embedding — extracted from the target audio itself. The model learns to generate music given a description of what that music sounds like. No text captions are needed. This allows training on a massive audio-only dataset of 280,000 hours — five million clips.

During inference, the MuLan text embedding from the user's prompt replaces the audio embedding. This works because MuLan was trained with contrastive learning to place matching audio-text pairs close together in embedding space. From the Transformer's perspective, the conditioning tokens look the same whether they came from audio or text.

This design is analogous to DALL·E 2's use of CLIP, but with a crucial simplification: MusicLM skips the "prior" model that DALL·E 2 uses to map text embeddings to image embeddings. The shared embedding space is tight enough that direct substitution works.

Residual vector quantization (RVQ) is the backbone of MusicLM's token system. Standard vector quantization (VQ) maps each audio frame to the nearest entry in a single codebook — but a single codebook can only capture so much detail. Increasing codebook size causes an exponential explosion in memory and lookup cost.

RVQ solves this by chaining multiple smaller codebooks. The first codebook quantizes the original signal. The second codebook quantizes the residual — the error left over after the first quantization. The third codebook quantizes the residual of the residual. And so on. The reconstruction is the sum of all codebook outputs.

In SoundStream, 12 codebooks with 1024 entries each give the same representational capacity as a single codebook with $1024^{12}$ entries — an astronomically large number — but with only $12 \times 1024$ entries in total. More importantly, the codebooks form a natural hierarchy: the first few levels capture the most important structure, and later levels add refinement. MusicLM exploits this by generating coarse levels (1–4) and fine levels (5–12) in separate stages.

Some aspects of music are easier to hum than to describe in words. MusicLM addresses this with melody conditioning: the user provides a hummed, whistled, or sung melody alongside a text description. The text controls the style (genre, instrumentation, mood), while the melody controls the melodic contour.

To capture melody invariant to timbre, the authors train a small ViT-based embedding model using triplet loss on pairs of audio clips with matching melodies but different acoustics (e.g., a song and its cover version). The resulting 192-dimensional melody embeddings are quantized with RVQ (24 quantizers, vocabulary 512) and concatenated with MuLan tokens as conditioning.

The result: you can whistle a tune, type "epic orchestral," and MusicLM will render your whistle as a full orchestral arrangement — same melody, completely different sound.

MusicLM was evaluated against Mubert (API-based, uses pre-recorded sounds by musicians) and Riffusion (fine-tuned Stable Diffusion on mel spectrograms) on the MusicCaps benchmark. Three metrics capture different quality dimensions.

Fréchet Audio Distance (FAD) measures audio quality without needing a reference. MusicLM achieved FAD$_\text{VGG}$ = 4.0, compared to 9.6 for Mubert and 13.4 for Riffusion, meaning its generated audio sounds more like real music.

KL Divergence (KLD) measures whether the generated music has similar acoustic characteristics to the reference. MusicLM scored 1.01, vs 1.58 for Mubert and 1.19 for Riffusion — tighter adherence to the text description.

MuLan Cycle Consistency (MCC) measures cosine similarity between the text embedding and the embedding of the generated audio. MusicLM scored 0.51, far above Mubert's 0.32 and Riffusion's 0.34.

In human listening tests, MusicLM was preferred over both baselines in pairwise comparisons, winning 312 out of 600 comparisons, while there remained a gap with the ground truth reference music (472 wins).

A critical concern with generative music models is whether they memorize and reproduce copyrighted music from the training set. The authors adapted a memorization methodology from large language models to study this.

They selected random training examples, fed the model a prompt (MuLan tokens + varying lengths of semantic token prefix), and compared generated continuations against the actual training data. The results: exact token matches remained below 0.2%, even with a 10-second prompt. Approximate matches (using an optimal transport measure between token histograms) reached about 1% — but closer inspection revealed these were low-diversity sequences (repeating patterns, drones) with an average entropy of just 1.0 bit versus 4.6 bits for normal sequences.

The second acoustic stage introduces further diversity even when semantic tokens match exactly, making literal reproduction of training audio extremely unlikely in practice.

MusicLM generates longer sequences by advancing with a sliding window: it uses the last 15 seconds as a prefix to generate the next 15 seconds, always conditioning on the same text description. This approach produces coherent audio over several minutes — far beyond the 30-second training window.

A variant called story mode changes the text description every 15 seconds. The model produces smooth transitions that maintain tempo consistency while shifting musical context — from "gentle piano" to "energetic drums" — creating a narrative arc through sound.

Several limitations are inherited from MuLan. The model misunderstands negations — "music without drums" may still produce drums. It also does not adhere to precise temporal ordering described in text, such as "start with piano, then add guitar after 10 seconds."

The MCC metric has a self-referential bias because it uses MuLan itself, which favors MusicLM since MuLan is part of its pipeline. Vocals quality remains limited, and the model does not generate lyrics. High-level song structure (intro, verse, chorus) is also not explicitly modeled.

MusicLM's deepest contribution is not any single component but the architectural recipe: use a pre-trained joint embedding model to sidestep paired data, decompose generation into hierarchical stages aligned with token granularity, and let each stage specialize. This recipe has since been adapted for video, 3D generation, and multimodal synthesis — confirming that the hierarchical token paradigm extends far beyond music.