Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

Before Tacotron 2, building a TTS system meant stitching together a long chain of hand-crafted modules:

• A text analyzer that converts raw text into linguistic features (phonemes, stress patterns, part-of-speech tags).
• A duration model that predicts how long each phoneme lasts.
• An F0 predictor that decides the pitch contour.
• A vocoder that synthesizes a waveform from these acoustic features.

Each module was engineered with domain-specific rules and signal-processing tricks. Errors at any stage — a wrong phoneme, an unnatural duration — would cascade downstream, and the speech would sound stilted. Two breakthroughs had appeared but remained incomplete:

WaveNet (2016) showed that a neural vocoder could generate astonishingly natural audio, but it needed hand-crafted linguistic features as input — the same fragile pipeline, just with a better last stage.

Tacotron 1 (2017) showed that a sequence-to-sequence model could predict spectrograms directly from characters, but it used the Griffin-Lim algorithm to convert spectrograms to audio, introducing phase-estimation artifacts that made the speech sound hollow.

The key insight is choosing the right intermediate representation — the contract between the text-understanding network and the audio-rendering network. Tacotron 2 uses 80-channel mel spectrograms computed from 50 ms frames with a 12.5 ms hop.

A mel spectrogram is a time-frequency picture of sound, but warped to match how the human ear actually perceives pitch. Low frequencies get more resolution (because our ears are most sensitive there), and high frequencies get compressed. Think of it as a heat map of sound: the x-axis is time, the y-axis is frequency (on the mel scale), and the color intensity is energy.

Why mel spectrograms instead of raw linguistic features? Three reasons:

• They are compact. 80 channels at 12.5 ms hops vs. 16,000+ raw audio samples per second — a 200× compression that still captures pitch, timbre, and rhythm.
• They are learnable. No hand-engineered phoneme dictionaries or prosody rules needed.
• They simplify WaveNet. Because the spectrogram already encodes long-range acoustic structure, WaveNet no longer needs an enormous receptive field to "figure out" what word it's in the middle of — it can focus on rendering fine audio detail.

The spectrogram prediction network is a sequence-to-sequence model with three main parts: an encoder, a location-sensitive attention mechanism, and an autoregressive decoder.

The encoder takes a sequence of characters, embeds each into a 512-dimensional vector, then passes them through 3 convolutional layers (512 filters, kernel size 5) with batch normalization and ReLU. These convolutions capture local character context — each filter sees a character and its two neighbors on each side. Finally, a bidirectional LSTM with 256 units per direction produces the encoded representation. Think of the encoder as a reader that understands both the individual letters and how they flow together into words and phrases.

Location-sensitive attention extends the standard additive attention mechanism by also looking at where the model was attending in the previous step. This prevents the common failure modes of attention in TTS: repeating a word (attention gets stuck) or skipping a word (attention jumps too far). The attention history acts like a bookmark that says "I've read up to here — move forward, don't repeat."

The decoder is autoregressive: at each step it predicts one frame of the mel spectrogram, then feeds that prediction back as input for the next step. The previous prediction passes through a pre-net (2 fully connected layers of 256 units with ReLU and dropout) that acts as an information bottleneck — it forces the model to learn a robust attention alignment rather than memorizing exact frame values. The pre-net output and attention context are concatenated and fed through 2 unidirectional LSTM layers (1024 units each). The LSTM output is used to predict the current mel frame and a stop token that signals end of utterance.

The decoder's raw mel predictions are reasonable but lack fine detail. A 5-layer convolutional post-net (512 filters, kernel size 5, with batch normalization and tanh on all but the last layer) predicts a residual — a correction to add to the initial prediction. This is a residual connection: instead of asking the post-net to predict the entire spectrogram from scratch, it only needs to learn the difference between "good enough" and "great."

This is exactly like an artist's workflow: the decoder does the rough sketch, and the post-net adds shading, highlights, and fine brushstrokes. The system trains with two loss terms: one on the raw decoder output and one on the post-net-enhanced output, so both the sketch and the refined version learn to be accurate.

The modified WaveNet vocoder takes the mel spectrogram and generates audio samples at 24 kHz — sample by sample, autoregressively. It uses the same core idea as the original WaveNet: stacked dilated causal convolutions that give each sample a large receptive field over past samples.

But there's a crucial simplification. The original WaveNet needed 30 dilated layers to achieve a receptive field large enough to capture long-range speech structure (what word is being said, what the intonation pattern is). Tacotron 2's WaveNet needs far fewer layers because the mel spectrogram already encodes that information. The vocoder just needs enough context to render smooth, natural-sounding audio at the sample level.

The paper's ablation studies confirmed this: reducing the receptive field by a factor of 25 still produced high-quality audio, but eliminating dilated convolutions entirely degraded quality significantly. There's a "sweet spot" — the vocoder needs some local context at the waveform level, but the heavy lifting of linguistic and prosodic planning is already done by the spectrogram network.

A key design choice: the two networks are trained independently. The spectrogram network is trained first using teacher forcing — at each decoder step, the ground-truth previous mel frame is fed as input (instead of the model's own prediction). This stabilizes training and helps the attention learn clean, diagonal alignments.

Once the spectrogram network is trained, it generates mel spectrograms for the training data. These predicted (not ground-truth) spectrograms are used to train the WaveNet vocoder. Why not use ground-truth spectrograms? Because at inference time, WaveNet will see predicted spectrograms — which may have small imperfections. Training on predicted spectrograms teaches WaveNet to be robust to these imperfections, bridging the train-test gap.

The loss function for the spectrogram network is straightforward mean squared error (MSE) on both the raw decoder output and the post-net output, plus binary cross-entropy on the stop token. No adversarial training, no complex perceptual losses — just MSE, and the system achieves near-human quality.

The authors systematically removed or modified components to measure their contribution:

• Mel spectrogram vs. linguistic features as WaveNet input. Mel spectrograms won decisively — they produced more natural speech without requiring any text processing pipeline.
• WaveNet receptive field size. Reducing the receptive field by 25× (from 256 ms to about 10 ms) barely affected quality. But removing dilated convolutions entirely (receptive field drops to ~0.5 ms) caused significant degradation.
• Post-net contribution. The 5-layer convolutional post-net consistently improved spectrogram quality by learning to add fine spectral detail that the autoregressive decoder misses.
• The system achieved MOS 4.53 — virtually indistinguishable from professional recordings at MOS 4.58.

The full Tacotron 2 pipeline flows from left to right: characters → encoder → attention → decoder → post-net → mel spectrogram → WaveNet → audio waveform. The mel spectrogram is the clean interface: the spectrogram network can be improved independently (better attention, better encoder) and the vocoder can be swapped (WaveNet → WaveRNN → HiFi-GAN) without retraining the other half.

This modular design is why the mel-spectrogram interface became the industry standard. Every subsequent TTS system — FastSpeech, VITS, Grad-TTS — predicts mel spectrograms as an intermediate step. The idea of separating "what to say" from "how to render" turned out to be one of the most influential design decisions in speech synthesis.