A Time Series Is Worth 64 Words: Long-Term Forecasting with Transformers

Before PatchTST, most Transformer-based time series models — Informer, Autoformer, FEDformer — treated each time step as a separate input token. This is equivalent to tokenizing English text one character at a time: it destroys local meaning.

A single temperature reading of 23.5°C carries almost no semantic information by itself. But a group of readings [22.1, 22.8, 23.2, 23.5] tells a story: the temperature is steadily rising. That local pattern — the trend, the shape — is exactly what point-wise tokenization loses.

Worse, the computational cost is devastating. The Transformer's self-attention has O(N²) time and memory complexity, where N is the number of input tokens. For a look-back window of 336 time steps, that's 336² = 112,896 attention entries. Training on the Traffic dataset (862 channels) with point-wise tokens took over 10,000 seconds — impractical for real-world deployment.

The coup de grâce came from Zeng et al. (2022): a single-layer linear model called DLinear outperformed every Transformer variant on common benchmarks. This wasn't a minor win — it was a fundamental challenge to the usefulness of Transformers for time series.

The patching idea comes directly from computer vision. Vision Transformer (ViT) splits an image into 16×16 pixel patches and treats each patch as a token. PatchTST applies the same principle to time series: instead of one point per token, group P consecutive time steps into a single patch token.

Imagine a long ribbon of temperature readings. You cut this ribbon into segments of length P = 16, with a stride S = 8 (so patches overlap by 8 steps). Each segment becomes one input token to the Transformer. This has three immediate benefits:

1. Local semantics are preserved. Each patch captures a local shape — a rising trend, a periodic oscillation, a plateau. This is the "word" that carries meaning, versus the meaningless "letter" of a single data point.

2. Quadratic attention cost drops dramatically. With stride S = 8 and look-back window L = 336, the number of tokens drops from 336 to roughly 336/8 ≈ 42. The attention cost drops from 336² to 42² — a 64× reduction. On the Traffic dataset, training time dropped from 10,040 seconds to 464 seconds — a 22× speedup.

3. The model can see longer history. Because each token now summarizes P time steps, the same 42 tokens can cover 336 steps instead of just 42. You can even extend to 512 steps (64 patches) within the same computational budget, giving the model more context for long-term forecasting.

A multivariate time series has M channels (variables) — for example, 862 road sensors in the Traffic dataset, or 21 weather measurements (temperature, humidity, pressure...). Previous Transformer models used channel-mixing: at each time step, they stacked all M values into a single vector and projected it into the embedding space. This forces the model to learn cross-channel correlations at every layer.

Think of it like an orchestra: channel-mixing forces all instruments to read from the same sheet of music, blending their sounds before understanding each one individually.

PatchTST takes the opposite approach: channel-independence. Each univariate series is fed through the Transformer backbone separately. The Transformer weights are shared across all channels (like a shared teacher), but each series generates its own attention maps. The final predictions from all channels are concatenated at the output.

Why does this work better? Three reasons:

• Adaptability: Different channels may have very different temporal patterns. In the Electricity dataset, some series are smooth and periodic while others are noisy. With channel-independence, each series learns its own attention pattern.
• Less overfitting: Channel-mixing models need to learn M×M cross-channel interactions, which requires much more data. Channel-independent models converge faster with less training data.
• Noise isolation: If one channel is noisy, channel-mixing projects that noise into all other channels via the embedding. Channel-independence keeps noise contained.

PatchTST's architecture is deliberately simple — it uses the vanilla Transformer encoder with no custom attention mechanisms. The innovation is entirely in how the input is prepared (patching + channel-independence), not in the Transformer itself.

The forward process for each univariate series goes through these stages:

1. Instance Normalization. Each series is normalized to zero mean and unit standard deviation. This handles distribution shift between training and testing data. The mean and standard deviation are saved and added back to the output predictions.

2. Patching. The normalized series is divided into N overlapping patches of length P with stride S, creating a patch matrix of shape P × N.

3. Projection + Position Embedding. Each patch is linearly projected into the Transformer's latent space of dimension D via a trainable projection matrix. Learnable position embeddings are added to preserve temporal order.

4. Transformer Encoder. The projected patches pass through standard encoder layers: multi-head attention, batch normalization, feed-forward network with GELU activation, and residual connections.

5. Flatten + Linear Head. The encoder output is flattened and a linear layer maps it to the prediction horizon T.

Default hyperparameters: 3 encoder layers, 16 attention heads, D = 128, FFN dimension = 256, dropout = 0.2, patch length P = 16, stride S = 8.

PatchTST isn't just a supervised forecasting model — it also supports self-supervised pre-training via masked patch reconstruction, inspired by masked autoencoders in computer vision and masked language modeling in NLP.

The idea is straightforward: randomly select 40% of the non-overlapping patches and replace them with zeros. The model is trained to reconstruct the original values of those masked patches using MSE loss. Because entire patches are masked (not individual points), the model can't simply interpolate from neighboring time steps — it must learn higher-level temporal representations.

Why is this important? Think of it as reading a paragraph with entire words blacked out rather than individual letters. You need to understand sentence structure and context to fill in the missing words — mere letter-level guessing won't work.

After pre-training, the model can be fine-tuned for forecasting in two ways:

• Linear probing: Freeze the encoder, train only the prediction head for 20 epochs. Already competitive with supervised training from scratch.
• End-to-end fine-tuning: First linear probing for 10 epochs, then unfreeze everything for 20 more epochs. This achieves the best results — even beating supervised training on large datasets like Weather, Traffic, and Electricity.

Crucially, channel-independence makes transfer learning natural: because each channel is processed independently with shared weights, you can pre-train on a dataset with 321 channels (Electricity) and fine-tune on one with 21 channels (Weather) — the number of variables doesn't need to match.

PatchTST was evaluated on 8 popular benchmarks: Weather, Traffic, Electricity, ILI, and the four ETT datasets. Two variants were tested: PatchTST/42 (look-back L = 336, 42 patches — fair comparison to DLinear) and PatchTST/64 (L = 512, 64 patches — pushing performance further).

Against Transformer baselines: PatchTST/64 achieved an overall 21.0% reduction in MSE and 16.7% reduction in MAE compared to the best Transformer-based models (FEDformer, Autoformer, Informer). This is an enormous margin.

Against DLinear: PatchTST outperformed DLinear on the large datasets — Weather, Traffic, and Electricity — which are the most reliable benchmarks because they have enough data to resist overfitting. On Weather with prediction horizon 96, PatchTST/64 achieved MSE 0.149 vs DLinear's 0.176. On Traffic at horizon 96: 0.360 vs 0.410.

Self-supervised boost: On large datasets, masked pre-training followed by fine-tuning beat supervised training from scratch. On Electricity at horizon 96: self-supervised MSE 0.126 vs supervised 0.130.

Transfer learning: Pre-training on Electricity and fine-tuning on Weather achieved MSE 0.145 at horizon 96 — better than supervised training from scratch (0.152) despite the domains being completely different.

Representation learning comparison: Against dedicated self-supervised methods (TS2Vec, TNC, TS-TCC, BTSF), PatchTST achieved 34–49% improvement in MSE on ETTh1.

The paper systematically tested every design choice. Removing patching while keeping channel-independence raised MSE on Weather from 0.152 to 0.164. Removing channel-independence while keeping patching raised it to 0.168. Removing both (original TST model) raised it to 0.177. Both pieces matter, and their combination is better than either alone.

Varying patch length: MSE remained stable across patch lengths from 4 to 40, showing robustness. Lengths between 8 and 16 were generally optimal.

Varying look-back window: PatchTST consistently improved with longer windows — unlike FEDformer, Autoformer, and Informer, which often degraded or plateaued when the look-back window exceeded 96. This is direct evidence that patching allows the model to absorb more historical information.

Instance normalization: Removing it degraded performance, especially on datasets with distribution shift (ILI, Traffic). But even without it, PatchTST still outperformed other Transformers — confirming that the gains come primarily from patching and channel-independence.