Are Transformers Effective for Time Series Forecasting?

Transformers conquered NLP and computer vision by finding semantic relationships between tokens — which word relates to which, which image patch connects to which. Self-attention excels at this because it treats every pair of elements equally, regardless of position. But this very property — permutation invariance — is a fundamental mismatch for time series.

In a time series, the order of data points is the signal. Stock prices, temperature readings, electricity consumption — they are continuous flows where position carries meaning. Self-attention, by contrast, is "anti-ordering": it sees every pair but ignores their sequential relationship. While positional encodings partially compensate, the fundamental architecture was designed for semantic correlation, not temporal dynamics.

The authors posed a sharp question: if we strip away all the architectural complexity and use the simplest possible model with direct multi-step forecasting, will the Transformer's supposed advantages survive?

Before comparing architectures, there is a critical distinction that previous papers glossed over: how multi-step predictions are generated.

Iterated Multi-Step (IMS) forecasting trains a single-step predictor and applies it recursively — predict step 1, feed it back, predict step 2, and so on. This is the autoregressive approach used by RNNs and classical methods. It works well for short horizons, but errors compound with each step. By the time you reach step 720, the accumulated noise can be devastating.

Direct Multi-Step (DMS) forecasting predicts all future steps at once. The model takes the look-back window and outputs the entire forecast horizon in a single pass. No error accumulation, because there is no feedback loop.

The key insight: all existing Transformer-based models (Informer, Autoformer, FEDformer) use DMS forecasting with non-autoregressive decoders. But the baselines they compared against — RNNs, DeepAR, classical methods — all use IMS. This is not a fair comparison. The Transformers' supposed advantage might come entirely from the forecasting strategy, not the architecture.

The DLinear architecture is built on two observations. First, a single linear layer is the simplest possible way to aggregate historical information for future prediction — it directly maps L input time steps to T output time steps through a learned weight matrix. Second, decomposing a time series into trend and remainder (as Autoformer showed) is a model-agnostic technique that can boost any architecture.

DLinear combines both: it applies a moving average with kernel size 25 to extract the trend component, subtracts it to get the remainder, then feeds each through its own single-layer linear network. The two outputs are summed to produce the final forecast.

That is the entire model. No attention heads, no multi-layer encoders, no positional embeddings, no feed-forward networks. Just decomposition plus two matrix multiplications.

A raw time series is a superposition of multiple dynamics operating at different scales. Think of it like ocean waves: there is the tide (the slow, long-term trend) and the ripples (the fast, seasonal fluctuations). Predicting both at once with a single model forces it to simultaneously learn slow dynamics and fast oscillations — a harder optimization problem.

The decomposition strategy borrowed from Autoformer applies a moving average to smooth out the ripples and extract the tide. The difference between the raw signal and the smoothed tide is the remainder — the seasonal or cyclical component.

The ablation study confirmed that decomposition helps most when there is a clear trend in the data (Exchange-Rate, ILI, ETT datasets) and provides less benefit for trendless data (Traffic, Weather). This makes intuitive sense: if there is no tide to separate, the decomposition step is redundant.

The results were striking. DLinear outperformed all Transformer-based methods in most settings across 9 benchmark datasets: ETTh1, ETTh2, ETTm1, ETTm2, Traffic, Electricity, Weather, Exchange-Rate, and ILI.

The margins were not small. On Exchange-Rate, DLinear beat the state-of-the-art FEDformer by roughly 50% in MSE. On Weather, the improvement exceeded 25%. Even the naive "Repeat-C" baseline (just repeat the last value) outperformed all Transformers on Exchange-Rate by over 30%.

Efficiency was equally dramatic. DLinear requires only 0.04G MACs compared to Informer's 3.93G, Autoformer's 4.41G, and FEDformer's 4.41G. Inference runs in 0.4 ms versus 40–164 ms for the Transformer variants. Parameters: 140K vs 14–21 million.

A model that truly captures temporal relations should improve when given more historical data. If you can see 30 days instead of 4, you should forecast better — period.

The authors tested all models with look-back windows from 24 to 720 steps. The results revealed a damning pattern: Transformer-based models either stagnated or got worse with larger windows. Informer, Autoformer, and Pyraformer showed flat or increasing MSE as the look-back window grew. FEDformer was slightly better but still plateaued.

DLinear, by contrast, consistently improved with longer look-back windows. On Traffic, its MSE dropped from 0.65 (L=24) to around 0.41 (L=720). On Electricity, from 0.23 to 0.13. The improvement was dramatic and consistent.

This is perhaps the paper's most devastating finding. It suggests that Transformers are not actually learning temporal dependencies from the data — they are merely mapping inputs to outputs without exploiting the temporal structure that additional history provides.

Because DLinear is a linear model, its learned weights W ∈ ℝᵀˣᴸ directly reveal which input time steps influence which output steps. This is a level of interpretability that Transformers, with their multi-headed attention layers, cannot match.

For the Electricity dataset (hourly granularity), the remainder weights showed clear diagonal bands spaced 24 steps apart — the model learned daily periodicity. The value at output step 1 depends most on input steps 1, 25, 49, 73 — i.e., the same hour on previous days. The Traffic weights revealed both daily (24-step) and weekly (168-step) periodicity.

For Exchange-Rate (no periodicity), the trend weights showed higher values near the most recent time steps — the model learned that recent values matter most for predicting aperiodic financial data. This interpretability is not just a nice feature; it provides validation that the model is learning sensible temporal patterns.

The authors systematically removed components from existing Transformer models to understand what actually contributes to performance. When self-attention layers and auxiliary designs were stripped from Informer step by step — gradually converting it into a linear model — performance improved at each step. The closer the model got to being a plain linear layer, the better it forecasted.

Embedding strategies also told a revealing story. Informer collapsed without positional embeddings (it uses single time steps as tokens), while FEDformer — which relies more on Fourier transforms than self-attention — was more robust to removing embeddings. This suggests that FEDformer's strength comes from its frequency analysis, not from the Transformer architecture itself.

Even training data size did not save the Transformers. On the Traffic dataset, reducing training data from 17,544 hours to 8,760 hours (one year) actually improved predictions for all models — contradicting the argument that Transformers would shine with more data.

The authors were transparent about DLinear's limitations. As a single-layer linear model, it has limited capacity — it cannot capture nonlinear dynamics, change points, or complex multi-scale interactions. It was designed as a baseline, not as the final answer to time series forecasting.

Later work, notably PatchTST, showed that Transformers can work for time series when the tokenization is redesigned. Instead of treating each time step as a token (which fragments the temporal signal), PatchTST groups consecutive time steps into patches — similar to how Vision Transformers handle images. This preserves local temporal structure before self-attention operates, and PatchTST consistently outperformed DLinear.

The debate DLinear started is still ongoing: simplicity as a baseline forced the community to be more rigorous about what Transformers actually contribute, leading to better Transformer designs rather than abandoning them entirely.