DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks

A large retailer may track demand for 500,000 products across 1,000 stores. Classical methods like ARIMA or Exponential Smoothing fit a separate model to each individual time series. This creates three serious problems:

Scale: fitting and maintaining half a million models is operationally expensive.

Sparsity: many products have short or intermittent sales histories, giving individual models too little data to learn meaningful patterns.

Cold start: a brand-new product has zero history. Individual models literally cannot start.

And even when classical models work, they typically output a single predicted value — a point forecast. But business decisions need uncertainty: should I order 100 or 200 units? The answer depends on how confident the forecast is. What's needed is a full probability distribution over future values.

DeepAR's two key insights are:

1. Global model: Instead of fitting one model per time series, train a single recurrent neural network on all related time series simultaneously. The network learns shared patterns — seasonality, trend shapes, covariate effects — that transfer across series. Series-specific behavior is captured through learned item embeddings and per-series scale normalization.

2. Probabilistic output: Instead of predicting a single number, the network outputs the parameters of a probability distribution at each time step. For real-valued data, it outputs the mean μ and standard deviation σ of a Gaussian. For count data, it outputs the parameters of a negative binomial distribution. This means DeepAR produces a full predictive distribution, not just a point estimate.

The architecture is autoregressive: at each time step, the model conditions on the previous observation and covariates, feeds them through an LSTM, and outputs distribution parameters. Think of it as a chain: each link (time step) uses the previous link's output to predict the next.

At each time step t, DeepAR performs four operations:

Step 1 — Gather inputs: Collect the previous observation z(t−1), the covariates x(t) (time features like day-of-week, month, holidays, plus any item-specific covariates), and the previous hidden state h(t−1).

Step 2 — LSTM forward pass: Feed these inputs into a multi-layer LSTM, which produces a new hidden state h(t). The hidden state is the network's compressed memory — like a summary of everything it has seen so far for this series.

Step 3 — Distribution parameters: Pass h(t) through two fully connected layers to produce distribution parameters. For a Gaussian likelihood: one layer outputs μ(t) (the predicted mean) and another outputs σ(t) (the predicted standard deviation, passed through a softplus to ensure positivity).

Step 4 — Likelihood: The likelihood ℓ(z(t) | θ(t)) measures how probable the actual observed value z(t) is under the predicted distribution. During training, we maximize this likelihood (equivalently, minimize the negative log-likelihood).

DeepAR is trained by maximizing the log-likelihood of the observed data. The loss function is:

During training, the model uses teacher forcing: at each time step, it receives the actual previous observation z(t−1) as input, not its own prediction. This is like a student practicing with an answer key — they always see the correct previous answer, which stabilizes and accelerates learning.

However, during inference there are no future observations to feed. The model must use its own sampled predictions as inputs — creating a fundamentally different regime. This train/test gap is a known challenge of autoregressive models, and DeepAR mitigates it through careful data augmentation (random window selection during training).

At inference time, DeepAR generates forecasts through Monte Carlo sampling. The process is:

1. Run the LSTM through the known history (conditioning range) to build up the hidden state — exactly like during training.

2. At the first prediction step, sample a value from the predicted distribution: z̃(t₀) ~ Gaussian(μ(t₀), σ(t₀)).

3. Feed that sampled value back as input to the next step (instead of the true observation, which we don't have).

4. Repeat for each step in the prediction horizon.

5. Do the entire process K times (e.g., K=100) to get K sample paths.

Each sample path is a plausible future trajectory. Together, the K paths form an empirical approximation of the full predictive distribution. From these paths you can compute any quantile (P10, P50, P90), prediction intervals, or expected values — whatever your downstream decision system needs.

Real-world time series datasets have two practical challenges that DeepAR addresses with elegant solutions:

Scale normalization: Product A might sell 10,000 units/day while Product B sells 2. If the network sees raw values, it will be dominated by high-volume series. DeepAR divides each series by a scale factor v_i (typically the mean of the absolute values in the conditioning range). The network learns patterns in the normalized space, and predictions are scaled back. This lets a single network handle series spanning many orders of magnitude.

Item embeddings: Each time series gets a learnable embedding vector — a compact fingerprint that captures series-specific characteristics the network can't infer from covariates alone. These embeddings are fed as additional input to the LSTM. For new items (cold start), the embedding starts at a default value and quickly adapts as data arrives. Think of it as giving each product a name tag that the network learns to read.

A powerful feature of DeepAR is its modularity around the likelihood function. The network architecture stays the same — only the output layer and loss function change:

Gaussian likelihood — for real-valued data (temperature, prices, energy consumption). The network outputs μ and σ, and the loss is the negative log of the Gaussian density evaluated at the true observation.

Negative binomial likelihood — for count data (product demand, website clicks, event counts). This distribution naturally handles the over-dispersion and zero-inflation common in count data, unlike a Gaussian which can predict negative values.

Other options are equally plug-and-play: Beta for data in [0, 1], Bernoulli for binary outcomes, or mixtures for complex distributions. The only requirement is that you can evaluate the log-likelihood and take gradients — which any standard parametric distribution satisfies.

DeepAR was evaluated on several large-scale forecasting datasets spanning retail demand, automotive parts, and electricity consumption:

~15% improvement in weighted quantile loss over state-of-the-art baselines including ARIMA, ETS (Exponential Smoothing), and Seasonal Naïve across most datasets.

Electricity dataset (370 hourly series): DeepAR significantly outperformed classical methods, especially at longer horizons where capturing cross-series patterns matters most.

Parts dataset (intermittent demand): The negative binomial likelihood excelled here, correctly modeling the many-zeros-then-a-spike pattern that Gaussian methods handle poorly.

Traffic dataset (862 road occupancy series): DeepAR captured daily and weekly seasonality patterns shared across sensors, producing well-calibrated uncertainty bands.

The key finding: a single global model consistently outperformed a basket of individually-fitted expert models — while requiring far less manual configuration.

DeepAR can be understood through the lens of the encoder-decoder framework used in sequence-to-sequence models:

The conditioning range (historical observations) acts as the encoder: the LSTM reads through the known history, accumulating context into its hidden state. At the end of this phase, the hidden state is a compressed summary of everything the model knows about this series.

The prediction range acts as the decoder: starting from the encoder's final hidden state, the LSTM generates future values step by step, feeding each sampled prediction back as input. The key difference from machine translation is that both encoder and decoder share the same LSTM — there is no separate decoder network. This weight sharing is what makes DeepAR efficient and what enables it to seamlessly transition from reading history to generating forecasts.