Natural Gradient Works Efficiently in Learning

When we train a neural network, we adjust its parameters $\theta$ to minimize a loss function $L(\theta)$. The standard approach — gradient descent — computes $\nabla L(\theta)$ and steps in the opposite direction:

$\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$

This implicitly assumes parameter space is flat — that a step of size $\epsilon$ in any direction costs the same. But for probabilistic models (which neural networks are), the parameter space is a curved manifold: changing $\theta_1$ by 0.01 might barely shift the output distribution, while changing $\theta_2$ by the same amount could flip the entire prediction. The gradient, blind to this geometry, wastes its step budget on the insensitive directions and under-steps the sensitive ones.

Here is the practical consequence: training gets stuck on plateaus — flat regions of the loss surface where the ordinary gradient is tiny even though a nearby valley exists. The model crawls for thousands of iterations, then suddenly drops once it escapes the plateau. This plateau phenomenon is especially severe in multilayer perceptrons trained with backpropagation.

Amari's key insight comes from information geometry: the parameter space of a statistical model is not a flat plane — it is a Riemannian manifold whose curvature is described by the Fisher information matrix $F(\theta)$.

The Fisher information matrix measures how sensitive the model's output distribution $p(x; \theta)$ is to small changes in each parameter. Formally, it is the expected outer product of the score function (the gradient of the log-likelihood):

$F(\theta) = \mathbb{E}\left[\nabla \log p(x;\theta) \, \nabla \log p(x;\theta)^\top\right]$

Think of $F(\theta)$ as a local ruler that tells you the "real" distance between two nearby parameter settings — not in centimeters on the parameter axis, but in how much the output distribution actually changes. Two parameter settings that produce nearly identical distributions are "close" in this ruler, even if their numerical values differ a lot.

The idea is simple once you see it: instead of stepping in the direction of the ordinary gradient $\nabla L$, step in the direction that decreases the loss fastest per unit of distributional change. To find this direction, we "undistort" the gradient by multiplying it by $F^{-1}$:

The update rule becomes:

$\theta_{t+1} = \theta_t - \eta \, F(\theta_t)^{-1} \nabla L(\theta_t)$

Compare this with Newton's method, which uses $H^{-1} \nabla L$ where $H$ is the Hessian. The natural gradient replaces the Hessian with the Fisher information matrix — which is always positive semi-definite (unlike the Hessian), making the update always a descent direction. It's a second-order method with guaranteed stability.

Suppose you write a model with parameters $\theta$, and your colleague writes the same model with a different parameterization $\phi = g(\theta)$. Ordinary gradient descent on $\theta$ and on $\phi$ will take completely different paths — the optimization depends on an arbitrary naming choice.

The natural gradient is invariant to this reparameterization. No matter how you write the model, the natural gradient always moves in the same direction in distribution space. This means the algorithm's behavior depends on the model itself, not on the accident of how you chose to write down the parameters.

Mathematically, under $\phi = g(\theta)$, the Fisher matrix transforms as a proper metric tensor: $F_\phi = J^\top F_\theta J$ where $J$ is the Jacobian of $g$. This transformation exactly cancels the coordinate change in the gradient, leaving the natural gradient unchanged.

One of the paper's strongest results is that natural gradient online learning is Fisher efficient. This is a technical term from statistics with a precise meaning: the estimator $\theta_k$ produced after $k$ online updates achieves the lowest possible variance among all unbiased estimators — matching the Cramér-Rao bound.

In practical terms: even though natural gradient online learning sees each data point only once (as in stochastic gradient descent), it extracts as much information from each sample as if it had collected all the data in a batch and run maximum likelihood estimation. No online algorithm can do better asymptotically.

The key condition is a decreasing learning rate $\eta_t = 1/t$, which satisfies the Robbins-Monro conditions for stochastic approximation convergence.

The plateau phenomenon is one of the most frustrating aspects of training multilayer perceptrons with backpropagation: the loss flattens for hundreds or thousands of iterations even though the model is far from a minimum.

Amari showed that these plateaus often arise because the ordinary gradient shrinks in regions where the Fisher curvature is low — the parameter space is "stretched" so that small gradients correspond to large distributional changes. The natural gradient compensates for this stretching by multiplying with $F^{-1}$, effectively "amplifying" the step in the stretched directions.

The intuition is like driving through fog: ordinary gradient descent slows down because it can't see far ahead (small gradient magnitude). But the natural gradient knows the road is straight even through the fog — it maintains speed because it measures progress in terms of how much the output distribution changes, not how big the gradient vector looks.

The paper applies the natural gradient framework to three concrete settings:

Multilayer perceptrons — For a network with weights $W$ and a probabilistic output $p(y|x; W)$, the Fisher matrix captures how parameter correlations interact with the data distribution. The natural gradient update decorrelates these interactions, leading to faster convergence and reduced plateau effects.

Blind source separation — Given mixed signals $x = As$ where $A$ is an unknown mixing matrix and $s$ contains independent sources, the natural gradient on the space of separating matrices $W$ yields an elegant update: $\Delta W \propto (I + \varphi(y)y^\top) W$ where $\varphi$ is a nonlinear activation. The matrix $W$ appearing on the right makes the update equivariant — it adapts to the current scale of the solution automatically.

Blind deconvolution — Extending the idea to linear dynamical systems with convolutive mixing, the natural gradient produces updates that are similarly efficient and scale-free.

The natural gradient has a practical bottleneck: computing and inverting the Fisher information matrix $F(\theta)$. For a model with $n$ parameters, $F$ is an $n \times n$ matrix, and inverting it costs $O(n^3)$. Modern deep networks have millions or billions of parameters — directly computing $F^{-1}$ is impossible.

This is where the paper's legacy truly begins. The computational challenge sparked decades of research into approximations of the natural gradient:

• Diagonal approximations — keep only the diagonal of $F$. This is essentially what AdaGrad, RMSProp, and the second moment of Adam do. Cheap but throws away parameter correlations.
• Block-diagonal approximations — compute $F$ within each layer independently. Captures some structure at manageable cost.
• Kronecker-factored approximations (K-FAC) — the Fisher for a neural network layer can be approximated as a Kronecker product of two smaller matrices (one from the input activations, one from the output gradients). This was the breakthrough that made natural gradient practical for deep learning.

The natural gradient paper is the ancestor of an entire family of optimization methods. Once you understand this connection, modern optimizers stop looking like unrelated tricks and start revealing their common DNA:

Adam (2014) — maintains running averages of the first moment (gradient) and second moment (squared gradient). The second moment acts as a diagonal approximation to the Fisher, scaling each parameter's learning rate by the inverse of its historical gradient magnitude.

K-FAC (Martens & Grosse, 2015) — the most direct descendant. Approximates the Fisher with Kronecker products, then performs the natural gradient update. Achieves near second-order convergence at near first-order cost.

TRPO (Schulman et al., 2015) — in reinforcement learning, the policy is a probability distribution over actions. TRPO constrains policy updates to a trust region defined by the Fisher information, ensuring each update doesn't change the policy too much. This is the natural gradient with a KL divergence constraint instead of a fixed step size.