Méthode Générale pour la Résolution des Systèmes d'Équations Simultanées

In Cauchy's time, astronomers wrestled with systems of equations arising from planetary motion — often six or more unknowns. Direct algebraic methods (like Gaussian elimination) worked beautifully for small systems, but their computational cost grew rapidly with size:

• Direct methods require on the order of n³ arithmetic operations, where n is the number of unknowns. For n = 6, that's already 216 operations — done by hand.

• No computers existed. Every multiplication was done with pen and paper. A method that required fewer total operations, even if approximate, was desperately needed.

• Nonlinear systems — where variables multiply each other or appear inside trigonometric functions — had no general solution procedure at all.

Cauchy's insight was that you don't need to solve the system in one shot. You can creep toward the solution by repeatedly reducing a single scalar quantity: the value of a function that measures how far you are from the answer.

Cauchy's method starts from one critical observation from calculus: the gradient of a function at a point tells you the direction of steepest ascent. Therefore, the negative gradient tells you the direction of steepest descent.

Here is the full procedure:

1. Start at an initial guess $x_0$.
2. Compute the gradient $\nabla f(x)$ — the vector of all partial derivatives.
3. Take a step in the opposite direction: $x_{t+1} = x_t - \gamma \nabla f(x_t)$.
4. Repeat until the function value stops changing significantly.

The step size $\gamma$ (called the learning rate in modern machine learning) controls how far you move per step. Too large and you overshoot; too small and you crawl. Cauchy recognized this tension and proposed choosing $\gamma$ to minimize f along the descent direction — what we now call a line search.

Before the formula: the goal of each iteration is to find a new point $x_{t+1}$ where the function value is lower than at $x_t$. The strategy is to move in the direction that decreases f the fastest — the negative gradient. The remaining question is how far to move.

Think of it as a compass rule: the gradient says "uphill is that way," so you turn 180° and walk. The step size $\gamma$ decides whether you take a cautious shuffle or a bold stride. Cauchy proposed choosing $\gamma$ optimally at each step by minimizing $f(x_t - \gamma \nabla f(x_t))$ as a function of $\gamma$ — an exact line search. Modern deep learning typically uses a fixed or scheduled $\gamma$ instead, because exact line search is too expensive with millions of parameters.

The mathematical guarantee behind gradient descent comes directly from the Taylor expansion. For a small enough step size $\gamma > 0$:

$f(x - \gamma \nabla f) \approx f(x) - \gamma \|\nabla f\|^2$

Since $\|\nabla f\|^2 \geq 0$, every step decreases (or at worst holds) the function value. The function keeps falling, step after step, until the gradient becomes zero — meaning you have reached a point where no direction leads downhill. That is a stationary point: a local minimum, a saddle point, or (if you're lucky) the global minimum.

Crucially, Cauchy showed that this works regardless of how many variables the function has. The gradient always exists for smooth functions, and the negative gradient always points downhill. The method scales to any dimensionality.

Gradient descent has a well-known pathology: on functions shaped like a narrow valley (technically, when the eigenvalues of the Hessian matrix are very different), the method zig-zags back and forth across the valley instead of sliding along it.

This happens because the steepest local direction at the valley wall points across the valley, not along it. Each step overshoots the narrow axis and lands on the opposite wall. The iterates oscillate, making slow progress.

This zig-zag problem motivated 170 years of improvements: conjugate gradients (1952), Newton's method with Hessian, momentum (1964), and modern adaptive optimizers like Adam (2014). All of them modify Cauchy's basic recipe to handle ill-conditioned landscapes better — but the core insight remains the same: use the gradient to decide where to step next.

Cauchy's original method spawned an entire family of optimization algorithms, each addressing one of its limitations. The genealogy connects directly to every optimizer used in deep learning today:

The convergence speed depends on the function's shape. For a convex quadratic $f(x) = \frac{1}{2} x^T A x$, the error at step $t$ satisfies:

$f(x_t) - f(x^*) \leq \left(\frac{\kappa - 1}{\kappa + 1}\right)^{2t} (f(x_0) - f(x^*))$

where $\kappa = \lambda_{\max} / \lambda_{\min}$ is the condition number of $A$ — the ratio of its largest eigenvalue to its smallest. The larger $\kappa$, the more elongated the contours and the worse the zig-zagging. For a well-conditioned function ($\kappa \approx 1$), convergence is fast. For an ill-conditioned function ($\kappa \gg 1$), convergence is painfully slow.

This is the fundamental trade-off: gradient descent is general (works on any smooth function) but first-order (uses only the gradient, not curvature information). Methods that use second-order information (Hessian) converge faster but are far more expensive per step.

Every neural network training loop is a direct descendant of Cauchy's algorithm:

1. Forward pass — compute the loss $L(\theta)$ (how wrong the predictions are).
2. Backward pass — compute $\nabla_\theta L$ using backpropagation (the chain rule).
3. Update — $\theta \leftarrow \theta - \gamma \nabla_\theta L$ (Cauchy's step).

The loss function $L$ plays the role of Cauchy's $f$, the model parameters $\theta$ replace his variables $(x_1, \ldots, x_n)$, and the learning rate $\gamma$ is his step size — except modern networks have billions of parameters instead of six, and the gradients come from mini-batches (stochastic gradient descent) rather than the full dataset.

The entire deep learning revolution rests on three legs: Cauchy's gradient descent (1847), Rumelhart/Hinton/Williams' backpropagation (1986) to compute the gradient efficiently, and GPU hardware to execute the loop fast enough. Remove any one of these, and modern AI does not exist.