An Information-Maximization Approach to Blind Separation and Blind Deconvolution

The classic setup: $n$ independent source signals $s_1, s_2, \ldots, s_n$ (voices at a party) are mixed by an unknown $n \times n$ mixing matrix $A$. What we observe is the mixture $\mathbf{x} = A\mathbf{s}$. Each sensor (microphone) gets a different linear combination of all sources.

The goal of blind source separation is to find an unmixing matrix $W$ such that $\mathbf{u} = W\mathbf{x} \approx \mathbf{s}$ — recovering the original sources without knowing $A$ or the distributions of the sources. The word "blind" means we work from the mixtures alone.

PCA can decorrelate the mixtures (remove linear correlations), but decorrelation only uses second-order statistics (covariances). Two signals can be uncorrelated yet statistically dependent — think of a signal and its square. True separation requires statistical independence, which involves all higher-order moments.

Consider two uniform source signals scattered inside a square. Mixing them rotates and scales the square into a parallelogram. PCA finds the axes of maximum variance — it "un-rotates" the parallelogram — but the result is still a rotated square, not the original one. PCA captures the spread of the data but is blind to the edges, which encode higher-order structure.

ICA goes further: it finds axes along which the data is not just uncorrelated but statistically independent. For uniform sources, this means finding the axes aligned with the edges of the parallelogram — which are exactly the directions of the original sources. The key difference: PCA uses only the covariance matrix (second-order), while ICA exploits the full shape of the distribution (all orders).

The brilliant insight of Bell and Sejnowski is an unexpected connection between two seemingly different goals:

Goal 1 — Information maximization: Pass the mixed signals through a network $\mathbf{y} = g(W\mathbf{x})$, where $g$ is a bounded nonlinear function (the sigmoid), and maximize the joint entropy $H(\mathbf{y})$ of the outputs. This is Linsker's "Infomax" principle: push as much information as possible through the network.

Goal 2 — Independence: Make the outputs statistically independent, i.e., minimize their mutual information.

The key theorem: because $g$ is fixed and invertible, maximizing $H(\mathbf{y})$ is equivalent to minimizing the mutual information among the components of $\mathbf{u} = W\mathbf{x}$ (the linear outputs before the nonlinearity). In other words, the network automatically finds independent components by trying to be an efficient information channel. The nonlinearity is what gives the network access to higher-order statistics — without it, we'd be back to PCA.

The network computes two stages. First, the linear unmixing: $\mathbf{u} = W\mathbf{x}$. Then, the element-wise nonlinearity: $y_i = g(u_i) = \frac{1}{1 + e^{-u_i}}$ (the logistic sigmoid).

The joint entropy of the output $\mathbf{y}$ can be written using the change-of-variables formula for probability densities. After simplification, the gradient of $H(\mathbf{y})$ with respect to $W$ yields a remarkably clean learning rule.

The term $W^{-T}$ has a geometric meaning: it prevents the determinant of $W$ from shrinking to zero (which would collapse all outputs into a single line). Think of it as an "anti-collapse" force that keeps the output representation spread out.

The term $\varphi(\mathbf{u})\mathbf{x}^T$ is where the magic happens: $\varphi(u_i) = 1 - 2g(u_i)$ measures how far each output is from being uniformly distributed after the sigmoid. It pushes each $u_i$ to match the distribution that would make $y_i$ uniform — which happens only when $u_i$ contains exactly one independent source. This is the term that goes beyond PCA: it uses the nonlinearity to access higher-order moments.

Why specifically a sigmoid? The logistic sigmoid $g(u) = 1/(1+e^{-u})$ is the cumulative distribution function (CDF) of the logistic distribution. When the network successfully separates a source, it pushes $u_i$ to match that source. Passing $u_i$ through the sigmoid then maps it to a uniform distribution on $[0,1]$ — because applying any random variable's CDF to itself yields a uniform. Uniform outputs have maximum entropy for a bounded variable, which is exactly what the algorithm is trying to achieve.

This means the sigmoid implicitly assumes that the sources have a distribution shaped like the logistic (super-Gaussian: peaked center, heavy tails). For sub-Gaussian sources (flatter than Gaussian, like uniform), the original algorithm struggles — which is why Lee, Girolami and Sejnowski (1999) later extended it with switchable nonlinearities.

The mutual information between the outputs can be decomposed as:

$I(u_1; u_2; \ldots; u_n) = \sum_i H(u_i) - H(\mathbf{u})$

This is the difference between the sum of individual entropies and the joint entropy. Independence means $I = 0$, which happens when the joint entropy equals the sum of marginals — the outputs carry no redundant information.

Because the sigmoid is a fixed invertible function, $H(\mathbf{y})$ and $H(\mathbf{u})$ differ only by a term that depends on $W$ through the log-Jacobian. Maximizing $H(\mathbf{y})$ with respect to $W$ simultaneously maximizes $H(\mathbf{u})$ — which, since the marginals $H(u_i)$ are bounded by the sigmoid's range, forces the mutual information $I$ toward zero. The algorithm achieves independence as a byproduct of efficient information transmission.

Bell and Sejnowski showed that the same principle extends to blind deconvolution: removing unknown echoes and reverberation from a signal. Instead of an instantaneous mixing matrix $A$, the signal is convolved with an unknown filter (impulse response). The unmixing becomes a filter too — a set of delay-weighted connections — and the same entropy-maximization rule adapts the filter weights.

Think of it this way: in blind separation, each sensor receives a spatial mixture (superposition of sources at one instant). In blind deconvolution, a single sensor receives a temporal mixture (superposition of a source with its own delayed copies, i.e., echoes). Both are linear mixing — one across channels, the other across time — and information maximization handles both.

Here is the complete Infomax ICA pipeline, from raw mixtures to recovered sources. The algorithm is surprisingly simple given what it achieves.

Infomax ICA did not just solve a signal processing problem — it created a bridge between information theory, neural networks, and neuroscience. The paper showed that a neural network can discover hidden structure in data by following a purely information-theoretic objective, with no labels, no teacher, and no knowledge of the data-generating process.

This unsupervised, information-driven approach directly inspired Olshausen and Field's sparse coding (1996), which applied similar principles to natural images and found that the learned features resemble the orientation-selective receptive fields of neurons in the primary visual cortex — suggesting that the brain itself might perform something like ICA.

Today, Infomax ICA is embedded in every major neuroscience toolbox (EEGLAB, MNE-Python, FieldTrip) for separating brain signals from artifacts. It is a standard preprocessing step in telecommunications for separating superimposed radio signals. The paper's deeper legacy is the demonstration that information theory provides a principled objective for unsupervised learning — an idea that echoes through variational autoencoders, contrastive learning, and modern representation learning.