Azimuth

Computer Vision1991foundational9 min read

Face Recognition Using Eigenfaces

التعرُّف على الوجوه باستخدام الوجوه الذاتية

Turk, M. A. · Pentland, A. P. — CVPR

The problem

In 1991, face recognition systems relied on hand-crafted geometric measurements — distances between eyes, nose width, jaw angle — that were brittle, required manual landmark annotation, and failed under even modest changes in lighting, pose, or expression. No system could handle the full complexity of a face image using all its information without drowning in thousands of dimensions.

The contribution

Eigenfaces: treat every face photo as a point in a very high-dimensional pixel space, then use Principal Component Analysis to find a low-dimensional "" that captures most of the variation among faces. The basis vectors of this space — the eigenfaces — are ghostly, face-like images. Any face can be described as a weighted sum of a few eigenfaces plus the . Recognition reduces to comparing these compact weight vectors using , enabling near-real-time identification with simple linear algebra.

The impact

Eigenfaces was the first practical, automatic face recognition system — no hand-crafted features, no manual landmarks. It proved that could compress a face to a handful of numbers and still tell people apart. The approach dominated the 1990s and directly inspired Fisherfaces (LDA), Bayesian face recognition, and the entire -methods lineage. While deep learning methods like FaceNet and ArcFace have since surpassed its accuracy, every modern face recognition pipeline inherits the core idea: represent faces in a learned low-dimensional space and compare them there.

Imagine a police sketch artist who draws every suspect from scratch — each portrait takes hours and captures only what the witness remembers.

Now imagine a different approach: the artist keeps a small book of transparent overlays — one shows a generic face shape, another adds wide cheekbones, a third adds a prominent nose, a fourth darkens the complexion. To build any face, the artist stacks a few overlays with different intensities.

Eigenfaces is exactly this system, discovered automatically from data. The overlays are the eigenfaces — learned patterns of variation — and each person's identity is a short recipe: "40% of overlay 1, minus 20% of overlay 2, plus 60% of overlay 3…"

The problem: why pixel space is too big

A grayscale face photo of 64×64 pixels is a of 4,096 numbers. Each face is a single point in a 4,096-dimensional space. Comparing two faces pixel-by-pixel means computing the between two points in this enormous space — but most of those dimensions carry , not identity.

Before Eigenfaces, systems tried to sidestep this by measuring hand-picked geometric features: the distance between the eyes, the width-to-height ratio of the nose, the angle of the jawline. But these features required manual annotation, ignored texture and shading, and broke down whenever lighting or pose changed even slightly.

The core insight of Turk and Pentland was: don't pick features — let the data tell you which dimensions matter. The tool for doing this had existed since the 1900s: Principal Component Analysis.

Open in Lab
Each slider changes one pixel. Notice how most pixel changes don't change the face identity — the information that matters lives in a small subspace.
The demo wakes as you arrive…

Building face space: the Eigenfaces recipe

The Eigenfaces algorithm has four clean steps. Before diving into the math, here is the intuition: we want to find the "building blocks" of face variation — the directions in pixel space along which faces differ the most from each other. These directions, sorted by importance, are the eigenfaces.

Ψ=1M∑i=1MΓi\Psi = \frac{1}{M}\sum_{i=1}^{M} \Gamma_i
Mean face — The mean face is created by averaging all training images pixel by pixel. It represents the typical appearance of the faces in the dataset and serves as a reference point for subsequent analysis. Individual faces are compared against this average so that the algorithm can focus on the variations that distinguish one person from another rather than on the common structure shared by all faces.
Open in Lab
Watch the mean face emerge as more training images are averaged together. The individual features blur, leaving only the shared structure.
The demo wakes as you arrive…
Φi=Γi−Ψ\Phi_i = \Gamma_i - \Psi
Difference face (mean-centered image) — Centering the data by subtracting the mean is the standard first step of PCA. It ensures that the subsequent covariance analysis captures directions of variance, not offset.
C=1MAAT⟹L=ATA(trick: M×M)C = \frac{1}{M} A A^T \quad \Longrightarrow \quad L = A^T A \quad (\text{trick: } M \times M)
The covariance trick — compute the small matrix instead — Directly analyzing the full covariance matrix would have been computationally infeasible because image vectors contain thousands of pixels. The key insight is that the same principal directions can be recovered by working with a much smaller matrix whose size depends on the number of training images rather than the number of pixels. This reduces both memory usage and computation dramatically, making Eigenfaces practical on the hardware available at the time.
Open in Lab
The first few eigenfaces capture broad variations (lighting, face shape). Later eigenfaces capture finer details. Drag the slider to see how many eigenfaces are needed for a good reconstruction.
The demo wakes as you arrive…
wk=ukT(Γnew−Ψ),ε=∥Ω−Ωj∥w_k = u_k^T (\Gamma_{\text{new}} - \Psi), \quad \varepsilon = \|\Omega - \Omega_j\|
Projection weight and recognition distance — To recognize a face, the image is first expressed as a combination of the learned eigenfaces. This produces a compact set of coefficients that captures the face's most important characteristics. Recognition is then performed by comparing this compact representation with those of known faces. Smaller distances indicate greater similarity, while larger distances suggest that the faces belong to different individuals.
Open in Lab
Select a test face to see it projected into face space. The system finds the closest match and shows the reconstruction from eigenfaces.
The demo wakes as you arrive…

Why it works: variance encodes identity

PCA sorts dimensions by how much the data varies along them. The first eigenface captures the single direction of greatest variation across the training set — often overall lighting. The second captures the most variation to the first — often face width or gender. As you go down the list, eigenfaces capture increasingly subtle differences: hairstyle, glasses, expression.

The key insight is that identity information is concentrated in just a few of these directions. A typical system uses 10–40 eigenfaces out of potentially thousands of pixel dimensions, achieving compression ratios of 100:1 or better with minimal loss of recognition accuracy.

This works because faces are not random pixel arrangements — they share an enormous amount of structure. The space of all possible face images is a tiny, low-dimensional embedded in the vast pixel space. PCA discovers this manifold.

Open in Lab
The eigenvalue spectrum drops off steeply: the first few eigenfaces capture most of the variance. The "elbow" shows where adding more eigenfaces yields diminishing returns.
The demo wakes as you arrive…

Beyond recognition: is it even a face?

Eigenfaces can also answer a more basic question: is the input image a face at all? When a non-face image (a tree, a chair) is projected into face space, its reconstruction from eigenfaces will be poor — the is high. This error, called the distance from face space (DFFS), acts as a face detector.

The system classifies every input into one of four categories using two thresholds — the distance within face space (how close the weight vector is to a known person) and the distance from face space (how well eigenfaces can reconstruct the image):

  • Known face: low DFFS, close to a stored person. Recognized.
  • Unknown face: low DFFS, far from all stored people. A face, but not in the database.
  • Not a face: high DFFS. The image cannot be well-represented by eigenfaces.
  • Near a face: borderline DFFS. Possibly a face seen under unusual conditions.
Open in Lab
Drag different images into the classifier. Watch the two distances determine whether the system sees a known face, unknown face, or non-face.
The demo wakes as you arrive…

The algorithm in code

Eigenfaces — full pipeline in NumPypython

Simplified to show the idea — not the real implementation.

import numpy as np
# 1. Load M training faces, each as a flattened vector of N pixels
# faces shape: (M, N)  e.g. (40, 4096) for 40 photos of 64x64
# 2. Compute the mean face
mean_face = faces.mean(axis=0)          # shape (N,)
# 3. Subtract the mean — center the data
diff = faces - mean_face                # shape (M, N)
# 4. The covariance trick: compute the small M×M matrix
L = diff @ diff.T                       # shape (M, M) — fast!
eigenvalues, eigenvectors = np.linalg.eigh(L)
# 5. Recover eigenfaces from the small eigenvectors
eigenfaces = (diff.T @ eigenvectors).T  # shape (M, N)
eigenfaces = eigenfaces / np.linalg.norm(eigenfaces, axis=1, keepdims=True)
# 6. Keep only the top K eigenfaces (highest eigenvalues)
K = 25
idx = np.argsort(eigenvalues)[::-1][:K]
top_eigenfaces = eigenfaces[idx]         # shape (K, N)
# 7. Project all training faces into face space weights = diff @ top_eigenfaces.T        # shape (M, K)
# 8. Recognize a new face
new_face = ...                           # shape (N,)
new_diff = new_face - mean_face
new_weights = top_eigenfaces @ new_diff  # shape (K,)
distances = np.linalg.norm(weights - new_weights, axis=1)
best_match = np.argmin(distances)

Strengths and limitations

The full pipeline at a glance

Open in Lab
The complete Eigenfaces pipeline from raw image to identity. Follow the arrows to see each transformation step.
The demo wakes as you arrive…

Impact: from eigenfaces to deep face recognition

Eigenfaces established the represent-then-compare paradigm that every face recognition system still follows: map faces to a learned low-dimensional space, then compare vectors in that space. What changed is how the mapping is learned — from linear PCA, to Fisher LDA, to handcrafted local features (LBPH), to convolutional neural networks that learn the entire end-to-end.

The paper also crystallized the idea that face recognition is fundamentally a problem. This perspective opened the door to the entire subspace methods literature and influenced how researchers think about far beyond faces.

  1. 1987

    Sirovich & Kirby

    First use of PCA on face images — proved that faces lie on a low-dimensional subspace and can be efficiently represented by a small number of "eigenpictures."

  2. 1991

    Eigenfaces (Turk & Pentland)

    Extended eigenpictures into a complete detection + recognition system operating in near real-time. The first automatic face recognition pipeline.

  3. 1997

    Fisherfaces (Belhumeur et al.)

    Applied Linear Discriminant Analysis after PCA, maximizing between-class variance. More robust to lighting changes than pure Eigenfaces.

  4. 2001

    Viola-Jones face detector

    Haar cascades enabled real-time face *detection* in images — solving the "where is the face" part before Eigenfaces or any recognition method runs.

  5. 2015

    FaceNet (Schroff et al.)

    Deep CNN with triplet loss that learns a 128-D face embedding directly optimized for recognition. Surpassed human-level accuracy on standard benchmarks.

  6. 2019

    ArcFace

    Angular margin loss pushed face embeddings to be even more discriminative, achieving state-of-the-art on million-scale face verification.

From Sirovich and Kirby's eigenpictures to FaceNet's deep embeddings, the thread is clear: compress faces into a small set of informative numbers, then match. Eigenfaces made this pipeline concrete and practical, and every system since has been a refinement of the same core idea.

CitationTurk, Pentland. Face Recognition Using Eigenfaces. CVPR, 1991.

Terms in this paper