Visualizing Data Using t-SNE

PCA finds the axes of greatest variance and projects data onto them. This is optimal when data lies on a flat plane. But real-world datasets often live on curved, folded surfaces — manifolds — embedded in high-dimensional space. Think of a Swiss roll: the data is really two-dimensional (position along the roll and height), but PCA sees a flat pancake and smashes points together that are far apart on the surface.

Earlier nonlinear methods like Sammon mapping and Isomap improved on PCA, but they still struggled to preserve both local and global structure simultaneously. What was needed was a method that makes the neighborhoods in low dimensions match those in high dimensions — even if global distances have to bend.

t-SNE starts with a simple question: for each point $x_i$ in high-dimensional space, how likely is it that $x_i$ would pick $x_j$ as its neighbor? The probability is defined as a Gaussian centered on $x_i$ — nearby points get high probability, distant points get near zero. Each point has its own Gaussian bandwidth $\sigma_i$, set automatically by the perplexity parameter (more on that shortly).

Then t-SNE asks the same question in the 2D map: for each low-dimensional point $y_i$, how likely is it that $y_i$ picks $y_j$ as its neighbor? Here the kernel is a Student-t distribution with one degree of freedom (a Cauchy distribution) — a bell curve with much heavier tails than a Gaussian.

The algorithm's goal: adjust the 2D positions until the low-dimensional probabilities match the high-dimensional ones. The mismatch is measured by KL divergence, and positions are optimized by gradient descent.

The first step converts every pairwise Euclidean distance in the original space into a conditional probability. Think of it as asking: "If point $x_i$ chooses a neighbor in proportion to a Gaussian centered on itself, what is the chance it picks $x_j$?"

To make the probabilities symmetric ($p_{ij} = p_{ji}$), t-SNE averages the two conditional probabilities: $p_{ij} = (p_{j|i} + p_{i|j}) / 2n$. This guarantees every point makes a significant contribution, even outliers.

In the 2D map, the similarity between the low-dimensional counterparts $y_i$ and $y_j$ uses a Student-t distribution with one degree of freedom instead of a Gaussian. The purpose of the heavy tail is critical — it gives moderately distant points room in the map, solving the crowding problem.

t-SNE wants the neighborhood structure in 2D to mirror the one in high-D. It measures the mismatch using KL divergence — a one-way measure of how different two probability distributions are. The goal is simple: make Q (the 2D neighborhoods) as close to P (the high-D neighborhoods) as possible.

Each point's Gaussian bandwidth $\sigma_i$ is chosen so that its probability distribution has a fixed perplexity — roughly the effective number of neighbors. Low perplexity (5–10) means each point only cares about its nearest few friends, producing tight local clusters. High perplexity (30–50) considers more neighbors, producing smoother, more globally structured maps.

The typical default is 30. But perplexity is not just a dial to set and forget — it controls the scale of structure you see. Running t-SNE at multiple perplexities is a good practice to separate real structure from artifact.

Consider 11 equidistant points on a 10-dimensional sphere. In 10D there is plenty of room for all of them to be far from a central point. But in 2D there is only a thin ring around the center — far fewer "moderately distant" positions are available. If you use a Gaussian kernel in 2D (as the earlier SNE algorithm did), those moderately distant points get crushed into the center. Everything becomes a single undifferentiated blob.

The Student-t distribution solves this by making the "acceptable distance" in 2D much wider. A point that is moderately similar in high-D can sit comfortably far from its partner in 2D without paying a large KL penalty, because $q_{ij}$ stays reasonably high even at that distance. This is the key insight of t-SNE over its predecessor SNE.

The gradient of the KL cost with respect to each map point $y_i$ has an elegant physical interpretation. Each pair of points exerts a "force" on each other. If $p_{ij} > q_{ij}$ (the points should be closer but aren't), there is an attractive spring pulling them together. If $p_{ij} < q_{ij}$ (the points are too close), there is a repulsive force pushing them apart. The $(1 + \|y_i - y_j\|^2)^{-1}$ factor means nearby points exert stronger forces — distant points are almost ignored.

t-SNE is a powerful tool, but its outputs can be misleading if you don't know what to trust. Here are the rules experienced practitioners follow:

• Cluster membership is meaningful — points in the same cluster are genuinely nearby in the original space.
• Distances between clusters are NOT meaningful — two clusters may look far apart but be close in the original space, or vice versa.
• Cluster sizes are NOT meaningful — t-SNE can inflate or shrink clusters depending on the local density of the data.
• Run it multiple times — t-SNE has a random initialization. Different runs can produce different layouts. True structure appears consistently across runs.
• Try multiple perplexities — structure that only appears at one perplexity may be an artifact.

Here is the complete algorithm laid out as a pipeline. Notice that the high-dimensional probabilities $P$ are computed once upfront, while the low-dimensional probabilities $Q$ are recomputed every iteration because the map points $y_i$ are moving.

UMAP (2018) builds on t-SNE's insight — optimize a neighborhood-based cost function — but uses a different mathematical framework (fuzzy simplicial sets from topological data analysis). Key differences:

• Speed: UMAP is much faster, especially on large datasets, because it uses approximate nearest neighbors and stochastic gradient descent.
• Global structure: UMAP tends to better preserve global relationships — the relative positions of clusters are more meaningful.
• Embeddings: UMAP can embed new, unseen data points using a learned transformation. t-SNE must re-run the entire optimization.
• Determinism: UMAP produces more consistent runs with different random seeds.

Both are valuable. t-SNE remains preferred when local cluster quality is paramount and dataset size is manageable. UMAP wins on speed, scale, and when global structure matters.

t-SNE's legacy is not just a visualization algorithm — it established the idea that neighborhood-based probabilistic optimization can reveal structure invisible to linear methods. Every time you see a colorful scatter plot of embeddings, cell types, or model activations, you are likely looking at t-SNE's intellectual offspring.