The Regression Analysis of Binary Sequences

In classical linear regression, you predict a continuous output $y$ from inputs $x$: $y = \alpha + \beta x$. But what if $y$ can only be 0 or 1 — a patient lives or dies, a part is defective or not?

If you fit a straight line through 0's and 1's, the line inevitably goes above 1 and below 0 for extreme values of $x$. Those predictions are meaningless as probabilities. You need a model whose output is always a valid probability — between 0 and 1.

Cox adopted the logistic function (also called the sigmoid) to map any real number to a probability between 0 and 1. The core idea has three layers — think of it as a pipeline of transformations:

Layer 1 — Linear score. Compute a weighted sum $z = \alpha + \beta x$ exactly as in ordinary regression. This score can range from $-\infty$ to $+\infty$.

Layer 2 — Sigmoid squeeze. Feed $z$ into the S-shaped sigmoid function $\sigma(z) = 1 / (1 + e^{-z})$. This «squeezes» any real number into the interval $(0, 1)$.

Layer 3 — Interpret as probability. The output $\sigma(z)$ is the model's estimate of $P(Y = 1 \mid x)$ — the probability that the outcome is 1 given the input $x$.

This three-step pipeline is the beating heart of logistic regression, and its influence extends far beyond statistics: the sigmoid function later became the canonical activation function in neural networks, and the entire pipeline is what every binary classifier in deep learning still performs at its output layer.

The sigmoid maps a linear score to a probability. Its inverse — the logit — maps a probability back to a linear score. This inverse view is what makes logistic regression elegant and interpretable.

Start with the odds of the event: if the probability is $p$, the odds are $p / (1 - p)$. A probability of 0.75 means odds of 3:1 — "three times more likely to happen than not." Now take the natural logarithm of the odds to get the log-odds: $\text{logit}(p) = \ln(p / (1 - p))$.

Cox's key insight: the logit of the probability is a linear function of the predictors. This is the simplest possible relationship between the inputs and the transformed outcome — and it is exactly why the model is called log-istic regression.

Interpreting coefficients becomes natural: $\beta$ is the change in log-odds per unit change in $x$. A positive $\beta$ means higher $x$ makes the event more likely; a negative $\beta$ means higher $x$ makes it less likely.

In ordinary regression you minimize squared errors. But for binary outcomes, squared error is not the right ruler — a prediction of 0.99 for an actual 1 should be rewarded more than a prediction of 0.51.

Cox used maximum likelihood estimation: find the values of $\alpha$ and $\beta$ that make the observed data most probable under the model. For each data point: if $y_i = 1$, the model's contribution is $p_i$; if $y_i = 0$, it is $1 - p_i$. The likelihood is the product of all these terms, and we maximize it.

In practice we maximize the log-likelihood — a sum instead of a product — because sums are numerically stable and easier to differentiate. This log-likelihood is exactly the negative of what modern deep learning calls the binary cross-entropy loss. Every neural network you have ever trained for classification is optimizing the same objective Cox wrote down in 1958.

One of Cox's most elegant contributions was identifying the sufficient statistics for logistic regression. A sufficient statistic is a summary of the data that captures all the information the data contains about a parameter — you can throw away the raw data and lose nothing.

Cox showed that the logistic model has the same sufficient statistics as a normal-theory linear model: $Y = \sum y_i$ (the total count of 1's) and $X = \sum y_i x_i$ (a weighted sum of the predictor values). $Y$ tells you everything about the intercept $\alpha$, and $X$ tells you everything about the slope $\beta$.

This led to a powerful technique: conditional inference. To estimate $\beta$ without worrying about the nuisance parameter $\alpha$, simply condition on the observed value of $Y$. This eliminates $\alpha$ from the analysis entirely — no approximation needed. It is the binary-data analogue of Fisher's exact test for contingency tables.

The model extends naturally to multiple predictors. With $p$ predictors $x_1, \ldots, x_p$, the linear score becomes:

$z = \alpha + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p$

Each $\beta_j$ measures the effect of predictor $j$ on the log-odds, holding all other predictors constant. This is the same logic as multiple linear regression, but operating in log-odds space.

In modern notation, this is $z = \mathbf{w}^\top \mathbf{x} + b$ — the same dot-product-plus-bias that is the fundamental building block of every neural network layer. The entire output layer of a binary classifier in deep learning is this equation followed by a sigmoid — literally the logistic regression model.

The logistic function itself dates to Verhulst (1845), who used it to model population growth. Berkson (1944) applied it to bioassay — estimating the dose of a drug at which 50% of subjects respond. But it was Cox's 1958 paper that unified these threads into a general regression framework with rigorous inference, making the logistic model a tool for any binary-outcome problem.