Fisher information as the curvature of the log-likelihood
The curvature of the log-likelihood around its maximum determines how much the data reveal about a parameter. The main properties of Fisher information follow from this interpretation, which a simulation illustrates for the normal mean.
Fisher information is often perceived as difficult. It is defined as "the variance of the derivative of the log of a density", it appears in several inequalities, and it is used in fields from quantum physics to machine learning. Students often memorise the formula without a clear interpretation.
It rests on one geometric idea: the sharper the peak of the likelihood, the more information the data carry. This article explains that idea and its consequences, and verifies the central result with simulated experiments.
R. A. Fisher introduced the concept in his foundational papers on estimation in the 1920s, notably "Theory of statistical estimation" (1925). The lower bound on the variance of unbiased estimators that bears Cramér's and Rao's names was established independently by Rao (1945) and Cramér (1946). Modern textbook treatments include Casella and Berger (2002) and Lehmann and Casella (1998). The link between Fisher information and the Kullback-Leibler divergence is the starting point of information geometry, and generalised, γ-order versions of Fisher's information are where the γ-order generalized normal distribution comes from, as described in the γ-order generalized normal distribution.
Curvature of the log-likelihood
Take a model with density and look at the log-likelihood as a function of the parameter. If the data are informative, has a sharp peak: moving away from the best value makes the data much less plausible. If the data are uninformative, the peak is flat.
Fisher information measures that sharpness:
The first form is the expected curvature. The second uses the score, the slope of the log-likelihood, which averages to zero at the true parameter; under mild conditions its variance equals the expected curvature.
Two examples
For a normal sample with known spread , one observation carries information about the mean. Lower noise gives more information. With independent observations, information adds up: .
For a yes/no trial with success probability , . It is smallest at , where outcomes are least predictable, and grows without bound near 0 and 1, where a single unexpected outcome is highly informative.
Opinion polls are the familiar application. With 1,000 respondents and support near 50%, the information is , so the standard error is percentage points and the usual margin of error is about ±3 points. For a party polling near 2%, the same sample gives a standard error of about 0.44 points. The poll measures the small party more precisely in absolute terms, although its relative uncertainty, about a fifth of the estimate, is much larger.
Consequences
- A floor on precision. The Cramér-Rao inequality says any unbiased estimator from independent observations has . For the normal mean, the sample average reaches this floor exactly.
- Standard errors. In large samples the maximum likelihood estimator is approximately normal with variance . The standard errors printed by statistical software are usually this, evaluated at the estimate.
- A local distance between models. For nearby parameters, , which links this note to the one on distances between distributions.
Dependent observations
Independence makes information additive. With dependence it must be computed for the whole model. For the Gaussian AR(1) model from the autoregressive model of a time series, the information about per observation is asymptotically : estimates become more precise as approaches 1, because each observation carries more of the past with it. Non-normal errors change that calculation, which is one reason heavy-tailed time-series models need their own theory.
The figure below illustrates the result: the curvature of the log-likelihood predicts the spread of the estimates. For the normal mean, the standard deviation of the sample average over repeated experiments equals .
- Curvature. Increase or decrease . The log-likelihood (solid) becomes sharper and the quadratic with curvature (dashed) follows it exactly: for the normal mean the log-likelihood is a parabola.
- Repeated experiments. Press "Repeat 2,000 times". Each repetition draws a fresh sample and records its estimate. Compare "observed SD of estimates" with "predicted SE". They agree to two or three decimals, at every setting.
- Square-root scaling. Quadruple the sample size, from 20 to 80. The SE halves rather than quartering: information grows like , precision only like . This is why national opinion polls rarely interview more than a few thousand people: halving a margin of error of three points requires four times as many interviews.
Derivation for the normal mean. With observations and known , . Its second derivative is whatever the data, so . The estimator has variance , so it attains the Cramér-Rao bound exactly, which is what the simulation confirms.
Fisher information is the curvature of the log-likelihood at its peak: a sharp peak means informative data and precise estimates, a flat one means the opposite. The Cramér-Rao bound, the standard errors in regression output and the local geometry of statistical models all follow from this interpretation.
References
- R. A. Fisher, "Theory of statistical estimation", Mathematical Proceedings of the Cambridge Philosophical Society, 22(5), 700-725, 1925.
- C. R. Rao, "Information and the accuracy attainable in the estimation of statistical parameters", Bulletin of the Calcutta Mathematical Society, 37, 81-91, 1945.
- H. Cramér, Mathematical Methods of Statistics, Princeton University Press, 1946.
- G. Casella and R. L. Berger, Statistical Inference, 2nd ed., Duxbury, 2002.
- E. L. Lehmann and G. Casella, Theory of Point Estimation, 2nd ed., Springer, 1998.