Jorge Quitério Search
Mathematics

Fisher information as the curvature of the log-likelihood

The curvature of the log-likelihood around its maximum determines how much the data reveal about a parameter. The main properties of Fisher information follow from this interpretation, which a simulation illustrates for the normal mean.

Fisher information is often perceived as difficult. It is defined as "the variance of the derivative of the log of a density", it appears in several inequalities, and it is used in fields from quantum physics to machine learning. Students often memorise the formula without a clear interpretation.

It rests on one geometric idea: the sharper the peak of the likelihood, the more information the data carry. This article explains that idea and its consequences, and verifies the central result with simulated experiments.

R. A. Fisher introduced the concept in his foundational papers on estimation in the 1920s, notably "Theory of statistical estimation" (1925). The lower bound on the variance of unbiased estimators that bears Cramér's and Rao's names was established independently by Rao (1945) and Cramér (1946). Modern textbook treatments include Casella and Berger (2002) and Lehmann and Casella (1998). The link between Fisher information and the Kullback-Leibler divergence is the starting point of information geometry, and generalised, γ-order versions of Fisher's information are where the γ-order generalized normal distribution comes from, as described in the γ-order generalized normal distribution.

Curvature of the log-likelihood

Take a model with density f(x;θ)f(x;\theta) and look at the log-likelihood ℓ(θ)=log⁡f(x;θ)\ell(\theta) = \log f(x;\theta) as a function of the parameter. If the data are informative, ℓ\ell has a sharp peak: moving θ\theta away from the best value makes the data much less plausible. If the data are uninformative, the peak is flat.

Fisher information measures that sharpness:

I(θ)=− E ⁣[∂2∂θ2log⁡f(X;θ)]=Var⁡ ⁣[∂∂θlog⁡f(X;θ)]. I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right] = \operatorname{Var}\!\left[\frac{\partial}{\partial\theta}\log f(X;\theta)\right].

The first form is the expected curvature. The second uses the score, the slope of the log-likelihood, which averages to zero at the true parameter; under mild conditions its variance equals the expected curvature.

Two examples

For a normal sample with known spread σ\sigma, one observation carries information I(μ)=1/σ2I(\mu) = 1/\sigma^2 about the mean. Lower noise gives more information. With nn independent observations, information adds up: n/σ2n/\sigma^2.

For a yes/no trial with success probability pp, I(p)=1/[p(1−p)]I(p) = 1/[p(1-p)]. It is smallest at p=1/2p = 1/2, where outcomes are least predictable, and grows without bound near 0 and 1, where a single unexpected outcome is highly informative.

Opinion polls are the familiar application. With 1,000 respondents and support near 50%, the information is 1000/(0.5×0.5)=40001000/(0.5 \times 0.5) = 4000, so the standard error is 1/4000≈1.61/\sqrt{4000} \approx 1.6 percentage points and the usual margin of error is about ±3 points. For a party polling near 2%, the same sample gives a standard error of about 0.44 points. The poll measures the small party more precisely in absolute terms, although its relative uncertainty, about a fifth of the estimate, is much larger.

Consequences

  • A floor on precision. The Cramér-Rao inequality says any unbiased estimator from nn independent observations has Var⁡(θ^)≥1/(nI(θ))\operatorname{Var}(\hat\theta) \ge 1/(n I(\theta)). For the normal mean, the sample average reaches this floor exactly.
  • Standard errors. In large samples the maximum likelihood estimator is approximately normal with variance 1/(nI(θ))1/(nI(\theta)). The standard errors printed by statistical software are usually this, evaluated at the estimate.
  • A local distance between models. For nearby parameters, KL(fθ ∥ fθ+δ)≈12I(θ) δ2\mathrm{KL}(f_\theta\,\|\,f_{\theta+\delta}) \approx \tfrac12 I(\theta)\,\delta^2, which links this note to the one on distances between distributions.

Dependent observations

Independence makes information additive. With dependence it must be computed for the whole model. For the Gaussian AR(1) model Xt=ϕXt−1+etX_t = \phi X_{t-1} + e_t from the autoregressive model of a time series, the information about ϕ\phi per observation is asymptotically 1/(1−ϕ2)1/(1-\phi^2): estimates become more precise as ∣ϕ∣|\phi| approaches 1, because each observation carries more of the past with it. Non-normal errors change that calculation, which is one reason heavy-tailed time-series models need their own theory.

The figure below illustrates the result: the curvature of the log-likelihood predicts the spread of the estimates. For the normal mean, the standard deviation of the sample average over repeated experiments equals 1/I=σ/n1/\sqrt{I} = \sigma/\sqrt{n}.

  1. Curvature. Increase nn or decrease σ\sigma. The log-likelihood (solid) becomes sharper and the quadratic with curvature n/σ2n/\sigma^2 (dashed) follows it exactly: for the normal mean the log-likelihood is a parabola.
  2. Repeated experiments. Press "Repeat 2,000 times". Each repetition draws a fresh sample and records its estimate. Compare "observed SD of estimates" with "predicted SE". They agree to two or three decimals, at every setting.
  3. Square-root scaling. Quadruple the sample size, from 20 to 80. The SE halves rather than quartering: information grows like nn, precision only like n\sqrt{n}. This is why national opinion polls rarely interview more than a few thousand people: halving a margin of error of three points requires four times as many interviews.

Derivation for the normal mean. With nn observations and known σ\sigma, ℓ(μ)=−∑(xi−μ)2/(2σ2)+const\ell(\mu) = -\sum (x_i-\mu)^2/(2\sigma^2) + \text{const}. Its second derivative is −n/σ2-n/\sigma^2 whatever the data, so I=n/σ2I = n/\sigma^2. The estimator xˉ\bar x has variance σ2/n=1/I\sigma^2/n = 1/I, so it attains the Cramér-Rao bound exactly, which is what the simulation confirms.

Fisher information is the curvature of the log-likelihood at its peak: a sharp peak means informative data and precise estimates, a flat one means the opposite. The Cramér-Rao bound, the standard errors in regression output and the local geometry of statistical models all follow from this interpretation.

References

  1. R. A. Fisher, "Theory of statistical estimation", Mathematical Proceedings of the Cambridge Philosophical Society, 22(5), 700-725, 1925.
  2. C. R. Rao, "Information and the accuracy attainable in the estimation of statistical parameters", Bulletin of the Calcutta Mathematical Society, 37, 81-91, 1945.
  3. H. Cramér, Mathematical Methods of Statistics, Princeton University Press, 1946.
  4. G. Casella and R. L. Berger, Statistical Inference, 2nd ed., Duxbury, 2002.
  5. E. L. Lehmann and G. Casella, Theory of Point Estimation, 2nd ed., Springer, 1998.