Kullback-Leibler divergence and Hellinger distance
The Kullback-Leibler divergence and the Hellinger distance both measure how different two models are, but they behave differently, and the difference determines which one to use. Both are computed numerically in the figure.
The difference between two models can be measured in more than one way, and the measures do not always agree.
Typical questions are whether a fitted model is close to the truth, how far a heavy-tailed error distribution is from the normal, and whether a data-generating process has changed. Two measures are most widely used: the Kullback-Leibler divergence, from information theory, and the Hellinger distance, from geometry. The first is asymmetric and can be infinite; the second is symmetric and bounded. This article explains both and how they relate.
Kullback and Leibler introduced their divergence in 1951 as a measure of the information for discriminating between two hypotheses; it is closely tied to Shannon's entropy and is treated in depth by Cover and Thomas (2006). Ernst Hellinger's distance dates from 1909 and his work on integral equations. Akaike's information criterion (1974) is an estimate of a Kullback-Leibler divergence. Dabak and Johnson (2002) relate the KL divergence to Fisher information. For the γ-order generalized normal family, Kitsos and Toulias (2017) derived the Hellinger distance between members in closed form.
Kullback-Leibler divergence
For densities and ,
It is the average log-likelihood ratio when the data really come from : how much evidence, per observation, accumulates against . In coding terms, it is the extra message length paid for using a code designed for when the source is : a compression scheme tuned to the letter frequencies of English text produces longer files, by this amount per character, when it is applied to French text. Three properties are important:
- it is not symmetric: in general;
- it does not satisfy the triangle inequality, so it is a divergence, not a distance;
- it can be infinite, for example when gives zero probability where does not.
Banks apply the divergence routinely. A credit-scoring model is built on past applicants, and each year the distribution of the new applicants' scores is compared with the original one, bin by bin, using the population stability index. That index is the sum of the two Kullback-Leibler divergences, one in each direction. A large value indicates that the applicants have changed and the model may need rebuilding. The third property above is familiar to practitioners: an empty bin makes the index infinite, and a small constant is usually added to every bin to prevent it.
For two normals there is a closed form:
Hellinger distance
The squared Hellinger distance compares square roots of the densities:
It is symmetric, between 0 and 1, always finite, and itself is a true metric.
How they relate
so a small KL divergence forces a small Hellinger distance, but not the other way round. Locally they agree up to a constant, both governed by the Fisher information: and for a small parameter change .
A worked example
Compare a standard normal with a Laplace distribution of the same variance. The two have the same centre and spread but different shapes. Then , and . Modelling heavy-tailed data as normal costs more than the reverse, because the normal model assigns low probability to the Laplace tails. For a model of insurance claims or market losses, this is the more expensive mistake: the normal model is repeatedly surprised by large losses that it treats as almost impossible.
Choosing between them
Use Kullback-Leibler for questions about likelihood and model choice; it is what AIC estimates and it has a natural direction, from the truth to the approximation. Use Hellinger when a symmetric, bounded distance is needed: comparing several fitted models with each other, building robust estimators, or when some densities vanish where others do not.
The figure below computes both and illustrates three properties: KL is asymmetric, Hellinger is symmetric and bounded, and in both directions.
- Worked example. Press "Laplace, same variance". The readouts show 0.056, 0.072 and 0.015, computed by numerical integration.
- Pure shape. Start from the normal and move only the shape slider. Means and variances never change, yet both measures grow: a difference in shape alone is a real difference.
- Pure location. Reset, then move only the shift. For two normals with equal variance, the two KL directions now agree exactly, both equal to half the squared shift. Asymmetry requires a difference in shape or spread.
- Different spread. Set the spread of to 0.4. KL from the wide to the narrow rises sharply, because assigns very low probability to values that produces. KL in the other direction stays modest.
- The inequality. No setting makes "KL ≥ 2H²" fail.
Non-negativity of KL, the simplest related inequality: since is concave, Jensen's inequality gives . Equality holds only when .
Kullback-Leibler and Hellinger measure the same thing, the difference between two models, from two viewpoints. KL is directional and tied to likelihood and information, which makes it suitable for model choice. Hellinger is symmetric and bounded, which makes it suitable when a true distance is required. For error distributions such as the γGN, both answer the practical question that recurs in this series: once the shape of the errors has been estimated, is it far enough from the normal to matter?
References
- E. Hellinger, "Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen", Journal für die reine und angewandte Mathematik, 136, 210-271, 1909.
- S. Kullback and R. A. Leibler, "On information and sufficiency", Annals of Mathematical Statistics, 22(1), 79-86, 1951.
- H. Akaike, "A new look at the statistical model identification", IEEE Transactions on Automatic Control, 19(6), 716-723, 1974.
- T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
- A. G. Dabak and D. H. Johnson, Relations between Kullback-Leibler distance and Fisher information, Technical report, Rice University, 2002.
- C. P. Kitsos and T. L. Toulias, "Hellinger distance between generalized normal distributions", British Journal of Mathematics & Computer Science, 21(2), 1-16, 2017.