Correlation, independence and transfer entropy beyond the normal distribution
For normal variables, zero correlation implies independence and Granger causality is equivalent to transfer entropy. For heavier- or lighter-tailed families neither holds, and the γ-order generalized normal shows how the difference arises. A scatter plot illustrates it.
Every statistics course warns that correlation is not causation. A related point receives less attention: zero correlation is not independence. Two variables can have a correlation of exactly zero and still be dependent, as financial markets illustrate when long calm periods are followed by simultaneous large movements.
A familiar example is electricity demand and outdoor temperature. Demand rises on cold days, for heating, and on hot days, for air conditioning. Over a year the two can show almost no correlation, although temperature remains one of the best predictors of demand. The relationship is strong but not linear, and correlation measures only linear association.
For normal variables the two notions coincide, and much of time-series practice relies on this implicitly. Outside the normal family they differ. This note shows how, using the multivariate γ-order generalized normal (γGN), whose sampling I described in random variate generation for the γGN. It then considers the consequence for time series: tests of whether Y helps predict X that cannot detect this kind of dependence.
The relevant literature is as follows.
- Maxwell (1860), studying the velocities of gas molecules, used the argument that became a classical characterisation: a spherically symmetric distribution with independent components must be normal.
- Granger (1969) defined predictive causality with regressions.
- Schreiber (2000) introduced transfer entropy, an information-theoretic counterpart with no linearity assumption.
- Kraskov, Stögbauer and Grassberger (2004) gave the nearest-neighbour estimator of mutual information now widely used.
- Barnett, Barrett and Seth (2009) proved that for jointly Gaussian variables, transfer entropy equals half the Granger statistic.
- Hlaváčková-Schindler, Toulias and Kitsos (2016) computed transfer entropy for processes with a multivariate γGN distribution.
A characterisation of the normal
Take a random vector whose distribution is spherical: it looks the same in every direction around the origin. Its components are always uncorrelated, by symmetry. A classical result going back to Maxwell's work on the velocities of gas molecules shows that they are independent only if the distribution is normal. Spherical symmetry plus independent components forces the Gaussian.
The multivariate γGN with identity scale matrix is spherical for every value of its shape parameter . So for its components are uncorrelated but dependent. The dependence is invisible to correlation, which only measures linear association, but it becomes visible in the squares of the components. As with electricity demand, which follows the squared distance of the temperature from a comfortable level, the relationship appears once the variables are squared.
Measuring the dependence
In a million simulated bivariate draws, the correlation between the components was zero in every case. The correlation between their squares was −0.23 at , zero at , 0.10 at and 0.16 at .
A cleaner measure is the mutual information, the Kullback-Leibler divergence between the joint distribution and the product of its marginals. It is zero if and only if the variables are independent, and it does not require the association to be linear. For the bivariate spherical γGN I computed it by numerical integration:
| 1.2 | 1.5 | 2 | 4 | 10 | 50 | |
|---|---|---|---|---|---|---|
| Mutual information (bits) | 0.048 | 0.008 | 0 | 0.010 | 0.023 | 0.030 |
The values are small, which is a reminder that this is a subtle form of dependence. They are not zero, and they grow as the distribution moves away from the normal in either direction. In the heavy-tailed members, large values tend to arrive together; in the light-tailed ones, a large value in one coordinate makes a large value in the other less likely.
Transfer entropy
Time series raise a more specific question: does the past of one series help predict another? Granger's 1969 definition addresses it with regressions. Series Granger-causes if adding the past of to a regression of on its own past reduces the prediction error. The usual statistic is the log ratio of the two residual variances.
Schreiber introduced an information-theoretic counterpart in 2000. The transfer entropy from to is the conditional mutual information between the present of and the past of , given the past of . It measures how much the past of reduces uncertainty about , with no assumption of linearity.
Barnett, Barrett and Seth proved in 2009 that, for jointly Gaussian variables, the two are equivalent: transfer entropy equals half the Granger statistic. For Gaussian data the regression-based test captures all predictive dependence.
Non-Gaussian families
Outside the Gaussian case the equivalence no longer holds, and the examples above show why. A linear Granger test only sees what correlations can see. If the dependence between the past of and the present of works through squares or tails, as in the spherical γGN, a regression on lagged values will find nothing while a conditional mutual information can.
The transmission of turbulence between stock markets is a practical case. After a large move in one market, the next session in another market often moves strongly as well, while the direction of that move remains unpredictable. A regression of returns on lagged returns finds nothing, because the dependence lies in the size of the moves and not in their sign. A test on squared returns, or a transfer entropy, detects it. Within a single market the same effect is called volatility clustering, the subject of volatility clustering and the GARCH model.
Hlaváčková-Schindler, Toulias and Kitsos computed transfer entropy for processes whose joint distribution is a multivariate γGN in 2016. They gave explicit expressions for the family and its Laplace limit, confirmed that the transfer entropy vanishes in the spherically contoured normal case, where the components are independent, and listed as an open problem the characterisation of which members give non-zero transfer entropy. The choice of error distribution can therefore determine whether a dependence is detected at all, not only its magnitude.
Estimation
Estimating mutual information or transfer entropy from data is harder than computing a correlation. Simple histogram estimators are badly biased; nearest-neighbour estimators such as the one of Kraskov, Stögbauer and Grassberger behave much better, but still need generous sample sizes and careful significance testing, typically against surrogate data. For short series, a linear test plus a test on squared residuals is a cheap first check.
The scatter plot below illustrates three properties: in the spherical γGN, the correlation of the components is zero for every γ; the correlation of their squares is negative for light tails, zero at the normal and positive for heavy tails; and the mutual information is zero only at γ = 2.
Each sample has 4,000 points, scaled to unit variance.
Suggested settings:
- The normal. Press "Normal (γ = 2)". The cloud is circular. Correlation zero, correlation of squares zero, mutual information zero: uncorrelated and independent.
- Heavy tails. Increase γ to 50. The cloud remains circular and the correlation still zero. But the squares are now positively correlated, about 0.16: an extreme value in one coordinate usually indicates a large radius, which also makes the other coordinate large. Extremes tend to occur together.
- Light tails. Reduce γ to 1.2. The cloud becomes a disc with a nearly sharp boundary. Now the squares are negatively correlated (about −0.23): the radius hardly varies, so if one coordinate accounts for most of it, the other must be small.
- Sampling variability. Press "New sample" a few times at γ = 4. The sample correlation of squares varies around the theoretical 0.10, and the ordinary correlation varies around zero. With 4,000 points, the dependence is visible but subtle, which is why it is easily missed in short series.
Derivation. Write a spherical vector as with uniform and independent of . Then : uncorrelated for any radius distribution. For the squares, and , so with ,
This is zero exactly when , which is the case for the Gaussian radius ( exponential with mean 2: , ). For the γGN, is Gamma, so , which gives the theoretical values in the widget: −0.23, 0, 0.10 and 0.16 at γ = 1.2, 2, 4 and 50. The mutual information is computed by numerical integration of the joint and marginal densities.
Conclusions:
- Zero correlation is a weak statement outside the normal family. Check squares, absolute values or a mutual information before concluding that two series are unrelated.
- Granger tests are linear by construction. Their equivalence with transfer entropy is a Gaussian result, so a negative Granger test on heavy-tailed data is not conclusive.
- The error distribution shapes the dependence. In elliptical families such as the γGN, the shape parameter decides how heavy the tails are and also how strongly large shocks move together.
References
- J. C. Maxwell, "Illustrations of the dynamical theory of gases", Philosophical Magazine, 4th series, 19, 19-32, 1860.
- C. W. J. Granger, "Investigating causal relations by econometric models and cross-spectral methods", Econometrica, 37(3), 424-438, 1969.
- T. Schreiber, "Measuring information transfer", Physical Review Letters, 85(2), 461-464, 2000.
- A. Kraskov, H. Stögbauer and P. Grassberger, "Estimating mutual information", Physical Review E, 69, 066138, 2004.
- L. Barnett, A. B. Barrett and A. K. Seth, "Granger causality and transfer entropy are equivalent for Gaussian variables", Physical Review Letters, 103, 238701, 2009.
- K. Hlaváčková-Schindler, T. L. Toulias and C. P. Kitsos, "Evaluating transfer entropy for normal and γ-order normal distributions", British Journal of Mathematics & Computer Science, 17(5), 1-20, 2016.