Jorge Quitério Search
Mathematics

Regression with autocorrelated errors: from Cochrane-Orcutt to Durbin's h

Regression with autocorrelated errors produced several well-known methods in econometrics. The most instructive case is the one in which these methods fail: a lagged dependent variable among the regressors. A simulation shows least squares converging to an incorrect value.

Long before the autoregressive model became a forecasting tool, econometricians met autocorrelation as a nuisance. They were fitting regressions to annual or quarterly data, and the residuals were not independent: a positive error in one year tended to be followed by another.

The methods they built between 1949 and 1970 remain standard in econometrics textbooks. The most instructive case is the one in which these methods fail: when the lagged value of the dependent variable is itself a regressor. Least squares then does not merely lose precision; it converges to an incorrect value as the sample grows, and the standard test fails to detect the problem.

Marketing analysis provides a typical case. An analyst explains monthly sales by advertising spend and by last month's sales, the second term representing customer habit: part of each month's sales carries over to the next. If unmeasured influences, such as a competitor's campaign or a period of poor weather, persist for several months, the errors are autocorrelated. Least squares then attributes their persistence to customer habit, and the carry-over appears stronger than it is.

The classical remedies:

  • Cochrane and Orcutt (1949): estimate, compute ρ\rho from the residuals, transform, re-estimate, repeat.
  • Prais and Winsten (1954) keep the first observation by scaling it by 1−ρ2\sqrt{1-\rho^2}, which matters in short samples.
  • Hildreth and Lu (1960) search a grid of ρ\rho values for the smallest residual sum of squares.
  • Durbin (1960) gave a two-step estimator using lagged values.
  • Beach and MacKinnon (1978) gave a full maximum-likelihood procedure.

The tests:

  • Durbin and Watson (1950, 1951) proposed the dd statistic and published bounds for it.
  • Griliches (1961) showed what goes wrong with a lagged dependent variable.
  • Durbin (1970) proposed the hh statistic for that case.
  • Breusch (1978) and Godfrey (1978) proposed the Lagrange multiplier test now reported by most software.

The problem

Take a regression whose errors follow a first-order autoregression,

yt=xt⊤β+ut,ut=ρ ut−1+et. y_t = \mathbf{x}_t^\top \boldsymbol\beta + u_t, \qquad u_t = \rho\, u_{t-1} + e_t .

If the regressors are unrelated to the errors, least squares still estimates β\boldsymbol\beta without bias. What goes wrong is the uncertainty: with positive ρ\rho, the usual standard errors are typically too small, so coefficients look more significant than they are.

The remedies

Subtracting ρ\rho times the previous equation removes the autocorrelation:

yt−ρ yt−1=(xt−ρ xt−1)⊤β+et. y_t - \rho\, y_{t-1} = (\mathbf{x}_t - \rho\,\mathbf{x}_{t-1})^\top \boldsymbol\beta + e_t .

If ρ\rho were known, least squares on these quasi-differenced data would be efficient. All the classical methods are different ways of estimating ρ\rho for this step.

The test

Durbin and Watson compare successive residuals:

d=∑t=2n(u^t−u^t−1)2∑t=1nu^t2  ≈  2 (1−ρ^). d = \frac{\sum_{t=2}^n (\hat u_t - \hat u_{t-1})^2}{\sum_{t=1}^n \hat u_t^2} \;\approx\; 2\,(1 - \hat\rho) .

Near 2 means no first-order autocorrelation; well below 2 means positive autocorrelation. Its exact distribution depends on the regressors, hence the published bounds and the inconclusive region between them.

A lagged dependent variable

Everything above assumes the regressors are unrelated to the errors. Dynamic models violate this assumption by construction. The simplest case:

yt=k yt−1+ut,ut=ρ ut−1+et. y_t = k\, y_{t-1} + u_t, \qquad u_t = \rho\, u_{t-1} + e_t .

Now yt−1y_{t-1} contains ut−1u_{t-1}, which is correlated with utu_t. Least squares is then inconsistent, which is much worse than inefficient. Griliches showed that

plim⁡k^=k+ρ1+kρ. \operatorname*{plim} \hat k = \frac{k + \rho}{1 + k\rho} .

The memory of the errors is absorbed into the coefficient of the lag. With k=ρ=0.5k = \rho = 0.5 the limit is 0.8, not 0.5.

In the marketing example the error has a direct cost. With a carry-over of 0.5, the total effect of a campaign over all following months is 1/(1−0.5)=21/(1-0.5) = 2 times its effect in the first month. With the estimated 0.8 it appears to be 1/(1−0.8)=51/(1-0.8) = 5 times, so the analysis would credit advertising with two and a half times its real long-run effect and justify a correspondingly larger budget.

The test fails in the same situation. Because the lag absorbs the autocorrelation, the residuals appear much less autocorrelated than the true errors. With ρ=0.5\rho = 0.5 a static regression would show a Durbin-Watson value near 1.0; here it sits near 1.6. The test is biased towards 2, that is, towards detecting no autocorrelation.

Durbin's h and later tests

Durbin's 1970 statistic,

h=(1−d2)n1−n Var⁡^(k^), h = \left(1 - \frac{d}{2}\right)\sqrt{\frac{n}{1 - n\,\widehat{\operatorname{Var}}(\hat k)}} ,

is approximately standard normal under no autocorrelation, but cannot be computed when n Var⁡^(k^)≥1n\,\widehat{\operatorname{Var}}(\hat k) \ge 1. The Breusch-Godfrey test avoids that gap: regress the residuals on the original regressors and on lagged residuals, and test whether the lagged residuals matter. It handles lagged dependent variables and higher-order autocorrelation.

The figure below illustrates the effect. With a lagged dependent variable and autocorrelated errors, least squares converges to (k+ρ)/(1+kρ)(k+\rho)/(1+k\rho) rather than kk, and the Durbin-Watson statistic drifts towards 2.

Each run simulates 400 series of 1,000 observations and plots the distribution of the least-squares estimates.

Suggested settings:

  1. Griliches' example. With k=ρ=0.5k = \rho = 0.5, the histogram sits on 0.80, far from the true 0.5, and the average Durbin-Watson is about 1.60 instead of the value of about 1.00 implied by the errors.
  2. No error autocorrelation. Set ρ=0\rho = 0. The histogram is centred on the true value and Durbin-Watson returns to about 2.
  3. Cancellation. Set k=0.5k = 0.5 and ρ=−0.5\rho = -0.5. The estimate falls to about 0: least squares reports that last period has no effect, when it has an effect of 0.5. The two effects cancel exactly.
  4. Sample size. These series already have 1,000 observations each. More data would only make the histogram narrower around the wrong value. This is the meaning of inconsistency.

Derivation of the limit. Substitute ut=yt−kyt−1u_t = y_t - k y_{t-1} into ut=ρut−1+etu_t = \rho u_{t-1} + e_t:

yt=(k+ρ) yt−1−kρ yt−2+et, y_t = (k + \rho)\, y_{t-1} - k\rho\, y_{t-2} + e_t ,

an AR(2) process. Least squares of yty_t on yt−1y_{t-1} alone converges to the first autocorrelation of this process, r1r_1. The Yule-Walker equation at lag 1 reads r1=ϕ1+ϕ2r1r_1 = \phi_1 + \phi_2 r_1 with ϕ1=k+ρ\phi_1 = k+\rho and ϕ2=−kρ\phi_2 = -k\rho, so

r1=ϕ11−ϕ2=k+ρ1+kρ. r_1 = \frac{\phi_1}{1 - \phi_2} = \frac{k + \rho}{1 + k\rho} .

The same AR(2) view explains the case ρ=−k\rho = -k: then ϕ1=0\phi_1 = 0 and the first autocorrelation vanishes.

Practical implications:

  • Dynamics and autocorrelation are hard to separate. Persistence can be described by a lag of yy, by autocorrelated errors, or both, and least squares can substitute one for the other.
  • Model the dynamics explicitly. Include enough lags of yy and x\mathbf{x} for the remaining errors to be white noise, then test that they are, with Breusch-Godfrey rather than Durbin-Watson.
  • Check the assumptions of the test. Durbin-Watson is still printed by default in many regression outputs, including for models where it is biased.

This is the same AR(1) whose estimator already misbehaves in least-squares estimation of the AR(1) coefficient in three regimes; here the difficulty arises from the errors rather than from the coefficient.

References

  1. D. Cochrane and G. H. Orcutt, "Application of least squares regression to relationships containing auto-correlated error terms", Journal of the American Statistical Association, 44(245), 32-61, 1949.
  2. J. Durbin and G. S. Watson, "Testing for serial correlation in least squares regression. I", Biometrika, 37(3/4), 409-428, 1950; "II", Biometrika, 38(1/2), 159-178, 1951.
  3. J. Durbin, "Estimation of parameters in time-series regression models", Journal of the Royal Statistical Society B, 22(1), 139-153, 1960.
  4. Z. Griliches, "A note on serial correlation bias in estimates of distributed lags", Econometrica, 29(1), 65-73, 1961.
  5. J. Durbin, "Testing for serial correlation in least-squares regression when some of the regressors are lagged dependent variables", Econometrica, 38(3), 410-421, 1970.
  6. C. M. Beach and J. G. MacKinnon, "A maximum likelihood procedure for regression with autocorrelated errors", Econometrica, 46(1), 51-58, 1978.
  7. T. S. Breusch, "Testing for autocorrelation in dynamic linear models", Australian Economic Papers, 17(31), 334-355, 1978.
  8. L. G. Godfrey, "Testing against general autoregressive and moving average error models when the regressors include lagged dependent variables", Econometrica, 46(6), 1293-1301, 1978.