Jorge Quitério Search
Mathematics

Least-squares estimation of the AR(1) coefficient in three regimes

The least-squares estimate of an AR(1) coefficient behaves differently depending on whether the process is stationary, has a unit root or is explosive: the rate of convergence and the limiting distribution differ in each case. A simulation shows the transition across the boundary.

The first-order autoregressive model, Xt=k Xt−1+etX_t = k\,X_{t-1} + e_t, is the simplest model in time series: one coefficient and one noise term. Its estimation theory is nevertheless subtle. I introduced it in the autoregressive model of a time series and described its origin in Yule's sunspot model.

It has one obvious estimator, the least-squares ratio

k^=∑tXt−1Xt∑tXt−12. \hat k = \frac{\sum_t X_{t-1} X_t}{\sum_t X_{t-1}^2} .

The three regimes correspond to familiar series. The daily departure of the temperature from its seasonal normal returns towards zero after a warm or cold spell: the stationary case. The price of a share, or an exchange rate, behaves much like a random walk, with no level to which it is drawn back: the unit root. Prices during a hyperinflation, or an asset price during a speculative bubble, grow at an accelerating rate: the explosive case. Tests for speculative bubbles in housing and stock markets ask precisely whether the coefficient has moved above 1.

It resembles an ordinary regression slope, but depending on the true kk it behaves like three different estimators. The theory took about fifty years to develop. The simulation below illustrates it.

Five contributions define the theory.

  • Mann and Wald (1943) settled the stationary case, ∣k∣<1|k| < 1: the estimator is consistent and asymptotically normal, and this holds for any independent errors with finite moments, normal or not.
  • White (1958) solved the explosive case, ∣k∣>1|k| > 1, with normal errors and a zero starting value: after the right scaling, the limit is a Cauchy distribution.
  • Anderson (1959) showed that in the explosive case the limit depends on the distribution of the errors.
  • Dickey and Fuller (1979) tabulated the unit-root case, k=1k = 1, where the limit is neither normal nor Cauchy. Their tables are why unit-root tests carry their own critical values.
  • Phillips (1987) and, independently, Chan and Wei (1987) bridged the regimes with local-to-unity asymptotics, letting k=1+c/nk = 1 + c/n.

Regime 1: stationary, ∣k∣<1|k| < 1

The process forgets its past and fluctuates around zero. Mann and Wald's result is

n (k^−k)  →  N ⁣(0, 1−k2). \sqrt{n}\,(\hat k - k) \;\to\; N\!\left(0,\ 1 - k^2\right).

This is the textbook regime: standard errors shrink like 1/n1/\sqrt{n} and the usual tt-statistics work. A simulation with k=0.5k = 0.5 and 20,000 replications gives a standard deviation of n(k^−k)\sqrt{n}(\hat k - k) of 0.881 at n=50n = 50, 0.872 at n=200n = 200 and 0.866 at n=1000n = 1000, against the theoretical 0.75≈0.866\sqrt{0.75} \approx 0.866.

Regime 2: unit root, k=1k = 1

At k=1k = 1 the process is a random walk. Shocks never fade, and the variance of XtX_t grows without bound. Two things change at once. The estimator converges faster, at rate nn instead of n\sqrt{n}. And the limit is not normal: n(k^−1)n(\hat k - 1) converges to a functional of Brownian motion, skewed to the left.

n=50n = 50 n=200n = 200 n=1000n = 1000
Mean of n(k^−1)n(\hat k - 1) −1.69 −1.75 −1.78
5% quantile of n(k^−1)n(\hat k - 1) −7.66 −7.99 −8.13
Share of estimates below 1 68% 68% 68%

About two thirds of the estimates fall below the truth, however long the series. A test that used normal critical values would reject a true unit root far too often.

Regime 3: explosive, ∣k∣>1|k| > 1

The series grows geometrically, and so does the information about kk. The estimator converges at the rate ∣k∣n|k|^n, which is extremely fast. White's result is

∣k∣nk2−1 (k^−k)  →  Cauchy. \frac{|k|^n}{k^2 - 1}\,(\hat k - k) \;\to\; \text{Cauchy}.

Anderson's result explains why. In the other two regimes the estimate averages over many comparable shocks, and a central limit theorem removes the effect of the shape of the errors. In the explosive regime the first few shocks are amplified by knk^n and dominate everything after them. Nothing averages out, so the shape of the errors survives into the limit, and White's Cauchy relies on those errors being normal.

Comparison

The standard deviation of k^\hat k at n=400n = 400:

  • k=0.5k = 0.5: about 0.044
  • k=1k = 1: about 0.008
  • k=1.05k = 1.05: about 6×10−86 \times 10^{-8}

With the same formula and sample size, the precision differs by seven orders of magnitude. The local-to-unity framework of Phillips, Chan and Wei connects these regimes: with k=1+c/nk = 1 + c/n, the limit is a functional of an Ornstein-Uhlenbeck process that moves continuously from stationary to explosive behaviour as cc changes sign. It is the basis of most modern inference near a unit root.

The figure below shows the effect of moving the coefficient across 1: below 1 the scaled error is normal with variance 1−k21-k^2; at 1 it is skewed with about 68% of estimates below the truth; above 1 it is Cauchy; and the precision differs by orders of magnitude.

Each run simulates 4,000 series and draws the histogram of the estimation error, scaled appropriately for the current regime.

Suggested settings:

  1. Unit root. At k=1k = 1 (the default), the histogram is skewed, the "below the truth" readout shows 68% and the 5% quantile is near −8. Change nn from 100 to 400: the shape barely changes, because the scaling by nn is correct.
  2. Stationary case. Set k=0.5k = 0.5. The histogram becomes a symmetric bell sitting on the dashed normal curve with variance 0.750.75, and "below the truth" falls to just over 50%.
  3. Near the unit root. Set k=0.95k = 0.95 with n=100n = 100. The normal approximation is visibly wrong: the histogram leans left. This is the region where local-to-unity theory is needed. Increase nn to 400 and the fit improves.
  4. Explosive case. Set k=1.05k = 1.05 and n=400n = 400. The histogram now matches the Cauchy curve, with its narrow peak and heavy tails, and the "SD of the estimate" readout falls to around 10−810^{-8}. Back at n=100n = 100 it is far larger and varies strongly from run to run: with Cauchy tails, a few extreme estimates dominate any standard deviation.

Derivation for the stationary case. Write k^−k=∑Xt−1et/∑Xt−12\hat k - k = \sum X_{t-1} e_t / \sum X_{t-1}^2. By the law of large numbers the denominator divided by nn tends to Var⁡(X)=1/(1−k2)\operatorname{Var}(X) = 1/(1-k^2) for unit-variance errors. The numerator is a sum of martingale differences with variance n/(1−k2)n/(1-k^2), so divided by n\sqrt n it tends to N(0,1/(1−k2))N(0, 1/(1-k^2)). The ratio gives n(k^−k)→N(0,1−k2)\sqrt n(\hat k - k) \to N(0, 1-k^2). At k=1k = 1 the denominator grows like n2n^2, not nn, which is where the faster rate and the non-normal limit come from.

Practical implications:

  • Standard errors are regime-dependent. The familiar (1−k^2)/n\sqrt{(1-\hat k^2)/n} is valid only well inside the stationary region. Near k=1k = 1 it misleads, and at k=1k = 1 it collapses to zero.
  • Normality of the errors matters least where it is usually checked. In the stationary case the asymptotics are robust to the error distribution. In the explosive case they are not.
  • The simplest model is a useful benchmark. A method that mishandles the AR(1) will not handle dependence in harder models either.

For an analyst asking whether house prices have entered a bubble, the first point is decisive. Near k=1k = 1 the textbook confidence interval is misleading, and the judgement has to rest on the non-normal distributions tabulated for the unit-root case.

References

  1. H. B. Mann and A. Wald, "On the statistical treatment of linear stochastic difference equations", Econometrica, 11(3/4), 173-220, 1943.
  2. J. S. White, "The limiting distribution of the serial correlation coefficient in the explosive case", Annals of Mathematical Statistics, 29(4), 1188-1197, 1958.
  3. T. W. Anderson, "On asymptotic distributions of estimates of parameters of stochastic difference equations", Annals of Mathematical Statistics, 30(3), 676-687, 1959.
  4. D. A. Dickey and W. A. Fuller, "Distribution of the estimators for autoregressive time series with a unit root", Journal of the American Statistical Association, 74(366), 427-431, 1979.
  5. P. C. B. Phillips, "Towards a unified asymptotic theory for autoregression", Biometrika, 74(3), 535-547, 1987.
  6. N. H. Chan and C. Z. Wei, "Asymptotic inference for nearly nonstationary AR(1) processes", Annals of Statistics, 15(3), 1050-1063, 1987.