Jorge Quitério Search
Mathematics

Choosing the order of an autoregressive model

The autocorrelation and partial autocorrelation functions suggest an order, and AIC and BIC formalise the choice. AIC overfits in about a quarter of cases regardless of sample size; BIC is consistent but tends to underfit in small samples. A simulation of 500 series per setting shows both effects.

For an autoregressive model, the dependence on the past is determined by the number of lags pp in

Xt=ϕ1Xt−1+⋯+ϕpXt−p+et ? X_t = \phi_1 X_{t-1} + \dots + \phi_p X_{t-p} + e_t \, ?

With too few lags the model misses genuine dependence; with too many it fits noise, which degrades forecasts and overstates confidence. The question arises whenever a utility forecasts tomorrow's electricity demand from recent days, or a retailer forecasts next week's sales from past weeks: the model has to decide how far back to look. Choosing pp is a routine decision in time-series modelling, and it is often made by fitting a few models and selecting the one with the smallest criterion, without considering how often that rule selects the correct order.

Box and Jenkins (1970) made order selection a visual procedure: compare the sample autocorrelation function (ACF) with the partial autocorrelation function (PACF) and match their patterns to known models. Quenouille (1949) had already given the yardstick for the PACF: for a series of length nn, sample partial autocorrelations beyond the true order are approximately normal with standard deviation 1/n1/\sqrt n. Akaike (1974) replaced the visual procedure with a numerical criterion, the AIC, designed to choose the model that predicts best. Schwarz (1978) derived the BIC from a Bayesian argument, designed to choose the true model when one is on the list. Shibata (1976) proved that AIC does not converge to the true order: even with infinite data, it overfits with positive probability. Hannan and Quinn (1979) found the slowest-growing penalty that still gives consistency.

The ACF and the PACF

The ACF measures the correlation between XtX_t and Xt−hX_{t-h}, including indirect paths through the lags in between. The PACF measures the correlation at lag hh after removing what lags 1,…,h−11, \dots, h-1 already explain; it is the last coefficient ϕhh\phi_{hh} of the best AR(hh) fit.

For an AR(pp) process, the PACF cuts off after lag pp (the extra lags explain nothing new), while the ACF decays gradually. For a moving-average process it is the other way round. This pair of patterns is the basis of the Box-Jenkins identification method. The Durbin-Levinson recursion computes all the partial autocorrelations at once from the sample autocorrelations, and gives the fitted innovation variance σ^p2\hat\sigma^2_p for every order as a by-product.

AIC and BIC

Both information criteria trade the fit, measured by σ^p2\hat\sigma^2_p, against the number of parameters:

AIC(p)=nln⁡σ^p2+2p,BIC(p)=nln⁡σ^p2+pln⁡n. \text{AIC}(p) = n \ln \hat\sigma^2_p + 2p, \qquad \text{BIC}(p) = n \ln \hat\sigma^2_p + p \ln n .

The only difference is the penalty. AIC charges 2 per parameter whatever the sample size; BIC charges ln⁡n\ln n, which exceeds 2 as soon as n≥8n \ge 8 and keeps growing. So BIC is stricter, increasingly so as data accumulate.

Behaviour in repeated samples

In simulations of 500 series each (Yule-Walker fits, orders 0 to 8), the following pattern emerges:

True model nn AIC correct AIC too large BIC correct BIC too small
AR(1), ϕ=0.7\phi = 0.7 150 73% 27% 96% 0%
AR(1), ϕ=0.7\phi = 0.7 1,000 72% 28% 99% 0%
AR(2), (0.75,−0.5)(0.75, -0.5) 150 74% 26% 96% 0%
AR(3), (0.5,0.2,−0.35)(0.5, 0.2, -0.35) 50 56% 18% 39% 58%
AR(3), (0.5,0.2,−0.35)(0.5, 0.2, -0.35) 1,000 73% 27% 99% 0%

AIC overfits about a quarter of the time, and more data do not help. BIC almost always finds the truth with enough data, but with 50 observations of an AR(3) whose third coefficient is modest, it underfits more often than not. Neither criterion is incorrect; they address different questions. AIC asks which model will forecast best, and a slightly too large model costs little in forecasting. BIC asks which model is true, and needs enough evidence before admitting another parameter.

The figure below shows both diagnostics and both criteria.

Suggested settings:

  1. Identification patterns. With AR(2) selected, the PACF has two clear stems outside the dashed band and nothing after; the ACF decays with oscillation. Switch to MA(1): now the ACF cuts off after lag 1 and the PACF decays.
  2. AIC overfitting. Press "Repeat 500 times" for AR(2) with n=150n = 150. About a quarter of AIC's choices land on 3 or more; BIC's land almost entirely on 2.
  3. Small samples. Choose AR(3) and set n=50n = 50. Repeat: BIC now picks 2 or fewer most of the time, while AIC finds 3 more often. Raise nn to 1,000 and BIC becomes the more accurate criterion.
  4. No finite order. With MA(1) there is no true finite AR order. AIC and BIC both choose an order that grows slowly with nn, which is the appropriate behaviour: an AR approximation with more lags fits better when there is more data to support it.

Inconsistency of AIC. Suppose the true order is pp and compare it with p+1p + 1. Under the true model, the extra sample partial autocorrelation ϕ^p+1,p+1\hat\phi_{p+1,p+1} is approximately N(0,1/n)N(0, 1/n), and nln⁡(σ^p2/σ^p+12)=−nln⁡(1−ϕ^p+1,p+12)≈nϕ^p+1,p+12n\ln(\hat\sigma^2_p/\hat\sigma^2_{p+1}) = -n\ln(1 - \hat\phi^2_{p+1,p+1}) \approx n\hat\phi^2_{p+1,p+1}, which is approximately χ12\chi^2_1. AIC prefers p+1p + 1 when this exceeds the penalty difference of 2: probability P(χ12>2)≈0.157P(\chi^2_1 > 2) \approx 0.157, constant in nn. Summing such chances over several larger orders gives the roughly 25% overfitting seen above. BIC's threshold is ln⁡n\ln n, so the same probability is P(χ12>ln⁡n)→0P(\chi^2_1 > \ln n) \to 0: consistency.

The appropriate criterion depends on the purpose of the model. For forecasting, a slightly generous order chosen by AIC is usually adequate. For interpretation, for example to claim that a cycle exists, the stricter BIC and a look at the PACF are safer. A utility forecasting electricity demand loses little from AIC's occasional extra lag. An economist who reports a three-year cycle in house prices on the basis of a lag that is not real reports a pseudo-cycle that does not exist. In both cases the criterion should be reported, and its error rate can be measured, as in statistical sampling in control testing.

References

  1. M. H. Quenouille, "Approximate tests of correlation in time-series", Journal of the Royal Statistical Society B, 11(1), 68-84, 1949.
  2. G. E. P. Box and G. M. Jenkins, Time Series Analysis: Forecasting and Control, Holden-Day, 1970.
  3. H. Akaike, "A new look at the statistical model identification", IEEE Transactions on Automatic Control, 19(6), 716-723, 1974.
  4. R. Shibata, "Selection of the order of an autoregressive model by Akaike's information criterion", Biometrika, 63(1), 117-126, 1976.
  5. G. Schwarz, "Estimating the dimension of a model", Annals of Statistics, 6(2), 461-464, 1978.
  6. E. J. Hannan and B. G. Quinn, "The determination of the order of an autoregression", Journal of the Royal Statistical Society B, 41(2), 190-195, 1979.