Choosing the order of an autoregressive model
The autocorrelation and partial autocorrelation functions suggest an order, and AIC and BIC formalise the choice. AIC overfits in about a quarter of cases regardless of sample size; BIC is consistent but tends to underfit in small samples. A simulation of 500 series per setting shows both effects.
For an autoregressive model, the dependence on the past is determined by the number of lags in
With too few lags the model misses genuine dependence; with too many it fits noise, which degrades forecasts and overstates confidence. The question arises whenever a utility forecasts tomorrow's electricity demand from recent days, or a retailer forecasts next week's sales from past weeks: the model has to decide how far back to look. Choosing is a routine decision in time-series modelling, and it is often made by fitting a few models and selecting the one with the smallest criterion, without considering how often that rule selects the correct order.
Box and Jenkins (1970) made order selection a visual procedure: compare the sample autocorrelation function (ACF) with the partial autocorrelation function (PACF) and match their patterns to known models. Quenouille (1949) had already given the yardstick for the PACF: for a series of length , sample partial autocorrelations beyond the true order are approximately normal with standard deviation . Akaike (1974) replaced the visual procedure with a numerical criterion, the AIC, designed to choose the model that predicts best. Schwarz (1978) derived the BIC from a Bayesian argument, designed to choose the true model when one is on the list. Shibata (1976) proved that AIC does not converge to the true order: even with infinite data, it overfits with positive probability. Hannan and Quinn (1979) found the slowest-growing penalty that still gives consistency.
The ACF and the PACF
The ACF measures the correlation between and , including indirect paths through the lags in between. The PACF measures the correlation at lag after removing what lags already explain; it is the last coefficient of the best AR() fit.
For an AR() process, the PACF cuts off after lag (the extra lags explain nothing new), while the ACF decays gradually. For a moving-average process it is the other way round. This pair of patterns is the basis of the Box-Jenkins identification method. The Durbin-Levinson recursion computes all the partial autocorrelations at once from the sample autocorrelations, and gives the fitted innovation variance for every order as a by-product.
AIC and BIC
Both information criteria trade the fit, measured by , against the number of parameters:
The only difference is the penalty. AIC charges 2 per parameter whatever the sample size; BIC charges , which exceeds 2 as soon as and keeps growing. So BIC is stricter, increasingly so as data accumulate.
Behaviour in repeated samples
In simulations of 500 series each (Yule-Walker fits, orders 0 to 8), the following pattern emerges:
| True model | AIC correct | AIC too large | BIC correct | BIC too small | |
|---|---|---|---|---|---|
| AR(1), | 150 | 73% | 27% | 96% | 0% |
| AR(1), | 1,000 | 72% | 28% | 99% | 0% |
| AR(2), | 150 | 74% | 26% | 96% | 0% |
| AR(3), | 50 | 56% | 18% | 39% | 58% |
| AR(3), | 1,000 | 73% | 27% | 99% | 0% |
AIC overfits about a quarter of the time, and more data do not help. BIC almost always finds the truth with enough data, but with 50 observations of an AR(3) whose third coefficient is modest, it underfits more often than not. Neither criterion is incorrect; they address different questions. AIC asks which model will forecast best, and a slightly too large model costs little in forecasting. BIC asks which model is true, and needs enough evidence before admitting another parameter.
The figure below shows both diagnostics and both criteria.
Suggested settings:
- Identification patterns. With AR(2) selected, the PACF has two clear stems outside the dashed band and nothing after; the ACF decays with oscillation. Switch to MA(1): now the ACF cuts off after lag 1 and the PACF decays.
- AIC overfitting. Press "Repeat 500 times" for AR(2) with . About a quarter of AIC's choices land on 3 or more; BIC's land almost entirely on 2.
- Small samples. Choose AR(3) and set . Repeat: BIC now picks 2 or fewer most of the time, while AIC finds 3 more often. Raise to 1,000 and BIC becomes the more accurate criterion.
- No finite order. With MA(1) there is no true finite AR order. AIC and BIC both choose an order that grows slowly with , which is the appropriate behaviour: an AR approximation with more lags fits better when there is more data to support it.
Inconsistency of AIC. Suppose the true order is and compare it with . Under the true model, the extra sample partial autocorrelation is approximately , and , which is approximately . AIC prefers when this exceeds the penalty difference of 2: probability , constant in . Summing such chances over several larger orders gives the roughly 25% overfitting seen above. BIC's threshold is , so the same probability is : consistency.
The appropriate criterion depends on the purpose of the model. For forecasting, a slightly generous order chosen by AIC is usually adequate. For interpretation, for example to claim that a cycle exists, the stricter BIC and a look at the PACF are safer. A utility forecasting electricity demand loses little from AIC's occasional extra lag. An economist who reports a three-year cycle in house prices on the basis of a lag that is not real reports a pseudo-cycle that does not exist. In both cases the criterion should be reported, and its error rate can be measured, as in statistical sampling in control testing.
References
- M. H. Quenouille, "Approximate tests of correlation in time-series", Journal of the Royal Statistical Society B, 11(1), 68-84, 1949.
- G. E. P. Box and G. M. Jenkins, Time Series Analysis: Forecasting and Control, Holden-Day, 1970.
- H. Akaike, "A new look at the statistical model identification", IEEE Transactions on Automatic Control, 19(6), 716-723, 1974.
- R. Shibata, "Selection of the order of an autoregressive model by Akaike's information criterion", Biometrika, 63(1), 117-126, 1976.
- G. Schwarz, "Estimating the dimension of a model", Annals of Statistics, 6(2), 461-464, 1978.
- E. J. Hannan and B. G. Quinn, "The determination of the order of an autoregression", Journal of the Royal Statistical Society B, 41(2), 190-195, 1979.