Jorge Quitério Search
Mathematics

Estimating an autoregression with non-negative innovations

When the shocks of an autoregressive process are non-negative, the smallest ratio of consecutive observations estimates the coefficient at rate n instead of √n. The result is notable but fragile, as a simulation with slightly negative shocks shows.

Many series cannot go negative: river flows, prices, the number of people in a queue, the time between failures. If such a series follows a first-order autoregression,

Xt=k Xt−1+et,0≤k<1, X_t = k\,X_{t-1} + e_t, \qquad 0 \le k < 1,

then its shocks ete_t are naturally non-negative too. A hospital ward illustrates the model. Each day a fraction kk of yesterday's patients remain, and new admissions arrive; admissions cannot be negative. On a day with almost no admissions, today's occupancy divided by yesterday's is almost exactly the share of patients who stayed. Over a long record, the smallest such ratio is therefore a precise estimate of kk.

This property changes the estimation problem fundamentally. It allows a very simple estimator that is about forty times more accurate than least squares on a thousand observations.

The advantage depends on a strict assumption. This article explains the estimator, proves why it works, and shows how it fails when the assumption is violated.

The relevant literature is as follows.

  • Bell and Smith (1986) studied inference for non-negative autoregressions, including the minimum-ratio estimator.
  • Anděl (1988, 1989) analysed AR(1) processes with exponential shocks and extended the idea to higher orders.
  • Davis and McCormick (1989) derived the limiting distribution for positive or bounded shocks using extreme-value theory rather than the central limit theorem.
  • Nielsen and Shephard (2003) gave the exact likelihood theory for exponential innovations and showed that the minimum ratio is then the maximum-likelihood estimator.

The AR(1) model itself, and the behaviour of least squares in it, are the subject of the autoregressive model of a time series and least-squares estimation of the AR(1) coefficient in three regimes.

The minimum ratio

Divide the model by Xt−1X_{t-1}:

XtXt−1=k+etXt−1. \frac{X_t}{X_{t-1}} = k + \frac{e_t}{X_{t-1}} .

Because et≥0e_t \ge 0 and Xt−1>0X_{t-1} > 0, every ratio of consecutive observations is at least kk. So the smallest ratio,

k~=min⁡tXtXt−1, \tilde k = \min_t \frac{X_t}{X_{t-1}} ,

can never fall below the true coefficient, and it approaches kk from above whenever a shock happens to be close to zero. With exponential shocks the likelihood is maximised on that boundary, so k~\tilde k is also the maximum-likelihood estimator.

Rate of convergence

Least squares converges at rate 1/n1/\sqrt{n}. The minimum ratio converges at rate 1/n1/n. A simulation with k=0.5k = 0.5, exponential shocks and 20,000 replications:

nn Least squares (with intercept) Minimum ratio Ratio
50 0.127 0.0139 9
200 0.062 0.0035 18
1000 0.027 0.0007 39

All 60,000 minimum-ratio estimates were at or above the truth, as the algebra implies. Each fourfold increase in sample size halves the least-squares error and quarters the minimum-ratio error.

Information at the boundary

The same phenomenon appears in a classic exercise: estimating the upper limit θ\theta of a uniform distribution on [0,θ][0, \theta]. The sample maximum converges at rate 1/n1/n, much faster than anything built from averages. In both cases the density has an edge, and the observations closest to it pin it down.

Regular asymptotic theory, with its 1/n1/\sqrt{n} rates, Fisher information and Cramér-Rao bound (see Fisher information as the curvature of the log-likelihood), assumes a smooth density whose support does not depend on the parameter. A sharp boundary breaks those assumptions, and the estimator exploits this.

Sensitivity to the assumption

The boundary is also the weakness. The argument needs et≥0e_t \ge 0 to hold exactly. Shift the exponential shocks down by just 0.1, so that about one in ten is negative, and at n=200n = 200 the minimum ratio averages 0.34 instead of 0.5, with an error of about 0.18, three times worse than least squares. One negative shock is enough to pull the minimum below kk, and with enough observations there will always be one.

Real data rarely respect a boundary exactly: measurement error, rounding or an approximate model all create small violations. In the ward example, a single day on which a recording error lists fewer patients than were present produces a ratio below kk, and the estimator then reports a shorter average stay than the true one. The minimum ratio is best seen as a diagnostic, or as the starting point for estimators that keep some of its efficiency while tolerating noise near the boundary.

The figure below illustrates two results: with non-negative shocks the minimum ratio is never below the truth and its error shrinks like 1/n1/n, while least squares shrinks like 1/n1/\sqrt n; and a small violation of the boundary removes the advantage.

Each run simulates 2,000 series with k=0.5k = 0.5 and exponential shocks.

Suggested settings:

  1. Default settings. At n=200n = 200, the minimum-ratio histogram is a thin spike just to the right of 0.5, while least squares is a wide bell. The readout says roughly 17 times more accurate.
  2. Rates of convergence. Go from n=50n = 50 to n=200n = 200 to n=1000n = 1000. The least-squares error roughly halves at each fourfold step; the minimum-ratio error roughly quarters. The advantage grows from about 9 to about 40.
  3. Negative shocks. Shift the shocks down by 0.1. Now about 10% of shocks are negative, the spike moves well to the left of the truth, and the readout flips to "worse".
  4. Threshold of failure. At n=1000n = 1000, set a shift of 0.01: only 1% of shocks are negative, and the advantage falls from about 40 times to about 2 times. At 0.03 the minimum ratio is already worse than least squares. More data does not help: with more observations, a rare violation becomes a certain one.

Derivation of the 1/n1/n rate. Given the past, the event {Xt/Xt−1>k+ε}\{X_t / X_{t-1} > k + \varepsilon\} is {et>εXt−1}\{e_t > \varepsilon X_{t-1}\}, which for unit-mean exponential shocks has probability e−εXt−1e^{-\varepsilon X_{t-1}}. Multiplying over tt, because each shock is independent of the past,

P(k~−k>ε)≈exp⁡ ⁣(−ε∑tXt−1)≈exp⁡(−ε n μ),μ=E[X]=11−k. P(\tilde k - k > \varepsilon) \approx \exp\!\Big(-\varepsilon \sum_t X_{t-1}\Big) \approx \exp(-\varepsilon\, n\, \mu), \qquad \mu = E[X] = \frac{1}{1-k}.

So n(k~−k)n(\tilde k - k) is approximately exponential with rate μ\mu, and the root mean squared error is 2/(nμ)=2(1−k)/n\sqrt{2}/(n\mu) = \sqrt{2}(1-k)/n. For k=0.5k = 0.5 and n=200n = 200 that is 0.00350.0035, the value in the table. Least squares has error about (1−k2)/n≈0.061\sqrt{(1-k^2)/n} \approx 0.061. The ratio grows like n\sqrt{n}.

Practical implications:

  • The shape of the shocks can matter more than the model. The same AR(1) equation admits estimators that differ by a factor of forty, depending on what is known about ete_t.
  • Efficiency and robustness conflict. The estimator that uses the assumption most is the one that breaks most easily when it fails.
  • Examine the tails and boundaries as well as the centre. Least squares uses only the first two moments of the shocks, so everything else about their distribution goes unused.

References

  1. C. B. Bell and E. P. Smith, "Inference for non-negative autoregressive schemes", Communications in Statistics to Theory and Methods, 15(8), 2267-2293, 1986.
  2. J. Anděl, "On AR(1) processes with exponential white noise", Communications in Statistics to Theory and Methods, 17(5), 1481-1495, 1988.
  3. J. Anděl, "Non-negative autoregressive processes", Journal of Time Series Analysis, 10(1), 1-11, 1989.
  4. R. A. Davis and W. P. McCormick, "Estimation for first-order autoregressive processes with positive or bounded innovations", Stochastic Processes and their Applications, 31(2), 237-250, 1989.
  5. B. Nielsen and N. Shephard, "Likelihood analysis of a first-order autoregressive model with exponential innovations", Journal of Time Series Analysis, 24(3), 337-344, 2003.