Autoregressive Models

Autoregressive models predict future values using past values.

Description

The simplest autoregressive model is AR(1), a model that regresses X_t on X_t-1.

This model is structured as

X_t = φX_t-1 + ε_t

which can be fit with OLS.

The intuition is that if a process is stationary, then future values should be predictable using past values. Errors carry forward infinitely in the model.

Importantly though, this model is only weak sense stationary (WSS) if 1 < |φ|. The intuition is that, while each period's error term carries forward infinitely, they must approach zero as t goes to infinity. Weak sense stationarity only requires that the first moment and autocovariance are stationary and that the second moment is finite. (Strict stationarity requires that all moments be stationary.)

If |φ| < 1 holds, future values of the time series would be trending away from past values. This clearly cannot be a stationary process. Note that if the parameter were equal to 1, then Var(X_t) depends on t, so the variance goes to infinity as the time series goes to infinity.

In the case that the time series is weak sense stationary:

E[X_t] = φE[X_t-1] + E[ε_t]

μ = φμ + 0

μ = 0

Var(X_t) = φ²Var(X_t-1) + σ_ε² = E[X_t²] - μ² = σ_ε² / (1-φ²)

Generalizations

An autoregressive model is notated as AR(p) where p is the number of lags that predict a future value.

Incorporating a moving average (MA) model makes this an autoregressive moving average (ARMA) model. The general ARMA(p, q) model is written as:

The important difference between the AR and MA models is that errors do not carry forward infinitely in the latter. Its general form is notated as MA(q) where q is the number of lags that predict a future value. The intuition is reversion to the mean and an MA model is always stationary.

So the ARMA model, notated as ARMA(p, q), is applicable to systems that feature both trending as well as errors (shocks) that revert to the mean. Interpretation relies upon the model being stationary, pending the φ parameters.

Generally p and q are set to the minimums that allow a good fit of the model.

If instead of being stationary, the system has a known or periodic trend, the stationary component can be isolated by differencing. This leads to the autoregressive integrated moving average (ARIMA) model, generally notated as ARIMA(p, d, q) where d is the number of lagged values subtracted to create a stationary time series.

In some cases, ARIMA models are structured and fit with zero components rather than using separate notations and software implementations. As an example, a random walk is sometimes described as ARIMA(0, 1, 0).

CategoryRicottone