Mathematics
Statistics, probability, time series, information measures and heuristic optimisation: methods I study and the problems they solve.
Recessions as a hidden state: Hamilton's regime-switching model
Whether an economy is in recession is not observed directly and is officially dated months later. Hamilton's Markov-switching model treats expansion and recession as a hidden state, estimates how long each lasts, and computes the probability of recession from the growth figures as they arrive.
7 min readMean reversion in interest rates: the Vasicek model
Interest rates wander but tend to return towards a long-run level. The Vasicek model describes this with a mean-reverting stochastic process, which is an AR(1) model in continuous time, gives bond prices and the yield curve in closed form, and allows negative rates, as the euro area experienced between 2014 and 2022.
7 min readGeometric Brownian motion and the price of an option
Geometric Brownian motion is the standard stochastic model of a share price. It explains why the typical outcome of an investment lies below the average one, and it leads to the Black-Scholes price of an option, which a Monte Carlo simulation reproduces.
7 min readVolatility clustering and the GARCH model
Daily stock returns are almost uncorrelated, yet calm and turbulent periods cluster. The GARCH model describes this with one recursion for the variance; a simulation shows its consequences for value at risk and for the backtesting of bank risk models.
8 min readThe logarithmic Sobolev inequality and the γ-order generalized normal
The normal distribution is the equality case of several inequalities linking entropy and Fisher information. Generalising one of them, the logarithmic Sobolev inequality, yields a family of equality cases: the γ-order generalized normal. A figure tests the inequality on a family of densities.
8 min readBregman divergences and the mean as optimal summary
Squared error, the Kullback-Leibler divergence and the Itakura-Saito distance appear unrelated, but all are Bregman divergences, and for every one of them the optimal single summary of the data is the mean. An interactive figure illustrates the result.
7 min readKullback-Leibler divergence and Hellinger distance
The Kullback-Leibler divergence and the Hellinger distance both measure how different two models are, but they behave differently, and the difference determines which one to use. Both are computed numerically in the figure.
5 min readEstimating the shape parameter of the γ-order generalized normal
Fitting the γ-order generalized normal requires estimating its shape parameter, which has no closed form. A simulation shows how much data this requires, why large values of γ are poorly determined, and which parametrisation should be used instead.
8 min readThe exponential power family and the γ-order generalized normal
The γ-order generalized normal has the same univariate shape as the exponential power family known since 1923. It differs in the origin of that shape and, in several dimensions, in the type of dependence between components. A translator between the two parametrisations is included.
7 min readEstimating an autoregression with non-negative innovations
When the shocks of an autoregressive process are non-negative, the smallest ratio of consecutive observations estimates the coefficient at rate n instead of √n. The result is notable but fragile, as a simulation with slightly negative shocks shows.
6 min readFisher information as the curvature of the log-likelihood
The curvature of the log-likelihood around its maximum determines how much the data reveal about a parameter. The main properties of Fisher information follow from this interpretation, which a simulation illustrates for the normal mean.
5 min readForecast intervals for autoregressive models
A point forecast without an interval is incomplete. For an autoregressive model the interval has a closed form and converges to a known width, but it is too narrow when the model is estimated from little data. A simulation compares nominal and realised coverage.
5 min readThe Wiener filter
To remove noise from a signal, each frequency should be scaled by the share of its power that belongs to the signal rather than cut sharply. Norbert Wiener showed that this is the optimal linear filter. A simulation shows the cost of misjudging the noise.
5 min readThe Kalman filter as recursive Bayesian estimation
The Kalman filter tracks a quantity observed through a noisy sensor by alternating prediction and update, with a gain that determines the weight given to each measurement. A simulation shows the effect of sensor outages and mis-specified noise.
6 min readBayesian inference for the autoregressive coefficient near a unit root
Near a unit root, the classical sampling distribution of the AR coefficient is skewed and requires special tables. The Bayesian posterior, computed from the same simulations, is not. Sims and Uhlig's simulation illustrates the difference.
6 min readRegression with autocorrelated errors: from Cochrane-Orcutt to Durbin's h
Regression with autocorrelated errors produced several well-known methods in econometrics. The most instructive case is the one in which these methods fail: a lagged dependent variable among the regressors. A simulation shows least squares converging to an incorrect value.
7 min readThe stationarity region of the AR(2) model
The stationarity region of a second-order autoregression is a triangle in the plane of its two coefficients. Below a parabola within it, the process has a pseudo-cycle; near the edges, shocks persist for a long time. An interactive map shows the behaviour for any pair of coefficients.
6 min readA spurious unit root from omitting the intercept
Fitting an autoregressive model without an intercept to data far from zero produces an apparent unit root. The effect is shown with Lake Huron's water levels, together with the small-sample bias that remains when the model is correctly specified.
6 min readLeast-squares estimation of the AR(1) coefficient in three regimes
The least-squares estimate of an AR(1) coefficient behaves differently depending on whether the process is stationary, has a unit root or is explosive: the rate of convergence and the limiting distribution differ in each case. A simulation shows the transition across the boundary.
7 min readThe autoregressive model of a time series
The AR model is the simplest way to state that the present depends on the past. This article explains the meaning of its parameter, how quickly a series forgets a shock, how the model forecasts, and where the distribution of the shocks matters.
6 min readSequential Bayesian updating of a failure rate
A prior belief about a failure rate is updated with test results to give a posterior belief. With a conjugate prior the update consists of two additions, the posterior narrows approximately as 1/√n, and it can be compared directly with the classical Clopper-Pearson bound.
5 min readCorrelation, independence and transfer entropy beyond the normal distribution
For normal variables, zero correlation implies independence and Granger causality is equivalent to transfer entropy. For heavier- or lighter-tailed families neither holds, and the γ-order generalized normal shows how the difference arises. A scatter plot illustrates it.
8 min readBiased randomisation of greedy heuristics
Greedy heuristics are fast but deterministic. Pure random search explores but wastes effort. Biased randomisation with a geometric distribution combines the two with a single parameter, as a comparison on a hard knapsack instance shows.
5 min readInput distributions in simulation models
Simulation models depend on their input distributions. Three small experiments, on distribution fitting, a newsvendor model and two queues, show how strongly the shape of the inputs affects the results and how easily it is misjudged.
7 min readRandom variate generation for the γ-order generalized normal
Simulation studies require a generator for the distribution under study rather than a normal approximation. This article presents two exact methods for sampling the γ-order generalized normal: a gamma transform and an adapted Ziggurat algorithm.
6 min readThe γ-order generalized normal distribution
One additional parameter allows a model's errors to have lighter or heavier tails than the normal distribution. This article presents the origin and properties of the γ-order generalized normal, with two interactive figures showing the behaviour of the tails.
6 min readThe power spectrum and its estimation
A stationary series can be described by the distribution of its variance over frequencies. Estimating this distribution is difficult: the periodogram does not converge, however much data is available. Welch averaging and the spectrogram address this, as an AR(2) simulation shows.
6 min readSearching for cycles in 150 years of Rhine discharge
Annual river flows are often expected to contain cycles. Three independent methods, the periodogram, autoregressive models and wavelets, agree that the annual flow of the Rhine at Basel is consistent with white noise. A simulation shows how convincing apparent cycles in pure noise can be.
6 min readChoosing the order of an autoregressive model
The autocorrelation and partial autocorrelation functions suggest an order, and AIC and BIC formalise the choice. AIC overfits in about a quarter of cases regardless of sample size; BIC is consistent but tends to underfit in small samples. A simulation of 500 series per setting shows both effects.
6 min readStatistical comparison of genetic algorithm parameter settings
Genetic algorithms have many parameters, and a single successful run proves little. A sound comparison of settings requires repeated runs and non-parametric tests, which can change the conclusion. The comparison is demonstrated with thirty seeds per setting.
6 min readThe value of the stochastic solution in two-stage planning
Replacing uncertain quantities by their averages yields a plan that is optimal only for the average scenario. Stochastic programming measures the cost of this simplification through the value of the stochastic solution, illustrated with a simple inventory problem.
5 min readYule's sunspot model and the origin of autoregression
In 1927 George Udny Yule replaced the idea of hidden periodicities with that of a damped pendulum disturbed by random shocks. This picture led to the autoregressive model. A simulation reproduces Yule's model of the sunspot cycle.
6 min readMarkov chain Monte Carlo and the random-walk Metropolis algorithm
Markov chain Monte Carlo draws samples from a posterior distribution that has no closed form. This article presents the random-walk Metropolis algorithm, the effect of the step size, and two convergence diagnostics: the effective sample size and the Gelman-Rubin statistic.
6 min read