The Wiener filter
To remove noise from a signal, each frequency should be scaled by the share of its power that belongs to the signal rather than cut sharply. Norbert Wiener showed that this is the optimal linear filter. A simulation shows the cost of misjudging the noise.
When a radio signal fades into noise, raising the volume amplifies both signal and noise, and reducing the treble attenuates the noise together with part of the signal. For each frequency there is an optimal gain, and in 1942 Norbert Wiener derived it.
His report was classified during the war, became known among engineers as "the yellow peril" for its yellow cover and its difficulty, and was published openly in 1949. The result can be stated as a single fraction. This note derives it, applies it to a noisy signal, and shows the effect of misjudging the noise.
Wiener's book (1949) solved the problem of estimating a stationary signal from a noisy observation with the smallest mean squared error, using spectral methods built on his own generalised harmonic analysis. Kolmogorov (1941) had solved the discrete-time prediction problem independently, which is why the theory is often credited to both. Kalman's 1960 recursive filter, the subject of the Kalman filter as recursive Bayesian estimation, later solved the same problem step by step and without assuming stationarity; Kailath's 1974 survey traces how the two lines of work connect. The frequency-domain view of a stationary series is the one introduced in the power spectrum.
The problem
The observation is : a signal plus independent noise, both stationary, with power spectra and . The objective is a linear filter whose output is as close as possible to in mean squared error.
The optimal filter
For stationary processes, components at different frequencies are uncorrelated. So the problem splits into independent problems, one per frequency. At frequency , the observation's component is the signal's component plus the noise's component, and a gain is chosen to multiply it by. The error at that frequency has power
the lost signal plus the transmitted noise. Minimising over gives
Where the signal dominates, : the component is passed. Where the noise dominates, : the component is suppressed. In between, the gain equals the signal's share of the power. There is no sharp cut-off and no parameter to tune.
A photograph taken in dim light shows the same structure in two dimensions. The large shapes of the scene carry most of their power at low spatial frequencies, while the grain from the camera sensor is spread evenly across all frequencies. A filter of this kind keeps the shapes and suppresses the fine grain, giving up some of the finest detail where grain and detail cannot be separated.
Minimum mean squared error
At the optimum, the error power at frequency is , so the smallest achievable mean squared error is
No linear filter can achieve a lower error. If signal and noise are jointly Gaussian, no filter at all can.
Required knowledge
The limitation lies in the formula: depends on and , which in practice must be estimated or assumed. Underestimate the noise and the filter lets too much of it through. Overestimate it and the filter smooths away real signal. In the photograph, the first error leaves visible grain; the second produces the waxy skin and smeared foliage familiar from aggressive noise reduction in phone cameras. Both errors raise the mean squared error above the MMSE, and the figure shows by how much.
The signal is a smooth AR(1) process; the noise is white, so its spectrum is flat.
Suggested settings:
- Default settings. With the defaults (smoothness 0.9, noise variance 2) the signal's power is concentrated at low frequencies, so the gain is close to 1 there and falls towards 0 where the flat noise spectrum takes over. The filtered error is around 0.3, from 2 for the raw observation, close to the theoretical minimum of about 0.31.
- Misjudged noise. Press "thinks it is 4× smaller": the gain stays high at frequencies where it should not, and the error rises. Press "4× larger": the estimate becomes too smooth and misses real variation. Either way the error exceeds the theoretical floor.
- White signal. Lower the smoothness to 0. Now the signal is white too: both spectra are flat, the gain is the same constant at every frequency, and filtering can only shrink the observation towards zero.
- Low noise. Set the noise to 0.05. The gain is close to 1 almost everywhere and the filter does almost nothing, which is the correct behaviour.
Optimality of S/(S + N). Fix one frequency and write the components as complex numbers with , and . The error is , so
For real , the derivative vanishes at , and substituting gives the minimum . Because the components at different frequencies of a stationary process are uncorrelated, minimising each frequency separately minimises the total. The widget applies the gain to a 256-point discrete Fourier transform, a circular approximation of the same idea.
The Wiener filter is the frequency-domain counterpart of the Kalman filter: one assumes stationarity and solves the problem all at once; the other works step by step and allows the world to change. Both weight each source by its share of signal. A deblurring algorithm, a noise-cancelling headset and a smoothing spline all use methods derived from that fraction.
References
- A. N. Kolmogorov, "Interpolation and extrapolation of stationary random sequences" (in Russian), Izvestiya Akademii Nauk SSSR, Seriya Matematicheskaya, 5, 3-14, 1941.
- N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series, MIT Press and Wiley, 1949.
- R. E. Kalman, "A new approach to linear filtering and prediction problems", Journal of Basic Engineering, 82(1), 35-45, 1960.
- T. Kailath, "A view of three decades of linear filtering theory", IEEE Transactions on Information Theory, 20(2), 146-181, 1974.