Frequentist model averaging

In prediction, considering multiple models, rather than just an individual model, is a way to manage the uncertainty about which of a set of individual models is likely to give the best results.

Model selection methods are a way to choose just one of multiple models and reject the rest. Model averaging methods are a way to combine multiple models in a weighted combination. Model averaging may have advantages over model selection, especially when (a) several of the models are likely to perform similarly well, and choosing just one of them is a fairly arbitrary choice, or (b) one is interested in ensuring that the entire range of reasonable possible predictions is captured, for example in risk management. On the other hand, model selection may have advantages over model averaging in a situation in which one of the models is likely to perform much better than all the other models.

Model averaging can be performed based on either frequentist or Bayesian ideas, or a mixture of the two. There are pros and cons to the two approaches.

This article is about frequentist model averaging (FMA) methods.

Types of Frequentist Model Averaging

What follows is a list of types of frequentist model averaging taken from the recent academic literature. The list is not exhaustive, and some of the items overlap. The list is ordered from simple, standard and well-used methods to more complex, more unusual methods.

Plug-in model averaging (PMA)

PMA is perhaps the simplest form of frequentist model averaging. The goal of PMA is to produce more accurate point forecasts, where more accurate is defined as lower predictive mean squared error (PMSE). To derive PMA, an expression for the predictive mean squared error PMSE is written in terms of the model weights and the true values for the model parameters. This expression is then maximised by varying the weights, giving the weights that would maximise PMSE, as a function of the true values for the model parameters. The true values for the model parameters are then replaced by the estimated values for the model parameters (the estimated values are simply ‘plugged in’). PMA has the advantage that the derivation is the simplest and easiest to understand of any of the FMA methods. Some authors have reported good results from PMA, including for probabilistic forecasting.

AIC model averaging (AICMA)

In AIC model averaging, models fitted using maximum likelihood are combined using weights derived from the Akaike information criterion (AIC) score, known as Akaike weights. The goal of AICMA is to produce more accurate probabilistic forecasts, where more accurate is defined as lower Kullback-Leibler divergence. AICMA has the advantages that AIC is well known and well accepted, and that AICMA is simple. It has the disadvantage that it only applies to models fitted using maximum likelihood estimation. AICMA can also be based on AICc, which is a small sample variant of AIC. There is also a variant of AICMA known as smoothed AICMA.

BIC model averaging (BICMA)

In BIC model averaging, models fitted using maximum likelihood are combined using weights based on the Bayesian information criterion (BIC) score. . Like AICMA, BICMA has the advantages that BIC is well known and well accepted, and that BICMA is simple. It again has the disadvantage that it only applies to models fitted using maximum likelihood estimation. There is also a variant of BICMA known as smoothed BICMA.

Cross-validation model averaging (CVMA)

In CVMA, cross-validation is used to determine weights for a given criterion. Many different criteria are possible, such as predictive mean squared error, or predictive mean log-likelihood (PMLL). CVMA can also be applied to models fitted in different ways (i.e., is not restricted to maximum likelihood models). This flexibility of criterion and models is the big advantage of CVMA relative to e.g., AIC and BIC.

Jack-knife model averagin (JMA)

JMA is a special case of CVMA, in which the cross-validation is performed using a jack-knife.

Focussed model averaging (FMA)

.

Least Squares model averaging (LSMA)

LSMA methods are a class of model averaging methods that are designed to minimise the predictive mean squared error (PMSE), by averaging across models fitted using least squares. Note that PMSE is known as the mean square forecast error (MSFE). Example of LSMA methods from this list are MMA, PMA, and also JMA if the criterion used in the JMA is predictive mean squared error (PMSE).

LSMA has also been known as mean squared error model averaging, although using the name least squares model averaging and the abbreviation LSMA avoids a clash with the abbreviation MMA which has been used for Mallows model averaging.

Mallows model averaging (MMA)

In MMA, models fitted using least squares are combined using weights derived from Mallows Cp. MMA is a type of LSMA.

Bayesian plug-in model averaging (BPMA)

Bayesian plug-in model averaging is an extension of PMA, in which the weights are calculated as the Bayesian expectation over all possible values for the weights. .

WAIC model averaging (WMA)

In WMA model averaging, models are combined using weights based on the WAIC score instead of the AIC score.

Performance

In simulation tests, no model averaging method dominates the other methods for all values of the real parameters. Typically each method performs well for a range of real parameter values. Simulation results are given in many places in the literature.

Applications in Atmospheric Science

FMA methods have been used to estimate the rate of climate change and make predictions of future climate. Since estimates of the rate of climate change may be highly uncertain, for certain climate variables, the best predictions may come from any of the following three methods:

• assuming that the rate of climate change is zero. This is likely to give the best results when the rate of climate change is so hard to estimate that assuming it is zero gives better predictions. This is typically when the uncertainty around the estimate of the rate is much greater than the rate itself.
• estimating the rate of climate change using unbiased estimation methods (such as maximum likelihood). This is likely to give the best results when the rate of climate change can be estimated relatively accurately. This is typically when the uncertainty around the estimate of the rate is much less than the rate itself.
• using FMA to combine a change of zero with an unbiased estimate, giving something in-between. This is likely to give the best results when the rate of climate change is roughly similar to the uncertainty around the estimate of the rate.

Specific examples from the atmospheric science literature include:

• Estimating the recent trend in local temperatures from local observations using PMA to combine ‘no trend’ and ‘OLS trend’.

• Estimating the recent trend in regional rainfall indices from observations using JMA, AICMA, BICMA and PMA.

• Estimating the future change in regional rainfall indices from climate models using AICMA, PMA and BPMA.

• add new reference

References

This article "Frequentist model averaging" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Frequentist model averaging. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

---