March 1, 2014

Calibrating Ensemble Forecasting Models with Sparse Data in the Social Sciences

We consider ensemble Bayesian model averaging (EBMA) in the context of small-n prediction tasks with high rates of missing component forecasts. With a large number of observations to calibrate ensembles and low rates of missing values for each component model, the standard approach to calibrating ensembles introduced by Raftery et al. (2005) performs well. However, data in the social sciences generally do not fulfill these requirements. The number of outcomes being predicted tend to be relatively small and missing predictions are neither random nor rare. In these circumstances, EBMA models may overweight components with low rates of missingness and those that that perform well on the limited calibration sample. This can seriously undermine the advantages of the ensemble approach to prediction. We demonstrate this problem and provide a solution that diminishes these undesirable outcomes by introducing a “wisdom of the crowds” parameter to the standard EBMA framework. We show that this solution improves predictive accuracy of EBMA forecasts in both political and economic applications.

2015.  International Journal of Forecasting 31(3) 930-942.  With Florian Hollenbach and Michael D. Ward.
Local copy (pdf) | External Link