Model Combination Methods Research Articles

Impact of combination methods on extreme precipitation projections

AbstractClimate change is expected to increase the frequency and intensity of extreme weather events. To properly assess the increased economical risk of these events, actuaries can gain in relying on expert models/opinions from multiple different sources, which requires the use of model combination techniques. From non-parametric to Bayesian approaches, different methods rely on varying assumptions potentially leading to very different results. In this paper, we apply multiple model combination methods to an ensemble of 24 experts in a pooling approach and use the differences in outputs from the different combinations to illustrate how one can gain additional insight from using multiple methods. The densities obtained from pooling in Montreal and Quebec City highlight the significant changes in higher quantiles obtained through different combination approaches. Areal reduction factor and quantile projected changes are used to show that consistency, or lack thereof, across approaches reflects the uncertainty of combination methods. This shows how an actuary using multiple expert models should consider more than one combination method to properly assess the impact of climate change on loss distributions, seeing as a single method can lead to overconfidence in projections.

Combining Predictions of Auto Insurance Claims

This paper aims to better predict highly skewed auto insurance claims by combining candidate predictions. We analyze a version of the Kangaroo Auto Insurance company data and study the effects of combining different methods using five measures of prediction accuracy. The results show the following. First, when there is an outstanding (in terms of Gini Index) prediction among the candidates, the “forecast combination puzzle” phenomenon disappears. The simple average method performs much worse than the more sophisticated model combination methods, indicating that combining different methods could help us avoid performance degradation. Second, the choice of the prediction accuracy measure is crucial in defining the best candidate prediction for “low frequency and high severity” (LFHS) data. For example, mean square error (MSE) does not distinguish well between model combination methods, as the values are close. Third, the performances of different model combination methods can differ drastically. We propose using a new model combination method, named ARM-Tweedie, for such LFHS data; it benefits from an optimal rate of convergence and exhibits a desirable performance in several measures for the Kangaroo data. Fourth, overall, model combination methods improve the prediction accuracy for auto insurance claim costs. In particular, Adaptive Regression by Mixing (ARM), ARM-Tweedie, and constrained Linear Regression can improve forecast performance when there are only weak learners or when no dominant learner exists.

Panel data nowcasting

This article promotes the use of panel data methods in nowcasting. This shifts the focus of the literature from national to regional nowcasting of variables like gross domestic product (GDP). We propose a mixed-frequency panel VAR model and a bias-corrected least squares estimator which attenuates the bias in fixed effects dynamic panel settings. Simulations show that panel forecast model selection and combination methods are successfully adapted to the nowcasting setting. Our novel empirical application of nowcasting quarterly U.S. state-level real GDP growth highlights the success of state-level nowcasting, as well as the gains from pooling information across states.

Mortality forecasting using stacked regression ensembles

Many alternative approaches for selecting mortality models and forecasting mortality have been proposed. The usual practice is to base forecasts on a single mortality model selected using in-sample goodness-of-fit measures. However, cross-validation measures are increasingly being used in model selection, and model combination methods are becoming a common alternative to using a single mortality model. We propose and assess a stacked regression ensemble that optimally combines different mortality models to reduce out-of-sample mean squared errors and mitigate model selection risk. Stacked regression uses a meta-learner to approximate horizon-specific weights by minimizing a cross-validation criterion for each forecasting horizon. The horizon-specific weights determine a mortality model combination customized to each horizon. We use 44 populations from the Human Mortality Database to compare the stacked regression ensemble with alternative methods. We show that, using one-year-ahead to 15-year-ahead out-of-sample mean squared errors, the stacked regression ensemble improves mortality forecast accuracy by 13% - 49% for males and 19% - 90% for females over individual mortality models. The stacked regression ensembles also have better predictive accuracy than other model combination methods, including Simple Model Averaging, Bayesian Model Averaging, and Model Confidence Set. We provide an R package, CoMoMo, that combines forecasts for Generalized-Age-Period-Cohort models.

Mortality Forecasting Using Stacked Regression Ensembles

There are many alternative approaches to selecting mortality models and forecasting mortality. The standard practice is to produce forecasts using a single model such as the Lee-Carter, the Cairns-Blake-Dowd, or the Age- Period-Cohort model, with model selection based on in-sample goodness of fit measures. However, increasingly cross-validation measures based on forecasts are used in mortality model selection, and model combination methods such as Bayesian Model Averaging and Model Confidence Set have been proposed as alternatives to using a single model. We present a stacked regression ensemble method that optimally combines different mortality models to reduce the mean squared errors of mortality rate forecasts and mitigate model selection risk. Stacked regression uses a supervised machine learning algorithm to approximate the horizon-specific weights by minimizing the cross-validation criterion for each forecasting horizon. The horizon-specific weights facilitate the development of a mortality model combination customized to each horizon. Unlike other model combination methods, stacked regression simultaneously solves model selection and estimates model combinations to improve model forecasts. We use 44 populations from the Human Mortality Database to compare the stacked regression mortality approach with other alternative methods. We show that using one-year-ahead to 15−year-ahead out-of-sample mean squared errors, the stacked regression ensemble approach improves mortality forecast accuracy by 13% - 49% and 19% - 90% for females over individual mortality models. The stacked regression ensemble has better predictive accuracy than other model combination methods, including Simple Model Averaging, Bayesian Model Averaging, and Model Confidence Set. We provide a user-friendly open-source R package, CoMoMo, that estimates models and provides mortality rate forecasts for different mortality model combination methods.

Does crude oil futures price really help to predict spot oil price? New evidence from density forecasting

AbstractIn this paper, we detect the contributions of various predictors in terms of density forecasts of monthly West Texas Intermediate (WTI) crude oil prices. In the first step, we use a simple predictive regression of crude oil prices on different predictors one‐by‐one as explanatory variables, and then two kinds of criteria, Log Score and Continuous Ranked Probability Score (CRPS), are employed to evaluate the density forecasting accuracy of them. In the second step, we utilize a CRPS‐weighted combination and an equal‐weighted combination, respectively, to assemble various density forecasting results in the first step. Finally, a novel test proposed by Rossi and Sekhposyan (2019) is adopted to verify the correct calibration of predictive densities by these two combination methods as well as an AR benchmark model. The empirical results indicate that those predictors proved to perform well (poor) in point forecasts of crude oil price in extant literature do not necessarily offer high (low) density forecasting accuracy. Interestingly, WTI oil futures price is the only predictor that can produce good out‐of‐sample density forecasts across various time horizons. In addition, we find that the two model combination methods can beat the AR benchmark in density forecasting of crude oil price. However, no model can produce correct calibration of predictive densities for crude oil price at time horizons longer than about 5 years.

Steganalysis by Recombining Spatial Rich Model

Steganalysis is an important topic in the field of information security. Steganalysis is mainly based on the traditional machine learning method, represented by the spatial rich model, and has high detection performance. Aiming at the problems of high feature dimension, strong redundancy and long feature extraction time in the spatial rich model, this paper conducts a series of feature selection operations and explores four different model combination methods: single quantization factor combination method, single filter kernel type combination method, mixed selecting combination method, Pearson correlation coefficient combination method. In addition, a feature preprocessing means are added. The experiment proves that the combination of mixed selecting combination and Pearson correlation coefficient can be stable with the performance of steganalysis before dimension reduction, but the performance dimension ratio is higher.

Panel Data Nowcasting

This paper promotes the use of panel data in nowcasting. We shift the existing focus of the literature, which has almost exclusively used time series models to nowcast national aggregate variables like gross domestic product (GDP). We propose a mixed-frequency panel VAR model and a bias-corrected least squares (BCLS) estimator which attenuates the bias inherent to fixed effects dynamic panel settings. We demonstrate how existing panel model selection and combination methods can be adapted to the mixed-frequency setting with different lag specifications. Detailed Monte Carlo simulations find that these methods outperform naive approaches. We present a novel application of our methods to the case of nowcasting quarterly U.S. state-level real GDP using timely employment data. Our results indicate that utilising relevant monthly information is important, while also highlighting the gains from pooling information across states and from the use of appropriate lag selection or combination methods. We also find particularly large gains from nowcasting in states such as California; a region which has higher real GDP than most developed economies and deserves rigorous attention.

Multiple model combination methods for annual maximum water level prediction during river ice breakup

AbstractThe Athabasca River is the largest unregulated river in Alberta, Canada, with ice jams frequently occurring in the vicinity of Fort McMurray. Modelling tools are desired to forecast ice‐related flood events. Multiple model combination methods can often obtain better predictive performances than any member models due to possible variance reduction of forecast errors or correction of biases. However, few applications of this method to river ice forecasting are reported. Thus, a framework of multiple model combination methods for maximum breakup water level (MBWL) Prediction during river ice breakup is proposed. Within the framework, the member models describe the relations between the MBWL (predicted variable) and their corresponding indicators (predictor variables); the combining models link the relations between the predicted MBWL by each member model and the observed MBWL. Especially, adaptive neuro‐fuzzy inference systems, artificial neural networks, and multiple linear regression are not only employed as member models but also as combining models. Simple average methods (SAM) are selected as the basic combining model due to simple calculations. In the SAM, an equal weight (1/n) is assigned to n member models. The historical breakup data of the Athabasca River at Fort McMurray for the past 36 years (1980 to 2015) are collected to facilitate the comparison of models. These models are examined using the leave‐one‐out cross validation and the holdout validation methods. A SAM, which is the average output from three optimal member models, is selected as the best model as it has the optimal validation performance (lowest average squared errors). In terms of lowest average squared errors, the SAM improves upon the optimal artificial neural networks, adaptive neuro‐fuzzy inference systems, and multiple linear regression member models by 21.95%, 30.97%, and 24.03%, respectively. This result sheds light on the effectiveness of combining different forecasting models when a scarce river ice data set is investigated. The indicators included in the SAM may indicate that the MBWL is affected by water flow conditions just after freeze‐up, overall freezing conditions during winter, and snowpack conditions before breakup.

A Bayesian statistical method for quantifying model form uncertainty and two model combination methods

Apart from parametric uncertainty, model form uncertainty as well as prediction error may be involved in the analysis of engineering system. Model form uncertainty, inherently existing in selecting the best approximation from a model set cannot be ignored, especially when the predictions by competing models show significant differences. In this research, a methodology based on maximum likelihood estimation is presented to quantify model form uncertainty using the measured differences of experimental and model outcomes, and is compared with a fully Bayesian estimation to demonstrate its effectiveness. While a method called the adjustment factor approach is utilized to propagate model form uncertainty alone into the prediction of a system response, a method called model averaging is utilized to incorporate both model form uncertainty and prediction error into it. A numerical problem of concrete creep is used to demonstrate the processes for quantifying model form uncertainty and implementing the adjustment factor approach and model averaging. Finally, the presented methodology is applied to characterize the engineering benefits of a laser peening process.

Survey of Model Selection and Model Combination

A general statistical modeling problem is that given a class of competing models and new data, how one can improve the overall model performance. In general, there exist two solutions for this problem, namely model selection and model combination. Model selection is to select a single best model while model combination builds a composite model by aggregating all available information. However, except for this difference model selection and model combination share the most important key elements such as model performance evaluation and are closely related to each other. A generalized flexible framework for designing predictive model performance evaluation method is put forward and possible choices for its two components, that is, model distance measure and generability estimation, are categorized and reviewed. After than, a unified framework is proposed to accommodate both model selection and model combination, in which model selection works as an extreme case of model combination. Finally, many model selection and combination methods in the literature, all of which can be fit into the unified framework, are reviewed.

Unbiased Model Combinations for Adaptive Filtering

In this paper, we consider model combination methods for adaptive filtering that perform unbiased estimation. In this widely studied framework, two adaptive filters are run in parallel, each producing unbiased estimates of an underlying linear model. The outputs of these two filters are combined using another adaptive algorithm to yield the final output of the system. Overall, we require that the final algorithm produce an unbiased estimate of the underlying model. We later specialize this framework where we combine one filter using the least-mean squares (LMS) update and the other filter using the least-mean fourth (LMF) update to decrease cross correlation in between the outputs and improve the overall performance. We study the steady-state performance of previously introduced methods as well as novel combination algorithms for stationary and nonstationary data. These algorithms use stochastic gradient updates instead of the variable transformations used in previous approaches. We explicitly provide steady-state analysis for both stationary and nonstationary environments. We also demonstrate close agreement with the introduced results and the simulations, and show for this specific combination, more than 2 dB gains in terms of excess mean square error with respect to the best constituent filter in the simulations.

Assessing the impact of land use change on hydrology by ensemble modelling (LUCHEM) II: Ensemble combinations and predictions

This paper reports on a project to compare predictions from a range of catchment models applied to a mesoscale river basin in central Germany and to assess various ensemble predictions of catchment streamflow. The models encompass a large range in inherent complexity and input requirements. In approximate order of decreasing complexity, they are DHSVM, MIKE-SHE, TOPLATS, WASIM-ETH, SWAT, PRMS, SLURP, HBV, LASCAM and IHACRES. The models are calibrated twice using different sets of input data. The two predictions from each model are then combined by simple averaging to produce a single-model ensemble. The 10 resulting single-model ensembles are combined in various ways to produce multi-model ensemble predictions. Both the single-model ensembles and the multi-model ensembles are shown to give predictions that are generally superior to those of their respective constituent models, both during a 7-year calibration period and a 9-year validation period. This occurs despite a considerable disparity in performance of the individual models. Even the weakest of models is shown to contribute useful information to the ensembles they are part of. The best model combination methods are a trimmed mean (constructed using the central four or six predictions each day) and a weighted mean ensemble (with weights calculated from calibration performance) that places relatively large weights on the better performing models. Conditional ensembles, in which separate model weights are used in different system states (e.g. summer and winter, high and low flows) generally yield little improvement over the weighted mean ensemble. However a conditional ensemble that discriminates between rising and receding flows shows moderate improvement. An analysis of ensemble predictions shows that the best ensembles are not necessarily those containing the best individual models. Conversely, it appears that some models that predict well individually do not necessarily combine well with other models in multi-model ensembles. The reasons behind these observations may relate to the effects of the weighting schemes, non-stationarity of the climate series and possible cross-correlations between models.

MAP-Based Perceptual Modeling for Noisy Speech Recognition

This study presents a maximum a posteriori (MAP) based perceptual modeling approach to deal with the issue of recognition degradation in noisy environment. In this approach, MAP-based noise detection is first applied to identify the noise segment in an utterance. Subtractive-type enhancement algorithm with masking properties of the human auditory system is then used to reduce the noise effect. Finally, MAP-based incremental noise model adaptation is developed to overcome the model inconsistencies between training and testing environments. For performance evaluation of the proposed approach, a Mandarin keyword recognition system was constructed. The experimental results show that the proposed approach achieves a better recognition rate compared to the audible noise suppression (ANS) and parallel model combination (PMC) methods.

IMPROVING PERFORMANCE OF INDUCTIVE MODELS THROUGH AN ALGORITHM AND SAMPLE COMBINATION STRATEGY

Multiple approaches have been developed for improving predictive performance of a system by creating and combining various learned models. There are two main approaches to creating model ensembles. This first is to create a set of learned models by applying an algorithm repeatedly to different training sample data, the second applies various learning algorithms to the same sample data. The predictions of the models are then combined accordings to a voting scheme. This paper presents a method for combining models that were developed using numerous samples, modeling algorithms, and modelers and compares it with the alternate approaches. The presented results are based on findings from an ongoing operational data mining initiative with respect to selecting a model set that is best able to meet defined goals from among trained models. The operational goals to be attained in this initiative are to deploy data mining model(s) that maximizes specificity with minimal negative impact to sensitivity. The results of the model combination methods are evaluated with respect to sensitivity and false alarm rates and are then compared against other approaches.

Cost functions and model combination for VaR-based asset allocation using neural networks

We introduce an asset-allocation framework based on the active control of the value-at-risk of the portfolio. Within this framework, we compare two paradigms for making the allocation using neural networks. The first one uses the network to make a forecast of asset behavior, in conjunction with a traditional mean-variance allocator for constructing the portfolio. The second paradigm uses the network to directly make the portfolio allocation decisions. We consider a method for performing soft input variable selection, and show its considerable utility. We use model combination (committee) methods to systematize the choice of hyperparameters during training. We show that committees using both paradigms are significantly outperforming the benchmark market performance.

Predictive Modular Neural Networks Methods for Prediction of Sugar Beet Crop Yield

In this paper we present several multiple model combination methods, utilizing neural as well as linear predictors, to predict sugar beet crop yield. The results are superior to previous prediction methocls which used only neural network or only linear regresison predictors.