Study on applicable coverage extension of theory-based generalization errors bounds to the variants of RVFL network and ELM

Abstract

This study investigates the coverage extendibility of generalized prediction error estimation for the random vector functional link (RVFL) network and generalized extreme learning machine (GELM). GELM is an ELM applying the limiting behavior of Moore–Penrose generalized inverse (M–P GI) to the output matrix of the last hidden layer of ELM. The RVFL network, the ELM, and their variants are currently employed in various areas due to their fast learning and are expected to be adopted in more applications. Additionally, obtaining prediction error bounds quickly for these networks could greatly increase their utilities. Therefore, exploring the error bounds of the variants is crucial. Recently, the upper and lower bounds of generalized prediction error and their associated tail probabilities for the RVFL network and GELM were analyzed. This study aims to extend the coverage of the existing error bounds to variants of RVFL and ELM. To this end, we investigate each solution of the objective functions for the existing RVFL and ELM variants, compare it with the solution obtained by using the limiting behavior of M–P GI, and then discuss the applicability of the probability inequalities for the prediction error to the variant. Specifically, we conceptualize a restricted solution vector and prove that the restricted solution of a variant of RVFL is identical to that of GELM, with an appropriately selected regularization parameter. The mechanism used in the proof can be applied to other variants, which have different objective functions but have similar forms of solution. Experiments are performed with the CAIDA (Center for Applied Internet Data Analysis) dataset. Numerical results verify the analysis, showing that the generalization prediction error bounds and their associated tail probabilities for ELM (RVFL) variants in applicability coverage are almost identical to those of the GELM (RVFL). In addition, performance-based network similarity is introduced and computed with the obtained results. Based on this analysis, we can conclude that the probability inequalities for prediction error derived can be directly applied to some variants of the RVFL network and ELM, such as robust regularized RVFL and ridge regression ELM. However, it cannot apply directly to the networks that use information obtained during the learning process in its learning, such as RVFL+ and its variants. The significance of this study is that it enables us to figure out the relationship between the size-restricted final weight vector of RVFL and the final weight vectors of ELM, the effect of the relationship on generalized prediction error estimation, and the performance-based similarity between networks. This study provides a theoretical basis for the generalized prediction error's upper and lower bounds and their associated tail probabilities for ELM (RVFL) variants.