Empirical Risk Minimization Principle Research Articles

Online imbalance learning with unpredictable feature evolution and label scarcity

Recently, online learning with imbalanced data streams has aroused wide concern, which reflects an uneven distribution of different classes in data streams. Existing approaches have conventionally been conducted on stationary feature space and they assume that we can obtain the entire labels of data streams in the case of supervised learning. However, in many real scenarios, e.g., the environment monitoring task, new features flood in, and old features are partially lost during the changing environment as the different lifespans of different sensors. Besides, each instance needs to be labeled by experts, resulting in expensive costs and scarcity of labels. To address the above problems, this paper proposes a novel Online Imbalance learning with unpredictable Feature evolution and Label scarcity (OIFL) algorithm. First, we utilize margin-based online active learning to selectively label valuable instances. After obtaining the labels, we handle imbalanced class distribution by optimizing F-measure and transforming F-measure optimization into a weighted surrogate loss minimization. When data streams arrive with augmented features, we combine the online passive-aggressive algorithm and structural risk minimization to update the classifier in the divided feature space. When data streams arrive with incomplete features, we leverage variance to identify the most informative features following the empirical risk minimization principle and continue to update the existing classifier as before. Finally, we obtain a sparse but reliable learner by the strategy of projecting truncation. We derive theoretical analyses of OIFL. Also, experiments on the synthetic datasets and real-world data streams to validate the effectiveness of our method.

Generalized Federated Learning via Gradient Norm-Aware Minimization and Control Variables

Federated Learning (FL) is a promising distributed machine learning framework that emphasizes privacy protection. However, inconsistencies between local optimization objectives and the global objective, commonly referred to as client drift, primarily arise due to non-independently and identically distributed (Non-IID) data, multiple local training steps, and partial client participation in training. The majority of current research tackling this challenge is mainly based on the empirical risk minimization (ERM) principle, while giving little consideration to the connection between the global loss landscape and generalization capability. This study proposes FedGAM, an innovative FL algorithm that incorporates Gradient Norm-Aware Minimization (GAM) to efficiently search for a local flat landscape. FedGAM specifically modifies the client model training objective to simultaneously minimize the loss value and first-order flatness, thereby seeking flat minima. To directly smooth the global flatness, we propose the more significant FedGAM-CV, which employs control variables to correct local updates, guiding each client to train models in a globally flat direction. Experiments on three datasets (CIFAR-10, MNIST, and FashionMNIST) demonstrate that our proposed algorithms outperform existing FL baselines, effectively finding flat minima and addressing the client drift problem.

Bats monitoring: a classification procedure of bats behaviours based on Hawkes processes

Abstract We are interested in the problem of classifying commuting and foraging behaviour of bats at delimited geographical areas, namely sites, throughout France. To predict the majority behaviour on these sites, we use echolocation call data recorded as part of Vigie-Chiro participatory project. As the temporal distribution of calls is a relevant indicator of behaviour, providing an adequate model of this distribution is a matter of great interest. Given the self-exciting dynamics observed in foraging behaviour, we propose to model bat calls by Hawkes processes. Specifically, we consider that the start time of each call emitted on a site is an event of a Hawkes process. Taking advantage of this modelling, we use a suitable procedure that relies on the empirical risk minimization principle to discriminate between the 2 classes. Then, the performance of the procedure is assessed on synthetic data through comprehensive numerical experiments. The overall methodology is evaluated with a goodness-of-fit test. Finally, we present the obtained results on the real data set. The classification results are convincing and show the relevance of our method, which could contribute to a better understanding of behavioural determinants and open up broad perspectives in spatial ecology.

Explainable empirical risk minimization

AbstractThe successful application of machine learning (ML) methods increasingly depends on their interpretability or explainability. Designing explainable ML (XML) systems is instrumental for ensuring transparency of automated decision-making that targets humans. The explainability of ML methods is also an essential ingredient for trustworthy artificial intelligence. A key challenge in ensuring explainability is its dependence on the specific human end user of an ML system. The users of ML methods might have vastly different background knowledge about ML principles, with some having formal training in the specific field and others having none. We use information-theoretic concepts to develop a novel measure for the subjective explainability of predictions delivered by a ML method. We construct this measure via the conditional entropy of predictions, given the user signal. Our approach allows for a wide range of user signals, ranging from responses to surveys to biophysical measurements. We use this measure of subjective explainability as a regularizer for model training. The resulting explainable empirical risk minimization (EERM) principle strives to balance subjective explainability and risk. The EERM principle is flexible and can be combined with arbitrary ML models. We present several practical implementations of EERM for linear models and decision trees. Numerical experiments demonstrate the application of EERM to weather prediction and detecting inappropriate language in social media.

Multiple instance learning from similarity-confidence bags

Multiple instance learning (MIL) is a classic weakly supervised learning approach, in which samples are grouped into bags that may contain varying numbers of instances. A bag is designated as positive if it contains at least one positive instance; otherwise, it is considered negative. Previous studies have consistently assumed that the bag labels are completely known. In fact, labeling every bag can be extremely challenging or even unfeasible due to the exorbitant expenses in terms of time and labor. Fortunately, it is much easier to obtain the similarity confidence, which represents the probability of two bags sharing the same label. How to employ it in MIL is worthy of study. Inspired by the above study, we present the first attempt to investigate MIL from similarity-confidence bags. Therefore, this paper proposes a new framework for training bag-level classifiers that adheres to the principle of empirical risk minimization. Moreover, we theoretically derive a generalization error bound to guarantee model convergence. Finally, we implement risk correction to mitigate potential over-fitting problem and provide theoretical consistency. Numerical experiments on eight datasets further validate the effectiveness of the proposed bag-level classifier.

A data-driven newsvendor problem: A high-dimensional and mixed-frequency method

In this paper, a data-driven newsvendor problem is studied by mapping high-dimensional and mixed-frequency features of historical data to replenishment decisions. Instead of relying on a known demand distribution, we propose using machine learning algorithms that incorporate demand features into the replenishment decisions to solve single- and multi-product newsvendor problems. In particular, our algorithms simultaneously optimize the demand estimation and replenishment decisions. To extract key features from the historical data, we propose a frequency alignment method to transform high-dimensional mixed-frequency data and historical data into the same frequency. We then propose two variable selection policies based on the empirical risk minimization principle, and employ the regularization method to tackle the parameter proliferation issue. In addition, a feature-based machine learning algorithm is designed to solve the multi-product newsvendor problem. Finally, we numerically justify the performances of proposed machine learning algorithms. We find that (1) data-informed replenishment decisions can effectively leverage the identified key demand features to avoid losing demand information, and (2) the optimal replenishment quantity behaves robustly and shows minimal variation across different cost structures. Our work provides meaningful insights for newsvendors making replenishment decisions under stochastic demand.

An improved multi-task least squares twin support vector machine

AbstractIn recent years, multi-task learning (MTL) has become a popular field in machine learning and has a key role in various domains. Sharing knowledge across tasks in MTL can improve the performance of learning algorithms and enhance their generalization capability. A new approach called the multi-task least squares twin support vector machine (MTLS-TSVM) was recently proposed as a least squares variant of the direct multi-task twin support vector machine (DMTSVM). Unlike DMTSVM, which solves two quadratic programming problems, MTLS-TSVM solves two linear systems of equations, resulting in a reduced computational time. In this paper, we propose an enhanced version of MTLS-TSVM called the improved multi-task least squares twin support vector machine (IMTLS-TSVM). IMTLS-TSVM offers a significant advantage over MTLS-TSVM by operating based on the empirical risk minimization principle, which allows for better generalization performance. The model achieves this by including regularization terms in its objective function, which helps control the model’s complexity and prevent overfitting. We demonstrate the effectiveness of IMTLS-TSVM by comparing it to several single-task and multi-task learning algorithms on various real-world data sets. Our results highlight the superior performance of IMTLS-TSVM in addressing multi-task learning problems.

Prediction using multi-objective slime mould algorithm optimized support vector regression model

Different from the traditional neural network models based on empirical risk minimization principle, support vector regression (SVR) minimizes the upper limit of generalization error by the structural risk minimization principle. SVR is suitable for solving nonlinear, small-sample, high-dimensional modelling, and pattern recognition problems. However, the selection of hyperparameters of SVR can significantly affect the prediction accuracy and computational time. This paper proposes a method to optimize the SVR hyperparameters through an intelligent optimization algorithm, specifically multi-objective slime mould algorithm (MOSMA). MOSMA is a new multi-objective optimization algorithm with strong global search ability and fast convergence. The principle and calculation process of MOSMA are first described in detail. The analytical method for transforming the SVR hyperparameter optimization problem into a multi-objective problem is then presented. The process of optimizing two SVR hyperparameters (the penalty coefficient and radial basis function kernel parameter) based on MOSMA is described in steps. Meanwhile, the procedures of algorithm parameter setting, fitness function design, and population update are given. To investigate the characteristics and performance of MOSMA-SVR, two datasets are used to predict the vibration trend of the spindle of CNC milling machine in one and the maximum bending normal stress in the cutting process in the other. The performance of MOSMA-SVR is evaluated by multiple statistical indexes and compared with seven other prediction models (GA-SVR, PSO-SVR, ABC-SVR, GWO-SVR, SSA-SVR, SMA-SVR and MOWOA-SVR).

Brain Age Prediction With Improved Least Squares Twin SVR.

Alzheimer's disease (AD) is the prevalent form of dementia and shares many aspects with the aging pattern of the abnormal brain. Machine learning models like support vector regression (SVR) based models have been successfully employed in the estimation of brain age. However, SVR is computationally inefficient than twin support vector machine based models. Hence, different twin support vector machine based models like twin SVR (TSVR), ε-TSVR, and Lagrangian TSVR (LTSVR) models have been used for the regression problems. ε-TSVR and LTSVR models seek a pair of ε-insensitive proximal planes for generation of end regressor. However, SVR and TSVR based models have several drawbacks- i) SVR model is computationally inefficient compared to the TSVR based models. ii) Twin SVM based models involve the computation of matrix inverse which is intractable in real world scenario's. iii) Both TSVR and LTSVR models are based on empirical risk minimization principle and hence may be prone to overfitting. iv) TSVR and LTSVR assume that the matrices appearing in their formulation are positive definite which may not be satisfied in real world scenario's. To overcome these issues, we formulate improved least squares twin support vector regression (ILSTSVR). The proposed ILSTSVR modifies the TSVR by replacing the inequality constraints with the equality constraints and minimizes the slack variables using squares of L2 norm instead of L1. Also, we introduce a different Lagrangian function to avoid the computation of matrix inverses. We evaluated the proposed ILSTSVR model on the subjects including cognitively healthy, mild cognitive impairment and Alzheimer's disease for brain-age estimation. Experimental evaluation and statistical tests demonstrate the efficiency of the proposed ILSTSVR model for brain-age prediction.

Multiclass Weighted Least Squares Twin Bounded Support Vector Machine for Intelligent Water Leakage Diagnosis

Monitoring the operating status of the pipeline and determining the location of the leak in time are very important to ensure the safe operation of the pipeline. Least squares twin support vector machine (LSTSVM) is a classic fast classification method, which has been used to identify different pipeline conditions. However, LSTSVM assumes that all samples share the same weight when generating the hyperplane, including data points that may be polluted in the sample (i.e., outliers), and outlier samples with equal weights will mislead the generation of the hyperplane. Inspired by the above research, this paper proposes a weighted least squares twin bounded support vector machine based on Gaussian mixture models (GMM), referred to as G-WLSTSVM. The proposed G-WLSTSVM introduces a weight matrix for the objective function through GMM, and assigns a larger weight to the normal samples and a smaller weight to the outliers, which reduces the impact effect of the outliers on the generation of the classification hyperplane. Furthermore, since LSTSVM only considers the empirical risk minimization principle, it may lead to overfitting. The proposed G-WLSTSVM introduces an extra regularization term based on the margin maximization idea to realize the principle of structural risk minimization, which improves the generalization performance of the model. Finally, since the practical problems are mostly multi-classification problems, the G-WLSTSVM for binary classification cannot be satisfied. Therefore, the proposed G-WLSTSVM combined with a "One-versus-One" strategy is extended to handle multi-classification problems, namely the multi-class G-WLSTSVM. We evaluate the effectiveness of the multi-class G-WLSTSVM in identifying different pipeline conditions and localize the identified leakage. Numerical experimental results on several UCI datasets further demonstrate that compared with other related methods, the proposed G-WLSTSVM not only retains the advantages of simplicity and speed of the LSTSVM, but also improves the classification accuracy and generalization ability. The code for this article is available at https://github.com/cmq-456/glstsvm.

Large-Scale Fuzzy Least Squares Twin SVMs for Class Imbalance Learning

Twin support vector machines (TSVMs) have been successfully employed for binary classification problems. With the advent of machine learning algorithms, data have proliferated and there is a need to handle or process large-scale data. TSVMs are not successful in handling large-scale data due to the following: 1) the optimization problem solved in the TSVM needs to calculate large matrix inverses, which makes it an ineffective choice for large-scale problems; 2) the empirical risk minimization principle is employed in the TSVM and, hence, may suffer due to overfitting; and 3) the Wolfe dual of TSVM formulation involves positive-semidefinite matrices, and hence, singularity issues need to be resolved manually. Keeping in view the aforementioned shortcomings, in this article, we propose a novel large-scale fuzzy least squares TSVM for class imbalance learning (LS-FLSTSVM-CIL). We formulate the LS-FLSTSVM-CIL such that the proposed optimization problem ensures that: 1) no matrix inversion is involved in the proposed LS-FLSTSVM-CIL formulation, which makes it an efficient choice for large-scale problems; 2) the structural risk minimization principle is implemented, which avoids the issues of overfitting and results in better performance; and 3) the Wolfe dual formulation of the proposed LS-FLSTSVM-CIL model involves positive-definite matrices. In addition, to resolve the issues of class imbalance, we assign fuzzy weights in the proposed LS-FLSTSVM-CIL to avoid bias in dominating the samples of class imbalance problems. To make it more feasible for large-scale problems, we use an iterative procedure known as the sequential minimization principle to solve the objective function of the proposed LS-FLSTSVM-CIL model. From the experimental results, one can see that the proposed LS-FLSTSVM-CIL demonstrates superior performance in comparison to baseline classifiers. To demonstrate the feasibility of the proposed LS-FLSTSVM-CIL on large-scale classification problems, we evaluate the classification models on the large-scale normally distributed clustered (NDC) dataset. To demonstrate the practical applications of the proposed LS-FLSTSVM-CIL model, we evaluate it for the diagnosis of Alzheimer’s disease and breast cancer disease. Evaluation on NDC datasets shows that the proposed LS-FLSTSVM-CIL has feasibility in large-scale problems as it is fast in comparison to the baseline classifiers.

Gradient Learning under Tilted Empirical Risk Minimization.

Gradient Learning (GL), aiming to estimate the gradient of target function, has attracted much attention in variable selection problems due to its mild structure requirements and wide applicability. Despite rapid progress, the majority of the existing GL works are based on the empirical risk minimization (ERM) principle, which may face the degraded performance under complex data environment, e.g., non-Gaussian noise. To alleviate this sensitiveness, we propose a new GL model with the help of the tilted ERM criterion, and establish its theoretical support from the function approximation viewpoint. Specifically, the operator approximation technique plays the crucial role in our analysis. To solve the proposed learning objective, a gradient descent method is proposed, and the convergence analysis is provided. Finally, simulated experimental results validate the effectiveness of our approach when the input variables are correlated.

KNN weighted reduced universum twin SVM for class imbalance learning

In real world problems, imbalance of data samples poses major challenge for the classification problems as the data samples of a particular class are dominating. Problems like fault and disease detection involve imbalance data and hence need attention to avoid the bias towards a particular class. The classification models like support vector machines (SVM) get biased to majority class samples and hence results in misclassification of the minority class samples. SVM suffers as no prior information related to the data is involved in the generation of hyperplanes. Also, local information of the neighbourhood is ignored in SVM samples and thus treats each sample equally for generating the hyperplanes. However, the data points may be contaminated and may mislead the generation of hyperplanes. Inspired by the idea of prior data information and local neighbourhood information, we propose K-nearest neighbour based weighted reduced universum twin SVM for class imbalance learning (KWRUTSVM-CIL). The proposed KWRUTSVM-CIL embodies the local neighbourhood information and uses universum data to balance the classes in class imbalance problems. Local neighbourhood information is incorporated via weight matrix in the objective function. In proposed KWRUTSVM-CIL model, weight vectors are used in the corresponding constraints of the objective functions to exploit the interclass information. The oversampling and undersampling approaches are followed to balance the data in class imbalance problems. Universum data gives prior information of the data. Twin SVM, universum twin SVM, and reduced universum twin SVM for class imbalance implement empirical risk minimization principle and thus may lead to overfitting. However, the proposed KWRUTSVM-CIL model embodies regularization term to maximize the margin and implement the structural risk minimization principle which is the marrow of statistical learning and overcomes the issues of overfitting. Experimental results and the statistical analysis signify that the generalization ability of proposed KWRUTSVM-CIL model is superior in comparison to other twin SVM based models. As an application, we use the proposed KWRUTSVM-CIL model for the diagnosis of Alzheimer’s disease and breast cancer disease. The proposed KWRUTSVM-CIL model showed better generalization performance compared to other twin SVM based models in biomedical datasets.

Semisupervised Ordinal Regression Based on Empirical Risk Minimization.

Ordinal regression is aimed at predicting an ordinal class label. In this letter, we consider its semisupervised formulation, in which we have unlabeled data along with ordinal-labeled data to train an ordinal regressor. There are several metrics to evaluate the performance of ordinal regression, such as the mean absolute error, mean zero-one error, and mean squared error. However, the existing studies do not take the evaluation metric into account, restrict model choice, and have no theoretical guarantee. To overcome these problems, we propose a novel generic framework for semisupervised ordinal regression based on the empirical risk minimization principle that is applicable to optimizing all of the metrics mentioned above. In addition, our framework has flexible choices of models, surrogate losses, and optimization algorithms without the common geometric assumption on unlabeled data such as the cluster assumption or manifold assumption. We provide an estimation error bound to show that our risk estimator is consistent. Finally, we conduct experiments to show the usefulness of our framework.

Prediction of S12-MKII rainfall simulator experimental runoff data sets using hybrid PSR-SVM-FFA approaches

Abstract Effective prediction of runoff is a substantial feature for the successful management of hydrological phenomena in arid regions. The present research findings reveal that a rainfall simulator (RS) can be a valuable instrument to estimate runoff as the intensity of rainfall is modifiable in the course of an experimental process, which turns out to be of great advantage. The rainfall-runoff process is a complex physical phenomenon caused by the effect of various parameters. In this research, a new hybrid technique integrating PSR (phase space reconstruction) with FFA (firefly algorithm) and SVM (support vector machine) has gained recognition in various modelling investigations in contrast to the principle of empirical risk minimization through ANN practices. Outcomes of SVM are contrasted against SVM-FFA and PSR-SVM-FFA models. The improvements in NSE (Nash–Sutcliffe efficiency), RMSE (Root Mean Square Error), and WI (Willmott's Index) by PSR-SVM-FFA over SVM models specify that the prediction accuracy of the hybrid model is better. The established PSR-SVM-FFA model generates preeminent WI values that range from 0.97 to 0.98, while the SVM and SVM-FFA models encompass 0.93–0.95 and 0.96–0.97, respectively. The proposed PSR-SVM-FFA model gives more accurate results and error limiting up to 2–3%.

Fine-grained Generalization Analysis of Vector-Valued Learning

Many fundamental machine learning tasks can be formulated as a problem of learning with vector-valued functions, where we learn multiple scalar-valued functions together. Although there is some generalization analysis on different specific algorithms under the empirical risk minimization principle, a unifying analysis of vector-valued learning under a regularization framework is still lacking. In this paper, we initiate the generalization analysis of regularized vector-valued learning algorithms by presenting bounds with a mild dependency on the output dimension and a fast rate on the sample size. Our discussions relax the existing assumptions on the restrictive constraint of hypothesis spaces, smoothness of loss functions and low-noise condition. To understand the interaction between optimization and learning, we further use our results to derive the first generalization bounds for stochastic gradient descent with vector-valued functions. We apply our general results to multi-class classification and multi-label classification, which yield the first bounds with a logarithmic dependency on the output dimension for extreme multi-label classification with the Frobenius regularization. As a byproduct, we derive a Rademacher complexity bound for loss function classes defined in terms of a general strongly convex function.

Model Karmaşıklığının Kontrolü

Model karmaşıklığı, modellerin başarısındaki en önemli ölçütlerden birisidir. Bu çalışmada bugüne dek model karmaşıklığının kontrolünde öne çıkan yaklaşımlar başlıklar halinde incelenmiştir. Bunlar Occam’ın usturası, Popper’ın yanlışlanabilirliği ve istatiksel öğrenme teorisidir. Occam’ın usturası ve Popper’ın yanlışlanabilirliği model karmaşıklığının kontrolünde, evet bir felsefi yaklaşım sağlamaktadırlar ve kabul de görmektedirler. Fakat model karmaşıklığının nasıl kontrol edileceği konusunda matematiksel bir formülasyon sağlamamaktadırlar. Fakat istatiksel öğrenme teorisi (diğer adıyla, VC teorisi) konuya yaklaşımı yalnızca felsefi bir düzeyde kalmamakta, aynı zamanda şimdiye dek geliştirilen modellerde kullanılan ampirik risk minimizasyonu (ARM) ilkesine VC katsayısını ilave ederek yeni bir risk minimizasyonu (yapısal risk minimizasyonu, YRM) ilkesi getirmektedir. Sonuç olarak Vapnik ve Chervonenkis tarafından geliştirilen VC teorisi bir kontrol modeli olarak, ispatlanmış matematiksel arka planı ve oldukça başarılı olan sonuçları itibariyle, model karmaşıklığının kontrolü konusunda, günümüz çerçevesinde, en tutarlı ve güvenilir bir yaklaşım olarak, model geliştiriciler için iyi bir ilham kaynağı olabilir.

From predictive to prescriptive analytics: A data-driven multi-item newsvendor model

This paper considers a multi-item newsvendor problem with a capacity constraint (Z). The problem has already been addressed in the literature using the classical newsvendor problem. However, provided solutions made assumptions for demand distributions, which are often incorrect and led to errors in the inventory optimization. This research proposes a distribution-free and completely data-driven solution approach to Z. The proposed approach uses sample demand data as input, and machine (and deep) learning methods with empirical risk minimization principle to find order quantities. A heuristic is developed using hierarchies of the retail products to perform multi-item inventory optimization when a capacity constraint is active. The proposed approach is tested on a real-world dataset of retail products. The results from the proposed method are compared with data-driven max-min and empirical inventory optimization methods, and it outperformed all of them. The machine (and deep) learning-based demand forecasting methods (part of the proposed approach) providing better results than neural networks, multiple regression, arima, arimax, etc. Finally, a comparison of total inventory cost from the proposed, max-min, and empirical inventory optimization methods are carried out, and it is observed that the proposed data-driven approach leads to a significant reduction in inventory cost.

Consistent procedures for multiclass classification of discrete diffusion paths

AbstractThe recent advent of modern technology has generated a large number of datasets which can be frequently modeled as functional data. This paper focuses on the problem of multiclass classification for stochastic diffusion paths. In this context we establish a closed formula for the optimal Bayes rule. We provide new statistical procedures which are built either on the plug‐in principle or on the empirical risk minimization principle. We show the consistency of these procedures under mild conditions. We apply our methodologies to the parametric case and illustrate their accuracy with a simulation study through examples.

A novel recursive modal parameter estimator for operational time-varying structural dynamic systems based on least squares support vector machine and time series model

Modal parameters are practically important for vibration control, structural dynamic design, health monitoring, etc. Presently, the commonly used recursive modal parameter estimators are generally based on the empirical risk minimization principle, and thus can result in overfitting problem easily. This paper presents a novel recursive modal parameter estimator for operational linear time-varying structures based on least squares support vector machine (LSSVM) and vector time-dependent autoregressive moving average model. A sliding-window forgetting mechanism is adapted to fix computational complexity of each update step and enhance tracking capability of the proposed estimator. A numerical example and a laboratory experiment are performed to demonstrate that the proposed structural risk minimization principle based estimator is robust to model structure and its computational complexities are independent of the dimension of output measurements comparing with the existing recursive extended least squares estimator.