Empirical Risk Function Research Articles

Capacitive empirical risk function-based bag-of-words and pattern classification processes

This paper proposes capacitive bag-of-words (Cap BoW) modeling and capacitive pattern classification processes for improved generalization ability. The Cap BoW and capacitive pattern classification are realized by introducing the notion of the capacitive empirical risk function (CERF). The CERF is used as a cost function to build a capacitive pattern classification process. The resulted capacitive pattern classification is used in the classification stage of the bag-of-words process, realizing thereby the Cap BoW process. The use of the CERF in building the capacitive pattern classification and Cap BoW processes is demonstrated to achieve simultaneous reduction to the empirical risk function (ERF) and confidence interval, reducing potential overfitting and enhancing the generalization ability of considered models. To have a thorough evaluation, two groups of experiments are carried out. The first group addresses the capacitive pattern classification process for 4 datasets. The second group addresses a more holistic impact of the CERF by using Cap Bow model to build capacitive dual ergodicity limits-based bag-of-words (Cap DEL-BoW) process, which is applied to 5 image datasets. In both groups of experiments, remarkable enhancement is demonstrated with the use of CERF-based pattern classification and Cap BoW processes. Comparison with corresponding conventional non-capacitive models demonstrates the tangible enhancement with the use of CERF-based models.

Robust High-Dimensional Regression with Coefficient Thresholding and Its Application to Imaging Data Analysis

It is important to develop statistical techniques to analyze high-dimensional data in the presence of both complex dependence and possible heavy tails and outliers in real-world applications such as imaging data analyses. We propose a new robust high-dimensional regression with coefficient thresholding, in which an efficient nonconvex estimation procedure is proposed through a thresholding function and the robust Huber loss. The proposed regularization method accounts for complex dependence structures in predictors and is robust against heavy tails and outliers in outcomes. Theoretically, we rigorously analyze the landscape of the population and empirical risk functions for the proposed method. The fine landscape enables us to establish both statistical consistency and computational convergence under the high-dimensional setting. We also present an extension to incorporate spatial information into the proposed method. Finite-sample properties of the proposed methods are examined by extensive simulation studies. An application concerns a scalar-on-image regression analysis for an association of psychiatric disorder measured by the general factor of psychopathology with features extracted from the task functional MRI data in the Adolescent Brain Cognitive Development (ABCD) study. Supplementary materials for this article are available online.

A Reproducing Kernel Hilbert Space Framework for Functional Classification

The intrinsic infinite-dimensional nature of functional data creates a bottleneck in the application of traditional classifiers to functional settings. These classifiers are generally either unable to generalize to infinite dimensions or have poor performance due to the curse of dimensionality. To address this concern, we propose building a distance-weighted discrimination (DWD) classifier on scores obtained by projecting data onto one specific direction. We choose this direction by minimizing, over a reproducing kernel Hilbert space, an empirical risk function containing the DWD classifier loss function. Our proposed classifier avoids overfitting and enjoys the appealing properties of DWD classifiers. We further extend this framework to accommodate functional data classification problems where scalar covariates are involved. In contrast to previous work, we establish a nonasymptotic estimation error bound on the relative misclassification rate. Through simulation studies and a real-world application, we demonstrate that the proposed classifier performs favorably relative to other commonly used functional classifiers in terms of prediction accuracy in finite-sample settings. Supplementary materials for this article are available online.

Nonregular and minimax estimation of individualized thresholds in high dimension with binary responses

Given a large number of covariates Z, we consider the estimation of a high-dimensional parameter θ in an individualized linear threshold θTZ for a continuous variable X, which minimizes the disagreement between sign(X−θTZ) and a binary response Y. While the problem can be formulated into the M-estimation framework, minimizing the corresponding empirical risk function is computationally intractable due to discontinuity of the sign function. Moreover, estimating θ even in the fixed-dimensional setting is known as a nonregular problem leading to nonstandard asymptotic theory. To tackle the computational and theoretical challenges in the estimation of the high-dimensional parameter θ, we propose an empirical risk minimization approach based on a regularized smoothed non-convex loss function. The Fisher consistency of the proposed method is guaranteed as the bandwidth of the smoothed loss is shrunk to 0. Statistically, we show that the finite sample error bound for estimating θ in ℓ2 norm is (slogd/n)β/(2β+1), where d is the dimension of θ, s is the sparsity level, n is the sample size and β is the smoothness of the conditional density of X given the response Y and the covariates Z. The convergence rate is nonstandard and slower than that in the classical Lasso problems. Furthermore, we prove that the resulting estimator is minimax rate optimal up to a logarithmic factor. The Lepski’s method is developed to achieve the adaption to the unknown sparsity s or smoothness β. Computationally, an efficient path-following algorithm is proposed to compute the solution path. We show that this algorithm achieves geometric rate of convergence for computing the whole path. Finally, we evaluate the finite sample performance of the proposed estimator in simulation studies and a real data analysis from the ChAMP (Chondral Lesions And Meniscus Procedures) Trial.

Optimal Window-Symbolic Time Series Analysis for Pattern Classification and Anomaly Detection

This article proposes the optimal window-symbolic time series analysis (OW-STSA) methodology to optimize parameters of feature extraction and pattern classification in industrial processes. The underlying theory is built upon minimization of an empirical risk function to discriminate between nominal and anomalous operations of the physical process under consideration. In particular, the proposed methodology produces: optimized windows of the time series used for pattern classification and anomaly detection, and optimized identification of feature extractors and classifier parameters. The algorithm is realized by segmenting a given time series into windows of equal size. Then, the stationary state probability vector is computed for each window in the sense of OW-STSA for anomaly prediction with locally optimal accuracy of detection performance. The proposed methodology has been experimentally validated in laboratory environment with different classifiers for two distinct industrial processes. The first experiment addresses detection of fatigue failure in polycrystalline alloy structures using time series of ultrasonic signals. The second experiment investigates detection of thermoacoustic instability in an emulated combustion system using time series of pressure-wave signals. In both experiments, the proposed OW-STSA methodology yielded excellent detection performance of anomalous behavior with multiple classification techniques.

Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, by analyzing the inherent randomness due to the use of a zeroth-order method, we show that D-ZOA is intrinsically endowed with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\epsilon,\delta)-$ </tex-math></inline-formula> differential privacy. In addition, we employ the moments accountant method to show that the total privacy leakage of D-ZOA grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially-private approaches in terms of accuracy while yielding similar privacy guarantee. We prove that D-ZOA reaches a neighborhood of the optimal solution whose size depends on the privacy parameter. The convergence analysis also reveals a practically important trade-off between privacy and accuracy. Simulation results verify the desirable privacy-preserving properties of D-ZOA and its superiority over the state-of-the-art algorithms as well as its network-wide convergence.

Statistical and Non-linear Dynamics Methods of Earthquake Forecast: Application in the Caucasus

In 20 th century more than 10 strong earthquakes (EQ) of magnitude from 6 to 7 hit the South Caucasus, causing thousands of casualties and gross economic losses. Thus, the problem of strong EQs forecast is an actual problem for the region. In this direction, we developed the physical percolation model of fracture, which considers the final failure of solid as a termination of prolonged process of destruction -generation and clustering of micro-cracks -till appearance, at some critical concentration, of the infinite cluster, marking the final failure. Percolation provides a model of preparation of an individual strong event (slip, earthquake). The natural seismic process contains many such events: the appropriate model is a non-linear stick-slip model, which is a particular case of the general theory of integrate-and-fire process. Nonlinearity of seismic process is in contradiction with a memory-less Poissonian approach to seismic hazard. The complexity theory offers a chance to improve strong earthquakes' forecast using analysis of hidden (nonlinear) patterns in seismic time series, such as attractors in the phase space plot.For a regional forecast we applied Bayesian approach to assess conditional probability of expected in the next 5 years strong EQ of magnitude 5 and more. Later on, in addition to Bayesian probability, we applied the pattern recognition technique to seismic time series, based on the assessment of empirical risk function (Generalized Portrait method): nowadays this approach is known as Support Vector Machines (SVM) technique. The preliminary analysis shows that application of GP technique predicts retrospectively 80% of M5 events in Caucasus.Besides regional forecast studies, intensive work is under way on short-term EQ prediction also. Here we present the results of multi-parametrical (hydrodynamic and magnetic) monitoring carried out in 2017-2019 on the territory of Georgia. In order to assess the reliability of the precursors, we used machine learning approach, namely the algorithm of deep learning ADAM, which optimizes target function by combination of optimization algorithm designed for neural networks and a method of stochastic gradient descent with momentum. Finally, we used the method of Receiver Operating Characteristics (ROC) to assess the forecast quality of this binary classifier system.

Adaptive composite estimation in small domains

Small area estimation techniques are used in sample surveys, where direct estimates for small domains are not reliable due to small sample sizes in the domains. We estimate the domain means by generalized linear compositions of the weighted sample means and the synthetic estimators that are obtained from the regression-synthetic model of fixed effects, based on the domain level auxiliary information. In the proposed method, the number of parameters of optimal compositions is reduced to a single unknown parameter, which is further evaluated by minimizing an empirical risk function. We apply various composite and related estimators to estimate proportions of the unemployed in a simulation study, based on the Lithuanian Labor Force Survey data. Conclusions on advantages and disadvantages of the proposed compositions are obtained from this empirical comparison.

An Empirical Study of Downstream Analysis Effects of Model Pre-Processing Choices

This study uses an empirical analysis to quantify the downstream analysis effects of data pre-processing choices. Bootstrap data simulation is used to measure the bias-variance decomposition of an empirical risk function, mean square error (MSE). Results of the risk function decomposition are used to measure the effects of model development choices on model bias, variance, and irreducible error. Measurements of bias and variance are then applied as diagnostic procedures for model pre-processing and development. Best performing model-normalization-data structure combinations were found to illustrate the downstream analysis effects of these model development choices. In additions, results found from simulations were verified and expanded to include additional data characteristics (imbalanced, sparse) by testing on benchmark datasets available from the UCI Machine Learning Library. Normalization results on benchmark data were consistent with those found using simulations, while also illustrating that more complex and/or non-linear models provide better performance on datasets with additional complexities. Finally, applying the findings from simulation experiments to previously tested applications led to equivalent or improved results with less model development overhead and processing time.

Model Misfit Minimization

Abstract In geophysical applications, solutions to ill‐posed inverse problems Ax=b are often obtained by analyzing the trade‐off between data residue ‖Ax−b‖2 and model norm ‖x‖2. In this study, we show that the traditional L‐curve analysis does not lead to solutions closest to the true models because the maximum curvature (or the corner of the L‐curve) depends on the relative scaling between data residue and model norm. A Bayes approach based on empirical risk function minimization using training datasets may be designed to find a statistically optimal solution, but its success depends on the true realization of the model. To overcome this limitation, we construct training models using eigenvectors of matrix ATA as well as spectral coefficients calculated from the correlation between observations and eigenvector projected data. This approach accounts for data noise level but does not require it as a priori knowledge. Using global tomography as an example, we show that the solutions are closest to true models.

Mortality impact of low annual crop yields in a subsistence farming population of Burkina Faso under the current and a 1.5°C warmer climate in 2100.

In subsistence farming populations of sub-Saharan Africa reliant on rainfed agriculture, years of low crop yields result in poorer child nutrition and survival. Estimates of such impacts are critical for their reduction and prevention. We developed a model to quantify such health impacts, and the degree to which they are attributable to weather variations, for a subsistence farming population in the Nouna district of Burkina Faso (89,000 people in 2010). The method combines data from a new weather-crop yield model with empirical epidemiological risk functions. We quantify the child mortality impacts for 1984-2012 using observed weather data and estimate potential future burdens in 2050 and 2100 using daily weather data generated by global climate models parameterized to simulate global warming of 1.5°C above pre-industrial levels. For 1984-2012, crop yields below 90% of the period average were estimated to result in the total of 109.8 deaths per 10,000 children <5years, or around 7122.0years of life lost, 72% of which are attributable to unfavourable weather conditions in the crop growing season. If all non-weather factors are assumed to remain unchanged, the mortality burden related to low crop yields would increase about twofold under 1.5°C global warming by 2100. These results emphasize the importance and value of developing strategies to protect against the effects of low crop yields and specifically the adverse impact of unfavourable weather conditions in such settings under the current and future climate.

An accelerated variance reducing stochastic method with Douglas-Rachford splitting

We consider the problem of minimizing the regularized empirical risk function which is represented as the average of a large number of convex loss functions plus a possibly non-smooth convex regularization term. In this paper, we propose a fast variance reducing (VR) stochastic method called Prox2-SAGA. Different from traditional VR stochastic methods, Prox2-SAGA replaces the stochastic gradient of the loss function with the corresponding gradient mapping. In addition, Prox2-SAGA also computes the gradient mapping of the regularization term. These two gradient mappings constitute a Douglas-Rachford splitting step. For strongly convex and smooth loss functions, we prove that Prox2-SAGA can achieve a linear convergence rate comparable to other accelerated VR stochastic methods. In addition, Prox2-SAGA is more practical as it involves only the stepsize to tune. When each loss function is smooth but non-strongly convex, we prove a convergence rate of $${\mathcal {O}}(1/k)$$ for the proposed Prox2-SAGA method, where k is the number of iterations. Moreover, experiments show that Prox2-SAGA is valid for non-smooth loss functions, and for strongly convex and smooth loss functions, Prox2-SAGA is prominently faster when loss functions are ill-conditioned.

Classification and Regression Using an Outer Approximation Projection-Gradient Method

This paper deals with sparse feature selection and grouping for classification and regression. The classification or regression problems under consideration consists of minimizing a convex empirical risk function subject to an $\ell ^1$ constraint, a pairwise $\ell ^\infty$ constraint, or a pairwise $\ell ^1$ constraint. Existing work, such as the Lasso formulation, has focused mainly on Lagrangian penalty approximations, which often require ad hoc or computationally expensive procedures to determine the penalization parameter. We depart from this approach and address the constrained problem directly via a splitting method. The structure of the method is that of the classical gradient-projection algorithm, which alternates a gradient step on the objective and a projection step onto the lower level set modeling the constraint. The novelty of our approach is that the projection step is implemented via an outer approximation scheme in which the constraint set is approximated by a sequence of simple convex sets consisting of the intersection of two half-spaces. Convergence of the iterates generated by the algorithm is established for a general smooth convex minimization problem with inequality constraints. Experiments on both synthetic and biological data show that our method outperforms penalty methods.

A Logical Approach for Empirical Risk Minimization in Machine Learning for Data Stratification

The data-driven methods capable of understanding, mimicking and aiding the information processing tasks of Machine Learning (ML) have been applied in an increasing range over the past years in diverse areas at a very high rate, and had achieved great success in predicting and stratifying given data instances of a problem domain. There has been generalization on the performance of the classifier to be the optimal based on the existing performance benchmarks such as accuracy, speed, time to learn, number of features, comprehensibility, robustness, scalability and interpretability. However, these benchmarks alone do not guarantee the successful adoption of an algorithm for prediction and stratification since there may be an incurring risk in its adoption. Therefore, this paper aims at developing a logical approach for using Empirical Risk Minimization (ERM) technique to determine the machine learning classifier with the minimum risk function for data stratification. The generalization on the performance of optimal algorithm was tested on BayesNet, Multilayered perceptron, Projective Adaptive Resonance Theory (PART) and Logistic Model Trees algorithms based on existing performance benchmarks such as correctly classified instances, time to build, kappa statistics, sensitivity and specificity to determine the algorithms with great performances. The study showed that PART and Logistic Model Trees algorithms perform well than others. Hence, a logical approach to apply Empirical Risk Minimization technique on PART and Logistic Model Trees algorithms is shown to give a detailed procedure of determining their empirical risk function to aid the decision of choosing an algorithm to be the best fit classifier for data stratification. This therefore serves as a benchmark for selecting an optimal algorithm for stratification and prediction alongside other benchmarks.

Learning Near-Optimal Cost-Sensitive Decision Policy for Object Detection.

Many popular object detectors, such as AdaBoost, SVM and deformable part-based models (DPM), compute additive scoring functions at a large number of windows in an image pyramid, thus computational efficiency is an important consideration in real time applications besides accuracy. In this paper, a decision policy refers to a sequence of two-sided thresholds to execute early reject and early accept based on the cumulative scores at each step. We formulate an empirical risk function as the weighted sum of the cost of computation and the loss of false alarm and missing detection. Then a policy is said to be cost-sensitive and optimal if it minimizes the risk function. While the risk function is complex due to high-order correlations among the two-sided thresholds, we find that its upper bound can be optimized by dynamic programming efficiently. We show that the upper bound is very tight empirically and thus the resulting policy is said to be near-optimal. In experiments, we show that the decision policy outperforms state-of-the-art cascade methods significantly, in several popular detection tasks and benchmarks, in terms of computational efficiency with similar accuracy of detection.

Minimizing response time in time series classification

Providing a timely output is one of the important criteria in applications of time series classification. Recent studies have been motivated to explore models of early prediction, prediction based on truncated temporal observations. The truncation of input improves the response time, but generally reduces the reliability of the prediction. The trade-off between the earliness and the accuracy is an inherent challenge of learning an early prediction model. In this paper, we present an optimization-based approach for learning an ensemble model for timely prediction with an intuitive objective function. The proposed model is comprised of time series classifiers with different response time, and a sequential aggregation procedure to determine the single timing of its output. We formalize the training of the ensemble classifier as a quadratic programming problem and present an iterative algorithm which minimizes an empirical risk function and the response time required to achieve the minimal risk simultaneously. We conduct an empirical study using a collection of behavior and time series datasets to evaluate the proposed algorithm. In the comparisons of the traditional and time-sensitive performance measures, the ensemble framework showed significant advantages over the existing methods on early prediction.

Feature selection based on empirical-risk function to detect lesions in vascular computed tomography

ObjectiveThe overall goal of the study is to detect coronary artery lesions regardless their nature, calcified or hypo-dense. To avoid explicit modelling of heterogeneous lesions, we adopted an approach based on machine learning and using unsupervised or semi-supervised classifiers. The success of the classifiers based on machine learning strongly depends on the appropriate choice of features differentiating between lesions and regular appearance. The specific goal of this article is to propose a novel strategy devised to select the best feature set for the classifiers used, out of a given set of candidate features. Materials and methodsThe features are calculated in image planes orthogonal to the artery centerline, and the classifier assigns to each of these cross-sections a label “healthy” or “diseased”. The contribution of this article is a feature-selection strategy based on the empirical risk function that is used as a criterion in the initial feature ranking and in the selection process itself. We have assessed this strategy in association with two classifiers based on the density-level detection approach that seeks outliers from the distribution corresponding to the regular appearance. The method was evaluated using a total of 13,687 cross-sections extracted from 53 coronary arteries in 15 patients. ResultsUsing the feature subset selected by the risk-based strategy, balanced error rates achieved by the unsupervised and semi-supervised classifiers respectively were equal to 13.5% and 15.4%. These results were substantially better than the rates achieved using feature subsets selected by supervised strategies. The unsupervised and semi-supervised methods also outperformed supervised classifiers using feature subsets selected by the corresponding supervised strategies. DiscussionSupervised methods require large data sets annotated by experts, both to select the features and to train the classifiers, and collecting these annotations is time-consuming. With these methods, lesions whose appearance differs from the training data may remain undetected. Lesion-detection problem is highly imbalanced, since healthy cross-sections usually are much more numerous than the diseased ones. Training the classifiers based on the density-level detection approach needs a small number of annotations or no annotations at all. The same annotations are sufficient to compute the empirical risk and to perform the selection. Therefore, our strategy associated with an unsupervised or semi-supervised classifier requires a considerably smaller number of annotations as compared to conventional supervised selection strategies. The approach proposed is also better suited for highly imbalanced problems and can detect lesions differing from the training set. ConclusionThe risk-based selection strategy, associated with classifiers using the density-level detection approach, outperformed other strategies and classifiers when used to detect coronary artery lesions. It is well suited for highly imbalanced problems, where the lesions are represented as low-density regions of the feature space, and it can be used in other anomaly detection problems interpretable as a binary classification problem where the empirical risk can be calculated.

A Risk Function for Behavioral Disruption of Blainville’s Beaked Whales (Mesoplodon densirostris) from Mid-Frequency Active Sonar

There is increasing concern about the potential effects of noise pollution on marine life in the world’s oceans. For marine mammals, anthropogenic sounds may cause behavioral disruption, and this can be quantified using a risk function that relates sound exposure to a measured behavioral response. Beaked whales are a taxon of deep diving whales that may be particularly susceptible to naval sonar as the species has been associated with sonar-related mass stranding events. Here we derive the first empirical risk function for Blainville’s beaked whales (Mesoplodon densirostris) by combining in situ data from passive acoustic monitoring of animal vocalizations and navy sonar operations with precise ship tracks and sound field modeling. The hydrophone array at the Atlantic Undersea Test and Evaluation Center, Bahamas, was used to locate vocalizing groups of Blainville’s beaked whales and identify sonar transmissions before, during, and after Mid-Frequency Active (MFA) sonar operations. Sonar transmission times and source levels were combined with ship tracks using a sound propagation model to estimate the received level (RL) at each hydrophone. A generalized additive model was fitted to data to model the presence or absence of the start of foraging dives in 30-minute periods as a function of the corresponding sonar RL at the hydrophone closest to the center of each group. This model was then used to construct a risk function that can be used to estimate the probability of a behavioral change (cessation of foraging) the individual members of a Blainville’s beaked whale population might experience as a function of sonar RL. The function predicts a 0.5 probability of disturbance at a RL of 150dBrms re µPa (CI: 144 to 155) This is 15dB lower than the level used historically by the US Navy in their risk assessments but 10 dB higher than the current 140 dB step-function.

Non-asymptotic oracle inequalities for the high-dimensional cox regression via lasso

We consider finite sample properties of the regularized high-dimensional Cox regression via lasso. Existing literature focuses on linear models or generalized linear models with Lipschitz loss functions, where the empirical risk functions are the summations of independent and identically distributed (iid) losses. The summands in the negative log partial likelihood function for censored survival data, however, are neither iid nor Lipschitz.We first approximate the negative log partial likelihood function by a sum of iid non-Lipschitz terms, then derive the non-asymptotic oracle inequalities for the lasso penalized Cox regression using pointwise arguments to tackle the difficulties caused by lacking iid Lipschitz losses.

Tensor Learning for Regression

In this paper, we exploit the advantages of tensorial representations and propose several tensor learning models for regression. The model is based on the canonical/parallel-factor decomposition of tensors of multiple modes and allows the simultaneous projections of an input tensor to more than one direction along each mode. Two empirical risk functions are studied, namely, the square loss and ε -insensitive loss functions. The former leads to higher rank tensor ridge regression (TRR), and the latter leads to higher rank support tensor regression (STR), both formulated using the Frobenius norm for regularization. We also use the group-sparsity norm for regularization, favoring in that way the low rank decomposition of the tensorial weight. In that way, we achieve the automatic selection of the rank during the learning process and obtain the optimal-rank TRR and STR. Experiments conducted for the problems of head-pose, human-age, and 3-D body-pose estimations using real data from publicly available databases, verified not only the superiority of tensors over their vector counterparts but also the efficiency of the proposed algorithms.