Kernel Ridge Regression Research Articles

Machine learning approach for screening alloy surfaces for stability in catalytic reaction conditions

A catalytic surface should be stable under reaction conditions to be effective. However, it takes significant effort to screen many surfaces for their stability, as this requires intensive quantum chemical calculations. To more efficiently estimate stability, we provide a general and data-efficient machine learning (ML) approach to accurately and efficiently predict the surface energies of metal alloy surfaces. Our ML approach introduces an element-centered fingerprint (ECFP) which was used as a vector representation for fitting models for predicting surface formation energies. The ECFP is significantly more accurate than several existing feature sets when applied to dilute alloy surfaces and is competitive with existing feature sets when applied to bulk alloy surfaces or gas-phase molecules. Models using the ECFP as input can be quite general, as we created models with good accuracy over a broad set of bimetallic surfaces including most d-block metals, even with relatively small datasets. For example, using the ECFP, we developed a kernel ridge regression ML model which is able to predict the surface energies of alloys of diverse metal combinations with a mean absolute error of 0.017 eV atom−1. Combining this model with an existing model for predicting adsorption energies, we estimated segregation trends of 596 single-atom alloys (SAAs)with and without CO adsorbed on these surfaces. As a simple test of the approach, we identify specific cases where CO does not induce segregation in these SAAs.

Using a machine learning-based risk prediction model to analyze the coronary artery calcification score and predict coronary heart disease and risk assessment

ObjectivesTo calculate the coronary artery calcification score (CACS) obtained from coronary artery computed tomography angiography (CCTA) examination and combine it with the influencing factors of coronary artery calcification (CAC), which is then analyzed by machine learning (ML) to predict the probability of coronary heart disease(CHD). MethodsAll patients who were admitted to the Affiliated Hospital of Traditional Chinese Medicine of Southwest Medical University from January 2019 to March 2022, suspected of CHD, and underwent CCTA inspection were retrospectively selected. The degree of CAC was quantified based on the Agatston score. To compare the correlation between the CACS and clinical-related factors, we collected 31 variables, including hypertension, diabetes, smoking, hyperlipidemia, among others. ML models containing the random forest (RF), radial basis function neural network (RBFNN),support vector machine (SVM),K-Nearest Neighbor algorithm (KNN) and kernel ridge regression (KRR) were used to assess the risk of CHD based on CACS and clinical-related factors. ResultsAmong the five ML models, RF achieves the best performance about accuracy (ACC) (78.96%), sensitivity (SN) (93.86%), specificity(Spe) (51.13%), and Matthew's correlation coefficient (MCC) (0.5192).It also has the best area under the receiver operator characteristic curve (ROC) (0.8375), which is far superior to the other four ML models. ConclusionComputer ML model analysis confirmed the importance of CACS in predicting the occurrence of CHD, especially the outstanding RF model, making it another advancement of the ML model in the field of medical analysis.

A comparative study of different machine learning methods for dissipative quantum dynamics

It has been recently shown that supervised machine learning (ML) algorithms can accurately and efficiently predict long-time population dynamics of dissipative quantum systems given only short-time population dynamics. In the present article we benchmarked 22 ML models on their ability to predict long-time dynamics of a two-level quantum system linearly coupled to harmonic bath. The models include uni- and bidirectional recurrent, convolutional, and fully-connected feedforward artificial neural networks (ANNs) and kernel ridge regression (KRR) with linear and most commonly used nonlinear kernels. Our results suggest that KRR with nonlinear kernels can serve as inexpensive yet accurate way to simulate long-time dynamics in cases where the constant length of input trajectories is appropriate. Convolutional gated recurrent unit model is found to be the most efficient ANN model.

Improving Genomic Prediction with Machine Learning Incorporating TPE for Hyperparameters Optimization.

Depending on excellent prediction ability, machine learning has been considered the most powerful implement to analyze high-throughput sequencing genome data. However, the sophisticated process of tuning hyperparameters tremendously impedes the wider application of machine learning in animal and plant breeding programs. Therefore, we integrated an automatic tuning hyperparameters algorithm, tree-structured Parzen estimator (TPE), with machine learning to simplify the process of using machine learning for genomic prediction. In this study, we applied TPE to optimize the hyperparameters of Kernel ridge regression (KRR) and support vector regression (SVR). To evaluate the performance of TPE, we compared the prediction accuracy of KRR-TPE and SVR-TPE with the genomic best linear unbiased prediction (GBLUP) and KRR-RS, KRR-Grid, SVR-RS, and SVR-Grid, which tuned the hyperparameters of KRR and SVR by using random search (RS) and grid search (Gird) in a simulation dataset and the real datasets. The results indicated that KRR-TPE achieved the most powerful prediction ability considering all populations and was the most convenient. Especially for the Chinese Simmental beef cattle and Loblolly pine populations, the prediction accuracy of KRR-TPE had an 8.73% and 6.08% average improvement compared with GBLUP, respectively. Our study will greatly promote the application of machine learning in GP and further accelerate breeding progress.

Computational simulation and target prediction studies of solubility optimization of decitabine through supercritical solvent

Computational analysis of drug solubility was carried out using machine learning approach. The solubility of Decitabine as model drug in supercritical CO2 was studied as function of pressure and temperature to assess the feasibility of that for production of nanomedicine to enhance the solubility. The data was collected for solubility optimization of Decitabine at the temperature 308–338 K, and pressure 120–400 bar used as the inputs to the machine learning models. A dataset of 32 data points and two inputs (P and T) have been applied to optimize the solubility. The only output is Y = solubility, which is Decitabine mole fraction solubility in the solvent. The developed models are three models including Kernel Ridge Regression (KRR), Decision tree Regression (DTR), and Gaussian process (GPR), which are used for the first time as a novel model. These models are optimized using their hyper-parameters tuning and then assessed using standard metrics, which shows R2-score, KRR, DTR, and GPR equal to 0.806, 0.891, and 0.998. Also, the MAE metric shows 1.08E−04, 7.40E−05, and 9.73E−06 error rates in the same order. The other metric is MAPE, in which the KRR error rate is 4.64E−01, DTR shows an error rate equal to 1.63E−01, and GPR as the best mode illustrates 5.06E−02. Finally, analysis using the best model (GPR) reveals that increasing both inputs results in an increase in the solubility of Decitabine. The optimal values are (P = 400, T = 3.38E + 02, Y = 1.07E−03).

Affinity based fuzzy kernel ridge regression classifier for binary class imbalance learning

The class imbalance learning (CIL) problem indicates when one class have very low proportions of samples (minority class) compared to the other class (majority class). Even though kernel ridge regression (KRR) shows high generalization ability at a considerably quicker learning speed than conventional machine learning algorithms, it fails to achieve an excellent result for CIL problems. To address this inherent limitation of KRR, a novel affinity-based fuzzy KRR (AFKRR) is proposed for dealing with the binary CIL problem. In AFKRR, an affinity-based fuzzy membership value is linked to each training sample. The affinity of the majority class datapoint is measured using the support vector data description (SVDD) trained by the majority class datapoints. The classification ability of the proposed AFKRR is calculated using the area under the receiver optimal characteristics curve (AUC), F-measure and Geometric mean. AFKRR’s performance is compared with the support vector machine (SVM), affinity and class probability-based fuzzy SVM (ACFSVM), entropy-based fuzzy SVM (EFSVM), improved density weighted least squares SVM (IDLSSVM-CIL), KRR and intuitionistic fuzzy KRR (IFKRR) models on a few real-world as well as a few artificial imbalanced datasets. Experimental outcomes reveal the usability and efficacy of the proposed AFKRR model.

Machine Learning Diffusion Monte Carlo Energies.

We present two machine learning methodologies that are capable of predicting diffusion Monte Carlo (DMC) energies with small data sets (≈60 DMC calculations in total). The first uses voxel deep neural networks (VDNNs) to predict DMC energy densities using Kohn-Sham density functional theory (DFT) electron densities as input. The second uses kernel ridge regression (KRR) to predict atomic contributions to the DMC total energy using atomic environment vectors as input (we used atom-centered symmetry functions, atomic environment vectors from the ANI models, and smooth overlap of atomic positions). We first compare the methodologies on pristine graphene lattices, where we find that the KRR methodology performs best in comparison to gradient boosted decision trees, random forest, Gaussian process regression, and multilayer perceptrons. In addition, KRR outperforms VDNNs by an order of magnitude. Afterward, we study the generalizability of KRR to predict the energy barrier associated with a Stone-Wales defect. Lastly, we move from 2D to 3D materials and use KRR to predict total energies of liquid water. In all cases, we find that the KRR models are more accurate than Kohn-Sham DFT and all mean absolute errors are less than chemical accuracy.

Improving asphalt mix design by predicting alligator cracking and longitudinal cracking based on machine learning and dimensionality reduction techniques

• Reasonable asphalt mixture design can mitigate the cracking problems. • Alligator cracking (AC) and longitudinal (LC) cracking are predicted from asphalt mixture properties. • The effect of three different dimension reduction methods on the performance of ML models is analyzed. • ML algorithms includes Support vector regression, Artificial Neural Networks, Kernel ridge regression, Gradient boosting, Extra-trees, eXtreme Gradient Boosting. • The paper improves the current asphalt mix design by adding the criteria of AC and LC and ML models. The asphalt mix design based on traditional laboratory fatigue cracking tests of asphalt mixture is not reasonable due to the difficulty of simulating the circumstance where asphalt mixture experiences in the pavement structure and under realistic climate and traffic conditions. The study aims to mitigate the cracking problems during the pavement design life, by improving the asphalt mix design process by introducing machine learning (ML) models, which are used to predict alligator cracking (AC) and longitudinal cracking (LC), and their criteria (required ranges). The data containing AC and LC were extracted from the NCHRP 1-37A report and the long-term pavement performance (LTPP) program. A total of 33 input features about climate condition, traffic, pavement structure, and materials properties of each pavement layer were selected. Support Vector Regression (SVR), Artificial Neural Networks (ANN), Kernel ridge regression (KRR), Gradient boosting (GB), Extra-trees, and eXtreme Gradient Boosting (XGBoost) were used. Meanwhile, three dimensionality reduction approaches, including Auto-Encoder (AE), Principal Component Analysis (PCA), and Recursive feature elimination with Random Forest, were combined with the six ML algorithms (18 hybrid models produced) in order to decrease the computation complexity and search for the optimum model. The 24 models (6 basic ML models plus 18 hybrid models) were trained, and their hyperparameters were tuned using Bayesian optimization. Different evaluation criteria R 2 , RMSE, MAE, SMAPE, and SI are calculated to evaluate the performance of these models. The results of the study showed that except for ANN and SVR, the basic ML models can get better for predicting AC and LC when they are combined with PCA or AE. Based on the performance indices, the optimum among these developed models proved to be PCA-ANN for predicting AC (R 2 = 0.84) and LC (R 2 = 0.83). Compared to other pavement layers, the asphalt surface course is the highest contributor to AC and LC. The improved mix design process was implemented to determine the mix proportion for the asphalt surface course of a construction project and Mix-2 with 5.1 % asphalt content was recommended.

Novel numerical simulation of drug solubility in supercritical CO2 using machine learning technique: Lenalidomide case study

In pharmaceutical industry, finding promising ways to enhance the solubility of disparate types of drugs is an important challenge for the orally administered drug delivery system. Disparate techniques based on drug characteristics, nature of dosage form and properties of excipients have recently been under extensive evaluation all over the world to improve the solubility of poorly water-soluble drugs. Among them, supercritical fluid carbon dioxide (SC-CO 2 ) has received paramount attentions due to having considerable advantages like cost-effectiveness and low flammability. Lenalidomide belongs is an orally administered anti-cancer agent, which has recently received indication for the treatment of adult patients with different bone marrow-related malignancies such as multiple myeloma, mantle cell lymphoma and follicular lymphoma. Predicting the optimized value of Lenalidomide inside the SC-CO 2 in a wide range of pressure and temperature via developing mathematical models based on artificial intelligence (AI) is the main objective of this paper. In this study, three different machine learning based models are selected to predict and optimized the drug solubility. The available data includes 28 rows of data with two inputs including temperature and pressure and two outputs including density and solubility. Selected models are Kernel Ridge Regression (KRR), least angle regression (LAR), and Multilayer Perceptron (MLP). After optimizing models and comparing the results, the MLP was selected as the primary model of this research. The models illustrated R-squared scores of 0.999 and 0.994 for density and solubility. The maximum errors are also 2.92 and 6.44 × 10 -2 for these outputs, which shows the accuracy and significant generality of the model.

Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime* *This article is an updated version of: Cui H, Loureiro B, Krzakala F and Zdeborová L 2021 Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin,

In this manuscript we consider kernel ridge regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the features co-variance. These decays were, however, provided for sizeably different setups, namely in the noiseless case with constant regularization and in the noisy optimally regularized case. Intermediary settings have been left substantially uncharted. In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization. In particular, we show the existence of a transition in the noisy setting between the noiseless exponents to its noisy values as the sample complexity is increased. Finally, we illustrate how this crossover can also be observed on real data sets.

Effective artificial neural network-based wind power generation and load demand forecasting for optimum energy management

The variability of power production from renewable energy sources (RESs) presents serious challenges in energy management (EM) and power system stability. Power forecasting plays a crucial role in optimal EM and grid security. Then, accurate power forecasting ensures optimum scheduling and EM. Therefore, this study proposes an artificial neural network- (ANN-) based paradigm to predict wind power (WP) generation and load demand, where the meteorological parameters, including wind speed, temperature, and atmospheric pressure, are fed to the model as inputs. The normalized root mean square error (NRMSE) and normalized mean absolute error (NMAE) criteria are used to evaluate the forecasting technique. The performance of ANN was compared to four machine learning methods: LASSO, decision tree (DT), regression vector machines (RVM), and kernel ridge regression (KRR). The obtained results show that ANN provides high effectiveness and accuracy for WP forecasting. Furthermore, ANN has proven to be an interesting tool in ensuring optimum scheduling and EM.

Machine learning-based predictive models for equivalent damping ratio of RC shear walls

Energy-based seismic design is being rapidly developed and suggests that the seismic demands are met by the energy dissipation capacity of the structural members. Equivalent damping ratio is a measure of energy dissipation in structural members that accounts for the post-elastic behavior of the member and provides insight regarding the dynamic response reduction during a seismic event. The present study implements a machine learning algorithm to estimate the equivalent damping ratio in reinforced concrete shear walls at displacements corresponding to a 1.0% lateral drift ratio. Five different machine learning models, namely, Robust Linear Regression, K-Nearest Neighbor Regression, Kernel Ridge Regression, Support Vector Regression, and Gaussian process regression were evaluated in order to choose the model with the highest accuracy. Among all models, Gaussian process regression, a machine learning method with successful implementation experiences in civil/structural engineering related problems, is selected to identify the equivalent damping ratio. The developed GPR-based algorithm uses a database of 161 rectangular shear walls subjected to quasi-static reversed cyclic loading with geometry and mechanical properties commonly found in building stocks of many earthquake-prone countries. The proposed algorithm estimates the equivalent damping ratio for each specimen by predicting the cyclic dissipated energy and lateral force values as two dependent variables. The model validation results show a mean coefficient of determination (R2) of about 0.89; a relative root mean square error of about 0.14 and a mean absolute percentage error of 10.44%, which is considered a substantially accurate prediction for such a complex problem. An open-source model and the entire database are provided which can be used by researchers and also design engineers. The proposed predictive model enables comparing the damping capacity of shear walls and the outcomes of this study are believed to contribute to the energy-based design or performance evaluation procedures in terms of predicting the energy capacity of shear walls.

Design of data-driven model for the pressurizer system in nuclear power plants using a TSK fuzzy neural network

In pressurized water reactors (PWRs), the main role of the pressurizer unit is to maintain the primary circuit pressure of the reactor within preset limits. To ensure this role, the pressure and the water level control systems for the pressurizer system are very significant. Hence, it is very important to monitor and predict them with great accuracy during operation. In this paper, a data-driven model using a TSK fuzzy neural network (TSKFNN) for the prediction of the pressure and the water level inside the pressurizer is developed. The identification task of the TSKFNN model of the pressurizer system is fulfilled in two steps: the structure and the parameters identification. In the first step, the fuzzy C-means clustering method is used to separate the input data into clusters and obtain the rule antecedent parameters of the TSKFNN model. Besides, the initial values of the rule consequent parameters are defined using the kernel ridge regression algorithm. In the second step, the sliding-window Kernel Recursive Least Squares (KRLS) algorithm and the gradient method are developed to adapt TSKFNN model parameters. By using input/ output data collected from the PCTran VVER-1200 simulator, the data-driven model was trained, tested, and evaluated. The simulation results demonstrate that the data-driven model using the TSKFNN structure can effectively predict the pressure and the water level in the pressurizer system. Furthermore, the simulation results also involve a comparison with two other mathematical models to confirm the effectiveness of the developed data-driven model.

Dynamic assessment of slope stability based on multi‐source monitoring data and ensemble learning approaches: A case study of Jiuxianping landslide

Accurate assessment of slope stability is the most important task in geological disaster prevention and control. This study developed an ensemble learning approach based on stacking strategy and eight commonly used machine learning (ML) models, for exploring the feasibility of the factor of safety (FS) prediction using dynamic multi‐source monitoring data of slopes and landslides. Based on long‐term and dynamic field monitoring and numerical calculation, a dataset for constructing the FS prediction model for the Jiuxianping landslide was established. The dataset includes five types of monitoring data namely rainfall, reservoir water level, groundwater level, surface displacement and deep displacement for a total of nine features, and one label FS. Four regularized regression models, kernel ridge regression (KRR), lasso, elastic net and support vector regression (SVR), as well as four ensemble learning models, random forest (RF), gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost) and light gradient boosting machine (LightGBM), were adopted to obtain the nonlinear association between the nine features and the label FS, respectively. Based on five repeated 5‐fold cross‐validation (CV) and successive halving (SH) hyperparameter searching method, the hyperparameters of each model were determined, and the prediction effects of each optimal model were compared. The results show that the ensemble learning models outperform the common regression models. Furthermore, with the help of the stacking ensemble learning thinking, four excellent ensemble models were combined, and the final stacking ensemble learning model was used to predict the FS of the Jiuxianping landslide. The results indicate that the stacking model has better robustness and generalization performance. Besides, the feature relative importance of four ensemble learning models was analysed, for enhancing the interpretability of ML models and pointing out the research direction of feature engineering in the future.

Uncovering Molecular Structure – Redox Potential Relationships for Organic Electrode Materials: A Hybrid DFT – Machine Learning Approach

Density functional theory and machine learning are used to investigate the structure-electrochemical performance relationships of organic moieties for use in Li-ion batteries. Namely, DFT calculations are performed to predict the redox potential of several novel organic molecules with an accuracy within ~0.1 V of experimental measurements. However, despite its ability to provide valuable insight regarding the electrochemical properties of novel organic molecules, our high efficacy DFT modeling protocol demands significant computational time and is therefore impractical for the vast screening of novel material candidate. As a result, we explore machine learning as a strategy for the accelerated discovery of novel organic materials. More critically, we use machine learning as a method for assessing the various structure-electrochemical relationships which can provide a more general guideline for the design of organic electrode materials. We are employing different learning models, including artificial neural networks, gradient-boosting regression, and kernel methods (such as kernel ridge regression and Gaussian process regression), via three different pipelines with varying sophistication with the aim of generating an advanced ML scheme for the accurate prediction and analysis of electrochemical activity. In addition to incorporating structural fingerprints, we are exploring an active learning framework, namely using the efficient global optimization scheme, to explore the materials space strategically using publicly available datasets. Through this approach, we can discover new electrode materials that could 1) have a higher probability for achieving enhanced electrochemical properties and 2) increase our learning model’s performance by increasing our dataset’s representation of the material space. Additionally, we are implementing a high-throughput virtual screening (HTVS) pipeline which consists of several surrogate learning models with increasing levels of fidelity. The material candidates are pruned at each stage to discard samples that are unlikely to possess a desirable redox potential for cathodic application, and the remaining samples move on to the next model. In this way, only a subset of the original dataset which has a higher probability for lying within the desired redox potential range is pursued using the most computationally expensive model.

Exploring soot inception rate with stochastic modelling and machine learning

A diverse range of polycyclic aromatic compounds (PACs) is thought to exist in flame environments before and during soot inception. This work seeks to develop a machine learning (ML)-based soot inception model that considers detailed and diverse PAC properties such as oxygenation, aliphatic content, radical character, size, and shape. To this end, temporal rates of change of PAC properties were computed by the stochastic modelling code SNapS2 and used as input to an ML model that predicts soot inception rate. The model is trained using experimentally-derived soot inception rates for three atmospheric pressure laminar premixed ethylene/air flames. An ML model (kernel ridge regression with a linear kernel) was developed to predict the soot inception rate in the three premixed flames. The soot inception rate predictions from this SNapS2-informed ML model outperformed the predictions from both the advanced soot modelling CFD code CoFlame and an ML model which used CFD-determined inputs (temperature and species concentrations). The final model had an R2 value of approximately 0.71 and a mean absolute error approximately 25% of the target values. The performance of the SNapS2-informed model suggests that detailed PAC properties are important to consider in inception modelling. While expanding this approach to other types of flames and fuels is crucial for future improvement to the model’s accuracy and generality, this methodology provides a successful framework for the current system. The success of this method demonstrates that ML can offer improvements in accuracy compared to current CFD inception models and the highlights the potential for ML in soot predictions.

Machine learning-enabled nanosafety assessment of multi-metallic alloy nanoparticles modified TiO2 system

Establishing toxicological predictive modeling frameworks for heterogeneous nanomaterials is crucial for rapid environmental and health risk assessment. However, existing structure-toxicity correlation models for such nanomaterials are only based on simple linear regression algorithms that are prone to underfitting the training data. These models rely heavily on experimental and expensive computational quantum mechanical descriptors, which significantly limit their practical use. Herein, we present the application of empirical descriptors and complex machine learning algorithms to the development of high-performance quantitative structure-toxicity relationship (QSTR) models of TiO2 hybridized with multi-metallic (Ag, Au, Pt) alloy nanoparticles (multi-metallic NPs/TiO2). To confirm the viability of empirical descriptors as model input, we selected five distinct machine learning algorithms for predicting the toxicity of multi-metallic alloy NPs/TiO2 system in Chinese hamster ovary cell line. Notably, an empirical descriptor-based QSTR model (kernel ridge regression) revealed a predictive performance that is on par with density functional theory (DFT) descriptor-based counterparts. More specifically, the results indicated that model selection is influenced by descriptor choice, such that complex DFT descriptors worked best with a complex algorithm (random forest regression; RMSET = 0.0954, MAET = 0.0811, RT2 = 0.9411), whereas more straightforward empirical descriptors were most suitable with a simpler algorithm (kernel ridge regression; RMSET = 0.1244, MAET = 0.1106, RT2 = 0.8999). Moreover, our model outperforms existing QSAR models built on the same data set. This study offers a new perspective on using empirical features to develop accurate predictive computational models for the rapid discovery and profiling of safe-by-design nanomaterials.

Manifold learning for global active noise control: Sensor interpolation

To achieve global control, unmeasured frequency responses (FRs) of a control source and preselected control points are required. In this paper, sensor interpolation (SI) approaches based on plane wave decomposition (PWD), kernel ridge regression, deep neural network (DNN) are presented. A broadened control region encompassing a large number of measured and fictitious control points can be obtained using the SI techniques. PWD is a two-stage procedure that begins with trimming down plane wave components, followed by an extraction of the associated complex amplitudes. In kernel ridge regression, a reproducing kernel in the Hilbert space (RKHS) is utilized to interpolate continuous relative transfer functions. The DNN-based SI approach is inspired by the kernel method. Convolutional neural networks and the like are exploited to interpolate the FRs at the fictitious points. Simulations are conducted for a thirty-microphone uniform linear array. From the results, an excellent fit between the FRs at the fictitious points and the ground truth generated by the image source method can be obtained below the spatial aliasing frequency. The efficacy of the interpolated FRs in relation to global control is demonstrated via an example of feedforward active noise control using a linear loudspeaker array.

Approximate kernel PCA: Computational versus statistical trade-off

Kernel methods are powerful learning methodologies that allow to perform nonlinear data analysis. Despite their popularity, they suffer from poor scalability in big data scenarios. Various approximation methods, including random feature approximation, have been proposed to alleviate the problem. However, the statistical consistency of most of these approximate kernel methods is not well understood except for kernel ridge regression wherein it has been shown that the random feature approximation is not only computationally efficient but also statistically consistent with a minimax optimal rate of convergence. In this paper, we investigate the efficacy of random feature approximation in the context of kernel principal component analysis (KPCA) by studying the trade-off between computational and statistical behaviors of approximate KPCA. We show that the approximate KPCA is both computationally and statistically efficient compared to KPCA in terms of the error associated with reconstructing a kernel function based on its projection onto the corresponding eigenspaces. The analysis hinges on Bernstein-type inequalities for the operator and Hilbert–Schmidt norms of a self-adjoint Hilbert–Schmidt operator-valued U-statistics, which are of independent interest.

The interpolation phase transition in neural networks: Memorization and generalization under lazy training

Modern neural networks are often operated in a strongly overparametrized regime: they comprise so many parameters that they can interpolate the training set, even if actual labels are replaced by purely random ones. Despite this, they achieve good prediction error on unseen data: interpolating the training set does not lead to a large generalization error. Further, overparametrization appears to be beneficial in that it simplifies the optimization landscape. Here, we study these phenomena in the context of two-layers neural networks in the neural tangent (NT) regime. We consider a simple data model, with isotropic covariates vectors in d dimensions, and N hidden neurons. We assume that both the sample size n and the dimension d are large, and they are polynomially related. Our first main result is a characterization of the eigenstructure of the empirical NT kernel in the overparametrized regime Nd≫n. This characterization implies as a corollary that the minimum eigenvalue of the empirical NT kernel is bounded away from zero as soon as Nd≫n and, therefore, the network can exactly interpolate arbitrary labels in the same regime. Our second main result is a characterization of the generalization error of NT ridge regression including, as a special case, min-ℓ2 norm interpolation. We prove that, as soon as Nd≫n, the test error is well approximated by the one of kernel ridge regression with respect to the infinite-width kernel. The latter is in turn well approximated by the error of polynomial ridge regression, whereby the regularization parameter is increased by a “self-induced” term related to the high-degree components of the activation function. The polynomial degree depends on the sample size and the dimension (in particular on logn/logd).