Regularized Learning Algorithm Research Articles

Online regularized learning algorithm for functional data

In recent years, functional linear models have attracted growing attention in statistics and machine learning for recovering the slope function or its functional predictor. This paper considers online regularized learning algorithm for functional linear models in a reproducing kernel Hilbert space. It provides convergence analysis of excess prediction error and estimation error with polynomially decaying step-size and constant step-size, respectively. Fast convergence rates can be derived via a capacity dependent analysis. Introducing an explicit regularization term extends the saturation boundary of unregularized online learning algorithms with polynomially decaying step-size and achieves fast convergence rates of estimation error without capacity assumption. In contrast, the latter remains an open problem for the unregularized online learning algorithm with decaying step-size. This paper also demonstrates competitive convergence rates of both prediction error and estimation error with constant step-size compared to existing literature.

Improving the Performance of Sentiment Analysis by Ensemble Hybrid Learning Algorithm With NLP And Cascaded Feature Extraction

Sentiment analysis is a challenging problem in Natural Language Processing since every language has its own character within several difficulties such as ambiguity, synonymy, negative suffixes…etc. Since words with ambiguity can have different sentiment scores depending on the meaning they have in their corresponding context, we accomplished a study on Turkish language to determine whether the polarity scores of these polysemous words may differ according to their meaning. For a word with ambiguity, we first made a polarity calculation module to calculate its polarity score. This way, we calculated the polarity scores of 100 Turkish polysemous words. Then, since negation directly affects the correct meaning of the word in the sentiment analysis, a negation handler module is also implemented. After that, we prepared a sentiment polarity corpus which consists of 159,876 Turkish words including 100 Turkish polysemous words. Actually, the main purpose of this study is to detect sentiment polarity of Turkish texts by considering and building a specialized module for polysemous words. In short, we built a system for Turkish sentiment polarity detection task including these modules: 1) Pre-processing, 2) Polarity Calculation Module, 3) Negation Handling Module, 4) Feature Generation Module, and 5) Classification Module. According to our knowledge, this is the first study which includes all of these modules in one Turkish sentiment analysis task. Finally, we conducted this corpus using an ensemble hybrid regularized learning algorithm on two self-collected Twitter-datasets. Experimental results show that the suggested approach improves the classification performance on Turkish sentiment analysis task.

Short-Term Electrical Load Forecasting Based on Neural Networks

The article sets out the use of an artificial neural network for electrical load consumption forecasting. Electrical load prediction is one of the most demanded areas of research in the electric power industry. The two-layer artificial neural network of direct distribution with the number of neurons in the hidden layer equal to 6 was proposed in the article. The following data was taken as an input for forecasting short-term electrical load: electrical load, time, day of the week, temperature, day, weekend and working day code. According to the studies, the value of the mean absolute percentage error was 2.35, using the Bayesian Regularization learning algorithm.

A Machine Learning Architecture Replacing Heavy Instrumented Laboratory Tests: In Application to the Pullout Capacity of Geosynthetic Reinforced Soils.

For economical and sustainable benefits, conventional retaining walls are being replaced by geosynthetic reinforced soil (GRS). However, for safety and quality assurance purposes, prior tests of pullout capacities of these materials need to be performed. Conventionally, these tests are conducted in a laboratory with heavy instruments. These tests are time-consuming, require hard labor, are prone to error, and are expensive as a special pullout machine is required to perform the tests and acquire the data by using a lot of sensors and data loggers. This paper proposes a data-driven machine learning architecture (MLA) to predict the pullout capacity of GRS in a diverse environment. The results from MLA are compared with actual laboratory pullout capacity tests. Various input variables are considered for training and testing the neural network. These input parameters include the soil physical conditions based on water content and external loading applied. The soil used is a locally available weathered granite soil. The input data included normal stress, soil saturation, displacement, and soil unit weight whereas the output data contains information about the pullout strength. The data used was obtained from an actual pullout capacity test performed in the laboratory. The laboratory test is performed according to American Society for Testing and Materials (ASTM) standard D 6706-01 with little modification. This research shows that by using machine learning, the same pullout resistance of a geosynthetic reinforced soil can be achieved as in laboratory testing, thus saving a lot of time, effort, and money. Feedforward backpropagation neural networks with a different number of neurons, algorithms, and hidden layers have been examined. The comparison of the Bayesian regularization learning algorithm with two hidden layers and 12 neurons each showed the minimum mean square error (MSE) of 3.02 × 10-5 for both training and testing. The maximum coefficient of regression (R) for the testing set is 0.999 and the training set is 0.999 for the prediction interval of 99%.

The kernel regularized learning algorithm for solving Laplace equation with Dirichlet boundary

In this paper, we give an investigation on the problem of solving Laplace equation with the kernel regularized regression. We provide a Sobolev type space corresponding to the Dirichlet boundary value problem on a compact domain, and with which define a reproducing kernel space (RKS), which is used as the hypothesis space for constructing kernel regularized learning algorithm. We give theory analysis for the convergence of the learning algorithm, bound an upper bound for the error. The discussions show that the learning rate is controlled by a [Formula: see text]-functional corresponding to the RKS. As an application we give the learning rate in case that the domain is the unit ball. The simulations show that the algorithm has better fitting performance. The investigations show that the problem of solving an elliptic boundary problem can be attributed to constructing an orthonormal basis with respect to a bilinear form corresponding to the boundary value condition.

Empirical approach for prediction of bearing pressure of spread footings on clayey soil using artificial intelligence (AI) techniques

This study investigates the applicability of two artificial intelligence (AI) techniques, namely, the support vector regression (SVR) and artificial neural network (ANN), in prediction of the bearing pressure of spread footings on clayey soils based on plate load test (PLT) data. A data set consisting of 576 numbers of data points from 58 numbers of full-scale and small-scale PLTs was collected from the literature. 70% of the data were used for training and the remaining 30% of data were used to test the AI models. The data division was checked with adequate statistical tests. A correlation analysis was performed to optimize the number of inputs. Three SVR models, namely, the polynomial kernel function (POLY), radial basis kernel function (RBF), and exponential radial basis kernel function (ERBF), and one ANN model with the Bayesian Regularization (BR) learning algorithm were used in the analysis. The model performances were evaluated by comparing various error parameters. The ANN-BR model showed the best performance with minimum error and a correlation co-efficient (R) of 0.9851 for the overall dataset. A sensitivity analysis indicates that the undrained cohesion has a maximum influence on the bearing pressure. Finally, an empirical approach is presented using the best-performing model to determine the soil bearing pressure.

Developing an approach for fast estimation of range of ion in interaction with material using the Geant4 toolkit in combination with the neural network

Precise modelling of the interaction of ions with materials is important for many applications including material characterization, ion implantation in devices, thermonuclear fusion, hadron therapy, secondary particle production (e.g. neutron), etc. In this study, a new approach using the Geant4 toolkit in combination with the Bayesian regularization (BR) learning algorithm of the feed-forward neural network (FFNN) is developed to estimate the range of ions in materials accurately and quickly. The different incident ions at different energies are interacted with the target materials. The Geant4 is utilized to model the interactions and to calculate the range of the ions. Afterward, the appropriate architecture of the FFNN-BR with the relevant input features is utilized to learn the modelled ranges and to estimate the new ranges for the new cases. The notable achievements of the proposed approach are: 1- The range of ions in different materials is given as quickly as possible and the time required for estimating the ranges can be neglected (i.e. less than 0.01 s by a typical personal computer). 2- The proposed approach can generalize its ability for estimating the new untrained cases. 3- There is no need for a pre-made lookup table for the estimation of the range values.

On a regularization of unsupervised domain adaptation in RKHS

We analyze the use of the so-called general regularization scheme in the scenario of unsupervised domain adaptation under the covariate shift assumption. Learning algorithms arising from the above scheme are generalizations of importance weighted regularized least squares method, which up to now is among the most used approaches in the covariate shift setting. We explore a link between the considered domain adaptation scenario and estimation of Radon-Nikodym derivatives in reproducing kernel Hilbert spaces, where the general regularization scheme can also be employed and is a generalization of the kernelized unconstrained least-squares importance fitting. We estimate the convergence rates of the corresponding regularized learning algorithms and discuss how to resolve the issue with the tuning of their regularization parameters. The theoretical results are illustrated by numerical examples, one of which is based on real data collected for automatic stenosis detection in cervical arteries.

Developing an approach for maximizing neutron activation reaction rate by optimizing moderator dimensions and target position using the Monte Carlo code in combination with the GA and ANN algorithms

In this study, in order to maximize the reaction rate of neutron activation (NA), an approach using combination of the MCNP code, the feed-forward neural network with the Bayesian regularization (FFNN-BR) learning algorithm, and the genetic algorithm (GA) is proposed. The MCNP code calculates the reaction rates based on the different moderator dimensions/ target positions. The calculated reaction rates with appropriate features (i.e. RT, R2S, and Z2S) are applied for training of the FFNN-BR. The trained neural network is utilized for estimating the reaction rates of the generated individuals by the GA. The results show that the trained neural network estimates the reaction rates with acceptable accuracy. For selection of the appropriate individual for maximizing the reaction rate, the distance of the source from the target as well as the value of reaction rate and the moderator dimensions are considered. The proposed approach in this study gives a simple and practical algorithm for maximizing the NA reaction rate without complexities related to the nature of the problem.

Compositional learning of mutually recursive procedural systems

This paper presents a compositional approach to active automata learning of Systems of Procedural Automata (SPAs), an extension of Deterministic Finite Automata (DFAs) to systems of DFAs that can mutually call each other. SPAs are of high practical relevance, as they allow one to efficiently learn intuitive recursive models of recursive programs after an easy instrumentation that makes calls and returns observable. Key to our approach is the simultaneous inference of individual DFAs for each of the involved procedures via expansion and projection: membership queries for the individual DFAs are expanded to membership queries of the entire SPA, and global counterexample traces are transformed into counterexamples for the DFAs of concerned procedures. This reduces the inference of SPAs to a simultaneous inference of the DFAs for the involved procedures for which we can utilize various existing regular learning algorithms. The inferred models are easy to understand and allow for an intuitive display of the procedural system under learning that reveals its recursive structure. We implemented the algorithm within the LearnLib framework in order to provide a ready-to-use tool for practical application which is publicly available on GitHub for experimentation.

Analysis of regularized least-squares in reproducing kernel Kreĭn spaces

In this paper, we study the asymptotic properties of regularized least squares with indefinite kernels in reproducing kernel Krein spaces (RKKS). By introducing a bounded hyper-sphere constraint to such non-convex regularized risk minimization problem, we theoretically demonstrate that this problem has a globally optimal solution with a closed form on the sphere, which makes approximation analysis feasible in RKKS. Regarding to the original regularizer induced by the indefinite inner product, we modify traditional error decomposition techniques, prove convergence results for the introduced hypothesis error based on matrix perturbation theory, and derive learning rates of such regularized regression problem in RKKS. Under some conditions, the derived learning rates in RKKS are the same as that in reproducing kernel Hilbert spaces (RKHS), which is actually the first work on approximation analysis of regularized learning algorithms in RKKS.

학교시설 실내공기질 환경 개선을 위한 실내 미세먼지(PM2.5) 예측모델 개발

Purpose: In this study, an indoor particulate matter (PMSUB2.5/SUB) prediction model was developed to improve air quality in the classrooms. Employing the artificial neural network, the developed model is able to conduct iterative self-training in real-time and adapt itself to the various class environments. Method: A school building, which was used for data acquisition and performance evaluation of predictive model, was modeled by coupling 3 simulation programs to consider various factors that influence the formation of indoor PMSUB2.5/SUB concentration. The ANN prediction model was developed using the Bayseian Regularization learning algorithm following the performance optimization. The optimized prediction model was applied to different classroom in the same building for the adaptive performance evaluation. Result: As a result of the performance evaluation, Cv(RMSE) of the optimized prediction model was 5% and R2 was 0.8757, indicating high accuracy and stability. According to the real-time training, the error gradually decreased after occurrence. Therefore, it was demonstrated that the developed ANN prediction model is able to be adapted to various environmental conditions and expected to be applied in the optimal control algorithm through future research.

Estimation of Residential Property Market Price: Comparison of Artificial Neural Networks and Hedonic Pricing Model

The correct real estate property price estimation is significant not only in the real estate market but also in the banking sector for collateral loans and the insurance sector for property insurance. The paper focuses on both traditional and advanced methods for real estate property valuation. Attention is paid to the analysis of the accuracy of valuation models. From traditional methods, a regression model is used for residential property price estimation, which represents the hedonic approach. Modern advanced valuation methods are represented by the artificial neural network, which is one of the soft computing techniques. The results of both methods in residential property market price estimation are compared. The analysis is performed using data on residential properties sold on the real estate market in the city of Nitra in the Slovak Republic. To estimate the residential property prices, artificial neural networks trained with the Levenberg-Marquart learning algorithm, the Bayesian Regularization learning algorithm, and the Scaled Conjugate Gradient learning algorithm, and the regression pricing model are used. Among the constructed neural networks, the best results are achieved with networks trained with the Regularization learning algorithm with two hidden layers. Its performance is compared with the performance of the regression pricing model, and it can state that artificial neural networks can considerably improve prediction accuracy in the estimation of residential property market price. Doi: 10.28991/esj-2020-01250 Full Text: PDF

Estimating buildup factor of alloys based on combination of Monte Carlo method and multilayer feed-forward neural network

Up to now, different methods have been developed for estimation of buildup factor (BF). However, either expensive estimation or time-consuming estimation are major restrictions/challenges of these methods. In this study a new technique utilizing combination of Monte Carlo method and the Bayesian regularization (BR) learning algorithm of multilayer feed-forward neural network (FFNN) is employed for estimation of BFs. First, the BFs of the different elements (i.e. Al, Cu, and Fe) at different energies and different mean free paths (MFPs) are modeled by the MCNP code. The results show that the calculated BFs by MCNP code are in good agreement with the reported values of American nuclear society (ANS). Afterwards, the appropriate architecture of FFNN (i.e. the appropriate number of hidden neurons and hidden layers) and the appropriate input patterns features are investigated. The resulted FFNN is trained using the modeled BFs and the selected category of features. In the test process, the BFs of the master alloys (i.e. Fe-Al%50, Cu-Fe50%, Al-Cu50%) are estimated. To evaluate the performance of the proposed FFNN for training/estimation of the new elements/alloys, Si is added to the training process and the BFs of the Al-Si35% is estimated. Average mean relative error (AMRE) and cumulative distribution function (CDF) of the errors show the acceptable accuracy of the estimating the BFs of the alloys. The noticeable advantages of the proposed technique are: 1- The BFs of the different alloys are estimated only by using the BFs of the constituent elements of the alloys. 2- The time needed to estimate the new BFs by the proposed technique can be neglected versus the time needed to model the new BFs by Monte Carlo. 3- The proposed technique can generalize its ability for estimating the BFs of the new alloys. 4- Monte Carlo codes need the trained person to model the BFs of the alloys while the FFNN generates the new BFs easily.

Comparative Study of Response Surface Methodology and Artificial Neural Network for Modeling and Optimization of Extraction Process Parameters on <I>Tetrapleura Tetraptera</I>

Bioactive compounds in the fruits of Tetrapleura tetraptera is widely used in food as a flavouring agent and for spices. In this study, bioactive compounds were extracted by solid-liquid extraction process and the yield was optimized by response surface methodology (RSM) and artificial neural network (ANN). The process parameters optimized were the extraction temperature, particle size and extraction time. Box-Behnken Design was used to study the effect of the process parameters on the extract yield. A quadratic model was obtained by RSM which was used topredict the extract yield. While for ANN, Bayesian Regularization learning algorithm with hyperbolic function (Tanh) for both hidden and output layers was the best model for predicting the extract yield. The performance of both models was established based on their R2 and RMSE values. (R2 and RMSE values were 0.9391 and 3.10 for RSM and 0.9637 and 0.8193 for ANN respectively). ANN gave the maximum extract yield of 29.15 % higher than that of RSM which evaluated a yield of 27.70 % with optimum conditions at extraction temperature of 90℃, particle size of 3.26 mm and extraction time of 50 mins. It was therefore concluded that ANN is better than RSM in the modeling and optimization of the extraction process parameters.
 Keywords: Tetrapleura tetraptera, bioactive compounds, process parameters, optimization

Error analysis on regularized regression based on the Maximum correntropy criterion

This paper aims at the regularized learning algorithm for regression associated with the correntropy induced losses in reproducing kernel Hilbert spaces. The main target is the error analysis for the regression problem in learning theory based on the maximum correntropy. Explicit learning rates are provided. From our analysis, when choosing a suitable parameter of the loss function, we obtain satisfactory learning rates. The rates depend on the regularization error and the covering numbers of the reproducing kernel Hilbert space.

Calibration of [formula omitted]insensitive loss in support vector machines regression

Support vector machines regression (SVMR) is an important tool in many machine learning applications. In this paper, we focus on the theoretical understanding of SVMR based on the ϵ−insensitive loss. For fixed ϵ ≥ 0 and general data generating distributions, we show that the minimizer of the expected risk for ϵ−insensitive loss used in SVMR is a set-valued function called conditional ϵ−median. We then establish a calibration inequality of ϵ−insensitive loss under a noise condition on the conditional distributions. This inequality also ensures us to present a nontrivial variance-expectation bound for ϵ−insensitive loss, and which is known to be important in statistical analysis of the regularized learning algorithms. With the help of the calibration inequality and variance-expectation bound, we finally derive an explicit learning rate for SVMR in some Lr−space.

Improving educational web search for question-like queries through subject classification

Students use general web search engines as their primary source of research while trying to find answers to school-related questions. Although search engines are highly relevant for the general population, they may return results that are out of educational context. Another rising trend; social community question answering websites are the second choice for students who try to get answers from other peers online. We attempt discovering possible improvements in educational search by leveraging both of these information sources. For this purpose, we first implement a classifier for educational questions. This classifier is built by an ensemble method that employs several regular learning algorithms and retrieval based approaches that utilize external resources. We also build a query expander to facilitate classification. We further improve the classification using search engine results and obtain 83.5% accuracy. Although our work is entirely based on the Turkish language, the features could easily be mapped to other languages as well. In order to find out whether search engine ranking can be improved in the education domain using the classification model, we collect and label a set of query results retrieved from a general web search engine. We propose five ad-hoc methods to improve search ranking based on the idea that the query-document category relation is an indicator of relevance. We evaluate these methods for overall performance, varying query length and based on factoid and non-factoid queries. We show that some of the methods significantly improve the rankings in the education domain.

Balancing principle in supervised learning for a general regularization scheme

We discuss the problem of parameter choice in learning algorithms generated by a general regularization scheme. Such a scheme covers well-known algorithms as regularized least squares and gradient descent learning. It is known that in contrast to classical deterministic regularization methods, the performance of regularized learning algorithms is influenced not only by the smoothness of a target function, but also by the capacity of a space, where regularization is performed. In the infinite dimensional case the latter one is usually measured in terms of the effective dimension. In the context of supervised learning both the smoothness and effective dimension are intrinsically unknown a priori . Therefore we are interested in a posteriori regularization parameter choice, and we propose a new form of the balancing principle. An advantage of this strategy over the known rules such as cross-validation based adaptation is that it does not require any data splitting and allows the use of all available labeled data in the construction of regularized approximants. We provide the analysis of the proposed rule and demonstrate its advantage in simulations. • For smooth kernels, a power decay of effective dimensions may not be appropriate. • Learning rates under general smoothness assumptions on target functions. • Optimal learning rates in all involved norms based on a novel bound. • An adaptive parameter choice which uses the whole dataset without data splitting.

Error Analysis of Least-Squares $l^{^{q}}$ -Regularized Regression Learning Algorithm With the Non-Identical and Dependent Samples

The selection of the penalty functional is critical for the performance of a regularized learning algorithm, and thus $l^{q}$ -regularizer $(1\leq q\leq 2)$ deserves special attention. We consider the regularized least-squares regression learning algorithm for the non-identical and weakly dependent samples. The dependent samples satisfy the polynomially $\beta $ -mixing condition and the sequence of the non-identical sampling marginal measures converges to a probability measure exponentially in the dual of a Holder space. We conduct the rigorous unified error analysis and derive the satisfactory learning rates of the algorithm by the stepping stone technique in the error decomposition and the independent-blocks technique in the sample error estimates.