Non-iterative Training Research Articles

An interpretable ensemble structure with a non-iterative training algorithm to improve the predictive accuracy of healthcare data analysis

The modern development of healthcare is characterized by a set of large volumes of tabular data for monitoring and diagnosing the patient's condition. In addition, modern methods of data engineering allow the synthesizing of a large number of features from an image or signals, which are presented in tabular form. The possibility of high-precision and high-speed processing of such large volumes of medical data requires the use of artificial intelligence tools. A linear machine learning model cannot accurately analyze such data, and traditional bagging, boosting, or stacking ensembles typically require significant computing power and time to implement. In this paper, the authors proposed a method for the analysis of large sets of medical data, based on a designed linear ensemble method with a non-iterative learning algorithm. The basic node of the new ensemble is an extended-input SGTM neural-like structure, which provides high-speed data processing at each level of the ensemble. Increasing prediction accuracy is ensured by dividing the large dataset into parts, the analysis of which is carried out in each node of the ensemble structure and taking into account the output signal from the previous level of the ensemble as an additional attribute on the next one. Such a design of a new ensemble structure provides both a significant increase in the prediction accuracy for large sets of medical data analysis and a significant reduction in the duration of the training procedure. Experimental studies on a large medical dataset, as well as a comparison with existing machine learning methods, confirmed the high efficiency of using the developed ensemble structure when solving the prediction task.

NON-ITERATIVE CALCULATION OF PARAMETERS OF A LINEAR CLASSIFIER WITH A THRESHOLD ACTIVATION FUNCTION

The relevance of artificial intelligence (AI) systems is growing every year. AI is being introduced into various fields of activity. One of the main technologies used in AI is artificial neural networks (hereinafter referred to as NN). With the help of neural networks, a huge class of problems is solved, such as classification, regression, autoregression, clustering, noise reduction, creating a vector representation of objects, and others. In this work, we consider the simplest case of operation of one neuron with the Heaviside activation function, we also consider fast ways to train it, and we reduce the learning problem to the problem of finding the normal vector to the separating hyperplane and the displacement weight. One of the promising areas for training NN is non-iterative training, especially in the context of processing and analyzing high-dimensional data. This article discusses a method of non-iterative learning, which allows you to greatly (by 1-2 orders of magnitude) speed up the training of one neuron. The peculiarity of the approach is to determine the hyperplane separating two classes of objects in the feature space, without the need for repeated recalculation of weights, which is typical for traditional iterative methods. Within the framework of the study, special attention is paid to cases when the main axes of the ellipsoids describing the classes are parallel. The function pln is defined to calculate the distances between objects and the centers of their classes, based on which the non-normalized normal vector to the hyperplane and the displacement weight are calculated. In addition, we provide a comparison of our method with support vector machines and logistic regression.

AN UNSUPERVISED-SUPERVISED ENSEMBLE TECHNOLOGY WITH NON-ITERATIVE TRAINING ALGORITHM FOR SMALL BIOMEDICAL DATA ANALYSIS

Improving the accuracy of intelligent data analysis is an important task in various application areas. Existing machine learning methods do not always provide a sufficient level of classification accuracy for their use in practice. That is why, in recent years, hybrid ensemble methods of intellectual data analysis have begun to develop. They are based on the combined use of clustering and classification procedures. This approach provides an increase in the accuracy of the classifier based on machine learning due to the expansion of the space of the input data of the task by the results of the clustering.
 In this paper, the tasks of modification and improvement of such technology for small data analysis are considered. The basis of the modification is the use of clustering with output at the first step of the method to increase the accuracy of the entire technology. Despite the high accuracy of the work, this approach requires a significant expansion of the inputs of the final linear classifier (labels of the obtained clusters are added to the initial inputs). To avoid this shortcoming, the paper proposes an improvement based on the introduction of a new classification procedure at the first step of the method and replaces all the initial inputs of the task with the results of its work. In parallel with it, clustering is performed taking into account the original attribute, the results of which are added to the output of the classifier of the first step. In this way, the formation of an expanded set of data of significantly lower dimensionality in comparison with the existing method takes place (here there is no longer a large number of initial features, which is characteristic of biomedical engineering tasks). This reduces the training time of the method and increases its generalization properties.
 Modeling of the method was based on the use of a short dataset contained in an open repository. After the preprocessing procedures, the dataset has only 294 vectors, each of which was characterized by 18 attributes. Data classification was done using an SGTM-based neural-like structure classifier. This linear classifier provides high accuracy of work. In addition, it does not provide for the implementation of an iterative training procedure and additional adjustment of work parameters. Data clustering was performed using the k-means method. This choice is due to both the simplicity and speed of its work. 
 The search for the optimal number of k-means clusters was carried out using 4 different methods. They all showed different results. That is why, some experiments were conducted to assess the influence of different numbers of clusters (from 3 to 7) on the accuracy of all 4 algorithms of the developed technology. The accuracy of the proposed technology has been established experimentally in comparison with the linear classifier and the existing hybrid method. In addition, by reducing the inputs of the final classifier, the developed technology reduces the duration of the training procedure compared to the basic method. All this ensures the possibility of using the proposed technology when solving various applied problems of medical diagnostics, in particular, based on the analysis of small data.
 Keywords: small data approach, non-iterative training, ensemble learning, unsupervised-supervised technology, biomedical engineering.

Optimized deterministic multikernel extreme learning machine for classification of COVID-19 chest Xray images

In this paper, a novel technique has been proposed to exploit the capability of residual network (ResNet) deep learning model to extract the features. It is utilized neither in pretrained form nor as a transfer learning model. ResNet uses shortcut connections to create shortcut blocks in order to skip blocks of convolutional layers (residual blocks). These stacked residual blocks significantly increase training effectiveness and address the degradation issue. For the purpose of classification, a multiple kernel learning based deterministic extreme learning machine (MKD-ELM) which uses a linear combination of different base kernels as target kernel function is designed to classify chest Xray images. Multiple kernels are used here to exploit their non-linear mapping capability on heterogeneous data. MKD-ELM is an enhanced classifier, which does not require iterative training of its parameters. The proposed technique has better feature extraction along with non-iterative training, thus it is having very fast training and very good generalization performance. The kernel and regularization parameters that influence how accurate MKD-ELM is at classifying data, are tuned through experimentation. So, an optimization technique called the genetic algorithm (GA) has been utilized to determine the ideal combination of these parameters for improved performance. The performance of the proposed technique is analysed for COVID-19 detection problem using chest Xray (ChXR) images by changing the training set, types of kernels and coefficients used for combining base kernels. The proposed algorithm achieves a 97.27% recognition rate on first dataset which comprises 5,856 images and 99.06% on the second dataset which consists of 13,808 images. A higher recognition rate is attained for these ChXR image datasets, in respect to modern techniques demonstrating the effectiveness of the proposed algorithm.

Algebraic Zero Error Training Method for Neural Networks Achieving Least Upper Bounds on Neurons and Layers

In the domain of artificial neural networks, it is important to know what their representation, classification and generalization capabilities are. There is also a need for time and resource-efficient training algorithms. Here, a new zero-error training method is derived for digital computers and single hidden layer networks. This method is the least upper bound on the number of hidden neurons as well. The bound states that if there are N input vectors expressed as rational numbers, a network having N − 1 neurons in the hidden layer and M neurons at the output represents a bounded function F: RD→RM for all input vectors. Such a network has massively shared weights calculated by 1 + M regular systems of linear equations. Compared to similar approaches, this new method achieves a theoretical least upper bound, is fast, robust, adapted to floating-point data, and uses few free parameters. This is documented by theoretical analyses and comparative tests. In theory, this method provides a new constructional proof of the least upper bound on the number of hidden neurons, extends the classes of supported activation functions, and relaxes conditions for mapping functions. Practically, it is a non-iterative zero-error training algorithm providing a minimum number of neurons and layers.

Blockwise Recursive Moore–Penrose Inverse for Network Learning

Training neural networks with the Moore–Penrose (MP) inverse has recently gained attention in view of its noniterative training nature. However, a significant drawback of learning based on the MP inverse is that the computational memory consumption grows along with the size of a dataset. In this article, based on the partitioning of the MP inverse, we propose a blockwise recursive MP inverse formulation (BRMP) for network learning with low-memory property while preserving its training effectiveness. The BRMP is an equivalent formulation to its batchwise counterpart since neither approximation nor assumption is made in the derivation process. Our further exploration of this recursive method leads to a switching structure among three different scenarios. This structure also reveals that the well-known recursive least squares method is a special case of our proposed technique. Subsequently, we apply BRMP to the training of radial basis function networks as well as multilayer perceptrons. The experimental validation covers both regression and classification tasks.

Neurocontrol methods in the context of development of technical solutions for transition to unmanned navigation

The paper considers the features and prospects of using neurocontrol methods in the context of development of technical solutions for transition to unmanned merchant vessels. The paper suggests a non-iterative training based artificial neural network (ANN), which is based on the principles of “direct inverse control” to control the speed and motion of unmanned surface vessels. The model is identified, and the structure of an artificial neural network and the diagram of the automatic control system (ACS) of an unmanned vessel (UV) are considered on the example of an electric propulsion vessel. A series of computational experiments is carried out to obtain a sufficiently complete training sample. and the control law is presented. The principle of the control system for an unmanned vessel is considered based on a neural network. At the next stage of the study, focus is on the synthesis of the optimal control system for UV navigation. The problem of the fastest motion of a third-order control object from one point (with any initial speed) to another (at the end point the vessel stops and the speed is zero) is considered. Based on the results of a series of experiments with the UV model, the controller parameters that provide the best indicators of control quality were set in the MATLAB Simulink environment.

A cepstrum analysis-based classification method for hand movement surface EMG signals.

It is of great importance to effectively process and interpret surface electromyogram (sEMG) signals to actuate a robotic and prosthetic exoskeleton hand needed by hand amputees. In this paper, we have proposed a cepstrum analysis-based method for classification of basic hand movement sEMG signals. Cepstral analysis technique primarily used for analyzing acoustic and seismological signals is effectively exploited to extract features of time-domain sEMG signals by computing mel-frequency cepstral coefficients (MFCCs). The extracted feature vector consisting of MFCCs is then forwarded to feed a generalized regression neural network (GRNN) so as to classify basic hand movements. The proposed method has been tested on sEMG for Basic Hand movements Data Set and achieved an average accuracy rate of 99.34% for the five individual subjects and an overall mean accuracy rate of 99.23% for the collective (mixed) dataset. The experimental results demonstrate that the proposed method surpasses most of the previous studies in point of classification accuracy. Discrimination ability of the cepstral features exploited in this study is quantified using Kruskal-Wallis statistical test. Evidenced by the experimental results, this study explores and establishes applicability and efficacy of cepstrum-based features in classifying sEMG signals of hand movements. Owing to the non-iterative training nature of the artificial neural network type adopted in the study, the proposed method does not demand much time to build up the model in the training phase. Graphical abstract.

Development of the Non-Iterative Supervised Learning Predictor Based on the Ito Decomposition and SGTM Neural-Like Structure for Managing Medical Insurance Costs

The paper describes a new non-iterative linear supervised learning predictor. It is based on the use of Ito decomposition and the neural-like structure of the successive geometric transformations model (SGTM). Ito decomposition (Kolmogorov–Gabor polynomial) is used to extend the inputs of the SGTM neural-like structure. This provides high approximation properties for solving various tasks. The search for the coefficients of this polynomial is carried out using the fast, non-iterative training algorithm of the SGTM linear neural-like structure. The developed method provides high speed and increased generalization properties. The simulation of the developed method’s work for solving the medical insurance costs prediction task showed a significant increase in accuracy compared with existing methods (common SGTM neural-like structure, multilayer perceptron, Support Vector Machine, adaptive boosting, linear regression). Given the above, the developed method can be used to process large amounts of data from a variety of industries (medicine, materials science, economics, etc.) to improve the accuracy and speed of their processing.

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

Recurrent neural networks (RNN) are a type of artificial neural networks (ANN) that have been successfully applied to many problems in artificial intelligence. However, they are expensive to train since the number of learned weights grows exponentially with the number of hidden neurons. Non-iterative training algorithms have been proposed to reduce the training time, mainly on feedforward ANN. In this work, the application of non-iterative randomized training algorithms to various RNN architectures, including Elman RNN, fully connected RNN, and long short-term memory (LSTM), are investigated. The mathematical formulation and theoretical computational complexity of the proposed algorithms are presented. Finally, their performance is empirically compared to other iterative RNN training algorithms on time series prediction and sequential decision-making problems. Non-iteratively-trained RNN architectures showed promising results as significant training speedup of up to 99%, and improved repeatability were achieved compared to backpropagation-trained RNN. Although the decrease in prediction accuracy was found to be statistically significant based on Friedman and ANOVA testing, some applications like real-time embedded systems can tolerate and make use of that.

Sensitivity of data matrix rank in non-iterative training

Abstract This paper focuses on the parameter pattern during the initialization of Extreme Learning Machines (ELMs). According to the algorithm, model performance is highly dependent on the matrix rank of its hidden layer. Previous research has already proved that the sigmoid activation function can transform input data to a full rank hidden matrix with probability 1, which secures the stability of ELM solution. In recent study, we notice that, under full-rank condition, the hidden matrix possibly has very small eigenvalue, which seriously affects the model generalization ability. Our study indicates such a negative impact is caused by the discontinuity of generalized inverse at the boundary of full and waning rank. Experiments show that each phase of ELM modeling possibly leads to this rank deficient phenomenon, which harms the test accuracy.

A simple feedforward convolutional conceptor neural network for classification

Training deep neural networks usually invokes error gradient backpropagation (BP), which costs a long training time even on high-performance computing devices or with professional skills of experienced experimenters. In order to accelerate and simplify the training process of deep networks, non-iterative training procedures have been proposed. Here we present another non-iterative approach, Feedforward Convolutional Conceptor Neural Network (FCCNN), for training feedforward networks on image classification tasks. Our work makes two major contributions: (1) a conceptor based classifier which is specific for non-temporal data; (2) a simple non-iterative neural network model. The architecture is established based on the combination of a Convolutional Neural Network (CNN) shaped by an unsupervised feature learning based on Principal Component Analysis (PCA), binary thresholding and the proposed non-temporal conceptor classifier. Through various experiments on MNIST variation datasets, FCCNN achieves classifying accuracy comparable to the state-of-the-art methods while requires significantly reduced training time.

An initial study on the rank of input matrix for extreme learning machine

It is noted that the rank of input data matrix has a critical impact on the performance of a trained classifier model. This paper presents a study on the rank of input data matrix based on a classification model of extreme learning machine which is a single hidden layer feed-forward neural network with non-iterative training. The changing tendency of model accuracy with the increase of input data matrix rank is experimentally investigated and the relationship between the input matrix rank and classification problem complexity is addressed. The analysis and experiments show some meaningful results.

Support Tool for the Combined Software/Hardware Design of On-Chip ELM Training for SLFF Neural Networks

Typically, hardware implemented neural networks are trained before implementation. Extreme learning machine (ELM) is a noniterative training method for single-layer feed-forward (SLFF) neural networks well suited for hardware implementation. It provides fixed-time learning and simplifies retraining of a neural network once implemented, which is very important in applications demanding on-chip training. This study proposes the data flow of a software support tool in the design process of a hardware implementation of on-chip ELM learning for SLFF neural networks. The software tool allows the user to obtain the optimal definition of functional and hardware parameters for any application, and enables the user to interact throughout the design process. Combining in a transparent way for the user, simulation and Xilinx synthesis tools, the tool recommends the optimal configuration, generating, finally, a synthesizable IP-core. As application, the field-programmable gate array implementation for real-time detection of brain areas in electrode positioning during a deep brain stimulation surgery is described. The generated IP-core can execute a peak of 95 ELM trainings per second on a low-cost Spartan 6 device, making possible its real-time use in this application.

Hardware implementation of real-time Extreme Learning Machine in FPGA: Analysis of precision, resource occupation and performance

Extreme Learning Machine (ELM) proposes a non-iterative training method for Single Layer Feedforward Neural Networks that provides an effective solution for classification and prediction problems. Its hardware implementation is an important step towards fast, accurate and reconfigurable embedded systems based on neural networks, allowing to extend the range of applications where neural networks can be used, especially where frequent and fast training, or even real-time training, is required. This work proposes three hardware architectures for on-chip ELM training computation and implementation, a sequential and two parallel. All three are implemented parameterizably on FPGA as an IP (Intellectual Property) core. Results describe performance, accuracy, resources and power consumption. The analysis is conducted parametrically varying the number of hidden neurons, number of training patterns and internal bit-length, providing a guideline on required resources and level of performance that an FPGA based ELM training can provide.

Computer aided analysis of breast MRI enhancement kinetics using mean shift c lustering and multifeature iterative region of interest selection

To evaluate automatic characterization of a breast MR lesion by its spatially coherent region of interest (ROI). The method delineated 247 enhancing lesions using Otsu thresholding after manually placing a sphere. Mean Shift Clustering subdivided each volume, based on features including pharmacokinetic parameters. An iteratively trained classifier to predict the most suspicious ROI (IsR) was used, to predict the malignancy likelihood of each lesion. Performance was evaluated using receiver operator characteristic (ROC) analysis, and compared with a previous prototype. IsR was compared with noniterative training. The effect of adding BI-RADS™ morphology (from a radiologist) to the classifier was investigated. The area under the ROC curve (AUC) was 0.83 (95% confidence interval [CI] of 0.77-0.88), and was 0.75 (95%CI = 0.68-0.81; P = 0.029) without pharmacokinetic features. IsR performed better than conventional selection, based on one feature (AUC 0.75, 95%CI = 0.68-0.81; P = 0.035). With morphology, the AUC was 0.84 (95%CI = 0.78-0.88) versus 0.82 without (P = 0.40). Breast lesions can be characterized by their most suspicious, contiguous ROI using multi-feature clustering and iterative training. Characterization was improved by including pharmacokinetic modeling, while in our experiments, including morphology did not improve characterization.

Comparative Evaluation of Two Neural Network Based Techniques for the Classification of Microcalcifications in Digital Mammograms

This paper investigates two neural network based techniques for the classification of microcalcifications in digital mammograms. Both techniques extract suspicious areas containing microcalcifications from digital mammograms and classify them into two categories: whether they contain benign or malignant clusters. The centroids and radii provided by expert radiologists are being used to locate and extract suspicious areas. Two neural networks based on iterative and non-iterative training methods are used to classify them into benign or malignant. The proposed techniques have been implemented in C++ on the SP2 supercomputer. The database from the Department of Radiology at the University of Nijmegen has been used for the experiments. The comparative results are very interesting and promising. Some of them are included in this paper.

Strategy for dynamic process modeling based on neural networks in macroscopic balances

AbstractGray box models combine the short development time of data‐driven black box models with extrapolation properties of knowledge‐driven first principles models (white box), which in (bio)chemical engineering are always based on macroscopic balances. By modeling the inaccurately known terms in a macroscopic balance with a black box model, one naturally obtains a so‐called serial gray box model configuration. The identification data must cover only the input‐output space of the inaccurately known terms, and the accurately known terms can be used for reliable extrapolation. In this way, the serial gray box configuration results in accurate models with known extrapolation properties with a limited experimental effort. This strategy is demonstrated for the modeling and control of a pressure vessel using real‐time experiments. For this case, the strategy is superior to a black box modeling approach that requires much more data and to a parallel gray box approach that results in a model with poor extrapolation properties. Moreover, neural networks are an accurate and convenient modeling tool for the black part in gray box model configurations, because a very fast noniterative training algorithm is used for training neural networks.

Random activation weight neural net (RAWN) for fast non-iterative training

General function approximation can be obtained by feed-forward neural nets consisting of just one hidden layer of non-linear neurons. The innovations described in this paper is that training of the weights between the input and hidden layers is not required. By taking these so-called “activation weights” as random, the resulting problem is linear in the parameters and can easily be solved by standard ordinary least-squares methods. This paper demonstrates that by using such “random activation weight nets” (RAWNs), excellent mappings can be obtained, provided that the activation weights are chosen so that the regression matrix is non-singular. Further improvements can be achieved by regularization, and by the proper choice of the excitation signal used for training. It is found that (i) much smaller errors are obtained as compared to backpropagation nets with the same degrees of freedom, (ii) the mapping depends only slightly upon the actual values of the hidden weights, provided the net has sufficient neurons, and (iii) since no iteration is needed, the speed of computation is incomparably much faster than with backpropagation. These properties make the RAW-Net particularly suitable for control applications. Various examples, both static and dynamic, are given to show the feasibility and advantages of the approach.