Fast Training Process Research Articles

Application of Artificial Neural Networks in Predicting the Thermal Performance of Heat Pipes

The loss of energy by heat is a common problem in almost all areas of industry, and heat pipes are essential to increase efficiency and reduce energy waste. However, in many cases, they have complex theoretical equations with high percentages of error, limiting their development and causing dependence on empirical methods that generate a waste of time and material, resulting in significant expenses and reducing the viability of their use. Thus, Artificial Neural Networks (ANNs) can be an excellent option to facilitate the construction and development of heat pipes without knowledge of the complex theory behind the problem. This investigation uses experimental data from previous studies to evaluate the ability of three different ANNs to predict the thermal performance of heat pipes with different capillary structures, each of them in various configurations of the slope, filling ratio, and heat load. The goal is to investigate results in as many different scenarios as possible to clearly understand the networks’ capacity for modeling heat pipes and their operating parameters. We chose two classic ANNs (the most used, Multilayer Perceptron (MLP) network, and the Radial Basis Function (RBF) network) and the Extreme Learning Machine (ELM), which has not yet been applied to heat pipes studies. The ELM is an Unorganized Machine with a fast training process and a simple codification. The ANN results were very close to the experimental ones, showing that ANNs can successfully simulate the thermal performance of heat pipes. Based on the RMSE (error metric being reduced during the training step), the ELM presented the best results (RMSE = 0.384), followed by MLP (RMSE = 0.409), proving their capacity to generalize the problem. These results show the importance of applying different ANNs to evaluate the system deeply. Using ANNs in developing heat pipes is an excellent option for accelerating and improving the project phase, reducing material loss, time, and other resources.

Efficient EEG Feature Learning Model Combining Random Convolutional Kernel with Wavelet Scattering for Seizure Detection.

Automatic seizure detection has significant value in epilepsy diagnosis and treatment. Although a variety of deep learning models have been proposed to automatically learn electroencephalography (EEG) features for seizure detection, the generalization performance and computational burden of such deep models remain the bottleneck of practical application. In this study, a novel lightweight model based on random convolutional kernel transform (ROCKET) is developed for EEG feature learning for seizure detection. Specifically, random convolutional kernels are embedded into the structure of a wavelet scattering network instead of original wavelet transform convolutions. Then the significant EEG features are selected from the scattering coefficients and convolutional outputs by analysis of variance (ANOVA) and minimum redundancy-maximum relevance (MRMR) methods. This model not only preserves the merits of the fast-training process from ROCKET, but also provides insight into seizure detection by retaining only the helpful channels. The extreme gradient boosting (XGboost) classifier was combined with this EEG feature learning model to build a comprehensive seizure detection system that achieved promising epoch-based results, with over 90% of both sensitivity and specificity on the scalp and intracranial EEG databases. The experimental comparisons showed that the proposed method outperformed other state-of-the-art methods for cross-patient and patient-specific seizure detection.

A short time series rolling bearing fault diagnosis method based on FMTF-CNN

The signal characteristics are extracted directly from the convolutional level when the Convolutional Neural Network (CNN) is used as a fault diagnosis method in most instances. Some scholars have proposed to use the Markov Transition Field (MTF) to extract the signal characteristics before convolution. However, for time-domain signals, the fault characteristics extraction is difficult, and the training process of the neural network parameters is slow. Therefore, in this paper, based on Fast Fourier Transform (FFT) and MTF algorithm, the Fourier Markov Transition Field (FMTF) characteristics extraction method in frequency domain is proposed, and a large number of experiments are carried on the public bearing data set of Jiangnan University. The final verification shows that the FMTF-CNN method has higher accuracy, faster training process and more accurate characteristics extraction than the traditional MTF-CNN diagnosis method.

A deep learning approach based on Richardson and Gauss–Seidel for massive MIMO detection

AbstractMassive multiple‐input multiple‐output (MIMO) systems can improve the spectrum utilization and the system capacity, but this also increases the computational complexity of the signal detection. The existing iterative algorithms can greatly reduce the computational complexity; however, the detection performance is limited. In order to achieve a better balance between the computational complexity and the detection performance, this article combines the model‐driven deep learning approached with Massive MIMO signal detection to construct RGNet (RIGS‐based deep learning Network). First, RIGS is proposed as a hybrid method of RI (Richardson) and GS (Gauss–Seidel). The RIGS algorithm combines these methods to achieve faster convergence. However, the performance of RIGS joint algorithm is limited to the spatially correlated channel scenarios. To improve robustness, we further extend RIGS, by adding learnable parameters in each iteration and introducing staircase activation functions to significantly improve detection performance. Simulation results show that the proposed RGNet has low computational complexity and a simple and fast training process. It can also achieve excellent detection performance in Rayleigh fading channel and spatially correlated channel.

Reinforcement Learning-Based Joint User Scheduling and Link Configuration in Millimeter-Wave Networks

In this paper, we develop algorithms for joint user scheduling and three types of mmWave link configuration: relay selection, codebook optimization, and beam tracking in millimeter wave (mmWave) networks. Our goal is to design an online controller that dynamically schedules users and configures their links to minimize system delay. To solve this complex scheduling problem, we model it as a dynamic decision-making process and develop two reinforcement learning-based solutions. The first solution is based on deep reinforcement learning (DRL), which leverages the proximal policy optimization to train a neural network-based solution. Due to the potential high sample complexity of DRL, we also propose an empirical multi-armed bandit (MAB)-based solution, which decomposes the decision-making process into a sequential of sub-actions and exploits classic maxweight scheduling and Thompson sampling to decide those sub-actions. Our evaluation of the proposed solutions confirms their effectiveness in providing acceptable system delay. It also shows that the DRL-based solution has better delay performance while the MAB-based solution has a faster training process.

A framework to build accurate Convolutional Neural Network models for melanoma diagnosis

In the past few years, Convolutional Neural Networks have achieved performance levels similar to those achieved by dermatologists. However, the diagnosis of melanoma remains a challenging task, mainly due to the high levels of inter and intra-class variability present in images of moles. With the aim of new methods for an effective melanoma diagnosis, a new framework is proposed. The training process is guided by an expert within an active learning approach where the architectures implicitly learn about the complexity of individual images through query strategies, which allows us to adjust the training process and achieve better performance. In addition, we propose a batch-based query strategy that enables a more stable and faster training process. Besides, the framework leverages segmentation, data augmentation and transfer learning to enhance melanoma diagnosis. The framework is composed by several specialized blocks, which allow us to measure how the diagnosis is improved after each step. In this sense, blocks could be customized and do not depend on specific models. An extensive experimental study was conducted on 16 skin image datasets, where five state-of-the-art models were significantly outperformed. This study corroborated that new active learning query strategies can be employed to effectively train neural networks architectures for the diagnosis of melanoma, achieving 182% better predictive performance in Xception, and an overall 11% and 20% better predictive performance in dermoscopic and non-dermoscopic images, respectively. It is worth mentioning that the informativeness value of each image is shown, which leads to identify the hardest images for the predictive models. Finally, the proposal required 2% of the total training time, and needed 61% less training epochs.

Flooding and Overflow Mitigation Using Deep Reinforcement Learning Based on Koopman Operator of Urban Drainage Systems

AbstractIn recent studies, deep reinforcement learning (RL) methods have been used for the real‐time control of urban drainage systems (UDSs). However, the training process of an RL agent is computationally expensive since collecting the training data requires multiple simulations of a UDS model. An effective solution to this issue is to replace the original UDS model with an emulator, a simpler model which emulates the specified output of the system. However, the adaptability and interpretability of the existing UDS emulators still require further improvements. To address these challenges, a new emulator with increased adaptability and interpretability, called Koopman emulator, was developed by using the Koopman operator and a deep learning technique in this study. Then, the Koopman emulator was used to replace the UDS model in RL to establish an emulator‐based RL framework for fast and efficient training. This framework was used to train emulator‐based RL agents for flooding and overflow mitigation in a UDS located in Eastern China. Baseline emulators were also employed for comparison. All the methods were evaluated through simulations. The following results were obtained: (a) The Koopman emulator demonstrated a better emulation performance than the baseline emulators. (b) The RL agents based on the Koopman emulator achieved a similar control effect with a faster training process and more efficient data usage compared with the RL agents based on the storm water management model. (c) The uncertainty analysis showed that the RL agents based on the Koopman emulator exhibited acceptable robustness when facing different rainfall events and input perturbations.

A robust deep attention dense convolutional neural network for plant leaf disease identification and classification from smart phone captured real world images

Plant diseases cause significant food loss and hence economic loss around the globe. Therefore, automatic plant disease identification is a primary task to take proper medications for controlling the spread of the diseases. Large variety of plants species and their dissimilar phytopathological symptoms call for the implementation of supervised machine learning techniques for efficient and reliable disease identification and classification. With the development of deep learning strategies, convolutional neural network (CNN) has paved its way for classification of multiple plant diseases by extracting rich features. However, several characteristics of the input images especially captured in real world environment, viz. complex or indistinguishable background, presence of multiple leaves with the diseased leaf, small lesion area, solemnly affect the robustness and accuracy of the CNN modules. Available strategies usually applied standard CNN architectures on the images captured in the laboratory environment and very few have considered practical in-field leaf images for their studies. However, those studies are limited with very limited number of plant species. Therefore, there is need of a robust CNN module which can successfully recognize and classify the dissimilar leaf health conditions of non-identical plants from the in-field RGB images. To achieve the above goal, an attention dense learning (ADL) mechanism is proposed in this article by merging mixed sigmoid attention learning with the basic dense learning process of deep CNN. The basic dense learning process derives new features at higher layer considering all lower layer features and that provides fast and efficient training process. Further, the attention learning process amplifies the learning ability of the dense block by discriminating the meaningful lesion portions of the images from the background areas. Other than adding an extra layer for attention learning, in the proposed ADL block the output features from higher layer dense learning are used as an attention mask to the lower layers. For an effective and fast classification process, five ADL blocks are stacked to build a new CNN architecture named DADCNN-5 for obtaining classification robustness and higher testing accuracy. Initially, the proposed DADCNN-5 module is applied on publicly available extended PlantVillage dataset to classify 38 different health conditions of 14 plant species from 54,305 images. Classification accuracy of 99.93% proves that the proposed CNN module can be used for successful leaf disease identification. Further, the efficacy of the DADCNN-5 model is checked after performing stringent experiments on a new real world plant leaf database, created by the authors. The new leaf database contains 10,851 real-world RGB leaf images of 17 plant species for classifying their 44 distinguished health conditions. Experimental outcomes reveal that the proposed DADCNN-5 outperforms the existing machine learning and standard CNN architectures, and achieved 97.33% accuracy. The obtained sensitivity, specificity and false positive rate values are 96.57%, 99.94% and 0.063% respectively. The module takes approximately 3235 min for training process and achieves 99.86% of training accuracy. Visualization of Class activation mapping (CAM) depicts that DADCNN-5 is able to learn distinguishable features from semantically important regions (i.e. lesion regions) on the leaves. Further, the robustness of the DADCNN-5 is established after experimenting with augmented and noise contaminated images of the practical database.

Simple phasor-based deep neural network for fluorescence lifetime imaging microscopy

Fluorescence lifetime imaging microscopy (FLIM) is a powerful technique to probe the molecular environment of fluorophores. The analysis of FLIM images is usually performed with time consuming fitting methods. For accelerating this analysis, sophisticated deep learning architectures based on convolutional neural networks have been developed for restrained lifetime ranges but they require long training time. In this work, we present a simple neural network formed only with fully connected layers able to analyze fluorescence lifetime images. It is based on the reduction of high dimensional fluorescence intensity temporal decays into four parameters which are the phasor coordinates, the mean and amplitude-weighted lifetimes. This network called Phasor-Net has been applied for a time domain FLIM system excited with an 80 MHz laser repetition frequency, with negligible jitter and afterpulsing. Due to the restricted time interval of 12.5 ns, the training range of the lifetimes was limited between 0.2 and 3.0 ns; and the total photon number was lower than 106, as encountered in live cell imaging. From simulated biexponential decays, we demonstrate that Phasor-Net is more precise and less biased than standard fitting methods. We demonstrate also that this simple architecture gives almost comparable performance than those obtained from more sophisticated networks but with a faster training process (15 min instead of 30 min). We finally apply successfully our method to determine biexponential decays parameters for FLIM experiments in living cells expressing EGFP linked to mCherry and fused to a plasma membrane protein.

Surgical Tool Segmentation Using Generative Adversarial Networks With Unpaired Training Data

Surgical tool segmentation is a challenging and crucial task for computer and robot-assisted surgery. Supervised learning approaches have shown great success for this task. However, they need a large number of paired training data. Based on Generative Adversarial Networks (GAN), unpaired image-to-image translation (I2I) techniques (like CycleGAN and dualGAN) have been proposed to avoid the requirement of paired data and have been employed for surgical tool segmentation. The unpaired I2I methods avoid annotating images for domain changes. Instead of using them directly for the segmentation task, we propose new GAN-based methods for unpaired I2I by embedding a specific constraint for segmentation, namely each pixel of input image belongs to either background or surgical tool. Our methods simplify the architectures of existing unpaired I2I with a reduced number of generators and discriminators. Compared with dualGAN, the proposed strategies have a faster training process without reducing the accuracy of the segmentation. Besides, we show that, using textured tool images as annotated samples to train discriminators, unpaired I2I (including our methods) can achieve simultaneous tool image segmentation and repair (such as reflection removal and tool inpainting). The proposed strategies are validated for image segmentation of a flexible tool and for in vivo images from the EndoVis dataset.

XSRU-IoMT: Explainable simple recurrent units for threat detection in Internet of Medical Things networks

The Internet of Medical Things (IoMT) is increasingly replacing the traditional healthcare systems. However, less focus has been paid to their security against cyber-threats in the implementation of the IoMT and its networks. One of the key reasons can be the challenging task of optimizing typical security solutions to the IoMT networks. And despite the rising admiration of machine learning and deep learning methods in the cyber-security domain (e.g., a threat detection system), most of these methods are acknowledged as a black-box model. The explainable AI (XAI) has become progressively vital to understand the employed learning models to improve trust level and empower security experts to interpret the prediction decisions. The authors propose a highly efficient model named XSRU-IoMT, for effective and timely detection of sophisticated attack vectors in IoMT networks. The proposed model is developed using novel bidirectional simple recurrent units (SRU) using the phenomenon of skip connections to eradicate the vanishing gradient problem and achieve a fast training process in recurrent networks. We also explore the concepts of XAI to improve trust level by providing explanations of the predictive decisions and enabling humans and security experts to understand the causal reasoning and underlying data evidence. The evaluation results on the ToN_IoT dataset demonstrate the effectiveness and superiority of the proposed XSRU-IoMT model as compared to the state-of-the-art compelling detection models, suggesting its usefulness as a viable deployment model in real-IoMT networks.

Model Fourier Untuk Prediksi Harga Saham Astrazeneca Menggunakan Algoritma Levenberg-Marquardt

The soaring cases of covid-19 prompted some countries to find solutions to save their people. One of the steps that is currently being taken is with vaccines. Several leading companies in the world that produce drugs are known to have produced vaccines for covid-19, one of which is AstraZeneca. AstraZeneca vaccine is known as the most widely used vaccine in all countries in the world. Interesting thing to research is how the development of the company's stock engaged in the medical field, especially companies that produce vaccines for covid-19. This study used Fourier's approach to modeling its curve fittings. As for the prediction process using levenberg-marquardt algorithm which is known to be reliable to perform the prediction process. Levenberg-Marquardt's algorithm has the advantage of a fast training process and reliable accuracy due to its work that combines Gauss-Newton and Steepest Descent. The result is root mean square error value from the test result that was lower than the Root Mean Square Error value in the training process. This indicates that the prediction went well.

Determining the Influencing Factors of COVID 19 on Mental Health Using Neural Network

In 2020 there is outbreak of the pandemic COVID 19. Due to the pandemic most of the countries enforced full lockdown. During lockdown the mental health of the common people of all ages are affected. The study aims to investigate the factors which influence the mental health. Mental health is influenced by the various factors. In this research the artificial neural network is used to predict the factors influencing the mental health due to COVID 19 pandemic. Experimental result showed that artificial neural network not only has better accuracy up to 81.5% but also faster training process. Based on these results, the determinant factors of mental health among people are: frustration, fear, sudden change in daily life style, anxiety to perceive changes as a challenge.

Study of Applications of Radial Basis Function Network in Forecasting

Forecasting is a broad research domain for multidisciplinary researchers. Forecasting climatic conditions is a challenging task that is carried out by scientists around the globe. The researchers for Forecasting of the various climatic phenomena have used multiple architectures of Artificial Neural Networks (ANN). Such tasks demand huge analysis on recent and past results to give appropriate with precise results of the forecast. The RBF network has a single hidden layer, simpler structure as well as a much faster training process; therefore, it is a popular alternative to the other ANN architectures. In this paper, we are proposing a Radial Basis Function Network (RBFN) as a machine learning tool for making forecasting. In a variety of real-time packages, that includes the prediction of weather, load forecasting, forecasting approximately a variety of traveller and in various programs RBFN applied.

Cooperative Multi-Agent Interaction and Evaluation Framework Considering Competitive Networks with Dynamic Topology Changes

In recent years, the problem of reinforcement learning has become increasingly complex, and the computational demands with respect to such processes have increased. Accordingly, various methods for effective learning have been proposed. With the help of humans, the learning object can learn more accurately and quickly to maximize the reward. However, the rewards calculated by the system and via human intervention (that make up the learning environment) differ and must be used accordingly. In this paper, we propose a framework for learning the problems of competitive network topologies, wherein the environment dynamically changes agent, by computing the rewards via the system and via human evaluation. The proposed method is adaptively updated with the rewards calculated via human evaluation, making it more stable and reducing the penalty incurred while learning. It also ensures learning accuracy, including rewards generated from complex network topology consisting of multiple agents. The proposed framework contributes to fast training process using multi-agent cooperation. By implementing these methods as software programs, this study performs numerical analysis to demonstrate the effectiveness of the adaptive evaluation framework applied to the competitive network problem depicting the dynamic environmental topology changes proposed herein. As per the numerical experiments, the greater is the human intervention, the better is the learning performance with the proposed framework.

Sparse Deep Neural Network Optimization for Embedded Intelligence

Deep neural networks become more popular as its ability to solve very complex pattern recognition problems. However, deep neural networks often need massive computational and memory resources, which is main reason resulting them to be difficult efficiently and entirely running on embedded platforms. This work addresses this problem by saving the computational and memory requirements of deep neural networks by proposing a variance reduced (VR)-based optimization with regularization techniques to compress the requirements of memory of models within fast training process. It is shown theoretically and experimentally that sparsity-inducing regularization can be effectively worked with the VR-based optimization whereby in the optimizer the behaviors of the stochastic element is controlled by a hyper-parameter to solve non-convex problems.

Incremental and decremental fuzzy bounded twin support vector machine

In this paper, we present an incremental variant of the Twin Support Vector Machine (TWSVM) called Fuzzy Bounded Twin Support Vector Machine (FBTWSVM) to deal with large datasets and to learn from data streams. We combine the TWSVM with a fuzzy membership function, so that each input has a different contribution to each hyperplane in a binary classifier. To solve the pair of quadratic programming problems (QPPs), we use a dual coordinate descent algorithm with a shrinking strategy, and to obtain a robust classification with a fast training we propose the use of a Fourier Gaussian approximation function with our linear FBTWSVM. Inspired by the shrinking technique, the incremental algorithm re-utilizes part of the training method with some heuristics, while the decremental procedure is based on a scoring window. The FBTWSVM is also extended for multi-class problems by combining binary classifiers using a Directed Acyclic Graph (DAG) approach. Moreover, we analyzed the theoretical foundation’s properties of the proposed approach and its extension, and the experimental results on benchmark datasets indicate that the FBTWSVM has a fast training and retraining process while maintaining a robust classification performance.

Nonlinear process modelling using echo state networks optimised by covariance matrix adaption evolutionary strategy

Echo state networks (ESN) have been shown to be an effective alternative to conventional recurrent neural networks (RNNs) due to its fast training process and good performance in dynamic system modelling. However, the performance of ESN can be affected by the randomly generated reservoir. This paper presents nonlinear process modelling using ESN optimized by covariance matrix adaption evolutionary strategy (CMA-ES). CMA-ES is used to optimize the structural parameters of ESN: reservoir size, spectral radius, and leak rate. The proposed method is applied to three case studies: modelling a time series, modelling a conic tank, and modelling a fed-batch penicillin fermentation process. The results are compared with those from the original ESN, long short-term memory network, GA-ESN (ESN optimized by genetic algorithm), and feedforward neural networks. It is shown that the proposed method gives much better performance than the original ESN and other networks on all the three case studies.

Intrusion Detection System for the MIL-STD-1553 Communication Bus

MIL-STD-1553 is a military standard that defines the specification of a serial communication bus that has been implemented in military and aerospace avionic platforms for over 40 years. MIL-STD-1553 was designed for a high level of fault tolerance while less attention was paid to cyber security issues. Thus, as indicated in recent studies, it is exposed to various threats. In this article, we suggest enhancing the security of MIL-STD-1553 communication buses by integrating a machine learning-based intrusion detection system (IDS); such anIDS will be capable of detecting cyber attacks in real time. The IDS consists of two modules: 1) a remote terminal (RT) authentication module that detects illegitimately connected components and data transfers and 2) a sequence-based anomaly detection module that detects anomalies in the operation of the system. The IDS showed high detection rates for both normal and abnormal behavior when evaluated in a testbed using real 1553 hardware, as well as a very fast and accurate training process using logs from a real system. The RT authentication module managed to authenticate RTs with +0.99 precision and +0.98 recall; and detect illegitimate component (or a legitimate component that impersonates other components) with +0.98 precision and +0.99 recall. The sequence-based anomaly detection module managed to perfectly detect both normal and abnormal behavior. Moreover, the sequencebased anomaly detection module managed to accurately (i.e., zero false positives) model the normal behavior of a real system in a short period of time (~22 s).

A Novel Unsupervised Learning Method Based on Cross-Normalization for Machinery Fault Diagnosis

Sparse representation is the important principle of unsupervised learning method. In order to accurately identify the fault condition of machines, the desired feature distribution should show population sparsity and lifetime sparsity. In this paper, to improve the accuracy and robustness of the classification, a novel fault diagnosis method named Cross-sparse Filtering (Cr-SF) is proposed based on the cross $l_{1/2}$ -norms of the feature matrix, which mean the population sparsity and lifetime sparsity terms. After the weights training process, a novel nonlinear activation function is used for feature extraction in the test process. Cr-SF can learn discriminative features from the raw data and accurately identify the fault condition. Rolling bearing fault and gear-box fault datasets are employed to validate the performance of the proposed method. The verification results confirm that Cr-SF is an effective tool for handling big data. The robustness and accuracy of the classification results using Cr-SF are comparable to convolutional networks with a much faster training process.