Neural Network Configuration Research Articles

Automated Configuration of Heterogeneous Graph Neural Networks With a Semantic Math Parser for IoT Systems

Efficient training of deep learning models from time series data for Internet of Things (IoT) systems requires a good understanding of the domain, particularly if the training is automated for large-scale applications. Heterogeneous graph neural networks (HGNNs) are a promising approach for incorporating domain knowledge into the modeling framework and consequently improving model performance. However, encoding domain knowledge into HGNNs is nontrivial for IoT systems and requires substantial manual effort. This complicates the adoption of HGNNs in practical settings. To overcome this drawback, we propose a framework for the automatic derivation of HGNN features by semantically parsing equations present in scientific and dedicated publications. We encode the derived features considering physical causation from these equations into an HGNN using an underlying Transformer for prediction and anomaly detection. We validate our approach using two IoT use cases, namely, the prediction of the remaining energy in the battery of an electric race car and the anomaly detection during pick and place operations in a robot workcell. We demonstrate that our approach significantly outperforms other competitive techniques.

Особенности построения нейросетей с учетом специфики их обучения для решения задач поиска сетевых атак

The problems of building neural networks to solve the problems of detecting network intrusions, taking into account modern publicly available technologies, are considered. Several configurations of neural networks are analyzed: a simple perceptron, a combined network consisting of two interconnected networks, simplified networks based on a simple perceptron, LSTM networks using hidden layers with data compression function. The weaknesses and strengths of neural network architectures are considered, taking into account the specifics of their training based on abnormal traffic datasets in intrusion detection tasks.

Deep Learning for Multi-Level Detection and Localization of Myocardial Scars Based on Regional Strain Validated on Virtual Patients

How well the heart is functioning can be quantified through measurements of myocardial deformation via echocardiography. Clinical assessment of cardiac function is generally focused on global indices of relative shortening; however, segmental strain indices have been shown to be abnormal in regions of myocardial disease such as scarring. In this work, we propose a single framework to predict myocardial scars at global, territorial, and segmental levels using regional myocardial strain traces as input to a convolutional neural network (CNN). An anatomically meaningful representation of the input data from the clinically standard bullseye representation to a multi-channel 2D image is proposed, thus enabling the use of state-of-the-art neural network configurations. A Fully Convolutional Network (FCN) is trained to detect and localize myocardial scar from regional left ventricular (LV) strain traces. Simulated regional strain data from a controlled dataset of virtual patients with varying degrees and locations of myocardial scar is used for training and validation. The proposed method successfully detects and localizes the scars on 98% of the 5490 left ventricle (LV) segments of the 305 patients in the test set using strain traces only. Due to the sparse existence of scar in the dataset, only 10% of the LV segments are scarred. Taking the imbalance into account, the class balanced accuracy is calculated as 95%. The proposed method proves successful on the strain traces of the virtual cohort and offers the potential to solve the regional myocardial scar detection problem on the strain traces of the real patient cohorts.

Data Augmentation for Deep Learning Algorithms that Perform Driver Drowsiness Detection

Driver drowsiness is one of the main causes of driver-related motor vehicle collisions, as this impairs a person’s concentration whilst driving. With the enhancements of computer vision and deep learning (DL), driver drowsiness detection systems have been developed previously, in an attempt to improve road safety. These systems experienced performance degradation under real-world testing due to factors such as driver movement and poor lighting. This study proposed to improve the training of DL models for driver drowsiness detection by applying data augmentation (DA) techniques that model these real-world scenarios. This paper studies six DL models for driver drowsiness detection: four configurations of a Convolutional Neural Network (CNN), two custom configurations as well as the architectures designed by the Visual Geometry Group (VGG) (i.e. VGG16 and VGG19); a Generative Adversarial Network (GAN) and a Multi-Layer Perceptron (MLP). These DL models were trained using two datasets of eye images, where the state of eye (open or closed) is used in determining driver drowsiness. The performance of the DL models was measured with respect to accuracy, F1-Score, precision, negative class precision, recall and specificity. When comparing the performance of DL models trained on datasets with and without DA in aggregation, it was found that all metrics were improved. After removing outliers from the results, it was found that the average improvement in both accuracy and F1 score due to DA was +4.3%. Furthermore, it is shown that the extent to which the DA techniques improve DL model performance is correlated with the inherent model performance. For DL models with accuracy and F1-Score ≤ 90%, results show that the DA techniques studied should improve performance by at least +5%.

Predicting intubation risk among COVID-19 hospitalized patients using artificial neural networks.

Accurately predicting the intubation risk in COVID-19 patients at the admission time is critical to optimal use of limited hospital resources, providing customized and evidence-based treatments, and improving the quality of delivered medical care services. This study aimed to design a statistical algorithm to select the best features influencing intubation prediction in coronavirus disease 2019 (COVID-19) hospitalized patients. Then, using selected features, multiple artificial neural network (ANN) configurations were developed to predict intubation risk. In this retrospective single-center study, a dataset containing 482 COVID-19 patients who were hospitalized between February 9, 2020 and July 20, 2021 was used. First, the Phi correlation coefficient method was performed for selecting the most important features affecting COVID-19 patients' intubation. Then, the different configurations of ANN were developed. Finally, the performance of ANN configurations was assessed using several evaluation metrics, and the best structure was determined for predicting intubation requirements among hospitalized COVID-19 patients. The ANN models were developed based on 18 validated features. The results indicated that the best performance belongs to the 18-20-1 ANN configuration with positive predictive value (PPV) = 0.907, negative predictive value (NPV) = 0.941, sensitivity = 0.898, specificity = 0.951, and area under curve (AUC) = 0.906. The results demonstrate the effectiveness of the ANN models for timely and reliable prediction of intubation risk in COVID-19 hospitalized patients. Our models can inform clinicians and those involved in policymaking and decision making for prioritizing restricted mechanical ventilation and other related resources for critically COVID-19 patients.

Segmentation-Based Angular Position Estimation Algorithm for Dynamic Path Planning by a Person-Following Robot

The study proposed a dynamic path planning (DPP) method that combines instance image segmentation and elementary matrix calculations to enable a robot to identify the angular position of entities in its surroundings. The DPP method fuses visual and depth information for scene understanding and path estimation with reduced computation resources. This study designed, developed, and evaluated a deep-learning based companion robot prototype for indoor navigation and obstacle avoidance using an RGB-D camera as the sole input sensor. A simulated environment was employed to evaluate the robot’s path-planning ability using visual sensors. The DPP method enables the person-following robot to perform intelligent curve manipulation for safe path planning to avoid objects in the initial trajectory. The approach offers a unique and straightforward technique for scene understanding without the burden of extensive neural network configuration. Its modular architecture and flexibility make it a promising candidate for future development and refinement in this domain. Its effectiveness in collision prevention and path planning has potential implications for various applications, including medical robotics.

Machine Learning Estimation of the Phase at the Fading Points of an OFDR-Based Distributed Sensor.

The paper reports a machine learning approach for estimating the phase in a distributed acoustic sensor implemented using optical frequency domain reflectometry, with enhanced robustness at the fading points. A neural network configuration was trained using a simulated set of optical signals that were modeled after the Rayleigh scattering pattern of a perturbed fiber. Firstly, the performance of the network was verified using another set of numerically generated scattering profiles to compare the achieved accuracy levels with the standard homodyne detection method. Then, the proposed method was tested on real experimental measurements, which indicated a detection improvement of at least 5.1 dB with respect to the standard approach.

Calibration of a Stereoscopic Vision System in the Presence of Errors in Pitch Angle

This paper proposes a novel method for the calibration of a stereo camera system used to reconstruct 3D scenes. An error in the pitch angle of the cameras causes the reconstructed scene to exhibit some distortion with respect to the real scene. To do the calibration procedure, whose purpose is to eliminate or at least minimize said distortion, machine learning techniques have been used, and more specifically, regression algorithms. These algorithms are trained with a large number of vectors of input features with their respective outputs, since, in view of the application of the procedure proposed, it is important that the training set be sufficiently representative of the variety that can occur in a real scene, which includes the different orientations that the pitch angle can take, the error in said angle and the effect that all this has on the reconstruction process. The most efficient regression algorithms for estimating the error in the pitch angle are derived from decision trees and certain neural network configurations. Once estimated, the error can be corrected, thus making the reconstructed scene appear more like the real one. Although the authors base their method on U-V disparity and employ this same technique to completely reconstruct the 3D scene, one of the most interesting features of the method proposed is that it can be applied regardless of the technique used to carry out said reconstruction.

Information Theory-based Evolution of Neural Networks for Side-channel Analysis

Profiled side-channel analysis (SCA) leverages leakage from cryptographic implementations to extract the secret key. When combined with advanced methods in neural networks (NNs), profiled SCA can successfully attack even those cryptocores assumed to be protected against SCA. Despite the rise in the number of studies devoted to NN-based SCA, a range of questions has remained unanswered, namely: how to choose an NN with an adequate configuration, how to tune the NN’s hyperparameters, when to stop the training, etc. Our proposed approach, “InfoNEAT,” tackles these issues in a natural way. InfoNEAT relies on the concept of neural structure search, enhanced by information-theoretic metrics to guide the evolution, halt it with novel stopping criteria, and improve time-complexity and memory footprint. The performance of InfoNEAT is evaluated by applying it to publicly available datasets composed of real side-channel measurements. In addition to the considerable advantages regarding the automated configuration of NNs, InfoNEAT demonstrates significant improvements over other approaches for effective key recovery in terms of the number of epochs (e.g.,x6 faster) and the number of attack traces compared to both MLPs and CNNs (e.g., up to 1000s fewer traces to break a device) as well as a reduction in the number of trainable parameters compared to MLPs (e.g., by the factor of up to 32). Furthermore, through experiments, it is demonstrated that InfoNEAT’s models are robust against noise and desynchronization in traces.

A Novel Memristive Neural Network Circuit and Its Application in Character Recognition.

The memristor-based neural network configuration is a promising approach to realizing artificial neural networks (ANNs) at the hardware level. The memristors can effectively simulate the strength of synaptic connections between neurons in neural networks due to their diverse significant characteristics such as nonvolatility, nanoscale dimensions, and variable conductance. This work presents a new synaptic circuit based on memristors and Complementary Metal Oxide Semiconductor(CMOS), which can realize the adjustment of positive, negative, and zero synaptic weights using only one control signal. The relationship between synaptic weights and the duration of control signals is also explained in detail. Accordingly, Widrow-Hoff algorithm-based memristive neural network (MNN) circuits are proposed to solve the recognition of three types of character pictures. The functionality of the proposed configurations is verified using SPICE simulation.

SNAL: sensitive non-associative learning network configuration for the automatic driving strategy

Nowadays, there is a huge gap between autonomous vehicles and mankind in terms of the decision response against some dangerous scenarios, which would has stressed the potential users out and even made them nervous. To efficiently identify the possible sensitivity scenarios, a new neural network configuration, named sensitive non-associative learning network (SNAL), is proposed. In such structure, the modulated interneurons, excited by abnormal scene stimulation for scene processing, are well processed and utilized to improve the training structure which refers to the sensitization mechanism in non-associative learning in neurobiology and the neural structure of Aplysia. When encountering the sensitivity scenes that the automatic driving agent is not good at or has not seen, the modulated interneuron facilitates the full connection layer neurons for the decision-making process, so as to change the final automatic driving strategy. In the process of constructing the model, a method to measure the similarity of the convolution feature map is proposed, which provides a new investigation tool for the properties of convolution networks after the feature extraction. Based on the Morris–Lecar equation in neurobiology, the dynamic model of modulating interneurons in the network is constructed. The automatic control optimization of the model is carried out by imitating the biological properties. The optimization method provides a reference for introducing neurobiological mechanism into deep learning and automatic control. To validate the effectiveness of the proposed method, the simulation test are executed and the existing methods are compared accordingly. The results show that the proposed SNAL algorithm can effectively recognize the sensitivity mechanism. Furthermore, compared with the existing algorithms, such as CNN, LSTM, ViT, the proposed algorithm can make better defensive strategies for potentially dangerous scenes rarely seen or not seen in the training stage. This sensitivity mechanism is more in line with the human driving intuition when dealing with abnormal driving scenes, and makes the decision more interpretable, significantly improving the traffic ability of autonomous vehicles under the sensitive scenes. In addition, this configuration can be easily combined with the existing mainstream neural network models and has good expansibility.

IMU-Based Classification of Locomotion Modes, Transitions, and Gait Phases with Convolutional Recurrent Neural Networks.

This paper focuses on the classification of seven locomotion modes (sitting, standing, level ground walking, ramp ascent and descent, stair ascent and descent), the transitions among these modes, and the gait phases within each mode, by only using data in the frequency domain from one or two inertial measurement units. Different deep neural network configurations are investigated and compared by combining convolutional and recurrent layers. The results show that a system composed of a convolutional neural network followed by a long short-term memory network is able to classify with a mean F1-score of 0.89 and 0.91 for ten healthy subjects, and of 0.92 and 0.95 for one osseointegrated transfemoral amputee subject (excluding the gait phases because they are not labeled in the data-set), using one and two inertial measurement units, respectively, with a 5-fold cross-validation. The promising results obtained in this study pave the way for using deep learning for the control of transfemoral prostheses with a minimum number of inertial measurement units.

Developing an Intelligent Tool for Breast Cancer Prognosis Using Artificial Neural Network

Today, there is ample scientific evidence that Breast Cancer (BC) is a global health challenge given its prevalence and invasive nature. Therefore, early detection of BC can help minimize the devastating effects of the disease. This study aimed to design a Clinical Decision Support System (CDSS) based on the best Artificial Neural Network (ANN) configuration to identify patients quickly. Using a single-center registry, we retrospectively reviewed the records of 3380 suspected BC cases. The independence test of Chi-Square at P<0.01 was utilized to select the most important criteria. Then the different ANN configuration was implemented in the Matlab R2013 environment and compared using some evaluation criteria. Finally, the best ANN configuration was obtained. After implementing feature selection, 20 variables were determined as the most relevant factors. The experimental results indicate that the best performance was obtained by the 20-25-1 configuration with PPV=90.9%, NPV=99.7%, Sensitivity=98.9%, Specificity=97.9%, Accuracy=98.1%, and AUC=0.958. The proposed software can identify cases of BC from healthy individuals with optimal diagnostic accuracy. Additionally, it might be integrated as a practical and helpful tool in natural clinical settings for easy and effective disease screening.

Gait Trajectory Prediction on an Embedded Microcontroller Using Deep Learning.

Achieving a normal gait trajectory for an amputee's active prosthesis is challenging due to its kinematic complexity. Accordingly, lower limb gait trajectory kinematics and gait phase segmentation are essential parameters in controlling an active prosthesis. Recently, the most practiced algorithm in gait trajectory generation is the neural network. Deploying such a complex Artificial Neural Network (ANN) algorithm on an embedded system requires performing the calculations on an external computational device; however, this approach lacks mobility and reliability. In this paper, more simple and reliable ANNs are investigated to be deployed on a single low-cost Microcontroller (MC) and hence provide system mobility. Two neural network configurations were studied: Multi-Layered Perceptron (MLP) and Convolutional Neural Network (CNN); the models were trained on shank and foot IMU data. The data were collected from four subjects and tested on a fifth to predict the trajectory of 200 ms ahead. The prediction was made for two cases: with and without providing the current phase of the gait. Then, the models were deployed on a low-cost microcontroller (ESP32). It was found that with fewer data (excluding the current gait phase), CNN achieved a better correlation coefficient of 0.973 when compared to 0.945 for MLP; when including the current phase, both network configurations achieved better correlation coefficients of nearly 0.98. However, when comparing the execution time required for the prediction on the intended MC, MLP was much faster than CNN, with an execution time of 2.4 ms and 142 ms, respectively. In summary, it was found that when training data are scarce, CNN is more efficient within the acceptable execution time, while MLP achieves relative accuracy with low execution time with enough data.

Automatic neural network hyperparameter optimization for extrapolation: Lessons learned from visible and near-infrared spectroscopy of mango fruit

Configuring a neural network’s architecture and hyperparameters often involves expert intuition and hand-tuning to extrapolate well without overfitting. This paper considers automatic methods for configuring a neural network for the domain of visible and near-infrared (Vis-NIR) spectroscopy. In particular, we study the effect of (a) validation set choice for validating configurations and (b) using ensembles. We consider several validation set choices: a random sample of 33% of non-test data (the technique used in previous work), samples from the latest year (a harvest season), and the first, middle, and latest 33% of samples sorted by time. To test these methods, we do a comprehensive study of a held-out 2018 harvest season of mango fruit given Vis-NIR spectra from 3 prior years. We find that ensembling improves the state-of-the-art model’s variance and accuracy. Furthermore, hyperparameter optimization experiments show that when ensembling is combined with using the latest 33% of samples as the validation set, a neural network configuration is found automatically that performs as well as an expertly-chosen configuration.

The investigation into the failure criteria of concrete based on the BP neural network

This paper proposes a method for modelling the failure criteria of concrete through the back propagation (BP) neural network. The mappings between the inputs and outputs are constructed through the mechanical analyses based on two types of failure mechanisms of concrete. The influences of the number of neurons and the activation functions in the hidden layer on the modelling results are studied by tuning the hyperparameters of the neural network. The BP neural network is trained systematically with the hidden layer of different numbers of neurons to construct the failure criteria based on the octahedral shear stress and various failure modes of concrete, respectively. The models are examined by comparing the mean square errors (MSEs) and coefficients of determination (R2). And the geometrical characteristics of the failure surfaces, the tensile and compressive meridians and the envelope features on the deviatoric plane are analyzed to evaluate the failure criteria models. The optimal configuration of the BP neural network is selected through the comparative analyses including the number of neuronal nodes, the activation functions in the hidden layer, and the input and output variables. Compared with the Ottosen failure criterion, the failure surfaces obtained through the BP neural network achieve higher accuracy, being consistent with the morphological characteristics of the failure surfaces described by previous experiments and the classical hypotheses.

PERSPECTIVE MAGNETIC NAVIGATION WITH USING THE SPLINE FUNCTIONS METHOD FOR OPTIMAL FORMATION OF THE MAP-AIDED STANDARD

An analytical review of the current problems of practical use of the planetary magnetic field as a geophysical basis for navigation is provided in the paper. A hypothesis about the possibility of orientation by an individual signature of a certain geographical area has been put forward and it is based on the fact that the Earth has a measurable magnetic field in any place and at any time, which makes the contours of abnormal magnetic intensity a reliable source of navigation. The results of domestic and foreign studies on the identification of the experimental mean square error of magnetic positioning for determining coordinates with an index of 10 m are analyzed, which in the perspective creates a precedent for supplementing magnetic navigation with the reliable backup global positioning systems. It is noted that navigation by the variable mutability of the magnetic field demonstrates a high-precision positioning potential in GPS-denied environment. As a result, the point of view of the necessity to search for an alternative method is emphasized. The actuality of creating a duplicate system is motivated by the fact that the reliability of global positioning is the subject of attention when studying the issue of cybernetic awareness for the both marine and aviation applications. The innovative approach is considered in a broad aspect, taking into account the possibility of constructing an effective configuration of an artificial neural network to remove the local magnetic field of a ship or aircraft from the measuring data of a magnetometer when using a machine learning algorithm to ensure the reliability of autonomous navigation both in near-Earth space and indoors. A three-dimensional visual representation of a digital model of a simulated magnetic field based on basic finite splines is implemented in two comparative versions: an approximated perspective of magnetic anomalies and its stylized frame model with a mathematical justification of the feasibility of using different methods as optimal standards for mapping the informativeness of magnetic positioning. The planning horizon for the incorporation of spline technology into navigation information processing has been theoretically expanded to a strategy for using a gradient approach in synthesizing the heterogeneous structure of the geophysical field in order to effectively position mobile objects based on realistic consideration of the architecture of multifactor orientation of the gradient vectors spectrum.

Prediction of the exergy performance of a hybrid photovoltaic/thermal-thermal wheel system using an optimal artificial neural network

In the present study, the aim is to estimate the annual exergy yield of a hybrid system consisting of a photovoltaic/thermal (PV/T) system and a thermal wheel. By using the hybrid system, the outside air can be cooled or heated and also, part of the electricity required by a building can be provided. The input parameters of the predictive model are the dimensions of the channel of PV/T system, the air mass flow rate and the thermal wheel length and diameter. For modeling the exergy of the studied hybrid system, a systematic method is proposed to find the optimal configuration of Artificial Neural Network (ANN). For this purpose, a novel method is proposed to discover the optimal architecture and training algorithm of the ANN by the Genetic Algorithm (GA) among 20,160 possible combinations. A dataset with 2000 input-target pairs is investigated using this method. The results indicate that a Feed-Forward (FF) network containing 19 and 10 “tansig” neurons in the first and second hidden layers that is trained by Bayesian regularization backpropagation (trainbr) can reach the best performance is the optimal ANN configuration. Based on the results, 74% of the investigated pairs have an error smaller than 0.05 %.

The effect of soiling on the performance of solar mirror materials: Experimentation and modeling

The main purpose of the current work is to examine and evaluate the impact of outdoor soiling on the performance of a sample concentrating solar power reflector. The exposure period is from April 2019 to October 2019, and during this period the mirror was left without cleaning. Achieved results from the experiment showed that the soiling index increased by almost 63 % with a maximum accumulation of dust on the mirror after approximately 20 weeks of exposure, then, it decreased due to rain event. Modeling the performance of the exposed mirror is also investigated in this paper. Thus, correlations between essential weather parameters (ambient temperature, wind speed, wind direction, relative humidity, rainfall, and dust optical depth) and soiling index are determined to select the most appropriate attributes for predicting daily soiling. The prediction is performed using Multilayer Perceptron Neural Networks approach with the adjustment of hidden layers, hidden neurons, and epoch iterations. Hence, three artificial neural networks configurations are trained, validated, and compared to other methods. The modeling study presented satisfying results, and the adapted structure includes two hidden layers and 20 hidden nodes for each layer. The best achieved performance showed a correlation coefficient of 1 and 0 % for the different calculated statistical metric errors.

On the predictability of long-term stock market returns: Design configuration of deep neural networks

In 1998, Robert J. Shiller and John Y. Campbell proposed that long-term stock market returns are not random walks and can be predicted by a valuation measure called the cyclically adjusted price-to-earnings (CAPE) ratio. This paper is set to identify the predictive power of long-term stock market returns with deep neural networks and trace the impact of different architectural components of deep neural networks. We present three network types — recurrent neural network (RNN), long short-term memory (LSTM) neural network, and gated recurrent units (GRU) neural network — to ascertain what impact the different networks have on predicting long-term stock market returns and whether a parsimonious neural network model (PNNM) can be identified for practical application. The networks above have different design features that allow returns to be predicted and the effects of the various elements of the networks to be understood. For our study, we use monthly CAPE ratios and real ten-year annualised excess returns of the S&P 500 from 1881-01 to 2012-06, with data from 1876-06 (real earnings) to 2022-06 (real total return price) needed to determine the two datasets. Our results show improved forecasting accuracy over linear regression for all analysed neural networks. Only the complex trial-and-error procedure leads to the network design with the optimal result of minimising the root-mean-squared error (RMSE). This approach is usually associated with a considerable time and cost factor. Therefore, for time series studies of the present type, we propose a parsimonious GRU architecture with low complexity and comparatively low out-of-sample error, which we call ‘GRU-101010’.