Neural Network Training Research Articles

Improving weakly-supervised object localization using adversarial erasing and pseudo label

Weakly-supervised learning approaches have gained significant attention due to their ability to reduce the effort required for human annotations in training neural networks. This paper investigates a framework for weakly-supervised object localization, which aims to train a neural network capable of predicting both the object class and its location using only images and their image-level class labels. The proposed framework consists of a shared feature extractor, a classifier, and a localizer. The localizer predicts pixel-level class probabilities, while the classifier predicts the object class at the image level. Since image-level class labels are insufficient for training the localizer, weakly-supervised object localization methods often encounter challenges in accurately localizing the entire object region. To address this issue, the proposed method incorporates adversarial erasing and pseudo labels to improve localization accuracy. Specifically, novel losses are designed to utilize adversarially erased foreground features and adversarially erased feature maps, reducing dependence on the most discriminative region. Additionally, the proposed method employs pseudo labels to suppress activation values in the background while increasing them in the foreground. The proposed method is applied to two backbone networks (MobileNetV1 and InceptionV3) and is evaluated on three publicly available datasets (ILSVRC-2012, CUB-200-2011, and PASCAL VOC 2012). The experimental results demonstrate that the proposed method outperforms previous state-of-the-art methods across all evaluated metrics.

Artificial neural networks with uniform norm-based loss functions

We explore the potential for using a nonsmooth loss function based on the max-norm in the training of an artificial neural network without hidden layers. We hypothesise that this may lead to superior classification results in some special cases where the training data are either very small or the class size is disproportional. Our numerical experiments performed on a simple artificial neural network with no hidden layer appear to confirm our hypothesis.

A Method for Identifying the Wear State of Grinding Wheels Based on VMD Denoising and AO-CNN-LSTM

Monitoring the condition of the grinding wheel in real-time during the grinding process is crucial as it directly impacts the precision and quality of the workpiece. Deep learning technology plays a vital role in analyzing the changes in sensor signals and identifying grinding wheel wear during the grinding process. Therefore, this paper innovatively proposes a grinding wheel wear recognition method based on Variational Mode Decomposition (VMD) denoising and Aquila Optimizer—Convolutional Neural Network—Long Short-Term Memory (AO-CNN-LSTM). The paper utilizes Acoustic Emission (AE) signals generated during grinding to identify the condition of the grinding wheel. To address noise interference, the study introduces the VMD algorithm for denoising the sample dataset, enhancing the effectiveness of neural network training. Subsequently, the dataset is fed into the designed Convolutional Neural Network—Long Short-Term Memory (CNN-LSTM) structure with AO-optimized parameters. Experimental results demonstrate that this method achieves high accuracy and performance.

ORDerly: Data Sets and Benchmarks for Chemical Reaction Data.

Machine learning has the potential to provide tremendous value to life sciences by providing models that aid in the discovery of new molecules and reduce the time for new products to come to market. Chemical reactions play a significant role in these fields, but there is a lack of high-quality open-source chemical reaction data sets for training machine learning models. Herein, we present ORDerly, an open-source Python package for the customizable and reproducible preparation of reaction data stored in accordance with the increasingly popular Open Reaction Database (ORD) schema. We use ORDerly to clean United States patent data stored in ORD and generate data sets for forward prediction, retrosynthesis, as well as the first benchmark for reaction condition prediction. We train neural networks on data sets generated with ORDerly for condition prediction and show that data sets missing key cleaning steps can lead to silently overinflated performance metrics. Additionally, we train transformers for forward and retrosynthesis prediction and demonstrate how non-patent data can be used to evaluate model generalization. By providing a customizable open-source solution for cleaning and preparing large chemical reaction data, ORDerly is poised to push forward the boundaries of machine learning applications in chemistry.

Security intrusion monitoring model for Internet of Things (IoT) using sniffing tools on wireless sensor networks

The Internet of Things (IoT) has revolutionized the way devices interact and share data over wireless sensor networks (WSN), enabling seamless connectivity and automation. However, the proliferation of IoT devices has raised serious security and privacy risks concerns due to their inherent vulnerabilities. This paper proposes a model for security intrusion monitoring by analyzing the existing literature and providing insights into the design, implementation, and effective deployment of the proposed model to detect intrusion in IoT using sniffing tools for network traffic analysis in real-time within WSN. The model passively monitors network traffic and identifies anomalous patterns, unauthorized access attempts, and abnormal device behavior. The review findings highlight the significance of the proposed model in enhancing the security of IoT systems. By detecting anomalous behavior and potential security breaches. The model enables timely response and mitigation actions to ensure the confidentiality, integrity and availability (CIA) of IoT devices data. The model includes consideration of network architecture, deployment of intrusion detection algorithms, and establishment of response mechanisms. It identifies various types of security threats, such as unauthorized access attempts, Denial-of-service, Distributed DoS, Brute-force, Heartbleed, Botnet, Inside Infiltration and device tampering, thereby providing response mechanisms that include generating alerts, isolating compromised devices, or blocking suspicious network traffic. The model incorporates a feedback loop to continuously update the detection mechanisms and adapt to evolving security threats in real-time. Series of experiments and simulations to be conducted using various IoT devices and network scenarios to evaluate model effectiveness. The model to comprise of wireless Router, MatLab for Deep Neural Network (DNN) training, Raspberry Pi, Wireshark setup and an array of Internet of Things (IoT) devices. The researcher to use dataset by extracting intrinsic, host-based and time-based attributes from Wireshark Sniffing tool. The datasets generated shall be piped by tshark to an output text file saved as a csv. Under-sampling technique to be used to address class imbalance of datasets. The model shall then be trained using the dataset to be able to detect intrusion in IoTs. The results is expected to demonstrate the model's ability to detect a wide range of security intrusions with high accuracy and minimal false positives. In conclusion, the model offers a proactive approach to safeguard IoT deployment. By leveraging sniffing tools and advanced analysis techniques, the model enhances the detection and response capabilities, enabling efficient protection against emerging threats in IoT. However, challenges associated with the model are identified, including the complexity of network monitoring and potential privacy concerns.

ВОЗМОЖНОСТИ ИСПОЛЬЗОВАНИЯ ГРАФОВЫХ НЕЙРОННЫХ СЕТЕЙ КАК ИНТЕЛЛЕКТУАЛЬНЫХ ПОМОЩНИКОВ ПРИ ПОСТРОЕНИИ ДИНАМИЧЕСКИХ КОГНИТИВНЫХ КАРТ

Approaches to the study of complex dynamic systems are considered. Methods of graph construc-tion with the help of neural networks are considered. The possibility of using trained generative neural networks to help in building dynamic cognitive maps is presented

A coupled data-physics computational framework for temperature, residual stress, and distortion modeling in autoclave process of composite materials

It is challenging to obtain the full-field temperature profile during autoclave processes to control the temperature uniformity and minimize the residual stress and distortion of cured composite structures. This paper proposes a coupled data-physics computational framework for full-field temperature reconstruction and the subsequent residual stress/distortion modeling by using limited monitoring temperature data. Firstly, a Long Short-Term Memory (LSTM) model is developed for full field temperature reconstruction. In this LSTM model, a cross-recombination method is proposed to maximize the value of monitored temperature data. The method effectively cuts through the bottleneck of neural network training with limited labeled data. The LSTM model’s prediction stability is enhanced based on the mean-teacher and ensemble learning strategy. To train and validate the proposed method, we perform experiments using an autoclave at the National Institute for Aviation Research (NIAR). The LSTM model’s accuracy is assessed by comparing its predicted results with the thermocouple (TC) data from measurements and high-fidelity simulation data from computational fluid dynamics (CFD). The study shows that the proposed LSTM model can effectively reconstruct the full-field temperature using limited monitoring data and significantly improve accuracy and efficiency compared with the CFD-based counterpart. Then, we create a coupled data-physics computational framework by embedding the data-driven LSTM model into a physics-based thermo-mechanical finite element model to predict residual stress and distortion. The simulation results show that the coupled data-physics framework provides an effective way for process-to-performance modeling and simulation of autoclave curing processes.

HybMED: A Hybrid Neural Network Training Processor with Multi-Sparsity Exploitation for Internet of Medical Things.

Cloud-based training and edge-based inference modes for Artificial Intelligence of Medical Things (AIoMT) applications suffer from accuracy degradation due to physiological signal variations among patients. On-chip learning can overcome this issue by online adaptation of neural network parameters for user-specific tasks. However, existing on-chip learning processors have limitations in terms of versatility, resource utilization, and energy efficiency. We propose HybMED, which is a novel neural signal processor that supports on-chip hybrid neural network training using a composite direct feedback alignment-based paradigm. HybMED is suitable for general-purpose health monitoring AIoMT devices. It improves resource utilization and area efficiency by the reconfigurable homogeneous core with heterogeneous data flow and enhances energy efficiency by exploiting sparsity at different granularities. The chip was fabricated by TSMC 40nm process and tested in multiple physiological signal processing tasks, demonstrating an average improvement in accuracy of 41.16% following online few-shot learning. The chip demonstrates an area efficiency of 1.17 GOPS/mm2 and an energy efficiency of 1.58 TOPS/W. Compared to the previous state-of-the-art physiological signal processors with on-chip learning, the chip achieves a 65× improvement in area efficiency and 1.48× improvement in energy efficiency, respectively.

Random vector functional link networks for function approximation on manifolds

The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this study, we begin to fill this theoretical gap. We then extend this result to the non-asymptotic setting using a concentration inequality for Monte-Carlo integral approximations. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error squared decaying asymptotically like O(1/n) for the number n of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided n is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Quantification of cervical spinal stenosis by automated 3D MRI segmentation of spinal cord and cerebrospinal fluid space

Abstract Design Prospective diagnostic study. Objectives Anatomical evaluation and graduation of the severity of spinal stenosis is essential in degenerative cervical spine disease. In clinical practice, this is subjectively categorized on cervical MRI lacking an objective and reliable classification. We implemented a fully-automated quantification of spinal canal compromise through 3D T2-weighted MRI segmentation. Setting Medical Center - University of Freiburg, Germany. Methods Evaluation of 202 participants receiving 3D T2-weighted MRI of the cervical spine. Segments C2/3 to C6/7 were analyzed for spinal cord and cerebrospinal fluid space volume through a fully-automated segmentation based on a trained deep convolutional neural network. Spinal canal narrowing was characterized by relative values, across sever segments as adapted Maximal Canal Compromise (aMCC), and within the index segment as adapted Spinal Cord Occupation Ratio (aSCOR). Additionally, all segments were subjectively categorized by three observers as “no”, “relative” or “absolute” stenosis. Computed scores were applied on the subjective categorization. Results 798 (79.0%) segments were subjectively categorized as “no” stenosis, 85 (8.4%) as “relative” stenosis, and 127 (12.6%) as “absolute” stenosis. The calculated scores revealed significant differences between each category (p ≤ 0.001). Youden’s Index analysis of ROC curves revealed optimal cut-offs to distinguish between “no” and “relative” stenosis for aMCC = 1.18 and aSCOR = 36.9%, and between “relative” and “absolute” stenosis for aMCC = 1.54 and aSCOR = 49.3%. Conclusion The presented fully-automated segmentation algorithm provides high diagnostic accuracy and objective classification of cervical spinal stenosis. The calculated cut-offs can be used for convenient radiological quantification of the severity of spinal canal compromise in clinical routine.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

Deep Learning Analysis with Gray Scale and Doppler Ultrasonography Images to Differentiate Graves' Disease.

Thyrotoxicosis requires accurate and expeditious differentiation between Graves' disease (GD) and thyroiditis to ensure effective treatment decisions. This study aimed to develop a machine learning algorithm using ultrasonography and Doppler images to differentiate thyrotoxicosis subtypes, with a focus on GD. This study included patients who initially presented with thyrotoxicosis and underwent thyroid ultrasonography at a single tertiary hospital. A total of 7,719 ultrasonography images from 351 patients with GD and 2,980 images from 136 patients with thyroiditis were used. Data augmentation techniques were applied to enhance the algorithm's performance. Two deep learning models, Xception and EfficientNetB0_2, were employed. Performance metrics such as accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score were calculated for both models. Image pre-processing, neural network model generation, and neural network training results verification were performed using DEEP:PHI® platform. The Xception model achieved 84.94% accuracy, 89.26% sensitivity, 73.17% specificity, 90.06% PPV, 71.43% NPV, and an F1 score of 89.66 for the diagnosis of GD. The EfficientNetB0_2 model exhibited 85.31% accuracy, 90.28% sensitivity, 71.78% specificity, 89.71% PPV, 73.05% NPV, and an F1 score of 89.99. Machine learning models based on ultrasound and Doppler images showed promising results with high accuracy and sensitivity in differentiating GD from thyroiditis.

A Novel Hybrid Approach to the Diagnosis of Simultaneous Imbalance and Shaft Bowing Faults in a Jeffcott Rotor-Bearing System

Ensuring optimal performance and reliability in rotor-bearing systems is crucial for industrial applications. Imbalances and shaft bowing in these systems can lead to decreased efficiency and increased vibrations. The early detection and mitigation of a rotor’s faults are essential, and model-based fault identification has gained much attention in the manufacturing industry over the years. Over the past two decades, however, the development of fault diagnosis rules with data-driven and artificial intelligence (AI) methods has become a trend, and in the foreseeable future the combination of AI with big data will become mainstream. Nevertheless, the critical role of rotating machinery in manufacturing introduces a challenge, as often insufficient fault data are available. This limitation renders the establishment of diagnostic rules using data-driven methods and AI technologies impractical. In light of these challenges, this study proposes a novel hybrid approach that combines a physical model with machine learning (ML) techniques for the diagnosis of multi-faults (imbalances and shaft bowing are demonstrated) in a Jeffcott rotor. To overcome the lack of real-world labeled fault datasets, a physics-based Jeffcott rotor model is first derived and then used to generate abundant fault datasets for ML. Subsequently, simulated data are employed for the training of an artificial neural network (ANN), enabling the network to learn from and analyze the vast array of generated data. The results prove that a well-trained feed-forward neural network (FNN) can accurately isolate and diagnose imbalance and shaft bowing faults using the simulated and real data from the Jeffcott rotor experiment. These physics-based and ML approaches prove effective particularly for multi-faults, offering new possibilities for advanced rotor system monitoring and maintenance strategies in industrial applications.

WiKA: A Vision Based Sign Language Recognition from Extracted Hand Joint Features using DeepLabCut

The use of hand gesture recognition for sign language translation to address the communication gap between the hearing majority and the deaf population has had significant breakthroughs over the years. While the contact-based approach uses wearable devices, a vision-based solution is preferred owing to the convenience it offers and since it obviates the need for complicated gears. This study presents the development of WiKA, an open-source software designed to track the joints of the hands and interpret them into their corresponding sign language counterparts. DeepLabCut, a markerless pose estimation software, was employed to develop the Hand-Joint Tracking Model through the training of a sequential Convolutional Neural Network, utilizing extracted Hand-Joint features to predict the sign language alphabets (A-Z) and numbers (1-9) based on the positioning of the joints. The developed Hand-Joint tracking model exhibited a 4.92% training error and a 5.74% test error with a p-cut-off of 60%. On the other hand, the developed sign language recognition achieved a 96.44% prediction accuracy with only 0.0356% misclass. This model can be further integrated into mobile phones for seamless conversations between the signing and non-signing populations.

Enhancing Training of Physics-Informed Neural Networks Using Domain Decomposition–Based Preconditioning Strategies

Enhancing Training of Physics-Informed Neural Networks Using Domain Decomposition–Based Preconditioning Strategies

Physics-Informed Neural Networks for High-Frequency and Multi-Scale Problems Using Transfer Learning

Physics-Informed Neural Network (PINN) is a data-driven solver for partial and ordinary differential equations (ODEs/PDEs). It provides a unified framework to address both forward and inverse problems. However, the complexity of the objective function often leads to training failures. This issue is particularly prominent when solving high-frequency and multi-scale problems. We proposed using transfer learning to boost the robustness and convergence of training PINN, starting training from low-frequency problems and gradually approaching high-frequency problems through fine-tuning. Through two case studies, we discovered that transfer learning can effectively train PINNs to approximate solutions from low-frequency problems to high-frequency problems without increasing network parameters. Furthermore, it requires fewer data points and less training time. We compare the PINN results using direct differences and L2 relative error showing the advantage of using transfer learning techniques. We describe our training strategy in detail, including optimizer selection, and suggest guidelines for using transfer learning to train neural networks to solve more complex problems.

Deep-learning-assisted design of multi-degree-of-freedom metamaterial absorber

In recent years, the development of metamaterial absorbers (MAs) based on deep learning methods has become a popular research topic. Nevertheless, conventional analytical approaches have limitations, leading to MAs characterized by low-degree-of-freedom cell structures, thereby restricting their absorption characteristics. In this study, we propose a multi-degree-of-freedom MA with multiple variable parameters in terms of structures, materials, and number of layers, which are distinguished by coding. Additionally, deep learning models, including a fully connected neural network, recurrent neural network, AlexNet, and residual neural network, are designed to predict structures and absorption spectra. We compare the prediction accuracy of these four neural network models and identify the optimal network model for spectral prediction and on-demand design. Using trained neural network models, we successfully designed broadband, dual-band, and single-band MAs in on-demand design. Compared with other design methods, this method provides a greater degree of freedom in device design, which allows the design of MAs to have more absorption characteristics

Analysis of the Effectiveness of Using Two-Stage Neural Network Models for Early Detection of Forest Fires

The purpose of the research – analysis of the effectiveness of two-stage neural network models for solving the problem of detecting forest fires in images obtained from unmanned aerial vehicles.Methods. А training dataset was synthesized for training neural network models for the purpose of detection and semantic segmentation of forest fires in images. Тwo-stage neural network models (“Faster R-CNN”, “Mask RCNN” and “Retina-Net”) were used for training. Тhe neural network models were trained according to the same parameters set for all models in order to ensure consistency and a common basis for experiments. Optimization of model parameters during the training process was carried out to minimize the classification loss function. Тo synthesize the test sample, we used a video sequence covering the events of forest fires in the /rkutsk region, which was filmed by an unmanned aerial vehicle. Using a specially developed script in the Рython programming language, the process of dividing this video sequence into separate frames was carried out, which were used as a test data set when assessing the quality of classification of trained neural network models.Results. Based on the analysis of the obtained values of the quality criterion, as well as visual analysis on the test data set produced as part of testing neural network models, the effectiveness of the studied models for detecting forest fires in images was assessed. Тo assess the quality of binary classification of neural network models, the quality criterion “Accuracy” (classification accuracy) was used.Conclusion. Еxperimental studies on a test data set showed that the Retina-Net model demonstrates the lowest, but acceptable, performance compared to other studied neural network models. Тhe two-stage neural network models “Faster R-CNN” and “Mask R-CNN” demonstrate similar classification accuracy values (0.9492 and 0.9521, respectively), which allows us to recommend them for use in early detection systems for forest fires.

Enhancing Microwave Photonic Interrogation Accuracy for Fiber-Optic Temperature Sensors via Artificial Neural Network Integration

In this paper, an application of an artificial neural network algorithm is proposed to enhance the accuracy of temperature measurement using a fiber-optic sensor based on a Fabry–Perot interferometer (FPI). It is assumed that the interrogation of the FPI is carried out using an optical comb generator realizing a microwave photonic approach. Firstly, modelling of the reflection spectrum of a Fabry–Perot interferometer is implemented. Secondly, probing of the obtained spectrum using a comb-generator model is performed. The resulting electrical signal of the photodetector is processed and is used to create a sample for artificial neural network training aimed at temperature detection. It is demonstrated that the artificial neural network implementation can predict temperature variations with an accuracy equal to 0.018 °C in the range from −10 to +10 °C and 0.147 in the range from −15 to +15 °C.

Innovative approach to predict soil moisture using the backpropagation‐Elman neural network

AbstractAccurate moisture prediction aids in taking early preventive measures to minimize damage caused by severe storms, droughts, and floods. The study collected 24 types of statistical data from the Xilingol Grassland, Inner Mongolia, spanning from 2011 to 2021. By integrating Elman, backpropagation (BP), and soil moisture prediction model (Elman‐BP) neural networks, soil moisture to 2 m was predicted. The soil moisture mechanism model analysis revealed that soil moisture is related to water supply rather than directly correlated with time. Based on these findings, a two‐stage soil moisture prediction model was established. In the first stage, the Elman neural network model predicted future precipitation data based on observed values. In the second stage, soil moisture was predicted using a BP neural network model based on predicted precipitation data. Comparing the mean squared errors of training and prediction data across the three models revealed that the Elman‐BP neural network had superior predictive accuracy. The maximum relative errors for the trained Elman neural network, BP neural network, and Elman‐BP model were 31.35%, 20.98%, and 3.94%. Overall, results showed a link between soil moisture and water supply and that soil moisture decreased with increasing temperatures. Soil moisture at 10‐ and 40‐cm depths was increased with increasing rainfall. Furthermore, a delay exists between the 40‐ and 100‐cm soil depths. Moreover, soil moisture stability increased with depth.