Generalization Capability Of Network Research Articles

Modulation of the emission spectrum of rare-earth ions using inverse-designed photonic crystals cavities.

Rare-earth elements play an indispensable role in the optical communication and laser industries, due to their superior luminescent properties. Nevertheless, the selective enhancement and suppression of different emission bands during energy level transitions for multi-band emitting rare-earth ions presents a significant research challenge, which we aim to address. This study explores the potential of leveraging an inverse-designed dual-cavity photonic crystals structure to manipulate the emission spectrum, thereby facilitating the augmentation or suppression of distinct emission bands. We utilized a convolutional neural network model to establish the relationship between geometric parameters and the local density of states, forecasting the optimal cavity geometry parameters for achieving the desired modulation outcomes. This paper delineates the neural network's generalization capabilities, along with the modulation efficacy of the dual-cavity configuration, both confirmed through numerical validation. Our findings highlight the modulatory capacity of Dy3+ ions, which exhibit three emission spectrum in the visible range, to achieve pure color light emission within the devised cavity structure. Notably, our approach yielded enhancements of up to 2.79-fold and 2.81-fold in pure yellow and red light emissions respectively, compared to free space emissions. The single-sided emission enhancement reaches 16.28-fold for yellow light and 30.79-fold for red light. This emphasizes the transformative potential of this methodology in crafting rare-earth-based luminescent materials with meticulously engineered emission attributes.

Accelerated Pure Shift NMR Spectroscopy with Deep Learning.

Pure shift nuclear magnetic resonance (NMR) spectroscopy presents a promising solution to provide sufficient spectral resolution and has been increasingly applied in various branches of chemistry, but the optimal resolution is generally accompanied by long experimental times. We present a proof of concept of deep learning for fast, high-quality, and reliable pure shift NMR reconstruction. The deep learning (DL) protocol allows one to eliminate undersampling artifacts, distinguish peaks with close chemical shifts, and reconstruct high-resolution pure shift NMR spectroscopy along with accelerated acquisition. More meaningfully, the lightweight neural network delivers satisfactory reconstruction performance on personal computers by several hundred simulated data learning, which somewhat lifts the prohibiting demand for a large volume of real training samples and advanced computing hardware generally required in DL projects. Additionally, an M-to-S strategy applicable to common DL cases is further exploited to boost the network generalization capability. As a result, this study takes a meaningful step toward deep learning protocols for broad chemical applications.

Event-triggered fixed/preassigned time stabilization of state-dependent switching neural networks with mixed time delays

<abstract><p>This study employed an event-triggered control (ETC) strategy to investigate the problems of fixed-time stabilization (FTS) and preassigned-time stabilization (PTS) for state-dependent switching neural networks (SDSNNs) that involved mixed time delays. To enhance the network's generalization capability and accelerate convergence stabilization, a more intricate weight-switching mechanism was introduced, then to mitigate transmission energy consumption, this paper proposed a tailored event-triggering rule that triggered the ETC solely at predetermined time points. This rule ensured the stability of the system while effectively reducing energy consumption. Using the Lyapunov stability theory and various inequality techniques, this paper presented new results for FTS and PTS of SDSNNs. The validity of these findings was supported by conducting data simulations in two illustrative examples.</p></abstract>

Transfer learning evaluation based on optimal convolution neural networks architecture for bearing fault diagnosis applications

Intelligent fault diagnosis utilizing deep learning algorithms is currently a topic of great interest. When developing a new Convolutional Neural Network (CNN) architecture to address machine diagnosis problem, it is common to use a deep model, with many layers, many feature maps, and large kernels. These models are capable of learning complex relationships and can potentially achieve superior performance on test data. However, not only does a large network potentially impose undue computational complexity for training and eventual deployment, it may also lead to more brittleness—where data outside of the curated dataset used in CNN training and evaluation is poorly handled. Accordingly, this paper will investigate a methodical approach for identifying a quasi-optimal CNN architecture to maximize robustness when a model is trained under one set of operating conditions, and deployed under a different set of conditions. Optuna software will be used to optimize a baseline CNN model for robustness to different rotational speeds and bearing Model #’s. To further improve the network generalization capabilities, this paper proposes the addition of white Gaussian noise to the raw vibration training data. Results indicate that the number of trainable weights and associated multiplications in the optimized model were reduced by almost 95% without jeopardizing the network classification accuracy. Additionally, moderate Additive White Gaussian Noise (AWGN) improved the model adaptation capabilities.

A spatial evaluation method for earthquake disaster using optimized BP neural network model

Rapid spatial evaluation of seismic disaster after earthquake occurrence is required in disaster emergency rescue management, because of its importance in decreasing casualties and property losses. Among many categories of seismic disaster, evaluation of earthquake-affected population is of great significance to clarify the severity of earthquake disaster. For simple classic regression model, it is difficult to describe the strong nonlinear relationship between multiple influencing factors and earthquake disasters. In present study, an optimized BP neural network model considering spatial characteristic of influencing factors is proposed to evaluate the population distribution affected by earthquake. The correlation between earthquake-affected population and influencing factors is analysed using data of 2013 Ms7.0 Lushan earthquake. Ten influencing factors including elevation, slope angle, population density, per capita GDP, distance to fault, distance to river, NDVI, PGA, PGV, and distance to the epicentre, were classified into environmental and seismic factors. Correlation analysis revealed that per capita GDP and PGA factor had a stronger correlation with the earthquake-affected population. The earthquake-affected population was evaluated using a BP neural network by optimizing training samples considering spatial characteristics of per capita GDP and PGA factors. Different numbers of sample points, instead of a random distribution of sample points, were generated in areas with different value intervals of the influencing factors. The optimized samples improved the convergence speed and generalization capability of neuron network compared to random samples. The trained network was applied to the 2017 Ms7.0 Jiuzhaigou earthquake to verify its prediction accuracy. The MAE of the estimated earthquake-affected populations of different counties under Jiuzhaigou earthquake were 1.276 people/km2 using network model from optimized samples, smaller than the results of network model from random samples and linear regression model. The results indicate that BP neural network, which considers correlation characteristics of factors, has capability to evaluate spatial earthquake disaster.

Mapping Soil Organic Matter Content during the Bare Soil Period by Using Satellite Data and an Improved Deep Learning Network

Soil function degradation has impaired global work in the implementation of sustainable development goals (SDGs), and soil organic matter (SOM) is a basic and the most important indicator. The deep learning neural network (i.e., DL network) has become a popular tool for mapping SOM content at a regional scale. However, outlier sample data caused by environmental factors (e.g., moisture and vegetation) and uncertain noise (e.g., random noise and instability effects) have interfered with the determination of the function mapping relationship between target soil properties and spectral features, leading to DL networks with low generalization capability. Therefore, we introduced a spatial association module into a deep neural network to remove outlier sample data in order to construct an optimal sample set for calibrating an improved deep learning (IDL) network for SOM mapping. A total of 707 soil samples and a Sentinel-2B multispectral image were acquired during the bare soil period in Weibei, China. The variable importance in the projection approach was used to select the SOM-responsive spectral features for model inputs. Measured SOM contents were taken as the dependent variable, and the IDL network was constructed and applied to map SOM at the regional scale. The results showed that Band 11 was the most important band for SOM prediction. The band difference transformation method was able to integrate multiple-band information and enhance the absorption signal of SOM. The optimal SOM-responsive spectral features included B11, B1–B11, B2−B11, B3−B11, B4−B11, and B1−B12. The IDL network exhibited better performance (R2 = 0.92; RPIQ = 4.57) regarding SOM estimation compared with the DL model performance (R2 = 0.84; RPIQ = 2.84), being improved by 9.52% (for R2) and 60.92% (for RPIQ). After introducing the spatial association module, the DL network generalization capability was enhanced. SOM distribution showed a high-value (>20 g kg−1) area in the south, and a low-value (<6 g kg−1) area in the north of the study area (the area affected by seawater intrusion). These results provide a strategy based on an IDL network and satellite data for effectively and accurately mapping SOM at the regional scale.

Expectation-maximization algorithm leads to domain adaptation for a perineural invasion and nerve extraction task in whole slide digital pathology images.

In addition to lymphatic and vascular channels, tumor cells can also spread via nerves, i.e., perineural invasion (PNI). PNI serves as an independent prognostic indicator in many malignancies. As a result, identifying and determining the extent of PNI is an important yet extremely tedious task in surgical pathology. In this work, we present a computational approach to extract nerves and PNI from whole slide histopathology images. We make manual annotations on selected prostate cancer slides once but then apply the trained model for nerve segmentation to both prostate cancer slides and head and neck cancer slides. For the purpose of multi-domain learning/prediction and investigation on the generalization capability of deep neural network, an expectation-maximization (EM)-based domain adaptation approach is proposed to improve the segmentation performance, in particular for the head and neck cancer slides. Experiments are conducted to demonstrate the segmentation performances. The average Dice coefficient for prostate cancer slides is 0.82 and 0.79 for head and neck cancer slides. Comparisons are then made for segmentations with and without the proposed EM-based domain adaptation on prostate cancer and head and neck cancer whole slide histopathology images from The Cancer Genome Atlas (TCGA) database and significant improvements are observed.

Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate Model Predictive Trajectory Tracking

Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. The model needs to capture the system behavior in multiple flight regimes and operating conditions, including those producing highly nonlinear effects such as aerodynamic forces and torques, rotor interactions, or possible system configuration modifications. Classical approaches rely on handcrafted models and struggle to generalize and scale to capture these effects. In this paper, we present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions. In addition, physics constraints are embedded in the training process to facilitate the network's generalization capabilities to data outside the training distribution. Finally, we design a model predictive control approach that incorporates the learned dynamics for accurate closed-loop trajectory tracking fully exploiting the learned model predictions in a receding horizon fashion. Experimental results demonstrate that our approach accurately extracts the structure of the quadrotor's dynamics from data, capturing effects that would remain hidden to classical approaches. To the best of our knowledge, this is the first time physics-inspired deep learning is successfully applied to temporal convolutional networks and to the system identification task, while concurrently enabling predictive control.

State of Charge Prediction of Lead Acid Battery using Transformer Neural Network for Solar Smart Dome 4.0

Renewable energy harvesting through solar photovoltaic with solar smart dome at rural area can help local farmers drying agricultural product such as coffee, spices, and dried fruit. To have a more viable and economical battery for the energy storage system, an accurate prediction battery State of Charge (SOC) is important to help control the battery charging and discharging, to extend the battery lifespan. This study explore correlation between SOC prediction with battery observable parameter such as voltage, current and temperature. Using Transformer Neural Network with comparison of Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), prediction model constructed utilizing two different datasets of laboratory lithium battery LiFePO4 and actual lead acid battery OPzS to measure model accuracy and its training time with extreme condition. Result show that voltage having strong positive correlation with SOC prediction for both battery type, while temperature having strong positive correlation only on lead acid battery. Current didn’t have direct correlation to SOC but have strong positive correlation with voltage for both battery dataset. Best prediction result gained from GRU at 45 epochs with MAE 0.642%, RMSE 0.885 %, R2 99.88% and training time of 10.74s. Transformer Neural Network accuracy placed third after LSTM with MAE 1.175%, RMSE 1.634%, R2 99.69% but it has faster training time at 7.13 second. Generalization capability of neural network in SOC prediction to produce great accuracy is proven in this study on GRU model with highest MAE of 1.19% given its challenge of limited data quantity, quality, and different battery type. Keywords— Battery State Prediction, Transformer Neural Network, State of Charge, Machine Learning, Smart Solar Dryer

Learning to localize image forgery using end-to-end attention network

Recent advancements have increased the prevalence of digital image tampering. Anyone can manipulate multimedia content using editing software to alter the semantic meaning of images to deceive viewers. Since manipulations appear realistic, both humans and machines face challenges detecting forgeries. We propose a novel algorithm for authenticating visual content by localizing forged regions in this work. Our proposed algorithm employs channel attention convolutional blocks in an end-to-end learning framework. The channel attention infers forged regions in an image by extracting attention-aware multi-resolution features in the spatial domain and features in the frequency domain. Therefore, the proposed network is divided into two subnetworks, for extracting attention-aware multi-resolution features in the spatial and frequency domain. To predict the resulting mask, we concatenate the features of both networks. The proposed channel attention network exclusively focuses on the forged region and increases network generalization capabilities on unseen manipulations. Rigorous experiments demonstrate that the proposed algorithm outperforms state-of-the-art methods on five benchmark datasets for localizing a wide range of manipulations.

A Deep Learning-Based GPR Forward Solver for Predicting B-Scans of Subsurface Objects

The forward full-wave modeling of ground-penetrating radar (GPR) facilitates the understanding and interpretation of GPR data. Traditional forward solvers require excessive computational resources, especially when their repetitive executions are needed in signal processing and/or machine learning algorithms for GPR data inversion. To alleviate the computational burden, a deep learning-based 2D GPR forward solver is proposed to predict the GPR B-scans of subsurface objects buried in the heterogeneous soil. The proposed solver is constructed as a bimodal encoder-decoder neural network. Two encoders followed by an adaptive feature fusion module are designed to extract informative features from the subsurface permittivity and conductivity maps. The decoder subsequently constructs the B-scans from the fused feature representations. To enhance the network's generalization capability, transfer learning is employed to fine-tune the network for new scenarios vastly different from those in training set. Numerical results show that the proposed solver achieves a mean relative error of 1.28%. For predicting the B-scan of one subsurface object, the proposed solver requires 12 milliseconds, which is 22,500x less than the time required by a classical physics-based solver.

Object-independent tilt detection for optical sparse aperture system with large-scale piston error via deep convolution neural network

The optical sparse aperture technique can improve the imaging resolution significantly under the ideal co-phase condition. However, the position deviation between different sub-apertures leads to notorious co-phase errors, seriously impacting the image quality. While the position deviation arises in practical applications, it is difficult to detect the errors in real-time for traditional iterative algorithms because of their narrow detection range and long-time iteration process. The deep neural network has shown its potential in optical information process, and it has some attempts in the detection of piston error. However, all existing deep learning-based methods just focus on the detection of piston error with the weak or corrected tilt error, which is not in line with reality. Here we implement the deep convolutional neural network to detect tilt error with large-scale piston error, and compare the detection performance of two kinds of network, one takes the point spread function as input while the other takes the phase diversity features as the input. The detection ability and generalization capability of network are verified under single wavelength, broadband light and turbulence aberration in simulation. The object-independent of tilt error detection ability is offered because the phase diversity features and point spread function are all unrelated to the object. In addition, the cyclic correction strategy is carried out to improve the generalization performance facing the larger errors. As a result, the deep learning-based method can detect the tilt error accurately with fast calculation, and the trained network is hopeful for the real-time correction with cyclic correction strategy.

Unsupervised domain adaptation based COVID-19 CT infection segmentation network.

Automatic segmentation of infection areas in computed tomography (CT) images has proven to be an effective diagnostic approach for COVID-19. However, due to the limited number of pixel-level annotated medical images, accurate segmentation remains a major challenge. In this paper, we propose an unsupervised domain adaptation based segmentation network to improve the segmentation performance of the infection areas in COVID-19 CT images. In particular, we propose to utilize the synthetic data and limited unlabeled real COVID-19 CT images to jointly train the segmentation network. Furthermore, we develop a novel domain adaptation module, which is used to align the two domains and effectively improve the segmentation network’s generalization capability to the real domain. Besides, we propose an unsupervised adversarial training scheme, which encourages the segmentation network to learn the domain-invariant feature, so that the robust feature can be used for segmentation. Experimental results demonstrate that our method can achieve state-of-the-art segmentation performance on COVID-19 CT images.

Calculation of scales in oil pipeline using gamma-ray scattering and artificial intelligence

This study investigates a methodology to study the deposition of barium sulfate scales (BaSO4) commonly found in the oil industry; it causes an internal diameter decrease, making it difficult for the flow. A measurement procedure was elaborated on gamma-ray scattering with three NaI(Tl) detectors and a 137Cs gamma-ray source to detect and quantify the maximum thickness of eccentric scale. The detectors data were used to train the artificial neural network for the prediction of the maximum scale thickness values regardless of oil, saltwater, gas and scale inside the tube. A data subset for training and evaluation of the artificial neural network generalization capability was generated using the MCNP6 code. Different thicknesses and positions of the maximum scale value were considered. The results show that more than 90% of the patterns presented relative errors lower than ±10%.

Intelligent Fault Diagnosis for Rotating Machines Using Deep Learning

Abstract The diagnosis of failures in high-speed machining centers and other rotary machines is critical in manufacturing systems, because early detection can save a representative amount of time and cost. Fault diagnosis systems generally have two blocks: feature extraction and classification. Feature extraction affects the performance of the prediction model, and essential information is extracted by identifying high-level abstract and representative characteristics. Deep learning (DL) provides an effective way to extract the characteristics of raw data without prior knowledge, compared with traditional machine learning (ML) methods. A feature learning approach was applied using one-dimensional (1-D) convolutional neural networks (CNN) that works directly with raw vibration signals. The network structure consists of small convolutional kernels to perform a nonlinear mapping and extract features; the classifier is a softmax layer. The method has achieved satisfactory performance in terms of prediction accuracy that reaches ∼99 % and ∼97 % using a standard bearings database: the processing time is suitable for real-time applications with ∼8 ms per signal, and the repeatability has a low standard deviation <2 % and achieves an acceptable network generalization capability.

Fast Multi-Objective Antenna Optimization Based on RBF Neural Network Surrogate Model Optimized by Improved PSO Algorithm

In this paper, a radial basis function neural network (RBFNN) surrogate model optimized by an improved particle swarm optimization (PSO) algorithm is developed to reduce the computation cost of traditional antenna design methods which rely on high-fidelity electromagnetic (EM) simulations. Considering parameters adjustment and update mechanism simultaneously, two modifications are proposed in this improved PSO. First, time-varying learning factors are designed to balance exploration and exploitation ability of particles in the search space. Second, the local best information is added to the updating process of particles except for personal and global best information for better population diversity. The improved PSO is applied to train RBFNN for determining optimal network parameters. As a result, the constructed improved PSO-RBFNN model can be used as a surrogate model for antenna performance prediction with better network generalization capability. By integrating the improved PSO-RBFNN surrogate model with multi-objective evolutionary algorithms (MOEAs), a fast multi-objective antenna optimization framework for multi-parameter antenna structures is then established. Finally, a Pareto-optimal planar miniaturized multiband antenna design is presented, demonstrating that the proposed model provides better prediction performance and considerable computational savings compared to those previously published approaches.

Impact of ECG Dataset Diversity on Generalization of CNN Model for Detecting QRS Complex

Detection of QRS complexes in electrocardiogram (ECG) signal is crucial for automated cardiac diagnosis. Automated QRS detection has been a research topic for over three decades and several of the traditional QRS detection methods show acceptable detection accuracy, however, the applicability of these methods beyond their study-specific databases was not explored. The non-stationary nature of ECG and signal variance of intra and inter-patient recordings impose significant challenges on single QRS detectors to achieve reasonable performance. In real life, a promising QRS detector may be expected to achieve acceptable accuracy over diverse ECG recordings and, thus, investigation of the model's generalization capability is crucial. This paper investigates the generalization capability of convolutional neural network (CNN) based-models from intra (subject wise leave-one-out and five-fold cross validation) and inter-database (training with single and multiple databases) points-of-view over three publicly available ECG databases, namely MIT-BIH Arrhythmia, INCART, and QT. Leave-one-out test accuracy reports 99.22%, 97.13%, and 96.25% for these databases accordingly and inter-database tests report more than 90% accuracy with the single exception of INCART. The performance variation reveals the fact that a CNN model's generalization capability does not increase simply by adding more training samples, rather the inclusion of samples from a diverse range of subjects is necessary for reasonable QRS detection accuracy.

Early Stopping Criterion Combining Probability Density Function with Validation Error for Improving the Generalization Capability of the Backpropagation Neural Network

A new early stopping criterion that combines the probability density function error and the validation error was designed to evaluate a training process. In this new criterion, the real status of the backpropagation neural network (BPNN) was jointly evaluated by the validation error, the probability density error of the traditional training data, and the unlabeled data that was not considered in the traditional training process to determine the appropriate stopping time. The proposed approach was tested on three classes of automatic stopping criteria. In addition, it was compared with the stopping criteria based solely on the validation error and the probability density error with the use of four different data sets selected from the UCI Machine Learning Repository. Results showed that the new early stopping criterion is beneficial for enhancing the generalization capability of the BPNN, demonstrating an improved forecasting precision by 2%-3%.

Data-Driven Modeling Using Improved Multi-Objective Optimization Based Neural Network for Coke Furnace System

The chamber pressure modeling of the industrial coke furnace is difficult due to the flame instability in the fuel burner and various disturbances. To deal with this issue, a new optimization method using radial basis function (RBF) neural network is proposed to improve the modeling accuracy and simplify the modeling structure. An improved multi-objective evolutionary algorithm (MOEA) is proposed to optimize the input layer, the hidden layer, and the parameters of the basis functions of the RBF neural network. The structure/parameter encoding and local search, prolong and pruning operators are designed to make MOEA suitable for optimization of the RBF neural network. Once a group of Pareto optimal solutions is derived, the RBF neural network with good generalization capability can be chosen succinctly in terms of root-mean-square error of a selected unused dataset. It shows that only a little prior knowledge of the plant is required and the approach has efficiently compromised between the generalization capability, approximation performance, and structure simplification of the RBF neural network when tested on a nonlinear dynamic function and the industrial chamber pressure.

A variable-structure online sequential extreme learning machine for time-varying system prediction

A variable-structure online sequential extreme learning machine (OS-ELM) is proposed by incorporating a hidden units pruning strategy. As conventional OS-ELM increases network dimensionality by adding newly-received samples as hidden units, the hidden layer dimension would expand and result in “dimensionality curse” finally. Furthermore, the vast number of hidden units cannot represent time-varying dynamics adaptively and would deteriorate the network generalization capability. Therefore, there is a practical need to adjust the dimension of OS-ELM not only by adding hidden units but also by simultaneously pruning superfluous units which contribute less to the output. To evaluate the individual contribution of existing hidden units, an index is proposed referred to as normalized error reduction ratio. The variable structure OS-ELM adds newly received samples in hidden units, and prunes those units contribute less to current dynamics from network simultaneously, thus the resulted network possesses parsimonious structure which can represent current system dynamics more efficiently. The online network structure adjustment approach can handle samples which are presented one-by-one or chuck-by-chuck. The variable-structure OS-ELM (VS-OSELM) can be implemented for online identification and prediction of time-varying systems. In this study, to evaluate the efficiency of VS-OSELM, it was implemented for real-time prediction of tidal level change which is a complex time-varying process. Online tidal prediction simulations is conducted based on the real measured tidal and meteorological data of Old Port Tampa in Florida, United States. Simulation results demonstrate that the proposed variable-structure OS-ELM is suitable for identification and prediction of complex time-varying systems with high prediction accuracy and fast computation speed.