Neural Network Framework Research Articles

Enhanced Wavelet Transform-Integrated MWRC-ResNet: A Novel Framework for Interpretable and Noise-Robust Rolling Bearing Fault Diagnosis

Abstract Research on efficient detection methods for rolling bearings is crucial for enhancing the reliability and safety of mechanical equipment. Statistics indicate that over 30% of failures in rotating machinery are attributed to rolling bearings. This paper proposes the Wavelet Retention Transformation (WRT) and integrates it seamlessly with a Residual Neural Network, resulting in a novel signal processing-based Residual Neural Network framework (MWRC-ResNet). This approach significantly improves the accuracy and interpretability of fault detection in high-noise environments. The proposed method was experimentally validated using both the CWRU dataset and the HIT dataset, and the experimental results show that its accuracy and noise resistance are superior to traditional models and other wavelet-based models. This approach not only improves the accuracy of fault detection but also offers better interpretability, providing an effective solution for rolling bearing fault diagnosis.

A hybrid dynamic graph neural network framework for real-time anomaly detection

ABSTRACT The timely and robust detection of anomalies is essential for resilient and secure operations of critical water infrastructures against operational faults or malicious actions. However, real-world systems exhibit diverse and evolving spatiotemporal relationships among their components, posing an intricate challenge in anomaly detection task. This study proposes a hybrid dynamic graph neural network (GNN) that jointly maps long- and short-term spatiotemporal relationships in multivariate data streams. Those relationships are encoded via a hybrid graph, comprising an optimally learned static subgraph for persistent structural relationships and a complementary dynamic subgraph for dynamically shifting relationships. Additionally, an attention mechanism captures time-varying relational importances and shifts the model's focus towards significant relationships, while minimising contributions of less importance to the final outputs. The proposed architecture is showcased through a synthetic case study of a water distribution system with multivariate data streams from both on-site and soft sensors, and shows strong detection and localisation accuracy against all scenarios of operational faults and malicious actions explored. A comparative analysis with an equivalent static graph model indicates that the addition of the hybrid dynamic component enhances detection accuracy and reduces false alarm rates through more robust characterisation of behaviours, thus allowing actionable insights for more resilient and secure operations.

A Group Regularization Framework of Convolutional Neural Networks Based on the Impact of Lₚ Regularizers on Magnitude

A Group Regularization Framework of Convolutional Neural Networks Based on the Impact of <i>Lₚ</i> Regularizers on Magnitude

A PINN framework for inverse physical design of metal-loaded electromagnetic devices

To solve the difficulty of inverse design of metal-loaded electromagnetic devices, we propose a physics-informed neural network (PINN) framework. With the emergence of PINNs, some scholars within the field of electromagnetism have utilized them to design dielectric-loaded electromagnetic devices. However, using PINN to design metal-loaded electromagnetic devices is still uncommon due to the difficulty of binarization of metal shapes. In light of this, we propose a PINN framework to address the inverse physical design of metal-loaded electromagnetic devices. In PINN, we embed mathematical-physics models into the network, which endows the training parameters of PINN with practical physical significance. The main innovation of this paper is to solve the binarization problem in the inverse design of metal-loaded devices within the PINN. To verify the feasibility of our method, we provide an example of designing a dual-layer metal diaphragm-loaded parallel plate waveguide device. The PINN presented with metal binarization in this paper offers a novel approach to the inverse physical design of metal-loaded electromagnetic devices.

Leveraging central-surrounding receptive fields for single image dehazing

Image dehazing is a representative low-level vision task that estimates latent haze-free images from hazy images. In recent years, Vision Transformers (ViT) have shown promising dehazing performance, leveraging their capacity for global perception through long-sequence dependencies in cost of high resource consumption. Therefore, our approach seeks to integrate the utilization of global information into the Convolutional Neural Network (CNN) framework in a more resource-efficient manner. In this paper, we introduce the Adaptive Center-Surround Receptive Field (ACSRF) network architecture inspired by the central-peripheral receptive field in biological vision for single-image haze removal. This leads to a unique receptive field mechanism that effectively combines both central and surrounding information. Our ACRSF addresses this by initially compressing global information and then merging it within the CNN, significantly boosting the capability to integrate local and global information, and effectively handling dominant color tones. Experimental results on four publicly available real-world image dehazing datasets show that our ACRSF outperforms current state-of-the-art methods in recovering global information, especially in dominant color tones. Importantly, this technology demonstrates its effectiveness in realistic scenarios, contributing significantly to improving traffic safety in adverse weather conditions. The code is available at https://github.com/JavanTang/ACSRF.

DsRNAPredictor-II: An improved predictor of identifying dsRNA and its silencing efficiency for Tribolium castaneum based on sequence length distribution

RNA interference (RNAi) has been widely utilized to investigate gene functions and has significant potential for control of pest insects. However, recent studies have revealed that the target insect species, dsRNA molecule length, target genes, and other experimental factors can affect the efficiency of RNAi mediated control, restricting the further development and application of this technology. Therefore, the aim of this study was to establish a deep learning model using bioinformatics to help researchers identify dsRNA fragments with the highest RNAi efficiency. In this study, we optimized an existing model, namely, dsRNAPredictor, by designing sub-models based on different sequence lengths. Accordingly, the data were divided into two groups: 130–399 bp and 400–616 bp long sequences. Then, one-hot encoding was employed to extract sequence information. The convolutional neural network framework comprising three convolutional layers, three average pooling layers, a flattened layer, and three dense layers was employed as the classifier. By adjusting the parameters, we established two sub-models for different sequence distributions. Using multiple independent test datasets and conducting hypothesis testing, we demonstrated that our model exhibits superior performance and strong robustness to dsRNAPredictor, respectively. Therefore, our model may help design dsRNAs with pre-screening potential and facilitate further research and applications.

A Review of State-of-the-art Mixed-Precision Neural Network Frameworks.

Mixed-precision Deep Neural Networks (DNNs) provide an efficient solution for hardware deployment, especially under resource constraints, while maintaining model accuracy. Identifying the ideal bit precision for each layer, however, remains a challenge given the vast array of models, datasets, and quantization schemes, leading to an expansive search space. Recent literature has addressed this challenge, resulting in several promising frameworks. This paper offers a comprehensive overview of the standard quantization classifications prevalent in existing studies. A detailed survey of current mixed-precision frameworks is provided, with an in-depth comparative analysis highlighting their respective merits and limitations. The paper concludes with insights into potential avenues for future research in this domain.

A meshless Runge-Kutta-based Physics-Informed Neural Network framework for structural vibration analysis

In recent years, Physics-Informed Neural Networks (PINN) have emerged as powerful meshless numerical methods for solving partial differential equations (PDEs) in engineering and science, including the field of structural vibration. However, PINN struggles due to the spectral bias when the target PDEs exhibit high-frequency features. In this work, a meshless Runge-Kutta-based PINN (R-KPINN) framework for structural vibration modelling is proposed for the first time. In the framework, the meshless features of traditional PINN are retained while applying Runge-Kutta (R-K) time integration to discretise the temporal domain, thereby reducing computational demands and enhancing temporal flexibility. Besides, the proposed R-KPINN allows for segmental training of PINN, reducing the dimensionality of the structural vibration problems and offering accurate solutions to relatively high-frequency functions. Through numerical examples including free, damped, and forced vibration of Euler-Bernoulli beams, the proposed framework provides effective and precise solutions for predicting the responses of structures. In summary, the proposed R-KPINN framework provides a flexible, efficient, and interpretable meshless method for solving structural vibration problems with state-of-the-art PINN techniques.

Gradient-based differential neural network to time-varying constrained quadratic programming

This paper introduces a novel approach to solving time-varying quadratic programming (TVQP) problems with time-dependent constraints, using gradient-based differential neural networks (GDNN). We establish the theoretical framework for both conventional gradient neural networks (CGNN) and GDNN models, highlighting their effectiveness in addressing dynamic optimization challenges. Comparative theoretical analyses show that the proposed GDNN model achieves higher accuracy than the CGNN model, significantly reducing solution errors with exponential convergence. Moreover, the use of a sign-bi-power activation function (SBPAF) ensures reasonable convergence times for the GDNN model. Our approach is validated through simulations of TVQP problems under specific constraints. The results demonstrate that while both models are capable of solving these problems, the GDNN model outperforms the CGNN model in minimizing optimization errors (residual errors), especially when varying the scaling factor γ, the GDNN model also shows superior performance and more efficient convergence.

Deep Regression Learning with Optimal Loss Function

In this article, we develop a novel efficient and robust nonparametric regression estimator under a framework of a feedforward neural network (FNN). There are several interesting characteristics for the proposed estimator. First, the loss function is built upon an estimated maximum likelihood function, which integrates the information from observed data as well as the information from the data distribution. Consequently, the resulting estimator has desirable optimal properties, such as efficiency. Second, different from the traditional maximum likelihood estimation (MLE), the proposed method avoids the specification of the distribution, making it adaptable to various distributions such as heavy tails and multimodal or heterogeneous distributions. Third, the proposed loss function relies on probabilities rather than direct observations as in least square loss, contributing to the robustness of the proposed estimator. Finally, the proposed loss function involves a nonparametric regression function only. This enables the direct application of the existing packages, simplifying the computational and programming requirements. We establish the large sample property of the proposed estimator in terms of its excess risk and minimax near-optimal rate. The theoretical results demonstrate that the proposed estimator is equivalent to the true MLE where the density function is known in terms of excess risk. Our simulation studies show that the proposed estimator outperforms the existing methods based on prediction accuracy, efficiency and robustness. Particularly, it is comparable to the MLE with the known density and even gets slightly better as the sample size increases. This implies that the adaptive and data-driven loss function from the estimated density may offer an additional avenue for capturing valuable information. We further apply the proposed method to four real data examples, resulting in significantly reduced out-of-sample prediction errors compared to existing methods. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

A novel deep neural network-based technique for network embedding.

In this paper, the graph segmentation (GSeg) method has been proposed. This solution is a novel graph neural network framework for network embedding that leverages the inherent characteristics of nodes and the underlying local network topology. The key innovation of GSeg lies in its encoder-decoder architecture, which is specifically designed to preserve the network's structural properties. The key contributions of GSeg are: (1) a novel graph neural network architecture that effectively captures local and global network structures, and (2) a robust node representation learning approach that achieves superior performance in various network analysis tasks. The methodology employed in our study involves the utilization of a graph neural network framework for the acquisition of node representations. The design leverages the inherent characteristics of nodes and the underlying local network topology. To enhance the architectural framework of encoder- decoder networks, the GSeg model is specifically devised to exhibit a structural resemblance to the SegNet model. The obtained empirical results on multiple benchmark datasets demonstrate that the GSeg outperforms existing state-of-the-art methods in terms of network structure preservation and prediction accuracy for downstream tasks. The proposed technique has potential utility across a range of practical applications in the real world.

IGTP: Learning interpretable cellular embedding for inferring biological mechanisms underlying single-cell transcriptomics.

Deep-learning models like Variational AutoEncoder have enabled low dimensional cellular embedding representation for large-scale single-cell transcriptomes and shown great flexibility in downstream tasks. However, biologically meaningful latent space is usually missing if no specific structure is designed. Here, we engineered a novel interpretable generative transcriptional program (iGTP) framework that could model the importance of transcriptional program (TP) space and protein-protein interactions (PPI) between different biological states. We demonstrated the performance of iGTP in a diverse biological context using gene ontology, canonical pathway, and different PPI curation. iGTP not only elucidated the ground truth of cellular responses but also surpassed other deep learning models and traditional bioinformatics methods in functional enrichment tasks. By integrating the latent layer with a graph neural network framework, iGTP could effectively infer cellular responses to perturbations. Lastly, we applied iGTP TP embeddings with a latent diffusion model to accurately generate cell embeddings for specific cell types and states. We anticipate that iGTP will offer insights at both PPI and TP levels and holds promise for predicting responses to novel perturbations.

RECNN: Restoration and extraction of incomplete Brillouin gain spectrum based on CNN framework combined with SVAE and attention mechanism

In this paper, we propose a novel method under convolutional neural network framework combined with a supervised variational autoencoder and an attention mechanism, named the Restoration and Extraction Convolutional Neural Network (RECNN), for the restoration and extraction of Incomplete Brillouin Gain Spectrum (IBGS). It is important to clarify that the IBGS discussed here does not encompass information around the Brillouin Frequency Shift (BFS). This omission complicates the task of reconstructing the original spectrum or determining the BFS value. The RECNN framework consists of two main components: Restoration Supervised Variational Autoencoder (RSVAE) with an attention module for IBGS restoration, and Residual Attention Convolutional Neural Network (RACNN) for BFS extraction. Different types of IBGS are discussed in detail. We define the K index to quantify the incompleteness of the IBGS and introduce the R-squared index to measure the restoration performance of RSVAE. Additionally, the Root Mean Square Error (RMSE) and uncertainty are used to evaluate the overall performance of RECNN. Both simulation and experimental results demonstrate that the R-squared index increases with increasing K and Signal-to-Noise Ratio (SNR), while both RMSE and uncertainty decrease with increasing K and SNR. In comparisons with various other methods including Linear Curve Fitting (LCF) and artificial neural networks, RECNN consistently outperforms them. Specifically, simulation results show that when K is 0.5 and SNR is 4dB, the R-squared value for RSVAE reaches 0.82, significantly higher than the 0.31 achieved by LCF method. Experimental results indicate that for an SNR of 6.78dB and K of 0.5, the RMSE and uncertainty of RECNN are 3.21MHz and 2.68MHz, respectively, representing reductions of 11.43MHz and 11.17MHz compared to LCF. It's noteworthy that the time consumption evaluation indicates that RECNN requires less than 7 milliseconds to restore the complete BGS and obtain the BFS value. These results are consistently observed in both simulations and experimental studies, which bodes well for the future of extracting valuable latent information from outdated and corrupted data stored in databases of distributed fiber sensing applications.

End-to-End CNN conceptual model for a biometric authentication mechanism for ATM machines

People commonly use automated teller machines, also known as ATMs, in contemporary culture. The use of ATMs for currency withdrawal is becoming increasingly widespread in the modern world. Furthermore, attacks and break-ins have increased, highlighting the need for an ATM that is more secure and equipped with extra safety measures. The existing method of confirming these transactions through credit cards and ATMs carries a high level of risk. Card trapping, cash larceny, shoulder surfing, and ATM skimming are among the examples. The proposed system, which is based on biometrics and deep convolutional neural networks, can effectively resolve the problems associated with conventional ATM cards and PINs. The proposed method relies on face biometrics, which offers a novel technique to address the current challenge. This article presents a novel end-to-end multiscale attention-based light-weighted deep convolutional neural network (MA-LW-DCNN) framework for enhancing the accuracy of recognizing faces. The multiscale attention mechanism in face recognition enhances feature extraction by applying convolution filters at various scales, allowing the network to capture detailed and global facial features. It generates an attention map by normalizing characteristic maps and refining them using activation functions, improving recognition accuracy by focusing on essential face regions. The suggested system is assessed using a synthetic dataset against conventional machine and deep learning-based face recognition techniques. The experiment results for different individuals demonstrate an accuracy rate of around 99.84% in authenticating test samples. The proposed approach produced significantly improved evaluation measures precision, recall, and loss by 0.99, 0.99, and 0.16, respectively.

Efficient out-of-distribution detection via layer-adaptive scoring and early stopping.

Multi-layer aggregation is key to the success of out-of-distribution (OOD) detection in deep neural networks. Moreover, in real-time systems, the efficiency of OOD detection is equally important as its effectiveness. We propose a novel early stopping OOD detection framework for deep neural networks. By attaching multiple OOD detectors to the intermediate layers, this framework can detect OODs early to save computational cost. Additionally, through a layer-adaptive scoring function, it can adaptively select the optimal layer for each OOD based on its complexity, thereby improving OOD detection accuracy. Extensive experiments demonstrate that our proposed framework is robust against OODs of varying complexity. Adopting the early stopping strategy can increase OOD detection efficiency by up to 99.1% while maintaining superior accuracy. OODs of varying complexity are better detected at different layers. Leveraging the intrinsic characteristics of inputs encoded in the intermediate latent space is important for achieving high OOD detection accuracy. Our proposed framework, incorporating early stopping, significantly enhances OOD detection efficiency without compromising accuracy, making it practical for real-time applications.

DMHGNN: Double multi-view heterogeneous graph neural network framework for drug-target interaction prediction

Accurate identification of drug-target interactions (DTIs) plays a crucial role in drug discovery. Compared with traditional experimental methods that are labor-intensive and time-consuming, computational methods for drug-target interactions prediction are more popular in recent years. Conventional computational methods almost simply view heterogeneous network constructed by the drug-related and protein-related dataset instead of comprehensively exploring drug-protein pair (DPP) information. To address this limitation, we proposed a Double Multi-view Heterogeneous Graph Neural Network framework for drug-target interaction prediction (DMHGNN). In DMHGNN, one multi-view heterogeneous graph neural network is based on meta-paths and denoising autoencoder for protein-, drug-related heterogeneous network learning, and another multi-view heterogeneous graph neural network is based on multi-channel graph convolutional network for drug-protein pair similarity network learning. First, a meta-path-based graph encoder with the attention mechanism is used for substructure learning of complex relationships from heterogeneous network constructed by proteins, drugs, side-effects and diseases, obtaining key information that is easy to be ignored in global learning of heterogeneous networks, and multi-source neighbouring features for drugs and proteins are learned from heterogeneous network via denoising auto-encoder model. Then, multi-view graphs of drug-protein pairs (DPPs) including the topology graph, semantics graph and collaborative graph with shared weights are constructed, and the multi-channel graph convolutional network (GCN) is utilized to learn the deep representation of DPPs. Finally, a multi-layer fully connection network is trained to predict drug-target interactions. Experiments have demonstrated its effectiveness and better performance than state-of-the-art methods.

A Cooperative Convolutional Neural Network Framework for Multisensor Fault Diagnosis of Rotating Machinery

A Cooperative Convolutional Neural Network Framework for Multisensor Fault Diagnosis of Rotating Machinery

Multisource data fusion for defect detection in composite additive manufacturing using explainable deep neural network

Composite additive manufacturing has the potential to replace traditional composite manufacturing for aerospace applications. Additive manufacturing provides greater adaptability to complex designs, reduce material waste, and streamline the production process. However, in comparison to conventional composite manufacturing methods, additive manufacturing is inherently more susceptible to processing anomalies and defects, primarily due to the novelty of the process and the absence of applied pressure. To effectively capitalize on the benefits of additive manufacturing, a quality and reliability assurance system is critical. In this study, we have generated multisource data to analyze the inner layer thermography and surface quality, employing a multi-camera system including thermal and charge-coupled device cameras. This multisource data is utilized in an explainable zero bias deep neural network framework to detect manufacturing defects. This deep learning algorithm introduces a new zero bias layer following the regular dense layer. After being trained on normal (defect-free) samples of each data stream, the trained model is transformed into an anomaly detector by extracting low dimension features from the zero-bias layer. This allowed identification various anomalies without having to train the model on any defective inputs. As a result, the model's accuracy to detect multiple types of anomalies is higher with multisource data compared to models based on single data source. Anomaly detection accuracy was 99.72% when using data from multiple sources, 98.28% when using only CCD images, and 95% when using only thermal camera data.

MT-RAM: Multi Task-Recurrent Attention Model for partially observable image anomaly classification and localization

With the rapid development of the digital manufacturing industry, the nature of quality data has transformed from simple univariate or multivariate characteristics to big data comprising multimedia elements such as images and videos. The utilization of image data for automated monitoring and anomaly detection has gained significant attention in recent years, which also poses new and complex challenges. A critical challenge is the substantial demand for sensing and computation resources. When these resources are limited, only a fraction of the image data can be observed and analyzed. Hence, adaptive sampling becomes imperative to select the most informative pixels that effectively capture anomaly information. In this article, we propose a novel recurrent neural network framework named Multi Task-Recurrent Attention Model (MT-RAM) which incorporates adaptive sampling for anomaly classification and localization in partially observable image data. MT-RAM emulates human-like perception by generating a sequence of glimpses to comprehend the image, with the location of each glimpse depending on the information gleaned from previous glimpses. Thorough numerical studies and case studies are conducted to evaluate the performance of MT-RAM in comparison to state-of-the-art adaptive-sampling-based anomaly detection methods.

A deep neural network framework with Analytic Continuation for predicting hypervelocity fragment flyout from satellite explosions

Hypervelocity breakup events contribute to the rapidly growing population of space debris orbiting Earth. To ensure a safe environment for future space missions, it is crucial to accurately characterize resulting fragments, which constitute a major portion of the orbital population but are often too small for conventional tracking methods. Currently, the Space Surveillance Network tracks satellites and debris within a specific altitude range and releases the data publicly as Two-Line Elements, though these datasets are limited in both size and accuracy. To supplement Two-Line Element data, this paper presents a novel Deep Neural Network approach to estimate orbital elements of hypervelocity fragments resulting from simulated satellite explosions. It employs initial conditions collected from realistic terrestrial explosions considered to be relative to 5 different detonation points on the polar orbit of the weather satellite, NOAA-16, to model the initial states of debris fragments immediately following an explosion. The debris trajectories are then propagated using a high-precision semi-analytic integration method, Analytic Continuation, considering J2−J6 zonal gravitational terms and Earth’s atmospheric drag perturbations. The collected data is used to train a Deep Neural Network for each of the orbital elements. Finally, a testing data set is used to validate the results, finding that this technique accurately estimates most of the orbital elements, but is not suited to predict the true anomaly of the fragments. Therefore, K-Nearest Neighbor regression is utilized instead to predict the nature of the final orbital element.