Neural Network Model Research Articles

TB-BGAT With TinyBERT and BiGRU in Personalized Course Recommendations

Aiming at the problems of inadequate user feature extraction, cold start and sparse data, a personalized course recommendation algorithm that utilizes TB-BGAT is suggested. First, the Tiny Bidirectional Encoder Representation from Transformers (TinyBERT) model is utilized to output character-level word vectors; then, Bidirectional Recurrent Neural Network (BiGRU) model is utilized to obtain the embedded contextual semantic features. Finally, the attention mechanism is utilized to allocate weights to various course features by assigning their importance and to obtain the output results. The results of experiment on the publicly available dataset MOOCs-Course prove that the proposed method improves at least 3.62%, 3.04%, and 3.33% in precision, recall, and F1-score, correspondingly, in contrast to several other state-of-the-art course resource recommendation algorithms. The proposed method can enhance the effectiveness of the course recommendation model, enhance the quality of learners' online learning, and provide good technical support for online education learning platforms.

Analysis of neural networks in text sentiment

The proliferation of digital media and online platforms has led to a significant increase in the production and distribution of textual data. Consequently, the study of text sentiment analysis has emerged as a crucial area of research within the domain of natural language processing. The objective of sentiment analysis is to detect and categorize emotional inclinations in text, which has substantial practical importance in comprehending public sentiment, monitoring brand reputation, and other related areas. This research does a comprehensive investigation of the utilization of neural network models in text sentiment analysis using literature review methods. It examines the difficulties encountered and the approaches employed to overcome these difficulties. The research encompasses the fundamental principles and techniques of text sentiment analysis utilizing neural networks, as well as the utilization of convolutional neural networks (CNN), recurrent neural networks (RNN)/long short-term memory networks (LSTM), attention mechanisms, and pre-trained models in the field of sentiment analysis. Furthermore, this study delves into the complexities associated with data noise, the interpretability of models, the analysis of long text and multimodal sentiment, as well as the utilization of transfer learning and domain adaptation techniques. The study's importance is in establishing a theoretical basis and practical recommendations for the advancement of sentiment analysis technology, particularly in enhancing model precision, comprehensibility, and range of applications, thereby delivering useful insights.

Real-time body movement tracking of athletes via a CNN-based approach

Sports injuries frequently stem from improper poses and movements. This paper explores the potential of using video and deep neural network model in mitigating such risks. A novel real-time body movement tracking system is designed to enable athletes to analyze and refine their techniques without the need for intrusive sensors. Central to our approach is the development of a user-friendly graphical interface, facilitating the effortless display and examination of tracked movements. This system not only enhances athletic performance but also aims to significantly reduce the sports injuries.

Towards an open pipeline for the detection of Critical Infrastructure from satellite imagery – a case study on electrical substations in the Netherlands

Abstract Critical infrastructure (CI) are at risk of failure due to the increased frequency and magnitude of climate extremes related to climate change. It is thus essential to include them in a risk management framework to identify risk hotspots, develop risk management policies and support adaptation strategies to enhance their resilience. However, the lack of information on the exposure of CI to natural hazards prevents their incorporation in large-scale risk assessments. This study sets out to improve the representation of CI for risk assessment studies by building a neural network model to detect CI assets from optical remote sensing imagery. We present a pipeline that extracts CI from OpenStreetMap (OSM), processes the imagery and assets’ masks, and trains a Mask R-CNN model that allows for instance segmentation of CI at the asset level. This study provides an overview of the pipeline and tests it with the detection of electrical substations assets in the Netherlands. Several experiments are presented for different under-sampling percentages of the majority class (25 %, 50 % and 100 %) and hyperparameters settings (batch size and learning rate). The highest scoring experiment achieved an Average Precision at an Intersection over Union of 50 % of 30.93 and a tile F-score of 89.88 %. This allows us to confirm the feasibility of the method and invite disaster risk researchers to use this pipeline for other infrastructure types. We conclude by exploring the different avenues to improve the pipeline by addressing the class imbalance, Transfer Learning and Explainable Artificial Intelligence.

Fundamentals and recent developments of free-space optical neural networks

Machine learning with artificial neural networks has recently transformed many scientific fields by introducing new data analysis and information processing techniques. Despite these advancements, efficient implementation of machine learning on conventional computers remains challenging due to speed and power constraints. Optical computing schemes have quickly emerged as the leading candidate for replacing their electronic counterparts as the backbone for artificial neural networks. Some early integrated photonic neural network (IPNN) techniques have already been fast-tracked to industrial technologies. This review article focuses on the next generation of optical neural networks (ONNs), which can perform machine learning algorithms directly in free space. We have aptly named this class of neural network model the free space optical neural network (FSONN). We systematically compare FSONNs, IPNNs, and the traditional machine learning models with regard to their fundamental principles, forward propagation model, and training process. We survey several broad classes of FSONNs and categorize them based on the technology used in their hidden layers. These technologies include 3D printed layers, dielectric and plasmonic metasurface layers, and spatial light modulators. Finally, we summarize the current state of FSONN research and provide a roadmap for its future development.

Employing an Artificial Neural Network Model to Predict Citrus Yield Based on Climate Factors

Employing an Artificial Neural Network Model to Predict Citrus Yield Based on Climate Factors

Prediction of hematoma expansion in spontaneous intracerebral hemorrhage using a multimodal neural network

Hematoma expansion occasionally occurs in patients with intracerebral hemorrhage (ICH), associating with poor outcome. Multimodal neural networks incorporating convolutional neural network (CNN) analysis of images and neural network analysis of tabular data are known to show promising results in prediction and classification tasks. We aimed to develop a reliable multimodal neural network model that comprehensively analyzes CT images and clinical variables to predict hematoma expansion. We retrospectively enrolled ICH patients at four hospitals between 2017 and 2021, assigning patients from three hospitals to the training and validation dataset and patients from one hospital to the test dataset. Admission CT images and clinical variables were collected. CT findings were evaluated by experts. Three types of models were developed and trained: (1) a CNN model analyzing CT images, (2) a multimodal CNN model analyzing CT images and clinical variables, and (3) a non-CNN model analyzing CT findings and clinical variables with machine learning. The models were evaluated on the test dataset, focusing first on sensitivity and second on area under the receiver operating curve (AUC). Two hundred seventy-three patients (median age, 71 years [59–79]; 159 men) in the training and validation dataset and 106 patients (median age, 70 years [62–82]; 63 men) in the test dataset were included. Sensitivity and AUC of a CNN model were 1.000 (95% confidence interval [CI] 0.768–1.000) and 0.755 (95% CI 0.704–0.807); those of a multimodal CNN model were 1.000 (95% CI 0.768–1.000) and 0.799 (95% CI 0.749–0.849); and those of a non-CNN model were 0.857 (95% CI 0.572–0.982) and 0.733 (95% CI 0.625–0.840). We developed a multimodal neural network model incorporating CNN analysis of CT images and neural network analysis of clinical variables to predict hematoma expansion in ICH. The model was externally validated and showed the best performance of all the models.

Using unsupervised learning based convolutional neural networks to solve Digital Image Correlation

Digital Image Correlation (DIC) is a robust, non-contact method for full-field deformation measurement widely used in photomechanics. Based on subset image matching and an iterative optimization architecture, traditional DIC deformation field analysis requires manual selection of such calculation parameters as subset size, rendering it challenging to balance spatial resolution and measurement resolution in non-uniform field measurements. The requirement of manual parameter adjustment is lowered by a recently emerged supervised learning-based neural network DIC method, as it establishes a direct correlation between images and deformation fields using deep neural networks (DNN). However, its generalization and accuracy are contingent on the scope and variety of the dataset because it requires training with a large number of simulated speckle images of deformation fields with known standards. This paper proposes an unsupervised learning neural network based DIC measurement method that circumvents the need for standard deformation fields and achieves high precision analysis of deformed fields. It constructs a loss function based on the photometric consistency assumption of speckle images before and after deformation and allows the neural network model to directly extract displacement information from the gray data of the “reference image-deformed image” pair. Meanwhile, the model is enhanced accordingly by further introducing displacement continuity constraints to enhance accuracy and mitigate noise interference. Validation through both simulated and real experiments shows that the proposed approach surpasses traditional subset DIC in measuring non-uniform deformation fields with higher precision. It also offers superior flexibility and adaptability over supervised learning-based neural network DIC methods by eliminating the need for a large training dataset, allowing for immediate application in displacement field analysis.

Neural Network Models of Process Equipment in a Monitoring and Predictive Analytics System

Neural Network Models of Process Equipment in a Monitoring and Predictive Analytics System

Self-Regulation of the Posterior–Frontal Brain Activity with Real-Time fMRI Neurofeedback to Influence Perceptual Discrimination

The Global Neuronal Workspace (GNW) hypothesis states that the visual percept is available to conscious awareness only if recurrent long-distance interactions among distributed brain regions activate neural circuitry extending from the posterior areas to prefrontal regions above a certain excitation threshold. To directly test this hypothesis, we trained 14 human participants to increase blood oxygenation level-dependent (BOLD) signals with real-time functional magnetic resonance imaging (rtfMRI)-based neurofeedback simultaneously in four specific regions of the occipital, temporal, insular and prefrontal parts of the brain. Specifically, we hypothesized that the up-regulation of the mean BOLD activity in the posterior–frontal brain regions lowers the perceptual threshold for visual stimuli, while down-regulation raises the threshold. Our results showed that participants could perform up-regulation (Wilcoxon test, session 1: p = 0.022; session 4: p = 0.041) of the posterior–frontal brain activity, but not down-regulation. Furthermore, the up-regulation training led to a significant reduction in the visual perceptual threshold, but no substantial change in perceptual threshold was observed after the down-regulation training. These findings show that the up-regulation of the posterior–frontal regions improves the perceptual discrimination of the stimuli. However, further questions as to whether the posterior–frontal regions can be down-regulated at all, and whether down-regulation raises the perceptual threshold, remain unanswered.

Optimization and Benefit Assessment of Production Supply Chain Networks Using Graph Neural Network Models

With the flourishing development of global economy, effective management of production supply chains is crucial for the competitiveness of enterprises. Optimizing supply chain networks can not only improve the efficiency of resource allocation but also enhance market responsiveness and systemic risk resistance. Traditional supply chain network optimization methods, focusing mostly on linear models and static analysis, fall short in addressing the growing complexity and dynamism. The emergence of Graph Neural Network (GNN) models in recent years has offered new opportunities to tackle non-linearity and structural dynamism in supply chain networks. However, existing research still faces methodological limitations in supply chain node relationship mining and benefit assessment. This study introduces an optimization and benefit assessment method for production supply chain networks based on GNNs. Firstly, by developing a node role type-aware graph neural network model, this paper achieves in-depth mining and optimization of node relationships within production supply chain networks. Secondly, a hierarchical factor analysis method is used to comprehensively assess the benefits of the production supply chain. This method can dynamically capture changes in node roles and relationships within the supply chain network, optimize the network structure, and provides a multidimensional, multilevel framework for benefit assessment. This study not only expands the application of GNN in the field of supply chain management but also provides a new analytical tool for the comprehensive assessment of supply chain benefits.

Fine-Grained Recognition of Fidgety-Type Emotions Using Multi-Scale One-Dimensional Residual Siamese Network

Fidgety speech emotion has important research value, and many deep learning models have played a good role in feature modeling in recent years. In this paper, the problem of practical speech emotion is studied, and the improvement is made on fidgety-type emotion using a novel neural network model. First, we construct a large number of phonological features for modeling emotions. Second, the differences in fidgety speech between various groups of people were studied. Through the distribution of features, the individual features of fidgety emotion were studied. Third, we propose a fine-grained emotion classification method, which analyzes the subtle differences between emotional categories through Siamese neural networks. We propose to use multi-scale residual blocks within the network architecture, and alleviate the vanishing gradient problem. This allows the network to learn more meaningful representations of fidgety speech signal. Finally, the experimental results show that the proposed method can provide the versatility of modeling, and that fidgety emotion is well identified. It has great research value in practical applications.

A review of emerging trends in photonic deep learning accelerators

Deep learning has revolutionized many sectors of industry and daily life, but as application scale increases, performing training and inference with large models on massive datasets is increasingly unsustainable on existing hardware. Highly parallelized hardware like Graphics Processing Units (GPUs) are now widely used to improve speed over conventional Central Processing Units (CPUs). However, Complementary Metal-oxide Semiconductor (CMOS) devices suffer from fundamental limitations relying on metallic interconnects which impose inherent constraints on bandwidth, latency, and energy efficiency. Indeed, by 2026, the projected global electricity consumption of data centers fueled by CMOS chips is expected to increase by an amount equivalent to the annual usage of an additional European country. Silicon Photonics (SiPh) devices are emerging as a promising energy-efficient CMOS-compatible alternative to electronic deep learning accelerators, using light to compute as well as communicate. In this review, we examine the prospects of photonic computing as an emerging solution for acceleration in deep learning applications. We present an overview of the photonic computing landscape, then focus in detail on SiPh integrated circuit (PIC) accelerators designed for different neural network models and applications deep learning. We categorize different devices based on their use cases and operating principles to assess relative strengths, present open challenges, and identify new directions for further research.

Impact of various DIII-D diagnostics on the accuracy of neural network surrogates for kinetic EFIT reconstructions

Kinetic equilibrium reconstructions make use of profile information such as particle density and temperature measurements in addition to magnetics data to compute a self-consistent equilibrium. They are used in a multitude of physics-based modeling. This work develops a multi-layer perceptron (MLP) neural network (NN) model as a surrogate for kinetic Equilibrium Fitting (EFITs) and trains on the 2019 DIII-D discharge campaign database of kinetic equilibrium reconstructions. We investigate the impact of including various diagnostic data and machine actuator controls as input into the NN. When giving various categories of data as input into NN models that have been trained using those same categories of data, the predictions on multiple equilibrium reconstruction solutions (poloidal magnetic flux, global scalars, pressure profile, current profile) are highly accurate. When comparing different models with different diagnostics as input, the magnetics-only model outputs accurate kinetic profiles and the inclusion of additional data does not significantly impact the accuracy. When the NN is tasked with inferring only a single target such as the EFIT pressure profile or EFIT current profile, we see a large increase in the accuracy of the prediction of the kinetic profiles as more data is included. These results indicate that certain MLP NN configurations can be reasonably robust to different burning-plasma-relevant diagnostics depending on the accuracy requirements for equilibrium reconstruction tasks.

Deep learning and explainable artificial intelligence for investigating dental professionals' satisfaction with CAD software performance.

This study aimed to examine the satisfaction of dental professionals, including dental students, dentists, and dental technicians, with computer-aided design (CAD) software performance using deep learning (DL) and explainable artificial intelligence (XAI)-based behavioral analysis concepts. This study involved 436 dental professionals with diverse CAD experiences to assess their satisfaction with various dental CAD software programs. Through exploratory factor analysis, latent factors affecting user satisfaction were extracted from the observed variables. A multilayer perceptron artificial neural network (MLP-ANN) model was developed along with permutation feature importance analysis (PFIA) and the Shapley additive explanation (Shapley) method to gain XAI-based insights into individual factors' significance and contributions. The MLP-ANN model outperformed a standard logistic linear regression model, demonstrating high accuracy (95%), precision (84%), and recall rates (84%) in capturing complex psychological problems related to human attitudes. PFIA revealed that design adjustability was the most important factor impacting dental CAD software users' satisfaction. XAI analysis highlighted the positive impacts of features supporting the finish line and crown design, while the number of design steps and installation time had negative impacts. Notably, finish-line design-related features and the number of design steps emerged as the most significant factors. This study sheds light on the factors influencing dental professionals' decisions in using and selecting CAD software. This approach can serve as a proof-of-concept for applying DL-XAI-based behavioral analysis in dentistry and medicine, facilitating informed software selection and development.

Form Uncertainty to Sustainable Decision-Making: A Novel MIDAS–AM–DeepAR-Based Prediction Model for E-Commerce Industry Development

Global efforts to build sustainable e-commerce ecosystems through various prediction tools have suffered due to uncertainty in politics, the economy, and the environment. This paper proposes a new integrative prediction model to track the sustainable development of e-commerce. Using US e-commerce data, this study explores the prediction accuracy of the mixed data sampling (MIDAS) model in combination with the attention mechanism (AM) approach, analyzing the performance differences between the model’s before and after improvements. More so, the paper evaluates the performance of the new prediction approach against other competing models using the prediction accuracy metric, the probability interval test, and the Diebold and Mariann (DM) test methods. The results indicate that the introduction of the MIDAS and the AM approaches allows the prediction model to fully utilize the effective information of the mixed-frequency data while simultaneously capturing the differences in the importance of the variables in terms of their time series and the non-linear relationship of the learning variables, thereby positively influencing the economic prediction of the e-commerce industry. Second, the proposed prediction model combines the ability of long-term and short-term high-precision prediction and performs multi-step probability prediction on the development of the e-commerce industry. It can better track abnormal changes in macroeconomic indicators and fit their fluctuation trends. Third, based on the results of the three evaluation indicators, the MIDAS–AM–Deep autoregressive recurrent neural network (DeepAR) model achieves optimal prediction accuracy, allowing it to provide more timely, accurate, and comprehensive predictions for e-commerce management decisions when macroeconomic conditions are undergoing significant transformations.

Prediction of oil pipeline process operating parameters based on mechanism and data mining

Abstract Precisely forecasting the operational characteristics of oil pipelines is essential for developing rational design, production, and operation strategies, as well as reducing energy consumption and saving energy. Due to significant disparities in the computation outcomes of conventional mechanism models and the inadequate performance of machine learning models when handling limited sample data, their conclusions likewise lack tangible significance. In this study, a novel physics-guided neural network (PGNN) model, which integrates mechanisms with machine learning models, is introduced. The proposed model incorporates essential physical intermediate factors that impact the temperature and pressure of oil pipelines as artificial neurons within the loss function. Additionally, an adaptive moment estimate approach is employed to optimize the parameters of the model. Through a comparative analysis of various models' predictive capabilities on an oil pipeline, it was shown that PGNN has the highest level of accuracy in forecasting pipeline temperature and pressure. Furthermore, PGNN demonstrates the ability to generate satisfactory prediction outcomes even with a limited sample size. Simultaneously, the predictive outcomes of PGNN exhibit a stronger correlation with variables that have a direct impact on temperature and pressure.

Forecasting chaotic behavior of the drill bit in real-time using a DNN model integrated with rock surface morphology

The multiple regenerative effects during the drilling process cause the drilling system to exhibit complex motions, even leading to chaotic behavior. Additionally, due to the inability to obtain real-time rock parameters, traditional drilling dynamics models and time-series forecasting models cannot accurately forecast the real-time operational status of the drilling system. In this paper, a deep neural network (DNN) model integrated with rock surface morphology model is proposed to forecast the real-time operational status of the drill bit. This new model can permanently record the changes in rock surface morphology and multiple regenerative effects during the drilling process, enhancing the perception of rock parameters and improving the forecasting accuracy of the drill bit’s operational status. Experimental results show that the new model forecasts the drill bit’s operational status with higher accuracy compared to other typical time-series forecasting models without integrating surface morphology model. Furthermore, this new model demonstrates superior convergence speed, stability, and generalization ability compared to other models. In addition, the influence of hyperparameters on the forecast accuracy of the new model has also been studied. These results play an important role in forecasting and preventing adverse vibrations such as stick slip, impact, reversal, and rebound of the drill bit.

Efficient segmentation of active and inactive plaques in FLAIR-images using DeepLabV3Plus SE with efficientnetb0 backbone in multiple sclerosis

This research paper introduces an efficient approach for the segmentation of active and inactive plaques within Fluid-attenuated inversion recovery (FLAIR) images, employing a convolutional neural network (CNN) model known as DeepLabV3Plus SE with the EfficientNetB0 backbone in Multiple sclerosis (MS), and demonstrates its superior performance compared to other CNN architectures. The study encompasses various critical components, including dataset pre-processing techniques, the utilization of the Squeeze and Excitation Network (SE-Block), and the atrous spatial separable pyramid Block to enhance segmentation capabilities. Detailed descriptions of pre-processing procedures, such as removing the cranial bone segment, image resizing, and normalization, are provided. This study analyzed a cross-sectional cohort of 100 MS patients with active brain plaques, examining 5000 MRI slices. After filtering, 1500 slices were utilized for labeling and deep learning. The training process adopts the dice coefficient as the loss function and utilizes Adam optimization. The study evaluated the model's performance using multiple metrics, including intersection over union (IOU), Dice Score, Precision, Recall, and F1-Score, and offers a comparative analysis with other CNN architectures. Results demonstrate the superior segmentation ability of the proposed model, as evidenced by an IOU of 69.87, Dice Score of 76.24, Precision of 88.89, Recall of 73.52, and F1-Score of 80.47 for the DeepLabV3+SE_EfficientNetB0 model. This research contributes to the advancement of plaque segmentation in FLAIR images and offers a compelling approach with substantial potential for medical image analysis and diagnosis.

MGPPI: multiscale graph neural networks for explainable protein–protein interaction prediction

Protein-Protein Interactions (PPIs) involves in various biological processes, which are of significant importance in cancer diagnosis and drug development. Computational based PPI prediction methods are more preferred due to their low cost and high accuracy. However, existing protein structure based methods are insufficient in the extraction of protein structural information. Furthermore, most methods are less interpretable, which hinder their practical application in the biomedical field. In this paper, we propose MGPPI, which is a Multiscale graph convolutional neural network model for PPI prediction. By incorporating multiscale module into the Graph Neural Network (GNN) and constructing multi convolutional layers, MGPPI can effectively capture both local and global protein structure information. For model interpretability, we introduce a novel visual explanation method named Gradient Weighted interaction Activation Mapping (Grad-WAM), which can highlight key binding residue sites. We evaluate the performance of MGPPI by comparing with state-of-the-arts methods on various datasets. Results shows that MGPPI outperforms other methods significantly and exhibits strong generalization capabilities on the multi-species dataset. As a practical case study, we predicted the binding affinity between the spike (S) protein of SARS-COV-2 and the human ACE2 receptor protein, and successfully identified key binding sites with known binding functions. Key binding sites mutation in PPIs can affect cancer patient survival statues. Therefore, we further verified Grad-WAM highlighted residue sites in separating patients survival groups in several different cancer type datasets. According to our results, some of the highlighted residues can be used as biomarkers in predicting patients survival probability. All these results together demonstrate the high accuracy and practical application value of MGPPI. Our method not only addresses the limitations of existing approaches but also can assists researchers in identifying crucial drug targets and help guide personalized cancer treatment.