Neural Layers Research Articles

Prediction of ball-on-plate friction and wear by ANN with data-driven optimization

For training artificial neural network (ANN), big data either generated by machine or measured from experiments are used as input to “learn” the unspecified functions defining the ANN. The experimental data are fed directly into the optimizer allowing training to be performed according to a predefined loss function. To predict sliding friction and wear at mixed lubrication conditions, in this study a specific ANN structure was so designed that deep learning algorithms and data-driven optimization models can be used. Experimental ball-on-plate friction and wear data were analyzed using the specific training procedure to optimize the weights and biases incorporated into the neural layers of the ANN, and only two independent experimental data sets were used during the ANN optimization procedure. After the training procedure, the ANN is capable to predict the contact and hydrodynamic pressure by adapting the output data according to the tribological condition implemented in the optimization algorithm.

One-shot lip-based biometric authentication: Extending behavioral features with authentication phrase information

Lip-based biometric authentication (LBBA) is an authentication method based on a person's lip movements during speech in the form of video data. LBBA can utilize both physical and behavioral characteristics of lip movements without requiring any additional sensory equipment apart from an RGB camera. Current approaches employ deep siamese neural networks trained with one-shot learning to generate embedding vectors from lip movement features. However, most of these approaches don't discriminate against speech content which makes them vulnerable to video replay attacks. Moreover, there is a lack of comprehensive analysis regarding the impact of distinct lip characteristics or difficult dataset phrases with significant word overlap on the performance of authentication in one-shot approaches. To address this, we introduce the GRID-CCP dataset and train a siamese neural network using 3D convolutions and recurrent neural network layers to additionally discriminate against speech content. For loss calculation, we propose a custom triplet loss function for efficient and customizable batch-wise hard-negative mining. Our experimental results, using an open-set protocol, demonstrate a False Acceptance Rate (FAR) of 3.2% and a False Rejection Rate (FRR) of 3.8% on the test set of the GRID-CCP dataset. Finally, we conduct an analysis to assess the influence and discriminative power of behavioral and physical features in LBBA.

DMGL-MDA: A dual-modal graph learning method for microbe-drug association prediction

The interaction between human microbes and drugs can significantly impact human physiological functions. It is crucial to identify potential microbe-drug associations (MDAs) before drug administration. However, conventional biological experiments to predict MDAs are plagued by drawbacks such as time-consuming, high costs, and potential risks. On the contrary, computational approaches can speed up the screening of MDAs at a low cost. Most computational models usually use a drug similarity matrix as the initial feature representation of drugs and stack the graph neural network layers to extract the features of network nodes. However, different calculation methods result in distinct similarity matrices, and message passing in graph neural networks (GNNs) induces phenomena of over-smoothing and over-squashing, thereby impacting the performance of the model. To address these issues, we proposed a novel graph representation learning model, dual-modal graph learning for microbe-drug association prediction (DMGL-MDA). It comprises a dual-modal embedding module, a bipartite graph network embedding module, and a predictor module. To assess the performance of DMGL-MDA, we compared it against state-of-the-art methods using two benchmark datasets. Through cross-validation, we illustrated the superiority of DMGL-MDA. Furthermore, we conducted ablation experiments and case studies to validate the effective performance of the model.

Deep learning-driven fragment ion series classification enables highly precise and sensitive de novo peptide sequencing.

Unlike for DNA and RNA, accurate and high-throughput sequencing methods for proteins are lacking, hindering the utility of proteomics in applications where the sequences are unknown including variant calling, neoepitope identification, and metaproteomics. We introduce Spectralis, a de novo peptide sequencing method for tandem mass spectrometry. Spectralis leverages several innovations including a convolutional neural network layer connecting peaks in spectra spaced by amino acid masses, proposing fragment ion series classification as a pivotal task for de novo peptide sequencing, and a peptide-spectrum confidence score. On spectra for which database search provided a ground truth, Spectralis surpassed 40% sensitivity at 90% precision, nearly doubling state-of-the-art sensitivity. Application to unidentified spectra confirmed its superiority and showcased its applicability to variant calling. Altogether, these algorithmic innovations and the substantial sensitivity increase in the high-precision range constitute an important step toward broadly applicable peptide sequencing.

Prediction and classification of IoT sensor faults using hybrid deep learning model

The quality and reliability of internet of thing (IoT) ecosystems heavily rely on accurate and dependable sensor data. However, resource limited sensors are prone to failure due to various factors like environmental disturbances and electrical noise in which they can produce erroneous and faulty measurements. These can have significant consequences across different domains, including a threat to safety in critical systems. Though many researches have been conducted, the existing literature primarily focuses on fault detection in the sensor data, while fault detection is useful, it is still a reactive approach that identifies the faults after they have occurred, meaning that actions are taken after the fault has already impacted the system, potentially leading to negative consequences. In this study, a proactive approach has been proposed by developing a two-stage solution. In the first stage, a hybrid convolutional neural network-long short term memory (CNN-LSTM) model was trained to forecast sensor measurements based on historical data, while in the second stage, the forecasted measurements were passed to a hybrid convolutional neural network-multi layer perceptron (CNN-MLP) model that has been trained to recognize different types of sensor faults and classify the new measurements accordingly. By passing the forecasted sensor values as input to the classification model and categorizing them as normal, bias, drift, random or poly-drift, anticipated the potential faults before they manifest. The publicly available Intel Lab data raw dataset is used, which has been annotated and fault-injected. For regression, gated recurrent unit (GRU), Long short term memory (LSTM), bidirectional long short term memory (BiLSTM), convolutional neural network-gated recurrent unit (CNN-GRU), convolutional neural network-long short term memory (CNN-LSTM), and convolutional neural network-bidirectional long short term memory (CNN-BiLSTM), were evaluated and compared their performance using root mean squared error (RMSE), mean squared error (MSE) and mean absolute error (MAE) with 2-split time series cross-validation. CNN-LSTM outperformed the other models with a Mean Absolute Error of 2.0957 for a 45 time steps forecast. For the classification task, convolutional neural network (CNN), multi-layer perceptron (MLP), and convolutional neural network-multi layer perceptron (CNN-MLP) evaluated using the metrics accuracy, precision, recall, and F1-score with 5 and tenfold cross-validations. CNN-MLP outperformed the others with accuracy of 96.11% for bias, 99.33% for drift, and 98.61% for random and 98.81% for poly-drift. The average accuracy across the 4 faults is 98.21%, which is a 0.3% increase from the baseline work 97.91%. By adopting a proactive approach to sensor fault prediction and classification, this research aims to enhance the reliability and efficiency of IoT systems, allowing for preventive measures to be taken before faults have a detrimental impact.

Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets.

In this article, we propose a set of transform-based neural network layers as an alternative to the 3 x 3 Conv2D layers in convolutional neural networks (CNNs). The proposed layers can be implemented based on orthogonal transforms, such as the discrete cosine transform (DCT), Hadamard transform (HT), and biorthogonal block wavelet transform (BWT). Furthermore, by taking advantage of the convolution theorems, convolutional filtering operations are performed in the transform domain using elementwise multiplications. Trainable soft-thresholding layers, that remove noise in the transform domain, bring nonlinearity to the transform domain layers. Compared with the Conv2D layer, which is spatial-agnostic and channel-specific, the proposed layers are location-specific and channel-specific. Moreover, these proposed layers reduce the number of parameters and multiplications significantly while improving the accuracy results of regular ResNets on the ImageNet-1K classification task. Furthermore, they can be inserted with a batch normalization (BN) layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy.

The hybrid of BERT and deep learning models for Indonesian sentiment analysis

<span>Artificial intelligence (AI) is one example of how data science innovation has advanced quickly in recent years and has greatly improved human existence. Neural networks, which are a type of machine learning model, are a fundamental component of deep learning in the field of AI. Deep learning models can carry out feature extraction and classification tasks in a single design because of their numerous neural network layers. Modern machine learning algorithms have been shown to perform worse than this model on tasks including text classification, audio recognition, imaginary, and pattern recognition. Deep learning models have outperformed AI-based methods in sentiment analysis and other text categorization tasks. Text data can originate from a number of places, including social media. Sentiment analysis is the computational examination of textual expressions of ideas and feelings. This study employs the convolutional neural network (CNN), long-short term memory (LSTM), CNN-LSTM, and LSTM-CNN models in a deep learning framework using bidirectional encoder representations from transformers (BERT) data representation to assess the performance of machine learning. The implementation of the model utilises YouTube discussion data pertaining to political films associated with the Indonesian presidential election of 2024. Confusion metrics, including as accuracy, precision, and recall, are then used to analyse the model’s performance.</span>

Boosting Communication Efficiency in Federated Learning for Multiagent-Based Multimicrogrid Energy Management.

Privacy of user is becoming increasingly significant in constructing efficient multiagent energy management systems for multimicrogrid (MMG). As an emerging privacy-protection method, federated learning (FL) has been used to prevent data breaches in the MMG-related field. However, with the ever-growing participants, the underlying communication burden existing in FL is evident. Besides, since the neural network layers collectively determine an agent's performance, the possible difference in layer convergence speeds would cause the inconsistency problem, that is, the FL may degrade the convergence rate of those fast-convergent layers, which weakens the overall performance of the agent. To address these issues, a communication-efficient FL (CEFL) algorithm is proposed in this study. Considering the cooperative relationship among layers, a layer evaluation (LE) mechanism is developed in CEFL to evaluate layer contribution through the Shapley value (SV), a profit distribution approach for coalitions. In this way, only partial layers with the highest contributions are selected to be uploaded to the server. In addition, instead of average parameters aggregation, a communication-efficient parameter aggregation method is proposed in CEFL to update the parameters of the global model (GM), in which an aggregation model (AM) is developed to receive parameters for aggregation. The performance of the proposed CEFL is verified by the numerical analysis of MMGs with 3-8 MGs participating. Furthermore, experiments investigate the influence of the hyperparameter in the CEFL and also demonstrate performance improvements, compared with the other four state-of-the-art algorithms.

Enhancing Load Forecasting Accuracy in Smart Grids: A Novel Parallel Multichannel Network Approach Using 1D CNN and Bi‐LSTM Models

Load forecasting plays a pivotal role in the efficient energy management of smart grid. However, the complex, intermittent, and nonlinear smart grids and the complexity of large dataset handling pose difficulty in accurately forecasting loads. The important issue is considering the cyclic features, which have not yet been adequately addressed through the trigonometric transformations. Furthermore, using long short‐term memory (LSTM) or 1D convolution neural network (1D CNN) and existing hybrid models involve stacked CNN‐LSTM architectures, employing 1D convolutions as a preprocessing step to downsample sequences and extract high‐ and low‐level spatial features. However, these models often overlook temporal features, emphasizing higher‐level features processed by the subsequent recurrent neural network layer. Therefore, this study considers a novel approach to independently process features for spatial and temporal characteristics using a parallel multichannel network comprising 1D CNN and bidirectional‐LSTM (Bi‐LSTM) models. The proposed model evaluated the National Transmission and Dispatch Company (NTDC) load dataset, with additional assessment on two datasets, American Electric Power and Commonwealth Edison, to ensure its generalizability. Performance evaluation on the NTDC dataset yields results of 3.4% mean absolute percentage error (MAPE), 513.95 mean absolute error (MAE), and 623.78 root mean square error (RMSE) for day‐ahead forecasting, and 0.56% MAPE, 94.84 MAE, and 115.67 RMSE for hour‐ahead load forecast. The experimental results demonstrate that the proposed model outperforms stacked CNN‐LSTM models, particularly in forecasting hour‐ and day‐ahead loads. Moreover, a comparative analysis with previous studies reveals superior performance in reducing the error gap between predicted and actual values.

Accelerating Neural ODEs Using Model Order Reduction.

Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time series naturally, and incorporate knowledge of physical systems into deep learning (DL) models. However, the practical applications of Neural ODEs are limited due to long inference times because the outputs of the embedded ODE layers are computed numerically with differential equation solvers that can be computationally demanding. Here, we show that mathematical model order reduction (MOR) methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and interpolation operations as layers of the neural network. We validate our approach by comparing it to neuron pruning and singular value decomposition (SVD)-based weight truncation methods from the literature in image and time-series classification tasks. The methods are evaluated by acceleration versus accuracy when adjusting the level of compression. On this spectrum, we achieve a favorable balance over existing methods by using MOR when compressing a convolutional Neural ODE. In compressing a recurrent Neural ODE, SVD-based weight truncation yields good performance. Based on our results, our integration of MOR with Neural ODEs can facilitate efficient, dynamical system-driven DL in resource-constrained applications.

Universal neural network potentials as descriptors: towards scalable chemical property prediction using quantum and classical computers

Using outputs from a pre-trained universal neural network potential's graph layer as descriptors enables efficient and accurate predictions of molecular properties. These descriptors are compact yet perform as well as the best current descriptors.

Prototype Learning Guided Hybrid Network for Breast Tumor Segmentation in DCE-MRI.

Automated breast tumor segmentation on the basis of dynamic contrast-enhancement magnetic resonance imaging (DCE-MRI) has shown great promise in clinical practice, particularly for identifying the presence of breast disease. However, accurate segmentation of breast tumor is a challenging task, often necessitating the development of complex networks. To strike an optimal tradeoff between computational costs and segmentation performance, we propose a hybrid network via the combination of convolution neural network (CNN) and transformer layers. Specifically, the hybrid network consists of a encoder-decoder architecture by stacking convolution and deconvolution layers. Effective 3D transformer layers are then implemented after the encoder subnetworks, to capture global dependencies between the bottleneck features. To improve the efficiency of hybrid network, two parallel encoder sub-networks are designed for the decoder and the transformer layers, respectively. To further enhance the discriminative capability of hybrid network, a prototype learning guided prediction module is proposed, where the category-specified prototypical features are calculated through online clustering. All learned prototypical features are finally combined with the features from decoder for tumor mask prediction. The experimental results on private and public DCE-MRI datasets demonstrate that the proposed hybrid network achieves superior performance than the state-of-the-art (SOTA) methods, while maintaining balance between segmentation accuracy and computation cost. Moreover, we demonstrate that automatically generated tumor masks can be effectively applied to identify HER2-positive subtype from HER2-negative subtype with the similar accuracy to the analysis based on manual tumor segmentation. The source code is available at https://github.com/ZhouL-lab/ PLHN.

Clinical decision support system using a machine learning model to assist simultaneous cardiopulmonary auscultation: Open-label randomized controlled trial.

The utility of a clinical decision support system using a machine learning (ML) model for simultaneous cardiac and pulmonary auscultation is unknown. This study aimed to develop and evaluate an ML system's utility for cardiopulmonary auscultation. First, we developed an ML system for cardiopulmonary auscultation, using cardiopulmonary sound files from our previous study. The technique involved pre-processing, feature extraction, and classification through several neural network layers. After integration, the output class was categorized as "normal," "abnormal," or "undetermined." Second, we evaluated the ML system with 24 junior residents in an open-label randomized controlled trial at a university hospital. Participants were randomly assigned to the ML system group (intervention) or conventional auscultation group (control). During training, participants listened to four cardiac and four pulmonary sounds, all of which were correctly classified. Then, participants classified a series of 16 simultaneous cardiopulmonary sounds. The control group auscultated the sounds using noise-cancelling headphones, while the intervention group did so by watching recommendations from the ML system. The total scores for correctly identified normal or abnormal cardiopulmonary sounds in the intervention group were significantly higher than those in the control group (366/384 [95.3%] vs. 343/384 [89.3%], P = 0.003). The cardiac test score in the intervention group was better (111/192 [57.8%] vs. 90/192 [46.9%], P = 0.04); there was no significant difference in pulmonary auscultation. The ML-based system improved the accuracy of cardiopulmonary auscultation for junior residents. This result suggests that the system can assist early-career physicians in accurate screening.

An End-to-End Deep Graph Clustering via Online Mutual Learning.

In clustering fields, the deep graph models generally utilize the graph neural network to extract the deep embeddings and aggregate them according to the data structure. The optimization procedure can be divided into two individual stages, optimizing the neural network with gradient descent and generating the aggregation with a machine learning-based algorithm. Hence, it means that clustering results cannot guide the optimization of graph neural networks. Besides, since the aggregating stage involves complicated matrix computation such as decomposition, it brings a high computational burden. To address these issues, a unified deep graph clustering (UDGC) model via online mutual learning is proposed in this brief. Specifically, it maps the data into the deep embedding subspace and extracts the deep graph representation to explore the latent topological knowledge of the nodes. In the deep subspace, the model aggregates the embeddings and generates the clustering assignments via the local preserving loss. More importantly, we train a neural layer to fit the clustering results and design an online mutual learning strategy to optimize the whole model, which can not only output the clustering assignments end-to-end but also reduce the computation complexity. Extensive experiments support the superiority of our model.

An effective framework for predicting drug–drug interactions based on molecular substructures and knowledge graph neural network

Drug–drug interactions (DDIs) play a central role in drug research, as the simultaneous administration of multiple drugs can have harmful or beneficial effects. Harmful interactions lead to adverse reactions, some of which can be life-threatening, while beneficial interactions can promote efficacy. Therefore, it is crucial for physicians, patients, and the research community to identify potential DDIs. Although many AI-based techniques have been proposed for predicting DDIs, most existing computational models primarily focus on integrating multiple data sources or combining popular embedding methods. Researchers often overlook the valuable information within the molecular structure of drugs or only consider the structural information of drugs, neglecting the relationship or topological information between drugs and other biological objects. In this study, we propose MSKG-DDI – a two-component framework that incorporates the Drug Chemical Structure Graph-based component and the Drug Knowledge Graph-based component to capture multimodal characteristics of drugs. Subsequently, a multimodal fusion neural layer is utilized to explore the complementarity between multimodal representations of drugs. Extensive experiments were conducted using two real-world datasets, and the results demonstrate that MSKG-DDI outperforms other state-of-the-art models in binary-class, multi-class, and multi-label prediction tasks under both transductive and inductive settings. Furthermore, the ablation analysis further confirms the practical usefulness of MSKG-DDI.

NA

NA

BioEGRE: a linguistic topology enhanced method for biomedical relation extraction based on BioELECTRA and graph pointer neural network

BackgroundAutomatic and accurate extraction of diverse biomedical relations from literature is a crucial component of bio-medical text mining. Currently, stacking various classification networks on pre-trained language models to perform fine-tuning is a common framework to end-to-end solve the biomedical relation extraction (BioRE) problem. However, the sequence-based pre-trained language models underutilize the graphical topology of language to some extent. In addition, sequence-oriented deep neural networks have limitations in processing graphical features.ResultsIn this paper, we propose a novel method for sentence-level BioRE task, BioEGRE (BioELECTRA and Graph pointer neural net-work for Relation Extraction), aimed at leveraging the linguistic topological features. First, the biomedical literature is preprocessed to retain sentences involving pre-defined entity pairs. Secondly, SciSpaCy is employed to conduct dependency parsing; sentences are modeled as graphs based on the parsing results; BioELECTRA is utilized to generate token-level representations, which are modeled as attributes of nodes in the sentence graphs; a graph pointer neural network layer is employed to select the most relevant multi-hop neighbors to optimize representations; a fully-connected neural network layer is employed to generate the sentence-level representation. Finally, the Softmax function is employed to calculate the probabilities. Our proposed method is evaluated on three BioRE tasks: a multi-class (CHEMPROT) and two binary tasks (GAD and EU-ADR). The results show that our method achieves F1-scores of 79.97% (CHEMPROT), 83.31% (GAD), and 83.51% (EU-ADR), surpassing the performance of existing state-of-the-art models.ConclusionThe experimental results on 3 biomedical benchmark datasets demonstrate the effectiveness and generalization of BioEGRE, which indicates that linguistic topology and a graph pointer neural network layer explicitly improve performance for BioRE tasks.

Synchronization in the two time-scale multiplex network with symplectic interactions

Mounting experimental evidence suggests a significant role for glial cells, particularly astrocytes, in shaping neuronal activity in both cell cultures and the brain, driving interest in exploring multi-component network dynamics [1]. These systems are modeled at varying levels of physiological detail, ranging from the Hodgkin-Huxley and Ullah models for neurons and astrocytes, respectively, to simple integrate-and-fire models or phase oscillators. Multiplex networks are commonly used to model generic dynamical phenomena, with an emphasis on network connectivity effects rather than the complexity of an individual cell’s behavior. This model is particularly advantageous due to its use of a two-layer network structure [2]. This structure accurately describes neural connectivity through the long-range random coupling, while glial cells interact locally through the diffusion of mediatory molecules. Furthermore, the oscillatory timescales between neural and glial cells differ by at least one order of magnitude. In this context, the analysis of synchronization is a focal point as it is a crucial process that underpins information processing, decision making, and movement control in living neural systems [3]. Network topology is known to be essential, where all-to-all coupling and random Erdos–Renyi connectivity support the second-order phase transition to global phase coherence. In contrast, regular local coupling such as a 2D lattice only permits frequency and phase locking. Additionally, symplectic networks are capable of demonstrating a first-order phase transition, which is commonly referred to as “explosive” synchronization [4]. Recently, studies have shown that combining model neural (random) and glial (regular lattice) oscillatory layers can lead to a variety of outcomes [5]. Specifically, this combination was found to induce a second-order phase transition in the regular glial layer. In addition, synchronization in both layers may be preceded by desynchronization as the coupling between the layers becomes stronger. Here we investigate synchronization in the multiplex neural-glial network, where the neural layer is symplectic and contains triadic interactions. We demonstrate that symplectic coupling in the neural layer can induce the first order transition in the regular (glial) layer. Additionally, we show that such a transition can be induced by strengthening the glial and interlayer coupling, even if the symplectic neural layer alone is below the synchronization threshold. Desynchronization in the neuronal layer, resulting from the moderated coupling to the glial layer, never occurs abruptly.

Learning lattice quantum field theories with equivariant continuous flows

We propose a novel machine learning method for sampling from the high-dimensional probability distributions of Lattice Field Theories, which is based on a single neural ODE layer and incorporates the full symmetries of the problem. We test our model on the \phi^4ϕ4 theory, showing that it systematically outperforms previously proposed flow-based methods in sampling efficiency, and the improvement is especially pronounced for larger lattices. Furthermore, we demonstrate that our model can learn a continuous family of theories at once, and the results of learning can be transferred to larger lattices. Such generalizations further accentuate the advantages of machine learning methods.

Predicting consumer acceptance of automobiles based on deep learning and traditional machine learning algorithms

Researchers have made significant progress in machine learning in recent years. Machine learning can learn and predict large and complex data sets. Researchers have divided machine learning algorithms into two categories: deep learning and traditional machine learning. Every problem can be predicted in both ways. This paper uses the "Car Data" dataset to investigate deep learning and traditional machine learning. In order to find a machine learning algorithm that is more conducive to analyzing and predicting consumers' acceptance of different cars, this paper mainly explores the differences in the prediction accuracy of the three methods of Neural Networks, Random Forest and Support Vector Machine (SVM). We construct 3-hidden layers neural networks and 4-hidden layers neural networks. After testing, it is known that the result predicted by Random Forest is the worst. The prediction accuracy of 3-hidden layers Neural Networks is similar to that by SVM. When we added an extra layer of hidden layers on the basis of 3-hidden layers, the prediction accuracy was higher than that of SVM. Adding a hidden layer can improve the prediction accuracy, and both SVM and Neural Network can be used to analyze Car Data. But not all methods have similar predictive accuracy.