Neural Network Module Research Articles

Statistical Prediction of [CII] Observations by Constructing Probability Density Functions Using SOFIA, Herschel, and Spitzer Observations

We present a statistical algorithm for predicting the [CII] emission from Herschel and Spitzer continuum images using probability density functions (PDFs) between the [CII] emission and continuum emission. The [CII] emission at 158[Formula: see text][Formula: see text]m is a critical tracer in studying the life cycle of interstellar medium and galaxy evolution. Unfortunately, its frequency is in the far-infrared (FIR), which is opaque through the troposphere and cannot be observed from the ground except for highly red-shifted sources ([Formula: see text]). Typically [CII] observations of closer regions have been carried out using suborbital or space observatories. Given the high cost of these facilities and limited time availability, it is important to have highly efficient observations/operations in terms of maximizing science returns. This requires accurate prediction of the strength of emission lines and, therefore, the time required for their observation. However, [CII] emission has been hard to predict due to a lack of strong correlations with other observables. Here we adopt a new approach to making accurate predictions of [CII] emission by relating this emission simultaneously to several tracers of dust emission in the same region. This is done using a statistical methodology utilizing PDFs among [CII] emission and Spitzer IRAC and Herschel PACS/SPIRE images. Our test result toward a star-forming region, RCW 120, demonstrates that our methodology delivers high-quality predictions with less than 30% uncertainties over 80% of the entire observation area, which is more than sufficient to test observation feasibility and maximize science return. The pickle dump files storing the PDFs and trained neural network module are accessible upon request and will support future FIR missions, for example, GUSTO and FIR Probe.

Estimation of interlayer textural relationships to discriminate the benignancy/malignancy of brain tumors

A computer-aided diagnosis system is a popular tool to predict the risk factors of brain tumors. However, the existing techniques are unable to provide high detection accuracy due to the inability of capturing the hidden features of brain tumors. In this view, the present work decomposes the MR modality images into different layers using non-sub-sampled shearlet transformation (NSST). Following this, the proposed detection pipeline employs adaptive pulse-coupled neural network (A-PCNN) module and a fuzzy c-means (FCM) clustering algorithm for enhancement and segmentation of suspicious regions respectively. Interlayer textural relationships of tumors are estimated in terms of the layer-wise difference of the gray-level cooccurrence matrix (GLCM), similarity indices, and entropy distributions. Thenceforth, those feature coefficients are coded with binary sequences to express the textural homogeneity/randomness of tumors. Finally, these handcrafted multi-scale feature quantifiers are fed to some standard classifiers such as the k-nearest neighbor (kNN) technique, the linear square support vector machine (LS-SVM), and the decision tree (DT) for discriminating the state of benignancy/malignancy of tumors. Experimental results ensure that the proposed detection model shows superior performance compared to existing methods.

Personalized recommendation via inductive spatiotemporal graph neural network

Graph neural network-based collaborative filtering methods have achieved excellent performance in recommender systems. However, previous works have primarily focused on representation using the entire graph or a single sampled subgraph, which fails to address the issue of noise introduced by prolonged ineffective interactions. Moreover, these approaches lack the capacity to model temporal information in an inductive manner, thereby limiting their ability to fully capture the evolving dynamic interests of users over time in a real world scenario. To address the aforementioned challenges, we propose a novel personalized inductive spatiotemporal graph neural network-based framework PistGNN. PistGNN extracts multiple temporal-aware subgraphs from user–item pairs to avoid long-time gap meaningless interactions, which are then fed into a spatiotemporal graph neural network module. It is worth noting that PistGNN trains the model only by relying on subgraphs extracted from bipartite graphs, without depending on global information, and is therefore capable of predicting for new users or items. Finally, we propose an attention-based meta-learning method to personalize the aggregation of subgraphs’ embeddings. Extensive experiments conducted on four real-world datasets demonstrate the superiority of PistGNN over both inductive and transductive baseline methods.

Investigations of brain-wide functional and structural networks of dopaminergic and CamKIIα-positive neurons in VTA with DREADD-fMRI and neurotropic virus tracing technologies

BackgroundThe ventral tegmental area (VTA) contains heterogeneous cell populations. The dopaminergic neurons in VTA play a central role in reward and cognition, while CamKIIα-positive neurons, composed mainly of glutamatergic and some dopaminergic neurons, participate in the reward learning and locomotor activity behaviors. The differences in brain-wide functional and structural networks between these two neuronal subtypes were comparatively elucidated.MethodsIn this study, we applied a method combining Designer Receptors Exclusively Activated by Designer Drugs (DREADD) and fMRI to assess the cell type-specific modulation of whole-brain neural networks. rAAV encoding the cre-dependent hM3D was injected into the right VTA of DAT-cre or CamKIIα-cre transgenic rats. The global brain activities elicited by DREADD stimulation were then detected using BOLD-fMRI. Furthermore, the cre-dependent antegrade transsynaptic viral tracer H129ΔTK-TT was applied to label the outputs of VTA neurons.ResultsWe found that DREADD stimulation of dopaminergic neurons induced significant BOLD signal changes in the VTA and several VTA-related regions including mPFC, Cg and Septum. More regions responded to selective activation of VTA CamKIIα-positive neurons, resulting in increased BOLD signals in VTA, Insula, mPFC, MC_R (Right), Cg, Septum, Hipp, TH_R, PtA_R, and ViC_R. Along with DREADD-BOLD analysis, further neuronal tracing identified multiple cortical (MC, mPFC) and subcortical (Hipp, TH) brain regions that are structurally and functionally connected by VTA dopaminergic and CamKIIα-positive neurons.ConclusionsOur study dissects brain-wide structural and functional networks of two neuronal subtypes in VTA and advances our understanding of VTA functions.

AttenSyn: An Attention-Based Deep Graph Neural Network for Anticancer Synergistic Drug Combination Prediction.

Identifying synergistic drug combinations is fundamentally important to treat a variety of complex diseases while avoiding severe adverse drug-drug interactions. Although several computational methods have been proposed, they highly rely on handcrafted feature engineering and cannot learn better interactive information between drug pairs, easily resulting in relatively low performance. Recently, deep-learning methods, especially graph neural networks, have been widely developed in this area and demonstrated their ability to address complex biological problems. In this study, we proposed AttenSyn, an attention-based deep graph neural network for accurately predicting synergistic drug combinations. In particular, we adopted a graph neural network module to extract high-latent features based on the molecular graphs only and exploited the attention-based pooling module to learn interactive information between drug pairs to strengthen the representations of drug pairs. Comparative results on the benchmark datasets demonstrated that our AttenSyn performs better than the state-of-the-art methods in the prediction of anticancer synergistic drug combinations. Additionally, to provide good interpretability of our model, we explored and visualized some crucial substructures in drugs through attention mechanisms. Furthermore, we also verified the effectiveness of our proposed AttenSyn on two cell lines by visualizing the features of drug combinations learnt from our model, exhibiting satisfactory generalization ability.

Enhancing Cross-Lingual Entity Alignment in Knowledge Graphs through Structure Similarity Rearrangement.

Cross-lingual entity alignment in knowledge graphs is a crucial task in knowledge fusion. This task involves learning low-dimensional embeddings for nodes in different knowledge graphs and identifying equivalent entities across them by measuring the distances between their representation vectors. Existing alignment models use neural network modules and the nearest neighbors algorithm to find suitable entity pairs. However, these models often ignore the importance of local structural features of entities during the alignment stage, which may lead to reduced matching accuracy. Specifically, nodes that are poorly represented may not benefit from their surrounding context. In this article, we propose a novel alignment model called SSR, which leverages the node embedding algorithm in graphs to select candidate entities and then rearranges them by local structural similarity in the source and target knowledge graphs. Our approach improves the performance of existing approaches and is compatible with them. We demonstrate the effectiveness of our approach on the DBP15k dataset, showing that it outperforms existing methods while requiring less time.

Flexible optical fiber channel modeling based on a neural network module.

Optical fiber channel modeling, which is essential in optical transmission system simulations and designs, is usually based on the split-step Fourier method (SSFM), making the simulation quite time-consuming owing to the iteration steps. Here, we train a neural network module termed NNSpan to learn the transfer function of a single fiber (G652 or G655) span with a length of 80 km and successfully emulate long-haul optical transmission systems by cascading multiple NNSpans, which gives remarkable prediction accuracy, even over a transmission distance of 1000 km. Even when trained without erbium-doped fiber amplifier (EDFA) noise, NNSpan performs quite well when emulating the systems affected by EDFA noise. An optical bandpass filter can optionally be added after EDFA, making the simulation more flexible. Comparison with the SSFM shows that NNSpan has a distinct computational advantage, with the computation time reduced by a factor of 12. This method based on NNSpan could be a supplementary option for optical transmission system simulations, thus contributing to system designs as well.

Diagnostic and Therapeutic Approaches in Neurorehabilitation after Traumatic Brain Injury and Disorders of Consciousness

Severe traumatic brain injury (TBI) may cause disorders of consciousness (DoC) in the form of coma, unresponsive wakefulness syndrome (UWS), or minimally conscious state (MCS). Despite significant advancements made over the last two decades in detecting, predicting, and promoting the recovery of consciousness in TBI patients with DoC, the available diagnostic and treatment choices remain limited. In cases of severe TBI, the dissolution of consciousness both in the acute and post-acute phases constitutes one of the major clinical findings and challenges. In clinical settings, neurologists and neurorehabilitation specialists are called on to discern the level of consciousness in patients who are unable to communicate, and to project outcomes and recommend approaches to treatment. Standards of care are not available to guide clinical decision-making for this population, often leading to inconsistent, inaccurate, and inappropriate care. Recent studies refer to network-based mechanisms of consciousness as a more promising method to predict outcomes and functional recovery. A further goal is the modulation of neural networks underlying awareness and arousal as the main components of consciousness. This review centers on the difficulties in characterizing individuals experiencing post-traumatic DoC and on the recent advancements made in the identification and prognostication of consciousness recovery through the utilization of advanced neuroimaging and electrophysiological techniques as well as biomarkers. Moreover, we discuss new treatment approaches and summarize recent therapeutic recommendations.

DBAN: An improved dual branch attention network combined with serum Raman spectroscopy for diagnosis of diabetic kidney disease

Diabetic kidney disease (DKD) is one of the most common kidney diseases worldwide. It is estimated that approximately 537 million adults worldwide have diabetes, and up to 30%–40% of diabetic patients are at risk of developing nephropathy. The pathogenesis of DKD is complex, and its onset is insidious. Currently, the clinical diagnosis of DKD primarily relies on the increase of urinary albumin and the decrease in glomerular filtration rate in diabetic patients. However, the excretion of urinary albumin is influenced by various factors, such as physical activity, infections, fever, and high blood glucose, making it challenging to achieve an objective and accurate diagnosis. Therefore, there is an urgent need to develop an efficient, fast, and low-cost auxiliary diagnostic technology for DKD. In this study, an improved Dual Branch Attention Network (DBAN) was developed to quickly identify DKD. Serum Raman spectroscopy samples were collected from 32 DKD patients and 32 healthy volunteers. The collected data were preprocessed using the adaptive iteratively reweighted penalized least squares (airPLS) algorithm, and the DBAN was used to classify the serum Raman spectroscopy data of DKD. The model consists of a dual branch structure that extracts features using Convolutional Neural Network (CNN) and bottleneck layer modules. The attention module allows the model to learn features specifically, and lateral connections are added between the dual branches to achieve multi-level and multi-scale fusion of shallow and deep features, as well as local and global features, improving the classification accuracy of the experiment. The results of the study showed that compared to traditional deep learning algorithms such as Artificial Neural Network (ANN), CNN, GoogleNet, ResNet, and AlexNet, our proposed DBAN classification model achieved 95.4% accuracy, 98.0% precision, 96.5% sensitivity, and 97.2% specificity, demonstrating the best classification performance. This is the best method for identifying DKD, and has important reference value for the diagnosis of DKD patients, as well as improving the accuracy of medical auxiliary diagnosis.

From IMU Measurement Sequence to Velocity Estimate Sequence: An Effective and Efficient Data-Driven Inertial Odometry Approach

We propose a novel deep learning-based inertial odometry framework to improve both localization accuracy and efficiency. The measured 3D accelerations and angular velocities are constructed as two independent channel 2D features. In each channel, the 3 axes of accelerations or angular velocities constitute the height dimension, whereas the temporal axis represents the width dimension. In the neural network module, we first employ 2D convolutional operations to convert the input features to 1D series with 4-fold downsampling along the temporal axis. We then apply the convolutional attention module to refine the generated 1D feature map and long short-term memory (LSTM) layers to generate the hidden state for each refined time index, which is followed by two separate 2-layer multilayer perceptrons (MLPs) to regress the velocity and the uncertainty. During backpropagation, we use the negative log-likelihood loss to optimize the velocity and the uncertainty objectives. Benefiting from the velocity sequence objective in a single network forward propagation procedure, we also add the average squared distance between the ground-truth relative position and that of the prediction to minimize the location error. Extensive experiments on both the RoNIN dataset and the real scenario collected data in the CUHK campus show higher localization accuracy with 10 times higher efficiency of the proposed framework than the comparing methods.

Knowledge fusion enhanced graph neural network for traffic flow prediction

Traffic flow prediction is a very important and challenging task in intelligent transportation systems. There has been a lot of related research work on this issue, especially the application of graph convolutional networks has achieved quite good results. However, the existing methods usually only consider the temporal and spatial dependence in traffic data, and cannot fully explore the implicit semantic relationship from traffic knowledge. To solve this problem, we model the transportation system as topological graphs containing different types of knowledge such as network structure, regional functionality, and traffic flow patterns. We propose a Knowledge Fusion Enhanced Graph Neural Network (KFGNN) module based on multiple graph convolutional networks. Specifically, topological graphs are represented by relation matrices obtained by calculating traffic semantic similarity, and are used as the input of the Graph Convolutional Network(GCN) layer to capture the semantic dependence. The KFGNN module finally fuses these features to obtain a complex semantic representation of the traffic flow. Finally, knowledge fusion enhanced models (KE-TGCN, KE-STGCN and KE-GWN) are proposed to verify the effectiveness and versatility of this module. Experimental results on real-world datasets show that knowledge-enhanced models have higher prediction performance compared with classic GCN-based models.

Deep discriminative clustering and structural constraint for cross-domain fault diagnosis of rotating machinery

With the rapid development of intelligent manufacturing, fault diagnostic methods based on deep learning have achieved impressive results. However, most methods require plentiful annotated samples and are based on the assumption that data from the source and target domains has the same distribution. These two conditions are difficult to satisfy in practical engineering. In light of these problems, an unsupervised domain adaptation approach named Deep Discriminative Clustering network with Structural Constraint (DDCSC) is proposed in this article. In our method, a Convolutional Neural Network (CNN) module is exploited for learning feature representations of raw data. Then a softmax module is employed to simultaneously predict class probabilities and cluster assignments of the source and target data, respectively. The learnable cluster centroids are introduced into the latent feature space to alleviate the data distribution discrepancy while better capturing the discriminative structure of the target data. In addition, geometric properties of the source data in a feature space are constrained to expand the scope of each category, which facilitates to improve prediction accuracy. An information-theoretic metric is considered as the objective function of discriminative clustering. Diagnostic experiments on a rolling bearing dataset demonstrate that our approach outperforms other popular intelligent approaches and confirms the effectiveness of discriminative clustering.

Modular Clinical Decision Support Networks (MoDN)-Updatable, interpretable, and portable predictions for evolving clinical environments.

Clinical Decision Support Systems (CDSS) have the potential to improve and standardise care with probabilistic guidance. However, many CDSS deploy static, generic rule-based logic, resulting in inequitably distributed accuracy and inconsistent performance in evolving clinical environments. Data-driven models could resolve this issue by updating predictions according to the data collected. However, the size of data required necessitates collaborative learning from analogous CDSS's, which are often imperfectly interoperable (IIO) or unshareable. We propose Modular Clinical Decision Support Networks (MoDN) which allow flexible, privacy-preserving learning across IIO datasets, as well as being robust to the systematic missingness common to CDSS-derived data, while providing interpretable, continuous predictive feedback to the clinician. MoDN is a novel decision tree composed of feature-specific neural network modules that can be combined in any number or combination to make any number or combination of diagnostic predictions, updatable at each step of a consultation. The model is validated on a real-world CDSS-derived dataset, comprising 3,192 paediatric outpatients in Tanzania. MoDN significantly outperforms 'monolithic' baseline models (which take all features at once at the end of a consultation) with a mean macro F1 score across all diagnoses of 0.749 vs 0.651 for logistic regression and 0.620 for multilayer perceptron (p < 0.001). To test collaborative learning between IIO datasets, we create subsets with various percentages of feature overlap and port a MoDN model trained on one subset to another. Even with only 60% common features, fine-tuning a MoDN model on the new dataset or just making a composite model with MoDN modules matched the ideal scenario of sharing data in a perfectly interoperable setting. MoDN integrates into consultation logic by providing interpretable continuous feedback on the predictive potential of each question in a CDSS questionnaire. The modular design allows it to compartmentalise training updates to specific features and collaboratively learn between IIO datasets without sharing any data.

A lightweight temporal attention-based convolution neural network for driver's activity recognition in edge

Low inference latency and accurate response to environment changes play a crucial role in the automated driving system, especially in the current Level 3 automated driving. Achieving the rapid and reliable recognition of driver's non-driving related activities (NDRAs) is important for designing an intelligent takeover strategy that ensures a safe and quick control transition. This paper proposes a novel lightweight temporal attention-based convolutional neural network (LTA-CNN) module dedicated to edge computing platforms, specifically for NDRAs recognition. This module effectively learns spatial and temporal representations at a relatively low computational cost. Its superiority has been demonstrated in an NDRA recognition dataset, achieving 81.01% classification accuracy and an 8.37% increase compared to the best result of the efficient network (MobileNet V3) found in the literature. The inference latency has been evaluated to demonstrate its effectiveness in real applications. The latest NVIDIA Jetson AGX Orin could complete one inference in only 63 ms.

Neural module networks: A review

The capability of deep neural networks to automatically learn informative features from data is their asset. For instance, a convolutional layer learns the filters based on their placement in the architecture, i.e., the low-level features in the early layers and more abstract features in the higher level. The question is whether we can learn at the sub-task level and automate the process. In other words, can a neural architecture learn to decompose a complex task into sub-tasks (i.e., elemental tasks), solve each sub-task, and aggregate the results? This way, we gain full transparency and explainability of how a complex task has been solved. This is the goal of neural module networks (NMN). Each module represents a sub-task that conveys a symbolic meaning. Each module learns the assigned sub-task based on its placement in the modules’ layout, internal architecture, or both. The NMN re-shapes itself for each sample by choosing the sample-specific modules (i.e., sub-tasks) and placing them into an appropriate layout. This review provides a comprehensive overview of neural module networks and their applications and assumes little prior knowledge. We showcased different applications of NMNs and compared their various implementations. To better compare the performance of each application, we also chose a few non-modular approaches for the completeness of the comparisons. We hope this review and the added benefit of NMN’s explainability attract more researchers to solve the current challenges.

Online Public Transit Ridership Monitoring Through Passive WiFi Sensing

Online public transit ridership information is helpful to enhance the service quality of urban public transportation and the travel experiences of passengers. Passive WiFi sensing collects WiFi probe (request) frames sent by nearby mobile devices in a non-intrusive manner, and can thus be employed to monitor ridership. Compared with the existing non-WiFi based approaches, passive WiFi sensing based approaches demonstrate the advantages of limited interferences, large coverage, low costs and lightweight calculations. More recently, although some dedicated passive WiFi sensing based methods have been proposed in an offline mode, due to sniffing opportunistically, unknown dynamic transmission boundary, MAC randomization and the difficulty in online feature extraction, how to utilize limited sensing data to provide accurate online ridership information is still challenging. To this end, an innovative public transit ridership monitoring system built upon a customized WiFi sniffer and an online ridership estimation algorithm is developed. In the algorithm, a convolutional neural network (CNN) module and a bidirectional long short-term memory (BiLSTM) neural network module are first adopted to find correlations among inputs and capture the bidirectional time-series features, respectively; furthermore, an attention module is incorporated to determine the importance of an input sequence at different times. Real-world experiments are carried out on 8 buses corresponding to 4 bus routes in Hohhot, China. The evaluation results show that the proposed algorithm outperforms the other 4 online algorithms and the state-of-the-art offline algorithm.

Artificial neural network-based mechanical properties prediction of cellulose polyvinyl alcohol bio composite

The deliberation of the environmental effect on synthetic polymer products has effectuated the development of the latest materials obtained from natural resources. This work tested the compatibility of a bio composite of polyvinyl alcohol (PVA) and cellulose. The mechanical effect of cellulose on PVA was investigated in the Synthia module of BIOVIA material studio and Artificial neural network. The exacted shear and bulk modulus were obtained with the variation of mass fraction of cellulose from 0 to 1 mass%. At 1% mass fraction of cellulose, the mechanical properties of PVA like shear modulus and bulk modulus had increased to 401% and 130%, respectively. The dry lab values of shear and bulk modulus were further analysed through a Bi-layer Artificial Neural Network (ANN). The designed ANN successfully predicted the bulk and shear modulus values. The implemented ANN prevailed with a root mean square error of 2.0692 and 1.865 for both properties.

Predictive Precoder Design for OTFS-Enabled URLLC: A Deep Learning Approach

This paper investigates the orthogonal time frequency space (OTFS) transmission for enabling ultra-reliable low-latency communications (URLLC). To guarantee excellent reliability performance, pragmatic precoder design is an effective and indispensable solution. However, the design requires accurate instantaneous channel state information at the transmitter (ICSIT) which is not always available in practice. Motivated by this, we adopt a deep learning (DL) approach to exploit implicit features from estimated historical delay-Doppler domain channels (DDCs) to directly predict the precoder to be adopted in the next time frame for minimizing the frame error rate (FER), that can further improve the system reliability without the acquisition of ICSIT. To this end, we first establish a predictive transmission protocol and formulate a general problem for the precoder design where a closed-form theoretical FER expression is derived serving as the objective function to characterize the system reliability. Then, we propose a DL-based predictive precoder design framework which exploits an unsupervised learning mechanism to improve the practicability of the proposed scheme. As a realization of the proposed framework, we design a DDCs-aware convolutional long short-term memory (CLSTM) network for the precoder design, where both the convolutional neural network and LSTM modules are adopted to facilitate the spatial-temporal feature extraction from the estimated historical DDCs to further enhance the precoder performance. Simulation results demonstrate that the proposed scheme facilitates a flexible reliability-latency tradeoff and achieves an excellent FER performance that approaches the lower bound obtained by a genie-aided benchmark requiring perfect ICSI at both the transmitter and receiver.

Learning Decentralized Power Control in Cell-Free Massive MIMO Networks

This paper studies learning-based decentralized power control methods for cell-free massive multiple-input multiple-output (MIMO) systems where a central processor (CP) controls access points (APs) through fronthaul coordination. To determine the transmission policy of distributed APs, it is essential to develop a network-wide collaborative optimization mechanism. To address this challenge, we design a cooperative learning (CL) framework which manages computation and coordination strategies of the CP and APs using dedicated deep neural network (DNN) modules. To build a versatile learning structure, the proposed CL is carefully designed such that its forward pass calculations are independent of the number of APs. To this end, we adopt a parameter reuse concept which installs an identical DNN module at all APs. Consequently, the proposed CL trained at a particular configuration can be readily applied to arbitrary AP populations. Numerical results validate the advantages of the proposed CL over conventional non-cooperative approaches.

Intelligent Reception of Frequency Hopping Signals Based on CVDP

The frequency hopping communication systems have been widely used in anti-jamming communication due to their anti-interception and anti-interference capabilities. With the increasingly complex electromagnetic environment, the transmitter of the frequency hopping communication system adopts more flexible frequency hopping patterns to cope with interference. Therefore, to address the problem of traditional frequency hopping receiver systems being unable to receive properly due to intelligent frequency hopping sequence decisions made at the transmitter, we design a CVDP (CNN_VIT_Dual Multi-head Probsparse Self-attention) network based on a convolutional neural network (CNN) module and Transformer Encoder module to realize the intelligent reception of frequency hopping signals. The designed CNN module extracts the global features and spatial information of the time-frequency spectrograms and serves as the input of the VIT (Vision Transformer) network, which not only solves the problem that the VIT network is difficult to converge on small data sets but also improves the stability and robustness of the VIT network. In the Transformer Encoder module of the VIT network, a dual multi-head self-attention mechanism is used to extract critical time-frequency features and a Probsparse self-attention idea is introduced to reduce the computational complexity. Simulation results on hopping frequency estimation and received BER(Bit Error Rate) of CVDP networks in different interference and channel environments show that when a hopping sequence is unknown, CVDP-based intelligent hopping receiver systems can achieve nearly the same receiver performance as conventional reception systems in which a hopping sequence is known.