Gating Units Research Articles

LGC-DBP: the method of DNA-binding protein identification based on PSSM and deep learning.

The recognition of DNA Binding Proteins (DBPs) plays a crucial role in understanding biological functions such as replication, transcription, and repair. Although current sequence-based methods have shown some effectiveness, they often fail to fully utilize the potential of deep learning in capturing complex patterns. This study introduces a novel model, LGC-DBP, which integrates Long Short-Term Memory (LSTM), Gated Inception Convolution, and Improved Channel Attention mechanisms to enhance the prediction of DBPs. Initially, the model transforms protein sequences into Position Specific Scoring Matrices (PSSM), then processed through our deep learning framework. Within this framework, Gated Inception Convolution merges the concepts of gating units with the advantages of Graph Convolutional Network (GCN) and Dilated Convolution, significantly surpassing traditional convolution methods. The Improved Channel Attention mechanism substantially enhances the model's responsiveness and accuracy by shifting from a single input to three inputs and integrating three sigmoid functions along with an additional layer output. These innovative combinations have significantly improved model performance, enabling LGC-DBP to recognize and interpret the complex relationships within DBP features more accurately. The evaluation results show that LGC-DBP achieves an accuracy of 88.26% and a Matthews correlation coefficient of 0.701, both surpassing existing methods. These achievements demonstrate the model's strong capability in integrating and analyzing multi-dimensional data and mark a significant advancement over traditional methods by capturing deeper, nonlinear interactions within the data.

Deciphering the Efficacy of No-Attention Architectures in Computed Tomography Image Classification: A Paradigm Shift

The burgeoning domain of medical imaging has witnessed a paradigm shift with the integration of AI, particularly deep learning, enhancing diagnostic precision and expediting the analysis of Computed Tomography (CT) images. This study introduces an innovative Multilayer Perceptron-driven model, DiagnosticMLP, which sidesteps the computational intensity of attention-based mechanisms, favoring a no-attention architecture that leverages Fourier Transforms for global information capture and spatial gating units for local feature emphasis. This study’s methodology encompasses a sophisticated augmentation and patching strategy at the input level, followed by a series of MLP blocks designed to extract hierarchical features and spatial relationships, culminating in a global average pooling layer before classification. Evaluated against state-of-the-art MLP-based models including MLP-Mixer, FNet, gMLP, and ResMLP across diverse and extensive CT datasets, including abdominal, and chest scans, DiagnosticMLP demonstrated a remarkable ability to converge efficiently, with competitive accuracy, F1 scores, and AUC metrics. Notably, in datasets featuring kidney and abdomen disorders, the model showcased superior generalization capabilities, underpinned by its unique design that addresses the complexity inherent in CT imaging. The findings in terms of accuracy and precision-recall balance posit DiagnosticMLP as an exceptional outperforming alternative to attention-reliant models, paving the way for streamlined, efficient, and scalable AI tools in medical diagnostics, reinforcing the potential for AI-augmented precision medicine without the dependency on attention-based architectures.

Attention-based gating units separate channels in neural radiance fields

Attention-based gating units separate channels in neural radiance fields

Research on the detection model of mental illness of online forum users based on convolutional network

Recently, there will be more than 4.62 billion social media users worldwide. A large number of users tend to publish personal emotional dynamics or express opinions on social media. These massive user data provide data support for the development of mental illness detection research and have achieved good results. However, it is difficult for current mental illness detection models to accurately identify key emotional features from a large number of posts issued by users to detect problem users. In view of the fact that the existing models cannot more accurately extract the words with high emotional contribution in the content of user posts, this paper proposes two hierarchical user post feature representation models, named Single-Gated LeakReLU-CNN (SGL-CNN) and Multi-Gated LeakyReLU-CNN (MGL-CNN). We leverage these 2 models to identify users with mental illness in online forums. For all posts published by each user within a certain time span, the model proposed in this paper can identify key emotional features in them and filter out other unimportant information as much as possible. In addition, the addition of gating units in this paper can significantly improve the performance of emotion detection tasks. The experimental results based on the task of RSDD dataset prove that the performance of the model proposed in this paper is superior to that of the existing methods.

A Novel Residual Gated Recurrent Unit Framework for Runoff Forecasting

Runoff forecasting is the key to the rational use and protection of water resources by mankind. The large-scale application of machine learning and neural networks in hydrological models has made accurate and reliable short-term runoff forecasting possible. In this paper, a novel short-term runoff forecasting framework called ResGRU Plus is proposed with gated recurrent unit (GRU) as the backbone. GRU has the characteristics of long short-term memory (LSTM) that can selectively memorize and forget information while merging gating units to reduce the amount of parameters. Residual network (ResNet) is also deeply integrated with GRU, and its unique shortcut connection effectively solves the degradation problem of traditional neural networks, making it possible to train deep neural networks based on recurrent architecture. Moreover, a lightweight attention mechanism module: squeeze-and-excitation network (SENet) is embedded in the framework. SENet explicitly models the interdependence between feature dimensions through one global average pooling layer (GAP) and two fully-connected layers, and rescales the original features through the learned weights to adaptively amplify or suppress features. Snapshot ensemble method is also used to train ResGRU Plus, which can integrate multiple homogeneous weak learners through one training process to improve the performance of the model at a small cost. In this paper, the hourly runoff of the Columbia River is used as the dataset. The Nash-Sutcliffe coefficient (NSE) of Efficiency and coefficient of determination (R2), which are two common evaluation metrics for hydrological models, are used to measure the performance of the models. Multiple sets of ablation experiments show that the proposed ResGRU Plus, which combines ResNet and attention mechanism, is able to improve depth by a factor of over 4 and accuracy by nearly 18% compared to the vanilla GRU, which further fully validates the effectiveness of combining the residual structure and attention mechanism with the recurrent architecture-based neural network and the feasibility of applying it to runoff forecasting. In addition, several sets of comparative experiments have also demonstrated the state-of-the-art performance of ResGRU Plus with significant improvement in accuracy compared to mainstream time series forecasting models.

A New Macromodeling Method Based on Deep Gated Recurrent Unit Regularized With Gaussian Dropout for Nonlinear Circuits

In this paper, for the first time, the deep gated recurrent unit (Deep GRU) is used as a new macromodeling approach for nonlinear circuits. Similar to Long Short-Term Memory (LSTM), the GRU has gating units that control the information flow and makes the network less prone to the vanishing gradient problem. Having a smaller number of gates causes GRU to have fewer parameters compared to LSTM leading to better model accuracy. Using the gates leads gradient formulations to have additive nature which helps them to be more resistant to vanishing and consequently learn long sequences of data. The proposed macromodeling method is capable of modeling nonlinear circuits more accurately and using fewer parameters compared to the conventional LSTM macromodeling method. To further improve the GRU performance, a regularization technique called Gaussian dropout is applied in this paper on deep GRU (GDGRU) to reduce the overfitting problem resulting in better test error. Additionally, the models obtained from the proposed techniques are remarkably faster than the original transistor-level models. To verify the superiority of the proposed method, time-domain modeling of three nonlinear circuits is provided. For these circuits, the comparisons of the accuracy and speed between the conventional recurrent neural network (RNN), the LSTM, and the proposed macromodeling methods are provided.

Physical Attention-Gated Spatial-Temporal Predictive Network for Weather Forecasting

Spatial-temporal sequence prediction is one of the hottest topics in the field of deep learning due to its wide range of potential applications in video-like data processing, specifically weather forecasting. Since most spatial-temporal observations evolve under physical laws, we adopt an attentional gating scheme to leverage the dynamic patterns captured by tailored convolution structures and propose a novel neural network, PastNet, to achieve accurate predictions. By highlighting useful parts of the whole feature map, the gating units help increase the efficiency of the architecture. Extensive experiments conducted on synthetic and real-world datasets reveal that PastNet bears the ability to accomplish this task with better performance than baseline methods.

MLP-JCG: Multi-Layer Perceptron With Joint-Coordinate Gating for Efficient 3D Human Pose Estimation

Various structural relations/dependencies exist among human body joints, which makes it possible to estimate 3D poses from 2D sources. The current research on 3D human pose estimation (3D-HPE for short) mainly focuses on structural information from a specific perspective. However, this information cannot facilitate 2D-to-3D pose lifting. This paper presents a novel and efficient multi-layer perceptron with a joint-coordinate gating (MLP-JCG) model, exploring and utilizing both the local and global structural information to perform 3D pose estimations. Specifically, MLP-JCG contains two independent MLP blocks, i.e., joint-mixing MLP and coordinate-mixing MLP, which solely act on the joint and coordinate axes in modelling their local structural information. For the global structural information, we first explore two kinds of global statistics from the pose matrix embeddings, which are referred to as the dynamics aggregated along the joint/coordinate axis. Then, we propose two kinds of gating units to elementwisely contextualize the features learned from MLP blocks. All the model components are designed based on MLP, making the MLP-JCG easy to implement and train. We conduct experiments on three 3D-HPE benchmarks, and the results demonstrate the superior effectiveness and efficiency of the proposed approach.

MLSFF: Multi-level structural features fusion for multi-modal knowledge graph completion.

With the rise of multi-modal methods, multi-modal knowledge graphs have become a better choice for storing human knowledge. However, knowledge graphs often suffer from the problem of incompleteness due to the infinite and constantly updating nature of knowledge, and thus the task of knowledge graph completion has been proposed. Existing multi-modal knowledge graph completion methods mostly rely on either embedding-based representations or graph neural networks, and there is still room for improvement in terms of interpretability and the ability to handle multi-hop tasks. Therefore, we propose a new method for multi-modal knowledge graph completion. Our method aims to learn multi-level graph structural features to fully explore hidden relationships within the knowledge graph and to improve reasoning accuracy. Specifically, we first use a Transformer architecture to separately learn about data representations for both the image and text modalities. Then, with the help of multimodal gating units, we filter out irrelevant information and perform feature fusion to obtain a unified encoding of knowledge representations. Furthermore, we extract multi-level path features using a width-adjustable sliding window and learn about structural feature information in the knowledge graph using graph convolutional operations. Finally, we use a scoring function to evaluate the probability of the truthfulness of encoded triplets and to complete the prediction task. To demonstrate the effectiveness of the model, we conduct experiments on two publicly available datasets, FB15K-237-IMG and WN18-IMG, and achieve improvements of 1.8 and 0.7%, respectively, in the Hits@1 metric.

GBC: An Energy-Efficient LSTM Accelerator With Gating Units Level Balanced Compression Strategy

Recurrent Neural Networks (RNNs) have emerged as one of the most popular neural networks for processing time-series problems, widely used in machine translation, automatic speech recognition, and other natural language processing applications. However, conventional RNNs suffered from vanishing and exploding gradients, resulting in poor network performance in applications with long-term input information. As a variant of RNN, Long Short-Term Memory (LSTM) had been proposed to tackle this issue. Nevertheless, at the same time, LSTM introduces gating units and many additional parameters, which makes it challenging to be implemented directly on resource-limited platforms, such as Field Programmable Gate Arrays (FPGAs). This work first investigated the overall maximum achievable compression rates of different gating units and their correlations. Then, Gating Units Level Balanced Compression (GBC) strategy is proposed. After Top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> pruning, the proposed GBC strategy can attain a compression rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$36.6\times $ </tex-math></inline-formula> for LSTM. Further, the theoretical analysis indicates that for the existing gating units level LSTM compression variants, the GBC strategy still has further potential for compression. A complementary compression of the GBC strategy is performed on the existing coupled-gate LSTM to verify the analysis. Experimental results show that GBC achieves an additional <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$32\times $ </tex-math></inline-formula> (overall <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$42.7\times $ </tex-math></inline-formula> ) compression rate with negligible accuracy loss. Finally, hardware experiments conducted on Xilinx ADM-PCIE-7V3 FPGAs also demonstrate that the accelerator designed in this paper achieves an improvement of 7.4%-191.5% in energy efficiency compared to the state-of-the-art designs.

ST-MGAT:Spatio-temporal multi-head graph attention network for Traffic prediction

Traffic flow prediction is an important part of the intelligent transportation system, and accurate traffic flow prediction can better perform the data management and control of the urban road network. Traditional methods often ignore the interaction between traffic flow factors and the spatiotemporal dependence of traffic networks. In this paper, a spatiotemporal multi-head graph attention network (ST-MGAT) is proposed for the traffic flow prediction task. In the input layer, multiple traffic flow variables are used as input to learn the nonlinearity and complexity existing in it. In terms of modeling, the structure of linear gating units is transformed using the full volume and applied to correlation learning in the temporal dimension. The spatial dimension features are then captured using a multi-head attention map network. Ablation experiments are designed to verify the use effect of each module. At the same time, several sets of parameter experiments are designed to obtain the best model parameters. The experimental results show that compared with the baselines, the RMSE of the ST-MGAT model is decreased by almost 2.340%∼22.471%, and the MAE is decreased by almost 7.417%∼26.142%. It is proved that the ST-MGAT model has high performance on the traffic flow prediction task.

Adaptive Intelligent Sliding Mode Control of a Dynamic System with a Long Short-Term Memory Structure

In this work, a novel fuzzy neural network (NFNN) with a long short-term memory (LSTM) structure was derived and an adaptive sliding mode controller, using NFNN (ASMC-NFNN), was developed for a class of nonlinear systems. Aimed at the unknown uncertainties in nonlinear systems, an NFNN was designed to estimate unknown uncertainties, which combined the advantages of fuzzy systems and neural networks, and also introduced a special LSTM recursive structure. The special three gating units in the LSTM structure enabled it to have selective forgetting and memory mechanisms, which could make full use of historical information, and have a stronger ability to learn and estimate unknown uncertainties than general recurrent neural networks. The Lyapunov stability rule guaranteed the parameter convergence of the neural network and system stability. Finally, research into a simulation of an active power filter system showed that the proposed new algorithm had better static and dynamic properties and robustness compared with a sliding controller that uses a recurrent fuzzy neural network (RFNN).

Dynamic Embedding Projection-Gated Convolutional Neural Networks for Text Classification

Text classification is a fundamental and important area of natural language processing for assigning a text into at least one predefined tag or category according to its content. Most of the advanced systems are either too simple to get high accuracy or centered on using complex structures to capture the genuinely required category information, which requires long time to converge during their training stage. In order to address such challenging issues, we propose a dynamic embedding projection-gated convolutional neural network (DEP-CNN) for multi-class and multi-label text classification. Its dynamic embedding projection gate (DEPG) transforms and carries word information by using gating units and shortcut connections to control how much context information is incorporated into each specific position of a word-embedding matrix in a text. To our knowledge, we are the first to apply DEPG over a word-embedding matrix. The experimental results on four known benchmark datasets display that DEP-CNN outperforms its recent peers.

An Energy Efficient Accelerator for Bidirectional Recurrent Neural Networks (BiRNNs) Using Hybrid-Iterative Compression With Error Sensitivity

Recurrent Neural Networks (RNNs) have been widely used in many sequential applications, such as machine translation, speech recognition and sentiment analysis. Long Term Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) are widely used variants of RNN due to their effectiveness in overcoming gradient vanishing and exploding problems; however, compared to conventional RNN, their massive storage and computation requirements hinder their application. In addition, the recurrent structure of RNNs makes them prone to accumulate errors, resulting in a severe loss of accuracy. In this work, we propose a hybrid-iterative compression (HIC) algorithm for LSTM/GRU. By exploiting the error sensitivity of RNN, the gating units are divided into error-sensitive and error-insensitive groups, that are compressed using different algorithms. By using this approach, a 37.1×/32.3× compression ratio is achieved with negligible accuracy loss for LSTM/GRU. Further, an energy efficient accelerator for bidirectional RNNs is proposed. In this accelerator, the data flow of the matrix operation unit based on the block structure matrix (MOU-S) is improved through rearranging weights; the utilization of BRAM is improved through a fine-grained parallelism configuration of matrix-vector multiplications (MVMs). Meanwhile, the timing matching strategy alleviates the load-imbalance problem between MOU-S and the matrix operation unit based on top- k pruning (MOU-P). When running at 200MHz on Xilinx ADM-PCIE-7V3 FPGA, the proposed design achieves an improvement in energy efficiency in a range of 5%-237% for LSTM networks, and an improvement of 58% for GRU networks compared with state-of-the-art designs.

PH- and miRNA-Responsive DNA-Tetrahedra/Metal-Organic Framework Conjugates: Functional Sense-and-Treat Carriers.

The synthesis of stimuli-responsive hybrid structures composed of drug-loaded UiO-66 metal-organic framework nanoparticles, NMOFs, locked by DNA tetrahedra gates is presented. The hybrid systems combine the high loading capacity of drugs in the porous NMOFs and the effective cell permeation properties of the DNA tetrahedra. The nucleic acid-functionalized UiO-66 NMOFs are loaded with drugs (doxorubicin, DOX, or camptothecin, CPT) or with dyes as drug models (Rhodamine 6G or fluorescein) and used to prepare stimuli-responsive carriers. In this study, two different stimuli-responsive NMOFs are presented. One system introduces the drug-loaded NMOFs locked by pH-responsive DNA tetrahedra. At acidic pH values, the gating tetrahedra are dissociated from the NMOFs through the formation of i-motif structures, resulting in the unlocking of the NMOFs and the release of the drugs. In addition, the tetrahedra gates are modified with AS1411 aptamer tethers, and these target the drug-loaded NMOFs to nucleolin receptors overexpressed in certain malignant cells. A second system involves the preparation of NMOFs loaded with drugs/dyes and gated by the microRNA (miRNA)-responsive tetrahedra (miRNA-21 or miRNA-155). In the presence of miRNAs, the dissociation of miRNA-responsive tetrahedra from the NMOFs leads to the unlocking of the NMOFs and the release of the loads. Further developments of the miRNA-responsive tetrahedra-gated hybrid carriers include the following. (i) By appropriate engineering of the miRNA gating units, the exonuclease III (Exo III)-amplified unlocking of the carriers, through the regeneration of the miRNA triggers, and the enhanced release of the loaded drugs are demonstrated. (ii) By applying mixtures of miRNA-21-responsive DNA tetrahedra-gated DOX-loaded NMOFs and miRNA-155-responsive DNA tetrahedra-gated CPT-loaded NMOFs, the multiplexed miRNA-21/miRNA-155-dictated release of the drugs is demonstrated. As compared to the analog DNA duplex-modified NMOFs, DNA tetrahedra-gated, drug-loaded NMOFs permeation into malignant MDA-MB-231 breast cancer cells presents more effective cell permeation. Effective and selective cytotoxicity toward the malignant cells, as compared to nonmalignant epithelial MCF-10A breast cells, is demonstrated due to the acidic pH, present in cancer cells, or the miRNA-21, present in MDA-MB-231 malignant cells.

On-Nanoparticle Gating Units Render an Ordinary Catalyst Substrate- and Site-Selective.

When an organometallic catalyst is tethered onto a nanoparticle and is embedded in a monolayer of longer ligands terminated in "gating" end-groups, these groups can control the access and orientation of the incoming substrates. In this way, a nonspecific catalyst can become enzyme-like: it can select only certain substrates from substrate mixtures and, quite remarkably, can also preorganize these substrates such that only some of their otherwise equivalent sites react. For a simple, copper-based click reaction catalyst and for gating ligands terminated in charged groups, both substrate- and site-selectivities are on the order of 100, which is all the more notable given the relative simplicity of the on-particle monolayers compared to the intricacy of enzymes' active sites. The strategy of self-assembling macromolecular, on-nanoparticle environments to enhance selectivities of "ordinary" catalysts presented here is extendable to other types of catalysts and gating based on electrostatics, hydrophobicity, and chirality, or the combinations of these effects. Rational design of such systems should be guided by theoretical models we also describe.

Aptamer-Functionalized Micro- and Nanocarriers for Controlled Release.

Sequence-specific nucleic acids recognizing low-molecular-weight ligands or macromolecules (aptamers) have found growing interest for biomedical applications. The present review article summarizes recent applications of aptamers as stimuli-responsive gating units of drug (or dye)-loaded nano- or microcarriers for controlled and targeted drug release. In the presence of cellular biomarkers, the nano-/microcarriers are unlocked by forming aptamer-ligand complexes. Different aptamer-functinalized nano-/microcarriers are presented, including inorganic nanomaterials, metal-organic framework nanoparticles, and soft materials. The chemistries associated with the preparation of the carriers and the mechanisms to unlock the carriers are discussed. Stimuli-responsive gated drug-loaded micro-/nanocarriers hold great promise as functional sense-and-treat materials for the targeted and selective release of drugs.

Second-Order Sensitivity Methods for Robustly Training Recurrent Neural Network Models

Adjoint methods are used in both control theory and machine learning (ML) to efficiently compute gradients of functionals. In ML, the adjoint method is a popular approach for training multilayer neural networks and is commonly referred to as backpropagation. Despite its importance in ML, the adjoint method suffers from two well documented shortcomings: (i) gradient decay/explosion and (ii) excessive training time. Until now, the gradient decay problem has primarily been addressed through modification to the network architecture with gating units that add additional parameters. This results in additional computational costs during evaluation and training which further exacerbates the excessive training time. In this letter, we introduce a powerful framework for addressing the gradient decay problem based on second-order sensitivity concepts from control theory. As a result, we are able to robustly train arbitrary network architectures without suffering from gradient decay. Furthermore, we demonstrate that this method is able to speed up training with respect to both wall-clock time and data efficiency. We demonstrate our method on a synthetic long time gap experiment, as well as three sequential modeling benchmarks with a simple recurrent neural network architecture.

Functional DNA Structures and Their Biomedical Applications

Since the discovery of the double-helix structure in 1953, nucleic acids have been developed from natural genetic codes into functional building blocks in a wide range of biotechnology and material...

A neural circuit model for a contextual association task inspired by recommender systems.

Behavioral data shows that humans and animals have the capacity to learn rules of associations applied to specific examples, and generalize these rules to a broad variety of contexts. This article focuses on neural circuit mechanisms to perform a context-dependent association task that requires linking sensory stimuli to behavioral responses and generalizing to multiple other symmetrical contexts. The model uses neural gating units that regulate the pattern of physiological connectivity within the circuit. These neural gating units can be used in a learning framework that performs low-rank matrix factorization analogous to recommender systems, allowing generalization with high accuracy to a wide range of additional symmetrical contexts. The neural gating units are trained with a biologically inspired framework involving traces of Hebbian modification that are updated based on the correct behavioral output of the network. This modeling demonstrates potential neural mechanisms for learning context-dependent association rules and for the change in selectivity of neurophysiological responses in the hippocampus. The proposed computational model is evaluated using simulations of the learning process and the application of the model to new stimuli. Further, human subject behavioral experiments were performed and the results validate the key observation of a low-rank synaptic matrix structure linking stimuli to responses.