Sequence Dependence Research Articles

Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

Multimodal sequence analysis aims to draw inferences from visual, language, and acoustic sequences. A majority of existing works focus on the aligned fusion of three modalities to explore inter-modal interactions, which is impractical in real-world scenarios. To overcome this issue, we seek to focus on analyzing unaligned sequences, which is still relatively underexplored and also more challenging. We propose Multimodal Graph, whose novelty mainly lies in transforming the sequential learning problem into graph learning problem. The graph-based structure enables parallel computation in time dimension (as opposed to recurrent neural network) and can effectively learn longer intra- and inter-modal temporal dependency in unaligned sequences. First, we propose multiple ways to construct the adjacency matrix for sequence to perform sequence to graph transformation. To learn intra-modal dynamics, a graph convolution network is employed for each modality based on the defined adjacency matrix. To learn inter-modal dynamics, given that the unimodal sequences are unaligned, the commonly considered word-level fusion does not pertain. To this end, we innovatively devise graph pooling algorithms to automatically explore the associations between various time slices from different modalities and learn high-level graph representation hierarchically. Multimodal Graph outperforms state-of-the-art models on three datasets under the same experimental setting.

EXPLORING DEPENDENCE STRUCTURES IN FINITE EXCHANGEABLE SEQUENCES

This study delves into the exploration of dependence structures in finite exchangeable sequences, shedding light on the intricate patterns that govern the relationships between elements within such sequences. Exchangeability, a fundamental concept in probability theory, posits that the joint distribution of a sequence remains invariant under permutations of its elements. By investigating finite exchangeable sequences, this research aims to uncover and comprehend the underlying dependence structures that influence the statistical behavior of the sequence elements. We employ mathematical modeling, statistical analysis, and empirical studies to elucidate the nature and extent of dependencies in finite exchangeable sequences, contributing to a deeper understanding of their probabilistic characteristics.

Structural predictions of protein-DNA binding: MELD-DNA.

Structural, regulatory and enzymatic proteins interact with DNA to maintain a healthy and functional genome. Yet, our structural understanding of how proteins interact with DNA is limited. We present MELD-DNA, a novel computational approach to predict the structures of protein-DNA complexes. The method combines molecular dynamics simulations with general knowledge or experimental information through Bayesian inference. The physical model is sensitive to sequence-dependent properties and conformational changes required for binding, while information accelerates sampling of bound conformations. MELD-DNA can: (i) sample multiple binding modes; (ii) identify the preferred binding mode from the ensembles; and (iii) provide qualitative binding preferences between DNA sequences. We first assess performance on a dataset of 15 protein-DNA complexes and compare it with state-of-the-art methodologies. Furthermore, for three selected complexes, we show sequence dependence effects of binding in MELD predictions. We expect that the results presented herein, together with the freely available software, will impact structural biology (by complementing DNA structural databases) and molecular recognition (by bringing new insights into aspects governing protein-DNA interactions).

Single molecule studies reveal that MutS's binding characteristics are mismatch type and context dependent and correlate with mismatch repair efficiency in vivo.

During DNA replication, DNA polymerases can incorrectly incorporate a non-complementary nucleotide against the parental DNA strand, leading to mutations and ultimately debilitating diseases if left unrepaired. All organisms possess a mismatch repair system that scans, recognizes, and repairs DNA mismatches. MutS is one of the first repair proteins to arrive at a mismatch. Upon recognition, MutS exchanges bound ADP molecules to ATP, and undergoes a conformational change to form a sliding clamp and slide away from the mismatch. MutS will then recruit other repair proteins to the site and resolve the error. Previous in vivo studies from our lab suggest that the mismatch repair efficiency in E. coli is hypervariable not only for different mismatched bases, but also for their local sequence context. Deletion studies showed that MutS is responsible for the observed hypervariability. To understand the molecular mechanism underlying sequence dependence of MutS, we employed single molecule FRET (fluorescence resonance energy transfer) to directly detect MutS binding to a mismatch, sliding clamp formation, and dissociation. The binding characteristics of E. coli MutS showed a striking dependence of the mismatch type and the overall residence time of MutS on a mismatch-containing DNA in vitro was strongly correlated with the in vivo mismatch repair efficiencies. Furthermore, the sliding clamp formation yield in the presence of ATP was correlated with in vivo mismatch repair efficiencies. These data support in vivo repair trends and help clarify mechanistic basis by which mismatches are differentially repaired. Future experiments will be conducted using additional DNA mismatches with different sequence contexts to understand the generalizability of this observation.

An improved LSTM-Seq2Seq-based forecasting method for electricity load

Power load forecasting has gained considerable research interest in recent years. The power load is vulnerable to randomness and uncertainty during power grid operations. Therefore, it is crucial to effectively predict the electric load and improve the accuracy of the prediction. This study proposes a novel power load forecasting method based on an improved long short-term memory (LSTM) neural network. Thus, an long short-term memory neural network model is established for power load forecasting, which supports variable-length inputs and outputs. The conventional convolutional neural network (CNN) and recurrent neural network (RNN) cannot reflect the sequence dependence between the output labels. Therefore, the LSTM-Seq2Seq prediction model was established by combining the sequence-to-sequence (Seq2Seq) structure with that of the long short-term memory model to improve the prediction accuracy. Four prediction models, i.e., long short-term memory, deep belief network (DBN), support vector machine (SVM), and LSTM-Seq2Seq, were simulated and tested on two different datasets. The results demonstrated the effectiveness of the proposed LSTM-Seq2Seq method. In the future, this model can be extended to more prediction application scenarios.

An Effective Hybrid Deep Neural Network for Arabic Fake News Detection

Recently, the phenomenon of the spread of fake news or misinformation in most fields has taken on a wide resonance in societies. Combating this phenomenon and detecting misleading information manually is rather boring, takes a long time, and impractical. It is therefore necessary to rely on the fields of artificial intelligence to solve this problem. As such, this study aims to use deep learning techniques to detect Arabic fake news based on Arabic dataset called the AraNews dataset. This dataset contains news articles covering multiple fields such as politics, economy, culture, sports and others. A Hybrid Deep Neural Network has been proposed to improve accuracy. This network focuses on the properties of both the Text-Convolution Neural Network (Text-CNN) and Long Short-Term Memory (LSTM) architecture to produce efficient hybrid model. Text-CNN is used to identify the relevant features, whereas the LSTM is applied to deal with the long-term dependency of sequence. The results showed that when trained individually, the proposed model outperformed both the Text-CNN and the LSTM. Accuracy was used as a measure of model quality, whereby the accuracy of the Hybrid Deep Neural Network is (0.914), while the accuracy of both Text-CNN and LSTM is (0.859) and (0.878), respectively. Moreover, the results of our proposed model are better compared to previous work that used the same dataset (AraNews dataset).

End-to-End Transformer-Based Models in Textual-Based NLP

Transformer architectures are highly expressive because they use self-attention mechanisms to encode long-range dependencies in the input sequences. In this paper, we present a literature review on Transformer-based (TB) models, providing a detailed overview of each model in comparison to the Transformer’s standard architecture. This survey focuses on TB models used in the field of Natural Language Processing (NLP) for textual-based tasks. We begin with an overview of the fundamental concepts at the heart of the success of these models. Then, we classify them based on their architecture and training mode. We compare the advantages and disadvantages of popular techniques in terms of architectural design and experimental value. Finally, we discuss open research, directions, and potential future work to help solve current TB application challenges in NLP.

Predicting the Sequence Specificities of DNA-Binding Proteins by DNA Fine-Tuned Language Model With Decaying Learning Rates.

DNA-binding proteins (DBPs) play vital roles in the regulation of biological systems. Although there are already many deep learning methods for predicting the sequence specificities of DBPs, they face two challenges as follows. Classic deep learning methods for DBPs prediction usually fail to capture the dependencies between genomic sequences since their commonly used one-hot codes are mutually orthogonal. Besides, these methods usually perform poorly when samples are inadequate. To address these two challenges, we developed a novel language model for mining DBPs using human genomic data and ChIP-seq datasets with decaying learning rates, named DNA Fine-tuned Language Model (DFLM). It can capture the dependencies between genome sequences based on the context of human genomic data and then fine-tune the features of DBPs tasks using different ChIP-seq datasets. First, we compared DFLM with the existing widely used methods on 69 datasets and we achieved excellent performance. Moreover, we conducted comparative experiments on complex DBPs and small datasets. The results show that DFLM still achieved a significant improvement. Finally, through visualization analysis of one-hot encoding and DFLM, we found that one-hot encoding completely cut off the dependencies of DNA sequences themselves, while DFLM using language models can well represent the dependency of DNA sequences. Source code are available at: https://github.com/Deep-Bioinfo/DFLM.

Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation

Despite the great progress in 3D human pose estimation from videos, it is still an open problem to take full advantage of a redundant 2D pose sequence to learn representative representations for generating one 3D pose. To this end, we propose an improved Transformer-based architecture, called Strided Transformer, which simply and effectively lifts a long sequence of 2D joint locations to a single 3D pose. Specifically, a Vanilla Transformer Encoder (VTE) is adopted to model long-range dependencies of 2D pose sequences. To reduce the redundancy of the sequence, fully-connected layers in the feed-forward network of VTE are replaced with strided convolutions to progressively shrink the sequence length and aggregate information from local contexts. The modified VTE is termed as Strided Transformer Encoder (STE), which is built upon the outputs of VTE. STE not only effectively aggregates long-range information to a single-vector representation in a hierarchical global and local fashion, but also significantly reduces the computation cost. Furthermore, a full-to-single supervision scheme is designed at both full sequence and single target frame scales applied to the outputs of VTE and STE, respectively. This scheme imposes extra temporal smoothness constraints in conjunction with the single target frame supervision, improving the capability of producing smoother and more accurate 3D poses for the target frame. The proposed Strided Transformer is evaluated on two challenging benchmark datasets, Human3.6M and HumanEva-I, and achieves state-of-the-art results with fewer parameters. Code and models are available at \url{https://github.com/Vegetebird/StridedTransformer-Pose3D}.

U-Shaped Transformer With Frequency-Band Aware Attention for Speech Enhancement

Recently, Transformer shows the potential to exploit the long-range sequence dependency in speech with self-attention. It has been introduced in single channel speech enhancement to improve the accuracy of speech estimation from a noise mixture. However, the amount of information represented across attention-heads is often huge, which leads to increased computational complexity. To address this issue, the axial attention is proposed i.e., to split a 2D attention into two 1-D attentions. In this paper, we develop a new method for speech enhancement by leveraging the axial attention, where we generate time and frequency sub-attention maps by calculating the attention map along time- and frequency-axis. Different from the conventional axial attention, the proposed method provides two parallel multi-head attentions for time- and frequency-axis, respectively. Moreover, the frequency-band aware attention is proposed i.e., high frequency-band attention (HFA), and low frequency-band attention (LFA), which facilitates the exploitation of the information related to speech and noise in different frequency bands in the noisy mixture. To re-use high-resolution feature maps from the encoder, we design a U-shaped Transformer, which helps recover lost information from the high-level representations to further improve the speech estimation accuracy. Extensive experiments on four public datasets are used to demonstrate the efficacy of the proposed method. The code of the proposed method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Yukino-3/U-shaped-Transformer-SE</uri> .

Mechanistic Aspects of the Effect of Flanking Nucleotide Sequence on CPD Formation and CPD Self-Repair in DNA.

A cyclobutane pyrimidine dimer (CPD) is a photolesion which is produced by a cycloaddition reaction between two stacked pyrimidine bases upon UV light absorption. Because of its harmful effect on important cellular processes involving DNA and especially its relevance to skin cancer, the mechanisms of how a CPD is formed or repaired have been studied extensively, and it has been demonstrated that flanking nucleotide sequences play a crucial role in CPD formation or self-repair. Understanding the mechanisms behind this sequence dependence of CPD formation or self-repair is of great importance because it can give us valuable information on which sequence will be vulnerable to this DNA photodamage. This Perspective focuses on the mechanisms of how flanking nucleotide sequences affect CPD formation or self-repair, especially highlighting the role of computational studies in this field.

Refocus attention span networks for handwriting line recognition

Recurrent neural networks have achieved outstanding recognition performance for handwriting identification despite the enormous variety observed across diverse handwriting structures and poor-quality scanned documents. We initially proposed a BiLSTM baseline model with a sequential architecture well-suited for modeling text lines due to its ability to learn probability distributions over character or word sequences. However, employing such recurrent paradigms prevents parallelization and suffers from vanishing gradients for long sequences during training. To alleviate these limitations, we propose four significant contributions to this work. First, we devised an end-to-end model composed of a split-attention CNN-backbone that serves as a feature extraction method and a self-attention Transformer encoder–decoder that serves as a transcriber method to recognize handwriting manuscripts. The multi-head self-attention layers in an encoder–decoder transformer-based enhance the model’s ability to tackle handwriting recognition and learn the linguistic dependencies of character sequences. Second, we conduct various studies on transfer learning (TL) from large datasets to a small database, determining which model layers require fine-tuning. Third, we attained an efficient paradigm by combining different strategies of TL with data augmentation (DA). Finally, since the robustness of the proposed model is lexicon-free and can recognize sentences not presented in the training phase, the model is only trained on a few labeled examples with no extra cost of generating and training on synthetic datasets. We recorded comparable and outperformed Character and Word Error Rates CER/WER on four benchmark datasets to the most recent (SOTA) models.

Single Molecule FRET Analysis of CRISPR Cas9 Single Guide RNA Folding Dynamics.

CRISPR Cas9 is an RNA guided endonuclease that is part of a bacterial adaptive immune system. Single guide RNA (sgRNA) can be designed to target genomic DNA, making Cas9 a programmable DNA binding/cutting enzyme and allowing applications such as epigenome editing, controlling transcription, and targeted DNA insertion. Some of the main hurdles against an even wider adoption are off-target effects and variability in Cas9 editing outcomes. Most studies that aim to understand the mechanisms that underlie these two areas have focused on Cas9 DNA binding, DNA unwinding, and target cleavage. The assembly of Cas9 RNA ribonucleoprotein complex (RNP) precedes all these steps and includes sgRNA folding and Cas9 binding to sgRNA. We know from the crystal structure of the Cas9 RNP what the final sgRNA conformation is. However, the assembly dynamics has not been studied in detail and a better understanding of RNP assembly could lead to better-designed sgRNAs and better editing outcomes. To study this process, we developed a single molecule FRET assay to monitor the conformation of the sgRNA and the binding of Cas9 to sgRNA. We labeled the sgRNA with a donor fluorophore and an acceptor fluorophore such that when the sgRNA folds, there are changes in FRET efficiency. We measured sgRNA folding dynamics under different ion conditions, under various methods of folding (refolding vs vectorial), and with or without Cas9. sgRNA that closely mimics the sgRNA construct used for high resolution structural analysis of the Cas9-gRNA complex showed two main FRET states without Cas9, and Cas9 addition shifted the distribution toward the higher FRET state attributed to the properly assembled complex. Even in the absence of Cas9, folding the sgRNA vectorially using a superhelicase-dependent release of the sgRNA in the direction of transcription resulted in almost exclusively high FRET state. An addition of Cas9 during vectorial folding greatly reduced a slow-folding fraction. Our studies shed light on the heterogeneous folding dynamics of sgRNA and the impact of co-transcriptional folding and Cas9 binding in sgRNA folding. Further studies of sequence dependence may inform rational design of sgRNAs for optimal function.

Temporal dependencies in sequences of perceptual learning stimuli impact choice behavior in Gabor orientation discrimination task

Temporal dependencies in sequences of perceptual learning stimuli impact choice behavior in Gabor orientation discrimination task

Deciphering the mechanical code of the genome and epigenome.

Diverse DNA-deforming processes are impacted by the local mechanical and structural properties of DNA, which in turn depend on local sequence and epigenetic modifications. Deciphering this mechanical code (that is, this dependence) has been challenging due to the lack of high-throughput experimental methods. Here we present a comprehensive characterization of the mechanical code. Utilizing high-throughput measurements of DNA bendability via loop-seq, we quantitatively established how the occurrence and spatial distribution of dinucleotides, tetranucleotides and methylated CpG impact DNA bendability. We used our measurements to develop a physical model for the sequence and methylation dependence of DNA bendability. We validated the model by performing loop-seq on mouse genomic sequences around transcription start sites and CTCF-binding sites. We applied our model to test the predictions of all-atom molecular dynamics simulations and to demonstrate that sequence and epigenetic modifications can mechanically encode regulatory information in diverse contexts.

High Pressure CPMG and CEST Reveal That Cavity Position Dictates Distinct Dynamic Disorder in the PP32 Repeat Protein.

Given the central role of conformational dynamics in protein function, it is essential to characterize the time scales and structures associated with these transitions. High pressure (HP) perturbation favors transitions to excited states because they typically occupy a smaller molar volume, thus facilitating characterization of conformational dynamics. Repeat proteins, with their straightforward architecture, provide good models for probing the sequence dependence of protein conformational dynamics. Investigations of chemical exchange by 15N CPMG relaxation dispersion analysis revealed that introduction of a cavity via substitution of isoleucine 7 by alanine in the N-terminal capping motif of the pp32 leucine-rich repeat protein leads to pressure-dependent conformational exchange detected on the 500 μs-2 ms CPMG time scale. Exchange amplitude decreased from the N- to C-terminus, revealing a gradient of conformational exchange across the protein. In contrast, introduction of a cavity in the central core of pp32 via the L60A mutation led to pressure-induced exchange on a slower (>2 ms) time scale detected by 15N-CEST analysis. Excited state 15N chemical shifts indicated that in the excited state detected by HP CEST, the N-terminal region is mostly unfolded, while the core retains native-like structure. These HP chemical exchange measurements reveal that cavity position dictates exchange on distinct time scales, highlighting the subtle, yet central role of sequence in determining protein conformational dynamics.

Cardiac Phase and Flow Compensation Effects on REnal Flow and Microstructure AnisotroPy MRI in Healthy Human Kidney.

Renal diffusion-weighted imaging (DWI) involves microstructure and microcirculation, quantified with diffusion tensor imaging (DTI), intravoxel incoherent motion (IVIM), and hybrid models. A better understanding of their contrast may increase specificity. To measure modulation of DWI with cardiac phase and flow-compensated (FC) diffusion gradient waveforms. Prospective. Six healthy volunteers (ages: 22-48 years, five females), water phantom. 3-T, prototype DWI sequence with 2D echo-planar imaging, and bipolar (BP) or FC gradients. 2D Half-Fourier Single-shot Turbo-spin-Echo (HASTE). Multiple-phase 2D spoiled gradient-echo phase contrast (PC) MRI. BP and FC water signal decays were qualitatively compared. Renal arteries and velocities were visualized on PC-MRI. Systolic (peak velocity), diastolic (end stable velocity), and pre-systolic (before peak velocity) phases were identified. Following mutual information-based retrospective self-registration of DWI within each kidney, and Marchenko-Pastur Principal Component Analysis (MPPCA) denoising, combined IVIM-DTI analysis estimated mean diffusivity (MD), fractional anisotropy (FA), and eigenvalues (λi) from tissue diffusivity (Dt ), perfusion fraction (fp ), and pseudodiffusivity (Dp , Dp,axial , Dp,radial ), for each tissue (cortex/medulla, segmented on b0/FA respectively), phase, and waveform (BP, FC). Monte Carlo water diffusion simulations aided data interpretation. Mixed model regression probed differences between tissue types and pulse sequences. Univariate general linear model analysis probed variations among cardiac phases. Spearman correlations were measured between diffusion metrics and renal artery velocities. Statistical significance level was set at P < 0.05. Water BP and FC signal decays showed no differences. Significant pulse sequence dependence occurred for λ1 , λ3 , FA, Dp , fp , Dp,axial , Dp,radial in cortex and medulla, and medullary λ2 . Significant cortex/medulla differences occurred with BP for all metrics except MD (systole [P= 0.224]; diastole [P= 0.556]). Significant phase dependence occurred for Dp , Dp,axial , Dp,radial for BP and medullary λ1 , λ2 , λ3 , MD for FC. FA correlated significantly with velocity. Monte Carlo simulations indicated medullary measurements were consistent with a 34 μm tubule diameter. Cardiac gating and flow compensation modulate of measurements of renal diffusion. 2 TECHNICAL EFFICACY STAGE: 2.

Analytical Formulation and Field-Theoretic Simulation of Sequence-Specific Phase Separation of Protein-Like Heteropolymers with Short- and Long-Spatial-Range Interactions.

A theory for sequence-dependent liquid-liquid phase separation (LLPS) of intrinsically disordered proteins (IDPs) in the study of biomolecular condensates is formulated by extending the random phase approximation (RPA) and field-theoretic simulation (FTS) of heteropolymers with spatially long-range Coulomb interactions to include the fundamental effects of short-range, hydrophobic-like interactions between amino acid residues. To this end, short-range effects are modeled by Yukawa interactions between multiple nonelectrostatic charges derived from an eigenvalue decomposition of pairwise residue-residue contact energies. Chain excluded volume is afforded by incompressibility constraints. A mean-field approximation leads to an effective Flory-Huggins χ parameter, which, in conjunction with RPA, accounts for the contact-interaction effects of amino acid composition and the sequence-pattern effects of long-range electrostatics in IDP LLPS, whereas FTS based on the formulation provides full sequence dependence for both short- and long-range interactions. This general approach is illustrated here by applications to variants of a natural IDP in the context of several different amino-acid interaction schemes as well as a set of different model hydrophobic-polar sequences sharing the same composition. Effectiveness of the methodology is verified by coarse-grained explicit-chain molecular dynamics simulations.

Aptamer-Based Sandwich Assay Formats for Detection and Discrimination of Human High- and Low-Molecular-Weight uPA for Cancer Prognosis and Diagnosis.

Urokinase-type plasminogen activator (urokinase, uPA) is a frequently discussed biomarker for prognosis, diagnosis, and recurrence of cancer. In a previous study, we developed ssDNA aptamers that bind to different forms of human urokinase, which are therefore assumed to have different binding regions. In this study, we demonstrate the development of aptamer-based sandwich assays that use different combinations of these aptamers to detect high molecular weight- (HMW-) uPA in a micro titer plate format. By combining aptamers and antibodies, it was possible to distinguish between HMW-uPA and low molecular weight- (LMW-) uPA. For the best performing aptamer combination, we calculated the limit of detection (LOD) and limit of quantification (LOQ) in spiked buffer and urine samples with an LOD up to 50 ng/mL and 138 ng/mL, respectively. To show the specificity and sequence dependence of the reporter aptamer uPAapt-02-FR, we have identified key nucleotides within the sequence that are important for specific folding and binding to uPA using a fluorescent dye-linked aptamer assay (FLAA). Since uPA is a much-discussed marker for prognosis and diagnosis in various types of cancers, these aptamers and their use in a micro titer plate assay format represent a novel, promising tool for the detection of uPA and for possible diagnostic applications.

Deep learning method for analysis and segmentation of fatigue damage in X-ray computed tomography data for fiber-reinforced polymers

Fiber-reinforced polymers (FRP) are becoming increasingly widespread in safety-critical components, e.g., in rotor blades of wind turbines, due to their excellent lightweight potential. In such applications mechanical, cyclic loads occur resulting in fatigue, which is characterized by an accumulation of damage. X-ray micro computed tomography scans are used to investigate different damage states in FRP. The detection rate and robustness against human bias are improved over the frequently used threshold algorithm by deep learning. A convolutional neuronal network (CNN) segmented the damage mechanisms. Thereby, the damage evolution of interfiber failure, matrix cracks, and delamination in various levels of stiffness reduction is observed in more detail. The CNN is used to generate quantitative mechanism differentiated damage analysis, with which the layer-dependent damage evolution is observed. That includes a high dependency of the stacking sequence for the intralaminar and interlaminar development of the mentioned damage mechanisms.