Representation Of Sequences Research Articles

Influence of process-based, stochastic and deterministic methods for representing heterogeneity in fluvial geothermal systems

Focus is on comparing stochastic, process-based and deterministic methods for modelling heterogeneity in hydraulic properties of fluvial geothermal reservoirs. Models are considered a generalized representation of a fluvial sequence in the upper part of the Gassum Formation in northern Denmark. Two ensemble realizations of process-based and stochastic heterogeneity were simulated using the state-of-the-art numerical modelling software, Delft Advanced Research Terra Simulator (DARTS), to assess differences on a statistically relevant sample size. Simulator settings were optimized to achieve two orders of magnitude improvement in simulation time. Our general findings show that the stochastic and process-based methods produce nearly identical results in terms of predicted breakthrough time and production temperature decline for high net-to-gross ratios (N/G). Simple homogenous and layered models overestimate breakthrough and underestimate temperature decline. More complex representation of facies in process-based models show smaller variance in results and stay within the limits of ensemble runs compared to simpler facies representation. Results indicate that a multivariate Gaussian based stochastic representation of heterogeneity provides comparable thermal response to a process-based model in fluvial systems of similar quality.

The Frobenius number associated with the number of representations for sequences of repunits

The Frobenius number associated with the number of representations for sequences of repunits

Automatic monitoring and detection of tail-biting behavior in groups of pigs using video-based deep learning methods.

Automated monitoring of pigs for timely detection of changes in behavior and the onset of tail biting might enable farmers to take immediate management actions, and thus decrease health and welfare issues on-farm. Our goal was to develop computer vision-based methods to detect tail biting in pigs using a convolutional neural network (CNN) to extract spatial information, combined with secondary networks accounting for temporal information. Two secondary frameworks were utilized, being a long short-term memory (LSTM) network applied to sequences of image features (CNN-LSTM), and a CNN applied to image representations of sequences (CNN-CNN). To achieve our goal, this study aimed to answer the following questions: (a) Can the methods detect tail biting from video recordings of entire pens? (b) Can we utilize principal component analyses (PCA) to reduce the dimensionality of the feature vector and only use relevant principal components (PC)? (c) Is there potential to increase performance in optimizing the threshold for class separation of the predicted probabilities of the outcome? (d) What is the performance of the methods with respect to each other? The study utilized one-hour video recordings of 10 pens with pigs prior to weaning, containing a total of 208 tail-biting events of varying lengths. The pre-trained VGG-16 was used to extract spatial features from the data, which were subsequently pre-processed and divided into train/test sets before input to the LSTM/CNN. The performance of the methods regarding data pre-processing and model building was systematically compared using cross-validation. Final models were run with optimal settings and evaluated on an independent test-set. The proposed methods detected tail biting with a major-mean accuracy (MMA) of 71.3 and 64.7% for the CNN-LSTM and the CNN-CNN network, respectively. Applying PCA and using a limited number of PCs significantly increased the performance of both methods, while optimizing the threshold for class separation did result in a consistent but not significant increase of the performance. Both methods can detect tail biting from video data, but the CNN-LSTM was superior in generalizing when evaluated on new data, i.e., data not used for training the models, compared to the CNN-CNN method.

Curriculum Pre-training Heterogeneous Subgraph Transformer for Top- N Recommendation

To characterize complex and heterogeneous side information in recommender systems, the heterogeneous information network (HIN) has shown superior performance and attracted much research attention. In HIN, the rich entities, relations, and paths can be utilized to model the correlations of users and items; such a task setting is often called HIN-based recommendation . Although HIN provides a general approach to modeling rich side information, it lacks special consideration on the goal of the recommendation task. The aggregated context from the heterogeneous graph is likely to incorporate irrelevant information, and the learned representations are not specifically optimized according to the recommendation task. Therefore, there is a need to rethink how to leverage the useful information from HIN to accomplish the recommendation task. To address the above issues, we propose a Curriculum pre-training based HEterogeneous Subgraph Transformer (called CHEST ) with new data characterization , representation model, and learning algorithm . Specifically, we consider extracting useful information from HIN to compose the interaction-specific heterogeneous subgraph, containing highly relevant context information for recommendation. Then, we capture the rich semantics (e.g., graph structure and path semantics) within the subgraph via a heterogeneous subgraph Transformer, where we encode the subgraph into multi-slot sequence representations. Besides, we design a curriculum pre-training strategy to provide an elementary-to-advanced learning process. The elementary course focuses on capturing local context information within the subgraph, and the advanced course aims to learn global context information. In this way, we gradually capture useful semantic information from HIN for modeling user-item interactions. Extensive experiments conducted on four real-world datasets demonstrate the superiority of our proposed method over a number of competitive baselines, especially when only limited training data is available.

Online portfolio management via deep reinforcement learning with high-frequency data

Recently, models that based on Transformer (Vaswani et al., 2017) have yielded superior results in many sequence modeling tasks. The ability of Transformer to capture long-range dependencies and interactions makes it possible to apply it in the field of portfolio management (PM). However, the built-in quadratic complexity of the Transformer prevents its direct application to the PM task. To solve this problem, in this paper, we propose a deep reinforcement learning-based PM framework called LSRE-CAAN, with two important components: a long sequence representations extractor and a cross-asset attention network. Direct Policy Gradient is used to solve the sequential decision problem in the PM process. We conduct numerical experiments in three aspects using four different cryptocurrency datasets, and the empirical results show that our framework is more effective than both traditional and state-of-the-art (SOTA) online portfolio strategies, achieving a 6x return on the best dataset. In terms of risk metrics, our framework has an average volatility risk of 0.46 and an average maximum drawdown risk of 0.27 across the four datasets, both of which are lower than the vast majority of SOTA strategies. In addition, while the vast majority of SOTA strategies maintain a poor turnover rate of approximately greater than 50% on average, our framework enjoys a relatively low turnover rate on all datasets, efficiency analysis illustrates that our framework no longer has the quadratic dependency limitation.

Identification of Phase Separating Proteins With Distributed Reduced Alphabet Representations of Sequences.

Phase separation of proteins play key roles in cellular physiology including bacterial division, tumorigenesis etc. Consequently, understanding the molecular forces that drive phase separation has gained considerable attention and several factors including hydrophobicity, protein dynamics, etc., have been implicated in phase separation. Data-driven identification of new phase separating proteins can enable in-depth understanding of cellular physiology and may pave way towards developing novel methods of tackling disease progression. In this work, we exploit the existing wealth of data on phase separating proteins to develop sequence-based machine learning method for prediction of phase separating proteins. We use reduced alphabet schemes based on hydrophobicity and conformational similarity along with distributed representation of protein sequences and biochemical properties as input features to Support Vector Machine (SVM) and Random Forest (RF) machine learning algorithms. We used both curated and balanced dataset for building the models. RF trained on balanced dataset with hydropathy, conformational similarity embeddings and biochemical properties achieved accuracy of 97%. Our work highlights the use of conformational similarity, a feature that reflects amino acid flexibility, and hydrophobicity for predicting phase separating proteins. Use of such "interpretable" features obtained from the ever-growing knowledgebase of phase separation is likely to improve prediction performances further.

Generating Natural Language From Logic Expressions With Structural Representation

Incorporating logic reasoning with deep neural networks (DNNs) is an important challenge in machine learning. In this article, we study the problem of converting logical expressions into natural language. In particular, given a sequential logic expression, the goal is to generate its corresponding natural sentence. Since the information in a logic expression often has a hierarchical structure, a sequence-to-sequence baseline struggles to capture the full dependencies between words, and hence it often generates incorrect sentences. To alleviate this problem, we propose a model to convert Structural Logic Expressions into Natural Language (SLEtoNL). SLEtoNL converts sequential logic expressions into structural representation and leverages structural encoders to capture the dependencies between nodes. The quantitative and qualitative analyses demonstrate that our proposed method outperforms the seq2seq model, which is based on the sequential representation, and outperforms strong pretrained language models ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , T5, BART, GPT3) with a large margin (28.6 in BLEU3) in out-of-distribution evaluation. The data and code will be available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com</uri> .

The Influence of Spatial-Numerical Associations to Visual-Spatial Attention—Based on Magnitude or Sequential Representation?

The Influence of Spatial-Numerical Associations to Visual-Spatial Attention—Based on Magnitude or Sequential Representation?

Sequence Representations and Their Utility for Predicting Protein-Protein Interactions.

Protein-Protein Interactions (PPIs) are a crucial mechanism underpinning the function of the cell. So far, a wide range of machine-learning based methods have been proposed for predicting these relationships. Their success is heavily dependent on the construction of the underlying feature vectors, with most using a set of physico-chemical properties derived from the sequence. Few work directly with the sequence itself. In this paper, we explore the utility of sequence embeddings for predicting protein-protein interactions. We construct a protein pair feature vector by concatenating the embeddings of their constituent sequence. These feature vectors are then used as input to a binary classifier to make predictions. To learn sequence embeddings, we use two established Word2Vec based methods - Seq2Vec and BioVec - and we also introduce a novel feature construction method called SuperVecNW. The embeddings generated through SuperVecNW capture some network information in addition to the contextual information present in the sequences. We test the efficacy of our proposed approach on human and yeast PPI datasets and on three well-known networks: CD9, the Ras-Raf-Mek-Erk-Elk-Srf pathway, and a Wnt-related network. We demonstrate that low dimensional sequence embeddings provide better results than most alternative representations based on physico-chemical properties while offering a far simple approach to feature vector construction.

Knowledge Guided Attention and Graph Convolutional Networks for Chemical-Disease Relation Extraction.

The automatic extraction of the chemical-disease relation (CDR) from the text becomes critical because it takes a lot of time and effort to extract valuable CDR manually. Studies have shown that prior knowledge from the biomedical knowledge base is important for relation extraction. The method of combining deep learning models with prior knowledge is worthy of our study. In this paper, we propose a new model called Knowledge Guided Attention and Graph Convolutional Networks (KGAGN) for CDR extraction. First, to make full advantage of domain knowledge, we train entity embedding as a feature representation of input sequence, and relation embedding to capture weighted contextual information further through the attention mechanism. Then, to make full advantage of syntactic dependency information in cross-sentence CDR extraction, we construct document-level syntactic dependency graphs and encode them using a graph convolution network (GCN). Finally, the chemical-induced disease (CID) relation is extracted by using weighted context features and long-range dependency features both of which contain additional knowledge information We evaluated our model on the CDR dataset published by the BioCreative-V community and achieves an F1-score of 73.3%, surpassing other state-of-the-art methods. the code implemented by PyTorch 1.7.0 deep learning library can be downloaded from Github: https://github.com/sunyi123/cdr.

RPPSP: A Robust and Precise Protein Solubility Predictor by Utilizing Novel Protein Sequence Encoder

Protein solubility prediction is essential to understand diverse types of biological processes and to explore the impact of different factors (ionic strength, temperature, PH of medium and electrostatic repulsion) on the productivity of proteins. It also plays an important role in disease analysis and drug development processes. Protein solubility prediction through experimental approaches is time-consuming, labour intensive and error-prone. To empower the process of protein solubility prediction and facilitate large scale analysis, 16 different computational predictors have been proposed. However, these predictors have low predictive performance mainly due to extraction of less semantic and discriminative features from raw protein sequences. Existing predictors either extract sequence order information or positional information, while both types of information are important to discriminate soluble and insoluble proteins. This paper presents a novel encoder CTAPAAC capable of generating statistical representations of protein sequences by extracting 4 different types of information correlation, distribution, composition and transition. Over 4 benchmark datasets a comprehensive intrinsic and extrinsic performance analysis of proposed and 14 most widely used existing protein sequence encoders reveals that proposed encoder has more potential in transforming soluble and insoluble protein sequences into statistical vectors having discriminative patterns among soluble and insoluble classes. Proposed encoder along with random forest classifier outperforms existing best performing protein solubility predictors with a significant margin of 6%, 7%, 25% and 10% over PSI:Biology, E.coli, price and Esol datasets in terms of accuracy. Source code of proposed predictor is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Faiza-Mehmood/RPPSP</uri> .

Pose-Appearance Relational Modeling for Video Action Recognition.

Recent studies of video action recognition can be classified into two categories: the appearance-based methods and the pose-based methods. The appearance-based methods generally cannot model temporal dynamics of large motion well by virtue of optical flow estimation, while the pose-based methods ignore the visual context information such as typical scenes and objects, which are also important cues for action understanding. In this paper, we tackle these problems by proposing a Pose-Appearance Relational Network (PARNet), which models the correlation between human pose and image appearance, and combines the benefits of these two modalities to improve the robustness towards unconstrained real-world videos. There are three network streams in our model, namely pose stream, appearance stream and relation stream. For the pose stream, a Temporal Multi-Pose RNN module is constructed to obtain the dynamic representations through temporal modeling of 2D poses. For the appearance stream, a Spatial Appearance CNN module is employed to extract the global appearance representation of the video sequence. For the relation stream, a Pose-Aware RNN module is built to connect pose and appearance streams by modelling action-sensitive visual context information. Through jointly optimizing the three modules, PARNet achieves superior performances compared with the state-of-the-arts on both the pose-complete datasets (KTH, Penn-Action, UCF11) and the challenging pose-incomplete datasets (UCF101, HMDB51, JHMDB), demonstrating its robustness towards complex environments and noisy skeletons. Its effectiveness on NTU-RGBD dataset is also validated even compared with 3D skeleton-based methods. Furthermore, an appearance-enhanced PARNet equipped with a RGB-based I3D stream is proposed, which outperforms the Kinetics pre-trained competitors on UCF101 and HMDB51. The better experimental results verify the potentials of our framework by integrating various modules.

IEdgeDTA: integrated edge information and 1D graph convolutional neural networks for binding affinity prediction.

Artificial intelligence has become more prevalent in broad fields, including drug discovery, in which the process is costly and time-consuming when conducted through wet experiments. As a result, drug repurposing, which tries to utilize approved and low-risk drugs for a new purpose, becomes more attractive. However, screening candidates from many drugs for specific protein targets is still expensive and tedious. This study aims to leverage computational resources to aid drug discovery by utilizing drug-protein interaction data and estimating their interaction strength, so-called binding affinity. Our estimation approach addresses multiple challenges encountered in the field. First, we employed a graph-based deep learning technique to overcome the limitations of drug compounds represented in string format by incorporating background knowledge of node and edge information as separate multi-dimensional features. Second, we tackled the complexities associated with extracting the representation and structure of proteins by utilizing a pre-trained model for feature extraction. Also, we employed graph operations over the 1D representation of a protein sequence to overcome the fixed-length problem typically encountered in language model tasks. In addition, we conducted a comparative analysis with a baseline model that creates a protein graph from a contact map prediction model, giving valuable insights into the performance and effectiveness of our proposed method. We evaluated the performance of our model using the same benchmark datasets with a variety of matrices as other previous work, and the results show that our model achieved the best prediction results while requiring no contact map information compared to other graph-based methods.

CATHe: detection of remote homologues for CATH superfamilies using embeddings from protein language models.

CATH is a protein domain classification resource that exploits an automated workflow of structure and sequence comparison alongside expert manual curation to construct a hierarchical classification of evolutionary and structural relationships. The aim of this study was to develop algorithms for detecting remote homologues missed by state-of-the-art hidden Markov model (HMM)-based approaches. The method developed (CATHe) combines a neural network with sequence representations obtained from protein language models. It was assessed using a dataset of remote homologues having less than 20% sequence identity to any domain in the training set. The CATHe models trained on 1773 largest and 50 largest CATH superfamilies had an accuracy of 85.6 ± 0.4% and 98.2 ± 0.3%, respectively. As a further test of the power of CATHe to detect more remote homologues missed by HMMs derived from CATH domains, we used a dataset consisting of protein domains that had annotations in Pfam, but not in CATH. By using highly reliable CATHe predictions (expected error rate <0.5%), we were able to provide CATH annotations for 4.62 million Pfam domains. For a subset of these domains from Homo sapiens, we structurally validated 90.86% of the predictions by comparing their corresponding AlphaFold2 structures with structures from the CATH superfamilies to which they were assigned. The code for the developed models is available on https://github.com/vam-sin/CATHe, and the datasets developed in this study can be accessed on https://zenodo.org/record/6327572. Supplementary data are available at Bioinformatics online.

NCYPred: A Bidirectional LSTM Network With Attention for Y RNA and Short Non-Coding RNA Classification.

Short non-coding RNAs (sncRNAs) are involved in multiple cellular processes and can be divided into dozens of classes. Among such classes, Y RNAs have been gaining attention, being essential factors for the initiation of DNA replication on vertebrates, as well as potential tumor biomarkers. Homologs have also been described in nematodes and insects, as well as related sequences in bacteria. Methods capable of accurately predicting Y RNA transcripts are lacking. In this work, we developed an attention-based LSTM network and built a classification model able to classify sncRNAs (including Y RNA) directly from nucleotide sequences. A dataset consisting of 45,447 sncRNA sequences, from a wide range of organisms, obtained from Rfam 14.3 was built. Performance evaluation demonstrated that our proposed method, NCYPred (Non-Coding/Y RNA Prediction), can accurately predict Y RNA sequences and their homologs, as well as 11 additional classes, achieving results comparable with state-of-the-art methods. We also demonstrate that applying t-SNE on learned sequence representations could be useful for sequence analysis. Our model is freely available as a web-server (https://www.gpea.uem.br/ncypred/).

Recognition of human emotion transition from video sequence using triangulation induced various centre pairs distance signatures

Human emotion recognition manifests the concept of contextual understanding of human communication in nonverbal form. This fascinates us to monitor the dynamic information of an emotion. In this article, we present an automatic recognition system for observing gradual changes in human emotion available in the form of a video sequence. In our proposed system, we introduce a triangulation mechanism for geometric feature generation from salient landmarks on the face image. Here, well known active appearance model (AAM) is applied to consecutive image frames in a sequence to track all locations of landmark points. In this work, we explore four centre points (Incentre, Circumcentre, Barycentre, and Orthocentre) from all the triangles formed by salient facial points selected in a triplet. Six different geometric distance signatures viz. Incentre–Circumcentre, Incentre–Barycentre, Incentre–Orthocentre, Circumcentre–Barycentre, Circumcentre–Orthocentre, and Barycentre–Orthocentre are extracted accordingly by taking all possible pairs of those four centres into account to signify several ways of the geometric representation of an emotional image sequence. The discriminative merit of each distance signature is analysed independently by employing them separately as input feature sets into a multilayer perceptron (MLP) classifier for recognition of six basic emotions(anger, disgust, fear, happiness, sadness, and surprise). We verify the performance of our proposed system through the experimentation on several publically available benchmark video datasets such as Extended Cohn–Kanade (CK＋), M&M Initiative (MMI), and Multimedia Understanding Group (MUG). Experimental results indicate impressive accuracy of 98.47% for CK＋, 92.57% for MMI, and 98.62% for the MUG dataset on Incentre–Circumcentre pair distance signature. Moreover, to justify the effectiveness of our proposed method we compare our experimental results with other existing states of the art available in the literature.

A novel medical text classification model with Kalman filter for clinical decision making

Medical text classification has long been a challenging issue in the construction of clinical decision support systems because medical texts contain medical terminologies and various medical abbreviations. This paper proposes an ALBERT-based fusion Kalman-filter model, named AFKF, to address word-level and sentence-level noises in electronic medical records. Specifically, a sliding window scheme is explored to deal with the coupling relationship among large sequences. Furthermore, we design a fusion block on the basis of Kalman filter to integrate representations of multiple segment sequences. Experiments indicate that our approach significantly outperforms the baseline methods with real-world medical texts. Our fusion strategy also improves the performance of other feature classification models in the medical text classification task.

Time course of learning sequence representations in action imagery practice

Action imagery practice (AIP) is effective to improve motor performance in a variety of tasks, though it is often less effective than action execution practice (AEP). In sequence learning, AIP and AEP result in the acquisition of effector-independent representations. However, it is unresolved whether effector-dependent representations can be acquired in AIP. In the present study, we investigated the acquisition of effector-independent representations and effector-dependent representations in AEP and AIP in an implicit sequence learning task (a visual serial-reaction-time task, involving a twelve-element sequence). Participants performed six sessions, each starting with tests. A practice sequence, a mirror sequence, and a different sequence were tested with the practice and transfer hand. In the first four sessions, after the tests, two groups performed either AIP (N = 50) or AEP (N = 54). Improvement in the different sequence indicated sequence-unspecific learning in both AEP and AIP. Importantly, reaction times of the practice hand became shorter in the practice sequence than in the other sequences, indicating implicit sequence learning in both, AEP and AIP. This effect was stronger in the practice hand than in the transfer hand, indicating effector-dependent sequence representations in both AEP and AIP. However, effector-dependent sequence representations were stronger in AEP than in AIP. No significant differences between groups were observed in the transfer hand, although effector-independent sequence representations were observed in AEP only. In conclusion, AIP promotes not only sequence-unspecific stimulus-response coupling and anticipations of the subsequent stimuli, but also anticipations of the subsequent responses.

ACPred-BMF: bidirectional LSTM with multiple feature representations for explainable anticancer peptide prediction

Cancer has become a major factor threatening human life and health. Under the circumstance that traditional treatment methods such as chemotherapy and radiotherapy are not highly specific and often cause severe side effects and toxicity, new treatment methods are urgently needed. Anticancer peptide drugs have low toxicity, stronger efficacy and specificity, and have emerged as a new type of cancer treatment drugs. However, experimental identification of anticancer peptides is time-consuming and expensive, and difficult to perform in a high-throughput manner. Computational identification of anticancer peptides can make up for the shortcomings of experimental identification. In this study, a deep learning-based predictor named ACPred-BMF is proposed for the prediction of anticancer peptides. This method uses the quantitative and qualitative properties of amino acids, binary profile feature to numerical representation for the peptide sequences. The Bidirectional LSTM network architecture is used in the model, and the attention mechanism is also considered. To alleviate the black-box problem of deep learning model prediction, we visualized the automatically extracted features and used the Shapley additive explanations algorithm to determine the importance of features to further understand the anticancer peptide mechanism. The results show that our method is one of the state-of-the-art anticancer peptide predictors. A web server as the implementation of ACPred-BMF that can be accessed via: http://mialab.ruc.edu.cn/ACPredBMFServer/.

DNA-MP: a generalized DNA modifications predictor for multiple species based on powerful sequence encoding method.

Accurate prediction of deoxyribonucleic acid (DNA) modifications is essential to explore and discern the process of cell differentiation, gene expression and epigenetic regulation. Several computational approaches have been proposed for particular type-specific DNA modification prediction. Two recent generalized computational predictors are capable of detecting three different types of DNA modifications; however, type-specific and generalized modifications predictors produce limited performance across multiple species mainly due to the use of ineffective sequence encoding methods. The paper in hand presents a generalized computational approach "DNA-MP" that is competent to more precisely predict three different DNA modifications across multiple species. Proposed DNA-MP approach makes use of a powerful encoding method "position specific nucleotides occurrence based 117 on modification and non-modification class densities normalized difference" (POCD-ND) to generate the statistical representations of DNA sequences and a deep forest classifier for modifications prediction. POCD-ND encoder generates statistical representations by extracting position specific distributional information of nucleotides in the DNA sequences. We perform a comprehensive intrinsic and extrinsic evaluation of the proposed encoder and compare its performance with 32 most widely used encoding methods on $17$ benchmark DNA modifications prediction datasets of $12$ different species using $10$ different machine learning classifiers. Overall, with all classifiers, the proposed POCD-ND encoder outperforms existing $32$ different encoders. Furthermore, combinedly over 5-fold cross validation benchmark datasets and independent test sets, proposed DNA-MP predictor outperforms state-of-the-art type-specific and generalized modifications predictors by an average accuracy of 7% across 4mc datasets, 1.35% across 5hmc datasets and 10% for 6ma datasets. To facilitate the scientific community, the DNA-MP web application is available at https://sds_genetic_analysis.opendfki.de/DNA_Modifications/.