Hidden Representations Research Articles

Extracting Sentence Embeddings from Pretrained Transformer Models

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT’s hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT’s sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

Deciphering the Feature Representation of Deep Neural Networks for High-Performance AI.

AI driven by deep learning is transforming many aspects of science and technology. The enormous success of deep learning stems from its unique capability of extracting essential features from Big Data for decision-making. However, the feature extraction and hidden representations in deep neural networks (DNNs) remain inexplicable, primarily because of lack of technical tools to comprehend and interrogate the feature space data. The main hurdle here is that the feature data are often noisy in nature, complex in structure, and huge in size and dimensionality, making it intractable for existing techniques to analyze the data reliably. In this work, we develop a computational framework named contrastive feature analysis (CFA) to facilitate the exploration of the DNN feature space and improve the performance of AI. By utilizing the interaction relations among the features and incorporating a novel data-driven kernel formation strategy into the feature analysis pipeline, CFA mitigates the limitations of traditional approaches and provides an urgently needed solution for the analysis of feature space data. The technique allows feature data exploration in unsupervised, semi-supervised and supervised formats to address different needs of downstream applications. The potential of CFA and its applications for pruning of neural network architectures are demonstrated using several state-of-the-art networks and well-annotated datasets across different disciplines.

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called Beyond Self-Attention for Sequential Recommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.

Redundant representations help generalization in wide neural networks * ,

Deep neural networks (DNNs) defy the classical bias-variance trade-off; adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ‘benign overfitting’ in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information and differ from each other only by statistically independent noise. The number of these groups increases linearly with the width of the layer, but only if the width is above a critical value. We show that redundant neurons appear only when the training is regularized and the training error is zero.

LaenNet: Learning robust GCNs by propagating labels

Graph Convolutional Networks (GCNs) can be acknowledged as one of the most significant methodologies for graph representation learning, and the family of GCNs has recently achieved great success in the community. However, in real-world scenarios, the graph data may be imperfect, e.g., with noisy and sparse features or labels, which poses a great challenge to the robustness of GCNs. To meet this challenge, we propose a simple-yet-effective LAbel-ENhanced Networks (LaenNet) architecture for GCNs, where the basic spirit is to propagate labels together with features. Specifically, we add an extra LaenNet module at one hidden layer of GCNs, which propagates labels along the graph and then integrates them with the hidden representations as the inputs to the deeper layer. The proposed LaenNet can be directly generalized to the variants of GCNs. We conduct extensive experiments to verify LaenNet on semi-supervised node classification tasks under four noisy and sparse graph data scenarios, including the graphs with noisy features, sparse features, noisy labels, and sparse labels. Empirical results indicate the superiority and robustness of LaenNet compared to the state-of-the-art baseline models. The implementation code is available to ease reproducibility11https://github.com/Zhangcx19/LAENNet..

Retrosynthesis Prediction with Local Template Retrieval

Retrosynthesis, which predicts the reactants of a given target molecule, is an essential task for drug discovery. In recent years, the machine learing based retrosynthesis methods have achieved promising results. In this work, we introduce RetroKNN, a local reaction template retrieval method to further boost the performance of template-based systems with non-parametric retrieval. We first build an atom-template store and a bond-template store that contains the local templates in the training data, then retrieve from these templates with a k-nearest-neighbor (KNN) search during inference. The retrieved templates are combined with neural network predictions as the final output. Furthermore, we propose a lightweight adapter to adjust the weights when combing neural network and KNN predictions conditioned on the hidden representation and the retrieved templates. We conduct comprehensive experiments on two widely used benchmarks, the USPTO-50K and USPTO-MIT. Especially for the top-1 accuracy, we improved 7.1% on the USPTO-50K dataset and 12.0% on the USPTO-MIT dataset.These results demonstrate the effectiveness of our method.

ReCoMIF: Reading comprehension based multi-source information fusion network for Chinese spoken language understanding

Spoken language understanding (SLU) plays a crucial role in the performance of dialogue systems. It usually includes slot filling and intent detection (SFID) tasks aiming at semantic parsing of utterances. At present, researchers focus mainly on English SLU tasks, while such investigations on Chinese utterances are not sufficient. In this paper, we first propose a reading comprehension based multi-source information fusion network, called ReCoMIF for Chinese SFID tasks by transforming the SLU task into a multi-turn question answering procedure comprising multiple choice for intent detection and span extraction for slot filling. Moreover, we present a TCM_CLS module with a concise architecture composed of TextCNN, MaxPooling, and feed forward network plus the [CLS] representation. Such three TCM_CLS modules are stacked in the proposed ReCoMIF network that can serve as sufficient integration of multi-source information originating from contexts, reading comprehension based queries, and hidden representations concerning intent and slot semantics. Finally, experimental results and ablation studies on three Chinese SLU datasets show that our proposed model can effectively fuse intent and slot information achieved by\\ state-of-the-art performance compared with other baseline methods.

TTS-Guided Training for Accent Conversion Without Parallel Data

Accent Conversion (AC) seeks to change the accent of speech from one (source) to another (target) while preserving the speech content and speaker identity. However, many existing AC approaches rely on source-target parallel speech data during training or reference speech at run-time. We propose a novel accent conversion framework without the need for either parallel data or reference speech. Specifically, a text-to-speech (TTS) system is first pretrained with target-accented speech data. This TTS model and its hidden representations are expected to be associated only with the target accent. Then, a speech encoder is trained to convert the accent of the speech under the supervision of the pretrained TTS model. In doing so, the source-accented speech and its corresponding transcription are forwarded to the speech encoder and the pretrained TTS, respectively. The output of the speech encoder is optimized to be the same as the text embedding in the TTS system. At run-time, the speech encoder is combined with the pretrained speech decoder to convert the source-accented speech toward the target. In the experiments, we converted English with two source accents (Chinese/Indian) to the target accent (American/British/Canadian). Both objective metrics and subjective listening tests successfully validate that the proposed approach generates speech samples that are close to the target accent with high speech quality.

Learning Representations by Graphical Mutual Information Estimation and Maximization.

The rich content in various real-world networks such as social networks, biological networks, and communication networks provides unprecedented opportunities for unsupervised machine learning on graphs. This paper investigates the fundamental problem of preserving and extracting abundant information from graph-structured data into embedding space without external supervision. To this end, we generalize conventional mutual information computation from vector space to graph domain and present a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graph and hidden representation. Except for standard GMI which considers graph structures from a local perspective, our further proposed GMI++ additionally captures global topological properties by analyzing the co-occurrence relationship of nodes. GMI and its extension exhibit several benefits: First, they are invariant to the isomorphic transformation of input graphs-an inevitable constraint in many existing methods; Second, they can be efficiently estimated and maximized by current mutual information estimation methods; Lastly, our theoretical analysis confirms their correctness and rationality. With the aid of GMI, we develop an unsupervised embedding model and adapt it to the specific anomaly detection task. Extensive experiments indicate that our GMI methods achieve promising performance in various downstream tasks, such as node classification, link prediction, and anomaly detection.

A novel feature integration and entity boundary detection for named entity recognition in cybersecurity

Owing to continuous cyberattacks, a large amount of threat intelligence is generated online every day. However, threat intelligence is mostly unstructured and multisource heterogeneous text. It is difficult for security analysts to understand the implicit threat in time. Knowledge Graph (KG) is an important research topic in recent years, which can perform automated and real-time analysis of threat intelligence in cybersecurity. As one of the critical technologies of KG, named entity recognition (NER) can identify cyberattack-related entities. It has been proved that long-distance structured information captured by dependency trees provides a rich semantic expression for the neural network. However, existing research works are more focused on the simple linear stack of neural networks when utilizing structured features. The interaction between different types of neural networks is vague. In addition, the existing models are insensitive to the boundaries of complex entity terms in cybersecurity. In this study, we propose a new feature integration and entity boundary detection (FIEBD) model. In our model, a new pretrained language model, PERT, is applied to obtain word embedding of cyber texts. Moreover, a novel neural network cell, namely GARU, is developed to incorporate different types of features extracted from graph neural networks and recurrent neural networks. It combines the graph encoder with the gate mechanism, aiming to obtain better hidden representation by explicit interaction. Furthermore, considering a large number of complex entities in cybersecurity, we contribute an entity boundary detection module to perform entity head and tail prediction as an augmentation task. We conduct extensive experiments on cybersecurity datasets. The results demonstrate that the proposed model achieves better performance than existing baseline methods.

Adaptation to CT Reconstruction Kernels by Enforcing Cross-Domain Feature Maps Consistency.

Deep learning methods provide significant assistance in analyzing coronavirus disease (COVID-19) in chest computed tomography (CT) images, including identification, severity assessment, and segmentation. Although the earlier developed methods address the lack of data and specific annotations, the current goal is to build a robust algorithm for clinical use, having a larger pool of available data. With the larger datasets, the domain shift problem arises, affecting the performance of methods on the unseen data. One of the critical sources of domain shift in CT images is the difference in reconstruction kernels used to generate images from the raw data (sinograms). In this paper, we show a decrease in the COVID-19 segmentation quality of the model trained on the smooth and tested on the sharp reconstruction kernels. Furthermore, we compare several domain adaptation approaches to tackle the problem, such as task-specific augmentation and unsupervised adversarial learning. Finally, we propose the unsupervised adaptation method, called F-Consistency, that outperforms the previous approaches. Our method exploits a set of unlabeled CT image pairs which differ only in reconstruction kernels within every pair. It enforces the similarity of the network’s hidden representations (feature maps) by minimizing the mean squared error (MSE) between paired feature maps. We show our method achieving a 0.64 Dice Score on the test dataset with unseen sharp kernels, compared to the 0.56 Dice Score of the baseline model. Moreover, F-Consistency scores 0.80 Dice Score between predictions on the paired images, which almost doubles the baseline score of 0.46 and surpasses the other methods. We also show F-Consistency to better generalize on the unseen kernels and without the presence of the COVID-19 lesions than the other methods trained on unlabeled data.

Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information

Stack Overflow is very helpful for software developers who are seeking answers to programming problems. Previous studies have shown that a growing number of questions are of low quality and thus obtain less attention from potential answerers. Gao et al. proposed an LSTM-based model (i.e., BiLSTM-CC) to automatically generate question titles from the code snippets to improve the question quality. However, only using the code snippets in the question body cannot provide sufficient information for title generation, and LSTMs cannot capture the long-range dependencies between tokens. This paper proposes CCBERT, a deep learning based novel model to enhance the performance of question title generation by making full use of the bi-modal information of the entire question body. CCBERT follows the encoder–decoder paradigm and uses CodeBERT to encode the question body into hidden representations, a stacked Transformer decoder to generate predicted tokens, and an additional copy attention layer to refine the output distribution. Both the encoder and decoder perform the multi-head self-attention operation to better capture the long-range dependencies. This paper builds a dataset containing around 200,000 high-quality questions filtered from the data officially published by Stack Overflow to verify the effectiveness of the CCBERT model. CCBERT outperforms all the baseline models on the dataset. Experiments on both code-only and low-resource datasets show the superiority of CCBERT with less performance degradation. The human evaluation also shows the excellent performance of CCBERT concerning both readability and correlation criteria. CCBERT is capable of automatically capturing the bi-modal semantic information from the entire question body and parsing the long-range dependencies to achieve better performance. Therefore, CCBERT is an effective approach for generating Stack Overflow question titles.

Deep Dynamic Patient Similarity Analysis: Model Development and Validation in ICU

BackgroundPersonalized medicine requires the patient similarity analysis for providing specific treatments tailed for each patient. However, the patient similarity analysis in personalized clinical scenarios encounters challenges, which are twofold. First, heterogeneous and multi-type data are usually recorded to Electronic Health Records (EHRs) during the course of admissions, which makes it difficult to measure the patient similarity. Second, disease progression manifests diverse disease states at different times, which brings sequential complexity to dynamically retrieve similar patients' sequences. Materials and methodsTo overcome the above-mentioned challenges, we propose a novel dynamic patient similarity analysis model based on deep learning. Specifically, the proposed model embeds the multi-type and heterogeneous data into hidden representations with a specially designed embedding and attention module. Thereafter, the proposed model retrieves similar patients' sequences based on these hidden representations in a dynamic manner. More importantly, we adopt two clinical tasks, i.e., diagnosis prediction and medication recommendation, to validate the effectiveness of the proposed model. It is worth noticing that the proposed model integrates a drug-drug interaction (DDI) knowledge graph in the medication recommendation task to reduce adverse reactions caused by combinational treatments, such that a more rational strategy can be realized. We evaluate our proposed model using the critical care database MIMIC-III, which includes 5,430 patients covering 14,096 clinical visits. ResultsThe proposed model outperforms several state-of-the-art methods. For diagnosis prediction, the average PR-AUC score of the proposed model reaches 0.6200, which is significantly higher than that of the baseline models (0.2497∼0.5407). Meanwhile, for medication recommendation, the average PR-AUC of the proposed model is 0.6682 (Jaccard: 0.4070; F1: 0.5672; Recall: 0.7832) whereas the K-nearest model can only reach 0.3805 (Jaccard: 0.3911; F1: 0.5465; Recall: 0.5705). In addition, our proposed model achieves a lower DDI rate. ConclusionWe propose a novel dynamic patient similarity analysis model, which can be implemented into a decision support system for clinical tasks including diagnosis prediction, surgical procedure selection, medication recommendation, etc. Also, the proposed model serves as an explainable protocol in clinical practice thanks to its analogy to real clinical reasoning where a doctor diagnoses diseases and prescribes medications according to the previous cured patients empirically.

Feature Encoding With Autoencoders for Weakly Supervised Anomaly Detection.

Weakly supervised anomaly detection aims at learning an anomaly detector from a limited amount of labeled data and abundant unlabeled data. Recent works build deep neural networks for anomaly detection by discriminatively mapping the normal samples and abnormal samples to different regions in the feature space or fitting different distributions. However, due to the limited number of annotated anomaly samples, directly training networks with the discriminative loss may not be sufficient. To overcome this issue, this article proposes a novel strategy to transform the input data into a more meaningful representation that could be used for anomaly detection. Specifically, we leverage an autoencoder to encode the input data and utilize three factors, hidden representation, reconstruction residual vector, and reconstruction error, as the new representation for the input data. This representation amounts to encode a test sample with its projection on the training data manifold, its direction to its projection, and its distance to its projection. In addition to this encoding, we also propose a novel network architecture to seamlessly incorporate those three factors. From our extensive experiments, the benefits of the proposed strategy are clearly demonstrated by its superior performance over the competitive methods. Code is available at: https://github.com/yj-zhou/Feature_Encoding_with_AutoEncoders_for_Weakly-supervised_Anomaly_Detection.

Masked Contrastive Representation Learning for Reinforcement Learning.

In pixel-based reinforcement learning (RL), the states are raw video frames, which are mapped into hidden representation before feeding to a policy network. To improve sample efficiency of state representation learning, recently, the most prominent work is based on contrastive unsupervised representation. Witnessing that consecutive video frames in a game are highly correlated, to further improve data efficiency, we propose a new algorithm, i.e., masked contrastive representation learning for RL (M-CURL), which takes the correlation among consecutive inputs into consideration. In our architecture, besides a CNN encoder for hidden presentation of input state and a policy network for action selection, we introduce an auxiliary Transformer encoder module to leverage the correlations among video frames. During training, we randomly mask the features of several frames, and use the CNN encoder and Transformer to reconstruct them based on context frames. The CNN encoder and Transformer are jointly trained via contrastive learning where the reconstructed features should be similar to the ground-truth ones while dissimilar to others. During policy evaluation, the CNN encoder and the policy network are used to take actions, and the Transformer module is discarded. Our method achieves consistent improvements over CURL on 14 out of 16 environments from DMControl suite and 23 out of 26 environments from Atari 2600 Games. The code is available at https://github.com/teslacool/m-curl.

МОДЕЛЬ РАСПРОСТРАНЕНИЯ ДЕСТРУКТИВНЫХ ВОЗДЕЙСТВИЙ В ОГРАНИЧЕННЫХ КОЛЛЕКТИВАХ НА ОСНОВЕ ОГРАНИЧЕННОЙ МАШИНЫ БОЛЬЦМАНА

The article considers a network model of spreading infectious diseases adapted for forecasting tasks in organizations. The authors propose the implementation of the current model based on Restricted Bolzmann Machine architecture. This approach allows reproducing the influence of its constituent nodes on each other due to hidden representations, which makes it possible to form potential scenarios for the spread of infection with relatively small amount of initial data. New coronavirus infection is used as an example of destructive impact.

Blind microscopy image denoising with a deep residual and multiscale encoder/decoder network.

In computer-aided diagnosis (CAD) focused on microscopy, denoising improves the quality of image analysis. In general, the accuracy of this process may depend both on the experience of the microscopist and on the equipment sensitivity and specificity. A medical image could be corrupted by several perturbations during image acquisition. Nowadays, CAD deep learning applications pre-process images with image denoising models to reinforce learning and prediction. In this work, an innovative and lightweight deep multiscale convolutional encoder-decoder neural network is proposed. Specifically, the encoder uses deterministic mapping to map features into a hidden representation. Then, the latent representation is rebuilt to generate the reconstructed denoised image. Residual learning strategies are used to improve and accelerate the training process using skip connections in bridging across convolutional and deconvolutional layers. The proposed model reaches on average 38.38 of PSNR and 0.98 of SSIM on a test set of 57458 images overcoming state-of-the-art models in the same application domain.Clinical relevance - Encoder-decoder based denoiser enables industry experts to provide more accurate and reliable medical interpretation and diagnosis in a variety of fields, from microscopy to surgery, with the benefit of real-time processing.

ML-ANet: A Transfer Learning Approach Using Adaptation Network for Multi-label Image Classification in Autonomous Driving

To reduce the discrepancy between the source and target domains, a new multi-label adaptation network (ML-ANet) based on multiple kernel variants with maximum mean discrepancies is proposed in this paper. The hidden representations of the task-specific layers in ML-ANet are embedded in the reproducing kernel Hilbert space (RKHS) so that the mean-embeddings of specific features in different domains could be precisely matched. Multiple kernel functions are used to improve feature distribution efficiency for explicit mean embedding matching, which can further reduce domain discrepancy. Adverse weather and cross-camera adaptation examinations are conducted to verify the effectiveness of our proposed ML-ANet. The results show that our proposed ML-ANet achieves higher accuracies than the compared state-of-the-art methods for multi-label image classification in both the adverse weather adaptation and cross-camera adaptation experiments. These results indicate that ML-ANet can alleviate the reliance on fully labeled training data and improve the accuracy of multi-label image classification in various domain shift scenarios.

Interpretable deep recommender system model for prediction of kinase inhibitor efficacy across cancer cell lines

Computational models for drug sensitivity prediction have the potential to significantly improve personalized cancer medicine. Drug sensitivity assays, combined with profiling of cancer cell lines and drugs become increasingly available for training such models. Multiple methods were proposed for predicting drug sensitivity from cancer cell line features, some in a multi-task fashion. So far, no such model leveraged drug inhibition profiles. Importantly, multi-task models require a tailored approach to model interpretability. In this work, we develop DEERS, a neural network recommender system for kinase inhibitor sensitivity prediction. The model utilizes molecular features of the cancer cell lines and kinase inhibition profiles of the drugs. DEERS incorporates two autoencoders to project cell line and drug features into 10-dimensional hidden representations and a feed-forward neural network to combine them into response prediction. We propose a novel interpretability approach, which in addition to the set of modeled features considers also the genes and processes outside of this set. Our approach outperforms simpler matrix factorization models, achieving R = 0.82 correlation between true and predicted response for the unseen cell lines. The interpretability analysis identifies 67 biological processes that drive the cell line sensitivity to particular compounds. Detailed case studies are shown for PHA-793887, XMD14-99 and Dabrafenib.

Information Flows of Diverse Autoencoders

Deep learning methods have had outstanding performances in various fields. A fundamental query is why they are so effective. Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data. The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers. In this study, we examine how the information flows are shaped by the network parameters, such as depth, sparsity, weight constraints, and hidden representations. Here, we adopt autoencoders as models of deep learning, because (i) they have clear guidelines for their information flows, and (ii) they have various species, such as vanilla, sparse, tied, variational, and label autoencoders. We measured their information flows using Rényi’s matrix-based -order entropy functional. As learning progresses, they show a typical fitting phase where the amounts of input-to-hidden and hidden-to-output mutual information both increase. In the last stage of learning, however, some autoencoders show a simplifying phase, previously called the “compression phase”, where input-to-hidden mutual information diminishes. In particular, the sparsity regularization of hidden activities amplifies the simplifying phase. However, tied, variational, and label autoencoders do not have a simplifying phase. Nevertheless, all autoencoders have similar reconstruction errors for training and test data. Thus, the simplifying phase does not seem to be necessary for the generalization of learning.