Text Sequences Research Articles

DeepJoin: Joinable Table Discovery with Pre-Trained Language Models

Due to the usefulness in data enrichment for data analysis tasks, joinable table discovery has become an important operation in data lake management. Existing approaches target equi-joins, the most common way of combining tables for creating a unified view, or semantic joins, which tolerate misspellings and different formats to deliver more join results. They are either exact solutions whose running time is linear in the sizes of query column and target table repository, or approximate solutions lacking precision. In this paper, we propose DeepJoin, a deep learning model for accurate and efficient joinable table discovery. Our solution is an embedding-based retrieval, which employs a pre-trained language model (PLM) and is designed as one framework serving both equi- and semantic (with a similarity condition on word embeddings) joins for textual attributes with fairly small cardinalities. We propose a set of contextualization options to transform column contents to a text sequence. The PLM reads the sequence and is fine-tuned to embed columns to vectors such that columns are expected to be joinable if they are close to each other in the vector space. Since the output of the PLM is fixed in length, the subsequent search procedure becomes independent of the column size. With a state-of-the-art approximate nearest neighbor search algorithm, the search time is sublinear in the repository size. To train the model, we devise the techniques for preparing training data as well as data augmentation. The experiments on real datasets demonstrate that by training on a small subset of a corpus, DeepJoin generalizes to large datasets and its precision consistently outperforms other approximate solutions'. DeepJoin is even more accurate than an exact solution to semantic joins when evaluated with labels from experts. Moreover, when equipped with a GPU, DeepJoin is up to two orders of magnitude faster than existing solutions.

Analysis of different transformer variants

Efficient self-attention models are crucial in applications where long sequences are modeled, such as documents, images, and videos, are often composed of relatively large numbers of pixels or tokens. Therefore, the efficiency of dealing with long sequences is crucial for the broad adoption of Transformers. This paper talks about four transformer variants: Longformer, Transformer-XL, Big Bird, and Star transformer. This paper makes a comparative analysis of the four varieties theoretical methods, and obtains the advantages of the four methods and their applicable fields. Longformer has lower complexity and can be used for various document-level tasks. Transformer-XL improves the accuracy by addressing context fragmentation. BigBird replaces the Bert-like whole attention mechanism with Block Sparse Attention, which has achieved the SOTA effect on many tasks with long text sequences, such as long text summaries, long text questions, and answers. Star-Transformer reduces the time complexity to and performs well in the synthetic data set.

Multimodal emotion recognition based on audio and text by using hybrid attention networks

Multimodal Emotion Recognition (MER) has recently become a popular and challenging topic. The most key challenge in MER is how to effectively fuse multimodal information. Most of prior works may not fully consider the inter-modal and intra-modal attention mechanism to jointly learn intra-modal and inter-modal emotional salient information for further improving the performance of MER. To address this problem, this paper proposes a new MER framework based on audio and text by using Hybrid Attention Networks (MER-HAN). The proposed MER-HAN combines three different attention mechanisms such as the local intra-modal attention, the cross-modal attention, and the global inter-modal attention to effectively learn intra-modal and inter-modal emotional salient features for MER. Specifically, an Audio and Text Encoder (ATE) block equipped with deep learning techniques with the local intra-modal attention mechanism is initially designed to learn high-level audio and text feature representations from the corresponding audio and text sequences, respectively. Then, a Cross-Modal Attention (CMA) block is presented to jointly capture high-level shared feature representations across audio and text modalities. Finally, a Multimodal Emotion Classification (MEC) block with the global inter-modal attention mechanism is provided to obtain final MER results. Extensive experiments conducted on two public multimodal emotional datasets, i.e., IEMOCAP and MELD datasets, show the advantage of the proposed MER-HAN on MER tasks.

Novel Heuristic Recurrent Neural Network Framework to Handle Automatic Telugu Text Categorization from Handwritten Text Image

In the near future, the digitization and processing of the current paper documents describe efficient role in the creation of a paperless environment. Deep learning techniques for handwritten recognition have been extensively studied by various researchers. Deep neural networks can be trained quickly thanks to a lot of data and other algorithmic advancements. Various methods for extracting text from handwritten manuscripts have been developed in literature. To extract features from written Telugu Text image having some other neural network approaches like convolution neural network (CNN), recurrent neural networks (RNN), long short-term memory (LSTM). Different deep learning related approaches are widely used to identification of handwritten Telugu Text; various techniques are used in literature for the identification of Telugu Text from documents. For automatic identification of Telugu written script efficiently to eliminate noise and other semantic features present in Telugu Text, in this paper, proposes Novel Heuristic Advanced Neural Network based Telugu Text Categorization Model (NHANNTCM) based on sequence-to-sequence feature extraction procedure. Proposed approach extracts the features using RNN and then represents Telugu Text in sequence-to-sequence format for the identification advanced neural network performs both encoding and decoding to identify and explore visual features from sequence of Telugu Text in input data. The classification accuracy rates for Telugu words, Telugu numerals, Telugu characters, Telugu sentences, and the corresponding Telugu sentences were 99.66%, 93.63%, 91.36%, 99.05%, and 97.73% consequently. Experimental evaluation describe extracted with revealed which are textured i.e. TENG shown considerable operations in applications such as private information protection, security defense, and personal handwriting signature identification.

A Study on the Strategic Application of Semantic Segmentation Based on Improved Convolutional Neural Networks on English-Chinese Interaction

Teachers need to provide numerous examples sentences for students to translate in the process of teaching English, but the number of sentences given by teachers to practice subjectively is not enough. Therefore, the study constructs a text generation model using an improved convolutional neural network semantic segmentation method, where the corpus utterances are keyword extracted and new shorter utterances are generated based on the keywords for language learners to practice translation. The research first uses the textRank algorithm to extract semantic keywords to obtain a dataset, and then uses CNN to construct an encoder to achieve semantic encoding of the keyword dataset. However, during the research process, it was found that traditional CNN models are relatively sensitive to the location of input data. Therefore, the research introduces the idea of Decomposition Machine (FM) to improve the encoder. In order to control text generation, research has introduced a weighted additive attention mechanism in the decoding process to associate the meaning of the generated text sequence with the meaning of the keyword set. Based on this, a text generation model for generating a translation related corpus in English teaching is constructed. This results in a text generation model that can be used to generate a translation-linked corpus for English language teaching. The average BLEU value of model 1 is 0.924, Inform value is 98.40, the Success value is 97.64, and the Combine value is 101.24, which can achieve high-quality text generation by the keyword lexical meaning and provide technical guarantee for the establishment of the corpus in educating in English.

Knowledge Adaptive Multi-Way Matching Network for Biomedical Named Entity Recognition via Machine Reading Comprehension.

Rapid and effective utilization of biomedical literature is paramount to combat diseases like COVID19. Biomedical named entity recognition (BioNER) is a fundamental task in text mining that can help physicians accelerate knowledge discovery to curb the spread of the COVID-19 epidemic. Recent approaches have shown that casting entity extraction as the machine reading comprehension task can significantly improve model performance. However, two major drawbacks impede higher success in identifying entities (1) ignoring the use of domain knowledge to capture the context beyond sentences and (2) lacking the ability to deeper understand the intent of questions. In this paper, to remedy this, we introduce and explore external domain knowledge which cannot be implicitly learned in text sequence. Previous works have focused more on text sequence and explored little of the domain knowledge. To better incorporate domain knowledge, a multi-way matching reader mechanism is devised to model representations of interaction between sequence, question and knowledge retrieved from Unified Medical Language System (UMLS). Benefiting from these, our model can better understand the intent of questions in complex contexts. Experimental results indicate that incorporating domain knowledge can help to obtain competitive results across 10 BioNER datasets, achieving absolute improvement of up to 2.02% in the f1 score.

Synwmd: Syntax-aware word Mover’s distance for sentence similarity evaluation

Word Mover’s Distance (WMD) computes the distance between words and models text similarity with the moving cost between words in two text sequences. Yet, it does not offer good performance in sentence similarity evaluation since it does not incorporate word importance and fails to take inherent contextual and structural information in a sentence into account. An improved WMD method using the syntactic parse tree, called Syntax-aware Word Mover’s Distance (SynWMD), is proposed to address these two shortcomings in this work. First, a weighted graph is built upon the word co-occurrence statistics extracted from the syntactic parse trees of sentences. The importance of each word is inferred from graph connectivities. Second, the local syntactic parsing structure of words is considered in computing the distance between words. To demonstrate the effectiveness of the proposed SynWMD, we conduct experiments on 6 textual semantic similarity (STS) datasets and 4 sentence classification datasets. Experimental results show that SynWMD achieves state-of-the-art performance on STS tasks. It also outperforms other WMD-based methods on sentence classification tasks.

JSUM: A Multitask Learning Speech Recognition Model for Jointly Supervised and Unsupervised Learning

In recent years, the end-to-end speech recognition model has emerged as a popular alternative to the traditional Deep Neural Network—Hidden Markov Model (DNN-HMM). This approach maps acoustic features directly onto text sequences via a single network architecture, significantly streamlining the model construction process. However, the training of end-to-end speech recognition models typically necessitates a significant quantity of supervised data to achieve good performance, which poses a challenge in low-resource conditions. The use of unsupervised representation significantly reduces this necessity. Recent research has focused on end-to-end techniques employing joint Connectionist Temporal Classification (CTC) and attention mechanisms, with some also concentrating on unsupervised presentation learning. This paper proposes a joint supervised and unsupervised multi-task learning model (JSUM). Our approach leverages the unsupervised pre-trained wav2vec 2.0 model as a shared encoder that integrates the joint CTC-Attention network and the generative adversarial network into a unified end-to-end architecture. Our method provides a new low-resource language speech recognition solution that optimally utilizes supervised and unsupervised datasets by combining CTC, attention, and generative adversarial losses. Furthermore, our proposed approach is suitable for both monolingual and cross-lingual scenarios.

Personalized Education Resource Recommendation Method Based on Deep Learning in Intelligent Educational Robot Environments

The goal of this article is to analyze the problem of low computational efficiency and propagation error rate in entity recognition and relation extraction. This paper proposes a personalized education resource recommendation algorithm framework XMAMBLSTM based on deep learning in an intelligent education robot environment. XMAMBLSTM uses XLNet to assign word vectors to text sequences, employs a Multi-Bi-LSTM layer to represent complex information of word vectors, and combines a multi-headed attention layer to realize weight distribution of each word vector. The experimental results show that compared with the traditional collaborative filtering algorithm, the comprehensive evaluation indexes of the proposed method, based on the intelligent education robot environment on the two platforms, are higher than 5.05% and 17.3%, respectively.

Research on Chinese Audio and Text Alignment Algorithm Based on AIC-FCM and Doc2Vec

‘‘Audiobook” is a multimedia-based reading technology that has emerged in recent years. Realizing the alignment of e-book text and book audio is the most important part of its processing. This article describes an audio and text alignment algorithm using deep learning and neural network technology to improve the efficiency and quality of audiobook production. The algorithm first uses dual-threshold endpoint detection technology to segment long audio into short audio with sentence dimensions and recognizes it as short text. The threshold is calculated by AIC-FCM optimized based on simulated annealing genetic algorithm. Then the algorithm uses Doc2vec optimized by the threshold prediction method based on the average length of the short text to calculate the text similarity. Finally, proofread and output the text sequence and audio segment aligned in the time dimension to meet the needs of audiobook production. Experiments show that compared to traditional audio and text alignment algorithms, the proposed algorithm is closer to the ideal segmentation result in long audio segmentation, and the alignment effect is basically the same as Doc2vec and the time complexity is reduced by about 35%.

Sentiment analysis of hotel online reviews using the BERT model and ERNIE model-Data from China.

The emotion analysis of hotel online reviews is discussed by using the neural network model BERT, which proves that this method can not only help hotel network platforms fully understand customer needs but also help customers find suitable hotels according to their needs and affordability and help hotel recommendations be more intelligent. Therefore, using the pretraining BERT model, a number of emotion analytical experiments were carried out through fine-tuning, and a model with high classification accuracy was obtained by frequently adjusting the parameters during the experiment. The BERT layer was taken as a word vector layer, and the input text sequence was used as the input to the BERT layer for vector transformation. The output vectors of BERT passed through the corresponding neural network and were then classified by the softmax activation function. ERNIE is an enhancement of the BERT layer. Both models can lead to good classification results, but the latter performs better. ERNIE exhibits stronger classification and stability than BERT, which provides a promising research direction for the field of tourism and hotels.

Ordinal analysis of lexical patterns.

Words are fundamental linguistic units that connect thoughts and things through meaning. However, words do not appear independently in a text sequence. The existence of syntactic rules induces correlations among neighboring words. Using an ordinal pattern approach, we present an analysis of lexical statistical connections for 11 major languages. We find that the diverse manners that languages utilize to express word relations give rise to unique pattern structural distributions. Furthermore, fluctuations of these pattern distributions for a given language can allow us to determine both the historical period when the text was written and its author. Taken together, our results emphasize the relevance of ordinal time series analysis in linguistic typology, historical linguistics, and stylometry.

Recognition of English speech – using a deep learning algorithm

Abstract The accurate recognition of speech is beneficial to the fields of machine translation and intelligent human–computer interaction. After briefly introducing speech recognition algorithms, this study proposed to recognize speech with a recurrent neural network (RNN) and adopted the connectionist temporal classification (CTC) algorithm to align input speech sequences and output text sequences forcibly. Simulation experiments compared the RNN-CTC algorithm with the Gaussian mixture model–hidden Markov model and convolutional neural network-CTC algorithms. The results demonstrated that the more training samples the speech recognition algorithm had, the higher the recognition accuracy of the trained algorithm was, but the training time consumption increased gradually; the more samples a trained speech recognition algorithm had to test, the lower the recognition accuracy and the longer the testing time. The proposed RNN-CTC speech recognition algorithm always had the highest accuracy and the lowest training and testing time among the three algorithms when the number of training and testing samples was the same.

Spatial–temporal transformer for end-to-end sign language recognition

Continuous sign language recognition (CSLR) is an essential task for communication between hearing-impaired and people without limitations, which aims at aligning low-density video sequences with high-density text sequences. The current methods for CSLR were mainly based on convolutional neural networks. However, these methods perform poorly in balancing spatial and temporal features during visual feature extraction, making them difficult to improve the accuracy of recognition. To address this issue, we designed an end-to-end CSLR network: Spatial–Temporal Transformer Network (STTN). The model encodes and decodes the sign language video as a predicted sequence that is aligned with a given text sequence. First, since the image sequences are too long for the model to handle directly, we chunk the sign language video frames, i.e., ”image to patch”, which reduces the computational complexity. Second, global features of the sign language video are modeled at the beginning of the model, and the spatial action features of the current video frame and the semantic features of consecutive frames in the temporal dimension are extracted separately, giving rise to fully extracting visual features. Finally, the model uses a simple cross-entropy loss to align video and text. We extensively evaluated the proposed network on two publicly available datasets, CSL and RWTH-PHOENIX-Weather multi-signer 2014 (PHOENIX-2014), which demonstrated the superior performance of our work in CSLR task compared to the state-of-the-art methods.

Genetic and genealogical (genealogical) study of the genus Nilogovoi (based on the DNA of autosomes)

The article deals with the problem of genetic and genealogical (genealogical) verification of the documentary pedigree of the Nilogovs, reconstructed by the methods of classical genealogy. Thanks to the testing of male and female representatives of the genus Nilogov on the DNA of autosomes, it was possible to genealogically confirm the documentary pedigree of the author of the article to his direct ancestor Semyon Danilovich Nilogov, who lived in the XVIII century. The object of study is the biological pedigree of the Nilogovs, and the subject is the archival and documentary reconstruction of the pedigree of the Nilogovs before the XVIII century. The source base of the work was the collections of such archives as: the State Archive of the Perm Territory, the Komi-Permyatsky District State Archive, the Russian State Archive of Ancient Acts, the archive of the city of Minusinsk; personal archives of several Nilogov families were also used. The novelty of the work is the application of modern genetic and genealogical methodology for the objectification of a specific documentary pedigree by certifying biological kinship in the "descendant – ancestor" chain. Specifically, the genetic relationship of two distant relatives N. A. Nelogov and R. A. Arinicheva was studied at the level of the sixth degree. Thanks to testing of autosomal DNA of these donors, it was possible to identify their common nucleotide segments (text sequences in the form of nitrogenous bases), after which, using the statistical database of genetic kinship, The Shared cM Project 4.0., it was possible to confirm the documented reconstructed pedigree. The introduction of modern methods of molecular genetics (biological science) into scientific genealogy (historical science) is an actual trend of systemic (interdisciplinary) research.

Heterogeneous Knowledge Network for Visual Dialog

Visual dialog requires an agent to answer successive questions considering an image and dialog history, which is a classic vision-language task. Despite progress, there are still two key challenges: 1) parsing long or complex questions and answers and 2) dealing with the visual scene containing complicated interactions among entities. These challenges bring about the unsatisfactory consequence of current visual dialog methods. In this paper, we propose a novel Heterogeneous Knowledge Network (HKNet), which leverages textual sequence knowledge and graph knowledge to address the above issues. Specifically, the textual sequence knowledge is derived from the sentences that are retrieved from the image captions of the visual dialog dataset. The textual sequence knowledge can supplement essential common sense for parsing long or complex questions and answers. The graph knowledge is constructed via scene graph, which provides complete visual relationships for understanding the complicated interactions. These two kinds of heterogeneous knowledge complement each other and jointly improve the logical reasoning ability of the visual dialog. Extensive experimental results on two benchmark datasets: VisDial v0.9 and v1.0 demonstrate the superiority of the proposed HKNet. Ablation studies and visualization results further verify the effectiveness of our method.

Context-aware style learning and content recovery networks for neural style transfer

Neural text transfer aims to change the style of a text sequence while keeping its original content. Due to the lack of parallel data, unsupervised learning-based approaches have gained considerable development. However, there are still several problems in these approaches: (1) The generated transferred sequences sometimes have inconsistencies between the transferred style and content, and (2) It is difficult to ensure sufficient preservation of the core semantics of original sequences in the transferred sequences. To address these defects, we propose Context-aware Style Learning and Content Recovery networks (CSLCR) for neural text transfer. Specifically, to improve the consistency between the transferred style and content, the designed context-aware style learning layer (CSL) retrieves target style samples with similar semantics to the original sequence, and promotes deep interactive fusion with the original sequence, so as to generate transferred sequence with context-aware style. To tackle the second problem, we explore content constraint recovery layer (CCR) from an indirect perspective, which decodes and recovers the core content semantics of the original sequence and the transferred sequence by both recovery decoding layers, respectively, and intensifies the preservation of the core semantics of both the sequences by a multi-level constraint mechanism. Experiments on two public datasets demonstrate the superiority of our proposed method.

SA-SGRU: Combining Improved Self-Attention and Skip-GRU for Text Classification

When reading texts for text classification tasks, a large number of words are irrelevant, and in text classification tasks, the traditional self-attention mechanism has the problem of weight distribution limitations. Therefore, a text classification model that combines an improved self-attention mechanism with a Skip-GRU (Skip-grate recurrent unit) network (SA-SGRU) is proposed in this paper. Firstly, Skip-GRU, the enhanced model of GRU (Grate Recurrent Unit), is used to skip the content that is not important for text classification when reading texts and only capture effective global information. Then, the improved self-attention mechanism is introduced to redistribute the weight of the deep text sequences. Secondly, the optimized CNN (convolutional neural network) is combined to bring up the local features of texts. Finally, a Softmax classifier is used to obtain the classification results of sample labels. Experimental results show that the proposed method can achieve better performance on three public datasets compared with other baseline methods. The ablation experiments also demonstrate the effectiveness of each module in the proposed model.

Imagens, links e vídeos: argumentação em campanhas eleitorais no Twitter

This article presents an excerpt from our ongoing doctoral research, which aims to identify and analyze linguistic aspects and argumentative strategies in electoral campaign publica-tions on social media from the official profiles/pages of Jair Bolsonaro and Fernando Haddad - the 2018 Brazilian presidential candidates – and Ana Gomes and André Ventura – the 2021 Portuguese presidential candidates – based on the theoretical framework of Discourse Analysis. Here, we pro-vide an analysis of two of the 15 most relevant publications from each candidate on the Twitter social media posted in the 15 days immediately preceding the elections. We draw on Textual Lin-guistics, following the works of Adam (1997, 2001a, 2001b), Bronckart (1985), and Plantin (1996), to identify and analyze how the structure of the social network (Bossetta, 2018), as a textual support, as conceived by Maingueneau (2001), influences the materialization of techno-discourses (Paveau, 2021), which is reflected in the construction of textual sequences identified in the corpus, specifi-cally regarding the presentation of arguments or data (premises) in argumentative textual sequences in which multimodal compositions involving static images, links, and videos were preferred.

A deep learning approach of financial distress recognition combining text

<abstract><p>The financial distress of listed companies not only harms the interests of internal managers and employees but also brings considerable risks to external investors and other stakeholders. Therefore, it is crucial to construct an efficient financial distress prediction model. However, most existing studies use financial indicators or text features without contextual information to predict financial distress and fail to extract critical details disclosed in Chinese long texts for research. This research introduces an attention mechanism into the deep learning text classification model to deal with the classification of Chinese long text sequences. We combine the financial data and management discussion and analysis Chinese text data in the annual reports of 1642 listed companies in China from 2017 to 2020 in the model and compare the effects of the data on different models. The empirical results show that the performance of deep learning models in financial distress prediction overcomes traditional machine learning models. The addition of the attention mechanism improved the effectiveness of the deep learning model in financial distress prediction. Among the models constructed in this study, the Bi-LSTM+Attention model achieves the best performance in financial distress prediction.</p></abstract>