Training Corpus Research Articles

Factors determining generalization in deep learning models for scoring COVID-CT images.

The COVID-19 pandemic has inspired unprecedented data collection and computer vision modelling efforts worldwide, focused on the diagnosis of COVID-19 from medical images. However, these models have found limited, if any, clinical application due in part to unproven generalization to data sets beyond their source training corpus. This study investigates the generalizability of deep learning models using publicly available COVID-19 Computed Tomography data through cross dataset validation. The predictive ability of these models for COVID-19 severity is assessed using an independent dataset that is stratified for COVID-19 lung involvement. Each inter-dataset study is performed using histogram equalization, and contrast limited adaptive histogram equalization with and without a learning Gabor filter. We show that under certain conditions, deep learning models can generalize well to an external dataset with F1 scores up to 86%. The best performing model shows predictive accuracy of between 75% and 96% for lung involvement scoring against an external expertly stratified dataset. From these results we identify key factors promoting deep learning generalization, being primarily the uniform acquisition of training images, and secondly diversity in CT slice position.

On the Training of Deep Neural Networks for Automatic Arabic-Text Diacritization

Automatic Arabic diacritization is one of the most important and challenging problems in Arabic natural language processing (NLP). Recurrent neural networks (RNNs) have proved recently to achieve state-of-the-art results for sequence transcription problems in general, and Arabic diacritization in specific. In this work, we investigate the effect of varying the size of the training corpus on the accuracy of diacritization. We produce a cleaned corpus of approximately 550k sequences extracted from the full dataset of Tashkeela and use subsets of this corpus in our training experiments. Our base model is a deep bidirectional long short-term memory (BiLSTM) RNN that transcribes undiacritized sequences of Arabic letters with fully diacritized sequences. Our experiments show that error rates improve as the size of training corpus increases. Our best performing model achieves average diacritic and word error rates of 1.45% and 3.89%, respectively. When compared with state-of-the-art diacritization systems, we reduce the word error rate by 12% over the best published results.

Sentiment analysis of Chinese text based on Elmo-RNN model

Traditional word embedding technologies such as Word2Vec and GloVe can only produce a single semantic vector, and can not get the specific meaning of polysemous words in the text combined with context. To solve this problem, a sentiment analysis model of Chinese text based on Elmo and recurrent neural network is proposed. The model uses Elmo model to learn the pre training corpus, and the BiLSTM network structure in Elmo model makes the word vector generated by Elmo model have context sensitive characteristics, which is more accurate in the expression of polysemous words, and this is not achieved by the traditional word embedding technology. Then, the model uses the recurrent neural network to extract the deep-seated features of the word vector and fuse the features. Finally, the softmax function is used to realize the sentiment classification of text. Experimental results show that Elmo-RNN can effectively improve the accuracy of text sentiment analysis.

News Video Title Extraction Algorithm Based on Deep Learning

In order to better serve the needs of news business, researchers apply the information extraction technology of news headlines to the news field to assist decision-making. This article designs the shared convolutional layer and RPN network in the detection network respectively, and improves the depth of the shared convolutional layer, using VGGNet instead of ZFNet. A classification method incorporating semantic enhancement is designed for fine-grained topic category news classification. Combined with the idea of modular fusion mechanism, a semantic-enhanced classification model Multiple-Fusion is proposed. The Bert module replaces the traditional Word2Vec for semantic vector characterization, and introduces Bi-LSTM to adaptively extract context features, strengthens feature expression through the self-attention mechanism and adjusts network weights, and finally makes the model achieve accurate classification. This article designs a novel word-level data enhancement strategy for text data enhancement, which solves the problems of fewer training corpus samples and model overfitting. This article proposes a video target extraction method based on deep learning. The algorithm framework includes particle filtering, pre-training features, convolutional neural networks, discriminative classifiers, and online parameter updates. This method deeply combines deep learning models and traditional target extraction methods and frameworks. The experimental results show that the method in this article has relatively outstanding performance, and can adapt to many interferences encountered in the extraction process and the change of the target itself, and has strong robustness.

Language Adaptation for Entity Relation Classification via Adversarial Neural Networks

Entity relation classification aims to classify the semantic relationship between two marked entities in a given sentence, and plays a vital role in various natural language processing applications. However, existing studies focus on exploiting mono-lingual data in English, due to the lack of labeled data in other languages. How to effectively benefit from a richly-labeled language to help a poorly-labeled language is still an open problem. In this paper, we come up with a language adaptation framework for cross-lingual entity relation classification. The basic idea is to employ adversarial neural networks (AdvNN) to transfer feature representations from one language to another. Especially, such a language adaptation framework enables feature imitation via the competition between a sentence encoder and a rival language discriminator to generate effective representations. To verify the effectiveness of AdvNN, we introduce two kinds of adversarial structures, dual-channel AdvNN and single-channel AdvNN. Experimental results on the ACE 2005 multilingual training corpus show that our single-channel AdvNN achieves the best performance on both unsupervised and semi-supervised scenarios, yield- ing an improvement of 6.61% and 2.98% over the state-of-the-art, respectively. Compared with baselines which directly adopt a machine translation module, we find that both dual-channel and single-channel AdvNN significantly improve the performances (F1) of cross-lingual entity relation classification. Moreover, extensive analysis and discussion demonstrate the appropriateness and effectiveness of different parameter settings in our language adaptation framework.

Exploring Significant Characteristics and Models for Classification of Structure Function of Academic Documents

With the increasing abundance of literature resources, how to acquire knowledge elements efficiently and accurately is the key to achieving accurate literature retrieval and utilization of available literature resources. The identification of the structure function of academic documents is a fundamental work to meet the above requirements. In this study, the proceedings of the Association for Computational Linguistics (ACL) conferences are used as the primitive corpus, and the training corpus of chapter category is obtained by manual annotation. Based on the chapter titles and the in-chapter texts, traditional machine learning and deep learning models are both used for classifier training. Our results show that the title of a chapter is more beneficial to the identification of the structure function of academic documents than the in-chapter texts. The highest F1 value in our experiments is 0.9249, which is obtained on the traditional logistic regression (LR) and support vector machine (SVM) models (slightly higher than on the convolutional neural network [CNN]). And through the experiment of adding other chapter characteristics based on the traditional model, we find that combining the relative position of chapters can effectively improve the classification performance. Finally, this study compares the results of experimental groups with different methods, analyzes the misclassification of the structure function of academic documents, and points out the main direction to improve the classification performance in the future.

CPM: A large-scale generative Chinese Pre-trained language model

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570 GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6 billion parameters and 100 GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning. The code and parameters are available at https://github.com/TsinghuaAI/CPM.

Recognition System for Libyan Entity Names

Named Entity Recognition (NER) is a computational linguistic concept that is used to find and classify appropriate nouns in a text such as person names, geographical locations, and organizations. Such a concept is fundamental in the field of natural language processing. In Libya, many private and public institutions suffer from using the proper translation of entity names from Arabic language into English. Therefore, in this paper, we are concerned with analyzing Arabic articles to extract and recognize entity names. A recognition system is developed for recognizing names of persons, academic institutions, and cities in Libya. At first, a training corpus and dictionaries are built for the intended entity names in this research. Then, the aspects of the entity names are studied, and their patterns and rules are designed. Then, the implementation is performed using Nooj linguistic language. The recognition of person names and Libyan cities and academic institutions was carried out. Statistics showed the frequencies of the appearance rate of person names, academic institutions, and cities in our training corpus. The obtained results are promised and met the research goals for tackling the problem of Arabic named entity recognition.

Uncovering Media Bias via Social Network Learning

It is known that media outlets, such as CNN and FOX, have intrinsic political bias that is reflected in their news reports. The computational prediction of such bias has broad application prospects. However, the prediction is difficult via directly analyzing the news content without high-level context. In contrast, social signals (e.g., the network structure of media followers) provide inspiring cues to uncover such bias. In this article, we realize the first attempt of predicting the latent bias of media outlets by analyzing their social network structures. In particular, we address two key challenges: network sparsity and label sparsity . The network sparsity refers to the partial sampling of the entire follower network in practical analysis and computing, whereas the label sparsity refers to the difficulty of annotating sufficient labels to train the prediction model. To cope with the network sparsity, we propose a hybrid sampling strategy to construct a training corpus that contains network information from micro to macro views. Based on this training corpus, a semi-supervised network embedding approach is proposed to learn low-dimensional yet effective network representations. To deal with the label sparsity, we adopt a graph-based label propagation scheme to supplement the missing links and augment label information for model training. The preceding two steps are iteratively optimized to reinforce each other. We further collect a large-scale dataset containing social networks of 10 media outlets together with about 300,000 followers and more than 5 million connections. Over this dataset, we compare our model to a range of state of the art. Superior performance gains demonstrate the merits of the proposed approach. More importantly, the experimental results and analyses confirm the validity of our approach for the computerized prediction of media bias.

Part-of-Speech Tagging via Deep Neural Networks for Northern-Ethiopic Languages

Deep Neural Networks (DNNs) have proven to be especially successful in the area of Natural Language Processing (NLP) and Part-Of-Speech (POS) tagging—which is the process of mapping words to their corresponding POS labels depending on the context. Despite recent development of language technologies, low-resourced languages (such as an East African Tigrinya language), have received too little attention. We investigate the effectiveness of Deep Learning (DL) solutions for the low-resourced Tigrinya language of the Northern-Ethiopic branch. We have selected Tigrinya as the testbed example and have tested state-of-the-art DL approaches seeking to build the most accurate POS tagger. We have evaluated DNN classifiers (Feed Forward Neural Network – FFNN, Long Short-Term Memory method – LSTM, Bidirectional LSTM, and Convolutional Neural Network – CNN) on a top of neural word2vec word embeddings with a small training corpus known as Nagaoka Tigrinya Corpus. To determine the best DNN classifier type, its architecture and hyper-parameter set both manual and automatic hyper-parameter tuning has been performed. BiLSTM method was proved to be the most suitable for our solving task: it achieved the highest accuracy equal to 92% that is 65% above the random baseline.

Candidate fusion: Integrating language modelling into a sequence-to-sequence handwritten word recognition architecture

Sequence-to-sequence models have recently become very popular for tackling handwritten word recognition problems. However, how to effectively integrate an external language model into such recognizer is still a challenging problem. The main challenge while training a language model is to deal with the language model corpus which is usually different to the one used for training the handwritten word recognition system. Thus, the bias between both word corpora leads to incorrectness on the transcriptions, providing similar or even worse performances on the recognition task. In this work, we introduce Candidate Fusion, a novel way to integrate an external language model to a sequence-to-sequence architecture. Moreover, it provides suggestions from an external language knowledge, as a new input to the sequence-to-sequence recognizer. Hence, Candidate Fusion provides two improvements. On the one hand, the sequence-to-sequence recognizer has the flexibility to not only combine the information from itself and the language model, but also choose the importance of the information provided by the language model. On the other hand, the external language model has the ability to adapt itself to the training corpus and even learn the most common errors produced from the recognizer. Finally, by conducting comprehensive experiments, the Candidate Fusion proves to outperform the state-of-the-art language models for handwritten word recognition tasks.

Grau de pertencimento como insumo para classificação automática de textos: uma abordagem sintática

Grouping documents into categories is one of the solutions adopted to streamline the information retrieval process, which is increasingly relevant due to the large amount of information available today. The manual localization of documents of a specific theme, available in digital repositories, involves reading the title, abstract and keywords, in addition to further detailed evaluation in order to identify whether the publication belongs to the desired thematic axis. Considering the number of publications in a digital repository, manually locating all the desired texts on a given topic can be laborious and time-consuming. This research proposes an architecture for automatic classification of texts that is based on syntactic questions, that is, it undertakes a comparison of n-grams, which are combinations of n-pairs of words that are identified throughout the text. An exploratory applied research was carried out, which applied a type of supervised learning, fundamentally based on the document representation model called bag-of-words (BoW). The paper’s macro objective was to classify texts in general, according to pre-defined categories, by generating and comparing degrees of belonging between texts, as one of the key criteria. The results of these comparisons, using n-gram = 3, demonstrate that in the use of classifications by n-grams, the greater the number of grams, and with the removal of the stop words, we obtain a reduced degree of belonging, demonstrating greater rigor in identifying the match during the classification. In order to have greater confidence in the results, a larger training corpus is necessary to expand the number of words that characterize the pre-defined categories, to be used in the classification of the texts.

On the Construction of Web NER Model Training Tool based on Distant Supervision

Named entity recognition (NER) is an important task in natural language understanding, as it extracts the key entities (person, organization, location, date, number, etc.) and objects (product, song, movie, activity name, etc.) mentioned in texts. However, existing natural language processing (NLP) tools (such as Stanford NER) recognize only general named entities or require annotated training examples and feature engineering for supervised model construction. Since not all languages or entities have public NER support, constructing a tool for NER model training is essential for low-resource language or entity information extraction. In this article, we study the problem of developing a tool to prepare training corpus from the Web with known seed entities for custom NER model training via distant supervision. The major challenge of automatic labeling lies in the long labeling time due to large corpus and seed entities as well as the concern to avoid false positive and false negative examples due to short and long seeds. To solve this problem, we adopt locality-sensitive hashing (LSH) for various length of seed entities. We conduct experiments on five types of entity recognition tasks, including Chinese person names, food names, locations, points of interest (POIs), and activity names to demonstrate the improvements with the proposed Web NER model construction tool. Because the training corpus is obtained by automatic labeling of the seed entity–related sentences, one could use either the entire corpus or the positive only sentences for model training. Based on the experimental results, we found the decision should depend on whether traditional linear chained conditional random fields (CRF) or deep neural network–based CRF is used for model training as well as the completeness of the provided seed list.

Automatic Recognition of Student Emotions Based on Deep Neural Network and Its Application in Depression Detection

The mental health rating scales combined with psychological expert consultation are easily influenced by subjective factors from psychological experts and lack objectivity and scientificalness. Due to the serious influence of depression tendency on learning and living of college students, a novel DNN network model framework based on context emotion information is designed to achieve the automatic auxiliary emotion classification. More corpus training sample can be received by heightening the sample length without reducing samples' number. Firstly, the existing sample feature is input into the recognition model to encode depression related features, and then the MADN features of the samples in the two adjacent segments are input into the above trained model in order for fine-tuning and optimization. Compared with the existing optimal method, the proposed model improves the recognition accuracy in the diagnosis of depression. From the analysis of the experimental results, it is known that deep learning network can monitor the emotional state of college students with high precision, which can accurately identify the patients with depression. The deep learning model can take effective measures to prevent the depression of college students, and discover the depression of college students, alleviate and treat the depression of college students, reduce the depression rate of college students.

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera.

We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. We propose a new encoder-decoder architecture with a novel multi-branch decoder designed specifically to account for the varying uncertainty in 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. To tackle the severe lack of labelled training data for egocentric 3D pose estimation we also introduced a large-scale photo-realistic synthetic dataset. xR-EgoPose offers 383K frames of high quality renderings of people with diverse skin tones, body shapes and clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.

“Corpus can be tricky”: revisiting teacher attitudes towards corpus-aided language learning and teaching

The present study explores the attitudes of novice teachers towards corpus-aided language learning and teaching in an undergraduate writing course for multilingual students at a large US public university. The participating instructors facilitated approximately 75 minutes of corpus training for their students and implemented 4-6 corpus activities over the course of the 16-week term. Participating instructors completed an initial survey documenting existing knowledge of, experiences with, and attitudes towards corpus-aided language learning and teaching, a second open-ended questionnaire at the mid-point of the academic term exploring their experiences, and a final instrument assessing how their attitudes had potentially evolved, either positively or negatively, and their likelihood for continued implementation of corpus-aided pedagogy in their classrooms. To offer a balanced perspective, data was also collected from three teachers who chose not to implement corpus activities in their courses. Findings display generally positive attitudes towards corpus study but also reveal emergent tensions regarding the use of ready-made corpus activities and the key affordances of discovery, authenticity, and autonomy often forwarded in support of corpus pedagogy.

TermInformer: unsupervised term mining and analysis in biomedical literature.

Terminology is the most basic information that researchers and literature analysis systems need to understand. Mining terms and revealing the semantic relationships between terms can help biomedical researchers find solutions to some major health problems and motivate researchers to explore innovative biomedical research issues. However, how to mine terms from biomedical literature remains a challenge. At present, the research on text segmentation in natural language processing (NLP) technology has not been well applied in the biomedical field. Named entity recognition models usually require a large amount of training corpus, and the types of entities that the model can recognize are limited. Besides, dictionary-based methods mainly use pre-established vocabularies to match the text. However, this method can only match terms in a specific field, and the process of collecting terms is time-consuming and labour-intensive. Many scenarios faced in the field of biomedical research are unsupervised, i.e. unlabelled corpora, and the system may not have much prior knowledge. This paper proposes the TermInformer project, which aims to mine the meaning of terms in an open fashion by calculating terms and find solutions to some of the significant problems in our society. We propose an unsupervised method that can automatically mine terms in the text without relying on external resources. Our method can generally be applied to any document data. Combined with the word vector training algorithm, we can obtain reusable term embeddings, which can be used in any NLP downstream application. This paper compares term embeddings with existing word embeddings. The results show that our method can better reflect the semantic relationship between terms. Finally, we use the proposed method to find potential factors and treatments for lung cancer, breast cancer, and coronavirus.

Neural machine translation for Tamil to English

The Tamil language is spoken by 80 million people around the world. The translation between Tamil and English leads to a significant impact by helping in the understanding of Tamil scripts, which otherwise would be a tedious, costly, and time-consuming process. Thus, developing an automated system to perform Tamil to English translation would save human time and effort. We publicly release a new high-quality corpus for standard training, evaluation, and report results experiments with two different architectures based on Encoder-Decoder to translate Tamil to English. We further tried to improve it by experimenting with pre-trained word embeddings and tuning hyperparameters. Although Google-Translator also provides Tamil to English and vice versa, our implemented architectures, along with the new dataset, completely outperformed the Google Translator with a margin of 7.5 BLEU score. Moreover, our proposed model solves out of vocabulary and polysemy problems up to a greater extent. Our dataset and implementation are available at: https://github.com/Ishikahooda/Tamil-English-Dataset

Cross-Domain Tibetan Word Segmentation Based on Deep Learning

Tibetan word segmentation is the primary task of various types of Tibetan natural language processing. However, when the test corpus and the training corpus are inconsistent, the accuracy of word segmentation will drop significantly. Therefore, in this paper firstly we build a BiLSTM-CRF Tibetan word segmentation model based on deep learning that includes dictionary features, and train the model parameters on the open domain word segmentation data. Then, taking the Tibetan medical clinical text as a small-scale word segmentation training corpus, and fine-tuning the parameters of the BiLSTM-CRF word segmentation model of the open domain corpus, and adding the domain vocabulary to the dictionary features of the model. So that the precision, recall rate and F value of the word segmentation model on the test data reached 93.21%, 92.10%, and 94.37% respectively.

Perception of brazilian Portuguese open and close mid vowels by native Russian speakers

The perception of Brazilian Portuguese (BP) mid vowels is mainly investigated by foreign researchers who work with learners of Brazilian Portuguese, native speakers of English or Spanish. In Brazil, this field has been poorly explored. This work contributes to filling this gap by investigating the perception of two Brazilian Portuguese minimal vowel pairs in the group of 103 native Russian speakers and nine French speakers, most of them residing in Brazil. The test has been designed to evaluate the perception of the mid high and low front vowels [e] - [ε], and the mid high and low back vowels [o] - [ɔ]. The test is conducted online and includes 30 trials distributed in three types of tasks: image identification (8 trials), vowel identification (4) and word discrimination (18). It also contains a short sociolinguistic questionnaire. Our findings indicate that native Russian speakers do not differentiate the Brazilian Portuguese mid vowels very well: the mean percentage of the correct answers in the Russophone group was 68%, while in the French group it was 87% and in the Brazilian control group 99%. No correlation was found between the percentage of correct answers and residence time in Brazil or type of language instruction. As next steps, we plan to investigate the differentiation of the Brazilian Portuguese open and close mid vowels by Russophones at the level of production. We also intend to create a training corpus and develop strategies for perception training of the Brazilian Portuguese mid vowels, monitoring their efficiency at different time points.