Large Pre-trained Language Models Research Articles

Applying Deep Learning Techniques for Sentiment Analysis to Assess Sustainable Transport

Users voluntarily generate large amounts of textual content by expressing their opinions, in social media and specialized portals, on every possible issue, including transport and sustainability. In this work we have leveraged such User Generated Content to obtain a high accuracy sentiment analysis model which automatically analyses the negative and positive opinions expressed in the transport domain. In order to develop such model, we have semiautomatically generated an annotated corpus of opinions about transport, which has then been used to fine-tune a large pretrained language model based on recent deep learning techniques. Our empirical results demonstrate the robustness of our approach, which can be applied to automatically process massive amounts of opinions about transport. We believe that our method can help to complement data from official statistics and traditional surveys about transport sustainability. Finally, apart from the model and annotated dataset, we also provide a transport classification score with respect to the sustainability of the transport types found in the use case dataset.

Importance-Aware Learning for Neural Headline Editing

Many social media news writers are not professionally trained. Therefore, social media platforms have to hire professional editors to adjust amateur headlines to attract more readers. We propose to automate this headline editing process through neural network models to provide more immediate writing support for these social media news writers. To train such a neural headline editing model, we collected a dataset which contains articles with original headlines and professionally edited headlines. However, it is expensive to collect a large number of professionally edited headlines. To solve this low-resource problem, we design an encoder-decoder model which leverages large scale pre-trained language models. We further improve the pre-trained model's quality by introducing a headline generation task as an intermediate task before the headline editing task. Also, we propose Self Importance-Aware (SIA) loss to address the different levels of editing in the dataset by down-weighting the importance of easily classified tokens and sentences. With the help of Pre-training, Adaptation, and SIA, the model learns to generate headlines in the professional editor's style. Experimental results show that our method significantly improves the quality of headline editing comparing against previous methods.

New Research on Transfer Learning Model of Named Entity Recognition

This paper integrates the current Google’s most powerful NLP transfer learning model BERT with the traditional state-of-the-art BiLSTM-CRF model to solve the problem of named entity recognition. A bi-directional LSTM model can consider an effectively infinite amount of context on both sides of a word and eliminates the problem of limited context that applies to any feed-forward models. Google’s model applied a feedforward neural network, causing its performance to weaken. We seek to solve these issues by proposing a more powerful neural network model named BT-BiLSTM. The new neural network model has obtained F1 scores on three Chinese datasets exceeds the previous BiLSTM-CRF model, especially on the value of recall. It shows the great value of the combination of large scale none-labelled data pre-trained language model with named entity recognition, which inspire new ideas on other future work.