Improving Bert-Based Model for Medical Text Classification with an Optimization Algorithm

Abstract

In the field of Natural Language Processing (NLP), automatic text classification is a classic topic that involves classifying textual material into predetermined categories based on its content. These models have been effectively applied to data containing a large number of dimensional features, some of which are inherently sparse. Machine learning and other statistical approaches, such as those used in medical text categorization, appear to be extremely successful for these tasks. However, much human work is still required to classify a large collection of training data. Recent research has shown the usefulness of pre-trained language models such as Bidirectional Encoder Representations from Transformers (BERT), all of which have demonstrated their ability to reduce the amount of work required for feature engineering. However, directly using the pre-trained BERT model in the classification task does not result in a statistically significant increase in performance. To improve the result of the BERT model, we propose an optimal deep learning model based on a BERT model and hyperparameter selection. The model consists of three steps: (1) processing medical text; (2) extracting medical text features using a BERT architecture; and (3) selecting hyperparameters for the Deep Learning model based on a Particle Swarm Optimization (PSO) algorithm. Finally, our approach uses a k-Nearest Neighbors algorithm (KNN) model to predict the matching response. Experiments conducted on the Hallmarks dataset have shown that the proposed method significantly increases the accuracy of the results.