Artificial intelligence based Chinese clinical trials eligibility criteria classification

Abstract

Subject recruitment is a key component that affects the progress and results of clinical trials, and generally conducted with eligibility criteria (includes inclusion criteria and exclusion criteria). The semantic category analysis of eligibility criteria can help optimizing clinical trials design and building automated patient recruitment system. This study explored the automatic semantic categories classification of Chinese eligibility criteria based on artificial intelligence by academic shared task. We totally collected 38 341 annotated eligibility criteria sentences and predefined 44 semantic categories. A total of 75 teams participated in competition, with 27 teams having submitted system outputs. Based on the results, we found out that most teams adopted mixed models. The mainstream resolution was applying pre-trained language models capable of providing rich semantic representation, which were combined with neural network models and used to fine-tune the models with reference to classifier tasks, and finally improved classification performance could be obtained by ensemble modeling. The best-performing system achieved a macro F1 score of 0.81 by using a pre-trained language model, i.e. bidirectional encoder representations from transformers (BERT) and ensemble modeling. With the error analysis we found out that from the point of data processing steps the data pre-processing and post-processing were very important for classification, while from the point of data volume these categories with less data volume showed lower classification performance. Finally, we hope that this study could provide a valuable dataset and state-of-the-art result for the research of Chinese medical short text classification.