Tooth-Marked Tongue Recognition Based on Mask Scoring R-CNN

Abstract

Tooth-marked tongue is one of the important tongue features in Traditional Chinese Medicine tongue diagnosis. It reflects the dysfunction of liver, spleen and kidney, and contains rich pathological information. However, the recognition of tooth-marked tongue is challenging. Most existing methods only focus on the classification of tooth-marked tongue, and the exact location and number of teeth marks were not involved, which had no great practical significance for the subsequent treatment based on syndrome differentiation. In this paper, we try to solve these problems by proposing a method based on Mask Scoring R-CNN framework and transfer learning which can extract, visualize the teeth marks and identify the number of teeth marks. First, the tongue image is fed into residual network of depth 101 layers (ResNet-101) to generate regions of interest (RoIs) via region proposal network (RPN) and RoI feature via RoIAlign. Next, MaskIOU is then predicted using the predicted mask and RoI characteristics as inputs. The proposed approach achieved F1 score of 0.95. According to these experimental results, the approach can robustly detect and segment teeth marks, which can provide a basis for the severity analysis of tooth-marked tongue.