Deep Learning with Metadata Augmentation for Classification of Diabetic Retinopathy Level

Abstract

Diabetic retinopathy (DR) is one of the most important and embarrassing problems in the medical, psychological, and social aspects of the working-age population in the world. The DR severity classification problem for single modality (with image input) model and multi modality (with image and text inputs) model is considered on the basis of RetinaMNIST dataset. The influence of additional data like subjective “patient” opinion or “expert” opinions about patient health state (that provide “data leakage” on some classes) can be helpful in some practical situations. These opinions were simulated by additional (augmented) metadata from simulated questionnaires. As a result the following variants of input values and the correspondent models were prepared: single modality model (SM) with input images only, and multi modality models with input images and patient opinion text like Multi modality model with Patient opinion (MP), Multi modality model with Expert opinion (ME), and Multi modality model with Patient and Expert opinions (MPE). All these multi modality models (MP, ME, MPE) allowed us to reach the various statistically significant improvements of classification performance by AUC value for all classes in the range from 4% to 27% that are rather beyond the limits of the standard deviation of 2–3% measured by cross-validation and can be estimated as significant ones. In general, this approach based on metadata augmentation, namely, usage of the additional modalities with “data leakage” on the extreme classes, for example, with the lowest (Class 0) and highest (Class 4) DR severity, and their combinations could be useful strategy for the better classification of some hardly classified DR severities like Classes 1–3 here and in the more general context.