Describe like a pathologist: Glomerular immunofluorescence image caption based on hierarchical feature fusion attention network

Abstract

Clinically, the histopathological assessment of glomerular immunofluorescence (IF) images is widely-used and indispensable for diagnosis of most chronic kidney diseases, however the histopathological assessment always requires human’s judgements and suffers from the inter-observer variability. In this paper, firstly, a hierarchical feature fusion attention network (HFANet) is proposed to enhance the feature extraction capability for glomerular IF images, which concatenates the weighted feature maps extracted from different depths. Secondly, a proposed intensity equalization (IE) algorithm improves the qualities of images and generated IF descriptions by adjusting all images to a uniform and appropriate intensity level. Thirdly, an encoder–decoder architecture with the HFANet as the core is utilized to generate descriptions that emulate the reading habits of clinicians. At last, we construct a large dataset containing 11,506 glomerular IF images and their corresponding IF descriptions to support the model training. In the testing phase, the proposed framework achieves the BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and CIDEr of 0.704, 0.563, 0.444, 0.387, 0.760, and 3.594, respectively. Extensive comparative experiments and ablation studies reveal the effectiveness of proposed modules and the generalization performance of the framework for various cases and IF descriptive indicators.