Enhancing structural illumination microscopy with hybrid CNN-transformer and dynamic frequency loss

Abstract

Deep learning techniques have augmented structured illumination microscopy (SIM) imaging by employing convolutional neural networks (CNNs) for the super-resolution reconstruction of SIM images. However, due to the intrinsic shortcomings of CNNs, i.e. limited receptive field and inadaptability to input content, their reconstruction faithfulness and imaging resolution are limited. Recently, another class of neural backbone, Transformer, has shown complementary advantages to CNNs. To this end, this work aims to borrow the recent Transformers to enhance CNN-based SIM reconstruction, thereby fully unleashing the power of neural networks for SIM imaging. Concretely, we first experimentally explore the pros and cons of CNNs and Transformers for SIM reconstruction. Then, we propose a Hybrid CNN-Transformer (HCT) model, to inherit merits of the both. To further encourage the recovery of high frequencies down-modulated by patterned illumination, we also devise a dynamic frequency loss based on the phase spectrum information, helping models focus on hard frequencies adaptively. Experiments on diverse subcellular structures demonstrate that the proposed method can evidently boost SIM reconstruction performance, even under low-light conditions or with fewer raw frames.