ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning

Charles N Christensen,Pietro Lio,Meng Lu,Clemens F Kaminski,Edward N Ward

doi:10.1364/boe.414680

Abstract

Structured illumination microscopy (SIM) has become an important technique for optical super-resolution imaging because it allows a doubling of image resolution at speeds compatible with live-cell imaging. However, the reconstruction of SIM images is often slow, prone to artefacts, and requires multiple parameter adjustments to reflect different hardware or experimental conditions. Here, we introduce a versatile reconstruction method, ML-SIM, which makes use of transfer learning to obtain a parameter-free model that generalises beyond the task of reconstructing data recorded by a specific imaging system for a specific sample type. We demonstrate the generality of the model and the high quality of the obtained reconstructions by application of ML-SIM on raw data obtained for multiple sample types acquired on distinct SIM microscopes. ML-SIM is an end-to-end deep residual neural network that is trained on an auxiliary domain consisting of simulated images, but is transferable to the target task of reconstructing experimental SIM images. By generating the training data to reflect challenging imaging conditions encountered in real systems, ML-SIM becomes robust to noise and irregularities in the illumination patterns of the raw SIM input frames. Since ML-SIM does not require the acquisition of experimental training data, the method can be efficiently adapted to any specific experimental SIM implementation. We compare the reconstruction quality enabled by ML-SIM with current state-of-the-art SIM reconstruction methods and demonstrate advantages in terms of generality and robustness to noise for both simulated and experimental inputs, thus making ML-SIM a useful alternative to traditional methods for challenging imaging conditions. Additionally, reconstruction of a SIM stack is accomplished in less than 200 ms on a modern graphics processing unit, enabling future applications for real-time imaging. Source code and ready-to-use software for the method are available at http://ML-SIM.github.io.

Highlights

Structured illumination microscopy (SIM) is an optical super-resolution imaging technique that was proposed more than a decade ago [1,2,3,4,5], and continues to stand as a powerful alternative to techniques such as Single Molecule Localization Microscopy (SMLM) [6,7] and Stimulated Emission Depletion (STED) microscopy [8]
We demonstrate and validate a SIM reconstruction method, ML-SIM, which takes advantage of transfer learning by training a model in an auxiliary domain consisting of simulated images and generalises to the target task of reconstructing experimental SIM images with no fine-tuning or retraining necessary
The training data was generated by simulating raw SIM image data from images obtained from common image repositories, serving as ground truths

Summary

Introduction

Structured illumination microscopy (SIM) is an optical super-resolution imaging technique that was proposed more than a decade ago [1,2,3,4,5], and continues to stand as a powerful alternative to techniques such as Single Molecule Localization Microscopy (SMLM) [6,7] and Stimulated Emission Depletion (STED) microscopy [8]. The super-resolution image is reconstructed by isolating the three superimposed spectra and shifting them into their correct location in frequency space. Isolating the three frequency spectra is mathematically analogous to solving three simultaneous equations. This requires the acquisition of three raw images, with the phase of the SIM patterns shifted with respect to one another along the direction of k0. These phase shifts are in increments of 2π/3 to ensure that the averaged illumination, i.e. the sum of all patterns, yields a homogeneous illumination field. To obtain isotropic resolution enhancement in all directions, this process is repeated twice, rotating the patterns by 2π/3 each time, to yield a total of 9 images (i.e. 3 phase shifts for each of the 3 pattern orientations)

Methods

Results

Conclusion