Fourier Ptychography Microscopy With Integrated Positional Misalignment Correction

Quantitative phase microscopy (QPM) is indispensable in biomedical research due to its advantages in unlabeled transparent sample thickness quantification and obtaining refractive index information. Fourier ptychographic microscopy (FPM) is among the most promising QPM methods, incorporating multi-angle illumination and iterative phase recovery for high-resolution quantitative phase imaging (QPI) of large cell populations over a wide field of-view (FOV) in a single pass. However, FPM is limited by data redundancy and sequential acquisition strategies, resulting in low imaging efficiency, which in turn limits its real-time application in in vitro label-free imaging. Here, we report a fast QPM based on Fourier ptychography (FQP-FPM), which uses an optimized annular downsampling and parallel acquisition strategy to minimize the amount of data required in the front end and reduce the iteration time of the back-end algorithm (3.3% and 4.4% of conventional FPM, respectively). Theoretical and data redundancy analyses show that FQP-FPM can realize high-throughput quantitative phase reconstruction at thrice the resolution of the coherent diffraction limit by acquiring only ten raw images, providing a precondition for in vitro label-free real-time imaging. The FQP-FPM application was validated for various in vitro label-free live-cell imaging. Cell morphology and subcellular phenomena in different periods were observed with a synthetic aperture of 0.75 NA at a 10× FOV, demonstrating its advantages and application potential for fast high-throughput QPI.

Fourier ptychographic microscopy (FPM) is a newly developed super-resolution technique, which employs angularly varying illuminations and a phase retrieval algorithm to surpass the diffraction limit of a low numerical aperture (NA) objective lens. In current FPM imaging platforms, accurate knowledge of LED matrix’s position is critical to achieve good recovery quality. Furthermore, considering such a wide field-of-view (FOV) in FPM, different regions in the FOV have different sensitivity of LED positional misalignment. In this work, we introduce an iterative method to correct position errors based on the simulated annealing (SA) algorithm. To improve the efficiency of this correcting process, large number of iterations for several images with low illumination NAs are firstly implemented to estimate the initial values of the global positional misalignment model through non-linear regression. Simulation and experimental results are presented to evaluate the performance of the proposed method and it is demonstrated that this method can both improve the quality of the recovered object image and relax the LED elements’ position accuracy requirement while aligning the FPM imaging platforms.

Quantitative phase imaging (QPI) plays a key role in many application areas such as Optical elements measuring and unstained biomedical live samples imaging. Therefore, several computational QPI methods have been developed and widely used, like transport-of-intensity equation (TIE) based techniques and differential phase contrast (DPC) based techniques. However, these phase retrieval approaches are fundamentally limited by the space-bandwidth product (SBP) of the optical microscopy system, resulting in a trade-off between the spatial resolution and field of view (FOV). Lately, this problem is effectively solved by a new computational imaging technique named Fourier ptychographic microscopy (FPM), which reconstructs high-resolution complex image from many angle-variable illuminated, low-resolution intensity images captured by a low numerical-aperture (NA) objective. Although FPM has great potential for finding application in digital pathology and cancer research, it still suffers from long acquisition time and low phase measuring accuracy. Due to the large number of low-resolution images (generally larger than 30 images) required in FPM for recovering one phase image, it is more difficult for FPM to realize real-time phase imaging comparing with TIE (at least 2-3 images) or DPC (at least 4 images). Herein, we report a real-time FPM technique based on annular illuminations (AIFPM) for quantitative phase imaging of unstained live samples in vitro. In AIFPM, we only need four low-resolution images, corresponding to four different illumination angles with 0.4NA, which equal to the NA obj . Therefore, using a 10X, 0.4NA objective lens with final effective imaging performance of 0.8 NA, we present the real-time imaging results of in vitro Hela cells mitosis and apoptosis at a frame rate of 25 Hz with a full-pitch resolution of 655 nm at a wavelength of 525 nm across a wide FOV of 1.77 mm 2 . Our work reveals an essential capability of FPM towards highspeed high-throughput phase imaging applications, such as biology and medicine screening, for the significant breakthrough in both space and time.

Fourier ptychographic microscopy (FPM) is a new, developing computational imaging technology. It can realize the quantitative phase imaging of a wide field of view and high-resolution (HR) simultaneously by means of multi-angle illumination via a light emitting diode (LED) array, combined with a phase recovery algorithm and the synthetic aperture principle. However, in the FPM reconstruction process, LED position misalignment affects the quality of the reconstructed image, and the reconstruction efficiency of the existing LED position correction algorithms needs to be improved. This study aims to improve the FPM correction method based on simulated annealing (SA) and proposes a position misalignment correction method (AA-C algorithm) using an improved phase recovery strategy. The spectrum function update strategy was optimized by adding an adaptive control factor, and the reconstruction efficiency of the algorithm was improved. The experimental results show that the proposed method is effective and robust for position misalignment correction of LED arrays in FPM, and the convergence speed can be improved by 21.2% and 54.9% compared with SC-FPM and PC-FPM, respectively. These results can reduce the requirement of the FPM system for LED array accuracy and improve robustness.

Fourier ptychographic microscopy (FPM) is a computational imaging technology used to achieve high-resolution imaging with a wide field-of-view. The existing methods of FPM suffer from the positional misalignment in the system, by which the quality of the recovered high-resolution image is determined. In this paper, a forward neural network method with correction of the positional misalignment (FNN-CP) is proposed based on TensorFlow, which consists of two models. Both the spectrum of the sample and four global position factors, which are introduced to describe the positions of the LED elements, are treated as the learnable weights in layers in the first model. By minimizing the loss function in the training process, the positional error can be corrected based on the trained position factors. In order to fit the wavefront aberrations caused by optical components in the FPM system for better recovery results, the second model is designed, in which the spectrum of the sample and coefficients of different Zernike modes are treated as the learnable weights in layers. After the training process of the second model, the wavefront aberration can be fit according to the coefficients of different Zernike modes and the high-resolution complex image can be obtained based on the trained spectrum of the sample. Both the simulation and experiment have been performed to verify the effectiveness of our proposed method. Compared with the state-of-art FPM methods based on forward neural network, FNN-CP can achieve the best reconstruction results.

Fourier ptychographic microscopy (FPM) is a newly developed computational imaging technique that can provide gigapixel images with both high resolution (HR) and wide field of view (FOV). However, there are two possible reasons for position misalignment, which induce a degradation of the reconstructed image. The first one is the position misalignment of the LED array, which can largely be eliminated during the experimental system building process. The more important one is the segment-dependent position misalignment. Note that, this segment-dependent positional misalignment still exists, even after we correct the central coordinates of every small segment. In this paper, we carefully analyze this segment-dependent misalignment and find that this global shift matters more, compared with the rotational misalignments. According to this fact, we propose a robust and fast method to correct the two factors of position misalignment of the FPM, termed as misalignment correction for the FPM misalignment correction (mcFPM). Although different regions in the FOV have different sensitivities to the position misalignment, the experimental results show that the mcFPM is robust with respect to the elimination of each region. Compared with the state-of-the-art methods, the mcFPM is much faster.

Fourier ptychography microscopy (FPM) is a recently developed computational imaging approach which surpasses the resolution barrier of a low numerical aperture (NA) imaging system. It is a powerful tool due to its ability to achieve super resolution of complex sample function, pupil aberration, LED misalignment, and beyond. However, recent studies have focused more on the optimization algorithms and set-ups instead of its theoretical background. Although some imaging laws about FPM have already been set forth, the formulas and laws are not fully defined, and the connection between diffraction theory and Fourier optics has a gap. Therefore, there exist a need for comprehensive research on physical and mathematical basis of FPM for future applications. Keeping this goal in mind, this manuscript utilizes scalar field diffraction theory to rigorously study the relationship between wavelength, the propagation mode, illumination direction of the incident wave, sample structure information and the direction of the output wave. The theoretical analysis of diffraction imaging in FPM provides a clear physical basis for not only the FPM systems, but also for the ptychography iterative engine (PIE) and any other coherent diffraction imaging techniques and systems. It can help to find the source of noise and therefore improve image quality in FPM technique and systems.

Over the past decade, the research field of Fourier Ptychographic Microscopy (FPM) has seen numerous innovative developments that significantly expands its utility. Here, we report a high numerical aperture (NA) FPM implementation that incorporates some of these innovations to achieve a synthetic NA of 1.9 - close to the maximum possible synthetic NA of 2 for a free space FPM system. At this high synthetic NA, we experimentally found that it is vital to homogenize the illumination field in order to achieve the best resolution. Our FPM implementation has a full pitch resolution of 266 nm for 465 nm light, and depth of field of 3.6 µm. In comparison, a standard transmission microscope (incoherent) with close to maximum possible NA of 0.95 has a full pitch resolution of 318 nm for 465 nm light, and depth of field of 0.65 µm. While it is generally assumed that a free-space coherent imaging system and a free-space incoherent imaging system operating at their respective maximum NA should give comparable resolution, we experimentally find that an FPM system significantly outperforms its incoherent standard microscopy counterpart in resolution by a factor of 20%. Coupled with FPM's substantially longer effective depth of field (5.5 times longer), our work indicates that, in the near-maximum NA operation regime, the FPM has significant resolution and depth of field advantages over incoherent standard microscopy.

Fourier ptychography microscopy (FPM) shares its roots with the synthetic aperture technique and phase retrieval method, and is a recently developed computational microscopic super-resolution technique. By turning on the light-emitting diode (LED) elements sequentially and acquiring the corresponding images that contain different spatial frequencies, FPM can achieve a wide field-of-view (FOV), high-spatial-resolution imaging and phase recovery simultaneously. Conventional FPM assumes that the sample is sufficiently thin and strictly in focus. Nevertheless, even for a relatively thin sample, the non-planar distribution characteristics and the non-ideal position/posture of the sample will cause all or part of FOV to be defocused. In this paper, we proposed a fast digital refocusing and depth-of-field (DOF) extended FPM strategy by taking the advantages of image lateral shift caused by sample defocusing and varied-angle illuminations. The lateral shift amount is proportional to the defocus distance and the tangent of the illumination angle. Instead of searching the optimal defocus distance with the optimization search strategy, which is time consuming, the defocus distance of each subregion of the sample can be precisely and quickly obtained by calculating the relative lateral shift amounts corresponding to different oblique illuminations. And then, digital refocusing strategy rooting in the angular spectrum (AS) method is integrated into FPM framework to achieve the high-resolution and phase information reconstruction for each part of the sample, which means the DOF of the FPM can be effectively extended. The feasibility of the proposed method in fast digital refocusing and DOF extending is verified in the actual experiments with the USAF chart and biological samples.

Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. Acquiring color images of histology slides often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learning approach for performing unsupervised image-to-image translation of FPM reconstructions. A cycle-consistent adversarial network with multiscale structure similarity loss is trained to perform virtual brightfield and fluorescence staining of the recovered FPM images. In the training stage, we feed the network with two sets of unpaired images: (1) monochromatic FPM recovery and (2) color or fluorescence images captured using a regular microscope. In the inference stage, the network takes the FPM input and outputs a virtually stained image with reduced coherent artifacts and improved image quality. We test the approach on various samples with different staining protocols. High-quality color and fluorescence reconstructions validate its effectiveness.

Adaptive denoising method for Fourier ptychographic microscopy

Fourier ptychography microscopy (FPM) is a lately developed technique, which achieves wide field, high resolution, and phase imaging, simultaneously. FPM stitches together the captured low-resolution images corresponding to angular varying illuminations in Fourier domain utilizing the concept of synthetic aperture and phase retrieval algorithms, which can surpass the space-bandwidth product limit of the objective lens and reconstruct a high-resolution complex image. In general FPM system, the LED source is important for the reconstructed quality and it is sensitive to the positions of each LED element. We find that the random positional deviations of each LED element can bring errors in reconstructed results, which is relative to a feedback parameter. To improve the reconstruction rate and correct random deviations, we combine an initial phase guess and a feedback parameter based on differential phase contrast and extended ptychographical iterative engine to propose an optimized iteration process for FPM. The simulated and experimental results indicate that the proposed method shows the reliability and validity towards the random deviations yet accelerates the convergence. More importantly, it is verified that this method can accelerate the convergence, reduce the requirement of LED array accuracy, and improve the quality of the reconstructed results.

The Fourier ptychographic microscopy (FPM) technique provides high-resolution images by combining a traditional imaging system, e.g. a microscope or a 4f-imaging system, with a multiplexing illumination system, e.g. an LED array and numerical image processing for enhanced image reconstruction. In order to numerically combine images that are captured under varying illumination angles, an iterative phase-retrieval algorithm is often applied. However, in practice, the performance of the FPM algorithm degrades due to the imperfections of the optical system, the image noise caused by the camera, etc. To eliminate the influence of the aberrations of the imaging system, an embedded pupil function recovery (EPRY)-FPM algorithm has been proposed [Opt. Express 22, 4960–4972 (2014)]. In this paper, we study how the performance of FPM and EPRY-FPM algorithms are affected by imperfections of the illumination system using both numerical simulations and experiments. The investigated imperfections include varying and non-uniform intensities, and wavefront aberrations. Our study shows that the aberrations of the illumination system significantly affect the performance of both FPM and EPRY-FPM algorithms. Hence, in practice, aberrations in the illumination system gain significant influence on the resulting image quality.

High-throughput quantitative phase imaging (QPI) is essential to cellular phenotypes characterization as it allows high-content cell analysis and avoids adverse effects of staining reagents on cellular viability and cell signaling. Among different approaches, Fourier ptychographic microscopy (FPM) is probably the most promising technique to realize high-throughput QPI by synthesizing a wide-field, high-resolution complex image from multiple angle-variably illuminated, low-resolution images. However, the large dataset requirement in conventional FPM significantly limits its imaging speed, resulting in low temporal throughput. In this talk, we report two optimum illumination schemes for FPM to achieve high-speed or even single-shot QPI. We present the high-speed imaging results of in vitro Hela cells mitosis and apoptosis at a frame rate of 25 Hz with a full-pitch resolution of 655 nm at a wavelength of 525 nm (effective NA = 0.8) across a wide field-of-view (FOV) of 1.77 mm2, corresponding to a space–bandwidth–time product of 411 megapixels per second. We also discuss how FPM can be extended to optical diffraction tomography (ODT) under Born or Rytov approximation, achieving super and depth resolved 3D imaging over a wide FOV.

Digital pathology with Fourier ptychography