Fourier Ptychographic Microscopy Research Articles

High-performance heterogeneous FPGA data-flow architecture for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique that can achieve both high-resolution and a wide field-of-view via a sequence of low-resolution images. FPM is a complex iterative process, and it is difficult to meet the needs of rapid reconstruction imaging with the conventional FPM deployed on general purpose processors. In this paper, we propose a high-performance heterogeneous field-programmable gate array (FPGA) architecture based on the principle of full pipeline and the data-flow structure for the iterative reconstruction procedure of FPM. By optimizing the architecture network at gate-level logic circuits, the running time of the FPGA-based FPM reconstruction procedure is nearly 20 times faster than conventional methods. Our proposed architecture can be used to develop FPM imaging equipment that meets resource and performance requirements.

Cellular analysis using label-free parallel array microscopy with Fourier ptychography.

Quantitative phase imaging (QPI) is an ideal method to non-invasively monitor cell populations and provide label-free imaging and analysis. QPI offers enhanced sample characterization and cell counting compared to conventional label-free techniques. We demonstrate this in the current study through a comparison of cell counting data from digital phase contrast (DPC) imaging and from QPI using a system based on Fourier ptychographic microscopy (FPM). Our FPM system offers multi-well, parallel imaging and a QPI-specific cell segmentation method to establish automated and reliable cell counting. Three cell types were studied and FPM showed improvement in the ability to resolve fine details and thin cells, despite limitations of the FPM system incurred by imaging artifacts. Relative to manually counted fluorescence ground-truth, cell counting results after automated segmentation showed improved accuracy with QPI over DPC.

Deep Multi-Feature Transfer Network for Fourier Ptychographic Microscopy Imaging Reconstruction.

Fourier ptychographic microscopy (FPM) is a potential imaging technique, which is used to achieve wide field-of-view (FOV), high-resolution and quantitative phase information. The LED array is used to irradiate the samples from different angles to obtain the corresponding low-resolution intensity images. However, the performance of reconstruction still suffers from noise and image data redundancy, which needs to be considered. In this paper, we present a novel Fourier ptychographic microscopy imaging reconstruction method based on a deep multi-feature transfer network, which can achieve good anti-noise performance and realize high-resolution reconstruction with reduced image data. First, in this paper, the image features are deeply extracted through transfer learning ResNet50, Xception and DenseNet121 networks, and utilize the complementarity of deep multiple features and adopt cascaded feature fusion strategy for channel merging to improve the quality of image reconstruction; then the pre-upsampling is used to reconstruct the network to improve the texture details of the high-resolution reconstructed image. We validate the performance of the reported method via both simulation and experiment. The model has good robustness to noise and blurred images. Better reconstruction results are obtained under the conditions of short time and low resolution. We hope that the end-to-end mapping method of neural network can provide a neural-network perspective to solve the FPM reconstruction.

Cross-Level Channel Attention Network for Fourier Ptychographic Microscopy Reconstruction

Cross-Level Channel Attention Network for Fourier Ptychographic Microscopy Reconstruction

Using Fourier ptychography microscopy to achieve high-resolution chromosome imaging: an initial evaluation.

.SignificanceSearching analyzable metaphase chromosomes is a critical step for the diagnosis and treatment of leukemia patients, and the searching efficiency is limited by the difficulty that the conventional microscopic systems have in simultaneously achieving high resolution and a large field of view (FOV). However, this challenge can be addressed by Fourier ptychography microscopy (FPM) technology.AimThe purpose of this study is to investigate the feasibility of utilizing FPM to reconstruct high-resolution chromosome images.ApproachAn experimental FPM prototype, which was equipped with or objective lenses to achieve a theoretical equivalent NA of 0.48 and 0.63, respectively, was developed. Under these configurations, we first generated the system modulation transfer function (MTF) curves to assess the resolving power. Next, a group of analyzable metaphase chromosomes were imaged by the FPM system, which were acquired from the peripheral blood samples of the leukemia patients. The chromosome feature qualities were evaluated and compared with the results accomplished by the corresponding conventional microscopes.Results The MTF curve results indicate that the resolving power of the FPM system is equivalent and comparable to the conventional microscope, whereas the performance of the FPM system is close to the conventional microscope. When imaging the chromosomes, the feature qualities of the FPM system are comparable to the results under the conventional lens, whereas the feature qualities of the FPM system are better than the conventional lens and comparable to the conventional lens.ConclusionsThis study initially verified that it is feasible to utilize FPM to develop a high-resolution and wide-field chromosome sample scanner.

ISAR Imaging Algorithm for Maneuvering Targets Based on Fourier Ptychographic Microscopy

Aiming at the problems of low resolution and poor noise suppression for maneuvering targets with traditional inverse synthetic aperture radar (ISAR) imaging algorithms, we propose a motion compensation-driven high-resolution imaging algorithm based on Fourier ptychographic microscopy (FPM) technology. FPM is a phase retrieval technology that uses the concept of synthetic aperture to obtain the high-resolution image of the target. The process of FPM imaging is composed of the enhancement of the known target information in the spatial domain and some fixed restricted alternating iteration in the frequency domain. This article analyzes the mathematical model of maneuvering target echo and then studies the rationality of applying the FPM phase retrieval algorithm to ISAR maneuvering target imaging. The proposed algorithm mainly includes two steps. First, a series of low-resolution ISAR images are generated by sampling echo data through an aperture function. Second, the FPM phase retrieval technology is applied to splice and reconstruct the spectrum of the low-resolution images to achieve the focused ISAR image. The effectiveness of the proposed algorithm is verified by using multiple groups of simulation data and real radar data.

SAR Imaging Method for Moving Targets Based on Omega-k and Fourier Ptychographic Microscopy

The performance of traditional SAR imaging methods is limited by the system bandwidth, and the imaging results are easily affected by the target motion state. In the process of SAR imaging, the relative motion between the radar and the target will cause the nonlinear phase error of the echo signal, which makes the imaging quality worse. To solve this problem, we propose a high-resolution moving targets SAR imaging method based on Omega-k and Fourier Ptychographic Microscopy (FPM) phase retrieval technology, which realizes phase retrieval by alternating iteration between time domain and 2-D frequency domain. The proposed method is mainly divided into two steps. Firstly, reference function multiply and Stolt mapping are performed to realize echo preprocessing in the 2-D frequency domain, and then the phase error correction of the compensated echo spectrum is completed by FPM method. Several groups of simulation data and measured data are used for experimental analysis, and the results verify the effectiveness of the proposed method.

All-in-focus fine needle aspiration biopsy imaging based on Fourier ptychographic microscopy

ContextCytology is the study of whole cells in diagnostic pathology. Unlike standard histologic thinly sliced specimens, cytologic preparations consist of preparations of whole cells where cells commonly cluster and aggregate. As such, cytology preparations are generally much thicker than histologic slides, resulting in large patches of defocus when examined under the microscope. A diagnostic aggregate of cells often cannot be viewed in focus together, requiring pathologists to continually manipulate the focal plane, complicating the task of accurately assessing the entire cellular aggregate and thus in making a diagnosis. Further, it is extremely difficult to acquire useful uniformly in-focus digital images of cytology preparations for applications such as remote diagnostic evaluations and artificial intelligence models. The predominant current method to address this issue is to acquire digital images at multiple focal planes of the entire slide, which demands long scanning time, complex and expensive scanning systems, and huge storage capacity. AimsHere we report a unique imaging method that can acquire cytologic images efficiently and computationally render all-in-focus digital images that are highly compact. Methods and materialThis method applies a metric-based digital refocusing to microscopy data collected with a Fourier ptychographic microscope (FPM). The digitally refocused patches of images are then synthesized into an all-in-focus image. ResultsWe report all-in-focus FPM results of thyroid fine needle aspiration (FNA) cytology samples, demonstrating our method’s ability to overcome the height variance of 30 μm caused by cell aggregation, and rendering images at high resolution (corresponds to a standard microscope with objective NA of 0.75) and that are all-in-focus. ConclusionsThis technology is applicable to standard microscopes, and we believe can have an impact on diagnostic accuracy as well as ease and speed of diagnosing challenging specimens. While we focus on cytology slides here, we anticipate this technology’s advantages will translate well for histology applications. This technique also addresses the issue of remote rapid evaluation of cytology preparations. Finally, we believe that by resolving the focus heterogeneity issues in standard digital images, this technique is a critical advance for applying machine learning to cytology specimens.

Enhanced image reconstruction of Fourier ptychographic microscopy with double-height illumination

Fourier ptychographic microscopy (FPM) is a recently developed computational microscopy approach that can produce both wide field-of-view and high-resolution intensity and phase image of the sample. Traditionally, a periodic LED array is used as the illumination source. The frequency spectrum is a grid structure that is sparse in the low-middle frequency region and dense in the high-frequency region. However, for most biological samples, the useful information is mostly concentrated in the low-middle frequency region. So the traditional sampling method may lead to distortion in the reconstructed images because of insufficient sampling. Therefore, we propose a convenient and feasible method to continuously improve the sampling conditions without increasing the LED units and changing the pitch, termed double-height illumination FPM. The LED array is moved from one plane to another to achieve the sample illuminations at two different heights and the low-resolution intensity images are totally utilized to reconstruct the high-resolution complex amplitude image. Both simulation results and experimental results illustrate that the proposed method not only improves the reconstruction accuracy and efficiency, but also has strong robustness under noisy conditions. Furthermore, the rule of choosing the two illumination heights is discussed, and the physical insight of the proposed method is analyzed from the view of the data redundancy. In addition, a wave vector correction method is employed to reduce the artifacts of the reconstructed image at the edge of the large full field of view.

Physics-based learning with channel attention for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) is a computational imaging technology for large field-of-view, high resolution and quantitative phase imaging. In FPM, low-resolution intensity images captured with angle-varying illumination are synthesized in Fourier space with phase retrieval approaches. However, system errors such as pupil aberration and light-emitting diode (LED) intensity error seriously affect the reconstruction performance. In this article, we propose a physics-based neural network with channel attention for FPM reconstruction. With the channel attention module, which is introduced into physics-based neural networks for the first time, the spatial distribution of LED intensity can be adaptively corrected. Besides, the channel attention module is used to synthesize different Zernike modes and recover the pupil function. Detailed simulations and experiments are carried out to validate the effectiveness and robustness of the proposed method. Theresults demonstrate that our method achieves better performance in high-resolution complex field reconstruction, LED intensity correction and pupil function recovery compared with the state-of-art methods. The combination with deep neural network structures like channel attention modules significantly enhance the performance of physics-based neural networks and will promote the application of FPM in practical use.

High-speed multiview imaging approaching 4pi steradians using conic section mirrors: theoretical and practical considerations

Illuminating or imaging samples from a broad angular range is essential in a wide variety of computational 3D imaging and resolution-enhancement techniques, such as optical projection tomography, optical diffraction tomography, synthetic aperture microscopy, Fourier ptychographic microscopy, structured illumination microscopy, photogrammetry, and optical coherence refraction tomography. The wider the angular coverage, the better the resolution enhancement or 3D-resolving capabilities. However, achieving such angular ranges is a practical challenge, especially when approaching ± 90 ∘ or beyond. Often, researchers resort to expensive, proprietary high numerical aperture (NA) objectives or to rotating the sample or source-detector pair, which sacrifices temporal resolution or perturbs the sample. Here, we propose several new strategies for multiangle imaging approaching 4pi steradians using concave parabolic or ellipsoidal mirrors and fast, low rotational inertia scanners, such as galvanometers. We derive theoretically and empirically relations between a variety of system parameters (e.g., NA, wavelength, focal length, telecentricity) and achievable fields of view (FOVs) and importantly show that intrinsic tilt aberrations do not restrict FOV for many multiview imaging applications, contrary to conventional wisdom. Finally, we present strategies for avoiding spherical aberrations at obliquely illuminated flat boundaries. Our simple designs allow for high-speed multiangle imaging for microscopic, mesoscopic, and macroscopic applications.

Self-adapting search algorithm for Fourier ptychographic microscopy

Self-adapting search algorithm for Fourier ptychographic microscopy

Erratum to: High-throughput fast full-color digital pathology based on Fourier ptychographic microscopy via color transfer [Sci. China-Phys. Mech. Astron. 64(11), 114211 (2021)

Erratum to: High-throughput fast full-color digital pathology based on Fourier ptychographic microscopy via color transfer [Sci. China-Phys. Mech. Astron. 64(11), 114211 (2021)

Accurate and stable two-step LED position calibration method for Fourier ptychographic microscopy.

.Significance: Fourier ptychography microscopy (FPM) is a computational optical imaging technology that employs angularly varying illuminations and a phase retrieval algorithm to achieve a wide field of view and high-resolution imaging simultaneously. In the FPM, LED position error will reduce the quality of the reconstructed high-resolution image. To correct the LED positions, current methods consider each of the LED positions as independent and use an optimization algorithm to get each of the positions. When the positional misalignment is large or the search position falls into a local optimal value, the current methods may lack stability and accuracy.Aim: We improve the model of the LED position and propose an accurate and stable two-step correction scheme (tcFPM) to calibrate the LED position error.Approach: The improved LED positions model combines the overall offset, which represents the relative deviation of the LED array and the optical axis, with the slight deviation of each LED’s independent position. In the tcFPM, the overall offset of the LED array is corrected at first, which obtains an approximate value of the overall offset of the LED array. Then the position of each LED is precisely adjusted, which obtains the slight offset of each LED.Results: This LED position error model is more in line with the actual situation. The simulation and experimental results show that the method has high accuracy in correcting the LED position. Furthermore, the reconstruction process of tcFPM is more stable and significantly improves the quality of the reconstruction results, which is compared with some LED position error correction methods.Conclusions: An LED position error correction technology is proposed, which has a stable iterative process and improves the reconstruction accuracy of complex amplitude.

High-speed Fourier ptychographic microscopy for quantitative phase imaging.

Fourier ptychographic microscopy (FPM), as an emerging computational imaging method, has been applied to quantitative phase imaging with resolution bypassing the physical limit of the detection objective. Due to the weak illumination intensity and long image acquisition time, the achieved imaging speed in current FPM methods is still low, making them unsuitable for real-time imaging applications. We propose and demonstrate a high-speed FPM method based on using laser illumination and digital micro-mirror devices for illumination angle scanning. In this new, to the best of our knowledge, FPM method, we realized quantitative phase imaging and intensity imaging at over 42 frames per second (fps) with around 1 µm lateral resolution. The quantitative phase images have revealed membrane height fluctuations of red blood cells with nanometer-scale sensitivity, while the intensity images have resolved subcellular features in stained cancer tissue slices.

The integration of neural network and physical reconstruction model for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging method for high resolution and wide field-of-view. In this paper, we propose a new FPM method based on an untrained neural network (FuNN), which integrates a physical reconstruction model for FPM into a convolutional neural network, to solve the FPM reconstruction problem. The weights and bias of FuNN are optimized based on the interaction of the network and the physical reconstruction model so that the requirement for thousands of training datasets is eliminated. To reduce the size of the network, a set of low-resolution images are pre-processed to synthesize a complex object field before fed as the input to FuNN. Pupil recovery is also included and the final input of FuNN is a 3D image tensor that consists of synthesized field and coherent transfer function. The output of FuNN is modeled as reconstructed high-resolution complex object field and pupil function. The function of the pupil recovery is achieved mainly by the physical reconstruction model, so that the main structure of the neural network does not need to be adjusted according to the pupil recovery process as other iterative-based FPM neural network does. The performance of FuNN is evaluated by both simulation and experiment. Compared with other iterative-based FPM neural network, which just utilize network to calculate the gradient, FuNN can improve the quality of reconstructed complex object field without a large number of training datasets.

Smart computational light microscopes (SCLMs) of smart computational imaging laboratory (SCILab)

Computational microscopy, as a subfield of computational imaging, combines optical manipulation and image algorithmic reconstruction to recover multi-dimensional microscopic images or information of micro-objects. In recent years, the revolution in light-emitting diodes (LEDs), low-cost consumer image sensors, modern digital computers, and smartphones provide fertile opportunities for the rapid development of computational microscopy. Consequently, diverse forms of computational microscopy have been invented, including digital holographic microscopy (DHM), transport of intensity equation (TIE), differential phase contrast (DPC) microscopy, lens-free on-chip holography, and Fourier ptychographic microscopy (FPM). These computational microscopy techniques not only provide high-resolution, label-free, quantitative phase imaging capability but also decipher new and advanced biomedical research and industrial applications. Nevertheless, most computational microscopy techniques are still at an early stage of “proof of concept” or “proof of prototype” (based on commercially available microscope platforms). Translating those concepts to stand-alone optical instruments for practical use is an essential step for the promotion and adoption of computational microscopy by the wider bio-medicine, industry, and education community. In this paper, we present four smart computational light microscopes (SCLMs) developed by our laboratory, i.e., smart computational imaging laboratory (SCILab) of Nanjing University of Science and Technology (NJUST), China. These microscopes are empowered by advanced computational microscopy techniques, including digital holography, TIE, DPC, lensless holography, and FPM, which not only enables multi-modal contrast-enhanced observations for unstained specimens, but also can recover their three-dimensional profiles quantitatively. We introduce their basic principles, hardware configurations, reconstruction algorithms, and software design, quantify their imaging performance, and illustrate their typical applications for cell analysis, medical diagnosis, and microlens characterization.

Large-scale phase retrieval

High-throughput computational imaging requires efficient processing algorithms to retrieve multi-dimensional and multi-scale information. In computational phase imaging, phase retrieval (PR) is required to reconstruct both amplitude and phase in complex space from intensity-only measurements. The existing PR algorithms suffer from the tradeoff among low computational complexity, robustness to measurement noise and strong generalization on different modalities. In this work, we report an efficient large-scale phase retrieval technique termed as LPR. It extends the plug-and-play generalized-alternating-projection framework from real space to nonlinear complex space. The alternating projection solver and enhancing neural network are respectively derived to tackle the measurement formation and statistical prior regularization. This framework compensates the shortcomings of each operator, so as to realize high-fidelity phase retrieval with low computational complexity and strong generalization. We applied the technique for a series of computational phase imaging modalities including coherent diffraction imaging, coded diffraction pattern imaging, and Fourier ptychographic microscopy. Extensive simulations and experiments validate that the technique outperforms the existing PR algorithms with as much as 17dB enhancement on signal-to-noise ratio, and more than one order-of-magnitude increased running efficiency. Besides, we for the first time demonstrate ultra-large-scale phase retrieval at the 8K level (7680times 4320 pixels) in minute-level time.

Fast digital refocusing and depth of field extended Fourier ptychography microscopy.

Fourier ptychography microscopy (FPM) shares its roots with the synthetic aperture technique and phase retrieval method, and is a recently developed computational microscopic super-resolution technique. By turning on the light-emitting diode (LED) elements sequentially and acquiring the corresponding images that contain different spatial frequencies, FPM can achieve a wide field-of-view (FOV), high-spatial-resolution imaging and phase recovery simultaneously. Conventional FPM assumes that the sample is sufficiently thin and strictly in focus. Nevertheless, even for a relatively thin sample, the non-planar distribution characteristics and the non-ideal position/posture of the sample will cause all or part of FOV to be defocused. In this paper, we proposed a fast digital refocusing and depth-of-field (DOF) extended FPM strategy by taking the advantages of image lateral shift caused by sample defocusing and varied-angle illuminations. The lateral shift amount is proportional to the defocus distance and the tangent of the illumination angle. Instead of searching the optimal defocus distance with the optimization search strategy, which is time consuming, the defocus distance of each subregion of the sample can be precisely and quickly obtained by calculating the relative lateral shift amounts corresponding to different oblique illuminations. And then, digital refocusing strategy rooting in the angular spectrum (AS) method is integrated into FPM framework to achieve the high-resolution and phase information reconstruction for each part of the sample, which means the DOF of the FPM can be effectively extended. The feasibility of the proposed method in fast digital refocusing and DOF extending is verified in the actual experiments with the USAF chart and biological samples.

Predictive searching method for fourier ptychography with weighted apodized coherent transfer function constraint

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technology with wide field-of-view (FOV) and high resolution. However, previous studies thought that the conventional FPM algorithm approximated the LED as a coherent point light source, so FPM imaging is regarded as a coherent imaging process, and the coherent transfer function(CTF)was constrained in the Fourier domain, which seemed to be problematic. At the same time, the conventional FPM data requires the acquisition of multiple data, which takes a long time and limits its real-time application. To investigate these two problems, this paper proposes the discrete cosine (DCT) coefficient Predictive searching method for FPM with Bessel function weighted apodized coherent transfer function (B-AC) constraint. The experimental results demonstrate that without adding any algorithmic complexity and computational burden, the proposed method can reduce the number of acquired images by about 60%, while retaining the accuracy of the reconstructed results.