Computational Imaging Technique Research Articles

Multimodal 2D and 3D microscopic mapping of growth cartilage by computational imaging techniques - a short review including new research.

Being able to image the microstructure of growth cartilage is important for understanding the onset and progression of diseases such as osteochondrosis and osteoarthritis, as well as for developing new treatments and implants. Studies of cartilage using conventional optical brightfield microscopy rely heavily on histological staining, where the added chemicals provide tissue-specific colours. Other microscopy contrast mechanisms include polarization, phase- and scattering contrast, enabling non-stained or "label-free" imaging that significantly simplifies the sample preparation, thereby also reducing the risk of artefacts. Traditional high-performance microscopes tend to be both bulky and expensive. Computational imaging denotes a range of techniques where computers with dedicated algorithms are used as an integral part of the image formation process. Computational imaging offers many advantages like 3D measurements, aberration correction and quantitative phase contrast, often combined with comparably cheap and compact hardware. X-ray microscopy is also progressing rapidly, in certain ways trailing the development of optical microscopy. In this study, we first briefly review the structures of growth cartilage and relevant microscopy characterization techniques with an emphasis on FPM and advanced X-ray microscopies. We next demonstrate with our own results computational imaging through Fourier ptychographic microscopy (FPM) and compare the images with hematoxylin eosin and saffron (HES)-stained histology. Zernike phase contrast, and the nonlinear optical microscopy techniques of second harmonic generation (SHG) and two-photon excitation fluorescence (TPEF) are reported. X-ray attenuation-, phase- and diffraction-contrast computed tomography (CT) images of the very same sample are also presented for comparisons. Future perspectives on the links to artificial intelligence, dynamic studies and in vivo possibilities conclude the article.

High-speed 4-dimensional scanning transmission electron microscopy using compressive sensing techniques.

Here we show that compressive sensing allows 4-dimensional (4-D) STEM data to be obtained and accurately reconstructed with both high-speed and reduced electron fluence. The methodology needed to achieve these results compared to conventional 4-D approaches requires only that a random subset of probe locations is acquired from the typical regular scanning grid, which immediately generates both higher speed and the lower fluence experimentally. We also consider downsampling of the detector, showing that oversampling is inherent within convergent beam electron diffraction (CBED) patterns and that detector downsampling does not reduce precision but allows faster experimental data acquisition. Analysis of an experimental atomic resolution yttrium silicide dataset shows that it is possible to recover over 25 dB peak signal-to-noise ratio in the recovered phase using 0.3% of the totaldata. Lay abstract: Four-dimensional scanning transmission electron microscopy (4-D STEM) is a powerful technique for characterizing complex nanoscale structures. In this method, a convergent beam electron diffraction pattern (CBED) is acquired at each probe location during the scan of the sample. This means that a 2-dimensional signal is acquired at each 2-D probe location, equating to a 4-D dataset. Despite the recent development of fast direct electron detectors, some capable of 100kHz frame rates, the limiting factor for 4-D STEM is acquisition times in the majority of cases, where cameras will typically operate on the order of 2kHz. This means that a raster scan containing 256^2 probe locations can take on the order of 30s, approximately 100-1000 times longer than a conventional STEM imaging technique using monolithic radial detectors. As a result, 4-D STEM acquisitions can be subject to adverse effects such as drift, beam damage, and sample contamination. Recent advances in computational imaging techniques for STEM have allowed for faster acquisition speeds by way of acquiring only a random subset of probe locations from the field of view. By doing this, the acquisition time is significantly reduced, in some cases by a factor of 10-100 times. The acquired data is then processed to fill-in or inpaint the missing data, taking advantage of the inherently low-complex signals which can be linearly combined to recover the information. In this work, similar methods are demonstrated for the acquisition of 4-D STEM data, where only a random subset of CBED patterns are acquired over the raster scan. We simulate the compressive sensing acquisition method for 4-D STEM and present our findings for a variety of analysis techniques such as ptychography and differential phase contrast. Our results show that acquisition times can be significantly reduced on the order of 100-300 times, therefore improving existing frame rates, as well as further reducing the electron fluence beyond just using a faster camera.

FPM-WSI: Fourier ptychographic whole slide imaging via feature-domain backdiffraction

Fourier ptychographic microscopy (FPM) theoretically provides a solution to the trade-off between spatial resolution and field of view (FOV), and has promising prospects in digital pathology. However, block reconstruction and then stitching has become an unavoidable procedure for reconstruction of large FOV due to vignetting artifacts. This introduces digital stitching artifacts, as the existing image-domain optimization algorithms are highly sensitive to systematic errors. Such obstacles significantly impede the advancement and practical implementation of FPM, explaining why, despite a decade of development, FPM has not gained widespread recognition in the field of biomedicine. We report a feature-domain FPM (FD-FPM) based on the structure-aware forward model to realize stitching-free, full-FOV reconstruction. The loss function is uniquely formulated in the feature domain of images, which bypasses the troublesome vignetting effect and algorithmic vulnerability via feature-domain backdiffraction. Through massive simulations and experiments, we show that FD-FPM effectively eliminates vignetting artifacts for full-FOV reconstruction, and still achieves impressive reconstructions despite the presence of various systematic errors. We also found it has great potential in recovering the data with a lower spectrum overlapping rate, and in realizing digital refocusing without a prior defocus distance. With FD-FPM, we achieved full-color and high-throughput imaging (4.7 mm diameter FOV, 336 nm resolution in the blue channel) free of blocking-and-stitching procedures on a self-developed Fourier ptychographic microscopy whole slide imaging platform. The reported FD-FPM shows the value of FPM for various experimental circumstances, and offers physical insights useful for the developments of models for other computational imaging techniques. The reported platform demonstrates high-quality, high-speed imaging and low cost, and could find applications in many fields of biomedical research, as well as in clinical applications.

Computational ghost imaging with adaptive intensity illumination for scenes featuring specular surfaces

Imaging a target scene with specular surfaces is a daunting challenge for both direct imaging and indirect computational imaging techniques. The intense specular reflection component during the measurement severely degrades the quality of the reconstructed image, resulting in a substantial loss of scene information. To address this issue, we propose a computational ghost imaging (CGI) method with adaptive intensity illumination. Capitalizing on the encoded imaging feature of CGI, this method enables effective imaging of target scenes with specular surfaces through two series of measurements, eliminating the necessity for additional optical components. Based on the position and intensity information of pixels in the specular regions from the first series of measurements, our method modulates the illumination patterns to weaken the intensity of the specular region in the second series of measurements. Simulation and experimental results demonstrate that the utilization of these modulated illumination patterns for target scene measurement effectively mitigates interference from the specular surface during imaging. Consequently, the reconstructed image is capable of presenting more detailed information about the target scene other than the specular regions. Our work introduces a novel approach for imaging target scenes with specular surfaces and broadens the scope of applications for CGI in reality.

Computational Imaging Encryption with a Steganographic and Holographic Authentication Strategy

AbstractRobust data security is of paramount importance, particularly for highly classified information. To achieve this, advanced encryption and decryption methods, along with stringent access control and physical security measures, are essential to mitigate the risk of unauthorized access and data exposure. Ghost imaging (GI), a computational imaging technique that operates indirectly, offers a distinct approach compared to traditional spatially resolved imaging techniques. By utilizing computational algorithms, GI enables indirect image reconstruction, making it well‐suited for encryption applications. This proposal introduces a new computational imaging encryption scheme that combines Fourier ghost imaging (FGI) and computational holography. The scheme utilizes an untrained‐tree‐network to generate two camouflaged computer‐generated holograms (CGHs). Integration of CGH‐based steganography and holographic authentication enhances the encoding and decoding processes of FGI encryption. The effectiveness and security of the proposed scheme are rigorously validated through numerical simulations and optical experiments. The results demonstrate promising potential for effectively merging direct and indirect imaging methods in security applications.

Single-shot Fourier ptychographic microscopy with isotropic lateral resolution via polarization-multiplexed LED illumination.

Fourier ptychographic microscopy (FPM) has emerged as a new wide-field and high-resolution computational imaging technique in recent years. To ensure data redundancy for a stable convergence solution, conventional FPM requires dozens or hundreds of raw images, increasing the time cost for both data collection and computation. Here, we propose a single-shot Fourier ptychographic microscopy with isotropic lateral resolution via polarization-multiplexed LED illumination, termed SIFPM. Three LED elements covered with 0°/45°/135° polarization films, respectively, are used to provide numerical aperture-matched illumination for the sample simultaneously. Meanwhile, a polarization camera is utilized to record the light field distribution transmitted through the sample. Based on weak object transfer functions, we first obtain the amplitude and phase estimations of the sample by deconvolution, and then we use them as the initial guesses of the FPM algorithm to refine the accuracy of reconstruction. We validate the complex sample imaging performance of the proposed method on quantitative phase target, unstained and stained bio-samples. These results show that SIFPM can realize quantitative imaging for general samples with the resolution of the incoherent diffraction limit, permitting high-speed quantitative characterization for cells and tissues.

Universal and High-Fidelity Resolution Extending for Fluorescence Microscopy Using a Single-Training Physics-Informed Sparse Neural Network

As a supplement to optical super-resolution microscopy techniques, computational super-resolution methods have demonstrated remarkable results in alleviating the spatiotemporal imaging trade-off. However, they commonly suffer from low structural fidelity and universality. Therefore, we herein propose a deep-physics-informed sparsity framework designed holistically to synergize the strengths of physical imaging models (image blurring processes), prior knowledge (continuity and sparsity constraints), a back-end optimization algorithm (image deblurring), and deep learning (an unsupervised neural network). Owing to the utilization of a multipronged learning strategy, the trained network can be applied to a variety of imaging modalities and samples to enhance the physical resolution by a factor of at least 1.67 without requiring additional training or parameter tuning. Given the advantages of high accessibility and universality, the proposed deep-physics-informed sparsity method will considerably enhance existing optical and computational imaging techniques and have a wide range of applications in biomedical research.

Comprehensive Review of Computational Imaging Techniques in Information Restoration and Enhancement (Invited)

Comprehensive Review of Computational Imaging Techniques in Information Restoration and Enhancement (Invited)

LinoSPAD2: an FPGA-based, hardware-reconfigurable 512×1 single-photon camera system.

We report on LinoSPAD2, a single-photon camera system, comprising a 512×1 single-photon avalanche diode (SPAD) front-end and one or two FPGA-based back-ends. Digital signals generated by the SPADs are processed by the FPGA in real time, whereas the FPGA offers full reconfigurability at a very high level of granularity both in time and space domains. The LinoSPAD2 camera system can process 512 SPADs simultaneously through 256 channels, duplicated on each FPGA-based back-end, with a bank of 64 time-to-digital converters (TDCs) operating at 133 MSa/s, whereas each TDC has a time resolution of 20 ps (LSB). To the best of our knowledge, LinoSPAD2 is the first fully reconfigurable SPAD camera system of large format. The SPAD front-end features a pitch of 26.2 μm, a native fill factor of 25.1%, and a microlens array achieving 2.3× concentration factor. At room temperature, the median dark count rate (DCR) is 80 cps at 7 V excess bias, the peak photon detection probability (PDP) is 53% at 520 nm wavelength, and the single-photon timing resolution (SPTR) is 50 ps FWHM. The instrument response function (IRF) is around 100 ps FWHM at system level. The LinoSPAD2 camera system is suitable for numerous applications, including LiDAR imaging, heralded spectroscopy, compressive Raman sensing, and other computational imaging techniques.

Redundant information model for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) is a computational optical imaging technique that overcomes the traditional trade-off between resolution and field of view (FOV) by exploiting abundant redundant information in both spatial and frequency domains for high-quality image reconstruction. However, the redundant information in FPM remains ambiguous or abstract, which presents challenges to further enhance imaging capabilities and deepen our understanding of the FPM technique. Inspired by Shannon's information theory and extensive experimental experience in FPM, we defined the specimen complexity and reconstruction algorithm utilization rate and reported a model of redundant information for FPM to predict reconstruction results and guide the optimization of imaging parameters. The model has been validated through extensive simulations and experiments. In addition, it provides a useful tool to evaluate different algorithms, revealing a utilization rate of 24%±1% for the Gauss-Newton algorithm, LED Multiplexing, Wavelength Multiplexing, EPRY-FPM, and GS. In contrast, mPIE exhibits a lower utilization rate of 19%±1%.

Large Field‐of‐View and Multi‐Color Imaging with GaP Quadratic Metalenses

AbstractMetalenses, in order to compete with conventional bulk optics in commercial imaging systems, often require large field of view (FOV) and broadband operation simultaneously. However, the strong chromatic and coma aberrations present in common metalens designs have so far limited their widespread use. Stacking of metalenses as one of the possible solutions increases the overall complexity of the optical system and hinders the main benefits of reduced thickness and light weight. To tackle both issues, this study proposes a single‐layer imaging system utilizing a recently developed class of metalenses providing large FOV. Using it, full‐color imaging with a FOV of is demonstrated. This approach, empowered by computational imaging techniques, produces high‐quality images, both in terms of color reproduction and sharpness. Suitable for real‐time unpolarized light operation with the standard color filters present in prevalent camera systems, these results might enable a pathway for consumer electronics applications of this emerging technology.

Software-defined meta-optics

Rapid advancements in autonomous systems and the Internet of Things have necessitated the development of compact and low-power image sensors to bridge the gap between the digital and physical world. To that end, sub-wavelength diffractive optics, commonly known as meta-optics, have garnered significant interest from the optics and photonics community due to their ability to achieve multiple functionalities within a small form factor. Despite years of research, however, the performance of meta-optics has often remained inferior compared to that of traditional refractive optics. In parallel, computational imaging techniques have emerged as a promising path to miniaturize optical systems, albeit often at the expense of higher power and latency. The lack of desired performance from either meta-optical or computational solutions has motivated researchers to look into a jointly optimized meta-optical–digital solution. While the meta-optical front end can preprocess the scene to reduce the computational load on the digital back end, the computational back end can in turn relax requirements on the meta-optics. In this Perspective, we provide an overview of this up-and-coming field, termed here as “software-defined meta-optics.” We highlight recent contributions that have advanced the current state of the art and point out directions toward which future research efforts should be directed to leverage the full potential of subwavelength photonic platforms in imaging and sensing applications. Synergistic technology transfer and commercialization of meta-optic technologies will pave the way for highly efficient, compact, and low-power imaging systems of the future.

Deep coded exposure: end-to-end co-optimization of flutter shutter and deblurring processing for general motion blur removal

Coded exposure photography is a promising computational imaging technique capable of addressing motion blur much better than using a conventional camera, via tailoring invertible blur kernels. However, existing methods suffer from restrictive assumptions, complicated preprocessing, and inferior performance. To address these issues, we proposed an end-to-end framework to handle general motion blurs with a unified deep neural network, and optimize the shutter’s encoding pattern together with the deblurring processing to achieve high-quality sharp images. The framework incorporates a learnable flutter shutter sequence to capture coded exposure snapshots and a learning-based deblurring network to restore the sharp images from the blurry inputs. By co-optimizing the encoding and the deblurring modules jointly, our approach avoids exhaustively searching for encoding sequences and achieves an optimal overall deblurring performance. Compared with existing coded exposure based motion deblurring methods, the proposed framework eliminates tedious preprocessing steps such as foreground segmentation and blur kernel estimation, and extends coded exposure deblurring to more general blind and nonuniform cases. Both simulation and real-data experiments demonstrate the superior performance and flexibility of the proposed method.

LED array microscopy system correction method with comprehensive error parameters optimized by phase smoothing criterion.

LED array microscopy is a novel computational imaging technique that can achieve two-dimensional (2D) phase imaging and three-dimensional (3D) refractive index imaging with both high resolution and a large field of view. Although its experimental setup is simple, the errors caused by LED array position and light source central wavelength obviously decrease the quality of reconstructed results. To solve this problem, comprehensive error parameters optimized by the phase smoothing criterion are put forward in this paper. The central wavelength error and 3D misalignment model with six freedom degree errors of LED array are considered as the comprehensive error parameters when the spatial positional and optical features of arbitrarily placed LED array are unknown. Phase smoothing criterion is also introduced to the cost function for optimizing comprehensive error parameters to improve the convergence results. Compared with current system correction methods, the simulation and experimental results show that the proposed method in this paper has the best reconstruction accuracy, which can be well applied to an LED array microscope system with unknown positional and optical features of the LED array.

Low-cost and simple optical system based onwavefront coding and deep learning.

With the development of computational imaging, the integration of optical system design and digital algorithms has made more imaging tasks easier to perform. Wavefront coding (WFC) is a typical computational imaging technique that is used to address the constraints of optical aperture and depth of field. In this paper, we demonstrated a low-cost and simple optical system based on WFC and deep learning. We constructed an optimized encoding method for the phase plate under the framework of deep learning, which reduces the requirement for aberration correction in the full field of view. Optical coding was achieved with just a double-bonded lens and a simple cubic phase mask, and digital decoding used the deep residual UNet++ network framework. The final image obtained has good resolution, whereas the depth of field of the system expanded by a factor of 13, which is of great significance for the high-precision inspection and attaching of small parts of machine vision.

Computational imaging and occluded objects perception method based on polarization camera array.

Traditional optical imaging relies on light intensity information from light reflected or transmitted by an object, while polarization imaging utilizes polarization information of light. Camera array imaging is a potent computational imaging technique that enables computational imaging at any depth. However, conventional imaging methods mainly focus on removing occlusions in the foreground and targeting, with limited attention to imaging and analyzing polarization characteristics at specific depths. Conventional camera arrays cannot be used for polarization layered computational imaging. Thus, to study polarization layered imaging at various depths, we devised a flexible polarization camera array system and proposed a depth-parallax relationship model to achieve computational imaging of polarization arrays and polarization information reconstruction under varying conditions and depths. A series of experiments were conducted under diverse occlusion environments. We analyzed the distinctive characteristics of the imaging results obtained from the polarization array, employing a range of array distribution methods, materials, occlusion density, and depths. Our research successfully achieved computational imaging that incorporates a layered perception of objects. Finally, we evaluated the object region's polarization information using the gray level co-occurrence matrix feature method.

Polarization-sensitive intensity diffraction tomography

Optical anisotropy, which is an intrinsic property of many materials, originates from the structural arrangement of molecular structures, and to date, various polarization-sensitive imaging (PSI) methods have been developed to investigate the nature of anisotropic materials. In particular, the recently developed tomographic PSI technologies enable the investigation of anisotropic materials through volumetric mappings of the anisotropy distribution of these materials. However, these reported methods mostly operate on a single scattering model, and are thus not suitable for three-dimensional (3D) PSI imaging of multiple scattering samples. Here, we present a novel reference-free 3D polarization-sensitive computational imaging technique—polarization-sensitive intensity diffraction tomography (PS-IDT)—that enables the reconstruction of 3D anisotropy distribution of both weakly and multiple scattering specimens from multiple intensity-only measurements. A 3D anisotropic object is illuminated by circularly polarized plane waves at various illumination angles to encode the isotropic and anisotropic structural information into 2D intensity information. These information are then recorded separately through two orthogonal analyzer states, and a 3D Jones matrix is iteratively reconstructed based on the vectorial multi-slice beam propagation model and gradient descent method. We demonstrate the 3D anisotropy imaging capabilities of PS-IDT by presenting 3D anisotropy maps of various samples, including potato starch granules and tardigrade.

A Deep Learning Framework to Remove the Off-Focused Voxels from the 3D Photons Starved Depth Images

Photons Counted Integral Imaging (PCII) reconstructs 3D scenes with both focused and off-focused voxels. The off-focused portions do not contain or convey any visually valuable information and are therefore redundant. In this work, for the first time, we developed a six-ensembled Deep Neural Network (DNN) to identify and remove the off-focused voxels from both the conventional computational integral imaging and PCII techniques. As a preprocessing step, we used the standard Otsu thresholding technique to remove the obvious and unwanted background. We then used the preprocessed data to train the proposed six ensembled DNNs. The results demonstrate that the proposed methodology can efficiently discard the off-focused points and reconstruct a focused-only 3D scene with an accuracy of 98.57%.

Frequency-domain compression imaging for extending the field of view of infrared thermometers.

We propose a computational imaging technique for expanding the field of view of infrared thermometers. The contradiction between the field of view and the focal length has always been a chief problem for researchers, especially in infrared optical systems. Large-area infrared detectors are expensive and technically arduous to be manufactured, which enormously limits the performance of the infrared optical system. On the other hand, the extensive use of infrared thermometers in COVID-19 has created a considerable demand for infrared optical systems. Therefore, improving the performance of infrared optical systems and increasing the utilization of infrared detectors is vital. This work proposes a multi-channel frequency-domain compression imaging method based on point spread function (PSF) engineering. Compared with conventional compressed sensing, the submitted method images once without an intermediate image plane. Furthermore, phase encoding is used without loss of illumination of the image surface. These facts can significantly reduce the volume of the optical system and improve the energy efficiency of the compressed imaging system. Therefore, its application in COVID-19 is of great value. We design a dual-channel frequency-domain compression imaging system to verify the proposed method's feasibility. Then, the wavefront coded PSF and optical transfer function (OTF) are used, and the two-step iterative shrinkage/thresholding (TWIST) algorithm is used to restore the image to get the final result. This compression imaging method provides a new idea for the large field of view monitoring systems, especially in infrared optical systems.

Comparison of radiomics models and dual-energy material decomposition to decipher abdominal lymphoma in contrast-enhanced CT.

The radiologists' workload is increasing, and computational imaging techniques may have the potential to identify visually unequivocal lesions, so that the radiologist can focus on equivocal and critical cases. The purpose of this study was to assess radiomics versus dual-energy CT (DECT) material decomposition to objectively distinguish visually unequivocal abdominal lymphoma and benign lymph nodes. Retrospectively, 72 patients [m, 47; age, 63.5 (27-87) years] with nodal lymphoma (n = 27) or benign abdominal lymph nodes (n = 45) who had contrast-enhanced abdominal DECT between 06/2015 and 07/2019 were included. Three lymph nodes per patient were manually segmented to extract radiomics features and DECT material decomposition values. We used intra-class correlation analysis, Pearson correlation and LASSO to stratify a robust and non-redundant feature subset. Independent train and test data were applied on a pool of four machine learning models. Performance and permutation-based feature importance was assessed to increase the interpretability and allow for comparison of the models. Top performing models were compared by the DeLong test. About 38% (19/50) and 36% (8/22) of the train and test set patients had abdominal lymphoma. Clearer entity clusters were seen in t-SNE plots using a combination of DECT and radiomics features compared to DECT features only. Top model performances of AUC = 0.763 (CI = 0.435-0.923) were achieved for the DECT cohort and AUC = 1.000 (CI = 1.000-1.000) for the radiomics feature cohort to stratify visually unequivocal lymphomatous lymph nodes. The performance of the radiomics model was significantly (p = 0.011, DeLong) superior to the DECT model. Radiomics may have the potential to objectively stratify visually unequivocal nodal lymphoma versus benign lymph nodes. Radiomics seems superior to spectral DECT material decomposition in this use case. Therefore, artificial intelligence methodologies may not be restricted to centers with DECT equipment.