High Space-bandwidth Product Research Articles

High sensitivity cameras can lower spatial resolution in high-resolution optical microscopy

High-resolution optical fluorescence microscopies and, in particular, super-resolution fluorescence microscopy, are rapidly adopting highly sensitive cameras as their preferred photodetectors. Camera-based parallel detection facilitates high-speed live cell imaging with the highest spatial resolution. Here, we show that the drive to use ever more sensitive, photon-counting image sensors in cameras can, however, have detrimental effects on the spatial resolution of the resulting images. This is particularly noticeable in applications that demand a high space-bandwidth product, where the image magnification is close to the Nyquist sampling limit of the sensor. Most scientists will often select image sensors based on parameters such as pixel size, quantum efficiency, signal-to-noise performance, dynamic range, and frame rate of the sensor. A parameter that is, however, typically overlooked is the sensor’s modulation transfer function (MTF). We have determined the wavelength-specific MTF of front- and back-illuminated image sensors and evaluated how it affects the spatial resolution that can be achieved in high-resolution fluorescence microscopy modalities. We find significant differences in image sensor performance that cause the resulting spatial resolution to vary by up to 28%. This result shows that the choice of image sensor has a significant impact on the imaging performance of all camera-based optical microscopy modalities.

EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM’s capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.

K-space holographic multiplexing for synthetic aperture diffraction tomography

Optical diffraction tomography can be performed with low phototoxicity and photobleaching to analyze 3D cells and tissues. It is desired to develop high throughput and powerful data processing capabilities. We propose high bandwidth holographic microscopy (HBHM). Based on the analyticity of complex amplitudes, the unified holographic multiplexing transfer function is established. A high bandwidth scattering field is achieved via the k-space optical origami of two 2D wavefronts from one interferogram. Scanning illumination modulates the high-horizontal and axial k-space to endow synthetic-aperture from 2D high space-bandwidth product (SBP) scattering fields. The bright-field counterpart SBP of a single scattering field from HBHM is 14.6 megapixels, while the number of pixels is only 13.7 megapixels. It achieves an eight-fold SBP enhancement under the same number of pixels and diffraction limit. The HBHM paves the way toward the performance of high throughput, large-scale, and non-invasive histopathology, cell biology, and industrial inspection.

High Space‐Bandwidth‐Product (SBP) Hologram Carriers Toward Photorealistic 3D Holography

Abstract3D holography capable of reproducing all necessary visual cues is considered the most promising route to present photorealistic 3D images. Three elements involving computer‐generated hologram (CGH) algorithms, hologram carriers, and optical systems are prerequisites to create high‐quality holographic displays for photorealistic 3D holography. Especially, the hologram carrier directly determines the holographic display capability and the design of high space‐bandwidth‐product (SBP) optical systems. Currently, two categories of hologram carriers, i.e., spatial light modulators (SLM) and metasurfaces, are regarded as promising candidates for photorealistic 3D holography. However, most of their SBP capability still cannot match the amount of information generated by the CGH. To address this issue, tremendous efforts are made to improve the capability of hologram carriers. Here, the main hologram carriers (from SLM to metasurfaces) that are widely utilized in holography systems to achieve high SBP capability (high resolution, wide viewing angles, and large sizes) are reviewed. The purpose of this review is to identify the key challenges and future directions of SLM‐based and metasurface‐based holography for photorealistic 3D holographic images.

Enhancing the spatial resolution of light-field displays without losing angular resolution by a computational subpixel realignment.

Low spatial resolution is an urgent problem in integral imaging light-field displays (LFDs). This study proposes a computational method to enhance the spatial resolution without losing angular resolution. How rays reconstruct voxels through lenslets is changed so that every ray through a lenslet merely provides a subpixel. The three subpixels of a pixel no longer form one voxel but three independent voxels. We further demonstrate imperfect integration of subpixels, called the sampling error, can be eliminated on specific image depths, including the central depth plane. By realigning subpixels in the above manner under no sampling error, the sampling rate of voxels is three times the conventional pixel-based LFDs. Moreover, the ray number of every voxel is preserved for an unaffected angular resolution. With unavoidable component alignment errors, resolution gains of 2.52 and 2.0 are verified in simulation and experiment by computationally updating the elemental image array. The proposed computational method further reveals that LFDs intrinsically have a higher space-bandwidth product than presumed.

Imaging process matched neural network for complex wavefront retrieval with a higher space-bandwidth product.

Recently, deep learning (DL) has shown great potential in complex wavefront retrieval (CWR). However, the application of DL in CWR does not match well with the physical diffraction process. The state-of-the-art DL-based CWR methods crop full-size diffraction patterns down to a smaller size to save computational resources. However, cropping reduces the space-bandwidth product (SBP). In order to solve the trade-off between computational resources and SBP, we propose an imaging process matched neural network (IPMnet). IPMnet accepts full-size diffraction patterns with a larger SBP as inputs and retrieves a higher resolution and a larger field of view of the complex wavefront. We verify the effectiveness of the proposed IPMnet through simulations and experiments.

Viewing angle enhancement for integral imaging display using two overlapped panels.

Integral imaging three-dimensional (3D) display relies on display panel to provide visual information, but the intrinsic trade-off between the wide viewing angle and high resolution refrains its application in high-throughput 3D display. We propose a method to enhance the viewing angle without sacrificing the resolution by using two overlapped panels. The additionally introduced display panel is composed of two parts: the information area and the transparent area. The transparent area loaded with blank information enables light passing through without any modulation, while the opaque information area is loaded with element image array (EIA) for 3D display. The configuration of the introduced panel can block crosstalk from the original 3D display and establish a new and viewable perspective. Experimental results show that the horizontal viewing angle can be effectively extended from 8° to 16°, demonstrating the feasibility and effectiveness of our proposed method. This method provides the 3D display system with a higher space-bandwidth product, making it a potential technique to be applied for high information-capacity display, including integral imaging and holography.

Lensfree time-gated photoluminescent imaging

Fluorescence and, more generally, photoluminescence enable high contrast imaging of targeted regions of interest through the use of photoluminescent probes with high specificity for different targets. Fluorescence can be used for rare cell imaging; however, this often requires a high space-bandwidth product: simultaneous high resolution and large field of view. With bulky traditional microscopes, high space-bandwidth product images require time-consuming mechanical scanning and stitching. Lensfree imaging can compactly and cost-effectively achieve a high space-bandwidth product in a single image through computational reconstruction of images from diffraction patterns recorded over the full field of view of standard image sensors. Many methods of lensfree photoluminescent imaging exist, where the excitation light is filtered before the image sensor, often by placing spectral filters between the sample and sensor. However, the sample-to-sensor distance is one of the limiting factors on resolution in lensfree systems and so more competitive performance can be obtained if this distance is reduced. Here, we show a time-gated lensfree photoluminescent imaging system that can achieve a resolution of 8.77 µm. We use europium chelate fluorophores because of their long lifetime (642 µs) and trigger camera exposure ∼50 µs after excitation. Because the excitation light is filtered temporally, there is no need for physical filters, enabling reduced sample-to-sensor distances and higher resolutions.

Diffuser-based fiber endoscopy for single-shot 3D fluorescence imaging

Minimally invasive endoscopy using coherent fiber bundles shows great potential for numerous applications in biomedical imaging. With a diffuser on the distal side of the fiber bundle and computational image recovery, single-shot 3D imaging is possible by encoding the image volume into 2D speckle patterns. In comparison to equivalent lens systems, a higher space-bandwidth product can be achieved. However, decoding the image with iterative algorithms is time-consuming. Thus, we propose utilizing a neural network for fast 2D and 3D image reconstruction at video rate. In this work, single-shot 3D fluorescence imaging with an ultra-thin endoscope is demonstrated, enabling applications like calcium imaging for in vivo brain diagnostics at cellular resolution.

Robust Fourier ptychographic microscopy via a physics-based defocusing strategy for calibrating angle-varied LED illumination.

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique for wide-field, high-resolution microscopy with a high space-bandwidth product. It integrates the concepts of synthetic aperture and phase retrieval to surpass the resolution limit imposed by the employed objective lens. In the FPM framework, the position of each sub-spectrum needs to be accurately known to ensure the success of the phase retrieval process. Different from the conventional methods with mechanical adjustment or data-driven optimization strategies, here we report a physics-based defocusing strategy for correcting large-scale positional deviation of the LED illumination in FPM. Based on a subpixel image registration process with a defocused object, we can directly infer the illumination parameters including the lateral offsets of the light source, the in-plane rotation angle of the LED array, and the distance between the sample and the LED board. The feasibility and effectiveness of our method are validated with both simulations and experiments. We show that the reported strategy can obtain high-quality reconstructions of both the complex object and pupil function even the LED array is randomly placed under the sample with both unknown lateral offsets and rotations. As such, it enables the development of robust FPM systems by reducing the requirements on fine mechanical adjustment and data-driven correction in the construction process.

Single-Shot High-Throughput Phase Imaging with Multibeam Array Interferometric Microscopy

High-throughput microscopic imaging is highly desirable in biomedical applications. Advances in computational microscopy have achieved high space-bandwidth products and even permitted gigapixel imaging in a stepwise fashion, yet temporal resolution remains challenging for investigating live-sample dynamics. Here, we report multibeam array interferometric microscopy (MAIM) for a single-shot high space-bandwidth product. The MAIM method overcomes the limitations of conventional digital holographic microscopy, providing complex field reconstruction with a maximum 5-fold field of view (FOV) increase in a single camera acquisition, while maintaining sub-nanometer optical path-length stability. This is achieved by integrating common-path holographic microscopy, multibeam interference technology, and holographic multiplexing technology. The temporal resolving power of MAIM is significantly higher than that of computational illumination microscopy. MAIM has major advantages over previous holographic multiplexing techniques in that it integrates more wavefronts and offers high temporal stability. The fundamentals of MAIM are analyzed theoretically. As a demonstration, we build MAIM prototypes to increase the FOV by factors of 5, 4, and 3, respectively. We present proof-of-concept MAIM imaging results of both natural and artificial samples and show biomedical applications such as monitoring sub-cellular dynamical phenomena in flowing live erythrocytes in vitro and label-free microrefractometry imaging of unstained cancer tissue slices. MAIM gives rise to (ultra)fast or long-term (time-lapse) imaging of nanoscale dynamics of unstained live samples in vitro with a high throughput.

Image reconstruction approach for a high space-bandwidth product structured illumination microscopy system

Conventional structured illumination microscopy (SIM) utilizes a sinusoidal excitation pattern of frequency within the detection passband and provides a maximum of twofold resolution enhancement over the diffraction limit. A transmission approach proposed in an earlier publication [J. Phys. D 53, 044006 (2019)JPAPBE0022-372710.1088/1361-6463/ab4e68] to further improve the lateral resolution requires sequential higher frequency illumination patterns. However, the existing reconstruction algorithms fail to deliver appropriate reconstruction when the excitation frequency lies far from the passband boundary. Here, we present a correlation-based SIM reconstruction approach for sequential high-frequency illumination patterns even if the pattern frequency lies far from the passband limit. The scheme can be suitably implemented in a variety of custom-built systems where illumination frequency lies beyond the passband support (e.g., non-linear SIM and plasmonic SIM).

High-throughput spatial sensitive quantitative phase microscopy using low spatial and high temporal coherent illumination

High space-bandwidth product with high spatial phase sensitivity is indispensable for a single-shot quantitative phase microscopy (QPM) system. It opens avenue for widespread applications of QPM in the field of biomedical imaging. Temporally low coherence light sources are implemented to achieve high spatial phase sensitivity in QPM at the cost of either reduced temporal resolution or smaller field of view (FOV). In addition, such light sources have low photon degeneracy. On the contrary, high temporal coherence light sources like lasers are capable of exploiting the full FOV of the QPM systems at the expense of less spatial phase sensitivity. In the present work, we demonstrated that use of narrowband partially spatially coherent light source also called pseudo-thermal light source (PTLS) in QPM overcomes the limitations of conventional light sources. The performance of PTLS is compared with conventional light sources in terms of space bandwidth product, phase sensitivity and optical imaging quality. The capabilities of PTLS are demonstrated on both amplitude (USAF resolution chart) and phase (thin optical waveguide, height ~ 8 nm) objects. The spatial phase sensitivity of QPM using PTLS is measured to be equivalent to that for white light source and supports the FOV (18 times more) equivalent to that of laser light source. The high-speed capabilities of PTLS based QPM is demonstrated by imaging live sperm cells that is limited by the camera speed and large FOV is demonstrated by imaging histopathology human placenta tissue samples. Minimal invasive, high-throughput, spatially sensitive and single-shot QPM based on PTLS will enable wider penetration of QPM in life sciences and clinical applications.

Review of bio-optical imaging systems with a high space-bandwidth product.

Optical imaging has served as a primary method to collect information about biosystems across scales-from functionalities of tissues to morphological structures of cells and even at biomolecular levels. However, to adequately characterize a complex biosystem, an imaging system with a number of resolvable points, referred to as a space-bandwidth product (SBP), in excess of one billion is typically needed. Since a gigapixel-scale far exceeds the capacity of current optical imagers, compromises must be made to obtain either a low spatial resolution or a narrow field-of-view (FOV). The problem originates from constituent refractive optics-the larger the aperture, the more challenging the correction of lens aberrations. Therefore, it is impractical for a conventional optical imaging system to achieve an SBP over hundreds of millions. To address this unmet need, a variety of high-SBP imagers have emerged over the past decade, enabling an unprecedented resolution and FOV beyond the limit of conventional optics. We provide a comprehensive survey of high-SBP imaging techniques, exploring their underlying principles and applications in bioimaging.

Phase-space synthesized digital holography for high space-bandwidth product imaging

High resolution and wide field of view (FOV) are always the goals of imaging, which are related to the space-bandwidth product (SBP) of the system. Currently, most methods focus on either resolution enhancement or FOV extension. Few works pay attention on both. There lacks a generalized framework for the joint space-frequency SBP expansion. We propose such a holographic imaging method, termed phase-space synthesized digital holography (PSH), which can improve and adjust resolution and FOV simultaneously, based on a phase-space analysis. Through a controllable SBP expansion in the phase space by multiangle divergent spherical wave illumination, a synthesized hologram is obtained to reconstruct a resolution-enhanced and FOV-extended image. As a general methodology of SBP expansion, the proposed method could open new insights for the imaging community.

Compressed off-axis digital holography

The Fourier transform method (FTM), which can reconstruct a wavefront from a single hologram, is widely used in digital holography (DH) as a reconstruction technique. However, the space-bandwidth product (SBP) is limited because of the frequency filtering required. We propose a wavefront reconstruction algorithm based on compressed sensing that can obtain a high SBP for use instead of the FTM. The proposed method can reconstruct a high-SBP wavefront by solving the hologram recording model with total variation regularization. We demonstrate the proposed method’s effectiveness via simulations and experiments. The method can be applied to conventional off-axis DH without alteration and thus can be applied in various fields.

High-Throughput Deep Learning Microscopy Using Multi-Angle Super-Resolution

Biomedical applications such as pathology and hematology expect microscopes with high space-bandwidth product (SBP) which is difficult to achieve with conventional microscope setup. By applying a deep neural network, we demonstrate a high space-bandwidth product microscopic technique termed multi-angle super-resolution microscopy (MASRM) to achieve high-resolution imaging with the low-magnification objective. We design a multiple-branch deep residual network which extracts high-frequency information and color information in obliquely-illuminated low-resolution input images and generates high-resolution output. To train our network, we build a well-registered dataset in which both low-resolution input and high-resolution target are real captured images. We carry out detailed experiments to demonstrate the effectiveness of MASRM and compare it with a computational imaging technique termed Fourier ptychographic microscopy (FPM). This data-driven technique unleashes the potential of traditional microscopes with low cost and has broad prospects in biomedical applications.

Illumination pattern design with deep learning for single-shot Fourier ptychographic microscopy.

Fourier ptychographic microscopy allows for the collection of images with a high space-bandwidth product at the cost of temporal resolution. In Fourier ptychographic microscopy, the light source of a conventional widefield microscope is replaced with a light-emitting diode (LED) matrix, and multiple images are collected with different LED illumination patterns. From these images, a higher-resolution image can be computationally reconstructed without sacrificing field-of-view. We use deep learning to achieve single-shot imaging without sacrificing the space-bandwidth product, reducing the acquisition time in Fourier ptychographic microscopy by a factor of 69. In our deep learning approach, a training dataset of high-resolution images is used to jointly optimize a single LED illumination pattern with the parameters of a reconstruction algorithm. Our work paves the way for high-throughput imaging in biological studies.

Kramers–Kronig holographic imaging for high-space-bandwidth product

Modern optical imaging possesses a huge information capacity whose corresponding space-bandwidth product (SBP) reaches tens of megapixels. However, despite the advances in optical and electronic devices, the SBP of an optical microscope is greatly limited, resulting in a reduced field of view or resolution of an image. In this paper, we exploit the Kramers–Kronig relations in digital holography to achieve high SBP imaging, demonstrating a complex amplitude image that can surpass the SBP of a bright-field image. The capability of the proposed method is demonstrated by imaging static samples and biological tissues. We successfully measure a 4.2-megapixel complex amplitude image whose bright-field counterpart exhibits 16.7 megapixels.

Efficient Colorful Fourier Ptychographic Microscopy Reconstruction With Wavelet Fusion

Fourier ptychographic microscopy (FPM) is a recently developed microscope that offers wide field, high resolution, and quantitative phase imaging. Combining the concepts of ptychography, synthetic aperture, and phase retrieval, FPM overcomes the space-bandwidth-product (SBP) limit of the optical system. In FPM, a LED array is used as the illumination module and specimen images illuminated with angular varying illuminations are captured. These images are synthesized to recover the high SBP complex field of the specimen. There have been many improvements of FPM since its appearance and FPM is of great potential in the field of hematology and pathology. However, low efficiency in capturing data, especially in the situation of capturing all three color channels, limits the application of FPM. Although spectral multiplex strategy for FPM is developed, the reconstruction quality and speed is decreased for exchanging the capture efficiency. On the other hand, the reconstruction quality of FPM is significantly degraded by the imaging noise in the dark-field images because the high-frequency information in dark-field images is of low energy and contaminated by noise. In this paper, we propose an efficient colorful FPM reconstruction method using multi-resolution wavelet color fusion. We also propose an adaptive denoising method by analyzing the noise information of the dark-frame. Both simulation and experiment results are carried out to validate our methods. Results demonstrate that the imaging noise is suppressed and the colorful reconstruction is of high efficiency and quality.