Multi-image Deconvolution Research Articles

Image scanning microscopy reconstruction by autocorrelation inversion

Abstract Confocal laser scanning microscopy (CLSM) stands out as one of the most widely used microscopy techniques thanks to its three-dimensional imaging capability and its sub-diffraction spatial resolution, achieved through the closure of a pinhole in front of a single-element detector. However, the pinhole also rejects useful photons, and beating the diffraction limit comes at the price of irremediably compromising the signal-to-noise ratio (SNR) of the data. Image scanning microscopy (ISM) emerged as the rational evolution of CLSM, exploiting a small array detector in place of the pinhole and the single-element detector. Each sensitive element is small enough to achieve sub-diffraction resolution through the confocal effect, but the size of the whole detector is large enough to guarantee excellent collection efficiency and SNR. However, the raw data produced by an ISM setup consists of a 4D dataset, which can be seen as a set of confocal-like images. Thus, fusing the dataset into a single super-resolved image requires a dedicated reconstruction algorithm. Conventional methods are multi-image deconvolution, which requires prior knowledge of the system point spread functions (PSFs), or adaptive pixel reassignment (APR), which is effective only on a limited range of experimental conditions. In this work, we describe and validate a novel concept for ISM image reconstruction based on autocorrelation inversion. We leverage unique properties of the autocorrelation to discard low-frequency components and maximize the resolution of the reconstructed image without any assumption on the image or any knowledge of the PSF. Our results push the quality of the ISM reconstruction beyond the level provided by APR and open new perspectives for multi-dimensional image processing.

Blind multi-Poissonian image deconvolution with sparse log-step gradient prior.

Blind image deconvolution plays a very important role in the fields such as astronomical observation and fluorescence microscopy imaging, in which the noise follows Poisson distribution. However, due to the ill-posedness, it is a very challenging task to reach a satisfactory result from a single blurred image especially when the power of the Poisson noise is at a high level. Therefore, in this paper, we try to achieve high-quality restoration results with multi-blurred images which are contaminated by Poisson noise. Firstly, we design a novel sparse log-step gradient prior which adopts a mixture of logarithm and step functions to regularize the image gradients and combine it with the Poisson distribution to formulate the blind multi-image deconvolution problem. Secondly, we incorporate the methods of variable splitting and Lagrange multiplier to convert the original problem into sub-problems, then we alternately solve them to achieve the estimation of all the blur kernels of corresponding blurred images. Besides, we also design a non-blind multi-image deconvolution algorithm which is based on the log-step gradient prior to reach the final restored image. Experimental results on both synthetic and real-world blurred images show that the proposed prior is very capable of suppressing negative artifacts caused by ill-posedness. The algorithm can achieve restored image of very high quality which is competitive with some state-of-the-art methods.

Reconstructing the image scanning microscopy dataset: an inverse problem

Confocal laser-scanning microscopy (CLSM) is one of the most popular optical architectures for fluorescence imaging. In CLSM, a focused laser beam excites the fluorescence emission from a specific specimen position. Some actuators scan the probed region across the sample and a photodetector collects a single intensity value for each scan point, building a two-dimensional image pixel-by-pixel. Recently, new fast single-photon array detectors have allowed the recording of a full bi-dimensional image of the probed region for each scan point, transforming CLSM into image scanning microscopy (ISM). This latter offers significant improvements over traditional imaging but requires an optimal processing tool to extract a super-resolved image from the four-dimensional dataset. Here we describe the image formation process in ISM from a statistical point of view, and we use the Bayesian framework to formulate a multi-image deconvolution problem. Notably, the single-photon detector suffers exclusively from the photon shot noise, enabling the development of an effective likelihood model. We derive an iterative likelihood maximization algorithm and test it on experimental and simulated data. Furthermore, we demonstrate that the ISM dataset is redundant, enabling the possibility of obtaining reconstruction sampled at twice the scanning step. Our results prove that in ISM, under appropriate conditions, the Nyquist–Shannon sampling criterium is effectively relaxed. This finding can be exploited to speed up the acquisition process by a factor of four, further improving the versatility of ISM systems.

Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo

RESOLFT fluorescence nanoscopy can nowadays image details far beyond the diffraction limit. However, signal to noise ratio (SNR) and temporal resolution are still a concern, especially deep inside living cells and organisms. In this work, we developed a non-deterministic scanning approach based on a real-time feedback system which speeds up the acquisition up to 6-fold and decreases the light dose by 70–90% for in vivo imaging. Also, we extended the information content of the images by acquiring the complete temporal evolution of the fluorescence generated by reversible switchable fluorescent proteins. This generates a series of images with different spatial resolution and SNR, from conventional to RESOLFT images, which combined through a multi-image deconvolution algorithm further enhances the effective resolution. We reported nanoscale imaging of organelles up to 35 Hz and actin dynamics during an invasion process at a depth of 20–30 µm inside a living Caenorhabditis elegans worm.

Boost Your Microscope by Exploring New Dimensions

Due to diffraction of light, the conventional fluorescent microscope is a system with a band-limited information spatial channel. Every super-resolved fluorescent microscope circumvents this limitation by avoiding the simultaneous signaling of adjacent (< diffraction limit distance) fluorophores recording them sequentially in time. Notably, the confocal microscope also agrees to this perspective. Thus, the current super-resolved systems encode extra spatial information in the temporal channel.Here we discuss if other channels (dimensions) can be used to further improve the resolution of a fluorescence microscope. In particular, we describe time-gated STED microscopy and image-scanning microscopy (ISM) as examples of exploring extra dimensions, respectively the photon arrival time and the spatial image plane dimensions.Confocal scanning microscopy (CSM) can theoretically surpass the diffraction limit by a factor of √2. Practically, this improvement is sacrificed to obtain a good signal-to-noise ratio (SNR). Image scanning microscopy (ISM) solved this limitation by substituting the single point detector with a 2D array of detector. We showed that ISM can be straightforwardly implemented by using a quadrant detector. This implementation offers resolution close to the CSM theoretical value, improves the SNR by factor of 1.5 with respect to the CSM counterpart, and may be implemented without losing the optical sectioning capability and the system versatility.Time-gated detection increases the spatial resolution of a gated continuous-wave-(CW-) STED microscope, with the drawback of reducing the SNR. Thus, in sub-optimal conditions, such as a low-photons budget regime, the SNR reduction can cancel-out the expected gain in resolution. We developed a method which does not discard photons, but instead sorts all the photons in different time-gates according to their arrival time and recombines them through a multi-image deconvolution. Our results show that the SNR of the restored image improves, thereby improving the effective resolution.

Two-Photon Excitation STED Microscopy with Time-Gated Detection.

We report on a novel two-photon excitation stimulated emission depletion (2PE-STED) microscope based on time-gated detection. The time-gated detection allows for the effective silencing of the fluorophores using moderate stimulated emission beam intensity. This opens the possibility of implementing an efficient 2PE-STED microscope with a stimulated emission beam running in a continuous-wave. The continuous-wave stimulated emission beam tempers the laser architecture’s complexity and cost, but the time-gated detection degrades the signal-to-noise ratio (SNR) and signal-to-background ratio (SBR) of the image. We recover the SNR and the SBR through a multi-image deconvolution algorithm. Indeed, the algorithm simultaneously reassigns early-photons (normally discarded by the time-gated detection) to their original positions and removes the background induced by the stimulated emission beam. We exemplify the benefits of this implementation by imaging sub-cellular structures. Finally, we discuss of the extension of this algorithm to future all-pulsed 2PE-STED implementationd based on time-gated detection and a nanosecond laser source.

Robust All-in-focus Super-Resolution for Focal Stack Photography

We present an unconventional image super-resolution algorithm targeting focal stack images. Contrary to previous works, which align multiple images with sub-pixel accuracy for image super-resolution, we analyze the correlation among the differently focused narrow depth-of-field images in a focal stack to infer high-resolution details for image super-resolution. In order to accurately model the defocus kernels at different depths, we use a cubic interpolation to parameterize the projection of defocus kernels, and apply the radon transform to accurately reconstruct the defocus kernels at arbitrary depth. In the image super-resolution, we utilize the multi-image deconvolution method with a l1 -norm regularization to suppress noise and ringing artifacts. We have also extended the depth-of-field of our inputs to produce an all-in-focus super-resolution image. The effectiveness of our algorithm is demonstrated with the quantitative analysis using synthetic examples and the qualitative analysis using real-world examples.

Multi-images deconvolution improves signal-to-noise ratio on gated stimulated emission depletion microscopy

Time-gated detection, namely, only collecting the fluorescence photons after a time-delay from the excitation events, reduces complexity, cost, and illumination intensity of a stimulated emission depletion (STED) microscope. In the gated continuous-wave- (CW-) STED implementation, the spatial resolution improves with increased time-delay, but the signal-to-noise ratio (SNR) reduces. Thus, in sub-optimal conditions, such as a low photon-budget regime, the SNR reduction can cancel-out the expected gain in resolution. Here, we propose a method which does not discard photons, but instead collects all the photons in different time-gates and recombines them through a multi-image deconvolution. Our results, obtained on simulated and experimental data, show that the SNR of the restored image improves relative to the gated image, thereby improving the effective resolution.

Three-dimensional PET reconstruction with time-of-flight measurement

The interest in fully three-dimensional image reconstruction, especially in positron emission tomography (PET) has significantly increased for the last few years. Taking into account the cross-plane gamma rays in a three-dimensional reconstruction algorithm improves the sensitivity. At LETI, the authors' speciality in PET is the time-of-flight (TOF) measurement. Thus, they present two reconstruction techniques for 3D TOF PET. The first is a backprojection-convolution algorithm. Due to the redundancy in the 3D data set, there exists an infinite number of filters. As Defrise et al. (1989) did for classical tomography, the authors establish a general condition that characterizes the filters and propose an algorithm with a factorizable filter. However, this first technique requires an acquisition system with revolution symmetry. Thus, the authors present a second one which is adapted to a detection geometry with a small number of angular positions. It consists of a multi-image deconvolution algorithm with Wiener filter.

Multi-image regeneration by white light processing

A technique of multi-image regeneration by white light processing is described. This white light method is very simple, versatile and economical, and it offers the advantages of multi-image regeneration and the elimination of coherent artifact noise that plagues coherent optical systems. Although this method is effective only in 1-D processing, nonetheless it also works very well for the 2-D multi-image deconvolution operation. A simple experimental demonstration of multi-image regeneration by this technique is also presented.