A Modular and Robust Physics-Based Approach for Lensless Image Reconstruction

Lensless imaging is an approach to microscopy in which a high-resolution image of an object is reconstructed from one or more measured diffraction patterns, providing a solution in situations where the use of imaging optics is not possible. However, current lensless imaging methods are typically limited by the need for a light source with a narrow, stable and accurately known spectrum. We have developed a general approach to lensless imaging without spectral bandwidth limitations or sample requirements. We use two time-delayed coherent light pulses and show that scanning the pulse-to-pulse time delay allows the reconstruction of diffraction-limited images for all the spectral components in the pulse. In addition, we introduce an iterative phase retrieval algorithm that uses these spectrally resolved Fresnel diffraction patterns to obtain high-resolution images of complex extended objects. We demonstrate this two-pulse imaging method with octave-spanning visible light sources, in both transmission and reflection geometries, and with broadband extreme-ultraviolet radiation from a high-harmonic generation source. Our approach enables effective use of low-flux ultra-broadband sources, such as table-top high-harmonic generation systems, for high-resolution imaging. Researchers in The Netherlands have overcome the restriction of monochromatic illumination when performing lensless imaging. Stefan Witte and co-workers from LaserLAB Amsterdam have developed a lensless scheme that employs two coherent time-delayed pulses and is compatible with broadband sources. Lensless imaging — whereby diffraction patterns are interpreted to reconstruct an image of a sample — is popular in the X-ray and extreme-ultraviolet regimes, where high-quality lenses for performing conventional imaging are not available. Because this approach has traditionally been limited to narrowband coherent radiation, scientists have been eager to make it compatible with broadband sources such as tabletop high-harmonic generation. This new broadband technique involves scanning the pulse-to-pulse time delay and then applying a phase-retrieval algorithm to produce high-resolution images of complex objects.

Lensless imaging with a mask is an attractive topic as it enables a compact configuration to acquire wavefront information of a sample with computational approaches. Most existing methods choose a customized phase mask for wavefront modulation and then decode the sample's wave field from modulated diffraction patterns. Different from phase masks, lensless imaging with a binary amplitude mask facilitates a cheaper fabrication cost, but high-quality mask calibration and image reconstruction have not been well resolved. Here we propose a self-calibrated phase retrieval (SCPR) method to realize a joint recovery of a binary mask and sample's wave field for a lensless masked imaging system. Compared with conventional methods, our method shows a high-performance and flexible image recovery without the help of an extra calibration device. Experimental results of different samples demonstrate the superiority of our method.

Lensless imaging represents a significant advancement in imaging technology, offering unique benefits over traditional optical systems due to its compact form factor, ideal for applications within the Internet of Things (IoT) ecosystem. Despite its potential, the intensive computational requirements of current lensless imaging reconstruction algorithms pose a challenge, often exceeding the resource constraints typical of IoT devices. To meet this challenge, a novel approach is introduced, merging multi-level image restoration with the pix2pix generative adversarial network architecture within the lensless imaging sphere. Building on the foundation provided by U-Net, a Multi-level Attention-based Lensless Image Restoration Network (MARN) is introduced to further augment the generator’s capabilities. In this methodology, images reconstructed through Tikhonov regularization are perceived as degraded images, forming the foundation for further refinement via the Pix2pix network. This process is enhanced by incorporating an attention-focused mechanism in the encoder--decoder structure and by implementing stage-wise supervised training within the deep convolutional network, contributing markedly to the improvement of the final image quality. Through detailed comparative evaluations, the superiority of the introduced method is affirmed, outperforming existing techniques and underscoring its suitability for addressing the computational challenges in lensless imaging within IoT environments. This method can produce excellent lensless image reconstructions when sufficient computational resources are available, and it consistently delivers optimal results across varying computational resource constraints. This algorithm enhances the applicability of lensless imaging in applications such as the Internet of Things, providing higher-quality image acquisition and processing capabilities for these domains.

Computational protein design promises the ability to build tailor-made proteins de novo. While a range of de novo proteins have been constructed so far, the majority of these designs have idealized topologies that lack larger cavities which are necessary for the incorporation of small molecule binding sites or enzymatic functions. One attractive target for enzyme design is the TIM-barrel fold, due to its ubiquity in nature and capability to host versatile functions. With the successful de novo design of a 4-fold symmetric TIM barrel, sTIM11, an idealized, minimalistic scaffold was created. In this work, we attempted to extend this de novo TIM barrel by incorporating a helix-loop-helix motif into its βα-loops by applying a physics-based modular design approach using Rosetta. Further diversification was performed by exploiting the symmetry of the scaffold to integrate two helix-loop-helix motifs into the scaffold. Analysis with AlphaFold2 and biochemical characterization demonstrate the formation of additional α-helical secondary structure elements supporting the successful extension as intended.

PurposeThe quantitative detection of failure modes is important for making deep neural networks reliable and usable at scale. We consider three examples for common failure modes in image reconstruction and demonstrate the potential of uncertainty quantification as a fine-grained alarm system.MethodsWe propose a deterministic, modular and lightweight approach called Interval Neural Network (INN) that produces fast and easy to interpret uncertainty scores for deep neural networks. Importantly, INNs can be constructed post hoc for already trained prediction networks. We compare it against state-of-the-art baseline methods (MCDrop, ProbOut).ResultsWe demonstrate on controlled, synthetic inverse problems the capacity of INNs to capture uncertainty due to noise as well as directional error information. On a real-world inverse problem with human CT scans, we can show that INNs produce uncertainty scores which improve the detection of all considered failure modes compared to the baseline methods.ConclusionInterval Neural Networks offer a promising tool to expose weaknesses of deep image reconstruction models and ultimately make them more reliable. The fact that they can be applied post hoc to equip already trained deep neural network models with uncertainty scores makes them particularly interesting for deployment.

Lensless image reconstruction is an ill-posed inverse problem in computational imaging, having several applications in machine vision. Existing approaches rely on large datasets for learning to perform deconvolution and are often specific to the point spread function of a particular lensless imager. Generating pairs of lensless images and their corresponding ground truths requires a specialized laboratory setup, thus making the dataset collection procedure challenging. We propose a reconstruction method using untrained neural networks that relies on the underlying physics of lensless image generation. We use an encoder-decoder network for reconstructing the lensless image for a known PSF. The same network can predict the PSF when supplied with a single example of input and ground-truth pair, thus acting as a one-time calibration step for any lensless imager. We used a physics-guided consistency loss function to optimize our model to perform reconstruction and PSF estimation. Our model generates accurate non-blind reconstructions with a PSNR of 24.55 dB.KeywordsLensless image reconstructionUntrained neural networksComputational imaging

Medical images provide experienced physicians with meaningful visual stimuli but their features are frequently hard to decipher. The development of a computational model to mimic physicians' expertise is a demanding task, especially if a significant and sophisticated preprocessing of images is required. Learning from well-expertised images may be a more convenient approach, inasmuch a large and representative bunch of samples is available. A four-stage approach has been designed, which combines image sub-sampling with unsupervised image coding, supervised classification and image reconstruction in order to directly extract medical expertise from raw images. The system has been applied (1) to the detection of some features related to the diagnosis of black tumors of skin (a classification issue) and (2) to the detection of virus-infected and healthy areas in retina angiography in order to locate precisely the border between them and characterize the evolution of infection. For reasonably balanced training sets, we are able to obtained about 90% correct classification of features (black tumors). Boundaries generated by our system mimic reproducibility of hand-outlines drawn by experts (segmentation of virus-infected area).

A mask-based lensless camera optically encodes the scene with a thin mask and reconstructs the image afterward. The improvement of image reconstruction is one of the most important subjects in lensless imaging. Conventional model-based reconstruction approaches, which leverage knowledge of the physical system, are susceptible to imperfect system modeling. Reconstruction with a pure data-driven deep neural network (DNN) avoids this limitation, thereby having potential to provide a better reconstruction quality. However, existing pure DNN reconstruction approaches for lensless imaging do not provide a better result than model-based approaches. We reveal that the multiplexing property in lensless optics makes global features essential in understanding the optically encoded pattern. Additionally, all existing DNN reconstruction approaches apply fully convolutional networks (FCNs) which are not efficient in global feature reasoning. With this analysis, for the first time to the best of our knowledge, a fully connected neural network with a transformer for image reconstruction is proposed. The proposed architecture is better in global feature reasoning, and hence enhances the reconstruction. The superiority of the proposed architecture is verified by comparing with the model-based and FCN-based approaches in an optical experiment.

Brain activity represents our perceptual experience. But the potential for reading out perceptual contents from human brain activity has not been fully explored. In this study, we demonstrate constraint-free reconstruction of visual images perceived by a subject, from the brain activity pattern. We reconstructed visual images by combining local image bases with multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Binary-contrast, 10 x 10-patch images (2100 possible states), were accurately reconstructed without any image prior by measuring brain activity only for several hundred random images. The results suggest that our approach provides an effective means to read out complex perceptual states from brain activity while discovering information representation in multi-voxel patterns.

The quality of the reconstructed image from lensless imaging is affected by the exposure time. This paper explores the influence of exposure time on imaging technology. Based on the lensless imaging system established by us, we compare and analyze the objective evaluation indexes of reconstructed images under different exposure times, which proves that our lensless imaging system is suitable for all-weather application fields.

We propose a lensless imaging system based on a full-aperture radial coded mask that produces a nearly depth-invariant point spread function (PSF), paving the way towards all-in-focus image reconstruction from a single PSF calibration. In contrast to conventional lens-based systems-where defocus blur is directly observed-lensless cameras rely on computational reconstruction using a calibrated PSF, making image quality highly sensitive to PSF mismatch. Our radial mask design ensures more consistent PSF structure across a wide depth range, mitigating this sensitivity and enabling captures that are less sensitive to depth variation. We validate the depth invariance of the proposed mask through experimental PSF analysis, demonstrating high correlation between PSFs captured at depths from 1 cm to 10 cm. We further evaluate the system through simulations and prototype experiments, showing improved robustness to depth mismatch when compared to conventional restricted-aperture masks. Finally, we introduce what we believe to be a novel shift-invariant reconstruction approach using an artificially extended PSF, enabled by the scale-invariant geometry of the radial pattern. This approximation allows for efficient and high-quality deconvolution across continuous depth ranges, demonstrating the practical feasibility of full-aperture lensless imaging.

Lensless cameras are characterized by several advantages (e.g., miniaturization, ease of manufacture, and low cost) as compared with conventional cameras. However, they have not been extensively employed due to their poor image clarity and low image resolution, especially for tasks that have high requirements on image quality and details such as text detection and text recognition. To address the problem, a framework of deep-learning-based pipeline structure was built to recognize text with three steps from raw data captured by employing lensless cameras. This pipeline structure consisted of the lensless imaging model U-Net, the text detection model connectionist text proposal network (CTPN), and the text recognition model convolutional recurrent neural network (CRNN). Compared with the method focusing only on image reconstruction, U-Net in the pipeline was able to supplement the imaging details by enhancing factors related to character categories in the reconstruction process, so the textual information can be more effectively detected and recognized by CTPN and CRNN with fewer artifacts and high-clarity reconstructed lensless images. By performing experiments on datasets of different complexities, the applicability to text detection and recognition on lensless cameras was verified. This study reasonably demonstrates text detection and recognition tasks in the lensless camera system, and develops a basic method for novel applications.

Mask-based lensless camera systems, replacing traditional lenses with a thin mask and sensor, offer unique imaging capabilities and enhanced privacy through optical encoding. This paper presents a facial identification system based on lensless camera images. The main contribution of this study is a framework that jointly learns lensless image reconstruction and recognition tasks using a dual-stream neural network. Unlike existing methods that focus solely on object recognition from lensless images, the proposed method leverages reconstructed image knowledge to improve face identification accuracy. By activating only the facial identification stream during inference, we ensure privacy protection while enhancing identification performance. Experiments demonstrate that our method outperforms state-of-the-art lensless facial identification methods.

Lensless light-field imaging is the process to encode the light field information of object through an optical encoder, and then recover the light field information of object through a reconstruction algorithm. In traditional lens-based light field imaging, each microlens in microlens-array corresponds to an angular sampling, and it needs to adjust the microlens-array to alter the angular resolution. In this paper, a commercial holographic diffuser is used as an optical encoder, and a flexible overlapping segmentation method of angular sampling is proposed for the Point Spread Function (PSF). The surface microstructures of holographic diffuser allow any region of it to encode and recover light information of object independently, corresponding to an angular sampling. Calibration of PSF for the lensless imaging system is done firstly, and encoded image of object is captured; then the PSF is divided into different regions as sub-PSFs, corresponding to different angular samplings; after that, light field images of object is reconstructed with corresponding sub-PSFs through reconstruction algorithm; with these light field images, digital refocusing can be achieved finally. Different from evenly segmentation of angular sampling in microlens-array, the overlapping segmentation method divides PSF into sub-PSFs while adjacent sub-PSFs overlap each other. This improves angular resolution of imaging system and ensures low error in reconstruction of light field images. Experiments show that, the overlapping segmentation method can guarantee the reconstruction accuracy of lensless light-field imaging system while flexibly adjusting and improving the angular resolution.

Lensless imaging microscopy has gained extensive application with the merits of system compactness and cost efficiency; however, its spatial resolution is usually compromised compared to conventional lens-based microscopes. To further enhance the spatial resolution, we built a lensless imaging system integrating a phase mask and a CMOS image sensor, and employed fluorescence fluctuation super-resolution microscopy (FF-SRM) algorithms to fully exploit the fluorescence intermittency (FI) characteristics of fluorescent molecules for high-resolution lensless image reconstruction. The study demonstrates that lensless image sequences processed by the Wiener deconvolution method can effectively retain the original fluorescence intermittency information, allowing for high-resolution reconstruction using FF-SRM algorithms. Furthermore, by combining expansion microscopy (ExM) and leveraging multi-algorithm synergy, we obtained additional improvements in spatial resolution and image quality for lensless imaging, facilitating clear visualization of biological subcellular organelles. This scheme offers a new pathway to achieve high spatial resolution imaging with practical advantages in simplicity and affordability.