Single Image Sensor Research Articles

Biologically inspired intraoral camera for multifunctional dental imaging

Biological vision offers intriguing inspiration for functional features in imaging systems with small form factors. We report biologically inspired intraoral camera (BIOC) for assorted dental imaging. This fully packaged BIOC features a convex-concave lens, inverted microlens arrays (iMLAs), LED module, and a single CMOS image sensor on a flexible printed circuit board in a handpiece holder. The iMLAs also collect light from wide angles by mounting the convex-concave lens to increase the viewing angle. The clinical trials have been successfully conducted for real-time and multifunctional intraoral monitoring of human teeth, including infinite depth of field, close-up, wide field-of-view, three-dimensional, and autofluorescence imaging. This biomedical camera provides insights for functional imaging not only in dental applications but also in surgical robots and endoscopy applications.

Effect of different imaging modalities on the performance of a CNN: An experimental study on damage segmentation in infrared, visible, and fused images of concrete structures

This study investigates the semantic segmentation of common concrete defects when using different imaging modalities. One pre-trained Convolutional Neural Network (CNN) model was trained via transfer learning and tested to detect concrete defect indications, such as cracks, spalling, and potential subsurface defects. We compared the model's performance using datasets of visible, thermal, and fused images. In addition, the impact of using different image enhancement techniques, such as histogram equalization and resolution improvement, was investigated. The data was collected from four different concrete structures using four infrared cameras with distinct sensitivities and resolutions, with imaging campaigns conducted during autumn, summer, and winter. Although specific defects can be detected in monomodal images, the results demonstrated that a larger number of defect classes could be detected using fused images with the same viewpoint and resolution as the single-sensor image without significant loss of information. In addition, the output of one hypothesis test showed that the image enhancement techniques provided no significant improvement in the CNN performance for this case of study, even though they resulted in enhanced images with higher information content (entropy) than the original images.

A wide-field and high-resolution lensless compound eye microsystem for real-time target motion perception

Optical measurement systems suffer from a fundamental tradeoff between the field of view (FOV), the resolution and the update rate. A compound eye has the advantages of a wide FOV, high update rate and high sensitivity to motion, providing inspiration for breaking through the constraint and realizing high-performance optical systems. However, most existing studies on artificial compound eyes are limited by complex structure and low resolution, and they focus on imaging instead of precise measurement. Here, a high-performance lensless compound eye microsystem is developed to realize target motion perception through precise and fast orientation measurement. The microsystem splices multiple sub-FOVs formed by long-focal subeyes, images targets distributed in a panoramic range into a single multiplexing image sensor, and codes the subeye aperture array for distinguishing the targets from different sub-FOVs. A wide-field and high resolution are simultaneously realized in a simple and easy-to-manufacture microelectromechanical system (MEMS) aperture array. Moreover, based on the electronic rolling shutter technique of the image sensor, a hyperframe update rate is achieved by the precise measurement of multiple time-shifted spots of one target. The microsystem achieves an orientation measurement accuracy of 0.0023° (3σ) in the x direction and 0.0028° (3σ) in the y direction in a cone FOV of 120° with an update rate ~20 times higher than the frame rate. This study provides a promising approach for achieving optical measurements with comprehensive high performance and may have great significance in various applications, such as vision-controlled directional navigation and high-dynamic target tracking, formation and obstacle avoidance of unmanned aerial vehicles.

Abstract 3416: Clearance of tumor-derived extracellular vesicle heterogeneity in aqueous humor after chemotherapy in retinoblastoma eyes

Abstract Introduction: Aqueous humor (AH), the clear fluid in front of the eye, maintains the pressure and vitality of ocular tissues. This fluid is accessible via the cornea which enables use of AH as a liquid biopsy for intraocular disease. Retinoblastoma (Rb), an intraocular cancer in children, is unique in that direct tumor biopsy is prohibited, thus liquid biopsy has great clinical potential. cfDNA in the AH as a biomarker for Rb patients has been investigated and extracellular vesicles (EVs) are detectable in the AH from adults. However, AH EVs in Rb have previously never been explored. We know very little about the heterogeneity of AH EV populations in ocular cancers. Materials and Methods: 27 AH samples from 19 patients from 13 tumor-free eyes and 11 Rb eyes were collected. Rb eyes were further divided into treatment-naïve (Rb_Tn) and treatment-active (Rb_Tx) subgroups. Unprocessed AH samples were subjected to Nanoparticle Tracking Analysis (NTA) (Nanosight NS300) for size distribution and concentration, and to Single Particle-Interferometric Reflectance Imaging Sensor (SP-IRIS) (Exoview R100) for fluorescent-based immunophenotyping of EV marker expression (CD63, CD81, and CD9). Results: NTA demonstrated the concentration of AH EVs is 3.11 x 109-1.38 x 1010 vesicles/mL; majority sized 76.8-103 nm. Study group comparisons showed that non-tumor containing eyes had a smaller nanoparticle mean size compared to Rb containing eyes (P = 0.002). More EVs were significantly detected in Rb_Tn containing eyes compared to Rb_Tx containing eyes (P = 0.022), suggesting the possible presence of tumor-derived EVs in Rb_Tn which were subsequently eliminated by treatment. SP-IRIS revealed distinct patterns of tetraspanin expression of AH small EVs (sEVs). Mono-CD63+ EVs were identified to be the most dominant sEV subpopulation from AH across non-tumor and Rb_Tx eyes. However, more diverse sEV subpopulation profile was detected in Rb_Tn AH samples. Significantly lower percentage of mono-CD63+ EVs could be determined in Rb_Tn eyes compared to Rb_Tx eyes (70.3% vs. 96.1%, P = 0.001). A significant accumulation of CD9/CD63 (4.3% vs. 1.1%, P = 0.035), CD63/CD81 (20.7% vs. 2.0%, P = 0.012) and CD9/CD63/CD81 (4.8% vs. 0.8%, P = 0.022) subpopulations in Rb_Tn was also observed. An enriched mono-CD63+ sEV subpopulation identified in AH indicates this is a potential AH-specific biomarker. In the setting of Rb there was a more heterogeneous population of sEVs which normalized with treatment. Conclusions: Small EVs are readily detectable in unprocessed AH with a dominant mono-CD63+ subpopulation in AH regardless of pediatric eye disease states. Tetraspanin colocalization analysis indicates the clearance of retinoblastoma-derived EV heterogeneity by chemotherapy. These novel finding suggests a potential clinical application of measurement of sEV subpopulations in AH samples to monitor response to therapy. Citation Format: Chen-Ching Peng, Deborah Im, Shreya Sirivolu, Bibiana Reiser, Aaron Nagiel, Paolo Neviani, Liya Xu, Jesse L. Berry. Clearance of tumor-derived extracellular vesicle heterogeneity in aqueous humor after chemotherapy in retinoblastoma eyes [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 3416.

Dual-wavelength and double-field-of-view quantitative phase microscopy using a Fresnel bi-prism

We present a novel stable quantitative phase measurement technique for extending the imaging area sensing with the capability of recording two wavelengths in a single shot. For this purpose, each wavelength is separated into three beams using a Fresnel bi-prism and they interfere in the CCD camera by a simple optic alignment. The final pattern created in the camera with the six beam contained both wavelengths information that their field of view is extended two times. The feasibility of this technique is experimentally demonstrated by dispersion measurement of silica beads using two wavelengths image of two different areas of silica beads with a single image sensor.

Microlens arrays enable variable-focus imaging

Microlens array (MLA) is a key part of optical element for biomedical inspection, lab-on-a-chip device, and light-field cameras. However, conventional MLAs always have only one focal plane, which makes them difficult to capture targets at different positions. To solve this problem, a planar multi-focus MLA (MF-MLA) was prepared on a polydimethylsiloxane (PDMS) substrate by a combination of femtosecond laser wet etching (FLWE) and soft lithography techniques. The modulation transfer function (MTF) values of the microlenses all exceeded 0.2 when the spatial frequency was less than 200 lp/mm, indicating the high imaging ability of the lenses. The reported MF-MLA had three different focal lengths, so that targets at different positions could be easily imaged on a single image sensor. In addition, MLAs with more focal points could be fabricated by selecting appropriate processing parameters. It is anticipated that the as-fabricated MF-MLA will become a promising device to improve the performance of the optical system, especially in optical remote sensing, biomedical imaging, and machine vision.

Single vesicle analysis of aqueous humor in pediatric ocular diseases reveals eye specific CD63‐dominant subpopulations

Aqueous humor (AH), the clear fluid in front of the eye, maintains the pressure and vitality of ocular tissues. This fluid is accessible via the clear cornea which enables use of AH as a liquid biopsy source of biomarkers for intraocular disease. Extracellular vesicles are detectable in the AH and small extracellular vesicles (sEVs) are present in the AH from adults. However, EVs in AH from pediatric eyes in vivo have never previously been explored. We know very little about the heterogeneity of AH EV populations in ocular disease. Twenty-seven processing-free AH samples from 19 patients across four different pediatric ocular diseases were subjected to Nanoparticle Tracking Analysis (NTA) and Single Particle-Interferometric Reflectance Imaging Sensor (SP-IRIS) analysis. NTA demonstrated the concentration of AH EV/EPs is 3.11 × 109-1.38 × 1010 particles/ml; the majority sized 76.8-103 nm. SP-IRIS revealed distinct patterns of tetraspanin expression of AH sEVs. An enriched mono-CD63+ sEV subpopulation identified in AH indicates this is a potential AH-specific biomarker. In the setting of retinoblastoma there was a more heterogeneous population of sEVs which normalized with treatment. This suggests a potential clinical application of direct measurement of sEV subpopulations in AH samples to monitor successful tumor response to therapy.

Adaptive and Progressive Multispectral Image Demosaicking

Common consumer RGB cameras use a single image sensor to capture RGB color images with a color filter array (CFA) placed in front of the image sensor. This system captures particularly one color sample ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> , or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$B$</tex-math></inline-formula> ) at each pixel location. This captured raw image is called a CFA image. The CFA demosaicking is used then to construct the complete color image. A similar approach of working with a single image sensor and extension of CFA to multispectral filter array (MSFA) enables us to develop a low-cost multispectral camera with the help of MSFA demosaicking. However, due to many spectral bands and their very sparse sampling, extending CFA demosaicking to MSFA demosaicking is not easy. This paper proposes an adaptive and progressive MSFA demosaicking method for various MSFA patterns for 5 to 15 bands multispectral images. Binary tree-based MSFA patterns are used to create MSFA images, and these MSFA patterns can be designed for any N-band MSFA image. The spectral bands are arranged specifically so that the middle band has the highest probability of appearance (PoA) in the MSFA pattern. The middle band is interpolated first using PoA based progressive adaptive interpolation, and the modified bilinear spectral difference method is used to estimate all other spectral bands with the interpolated middle band’s help. Experimental results and comparative analysis on two different datasets reveal that our proposed MSFA demosaicking method outperforms the existing generic MSFA demosaicking methods in terms of computational time and various image quality metrics considered.

CFNet: A Cross Fusion Network for Joint Land Cover Classification Using Optical and SAR Images

As two of the most widely used remote sensing images, optical and synthetic aperture radar (SAR) images show abundant and complementary information on the same target owing to their individual imaging mechanisms. Consequently, using optical and SAR images simultaneously can better describe the inherent features of the target, and thus, be beneficial for subsequent remote sensing applications. In this article, we propose a novel modular fully convolutional network model to improve the accuracy of land cover classification by fully exploiting the complementary features of the two sensors. We investigate where and how to fuse the two images in the joint classification network. A cross-gate module with a bidirectional information flow is proposed to achieve the best fusion performance. In addition, to validate the proposed model, we construct a multiclass land cover classification dataset. Exhaustive experiments show that the proposed joint classification network presents superior results than state-of-the-art classification models using single-sensor images.

Holographic single-image depth reconstruction

In this article a camera-based single-image sensor is presented, that is able to measure the distance of multiple object points. The experimental results show an accuracy of 8,51 μm within a depth range of 20 mm. The sensor consists of a camera, whose lens is upgraded with a diffractive optical element (DOE). It fulfils two tasks: adding a vortex point spread function (PSF) and replication of the vortex PSFs to a predefined pattern of K spots. Both, shape and rotation of the vortex PSF is sensitive to defocus. By applying the depth reconstruction to each of the K replications and averaging the results, we experimentally show, that the reconstruction of the depth signal can be improved by a factor of up to 3.

A General Framework of Remote Sensing Epipolar Image Generation

Epipolar images can improve the efficiency and accuracy of dense matching by restricting the search range of correspondences from 2-D to 1-D, which play an important role in 3-D reconstruction. As most of the satellite images in archives are incidental collections, which do not have rigorous stereo properties, in this paper, we propose a general framework to generate epipolar images for both in-track and cross-track stereo images. We first investigate the theoretical epipolar constraints of single-sensor and multi-sensor images and then introduce the proposed framework in detail. Considering large elevation changes in mountain areas, the publicly available digital elevation model (DEM) is applied to reduce the initial offsets of two stereo images. The left image is projected into the image coordinate system of the right image using the rational polynomial coefficients (RPCs). By dividing the raw images into several blocks, the epipolar images of each block are parallel generated through a robust feature matching method and fundamental matrix estimation, in which way, the horizontal disparity can be drastically reduced while maintaining negligible vertical disparity for epipolar blocks. Then, stereo matching using the epipolar blocks can be easily implemented and the forward intersection method is used to generate the digital surface model (DSM). Experimental results on several in-track and cross-track images, including optical-optical, SAR-SAR, and SAR-optical pairs, demonstrate the effectiveness of the proposed framework, which not only has obvious advantages in mountain areas with large elevation changes but also can generate high-quality epipolar images for flat areas. The generated epipolar images of a ZiYuan-3 pair in Songshan are further utilized to produce a high-precision DSM.

Adaptive Weight Fusion Algorithm of Infrared and Visible Image Based on High-Frequency Domain CNN

Aiming at covering the shortage of single source sensor imaging and improving the contrast between the target and the background in image, this paper proposes an adaptive weight fusion algorithm of infrared and visible image based on a High-frequency Domain Convolutional Neural Network (HDCNN). Firstly, the high and low frequency components of the original image are obtained by using the Daubechies wavelet transform, and then a high-frequency domain convolutional neural network which can detect the frequency information ratio of infrared and visible light in the high-frequency subband is trained. Secondly, the network is used to perform adaptive weight fusion for the high frequency components and regional energy is used for fusion of the low frequency components. Finally, the fusion image is obtained by inverse wavelet transform. A large number of experiments have proved that the algorithm in this paper has a greater improvement over similar comparison algorithms in objective evaluation metrics such as standard deviation, spatial frequency and average gradient. The algorithm enhances the contrast between the target and the background in the fusion image, and enriches the characteristic information of the target itself.

Single-sensor filter-intensified fiber optic 4D tomographic CH* chemiluminesence flame measurements.

Chemiluminesence is one of the most common reacting flow visualization techniques, in which a narrowband wavelength associated with species radicals is captured on an image sensor. This technique can provide key insights into flame physics, particularly when a full 3D field can be extracted. Tomographic imaging can capture 3D instantaneous information, but becomes impractical to use when the technique is extended to larger, heavier high-speed imaging equipment. This has resulted in many researchers using fiber-based endoscopes (FBE) to capture multiple views on a single image sensor. However, the introduction of the fiber bundle and corresponding equipment to image specific combustion radicals results in a low SNR imaging environment when captured at high repetition rates. Additionally, when varying equivalence ratios, the signal can become further diminished because the combustion radicals signal weakens at fuel rich and fuel lean conditions. Therefore, this work determines the efficacy of using a filter-intensified FBE approach to capture the CH* radical of a hydrocarbon flame under varying equivalence ratio conditions by comparing the resulting reconstructions to previous quantitative flame measurements.

Biologically Inspired Ultrathin Contact Imager for High‐Resolution Imaging of Epidermal Ridges on Human Finger

AbstractCompound eyes exhibit compelling vision schemes due to their unique visual ultrastructures. Here, a biologically inspired ultrathin contact imager (BUCI) for high‐resolution contact fingerprint imaging is reported. The BUCI contains ultrathin arrayed camera, light‐emitting diode arrays, and light guide plate (LGP). The ultrathin arrayed camera is fully packaged with microlens arrays, variable micropinhole arrays (MLA‐VMA), and a single complementary metal oxide semiconductor image sensor. The ultrathin arrayed camera shows a large field of view of 77.3° and a constant MTF50 of 89.4 cycles mm–1, relatively higher than a comparable single lens camera at the distance less than 36.8 mm. The channel images of a human fingertip are directly obtained by using frustrated total internal reflection and stitched to a single and high‐resolution full‐field image for a fingertip of 5.7 mm × 4.9 mm. The BUCI provides a new platform for high‐resolution contact imaging in biometric or healthcare applications.

Deep image enhancement for ill light imaging.

Imaging in the natural scene under ill lighting conditions (e.g., low light, back-lit, over-exposed front-lit, and any combinations of them) suffers from both over- and under-exposure at the same time, whereas processing of such images often results in over- and under-enhancement. A single small image sensor can hardly provide satisfactory quality for ill lighting conditions with ordinary optical lenses in capturing devices. Challenges arise in the maintenance of a visual smoothness between those regions, while color and contrast should be well preserved. The problem has been approached by various methods, including multiple sensors and handcrafted parameters, but extant model capacity is limited to only some specific scenes (i.e., lighting conditions). Motivated by these challenges, in this paper, we propose a deep image enhancement method for color images captured under ill lighting conditions. In this method, input images are first decomposed into reflection and illumination maps with the proposed layer distribution loss net, where the illumination blindness and structure degradation problem can be subsequently solved via these two components, respectively. The hidden degradation in reflection and illumination is tuned with a knowledge-based adaptive enhancement constraint designed for ill illuminated images. The model can maintain a balance of smoothness and contribute to solving the problem of noise besides over- and under-enhancement. The local consistency in illumination is achieved via a repairing operation performed in the proposed Repair-Net. The total variation operator is optimized to acquire local consistency, and the image gradient is guided with the proposed enhancement constraint. Finally, a product of updated reflection and illumination maps reconstructs an enhanced image. Experiments are organized under both very low exposure and ill illumination conditions, where a new dataset is also proposed. Results on both experiments show that our method has superior performance in preserving structural and textural details compared to other states of the art, which suggests that our method is more practical in future visual applications.

Snapshot compressive spectral depth imaging from coded aberrations.

Compressive spectral depth imaging (CSDI) is an emerging technology aiming to reconstruct spectral and depth information of a scene from a limited set of two-dimensional projections. CSDI architectures have conventionally relied on stereo setups that require the acquisition of multiple shots attained via dynamically programmable spatial light modulators (SLM). This work proposes a snapshot CSDI architecture that exploits both phase and amplitude modulation and uses a single image sensor. Specifically, we modulate the spectral-depth information in two steps. Firstly, a deformable mirror (DM) is used as a phase modulator to induce a focal length sweeping while simultaneously introducing a controlled aberration. The phase-modulated wavefront is then spatially modulated and spectrally dispersed by a digital micromirror device (DMD) and a prism, respectively. Therefore, each depth plane is modulated by a variable phase and binary code. Complimentary, we also propose a computational methodology to recover the underlying spectral depth hypercube efficiently. Through simulations and our experimental proof-of-concept implementation, we demonstrate that the proposed computational imaging system is a viable approach to capture spectral-depth hypercubes from a single image.

Compressive Spectral Imaging Via Virtual Side Information

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

An Over 120 dB Single Exposure Wide Dynamic Range CMOS Image Sensor With Two-Stage Lateral Overflow Integration Capacitor

This article presents a prototype linear response single exposure CMOS image sensor with two-stage lateral overflow integration capacitors (LOFIC) exhibiting over the 120-dB dynamic range (DR) with 11.4 Me <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-</sup> full well capacity (FWC) and maximum signal-to-noise ratio (SNR) of 70 dB. The measured SNR at all switching points were over 35 dB thanks to the proposed two-stage LOFIC.

Light-field multi-spectral radiation thermometry.

This Letter proposes light-field multi-spectral radiation thermometry based on an unfocused light-field camera, which can simultaneously record directions and intensities of incident rays. In this method, the direction information of rays is substituted by radiation spectrums via placing an array of filters in front of camera main lens, such that the image sensor can simultaneously acquire spectrums and intensities of rays. By decoupling a raw multi-spectral light-field (MSLF) image and utilizing traditional multi-wavelength pyrometer algorithms, the scalar field of surface temperature distribution can be achieved. To verify the method, measurement errors of different temperature levels on several typical areas of MSLF images are analyzed. In addition, the validation experiment demonstrates that accurate surface temperature measurement can be achieved with a single lens, single monochromatic image sensor, and just one snapshot in the proposed method.

U-GAN Model for Infrared and Visible Images Fusion

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.