Conventional Camera Research Articles

Large depth-of-field fringe projection profilometry applied to defocused scenes.

Fringe projection profilometry (FPP) is a widely adopted technique for three-dimensional (3D) reconstruction. However, its depth-of-field (DOF) is constrained when reconstructing defocused scenes, mainly due to limitations in the camera model and image blur. This study introduces a camera model based on the ideal optical system, which effectively reduces the systematic errors associated with the conventional pinhole camera model. A calibration method to determine the optical system parameters of the improved camera model is proposed. Additionally, the point spread function (PSF) of the camera is calibrated and the image is deblurred through non-blind deconvolution, thereby minimizing the phase aliasing resulting from defocus. Experimental results validate the potential of the proposed method for accurate 3D reconstruction in scenes with a wide depth range.

Enabling Space-Based Computed Cloud Tomography with a Mixed Integer Linear Programming Scheduler

Cumuliform clouds in Earth’s lower atmosphere play important roles in the radiative balance and hydrological cycle of the climate system. Yet key processes unfolding in these clouds, such as aerosol interactions and precipitation initiation, are still poorly understood. Current space-based cloud observation methods are inadequate for inferring the internal structure of clouds outside of a narrow transect that is probed actively by radar and lidar. Cloud tomography is an emerging technique that uses passive imaging of a cloud target from multiple locations with a large angular range to infer internal cloud structure. Many low convective clouds only have a lifetime of 15–25 min, necessitating autonomous scheduling of observation targets as they appear. Onboard autonomous scheduling is formulated as a mixed-integer optimization problem (MILP) with a finite time horizon and a reward scheme designed to maximize the angular range of the observations. A finite time horizon MILP scheduler is well suited to this mission because the short lifetime of low convective clouds creates a natural time horizon. This MILP can solve for an optimal observation schedule in a maximum time of 15 ms on a desktop CPU. The MILP scheduler is able to observe nearly 60% more targets than a conventional push-broom camera configuration. This initial result is promising and demonstrates the need for continued research in this area.

Efficient Visual Computing With Camera RAW Snapshots.

Conventional cameras capture image irradiance (RAW) on a sensor and convert it to RGB images using an image signal processor (ISP). The images can then be used for photography or visual computing tasks in a variety of applications, such as public safety surveillance and autonomous driving. One can argue that since RAW images contain all the captured information, the conversion of RAW to RGB using an ISP is not necessary for visual computing. In this paper, we propose a novel ρ-Vision framework to perform high-level semantic understanding and low-level compression using RAW images without the ISP subsystem used for decades. Considering the scarcity of available RAW image datasets, we first develop an unpaired CycleR2R network based on unsupervised CycleGAN to train modular unrolled ISP and inverse ISP (invISP) models using unpaired RAW and RGB images. We can then flexibly generate simulated RAW images (simRAW) using any existing RGB image dataset and finetune different models originally trained in the RGB domain to process real-world camera RAW images. We demonstrate object detection and image compression capabilities in RAW-domain using RAW-domain YOLOv3 and RAW image compressor (RIC) on camera snapshots. Quantitative results reveal that RAW-domain task inference provides better detection accuracy and compression efficiency compared to that in the RGB domain. Furthermore, the proposed ρ-Vision generalizes across various camera sensors and different task-specific models. An added benefit of employing the ρ-Vision is the elimination of the need for ISP, leading to potential reductions in computations and processing times.

Full-scale structural displacement measurement with camera ego-motion compensation using RGB and LiDAR cameras

Displacement measurements based on computer vision techniques have shown significant potential in structural health monitoring. However, as the inevitable camera ego-motion results in a drift in the measured displacement, compensation for such errors must be performed carefully. Although an additional RGB camera is considered in conventional methods for error compensation, the need for the sub-target limits its application. This study presents a dual-camera system, consisting of RGB and light detection and ranging (LiDAR) cameras, that enables accurate camera ego-motion compensation for long-term structural displacement measurements. In particular, the LiDAR camera of the proposed system can measure the six-degrees-of-freedom (6-DOF) ego-motion of the camera body regardless of the sub-target and its elaborate design. The accuracy of the proposed method is evaluated by comparing it with that of a conventional RGB camera. The resulting displacement can be employed to monitor the long-term behavior of full-scale bridges.

Beyond supervision: An unsupervised spatio-temporal point cloud noise modeling for event vision sensor

Noise modeling is a fundamental unexplored problem in many event camera applications. The noise modeling methods of conventional cameras are inadequate for capturing the intricate noise patterns unique to event cameras. To address this gap, we propose a novel event camera-driven noise modeling approach specifically designed to account for the unique characteristics of event camera data. Given the challenges in generating ground truth data, the proposed method is designed using unsupervised statistical techniques. The proposed noise model is camera-aware, i.e. it allows adaptability to the unique noise patterns of each sensor. It also facilitates asynchronous event processing as applied to downstream computer vision tasks. To validate the proposed noise modeling approach, we conducted extensive experiments in the context of denoising tasks. The results indicate that our noise model accurately represents the noise, leading to state-of-the-art performance across various datasets.

Characterization of a SiPM-based monolithic neutron scatter camera using dark counts

The Single Volume Scatter Camera (SVSC) Collaboration aims to develop portable neutron imaging systems for a variety of applications in nuclear non-proliferation. Conventional double-scatter neutron imagers are composed of several separate detector volumes organized in at least two planes. A neutron must scatter in two of these detector volumes for its initial trajectory to be reconstructed. As such, these systems typically have a large footprint and poor geometric efficiency. We report on the design and characterization of a prototype monolithic neutron scatter camera that is intended to significantly improve upon the geometrical shortcomings of conventional neutron cameras. The detector consists of a 50 mm×56 mm× 60 mm monolithic block of EJ-204 plastic scintillator instrumented on two faces with arrays of 64 Hamamatsu S13360-6075PE silicon photomultipliers (SiPMs). The electronic crosstalk is limited to < 5% between adjacent channels and < 0.1% between all other channel pairs. SiPMs introduce a significantly elevated dark count rate over PMTs, as well as correlated noise from after-pulsing and optical crosstalk. In this article, we characterize the dark count rate and optical crosstalk and present a modified event reconstruction likelihood function that accounts for them. We find that the average dark count rate per SiPM is 4.3 MHz with a standard deviation of 1.5 MHz among devices. The analysis method we employ to measure internal optical crosstalk also naturally yields the mean and width of the single-electron pulse height. We calculate separate contributions to the width of the single-electron pulse-height from electronic noise and avalanche fluctuations. We demonstrate a timing resolution for a single-photon pulse to be (128 ± 4) ps. Finally, coincidence analysis is employed to measure external (pixel-to-pixel) optical crosstalk. We present a map of the average external crosstalk probability between 2×4 groups of SiPMs, as well as the in-situ timing characteristics extracted from the coincidence analysis. Further work is needed to characterize the performance of the camera at reconstructing single- and double-site interactions, as well as image reconstruction.

Correction for geometric distortion in the flattened representation of pipeline external surface

Conventional cameras often produce external pipeline surface images with significant perspective-projection distortions, making determining the type and size of defects on the pipeline surface difficult. To address this issue, we present a novel approach for correcting the geometric distortion of a constant diameter pipeline (CDP) external surface image based on apparent contour pairs (ACPs). First, the camera is calibrated and the real projection position of ACPs accurately extracted. Second, a single-view perspective projection model of the CDP is adopted to realize a three-dimensional reconstruction of the pipeline. Subsequently, the flattened image of the CDP’s outer surface is obtained by performing equidistant sampling of cross sections, determining the visibility of discrete points, and mapping projection points onto a flat surface. Simulations and real experiments demonstrated the high accurate method in the correction of CDP images. Experiments conducted under different outdoor scenarios confirmed the robustness of the proposed method.

Widefield Super-Resolution Infrared Spectroscopy and Imaging of Autofluorescent Biological Materials and Photosynthetic Microorganisms Using Fluorescence Detected Photothermal Infrared (FL-PTIR).

We have demonstrated high-speed, super-resolution infrared (IR) spectroscopy and chemical imaging of autofluorescent biomaterials and organisms using camera-based widefield photothermal detection that takes advantage of temperature-dependent modulations of autofluorescent emission. A variety of biological materials and photosynthetic organisms exhibit strong autofluorescence emission under ultraviolet excitation and the autofluorescent emission has a very strong temperature dependence, of order 1%/K. Illuminating a sample with pulses of IR light from a wavelength-tunable laser source causes periodic localized sample temperature increases that result in a corresponding transient decrease in autofluorescent emission. A low-cost light-emitting diode-based fluorescence excitation source was used in combination with a conventional fluorescence microscopy camera to detect localized variations in autofluorescent emission over a wide area as an indicator of localized IR absorption. IR absorption image stacks were acquired over a range of IR wavelengths, including the fingerprint spectral range, enabling extraction of localized IR absorption spectra. We have applied widefield fluorescence detected photothermal IR (FL-PTIR) to an analysis of autofluorescent biological materials including collagen, leaf tissue, and photosynthetic organisms including diatoms and green microalgae cells. We have also demonstrated the FL-PTIR on live microalgae in water, demonstrating the potential for label-free dynamic chemical imaging of autofluorescent cells.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy

High-speed wide-field fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio at high frame rates, limiting their ability to detect faint fluorescent events. Here, we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high signal-to-noise ratio. In high-speed voltage imaging experiments, our image sensor significantly increases the output signal-to-noise ratio compared to a low-noise scientific CMOS camera (~2–3 folds). This signal-to-noise ratio gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

Single-layer Compton camera based on a 1 mm thick pixelated CDTE detectors

Compton camera is a novel γ-camera paradigm that relies upon the kinematics of Compton scattering for image reconstruction. A Conventional Compton camera uses two layers of sensors - the Absorber and the Scatterer. This work reports the proof of concept of a single-layer Compton camera (SLCC) where simultaneous Compton pairs events are registered in a single High-Z semiconductor sensor. The Hybrid pixel detector (HPD) is 1 mm thick, 256 x 256 square pixels with 55 μm pixel pitch CdTe sensor bonded to a Timepix3 readout ASIC. The superior spectroscopic imaging and fast timing capabilities of the Timepix3 readout ASIC coupled with a microscopic and highly pixelated CdTe enable simultaneous event detection of multi-energies occurring at multiple positions. The concept was exemplified by measuring the 122 KeV γ-ray emitted from a 57Co radioisotope source at two positions. Data were captured in the Timepix3 data-driven mode with the KATHERINE readout system via Gigabyte Ethernet data transfer. A bespoke Compton kinematics criterion algorithm implemented in Python 3 IDE was used for data analysis and Compton’s image reconstruction. Numerous events (5.2 million) were captured for 30-minute acquisitions. However, due to the thin nature of CdTe, fewer events (≈ 0.01 %) met the Compton event selection criteria. Nevertheless, the algorithm accurately pinpointed the radioisotope’s location, demonstrating proof of concept of the SLCC system.

Super-Resolution Microscopy with Dense Grid After Interpolation

The conventional camera image’s pixel size of super-resolution (SR) microscopy is almost the point spread function’s standard deviation, and the grid of a SR image is 1/8 of the pixel size in conventional compressed sensing-based SR microscopy. Here, based on smaller grid size and smaller pixel size, we proposed and generated different measurement matrices, and then compared and analyzed the SR reconstruction results based on the interpolated conventional camera image and different measurement matrices. The quality of the measurement matrix is related to the interpolation’s size. The larger the interpolation’s size, the better its performance. The quality of SR reconstruction depends not only on the measurement matrix’s performance, but also on the grid size. It is found that dense grid based on the size of interpolation equal to 2 can help to obtain the best SR reconstruction in simulation experiments when added Gaussian noise is lower.

Advancing 2D imaging of ammonium for biological application: Planar and nanoparticle-based ion-selective optodes

This study investigates the development and optimization of ammonium-selective chromoionophore-based optodes for two-dimensional (2D) imaging with conventional cameras. Ammonium, vital in various fields like agriculture, ecology, and health, requires accurate measurement. Traditional methods have limitations, leading to the exploration of optical sensors, or optodes, for direct imaging of ammonium gradients. The research covers planar and nanoparticle designs, exploring absorbance and fluorescence readouts with commercial cameras. The study extends to repeatability, response time, and selectivity of the optodes. Planar optodes, utilizing chromoionophores under different background colors, demonstrate improved characteristics with the addition of TiO2. The impact of plasticizer choice on response characteristics, including hysteresis, is explored. Nanoparticle designs using PVC and F127 are compared, with autoclaving identified as the superior sterilization method. The work effectively demonstrates 2D real-time NH4+ visualization using optode nanoparticles in agar gels. Finally, a proof-of-concept was performed to showcase a biological application for identifying urease-producing pathogens (Bacillus subtilis) without relying on pH changes. Overall, this study provides a comprehensive exploration of ammonium-selective optodes with promising applications in environmental monitoring and biological studies.

Sensitive assessment of bilirubin using various color space signals derived from captured images of microfluidic paper-based analytical devices

Many neonates’ manifest hyperbilirubinemia and jaundice during the first week following birth because of bilirubin accumulation in the blood. Newborns with severe hyperbilirubinemia develop neurotoxicity. Therefore, sensitive, accurate, cost-effective, and timely determination of bilirubin levels is highly desired. Herein, we present three simple systems of bilirubin assessment based on spectrophotometric, and digital imaging tools. Image-capturing devices pose a formidable challenge to sophisticated spectrophotometers and field-monitoring devices because of their simplicity and sensitivity. The first and second systems of bilirubin assessment, in solution, relied on spectrophotometric and conventional color responses of digital images collected with conventional or android smartphone camera, with LODs of 0.060–0.082 and LOQs of 0.200–0.273 ppm bilirubin. The third system utilized a microfluidic paper-based analytical device (μ-PAD) that is patterned with a wax printer on Whatman grade-1 chromatographic papers, with linear calibration graphs of up to 2.70,s and LODs of 0.035–0.040 and LOQs of 0.117–0.133 μg bilirubin/μ-PAD.

Analysis of the hybrid light field reconstruction and comparison with Richardson-Lucy Light Field Deconvolution

Conventional microscopes have a high spatial resolution and a low depth-of-field. Light field microscopes have a high depth-of-field but low spatial resolution. A new hybrid approach uses information from both systems to reconstruct a high-resolution light field [Appl. Opt. 58, A142 (2019)APOPAI0003-693510.1364/AO.58.00A142]. The resolution of the resulting light field is said to be limited only by diffraction and the size of the pixels. In this paper, we evaluate this method. Using simulation data we compare the output of the hybrid reconstruction algorithm with its simulated ground truth. Our analyses reveal that the observed improvement in the light field quality is not a consequence of data fusion or incorporation of information from a conventional camera, but rather the results of an intermediate interpolation step within the light field itself. This suggests that the required information is already inherent to the light field. By employing the Richardson-Lucy Light Field Deconvolution algorithm, we demonstrate that existing algorithms have already utilized this information.

Kidney dynamic SPECT acquisition on a CZT swiveling-detector ring camera: an in vivo pilot study

BackgroundLarge field of view CZT SPECT cameras with a ring geometry are available for some years now. Thanks to their good sensitivity and high temporal resolution, general dynamic SPECT imaging may be performed more easily, without resorting to dedicated systems. To evaluate the dynamic SPECT imaging by such cameras, we have performed an in vivo pilot study to analyze the kidney function of a pig and compare the results to standard dynamic planar imaging by a conventional gamma camera.MethodsA 7-week-old (12 kg) female Landrace pig was injected with [99mTc]Tc-MAG3 and a 30 min dynamic SPECT acquisition of the kidneys was performed on a CZT ring camera. A fast SPECT/CT was acquired with the same camera immediately after the dynamic SPECT, without moving the pig, and used for attenuation correction and drawing regions of interest. The next day the same pig underwent a dynamic planar imaging of the kidneys by a conventional 2-head gamma camera. The dynamic SPECT acquisition was reconstructed using a MLEM algorithm with up to 20 iterations, with and without attenuation correction. Time-activity curves of the total counts of each kidney were extracted from 2D and 3D dynamic images. An adapted 2-compartment model was derived to fit the data points and extract physiological parameters. Comparison of these parameters was performed between the different reconstructions and acquisitions.ResultsTime-activity curves were nicely fitted with the 2-compartment model taking into account the anesthesia and bladder filling. Kidney physiological parameters were found in agreement with literature values. Good agreement of these parameters was obtained for the right kidney between dynamic SPECT and planar imaging. Regional analysis of the kidneys can be performed in the case of the dynamic SPECT imaging and provided good agreement with the whole kidney results.ConclusionsDynamic SPECT imaging is feasible with CZT swiveling-detector ring cameras and provides results in agreement with dynamic planar imaging by conventional gamma cameras. Regional analysis of organs uptake and clearance becomes possible. Further studies are required regarding the optimization of acquisition and reconstruction parameters to improve image quality and enable absolute quantification.

Analysis of the Photogrammetric Use of 360-Degree Cameras in Complex Heritage-Related Scenes: Case of the Necropolis of Qubbet el-Hawa (Aswan Egypt).

This study shows the results of the analysis of the photogrammetric use of 360-degree cameras in complex heritage-related scenes. The goal is to take advantage of the large field of view provided by these sensors and reduce the number of images used to cover the entire scene compared to those needed using conventional cameras. We also try to minimize problems derived from camera geometry and lens characteristics. In this regard, we used a multi-sensor camera composed of six fisheye lenses, applying photogrammetric procedures to several funerary structures. The methodology includes the analysis of several types of spherical images obtained using different stitching techniques and the comparison of the results of image orientation processes considering these images and the original fisheye images. Subsequently, we analyze the possible use of the fisheye images to model complex scenes by reducing the use of ground control points, thus minimizing the need to apply surveying techniques to determine their coordinates. In this regard, we applied distance constraints based on a previous extrinsic calibration of the camera, obtaining results similar to those obtained using a traditional schema based on points. The results have allowed us to determine the advantages and disadvantages of each type of image and configuration, providing several recommendations regarding their use in complex scenes.

TinyMSI: A cost-effective handheld device for non-contact diabetic wound monitoring

Devices characterizing diabetic foot ulcers and other wounds currently fall into two categories. Expensive clinically-oriented devices that use mature technologies such as X-ray CT and hyperspectral imaging or low-cost solutions that leverage deep learning to infer wound characterization from conventional smartphone camera images or simple surrogate markers. Mature medical-grade devices are too expensive for primary care and assisted living facilities. Low-cost solutions rely too much on indirect statistical inference to be clinically suitable. Therefore, we propose a device that leverages mature, clinically suitable optical technologies to provide a solution for these facilities. Recognizing that individual combinations of 1–2 bands of active illumination are used individually to capture pulsation, vascular, and oxygenation images. We combine all these bands into a single multispectral lighting source to create a multi-functional, reliable device for wound assessment. We selected these bands to leverage CMOS cameras near orthogonality between the RGB channels and leverage that CMOS cameras can also sense near IR light if a filter is not present, reducing overall system complexity and needed bands. For each function, the necessary lights are turned on, and the captured raw video is then fed to the corresponding sequence of image processing steps. No deep learning models are used, so large training datasets are not required. Our device is also small, lightweight, and handheld.

Instant plastic waste detection on shores using laser-induced fluorescence and associated hyperspectral imaging

Plastic pollution is a rising environmental issue, with millions of tons of plastic debris collecting in the world's seas and on its shores. Hyperspectral imaging (HSI) has become increasingly widely used as a more precise approach that can identify targets in remote sensing aquatic missions. The interference from other beach materials, and the need for proper identification of litter types can make identifying dumped plastics on sand-surrounded beaches challenging. This study lays the groundwork for a physical laboratory setting for images captured by a hyperspectral (HS) imager. The suggested testing setup included the development of a fluorescence signature for the target theater of operations (low-density polyethylene (LD-PE) and wood surrounded by sand) for detecting polymers in a simulated beach environment using the laser-induced fluorescence (LIF) approach. Initially using broadband-spectrum light, strong sample diffuse reflectance contrast is observed in the imaging at wavelengths between 400 and 460 nm. Next, a dedicated LIF system for plastic litter discovery was developed using an ultraviolet (UV) laser source. Initial findings show that there is a distinct fluorescence signal for plastics at 450 nm and at 750 nm for wood. Our pilot studies support current efforts to determine the optimum spectral signature that these polymers will appear with clarity on shorelines using an inexpensive imagery combined with our UV LIF approach, which may have an impact on applications for the detection of beach pollution. The knowledge gained from this study can be used to construct reliable aerial conventional cameras for plastic waste environmental monitoring and management.

Single-shot synthetic wavelength imaging: Sub-mm precision ToF sensing with conventional CMOS sensors

We present a novel single-shot interferometric ToF camera targeted for precise 3D measurements of dynamic objects. The camera concept is based on Synthetic Wavelength Interferometry, a technique that allows retrieval of depth maps of objects with optically rough surfaces at submillimeter depth precision. In contrast to conventional ToF cameras, our device uses only off-the-shelf CCD/CMOS detectors and works at their native chip resolution (as of today, theoretically up to 20 Mp and beyond). Moreover, we can obtain a full 3D model of the object in single-shot, meaning that no temporal sequence of exposures or temporal illumination modulation (such as amplitude or frequency modulation) is necessary, which makes our camera robust against object motion.In this paper, we introduce the novel camera concept and show the first measurements that demonstrate the capabilities of our system. We present 3D measurements of small (cm-sized) objects with > 2 Mp point cloud resolution (the native pixel resolution of our used detector) and up to sub-mm depth precision. We also report a “single-shot 3D video” acquisition and a first single-shot “Non-Line-of-Sight” measurement. Our technique has great potential for high-precision applications with dynamic object movement, e.g., in AR/VR, industrial inspection, medical imaging, and imaging through scattering media like fog or human tissue.

Employing an innovative underwater camera to improve electronic monitoring in the commercial Gulf of Mexico reef fish fishery.

Vessel electronic monitoring (EM) systems used in fisheries around the world apply a variety of cameras to record catch as it is brought on deck and during fish processing activities. In EM work conducted by the Center for Fisheries Electronic Monitoring at Mote (CFEMM) in the Gulf of Mexico commercial reef fish fishery, there was a need to improve upon current technologies to enhance camera views for accurate species identification of large sharks, particularly those that were released while underwater at the vessel side or underneath the hull. This paper describes how this problem was addressed with the development of the first known EM system integrated underwater camera (UCAM) with a specialized vessel-specific deployment device on a bottom longline reef fish vessel. Data are presented based on blind video reviews from CFEMM trained reviewers of the resulting UCAM video footage compared with video from only the overhead EM cameras from 68 gear retrievals collected from eight fishing trips. Results revealed that the UCAM was a successful tool for capturing clear underwater video footage of released large (>2m) sharks to enable reviewers to improve individual species identification, determination, and fate by 34.4%. This was particularly important for obtaining data on incidental catches of large protected shark species. It also provided clear underwater imagery of the presence of potential predators such as marine mammals close to the vessel, more specifically bottlenose dolphin (Tursiops truncatus) during gear retrieval, which often damaged or removed catch. This information is intended to assist researchers in need of gathering critical data on bycatch in close proximity to a vessel in which conventional overhead EM cameras are limited.