Pixel Camera Research Articles

Parallel Single-Pixel Imaging: A General Method for Direct\u2013Global Separation and 3D Shape Reconstruction Under Strong Global Illumination

We present parallel single-pixel imaging (PSI), a photography technique that captures light transport coefficients and enables the separation of direct and global illumination, to achieve 3D shape reconstruction under strong global illumination. PSI is achieved by extending single-pixel imaging (SI) to modern digital cameras. Each pixel on an imaging sensor is considered an independent unit that can obtain an image using the SI technique. The obtained images characterize the light transport behavior between pixels on the projector and the camera. However, the required number of SI illumination patterns generally becomes unacceptably large in practical situations. We introduce local region extension (LRE) method to accelerate the data acquisition of PSI. LRE perceives that the visible region of each camera pixel accounts for a local region. Thus, the number of detected unknowns is determined by local region area, which is extremely beneficial in terms of data acquisition efficiency. PSI possesses several properties and advantages. For instance, PSI captures the complete light transport coefficients between the projector–camera pair, without making specific assumptions on measured objects and without requiring special hardware and restrictions on the arrangement of the projector–camera pair. The perfect reconstruction property of LRE can be proven mathematically. The acquisition and reconstruction stages are straightforward and easy to implement in the existing projector–camera systems. These properties and advantages make PSI a general and sound theoretical model to decompose direct and global illuminations and perform 3D shape reconstruction under global illumination.

Auto-Tuned Event-Based Perception Scheme for Intrusion Monitoring With UAS

This paper presents an asynchronous event-based scheme for automatic intrusion monitoring using Unmanned Aerial Systems (UAS). Event cameras are neuromorphic sensors that capture the illumination changes in the camera pixels with high temporal resolution and dynamic range. In contrast to conventional frame-based cameras, they are naturally robust against motion blur and lighting conditions, which make them ideal for outdoor aerial robot applications. The presented scheme includes two main perception components. First, an asynchronous event-based processing system efficiently detects intrusions by combining several asynchronous event-based algorithms that exploit the advantages of the sequential nature of the event stream. The second is an off-line training mechanism that adjusts the parameters of the event-based algorithms to a particular surveillance scenario and mission. The proposed perception system was implemented in ROS for on-line execution on board UAS, integrated in an autonomous aerial robot architecture, and extensively validated in challenging scenarios with a wide variety of lighting conditions, including day and night experiments in pitch dark conditions.

Adaptive Phase Correction for Phase Measuring Deflectometry Based on Light Field Modulation

The phase measuring deflectometry is a powerful measuring method for complex optical surfaces, which captures the reflected fringe images associated with a displaying screen and calculates the normal vectors of the surface under test (SUT) accordingly. The captured images are usually set conjugate to the SUT, which in turn makes the screen defocused. As a result, the blurring effect caused by the point spread function (PSF) of the off-axis catadioptric imaging system can bias the solved phases. In order to correct the phase errors, a light field is constructed based on the Fourier compressive sensing method to describe the light transmission between the screen and camera pixels. Fringe modulation is conducted to enhance the robustness against noise, and then, space-variant PSFs can be extracted from the light field. The true phases are obtained by solving a Wiener deconvolution problem, with the merit function adaptively regularized by adjusting the damping parameter. The proposed method can correct adaptively the phase errors induced by the complex aberrations. Compared to the reference measurements, the form accuracy can be improved by four times.

Influence of spectral characteristics of light sources on measuring space camera modulation transfer function

Modulation transfer function (MTF) measurement is a major means to evaluate the imaging quality of a space camera. The influence caused by the spectral characteristic of light source on the MTF results is not negligible, because the transmittance and color aberration of optical systems, and quantum efficiency of the space camera detectors are all spectrally related. Thus, MTF results tested by different light sources are different from each other. To address this problem, we propose a method to analyze the influence of spectral characteristics of light sources on measuring the MTF of space cameras. In addition, the devices and methods are designed to calibrate the spectral response and monochrome point spread function (PSF) of space camera. A Sigma lens (focal length: 1000mm, F number: 5.6) and a Cannon EOS 5DSR camera (pixel size: 4.14 μm) are combined into an experimental space camera, whose spectral response is calibrated with a monochromator (Omno30300, NBeT) and a spectral radiometer (FieldSpec, ASD). We calibrate the monochrome PSF of the Sigma lens with the same monochromator and a CCD (PIXIS 1024, Princeton Instruments, pixel size: 13 μm) micro-measuring system (20X objective). During the calibration of spectral response and monochrome PSF, the same collimator (focus: 5000 mm, F number: 10) is used. With using the proposed method and those calibrating data, we compute the theoretical values of the MTF of a space camera measured separately with five different light sources. The results indicate that MTF measured by a xenon lamp is greatly different from those MTFs measured by the other four light sources. Comparisons of those theoretically calculated MTFs, separately, show that the MTF measured by a tungsten halogen lamp is greater than the MTF measured by a xenon lamp at each spatial frequency. The deviation between those two lamps reaches a maximum value of 0.075 in the medium-high frequency zone. Furthermore, in order to verify those theoretical conclusions, a platform including a collimator and the previous space camera is constructed. The MTFs measured by a tungsten halogen lamp and a xenon lamp are computed with the slanted-edge method respectively. The results demonstrate that the distributions and deviations of the MTFs tested by those two lamps are identical to those theoretical results at each spatial frequency, with the maximum deviation being 0.057. The theoretical and experimental results demonstrate that the suggested method can accurately calculate the influence of spectral characteristics of light sources on measuring MTF of space cameras. The proposed method can also be adopted to investigate the influence of spectral characteristics of light sources on MTF of optical systems in the design or test stages.

PySTACHIO: Python Single-molecule TrAcking stoiCHiometry Intensity and simulatiOn, a flexible, extensible, beginner-friendly and optimized program for analysis of single-molecule microscopy data

As camera pixel arrays have grown larger and faster, and optical microscopy techniques ever more refined, there has been an explosion in the quantity of data acquired during routine light microscopy. At the single-molecule level, analysis involves multiple steps and can rapidly become computationally expensive, in some cases intractable on office workstations. Complex bespoke software can present high activation barriers to entry for new users. Here, we redevelop our quantitative single-molecule analysis routines into an optimized and extensible Python program, with GUI and command-line implementations to facilitate use on local machines and remote clusters, by beginners and advanced users alike. We demonstrate that its performance is on par with previous MATLAB implementations but runs an order of magnitude faster. We tested it against challenge data and demonstrate its performance is comparable to state-of-the-art analysis platforms. We show the code can extract fluorescence intensity values for single reporter dye molecules and, using these, estimate molecular stoichiometries and cellular copy numbers of fluorescently-labeled biomolecules. It can evaluate 2D diffusion coefficients for the characteristically short single-particle tracking data. To facilitate benchmarking we include data simulation routines to compare different analysis programs. Finally, we show that it works with 2-color data and enables colocalization analysis based on overlap integration, to infer interactions between differently labelled biomolecules. By making this freely available we aim to make complex light microscopy single-molecule analysis more democratized.

Compressive Sampling Using a Pushframe Camera

The recently described pushframe imager, a parallelized single pixel camera capturing with a pushbroom-like motion, is intrinsically suited to both remote-sensing and compressive sampling. It optically applies a 2D mask to the imaged scene, before performing light integration along a single spatial axis, but previous work has not made use of the architecture's potential for taking measurements sparsely. In this paper we develop a strongly performing static binarized noiselet compressive sampling mask design, tailored to pushframe hardware, allowing both a single exposure per motion time-step, and retention of 2D correlations in the scene. Results from simulated and real-world captures are presented, with performance shown to be similar to that of immobile and hence inappropriate for satellite use whole-scene imagers. A particular feature of our sampling approach is that the degree of compression can be varied without altering the pattern, and we demonstrate the utility of this for efficiently storing and transmitting multi-spectral images.

Urban surface temperature observations from ground-based thermography: intra- and inter-facet variability

Ground based thermal cameras are used to observe urban surface temperatures (Ts) with an unprecedented combination of: temporal and spatial resolution (5 min and ~ 0.5 m → 2.5 m), spatial extent (3.9 ha), instrument number (6 static cameras) and surface heterogeneity (mixed high rise and vegetation). Unsupervised classification of images by geometry and material properties (surface orientation, albedo, solar irradiance, and shadow history) is facilitated by a detailed three-dimensional surface model (430 m × 430 m extent) and sensor view modelling. From detailed source area analysis, 9.5% of the area is observed by the cameras. Across all camera pixels, the 5th - 95th percentile Ts differences reach 37.5 K around midday. Roofs have the greatest diurnal Ts range (290.6 K → 329.0 K). Ts differences across sunlit sloped roofs reach 23.3 K. Walls of different cardinal orientations consistently differ by >10 K between 10:00 and 15:00. Shadow tracking within images is used to model cooling rates, where recently shaded (<30 min) ground can be 18.6 K warmer than equivalent unshaded Ts. West walls remain warm past sunset and are 1.2 K warmer than north walls at 23:00 (~4 h after sunset). Recently shaded walls cool exponentially to ambient Ts at a similar rate as the ground, but four times slower than roofs. The observed Ts characteristics are anticipated to have a wide range of applications (e.g. evaluation of urban surface energy balance models, ground-truthing of satellite thermal remote sensing).

High-Resolution Sampling of a Broad Marine Life Size Spectrum Reveals Differing Size- and Composition-Based Associations With Physical Oceanographic Structure

Observing multiple size classes of organisms, along with oceanographic properties and water mass origins, can improve our understanding of the drivers of aggregations, yet acquiring these measurements remains a fundamental challenge in biological oceanography. By deploying multiple biological sampling systems, from conventional bottle and net sampling to in situ imaging and acoustics, we describe the spatial patterns of different size classes of marine organisms (several microns to ∼10 cm) in relation to local and regional (m to km) physical oceanographic conditions on the Delaware continental shelf. The imaging and acoustic systems deployed included (in ascending order of target organism size) an imaging flow cytometer (CytoSense), a digital holographic imaging system (HOLOCAM), an In Situ Ichthyoplankton Imaging System (ISIIS, 2 cameras with different pixel resolutions), and multi-frequency acoustics (SIMRAD, 18 and 38 kHz). Spatial patterns generated by the different systems showed size-dependent aggregations and differing connections to horizontal and vertical salinity and temperature gradients that would not have been detected with traditional station-based sampling (∼9-km resolution). A direct comparison of the two ISIIS cameras showed composition and spatial patchiness changes that depended on the organism size, morphology, and camera pixel resolution. Large zooplankton near the surface, primarily composed of appendicularians and gelatinous organisms, tended to be more abundant offshore near the shelf break. This region was also associated with high phytoplankton biomass and higher overall organism abundances in the ISIIS, acoustics, and targeted net sampling. In contrast, the inshore region was dominated by hard-bodied zooplankton and had relatively low acoustic backscatter. The nets showed a community dominated by copepods, but they also showed high relative abundances of soft-bodied organisms in the offshore region where these organisms were quantified by the ISIIS. The HOLOCAM detected dense patches of ciliates that were too small to be captured in the nets or ISIIS imagery. This near-simultaneous deployment of different systems enables the description of the spatial patterns of different organism size classes, their spatial relation to potential prey and predators, and their association with specific oceanographic conditions. These datasets can also be used to evaluate the efficacy of sampling techniques, ultimately aiding in the design of efficient, hypothesis-driven sampling programs that incorporate these complementary technologies.

Precise Nanosizing with High Dynamic Range Holography.

Optical sensing is one of the key enablers of modern diagnostics. Especially label-free imaging modalities hold great promise as they eliminate labeling procedures prior to analysis. However, scattering signals of nanometric particles scale with their volume square. This unfavorable scaling makes it extremely difficult to quantitatively characterize intrinsically heterogeneous clinical samples, such as extracellular vesicles, as their signal variation easily exceeds the dynamic range of currently available cameras. Here, we introduce off-axis k-space holography that circumvents this limitation. By imaging the back-focal plane of our microscope, we project the scattering signal of all particles onto all camera pixels, thus dramatically boosting the achievable dynamic range to up to 110 dB. We validate our platform by detecting and quantitatively sizing metallic and dielectric particles over a 200 × 200 μm field of view and demonstrate that independently performed signal calibrations allow correctly sizing particles made from different materials. Finally, we present quantitative size distributions of extracellular vesicle samples.

Similarity analysis of pixelated CdTe semiconductor gamma camera image using a quadrant bar phantom for nuclear medicine: Monte Carlo simulation study

In the nuclear medicine imaging, quality control (QC) process using quadrant bar phantom is fundamental aspect of evaluating the spatial resolution. In addition, QC process of gamma camera is performed by daily or weekly. Recently, Monte Carlo simulation using the Geant4 application for tomographic emission (GATE) is widely applied in the pre-clinical nuclear medicine field for modeling gamma cameras with pixelated cadmium telluride (CdTe) semiconductor detector. In this study, we modeled a pixelated CdTe semiconductor detector and quadrant bar phantom (0.5, 1.0, 1.5, and 2.0 mm bar thicknesses) using the GATE tool. Similarity analysis based on correlation coefficients and peak signal-to-noise ratios was performed to compare image qualities for various source to collimator distances (0, 2, 4, 6, and 8 cm) and collimator lengths (0.2, 0.4, 0.6, 0.8, and 1.0 cm). To this end, we selected reference images based on collimator length and source to collimator distance settings. The results demonstrate that as the collimator length increases and the source to collimator distance decreases, the similarity to reference images improves. Therefore, our simulation results represent valuable information for the modeling of CdTe-based semiconductor gamma imaging systems and QC phantoms in the field of nuclear medicine.

Rapid and nondestructive evaluation of soluble solids content (SSC) and firmness in apple using Vis–NIR spatially resolved spectroscopy

Abstract Visible–near infrared (Vis–NIR) spectroscopy is a rapid and nondestructive method used to characterize organic compounds in postharvest fruit and vegetable assessment. However, developing robust calibration models is a challenge as conventional spectrometers collect only the cumulative effects of light absorption and scattering. In this study, a multifiber-based Vis–NIR spatially resolved spectra measurement system was designed for simultaneous evaluation of soluble solid content (SSC) and firmness in apple. Thirty silica fibers separated into five groups at 1, 2, 3, 4, and 5 mm away from the light illumination point and connected to a cost-effective Vis–NIR hyperspectral imaging camera were used to acquire spectral data with an improved signal-to-noise ratio (S/N) by a two-step signal averaging process (i.e., 30 camera pixels per fiber and six optical fibers per group). Reflectance ratio spectra were then calculated by dividing the diffusely reflected light intensity detected at distance d +△ by that detected at distance d to realize a light reference-free approach. Finally, the useful explanatory variables were selected by competitive adaptive reweighted sampling (CARS) to construct individual calibration models for various regions. The coefficients of determination ( R c a l 2 ) and the root mean square errors (RMSEcal) of the best-performing calibration models were approximately 0.97 and 0.20 % for SSC and 0.96 and 0.37 N for firmness, respectively. Furthermore, the predicted results were 0.92 and 0.35 % for SSC and 0.87 and 0.71 N for firmness. Our method offers low-cost and portable detection of SSC and firmness for postharvest fruit evaluation.

On-line Dense 3D Reconstruction Method based on ANN us-ing a Single Structured Light Pattern

Dense 3D reconstruction techniques to measuring dynamic scenes and deformable objects with little texture have been widely studied for various applications like surface deformation measurement. We assume that the actual situation of scanning is acquiring sequential shape and measuring surface deformation online at the moment when an object deforms under the action of aerodynamic forces. This paper presents such a technique for on-line acquisition of 3D surface data based on a one-shot scanning method that reconstructs 3D shape from a single image where a simple color-coded pattern using de Bruijn sequence is projected onto an object. The proposed approach has advantages that it requires no assumption of global smoothness and continuous surface. To realize 3D reconstruction from a single image, there are sever-al issues to be solved, for example, difficulty on decoding structured light pattern because of influence of chromatic aberration in both projection systems and imaging systems, and difficul-ty on establishing accurate correspondences between projector and camera pixels. This paper describes the solutions of the issues by combining two methods, that is (1) an efficient and robust pattern decoding method based on artificial neural networks (ANN), and (2) a sub-pixel matching method using phase map and multiple view geometry. Furthermore, we adopt some effective strategies to improve algorithm robustness. Practical experiments were carried out to test the accuracy and efficiency of the scanning system in typical configuration. The results show that the scanning system can reconstruct 3D shape on-line in high-resolution with the accuracy of 0.09 mm and efficiency of 10 fps.

Spectrally resolved polarizing interferometer for single-shot line profiling.

In this investigation, we describe a spectrally resolved polarizing interferometer to modify the typical spectrally resolved interferometer based on the Fourier method, which has a dead zone caused by the minimum measurable range. By use of a polarization pixelated CMOS camera, which enables us to obtain spatially phase-shifted spectral interferograms, the spectral phase can be extracted with a single image, and the proposed interferometer can obtain a line profile of a specimen at once. By the nature of phase-shifting technique, moreover, the directions of the measured distances can be identified, and the measuring range is extended to twice that of the typical spectrally resolved interferometer. In the experimental results, the measuring range was 104 µm, and the capability to obtain a line profile of a specimen with a single image was confirmed. For 3D surface profiling, lateral scanning was adopted, and the measurement results were compared with the result of the Fourier method.

Image Compression Based on Compressive Sensing: End-to-End Comparison With JPEG

We present an end-to-end image compression system based on compressive sensing. The presented system integrates the conventional scheme of compressive sampling and reconstruction with quantization and entropy coding. The compression performance, in terms of decoded image quality versus data rate, is shown to be comparable with JPEG and significantly better at the low rate range. We study the parameters that influence the system performance, including (i) the choice of sensing matrix, (ii) the trade-off between quantization and compression ratio, and (iii) the reconstruction algorithms. We propose an effective method to jointly control the quantization step and compression ratio in order to achieve near optimal quality at any given bit rate. Furthermore, our proposed image compression system can be directly used in the compressive sensing camera, e.g. the single pixel camera, to construct a hardware compressive sampling system.

Diverging cyclic radial shearing interferometry for single-shot wavefront sensing.

In this investigation, we describe a simple cyclic radial shearing interferometer for single-shot wavefront sensing. Instead of using the telescope lens system used in typical radial shearing interferometry, a single lens is used to generate two diverging radial shearing beams. This simple modification leads to the advantages of conveniently adjusting the radial shearing ratio, compactness of the system, and practical ease of alignment. With the aid of a polarization pixelated CMOS camera, the spatial phase-shifting technique is used to extract the phase with a single image. The most important feature is the fringe contrast enhancement by reducing the aberrations caused by the complicated optical system even though an incoherent light is used. The experimental results show the fringe contrast enhancement is at least 0.1 better than that of the conventional method, and the wavefronts are properly reconstructed with less than 0.071λ root-mean-squared wavefront error regardless of the coherence of the light.

Rapid thickness mapping of free-standing smectic films using colour information of reflected light

ABSTRACT A practical method to rapidly obtain the 2D thickness distribution in free-standing smectic films is developed based on the colour information of reflected image of the film. The red(R), green(G) and blue(B) colour coordinates from each pixel of a colour camera is normalised by the local intensity and is then quantitatively analysed in reference to the theoretical reflection spectra evaluated as a function of the film thickness. Results from the optical system employing a standard stereomicroscope can clearly discriminate a single-layer step over a wide range of thicknesses and provide thickness values with a precision better than 1 nm.

First analysis of the misaligned leading edges of ITER-like plasma facing units using a very high resolution infrared camera in WEST

Twelve ITER-like plasma-facing units made of tungsten were exposed in the WEST tokamak divertor, with three plasma-facing units (PFUs) significantly overexposed to plasma heat flux: one sharp-edged PFU (vertical misalignment h = 0.8 mm) and two chamfered PFUs (h =0.6 mm and 0.3 mm, respectively). This paper describes the first temperature analysis obtained with a very high spatial resolution infrared camera (pixel size ∼ 0.1 mm) on the misaligned PFU edges and shows the consistency obtained on the parallel heat flux derived from these measurements. The analysis is focused on the hottest areas of the PFU misaligned leading edges, since the temperature detection threshold of the VHR camera is high (Tthreshold,BB ≈ 370 °C). The heat flux parallel to the magnetic field lines is assessed by matching the toroidal temperature profile in the vicinity of the leading edge with 3D finite element modelling. The tungsten emissivity assumed in this study is 0.6, which is consistent with laboratory measurements for damaged PFUs. For the three PFUs studied (with different vertical misalignments, incident angles, geometries), the derived parallel heat flux is similar, and consistent with independent measurements by a Fiber Bragg grating embedded in a graphite PFU at another toroidal location, giving confidence in future experiments using the same settings.

Point-to-point compensation method for color coupling and imbalance in color-fringe pattern profilometry

Color-fringe pattern profilometry is widely developed for high-speed 3D shape measurement due to more information can be extracted through a single color pattern. However, previous researches show that the color coupling and imbalance of the system cause significant errors, which severely influence the accuracy of the color-fringe pattern profilometry. We proposed a point-to-point method to compensate the errors caused by color coupling and imbalance. Different from previous methods, the proposed method bases on the correspondence between camera pixels and projector pixels, which achieves point-to-point compensation of the full field. Experimental results verify that the proposed easy-to-operated method reduces the measurement errors significantly.

Polarization Navigation Simulation System and Skylight Compass Method Design Based upon Moment of Inertia

Unpolarized sunlight becomes polarized by atmospheric scattering and produces a skylight polarization pattern in the sky, which is detected for navigation by several species of insects. Inspired by these insects, a growing number of research studies have been conducted on how to effectively determine a heading angle from polarization patterns of skylight. However, few research studies have considered that the pixels of a pixelated polarization camera can be easily disturbed by noise and numerical values among adjacent pixels are discontinuous caused by crosstalk. So, this paper proposes a skylight compass method based upon the moment of inertia (MOI). Inspired by rigid body dynamics, the MOI of a rigid body with uniform mass distribution reaches the extreme values when the rigid body rotates on its symmetry axes. So, a whole polarization image is regarded as a rigid body. Then, orientation determination is transformed into solving the extreme value of MOI. This method makes full use of the polarization information of a whole polarization image and accordingly reduces the influence of the numerical discontinuity among adjacent pixels and measurement noise. In addition, this has been verified by numerical simulation and experiment. And the compass error of the MOI method is less than 0.44° for an actual polarization image.

Compact snapshot dual-mode interferometric system for on-machine measurement

To meet the need of on-machine metrology in optical manufacturing, a compact and snapshot dual-mode interferometric system is proposed for surface shape and roughness measurement. To simplify the measurement process between surface shape and roughness, a novel concept of using optical filters to separate the beam paths in the reference arm is introduced. A pixelated camera with a micro-polarizer array acquires four pi/2 phase-shifted interferograms simultaneously to minimize the environmental disturbance. Besides, the configuration-optimization-based subaperture stitching technique is introduced to extend the measurable aperture range. Both numerical analysis and experiments have been carried out to demonstrate the feasibility of the proposed compact snapshot dual-mode interferometer. The proposed system provides a powerful and portable tool to achieve on-machine surface characterization of various optical elements over a wide range of spatial frequencies and aperture sizes.