Array Camera Research Articles

Dynamic light field reconstruction via densely connected deep equilibrium model

High-resolution consumer plenoptic cameras usually feature low frame rates, making them not well-suited for capturing high-speed motion scenes. To compensate for this limitation, we extend the original snapshot compressive imaging system to plenoptic cameras and propose a densely connected deep equilibrium (DEQ) model for high-quality dynamic light field (LF) reconstruction, abbreviated as DLFDEQ. Specifically, we perform temporal compression encoding on a dynamic LF and model the reconstruction process as an inverse problem with an implicit regularization term. To solve this inverse problem, we present a densely connected DEQ model based on gradient descent. Our approach demonstrates stronger robustness and better detail retention than existing methods. We can practically quadruple the original camera’s frame rate by continually capturing and retrieving these measurement frames with high reconstruction accuracy.

Blind aberration correction for light field photography

Blind aberration correction for light field photography

Investigation of ultrasmall sidewall-insulated GaN via with large aspect ratio for the strategy of vertically stacked full-color Micro-LEDs

Micro light-emitting diode (Micro-LED) is considered as an ideal candidate for near-eye, outdoor display, and light field photography applications. At present, the commercialization of full-color Micro-LED is limited by the mass transfer of red, green, and blue (R-B-G) sub-pixels. Hence, we proposed a full-color scheme of vertically stacked tricolor Micro-LED layers to avoid mass transfer, which owns over 1000 PPI (pixel per inch). In this solution, the sidewall-insulated via in the epilayer plays a critical role to achieve the electrical and mechanical integration of three monochromatic Micro-LEDs with Si-based complementary metal-oxide semiconductor (CMOS) driver. Therefore, the via processes of GaN-based epilayer were investigated systematically using available semiconductor processes in this article. The inductively coupled plasma (ICP) etching was employed to create the ultra-small micro-structure array using SiO2 thin film as hard mask. Sidewall-insulated vias were fabricated with different aperture sizes (9.3, 7.5, and 3.4 μm) and a depth of about 4-μm. The vias with large aspect ratio are completely satisfy the requirement of designed vertical interconnection. This study aims to provide valuable reference for the commercial progress of high-resolution and full-color Micro-LEDs.

Rapid acquisition and surface defects recognition based on panoramic image of small-section hydraulic tunnel

Small-section hydraulic tunnels are characterized by small spaces and various section forms, under complex environments, which makes it difficult to carry out an inspection by the mobile acquisition equipment. To resolve these problems, an arbitrarily adjustable camera module deployment method and the corresponding automatic image acquisition equipment with multi-area array cameras are proposed and developed. Such method enables the acquisition of full-length surface images of the hydraulic tunnels with different cross-section forms and diameters by a one-way travel, and the overlap rate and accuracy of the acquired image sets meet the requirements of three-dimensional reconstruction and panoramic image generation. In addition, to improve the speed and accuracy of traditional algorithms for tunnel surface defects detection, this paper proposes an improved YOLOv5s-DECA model. The algorithm introduces DenseNet to optimize the backbone feature extraction network and incorporates an efficient channel attention ECA module to make a better extraction of features of defects. The experimental results show that mAP, and F1-Score of YOLOv5-DECA are 73.4% and 74.6%, respectively, which are better than the common model in terms of accuracy and robustness. The proposed YOLOv5-DECA has great detection performance for targets with variable shapes and can solve the problem of classification imbalance in surface defects. Then, by combining YOLOv5-DECA with the direction search algorithm, a “point-ring-section” method is established to allow rapid identification of common surface defects by detecting them layer by layer with the bottom image of the stitched panorama as the seed. The presented method in this paper effectively solves the problem that a single image fails to show the overall distribution of the defects and their accurate positioning in a whole large tunnel section and the effective features of defects in an excessively large panoramic image size are difficult to be captured by the neural network. Field applications demonstrated that the presented method is adequate for high-precision and intelligent surface defect detection and positioning for different small-section hydraulic tunnels such as circular, arch-wall, and box-shaped hydraulic tunnels.

Ptychographic wavefront cameras.

We introduce a snapshot wavefront camera capable of reconstructing the complex field of an object with near quantum-limited accuracy. This system incorporates a diffraction grating at the entrance aperture and an array of cameras positioned to align with the grating orders, capturing the field with varying impulse responses. The camera effectively reconstructs both the amplitude and phase of the object. Key performance factors include the aperture size, the number of apertures, and the overlap ratio between adjacent apertures. The minimal imaging and computational performance could be obtained using a 3 × 3 camera array configuration.

Three-dimensional reconstruction of a light field based on phase restoration for highly reflective surfaces

This paper proposes, a novel, to our knowledge, phase-restoration-based light field method to achieve 3D reconstruction of highly reflective surfaces. First, a focused light field camera whose angular and spatial resolutions can be adjusted according to the needs has been designed and fabricated to capture 4D light field information. Then, according to the pixel offsets between different sub-aperture images, a phase restoration method based on multi-view complementary information is proposed to restore the missing absolute phase information caused by highlights. Finally, a cubic B-spline curve method is used to directly fit the relationship between absolute phase and coordinates to achieve 3D reconstruction of highly reflective surfaces. The experimental results demonstrate that the proposed method effectively utilizes the multi-view information from the light field to restore missing absolute phase data in the phase unwrapping, ensuring accurate 3D reconstruction of highly reflective surfaces. What is more, our method requires no additional hardware, camera angle calibration, or point cloud fusion, which significantly reduces both hardware complexity and computational demands.

Super-resolution imaging quality enhancement method for distributed array infrared camera

Abstract To address issues related to low resolution and high noise in infrared cameras, a distributed array infrared camera imaging system utilizing four cameras is proposed. The four cameras are arranged in an unconstrained array, and the combination algorithm of Projections onto Convex Sets (POCS) and Real-Enhanced Super-Resolution Generative Adversarial Networks (Real-ESRGAN) is applied to achieve high-quality super-resolution infrared imaging. The wavelet fusion algorithm is used to preprocess four low-resolution infrared images to reduce noise. Then, the POCS algorithm is used to reconstruct the preprocessed image. Finally, the Real-ESRGAN is employed for image reconstruction, resulting in an ultra-high-resolution infrared image. The results show that compared to single infrared camera imaging, the resolution of images reconstructed using the distributed infrared camera array is increased by 0.58 times, with the Modulation Transfer Function (MTF) increased by 1.2 times. Additionally, the entropy is increased by 18.87%, the standard deviation is increased by 13.51%, and the Naturalness Image Quality Evaluator (NIQE) is reduced by 18.87%. This demonstrates a significant enhancement in the super-resolution imaging quality of the distributed infrared camera array.

Long-term observations of a sponge in situ reveal a rich repertoire of contractile behaviors, including winter dormancy

Sponges are remarkably active animals with a range of behaviors that are broadly termed sneezes. Little is known about what triggers sneezes or how sneezes propagate in an animal that lacks either muscular or nervous systems. We explored the triggers of sponge behavior in situ in the demosponge Suberites concinnus. The placement of this tennis ball-sized sponge at the focal point of a camera array on an instrument platform in Barkley Sound, BC, Canada, provided a unique opportunity for our investigation. The observatory has a suite of environmental sensors measuring parameters such as photosynthetically active radiation (PAR), oxygen, turbidity, salinity, pressure, current speed, and temperature. An array of 8 cameras designed for 3D imaging was used to capture hourly images. Our findings revealed both long-term (annual) and short-term (daily) behaviors in S. concinnus. The sponge contracted for 2 mo each winter. In February, it gradually expanded and became very active with short-term full-body contractions each lasting 11 h through the summer. A meta-analysis of contraction behaviors in other species of sponge showed that contraction duration is a factor of sponge size and that all sponges contract faster than they expand. Invertebrates and fish were frequently in contact with the sponge, but no single interaction was found to correlate with patterns of sponge behavior. However, significant correlations between sponge behaviors and chlorophyll and turbidity were found using generalized additive mixed models. Our study underscores the dynamic nature of these sessile filter feeders and their responses to their environment.

Fringe photometric stereo

In fringe projection profilometry (FPP), a high spatial frequency of fringe typically indicates powerful error suppression capability; however, it complicates phase unwrapping, necessitating a greater number of patterns and thus compromising the real-time performance of scanning. In this letter, we report a fringe photometric stereo (FPS) to completely bypass phase unwrapping for recovering the continuous 3D surface. We reveal the linear mapping relationship between the phase gradient and depth gradient, thereby facilitating the computation of depth gradients from phase gradients. Thus, we can employ depth gradients to reconstruct high-quality 3D profiles. Experimental results demonstrate that our FPS surpasses traditional phase-shifting profilometry in mitigating Gaussian noise. Our approach enables high-quality and unambiguous 3D reconstruction using single-frequency fringe patterns, relying solely on a camera and a projector without the need for dual, multiple, or light field cameras, thereby paving a path to high-speed and precision 3D imaging.

Light field images super-resolution method based on hybrid low-dimensional spatial–angular interaction feature and linear complementation epipolar feature

Light-field (LF) cameras record 4D information about the intensity and angle of light, but limited by the resolution of the sensor, LF images require a trade-off between spatial and angular resolution, which results in generally low spatial resolution for LF images. We work on spatial super-resolution reconstruction (SR) of LF images, but due to the high dimensionality of 4D LF, it is difficult to process it directly, so we decompose 4D LF into low-dimensional sub-spaces. Since the 4D features of interest to different low-dimensional sub-spaces are different, we design a hybrid network structure—HSAESR, which processes multiple low-dimensional sub-spaces individually to adequately model the texture information in the 4D LF. HSAESR is designed to reconstruct Macro Pixel Images (MacPI) and Extremely Planar Images (EPI) respectively, and the final reconstruction results are weighted and fused. For MacPI, a CNN-based feature interaction module is designed using a novel asymptotic ’grid’ interaction approach to fully utilize the correlation between the 2D spatial dimension and the 2D angular dimension sub-spaces in 4D LF. For EPI, a Transformer-based epipolar feature linear complementation module is designed learning global and non-global multi-directional epipolar feature correlations, which learns feature linear correlation factor (α) and linear bias weight (β) through the correlation of non-global epipolar features, and corrects and complements the global epipolar correlation features through feature linear complementation. Extensive experiments are conducted on five datasets, the results show that HSAESR further improves the performance of LF images spatial SR, the reconstruction results are better than the comparison methods.

Noise sensitivity analysis of focal scanning light field imaging.

Light field imaging can simultaneously record spatial and angular information of light signals to provide various computational imaging functions. However, traditional microlens array-based light field cameras usually suffer from a trade-off between spatial and angular resolutions. In contrast, focal scanning light field imaging (FSLFI) can digitally modulate an incident light field through an image stack captured at different focal planes and then utilize the transport-of-intensity property to computationally recover the full-resolution light field. This paper presents a unified light field reconstruction algorithm framework, which involves different types of algorithms, such as back-projection reconstruction and additive/multiplicative iterative reconstruction, for FSLFI. Based on the unified algorithm framework, we systematically analyze and investigate the FSLFI performance on noise sensitivity. Light fields are reconstructed at different noise levels to quantitatively analyze the FSLFI performances with different types of algorithms. Both simulation and actual experimental results demonstrate that the noise sensitivity and reconstruction accuracy are constrained by each other for FSLFI. Back-projection reconstruction is appropriate in high-efficiency light field reconstruction, while additive/multiplicative iterative reconstruction is suitable for high-accuracy light field imaging at high/low noise levels. These conclusions can apply to any FSLFI method covered by the unified algorithm framework, in which appropriate algorithms can be selected for high-quality light field imaging and measurement according to specific application scenarios.

Binocular Scheimpflug light-field PIV

This work presents theoretical and experimental studies on a binocular Scheimpflug light-field based PIV. To investigate the effects of key parameters such as magnification, camera separation angle and tilt angle, a simulation study and quantitative comparison between conventional and binocular Scheimpflug light-field PIV have been performed. To validate the performance of binocular Scheimpflug light-field PIV, a series of vortex ring experiments have been carried out with different parameters. A comparison of the reconstructed volume and quality of the resolved vortex ring structures shows the advantages of binocular Scheimpflug light-field PIV. With Scheimpflug imaging, the focal plane of two light-field cameras can be fully overlapped which results in an extended reconstructed particle volume and less distortion in the resolved flow field than conventional binocular light-field PIV.

Real-Time Table Tennis Scoring and Player Position Tracking System Using Computer Vision

This study focuses on developing a system that uses state-of-the-art computer vision techniques to improve the accuracy and efficiency of table tennis tracking. The table tennis ball's trajectory and movement are tracked automatically by the system using a mobile camera array, allowing for real-time score updates without the need for human participation. This method guarantees precise and seamless scoring, freeing players from worrying about maintaining score and allowing them to concentrate fully on the game. The system uses player pose estimation technologies in addition to real-time scoring to track and record player positions and motions during the match. The system may provide a thorough study of player motions, including tracking posture and finding important motion patterns, by gathering and analyzing skeletal data.

Design of a Novel Microlens Array and Imaging System for Light Fields.

Light field cameras are unsuitable for further acquisition of high-quality images due to their small depth of field, insufficient spatial resolution, and poor imaging quality. To address these issues, we proposed a novel four-focal-square microlens and light field system. A square aspheric microlens array with four orthogonal focal lengths was designed, in which the aperture of a single lens was 100 μm. The square arrangement improves pixel utilization, the four focal lengths increase the depth of field, and the aspheric improves image quality. The simulations demonstrate pixel utilization rates exceeding 90%, depth-of-field ranges 6.57 times that of a single focal length, and image quality is significantly improved. We have provided a potential solution for improving the depth of field and image quality of the light field imaging system.

Optimizing Additive Manufacturable Structures with Computer Vision to Enhance Material Efficiency and Structural Stability

This study introduces an innovative technique that merges computer vision with topology optimization to advance additive manufacturing. Employing advanced photogrammetry software, we obtain high-resolution images of the design domain, which are then used to develop accurate 3D models through meticulous scanning procedures. These models are transformed into an STL file format and remeshed using an adaptive algorithm within COMSOL 5.3 Multiphysics, facilitated by a custom MATLAB 2023 application. This integration achieves the optimal mesh resolution and precision in analytical assessments. We applied this technique to the design of a concrete pillar for 3D printing, targeting a 75% reduction in volume to improve the material efficiency and structural stability—critical factors for extraterrestrial applications. The design, captured with a 360-degree camera array, guided the MATLAB-based topology optimization process. By combining MATLAB’s optimization algorithms with COMSOL’s meshing and finite element analysis tools, we investigated various material-efficient configurations. The findings reveal a substantial volume reduction, especially in the central region of the design, effectively optimizing material utilization while preserving structural integrity. The optimization algorithm exhibited a swift and stable convergence, reaching near-optimal solutions within approximately 20 iterations.

Aberration-robust monocular passive depth sensing using a meta-imaging camera

Depth sensing plays a crucial role in various applications, including robotics, augmented reality, and autonomous driving. Monocular passive depth sensing techniques have come into their own for the cost-effectiveness and compact design, offering an alternative to the expensive and bulky active depth sensors and stereo vision systems. While the light-field camera can address the defocus ambiguity inherent in 2D cameras and achieve unambiguous depth perception, it compromises the spatial resolution and usually struggles with the effect of optical aberration. In contrast, our previously proposed meta-imaging sensor1 has overcome such hurdles by reconciling the spatial-angular resolution trade-off and achieving the multi-site aberration correction for high-resolution imaging. Here, we present a compact meta-imaging camera and an analytical framework for the quantification of monocular depth sensing precision by calculating the Cramér–Rao lower bound of depth estimation. Quantitative evaluations reveal that the meta-imaging camera exhibits not only higher precision over a broader depth range than the light-field camera but also superior robustness against changes in signal-background ratio. Moreover, both the simulation and experimental results demonstrate that the meta-imaging camera maintains the capability of providing precise depth information even in the presence of aberrations. Showing the promising compatibility with other point-spread-function engineering methods, we anticipate that the meta-imaging camera may facilitate the advancement of monocular passive depth sensing in various applications.

Enhancing Facial Expression Recognition through Light Field Cameras.

In this paper, we study facial expression recognition (FER) using three modalities obtained from a light field camera: sub-aperture (SA), depth map, and all-in-focus (AiF) images. Our objective is to construct a more comprehensive and effective FER system by investigating multimodal fusion strategies. For this purpose, we employ EfficientNetV2-S, pre-trained on AffectNet, as our primary convolutional neural network. This model, combined with a BiGRU, is used to process SA images. We evaluate various fusion techniques at both decision and feature levels to assess their effectiveness in enhancing FER accuracy. Our findings show that the model using SA images surpasses state-of-the-art performance, achieving 88.13% ± 7.42% accuracy under the subject-specific evaluation protocol and 91.88% ± 3.25% under the subject-independent evaluation protocol. These results highlight our model's potential in enhancing FER accuracy and robustness, outperforming existing methods. Furthermore, our multimodal fusion approach, integrating SA, AiF, and depth images, demonstrates substantial improvements over unimodal models. The decision-level fusion strategy, particularly using average weights, proved most effective, achieving 90.13% ± 4.95% accuracy under the subject-specific evaluation protocol and 93.33% ± 4.92% under the subject-independent evaluation protocol. This approach leverages the complementary strengths of each modality, resulting in a more comprehensive and accurate FER system.

Realization of a gas puff imaging system on the Wendelstein 7-X stellarator.

A system for studying the spatiotemporal dynamics of fluctuations in the boundary of the W7-X plasma using the "Gas-Puff Imaging" (GPI) technique has been designed, constructed, installed, and operated. This GPI system addresses a number of challenges specific to long-pulse superconducting devices, such as W7-X, including the long distance between the plasma and the vacuum vessel wall, the long distance between the plasma and diagnostic ports, the range of last closed flux surface (LCFS) locations for different magnetic configurations in W7-X, and management of heat loads on the system's plasma-facing components. The system features a pair of "converging-diverging" nozzles for partially collimating the gas puffed locally ≈135mm radially outboard of the plasma boundary, a pop-up turning mirror for viewing the gas puff emission from the side (which also acts as a shutter for the re-entrant vacuum window), and a high-throughput optical system that collects visible emission resulting from the interaction between the puffed gas and the plasma and directs it along a water-cooled re-entrant tube directly onto the 8 × 16 pixel detector array of the fast camera. The DEGAS 2 neutral code was used to simulate the Hα (656nm) and HeI (587nm) line emission expected from well-characterized gas-puffs of H2 and He and excited within typical edge plasma profiles in W7-X, thereby predicting line brightnesses used to reduce the risks associated with system sensitivity and placement of the field of view. Operation of GPI on W7-X shows excellent signal-to-noise ratios (>100 at 2 Mframes/s) over the field of view for minimally perturbing gas puffs. The GPI system provides detailed measurements of the two-dimensional (radial and poloidal) dynamics of plasma fluctuations in the W7-X edge and scrape-off layer and in and around the magnetic islands outside the LCFS that make up the island divertor configuration employed on W7-X.

The progenitor star of SN 2023ixf: a massive red supergiant with enhanced, episodic pre-supernova mass loss

ABSTRACT We identify the progenitor star of SN 2023ixf in Messier 101 using Keck/NIRC2 adaptive optics imaging and pre-explosion Hubble Space Telescope (HST)/Advanced Camera for Surveys (ACS) images. The supernova, localized with diffraction spikes and high-precision astrometry, unambiguously coincides with a progenitor candidate of $m_\text{F814W}=24.87\pm 0.05$ (AB). Given its reported infrared excess and semiregular variability, we fit a time-dependent spectral energy distribution (SED) model of a dusty red supergiant (RSG) to a combined data set of HST optical, ground-based near-infrared, and Spitzer Infrared Array Camera (IRAC) [3.6], [4.5] photometry. The progenitor resembles an RSG of $T_\text{eff}=3488\pm 39$ K and $\log (L/\mathrm{L}_\odot)=5.15\pm 0.02$, with a $0.13\pm 0.01$ dex ($31.1\pm 1.7$ per cent) luminosity variation at a period of $P=1144.7\pm 4.8$ d, obscured by a dusty envelope of $\tau =2.92\pm 0.02$ at $1\, \mu \text{m}$ in optical depth (or $A_\text{V}=8.43\pm 0.11$ mag). The signatures match a post-main-sequence star of $18.2_{-0.6}^{+1.3}\, \mathrm{M}_\odot$ in zero-age main-sequence mass, among the most massive SN II progenitor, with a pulsation-enhanced mass-loss rate of $\dot{M}=(4.32\pm 0.26)\times 10^{-4} \, \mathrm{M}_\odot \, \text{yr}^{-1}$. The dense and confined circumstellar material is ejected during the last episode of radial pulsation before the explosion. Notably, we find strong evidence for variations of $\tau$ or $T_\text{eff}$ along with luminosity, a necessary assumption to reproduce the wavelength-dependent variability, which implies periodic dust sublimation and condensation. Given the observed SED, partial dust obscuration remains possible, but any unobstructed binary companion over $5.6\, \mathrm{ M}_\odot$ can be ruled out.

A Study on the 3D Reconstruction Strategy of a Sheep Body Based on a Kinect v2 Depth Camera Array.

Non-contact measurement based on the 3D reconstruction of sheep bodies can alleviate the stress response in sheep during manual measurement of body dimensions. However, data collection is easily affected by environmental factors and noise, which is not conducive to practical production needs. To address this issue, this study proposes a non-contact data acquisition system and a 3D point cloud reconstruction method for sheep bodies. The collected sheep body data can provide reference data for sheep breeding and fattening. The acquisition system consists of a Kinect v2 depth camera group, a sheep passage, and a restraining pen, synchronously collecting data from three perspectives. The 3D point cloud reconstruction method for sheep bodies is implemented based on C++ language and the Point Cloud Library (PCL). It processes noise through pass-through filtering, statistical filtering, and random sample consensus (RANSAC). A conditional voxel filtering box is proposed to downsample and simplify the point cloud data. Combined with the RANSAC and Iterative Closest Point (ICP) algorithms, coarse and fine registration are performed to improve registration accuracy and robustness, achieving 3D reconstruction of sheep bodies. In the base, 135 sets of point cloud data were collected from 20 sheep. After 3D reconstruction, the reconstruction error of body length compared to the actual values was 0.79%, indicating that this method can provide reliable reference data for 3D point cloud reconstruction research of sheep bodies.