Array Camera Research Articles

A novel audio-visual based beamformer using a 360° camera and planar microphone array system

Beamforming systems that rely only on microphone arrays may suffer from poor performance in complex acoustic environments with multiple sources and room reverberation. To overcome this limitation, we propose a novel method that combines a planar microphone array and a 360° camera to steer the beamformer. The camera provides visual information about the location and orientation of sound sources, which can be used to adjust the beamformer parameters. We will present results that compare the signal-to-noise ratio (SNR) of the proposed method with a conventional beamforming microphone-based method in a simulated room with different source positions, orientations, and noise levels to investigate the benefit of visual information for beamforming in different environments. The proposed method has the potential to improve the performance of beamforming systems in applications such as speech enhancement, speech recognition, and speaker localization.

Sky Brightness Evaluation and First Coronal Signal Detection from Concordia Base (Antarctica) with a Calibrated Micropolarizer Array Camera

Sky Brightness Evaluation and First Coronal Signal Detection from Concordia Base (Antarctica) with a Calibrated Micropolarizer Array Camera

Morphological Decomposition of the Early-type Spiral Galaxy M81 (NGC 3031)

This paper presents a morphological investigation of the early-type spiral galaxy M81 (NGC 3031) through decomposition by fitting radial profiles of surface brightness using the software GALFIT, aimed at exploring structural components of M81 and quantifying their morphology. In this work, we adopt 6 types of decompositions with different numbers of morphological components, among which the most complicated one contains 5 components such as a bulge, a disk, a pair of outer spiral arms, a pair of inner spiral arms, and a galactic nucleus. The results show that, M81 hosts a classical bulge with the Sérsic index ∼ 5.0; the morphology and the luminosity for the bulge are almost constant in the different decomposition types. The disk of M81 has the Sérsic index ∼ 1.2 but the morphology and the luminosity are found sensitive to the inclusion of the inner spiral arms in the decomposition or not. The results of this work indicate that the combination of individual substructures has a considerable impact on the morphology of the galaxy as a mixture. On the basis of the results, the usability of the different types of decomposition is suggested in this work. The three-component decomposition, i.e., bulge + disk + nucleus, is applicable to statistics of large samples of galaxies; more complicated decomposition with spiral arms taken into account is suitable for precise measurements of individual galaxies in small samples. This work is based on the single-band Spitzer-The Infrared Array Camera (IRAC) 4.5 μm image. In the future, we will carry out multi-wavelength decomposition, for the purpose of investigating spectral energy distributions and stellar population properties for the galactic substructures, and thereby deduce their formation history and evolution processes.

Digital Classification of Chilean Pelagic Species in Fishing Landing Lines.

Fishing landings in Chile are inspected to control fisheries that are subject to catch quotas. The control process is not easy since the volumes extracted are large and the numbers of landings and artisan shipowners are high. Moreover, the number of inspectors is limited, and a non-automated method is utilized that normally requires months of training. In this work, we propose, design, and implement an automated fish landing control system. The system consists of a custom gate with a camera array and controlled illumination that performs automatic video acquisition once the fish landing starts. The imagery is sent to the cloud in real time and processed by a custom-designed detection algorithm based on deep convolutional networks. The detection algorithm identifies and classifies different pelagic species in real time, and it has been tuned to identify the specific species found in landings of two fishing industries in the Biobío region in Chile. A web-based industrial software was also developed to display a list of fish detections, record relevant statistical summaries, and create landing reports in a user interface. All the records are stored in the cloud for future analyses and possible Chilean government audits. The system can automatically, remotely, and continuously identify and classify the following species: anchovy, jack mackerel, jumbo squid, mackerel, sardine, and snoek, considerably outperforming the current manual procedure.

Specular highlight removal based on dichromatic reflection model and priority-based adaptive direction with light field camera

In this paper, an image restoration method with light field camera is proposed to effectively remove specular highlight, and it only needs single shot and has no special requirements for light source and environment. The x-means clustering method based on dichromatic reflection model is proposed to recover unsaturated highlight pixels, and a priority-based adaptive direction method is designed to eliminate saturated specular highlight pixels. After that, a method based on information entropy and mean gradient ratio, called EMGR, is established to quantitatively evaluate the image quality of specular highlight removal. The experimental results showed that our method can effectively utilized the multi-view advantage of the light field to remove the specular highlight of images in different scenes while completely preserving the images texture information, and has the advantages of low complexity and wide applicability.

Long-range Fourier ptychographic imaging of the dynamic object with a single camera.

Fourier ptychographic imaging technology is a new imaging method proposed in recent years. This technology captures multiple low-resolution images, and synthesizes them into a high-resolution image in the Fourier domain by a phase retrieval algorithm, breaking through the diffraction limit of the lens. In the field of macroscopic Fourier ptychographic imaging, most of the existing research generally focus on high-resolution imaging of static objects, and applying Fourier ptychographic imaging technology to dynamic objects is a hot research area now. At present, most of the researches are to use camera arrays combined with multiplexed lighting, deep learning or other algorithms, but the implementation of these methods is complicated or costly. Based on the diffraction theory of Fourier optics, this paper proposes that by expanding and focusing the illumination area, we can apply Fourier ptychographic imaging technology with a single camera to moving objects within a certain range. Theoretical analysis and experiments prove the feasibility of the proposed method. We successfully achieve high-resolution imaging of the dynamic object, increasing the resolution by about 2.5 times. This paper also researches the impact of speckles in the illuminated area on imaging results and proposes a processing method to reduce the impact of speckles.

Per-pixel distortion correction method of light field camera with homography matrix and phase calculation

A light field camera is constructed by positioning a microlens array between the main lens and an imaging plane, allowing it to record depth information. One of the factors affecting the reconstruction accuracy is distortion. To address this, we propose a novel method for correcting distortion in light field cameras using a homography matrix and phase calculation of the fringe projection technique. In this method, the distortion characteristic of each pixel in light field cameras can be directly measured. Firstly, the phase values of the concentric circular target captured by the light field camera are acquired using phase calculation technique. Next, the homography matrix is obtained by extracting the center coordinates of the target and their corresponding phase values. The phase value of each ideal point is obtained using the homography matrix, whereas the phase value of the corresponding real point is obtained through phase calculation technique. Finally, the distortion mapping table is generated based on the per-pixel ideal and real point pairs. By comparing the full-field error of the images before and after correction, the results show an average improvement of 15.992 % using this method.

Learning based compressive snapshot spectral light field imaging with RGB sensors.

The application of multidimensional optical sensing technologies, such as the spectral light field (SLF) imager, has become increasingly common in recent years. The SLF sensors provide information in the form of one-dimensional spectral data, two-dimensional spatial data, and two-dimensional angular measurements. Spatial-spectral and angular data are essential in a variety of fields, from computer vision to microscopy. Beam-splitters or expensive camera arrays are required for the usage of SLF sensors. The paper describes a low-cost RGB light field camera-based compressed snapshot SLF imaging method. Inspired by the compressive sensing paradigm, the four dimensional SLF can be reconstructed from a measurement of an RGB light field camera via a network which is proposed by utilizing a U-shaped neural network with multi-head self-attention and unparameterized Fourier transform modules. This method is capable of gathering images with a spectral resolution of 10 nm, angular resolution of 9 × 9, and spatial resolution of 622 × 432 within the spectral range of 400 to 700 nm. It provides us an alternative approach to implement the low cost SLF imaging.

Effectiveness of a vision-based handle trajectory monitoring system in studying robotic suture operation.

Data on surgical robots are not openly accessible, limiting further study of the operation trajectory of surgeons' hands. Therefore, a trajectory monitoring system should be developed to examine objective indicators reflecting the characteristic parameters of operations. 20 robotic experts and 20 first-year residents without robotic experience were included in this study. A dry-lab suture task was used to acquire relevant hand performance data. Novices completed training on the simulator and then performed the task, while the expert team completed the task after warm-up. Stitching errors were measured using a visual recognition method. Videos of operations were obtained using the camera array mounted on the robot, and the hand trajectory of the surgeons was reconstructed. The stitching accuracy, robotic control parameters, balanceanddexterityparameters, and operation efficiency parameters were compared. Experts had smaller center distance (p < 0.001) and larger proximal distance between the hands (p < 0.001) compared with novices. The path and volume ratios between the left and right hands of novices were larger than those of experts (both p < 0.001) and the total volume of the operation range of experts was smaller (p < 0.001). The surgeon trajectory optical monitoring system is an effective and non-subjective method to distinguish skill differences. This demonstrates the potential of pan-platform use to evaluate task completion and help surgeons improve their robotic learning curve.

3D Hyperspectral Light-Field Imaging: a first intraoperative implementation

Abstract Hyperspectral imaging is an emerging technology that has gained significant attention in the medical field due to its ability to provide precise and accurate imaging of biological tissues. The current methods of hyperspectral imaging, such as filter-wheel, snapshot, line-scanning, and push-broom cameras have limitations such as low spatial and spectral resolution, slow acquisition time. New developments on the field of light field cameras show the potential to overcome these limitations. In this paper, we use a novel hyperspectral lightfield camera and try to combine the capability of hyperspectral and 3D analysis. For this purpose we calibrate our system and test it during two ENT-surgeries to show its potential for improving surgical outcomes. The micro-lenses of the camera map 66 spectral sub-images onto the sensor allowing to reconstruct the spectral behavior of the captured scene in the spectral range of 350-1000nm. In addition, we use the sensor data to apply a 3D camera calibration pipeline to allow 3D surface reconstruction. We captured 26 calibration images and achieved calibration results in accordance to stated company data. The best calibration showed a re-projection error of 0.55 px. Further, we tested the camera during a parotidectomy and a neck-dissection. The extracted reflectance spectra of the selected venal and arterial regions correspond perfectly to the spectrum of oxygenated and deoxygenated hemoglobin. For the first time, up to our knowledge, a hyperspectral lightfield camera has been used during a surgery. We were able to continuously capture images and analyze the reconstructed spectra of specific tissue types. Further, we are able to use the sensor data of the micro-lens projections to calibrate the multilens camera system for later intraoperative measurement tasks.

Experimental observation on water entry of a sphere in regular wave

This paper presents a novel experiment to observe the whole water entry process of a free-falling sphere into a regular wave. A time-accurate synchronizing system modulates the moment elaborately to ensure the sphere impacting onto the water surface at the desirable wave phase. Four high-speed cameras focus locally to measure the high-precision size of the cavity evolution. Meanwhile, the aggregated field view of the camera array covers both the splash above the free surface and the entire cavity in the wave. The detailed methodologies are described and verified for the hardware set-up and the image post-processing. The theoretical maximum deviation is 1.7% on the space scale. The integral morphology of the cavity is captured precisely in the coordinate system during the sphere penetrates through the water at four representative wave phases and the still water. The result shows that the horizontal velocity of the fluid particle in the wave impels the cavity and changes the shape distinctly. Notably, the wave motion causes the cavity to pinch off earlier at the wave trough phase and later at the wave crest phase than in the still water. The wave motion influences the falling process of the sphere slightly in the present parameters.

High-resolution multi-spectral snapshot 3D imaging with a SPAD array camera.

Currently, mainstream light detection and ranging (LiDAR) systems usually involve a mechanical scanner component, which enables large-scale, high-resolution and multi-spectral imaging, but is difficult to assemble and has a larger system size. Furthermore, the mechanical wear on the moving parts of the scanner reduces its usage lifetime. Here, we propose a high-resolution scan-less multi-spectral three-dimensional (3D) imaging system, which improves the resolution with a four-times increase in the pixel number and can achieve multi-spectral imaging in a single snapshot. This system utilizes a specially designed multiple field-of-view (multi-FOV) system to separate four-wavelength echoes carrying depth and spectral reflectance information with predetermined temporal intervals, such that one single pixel of the SPAD array can sample four adjacent positions through the four channels' FOVs with subpixel offset. The positions and reflectivity are thus mapped to wavelengths in different time-bins. Our results show that the system can achieve high-resolution multi-spectral 3D imaging in a single exposure without scanning component. This scheme is the first to realize scan-less single-exposure high-resolution and multi-spectral imaging with a SPAD array sensor.

FPattNet: A Multi-Scale Feature Fusion Network with Occlusion Awareness for Depth Estimation of Light Field Images.

A light field camera can capture light information from various directions within a scene, allowing for the reconstruction of the scene. The light field image inherently contains the depth information of the scene, and depth estimations of light field images have become a popular research topic. This paper proposes a depth estimation network of light field images with occlusion awareness. Since light field images contain many views from different viewpoints, identifying the combinations that contribute the most to the depth estimation of the center view is critical to improving the depth estimation accuracy. Current methods typically rely on a fixed set of views, such as vertical, horizontal, and diagonal, which may not be optimal for all scenes. To address this limitation, we propose a novel approach that considers all available views during depth estimation while leveraging an attention mechanism to assign weights to each view dynamically. By inputting all views into the network and employing the attention mechanism, we enable the model to adaptively determine the most informative views for each scene, thus achieving more accurate depth estimation. Furthermore, we introduce a multi-scale feature fusion strategy that amalgamates contextual information and expands the receptive field to enhance the network's performance in handling challenging scenarios, such as textureless and occluded regions.

Deep focus light-field camera for handheld 3D intraoral scanning using crosstalk-free solid immersion microlens arrays.

3D in vivo imaging techniques facilitate disease tracking and treatment, but bulky configurations and motion artifacts limit practical clinical applications. Compact light-field cameras with microlens arrays offer a feasible option for rapid volumetric imaging, yet their utilization in clinical practice necessitates an increased depth-of-field for handheld operation. Here, we report deep focus light-field camera (DF-LFC) with crosstalk-free solid immersion microlens arrays (siMLAs), allowing large depth-of-field and high-resolution imaging for handheld 3D intraoral scanning. The siMLAs consist of thin PDMS-coated microlens arrays and a metal-insulator-metal absorber to extend the focal length with low optical crosstalk and specular reflection. The experimental results show that the immersion of MLAs in PDMS increases the focal length by a factor of 2.7 and the transmittance by 5.6%-27%. Unlike conventional MLAs, the siMLAs exhibit exceptionally high f-numbers up to f/6, resulting in a large depth-of-field for light-field imaging. The siMLAs were fully integrated into an intraoral scanner to reconstruct a 3D dental phantom with a distance measurement error of 82 ± 41 μm during handheld operation. The DF-LFC offers a new direction not only for digital dental impressions with high accuracy, simplified workflow, reduced waste, and digital compatibility but also for assorted clinical endoscopy and microscopy.

Testing 2D temperature models in Bayesian retrievals of atmospheric properties from hot Jupiter phase curves

ABSTRACT Spectroscopic phase curves of transiting hot Jupiters are spectral measurements at multiple orbital phases, giving a set of disc-averaged spectra that probe multiple hemispheres. By fitting model phase curves to observations, we can constrain the atmospheric properties of hot Jupiters, such as molecular abundance, aerosol distribution, and thermal structure, which offer insights into their atmospheric dynamics, chemistry, and formation. We propose a novel 2D temperature parametrization consisting of a dayside and a nightside to retrieve information from near-infrared phase curves and apply the method to phase curves of WASP-43b observed by HST/Wide Field Camera 3 and Spitzer/Infra-Red Array Camera. In our scheme, the temperature is constant on isobars on the nightside and varies with cosn(longitude/ϵ) on isobars on the dayside, where n and ϵ are free parameters. We fit all orbital phases simultaneously using the radiative transfer package nemesispy coupled to a Bayesian inference code. We first validate the performance of our retrieval scheme with synthetic phase curves generated from a Global Circulation Model and find that our 2D scheme can accurately retrieve the latitudinally averaged thermal structure and constrain the abundance of H2O and CH4. We then apply our 2D scheme to the observed phase curves of WASP-43b and find: (1) The dayside temperature–pressure profiles do not vary strongly with longitude and are non-inverted. (2) The retrieved nightside temperatures are extremely low, suggesting significant nightside cloud coverage. (3) The H2O volume mixing ratio is constrained to 5.6 × 10−5–4.0 × 10−4, and we retrieve an upper bound for CH4 mixing ratio at ∼10−6.

Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging.

We report a microlens array camera with variable apertures (MACVA) for high dynamic range (HDR) imaging by using microlens arrays with various sizes of apertures. The MACVA comprises variable apertures, microlens arrays, gap spacers, and a CMOS image sensor. The microlenses with variable apertures capture low dynamic range (LDR) images with different f-stops under single-shot exposure. The reconstructed HDR images clearly exhibit expanded dynamic ranges surpassing LDR images as well as high resolution without motion artifacts, comparable to the maximum MTF50 value observed among the LDR images. This compact camera provides, what we believe to be, a new perspective for various machine vision or mobile devices applications.

Consistent foraging on marine resources by coyotes (Canis latrans) on the Southern California coast

The marine-terrestrial interface provides consumers with access to nutrient rich ocean-based resource subsidies. Coyotes, omnivorous generalists with large ranges that can span many landscape types including the coastal ecotone, might be a critical component of the intertidal food web, contributing to resource subsidy movement into the less productive adjacent coastal scrub habitat. This study utilized an array of camera traps along identified coastal game trails in two protected coastal properties in California to assess the consumption of marine fauna by coyotes in intertidal habitats. We captured coyotes consuming a variety of marine resources, including marine mammals, fish, and intertidal invertebrates. We therefore suggest that coyotes are a more consistent consumer in intertidal habitats than previously thought. We highlight the need for further research into the role of coyotes as a marine to terrestrial subsidy vector, especially under the pressure of anthropogenic change in coastal systems.

Blur aware metric depth estimation with multi-focus plenoptic cameras

While a traditional camera only captures one point of view of a scene, a plenoptic or light-field camera, is able to capture spatial and angular information in a single snapshot, enabling depth estimation from a single acquisition. In this paper, we present a new metric depth estimation algorithm using only raw images from a multi-focus plenoptic camera. The proposed approach is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. The main goal of our blur aware depth estimation (BLADE) approach is to improve disparity estimation for defocus stereo images by integrating both correspondence and defocus cues. We thus leverage blur information where it was previously considered a drawback. We explicitly derive an inverse projection model including the defocus blur providing depth estimates up to a scale factor. A method to calibrate the inverse model is then proposed. We thus take into account depth scaling to achieve precise and accurate metric depth estimates. Our results show that introducing defocus cues improves the depth estimation. We demonstrate the effectiveness of our framework and depth scaling calibration on relative depth estimation setups and on real-world 3D complex scenes with ground truth acquired with a 3D lidar scanner.

Wide-baseline view synthesis for light-field display based on plane-depth-fused sweep volume

The advancements in three-dimensional (3D) display technology have led to a wide interest in light-field display. However, the need to simultaneously capture a large number of object views made content generation for light-field displays still a bottleneck. In this paper, we propose a method for light-field content generation based on plane-depth-fused sweep volume (PDFSV), focusing on handling wide-baseline views and exhibiting scene generalization when the camera array remains unchanged. Specifically, the proposed PDFSV exploits the prior depth of the images captured by a 4 × 4 spherical camera array to represent 3D information of scenes. Then two optimized sequential convolutional neural networks (CNN) are employed for implicit depth modeling and final color calculation, respectively. By doing these, the prior depth facilitates the synthesis of regions with complex textures in the target view. We produce a Wide-baseline Multi-view Image Set (WMIS) which has a field of view (FOV) angle reaching 54°and could be publicly available for access. In our experiments, we use only the 4 vertex views as input. Results demonstrate that the proposed approach can synthesize high-quality views at arbitrary positions between sparse views, outperforming existing neural-radiance-fields-based (NeRF-based) methods. Finally, we conduct autostereoscopic display experiments, achieving satisfactory results.

Progressive Moire Removal and Texture Complementation for Image Demoireing

Taking photos of digital screens often produces color-distorting moire patterns caused by inconsistency between the color filter array of cameras and the sub-pixel layout of screens, which severely degrades the quality of photos. Most existing demoireing methods employ multi-stream network architecture to simultaneously process the same moire image with different resolutions, but they neglect the complementarity among different resolutions. In this paper, we propose a novel moire removal model to address this issue. Unlike the existing multi-stream based approaches, in this model, we present a progressive texture complementation block to exploit the complementary information from different resolutions in order to progressively remove moire textures and restore image content. Additionally, we propose a residual moire removal block, in which the depthwise separable convolution is utilized to remove moire from image while reducing computation overhead. This block also includes a local color correction structure, which is used to correct color shifts presented in the moire images. Experimental results on two public datasets show that our method outperforms state-of-the-art methods. Besides, the quantity of parameters and FLOPs of our model are tens of times fewer than the off-the-shelf models. Furthermore, our network framework can adapt well to another low-level vision task, rain removal, in which our model also achieves state-of-the-art performance.