Array Camera Research Articles

Roving Multiple Camera Array with Structure-from-Motion for Coastal Monitoring

Regular monitoring is essential for vulnerable coastal locations such as areas of landward retreat. However, for coastal practitioners, surveying is limited by budget, specialist personnel/equipment and weather. In combination structure-from-motion and multi-view stereo (SfM-MVS) has helped to improve accessibility to topographic data acquisition. Pole-mounted cameras with SfM-MVS have gained traction but to guarantee coverage and reconstruction quality, greater understanding of camera position and interaction is required. This study uses a multi-camera array for image acquisition and reviews processing procedures in Agisoft Photoscan (Metashape). The camera rig was deployed at three sites and results were verified against a terrestrial laser scanner (TLS) and independent precision estimates. The multi-camera approach provided effective image acquisition ~11 times faster than the TLS. Reconstruction quality equalled (>92% similarity) the TLS, subject to processing parameters. A change in the image alignment parameter demonstrated a significant influence on deformation, reducing reprojection error by~94%. A lower densification parameter (‘High’) offered results ~4.39% dissimilar from the TLS at 1/8th of the processing time of other parameters. Independent precision estimates were <8.2 mm for x, y and z dimensions. These findings illustrate the potential of multi-camera systems and the influence of processing on point cloud quality and computation time.

Attention-based encoder–decoder network for depth estimation from color-coded light fields

Compressive light field cameras have attracted notable attention over the past few years because they can efficiently determine redundancy from light fields. However, much of the research has only concentrated on reconstructing the entire light field from compressed sampling, which ignores the possibility of directly extracting information such as depth from it. In this paper, we introduce a light field camera configuration with a random color-coded microlens array. Considering the color-coded light fields, we propose a novel attention-based encoder–decoder network. Specifically, the encoder part compresses the coded measurement into a low-dimensional representation that removes most redundancy, and the decoder part constructs the depth map directly from the latent representation. The attention mechanism enables the network to process spatial and angular features dynamically and effectively, thus significantly improving performance. Extensive experiments on synthetic and real-world datasets show that our method outperforms the state-of-the-art light field depth estimation method designed for non-coded light fields. To our knowledge, this is the first study that combines the color-coded light field with the attention-based deep learning approach, which provides a crucial insight into the design of enhanced light field photography systems.

Consistency analysis of focal stack-based light field reconstruction

Light field camera with a microlens array can realize spatio-angular joint sampling of light ray field at the cost of a trade-off between spatial and angular resolutions. Alternatively, focal stack-based light field reconstruction can computationally retrieve full-pixel-resolution light fields in object space by virtue of the transport-of-intensity property in an image sequence recorded at different focal depths in image space. However, traditional camera imaging generally involves a nonlinear mapping between object and image spaces. The inconsistency of image-space recording and object-space reconstruction will reduce the accuracy of reconstructed light fields. In this work, we focus on analyzing and addressing the problem caused by the object-image space inconsistency for high-resolution, high-accuracy focal stack-based light field reconstruction. With a pre-calibrated light field camera as a reference, light field reconstructions in object and image spaces are experimentally compared and discussed from different aspects, such as digital refocusing, viewpoint switching, angular resolution, and depth range and sampling rate. All experimental results demonstrate that the light field reconstruction accuracy can be significantly improved when satisfying the object-image space consistency, which can serve as a mechanism for the realization and application of high-quality computational light field imaging and measurement in the situation of nonlinear recording and reconstruction.

Visual Object Tracking Based on Light-Field Imaging in the Presence of Similar Distractors

Visual object tracking is of great importance in the field of computer vision. One of the main challenges is the difficulty of identifying moving targets from nearby similar distractors with a single-view image of the scene. To overcome this challenge, in this article, we acquire multiview images of the scenes by using a light-field camera. The multiview images are able to capture the 4-D structure instead of the 2-D plane of the objects but are more difficult to process. Therefore, we propose a novel representation for multiview images, i.e., the macro-epipolar plane image (macro-EPI), which highlights both spatial topological and angular information of the target and distractors. It is obtained by slicing the original multiview images into pieces and properly restacking these pieces in an ordinal manner. The resulting macro-EPI is mapped into the 2-D space; therefore, we adapt a modified autoencoder network to train a macro-EPI feature extractor. Thereafter, we design a composite framework of two-pattern convolution filters based on a discriminative correlation filter for object tracking, which successfully discriminates the target from the distractors by merging the macro-EPI features and the single-view image features. The experiments also show that our method outperforms the state-of-the-art methods in the presence of similar distractors.

Multi-Modal Late Fusion Rice Seed Variety Classification Based on an Improved Voting Method

Rice seed variety purity, an important index for measuring rice seed quality, has a great impact on the germination rate, yield, and quality of the final agricultural products. To classify rice varieties more efficiently and accurately, this study proposes a multimodal l fusion detection method based on an improved voting method. The experiment collected eight common rice seed types. Raytrix light field cameras were used to collect 2D images and 3D point cloud datasets, with a total of 3194 samples. The training and test sets were divided according to an 8:2 ratio. The experiment improved the traditional voting method. First, multiple models were used to predict the rice seed varieties. Then, the predicted probabilities were used as the late fusion input data. Next, a comprehensive score vector was calculated based on the performance of different models. In late fusion, the predicted probabilities from 2D and 3D were jointly weighted to obtain the final predicted probability. Finally, the predicted value with the highest probability was selected as the final value. In the experimental results, after late fusion of the predicted probabilities, the average accuracy rate reached 97.4%. Compared with the single support vector machine (SVM), k-nearest neighbors (kNN), convolutional neural network (CNN), MobileNet, and PointNet, the accuracy rates increased by 4.9%, 8.3%, 18.1%, 8.3%, and 9%, respectively. Among the eight varieties, the recognition accuracy of two rice varieties, Hannuo35 and Yuanhan35, by applying the voting method improved most significantly, from 73.9% and 77.7% in two dimensions to 92.4% and 96.3%, respectively. Thus, the improved voting method can combine the advantages of different data modalities and significantly improve the final prediction results.

The Nature of Blue Stars with Mid-infrared Excesses in the Large Magellanic Cloud

We present low-resolution optical spectra and classifications of 92 blue objects with mid-infrared excesses in the Large Magellanic Cloud. The majority of these objects were selected with the criteria of U − B < 0 and V < 17 from the potential young stellar object (YSO) candidates in Gruendl & Chu (GC09), which were identified based on Spitzer Infrared Array Camera and Multiband Imaging Photometer for Spitzer observations in conjunction with optical photometry from the Magellanic Clouds Photometric Survey. Many of the sample objects have ambiguous classifications. We examined the properties of these 92 objects using low-resolution optical spectra obtained with the SOAR 4.1 m Telescope at Cerro Pachon and the Blanco 4 m Telescope at Cerro Tololo Inter-American Observatory, supplemented by available photometric and imaging observations. We estimated the spectral types, temperatures, and luminosities of these objects from the optical to near-IR spectral energy distributions based on the photometric data, and further examined stellar absorption line features in the optical spectra to verify the spectral types. The interstellar/circumstellar environments, assessed from nebular line imaging observations and nebular lines detected in the stellar spectra, further helped constrain the nature of stars. Among these 92 objects, we confirm 42 stars as YSOs, and the remaining 50 objects as protoplanetary nebulae, post-AGB/RGB stars, blue evolved massive stars, stars with dust in vicinity, or uncertain classifications. Our results show that the photometric criteria in GC09 are generally effective in the initial selection of YSO candidates, and the low-resolution spectroscopy combined with environment assessment can be useful to better constrain the classifications and ameliorate most ambiguities.

Image fusion of multi-vision system with one high resolution image for the measurement of long distance

The mutual restriction between angle detection accuracy and spatial resolution of image will restrict the detection of long distance by multi-vision system. One higher resolution image of a multi-vision system can be used to improve the resolution of the images of each camera. In order to enhance the fusion effect of planar objects, a novel method has been proposed for dealing with high-resolution images and tested for applications. As an example, an image fusion for the images captured by the combining optical system of camera arrays and the synthetic aperture is studied. Both of the simulation and experiment shows that the proposed method is simple and accuracy.

Study of central light distribution in nearby early-type galaxies hosting nuclear star clusters

ABSTRACT We present analysis of 63 nearby (&amp;lt;44 Mpc) early-type galaxies hosting nuclear star clusters using the recently discovered parameter central intensity ratio (CIRI) determined from near-infrared (3.6 $\mu$m) observations with the Infrared Array Camera of Spitzer Space Telescope. The CIRI, when combined with filters involving age and B − K colour of host galaxies, helps identify two distinct classes of galaxies hosting nuclear star clusters. This is independently verified using Gaussian mixture model. CIR shows a positive trend with faint, low-mass, and blue galaxies in the sample, while the opposite is true for bright, high-mass, and red galaxies, albeit with large scatter. The variation of CIRI with central velocity dispersion, absolute B-band magnitude, dynamical mass, and stellar mass of host galaxies suggests that the mass of nuclear star clusters increases with that of host galaxies, for faint, low-mass, young, and blue galaxies in the sample. In bright, high-mass, old, and red galaxies, on the other hand, the evolution of nuclear star clusters appears complex, with no apparent trends. The analysis also reveals that redder galaxies (B − K &amp;gt; 3.76) are more likely to be dominated by the central black hole than the nuclear star clusters, while for bluer galaxies (B − K &amp;lt; 3.76) in the sample the situation is quite opposite.

Fusion and Allocation Network for Light Field Image Super-Resolution

Light field (LF) images taken by plenoptic cameras can record spatial and angular information from real-world scenes, and it is beneficial to fully integrate these two pieces of information to improve image super-resolution (SR). However, most of the existing approaches to LF image SR cannot fully fuse the information at the spatial and angular levels. Moreover, the performance of SR is hindered by the ability to incorporate distinctive information from different views and extract informative features from each view. To solve these core issues, we propose a fusion and allocation network (LF-FANet) for LF image SR. Specifically, we have designed an angular fusion operator (AFO) to fuse distinctive features among different views, and a spatial fusion operator (SFO) to extract deep representation features for each view. Following these two operators, we further propose a fusion and allocation strategy to incorporate and propagate the fusion features. In the fusion stage, the interaction information fusion block (IIFB) can fully supplement distinctive and informative features among all views. For the allocation stage, the fusion output features are allocated to the next AFO and SFO for further distilling the valid information. Experimental results on both synthetic and real-world datasets demonstrate that our method has achieved the same performance as state-of-the-art methods. Moreover, our method can preserve the parallax structure of LF and generate faithful details of LF images.

Polarization Orientation Method Based on Remote Sensing Image in Cloudy Weather

Autonomous navigation technology is a core technology for intelligent operation, allowing the vehicles to perform tasks without relying on external information, which effectively improves the concealability and reliability. In this paper, based on the previous research on the bionic compound eye, a multi-channel camera array with different polarization degrees was used to construct the atmospheric polarization state measurement platform. A polarization trough threshold segmentation algorithm was applied to study the distribution characteristics and characterization methods of polarization states in atmospheric remote sensing images. In the extracted polarization feature map, the tilting suggestion box was obtained based on the multi-direction window extraction network (similarity-based region proposal networks, SRPN) and the rotation of the suggestion box (Rotation Region of interests, RRoIs). Fast Region Convolutional Neural Networks (RCNN) was used to screen the suggestion boxes, and the Non-maximum suppression (NMS) method was used to select the angle, corresponding to the label of the suggestion box with the highest score, as the solar meridian azimuth in the vehicle coordinate system. The azimuth angle of the solar meridian in the atmospheric coordinate system can be calculated by the astronomical formula. Finally, the final heading angle can be obtained according to the conversion relationship between the coordinate systems. By fitting the measured data based on the least Square method, the slope K value is −1.062, RMSE (Root Mean Square Error) is 6.984, and the determination coefficient R-Square is 0.9968. Experimental results prove the effectiveness of the proposed algorithm, and this study can construct an autonomous navigation algorithm with high concealment and precision, providing a new research idea for the research of autonomous navigation technology.

Real-time grouping of tobacco through channel weighting and dynamic loss regulation

Grouping of tobacco is a key technology in intelligent management of tobacco planting and fine blending in cigarette industry. However, existing grouping methods based on deep learning are limited by lacking key feature expression due to excessive abstraction at higher scales. In addition, the inter-class discrimination ability is limited and the model prefers to majority classes during training. To tackle these problems, we propose a real-time tobacco grouping network (TGNet) based on channel weighting of higher scale features and dynamic loss regulation. Firstly, the industrial array camera equipped on a machine vision detection platform is used to capture the original tobacco images, which including the orange (F), lemon (L), variegated (K), greenish (V), and green-yellow (GY) categories manually labeled by grading experts. Then, to tackle the deficiency of key feature expression in the higher scale abstraction layers, a feature channel weighting module is proposed, which generates weighting factors proportional to different channels to re-encode higher scale features and emphatically highlight the key area and color information. The lightweight structure of feature channel weighting module compresses the model volume and improves forward inference speed. Finally, aiming at the problems of inter-group ambiguity and class imbalance dilemma, the focal and dynamic margin (FADM) loss with focusing on minority classes training and adaptative margin is proposed. The dynamic margin is adaptatively encoded by a Cosine function of similarity, which uses tobacco category recognition probability to measure the similarity of tobacco. Extensive performance evaluations show that TGNet outperforms other classification models in terms of the average classification accuracy 77.16%, inference time 112FPS, model size 2.5MB, model computation 0.12GFlops and model parameter number 0.613 M. The classification accuracy of minority classes V and GY is improved.

High-throughput sorting of two-color fluorescent-labeled zebrafish embryos

The zebrafish embryos were widely employed in genetics, development and drug discovery studies as miniatured animal models. Sorting of two-color fluorescent embryos is often required in large-scale experiments but it is challenging to manually sort with high efficiency. Here, we reported a high-throughput sorting system for two-color fluorescent zebrafish embryos. The embryos can be automatically loaded from a sample pool and sorted based on the average fluorescent intensity. The two-color fluorescent signals were split into two lines and detected by an area array camera. The system achieves the sorting of 100 embryos in less than 10[Formula: see text]min with an accuracy of greater than 95%.

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image.

Light field reconstruction and synthesis algorithms are essential for improving the lower spatial resolution for hand-held plenoptic cameras. Previous light field synthesis algorithms produce blurred regions around depth discontinuities, especially for stereo-based algorithms, where no information is available to fill the occluded areas in the light field image. In this paper, we propose a light field synthesis algorithm that uses the focal stack images and the all-in-focus image to synthesize a 9 × 9 sub-aperture view light field image. Our approach uses depth from defocus to estimate a depth map. Then, we use the depth map and the all-in-focus image to synthesize the sub-aperture views, and their corresponding depth maps by mimicking the apparent shifting of the central image according to the depth values. We handle the occluded regions in the synthesized sub-aperture views by filling them with the information recovered from the focal stack images. We also show that, if the depth levels in the image are known, we can synthesize a high-accuracy light field image with just five focal stack images. The accuracy of our approach is compared with three state-of-the-art algorithms: one non-learning and two CNN-based approaches, and the results show that our algorithm outperforms all three in terms of PSNR and SSIM metrics.

Geometric Parameters Calibration of Focused Light Field Camera Based on Edge Spread Information Fitting

In this paper, a novel method based on edge spread information fitting (ESIF) is proposed to accurately calibrate the geometric parameters of a focused light field camera. A focused light field camera with flexible adjustment of spatial resolution and angular resolution is designed and built to capture the four-dimensional light field information of the scenes, and the geometric relationship between the focus plane of the camera and its internal parameters is derived to establish and simplify the calibration model. After that, a new algorithm based on sub-pixel edge fitting is designed to accurately detect corners, and the minimum corner size can be calculated to confirm the focus plane. In the simulation experiments, the error is 0.083% between the ground truth and the calibration result. The physical experiments show that our method is effective and reliable for the geometric calibration of a focused light field camera. Our method cleverly utilizes the special imaging geometric relationship of the focused light field camera to ensure the better calibration accuracy, which makes the calibration process more reliable.

Do golden snub-nosed monkeys use deceptive alarm calls during competition for food?

Tactical deception can be beneficial for social animals during intra-specific competition. However, the use of tactical deception in wild mammals is predicted to be rare. We tested whether a food-provisioned free-ranging band of golden snub-nosed monkeys (Rhinopithecus roxellana) use alarm calls in a functionally deceptive manner to gain access to food resources, whether the rate of deceptive alarm calls varies among individuals, and whether there are any counter-deception behaviors. We used a hexagonal camera array consisting of 10 cameras to record videos during feeding, which allowed us to identify individual alarm callers. We found evidence that these monkeys use deceptive alarms and that adult females were more likely to use such calls than other individuals. The monkeys increased their rates of response to alarm calls when competition for food was high. However, we found no direct evidence of any counter-deception strategies.

Curved Computer-Generated Integral Imaging Based on Backward Ray-Tracing Technique

In traditional computer-generated integral imaging (CGII) methods, the generated elemental image array (EIA) is flat, and it doesn't fit for curved integral imaging (II) systems. Here, a curved CGII method was proposed for curved II system. In the proposed method, multiple virtual camera arrays (VCAs) were built up to determine the origins and directions of the traced rays. Combined with the backward ray-tracing (BRT) technique, a curved EIA was generated, and the rendering time was the same as the BRT-CGII of flat II system. A curved II system consisting of a curved organic light emitting diode display screen, a curved lens array and a curved optical diffuser screen was built up in the experiment. The proposed curved CGII method generated correct curved EIA for the curved II system, and the system presented vivid 3D images with the viewing angle as wide as 80° × 50°.

Development of a prototype 16-channel dual gain waveform readout and data transfer system for IACT based telescope camera

A 256-pixel camera for a 4-meter class telescope based on Imaging Atmospheric Cherenkov Technique (IACT) is under development. It would be mounted at the focal plane of TeV Atmospheric Cherenkov Telescope with Imaging Camera (TACTIC) array element at Mount Abu (Rajasthan, India). The telescope would be used for detection of very high energy gamma rays of celestial origin. Instead of the traditional choice of photo-multiplier tube, the camera employs silicon photo-multiplier (SiPM) as pixel sensor. The front-end electronics of the camera processes signals from 256 pixel sensors. The back-end electronics samples the processed pixel signals, generates event triggers and records the sampled signals. The critical design objectives to be met by the back-end electronics include large dynamic range of photon detection, capability of in-situ gain calibration for each pixel chain, low chance trigger rate, recording the pixel signal profile with a sampling rate of 1 Giga Samples Per Second (GSPS), low dead time (<1 millisecond) and run-time control of the camera operation. Also, the camera cost, power, size and weight should be as low as possible. The important back-end tasks like pixel signal processing, generating pre-triggers, digitization of the pixel pulse profiles, packing and sending the digitized data are performed by the custom designed Cluster Digitizer Module (CDM). CDM is a 16-channel readout system, and its design revolves around an ultra-fast analog sampler chip and a set of FPGAs. The CDM is thoroughly tested in the lab using test signals from SiPM as well as high performance function generator. The paper describes the design, development and performance of the prototype CDM.

Evaluation of low-cost Raspberry Pi sensors for structure-from-motion reconstructions of glacier calving fronts

Abstract. Glacier calving fronts are highly dynamic environments that are becoming ubiquitous as glaciers recede and, in many cases, develop proglacial lakes. Monitoring of calving fronts is necessary to fully quantify the glacier ablation budget and to warn nearby communities of the threat of hazards, such as glacial lake outburst floods (GLOFs), tsunami waves, and iceberg collapses. Time-lapse camera arrays, with structure-from-motion photogrammetry, can produce regular 3D models of glaciers to monitor changes in the ice but are seldom incorporated into monitoring systems owing to the high cost of equipment. In this proof-of-concept study at Fjallsjökull, Iceland, we present and test a low-cost, highly adaptable camera system based on Raspberry Pi computers and compare the resulting point cloud data to a reference cloud generated using an unoccupied aerial vehicle (UAV). The mean absolute difference between the Raspberry Pi and UAV point clouds is found to be 0.301 m with a standard deviation of 0.738 m. We find that high-resolution point clouds can be robustly generated from cameras positioned up to 1.5 km from the glacier (mean absolute difference 0.341 m, standard deviation 0.742 m). Combined, these experiments suggest that for monitoring calving events in glaciers, Raspberry Pi cameras are an affordable, flexible, and practical option for future scientific research. Owing to the connectivity capabilities of Raspberry Pi computers, this opens the possibility for real-time structure-from-motion reconstructions of glacier calving fronts for deployment as an early warning system to calving-triggered GLOFs.

Calibrating an unfocused plenoptic camera based on parameters grouping and the light field structure point.

Accurately calibrating an unfocused plenoptic camera is essential to its applications. Rapid progress has been made in this area in the past decades. In this paper, detailed analysis is first performed toward the state-of-the-art projection model. Based on the analysis, parameters in the projection model are divided into two groups. Then, based on the parameter analysis, a new, to the best of our knowledge, form of the projection model, together with a new image feature light field structure point (LF-structure-point), is proposed. The LF-structure-point provides a completely non-redundant representation of the signal structure of the recorded light field raw data and induces a virtual space, "light field structure space," which is related to the real physical space by a 3D-to-3D projective transformation. The extracting algorithm of the LF-structure-point is also presented. Finally, based on the 3D-to-3D projective transformation and parameter grouping, a simple two-step calibration method using the LF-structure-point as the input data is then proposed and achieves satisfactory experimental results.

On-Orbit Relative Radiometric Calibration of the Bayer Pattern Push-Broom Sensor for Zhuhai-1 Video Satellites

The two video satellites of the second and third batch of Zhuhai-1 microsatellites (referred to as OVS-2A/3A) are operational with their hyperspectral satellites, which improves the data acquisi-tion capability of the Zhuhai-1 remote sensing satellite constellation. Contrary to the linear array push-broom hyperspectral satellites and plane array CCD video satellites, the OVS satellite is equipped with a planar array Bayer pattern sensor, which can obtain single-band grayscale images by push-broom imaging. Additionally, the Bayer color reconstruction algorithm can interpolate sensor data to provide RGB color band information. Therefore, for the Bayer pattern push-broom sensor, the relative calibration method of linear push-broom or array cameras cannot be directly applied. The radiometric calibration of the Bayer pattern push-broom imaging mode has become a matter of concern; therefore, this study developed a radiometric calibration method for the Bayer pattern push-broom sensor of the OVS satellite and verified its effectiveness and accuracy. OVS images were used to perform on-orbit relative radiometric calibration, and the calibration accu-racy, including streaking metrics and root-mean-square error, was better than 1%, meeting the specification requirements for the OVS satellite. Visually, after calibration correction, the streaking and striping noise of the Bayer images was removed, and the radiometric quality of the image was considerably improved, providing a good data basis for subsequent research in remote sensing applications.