Scene Light Field Research Articles

A New Parallel Intelligence Based Light Field Dataset for Depth Refinement and Scene Flow Estimation

Computer vision tasks, such as motion estimation, depth estimation, object detection, etc., are better suited to light field images with more structural information than traditional 2D monocular images. However, since costly data acquisition instruments are difficult to calibrate, it is always hard to obtain real-world scene light field images. The majority of the datasets for static light field images now available are modest in size and cannot be used in methods such as transformer to fully leverage local and global correlations. Additionally, studies on dynamic situations, such as object tracking and motion estimates based on 4D light field images, have been rare, and we anticipate a superior performance. In this paper, we firstly propose a new static light field dataset that contains up to 50 scenes and takes 8 to 10 perspectives for each scene, with the ground truth including disparities, depths, surface normals, segmentations, and object poses. This dataset is larger scaled compared to current mainstream datasets for depth estimation refinement, and we focus on indoor and some outdoor scenarios. Second, to generate additional optical flow ground truth that indicates 3D motion of objects in addition to the ground truth obtained in static scenes in order to calculate more precise pixel level motion estimation, we released a light field scene flow dataset with dense 3D motion ground truth of pixels, and each scene has 150 frames. Thirdly, by utilizing the DistgDisp and DistgASR, which decouple the angular and spatial domain of the light field, we perform disparity estimation and angular super-resolution to evaluate the performance of our light field dataset. The performance and potential of our dataset in disparity estimation and angular super-resolution have been demonstrated by experimental results.

Semantic Segmentation With Light Field Imaging and Convolutional Neural Networks

Semantic segmentation is of great importance and a challenge in computer vision. One of its main problems is how to efficiently obtain rich information (geometric structure) and identify useful features from higher dimensions. A light field camera, due to its special microlens array structure, can completely record the angular-spatial information of scenes, which is attractive and has great potential to improve the performance of semantic segmentation tasks. Inspired by this, we propose an end-to-end semantic segmentation network that can process light field macropixel images robustly and extract their features. In addition, this network can flexibly and efficiently load the different popular deep learning backbones. Furthermore, we propose an efficient angular model, which, to learn the angular features between the different viewpoints of the macropixel image, improves the nonlinearity of angular-spatial features and enhances multichannel semantic correlations. To evaluate the network, we construct a new real scene light field dataset comprising 800 high-quality samples. The quantitative and qualitative results show that the highest mean intersection over union (mIoU) based on our algorithm is greater than 57%. Our algorithm achieves a 10.30% increase compared with state-of-the-art semantic segmentation algorithms. In combination with different backbones or multiscale light field macropixel images, the network can also achieve comparable results. This preliminary work demonstrates that the combination of light field imaging and deep learning technology has potential applications in the future study of semantic segmentation.

Depth Estimation From a Light Field Image Pair With a Generative Model

In this paper, we propose a novel method to estimate the disparity maps from a light field image pair captured by a pair of light field cameras. Our method integrates two types of critical depth cues, which are separately inferred from the epipolar plane images and binocular stereo vision into a global solution. At the same time, in order to produce highly accurate disparity maps, we adopt a generative model, which can estimate a light field image only with the central subaperture view and corresponding hypothesized disparity map. The objective function of our method is formulated to minimize two energy terms/differences. One is the difference between the two types of previously extracted disparity maps and the target disparity maps, directly optimized in the gray-scale disparity space. The other indicates the difference between the estimated light field images and the input light field images, optimized in the RGB color space. Comprehensive experiments conducted on real and virtual scene light field image pairs demonstrate the effectiveness of our method.

Fabrication of flexible highly reflective micromirrors arrays for sunlight reflectors

Micromirrors have applications in optical displays, optical switches and biological imaging systems. In this paper, we developed the notion of flexible micromirrors solar light reflecting system. This proposed fabrication enables the system to select rays from the scene's light field based on the needs of the application at hand. Such a system achieves high reflective metal mirrors from 1064 nm – the near infrared (NIR) – down to 150 nm – the vacuum ultraviolet (VUV) – spectral region and solves problems with many micromirror designs including size limitations and fabrication difficulty. This flexibility is achieved using an array of micro-mirrors fabricated using inverse chemical vapour deposition and photolithography.

Programmable Imaging: Towards a Flexible Camera

In this paper, we introduce the notion of a programmable imaging system. Such an imaging system provides a human user or a vision system significant control over the radiometric and geometric characteristics of the system. This flexibility is achieved using a programmable array of micro-mirrors. The orientations of the mirrors of the array can be controlled with high precision over space and time. This enables the system to select and modulate rays from the scene's light field based on the needs of the application at hand. We have implemented a programmable imaging system that uses a digital micro-mirror device (DMD), which is used in digital light processing. Although the mirrors of this device can only be positioned in one of two states, we show that our system can be used to implement a wide variety of imaging functions, including, high dynamic range imaging, feature detection, and object recognition. We also describe how a micro-mirror array that allows full control over the orientations of its mirrors can be used to instantly change the field of view and resolution characteristics of the imaging system. We conclude with a discussion on the implications of programmable imaging for computer vision.