Spatio-angular Information Research Articles

SUPER-RESOLUTION OF DIFFUSION MRI VIA JOINT CONSIDERATION OF SPATIAL AND ANGULAR SPACES USING GRAPH CNN

Diffusion magnetic resonance imaging (dMRI) is an essential technique for studying brain white matter. However, its application suffers from the tradeoff between resolution, signal-to-noise ratio (SNR), and scanning time. Under a given setting, post-acquisition super-resolution (SR) has displayed the exciting performance of increasing the spatial or angular resolution of dMRI without prolonging the acquisition time. dMRI is acquired from a heterogeneous space composed of the spatial domain (x-space) and the diffusion wave vector domain (q-space). We introduced a graph convolution neural network (GCNN) to fit the mapping from low-resolution (LR) dMRI to high-resolution (HR) counterpart by jointly considering x-space and q-space to make an effective learning in this non-Euclidean space to increase the spatial resolution. Real brain dMRI data were downloaded from the Human Connectome Project (HCP). Comparisons were made between different spaces, including x-space, q-space, and xq-space. The results indicated that the best performance was obtained when two spaces were considered jointly using GCNN. Compared with the traditional methods, our network fully used spatio-angular information to improve the quality of dMRI SR. Furthermore, some feature metrics, including fractional anisotropic mapping, fiber orientation distribution (FOD), and fiber tractography, were presented to support the improvement. The cross-phantom test on synthetic data generated by the Phantomas tool was performed to further evaluate our model. Extended experiments revealed that GCNN might serve as a new way of enhancing dMRI.

Miniaturized light-field endoscope via a GRIN lens array

Endoscopy provides an intuitive observation of the internal organ structures for surgeons. Most clinical 3D endoscopes generally rely on visual parallax to create a stereoscopic vision, which is incapable of obtaining quantitative depth information for precise diagnosis. Hence a compact probe-type 3D quantitative imaging remains a challenging task for the endoscopic system. In this work, we propose an integrated light-field endoscope that realizes 3D morphology observation with cellular-level resolution from a single-frame data record. In our prototype, a compact customized GRIN lens array (GLA) is used as an objective lens to capture the spatio-angular information directly from object space, as well as maintain consistent spatial measurements. In data processing, we employ a calibration method to estimate the 3D point spread function. After decoupling the spatio-angular information, the 3D morphology of sample can be reconstructed by a deconvolution operation. The system has a lateral resolution of about 15∼40 μm and an axial resolution of 100∼200 μm over an axial operating range of 10 mm. In experiment, we demonstrate the 3D imaging capacity of our platform via fixed structures and living worms.

Compressive HDR Light Field Imaging Using a Single Multi-ISO Sensor

In this paper, we propose a new design for single sensor compressive HDR light field cameras, combining multi-ISO photography with coded mask acquisition, placed in a compressive sensing framework. The proposed camera model is based on a main lens, a multi-ISO sensor and a coded mask located in the optical path between the main lens and the sensor that projects the coded spatio-angular information of the light field onto the 2D sensor. The model encompasses different acquisition scenarios with different ISO patterns and gains. Moreover, we assume that the sensor has a built-in color filter array (CFA), making our design more suitable for consumer-level cameras. We propose a reconstruction algorithm to jointly perform color demosaicing, light field angular information recovery, HDR reconstruction, and denoising from the multi-ISO measurements formed on the sensor. This is achieved by enabling the sparse representation of HDR light fields using an overcomplete HDR dictionary. We also provide two HDR light field data sets: one synthetic data set created using the Blender rendering software with two baselines, and a real light field data set created from the fusion of multi-exposure low dynamic range (LDR) images captured using a Lytro Illum light field camera. Experimental results show that, with a sampling rate as low as 2.67%, using two shots, our proposed method yields a higher light field reconstruction quality compared to the fusion of multiple LDR light fields captured with different exposures, and with the fusion of multiple LDR light fields captured with different ISO settings.

A Double-Deep Spatio-Angular Learning Framework for Light Field-Based Face Recognition

Face recognition has attracted increasing attention due to its wide range of applications, but it is still challenging when facing large variations in the biometric data characteristics. Lenslet light field cameras have recently come into prominence to capture rich spatio-angular information, thus offering new possibilities for advanced biometric recognition systems. This paper proposes a double-deep spatio-angular learning framework for light field-based face recognition, which is able to model both the intra-view/spatial and inter-view/angular information using two deep networks in sequence. This is a novel recognition framework that has never been proposed in the literature for face recognition or any other visual recognition task. The proposed double-deep learning framework includes a long short-term memory (LSTM) recurrent network, whose inputs are VGG-Face descriptions, computed using a VGG-16 convolutional neural network (CNN). The VGG-Face spatial descriptions are extracted from a selected set of 2D sub-aperture (SA) images rendered from the light field image, corresponding to different observation angles. A sequence of the VGG-Face spatial descriptions is then analyzed by the LSTM network. A comprehensive set of experiments has been conducted using the IST-EURECOM light field face database, addressing varied and challenging recognition tasks. The results show that the proposed framework achieves superior face recognition performance when compared to the state of the art.

Long Short-Term Memory With Gate and State Level Fusion for Light Field-Based Face Recognition

Long Short-Term Memory (LSTM) is a prominent recurrent neural network for extracting dependencies from sequential data such as time-series and multi-view data, having achieved impressive results for different visual recognition tasks. A conventional LSTM network, hereafter referred only as LSTM network, can learn a model to posteriorly extract information from one input sequence. However, if two or more dependent sequences of data are simultaneously acquired, the LSTM networks may only process those sequences consecutively, not taking benefit of the information carried out by their mutual dependencies. In this context, this paper proposes two novel LSTM cell architectures that are able to jointly learn from multiple sequences simultaneously acquired, targeting to create richer and more effective models for recognition tasks. The efficacy of the novel LSTM cell architectures is assessed by integrating them into deep learning-based methods for face recognition with multi-view, light field images. The new cell architectures jointly learn the scene horizontal and vertical parallaxes available in a light field image, to capture richer spatio-angular information from both directions. A comprehensive evaluation, with the IST-EURECOM LFFD dataset using three challenging evaluation protocols, shows the advantage of using the novel LSTM cell architectures for face recognition over the state-of-the-art light field-based methods. These results highlight the added value of the novel cell architectures when learning from correlated input sequences.

Three-dimensional real-time darkfield imaging through Fourier lightfield microscopy.

We report a protocol that takes advantage of the Fourier lightfield microscopy concept for providing 3D darkfield images of volumetric samples in a single-shot. This microscope takes advantage of the Fourier lightfield configuration, in which a lens array is placed at the Fourier plane of the microscope objective, providing a direct multiplexing of the spatio-angular information of the sample. Using the proper illumination beam, the system collects the light scattered by the sample while the background light is blocked out. This produces a set of orthographic perspective images with shifted spatial-frequency components that can be recombined to produce a 3D darkfield image. Applying the adequate reconstruction algorithm high-contrast darkfield optical sections are calculated in real time. The presented method is applied for fast volumetric reconstructions of unstained 3D samples.

Light-field depth estimation considering plenoptic imaging distortion.

Light-field imaging can simultaneously record spatio-angular information of light rays to carry out depth estimation via depth cues which reflect a coupling of the angular information and the scene depth. However, the unavoidable imaging distortion in a light-field imaging system has a side effect on the spatio-angular coordinate computation, leading to incorrectly estimated depth maps. Based on the previously established unfocused plenoptic metric model, this paper reports a study on the effect of the plenoptic imaging distortion on the light-field depth estimation. A method of light-field depth estimation considering the plenoptic imaging distortion is proposed. Besides, the accuracy analysis of the light-field depth estimation was performed by using standard components. Experimental results demonstrate that efficiently compensating the plenoptic imaging distortion results in a six-fold improvement in measuring accuracy and more consistency across the measuring depth range. Consequently, the proposed method is proved to be suitable for light-field depth estimation and three-dimensional measurement with high quality, enabling unfocused plenoptic cameras to be metrological tools in the potential application scenarios such as industry, biomedicine, entertainment, and many others.

Light Field Synthesis by Training Deep Network in the Refocused Image Domain.

Light field imaging, which captures spatial-angular information of light incident on image sensors, enables many interesting applications such as image refocusing and augmented reality. However, due to the limited sensor resolution, a trade-off exists between the spatial and angular resolutions. To increase the angular resolution, view synthesis techniques have been adopted to generate new views from existing views. However, traditional learning-based view synthesis mainly considers the image quality of each view of the light field and neglects the quality of the refocused images. In this paper, we propose a new loss function called refocused image error (RIE) to address the issue. The main idea is that the image quality of the synthesized light field should be optimized in the refocused image domain because it is where the light field is viewed. We analyze the behavior of RIE in the spectral domain and test the performance of our approach against previous approaches on both real (INRIA) and software-rendered (HCI) light field datasets using objective assessment metrics such as MSE, MAE, PSNR, SSIM, and GMSD. Experimental results show that the light field generated by our method results in better refocused images than previous methods.

Scene-Independent Motion Pattern Segmentation in Crowded Video Scenes Using Spatio-Angular Density-Based Clustering

Motion pattern segmentation for crowded video scenes is an open problem because of the inability of existing approaches to tackle unpredictable crowd behaviour across varied scenes. To address this problem, we propose a Spatio-Angular Density-based Clustering (SADC) approach, which performs motion pattern segmentation by clustering the spatial and angular information obtained from the input trajectories. The k-nearest neighbours of each trajectory and the angular deviation between trajectories constitute the spatial and angular information, respectively. Effective integration of the spatio-angular information with an improvised density-based clustering algorithm makes this approach scene-independent. The performance of most clustering algorithms in the literature is parameter-driven. Choosing a single parameter value for different types of scenes decreases the overall clustering performance. In this article, we have shown that our approach is robust to scene changes using a single threshold, and, through the analysis of parameters across eight commonly occurring crowded scenarios, we point out the range of thresholds that are suitable for each scene category. We evaluate the proposed approach on the benchmarked CUHK dataset. The experimental results show the superior clustering performance and execution speed of the proposed approach when compared to the state-of-the-art over different scene categories.

LightGAN: A Deep Generative Model for Light Field Reconstruction

A light field image captured by a plenoptic camera can be considered a sampling of light distribution within a given space. However, with the limited pixel count of the sensor, the acquisition of a high-resolution sample often comes at the expense of losing parallax information. In this work, we present a learning-based generative framework to overcome such tradeoff by directly simulating the light field distribution. An important module of our model is the high-dimensional residual block, which fully exploits the spatio-angular information. By directly learning the distribution, our approach can generate both high-quality sub-aperture images and densely-sampled light fields. Experimental results on both real-world and synthetic datasets demonstrate that the proposed method outperforms other state-of-the-art approaches and achieves visually more realistic results.

High-Accuracy Correction of a Microlens Array for Plenoptic Imaging Sensors.

Microlens array (MLA) errors in plenoptic cameras can cause the confusion or mismatching of 4D spatio-angular information in the image space, significantly affecting the accuracy and efficiency of target reconstruction. In this paper, we present a high-accuracy correction method for light fields distorted by MLA errors. Subpixel feature points are extracted from the microlens subimages of a raw image to obtain correction matrices and perform registration of the corresponding subimages at a subpixel level. The proposed method is applied for correcting MLA errors of two different categories in light-field images, namely form errors and orientation errors. Experimental results show that the proposed method can rectify the geometric and intensity distortions of raw images accurately and improve the quality of light-field refocusing. Qualitative and quantitative comparisons between images before and after correction verify the performance of our method in terms of accuracy, stability, and adaptability.

Identification and correction of microlens array rotation error in plenoptic imaging systems

Plenoptic imaging systems can acquire spatio-angular information of light rays, allowing complex scene measurements with a single exposure. The measurement accuracy is significantly affected by the orientation parameters of internal components in the system, especially the rotation error between the microlens array (MLA) and the image sensor. In this paper, we formulate an MLA rotation error model to derive the transformations between the error point and its ideal point of ray projection. Furthermore, we present a method for identifying and correcting the rotation error, including error angle estimation, raw image rotation, and refocused image reverse rotation. Through simulation experiments, the imaging characteristics of the MLA rotation error are analyzed and the performance of the proposed model and method are verified. The results demonstrate that the integral distribution of raw image distortion caused by the error is centrosymmetric. In the error range of 0.1–1.0°, the mean square error (MSE) and structural similarity index metric (SSIM) of the corrected refocused images are improved to 0.2146–3.8190 and 0.9986–0.9999, respectively.

Ear recognition in a light field imaging framework: a new perspective

Ear recognition is an emerging research area in image-based biometrics. The commercial availability of lenslet light field cameras able to capture full spatio-angular information has brought momentum to biometric and forensic research exploiting this new type of imaging sensors. This study is the first to consider the usage of light field cameras for ear recognition, proposing both an appropriate content database and an ear recognition solution. The novel Lenslet Light Field Ear DataBase (LLFEDB) includes 536 light field images corresponding to four different poses, from 67 subjects, captured with a Lytro ILLUM lenslet light field camera. The LLFEDB includes critical cases such as ear images partly occluded by ear piercing, earing, hair and combination of occlusions. The novel light field-based ear recognition solution proposed exploits the richer spatio-angular information available in the light field images. A comparative performance evaluation study using the LLFEDB, and focusing on the most recent light field and non-light field based descriptors for ear recognition, shows a very promising performance of the proposed descriptor, outperforming all the assessed descriptors.

A Light Transport Framework for Lenslet Light Field Cameras

Light field cameras capture full spatio-angular information of the light field, and enable many novel photographic and scientific applications. It is often stated that there is a fundamental trade-off between spatial and angular resolution, but there has been limited understanding of this trade-off theoretically or numerically. Moreover, it is very difficult to evaluate the design of a light field camera because a new design is usually reported with its prototype and rendering algorithm, both of which affect resolution. In this article, we develop a light transport framework for understanding the fundamental limits of light field camera resolution. We first derive the prefiltering model of lenslet-based light field cameras. The main novelty of our model is in considering the full space-angle sensitivity profile of the photosensor—in particular, real pixels have nonuniform angular sensitivity , responding more to light along the optical axis rather than at grazing angles. We show that the full sensor profile plays an important role in defining the performance of a light field camera. The proposed method can model all existing lenslet-based light field cameras and allows to compare them in a unified way in simulation, independent of the practical differences between particular prototypes. We further extend our framework to analyze the performance of two rendering methods: the simple projection-based method and the inverse light transport process. We validate our framework with both flatland simulation and real data from the Lytro light field camera.

Capture of the spatio-angular information of a 3D scene

En este trabajo se han estudiado los principios basicos de los sistemas de captura plenopticos, mediante los cuales es posible capturar no solo la informacion espacial, sino tambien la informacion angular de un objeto o escena 3D. Esta informacion espacio-angular queda determinada a traves del concepto de lightfield o funcion plenoptica. Para la verificacion de los conceptos presentados, se ha procedido a la captura de una imagen plenoptica mediante un sistema compuesto por un conjunto de microlentes distribuidas en disposicion hexagonal. De esta manera, se ha podido comprobar que el registro de la informacion espacio-angular permite reconstruir la escena 3D a partir de una unica captura. Esta reconstruccion se ha llevado a cabo, por una parte, mediante un reenfoque en profundidad de la escena. Por otro lado, tambien ha sido posible llevar a cabo una reconstruccion angular, permitiendo al observador observar la escena desde diferentes perspectivas.