Exploring Extrinsic Spectral Properties of Hyperspectral Image for 3D Computer Vision

Abstract

A hyperspectral image (HSI) consists of band images that capture the response from numerous light wavelengths in the electromagnetic spectrum. Therefore, HSI has a higher spectral resolution than RGB and grayscale images, which gives HSI enhanced characteristics that are useful in solving various real-world problems. In particular, hyperspectral imaging has been used for analysis and interpretation of material-specific properties in an image. Traditionally, most of the research on this topic has focused on the remote sensing. With the development of sensor technology, HSI has been introduced to other areas including scene understanding, medicine, food quality assessment etc. With the wide adoption of HSI, there is a demand to more effectively understand and exploit the information in HSI by exploring both intrinsic and extrinsic properties of HSI. The intrinsic properties in HSI consist of inherent information in the image due to various optical phenomenons. In contrast, the extrinsic property consists of compound information driven from intrinsic information. For this thesis, we focus mainly on how different properties in a hyperspectral image can contribute towards solving computer vision problems effectively. Although extrinsic properties of HSI can be explored for various research areas, we focus out research towards 3D computer vision due to its broad potential in both theoretical research and real-world applications. In this thesis, three main technical approaches are presented. Our first proposed approach focuses on the 3D reconstruction problem. We exploited spectral details in HSI to get the 3D structure, and we call it as structure from spectra. This is a challenging problem because 3D models reconstructed from different spectral bands demonstrate different structural properties. Our proposed method first generates 3D point sets from images at each wavelength using the typical structure from motion approach. A structural descriptor is developed to characterize the spatial relationship between the points, which allows robust point matching between two 3D models at different wavelengths. Then a 3D registration method is introduced to combine all band-level models into a single and complete hyperspectral 3D model. As far as we know, this is the first attempt in reconstructing a complete 3D model from hyperspectral images. This work allows fine structural-spectral information of an object to be captured and integrated into the 3D model, which can be used to support further research and applications. Our second method focuses on relative depth estimation. Here we exploited chromatic aberration and defocus blur in the monocular HSI to get depth cues. We propose that change in focus across band images of HSI due to chromatic aberration and band-wise defocus blur can be integrated for depth estimation. Then, novel methods are developed to estimate sparse depth maps based on different integration models. After that, by adopting manifold learning, an effective objective function is designed to combine all sparse depth maps into a final optimized sparse depth map. Lastly, a new dense depth map generation approach is proposed, which extrapolates sparse depth cues by using material-based properties on graph Laplacian. Experimental results show that our methods successfully exploit HSI properties to generate depth cues. We also compare our method with state-of-the-art RGB image-based approaches, which shows that our methods produce better sparse and dense depth maps than those from the benchmark methods. Our third novel approach deals with the occluded object detection problem. We use depth cues and material based information (reflectance) to get extrinsic property of continuous labels for detecting objects in the occluded environment. We propose the first method that exploits the unique fine reflectance properties of HSI to explore joint spectral and spatial information in HSI for occlusion detection. Specifically, our approach combines material distribution with depth cues extracted from a monocular HSI. The material distribution is estimated using a blind unmixing method based on hierarchical clustering. Then depths of objects are calculated using defocus blur and chromatic aberration. Finally, we combine both information using graph Laplacian to detect occluded objects. Lastly, in the Appendix, two supplementary methods are presented, which deals with spectral-spatial feature detection and object boundary detection problems in HSI. These two methods were partially derived from the research related to this thesis. They employ reflectance with structural and geometric cue present in HSI in different ways to achieve the goal.