Three-dimensional Data Cube Research Articles

Freeform-surface-based optical design of a broadband compressive spectral imager with co-aperture coding

Compressive spectral imaging (CSI) is an advanced computational imaging approach to reconstruct the three-dimensional (3D) spatio-spectral data cube of a target scene through a single or a few snapshots. However, limited by the response range of the image detector, the existing CSI systems mostly work within narrow spectral bands, such as the visible or shortwave-infrared (SWIR) spectral band. The work band of the CSI system constrains the detection capacity for the targets under complex environments (such as rain, snow, haze, etc.). In addition, most of the current CSI prototypes lack engineering design for practical applications. This paper develops a novel, to the best of our knowledge, optical design scheme of a broadband CSI system with co-aperture coding to simultaneously realize visible multi-spectral imaging (10 channels) and SWIR super-resolution imaging (16×). The freeform surfaces are used to design the front-end reflective objective lens, thus significantly improving the image quality and spatial modulation precision of the system. By means of performance evaluation and tolerance analysis, excellent image quality and manufacturability of the proposed system are demonstrated.

Choledochal cancer region detection in hyperspectral images using U-Net based models

Cholangiocarcinoma (CCA) is a type of cancer that forms in the bile duct that carry digestive fluid from the liver. CCA is the primary form of liver cancer that affects population ranging from age 60 to 69 years. CCA is difficult to diagnose at an early stage. Hyperspectral (HS) imaging is an advanced imaging technique that combines spectroscopy with conventional imaging. HS imaging is an emerging field of study which can be used for early CCA detection. HS imaging involves capturing images across various spectral bands, which forms a three-dimensional data cube often called as hyperspectral data cube. In this study, we have utilized U-Net based models, namely U-Net and DenseUNet were used to perform semantic segmentation on the HS images of CCA tissues. A band selective approach was employed to derive a subset of meaningful bands based on the spectrum plot from the HS image. The HS images are further preprocessed with Principal Component Analysis (PCA). The models were further evaluated by computing the accuracy, AUC (Area under the ROC curve), sensitivity and specificity metrics. The proposed models, namely, U-Net and DenseUNet reported an overall accuracy of 73.47% and 77.09% respectively. The DenseUNet models outperforms the U-Net model on every evaluation metric. The proposed models were also compared with other state-of-the-art (SOTA) models trained on various HS dataset. This study explores the application of HS imaging in carcinoma detection. The findings of this study could be used for further enhancement of the approach.

Fast Nano-IR Hyperspectral Imaging Empowered by Large-Dataset-Free Miniaturized Spatial-Spectral Network.

The emerging field of nanoscale infrared (nano-IR) offers label-free molecular contrast, yet its imaging speed is limited by point-by-point traverse acquisition of a three-dimensional (3D) data cube. Here, we develop a spatial-spectral network (SS-Net), a miniaturized deep-learning model, together with compressive sampling to accelerate the nano-IR imaging. The compressive sampling is performed in both the spatial and spectral domains to accelerate the imaging process. The SS-Net is trained to learn the mapping from small nano-IR image patches to the corresponding spectra. With this elaborated mapping strategy, the training can be finished quickly within several minutes using the subsampled data, eliminating the need for a large-labeled dataset of common deep learning methods. We also designed an efficient loss function, which incorporates the image and spectral similarity to enhance the training. We first validate the SS-Net on an open stimulated Raman-scattering dataset; the results exhibit the potential of 10-fold imaging speed improvement with state-of-the-art performance. We then demonstrate the versatility of this approach on atomic force microscopy infrared (AFM-IR) microscopy with 7-fold imaging speed improvement, even on nanoscale Fourier transform infrared (nano-FTIR) microscopy with up to 261.6 folds faster imaging speed. We further showcase the generalization of this method on AFM-force volume-based multiparametric nanoimaging. This method establishes a paradigm for rapid nano-IR imaging, opening new possibilities for cutting-edge research in materials, photonics, and beyond.

Wide-field mid-infrared hyperspectral imaging beyond video rate

Mid-infrared hyperspectral imaging has become an indispensable tool to spatially resolve chemical information in a wide variety of samples. However, acquiring three-dimensional data cubes is typically time-consuming due to the limited speed of raster scanning or wavelength tuning, which impedes real-time visualization with high spatial definition across broad spectral bands. Here, we devise and implement a high-speed, wide-field mid-infrared hyperspectral imaging system relying on broadband parametric upconversion of high-brightness supercontinuum illumination at the Fourier plane. The upconverted replica is spectrally decomposed by a rapid acousto-optic tunable filter, which records high-definition monochromatic images at a frame rate of 10 kHz based on a megapixel silicon camera. Consequently, the hyperspectral imager allows us to acquire 100 spectral bands over 2600-4085 cm−1 in 10 ms, corresponding to a refreshing rate of 100 Hz. Moreover, the angular dependence of phase matching in the image upconversion is leveraged to realize snapshot operation with spatial multiplexing for multiple spectral channels, which may further boost the spectral imaging rate. The high acquisition rate, wide-field operation, and broadband spectral coverage could open new possibilities for high-throughput characterization of transient processes in material and life sciences.

Reconstructing robust background integral field unit spectra using machine learning

ABSTRACT In astronomy, spectroscopy consists of observing an astrophysical source and extracting its spectrum of electromagnetic radiation. Once extracted, a model is fit to the spectra to measure the observables, leading to an understanding of the underlying physics of the emission mechanism. One crucial, and often overlooked, aspect of this model is the background emission, which contains foreground and background astrophysical sources, intervening atmospheric emission, and artefacts related to the instrument such as noise. This paper proposes an algorithmic approach to constructing a background model for SITELLE observations using statistical tools and supervised machine learning algorithms. SITELLE is an imaging Fourier transform spectrometer located at the Canada-France-Hawaii Telescope, which produces a three-dimensional data cube containing the position of the emission (two dimensions) and the spectrum of the emission. SITELLE has a wide field of view (11 arcmin × 11 arcmin), which makes the background emission particularly challenging to model. We apply a segmentation algorithm implemented in photutils to divide the data cube into background and source spaxels. After applying a principal component analysis (PCA) on the background spaxels, we train an artificial neural network to interpolate from the background to the source spaxels in the PCA coefficient space, which allows us to generate a local background model over the entire data cube. We highlight the performance of this methodology by applying it to SITELLE observations obtained of a Star-formation, Ionized Gas and Nebular Abundances Legacy Survey galaxy, NGC 4449, and the Perseus galaxy cluster of galaxies, NGC 1275. We discuss the physical interpretation of the principal components and noise reduction in the resulting PCA-based reconstructions. Additionally, we compare the fit results using our new background modelling approach with standard methods used in the literature and find that our method better captures the emission from H ii regions in NGC 4449 and the faint emission regions in NGC 1275. These methods also demonstrate that the background does change as a function of the position of the data cube. While the approach is applied explicitly to SITELLE data in this study, we argue that it can be readily adapted to any integral field unit style data, enabling the user to obtain more robust measurements on the flux of the emission lines.

SAR time series despeckling via nonlocal matrix decomposition in logarithm domain

Advanced satellite and signal processing techniques have made Synthetic Aperture Radar (SAR) capable of monitoring the dynamic processes of the earth with high temporal frequency through co-registered time series. There are several challenges in interpreting these three-dimensional data cubes, including the inherent speckle effect and outliers caused by abrupt changes or sporadically appearing objects. In this paper, we propose a novel despeckling method, LogSAR-NLMD, based on nonlocal matrix decomposition in the logarithm domain to despeckle SAR time series without producing the artifacts caused by outliers. A maximum a posteriori (MAP) problem is developed by taking into account smoothness and low-rank prior knowledge in addition to the likelihood of SAR reflectivities following the Fisher-Tippett distribution. Additionally, nonlocal self-similarity is exploited to construct time series matrices that satisfy the prior better than the original data. In order to solve the MAP problem, the variable splitting procedure and the alternating direction method of multipliers (ADMM) framework are used. The proposed method has been validated by multiple experiments on synthetic and real data by comparing it with other state-of-the-art methods for despeckling SAR time series.

CTIS-GAN: computed tomography imaging spectrometry based on a generative adversarial network.

Computed tomography imaging spectrometry (CTIS) is a snapshot hyperspectral imaging technique that can obtain a three-dimensional (2D spatial+1D spectral) data cube of the scene captured within a single exposure. The CTIS inversion problem is typically highly ill-posed and is usually solved by time-consuming iterative algorithms. This work aims to take the full advantage of the recent advances in deep-learning algorithms to dramatically reduce the computational cost. For this purpose, a generative adversarial network is developed and integrated with self-attention, which cleverly exploits the clearly utilizable features of zero-order diffraction of CTIS. The proposed network is able to reconstruct a CTIS data cube (containing 31 spectral bands) in milliseconds with a higher quality than traditional methods and the state-of-the-art (SOTA). Simulation studies based on real image data sets confirmed the robustness and efficiency of the method. In numerical experiments with 1000 samples, the average reconstruction time for a single data cube was ∼16m s. The robustness of the method against noise is also confirmed by numerical experiments with different levels of Gaussian noise. The CTIS generative adversarial network framework can be easily extended to solve CTIS problems with larger spatial and spectral dimensions, or migrated to other compressed spectral imaging modalities.

High-speed hyperspectral imaging enabled by compressed sensing in time domain

Hyperspectral imaging (HSI) is a powerful tool widely used for various scientific and industrial applications due to its ability to provide rich spatiospectral information. However, in exchange for multiplex spectral information, its image acquisition rate is lower than that of conventional imaging, with up to a few colors. In particular, HSI in the infrared region and using nonlinear optical processes is impractically slow because the three-dimensional (3D) data cube must be scanned in a point-by-point manner. In this study, we demonstrate a framework to improve the spectral image acquisition rate of HSI by integrating time-domain HSI and compressed sensing. Specifically, we simulated broadband coherent Raman imaging at a record high frame rate of 25 frames per second (fps) with 100 pixels × 100 pixels, which is 10 × faster than that of previous work, based on an experimentally feasible sampling scheme utilizing 3D Lissajous scanning.

Detecting corporate financial fraud via two-stage mapping in joint temporal and financial feature domain

Corporate financial fraud (CFF) has great harmful influences on firms, stakeholders, and capital markets. For CFF detection, firm-year or firm-quarter data are frequently utilized to construct financial feature variables and they have been proved to be effective. However, the firm-year or firm-quarter data are sampled at a single time step only, and the information in the temporal domain has not been fully exploited. As such, we model the corporate financial data into a three-dimensional (3-D) data cube that incorporates both temporal and financial feature domain information. This information can be jointly processed by a two-stage mapping scheme. The first stage maps the joint domain data into discrete sparsity patterns, and the second stage learn some high-level low-dimensional feature representation from the first stage output. The proposed two-stage mapping scheme is tested with real financial data obtained from China Stock Market and Accounting Research (CSMAR) database, and is found to work well. We find that the detection performance is dramatically improved when the accumulated number of quarters increased, and a 3-year, i.e., 12-quarter processing interval will be enough for CFF detection. This finding is important because it is intuitive and similar joint processing interval can be found on other data set. We also find out that joint temporal and financial feature domain processing is helpful for CFF detection. Furthermore, the proposed two-stage mapping scheme for joint processing in temporal and financial feature domain is intuitive for developing intelligent CFF detection systems.

Bearing Fault Diagnosis Based on Mel Frequency Cepstrum Coefficient and Deformable Space-Frequency Attention Network

The main bearing is the core component of gas-fired generator, and its reliability directly affects the stability of the whole system. Therefore, it is of great significance to study the fault diagnosis of the main bearing of gas-fired generator. In the bearing fault diagnosis based on vibration signal, how to extract the signature features of fault effectively is the key to achieving accurate fault diagnosis. Based on extracting the signature features of faults, how to classify the fault features efficiently is another key to achieving accurate fault diagnosis. Based on this, we propose a bearing fault diagnosis method based on Mel frequency cepstrum coefficient (MFCC) and deformable space-frequency attention network (DSFAN). In view of the inconsistent feature distribution of different types of faults, the MFCC algorithm is introduced to preprocess the original fault signals and extract their signature features. Then, the network model DSFAN is constructed based on the space-frequency feature attention mechanism (SFFAM). DSFAN can extract the global constraint features and distributed constraint features of fault signals and realize bearing fault diagnosis. To make full use of classification information, the data processed by MFCC is constructed into a three-dimensional data cube as the input of DSFAN. Finally, the validity of the proposed method MFCC-DSFAN is verified on CWRU, XJTU, and gas-fired generator data sets. The experimental results show the excellent performance of MFCC-DSFAN for fault diagnosis and prove the effectiveness of the attention module in feature extraction.

Tomographic imaging for solar extreme ultraviolet He II 30.4 nm and spectral data inversion

Highly dynamic solar eruptive activities occurring over the corona and transition region, triggered off by magnetic field reconnection, are the driving source of disastrous space weather, and the space imaging and spectroscopic measurements of solar eruptive activities are a key data source for accurate space weather forecasting. The He II 30.4 nm resonance line comes from the Lyman <i>α</i> transition of singly ionized helium, which has an anomalous intensity, an order of magnitude higher than the intensities of other transition region lines. In this paper, we propose and design a two-dimensional spectroscopic tomographic imaging instrument operating at He II 30.4 nm wavelength to make up for the shortcomings of conventional solar extreme ultraviolet imager and imaging spectrometer, and adopt a slitless three-order (–1, 0, +1) simultaneous diffraction imaging configuration with a single snapshot to achieve two-dimensional spectroscopy instantaneous imaging with a large field of view. Owing to the confusion of spatial and spectral information of the three orders of images, the three-dimensional data cube <i>I</i> (<i>x</i>, <i>y</i>, <i>λ</i>) of the observed target is reconstructed using a spectral data inversion algorithm with a limited tomographic projection angle.

Recovery and analysis of rest-frame UV emission lines in 2052 galaxies observed with MUSE at 1.5 <z< 6.4

Rest-frame ultraviolet (UV) emission lines probe electron densities, gas-phase abundances, metallicities, and ionization parameters of the emitting star-forming galaxies and their environments. The strongest main UV emission line, Lyα, has been instrumental in advancing the general knowledge of galaxy formation in the early universe. However, observing Lyαemission becomes increasingly challenging atz ≳ 6 when the neutral hydrogen fraction of the circumgalactic and intergalactic media increases. Secondary weaker UV emission lines provide important alternative methods for studying galaxy properties at high redshift. We present a large sample of rest-frame UV emission line sources at intermediate redshift for calibrating and exploring the connection between secondary UV lines and the emitting galaxies’ physical properties and their Lyαemission. The sample of 2052 emission line sources with 1.5 < z < 6.4 was collected from integral field data from the MUSE-Wide and MUSE-Deep surveys taken as part of Guaranteed Time Observations. The objects were selected through untargeted source detection (i.e., no preselection of sources as in dedicated spectroscopic campaigns) in the three-dimensional MUSE data cubes. We searched optimally extracted one-dimensional spectra of the full sample for UV emission features via emission line template matching, resulting in a sample of more than 100 rest-frame UV emission line detections. We show that the detection efficiency of (non-Lyα) UV emission lines increases with survey depth, and that the emission line strength of He IIλ1640 Å, [O III]λ1661 + O III]λ1666, and [Si III]λ1883 + Si III]λ1892 correlate with the strength of [C III]λ1907 + C III]λ1909. The rest-frame equivalent width (EW0) of [C III]λ1907 + C III]λ1909 is found to be roughly 0.22 ± 0.18 of EW0(Lyα). We measured the velocity offsets of resonant emission lines with respect to systemic tracers. For C IVλ1548 + C IVλ1551 we find that ΔvC IV≲ 250 km s−1, whereas ΔvLyαfalls in the range of 250−500 km s−1which is in agreement with previous results from the literature. The electron densitynemeasured from [Si III]λ1883 + Si III]λ1892 and [C III]λ1907 + C III]λ1909 line flux ratios is generally < 105cm−3and the gas-phase abundance is below solar at 12 + log10(O/H)≈8. Lastly, we used “PhotoIonization Model Probability Density Functions” to infer physical parameters of the full sample and individual systems based on photoionization model parameter grids and observational constraints from our UV emission line searches. This reveals that the UV line emitters generally have ionization parameter log10(U) ≈ −2.5 and metal mass fractions that scatter aroundZ ≈ 10−2, that isZ ≈ 0.66 Z⊙. Value-added catalogs of the full sample of MUSE objects studied in this work and a collection of UV line emitters from the literature are provided with this paper.

Latent fingerprint residue detection method using Sagnac Fourier transform imaging spectroscopy.

We propose a new method to detect latent fingerprints and their residues based on Sagnac ultraviolet Fourier transform imaging spectroscopy. The three-dimensional data cube including two-dimensional images and spectrum dimensions can be obtained by the new hyperspectral imaging technique. The method to inhibit the redundancy from the spectra-image data is also presented, which includes the self-adaptive differential filtering, the apodization algorithm, and a fast Fourier transform method. The whole process is also discussed in detail. Not only the latent fingerprint but also its residues' distribution are provided in experimental results, and the proposed method is demonstrated.

Computed tomography imaging spectrometry based on superiorization and guided image filtering.

Computed tomography imaging spectrometry (CTIS) is a snapshot hyperspectral imaging technique that can obtain a three-dimensional (${2D +}\lambda$) data cube of the target scene within a single exposure. Previous studies of CTIS suggest that reconstructions usually suffer from severe artifacts due to the limited number of projections available. To overcome this limitation, an iterative algorithm combining superiorization and guided image filtering is proposed to explore the intrinsic properties of the hyperspectral data cube as well as the characteristics of zero-order diffraction for the first time, to the best of our knowledge. Results from both simulative studies and proof-of-concept experiments demonstrate its superiority in suppressing artifacts and improving precision over the frequently used expectation maximization algorithm.

Multilayer Huygens’ metasurface absorber toward snapshot multispectral imaging

Multispectral or hyperspectral imaging requires the collection of a three-dimensional (3D) data cube that is composed of 2D spatial information and 1D spectral information of the scene. However, conventional cameras can only record 2D information, such that to obtain the 3D data cube, one needs to sacrifice either spatial or spectral resolution, or spatial or spectral scanning is required. Here, we introduce a multilayer metasurface absorber platform that may potentially allow the collection of a multispectral data cube in a single shot, without sacrificing the resolution. We design each absorber layer composed of silicon nano-resonators to support a Huygens’ mode in the visible spectrum, such that each layer may detect an impinging scene only within a narrow spectral band. We envision that the multilayer metasurface absorber, if integrated with proper electronics, may be used for snapshot multispectral imaging, with potential applications in target identification with high spatial and spectral resolution.

Microlens array snapshot hyperspectral microscopy system for the biomedical domain.

We propose a microlens array-type snapshot hyperspectral microscope system that can provide spatial spectrum sampling according to detector frame rates for the biomedical domain. The system uses a shared optical path design. One path is used to perform direct microscopic imaging with high spatial resolution, while the other is used to collect microscopic images through a microlens array; the images are then spatially cut and reimaged such that they are spaced simultaneously by the prism-grating type hyperspectral imager's dispersion. Rapid acquisition of a three-dimensional data cube measuring 28×14×180 (x×y×λ) can be performed at the detector's frame rate. The system has a spatial resolution of 2.5 µm and can achieve 180-channel sampling of a 100 nm spectrum in the 400-800 nm spectral range with spectral resolution of approximately 0.56 nm. Spectral imaging results from biological samples show that the microlens array-type snapshot hyperspectral microscope system may potentially be applied in real-time biological spectral imaging.

Acquisition and Analysis of Hyperspectral Thermal Images for Sample Segregation.

This study presents the first results of a new type of hyperspectral imager in the long-wave thermal radiation range from 8.0 to 14.0 µm which is simpler than readily available Fourier transform infrared spectroscopy-based imagers. Conventional thermography images the thermal radiation from hot objects, but an accurate determination of temperature is hampered by the often unknown emissivities of different materials present in the same image. This paper describes the setup and development of a hyperspectral thermal camera based on a low-order scanning Fabry-Pérot interferometer acting as a bandpass filter. A three-dimensional hyperspectral data cube (two spatial and one spectral dimension) was measured by imaging a high-emissivity carbon nanotube-coated surface (Vantablack), black painted aluminum, borosilicate glass, Kapton tape, and bare aluminum. A principal component analysis (PCA) of the hyperspectral thermal image clearly segregates the individual samples. The most distinguishable sample from the PCA is the borosilicate Petri dish of which the Si-O-Si bond in borosilicate glass was the most noticeable. Additionally, it was found that the relatively large 1024 × 768 × 70 data cube can be reduced to a much smaller cube of size 1024 × 768 × 5 containing 92% of the variance in the original dataset. The possibility of discriminating between the samples by their spectroscopic signature was tested using a logistic regression classifier. The model was fitted to a chosen set of principal components obtained from a PCA of the original hyperspectral data cube. The model was used to predict all pixels in the original data cube resulting in estimates with very high true positive rate (TPR). The highest TPR was obtained for borosilicate glass with a value of 99% correctly predicted pixels. The remaining TPRs were 94% for black painted aluminum, 81% for bare aluminum, 79% for Kapton tape, and 70% for Vantablack. A standard thermographic image was acquired of the same objects where it was found that the samples were mutually indistinguishable in this image. This shows that the hyperspectral thermal image contains sample characteristics which are material related and therefore outperforms standard thermography in the amount of information contained in an image.

JL-GFDN: A Novel Gabor Filter-Based Deep Network Using Joint Spectral-Spatial Local Binary Pattern for Hyperspectral Image Classification

The traditional local binary pattern (LBP, hereinafter we also call it a two-dimensional local binary pattern 2D-LBP) is unable to depict the spectral characteristics of a hyperspectral image (HSI). To cure this deficiency, this paper develops a joint spectral-spatial 2D-LBP feature (J2D-LBP) by averaging three different 2D-LBP features in a three-dimensional hyperspectral data cube. Subsequently, J2D-LBP is added into the Gabor filter-based deep network (GFDN), and then a novel classification method JL-GFDN is proposed. Different from the original GFDN framework, JL-GFDN further fuses the spectral and spatial features together for HSI classification. Three real data sets are adopted to evaluate the effectiveness of JL-GFDN, and the experimental results verify that (i) JL-GFDN has a better classification accuracy than the original GFDN; (ii) J2D-LBP is more effective in HSI classification in comparison with the traditional 2D-LBP.

Optimization modulation method for 3D spectral data cube using linear encoding of intrinsic chromatic aberration

We develop herein a new optimization modulation method for the three-dimensional (3D) spectral data cube (SDC) based on the use of a linear encoding scheme of the intrinsic chromatic aberration, especially for snapshot multispectral or hyperspectral imaging systems. A mathematical model of the linear chromatic aberration modulation is established that provides theoretical guidelines on the optimal design of 3D SDC for snapshot spectral imaging systems. Spectral simulations and experiments of an optical system are conducted and are shown to match closely, thus demonstrating that the proposed linear encoding scheme can be used as a valid modulation method for 3D SDC. The feasibility of the proposed method for snapshot spectral imaging is verified by the spectral imaging experiments of a polychromatic slit with five wavelengths of 470 nm, 516 nm, 550 nm, 570 nm, 610 nm. The results of the recovered images also confirm that the mutual restriction of spectral and spatial resolutions is addressed. Overall, it is expected for the proposed modulation method for 3D SDC to create new possibilities for high-resolution, compact spectral imaging system with low-cost.

Fluorescence fingerprints of oral bacteria.

The rapid detection and identification of microorganisms is one of the most important factors in many cases of ill health. The purpose of this study was to determine the fluorescence characteristics of seven oral bacteria using emission spectra with the aim of distinguishing between the bacteria, and to compare fluorescence imaging methods for the direct assessment of oral bacteria. Fluorescence images of each bacterium were obtained under a 405-nm light source using a two-filter system. The emissions of all samples were measured with a fluorescence spectrometer. The complete fluorescence data set collected for each sample employed a three-dimensional data cube. The differences in the autofluorescence characteristics of the seven oral bacteria were determined by principal components analysis (PCA). The fluorescence images of the oral bacteria varied with the genus and the filter system. The three-dimensional excitation-emission matrix fluorescence spectra exhibited distinctive fluorescence features associated with intracellular fluorophores. The seven bacteria could be clearly differentiated on the PCA score plot. The findings of this study indicate that oral bacteria can be identified based on their autofluorescence characteristics. Fluorescence spectroscopy coupled with PCA can be used to detect and classify oral bacteria.