Temporal Unmixing Research Articles

A Spatial–Temporal Bayesian Deep Image Prior Model for Moderate Resolution Imaging Spectroradiometer Temporal Mixture Analysis

Time-series remote sensing images are important in agricultural monitoring and investigation. However, most time-series data with high temporal resolution have the problem of insufficient spatial resolution which cannot meet the requirement of precision agriculture. The unmixing technique can obtain the object abundances with richer spatial information from the coarse-resolution images. Although the unmixing technique is widely used in hyperspectral data, it is insufficiently researched for time-series data. Temporal unmixing extends spectral unmixing to the time domain from the spectral domain, and describes the temporal characteristics rather than the spectral characteristics of different ground objects. Deep learning (DL) techniques have achieved promising performance for the unmixing problem in recent years, but there are still few studies on temporal mixture analysis (TMA), especially in the application of crop phenological monitoring. This paper presents a novel spatial–temporal deep image prior method based on a Bayesian framework (ST-Bdip), which innovatively combines the knowledge-driven TMA model and the DL-driven model. The normalized difference vegetation index (NDVI) time series of moderate resolution imaging spectroradiometer (MODIS) data is used as the object for TMA, while the extracted seasonal crop signatures and the fractional coverages are perceived as the temporal endmembers (tEMs) and corresponding abundances. The proposed ST-Bdip method mainly includes the following contributions. First, a deep image prior model based on U-Net architecture is designed to efficiently learn the spatial context information, which enhances the representation of abundance modeling compared to the traditional non-negative least squares algorithm. Second, The TMA model is incorporated into the U-Net training process to exploit the knowledge in the forward temporal model effectively. Third, the temporal noise heterogeneity in time-series images is considered in the model optimization process. Specifically, the anisotropic covariance matrix of observations from different time dimensions is modeled as a multivariate Gaussian distribution and incorporated into the calculation of the loss function. Fourth, the "purified means" approach is used to further optimize crop tEMs and the corresponding abundances. Finally, the expectation–maximization (EM) algorithm is designed to solve the maximum a posterior (MAP) problem of the model in the Bayesian framework. Experimental results on three synthetic datasets with different noise levels and two real MODIS datasets demonstrate the superiority of the proposed approach in comparison with seven traditional and advanced unmixing algorithms.

A novel spectral-temporal Bayesian unmixing algorithm with spatial prior for Sentinel-2 time series

ABSTRACT Temporal unmixing of Sentinel-2 time series is very challenging due to the heterogeneity in spatial, spectral and temporal domains. To tackle this problem, we develop an advanced spectral-temporal Bayesian unmixing approach, with the following characteristics. First, a heterogenous noise model is designed to address the noise variation across spectral bands and time frames. Second, a conditional distribution of endmembers is designed to better characterize endmembers in heterogeneous mixed pixels. Third, the spatial prior is used to better exploit the spatial information for enhanced abundance estimates and noise resistance. Last, the proposed Bayesian framework is solved by a block coordinate descent strategy to better estimate endmembers and abundances. Experiments using both synthetic and real Sentinel-2 time series demonstrate that the proposed approach provides improved unmixing result compared with existing methods.

Decontaminate Traces From Fluorescence Calcium Imaging Videos Using Targeted Non-negative Matrix Factorization.

Fluorescence microscopy and genetically encoded calcium indicators help understand brain function by recording large-scale in vivo videos in assorted animal models. Extracting the fluorescent transients that represent active periods of individual neurons is a key step when analyzing imaging videos. Non-specific calcium sources and background adjacent to segmented neurons contaminate the neurons’ temporal traces with false transients. We developed and characterized a novel method, temporal unmixing of calcium traces (TUnCaT), to quickly and accurately unmix the calcium signals of neighboring neurons and background. Our algorithm used background subtraction to remove the false transients caused by background fluctuations, and then applied targeted non-negative matrix factorization to remove the false transients caused by neighboring calcium sources. TUnCaT was more accurate than existing algorithms when processing multiple experimental and simulated datasets. TUnCaT’s speed was faster than or comparable to existing algorithms.

Temporal and spectral unmixing of photoacoustic signals by deep learning.

Improving the imaging speed of multi-parametric photoacoustic microscopy (PAM) is essential to leveraging its impact in biomedicine. However, to avoid temporal overlap, the A-line rate is limited by the acoustic speed in biological tissues to a few megahertz. Moreover, to achieve high-speed PAM of the oxygen saturation of hemoglobin, the stimulated Raman scattering effect in optical fibers has been widely used to generate 558 nm from a commercial 532 nm laser for dual-wavelength excitation. However, the fiber length for effective wavelength conversion is typically short, corresponding to a small time delay that leads to a significant overlap of the A-lines acquired at the two wavelengths. Increasing the fiber length extends the time interval but limits the pulse energy at 558 nm. In this Letter, we report a conditional generative adversarial network-based approach that enables temporal unmixing of photoacoustic A-line signals with an interval as short as ${\sim}{38}\;{\rm ns}$, breaking the physical limit on the A-line rate. Moreover, this deep learning approach allows the use of multi-spectral laser pulses for PAM excitation, addressing the insufficient energy of monochromatic laser pulses. This technique lays the foundation for ultrahigh-speed multi-parametric PAM.

Mapping Understory Invasive Plants in Urban Forests with Spectral and Temporal Unmixing of Landsat Imagery

Successful eradication and management of invasive plants require frequent and accurate maps. Detection of invasive plants is difficult at moderate resolution because target species are often located in the forest understory among other vegetation types, and so produce mixed spectral signatures. Spectral unmixing approaches can help to decompose these spectral mixtures; however, they are typically applied to only one or a few images, and thus neglect phenological variability that may improve invasive species discrimination. We compared two approaches to multiple endmember spectral mixture analysis for detecting Ligustrum sinense in the southeastern United States: the use of temporal signatures of endmembers from full and select-date normalized difference vegetation index time series, and conventional spectral unmixing using a single image date. Our results suggest that using temporal signatures from all available imagery may be a good choice, with minimal impact to achievable accuracy, if a priori information on phenological differences between endmembers is unavailable, or imagery for periods of high phenological difference are unavailable.

Characterization of Reversibly Switchable Fluorescent Proteins in Optoacoustic Imaging.

Reversibly switchable fluorescent proteins (rsFPs) have had a revolutionizing effect on life science imaging due to their contribution to sub-diffraction-resolution optical microscopy (nanoscopy). Initial studies showed that their use as labels could also be highly beneficial for emerging photo- or optoacoustic imaging. It could be shown that their use in optoacoustics (i) strongly improves the imaging contrast-to-noise ratio due to modulation and locked-in detection, (ii) facilitates fluence calibration, affording precise measurements of physiological parameters, and finally (iii) could boost spatial resolution following similar concepts as used for nanoscopy. However, rsFPs show different photophysical behavior in optoacoustics than in optical microscopy because optoacoustics requires pulsed illumination and depends on signal generation via nonradiative energy decay channels. This implies that rsFPs optimized for fluorescence imaging may not be ideal for optoacoustics. Here, we analyze the photophysical behavior of a broad range of rsFPs with optoacoustics and analyze how the experimental factors central to optoacoustic imaging influence the different types of rsFPs. Finally, we discuss how knowledge of the switching behavior can be exploited for various optoacoustic imaging approaches using sophisticated temporal unmixing schemes.

Detection of Decreasing Vegetation Cover Based on Empirical Orthogonal Function and Temporal Unmixing Analysis

Vegetation plays an important role in the energy exchange of the land surface, biogeochemical cycles, and hydrological cycles. MODIS (MODerate-resolution Imaging Spectroradiometer) EVI (Enhanced Vegetation Index) is considered as a quantitative indicator for examining dynamic vegetation changes. This paper applied a new method of integrated empirical orthogonal function (EOF) and temporal unmixing analysis (TUA) to detect the vegetation decreasing cover in Jiangsu Province of China. The empirical orthogonal function (EOF) statistical results provide vegetation decreasing/increasing trend as prior information for temporal unmixing analysis. Temporal unmixing analysis (TUA) results could reveal the dominant spatial distribution of decreasing vegetation. The results showed that decreasing vegetation areas in Jiangsu are distributed in the suburbs and newly constructed areas. For validation, the vegetation’s decreasing cover is revealed by linear spectral mixture from Landsat data in three selected cities. Vegetation decreasing areas pixels are also calculated from land use maps in 2000 and 2010. The accuracy of integrated empirical orthogonal function and temporal unmixing analysis method is about 83.14%. This method can be applied to detect vegetation change in large rapidly urbanizing areas.

Temporal partial unmixing of exotic salt cedar using Landsat time series

ABSTRACTIn the south-western United States, the rapid colonization of exotic salt cedar (Tamarix spp.) along the riparian corridors poses a serious threat to the native plant diversity. Monitoring the salt cedar abundance in diversified riparian ecosystems is critically essential for evaluating the biodiversity loss and developing corresponding restoration strategies. To date, it still remains a significant challenge to estimate the abundance of plant species within a pixel, as many plants share similar spectral signatures. This challenge is even more daunting in species diverse ecosystems due to the difficulty of securing all the endmembers (e.g. plant species) in unmixing procedures. The objective of this study is to introduce a new concept, temporal partial unmixing, to estimate the relative abundance of plant species using Landsat time series. Temporal partial unmixing only requires the signature of the target endmember, but it emphasizes the importance of the temporal dimension of the target signature in estimating the corresponding abundance. Two types of temporal signatures that capture the phenological trajectory of the salt cedar growth were devised in this study. Compared to the traditional partial unmixing method (R2 = 0.35), temporal partial unmixing that exploited the temporal signatures greatly improved the relative abundance estimation accuracy (R2 = 0.56 or 0.49). It is concluded that the proposed temporal partial unmixing, along with the readily available Landsat imagery, offers unprecedented opportunities to monitor the abundance dynamics of plant species, especially in spatially heterogeneous landscapes.

High-contrast imaging of reversibly switchable fluorescent proteins via temporally unmixed multispectral optoacoustic tomography.

Photocontrol of reversibly switchable fluorescent proteins (RSFPs) was used to program optoacoustic signal time courses that were temporally unmixed to increase the proteins' contrast-to-noise-ratios (CNRs) in optoacoustic imaging. In this way, two variants of the RSFP Dronpa with very similar optoacoustic spectra could be readily discriminated in the presence of highly absorbing blood. Addition of temporal unmixing to multispectral optoacoustic tomography (tuMSOT) in conjunction with synthetic or genetically encoded photochromic contrast agents and customized photoswitching schedules can increase the performance of multiplexed and high-contrast molecular optoacoustic imaging.

Temporal Unmixing of Dynamic Fluorescent Images by Blind Source Separation Method with a Convex Framework.

By recording a time series of tomographic images, dynamic fluorescence molecular tomography (FMT) allows exploring perfusion, biodistribution, and pharmacokinetics of labeled substances in vivo. Usually, dynamic tomographic images are first reconstructed frame by frame, and then unmixing based on principle component analysis (PCA) or independent component analysis (ICA) is performed to detect and visualize functional structures with different kinetic patterns. PCA and ICA assume sources are statistically uncorrelated or independent and don't perform well when correlated sources are present. In this paper, we deduce the relationship between the measured imaging data and the kinetic patterns and present a temporal unmixing approach, which is based on nonnegative blind source separation (BSS) method with a convex analysis framework to separate the measured data. The presented method requires no assumption on source independence or zero correlations. Several numerical simulations and phantom experiments are conducted to investigate the performance of the proposed temporal unmixing method. The results indicate that it is feasible to unmix the measured data before the tomographic reconstruction and the BSS based method provides better unmixing quality compared with PCA and ICA.

Estimation of Pure NDVI Seasonal Profiles of Crops by Temporal Unmixing of MODIS Dataset

Estimation of Pure NDVI Seasonal Profiles of Crops by Temporal Unmixing of MODIS Dataset

Blind source unmixing in multi-spectral optoacoustic tomography

Multispectral optoacoustic (photoacoustic) tomography (MSOT) is a hybrid modality that can image through several millimeters to centimeters of diffuse tissues, attaining resolutions typical of ultrasound imaging. The method can further identify tissue biomarkers by decomposing the spectral contributions of different photo-absorbing molecules of interest. In this work we investigate the performance of blind source unmixing methods and spectral fitting approaches in decomposing the contributions of fluorescent dyes from the tissue background, based on MSOT measurements in mice. We find blind unmixing as a promising method for accurate MSOT decomposition, suitable also for spectral unmixing in fluorescence imaging. We further demonstrate its capacity with temporal unmixing on real-time MSOT data obtained in-vivo for enhancing the visualization of absorber agent flow in the mouse vascular system.

Spectral analysis of doxorubicin accumulation and the indirect quantification of its DNA intercalation

There is a wide range of techniques utilizing fluorescence of doxorubicin (Dox) commonly used for analysis of intracellular accumulation and destiny of various drug delivery systems containing this anthracycline antibiotic. Unfortunately, results of these studies can be significantly influenced by doxorubicin degradation product, 7,8-dehydro-9,10-desacetyldoxorubicinone (D*) forming spontaneously in aqueous environment, whose fluorescence strongly interfere with that of doxorubicin. Here, we define two microscopy techniques enabling to distinguish and separate Dox and D* emission based either on its spectral properties or on fluorescence lifetime analysis. To analyze influx and nuclear accumulation of Dox (free or polymer-bound) by flow cytometry, we propose using an indirect method based on its DNA intercalation competition with Hoechst 33342 rather than a direct measurement of doxorubicin fluorescence inside the cells.

The spatial distribution of crop types from MODIS data: Temporal unmixing using Independent Component Analysis

Remote sensing plays an important role in delivering accurate and timely information on the location and area of major crop types with important environmental, economic, and policy considerations. The purpose of this paper is to report on an unsupervised signal processing algorithm called Independent Component Analysis (ICA) to temporally decompose MODerate-resolution Imaging Spectroradiometer (MODIS) data to automatically map major crop types in three agricultural regions covering parts of Kansas and Nebraska in the US and a third in northwestern Turkey. The approach proposed here is based on the premise that the temporal profiles of individual crops are observed as mixtures with a moderate-resolution sensor when cultivated fields are smaller than the spatial resolution of the observing sensor. The purpose of ICA is to decompose these mixed observations by a remote sensor into individual crop signals, using only the mixed observations without the aid of information about the crop signatures and the mixing process. Results using both synthetic data and real observations suggest that the ICA approach can successfully separate generalized, landscape-level cropping patterns using only available temporal measurements. There is very little need to use complicated indices or derivative spectral products to map crop types using ICA: availability of high temporal observations, either as raw spectral bands or simple vegetation indices is sufficient to identify crop types at the scale of landscapes. Results also suggest that crop map predictions aggregated to coarser resolutions have better accuracy than at native resolution when compared to maps made from fine-scale observations used as ground truth. These accuracies range from RMSE of 15–30% at 500 m to less than 10% at 2000 m. The success of the initial results presented here to automatically map crop distributions across large areas using MODIS data is particularly encouraging given the existing and planned worldwide observations of agriculturally important regions. However, the use of ICA for operational crop monitoring will require algorithms that will take into account prior information on crop growth curves, constrained estimation of independent components, and scaling of mixing vectors to obtain physically possible ranges.

Cropland distributions from temporal unmixing of MODIS data

Knowledge of the distribution of crop types is important for land management and trade decisions, and is needed to constrain remotely sensed estimates of variables, such as crop stress and productivity. The Moderate Resolution Imaging Spectroradiometer (MODIS) offers a unique combination of spectral, temporal, and spatial resolution compared to previous global sensors, making it a good candidate for large-scale crop type mapping. However, because of subpixel heterogeneity, the application of traditional hard classification approaches to MODIS data may result in significant errors in crop area estimation. We developed and tested a linear unmixing approach with MODIS that estimates subpixel fractions of crop area based on the temporal signature of reflectance throughout the growing season. In this method, termed probabilistic temporal unmixing (PTU), endmember sets were constructed using Landsat data to identify pure pixels, and uncertainty resulting from endmember variability was quantified using Monte Carlo simulation. This approach was evaluated using Landsat classification maps in two intensive agricultural regions, the Yaqui Valley (YV) of Mexico and the Southern Great Plains (SGP). Performance of the mixture model varied depending on the scale of comparison, with R 2 ranging from roughly 50% for estimating crop area within individual pixels to greater than 80% for crop cover within areas over 10 km 2. The results of this study demonstrate the importance of subpixel heterogeneity in cropland systems, and the potential of temporal unmixing to provide accurate and rapid assessments of land cover distributions using coarse resolution sensors, such as MODIS.