Spectral Ensemble Research Articles

Toward electron temperature profiles in hot-dense plasmas from x-ray spectral ensembles

High repetition rate laser systems enable new strategies for diagnosing plasma behavior with large datasets. Here, we define an ensemble technique that relies on randomized targeting of x-ray tracer micro-stripes. On each shot, a high-intensity laser pulse is focused on a solid target with Ti tracer stripes embedded in an Al foil, randomly targeting a micro-stripe, a portion of a stripe, or a gap between stripes. High-resolution, time-integrated x-ray spectrometers capture line emission from the portion of the micro-stripe that is heated to sufficiently high electron temperatures. Accumulation of many such cases is used to construct ensemble distributions of x-ray line intensities that encompass all relative offsets of the laser focus to the micro-stripe centers. Synthetic intensity distributions are likewise generated using collisional-radiative modeling. Bayesian fitting of modeled to measured intensity distributions establishes the most likely radial temperature profiles, enabling comparison to hydrodynamic models and calling into question the cylindrical symmetry of these micro-stripe-embedded systems. Ensemble techniques have significant potential for high-energy-density plasma diagnostics, especially with the advent of high repetition rate experiments.

Probabilistic Mixture Model-Based Spectral Unmixing

Spectral unmixing attempts to decompose a spectral ensemble into the constituent pure spectral signatures (called endmembers) along with the proportion of each endmember. This is essential for techniques like hyperspectral imaging (HSI) used in environment monitoring, geological exploration, etc. Several spectral unmixing approaches have been proposed, many of which are connected to hyperspectral imaging. However, most extant approaches assume highly diverse collections of mixtures and extremely low-loss spectroscopic measurements. Additionally, current non-Bayesian frameworks do not incorporate the uncertainty inherent in unmixing. We propose a probabilistic inference algorithm that explicitly incorporates noise and uncertainty, enabling us to unmix endmembers in collections of mixtures with limited diversity. We use a Bayesian mixture model to jointly extract endmember spectra and mixing parameters while explicitly modeling observation noise and the resulting inference uncertainties. We obtain approximate distributions over endmember coordinates for each set of observed spectra while remaining robust to inference biases from the lack of pure observations and the presence of non-isotropic Gaussian noise. As a direct impact of our methodology, access to reliable uncertainties on the unmixing solutions would enable robust solutions to noise, as well as informed decision-making for HSI applications and other unmixing problems.

Stacking and ridge regression-based spectral ensemble preprocessing method and its application in near-infrared spectral analysis

Spectral preprocessing techniques can, to a certain extent, eliminate irrelevant information, such as current noise and stray light from spectral data, thereby enhancing the performance of prediction models. However, current preprocessing techniques mostly attempt to find the best single preprocessing method or their combination, overlooking the complementary information among different preprocessing methods. These preprocessing techniques fail to maximize the utilization of useful information in spectral data and restrict the performance of prediction models. This study proposed a spectral ensemble preprocessing method based on the rapidly developing ensemble learning methods in recent years and the ridge regression (RR) model, named stacking preprocessing ridge regression (SPRR), to address the aforementioned issues. Different from conventional ensemble learning methods, the proposed SPRR method applied multiple different preprocessing techniques to the original spectral data, generating multiple preprocessed datasets. These datasets were then individually inputted into RR base models for training. Ultimately, RR still served as the meta-model, integrating the output results of each RR base model through stacking. This approach not only produced diversity in base models but also achieved higher accuracy and lower computational complexity by using a single type of base model. On the apple spectral dataset collected by our team, correlation analysis showed significant complementary information among the data produced by different preprocessing techniques. This provided robust theoretical support for the proposed SPRR method. By introducing the currently popular averaging ensemble preprocessing method in a comparative experiment, the results of applying the proposed SPRR method to six datasets (apple, meat, wheat, olive oil, tablet, and corn) demonstrated that compared to the single preprocessing method and averaging ensemble preprocessing method, SPRR yielded the best accuracy and reliability for all six datasets. Furthermore, under the same conditions of the training and test datasets, the proposed SPRR method demonstrated better performance than the four commonly used ensemble preprocessing methods.

Tensorized Graph Learning for Spectral Ensemble Clustering

Tensorized Graph Learning for Spectral Ensemble Clustering

LiteWSEC: A Lightweight Framework for Web-Scale Spectral Ensemble Clustering

Spectral Clustering (SC) is an effective clustering method for its excellent performance in partitioning non-linearly distributed data. On the other hand, Ensemble Clustering (EC), a different clustering technology, can promote cluster quality by ensembling the results of base clusterings. In this work, we concentrate on an EC framework that utilizes SC as the base method. Nevertheless, SC suffers from scalability due to its high computational complexity in constructing the Laplacian graph and computing the corresponding eigendecomposition. In the past decades, many efforts have been made to it. However, SC suffers from the scalability issue in processing extensive data, especially in web-scale scenarios. Additionally, EC requires multiple clustering results as the ensemble bases, which further aggravates resource consumption. To address this issue, LiteWSEC, a simple yet efficient Lightweight Framework for Web-scale Spectral Ensemble Clustering, is proposed to cluster web-scale data with limited resource requirements. It adopts the Web-scale Spectral Clustering (WSC) as the base method, which has minimal space overhead without computing overall embedding explicitly. LiteWSEC is highly flexible in the memory requirement, which is adaptive to the available resource. It can partition web-scale data (e.g., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n = 8,000~k$</tex-math></inline-formula> ) in an resource-limited host (e.g., memory is restricted to 1 GB). Experiments on real-world, large-scale, and web-scale datasets demonstrate both the efficiency and effectiveness of LiteWSEC over state-of-the-art SC and EC methods.

Time series classification, augmentation and artificial-intelligence-enabled software for emergency response in freight transportation fires

In responding to freight transportation fire incidents, first responders refer to the terials labeled on the freights and the Emergency Response Guidebook (ERG) for guidance on the initial response. However, when the burning goods are mixed or unknown, first responders require support on the appropriate response strategy. In this work, we propose an artificial intelligence (AI) enabled tool to aid first responders in the initial emergency response to fire incidents involving mixed or unknown materials. We used many different machine learning models, including MultiRocket, Omni-Scale CNN (OS-CNN), Ultra-Fast Shapelets (UFA), WEASEL+MUSE, Dynamic Time Warping (DTW), Time Series Forest (TSF), the Random Interval Spectral Ensemble (RISE), and InceptionTime, to develop a trained machine learning (ML) model integrated into the AI tool. The ML model identifies the hazard characteristics (e.g., toxicity and explosivity) of a given fire, based on chemicals found in the effluent. The effluent data was collected from samples of burning materials using a cone calorimeter and a Fourier Transform Infrared Spectroscopy (FTIR) gas analyzer. We discuss the methodologies behind developing the ML model and demonstrate its high classification accuracy with time series augmentations to enhance the training dataset. We also implemented Self-Supervised Learning, Domain-Agnostic Contrastive Learning (DACL), and Adversarial Perturbation-based Latent Construction (APLR) to further improve the model. The results show that MultiRocket outperforms other models in terms of accuracy and running time. Furthermore, we developed a user interface for our AI-enabled tool, which uses a data sample as input to provide not only predictions of the fire hazards but also actionable intelligence to the first responders for safe and effective fire suppression strategies. Our proposed toolkit benefiting from AI will pave the way to further research in accessible, easily-deployable AI-enabled tools in fire sciences and management.Environmental Implications We leverage machine learning to predict the hazard classification of materials that may not be considered hazardous by Emergency Response Guidebook (ERG) used by Transport Canada, but may release toxic substances when burning. For example, polyvinyl chloride is not considered as hazardous by ERG, but toxic and corrosive hydrochloric gas is produced when burning. We aim to provide more decision support to first responders in freight transportation fires involving non-hazardous or unknown materials to identify the scope of the emergency from the early stage of the fire, which is critical to protect the environment from the effects of the fire.

Fuzzy-Rough induced spectral ensemble clustering

Ensemble clustering helps achieve fast clustering under abundant computing resources by constructing multiple base clusterings. Compared with the standard single clustering algorithm, ensemble clustering integrates the advantages of multiple clustering algorithms and has stronger robustness and applicability. Nevertheless, most ensemble clustering algorithms treat each base clustering result equally and ignore the difference of clusters. If a cluster in a base clustering is reliable/unreliable, it should play a critical/uncritical role in the ensemble process. Fuzzy-rough sets offer a high degree of flexibility in enabling the vagueness and imprecision present in real-valued data. In this paper, a novel fuzzy-rough induced spectral ensemble approach is proposed to improve the performance of clustering. Specifically, the significance of clusters is differentiated, and the unacceptable degree and reliability of clusters formed in base clustering are induced based on fuzzy-rough lower approximation. Based on defined cluster reliability, a new co-association matrix is generated to enhance the effect of diverse base clusterings. Finally, a novel consensus spectral function is defined by the constructed adjacency matrix, which can lead to significantly better results. Experimental results confirm that the proposed approach works effectively and outperforms many state-of-the-art ensemble clustering algorithms and base clustering, which illustrates the superiority of the novel algorithm.

LSEC: Large-scale spectral ensemble clustering

A fundamental problem in machine learning is ensemble clustering, that is, combining multiple base clusterings to obtain improved clustering result. However, most of the existing methods are unsuitable for large-scale ensemble clustering tasks owing to efficiency bottlenecks. In this paper, we propose a large-scale spectral ensemble clustering (LSEC) method to balance efficiency and effectiveness. In LSEC, a large-scale spectral clustering-based efficient ensemble generation framework is designed to generate various base clusterings with low computational complexity. Thereafter, all the base clusterings are combined using a bipartite graph partition-based consensus function to obtain improved consensus clustering results. The LSEC method achieves a lower computational complexity than most existing ensemble clustering methods. Experiments conducted on ten large-scale datasets demonstrate the efficiency and effectiveness of the LSEC method. The MATLAB code of the proposed method and experimental datasets are available at https://github.com/Li-Hongmin/MyPaperWithCode.

MEMO: Mass Spectrometry-Based Sample Vectorization to Explore Chemodiverse Datasets.

In natural products research, chemodiverse extracts coming from multiple organisms are explored for novel bioactive molecules, sometimes over extended periods. Samples are usually analyzed by liquid chromatography coupled with fragmentation mass spectrometry to acquire informative mass spectral ensembles. Such data is then exploited to establish relationships among analytes or samples (e.g., via molecular networking) and annotate metabolites. However, the comparison of samples profiled in different batches is challenging with current metabolomics methods since the experimental variation—changes in chromatographical or mass spectrometric conditions - hinders the direct comparison of the profiled samples. Here we introduce MEMO—MS2 BasEd SaMple VectOrization—a method allowing to cluster large amounts of chemodiverse samples based on their LC-MS/MS profiles in a retention time agnostic manner. This method is particularly suited for heterogeneous and chemodiverse sample sets. MEMO demonstrated similar clustering performance as state-of-the-art metrics considering fragmentation spectra. More importantly, such performance was achieved without the requirement of a prior feature alignment step and in a significantly shorter computational time. MEMO thus allows the comparison of vast ensembles of samples, even when analyzed over long periods of time, and on different chromatographic or mass spectrometry platforms. This new addition to the computational metabolomics toolbox should drastically expand the scope of large-scale comparative analysis.

Experimental Assessments on the Evaluation of Wire Rope Characteristics as Helical Symmetrical Multi-body Ensembles

The existing literature provides various computational models related to the dynamic behavior of strand wire ropes. It starts from the simple longitudinally oscillating beam, to the complex nonlinear multi-body configuration based on helical structural symmetry. The challenge is the prior availability of characteristic parameters for material behavior, structural configuration, and functional capability. Experimental investigation is the main source for evaluation of these characteristics. However, tests have specifically been performed according to each case, minimizing the generalization aspect. This is the main frame of this study. Hereby, the authors propose an ensemble of spectral investigations, applied to a reduced set of experimental tests regarding wire rope dynamics. The research goal consists of wire rope characterization in terms of the flexible and adaptive groups of parameters, related to the conservative and dissipative behaviors. An experimental setup is considered here according to the rope exploitation conditions in order to enable an extension of the method application from the experimental mode to the operational mode. Experiments are conducted based on classical vibration measurement procedures. The analysis is performed using a spectral method ensemble, including discrete Fourier transform, time-frequency joint analysis, and the Prony method. The result show that the proposed assessments can provide suitable information related to a large group of wire rope models.

Robust Spectral Ensemble Clustering via Rank Minimization

Ensemble Clustering (EC) is an important topic for data cluster analysis. It targets to integrate multiple Basic Partitions (BPs) of a particular dataset into a consensus partition. Among previous works, one promising and effective way is to transform EC as a graph partitioning problem on the co-association matrix, which is a pair-wise similarity matrix summarized by all the BPs in essence. However, most existing EC methods directly utilize the co-association matrix, yet without considering various noises (e.g., the disagreement between different BPs and the outliers) that may exist in it. These noises can impair the cluster structure of a co-association matrix, and thus mislead the final graph partitioning process. To address this challenge, we propose a novel Robust Spectral Ensemble Clustering (RSEC) algorithm in this article. Specifically, we learn low-rank representation (LRR) for the co-association matrix to uncover its cluster structure and handle the noises, and meanwhile, we perform spectral clustering with the learned representation to seek for a consensus partition. These two steps are jointly proceeded within a unified optimization framework. In particular, during the optimizing process, we leverage consensus partition to iteratively enhance the block-diagonal structure of LRR, in order to assist the graph partitioning. To solve RSEC, we first formulate it by using nuclear norm as a convex proxy to the rank function. Then, motivated by the recent advances in non-convex rank minimization, we further develop a non-convex model for RSEC and provide it a solution by the majorization--minimization Augmented Lagrange Multiplier algorithm. Experiments on 18 real-world datasets demonstrate the effectiveness of our algorithm compared with state-of-the-art methods. Moreover, several impact factors on the clustering performance of our approach are also explored extensively.

Better Baltic Sea wave forecasts: improving resolution or introducing ensembles?

Abstract. The performance of short-range operational forecasts of significant wave height (SWH) in the Baltic Sea is evaluated. Forecasts produced by a base configuration are intercompared with forecasts from two improved configurations: one with improved horizontal and spectral resolution and one with ensembles representing uncertainties in the physics of the forcing wind field and the initial conditions of this field. Both of the improved forecast classes represent an almost equal increase in computational costs. Therefore, the intercomparison addresses the question of whether more computer resources would be more favorably spent on enhancing the spatial and spectral resolution or, alternatively, on introducing ensembles. The intercomparison is based on comparisons with hourly observations of significant wave height from seven observation sites in the Baltic Sea during the 3-year period from 2015 to 2017. We conclude that for most wave measurement sites, the introduction of ensembles enhances the overall performance of the forecasts, whereas increasing the horizontal and spectral resolution does not. These sites represent offshore conditions, in that they are well exposed from all directions, are a large distance from the nearest coast and in deep water. Therefore, there is the a priori expectation that a detailed shoreline and bathymetry will not have any impact. Only at one site do we find that increasing the horizontal and spectral resolution significantly improves the forecasts. This site is situated in nearshore conditions, close to land and a nearby island, and is therefore shielded from many directions. Consequently, this study concludes that to improve wave forecasts in offshore areas, ensembles should be introduced. For near shore areas, in comparison, the study suggests that additional computational resources should be used to increase the resolution.

Spectral Reflectance Estimation Using Gaussian Processes and Combination Kernels.

This paper explores hyperspectral reflectance factor estimation using Gaussian process regression with multispectral- and trichromatic measurements. Estimations are performed in visible- (400-700 nm) or visible-near infrared (400-980 nm) wavelength ranges using the learning-based approach, where sensor and light spectral characteristics are not required. We first construct new estimation models via Gaussian processes, show connection to previous kernel-based models, and then evaluate new models by using marginal likelihood optimization within the probabilistic interpretation. By using standard spectral ensembles and several images in experiments, we evaluate new models with anisotropic radial- and combination kernels (process covariance), marginal likelihood optimization (parameter selection), as well as with input data transformations (pre-processing). Several new Gaussian process models provide spectral accuracy improvements for simulated and real data, when compared with the previous kernel-based models. Most versatile new model is using spectral subspace coordinate learning and combination kernels, and can be efficiently optimized via marginal likelihood. Preliminary results suggest that new models provide uncertainty estimates, which can be used for iterative training set augmentation.

Toward Reliable Lipoprotein Particle Predictions from NMR Spectra of Human Blood: An Interlaboratory Ring Test

Lipoprotein profiling of human blood by 1H nuclear magnetic resonance (NMR) spectroscopy is a rapid and promising approach to monitor health and disease states in medicine and nutrition. However, lack of standardization of measurement protocols has prevented the use of NMR-based lipoprotein profiling in metastudies. In this study, a standardized NMR measurement protocol was applied in a ring test performed across three different laboratories in Europe on plasma and serum samples from 28 individuals. Data was evaluated in terms of (i) spectral differences, (ii) differences in LPD predictions obtained using an existing prediction model, and (iii) agreement of predictions with cholesterol concentrations in high- and low-density lipoproteins (HDL and LDL) particles measured by standardized clinical assays. ANOVA-simultaneous component analysis (ASCA) of the ring test spectral ensemble that contains methylene and methyl peaks (1.4–0.6 ppm) showed that 97.99% of the variance in the data is related to subject, 1.62% to sample type (serum or plasma), and 0.39% to laboratory. This interlaboratory variation is in fact smaller than the maximum acceptable intralaboratory variation on quality control samples. It is also shown that the reproducibility between laboratories is good enough for the LPD predictions to be exchangeable when the standardized NMR measurement protocol is followed. With the successful implementation of this protocol, which results in reproducible prediction of lipoprotein distributions across laboratories, a step is taken toward bringing NMR more into scope of prognostic and diagnostic biomarkers, reducing the need for less efficient methods such as ultracentrifugation or high-performance liquid chromatography (HPLC).

Spectral Ensemble Clustering via Weighted K-Means: Theoretical and Practical Evidence

As a promising way for heterogeneous data analytics, consensus clustering has attracted increasing attention in recent decades. Among various excellent solutions, the co-association matrix based methods form a landmark, which redefines consensus clustering as a graph partition problem. Nevertheless, the relatively high time and space complexities preclude it from wide real-life applications. We, therefore, propose Spectral Ensemble Clustering (SEC) to leverage the advantages of co-association matrix in information integration but run more efficiently. We disclose the theoretical equivalence between SEC and weighted K-means clustering, which dramatically reduces the algorithmic complexity. We also derive the latent consensus function of SEC, which to our best knowledge is the first to bridge co-association matrix based methods to the methods with explicit global objective functions. Further, we prove in theory that SEC holds the robustness, generalizability, and convergence properties. We finally extend SEC to meet the challenge arising from incomplete basic partitions, based on which a row-segmentation scheme for big data clustering is proposed. Experiments on various real-world data sets in both ensemble and multi-view clustering scenarios demonstrate the superiority of SEC to some state-of-the-art methods. In particular, SEC seems to be a promising candidate for big data clustering.

Fast Constrained Spectral Clustering and Cluster Ensemble with Random Projection.

Constrained spectral clustering (CSC) method can greatly improve the clustering accuracy with the incorporation of constraint information into spectral clustering and thus has been paid academic attention widely. In this paper, we propose a fast CSC algorithm via encoding landmark-based graph construction into a new CSC model and applying random sampling to decrease the data size after spectral embedding. Compared with the original model, the new algorithm has the similar results with the increase of its model size asymptotically; compared with the most efficient CSC algorithm known, the new algorithm runs faster and has a wider range of suitable data sets. Meanwhile, a scalable semisupervised cluster ensemble algorithm is also proposed via the combination of our fast CSC algorithm and dimensionality reduction with random projection in the process of spectral ensemble clustering. We demonstrate by presenting theoretical analysis and empirical results that the new cluster ensemble algorithm has advantages in terms of efficiency and effectiveness. Furthermore, the approximate preservation of random projection in clustering accuracy proved in the stage of consensus clustering is also suitable for the weighted k-means clustering and thus gives the theoretical guarantee to this special kind of k-means clustering where each point has its corresponding weight.

Spectral diagonal ensemble Kalman filters

Abstract. A new type of ensemble Kalman filter is developed, which is based on replacing the sample covariance in the analysis step by its diagonal in a spectral basis. It is proved that this technique improves the approximation of the covariance when the covariance itself is diagonal in the spectral basis, as is the case, e.g., for a second-order stationary random field and the Fourier basis. The method is extended by wavelets to the case when the state variables are random fields which are not spatially homogeneous. Efficient implementations by the fast Fourier transform (FFT) and discrete wavelet transform (DWT) are presented for several types of observations, including high-dimensional data given on a part of the domain, such as radar and satellite images. Computational experiments confirm that the method performs well on the Lorenz 96 problem and the shallow water equations with very small ensembles and over multiple analysis cycles.

An Approximate Spectral Clustering Ensemble for High Spatial Resolution Remote-Sensing Images

Unsupervised clustering of high spatial resolution remote-sensing images plays a significant role in detailed land-cover identification, especially for agricultural and environmental monitoring. A recently promising method is approximate spectral clustering (SC) which enables spectral partitioning for large datasets to extract clusters with distinct characteristics without a parametric model. It also facilitates the use of various information types via advanced similarity criteria. However, it requires an empirical selection of a similarity criterion optimal for the corresponding application. To address this challenge, we propose an approximate SC ensemble (ASCE2) which fuses partitionings obtained by different similarity representations. Contrary to existing spectral ensembles for remote-sensing applications, the proposed ASCE2 employs neural gas quantization instead of random sampling, advanced similarity criteria instead of traditional distance-based Gaussian kernel with different decay parameters, and a two-level ensemble. We evaluate the proposed ASCE2 with three measures (accuracy, adjusted Rand index, and normalized mutual information) using five remote-sensing images, two of which are commonly available. We apply the ASCE2 in two applications for agricultural monitoring: 1) land-cover identification to determine orchard fields using a WorldView-2 image (0.5-m spatial resolution) and 2) finding lands in good agricultural condition using multitemporal RapidEye images (5-m spatial resolution). Experimental results indicate a significant betterment of the resulting partitionings obtained by the proposed ensemble, with respect to the evaluation measures in these applications.

Spectral co-clustering ensemble

The goal of co-clustering is to simultaneously cluster the rows and columns of an input data matrix. It overcomes several limitations associated with traditional clustering methods by allowing automatic discovery of similarity based on a subset of attributes. However, different co-clustering models usually produce very distinct results since each algorithm has its own bias due to the optimization of different criteria. The idea of combining different co-clustering results emerged as an alternative approach for improving the performance of co-clustering algorithms. Similar to clustering ensembles, co-clustering ensembles provide a framework for combining multiple base co-clusterings of a dataset to generate a stable and robust consensus co-clustering result. In this paper, a novel co-clustering ensemble algorithm named spectral co-clustering ensemble (SCCE) is presented. SCCE performs ensemble tasks on base row clusters and column clusters of a dataset simultaneously, and obtains an optimization co-clustering result. Meanwhile, SCCE is a matrix decomposition based approach which can be formulated as a bipartite graph partition problem and solve it efficiently with the selected eigenvectors. To the best of our knowledge, this is the first work on using spectral algorithm for co-clustering ensemble. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method. Our study also shows that SCCE has some favorable merits compared with many state of the art methods.

Distribution of the smallest eigenvalue in complex and real correlated Wishart ensembles

For the correlated Gaussian–Wishart ensemble we compute the distribution of the smallest eigenvalue and a related gap probability. We obtain exact results for the complex (β = 2) and for the real case (β = 1). For a particular set of empirical correlation matrices we find universality in the spectral density, for both real and complex ensembles and all kinds of rectangularity. We calculate the asymptotic and universal results for the gap probability and the distribution of the smallest eigenvalue. We use the supersymmetry method, in particular the generalized Hubbard–Stratonovich transformation and superbosonization.