Dictionary Pruning Research Articles

Authenticated and Prunable Dictionary for Blockchain-Based VNF Management

Network function virtualization is a key enabling technology in future wireless networks for flexible and efficient sharing of network resources. Due to the increasing heterogeneity of network resource providers, a blockchain-based distributed architecture is a promising solution to enable reliable and transparent virtualized network function (VNF) management. However, since on-chain storage and computation are costive, it becomes a challenging task to achieve efficient VNF management with blockchain. In this paper, we first introduce a consortium blockchain for collaborative VNF management among network resource providers. Then, we propose an authenticated VNF dictionary that can be stored as a succinct authenticator on blockchain to support rich VNF query functionalities and efficient verifications of query results. Moreover, we design a dictionary pruning strategy to securely generate a compact authenticator for a given query, which reduces unnecessary memory accesses of the original dictionary when VNF queries are represented as arithmetic circuits. Finally, we conduct extensive experiments with a consortium blockchain network. The experimental results demonstrate that our pruning strategy is efficient for both on-chain and off-chain VNF management.

Hyperspectral sparse unmixing based on multiple dictionary pruning

ABSTRACT When a spectral library is known, hyperspectral sparse unmixing could obtain the abundance images, which estimate the fractional proportions in each pixel in a hyperspectral image scene. Dictionary pruning (DP) methods furtherly improve the performance of the hyperspectral sparse unmixing by reducing the spectral dictionary to a smaller subset. However, the current hyperspectral sparse unmixing algorithms usually only use a single DP method to obtain the subset, which affects the sparse unmixing results. In this study, we propose a multiple dictionary pruning (MDP) method to improve the performance of hyperspectral sparse unmixing algorithms, making them more accurate. MDP consists of three DP methods, namely, spectral angle mapping (SAM), spectral information divergence (SID) and robust multiple signal classification (RMUSIC). In MDP, we first make a difference between SAM and SID to produce the first level subset named as SS, reducing the impact of non-target information spectrums on the accuracy of sparse unmixing. We then use the RMUSIC to handle the SS to obtain the second level subset named as SSTR, reducing the endmember variability and calibration errors. Finally, we introduce the more accurate subset SSTR into the current hyperspectral sparse unmixing algorithms to produce the sparse unmixing results. Experimental results on two simulated data sets and one real data set show that the proposed MDP method is superior to the traditional DP methods.

Total Utility Metric Based Dictionary Pruning for Sparse Hyperspectral Unmixing

Given a spectral library, sparse unmixing aims to estimate the fractional proportions in each pixel of a hyperspectral image scene. However, the ever-growing dimensionality of spectral dictionaries strongly limits the performance of sparse unmixing algorithms. In this study, we propose a novel dictionary pruning (DP) approach to improve the performance of sparse unmixing algorithms, making them more accurate and time-efficient. We quantify the relative importance of each spectral dictionary atom using the total utility metric at virtually no cost. In this way, we have quantitative insights into how well the elements in the dictionary represent the hyperspectral scene. We evaluate the performance of the proposed dictionary pruning approach on several simulated data sets and one real data. We also compare the experimental results with two well-known dictionary pruning methods both visually and quantitatively and demonstrate the superiority of our proposed method through extensive experimental analysis.

Hyperspectral Unmixing Via Nonconvex Sparse and Low-Rank Constraint

In recent years, sparse unmixing has attracted significant attention, as it can effectively avoid the bottleneck problems associated with the absence of pure pixels and the estimation of the number of endmembers in hyperspectral scenes. The joint-sparsity model has outperformed the single sparse unmixing method. However, the joint-sparsity model might cause some aliasing artifacts for the pixels on the boundaries of different constituent endmembers. To address this shortcoming, researchers have developed many unmixing algorithms based on low-rank representation, which makes good use of the global structure of data. In addition, the high mutual coherence of spectral libraries strongly affects the applicability of sparse unmixing. In this study, adopting combined constraints imposing sparsity and low rankness, a novel algorithm called nonconvex joint-sparsity and low-rank unmixing with dictionary pruning is developed In particular, we impose sparsity on the abundance matrix using the $\ell _{2,p}$ mixed norm, and we also employ the weighted Schatten $p$ -norm instead of the convex nuclear norm as an approximation for the rank. The key parameter $p$ is set between 0.4 and 0.6, and a good quality sparse solution is generated. The effectiveness of the proposed algorithm is demonstrated on both simulated and real hyperspectral datasets.

Region-Based Multiview Sparse Hyperspectral Unmixing Incorporating Spectral Library

Hyperspectral image (HSI) is characterized by its huge contiguous set of wavelengths. It is possible and needed to benefit from the “hyper” spectral information as well as the spatial information. For this purpose, we propose a new multiview data generation approach that takes full advantage of the rich spectral and spatial information in HSI, by dividing the original HSI into several spatially homogeneous regions with different band margins. Then, a new sparse unmixing algorithm, called region-based multiview sparse unmixing (RMSU), is presented to tackle such a multiview data model in this letter. The RMSU algorithm combines the multiview learning and a priori information to improve the performance of sparse unmixing by incorporating the multiview information and spectral library into the dictionary learning framework. We also show that RMSU can serve as a dictionary pruning algorithm, which provides a possibility that unmixing algorithms could have higher accuracy and efficiency. Experimental results on both simulated and real hyperspectral data demonstrate the effectiveness of the proposed RMSU algorithm both visually and quantitatively.

Covariance Similarity Approach for Semiblind Unmixing of Hyperspectral Image

Hyperspectral sparse unmixing methods estimate abundance of endmembers, assuming spectral library as an overcomplete set of endmembers. In this letter, we present a novel, fast and efficient dictionary pruning approach for hyperspectral unmixing. We quantify the change in the latent structure of data due to augmentation of spectral library element using covariance similarity measure. Since the covariance matrices form a nonlinear manifold, choosing an appropriate similarity measure is a nontrivial task. We explored prevalent similarity measures, which motivated us to employ Jeffrey’s Kullback–Leibler divergence measure due to its tighter bounds and better noise performance. We also present analytical formulations for faster implementation of covariance similarity. We evaluate the performance of dictionary pruning algorithms on several synthetic and real hyperspectral images and demonstrate the proficiency of our proposed work in diverse scenarios.

A novel joint dictionary framework for sparse hyperspectral unmixing incorporating spectral library

The dictionary-aided sparse representation has recently become a promising method in hyperspectral unmixing. Under the assumption of linear mixture model, sparse unmixing aims at selecting a small subset of spectral signals in the spectral library known in advance to represent the whole hyperspectral image. Unfortunately, the high mutual coherence of spectral libraries, along with their ever-growing dimensionality, strongly limits the performance of sparse unmixing. For this purpose, we propose a joint dictionary framework for sparse unmixing (JDSU) to tackle such a limitation in this paper. The proposed approach combines the learning capacity and priori information to improve the performance of sparse unmixing by incorporating the spectral library into the dictionary learning method. A multi-dictionary learning model has been developed based on cluster analysis, and it takes advantage of the collaborative effect of endmembers in the local hyperspectral image to learn several local dictionaries which comprise the joint dictionary. Moreover, JDSU can act as a dictionary pruning algorithm which provides a possibility that sparse unmixing algorithms could have higher accuracy and efficiency. Experimental results illustrate that, under certain conditions, JDSU can recover the optimal endmembers from the spectral library. The effectiveness of the proposed approach is extensively validated on both simulated and real hyperspectral data sets.

Fast Semi-Supervised Unmixing of Hyperspectral Image by Mutual Coherence Reduction and Recursive PCA

Dictionary pruning step is often employed prior to the sparse unmixing process to improve the performance of library aided unmixing. This paper presents a novel recursive PCA approach for dictionary pruning of linearly mixed hyperspectral data motivated by the low-rank structure of a linearly mixed hyperspectral image. Further, we propose a mutual coherence reduction method for pre-unmixing to enhance the performance of pruning. In the pruning step we, identify the actual image endmembers utilizing the low-rank constraint. We obtain an augmented version of the data by appending each image endmember and compute PCA reconstruction error, which is a convex surrogate of matrix rank. We identify the pruned library elements according to PCA reconstruction error ratio (PRER) and PCA reconstruction error difference (PRED) and employ a recursive formulation for repeated PCA computation. Our proposed formulation identifies the exact endmember set at an affordable computational requirement. Extensive simulated and real image experiments exhibit the efficacy of the proposed algorithm in terms of its accuracy, computational complexity and noise performance.

Sparse Unmixing With Dictionary Pruning for Hyperspectral Change Detection

The localization of changes that occur between the images in a multitemporal series is crucial for many applications, ranging from environmental monitoring to military surveillance. In contrast to traditional change detection methods, unmixing-based change detection has been shown to have the important added benefit of providing subpixel-level information on the nature of the changes, instead of only providing the location of the changes. Recently, sparse unmixing has also been introduced to hyperspectral change detection, resulting in a method that circumvents the drawbacks of regular spectral unmixing approaches. Sparse unmixing-based change detection reveals the changes that occur in a multitemporal series, at subpixel level, and in terms of the library spectra and their sparse abundances, and provides enhanced change detection performance, especially when subpixel-level changes have occurred. However, sparse unmixing is generally an ill-conditioned and time-consuming process, especially as the size of the utilized spectral library increases. In this paper, dictionary pruning is exploited for the first time for hyperspectral change detection using sparse unmixing, in order to alleviate the ill-conditioning of the problem and achieve decreased computation times and enhanced change detection performance. Experimental results on both realistic synthetic and real datasets are used to validate the proposed approach.

Semiblind Hyperspectral Unmixing in the Presence of Spectral Library Mismatches

The dictionary-aided sparse regression (SR) approach has recently emerged as a promising alternative to hyperspectral unmixing (HU) in remote sensing. By using an available spectral library as a dictionary, the SR approach identifies the underlying materials in a given hyperspectral image by selecting a small subset of spectral samples in the dictionary to represent the whole image. A drawback with the current SR developments is that an actual spectral signature in the scene is often assumed to have zero mismatch with its corresponding dictionary sample, and such an assumption is considered too ideal in practice. In this paper, we tackle the spectral signature mismatch problem by proposing a dictionary-adjusted nonconvex sparsity-encouraging regression (DANSER) framework. The main idea is to incorporate dictionary correcting variables in an SR formulation. A simple and low per-iteration complexity algorithm is tailor-designed for practical realization of DANSER. Using the same dictionary correcting idea, we also propose a robust subspace solution for dictionary pruning. Extensive simulations and real-data experiments show that the proposed method is effective in mitigating the undesirable spectral signature mismatch effects.

Correlation Based Online Dictionary Learning Algorithm

The goal of dictionary learning algorithms is to learn a set of atoms called dictionary from a set of training data such that each training data can be represented sparsely by the dictionary. Most dictionary learning algorithms use two alternating steps, sparse coding and dictionary update, to solve this problem. In this paper, we propose a new online dictionary learning algorithm with a novel dictionary update step. In the new update method, only the atoms involved in the sparse representation of new training data are adaptively updated. The adaptive update also includes using the previous training data that have common atoms with the new training data in their sparse representation. The proposed algorithm reduces the computational complexity by reducing the number of atoms updating at each iteration and the number of training data contributing in the dictionary update. In the proposed algorithm, different weights are given to training data, so each has the proper level of influence in the dictionary update. The simulation results show this improves the performance of the algorithm, in convergence speed and representation error. We also show that our algorithm has the ability of full recovery of dictionary with only one trial over training data and does not need any dictionary pruning to remove the unused atoms of the dictionary. We compare our proposed algorithm with well-known online and batch based methods using synthetic and autoregressive data.

A Dynamic Unmixing Framework for Plant Production System Monitoring

Hyperspectral remote sensing or imaging spectroscopy is an emerging technology in plant production monitoring and management. The continuous reflectance spectra allow for the intensive monitoring of biophysical and biochemical tree characteristics during growth, through for instance the use of vegetation indices. Yet, since most of the pixels in hyperspectral images are mixed, the evaluation of the actual vegetation state on the ground directly from the measured spectra is degraded by the presence of other endmembers, such as soil. Spectral unmixing, then, becomes a necessary processing step to improve the interpretation of vegetation indices. In this sense, an active research direction is based on the use of large collections of pure spectra, called spectral libraries or dictionaries, which model a wide variety of possible states of the endmembers of interest on the ground, i.e., vegetation and soil. Under the linear mixing model (LMM), the observed spectra are assumed to be linear combinations of spectra from the available dictionary. Combinatorial techniques (e.g., MESMA) and sparse regression algorithms (e.g., SUnSAL) are widely used to tackle the unmixing problem in this case. However, both combinatorial and sparse techniques benefit from appropriate library reduction strategies. In this paper, we develop a new efficient method for library reduction (or dictionary pruning), which exploits the fact that hyperspectral data generally lives in a lower-dimensional subspace. Specifically, we present a slight modification of the MUSIC-CSR algorithm, a two-step method which aims first at pruning the dictionary and second at infering high-quality reconstruction of the vegetation spectra on the ground (this application being called signal unmixing in remote sensing), using the pruned dictionary as input to available unmixing methods. Our goal is two-fold: 1) to obtain high-accuracy unmixing output using sparse unmixing, with low-execution time; and 2) to improve MESMA performances in terms of accuracy. Our experiments, which have been conducted in a multi-temporal case study, show that the method achieves these two goals and proposes sparse unmixing as a reliable and robust alternative to the combinatorial methods in plant production monitoring applications. We further demonstrate that the proposed methodology of combining a library pruning approach with spectral unmixing provides a solid framework for the year-round monitoring of plant production systems.

Subspace Matching Pursuit for Sparse Unmixing of Hyperspectral Data

Sparse unmixing assumes that each mixed pixel in the hyperspectral image can be expressed as a linear combination of only a few spectra (endmembers) in a spectral library, known a priori. It then aims at estimating the fractional abundances of these endmembers in the scene. Unfortunately, because of the usually high correlation of the spectral library, the sparse unmixing problem still remains a great challenge. Moreover, most related work focuses on the l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> convex relaxation methods, and little attention has been paid to the use of simultaneous sparse representation via greedy algorithms (GAs) (SGA) for sparse unmixing. SGA has advantages such as that it can get an approximate solution for the l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> problem directly without smoothing the penalty term in a low computational complexity as well as exploit the spatial information of the hyperspectral data. Thus, it is necessary to explore the potential of using such algorithms for sparse unmixing. Inspired by the existing SGA methods, this paper presents a novel GA termed subspace matching pursuit (SMP) for sparse unmixing of hyperspectral data. SMP makes use of the low-degree mixed pixels in the hyperspectral image to iteratively find a subspace to reconstruct the hyperspectral data. It is proved that, under certain conditions, SMP can recover the optimal endmembers from the spectral library. Moreover, SMP can serve as a dictionary pruning algorithm. Thus, it can boost other sparse unmixing algorithms, making them more accurate and time efficient. Experimental results on both synthetic and real data demonstrate the efficacy of the proposed algorithm.

Multi-stage image denoising based on correlation coefficient matching and sparse dictionary pruning

We present a novel image denoising method based on multiscale sparse representations. In tackling the conflicting problems of structure extraction and artifact suppression, we introduce a correlation coefficient matching criterion for sparse coding so as to extract more meaningful structures from the noisy image. On the other hand, we propose a dictionary pruning method to suppress noise. Based on the above techniques, an effective dictionary training method is developed. To further improve the denoising performance, we propose a multi-stage sparse coding framework where sparse representations are obtained in different scales to capture multiscale image features for effective denoising. The multi-stage coding scheme not only reduces the computational burden of previous multiscale denoising approaches, but more importantly, it also contributes to artifact suppression. Experimental results show that the proposed method achieves a state-of-the-art denoising performance in terms of both objective and subjective quality and provides significant improvements over other methods at high noise levels.