Sparse Detection Research Articles

Machine Learning Evaluation of Fruit Ripeness with Multidimensional Sparse-Dataset Calibration

HighlightsThis article provides a machine-learning method for estimation of sugar content in target fruit using color appearance.Complex networks based on multidimensional datasets work effectively for color sensor calibration and data rebalancing.Beyond datapoint sparsity in datasets, estimation accuracy in sugar content is assured when an unsupervised-learning algorithm classifies the multidimensional datapoints composed of fruit color and relevant quantities.Abstract. In agriculture, harvests of fruits and vegetables depend on their ripeness, in which farmers should rely not only on the eye-sensed outlook but rigorously on constituents inside the targets, like sugar content. So far, detail measurements on the outlook color of fruits have been widely reported to detect suitable harvest times, where non-destructive instruments, which are costly in general agriculture fields, work well for color estimation whose resolution level is much higher than human eyes. In this study, we propose a scheme with a calibration procedure for sugar content prediction from outlook-color quantities, which is applicable even if a sensor includes systematic errors in output signals or the base dataset includes some datapoint sparsity. Most multidimensional data, which are both in agriculture and in other general cases, are imperfect in terms of several aspects, but they can be useful when we take appropriate sparsity detection and setting of rebalancing weights into account for machine learning procedures. We newly introduce a complex-network method for this purpose, and after the color calibration, we successfully obtain accurate hue and chroma, which are quantities representing a position on calibrated color coordinate, to predict °Brix sugar content in Japanese pears. Keywords: Asian pear, Color Elements, Fruit Ripening, °Brix Sugar Content, OHD.

Distributed Quantized Detection of Sparse Signals Under Byzantine Attacks

Distributed Quantized Detection of Sparse Signals Under Byzantine Attacks

Minimax rates for sparse signal detection under correlation

Abstract We fully characterize the nonasymptotic minimax separation rate for sparse signal detection in the Gaussian sequence model with $p$ equicorrelated observations, generalizing a result of Collier, Comminges and Tsybakov. As a consequence of the rate characterization, we find that strong correlation is a blessing, moderate correlation is a curse and weak correlation is irrelevant. Moreover, the threshold correlation level yielding a blessing exhibits phase transitions at the $\sqrt{p}$ and $p-\sqrt{p}$ sparsity levels. We also establish the emergence of new phase transitions in the minimax separation rate with a subtle dependence on the correlation level. Additionally, we study group structured correlations and derive the minimax separation rate in a model including multiple random effects. The group structure turns out to fundamentally change the detection problem from the equicorrelated case and different phenomena appear in the separation rate.

Likelihood Scores for Sparse Signal and Change-Point Detection

We consider here the identification of change-points on large-scale data streams. The objective is to find the most efficient way of combining information across data stream so that detection is possible under the smallest detectable change magnitude. The challenge comes from the sparsity of change-points when only a small fraction of data streams undergo change at any point in time. The most successful approach to the sparsity issue so far has been the application of hard thresholding such that only local scores from data streams exhibiting significant changes are considered and added. However the identification of an optimal threshold is a difficult one. In particular it is unlikely that the same threshold is optimal for different levels of sparsity. We propose here a sparse likelihood score for identifying a sparse signal. The score is a likelihood ratio for testing between the null hypothesis of no change against an alternative hypothesis in which the change-points or signals are barely detectable. By the Neyman-Pearson Lemma this score has maximum detection power at the given alternative. The outcome is that we have a scoring of data streams that is successful in detecting at the boundary of the detectable region of signals and change-points. The likelihood score can be seen as a soft thresholding approach to sparse signal and change-point detection in which local scores that indicate small changes are down-weighted much more than local scores indicating large changes. We are able to show sharp optimality of the sparsity likelihood score in the sense of achieving successful detection at the minimum detectable order of change magnitude as well as the best constant with respect this order of change.

Compressive Detection of Stochastic Sparse Signals With Unknown Sparsity Degree

Compressive Detection of Stochastic Sparse Signals With Unknown Sparsity Degree

Sparse block signal detection and identification for shared cross-trait association analysis

Genome-wide association studies (GWAS) have identified thousands of single nucleotide polymorphisms (SNPs) that are associated with complex traits. GWAS data allows us to investigate the shared genetic etiologies among different traits. However, linkage disequilibrium (LD) between the SNPs complicates the detection and identification of shared genetic effects. In this paper we model the LD by dividing the genome into LD blocks and linking the genetic variants within a block to a possible latent causal variant. An eigenvector-projected score statistic that leverages the set of variants in LD and a maxtype test statistic (Max-block) are proposed to detect the existence of cross-trait genetic association. The Max-block is easy to calculate and is shown to control the genome-wide error rate. After the detection a stepwise procedure is proposed to identify the significant blocks that explain the genetic sharing between two traits. Simulation experiments show that Max-block is more powerful than standard approaches in the sparse settings and is robust to different signal strengths or levels of sparsity. The method is applied to study shared cross-trait associations in 10 pediatric autoimmune diseases and identified several regions that explain the genetic sharing between juvenile idiopathic arthritis (JIA) and ulcerative colitis (UC) and between UC and Crohn’s disease (CD). In addition, our analysis also indicates the genetic sharing in the MHC region among systemic lupus (SLE), celiac disease (CEL) and common variable immunodeficiency (CVID). Results from real data and simulation studies show that Max-block provides an important alternative to commonly used genetic correlation estimation in understanding genetic correlation among complex diseases.

Large-scale inference with block structure

The detection of weak and rare effects in large amounts of data arises in a number of modern data analysis problems. Known results show that in this situation the potential of statistical inference is severely limited by the large-scale multiple testing that is inherent in these problems. Here, we show that fundamentally more powerful statistical inference is possible when there is some structure in the signal that can be exploited, for example, if the signal is clustered in many small blocks, as is the case in some relevant applications. We derive the detection boundary in such a situation where we allow both the number of blocks and the block length to grow polynomially with sample size. We derive these results both for the univariate and the multivariate settings as well as for the problem of detecting clusters in a network. These results recover as special cases the sparse signal detection problem (Ann. Statist. 32 (2004) 962–994) where there is no structure in the signal, as well as the scan problem (Statist. Sinica 23 (2013) 409–428) where the signal comprises a single interval. We develop methodology that allows optimal adaptive detection in the general setting, thus exploiting the structure if it is present without incurring a relevant penalty in the case where there is no structure. The advantage of this methodology can be considerable, as in the case of no structure the means need to increase at the rate logn to ensure detection, while the presence of structure allows detection even if the means decrease at a polynomial rate.

Variational Inference Based Sparse Signal Detection for Next Generation Multiple Access

The next generation multiple access (NGMA) schemes are considered to support massive access for a large number of devices, which motivates us to develop a low-complexity approach for next generation systems. Since the generalized spatial modulation (SM) can be adopted to the system, a number of compressive sensing (CS) reconstruction algorithms are deployed for the detection of sparse signals, while the complexity of CS-based approaches is proportional to the number of antennas. In order to decrease the complexity, we propose a two-stage approach to detect sparse signals, where the received signals are divided into groups. Then, the activity variables of aggregated signals are decided and the sparse signal detection is carried out at the signals belonging to active groups. During the activity variable detection, the variational inference algorithm is applied to determine the activity variables. Moreover, in order to analyze the performance of activity variable detection, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$J$ </tex-math></inline-formula> -divergence is proposed to measure the distance between the distributions, while the approximate expression of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$J$ </tex-math></inline-formula> -divergence is derived. Simulation results show that the proposed approach is able to provide good detection performance with low complexity. In addition, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$J$ </tex-math></inline-formula> -divergence is confirmed to be useful as an evaluation metric to measure the detection performance.

Distributed detection of sparse signals with censoring sensors in clustered sensor networks

In this paper, we explore the distributed detection of sparse signals in energy-limited clustered sensor networks (CSNs). For this problem, the centralized detector based on locally most powerful test (LMPT) methodology that uses the analog data transmitted by all the sensor nodes in CSNs can be easily realized according to the prior work. However, for the centralized LMPT detector, the energy consumption caused by data transmission is excessively high, which makes its implementation in CSNs with limited energy supply impractical. To address this issue, we propose a new detector by combining the advantages of censoring and LMPT strategies, in which both the cluster head (CLH) nodes and the ordinary (ORD) nodes only send data deemed to be informative enough and the fusion center (FC) fuses the received data based on LMPT methodology. The detection performance of the proposed detector, characterized by Fisher Information, is analyzed in the asymptotic regime. Also, we analytically derive the relationship between the detection performance of the proposed censoring-based LMPT (cens-LMPT) detector and the communication rates, both of which are controlled by the censoring thresholds. We present an illustrative example by considering the detection problem with 2-CSNs, i.e., CSNs in which each cluster contains two nodes, and provide corresponding theoretical analysis and simulation results.

Generalized Locally Most Powerful Tests for Distributed Sparse Signal Detection

In this paper we tackle distributed detection of a localized phenomenon of interest (POI) whose signature is sparse via a wireless sensor network. We assume that both the position and the emitted power of the POI are unknown, other than the sparsity degree associated to its signature. We consider two communication scenarios in which sensors send either (<inline-formula><tex-math notation="LaTeX"><?TeX $i$?></tex-math></inline-formula>) their compressed observations or (<inline-formula><tex-math notation="LaTeX"><?TeX $ii$?></tex-math></inline-formula>) a 1-bit quantization of them to the fusion center (FC). In the latter case, we consider non-ideal reporting channels between the sensors and the FC. We derive generalized (i.e. based on Davies&#x2019; framework&#x00A0;[1]) locally most powerful detectors for the considered problem with the aim of obtaining computationally-efficient fusion rules. Moreover, we obtain their asymptotic performance and, based on such result, we design the local quantization thresholds at the sensors by solving a 1-D optimization problem. Simulation results confirm the effectiveness of the proposed design and highlight only negligible performance loss with respect to counterparts based on the (more-complex) generalized likelihood ratio.

Sparsity Signal Detection for Indoor GSSK-VLC System

In this article, the signal detection problem in indoor visible light communication (VLC) system aided by generalized space shift keying (GSSK) is modeled as a sparse signal reconstruction problem, which has lower computational complexity by exploiting the sparse reconstruction algorithms in compressed sensing (CS). In order to satisfy the measurement matrix property to perform sparse signal reconstruction, a preprocessing approach of measurement matrix is proposed based on singular value decomposition (SVD), which theoretically guarantees the feasibility of utilizing CS based sparse signal detection method in indoor GSSK-VLC system. Then, by adopting classical orthogonal matching pursuit (OMP) algorithm and compressed sampling matching pursuit (CoSaMP) algorithm, the GSSK signals are efficiently detected in the considered indoor GSSK-VLC system. Furthermore, a more efficient detection algorithm combined with OMP and maximum likelihood (ML) is also presented especially for SSK scenario. Finally, the effectiveness of the proposed sparsity aided detection algorithms in indoor GSSK-VLC system are verified by computer simulations. The results show that the proposed algorithms can achieve better bit error rate (BER) and lower computation complexity than ML based detection method. Specifically, a signal-to-noise ratio (SNR) gain as high as 12 dB is observed in the SSK scenario and about 5 dB in case of a GSSK scenario upon employing our proposed detection methods.

Improved SparseEA for sparse large-scale multi-objective optimization problems

Sparse large-scale multi-objective optimization problems (LSMOPs) widely exist in real-world applications, which have the properties of involving a large number of decision variables and sparse Pareto optimal solutions, i.e., most decision variables of these solutions are zero. In recent years, sparse LSMOPs have attracted increasing attentions in the evolutionary computation community. However, all the recently tailored algorithms for sparse LSMOPs put the sparsity detection and maintenance in the first place, where the nonzero variables can hardly be optimized sufficiently within a limited budget of function evaluations. To address this issue, this paper proposes to enhance the connection between real variables and binary variables within the two-layer encoding scheme with the assistance of variable grouping techniques. In this way, more efforts can be devoted to the real part of nonzero variables, achieving the balance between sparsity maintenance and variable optimization. According to the experimental results on eight benchmark problems and three real-world applications, the proposed algorithm is superior over existing state-of-the-art evolutionary algorithms for sparse LSMOPs.

Spatial Shrinkage Via the Product Independent Gaussian Process Prior

We study the problem of sparse signal detection on a spatial domain. We propose a novel approach to model continuous signals that are sparse and piecewise-smooth as the product of independent Gaussian (PING) processes with a smooth covariance kernel. The smoothness of the PING process is ensured by the smoothness of the covariance kernels of the Gaussian components in the product, and sparsity is controlled by the number of components. The bivariate kurtosis of the PING process implies that more components in the product results in the thicker tail and sharper peak at zero. We develop an efficient computation algorithm based on spectral methods. The simulation results demonstrate superior estimation using the PING prior over Gaussian process prior for different image regressions. We apply our method to a longitudinal magnetic resonance imaging dataset to detect the regions that are affected by multiple sclerosis computation in this domain. Supplementary materials for this article are available online.

Optimal post-selection inference for sparse signals: a nonparametric empirical Bayes approach

Summary Many recently developed Bayesian methods focus on sparse signal detection. However, much less work has been done on the natural follow-up question: how does one make valid inferences for the magnitude of those signals after selection? Ordinary Bayesian credible intervals suffer from selection bias, as do ordinary frequentist confidence intervals. Existing Bayesian methods for correcting this bias produce credible intervals with poor frequentist properties. Further, existing frequentist approaches require sacrificing the benefits of shrinkage typical in Bayesian methods, resulting in confidence intervals that are needlessly wide. We address this gap by proposing a nonparametric empirical Bayes approach to constructing optimal selection-adjusted confidence sets. Our method produces confidence sets that are as short as possible on average, while both adjusting for selection and maintaining exact frequentist coverage uniformly over the parameter space. We demonstrate an important consistency property of our procedure: under mild conditions, it asymptotically converges to the results of an oracle-Bayes analysis in which the prior distribution of signal sizes is known exactly. Across a series of examples, the method is found to outperform existing frequentist techniques for post-selection inference, producing confidence sets that are notably shorter, but with the same coverage guarantee.

Distributed Detection of Sparse Stochastic Signals With 1-Bit Data in Tree-Structured Sensor Networks

In this paper, we consider the problem of detection of sparse stochastic signals based on 1-bit data with tree-structured sensor networks (TSNs). In the literature, distributed detection of sparse signals with parallel sensor networks (PSNs) has previously been studied, and detectors using the locally most powerful test (LMPT) strategies with analog data and 1-bit data have been proposed, respectively. However, parallel topology does not always reflect the practical scenario, such as in the case where some nodes are outside the communication range of the fusion center (FC). In this paper, we design a new detector for TSNs with extremely limited resources and let each local sensor node send 1-bit data to its immediate successor. For the proposed detector, we devise 1-bit quantizers at the local sensor nodes and the decision fusion rule with the 1-bit data collected at the FC, based on quantization of likelihood ratios and the LMPT strategy. We also present the procedure to obtain the near optimal quantization thresholds numerically for nodes at different levels by characterizing the detection performance in terms of the Fisher Information. In particular, in two-level two-degree (2L-2D) TSNs, compared with the analog LMPT detector with Q nodes that transmit analog data hierarchically, the proposed 1-bit LMPT detector asymptotically needs 1.74 Q nodes to compensate for the performance loss induced by local quantization. Simulation results validate our theoretical analysis.

Distributed Detection of Sparse Signals With Physical Layer Secrecy Constraints: A Falsified Censoring Strategy

In this paper, we investigate the problem of distributed detection of sparse signals in wireless sensor networks (WSNs) with censoring sensors in the presence of an Eavesdropper (Eve). The Eve, which is able to perfectly monitor the “idle” and “busy” states of the communication channels between the local sensors and the fusion center (FC), also wants to detect the sparse signals. For the classical problem of distributed detection with censoring sensors, applying appropriate censoring thresholds to attain the same transmission probability under either hypothesis to ensure perfect secrecy has previously been studied. We refer to it as the clairvoyant censoring method since it requires full knowledge of the distributions of the observations. However, the clairvoyant censoring method is not practical to implement for the detection of sparse signals with an unknown sparsity level. In this paper, a falsified censoring (FACE) strategy is proposed, in which a group of cooperating deceitful nodes censor their local observations in a way that is opposite to what would be done at the regular nodes. Based on this setup, the optimization problem to maximize the detection performance at the FC under communication and secrecy constraints is formulated and numerical methods are provided to find the near optimal system parameters. Simulation results exhibit excellent performance of our proposed strategy.

Distributed Detection of Sparse Signals With Censoring Sensors Via Locally Most Powerful Test

In this letter, we consider the problem of distributed detection of stochastic sparse signals in battery-powered sensor networks (SNs). For this problem, an original locally most powerful test (oLMPT) detector has previously been developed, where compressed measurements are collected from all local sensors and then fused at the fusion center (FC) for making the global decision. However, since the sensors always operate on limited energy resources, allowing all the nodes to send their observations to the FC all the time exerts tremendous pressure on their energy consumption and hinders the longevity of the sensors. To solve this problem, we propose a new censoring LMPT (cen-LMPT) detector by combining the strengths of censoring strategy and the oLMPT detector, where sensors are designated to merely send observations deemed informative enough so as to utilize the local energy more efficiently, and the FC still makes the global decision based on LMPT. We analytically derive the relationship between the detection performance and the communication rate for the proposed detector. It is shown that, compared with the oLMPT detector, the proposed cen-LMPT detector with the same number of nodes can achieve almost the same detection performance with significantly lower communication rate and, therefore, much lower local energy consumption. The simulation results verify our theoretical findings.

Sub-Nyquist Spectrum Sensing of Sparse Wideband Signals Using Low-Density Measurement Matrices

The problem of wideband spectrum sensing/sampling in the sub-Nyquist domain is solved in this paper using sparse (low-density) binary-valued measurement matrices. Key objectives are (i) to achieve an efficient compression ratio, and (ii) improve the signal reconstruction performance. We propose a novel RF front-end with parallel branches that we have called Low-Density Wideband Converter (LDWC). We show that the LDWC implements a binary Low-Density Parity Check (LDPC) matrix as the compressive sensing (CS) measurement matrix. We evaluate, using an Information-Theoretic approach, the asymptotic bound on the required number of LDWC parallel branches for sparsity detection. We develop two new belief propagation (BP) algorithms that operate on the Tanner graph of the CS measurements. We have derived the first algorithm by assuming independence among the variable nodes (VNs) of the Tanner graph. For the second method, we have accounted for the joint probability distribution of the VNs. Analytical and simulated performance results prove the concepts of the LDWC and the proposed BP algorithms and quantify the attainment of objectives (i) and (ii) stated above.

Distributed Detection of Sparse Stochastic Signals via Fusion of 1-bit Local Likelihood Ratios

In this letter, we consider the detection of sparse stochastic signals with sensor networks (SNs), where the fusion center (FC) collects 1-bit data from the local sensors and then performs global detection. For this problem, a newly developed 1-bit locally most powerful test (LMPT) detector requires 3.3Q sensors to asymptotically achieve the same detection performance as the centralized LMPT (cLMPT) detector with Q sensors. This 1-bit LMPT detector is based on 1-bit quantized observations without any additional processing at the local sensors. However, direct quantization of observations is not the most efficient processing strategy at the sensors since it incurs unnecessary information loss. In this letter, we propose an improved-1-bit LMPT (Im-1-bit LMPT) detector that fuses local 1-bit quantized likelihood ratios (LRs) instead of directly quantized local observations. In addition, we design the quantization thresholds at the local sensors to ensure asymptotically optimal detection performance of the proposed detector. It is shown theoretically and numerically that, with the designed quantization thresholds, the proposed Im-1-bit LMPT detector for the detection of sparse signals requires less number of sensor nodes to compensate for the performance loss caused by 1-bit quantization.

Graph representation of random signal and its application for sparse signal detection

In this paper, a novel graph-based adequate and concise signal representation paradigm is explored. Amplitudes of a signal can be quantized first. Then the quantization levels and transitions among these levels can be specified as vertices and edges so that the signal waveform (spectrum) can be converted to a graph thereby. This new signal representation framework can provide a promising alternative for manifesting the essential structure of random signals in the graph domain. The randomness of signal samples can be characterized by the topological features of the converted graph. In this work, we explore the connectivity of a graph to measure such randomness. A typical application, namely sparse communication signal detection, can be undertaken by the graph-connectivity metric. New pertinent theoretical analyses are also conducted here. First, the analysis on the minimum sample size for constructing a fully-connected graph from random samples is established. Second, the probability analysis of constructing a fully-connected graph with respect to a particular sample size is also derived. Third, if a graph is not fully connected, the probability distribution function of the number of edges is derived. According to Monte Carlo simulations, our proposed graph-based signal-detection method leads to outstanding performance, compared to a popular existing signal-detection technique especially when the signal-to-noise ratio is rather small.