Leibler Distance Research Articles

Bias-corrected Kullback–Leibler distance criterion based model selection with covariables missing at random

A model selection problem for the conditional probability function of the response variable Y given the covariable vector (X,Z) is considered under the case where X is missing at random. And two novel model selection criteria are suggested. It is shown that the model selection by these two criteria is consistent and that the population parameter estimators, corresponding to the selected model, are also consistent and asymptotically normal. Extensive simulation studies are conducted to investigate the finite-sample performances of the proposed two criteria and a thorough comparison is made with some related model selection strategies. Moreover, two real data analyses are presented for illustrating the practical application of the proposed two criteria.

Data prediction approaches for efficient data transmission using optimized Leibler distance matrix-based data aggregation in wireless sensor network

In wireless sensor networks, sensor nodes are distributed in different geographically dispersed areas, sink nodes are used for sensing data and sink nodes are used for data collected from different sensor nodes. Therefore, data collection is a key issue in wireless sensor networks. In addition to computational overhead, many challenging security issues have been introduced to in-network data processing. Data collection and communication methods are also proposed based on the Optimized Leibler Distance Matrix-based data aggregation (OLDMDA) algorithm. These sensor nodes belong to the category forming the OLDMDA plane data aggregation technology, which includes homogeneous nodes and data collection protocols. Leibler Distance Matrix for Node Location to find the location and multi sink node selection is based on node location distance using a Leibler matrix. In this distance matrix to collect the information of node energy, transmission rate to identify the higher performance node on the network. The Optimized Johnson's shortest path algorithm to solve the path problem source to destination. In this path selection method proposed to find the all the possible multi path randomly generate the graph. The performance of the new technology has been implemented using the network emulator (NS2). Simulation results of the proposed method OLDMDA higher performance with respect to the average packet transfer rate and metrics, as compared with the delay and network lifetime.

Distance-based tests for planar shape

Statistical shape analysis (SSA) may be understood like a field of Multivariate Analysis and has assumed a prominent position due to its applicability in various areas; such as in imagery processing, biology, anatomy, among others. Biological hypotheses have been often formulating in the SSA framework, which is based on the concept of “shape” whose related data are in a non Euclidian space. Thus, tailored tests are sought to take accurate decision in this area. To that end, we first assume the complex Bingham (CB) distribution to describe preshape data which are extracted from two-dimensional objects, called planar shape. Adopting adequate distances is crucial in SSA; particularly at hypothesis tests, discriminant analysis and clustering processes. In this paper, we derive the Rényi, Kullback–Leibler (KL), Bhattacharyya and Hellinger distances for the CB distribution. These quantities are then used as distance-based hypothesis tests to assess if two planar shape samples come from the same distribution. Even though useful for many models having Euclidean supports, we prove that the CB KL discrepancy is invariant in rotation, what restricts its use. Four new homogeneity hypothesis tests are proposed, three among them may also be used like tests involving mean shape. The performance of the proposed tests is quantified and compared (in terms of both test size and power and robustness) to that due to other four mean shape tests of the SSA literature: Hotelling T2, Goodall, James and lambda. Finally, an application in evolutionary biology is done to assess sexual dimorphism in the Pan troglodytes (chimpanzee) species. Results point out our proposals may offer meaningfully advantages comparatively to considered literature tests.

A novel generalized grey target decision method with index and weight both containing mixed types of data

PurposeThis paper is to propose a novel generalized grey target decision method (GGTDM) with index and weight both containing mixed types of data.Design/methodology/approachThe decision-making steps of the proposed approach are as follows. First, all mixed attribute values of alternatives and weights are transformed into binary connection numbers and also comprised two-tuple (determinacy, uncertainty) numbers. Then, the two-tuple (determinacy, uncertainty) numbers of target center indices are calculated. Next, the certain weights are determined by the Gini–Simpson (G–S) index-based method. Following this, the comprehensive-weighted Kullback–Leibler distances (CWKLDs) of all alternatives and the target center are obtained. Finally, the alternative ranking relies on the CWKLD considering the smaller value as the better option.FindingsThe certain weights determined by the improved Gini–Simpson index (IGSI) based method are more accurate in compared with that by the proximity-based method and the weight function method. The discrimination ability of alternatives ranking of the proposed approach is stronger than that of the compared comprehensive-weighted proximity (CWP) based method and comprehensive-weighted Gini–Simpson index (CWGSI) based method.Research limitations/implicationsThe proposed method fulfills the decision-making task relying on CWKLD, which solves the uncertain measurement from the viewpoint of entropy.Originality/valueThe proposed approach adopts the IGSI to transform uncertain weights into certain ones and takes the CWKLD as the basis for the decision-making.

A Novel Contrast Enhancement Technique using Gradient-Based Joint Histogram Equalization

Image enhancement by histogram equalization reduces the number of gray levels that lead to information loss and unnatural appearance. This paper aims to improve the contrast and preserve information and edge details by employing gradient-based joint histogram equalization. It is achieved by a multiscale-based dark pass filter, which gives the pixel’s edge information. A joint histogram is computed from the edge information and the gray-level distribution of the low contrast image to develop a discrete function. This discrete function is mapped to uniform distribution to get the final enhanced image. The proposed method is experimented on Kodak, USC-SIPI, and CSIQ databases and analyzed using various performance measures such as Contrast, standard deviation, contrast improvement index, structural similarity index, normalized entropy, and normalized mean brightness error. It is observed that the proposed method provides the highest Contrast values of 86.2, 85.79, and 86.02 in Kodak, USC-SIPI, and CSIQ databases, respectively. Normalized entropy value is found to be highest for the proposed method for all the databases. This is noticed to be 0.89, 0.84, and 0.85 for the databases Kodak, USC-SIPI, and CSIQ, respectively. The degree of the uniform distribution is measured by Kullback–Leibler distance. The proposed method produces more uniformity than other techniques available in the literature for all the three databases.

Statistical foundation of Variational Bayes neural networks

Despite the popularism of Bayesian neural networks (BNNs) in recent years, its use is somewhat limited in complex and big data situations due to the computational cost associated with full posterior evaluations. Variational Bayes (VB) provides a useful alternative to circumvent the computational cost and time complexity associated with the generation of samples from the true posterior using Markov Chain Monte Carlo (MCMC) techniques. The efficacy of the VB methods is well established in machine learning literature. However, its potential broader impact is hindered due to a lack of theoretical validity from a statistical perspective. In this paper, we establish the fundamental result of posterior consistency for the mean-field variational posterior (VP) for a feed-forward artificial neural network model. The paper underlines the conditions needed to guarantee that the VP concentrates around Hellinger neighborhoods of the true density function. Additionally, the role of the scale parameter and its influence on the convergence rates has also been discussed. The paper mainly relies on two results (1) the rate at which the true posterior grows (2) the rate at which the Kullback–Leibler (KL) distance between the posterior and variational posterior grows. The theory provides a guideline for building prior distributions for BNNs along with an assessment of accuracy of the corresponding VB implementation.

A Smart Privacy-Preserving Learning Method by Fake Gradients to Protect Users Items in Recommender Systems

In this paper, we study the problem of protecting privacy in recommender systems. We focus on protecting the items rated by users and propose a novel privacy-preserving matrix factorization algorithm. In our algorithm, the user will submit a fake gradient to make the central server not able to distinguish which items are selected by the user. We make the Kullback–Leibler distance between the real and fake gradient distributions to be small thus hard to be distinguished. Using theories and experiments, we show that our algorithm can be reduced to a time-delay SGD, which can be proved to have a good convergence so that the accuracy will not decline. Our algorithm achieves a good tradeoff between the privacy and accuracy.

Fully Statistical, Wavelet-based conditional random field (FSWCRF) for SAR image segmentation

Recently, the conditional random field (CRF) model has been greatly considered in synthetic aperture radar (SAR) image segmentation. This model not only directly considers the posterior distribution of the label field conditioned on images but also gives the interactions between the observations. In this paper, we propose a new CRF-based algorithm for SAR image segmentation. We consider the statistical approach jointly in feature extraction and similarity measurement in the proposed conditional random field model. Using the benefit of the 2-D wavelet transform, we define the generalized Gaussian distribution (GGD) on the wavelet coefficients to extract texture-based features. Then, to improve the CRF potential functions a new unary function is proposed which exactly matches the statistical properties of the wavelet coefficients and produces more accurate parameters for different regions. As the advantage of this function, it is no longer necessary to apply the multinomial logistic regression (MLR) model used in previous CRFs. Moreover, using the Kullback–Leibler distance (KLD) between distribution functions, the similarity measure in our pairwise potential is proposed very effectively and efficiently. The superiority of this scheme is that the similarity measure can be entirely computed using the parameters of the GGD that are typically of small size compared with the feature vectors in the previous methods. Comprehensive experiments on both synthetic and real SAR images indicate that our proposed algorithm achieves accuracy improvement in SAR image segmentation.

Detecting network cyber-attacks using an integrated statistical approach

Anomaly detection in the Internet of Things (IoT) is imperative to improve its reliability and safety. Detecting denial of service (DOS) and distributed DOS (DDOS) is one of the critical security challenges facing network technologies. This paper presents an anomaly detection mechanism using the Kullback–Leibler distance (KLD) to detect DOS and DDOS flooding attacks, including transmission control protocol (TCP) SYN flood, UDP flood, and ICMP-based attacks. This mechanism integrates the desirable properties of KLD, the capacity to quantitatively discriminate between two distributions, with the sensitivity of an exponential smoothing scheme. The primary reason for exponentially smoothing KLD measurements (ES–KLD) is to aggregate all of the information from past and actual samples in the decision rule, making the detector sensitive to small anomalies. Furthermore, a nonparametric approach using kernel density estimation has been used to set a threshold for ES-KLD decision statistic to uncover the presence of attacks. Tests on three publicly available datasets show improved performances of the proposed mechanism in detecting cyber-attacks compared to other conventional monitoring procedures.

Performance limits of one-bit compressive classification

Classification is an important task in the fields of signal processing and machine learning. Recently, compressive classification (CC) appears to enable signal classification directly with the compressive measurements, which are acquired by the technique of compressive sensing (CS). However, the existing works of CC ignore the practical quantization operation. This paper studies the one-bit CC based on one-bit quantized compressive measurements, which is appealing in saving the transmission and storage bits in many applications when the bandwidth and energy are constrained. For the signals that are not necessarily sparse as well as exactly sparse, we provide the performance limits of one-bit CC. We first analyze the Chernoff distance and Kullback–Leibler (KL) distance between two probability density functions under any two hypotheses. The results are then used to derive the upper and lower bounds on the probability of misclassification in the high-dimensional setting. The analytical results show that one-bit quantized CC has the same performance trend as conventional not-quantized CC when the number of measurements increases.

Super-Resolution Mapping Based on Spatial–Spectral Correlation for Spectral Imagery

Due to the influences of imaging conditions, spectral imagery can be coarse and contain a large number of mixed pixels. These mixed pixels can lead to inaccuracies in the land-cover class (LC) mapping. Super-resolution mapping (SRM) can be used to analyze such mixed pixels and obtain the LC mapping information at the subpixel level. However, traditional SRM methods mostly rely on spatial correlation based on linear distance, which ignores the influences of nonlinear imaging conditions. In addition, spectral unmixing errors affect the accuracy of utilized spectral properties. In order to overcome the influence of linear and nonlinear imaging conditions and utilize more accurate spectral properties, the SRM based on spatial–spectral correlation (SSC) is proposed in this work. Spatial correlation is obtained using the mixed spatial attraction model (MSAM) based on the linear Euclidean distance. Besides, a spectral correlation that utilizes spectral properties based on the nonlinear Kullback–Leibler distance (KLD) is proposed. Spatial and spectral correlations are combined to reduce the influences of linear and nonlinear imaging conditions, which results in an improved mapping result. The utilized spectral properties are extracted directly by spectral imagery, thus avoiding the spectral unmixing errors. Experimental results on the three spectral images show that the proposed SSC yields better mapping results than state-of-the-art methods.

Expert Refined Topic Models to Edit Topic Clusters in Image Analysis Applied to Welding Engineering

This paper proposes a new method to generate edited topics or clusters to analyze images for prioritizing quality issues. The approach is associated with a new way for subject matter experts to edit the cluster definitions by “zapping” or “boosting” pixels. We refer to the information entered by users or experts as “high-level” data and we are apparently the first to allow in our model for the possibility of errors coming from the experts. The collapsed Gibbs sampler is proposed that permits efficient processing for datasets involving tens of thousands of records. Numerical examples illustrate the benefits of the high-level data related to improving accuracy measured by Kullback–Leibler (KL) distance. The numerical examples include a Tungsten inert gas example from the literature. In addition, a novel laser aluminum alloy image application illustrates the assignment of welds to groups that correspond to part conformance standards.

Features detection based blind handover using kullback leibler distance for 5G HetNets systems

<p>The Fifth Generation of Mobile Networks (5G) is changing the cellular network infrastructure paradigm, and Small Cells are a key piece of this shift. But the high number of Small Cells and their low coverage involve more Handovers to provide continuous connectivity, and the selection, quickly and at low energy cost, of the appropriate one in the vicinity of thousands is also a key problem. In this paper, we propose a new method, to have an efficient, blind and rapid handover just by analysing Received Signal probability density function instead of demodulating and analysing Received Signal itself as in classical handover. The proposed method exploits KL Distance, Akaike Information Criterion (AIC) and Akaike weights, in order to decide blindly the best handover and the best Base Station (BS) for each user</p>

Features detection based blind handover using kullback leibler distance for 5G HetNets systems

Features detection based blind handover using kullback leibler distance for 5G HetNets systems

Statistical Evaluation of the Behavior of 5-GHz Radio LAN Devices

The goal of this paper is to develop an experimental hardware testbed for verifying the compliance of the medium access mechanism of commercially available Wi-Fi certified units with the IEEE 802.11 standard. The core of this testbed is based on a Universal Software Radio Peripheral (USRP), which is used to capture and downconvert the Wi-Fi frames to the baseband. The data captured by the USRP are used to extract the key medium access parameters of the commercial unit under test (UUT). Our measurement methodology involves a novel compliance metric based on the Kullback–Leibler distance between probability distributions. This metric is in the form of a single scalar, which will hence enable us to rank the UUTs based on their level of compliance. Furthermore, we develop a method based on the harmonized European Standard ETSI EN 301 893 to identify UUTs with noncompliant access mechanisms that tend to monopolize the wireless medium and UUTs for which compliance is an option that deviates from default settings.

INFORMATION MEASURES IN A SMALL NETWORK OF SPIKING NEURONS

The information transmission among the elements of a small network of spiking neurons is studied using the normalized differential entropy and the Kullback Leibler distance information measures. The attention is devoted to the information content of the spiking activity of a reference neuron subject to excitatory and inhibitory stimuli coming from other elements of the considered network. The use of information measures allows to quantify the effects of the input contributions. The role of inhibition in the spiking activity of the reference neuron is enlighted and the effect of considering different distributions for the time events of the stimuli is discussed. 1 Introduction The classical characterization of the input-output properties of neurons, as well as of the neuronal models, makes use of the so called frequency transfer functions. Several definitions exist for these functions but they all share the feature of being plots of the output frequency of firing against the strength of the input signal. The differences are confined to the measure employed to quantify the input strenght. Use of the transfer function tacitely assumes that the information is coded by the frequency of action potentials ((2)). However, frequency is only one possible descriptor of the properties of a spike train and it is evident that different spike records may have the same frequency but at the same time they may be very dissimilar. Therefore, other tools to quantify the differences among spike records have been investigated. Considering the well established relevance of noise in neuronal coding (cf. for example (3, 15, 21, 19)), alternative measures that could be able to catch the random properties of the firing activity have been introduced. In this direction many efforts have been made to apply Shannon theory of communi- cation and information transmission ((20)) to the study of the properties of the nervous system. Among the wide literature on this topic, we cite here for example (11), where the Fisher information is used to study the input-output effect for some neuronal models. In (12) and (13) this measure is applied to the stochastic leaky integrate-and-fire (LIF) model, iden- tifying the input signal as a constant additive term in the drift of the Ornstein-Uhlenbeck process used to describe the underthreshold membrane potential dynamics. In (9) a nor- malized version of the differential entropy is introduced to study the randomness for the same neuronal model. Information theory is applied here to the study of the information transmission among the units of a small network of spiking neurons. In particular we are interested in the quantification of the response of a reference neuron to excitatory and inhibitory inputs coming from other units of the network. The problem has been already considered in (22) and (23) but the study is performed there with different tools of analysis, as histograms, autocorrelation functions and crosscorrelation functions.

Mis-specification analysis of the impact of covariates on the diffusion coefficient in Wiener degradation process

The diffusion coefficients are often assumed irrelevant to stresses in Wiener degradation process. However, there is evidence that the diffusion coefficients under different accelerated stress levels are different in an accelerated degradation test where higher accelerated stress could lead to larger diffusion. Therefore, in contrast to the existing models in which the diffusion coefficients are assumed as a constant or irrelevant to stress levels, this paper allows both the drift coefficients and the diffusion coefficients to be defined as the functions of accelerated stresses. Then, the consequence of model mis-specification in which the proposed model is wrongly fitted by its special case is analyzed based on Kullback–Leibler distance. In addition, the influence of each degradation parameter on the relative bias and the relative variation of the mean time to failure and 100p th percentage of first hitting time is analyzed by simulation methods. Finally, some important differences between the two models are verified by two case studies.

A Practical Plateau Lake Extraction Algorithm Combining Novel Statistical Features and Kullback–Leibler Distance Using Synthetic Aperture Radar Imagery

Due to wind-induced waves, dry sand, wet snow, and terrain shadows, the lake extraction from synthetic aperture radar (SAR) imagery in the Qinghai-Tibet plateau is accompanied by false alarms. In this article, a practical plateau lake extraction algorithm combining novel statistical features and Kullback–Leibler distance (KLD) using SAR imagery has been proposed. First, a mathematical description for the plateau lake surface called object-based generalized gamma distribution (OGΓD) features has been proposed, which is able to suppress the false alarms by using spatial context information as the large-scale descriptor. Second, the random forest classifier is used to train a multifeature set, including conventional texture features and OGΓD features, and output an initial labeling result. Finally, to suppress the false alarms in the initial lake extraction results, automatic postprocessing based on KLD has been used. The algorithm is tested by several experiments using Sentinel-1 SAR data, performing better than the state-of-the-art algorithms, achieving the overall accuracy of 99.54% while maintaining a false-alarm rate of 0.32%.

CNN-Based Polarimetric Decomposition Feature Selection for PolSAR Image Classification

In order to better interpret polarimetric synthetic aperture radar (PolSAR) images, many scholars tend to do target decomposition for PolSAR images and utilize the obtained features to perform subsequent classification. These target decomposition features play an important role in terrain classification but completely utilizing them produces a high computational complexity. Furthermore, some features have a negative impact on the classification task. Therefore, selecting the appropriate amount of high-quality features is of great significance to the classification task. In this paper, we propose a convolutional neural network (CNN)-based feature selection algorithm for PolSAR image classification. First, we design a 1-D CNN for feature selection, then train the designed network with all the decomposition features to obtain a trained model. Second, the Kullback–Leibler distance (KLD) between different features is utilized as a standard to select feature subsets. Third, feature subsets with excellent performance form the final results. Due to the special structure of the 1-D CNN, repetitively training model is avoided when the input changes. Different from traditional feature selection methods, our method considers the performance of features combination rather than single feature contribution. To this end, the feature subsets selected by the proposed method are more useful to the classification task. Innovatively introducing KLD in the selection stage avoids random selection and improves the selection efficiency. Finally, we validate the performance of selected feature subsets in traditional and deep learning classification frameworks. Experiments demonstrate that features selected by the proposed method have a good performance comparing with others on three real PolSAR data sets.

The Dominant Influencer of Voltage Fluctuation (DIVF) for Power Distribution System

Variability in power generation at multiple consumer-owned renewable energy sources can cause random voltage fluctuations, which may result in voltage violations at multiple nodes in a power distribution system. Traditional methods of voltage control are ineffective to address fast voltage variations. New dynamic methods of voltage control using end-user devices can significantly benefit from the knowledge of the dominant influencer of voltage fluctuation (DIVF). This paper introduces information theoretic metrics to identify DIVF for every node in the power distribution system. A node can be DIVF due to its location in the distribution system and its capacity to generate and change its power. Use of traditional power flow algorithms to identify the DIVF is computationally complex, which limits their use in real-time applications. In this paper, information theoretic indicators of the DIVF are computed analytically in a computationally efficient manner, and their effectiveness to identify the DIVF is tested using the IEEE 69 node test system. From simulations, it is shown that differential entropy, Kullback–Leibler distance, and Frechet distance are effective indicators of the DIVF and can successfully identify the most influential nodes providing the foundation for intelligent voltage control.