Entropy-based Measure Research Articles

UniDis: a universal discretization technique

Discretization techniques have played an important role in machine learning and data mining as most methods in such areas require that the training data set contains only discrete attributes. Data discretization unification (DDU), one of the state-of-the-art discretization techniques, trades off classification errors and the number of discretized intervals, and unifies existing discretization criteria. However, it suffers from two deficiencies. First, the efficiency of DDU is very low as it conducts a large number of parameters to search good results, which does not still guarantee to obtain an optimal solution. Second, DDU does not take into account the number of inconsistent records produced by discretization, which leads to unnecessary information loss. To overcome the above deficiencies, this paper presents a Uni versal Dis cretization technique, namely UniDis. We first develop a non-parametric normalized discretization criteria which avoids the effect of relatively large difference between classification errors and the number of discretized intervals on discretization results. In addition, we define a new entropy-based measure of inconsistency for multi-dimensional variables to effectively control information loss while producing a concise summarization of continuous variables. Finally, we propose a heuristic algorithm to guarantee better discretization based on the non-parametric normalized discretization criteria and the entropy-based inconsistency. Besides theoretical analysis, experimental results demonstrate that our approach is statistically comparable to DDU evaluated by a popular statistical test and it yields a better discretization scheme which significantly improves the accuracy of classification than previously other known discretization methods except for DDU by running J4.8 decision tree and Naive Bayes classifier.

Determining lower bounds on a measure of multipartite entanglement from few local observables

We introduce a method to lower bound an entropy-based measure of genuine multipartite entanglement via nonlinear entanglement witnesses. We show that some of these bounds are tight and explicitly work out their connection to a framework of nonlinear witnesses that were published recently. Furthermore we provide a detailed analysis of these lower bounds in the context of other possible bounds and measures. In exemplary cases we show that only a few local measurements are necessary to determine these lower bounds.

Entropy coding for training deep belief networks with imbalanced and unlabeled data

Training deep belief networks (DBNs) is normally done with large data sets. In this work, the goal is to predict traces of the surface of the tongue in ultrasound images of the mouth during speech. Performance on this task can be dramatically enhanced by pre-training a DBN jointly on human-supplied traces and ultrasound images, then training a modified version of the network to predict traces from ultrasound only. However, hand-tracing the entire dataset of ultrasound images is extremely labor intensive. Moreover, the dataset is highly imbalanced since many images are extremely similar. This work presents a bootstrapping method which takes advantage of this imbalance, iteratively selecting a small subset of images to be hand-traced, then (re)training the DBN, making use of an entropy-based diversity measure for the initial selection. With this approach, a three-fold reduction in human time required to trace an entire dataset with human-level accuracy was achieved.

The effects of related and unrelated technological diversification on innovation performance and corporate growth in the Taiwan’s semiconductor industry

This study uses the entropy-based patent measure to discuss the effects of related technological diversification (RTD) and unrelated technological diversification (UTD) on innovation performance an...

Measuring the Impact of Observations on the Predictability of the Kuroshio Extension in a Shallow-Water Model

Abstract In this paper, sequential importance sampling is used to assess the impact of observations on an ensemble prediction for the decadal path transitions of the Kuroshio Extension. This particle-filtering approach gives access to the probability density of the state vector, which allows the predictive power—an entropy-based measure—of the ensemble prediction to be determined. The proposed setup makes use of an ensemble that, at each time, samples the climatological probability distribution. Then, in a postprocessing step, the impact of different sets of observations is measured by the increase in predictive power of the ensemble over the climatological signal during one year. The method is applied in an identical-twin experiment for the Kuroshio Extension using a reduced-gravity shallow-water model. This study investigates the impact of assimilating velocity observations from different locations during the elongated and the contracted meandering states of the Kuroshio Extension. Optimal observation locations correspond to regions with strong potential vorticity gradients. For the elongated state the optimal location is in the first meander of the Kuroshio Extension. During the contracted state it is located south of Japan, where the Kuroshio separates from the coast.

Weighted Entropy-based Measure for Image Segmentation

Image segmentation is one of the fundamental and important steps that is needed to prepare an image for further processing in many computer vision applications. Over the last few decades, many image segmentation methods have been proposed, as accurate image segmentation is vitally important for many image, video and computer vision applications. A common approach is to look at the grey level colours of the image to perform multi-level-thresholding. The ability to quantify and compare the resulting segmented images is of vital importance even though it can be a major challenge. One measure used here computes the total distances of the pixels to its centroid for each region to provide a quantifiable measure of the segmented images. We also suggest an improved Zhang's entropy measure for image segmentation based on computing the entropy of the image and segmented regions. In this paper, we will present the results from both of these approaches.

Using the entropy-based patent measure to explore the influences of related and unrelated technological diversification upon technological competences and firm performance

This study applies the entropy-based patent measure to explore the influences of related technological diversification (RTD) and unrelated technological diversification (UTD) upon technological competences and firm performance. The results show that RTD has a monotonically positive effect on technological competences and UTD has an inverse U-shaped effect on technological competences. Besides, the results demonstrate that the extent of the positive influence of RTD upon technological competences is better than that of UTD upon technological competences. If American pharmaceutical companies would like to adopt technological diversification, this study suggests that they should undertake RTD, rather than UTD. In addition, this study finds out that technological competences mediate the relationship between firm performance and both of RTD and UTD. Although RTD and UTD cannot significantly influence firm performance directly, they can positively affect firm performance indirectly through technological competences.

Multi-Objective Particle Swarm Optimization with Dynamic Crowding Entropy-Based Diversity Measure

A multi-objective particle swarm optimization with dynamic crowding entropy-based diversity measure is proposed in this paper. Firstly, the elitist strategy is used in external archive in order to improve the convergence of this algorithm. Then the new diversity strategy called dynamic crowding entropy strategy and the global optimization update strategy are used to ensure sufficient diversity and uniform distribution amongst the solution of the non-dominated fronts. The results show that the proposed algorithm is able to find better spread of solutions with the better convergence to the Pareto front and preserve diversity of Pareto optimal solutions the more efficiently.

A Spatial Correlation-Based Image Compression Framework for Wireless Multimedia Sensor Networks

Data redundancy caused by correlation has motivated the application of collaborative multimedia in-network processing for data filtering and compression in wireless multimedia sensor networks (WMSNs). This paper proposes an information theoretic image compression framework with an objective to maximize the overall compression of the visual information gathered in a WMSN. The novelty of this framework relies on its independence of specific image types and coding algorithms, thereby providing a generic mechanism for image compression under different coding solutions. The proposed framework consists of two components. First, an entropy-based divergence measure (EDM) scheme is proposed to predict the compression efficiency of performing joint coding on the images collected by spatially correlated cameras. The EDM only takes camera settings as inputs without requiring statistics of real images. Utilizing the predicted results from EDM, a distributed multi-cluster coding protocol (DMCP) is then proposed to construct a compression-oriented coding hierarchy. The DMCP aims to partition the entire network into a set of coding clusters such that the global coding gain is maximized. Moreover, in order to enhance decoding reliability at data sink, the DMCP also guarantees that each sensor camera is covered by at least two different coding clusters. Experiments on H.264 standards show that the proposed EDM can effectively predict the joint coding efficiency from multiple sources. Further simulations demonstrate that the proposed compression framework can reduce 10%–23% total coding rate compared with the individual coding scheme, i.e., each camera sensor compresses its own image independently.

Voice Activity Detector for Noise Spectrum Estimation Using a Dynamic Band-Splitting Entropy Estimate

To obtain reliable performance of Voice Activity Detector (VAD) algorithm, the straight lines on spectrogram of speech-activity being robust against noise is characterized by an entropy-based measure in this paper. A measure of entropy will be defined on the energy domain of harmonic subband. It is shown that the entropy-based measure is well suited for detecting speech in white or quasi-white noises, but will perform poorly for coloured noises. To compensate the limitation, the refined minima controlled recursive averaging, which be updated quickly and accurately even given rapidly increasing levels of noise, is required to desensitize the measure of entropy to various types of noise. Consequently, the proposed VAD algorithm is shown significantly outperform the commonly used energy-based algorithm when SNR drops rapidly, and moreover is insensitive to the changing-level of noise. Experimental results demonstrate that the performance of the proposed VAD is comparable to modern standard VADs such that ITU-T G.729B and ETSI front-end VAD or statistical model-based VADs.

An entropy-based uncertainty measure of process models

In managing business processes, the process uncertainty and variability are significant factors causing difficulties in prediction and decision making, which evokes and augments the importance and need of process measures for systematic analysis. We propose an entropy-based process measure to quantify the uncertainty of business process models. The proposed measure enables capturing the dynamic behavior of processes, in contrast to previous work which focused on providing measures for the static aspect of process models.

An entropy-based measurement of working time flexibility

Abstract Working time flexibility is of major concern to companies, and must be established by reaching arrangements with workers. Comparing flexibility arrangements is far from easy. Measures of flexibility that assume a probability distribution of future demand are not really measuring flexibility, but rather risk. We define a space of states made up of possible working hours over several periods, and apply three measures of flexibility to it: the proportion of feasible states, the average cost and a new entropy-based measure of flexibility (EMF). The EMF is based on Shannon’s entropy. We propose the use of three measures simultaneously for comparing the flexibility provided by a working time arrangement. Two examples are given: one that assesses the flexibility generated by using time accounts and overtime in a working time accounts (WTAs) modality, and one that compares the WTAs and hiring and firing (H&F) modalities.

Application of Tsallis Entropy to EEG: Quantifying the Presence of Burst Suppression After Asphyxial Cardiac Arrest in Rats

Burst suppression (BS) activity in EEG is clinically accepted as a marker of brain dysfunction or injury. Experimental studies in a rodent model of brain injury following asphyxial cardiac arrest (CA) show evidence of BS soon after resuscitation, appearing as a transitional recovery pattern between isoelectricity and continuous EEG. The EEG trends in such experiments suggest varying levels of uncertainty or randomness in the signals. To quantify the EEG data, Shannon entropy and Tsallis entropy (TsEn) are examined. More specifically, an entropy-based measure named TsEn area (TsEnA) is proposed to reveal the presence and the extent of development of BS following brain injury. The methodology of TsEnA and the selection of its parameter are elucidated in detail. To test the validity of this measure, 15 rats were subjected to 7 or 9 min of asphyxial CA. EEG recordings immediately after resuscitation from CA were investigated and characterized by TsEnA. The results show that TsEnA correlates well with the outcome assessed by evaluating the rodents after the experiments using a well-established neurological deficit score (Pearson correlation = 0.86, p << 0.01 ). This research shows that TsEnA reliably quantifies the complex dynamics in BS EEG, and may be useful as an experimental or clinical tool for objective estimation of the gravity of brain damage after CA.

The Complexity of Discourse*

An entropy-based measure is used to calculate the complexity of discourse patterns. This measure of complexity takes into account both the number of patterns (possibly generated), as well as the frequency of each pattern (actually instantiated). The discourse patterns are modelled as random walks in a multi-dimensional and value-weighted space. The multiple dimensions are theoretically specifiable within the framework of conversational analysis (e.g. number of moves, number of participants, types of adjacency pairs, etc.). And the weighted values are empirically measurable within a corpus of texts (e.g. relative frequency participants take the floor, relative frequency first pair-part is a command versus a question, relative frequency an embedding occurs, etc.). The way complexity correlates with various social and discursive factors is described. And the way this method may be extended to analyse successively more complicated patterns is detailed.

Multi-objective self-adaptive differential evolution with elitist archive and crowding entropy-based diversity measure

A self-adaptive differential evolution algorithm incorporate Pareto dominance to solve multi-objective optimization problems is presented. The proposed approach adopts an external elitist archive to retain non-dominated solutions found during the evolutionary process. In order to preserve the diversity of Pareto optimality, a crowding entropy diversity measure tactic is proposed. The crowding entropy strategy is able to measure the crowding degree of the solutions more accurately. The experiments were performed using eighteen benchmark test functions. The experiment results show that, compared with three other multi-objective optimization evolutionary algorithms, the proposed MOSADE is able to find better spread of solutions with better convergence to the Pareto front and preserve the diversity of Pareto optimal solutions more efficiently.

Note on measures for spectral flatness

Spectral flatness is a feature of acoustic signals that has been useful in many audio signal processing applications. The traditional definition of spectral flatness is the ratio of the geometric mean to the arithmetic mean of the magnitude spectrum of the signal, as obtained from the DFT. Presented is an analysis of this measure and the shortcomings. To overcome these shortcomings, a robust measure based on the concept of entropy is proposed. Although this entropy-based measure is derived from a completely different standpoint, it is closely related to the traditional measure, the relation of which is also derived here. Given the importance that spectral flatness is gaining in the field of audio signal processing, it is believed that an understanding of this measure and the development of a robust version as proposed is exigent.

An H-index for rodents: development of entropy-based measure for evaluation of spatial performance in the water maze

An H-index for rodents: development of entropy-based measure for evaluation of spatial performance in the water maze

Development and validation of a sensitive entropy-based measure for the water maze

In the water maze, mice are trained to navigate to an escape platform located below the water's surface, and spatial learning is most commonly evaluated in a probe test in which the platform is removed from the pool. While contemporary tracking software provides precise positional information of mice for the duration of the probe test, existing performance measures (e.g., percent quadrant time, platform crossings) fail to exploit fully the richness of this positional data. Using the concept of entropy (H), here we develop a new measure that considers both how focused the search is and the degree to which searching is centered on the former platform location. To evaluate how H performs compared to existing measures of water maze performance we compiled five separate databases, containing more than 1600 mouse probe tests. Random selection of individual trials from respective databases then allowed us to simulate experiments with varying sample and effect sizes. Using this Monte Carlo-based method, we found that H outperformed existing measures in its ability to detect group differences over a range of sample or effect sizes. Additionally, we validated the new measure using three models of experimentally induced hippocampal dysfunction: (1) complete hippocampal lesions, (2) genetic deletion of αCaMKII, a gene implicated in hippocampal behavioral and synaptic plasticity, and (3) a mouse model of Alzheimer's disease. Together, these data indicate that H offers greater sensitivity than existing measures, most likely because it exploits the richness of the precise positional information of the mouse throughout the probe test.

Entropy-Based Distortion Measure and Bit Allocation for Wavelet Image Compression

Quality criteria for image coding are often based on mean square error. However, this is not always a relevant measure of visual quality at low bit rates. Here, we investigate the properties of a distortion measure based on the conditional differential entropy of the input signal given its quantized value. The proposed measure appears to be a correct representation of the amount of information lost by quantization. An adaptive bit allocation algorithm is proposed in order to take advantage of this criterion. Experimental results illustrate the behavior of the proposed distortion measure and exhibit interesting visual properties for low bit-rate subband image coding.

An Entropy-Based Measure for QTL Mapping Using Extreme Samples of Population

Quantitative trait locus (QTL) mapping can be accomplished through the method of selective genotyping, which is based on the differences of frequencies between an upper sample and a lower sample in population. However, amplifying the differences in marker allele frequencies in extreme samples may increase the probability for QTL mapping. Shannon entropy, which is a nonlinear function of allele frequencies, can be used to amplify the differences in marker allele frequencies. In this paper, we present a novel measure for linkage disequilibrium (LD) between a marker and single QTL, that is based on the comparison of the entropy and conditional entropy in a marker in extreme samples of population. This measure of LD between the marker and the trait locus can be used when the marker allele frequencies are known in the extreme samples of a population. We investigate the mapping performance in both analytic and simulation scenarios of a single QTL linked to a single marker. Our results show that the measure has very reasonable performance. In addition, a simulation study is performed on the basis of the haplotype frequencies of 10 SNPs of angiotensin-I converting enzyme (ACE) genes.