Entropy-based Criterion Research Articles

AdaDet: An Adaptive Object Detection System Based on Early-Exit Neural Networks

Recent researchers have proposed adaptive inference methods with an early-exiting mechanism, which stops the inference procedure of input if the prediction is with high confidence, leading to a fine-grained resource allocation based on the complexity of inputs. However, the adaptive inference strategy can only be applied for image classification tasks, such a system for object detection is still under-investigated since it lacks an appropriate way to measure the uncertainty of detection results for multiple objects in an image. To address this issue, in this paper, we propose a novel adaptive object detection system (AdaDet) based on early-exit neural networks. By using stochastic variables to represent localization predictions, our method can measure the uncertainty of the detection result for an object, based on which we further develop an entropy-based criterion to estimate the reliability of the whole detection results for an input image. With the proposed method, samples that achieve high-confidence detection results will exit early from the detection model, leading to low computational costs for inferring these samples. While only complex images need to finish the whole computational graph to achieve better detection results during inference. We experimentally demonstrate that the proposed AdaDet enhances detection performance under anytime prediction and adaptive inference settings on the most common dataset (MS COCO) used in this field.

Smallest unit of maximal entropy as novel experimental criterion for parametric characterization of middle- and high-entropy materials.

Materials with multiple principal elements (middle- and high-entropy materials), are used in emerging applications in various fields due to their unique properties, driven by configuration entropy. Improved understanding and experimental investigations of the impact of the entropy of mixing on the properties of these materials are of large practical interest. Here we show a simplified limited area calculation approach for assessing the entropy of mixing using a CoCuFeNi model nanoalloy. Based on our calculations we propose a new parametric entropy-based criterion, which defines critical scale parameter transition from the maximal entropy state to the entropy-depleted state of the system. The criterion could be used for generalized mechanistic assessment of the effect of the entropy of mixing on the characteristics of the materials with multiple principal elements and for the development and characterization of existing and new middle- and high-entropy materials with both simple single-, and more complex, multiple-sublattice structures.

Adaptive decoder-block selection with filter reweighting for medical image segmentation

Image segmentation models are mostly designed for specific tasks and trained on large-scale datasets. Medical image segmentation tasks, however, are highly diverse and usually have a small training dataset, making it difficult for a segmentation model to generalize well on other tasks. This architecture bias motivates a lot of research on designing a task-adaptive segmentation network with the most suitable settings including the network’s depth and width and feature fusion strategy. In this paper, we propose the adaptive Decoder-block Selection with Filter Reweighting (DeSFiR) algorithm to generate an efficient and effective medical image segmentation model for each specific dataset. This algorithm is a two-step iterative process. In the adaptive decoder-block selection step, we develop an auxiliary indicator function layer to bypass the decoder block whose individual responses are lower than a pre-defined threshold, aiming to generate a lightweight model. In the filter reweighing step, we replace invalid layers in the segmentation network with the external information from a simultaneously trained twin network. To this end, we utilize an entropy-based criterion to measure the informativeness of each layer and an adaptive weighting strategy to balance the replaced information between two networks. We have evaluated our algorithm on three tasks, including the segmentation of neural structures, nuclei, and liver. Our results indicate that the medical image segmentation model generated by the proposed DeSFiR algorithm has a significantly reduced complexity and comparable accuracy.

Entropy-Based Logic Explanations of Neural Networks

Explainable artificial intelligence has rapidly emerged since lawmakers have started requiring interpretable models for safety-critical domains. Concept-based neural networks have arisen as explainable-by-design methods as they leverage human-understandable symbols (i.e. concepts) to predict class memberships. However, most of these approaches focus on the identification of the most relevant concepts but do not provide concise, formal explanations of how such concepts are leveraged by the classifier to make predictions. In this paper, we propose a novel end-to-end differentiable approach enabling the extraction of logic explanations from neural networks using the formalism of First-Order Logic. The method relies on an entropy-based criterion which automatically identifies the most relevant concepts. We consider four different case studies to demonstrate that: (i) this entropy-based criterion enables the distillation of concise logic explanations in safety-critical domains from clinical data to computer vision; (ii) the proposed approach outperforms state-of-the-art white-box models in terms of classification accuracy.

Probabilistic unsupervised classification for large-scale analysis of spectral imaging data

Land cover classification of remote sensing data is a fundamental tool to study changes in the environment such as deforestation or wildfires. A current challenge is to quantify land cover changes with real-time, large-scale data from modern hyper- or multispectral sensors. A range of methods are available for this task, several of them being based on the k-means classification method which is efficient when classes of land cover are well separated. Here a new algorithm, called probabilistic k-means, is presented to solve some of the limitations of the standard k-means. It is shown that the new algorithm performs better than the standard k-means when the data are noisy. If the number of land cover classes is unknown, an entropy-based criterion can be used to select the best number of classes. The proposed new algorithm is implemented in a combination of R and C computer codes which is particularly efficient with large data sets: a whole image with more than 3 million pixels and covering more than 10,000 km2 can be analysed in a few minutes. Four applications with hyperspectral and multispectral data are presented. For the data sets with ground truth data, the overall accuracy of the probabilistic k-means was substantially improved compared to the standard k-means. One of these data sets includes more than 120 million pixels, demonstrating the scalability of the proposed approach. These developments open new perspectives for the large scale analysis of remote sensing data. All computer code are available in an open-source package called sentinel.

Double Thresholding with Sine Entropy for Thermal Image Segmentation

Traditional thresholding methods are often used for image segmentation of real images. However, due to distinct characteristics of infrared thermal images, it is difficult to ensure an optimal image segmentation using the traditional thresholding algorithms, and therefore, sometimes this can lead to over-segmentation, missing object information, and/or spurious responses in the output. To overcome these issues, we propose a new thresholding technique that makes use of the sine entropy-based criterion. Moreover, we build a double thresholding technique that makes use of two thresholds to get the final image thresholding result. Besides, we introduce the sine entropy concept as a supplement of the Shannon entropy in creating threshold-dependent criterion derived from the grayscale histogram. We found that the sine entropy is more robust in interpreting the strength of the long-range correlation in the gray levels compared to the Shannon entropy. We have experimented with our method on several infrared thermal images collected from standard image databases to describe the performance. On comparing with the state-of-art methods, the qualitative results from the experiments show that the proposed method achieves the best performance with an average sensitivity of 0.98 and an average misclassification error of 0.01, and second-best performance with an average sensitivity of 0.99 and an average specificity of 0.93.

Toward Multidiversified Ensemble Clustering of High-Dimensional Data: From Subspaces to Metrics and Beyond

The rapid emergence of high-dimensional data in various areas has brought new challenges to current ensemble clustering research. To deal with the curse of dimensionality, recently considerable efforts in ensemble clustering have been made by means of different subspace-based techniques. However, besides the emphasis on subspaces, rather limited attention has been paid to the potential diversity in similarity/dissimilarity metrics. It remains a surprisingly open problem in ensemble clustering how to create and aggregate a large population of diversified metrics, and furthermore, how to jointly investigate the multilevel diversity in the large populations of metrics, subspaces, and clusters in a unified framework. To tackle this problem, this article proposes a novel multidiversified ensemble clustering approach. In particular, we create a large number of diversified metrics by randomizing a scaled exponential similarity kernel, which are then coupled with random subspaces to form a large set of metric-subspace pairs. Based on the similarity matrices derived from these metric-subspace pairs, an ensemble of diversified base clusterings can be thereby constructed. Furthermore, an entropy-based criterion is utilized to explore the cluster wise diversity in ensembles, based on which three specific ensemble clustering algorithms are presented by incorporating three types of consensus functions. Extensive experiments are conducted on 30 high-dimensional datasets, including 18 cancer gene expression datasets and 12 image/speech datasets, which demonstrate the superiority of our algorithms over the state of the art. The source code is available at https://github.com/huangdonghere/MDEC.

A Damage Detection Method Using Neural Network Optimized by Multiple Particle Collision Algorithm

A critical task of structural health monitoring is damage detection and localization. Lamb wave propagation methods have been successfully applied for damage identification in plate‐like structures. However, Lamb wave processing is still a challenging task due to its multimodal and dispersive characteristics. To address this issue, data‐driven machine learning approaches as artificial neural network (ANN) have been proposed. However, the effectiveness of ANN can be improved based on its architecture and the learning strategy employed to train it. The present paper proposes a Multiple Particle Collision Algorithm (MPCA) to design an optimum ANN architecture to detect and locate damages in plate‐like structures. For the first time in the literature, the MPCA is applied to find damages in plate‐like structures. The present work uses one piezoelectric transducer to generate Lamb wave signals on an aluminum plate structure and a linear array of four transducers to capture the scattered signals. The continuous wavelet transform (CWT) processes the captured signals to estimate the time‐of‐flight (ToF) that is the ANN inputs. The ANN output is the damage spatial coordinates. In addition to MPCA optimization, this paper uses a quantitative entropy‐based criterion to find the best mother wavelet and the scale values. The presented experimental results show that MPCA is capable of finding a simple ANN architecture with good generalization performance in the proposed damage localization application. The proposed method is compared with the 1‐dimensional convolutional neural network (1D‐CNN). A discussion about the advantages and limitations of the proposed method is presented.

Near-equiatomic high-entropy decagonal quasicrystal in Al20Si20Mn20Fe20Ga20

High-entropy alloys (HEAs) contain multiple principal alloying elements, but usually with simple crystal structures. Quasicrystals are structurally complex phases, but are generally dominated by only one element. However, near-equiatomic high-entropy quasicrystals have rarely been reported because they are difficult to prepare experimentally and predict theoretically. Therefore, the preparation and crystal structures of near-equiatomic high-entropy quasicrystals have drawn much interest. We report a quinary decagonal quasicrystal (DQC) with near-equiatomic alloying elements in Al20Si20Mn20Fe20Ga20 melt-spun ribbons, which is the first to our knowledge. Meanwhile, the structural features of the DQC are characterized in detail. The configurational entropy of both the alloy and DQC satisfies the entropy-based criterion for HEAs, suggesting a high-entropy DQC. Our findings provide a new strategy to develop high-entropy quasicrystals.

Automatic Weakly Supervised Object Detection From High Spatial Resolution Remote Sensing Images via Dynamic Curriculum Learning

In this article, we focus on tackling the problem of weakly supervised object detection from high spatial resolution remote sensing images, which aims to learn detectors with only image-level annotations, i.e., without object location information during the training stage. Although promising results have been achieved, most approaches often fail to provide high-quality initial samples and thus are difficult to obtain optimal object detectors. To address this challenge, a dynamic curriculum learning strategy is proposed to progressively learn the object detectors by feeding training images with increasing difficulty that matches current detection ability. To this end, an entropy-based criterion is firstly designed to evaluate the difficulty for localizing objects in images. Then, an initial curriculum that ranks training images in ascending order of difficulty is generated, in which easy images are selected to provide reliable instances for learning object detectors. With the gained stronger detection ability, the subsequent order in the curriculum for retraining detectors is accordingly adjusted by promoting difficult images as easy ones. In such way, the detectors can be well prepared by training on easy images for learning from more difficult ones and thus gradually improve their detection ability more effectively. Moreover, an effective instance-aware focal loss function for detector learning is developed to alleviate the influence of positive instances of bad quality and meanwhile enhance the discriminative information of class-specific hard negative instances. Comprehensive experiments and comparisons with state-of-the-art methods on two publicly available data sets demonstrate the superiority of our proposed method.

Multi-Criteria Decision-Making Approach for Selecting Wind Energy Power Plant Locations

In pursuit of green technology innovations, the energy industry is showing an interest in sustainable sources such as wind energy generation. The Saudi Arabian energy industry has a 2030 target to generate and transmit electricity to major customers nationwide and other neighboring Gulf countries. However, the selection of wind energy power plant locations is a concern because the decision process involves social, technological, economical, and environmental factors. The originality of this study lies in (1) proposing an integrated quantitative and qualitative multi-criteria decision making framework for selecting wind-energy power plant locations; (2) applying the proposed framework in the context of the energy industry in a gulf region country and investigating expert-based and entropy-based criterion weight assignments; (3) choosing five possible alternative wind energy power plant locations with 17 response criteria for each alternative to help decision makers identify the best possible alternative; and (4) establishing the superiority of one alternative over another (if it exists). The presented approach extends considerable support to the comparing and ranking of alternatives along with its validation and sensitivity analysis. Based on the proposed multi-criteria decision-making approach, an appropriate wind energy power plant location has been successfully selected among the five alternatives.

Quantifying the non-equilibrium characteristics of heterogeneous gas–solid flow of smooth, inelastic spheres using a computational fluid dynamics–discrete element method

Continuum modelling of dense gas–solid flows strongly depends on the constitutive relations used, including the interphase drag force, particle phase stress and the boundary condition for particle–wall interactions. The lack of scale separation is usually claimed to cause the breakdown of the Navier–Stokes (NS) order continuum theory. In this study, computational fluid dynamics–discrete element method (CFD-DEM) simulations of bubbling, turbulent and fast fluidization of smooth, inelastic spheres were conducted to systematically analyse the valid range of NS theory. An entropy-based criterion $I_{s}$ and Knudsen numbers defined using different characteristic length scales ($Kn_{frac}$, $Kn_{gran}$ and $Kn_{vel}$) were quantified. It was found that (i) except at the centre of bubbles where the solid concentration is quite low, NS theory for discrete particles was valid for bubbling fluidization irrespective of the breakdown criterion. Even at the boundary of the bubbles, values of $I_{s}$ and $Kn$ were small; and (ii) the conclusion depended on the criterion used in turbulent and fast fluidization. If $Kn_{frac}$ was used, NS theory would be generally valid. If $I_{s}$ was chosen, NS theory would be still valid but with lower confidence. However, if $Kn_{vel}$ or $Kn_{gran}$ was selected, NS theory broke down. Because $I_{s}$ includes the non-equilibrium effects caused by the gradient of hydrodynamic fields and particle inelasticity, we may conclude that NS theory was valid for all tested cases. This means that the continuum description of discrete particles is not the main source of the breakdown of NS theory.

A generalized Masi entropy based efficient multilevel thresholding method for color image segmentation

Multilevel thresholding for image segmentation is a crucial process in several applications such as feature extraction and pattern recognition. In this paper, a novel Masi entropy-based criterion f...

A sequential ensemble clusterings generation algorithm for mixed data

Ensemble clustering has attracted much attention for its robustness, stability, and accuracy in academic and industry communities. In order to yield base clusterings with high quality and diversity simultaneously in ensemble clustering, many efforts have been done by exploiting different clustering models and data information. However, these methods neglect correlation between different base clusterings during the process of base clusterings generation, which is important to obtain a quality and diverse clustering decision. To overcome this deficiency, a sequential ensemble clusterings generation algorithm for mixed data is developed in this paper based on information entropy. The first high quality base clustering is yield by maximizing the entropy-based criterion. Afterward, a sequential paradigm is utilized to incrementally find more base clusterings, in which the diversity between a new base clustering and the former base partitions is measured by the normalized mutual information. Extensive experiments conducted on various data sets have demonstrated the superiority of our proposal as compared to several existing base clusterings generation algorithms.

Minimum aberration criterion for screening experiments at two levels from an entropy-based perspective

ABSTRACTThis paper presents a new criterion for selecting a two-level fractional factorial design. The theoretical underpinning for the criterion is the Shannon entropy. The criterion, which is referred to as the entropy-based minimum aberration criterion, has several advantages. The advantage of the entropy-based criterion over the classical minimum aberration criterion is that it utilizes a measure of uncertainty on the skewness of the distribution of word length patterns in the selection of the “best” design in a family of two-level fractional factorial plans. The criterion evades the trauma associated with the lack of prior knowledge on the important effects.

Robust and efficient Bayesian adaptive psychometric function estimation

The efficient measurement of the threshold and slope of the psychometric function (PF) is an important objective in psychoacoustics. This paper proposes a procedure that combines a Bayesian estimate of the PF with either a look one-ahead or a look two-ahead method of selecting the next stimulus presentation. The procedure differs from previously proposed algorithms in two respects: (i) it does not require the range of possible PF parameters to be specified in advance and (ii) the sequence of probe signal-to-noise ratios optimizes the threshold and slope estimates at a performance level, ϕ, that can be chosen by the experimenter. Simulation results show that the proposed procedure is robust and that the estimates of both threshold and slope have a consistently low bias. Over a wide range of listener PF parameters, the root-mean-square errors after 50 trials were ∼1.2 dB in threshold and 0.14 in log-slope. It was found that the performance differences between the look one-ahead and look two-ahead methods were negligible and that an entropy-based criterion for selecting the next stimulus was preferred to a variance-based criterion.

Enhancing mobile sensor selection in large participatory sensing systems

The large number of sensor-rich mobile devices that are commonly available in urban environments bring new venues for the study of the environment and human interactions. Better approaches for the exploitation of the spatial-temporal properties of mobile communities in self-selected social-sensing networks are still needed, in particular, for very large participatory sensing systems where incentives and rewards create a cost for the system. We introduce an entropy-based criterion to evaluate the measurement diversity that can be achieved by a mobile group of sensors and propose a parallel randomised search algorithm to find the best group composition. The problem is non-trivial as the aggregated group entropy does not equal the sum of individual contributions. We discuss applications of the approach to achieve area coverage and anomaly tracking and show that the size of mobile sensor groups can be reduced to minimise costs without compromising the quality of the sensing objective.

Optimization of quadratic time–frequency distributions using the local Rényi entropy information

This paper presents two TFD optimization schemes, extending an optimization method so far limited to the spectrogram. The first method is focused on the enhancement of the spectrogram concentration, by an adaptive realization of the S-method TFD which prevents cross-terms generation. The second approach, a generalization of the first one to the Quadratic class of TFDs, operates on a set of TFDs with different kernel parameters, selecting for each time instant the best performing one. Both methods use an entropy-based criterion for concentration enhancement, and a here-proposed method for the detection of cross-terms. The combination of the two criteria allows us to generate optimal TFDs, i.e. TFDs with highly concentrated components, but at the same time avoiding the appearance of undesirable cross-terms in the resulting TFD.

Latent class analysis of incomplete data via an entropy-based criterion

Latent class analysis is used to group categorical data into classes via a probability model. Model selection criteria then judge how well the model fits the data. When addressing incomplete data, the current methodology restricts the imputation to a single, pre-specified number of classes. We seek to develop an entropy-based model selection criterion that does not restrict the imputation to one number of clusters. Simulations show the new criterion performing well against the current standards of AIC and BIC, while a family studies application demonstrates how the criterion provides more detailed and useful results than AIC and BIC.

System Identification with a MinMax Entropy-based Criterion

Abstract: A minmax problem statement of a problem of the parameter identification of the input/output mapping of systems is considered, based on the entropy (in the Shannon sense) of the identification error. Using the entropy based approach, a Gauss closeness measure is proposed, that simultaneously may serve as a simple test for the normality for any continuous probability distribution, suitable for applications within depended observations. A case study is considered, with confirming the full practical suitability of the theoretical considerations of the paper.