Entropy Regularization Research Articles

Positive-Negative Receptive Field Reasoning for Omni-Supervised 3D Segmentation.

Hidden features in the neural networks usually fail to learn informative representation for 3D segmentation as supervisions are only given on output prediction, while this can be solved by omni-scale supervision on intermediate layers. In this paper, we bring the first omni-scale supervision method to 3D segmentation via the proposed gradual Receptive Field Component Reasoning (RFCR), where target Receptive Field Component Codes (RFCCs) is designed to record categories within receptive fields for hidden units in the encoder. Then, target RFCCs will supervise the decoder to gradually infer the RFCCs in a coarse-to-fine categories reasoning manner, and finally obtain the semantic labels. To purchase more supervisions, we also propose an RFCR-NL model with complementary negative codes (i.e., Negative RFCCs, NRFCCs) with negative learning. Because many hidden features are inactive with tiny magnitudes and make minor contributions to RFCC prediction, we propose Feature Densification with a centrifugal potential to obtain more unambiguous features, and it is in effect equivalent to entropy regularization over features. More active features can unleash the potential of omni-supervision method. We embed our method into three prevailing backbones, which are significantly improved in all three datasets on both fully and weakly supervised segmentation tasks and achieve competitive performances.

Weakly supervised semantic segmentation of mobile laser scanning point clouds via category balanced random annotation and deep consistency‐guided self‐distillation mechanism

AbstractScene understanding of mobile laser scanning (MLS) point clouds is vital in autonomous driving and virtual reality. Most existing semantic segmentation methods rely on a large number of accurately labelled points, which is time‐consuming and labour‐intensive. To cope with this issue, this paper explores a weakly supervised learning (WSL) framework for MLS data. Specifically, a category balanced random annotation (CBRA) strategy is employed to obtain balanced labels and enhance model performance. Next, based on KPConv‐Net as a backbone network, a WSL semantic segmentation framework is developed for MLS point clouds via a deep consistency‐guided self‐distillation (DCS) mechanism. The DCS mechanism consists of a deep consistency‐guided self‐distillation branch and an entropy regularisation branch. The self‐distillation branch is designed by constructing an auxiliary network to maintain the consistency of predicted distributions between the auxiliary network and the original network, while the entropy regularisation branch is designed to increase the confidence of the network predicted results. The proposed WSL framework was evaluated on the WHU‐MLS, NPM3D and Toronto3D datasets. By using only 0.1% labelled points, the proposed WSL framework achieved a competitive performance in MLS point cloud semantic segmentation with the mean Intersection over Union (mIoU) scores of 60.08%, 72.0% and 67.42% on the three datasets, respectively.

Deep multi-view fuzzy k-means with weight allocation and entropy regularization

Deep multi-view fuzzy k-means with weight allocation and entropy regularization

An information–Theoretic approach to partially identified auction models

We consider a situation in which we have data from ascending auctions with symmetric bidders, independent private values, and exogenous entry in which the bidders’ value distribution is partially identified. Focusing on the case in which the seller intends to use a second price auction, we discuss how to determine an optimal reserve price. We justify the use of maximum entropy, explore the properties of the estimand, determine the asymptotic properties of our maximum entropy estimator, evaluate its behavior in a simulation study, and demonstrate its use in a modest application. As an extension, we propose a maxmin decision rule with entropy regularization, which includes Aryal and Kim (2013) and the maximum entropy solution as extreme cases.

Possibilistic distribution distance metric: a robust domain adaptation learning method.

The affective Brain-Computer Interface (aBCI) systems, which achieve predictions for individual subjects through training on multiple subjects, often cannot achieve satisfactory results due to the differences in Electroencephalogram (EEG) patterns between subjects. One tried to use Subject-specific classifiers, but there was a lack of sufficient labeled data. To solve this problem, Domain Adaptation (DA) has recently received widespread attention in the field of EEG-based emotion recognition. Domain adaptation (DA) learning aims to solve the problem of inconsistent distributions between training and test datasets and has received extensive attention. Most existing methods use Maximum Mean Discrepancy (MMD) or its variants to minimize the problem of domain distribution inconsistency. However, noisy data in the domain can lead to significant drift in domain means, which can affect the adaptability performance of learning methods based on MMD and its variants to some extent. Therefore, we propose a robust domain adaptation learning method with possibilistic distribution distance measure. Firstly, the traditional MMD criterion is transformed into a novel possibilistic clustering model to weaken the influence of noisy data, thereby constructing a robust possibilistic distribution distance metric (P-DDM) criterion. Then the robust effectiveness of domain distribution alignment is further improved by a fuzzy entropy regularization term. The proposed P-DDM is in theory proved which be an upper bound of the traditional distribution distance measure method MMD criterion under certain conditions. Therefore, minimizing P-DDM can effectively optimize the MMD objective. Secondly, based on the P-DDM criterion, a robust domain adaptation classifier based on P-DDM (C-PDDM) is proposed, which adopts the Laplacian matrix to preserve the geometric consistency of instances in the source domain and target domain for improving the label propagation performance. At the same time, by maximizing the use of source domain discriminative information to minimize domain discrimination error, the generalization performance of the learning model is further improved. Finally, a large number of experiments and analyses on multiple EEG datasets (i.e., SEED and SEED-IV) show that the proposed method has superior or comparable robustness performance (i.e., has increased by around 10%) in most cases.

Self-Learned Autonomous Driving at Unsignalized Intersections: A Hierarchical Reinforced Learning Approach for Feasible Decision-Making

Reinforcement learning-based techniques, empowered by deep-structured neural nets, have demonstrated superiority over rule-based methods in terms of making high-level behavioral decisions due to qualities related to handling large state spaces. Nonetheless, their training time, sample efficiency and the feasibility of the learnt behaviors remain key concerns. In this paper, we propose a novel hierarchical reinforcement learning-based decision-making architecture for learning left-turn policies at unsignalized intersections with feasibility guarantees. The proposed technique is comprised of two layers; a high-level learning-based behavioral planning layer which adopts soft actor-critic (SAC) principles to learn high-level, non-conservative yet safe, driving behaviors, and a low-level motion planning layer that uses Model Predictive Control (MPC) framework to ensure feasibility of the two-dimensional left-turn maneuver. The high-level layer generates reference signals of velocity and yaw angles for the ego vehicle taking into account safety and collision avoidance with the intersection vehicles, whereas the low-level motion planning layer solves an optimization problem to track these reference commands taking into account several vehicle dynamic constraints and ride comfort. While training the behavioral SAC-based planning layer, We develop an adaptive entropy regularization technique that results in faster convergence, higher mean rewards, and lower collision rates. We validate the proposed decision-making scheme in simulated environments and compare with other model free Reinforcement Learning (RL) baselines. The results demonstrate that the proposed integrated framework possesses better training and navigation capabilities.

Non-iterative scribble-supervised learning with pacing pseudo-masks for medical image segmentation

Scribble-supervised medical image segmentation tackles the limitation of sparse masks. Conventional approaches alternate between: labeling pseudo-masks and optimizing network parameters. However, such iterative two-stage paradigm is unwieldy and could be trapped in poor local optima since the networks undesirably regress to the erroneous pseudo-masks. To address these issues, we propose a non-iterative method where a stream of varying (pacing) pseudo-masks teach a network via consistency training, named PacingPseudo. Our contributions are summarized as follows. First, we design a non-iterative process. This process is achieved gracefully by a siamese architecture that comprises two weight-sharing networks. The siamese architecture naturally allows a stream of pseudo-masks to assimilate a stream of predicted-masks during training. Second, we make the consistency training effective with two necessary designs: (i) entropy regularization to obtain high-confidence pseudo-masks for effective teaching; and (ii) distorted augmentations to create discrepancy between the pseudo-mask and predicted-mask streams for consistency regularization. Third, we devise a new memory bank mechanism that provides an extra source of ensemble features to complement scarce labeled pixels. We evaluate the proposed PacingPseudo on public abdominal organ, cardiac structure, and myocardium datasets, named CHAOS T1&T2, ACDC, and LVSC. Evaluation metrics include the Dice similarity coefficient (DSC) and the 95th percentile of Hausdorff distance (HD95). Experimental results show that PacingPseudo achieves a 68.0% DSC and 14.1 mm HD95 on CHAOS T1, 73.7% DSC and 12.2 mm HD95 on CHAOS T2, 82.9% DSC and 4.3 mm HD95 on ACDC, and 61.4% DSC and 11.9 mm HD95 on LVSC. These results improve the baseline method by ≥3.1% in DSC and ≥14.2 mm in HD95. These results also outcompete previous methods. The fully-supervised method attains a 67.0% DSC and 16.7 mm HD95 on CHAOS T1, 71.2% DSC and 12.6 mm HD95 on CHAOS T2, 84.0% DSC and 3.9 mm HD95 on ACDC, and 72.9% DSC and 7.6 mm HD95 on LVSC. PacingPseudo’s performance is comparable to the fully-supervised method on CHAOS T1&T2 and ACDC. Overall, the above results demonstrate the feasibility of PacingPseudo for the challenging scribble-supervised segmentation tasks. The source code is publicly available athttps://github.com/zefanyang/pacingpseudo.

Trust-aware conditional adversarial domain adaptation with feature norm alignment

Adversarial learning has proven to be an effective method for capturing transferable features for unsupervised domain adaptation. However, some existing conditional adversarial domain adaptation methods assign equal importance to different samples, ignoring the fact that hard-to-transfer samples might damage the conditional adversarial adaptation procedure. Meanwhile, some methods can only roughly align marginal distributions across domains, but cannot ensure category distributions alignment, causing classifiers to make uncertain or even wrong predictions for some target data. Furthermore, we find that the feature norms of real images usually follow a complex distribution, so directly matching the mean feature norms of two domains cannot effectively reduce the statistical discrepancy of feature norms and may potentially induce feature degradation.In this paper, we develop a Trust-aware Conditional Adversarial Domain Adaptation (TCADA) method for solving the aforementioned issues. To quantify data transferability, we suggest utilizing posterior probability modeled by a Gaussian-uniform mixture, which effectively facilitates conditional domain alignment. Based on this posterior probability, a confidence-guided alignment strategy is presented to promote precise alignment of category distributions and accelerate the learning of shared features. Moreover, a novel optimal transport-based strategy is introduced to align the feature norms and facilitate shared features becoming more informative. To encourage classifiers to make more accurate predictions for target data, we also design a mixed information-guided entropy regularization term to promote deep features being away from the decision boundaries. Extensive experiments show that our method greatly improves transfer performance on various tasks.

Three Fuzzy c-Shapes Clustering Algorithms for Series Data

Various fuzzy clustering algorithms have been proposed for vectorial data. However, most of these methods have not been applied to series data. This study presents three fuzzy clustering algorithms for series data based on shape-based distances. The first algorithm involves Shannon entropy regularization of the k-shape objective function. The second algorithm is similar to the revised Bezdek-type fuzzy c-means algorithm obtained by replacing the membership of the hard c-means objective function with its power. The third algorithm involves Tsallis entropy regularization of the objective function of the second algorithm. Theoretical observations revealed that the third algorithm is a generalization of the first and second algorithms, which was validated by numerical experiments. Furthermore, numerical experiments were performed using 11 benchmark datasets to demonstrate that the third algorithm outperforms the others in terms of accuracy.

An Entropy Weighted Nonnegative Matrix Factorization Algorithm for Feature Representation.

Nonnegative matrix factorization (NMF) has been widely used to learn low-dimensional representations of data. However, NMF pays the same attention to all attributes of a data point, which inevitably leads to inaccurate representations. For example, in a human-face dataset, if an image contains a hat on a head, the hat should be removed or the importance of its corresponding attributes should be decreased during matrix factorization. This article proposes a new type of NMF called entropy weighted NMF (EWNMF), which uses an optimizable weight for each attribute of each data point to emphasize their importance. This process is achieved by adding an entropy regularizer to the cost function and then using the Lagrange multiplier method to solve the problem. Experimental results with several datasets demonstrate the feasibility and effectiveness of the proposed method. The code developed in this study is available at https://github.com/Poisson-EM/Entropy-weighted-NMF.

Entropy regularization in probabilistic clustering

Bayesian nonparametric mixture models are widely used to cluster observations. However, one major drawback of the approach is that the estimated partition often presents unbalanced clusters’ frequencies with only a few dominating clusters and a large number of sparsely-populated ones. This feature translates into results that are often uninterpretable unless we accept to ignore a relevant number of observations and clusters. Interpreting the posterior distribution as penalized likelihood, we show how the unbalance can be explained as a direct consequence of the cost functions involved in estimating the partition. In light of our findings, we propose a novel Bayesian estimator of the clustering configuration. The proposed estimator is equivalent to a post-processing procedure that reduces the number of sparsely-populated clusters and enhances interpretability. The procedure takes the form of entropy-regularization of the Bayesian estimate. While being computationally convenient with respect to alternative strategies, it is also theoretically justified as a correction to the Bayesian loss function used for point estimation and, as such, can be applied to any posterior distribution of clusters, regardless of the specific model used.

Efficient Information-Theoretic Large-Scale Semi-Supervised Metric Learning via Proxies

Semi-supervised metric learning intends to learn a distance function from the limited labeled data as well as a large amount of unlabeled data to better gauge the similarities of any two instances than using a general distance function. However, most existing semi-supervised metric learning methods rely on the manifold assumptions to mine the rich discriminant information of the unlabeled data, which breaks the intrinsic connection between the manifold regularizer-building process and the subsequent metric learning. Moreover, these methods usually encounter high computational or memory overhead. To solve these issues, we develop a novel method entitled Information-Theoretic Large-Scale Semi-Supervised Metric Learning via Proxies (ISMLP). ISMLP aims to simultaneously learn multiple proxy vectors as well as a Mahalanobis matrix and forms the semi-supervised metric learning as the probability distribution optimization parameterized by the Mahalanobis distance between the instance and each proxy vector. ISMLP maximizes the entropy of the labeled data and minimizes that of the unlabeled data to follow the entropy regularization, in this way, the labeled part and unlabeled part can be integrated in a meaningful way. Furthermore, the time complexity of the proposed method has a linear dependency concerning the number of instances, thereby, can be extended to the large-scale dataset without incurring too much time. Experiments on multiple datasets demonstrate the superiority of the proposed method over the compared methods used in the experiments.

Fuzzy clustering of spatial interval-valued data

In this paper, two fuzzy clustering methods for spatial interval-valued data are proposed, i.e. the fuzzy C-Medoids clustering of spatial interval-valued data with and without entropy regularization. Both methods are based on the Partitioning Around Medoids (PAM) algorithm, inheriting the great advantage of obtaining non-fictitious representative units for each cluster.In both methods, the units are endowed with a relation of contiguity, represented by a symmetric binary matrix. This can be intended both as contiguity in a physical space and as a more abstract notion of contiguity. The performances of the methods are proved by simulation, testing the methods with different contiguity matrices associated to natural clusters of units. In order to show the effectiveness of the methods in empirical studies, three applications are presented: the clustering of municipalities based on interval-valued pollutants levels, the clustering of European fact-checkers based on interval-valued data on the average number of impressions received by their tweets and the clustering of the residential zones of the city of Rome based on the interval of price values.

A discriminative information-theoretical analysis of the regularity gradient in inflectional morphology

Over the last decades, several independent lines of research in morphology have questioned the hypothesis of a direct correspondence between sublexical units and their mental correlates. Word and paradigm models of morphology shifted the fundamental part-whole relation in an inflection system onto the relation between individual inflected word forms and inflectional paradigms. In turn, the use of artificial neural networks of densely interconnected parallel processing nodes for morphology learning marked a radical departure from a morpheme-based view of the mental lexicon. Lately, in computational models of Discriminative Learning, a network architecture has been combined with an uncertainty reducing mechanism that dispenses with the need for a one-to-one association between formal contrasts and meanings, leading to the dissolution of a discrete notion of the morpheme.The paper capitalises on these converging lines of development to offer a unifying information-theoretical, simulation-based analysis of the costs incurred in processing (ir)regularly inflected forms belonging to the verb systems of English, German, French, Spanish and Italian. Using Temporal Self-Organising Maps as a computational model of lexical storage and access, we show that a discriminative, recurrent neural network, based on Rescorla-Wagner’s equations, can replicate speakers’ exquisite sensitivity to widespread effects of word frequency, paradigm entropy and morphological (ir)regularity in lexical processing. The evidence suggests an explanatory hypothesis linking Word and paradigm morphology with principles of information theory and human perception of morphological structure. According to this hypothesis, the ways more or less regularly inflected words are structured in the mental lexicon are more related to a reduction in processing uncertainty and maximisation of predictive efficiency than to economy of storage.

Intelligent Resource Allocation in Backscatter-NOMA Networks: A Soft Actor Critic Framework

With the use of power domain non-orthogonal multiple access (NOMA) and backscatter communication (BAC), future sixth-generation ultra massive machine-type communications networks are expected to connect large-scale internet of things (IoT) devices. However, due to NOMA co-channel interference, the power allocation to large-scale IoT devices becomes critical. The existing convex optimization-based solutions are highly complex, and therefore it is difficult to find the optimal solution to the resource allocation problem in a highly dynamic environment. To alleviate this problem, this work develops an efficient model-free BAC approach with a NOMA system to assist the base station with complex resource scheduling tasks in a dynamic BAC-NOMA IoT network. The objective is to increase the sum rate of uplink backscatter devices. More specifically, we jointly optimize the transmit power of downlink IoT users and the reflection coefficient of uplink backscatter devices using a reinforcement learning algorithm, namely, the soft-actor critic (SAC) algorithm. With the advantage of entropy regularization, the SAC agent learns to explore and exploit the dynamic BAC-NOMA network efficiently. The proposed algorithm ensures the quality of service (QoS) requirements of downlink users while enhancing the sum rate of uplink backscatter devices. Numerical results reveal the superiority of the proposed algorithm over the conventional optimization (benchmark) approach in terms of the average sum rate of uplink backscatter devices. We show that the network with multiple downlink users obtained a higher reward with respect to a large number of iterations compared to episodes with a lower number of iterations. Moreover, the proposed algorithm outperforms the benchmark scheme and BAC with orthogonal multiple access in terms of the average sum rate with the different number of backscatter devices. Additionally, we show that our proposed algorithm enhances sum rate efficiency with respect to different self-interference coefficients and different noise levels. Finally, we evaluate and show the sum rate efficiency of the proposed algorithm with different QoS requirements and cell radii.

Convergence Rates for Regularized Optimal Transport via Quantization

We study the convergence of divergence-regularized optimal transport as the regularization parameter vanishes. Sharp rates for general divergences including relative entropy or Lp regularization, general transport costs, and multimarginal problems are obtained. A novel methodology using quantization and martingale couplings is suitable for noncompact marginals and achieves, in particular, the sharp leading-order term of entropically regularized 2-Wasserstein distance for marginals with a finite [Formula: see text]-moment. Funding: This work was supported by the Alfred P. Sloan Foundation [Grant FG-2016-6282] and the Division of Mathematical Sciences [Grants DMS-1812661 and DMS-2106056].

Survival Prediction After Transarterial Chemoembolization for Hepatocellular Carcinoma: a Deep Multitask Survival Analysis Approach.

The accurate prediction of postoperative survival time of patients with Barcelona Clinic Liver Cancer (BCLC) stage B hepatocellular carcinoma (HCC) is important for postoperative health care. Survival analysis is a common method used to predict the occurrence time of events of interest in the medical field. At present, the mainstream survival analysis models, such as the Cox proportional risk model, should make strict assumptions about the potential random process to solve the censored data, thus potentially limiting their application in clinical practice. In this paper, we propose a novel deep multitask survival model (DMSM) to analyze HCC survival data. Specifically, DMSM transforms the traditional survival time prediction problem of patients with HCC into a survival probability prediction problem at multiple time points and applies entropy regularization and ranking loss to optimize a multitask neural network. Compared with the traditional methods of deleting censored data and strong hypothesis, DMSM makes full use of all the information in the censored data but does not need to make any assumption. In addition, we identify the risk factors affecting the prognosis of patients with HCC and visualize the importance of ranking these factors. On the basis of the analysis of a real dataset of patients with BCLC stage B HCC, experimental results on three different validation datasets show that the DMSM achieves competitive performance with concordance index of 0.779, 0.727, and 0.780 and integrated Brier score (IBS) of 0.172, 0.138, and 0.135, respectively. Our DMSM has a comparatively small standard deviation (0.002, 0.002, and 0.003) for IBS of bootstrapping 100 times. The DMSM we proposed can be utilized as an effective survival analysis model and provide an important means for the accurate prediction of postoperative survival time of patients with BCLC stage B HCC.

UNIT: A unified metric learning framework based on maximum entropy regularization

UNIT: A unified metric learning framework based on maximum entropy regularization

User-Controllable Arbitrary Style Transfer via Entropy Regularization

Ensuring the overall end-user experience is a challenging task in arbitrary style transfer (AST) due to the subjective nature of style transfer quality. A good practice is to provide users many instead of one AST result. However, existing approaches require to run multiple AST models or inference a diversified AST (DAST) solution multiple times, and thus they are either slow in speed or limited in diversity. In this paper, we propose a novel solution ensuring both efficiency and diversity for generating multiple user-controllable AST results by systematically modulating AST behavior at run-time. We begin with reformulating three prominent AST methods into a unified assign-and-mix problem and discover that the entropies of their assignment matrices exhibit a large variance. We then solve the unified problem in an optimal transport framework using the Sinkhorn-Knopp algorithm with a user input ε to control the said entropy and thus modulate stylization. Empirical results demonstrate the superiority of the proposed solution, with speed and stylization quality comparable to or better than existing AST and significantly more diverse than previous DAST works. Code is available at https://github.com/cplusx/eps-Assign-and-Mix.

Entropy Regularization for Population Estimation

Entropy regularization is known to improve exploration in sequential decision-making problems. We show that this same mechanism can also lead to nearly unbiased and lower-variance estimates of the mean reward in the optimize-and-estimate structured bandit setting. Mean reward estimation (i.e., population estimation) tasks have recently been shown to be essential for public policy settings where legal constraints often require precise estimates of population metrics. We show that leveraging entropy and KL divergence can yield a better trade-off between reward and estimator variance than existing baselines, all while remaining nearly unbiased. These properties of entropy regularization illustrate an exciting potential for bringing together the optimal exploration and estimation literature.