Minimum Mutual Information Research Articles

Avoiding Shortcut-Learning by Mutual Information Minimization in Deep Learning-Based Image Processing

Avoiding Shortcut-Learning by Mutual Information Minimization in Deep Learning-Based Image Processing

Deep Margin-Sensitive Representation Learning for Cross-Domain Facial Expression Recognition

Cross-domain Facial Expression Recognition (FER) aims to safely transfer the learned knowledge from labeled source data to unlabeled target data, which is challenging due to the subtle difference between various expressions and the large discrepancy between domains. Existing methods mainly focus on reducing the domain shift for transferable features but fail to learn discriminative representations for recognizing facial expression, which may result in negative transfer under cross-domain settings. To this end, we propose a novel Deep Margin-Sensitive Representation Learning (DMSRL) framework, which can extract multi-level discriminative features during sematic-aware domain adaptation. Specifically, we design a semantic metric learning module based on the category prior of source data and generated pseudo labels of target data, which can facilitate discriminative intra-domain representation learning and transferable inter-domain knowledge discovery by enlarging the category margin. Moreover, we develop a mutual information minimization module by simultaneously distilling the domain-invariant components and eliminating the domain-sensitive ones, which benefits discriminative transferable feature learning by generating accurate pseudo target labels. Furthermore, instead of only utilizing the global features, we formulate a multi-level feature extracting module to concurrently get the local ones, which contain detailed information to distinguish the small changes among different expressions. These modules are jointly utilized in our DMSRL in an end-to-end manner to ensure the positive transfer of source knowledge. Extensive experimental results on seven databases demonstrate that our DMSRL can achieve superior performance against state-of-the-art baselines.

Speaker-Independent Emotional Voice Conversion via Disentangled Representations

Emotional Voice Conversion (EVC) technology aims to transfer emotional state in speech while keeping the linguistic information and speaker identity unchanged. Prior studies on EVC have been limited to perform the conversion for a specific speaker or a predefined set of multiple speakers seen in the training stage. When encountering arbitrary speakers that may be unseen during training (outside the set of speakers used in training), existing EVC methods have limited conversion capabilities. However, converting the emotion of arbitrary speakers, even those unseen during the training procedure, in one model is much more challenging and much more attractive in real-world scenarios. To address this problem, in this study, we propose SIEVC, a novel speaker-independent emotional voice conversion framework for arbitrary speakers via disentangled representation learning. The proposed method employs the autoencoder framework to disentangle the emotion information and emotion-independent information of each input speech into separated representation spaces. To achieve better disentanglement, we incorporate mutual information minimization into the training process. In addition, adversarial training is applied to enhance the quality of the generated audio signals. Finally, speaker-independent EVC for arbitrary speakers could be achieved by only replacing the emotion representations of source speech with the target ones. The experimental results demonstrate that the proposed EVC model outperforms the baseline models in terms of objective and subjective evaluation for both seen and unseen speakers.

Robust blind separation of smooth graph signals using minimization of graph regularized mutual information

The smoothness of the graph signals on predefined/constructed graphs appears in many natural applications of processing unstructured (i.e., graph-based) data. In the case of latent sources being smooth graph signals, blind source separation (BSS) quality can be significantly improved by exploiting graph signal smoothness along with the classic measures of statistical independence. In this paper, we propose a BSS method benefiting from the minimization of mutual information as a well-known independence criterion and also graph signal smoothness term of the estimated latent sources, and show that its performance is superior and fairly robust to the state-of-the-art classic and Graph Signal Processing (GSP)-based methods, even in the case of the smooth graph signal sources being heavily corrupted. In addition to the comprehensive complexity and convergence analysis, the asymptotic performance of the proposed method is comprehensively compared with that of the Cramér-Rao Bound (CRB) to illustrate the asymptotic efficiency of the proposed method. Besides, the comprehensive real-world analysis of separating mixed images and speech signals illustrates the applicability and efficiency of the proposed method in real-world scenarios.

Knowledge graph completion using topological correlation and multi-perspective independence

Knowledge graphs (KGs) are large-scale semantic networks designed to describe real-world facts. Existing KGs contain only a fraction of what is really happening in the real life. Knowledge graph completion (KGC) has attracted attention because it can automatically infer and predict missing links in the KGs. The classical geometric, tensor decomposition and convolutional neural network (CNN) based models generate expressive feature embeddings, but these models treat triples independently and thus fail to cover the hidden information that is inherently implicit in the local neighborhood surrounding a triple. Recent works have introduced Graph Convolutional Network (GCN) into KGC task for leveraging the rich structural information in complex graphs. However, existing GCN-based models have limitations in relational topology distinction and multi-perspective feature aggregation. As multi-relational graphs, KGs display intrinsic heterogeneous structures and rich entity types. Therefore, ignoring the rich structural information and perspective features will greatly limit the expressive power of these models. In this paper, we propose multi-relational GCNs for modeling topological correlation and multi-perspective independence (CorIn). Specifically, we propose a multi-integration ring relational topology to fine-grained select neighbors according to relational correlation patterns, and capture the independence of multi-perspective feature groups by mutual information minimization. Our model adaptively utilizes an embedding learning design that can leverage a variety of entity-relation composition operations from classical KGC models. We evaluate our model on the public benchmark datasets and achieve marked performance gains in comparison to state-of-the-art methods.

Waveform Design for Multi-Target Detection Based on Two-Stage Information Criterion.

Parameter estimation accuracy and average sample number (ASN) reduction are important to improving target detection performance in sequential hypothesis tests. Multiple-input multiple-output (MIMO) radar can balance between parameter estimation accuracy and ASN reduction through waveform diversity. In this study, we propose a waveform design method based on a two-stage information criterion to improve multi-target detection performance. In the first stage, the waveform is designed to estimate the target parameters based on the criterion of single-hypothesis mutual information (MI) maximization under the constraint of the signal-to-noise ratio (SNR). In the second stage, the objective function is designed based on the criterion of MI minimization and Kullback–Leibler divergence (KLD) maximization between multi-hypothesis posterior probabilities, and the waveform is chosen from the waveform library of the first-stage parameter estimation. Furthermore, an adaptive waveform design algorithm framework for multi-target detection is proposed. The simulation results reveal that the waveform design based on the two-stage information criterion can rapidly detect the target direction. In addition, the waveform design based on the criterion of dual-hypothesis MI minimization can improve the parameter estimation performance, whereas the design based on the criterion of dual-hypothesis KLD maximization can improve the target detection performance.

Partial Alignment for Object Detection in the Wild

Conventional object detectors often encounter remarkable performance drops due to the domain shift caused by environmental changes. However, labeling sufficient training data drawn from various domains is cost-ineffective and labor-intensive. To this end, unsupervised domain adaptive object detection (DAOD) has attracted much attention, in which the detector is transferred from the label-rich source domain to a label-agnostic target domain. Most of the existing cross-domain object detectors are designed with a parameter-shared network architecture, which, however, has an inherent flaw to accumulate the source errors caused by the inaccurate target distribution. As a result, the target data instead of decisive source data may dominate the learning towards model collapse. To overcome the risk, we propose an Asymmetric Tri-way Faster-RCNN (ATF) for DAOD tasks, in which a novel ancillary net is deployed with the chief net and it brings two merits. 1) The ancillary net enables the source distribution to be preserved without accumulating source error/distortion, such that the target distribution is dominated by source data and the model collapse is alleviated. 2) Ancillary target features can be generated by the ancillary net, which further enhances the discrimination of the target domain object detector. Furthermore, in order to remove the useless source-specific knowledge and exploit the informative domain-invariant knowledge, we propose a Partial Alignment based ATF (PA-ATF) model, in which an adversarial shuffling based mutual-information minimization strategy for domain disentanglement is provided. Intuitively, transferring only the source-domain invariant information to the target domain is more conscionable. Extensive experiments on benchmark datasets, including the Cityscapes, Foggy Cityscapes, Pascal VOC, Clipart, Watercolor, SIM10K, and KITTI demonstrate the remarkable performance of our models over other state-of-the-art approaches.

One-shot emotional voice conversion based on feature separation

The task of emotional voice conversion (EVC) aims to convert speech from one emotional state into another, while keeping linguistic content, speaker identity and other emotion-independent information unchanged. Because previous studies were limited to a specific set of emotions, it is challenging to realize the conversion of emotions never seen in training stage. In this paper, we propose a one-shot emotional voice conversion model based on feature separation. The proposed method could control emotional characteristics with Global Emotion Embeddings (GEEs), and introduce activation guidance (AG) and mutual information (MI) minimization to reduce the correlations between emotion embedding and emotion-independent representation. At run-time conversion, it could produce the desired emotional utterance from a single pairwise utterance without any emotion labels, whether the target emotion appears in the training set or not. The subjective and objective evaluations validate the effectiveness of our proposed model for both the degree of feature separation and emotion expression, even it could achieve unseen emotion conversion.

A new patterns of self-organization activity of brain: Neural energy coding

According to the basic principles and methods of information theory, the operation way of neural coding is studied and analyzed by using the minimum mutual information and the maximum entropy principle. This paper describes how the principles of minimum mutual information and maximum entropy are used to evaluate the amount of information in neural responses. Its main contribution is as follows: (1) that the expression of neural information is closely related to the utilization of neural energy, and it is found that the highly evolved nervous system strictly follows the two basic principles of economy and efficiency in energy consumption and utilization; (2) In order to verify the relationship between neural information processing and energy utilization, this paper uses the concept of energy-efficiency ratio to measure the economy and high efficiency of the nervous system in term of energy utilization by using the maximum entropy principle; (3) The numerical results show that the energy consumed by the nervous system reflects not only the internal law of neural information conduction and processing, but also the self-organization structure of neural information coding. The results suggest that energy neural coding, a novel neural information processing method, can be used to understand how brain activity works. Such a coding pattern can not only be extended to research the large-scale neuroscience field, but also unify brain models at all levels by use of the energy theory. This will provide a scientific theoretical basis for the exploration of how the brain works and the computational principles of brain-like artificial intelligence.

Deep Representation Debiasing via Mutual Information Minimization and Maximization (Student Abstract)

Deep representation learning has succeeded in several fields. However, pre-trained deep representations are usually biased and make downstream models sensitive to different attributes. In this work, we propose a post-processing unsupervised deep representation debiasing algorithm, DeepMinMax, which can obtain unbiased representations directly from pre-trained representations without re-training or fine-tuning the entire model. The experimental results on synthetic and real-world datasets indicate that DeepMinMax outperforms the existing state-of-the-art algorithms on downstream tasks.

Abstract 2457: PICASSO: Ultra-multiplexed imaging technique for spatial protein analysis of tumor microenvironments

Abstract In recent cancer research in medicine, various combinatorial biomarkers are simultaneously visualized to study molecular heterogeneity of the tumor microenvironment in detail. These kinds of study aim to provide enhanced precise diagnostic system, optimal treatment decision, and more accurate prediction of survival or recurrence rates for each independent patient case. Conventional immunofluorescence-based approach, however, has been faced an intrinsic limitation in terms of spectral overlap among fluorophores, which allows only 3-4 fluorescent channels per single imaging cycle. Various alternatives, including mass cytometry or linear unmixing, have been proposed to overcome and enhance the multiplexing capability, while they could not totally replace the conventional immunofluorescence technique due to its requirement of specialized equipment and high barrier to entry. Here we report a state-of-the-art multiplexed imaging technique, termed PICASSO, which can perform an unprecedented high multiplexing capability even with a conventional microscopy platform. PICASSO is mainly composed of three parts; 1. antibody complex preformation technique for multiple target staining without host issue, 2. acquiring spectrally mixed images at different spectral ranges, in which each channel included one of those fluorophores’ emission peaks, indicating the number of image acquisitions equal to the number of fluorophores, and 3. fluorescent signals unmixing via mutual information (MI) minimization algorithm. Fifteen-color multiplexed imaging has been experimentally demonstrated within a single staining and imaging cycle via PICASSO. As a cyclic staining method, we demonstrated that PICASSO enables more than 40-color imaging within 3 staining and imaging rounds. Moreover, we demonstrated multiplexed imaging of mouse brains using commercialized bandpass filter-based microscopy. Other than mouse brain samples, various formalin-fixed paraffin-embedded (FFPE) specimens (e.g., breast, kidney, bladder, colon, prostate, thymus, etc.) could be utilized for PICASSO unmixing demonstration, which infers the feasibility of PICASSO technique in various pathological studies in medicine. In addition, PICASSO can be applied to 3D multiplexed imaging, mRNA multiplexed imaging, and large-area multiplexed imaging. We anticipate that PICASSO will provide a new perspective in multiplexed imaging field with high accessibility in cancer research and promote how we see and understand the molecular profiles of tumors and their microenvironments. Acknowledgments: This work was supported by Samsung Research Funding & Incubation Center for Future Technology (SRFC-IT1702-09). Citation Format: Junyoung Seo, Yeonbo Sim, Jeewon Kim, Hyunwoo Kim, In Cho, Hoyeon Nam, Young-Gyu Yoon, Jae-Byum Chang. PICASSO: Ultra-multiplexed imaging technique for spatial protein analysis of tumor microenvironments [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 2457.

Improving RGB-Infrared Object Detection by Reducing Cross-Modality Redundancy

In the field of remote sensing image applications, RGB and infrared image object detection is an important technology. The object detection performance can be improved and the robustness of the algorithm will be enhanced by making full use of their complementary information. Existing RGB-infrared detection methods do not explicitly encourage RGB and infrared images to achieve effective multimodal learning. We find that when fusing RGB and infrared images, cross-modal redundant information weakens the degree of complementary information fusion. Inspired by this observation, we propose a redundant information suppression network (RISNet) which suppresses cross-modal redundant information and facilitates the fusion of RGB-Infrared complementary information. Specifically, we design a novel mutual information minimization module to reduce the redundancy between RGB appearance features and infrared radiation features, which enables the network to take full advantage of the complementary advantages of multimodality and improve the object detection performance. In addition, in view of the drawbacks of the current artificial classification of lighting conditions, such as the subjectivity of artificial classification and the lack of comprehensiveness (divided into day and night only), we propose a method based on histogram statistics to classify lighting conditions in more detail. Experimental results on two public RGB-infrared object detection datasets demonstrate the superiorities of our proposed method over the state-of-the-art approaches, especially under challenging conditions such as poor illumination, complex background, and low contrast.

Distortion based potential game for distributed coverage control

In this work, we propose a multi-agent learning framework to address the mobile sensor coverage problem in which the minimum mutual information between the agents (mobile sensors) and their environment defines the agent-strategy selection rule towards the system Nash equilibrium in a potential game setting. Initially, the agents infer the environment behavior by means of the Gaussian process regression (GPR) approach, using their own information and the information provided by their neighborhood. Then, the rate distortion function (RDF) is used to minimize the mutual information between each agent and its environment by means of the Blahut-Arimoto algorithm, from which, the resulting conditional probability and the parameter λ have the highest relevance, since, the former describes the similitude between the agent information and its environment, and the latter, due to its influence in the distortion and gathered environment information, defines the rationality measure in the learning process. Finally, the expected distortion defined in the RDF, allows the formulation of a distortion based potential function and the consequent equilibrium convergence.

Entropic Regularization of Wasserstein Distance Between Infinite-Dimensional Gaussian Measures and Gaussian Processes

This work studies the entropic regularization formulation of the 2-Wasserstein distance on an infinite-dimensional Hilbert space, in particular for the Gaussian setting. We first present the minimum mutual information property, namely, the joint measures of two Gaussian measures on Hilbert space with the smallest mutual information are joint Gaussian measures. This is the infinite-dimensional generalization of the maximum entropy property of Gaussian densities on Euclidean space. We then give closed-form formulas for the optimal entropic transport plan, entropic 2-Wasserstein distance, and Sinkhorn divergence between two Gaussian measures on a Hilbert space, along with the fixed point equations for the barycenter of a set of Gaussian measures. Our formulations fully exploit the regularization aspect of the entropic formulation and are valid both in singular and in nonsingular settings. In the infinite-dimensional setting, both the entropic 2-Wasserstein distance and Sinkhorn divergence are Fréchet differentiable, in contrast to the exact 2-Wasserstein distance, which is not differentiable. Our Sinkhorn barycenter equation is new and always has a unique solution. In contrast, the finite-dimensional barycenter equation for the entropic 2-Wasserstein distance fails to generalize to the Hilbert space setting. In the setting of reproducing kernel Hilbert spaces, our distance formulas are given explicitly in terms of the corresponding kernel Gram matrices, providing an interpolation between the kernel maximum mean discrepancy and the kernel 2-Wasserstein distance.

A Minimum Mutual Information Optimized SSD and Application In Bearing Fault Detection

For purpose of efficiently extracting the fault detailed characteristics of rolling bearing vibration signals in the midst of heavy noise interference, the author combines the mutual information theory and proposes a singular spectrum decomposition effective component discrimination method based on the minimum mutual information criterion. First, the SSD algorithm is used to decompose the original vibration signal into a number of SSC; then the mutual information value between the original signal and each singular spectrum component is calculated separately, based on the nature of the bearing fault vibration signal and the mutual information theory for derivation and analysis, and the selection is determined the singular spectrum component with the smallest mutual information value among the original signal and each component is taken as the best component; finally, the best component selected based on the minimum mutual information value criterion is analysed by envelope spectrum to extract the fault characteristics of rolling bearing. Through the analysis of bearing fault simulation signal and the comparison with other indicators, it is shown that this set of experiments significantly extracts the characteristics of the vibration signals emitted by rolling bearings due to faults in noisy environments, thus revealing promising applications and development prospects.

A feature reduction and selection algorithm for improved obstructive sleep apnea classification process.

Feature reduction and selection of the best features for classification is a crucial stage for data analysis to reduce training time and overfitting. Commonly, feature reduction techniques rely on selecting features with high relevance to the outcome and minimum mutual information among one another. However, satisfying these criteria does not guarantee the selected features have a high classification power. In this study, we provide an algorithm to build predictive models for the desired outcome, while selecting the most useful features for high classification outcomes. The results of the proposed algorithm are compared with the outcomes of five popular feature reduction and selection techniques. A dataset from an obstructive sleep apnea study (data of 113 and 86 participants as training and blind testing datasets, respectively) was used to illustrate the algorithm's performance. The extracted features used in the algorithm were modeled using three-, four-, and five-feature combinations. The models with a high correlation to the outcome and low overlap percentages among specific subgroups of the data were selected and examined. Then, a set of the best selected models were averaged to provide better classification accuracy. The accuracy of the proposed algorithm has been 25% higher than that when using five popular feature reduction/selection techniques. Furthermore, the proposed algorithm is about 20 times faster than the five popular techniques.

Using the Semantic Information G Measure to Explain and Extend Rate-Distortion Functions and Maximum Entropy Distributions.

In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual Information (MMI) distributions and ME distributions are expressed by Bayes-like formulas, including Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other hand, the rate-distortion function has three disadvantages: (1) the distortion function is subjectively defined; (2) the definition of the distortion function between instances and labels is often difficult; (3) it cannot be used for data compression according to the labels’ semantic meanings. The author has proposed using the semantic information G measure with both statistical probability and logical probability before. We can now explain NEFs as truth functions, partition functions as logical probabilities, Bayes-like formulas as semantic Bayes’ formulas, MMI as Semantic Mutual Information (SMI), and ME as extreme ME minus SMI. In overcoming the above disadvantages, this paper sets up the relationship between truth functions and distortion functions, obtains truth functions from samples by machine learning, and constructs constraint conditions with truth functions to extend rate-distortion functions. Two examples are used to help readers understand the MMI iteration and to support the theoretical results. Using truth functions and the semantic information G measure, we can combine machine learning and data compression, including semantic compression. We need further studies to explore general data compression and recovery, according to the semantic meaning.

Mutual information regularized identity-aware facial expression recognition in compressed video

How to extract effective expression representations that invariant to the identity-specific attributes is a long-lasting problem for facial expression recognition (FER). Most of the previous methods process the RGB images of a sequence, while we argue that the off-the-shelf and valuable expression-related muscle movement is already embedded in the compression format. In this paper, we target to explore the inter-subject variations eliminated facial expression representation in the compressed video domain. In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames and possibly extract identity factors from the I frame with a pre-trained face recognition network. By enforcing the marginal independence of them, the expression feature is expected to be purer for the expression and be robust to identity shifts. Specifically, we propose a novel collaborative min-min game for mutual information (MI) minimization in latent space. We do not need the identity label or multiple expression samples from the same person for identity elimination. Moreover, when the apex frame is annotated in the dataset, the complementary constraint can be further added to regularize the feature-level game. In testing, only the compressed residual frames are required to achieve expression prediction. Our solution can achieve comparable or better performance than the recent decoded image-based methods on the typical FER benchmarks with about 3 times faster inference.

Universal Active Learning via Conditional Mutual Information Minimization

Modern machine learning systems require massive amounts of labeled training data in order to achieve high accuracy rates which is very expensive in terms of time and cost. Active learning is an approach which uses feedback to only label the most informative data points and significantly reduce the labeling effort. Many heuristics for selecting data points have been developed in recent years which are usually tailored to a specific task and a general unified framework is lacking. In this work, a new information theoretic criterion is proposed based on a minimax log-loss regret formulation of the active learning problem. First, a Redundancy Capacity theorem for active learning is derived along with an optimal learner. This leads to a new active learning criterion which naturally induces an exploration - exploitation trade-off in feature selection and generalizes previously proposed heuristic criteria. The new criterion is compared analytically and via empirical simulation to other commonly used information theoretic active learning criteria. Next, the linear hyper-plane hypotheses class with possibly asymmetric label noise is considered. The achievable performance for the proposed criterion is analyzed using a new low complexity greedy algorithm based on the Posterior Matching scheme for communication with feedback. It is shown that for general label noise and bounded feature distribution, the new information theoretic criterion decays exponentially fast to zero.

Face recognition using particle swarm optimization based block ICA

Face recognition is one of the most important and widely applicable research problems in the subject area of machine learning and computer vision. Extraction of features, local or holistic, is the fundamental step and subspace method has been a natural choice for facial feature extraction. Among these, methods like PCA, ICA, LDA aim to reduce the dimension of the data while retaining the statistical separation property between distinct classes. Unlike the traditional ICA, in which the whole face image is stretched into a vector before calculating the independent components (ICs), Block ICA (B-ICA) partitions the facial images into blocks and takes the block as the training vector. Since the dimensionality of the training vector in B-ICA is much smaller than that in traditional ICA, reduction in face recognition error is expected. The objective of ICA is to find a separation matrix and it is achieved by a process of optimization, such as maximization of non-Gaussianity, maximum likelihood estimation, and minimization of mutual information. We observe here that the gradient-based learning can be efficiently and effectively achieved by the application of swarm-based optimization. We propose here the application of our Gradient-based Swarm Optimization method for Block ICA, where gradient information is combined with conventional swarm search to optimize the contrast function. We compare our method with B-ICA on three benchmark image data sets and show that our method achieved a better recognition rate compared to B-ICA in different block sizes with 70%, 80% and 90% data used for training the model.