KL Divergence Research Articles

Efficient Deep Reinforcement Learning With Imitative Expert Priors for Autonomous Driving.

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this article proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely, expert demonstration, policy derivation, and RL. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the RL step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60% improvement compared with soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert. We also find that using the imitative expert policy trained with the ensemble method that estimates both policy and model uncertainties, as well as increasing the training sample size, can result in better training and testing performance, especially for more difficult tasks. As a result, the proposed method has shown its potential to facilitate the applications of DRL-enabled human-like autonomous driving systems in practice. The code and supplementary videos are also provided. [https://mczhi.github.io/Expert-Prior-RL/].

Enhancing Robustness of Medical Image Segmentation Model with Neural Memory Ordinary Differential Equation.

Deep neural networks (DNNs) have emerged as a prominent model in medical image segmentation, achieving remarkable advancements in clinical practice. Despite the promising results reported in the literature, the effectiveness of DNNs necessitates substantial quantities of high-quality annotated training data. During experiments, we observe a significant decline in the performance of DNNs on the test set when there exists disruption in the labels of the training dataset, revealing inherent limitations in the robustness of DNNs. In this paper, we find that the neural memory ordinary differential equation (nmODE), a recently proposed model based on ordinary differential equations (ODEs), not only addresses the robustness limitation but also enhances performance when trained by the clean training dataset. However, it is acknowledged that the ODE-based model tends to be less computationally efficient compared to the conventional discrete models due to the multiple function evaluations required by the ODE solver. Recognizing the efficiency limitation of the ODE-based model, we propose a novel approach called the nmODE-based knowledge distillation (nmODE-KD). The proposed method aims to transfer knowledge from the continuous nmODE to a discrete layer, simultaneously enhancing the model's robustness and efficiency. The core concept of nmODE-KD revolves around enforcing the discrete layer to mimic the continuous nmODE by minimizing the KL divergence between them. Experimental results on 18 organs-at-risk segmentation tasks demonstrate that nmODE-KD exhibits improved robustness compared to ODE-based models while also mitigating the efficiency limitation.

Bayesian-based Component Lifetime Prediction Model Using Workshop and Telematics Data

This paper presents a Bayesian approach to predicting brake pad and battery life based on field service data from a fleet management system(FMS). The data includes vehicle driving data collected via telematics and maintenance record data managed by the workshop. The proposed approach consists of three modules: component health diagnosis, workshop data analysis and driving pattern analysis. The health diagnosis module detects domain-based transformed feature, from the driving data, changes using KL divergence. The maintenance record data from workshop analysis module estimates the prior probability of maintenance cycles. The censored nature of workshop data is validated by updating the posterior probability using driving patterns from driving data. The driving pattern analysis module classifies driving patterns for lifetime prediction. This study develops a predictive maintenance model for brakes and batteries without additional sensors using the data required for fleet operation. The mileagebased cycle maintenance approach commonly used for fleet management is improved by this model. Future FMS systems are expected to make extensive use of this concept.

Data-Weighted Multivariate Generalized Gaussian Mixture Model: Application to Point Cloud Robust Registration.

In this paper, a weighted multivariate generalized Gaussian mixture model combined with stochastic optimization is proposed for point cloud registration. The mixture model parameters of the target scene and the scene to be registered are updated iteratively by the fixed point method under the framework of the EM algorithm, and the number of components is determined based on the minimum message length criterion (MML). The KL divergence between these two mixture models is utilized as the loss function for stochastic optimization to find the optimal parameters of the transformation model. The self-built point clouds are used to evaluate the performance of the proposed algorithm on rigid registration. Experiments demonstrate that the algorithm dramatically reduces the impact of noise and outliers and effectively extracts the key features of the data-intensive regions.

Research on a demand design method of a cross polarization converter metasurface based on a depth generation model

As a two-dimensional electromagnetic metamaterial, the cross-polarization conversion (CPC) metasurface is thin, easy to develop, and has attracted wide attention. However, existing CPC cell surface designs still rely on inefficient full-wave numerical simulation. Although some researchers have explored deep learning CPC metasurface structure design methods, the generated metasurface patterns are of poor quality. In this paper, an on-demand design method for cross-polarization conversion metasurface based on depth-generation model is proposed. Firstly, Wasserstein generative adversarial network (WGAN) is used to reverse design CPC metasurface, and Wasserstein distance is introduced to replace JS divergence and KL divergence to optimize the target. The problem of training difficulty caused by gradient elimination of original generative adversarial network (GAN) is fundamentally solved. Secondly, in the WGAN model, U-Net architecture generator is used to generate images, which greatly improves the surface image quality of CPC. In addition, a simulator composed of convolutional neural network (CNN) is also added in this paper to carry out forward prediction of S-parameter spectrum diagram. By inputting the patterns generated by WGAN into the simulator, the corresponding S-parameter spectrum diagram is generated and compared with the real S-parameter spectrum diagram, so as to verify whether the surface patterns of generated elements meet the requirements. The depth generation model proposed in this paper organically combines the forward spectrum prediction model and the reverse CPC metasurface structure design model, so that the CPC metasurface structure satisfying the expected electromagnetic response can be designed quickly on demand. This on-demand design method is expected to promote the rapid design, fabrication and application of electromagnetic devices.

Relation Extraction Based on Prompt Information and Feature Reuse

ABSTRACT To alleviate the problem of under-utilization features of sentence-level relation extraction, which leads to insufficient performance of the pre-trained language model and underutilization of the feature vector, a sentence-level relation extraction method based on adding prompt information and feature reuse is proposed. At first, in addition to the pair of nominals and sentence information, a piece of prompt information is added, and the overall feature information consists of sentence information, entity pair information, and prompt information, and then the features are encoded by the pre-trained language model ROBERTA. Moreover, in the pre-trained language model, BIGRU is also introduced in the composition of the neural network to extract information, and the feature information is passed through the neural network to form several sets of feature vectors. After that, these feature vectors are reused in different combinations to form multiple outputs, and the outputs are aggregated using ensemble-learning soft voting to perform relation classification. In addition to this, the sum of cross-entropy, KL divergence, and negative log-likelihood loss is used as the final loss function in this paper. In the comparison experiments, the model based on adding prompt information and feature reuse achieved higher results of the SemEval-2010 task 8 relational dataset.

Improving adversarial robustness through a curriculum-guided reliable distillation

Adversarial distillation (AD) which combines adversarial training with knowledge distillation has become a powerful procedure to mitigate the effects of adversarial examples to deep neural networks, which aims at distilling a robust student network from a pre-trained robust teacher network. In AD, the teacher's reliability has been a crucial problem and recent work incorporates a self-distillation loss on the student into the AD framework, encouraging the student to partially trust the teacher and gradually trust itself more. However, the only key factor to control this trust level cannot be adaptive enough for different training samples, guiding the student to trust itself inappropriately and inspiring us to pursue a better way to obtain more reliable supervision. In this paper we revisit the performance variation of all training samples from the teacher to student, showing that previous work is not always adaptive with the distillation and can be further refined. Accordingly, a more effective and justified supervision, namely Curricular Adversarial Distillation (CAD), is proposed to help boost the self-distillation process. In CAD, the KL divergences between both the clean and adversarial outputs of teacher and the smoothed labels are calculated for the supervision to address the teacher's unreliability in the self-distillation. Besides, the smoothed labels follow a curriculum-style scheduler, getting smoothed to different degrees at different training stages and helping the teacher continuously be adaptive to the varying adversarial examples from student. Extensive experiments have shown the superiority of our strategy in distilling a robust student network against various attacks.

Pipeline leakage aperture identification method based on pseudolabel learning

Aiming at the problem of insufficient label data in the pipeline leak detection field, this paper proposes a pseudolabel (PL) adaptive learning method based on multiscale convolutional neural network (MSCNN) with the idea of transfer learning for pipeline leak aperture identification. First, the convolutional and pooling layers for transfer learning feature extraction are improved by using a dual-channel MSCNN. Second, the KL divergence function after dimensionality reduction is used to calculate the distribution distance between the source domain and the target domain to improve the robustness of distribution alignment in high-noise environments. In addition, considering the interference of PL noise, this paper develops a pseudolabel (PL) dynamic threshold to achieve the purpose of PL adaptive updating. Compared with the fixed threshold, the improved PL learning (PLL) can effectively improve the prediction accuracy of the model. The effectiveness of the method proposed in this paper is verified by predicting pipeline leakage conditions at different distances and under different pressures. The comparative analysis results show that the method in this paper is superior to other transfer learning methods in terms of prediction accuracy, stability, and convergence speed.

GenKL: An Iterative Framework for Resolving Label Ambiguity and Label Non-conformity in Web Images Via a New Generalized KL Divergence

Web image datasets curated online inherently contain ambiguous in-distribution instances and out-of-distribution instances, which we collectively call non-conforming (NC) instances. In many recent approaches for mitigating the negative effects of NC instances, the core implicit assumption is that the NC instances can be found via entropy maximization. For “entropy” to be well-defined, we are interpreting the output prediction vector of an instance as the parameter vector of a multinomial random variable, with respect to some trained model with a softmax output layer. Hence, entropy maximization is based on the idealized assumption that NC instances have predictions that are “almost” uniformly distributed. However, in real-world web image datasets, there are numerous NC instances whose predictions are far from being uniformly distributed. To tackle the limitation of entropy maximization, we propose (alpha , beta )-generalized KL divergence, {mathcal {D}}_{text {KL}}^{alpha , beta }(pVert q), which can be used to identify significantly more NC instances. Theoretical properties of {mathcal {D}}_{text {KL}}^{alpha , beta }(pVert q) are proven, and we also show empirically that a simple use of {mathcal {D}}_{text {KL}}^{alpha , beta }(pVert q) outperforms all baselines on the NC instance identification task. Building upon (alpha ,beta )-generalized KL divergence, we also introduce a new iterative training framework, GenKL, that identifies and relabels NC instances. When evaluated on three web image datasets, Clothing1M, Food101/Food101N, and mini WebVision 1.0, we achieved new state-of-the-art classification accuracies: 81.34%, 85.73% and 78.99%/92.54% (top-1/top-5), respectively.

Addressing Posterior Collapse in Variational Autoencoders with β-VAE

Posterior collapse is a pervasive issue in Variational Autoencoders (VAEs) that leads to the learned latent representations becoming trivial and devoid of meaningful information. To address this problem, this paper presents a novel β-VAE approach, which incorporates a hyperparameter β to strike an optimal balance between the reconstruction loss and the KL divergence loss. By conducting a comprehensive series of experiments and drawing comparisons with existing methods, robust evidence is provided that the proposed β-VAE method effectively mitigates posterior collapse and yields more expressive and informative latent representations.The experimental setup involves various architectures and datasets to demonstrate the versatility and efficacy of the β-VAE approach in diverse settings. Additionally, ablation studies are performed to investigate the impact of different β values on the model's performance, elucidating the role of this hyperparameter in controlling the trade-off between reconstruction quality and latent representation expressiveness. Furthermore, the disentanglement properties of the learned latent space are analyzed, which is a crucial aspect of VAEs, especially when applied to complex, real-world data.In-depth analysis of the results offers valuable insights into the underlying mechanisms of β-VAE, thereby contributing to a more profound understanding of VAEs and their inherent limitations. The findings not only establish the effectiveness of the β-VAE method in preventing posterior collapse but also pave the way for future research on improving VAEs' performance in various applications. Potential future work could explore alternative techniques for balancing the competing objectives of reconstruction and latent representation learning or delve into the theoretical properties of β-VAE, providing a more rigorous foundation for this approach.

Entropy Regularization for Population Estimation

Entropy regularization is known to improve exploration in sequential decision-making problems. We show that this same mechanism can also lead to nearly unbiased and lower-variance estimates of the mean reward in the optimize-and-estimate structured bandit setting. Mean reward estimation (i.e., population estimation) tasks have recently been shown to be essential for public policy settings where legal constraints often require precise estimates of population metrics. We show that leveraging entropy and KL divergence can yield a better trade-off between reward and estimator variance than existing baselines, all while remaining nearly unbiased. These properties of entropy regularization illustrate an exciting potential for bringing together the optimal exploration and estimation literature.

Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching

Offline imitation learning (IL) promises the ability to learn performant policies from pre-collected demonstrations without interactions with the environment. However, imitating behaviors fully offline typically requires numerous expert data. To tackle this issue, we study the setting where we have limited expert data and supplementary suboptimal data. In this case, a well-known issue is the distribution shift between the learned policy and the behavior policy that collects the offline data. Prior works mitigate this issue by regularizing the KL divergence between the stationary state-action distributions of the learned policy and the behavior policy. We argue that such constraints based on exact distribution matching can be overly conservative and hamper policy learning, especially when the imperfect offline data is highly suboptimal. To resolve this issue, we present RelaxDICE, which employs an asymmetrically-relaxed f-divergence for explicit support regularization. Specifically, instead of driving the learned policy to exactly match the behavior policy, we impose little penalty whenever the density ratio between their stationary state-action distributions is upper bounded by a constant. Note that such formulation leads to a nested min-max optimization problem, which causes instability in practice. RelaxDICE addresses this challenge by supporting a closed-form solution for the inner maximization problem. Extensive empirical study shows that our method significantly outperforms the best prior offline IL method in six standard continuous control environments with over 30% performance gain on average, across 22 settings where the imperfect dataset is highly suboptimal.

Near-Optimal Learning of Tree-Structured Distributions by Chow and Liu

We provide finite sample guarantees for the classical Chow–Liu algorithm [Chow and Liu, IEEE Trans. Inform. Theory, 14 (1968), pp. 462–467] to learn a tree-structured graphical model of a distribution. For a distribution on and a tree on nodes, we say is an -approximate tree for if there is a -structured distribution such that is at most more than the best possible tree-structured distribution for . We show that if itself is tree-structured, then the Chow–Liu algorithm with the plug-in estimator for mutual information with independent and identically distributed samples outputs an -approximate tree for with constant probability. In contrast, for a general (which may not be tree-structured), samples are necessary to find an -approximate tree. Our upper bound is based on a new conditional independence tester that addresses an open problem posed by Canonne et al. [Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, ACM, 2018, pp. 735–748]: we prove that for three random variables each over , testing if is 0 or is possible with samples. Finally, we show that for a specific tree , with samples from a distribution over , one can efficiently learn the closest -structured distribution in KL divergence by applying the add-1 estimator at each node.

Operational adjustment modeling approach based on Bayesian network transfer learning for new flotation process under scarce data

Establishing an intelligent operational adjustment model is significant for optimizing flotation performance and realizing the stable control of the flotation process. When sufficient historical data is available, the Bayesian network (BN) has been used to achieve this goal due to its powerful uncertainty modeling and reasoning capabilities. However, for a new flotation process in the early stage with scarce data, an accurate BN model can hardly be established to make reliable decisions. In this scenario, it is an effective way to transfer data or knowledge from other similar flotation processes to the new flotation process to help complete the target BN modeling task. Therefore, the operational adjustment modeling problem is transformed into the BN transfer learning problem. Moreover, this article proposes a new BN transfer learning approach, including structure and parameters transfer learning techniques. For structure transfer learning, a majority voting mechanism is proposed to get an integrated result of conditional independence tests and generates the network structure based on them. For parameters transfer learning, the sources are judged whether to be used to learn the parameters first, and then the suitable sources are used to learn the network parameters based on the weight determined by the similarity. The Asia network first assesses the proposed approach and then applies it to real-world flotation operational adjustment decisions. Experimental results on the Asia network show that the proposed approach can learn more accurate structure and parameters according to F1 score, Hamming distance, and KL divergence indicators compared to existing advanced methods. Furthermore, the experimental results of the flotation process show that the inference accuracy is improved from 57.1% to 85.7% compared with the traditional modeling method.

Behavior-Aware English Reading Article Recommendation System Using Online Distilled Deep Q-Learning

Due to the differences of students' English proficiency and the rapid changes in reading interests, online personalized English reading recommendation is a highly challenging problem. Although some works have been proposed to address the dynamic change of recommendation, there are two issues with these methods. First, it only considers whether students have read the recommended articles. Second, these methods often fail to capture the real-time changing interests of users. To address the above challenges, a deep Q-network based recommendation framework was proposed. The authors further use the user's behavior and scores as reward information to get more user's feedback. In addition, a personalized adaptive module was introduced to capture the short-term interests on the fly and utilized the consistent loss of KL divergence to distill the knowledge from the online model. Extensive experiments on the offline and online dataset in the IWiLL website demonstrate the superior performance of the method.

Belief f-divergence for EEG complexity evaluation

Evidence theory, as a useful uncertain reasoning method, is widely used in various fields. Nevertheless, how to quantify the divergence between the basic belief assignments in belief theory and how to apply it are still open issues. To measure the discrepancy between basic belief assignments, we first introduce belief f-divergence (BFD) in this paper. We also analyze and verify its related properties that are expected to significantly improve the performance of electroencephalogram (EEG) complexity evaluation. Then, we employ empirical mode decomposition to detrend EEG noise and reconstruct EEG signals with inherent components and develop a belief f-divergence-based complexity (BFDC) evaluation method with five different functions, including belief KL divergence, belief Hellinger divergence, belief χ2 divergence, belief triangular divergence, and belief total variation divergence, to evaluate the complexity of the EEG signals. Based on a benchmark lane-keeping EEG driving experiment, our proposed BFDC evaluation approach can identify the differences among keep–drift events more obviously and significantly reduce the root mean squared error up to the order of 102 compared to the existing complexity measurement of fuzzy entropy. We believe that our proposed BFDC evaluation method has the potential to be a new and robust approach to precisely evaluate the dynamic complexity of physiological signals.

Exploring the Advantages of Quantum Generative Adversarial Networks in Generative Chemistry.

De novo drug design with desired biological activities is crucial for developing novel therapeutics for patients. The drug development process is time- and resource-consuming, and it has a low probability of success. Recent advances in machine learning and deep learning technology have reduced the time and cost of the discovery process and therefore, improved pharmaceutical research and development. In this paper, we explore the combination of two rapidly developing fields with lead candidate discovery in the drug development process. First, artificial intelligence has already been demonstrated to successfully accelerate conventional drug design approaches. Second, quantum computing has demonstrated promising potential in different applications, such as quantum chemistry, combinatorial optimizations, and machine learning. This article explores hybrid quantum-classical generative adversarial networks (GAN) for small molecule discovery. We substituted each element of GAN with a variational quantum circuit (VQC) and demonstrated the quantum advantages in the small drug discovery. Utilizing a VQC in the noise generator of a GAN to generate small molecules achieves better physicochemical properties and performance in the goal-directed benchmark than the classical counterpart. Moreover, we demonstrate the potential of a VQC with only tens of learnable parameters in the generator of GAN to generate small molecules. We also demonstrate the quantum advantage of a VQC in the discriminator of GAN. In this hybrid model, the number of learnable parameters is significantly less than the classical ones, and it can still generate valid molecules. The hybrid model with only tens of training parameters in the quantum discriminator outperforms the MLP-based one in terms of both generated molecule properties and the achieved KL divergence. However, the hybrid quantum-classical GANs still face challenges in generating unique and valid molecules compared to their classical counterparts.

A novel scheme based on information theory and transfer learning for multi classes motor imagery decoding

AbstractThe most important challenges of classifying Motor Imagery tasks based on the EEG signal are low signal‐to‐noise ratio, non‐stationarity, and the high subject dependence of the EEG signal. In this study, a framework for multi‐class decoding of Motor Imagery signals is presented. This framework is based on information theory and hybrid deep learning along with transfer learning. In this study, the OVR‐FBDiv method, which is based on the symmetric Kullback—Leibler divergence, is used to differentiate between features of different classes and highlight them. Then, the mRMR algorithm is used to select the most distinctive features obtained from the filters of symmetric KL divergence. Finally, a hybrid deep neural network consisting of CNN and LSTM is used to learn the spatial and temporal features of the EEG signal along with the transfer learning technique to overcome the problem of subject dependence in EEG signals. The average value of Kappa for the classification of 4‐class Motor Imagery data on BCI competition IV dataset 2a by the proposed method is 0.84. Also, the proposed method is compared with other state‐of‐the‐art methods.

Supplementary Materials: Stability Analysis of MD trajectories via KL divergence

Supplementary Materials: Stability Analysis of MD trajectories via KL divergence

Semi-Supervised Lexicon-Aware Embedding for News Article Time Estimation

In the information retrieval community, Temporal Information Retrieval (TIR) has become increasingly popular. Documents focused on the time surrounding their publication are more likely to be accurate and contain information relevant to the reader. In this study, we explore the inverted pyramid paradigm by extracting temporal expressions from news documents, standardizing their values, and evaluating them based on their position within the text. We present a lexicon expansion method that employs WordNet as input. This approach enhances the lexicon by grouping words with similar meanings, potentially improving the accuracy of event detection algorithms. Additionally, this process can introduce new words and phrases to the lexicon, expanding the vocabulary. Using each tagged dataset, a classifier is trained with a pre-trained network. A pool of unlabeled data are processed, and high-confidence pseudo-labels are assigned. Pseudo-labels are generated by leveraging the partially trained model and the original labelled data. As the classifier predicts the correct label for a data sample, the pseudo-labels of other data samples are updated, and vice versa. At the end of this process, the predictions from different matching classifiers are combined. It takes several rounds to label the unlabeled inputs using this method. To evaluate the proposed solutions, we conducted experiments on 4,500 online news articles relevant to temporal retrieval. LSTM, BiLSTM, and BERT models with and without lexicon expansion were assessed based on log loss and relative divergence of entropy. A jointly trained semi-supervised learning model achieved a mean KL divergence of 0.89, an F1 score of 0.74 for temporal events, and 0.63 for non-temporal events. Besides alleviating data sparsity issues and enabling the training of more complex networks, this technique can also serve as an alternative to data augmentation methods.