Loss Constraints Research Articles

Adaptive pooling-based convolution factorization for deploying CNNs on energy-constrained IoT edge devices

Convolutional neural networks (CNNs) involve a tremendous amount of multiply-and-accumulate (MAC) computations, leading to energy issues when deployed on energy-constrained IoT edge devices for computer vision purposes. However, a great number of these computations are ineffectual since they lead to meaningless zero activations, which could be avoided if predicted appropriately. This has been the focus of many studies, whereas it has only been done through zero-activation simplifications. This paper goes beyond those simplifications through a convolution factorization approach to convert the MAC-intensive convolution operations into less complex pooling operations. The main characteristic of the proposed idea is that, given a target accuracy, the applicability and type of factorization are adaptively decided for each individual activation. This is achieved by providing a lightweight multi-class CNN-based predictor, namely RedZAP. The selected types of pooling operations for our convolution factorizations are supported by existing CNN architectures. The experimental results show that, given a top-1 accuracy loss constraint of 1%, the proposed approach saves another 8% MAC operations in comparison to the existing zero-prediction approaches in the literature.

Simulation of Postoperative Facial Appearances via Geometric Deep Learning for Efficient Orthognathic Surgical Planning.

Orthognathic surgery corrects jaw deformities to improve aesthetics and functions. Due to the complexity of the craniomaxillofacial (CMF) anatomy, orthognathic surgery requires precise surgical planning, which involves predicting postoperative changes in facial appearance. To this end, most conventional methods involve simulation with biomechanical modeling methods, which are labor intensive and computationally expensive. Here we introduce a learning-based framework to speed up the simulation of postoperative facial appearances. Specifically, we introduce a facial shape change prediction network (FSC-Net) to learn the nonlinear mapping from bony shape changes to facial shape changes. FSC-Net is a point transform network weakly-supervised by paired preoperative and postoperative data without point-wise correspondence. In FSC-Net, a distance-guided shape loss places more emphasis on the jaw region. A local point constraint loss restricts point displacements to preserve the topology and smoothness of the surface mesh after point transformation. Evaluation results indicate that FSC-Net achieves 15× speedup with accuracy comparable to a state-of-the-art (SOTA) finite-element modeling (FEM) method.

Random clustering analysis in deterministic DG-based distribution planning

New feeders’ allocation, new lines connecting the substations to new feeders and other feeders to tie-line routes, should be defined for minimizing the costs of new facilities’ installation with some limitations as voltage loss, capacity of current, geographical restrictions of installation, and balance between loads, while feeders have been recently distributed in a system of distribution. However, finding the optimum plan for feeder distribution among different applicants, is difficult for planners. To confirm decision of planners of feeder distribution plan in this paper, the author proposes a technique to obtain the minimum cost distribution facilities by all the applicants’ searching that fulfill such distribution limitations as voltage loss, also geographical constraints of installation and power loss and etc. In this paper, optimal deterministic planning of distribution networks with (DG) will be discussed and all calculations will be done in medium voltage (MV) level. Weight clustering in distribution networks planning is a major topic that can be discussed by different methods like fuzzy algorithm. Weight clustering in reliability concepts of deterministic DG-based distribution planning such as failure rate calculations can be discussed, but its meaning in robust optimization (RO) will be very complex and profound. In this paper, the study of random clustering in both single and multi-cluster states has been investigated and concluded. The results of this research can be used in the field of robust optimization of distribution networks at the MV networks. The sample candidate of random clustering studies has been selected the Greenfield residential area. According flowcharts I and II in part of the appendix, detailed analytical results based on robust optimization of 32 candidate substations of case study has been extracted in two state consist of both single and multi-cluster states.

Federated learning for intelligent fault diagnosis based on similarity collaboration

In recent years, federated learning has received much attention because it involves the collaboration of each client to train a strong model without infringing data privacy. Machinery fault diagnosis also benefits from this technology. However, the different operating conditions and diagnostic tasks for each client lead to data heterogeneity among the devices, which brings a tough challenge to federated learning for machinery fault diagnosis. To solve this problem, we explore a federated learning method for machinery fault diagnosis based on similarity collaboration (FedSC). Considering the data heterogeneity of each client, the FedSC customizes a personalized model for each client, and then a similarity mechanism is used to weigh the aggregation of each personalization model. In addition, when a client’s model is updated, the distance constraint loss is employed to ensure that local model updates do not deviate from their personalized cloud model. Comprehensive experiments on two rotating machinery datasets demonstrate that our method achieves higher accuracy and faster convergence, providing promising application prospects in realistic industrial scenarios.

A hybrid dynamic economics emissions dispatch model: Distributed renewable power systems based on improved COOT optimization algorithm

This study proposes a hybrid dynamic economics emissions dispatch (HDEED) model for a distributed power system containing thermal generating units, wind farms and photovoltaic plants. The construction of novel distributed power system has been a significant means of tackling the energy crisis. (1) the relationship between each generation unit of the power generation system is analyzed, as well as the power balance constraint, transmission loss constraint, output capacity of each generation unit and slope constraint of the power system; and (2) the operating cost objective function is established with the objective of minimizing unit's generation cost, pollutant emission objective function, and the satisfaction weight coefficient. A novel and improved COOT optimization algorithm is presented to enhance convergence performance and solution speed in solving the problem by introducing a chaotic initialization strategy. A mutation strategy and an improved chain movement of the model solution are verified. The result shows that for the HDEED problem, the ICOOT algorithm reduces the operating cost targets by 1.28%, 6.99% and 7.44% and the pollutant emission targets by 2.98%, 5.46% and 10.88% compared to other algorithms. The developed model provides an effective solution for improving the operational stability, economy and cleanliness of system.

Portfolio performance under benchmarking relative loss and portfolio insurance: From omega ratio to loss aversion

AbstractWe study an optimal investment problem under a joint limited expected relative loss and portfolio insurance constraint with a general random benchmark. By making use of a static Lagrangian method in a complete market setting, the optimal wealth and investment strategy can be fully determined along with the existence and uniqueness of the Lagrangian multipliers. Our numerical demonstration for various commonly used random benchmarks shows a trade-off between the portfolio outperformance and underperformance relative to the benchmark, which may not be captured by the widely used Omega ratio and its utility-transformed version, reflecting the impact of the benchmarking loss constraint. Furthermore, we develop a new portfolio performance measurement indicator that incorporates the agent’s utility loss aversion relative to the benchmark via solving an equivalent optimal asset allocation problem with a benchmark-reference-based preference. We show that the expected utility performance is well depicted by looking at this new portfolio performance ratio, suggesting a more suitable portfolio performance measurement under a limited loss constraint relative to a possibly random benchmark.

Trustworthiness of Process Monitoring in IIoT Based on Self-Weighted Dictionary Learning

Process monitoring, a typical application of industrial Internet of Things (IIOT), is crucial to ensure the reliable operation of the industrial system. In practice, due to the harsh environment and unreliable sensors and actuators, it is often difficult for IIoT to collect enough tagged and highly reliable data, which further degrades the process monitoring performance and makes the monitoring results not trustworthy. In order to reduce the negative impact of these unreliable factors, a self-weighted dictionary learning process monitoring method is proposed. In particular, a label propagation classifier is implemented from the labeled data to unlabeled data to obtain a credible label prediction. Subsequently, considering the interference of low-quality data and label information, we reweight the classification loss and label-consistency constraints to enhance the trustworthiness of feature extraction. Finally, a novel iterative optimization algorithm that combines the block coordinate descent method with the alternating direction multiplier method is developed to ensure the convergence speed of the learned classifier and dictionary. Extensive experiments indicate that the proposed method can guarantee the trustworthiness of the process monitoring results.

An effective differential privacy protection method of location data based on perturbation loss constraint

An effective differential privacy protection method of location data based on perturbation loss constraint

Optimal privacy protection of mobility data: a predictive approach

Location data are extensively used to provide geo-personalized contents to mobile devices users. Sharing such personal data is a major threat to privacy, with risks of re-identification or inference of sensitive information. Location data broadcasted to services can be sanitized, i.e., by adding noise to spatial coordinates. Such protection mechanisms from the literature are widely generic, e.g., not specific to a user and mobility properties. In this work, we advocate that taking into account the specificities of location data (temporal correlation, human mobility patterns, etc.) enables to gain in the privacy-utility trade-off. specifically, using future mobility prediction greatly improves privacy. We present a novel protection mechanism, based on model predictive control (MPC). The sanitized location is optimally computed so that it maximizes privacy while guaranteeing a utility loss constraint, for present and future locations. Our formulation explicitly takes into account non-constant sampling time, due to moments when no location data is broadcasted. We evaluate experimentally our control on real mobility dataset and compare to the state of the art. Results show that with knowledge of user's future mobility over a few of minutes, we can gain up to 10% of privacy compared to state of the art, while preserving data utility.

A review of disentangled representation learning for visual data processing and analysis

表示学习是机器学习研究的核心问题之一。机器学习算法的输入表征从过去主流的手工特征过渡到现在面向多媒体数据的潜在表示，使算法性能获得了巨大提升。然而，视觉数据的表示通常是高度耦合的，即输入数据的所有信息成分被编码进同一个特征空间，从而互相影响且难以区分，使得表示的可解释性不高。解耦表示学习旨在学习一种低维的可解释性抽象表示，可以识别并分离出隐藏在高维观测数据中的不同潜在变化因素。通过解耦表示学习，可以捕获到单个变化因素信息并通过相对应的潜在子空间进行控制，因此解耦表示更具有可解释性。解耦表征可用于提高样本效率和对无关干扰因素的容忍度，为数据中的复杂变化提供一种鲁棒性表示，提取的语义信息对识别分类、域适应等人工智能下游任务具有重要意义。本文首先介绍并分析解耦表示的研究现状及其因果机制，总结解耦表示的3个重要性质。然后，将解耦表示学习算法分为4类，并从数学描述、类型特点及适用范围3个方面进行归纳及对比。随后，分类总结了现有解耦表示工作中常用的损失函数、数据集及客观评估指标。最后，总结了解耦表示学习在实际问题中的各类应用，并对其未来发展进行了探讨。;Representation learning is essential for machine learning technique nowadays. The transition of input representations have been developing intensively in algorithm performance benefited from the growth of hand-crafted features to the representation for multi-media data. However，the representations of visual data are often highly entangled. The interpretation challenges are to be faced because all information components are encoded into the same feature space. Disentangled representation learning（DRL）aims to learn a low-dimensional interpretable abstract representation that can sort the multiple factors of variation out in high-dimensional observations. In the disentangled representation，we can capture and manipulate the information of a single factor of variation through the corresponding latent subspace，which makes it more interpretable. DRL can improve sample efficiency and tolerance to the nuisance variables and offer robust representation of complex variations. Their semantic information is extracted and beneficial for artificial intelligence（AI）downstream tasks like recognition，classification and domain adaptation. Our summary is focused on brief introduction to the definition， research development and applications of DRL. Some of independent component analysis（ICA）-nonlinear DRL researches are covered as well since the DRL is similar to the identifiability issue of nonlinear independent component analysis（nonlinear ICA）. The cause and effects mechanism of DRL as high-dimensional ground truth data is generated by a set of unobserved changing factors（generating factors）. The DRL can be used to model the factors of variation in terms of latent representation，and the observed data generation process is restored. We summarize the key elements that a well-defined disentangled representation should be qualified into three aspects，which are 1）modularity，2）compactness，and 3）explicitness. First，explicitness consists of the two sub-requirements of completeness and informativeness. Then，current DRL types are categorized into 1）dimension-wise disentanglement，2）semantic-based disentanglement，3）hierarchical disentanglement，and 4） nonlinear ICA four types in terms of its formulation，characteristics，and scope of application. Dimension-wise disentanglement is assumed that the generative factors are solely and each dimension of latent vector can be separated and mapped，which is suitable for learning the disentangled representation of simple synthetic visual data. Semantic-based disentanglement is hypnotized that some semantic information is solely as well. The generative factors are group-disentangled in terms of specific semantics and they are mapped to different latent spaces，which is suitable for complicated ground truth data. Hierarchical disentanglement is based on the assumption that there is a correlation between generative factors at different levels of abstraction. The generative factors are disentangled by group from the bottom up and they can be mapped to latent space of different semantic abstraction levels to form a hierarchical disentangled representation. Nonlinear ICA provides an identifiable method for observed data-mixed disentangling unknown generative factors through a nonlinear reversible generator. For the motivation of loss functions，the loss functions can be commonly used in disentangled representation learning，which are grouped into three categories：1）modularity constraint：a single latent variable-constrained in the disentangled representation to capture only a single or a single group of factors of variation，and it promotes the separation of factors of variation mutually；2）explicitness constraint：current latent variable of the latent representation is activated to encode the ground truth of the corresponding generating factor effectively，and the entire latent representation contains complete information about all generative factors；and 3）multi-purpose constraint：lossrelated can optimize multiple disentangled representation，including modularity，compactness，and explicitness of the disentangled representation at the same time. The model-relevant can combine multiple loss constraint terms to form the final hybrid objective function. We compare the scope of application and limitations of each type of loss functions and summarize the classical disentangled representation works using the hybrid objective function further.

An Effective Differential Privacy Protection Method of Location data Based on Perturbation Loss Constraint

An Effective Differential Privacy Protection Method of Location data Based on Perturbation Loss Constraint

Dual Attention and Patient Similarity Network for drug recommendation.

Artificially making clinical decisions for patients with multi-morbidity has long been considered a thorny problem due to the complexity of the disease. Drug recommendations can assist doctors in automatically providing effective and safe drug combinations conducive to treatment and reducing adverse reactions. However, the existing drug recommendation works ignored two critical information. (i) Different types of medical information and their interrelationships in the patient's visit history can be used to construct a comprehensive patient representation. (ii) Patients with similar disease characteristics and their corresponding medication information can be used as a reference for predicting drug combinations. To address these limitations, we propose DAPSNet, which encodes multi-type medical codes into patient representations through code- and visit-level attention mechanisms, while integrating drug information corresponding to similar patient states to improve the performance of drug recommendation. Specifically, our DAPSNet is enlightened by the decision-making process of human doctors. Given a patient, DAPSNet first learns the importance of patient history records between diagnosis, procedure and drug in different visits, then retrieves the drug information corresponding to similar patient disease states for assisting drug combination prediction. Moreover, in the training stage, we introduce a novel information constraint loss function based on the information bottleneck principle to constrain the learned representation and enhance the robustness of DAPSNet. We evaluate the proposed DAPSNet on the public MIMIC-III dataset, our model achieves relative improvements of 1.33%, 1.20% and 2.03% in Jaccard, F1 and PR-AUC scores, respectively, compared to state-of-the-art methods. The source code is available at the github repository: https://github.com/andylun96/DAPSNet.

Hybrid Contrastive Learning for Unsupervised Person Re-Identification

Unsupervised person re-identification (Re-ID) aims to learn discriminative features without human-annotated labels. Recently, contrastive learning provides a new prospect for unsupervised person Re-ID, and existing methods mainly constrain the feature similarity among easy sample pairs. However, the feature similarity among hard sample pairs is neglected, which causes suboptimal performance in unsupervised person Re-ID. In the paper, we propose a novel Hybrid Contrastive Model (HCM) to perform the identity-level contrastive learning and the image-level contrastive learning for unsupervised person Re-ID, which adequately explores the feature similarity among hard sample pairs. Specifically, for the identity-level contrastive learning, an identity-based memory is constructed to store pedestrian features. Accordingly, we define the dynamic contrast loss to identify identity information with dynamic factor for distinguishing hard/easy samples. As for the image-level contrastive learning, an image-based memory is established to store each image feature. We design the sample constraint loss to explore the similarity relationship between hard positive and negative sample pairs. Furthermore, we optimize the two contrastive learning processes in one unified framework to make use of their own advantages as so to constrain the feature distribution for extracting potential information. Extensive experiments prove that the proposed HCM distinctly outperforms the state-of-the-art methods.

Unsupervised Adversarial Domain Adaptation for Sim-to-Real Transfer of Tactile Images

Transferring optical tactile skills learned from simulated environments to the real world benefits many robotic tactile applications, which can reduce the cost of data collection. However, the models purely trained on simulated data are often difficult to generalize well to the unseen real world due to the domain gap between the faultless training images and testing images with unpredictable manufacturing defects or natural wear. In this paper, we propose an Adaptively Correlation-attentive and Task-related Network (ACTNet) for tactile image transfer, a novel unsupervised adversarial domain adaptation method to narrow the domain gap for pixel-level tactile perception tasks. An adaptively correlative attention mechanism is introduced to improve the generator, which is capable of leveraging global information and focusing on salient regions. We also construct a task-related constraint loss based on the tactile image classification task to facilitate the zero-shot sim-to-real transfer of the tactile image classifier. Extensive experiment results on tactile image generation and tactile image classification indicate that our method outperforms the state-of-the-art method. Classification accuracy of 92.85% in the real-world environment is achieved without using any real-world classification labels.

Notch Effects on the Stress Intensity Factor and on the Fatigue Crack Path for Eccentric Circular Internal Cracks in Elliptically Notched Round Bars under Tensile Loading.

In this paper, stress intensity factor (SIF) solutions are numerically obtained for notched bars subjected to tensile loading containing an eccentric circular inner crack located in the cross-section corresponding to the notch root. The finite element method and the J-integral have been used to obtain the SIF and to analyze the effect on it of three elliptical notch geometries (of equal radial depth). The results show how the SIF is greater in the notched bars than in the smooth bar and within the former when the axial semi-axis of the notch rises, its effect being greater as the diameter and eccentricity of the inner crack increase. In addition, the fatigue growth of an eccentric crack induces an increase in such eccentricity, greater as the notch axial semi-axis increases. The cause of these phenomena can be attributed to the constraint loss caused by the notch, which also facilitates bending of the specimen due to the asymmetry generated by the crack eccentricity.

Highly Efficient Fault Diagnosis of Rotating Machinery Under Time-Varying Speeds Using LSISMM and Small Infrared Thermal Images

The existing fault diagnosis methods of rotating machinery constructed with both shallow learning and deep learning models are mostly based on vibration analysis under steady rotating speed. However, the rotating speed frequently changes to meet practical engineering needs. The shallow learning models largely depend on domain experience of feature extraction, and training a deep learning model requires large samples and a long time. In addition, vibration monitoring has the shortcomings of contact measurement, small coverage, and noise interference. To address these problems, this article proposes a new fault diagnosis method with the least square interactive support matrix machine (LSISMM) and infrared thermal images. In this method, a novel matrix-form classifier called LSISMM is constructed under the concept of nonparallel interactive hyperplanes to fully leverage the structure information of infrared thermal images. To improve the computation efficiency, a new least square loss constraint is designed for LSISMM. Besides, we derive an effective solution framework based on the alternating direction method of the multiplier (ADMM) framework. The constructed LSISMM is directly used to analyze the collected thermal images of rotating machinery under time-varying speeds. Experiment results demonstrate that the proposed method is superior to state-of-the-art methods in terms of diagnosis accuracy and efficiency, especially under small thermal image samples.

Self-Supervised Reinforcement Learning with dual-reward for knowledge-aware recommendation

To improve the recommendation accuracy and offer explanations for recommendations, Reinforcement Learning (RL) has been applied to path reasoning over knowledge graphs. However, in recommendation tasks, most existing RL methods learn the path-finding policy using only a short-term or single reward, leading to a local optimum and losing some potential paths. To address these issues, we propose a Self-Supervised Reinforcement Learning (SSRL) framework combined with dual-reward for knowledge-aware recommendation reasoning over knowledge graphs. Then, we improve Actor–Critic algorithm by using a dual-reward driven strategy, which combines short-term reward with long-term incremental evaluation. The improved algorithm helps the policy guide path reasoning in an overall situation. In addition, to find the most potential paths, in the improved Actor–Critic algorithm, a loss constraint of each sample is used as a reinforced signal to update the gradients. With some improvements against baselines, experimental results demonstrate the effectiveness of our framework.

Binary feature learning with local spectral context-aware attention for classification of hyperspectral images

The classification of hyperspectral images (HSIs) has achieved success in applications. For many approaches, features are directly extracted from whole spectral pixels, which can not well describe local characteristics. These methods are also susceptible to noise since each feature code is learned individually. Accordingly, a binary feature learning method with local spectral context-aware attention (BFLSC) is proposed for the classification. Specifically, for training samples, we first build the local spectrum models (LSMs) to describe local spectral properties, where each training sample is segmented into some parts and the difference between the central value and its neighborhoods is calculated in each part. Then, we construct the BFLSC model to learn a projection and binary features of training samples. In such model, the spectral context-awareness attention is established to collaboratively learn binary feature codes by enforcing one shift between 0/1 of each LSM, which enhances the robustness and stability of binary leaning. We also introduce the loss constraint, even distribution constraint, and variance constraint to reduce information loss and improve the quality of learned feature distribution. Additionally, an optimization scheme is designed to obtain the solution of the BFLSC model. Further, the learned binary features are added to train the support vector machine (SVM). For each testing sample, the LSMs are first extracted, and then mapped into binary features by the learned projection. The trained SVM is finally used for the mapped binary features to predict the label of the testing sample. Experimental results validate that our BFLSC realizes the better performance compared with some advanced approaches.

Design of a Chamfered Structure on Consequent-Pole Vernier Permanent-Magnet Machine

The consequent-pole (CP) Vernier permanent-magnet (VPM) machine has been developed over the last decade. In VPM machine, the CP structure can produce considerable torque with a half volumn of the PMs compared with the regular structure, so the cost is reduced and the mechanical strength is increased. In this paper, an improvement of chamfering structure on a CPVPM machine is proposed to alleviate the flux leakage and increase the torque density. The chamfered structure is easily machined and will not influence the robustness of the rotor. The comparison results show that under the same volume and copper loss constraint, the proposed structure has smaller cogging torque, smaller torque ripple, larger torque density and larger power factor.

Robust self-supervised monocular visual odometry based on prediction-update pose estimation network

Visual odometry aims at estimating the camera pose from video sequence, which is an important part of visual Simultaneous Localization and Mapping (SLAM). In this paper, we propose a novel prediction-update pose estimation network, PU-PoseNet, for self-supervised monocular visual odometry. It allows the network to use the effective information of the previous frame in estimating the current pose. The long-time pose consistency constraint-based motion weighted photometric loss is designed to make the network to pay more attention to the pixels of stationary objects and enhance the time consistency of estimation results. The depth image-based occlusion detection, the depth smoothness loss and auto-Masking are used to construct the depth consistency constraint loss term to reduce the influences of interferences such as occlusion. To further improve the robustness and the accuracy of the proposed method, both the depth consistency constraint and the variational auto-encoder are used for network training. For frame missing cases, a novel frame missing training strategy is used to make our method adapt frame missing cases. Extensive experiments on the KITTI dataset have validated the effectiveness of our proposed method.