Weighted Loss Function Research Articles

Adaptive Weighted Discriminator for Training Generative Adversarial Networks.

Generative adversarial network (GAN) has become one of the most important neural network models for classical unsupervised machine learning. A variety of discriminator loss functions have been developed to train GAN's discriminators and they all have a common structure: a sum of real and fake losses that only depends on the actual and generated data respectively. One challenge associated with an equally weighted sum of two losses is that the training may benefit one loss but harm the other, which we show causes instability and mode collapse. In this paper, we introduce a new family of discriminator loss functions that adopts a weighted sum of real and fake parts, which we call adaptive weighted loss functions or aw-loss functions. Using the gradients of the real and fake parts of the loss, we can adaptively choose weights to train a discriminator in the direction that benefits the GAN's stability. Our method can be potentially applied to any discriminator model with a loss that is a sum of the real and fake parts. For our experiments, SN-GAN, AutoGAN, and BigGAN are used. Experiments validated the effectiveness of our loss functions on unconditional and conditional image generation tasks, improving the baseline results by a significant margin on CIFAR-10, STL-10, and CIFAR-100 datasets in Inception Scores (IS) and Fréchet Inception Distance (FID) metrics.

Multitask Learning with Knowledge Base for Joint Intent Detection and Slot Filling

Intent detection and slot filling are important modules in task-oriented dialog systems. In order to make full use of the relationship between different modules and resource sharing, solving the problem of a lack of semantics, this paper proposes a multitasking learning intent-detection system, based on the knowledge-base and slot-filling joint model. The approach has been used to share information and rich external utility between intent and slot modules in a three-part process. First, this model obtains shared parameters and features between the two modules based on long short-term memory and convolutional neural networks. Second, a knowledge base is introduced into the model to improve its performance. Finally, a weighted-loss function is built to optimize the joint model. Experimental results demonstrate that our model achieves better performance compared with state-of-the-art algorithms on a benchmark Airline Travel Information System (ATIS) dataset and the Snips dataset. Our joint model achieves state-of-the-art results on the benchmark ATIS dataset with a 1.33% intent-detection accuracy improvement, a 0.94% slot filling F value improvement, and with 0.19% and 0.31% improvements respectively on the Snips dataset.

Multi-Resolution Supervision Network with an Adaptive Weighted Loss for Desert Segmentation

Desert segmentation of remote sensing images is the basis of analysis of desert area. Desert images are usually characterized by large image size, large-scale change, and irregular location distribution of surface objects. The multi-scale fusion method is widely used in the existing deep learning segmentation models to solve the above problems. Based on the idea of multi-scale feature extraction, this paper took the segmentation results of each scale as an independent optimization task and proposed a multi-resolution supervision network (MrsSeg) to further improve the desert segmentation result. Due to the different optimization difficulty of each branch task, we also proposed an auxiliary adaptive weighted loss function (AWL) to automatically optimize the training process. MrsSeg first used a lightweight backbone to extract different-resolution features, then adopted a multi-resolution fusion module to fuse the local information and global information, and finally, a multi-level fusion decoder was used to aggregate and merge the features at different levels to get the desert segmentation result. In this method, each branch loss was treated as an independent task, AWL was proposed to calculate and adjust the weight of each branch. By giving priority to the easy tasks, the improved loss function could effectively improve the convergence speed of the model and the desert segmentation result. The experimental results showed that MrsSeg-AWL effectively improved the learning ability of the model and has faster convergence speed, lower parameter complexity, and more accurate segmentation results.

Effective Prediction of Bug-Fixing Priority via Weighted Graph Convolutional Networks

With the increasing number of software bugs, bug fixing plays an important role in software development and maintenance. To improve the efficiency of bug resolution, developers utilize bug reports to resolve given bugs. Especially, bug triagers usually depend on bugs' descriptions to suggest priority levels for reported bugs. However, manual priority assignment is a time-consuming and cumbersome task. To resolve this problem, recent studies have proposed many approaches to automatically predict the priority levels for the reported bugs. Unfortunately, these approaches still face two challenges that include words' nonconsecutive semantics in bug reports and the imbalanced data. In this article, we propose a novel approach that graph convolutional networks (GCN) based on weighted loss function to perform the priority prediction for bug reports. For the first challenge, we build a heterogeneous text graph for bug reports and apply GCN to extract words' semantics in bug reports. For the second challenge, we construct a weighted loss function in the training phase. We conduct the priority prediction on four open-source projects, including Mozilla, Eclipse, Netbeans, and GNU compiler collection. Experimental results show that our method outperforms two baseline approaches in terms of the F-measure by weighted average of 13.22%.

GSNet: Learning Spatial-Temporal Correlations from Geographical and Semantic Aspects for Traffic Accident Risk Forecasting

Traffic accident forecasting is of great importance to urban public safety, emergency treatment, and construction planning. However, it is very challenging since traffic accidents are affected by multiple factors, and have multi-scale dependencies on both spatial and temporal dimensional features. Meanwhile, traffic accidents are rare events, which leads to the zero-inflated issue. Existing traffic accident forecasting methods cannot deal with all above problems simultaneously. In this paper, we propose a novel model, named GSNet, to learn the spatial-temporal correlations from geographical and semantic aspects for traffic accident risk forecasting. In the model, a Spatial-Temporal Geographical Module is designed to capture the geographical spatial-temporal correlations among regions, while a Spatial-Temporal Semantic Module is proposed to model the semantic spatial-temporal correlations among regions. In addition, a weighted loss function is designed to solve the zero-inflated issue. Extensive experiments on two real-world datasets demonstrate the superiority of GSNet against the state-of-the-art baseline methods.

Uncertain Graph Neural Networks for Facial Action Unit Detection

Capturing the dependencies among different facial action units (AU) is extremely important for the AU detection task. Many studies have employed graph-based deep learning methods to exploit the dependencies among AUs. However, the dependencies among AUs in real world data are often noisy and the uncertainty is essential to be taken into consideration. Rather than employing a deterministic mode, we propose an uncertain graph neural network (UGN) to learn the probabilistic mask that simultaneously captures both the individual dependencies among AUs and the uncertainties. Further, we propose an adaptive weighted loss function based on the epistemic uncertainties to adaptively vary the weights of the training samples during the training process to account for unbalanced data distributions among AUs. We also provide an insightful analysis on how the uncertainties are related to the performance of AU detection. Extensive experiments, conducted on two benchmark datasets, i.e., BP4D and DISFA, demonstrate our method achieves the state-of-the-art performance.

Reactor Coolant Pump Leakage Estimation of PWR Based on Broad Learning System

Loss-of-coolant accident (LOCA) of reactor coolant pump is considered as a critical issue in pressurized water reactor (PWR) accident analysis. Much work focus on the detection of LOCA, while little work has been done on leakage quantity estimation. In this paper, a reactor coolant leakage estimation model is proposed based on broad learning system (BLS) model which takes the advantages of the flatted structure and incremental learning. Considering that larger leakage is more concerned, the BLS model is improved by a weighted loss function. Dropout layer and white noise are added to improve the robustness of the model. This model is designed to estimate the coolant leakage in an online manner with high precision. The proposed model is evaluated on real leakage data of a reactor coolant pump. Experiments show that, with similar accuracy, the proposed model is significantly faster than the state-of-art deep neural networks.

Deep learning-based automated and universal bubble detection and mask extraction in complex two-phase flows

While investigating multiphase flows experimentally, the spatiotemporal variation in the interfacial shape between different phases must be measured to analyze the transport phenomena. For this, numerous image processing techniques have been proposed, showing good performance. However, they require trial-and-error optimization of thresholding parameters, which are not universal for all experimental conditions; thus, their accuracy is highly dependent on human experience, and the overall processing cost is high. Motivated by the remarkable improvements in deep learning-based image processing, we trained the Mask R-CNN to develop an automated bubble detection and mask extraction tool that works universally in gas–liquid two-phase flows. The training dataset was rigorously optimized to improve the model performance and delay overfitting with a finite amount of data. The range of detectable bubble size (particularly smaller bubbles) could be extended using a customized weighted loss function. Validation with different bubbly flows yields promising results, with AP50 reaching 98%. Even while testing with bubble-swarm flows not included in the training set, the model detects more than 95% of the bubbles, which is equivalent or superior to conventional image processing methods. The pure processing speed for mask extraction is more than twice as fast as conventional approaches, even without counting the time required for tedious threshold parameter tuning. The present bubble detection and mask extraction tool is available online (https://github.com/ywflow/BubMask).

Effects of Low-Carbohydrate versus Mediterranean Diets on Weight Loss, Glucose Metabolism, Insulin Kinetics and β-Cell Function in Morbidly Obese Individuals

Low-calorie Mediterranean-style or low-carbohydrate dietary regimens are widely used nutritional strategies against obesity and associated metabolic diseases, including type 2 diabetes. The aim of this study was to compare the effectiveness of a balanced Mediterranean diet with a low-carbohydrate diet on weight loss and glucose homeostasis in morbidly obese individuals at high risk to develop diabetes. Insulin secretion, insulin clearance, and different β-cell function components were estimated by modeling plasma glucose, insulin and C-peptide profiles during 75-g oral glucose tolerance tests (OGTTs) performed at baseline and after 4 weeks of each dietary intervention. The average weight loss was 5%, being 58% greater in the low-carbohydrate-group than Mediterranean-group. Fasting plasma glucose and glucose tolerance were not affected by the diets. The two dietary regimens proved similarly effective in improving insulin resistance and fasting hyperinsulinemia, while enhancing endogenous insulin clearance and β-cell glucose sensitivity. In summary, we demonstrated that a low-carbohydrate diet is a successful short-term approach for weight loss in morbidly obese patients and a feasible alternative to the Mediterranean diet for its glucometabolic benefits, including improvements in insulin resistance, insulin clearance and β-cell function. Further studies are needed to compare the long-term efficacy and safety of the two diets.

MULTI-DOMAIN LEARNING BY META-LEARNING: TAKING OPTIMAL STEPS IN MULTI-DOMAIN LOSS LANDSCAPES BY INNER-LOOP LEARNING.

We consider a model-agnostic solution to the problem of Multi-Domain Learning (MDL) for multi-modal applications. Many existing MDL techniques are model-dependent solutions which explicitly require nontrivial architectural changes to construct domain-specific modules. Thus, properly applying these MDL techniques for new problems with well-established models, e.g. U-Net for semantic segmentation, may demand various low-level implementation efforts. In this paper, given emerging multi-modal data (e.g., various structural neuroimaging modalities), we aim to enable MDL purely algorithmically so that widely used neural networks can trivially achieve MDL in a model-independent manner. To this end, we consider a weighted loss function and extend it to an effective procedure by employing techniques from the recently active area of learning-to-learn (meta-learning). Specifically, we take inner-loop gradient steps to dynamically estimate posterior distributions over the hyperparameters of our loss function. Thus, our method is model-agnostic, requiring no additional model parameters and no network architecture changes; instead, only a few efficient algorithmic modifications are needed to improve performance in MDL. We demonstrate our solution to a fitting problem in medical imaging, specifically, in the automatic segmentation of white matter hyperintensity (WMH). We look at two neuroimaging modalities (T1-MR and FLAIR) with complementary information fitting for our problem.

PROSTATE CANCER DIAGNOSIS WITH SPARSE BIOPSY DATA AND IN PRESENCE OF LOCATION UNCERTAINTY.

Prostate cancer is the second most prevalent cancer in men worldwide. Deep neural networks have been successfully applied for prostate cancer diagnosis in magnetic resonance images (MRI). Pathology results from biopsy procedures are often used as ground truth to train such systems. There are several sources of noise in creating ground truth from biopsy data including sampling and registration errors. We propose: 1) A fully convolutional neural network (FCN) to produce cancer probability maps across the whole prostate gland in MRI; 2) A Gaussian weighted loss function to train the FCN with sparse biopsy locations; 3) A probabilistic framework to model biopsy location uncertainty and adjust cancer probability given the deep model predictions. We assess the proposed method on 325 biopsy locations from 203 patients. We observe that the proposed loss improves the area under the receiver operating characteristic curve and the biopsy location adjustment improves the sensitivity of the models.

Resveratrol Enhances Inhibition Effects of Cisplatin on Cell Migration and Invasion and Tumor Growth in Breast Cancer MDA-MB-231 Cell Models In Vivo and In Vitro.

Triple-negative breast cancer (TNBC) is a refractory type of breast cancer that does not yet have clinically effective drugs. The aim of this study is to investigate the synergistic effects and mechanisms of resveratrol combined with cisplatin on human breast cancer MDA-MB-231 (MDA231) cell viability, migration, and invasion in vivo and in vitro. In vitro, MTS assays showed that resveratrol combined with cisplatin inhibits cell viability as a concentration-dependent manner, and produced synergistic effects (CI < 1). Transwell assay showed that the combined treatment inhibits TGF-β1-induced cell migration and invasion. Immunofluorescence assays confirmed that resveratrol upregulated E-cadherin expression and downregulated vimentin expression. Western blot assay demonstrated that resveratrol combined with cisplatin significantly reduced the expression of fibronectin, vimentin, P-AKT, P-PI3K, P-JNK, P-ERK, Sma2, and Smad3 induced by TGF-β1 (p < 0.05), and increased the expression of E-cadherin (p < 0.05), respectively. In vivo, resveratrol enhanced tumor growth inhibition and reduced body weight loss and kidney function impairment by cisplatin in MDA231 xenografts, and significantly reduced the expressions of P-AKT, P-PI3K, Smad2, Smad3, P-JNK, P-ERK, and NF-κB in tumor tissues (p < 0.05). These results indicated that resveratrol combined with cisplatin inhibits the viability of breast cancer MDA231 cells synergistically, and inhibits MDA231 cells invasion and migration through Epithelial-mesenchymal transition (EMT) approach, and resveratrol enhanced anti-tumor effect and reduced side of cisplatin in MDA231 xenografts. The mechanism may be involved in the regulations of PI3K/AKT, JNK, ERK and NF-κB expressions.

Wasserstein Distance-Based Auto-Encoder Tracking

Most of the existing visual object trackers are based on deep convolutional feature maps, but there have fewer works about finding new features for tracking. This paper proposes a novel tracking framework based on a full convolutional auto-encoder appearance model, which is trained by using Wasserstein distance and maximum mean discrepancy . Compared with previous works, the proposed framework has better performance in three aspects, including appearance model, update scheme, and state estimation. To address the issues of the original update scheme including poor discriminant performance under limited supervisory information, sample pollution caused by long term object occlusion, and sample importance unbalance, in this paper, a novel latent space importance weighting algorithm, a novel sample space management algorithm, and a novel IOU-based label smoothing algorithm are proposed respectively. Besides, an improved weighted loss function is adopted to address the sample imbalance issue. Finally, to improve the state estimation accuracy, the combination of Kullback-Leibler divergence and generalized intersection over union is introduced. Extensive experiments are performed on the three widely used benchmarks, and the results demonstrate the state-of-the-art performance of the proposed method. Code and models are available at https://github.com/wahahamyt/CAT.git .

A new deep distortion convolutional neural network for semantic segmentation of panoramic images

Semantic segmentation of panoramic images plays a crucial role in many applications, such as scene understanding, autonomous navigation, and community security. However, the traditional deep learning algorithms achieve lower accuracy for panoramic images due to the serious image distortion. This paper proposes a new deep convolutional neural network to segment panoramic images of outdoor scenes semantically. The proposed network includes two branches: the semantic segmentation sub-branch (SS-branch) and the feature enhancement sub-branch (FE-branch). The SS-branch contains two parts: Encoder and decoder. In the encoder, a new distortion convolution layer (DCL) is defined to reduce the distortion of panoramic images. In the FE-branch, a canny is used to detect the edge and a lightweight network is designed to enhance the features of image boundaries. Finally, a weighted loss function is used. The experimental results show that the proposed semantic segmentation method has better performance for different outdoor scenes.

Image restoration for irregular holes based on dual discrimination generation countermeasure network

In order to solve the problem that the global and local generated countermeasure network cannot inpaint the random irregular large holes, and to improve the standard convolution generator, which demonstrates the defects of color difference and blur, a network architecture of inpainting irregular large holes in an image based on double discrimination generation countermeasure network is proposed. Firstly, the image generator is a U-net architecture defined by partial convolution. The normalized partial convolution only completes the end-to-end mask update for the effective pixels. The skip link in U-net propagates the context information of the image to the higher resolution, and optimizes the training results of the model with the weighted loss function of reconstruction loss, perception loss and wind grid loss. Subsequently, the adversary loss function, the dual discrimination network including the synthetic discriminator and the global discriminator are trained separately to judge the consistency between the generated image and the real image. Finally, the weighted loss functions are trained together with generating network and double discrimination network to further enhance the detail and overall consistency of the inpainted area and make the inpainted results more natural. The simulation experiment is carried out on the Place 365 standard database. The subjective and objective experimental results show that the results of the proposed method has reasonable overall and detail semantic consistency than those of the existing methods when they are used to repair random, irregular and large-area holes. The proposed method effectively overcomes the defects of blurry details, color distortion and artifacts.

Recognition of interface and category of roadway roof strata based on drilling parameters

Measurement while drilling is an important part of the intelligent development of coal mines and the development trend of roof formation detection. The main purpose of this paper is to study the best identification method of rock interface and category based on drilling parameters. Firstly, the borehole drilling device was independently developed; Secondly, 4 concrete blocks were poured to simulate the different rock formations; Thirdly, displacement, rotational speed, torque, and sound pressure level data were collected during drilling the concrete blocks, and the original data was processed by the local linear kernel estimation method; Finally, K-means clustering and change points detection method, was used to identify the rock formation interface respectively, and BP neural network and support vector machine method was used to identify the rock category respectively. Experimental results show that the local linear kernel estimation method based on exponentially weighted loss function can better reflect the response characteristics of drilling parameters. The change point detection method based on the penetration rate is better for the rock interface prediction (the sum of the two interface errors is 10.25 mm). Besides, with the penetration rate and sound pressure level as input parameters, BP neural network and support vector machine can achieve the operational classification of rock formations. This research is of great significance for understanding the prediction of the interface and category of rock formation in coal mines, which can not only promote safe mining but also reflect efficient coal production.

Triangulate geometric constraint combined with visual-flow fusion network for accurate 6DoF pose estimation

Estimating the 6D object pose based on a monocular RGB image is a challenging task in computer vision, which produces false positives under the influence of occlusion or cluttered environments. In addition, the prediction of translation is affected by changes of the image size. In this work, we present a novel two-stage method TGCPose6D for robust 6DoF object pose estimation which is composed of 2D keypoint detection and translation refinement. In the first stage, the 2D keypoint regression space is constrained by triangulate geometric feature vectors, and the low-quality prediction is suppressed by the center-heatmap weighted loss function, thereby the performance of keypoint detection is significantly improved. In the second stage, the Visual-Flow Fusion network (VFFNet) is used to extract the visual feature and optical flow feature of the rendered image and the observed image, and to predict the relative translation based on the difference of features. Specifically, the VFFNet is trained iteratively to gain the ability to predict the relative translation deviation. Extensive experiments are conducted to demonstrate the effectiveness of the proposed TGCPose6D method. Our overall pose estimation pipeline outperforms state-of-the-art object pose estimation methods on several benchmarks.

Multi-Crop Convolutional Neural Networks for Fast Lung Nodule Segmentation

Computed tomography (CT) images are formally taken as an assistance of early diagnosis in lung nodule analysis. Thus the accurate lung nodule segmentation is in great need for image-driven tasks. However, as heterogeneity exists between different types of lung nodules, the similar visual appearance between the pixels of nodules and pixels of non-nodule area make it difficult for automatic lung nodule segmentation. In this article, we propose a fast end-to-end framework, called Fast Multi-crop Guided Attention (FMGA) network, to accurately segment lung nodules in CT images. Our method utilizes multi-crop nodule slices as input to aggregate contextual information (2D context from current image slice and 3D context from adjacent axial slices), and exploits a global convolutional layer for nodule pixel embedding matching. To further make use of the information from border pixels near the nodule margin for better segmentation, we develop a weighted loss function to facilitate the model training by considering a balanced class samples of pixels around the nodule margin. Moreover, we utilize a central pooling layer to facilitate the contexts feature propagation in pixel neighbors. We evaluate our method on the largest public lung CT dataset LIDC and the collected lung CT data from Wuhan local hospital, respectively. Experimental results show that FMGA achieves superior performance among the state-of-the-arts. In addition, we give an ablation study and visualization results to illustrate how each component works for accurate lung nodule segmentation.

Scalable multi-channel dilated CNN–BiLSTM model with attention mechanism for Chinese textual sentiment analysis

Due to the complex semantics of natural language, the multi-sentiment polarity of words, and the long-dependence of sentiments between words, the existing sentiment analysis methods (especially Chinese textual sentiment analysis) still face severe challenges. Aware of these issues, this paper proposes a scalable multi-channel dilated joint architecture of convolutional neural network and bidirectional long short-term memory (CNN–BiLSTM) model with an attention mechanism to analyze the sentiment tendency of Chinese texts. Through the multi-channel structure, this model can extract both the original context features and the multiscale high-level context features. Importantly, the number of the model channel can be optimally expanded according to the actual corpus. Furthermore, the attention mechanism including local attention and global attention is adopted to further distinguish the difference of features. The former is employed to weight the output features of each channel, and the latter is used to weight the fused features of all channels. Besides, an adaptive weighted loss function is designed to effectively avoid the imbalance of classes in training data. Finally, several experiments are performed to demonstrate the superior performance of the proposed model on two public datasets. Compared with word-level methods, the accuracy and Macro-F1 are respectively increased by over 1.19% and 0.9% on NLPCC2017-ECGC corpus, the accuracy and F1 are respectively increased by more than 1.7% and 1.214% on ChnSentiCorp-Htl-unba-10000 corpus. Compared with char-level pre-training methods, the accuracy and Macro-F1 also respectively achieve an improvement of over 3.416% and 4.324% on the NLPCC2017-ECGC corpus, the accuracy and F1 are respectively increased by more than 0.14% and 3% on the ChnSentiCorp-Htl-unba-10000 corpus.

Matching Large Baseline Oblique Stereo Images Using an End-to-End Convolutional Neural Network

The available stereo matching algorithms produce large number of false positive matches or only produce a few true-positives across oblique stereo images with large baseline. This undesired result happens due to the complex perspective deformation and radiometric distortion across the images. To address this problem, we propose a novel affine invariant feature matching algorithm with subpixel accuracy based on an end-to-end convolutional neural network (CNN). In our method, we adopt and modify a Hessian affine network, which we refer to as IHesAffNet, to obtain affine invariant Hessian regions using deep learning framework. To improve the correlation between corresponding features, we introduce an empirical weighted loss function (EWLF) based on the negative samples using K nearest neighbors, and then generate deep learning-based descriptors with high discrimination that is realized with our multiple hard network structure (MTHardNets). Following this step, the conjugate features are produced by using the Euclidean distance ratio as the matching metric, and the accuracy of matches are optimized through the deep learning transform based least square matching (DLT-LSM). Finally, experiments on Large baseline oblique stereo images acquired by ground close-range and unmanned aerial vehicle (UAV) verify the effectiveness of the proposed approach, and comprehensive comparisons demonstrate that our matching algorithm outperforms the state-of-art methods in terms of accuracy, distribution and correct ratio. The main contributions of this article are: (i) our proposed MTHardNets can generate high quality descriptors; and (ii) the IHesAffNet can produce substantial affine invariant corresponding features with reliable transform parameters.