Classification Loss Function Research Articles

P-Norm Attention Deep CORAL: Extending Correlation Alignment Using Attention and the P-Norm Loss Function

CORrelation ALignment (CORAL) is an unsupervised domain adaptation method that uses a linear transformation to align the covariances of source and target domains. Deep CORAL extends CORAL with a nonlinear transformation using a deep neural network and adds CORAL loss as a part of the total loss to align the covariances of source and target domains. However, there are still two problems to be solved in Deep CORAL: features extracted from AlexNet are not always a good representation of the original data, as well as joint training combined with both the classification and CORAL loss may not be efficient enough to align the distribution of the source and target domain. In this paper, we proposed two strategies: attention to improve the quality of feature maps and the p-norm loss function to align the distribution of the source and target features, further reducing the offset caused by the classification loss function. Experiments on the Office-31 dataset indicate that our proposed methodologies improved Deep CORAL in terms of performance.

Improved YOLO Network for Free-Angle Remote Sensing Target Detection

Despite significant progress in object detection tasks, remote sensing image target detection is still challenging owing to complex backgrounds, large differences in target sizes, and uneven distribution of rotating objects. In this study, we consider model accuracy, inference speed, and detection of objects at any angle. We also propose a RepVGG-YOLO network using an improved RepVGG model as the backbone feature extraction network, which performs the initial feature extraction from the input image and considers network training accuracy and inference speed. We use an improved feature pyramid network (FPN) and path aggregation network (PANet) to reprocess feature output by the backbone network. The FPN and PANet module integrates feature maps of different layers, combines context information on multiple scales, accumulates multiple features, and strengthens feature information extraction. Finally, to maximize the detection accuracy of objects of all sizes, we use four target detection scales at the network output to enhance feature extraction from small remote sensing target pixels. To solve the angle problem of any object, we improved the loss function for classification using circular smooth label technology, turning the angle regression problem into a classification problem, and increasing the detection accuracy of objects at any angle. We conducted experiments on two public datasets, DOTA and HRSC2016. Our results show the proposed method performs better than previous methods.

Classification and Prediction on the Effects of Nutritional Intake on Overweight/Obesity, Dyslipidemia, Hypertension and Type 2 Diabetes Mellitus Using Deep Learning Model: 4-7th Korea National Health and Nutrition Examination Survey.

Few studies have been conducted to classify and predict the influence of nutritional intake on overweight/obesity, dyslipidemia, hypertension and type 2 diabetes mellitus (T2DM) based on deep learning such as deep neural network (DNN). The present study aims to classify and predict associations between nutritional intake and risk of overweight/obesity, dyslipidemia, hypertension and T2DM by developing a DNN model, and to compare a DNN model with the most popular machine learning models such as logistic regression and decision tree. Subjects aged from 40 to 69 years in the 4–7th (from 2007 through 2018) Korea National Health and Nutrition Examination Survey (KNHANES) were included. Diagnostic criteria of dyslipidemia (n = 10,731), hypertension (n = 10,991), T2DM (n = 3889) and overweight/obesity (n = 10,980) were set as dependent variables. Nutritional intakes were set as independent variables. A DNN model comprising one input layer with 7 nodes, three hidden layers with 30 nodes, 12 nodes, 8 nodes in each layer and one output layer with one node were implemented in Python programming language using Keras with tensorflow backend. In DNN, binary cross-entropy loss function for binary classification was used with Adam optimizer. For avoiding overfitting, dropout was applied to each hidden layer. Structural equation modelling (SEM) was also performed to simultaneously estimate multivariate causal association between nutritional intake and overweight/obesity, dyslipidemia, hypertension and T2DM. The DNN model showed the higher prediction accuracy with 0.58654 for dyslipidemia, 0.79958 for hypertension, 0.80896 for T2DM and 0.62496 for overweight/obesity compared with two other machine leaning models with five-folds cross-validation. Prediction accuracy for dyslipidemia, hypertension, T2DM and overweight/obesity were 0.58448, 0.79929, 0.80818 and 0.62486, respectively, when analyzed by a logistic regression, also were 0.52148, 0.66773, 0.71587 and 0.54026, respectively, when analyzed by a decision tree. This study observed a DNN model with three hidden layers with 30 nodes, 12 nodes, 8 nodes in each layer had better prediction accuracy than two conventional machine learning models of a logistic regression and decision tree.

DGANet: Dynamic Gradient Adjustment Anchor-Free Object Detection in Optical Remote Sensing Images

Remote sensing image object detection has been studied by many researchers in recent years using deep neural networks. However, optical remote sensing images contain many scenes with small and dense objects, resulting in a high rate of misrecognition. Firstly, in this work we selected a deep layer aggregation network with updated deformable convolution layers as the backbone to extract object features. The detection and classification of objects was based on the center-point network without non-maximum suppression. Secondly, the dynamic gradient adjustment embedded into the classification loss function was put forward to harmonize the quantity imbalance between easy and hard examples, as well as between positive and negative examples. Furthermore, the complete intersection over union (CIoU) loss function was selected as the objective function of bounding box regression, which achieves better convergence speed and accuracy. Finally, in order to validate the effectiveness and precision of the dynamic gradient adjustment network (DGANet), we conducted a series of experiments in remote sensing public datasets UCAS-AOD and LEVIR. The comparison experiments demonstrate that the DGANet achieves a more accurate detection result in optical remote sensing images.

On Lagrangian L2-norm pinball twin bounded support vector machine via unconstrained convex minimization

With the introduction of the regularization term in the formulation of the well-known twin support vector machine (TWSVM) for classification , twin bounded support vector machine (TBSVM) method was proposed recently as an improved version by implementing the structural risk minimization principle. However, TBSVM employs hinge loss function and it is sensitive to noise and unstable to re-sampling. Since the pinball loss function related to quantile distance enjoys noise insensitivity property , a novel TBSVM method with squared pinball loss function for classification is proposed. The noise insensitivity and scatter minimization properties are discussed. Our formulation is further simplified as a pair of unconstrained strongly convex minimization problems in the dual space free of matrix inversion terms and having only m variables where m is the number of training examples. As opposed to TWSVM and TBSVM wherein approximate kernel generated surfaces are constructed, kernel trick is applied directly in our formulation and thereby elegant formulation as in the classical support vector machine (SVM) is achieved. Numerical experiments performed on a synthetic and thirteen benchmark datasets with noise where better or comparable generalization performance with faster learning speed by the proposed method confirms its suitability and applicability to problems of interest.

Handwritten Text Recognition using Deep Learning and Word Beam Search

This paper offers a solution to traditional handwriting recognition techniques using concepts of Deep learning and Word Beam Search. This paper explains about how an individual handwritten word is classified from the handwritten text by translating into a digital form. The digital form when trained with the Connectionist Temporal Classification (CTC) loss function, the output produced is a RNN. This is a matrix containing character probabilities for each time-step. The final text is mapped using a CTC decoding algorithm by converting the character probabilities. The recognized text is constructed by a list of words from the dictionary by using the token passing algorithm. It is found the running time of token passing depends on the size of dictionary. Also the numbers like arbitrary character strings will not able to decode. In this paper the decoding search algorithm word beam search is proposed, in order to tackle these types of problems. This methodology support to constrain words similar to those contained in a dictionary. It allows the character strings such as arbitrary non-word between the words, and integrates into a word-level language model. It is found the running time is better when compared with the token passing. The proposed algorithm comprises of the decoding algorithm named vanilla beam search and token passing using the IAM dataset and Bentham data set.

RETRACTED ARTICLE: Enhanced classification loss functions and regularization loss function (ECLFaRLF) algorithm for bowel cancer feature classification

Bowel cancer is one of the most common cancers as stated in the bowel cancer cases statistics. The proposed technique is to recognize the pattern of tissue affected by bowel cancer by using support vector machine (SVM) classification. This research aims to increase the accuracy of detecting and classifying bowel cancer with reduced processing time. The proposed method considered a feature extraction and image classification by Eenhanced Classification Loss Functions and Regularization Loss Function (ECLFaRLF) algorithm. This method allowed for more precise interpretations regarding the best associations for bowel cancer. The proposed was tested on colorectal images from different datasets commonly investigated in the proposed solution. The test was evaluated by applying 10-fold cross-validation method. All classification methods provide differentiation rate above processing time 0.413 s, and accuracy 95.67% for the state of the art solution, but by introducing SVM2 classification algorithm produce high accuracy rate with average accuracy is 97.02% over 95.67% and with processing time 0.359 s over 0.413 s. This reality shows the significance of the discriminating power of the SVM2 classifier. The proposed framework has presented an examination of feature extraction and classification techniques to help pathologists in the identifying of benign and malignant diagnosis of bowel cancer.

Deep learning for post‐hurricane aerial damage assessment of buildings

AbstractThis study aims to improve post‐disaster preliminary damage assessment (PDA) using artificial intelligence (AI) and unmanned aerial vehicle (UAV) imagery. In particular, a stacked convolutional neural network (CNN) architecture is introduced and trained on an in‐house visual dataset from Hurricane Dorian. To account for the ordinality of damage level classes, the cross‐entropy classification loss function is replaced with the square of earth mover's distance (EMD2) loss. The trained model achieves 65.6% building localization precision and 61% (90% considering ±1 class deviation from ground‐truth) classification accuracy. It also exhibits a positive accuracy–confidence correlation, which is valuable for model assessment in situations where ground‐truth information is not readily available. Finally, the outcome of damage assessment is compared with the literature by examining the relationship between building size and number of stories, and severity of induced disaster damage.

Empirical study on tangent loss function for classification with deep neural networks

Deep neural networks have been widely applied in natural language processing and computer vision tasks, which have achieved great successes due to their powerful ability for capturing sophisticated deep features. Currently, most of the neural networks are trained with cross-entropy (CE). However, the traditional CE loss function is sensitive to randomness induced from training samples. In this paper, we propose a novel loss function, namely tangent loss (TG), aiming to make classification models more stable while achieving comparable performance. The TG loss function trains the neural network in a way that emphasizes samples whose predictions deviate greatly from the targets at each training step. We make a systematical empirical study on TG loss, which is compared with CE loss on various classification tasks. Extensive experimental results on the real-world datasets demonstrate that the TG loss function can be readily applied with the existing neural networks and improves the stability of classification models, which can obtain better or comparable classification performance than the CE loss function.

Generalization of Cross-Entropy Loss Function for Image Classification

Classification task is one of the most common tasks in machine learning. This supervised learning problem consists in assigning each input to one of a finite number of discrete categories. Classification task appears naturally in numerous applications, such as medical image processing, speech recognition, maintenance systems, accident detection, autonomous driving etc.In the last decade methods of deep learning have proven to be extremely efficient in multiple machine learning problems, including classification. Whereas the neural network architecture might depend a lot on data type and restrictions posed by the nature of the problem (for example, real-time applications), the process of its training (i.e. finding model’s parameters) is almost always presented as loss function optimization problem.Cross-entropy is a loss function often used for multiclass classification problems, as it allows to achieve high accuracy results.Here we propose to use a generalized version of this loss based on Renyi divergence and entropy. We remark that in case of binary labels proposed generalization is reduced to cross-entropy, thus we work in the context of soft labels. Specifically, we consider a problem of image classification being solved by application of convolution neural networks with mixup regularizer. The latter expands the training set by taking convex combination of pairs of data samples and corresponding labels. Consequently, labels are no longer binary (corresponding to single class), but have a form of vector of probabilities. In such settings cross-entropy and proposed generalization with Renyi divergence and entropy are distinct, and their comparison makes sense.To measure effectiveness of the proposed loss function we consider image classification problem on benchmark CIFAR-10 dataset. This dataset consists of 60000 images belonging to 10 classes, where images are color and have the size of 32×32. Training set consists of 50000 images, and the test set contains 10000 images.For the convolution neural network, we follow [1] where the same classification task was studied with respect to different loss functions and consider the same neural network architecture in order to obtain comparable results.Experiments demonstrate superiority of the proposed method over cross-entropy for loss function parameter value α < 1. For parameter value α > 1 proposed method shows worse results than cross-entropy loss function. Finally, parameter value α = 1 corresponds to cross-entropy.

Neighborhood based decision theoretic rough set under dynamic granulation for BCI motor imagery classification

Brain Computer Interface is an interesting and important research field that has contributed widespread application systems. In the medical field, it is important for physically challenged persons to aid in rehabilitation and restoration. In Brain Computer Interface, computer acts as interface between brain signals and external device. The computer processes the brain signals and sends necessary instructions to external device. The external device helps in restoring the movement ability of patient. Motor imagery is the imagination of motor movements like hand, foot and tongue. There is an associated brain signal when the normal person moves their hand, foot and tongue. Similarly, there is an associated brain signal when the physically challenged person imagines moving their hand, foot and tongue. When this brain signal is analyzed by brain computer interface, it can facilitate motor movements through external device. The aim of this work is to analyze and classify the brain signals for motor movements to aid in rehabilitation and restoration. In this paper BCI Competition IV Dataset I, Dataset IIa, BCI Competition III Dataset IIIa and Neuroprosthetic EEG Dataset are analyzed A novel optimization technique with Neighborhood Decision Theoretic Rough Set under Dynamic Granulation is proposed for motor imagery classification. Neighborhood based Decision Theoretic Rough Set under Dynamic Granulation (NDTRS under DG) is hybrid approach combining two algorithms Neighborhood Rough Set and Decision Theoretic Rough Set under Dynamic Granulation ((DTRS under DG). Neighborhood Rough Set overcomes the drawback of discretization step in Rough Set. Decision Theoretic Rough Set under Dynamic Granulation algorithm has loss function for classification. The effectiveness of classification is improved since the loss function is involved in the construction of algorithm. The proposed method Neighborhood based Decision Theoretic Rough Set under Dynamic Granulation gives higher classification accuracy compared to existing approaches.

Improved Loss Function for Image Classification

An improved loss function free of sampling procedures is proposed to improve the ill‐performed classification by sample shortage. Adjustable parameters are used to expand the loss scope, minimize the weight of easily classified samples, and further substitute the sampling function, which are added to the cross‐entropy loss and the SoftMax loss. Experiment results indicate that improvements in all classification performance of our loss function are shown in various network architectures and on different datasets. To summarize, compared with traditional loss functions, our improved version not only elevates classification performance but also lowers the difficulty of network training.

Fully Convolutional Neural Network Structure and its Loss Function for Image Classification

Fully Convolutional Neural Network Structure and its Loss Function for Image Classification

A Method for MBTI Classification Based on Impact of Class Components

Predicting the personality type of text authors has a well-known usage in psychology with practical applications in business. From the data science perspective, we can look at this problem as a text classification task that can be tackled using natural language processing (NLP) and deep learning. This paper proposes a method and a novel loss function for multiclass classification using the Myers–Briggs Type Indicator (MBTI) approach for predicting the author’s personality type. Furthermore, this paper proposes an approach that improves the current results of the MBTI multiclass classification because it considers components of compound class labels as supportive elements for better classification according to MBTI. As such, it also provides a new perspective on this classification problem. The experimental results on long short-term memory (LSTM) and convolutional neural network (CNN) models outperform baseline models for multiclass classification, related research on multiclass classification, and most research with four binary approaches to MBTI classification. Moreover, other classification problems that target compound class labels and label parts with binary mutually exclusive values can benefit from this approach.

Few-Shot Object Detection via Sample Processing

Few-shot object detection (FSOD) eliminates the dependence on tremendous instances with manual annotations in conventional object detection. We deem that the scarcity of positive samples is the main reason that restricts the performance of FSOD detectors. In this paper, a novel FSOD model via sample processing, namely, FSSP, is proposed to detect objects accurately with only a few annotated samples, which is based on the structural design of the Siamese network and uses YOLOv3-SPP as the baseline. Central to FSSP are our designed self-attention (SAM) and positive-sample augmentation (PSA) modules. The former attempts to better extract the representative features of hard samples, and the latter expands the number and enriches the scale distribution of positive samples, inhibiting the growth of negative samples. For the fine-tuning phase, we modify the classification loss function to increase the punishment for hard samples. Experiments conducted on the PASCAL VOC and MS COCO datasets confirm that the proposed FSSP achieves competitive detection performance compared with state-of-the-art detectors.

Real-Time Image Semantic Segmentation Based on Attention Mechanism and Multi-Label Classification

<p indent=0mm>Improving the accuracy is the goal in real-time semantic segmentation, especially for fuzzy boundary pixel segmentation. We proposed a high-precision and real-time semantic segmentation algorithm based on cross-level attention mechanism and multi-label classification. The procedure started with an optimization of DeepLabv3 to achieve real-time segmentation speed. Then, a cross-level attention module was added, so that the high-level features provided pixel-level attention for the low-level features, so as to inhibit the output of inaccurate semantic information in the low-level features. In the training phase, the multi-label classification loss function was introduced to assist the supervised training. The experimental results on Cityscapes dataset and CamVid dataset show that the segmentation accuracy is 68.1% and 74.1% respectively, and the segmentation speed is 42 frames/s and 89 frames/s respectively. It achieves a good balance between segmentation speed and accuracy, can optimize edge segmentation, and has strong robustness in complex scene segmentation.

Sentiment Classification for Financial Texts Based on Deep Learning.

Sentiment classification for financial texts is of great importance for predicting stock markets and financial crises. At present, with the popularity of applications in the field of natural language processing (NLP) adopting deep learning, the application of automatic text classification and text-based sentiment classification has become more and more extensive. However, in the field of financial text-based sentiment classification, due to a lack of labeled samples, such applications are limited. A domain-adaptation-based financial text sentiment classification method is proposed in this paper, which can adopt source domain (SD) text data with sentiment labels and a large amount of unlabeled target domain (TD) financial text data as training samples for the proposed neural network. The proposed method is a cross-domain transfer-learning-based method. The domain classification subnetwork is added to the original neural network, and the domain classification loss function is also added to the original training loss function. Therefore, the network can simultaneously adapt to the target domain and then accomplish the classification task. The experiment of the proposed sentiment classification transfer learning method is carried out through an open-source dataset. The proposed method in this paper uses the reviews of Amazon Books, DVDs, electronics, and kitchen appliances as the source domain for cross-domain learning, and the classification accuracy rates can reach 65.0%, 61.2%, 61.6%, and 66.3%, respectively. Compared with nontransfer learning, the classification accuracy rate has improved by 11.0%, 7.6%, 11.4%, and 13.4%, respectively.

Study on the Identification Method of Human Upper Limb Flag Movements Based on Inception-ResNet Double Stream Network

To investigate the recognition effect of flag motions based on 9 axis sensors, starting with deep learning methods, this paper proposes a framework for feature extraction and recognition of flag movements recognition by employing an improved Inception-ResNet dual-stream network. In traditional signal recognition studies, Support Vector Machine (SVM), Random Forest (RF) and One Dimensional Convolutional Neural Network (CNN) are usually used to extract signals' features. In the meanwhile, the time series data sets such as flag movements are usually the standard data set after processing. Therefore, there are usually some limitations in traditional systems. First, in actual environment, there exists a lack of effective segmentation detection method for the samples of long time series, resulting in the deviation of the data set in the recognition process. Second, the One-Dimensional CNN framework used and the machine learning frameworks used in previous studies are difficult to process large quantities of data with too high computational memory. Based on these problems, this study proposes a signal change point detection model based on the diversity factory function in the signal segmentation and detection stage, miniaturizes the convolution kernel in the original CNN by using the Inception-ResNet(I-R) dual-flow network separable convolution method, and proposes a CrossEntropy-Logistic(C-L) joint classification loss function. Through conducting comparative experiments, it is found that the average calculation parameter of the CNN framework based on the Inception-ResNet model is 2.7×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">7</sup> , which is approximately 37% lower than the number of 3.7×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">7</sup> in the original CNN model. Finally, the recognition rate between C-L joint loss function and other models such as Multi-Layer Perceptron and Ensemble Learning in recent years are compared. Compared with Ensemble Learning-CrossEntropy (ELC) model, the C-L joint loss function can improve the recognition rate by nearly 5% according to the results of flag movements identification measured by several classification models.

Focal Gradient Harmonized Loss Function for Object Detection

Mainstream detectors in the field of object detection usually can be divided into one-stage and two-stage detectors. Compared to the two-stage detectors, one-stage detectors are more efficient and elegant, but are more easily disturbed by imbalanced data during the training. Inspired by the gradient harmonized single-stage detector (GHM detector)[1], we propose a focal gradient harmonizing loss function for classification (FGHM-C). FGHM-C loss function optimizes the method of generating example weights during training, thereby alleviating the impact of long-tailed distribution. Our proposed model achieves 42.0 mAP on COCO test-dev set, which is 0.4 points higher than the original GHM detector.

Self-paced active learning for deep CNNs via effective loss function

Deep convolutional neural networks have achieved great success on computer vision tasks, but the lack of labeled samples hinders the application of it in the real world. Active learning is one of the effective methods to address large volumes of unlabeled data. It interactively selects a few samples based on a certain criterion and queries their labels from annotators. In order to reduce the labor of annotation, we propose a novel framework “self-paced multi-criteria active learning” on the image classification task. Unlike previous work, we treat each iteration of active learning as the step of self-paced learning which considers the model should gradually proceed from simplicity to complexity in training. In the stage of selecting samples, we combine clustering with a multiple criteria selection method to reduce the negative effects of hard samples. In the training phase, we design a similarity classification loss function to mitigate the impact of the deficiency of labeled data. Experiments on multiple datasets demonstrate that our proposed method can achieve higher performance than current approaches.