Optimization Of Loss Function Research Articles

A Novel Real-Time Filtering Method to General Nonlinear Filtering Problem Without Memory

In this paper, the filtering problem for the general time-invariant nonlinear state-observation system is considered. Our work is based on the Yau-Yau filtering framework developed by S.-T. Yau and the third author in 2008. The key problem of Yau-Yau filtering framework is how to compute the solution to forward Kolmogorov equation (FKE) off-line effectively. Motivated by the supervised learning in machine learning, we develop an efficient method to numerically solve the FKE off-line from the point of view of optimization. Specifically, for the off-line computation part, the computation of the solution to a FKE is reduced to computing a linear system of equations by making the temporal inverse transformation and the loss function optimization, and we store the results for the preparation of on-line computation. For the on-line computation part, the unnormalized density function is approximated by a complete polynomial basis, and then the estimation of the state is computed using the stored off-line data. Our method has the merits of easily implementing, real-time and memoryless. More importantly, it can be applicable for moderate-high dimensional cases. Numerical experiments have been carried out to verify the feasibility of our method. Our algorithm outperforms extended Kalman filter, unscented Kalman filter and particle filter both in accuracy and costing time.

Application of chaotic Fish School Search optimization algorithm with exponential step decay in neural network loss function optimization

The Fish School Search (FSS) algorithm is a heuristic technique for finding globally optimal solutions. This algorithm is characterized by its simplicity in implementation, and high performance. Since the first mention of FSS, this effective optimization algorithm has been of a great interest among researches and practitioners around the globe. Modifications of FSS exist, applied to solve practical problems, including image reconstruction in electrical impedance tomography, finding optimal solutions in assembly line balancing problems, neural network structure optimization. In this paper, we consider a modification of the FSS algorithm, which uses chaos theory to generate uniformly distributed pseudorandom numbers, and incorporates exponential step decay. The described modified optimization algorithm is known as ETFSS, and is characterized by faster convergence speed and better performance. In order to further investigate the performance of the novel optimization algorithm, we apply ETFSS to neural network loss function optimization. In addition, we compare the described approach with other machine learning techniques, such as the support vector machine (SVM) algorithm, k-nearest neighbors (KNN) algorithm and back propagation-based neural network, trained using the adaptive moment estimation (Adam) optimizer. We visualize classification results using T-distributed stochastic neighbor embedding (TSNE) method, and uniform manifold approximation and projection (UMAP) method, in order to provide more details considering classification performance and dataset shape. The obtained results confirm, that ETFSS can produce slightly more accurate classifications when compared to backpropagation.

Survey on the Loss Function of Deep Learning in Face Recognition

With the continuous development of face recognition network, the selection of loss function plays an increasingly important role in improving accuracy. The loss function of face recognition network needs to minimize the intra-class distance while expanding the inter-class distance. So far, one of our mainstream loss function optimization methods is to add penalty terms, such as orthogonal loss, to further constrain the original loss function. The other is to optimize using the loss based on angular/cosine margin. The last is Triplet loss and a new type of joint optimization based on HST Loss and ACT Loss. In this paper, based on the three methods with good practical performance and the joint optimization method, various loss functions are thoroughly reviewed.

Multivariate Taguchi loss function optimization based on principal components analysis and normal boundary intersection

Optimization methods are widely used to improve industrial processes and enhance the quality characteristics of product, where process costs are directly linked. Given this assumption, this study aims to present a multivariate proposal of the Taguchi loss function, to model and optimize manufacturing processes, searching to establish values that prioritize quality and provide the minimum loss in view of the process costs. For this, design of experiments techniques will be used to model the process and the calculated loss functions. The strategy of principal components analysis is used to minimize the data dimension, considering the structure of variance–covariance. Then, the normal boundary intersection method is used to find the Pareto frontier. Based on the values, the method also proposes a total loss function equation, which is characterized as an approach to choose the optimal point based on the sum of the loss functions for the Pareto frontier through the process cost. To demonstrate the behavior of the method, the flux-cored arc welding of stainless-steel cladding process was applied. In view of the results, the method provided an optimal value at the Pareto frontier, contemplating an appropriate balance between minimal loss and higher quality, which were compared with other studies in the literature. The method also provided a reduction in computational effort of approximately 90% (from 210 to 21 subproblems), obtaining the best solution and contemplating the multivariate nature of the data.

Triple loss for hard face detection

Although face detection has been well addressed in the last decades, despite the achievements in recent years, effective detection of small, blurred and partially occluded faces in the wild remains a challenging task. Meanwhile, the trade-off between computational cost and accuracy is also an open research problem in this context. To tackle these challenges, in this paper, a novel context enhanced approach is proposed with structural optimization and loss function optimization. For loss function optimization, we introduce a hierarchical loss, referring to ``triple loss'' in this paper, to optimize the feature pyramid network (FPN) (Lin et al., 2017) based face detector. Additional layers are only applied during the training process. As a result, the computational cost is the same as FPN during inference. For structural optimization, we propose a context sensitive structure to increase the capacity of the prediction network to improve the accuracy of the output. In details, a three-branch inception subnet (Szegedy et al., 2015) based feature fusion module is employed to refine the original FPN without increasing the computational cost significantly, further improving low-level semantic information, which is originally extracted from a single convolutional layer in the backward pathway of FPN. The proposed approach is evaluated on two publicly available face detection benchmarks, FDDB and WIDER FACE. By using a VGG-16 based detector, experimental results indicate that the proposed method achieves a good balance between the accuracy and computational cost of face detection.

Fast location and classification of small targets using region segmentation and a convolutional neural network

The hickory nut has rich nutritional and high economic value, but its inner wall structure is complex, making it inconvenient to separate the kernel from the hickory nut fragments for consumption. In this study, we apply a deep convolutional neural network algorithm to hickory nut fragments after shell breaking, to accurately sort kernels, shells, and unseparated bodies. Region segmentation is used to eliminate interference when locating multiple small targets of different sizes, and a custom dataset of 15,000 target images after shell breaking is created. Then, three different loss function optimization methods are evaluated for model optimization. Results revealed that the model only minimizes the average cross-entropy loss, and that using dropout at the first fully connected layer with a dropout rate 0.1 provides the best shell, kernel, and unseparated body identification accuracy of 97.14%, 99.66%, and 96.48%, respectively. Notably, the kernel and unseparated body recognition rates are considerably improved over other methods. Finally, a total of 450 hickory nut fragments were divided into 30 batches for a sorting experiment. The resulting shell, kernel, and unseparated body identification accuracy is 94.00%, 98.67%, and 95.33%, respectively. These results demonstrate that the proposed method provides excellent classification of hickory nut fragments according to the material. This approach can be easily applied to any small target sorting application.

Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.

In medical image processing, Brain tumor segmentation plays an important role. Early detection of these tumors is highly required to give Treatment of patients. The patient's life chances are improved by the early detection of it. The process of diagnosing the brain tumoursby the physicians is normally carried out using a manual way of segmentation. It is time consuming and a difficult one. To solve these problems, Enhanced Convolutional Neural Networks (ECNN) is proposed with loss function optimization by BAT algorithm for automatic segmentation method. The primary aim is to present optimization based MRIs image segmentation. Small kernels allow the design in a deep architecture. It has a positive consequence with respect to overfitting provided the lesser weights are assigned to the network. Skull stripping and image enhancement algorithms are used for pre-processing. The experimental result shows the better performance while comparing with the existing methods. The compared parameters are precision, recall and accuracy. In future, different selecting schemes can be adopted to improve the accuracy.

Deep Learning based MURA Defect Detection

INTRODUCTION: MURA defects in LED/LCD panels are one of the most challenging defects for Automatic Defect Classification and Localization (ADC) due to their extremely low contrast when compared with the background. Manual detection is subjective, error prone, very tedious and time consuming. Even when the type of MURA defects can be ascertained manually, the exact bounding box for defect is hard to determine. Various heuristic based image processing techniques have been applied giving sub-optimal accuracy over generic datasets. OBJECTIVES: The primary objective of this paper is to check whether the state of the art DL (Deep Learning) network for general object classification and localization (MSCOCO PASCAL VOC etc.) can be applied successfully for MURA Defect Classification and Localization. METHODS: In this paper we present a single DL pipeline for classification and localization which for the first time is applied for MURA defects. Naive DL network - Single Shot multi-box Detector (SSD, pre-trained on ImageNet) was not sufficient to give a good F1 score because of the nature of the defect. Accuracy improved a little after applying various DL specific optimization methods such as loss function optimization, network optimization etc. Utilizing the knowledge from MURA domain for data augmentation, like filtering based on image capture wavelength etc. improved the results significantly. RESULTS: Using optimization techniques that are from both DL domain as well as specific to MURA domain, we show improvement in the accuracy of the base DL pipeline from ~30% to ~80%. Minimum heuristics were used to define the pipeline so that it can easily adapt to any new MURA dataset. The paper shows the importance of domain specific preprocessing steps for the designed network in case of MURA defects. CONCLUSION: Using DL, MURA classification and localization had not been tried before. For the first time we demonstrated results for both classification and localization of MURA defects using state-of-the-art DL network with F1~80%. We also conclude that state-of-the-art network for general object detection can be reused with the help of Transfer Learning (TL) concept and fine-tuned with MURA domain specific optimizations mentioned in paper for optimal performances in MURA domain.

Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification.

Top-k error is currently a popular performance measure on large scale image classification benchmarks such as ImageNet and Places. Despite its wide acceptance, our understanding of this metric is limited as most of the previous research is focused on its special case, the top-1 error. In this work, we explore two directions that shed more light on the top-k error. First, we provide an in-depth analysis of established and recently proposed single-label multiclass methods along with a detailed account of efficient optimization algorithms for them. Our results indicate that the softmax loss and the smooth multiclass SVM are surprisingly competitive in top-k error uniformly across all k, which can be explained by our analysis of multiclass top-k calibration. Further improvements for a specific k are possible with a number of proposed top-k loss functions. Second, we use the top-k methods to explore the transition from multiclass to multilabel learning. In particular, we find that it is possible to obtain effective multilabel classifiers on Pascal VOC using a single label per image for training, while the gap between multiclass and multilabel methods on MS COCO is more significant. Finally, our contribution of efficient algorithms for training with the considered top-k and multilabel loss functions is of independent interest.

Hierarchical Rank Aggregation with Applications to Nanotoxicology.

The development of high throughput screening (HTS) assays in the field of nanotoxicology provide new opportunities for the hazard assessment and ranking of engineered nanomaterials (ENMs). It is often necessary to rank lists of materials based on multiple risk assessment parameters, often aggregated across several measures of toxicity and possibly spanning an array of experimental platforms. Bayesian models coupled with the optimization of loss functions have been shown to provide an effective framework for conducting inference on ranks. In this article we present various loss-function-based ranking approaches for comparing ENM within experiments and toxicity parameters. Additionally, we propose a framework for the aggregation of ranks across different sources of evidence while allowing for differential weighting of this evidence based on its reliability and importance in risk ranking. We apply these methods to high throughput toxicity data on two human cell-lines, exposed to eight different nanomaterials, and measured in relation to four cytotoxicity outcomes. This article has supplementary material online.

Un metodo dinamico per l'analisi dell'omogeneità non lineare

Aim of this paper is to propose a new method for Adaptive Fuzzy Homogeneity Analysis. Van Rijckevorsel et al. (1987) define Fuzzy Homogeneity Analysis (FHA) as the Homogeneity analysis of non linear transformed data, where the transformation is performed using B-splines. The originality of the method consists of the inclusion of the knot location among the parameters to be defined and for the adaptivity of the process of optimization of the loss function. An algorithm to perform the proposed method is described. A comparison with the results of an exhaustive algorithm is reported to illustrate the performance of the proposed adaptive algorithm.