Terms Of Loss Function Research Articles

Novel imbalanced fault diagnosis method based on generative adversarial networks with balancing serial CNN and Transformer (BCTGAN)

In industrial production engineering, machines cannot sustain faults for extended periods, resulting in limited fault data collection. This scarcity of data impedes the advancement of accurate fault diagnosis models, particularly in the context of small-sample, extremely imbalanced fault modeling, which poses a significant and challenging problem. In this paper, we introduce an innovative data augmentation approach for addressing the small-sample imbalanced problem: the balanced conditioning CNN (Convolutional Neural Networks)-Transformer architecture based Generative Adversarial Network (BCTGAN). Firstly, we employ a serial CNN and Transformer architecture as the foundation of the generator. Secondly, we introduce micro-enhancement and SMOTE (Synthetic Minority Oversampling Technique) enhancement of samples to facilitate model training. Micro-enhancement simply augments the sample size, while SMOTE enhancement enhances diversity. Together, these techniques balance the training process by introducing new loss function terms. Finally, outlier samples are mitigated through a designed multi-domain feature screening method. Despite the added time cost and computational complexity, results from three instances demonstrate that the proposed method holds superior diagnostic results with great potential compared to traditional and general GAN-based fault diagnosis methods. Our work provides theoretical and practical insights for modeling the field of small-sample blended disequilibrium fault diagnosis.

Radiological segmentation of knee meniscus ultrasound images based on boundary constraints and multi-scale fusion network

Knee osteoarthritis (KOA) is one of the most popular joint diseases endangering human health because of its high incidence, disability, and younger onset. However, there is no suitable method for early diagnosis, evaluation, and treatment of KOA. In recent years, some clinical studies have found that ultrasound can detect early changes in KOA in advance, and the automatic segmentation of ultrasound images can achieve rapid and effective quantitative research on KOA. However, in ultrasound images, the soft tissue boundaries of the lesion are blurred, making lesion segmentation difficult. Although the U-Net family is one of the best networks for image segmentation, they still have defects such as blurred segmentation boundaries, distorted morphology, and insufficient accuracy when segmenting ultrasound images. To address this issue, we added attention, atrous spatial pyramid pooling (ASSP), and edge loss function terms into the Unet3+ network, which improved the contour clarity and accuracy of output images (the improved Unet3+ Dice acc = 78.74%. Then, we extract the key features of the improved Unet3+ for outputting meniscus images: calculating meniscus area and distance, where the average accuracy of the area is: area_ avg_ acc = 91.12%, with an average distance accuracy of distance_ avg_ acc = 91.14%. This thesis creates a new dataset collection from West China Hospital, Sichuan University, and automated measurement of knee meniscus protrusion area has been achieved for the first time. This article is the first to apply deep learning to ultrasound image segmentation of the knee meniscus, helping doctors conduct qualitative and quantitative analysis of the diagnosis and treatment of early KOA. The results indicate that the improved Unet3+ can assist doctors in automatically diagnosing and evaluating KOA based on meniscus ultrasound images, which is beneficial for guiding early clinical intervention.

Knowledge-aware Multisite Adaptive Graph Transformer for Brain Disorder Diagnosis.

Brain disorder diagnosis via resting-state functional magnetic resonance imaging (rs-fMRI) is usually limited due to the complex imaging features and sample size. For brain disorder diagnosis, the graph convolutional network (GCN) has achieved remarkable success by capturing interactions between individuals and the population. However, there are mainly three limitations: 1) The previous GCN approaches consider the non-imaging information in edge construction but ignore the sensitivity differences of features to non-imaging information. 2) The previous GCN approaches solely focus on establishing interactions between subjects (i.e., individuals and the population), disregarding the essential relationship between features. 3) Multisite data increase the sample size to help classifier training, but the inter-site heterogeneity limits the performance to some extent. This paper proposes a knowledge-aware multisite adaptive graph Transformer to address the above problems. First, we evaluate the sensitivity of features to each piece of non-imaging information, and then construct feature-sensitive and feature-insensitive subgraphs. Second, after fusing the above subgraphs, we integrate a Transformer module to capture the intrinsic relationship between features. Third, we design a domain adaptive GCN using multiple loss function terms to relieve data heterogeneity and to produce the final classification results. Last, the proposed framework is validated on two brain disorder diagnostic tasks. Experimental results show that the proposed framework can achieve state-of-the-art performance.

Universal optimality and robust utility bounds for metric differential privacy1

We study the privacy-utility trade-off in the context of metric differential privacy. Ghosh et al. introduced the idea of universal optimality to characterise the “best” mechanism for a certain query that simultaneously satisfies (a fixed) ε-differential privacy constraint whilst at the same time providing better utility compared to any other ε-differentially private mechanism for the same query. They showed that the Geometric mechanism is universally optimal for the class of counting queries. On the other hand, Brenner and Nissim showed that outside the space of counting queries, and for the Bayes risk loss function, no such universally optimal mechanisms exist. Except for the universal optimality of the Laplace mechanism, there have been no generalisations of these universally optimal results to other classes of differentially-private mechanisms. In this paper, we use metric differential privacy and quantitative information flow as the fundamental principle for studying universal optimality. Metric differential privacy is a generalisation of both standard (i.e., central) differential privacy and local differential privacy, and it is increasingly being used in various application domains, for instance in location privacy and in privacy-preserving machine learning. Similar to the approaches adopted by Ghosh et al. and Brenner and Nissim, we measure utility in terms of loss functions, and we interpret the notion of a privacy mechanism as an information-theoretic channel satisfying constraints defined by ε-differential privacy and a metric meaningful to the underlying state space. Using this framework we are able to clarify Nissim and Brenner’s negative results by (a) that in fact all privacy types contain optimal mechanisms relative to certain kinds of non-trivial loss functions, and (b) extending and generalising their negative results beyond Bayes risk specifically to a wide class of non-trivial loss functions. Our exploration suggests that universally optimal mechanisms are indeed rare within privacy types. We therefore propose weaker universal benchmarks of utility called privacy type capacities. We show that such capacities always exist and can be computed using a convex optimisation algorithm. Further, we illustrate these ideas on a selection of examples with several different underlying metrics.

Steganography for medical record image

Medical record images in EHR system are users’ privacy and an asset, and there is an urgent need to protect this data. Image steganography can offer a potential solution. A steganographic model for medical record images is therefore developed based on StegaStamp. In contrast to natural images, medical record images are document images, which can be very vulnerable to image cropping attacks. Therefore, we use text region segmentation and watermark region localization to combat the image cropping attack. The distortion network has been designed to take into account the distortion that can occur during the transmission of medical record images, making the model robust against communication induced distortions. In addition, based on StegaStamp, we innovatively introduced FISM as part of the loss function to reduce the ripple texture in the steganographic image. The experimental results show that the designed distortion network and the FISM loss function term can be well suited for the steganographic task of medical record images from the perspective of decoding accuracy and image quality.

Trajectory Planning With Deep Reinforcement Learning in High-Level Action Spaces

This paper presents a technique for trajectory planning based on parameterized high-level actions. These high-level actions are sub trajectories that have variable shape and duration. The use of high-level actions can improve the performance of guidance algorithms. Specifically, we show how the use of high-level actions improves the performance of guidance policies that are generated via reinforcement learning (RL). Reinforcement learning has shown great promise for solving complex control, guidance, and coordination problems but can still suffer from long training times and poor performance. This work shows how the use of high-level actions reduces the required number of training steps and increases the path performance of an RL-trained guidance policy. We demonstrate the method on a space-shuttle guidance example. We show the proposed method increases the path performance (latitude range) by 18% percent compared to a baseline RL implementation. Similarly, we show the proposed method achieves steady state during training with approximately 75% fewer training steps. We also show how the guidance policy enables effective performance in an obstacle field. Finally, this paper develops a loss function term for policy-gradient-based Deep RL, which is analogous to an anti-windup mechanism in feedback control. We demonstrate that the inclusion of this term in the underlying optimization increases the average policy return in our numerical example.

An Efficient Approach of Face Detection and Prediction of Drowsiness Using SVM

This article investigates an issue of road safety and a method for detecting drowsiness in images. More fatal accidents may be averted if fatigued drivers are using this technology accurately and the proposed models provide quick response by recognising the driver’s state of falling asleep. There are the following drowsiness models for depicting the possible eye state classifications as VGG16, VGG19, RESNET50, RESNET101 and MobileNetV2. The absence of a readily available and trustworthy eye dataset is perceived acutely in the realm of eye closure detection. On extracting the deep features of faces with VGG16, 98.68% accuracy has been achieved, VGG19 provides an accuracy of 98.74%, ResNet50 works with 65.69% accuracy, ResNet101 has achieved 95.77%, and MobileNetV2 is achieving 96.00% accuracy with the proposed dataset. The put forth model using the support vector machine (SVM) has been used to evaluate several models, and the present results in terms of loss function and accuracy have been obtained. In the proposed dataset, 99.85% accuracy in detecting facial expressions has been achieved. These experimental results show that the eye closure estimation has a higher accuracy and cheap processing cost, as well as the ability of the proposed framework for drowsiness.

Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression

Many real-world systems can be described by mathematical models that are human-comprehensible, easy to analyze and can be helpful in explaining the system’s behavior. Symbolic regression is a method that can automatically generate such models from data. Historically, symbolic regression has been predominantly realized by genetic programming, a method that evolves populations of candidate solutions that are subsequently modified by genetic operators crossover and mutation. However, the approach suffers from several deficiencies: it does not scale well with the number of variables and samples in the training data – models tend to grow in size and complexity without an adequate accuracy gain, and it is hard to fine-tune the inner model coefficients using just genetic operators. Recently, neural networks have been applied to learn the whole analytic model, i.e., its structure as well as the coefficients, by means of gradient-based optimization algorithms. In this paper, we propose a novel neural network-based symbolic regression method that constructs physically plausible models based on very small training data sets and prior knowledge about the system. The method employs an adaptive weighting scheme to effectively deal with multiple loss function terms and an epoch-wise learning process to reduce the chance of getting stuck in poor local optima. Furthermore, we propose a parameter-free method for choosing the model with the best interpolation and extrapolation performance out of all models generated throughout the whole learning process. We experimentally evaluate the approach on four test systems: the TurtleBot 2 mobile robot, the magnetic manipulation system, the equivalent resistance of two resistors in parallel, and the longitudinal force of the anti-lock braking system. The results clearly show the potential of the method to find parsimonious models that comply with the prior knowledge provided.

Clipping-Based Post Training 8-Bit Quantization of Convolution Neural Networks for Object Detection

Fueled by the development of deep neural networks, breakthroughs have been achieved in plenty of computer vision problems, such as image classification, segmentation, and object detection. These models usually have handers and millions of parameters, which makes them both computational and memory expensive. Motivated by this, this paper proposes a post-training quantization method based on the clipping operation for neural network compression. By quantizing parameters of a model to 8-bit using our proposed methods, its memory consumption is reduced, its computational speed is increased, and its performance is maintained. This method exploits the clipping operation during training so that it saves a large computational cost during quantization. After training, this method quantizes the parameters to 8-bit based on the clipping value. In addition, a fully connected layer compression is conducted using singular value decomposition (SVD), and a novel loss function term is leveraged to further diminish the performance drop caused by quantization. The proposed method is validated on two widely used models, Yolo V3 and Faster R-CNN, for object detection on the PASCAL VOC, COCO, and ImageNet datasets. Performances show it effectively reduces the storage consumption at 18.84% and accelerates the model at 381%, meanwhile avoiding the performance drop (drop < 0.02% in VOC).

Solving PDEs by variational physics-informed neural networks: an a posteriori error analysis

We consider the discretization of elliptic boundary-value problems by variational physics-informed neural networks (VPINNs), in which test functions are continuous, piecewise linear functions on a triangulation of the domain. We define an a posteriori error estimator, made of a residual-type term, a loss-function term, and data oscillation terms. We prove that the estimator is both reliable and efficient in controlling the energy norm of the error between the exact and VPINN solutions. Numerical results are in excellent agreement with the theoretical predictions.

Deep-green inversion to extract traction-separation relations at material interfaces

The traction-separation relationship of an interface is a critical component to understand and model the delamination behavior of multi-layer composites in situations where large scale bridging is active. Limited by current experimental techniques, the extraction of traction-separation relationships often relies on inverse approaches, where far field measurements are used as input data. In this article, a data-driven model based on Green’s function embedded neural networks is proposed (namely Deep Green Inversion, or DGI), where the input consists of far field displacement fields while the output is the desired but initially unknown tractions and separations on the interface. Specifically, Green’s functions are embedded as loss function terms along with other terms based on mean squared error and the field equations associated with loaded elastic bodies. The approach is first verified via analytical solutions to a simply supported beam subject to end moments and a non-uniform traction applied to a portion of one boundary of the beam. Hyperparameter training is included in order to elucidate the influence of weight factors that are applied to the different constraints that are considered. The developed approach is then successfully validated for mode-I and mixed-mode cohesive zone extraction problems using only far-field displacement synthetic data that are generated from numerical solutions to the problems. As part of the validation process and consideration of any limitations of the approach, the amount of displacement data required to produce robust traction-separation relations is deliberated. The tractions and separations within the cohesive zone are extracted with an average global error of 9.06 % and local error of 9.24%, using only 55–60% of the available experimental data. The proposed approach is then applied to double cantilever beam experiments with six different types of interaction between the beams, where the input displacement fields are obtained using digital image correlation. The traction-separation relationships extracted via the DGI neural network developed here agree very well with the results obtained via a direct extraction approach. These results suggest that the proposed approach is quite general, considering the range of traction-separation relations that were explored.

Improving neural topic modeling via Sinkhorn divergence

Textual data have been a major form to convey internet users’ content. How to effectively and efficiently discover latent topics among them has essential theoretical and practical value. Recently, neural topic models(NTMs), especially Variational Auto-encoder-based NTMs, proved to be a successful approach for mining meaningful and interpretable topics. However, they usually suffer from two major issues:(1)Posterior collapse: KL divergence will rapidly reach zeros resulting in low-quality representation in latent distribution; (2)Unconstrained topic generative models: Topic generative models are always unconstrained, which potentially leads to discovering redundant topics. To address these issues, we propose Autoencoding Sinkhorn Topic Model based on Sinkhorn Auto-encoder(SAE) and Sinkhorn divergence. SAE utilizes Sinkhorn divergence rather than problematic KL divergence to optimize the difference between posterior and prior, which is free of posterior collapse. Then, to reduce topic redundancy, Sinkhorn Topic Diversity Regularization(STDR) is presented. STDR leverages the proposed Salient Topic Layer and Sinkhorn divergence for measuring distance between salient topic features and serves as a penalty term in loss function facilitating discovering diversified topics in training. Several experiments have been conducted on 2 popular datasets to verify our contribution. Experiment results demonstrate the effectiveness of the proposed model.

Learning time-dependent PDEs with a linear and nonlinear separate convolutional neural network

Partial differential equations (PDEs) that describe many physical phenomena are discovered or derived based on professional knowledge or empirical observations. However, the rapid development of machine learning technology interests us to discover the PDEs with a data-driven approach based on neural networks. We introduce the convolution neural network (CNN) to learn the time-dependent PDEs. Instead of customizing a complex regularization term in loss function, we introduce a series of constraints on the structure of neural network model to avoid the overfitting problems. Firstly, since the linear PDEs can be accurately approximated by a linear CNN, we construct a network consisting of a linear CNN and a nonlinear CNN. The linear CNN helps to mitigate the overfitting problems of learning nonlinear PDEs. Secondly, the boundary conditions are hard encoded into the neural networks by custom padding operations. Finally, the time-series data are learned by an auto-regressive framework corresponding with the Euler scheme. We test the proposed framework with a series of PDEs, including heat equation, Burgers equation, reaction-diffusion equation and Kuramoto–Sivashinsky equation. Moreover, we also test the two-dimensional equations, and it shows similar accuracy as one-dimensional case. These numerical results demonstrate that the proposed framework is able to learn the PDEs from few data more accurately and stably.

Low-Dose CT Image Denoising with Improving WGAN and Hybrid Loss Function.

The X-ray radiation from computed tomography (CT) brought us the potential risk. Simply decreasing the dose makes the CT images noisy and diagnostic performance compromised. Here, we develop a novel denoising low-dose CT image method. Our framework is based on an improved generative adversarial network coupling with the hybrid loss function, including the adversarial loss, perceptual loss, sharpness loss, and structural similarity loss. Among the loss function terms, perceptual loss and structural similarity loss are made use of to preserve textural details, and sharpness loss can make reconstruction images clear. The adversarial loss can sharp the boundary regions. The results of experiments show the proposed method can effectively remove noise and artifacts better than the state-of-the-art methods in the aspects of the visual effect, the quantitative measurements, and the texture details.

Foveal blur-boosted segmentation of nuclei in histopathology images with shape prior knowledge and probability map constraints.

In most tissue-based biomedical research, the lack of sufficient pathology training images with well-annotated ground truth inevitably limits the performance of deep learning systems. In this study, we propose a convolutional neural network with foveal blur enriching datasets with multiple local nuclei regions of interest derived from original pathology images. We further propose a human-knowledge boosted deep learning system by inclusion to the convolutional neural network new loss function terms capturing shape prior knowledge and imposing smoothness constraints on the predicted probability maps. Our proposed system outperforms all state-of-the-art deep learning and non-deep learning methods by Jaccard coefficient, Dice coefficient, Accuracy and Panoptic Quality in three independent datasets. The high segmentation accuracy and execution speed suggest its promising potential for automating histopathology nuclei segmentation in biomedical research and clinical settings. The codes, the documentation and example data are available on an open source at: https://github.com/HongyiDuanmu26/FovealBoosted. Supplementary data are available at Bioinformatics online.

EAR-UNet: A deep learning-based approach for segmentation of tympanic membranes from otoscopic images

This paper presents a method for automatic segmentation of tympanic membranes (TMs) from video-otoscopic images based on deep fully convolutional neural network. Built upon the UNet architecture, the proposed EAR scheme is based on three main paradigms: EfficientNet for the encoder, Attention gate for the skip connection path, and Residual blocks for the decoder. The paper also introduces a new loss function term for the neural networks to perform segmentation tasks. Particularly, we propose to integrate EfficientNet-B4 into the encoder part of the UNet. In addition, the decoder part of the proposed network is constructed based on residual blocks from ResNet architecture. By this way, the proposed approach could take advantages of the EfficientNet and ResNet architectures such as preserving efficient reception field size for the model and avoiding overfitting problem. In addition, in the skip connection path, we employ the attention gate that can handle the varieties in shapes and sizes of interested objects, which are common issues in TM regions. Moreover, for network training, we proposed a new loss function term based on the shape distance between predicted and ground truth masks, and exploited the stochastic weight averaging to avoid being trapped in local minima. We evaluate the proposed approach on a TM dataset which includes 1012 otoscopic images from patients diagnosed with and without otitis media. Experimental results show that the proposed approach achieves high segmentation performance with the average Dice similarity coefficient of 0.929, without any pre- or post-processing steps, that outperforms other state-of-the-art methods.

Sdgan: Improve Speech Enhancement Quality by Information Filter

The speech denoising model based on adversarial generative network has achieved better results than the traditional machine learning model. In this paper, for the short cut connection in the generator, we discuss its influence on the information transfer between encoder and decoder, and propose SDGAN at target. SDGAN sets linear and convolution filters in the short cut connection which adaptively learn the optimal information processing. The information filter still enables the generator to solve the gradient vanishing problem, and it can also avoid information redundancy and improve expression ability. In addition, SDGAN replaces the L1 regularization term in loss function with the L2 regularization term, which not only makes the output speech of the generator closer to the clean speech, but also avoids sparsity. In the experiments, SDGAN significantly performs better than other traditional GAN in five performance metrics (such as PESQ), and the effect of convolution filter is better than that of linear filter.

An Effective and Comprehensive Image Super Resolution Algorithm Combined With a Novel Convolutional Neural Network and Wavelet Transform

In order to further improve the reconstruction effect of the image super resolution algorithm, this paper proposes an image super resolution algorithm combining deep learning and wavelet transform (ISRDW). In terms of network design, it is not only simple in structure, but also more effective in capturing image details compared with other neural network structures. At the same time, cross-connection and residual learning methods are used to reduce the difficulty of the training model. In terms of loss function, this paper uses the loss generated in the original image space domain and the wavelet domain to strengthen the constraint of network training. Experimental results show that the algorithm proposed in this paper achieves better results under different data sets and different evaluation indexes.

Theoretical study on the electronic structures and electromagnetic wave absorption properties of Fe3Si/PPy composites

Electronic structures and electromagnetic wave absorption properties of Fe3Si/polypyrrole (PPy) composites are investigated by using first principles methods based on plane–wave pseudopotential theory. The calculation of electric structures shows that the Fe3Si/PPy composite presents typical metal properties. The real and imaginary parts of the dielectric constant of Fe3Si/PPy are larger than those of Fe3Si. The maximum absorption coefficient of Fe3Si is larger than that of Fe3Si/PPy, and the maximum absorption coefficients of both materials all increase first and then decrease. However, the variation range of the absorption coefficient of Fe3Si/PPy is wider than that of Fe3Si. The reflection coefficient and maximum extinction coefficient of Fe3Si/PPy are larger than those of Fe3Si. The maximum reflection coefficient of Fe3Si/PPy is greater than that of Fe3Si, and the maximum reflection coefficients of Fe3Si/PPy and Fe3Si are 0.81 and 0.91, respectively. In terms of loss function and refractive index, Fe3Si/PPy has more practical value than Fe3Si.

Super-resolution and denoising of 4D-Flow MRI using physics-Informed deep neural nets

Background and Objective: Time resolved three-dimensional phase contrast magnetic resonance imaging (4D-Flow MRI) has been used to non-invasively measure blood velocities in the human vascular system. However, issues such as low spatio-temporal resolution, acquisition noise, velocity aliasing, and phase-offset artifacts have hampered its clinical application. In this research, we developed a purely data-driven method for super-resolution and denoising of 4D-Flow MRI.Methods: The flow velocities, pressure, and the MRI image magnitude are modeled as a patient-specific deep neural net (DNN). For training, 4D-Flow MRI images in the complex Cartesian space are used to impose data-fidelity. Physics of fluid flow is imposed through regularization. Creative loss function terms have been introduced to handle noise and super-resolution. The trained patient-specific DNN can be sampled to generate noise-free high-resolution flow images. The proposed method has been implemented using the TensorFlow DNN library and tested on numerical phantoms and validated in-vitro using high-resolution particle image velocitmetry (PIV) and 4D-Flow MRI experiments on transparent models subjected to pulsatile flow conditions.Results: In case of numerical phantoms, we were able to increase spatial resolution by a factor of 100 and temporal resolution by a factor of 5 compared to the simulated 4D-Flow MRI. There is an order of magnitude reduction of velocity normalized root mean square error (vNRMSE). In case of the in-vitro validation tests with PIV as reference, there is similar improvement in spatio-temporal resolution. Although the vNRMSE is reduced by 50%, the method is unable to negate a systematic bias with respect to the reference PIV that is introduced by the 4D-Flow MRI measurement.Conclusions: This work has demonstrated the feasibility of using the readily available machinery of deep learning to enhance 4D-Flow MRI using a purely data-driven method. Unlike current state-of-the-art methods, the proposed method is agnostic to geometry and boundary conditions and therefore eliminates the need for tedious tasks such as accurate image segmentation for geometry, image registration, and estimation of boundary flow conditions. Arbitrary regions of interest can be selected for processing. This work will lead to user-friendly analysis tools that will enable quantitative hemodynamic analysis of vascular diseases in a clinical setting.