Cross-entropy Training Research Articles

Transforming the bootstrap: using transformers to compute scattering amplitudes in planar N=4 super Yang–Mills theory

Abstract We pursue the use of deep learning methods to improve state-of-the-art computations in theoretical high-energy physics. Planar N = 4 Super Yang-Mills theory is a close cousin to the theory that describes Higgs boson production at the Large Hadron Collider; its scattering amplitudes are large mathematical expressions containing integer coefficients. In this paper, we apply Transformers to predict these coefficients. The problem can be formulated in a language-like representation amenable to standard cross-entropy training objectives. We design two related experiments and show that the model achieves high accuracy (> 98%) on both tasks. Our work shows that Transformers can be applied successfully to problems in theoretical physics that require exact solutions.

Enhancing Brain Stroke Detection: A Novel Deep Neural Network with Weighted Binary Cross Entropy Training

Enhancing Brain Stroke Detection: A Novel Deep Neural Network with Weighted Binary Cross Entropy Training

GraphECC: Enhancing GraphSMOTE with enhanced complementary classifier

Node classification in graph learning faces significant challenges due to imbalanced data, particularly for under-represented samples from minority classes. To address this issue, existing methods often rely on synthetic minority over-sampling techniques, introducing additional complexity during model training. In light of the challenges faced, we introduce GraphECC, an innovative approach that addresses numerical anomalies in large-scale datasets by supplanting the traditional CE loss function with an Enhanced Complementary Classifier (ECC) loss function’a novel modification to the CCE loss. This alteration ensures computational stability and mitigates potential numerical anomalies by incorporating a slight offset in the denominator during the computation of the complementary probability distribution. In this paper, we present a novel training paradigm, the Enhanced Complementary Classifier (ECC), which offers “imbalance defense for free” without the need for extra procedures to improve node classification accuracy.The ECC approach optimizes model probabilities for the ground-truth class, akin to the cross-entropy method. Additionally, it effectively neutralizes probabilities associated with incorrect classes through a “guided” term, achieving a balanced trade-off between the two aspects. Experimental results demonstrate that our proposed method not only enhances model robustness but also surpasses the widely used cross-entropy training objective.Moreover, we demonstrate the versatility of our method by seamlessly integrating it with various well-known adversarial training techniques, resulting in significant gains in robustness. Notably, our approach represents a breakthrough, as it enhances model robustness without compromising performance, distinguishing it from previous attempts.The code for GraphECC can be accessed from the following link:https://github.com/12chen20/GraphECC.

Transformers in High-Frequency Trading

Transformer is a deep learning model that, having an innovative performance in many tasks, has uniquely and significantly modified all the cast of mind of the AI scientific community. In this paper, we introduce a Transformer model that is applied to 1-minute timescale in the EURUSD and GBPUSD instruments of forex trading. We use the classical Transformer architecture without the Decoder since the Encoder is enough. Moreover, the Exponential Moving Average (EMA) model is applied to the input, while different values of its smoothing factor α are tested. With cross-entropy training loss less than 0.2, it is exclamatory that Transformers are a promising tool for lucrative strategies in high-frequency trading.

Curricular-balanced long-tailed learning

The real-world data distribution is essentially long-tailed, which poses a significant challenge to the deep model. Classification models minimizing cross-entropy loss struggle to classify the tail classes, although cross-entropy training is successful on balanced data. We reveal that minimizing cross-entropy loss under long-tailed distribution leads to the Tail Collapse phenomenon, which fundamentally limits the performance of neural networks. To correct the optimization behavior of cross-entropy training, we propose a new Curricular Balanced Loss (CurB Loss) to alleviate the imbalance. The CurB loss has two factors: the re-weighting factor and the curriculum learning factor. We design the re-weighting factor based on the margin-based training that can theoretically reach the optimums of networks. Then, we incorporate the idea of Curriculum Learning into the re-weighting loss in an adaptive manner. We design the curriculum learning factor to make the model gradually emphasize the hard classes. The empirical results demonstrate the complementary of the two factors. Our method outperforms previous state-of-the-art methods by 0.9%, 2.7%, 1.2% on CIFAR10-LT, CIFAR-100-LT and ImageNet-LT, demonstrating the effectiveness of CurB Loss for long-tailed visual recognition.

Multi-Loss U-Net Reformulation as an Efficient Solution to the Colony-Forming Unit Counting Problem.

U-Net is undoubtedly the most cited and popularized deep learning architecture in the biomedical domain. Starting with image, volume, or video segmentation in numerous practical applications, such as digital pathology, and continuing to Colony-Forming Unit (CFU) segmentation, new emerging areas require an additional U-Net reformulation to solve some inherent inefficiencies of a simple segmentation-tailored loss function, such as the Dice Similarity Coefficient, being applied at the training step. One of such areas is segmentation-driven CFU counting, where after receiving a segmentation output map one should count all distinct segmented regions belonging to different detected microbial colonies. This problem can be a challenging endeavor, as a pure segmentation objective tends to produce many irrelevant artifacts or flipped pixels. Our novel multi-loss U-Net reformulation proposes an efficient solution to the aforementioned problem. It introduces an additional loss term at the bottom-most U-Net level, which focuses on providing an auxiliary signal of where to look for distinct CFUs. In general, our experiments show that all probed multi-loss U-Net architectures consistently outperform their single-loss counterparts, which embark on the Dice Similarity Coefficient and Cross-Entropy training losses alone.

Deep Motion Prior for Weakly-Supervised Temporal Action Localization.

Weakly-Supervised Temporal Action Localization (WSTAL) aims to localize actions in untrimmed videos with only video-level labels. Currently, most state-of-the-art WSTAL methods follow a Multi-Instance Learning (MIL) pipeline: producing snippet-level predictions first and then aggregating to the video-level prediction. However, we argue that existing methods have overlooked two important drawbacks: 1) inadequate use of motion information and 2) the incompatibility of prevailing cross-entropy training loss. In this paper, we analyze that the motion cues behind the optical flow features are complementary informative. Inspired by this, we propose to build a context-dependent motion prior, termed as motionness. Specifically, a motion graph is introduced to model motionness based on the local motion carrier (e.g., optical flow). In addition, to highlight more informative video snippets, a motion-guided loss is proposed to modulate the network training conditioned on motionness scores. Extensive ablation studies confirm that motionness efficaciously models action-of-interest, and the motion-guided loss leads to more accurate results. Besides, our motion-guided loss is a plug-and-play loss function and is applicable with existing WSTAL methods. Without loss of generality, based on the standard MIL pipeline, our method achieves new state-of-the-art performance on three challenging benchmarks, including THUMOS'14, ActivityNet v1.2 and v1.3.

Cross-Entropy Training of DNN Ensemble Acoustic Models for Low-Resource ASR

Deep neural networks (DNNs) have shown a great promise in exploiting out-of-language data, particularly for under-resourced languages. The common trend is to merge data from various source languages to train a multilingual DNN and then reuse the hidden layers as language-independent feature extractors for a low-resource target language. While there is a consensus that using as much data from various languages results in a better and more general multilingual DNN, employing only source languages similar to the target language has proven effective. In this study, we propose a novel framework for multilingual DNN training, which employs all the available training data and exploits complementary information from individual source languages at the same time. Toward this goal, we borrow the idea of an ensemble with one generalist and many specialists . The generalist is derived from a multilingual DNN acoustic model trained on all available multilingual data; the specialists are the DNNs derived from the source languages individually. Then, the constituents in the ensemble are combined using weighted averaging schemes, where the combination weights are trained to minimize the cross-entropy objective function. In this framework, we seek for complementary information among the constituents while it is possible to get at least the performance equal to the baseline. Moreover, unlike previous well-known system combination schemes, only one model is required during decoding. We successfully examined two combination methodologies and demonstrated their usefulness in different scenarios using the multilingual GlobalPhone dataset. It is observed that, specifically, speech recognition systems developed in low-resource settings profit from the proposed strategy.

Image Captioning via Semantic Guidance Attention and Consensus Selection Strategy

Recently, a series of attempts have incorporated spatial attention mechanisms into the task of image captioning, which achieves a remarkable improvement in the quality of generative captions. However, the traditional spatial attention mechanism adopts latent and delayed semantic representations to decide which area should be paid more attention to, resulting in inaccurate semantic guidance and the introduction of redundant information. In order to optimize the spatial attention mechanism, we propose the Semantic Guidance Attention (SGA) mechanism in this article. Specifically, SGA utilizes semantic word representations to provide an intuitive semantic guidance that focuses accurately on semantic-related regions. Moreover, we reduce the difficulty of generating fluent sentences by updating the attention information in time. At the same time, the beam search algorithm is widely used to predict words during sequence generation. This algorithm generates a sentence according to the probabilities of words, so it is easy to push out a generic sentence and discard some distinctive captions. In order to overcome this limitation, we design the Consensus Selection (CS) strategy to choose the most descriptive and informative caption, which is selected by the semantic similarity of captions instead of the probabilities of words. The consensus caption is determined by selecting the one with the highest cumulative semantic similarity with respect to the reference captions. Our proposed model (SGA-CS) is validated on Flickr30k and MSCOCO, which shows that SGA-CS outperforms state-of-the-art approaches. To our best knowledge, SGA-CS is the first attempt to jointly produce semantic attention guidance and select descriptive captions for image captioning tasks, achieving one of the best performance ratings among any cross-entropy training methods.

A Continuous Relaxation of Beam Search for End-to-End Training of Neural Sequence Models

Beam search is a desirable choice of test-time decoding algorithm for neural sequence models because it potentially avoids search errors made by simpler greedy methods. However, typical cross entropy training procedures for these models do not directly consider the behaviour of the final decoding method. As a result, for cross-entropy trained models, beam decoding can sometimes yield reduced test performance when compared with greedy decoding. In order to train models that can more effectively make use of beam search, we propose a new training procedure that focuses on the final loss metric (e.g. Hamming loss) evaluated on the output of beam search. While well-defined, this "direct loss" objective is itself discontinuous and thus difficult to optimize. Hence, in our approach, we form a sub-differentiable surrogate objective by introducing a novel continuous approximation of the beam search decoding procedure.In experiments, we show that optimizing this new training objective yields substantially better results on two sequence tasks (Named Entity Recognition and CCG Supertagging) when compared with both cross entropy trained greedy decoding and cross entropy trained beam decoding baselines.

Nonrecurrent Neural Structure for Long-Term Dependence

In this paper, we propose a novel neural network structure, namely feedforward sequential memory networks FSMN, to model long-term dependence in time series without using recurrent feedback. The proposed FSMN is a standard fully connected feedforward neural network equipped with some learnable memory blocks in its hidden layers. The memory blocks use a tapped-delay line structure to encode the long context information into a fixed-size representation as short-term memory mechanism which are somehow similar to the time-delay neural networks layers. We have evaluated the FSMNs in several standard benchmark tasks, including speech recognition and language modeling. Experimental results have shown that FSMNs outperform the conventional recurrent neural networks RNN while can be learned much more reliably and faster in modeling sequential signals like speech or language. Moreover, we also propose a compact feedforward sequential memory networks cFSMN by combining FSMN with low-rank matrix factorization and make a slight modification to the encoding method used in FSMNs in order to further simplify the network architecture. On the speech recognition Switchboard task, the proposed cFSMN structures can reduce the model size by 60% and speed up the learning by more than seven times while the model can still significantly outperform the popular bidirectional LSTMs for both frame-level cross-entropy criterion-based training and MMI-based sequence training.

A Bayesian Framework for XML Information Retrieval: Searching and Learning with the INEX Collection

Most recent document standards like XML rely on structured representations. On the other hand, current information retrieval systems have been developed for flat document representations and cannot be easily extended to cope with more complex document types. The design of such systems is still an open problem. We present a new model for structured document retrieval which allows computing scores of document parts. This model is based on Bayesian networks whose conditional probabilities are learnt from a labelled collection of structured documents--which is composed of documents, queries and their associated assessments. Training these models is a complex machine learning task and is not standard. This is the focus of the paper: we propose here to train the structured Bayesian Network model using a cross-entropy training criterion. Results are presented on the INEX corpus of XML documents.