Embedding Loss Research Articles

MRDPDA: A multi-Laplacian regularized deepFM model for predicting piRNA-disease associations.

PIWI-interacting RNAs (piRNAs) are a typical class of small non-coding RNAs, which are essential for gene regulation, genome stability and so on. Accumulating studies have revealed that piRNAs have significant potential as biomarkers and therapeutic targets for a variety of diseases. However current computational methods face the challenge in effectively capturing piRNA-disease associations (PDAs) from limited data. In this study, we propose a novel method, MRDPDA, for predicting PDAs based on limited data from multiple sources. Specifically, MRDPDA integrates a deep factorization machine (deepFM) model with regularizations derived from multiple yet limited datasets, utilizing separate Laplacians instead of a simple average similarity network. Moreover, a unified objective function to combine embedding loss about similarities is proposed to ensure that the embedding is suitable for the prediction task. In addition, a balanced benchmark dataset based on piRPheno is constructed and a deep autoencoder is applied for creating reliable negative set from the unlabeled dataset. Compared with three latest methods, MRDPDA achieves the best performance on the pirpheno dataset in terms of the five-fold cross validation test and independent test set, and case studies further demonstrate the effectiveness of MRDPDA.

Analysis of the effectiveness of two different loss functions in training a neural network in lung image reconstruction using impedance tomography

The article presents research findings on developing a medical diagnostic system based on electrical impedance tomography technology. One of the key components of this project is developing a method for reconstructing the structure of human lungs using this technology. The authors of the article compared the effectiveness of two different loss functions in training a neural network, which is tasked with accurately replicating the lung structure based on electrical impedance tomography data. The researchers analyzed various approaches to calculating loss functions, including cosine embedding loss and InfoNCE loss. They compared the results obtained using these two functions to identify which one performs better in lung structure reconstruction. The findings of these studies may have significant implications for the development of diagnostic systems based on electrical impedance tomography and for improving the effectiveness of lung disease diagnosis. Additionally, the authors discuss potential future directions for the project, including possible applications of the research findings in clinical practice. Future research efforts may focus on optimizing neural network parameters, exploring alternative loss functions, or utilizing advanced machine learning techniques for even more precise lung structure reconstruction. The pursuit of improving such diagnostic systems could lead to significant advancements in the field of medicine, particularly in diagnosing and treating respiratory diseases.

Effect of padeye offset ratio on keying behaviours and capacity of square plate anchors in clay

This paper reports the effect of padeye offset ratio on the keying behaviours and capacity of square plate anchors in clay. The investigation is conducted through three-dimensional large deformation finite element analyses using the Remeshing and Interpolation Technique with Small Strain in ABAQUS. Strain rate dependency of the undrained shear strength and soil remoulding are accounted for adopting an extended Tresca model. The results from LDFE analyses are compared with existing analytical solutions, centrifuge test data, and numerical results; with reasonable agreement obtained. Parametric analyses are then carried out varying several critical factors including initial embedment depth, load eccentricity, padeye offset ratio, soil sensitivity, and soil strength heterogeneity. It is seen that the padeye offset has significant influence on the ultimate loss of embedment and capacity under vertical loading. To assess the keying behaviour of square plate anchors in clay, an equivalent soil strength incorporating strain rate dependency and remoulding is adopted. A combined factor defined as a function of the initial embedment depth and equivalent soil strength is used to estimate the ultimate loss of embedment. An equivalent ultimate bearing capacity factor is shown to be a function of the initial embedment depth, padeye offset ratio, and load eccentricity.

GrassCaré: Visualizing the Grassmannian on the Poincaré Disk

This paper introduces a novel method for visualizing high-dimensional Grassmannians through 2D embeddings on the Poincaré disk. The proposed approach involves the construction of affinity matrices on each manifold, followed by the minimization of KL-divergence between the geodesics affinity. This process enables the identification of an optimal projection that effectively preserves the geometry of the original high-dimensional Grassmannian. Our main theoretical contribution lies in bounding the embedding loss with respect to two factors: the logarithm of the number of subspaces and a term dependent on the distribution of subspaces within the Grassmannian. Notably, this term is smaller when subspaces exhibit well-defined clusters, and larger in the absence of any discernible structure. We complement our theoretical analysis with comprehensive experiments on both synthetic and real datasets. The experimental results showcase the superiority of our embedding in accurately visualizing Grassmannians compared to existing representations.

Semi-supervised class incremental broad network for continuous diagnosis of rotating machinery faults with limited labeled samples

Machine learning has demonstrated remarkable success in the field of intelligent fault diagnosis. In current machine learning systems, models tend to be fixed after training, which makes them can only generalize to classes that appear in the training set, and cannot continuously learn newly emerging classes. However, in real industrial scenarios, industrial data is constantly growing, which requires models to be able to acquire new knowledge while retaining existing knowledge. Furthermore, a substantial proportion of the acquired samples in industry are unlabeled and only few samples are labeled. To tackle the above two challenges, this article proposes a novel semi-supervised class incremental broad network (SSCIBN) for incremental intelligent diagnosis of rotating machinery faults under limited labeled samples. Specifically, a semi-supervised graph embedding loss function is designed. On the one hand, this loss function can learn the structural information of a large amount of unlabeled data, which overcomes the limitation that the scarcity of labeled samples leads to poor model performance. On the other hand, local structure information within old and new classes is considered in the continuous learning process to fully learn the unique manifold structure information of different classes, which in turn enhances the discriminative performance between old and new classes. Furthermore, a novel semi-supervised class incremental learning mechanism is proposed, which does not need to utilize the old class data during the incremental learning process, but can effectively retain the old class knowledge. The effectiveness of the proposed method is evaluated through multiple mechanical failure increment cases. Experimental analysis demonstrates that the designed SSCIBN has significant advantages in the class incremental application with the lack of labeled samples.

Effects of soil remoulding on keying behaviours of strip plate anchors with padeye offset

This paper reports the effect of soil remoulding on embedment loss and pullout capacity of strip plate anchors with padeye offset in normally consolidated clay during keying through large deformation finite element (LDFE) analysis. The effects of strain rate and soil remoulding are considered during keying process of plate anchors. First, the LDFE results are compared with theoretical solutions, centrifuge test results, and LDFE data available in the literature, which shows a good agreement. Then, the influences of initial embedment depth, load eccentricity, padeye offset and soil sensitivity on the embedment loss and pullout capacity of strip plate anchors during keying are investigated. The LDFE results show that the padeye offset and the soil remoulding have significant effects on the ultimate embedment loss and pullout capacity. To assess the keying behaviour of plate anchors, an equivalent soil strength is introduced. A combined factor and an equivalent ultimate pullout capacity are adopted to estimate the embedment loss and ultimate pullout capacity of strip plate anchors in normally consolidated clay, respectively.

Numerical investigation of the keying process of a new type of expandable plate anchor

High efficient anchors are increasing demanded to moor floating structures with oil and gas and renewable energy developments move into deeper water. One effective way to increase efficiency is by maximizing the bearing area of the anchor during operation. Along this line of design, a new high efficiency plate anchor composed of two pieces of fluke plates, which are expandable after installation under preloading, is proposed in this paper. The keying process plays a significant role in the anchor's bearing performance. To gain insights into the keying process and optimize the geometry design of this anchor, a series of Large Deformation Finite Element (LDFE) analyses are carried out. From the comprehensive LDFE analysis, this proposed anchor with optimal padeye eccentricity has lower loss of embedment compared with traditional upright anchor and the accompanying higher holding capacity. Besides, the newly proposed expandable anchor also has a much stiffer response, the anchor can bear 3 times the keying load (proof loading) without losing much embedment during operation stage. This has an important implication for saving offshore cost, where the proposed anchor could be keyed using small vessel with limited winch capacity.

Face identity and expression consistency for game character face swapping

Customizing the appearance of game characters according to individual preferences is an important application in the gaming industry. The traditional solutions such as manual editing within the game engine require much time and some professional experience. To address this problem, this paper proposes a novel face-swapping framework to swap real faces onto game characters while maintaining the art style of the game. This approach helps players speed up the character customization process, as they can input any face and quickly see the results of swapping it onto the character. Specifically, our framework extracts a more robust identity embedding from compound face identity models. Then the identity embedding is sent into AdaIN for feature fusion and changes the identity attribute for the output of the decoder. To maintain the expression consistency between the swapped and target faces, we utilize a novel expression embedding loss which effectively constrains the fine-grain expression similarity. To reduce the cross-domain gap between human and game faces, we also construct a game face dataset and propose to use fine-tuning to improve the image quality of cross-domain face swapping. Extensive qualitative and quantitative experiments indicate that our method achieves leading swapping results in both pure natural human faces and game faces datasets.

Numerical modelling of the keying process for a suction embedded plate anchor in spatially varying clays

ABSTRACT As the exploration of oil and gas moves into deeper water, suction embedded plate anchors (SEPLAs) are developed to answer the growing demand for anchor-mooring systems. The keying process of SEPLAs has been extensively reported assuming a spatially homogeneous soil which is a departure from reality. This study conducts a comprehensive investigation of the keying process of SEPLA in spatially varying clay through a large deformation random finite element method coupled with the Monte Carlo simulations. The numerical model is validated with the results from the existing centrifuge test and numerical analysis. The spatial randomness of soil strength is assumed to be a three-dimensional (3D) random field to explore the failure mechanism and statistical characteristics of rotational response and pullout behaviour. The normalised factor of the upper 5% fractile of embedment loss is presented as an algebraic expression of the coefficient of variation of soil strength. The relationship between safety factor and failure probability is determined based on the requirement of pullout capacity. Finally, the applicability of the present results to different soil properties is verified.

Numerical study on the performance of a novel gravity installed anchor with ring fins in clay

A novel gravity installed anchor with ring fins (RGIA) is proposed in the present work, which combines features of the conventional gravity installed anchor (GIA) and the drag embedment plate anchor, with two soft shanks attached to the anchor body. A systematic study on the performance of the RGIA in clay is performed by large deformation finite element analyses using the coupled Eulerian-Lagrangian technique (CEL), with emphasis on the initial penetration, the behaviors of keying, diving and pulling out, the optimization of the shackle point, and the effect of loading rates on the maximum embedment loss (MEL). The initial penetration of the anchor is simulated under various impact velocities, from which the maximum initial embedment depth in the seabed is acquired. Two factors of the shackle point, en and eh, are defined and the detailed criteria for optimizing the shackle point are put forward. A series of analytical cases are designed to thoroughly explore the effects of the shackle point on the performance of the RGIA in clay, focusing on the keying and diving behaviors and the pullout capacity. The formulas for calculating the diving angle and the MEL are then constructed with high precision, by taking into account the effect of the shackle point, and can be utilized to optimize the position of the shackle point. Comprehensive drag effects under multiple factors, especially the high loading rate, on the performance of the RGIA in clay are quantitatively investigated by performing six groups of cases, from which the effects of drag velocity, initial embedment depth, gradient of shear strength of clay, and cable parameters can be known. The formula for calculating the MEL of the RGIA in clay is also constructed with high precision by taking into account comprehensive drag effects. The numerical study demonstrates that, compared with the OMNI-Max anchor, the RGIA achieves deeper initial penetration depth during installation, requires shallower initial penetration depth that ensures subsequent keying and diving, and possesses stronger ability of resisting dynamic loading with high loading rates.

Learning Structure Aware Deep Spectral Embedding.

Spectral Embedding (SE) has often been used to map data points from non-linear manifolds to linear subspaces for the purpose of classification and clustering. Despite significant advantages, the subspace structure of data in the original space is not preserved in the embedding space. To address this issue subspace clustering has been proposed by replacing the SE graph affinity with a self-expression matrix. It works well if the data lies in a union of linear subspaces however, the performance may degrade in real-world applications where data often spans non-linear manifolds. To address this problem we propose a novel structure-aware deep spectral embedding by combining a spectral embedding loss and a structure preservation loss. To this end, a deep neural network architecture is proposed that simultaneously encodes both types of information and aims to generate structure-aware spectral embedding. The subspace structure of the input data is encoded by using attention-based self-expression learning. The proposed algorithm is evaluated on six publicly available real-world datasets. The results demonstrate the excellent clustering performance of the proposed algorithm compared to the existing state-of-the-art methods. The proposed algorithm has also exhibited better generalization to unseen data points and it is scalable to larger datasets without requiring significant computational resources.

Discrimination Neural Network Model for Binary Classification Tasks on Tabular Data

For the classification task, neural network-based approaches attempt to distinguish between two distributions by determining the joint distribution of input variables for each class. However, the most challenging task is still to classify the observations in the overlapping region of two classes. In this work, we propose a new discrimination neural network (DiscNN) architecture to address this issue. Our DiscNN learns to embed the initial input into more informative representations with better discriminability between the two distributions based on the cosine embedding loss. We also train our proposed model using the few-shot learning method to extract better-generalized representations from the initial input. We applied the DiscNN model to 35 tabular datasets from the OpenML-CC18 benchmark for a binary classification task. Our model showed superior performances on 28 datasets of them. In addition, we also performed experiments on 95 imbalanced datasets from the KEEL repository. The experiment results showed that the DiscNN outperformed the state-of-the-art models, including CatBoost, LightGBM, TabNet, VIME and Scarf, by around 0.23% AUC, 0.20% G-mean, and 1.06% F1 score.

A Joint Network Based CNN for Yoga Pose Classification and Scoring

Comparing to traditional rehabilitation, rehabilitation at home becomes a need during pandemic. The technique brought up in this paper allows patients and yoga fans exercise at home with low cost and comfort while can also evaluate their postures. Previous works focus either on classifying poses or scoring on the sameness between the two input branches of patients’ poses and normative poses, but they ignore the combination of them in one single network. In this study, a residual block based Siamese CNN network with classification and scoring modules is proposed, aiming at providing accurate pose matching scores and classify pose types on yoga postures simultaneously. The Siamese network takes two inputs of learner’s pose and standard pose, which are preprocessed skeleton images by OpenPose. With the addition of residual block on the first convolutional module, back propagation is facilitated, which boosts up the process of updating parameters and optimization. The model calculates total loss by summing up cosine embedding loss and cross entropy loss in which the weight parameter lambda could be modified based on need. As for the scoring module, cosine similarity is used to calculate pose resemblance on batch level. The improvement in model performance is obvious when comparing the loss and accuracy between the Siamese network with residual block and VGG-16. Experimental results indicate that the residual block based Siamese network achieves competitive performance compared to the VGG-16 and can provide scoring feedback to learner’s yoga poses.

The failure induced by embedment loss of gravity installed anchors in clay

The reliability of mooring systems is a critical issue for deepwater floating structures. Gravity installed anchors (GIAs) have been effectively employed in a significant number of mooring systems, whose embedment loss and kinematic behaviors under the cable loading conditions also attract attention to the reliability of the anchors during service. Particularly, the embedment loss of the anchor may result in the failure of the anchor. The present work focuses on systematically investigating the failure state of anchors in clay induced by embedment loss under various complex conditions. A theoretical model for analyzing the comprehensive behaviors of the anchor is constructed, which is validated by previous centrifuge tests and 1g model tests. A rational criterion for determining the state of failure induced by embedment loss is proposed and applied to the analytical results of the theoretical model. The factors that have a substantial impact on embedment loss of the anchors, including the anchor embedment depth, anchor orientation, cable loading angle at the seafloor, soil adhesion factor, soil strength, and cable properties are investigated in the analytical cases. OMNI-Max anchors, being one of the most remarkable GIAs and having complex kinematic behaviors, are selected to reveal the influences of these factors on the failure state of the anchors. Based on the results of numerous analytical cases, the failure surfaces in the three-dimensional spatial data field are presented, which reveal the critical state of the anchor under different conditions. Furthermore, these failure surfaces are stated using a surface function. The function can be used as a simple judgment condition for the state of failure owing to anchor embedment loss. The findings of earlier investigations support the applicability of the surface function. In addition, the promising application of the surface function in practice is highlighted.

Speeding up cardiac MR segmentation with semi-supervision: applications in cine imaging

Abstract Funding Acknowledgements Type of funding sources: Private company. Main funding source(s): This material is based upon work supported by the Google Cloud Research Credits program with the award GCP19980904. Background Segmentation is an important postprocessing step in cardiac magnetic resonance (CMR) imaging that enables quantitative assessment of functional parameters. Deep learning can automate the segmentation process, producing accurate contours of cardiac structures while reducing the time required to analyze images and the interobserver variation compared to manual analysis. However, common approaches to training neural networks (NNs) require large amounts of labeled data, which is costly to generate and slows down the development of CMR segmentation NNs for new applications. Semi-supervision is an approach to alleviate this labeling burden by relying on abundant unlabeled data and a smaller amount of labeled data to train NNs. Purpose We propose a novel semi-supervised method to train CMR segmentation NNs. We use the proposed method to train NNs to segment the left ventricle in CMR cine images. Ultimately, we aim to show that semi-supervision can drastically reduce the amount of labeled data required to develop machine learning segmentation applications for CMR while maintaining high performance. Methods Our dataset consists of 1,208 short-axis cine CMR images and 1,244 long-axis cine CMR images. An expert annotator manually segmented the endocardium on the end-diastolic and end-systolic short-axis and long-axis images and the epicardium on the end-diastolic short-axis images. We split the dataset randomly by patient into 60% training, 20% validation, and 20% testing data. We train semi-supervised segmentation networks using a supervised cross-entropy loss to learn from the labeled training data and a cosine embedding loss in addition to a pseudo-labeling step to learn from the unlabeled training data. To evaluate how performance changes with different amounts of labeled training data, we vary the percent of training data that has labels from <1%-100%. We evaluate the predicted segmentation masks using the Dice coefficient. Results Using only 100 labeled image slices, the semi-supervised segmentation NNs achieve a mean Dice coefficient within 1.10% of networks trained with fully labeled training sets, corresponding to >85% reduction in required labeled training data (Table 1). The proposed semi-supervised method improves performance over naïve training by 6.21% for the most limited labeled data setting (i.e., 10 labeled image slices; Figure 1). Conclusion We have shown that NNs trained with limited labeled data achieve high performance on left ventricle segmentation in short-axis and long-axis CMR cines. The proposed approach is flexible and broadly applicable to different CMR segmentation tasks, enabling rapid development of segmentation networks for many cardiac structures and applications. Table 1Figure 1

The diving and keying capacities of OMNI-Max anchors in clay

Different to the conventional gravity installed anchors (GIAs), the OMNI-Max anchor is capable of deriving higher capacity through the diving behavior (obtaining deeper embedment depth) after the keying process (usually accompanied by the loss of embedment depth) subjected to excessive loading, due to its three-dimensional structure and the various way to bear applied loading. The capacity properties of the anchor during keying and diving not only determine the keying and diving behaviors of the anchor, but also influence the other anchor behaviors, such as the embedment loss, pulling out and kinematic trajectory. Existing knowledge of the diving and keying behaviors is rather rare, stimulating further exploration of diving and keying capacities for OMNI-Max anchors. A systematic and quantitative study is performed on the complex capacities of OMNI-Max anchors during diving and keying in clay under multiple factors, including the anchor embedment depth, the anchor orientation, the interface friction, the loading direction, the bearing area of the anchor, the rotational center of the anchor, and the property of soil. Numerous cases of large deformation finite element (LDFE) analyses are conducted by employing the technique of coupled Eulerian-Lagrangian (CEL), through which the impacts of different factors can be analyzed. The position of the rotational center affects most the moment capacity factor among all the factors, while the anchor orientation has only a tiny influence on the moment capacity factor. The minimum value of the moment capacity factor results from the rotational center located at approximately one-third to half of the shaft length away from the anchor rear. The end bearing during diving when the movement is parallel to the shaft, is about one-quarter to one-third of that during pulling out when the movement is perpendicular to the shaft. Explicit expressions of bearing capacity factors composed of multiple factors are eventually formulated, which are validated by orthogonal tests and published data, and can be used conveniently not only to quantify the effects of different factors but also to evaluate the diving and keying capacities of OMNI-Max anchors.

Data Assimilation Network for Generalizable Person Re-Identification

In this paper, a data assimilation network is proposed to tackle the challenges of domain generalization for person re-identification (ReID). Most of the existing research efforts only focus on single-dataset issues, and the trained models are difficult to generalize to unseen scenarios. This paper presents a distinctive idea to improve the generality of the model by assimilating three types of images: style-variant images, misaligned images and unlabeled images. The latter two are often ignored in the previous domain generalization ReID studies. In this paper, a non-local convolutional block attention module is designed for assimilating the misaligned images, and an attention adversary network is introduced to correct it. A progressive augmented memory is designed for assimilating the unlabeled images by progressive learning. Moreover, we propose an attention adversary difference loss for attention correction, and a labeling-guide discriminative embedding loss for progressive learning. Rather than designing a specific feature extractor that is robust to style shift as in most previous domain generalization work, we propose a data assimilation meta-learning procedure to train the proposed network, so that it learns to assimilate style-variant images. It is worth mentioning that we add an unlabeled augmented dataset to the source domain to tackle the domain generalization ReID tasks. Extensive experiments demonstrate that our approach significantly outperforms the state-of-the-art domain generalization methods.

SPATE-GAN: Improved Generative Modeling of Dynamic Spatio-Temporal Patterns with an Autoregressive Embedding Loss

From ecology to atmospheric sciences, many academic disciplines deal with data characterized by intricate spatio-temporal complexities, the modeling of which often requires specialized approaches. Generative models of these data are of particular interest, as they enable a range of impactful downstream applications like simulation or creating synthetic training data. Recently, COT-GAN, a new GAN algorithm inspired by the theory of causal optimal transport (COT), was proposed in an attempt to improve generation of sequential data. However, the task of learning complex patterns over time and space requires additional knowledge of the specific data structures. In this study, we propose a novel loss objective combined with COT-GAN based on an autoregressive embedding to reinforce the learning of spatio-temporal dynamics. We devise SPATE (spatio-temporal association), a new metric measuring spatio-temporal autocorrelation. We compute SPATE for real and synthetic data samples and use it to compute an embedding loss that considers space-time interactions, nudging the GAN to learn outputs that are faithful to the observed dynamics. We test our new SPATE-GAN on a diverse set of spatio-temporal patterns: turbulent flows, log-Gaussian Cox processes and global weather data. We show that our novel embedding loss improves performance without any changes to the architecture of the GAN backbone, highlighting our model's increased capacity for capturing autoregressive structures.

Alleviating ASR Long-Tailed Problem by Decoupling the Learning of Representation and Classification

Recently, we have witnessed excellent improvement of end-to-end (E2E) automatic speech recognition (ASR). However, how to tackle the long-tailed data distribution problem while maintaining E2E ASR models' performance for high-frequency tokens is still challenging. To solve this challenge, we propose a novel decoupled ASR learning method for the sequence-to-sequence ASR architecture in this paper. Our method decouples the learning procedure of this model into two stages: representation learning and classification learning. In the representation learning stage, we use the encoder output of a pretrained language model as one of the ASR model’s learning targets, and propose threshold log cosine embedding loss (TLCE-loss) as the objective function. A frequency-mask cross-entropy loss (FMCE-loss) is also designed as an auxiliary loss. In the classification learning stage, we find that introducing a temperature into softmax function helps reduce the influence of negative samples on tail classes, thus mitigating the biased learning process for the classifier. Furthermore, we propose a weighted softmax (w-softmax) to adjust ASR posterior probabilities according to the token appearing frequency during inference. Additionally, we introduce tail word/character error rate (TWER / TCER) and head word/character error rate (HWER / HCER) that respectively evaluate the ASR accuracy for tail and head words/characters. Experimental results on the Switchboard and HKUST corpora show that our proposed method greatly outperforms the baseline, especially in TWER / TCER reduction. To the best of our knowledge, this is the first work to use a decoupled ASR learning method to alleviate the long-tailed problem in sequence-to-sequence ASR.

Cross-knowledge-graph entity alignment via relation prediction

The entity alignment task aims to align entities corresponding to the same object in different KGs. The recent work focuses on applying knowledge embedding or graph neural networks to obtain entity embedding for entity alignment. However, there are two challenges encountered by these models: one is some models need to design hyper-parameter to balance embedding loss and alignment loss, the other is the limited training data size. In this paper, we propose a novel entity alignment framework named RpAlign (Relation prediction based cross-knowledge-graph entity Alignment) to address these two issues. Specifically, RpAlign transforms the entity alignment task to the KG completion task to solve and does not need to design any extra alignment component. Unlike the existing models that predict aligned entities by using entity vector distance, the RpAlign defines a new relation called ‘anchor’ for aligned entities, and it predicts new aligned entities based on the relational predictions between the entities. RpAlign employs several data augmentation and improved self-training techniques to mitigate the impact of the data limitation. We conduct experiments on two datasets, and the experimental results show that the RpAlign model significantly outperforms the current state-of-the-art models.