Embedding Layer Research Articles

Multi-Modal Vision Transformer with Explainable Shapley Additive Explanations Value Embedding for Cymbidium goeringii Quality Grading

Cymbidium goeringii (Rchb. f.) is a traditional Chinese flower with highly valued biological, cultural, and artistic properties. However, the valuation of Rchb. f. mainly relies on subjective judgment, lacking a standardized digital evaluation and grading methods. Traditional grading methods solely rely on unimodal data and are based on fuzzy grading standards; the key features for values are especially inexplicable. Accurately evaluating Rchb. f. quality through multi-modal algorithms and clarifying the impact mechanism of key features on Rchb. f. value is essential for providing scientific references for online orchid trading. A multi-modal Transformer for Rchb. f. quality grading combined with the Shapley Additive Explanations (SHAP) algorithm was proposed, which mainly includes one embedding layer, one UNet, one Vision Transformer (ViT) and one Encoder layer. A multi-modal orchid dataset including images and text was obtained from Orchid Trading Website, and seven key features were extracted. Based on petals’ RGB segmented from UNet and global fine-grained features extracted from ViT, text features and image features were organically fused into Transformer Encoders throughout concatenation operation, a 93.13% accuracy was achieved. Furthermore, SHAP algorithm was utilized to quantify and rank the importance of seven features, clarifying the impact mechanism of key features on Rchb. f. quality and value. This multi-modal Transformer with SHAP algorithm for Rchb. f. grading provided a novel idea to represent the explainable features accurately, exhibiting good potential for establishing a reliable digital evaluation method for agricultural products with high value.

Exploring protein natural diversity in environmental microbiomes with DeepMetagenome.

Protein natural diversity offers a vast sequence space for protein engineering, and deep learning enables its detection from metagenomes/proteomes without prior assumptions. DeepMetagenome, a Python-based method, explores protein diversity through modules for training and analyzing sequence datasets. The deep learning model includes Embedding, Conv1D, LSTM, and Dense layers, with sequence feature analysis for data cleaning. Applied to metallothioneins from a database of over 146 million coding features, DeepMetagenome identified over 500 high-confidence metallothionein sequences, outperforming DIAMOND and CNN-based models. It showed stable performance compared to a Transformer-based model over 25 epochs. Among 23 synthesized sequences, 20 exhibited metal resistance. The tool also successfully explored the diversity of three additional protein families and is freely available on GitHub with detailed instructions.

BCBA: An IIoT encrypted traffic classifier based on a serial network model

With the rapid development of the Industrial Internet of Things (IIoT), ensuring the security and privacy of network traffic has become particularly important. Classifying and identifying encrypted traffic is a critical step in enhancing network security, but traditional traffic classification methods often struggle to handle the complexities of the IIoT environment. In this paper, we propose the BCBA model, a bidirectional encoder representation from transformers (BERT)-based serial network model, which significantly improves the encrypted traffic classification performance and can be trained more than four times faster than the original BERT classifier. The BCBA method obtains word vector representations from the embedding layer of the pretrained BERT model, uses convolutional neural networks (CNNs) to extract local features, employs bidirectional long short-term memory (BiLSTM) networks to capture temporal dependencies in traffic data, and leverages a multihead self-attention mechanism to improve global dependency understanding. In experiments on six types of regular encrypted traffic from the ISCXVPN2016 dataset, the BCBA model achieved an F1 score of 99.10%, outperforming traditional traffic classification techniques across multiple performance metrics. This study demonstrates the effectiveness of deep learning in enhancing the security of the IIoT. It also provides new perspectives and technical routes for future research, particularly in encrypted traffic processing and classification applications.

CNN-Based Models for Emotion and Sentiment Analysis Using Speech Data

The study aims to present an in-depth Sentiment Analysis (SA) grounded by the presence of emotions in the speech signals. Nowadays, all kinds of web-based applications ranging from social media platforms and video-sharing sites to e-commerce applications provide support for Human–Computer Interfaces (HCIs). These media applications allow users to share their experiences in all forms such as text, audio, video, GIF, and so on. The most natural and fundamental form of expressing oneself is through speech. Speech-Based Sentiment Analysis (SBSA) is the task of gaining insights into speech signals. It aims to classify the statement as neutral, negative, or positive. On the other hand, Speech Emotion Recognition (SER) categorizes speech signals into the following emotions: disgust, fear, sadness, anger, happiness, and neutral. It is necessary to recognize the sentiments along with the profoundness of the emotions in the speech signals. To cater to the above idea, a methodology is proposed defining a text-oriented SA model using the combination of CNN and Bi-LSTM techniques along with an embedding layer, applied to the text obtained from speech signals; achieving an accuracy of 84.49%. Also, the proposed methodology suggests an Emotion Analysis (EA) model based on the CNN technique highlighting the type of emotion present in the speech signal with an accuracy measure of 95.12%. The presented architecture can also be applied to different other domains like product review systems, video recommendation systems, education, health, security, and so on.

Proton dose calculation with LSTM networks in presence of a magnetic field

Objective.To present a long short-term memory (LSTM) network-based dose calculation method for magnetic resonance (MR)-guided proton therapy.Approach.35 planning computed tomography (CT) images of prostate cancer patients were collected for Monte Carlo (MC) dose calculation under a perpendicular 1.5 T magnetic field. Proton pencil beams (PB) at three energies (150, 175, and 200 MeV) were simulated (7560 PBs at each energy). A 3D relative stopping power cuboid covering the extent of the PB dose was extracted and given as input to the LSTM model, yielding a 3D predicted PB dose. Three single-energy (SE) LSTM models were trained separately on the corresponding 150/175/200 MeV datasets and a multi-energy (ME) LSTM model with an energy embedding layer was trained on either the combined dataset with three energies or a continuous energy (CE) dataset with 1 MeV steps ranging from 125 to 200 MeV. For each model, training and validation involved 25 patients and 10 patients were for testing. Two single field uniform dose prostate treatment plans were optimized and recalculated with MC and the CE model.Results.Test results of all PBs from the three SE models showed a mean gamma passing rate (2%/2 mm, 10% dose cutoff) above 99.9% with an average center-of-mass (COM) discrepancy below 0.4 mm between predicted and simulated trajectories. The ME model showed a mean gamma passing rate exceeding 99.8% and a COM discrepancy of less than 0.5 mm at the three energies. Treatment plan recalculation by the CE model yielded gamma passing rates of 99.6% and 97.9%. The inference time of the models was 9-10 ms per PB.Significance.LSTM models for proton dose calculation in a magnetic field were developed and showed promising accuracy and efficiency for prostate cancer patients.

Chinese Named Entity Recognition Based on Multi-Level Representation Learning

Named Entity Recognition (NER) is a crucial component of Natural Language Processing (NLP). When dealing with the high diversity and complexity of the Chinese language, existing Chinese NER models face challenges in addressing word sense ambiguity, capturing long-range dependencies, and maintaining robustness, which hinders the accuracy of entity recognition. To this end, a Chinese NER model based on multi-level representation learning is proposed. The model leverages a pre-trained word-based embedding to capture contextual information. A linear layer adjusts dimensions to fit an Extended Long Short-Term Memory (XLSTM) network, enabling the capture of long-range dependencies and contextual information, and providing deeper representations. An adaptive multi-head attention mechanism is proposed to enhance the ability to capture global dependencies and comprehend deep semantic context. Additionally, GlobalPointer with rotational position encoding integrates global information for entity category prediction. Projected Gradient Descent (PGD) is incorporated, introducing perturbations in the embedding layer of the pre-trained model to enhance stability in noisy environments. The proposed model achieves F1-scores of 96.89%, 74.89%, 72.19%, and 80.96% on the Resume, Weibo, CMeEE, and CLUENER2020 datasets, respectively, demonstrating improvements over baseline and comparison models.

Generative Adversarial Network Based Music Generation

Music has been an integral part of human civilization personally and culturally. Historically, music has been generated using various instruments, or natural sounds like water drops, or unconventional musical instruments like metal or glass-wares. At present, technologies like Musical Instrument Digital Interface (MIDI) are used to generate music electronically. This research investigates the use of Generative Adversarial Networks (GANs) for beginner-friendly music production. This model uses Long Short-Term Memory (LSTM) generator and Patch GAN as discriminator for the GAN architecture. The generator consists of input layer, embedding layer, LSTM layer and generates output with a SoftMax function. Similarly, the discriminator consists of a convolution layer, the output of which is averaged by the global average pooling layer, and output is generated by the sigmoid function. The model is trained on Maestro MIDI dataset. We make the process understandable by delving into the implementation specifics and outlining the fundamental concepts of music. Our effective model highlights the potential of GANs in music composition by producing cohesive music. After training for 50 epochs, the model exhibited a remarkable precision of 91.82 percent. This project uses the combination of Artificial Intelligence (AI) with music theory to provide intriguing new opportunities in the field of music. The model can be beneficial for different industries like gaming, music, entertainment, education, etc.

FELRec: efficient handling of item cold-start with dynamic representation learning in recommender systems

AbstractRecommender systems suffer from the cold-start problem whenever a new user joins the platform or a new item is added to the catalog. To address item cold-start, we propose to replace the embedding layer in sequential recommenders with a dynamic storage that has no learnable weights and can keep an arbitrary number of representations. In this paper, we present , a large embedding network that refines the existing representations of users and items in a recursive manner, as new information becomes available. In contrast to similar approaches, our model represents new users and items without side information and time-consuming finetuning, instead it runs a single forward pass over a sequence of existing representations. During item cold-start, our method outperforms similar method by 29.50–47.45%. Further, our proposed model generalizes well to previously unseen datasets in zero-shot settings. The source code is publicly available at https://github.com/kweimann/FELRec.

Exploiting explicit item–item correlations from knowledge graphs for enhanced sequential recommendation

In recent years, the research of employing knowledge graphs (KGs) in sequential recommendation (SR) has received a lot of attention, since the side information extracted from KGs, especially the information of the correlations between items, indeed helps the SR models achieve better performance. However, many previous KG-based SR models tend to introduce some noise information when learning item embeddings, or insufficiently fuse item–item correlations into their sequential modeling, thus limiting their performance improvements. In this paper, we propose a Distance-Aware Knowledge-based Sequential Recommendation model (DAKSR), which exploits the explicit item–item correlations from KGs to achieve enhanced SR. Specifically, as one critical component in our DAKSR, the distance score matrix (DSM) is first obtained to indicate the correlations between items, and then leveraged in the following three major modules of DAKSR. First, in the Item-Set Embedding layer (ISE) all item embeddings are learned based on DSM, in which the noise information is eliminated effectively. Meanwhile, the Knowledge-Infused Transformer (KIT) incorporates DSM into its attention mechanism to improve the feature extraction. Furthermore, the Knowledge Contrastive Learning module (KCL) also leverages the item–item correlations presented in DSM to generate two credible sequence views, which are used to refine sample representations through a contrastive learning strategy, and thus improve the model’s robustness. Our extensive experiments on three SR benchmarks obviously demonstrate our DAKSR’s superior performance over the state-of-the-art (SOTA) KG-based recommendation models. The implementation of our DAKSR is available at https://github.com/Easonsi/DAKSR for reproducing our experiment results conveniently.

Generating comprehensive lithium battery charging data with generative AI

In optimizing the performance and extending the lifespan of lithium batteries, accurate state prediction is crucial. Traditional regression and classification methods have achieved some success in predicting battery states. However, the effectiveness of these data-driven approaches largely depends on the availability and quality of public datasets. Additionally, generating electrochemical data primarily through battery experiments is a lengthy and costly process, making it extremely difficult to obtain high-quality data. This difficulty, combined with data incompleteness, significantly affects prediction accuracy. To address these challenges, this study introduces End of Life (EOL) and Equivalent Cycle Life (ECL) as supervised conditions for generative AI models. By integrating an embedding layer into the CVAE model, we developed the Refined Conditional Variational Autoencoder (RCVAE). Through preprocessing data into a quasi-video format, coupled with customized training and inference algorithms, the RCVAE generates the required charging data in real time based on the battery’s ECL and EOL. This avoids storing irrelevant information, saving space and resources. RCVAE uses a lightweight architecture, enabling fast generation of necessary voltage, current, temperature, and charging capacity data. This approach provides users with a comprehensive electrochemical dataset, pioneering a new research domain for the artificial synthesis of lithium battery data. Furthermore, based on the detailed synthetic data, various battery state indicators can be calculated, offering new perspectives and possibilities for lithium battery performance prediction.

New Adversarial Image Detection Based on Sentiment Analysis.

Deep neural networks (DNNs) are vulnerable to adversarial examples, while adversarial attack models, e.g., DeepFool, are on the rise and outrunning adversarial example detection techniques. This article presents a new adversarial example detector that outperforms state-of-the-art detectors in identifying the latest adversarial attacks on image datasets. Specifically, we propose to use sentiment analysis for adversarial example detection, qualified by the progressively manifesting impact of an adversarial perturbation on the hidden-layer feature maps of a DNN under attack. Accordingly, we design a modularized embedding layer with the minimum learnable parameters to embed the hidden-layer feature maps into word vectors and assemble sentences ready for sentiment analysis. Extensive experiments demonstrate that the new detector consistently surpasses the state-of-the-art detection algorithms in detecting the latest attacks launched against ResNet and Inception neutral networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The detector only has about 2 million parameters and takes less than 4.6 ms to detect an adversarial example generated by the latest attack models using a Tesla K80 GPU card.

Medical Visual Question‐Answering Model Based on Knowledge Enhancement and Multi‐Modal Fusion

ABSTRACTThis paper aims to utilize a knowledge graph for importing external knowledge. It combines multi‐modal fusion mechanisms and confidence detection mechanisms to explore the correlation between clinical problems and medical images, enhancing their effectiveness in medical visual question‐answering tasks. The proposed medical visual question answering model comprises a text knowledge enhancement layer, an image embedding layer, a multimodal fusion layer, a confidence detection layer, and a prediction layer. The experimental results demonstrate that the medical vision question‐answering model, based on knowledge enhancement and multi‐modal fusion, achieves an optimal accuracy of 59.3% and 16.2% in open‐domain question‐answering tasks on the VQA‐RAD and PathVQA datasets, respectively, thus validating the effectiveness of the proposed model.

AGNN: Alternating Graph-Regularized Neural Networks to Alleviate Over-Smoothing.

Graph convolutional network (GCN) with the powerful capacity to explore graph-structural data has gained noticeable success in recent years. Nonetheless, most of the existing GCN-based models suffer from the notorious over-smoothing issue, owing to which shallow networks are extensively adopted. This may be problematic for complex graph datasets because a deeper GCN should be beneficial to propagating information across remote neighbors. Recent works have devoted effort to addressing over-smoothing problems, including establishing residual connection structure or fusing predictions from multilayer models. Because of the indistinguishable embeddings from deep layers, it is reasonable to generate more reliable predictions before conducting the combination of outputs from various layers. In light of this, we propose an alternating graph-regularized neural network (AGNN) composed of graph convolutional layer (GCL) and graph embedding layer (GEL). GEL is derived from the graph-regularized optimization containing Laplacian embedding term, which can alleviate the over-smoothing problem by periodic projection from the low-order feature space onto the high-order space. With more distinguishable features of distinct layers, an improved Adaboost strategy is utilized to aggregate outputs from each layer, which explores integrated embeddings of multi-hop neighbors. The proposed model is evaluated via a large number of experiments including performance comparison with some multilayer or multi-order graph neural networks, which reveals the superior performance improvement of AGNN compared with the state-of-the-art models.

Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation.

The MRI-derived brain network serves as a pivotal instrument in elucidating both the structural and functional aspects of the brain, encompassing the ramifications of diseases and developmental processes. However, prevailing methodologies, often focusing on synchronous BOLD signals from functional MRI (fMRI), may not capture directional influences among brain regions and rarely tackle temporal functional dynamics. In this study, we first construct the brain-effective network via the dynamic causal model. Subsequently, we introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE). This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic inter-play between structural and effective networks via an ordinary differential equation (ODE) model, which characterizes spatial-temporal brain dynamics. Our framework is validated on several clinical phenotype prediction tasks using two independent publicly available datasets (HCP and OASIS). The experimental results clearly demonstrate the advantages of our model compared to several state-of-the-art methods.

Novel static decoupling algorithm for the multi-axis wheel force sensor based on the Informer network

Ensuring the stability of heavy multi-axle vehicles necessitates the accurate calibration and decoupling of Multi-axis Wheel Force Sensors (MWFS). Traditional methods often neglect the temporal coupling present in MWFS output data, leading to reduced accuracy. This paper introduces an Improved Decoupling Algorithm (IDI) based on the Informer network, designed to temporally decouple MWFS and enhance precision. The Decoupling embedding layer (DE) performs linear decoupling of the MWFS, while the Token embedding layer (TE) and Informer encoder extract timing coupling features. The highway network and linear fully-connected layer then provide nonlinear decoupling compensation. Experimental results demonstrate that the IDI algorithm significantly outperforms traditional methods like Extreme Learning Machine (ELM) and Back Propagation Network (BPNN), achieving at least a 41.12% improvement in accuracy in the highly coupled Mz channel. In conclusion, the IDI algorithm not only achieves high-precision decoupling of MWFS but also presents a robust framework for modeling various sensor types.

Impact of Feature Extraction on Multi-Aspect Sentiment Classification for Livin'byMandiri Using BiLSTM

Mobile applications are currently experiencing very rapid development including applications in the financial sector. Livin'byMandiri is one of the mobile applications used to transact online without the need to go to the bank. This makes it very easy for customers to transact anywhere and anytime. Application reviews are user reviews that reflect the reputation of the application among the community, these application reviews can be found anywhere, so many companies use application reviews as a reference in developing their applications in the future. However, people's opinions on apps can vary and are influenced by many aspects. Therefore, aspect-based sentiment analysis can be applied to app reviews to get better results. This research focuses on analyzing the sentiment of Livin'byMandiri app reviews on the Google Play Store. In this research, the Bidirectional LSTM (Bi-LSTM) method is combined with TF-IDF and Word2Vec feature extraction. From the results of the experiments that have been carried out, the best accuracy results for the access aspect are 81.18% and F1-Score of 81.03%, the service aspect produces an accuracy of 82.82% and F1-Score of 82.74%, and for the convenience aspect produces an accuracy of 77.28% and F1-Score of 77.19%. In this experiment, it is also found that feature extraction has an effect on sentiment analysis, this is evidenced by an increase in accuracy of more than 1% for each aspect when TF-IDF feature extraction is added and also the combination of TF-IDF and Word2vec in the initial model built using only the Neural Network embedding layer.

Compositional Kronecker Context Optimization for vision–language models

Context Optimization (CoOp) has emerged as a simple yet effective technique for adapting CLIP-like vision–language models to downstream image recognition tasks. Nevertheless, learning context with satisfactory base-to-new, domain and cross-task generalization ability simultaneously while adapting to new tasks is a challenge. To tackle such a challenge, existing methods mainly exploit knowledge distillation with auxiliary text data written by human experts. However, we instead explore a new technique route by structuring the prompts without resorting to extra text data. As a result, we obtain a new lightweight yet generalizable approach termed Compositional Kronecker Context Optimization (CK-CoOp). Technically, the prompt’s context words in CK-CoOp are learnable vectors, which are crafted by linearly combining base vectors from a dictionary. These base vectors consist of a non-learnable component obtained by quantizing the weights in the token embedding layer, and a learnable component constructed by applying Kronecker product on several learnable tiny matrices. Intuitively, the compositional structure mitigates the risk of overfitting on training data and the Kronecker product breaks the non-learnable restrictions of the dictionary, thereby enhancing representation ability with minimal additional parameters. Extensive experiments confirm that, compared with existing methods, CK-CoOp can not only achieve comparable or even better performance under base-to-new, domain and cross-task generalization evaluation without the help of auxiliary text data, but also has the merits of fewer learnable parameters and efficient training and inference speed.

Dynamic chaos unveiled: enhancing ship’s attitude time series prediction through spatiotemporal embedding and improved transformer model

Abstract During ship operations at sea, the vessel's attitude undergoes continuous changes due to various factors such as wind, waves, and its own motion. These influences are challenging to mathematically describe, and the changes in attitude are also influenced by multiple interconnected factors. Consequently, accurately predicting the ship's attitude presents significant challenges. Previous studies have demonstrated that phenomena like wind speed and wave patterns exhibit chaotic characteristics when affecting attitude changes. However, research on predicting ship attitudes lacks an exploration of whether chaotic characteristics exist and how they can be described and applied. This paper initially identifies the chaotic characteristics of ship attitude data through phase space reconstruction analysis and provides mathematical representations for them. Based on these identified chaotic characteristics, a
Transformer model incorporating feature embedding layers is employed for time series prediction. Finally, a comparison with traditional methods validates the superiority of our proposed approach.

Research on personalised knowledge graph recommendation algorithm based on self-supervised learning

A recommendation algorithm that integrates generative graph self-supervised learning and contrastive graph self-supervised learning is proposed in this paper to address the issues of noisy data and graph incompleteness in knowledge graphs. Firstly, implicit semantics are captured by the self-supervised generative learning method through the masking and reconstruction of the edges of knowledge graph triples. Secondly, a multi-level contrastive learning method is proposed, which includes strategies for edge deletion and feature dropout in the graph structure and embedding layers, respectively.Finally, a contrastive loss function is introduced for contrastive learning to improve the quality of the embedded representations, and the BPR loss is incorporated and combined with the loss from the graph self-supervised learning module to optimize the overall performance of the recommender system. The results demonstrate that on the Last.fm, MovieLens, and Yelp2018 datasets, the model proposed in this paper significantly improves Recall and NDCG evaluation metrics compared to other traditional methods.

HEalthRecordBERT (HERBERT): Leveraging Transformers on Electronic Health Records for Chronic Kidney Disease Risk Stratification

Risk stratification is an essential tool in the fight against many diseases, including chronic kidney disease. Recent work has focused on applying techniques from machine learning and leveraging the information contained in a patient’s electronic health record (EHR). Irregular intervals between data entries and the large number of variables tracked in EHR datasets can make them challenging to work with. Many of the difficulties associated with these datasets can be overcome by using large language models, such as bidirectional encoder representations from transformers (BERT). Previous attempts to apply BERT to EHR for risk stratification have shown promise. In this work we propose HERBERT, a novel application of BERT to EHR data. We identify two key areas where BERT models must be modified to adapt them to EHR data, namely: the embedding layer and the pretraining task. We show how changes to these can lead to improved performance, relative to the previous state of the art. We evaluate our model by predicting the transition of chronic kidney disease patients to end stage renal disease. The strong performance of our model justifies our architectural changes and suggests that large language models could play an important role in future renal risk stratification.