Embedding Layer Research Articles

Using upsampling CONV-LSTM with metadata embedding for respiratory sound classification

Respiratory diseases are one of the leading causes of death around the world and they severely affect patient quality of life. Auscultation is an essential method for diagnosing respiratory diseases, and it is low-cost and convenient. However, auscultation requires experts who are highly experienced. Medical trainees suffer from misdiagnosis inevitably. To address this issue, a novel machine learning model is proposed, which consists of upsampling convolutional neural network (CNN), a long short-term memory network (LSTM), and a fully connected network (FCNN) with embedding layers to classify respiratory sounds into seven categories: Normal (N), Rhonchi (R), Wheeze (W), Stridor (S), Coarse Crackle (CC), Fine Crackle (FC), Wheeze & Crackle (WC). The model is trained and evaluated on the SPRSound dataset and obtained the result on the test dataset with a sensitivity of 0.5716, specificity of 0.7882, average score of 0.6799, harmonic score of 0.6626, and total score of 0.6756.

Capturing natural position relationships: A neural differential equation approach

The Transformer has emerged as the predominant model in Natural Language Processing due to its exceptional performance in various sequence modeling tasks, particularly in handling long-term dependencies. However, the traditional absolute and relative position encoding methods, which do not learn from data, tend to ignore the inherent structure of natural language sequences due to the position embedding layer at the input end of the Transformer model. This paper introduces a novel learnable neural Ordinary Differential Equation Position Encoding (ODEPE) method that can implicitly capture the natural position relationships within a sequence without requiring additional position embeddings. ODEPE can model continuous sequences and leverage differential equations to simulate the evolution of position information along the sequence, enabling position information to flow seamlessly between sequences. Additionally, a highly effective recurrent attention framework is proposed, which hybridizes attention with the ODEPE method to improve model performance. Compared to the Transformer-based sequence modeling network, our framework demonstrates a performance improvement of 24.0 points on the WikiText-103 dataset, while also achieving a performance improvement of 1.06 points on the Enwik8 dataset. This corresponds to an improvement of 4.9% and 0.17%, respectively.

A deep neural network based reverse radio spectrogram search algorithm

Abstract Modern radio astronomy instruments generate vast amounts of data, and the increasingly challenging radio frequency interference (RFI) environment necessitates ever-more sophisticated RFI rejection algorithms. The ‘needle in a haystack’ nature of searches for transients and technosignatures requires us to develop methods that can determine whether a signal of interest has unique properties, or is a part of some larger set of pernicious RFI. In the past, this vetting has required onerous manual inspection of very large numbers of signals. In this paper, we present a fast and modular deep learning algorithm to search for lookalike signals of interest in radio spectrogram data. First, we trained a β-variational autoencoder on signals returned by an energy detection algorithm. We then adapted a positional embedding layer from classical transformer architecture to a embed additional metadata, which we demonstrate using a frequency-based embedding. Next we used the encoder component of the β-variational autoencoder to extract features from small (∼715 Hz, with a resolution of 2.79 Hz per frequency bin) windows in the radio spectrogram. We used our algorithm to conduct a search for a given query (encoded signal of interest) on a set of signals (encoded features of searched items) to produce the top candidates with similar features. We successfully demonstrate that the algorithm retrieves signals with similar appearance, given only the original radio spectrogram data. This algorithm can be used to improve the efficiency of vetting signals of interest in technosignature searches, but could also be applied to a wider variety of searches for ‘lookalike’ signals in large astronomical data sets.

A Service Recommendation System Based on Dynamic User Groups and Reinforcement Learning

Recently, advancements in machine-learning technology have enabled platforms such as short video applications and e-commerce websites to accurately predict user behavior and cater to their interests. However, the limited nature of user data may compromise the accuracy of these recommendation systems. To address personalized recommendation challenges and adapt to changes in user preferences, reinforcement-learning algorithms have been developed. These algorithms strike a balance between exploring new items and exploiting existing ones, thereby enhancing recommendation accuracy. Nevertheless, the cold-start problem and data sparsity continue to impede the development of these recommendation systems. Hence, we proposed a joint-training algorithm that combined deep reinforcement learning with dynamic user groups. The goal was to capture user preferences for precise recommendations while addressing the challenges of data sparsity and cold-start. We used embedding layers to capture representations and make decisions before the reinforcement-learning process, executing this approach cyclically. Through this method, we dynamically obtained more accurate user and item representations and provide precise recommendations. Additionally, to address data sparsity, we introduced a dynamic user grouping algorithm that collectively enhanced the recommendations using group parameters. We evaluated our model using movie-rating and e-commerce datasets. As compared to other baseline algorithms, our algorithm not only improved recommendation accuracy but also enhanced diversity by uncovering recommendations across more categories.

Research on English Translation of Intangible Cultural Heritage in the Age of AIGC

Abstract Cultural dissemination and display methods with the progress of science and technology requirements continue to improve, especially in the AIGC era. The traditional cultural dissemination methods are no longer applicable, and “non-legacy digitization” needs to become more and more urgent. This paper analyzes the neural machine translation method and constructs a back-translation model by synthesizing a corpus and generating parameters. BLEU and other evaluation indexes are used to optimize the factors that affect the computation of synthetic data. An enhanced synthetic corpus is created using the kNN algorithm to construct a monolingual translation memory. Using the semantic information-sharing strategy, generate the parameters of the embedding layer of the non-legacy translation model while introducing the encoding layer to complete the overall construction of the non-legacy translation model. Comparative experiments and empirical analyses are set up to study the heterogeneity of the quality and dissemination effect of English translation of NRLs, and the experiments show that in the first set of translation tasks, the BLEU scores of the model proposed in this paper are 1 and 1.4 higher than those of the wait-k model when the value of k is 5 and 7, and the translation quality of the model proposed in this paper is better. The values of primary and secondary term coefficients of the English translation of non-legacy are 14.974 and -3.071, respectively, in the case of not establishing samples, showing an inverted U-shaped linear relationship.

Research on neural network-based fault diagnosis and prediction method for power communication equipment

Abstract In this paper, facing the digital development of the power grid and the status quo of massive power communication equipment access and targeting the demand for highly intelligent operation and maintenance management of the power grid, combined with neural network technology, we propose an intelligent diagnosis model of power communication equipment faults. Adopting BERT as the vector embedding layer to obtain the vector sequence of fault text, designing a fault entity recognition model for power communication equipment based on BERT-BiGRU-CRF, and completing the construction of the relationship set of fault text. The proposed knowledge graph-based power communication equipment fault intelligent diagnosis model combined with the WBLA-based power communication equipment fault severity level recognition algorithm to obtain different severity fault information, from which a TFIDF-COS-based power communication equipment fault intelligent diagnosis algorithm is designed to realize intelligent diagnosis of power communication equipment faults. After testing, the TFIDF-COS algorithm can get the best optimization effect when the number of hidden layers of the selected algorithm is 1, and the initial learning rate is 0.05, and its accuracy rate can be kept above 98%. Compared with the traditional fault diagnosis system, in terms of the order of magnitude 100M, 500M, 1G, and 5G, the running time is reduced by 322s, 1874s, 4617s, and 7467s, and the accuracy rate is increased by 2.33%, 2.6%, 32.02%, and 61.4%, respectively. Therefore, this paper realizes the accurate positioning of power communication equipment faults and provides technical support for intelligent operation and maintenance of the power grid.

Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding

When writing academic papers, referencing statements, claims and previous studies is always an important activity. However, it is considered challenging for scientists to find relevant and appropriate scientific articles which are closely related to their current works in order to reference them in their research. As a consequence of the rapid growth in scientific papers being published every year, researchers might easily get overwhelmed within a huge number of resources. One way to help them find the desired references more easily is to use context-aware citation recommendation. The citation recommender system can automatically provide a list of suitable papers as references based on specified inputs which reflect the researchers’ interests. Among the outstanding achievements of deep learning and natural language processing in recent years, the utilization of deep neural learning architectures have supported to address the problem of citation recommendation. As the result, the neural citation recommendation area has received much attention from the academic community, with the aim of enhancing the precision and correctness of the results of existing citation recommendation systems. Following this research direction, in our paper we present a novel context-aware citation recommendation model, called RHN-DualLCR (Recurrent Highway Networks – Dual Local Citation Recommendation), which integrates Recurrent Highway Networks (RHN), an improved model of the original Bidirectional Long Short-Term Memory (BiLSTM) architecture, and uses a SciBERT-based (Science text of Bidirectional Encoder Representations from Transformers) embedding layer to build up the efficiency of the state-of-the-art local citation recommendation model, which enriches context representation with global information. Our research demonstrates its originality and relevance because we used one of the latest achievements of deep models (RHN model) and natural language processing (SciBERT) applied to the citation recommendation problem. We have conducted experiments on the RHN-DualLCR model on 3 widely known datasets for the citation recommendation problem: ACL-200 (Association for Computational Linguistics), ACL-600 and RefSeer, and also used 2 common evaluation standards Mean Reciprocal Rank (MRR) and the Recall@K (R@K for short) to evaluate the performance of our model. Experimental results show that our proposed model is 3% to 16% better than the original models or state-of-the-art models.

SMAAMA: A named entity alignment method based on Siamese network character feature and multi-attribute importance feature for Chinese civil aviation

With the growth of demand for air-space integrated transportation industry, the civil aviation transportation industry plays an increasingly important role in the national economy and transportation. The diversity and diversified development of civil aviation emergencies have greatly affected the decision-making efficiency for emergencies in civil aviation emergency management systems. Constructing a knowledge graph of civil aviation and improving the reliability and richness of the knowledge graph has become an urgent problem to be solved. There are some problems during the construction of a civil aviation domain knowledge graph such as long entity length, existing hybrid and composite entities, the similarity of the font features of domain entity names, a difference of information between entities, separated coding between entities, and error-prone in the transmission process of coding. So we research the construction method of knowledge graph for civil aviation emergencies to resolve these problems. (1) We propose a font pre-training model and incorporate a character feature layer into the embedding layer. (2) We propose a multi-attribute attention alignment method based on the Siamese network; entities with similar structures in the civil aviation knowledge base are input into the font layer for pre-training; the complete semantic information of entities and the importance of different attributes to entities are learned through two sub-networks respectively to improve the integrity of civil aviation knowledge graph. Experiments were carried out on the self-built aviation emergency and public data sets, respectively. The experimental results show that compared with other models, the F1 value of the proposed model can reach 96.97%, which verifies the feasibility of the model.

DMMP: A distillation-based multi-task multi-tower learning model for personalized recommendation

Recommendation systems involve the matching and ranking stages. The click-through rate (CTR) and the conversion rate (CVR) predictions are two fundamental modules in recommendation systems. Most candidate generation matching models leverage a two-tower architecture to model the CTR prediction task. However, items with low-quality but attractive titles, i.e., click baits, may be recommended to the user, which worsens the user’s experience. Therefore, both click and conversion tasks should be modeled during the matching stage to improve user engagement. An intuitive way is to model these two tasks with a three-tower matching model. However, its efficiency and effectiveness are limited. By inheriting the merits of knowledge distillation and multi-task learning, we propose a distillation-based multi-task multi-tower model (DMMP) for personalized recommendation. Specifically, the MTL-based teacher network builds the task-shared and task-specific expert networks, and employs a customized multi-gate control network to merge these expert networks adaptively within the task embedding layer. Two auxiliary pCTR and pCTCVR tasks are incorporated to model CVR directly across the whole space. By sharing the feature representation parameters with the CTR modeling, the CVR modeling can be trained with richer samples. Three-tower-based candidate generation student network incorporates a user preference gating network to learn the task scores so that personalized candidates can be produced. Furthermore, we design an adaptive weighting strategy for total loss to eventually adjust the importance and relevance of different networks. Extensive experiments on public and industrial datasets validate the effectiveness of DMMP.

SwinCrack: Pavement crack detection using convolutional swin-transformer network

By leveraging deep learning methods, pavement crack detection can be more automatic, efficient, and accurate than manual inspection. To solve the problem of limited receptive field in pure CNN-based crack detection networks, we proposed an end-to-end detection network based on Swin-Transformer, called SwinCrack. SwinCrack can produce more accurate and continuous descriptions of pavement cracks by modeling long-range interactions and adaptive spatial aggregation compared to CNN-based detection models. Furthermore, to delineate crisp and accurate crack boundaries, we introduced convolution operations to Swin-Transformer for more local and detailed crack information. Convolutional Patch Embedding Layer (CPEL), Convolutional Swin-Transformer Block (CSTB), and Depth-convolution Forward Network (DFN) are proposed and embedded into SwinCrack to capture more spatial contexts. Also, Convolutional Attention Gated Skip Connection (CAGSC) is designed to suppress background interference in low-level features. Furthermore, five evaluation experiments on SwinCrack and an ablation study on the four proposed modules are performed. The attention maps of the SwinCrack are visualized to give a better insight into the contribution of each convolutional module embedded. Evaluation results show that SwinCrack gains OIS values of 0.781 to 0.849 and a maximum 4.4% improvement on OIS among the six public crack datasets.

SENTIMENT ANALYSIS USING LONG TERM MEMORY (LSTM) BOOK CASE STUDY: UNSOLICITED ADVICE FOR MURDERERS BY VERA WONG'S

Reading books is one of the most effective ways to reduce stress. In today's digital era, access to finding and buying books is getting easier, so reader reviews are important in choosing books that match interests. However, with a large number of reviews, Natural Language Processing (NLP) with the Long Short-Term Memory (LSTM) method is used to help analyze positive and negative sentiments from many book reviews. This sentiment analysis is useful for readers to evaluate the quality of books, as well as for authors and sellers to find out the opinions of readers and improve the quality of their work. In this study, the book review dataset "Vera Wong's Unsolicited Advice for Murderers" from the Goodreads website is used, which is then divided into training data and validation data with a ratio of 75%: 25%. The Long Short-Term Memory (LSTM) method is used to analyze the sentiment of the reviews. The model architecture built consists of Embedding Layer, LSTM Layer with 128 neuron units, 3 Dense Layer with ReLU activation function, 3 Dropout Layer, and Fully Connected Layer with and Sigmoid activation function, Binary Cross Entropy loss function, and RMSprop optimizer. The model training process was conducted with 30 epochs. The evaluation results show that the model achieved an accuracy of 90%, indicating the model performs relatively well in correctly classifying positive sentiments.

Multi-Channel 2D-CNN Attention Based BiLSTM Method for Low-Resource Ewe Sentiment Analysis

The unavailability of an annotated dataset for a low-resource Ewe language makes it difficult to develop an automated system to appropriately evaluate public opinion on events, news, policies, and regulations. In this study, we collected and preprocessed a low-resourced document-level Ewe sentiment dataset based on social media comments. We used three (3) features learned by word embeddings (Global vectors, word-to-vector, and FastText) rather than hand-crafted features. We further proposed a novel method termed MC2D-CNN+BiLSTM-Attn to detect the exact sentiment feature from the Ewe dataset. Extensive experiments indicate that the proposed method efficiently classifies various sentiments and is superior to benchmark deep learning methods. Results show that in detecting the precise sentiments from raw Ewe textual context, the BiLSTM incorporating Glove outperforms word2vec and FastText embedding with an accuracy of 0.727. Furthermore, Attn+BiLSTM and Multi-channel CNN methods incorporating the word2vec embedding layer perform better than Glove and FastText embedding with an accuracy of 0.848 and 0.896. In contrast, our proposed method with the same word2vec embedding recorded 0.949.

Spline-based neural network interatomic potentials: Blending classical and machine learning models

While machine learning (ML) interatomic potentials (IPs) are able to achieve accuracies nearing the level of noise inherent in the first-principles data to which they are trained, it remains to be shown if their increased complexities are strictly necessary for constructing high-quality IPs. In this work, we introduce a new MLIP framework which blends the simplicity of spline-based MEAM (s-MEAM) potentials with the flexibility of a neural network (NN) architecture. The proposed framework, which we call the spline-based neural network potential (s-NNP), is a simplified version of the traditional NNP that can be used to describe complex datasets in a computationally efficient manner. We demonstrate how this framework can be used to probe the boundary between classical and ML IPs, highlighting the benefits of key architectural changes. Furthermore, we show that using spline filters for encoding atomic environments results in a readily interpreted embedding layer which can be coupled with modifications to the NN to incorporate expected physical behaviors and improve overall interpretability. Finally, we test the flexibility of the spline filters, observing that they can be shared across multiple chemical systems in order to provide a convenient reference point from which to begin performing cross-system analyses.

Correcting spelling mistakes in Persian texts with rules and deep learning methods

This study aims to develop a system for automatically correcting spelling errors in Persian texts using two approaches: one that relies on rules and a common spelling mistake list and another that uses a deep neural network. The list of 700 common misspellings was compiled, and a database of 55,000 common Persian words was used to identify spelling errors in the rule-based approach. 112 rules were implemented for spelling correction, each providing suggested words for misspelled words. 2500 sentences were used for evaluation, with the word with the shortest Levenshtein distance selected for evaluation. In the deep learning approach, a deep encoder-decoder network that utilized long short-term memory (LSTM) with a word embedding layer was used as the base network, with FastText chosen as the word embedding layer. The base network was enhanced by adding convolutional and capsule layers. A database of 1.2 million sentences was created, with 800,000 for training, 200,000 for testing, and 200,000 for evaluation. The results showed that the network's performance with capsule and convolutional layers was similar to that of the base network. The network performed well in evaluation, achieving accuracy, precision, recall, F-measure, and bilingual evaluation understudy (Bleu) scores of 87%, 70%, 89%, 78%, and 84%, respectively.

Abstract 18281: Identifying Predictors of ICU Mortality Outcomes Among Critically Ill Patients With Atrial Fibrillation Using Machine Learning Models

Introduction: New onset atrial fibrillation (Afib) is a frequent complication of intensive care admission and is linked to higher mortality in the ICU. We aimed to develop machine learning (ML) models to predict the risk of death in the ICU among these patients and assess their long-term prognosis. Methods: Eligible patients with Afib from the Medical Information Mart for Intensive Care IV database were included. The cases were randomly divided into training (75%) and test (25%) groups. Machine learning algorithms such as light GBM, random forest, gradient boosted classifier, and XGBoost were used. In addition, we utilized a tabular deep learning model architecture comprised of an embedding layer for categorical variables. The robustness of the results was ensured through K-fold cross-validation. AUROC and accuracy were used to compare model performance. Models accounted for variables from demographics, labs, charting events, and comorbidity domains. A web-based calculator was also developed to encourage the model's practical application. Results: The sample (n = 8455) consisted of 3408 females (40.3%), 6004 whites (71.0%), and 4876 (57.7%) patients with Medicare insurance. The ICU mortality was 9.5% (n = 802). The median age (79 vs. 75 years; p < 0.001) and ICU LOS (4 vs. 2 days; p < 0.001) were higher among the patients who expired. Congestive heart failure was reported in 15% of cases. The lightGBM model (AUROC: 84%) performed best and showed that mean arterial BP, WBC, PO2, platelet count, age, hemoglobin, BUN, pH, ICU LOS and anion gap were the top predictors of ICU mortality among this population. Conclusion: Death in the ICU among patients with Afib was associated with several clinical features, especially BP, WBC count, PO2, platelet count, age, and hemoglobin. Predictors of ICU mortality were identified with high accuracy and this may aid in the creation of a real time risk score for selective intervention to reduce the risk of death in this vulnerable patient population.

Ideological Analysis of Digital Capitalism in the Context of Marxist Political Economy Critique

Abstract This paper focuses on analyzing and studying digital capitalism using a textual analysis model in the context of Marxist political economy. The magnitude of the angle between text vectors can be used to measure the similarity between digital capitalism and political economy. An iterative computational approach is employed to address the issue of overfitting in text. Constructing a BERT model involves formulating a self-attentive mechanism and setting up an embedding layer. The direction of digital capitalism’s ideology in the political economy context is examined about its characteristics. The results show that digital labor is more secretive regarding office location than traditional labor, with a secrecy rate of 60%. Unpaidness, large quantity, and potential value dominate data characteristics, with 90% unpaid data.

HA-ResNet: Residual Neural Network With Hidden Attention for ECG Arrhythmia Detection Using Two-Dimensional Signal.

Arrhythmia is an abnormal heart rhythm, a common clinical problem in cardiology. Long-term or severe arrhythmia may lead to stroke and sudden cardiac death. The electrocardiogram (ECG) is the most commonly used tool to diagnose arrhythmia. However, the traditional diagnosis relies on experts for manual interpretation, which is time-consuming and laborious. In recent years, many automatic arrhythmia detection methods have emerged due to advancements in deep learning. These methods can reduce manual intervention and improve diagnostic efficiency. However, extracting useful features from raw ECG signals for arrhythmia detection is still challenging due to the low frequency of ECG signals and noise distribution. In this paper, we propose a novel hidden attention residual network (HA-ResNet) for automated arrhythmia classification. In this model, the one-dimensional ECG signals are first converted into two-dimensional images and fed into an embedding layer to obtain the relevant shallow features in ECG. Then, a hidden attention layer combining Squeeze-and-Excitation (SE) block and Bidirectional Convolutional LSTM (BConvLSTM) is used to further capture the deep Spatio-temporal features. We evaluate our HA-ResNet on two public datasets and achieve F1 scores of 96.0%, 96.7%, and 87.6% on 2s segments, 5s segments, and 10s segments, respectively, which significantly outperform the existing state-of-the-art approaches. The experimental results demonstrate the effectiveness and generalization of our method.

ASENN: attention-based selective embedding neural networks for road distress prediction

This study proposes an innovative neural network framework, ASENN (Attention-based Selective Embedding Neural Network), for the prediction of pavement deterioration. Considering the complexity and uncertainty associated with the pavement deterioration process, two fundamental frameworks, SEL (Selective Embedding Layer) and MDAL (Multi-Dropout Attention Layer), are combined to enhance feature abstraction and prediction accuracy. This approach is significant while analyzing the pavement deterioration process due to the high variability of the contributing deterioration factors. These factors, represented as tabular data, undergo filtering, embedding, and fusion stages in the SEL, to extract crucial features for an effective representation of pavement deterioration. Further, multiple attention-weighted combinations of raw data are obtained through the MDAL. Several SELs and MDALs were combined as basic cells and layered to form an ASENN. The experimental results demonstrate that the proposed model outperforms existing tabular models on four road distress parameter datasets corresponding to cracking, deflection, international roughness index, and rutting. The optimal number of cells was determined using different ablation settings. The results also show that the feature learning capabilities of the ASENN model improved as the number of cells increased; however, owing to the limited combination space of feature fields, extreme depths were not preferred. Furthermore, the ablation investigation demonstrated that MDAL can improve performance, particularly on the cracking dataset. Notably, compared with mainstream transformer models, ASENN requires significantly less storage and achieves faster execution speed.

Design of a Modified Transformer Architecture Based on Relative Position Coding

Natural language processing (NLP) based on deep learning provides a positive performance for generative dialogue system, and the transformer model is a new boost in NLP after the advent of word vectors. In this paper, a Chinese generative dialogue system based on transformer is designed, which only uses a multi-layer transformer decoder to build the system and uses the design of an incomplete mask to realize one-way language generation. That is, questions can perceive context information in both directions, while reply sentences can only output one-way autoregressive. The above system improvements make the one-way generation of dialogue tasks more logical and reasonable, and the performance is better than the traditional dialogue system scheme. In consideration of the long-distance information weakness of absolute position coding, we put forward the improvement of relative position coding in theory, and verify it in subsequent experiments. In the transformer module, the calculation formula of self-attention is modified, and the relative position information is added to replace the absolute position coding of the position embedding layer. The performance of the modified model in BLEU, embedding average, grammatical and semantic coherence is ideal, to enhance long-distance attention.

A BiLSTM-attention-based point-of-interest recommendation algorithm

Abstract Aiming at the problem that users’ check-in interest preferences in social networks have complex time dependences, which leads to inaccurate point-of-interest (POI) recommendations, a location-based POI recommendation model using deep learning for social network big data is proposed. First, the original data are fed into an embedding layer of the model for dense vector representation and to obtain the user’s check-in sequence (UCS) and space-time interval information. Then, the UCS and spatiotemporal interval information are sent into a bidirectional long-term memory model for detailed analysis, where the UCS and location sequence representation are updated using a self-attention mechanism. Finally, candidate POIs are compared with the user’s preferences, and a POI sequence with three consecutive recommended locations is generated. The experimental analysis shows that the model performs best when the Huber loss function is used and the number of training iterations is set to 200. In the Foursquare dataset, Recall@20 and NDCG@20 reach 0.418 and 0.143, and in the Gowalla dataset, the corresponding values are 0.387 and 0.148.