Word Representations Research Articles

SeMalBERT: Semantic-based malware detection with bidirectional encoder representations from transformers

Machine learning models are widely used for identifying malicious software. However, existing models suffer from issues such as imprecise polysemous representations and a lack of contextual semantic representations, leading to the failure to recognize certain types of malicious software. In this paper, we propose a semantic-based intelligent malware detection model called SeMalBERT for identifying malicious software in Windows systems. Specifically, the model utilizes the API function sequences of malicious software as the learned features. Firstly, BERT is applied to accomplish word representation tasks and extract semantic information from the sequences. Secondly, a hybrid discriminator based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is used to explore the chaining relationships between functions. Lastly, an external attention mechanism is added after the LSTM to enable the model to better focus on key information within the text. Experimental results demonstrate that SeMalBERT outperforms existing malware detection techniques in terms of accuracy, F1 score, and loss function value on a general dataset.

Understanding the Concept of Phonetic and Visual Similarity Vis-A-Vis To Letter Trademarks Through Judicial Precedents

The assessment of the distinctiveness of trademarks, despite the criteria outlined in the Trademark’s Act of 1999, specifically under sections 9 and 11, presents a formidable challenge, often necessitating the discerning perspective of the judiciary. As a fundamental guiding principle, both examiner and courts are obliged to evaluate a trademark as a unified entity. In recent years, there has been a discernible rise in the scrutiny of phonetic similarities between marks. The legal framework for dealing with the infringement of a registered mark due to phonetic resemblance is encapsulated within section 29(9) of trademarks act 1999. This section expressly stipulates that a mark may be infringed by the oral as well as the visual representation of words. The Supreme Court has consistently affirmed the imperative need to consider both auditory and visual aspects when comparing trademark. This paper delves into an analysis of numerous judgements handed down by the Supreme Court and various High Courts, primarily focusing on the assessment of both phonetic and visual aspects of trademarks. Special emphasis has been placed upon letter trademarks and their susceptibility to both visual and phonetic infringement. While trademarks typically encompass various elements such as devices, color combinations, letters and images, letter trademarks, often consisting of a single letter with descriptive trade connotations, are generally considered weaker trade makers. To establish trademark rights in such instances compelling evidence of usage is requisite. Conversely, trademarks comprised of more than one letter, devoid of descriptive elements pertaining to goods and trade, are regarded as stronger trademarks. In cases where a letter trademark incorporates a unique device, it offers robust protection for the trademark. This paper references multiple case laws where esteemed courts have recognised unequivocally rights with respect to letter trademarks.

The membership problem for subsemigroups of [formula omitted] is NP-complete

We show that the problem of determining if the identity matrix belongs to a finitely generated semigroup of 2×2 matrices from the modular group PSL2(Z), the Special Linear group SL2(Z) and the General Linear Group GL2(Z) is solvable in NP. We extend this to prove that the membership problem is decidable in NP for GL2(Z) and for any arbitrary regular expression over matrices from SL2(Z). We then derive that the problems of whether a given finite set of matrices from SL2(Z) or PSL2(Z) generates a group or a free semigroup are both decidable in NP. The previous algorithm for these problems, shown in 2005 by Choffrut and Karhumäki, was in EXPSPACE. Our algorithm is based on new techniques allowing us to operate on compressed word representations of matrices without explicit expansions. When combined with the known NP-hard lower bound, this proves that the identity (and thus membership) problem over GL2(Z) is NP-complete, and the group problem and the non-freeness problem in SL2(Z) are NP-complete. Thus the paper answer the long standing open question on the complexity of the membership problem in semigroups generated by matrices from GL2(Z). We develop novel techniques that can be used for solving numerical matrix problems in symbolic form, which are applicable for solving compressed word problems for groups and semigroups, bridging the gap between combinatorial group theory, computational problems on matrices and complexity theory.1

Minimal gains for minimal pairs: Difficulty in learning similar-sounding words continues into preschool

A critical indicator of spoken language knowledge is the ability to discern the finest possible distinctions that exist between words in a language—minimal pairs, for example, the distinction between the novel words beesh and peesh. Infants differentiate similar-sounding novel labels like “bih” and “dih” by 17 months of age or earlier in the context of word learning. Adult word learners readily distinguish similar-sounding words. What is unclear is the shape of learning between infancy and adulthood: Is there a nonlinear increase early in development, or is there protracted improvement as experience with spoken language amasses? Three experiments tested monolingual English-speaking children aged 3 to 6 years and young adults. Children underperformed when learning minimal-pair words compared with adults (Experiment 1), compared with learning dissimilar words even when speech materials were optimized for young children (Experiment 2), and when the number of word instances during learning was quadrupled (Experiment 3). Nonetheless, the youngest group readily recognized familiar minimal pairs (Experiment 3). Results are consistent with a lengthy trajectory for detailed sound pattern learning in one’s native language(s), although other interpretations are possible. Suggestions for research on developmental trajectories across various age ranges are made.

Language with vision: A study on grounded word and sentence embeddings

Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations by incorporating perceptual knowledge from vision into text-based representations. Despite many attempts at language grounding, achieving an optimal equilibrium between textual representations of the language and our embodied experiences remains an open field. Some common concerns are the following. Is visual grounding advantageous for abstract words, or is its effectiveness restricted to concrete words? What is the optimal way of bridging the gap between text and vision? To what extent is perceptual knowledge from images advantageous for acquiring high-quality embeddings? Leveraging the current advances in machine learning and natural language processing, the present study addresses these questions by proposing a simple yet very effective computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information while simultaneously preserving the distributional statistics that characterize word usage in text corpora. By applying a learned alignment, we are able to indirectly ground unseen words including abstract words. A series of evaluations on a range of behavioral datasets shows that visual grounding is beneficial not only for concrete words but also for abstract words, lending support to the indirect theory of abstract concepts. Moreover, our approach offers advantages for contextualized embeddings, such as those generated by BERT (Devlin et al, 2018), but only when trained on corpora of modest, cognitively plausible sizes. Code and grounded embeddings for English are available at (https://github.com/Hazel1994/Visually_Grounded_Word_Embeddings_2).

Optimizing Vocational Education Design with Big Data and Educational Technology

Amidst the rapid advancements in big data and internet technologies, profound transformations have been witnessed in modern societal and economic structures. Vocational education, a pivotal component within the educational domain, faces challenges in course design and optimization. While the majority of existing research focuses on utilizing educational technology to optimize courses, most methodologies still rely on traditional data analysis and mining techniques, with little exploration of deep learning and big data technologies. This research proposes a novel strategy for course design and optimization. Initially, the course theme optimization is addressed through a short text clustering algorithm that utilizes a linear fusion of the biterm topic model (BTM) and global vectors for word representation (GloVe) similarity. Subsequently, course content is planned using a deep interest network (DIN) recommendation algorithm in conjunction with the gated recurrent unit (GRU) time series model. This study provides the vocational education sector with a new perspective on course design and optimization, which has important theoretical and practical implications.

RESEARCH ON NAMED ENTITY RECOGNITION METHODS FOR URBAN UNDERGROUND SPACE DISASTERS BASED ON TEXT INFORMATION EXTRACTION

Abstract. Urban underground space is a complex spatial scenario that is highly susceptible to disasters. To achieve entity recognition in textual information related to urban underground space disasters, this study proposes the ALBERT-BiLSTM-CRF model. The urban underground space disaster text data is firstly encoded using the ALBERT model, which captures the deep semantic information of words in the context. The encoded data is then fed into a BiLSTM network to obtain hidden state vectors for each word, enhancing the feature representation of words. Finally, these vectors are input into a CRF layer to obtain the optimal label sequence and complete named entity recognition. The proposed model achieves an accuracy of 95.41%, a recall of 94.08%, and an F1 score of 94.74%. Comparative experiments with the BiLSTM-CRF, BERT-CRF, and BERT-BiLSTM-CRF models are conducted on the Boson dataset and our experimental dataset, demonstrating the superior performance of the ALBERT-BiLSTM-CRF model.

Modelling orthographic similarity effects in recognition memory reveals support for open bigram representations of letter coding

A variety of letter string representations has been proposed in the reading literature to account for empirically established orthographic similarity effects from masked priming studies. However, these similarity effects have not been explored in episodic memory paradigms and very few memory models have employed orthographic representation of words. In the current work, through two recognition memory experiments employing word and pseudoword stimuli respectively, we empirically established a set of key orthographic similarity effects for the first time in recognition memory – namely the substitution effect, transposition effect and reverse effect in recognition memory of words and pseudowords, and a start-letter importance in recognition memory of words. Subsequently, we compared orthographic representations from the reading literature including slot coding, closed-bigram, open-bigram and the overlap model. Each of these representations was situated in a global matching model and fitted to recognition performance via Luce’s choice rule in a hierarchical Bayesian framework. Model selection results showed support for the open-bigram representation in both experiments.

M5UMCB: Prediction of RNA 5-methyluridine sites using multi-scale convolutional neural network with BiLSTM

As a prevalent RNA modification, 5-methyluridine (m5U) plays a critical role in diverse biological processes and disease pathogenesis. High-throughput identification of m5U typically relies on labor-intensive biochemical experiments using various sequencing-based techniques, which are not only time-consuming but also expensive. Consequently, there is a pressing need for more efficient and cost-effective computational methods to complement these high-throughput techniques. In this study, we present m5UMCB, a novel approach that harnesses a multi-scale convolutional neural network (CNN) in tandem with bidirectional long short-term memory (BiLSTM) to recognize m5U sites. Our method involves segmenting RNA sequences into smaller fragments based on a 3-mer length and subsequently mapping each fragment to a lower-dimensional vector representation using the global vectors for word representation (GloVe) technique. Through a series of multi-scale convolution and pooling operations, local features are extracted from RNA sequences and transformed into abstract, high-level features. The feature matrix is then inputted into a BiLSTM network, enabling the capture of contextual information and long-term dependencies within the sequence. Ultimately, a fully connected layer is employed to classify m5U sites. The validation results from 5-fold cross-validation (5-fold CV) test indicate that m5UMCB outperforms existing state-of-the-art predictive methods, demonstrating a 1.98% increase in the area under ROC curve (AUC) and significant improvements in relevant evaluation metrics. We are confident that m5UMCB will serve as a valuable tool for m5U prediction.

A Local Self-Attention Sentence Model for Answer Selection Task in CQA Systems

Current evidence indicates that the semantic representation of question and answer sentences is better generated by deep neural network-based sentence models than traditional methods in community answer selection tasks. In particular, as a widely recognized language model, the self-attention model computes the similarity between the specific word and the whole sets of words in the same sentence and generates new semantic representation through the similarity-weighted summation of semantic representations of the whole words. However, the self-attention operation entirely considers all the signals with a weighted sum operation, which disperses the distribution of attention, which may result in overlooking the relation of neighboring signals. This issue becomes serious when applying the self-attention model to online community question answering platforms because of the varied length of the user-generated questions and answers. To address this problem, we introduce an attention mechanism enhanced local self-attention (LSA), which restricts the range of original self-attention by a local window mechanism, thereby scaling linearly when increasing the sequence length. Furthermore, we propose stacking multiple LSA layers to model the relationship of multiscale <inline-formula> <tex-math notation="LaTeX">$n$</tex-math> </inline-formula>-gram features. It captures the word-to-word relationship in the first layer and then captures the chunk-to-chunk (such as lexical <inline-formula> <tex-math notation="LaTeX">$n$</tex-math> </inline-formula>-gram phrases) relationship in its deeper layers. We also test the effectiveness of the proposed model by applying the learned representation through the LSA model to a Siamese and a classification network in community question answer selection tasks. Experiments on the public datasets show that the proposed LSA achieves a good performance.

A novel meta-embedding technique for drug reviews sentiment analysis

&lt;p&gt;Traditional word embedding models have been used in the feature extraction process of deep learning models for sentiment analysis. However, these models ignore the sentiment properties of words while maintaining the contextual relationships and have inadequate representation for domainspecific words. This paper proposes a method to develop a meta embedding model by exploiting domain sentiment polarity and adverse drug reaction (ADR) features to render word embedding models more suitable for medical sentiment analysis. The proposed lexicon is developed from the medical blogs corpus. The polarity scores of the existing lexicons are adjusted to assign new polarity score to each word. The neural network model utilizes sentiment lexicons and ADR in learning refined word embedding. The refined embedding obtained from the proposed approach is concatenated with original word vectors, lexicon vectors, and ADR feature to form a meta-embedding model which maintains both contextual and sentimental properties. The final meta-embedding acts as a feature extractor to assess the effectiveness of the model in drug reviews sentiment analysis. The experiments are conducted on global vectors (GloVE) and skip-gram word2vector (Word2Vec) models. The empirical results demonstrate the proposed meta-embedding model outperforms traditional word embedding in different performance measures.&lt;/p&gt;

Modeling Brain Representations of Words' Concreteness in Context Using GPT-2 and Human Ratings.

The meaning of most words in language depends on their context. Understanding how the human brain extracts contextualized meaning, and identifying where in the brain this takes place, remain important scientific challenges. But technological and computational advances in neuroscience and artificial intelligence now provide unprecedented opportunities to study the human brain in action as language is read and understood. Recent contextualized language models seem to be able to capture homonymic meaning variation ("bat", in a baseball vs. a vampire context), as well as more nuanced differences of meaning-for example, polysemous words such as "book", which can be interpreted in distinct but related senses ("explain a book", information, vs. "open a book", object) whose differences are fine-grained. We study these subtle differences in lexical meaning along the concrete/abstract dimension, as they are triggered by verb-noun semantic composition. We analyze functional magnetic resonance imaging (fMRI) activations elicited by Italian verb phrases containing nouns whose interpretation is affected by the verb to different degrees. By using a contextualized language model and human concreteness ratings, we shed light on where in the brain such fine-grained meaning variation takes place and how it is coded. Our results show that phrase concreteness judgments and the contextualized model can predict BOLD activation associated with semantic composition within the language network. Importantly, representations derived from a complex, nonlinear composition process consistently outperform simpler composition approaches. This is compatible with a holistic view of semantic composition in the brain, where semantic representations are modified by the process of composition itself. When looking at individual brain areas, we find that encoding performance is statistically significant, although with differing patterns of results, suggesting differential involvement, in the posterior superior temporal sulcus, inferior frontal gyrus and anterior temporal lobe, and in motor areas previously associated with processing of concreteness/abstractness.

Neural tracking measures of speech intelligibility: Manipulating intelligibility while keeping acoustics unchanged.

Neural speech tracking has advanced our understanding of how our brains rapidly map an acoustic speech signal onto linguistic representations and ultimately meaning. It remains unclear, however, how speech intelligibility is related to the corresponding neural responses. Many studies addressing this question vary the level of intelligibility by manipulating the acoustic waveform, but this makes it difficult to cleanly disentangle the effects of intelligibility from underlying acoustical confounds. Here, using magnetoencephalography recordings, we study neural measures of speech intelligibility by manipulating intelligibility while keeping the acoustics strictly unchanged. Acoustically identical degraded speech stimuli (three-band noise-vocoded, ~20 s duration) are presented twice, but the second presentation is preceded by the original (nondegraded) version of the speech. This intermediate priming, which generates a "pop-out" percept, substantially improves the intelligibility of the second degraded speech passage. We investigate how intelligibility and acoustical structure affect acoustic and linguistic neural representations using multivariate temporal response functions (mTRFs). As expected, behavioral results confirm that perceived speech clarity is improved by priming. mTRFs analysis reveals that auditory (speech envelope and envelope onset) neural representations are not affected by priming but only by the acoustics of the stimuli (bottom-up driven). Critically, our findings suggest that segmentation of sounds into words emerges with better speech intelligibility, and most strongly at the later (~400 ms latency) word processing stage, in prefrontal cortex, in line with engagement of top-down mechanisms associated with priming. Taken together, our results show that word representations may provide some objective measures of speech comprehension.

A Synergistic Bidirectional LSTM and N-gram Multi-channel CNN Approach Based on BERT and FastText for Arabic Event Identification

Event extraction from texts continues to pose a challenge for many NLP systems. This article presents a novel neural network architecture that can extract and classify events from Arabic sentences. The model combines word representations and Part-Of-Speech (POS) tags and uses a bidirectional LSTM layer and a dual combined convolutional neural network. The first layer of the network focuses on sentence representations, while the second layer focuses on POS representations. The model takes advantage of both N-gram character features from FastText and contextual representations from bidirectional encoder representations from transformers. This combination proves to be successful, as evidenced by the good results obtained from evaluating the model on the Arabic TimeML corpus. Our results show that combining both contextual and N-gram representations outperforms the traditional skip-gram model.

How a Deep Contextualized Representation and Attention Mechanism Justifies Explainable Cross-Lingual Sentiment Analysis

The number of applications in sentiment analysis is growing daily, and research in this field is increasing. Despite the rapid growth of data sources in English, low-resource languages suffer from a lack of data for accurate training models. Moreover, users cannot trust such systems without explaining the output. In this study, we propose a cross-lingual deep neural model to improve the accuracy of sentiment analysis for low-resource languages while providing an explainable description of the predictions. The proposed model contains a word representation model where we use XLM-RoBERTa, a pre-trained contextualized transformer-based cross-lingual language model, and a long short-term memory network together with an attention mechanism that helps improve the explainability of the model and detect the informative words that impact text polarity. Our experiments show the superiority of the proposed model compared to the state-of-the-art mono-lingual techniques and cross-lingual models. The results show 0.55% improvement compared to the cross-lingual sentiment analysis proposed by Ghasemi et al. and 15.08% improvement compared to the mono-lingual contextualized sentiment analysis. Moreover, we achieve 0.54% further improvement when using attention mechanisms for enhancing the model with explainability.

An eye movement study on the mechanisms of reading fluency development

Little is known about how word recognition processes, such as decoding, change when reading fluency improves during the school year. Such knowledge may have practical importance by determining which aspects of reading are most malleable at a certain age and reading level. The development of word-recognition subprocesses of third- and fourth-grade Finnish students (n = 81) with variable reading fluency was explored from longitudinal (6-month) text reading eye-tracking data. Generic development of the word recognition system was assessed from longitudinal changes in first fixation, average refixation durations and the number of first-pass fixations. The development of orthographic word representations and decoding was studied by examining the longitudinal changes in word frequency and word length effects, respectively. According to the results, the gain in reading fluency was mainly associated with decreases in first fixation and refixation durations. These decreases, in turn, inhibited the reduction in the number of fixations. However, students who could overcome this inhibitory effect, that is, by reading both with shorter fixation durations and with fewer fixations, developed most in reading fluency. The results seem to indicate that reading fluency development is driven by increased efficiency in representing letter strings in working memory. Over time, this development may lead to fewer fixations made into a word and, thus, more letters processed during each fixation.

Research of methods based on neural networks for the analysis of the tonality of the corps of the texts

The object of the study is methods based on neural networks for analyzing the tonality of a corpus of texts. To achieve the goal set in the work, it is necessary to solve the following tasks: study the theoretical material for learning deep neural networks and their features in relation to natural language processing; study the documentation of the Tensorflow library; develop models of convolutional and recurrent neural networks; to develop the implementation of linear and non-linear classification methods on bag of words and Word2Vec models; to compare the accuracy and other quality indicators of implemented neural network models with classical methods. Tensorboard is used for learning visualization. The work shows the superiority of classifiers based on deep neural networks over classical classification methods, even if the Word2Vec model is used for vector representations of words. The model of recurrent neural network with LSTM blocks has the highest accuracy for this corpus of texts.

Breakdown of category-specific word representations in a brain-constrained neurocomputational model of semantic dementia

The neurobiological nature of semantic knowledge, i.e., the encoding and storage of conceptual information in the human brain, remains a poorly understood and hotly debated subject. Clinical data on semantic deficits and neuroimaging evidence from healthy individuals have suggested multiple cortical regions to be involved in the processing of meaning. These include semantic hubs (most notably, anterior temporal lobe, ATL) that take part in semantic processing in general as well as sensorimotor areas that process specific aspects/categories according to their modality. Biologically inspired neurocomputational models can help elucidate the exact roles of these regions in the functioning of the semantic system and, importantly, in its breakdown in neurological deficits. We used a neuroanatomically constrained computational model of frontotemporal cortices implicated in word acquisition and processing, and adapted it to simulate and explain the effects of semantic dementia (SD) on word processing abilities. SD is a devastating, yet insufficiently understood progressive neurodegenerative disease, characterised by semantic knowledge deterioration that is hypothesised to be specifically related to neural damage in the ATL. The behaviour of our brain-based model is in full accordance with clinical data—namely, word comprehension performance decreases as SD lesions in ATL progress, whereas word repetition abilities remain less affected. Furthermore, our model makes predictions about lesion- and category-specific effects of SD: our simulation results indicate that word processing should be more impaired for object- than for action-related words, and that degradation of white matter should produce more severe consequences than the same proportion of grey matter decay. In sum, the present results provide a neuromechanistic explanatory account of cortical-level language impairments observed during the onset and progress of semantic dementia.

Text sentiment analysis based on BERT-CBLBGA

Current word vector techniques can generate effective word representations from massive text data, and different deep learning models can extract various textual semantic and grammatical features from different perspectives and levels. To extract richer text sentiment features, we propose a fused model of text sentiment classification called BERT-CBLBGA, which combines BERT with multiple feature extraction deep learning models. First, BERT is utilized to generate word embedding of the text, and the obtained feature representations are then fed into TextCNN, BiLSTM, and BiGRU to extract local and global sentiment features of the text. Then we use self-attention to fuse the output of TextCNN, BiLSTM, and BiGRU to enhance the ability to extract text sentiment classification features. Finally, a classifier is employed to classify the extracted features. Experimental results on four different domain text datasets demonstrate that the BERT-CBLBGA outperforms the baseline models in terms of extracting text sentiment features and achieves the best classification performance.

ALBERT-Based Personalized Educational Recommender System: Enhancing Students’ Learning Outcomes in Online Learning

Online learners must navigate vast educational resources to find materials that meet their needs. This study introduces an ALBERT-based personalized educational recommender system to improve student learning. ALBERT (A Lite BERT), an optimized variant of the BERT algorithm, captures contextualized word representations and understands the semantic meaning of learning resources, student profiles, and interactions. This study evaluates the ALBERT-based recommender system’s personalized learning recommendations. To assess learning outcomes, a diverse group of students from different educational domains is evaluated. Before and after the recommender system, academic performance, knowledge retention, and engagement are assessed. User satisfaction surveys assess recommendation quality, relevance, and user experience. The recommender system uses ALBERT’s model optimization to improve recommendation accuracy, learner engagement, and personalized learning. The evaluation shows the ALBERT-based personalized recommender system improves online learning outcomes. System-generated recommendations boost student engagement, knowledge retention, and academic performance. User satisfaction surveys show that the ALBERT-based system meets learners’ needs by providing relevant and high-quality recommendations. This research shows how advanced deep learning algorithms like ALBERT can improve personalized online learning. ALBERT’s optimized training and inference speeds up the recommender system’s scalability. This empowers learners to access tailored and high-quality educational resources, maximizing their learning outcomes and potential in online learning.