Embedding Quality Research Articles

Character and Word Embeddings for Phishing Email Detection

Phishing attacks are among the most common malicious activities on the Internet. During a phishing attack, cybercriminals present themselves as a trusted organization or individual. Their goal is to lure people to enter their private information, such as passwords and bank card numbers, while believing that nothing malicious is happening. The attack often starts with a phishing email, which is an email that is very similar to a legitimate email, but usually contains links to malicious websites or uses some other techniques to mislead victims. To prevent phishing attacks, it is crucial to detect phishing emails and remove them from email inbox folders. In this paper, a neural network based phishing email detection model is proposed. In comparison to some earlier approaches, our model does not use manually engineered input features. It learns character and word embeddings directly from email texts, and uses them to extract local and global features using convolutional and recurrent layers, respectively. Our model is tested on the two commonly used datasets for phishing email detection, the SpamAssassin Public Corpus and Nazario Phishing Corpus, and it achieves an accuracy of 99.81 % and F_1-score of 99.74 %, which is on par or better than the current state-of-the-art approaches.

A Chinese Named Entity Recognition Model of Maintenance Records for Power Primary Equipment Based on Progressive Multitype Feature Fusion

Presently, the State Grid Corporation of China has accumulated a large amount of maintenance records for power primary equipment. Unfortunately, most of these records are unstructured data which lead to difficultly analyze and utilize them. The emergence of natural language processing technology and deep learning methods provide a solution for unstructured text data. This paper proposes a progressive multitype feature fusion model to recognize Chinese named entity of unstructured maintenance records for power primary equipment. Firstly, the textual characteristics and word separation difficulties of maintenance records are analyzed, then 7 main entity categories of power technical terms from unstructured maintenance records are chosen, and 3452 maintenance records are labeled by these categories, which is so called EPE‐MR training dataset. Secondly, the standard test reports, standard maintenance, and fault analysis reports for three types of power primary equipment (namely, main transformer, circuit breaker, and isolating switch) are employed as corpus to train character embedding in order to obtain certain words representation ability of maintenance records. After that, progressive multilevel radicals feature extraction module is designed to get detailed and fine semantic information in a hierarchical manner. Further, radicals feature representation and character embedding are concatenated and sent to BiLSTM module to extract contextual information in order to improve Chinese entity recognition ability. Moreover, CRF is introduced to handle the dependencies among prediction labels and to output the optimal prediction sequence, which can easily obtain structured data of maintenance records. Finally, comparative experiments on public MSRA dataset, China People’s Daily corpus, and EPE‐MR dataset are implemented, respectively, which show the effectiveness of the proposed method.

Definition modeling: literature review and dataset analysis

<abstract><p>Definition modeling, the task of generating a definition for a given term, is a relatively new area of research applied in evaluating word embeddings. Automatic generation of dictionary quality definitions has many applications in natural language processing, such as sentiment analysis, machine translation, and word sense disambiguation. Additionally, definition modeling is also helpful for evaluating the quality of word embeddings. As more research is done in this field, the need for a summary of different applications, approaches, and obstacles grows apparent. This review provides an overview of the current research in definition modeling and a list of future directions and trends.</p></abstract>

Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition

The variety and complexity of accents pose a huge challenge to robust Automatic Speech Recognition&#x00A0;(ASR). Some previous work has attempted to address such problems, however most of the current approaches either require prior knowledge about the target accent, or cannot handle unseen accents and accent-unspecific standard speech. In this work, we aim to improve multi-accent speech recognition in the end-to-end&#x00A0;(E2E) framework with a novel layer-wise adaptation architecture. Firstly, we propose a robust deep accent representation learning architecture to obtain accurate accent embedding, and some advanced schemes are designed to further boost the quality of accent embeddings, including phone posteriorgram&#x00A0;(PPG) feature, TTS based data augmentation in the training stage, test-time augmentation and multi-embedding fusion in the testing stage. Then, the layer-wise adaptation with accent embeddings is developed for fast accent adaptation in ASR, and two types of adapter layers are designed, including the gated adapter layer and multi-basis adapter layer. Compared to the usual two-pass adaptation, these adapter layers are injected between the ASR encoder layers to encode the accent information in ASR flexibly, and perform fast adaption on the corresponding speech accent. The experiments on Accent AESRC corpus show that the proposed deep accent representation learning can capture accurate accent knowledge, and get high performance on accent classification. The new layer-wise adaptation architecture with the accurate accent embedding outperforms the other traditional methods, and obtains consistent <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula>15&#x0025; relative word error rate&#x00A0;(WER) reduction on all kinds of testing scenarios, including seen accents, unseen accents and accent-unspecific standard speech.

Unsupervised Character Embedding Correction and Candidate Word Denoising

Inthis paper, we take Indonesian as the research object, and propose a multiple filter correction framework (MFCF). The main idea of MFCF is to remove noise from candidate words to increase the probability of correct words being selected. In MFCF, we use window search algorithm (WSA) to filter the candidate words in the dictionary. When searching for candidate words whose Levenshtein distance is 1, WSA reduces the candidate word search space by an average of 71%. When searching for candidate words whose Levenshtein distance is 2, the search space is reduced by an average of 55%. The reduction in search space has brought about an increase in search speed. When WSA searches for candidate words with Levenshtein distance equal to 1 and 2, the speed exceeds the current advanced search algorithm. A character vector-based candidate word scoring model (CWSM-CV) is also introduced in this paper. CWSM-CV is a simple but unsupervised method. In MFCF, we use CWSM-CV to filter the correct word in the candidate word list. Through exploring the feasibility of using word vector-based candidate word scoring model to score candidate words (CWSM-WV), we find the necessity of denoising the candidate word list and verified it with experiments. In order to apply this finding to the text correction, a new set of evaluation indicators are proposed to replace accuracy. Finally, we recommend that researchers who correct text in low-resource languages ​​make the model an open system and publish it for users to use. The system receives user feedback as new data to gradually reduce the negative impact of data volume.

Predicting User Preferences of Dimensionality Reduction Embedding Quality.

A plethora of dimensionality reduction techniques have emerged over the past decades, leaving researchers and analysts with a wide variety of choices for reducing their data, all the more so given some techniques come with additional hyper-parametrization (e.g., t-SNE, UMAP, etc.). Recent studies are showing that people often use dimensionality reduction as a black-box regardless of the specific properties the method itself preserves. Hence, evaluating and comparing 2D embeddings is usually qualitatively decided, by setting embeddings side-by-side and letting human judgment decide which embedding is the best. In this work, we propose a quantitative way of evaluating embeddings, that nonetheless places human perception at the center. We run a comparative study, where we ask people to select "good" and "misleading" views between scatterplots of low-dimensional embeddings of image datasets, simulating the way people usually select embeddings. We use the study data as labels for a set of quality metrics for a supervised machine learning model whose purpose is to discover and quantify what exactly people are looking for when deciding between embeddings. With the model as a proxy for human judgments, we use it to rank embeddings on new datasets, explain why they are relevant, and quantify the degree of subjectivity when people select preferred embeddings.

An integrated computer vision system for real-time monitoring and control of long-fiber embedded hydrogel 3D printing

3D printed hydrogel components have been used in a wide range of applications including tissue engineering and soft and biohybrid robotics. Advances in embedded printing, such as the Freeform Reversible Embedding of Suspended Hydrogels (FRESH), have further improved the geometric fidelity of 3D printed hydrogels using a fugitive support bath. Recently, it has been shown that the structural rigidity of hydrogels fabricated with embedded 3D printing can be significantly reinforced for a wider range of applications using the Long-fiber Embedded FRESH (LFE-FRESH) technique, an extension of FRESH by combing fiber embedding and hydrogel 3D printing. However, fibers are prone to buckling under compression load due to the high slenderness ratio and maintaining fiber stability during embedding is vital for LFE-FRESH. In this study, we introduce an integrated computer vision (CV) system for the continuous monitoring and control of LFE-FRESH, which actively adjusts the fiber embedding process in real-time by quantifying fiber deformation from the video data and controlling the fiber extrusion motor. Using the prototype, we demonstrated that the integrated CV system effectively prevents fiber buckling, corrects for over-extrusion during the LFE-FRESH process, and improves fiber embedding quality. Moreover, this technique was implemented with low-cost, mass-produced components and can be conveniently integrated with existing open-source LFE-FRESH software and hardware, which improves its accessibility and facilitates future adaptations for new research applications.

Novel Ransomware Hiding Model Using HEVC Steganography Approach

Ransomware is considered one of the most threatening cyberattacks. Existing solutions have focused mainly on discriminating ransomware by analyzing the apps themselves, but they have overlooked possible ways of hiding ransomware apps and making them difficult to be detected and then analyzed. Therefore, this paper proposes a novel ransomware hiding model by utilizing a block-based High-Efficiency Video Coding (HEVC) steganography approach. The main idea of the proposed steganography approach is the division of the secret ransomware data and cover HEVC frames into different blocks. After that, the Least Significant Bit (LSB) based Hamming Distance (HD) calculation is performed amongst the secret data’s divided blocks and cover frames. Finally, the secret data bits are hidden into the marked bits of the cover HEVC frame-blocks based on the calculated HD value. The main advantage of the suggested steganography approach is the minor impact on the cover HEVC frames after embedding the ransomware while preserving the histogram attributes of the cover video frame with a high imperceptibility. This is due to the utilization of an adaptive steganography cost function during the embedding process. The proposed ransomware hiding approach was heavily examined using subjective and objective tests and applying different HEVC streams with diverse resolutions and different secret ransomware apps of various sizes. The obtained results prove the efficiency of the proposed steganography approach by achieving high capacity and successful embedding process while ensuring the hidden ransomware’s undetectability within the video frames. For example, in terms of embedding quality, the proposed model achieved a high peak signal-to-noise ratio that reached 59.3 dB and a low mean-square-error of 0.07 for the examined HEVC streams. Also, out of 65 antivirus engines, no engine could detect the existence of the embedded ransomware app.

Embedded electronic medical record text data mining using neural network association classification algorithm

AbstractThe embedded electronic medical record (EMR) information is analysed through a neural network algorithm in the field of deep learning to mine the main sentence of EMR text. The pure convolutional neural network model (PCNN) based on pooled and non‐pooled analysis is constructed based on deep learning CNN to mine EMR text. An association classification algorithm based on neural network is proposed, which integrates classification rules into the neural network and realizes the rapid establishment of neural network structure and parameter setting. Character embedding is imported and combined with a sliding window. Then, CNN is adopted to classify the Chinese character tags of the neural network model. Finally, the model is applied to the word segmentation of the actual pregnant women's EMR text. The results show that the scores of the Biomedical Engineering Society and Bioelectronics in Peking University and Microsoft Reserved Partition datasets are between 0.9516 and 0.9684 and the performance is basically the same, which shows that the model has strong stability. The performance of the max‐pooling, n‐max pooling, and mean‐pooling are all between 0.8004 and 0.8634, but the above results are not as good as the model without pooling layer. For this new fast method to obtain neural network parameters, once the classification rules are given, the parameters in PCNN can be set quickly, and provide a practical basis for the development of deep learning in the medical field.

Phishing Website Detection Based on Deep Convolutional Neural Network and Random Forest Ensemble Learning.

Phishing has become one of the biggest and most effective cyber threats, causing hundreds of millions of dollars in losses and millions of data breaches every year. Currently, anti-phishing techniques require experts to extract phishing sites features and use third-party services to detect phishing sites. These techniques have some limitations, one of which is that extracting phishing features requires expertise and is time-consuming. Second, the use of third-party services delays the detection of phishing sites. Hence, this paper proposes an integrated phishing website detection method based on convolutional neural networks (CNN) and random forest (RF). The method can predict the legitimacy of URLs without accessing the web content or using third-party services. The proposed technique uses character embedding techniques to convert URLs into fixed-size matrices, extract features at different levels using CNN models, classify multi-level features using multiple RF classifiers, and, finally, output prediction results using a winner-take-all approach. On our dataset, a 99.35% accuracy rate was achieved using the proposed model. An accuracy rate of 99.26% was achieved on the benchmark data, much higher than that of the existing extreme model.

Extracting value from Brazilian Court decisions

We propose a methodology to extract value from Brazilian Court decisions to support judges and lawyers in their decision-making. We instantiate our methodology in one information system we have developed. Such system (i) extracts plaintiff’s legal claims and each specific provision on legal opinions enacted by lower and Appellate Courts, and (ii) connects each legal claim with the corresponding judicial provision. The information system presents the results through visualizations. Information Extraction for legal texts has been previously approached in the literature for different languages, using different methods. Our proposal is different from previous work, since our corpora comprise Brazilian lower and Appellate Court decisions, in which we look for a set of plaintiff’s legal claims and judicial provisions commonly judged by the Court. We use the following methods to tackle the information extraction tasks: Bidirectional Long Short-Term Memory network; Conditional Random Fields; and a combination of Bidirectional Long Short-Term Memory network and Conditional Random Fields. In addition to the well-known distributed representation of words in word embeddings, we use character-level representation of words in character embeddings. We have built three corpora – Kauane Insurance Report, Kauane Insurance Lower, and Kauane Insurance Upper – to train and evaluate the system, using public data from the State Court of Rio de Janeiro. Our methods achieved good quality for Kauane Insurance Lower and Kauane Insurance Upper, and promising results for Kauane Insurance Report.

Disaster related social media content processing for sustainable cities

The current study offers a hybrid convolutional neural networks (CNN) model that filters relevant posts and categorises them into several humanitarian classifications using both character and word embedding of textual content. The distinct embeddings for words and characters are used as input to the CNN model’s various channels. A hurricane, flood, and wildfire dataset are used to validate the proposed model. The model performed similarly across all datasets, with the F1-score ranging from 0.66 to 0.71. Because it uses existing social media posts and may be used as a layer with any social media, the model provides a sustainable solution for disaster analysis. With domain-specific training, the suggested approach can be used to locate useful information in other domains such as traffic accidents and civil unrest also.

Scalable robust graph embedding with Spark

Graph embedding aims at learning a vector-based representation of vertices that incorporates the structure of the graph. This representation then enables inference of graph properties. Existing graph embedding techniques, however, do not scale well to large graphs. While several techniques to scale graph embedding using compute clusters have been proposed, they require continuous communication between the compute nodes and cannot handle node failure. We therefore propose a framework for scalable and robust graph embedding based on the MapReduce model, which can distribute any existing embedding technique. Our method splits a graph into subgraphs to learn their embeddings in isolation and subsequently reconciles the embedding spaces derived for the subgraphs. We realize this idea through a novel distributed graph decomposition algorithm. In addition, we show how to implement our framework in Spark to enable efficient learning of effective embeddings. Experimental results illustrate that our approach scales well, while largely maintaining the embedding quality.

Efficient Estimate of Low-Frequency Words’ Embeddings Based on the Dictionary: A Case Study on Chinese

Obtaining high-quality embeddings of out-of-vocabularies (OOVs) and low-frequency words is a challenge in natural language processing (NLP). To efficiently estimate the embeddings of OOVs and low-frequency words, we propose a new method that uses the dictionary to estimate the embeddings of OOVs and low-frequency words. More specifically, the explanatory note of an entry in dictionaries accurately describes the semantics of the corresponding word. Naturally, we adopt the sentence representation model to extract the semantics of the explanatory note and regard the semantics as the embedding of the corresponding word. We design a new sentence representation model to encode sentences to extract the semantics from the explanatory notes of entries more efficiently. Based on the assumption that the higher quality of word embeddings will lead to better performance, we design an extrinsic experiment to evaluate the quality of low-frequency words’ embeddings. The experimental results show that the embeddings of low-frequency words estimated by our proposed method have higher quality. In addition, both intrinsic and extrinsic experiments show that our proposed sentence representation model can represent the semantics of sentences well.

Information extraction for different layouts of invoice images

ABSTRACT In the organization, they purchased goods or services from different suppliers and used invoice documents to confirm the payment. The invoice documents contained information that can be used for a business decision but, the process of information extraction required many resources to collect the data. The traditional way used template matching-based methods. This process identifies the parts on an image that match a predefined template and requires new manual annotation when processing the new image layout. Therefore, developing a system for robustly extracting entities from different layouts of invoices is necessary. Existing research applied deep learning and Name Entity Recognition (NER) for information extraction but, extracting invoice information was widely done in English and Chinese languages. In this study, we constructed a deep learning model using BiLSTM-CRF (Bidirectional Long Short-Term Memory-Conditional Random Fields) with word and character embedding for information extraction from different layouts of Thai invoice images. The model was evaluated by Semantic Evaluation at a full named-entity level. Our experimental results showed that this method can achieve a precision of 0.9557, recall of 0.9486, and F1-score of 0.9521 for the partial match; precision of 0.9329, recall of 0.9259, and F1-score of 0.9294 for the exact match and the result of the F1-score was significantly influenced by the quality of images and text result from Optical Character Recognition (OCR). Abbreviations: BERT: bidirectional encoder representations from transformers; BiLSTM: bidirectional long short-term memory; COR: correct; CRF: conditional random fields; CV: computer vision; ELMO: embeddings from language model; INC: incorrect; MIS: missing; MSE: mean squared error; MUC: message understanding conference; NER: named entity recognition; NLP: natural language processing; OCR: optical character recognition; PAR: partial; SemEval: semantic evaluation; SPU: spurius

Parallel training of knowledge graph embedding models

Knowledge graph embedding (KGE) models represent the entities and relations of a knowledge graph (KG) using dense continuous representations called embeddings. KGE methods have recently gained traction for tasks such as knowledge graph completion and reasoning as well as to provide suitable entity representations for downstream learning tasks. While a large part of the available literature focuses on small KGs, a number of frameworks that are able to train KGE models for large-scale KGs by parallelization across multiple GPUs or machines have recently been proposed. So far, the benefits and drawbacks of the various parallelization techniques have not been studied comprehensively. In this paper, we report on an experimental study in which we presented, re-implemented in a common computational framework, investigated, and improved the available techniques. We found that the evaluation methodologies used in prior work are often not comparable and can be misleading, and that most of currently implemented training methods tend to have a negative impact on embedding quality. We propose a simple but effective variation of the stratification technique used by PyTorch BigGraph for mitigation. Moreover, basic random partitioning can be an effective or even the best-performing choice when combined with suitable sampling techniques. Ultimately, we found that efficient and effective parallel training of large-scale KGE models is indeed achievable but requires a careful choice of techniques.

Clinical knowledge extraction via sparse embedding regression (KESER) with multi-center large scale electronic health record data

The increasing availability of electronic health record (EHR) systems has created enormous potential for translational research. However, it is difficult to know all the relevant codes related to a phenotype due to the large number of codes available. Traditional data mining approaches often require the use of patient-level data, which hinders the ability to share data across institutions. In this project, we demonstrate that multi-center large-scale code embeddings can be used to efficiently identify relevant features related to a disease of interest. We constructed large-scale code embeddings for a wide range of codified concepts from EHRs from two large medical centers. We developed knowledge extraction via sparse embedding regression (KESER) for feature selection and integrative network analysis. We evaluated the quality of the code embeddings and assessed the performance of KESER in feature selection for eight diseases. Besides, we developed an integrated clinical knowledge map combining embedding data from both institutions. The features selected by KESER were comprehensive compared to lists of codified data generated by domain experts. Features identified via KESER resulted in comparable performance to those built upon features selected manually or with patient-level data. The knowledge map created using an integrative analysis identified disease-disease and disease-drug pairs more accurately compared to those identified using single institution data. Analysis of code embeddings via KESER can effectively reveal clinical knowledge and infer relatedness among codified concepts. KESER bypasses the need for patient-level data in individual analyses providing a significant advance in enabling multi-center studies using EHR data.

A Neural N-Gram-Based Classifier for Chinese Clinical Named Entity Recognition

Clinical Named Entity Recognition (CNER) focuses on locating named entities in electronic medical records (EMRs) and the obtained results play an important role in the development of intelligent biomedical systems. In addition to the research in alphabetic languages, the study of non-alphabetic languages has attracted considerable attention as well. In this paper, a neural model is proposed to address the extraction of entities from EMRs written in Chinese. To avoid erroneous noise being caused by the Chinese word segmentation, we employ the character embeddings as the only feature without extra resources. In our model, concatenated n-gram character embeddings are used to represent the context semantics. The self-attention mechanism is then applied to model long-range dependencies of embeddings. The concatenation of the new representations obtained by the attention module is taken as the input to bidirectional long short-term memory (BiLSTM), followed by a conditional random field (CRF) layer to extract entities. The empirical study is conducted on the CCKS-2017 Shared Task 2 dataset to evaluate our method and the experimental results show that our model outperforms other approaches.

The first named entity recognizer in Maithili: Resource creation and system development

In this paper, we present our effort on the development of a Maithili Named Entity Recognition (NER) system. Maithili is one of the official languages of India, with around 50 million native speakers. Although various NER systems have been developed in several Indian languages, we did not find any openly available NER resource or system in Maithili. For the development, we manually annotated a Maithili NER corpus containing around 200K words. We prepared a baseline classifier using Conditional Random Fields (CRF). Then we ran many experiments using various recurrent neural networks (RNN). We collected larger raw corpus to obtain better word embedding and character embedding. In our experiments, we found, neural models are better than CRF; a CRF layer is effective for the prediction of the final output in the RNN models; character embedding is effective in Maithili language. We also investigated the effectiveness of gazetteer lists in neural models. We prepared a few gazetteer lists from various web resources and used those in the neural models. The incorporation of the gazetteer layer caused performance improvement. The final system achieved an f-measure of 91.6% with 94.9% precision and 88.53% recall.

A hybrid DNN-LSTM model for detecting phishing URLs.

Phishing is an attack targeting to imitate the official websites of corporations such as banks, e-commerce, financial institutions, and governmental institutions. Phishing websites aim to access and retrieve users’ important information such as personal identification, social security number, password, e-mail, credit card, and other account information. Several anti-phishing techniques have been developed to cope with the increasing number of phishing attacks so far. Machine learning and particularly, deep learning algorithms are nowadays the most crucial techniques used to detect and prevent phishing attacks because of their strong learning abilities on massive datasets and their state-of-the-art results in many classification problems. Previously, two types of feature extraction techniques [i.e., character embedding-based and manual natural language processing (NLP) feature extraction] were used in isolation. However, researchers did not consolidate these features and therefore, the performance was not remarkable. Unlike previous works, our study presented an approach that utilizes both feature extraction techniques. We discussed how to combine these feature extraction techniques to fully utilize from the available data. This paper proposes hybrid deep learning models based on long short-term memory and deep neural network algorithms for detecting phishing uniform resource locator and evaluates the performance of the models on phishing datasets. The proposed hybrid deep learning models utilize both character embedding and NLP features, thereby simultaneously exploiting deep connections between characters and revealing NLP-based high-level connections. Experimental results showed that the proposed models achieve superior performance than the other phishing detection models in terms of accuracy metric.