Pair Encoding Research Articles

Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting

This work proposes a multimodal Hindi-to-English machine translation on a news corpus by integrating multiple input modalities. The experimental dataset comprises of an image from a news article and its caption in the English-Hindi language. The parallel English-Hindi dataset is constructed by translating the image caption from English-language news articles into Hindi. Unlike standard images used for NLP tasks, images in news articles frequently depict significant events, locations, or individuals. The caption provides a more extensive description of the image, resulting in the inclusion of multiple identified entities. Our methodology consists of encoders for each input modality, allowing for simultaneous consideration of both text and image. We undertake two tests to examine how effectively the image content improves MT systems when 1) the entire image is considered first and 2) the largest object recognized in the image is considered. We employ Byte Pair Encoding (BPE) to represent the text, and the visual attributes are extracted from the image with VGG-19, a pre-trained CNN model. In terms of BLEU and chrF, our Multimodal Machine Translation (MMT) systems are superior than the baseline unimodal NMT system. Our MMT system outperforms the NMT model by +1.8 BLEU and +0.03 chrF.

IMPLEMENTASI ALGORITMA BYTE PAIR ENCODING UNTUK KOMPRESI FILE

ABSTRACT Data compression is a field that focuses on forming a small output file from a large file. The need for compression has given birth to several methods that can be implemented in communication activities on the network as well as on storage activities. The byte pair encoding algorithm is a compression algorithm that takes advantage of the emergence of repeated or frequent character byte pairs. The repeating character byte pairs are in the data and are substituted with a wildcard byte where the wildcard byte is not in the character byte series of the file. One of the well-known compression methods is the byte pair encoding compression method. The implementation of the system uses the Visual Basic 2010 programming language and the method used in this research is an applied research method. 
 Keywords: Compression, Byte Pair Encoding, Document.

Blind key-generation attribute-based encryption for general predicates

Attribute-based encryption (ABE) is a form of public-key encryption that allows fine-grained access control on encrypted data. Blind key-generation (BKG) attribute-based encryption (Rial, Designs, Codes and Cryptography 2016) is a variant in which the master authority issues secret keys without learning any information about the attributes associated to them. This extra functionality makes it an appealing building block for several applications. In this work, we extend the generic framework of ABE based on pair encodings (Attrapadung, Eurocrypt 2014) to support blind key-generation. In particular, we define two new notions of pair encodings that we coin BKG-compatible and algebraic pair encoding. We show that every encoding satisfies the former without loss of generality, whereas the latter is satisfied by all existing pair encodings from the literature. We then show how to enhance any ABE based on a BKG-compatible pair encoding to achieve honest-but-curious blind key-generation. In the case of algebraic encodings, our protocol admits a very efficient version, secure against malicious parties. The main advantage of our work is generality. Our protocol is designed over the recent and most advanced modular frameworks of ABE that can handle a rich variety of predicates.

Graph convolutional network based virus-human protein-protein interaction prediction for novel viruses

Computational identification of human-virus protein-protein interactions (PHIs) is a worthwhile step towards understanding infection mechanisms. Analysis of the PHI networks is important for the determination of pathogenic diseases. Prediction of these interactions is a popular problem since experimental detection of PHIs is both time-consuming and expensive. The available methods use biological features like amino acid sequences, molecular structure, or biological activities for prediction. Recent studies show that the topological properties of proteins in protein-protein interaction (PPI) networks increase the performance of the predictions. The basic network projections, random-walk-based models, or graph neural networks are used for generating topologically enriched (hybrid) protein embeddings. In this study, we propose a three-stage machine learning pipeline that generates and uses hybrid embeddings for PHI prediction. In the first stage, numerical features are extracted from the amino acid sequences using the Doc2Vec and Byte Pair Encoding method. The amino acid embeddings are used as node features while training a modified GraphSAGE model, which is an improved version of the graph convolutional network. Lastly, the hybrid protein embeddings are used for training a binary interaction classifier model that predicts whether there is an interaction between the given two proteins or not. The proposed method is evaluated with comprehensive experiments to test its functionality and compare it with the state-of-art methods. The experimental results on the benchmark dataset prove the efficiency of the proposed model by having a 3–23% better area under curve (AUC) score than its competitors.

Human Evaluation of English–Irish Transformer-Based NMT

In this study, a human evaluation is carried out on how hyperparameter settings impact the quality of Transformer-based Neural Machine Translation (NMT) for the low-resourced English–Irish pair. SentencePiece models using both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations in model architectures included modifying the number of layers, evaluating the optimal number of heads for attention and testing various regularisation techniques. The greatest performance improvement was recorded for a Transformer-optimized model with a 16k BPE subword model. Compared with a baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model demonstrated a BLEU score improvement of 7.8 points. When benchmarked against Google Translate, our translation engines demonstrated significant improvements. Furthermore, a quantitative fine-grained manual evaluation was conducted which compared the performance of machine translation systems. Using the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation of the error types generated by an RNN-based system and a Transformer-based system was explored. Our findings show the best-performing Transformer system significantly reduces both accuracy and fluency errors when compared with an RNN-based model.

Compact N-gram Language Models for Armenian

Applications such as speech recognition and machine translation use language models to select the most likely translation among many hypotheses. For on-device applications, inference time and model size are just as important as performance. In this work, we explored the fastest family of language models: the N-gram models for the Armenian language. In addition, we researched the impact of pruning and quantization methods on model size reduction. Finally, we used Bye Pair Encoding to build a subword language model. As a result, we obtained a compact (100 MB) subword language model trained on massive Armenian corpora.

A comparative study of neural machine translation models for Turkish language

Machine translation (MT) is an important challenge in the fields of Computational Linguistics. In this study, we conducted neural machine translation (NMT) experiments on two different architectures. First, Sequence to Sequence (Seq2Seq) architecture along with a variation that utilizes attention mechanism is performed on translation task. Second, an architecture that is fully based on the self-attention mechanism, namely Transformer, is employed to perform a comprehensive comparison. Besides, the contribution of employing Byte Pair Encoding (BPE) and Gumbel Softmax distributions are examined for both architectures. The experiments are conducted on two different datasets: TED Talks that is one of the popular benchmark datasets for NMT especially among morphologically rich languages like Turkish and WMT18 News dataset that is provided by The Third Conference on Machine Translation (WMT) for shared tasks on various aspects of machine translation. The evaluation of Turkish-to-English translations’ results demonstrate that the Transformer model with combination of BPE and Gumbel Softmax achieved 22.4 BLEU score on TED Talks and 38.7 BLUE score on WMT18 News dataset. The empirical results support that using Gumbel Softmax distribution improves the quality of translations for both architectures.

Retracted] Design of Intelligent Recognition English Translation Model Based on Deep Learning

Nowadays, the intercommunication and translation of global languages has become an indispensable condition for friendly communication among human beings around the world. The advancement of computer technology developed the machine translation from academic research to industrial applications. Additionally, a new and popular branch of machine learning is deep learning which has achieved excellent results in research fields such as natural language processing. This paper improved the performance of machine translation based on deep learning network and studied the intelligent recognition of English‐Chinese machine translation models. This research mainly focused on solving out‐of‐vocabulary (OOV) problem of machine translation on unregistered words and rare words. Moreover, it combined stemming technology and data compression algorithm Byte Pair Encoding (BPE) and proposed a different subword‐based word sequence segmentation method. Using this method, the English text is segmented into word sequences composed of subword units, and, at the same time, the Chinese text is segmented into character sequences composed of Chinese characters using unigram. Secondly, the current research also prevented the decoder from experiencing incomplete translation. Furthermore, it adopted a deep‐attention mechanism that can improve the decoder's ability to obtain context information. Inspired by the traditional attention calculation process, this work uses a two‐layer calculation structure in the improved attention to focus on the connection between the context vectors at different moments of the decoder. Based on the neural machine translation model Google Neural Machine Translation (GNMT), this paper conducted experimental analysis on the above improved methods on three different scale datasets. Experimental results verified that the improved method can solve OOV problem and improve accuracy of model translation.

GPU based Image Compression with Scaling-Hard-Threshold Using HWT (Haar-Wavelet-Transform) + RL (Run-Length) Pair Encoding

In 5G MASSIVE MIMO the main problem for better video communication is of transmission bandwidth issues which in turn reflects in video conferencing, Mobile-learning, gaming, graphics applications etc. However, in regard to a versatile framework i.e. a computerized camera with web and correspondence office, at that point the transmission capacity for correspondence and additionally stockpiling are of genuine concern. Tremendous valuable data must be put away and recovered proficiently for down to earth purposes. This is the novel methodology that scaling technique has proposed for 2D Haar wavelet transform based compression of picture utilizing parameter CR- ratio of Compression with PSNR improvement, with maintaining picture quality, less bandwidth usage with better compression ratio which increase by increasing speed with the help of NVIDIA GPU.

Tag-based ABE in prime-order groups via pair encoding

Tag-based ABE in prime-order groups via pair encoding

Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition.

With the rapid development of speech assistants, adapting server-intended automatic speech recognition (ASR) solutions to a direct device has become crucial. For on-device speech recognition tasks, researchers and industry prefer end-to-end ASR systems as they can be made resource-efficient while maintaining a higher quality compared to hybrid systems. However, building end-to-end models requires a significant amount of speech data. Personalization, which is mainly handling out-of-vocabulary (OOV) words, is another challenging task associated with speech assistants. In this work, we consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate, embodied in Babel Turkish and Babel Georgian tasks. We propose a method of dynamic acoustic unit augmentation based on the Byte Pair Encoding with dropout (BPE-dropout) technique. The method non-deterministically tokenizes utterances to extend the token’s contexts and to regularize their distribution for the model’s recognition of unseen words. It also reduces the need for optimal subword vocabulary size search. The technique provides a steady improvement in regular and personalized (OOV-oriented) speech recognition tasks (at least 6% relative word error rate (WER) and 25% relative F-score) at no additional computational cost. Owing to the BPE-dropout use, our monolingual Turkish Conformer has achieved a competitive result with 22.2% character error rate (CER) and 38.9% WER, which is close to the best published multilingual system.

Experience of neural machine translation between Indian languages

In this paper we explore neural machine translation (NMT) for Indian languages. Reported work on Indian language Statistical Machine Translation (SMT) demonstrated good performance within the Indo-Aryan family, but relatively poor performance within the Dravidian family as well as between the two families. Interestingly, by common observation NMT generates more fluent output than SMT. This led us to investigate NMT’s potential for translation involving Indian languages. The current practice in NMT is to train the models with subword units. Among subwording methods, byte pair encoding (BPE) is a popular choice. We conduct extensive experiments with BPE-based NMT models for Indian languages. An interesting outcome of our study is the finding that the optimal value for BPE merge for Indian language pairs seems to be falling in the range of 0–5000 which is fairly low compared to that observed for European Languages. Additionally, we apply other techniques such as phrase table injection and linguistic feature based enhancements on corpora, plus BERT augmented NMT to boost performance. To the best of our knowledge, this is the first comprehensive study on Indian language NMT (ILNMT) covering major languages in India. As an empirical paper, we expect this work could serve as a benchmark for ILNMT research.

SMILES Pair Encoding: A Data-Driven Substructure Tokenization Algorithm for Deep Learning.

Simplified molecular input line entry system (SMILES)-based deep learning models are slowly emerging as an important research topic in cheminformatics. In this study, we introduce SMILES pair encoding (SPE), a data-driven tokenization algorithm. SPE first learns a vocabulary of high-frequency SMILES substrings from a large chemical dataset (e.g., ChEMBL) and then tokenizes SMILES based on the learned vocabulary for the actual training of deep learning models. SPE augments the widely used atom-level tokenization by adding human-readable and chemically explainable SMILES substrings as tokens. Case studies show that SPE can achieve superior performances on both molecular generation and quantitative structure-activity relationship (QSAR) prediction tasks. In particular, the SPE-based generative models outperformed the atom-level tokenization model in the aspects of novelty, diversity, and ability to resemble the training set distribution. The performance of SPE-based QSAR prediction models were evaluated using 24 benchmark datasets where SPE consistently either did match or outperform atom-level and k-mer tokenization. Therefore, SPE could be a promising tokenization method for SMILES-based deep learning models. An open-source Python package SmilesPE was developed to implement this algorithm and is now freely available at https://github.com/XinhaoLi74/SmilesPE.

Prediction and Analysis of Protein Ubiquitin Sites in the Model Plant A. thaliana

Ubiquitin is an important type of protein after translational modification. Ubiquitin has the ability to take part in several cellular regulations among several biological processions. At the same time, ubiquitin plays key roles in the enzymatic process. So as to construct the new tool to classify the ubiquitin amino acid residues, we employed the random forest model to classify the ubiquitin sites utilizing the experimentally identified ubiquitinated protein sequences of A. thaliana. More detailed, we utilized the k-spaced amino acid pair (CKSAAP) encoding and binary encoding to deal with the potential protein segments. The proposed tools can obtain 72.83% in Sp, 72.42% in Sn, 72.63% in Acc, and 0.4525 in MCC. With these performances, such tools can obtain the available results in the dataset of Arabidopsis.

Is neural always better? SMT versus NMT for Dutch text normalization

Social media texts have become one of the most used forms of written language and a valuable source of information for companies. However, despite the wealth of information hidden in social media language for profiling, information extraction and sentiment analysis purposes, this user-generated content is also characterized by non-standard language. This makes it difficult to process using standard NLP techniques as these are typically trained on standard text. Text normalization is often used as a preprocessing step to overcome this problem. In this work, we take a machine translation perspective on text normalization and investigate both statistical and neural methods. We perform text normalization in a low-resource scenario and report both SMT and NMT experiments for three different Dutch user-generated text types: tweets, message board posts and text messages. Furthermore, we look into overcoming the over-normalization problem in NMT using different techniques, viz. teacher forcing, copy mechanism and byte pair encoding. Our results reveal that even though the SMT approach obtains the best results, the NMT system using CopyNet shows promising results even in this low-resource setting, solving more normalization issues than SMT although not solving all the over-normalization problems.

Memory Reactivation during Learning Simultaneously Promotes Dentate Gyrus/CA2,3 Pattern Differentiation and CA1 Memory Integration.

Events that overlap with previous experience may trigger reactivation of existing memories. However, such reactivation may have different representational consequences within the hippocampal circuit. Computational theories of hippocampal function suggest that dentate gyrus and CA2,3 (DG/CA2,3) are biased to differentiate highly similar memories, whereas CA1 may integrate related events by representing them with overlapping neural codes. Here, we tested whether the formation of differentiated or integrated representations in hippocampal subfields depends on the strength of memory reactivation during learning. Human participants of both sexes learned associations (AB pairs, either face-shape or scene-shape), and then underwent fMRI scanning while they encoded overlapping associations (BC shape-object pairs). Both before and after learning, participants were also scanned while viewing indirectly related elements of the overlapping memories (A and C images) in isolation. We used multivariate pattern analyses to measure reactivation of initial pair memories (A items) during overlapping pair (BC) learning, as well as learning-related representational change for indirectly related memory elements in hippocampal subfields. When prior memories were strongly reactivated during overlapping pair encoding, DG/CA2,3 and subiculum representations for indirectly related images (A and C) became less similar, consistent with pattern differentiation. Simultaneously, memory reactivation during new learning promoted integration in CA1, where representations for indirectly related memory elements became more similar after learning. Furthermore, memory reactivation and subiculum representation predicted faster and more accurate inference (AC) decisions. These data show that reactivation of related memories during new learning leads to dissociable coding strategies in hippocampal subfields, in line with computational theories.SIGNIFICANCE STATEMENT The flexibility of episodic memory allows us to remember both the details that differentiate similar events and the commonalities among them. Here, we tested how reactivation of past experience during new learning promotes formation of neural representations that might serve these two memory functions. We found that memory reactivation during learning promoted formation of differentiated representations for overlapping memories in the dentate gyrus/CA2,3 and subiculum subfields of the hippocampus, while simultaneously leading to the formation of integrated representations of related events in subfield CA1 Furthermore, memory reactivation and subiculum representation predicted success when inferring indirect relationships among events. These findings indicate that memory reactivation is an important learning signal that influences how overlapping events are represented within the hippocampal circuit.

A Neural-Network-Based Approach to Chinese–Uyghur Organization Name Translation

The recognition and translation of organization names (ONs) is challenging due to the complex structures and high variability involved. ONs consist not only of common generic words but also names, rare words, abbreviations and business and industry jargon. ONs are a sub-class of named entity (NE) phrases, which convey key information in text. As such, the correct translation of ONs is critical for machine translation and cross-lingual information retrieval. The existing Chinese–Uyghur neural machine translation systems have performed poorly when applied to ON translation tasks. As there are no publicly available Chinese–Uyghur ON translation corpora, an ON translation corpus is developed here, which includes 191,641 ON translation pairs. A word segmentation approach involving characterization, tagged characterization, byte pair encoding (BPE) and syllabification is proposed here for ON translation tasks. A recurrent neural network (RNN) attention framework and transformer are adapted here for ON translation tasks with different sequence granularities. The experimental results indicate that the transformer model not only outperforms the RNN attention model but also benefits from the proposed word segmentation approach. In addition, a Chinese–Uyghur ON translation system is developed here to automatically generate new translation pairs. This work significantly improves Chinese–Uyghur ON translation and can be applied to improve Chinese–Uyghur machine translation and cross-lingual information retrieval. It can also easily be extended to other agglutinative languages.

Efficient fully CCA-secure predicate encryptions from pair encodings

<p style='text-indent:20px;'>Attrapadung (Eurocrypt 2014) proposed a generic framework for fully (adaptively) CPA-secure predicate encryption (PE) based on a new primitive, called <i>pair encodings</i>. Following the CCA conversions of Yamada et al. (PKC 2011, 2012) and Nandi et al. (ePrint Archive: 2015/457, AAECC 2018), one can have CCA-secure PE from CPA-secure PE if the primitive PE has either verifiability or delegation. These traditional approaches degrade the performance of the resultant CCA-secure PE scheme as compared to the primitive CPA-secure PE. As an alternative, we provide a direct fully secure CCA-construction of PE from the pair encoding scheme. This costs an extra computation of group element in encryption, three extra pairing computations and one re-randomization of key in decryption as compared to the CPA-construction of Attrapadung.</p><p style='text-indent:20px;'>Recently, Blömer et al. (CT-RSA 2016) proposed a direct CCA-secure construction of predicate encryptions from pair encodings. Although they did not use the aforementioned traditional approaches, a sort of verifiability checking is still involved in the CCA-decryption. The number of pairing computations for this checking is nearly equal to the number of paring computations in CPA-decryption. Therefore, the performance of our direct CCA-secure PE is far better than Blömer et al.</p>

Emerging trends: Subwords, seriously?

AbstractSubwords have become very popular, but the BERTa and ERNIEb tokenizers often produce surprising results. Byte pair encoding (BPE) trains a dictionary with a simple information theoretic criterion that sidesteps the need for special treatment of unknown words. BPE is more about training (populating a dictionary of word pieces) than inference (parsing an unknown word into word pieces). The parse at inference time can be ambiguous. Which parse should we use? For example, “electroneutral” can be parsed as electron-eu-tral or electro-neutral, and “bidirectional” can be parsed as bid-ire-ction-al and bi-directional. BERT and ERNIE tend to favor the parse with more word pieces. We propose minimizing the number of word pieces. To justify our proposal, a number of criteria will be considered: sound, meaning, etc. The prefix, bi-, has the desired vowel (unlike bid) and the desired meaning (bi is Latin for two, unlike bid, which is Germanic for offer).

Ancient Korean Neural Machine Translation

Translation of the languages of ancient times can serve as a source for the content of various digital media and can be helpful in various fields such as natural phenomena, medicine, and science. Owing to these needs, there has been a global movement to translate ancient languages, but expert minds are required for this purpose. It is difficult to train language experts, and more importantly, manual translation is a slow process. Consequently, the recovery of ancient characters using machine translation has been recently investigated, but there is currently no literature on the machine translation of ancient Korean. This paper proposes the first ancient Korean neural machine translation model using a Transformer. This model can improve the efficiency of a translator by quickly providing a draft translation for a number of untranslated ancient documents. Furthermore, a new subword tokenization method called the Share Vocabulary and Entity Restriction Byte Pair Encoding is proposed based on the characteristics of ancient Korean sentences. This proposed method yields an increase in the performance of the original conventional subword tokenization methods such as byte pair encoding by 5.25 BLEU points. In addition, various decoding strategies such as n-gram blocking and ensemble models further improve the performance by 2.89 BLEU points. The model has been made publicly available as a software application.