Subword Levels Research Articles

PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention network

The expansion of the Internet has led to the widespread proliferation of malicious URLs, becoming a primary vector for cyber threats. Detecting malicious URLs is now essential for improving network security. The technological revolution spurred by pre-trained language models holds great promise for advancing the detection of malicious URLs. However, current research applying these models to URLs fails to address several crucial factors, including the lack of domain-specific adaptability, the omission of character-level information, and the neglect of both local detail extraction and low-order encoding information. In this paper, we propose PMANet, a pre-trained Language Model-Guided multi-level feature attention network, for addressing these issues. To facilitate a smooth transition of the pre-trained Transformer into the URL domain and to enable it to effectively capture information at both subword and character levels, we propose a post-training program that continues training the model on URLs using three self-supervised learning objectives: masked language model, noisy language model, and domain discrimination task. Subsequently, we develop a module to capture the output of each encoding layer, thus extracting hierarchical representations of URLs spanning from low-level to high-level. In addition, we propose a layer-wise attention mechanism that dynamically assigns weight coefficients to these feature layers based on their relevance. Finally, we apply spatial pyramid pooling to perform multi-scale down-sampling in order to obtain both local features and global context. PMANet achieves multifaceted integration in URL feature extraction, including capturing information at both the lexical and character levels, extracting features from low to high order, and discerning patterns at both global and local scales. We evaluate PMANet against challenging real-world scenarios, such as small-scale data, class imbalance, cross-dataset, adversarial attacks, and case studies on active malicious URLs. All experiments demonstrate that PMANet exhibits superiority over both the previous state-of-the-art pre-trained models and conventional deep learning models. Specifically, PMANet still achieves a 0.9941 AUC under adversarial attacks and correctly identifies all 20 actively malicious URLs in the case study. The code and data for our research are available at: https://github.com/Alixyvtte/Malicious-URL-Detection-PMANet.

HierTTS: Expressive End-to-End Text-to-Waveform Using a Multi-Scale Hierarchical Variational Auto-Encoder

End-to-end text-to-speech (TTS) models that directly generate waveforms from text are gaining popularity. However, existing end-to-end models are still not natural enough in their prosodic expressiveness. Additionally, previous studies on improving the expressiveness of TTS have mainly focused on acoustic models. There is a lack of research on enhancing expressiveness in an end-to-end framework. Therefore, we propose HierTTS, a highly expressive end-to-end text-to-waveform generation model. It deeply couples the hierarchical properties of speech with hierarchical variational auto-encoders and models multi-scale latent variables, at the frame, phone, subword, word, and sentence levels. The hierarchical encoder encodes the speech signal from fine-grained features into coarse-grained latent variables. In contrast, the hierarchical decoder generates fine-grained features conditioned on the coarse-grained latent variables. We propose a staged KL-weighted annealing strategy to prevent hierarchical posterior collapse. Furthermore, we employ a hierarchical text encoder to extract linguistic information at different levels and act on both the encoder and the decoder. Experiments show that our model performs closer to natural speech in prosody expressiveness and has better generative diversity.

Song authorship attribution: a lyrics and rhyme based approach

In this work, we apply authorship attribution to a large-scale corpus of song lyrics. As a sub-category of poetry, song lyrics embody cultural elements as well as stylistic attributes that are not present in prose. We draw attention to special characteristics such as repetitive sound patterns and rhyme based structures in lyrics that can be key to ownership, and present opportunities that cannot be employed for authorship attribution of other types of text such as tweets, emails, and blog posts. We first create a new balanced, large-scale data set of 12,000 song lyrics from 120 different artists. We propose CNN models for authorship attribution on this song lyric data set, in order to use structural information included in the lyrics, similarly to image classification. We conduct experiments at the character and sub-word levels that mostly reflect positional information. In addition, we use phoneme level features, which intrinsically involve attributes such as repetitions, rhyme, and meter, and represent elements unique to verse-based textual compositions. We attempt to discover idiosyncratic features and consequently author and genre associations by working with variants of CNN architectures that have been successfully used in other text classification domains. Our architecture choice results in a particular focus on lyric attributes residing in neighboring regions, since CNNs fail to apprehend long term textual dependencies. Finally, we empirically evaluate our results in comparison with the findings of previous test classification research from different domains.

An evaluation of recent neural sequence tagging models in Turkish named entity recognition

Named entity recognition (NER) is an extensively studied task that extracts and classifies named entities in a text. NER is crucial not only in downstream language processing applications such as relation extraction and question answering but also in large scale big data operations such as real-time analysis of online digital media content. Recent research efforts on Turkish, a less studied language with morphologically rich nature, have demonstrated the effectiveness of neural architectures on well-formed texts and yielded state-of-the art results by formulating the task as a sequence tagging problem. In this work, we empirically investigate the use of recent neural architectures (Bidirectional long short-term memory (BiLSTM) and Transformer-based networks) proposed for Turkish NER tagging in the same setting. Our results demonstrate that transformer-based networks which can model long-range context overcome the limitations of BiLSTM networks where different input features at the character, subword, and word levels are utilized. We also propose a transformer-based network with a conditional random field (CRF) layer that leads to the state-of-the-art result (95.95% f-measure) on a common dataset. Our study contributes to the literature that quantifies the impact of transfer learning on processing morphologically rich languages.

Cultural literacy in Chinese and Malay

The discussion suggests that there are many meaningful ways of making important messages beyond lexical meaning. Speakers of Malay and Chinese have been applying meaningful linguistic systems consistently by accentuating meaning at subword levels to communicate, preach, scorn, and interact with members in their respective speech communities. Such cultural-specific verbal styling in speech, however, is not accounted for in language standardization. The word is commonly accepted as the basic meaning unit conducive to language planning and lexicography. Nonetheless, effort to understand meaningful linguistic units below the level of a word stem is relevant to both cultural literacy and group cohesion. The failure to understand and participate in the production of meaning in subword level sounds is a form of cultural autism. This results in a loss of interactive essence which is conventionally signaled subtly in native speech style. By highlighting versatile meaning-making in the selected language data, the discussion complements the focus on current lexical expansion in standard written literacy.

Talking policy into practice: probing the debates around the effective teaching of early reading

This article examines the debates around the teaching of early reading and explores the possible consequences of policy recommendations for teachers' knowledge and understanding of effective pedagogic practices. These include teachers' abilities to adapt patterns of talk (pedagogic discourses) to respond to individual needs and the aims of instruction at any point in a lesson. The case study reported here involves a group of eight struggling readers, aged six to seven years, as they learn to decode text and comprehend meaning in an episode of shared reading. Integration of these dual purposes challenges current recommendations for discrete teaching of phonic skills prior to the development of comprehension strategies. Using lesson transcripts and drawing on notions of dialogic teaching, I argue that instruction at sub-word and text levels might be effectively integrated through appropriately judged changes of discourse that match different pedagogic purposes. Attention to the development of teachers' repertoire of talk forms would, I argue, eliminate the need for pedagogic prescription.

Reading with meaning: the contributions of meaning-related variables at the word and subword levels to early Chinese reading comprehension

This study examined the associations of three levels of meaning acquisition, i.e., whole word (vocabulary), morpheme (morphological awareness), and semantic radical (orthography-semantic awareness) to early Chinese reading comprehension among 164 Hong Kong Chinese primary school students, ages 7 and 8 years old, across 1 year. With time 1 word reading, phonological awareness and speeded naming controlled, morphological awareness was uniquely associated with concurrent and subsequent reading comprehension; orthography-semantic awareness uniquely explained concurrent reading comprehension at time 2. Together, the meaning acquisition variables explained between 2 and 6% unique variance in reading comprehension across time, underscoring the importance of acquisition of meaning for early reading comprehension among Chinese children.

Brand name suggestiveness: a Chinese language perspective

The selection of the right brand name is one of the most important marketing decisions and brand memorability has been proposed to be one of the key brand name selection criteria. Prior brand naming research has primarily dealt with word-level analysis in examining English brand name characteristics that are desirable for improving brand memorability. In this regard, this research examines the effects of suggestiveness at both the word and sub-word levels on consumer responses to Chinese brand names. Through two experiments, the findings demonstrate the ability of both character-level and radical-level suggestiveness in improving the memory for brand name and target claim. This research also extends the findings of previous studies on suggestiveness by demonstrating its favorable attitudinal effects. Based on these findings, theoretical and managerial implications are discussed.

Early Intervention for Reading Disabilities

Forty-eight children referred by teachers at the end of first grade for difficulty in reading were randomly assigned to three treatments, all of which modeled connections between written and spoken words but did not teach phonics rules, for eight half-hour individual tutoring sessions. The children were taught 48 words of varying orders of spelling-sound predictability (Venezky, 1995) using a whole-word method, for making connections between a word's name and its constituent letters; a subword method, for making connections between each color-coded spelling unit and its corresponding phonemes; or a combined whole-word and subword method. Regardless of the method used, children improved reliably on standardized reading measures and the taught words, showing that they could make connections between written and spoken words at the whole word and subword levels, even when rules were not taught. By posttest, the subword method showed a reliable advantage on a standardized test of real word reading. Knowledge of sounds associated with both multiletter and single-letter spelling units predicted reading achievement. Order of spelling-sound predictability (easy, moderate, difficult) was correlated with standardized measures of reading at pretest and posttest, and the magnitude of the relationship increased as a result of the intervention. Individual differences in verbal intelligence, rapid automatized naming, and phonological and orthographic skills predicted response to the intervention. Instructional implications of the results are discussed.

Teaching Spelling to Children with Specific Learning Disabilities: The Mind's Ear and Eye Beat the Computer or Pencil

Children with only spelling ( n=24) or handwriting and spelling disabilities ( n=24) were randomly assigned to a pencil or computer response mode. They were taught 48 words of varying orders of sound-spelling predictability using a method that emphasized hearing the word in the mind's ear and seeing the word in the mind's eye and making connections between the phonological and orthographic representations at the whole word and subword levels. Although no main effects were found for response mode, at posttest the pencil was superior to the computer for easy orders (single-letter spelling units with high sound-spelling predictability), but the computer was superior to the pencil for moderate and difficult orders (multiletter spelling units with intermediate or low sound-spelling predictability). Prior to and in response to treatment, children with handwriting and spelling problems spelled less well than children with only spelling problems. Multiletter spelling units of moderate or difficult orders of sound-spelling predictability explained unique increments of variance in spelling achievement, whereas single-letter spelling units of easy order of sound-spelling predictability did not. Based on these findings, instructional recommendations are to provide explicit instruction in the correspondence between sound and multiletter spelling units.

Comparisons of Beginning Literacy Programs: Alternative Paths to the Same Learning Outcome

University-based researchers consulted with first-grade teachers in a comparison of Integrated Reading-Writing (IR-W) and DISTAR programs in comparable urban schools with students “at risk” for literacy problems. Students in the two programs made comparable relative gains in achievement outcomes in word recognition from the beginning to the end of first grade. Level of orthographic and phonological awareness at the end of the first grade, but not instructional program, predicted level of reading achievement at the end of first grade. However, differences in processes underlying end-of-year achievement outcome were observed. Children in Integrated Reading-Writing tended to acquire orthographic-phonological connections at the whole word and subword levels, whereas children in DISTAR tended to acquire only subword connections. No differences were noted in developmental level of composition between the two instructional programs at the middle or end of first grade when coded compositions were available. The results are consistent with the notion that there is more than one way to learn to read and write.