Word Segmentation Research Articles

Exploring the Festival Attendees’ Experiences on Social Media: A Study on the Guangzhou International Light Festival

This article explores festival attendees’ experiences on social media. Data on festival attendees’ posts on the Guangzhou International Light Festival were collected from TikTok. The Jieba Chinese word segmentation tool was applied to tokenize the texts and then classify the high-frequency words and phrases into five dimensions of festival experiences. The five dimensions of festival attendees’ experiences are cognitive, novel, affective, physical participation, and social interaction. A combination of the modularity algorithm and Gephi was used to cluster and visualize the festival attendees’ overall experiences and the different experiences in terms of gender and age. The results indicate that, in festival attendees’ overall experience, cognitive experience accounted for the largest proportion, followed by physical participation, and the proportion of social interaction, emotional experience, and novel experience was less. Moreover, the demographic characteristics of gender and age differ in the festival experience.

High-Speed Binary-to-Residue Converter Design Using 2-Bit Segmentation of the Input Word

In this paper a new approach to the design of the high-speed binary-to-residue converter is proposed that allows the attaining of high pipelining rates by eliminating memories used in modulo m generators. The converter algorithm uses segmentation of the input binary word into 2-bit segments. The use and effects of the input word segmentation for the synthesis of converters for five-bit moduli are presented. For the number represented by each segment, the modulo m reduction using a segment modulo m generator is performed. The use of 2-bit segments substantially reduces the hardware amount of the layer of input modulo m generators. The generated residues are added using the multi-operand modulo m adder based on the carry-save adder (CSA) tree, reduction of the number represented by the output CSA tree vectors to the 2m range and fast two-operand modulo m additions. Hardware amount and time delay analyses are also included.

SPELLING ERROR ANALYSIS IN WRITING RECOUNT TEXT BY THE NINTH-GRADE STUDENTS OF SMP NEGERI 1 SUMBAWA ACADEMIC YEAR 2022/2023

The aims of this study are to find out the types of spelling errors made by the ninth-grade students of SMPN 1 Sumbawa by determining students writing and to find out the causes of spelling error made by students. In addition, this study uses the theory from Bestgen and Granger (2011) in their journal entitled “Categorizing Spelling Error to Assess L2 Writing” that discussed about the category of spelling errors. Furthermore, this research applies descriptive qualitative in which the researchers collected the data from students writing tasks and found the spelling error of three words classification which are adjective, noun and verb. The researchers found six types of spelling errors which are addition, omission, substitution, transposition, word segmentation, and multiple error letter. In addition, there are three main causes of spelling error found such as the influence of Indonesian spelling, lack of vocabulary, and spelling difficulties

Review of techniques for morphological analysis in natural language processing

Natural language is a crucial tool to facilitate communication in our day-to-day activities. This can be achieved either in text or speech forms. Natural language processing (NLP) involves making computers understand and process natural language. NLP has enhanced the way humans interact with computers, from having computers use speech to talk to humans as well as having computers translate human speech. Apart from speech, computers also create and understand sentences in natural language in a process called morphological analysis. Morphological analysis is an important part in natural language processing, being applied as a preprocessing step in most NLP tasks. Morphological analysis consists of four subtasks, that is, lemmatization, part-of-speech (POS) tagging, word segmentation and stemming. In this paper, we explore in detail each of these tasks of morphological analysis. We then evaluate the techniques used in this NLP field. Finally, we give a summary of the results of each of these techniques.

Developmental consistency in the use of subphonemic information during real-time sentence processing

ABSTRACT Apparently homophonous sequences contain acoustic information that differentiates their meanings [Gahl. (2008). Time and thyme are not homophones: The effect of lemma frequency on word durations in spontaneous speech. Language, 84(3), 474–496; Quené. (1992). Durational cues for word segmentation in Dutch. Journal of Phonetics, 20(3), 331–350]. Adults use this information to segment embedded homophones [e.g. ham vs. hamster; Salverda et al. (2003). The role of prosodic boundaries in the resolution of lexical embedded in speech comprehension. Cognition, 90(1), 51–89] in fluent speech. Whether children also do this is unknown, as is whether listeners of any age use such information to disambiguate lexical homophones. In two experiments, 48 English-speaking adults and 48 English-speaking 7 to 10-year-old children viewed sets of four images and heard sentences containing phonemically identical sequences while their eye movements were continuously tracked. As in previous research, adults showed greater fixation of target meanings when the acoustic properties of an embedded homophone were consistent with the target than when they were consistent with the alternate interpretation. They did not show this difference for lexical homophones. Children’s behaviour was similar to that of adults, indicating that the use of subphonemic information in homophone processing is consistent over development.

Statistical word segmentation: Anchoring learning across contexts.

The present experiments were designed to assess infants' abilities to use syllable co-occurrence regularities to segment fluent speech across contexts. Specifically, we investigated whether 9-month-old infants could use statistical regularities in one speech context to support speech segmentation in a second context. Contexts were defined by different word sets representing contextual differences that might occur across conversations or utterances. This mimics the integration of information across multiple interactions within a single language, which is critical for language acquisition. In particular, we performed two experiments to assess whether a statistically segmented word could be used to anchor segmentation in a second, more challenging context, namely speech with variable word lengths. The results of Experiment 1 were consistent with past work suggesting that statistical learning may be hindered by speech with word-length variability, which is inherent to infants' natural speech environments. In Experiment 2, we found that infants could use a previously statistically segmented word to support word segmentation in a novel, challenging context. We also present findings suggesting that this ability was associated with infants' early word knowledge but not their performance on a cognitive development assessment.

A New Text-Mining–Bayesian Network Approach for Identifying Chemical Safety Risk Factors

The frequent occurrence of accidents in the chemical industry has caused serious economic loss and negative social impact. The chemical accident investigation report is of great value for analyzing the risk factors involved. However, traditional manual analysis is time-consuming and labor-intensive, while existing keyword extraction methods still need to be improved. This study aims to propose an improved text-mining method to analyze a large number of chemical accident reports. A workflow was designed for building and updating lexicons of word segmentation. An improved keyword extraction algorithm was proposed to extract the top 100 keywords from 330 incident reports. A total of 51 safety risk factors was obtained by standardizing these keywords. In all, 294 strong association rules were obtained by Apriori. Based on these rules, a Bayesian network was built to analyze safety risk factors. The mean accuracy and mean recall of the BM25 model in the comparison experiments were 10.5% and 14.38% higher than those of TF-IDF, respectively. The results of association-rule mining and Bayesian network analysis can clearly demonstrate the interrelationship between the safety risk factors. The methodology of this study can quickly and efficiently extract key information from incident reports which can provide managers with new insights and suggestions.

Moving toward a standardized diagnostic statement of pituitary adenoma using an information extraction model: a real-world study based on electronic medical records

PurposeDiagnostic statements for pituitary adenomas (PAs) are complex and unstandardized. We aimed to determine the most commonly used elements contained in the statements and their combination patterns and variations in real-world clinical practice, with the ultimate goal of promoting standardized diagnostic recording and establishing an efficient element extraction process.MethodsPatient medical records from 2012 to 2020 that included PA among the first three diagnoses were included. After manually labeling the elements in the diagnostic texts, we obtained element types and training sets, according to which an information extraction model was constructed based on the word segmentation model “Jieba” to extract information contained in the remaining diagnostic texts.ResultsA total of 576 different diagnostic statements from 4010 texts of 3770 medical records were enrolled in the analysis. The first ten diagnostic elements related to PA were histopathology, tumor location, endocrine status, tumor size, invasiveness, recurrence, diagnostic confirmation, Knosp grade, residual tumor, and refractoriness. The automated extraction model achieved F1-scores that reached 100% for all ten elements in the second round and 97.3–100.0% in the test set consisting of an additional 532 diagnostic texts. Tumor location, endocrine status, histopathology, and tumor size were the most commonly used elements, and diagnoses composed of the above elements were the most frequent. Endocrine status had the greatest expression variability, followed by Knosp grade. Among all the terms, the percentage of loss of tumor size was among the highest (21%). Among statements where the principal diagnoses were PAs, 18.6% did not have information on tumor size, while for those with other diagnoses, this percentage rose to 48% (P < 0.001).ConclusionStandardization of the diagnostic statement for PAs is unsatisfactory in real-world clinical practice. This study could help standardize a structured pattern for PA diagnosis and establish a foundation for research-friendly, high-quality clinical information extraction.

Foveal Splitting of Compounds and Pseudocompounds using Anaglyphs

We investigated the nature of linguistic codes at the earliest moments of visual word recognition by employing red-blue anaglyph glasses to split compounds (e.g., snowball) and pseudocompounds (e.g., cartridge) along the fovea’s vertical meridian. By hypothesis, anaglyphs allowed us to manipulate the role of retinotopic projections during visual processing, whereby word segments were projected to the right or left hemisphere via the ipsilateral or contralateral pathways. Furthermore, this technique allows the embedded words (i.e., constituents) of compounds and pseudocompounds to be presented independently in the visual word form area. Seventy-one participants performed a visual masked lexical decision task, where they made word-nonword judgements. Stimuli were presented for 133 milliseconds either completely in black (both pathways), red/blue (ipsilateral pathways) and blue/red (contralateral pathways). The compounds and pseudocompounds were split into their constituents (legal split) or one letter to the left or to right of the morpheme boundary (illegal split). Compounds varied in the degree of semantic transparency, either transparent (T) or opaque (O), of constituents (e.g., TT, snowball; OT, crowbar; TO, jailbird; OO, hogwash). The accuracy and response times (RTs) to the lexical decision task were analyzed using linear mixed effects models. Results suggest an advantage for words processed through both visual pathways than when they are projected contralaterally (both in accuracy and RTs) and ipsilaterally (in accuracy only). Furthermore, legally split stimuli were judged faster and more accurately than illegal split stimuli regardless of word type. Responses to compounds were more accurate when compared to pseudocompounds. Taken together, these findings suggest that the early visual word recognition system is sensitive to the internal structure of compounds and pseudocompounds, but blind to the semantic contribution of their constituents.

Segmenting Speech by Mouth: The Role of Oral Prosodic Cues for Visual Speech Segmentation.

Adults are able to use visual prosodic cues in the speaker's face to segment speech. Furthermore, eye-tracking data suggest that learners will shift their gaze to the mouth during visual speech segmentation. Although these findings suggest that the mouth may be viewed more than the eyes or nose during visual speech segmentation, no study has examined the direct functional importance of individual features; thus, it is unclear which visual prosodic cues are important for word segmentation. In this study, we examined the impact of first removing (Experiment 1) and then isolating (Experiment 2) individual facial features on visual speech segmentation. Segmentation performance was above chance in all conditions except for when the visual display was restricted to the eye region (eyes only condition in Experiment 2). This suggests that participants were able to segment speech when they could visually access the mouth but not when the mouth was completely removed from the visual display, providing evidence that visual prosodic cues conveyed by the mouth are sufficient and likely necessary for visual speech segmentation.

Validation of a Thai artificial chatmate designed for cheering up the elderly during the COVID-19 pandemic

Background: The COVID-19 pandemic severely affected populations of all age groups. The elderly are a high-risk group and are highly vulnerable to COVID-19. Assistive software chatbots can enhance the mental health status of the elderly by providing support and companionship. The objective of this study was to validate a Thai artificial chatmate for the elderly during the COVID-19 pandemic and floods. Methods: Chatbot design includes the establishment of a dataset and emotional word vectors in which data consisting of emotional sentences were converted into the word vector form using a pre-trained word2vec model. A word vector was then input into a convolutional neural network (CNN) and trained until the model converges using sentence embedding and similarity word segmentation. Sentence vectors were generated by averaging each word vector using an averaged vector method. For approximate similarity matching, the Annoy library was used to create the indices in tree sorting. Data were collected from 22 elderly and assessed by the Post-Study System Usability Questionnaire (PSSUQ). Results: The study revealed that 72.73% of the respondents found the chatbot easy to learn and use, 63.64% of the respondents found the chatbot can autonomously determine the next course of action, and 59.09% of the respondents believed that troubleshooting guidelines were provided for overcoming errors. The accuracy of the chatbot providing a reasonable response is 56.20±13.99%. Conclusions: Most users were satisfied with the chatbot system. The proposed chatbot provided considerable essential insights into the development of assistance systems for the elderly during the coronavirus pandemic (COVID-19) and during the period of national disasters. The model can be expanded to other applications in the future.

Unsupervised Word Segmentation with Bi-directional Neural Language Model

We propose an unsupervised word segmentation model, in which for each unlabelled sentence sample, the learning objective is to maximize the generation probability of the sentence given its all possible segmentations. Such a generation probability can be factorized into the likelihood of each possible segment given the context in a recursive way. To capture both the long- and short-term dependencies, we propose to use a bi-directional neural language model to better extract the features of the segment’s context. Two decoding algorithms were also developed to combine the context features from both directions to generate the final segmentation at the inference time, which helps to reconcile word-boundary ambiguities. Experimental results show that our context-sensitive unsupervised segmentation model achieved state-of-the-art at different evaluation settings on various datasets for Chinese, and the comparable result for Thai.

Feature Screening for Interval-Valued Response with Application to Study Association between Posted Salary and Required Skills

It is important to quantify the differences in returns to skills using the online job advertisements data, which have attracted great interest in both labor economics and statistics fields. In this article, we study the relationship between the posted salary and the job requirements in online labor markets. There are two challenges to deal with. First, the posted salary is always presented in an interval-valued form, for example, 5k–10k yuan per month. Simply taking the mid-point or the lower bound as the alternative for salary may result in biased estimators. Second, the number of the potential skill words as predictors generated from the job advertisements by word segmentation is very large and many of them may not contribute to the salary. To this end, we propose a new feature screening method, Absolute Distribution Difference Sure Independence Screening (ADD-SIS), to select important skill words for the interval-valued response. The marginal utility for feature screening is based on the difference of estimated distribution functions via nonparametric maximum likelihood estimation, which sufficiently uses the interval information. It is model-free and robust to outliers. Numerical simulations show that the new method using the interval information is more efficient to select important predictors than the methods only based on the single points of the intervals. In the real data application, we study the text data of job advertisements for data scientists and data analysts in a major China’s online job posting website, and explore the important skill words for the salary. We find that the skill words like optimization, long short-term memory (LSTM), convolutional neural networks (CNN), collaborative filtering, are positively correlated with the salary while the words like Excel, Office, data collection, may negatively contribute to the salary. Supplementary materials for this article are available online.

No need to forget, just keep the balance: Hebbian neural networks for statistical learning

Language processing in humans has long been proposed to rely on sophisticated learning abilities including statistical learning. Endress and Johnson (E&J, 2021) recently presented a neural network model for statistical learning based on Hebbian learning principles. This model accounts for word segmentation tasks, one primary paradigm in statistical learning. In this discussion paper we review this model and compare it with the Hebbian model previously presented by Tovar and Westermann (T&W, 2017a; 2017b; 2018) that has accounted for serial reaction time tasks, cross-situational learning, and categorization paradigms, all relevant in the study of statistical learning. We discuss the similarities and differences between both models, and their key findings. From our analysis, we question the concept of “forgetting” in the model of E&J and their suggestion of considering forgetting as the critical ingredient for successful statistical learning. We instead suggest that a set of simple but well-balanced mechanisms including spreading activation, activation persistence, and synaptic weight decay, all based on biologically grounded principles, allow modeling statistical learning in Hebbian neural networks, as demonstrated in the T&W model which successfully covers learning of nonadjacent dependencies and accounts for differences between typical and atypical populations, both aspects that have not been fully demonstrated in the E&J model. We outline the main computational and theoretical differences between the E&J and T&W approaches, present new simulation results, and discuss implications for the development of a computational cognitive theory of statistical learning.

BioByGANS: biomedical named entity recognition by fusing contextual and syntactic features through graph attention network in node classification framework

BackgroundAutomatic and accurate recognition of various biomedical named entities from literature is an important task of biomedical text mining, which is the foundation of extracting biomedical knowledge from unstructured texts into structured formats. Using the sequence labeling framework and deep neural networks to implement biomedical named entity recognition (BioNER) is a common method at present. However, the above method often underutilizes syntactic features such as dependencies and topology of sentences. Therefore, it is an urgent problem to be solved to integrate semantic and syntactic features into the BioNER model.ResultsIn this paper, we propose a novel biomedical named entity recognition model, named BioByGANS (BioBERT/SpaCy-Graph Attention Network-Softmax), which uses a graph to model the dependencies and topology of a sentence and formulate the BioNER task as a node classification problem. This formulation can introduce more topological features of language and no longer be only concerned about the distance between words in the sequence. First, we use periods to segment sentences and spaces and symbols to segment words. Second, contextual features are encoded by BioBERT, and syntactic features such as part of speeches, dependencies and topology are preprocessed by SpaCy respectively. A graph attention network is then used to generate a fusing representation considering both the contextual features and syntactic features. Last, a softmax function is used to calculate the probabilities and get the results. We conduct experiments on 8 benchmark datasets, and our proposed model outperforms existing BioNER state-of-the-art methods on the BC2GM, JNLPBA, BC4CHEMD, BC5CDR-chem, BC5CDR-disease, NCBI-disease, Species-800, and LINNAEUS datasets, and achieves F1-scores of 85.15%, 78.16%, 92.97%, 94.74%, 87.74%, 91.57%, 75.01%, 90.99%, respectively.ConclusionThe experimental results on 8 biomedical benchmark datasets demonstrate the effectiveness of our model, and indicate that formulating the BioNER task into a node classification problem and combining syntactic features into the graph attention networks can significantly improve model performance.

Using a classification model for determining the value of liver radiological reports of patients with colorectal cancer.

Medical imaging is critical in clinical practice, and high value radiological reports can positively assist clinicians. However, there is a lack of methods for determining the value of reports. The purpose of this study was to establish an ensemble learning classification model using natural language processing (NLP) applied to the Chinese free text of radiological reports to determine their value for liver lesion detection in patients with colorectal cancer (CRC). Radiological reports of upper abdominal computed tomography (CT) and magnetic resonance imaging (MRI) were divided into five categories according to the results of liver lesion detection in patients with CRC. The NLP methods including word segmentation, stop word removal, and n-gram language model establishment were applied for each dataset. Then, a word-bag model was built, high-frequency words were selected as features, and an ensemble learning classification model was constructed. Several machine learning methods were applied, including logistic regression (LR), random forest (RF), and so on. We compared the accuracy between priori choosing pertinent word strings and our machine language methodologies. The dataset of 2790 patients included CT without contrast (10.2%), CT with/without contrast (73.3%), MRI without contrast (1.8%), and MRI with/without contrast (14.6%). The ensemble learning classification model determined the value of reports effectively, reaching 95.91% in the CT with/without contrast dataset using XGBoost. The logistic regression, random forest, and support vector machine also achieved good classification accuracy, reaching 95.89%, 95.04%, and 95.00% respectively. The results of XGBoost were visualized using a confusion matrix. The numbers of errors in categories I, II and V were very small. ELI5 was used to select important words for each category. Words such as "no abnormality", "suggest", "fatty liver", and "transfer" showed a relatively large degree of positive correlation with classification accuracy. The accuracy based on string pattern search method model was lower than that of machine learning. The learning classification model based on NLP was an effective tool for determining the value of radiological reports focused on liver lesions. The study made it possible to analyze the value of medical imaging examinations on a large scale.

Script-Independent Text Segmentation from Document Images

Document image analysis finds broad application in the digital world for the purpose of information retrieval. This includes optical character recognition (OCR), indexing of digital libraries, web image processing, etc. One of the important steps in this field is text segmentation. This segmentation becomes complicated for the documents containing text of uneven spacing and characters of varying font sizes. In this paper, script-independent text-line segmentation and word segmentation algorithms are presented. Fast marching method is used for text-line segmentation, whereas wavelet transform with connected components (CCs) labeling is used for word segmentation. Fast marching method is used as a region growing process that detects potential text-lines. For word segmentation, energy map is calculated using wavelet transform to create text-blocks. Both the proposed algorithms are evaluated on different databases containing documents of different scripts, where highest text-line and word segmentation accuracies of 98.9% and 99.1%, respectively, are obtained.

Analysis of album comments based on NetEase Cloud Music

This paper aims to explore the user's comment behavior and the user's emotional tendency towards each song in the album. In this paper, the comments in the Netease Cloud Music are taken as the analysis objects, and Python is used as a tool to crawl data, clean data, segment words, generate themes, and analyze emotions. Finally, visual analysis is carried out through R language.

Lexicon and attention-based named entity recognition for kiwifruit diseases and pests: A Deep learning approach.

Named Entity Recognition (NER) is a crucial step in mining information from massive agricultural texts, which is required in the construction of many knowledge-based agricultural support systems, such as agricultural technology question answering systems. The vital domain characteristics of Chinese agricultural text cause the Chinese NER (CNER) in kiwifruit diseases and pests to suffer from the insensitivity of common word segmentation tools to kiwifruit-related texts and the feature extraction capability of the sequence encoding layer being challenged. In order to alleviate the above problems, effectively mine information from kiwifruit-related texts to provide support for agricultural support systems such as agricultural question answering systems, this study constructed a novel Chinese agricultural NER (CANER) model KIWINER by statistics-based new word detection and two novel modules, AttSoftlexicon (Criss-cross attention-based Softlexicon) and PCAT (Parallel connection criss-cross attention), proposed in this paper. Specifically, new words were detected to improve the adaptability of word segmentation tools to kiwifruit-related texts, thereby constructing a kiwifruit lexicon. The AttSoftlexicon integrates word information into the model and makes full use of the word information with the help of Criss-cross attention network (CCNet). And the PCAT improves the feature extraction ability of sequence encoding layer through CCNet and parallel connection structure. The performance of KIWINER was evaluated on four datasets, namely KIWID (Self-annotated), Boson, ClueNER, and People's Daily, which achieved optimal F1-scores of 88.94%, 85.13%, 80.52%, and 92.82%, respectively. Experimental results in many aspects illustrated that methods proposed in this paper can effectively improve the recognition effect of kiwifruit diseases and pests named entities, especially for diseases and pests with strong domain characteristics.

The positive energy of netizens: development and application of fine-grained sentiment lexicon and emotional intensity model.

The outbreak of COVID-19 has led to a global health crisis and caused huge emotional swings. However, the positive emotional expressions, like self-confidence, optimism, and praise, that appear in Chinese social networks are rarely explored by researchers. This study aims to analyze the characteristics of netizens' positive energy expressions and the impact of node events on public emotional expression during the COVID-19 pandemic. First, a total of 6,525,249 Chinese texts posted by Sina Weibo users were randomly selected through textual data cleaning and word segmentation for corpus construction. A fine-grained sentiment lexicon that contained POSITIVE ENERGY was built using Word2Vec technology; this lexicon was later used to conduct sentiment category analysis on original posts. Next, through manual labeling and multi-classification machine learning model construction, four mainstream machine learning algorithms were selected to train the emotional intensity model. Finally, the lexicon and optimized emotional intensity model were used to analyze the emotional expressions of Chinese netizens. The results show that POSITIVE ENERGY expression accounted for 40.97% during the COVID-19 pandemic. Over the course of time, POSITIVE ENERGY emotions were displayed at the highest levels and SURPRISES the lowest. The analysis results of the node events showed after the outbreak was confirmed officially, the expressions of POSITIVE ENERGY and FEAR increased simultaneously. After the initial victory in pandemic prevention and control, the expression of POSITIVE ENERGY and SAD reached a peak, while the increase of SAD was the most prominent. The fine-grained sentiment lexicon, which includes a POSITIVE ENERGY category, demonstrated reliable algorithm performance and can be used for sentiment classification of Chinese Internet context. We also found many POSITIVE ENERGY expressions in Chinese online social platforms which are proven to be significantly affected by nod events of different nature.